Upstream Mercurial no longer always calls bookmark.write to serialize the
bookmarks. Now it sometimes goes through the transaction API which calls
bookmark._write.
Summary:
The sqlstrip command had a bug where a multidigit revision, like '5381' would be
treated like a string, sorted, and the first digit taken. So we always
sqlstripped one of the first revs in the repository.
This fixes that to treat rev like an integer, and removes the sort (which was a
legacy bit of code from when sqlstrip supported multiple integers).
Test Plan: Ran the tests.
Reviewers: sid0, pyd, davidsp
Differential Revision: https://phabricator.fb.com/D1580556
Summary: Mercurial rev f8c397b4362c changed the API for `phases.advanceboundary`. A different rev also added a benign 'activating bookmark' message.
Test Plan: Ran the tests.
Reviewers: durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D1561665
Tasks: 5170539
The previous change that allowed transactions that didn't touch revlogs was
broken. It allowed transactions like 'strip', etc. This reverts that.
To fix the actual issue we were trying to fix before, we now add a sql
transaction around local phase moves as part of exchange.
Recent upstream Mercurial changes have moved the phase cache into a transaction.
We don't sync phases as part of hgsql (everything is just public), so lets allow
transactions as long as they don't touch revlogs.
This was necessary because pushing between local repos was broken by this.
Summary:
Adds an 'hg sqlstrip' command that strips commits from the local repo and from
the database. Since this modifes the database, all other server hosts will
suddenly become out of sync, so ideally this command should be run on all the
server hosts at the same time.
Test Plan: Added a test and ran it
Reviewers: sid0, pyd, dschleimer, davidsp
Reviewed By: davidsp
Differential Revision: https://phabricator.fb.com/D1422865
Commit 70d94df78f11 tried to hide this benign warning, but did it at the wrong
spot. This does it at the right spot (around cursor.close()).
This is the description from the original commit:
The revision_references table has multiple unique keys (primary key, and a unique
name index). Using ON DUPLICATE KEY in an INSERT causes a warning, since it will
only update one of the rows, even if there are two duplicates. We can safely
ignore this since we know there will only ever be one row that has a duplicate
key.
Previously we escaped path names ourselves, but this didn't remove %'s from the
paths, which could break the sql libraries ability to format queries with args.
Now we use the actual sql libraries builtin formatting.
The revision_references table has multiple unique keys (primary key, and a unique
name index). Using ON DUPLICATE KEY in an INSERT causes a warning, since it will
only update one of the rows, even if there are two duplicates. We can safely
ignore this since we know there will only ever be one row that has a duplicate
key.
Check with the actual database to ensure we are the ones with the write lock
before performing the actual lock. We encountered an issue previously where two
writers managed to write at once, which this should now catch.
Summary:
In certain pull use cases (hg pull -B foo), it tries to update the bookmarks
outside of the normal bookmark.updatefromremote() function. This takes the lock
at the very beginning of the pull so all inner bookmark manipulations should
succeed.
Test Plan:
Ran the new test with and without the fix, verified it failed before
and passed afterwards.
Reviewers: sid0, davidsp
Differential Revision: https://phabricator.fb.com/D1214148
Pushing and pulling local bookmarks was a bit broken, since as soon as the
bookmark dict tried to write, it would sync with the db, which overwrote the
current bookmarks.
This changes it to require the sql connection to be set up and locked before
the bookmark modifications even start to happen.
Previously, if the repo needed to be synced, every command would block and
wait until the repo was synced. So if one command was doing the update, all
the other parallel commands (even the readonly ones) would be blocked.
This changes it so readonly commands (i.e. commands that don't require locking
the db) can be executed even if the repo is slightly out of date. The downside
of this is a user may get a slightly old (by seconds) version of the repo when
doing a pull.
Bookmarks were failing to write to the database when pushed (but worked fine
when pulled) because the executewithsql around the bookmark writing would sync
the bookmark dict and overwrite the pending bookmarks.
The new logic takes the lock up at the pushkey level. Also had to add support
for nesting locks so both the pushkeys and the bookmark write could take the
bookmarks_lock.
MySQL has limits on how large a transaction can become, so if a commit is
larger than that we need to break it into several database transactions. To
keep the atomic nature of the repo table, we add a new pushkey namespace 'tip'
that contains the latest valid linkrev for that repo in the table. Then at the
very end we update the 'tip' marker (along with the heads).
When syncing, the repo only syncs up to the tip commit.
When making a commit, it first cleans up any abandon commits by deleting
any rows that have a linkrev greater than the tip.
Before writing revisions to the database we now validate that they are
correct. This consists of two checks:
1. Are we appending to the same linkrev in the database as we are in the local
repo.
2. Every rev that we are writing to the database has a base, p1, and p2 rev
dependency. Before we write the rev, verify that the base, p1, and p2 revs in
the database have the same nodes as in the localrepo. This prevents us from
writing rev dependencies that then point at the incorrect node (due to
different ordering in the db vs local).
In order to validate that the incoming revisions are valid, we need node
information in the revisions table. A future patch will add the actual
validation.
Check that heads and bookmarks match after syncing.
Verify that the new commits are being applied on top of the correct linkrev on
the server.
Remove excess invalidate steps. They are no longer necessary since we do the
lock.release() much later in the process (which was what caused problems
originally).
I added revlog._writeentry to upstream Mercurial which allows us to easily
intercept revlog writes instead of building crazy file-like objects to
intercept the calls.
Moves the sql connect, lock, unlock, close logic to a single executewithsql
function so all the complicated try/finally logic is in one place.
Move pre-transaction close logic out of the hook and into a transaction.close
wrapper. This makes it more symettrical (matches transaction open/close) and
allows hooks to fail the transaction before the db commit.
Switches to using a single table for many repos instead of multiple databases.
Changes the headbookmarks table to be a pushkeys table using the pushkeys
namespace convention.
Move public phase boundary forward during sync
Thread syncdb so we pull from the db at the same time we are writing to the
revlogs. Also use a buffered revlog so we only flush at the very end.
Clear repo._filecache after syncing in order to force the changelog to update.