The revision_references table has multiple unique keys (primary key, and a unique
name index). Using ON DUPLICATE KEY in an INSERT causes a warning, since it will
only update one of the rows, even if there are two duplicates. We can safely
ignore this since we know there will only ever be one row that has a duplicate
key.
Check with the actual database to ensure we are the ones with the write lock
before performing the actual lock. We encountered an issue previously where two
writers managed to write at once, which this should now catch.
Summary:
In certain pull use cases (hg pull -B foo), it tries to update the bookmarks
outside of the normal bookmark.updatefromremote() function. This takes the lock
at the very beginning of the pull so all inner bookmark manipulations should
succeed.
Test Plan:
Ran the new test with and without the fix, verified it failed before
and passed afterwards.
Reviewers: sid0, davidsp
Differential Revision: https://phabricator.fb.com/D1214148
Pushing and pulling local bookmarks was a bit broken, since as soon as the
bookmark dict tried to write, it would sync with the db, which overwrote the
current bookmarks.
This changes it to require the sql connection to be set up and locked before
the bookmark modifications even start to happen.
Previously, if the repo needed to be synced, every command would block and
wait until the repo was synced. So if one command was doing the update, all
the other parallel commands (even the readonly ones) would be blocked.
This changes it so readonly commands (i.e. commands that don't require locking
the db) can be executed even if the repo is slightly out of date. The downside
of this is a user may get a slightly old (by seconds) version of the repo when
doing a pull.
Adds test to test synchronizing between master repos and to test the
prevention of non-push/pull transactions (such as commit and strip).
Adds tests/getdb.sh to the ignored list. This is a script that provides the
database host, port, and name for the test. It must be in the form
"ip:port:databasename". The test assumes that the database name is unique and
doesn't already exist each time getdb.sh is called.
Bookmarks were failing to write to the database when pushed (but worked fine
when pulled) because the executewithsql around the bookmark writing would sync
the bookmark dict and overwrite the pending bookmarks.
The new logic takes the lock up at the pushkey level. Also had to add support
for nesting locks so both the pushkeys and the bookmark write could take the
bookmarks_lock.
MySQL has limits on how large a transaction can become, so if a commit is
larger than that we need to break it into several database transactions. To
keep the atomic nature of the repo table, we add a new pushkey namespace 'tip'
that contains the latest valid linkrev for that repo in the table. Then at the
very end we update the 'tip' marker (along with the heads).
When syncing, the repo only syncs up to the tip commit.
When making a commit, it first cleans up any abandon commits by deleting
any rows that have a linkrev greater than the tip.
Before writing revisions to the database we now validate that they are
correct. This consists of two checks:
1. Are we appending to the same linkrev in the database as we are in the local
repo.
2. Every rev that we are writing to the database has a base, p1, and p2 rev
dependency. Before we write the rev, verify that the base, p1, and p2 revs in
the database have the same nodes as in the localrepo. This prevents us from
writing rev dependencies that then point at the incorrect node (due to
different ordering in the db vs local).
In order to validate that the incoming revisions are valid, we need node
information in the revisions table. A future patch will add the actual
validation.
Check that heads and bookmarks match after syncing.
Verify that the new commits are being applied on top of the correct linkrev on
the server.
Remove excess invalidate steps. They are no longer necessary since we do the
lock.release() much later in the process (which was what caused problems
originally).
I added revlog._writeentry to upstream Mercurial which allows us to easily
intercept revlog writes instead of building crazy file-like objects to
intercept the calls.
Moves the sql connect, lock, unlock, close logic to a single executewithsql
function so all the complicated try/finally logic is in one place.
Move pre-transaction close logic out of the hook and into a transaction.close
wrapper. This makes it more symettrical (matches transaction open/close) and
allows hooks to fail the transaction before the db commit.
Switches to using a single table for many repos instead of multiple databases.
Changes the headbookmarks table to be a pushkeys table using the pushkeys
namespace convention.
Move public phase boundary forward during sync
Thread syncdb so we pull from the db at the same time we are writing to the
revlogs. Also use a buffered revlog so we only flush at the very end.
Clear repo._filecache after syncing in order to force the changelog to update.
