The previous commits added treemanifest integration, but the tests weren't
actually accurate. By enabling treemanifest.server=True before making any
commits, it turns out we were creating trees during commit. So all the backfill
and replay logic wasn't actually doing anything.
This patch fixes the tests by moving the treemanifest.server=True bit to later
on, and fixes a bug in the code around needing to not validate that the added
revision linkrevs are newer than tip, since backfilltree and sqlreplay are meant
to add old data.
Summary:
The sqlreplay command goes through all the revisions in the database and
attempts to replay them onto the local revlogs. Any revisions which already
exist in the revlog are skipped, but revisions that aren't in the revlog are
applied to the end of the revlog, with the same verification that is normally
used when applying revisions (i.e. it's being applied at the right file offset
and the parents and delta base nodes are already present).
This is usual when migrating the repository to a new format, like when switching
to treemanifests.
Test Plan: Added a test
Reviewers: #mercurial, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D4801108
Signature: t1:4801108:1490869053:920db3a86801612ec56d6c076b5854bb9a44ca62
Summary:
treemanifest interacts with pushrebase to create trees during push, even if the
client sends a flat manifest. This patch adds a test to verify that those trees
are sent to sql and replicated.
Test Plan: Added a test
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Differential Revision: https://phabricator.intern.facebook.com/D4801107
Signature: t1:4801107:1490883109:6818cbc96038ac6e63e4e6dd3d619775c6d8c26c
Summary:
The backfilltree command is a command in the treemanifest extension that can
retroactively create new revlog entries for treemanifests. hgsql needs to
provide a sql connection for this write so the new entries can be synced to sql.
A future patch will add a new hgsql command for replaying these old writes into
an existing repository, since they won't automatically be replayed since the
linkrevs are older than the tip linkrev (and syncing works by only replaying
newly added revisions).
Test Plan: Added a test. Also ran it on an internal repo.
Reviewers: #mercurial, simonfar
Reviewed By: simonfar
Differential Revision: https://phabricator.intern.facebook.com/D4801101
Signature: t1:4801101:1490883142:a8bd2968a22d7d7f89a9ce626502504a464c9f11
Summary:
Debugging showed that readding all the bookmarks (especially the old ones) took
a significant amount of time. Let's optimize the bookmark sync to do the minimal
number of adds/deletes instead.
Test Plan: Adds a test for bookmark deletes since it didn't exist before.
Reviewers: simonfar, #mercurial, quark
Reviewed By: quark
Subscribers: quark
Differential Revision: https://phabricator.intern.facebook.com/D4792724
Signature: t1:4792724:1490749918:33e661ffd042a3281c42eee6164f2f669398178b
Summary:
The upstream manifestctx class has changed to keep a reference to the
manifestlog instead of to the repo. This means that if we access an old mfctx
after repo.manifestlog has been invalidated, it will still have a reference to
the old manifestlog and the old manifestrevlog.
This created a bug where writes would fail because they saw the old
manifestrevlog, and even worse, they would truncate the manifest revlog back to
the last state that process had seen it (i.e. from before the latest db sync),
instead of truncating just the writes done by this transaction.
Test Plan:
I copied it to a server that had issues before the change, and
verified that it no longer truncated the manifest revlog incorrectly. I'm still
trying to think how I could reproduce the issue in a test.
Reviewers: #mercurial, phillco
Reviewed By: phillco
Differential Revision: https://phabricator.intern.facebook.com/D4789920
Signature: t1:4789920:1490735433:187e8b22d7ce89bfb2f88d9fc3c9932842bab4f8
Summary:
The test is to check whether `varchar CHARACTER SET latin1 COLLATE latin1_bin`
works just fine or not.
Sadly it does not. And `varbinary` makes the test pass, as expected.
For now, this diff only changes the `path` field, other fields like
`revision_references.name` etc should use `varbinary` in theory. But they are
less concerning than `path` because they don't impact real repo content. Using
`varchar`, MySQL clients are happy to display the contents, instead of showing
a placeholder like "(BLOB)".
Test Plan: `run-tests.py *.t`
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D4774735
Signature: t1:4774735:1490700636:61a3dc46476b37b3c32e6a022d5791be2c3ea965
Summary:
MariaDB is a well-known project as a drop-in replacement for MySQL. Some
Linux distributions (ex. Arch Linux) ship it by default.
I cannot get MariaDB pass `test-sync.t`. The server seems to close the
connection early and the client (the hg server) gets something like:
2055: Lost connection to MySQL server at 'localhost:3306', system error: 32 Broken pipe
I gave up figuring out what's going on after a few hours, and think it may
save others some time if we write down the issue.
Test Plan: spellchecker
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D4774842
Signature: t1:4774842:1490401350:9937e8021953fcbaee0cbc7809c243d9bb24d744
Summary:
Add two tests so tests can be skipped early instead of dumping stack traces:
- If the Python module `mysql.connector` is not available, skip.
- If the `mysql` command fails, skip.
Test Plan:
Normally, test passed:
$ run-tests.py test-sync.t
.
Without mysql.connector:
$ mv /usr/lib/python2.7/site-packages/mysql/connector{,2}
$ run-tests.py test-sync.t
s
Skipped test-sync.t: mysql-connector-python missing
With a wrong `getdb.sh`:
$ echo 'DBPORT=9999' >> getdb.sh
$ run-tests.py test-sync.t
s
Skipped test-sync.t: unable to initialize the database. check your getdb.sh
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D4772997
Signature: t1:4772997:1490400442:cf67ee406ee12905042cb0faf1053e355eda2ea7
Summary:
Previously, when running with an external MySQL setup, it's easy to get the
following "mysterious" error when trying to create the "revisions" table:
ERROR 1071 (42000): Specified key was too long; max key length is 767 bytes
This is a cleanup of many minor areas to make it easier for external users
to run the test.
* Move database schema to `schema.sql`, which could be used as `mysql`
input directly. Remove schema in other places so `schema.sql` is the
single source of truth.
* Change `repo`, `namespace` fields to `varchar` to save space.
* Explicitly set `latin1(_bin)` in schema to avoid potential encoding
issues. Also specify `InnoDB` to avoid issues of other engines.
* Source `getdb.sh` instead of parsing its output. Simplifies `getdb.sh`.
* Provide an example of `getdb.sh` as `getdb.sh.example`.
* Add `DROP DATABASE` cleanup. External setup probably don't have `testdb`
dropped automatically.
* Change `README.md` with the latest instruction about database and test.
* Bundle `hg-ssh` so `test-pushpull.t` won't fail if it's missing.
Test Plan: `run-tests.py` and it mostly works on a local, non-prod Linux system.
Reviewers: #sourcecontrol, rmcelroy
Reviewed By: rmcelroy
Subscribers: stash, rmcelroy, tja, durham, andrasbelo
Differential Revision: https://phabricator.intern.facebook.com/D4313981
Signature: t1:4313981:1490381520:7c0c9af72ac3446c5ee31f8c9b4376c4198a6dc8
Summary:
Previously we were not logging sqllock wait times for waits that timed out. This
adds that, and adds a variable to indicate if the lock succeed or failed.
Test Plan: Added a test for the ui.log metrics
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark
Differential Revision: https://phabricator.intern.facebook.com/D4295786
Signature: t1:4295786:1481154111:442b55f230b53add4e0966223af165f5195ba8b4
Summary:
Previously we detected if we were in the middle of a sync by checking if the
sync limiter lock was taken. This lock is not actually taken during a sync where
we force it to sync. So let's replace _synclock with a more robust
self._issyncing boolean.
Test Plan:
Did some manual testing with break points. My last attempt was just
naive, so this isn't a big change.
Reviewers: #mercurial, quark
Reviewed By: quark
Differential Revision: https://phabricator.intern.facebook.com/D4289240
Signature: t1:4289240:1481071402:e2a448a2739bff3eecfea90a6fd7a0376b944adb
Summary:
We've been seeing some issues where non-write processes (like clone) are taking
the wlock/lock without taking the sqllock. This causes processes which do want
the sqllock (like pushes) to hang waiting. Streaming clones are one known case
of this, but this extra logging will help us catch other cases.
Once we're confident that nothing takes the wrong lock ordering, we can change
this to throw an exception instead of logging the error.
This required a minor refactor the synclimit code, so we can detect if we're in
a sync lock (which is a case where it's ok to take the wlock/lock without taking
the sqlock). And it configures the tests to use streaming clones when possible,
which trigger the current known bug.
Test Plan: Added a test
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark
Differential Revision: https://phabricator.intern.facebook.com/D4287384
Signature: t1:4287384:1481065929:dfac78e3537c3cdf44c810bc99c5f07ed5b12f4c
Summary:
Internally we are now logging the sqllock times to our metrics infrastructure.
It's easier if the numbers are integers, so let's turn the elapsed sqllock time
into milliseconds. We leave the user readable message alone though.
Test Plan: doitlive
Reviewers: #mercurial, quark
Reviewed By: quark
Differential Revision: https://phabricator.intern.facebook.com/D4287378
Signature: t1:4287378:1481065691:5ae3b259598692a41927c793dab1bc2c847acae0
Summary:
The manifest has been replaced with a manifestlog in upstream, so we need to
update our usage accordingly.
Test Plan: Ran the tests
Reviewers: #mercurial, quark
Reviewed By: quark
Differential Revision: https://phabricator.intern.facebook.com/D4198744
Signature: t1:4198744:1479408675:94fd5a5de367fcde1fc1ff16c0a88f052ef60a8d
Summary:
In the past we've introduced code into hgsql that tries to load bookmarks into
memory very early in the process, so there's no chance of the bookmarks being
out of sync with the changelog. When we added the synclimiter short circuit code
path, we lost the ordered loading which has caused the master bookmark to seem
to disappear when users were doing a pull and the bookmark file changed after
they had already loaded the changelog.
A real fix should go upstream, but since hgsql wraps the repo object so tightly
(before anything else gets a chance to operate on it), putting a work around
here is a simple, high impact fix.
Test Plan: Ran the tests, but really the tests won't catch anything here.
Reviewers: #mercurial, quark
Reviewed By: quark
Subscribers: quark
Differential Revision: https://phabricator.intern.facebook.com/D4121849
Signature: t1:4121849:1478133009:3cad7decdc866e68872c93b884467dfbc17f850c
Summary:
We had tried to be smart about incrementally updating only the heads and
bookmarks that actually changed, but the old logic was broken. It compared
binary and hex nodes against each other, and therefore didn't actually do
anything incrementally.
This patch makes us convert the Mercurial data to hex at the very beginning, so
the incremental check uses hex as well.
Test Plan:
Inserted print statements around the insert/delets and verified they
changed before and after
Reviewers: #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D4012317
Summary:
We sometimes "succeeded" in syncing to the database if we could not get local write locks, because we assumed that the only time we can't get those locks is if we're not waiting for them. However, if another process holds the local write locks and is not releasing them, we hit a timeout.
If you're doing a write operation, this is harmless - the MySQL lock ensures that only one client at a time can be syncing anyway. However, for read operations run with `--forcesync`, this can cause us to succeed without syncing, breaking our contract with our users.
Test Plan:
Run the test suite and confirm no change.
Manually take the local write locks, run against the test DB `hg log -r . --forcesync`, and confirm that we fail correctly with this change but not without it.
Confirm that even with the local locks taken, `hg log -r .` returns success
Reviewers: durham
Reviewed By: durham
Subscribers: tja, #mercurial
Differential Revision: https://phabricator.intern.facebook.com/D3976452
Tasks: 13484857
Signature: t1:3976452:1475757832:e8370819bc345ce8e1bde0b057755803adb9896c
Summary:
Upstream a099aba68382 changed the words about backup bundles.
Let's update our tests.
Test Plan: Run the changed tests.
Reviewers: #sourcecontrol, stash
Reviewed By: stash
Differential Revision: https://phabricator.intern.facebook.com/D3949507
Signature: t1:3949507:1475221527:712dc13005a2860aac64293968472ab8de074f8c
Test Plan: run all tests; no more skipped tests
Reviewers: #mercurial, ttung, quark
Reviewed By: quark
Subscribers: quark
Differential Revision: https://phabricator.intern.facebook.com/D3819316
Signature: t1:3819316:1473091353:c695355d74576f879062865c22f0c675c9864367
Summary:
Currently test-race-conditions.t fails due to the
fact that we don't wait enough to cover the duration of
presyncdb hook.
Test Plan: run test-race-conditions.t a couple of times
Reviewers: simonfar, durham
Differential Revision: https://phabricator.intern.facebook.com/D3605295
Summary:
Previously, if we needed to disable hgsql for some reason (like testing
corruption, etc) we would just disable the extension. Since we now add a
requires flag to repos that are created with hgsql, this no longer works.
Let's add a config option that can disable hgsql instead and update the tests to
use it. I'm not sure how these test failures slipped through the original
commit.
Test Plan: Ran the tests
Reviewers: #mercurial, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar
Differential Revision: https://phabricator.intern.facebook.com/D3289119
Signature: t1:3289119:1462999047:f33337610a1777c379adb70ec307971823f7ae59
Summary:
Upstream mercurial has deleted bookmarks.write() since all bookmark writes
happen via transactions now. So we can delete our write wrapper for that. We
already wrap transaction closes, so nothing new needs to be added.
Test Plan: Ran the tests
Reviewers: #mercurial, ttung, rmcelroy
Reviewed By: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D3289117
Signature: t1:3289117:1463000038:aaf3454059514d92a017bd579b6de0691b334795
Summary:
We don't want to allow commands to run in the repo if hgsql is not
enabled, so let's add a requirement.
This will not affect existing repositories, since they don't have the flag in
their requires, but the flag can be added manually once this is deployed.
Test Plan: Added a test
Reviewers: #sourcecontrol, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar
Differential Revision: https://phabricator.intern.facebook.com/D3239263
Signature: t1:3239263:1461923217:b8fb59498292e1f9d317d98d5de7ce8513685ba6
Summary:
This adds a config option that can be enabled to profile what is
happening when the sqlock is taken (since that is the time spent that limits how
fast we can accept commits). It writes the profile out to an external file for
reading.
Test Plan:
Added a test. Also manually copied this to an hg server and did a
push.
Reviewers: #sourcecontrol, ttung, simonfar
Reviewed By: simonfar
Subscribers: simonfar
Differential Revision: https://phabricator.intern.facebook.com/D3239204
Signature: t1:3239204:1461922938:45298ada2955df8e5e145805f137ccc1974becea
Summary:
This fixes a bug where the initial sync of a generaldelta repo could
cause the changelog.i to have a bad header byte that indicated the revlog was
generaldelta when it really wasn't. This fixes by applying the changelog
hardcoded value to the lightweight hgsql revlog class.
Test Plan: Added a test. It failed before, and passes now.
Reviewers: #sourcecontrol, ttung, quark
Reviewed By: quark
Subscribers: quark
Differential Revision: https://phabricator.fb.com/D3194429
Signature: t1:3194429:1461030817:6cc836e2dcd8b1feeeec16ee46548eac443695bf
Summary: we no longer print "activating" if the bookmark is already active
Test Plan: all tests are passing
Reviewers: #sourcecontrol, ttung
Reviewed By: ttung
Differential Revision: https://phabricator.fb.com/D3117128
Signature: t1:3117128:1459372145:64b3a523d290fa18d7146d48adaa953f698d9d02
Summary: For fixing tests
Test Plan: this is fixing some tests
Reviewers: #sourcecontrol, ttung
Reviewed By: ttung
Differential Revision: https://phabricator.fb.com/D3117108
Signature: t1:3117108:1459372132:0daf2049cd66a192e96a551661fcea2a81fb55a8
Test Plan: this is fixing some tests
Reviewers: #sourcecontrol, quark, ttung
Reviewed By: quark
Differential Revision: https://phabricator.fb.com/D3117092
Signature: t1:3117092:1459371915:231c133209a06f3e0e207482858775802d5820e7
Summary:
We've been seeing bugs where pulls result in bookmarks missing for clients. We
suspect it to be a race condition on the server side. One possible race is that
we load the changelog data before the bookmarks, and therefore may receive
bookmarks that point to a node that isn't in memory.
A simple fix to decrease the chances of this is to force loading bookmarks
first. Since hgsql is the first bit of code that access the repo, we can just
change the order here to ensure bookmarks are loaded first.
Test Plan:
It's not really a correctness change, so I wasn't able to test it.
What I did do however was insert exceptions into the bookmark loading and
changelog loading logic on one of our servers. This showed me that the changelog
was indeed being loaded first before because of this code.
Reviewers: #sourcecontrol, ttung
Reviewed By: ttung
Differential Revision: https://phabricator.fb.com/D3063659
Signature: t1:3063659:1458244130:7448377065fa0f0f6d98e43eaa00bdfc380948b6
Summary:
We are overriding the _afterlock to mind one more lock before executing
callbacks. We need that so our post commit hooks are executed out of the lock.
Test Plan: run tests
Reviewers: #sourcecontrol, ttung, durham
Reviewed By: durham
Differential Revision: https://phabricator.fb.com/D2809038
Signature: t1:2809038:1452124328:3b1d2da7711cdf1c97fe06d5dcf07be253582fcd
Summary:
Upstream has moved almost all bookmark writes to use bookmark.recordchange()
instead of write, which works inside a transaction. For hgsql, we need to
perform our bookmark writes inside a transaction now.
Additionally, we need to check self.disablesync inside committodb. disablesync
is used to allow us to write bookmarks during a syncdb without triggering the
bookmark writes to the db (since that would be a infinite loop). Previously we
only needed the syncdb check for bookmark writes, but since that has moved to
transactions, we need to do the check there too (committodb fires on
pretxnclose).
Lastly, in theory we could remove our wrapping of bookmark.write now, since
nothing should use it. But since upstream still has code for bookmark.write to
write bookmarks when outside a transaction, we can't remove our code just yet,
in case some extension still uses bookmark.write.
Test Plan: Ran the tests
Reviewers: ttung, lcharignon, #sourcecontrol, mitrandir
Reviewed By: mitrandir
Differential Revision: https://phabricator.fb.com/D2808601
Signature: t1:2808601:1452116992:ed34107a97ac633281ad8934266b35e3375cb247
Summary:
We've seen load issues on the mysql server in certain situations. This is caused
by large revision_reference query load from every read connection checking the
database.
To fix it, let's use Mercurial locks to enforce that only one reader is checking
and syncing at any given time. All other will just serve the existing on disk
data immediately.
Writer clients are unaffected. They do not obey the rate limiting lock and do
exactly what they did before.
Test Plan: I'll try to add a test
Reviewers: #sourcecontrol, ttung
Differential Revision: https://phabricator.fb.com/D2756217
Since upstream Mercurial now defaults to generaldelta, this affects the tests.
In this particular test, the offset value (which contains the revlog type if
it's rev 0, like in this case), is different because it now contains the
generaldelta bit.
Summary:
test-sync tested a race condition by introducing a sleep in a hook inside the
lock. It was flakey though. This extends the lock hook to 2 seconds and makes
the second command wait 1 second before beginning, so we can ensure the first
command is in the lock before the second one starts.
Test Plan:
Ran the tests a lot, saw no failures. Previously I would see them
relatively frequently.
Reviewers: #sourcecontrol
Differential Revision: https://phabricator.fb.com/D2665352
Summary:
hgsql failed when run against bundle repos before. We've fixed it so that no
sync is performed when reading from a bundle repo, but writes will be rejected.
Test Plan: Added a test and ran the tests
Reviewers: #sourcecontrol
Differential Revision: https://phabricator.fb.com/D2665347
Summary:
The old MySQLdb library did not support IPv6 so we've switched to using
mysql.connector, which is more recent and does support IPv6.
The most notable change in functionality is that it returns byte arrays instead
of strings, so I had to add a custom type converter that returns strings.
Test Plan: Ran the tests (after installing the new rpm)
Reviewers: #sourcecontrol
Differential Revision: https://phabricator.fb.com/D2665344
Summary:
We're going to be refactoring which functions actually sync with sql soon, and
putting this logic in a function will help us stay clean.
Test Plan: Ran the tests
Reviewers: #sourcecontrol
Differential Revision: https://phabricator.fb.com/D2665341
