Summary:
This adds a new queue to replay scratch bookmark changes into Mononoke, which will allow us to replay them there.
There's not a lot going on in Mercurial (i.e. in this diff) to achieve this: we simply record bookmark changes as they happen in infinitepush, and allow for excluding some bookmarks. Specifically, we'll want to exclude backup branches, which we don't want to copy, since a) there's way too many of them and b) they're deprecated in favor of Commit Cloud.
Currently, this does not allow for replaying deletions. That would require further rework of how we delete things, since right now we do it my matching on bookmark names in the DB, which means the Python side of things is not aware of which bookmarks exactly were deleted. I'm not aware of how much use this is currently getting, but I'll research that and add it in if necessary.
Finally, one thing that's worth calling out here is the `bookmark_hash` column in this table. This is here in case we need to scale out the replication of bookmarks across multiple workers.
Indeed, we'll always want the replication of any given bookmark to happen sequentially, so we should perform it in a single worker. However, if we have too many bookmarks to replicate, then that could become a bottleneck. If that happens, we'll want to scale out workers, which we can do by having each worker operate on separate bookmarks.
The `bookmark_hash` column allows us to evenly divide up the space of bookmarks across workers if that becomes necessary (e.g. we could have 16 workers: one for each first hex digit of the hash). We won't use `bookmark_hash` immediately, but since it's very cheap to add (just compute one hash in Mercurial and put it in the table), I'm adding it in this diff now in case we need it later to avoid the friction of having to re-redeploy hg servers for that.
Reviewed By: StanislavGlebik
Differential Revision: D15778665
fbshipit-source-id: c34898c1a66e5bec08663a0887adca263222300d
Summary: This adds (behing a feature flag) the option for Mercurial servers to record bundles they ingest into a `forwardfillerqueue`. This queue is consumed by the commit cloud forwardfiller (see stack rooted at D15712911 for that) to replay those commits into Mononoke.
Reviewed By: farnz
Differential Revision: D15759638
fbshipit-source-id: 70660343408bbaa865fc8f51f49f590eaf525cce
Summary:
Merge the functionality of the infinitepushbackup extension (backing up commits
to commit cloud) into the commitcloud extension.
These two extensions are highly coupled, and the commitcloud extension
monkey-patches the infinitepushbackup extension for a lot of its functionality.
There is also a lot of code duplication between the two extensions which we can
remove if they are part of the same extension.
The infinitepushbackup commands (`hg pushbackup`, ...) are moved to subcommands
of the `hg cloud` command, e.g. `hg cloud backup`.
Each feature of the infinitepushbackup extension is moved to a new module
in the commit cloud extension:
The `background` module controls background execution of `hg cloud backup` and
`hg cloud sync`.
The `backupbookmarks` module tracks and updates scratch bookmarks for backups.
This will be deprecated in the future.
The `backupstate` module tracks whether or not a commit has been backed up.
This is now tracked separately from backup bookmarks in a new file:
`.hg/commitcloud/backedupheads.<remote-identifier>`. This also covers hidden
commits, preventing a re-backup of previously backed up commits when they are
unhidden.
Previously the commitcloud extension customized the smartlog annotations: `Backing up`
became `Syncing`, etc. This is now removed for consistency.
Previously the infinitepushbackup extension disabled background backup by
injecting an `infinitepushbackup.disableduntil` config entry into the user's
config. This is now replaced with a state file at `.hg/commitcloud/autobackup`.
Either option can be set to disable auto backup. Commit cloud will wait until
both have expired before starting to run background backups again.
Reviewed By: DurhamG
Differential Revision: D15276939
fbshipit-source-id: 1d28989a157286e47d3dd97ca9c70b27f692dda1
Summary:
There is redundant code in the codebase relating to fetching
configuration for database for the purpose of testing. This commit refactors
some code which enables removing the redundant code.
Reviewed By: quark-zju
Differential Revision: D8961806
fbshipit-source-id: f673bfffb80304d98987c9727f6e709fdb742813
Summary:
Split the infinitepush backups code into a separate module.
The infinitepush extension is responsible for providing scratch pushes, as well
as the server-side implementation for storing scratch bundles.
The infinitepushbackup extension uses this feature to provide automatic backup
support. It is an alternative to the commitcloud extension, and the two will
be mutually exclusive.
Reviewed By: DurhamG
Differential Revision: D8204418
fbshipit-source-id: 9279c426da35d451a5e7b39a72f2c91d364e3fd4
Summary:
We broke infinitepush with a commitcloud change. This would have been caught
if we had the commitcloud extension enabled during the infinitepush tests.
Differential Revision: D8257849
fbshipit-source-id: 8a0da2e3268a9545502d432fcbce4b4c1eae5bcb
Summary:
Before this change `hg pullbackup` did not set correct markers on commits.
This change make possible to see what changes already landed even when we are restoring repository from backup.
Before the change `fbclone` + `hg pullbackup` of repo with `C1` commit landed would result in:
```
o o C2
| |
o o C1
| /
|
o
```
after:
```
o o C2
| |
o x C1
| /
|
o
```
Reviewed By: StanislavGlebik
Differential Revision: D7032572
fbshipit-source-id: ffee3c7cc23c24a3df9a89c999c9dd2de226dbff
Summary:
`-r` seems to be unneeded, as tests pass without it.
Also, it does not look like the regex itself uses anything not mentioned in `man re_format` on OSX, so we can just use the non-extended re.
Reviewed By: StanislavGlebik
Differential Revision: D7167503
fbshipit-source-id: 3c5c520e9bf2627523cabc771226fc37dc2e9171
Summary:
Added passing BatchMode option to SSH call only when puchbackup runs in background.
Also fixed dummyssh in skipping options before hostname, added unittest.
Reviewed By: quark-zju
Differential Revision: D7119123
fbshipit-source-id: 2c8e66fee44cca5b23389cba8e21e3a0b237268e
Summary:
Make `library.sh` read `$HGTEST_GETDB_PATH` for `getdb.sh` location. Unify
`getdb.sh` used by infinitepush and hgsql into one.
Reviewed By: ryanmce
Differential Revision: D6974635
fbshipit-source-id: 55bcebfbdaca7f43fb0cfca0d3e089e76cf9f24e
Summary:
[infinitepush] Implement support for short hashes
Also, fix test that was broken and lint errors
Reviewed By: StanislavGlebik
Differential Revision: D6814967
fbshipit-source-id: fef1ae79ef85ebf84aa88cd99367946256049192
Summary: The time column was missing in the bookmark test table
Reviewed By: StanislavGlebik
Differential Revision: D6819122
fbshipit-source-id: c0b4ede651ba5c598f0384580942cfb727b5fe61
Summary: Makes more sense to have this structure after D6691821.
Test Plan: Ran all the tests.
Reviewers: quark, #mercurial, #sourcecontrol
Reviewed By: quark
Differential Revision: https://phabricator.intern.facebook.com/D6694171
Signature: 6694171:1515609818:1142419346e3458db7674b1b69036d88e82a1903
Summary:
Now that we moved infinitepush to hgext in D6691670, we can also move
the tests. The sql related tests need special attention because of the db
connection and mysql connector.
Test Plan:
Ran all the tests. Removed the mysql connector and checked that the
test is skipped. Also checked that the test is skipped if the db connection
details are not specified.
Reviewers: ikostia, singhsrb, #mercurial, #sourcecontrol
Reviewed By: ikostia
Subscribers: rmcelroy
Differential Revision: https://phabricator.intern.facebook.com/D6691821
Signature: 6691821:1515600428:279dc4796c9be721c5fa81ac2dcd6e065ac30f18
