Summary:
Small refactoring before the next diffs.
Before
mononoke_hg_sync ... --mode sync-once ssh://PATH --start-id 0
Now
mononoke_hg_sync ... ssh://PATH sync-once --start-id 0
Reviewed By: HarveyHunt
Differential Revision: D14226940
fbshipit-source-id: 1b6db6f194aecb2f4bdb7bbd9e846aaba180e098
Summary:
We didn't process parts like `error:abort` and so we might have easily missed
an error. This diff fixes it.
Reviewed By: quark-zju
Differential Revision: D14185378
fbshipit-source-id: e68e365fd939a4bd6a0c2835a513ebc94530aa87
Summary:
Pretty big cleanup. The biggest part is simplifying comparing logic.
Pushrebase replay compared replayed pushrebase commits with their hg
counterparts. Because previously we didn't log enough data from hg servers we
had to jump through a lot of hoops to find out which commits should be compared
to which.
Now we are logging `ordered_added_revs` i.e. which commits mercurial produced
after the hg pushrebase, which is exactly what we need for comparison. So a lot
of code can be simplified.
That does mean though that some of the recorded pushrebases we won't be able to
replay because they don't have `ordered_added_revs`. At the moment we have
~300K pushrebases without `ordered_added_revs` and ~100K with. I think this
change is worth it given how simpler the code is and I'd argue 100K is a pretty
big number. For the rest 300K pushrebases we can later manually fill in
`ordered_added_revs` field if we consider it's necessary.
After this diff pushrebase replay won't use hg repo, and so we can run it on
normal twshared jobs instead of on hg servers.
A few smaller changes:
1) Also note that for now this diff makes pushrebase replay single threaded, while
previously we ran comparison in parallel. That will be fixed in the next diffs.
2) Pushrebase replay now always compare commits i.e. `--compare-commits` option
was removed
Also change repo id in tw spec to 20
Reviewed By: HarveyHunt
Differential Revision: D14122963
fbshipit-source-id: 0f8da7cffb13899f11143a01d1a301fdf8ea7f00
Summary:
First stab at a job that will keep hg in sync with Mononoke when Mononoke
becomes a source of truth.
Reviewed By: ikostia
Differential Revision: D14018269
fbshipit-source-id: f88c5eba8bf5482f2f162b7807ca8e41a3b4291d
Summary:
We can't run in parallel at the moment as the log file and the lock file are
shared.
Every path maintains independent backup state (the previous diff).
The secondary backup state doesn't affect smartlog (only the main one)
The issue with this approach is that we maintain backup lock a bit longer.
Unfortunately, the progress in smartlog doesn't show anything about the second backup.
I added 'finished', it makes it easier to compare in the logs.
Reviewed By: markbt
Differential Revision: D14149399
fbshipit-source-id: f90e8aac6cb8dee53d5c7468bd6adba067e13362
Summary: Named branches are going away. Remove the logic around it.
Reviewed By: phillco
Differential Revision: D13978575
fbshipit-source-id: d6e28d7cadffa612f74a2afc12800829d6113dfa
Summary: This is adds a metaconfig option to preserve push/pushrebase bundles in the blobstore.
Reviewed By: StanislavGlebik
Differential Revision: D14020299
fbshipit-source-id: 94304d69e0ac5d81232f058c6d94eec61eb0020a
Summary:
This diff does not change anything on it's own, but rather adds the not
reachable (but already somewhat tested) code to preserve bundles when doing
pushes and pushrebases.
I want to land it now so that conflict resolution is easier.
Reviewed By: StanislavGlebik
Differential Revision: D14001738
fbshipit-source-id: e3279bc34946400210d8d013910e28f8d519a5f8
Summary:
Together with logging bookmark moves, let's also log bundle handle. It will be
used during replay from Mononoke to mercurial.
Reviewed By: ikostia
Differential Revision: D13990929
fbshipit-source-id: 4039322903b13e84fb31c8e65cc2e097ca765213
Summary:
This is the first step in adding support for tracking all bookmark moves. They
will be recorded in the separate mysql table in the same transaction as
bookmark is updated.
That gives us two things:
1) Ability to inspect all bookmark moves and debug issues with them
2) Also record which mercurial bundle moved a bookmark if any so that we could
later replay these bundles in correct order on hg
Add a struct that let us track bookmark moves.
Reviewed By: ikostia
Differential Revision: D13958872
fbshipit-source-id: 9adfee6d977457db5af4ad5d3a6734c73fcbcd76
Summary: Additionally parts of the hook were adjusted to better fit Mononoke design rather than being a copy of Mercurial version of the hook and so that kill switches are configs and test script is being passed through config
Reviewed By: StanislavGlebik
Differential Revision: D13964487
fbshipit-source-id: 8ab2784083ece80dfb781ddd1245172e65699ad7
Summary: Passing input on command line is restricted to the maximum length of a command, it is much better to pass the file descriptor, this way one can pass very long inputs inside the bash functions.
Reviewed By: StanislavGlebik
Differential Revision: D13964484
fbshipit-source-id: da010ecfcf05da8c5860c8b5ee0860a8aeda0502
Summary:
I decided against using the naming convention maybe_ancestor and maybe_descendant as I believe the name is_ancestor already implies this uncertainty.
Skiplist is used instead of bfs, this improves performance significantly when using very old commits. For now we do not update skiplist as more recent commits are in cache anyway.
Reviewed By: StanislavGlebik
Differential Revision: D13917333
fbshipit-source-id: 21b49d920ff473c953a952ee3c6d7b55565f98ac
Summary: Test that we can non-forward push a bookmark. Hooks should prevent that for some bookmarks, but they are a different matter.
Reviewed By: lukaspiatkowski
Differential Revision: D13803097
fbshipit-source-id: f2dfba9a21f83127e8c92397c728a074037da046
Summary: The Copy trait means that something is so cheap to copy that you don't even need to explicitly do `.clone()` on it. As it doesn't make much sense to pass &i64 it also doesn't make much sense to pass &<Something that is Copy>, so I have removed all the occurences of passing one of ouf hashes that are Copy.
Reviewed By: fanzeyi
Differential Revision: D13974622
fbshipit-source-id: 89efc1c1e29269cc2e77dcb124964265c344f519
Summary:
Can we rebase a merge commit whose one parent (p2) is an ancestor of our public
bookmark and the other - not?
Reviewed By: lukaspiatkowski
Differential Revision: D13802863
fbshipit-source-id: f12cbc0684a001a2b9ac6eb903d825e73777ab58
Summary:
Are we sure we're prohibiting rebases over the merge commit in a public
history?
Reviewed By: lukaspiatkowski
Differential Revision: D13802865
fbshipit-source-id: d9556ea0616825f82d1f781c4e57bc01a5c5a0b0
Summary: Can we rebase a merge where both parents are not ancestors of current master?
Reviewed By: lukaspiatkowski
Differential Revision: D13802864
fbshipit-source-id: 0003c72c54e8689e1a26a252d48008bac31d5c0b
Summary: Can we do a no-op push which neither adds commits nor advances a bookmark?
Reviewed By: lukaspiatkowski
Differential Revision: D13802862
fbshipit-source-id: 7ef3b7a2f79c9dc9619ffb4f0fad925d263ff425
Summary: Do we successfully push a commit when no rebasing is needed?
Reviewed By: lukaspiatkowski
Differential Revision: D13802868
fbshipit-source-id: 5ca73f59b4057d8c00f10320a2d6b76f0197187d
Summary: Similarly to Mercurial's `pushrebase` tests, we want to test both cases.
Reviewed By: lukaspiatkowski
Differential Revision: D13802867
fbshipit-source-id: 77dc875857602e5abb177af4f61af9f1fda072e3
Summary: Just a helper to print more concise logs.
Reviewed By: StanislavGlebik
Differential Revision: D13802866
fbshipit-source-id: 77c44735a416612320b22e20fa4413923334f79c
Summary: Rename Mononoke API to Eden API, per war room discussion.
Reviewed By: quark-zju
Differential Revision: D13908195
fbshipit-source-id: 94a2fe93f8a89d0c5e9b6a24939cc4760cfaade0
Summary:
There were lots of small stupid mistakes with pushrebase replay in past.
Since pushrebase replay will stay with us for quite some time,
let's add a test to prevent these regressions in future.
In tests we'll use filesystem+sqlite instead of everstore + xdb to fetch
requests to replay.
Reviewed By: HarveyHunt
Differential Revision: D13940390
fbshipit-source-id: a4398d1c8c22bf16a85d6391a1f5665ce4b73eb1
Summary:
`blobrepo_factory` is a crate that knows how to create blobrepo given
a configuration i.e. it creates blobstores, filenodes, changesets etc and
initializes blobrepo with them.
`post_commit` is a small part of blobrepo which can also be extracted from
blobrepo crate.
There are a few upsides with this approach
1) Less dependencies on Blobrepo, meaning we have to rebuild it fewer times
2) BlobRepo compilation is faster
Reviewed By: jsgf
Differential Revision: D13896334
fbshipit-source-id: 1f5701341f01fcefff4e5f9430ddc914b9496064
Summary: Add metadata to each delta entry written to the datapack. Since the HTTP API never serves LFS files, and the only flag currently used simple indicates whether a file should use LFS, the flag field is intentionally set to `None`, leaving only the size in the metadata (which, since we're storing full file content, is the same as the content length).
Differential Revision: D13894292
fbshipit-source-id: 36db25adb0c46cd1c7fde841a69d3e6d48d08d06
Summary: Previously, `hg debuggetfile` would allow fetching a single file from the API server. This diff updates the command to `hg debuggetfiles`, which accepts a list of filenode/path pairs on stdin, and fetches all the files concurrently, writing the contents to a single datapack.
Reviewed By: DurhamG
Differential Revision: D13893894
fbshipit-source-id: 36fc54f1870273ab4b447de5d615b3fb6aaa3378
Summary: Add a new `get_file()` method to `MononokeClient` that fetches Mercurial file content from the API server and writes it to a datapack in the cache. This functionality is exposed via the new `hg debuggetfile` debug command, which takes a filenode and file path and fetches the corresponding file.
Differential Revision: D13889829
fbshipit-source-id: 2b68bf114ee72d641de7a1043cca1975e34cf4e6
Summary: Add a `/getfile/` endpoint that takes a filenode and returns Mercurial file content. This is similar to the `/blob/` endpoint, except the returned content can also include copy information in the case that the file was copied. This matches the exact format in which Mercurial stores file content, so the resulting output can be directly written to disk.
Reviewed By: chadaustin
Differential Revision: D13876096
fbshipit-source-id: 0417cb1f993c4c93687e364b1647490c3a9040bc
Summary:
At From<Error>::from errors are automatically downcast into an InternalError. While we unwrap them for http this was not done for thrift, resulting in an InternalError being returned for everything. By unwrapping the error first thrift now returns more meaningful errors.
In addition I added a new errorkind for ls_v2, this is purely for internal structure as the client will still see an InvalidInput error.
Reviewed By: StanislavGlebik
Differential Revision: D13881374
fbshipit-source-id: bd2e43a0ac7e5abdbf41068c3b1ab37681f03009
Summary: Rename this debug command to make it obvious what it does (i.e., perform a health check) and make it print more useful output (i.e., the name of the server for which it performs the health check).
Reviewed By: DurhamG
Differential Revision: D13890867
fbshipit-source-id: 8fc96bcc06d04611a308ecc0b870049936f1cb29
Summary:
I already made them the same (but copy) in a different diff. As we discussed at
war room, symlinks is a better solution.
Also removed unused import and some functions that were copied from mercurial.
This is no longer needed as we now share tinit.sh with general purpose
functions.
The only mononoke specific file left is dummyssh.py
I think later we could pack them in a separate buck target or something and share in a better way that symlinks but for now it is the easiest solution.
Reviewed By: DurhamG
Differential Revision: D13881960
fbshipit-source-id: 36f425d6f0ddbae2c9d083de35d2779669dc01e7
Summary:
This is the first step, I just copied the files from mercurial to mononoke/tests/integration/third_party and made it work.
Further unification is required to reuse the same files.
It renders progress bars and other fancy things and matches our mercurial test framework.
Also it allows to use embedded python code!
Some things are not supported: like aliases, so changes in the tests are required.
Some tests used urandom and cat in a wrong way, I had to fix it as well.
Reviewed By: lukaspiatkowski
Differential Revision: D13871531
fbshipit-source-id: c723bb18c2233639bb36bc7964e57baddff4c0b9
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary:
Previously pushrebasing an empty commit failed because we assumed that root
manifest of a commit is always sent in a bundle. This diff removes this
assumption
Reviewed By: lukaspiatkowski
Differential Revision: D13818556
fbshipit-source-id: 44e96374ae343074f48e42a90c691b21e3c41386
Summary:
Pushrebase wasn't returning a response to a pushkey part, and we get `server
ignored bookmark ... update` messages. This diff fixes it by returning the
reply to a pushkey part.
Note that behaviour is different from mercurial. In mercurial many pushkey
parts are allowed, while we allow only which moves `onto` bookmark. That
shouldn't be restrictive, however we can change this behaviour later if needed.
Reviewed By: aslpavel
Differential Revision: D13781546
fbshipit-source-id: edb0fdc7dc10c7a5cf4c49157fce0887e71fcf8a
Summary: This allows us to run pushbackup and cloud sync commands for Read Only Mononoke repos.
Reviewed By: ikostia
Differential Revision: D13804545
fbshipit-source-id: 8026fc4668afc8bb5c2c0a9587ca024e3c6920da
Summary:
Next step is to change infinitepush / cloud sync to pass this pushvar to
Mononoke
Reviewed By: StanislavGlebik
Differential Revision: D13802402
fbshipit-source-id: 25e3d699bd934e3e015b9784040bd2dc4b43188d
