Summary: This adds some Scuba reporting to unbundle_replay.
Differential Revision: D20674162
fbshipit-source-id: 59e12de90f5fca8a7c341478048e68a53ff0cdc1
Summary:
This updates unbundle_replay to do things concurrently where possible.
Concretely, this means we do ingest unbundles concurrently, and filenodes
derivation concurrently, and only do the actual pushrebase sequentially. This
lets us get ahead on work wherever we can, and makes the process faster.
Doing unbundles concurrently isn't actually guaranteed to succeed, since it's
*possible* that an unbundle coming in immediately after a pushrebase actually
depends the commits created in said pushrebase. In this case, we simply retry
the unbundle when we're ready to proceed with the pushrebase (in the code, this
is the `Deferred` variant). This is fine from a performance perspective
As part of this, I've also moved the loading of the bundle to processing, as
opposed to the hg recording client (the motivation for this is that we want to
do this loading in parallel as well).
This will also let us run hooks in parallel once I add this in.
Reviewed By: StanislavGlebik
Differential Revision: D20668301
fbshipit-source-id: fe2c62ca543f29254b4c5a3e138538e8a3647daa
Summary:
pushrebase_errmsg is NULL when we have conflicts, but we still shouldn't replay
the entry (because it'll fail, with conflicts). Let's exclude those.
Reviewed By: StanislavGlebik
Differential Revision: D20668304
fbshipit-source-id: a058bb466e0a8a53ec81e41db7ba138d6aedf3f9
Summary:
This updates unbundle_replay to support sleeping when watching for updates in a
bookmark and said bookmark isn't moving. This will be useful so it can run as a
service.
Reviewed By: StanislavGlebik
Differential Revision: D20645157
fbshipit-source-id: 6edeb66b65b2ef8b88c8db5e664982756acbfaf1
Summary:
I accidentally forgot to insert the entry, so that made this test a bit
useless. Let's make it not useless.
Reviewed By: StanislavGlebik
Differential Revision: D20645158
fbshipit-source-id: 0f0eb0cf9d16e8c346897088891aa3277b4d9c07
Summary:
This adds support for replaying the updates to a bookmark through unbundle
replay. The goal is to be able to run this as a process that keeps a bookmark
continuously updated.
There is still a bit of work here, since we don't yet allow the stream to pause
until bookmark update becomes available (i.e. once caught up, it will exit).
I'll introduce this in another diff.
Note that this is only guaranteed to work if there is a single bookmark in the
repo. With more, it could fail if a commit is first introduced in a bookmark that
isn't the one being replayed here, and later gets introduced in said bookmark.
Reviewed By: StanislavGlebik
Differential Revision: D20645159
fbshipit-source-id: 0aa11195079fa6ac4553b0c1acc8aef610824747
Summary:
I'm going to update this to run in a loop, so to do that it would be nice to
represent the things to replay as a stream. This does that change, but for now
all our streams have just one element.
Reviewed By: StanislavGlebik
Differential Revision: D20645156
fbshipit-source-id: fce7536d0ccbc1911335704816b71c17e80f2116
Summary:
We normally derive those lazily when accepting pushrebase, but we do derive
them eagerly in blobimport. For now, let's be consistent with blobimport.
This ensures that we don't lazily generate them, which would require read traffic,
and gives a picture a little more consistent with what an actual push would look like.
Reviewed By: ikostia
Differential Revision: D20623966
fbshipit-source-id: 2209877e9f07126b7b40561abf3e6067f7a613e6
Summary:
This makes it easier to realize if you used the wrong entry ID when replaying
(instead of telling you the bookmark isn't at `None` as expected, it tells you
the Hg Changeset could not be mapped to a Bonsai).
Reviewed By: ikostia
Differential Revision: D20623847
fbshipit-source-id: aaa66e7825f12373742efd4f779ae20ff21f0b46
Summary:
This updates unbundle_replay to account for pushrebase hooks, notably to assign
globalrevs.
To do so, I've extracted the creation of pushrebase hooks in repo_client and
reused it in unbundle_replay. I also had to update unbundle_replay to no longer
use `args::get_repo` since that doesn't give us access to the config (which we
need to know what pushrebase hooks to enable).
Reviewed By: ikostia
Differential Revision: D20622723
fbshipit-source-id: c74068c920822ac9d25e86289a28eeb0568768fc
Summary:
This adds a unbundle_replay Rust binary. Conceptually, this is similar to the
old unbundle replay Python script we used to have, but there are a few
important differences:
- It runs fully in-process, as opposed to pushing to a Mononoke host.
- It will validate that the pushrebase being produced is consistent with what
is expected before moving the bookmark.
- It can find sources to replay from the bookmarks update log (which is
convenient for testing).
Basically, this is to writes and to the old unbundle replay mechanism what
Fastreplay is to reads and to the traffic replay script.
There is still a bit of work to do here, notably:
- Make it possible to run this in a loop to ingest updates iteratively.
- Run hooks.
- Log to Scuba!
- Add the necessary hooks (notably globalrevs)
- Set up pushrebase flags.
I would also like to see if we can disable the presence cache here, which would
let us also use this as a framework for benchmarking work on push performance,
if / when we need that.
Reviewed By: StanislavGlebik
Differential Revision: D20603306
fbshipit-source-id: 187c228832fc81bdd30f3288021bba12f5aca69c
