Summary:
Fixed the path to generate the binary in order to able to use repo_import as command.
Changed contbuild location for publishing repo_import packman : mononoke contbuild uses the binary for testing, so it would be cheaper to just build there.
Reviewed By: StanislavGlebik
Differential Revision: D22526542
fbshipit-source-id: 926f8ed08169c34833ccc5711ec3fa26c3784615
Summary:
Increase the max limits for repo_list_bookmarks and commit_list_descendant_bookmarks
from 1,000 to 10,000. The higher number is still reasonable for a single request,
and this reduces the number of round-trips when there are lots of bookmarks.
This updates the server. A later diff will increase the constant so that clients
can make use of it, but this diff must be landed and deployed before that change
can be made.
Reviewed By: mitrandir77
Differential Revision: D22524892
fbshipit-source-id: c216f4ba7fa60774990d87747c9d8ea9d551dc85
Summary:
Follow up on this diff: D22432330 (b7817ffbd8)
Renamed xdiff functions to avoid linking issues when using both libgit2-sys and xdiff.
Reviewed By: farnz
Differential Revision: D22511368
fbshipit-source-id: e4be20e3112a8e8829298d5748657e9bdbde8588
Summary:
We already have a function to fetch all public changesets - let's use it
instead of re-implementing it.
The small caveat is that function in skiplist subcommand fetched all the
changesets (i.e. public and draft), so using bulkops function looks like a change in
behaviour. However it's actually the same - we index only public changesets for
skiplists anyway.
Reviewed By: krallin
Differential Revision: D22499940
fbshipit-source-id: ac8ad7d2b6ff0208e830a344877d7d2e93693abc
Summary:
If a particular is blob is too popular, we can saturate a LFS host through
consistent routing, and possibly OOM the host as well.
Historically, we haven't had enough traffic to LFS to make this a problem, but
we're getting there now.
This diffs adds support for reporting the popularity of a blob through SCS (not
Mononoke SCS — the couting one), and for using this popularity to identify when
we should stop consistently-routing a given blob.
The idea is that if e.g. something was requested 300 times in the last 20
seconds, it'll take a second for all the hosts to have it in cache, so we might
as well distribute this load.
There are plenty of things we could do slightly better here, such as making the
interval configurable, or having something in-between "consistently route to a
single host" and "don't consistently route at all". That said, I don't think
those are necessary right now, so let's start simple and find out.
Reviewed By: HarveyHunt
Differential Revision: D22503748
fbshipit-source-id: 48827bcfb7658ad22c88a8433359e29b0d56ad5a
Summary:
Computing descendant bookmarks is done async-but-serially (i.e. in a single
tokio Task). That means it is slow for repos with many bookmarks if you make a
request for an early commit.
This change makes it use multiple tokio Tasks, each one computing a batch of 100
bookmarks, giving true parallelism.
Reviewed By: krallin
Differential Revision: D22461931
fbshipit-source-id: a8908f7c20173c61b83d69c9dc37a5275937e2dc
Summary: After we shifted the bonsaichangesets (D22307039 (f5db2fdcdc)), we want to derive all the data types available in the target repo. Previously, we used a script to specify what we want to derive (https://fburl.com/wiki/ivk76muf step 4). This diff adds this functionality to repo_import and automate this step.
Reviewed By: StanislavGlebik
Differential Revision: D22455297
fbshipit-source-id: b38ac68606687350ace57d68464e68ca8229f7a5
Summary:
Logs showed that one of the repo is missing a bookmark, however it wasn't clear
which repo. Let's log it to make it clearer.
Reviewed By: HarveyHunt
Differential Revision: D22500793
fbshipit-source-id: c1d5fce66d7b2b119c7365d13511a7e9a6d6ed3f
Summary:
As part of modernising MultiplexedBlobstore, I want to fully asyncify the blobstore_sync_queue; that means I need this fully asyncified.
Fully asyncify everything but the bits that interact with blobstore_sync_queue; those have to wait for MultiplexedBlobstore to be asyncified
End goal is to reduce the number of healer overloads, by adding a mode of operation in which writes (e.g. from backfills or derived data) can avoid a sync queue write when all blobstores are working
Reviewed By: StanislavGlebik
Differential Revision: D22460059
fbshipit-source-id: 5792c4a8daf17ffe99a04d792792f568c40fde37
Summary: I'm about to asyncify the healer - move 2/3rds of the file content (tests) into their own file.
Reviewed By: ikostia
Differential Revision: D22460166
fbshipit-source-id: 18c0dde5f582c4c7006e3f023816ac457d38234b
Summary: Stage 1 of a migration - next step is to make all users of this trait use new futures, and then I can come back, add lifetimes and references, and leave it modernised
Reviewed By: StanislavGlebik
Differential Revision: D22460164
fbshipit-source-id: 94591183912c0b006b7bcd7388a3d7c296e60577
Summary: Use `pub(crate)` visibility to share the `SelectBookmark` query between modules.
Reviewed By: StanislavGlebik
Differential Revision: D22464059
fbshipit-source-id: 269561f5ab936b730ce2052e50173134ce241ff8
Summary:
Remove the `repo_id` parameter from the `Bookmarks` trait methods.
The `repo_id` parameters was intended to allow a single `Bookmarks` implementation
to serve multiple repos. In practise, however, each repo has its own config, which
results in a separate `Bookmarks` instance for each repo. The `repo_id` parameter
complicates the API and provides no benefit.
To make this work, we switch to the `Builder` pattern for `SqlBookmarks`, which
allows us to inject the `repo_id` at construction time. In fact nothing here
prevents us from adding back-end sharing later on, as these `SqlBookmarks` objects
are free to share data in their implementation.
Reviewed By: StanislavGlebik
Differential Revision: D22437089
fbshipit-source-id: d20e08ce6313108b74912683c620d25d6bf7ca01
Summary:
The dbbookmarks crate is getting too large for a single file. Split it up into
a `store` module, which implements the bookmarks traits, and a `transaction`
module, which handles bookmark transactions.
Reviewed By: krallin
Differential Revision: D22437088
fbshipit-source-id: 629b62de151400cdbf56d502aef061df46c3da81
Summary:
Separate out the `BundleReplayData` from the `BookmarkUpdateReason` enum. There's
no real need for this to be part of the reason, and removing it means we can
abstract away the remaining dependency on Mercurial changeset IDs from
the main bookmarks traits.
Reviewed By: mitrandir77, ikostia
Differential Revision: D22417659
fbshipit-source-id: c8e5af7ba57d10a90c86437b59c0d48e587e730e
Summary: For populating the XDB blobstore, we'd like to copy data from Manifold - the easiest way to do that is to exploit MultiplexedBlobstore's scrub mode to copy data directly.
Reviewed By: krallin
Differential Revision: D22373838
fbshipit-source-id: 550a9c73e79059380337fa35ac94fe1134378196
Summary:
Previously backfill_batch_dangerous method was calling internal derive_impl() method
directly. That wasn't great (after all, we are calling a function whose name suggests it should only be called from inside derive data crate) and this diff changes it so that we call batch_derive() method instead.
This gives a few benefits:
1) We no longer call internal derive_impl function
2) It allows different types of derived data to override batching behaviour.
For example, we've already overriden it for fsnodes and next diff will override
it for blame as well.
To make it compatible with derive_impl() batch_derive() now accepts derive data mode and mapping
Reviewed By: krallin
Differential Revision: D22435044
fbshipit-source-id: a4d911606284676566583a94199195860ffe2ecf
Summary:
D22206317 (9a6ed4b6ca) added requesting of predecessor information for suspected primordials
by the successor ID. This allows recovery of earlier predecessors when partial
data upload resulted in the history of a commit being extended backwards.
Unfortunately, while the individual requests are fast, the combined request
using `OR` in SQL ended up being very slow for some requests.
Separate out the requests at the application level, and aggregate the results
by concatenating them. `collect_entries` already handles duplicates should any
arise.
Most of the time the successor query will very quickly return no rows, as
it only matters when history is extended backwards, which is expected to be
rare.
Reviewed By: ikostia
Differential Revision: D22456062
fbshipit-source-id: 1e6094b4ac1590a5824e9ae6ef48468766560188
Summary:
Bypass truncation-based transaction if narrow-heads is on.
The transaction abort still works logically because commit references stay
unchanged on abort.
Related EdenFS and Mononoke tests are updated. Mononoke tests probably
shouldn't rely on revlog / fncache implementation details in hg.
Reviewed By: DurhamG
Differential Revision: D22240186
fbshipit-source-id: f97efd60855467b52c9fb83e7c794ded269e9617
Summary:
The goal is to make it easier to implement unit tests, which depend on `LiveCommitSyncConfig`. Specifically, `scs` has a piece of code, which instantiates `mononoke_api::Repo` with a test version of `CommitSyncConfig`. To migrate it to `LiveCommitSyncConfig`, I need to be able to create a test version of that. It **is** possible now, but would require me to turn a supplied instance of `CommitSyncConfig` back into `json`, which is cumbersome. Using a `dyn LiveCommitSyncConfig` there, instead of a concrete struct seems like a good idea.
Note also that we are using this technique in many places: most (all?) of our DB tables are traits, which we then implement for SQL-specific structs.
Finally, this diff does not actually migrate all of the current users of `LiveCommitSyncConfig` (the struct) to be users of `LiveCommitSyncConfig` (the trait), and instead makes them use `CfgrLiveCommitSyncConfig` (the trait impl). The idea is that we can migrate bits to use traits when needed (for example, in an upcoming `scs` diff). When not needed, it's fine to use concrete structs. Again, this is already the case in a a few places: we sometimes use `SqlSyncedCommitMapping` struct directly, instead of `T: SyncedCommitMapping` or `dyn SyncedCommitMapping`.
Reviewed By: StanislavGlebik
Differential Revision: D22383859
fbshipit-source-id: 8657fa39b11101684c1baae9f26becad6f890302
Summary:
This updates the AsyncRead implementations we use in hgproto and
mercurial_bundles to use a LimitedAsyncRead. The upshot of this change is that
we eliminate O(N^2) behavior when parsing the data we receive from clients.
See the earlier diff on this stack for more detail on where this happens, but
the bottom line is that Framed presents a full-size buffer that we zero out
every time we try to read data. With this change, the buffer we zero out is
comparable to the amount of data we are reading.
This matters in commit cloud because bundles might be really big, and a single
big bundle is enough to take an entire core for a spin or 20 minutes (and they
achieve nothing but time out in the end). That being said, it's also useful for
non-commit cloud bundles: we do occasionally receive big bundles (especially
for WWW codemods), and those will benefit from the exact same speedup.
One final thing I should mention: this is all in a busy CPU poll loop, and as I noted
in my earlier diff, the effect persists across our bundle receiving code. This means
it will sometimes result in not polling other futures we might have going.
Reviewed By: farnz
Differential Revision: D22432350
fbshipit-source-id: 33f1a035afb8cdae94c2ecb8e03204c394c67a55
Summary:
I want to update the health check to stop averaging averages (like in
D22394014). To do this, I need those counters.
Reviewed By: ahornby
Differential Revision: D22410196
fbshipit-source-id: aa5cbfe6607be3b953887f1639e1de54baac7389
Summary:
Just knowing the number of fetched undesired files doesn't give the full
picture. e.g. fetching lots of small files is better than fetching single
multi-Gb file.
So knowing the size of files is helpful
Reviewed By: krallin
Differential Revision: D22408400
fbshipit-source-id: 7653c1cdceccf50aeda9ce8a4880ee5178d4b107
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.
Reviewed By: dtolnay
Differential Revision: D22403809
fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
Summary:
Sometimes you want to fetch a file. Using curl and the LFS server works, but
this really should be part of Mononoke admin.
Reviewed By: ikostia
Differential Revision: D22397472
fbshipit-source-id: 17decf4aa2017a2c1be52605a254692f293d1bcd
Summary:
This got broken when we moved to Tokio 0.2. Let's fix it and add a test to make
sure it does not regress.
Reviewed By: ikostia
Differential Revision: D22396261
fbshipit-source-id: a8359aee33b4d6d840581f57f91af6c03125fd6a
Summary:
This diff adds two new bits of functionality to `LiveCommitSyncConfig`:
- getting all possible versions of `CommitSyncConfig` for a given repo
- getting `CommitSyncConfig` for a repo by version name
These bits are meant to be used in:
- `commit_validator` and `bookmarks_validator`, which would
need to run validation against a specific config version
- `mononoke_admin`, which would need to be able to query all versions,
display the version used to sync two commits and so on
Reviewed By: StanislavGlebik
Differential Revision: D22235381
fbshipit-source-id: 42326fe853b588849bce0185b456a5365f3d8dff
Summary:
This updates the virtually_sharded_blobstore to deduplicate puts only if the
data being put is actually the data we have put in the past. This is done by
keeping track of the hash of things we've put in the presence cache.
This has 2 benefits:
- This is safer. We only dedupe puts we 100% know succeeded (because this
particular instance was the one to attempt the put).
- This is creates less surprises, notably it lets us overwrite data in the
backing store (if we are writing something different).
Reviewed By: StanislavGlebik
Differential Revision: D22392809
fbshipit-source-id: d76a49baa9a5749b0fb4865ee1fc1aa5016791bc
Summary:
Running those on my devserver, I noticed they can be a bit flaky. They're are
racy on the purpose, but let's relax them a bit.
We have a lot of margin here — our blobstore is rate limited at once request
every 10ms, and we need to do 100 requests (the goal is to show that they don't
all wait), so 100ms is fine to prove that they're not rate limited when sharing
the same data.
Reviewed By: StanislavGlebik
Differential Revision: D22392810
fbshipit-source-id: 2e3c9cdf19b0e4ab979dfc000fbfa8da864c4fd6
Summary:
When we look up how a commit was synced, we frequently need to know which version of `CommitSyncConfig` was used to sync it. Specifically, this is useful for admin tooling and commit validator, which I am planning to migrate to use versioned `CommitSyncConfig` in the near future.
Later I will also include this information into `RewrittenAs` variant of `CommitSyncOutcome`, so that we expose it to real users. I did not do it in this diff to keep is small and easy to review. And because the other part is not ready :P
Reviewed By: StanislavGlebik
Differential Revision: D22255785
fbshipit-source-id: 4312e9b75e2c5f92ba018ff9ed9149efd3e7b7bc
Summary: When I've implemented this method I didn't test it for preserving the order of the input changesets and I've noticed my mistake when I was testing the scmquery part.
Reviewed By: StanislavGlebik
Differential Revision: D22374981
fbshipit-source-id: 4529f01370798377b27e4b6a706fc192a1ea928e
Summary:
Add the `scsc list-bookmarks` command, which lists bookmarks in a repository.
If a commit id is also provided, `list-bookmark` will be limited to bookmarks that
point to that commit of one of its descendants.
Reviewed By: mitrandir77
Differential Revision: D22361240
fbshipit-source-id: 17067ba47f9285b8137a567a70a87fadcaabec80
Summary:
There is inevitably interaction between caching, deduplication and rate
limiting:
- You don't want the rate limiting to be above caching (in the blobstore stack,
that is), because you shouldn't rate limits cache hits (this is where we are
today).
- You don't want the rate limiting to below deduplication, because then you get
priority inversion where a low-priority rate-limited request might hold the
semaphore while a higher-priority, non rate limited request wants to do the
same fetch (we could have moved rate limiting here prior to introducing
deduplication, but I didn't do it earlier because I wanted to eventually
introduce deduplication).
So, now that we have caching and deduplication in the same blobstore, let's
also incorporate rate limiting there!.
Note that this also brings a potential motivation for moving Memcache into this
blobstore, in case we don't want rate limiting to apply to requests before they
go to the _actual_ blobstore (I did not do this in this diff).
The design here when accessing the blobstore is as follows:
- Get the semaphore
- Check if the data is in cache, if so release the semaphore and return the
data.
- Otherwise, check if we are rater limited.
Then, if we are rate limited:
- Release the semaphore
- Wait for our turn
- Acquire the semaphore again
- Check the cache again (someone might have put the data we want while we were
waiting).
- If the data is there, then return our rate limit token.
- If the data isn't there, then proceed to query the blobstore.
If we aren't rate limited, then we just proceed to query the blobstore.
There are a couple subtle aspects of this:
- If we have a "late" cache hit (i.e. after we waited for rate limiting), then
we'll have waited but we won't need to query the blobstore.
- This is important when a large number of requests from the same key
arrive at the same time and get rate limited. If we don't do this second
cache check or if we don't return the token, then we'll consume a rate
limiting token for each request (instead of 1 for the first request).
- If a piece of data isn't cacheable, we should treat it like a cache hit with
regard to semaphores (i.e. release early), but like a miss with regard to
rate limits (i.e. wait).
Both of those are addressed captured in the code by returning the `Ticket` on a
cache hit. We can then choose to either return the ticket on a cache hit, or wait
for it on a cache miss.
(all of this logic is captured in unit tests, we can remove any of the blocks
there in `Shards::acquire` and a test will fail)
Reviewed By: farnz
Differential Revision: D22374606
fbshipit-source-id: c3a48805d3cdfed2a885bec8c47c173ee7ebfe2d
Summary:
Sometimes we take a token then realize we don't want it. In this case, giving it back is convenient.
This adds this!
Reviewed By: farnz
Differential Revision: D22374607
fbshipit-source-id: ccf47e6c75c37d154704645c9e826f514d6f49f6
Summary:
This is a mirror image of a diff, which made backsyncer use `LiveCommitSyncConfig`: we want to use configerator-based live configs, when we run in the continuous tailing mode.
As no-op iteration time used to be 10s and that's a bit wasteful for tests, this diff changes it to be configurable.
Finally, because of instantiating various additional `CommitSyncerArgs` structs, this diff globs out some of the `using repo` logs (which aren't very useful as test signals anyway, IMO).
Reviewed By: StanislavGlebik
Differential Revision: D22209205
fbshipit-source-id: fa46802418a431781593c41ee36f468dee9eefba
Summary: This diff actually start to use the option
Reviewed By: krallin
Differential Revision: D22373943
fbshipit-source-id: fe23da9c3daa1f9f91a5ee5e368b33e0091aa9c1
Summary:
Previously if a blame request was rejected (e.g. because a file was too large)
then we returned BlameError::Error.
This doesn't look correct, because there's BlameError::Rejected. This diff
makes it so that fetch_blame function returns BlameError::Rejected
Reviewed By: aslpavel
Differential Revision: D22373948
fbshipit-source-id: 4859809dc315b8fd66f94016c6bd5156cffd7cc2
Summary:
In the next diffs we'll need to read override_blame_filesize_limit from derived
data config, and this config is stored in BlobRepo.
this diff makes a small refactoring to pass BlobRepo to fetch_full_file_content
Reviewed By: krallin
Differential Revision: D22373946
fbshipit-source-id: b209abce82c0279d41173b5b25f6761659a92f3d
Summary: This will make adding blame file size limit override the next diffs easier
Reviewed By: krallin
Differential Revision: D22373945
fbshipit-source-id: 4857e43c5d80596340878753ea90bf31d7bb3367
Summary:
We're always yielding zero or one child during traversal, bounded traversal is
unnecessary here
Differential Revision: D22242148
fbshipit-source-id: b4c8a1279ef7bd15e9d0b3b2063683f45e30a97a
Summary:
Let's use new option in CLI. Unfortunately we can't easily accept commit ids in
named params so it has to be a postional one.
Differential Revision: D22234412
fbshipit-source-id: a9c27422fa65ae1c42cb1c243c7694507a957437
Summary:
If anything were to go wrong, we'd be happy to know which puts we ignored. So,
let's log them.
Reviewed By: farnz
Differential Revision: D22356714
fbshipit-source-id: 5687bf0fc426421c5f28b99a9004d87c97106695
Summary:
Eventually, I plan to make this the default, but for now I'd like to make it
something we can choose to turn on or off as a cmd argument (so we can start
with the experimental tier and Fastreplay).
Note that this mixes volatile vs. non-volatile pools when accessing the pools
for cacheblob. In practice, those pools are actually volatile, it's just that
things don't break if you access them as non-volatile.
Reviewed By: farnz
Differential Revision: D22356537
fbshipit-source-id: 53071b6b21ca5727d422e10f685061c709114ae7
Summary:
I canaried this on Fastreplay, but unfortunately that showed that sometimes we
just deadlock, or get so slow we might as well be deadlocked (and it happens
pretty quickly, after ~20 minutes). I tried spawning all the `get()` futures,
and that fixes the problem (but it makes gettreepack noticeably slower), so
that suggests something somewhere is creating futures, polling them a little
bit, then never driving them to completion.
For better or worse, I'd experienced the exact same problem with the
ContextConcurrencyBlobstore (my initial attempt at QOS, which also used a
semaphore), so I was kinda expecting this to happen.
In a sense, this nice because I we've suspected there were things like that in
the codebase for a while (e.g. with the occasional SQL timeout we see where it
looks like MySQL responds fast but we don't actually poll it until past the
timeout), and it gives us a somewhat convenient repro.
In another sense, it's annoying because it blocks this work :)
So, to work around the problem, for now, let's spawn futures to force the work
to complete when a semaphore is held. I originally had an unconditional spawn
here, but that is too expensive for the cache-hit code path and slows things
down (by about ~2x).
However, having it only if we'll query the blobstore isn't not as expensive,
and that seems to be fine (in fact it is a ~20% p99 perf improvement,
though the exact number depends on the number of shard we use for this, which I've had to tweak a bit).
https://pxl.cl/1c18H
I did find what I think is one potential instance of this problem in
`bounded_traversal_stream`, which is that we never try to poll `scheduled` to
completion. Instead, we just poll for the next ready future in our
FuturesUnordered, and if that turns out to be synchronous work then we'll just
re-enqueue more stuff (and sort of starve async work in this FuturesUnordered).
I tried updating bounded traversal to try a fairer implementation (which polls
everything), but that wasn't sufficient to make the problem go away, so I think
this is something we have to just accept for now (note that this actually has
some interesting perf impact in isolation: it's a free ~20% perf improvement on
p95+: https://pxl.cl/1c192
see 976b6b92293a0912147c09aa222b2957873ef0df if you're curious
Reviewed By: farnz
Differential Revision: D22332478
fbshipit-source-id: 885b84cda1abc15c51fbc5dd34473e49338e13f4
Summary: Like it says in the title. Those are useful!
Reviewed By: farnz
Differential Revision: D22332479
fbshipit-source-id: f9bddad75fcbed2593c675f9ba45965bd87f1575
Summary:
The goal of this blobstore is to dedupe reads by waiting for them to finish and
hit cache instead (and also to dedupe writes, but that's not relevant here).
However, this is not a desirable feature if a blob cannot be stored in cache,
because then we're serializing accesses for no good reason. So, when that
happens, we store "this cannot be stored in cache", and we release reads
immediately.
Reviewed By: farnz
Differential Revision: D22285269
fbshipit-source-id: be7f1c73dc36b6d58c5075172e5e3c5764eed894
Summary:
I'm going to store things that aren't quite the exact blobs in here, so on the
off chance that we somehow have two caching blobstores (the old one and this
one) that use the same pools, we should avoid collisions by using a prefix.
And, since I'm going to use a prefix, I'm adding a newtype wrapper to not use
the prefixed key as the blobstore key by accident.
Differential Revision: D22285271
fbshipit-source-id: e352ba107f205958fa33af829c8a46896c24027e
Summary:
This introduces a caching blobstore that deduplicates reads and writes. The
underlying motivation is to improve performance for processes that might find
themsleves inadvertently reading the same data concurrently from a bunch of
independent callsites (most of Mononoke), or writing the same bit of data over
and over again.
The latter is particularly useful for things like commit cloud backfilling in
WWW, where some logger commits include the same blob being written hundreds or
thousands of times, and cause us to overload the underlying Zippy shard in
Manifold. This is however a problem we've also encountered in the past in e.g.
the deleted files manifest and had to solve there. This blobstore is a little
different in the sense that it solves that problem for all writers.
This comes at the cost of writes being dropped if they're known to be
redundant, which prevents updates through this blobstore. This is desirable for
most of Mononoke, but not all (notably, for skiplist updates it's not great).
For now, I'm going to add this behind an opt-in flag, and later on I'm planning
to make it opt-out and turn it off there (I'm thinking to use the CoreContext
for this).
Reviewed By: farnz
Differential Revision: D22285270
fbshipit-source-id: 4e3502ab2da52a3a0e0e471cd9bc4c10b84a3cc5
Summary: This allowed me to compare two alternative approaches to queue draining, and generally seems like a useful thing to do.
Reviewed By: krallin
Differential Revision: D22364733
fbshipit-source-id: b6c76295c85b4dec6f0bfd7107c30bb4e4a28942
Summary: It's useful to derive all enabled derived data at once
Reviewed By: krallin
Differential Revision: D22336338
fbshipit-source-id: 54bc27ab2c23c175913fc02e6bf05d18a54c249c
Summary:
We've recently added an option to perform a stack move in megarepolib. A "stack
move" it's a stack of commits that move a files according to a mover. Now let's
expose it in the megarepotool
Reviewed By: ikostia
Differential Revision: D22312486
fbshipit-source-id: 878d4b2575ed2930bbbf0b9b35e51bb41393e622
Summary:
Implement `commit_list_descendant_bookmarks` by iterating over all bookmarks and
checking if the target commit is an ancestor of the bookmark's current target.
Reviewed By: mitrandir77
Differential Revision: D22357988
fbshipit-source-id: e1b1d7387742ba7133370f52c4d36c0b1a77f4e3
Summary:
Currently the `move_bookmark` API needs to get the old bookmark location in order
to move the bookmark. We'll fix that in general later, but for now we need to
make sure the value we use doesn't come from an out-of-date cache (e.g. the
warm_bookmarks_cache), as it may prevent the move from working.
Reviewed By: krallin
Differential Revision: D22358467
fbshipit-source-id: 4d46a6be717644b24663318326fdcd81249481c9
Summary:
Bookmark requests that are truncated because the requested limit is reached now return a `continue_after` value, containing the last bookmark that was processed.
Callers can make a subsequent request with the same parameters, but `after` set to the value received in `continue_after` to continue their request where it left off.
Reviewed By: krallin
Differential Revision: D22338301
fbshipit-source-id: 81e398bee444e0960e65dc3b4cdbbe877aff926d
Summary:
Add `commit_list_descendant_bookmarks` which will list all bookmarks that are
descendants of a particular commit.
We will also use this opportunity to complete the implementation of pagination
for regular bookmark listing, so add the appropriate fields to the
`repo_list_bookmarks` request and response structs.
Reviewed By: StanislavGlebik
Differential Revision: D22338300
fbshipit-source-id: defd019795c2a2ac9e5573d58de187c10848397f
Summary:
Add a new parameter, `pagination`, to the `list` method of the `Bookmarks` trait.
This restricts the returned bookmarks to those lexicographically after the
given bookmark name (exclusive). This can be use to implement pagination:
callers can provide the last bookmark in the previous page to fetch the
next page of bookmarks.
Reviewed By: krallin
Differential Revision: D22333943
fbshipit-source-id: 686df545020d936095e29ae5fee24258511f4083
Summary:
Rework the bookmarks traits:
* Split out log functions into a separate `BookmarkUpdateLog` trait. The cache doesn't care about these methods.
* Simplify `list` down to a single method with appropriate filtering parameters. We want to add more filtering types, and adding more methods for each possible combination will be messier.
* The `Bookmarks` and `BookmarkUpdateLog` traits become `attributes` on `BlobRepo`, rather than being a named member.
Reorganise the bookmarks crate to separate out the bookmarks log and transactions into their own modules.
Reviewed By: krallin
Differential Revision: D22307781
fbshipit-source-id: 4fe514df8b7ef92ed3def80b21a16e196d916c64
Summary:
The LIKE pattern used by bookmark prefixes needs to be escaped, otherwise
users looking for bookmarks containing `\`, `_` or `%` will get the
wrong results.
Reviewed By: krallin
Differential Revision: D22336716
fbshipit-source-id: 99b0ad6097f096358e66042752e4d153359935be
Summary: We were monitoring the wrong lag so far.
Reviewed By: farnz
Differential Revision: D22356455
fbshipit-source-id: abe41a4154c2a8d53befed4760e2e9544797c845
Summary:
`bulk_add()` method was checking for conflicts correctly i.e. it wouldn't fail
if we try to insert the same mapping twice.
`bulk_add_git_mapping_in_transaction` wasn't doing this check i.e. it would
fail.
This caused us a few problems and this diff fixes them - now
`bulk_add_git_mapping_in_transaction` would do the same checks as bulk_add was
doing previously.
There is another important change in behaviour: if we try to insert two entries, one of them
has a conflict another don't then previously we'd insert the second entry.
Now we don't insert any, arguably that's a preferred behaviour.
Reviewed By: krallin
Differential Revision: D22332001
fbshipit-source-id: 86fff8c23c43eeca0fb36b01b10cdaa73b3ce4ab
Summary:
EdenAPI's `make_req` tools allows developers to create ad-hoc CBOR request payloads for debugging purposes (e.g., for use with `curl`). The tool generates requests from human-created JSON, which are particularly useful in Mercurial and Mononoke's integration tests.
Later in this stack, the use of this JSON format will be extended beyond just this one tool. As such, it is important that the representation be sufficiently extensible so accommodate future changes to the request structs. In the case of the JSON representation of `DataRequest`, this means changing from an array to a single-attribute object, so that additional fields can potentially be added in the future.
Reviewed By: quark-zju
Differential Revision: D22319314
fbshipit-source-id: 5931bc7ab01ca48ceab5ffd1c9177dd3035b643c
Summary:
The (re)construction process for the IdMap will generate millions of rows
to be inserted in our database. We want to throttle the inserts so that
the database doesn't topple over.
Reviewed By: ikostia
Differential Revision: D22104349
fbshipit-source-id: 73b7c2bab12ae0cd836080bcf1eb64586116e70f
Summary:
Simple implementation that queries the MyAdmin service to fetch replication
lag.
Caching like in sqlblob::facebook::myadmin will probably come in a follow
up change.
Reviewed By: StanislavGlebik
Differential Revision: D22104350
fbshipit-source-id: fbd90174d528ddae4045e957c343e6c213f70d26
Summary:
ReplicaLagMonitor is aimed to generalize over different stategies of fetching
the replication lag in a SQL database. Querying a set of connections is one
such strategy.
Reviewed By: ikostia
Differential Revision: D22104348
fbshipit-source-id: bbbeccb55a664e60b3c14ee17f404982d09f2b25
Summary:
SQLite's `LIKE` operator is case insensitive by default. This doesn't match MySQL, and
also seems like a surprising default. Set the pragma on every connection to make it
case sensitive.
Reviewed By: farnz
Differential Revision: D22332419
fbshipit-source-id: 4f503eeaa874e110c03c27300467ddc02dc9b365
Summary:
Whether a bookmark is publishing or not is not specific to Mercurial - it also affects
whether a commit is draft, so it is interesting to the Bonsai world.
Rename `BookmarkHgKind` to `BookmarkKind` to make this clear.
Since negatives are more awkward to work with, rename `PublishingNotPullDefault` to
`Publishing` and `PullDefault` to `PullDefaultPublishing` to make it clearer that
pull-default bookmarks are also publishing.
We can't rename the database column, so that remains as `hg_kind`.
Reviewed By: StanislavGlebik
Differential Revision: D22307782
fbshipit-source-id: 9e686a98cc5eaf9af722fa62fac5ffd4844967fd
Summary:
Blobstore healer has a logic, which prevents it from doing busy work, when the
queue is empty. This is implemented by means of checking whether the DB query
fetched the whole `LIMIT` of values. Or that is the idea, at least. In
practice, here's what happens:
1. DB query is a nested one: first it gets at most `LIMIT` distinct
`operation_key` entries, then it gets all rows with such entries. In practice
this almost always means `# of blobstores * LIMIT` rows, as we almost always
succeed writing to every blobstore
2. Once this query is done, the rows are grouped by the `blobstore_key`, and a
future is created for each such row (for simplicity, ignore that future may not
be created).
3. We then compare the number of created futures with `LIMIT` and report an
incomplete batch if the numbers are different.
This logic has a flaw: same `blobstore_key` may be written multiple times with
different `operation_key` values. One example of this: `GitSha1` keys for
identical contents. When this happens, grouping from step 2 above will produce
fewer than `LIMIT` groups, and we'll end up sleeping for nothing.
This is not a huge deal, but let's fix it anyway.
My fix also adds some strictly speaking unnecessary logging, but I found it
helpful during this investigation, so let's keep it.
The price of this change is collecting two `unique_by` calls, both of which
allocates a temporary hash set [1] of the size `LIMIT * len(blobstore_key) * #
blobstores` (and another one with `operation_key`). For `LIMIT=100_000`
`len(blobstore_key)=255`, `# blobstores = 3` we have roughly 70 mb for the
larger one, which should be ok.
[1] https://docs.rs/itertools/0.9.0/itertools/trait.Itertools.html#method.unique
Reviewed By: ahornby
Differential Revision: D22293204
fbshipit-source-id: bafb7817359e2c867cf33c319a886653b974d43f
Summary:
Previous commit: D22233127 (fa1caa8c4e)
In this diff, I added rewrite commit path functionality using Mover https://fburl.com/diffusion/6rnf9q2f to repo_import.
Given a prefix (e.g. new_repo), we prepend the paths of the files extracted from the bonsaichangesets given by gitimport (e.g. folder1/file1 => new_repo/folder1/file1). Previously, we did this manually when importing a git repo (https://www.internalfb.com/intern/wiki/Mercurial/Admin/ImportingRepos/) using convert extension.
Reviewed By: StanislavGlebik
Differential Revision: D22307039
fbshipit-source-id: 322533e5d6cbaf5d7eec589c8cba0c1b9c79d7af
Summary: Also fix up the parser test that now fails with this change
Reviewed By: StanislavGlebik
Differential Revision: D22306340
fbshipit-source-id: 820aad48068471b03cbc1c42107c443bfa680607
Summary: This will be used for commits_between replacement
Differential Revision: D22234236
fbshipit-source-id: c0c8550d97a9e8b42034d605e24ff54251fbd13e
Summary: Some SCMQuery queries need just a list of commit hashes instead of full coverage.
Reviewed By: markbt
Differential Revision: D22165006
fbshipit-source-id: 9eeeab72bc4c88ce040d9d2f1a7df555a11fb5ae
Summary: This way we can go from list of changesets into changet ids that we're returning as an answer in few queries.
Differential Revision: D22165005
fbshipit-source-id: 4da8ab2a89be0de34b2870044e44d35424be5510
Summary: It can be useful in other places as well, not only in blobimport
Reviewed By: krallin
Differential Revision: D22307314
fbshipit-source-id: f7d8c91101edc2ed4f230f7ef6796e39fbea5117
Summary: Convert the bookmarks traits to use new-style `BoxFuture<'static>` and `BoxStream<'static>`. This is a step along the path to full `async`/`await`.
Reviewed By: farnz
Differential Revision: D22244489
fbshipit-source-id: b1bcb65a6d9e63bc963d9faf106db61cd507e452
Summary:
Enable narrow-heads.
Changed log revset from `:` to `all()` to make the test compatible.
Reviewed By: krallin
Differential Revision: D22200495
fbshipit-source-id: 148a82e77c953b9e7dbed055ef464c318e56cafa
Summary:
Enable narrow-heads, and mutation. Disable obsmarker related features.
Change phase manipulation to `debugmakepublic` which works with narrow-heads.
Reviewed By: krallin
Differential Revision: D22200511
fbshipit-source-id: 8dec050f137e6cc055015fe084eb4cc67faa1216
Summary:
Enable narrow-heads.
The test output seems a bit unstable - sometimes I got 28 there. So I globbed
it out.
Reviewed By: krallin
Differential Revision: D22200497
fbshipit-source-id: f005381a341d88c0bcbb09150e7d1878df7a38f3
Summary:
Enable narrow-heads.
Change the revset `:` to `all()`. With narrow-heads, `:` selects all commits
including those that are not referred by visible heads. The `all()` revset
only selects commits reachable from visible heads.
Reviewed By: krallin
Differential Revision: D22200498
fbshipit-source-id: beb863d42069ae898e419a4a75b3a707c72ae1f9
Summary:
Enable remotenames, selectivepull, and narrow-heads. Use the new stream clone
code path.
Selectivepull makes a difference. `hg pull -r HASH` also pulls the selected
bookmarks so an extra `pull` was unnecessary. Change the clone command to use
`-U` to trigger the new clone code path.
Reviewed By: krallin
Differential Revision: D22200499
fbshipit-source-id: 764202098c7e8afdbb5e2ee83679da7570c08c90
Summary:
Enable remotenames and narrow-heads.
Local bookmarks are replaced by remote bookmarks, causing the test change.
Reviewed By: krallin
Differential Revision: D22200500
fbshipit-source-id: aeee528d1766e0642c12e78a6c1a50cadc8a579a
Summary:
Enable remotenames and narrow-heads.
The commits become 'draft' because there are no remote bookmarks.
Reviewed By: krallin
Differential Revision: D22200514
fbshipit-source-id: 04d0befa7c22756e936a28ffdcdf1305057cf062
Summary:
Enable remotenames and narrow-heads.
The test was migrated cleanly. The only change is that local bookmarks are
replaced by remote bookmarks.
Reviewed By: krallin
Differential Revision: D22200510
fbshipit-source-id: f5b8cd2ed125e9fc4e5daac897851d91fef5693f
Summary:
Enable remotenames and narrow-heads.
Local bookmarks are replaced by remote bookmarks.
Reviewed By: krallin
Differential Revision: D22200503
fbshipit-source-id: 41ac4f4f606011dcaf6d0d9867b01fb77b9a79d8
Summary:
Enable remotenames and narrow-heads.
Phase exchange is gone because of narrow-heads.
The remtoenames extension was written suboptimally so it issued a second
bookmarks request (which, hopefully can be removed by having selective
pull everywhere and migrate pull to use the new API).
Reviewed By: krallin
Differential Revision: D22200506
fbshipit-source-id: c522bb9fc1396d813e0f1f380c4290445bab3db3
Summary:
Enable remotenames and narrow-heads. The `master_bookmark` is no longer a local
bookmark in the client repo.
Reviewed By: krallin
Differential Revision: D22200513
fbshipit-source-id: bc3c1715ce21f45a35bc67148eb00e44944bea6e
Summary:
Enable remotenames and narrow-heads. The server gets one more request from
remotenames.
Reviewed By: krallin
Differential Revision: D22200502
fbshipit-source-id: 26bc28b19438c7be4a19eae6be728c83b113f822
Summary:
Enable remotenames and narrow-heads. The client gets remote bookmarks instead
of local bookmarks during clone and phases are decided by remote bookmarks.
Reviewed By: krallin
Differential Revision: D22200515
fbshipit-source-id: 12a9e892855b3a8f62f01758565de5f224c4942b
Summary:
Change the template to show remote bookmarks, which will be more relevant once
we migrate to modern configs. Namely, phases will be decided by remote bookmarks.
The named branches logic was mostly removed from the code base. Therefore
drop the `{branches}` template.
Reviewed By: StanislavGlebik
Differential Revision: D22200512
fbshipit-source-id: 8eca3a71ff88b8614023f4920a448156fcd712d5
Summary:
Most tests pass without changes. Some incompatible tests are added to the
special list.
Reviewed By: krallin
Differential Revision: D22200505
fbshipit-source-id: 091464bbc7c9c532fed9ef91f2c955d6e4f2df0b
Summary: This is the final diff of the stack - it starts logging pushed commits to scribe
Reviewed By: farnz
Differential Revision: D22212755
fbshipit-source-id: ec09728408468acaeb1c214d43f930faac30899b
Summary:
Failing push if we failed to log to scribe doesn't make a lot of sense. By that
time the ship has sailed - commit has already been pushed and by failing the
request we can't undo that. It will just create an annoyance by whoever is
pushing.
Instead let's log it to scuba
Reviewed By: farnz
Differential Revision: D22256687
fbshipit-source-id: 2428bbf1db4cef6fa80777ad65184fab1804fa9c
Summary:
At the moment we can't test logging to scribe easily - we don't have a way to
mock it. Scribe are supposed to help with that.
They will let us to configure all scribe logs to go to a directory on a
filesystem similar to the way we configure scuba. The Scribe itself will
be stored in CoreContext
Reviewed By: farnz
Differential Revision: D22237730
fbshipit-source-id: 144340bcfb1babc3577026191428df48e30a0bb6
Summary: Many tests are incompatible. But many are passing.
Reviewed By: kulshrax
Differential Revision: D22052475
fbshipit-source-id: 1f30ac2b0fe034175d5ae818ec2be098dbd5283d
Summary:
For Lua hooks, we needed to know whether to run the hook per file, or per changeset. Rust hooks know this implicitly, as they're built-in to the server.
Stop having the tests set an unnecessary config
Reviewed By: krallin
Differential Revision: D22282799
fbshipit-source-id: c9f6f6325823d06d03341f04ecf7152999fcdbe7
Summary:
D21642461 (46d2b44c0e) converted Mononoke server to use the
`--mononoke-config-path` common argument style to select a config path.
Now that this change has been running for a while, remove the extra logic in
the server that allowed it to accept both the deprecated `--config_path / -P`
and the new arg.
Reviewed By: ikostia
Differential Revision: D22257386
fbshipit-source-id: 7da4ed4e0039d3659f8872693fa4940c58bae844
Summary:
`get_entry_with_small_repo_mapings` is a function that turns a `CommitEntry`
struct into `CommitEntryWithSmallReposMapped` struct - the idea being that this
function looks up hashes of commits into which the original commit from the
large repo got rewritten (in practice rewriting may have happened in the small
-> large direction, but it is not important for the purpose of this job). So it
establishes a mapping. Before this
diff, it actually established `Large<ChangesetId> ->
Option<(Small<ChangesetId>, Option<BookmarkName>)>` mapping, meaning that it
recorded into which bookmark large bookmark was rewritten. This was a useless
information (as evidenced by the fact that it was ignored by the
`prepare_entry` function, which turns `CommitEntryWithSmallReposMapped` into
`EntryPreparedForValidation`. It is useless because bookmarks are mutable and
it is impossible to do historic validation of the correctness of bookmark
renaming: bookmarks may have been correctly renamed when commits where pushes,
but they may be incorrectly renamed now and vice-versa. To deal with bookmarks,
we have a separate job, `bookmarks_validator`.
So this diff stops recording this useless information. As a bonus, this will
make migration onto `LiveCommitSyncConfig` easier.
Reviewed By: StanislavGlebik
Differential Revision: D22235389
fbshipit-source-id: c02b3f104a8cbd1aaf76100aa0930efeac475d42
Summary: We need to be able to query `synced_commit_mapping` to understand which `version_name` was used to sync commits. That `version_name` will be needed to produce `CommitSyncConfig` by utilizing upcoming `LiveCommitSyncConfig` APIs. And `CommitSyncConfig` is needed to create `CommitSyncer`. So let's extract this fn out of `CommitSyncer`, as it's an independent functionality really
Reviewed By: farnz
Differential Revision: D22244952
fbshipit-source-id: 53e55139efd423174176720c8bf7e3ecc0dcb0d7
Summary:
This diff migrates `backsyncer_cmd` (the thing that runs in the separate backsyncer job, as opposed to bakcsyncer, triggered from push-redirector) onto `LiveCommitSyncConfig`. Specifically, this means that on every iteration of the loop, which calls `backsync_latest` we reload `CommitSyncConfig` from configerator, build a new `CommitSyncer` from it, and then pass that `CommitSyncer` to `backsync_latest`.
One choice made here is to *not* create `CommitSyncer` on every iteration of the inner loop of `backsync_latest` and handle live configs outside. The reason for this is twofold:
- `backsync_latest` is called form `PushRedirector` methods, and `PushRedirector` is recreated on each `unbundle` using `LiveCommitSyncConfig`. That call provides an instance of `CommitSyncer` used to push-redirect a commit we want to backsync. It seems strictly incorrect to try and maybe use a different instance.
- because of some other consistency concerns (different jobs getting `CommitSyncConfig` updates at different times), any sync config change needs to go through the following loop:
- lock the repo
- land the change
- wait some time, until all the possible queues (x-repo sync and backsync) are drained
- unlock the repo
- this means that it's ok to have the config refreshed outside of `backsync_latest`
Reviewed By: farnz
Differential Revision: D22206992
fbshipit-source-id: 83206c3ebdcb2effad7b689597a4522f9fd8148a
Summary:
Before this diff only the main Mononoke server binary was able to use fs-based
`ConfigStore`, which is pretty useful in integration tests.
Reviewed By: farnz
Differential Revision: D22256618
fbshipit-source-id: 493a064a279250d01469c9ff7f747585581caf51
Summary: We designed the schema to make this simple to implement - it's literally a metadata read and a metadata write.
Reviewed By: ikostia
Differential Revision: D22233922
fbshipit-source-id: b392b4a3a23859c6106934f73ef60084cc4de62c
Summary:
Eventually, we want everything to be `async`/`await`; as a stepping stone in that direction, switch the remaining lobstore traits to new-style futures.
This just pushes the `.compat()` out to old-style futures, but it makes the move to non-'static lifetimes easier, as all the compile errors will relate to lifetime issues.
Reviewed By: krallin
Differential Revision: D22183228
fbshipit-source-id: 3fe3977f4469626f55cbf5636d17fff905039827
Summary:
This is to avoid passing `String` around. Will be useful in one of the next
diffs, where I add querying `LiveCommitSyncConfig` by versions.
Reviewed By: krallin
Differential Revision: D22243254
fbshipit-source-id: c3fa92b62ae32e06d7557ec486d211900ff3964f
Summary: I have previously moved the gitimport functionality (D22159880 (2cf5388835)) into a separate library, since repo_import shares similar behaviours. In this diff, I setup repo_import to be able to call gitimport to get the commits and changes. (Next steps include using Mover to set the paths of the files in the commits given by gitimport)
Reviewed By: StanislavGlebik
Differential Revision: D22233127
fbshipit-source-id: 4680c518943936f3e29d21c91a2bad60108e49dd
Summary:
Eventually, we want everything to be `async`/`await`; as a stepping stone in that direction, switch some of the blobstore interfaces to new-style `BoxFuture` with a `'static` lifetime.
This does not enable any fixes at this point, but does mean that `.compat()` moves to the places that need old-style futures instead of new. It also means that the work needed to make the transition fully complete is changed from a full conversion to new futures, to simply changing the lifetimes involved and fixing the resulting compile failures.
Reviewed By: krallin
Differential Revision: D22164315
fbshipit-source-id: dc655c36db4711d84d42d1e81b76e5dddd16f59d
Summary:
If the first client to send mutation data for a commit is only aware of partial
history for that commit, the primordial commit that is determined will be the
earliest of those commits. If another client comes along later with a longer
history, the new set of commits will be assigned a different primordial commit.
Make sure that when this happens, we still fetch the full history. We do this
by including the successor in the search-by-primordial case, which allows us
to join together disconnected histories at the cost of one extra round-trip to
the database.
Note that the fast path for addition of a single mutation will not fill in the
missing history. This is an acceptable trade-off for the faster performance
in the usual case.
Reviewed By: mitrandir77
Differential Revision: D22206317
fbshipit-source-id: 49141d38844d6cddc543b6388f0c31dbc70dcbc5
Summary:
By design, the mutation history of a commit should not have any cycles. However,
synthetic entries created by backfilling obsmarkers may inadvertently create
erroneous cycles, which must be correctly ignored by the mutation store.
The mutation store is affected by cycles in two ways:
* Self-referential entries (created by backfilling "revive" obsmarkers) must
be dropped early on, as these will overwrite any real mutation data for
that successor.
* Larger cycles will prevent determination of the primordial commit for
primordial optimization. Here we drop all entries that are part of the cycle.
These entries will not be shareable via the mutation store.
Note that it is still possible for cycles to form in the store if they are
added in multiple requests - the first request with a partial cycle will
allow determination of a primordial commit which is then used in subsequent
requests. That's ok, as client-side cycle detection will break the cycle in
these entries.
As we move away from history that has been backfilled from obsmarkers, this
will become less of a concern, as cycles in pure mutation data are impossible
to create.
Reviewed By: mitrandir77
Differential Revision: D22206318
fbshipit-source-id: a57f30a19c482c7cde01cbd26deac53b7bb5973f
Summary:
Push supported multiple bookmarks in theory, but in practice we never used it.
Since we want to start logging pushed commits in the next diffs we need to decide what to do with
bookmarks, since at the moment we can log only a single bookmark to scribe
let's just allow a single bookmark push
Reviewed By: farnz
Differential Revision: D22212674
fbshipit-source-id: 8191ee26337445ce2ef43adf1a6ded3e3832cc97
Summary:
In the next diffs it will be passed to unbundle processing so that we can use
scribe category to log pushed commits
Reviewed By: krallin
Differential Revision: D22212616
fbshipit-source-id: 17552bda11f102041a043f810125dc381e478611
Summary: That was like 50% of the point of this change, and somehow I forgot to do it.
Reviewed By: farnz
Differential Revision: D22231923
fbshipit-source-id: 4a4daaeaa844acd219680907c0b5a5fdacdf535c
Summary:
Similarly to how we have `PushRedirectorArgs`, we need `CommitSyncerArgs`: a struct, which a long-living process can own and periodically create a real `CommitSyncer` out of it, by consuming freshly reloaded `CommitSyncConfig`.
It is a little unfortunate that I am introducing yet another struct to `commit_rewriting/cross_repo_sync`, as it's already pretty confusing with `CommitSyncer` and `CommitSyncRepos`, but hopefully `CommitSyncerArgs`'s purpose is simple enough that it can be inferred from the name. Note that this struct does have a few convenience methods, as we need to access things like `target_repo` and various repo ids before we even create a real `CommitSyncer`. This makes it's purpose a little less singular, but still fine IMO.
Reviewed By: StanislavGlebik
Differential Revision: D22197123
fbshipit-source-id: e2d993e186075e33acec00200d2aab10fb893ffd
Summary:
This fn is not used anywhere except tests, and its only difference from
`backsync_all_latest` is in the fact that it accepts a limit. So let's rename
`backsync_all_latest` into `backsync_latest` and make it accept a limit arg.
I decided to use a custom enum instead of `Option` so that people don't have to
open fn definition to understand what `BacksyncLimit::Limit(2)` or
`BacksyncLimit::NoLimit` mean.
Reviewed By: StanislavGlebik
Differential Revision: D22187118
fbshipit-source-id: 6bd97bd6e6f3776e46c6031f775739ca6788ec8c
Summary:
This diff enables `unbundle` flow to start creating `push_redirector` structs from hot-reloaded `CommitSyncConfig` (by using the `LiveCommitSyncConfig` struct).
Using `LiveCommitSyncConfig` unfortunately means that we need to make sure those tests, which don't use standard fixtures, need to have both the `.toml` and the `.json` commit sync configs present, which is a little verbose. But it's not too horrible.
Reviewed By: StanislavGlebik
Differential Revision: D21962960
fbshipit-source-id: d355210b5dac50d1b3ad277f99af5bab56c9b62e
Summary:
`LiveCommitSyncConfig` is intended to be a fundamental struct, on which live push-redirection and commit sync config for push-redurector, x-repo sync job, backsyncer, commit and bookmark validators are based.
The struct wraps a few `ConfigStore` handles, which allows it to query latest values every time one of the public methods is called. Callers receive parsed structs/values (`true`/`false` for push redirection config, `CommitSyncConfig` for the rest), which they later need to use to build things like `Mover`, `BookmarkRenamer`, `CommitSyncer`, `CommitRepos` and so on. For now the idea is to rebuild these derived structs every time, but we can later add a memoization layer, if the overhead is going to be large.
Reviewed By: StanislavGlebik
Differential Revision: D22095975
fbshipit-source-id: 58e1f1d8effe921b0dc264fffa785593ef188665
Summary:
This diff does three things:
- moves existing `CommitSyncConfig` validation from `config.rs` into
`convert/commit_sync.rs`, so that any user of `impl Convert for
RawCommitSyncConfig` gets it for free
- adds another thing to validate `CommitSyncConfig` against (large repo is one
of the small repos)
- adds `RawCommitSyncConfig` validation for something that can be lost when
looking at `CommitSyncConfig` (no duplicate small repos).
Reviewed By: markbt
Differential Revision: D22211897
fbshipit-source-id: a9820cc8baf427da66ce7dfc943e25eb67e1fd6e
Summary: When running integration tests we should make the paths absolute, but kept it relative so far. This results it breaking the tests.
Reviewed By: krallin
Differential Revision: D22209498
fbshipit-source-id: 54ca3def84abf313db32aecfac503c7a42ed6576
Summary:
If we don't read the body for a response, then Hyper cannot return the
connection to the pool. So, let's do it automatically upon dropping. This will
typically happen when we send a request to upstream then don't read the
response.
I seem to remember this used to work fine at some point, but looking at the
code I think it's actually broken now and we don't reuse upstream connections
if we skip waiting for upstream in a batch request. So, let's fix it once and
for all with a more robust abstraction.
Reviewed By: HarveyHunt
Differential Revision: D22206742
fbshipit-source-id: 2da1c008556e1d964c1cc337d58f06f8d691a916
Summary:
This was old Tokio 0.1 code that needed channels for spawns, but in 0.2 that
actually is built-in to tokio::spawn, so let's use this.
Reviewed By: HarveyHunt
Differential Revision: D22206738
fbshipit-source-id: 8f89ca4f7afc8dd888fe289e8d597148976cc54c
Summary:
This fixes a bit of a tech debt item in the LFS Server. We've had this
discard_stream functon for a while, which was necessary because if you just
drop the data stream, you get an error on the sending end.
This makes the code more complex than it needs to be, since you need to always
explicitly discard data streams you don't want instead of just dropping them.
This fixes that by letting us support a sender that tolerates the receiver
being closed, and just ignores those errors.
Reviewed By: HarveyHunt
Differential Revision: D22206739
fbshipit-source-id: d209679b20a3724bcd2e082ebd0d2ce10e9ac481
Summary:
We have a lot of integration tests for LFS, but a handful of unit tests don't
hurt for some simpler changes. Let's make it easier to write those.
Reviewed By: HarveyHunt
Differential Revision: D22206741
fbshipit-source-id: abcb73b35c01f28dd54cc543cd0a746327d3787b
Summary:
This diff is probably going to sound weird ... but xavierd and I both think
this is the best approach for where we are right now. Here is why this is
necessary.
Consider the following scenario
- A client creates a LFS object. They upload it to Mononoke LFS, but not
upstream.
- The client shares this (e.g. with Sandcastle), and includes a LFS pointer.
- The client tries to push this commit
When this happens, the client might not actually have the object locally.
Indeed, the only pieces of data the client is guaranteed to have is
locally-authored data.
Even if the client does have the blob, that's going to be in the hgcache, and
uploading from the hgcache is a bit sketchy (because, well, it's a cache, so
it's not like it's normally guaranteed to just hold data there for us to push
it to the server).
The problem boils down to a mismatch of assumptions between client and server:
- The client assumes that if the data wasn't locally authored, then the server
must have it, and will never request this piece of data again.
- The server assumes that if the client offers a blob for upload, it can
request this blob from the client (and the client will send it).
Those assumptions are obviously not compatible, since we can serve
not-locally-authored data from LFS and yet want the client to upload it, either
because it is missing in upstream or locally.
This leaves us with a few options:
- Upload from the hg cache. As noted above, this isn't desirable, because the
data might not be there to begin with! Populating the cache on demand (from
the server) just to push data back to the server would be quite messy.
- Skip the upload entirely, either by having the server not request the upload
if the data is missing, by having the server report that the upload is
optional, or by having the client not offer LFS blobs it doens't have to the
server, or finally by having the client simply disobey the server if it
doesn't have the data the server is asking for.
So, why can we not just skip the upload? The answer is: for the same reason we
upload to upstream to begin with. Consider the following scenario:
- Misconfigured client produces a commit, and upload it to upstream.
- Misconfigured client shares the commit with Sandcastle, and includes a LFS
pointer.
- Sandcastle wants to push to master, so it goes to check if the blob is
present in LFS. It isn't (Mononoke LFS checks both upstream and internal, and
only finds the blob in upstream, so it requests that the client submit the
blob), but it's also not not locally authored, so we skip the push.
- The client tries to push to Mononoke
This push will fail, because it'll reference LFS data that is not present in
Mononoke (it's only in upstream).
As for how we fix this: the key guarantee made by our proxying mechanism is
that if you write to either LFS server, your data is readable in both (the way
we do this is that if you write to Mononoke LFS, we write it to upstream too,
and if you write to upstream, we can read it from Mononoke LFS too).
What does not matter there is where the data came from. So, when the client
uploads, we simply let it submit a zero-length blob, and if so, we take that to
mean that the client doesn't think it authored data (and thinks we have it), so
we try to figure out where the blob is on the server side.
Reviewed By: xavierd
Differential Revision: D22192005
fbshipit-source-id: bf67e33e2b7114dfa26d356f373b407f2d00dc70
Summary:
Due to Thrift design of "include" statements in fbcode the thrift structures has to be contained in folders that are identical to the folder layout inside fbcode.
This diff changes the folder layout on Cargp.toml files and in fbcode_builder, there will be a next diff that changes this for ShipIt as well.
Reviewed By: ikostia
Differential Revision: D22208707
fbshipit-source-id: 65f9cafed2f0fcc8398a3887dfa622de9e139f68
Summary:
If a commit changes modes (i.e. executable, symlink or regular) of a lot of files but
doesn't change their content then we don't need to put these filenodes to the
generated bundle. Mercurial stores mode in manifest, so changing the mode
doesn't change the filenode.
Reviewed By: ikostia
Differential Revision: D22206736
fbshipit-source-id: f64ee8a34281cd207c92653b927bf9109ccbe1b4
Summary:
I landed D22118926 (e288354caf) yesterday expecting those messages at about the same time
xavierd landed D21987918 (4d13ce1bcc), which removed them. This removes them from the
tests.
Reviewed By: StanislavGlebik
Differential Revision: D22204980
fbshipit-source-id: 6b1d696c93a07e942f86cd8df9a8e43037688728
Summary:
The Rust store code has been enabled everywhere for a few weeks now, let's
enable it by default in the code. Future changes will remove the config as well
as all the code associated with the non Rust store code.
The various tests changes are due to small difference between the Rust code and
the Python one, the biggest one being it's handling of corrupted packfiles. The
old code silently ignored them, while the new one errors out for local
packfiles. The test-lfs-bundle.t difference is just due to an ordering
difference between Python and Rust.
Reviewed By: kulshrax
Differential Revision: D21985744
fbshipit-source-id: 10410560193476bc303a72e7583f84924a6de820
Summary: Let's not heal 10000 blobs in parallel, that's a little too much data.
Reviewed By: farnz
Differential Revision: D22186543
fbshipit-source-id: 939fb5bc83b283090e979ac5fe3efc96191826d3
Summary:
If we're going to iterate through the whole manifest, we should probably
prefetch it. Otherwise, we might end up doing a whole lot of sequential
fetching. We saw this this week when a change landed in sparse profiles that
caused requests to Mononoke to increase 100-fold.
Unfortunately, I don't think we can selectively only fetch the things we are
missing, so this just goes ahead and fetches everything unconditionally. If
there is a better way to do this, I'm all ears.
Reviewed By: StanislavGlebik, xavierd
Differential Revision: D22118926
fbshipit-source-id: f809fa48a7ff7b449866b42b247bf1da30097caa
Summary: This got broken in D22115015 — this fixes it.
Reviewed By: farnz
Differential Revision: D22186138
fbshipit-source-id: 54c05466cdbd3be4f6887a852f099351ea5e891e
Summary: This will be used in the following diffs. It just adds commitsync fixtures in a single place, so that we can later play with them in integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D21952665
fbshipit-source-id: 2933a9f7ea8343d5d52e6c3207e7d78a3ef0be25
Summary: `HgPhase` type is redundant and was adding dependency on mercurial in phases crate.
Reviewed By: farnz
Differential Revision: D22162716
fbshipit-source-id: 1c21841d34897d0072ff6fe5e4ac89adddeb3c68
Summary:
During expensive getbundle request traversing skiplist uses a lot of cpu, and
in fact it's blocking the whole CPU. krallin suggested to yield since it
should avoid blocking cpus, and that's what this diff is doing.
Reviewed By: krallin
Differential Revision: D22160477
fbshipit-source-id: 5bd881d5c50f4d8e64f2cc90389abc8568ec1df6
Summary: DangerousOverride is moved into a separate crate. Not only it is usually not needed but it was introducing dependencies on mercurial crate.
Reviewed By: StanislavGlebik
Differential Revision: D22115015
fbshipit-source-id: c9646896f906ea54d11aa83a8fbd8490a5b115ea
Summary: This change will ensure that cloning blobrepo is cheap, even if someone adds field that is expensive to clone. Plus it will result in just one arc-clone instead of cloning all the fields one by one.
Reviewed By: mitrandir77
Differential Revision: D22114066
fbshipit-source-id: ca0c3c78033b4c74872da314a32deb37c05b70ca
Summary: Globalrev does not have any dependencies on mercurial so it can be moved to mononoke_types since it is used in BlobRepo
Reviewed By: StanislavGlebik
Differential Revision: D22092491
fbshipit-source-id: 1dded88eb2ace08e8c6c3673e2d50ae1fbb9850d
Summary: Move all mercurial changeset generation logic to `blobrepo_hg`. This is preliminary step is required to decouples BlobRepo from mercurial, and in later stages it will be moved to derived data infra once blobrepo is free of mercurial.
Reviewed By: StanislavGlebik
Differential Revision: D22089677
fbshipit-source-id: bca28dedda499f80899e729e4142e373d8bec0b8
Summary: move HgMutationStore to attributes, and all related methods to BlobRepoHg
Reviewed By: StanislavGlebik
Differential Revision: D22089657
fbshipit-source-id: 8fe87418ccb8a7ad43828758844bdbd73dc0573d
Summary: Move `Filenodes` to `BlobRepo::attributes` as it is mercurial specific.
Reviewed By: ikostia
Differential Revision: D21662418
fbshipit-source-id: 87648a3e6fd7382437424df3ee60e1e582b6b958
Summary: This diff introduces `BlobRepoHg` extension trait for `BlobRepo` object. Which contains mercurial specific methods that were previously part of `BlobRepo`. This diff also stars moving some of the methods from BlobRepo to BlobRepoHg.
Reviewed By: ikostia
Differential Revision: D21659867
fbshipit-source-id: 1af992915a776f6f6e49b03e4156151741b2fca2
Summary:
This diff adds additional filed `BlobRepo::attributes` which can store attributes of arbitrary type. This will help store opaque types inside blobrepo without creating dependency on a crate which contains type definition for this attribute. This diff also moves `BonsaiHgMapping` inside attributes set.
- This work will allow to move mercurial changeset generation logic to derive data infrastructure
Reviewed By: ikostia
Differential Revision: D21640438
fbshipit-source-id: 3abd912e7227738a73ea9b17aabdda72a33059aa
Summary: Not all facebook-specific code was moved out of integration_runner_real.py, but removing part of the code that is left would made the code less readable, the rest of it will be removed while the integration_runner_real.py is made usable for OSS
Reviewed By: farnz
Differential Revision: D22114948
fbshipit-source-id: d9c532a6a9ea653de2b12cffc92fbf45826dad37
Summary: Add link support to CountedBlobstore
Reviewed By: StanislavGlebik
Differential Revision: D22090644
fbshipit-source-id: 36dc5454f1ca12c91d0eac6e5059f554ac5cb352
Summary:
Previously perform_move was able to only create a single commit that moves all
files. However this commit can be very large and it can cause problems for
different sync jobs. Let's add a method that all the files in the stack of
commits rather than in a single commit.
Reviewed By: ikostia
Differential Revision: D22067851
fbshipit-source-id: aff4d01cd547eb98dd6f0c45704371724acee9fc
Summary: If a value is above cachelib limit let's try to compress it.
Reviewed By: krallin
Differential Revision: D22139644
fbshipit-source-id: 9eb366e8ec94fe66529d27892a988b035989332a
Summary:
We support unicode file paths, and in python 3 those get passed to
python libraries as unicode strings. The tests set LANG=C which mean the python
library tries to convert the path to ascii, but fails for any non-ascii
characters. Let's switch to LANG="en_US.UTF-8" to match our production
behavior and make tests about unicode paths work.
Reviewed By: xavierd
Differential Revision: D22098359
fbshipit-source-id: c3057edc66e6e32f7b8b49374e622d02bd05711f
Summary: Moved GitimportPreferences and GitimportTarget objects from gitimport to import_tools. Reasoning: will need to move gitimport function which needs these objects. Since it's a rust_binary, we can't add dependency on it in import_tools. Therefore, I moved the objects to import_tools.
Reviewed By: StanislavGlebik
Differential Revision: D22135765
fbshipit-source-id: f31f96e689dccaac171a9d0573019b1fff757885
Summary: A bunch of our dependencies weren't really used, and this fact has recently became a source of hard failures. This diff is an attempt to fix it.
Reviewed By: StanislavGlebik
Differential Revision: D22136288
fbshipit-source-id: 4ae265a93e155312ae086647a27ecf1cbbd57e9c
Summary:
All of these are instances of `CommitSyncer`, named `commit_sync_config`. It
makes a reader think they may be instances of `CommitSyncConfig`.
Reviewed By: farnz
Differential Revision: D22113917
fbshipit-source-id: bb682b24d7cadf87ca7721f341dbc8231636c99a
Summary: Megarepo is simplified if we can avoid copying hooks everywhere - run megarepo hooks as well as small repo hooks during pushredirection.
Reviewed By: StanislavGlebik
Differential Revision: D20652331
fbshipit-source-id: f42216797b9061db10b50c1440253de1f56d6b85
Summary:
Tooling can't handle named_deps yet, but it can warn about them
P133451794
Reviewed By: StanislavGlebik
Differential Revision: D22083499
fbshipit-source-id: 46de533c19b13b2469e912165c1577ddb63d15cd
Summary:
Remove unused dependencies for Rust targets.
This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.
Manual edits (named_deps) and misc output in P133451794
Reviewed By: dtolnay
Differential Revision: D22083498
fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
Summary:
Two changes here:
1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches.
2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file.
Reviewed By: farnz
Differential Revision: D22023144
fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
Summary:
The goal of the stack is to support hot reloading of `CommitSyncConfig`s everywhere: in `push_redirector`, `backsyncer`, `x-repo sync job` and so forth.
This diff in particular is a refactoring of how we instantiate the `PushRedirector` struct for the `unbundle` flow. Previously the struct would be instantiated when `RepoHandler` struct was built and would later be reused by `RepoClient`. Now we want to instantiate `PushRedirector` before we start processing the `unbundle` request, so that we can request the newest `CommitSyncConfig`. Note that this diff does not introduce the hot reload itself, it just lays the groundwork: instantiation of `PushRedirector` at request start.
To achieve this goal, `RepoClient` now contains a somewhat modified `PushRedirectorArgs` struct, whose goal is to own the unchanging stuff, needed to create a full `PushRedirector`.
Here are a few explicit non-goals for this hot reloading:
- the overall decision whether the repo is part of any `CommitSyncConfig` or not is still made at `RepoHandler` creation time. What this means is that if `CommitSyncConfig` is changed to have another small repo and Mononoke servers happens to know about that repo, it would not automatically pick up the fact that the repo should be a part of `CommitSyncConfig`
- same for removal (stopping push redirector is already possible via a different hot-reloaded config)
- changing anything about a large/small relationship is likely to be very complicated under the best circumstances of everything being down, let alone during a hot reload. This means that target repo cannot be changed via this mechanizm.
Essentially, the goal is only to be able to do a live change of how paths change between repos.
Reviewed By: StanislavGlebik
Differential Revision: D21904799
fbshipit-source-id: e40e6a9c39f4f03a436bd974f3cba26c690c5f27
Summary: Added basic main and target files for the repo import tool
Reviewed By: StanislavGlebik
Differential Revision: D22067642
fbshipit-source-id: 5ed97e3f2446c86314918f57103c2ecb911e30b0
Summary:
Add optional compress on put controlled by a command line option.
Other than costing some CPU time, this may be a good option when populating repos from existing uncompressed stores to new stores.
Reviewed By: farnz
Differential Revision: D22037756
fbshipit-source-id: e75190ddf9cfd4ed3ea9a18a0ec6d9342a90707b
Summary: Add zstd decoding support to packblob so that if store contains individually zstd compressed blobs we can load them on get()
Reviewed By: farnz
Differential Revision: D22037755
fbshipit-source-id: 41d85be6fcccf14fb198f6ea33a7ca26c4527a46
Summary:
Add a regex for repo prefix detection, and use it in prefixblob for removal of repo prefix from blob-embedded keys.
This is important to keep blobs copyable between repos, and to allow dedupe between same blob in two repos.
I looked at alternate approaches of passing down the prefix from PrefixBlob::new(), but that was fragile and didn't cover the use cases from things like blobstore_healer where no prefixblob is configured at all but packblob will be in the blobstore stack. Using a pattern is the only real option in such "all repo" cases.
The aim of binding the pattern and its tests closely to the prefix generation is to make it hard for someone to get them out of sync, and provide a clear local test failure if they do.
Reviewed By: farnz
Differential Revision: D22034638
fbshipit-source-id: 95a1c2e1ef81432cba606c22afa16b026f59fd5f
Summary:
Add pack and unpack logic to packblob.
Loading a packed form is possible from a regular get(), as the store may contain packed data.
Storing a packed form is only possible from a new put_packed() method, intended for use from the packer (and tests).
NB As noted in the TODO in get, this does not yet handle prefix removal on get, will address that in a separate diff.
Reviewed By: StanislavGlebik
Differential Revision: D21980498
fbshipit-source-id: f534b0e754aa29c42bf00bb7e764f93f1446c14c
Summary: Add blobstore link trait so we can use hardlink style links in fileblob and memblob for testing and later sqlblob et al for prod.
Reviewed By: StanislavGlebik
Differential Revision: D21935647
fbshipit-source-id: f76eaca26b6b226c77d6e39e9c64e02b4145b614
Summary: Fix the capitalization in the error message to match the other variants.
Reviewed By: singhsrb
Differential Revision: D22078125
fbshipit-source-id: 4a1de0111fa4dceea2d401c47ef145669a3cc6f5
Summary: asyncify get_file_history_using_prefetched. Initially I was planning to change this function in the next diff, but then changed my mind. However asyncifying this function won't hurt to land.
Reviewed By: krallin
Differential Revision: D21881121
fbshipit-source-id: 0b2e92ae015b72e1f4578153c56708faabc4bf13
Summary: this is the last diff to complete the `reachabilityindex` module migration
Reviewed By: farnz
Differential Revision: D22049853
fbshipit-source-id: d8510f2d1b1c12a1ba903c2afda37096aadd8347
Summary:
When inserting multiple rows using `INSERT IGNORE`, it's possible to hit
deadlocks if two competing transactions insert the same rows in a different
order.
This can be triggered in the mutation store if two clients are pushing
overlapping sets of commits, and the arbitrary changeset order from the
`HashSet` gives the commits in a different order. For example the first
transaction might be inserting "A, B, C", be holding the lock on row A, and be
waiting for the lock on row C, meanwhile another transaction is inserting "C,
D, A", holding the lock on row C, and waiting for the lock on row A.
To avoid these deadlocks, always insert the mutation changesets in a stable
order (sorted by changeset hash or successor hash).
Reviewed By: krallin
Differential Revision: D22067338
fbshipit-source-id: aa54c8a9c0cac4e4ae35855b44e759f8b6cb4f59
Summary:
async the blobstore test functions, in preparation for adding test coverage for BlobstoreWithLink::link
The boxable test was duplicating the simple roundtrip test which seemed unnecessary, now it just checks that we can Box the blobstore type.
Reviewed By: krallin
Differential Revision: D21974471
fbshipit-source-id: a3cd71a7503a3a670b3c6223812a2e870a20a16e
Summary: update memblob imports to futures_old in preparation for using new futures to implement BlobstoreWithLink
Reviewed By: krallin
Differential Revision: D21973642
fbshipit-source-id: dea4a686db41a0d618395c4dc411603f922a9e22
Summary: Update imports in preparation for using new futures for the BlobstoreWithLink mixin trait
Reviewed By: StanislavGlebik
Differential Revision: D21973587
fbshipit-source-id: 094aa82abc21848d43f50d6d169087e19339ba04
Summary: Switch blobstore imports to use futures_old naming in preparation for using new futures in BlobstoreWithLink (and later BlobstoreWithEnumeration) traits.
Reviewed By: krallin
Differential Revision: D21972842
fbshipit-source-id: 6022c98e7b7254f7c7beb46dc1c7f83609810853
Summary: Add basic packblob store which wraps blobs in the thrift StorageEnvelope type on put() and unwraps them on get()
Reviewed By: farnz
Differential Revision: D21924405
fbshipit-source-id: 81eccf5da6b5b50bdb9aae13622301a20dca0eac
Summary:
Thift definitions for a new blobstore layer known as packblob, which is an enabling part of compression at the mononoke blobstore layer. Its storage formats will support raw data, individual blob compression and blob delta compression.
The included README.md has more detail on the expected use of these types.
Reviewed By: farnz
Differential Revision: D21761723
fbshipit-source-id: a277800f4b4aed5972c4a5bb277e984ff12660f8
Summary:
I'm going to change it in the next diffs, so let's migrate them to new futures
first.
Reviewed By: krallin
Differential Revision: D22065216
fbshipit-source-id: b06fcac518d684f40b1d19fcef9118ca51236873
Summary: Without this, the most lagged shard will slow down all shards; with this, a single lagging shard doesn't slow down other shards.
Reviewed By: ahornby
Differential Revision: D22018264
fbshipit-source-id: c23f25ea1d0b0b6891d2a6f76dfed467101d2f4d
Summary: I'll want the delay operation to respect shard_id - move it into the stores. While here, do some refactoring that will reduce the size of the next diff.
Reviewed By: ahornby
Differential Revision: D22018317
fbshipit-source-id: d2333dfc6749b2a13bc8a67bfa953ed25cedf847
Summary: Prep for 1.44 but also general cleanups.
Reviewed By: dtolnay
Differential Revision: D22024428
fbshipit-source-id: 8e1d39a1e78289129b38554674d3dbf80681f4c3
Summary:
Rename the `subtree` endpoint on the EdenAPI server to `complete_trees` to better express what it does (namely, fetching complete trees, in contrast to the lighter weight `/trees` endpoint that serves individual tree nodes). This endpoint is not used by anything yet, so there isn't much risk in renaming it at this stage.
In addition to renaming the endpoint, the relevant request struct has been renamed to `CompleteTreeRequest` to better evoke its purpose, and the relevant client and test code has been updated accordingly. Notably, now that the API server is gone, we can remove the usage of this type from Mononoke's `hgproto` crate, thereby cleaning up our dependency graph a bit.
Reviewed By: krallin
Differential Revision: D22033356
fbshipit-source-id: 87bf6afbeb5e0054896a39577bf701f67a3edfec
Summary:
After being trimmed down to store only a single edge the skiplist update
algoritm was working incorrectly. This diff introduces a way of fixing things
up.
Reviewed By: StanislavGlebik
Differential Revision: D21976995
fbshipit-source-id: 3bc4678011b332419d9f6f7b983a33fa066674bf
Summary: Make most of the handlers on the EdenAPI server return streaming responses.
Reviewed By: krallin
Differential Revision: D22014652
fbshipit-source-id: 177e540e1372e7dfcba73c594614f0225da3a10f
Summary:
In production we trim skiplists to hold a single entry which causes
the update algorithm to behave incorrectly
Reviewed By: StanislavGlebik
Differential Revision: D21976996
fbshipit-source-id: 4946be11512ac9577bdd3320913fe092d513ecdd
