Summary:
As a starting point for tuning up caching, ask for huge evenly sized
pools. We will gather data from here via Cachelib's logging, and then come back
to tune this properly
Reviewed By: StanislavGlebik
Differential Revision: D9323841
fbshipit-source-id: 721f24acaca4542276b73d25985e54b111e2ca8a
Summary:
Now that we can configure access configs, ask for one suited for a
larger cache
Reviewed By: StanislavGlebik
Differential Revision: D9323842
fbshipit-source-id: 40d94090121214ed3310b40ce8de8a7c350ed4af
Summary: Store where it was copied from
Reviewed By: farnz
Differential Revision: D9132560
fbshipit-source-id: a7a73e1f3de08340f5add5fffa32dd0373eb27fa
Summary:
The issues were fixed, and also thrift manifold works better during bulk
blobimport.
Reviewed By: farnz
Differential Revision: D9132384
fbshipit-source-id: ab4a04eeff86bb4968b80af00c404fad710db183
Summary:
Update crates.
`kvfilter` has minor API changes. Fixed Mononoke in place.
Reviewed By: jsgf
Differential Revision: D9325139
fbshipit-source-id: 13e266519b242f47c0bf022455fc8ca33c205382
Summary: Mononoke stopped logging such a fine graned logs per commit because they are noisy, the remainder of the gettreepack test should be good enough to verify that the thing in test works
Reviewed By: sunshowers
Differential Revision: D9318050
fbshipit-source-id: 141a673731ceff0092f8f3e4209efb69cf481640
Summary: The "path" here is only used for logging. Also add another constructor which can be used to provide custom naming.
Reviewed By: StanislavGlebik
Differential Revision: D9309237
fbshipit-source-id: 0192a3f74129ad063aa41ed61125b3b1d2711a77
Summary: Those futures started calling tokio_threadpool::blocking since aslpavel added bonsai to hg mapping for bookmarks, this causes the server to fail to start up because fetching config repo was happening outside of tokio.
Reviewed By: farnz
Differential Revision: D9316711
fbshipit-source-id: 64188028537881baf1b1c713adc39b22c09a78cc
Summary:
The process size shrinker has issues in Sandcastle CI; the Tupperware
shrinker will only enable shrinking if we're inside an identifiable Tupperware
task.
Switch to the Tupperware shrinker, which also means that we're using
`cachelib`'s default recommendations for shrinking the cache under memory
pressure in Tupperware.
Reviewed By: lukaspiatkowski
Differential Revision: D9297077
fbshipit-source-id: 0ec788e32409e2a5d058a89e11dc1f7e37c7552f
Summary: This diff implements a file length function for file hooks.
Reviewed By: lukaspiatkowski
Differential Revision: D9266987
fbshipit-source-id: f53fa0349e898a3bdc314f59dbae0a61c6d7a323
Summary: - Make `Bookmakrs` work with `ChangsetId` instead of `HgChangesetId`
Reviewed By: StanislavGlebik, farnz
Differential Revision: D9297139
fbshipit-source-id: e18661793d144669354e509271044410caa3502a
Summary:
The previous implementation of `lazy_index` would hit max recursion depth, due to long chains of `Future`s all calling poll on each other when the `Future` representing indexing of a node is waiting for the `Future` of the parent, all the way down. This modification avoids this by:
* Doing a BFS down to the desired depth, remembering all the nodes seen and returning them in topological order (oldest to newest). This is done in a `loop_fn` and so doesn't have the long chain of futures polling each other.
* Synchronously indexing the list of discovered nodes.
Reviewed By: StanislavGlebik
Differential Revision: D9228578
fbshipit-source-id: 0f472d13ee5a0a33472700d1fea29bd7a9938011
Summary: Implements the functionality to use SkiplistIndex to quickly find ancestors of a given set of nodes with generation <= a given parameter. This will be used to help speed up "A % B" revset operations.
Reviewed By: StanislavGlebik
Differential Revision: D9120502
fbshipit-source-id: 5e51057ab23ec6bdaf727e3e71870aad6ab27e30
Summary: There's a hidden dependency here -- ideally it'd be made explicit.
Reviewed By: jsgf
Differential Revision: D9307939
fbshipit-source-id: e00a201e20bb8d83be09eab10cfdd2ce03a3e22f
Summary:
Should be functionally equivalent and semantically more appropriate
This also makes a couple of small API changes:
- The inner function is expected to just return a Result - IntoFuture is
overkill if its supposed to be synchronous in the first place
- `asynchronize` itself returns `impl Future` rather than being intrinsically
boxed.
- Restructure dieselfilenodes::add_filenodes to only asynchronize the insert
itself.
Reviewed By: farnz
Differential Revision: D8959535
fbshipit-source-id: fef9164e3be0069bd0d93573642cd57bb5babb73
Summary:
Make panics more obvious by printing a backtrace and aborting the
process.
Reviewed By: lukaspiatkowski
Differential Revision: D9282766
fbshipit-source-id: 9e22c1ced5a651dda3b58fa3dd84465699235d2c
Summary: all that places logged all list of HgNodeHash
Reviewed By: StanislavGlebik
Differential Revision: D9242904
fbshipit-source-id: 42a98b04986f2ed432a8956828356bd5b9bcaa88
Summary:
Added a definition for a trait `LeastCommonAncestorHint`, for indexes which can compute an advanced frontier from a starting set of nodes.
Any implementation of this trait will need a method which takes a set of `HgNodeHash` "nodes", and a `Generation` value "gen", and returns a set of nodes "C" which satisfies:
* Max generation in "C" is <= "gen"
* Any ancestor of "nodes" with generation <= gen is also an ancestor of "C"
The plan is to implement this trait for `SkiplistIndex` and introduce it into parts of the revset code.
To elaborate, the current way `DifferenceOfUnionOfAncestorsNodeStream` operates, it assumes that the nodes to exclude can be used as a stream, producing (Generation, HashSet<HgNodeHash>) pairs when polled. In order to use this work to improve that, this assumption will have to be removed, because we will want to advance the exclude_nodes based on the current maximum generation of nodes to include that we know about. Then, the only structure that needs to be replaced in `DifferenceOfUnionOfAncestorsNodeStream` is exclude_nodes. It is used in the following way:
* When trying to determine if an include node should be yielded, the exclude_stream is peeked at. If it is the generation equal to the node being checked, it is checked if that node is in the exclude_stream node set. Else if the generation is less than the node being checked, the node is yielded by default.
* If the generation of the peeked exclude_stream is larger than that of the node being checked, it cannot yet be determined if that node should be yielded or not. And so the exclude_stream is "polled", which forces it to move to the next generation value.
Replacing this structure with a `NodeFrontier`, and the code that uses it so that it can use `LeastCommonAncestorHint` to move the exclude_node frontier forward, should suffice to introduce this work into `DifferenceOfUnionOfAncestorsNodeStream`.
Reviewed By: jsgf
Differential Revision: D9120503
fbshipit-source-id: 317af81a1e335e66cf72603899aa06f28c85b027
Summary: the current 100 is just too small, which can be seen while tracing the requests
Reviewed By: farnz
Differential Revision: D9032184
fbshipit-source-id: adbd274d2bd275cc57af635ff42321f06fa358de
Summary: This makes it easier to look at traces as a whole client session, it also makes it possible to see how long did the entire request took time (previously it would not count the time to send data back).
Reviewed By: farnz
Differential Revision: D9032172
fbshipit-source-id: 048fcb21d4eb70424643f7bb2afa426e09904ada
Summary: As a bonus this diff also contains unifying the linknode family of methods (they all now accept arguments via reference) and better tracing for get_files request
Reviewed By: StanislavGlebik
Differential Revision: D9031283
fbshipit-source-id: 4526a8446984904bce7d4dcef240088c7f2ffaa3
Summary:
Set a panichandler by default in cmdlib::get_logger to make sure
everyone gets one set. It configures itself to exit the process so that we
don't leave it in a half-broken state.
The Mononoke server was already using a panic hook, but this replaces it with
one that prints more detail about what was going on at the time.
Reviewed By: StanislavGlebik
Differential Revision: D9234587
fbshipit-source-id: bb51790a60b1ee545a364b4b92e09ec950788684
Summary:
Backout D9124508.
This is actually more complex than it seems. It breaks non-buck build
everywhere:
- hgbuild on all platforms. POSIX platforms break because `hg archive` will
miss `scm/common`. Windows build breaks because of symlink.
- `make local` on GitHub repo because `failure_ext` is not public. The `pylz4`
Cargo.toml has missing dependencies.
Fixing them correctly seems non-trivial. Therefore let's backout the change to
unblock builds quickly.
The linter change is kept in case we'd like to try again in the future.
Reviewed By: simpkins
Differential Revision: D9225955
fbshipit-source-id: 4170a5f7664ac0f6aa78f3b32f61a09d65e19f63
Summary: Moved the lz4 compression code into a separate module in `scm/common/pylz4` and redirected code referencing the former two files to the new module
Reviewed By: quark-zju, mitrandir77
Differential Revision: D9124508
fbshipit-source-id: e4796cf36d16c3a8c60314c75f26ee942d2f9e65
Summary:
This diff implements a service which tails a configurable bookmark and runs hooks against it.
It uses the standard Mononoke config from the meta config repo.
Reviewed By: StanislavGlebik
Differential Revision: D8898637
fbshipit-source-id: f710fe4c9bda1b78bd17eb6cd6abf2abda4fdb8e
Summary:
This diff extends Lua file hooks to allow them to access file content. Instead of exposing just the file path we now expose a Lua table which contains two fields:
'path' - the file path
'contains_string' - a function that can be called to see whether the file content contains the given string.
contains_string is executed as a coroutine which is immediately yielded so the hook does not block while waiting for the result (this potentially might involve network access).
Reviewed By: StanislavGlebik
Differential Revision: D8889838
fbshipit-source-id: 48b49e8ba2acb2d5479405f2ac27fbf5fc2d642a
Summary:
Simply make the right init call, so that we have data in Scuba that
the cachelib team can use to see if we're doing the right thing
Reviewed By: StanislavGlebik
Differential Revision: D9195015
fbshipit-source-id: ff31dd9462a489bc5f8070068effd8a1bfb2fa58
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
cachelib can throw exceptions from any method. Make all our bindings
exception-safe
Reviewed By: jsgf
Differential Revision: D9158062
fbshipit-source-id: 57948aa6ead304fac6e037621a1f6641826345c5
Summary:
These are the types that we currently need to be able to serialize if we're to
replace `Asyncmemo`'s caching uses with cachelib. Derive the `Abomonation`
trait for all of them.
Reviewed By: jsgf
Differential Revision: D9082597
fbshipit-source-id: 910e90476a3cc4d18ba758226b8572c3e8d264c6
Summary:
The pool rebalancer allows cachelib to shed mostly-dead slabs in
favour of new allocations. Turn it on.
Reviewed By: jsgf
Differential Revision: D9035995
fbshipit-source-id: 1be8073381020b4119df5ffba1a16e90ab7233be
Summary:
We'll be running in Tupperware, and want to shrink when we get too
large to avoid OOM due to caches. Configure cachelib appropriately
Reviewed By: StanislavGlebik
Differential Revision: D8900371
fbshipit-source-id: 4f1f64c2508c64e4ce2d201e0a0e86446f84ffef
Summary: This fixes the `blocking` cannot run without threadpool error.
Reviewed By: farnz
Differential Revision: D9017757
fbshipit-source-id: 037fd6f30598f56a83c1dd91c9b8c4f3c8e413b3
Summary:
Since we have bonsai changesets now, let's add a subcommand to print bonsai
changesets given an hg changeset
Reviewed By: farnz
Differential Revision: D9179315
fbshipit-source-id: fad8a7454b45608482ac9fbf0dae9be16d5cbd69
Summary: The 'mut' requirement wasn't required for structs implementing `ReachabilityIndex`, and will get in the way when incorporating this work into the Mononoke server / API server.
Reviewed By: StanislavGlebik
Differential Revision: D9142238
fbshipit-source-id: 4853b468bf04493289fb017bf56b3a1753f29dcd
Summary:
Alas, the diff is huge. One thing is changing Changesets to use ChangesetId.
This is actually quite straightforward. But in order to do this we need to
adapt our test fixtures to also use bonsai changesets. Modifying existing test
fixtures to work with bonsai changesets is very tricky. Besides, existing test
fixtures is a big pile of tech debt anyway, so I used this chance to get rid of
them.
Now test fixtures use `generate_new_fixtures` binary to generate an actual Rust
code that creates a BlobRepo. This Rust code creates a bonsai changeset, that
is converted to hg changeset later.
In many cases it results in the same hg hashes as in old test fixtures.
However, there are a couple of cases where the hashes are different:
1) In the case of merge we are generating different hashes because of different
changed file list (lukaspiatkowski, aslpavel, is it expected?). this is the case for test
fixtures like merge_even, merge_uneven and so on.
2) Old test fixtures used flat manifest hashes while new test fixtures are tree
manifest only.
Reviewed By: jsgf
Differential Revision: D9132296
fbshipit-source-id: 5c4effd8d56dfc0bca13c924683c19665e7bed31
Summary: To make them more explicit
Reviewed By: matthewdippel
Differential Revision: D9132294
fbshipit-source-id: a365d7b58ba095d11fb0570e5ab6994a158873b3
Summary:
This is a split from D8893504. It just enables functionality to create bonsai changesets.
The split was done so that I can land the biggest chunk of work.
Reviewed By: farnz
Differential Revision: D9081430
fbshipit-source-id: 7437c7789998f5691afe83d5b16a8f2c5faac8b4
Summary:
This diff gets the machinery in place to allow file hooks to access file content. It uses the manifest vfs crate to do this.
Initially we just expose a simple contains_string() method which checks if the file content contains the given string (this is the most common thing file hooks want to do with content), but we can extend this with further methods as the need arises.
Reviewed By: StanislavGlebik
Differential Revision: D8889837
fbshipit-source-id: fb5191d6d1286f070a99108782ff529914aeb198
Summary: This verifier works on manifests -- we'll need other ones for filenodes etc.
Reviewed By: StanislavGlebik
Differential Revision: D9105976
fbshipit-source-id: 939b72b6a3f69b716385315f69a57e91d71a45f3
Summary: The content-fetch subcommand will try to lookup a changeset by bookmark first and then falls back to a hash.
Reviewed By: StanislavGlebik
Differential Revision: D9110531
fbshipit-source-id: 3bd0e452739fc1c8b41dc559864f73e3a1f3f782
Summary:
It was using manifest hash as parent hash. One more reason to fix all the types
in Mononoke
Reviewed By: aslpavel
Differential Revision: D9123653
fbshipit-source-id: 0841f7ac64e50e9234d80040b7f286930af53420
