Summary: ugh, yet another case of a hidden dependency.
Reviewed By: StanislavGlebik
Differential Revision: D9318498
fbshipit-source-id: 5fcd25081b5033cbef9c5f137e616348c5d6ced9
Summary: Remove `Arc<BlobRepo>` from more places since `BlobRepo` will do its own internal `Arc`ing.
Reviewed By: StanislavGlebik
Differential Revision: D9317987
fbshipit-source-id: 899e8b2ede278e62a83e64c144eb18c8cc7e57c6
Summary: Adds proper url decoding for is_ancestor, so that special characters can be encoded in the url.
Reviewed By: kulshrax
Differential Revision: D9325467
fbshipit-source-id: d3ff60e004be8d254ea6f7288188adf54ab7ff5f
Summary: Beginnings of a container with various essential bits.
Reviewed By: StanislavGlebik
Differential Revision: D9322148
fbshipit-source-id: b69bd17aa88acd69e81b90e9a1efb672247dc887
Summary:
It's really part of the server, and isn't likely to be useful
elsewhere.
Reviewed By: StanislavGlebik
Differential Revision: D9322149
fbshipit-source-id: 0dc3ca41f2779b3cc9e1c32f8e09e369038c3d53
Summary:
There's no need to use the more general syntax here, and it helps with
readability.
Reviewed By: StanislavGlebik
Differential Revision: D9322151
fbshipit-source-id: 6273083048657f79a990555d8d35cc5d0d51714b
Summary:
This causes `buck test` to return an error code, as it skipped this
test. It's trivial to change `extras_roundtrip` to get a longer test run if
developing this code, and I'd like to be in a place where `buck test && jf
submit` is a good idiom to use for only submitting tested code
Reviewed By: StanislavGlebik
Differential Revision: D9334941
fbshipit-source-id: 98edb18346f75b27175b00096232471839caed71
Summary:
I was seeing this test fail intermittently - increasing the loop from
40 iterations (4 seconds) to 100 iterations (10 seconds) makes it reliably pass
Reviewed By: StanislavGlebik
Differential Revision: D9333501
fbshipit-source-id: 31d842e2ec9a5bfd2e6d5d31ada85a6a891ff037
Summary:
As a starting point for tuning up caching, ask for huge evenly sized
pools. We will gather data from here via Cachelib's logging, and then come back
to tune this properly
Reviewed By: StanislavGlebik
Differential Revision: D9323841
fbshipit-source-id: 721f24acaca4542276b73d25985e54b111e2ca8a
Summary:
Now that we can configure access configs, ask for one suited for a
larger cache
Reviewed By: StanislavGlebik
Differential Revision: D9323842
fbshipit-source-id: 40d94090121214ed3310b40ce8de8a7c350ed4af
Summary: Store where it was copied from
Reviewed By: farnz
Differential Revision: D9132560
fbshipit-source-id: a7a73e1f3de08340f5add5fffa32dd0373eb27fa
Summary:
The issues were fixed, and also thrift manifold works better during bulk
blobimport.
Reviewed By: farnz
Differential Revision: D9132384
fbshipit-source-id: ab4a04eeff86bb4968b80af00c404fad710db183
Summary:
Update crates.
`kvfilter` has minor API changes. Fixed Mononoke in place.
Reviewed By: jsgf
Differential Revision: D9325139
fbshipit-source-id: 13e266519b242f47c0bf022455fc8ca33c205382
Summary: Mononoke stopped logging such a fine graned logs per commit because they are noisy, the remainder of the gettreepack test should be good enough to verify that the thing in test works
Reviewed By: sunshowers
Differential Revision: D9318050
fbshipit-source-id: 141a673731ceff0092f8f3e4209efb69cf481640
Summary: The "path" here is only used for logging. Also add another constructor which can be used to provide custom naming.
Reviewed By: StanislavGlebik
Differential Revision: D9309237
fbshipit-source-id: 0192a3f74129ad063aa41ed61125b3b1d2711a77
Summary: Those futures started calling tokio_threadpool::blocking since aslpavel added bonsai to hg mapping for bookmarks, this causes the server to fail to start up because fetching config repo was happening outside of tokio.
Reviewed By: farnz
Differential Revision: D9316711
fbshipit-source-id: 64188028537881baf1b1c713adc39b22c09a78cc
Summary:
The process size shrinker has issues in Sandcastle CI; the Tupperware
shrinker will only enable shrinking if we're inside an identifiable Tupperware
task.
Switch to the Tupperware shrinker, which also means that we're using
`cachelib`'s default recommendations for shrinking the cache under memory
pressure in Tupperware.
Reviewed By: lukaspiatkowski
Differential Revision: D9297077
fbshipit-source-id: 0ec788e32409e2a5d058a89e11dc1f7e37c7552f
Summary: This diff implements a file length function for file hooks.
Reviewed By: lukaspiatkowski
Differential Revision: D9266987
fbshipit-source-id: f53fa0349e898a3bdc314f59dbae0a61c6d7a323
Summary: - Make `Bookmakrs` work with `ChangsetId` instead of `HgChangesetId`
Reviewed By: StanislavGlebik, farnz
Differential Revision: D9297139
fbshipit-source-id: e18661793d144669354e509271044410caa3502a
Summary:
The previous implementation of `lazy_index` would hit max recursion depth, due to long chains of `Future`s all calling poll on each other when the `Future` representing indexing of a node is waiting for the `Future` of the parent, all the way down. This modification avoids this by:
* Doing a BFS down to the desired depth, remembering all the nodes seen and returning them in topological order (oldest to newest). This is done in a `loop_fn` and so doesn't have the long chain of futures polling each other.
* Synchronously indexing the list of discovered nodes.
Reviewed By: StanislavGlebik
Differential Revision: D9228578
fbshipit-source-id: 0f472d13ee5a0a33472700d1fea29bd7a9938011
Summary: Implements the functionality to use SkiplistIndex to quickly find ancestors of a given set of nodes with generation <= a given parameter. This will be used to help speed up "A % B" revset operations.
Reviewed By: StanislavGlebik
Differential Revision: D9120502
fbshipit-source-id: 5e51057ab23ec6bdaf727e3e71870aad6ab27e30
Summary: There's a hidden dependency here -- ideally it'd be made explicit.
Reviewed By: jsgf
Differential Revision: D9307939
fbshipit-source-id: e00a201e20bb8d83be09eab10cfdd2ce03a3e22f
Summary:
Should be functionally equivalent and semantically more appropriate
This also makes a couple of small API changes:
- The inner function is expected to just return a Result - IntoFuture is
overkill if its supposed to be synchronous in the first place
- `asynchronize` itself returns `impl Future` rather than being intrinsically
boxed.
- Restructure dieselfilenodes::add_filenodes to only asynchronize the insert
itself.
Reviewed By: farnz
Differential Revision: D8959535
fbshipit-source-id: fef9164e3be0069bd0d93573642cd57bb5babb73
Summary:
Make panics more obvious by printing a backtrace and aborting the
process.
Reviewed By: lukaspiatkowski
Differential Revision: D9282766
fbshipit-source-id: 9e22c1ced5a651dda3b58fa3dd84465699235d2c
Summary: all that places logged all list of HgNodeHash
Reviewed By: StanislavGlebik
Differential Revision: D9242904
fbshipit-source-id: 42a98b04986f2ed432a8956828356bd5b9bcaa88
Summary:
Added a definition for a trait `LeastCommonAncestorHint`, for indexes which can compute an advanced frontier from a starting set of nodes.
Any implementation of this trait will need a method which takes a set of `HgNodeHash` "nodes", and a `Generation` value "gen", and returns a set of nodes "C" which satisfies:
* Max generation in "C" is <= "gen"
* Any ancestor of "nodes" with generation <= gen is also an ancestor of "C"
The plan is to implement this trait for `SkiplistIndex` and introduce it into parts of the revset code.
To elaborate, the current way `DifferenceOfUnionOfAncestorsNodeStream` operates, it assumes that the nodes to exclude can be used as a stream, producing (Generation, HashSet<HgNodeHash>) pairs when polled. In order to use this work to improve that, this assumption will have to be removed, because we will want to advance the exclude_nodes based on the current maximum generation of nodes to include that we know about. Then, the only structure that needs to be replaced in `DifferenceOfUnionOfAncestorsNodeStream` is exclude_nodes. It is used in the following way:
* When trying to determine if an include node should be yielded, the exclude_stream is peeked at. If it is the generation equal to the node being checked, it is checked if that node is in the exclude_stream node set. Else if the generation is less than the node being checked, the node is yielded by default.
* If the generation of the peeked exclude_stream is larger than that of the node being checked, it cannot yet be determined if that node should be yielded or not. And so the exclude_stream is "polled", which forces it to move to the next generation value.
Replacing this structure with a `NodeFrontier`, and the code that uses it so that it can use `LeastCommonAncestorHint` to move the exclude_node frontier forward, should suffice to introduce this work into `DifferenceOfUnionOfAncestorsNodeStream`.
Reviewed By: jsgf
Differential Revision: D9120503
fbshipit-source-id: 317af81a1e335e66cf72603899aa06f28c85b027
Summary: the current 100 is just too small, which can be seen while tracing the requests
Reviewed By: farnz
Differential Revision: D9032184
fbshipit-source-id: adbd274d2bd275cc57af635ff42321f06fa358de
Summary: This makes it easier to look at traces as a whole client session, it also makes it possible to see how long did the entire request took time (previously it would not count the time to send data back).
Reviewed By: farnz
Differential Revision: D9032172
fbshipit-source-id: 048fcb21d4eb70424643f7bb2afa426e09904ada
Summary: As a bonus this diff also contains unifying the linknode family of methods (they all now accept arguments via reference) and better tracing for get_files request
Reviewed By: StanislavGlebik
Differential Revision: D9031283
fbshipit-source-id: 4526a8446984904bce7d4dcef240088c7f2ffaa3
Summary:
Set a panichandler by default in cmdlib::get_logger to make sure
everyone gets one set. It configures itself to exit the process so that we
don't leave it in a half-broken state.
The Mononoke server was already using a panic hook, but this replaces it with
one that prints more detail about what was going on at the time.
Reviewed By: StanislavGlebik
Differential Revision: D9234587
fbshipit-source-id: bb51790a60b1ee545a364b4b92e09ec950788684
Summary:
Backout D9124508.
This is actually more complex than it seems. It breaks non-buck build
everywhere:
- hgbuild on all platforms. POSIX platforms break because `hg archive` will
miss `scm/common`. Windows build breaks because of symlink.
- `make local` on GitHub repo because `failure_ext` is not public. The `pylz4`
Cargo.toml has missing dependencies.
Fixing them correctly seems non-trivial. Therefore let's backout the change to
unblock builds quickly.
The linter change is kept in case we'd like to try again in the future.
Reviewed By: simpkins
Differential Revision: D9225955
fbshipit-source-id: 4170a5f7664ac0f6aa78f3b32f61a09d65e19f63
Summary: Moved the lz4 compression code into a separate module in `scm/common/pylz4` and redirected code referencing the former two files to the new module
Reviewed By: quark-zju, mitrandir77
Differential Revision: D9124508
fbshipit-source-id: e4796cf36d16c3a8c60314c75f26ee942d2f9e65
Summary:
This diff implements a service which tails a configurable bookmark and runs hooks against it.
It uses the standard Mononoke config from the meta config repo.
Reviewed By: StanislavGlebik
Differential Revision: D8898637
fbshipit-source-id: f710fe4c9bda1b78bd17eb6cd6abf2abda4fdb8e
Summary:
This diff extends Lua file hooks to allow them to access file content. Instead of exposing just the file path we now expose a Lua table which contains two fields:
'path' - the file path
'contains_string' - a function that can be called to see whether the file content contains the given string.
contains_string is executed as a coroutine which is immediately yielded so the hook does not block while waiting for the result (this potentially might involve network access).
Reviewed By: StanislavGlebik
Differential Revision: D8889838
fbshipit-source-id: 48b49e8ba2acb2d5479405f2ac27fbf5fc2d642a
Summary:
Simply make the right init call, so that we have data in Scuba that
the cachelib team can use to see if we're doing the right thing
Reviewed By: StanislavGlebik
Differential Revision: D9195015
fbshipit-source-id: ff31dd9462a489bc5f8070068effd8a1bfb2fa58
Summary:
Asyncmemo has two issues for our use:
1. Separate memory pool from cachelib caches.
2. Future fusion means that a `get` that should succeed will fail because there
was an earlier get still in progress.
The second is good for memoization, where the worst case from a failed get is
extra CPU work, but not so good for caching. Replace uses of Asyncmemo for
caches with a cachelib based cache
Reviewed By: StanislavGlebik
Differential Revision: D9013679
fbshipit-source-id: b85d4eec7294e0c8ee08faa671d26901b35cf1fc
Summary:
cachelib can throw exceptions from any method. Make all our bindings
exception-safe
Reviewed By: jsgf
Differential Revision: D9158062
fbshipit-source-id: 57948aa6ead304fac6e037621a1f6641826345c5
Summary:
These are the types that we currently need to be able to serialize if we're to
replace `Asyncmemo`'s caching uses with cachelib. Derive the `Abomonation`
trait for all of them.
Reviewed By: jsgf
Differential Revision: D9082597
fbshipit-source-id: 910e90476a3cc4d18ba758226b8572c3e8d264c6
Summary:
The pool rebalancer allows cachelib to shed mostly-dead slabs in
favour of new allocations. Turn it on.
Reviewed By: jsgf
Differential Revision: D9035995
fbshipit-source-id: 1be8073381020b4119df5ffba1a16e90ab7233be
Summary:
We'll be running in Tupperware, and want to shrink when we get too
large to avoid OOM due to caches. Configure cachelib appropriately
Reviewed By: StanislavGlebik
Differential Revision: D8900371
fbshipit-source-id: 4f1f64c2508c64e4ce2d201e0a0e86446f84ffef
Summary: This fixes the `blocking` cannot run without threadpool error.
Reviewed By: farnz
Differential Revision: D9017757
fbshipit-source-id: 037fd6f30598f56a83c1dd91c9b8c4f3c8e413b3
Summary:
Since we have bonsai changesets now, let's add a subcommand to print bonsai
changesets given an hg changeset
Reviewed By: farnz
Differential Revision: D9179315
fbshipit-source-id: fad8a7454b45608482ac9fbf0dae9be16d5cbd69
