Summary:
We have plans to add a cache of many changeset entries and store it in the
blobstore. The main reason is to speed up Mononoke's revsets and in turn speed
up getbundle wireproto request.
To cache we first need to serialize them. Let's use thrift serialization for
that.
Reviewed By: lukaspiatkowski
Differential Revision: D9738637
fbshipit-source-id: ba771545de9a955956acb6d169ee7bc424ef271b
Summary: It will be useful outside of pushrebase library as well
Reviewed By: farnz
Differential Revision: D9789811
fbshipit-source-id: c851df8a8cce8b1c26daa09b7fe2ffa40f290160
Summary:
There were quite a lot of pushes that use pushvars.
This diff adds a parsing of it.
After I added the parsing of pushvars it started to fail because it seems to be
a flat manifest push. But having parsing of pushvars probably won't hurt.
Reviewed By: farnz
Differential Revision: D9751962
fbshipit-source-id: 49796e91edfad76fb022a2e0fc049a79859de1b7
Summary:
In `fetch_file_contents()` `blobstore_bytes.into()` converted the bytes to
`Blob<Id>`. This code calls `MononokeId::from_data()` which calls blake2
hashing. Turns out it causes big problems for large many large files that
getfiles can return.
Since this hash is not used at all, let's avoid generating it.
Reviewed By: jsgf
Differential Revision: D9786549
fbshipit-source-id: 65de6f82c1671ed64bdd74b3a2a3b239f27c9f17
Summary: Use failure rather than error-chain for errors.
Reviewed By: StanislavGlebik
Differential Revision: D9780341
fbshipit-source-id: 4d41855093cf812e83b6c348a7499e85d9472daf
Summary:
Split the decoder out into its own function. This can handle partial results,
but the Decoder trait API cannot, so make sure the Decoder still only returns
complete results.
Reviewed By: farnz
Differential Revision: D9780342
fbshipit-source-id: b2439cba95b1e42444adbf2ee4b6e3792703a188
Summary:
Profiling showed that since we are inserting objects into blobstore
sequentially it takes a lot of time for long stacks of commit. Let's do it in
parallel.
Note that we are still inserting sequentially into changesets table
Reviewed By: farnz
Differential Revision: D9683037
fbshipit-source-id: 8f9496b97eaf265d9991b94243f0f14133f463da
Summary:
The "path" in manifold blobrepo is used for logging, but it has been quite confusing with "fbsource" and "fbsource-pushrebase" to be logged in an identical way - both are "fbsource", because of the "path" config. Lets not use the "path" for logging, instead use the "reponame" from metaconfig repo.
In case we ever want to have two repos that are named the same (please don't) or have logging under a different name than "reponame" from config then we can add a proper optional "name" parameter, but for now we don't require this confusing feature.
Reviewed By: StanislavGlebik
Differential Revision: D9769514
fbshipit-source-id: 89f2291df90a7396749e127d8985dc12e61f4af4
Summary:
Let's log how long it takes to do pushrebase, how many retry attempts
pushrebase has done, and how long it takes to generate
the response to the client.
Reviewed By: farnz
Differential Revision: D9683036
fbshipit-source-id: 3ad57c2925bdceb3839cae1ff4215c3dd8cd0cc2
Summary:
We had a lot of requests that took > 15 mins on Mononoke, while taking few
seconds on mercurial. Turned out that hgcli doesn't play well with big chunks.
Looks like AsyncRead very inefficiently tries to allocate memory, and that
causes huge slowness (T33775046 for more details).
As a short-term fix let's chunk the data on the server. Note that now we have
to make getfiles request streamable and manually insert the size of the
request.
Reviewed By: lukaspiatkowski
Differential Revision: D9738591
fbshipit-source-id: f504cf540bc7d90e2cbebba9808455b6e89c92c6
Summary:
We were using incorrect buffer size. That's *very* surprising that our servers
weren't continuously crashing. However, see the test plan - it really looks
like `LZ4_compressBound()` is the correct option here.
Reviewed By: farnz
Differential Revision: D9738590
fbshipit-source-id: d531f32e79ab900f40d46b7cb6dac01dff8e9cdc
Summary:
See the comment near "DecompressionType::OverreadingZstd" to see what it does.
Why OverreadingZstd works for Mononoke's use case? Answer:
Because we use it in bundle2 parsing, which is already chunked by the outer Reader. This means that when we have a stream of bytes:
```
uncompressed -> compressed bundle2 -> uncompressed
```
thanks to chunking we extract the compressed part:
```
do_stuff(uncompressed)
ZstdDecoder(compressed bundle2)
do_stuff(uncompressed)
```
rather than
```
do_stuff(uncompressed)
ZstdDecoder(compressed bundle2 -> uncompressed)
```
So overreading doesn't hurt us here
Reviewed By: StanislavGlebik
Differential Revision: D9700778
fbshipit-source-id: 70dd6f405ffa00fb981791aff25c60f60831ea6b
Summary:
Use .chain_err() where appropriate to give context to errors coming up from
below. This requires the outer errors to be proper Fail-implementing errors (or
failure::Error), so leave the string wrappers as Context.
Reviewed By: lukaspiatkowski
Differential Revision: D9439058
fbshipit-source-id: 58e08e6b046268332079905cb456ab3e43f5bfcd
Summary: Cleans things up a bit, esp when matching Context/Chain.
Reviewed By: lukaspiatkowski
Differential Revision: D9439062
fbshipit-source-id: cde8727437f58b288bed9dfacb864bdcd7dea45c
Summary:
Use the err_downcast macros instead of manual downcasting. Doesn't make
a huge code-size difference in this case, but a little neater?
Reviewed By: kulshrax, fanzeyi
Differential Revision: D9405014
fbshipit-source-id: 170665f3ec3e78819c5c8a78d458636de253bb6f
Summary:
Add a type to explicitly model a causal chain of errors, akin to
error_chain. This looks a lot like Context, but is intended to show the entire
stack of errors rather than deciding that only the top-level one is
interesting.
This adds a `ChainExt` trait, which adds a `.chain_ext(OuterError)` method to
add another step to the causal chain. This is implemented for:
- `F` where `F: Fail`
- `Error`
- `Result<_, F>` where `F: Fail`
- `Result<_, Error>`
- `Future`/`Stream<Error=F>` where `F: Fail`
- `Future`/`Stream<Error=Error>`
- `Chain`
Using it is simple:
```
let res = something_faily().chain_err(LocalError::new("Something amiss"))?;
```
where `something_faily()` returns any of the above types.
(This is done by adding an extra dummy marker type parameter to the `ChainExt`
trait so that it can avoid problems with the coherence rules - thanks for the idea @[100000771202578:kulshrax]!)
Reviewed By: lukaspiatkowski
Differential Revision: D9394192
fbshipit-source-id: 0817844d283b3900d2555f526c2683231ca7fe12
Summary:
Add a pair of macros to make downcasting errors less tedious:
```
let res = err_downcast! {
err, // failure::Error
foo: FooError => { println!("err is an FooError! {:?}", foo) },
bar: BarError => { println!("err is a BarError! {:?}", bar) },
};
```
`err_downcast` takes `failure::Error` and deconstructs it into one of the
desired types and returns `Ok(match action)`, or returning it as `Err(Error)`
if nothing matches.
`err_downcast_ref` takes `&failure::Error` and gives a reference type. It
returns `Some(match action)` or `None` if nothing matches.
The error types are required to implement `failure::Fail`.
`err_downcast_ref` also matches each error type `E` as `Context<E>`.
Reviewed By: lukaspiatkowski
Differential Revision: D9394193
fbshipit-source-id: c56d91362d5bed8ab3e254bc44bb6f8a0eb376a2
Summary:
Pushrebase should send back the newly created commits. This diff adds this
functionality.
Note that it fetches both pushrebased commit and current "onto" bookmark.
Normally they should be the same, however they maybe different if bookmark
suddenly moved before current pushrebase finished.
Reviewed By: lukaspiatkowski
Differential Revision: D9635433
fbshipit-source-id: 12a076cc95f55b1af49690d236cee567429aef93
Summary: We are going to use it in pushrebase as well
Reviewed By: lukaspiatkowski
Differential Revision: D9635432
fbshipit-source-id: 5cbe0879d002d9b6c21431b0938562357347a67f
Summary:
`asynchronize` does two conceptually separate things:
1. Given a closure that can do blocking I/O or is CPU heavy, create a future
that runs that closure inside a Tokio task.
2. Given a future, run it on a new Tokio task and shuffle the result back to
the caller via a channel.
Split these two things out into their own functions - one to make the future,
one to spawn it and recover the result. For now, this is no net change - but
`spawn_future` is likely to come in useful once we need more parallelism than
we get from I/O alone, and `closure_to_blocking_future` at least signals intent
when we allow a long-running function to take over a Tokio task.
Reviewed By: jsgf
Differential Revision: D9635812
fbshipit-source-id: e15aeeb305c8499219b89a542962cb7c4b740354
Summary:
`asynchronize` currently does not warn the event loop that it's
running blocking code, so we can end up starving the thread pool of threads.
We can't use `blocking` directly, because it won't spawn a synchronous task
onto a fresh Tokio task, so your "parallel" futures end up running in series.
Instead, use it inside `asynchronize` so that we can pick up extra threads in
the thread pool as and when we need them due to heavy load.
While in here, fix up `asynchronize` to only work on synchronous tasks and
push the boxing out one layer. Filenodes needs a specific change that's
worth extra eyes.
Reviewed By: jsgf
Differential Revision: D9631141
fbshipit-source-id: 06f79c4cb697288d3fadc96448a9173e38df425f
Summary:
We have suspect timings in Mononoke where `asynchronize` is used to
turn a blocking function into a future. Add a test case to ensure that
`asynchronize` itself cannot be causing accidental serialization.
Reviewed By: jsgf
Differential Revision: D9561367
fbshipit-source-id: 14f03e3f003f258450bb897498001050dee0b40d
Summary: In case `max_depth=1` we should only return the topmost entry, which in this case always is the root-entry. This fixes it so that we always return-fast in case `max_depth=1`.
Reviewed By: StanislavGlebik
Differential Revision: D9614259
fbshipit-source-id: a6b82bd5aac74d004f61a07bc24f5d26e5c56412
Summary: The latest release of `tokio` updates `tokio::timer` to include a new `Timeout` type and a `.timeout()` method on `Future`s. As such, our internal implementation of `.timeout()` in `FutureExt` is no longer needed.
Reviewed By: jsgf
Differential Revision: D9617519
fbshipit-source-id: b84fd47a3ee4fc1f7c0a52e308317b93f28f04da
Summary: While I was working on `actix-srserver`, I realized the current design of the API server is quite unnecessary. The "MononokeActor" and "MononokeRepoActor" are only returning futures without much CPU computation cost. So it don't need to be placed in a separate thread.
Reviewed By: jsgf
Differential Revision: D9472848
fbshipit-source-id: 618ec39c42d90717fa6985fee7d6308420962d3f
Summary: It avoids a heap of copies
Reviewed By: StanislavGlebik
Differential Revision: D9595689
fbshipit-source-id: a64f0a383acd517830d08cf0be9fc0a1b6903382
Summary:
Looks like we shouldn't have raised in the first place. Big getfiles buffer
causes OOMs on the servers. Also memory profiling shows that quite often most
of the Mononoke server's memory is used for serving remotefilelog requests.
Reviewed By: purplefox
Differential Revision: D9601990
fbshipit-source-id: 356a65d0749b064486436fb737bd5a47b3beecfa
Summary:
Add the ability to get bookmarks using the mononoke admin tool.
Usage: `mononoke_admin --repo-id <repo-id> bookmarks get --changeset-type <hg|bonsai> <BOOKMARK_NAME>`
The changeset-type defaults to HG.
Reviewed By: StanislavGlebik
Differential Revision: D9556742
fbshipit-source-id: c5e64981947aabb9059295622501bc359ed57cc6
Summary:
Add the ability to set bookmarks using the mononoke admin tool.
Usage: `mononoke_admin --repo-id <repo-id> bookmarks set <BOOKMARK_NAME> <HG_CHANGESET_ID>`
Reviewed By: StanislavGlebik
Differential Revision: D9539550
fbshipit-source-id: 7114a6a51711eae6784eb30d820c2ce11672679c
Summary:
Pushkey parts can be sent as part of pushrebase. Phases pushkeys are ignored
because we don't yet support them in Mononoke.
It's not really clear what to do with bookmark pushkey parts, since we are
already moving the `onto` bookmark. For now I suggest to ignore moves of the
`onto` bookmark, and error if there is a pushkey of another bookmark.
Reviewed By: farnz
Differential Revision: D9554385
fbshipit-source-id: 07aff1bd9034c0f2d56a2a5a66ea33c91835ef98
Summary:
I was debugging pushrebase bugs, and the only error I got was
'oneshot::Cancelled'. Let's add a bit more context around it
Reviewed By: farnz
Differential Revision: D9554384
fbshipit-source-id: b3111ef1b5c743d65740f7fa3fd1a92eef9ab784
Summary:
This diff fills missing parts of push-rebase implementation
- `find_closest_root` - find closest root to specified bookmark
- `find_changed_files` - find file affected by changesets between provided `ancestor` and `descendant`
- `intersect_changed_files` - rejects push rebase if any conflicts have been found
- `create_rebased_changes` - support for merges
- `do_pushrebase` - returns updated bookmark value
Reviewed By: StanislavGlebik
Differential Revision: D9458416
fbshipit-source-id: c0cb53773eba6e966f1a5928c43ebdec761a78d3
Summary:
Modify the lookup() RPC function to be able to accept either a
bookmark or commit hash. A commit hash lookup is attempted first, falling
back to a bookmark lookup if it fails.
Reviewed By: StanislavGlebik
Differential Revision: D9457349
fbshipit-source-id: 78db21c01c498b045f5781097cb12f7220a40999
Summary: Added a thrift client library and binary for Mononoke API Server that allows us to play with the API Server's thrift port.
Reviewed By: farnz
Differential Revision: D9110899
fbshipit-source-id: 603cc5e2b5e0419a73c9eccb35f8c95455ada9ce
Summary: This commit adds a basic thrift server that responds to fb303 status check queries to Mononoke API Server.
Reviewed By: farnz
Differential Revision: D9092291
fbshipit-source-id: d1e4ddb280c252f549d40a0bb03d05afccbf73b8
Summary: This commits change `HgBlob` from an enum into a struct that only contains one Bytes field, completely removes `HgBlobHash` and changes the methods of `HgBlob` from returning `Option`s into directly returning results.
Reviewed By: farnz
Differential Revision: D9317851
fbshipit-source-id: 48030a621874d628602b1c5d3327e635d721facf
Summary: Since this data is specific to TimedStream and not TimedFuture I split the Stats struct into FutureStats and StreamStats
Reviewed By: StanislavGlebik
Differential Revision: D9355421
fbshipit-source-id: cc2055706574756e2e53f3ccc57abfc50c3a02ba
Summary: gettreepack doesn't care for deleted entries, only about added or modified ones
Reviewed By: StanislavGlebik
Differential Revision: D9378909
fbshipit-source-id: 2935e6b74fbb0208f7cf89ab4b1e761bb9c6000b
Summary: Now pushrebasing stacks as well. Again, still no conflicts checks
Reviewed By: aslpavel
Differential Revision: D9359807
fbshipit-source-id: 9f6e7a05b45fb80b40faaaaa4fe2434b7a591a7c
