Summary:
This adds the options to `eden stats` for collecting only fast stats and printing in JSON.
`eden stats` can be slow especially due to collecting fb303 counters and private bytes. An example use case of this new lightweight endpoint is that Buck can poll it to display Eden related info in its cli (see [post](https://fb.workplace.com/groups/132499338763090/permalink/210396380973385/) for context).
Reviewed By: xavierd
Differential Revision: D29687041
fbshipit-source-id: a663e71231527c5dfb822acbf238af0ac6ce4a00
Summary: Currently we have to manually save the id returned from `start_recording`. After this, we can simply ask for the list of all active recorder sessions.
Reviewed By: genevievehelsel
Differential Revision: D30056117
fbshipit-source-id: 7fd69b70e7b04fcd0b3724f4ee16c5e5e86badaf
Summary:
This improves buck build(and likely other tools) execution time significantly on the cold cache, with eden api enabled(e.g. no import helper)
I have noticed during the tests that we don't urilize network well when accessing files not in cache.
Adding some instrumentation shows that we only do up to 8 parallel fetches on mercurial side.
This seem to be limited by number of hg queue workers, simply boosting this number from 8->32 improves performance a lot, and it is still manageable number of threads
We probably need some better solution to allow for greater parallelism.
With this diff time for `buck build fb4a` on the cold cache is reduced **from 121 to 95 minutes(-30%)**.
'Parsing buck files' stage is reduced **from 07:16 min to 05:53 min(-25%)**.
During buck build we get to actual parallelism of about 16 fetches which is limited by number of buck threads.
Bumping number of buck threads with `-j 32` further reduces build time to **54 minutes (-60% from original)**
Reviewed By: xavierd
Differential Revision: D30025720
fbshipit-source-id: 6bcae1f353a0d31d8ce632ccb991c2a02f3136fc
Summary: Switch from chashmap to dashmap as dashmap upstream is more responsive.
Reviewed By: krallin
Differential Revision: D30046524
fbshipit-source-id: b4377000d33421c9d23c499fd5d6c9ef4b302ee5
Summary: Switch from chashmap to dashmap as dashmap upstream is more responsive.
Reviewed By: krallin
Differential Revision: D30045037
fbshipit-source-id: 252e231d426389ccf726613f0b073159a7a70d68
Summary:
One of the things that megarepo_tool can do is to manually rewrite a commit
from one repo to another with a particular commit remapping version.
e.g.
```
source repo
X
|
P
target repo
X' <- rewritten X
|
A <- commit that exists only in target repo
|
P' <- rewritten P
```
Previously it always required manually setting the parents in the target repo
i.e. in the example above we'd need to tell that A is the new parent of
rewritten commit.
However this is not always convenient. Sometimes we just want megarepo_tool to take
parents in the source repo (i.e. P in the example above), remap them to large repo
(i.e. P' in the example above), and use P' as a target repo parent.
This diff adds a special option that lets us do so.
Reviewed By: farnz
Differential Revision: D30040016
fbshipit-source-id: 116dbe1803857053336ca76d0a65dbca8b14bd73
Summary:
Preupload LFS blobs for big file via LFS protocol rather than new EdenApi Uploads.
The `store.upload` function "skips" non lfs stuff but uploads lfs stuff remotely via LFS protocol.
The lfs implementation also uploads only what is missing, no difference from our implementation.
So, it seems it should be called right before our new Eden Api uploads.
Once it is called, our EdenApi uploads recognises the blobs that have been already uploaded and LFS blobs won't go via our new upload protocol.
Reviewed By: DurhamG
Differential Revision: D30012307
fbshipit-source-id: 39f09055ded71dc69fa0b12fe0ac74ee1a7d3659
Summary: Switch from chashmap to dashmap as dashmap upstream is more responsive.
Reviewed By: StanislavGlebik
Differential Revision: D30044747
fbshipit-source-id: a8eef5140542ddce4199bd052af01f41c75b53e8
Summary:
automv is quite slow if there are lots of files.
Let's disable it automatically if there are too many changed files.
Reviewed By: kulshrax
Differential Revision: D30052694
fbshipit-source-id: deff55e1cee9ce65f459c32e320d2389ea7cfe89
Summary:
* `FuseRequestContext` currently uses `fuseHeader_.opcode` to get the cause detail.
* However, `fuseHeader_.opcode` is set to 0 by `stealReq` to indicate that the request has been released.
* This lead to `getCauseDetail` returning `<unknown>` if we call it after the request is released (e.g. in the clean up future).
This adds the `opcode_` member so that `getCauseDetail` always return the correct cause detail regardless of whether the request has been released. This also removes `examineReq` as it's not used.
Reviewed By: xavierd
Differential Revision: D30050518
fbshipit-source-id: 0c3e4d31e59a5b5fd87967d6c976b573692a2af9
Summary:
Per comment in D30017261 (e9c039ab4a)
> [...] we can get rid of this ESTALE check. That dates back to when edenfs had a bug that returned ESTALE when reading the .eden directory sometimes.
Reviewed By: xavierd
Differential Revision: D30024979
fbshipit-source-id: 645097c8f689c916245845561fd3d824ff7df8b4
Summary: Added a new `edenapi.encoding` config which allows specifying the `Accept-Encoding` for EdenAPI HTTP requests.
Reviewed By: DurhamG
Differential Revision: D30028835
fbshipit-source-id: e80c49e31b0f2180d75fabe28080a7b0dd7383ba
Summary: Add support for setting the `Accept-Encoding` request header, which will allow for compressed responses.
Reviewed By: DurhamG
Differential Revision: D30004651
fbshipit-source-id: 5db784824e1c446c55e4aef5038eb4ab71f8a10b
Summary: Tweak the tracing logs so it does not print duplicated hashes.
Reviewed By: andll
Differential Revision: D29980948
fbshipit-source-id: 2244830f85b3f3beacb9a1264b84da7182ddae1b
Summary: This no longer depends on old-style futures, so we can remove the dependency.
Reviewed By: quark-zju
Differential Revision: D27708962
fbshipit-source-id: fd66fb2934ff631abe0bfcdae843fcc9b10d5fdc
Summary:
In integration tests, we want to be able to run through the megarepo processing, and then check that configs have persisted correctly, so that we can start async workers after sending a config change down, and see the change be picked up.
Make it possible
Reviewed By: StanislavGlebik
Differential Revision: D30012106
fbshipit-source-id: f944165e7b93451180a78d8287db8a59d71bbe13
Summary:
FailScenario nested in a single thread can cause deadlock on a Mutex, confirmed
by gdb backtrace. Fix that by avoiding nested FailScenario in a single thread.
Reviewed By: kulshrax
Differential Revision: D30052693
fbshipit-source-id: 8d38b2e40ec30c8024e563f079d2595d76bbae1b
Summary: Left over of D29989181 (ab8d843ebd).
Reviewed By: kulshrax
Differential Revision: D30052695
fbshipit-source-id: 3e126055c76f9d9e48566708abe1407bd2fecaa3
Summary: Switch from chashmap to dashmap as dashmap upstream is more responsive.
Reviewed By: StanislavGlebik
Differential Revision: D30044510
fbshipit-source-id: 8003ecba2f9c5d16e9cb6dced28f3785a062870d
Summary: 2 of the disabled are passing, and the last one just needed a simple tweak.
Reviewed By: fanzeyi
Differential Revision: D30029260
fbshipit-source-id: ad386632e8b6fbf9c76530c8ce2df7d38990bbbd
Summary:
The mode bits don't make a lot of sense on Windows, so let's use the plain
0o777 as that's what they will look like on Windows.
Reviewed By: chadaustin
Differential Revision: D30028892
fbshipit-source-id: ced737f1290438708536b732095cee18e1b1c073
Summary: The test is passing, enable it.
Reviewed By: chadaustin
Differential Revision: D30028894
fbshipit-source-id: b268fc63220704466c72c688a1f0f04a47573641
Summary:
The create_editor_that_writes_commit_messages function uses bash which is not
available on Windows. Instead, we can simply pass --message to Mercurial for
the same effect.
Reviewed By: chadaustin
Differential Revision: D30028893
fbshipit-source-id: 0a588c37bd4da85176546568126893192b29a574
Summary: This is unused, no need to keep it around.
Reviewed By: genevievehelsel
Differential Revision: D30046503
fbshipit-source-id: 1d20d9b4ce672d5d79410203807dbc93b4bce31a
Summary:
In current location cargo build generates binary file with name 'main'.
Moving main.rs up makes it generate proper binary name 'fsprobe'
Reviewed By: DurhamG
Differential Revision: D29998232
fbshipit-source-id: b33ffb3ca7209a7e19ec15705ba03a31c0449447
Summary:
This option allows to perfrom operations in parallel to test throughput
Using fairly simple approach of sending to first available worker to build parallelism. There is probably some crate that provides multi-consumer queue, but current implementation is likely sufficient for our purposes
Reviewed By: DurhamG
Differential Revision: D29997918
fbshipit-source-id: fe90b88b9c19546fefbb7643c5a8cd5ea8c565aa
Summary:
Let's log repo name since it's clearer for people than repo ids. And in my mind logging
repo ids was a mistake - repo id is an implementation detail (we use repo ids
because they are more efficient to store in xdb table than strings), and
Mononoke users shouldn't need to care about repo ids. So let's start loggin
repo names.
Reviewed By: krallin
Differential Revision: D30040409
fbshipit-source-id: 71c2794d8122e616850662cda27c8092d382de7a
Summary: Same as previous diff, but for wire equivalents. Code was also very duplicated. Now it should be easy to create a new strongly typed hash for edenapi.
Reviewed By: liubov-dmitrieva
Differential Revision: D29994118
fbshipit-source-id: 304c9500ba2aa35cc975612815dc40689845ba93
Summary:
Code was very duplicated between ContentId, Sha1, Sha256. And in this stack I plan to implement even more hashes, so I made this macro which makes it really easy to do so.
I took the opportunity to make the inner field not public, and only accessible via from/into.
Reviewed By: liubov-dmitrieva
Differential Revision: D29992279
fbshipit-source-id: c0b7225a3634071a1b1513119ec516d14bd8fd9e
Summary:
On `createremote`, create a changeset using the create_bonsai_changesets method created earlier.
For now, this changeset is create with a bunch of placeholders, but I added todo for all of those things and will tackle them over the next diffs, otherwise this would be a massive diff and hard to review.
Reviewed By: liubov-dmitrieva
Differential Revision: D29990295
fbshipit-source-id: 6b4c97887c0b4c017c586bf0ea06f12df9d07d23
Summary: This makes the `createremote` command also upload modified files to Mononoke, which will later be used to populate the snapshot.
Reviewed By: liubov-dmitrieva
Differential Revision: D29989181
fbshipit-source-id: 5a3b8d7133d6b27ea291ca01d14432a38d92f866
Summary:
This starts adding very basic behaviour to the createremote command.
- Added a `uploadsnapshot` method to the python/rust api. This will be used by the `createremote` command. It will create a bubble, and upload a snapshot to it. For now it just creates a bubble. The request/response objects are still subject to change.
- Added basic code to `createremote` that calls `uploadsnapshot`. It gets the modified files, but for now does nothing with them. I believe I'll have to read their content on rust code, as they are not in the "hg filestore" I believe, as they're not commited.
Reviewed By: StanislavGlebik
Differential Revision: D29987594
fbshipit-source-id: d1e332bb6d1baf9e90efdd2173474e8f3ebcc0e7
Summary: Add ephemeral_prepare to EdenApi, which allows calling the `/ephemeral/prepare` endpoint. This will be used when creating snapshots.
Reviewed By: liubov-dmitrieva
Differential Revision: D29987593
fbshipit-source-id: 2af2d9e051a8b4c124adf023ea4fd1bf2ec69e7c
Summary: "hg debugsegmentgraph --level LEVEL --group GROUP" prints a DAG of the segments belonging to the specified group and level. This is somewhat easier to read than the segment list from "hg debugchangelog --debug".
Reviewed By: quark-zju
Differential Revision: D29889712
fbshipit-source-id: a304a0a87f733b66f456f468cbbec5d8b03a3aeb
Summary: For `future_` endpoints, we wrap the final return statement with `wrapFuture(std::move(helper), ...` to ensure we keep the ThriftLogHelper alive through the whole call.
Differential Revision: D30018980
fbshipit-source-id: 2c63fe5d7b4504912cc46a32ca04f16e98b0805f
Summary: `eden mount` currently swallows all but one type of exceptions, which resulted in cli reporting false positive mount success when the thrift endpoint throws. The try-catch was added in D21934538 (51df752a46) to provide clear error message for a particular windows error and we can just re-raise so that the other exceptions are propagated.
Reviewed By: chadaustin
Differential Revision: D30017261
fbshipit-source-id: 6a8a44330a90275b3c044301ff644dce0d6dee13
Summary:
Today the globing code is not able to parse `\` as a path separator
which causes an error like this:
```
facebook.eden.ttypes.EdenError: Invalid glob (attempt to construct a PathComponent from a string containing a directory separator: tools\arcanist): tools\arcanist
```
Many other tools like hg files still use these separators, so it would be
nice if we could support `\`.
This change will mean that Windows globs will not support special characters.
But they were already broken so we can not have any users successfully using
them today.
Ideally we will not have to support special charcters on Windows because it is
icky, But I left a note in the code about how we could do this, if its ever
needed in the future
Reviewed By: chadaustin
Differential Revision: D29971586
fbshipit-source-id: 32f54986e801b2aa3b3b1c90c650bd45531e8c0a
Summary:
Copying PathPiece onto futures is a recipe for a use-after-free due to the
underlying string being likely freed when the future executes, let's instead
use plain Path and have the caller move/copy them onto the futures.
Differential Revision: D29989957
fbshipit-source-id: 2e6cebf8c9dfc8f55df8edbc410c2a7f681b8f22
Summary: The code is no longer necessary with `maybe_reuse_caches_from` below.
Reviewed By: andll
Differential Revision: D29980947
fbshipit-source-id: fc81d033f6afdcf484f305be1901f338478bd363
Summary:
There are 2 issues:
- `path.exists()` seems to return `false` for the symlink of the Thrift
compiler. Potentially a rust stdlib issue.
- Generated Thrift code requires a new dep: `tokio_shim`.
Fix it by following the symlink and add the dependency.
Reviewed By: DurhamG
Differential Revision: D29980884
fbshipit-source-id: 318269da51295b296dd5ed2eab850bf9ecdc6047
Summary:
The thrift-types dep is quite heavyweight. It depends on fbthrift, and then
tokio-shim, and needs complex code gen logic. It is especially annoying in open
source builds (ex on non-fbcode Linux) since building the fbthrift with
compatible dependencies is quite slow and error prone.
The configparser is used by backingstore, which is used by edenfs. Decoupling
from thrift-types (and fbthrift, tokio-shim, etc.) avoids a lot of headache for edenfs
build.
The actual logic we need here is only JSON serialization for some simple
structs. There is no need to use Thrift serialization at all.
Let's just check-in the lightweight Rust structs that support serde JSON
serialization.
Ideally, the Thrift compiler could generate code that is serde struct only
without the fbthrift stuff. However, it does not support that for now.
A `build.rs` script is added to prevent the code from being out of sync
with the Thrift source. It seems fine given that the Thrift source is
rarely changed.
Reviewed By: DurhamG
Differential Revision: D30008264
fbshipit-source-id: 1822c3f0d47445a9e59d26575a567cb51b51cec0
