Summary:
It turns out that D30052693 (fd10938296) was not enough. The FailScenario cannot be setup
from different threads too. Doing that would cause deadlock. So let's avoid
that too.
Reviewed By: DurhamG
Differential Revision: D30124930
fbshipit-source-id: 0095bcf5ad3a99831d9b9c75a1e9f2c50729819b
Summary:
Now we validate the bonsai changeset uploaded via edenapi, by using the `RepoWriteContext.create_changeset` function, instead of directly creating the changeset using `BonsaiChangesetMut`.
I left a comment with a possible future improvement, where we can use upload tokens on `create_changeset` to avoid querying the blobstore for file size.
Differential Revision: D30045939
fbshipit-source-id: 84bb383879f8a25464044487eb99bd38b2849537
Summary:
This simplifies both client and server code to make bonsai changeset uploading be simpler for snapshots, as we only need a single commit, no mutations, etc.
This will make it easier to validate the bonsai changeset on the next diff.
It is fine to change both client and server as this code is not still in production, so we don't need to worry about rollout order.
Reviewed By: StanislavGlebik
Differential Revision: D30044542
fbshipit-source-id: d14bf58d671bc3bb5ff54b07c21f1781a043e0cf
Summary:
Using bonsai upload on commitcloud was useful to get things started, but it's not really necessary, and by removing we can simplify things a bit, since for snapshots it's a single commit.
This diff removes the client code. The server code is still used by snapshots, but will be simplified in a following diff.
Reviewed By: liubov-dmitrieva
Differential Revision: D30018157
fbshipit-source-id: 98bf0d30959de0c63b01bd11442f0bb3c538fcc7
Summary:
This diff addresses [this comment](https://www.internalfb.com/diff/D29849964 (4bde7b7488)?dst_version_fbid=244353817531864&transaction_fbid=342353780770798).
- It removes the bit of code in `process_files_upload` that adds file size to the metadata.
- In order for this not to break the bonsai upload, I made it so the lookup call returns upload tokens with file size when looking up a file.
- Took the opportunity to do some refactoring
- Consolidated duplicated functions in `convert_file_to_content_id`, and added some helpful From implementations to make calling it more ergonomic.
- `convert_file_to_content_id` now doesn't fail when the file doesn't exist, instead returns option (also fixed the callsite)
Reviewed By: liubov-dmitrieva
Differential Revision: D30016963
fbshipit-source-id: aae8a085d7a207e50679bb1210277a9e21a32de8
Summary:
These fields had placeholder values and are now populated properly. I found out about `workingctx` which has most information I need about uncommitted changes that will create the snapshot.
TODO on next diffs:
- Remove code for uploading commit cloud stuff via bonsai, and simplify API.
- Verify created bonsai is valid before persisting it
- Store files and bonsai blobs in the created ephemeral bubble
- Modify bonsai changeset objects so they can contain snapshot information
Reviewed By: liubov-dmitrieva
Differential Revision: D30013238
fbshipit-source-id: c509d25c59d6396d922435cc3a97fc837d3fe51b
Summary: Using changes from D29995429, this returns the upload token of the changeset upload in the uploadsnapshot response.
Reviewed By: StanislavGlebik
Differential Revision: D30012368
fbshipit-source-id: 5ca54763153a474d1ce3c38ddeaa0efff071b09c
Summary:
Using the new macros from previous diffs, this creates a new `ChangesetId` edenapi type and adds it to AnyId, which allows it to be used from UploadToken.
It then adds the lookup method for it, and returns it from upload_bonsai_changeset call (instead of a fake HgId UploadToken).
This will be used so that the client can know the changeset id of the uploaded snapshot.
Reviewed By: StanislavGlebik
Differential Revision: D29995429
fbshipit-source-id: e2ee4b9b0ac21d6f5394afacbfed1802da64013b
Summary:
This newly added config controls the default behavior of `eden clone` and
whether a new clone will use NFS or FUSE. This is intended to facilitate the
transition to NFS from FUSE on macOS.
Differential Revision: D30110056
fbshipit-source-id: ea6b493aa803297952b46434f6d11d8edf58e40b
Summary:
Same as the previous diff, but for the scmstore path.
Previously file fetching during native checkout wasn't done in parallel, so if
the fetcher was stuck downloading one large file, other files would not be
downloaded while it waited. This diff breaks the file fetching into smaller
chunks and has the chunks fetched in parallel.
Reviewed By: andll
Differential Revision: D30075316
fbshipit-source-id: 42c0401212373296c0ccbf65a676dec80a8e7a90
Summary:
Previously file fetching during native checkout wasn't done in
parallel, so if the fetcher was stuck downloading one large file, other files
would not be downloaded while it waited. This diff breaks the file fetching into
smaller chunks and has the chunks fetched in parallel.
Reviewed By: andll
Differential Revision: D29948891
fbshipit-source-id: fc3de341f87bcfd5badb530769857fa52a9d3e08
Summary: It was not used and produced unused warning
Reviewed By: quark-zju
Differential Revision: D30113295
fbshipit-source-id: 9a0d63269c3d6fd07561f50d6f1879322bb926a8
Summary: This will make LFS and mononoke wireproto traffic go through a http proxy. It's behind `--config auth_proxy.http_proxy`.
Reviewed By: farnz
Differential Revision: D29935440
fbshipit-source-id: be9a5fb7579ad8d750edf4b3c3a24fac7005679c
Summary: Similar to D25615788 (06e016600b) for LFS, add EdenAPI settings for configuring `CURLOPT_LOW_SPEED_LIMIT` and `CURLOPT_LOW_SPEED_TIME`.
Reviewed By: quark-zju
Differential Revision: D30088120
fbshipit-source-id: b567eca22bc4f86aef4c5565756b213a973e3113
Summary:
Previously, if the server returned an HTTP status code indicating an error, we'd use the entire response body as the error message. Sometimes these response bodies are huge (for example, if we hit the wrong server and get served an HTML webpage), and in these cases the error message is basically useless. An actual error message from the server should be fairly short.
To avoid spewing the entire response body onto the user's terminal, let's truncate long messages. Additionally, if the error message looks like a web page (as determined via a crude heuristic), let's omit it entirely and replace it with a more helpful message.
Reviewed By: quark-zju
Differential Revision: D30078934
fbshipit-source-id: 40836bb9187f7479360ce73f581fd15980ab183c
Summary:
Split extension filtering functionality from `no_bad_filenames`.
This diff does not modify `no_bad_filenames`. I am planning to
land and deploy this diff, then update the Mononoke configuration
to start using the new hooks, then land and deploy the next diff
which will remove redundant functionality from `no_bad_filenames`.
Reviewed By: krallin
Differential Revision: D29997126
fbshipit-source-id: 2b76b6275b491f3e8950ec4cfd2b4a3dacb929c9
Summary: Previous diffs switched all our usage from chashmap to dashmap as dashmap upstream is more responsive. Now remove chashmap from the cargo vendoring.
Reviewed By: dtolnay
Differential Revision: D30046522
fbshipit-source-id: 111ef9375bd8095f8b7c95752ecbc1988fb0438d
Summary: As title. This was added as `thrift-py3` doesn't support CMake well but we can workaround by importing from py.
Reviewed By: xavierd
Differential Revision: D30078300
fbshipit-source-id: 277866f8b226f164b5e30231aa10b59c0aba5807
Summary:
This adds the options to `eden stats` for collecting only fast stats and printing in JSON.
`eden stats` can be slow especially due to collecting fb303 counters and private bytes. An example use case of this new lightweight endpoint is that Buck can poll it to display Eden related info in its cli (see [post](https://fb.workplace.com/groups/132499338763090/permalink/210396380973385/) for context).
Reviewed By: xavierd
Differential Revision: D29687041
fbshipit-source-id: a663e71231527c5dfb822acbf238af0ac6ce4a00
Summary: Currently we have to manually save the id returned from `start_recording`. After this, we can simply ask for the list of all active recorder sessions.
Reviewed By: genevievehelsel
Differential Revision: D30056117
fbshipit-source-id: 7fd69b70e7b04fcd0b3724f4ee16c5e5e86badaf
Summary:
This improves buck build(and likely other tools) execution time significantly on the cold cache, with eden api enabled(e.g. no import helper)
I have noticed during the tests that we don't urilize network well when accessing files not in cache.
Adding some instrumentation shows that we only do up to 8 parallel fetches on mercurial side.
This seem to be limited by number of hg queue workers, simply boosting this number from 8->32 improves performance a lot, and it is still manageable number of threads
We probably need some better solution to allow for greater parallelism.
With this diff time for `buck build fb4a` on the cold cache is reduced **from 121 to 95 minutes(-30%)**.
'Parsing buck files' stage is reduced **from 07:16 min to 05:53 min(-25%)**.
During buck build we get to actual parallelism of about 16 fetches which is limited by number of buck threads.
Bumping number of buck threads with `-j 32` further reduces build time to **54 minutes (-60% from original)**
Reviewed By: xavierd
Differential Revision: D30025720
fbshipit-source-id: 6bcae1f353a0d31d8ce632ccb991c2a02f3136fc
Summary: Switch from chashmap to dashmap as dashmap upstream is more responsive.
Reviewed By: krallin
Differential Revision: D30046524
fbshipit-source-id: b4377000d33421c9d23c499fd5d6c9ef4b302ee5
Summary: Switch from chashmap to dashmap as dashmap upstream is more responsive.
Reviewed By: krallin
Differential Revision: D30045037
fbshipit-source-id: 252e231d426389ccf726613f0b073159a7a70d68
Summary:
One of the things that megarepo_tool can do is to manually rewrite a commit
from one repo to another with a particular commit remapping version.
e.g.
```
source repo
X
|
P
target repo
X' <- rewritten X
|
A <- commit that exists only in target repo
|
P' <- rewritten P
```
Previously it always required manually setting the parents in the target repo
i.e. in the example above we'd need to tell that A is the new parent of
rewritten commit.
However this is not always convenient. Sometimes we just want megarepo_tool to take
parents in the source repo (i.e. P in the example above), remap them to large repo
(i.e. P' in the example above), and use P' as a target repo parent.
This diff adds a special option that lets us do so.
Reviewed By: farnz
Differential Revision: D30040016
fbshipit-source-id: 116dbe1803857053336ca76d0a65dbca8b14bd73
Summary:
Preupload LFS blobs for big file via LFS protocol rather than new EdenApi Uploads.
The `store.upload` function "skips" non lfs stuff but uploads lfs stuff remotely via LFS protocol.
The lfs implementation also uploads only what is missing, no difference from our implementation.
So, it seems it should be called right before our new Eden Api uploads.
Once it is called, our EdenApi uploads recognises the blobs that have been already uploaded and LFS blobs won't go via our new upload protocol.
Reviewed By: DurhamG
Differential Revision: D30012307
fbshipit-source-id: 39f09055ded71dc69fa0b12fe0ac74ee1a7d3659
Summary: Switch from chashmap to dashmap as dashmap upstream is more responsive.
Reviewed By: StanislavGlebik
Differential Revision: D30044747
fbshipit-source-id: a8eef5140542ddce4199bd052af01f41c75b53e8
Summary:
automv is quite slow if there are lots of files.
Let's disable it automatically if there are too many changed files.
Reviewed By: kulshrax
Differential Revision: D30052694
fbshipit-source-id: deff55e1cee9ce65f459c32e320d2389ea7cfe89
Summary:
* `FuseRequestContext` currently uses `fuseHeader_.opcode` to get the cause detail.
* However, `fuseHeader_.opcode` is set to 0 by `stealReq` to indicate that the request has been released.
* This lead to `getCauseDetail` returning `<unknown>` if we call it after the request is released (e.g. in the clean up future).
This adds the `opcode_` member so that `getCauseDetail` always return the correct cause detail regardless of whether the request has been released. This also removes `examineReq` as it's not used.
Reviewed By: xavierd
Differential Revision: D30050518
fbshipit-source-id: 0c3e4d31e59a5b5fd87967d6c976b573692a2af9
Summary:
Per comment in D30017261 (e9c039ab4a)
> [...] we can get rid of this ESTALE check. That dates back to when edenfs had a bug that returned ESTALE when reading the .eden directory sometimes.
Reviewed By: xavierd
Differential Revision: D30024979
fbshipit-source-id: 645097c8f689c916245845561fd3d824ff7df8b4
Summary: Added a new `edenapi.encoding` config which allows specifying the `Accept-Encoding` for EdenAPI HTTP requests.
Reviewed By: DurhamG
Differential Revision: D30028835
fbshipit-source-id: e80c49e31b0f2180d75fabe28080a7b0dd7383ba
Summary: Add support for setting the `Accept-Encoding` request header, which will allow for compressed responses.
Reviewed By: DurhamG
Differential Revision: D30004651
fbshipit-source-id: 5db784824e1c446c55e4aef5038eb4ab71f8a10b
Summary: Tweak the tracing logs so it does not print duplicated hashes.
Reviewed By: andll
Differential Revision: D29980948
fbshipit-source-id: 2244830f85b3f3beacb9a1264b84da7182ddae1b
Summary: This no longer depends on old-style futures, so we can remove the dependency.
Reviewed By: quark-zju
Differential Revision: D27708962
fbshipit-source-id: fd66fb2934ff631abe0bfcdae843fcc9b10d5fdc
Summary:
In integration tests, we want to be able to run through the megarepo processing, and then check that configs have persisted correctly, so that we can start async workers after sending a config change down, and see the change be picked up.
Make it possible
Reviewed By: StanislavGlebik
Differential Revision: D30012106
fbshipit-source-id: f944165e7b93451180a78d8287db8a59d71bbe13
Summary:
FailScenario nested in a single thread can cause deadlock on a Mutex, confirmed
by gdb backtrace. Fix that by avoiding nested FailScenario in a single thread.
Reviewed By: kulshrax
Differential Revision: D30052693
fbshipit-source-id: 8d38b2e40ec30c8024e563f079d2595d76bbae1b
Summary: Left over of D29989181 (ab8d843ebd).
Reviewed By: kulshrax
Differential Revision: D30052695
fbshipit-source-id: 3e126055c76f9d9e48566708abe1407bd2fecaa3
Summary: Switch from chashmap to dashmap as dashmap upstream is more responsive.
Reviewed By: StanislavGlebik
Differential Revision: D30044510
fbshipit-source-id: 8003ecba2f9c5d16e9cb6dced28f3785a062870d
Summary: 2 of the disabled are passing, and the last one just needed a simple tweak.
Reviewed By: fanzeyi
Differential Revision: D30029260
fbshipit-source-id: ad386632e8b6fbf9c76530c8ce2df7d38990bbbd
Summary:
The mode bits don't make a lot of sense on Windows, so let's use the plain
0o777 as that's what they will look like on Windows.
Reviewed By: chadaustin
Differential Revision: D30028892
fbshipit-source-id: ced737f1290438708536b732095cee18e1b1c073
Summary: The test is passing, enable it.
Reviewed By: chadaustin
Differential Revision: D30028894
fbshipit-source-id: b268fc63220704466c72c688a1f0f04a47573641
Summary:
The create_editor_that_writes_commit_messages function uses bash which is not
available on Windows. Instead, we can simply pass --message to Mercurial for
the same effect.
Reviewed By: chadaustin
Differential Revision: D30028893
fbshipit-source-id: 0a588c37bd4da85176546568126893192b29a574
Summary: This is unused, no need to keep it around.
Reviewed By: genevievehelsel
Differential Revision: D30046503
fbshipit-source-id: 1d20d9b4ce672d5d79410203807dbc93b4bce31a
Summary:
In current location cargo build generates binary file with name 'main'.
Moving main.rs up makes it generate proper binary name 'fsprobe'
Reviewed By: DurhamG
Differential Revision: D29998232
fbshipit-source-id: b33ffb3ca7209a7e19ec15705ba03a31c0449447
