Summary: Allow this to be used by more utilities.
Reviewed By: DurhamG
Differential Revision: D29958512
fbshipit-source-id: b6a1a7017102a4ff4ad252246d3252903bbb793f
Summary: Prepare before moving to a common directory
Reviewed By: StanislavGlebik
Differential Revision: D29958513
fbshipit-source-id: 48c04a9bf6e8c3800b3f187e7741d240ecff9e9c
Summary: This allows us to remove older files to free disk space.
Reviewed By: DurhamG
Differential Revision: D30092445
fbshipit-source-id: 20770968cfaeba4fb6f6c45b247cc1cdafb79a08
Summary: This makes upcoming changes a bit easier.
Reviewed By: yancouto
Differential Revision: D30092446
fbshipit-source-id: f62f1ced45c46dc48c3b1112af661d8fbcd80706
Summary: This removes some redundant lookups when using the `*_batch` APIs.
Reviewed By: andll
Differential Revision: D30094155
fbshipit-source-id: f64702a09f67b67a10a2d51545f3b6940e80022f
Summary: The test exposes issues that the local cache isn't used in some cases.
Reviewed By: andll
Differential Revision: D30094158
fbshipit-source-id: 0122154b8bd2f9d321e248720632e28b6ca0559e
Summary:
Similar to D29888937 (8cb3d3dd75). This helps getting the Rust traceback of why certain
code path gets triggered.
Reviewed By: andll
Differential Revision: D30094157
fbshipit-source-id: ef5bacbf4612ed115fd631056f6c004fe6d1d93f
Summary: This will be used to debug some over fetching issues.
Reviewed By: andll
Differential Revision: D30094159
fbshipit-source-id: cfbdff497fbf53f97f1c957bf77c25e26ee79286
Summary:
Prefetch vertex on checkout (which is more expected to require network to download
files) so committing on them does not trigger remote lookup.
Reviewed By: andll
Differential Revision: D30094156
fbshipit-source-id: 917df2bbad7cee51a06f2fa38172775423893c37
Summary:
See the added comments. In some cases we can avoid remote lookups. This would
help make commit/amend more offline friendly.
Reviewed By: DurhamG
Differential Revision: D30004908
fbshipit-source-id: 94fbc7934a1eb3ae1058d8c542211a885d5ad8e6
Summary:
Similar to D30094157. This is helpful to get a Rust backtrace on "not found"
errors. This is useful because the Python traceback might not have Rust
stack.
Ideally, it'd be better if we can integrate backtrace into the error types.
However, the stdlib backtrace is still unstable and the thirdparty backtrace
seems too heavyweight.
Reviewed By: DurhamG
Differential Revision: D30123355
fbshipit-source-id: 7af76422143340194d8499a75e8ed65e9bf1fcae
Summary:
We have seen slicing panics on NFS, like:
thread 'main' panicked at 'range end index 14079379 out of range for slice of length 0', eden/scm/lib/indexedlog/src/log/mod.rs:1678:14
Possibly when some files were stale or out of sync (ex. `log` got shorter than
expected somehow) due to NFS weirdness (bad locking?). Let's make them Rust
errors instead of panics for easier error reporting and investigation.
Reviewed By: DurhamG
Differential Revision: D30166177
fbshipit-source-id: 25cbd9c178b24580c176fe9469362fe0020ede95
Summary:
commitcloud CLEAN UP: remove last bits code related to an old interngraph commit cloud proxy
The configs are not used, the service_url in scm daemon is not used, the interngraph proxy is not used
Reviewed By: singhsrb
Differential Revision: D30160471
fbshipit-source-id: 07fee2b8733afa29659467cb296df3b47fe1b54b
Summary: These were accidentally left over from a previous diff.
Reviewed By: singhsrb
Differential Revision: D30133799
fbshipit-source-id: 311f746848fc823a6fbae9e58bafda6641104b1e
Summary:
file upload: pass content size as a parameter
We shouldn't rely on body size, because it will not allow us to build compression and also it won't allow us to implement streaming.
Reviewed By: yancouto
Differential Revision: D30100887
fbshipit-source-id: c16f79fa71fe320f61d15e1328b67026f586a1dc
Summary:
On Windows, EPIPE's raw os error might be 232, different from libc::EPIPE (32).
That means the Python world cannot check the errno properly to hide the EPIPE
error message. Fix it by reading io::ErrorKind first, and fallback to
raw_os_error.
Reviewed By: yancouto
Differential Revision: D30118006
fbshipit-source-id: b34a2142b24714e68d11a03a0771a5f9c037ed0f
Summary:
The "local log" algorithm in the fastlog extension is poorly implemented.
It can be much slower than the non-fastlog algorithm. This means, if the
"remote" fastlog thread does not work in production, then enabling fastlog
can actually seriously regress log performance.
The fastlog test (already) cannot meaningfully test the remote behavior. So it
is removed for now.
Differential Revision: D30145651
fbshipit-source-id: 601a38e03920fbcae86e23262949ae3e3a16b9af
Summary:
Original commit changeset: 34fe02ddf580
We are seeing reports of EdenFS only showing partial directories to our user. Local testing shows this commit seems to be causing the issue. Reverting it for now.
Reviewed By: kmancini
Differential Revision: D30136949
fbshipit-source-id: 7fcc81506c132055a5b639a383b8c9be68118dc5
Summary:
Practically only checking the "root" vertexes is also "correct". Let's do
that to reduce some remote lookups.
Reviewed By: andll
Differential Revision: D30004909
fbshipit-source-id: e46e8ee6b072270b860c5275b13f1e2103ef2b6a
Summary:
The test shows the offline commit problem that will be addressed by upcoming
changes.
Reviewed By: andll
Differential Revision: D30094154
fbshipit-source-id: c5f38ff6f0d33d241501050a1f6257633b90b31e
Summary:
IPython assumes time.clock exists but it is missing on Windows. Polyfill it
on Windows.
Reviewed By: yancouto
Differential Revision: D30106738
fbshipit-source-id: fd49417917b0b17e81eba599a474fc96f5624b42
Summary:
With lazy changelog (and lazy tree, files, etc), scanning the local repo is
painfully slow. Moreover, the remote request is fast and fastlog does not
seem to know to end the local thread immediately if the remote request has
ended.
Let's just disable the local scan by default to avoid excessive remote
lookups.
Reviewed By: singhsrb
Differential Revision: D30137868
fbshipit-source-id: 23eeb2a88d83ebb2bbea4993408da181d2e8003d
Summary:
It turns out that D30052693 (fd10938296) was not enough. The FailScenario cannot be setup
from different threads too. Doing that would cause deadlock. So let's avoid
that too.
Reviewed By: DurhamG
Differential Revision: D30124930
fbshipit-source-id: 0095bcf5ad3a99831d9b9c75a1e9f2c50729819b
Summary:
This simplifies both client and server code to make bonsai changeset uploading be simpler for snapshots, as we only need a single commit, no mutations, etc.
This will make it easier to validate the bonsai changeset on the next diff.
It is fine to change both client and server as this code is not still in production, so we don't need to worry about rollout order.
Reviewed By: StanislavGlebik
Differential Revision: D30044542
fbshipit-source-id: d14bf58d671bc3bb5ff54b07c21f1781a043e0cf
Summary:
Using bonsai upload on commitcloud was useful to get things started, but it's not really necessary, and by removing we can simplify things a bit, since for snapshots it's a single commit.
This diff removes the client code. The server code is still used by snapshots, but will be simplified in a following diff.
Reviewed By: liubov-dmitrieva
Differential Revision: D30018157
fbshipit-source-id: 98bf0d30959de0c63b01bd11442f0bb3c538fcc7
Summary:
This diff addresses [this comment](https://www.internalfb.com/diff/D29849964 (4bde7b7488)?dst_version_fbid=244353817531864&transaction_fbid=342353780770798).
- It removes the bit of code in `process_files_upload` that adds file size to the metadata.
- In order for this not to break the bonsai upload, I made it so the lookup call returns upload tokens with file size when looking up a file.
- Took the opportunity to do some refactoring
- Consolidated duplicated functions in `convert_file_to_content_id`, and added some helpful From implementations to make calling it more ergonomic.
- `convert_file_to_content_id` now doesn't fail when the file doesn't exist, instead returns option (also fixed the callsite)
Reviewed By: liubov-dmitrieva
Differential Revision: D30016963
fbshipit-source-id: aae8a085d7a207e50679bb1210277a9e21a32de8
Summary:
These fields had placeholder values and are now populated properly. I found out about `workingctx` which has most information I need about uncommitted changes that will create the snapshot.
TODO on next diffs:
- Remove code for uploading commit cloud stuff via bonsai, and simplify API.
- Verify created bonsai is valid before persisting it
- Store files and bonsai blobs in the created ephemeral bubble
- Modify bonsai changeset objects so they can contain snapshot information
Reviewed By: liubov-dmitrieva
Differential Revision: D30013238
fbshipit-source-id: c509d25c59d6396d922435cc3a97fc837d3fe51b
Summary: Using changes from D29995429, this returns the upload token of the changeset upload in the uploadsnapshot response.
Reviewed By: StanislavGlebik
Differential Revision: D30012368
fbshipit-source-id: 5ca54763153a474d1ce3c38ddeaa0efff071b09c
Summary:
Using the new macros from previous diffs, this creates a new `ChangesetId` edenapi type and adds it to AnyId, which allows it to be used from UploadToken.
It then adds the lookup method for it, and returns it from upload_bonsai_changeset call (instead of a fake HgId UploadToken).
This will be used so that the client can know the changeset id of the uploaded snapshot.
Reviewed By: StanislavGlebik
Differential Revision: D29995429
fbshipit-source-id: e2ee4b9b0ac21d6f5394afacbfed1802da64013b
Summary:
Same as the previous diff, but for the scmstore path.
Previously file fetching during native checkout wasn't done in parallel, so if
the fetcher was stuck downloading one large file, other files would not be
downloaded while it waited. This diff breaks the file fetching into smaller
chunks and has the chunks fetched in parallel.
Reviewed By: andll
Differential Revision: D30075316
fbshipit-source-id: 42c0401212373296c0ccbf65a676dec80a8e7a90
Summary:
Previously file fetching during native checkout wasn't done in
parallel, so if the fetcher was stuck downloading one large file, other files
would not be downloaded while it waited. This diff breaks the file fetching into
smaller chunks and has the chunks fetched in parallel.
Reviewed By: andll
Differential Revision: D29948891
fbshipit-source-id: fc3de341f87bcfd5badb530769857fa52a9d3e08
Summary: It was not used and produced unused warning
Reviewed By: quark-zju
Differential Revision: D30113295
fbshipit-source-id: 9a0d63269c3d6fd07561f50d6f1879322bb926a8
Summary: This will make LFS and mononoke wireproto traffic go through a http proxy. It's behind `--config auth_proxy.http_proxy`.
Reviewed By: farnz
Differential Revision: D29935440
fbshipit-source-id: be9a5fb7579ad8d750edf4b3c3a24fac7005679c
Summary: Similar to D25615788 (06e016600b) for LFS, add EdenAPI settings for configuring `CURLOPT_LOW_SPEED_LIMIT` and `CURLOPT_LOW_SPEED_TIME`.
Reviewed By: quark-zju
Differential Revision: D30088120
fbshipit-source-id: b567eca22bc4f86aef4c5565756b213a973e3113
Summary:
Previously, if the server returned an HTTP status code indicating an error, we'd use the entire response body as the error message. Sometimes these response bodies are huge (for example, if we hit the wrong server and get served an HTML webpage), and in these cases the error message is basically useless. An actual error message from the server should be fairly short.
To avoid spewing the entire response body onto the user's terminal, let's truncate long messages. Additionally, if the error message looks like a web page (as determined via a crude heuristic), let's omit it entirely and replace it with a more helpful message.
Reviewed By: quark-zju
Differential Revision: D30078934
fbshipit-source-id: 40836bb9187f7479360ce73f581fd15980ab183c
Summary: Switch from chashmap to dashmap as dashmap upstream is more responsive.
Reviewed By: krallin
Differential Revision: D30045037
fbshipit-source-id: 252e231d426389ccf726613f0b073159a7a70d68
Summary:
Preupload LFS blobs for big file via LFS protocol rather than new EdenApi Uploads.
The `store.upload` function "skips" non lfs stuff but uploads lfs stuff remotely via LFS protocol.
The lfs implementation also uploads only what is missing, no difference from our implementation.
So, it seems it should be called right before our new Eden Api uploads.
Once it is called, our EdenApi uploads recognises the blobs that have been already uploaded and LFS blobs won't go via our new upload protocol.
Reviewed By: DurhamG
Differential Revision: D30012307
fbshipit-source-id: 39f09055ded71dc69fa0b12fe0ac74ee1a7d3659
Summary:
automv is quite slow if there are lots of files.
Let's disable it automatically if there are too many changed files.
Reviewed By: kulshrax
Differential Revision: D30052694
fbshipit-source-id: deff55e1cee9ce65f459c32e320d2389ea7cfe89
Summary:
Per comment in D30017261 (e9c039ab4a)
> [...] we can get rid of this ESTALE check. That dates back to when edenfs had a bug that returned ESTALE when reading the .eden directory sometimes.
Reviewed By: xavierd
Differential Revision: D30024979
fbshipit-source-id: 645097c8f689c916245845561fd3d824ff7df8b4
Summary: Added a new `edenapi.encoding` config which allows specifying the `Accept-Encoding` for EdenAPI HTTP requests.
Reviewed By: DurhamG
Differential Revision: D30028835
fbshipit-source-id: e80c49e31b0f2180d75fabe28080a7b0dd7383ba
Summary: Add support for setting the `Accept-Encoding` request header, which will allow for compressed responses.
Reviewed By: DurhamG
Differential Revision: D30004651
fbshipit-source-id: 5db784824e1c446c55e4aef5038eb4ab71f8a10b
Summary: Tweak the tracing logs so it does not print duplicated hashes.
Reviewed By: andll
Differential Revision: D29980948
fbshipit-source-id: 2244830f85b3f3beacb9a1264b84da7182ddae1b
Summary:
FailScenario nested in a single thread can cause deadlock on a Mutex, confirmed
by gdb backtrace. Fix that by avoiding nested FailScenario in a single thread.
Reviewed By: kulshrax
Differential Revision: D30052693
fbshipit-source-id: 8d38b2e40ec30c8024e563f079d2595d76bbae1b
