Summary:
During flush, there could be newly added vertexes in the master group,
overlay_map_next_id needs to be updated accordingly so lazy lookup
can still work. This is especially needed when the non-master group
needs to be re-assigned.
Reviewed By: andll
Differential Revision: D30314051
fbshipit-source-id: cce7080f62aec2617b8f3d7194864df41dfff7e0
Summary:
In case anything went wrong in the pull fast path, preserve `self` as unchanged
to avoid possible issues.
Reviewed By: andll
Differential Revision: D30314052
fbshipit-source-id: 127a7aeda8f27f7862d1ad92847c2e911131b4b4
Summary:
Will be used in tests to check what data is inserted into IdMap during clone or
pull.
Reviewed By: andll
Differential Revision: D30314054
fbshipit-source-id: 8a58473ef3d85263985c1be8a6c57a9a10b3fdb4
Summary:
In rare cases, the local IdMap might miss parents of draft roots.
This provides a way to emulate that in tests.
Reviewed By: andll
Differential Revision: D30314050
fbshipit-source-id: 09983604bf3259e62a41224a579774265ae0b272
Summary: This would help investigate some errors.
Reviewed By: andll
Differential Revision: D30314057
fbshipit-source-id: 070e75cf0a39180c544b49c9ed1292d8536040fa
Summary:
- This diff adds validation so that changesets that are not snapshots cannot have untracked or missing files.
- It removes the THIS IS A SNAPSHOT commit message.
- It makes the snapshot created by `hg snapshot createremote` be an actual snapshot.
Differential Revision: D30159184
fbshipit-source-id: 976968c0c2222f950a4a937aa805b25dc07c9207
Summary:
This diff makes the snapshot command upload all types of files that were missing (added/untracked/missing/deleted), using the new types of file changes added on the previous diff.
Next steps:
- Add some indicator to Bonsai Changeset saying it is a snapshot. Verify only snapshots can have certain file changes (untracked/missing).
- Upload the files and the changeset inside an ephemeral bubble instead of in the main blobstore
- Start writing the `snapshot restore` command
Differential Revision: D30137673
fbshipit-source-id: 555238f1d64a5438cde35a843043884a939de4fe
Summary: Allow `manifest-tree::Entry` to be constructed directly by other crates, and introduce a `TryFrom` conversion to `manifest::List::Directory`. This will be used by the scmstore `BackingStore` implementation to avoid having to rely on the `TreeStore` trait, which does not support batching.
Reviewed By: kulshrax
Differential Revision: D30282738
fbshipit-source-id: 590350dd53217fa8a181e91b194abca753a8adbe
Summary: Add a new results adapter for `FileStore` (which will probably become the primary method of consuming batch errors in the future), which iterates over all keys, and explicitly encodes not found with `Option`, separate from the outer `Result`.
Reviewed By: kulshrax
Differential Revision: D30254000
fbshipit-source-id: 074c6ac4c279fdf72078dca17231e58d0c704956
Summary:
Add a `local` method to `TreeStore` like the one on `FileStore` which returns a `TreeStore` with only the local subset of backends.
Required for `BackingStore` implementation.
Reviewed By: kulshrax
Differential Revision: D30253980
fbshipit-source-id: 142f2d88454826ff9cb9c34f30b7d21bf62b297c
Summary: Add a new method, `edenapi_override`, to the `TreeStore` and `FileStore` builders for use in the `BackingStore` implementation.
Reviewed By: kulshrax
Differential Revision: D30253837
fbshipit-source-id: 4a42e83621fb2634024e4ee8529c26aeae0256a4
Summary:
Instrument all the fetching-related legacy API implementations (ie, not `get_logged_fetches` and `get_shared_mutable`) to track the number of calls, keys, and single-key calls.
This introduces an additional lock acquisition to each of these implementations (and it'd be awkward to merge it with the one in `fetch`), but I think that's probably fine.
For APIs which do not support a variable number of keys, I use `.call(0)` so we simply track the total number of API calls.
Reviewed By: DurhamG
Differential Revision: D30003444
fbshipit-source-id: 8756d2669ca038b3f6a08e211e44e8ccb9251312
Summary:
Adds tracking of FileStore write errors.
Write batches will still abort after the first failure as written.
Reviewed By: DurhamG
Differential Revision: D29997203
fbshipit-source-id: e1cc2ffc4a8d97ca935a7fc9aab30bde3dc548b2
Summary: Refactor the `FileStore::write_batch` method, extracting the various write cases into other methods.
Reviewed By: DurhamG
Differential Revision: D29997096
fbshipit-source-id: 8efac4b2efa2e2225d39583a5c6893efc5096fec
Summary:
Previously errors from `prefetch` weren't counted. With this change I move calculating the pending keys out of the `inner` method, making it easier to handle all batch error metrics from `?` calls right beside the existing `other_errors` call.
This is probably how I'll structure the other methods too, though it's still a little subtle - currently the batch errors only come from `prefetch`, but if batch errors were returned after the `drain` call (which always drains the vector, even if you don't consume the whole iterator), the second `pending.len()` call could incorrectly report 0. To address this, I could separate out the "single key fetching" part of the method, or just let it be understood that you should repeatedly call `pop` in such cases.
Reviewed By: DurhamG
Differential Revision: D29995789
fbshipit-source-id: 09753081c5e7076b83cb26fdddb949205ad7cd43
Summary: Previously, contentstore metrics were tracked as a special case. Now that I'm tracking metrics on each backend, I've standardized the contentstore metrics to match.
Reviewed By: DurhamG
Differential Revision: D29995500
fbshipit-source-id: 3133dca5912d9cd6c26bd88d9aeb6a572f6eca40
Summary:
Increase [CURLOPT_LOW_SPEED_TIME](`https://curl.se/libcurl/c/CURLOPT_LOW_SPEED_TIME.html`) because users were hitting low speed timeouts during normal usage.
2 minutes was arbitrarily chosen, but it seems unlikely that we'd have situations where connections legitimately sit idle for more than 2 minutes. (If it turns out we do, that would be a bit concerning.)
This diff also updates the config option to specify the time in seconds instead of milliseconds because libcurl only supports second-level precision when specifying `CURLOPT_LOW_SPEED_TIME`. The config option name has been updated to explicitly include the unit to avoid confusion.
Differential Revision: D30272017
fbshipit-source-id: 0ae5e912c0dafc897fe19de3ea8e5aeb4ed8459c
Summary:
Client::fetch was async method for no reason(it does not contain any awaits)
This diff removes async and makes it clear that fetch only prepares futures but does not itself contain async code
Reviewed By: DurhamG
Differential Revision: D30137299
fbshipit-source-id: 87f80cf906bbea856f9f54b88a3609d0aad03a73
Summary: This allow to render metrics in the real time, similar to how we render network usage
Reviewed By: DurhamG
Differential Revision: D30137295
fbshipit-source-id: 8843b574971c1ccaaae0727be6f9d46d1d90625f
Summary: See docstring to init_from_env for the description
Reviewed By: quark-zju
Differential Revision: D30137304
fbshipit-source-id: ad3fb5e0cbb52ca405959eb0d394f5a2c9761ad9
Summary: This allows to count number of parallel operations, like parallel entrance to some fn or parallel requests
Reviewed By: quark-zju
Differential Revision: D30137297
fbshipit-source-id: 74585dcab353091619a0a1f217548121079b463c
Summary: Allow this to be used by more utilities.
Reviewed By: DurhamG
Differential Revision: D29958512
fbshipit-source-id: b6a1a7017102a4ff4ad252246d3252903bbb793f
Summary: Prepare before moving to a common directory
Reviewed By: StanislavGlebik
Differential Revision: D29958513
fbshipit-source-id: 48c04a9bf6e8c3800b3f187e7741d240ecff9e9c
Summary: This removes some redundant lookups when using the `*_batch` APIs.
Reviewed By: andll
Differential Revision: D30094155
fbshipit-source-id: f64702a09f67b67a10a2d51545f3b6940e80022f
Summary: The test exposes issues that the local cache isn't used in some cases.
Reviewed By: andll
Differential Revision: D30094158
fbshipit-source-id: 0122154b8bd2f9d321e248720632e28b6ca0559e
Summary:
Similar to D29888937 (8cb3d3dd75). This helps getting the Rust traceback of why certain
code path gets triggered.
Reviewed By: andll
Differential Revision: D30094157
fbshipit-source-id: ef5bacbf4612ed115fd631056f6c004fe6d1d93f
Summary: This will be used to debug some over fetching issues.
Reviewed By: andll
Differential Revision: D30094159
fbshipit-source-id: cfbdff497fbf53f97f1c957bf77c25e26ee79286
Summary:
See the added comments. In some cases we can avoid remote lookups. This would
help make commit/amend more offline friendly.
Reviewed By: DurhamG
Differential Revision: D30004908
fbshipit-source-id: 94fbc7934a1eb3ae1058d8c542211a885d5ad8e6
Summary:
Similar to D30094157. This is helpful to get a Rust backtrace on "not found"
errors. This is useful because the Python traceback might not have Rust
stack.
Ideally, it'd be better if we can integrate backtrace into the error types.
However, the stdlib backtrace is still unstable and the thirdparty backtrace
seems too heavyweight.
Reviewed By: DurhamG
Differential Revision: D30123355
fbshipit-source-id: 7af76422143340194d8499a75e8ed65e9bf1fcae
Summary:
We have seen slicing panics on NFS, like:
thread 'main' panicked at 'range end index 14079379 out of range for slice of length 0', eden/scm/lib/indexedlog/src/log/mod.rs:1678:14
Possibly when some files were stale or out of sync (ex. `log` got shorter than
expected somehow) due to NFS weirdness (bad locking?). Let's make them Rust
errors instead of panics for easier error reporting and investigation.
Reviewed By: DurhamG
Differential Revision: D30166177
fbshipit-source-id: 25cbd9c178b24580c176fe9469362fe0020ede95
Summary:
commitcloud CLEAN UP: remove last bits code related to an old interngraph commit cloud proxy
The configs are not used, the service_url in scm daemon is not used, the interngraph proxy is not used
Reviewed By: singhsrb
Differential Revision: D30160471
fbshipit-source-id: 07fee2b8733afa29659467cb296df3b47fe1b54b
Summary: These were accidentally left over from a previous diff.
Reviewed By: singhsrb
Differential Revision: D30133799
fbshipit-source-id: 311f746848fc823a6fbae9e58bafda6641104b1e
Summary:
file upload: pass content size as a parameter
We shouldn't rely on body size, because it will not allow us to build compression and also it won't allow us to implement streaming.
Reviewed By: yancouto
Differential Revision: D30100887
fbshipit-source-id: c16f79fa71fe320f61d15e1328b67026f586a1dc
Summary:
On Windows, EPIPE's raw os error might be 232, different from libc::EPIPE (32).
That means the Python world cannot check the errno properly to hide the EPIPE
error message. Fix it by reading io::ErrorKind first, and fallback to
raw_os_error.
Reviewed By: yancouto
Differential Revision: D30118006
fbshipit-source-id: b34a2142b24714e68d11a03a0771a5f9c037ed0f
Summary:
Original commit changeset: 34fe02ddf580
We are seeing reports of EdenFS only showing partial directories to our user. Local testing shows this commit seems to be causing the issue. Reverting it for now.
Reviewed By: kmancini
Differential Revision: D30136949
fbshipit-source-id: 7fcc81506c132055a5b639a383b8c9be68118dc5
Summary:
Practically only checking the "root" vertexes is also "correct". Let's do
that to reduce some remote lookups.
Reviewed By: andll
Differential Revision: D30004909
fbshipit-source-id: e46e8ee6b072270b860c5275b13f1e2103ef2b6a
Summary:
It turns out that D30052693 (fd10938296) was not enough. The FailScenario cannot be setup
from different threads too. Doing that would cause deadlock. So let's avoid
that too.
Reviewed By: DurhamG
Differential Revision: D30124930
fbshipit-source-id: 0095bcf5ad3a99831d9b9c75a1e9f2c50729819b
Summary:
This simplifies both client and server code to make bonsai changeset uploading be simpler for snapshots, as we only need a single commit, no mutations, etc.
This will make it easier to validate the bonsai changeset on the next diff.
It is fine to change both client and server as this code is not still in production, so we don't need to worry about rollout order.
Reviewed By: StanislavGlebik
Differential Revision: D30044542
fbshipit-source-id: d14bf58d671bc3bb5ff54b07c21f1781a043e0cf
Summary:
This diff addresses [this comment](https://www.internalfb.com/diff/D29849964 (4bde7b7488)?dst_version_fbid=244353817531864&transaction_fbid=342353780770798).
- It removes the bit of code in `process_files_upload` that adds file size to the metadata.
- In order for this not to break the bonsai upload, I made it so the lookup call returns upload tokens with file size when looking up a file.
- Took the opportunity to do some refactoring
- Consolidated duplicated functions in `convert_file_to_content_id`, and added some helpful From implementations to make calling it more ergonomic.
- `convert_file_to_content_id` now doesn't fail when the file doesn't exist, instead returns option (also fixed the callsite)
Reviewed By: liubov-dmitrieva
Differential Revision: D30016963
fbshipit-source-id: aae8a085d7a207e50679bb1210277a9e21a32de8
