Summary:
Generic container for a bunch of uniform objects. This is primarily intended
for requests and responses which can be difficult to evolve when the top level
object is an array. For cases where evolution is required we would
probably replace the Batch wrapper with a specialized type. For example,
starting from `Batch<MyRequest>` we would change to:
struct MyRequestBatch {
pub batch: Vec<T>,
pub evolution: Evolution,
}
Reviewed By: quark-zju
Differential Revision: D28034534
fbshipit-source-id: d231c063eeacf3500b75ae76bcc101ccbcda8881
Summary: Those methods only access store/network to fetch content but does not write to disk
Differential Revision: D28040640
fbshipit-source-id: e45dd08e12d128d54b3446e1137465981cde8f13
Summary:
This and following diff will refactor CheckoutPlan creation.
Right now we create CheckoutPlan from manifest diff and then manipulate it with methods like `with_sparse_profile` to adjust plan for different cases.
Those 'adjustment' do not work great with the structure of CheckoutPlan, for example `with_sparse_profile` has to create temporary HashSet just to index files in CheckoutPlan
We are going to add more adjustments in the future (for example, checkout --clean), and will run into same issues with current structure of CheckoutPlan
To avoid those issues, we are going to refactor this code, so that instead of Diff -> CheckoutPlan -> adjustments, we are going to have Diff -> ActionMap -> adjustments -> CheckoutPlan
The structure of CheckoutPlan is still good for it's direct purpose (execution), but all the 'changes' to the plan will be done in ActionMap instead.
Reviewed By: DurhamG
Differential Revision: D27980390
fbshipit-source-id: 403f371fd2fe7760984925a38429e1bfb88d8e3f
Summary: When checking out on dirty copy without --clean this function can be used to check if checkout operation conflicts with currently modified files
Reviewed By: quark-zju
Differential Revision: D27953965
fbshipit-source-id: 4096506e4cbf8b102e0afa1a929c066dfa474825
Summary:
This crate introduces consumer API for status in rust
Currently the implementation will just take status from Python and convert it into this struct
But in the future we can get pure Rust implementation to get status
Reviewed By: quark-zju
Differential Revision: D27953963
fbshipit-source-id: 29c876400c82056eaf81fffa4adc814473853c1e
Summary: This method can be used to 'normalize' path for case insentive use cases
Reviewed By: quark-zju
Differential Revision: D27953964
fbshipit-source-id: 421832af22af9a3b56eec0d045b9f983592ed192
Summary: It has been fixed and we now set auth config with higher priority anyway.
Reviewed By: johansglock
Differential Revision: D28026081
fbshipit-source-id: 7086b48139bb05ffadd782898a1758ae06236aca
Summary:
The check unknown logic would block checkout for any unknown files that
were to be overwritten. We want to allow checkouts where the unknown file has
the same content as the desired checkout value. Ideally we'd check it against
the SHA1 hash of the file we're about to checkout, but since content hashes
aren't available yet we can limit our check to resumed checkouts for now.
Reviewed By: andll
Differential Revision: D27804719
fbshipit-source-id: e129ca694080051420e2cb685c7eeb5f1adee005
Summary:
Every function on CheckoutPlan required the VFS already, and the
CheckoutProgress is storing the VFS and living on the CheckoutPlan, so it makes
sense to just store the VFS on the CheckoutPlan.
Reviewed By: andll
Differential Revision: D27825088
fbshipit-source-id: 3d063fdfd1a50983b60d00a3992a893e71732f94
Summary:
Now that CheckoutPlan can look for untracked files, it breaks the
ability to continue a checkout since those untracked files are considered dirty.
In a later diff we'll use the CheckoutProgress to inspect the dirty files and
determine which are actually dirty and which can be overwritten. To do so
though, we need access to the CheckoutProgress earlier. So let's just store it
on the CheckoutPlan.
This is a little awkward because we're passing the root VFS to the constructor
so CheckoutProgress can be instantiated, but then also passing it to every
CheckoutPlan function as well. We should probably just store the vfs on the
CheckoutPlan. If others agree, I can make a diff to do that.
Reviewed By: andll
Differential Revision: D27804720
fbshipit-source-id: e819c27fa8580c82a8cf8f0baf22ac1ea707ee54
Summary:
Add a way to extend the graph with concrete commit hashes, without specifying
exact commit messages.
Reviewed By: sfilipco
Differential Revision: D27897894
fbshipit-source-id: fccd64b2fef1386d79cddd841208da6a938a5217
Summary:
Current implementation had a bug(demonstrated in test case) in handling unknown files on case insensitive fs.
When file is replaced with another file, whose name only differs in case, we get two distinct update operations - rm file, and create file.
Create operation checks against unknown files, and see that file "exists". In this case operation is aborted.
However, we should proceed in this case, and this diff fixes it.
Reviewed By: quark-zju
Differential Revision: D27926953
fbshipit-source-id: 48c8824322d6e5dd9ae57fee1f849b57dc11a4df
Summary: Will be useful on case insensitive fs
Reviewed By: quark-zju
Differential Revision: D27946982
fbshipit-source-id: e7a2fd0ee503c4a580531e6f52225fe2316e5b76
Summary: This diff adds flag to VFS to detect whether FS is case sensitive. The logic in this code losely follows similar logic in Python
Reviewed By: quark-zju
Differential Revision: D27926952
fbshipit-source-id: 36fdf4187ae513b25346f704050c64f9a1a4ec74
Summary:
Update the zstd crates.
This also patches async-compression crate to point at my fork until upstream PR https://github.com/Nemo157/async-compression/pull/117 to update to zstd 1.4.9 can land.
Reviewed By: jsgf, dtolnay
Differential Revision: D27942174
fbshipit-source-id: 26e604d71417e6910a02ec27142c3a16ea516c2b
Summary: Those conflicts can be resolved in Python using textual 3-way merge
Reviewed By: DurhamG
Differential Revision: D27752770
fbshipit-source-id: 816a601112ee2e747d780f8b17473049df46b469
Summary: This diff exposes manifestbuilder that can be used to construct memctx from Python
Reviewed By: DurhamG
Differential Revision: D27639395
fbshipit-source-id: ed047d3d7533f9d2bc592a5d948dc01e429692a7
Summary: Add bookmark method to Edenapi client for use in Mercurial client.
Reviewed By: kulshrax
Differential Revision: D27174441
fbshipit-source-id: cdc324e9115e87eab2e078209bbbc266e4e1dcdc
Summary:
It is considered an anti-pattern (https://rust-unofficial.github.io/patterns/anti_patterns/deny-warnings.html)
and is causing Github CI breakage unnecessarily (https://github.com/facebookexperimental/eden/runs/2402094456):
error: function is never used: `example_blob`
--> src/lfs.rs:1860:8
|
1860 | fn example_blob() -> (Sha256, usize, Bytes) {
| ^^^^^^^^^^^^
|
note: the lint level is defined here
--> src/lib.rs:125:9
|
125 | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(dead_code)]` implied by `#[deny(warnings)]`
Reviewed By: andll
Differential Revision: D27911477
fbshipit-source-id: df8d642fe74fe311eb0f329d977b9b8270c27bf4
Summary:
In case the stderr is redirected but stdout is not:
EDENSCM_LOG=edenscm::mercurial=trace lhg log archival.py 2>/tmp/1
cat /tmp/1
The expected behavior is to still use the pager for stdout output, and the
stderr should be redirected to the specified file.
Reviewed By: andll
Differential Revision: D27867884
fbshipit-source-id: c369bc6be40fc200c4c0e2c9bb38b5faeb1208f2
Summary:
This will be useful to test if a (frequently called) callsite is enabled or not
without having the logging side effect.
Reviewed By: andll
Differential Revision: D27867672
fbshipit-source-id: 361cb18a7a4680932dcfc9d5496d2906e1dc1f9f
Summary:
The previous choice of the upload / download arrows do not render in Windows
`cmd.exe` terminal using common fonts:
- Cascadia Code
- Cascadia Mono
- Consolas
- Courier New
- JetBrains Mono
- Lucida Console
Replace them with tri-angles that can be rendered using the above fonts.
Note: newer terminals like Microsoft Terminal and WezTerm can render the characters
just fine.
Reviewed By: andll, ikostia
Differential Revision: D27894085
fbshipit-source-id: c53d995355e66ba793d80afc1b36fc83853d07f8
Summary:
Integration test showing that MismatchedHeads errors are sent over the wire to
the client.
Reviewed By: quark-zju
Differential Revision: D27798555
fbshipit-source-id: b14a213e9055486bf07ecbb4b5453385df701b48
Summary:
This error occurs when the client sends us a set of heads that we cannot match.
For example when the client forces a commit in the master group; this was
possible with revlogs but should be a bug with Segmented Changelog. This can
also happen when master moves backwards, clients and server have multiple heads
then the server reseeds.
Clients that get this error should reclone.
Reviewed By: quark-zju
Differential Revision: D27786602
fbshipit-source-id: 9854ccee929ae0a845236ebd83ed68158c93fc7a
Summary:
We want to distinguish between no location and failure to compute location.
It is useful to know on the client side if we failed to compute the location of
some hash or we are not sending it because we don't have a location for it.
We have different options for hash-to-location in particular but the problem is
a general one. Does it make sense for us to skip sending error at the EdenApi
handler layer? I don't think so, I think that's an old hask. We should be
evolving the EdenApi handler layer to handle application errors.
This is still at the exploration phase. I haven't fixed the client side to
handle the errors.
Reviewed By: quark-zju, kulshrax
Differential Revision: D27549926
fbshipit-source-id: 7094160fe997af0e69c8d006b9731ea7dfb3e3f8
Summary:
The goal here is to add an easy way of propagating application errors from
server to the client.
The most convenient form for an error message is a message string so we just
start with that. I think the most important thing to add is some way of
communicating whether the error is retryable or not.
With conversions for anyhow::Error to WireError and from WireError to
ServerError it should be trivial for application code to pass application
errors down to the client.
The format here is driven by the fact that we have streams in the response
object. A batch oriented format for responses has more options. For example
with batches it is common to have a response object that has 3 categories:
1. found: HashMap, 2. not_found: Vec, 3. errors: Vec/HashMap
Reviewed By: kulshrax
Differential Revision: D27549923
fbshipit-source-id: 33b790253adc4761ea9ac7caced4148f4026e620
Summary:
On Mac this introduces 150ms delays in every indexedlog flush. During
an amend of a stack with 32 commits, this fsync accounted for 2/3rds of the time
spent.
Since commitcloud is pretty reliable these days, I think this is no longer
necessary.
Reviewed By: andll
Differential Revision: D27896589
fbshipit-source-id: a13a494c54ffea5a670ed942b366620619af2bd0
Summary: The Python `[auth]` matching code does not take cert validity into account when performing certificate matching, whereas the Rust version of the code does. In practice, the existing call sites for the Rust code disable match-time validation, and instead validate the certificate at time-of-use. This diff makes the Rust code's behavior match Python so we can remove the latter entirely.
Reviewed By: DurhamG
Differential Revision: D27837343
fbshipit-source-id: 0bfb5ebc3a36c8fa748cb289e2d8d1495ba8b296
Summary:
Like it says in the title. We'd like to ask for the exact size that was
configured, because this way we can set the chunk size to the LFS threshold and
it avoids overlapping any file chunks server side.
Reviewed By: DurhamG
Differential Revision: D27824418
fbshipit-source-id: 43f40eb87080ec58e813ba1f1dda5b6a5e9f98ee
Summary:
This replicates behaviour of Python code - if unknown file content matches content of the file to be check out, do not abort checkout
This is useful for resuming interrupted checkouts / clones
Reviewed By: DurhamG
Differential Revision: D27799147
fbshipit-source-id: 7d2582583525da18fba08dfcd8bced2b619304de
Summary: It's been showing up while building mononoke. Let's fix it
Reviewed By: sfilipco
Differential Revision: D27789928
fbshipit-source-id: a15912f66b9ad3370545aed88405dbeb800e63de
Summary: This seems to have broken the EdenFS HgPrefetch test.
Reviewed By: xavierd
Differential Revision: D27795192
fbshipit-source-id: 80a748036961aa6a5750182bf65637fb76825341
Summary: This will show proper checkout progress when using native checkout
Reviewed By: quark-zju
Differential Revision: D27775423
fbshipit-source-id: 79f2afa02bd1fab7d5f747da1c714d4d1126ce7c
Summary:
EdenAPI makes heavy use of streaming HTTP responses consisting of a series of serialized CBOR values. In order to process the data in a streaming manner, we use the `CborStream` combinator, which attempts to deserialize the CBOR values as they are received.
`CborStream` hits a pathological case when it receives a very large CBOR value. Previously, it would always buffer the input stream into 1 MB chunks, and attempt to deserialize whenever a new chunk was received. In the case of downloading values that are >1GB in size, this meant potentially thousands of wasted deserialization attempts. In practice, this meant that EdenAPI would hang when receiving the content of large files.
To address this problem, this diff adds a simple heuristic: If a partial CBOR value exceeds the current buffer size, double the size threshold before attempting to deserialize again. This reduces the pathological case from `O(n^2)` to `O(log(n))` (with some caveats, described in the comment in the code).
Reviewed By: krallin
Differential Revision: D27759698
fbshipit-source-id: 67882c31ce95a934b96c61f1c72bd97cad942d2e
Summary:
Previously we'd skip dynamicconfigs when there wasn't a repo available.
Now that dynamicconfig can represent the NoRepo state, let's load dynamicconfigs
in that situation.
This also supports the case where there is no user name.
Reviewed By: kulshrax
Differential Revision: D26801059
fbshipit-source-id: 377cfffb6695a7fbe31303868a88862259ebf8c4
Summary: Add a new `edenapi.maxrequests` config option to allow controlling the number of parallel in-flight requests. This can be used to bound resource usage of the client when requesting large amounts of data.
Reviewed By: sfilipco
Differential Revision: D27724817
fbshipit-source-id: 8d607efa83d8b0b94074d1a6e06f6c536cc0c791
Summary: Add a method to allow setting `CURLMOPT_MAX_TOTAL_CONNECTIONS`, which limits the number of concurrent requests within a curl multi session. If the number of requests in the session exceeds this number, they will be queued and sent once earlier requests have completed.
Reviewed By: sfilipco
Differential Revision: D27724818
fbshipit-source-id: 436384aed9d6d29f426e5e45aebb7a72c24ba667
Summary:
Without this, `make local` will build `hostcaps` without fb-specific logic and
cause wrong configs being used. `hg up master` will error out like:
File "treemanifest/__init__.py", line 690, in _httpgetdesignatednodes
self.edenapi.trees(dpack, self.name, keys)
RustError: Server reported an error (403 Forbidden)
Reviewed By: quark-zju
Differential Revision: D27759821
fbshipit-source-id: d42895f44bc53003f2578b65640ebe4ee05d52e6
Summary:
Right now, if prefetch fails, we just give the client back an error saying
"content not found".
This isn't super helpful, because usually the reason the content is not found
is because we cannot talk to the server that has the content, so showing the
user why we cannot talk to said server is more useful.
I'd like to ship this gradually, so I also added a config flag to turn it off.
Initially I'll have the flag set, but I did default it to not-set in the code
so that our tests run with the desired configuration.
Note: I initially contemplated adding logging for this here, but after
discussion with xavierd it looks like just failing instead of eating the error
is probably a better approach (and it's much simpler). This is also consistent
with what EdenAPI does.
Reviewed By: mzr
Differential Revision: D27761572
fbshipit-source-id: 3506d9c97a00e3f076bd346883e07f49194b0b06
Summary:
Right now, if the server ever tells us a file is missing, we fail the entire
batch download. This is a bit unfortunate because other objects could still be
downloaded, but also because we lose the distinction between "server crashed"
and "server told us the data we want does not exist".
Besides, it's actually a bit unnecessary. Indeed, this fails, we just ignore
the error anyway up the stack, so it never actually gets accessed.
Reviewed By: mzr
Differential Revision: D27761574
fbshipit-source-id: cb4fb0526a3bf19c04ecb81c05d44d4d8afb81ad
Summary: We can just return the actual error here now.
Reviewed By: sfilipco
Differential Revision: D27761573
fbshipit-source-id: 0866f976b4ed434deffd96be6820ad05d27b7b93
Summary:
NOTE: The revisionstore LFS tests talk to prod Mononoke LFS, so the test here
will fail until that is deployed.
Like it says in the title, this adds support for downloading content in chunks
instead of all in one go. The underlying goal here is to let us do better load
balancing server side. This earlier diff in this stack explains that in more
detail: D27188610 (820b538001).
Reviewed By: quark-zju
Differential Revision: D27191594
fbshipit-source-id: 19b1be9bddc6b32b1fabfe4ab5738dfcf71b1a85
Summary: Introduce `FetchKey` and `FetchValue` traits to simplify repeated trait bounds in many `ReadStore` implementations. We also newly require `Clone` for both keys and values, which was already required by the fallback combinator.
Reviewed By: DurhamG
Differential Revision: D27652499
fbshipit-source-id: 6a3d5eb18a904b982fdb9946b80fcc9025d391ea
Summary:
Extend debugscmstore command to fetch arbitrary files / trees by key.
Replace debugpyscmstore with a python fallback for debugscmstore (allowing you to test with the store as it is constructed for Python, with legacy fallback).
Refactor some functionality so it is shared between the rust and python versions of debugscmstore.
Currently the output is pretty ugly. It uses the `{:#?}` format for everything. In the next change, I propose modifying the `Debug` implementation for `minibytes::Bytes` to use ascii-escaped bytestrings rather than the default slice formatter to make things much nicer.
This new `debugscmstore` functionality should be useful in integration tests for testing scmstore under different repo configurations, and for test harnesses and performance testing (fetch a specific set of things easily, simulate delays in the key stream by delaying the input pipe, etc).
Reviewed By: andll
Differential Revision: D27351321
fbshipit-source-id: 8650480e3f5b045b279472643570309c48d7fe6b
Summary: Like `FileScmStoreBuilder`, but for trees. As LFS is not used for trees, `TreeScmStoreBuilder` defaults to `ExtStoredPolicy::Use` (pass along anything you find without LFS-specific checks).
Reviewed By: DurhamG
Differential Revision: D27641290
fbshipit-source-id: 637340a23cef058e7e37a41ae7f5b4fcc9481190
Summary: Introduce a new `FileScmStoreBuilder` structured much like `ContentStoreBuilder`, but supporting the features needed for the intermingling of `contentstore` and `filescmstore` construction (shared indexedlog, scmstore fallback to contentstore).
Reviewed By: DurhamG
Differential Revision: D27640702
fbshipit-source-id: e9771e6f61d80698a9dd761a0db66407b565c010
Summary: This test fails without other diffs in stack because previously native checkout was overwriting untracked files
Reviewed By: DurhamG
Differential Revision: D27667151
fbshipit-source-id: 9b3aea37ba5c2d07fe4fc975dd40b4d7bea9d810
Summary:
The `add_heads_and_flush` method might add new nodes in the master group,
and it should update `overlay_map_next_id` accordingly. Without it, it
might error out like:
RustError: ProgrammingError: Server returned x~n (x = 9ebc9ebc49f1819767b40f9ceb22c37547a10c37 8459584, n = 1411).
But x exceeds the head in the local master group {}. This is not expected and indicates some logic error on the server side.
Full error: P387088806
Reviewed By: sfilipco
Differential Revision: D27637278
fbshipit-source-id: b45370db0561dec52cd513a12e2fd0110f18e0e5
Summary:
The idset is not fully backed by Rust and do not batch resolve vertexes.
The nameset is backed by Rust NameSet and has proper batch prefetching.
Use nameset if possible but fallback to idset if the backend is not in
Rust (rare, only used by hgsql repos now).
Reviewed By: sfilipco
Differential Revision: D27630092
fbshipit-source-id: cf847dd1a78bd5273a8928ecb6616fe11f2c7026
