Summary:
This replicates behaviour of Python code - if unknown file content matches content of the file to be check out, do not abort checkout
This is useful for resuming interrupted checkouts / clones
Reviewed By: DurhamG
Differential Revision: D27799147
fbshipit-source-id: 7d2582583525da18fba08dfcd8bced2b619304de
Summary: It's been showing up while building mononoke. Let's fix it
Reviewed By: sfilipco
Differential Revision: D27789928
fbshipit-source-id: a15912f66b9ad3370545aed88405dbeb800e63de
Summary: This seems to have broken the EdenFS HgPrefetch test.
Reviewed By: xavierd
Differential Revision: D27795192
fbshipit-source-id: 80a748036961aa6a5750182bf65637fb76825341
Summary: This will show proper checkout progress when using native checkout
Reviewed By: quark-zju
Differential Revision: D27775423
fbshipit-source-id: 79f2afa02bd1fab7d5f747da1c714d4d1126ce7c
Summary:
EdenAPI makes heavy use of streaming HTTP responses consisting of a series of serialized CBOR values. In order to process the data in a streaming manner, we use the `CborStream` combinator, which attempts to deserialize the CBOR values as they are received.
`CborStream` hits a pathological case when it receives a very large CBOR value. Previously, it would always buffer the input stream into 1 MB chunks, and attempt to deserialize whenever a new chunk was received. In the case of downloading values that are >1GB in size, this meant potentially thousands of wasted deserialization attempts. In practice, this meant that EdenAPI would hang when receiving the content of large files.
To address this problem, this diff adds a simple heuristic: If a partial CBOR value exceeds the current buffer size, double the size threshold before attempting to deserialize again. This reduces the pathological case from `O(n^2)` to `O(log(n))` (with some caveats, described in the comment in the code).
Reviewed By: krallin
Differential Revision: D27759698
fbshipit-source-id: 67882c31ce95a934b96c61f1c72bd97cad942d2e
Summary:
Previously we'd skip dynamicconfigs when there wasn't a repo available.
Now that dynamicconfig can represent the NoRepo state, let's load dynamicconfigs
in that situation.
This also supports the case where there is no user name.
Reviewed By: kulshrax
Differential Revision: D26801059
fbshipit-source-id: 377cfffb6695a7fbe31303868a88862259ebf8c4
Summary: Add a new `edenapi.maxrequests` config option to allow controlling the number of parallel in-flight requests. This can be used to bound resource usage of the client when requesting large amounts of data.
Reviewed By: sfilipco
Differential Revision: D27724817
fbshipit-source-id: 8d607efa83d8b0b94074d1a6e06f6c536cc0c791
Summary: Add a method to allow setting `CURLMOPT_MAX_TOTAL_CONNECTIONS`, which limits the number of concurrent requests within a curl multi session. If the number of requests in the session exceeds this number, they will be queued and sent once earlier requests have completed.
Reviewed By: sfilipco
Differential Revision: D27724818
fbshipit-source-id: 436384aed9d6d29f426e5e45aebb7a72c24ba667
Summary:
Without this, `make local` will build `hostcaps` without fb-specific logic and
cause wrong configs being used. `hg up master` will error out like:
File "treemanifest/__init__.py", line 690, in _httpgetdesignatednodes
self.edenapi.trees(dpack, self.name, keys)
RustError: Server reported an error (403 Forbidden)
Reviewed By: quark-zju
Differential Revision: D27759821
fbshipit-source-id: d42895f44bc53003f2578b65640ebe4ee05d52e6
Summary:
Right now, if prefetch fails, we just give the client back an error saying
"content not found".
This isn't super helpful, because usually the reason the content is not found
is because we cannot talk to the server that has the content, so showing the
user why we cannot talk to said server is more useful.
I'd like to ship this gradually, so I also added a config flag to turn it off.
Initially I'll have the flag set, but I did default it to not-set in the code
so that our tests run with the desired configuration.
Note: I initially contemplated adding logging for this here, but after
discussion with xavierd it looks like just failing instead of eating the error
is probably a better approach (and it's much simpler). This is also consistent
with what EdenAPI does.
Reviewed By: mzr
Differential Revision: D27761572
fbshipit-source-id: 3506d9c97a00e3f076bd346883e07f49194b0b06
Summary:
Right now, if the server ever tells us a file is missing, we fail the entire
batch download. This is a bit unfortunate because other objects could still be
downloaded, but also because we lose the distinction between "server crashed"
and "server told us the data we want does not exist".
Besides, it's actually a bit unnecessary. Indeed, this fails, we just ignore
the error anyway up the stack, so it never actually gets accessed.
Reviewed By: mzr
Differential Revision: D27761574
fbshipit-source-id: cb4fb0526a3bf19c04ecb81c05d44d4d8afb81ad
Summary: We can just return the actual error here now.
Reviewed By: sfilipco
Differential Revision: D27761573
fbshipit-source-id: 0866f976b4ed434deffd96be6820ad05d27b7b93
Summary:
NOTE: The revisionstore LFS tests talk to prod Mononoke LFS, so the test here
will fail until that is deployed.
Like it says in the title, this adds support for downloading content in chunks
instead of all in one go. The underlying goal here is to let us do better load
balancing server side. This earlier diff in this stack explains that in more
detail: D27188610 (820b538001).
Reviewed By: quark-zju
Differential Revision: D27191594
fbshipit-source-id: 19b1be9bddc6b32b1fabfe4ab5738dfcf71b1a85
Summary: Introduce `FetchKey` and `FetchValue` traits to simplify repeated trait bounds in many `ReadStore` implementations. We also newly require `Clone` for both keys and values, which was already required by the fallback combinator.
Reviewed By: DurhamG
Differential Revision: D27652499
fbshipit-source-id: 6a3d5eb18a904b982fdb9946b80fcc9025d391ea
Summary:
Extend debugscmstore command to fetch arbitrary files / trees by key.
Replace debugpyscmstore with a python fallback for debugscmstore (allowing you to test with the store as it is constructed for Python, with legacy fallback).
Refactor some functionality so it is shared between the rust and python versions of debugscmstore.
Currently the output is pretty ugly. It uses the `{:#?}` format for everything. In the next change, I propose modifying the `Debug` implementation for `minibytes::Bytes` to use ascii-escaped bytestrings rather than the default slice formatter to make things much nicer.
This new `debugscmstore` functionality should be useful in integration tests for testing scmstore under different repo configurations, and for test harnesses and performance testing (fetch a specific set of things easily, simulate delays in the key stream by delaying the input pipe, etc).
Reviewed By: andll
Differential Revision: D27351321
fbshipit-source-id: 8650480e3f5b045b279472643570309c48d7fe6b
Summary: Like `FileScmStoreBuilder`, but for trees. As LFS is not used for trees, `TreeScmStoreBuilder` defaults to `ExtStoredPolicy::Use` (pass along anything you find without LFS-specific checks).
Reviewed By: DurhamG
Differential Revision: D27641290
fbshipit-source-id: 637340a23cef058e7e37a41ae7f5b4fcc9481190
Summary: Introduce a new `FileScmStoreBuilder` structured much like `ContentStoreBuilder`, but supporting the features needed for the intermingling of `contentstore` and `filescmstore` construction (shared indexedlog, scmstore fallback to contentstore).
Reviewed By: DurhamG
Differential Revision: D27640702
fbshipit-source-id: e9771e6f61d80698a9dd761a0db66407b565c010
Summary: This test fails without other diffs in stack because previously native checkout was overwriting untracked files
Reviewed By: DurhamG
Differential Revision: D27667151
fbshipit-source-id: 9b3aea37ba5c2d07fe4fc975dd40b4d7bea9d810
Summary:
The `add_heads_and_flush` method might add new nodes in the master group,
and it should update `overlay_map_next_id` accordingly. Without it, it
might error out like:
RustError: ProgrammingError: Server returned x~n (x = 9ebc9ebc49f1819767b40f9ceb22c37547a10c37 8459584, n = 1411).
But x exceeds the head in the local master group {}. This is not expected and indicates some logic error on the server side.
Full error: P387088806
Reviewed By: sfilipco
Differential Revision: D27637278
fbshipit-source-id: b45370db0561dec52cd513a12e2fd0110f18e0e5
Summary:
The idset is not fully backed by Rust and do not batch resolve vertexes.
The nameset is backed by Rust NameSet and has proper batch prefetching.
Use nameset if possible but fallback to idset if the backend is not in
Rust (rare, only used by hgsql repos now).
Reviewed By: sfilipco
Differential Revision: D27630092
fbshipit-source-id: cf847dd1a78bd5273a8928ecb6616fe11f2c7026
Summary:
While it is in theory "correct" going through the remote resolution even if the
protocol is "local". The overhead turns out to be something. And the tracing
message "resolve .. remotely" can be quite noisy. Let's just skip remote
resolutions early in IdConvert implementations.
Reviewed By: sfilipco
Differential Revision: D27630094
fbshipit-source-id: 7d87079876f040cf8f764f7362021dddba0d4723
Summary:
Currently the "contains vertex" check can trigger excessive
fetches for add_heads (and add_heads_and_flush used by flush).
Add a test to demonstrate the problem.
Reviewed By: sfilipco
Differential Revision: D27630091
fbshipit-source-id: ce3639c2a13226ba5681b4e8696edd7acbcb57f9
Summary:
Otherwise it can cause a lazy dag to think vertexes as "missing", insert
vertexes unnecessarily, and potentially break key graph properties (a
vertex should only have one Id).
Reviewed By: sfilipco
Differential Revision: D27629315
fbshipit-source-id: 1688d13cb94015bbc675613ecf9225556ff48372
Summary:
Also move related functions.
Similar to D27547584 (af3c3b3fd0), this allows `add_heads_and_flush` use `IdConvert`
on the `NameDag`, instead of the `IdMap` to trigger remote fetches properly.
This diff is easier to view with whitespace changes ignored.
Reviewed By: sfilipco
Differential Revision: D27629314
fbshipit-source-id: 8f79223c5d324aabfc5ab9813a9f65400fc533ec
Summary:
See the previous diff for context. Drop Locked and related APIs (prepare_filesystem_sync).
This makes it easier to use operate on a mut NameDag on flush because it does not need
to use separate types (Locked) for writing which has issues of not having the remote protocol
involved.
Reviewed By: sfilipco
Differential Revision: D27629306
fbshipit-source-id: 301445b242321ad5f424741ea91ebf6c075bff1c
Summary:
See the previous diff for context. Drop SyncableIdMap so we are one step
closer to using mut NameDag directly on add_heads, which knows when and how to
do remote fetching properly.
Reviewed By: sfilipco
Differential Revision: D27629310
fbshipit-source-id: 883606e40bb83907dfa6142ddd2c3030de61698f
Summary:
By using SyncableIdDag and SyncableIdMap, it's harder to use extra features
around them (ex. remote fetching). Drop SyncableIdDag so we are one step
closer to using mut NameDag directly on add_heads, which knows when and how to
do remote fetching properly.
Reviewed By: sfilipco
Differential Revision: D27629307
fbshipit-source-id: 8e9a5a4348a42b9751752b82feb3f3d2d7c4ba45
Summary:
The `Parents` trait is used for input of adding (non-lazy) vertexes to the
graph. The API will be used to provide extra hints to optimize network
fetches.
With the current logic, `assign_head` will ask the server to resolve the heads
first, to check if it is already assigned, then to resolve the parents, etc. to
the roots (in the "to assign" set). Ideally the `assign_head` logic can ask
the server to resolve the roots first, and if that's unassigned, then just mark
all descendants of the roots as unassigned, do not send more requests.
Note: the current pull logic has all the hashes ready (hashes are known).
But whether the hashes have Id assigned are unknown. It is more tricky
taking the "lock" and "reload" into consideration - hashes without Ids might
"become" having ids assigned after we obtain the lock to write data to disk.
Practically, `pull` using the current logic would take a very long time because
it tries to resolve things remotely for every "to assign" commits.
Reviewed By: sfilipco
Differential Revision: D27629317
fbshipit-source-id: e02f54f43ef65228ce6e3a8a8723dd9ae0a07008
Summary: This just simplifies the test code a bit.
Reviewed By: sfilipco
Differential Revision: D27629308
fbshipit-source-id: 04eac5cd045c71123e7fc410af74609dbadb8fb7
Summary: This avoids triggering remote lookups if an unknown name was looked up multiple times.
Reviewed By: sfilipco
Differential Revision: D27629316
fbshipit-source-id: 64c1ce5e872a5ce4f14c650a946ae8396f4cc74c
Summary:
When translating RequestNameToLocation to ResponseIdNamePair. If "heads" are
known, but some "names" aren't. Do not treat it as an error. This will be
used by the client-side to properly handle the "contains" check.
Reviewed By: sfilipco
Differential Revision: D27629309
fbshipit-source-id: 206ec5df956b33f4e816ab8d6a67ce776fd7bd74
Summary: This will make it easier to test client / server dags in upcoming changes.
Reviewed By: sfilipco
Differential Revision: D27629318
fbshipit-source-id: e3137654613aa3208a8f2e4b9f4ddfe73871f2c5
Summary: This will be used in upcoming changes. It just delegates to the Arc inner.
Reviewed By: sfilipco
Differential Revision: D27629313
fbshipit-source-id: ba6cd7cac2b9f5c1a2898c439c53768995a2dc42
Summary: This will be used by upcoming changes.
Reviewed By: sfilipco
Differential Revision: D27629312
fbshipit-source-id: 6c56e73caf4e1a398ac3a8b4294bd9f380af3764
Summary: This will be used by upcoming changes.
Reviewed By: sfilipco
Differential Revision: D27629319
fbshipit-source-id: d19e490268561f3154642e5bb1e415da4c5d03ee
Summary:
See the previous diff for context. A concrete HashMap can provide
"hint_pending_subdag". But a parent function cannot.
Reviewed By: sfilipco
Differential Revision: D27629311
fbshipit-source-id: 65168a8d00d9a672396312200016d6749f416d02
Summary: The lazy backend can now (partially) support the non-full IdMap segment clone.
Reviewed By: sfilipco
Differential Revision: D27581488
fbshipit-source-id: 51eded6acdbe82d22f5bb73eb4a715e2c22f4d75
Summary:
Make mutationstore more friendly to async.
This resolves an issue with smartlog with the lazy commit hash backend.
Reviewed By: sfilipco
Differential Revision: D27583844
fbshipit-source-id: 5b0b0b9b8ab82399f80eb2b410a0c4b84bd6a444
Summary:
Otherwise it might panic (ex. calling into tokio without entering a tokio
runtime). This can happen in, for example, `Debug::fmt(&IdStaticSet, ...)`.
Reviewed By: sfilipco
Differential Revision: D27581487
fbshipit-source-id: feec53e088706adcc6710afcf24fa70598f886cf
Summary:
This will be used by "add_heads" logic to detect what vertexes to insert
and might reduce remote fetches.
Reviewed By: sfilipco
Differential Revision: D27572359
fbshipit-source-id: d0bf027a69d180663a1587dfde613cb76b05072a
Summary: The API returns entries buffered in memory not persisted.
Reviewed By: sfilipco
Differential Revision: D27572360
fbshipit-source-id: 555988f7c891f2d928bfa1e486a0fc1d089b4ad5
Summary: This will be used to select "dirty" (not written to disk) set in the IdDag.
Reviewed By: sfilipco
Differential Revision: D27572361
fbshipit-source-id: 0b4d2e092ece835e3d4b6aa831d32ffffc7087ca
Summary:
Before this change, overlap IdMap was not considered for prefix lookup. That
results in "shortest" template not working and smartlog prints full hashes
for remote/stable etc.
Reviewed By: sfilipco
Differential Revision: D27547582
fbshipit-source-id: 7a56590775eed9d509f2212f8e5009c04aaf4e9d
Summary: It will be reused in NameDag.
Reviewed By: sfilipco
Differential Revision: D27547583
fbshipit-source-id: da85fc7504d20877210e8ed1a97cbec18d06eede
Summary:
Now NameSet iteration can be blocking, SyncNameSetQuery is no longer sound.
Remove SyncNameSetQuery in key logic (namedag and ops) and replace them with
async logic.
Reviewed By: sfilipco
Differential Revision: D27547581
fbshipit-source-id: af69b1a8219e97d10278939407ee79f9b333a77f
Summary: Dag algorithms like `parent_names` need to fetch vertexes via remote automatically.
Reviewed By: sfilipco
Differential Revision: D27547584
fbshipit-source-id: 8106931d6f642c9a4bf0f3c546ba881c2ca73fea
Summary: This crate does not panic on Windows.
Reviewed By: DurhamG
Differential Revision: D27640362
fbshipit-source-id: f50f6b8e0bd31e5f80fa939bcfb6846bc8fd4a63
Summary:
Recently we saw some progress rendering issues. Add a command to attempt to
reproduce them.
Reviewed By: DurhamG
Differential Revision: D27669184
fbshipit-source-id: 62fcf82d8261fd27e91ba5a116c61f4df1919007
Summary: This will be used later.
Reviewed By: skotchvail
Differential Revision: D27744058
fbshipit-source-id: 411ab66ccc38b306c6bffb190e936ba1e455f07a
Summary: This will make RotateLog achieve zero-copy reading more easily.
Reviewed By: kulshrax
Differential Revision: D27724331
fbshipit-source-id: 57915516dc6bd1935838bd099a60c104f0bdef3d
Summary: This makes it more flexible.
Reviewed By: kulshrax
Differential Revision: D27724332
fbshipit-source-id: 43ad670519f0617a97e0b7d38b374f497e9c01af
Summary:
This allows setting the wrapping mode. For example:
lhg log -pv
# copy paste long lines works.
lhg log -pv --config pager.wrapping-mode=unwrapped
# lines are not wrapped, ">" is shown for long lines.
lhg log -pv --config pager.wrapping-mode=word
# long lines wrapped at word level.
The default value matches "less" behavior.
Reviewed By: DurhamG
Differential Revision: D27720767
fbshipit-source-id: e29d6b13656407c0a1e63287fb96e2f8d914cfc8
Summary: This method checks if any of files that checkout writes is not tracked in hg and exists on disk
Reviewed By: DurhamG
Differential Revision: D27667153
fbshipit-source-id: 4ad8bc08520678ea0b51008ed14fb51ca4a98f76
Summary:
`hostcaps` abstracts the logic for determining whether we have a prod or corp
environment.
Reviewed By: DurhamG
Differential Revision: D27684641
fbshipit-source-id: 50df9a60b6a613b4cb5c9aed6cad2844aae85a6f
Summary:
We want to use it in Mercurial and the directory structure was playing bad
with Mononoke's OSS build.
Reviewed By: xavierd
Differential Revision: D27684642
fbshipit-source-id: 8827645eee58fa671f9c9e1964a34c34e3a8eeb6
Summary:
This diff introduces CheckoutPlan::apply_read_store to apply checkout plan using ReadStore as a data source
This requires some minor changes in apply_stream flow as ReadStore does not guarantee ordering of returned files
Reviewed By: DurhamG
Differential Revision: D27658346
fbshipit-source-id: 5a289554d8dd7b6bb4b5a996659cd0661779ad5f
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641665
fbshipit-source-id: d46f62f9067eb9cb4c8517a62efa6f663d4b6732
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641669
fbshipit-source-id: d907407f5a6e868862fe37f1f67fbe99ee378156
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641667
fbshipit-source-id: adce5a39fcb5d8e8d5d989fed46991e20ab3710d
Summary: Provides a way to read config with lighter dependencies.
Reviewed By: DurhamG
Differential Revision: D27641668
fbshipit-source-id: fc99a78f5f51e63f61d1b049af74f61f5d1916a3
Summary:
The `configparser` is now too heavyweight. Some other crates (ex. io, auth,
revisionstore) just want to extract config values without complicated parsing /
remote hotfix logic.
Add a configmodel crate to satisfy the need.
Reviewed By: DurhamG
Differential Revision: D27641666
fbshipit-source-id: 26bd0b606ae3d286b3ec218927aef726d6802c63
Summary: Add a new `http.verbose` config option that turns on verbose output for libcurl (similar to the output printed by `curl -v`). This can be very useful for debugging HTTP issues.
Reviewed By: DurhamG
Differential Revision: D27693304
fbshipit-source-id: 2ad7a08889f40ffbcd2f14ac9c21d70433629da4
Summary:
`const_cstr::ConstCStr` is represented internally as a fat pointer with fixed size: `&'static str`. See https://docs.rs/const-cstr/0.3.0/const_cstr/struct.ConstCStr.html. Notably this is **different** from the representation of `std::ffi::CStr`, which is a dynamically sized type and normally passed around behind a reference, as `&CStr`. Using `&ConstCStr` in signatures, which is effectively like `&'a &'static CStr`, is confusing due to the discrepancy between the two relatedly named types. Additionally having two different lifetimes involved -- the static lifetime of the underlying bytes, and the short lifetime of the fat pointer -- is unnecessarily confusing when async code and a language boundary are involved.
The utf8-cstr crate uses what seems like a better representation to me than the const-cstr crate. See https://docs.rs/utf8-cstr/0.1.6/utf8_cstr/struct.Utf8CStr.html. `Utf8CStr` is the dynamically sized type, just like `CStr`. Then `&'static Utf8CStr` is how it would commonly be passed around, just like `&CStr`.
Reviewed By: krallin
Differential Revision: D27698169
fbshipit-source-id: ffe172c2c2fc77aeab6b0a0a8aed3e3c196098cc
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the client code to thread the list of master commits to the
EdenApi client.
Reviewed By: quark-zju
Differential Revision: D27523868
fbshipit-source-id: db4148a3f1d0e8b0b162e0ecc934e07f041c5511
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the request object for HashToLocation to send over all the
master heads.
When the server builds non-master commits, we will want to send over non-master
heads too. We may consider having one big list of heads but I think that we
would still want to distinguish the non-master commit case in order to optimize
that use-case.
Reviewed By: quark-zju
Differential Revision: D27521778
fbshipit-source-id: cc83119b47ee90f902c186528186ad57bf023804
Summary:
`panic!()`, and things which use `panic!()` like `assert!()`, take a literal format
string, and whatever parameters to format. There's no need to use `format!()`
with it, and it is incorrect to pass a non-literal string.
Mostly it's harmless, but there are some genuinely confusing asserts which
trigger this warning.
Reviewed By: dtolnay
Differential Revision: D27672891
fbshipit-source-id: 73929cc77c4b3d354bd315d8d4b91ed093d2946b
Summary:
Modify the `Debug` implementation for `minibytes::Bytes` to use `std::ascii::escape_default` to debug print a `Bytes` as an ascii-escaped bytestring.
For comparison, the `bytes` crate `Bytes` type provides the same functionality, though it doesn't use the standard library `escape_default` function: https://docs.rs/bytes/1.0.1/src/bytes/fmt/debug.rs.html#39-43
This change greatly improves the output of the `debugscmstore` command. If we don't want to make this the default behavior, we can provide a formatting wrapper type or I can specialize the output in `debugscmstore`, but I can't see any real downsides, especially given the `bytes` crate does the same thing, and we have a similar specialization for `HgId` (hex format in that case).
Reviewed By: quark-zju
Differential Revision: D27642721
fbshipit-source-id: 8faba421fa5082a2098b13ef7d286e05eccb6400
Summary: Add the `with_key` function to `Entry`, which replaces it's key with a provided key. Currently, scmstore returns incorrect results when multiple entries exist with different paths but the same HgId (as scmstore directly returns the path found on disk locally). This isn't a problem in the legacy API, which returns a bare `Vec<u8>` content, which is implicitly associated with the requesting key because it is the result of a single `get` call, or is irrelevant because the `prefetch` method doesn't directly return the results.
Reviewed By: andll
Differential Revision: D27664025
fbshipit-source-id: 014d44ca9a1dc2721685622fd2b077ed3483838f
Summary:
Those are not used. Recently we saw build issues like:
lib/third-party/sha1dc/sha1.c:8:10: fatal error: string.h: No such file or directory
#include <string.h>
^~~~~~~~~~
Possibly by some compiler flags disabling stdlib. Since we don't need
the C code let's just remove them.
Reviewed By: singhsrb
Differential Revision: D27659634
fbshipit-source-id: b8fcac4f05be95cac470f9b4830baf76e06f98ad
Summary:
The "filter" set's filter function might not be prepared for inputs outside
the parent set. So let's the "contains" function to test against the parent
set first, then test the filter function.
This fixes the "merge()" set's "contains" check using the revlog backend:
In [1]: v=repo.revs('draft() & merge()')
In [2]: v._set
Out[2]: <meta ?>
In [3]: m.node.nullid in v._set
Out[3]: False
Before this diff it would be:
In [3]: m.node.nullid in v._set
CommitLookupError: '0000000000000000000000000000000000000000 cannot be found'
Note: Segmented changelog backend is not affected because it does not use
filter sets.
Reviewed By: xavierd
Differential Revision: D27657502
fbshipit-source-id: 30bb261fea59bdc5644580e98796f52fa93c2705
Summary: We were accidentally not setting the TLS key here; this diff fixes it.
Reviewed By: quark-zju
Differential Revision: D27634276
fbshipit-source-id: 9aac3a34b6f6655059a8d3332eea8ba02d062651
Summary:
This diff contains proposal for basic rebase model for new checkout.
This diff introduces two types of file changes - basic no conflict Actions(that will replace CheckoutPlan actions) and ConflictAction for actions that contain conflicts
No conflict actions are produced when comparing two manifests
Conflict actions can be produced when rebasing manifests.
Eventually we will adjust checkout code to take either list of Actions or Conflict actions and perform corresponding operations on the file.
Rebase::rebase is used to generate list of conflict actions when rebasing.
Currently `Rebase::rebase` produces list of actions on top of 'base' commit.
Before doing actual checkout we will need to take into account what current commit is and adjust this list of actions accordingly (however, this adjustment will be trivial, it won't produce more conflicts)
After getting adjusted list of actions, checkout code can roll it out to fs and call merge drivers if needed.
In the future, if we will want to allow check in conflicts, we will need to adjust `Rebase::resolve_conflict` to take `ConflictAction` as input, instead of an Action.
Reviewed By: DurhamG
Differential Revision: D27028001
fbshipit-source-id: b02b7ad8030fcf146b53639d69e66fdc8f38a62d
Summary:
Only used by one test that can define the constaint itself.
The problem with having it on the trait is that it's a bit noisy when
things operate on ToApi at the generic level. It adds to the list of
constaints they these users of the ToApi trait need to add.
Reviewed By: kulshrax
Differential Revision: D27549922
fbshipit-source-id: fff9e513eb4c06862111ce6eecc84ab981eea893
Summary:
This is only used in one utility which can define the constaint itself.
I am looking to simplify the Requirements for ToWire so that we can more
easily iterate on them. Debug as a requirement produces too much noise.
There is the risk of ending up in a scenario where we want to print the Wire
type but it's more practical to annotate structs with derive Debug when that
happens than to add the constaint in the trait.
Reviewed By: kulshrax
Differential Revision: D27549925
fbshipit-source-id: aacf7c1c465c94414be02aa143187897c7084980
Summary:
There is no use for it outside of one test which can describe that constraint
itself.
I think that requiring ToWire and ToApi to use the same objects is too much
for the general use case. We regularly convert between different object types
that are the same logical thing but have different representations. A good
example for that is the HgId. It makes sense to implement ToWire for all HgId
variations.
Reviewed By: kulshrax
Differential Revision: D27549924
fbshipit-source-id: d76d7a4beb528634bed46ae93dbd634d850547e5
Summary:
For async requests, we perform a blocking request in a separate thread, and stream the results back through a channel. However, if the curl handle for the request is dropped before starting the request (for example, because of a configuration error), this function would return a `oneshot::Canceled` error (from the channel consumer) instead of the real error message from the IO thread.
This diff fixes the issue by ensuring that the function waits for and returns the error message from the IO thread in the event that the IO thread returns before starting the request.
Reviewed By: quark-zju
Differential Revision: D27584502
fbshipit-source-id: 8447c158d253c3f28f03fcc4c36a28698fe6e83d
Summary:
Since we're rolling out native checkout and resumable checkout around
the same time, let's make resumable checkout optional so we can turn it off it
causes issues, without turning off native checkout.
Reviewed By: quark-zju
Differential Revision: D27481986
fbshipit-source-id: a0a3e68567ca2a468e852dce95c03c4b606aaf22
Summary: This makes it easier to use.
Reviewed By: kulshrax
Differential Revision: D27406589
fbshipit-source-id: 11bef407ab620859381c6ee952e6ef00494551e1
Summary:
The issue is that `mut i: usize` is no longer shared across multiple `async
move` blocks (introduced by D27308798 (0df4efa969)).
Rewrite the logic to collect `ids` first, then use `vertex_name_batch`
query instead.
Reviewed By: sfilipco
Differential Revision: D27406586
fbshipit-source-id: b41fe3a13114dc34aa5763e6e2bebe0571decc87
Summary:
Merge paths like `x~n` and `x~(n+1)` to `x~n (batch_size = 2)`.
This could be more efficient bandwidth-wise and algorithm-wise.
Reviewed By: sfilipco
Differential Revision: D27406587
fbshipit-source-id: f2a67352ad627945685e33667e8299a2bc652930
Summary: IdSet is more compact. This changes the order a bit.
Reviewed By: sfilipco
Differential Revision: D27339279
fbshipit-source-id: e9b50a47beba081b892eccd7711dbd6ab5c3a886
Summary: This will be used by the next change.
Reviewed By: sfilipco
Differential Revision: D27406591
fbshipit-source-id: fcacc35a9ae8ed96cebb2af804d26d1e5e83ad9e
Summary:
Add a way to flush the overlay map to disk so we can avoid network fetches over
and over.
Reviewed By: sfilipco
Differential Revision: D27406592
fbshipit-source-id: 7086ad665119cc3a0834f533690325c7a2363442
Summary: It will be reused elsewhere.
Reviewed By: sfilipco
Differential Revision: D27406593
fbshipit-source-id: 296cf5f50830bb7285e0cb9c7c15a9b374689819
Summary:
I spent some time thinking about how to flush the "overlay_map" to disk.
It is a bit tricky because the on-disk IdMap might have changed in an
incompatible way. I tried a few ways to verify the on-disk IdMap remains
compatible and didn't find a way that looks good (either slow - calculating
universal_ids, or is not 100% correct in corner cases).
Now I come up with this "just track x~n" idea. It trades memory usage (~2x
overlay_map) for easy-to-verify correctness, and efficient overlay_map
flush.
Reviewed By: sfilipco
Differential Revision: D27406583
fbshipit-source-id: 0b7fb3186a9c15f376c1dc4afe7f0516c25d3dec
Summary: It is not obvious. So let's add more comments.
Reviewed By: sfilipco
Differential Revision: D27406584
fbshipit-source-id: 9ce1215efc1a6d4849180c6693616613c08f2a51
Summary:
A sparse dag does not have full IdMap. Its IdMap only contains "universally known" entries.
Add a basic test about cloning from a sparse clone data and resolve vertex <-> id mapping
on the fly.
Reviewed By: sfilipco
Differential Revision: D27352018
fbshipit-source-id: 4a3f5f50be52e91bf7b2021cdc858bcab9c99e80
Summary:
The `NameDag::flush` API will actually rebuild the graph using a "parent" function.
That is not necessary if we got clone data, and won't work well for a lazy graph
(since the parent function talks about vertex names and some names are missing).
Let's bypass the `flush` function and write data directly in `import_clone_data`.
Reviewed By: sfilipco
Differential Revision: D27352019
fbshipit-source-id: a79569d25d858447b8c5eb86902b8d39ae0429a3
Summary: This will be used in tests.
Reviewed By: sfilipco
Differential Revision: D27343882
fbshipit-source-id: 5a2d94a9f755eed0fc27e5a11093b55c810dc8da
Summary:
Implement logic to export the clone data. This allows us to construct a sparse/lazy
dag via export + import CloneData.
Reviewed By: sfilipco
Differential Revision: D27343885
fbshipit-source-id: 71dc0d31e36876a8b6a8c3d7f3498be3262ce297
Summary:
Clone data can only be imported to an empty Dag and universally known vertexes
should be present in the IdMap.
Reviewed By: sfilipco
Differential Revision: D27343888
fbshipit-source-id: ba150d6afdbe15f0902ec20ff150a70657e24c80
Summary: Collect "missing" items and only use one request to fetch them.
Reviewed By: sfilipco
Differential Revision: D27406588
fbshipit-source-id: a5cd091b39d90c1ad0e7c5d509673c4665232304
Summary:
This will be used by upcoming changes. Sparse/Lazy NameSet will override it
to reduce network round-trips.
Reviewed By: sfilipco
Differential Revision: D27406590
fbshipit-source-id: a44a73b4aec6e14d6e82d55285fe1cfc0fcfd482
Summary: This will be used by upcoming changes.
Reviewed By: sfilipco
Differential Revision: D27343884
fbshipit-source-id: 0938b1fb3d90b35f9d51c468cffca53e3f421bb8
Summary: The `mut` was unused. Remove it to make the interface more flexible.
Reviewed By: sfilipco
Differential Revision: D27406594
fbshipit-source-id: 1cfa4921015fc89b6c71ed4a97d9c351f56c7370
Summary:
Remove some TODOs. This serves as fallback paths where batch prefetch didn't happen.
I'd expect most use-cases will trigger IdStatic set's batch prefetch logic (to be
added).
I haven't decided what to do exactly with "contains". Fetching remotely there seems
to require some kind of negative cache (ex. in mutation records there might be nodes
not in the graph). But it _might_ be okay to say the "contains" is a local-only
operation, too. I leave it as-is and we'll see how the Python world uses "contains"
later.
Reviewed By: sfilipco
Differential Revision: D27339275
fbshipit-source-id: ba70b3c84a391a8e395c73ccd1d7e08f92b0cbd0
Summary:
Put everything together. I used "programming error" extensively to
provide more context if we have to investigate issues in the future.
Reviewed By: sfilipco
Differential Revision: D27339278
fbshipit-source-id: 574a2c048dc1d24dbe690f862fec3e5078cb067a
Summary: Provide more context about what invariants we expect. Not just show "vertex not found".
Reviewed By: sfilipco
Differential Revision: D27339273
fbshipit-source-id: 1c6c92537ff37666ff603783adfd8f9ea770fbaa
Summary:
Makes NameDag own the state (logic) about how to send remote requests.
So NameDag can send requests on its own.
Reviewed By: sfilipco
Differential Revision: D27339282
fbshipit-source-id: 3cb6327dfeaefae45d4e7b88a3535463a84b195b
Summary: This will make the overlay IdMap effective when query against the NameDag.
Reviewed By: sfilipco
Differential Revision: D27339276
fbshipit-source-id: 80712bf651beb6c7e9f23bd4233c6d916101696a
Summary:
We already checked size and mtime, which should allow us to identify if
a file has changed since the last update, but we need to check hgid as well, to
make sure the previously-written content is supposed to be the same as the
about-to-be-written content
Reviewed By: andll
Differential Revision: D26955401
fbshipit-source-id: 859ad35b008e68d699601217f2a4398c6789913e
Summary:
Updates native checkout to store which files have already been written
in .hg/upgradeprogress. Then enables it to load that file and skip writing those
files if they're already on disk and their mtime and size match the previously
written values. In theory we could record and check file hashes as well, but
that'd likely slow things down quite a bit.
Future diffs will add:
- Recording and checking the hgid that was written before vs what is about to be
written. Just an hgid comparison, not a full hash computation.
- Some UX to inform the user when hg checkout can be continued, and possibly to
implement 'hg checkout --continue'.
Reviewed By: andll
Differential Revision: D26830249
fbshipit-source-id: 88a75080966dae5241550ed7eedbc057c65966dd
Summary: Vendor updated zstd crates. Started on this as a prerequisite for vendoring updated async-compression crate, however dependency resolution means this actually updates async-compression as well.
Reviewed By: farnz
Differential Revision: D27467760
fbshipit-source-id: 74ca9e4da9f336cf609cf06c62559c9ef913c9a2
Summary:
Add a `set_global_config` function to `hg-http` that gets called prior to command execution. This function can be used to configure the global behavior of all HTTP requests by setting up a callback that modifies every `Request` prior to sending.
Right now, the use case for this is to support the `--insecure` flag, as can be seen in the code.
Reviewed By: quark-zju
Differential Revision: D27243638
fbshipit-source-id: 61a52c1f4b56fffd860c2a7e9c7b03e146b2240a
Summary: Add the `--insecure` flag to the list of global hg flags. This flag is already supported by Python, and adding it here will make it available to Rust code as well. This is used later in the stack.
Reviewed By: quark-zju
Differential Revision: D27243029
fbshipit-source-id: 150d42ee96e1e3194ff1be6a33d9b36887d86f2c
Summary:
Add bindings for `pysptui`. This allows using `streampager` as a TUI, using
streampager's controlled file mode to manage the display.
Reviewed By: quark-zju
Differential Revision: D27268252
fbshipit-source-id: d191a09c44ca4ed013647feb81e6f031d553b2f2
Summary: In a later diff, I plan to make this `pub` in order to parse HTTP versions from the user's config.
Reviewed By: quark-zju
Differential Revision: D27449576
fbshipit-source-id: 28a60080393eff73399c65b9e808647b39603719
Summary: Expand the IdConvert implementation so we can change it.
Reviewed By: sfilipco
Differential Revision: D27339280
fbshipit-source-id: eb55c63529c895502a25bb279bcba3c13737452a
Summary:
Add an overlay IdMap field for NameDag to store temporary remote IdMap results.
This diff just adds the field. It's not used yet.
Reviewed By: sfilipco
Differential Revision: D27339274
fbshipit-source-id: dbbde227f26de15d10c84f5d7c61ca8054577752
Summary:
In a future diff, AbstractNameDag wants an "overlay" IdMap to store temporary
remote IdMap results. The MemIdMap is suitable but has extra features (next
free id, version). Extract the core in-memory IdMap logic for the "overlay"
purpose.
Reviewed By: sfilipco
Differential Revision: D27339277
fbshipit-source-id: 4e73032b8bc6670264e3fa1dd5515ea3bc853d10
Summary:
The `Process::process` contains logic to resolve Id <-> Vertex using remote
service. The remote service is async so let's make `Process` async.
Reviewed By: sfilipco
Differential Revision: D27308798
fbshipit-source-id: 30c2c3eda124d542d0867d278ce56a7a174f33e0
Summary:
The config turns on fsync for all indexedlog writes. It can be useful in places
where we want reliability more than performance with unknown kernel issues.
Reviewed By: sfilipco
Differential Revision: D27347595
fbshipit-source-id: c0b31928684e8805a9e6441062f96b05ad311ea2
Summary:
Add a global flag, if turned on, ensure all atomic files, and indexes and
primary logs use fsync.
Also enhance fsync so it syncs the directory too.
Reviewed By: sfilipco
Differential Revision: D27347596
fbshipit-source-id: 831e27e494cc343a33ca675619c030ead8023210
Summary: Use Edenapi book request and response type in bookmarks edenapi endpoint. Serialize the response as cbor.
Reviewed By: kulshrax
Differential Revision: D27174122
fbshipit-source-id: 6bc7295c25bd355db4625da3c1f8c68349e7b0b7
Summary: Add edenapi types for the bookmarks endpoint. Now the endpoint can handle a request for a batch of bookmarks instead of a single bookmark. The request type still needs to be modified at some point to allow for bookmark prefix listing.
Reviewed By: kulshrax
Differential Revision: D27133284
fbshipit-source-id: c3960629cad76504e222f726a151eb3390850276
Summary:
With the mix of external pager and progress suspension, the progress might
be enabled by accident:
# pager: disable forever
disable_progress(True)
# suspension
with progress.suspend()
...
# on __exit__, re-enables pager
Update the pager disabling logic be nested to avoid the potential issue.
Reviewed By: andll
Differential Revision: D27275016
fbshipit-source-id: 35ca7aef1890a981e6a1f0e5313b6a482ed43368
Summary:
Refactor `scmstore::types` into separate `file` and `tree` modules and introduce a new `StoreTree` type to represent trees in the scmstore API.
Introduce a minimal `StoreTree` type. Importantly, for now this type does not provide any methods for deserializing the tree manifest and inspecting it's contents. This functionality won't be too hard to implement, though - it'll require some revisions to the `manifest-tree` crate and / or moving the `StoreTree` and `StoreFile` types to `revisionstore_types`.
Reviewed By: kulshrax
Differential Revision: D27310878
fbshipit-source-id: 712330fba87f33c49587fa895efea3601ce377af
Summary:
Rename "newstore" to "scmstore"
The three main options I'm considering are "edenstore", "scmstore", and "hgstore". They all describe the project sufficiently well, in my opinion. "edenstore" might be a stretch, as I have no reals plans for Mononoke to use it, while "hgstore" might be a bit limited, as integration with EdenFS is a core goal, and it'll be used by EdenFS to fetch remote data via EdenApi, not just Mercurial's local storage. I feel like "scmstore" rolls off the tongue the easiest, particularly vs. "hgstore" (both "H.G. Store" and "Mercurial Store"), and is a bit easier to type than "edenstore" (which has "ede" all with the same finger). I don't have a strong opinion on this matter, so If you'd like a different name I'm open to suggestions.
Speak now or forever hold your peace.
Reviewed By: andll
Differential Revision: D27180332
fbshipit-source-id: 19e6972ea0f6527e671792845dcfd339cf1ab767
Summary:
Introduce a new `StoreFile` type for the `revisionstore` crate. This is an enum of `File`, `LfsPointer`, and `RedactedFile`, which represent the different cases the `Entry` type's `content` might actually represent. The `File` variant also handles stripping the copy header, if present, and stores both the `copied_from`, raw unstripped `content`, and stripped `content`, though only the latter can be accessed through the public API right now. Ideally, we'll move copy information into the history API and never actually need to make it public here.
Conversions are provided from `Entry` and EdenApi's `FileEntry` type (which also supports redaction and LFS pointers, but as errors instead of first-class values).
Modify the output type used in the Python bindings to use this new `StoreFile` type.
Currently, the `LfsPointer` variant is unused. This will be used later when we add first-class LFS support.
Reviewed By: kulshrax
Differential Revision: D26862710
fbshipit-source-id: 8326921f3ee43bf2e253847d5735c61f5a50bfa6
Summary: Adds `ExtStoredPolicy` support to the EdenApi and IndexedLog ReadStore implementations. This flag controls how LfsPointers found during a file query are handled (either returned as a result, or treated as "not found").
Reviewed By: kulshrax
Differential Revision: D27171814
fbshipit-source-id: 14dda47f32184c3ee703fbc77106885ca4d3ea27
Summary:
Make the `RequestCreationEventListeners::new_request` event listener take an `&mut Request` instead of an `&mut RequestContext` as a parameter.
In the existing code (particularly in the `hg-http` crate), this event listener is used to configure the `RequestContext` for reporting progress. This diff just generalizes this idea, allowing the listener to modify the entire `Request`.
This is useful when we need to hijack request creation in `hg-http` to do Mercurial-specific configuration. The specific use case here is to disable TLS certificate checking when the global `--insecure` flag is set.
(Note that `http-client` itself is application-agnostic, so Mercurial specific configuration should not happen in this crate. This is why `hg-http` exists at all.)
Reviewed By: quark-zju
Differential Revision: D27242947
fbshipit-source-id: 019e19037642fe24acaa8c2917d446b91e7bcb26
Summary: Add `verify_tls_cert` and `verify_tls_host` settings to `http-client::Request` that are equivalent to [`CURLOPT_SSL_VERIFYPEER`](https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYPEER.html) and [`CURLOPT_SSL_VERIFYHOST`](https://curl.se/libcurl/c/CURLOPT_SSL_VERIFYHOST.html). These will be used to allow skipping cert validation with the `--insecure` flag.
Reviewed By: quark-zju, sfilipco
Differential Revision: D27242946
fbshipit-source-id: cfa0fe800d0d132ca10ec0203bfd20b53c68b814
Summary:
The `Request` builder's methods currently take `self` by value, which was intended to make it easy to create new `Request`s in a fluent style without assignment. Unfortunately, it turns out that this complicates situations where we need to modify a `Request` in-place.
The ideal solution would be to change `Request`'s builder methods to take `&mut self` instead [1]. I tried doing this, but unfortunately there are too many other parts of the current design that rely on the existing behavior in ways that are difficult to change (particularly around the creation of streaming and async requests).
As a workaround, this diff simply adds matching `fn set_X(&mut self, ...) -> &mut Self` methods for each builder method on `Request`. The existing consuming methods have been rewritten in terms of those methods to prevent duplication of the actual method contents.
Given that all of the consuming builders now contain essentially the same body, it seems like we could reduce the verbosity here by using a macro. Unfortunately, I'm not sufficiently experienced with writing nontrivial macros to come up with something quickly, but I do think that it would be a good idea to eventually use a macro here.
---
[1]: In fact, it turns out that [this is considered a better practice](https://github.com/rust-lang/api-guidelines/discussions/81) when designing builders in Rust -- as long as the terminal method of the builder chain returns the built struct by value, it is still possible to create a new instance without assignment. This does mean, however, that we cannot easily move things from the builder to the final struct without using tricks like `Option::take` and `mem::swap`, since the signature of the terminal method would be `(&mut self) -> Self`).
Reviewed By: andll
Differential Revision: D27256048
fbshipit-source-id: 14f770a87abc839d358e5ba211a096226d3e0dc6
Summary:
This is a follow-up to D27093942 (59b8287c85). But D27093942 (59b8287c85) forgot one `read_log` spot, making it
possible to lose data.
Reviewed By: andll
Differential Revision: D27265233
fbshipit-source-id: ab1e73cdcfd3cc9da2e19000ee5fd5761977dc4a
Summary:
Turns out D27093942 (59b8287c85) was not a complete fix. It misses a `read_log`. Add a
test showing the problem.
Reviewed By: andll
Differential Revision: D27265232
fbshipit-source-id: 103e543abacbb2cbaf10aa3fe3d9693922d9daad
Summary:
There are a few abstractions in this that make it a little hard to
have different behavior between the download & upload paths.
This makes it hard to have different behavior for the two. It's a bit of a
problem right now in the sense that we end up doing things like sending a
Content-Length on a GET, but the reason I'm changing it is because I want to
chunk downloads and that requires having different logic in the upload &
download paths.
As part of doing this, I also moved a bunch of parameters away from
HttpLfsRemote and into HttpOptions, which makes the function signatures a
little more manageable.
Reviewed By: quark-zju
Differential Revision: D27191593
fbshipit-source-id: c332229bb3c5a4c1eaedb54dc12c3ddc19205050
Summary:
eden/scm/lib: default cpython_ext to python3
```
CI internally, try a PR on github once its landed.
Reviewed By: quark-zju
Differential Revision: D27237636
fbshipit-source-id: 1768f778d75b5d6c62dfd5641f911604a37d3163
Summary: Fix malformed module-level doc comment and rearrange import groups (based on the de facto Rust style conventions at FB).
Reviewed By: quark-zju
Differential Revision: D27243739
fbshipit-source-id: 21614d16e5fbfecd793939325ef52819db9a4cb8
Summary:
Remove use of `#[async_trait]` in the new storage API, and eliminate the outer Future on the `fetch_stream` and `write_stream` methods, which was not used. The methods are now normal trait methods, which accept and return a stream.
It's possible we'll want to make these methods `async` again in the future, but until this this is more ergonomic, faster (one less layer of indirection), and avoids some of the type system limitations of `#[async_trait]`.
Reviewed By: andll
Differential Revision: D26695517
fbshipit-source-id: 2db60a3f37d594b0b9d1e1a2708532ef0ddaf585
Summary:
Introduce `FilterMapStore` combinator to support cases like LFS, where values of the same type may or may not be supported by a given store implementation.
Use `FilterMapStore` in Python bindings to prevent LFS files from being accidentally written to the non-LFS indexedlog.
With this change, the new storage API should be safe to use without corrupting local storage.
Reviewed By: kulshrax
Differential Revision: D26651254
fbshipit-source-id: 629cb43d85f43117a32b577777e13ff8fb801d57
Summary:
Introduce `HashMapStore`, a HashMap-based, generic, in-memory `ReadStore` / `WriteStore` implementation for testing purposes (and perhaps other uses in the future).
Introduce a minimal `KeyedValue` trait so that `HashMapStore` can determine the key to be associated with a written value in a generic way.
Reviewed By: kulshrax
Differential Revision: D26859235
fbshipit-source-id: 66032f072148796bde7a3a176fb2bb6707b95287
Summary:
Introduce `WriteError` type (similar to `FetchError` for reads) to support strongly-typed write errors. Eventually we'll probably want this type to carry the value that we tried to write (to allow write fallbacks, retries, etc) and perhaps add more enum variants (rejects, unavailable, etc).
Introduce `ReadWriteStore` trait for stores that support both reading and writing. This trait is automatically implemented for types which implement `ReadStore` and `WriteStore` with the same key and value types. This trait is intended to be used via the `BoxedRWStore` trait object. This type will be used in the Python bindings when `FallbackCache` is extended to implement `WriteStore` (in a future change).
Reviewed By: kulshrax
Differential Revision: D26646393
fbshipit-source-id: 96058cf1e826fdb6076a4162389829b3fe053686
Summary:
Introduce newfilestore class in pyrevisionstore, which constructs a newfilestore (BoxedReadStore for files) and a corresponding `ContentStore` which share the underlying IndexedLog object and EdenApi client.
Modify remotefilelog to construct ContentStore via this new class.
Currently, no methods are provided for the newfilestore - it is meant to be passed back into Rust code, where you may call it's Rust methods as shown in the `test_newstore` method (which will be removed in the future).
Currently the `util` module is made public for access from pyrevisionstore. In the future, this will be replaced in favor of a `NewFileStoreBuilder` which handles these concerns internally.
Reviewed By: DurhamG
Differential Revision: D26526331
fbshipit-source-id: c0f439fbee4c303db4a82171c866a3f3a5fc2324
Summary:
Split `FallbackStore` (which supports writing fallback fetches) into `FallbackCache` (which behaves the same) and `Fallback` (which doesn't support writing fallback fetches). This eliminates the `From<>` bound on `Fallback` which is nonsensical in some cases where we don't want writing.
Replace `write_store` and `write` fields with a single `Option<>` `write_store` in `FallbackCache`, which cleans up the code a bit.
Separate the "fallback value type", "preferred value type", "write value type", and "output value type" separately control the output value type independent of the preferred store.
Reviewed By: kulshrax
Differential Revision: D26526149
fbshipit-source-id: aa9f25f5efb8a9ddab03a7d86a7a007f24d156ce
Summary:
Add an `edenapi.logdir` config option, which, when set, will cause the EdenAPI client to write a JSON version of every request it sends to the specified directory. This is intended for use when reproducing user issues (so that the developer can inspect and replay the requests that were sent).
The JSON request files can be directly given to the `edenapi_cli` or converted to CBOR using the `make_req` tool for manual testing with `curl`.
The approach right now is a bit simplistic, in that we just write a JSON file whenever the EdenAPI client makes a request. I'm open to suggestions on how to improve this. (For example, if there were a way for us to always have a record of the last N requests, that would be helpful for debugging user issues.)
Reviewed By: DurhamG
Differential Revision: D27145093
fbshipit-source-id: 3834c2052b0c5efa05d1d209962953b29f545a3f
Summary: noticed these in passing while working on previous diff
Reviewed By: mitrandir77
Differential Revision: D27227682
fbshipit-source-id: e7858c81951b780722b0836ecf6ee72aeb1ffa09
Summary:
When testing "disable_progress" with chg (next diff) I found it was not
effective because there are 2 separate IO structs. The one we disable
from Python is different from the one the Rust progress thread uses.
I traced it down here. Since the Python IOs are just wrappers of
Rust IOs in the chg use-case. There is no need to recreate an IO
struct.
The "creating IO" struct is still useful, for things like "-t.py" testing where
the output needs to be captured into different Python variables per different
commands.
Reviewed By: DurhamG
Differential Revision: D27149243
fbshipit-source-id: 6e27adcc9f48b21fc24fba120be8c4a8fef1f909
Summary:
In some cases (ex. using an external pager). The IO states are changed outside
the IO struct's control. Ideally we should implement the external pager logic
on IO too but for now let's just add an API so the Python external pager logic
can disable progress output after starting an external pager.
Reviewed By: kulshrax
Differential Revision: D27149242
fbshipit-source-id: ff51fc153d3cc211cfa8ef697923d36f7c0f0d9b
Summary:
Detecting prod on Windows wasn't working because we used a posix path.
Let's add the Windows equivalent.
Also moves us to use the new hostcaps crate.
Reviewed By: chadaustin
Differential Revision: D27126497
fbshipit-source-id: 4035012fb7701378fb6e2e902c0efcd54ef42ea9
Summary:
I'd like to add support for event handlers in the Rust ServiceRouter client (I
need this in order to inject CATs in calls made by the SMC client). To do so, I
need to be able to instantiate a `ContextStack`, and to do so, I need a static
c-string representing the service name & function name, which is what this diff
does.
Note that we actually do the same thing for Rust servers already.
#forcetdhashing
Reviewed By: farnz
Differential Revision: D27088187
fbshipit-source-id: be2ad541d5123b31f0dab73da16b35dbfd308d6f
Summary:
This creates .hg directories when there is no repository at all, which
breaks jf submit in git repos. D26801059 is the real fix, but it has some other
complications. Let's drop the creation here, since it really isn't necessary.
Reviewed By: quark-zju
Differential Revision: D27134087
fbshipit-source-id: d15048b2d1022d38393b62cc02ebf022e617ed4f
Summary:
`CommitHashToLocationRequestBatch` and `CommitRevlogDataRequest` implemented `FromJson` but not `ToJson`. The latter is required so that the EdenAPI client can log JSON representations of all request types.
In the process of adding tests for those two types, I discovered a bug in `CommitLocationToHashRequestBatch`'s `ToJson` implementation which prevented it from passing JSON roundtrip tests (which I added because this type didn't have test coverage here). The bug is now fixed and the roundtrip test passes.
Reviewed By: krallin
Differential Revision: D27105414
fbshipit-source-id: d2e23e21f996ac8062bca948ed9abe3779e4669d
Summary: Use a macro to define quickcheck tests that perform roundtrip conversions for EdenAPI types that implement `ToJson` and `FromJson`. This makes it easier to add additional roundtrip tests for new types.
Reviewed By: krallin
Differential Revision: D27105416
fbshipit-source-id: c6059f9527d57b88cd6e3d1647fa89d8f826b1e3
Summary:
Rework the progress. Always move the cursor to the top-left corner of the
progress output. So the clear progress instruction is just to erase till
the end of the screen. There is no need to track the height of the progress.
Update flush logic so we only do flush when crossing the progress/non-progress
boundary: When we write progress (to stderr), flush stdout, write progress,
then flush stderr.
Also, disable progress unconditionally if the current output line is incomplete.
Reviewed By: sfilipco
Differential Revision: D27109228
fbshipit-source-id: 717345e9c7eaeebeb378ce090f7b2f60957fd150
Summary: This makes write_err() consistent with error().write().
Reviewed By: sfilipco
Differential Revision: D27109227
fbshipit-source-id: 4e7ea1fe464c17548c34e5f9bf78085868a20256
Summary: This line just collects a vector into a vector. Probably a remnant of a refactor. Delete it.
Reviewed By: quark-zju
Differential Revision: D27091643
fbshipit-source-id: fb611aabea375b8495476401b2d9cdf7ba12fde1
Summary: This makes the new software respects writes by older software.
Differential Revision: D27093942
fbshipit-source-id: 097b57c61b5ee1f0264babb88737306113fe356a
Summary:
The error was triggered but it's unclear what's wrong. Make the error
more detailed.
Reviewed By: xavierd
Differential Revision: D27058212
fbshipit-source-id: 3f6220e2d100d9118c05a8b4c75c5ba19c9181db
Summary:
By implementing DefaultOpenOptions, indexedlog provides `repair()` for free.
Re-export the `Repair` trait so other crates can use `repair()` without
importing indexedlog.
Reviewed By: sfilipco
Differential Revision: D27053352
fbshipit-source-id: 8fa952f0e51e007b9d348bc12699ef1d65000c6b
Summary:
With the new log for MultiMeta. It's now possible to repair a MultiLog by:
- Repair all Logs
- Scanning through the MultiMeta Log and find a valid MultiMeta.
- Set the current MultiMeta to the picked MultiMeta.
Reviewed By: sfilipco
Differential Revision: D27053346
fbshipit-source-id: d60596fb00323b3bcadd5ade2e34cad29a37d64a
Summary:
We recently saw a few reports about "multimeta" being 0-sized. MultiLog cannot
be repaired like other logs because the logs (ex. IdDag and IdMap) have to be
in sync. To implement Repair for MultiLog, let's track MultiMeta in a Log so
we can check its previous entries and fix the multimeta.
Reviewed By: sfilipco
Differential Revision: D27053347
fbshipit-source-id: af99b13d658ee62bfe63973ab9d37338d14a7d4a
Summary: This makes it easier to spot problematic segments.
Differential Revision: D27025039
fbshipit-source-id: 44ae9eaf03ca54751b64c0cf977b78737eaa58f0
Summary:
We saw the `unwrap()` panic at `iddag.rs:453`:
let (new_idx, low, high, parent_count, has_root) = candidate.unwrap();
Reading the code, it seems it can only happen if one of `parents` is `high` in a segment.
Parents in a segment should all be < low. Add some validations for that.
Reviewed By: sfilipco
Differential Revision: D27021394
fbshipit-source-id: 2c51713dfb8f07f2921229d8866f7e8a0bd9490f
Summary:
For example, `hg log > a.txt`, writing to `a.txt` does not need to clear the
progress bars.
Reviewed By: andll
Differential Revision: D26886281
fbshipit-source-id: 392c50e62a854e03ea0c400a1dac8bfb900815a2
Summary:
The Rust progress bars now integrates with hg-http and all requests get
progress bars for free. There is no need for a separate bar.
Reviewed By: andll
Differential Revision: D26970747
fbshipit-source-id: c5e59c745e75a93e5da0541baab21c3f23dc1322
Summary:
Render the progress bars in a Rust thread.
Also make it compatible with traditional Python progress bars (not the most efficient way because the Python still have its own progress thread, but the code change is minimal).
Reviewed By: andll
Differential Revision: D26886282
fbshipit-source-id: ad1f862ced278cc1cb90c37f576eb05f52a50b13
Summary: 100 and 5k seems too small - requests are too frequent.
Reviewed By: andll
Differential Revision: D26886276
fbshipit-source-id: cb5cce1111120f8a437fbc8c60906940588f38bc
Summary:
Split `dispatch()` into 2 parts: config parsing and command execution.
This allows callsite to do extra work based on the config before running the
command.
Reviewed By: andll
Differential Revision: D26853749
fbshipit-source-id: 459918b43341f7e14df6958056b6e07453bb691b
Summary: This method should be preferred to use from python binding. It is similar to block_on but allow to interrupt future on Ctrl+C
Reviewed By: quark-zju
Differential Revision: D26894586
fbshipit-source-id: 02d8fbb2f9c4d9e585f8fbee51bfc400c402d0d2
Summary:
During sparse profile change, there might be a case when file has changed between checkouts, and also is added due to sparse profile change
Currently this would lead to creating duplicated update action that would waste resources. This diff deduplicates such action
Reviewed By: quark-zju
Differential Revision: D26953657
fbshipit-source-id: ccd9084cc3984ac0692ee6178fce018a94cd56e7
Summary:
Current store implementation returns value that may contain matadata.
We need to strip it before writing out file to disk
This is not going to be a problem with new Eden API, but it is not erady yet and rolling it out may take time
So in order to start testing native checkout now, we are incorporating metadata removal in the checkout code
Reviewed By: quark-zju
Differential Revision: D26953658
fbshipit-source-id: 05d8be55bb96e7c3162594498ef0b9944b422c90
Summary:
Add a way to register an IO time series showing total network I/O, and register
download progress bars for requests.
Reviewed By: andll
Differential Revision: D26853750
fbshipit-source-id: 42c4ab9724c60db9612a2bc419ca3bceaa667647
Summary:
It turns out there are multiple users sending requests bypassing the
HttpClient, like the LFS in revisionstore, or the segmented changelog
clone.
Requests bypassing HttpClient means HttpClient event callbacks do not
have a chance to insert progress and bandwidth monitoring. So let's
add another callback that can capture what HttpClient misses. This would allow
us to get proper progress bars of revisionstore LFS and segmented clone without
changing their code.
Reviewed By: andll
Differential Revision: D26970748
fbshipit-source-id: 5133bc6f9eeb14a6d2944d253bc66cefd49c83c5
Summary: Add a simple way to render progress into a multi-line string.
Reviewed By: andll
Differential Revision: D26853751
fbshipit-source-id: 4f1de55e7bb03f607d683eff16e035aa5d1476c1
Summary:
Add a struct to "register" progress data for Rust to render them.
Note: The "registry" only tracks the data. Rendering will be
implemented separately.
Reviewed By: andll
Differential Revision: D26615606
fbshipit-source-id: affa663660bd52a2583d40e3b28f0b65480851a3
Summary: Add an API to render the time series samples into ASCII graph.
Reviewed By: andll
Differential Revision: D26836633
fbshipit-source-id: 9c7a757e46c28c810fb3e0671a775fde16318f54
Summary: This could be used for showing overall statistics
Reviewed By: andll
Differential Revision: D26886280
fbshipit-source-id: a52f057dbbebb441d0eee14860275f2690c7a3a3
Summary: Add an API to render the time series samples into an array that can be further turned into an ASCII graph.
Reviewed By: andll
Differential Revision: D26615170
fbshipit-source-id: 4a2f178869c235e7dda3edef7930a6e04bc4bda3
Summary: Continue with last change. Add a time series type that keeps samples.
Reviewed By: andll
Differential Revision: D26615171
fbshipit-source-id: 5dcc775820d782ac502d84b6af5e270f2a9072ee
Summary:
Going to add some kind of "time series" graph based on periodical sampling.
Add the sample structure for IO with some simple math helpers.
Reviewed By: kulshrax
Differential Revision: D26615173
fbshipit-source-id: 4d610f435a3c7913e8bb31a8de0738ffb3de1468
Summary:
Going to add some kind of data model definitions for pure Rust logic.
Use a dedicated crate so progress model and rendering are separate. If the
rendering logic becomes bloated somehow, it won't affect the data models.
The idea is that only application-level logic like `hgcommands` care about
rendering. Other use-cases only care about pure progress states.
Reviewed By: andll
Differential Revision: D26836634
fbshipit-source-id: 5222d2f9fb4894bd246c62ad48e1c5fc982e2fa6
Summary:
Previously we referred to the config location as the repo_path or
repo_dir. In an upcoming diff we'll support NoRepo configuration, so let's
rename this to config dir and support using cache_dir as a config dir if there
is no repo. That code path isn't actually used in this diff though.
Reviewed By: kulshrax
Differential Revision: D26801061
fbshipit-source-id: 72cd672f65ca10821882ee9124fd4a293ead7257
Summary:
Previously we assumed there was always a repository. In an upcoming
diff we'll support configuring Mercurial without any repository. To support that
let's add a representation of NoRepo to the dynamicconfig structure and update
all uses to use that.
Nothing will currently create a NoRepo, but that will come in a later diff.
Reviewed By: kulshrax
Differential Revision: D26801060
fbshipit-source-id: d8e9066c865921f45e068c799ac8acb8eec32acd
Summary:
Pure Rust commands weren't loading dynamicconfigs correctly because the
validation was happening in Python. This diff moves it down into Rust so that
all config loading gets the same validation.
Also removes the hard coded list of files to validate. I'll add this to
configerator to make it easier for us to edit.
Also removes the configs.validatedynamicconfig option. Validation is on
everywhere and should just be on by default.
Reviewed By: kulshrax
Differential Revision: D26680075
fbshipit-source-id: f206198154801f655b4b277cdb22a728d0004b5f
Summary: These were generally useful when debugging native checkout.
Reviewed By: quark-zju
Differential Revision: D26955400
fbshipit-source-id: 967a4fa2d831b8b060409a5f7be31a1416c1ba63
Summary:
Similar to `output()`, and `error()`, provide a `progress()` to support setting
progress without needing `&IO`.
Reviewed By: andll
Differential Revision: D26886278
fbshipit-source-id: a54563a9cf1d9d1cdb9dabe945aeb5ed1d84e8fb
Summary: Make sure the progress is not hidden because of buffering.
Reviewed By: singhsrb
Differential Revision: D26886277
fbshipit-source-id: d44e48e67b4529a181dd36c30e92608654d8fea6
Summary:
Add a check to ensure the `into_vec()` fast path do not take the full vec if
the `Bytes` is sliced.
Reviewed By: andll
Differential Revision: D26966453
fbshipit-source-id: 538dfb8ca2f01a46d1ede7b98b7f0a30fc7a786e
Summary:
Current `block_on_future` implementation is not safe, as it does not panic on nested calls. Nesting `block_on_future` calls causes starvation of tokio scheduler processes and eventually will lead to deadlock if nested too many times
This diff simply calls `block_on_exclusive` from `block_on_future`. When we see that this does not cause problems we can simply remove `block_on_future`
Reviewed By: sfilipco
Differential Revision: D26899522
fbshipit-source-id: 93794bf2c5908421691dfb094d1807266c9ecd8d
Summary:
Johan Schuijt-Li found they are broken on some CI hosts.
(But other store paths seem okay, which might indicate the fsync list is effective)
Reviewed By: DurhamG
Differential Revision: D26950101
fbshipit-source-id: e3bcd3f77636325be9e9ce8dfded8b17ec68f436
Summary: This API will be used to spawn tasks in the runtime.
Reviewed By: andll
Differential Revision: D26615607
fbshipit-source-id: 6b9892fad755bbe8feb775e9dad457697b2ea1b7
Summary: We weren't passing a repo path when initially loading the repo's config in `clidispatch`. This meant that the resulting `ConfigSet` didn't contain values from the shared-repo `.hgrc.dynamic`. Evidently, some other code path in Python would eventually add these values, but this meant that pure-Rust commands could not see config values set via dynamicconfig. Passing the path fixes the problem.
Reviewed By: DurhamG
Differential Revision: D26508980
fbshipit-source-id: 65f187d18098a08c81325e78cb02a8ed854c739a
Summary: The goal is to reduce load on tokio scheduler by using threads & channels instead of spawning new task every time
Reviewed By: DurhamG
Differential Revision: D26801249
fbshipit-source-id: a8d9accc721c7ffc981fd538c06ab8cbd908f715
Summary:
AsyncVfs provides async vfs interface.
It will be used in the native checkout instead of current use case that spawns blocking tokio tasks for VFS action
Reviewed By: quark-zju
Differential Revision: D26801250
fbshipit-source-id: bb26c4fc8acac82f4b55bb3f2f3964a6d0b64014
Summary: This function is now internal details of vfs, and is not used outside of it. Making it private so it won't be accessed outside
Reviewed By: quark-zju
Differential Revision: D26801251
fbshipit-source-id: 03434e235978fa0745128d90ea0c5975ea662ff1
Summary: This diff removes the split between manually managed and autocargo managed Cargo.toml files in `eden/scm/lib`, now all files are autogenerated.
Reviewed By: quark-zju
Differential Revision: D26830884
fbshipit-source-id: 3a5d8409a61347c7650cc7d8192fa426c03733dc
Summary:
On POSIX systems it's a good practice to fsync directories to ensure metadata
about a file is properly written.
Reviewed By: DurhamG
Differential Revision: D26822211
fbshipit-source-id: fca10c702b480f22020ad184eb55c8879999f0ee
Summary:
Memcache is dogslow to initialize, taking >30s on debug build. As a
consequence, this slows down every single test by that amount time, with the
guarantee that no blobs will be found in memcache, ie: a total waste of time.
On release builds, Memcache is significantly faster to initialize, so let's
only disable initializing Memcache for debug builds only.
Reviewed By: fanzeyi
Differential Revision: D26800265
fbshipit-source-id: 8b393c603414de68268fdadb385de177e214a328
Summary:
## Why this diff
we want hostname prefix to support targeting configs at clients in corp ("corp" means laptop, labs, and other machines that are not in "prod" datacenters), like FRL machines, that don't support our existing tier mechanism.
## Changes
* Extract hostname prefix in `dynamicconfig.rs` and add a getter function `hostname_prefix()` for it.
*A hostname prefix only consists of alphabetical letters and dashes, which is followed by one or more digits in the hostname. If no valid match, the prefix is set to the empty string.*
* Use `gen.hostname_prefix()` in the `evaluate()` fn inside `mod.rs` to check the generator's prefix against a list of given prefixes.
* Copy changes from `configerator/source/scm/hg/hgclientconf/hgclient.thrift` to `fbsource/fbcode/configerator/structs/scm/hg/hgclientconf/hgclient.thrift`.
* Rebuild in `eden/scm/`.
Reviewed By: DurhamG
Differential Revision: D26706686
fbshipit-source-id: 725506a1c1f0983e981b0b3f3993c7c14510b1db
Summary: This simplifies the code a bit.
Reviewed By: kulshrax
Differential Revision: D26681779
fbshipit-source-id: 393565790ab711dd09ae6cfa6f9c4b19c930eb93
Summary:
Similar to D26670318, use the EventListeners APIs to implement the progress
callback. Now HandlerExt only has RequestContext related "as_ref" logic.
Reviewed By: kulshrax
Differential Revision: D26681778
fbshipit-source-id: b7f6e07ced43e0ae043e859337c06b69bd5dfc95
Summary: This makes it useful in non-mut callbacks.
Reviewed By: kulshrax
Differential Revision: D26681784
fbshipit-source-id: 97312df8bf3f900a36cbeb27206a2946bb6c702c
Summary: This makes ProgressUpdater Send + Sync so it can be used in the new callback APIs once `mut` gets dropped.
Reviewed By: kulshrax
Differential Revision: D26681781
fbshipit-source-id: 9c622b1d78b4091e3359c28972b6624f0b53734d
Summary:
This removes more mutable fields. Note the new code is more correct because
curl can call the `progress` callback periodically even if no progress is made.
According to https://curl.se/libcurl/c/CURLOPT_PROGRESSFUNCTION.html:
This function gets called by libcurl instead of its internal equivalent with
a frequent interval. While data is being transferred it will be called very
frequently, and during slow periods like when nothing is being transferred it
can slow down to about one call per second.
Reviewed By: kulshrax
Differential Revision: D26681780
fbshipit-source-id: 19aa4bcb4c56623e3f0408b06041b3a894f197e7
Summary:
This makes the `total_progress` field use lock-free interior mutability. The
goal is to eventually drop Rc and RefCell.
Reviewed By: kulshrax
Differential Revision: D26681782
fbshipit-source-id: ec0a6abbb2115c17c674db2255d196aaec847705
Summary: This removes the need for `mut` for this field.
Reviewed By: kulshrax
Differential Revision: D26681783
fbshipit-source-id: 10ed9adfb62081b0e6839abd9534db92d4e056c5
Summary:
The ProgressInner only exposes APIs for total (aggregated) progress. There is
no API to read individual progress. Make it only track the total progress. This
would make it easier to implement interior, lock-free mutability on ProgressInner.
The updater now needs `mut` temporarily, which will be dropped later (D26681784).
A test case is tweaked so progress does not go backwards.
Reviewed By: kulshrax
Differential Revision: D26681777
fbshipit-source-id: 4ad1b9173d5a2c2326e00c030d51f77e9b9458f3
Summary: Test both the HttpClient and Request events.
Reviewed By: kulshrax
Differential Revision: D26670325
fbshipit-source-id: ffbc4268f7de698830411434a769c8b1a4acd863
Summary:
This simplifies the code a bit and makes it look consistent with other event
listeners.
Reviewed By: kulshrax
Differential Revision: D26670318
fbshipit-source-id: f6eda9403bb6eb09a074544e672a45c84f38e2b1
Summary:
Add `RequestContext.event_listeners()` to register callbacks, and trigger the callbacks
when related events happen.
Reviewed By: kulshrax
Differential Revision: D26670323
fbshipit-source-id: 9b92b715444e83957c06b06f1ce696d4de3c0023
Summary:
This simplifies the logic a bit. There is no need for
`HandlerExt::with_payload` that is similar to `Request::body`.
Reviewed By: kulshrax
Differential Revision: D26670326
fbshipit-source-id: 9fe755821062ad6f2a74d6d5ba345620669f0f63
Summary:
This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here.
Arc lint support:
V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff:
- Adds running `autocargo` as part of `arc lint-rust`
Mergedriver update:
- Mergedriver used in resolving conflicts on commits is now pointing to V2
- It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble
Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for:
- Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section)
- "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string
- "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo
- "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added)
- Files generated from thrift_library rules that do not support "rust" language are removed
- Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored
Others:
- Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo`
- Updated following files to use V2:
- `common/rust/tools/reindeer/version-bump`
- `remote_execution/rust/setup.sh`
- Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing
Reviewed By: zertosh
Differential Revision: D26728789
fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
Summary:
Add a way to define event listeners. Define events on HttpClient and
Request(Info). They will be used by upcoming changes.
The idea is similar to GUI programming. A "control" has a list of "events" that
handlers can be defined on them.
This diff defines lists of events on the "client" and "request" types.
Areas this diff tries to improve:
- Make it easier to add new events (by using macro_rules).
- Make it easier to visually see all possible events.
Reviewed By: kulshrax
Differential Revision: D26670324
fbshipit-source-id: 92f74779f8e546491d0e922db27a4b87f527a5e9
Summary:
This makes it easier to access states defined in `RequestContext` from
`curl::Easy2<H>` types.
Reviewed By: kulshrax
Differential Revision: D26670317
fbshipit-source-id: 27eca9dcc11b14b5d41c8327448f7748ebc62e10
Summary:
Upcoming diffs extend the trait with new methods unrelated to configuration.
Rename to clarify.
Reviewed By: kulshrax
Differential Revision: D26670314
fbshipit-source-id: 7d33ebe22b26f1a286ae40c78f51f31a1a64957e
Summary:
Similar to the Streaming curl handler, add a RequestContext field to the Buffered
handler so the curl callbacks on the Handler can provide the Request
information like urls.
Reviewed By: kulshrax
Differential Revision: D26670321
fbshipit-source-id: de7abecf162c4aaed03d927c35516b6f8ac523ce
Summary:
Make `RequestContext` available in the streaming request. The `clone` will
be removed by a later change.
`dead_code` is temporarily allowed so the following won't be an error.
error: field is never read: `request_info`
--> src/handler/streaming.rs:24:5
|
24 | request_info: RequestContext,
| ^^^^^^^^^^^^^^^^^^^^^^^^^
|
Reviewed By: kulshrax
Differential Revision: D26670319
fbshipit-source-id: 53a1deeece5a2059e7caa9d28ef00e083a27b722
Summary:
Add an "auto incremental" Id to uniquely identify requests. This allows external
logic to have a map from the Id to extra metadata not owned by this crate. Without
the Id, there is no way to tell if `RequestContext`s with a same url and method are
actually a same request or not.
Reviewed By: kulshrax
Differential Revision: D26670327
fbshipit-source-id: 60fa760432b23ab5334f22806e01304f9c160182
Summary:
The `RequestContext` is a subset of `Request` that are:
- Independent from curl types.
- Carry useful states, and make them available on Handler callbacks.
- For now, the "useful states" include url and method. They can be extended
later.
Reviewed By: kulshrax
Differential Revision: D26670320
fbshipit-source-id: 3d71d5fee8927dd57a52f51b212397710379e7fa
Summary: This treats update_meta collection same way as update_content
Reviewed By: DurhamG
Differential Revision: D26694077
fbshipit-source-id: f2a7afb988eb99afa6e8f62ae6cda57a108a987d
Summary: This diff adds unit test for sparse checkout plan
Reviewed By: DurhamG
Differential Revision: D26673701
fbshipit-source-id: 8f810df317c3447f8445765e3a510559d4f1b378
Summary:
This method updates checkout plan to account for sparse profile change
The logic is somewhat similar to sparse.py, except it is smaller because it does not account for merges
Reviewed By: DurhamG
Differential Revision: D26670413
fbshipit-source-id: 54294ad7f1c61e39400450ff139f40ac8fffdd62
Summary:
This matcher combinator is useful when checking out across sparse profile change, since it allows to list files that has been affected by the sparse profile chane
Python has similar combinator xormatcher
Reviewed By: DurhamG
Differential Revision: D26670412
fbshipit-source-id: 0f84835aad16177e79e87d15b41b2caef4374605
Summary:
There are some reports about truncated latest files on NFS hosts (where the
hgcache portion might be outside NFS).
Fsync files in hgcache too. With indexedlog the total number of files is
bounded so listing them is considered bounded time.
Reviewed By: kulshrax
Differential Revision: D26741889
fbshipit-source-id: a789396cc5802110eb55fa6b52b5124cd9a3dbf3
Summary: It seems there is no "tty" check. So let's disable colors for tests explicitly.
Reviewed By: DurhamG
Differential Revision: D26708620
fbshipit-source-id: 802537af0d3f580c39a746b6aa6a617c9fb4c9c1
Summary: A few new config options were added to `EdenApiBuilder` without corresponding setting methods, meaning that they could only be set via an hgrc config file. This diff adds the methods so that Rust code can manually configure these settings.
Differential Revision: D26709471
fbshipit-source-id: 6af645961ed50526ccc04990b82ee201211b6ad2
Summary: Add `Content-Length` to `ResponseMeta` struct. Will be used for logging purposes.
Differential Revision: D26709386
fbshipit-source-id: 4346627e3883d00350738cca1e71004f8d407c0d
Summary:
Currently running native checkout shows much higher sys time then regular checkout
One theory that this is due to high scheduler overhead due to large number small tasks spawned with spawn_blocking
This diff batches 100 operations before spawning blocking proc
Reviewed By: quark-zju
Differential Revision: D26620395
fbshipit-source-id: f5be84c3ee14ffc55e4332e714a48f55701d5f8f
Summary:
Recently there are reports about "remote/master" pointing to a non-existed
commit, missing tree data, and general hard-reboot issues. Attempt to reduce
the chance of losing data by adding the critical paths to the fsync list.
Reviewed By: kulshrax
Differential Revision: D26708225
fbshipit-source-id: ae52534fdbf8d9d35498d30c24bdce7f5d9a96a3
