Summary: Previously, it was not possible to interrupt `hg` during EdenAPI fetch operations. This made it impossible to interrupt long-running fetches, which is very frustating to users. This can be simply fixed by using `block_unless_interrupted` in place of `block_on`.
Reviewed By: quark-zju
Differential Revision: D29344670
fbshipit-source-id: 3b0d36dda28f5f7cc812a07981f295f8d0fbdd8a
Summary:
This is simple command mostly to be used by testing before we fully integrate with hg pull
This command does not perform discovery and requires from/to revision to be passed in cmd line
Reviewed By: quark-zju
Differential Revision: D29315647
fbshipit-source-id: 26d67031e566b7c99af1e2a5ab287f02b52f7db0
Summary: This is breaking the Windows release, reverting.
Reviewed By: fanzeyi
Differential Revision: D29339787
fbshipit-source-id: 22d8ff5db5619194e4597754dc37343cf0bc3286
Summary:
Introduce basic contentstore fallback tracking to help monitor the scmstore shim rollout.
This will be expanded to a general fetch metrics system for scmstore in a future change.
Reviewed By: kulshrax
Differential Revision: D29305839
fbshipit-source-id: c6cc3ea15a3bb7b90f4ec298febc911ec4e2af91
Summary:
Prevent `FileStore` from deadlocking when a write falls back to contentstore and attempts to write to the same indexedlog_local which is held lock for the batch.
Note: this shouldn't need to block release, we current expect writing raw LFS pointers to only happen with non-remotefilelog LFS.
Reviewed By: kulshrax
Differential Revision: D29299050
fbshipit-source-id: bf39f87b9956165a558f3a19960d3d055685db9a
Summary:
Introduce `LegacyStore` trait, which contains ContentStore methods not covered by other datastore traits.
Implement this trait for both contentstore and scmstore, and modify rust code which consumes `contentstore` directly to use `PyObject` and `LegacyStore` to abstract over both contentstore and scmstore instead.
Reviewed By: DurhamG
Differential Revision: D29043162
fbshipit-source-id: 26e10b23efc423265d47a8a13b25f223dbaef25c
Summary: Previously, we just fetched "best effort", and logged any encountered errors using `tracing`, leaving it up to the client to inspect errors if necessary. Python relies on catching these fetch errors as exceptions, though, so this change introduces some utility methods to help propagate them correctly.
Reviewed By: DurhamG
Differential Revision: D29211683
fbshipit-source-id: 5e9dee942c2b60e0f77a051624d7f393a811fc4e
Summary: My previous fix was actually incorrect, we now log actual remote requests, but join that with the logs from the contentstore fallback.
Reviewed By: DurhamG
Differential Revision: D29206878
fbshipit-source-id: d22e58792bf380c274e8086ce08aebe20dd9b848
Summary:
Previously, when fetching data using several concurrent requests, the EdenAPI client would wait for the headers for every request to finish coming in before starting to deserialize and yield entries from the bodies of any of the requests.
Normally, this isn't a huge deal since the response headers on all of the requests are usually roughly the same size, so they all finish downloading at roughly the same time when the requests are run concurrently. However, this does become an issue when `edenapi.maxrequests` is set. This option makes EdenAPI configure libcurl to queue outgoing connections once the configured limit is hit.
This means that although from EdenAPI's perspective all of the requests are running concurrently, they are not actually running in parallel. The result is that the EdenAPI client ends up waiting for all of the queued requests to be sent before yielding any data to the caller, which forces it to buffer all of the received data, resulting in massive memory consumption.
This diff fixes the problem by rearranging the structure of the Futures/Streams involved such that the client immediately begins yielding entries when they are received from any of the underlying transfers.
Reviewed By: quark-zju
Differential Revision: D29204196
fbshipit-source-id: b6b56bb7d60457de3c4046a07a5965749e9dd371
Summary:
When the `send_async` method is used to dispatch multiple concurrent requests, the method needs to return an `AsyncResponse` for each request. Since `AsyncResponse`'s constructor is itself `async` (it waits for all of the headers to be received), internally the method ends up with a collection of `AsyncResponse` futures.
Previously, in an attempt to simplify the API, the method would insert all of these futures into a `FuturesUnordered`, thereby conceptually returning a `Stream` of `AsyncResponses`. Unfortunately, this API ends up making it harder to consume the resulting `AsyncResponses` concurrently, as one might want to do when streaming lots of data over several concurrent requests.
This diff changes the API to just insert the `AsyncResponse` futures into a `Vec` to allow the caller to use them as desired. To maintain compatibility with the old behavior for the sake of this diff, the one current callsite has been updated to just dump the returned `Vec` into a `FuturesUnordered`. This will be changed later in the stack.
Reviewed By: quark-zju
Differential Revision: D29204195
fbshipit-source-id: ecee8cff430badd8213c2efef62fc68fbd91fde9
Summary: Nothing was using this metadata, and removing it simplifies the subsequent diffs in this stack.
Reviewed By: quark-zju
Differential Revision: D29147228
fbshipit-source-id: aa4828b710c3ef719f4d66adec5f66cd5b7d05d1
Summary:
This dep got updated in D29165283 (b82c5672fc) across a major version but the code depending
on it wasn't so now it's broken.
Reviewed By: mitrandir77
Differential Revision: D29229087
fbshipit-source-id: 5f2a14dd9f0447dd4578e8321991dfb3df32dcc2
Summary: Update versions for several of the crates we depend on.
Reviewed By: danobi
Differential Revision: D29165283
fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
Summary:
add an option to pass some metadata in the token
This will be used for content tokens, for example. We would like to guarantee that the specific content has been uploaded and it had the specific length. This will be used for hg filenodes upload.
Reviewed By: markbt
Differential Revision: D29136295
fbshipit-source-id: 2fbd3917ee0a55f43216351fdbc1a6686eb80176
Summary:
use `commitknown` edenapi api for checking the existing commits
it uses the same `lookup_commits` under the hood but a bit shorter to use
we won't need the tokens for existing changesets, so can use a simpler api
also, make `lookupfilenodes` function a bit shorter
Reviewed By: markbt
Differential Revision: D29134677
fbshipit-source-id: 257624d64480102c34761560b2bd768049cbfa83
Summary: They will be reused in import_pull_data
Reviewed By: quark-zju
Differential Revision: D29147950
fbshipit-source-id: 192bf33c30067f43c4fcaaf3054741b39efb4e25
Summary: This is an interface for importing pull data into dag
Reviewed By: quark-zju
Differential Revision: D29142979
fbshipit-source-id: b40b94403a044c0b74d1574528aa374ec309a0cf
Summary: This will be used to import pull data into segmented changelog
Reviewed By: quark-zju
Differential Revision: D29142981
fbshipit-source-id: 2d19a035ee0b6cefef8fc0547a5dfb79f284a1de
Summary:
The only real change here is: https://github.com/BurntSushi/ripgrep/pull/1756
This is a patch release but fixes a very glaring bug that others have
depended on. This diff fixes the uses to match the old behavior.
Although it's billed as a "fix", it's actually a huge perf improvement
for Linttool, which uses predominantly recursive suffix globs. The fact
that we don't have to compile ~5,000 regexps at Linttool startup anymore
makes such a huge difference that I am going to do write up soon.
Reviewed By: ndmitchell
Differential Revision: D29085977
fbshipit-source-id: 304470e5fa8cb986738aa0d9dd941641684a9194
Summary:
For a while we had two methods in async runtime: block_on_future and block_on_exclusive, due to historic reasons
Recently those methods were calling same code, and now it is time to replace both of them and rename to block_on
Reviewed By: quark-zju
Differential Revision: D29121107
fbshipit-source-id: 5faa76ae181e491b55d799c23c9de1b4e80298f3
Summary:
The added command gives access to read (execute) dynamicconfig without
using an on-disk repo.
It can be used by the clone script to stage rollout lazy changelog, or just to
verify dynamic config changes without using a repo.
Reviewed By: DurhamG
Differential Revision: D29123072
fbshipit-source-id: e8856d816a636fa860bfcc9694306a4a37552523
Summary:
implement uploading file content via Eden API
* in this diff I aim to upload file content for the given set of filenodes
* also, the code would check with Mononoke using the lookup request what is already there and skip those
* also, this diff introduces calculation of blake2 hash (called ContentId) for file contents (we would probably need to store/cache those and the mapping from hg filenode id to the canonical Mononoke content_id)
* for every uploaded content EdenApi returns a token that we would also need to store later
Reviewed By: markbt
Differential Revision: D29063229
fbshipit-source-id: 739a44bc3ff904cb04a39514ba5efd01c80ba6d0
Summary: This will be used in eager repo integration tests
Reviewed By: quark-zju
Differential Revision: D29113218
fbshipit-source-id: a24232bd6c19010d8ac90d1305f57f1094b06323
Summary:
The mutationstore might contain lots of names (commit hashes) that are unknown
to the repo. When using a lazy changelog, by default, those names will be
looked up remotely, and most of the time the server just returns "name is outside
the graph".
We don't have negative cache to speed it up, because cache invalidation is
tricky - every lazy pull would in theory invalidate them.
To make things faster, let's just skip names in the mutationstore that are unknown
_locally_ without asking the server.
In theory this might affect correctness. Practically, this should only affect "landed as" markers,
because all drafts should be non-lazy. If the "landed as" correctness is an issue, we can fix
forward "landed as" later (ex. by writing down the public commit hash explicitly in
debugmarklanded).
Reviewed By: andll
Differential Revision: D29111710
fbshipit-source-id: 2c1b16a8140ca4f7195bf1206ffe1db4750185b9
Summary: This allows BTreeMap to be used as a provide of config for testing.
Reviewed By: andll
Differential Revision: D28019023
fbshipit-source-id: 94a1585139d6c6612fa163f2ff1aecc909db3a72
Summary: Bumping the crossbeam version so we can use `recv_deadline`. This also necessitates updating the lsp and lsp-types crates.
Reviewed By: alunyov, dtolnay
Differential Revision: D29056473
fbshipit-source-id: 9434e9e0895d82482f4c70afa01a2f77702b965f
Summary:
This functionality is exercised in `test-lfs-copytracing.t` and a few other tests, though I'm not sure if it's necessary outside of that. I've added a `TODO` to investigate and try to eliminate such cases.
Because we'd rather not support this, I implemented it on top of ContentStore - it looks like we might have some lingering datapack-related tests where I think this comes up, too.
Reviewed By: DurhamG
Differential Revision: D29056647
fbshipit-source-id: e6bd5ec31dde06d8c509665f738cfe19270ddc76
Summary: Temporarily implement `upload` on `FileStore` by forwarding to the existing `upload` implementation using `LfsRemote`. Eventually we'll want to re-implement this entirely inside `FileStore`.
Reviewed By: DurhamG
Differential Revision: D29045264
fbshipit-source-id: 8ebbaead4c1032949e5bdd1dcf9a733d0086a77c
Summary: Add a `get_shared_mutable` method to `TreeStore` which behaves like `ContentStore::get_shared_mutable`, returning a store which will read/write only to the local cache stores (aka shared stores).
Reviewed By: DurhamG
Differential Revision: D29042716
fbshipit-source-id: f2236ff8b47ef213b2ffc61501ca301da02dc492
Summary: Add a `get_shared_mutable` method to `FileStore` which behaves like `ContentStore::get_shared_mutable`, returning a store which will read/write only to the local cache stores (aka shared stores).
Reviewed By: DurhamG
Differential Revision: D29042485
fbshipit-source-id: e28af0bafac5eba87523b0ef522f32355106b467
Summary: Add `FetchLogger` to `FileStore` to track remote fetches which match `remotefilelog.undesiredfileregex`.
Reviewed By: DurhamG
Differential Revision: D29042243
fbshipit-source-id: 08ec69d979d66ef93e8bbe8659171a412ade7e22
Summary: `ReportingRemoteDataStore` wraps a store and records all fetches which match `remotefilelog.undesiredfileregex`. Since scmstore doesn't use a hierarchical store composition, this change extracts out the matching/reporting logic into `FetchLogger` so that it can be re-used by both `ReportingRemoteDataStore` and scmstore.
Reviewed By: DurhamG
Differential Revision: D29041488
fbshipit-source-id: db0c4b545886ffebcf2a2841a506d301b2f2f230
Summary: Straightforward update of `debugscmstore` from old scmstore to new scmstore. I'll want to improve this command a bit more in the future, but this at least enables you to easily test arbitrary fetches.
Reviewed By: DurhamG
Differential Revision: D29047527
fbshipit-source-id: 4e10cb88cba4b572d3e413640ca3d800940d675d
Summary:
Our log deletion is a two step process. First it deletes the meta file,
then it deletes the log. There was a bug where if the meta file deletion
succeeded but the log deletion failed (often because Eden has the log open) then
future attempts to delete the log will fail because it fails to delete the meta
file.
Reviewed By: quark-zju
Differential Revision: D29035913
fbshipit-source-id: 33ab9a4e46b285819c4bf14031426983d2b4342f
Summary:
Since all the errors are eaten, we don't have good insight into the
rotatelog cleanup process. Windows users were seeing large growth in their
hgcache and this logging would've been useful, so let's add it.
Reviewed By: quark-zju
Differential Revision: D29035914
fbshipit-source-id: 54f0f05aa8bed1911b6b95438cd2d3ccc45e8ec9
Summary: This method will be used to get all Ids that needs to be included into CloneData::id_map during fast forward pull.
Reviewed By: quark-zju
Differential Revision: D29045538
fbshipit-source-id: f6eb04f537aa5365f7588391ea4c60b3dc010a53
Summary: revert the zstd crates back to previous version
Reviewed By: johansglock
Differential Revision: D29038514
fbshipit-source-id: 3cbc31203052034bca428441d5514557311b86ae
Summary:
implement `known` method based on the existing `lookup` API that has been already implemented
this might be useful for eden api pull project / wireproto deprecation, as the pull algorithm queries the server for whether commits are known by the server
Differential Revision: D29030429
fbshipit-source-id: 4d78c8b1e7983d89de917951d9765b1bd1d90fb0
Summary: This will be used in fastpath pull
Reviewed By: quark-zju
Differential Revision: D29010304
fbshipit-source-id: 9a7464fbc42544f8c5cc52f6fa25db5f7b5d6bd9
Summary: idset_to_flat_segments allows to return intersection between flat segments and span set
Reviewed By: quark-zju
Differential Revision: D28980521
fbshipit-source-id: da90c66a021a1e8bb95edbbc2e06cd2a850d7dcd
Summary:
This diff introduces Subspan trait for objects that contains spans and allows to take a 'subspan' of an object
Implementing Subspan trait allows to intersect arbitrary objects and spans
Reviewed By: quark-zju
Differential Revision: D28980523
fbshipit-source-id: 29a2e58ec3c79a838db751d8a37227b4cdaaa372
Summary: Update to latest version. This includes a patch to async-compression crate from [my PR updating it](https://github.com/Nemo157/async-compression/pull/125), I will remove once the crate is released.
Reviewed By: mitrandir77
Differential Revision: D28897019
fbshipit-source-id: 07c72f2880e7f8b85097837d084178c6625e77be
Summary: This will be used for rate limiting decisions. Also, could be logged to scuba tables to get more info about clients.
Reviewed By: quark-zju
Differential Revision: D28750197
fbshipit-source-id: 83f54e38f998c9dd824ef2d3834c777a44d0ffed
Summary: Let clients connect to lfs with HTTP through unix socket so we don't have to worry about certificates presence.
Reviewed By: johansglock
Differential Revision: D28683392
fbshipit-source-id: f6228b4099ef04fe584e320cb1892e6cb513e355
Summary:
create end to end intergation for the lookup API on the client
Start prototyping of `hg cloud upload` command.
Currently, it just performs lookup for existing heads.
This way we can end to end test the new APIs.
Reviewed By: markbt
Differential Revision: D28848205
fbshipit-source-id: 730c1ed4a21c1559d5d9b54d533b0cf551c41b9c
Summary:
Files upload will be executed in 2 stages:
* check if content is already present
* upload missing files
The check api is generic, it could be used for any id type. Called 'lookup' API.
Reviewed By: markbt
Differential Revision: D28708934
fbshipit-source-id: 654c73b054790d5a4c6e76f7dac6c97091a4311f
Summary:
Time 0.2 is current, and 0.1 is long obsolete. Unfortunately there's a
large 0.1 -> 0.2 API change, so I preserved 0.1 and updated the targets of its
users. Also unfortunate that `chrono` has `oldtime` as a default feature, which
makes it use `time-0.1`'s `Duration` type. Excluding it from the features
doesn't help because every other user is specifying it by default.
Reviewed By: dtolnay
Differential Revision: D28854148
fbshipit-source-id: 0c41ac6b998dfbdcddc85a22178aadb05e2b2f2b
Summary:
Instrument file scmstore with tracing logging. There's more we should add here, but this will be a good starting place - I've already discovered some issues from looking at the log output. (Why does drop run twice? How does it run twice?)
It'd also probably be nice to support formatting the output like https://crates.io/crates/tracing-tree, which will be a lot less cluttered by the logged fields (like `attrs` on `fetch`).
Reviewed By: DurhamG
Differential Revision: D28750954
fbshipit-source-id: 63baa602f7147d24ac3e34defa969a70a92f96a4
Summary:
Now that EdenFS is using EdenAPI more, let's let it take advantage of
EdenAPI's better batching. We alread have a batch API for files, let's copy the
pattern for trees as well. This adds the C++ bindings. The next diff consumes
this from EdenFS
This is largely just a copy of how batch blob fetching does this. But I'm a C++
noob, so feel free to tear this apart with nits.
Reviewed By: chadaustin
Differential Revision: D28426789
fbshipit-source-id: 88d359985e849018fb3c2b4ef9e52d07c96bf31a
Summary:
Now that EdenFS is using EdenAPI more, let's let it take advantage of
EdenAPI's better batching. We alread have a batch API for files, let's copy the
pattern for trees as well. This first diff just produces the Rust code. Future
diffs will add the C++ bindings then integrate it into EdenFS.
This is largely just a copy of how batch blob fetching does it.
Reviewed By: chadaustin
Differential Revision: D28426790
fbshipit-source-id: 822ef6e7b3458df5dba7a007657e85351162b9ff
Summary:
Add the `fetch_contentsha256` python method to `filescmstore`, which accepts a list of keys and returns a list of (key, sha256).
This is intended to be used by the modified `status` command implementation, which will prefer comparing content hashes to directly comparing file content.
Reviewed By: DurhamG
Differential Revision: D28696618
fbshipit-source-id: a0304319b0a19d4f09d07bec02dc41964aec7255
Summary:
Merge `found_file` and `found_aux_indexedlog` into a new `found_attributes` method, which simply "or"s the newly found attributes into the `found` map.
Replaces the `satisfies` concept with a new `pending` check, used the same way by each `pending_*` method, which considers a key pending if fetching from a store which returns a given set of attributes would allow us to resolve any requested by missing attributes, optionally taking into account attributes that can be computed from those already found. This will still need to be adjusted to support preferring remote fetching of attributes to local computation, but it is no longer as brittle as the previous implementation: there's no requirement that aux data be computed as content is fetched in order to avoid redundantly fetching content.
Move attribute computation to a separate phase, and filter out un-requested attributes in the `finish` function.
Reviewed By: DurhamG
Differential Revision: D28694192
fbshipit-source-id: 9b096c056736cadc0f97ff09243ed09d5266504d
Summary: Use associated constants instead of methods for `FileAttributes` bit masks.
Reviewed By: DurhamG
Differential Revision: D28724729
fbshipit-source-id: 441c0d2361166824c4ee7cfd5ad0b6f21ee1ac26
Summary:
Previously, the `found_error` required `&mut self`, even though it only ever interacted with the error fields. This prevents Rust's type checker from validating the safety of logging errors while iterating over the `found` map, for instance.
Replacing the `&mut self` method call with a field access into an existing `&mut self` resolves this problem, and allows logging errors while mutating other fetch state.
Reviewed By: DurhamG
Differential Revision: D28722547
fbshipit-source-id: 59c6a530cbf331282d6f654a56e492d47cafcd2f
Summary:
Don't try to fetch from a store if we don't have any pending keys.
Handle missing content when writing to cache after fetching from remote stores. Currently, `found_in_*` will be populated even if we don't store the content, having just used it for aux data computation. This change won't be necessary, but won't cause any problems either, after the next change which only prunes overfetching in the `finish` method, allowing remote blobs to be written to local cache even if we only fetched them to compute their attributes. I might revert this portion of the change, or warn if content is unexpectedly unavailable.
Reviewed By: DurhamG
Differential Revision: D28694964
fbshipit-source-id: 465211c9257cbf49b1cb68856473323fc940f10b
Summary: Extends the previous change to add support for computing aux data (currently only Content Sha256) and caching it locally. Introduces a `FetchState` config option, `compute_aux_data`, which controls if content will be fetched in order to compute aux data, or if unavailable aux data will be treated as "not found".
Reviewed By: DurhamG
Differential Revision: D28528456
fbshipit-source-id: 26189d18c8e453040f3c1f6e22a34d623a5aa40d
Summary:
Extends the `FileScmStoreBuilder` to construct two new indexedlog stores for caching aux data. The stores will be created in a directory adjacent to the normal non-LFS indexedlog stores.
Currently aux data stores will not be constructed for production users, a configuration option will be introduced to gate this before `.store_aux_data()` is called in the `filescmstore` constructor bindings.
Reviewed By: DurhamG
Differential Revision: D28689693
fbshipit-source-id: e3ad1594e5beee00b1a8b9fe489e3b6af3a2e93e
Summary:
Modify `FileStore` to introduce basic aux data fetching. Aux data is currently read from a separate IndexedLog store, serialized with `serde_json` (chosen for expediency / ease of debugging, I intend to optimize the storage format before releasing this, at the very least to avoid unnecessarily serializing the key path).
Currently aux data fetching will never succeed, as aux data fetching is not supported in the EdenApi "files" API and nothing else exists to populate the local aux data stores. Later in this stack, computing aux data (currently only content sha256) to populate the aux data storage is implemented.
Reviewed By: DurhamG
Differential Revision: D28526788
fbshipit-source-id: c8e21a1377689d7913a68426a3a480d53148da66
Summary:
Simplify tracking of incomplete fetches in preparation for attributes support in the next change.
Now, all keys which have not been completely and successfully fetched are recorded in `pending`, and are removed only when the complete fetch is recorded in `found`. Keys are now removed from `lfs_pointers` and `pointer_origin` as they are completed, as they aren't needed for anything other than fetching from local LFS and remote LFS respectively.
Reviewed By: DurhamG
Differential Revision: D28546515
fbshipit-source-id: c657e5c6350cadc8da970f57bb7694ed71022efb
Summary: I have modified the places where most of the errors were raised that users reported and were resolved by renewal of certificates.
Reviewed By: krallin
Differential Revision: D28568561
fbshipit-source-id: 44fb127a49bde83efee1c934e0435b31f8602a8d
Summary:
Like it says in the title. The API between Bytes 1.x has changed a little bit,
but the concepts are basically the same, so we just need to change the
callsites that were calling `bytes()` and have them ask for `chunk()` instead.
This diff attempts to be as small as it can (and it's already quite big). I
didn't attempt to update *everything*: I only updated whatever was needed to
keep `common/rust/tools/scripts/check_all.sh` passing.
However, there are a few changes that fall out of this. I'll outline them here:
## `BufExt`
One little caveat is the `copy_to_bytes` we had on `BufExt`. This was
introduced into Bytes 1.x (under that name), but we can't use it here directly.
The reason we can't is because the instance we have is a `Cursor<Bytes>`, which
receives an implementation of `copy_from_bytes` via:
```
impl<T: AsRef<[u8]>> Buf for std::io::Cursor<T>
```
This means that implementation isn't capable of using the optimized
`Bytes::copy_from_bytes` which doesn't do a copy at all. So, instead, we need
to use a dedicated method on `Cursor<Bytes>`: `copy_or_reuse_bytes`.
## Calls to `Buf::to_bytes()`
This method is gone in Bytes 1.x, and replaced by the idiom
`x.copy_to_bytes(x.remaining())`, so I updated callsites of `to_bytes()`
accordingly.
## `fbthrift_ext`
This set of crates provides transports for Thrift calls that rely on Tokio 0.2
for I/O. Unfortunately, Tokio 0.2 uses Bytes 0.5, so that doesn't work well.
For now, I included a copy here (there was only one required, when reading from
the socket). This can be removed if we update the whole `fbthrift_ext` stack to
Bytes 1.x. fanzeyi had been wanting to update this to Tokio 1.x, but was blocked on `thrift/lib/rust` using Bytes 0.5, and confirmed that the overhead of a copy here is fine (besides, this code can now be updated to Tokio 1.x to remove the copy).
## Crates using both Bytes 0.5 & Bytes 1.x
This was mostly the case in Mononoke. That's no coincidence: this is why I'm
working on this. There, I had to make changes that consist of removing Bytes
0.5 to Bytes 1.x copies.
## Misuse of `Buf::bytes()`
Some places use `bytes()` when they probably mean to use `copy_to_bytes()`. For
now, I updated those to use `chunk()`, which keeps the behavior the same but
keeps the code buggy. I filed T91156115 to track fixing those (in all
likelihood I will file tasks for the relevant teams).
Reviewed By: dtolnay
Differential Revision: D28537964
fbshipit-source-id: ca42a614036bc3cb08b21a572166c4add72520ad
Summary: Modifies `treescmstore` and `filescmstore` to also construct `TreeStore` and `FileStore` respectively. Currently these newly constructed stores are not used anywhere, no application code behavior should change as a result of this.
Reviewed By: DurhamG
Differential Revision: D28237680
fbshipit-source-id: 2bf3fd4b96be8c26e5c1e55cfd2e865f98e6ba91
Summary:
Implement `HgIdDataStore`, `RemoteDataStore`, `LocalStore`, `HgIdMutableDeltaStore`, and `ContentDataStore` for `FileStore`.
Currently I've left `RemoteDataStore::upload` unimplemented, as it's a little more complicated than the other functionality (with lots of private field accesses), and is probably worth building a good API for first. As a temporary workaround, I can store an `LfsRemote` (which requires an associated `LfsStore` for cache) and just call upload on that for now, but that's pretty ugly with the current design. I could also construct one on the fly, but it currently stores a bare `LfsRemoteInner`, not an `Arc<LfsRemoteInner>`. I'll take one of these three approaches after getting the integration tests running with the new `TreeStore` and `FileStore`.
Reviewed By: DurhamG
Differential Revision: D28235602
fbshipit-source-id: 13c72cd9379cba70a2ca7038dad419346fe0b14a
Summary:
Implement `HgIdDataStore`, `RemoteDataStore`, `LocalStore`, `HgIdMutableDeltaStore`, and `ContentDataStore` for `TreeStore`.
Also add a `Drop` impl that flushes the local stores, which matches the behavior of `ContentStore` (such as impl does not exist for the underlying stores, but it might be more appropriate there).
Reviewed By: DurhamG
Differential Revision: D28235060
fbshipit-source-id: 5a12d8c2ecff9fcc204cf437bf6f2a98f08645b4
Summary:
Introduce a new, flat, FileStore implementation. This `FileStore`, like the previously submitted `TreeStore`, directly handles all the fallback, local caching, etc, necessary to implement our storage system.
The API supports fetching batches of `Key`s, writing batches of entries (currently only in the "hg file blob" format, with copy header embedded), and querying only the local subset of underlying stores (to allow implementing `get_missing`). Other store subsets and write features will be added in the future.
Reviewed By: DurhamG
Differential Revision: D28138800
fbshipit-source-id: ca5bb91c66fa078019a19180235dd632ea73a0b3
Summary:
Introduce `from_hg_file_blob` and `from_content` LfsPointersEntry constructors, which are used for creating the correct `LfsPointersEntry` for a `Delta` (HgId + file content).
Add `sha256` accessor to `LfsPointersEntry`. Comments on `LfsPointersEntry` and looking at the construction logic suggest there should always be an associated Sha256 content hash. We use it often, so an accessor is useful to avoid the cumbersome HashMap access + match.
Add `fetch_available` to `LfsStore`, which is used by scmstore for handling cases where either only the pointer, or both the pointer and data are available. Existing LFS code directly accesses the underlying blob and pointer store.
Reviewed By: kulshrax
Differential Revision: D28231747
fbshipit-source-id: e6b1f210605d821f542fcb8e87aea366a0864d44
Summary:
Convert client certificates (which are expected to be supplied as PEM files) into an in-memory PKCS#12 archive to pass into libcurl. This is necessary on certain platforms (such as Windows) whose native crypto APIs do not support loading PEM files.
This was previously landed as D27637069 (5b759a2b52), which unconditionally converted the certificates under the assumption that all major TLS backends support PKCS#12. That assumption is still true, but it did not account for the fact that libcurl itself is dynamically linked on some platforms (such as MacOS), and the system libcurl may be too old to support support in-memory certs (via `CURLOPT_SSLCERT_BLOB` added in libcurl version 7.71.0). This diff gates this feature behind the `http.convert-cert` config option, which we can selectively set on platforms where it is needed.
Reviewed By: mzr
Differential Revision: D28524444
fbshipit-source-id: 4af9cdd60b8ef3977ad81abdb8e406c63795e628
Summary: Instead of passing a client certificate path to libcurl, load the certificate into memory and pass it to libcurl as a blob using `CURLOPT_SSLCERT_BLOB`. This allows us to convert the certificate format in-memory from PEM to PKCS#12, the latter of which is supported by the TLS engines on all platform (and notably SChannel on Windows, which does not support PEM certificate).
Reviewed By: quark-zju
Differential Revision: D27637069
fbshipit-source-id: f7f8eaafcd1498fabf2ee91c172e896a97ceba7e
Summary:
Adding mappng to keep track of two things:
1) keep track of the latest source commit that was synced into a given target - this will be used during sync_changeset() method to validate if a parent changeset of a given changeset was already synced
2) which source commit maps to what target commit
Reviewed By: ikostia
Differential Revision: D28319908
fbshipit-source-id: f776d294d779695e99d644bf5f0a5a331272cc14
Summary: Right now this is not very useful. Let's make it more useful.
Reviewed By: DurhamG
Differential Revision: D28281653
fbshipit-source-id: ef3d7acb61522549cca397048c841d1afb089b9b
Summary:
This makes it easier to see what builder functions were registered:
% EDENSCM_LOG=edenapi=debug lhg log -r .
May 06 16:40:29.355 DEBUG edenapi::builder: registered eagerepo::api::edenapi_from_config to edenapi Builder
Reviewed By: DurhamG
Differential Revision: D28271366
fbshipit-source-id: f6c7c3aa9f29c3e47c2449e3d5fc16474aa338b0
Summary:
The server1 was not used after D27629318 (ba7e1c6952) while the test intentionally wants to
exercise graph isomorphism. So let's revive server1 in the test.
Reviewed By: andll
Differential Revision: D28269926
fbshipit-source-id: 0a04031415f559f8a6eb81f1e2f2530329a2a3bc
Summary: This makes it easier to use it in tests.
Reviewed By: DurhamG
Differential Revision: D28006549
fbshipit-source-id: 90e29b220453a3d7a260d0a62d697d64363d9a6c
Summary:
This makes it easier to filter logs related to remote fetching.
The `DEBUG dag::protocol: resolve ids [0] remotely` means the lazy hash resolution is working.
Reviewed By: kulshrax
Differential Revision: D27971117
fbshipit-source-id: f2492204c70d793997d0c3865e500bbad56b1953
Summary:
Write commit to master group. This provides proper "CloneData" and allows us to
actually test lazy commit hash backend (since only commits in the master group
can have lazy hashes).
Reviewed By: DurhamG
Differential Revision: D27971123
fbshipit-source-id: 4e19486007ddc89de7468be65445559f34d796f5
Summary:
The trees endpoint is another example where we try to send errors to the
client. As it was done previously we would fail to log any errors on the
server side. This diff corrects that by using custom_cbor_stream.
Reviewed By: kulshrax
Differential Revision: D28111102
fbshipit-source-id: 468095d024647f472b8ad9a9e17ca8364605ff98
Summary:
eagerepo -> metalog -> git2 -> libgit2-sys -> libgit2 conflicts with edenfs'
non-Rust libgit2 dependency. Rust git2 crate does not seem to provide a way to
depend on specified libgit2.
Quote https://github.com/rust-lang/git2-rs/issues/263#issuecomment-450934287:
> It's expected that git2-rs builds its own copy of libgit2 and doesn't use the
> system version, as the system version is likely incompatible
It also seems non-trivial to make buck C++ use the libgit2 frm `libgit2-sys` crate.
Let's just avoid depending on eagerepo from edenapi directly for now to solve the
issue. This basically revives D27948369 and D27951632.
Reviewed By: xavierd
Differential Revision: D28243784
fbshipit-source-id: 0c38c20c2d3a80c550732129da572fe26a229799
Summary:
We have a linker issue on Windows when building EdenFS with CMake:
```
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpSetStatusCallback referenced in function winhttp_connect
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpOpen referenced in function winhttp_connect
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpCloseHandle referenced in function winhttp_close_connection
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpConnect referenced in function winhttp_connect
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpReadData referenced in function winhttp_stream_read
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpWriteData referenced in function winhttp_stream_read
backingstore.lib(winhttp.o) : error LNK2019: unresolved external symbol __imp_WinHttpQueryOption referenced in function certificate_check
```
This fixes that.
Reviewed By: xavierd
Differential Revision: D28230163
fbshipit-source-id: f74e42ee30ec8f3b81c1f80b7cf63a21ea97732c
Summary: Windows path like `eagerepo:///C:\foo\bar` needs special handling.
Reviewed By: kulshrax
Differential Revision: D27971119
fbshipit-source-id: 9d4b87782eca2734b708565f0ee22a7495253cff
Summary: Add a way to fetch tree content without going through store.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200387
fbshipit-source-id: 8f5b2214aafba39c7674f0f6b27af0c985f0ea72
Summary:
Expose NameDag's `import_clone_data` API so this can be then exposed via
`pydag`.
Reviewed By: kulshrax
Differential Revision: D27971118
fbshipit-source-id: c9d869ffbbc8ba5a7a6ae98d17a2b7ea713bc675
Summary: Make it easier to check whether APIs in EagerRepo is called or not.
Reviewed By: andll
Differential Revision: D27955426
fbshipit-source-id: 27ca505c63596368cff98642de010b5b5717454c
Summary: It's not an error case. It just means all nodes are unknown to the repo.
Reviewed By: kulshrax
Differential Revision: D27951619
fbshipit-source-id: 672932af3a54ffa5adfa5cccbfff7edbf4f24022
Summary:
Now EdenApi trait is moved to a separate crate, we can inline the EdenApi
backed by EagerRepo without using dynamic registration functions.
Reviewed By: andll
Differential Revision: D28006553
fbshipit-source-id: 427513da94db228745b1a7e90af0e62296056128
Summary: So that we don't duplicate the URL handling in Python.
Reviewed By: andll
Differential Revision: D28006552
fbshipit-source-id: 2efda622fe86787373fa4ec5978537588defec28
Summary:
There have been a bunch of problems with the previous approach to scmstore, so I'm going to try to start simple, make it feature complete, and then add async integration and factor out generic functionality as appropriate.
This change contains a `TreeStore` implementation with a single, synchronous, batch read method (supporting local storage, memcache, and legacy fallback, with writing missing to cache).
Add `TreeStoreBuilder`, which duplicates the existing `TreeScmStoreBuilder`, which some changes that make it easier to use for this case. I intend to unify these in the future.
Add an inherent impl for `EdenApiTreeStore` that provides subset of the `BlockingEdenApi` trait, which eliminates the need to unpack this type into a different adapter as the old `scmstore` code does. This might not be the right approach here, in reality we only need a `(client: Arc<dyn EdenApi>, repo: String)` here for trees, and that + `ExtStoredPolicy` for files, so we could take the `EdenApiAdapter` approach here too. The only reason we have to do any of this is because when `pyrevisionstore` is called to construct `scmstore` / `contentstore`, all we have is `Arc<EdenApiTreeStore>`. We could also just make the `EdenApiRemoteStore` fields public, and access them through the `Arc`.
Add `add_mcdata` method to `MemcacheStore`, `impl TryFrom<Entry> for McData`, and `impl From<McData> for Entry` for convenience when working with `MemcacheStore` (so we don't need to manually unpack the type and build `Entry`, or manually build a fake `Delta` from `Entry` to write).
Reviewed By: DurhamG
Differential Revision: D28076900
fbshipit-source-id: 7fdb5e8a42d052879eff449f60d40a83cfa7145d
Summary:
Both `get_local_path` and `get_cache_path` take suffix as as `PathBuf`, even though they only ever use it as a reference. `get_local_path` also takes a reference to a `PathBuf`, even though it always clones it internally, and takes an `Option`, even though it just maps across the contents of the option.
I modified `get_local_path` to accept a `PathBuf` by move, which it uses directly, and to not take an `Option` (instead just calling `map` externally, removing some unnecessary unwraps), and for both functions to accept `impl AsRef<Path>` for suffix.
Reviewed By: DurhamG
Differential Revision: D28100527
fbshipit-source-id: df28b51c8005f3d95acc8e082b40adaab18e31c9
Summary: Add a Read/Write Guard API to IndexedLogHgIdDataStore which allows client code outside the module to perform a series of reads and writes without locking for each individually.
Reviewed By: kulshrax
Differential Revision: D28075788
fbshipit-source-id: 2a65a426f443e1a421198ad8b4c610e4822574f7
Summary:
Add get_entry, put_entry, and flush_log inherent methods to IndexedLogHgIdDataStore. Refactor callers to use them in cases where they don't lock across multiple reads / writes (to avoid performance regressions).
This should allow `ReadStore` and `WriteStore` to be moved out of the module.
Reviewed By: DurhamG
Differential Revision: D27979828
fbshipit-source-id: c9fb8c4ac68f67b285c72396509aa17928aa54ed
Summary: This is step towards unifying native merge/rebase structs with native checkout - we now construct native checkout plan from the action map, instead of directly making it from the diff
Reviewed By: quark-zju
Differential Revision: D28078156
fbshipit-source-id: 318d7e419ca9fef15a4aebf7494451f69a3bbbe5
Summary:
This diff makes concurrency of native checkout to be configurable
This config can be used to reduce concurrency on platforms that are known to cause issues with watchman due to too many checkout operations
Reviewed By: quark-zju
Differential Revision: D28074993
fbshipit-source-id: 0a09fcf3ae48d08cead36da56c06b546aecd16b4
Summary: This diff refactors out `Checkout` component from checkout plan and allows to configure parallelism in checkout
Reviewed By: quark-zju
Differential Revision: D28074994
fbshipit-source-id: 72933c757d6e27615d1ef2bb4652bc67c9c3253d
Summary:
From what I can see, this was added when EdenFS had a Mononoke store, which is
now long gone, thus we should be able to remove the Curl dependency altogether.
Reviewed By: fanzeyi
Differential Revision: D28037816
fbshipit-source-id: 834f7db64bab5dda1748ad2f033c27a2854b0ba4
Summary:
This updates hg to have a different amount of retry for backoffs requested by
the server and errors.
The rationale is that backoffs are fairly well understood and usually caused by
a surge in traffic where everybody wants the same data (in which case we should
be willing to wait to get it because there is literally no alternative),
whereas general errors aren't predictable in the same way.
We're now effectively at a point on the server side where _all_ our instances
have the exact same load, so if any server is telling you to backoff, that
pretty much guarantees that the whole tier has too much traffic to deal with.
This leaves us with two options:
- Tell clients to wait longer and smooth out the traffic surge.
- Add enough capacity that even our biggest surges don't result in _any_
throttling.
The latter is a bit unrealistic unrealistic given we routinely get egress
variations in excess of 5x (here's an example: https://fburl.com/ods/pidsrqnl),
so this does the former.
This also updates the client to tell the server how many attempts it has left
in addition to how many it used up so far. How many are left is more meaningful
for alerting!
Finally, it adds a bit of logging so that in debug mode you can see this
happening.
Reviewed By: quark-zju
Differential Revision: D28092797
fbshipit-source-id: f61410e39c4a3e3356371a3c7bd7892de4beacc8
Summary: They will be useful in the pull exchange path.
Reviewed By: kulshrax
Differential Revision: D27951625
fbshipit-source-id: 31dfd54cda7a0d0d0b2565e0ecddc2e82eebcc05
Summary:
This will be used to migrate "pull" from bundle2 to EdenApi.
This diff just defines the interface in a minimal way.
Actual implementation is TBD.
Differential Revision: D27951634
fbshipit-source-id: 66210a833d8fc87452a19d1935e9d208a1d31b14
Summary:
This will be used to migrate "pull" from bundle2 to EdenApi.
This diff just defines the interface in a minimal way. Actual implementation is TBD.
I dropped the progress callback parameter since the native Rust progress bars
make it unnecessary. I also avoided the blocking API interface, since we would
like pure Rust code to not call blocking APIs which is likely a mistake, and
only use `async_runtime::block_on` in Python bindings - in that case blocking
API is not needed.
Reviewed By: andll
Differential Revision: D27951624
fbshipit-source-id: 3844dd96df265cc6e61d7cf5e79f39c891e8117d
Summary: Implement the endpoint. Most of the complexity is type conversion.
Reviewed By: andll
Differential Revision: D27929203
fbshipit-source-id: ff865ae727e383cd2b465f8bcd1e29f0c9316ff6
Summary: Implement the endpoint. Most of the complexity is type conversion.
Reviewed By: andll
Differential Revision: D27929201
fbshipit-source-id: b96f65a0173a31f716272c8e0dd47ce8a90759cd
Summary: This makes CloneData possible to represent an empty repo.
Reviewed By: sfilipco
Differential Revision: D27926246
fbshipit-source-id: 0bcead224ef5b89c66d07a34d8217edaef62177f
Summary: Implement the files API. It's just reading content from the zstore.
Reviewed By: kulshrax
Differential Revision: D27926251
fbshipit-source-id: 54d04caa63e01b6ce5b9c785990c14043f7f22ad
Summary: The will be useful for "push" logic.
Reviewed By: kulshrax
Differential Revision: D27951633
fbshipit-source-id: 38bbdc554f017d5776df0577b82fbb0c78d18a83
Summary:
This will be useful for "push" related logic.
The name "eager" is to make it explicit that the repo is not lazy.
Reviewed By: kulshrax
Differential Revision: D27951618
fbshipit-source-id: 8039059beba68d269c752bc8ed3e72bde0c55790
Summary:
Currently it's hard to test EdenApi related features in hg tests. The Mononoke
test suite can do it but it's too heavyweight. Looking at the API surface of
EdenApi it's actually quite small. So let's add a minimal Rust struct that can
serve as an EdenApi server.
This diff just adds a few minimal features. EdenApi related features and
push/pull support will be added later.
The name "eager" is to make it explicit that the repo is not lazy. I thought
about names like "testrepo" or "serverrepo", but the implementation is
somewhat "sound" to be used as a client, non-test repo. It can potentially
be used as starting point for a real "repo" in pure Rust. So I didn't choose
those names.
(I'm not entirely happy with the name but it's more like a placeholder
that makes it look different from other names for now).
Reviewed By: kulshrax
Differential Revision: D27926255
fbshipit-source-id: ad7a023de5e77605a553509de82ff13ae8112439
Summary:
This allows an external crate C that implements `EdenApi` to depend on a more
lightweight library just providing `EdenApi` without things like `hg_http`.
Then the `edenapi` crate can depend on C too.
Didn't move it to `edenapi_types` because it would add extra dependencies
(http, http_client, auth, etc.)
Reviewed By: kulshrax
Differential Revision: D28006548
fbshipit-source-id: 6e828974fd3f78fec70d4a04ae7be85abc459b36
Summary:
The `Builder` API is the main API used by external users to obtain an `EdenApi`
client. In the future we want to support different kinds of `EdenApi`, like a
local repo serving it, if `paths.default` is set to something like
`myrepotype:path`. Make `Builder` more flexible to support non-HTTP `EdenApi`s,
by returning `EdenApi` trait objects.
The old builder that is coupled with HTTP is renamed to HttpClientBuilder.
Reviewed By: kulshrax
Differential Revision: D28018586
fbshipit-source-id: 1eff7bbb8f0e5521a9bcf5a225ac361ddf7c310f
Summary:
This ensures the User-Agent is always set. It also makes the `header` less
unnecessary.
Reviewed By: DurhamG
Differential Revision: D28018587
fbshipit-source-id: 1125d2122431579f127e81c4713de45135b1f972
Summary:
Make it easier to use.
This makes it easier for other crates to use `edenapi::Result<T>`, which is
a bit shorter than `Result<T, EdenApiError>`. Also re-export `Metadata`
from revisionstore-types so callsite does not need to depend on
revisionstore-types explicitly.
Reviewed By: kulshrax
Differential Revision: D27926250
fbshipit-source-id: c85198b5c151e10a2d4d2567e23e32605a3e7c36
Summary:
New endpoint. This endpoint can be used for prefix lookup and the contains
check.
Reviewed By: quark-zju
Differential Revision: D28034533
fbshipit-source-id: d724b85c3816414475b142215e3052d0b555cf59
Summary:
These structures are going to be used to implement the `commit/hash_lookup`
endpoint in EdenApi.
Reviewed By: quark-zju
Differential Revision: D28034532
fbshipit-source-id: 7b00d0d97dd0593dfa43834cda9fc9e9ab9021c5
Summary:
Generic container for a bunch of uniform objects. This is primarily intended
for requests and responses which can be difficult to evolve when the top level
object is an array. For cases where evolution is required we would
probably replace the Batch wrapper with a specialized type. For example,
starting from `Batch<MyRequest>` we would change to:
struct MyRequestBatch {
pub batch: Vec<T>,
pub evolution: Evolution,
}
Reviewed By: quark-zju
Differential Revision: D28034534
fbshipit-source-id: d231c063eeacf3500b75ae76bcc101ccbcda8881
Summary: Those methods only access store/network to fetch content but does not write to disk
Differential Revision: D28040640
fbshipit-source-id: e45dd08e12d128d54b3446e1137465981cde8f13
Summary:
This and following diff will refactor CheckoutPlan creation.
Right now we create CheckoutPlan from manifest diff and then manipulate it with methods like `with_sparse_profile` to adjust plan for different cases.
Those 'adjustment' do not work great with the structure of CheckoutPlan, for example `with_sparse_profile` has to create temporary HashSet just to index files in CheckoutPlan
We are going to add more adjustments in the future (for example, checkout --clean), and will run into same issues with current structure of CheckoutPlan
To avoid those issues, we are going to refactor this code, so that instead of Diff -> CheckoutPlan -> adjustments, we are going to have Diff -> ActionMap -> adjustments -> CheckoutPlan
The structure of CheckoutPlan is still good for it's direct purpose (execution), but all the 'changes' to the plan will be done in ActionMap instead.
Reviewed By: DurhamG
Differential Revision: D27980390
fbshipit-source-id: 403f371fd2fe7760984925a38429e1bfb88d8e3f
Summary: When checking out on dirty copy without --clean this function can be used to check if checkout operation conflicts with currently modified files
Reviewed By: quark-zju
Differential Revision: D27953965
fbshipit-source-id: 4096506e4cbf8b102e0afa1a929c066dfa474825
Summary:
This crate introduces consumer API for status in rust
Currently the implementation will just take status from Python and convert it into this struct
But in the future we can get pure Rust implementation to get status
Reviewed By: quark-zju
Differential Revision: D27953963
fbshipit-source-id: 29c876400c82056eaf81fffa4adc814473853c1e
Summary: This method can be used to 'normalize' path for case insentive use cases
Reviewed By: quark-zju
Differential Revision: D27953964
fbshipit-source-id: 421832af22af9a3b56eec0d045b9f983592ed192
Summary: It has been fixed and we now set auth config with higher priority anyway.
Reviewed By: johansglock
Differential Revision: D28026081
fbshipit-source-id: 7086b48139bb05ffadd782898a1758ae06236aca
Summary:
The check unknown logic would block checkout for any unknown files that
were to be overwritten. We want to allow checkouts where the unknown file has
the same content as the desired checkout value. Ideally we'd check it against
the SHA1 hash of the file we're about to checkout, but since content hashes
aren't available yet we can limit our check to resumed checkouts for now.
Reviewed By: andll
Differential Revision: D27804719
fbshipit-source-id: e129ca694080051420e2cb685c7eeb5f1adee005
Summary:
Every function on CheckoutPlan required the VFS already, and the
CheckoutProgress is storing the VFS and living on the CheckoutPlan, so it makes
sense to just store the VFS on the CheckoutPlan.
Reviewed By: andll
Differential Revision: D27825088
fbshipit-source-id: 3d063fdfd1a50983b60d00a3992a893e71732f94
Summary:
Now that CheckoutPlan can look for untracked files, it breaks the
ability to continue a checkout since those untracked files are considered dirty.
In a later diff we'll use the CheckoutProgress to inspect the dirty files and
determine which are actually dirty and which can be overwritten. To do so
though, we need access to the CheckoutProgress earlier. So let's just store it
on the CheckoutPlan.
This is a little awkward because we're passing the root VFS to the constructor
so CheckoutProgress can be instantiated, but then also passing it to every
CheckoutPlan function as well. We should probably just store the vfs on the
CheckoutPlan. If others agree, I can make a diff to do that.
Reviewed By: andll
Differential Revision: D27804720
fbshipit-source-id: e819c27fa8580c82a8cf8f0baf22ac1ea707ee54
Summary:
Add a way to extend the graph with concrete commit hashes, without specifying
exact commit messages.
Reviewed By: sfilipco
Differential Revision: D27897894
fbshipit-source-id: fccd64b2fef1386d79cddd841208da6a938a5217
Summary:
Current implementation had a bug(demonstrated in test case) in handling unknown files on case insensitive fs.
When file is replaced with another file, whose name only differs in case, we get two distinct update operations - rm file, and create file.
Create operation checks against unknown files, and see that file "exists". In this case operation is aborted.
However, we should proceed in this case, and this diff fixes it.
Reviewed By: quark-zju
Differential Revision: D27926953
fbshipit-source-id: 48c8824322d6e5dd9ae57fee1f849b57dc11a4df
Summary: Will be useful on case insensitive fs
Reviewed By: quark-zju
Differential Revision: D27946982
fbshipit-source-id: e7a2fd0ee503c4a580531e6f52225fe2316e5b76
Summary: This diff adds flag to VFS to detect whether FS is case sensitive. The logic in this code losely follows similar logic in Python
Reviewed By: quark-zju
Differential Revision: D27926952
fbshipit-source-id: 36fdf4187ae513b25346f704050c64f9a1a4ec74
Summary:
Update the zstd crates.
This also patches async-compression crate to point at my fork until upstream PR https://github.com/Nemo157/async-compression/pull/117 to update to zstd 1.4.9 can land.
Reviewed By: jsgf, dtolnay
Differential Revision: D27942174
fbshipit-source-id: 26e604d71417e6910a02ec27142c3a16ea516c2b
Summary: Those conflicts can be resolved in Python using textual 3-way merge
Reviewed By: DurhamG
Differential Revision: D27752770
fbshipit-source-id: 816a601112ee2e747d780f8b17473049df46b469
Summary: This diff exposes manifestbuilder that can be used to construct memctx from Python
Reviewed By: DurhamG
Differential Revision: D27639395
fbshipit-source-id: ed047d3d7533f9d2bc592a5d948dc01e429692a7
Summary: Add bookmark method to Edenapi client for use in Mercurial client.
Reviewed By: kulshrax
Differential Revision: D27174441
fbshipit-source-id: cdc324e9115e87eab2e078209bbbc266e4e1dcdc
Summary:
It is considered an anti-pattern (https://rust-unofficial.github.io/patterns/anti_patterns/deny-warnings.html)
and is causing Github CI breakage unnecessarily (https://github.com/facebookexperimental/eden/runs/2402094456):
error: function is never used: `example_blob`
--> src/lfs.rs:1860:8
|
1860 | fn example_blob() -> (Sha256, usize, Bytes) {
| ^^^^^^^^^^^^
|
note: the lint level is defined here
--> src/lib.rs:125:9
|
125 | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(dead_code)]` implied by `#[deny(warnings)]`
Reviewed By: andll
Differential Revision: D27911477
fbshipit-source-id: df8d642fe74fe311eb0f329d977b9b8270c27bf4
Summary:
In case the stderr is redirected but stdout is not:
EDENSCM_LOG=edenscm::mercurial=trace lhg log archival.py 2>/tmp/1
cat /tmp/1
The expected behavior is to still use the pager for stdout output, and the
stderr should be redirected to the specified file.
Reviewed By: andll
Differential Revision: D27867884
fbshipit-source-id: c369bc6be40fc200c4c0e2c9bb38b5faeb1208f2
Summary:
This will be useful to test if a (frequently called) callsite is enabled or not
without having the logging side effect.
Reviewed By: andll
Differential Revision: D27867672
fbshipit-source-id: 361cb18a7a4680932dcfc9d5496d2906e1dc1f9f
Summary:
The previous choice of the upload / download arrows do not render in Windows
`cmd.exe` terminal using common fonts:
- Cascadia Code
- Cascadia Mono
- Consolas
- Courier New
- JetBrains Mono
- Lucida Console
Replace them with tri-angles that can be rendered using the above fonts.
Note: newer terminals like Microsoft Terminal and WezTerm can render the characters
just fine.
Reviewed By: andll, ikostia
Differential Revision: D27894085
fbshipit-source-id: c53d995355e66ba793d80afc1b36fc83853d07f8
Summary:
Integration test showing that MismatchedHeads errors are sent over the wire to
the client.
Reviewed By: quark-zju
Differential Revision: D27798555
fbshipit-source-id: b14a213e9055486bf07ecbb4b5453385df701b48
Summary:
This error occurs when the client sends us a set of heads that we cannot match.
For example when the client forces a commit in the master group; this was
possible with revlogs but should be a bug with Segmented Changelog. This can
also happen when master moves backwards, clients and server have multiple heads
then the server reseeds.
Clients that get this error should reclone.
Reviewed By: quark-zju
Differential Revision: D27786602
fbshipit-source-id: 9854ccee929ae0a845236ebd83ed68158c93fc7a
Summary:
We want to distinguish between no location and failure to compute location.
It is useful to know on the client side if we failed to compute the location of
some hash or we are not sending it because we don't have a location for it.
We have different options for hash-to-location in particular but the problem is
a general one. Does it make sense for us to skip sending error at the EdenApi
handler layer? I don't think so, I think that's an old hask. We should be
evolving the EdenApi handler layer to handle application errors.
This is still at the exploration phase. I haven't fixed the client side to
handle the errors.
Reviewed By: quark-zju, kulshrax
Differential Revision: D27549926
fbshipit-source-id: 7094160fe997af0e69c8d006b9731ea7dfb3e3f8
Summary:
The goal here is to add an easy way of propagating application errors from
server to the client.
The most convenient form for an error message is a message string so we just
start with that. I think the most important thing to add is some way of
communicating whether the error is retryable or not.
With conversions for anyhow::Error to WireError and from WireError to
ServerError it should be trivial for application code to pass application
errors down to the client.
The format here is driven by the fact that we have streams in the response
object. A batch oriented format for responses has more options. For example
with batches it is common to have a response object that has 3 categories:
1. found: HashMap, 2. not_found: Vec, 3. errors: Vec/HashMap
Reviewed By: kulshrax
Differential Revision: D27549923
fbshipit-source-id: 33b790253adc4761ea9ac7caced4148f4026e620
Summary:
On Mac this introduces 150ms delays in every indexedlog flush. During
an amend of a stack with 32 commits, this fsync accounted for 2/3rds of the time
spent.
Since commitcloud is pretty reliable these days, I think this is no longer
necessary.
Reviewed By: andll
Differential Revision: D27896589
fbshipit-source-id: a13a494c54ffea5a670ed942b366620619af2bd0
Summary: The Python `[auth]` matching code does not take cert validity into account when performing certificate matching, whereas the Rust version of the code does. In practice, the existing call sites for the Rust code disable match-time validation, and instead validate the certificate at time-of-use. This diff makes the Rust code's behavior match Python so we can remove the latter entirely.
Reviewed By: DurhamG
Differential Revision: D27837343
fbshipit-source-id: 0bfb5ebc3a36c8fa748cb289e2d8d1495ba8b296
Summary:
Like it says in the title. We'd like to ask for the exact size that was
configured, because this way we can set the chunk size to the LFS threshold and
it avoids overlapping any file chunks server side.
Reviewed By: DurhamG
Differential Revision: D27824418
fbshipit-source-id: 43f40eb87080ec58e813ba1f1dda5b6a5e9f98ee
Summary:
This replicates behaviour of Python code - if unknown file content matches content of the file to be check out, do not abort checkout
This is useful for resuming interrupted checkouts / clones
Reviewed By: DurhamG
Differential Revision: D27799147
fbshipit-source-id: 7d2582583525da18fba08dfcd8bced2b619304de
Summary: It's been showing up while building mononoke. Let's fix it
Reviewed By: sfilipco
Differential Revision: D27789928
fbshipit-source-id: a15912f66b9ad3370545aed88405dbeb800e63de
Summary: This seems to have broken the EdenFS HgPrefetch test.
Reviewed By: xavierd
Differential Revision: D27795192
fbshipit-source-id: 80a748036961aa6a5750182bf65637fb76825341
Summary: This will show proper checkout progress when using native checkout
Reviewed By: quark-zju
Differential Revision: D27775423
fbshipit-source-id: 79f2afa02bd1fab7d5f747da1c714d4d1126ce7c
Summary:
EdenAPI makes heavy use of streaming HTTP responses consisting of a series of serialized CBOR values. In order to process the data in a streaming manner, we use the `CborStream` combinator, which attempts to deserialize the CBOR values as they are received.
`CborStream` hits a pathological case when it receives a very large CBOR value. Previously, it would always buffer the input stream into 1 MB chunks, and attempt to deserialize whenever a new chunk was received. In the case of downloading values that are >1GB in size, this meant potentially thousands of wasted deserialization attempts. In practice, this meant that EdenAPI would hang when receiving the content of large files.
To address this problem, this diff adds a simple heuristic: If a partial CBOR value exceeds the current buffer size, double the size threshold before attempting to deserialize again. This reduces the pathological case from `O(n^2)` to `O(log(n))` (with some caveats, described in the comment in the code).
Reviewed By: krallin
Differential Revision: D27759698
fbshipit-source-id: 67882c31ce95a934b96c61f1c72bd97cad942d2e
Summary:
Previously we'd skip dynamicconfigs when there wasn't a repo available.
Now that dynamicconfig can represent the NoRepo state, let's load dynamicconfigs
in that situation.
This also supports the case where there is no user name.
Reviewed By: kulshrax
Differential Revision: D26801059
fbshipit-source-id: 377cfffb6695a7fbe31303868a88862259ebf8c4
Summary: Add a new `edenapi.maxrequests` config option to allow controlling the number of parallel in-flight requests. This can be used to bound resource usage of the client when requesting large amounts of data.
Reviewed By: sfilipco
Differential Revision: D27724817
fbshipit-source-id: 8d607efa83d8b0b94074d1a6e06f6c536cc0c791
Summary: Add a method to allow setting `CURLMOPT_MAX_TOTAL_CONNECTIONS`, which limits the number of concurrent requests within a curl multi session. If the number of requests in the session exceeds this number, they will be queued and sent once earlier requests have completed.
Reviewed By: sfilipco
Differential Revision: D27724818
fbshipit-source-id: 436384aed9d6d29f426e5e45aebb7a72c24ba667
Summary:
Without this, `make local` will build `hostcaps` without fb-specific logic and
cause wrong configs being used. `hg up master` will error out like:
File "treemanifest/__init__.py", line 690, in _httpgetdesignatednodes
self.edenapi.trees(dpack, self.name, keys)
RustError: Server reported an error (403 Forbidden)
Reviewed By: quark-zju
Differential Revision: D27759821
fbshipit-source-id: d42895f44bc53003f2578b65640ebe4ee05d52e6
Summary:
Right now, if prefetch fails, we just give the client back an error saying
"content not found".
This isn't super helpful, because usually the reason the content is not found
is because we cannot talk to the server that has the content, so showing the
user why we cannot talk to said server is more useful.
I'd like to ship this gradually, so I also added a config flag to turn it off.
Initially I'll have the flag set, but I did default it to not-set in the code
so that our tests run with the desired configuration.
Note: I initially contemplated adding logging for this here, but after
discussion with xavierd it looks like just failing instead of eating the error
is probably a better approach (and it's much simpler). This is also consistent
with what EdenAPI does.
Reviewed By: mzr
Differential Revision: D27761572
fbshipit-source-id: 3506d9c97a00e3f076bd346883e07f49194b0b06
Summary:
Right now, if the server ever tells us a file is missing, we fail the entire
batch download. This is a bit unfortunate because other objects could still be
downloaded, but also because we lose the distinction between "server crashed"
and "server told us the data we want does not exist".
Besides, it's actually a bit unnecessary. Indeed, this fails, we just ignore
the error anyway up the stack, so it never actually gets accessed.
Reviewed By: mzr
Differential Revision: D27761574
fbshipit-source-id: cb4fb0526a3bf19c04ecb81c05d44d4d8afb81ad
Summary: We can just return the actual error here now.
Reviewed By: sfilipco
Differential Revision: D27761573
fbshipit-source-id: 0866f976b4ed434deffd96be6820ad05d27b7b93
Summary:
NOTE: The revisionstore LFS tests talk to prod Mononoke LFS, so the test here
will fail until that is deployed.
Like it says in the title, this adds support for downloading content in chunks
instead of all in one go. The underlying goal here is to let us do better load
balancing server side. This earlier diff in this stack explains that in more
detail: D27188610 (820b538001).
Reviewed By: quark-zju
Differential Revision: D27191594
fbshipit-source-id: 19b1be9bddc6b32b1fabfe4ab5738dfcf71b1a85
Summary: Introduce `FetchKey` and `FetchValue` traits to simplify repeated trait bounds in many `ReadStore` implementations. We also newly require `Clone` for both keys and values, which was already required by the fallback combinator.
Reviewed By: DurhamG
Differential Revision: D27652499
fbshipit-source-id: 6a3d5eb18a904b982fdb9946b80fcc9025d391ea
Summary:
Extend debugscmstore command to fetch arbitrary files / trees by key.
Replace debugpyscmstore with a python fallback for debugscmstore (allowing you to test with the store as it is constructed for Python, with legacy fallback).
Refactor some functionality so it is shared between the rust and python versions of debugscmstore.
Currently the output is pretty ugly. It uses the `{:#?}` format for everything. In the next change, I propose modifying the `Debug` implementation for `minibytes::Bytes` to use ascii-escaped bytestrings rather than the default slice formatter to make things much nicer.
This new `debugscmstore` functionality should be useful in integration tests for testing scmstore under different repo configurations, and for test harnesses and performance testing (fetch a specific set of things easily, simulate delays in the key stream by delaying the input pipe, etc).
Reviewed By: andll
Differential Revision: D27351321
fbshipit-source-id: 8650480e3f5b045b279472643570309c48d7fe6b
Summary: Like `FileScmStoreBuilder`, but for trees. As LFS is not used for trees, `TreeScmStoreBuilder` defaults to `ExtStoredPolicy::Use` (pass along anything you find without LFS-specific checks).
Reviewed By: DurhamG
Differential Revision: D27641290
fbshipit-source-id: 637340a23cef058e7e37a41ae7f5b4fcc9481190
Summary: Introduce a new `FileScmStoreBuilder` structured much like `ContentStoreBuilder`, but supporting the features needed for the intermingling of `contentstore` and `filescmstore` construction (shared indexedlog, scmstore fallback to contentstore).
Reviewed By: DurhamG
Differential Revision: D27640702
fbshipit-source-id: e9771e6f61d80698a9dd761a0db66407b565c010
Summary: This test fails without other diffs in stack because previously native checkout was overwriting untracked files
Reviewed By: DurhamG
Differential Revision: D27667151
fbshipit-source-id: 9b3aea37ba5c2d07fe4fc975dd40b4d7bea9d810
Summary:
The `add_heads_and_flush` method might add new nodes in the master group,
and it should update `overlay_map_next_id` accordingly. Without it, it
might error out like:
RustError: ProgrammingError: Server returned x~n (x = 9ebc9ebc49f1819767b40f9ceb22c37547a10c37 8459584, n = 1411).
But x exceeds the head in the local master group {}. This is not expected and indicates some logic error on the server side.
Full error: P387088806
Reviewed By: sfilipco
Differential Revision: D27637278
fbshipit-source-id: b45370db0561dec52cd513a12e2fd0110f18e0e5
Summary:
The idset is not fully backed by Rust and do not batch resolve vertexes.
The nameset is backed by Rust NameSet and has proper batch prefetching.
Use nameset if possible but fallback to idset if the backend is not in
Rust (rare, only used by hgsql repos now).
Reviewed By: sfilipco
Differential Revision: D27630092
fbshipit-source-id: cf847dd1a78bd5273a8928ecb6616fe11f2c7026
Summary:
While it is in theory "correct" going through the remote resolution even if the
protocol is "local". The overhead turns out to be something. And the tracing
message "resolve .. remotely" can be quite noisy. Let's just skip remote
resolutions early in IdConvert implementations.
Reviewed By: sfilipco
Differential Revision: D27630094
fbshipit-source-id: 7d87079876f040cf8f764f7362021dddba0d4723
Summary:
Currently the "contains vertex" check can trigger excessive
fetches for add_heads (and add_heads_and_flush used by flush).
Add a test to demonstrate the problem.
Reviewed By: sfilipco
Differential Revision: D27630091
fbshipit-source-id: ce3639c2a13226ba5681b4e8696edd7acbcb57f9
Summary:
Otherwise it can cause a lazy dag to think vertexes as "missing", insert
vertexes unnecessarily, and potentially break key graph properties (a
vertex should only have one Id).
Reviewed By: sfilipco
Differential Revision: D27629315
fbshipit-source-id: 1688d13cb94015bbc675613ecf9225556ff48372
Summary:
Also move related functions.
Similar to D27547584 (af3c3b3fd0), this allows `add_heads_and_flush` use `IdConvert`
on the `NameDag`, instead of the `IdMap` to trigger remote fetches properly.
This diff is easier to view with whitespace changes ignored.
Reviewed By: sfilipco
Differential Revision: D27629314
fbshipit-source-id: 8f79223c5d324aabfc5ab9813a9f65400fc533ec
Summary:
See the previous diff for context. Drop Locked and related APIs (prepare_filesystem_sync).
This makes it easier to use operate on a mut NameDag on flush because it does not need
to use separate types (Locked) for writing which has issues of not having the remote protocol
involved.
Reviewed By: sfilipco
Differential Revision: D27629306
fbshipit-source-id: 301445b242321ad5f424741ea91ebf6c075bff1c
Summary:
See the previous diff for context. Drop SyncableIdMap so we are one step
closer to using mut NameDag directly on add_heads, which knows when and how to
do remote fetching properly.
Reviewed By: sfilipco
Differential Revision: D27629310
fbshipit-source-id: 883606e40bb83907dfa6142ddd2c3030de61698f
Summary:
By using SyncableIdDag and SyncableIdMap, it's harder to use extra features
around them (ex. remote fetching). Drop SyncableIdDag so we are one step
closer to using mut NameDag directly on add_heads, which knows when and how to
do remote fetching properly.
Reviewed By: sfilipco
Differential Revision: D27629307
fbshipit-source-id: 8e9a5a4348a42b9751752b82feb3f3d2d7c4ba45
Summary:
The `Parents` trait is used for input of adding (non-lazy) vertexes to the
graph. The API will be used to provide extra hints to optimize network
fetches.
With the current logic, `assign_head` will ask the server to resolve the heads
first, to check if it is already assigned, then to resolve the parents, etc. to
the roots (in the "to assign" set). Ideally the `assign_head` logic can ask
the server to resolve the roots first, and if that's unassigned, then just mark
all descendants of the roots as unassigned, do not send more requests.
Note: the current pull logic has all the hashes ready (hashes are known).
But whether the hashes have Id assigned are unknown. It is more tricky
taking the "lock" and "reload" into consideration - hashes without Ids might
"become" having ids assigned after we obtain the lock to write data to disk.
Practically, `pull` using the current logic would take a very long time because
it tries to resolve things remotely for every "to assign" commits.
Reviewed By: sfilipco
Differential Revision: D27629317
fbshipit-source-id: e02f54f43ef65228ce6e3a8a8723dd9ae0a07008
Summary: This just simplifies the test code a bit.
Reviewed By: sfilipco
Differential Revision: D27629308
fbshipit-source-id: 04eac5cd045c71123e7fc410af74609dbadb8fb7
Summary: This avoids triggering remote lookups if an unknown name was looked up multiple times.
Reviewed By: sfilipco
Differential Revision: D27629316
fbshipit-source-id: 64c1ce5e872a5ce4f14c650a946ae8396f4cc74c
Summary:
When translating RequestNameToLocation to ResponseIdNamePair. If "heads" are
known, but some "names" aren't. Do not treat it as an error. This will be
used by the client-side to properly handle the "contains" check.
Reviewed By: sfilipco
Differential Revision: D27629309
fbshipit-source-id: 206ec5df956b33f4e816ab8d6a67ce776fd7bd74
Summary: This will make it easier to test client / server dags in upcoming changes.
Reviewed By: sfilipco
Differential Revision: D27629318
fbshipit-source-id: e3137654613aa3208a8f2e4b9f4ddfe73871f2c5
Summary: This will be used in upcoming changes. It just delegates to the Arc inner.
Reviewed By: sfilipco
Differential Revision: D27629313
fbshipit-source-id: ba6cd7cac2b9f5c1a2898c439c53768995a2dc42
Summary: This will be used by upcoming changes.
Reviewed By: sfilipco
Differential Revision: D27629312
fbshipit-source-id: 6c56e73caf4e1a398ac3a8b4294bd9f380af3764
Summary: This will be used by upcoming changes.
Reviewed By: sfilipco
Differential Revision: D27629319
fbshipit-source-id: d19e490268561f3154642e5bb1e415da4c5d03ee
Summary:
See the previous diff for context. A concrete HashMap can provide
"hint_pending_subdag". But a parent function cannot.
Reviewed By: sfilipco
Differential Revision: D27629311
fbshipit-source-id: 65168a8d00d9a672396312200016d6749f416d02
Summary: The lazy backend can now (partially) support the non-full IdMap segment clone.
Reviewed By: sfilipco
Differential Revision: D27581488
fbshipit-source-id: 51eded6acdbe82d22f5bb73eb4a715e2c22f4d75
Summary:
Make mutationstore more friendly to async.
This resolves an issue with smartlog with the lazy commit hash backend.
Reviewed By: sfilipco
Differential Revision: D27583844
fbshipit-source-id: 5b0b0b9b8ab82399f80eb2b410a0c4b84bd6a444
Summary:
Otherwise it might panic (ex. calling into tokio without entering a tokio
runtime). This can happen in, for example, `Debug::fmt(&IdStaticSet, ...)`.
Reviewed By: sfilipco
Differential Revision: D27581487
fbshipit-source-id: feec53e088706adcc6710afcf24fa70598f886cf
Summary:
This will be used by "add_heads" logic to detect what vertexes to insert
and might reduce remote fetches.
Reviewed By: sfilipco
Differential Revision: D27572359
fbshipit-source-id: d0bf027a69d180663a1587dfde613cb76b05072a
Summary: The API returns entries buffered in memory not persisted.
Reviewed By: sfilipco
Differential Revision: D27572360
fbshipit-source-id: 555988f7c891f2d928bfa1e486a0fc1d089b4ad5
Summary: This will be used to select "dirty" (not written to disk) set in the IdDag.
Reviewed By: sfilipco
Differential Revision: D27572361
fbshipit-source-id: 0b4d2e092ece835e3d4b6aa831d32ffffc7087ca
Summary:
Before this change, overlap IdMap was not considered for prefix lookup. That
results in "shortest" template not working and smartlog prints full hashes
for remote/stable etc.
Reviewed By: sfilipco
Differential Revision: D27547582
fbshipit-source-id: 7a56590775eed9d509f2212f8e5009c04aaf4e9d
Summary: It will be reused in NameDag.
Reviewed By: sfilipco
Differential Revision: D27547583
fbshipit-source-id: da85fc7504d20877210e8ed1a97cbec18d06eede
Summary:
Now NameSet iteration can be blocking, SyncNameSetQuery is no longer sound.
Remove SyncNameSetQuery in key logic (namedag and ops) and replace them with
async logic.
Reviewed By: sfilipco
Differential Revision: D27547581
fbshipit-source-id: af69b1a8219e97d10278939407ee79f9b333a77f
Summary: Dag algorithms like `parent_names` need to fetch vertexes via remote automatically.
Reviewed By: sfilipco
Differential Revision: D27547584
fbshipit-source-id: 8106931d6f642c9a4bf0f3c546ba881c2ca73fea
Summary: This crate does not panic on Windows.
Reviewed By: DurhamG
Differential Revision: D27640362
fbshipit-source-id: f50f6b8e0bd31e5f80fa939bcfb6846bc8fd4a63
Summary:
Recently we saw some progress rendering issues. Add a command to attempt to
reproduce them.
Reviewed By: DurhamG
Differential Revision: D27669184
fbshipit-source-id: 62fcf82d8261fd27e91ba5a116c61f4df1919007
Summary: This will be used later.
Reviewed By: skotchvail
Differential Revision: D27744058
fbshipit-source-id: 411ab66ccc38b306c6bffb190e936ba1e455f07a
Summary: This will make RotateLog achieve zero-copy reading more easily.
Reviewed By: kulshrax
Differential Revision: D27724331
fbshipit-source-id: 57915516dc6bd1935838bd099a60c104f0bdef3d
Summary: This makes it more flexible.
Reviewed By: kulshrax
Differential Revision: D27724332
fbshipit-source-id: 43ad670519f0617a97e0b7d38b374f497e9c01af
Summary:
This allows setting the wrapping mode. For example:
lhg log -pv
# copy paste long lines works.
lhg log -pv --config pager.wrapping-mode=unwrapped
# lines are not wrapped, ">" is shown for long lines.
lhg log -pv --config pager.wrapping-mode=word
# long lines wrapped at word level.
The default value matches "less" behavior.
Reviewed By: DurhamG
Differential Revision: D27720767
fbshipit-source-id: e29d6b13656407c0a1e63287fb96e2f8d914cfc8
Summary: This method checks if any of files that checkout writes is not tracked in hg and exists on disk
Reviewed By: DurhamG
Differential Revision: D27667153
fbshipit-source-id: 4ad8bc08520678ea0b51008ed14fb51ca4a98f76
Summary:
`hostcaps` abstracts the logic for determining whether we have a prod or corp
environment.
Reviewed By: DurhamG
Differential Revision: D27684641
fbshipit-source-id: 50df9a60b6a613b4cb5c9aed6cad2844aae85a6f
Summary:
We want to use it in Mercurial and the directory structure was playing bad
with Mononoke's OSS build.
Reviewed By: xavierd
Differential Revision: D27684642
fbshipit-source-id: 8827645eee58fa671f9c9e1964a34c34e3a8eeb6
Summary:
This diff introduces CheckoutPlan::apply_read_store to apply checkout plan using ReadStore as a data source
This requires some minor changes in apply_stream flow as ReadStore does not guarantee ordering of returned files
Reviewed By: DurhamG
Differential Revision: D27658346
fbshipit-source-id: 5a289554d8dd7b6bb4b5a996659cd0661779ad5f
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641665
fbshipit-source-id: d46f62f9067eb9cb4c8517a62efa6f663d4b6732
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641669
fbshipit-source-id: d907407f5a6e868862fe37f1f67fbe99ee378156
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641667
fbshipit-source-id: adce5a39fcb5d8e8d5d989fed46991e20ab3710d
Summary: Provides a way to read config with lighter dependencies.
Reviewed By: DurhamG
Differential Revision: D27641668
fbshipit-source-id: fc99a78f5f51e63f61d1b049af74f61f5d1916a3
Summary:
The `configparser` is now too heavyweight. Some other crates (ex. io, auth,
revisionstore) just want to extract config values without complicated parsing /
remote hotfix logic.
Add a configmodel crate to satisfy the need.
Reviewed By: DurhamG
Differential Revision: D27641666
fbshipit-source-id: 26bd0b606ae3d286b3ec218927aef726d6802c63
Summary: Add a new `http.verbose` config option that turns on verbose output for libcurl (similar to the output printed by `curl -v`). This can be very useful for debugging HTTP issues.
Reviewed By: DurhamG
Differential Revision: D27693304
fbshipit-source-id: 2ad7a08889f40ffbcd2f14ac9c21d70433629da4
Summary:
`const_cstr::ConstCStr` is represented internally as a fat pointer with fixed size: `&'static str`. See https://docs.rs/const-cstr/0.3.0/const_cstr/struct.ConstCStr.html. Notably this is **different** from the representation of `std::ffi::CStr`, which is a dynamically sized type and normally passed around behind a reference, as `&CStr`. Using `&ConstCStr` in signatures, which is effectively like `&'a &'static CStr`, is confusing due to the discrepancy between the two relatedly named types. Additionally having two different lifetimes involved -- the static lifetime of the underlying bytes, and the short lifetime of the fat pointer -- is unnecessarily confusing when async code and a language boundary are involved.
The utf8-cstr crate uses what seems like a better representation to me than the const-cstr crate. See https://docs.rs/utf8-cstr/0.1.6/utf8_cstr/struct.Utf8CStr.html. `Utf8CStr` is the dynamically sized type, just like `CStr`. Then `&'static Utf8CStr` is how it would commonly be passed around, just like `&CStr`.
Reviewed By: krallin
Differential Revision: D27698169
fbshipit-source-id: ffe172c2c2fc77aeab6b0a0a8aed3e3c196098cc
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the client code to thread the list of master commits to the
EdenApi client.
Reviewed By: quark-zju
Differential Revision: D27523868
fbshipit-source-id: db4148a3f1d0e8b0b162e0ecc934e07f041c5511
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the request object for HashToLocation to send over all the
master heads.
When the server builds non-master commits, we will want to send over non-master
heads too. We may consider having one big list of heads but I think that we
would still want to distinguish the non-master commit case in order to optimize
that use-case.
Reviewed By: quark-zju
Differential Revision: D27521778
fbshipit-source-id: cc83119b47ee90f902c186528186ad57bf023804
Summary:
`panic!()`, and things which use `panic!()` like `assert!()`, take a literal format
string, and whatever parameters to format. There's no need to use `format!()`
with it, and it is incorrect to pass a non-literal string.
Mostly it's harmless, but there are some genuinely confusing asserts which
trigger this warning.
Reviewed By: dtolnay
Differential Revision: D27672891
fbshipit-source-id: 73929cc77c4b3d354bd315d8d4b91ed093d2946b
Summary:
Modify the `Debug` implementation for `minibytes::Bytes` to use `std::ascii::escape_default` to debug print a `Bytes` as an ascii-escaped bytestring.
For comparison, the `bytes` crate `Bytes` type provides the same functionality, though it doesn't use the standard library `escape_default` function: https://docs.rs/bytes/1.0.1/src/bytes/fmt/debug.rs.html#39-43
This change greatly improves the output of the `debugscmstore` command. If we don't want to make this the default behavior, we can provide a formatting wrapper type or I can specialize the output in `debugscmstore`, but I can't see any real downsides, especially given the `bytes` crate does the same thing, and we have a similar specialization for `HgId` (hex format in that case).
Reviewed By: quark-zju
Differential Revision: D27642721
fbshipit-source-id: 8faba421fa5082a2098b13ef7d286e05eccb6400
Summary: Add the `with_key` function to `Entry`, which replaces it's key with a provided key. Currently, scmstore returns incorrect results when multiple entries exist with different paths but the same HgId (as scmstore directly returns the path found on disk locally). This isn't a problem in the legacy API, which returns a bare `Vec<u8>` content, which is implicitly associated with the requesting key because it is the result of a single `get` call, or is irrelevant because the `prefetch` method doesn't directly return the results.
Reviewed By: andll
Differential Revision: D27664025
fbshipit-source-id: 014d44ca9a1dc2721685622fd2b077ed3483838f
Summary:
Those are not used. Recently we saw build issues like:
lib/third-party/sha1dc/sha1.c:8:10: fatal error: string.h: No such file or directory
#include <string.h>
^~~~~~~~~~
Possibly by some compiler flags disabling stdlib. Since we don't need
the C code let's just remove them.
Reviewed By: singhsrb
Differential Revision: D27659634
fbshipit-source-id: b8fcac4f05be95cac470f9b4830baf76e06f98ad
Summary:
The "filter" set's filter function might not be prepared for inputs outside
the parent set. So let's the "contains" function to test against the parent
set first, then test the filter function.
This fixes the "merge()" set's "contains" check using the revlog backend:
In [1]: v=repo.revs('draft() & merge()')
In [2]: v._set
Out[2]: <meta ?>
In [3]: m.node.nullid in v._set
Out[3]: False
Before this diff it would be:
In [3]: m.node.nullid in v._set
CommitLookupError: '0000000000000000000000000000000000000000 cannot be found'
Note: Segmented changelog backend is not affected because it does not use
filter sets.
Reviewed By: xavierd
Differential Revision: D27657502
fbshipit-source-id: 30bb261fea59bdc5644580e98796f52fa93c2705
Summary: We were accidentally not setting the TLS key here; this diff fixes it.
Reviewed By: quark-zju
Differential Revision: D27634276
fbshipit-source-id: 9aac3a34b6f6655059a8d3332eea8ba02d062651
Summary:
This diff contains proposal for basic rebase model for new checkout.
This diff introduces two types of file changes - basic no conflict Actions(that will replace CheckoutPlan actions) and ConflictAction for actions that contain conflicts
No conflict actions are produced when comparing two manifests
Conflict actions can be produced when rebasing manifests.
Eventually we will adjust checkout code to take either list of Actions or Conflict actions and perform corresponding operations on the file.
Rebase::rebase is used to generate list of conflict actions when rebasing.
Currently `Rebase::rebase` produces list of actions on top of 'base' commit.
Before doing actual checkout we will need to take into account what current commit is and adjust this list of actions accordingly (however, this adjustment will be trivial, it won't produce more conflicts)
After getting adjusted list of actions, checkout code can roll it out to fs and call merge drivers if needed.
In the future, if we will want to allow check in conflicts, we will need to adjust `Rebase::resolve_conflict` to take `ConflictAction` as input, instead of an Action.
Reviewed By: DurhamG
Differential Revision: D27028001
fbshipit-source-id: b02b7ad8030fcf146b53639d69e66fdc8f38a62d
Summary:
Only used by one test that can define the constaint itself.
The problem with having it on the trait is that it's a bit noisy when
things operate on ToApi at the generic level. It adds to the list of
constaints they these users of the ToApi trait need to add.
Reviewed By: kulshrax
Differential Revision: D27549922
fbshipit-source-id: fff9e513eb4c06862111ce6eecc84ab981eea893
Summary:
This is only used in one utility which can define the constaint itself.
I am looking to simplify the Requirements for ToWire so that we can more
easily iterate on them. Debug as a requirement produces too much noise.
There is the risk of ending up in a scenario where we want to print the Wire
type but it's more practical to annotate structs with derive Debug when that
happens than to add the constaint in the trait.
Reviewed By: kulshrax
Differential Revision: D27549925
fbshipit-source-id: aacf7c1c465c94414be02aa143187897c7084980
Summary:
There is no use for it outside of one test which can describe that constraint
itself.
I think that requiring ToWire and ToApi to use the same objects is too much
for the general use case. We regularly convert between different object types
that are the same logical thing but have different representations. A good
example for that is the HgId. It makes sense to implement ToWire for all HgId
variations.
Reviewed By: kulshrax
Differential Revision: D27549924
fbshipit-source-id: d76d7a4beb528634bed46ae93dbd634d850547e5
Summary:
For async requests, we perform a blocking request in a separate thread, and stream the results back through a channel. However, if the curl handle for the request is dropped before starting the request (for example, because of a configuration error), this function would return a `oneshot::Canceled` error (from the channel consumer) instead of the real error message from the IO thread.
This diff fixes the issue by ensuring that the function waits for and returns the error message from the IO thread in the event that the IO thread returns before starting the request.
Reviewed By: quark-zju
Differential Revision: D27584502
fbshipit-source-id: 8447c158d253c3f28f03fcc4c36a28698fe6e83d
Summary:
Since we're rolling out native checkout and resumable checkout around
the same time, let's make resumable checkout optional so we can turn it off it
causes issues, without turning off native checkout.
Reviewed By: quark-zju
Differential Revision: D27481986
fbshipit-source-id: a0a3e68567ca2a468e852dce95c03c4b606aaf22
Summary: This makes it easier to use.
Reviewed By: kulshrax
Differential Revision: D27406589
fbshipit-source-id: 11bef407ab620859381c6ee952e6ef00494551e1
Summary:
The issue is that `mut i: usize` is no longer shared across multiple `async
move` blocks (introduced by D27308798 (0df4efa969)).
Rewrite the logic to collect `ids` first, then use `vertex_name_batch`
query instead.
Reviewed By: sfilipco
Differential Revision: D27406586
fbshipit-source-id: b41fe3a13114dc34aa5763e6e2bebe0571decc87
Summary:
Merge paths like `x~n` and `x~(n+1)` to `x~n (batch_size = 2)`.
This could be more efficient bandwidth-wise and algorithm-wise.
Reviewed By: sfilipco
Differential Revision: D27406587
fbshipit-source-id: f2a67352ad627945685e33667e8299a2bc652930
