Summary: We're going to remove the fields. Those add noise to the serialized data.
Reviewed By: sfilipco
Differential Revision: D26504916
fbshipit-source-id: 7fd13bdab4511ceea50195595514fb89a4169419
Summary:
By default it spawns `nproc` threads, which is an overkill on hosts with many cores.
Data shows the max CPU busy % for tokio threads is 30%, avg is 0.1%.
Testing framework will spawn `nproc` tests, meaning it's `nproc ^ 2` threads, which
seems too many.
The number 8 is picked to match the checkout worker count (worker.numcpus), to
ensure checkout tasks won't have bottleneck on the count of tokio threads doing
work like parsing manifest etc.
Reviewed By: sfilipco
Differential Revision: D26437442
fbshipit-source-id: 4dccef83294e64c432ef5ab4be259d41dd890ece
Summary: This is most universal/flexible interface
Reviewed By: quark-zju
Differential Revision: D26410641
fbshipit-source-id: cae74675f89c0c1c05c84331c3ce0e78fd0b0b2f
Summary: The latest libc has fixes for compiling on the M1 mac, let's update it.
Reviewed By: dtolnay
Differential Revision: D26476625
fbshipit-source-id: d9a997e69c428d53c51fc52353289a6510314c50
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary:
Make the struct and method names in this crate more clearly reflective of what they do:
- `Auth` -> `AuthGroup`
- `Auth::try_from` -> `AuthGroup::new`
- `AuthConfig` -> `AuthSection`
- `AuthConfig::new` -> `AuthSection::from_config`
- `AuthConfig::auth_for_url` -> `AuthSection::best_match_for`
Reviewed By: singhsrb
Differential Revision: D26436095
fbshipit-source-id: a5ec5d9c48d3b75a0ee166b74e5340f9c529eeae
Summary: Make `AuthConfig::auth_for_url` match the behavior of the Python `httpconnection.readauthforuri` function when an auth group has a URL prefix with a scheme. Previously, the Rust version did not allow schemes to be specified on the prefix, and instead required them to be specified in the "schemes" fields. The Python code allows a scheme in the prefix, which overrides the contents of the "schemes" field for that auth group if present.
Reviewed By: DurhamG
Differential Revision: D26419721
fbshipit-source-id: 909b52a2e37e2fc908d5eb56740dd41dda826033
Summary:
Move these conversion related function and trait out of `hg` module so EdenFS can use it too. Changes:
* Moved `get_opt`, `get_or` and `get_or_default` directly into `ConfigSet`.
* Moved `FromConfigValue` and `ByteCount` into `configparser::convert`.
Reviewed By: quark-zju
Differential Revision: D26355403
fbshipit-source-id: 9096b7b737bc4a0cccee1a3883e89a323f864fac
Summary: The Python `readauthforuri` method will include *any* fields specified in the `[auth]` section for a given auth group, even if those fields aren't one of the expect ones (such as `cert`, `key`, etc). This is sometimes used in tests to attach additional information to an auth group. To support this in Rust, let's just collect all unknown fields into a `HashMap`.
Reviewed By: quark-zju
Differential Revision: D26416086
fbshipit-source-id: 0252e340e38850a54e24d54810e9abd77c566f63
Summary:
Add a new `HttpClientError::Tls` variant specifically for TLS errors, separating them from other `curl::Error`s from libcurl.
To determine whether a particular `curl::Error` is a TLS error, we check both the error code and the contents of the error message.
Reviewed By: quark-zju
Differential Revision: D26385845
fbshipit-source-id: fd58f86a3a61fcfb845d19e262fdcb132dc7ec9f
Summary: This diff adds auto-generated test cases to checkout code. It generates partially overlapping trees and tests transitions between them
Reviewed By: quark-zju
Differential Revision: D26384962
fbshipit-source-id: 6140bbb7ff8b87843a2235f8325f57829cdd8cae
Summary: Currently PathComponentBuf::arbitrary generates any characters. Those characters are ok for unit tests on hg abstractions where they are currently used, but many of them do not work with real filesystems
Reviewed By: quark-zju
Differential Revision: D26384961
fbshipit-source-id: dde1e9276114b30262bc477a3e0f828645f1f32a
Summary:
Currently if VFS overwrites executable file with regular, it preserves exec bit[see added test].
This diff makes sure that file has correct permissions after overwrite
This diff also slightly optimizes write_executable, by calling set_mode on the file handle, instead of path
TODO - we can check if calling stats() before set_permissions will save some time
Reviewed By: quark-zju
Differential Revision: D26379824
fbshipit-source-id: 42d0b2fb79ed860ac37b2de077388002ade69449
Summary:
Before this diff VFS::write_regular did not handle correctly use case when file already existed as as symlink - it would write into symlink location, instead of replacing symlink with a regular file (see updated test_symlink_overwrite that is failing on old implementation)
This diff adds O_NOFOLLOW option on unix when overwriting the file. When destination is a symlink, attempt to write fails with E_LOOP and triggers clear_conflict that removes symlink and allows retry write to succeed.
This also allows one of test cases in checkout that previously did not work
Reviewed By: quark-zju
Differential Revision: D26378893
fbshipit-source-id: 28bcdaba78db283ac7a25bb232c198d3d8f73e5d
Summary: This diff contains basic test setup for checkout tests - we compare transition between two trees without dirty changes
Reviewed By: quark-zju
Differential Revision: D26359502
fbshipit-source-id: ef670c944200bae1652863c91ada92c6fecce4ac
Summary:
One of the primary use cases for hash_to_location is translating user provided
hashes. It is then perfectly valid for the hashes that are provided to not
exist. Where we would previously return an error for the full request if a
hash was invalid, we now omit the hash from the response.
Reviewed By: quark-zju
Differential Revision: D26389472
fbshipit-source-id: c59529d43f44bed7cdb2af0e9babc96160e0c4a7
Summary:
The approach is very similar to what commitrevlogdata does. You could say
that it's cargo culted.
I am not sure how appropriate it is to return CommitLocationToHashResponse
but I think that it's fine for now.
Reviewed By: quark-zju
Differential Revision: D26374219
fbshipit-source-id: 61d851d5a4fc4223c65078ef434a0c67314a90cd
Summary:
We've rolled both of these out to 100%. Let's make this the default so
we can delete those configs.
Reviewed By: quark-zju
Differential Revision: D26233645
fbshipit-source-id: cd7a08c404483f78ab714763870f5bf0fa801e7a
Summary:
We're seeing cases where Mercurial is creating unused .tmp pack files
and leaving them around. It looks like there are two places this can happen, 1)
in the treemanifest python code we manually instantiate some mutable packs, and
2) when doing a read from the rust data store, when it does a read against the
mutable pack store it will unnecessarily create the MutableDataPackInner struct,
which creates the temp file.
Let's fix those.
Reviewed By: quark-zju
Differential Revision: D26387205
fbshipit-source-id: 5a567c886849084bcc8121949dd2fb0f9e66d570
Summary:
In our upcoming migration away from chef/static rc files, we'll be
marking certain files as "allowed". Our hope is that that list only includes
things like .hg/hgrc, ~/.hgrc, etc.
There are cases however where it's convienent to continue to use chef, for
instance when we condition on machine type. To support this, let's add an
allowed_config option, which will allow configs from non-supported locations.
This will also be useful when remediating issues that come up when we start
enforcing allow_location, without rolling back the entire thing.
Reviewed By: quark-zju
Differential Revision: D26233451
fbshipit-source-id: 71789e0361923a6f80de4aef7f012afc0269440d
Summary:
The backingstore crate depends on mio which depends on ntdll. It's not entirely
clear to me why we need to do this manually and why cargo/cmake doesn't pick it
up automatically, but let's fix it this way for now.
Reviewed By: genevievehelsel
Differential Revision: D26376606
fbshipit-source-id: 26714b3f03aabefafdf48c7fb0f442cad501a058
Summary:
Previously the `IdDag` struct has max_level caching. With the previous diff the
cache was gone. Re-implement it on the indexedlog IdDag to maintain
performance. This has visible performance wins:
Before:
building segments 115.949 ms
ancestors 93.072 ms
children (spans) 495.732 ms
children (1 id) 10.384 ms
common_ancestors (spans) 3.567 s
descendants (small subset) 25.829 ms
gca_one (2 ids) 258.997 ms
gca_one (spans) 3.718 s
gca_all (2 ids) 440.764 ms
gca_all (spans) 3.732 s
heads 322.552 ms
heads_ancestors 67.567 ms
is_ancestor 165.046 ms
parents 304.392 ms
parent_ids 11.765 ms
range (2 ids) 21.466 ms
range (spans) 18.663 ms
roots 471.934 ms
After:
benchmarking dag::iddagstore::indexedlog_store::IndexedLogStore
building segments 103.177 ms
ancestors 68.485 ms
children (spans) 451.383 ms
children (1 id) 9.817 ms
common_ancestors (spans) 3.096 s
descendants (small subset) 24.845 ms
gca_one (2 ids) 201.458 ms
gca_one (spans) 3.185 s
gca_all (2 ids) 357.899 ms
gca_all (spans) 3.239 s
heads 295.462 ms
heads_ancestors 50.991 ms
is_ancestor 146.798 ms
parents 296.667 ms
parent_ids 12.305 ms
range (2 ids) 7.781 ms
range (spans) 17.630 ms
roots 478.574 ms
Reviewed By: sfilipco
Differential Revision: D26360564
fbshipit-source-id: 51f55a5bb4e69321515e02f45545618320c1bce5
Summary:
Previously, algorithms are defined on `IdDag<Store>`, which requires a type
parameter to use. Move them to be defined directly on `Store` so using the
algorithms no longer require a type parameter. This will make it easier
to write code that calls algorithms on different IdDagStores.
Reviewed By: sfilipco
Differential Revision: D26360561
fbshipit-source-id: 8e0faf741019c4ed4119ad8e754aea9057b31866
Summary: It is not used. The definitions were moved to dag-types.
Reviewed By: sfilipco
Differential Revision: D26360562
fbshipit-source-id: 35e672194918e3f35294d69cad9e6990d8921900
Summary:
This allows us to "fork" the iterator so we can "peek ahead" multiple items,
without affecting the original iterator.
Reviewed By: sfilipco
Differential Revision: D26360566
fbshipit-source-id: 4cba280e64338b20fb3e1584609be8fda9b3d616
Summary:
Implement Iterator::{size_hint,count,last} for potential fast paths (ex.
Collect into a Vec).
Reviewed By: sfilipco
Differential Revision: D26360565
fbshipit-source-id: 227d9c5e615c2a0a624ba88d6d4c3f81b10d7795
Summary:
Benchmark code is out of sync. Fix it.
Example run:
benchmarking dag::iddag::IdDag<dag::iddagstore::in_process_store::InProcessStore> serde
serializing inprocess iddag with mincode 0.664 ms
mincode serialized blob has 707565 bytes
deserializing inprocess iddag with mincode 42.477 ms
Reviewed By: sfilipco
Differential Revision: D26360563
fbshipit-source-id: f87e7ad53e6b6dadecaa0976e1c61f0399814104
Summary:
Benchmark code is out of sync. This is an important benchmark.
Example run:
benchmarking dag::iddagstore::indexedlog_store::IndexedLogStore
building segments 111.814 ms
ancestors 67.953 ms
children (spans) 484.954 ms
children (1 id) 12.599 ms
common_ancestors (spans) 3.337 s
descendants (small subset) 27.979 ms
gca_one (2 ids) 216.430 ms
gca_one (spans) 3.297 s
gca_all (2 ids) 371.049 ms
gca_all (spans) 3.348 s
heads 303.232 ms
heads_ancestors 52.821 ms
is_ancestor 149.525 ms
parents 294.633 ms
parent_ids 12.173 ms
range (2 ids) 7.612 ms
range (spans) 16.991 ms
roots 459.869 ms
benchmarking dag::iddagstore::in_process_store::InProcessStore
building segments 68.869 ms
ancestors 6.683 ms
children (spans) 175.711 ms
children (1 id) 2.061 ms
common_ancestors (spans) 408.220 ms
descendants (small subset) 6.990 ms
gca_one (2 ids) 16.983 ms
gca_one (spans) 411.237 ms
gca_all (2 ids) 27.921 ms
gca_all (spans) 415.704 ms
heads 110.486 ms
heads_ancestors 5.228 ms
is_ancestor 11.223 ms
parents 108.636 ms
parent_ids 0.746 ms
range (2 ids) 1.539 ms
range (spans) 5.885 ms
roots 172.910 ms
benchmarking NameDag with many heads
range (master::draft) 55.400 ms
range (recent_draft::drafts) 12.439 ms
Reviewed By: sfilipco
Differential Revision: D26360567
fbshipit-source-id: 6d3244e3f4655634c239f84a7304540860a7d34a
Summary:
This endpoint is used by the lazy Segmented Changelog to understand the
location of commit described by hashes. For example users may say that they
want to check out an older commit by hash. The client would then call this
endpoint to understand the relationship of the destination commit relative to
the graph.
Reviewed By: quark-zju
Differential Revision: D26289622
fbshipit-source-id: 4bbfd4bd4f91c984384fff5a6480b8d9d77cf8d3
Summary:
Get the graph location of a given commit identifier.
The client using segmented changelog will have only a set of identifiers for
the commits in the graph. The client needs a way to translate user input to
data that it has locally. For example, when checking out an older commit by
hash the client will have to retrieve a location to understand the place in the
graph of the commit.
Reviewed By: quark-zju
Differential Revision: D26289623
fbshipit-source-id: 4192d91a4cce707419fb52168c5fdff53ac3a9d0
Summary: If a URL's path does not end in a trailing slash (e.g., https://example.com/foo/bar.html), `Url::join` will strip off the final path component under the assumption that it is a filename. This is problematic in the case of EdenAPI base URLs, since the endpoints might be under some base URL (e.g., https://example.com/edenapi/v1). This change ensures the final path component will not be stripped off in such cases.
Reviewed By: quark-zju
Differential Revision: D26353232
fbshipit-source-id: 2ec3dc56fd6d5a0ccaf522fcfc34e6741ec95c68
Summary: Allow different type of storages for CheckoutPlan::apply
Reviewed By: quark-zju
Differential Revision: D26291577
fbshipit-source-id: a1c9ba4dbef09e844727ae32eac25c37dd01358e
Summary:
Previously, `write` can fail because the destination file exists as a
directory, or the parent directory is missing. pyworker handles those cases
by calling `clear_conflicts` to remove conflicted directories and create
missing parent directories and retry `write`. Practically, for all `write`
usecases (including checkout) we always want the `clear_conflicts` behavior.
Therefore, move `clear_conflicts` to vfs `write` and make it private.
Reviewed By: quark-zju
Differential Revision: D26257829
fbshipit-source-id: 03d1da0767202edba61c47ae5654847c0ea3b33e
Summary: Currently we pick simple approach and just delete existing file before re-creating it if there is some non-trivial flag change
Reviewed By: quark-zju
Differential Revision: D26239977
fbshipit-source-id: 167efa1bf6018e7f967ef3a9e3c8c62781486ec9
Summary:
This is removing `edenapi::CommitLocation` in order to use
`dag_types::Location`.
First, `edenapi::CommitLocation` has a `count` field and `dag_types::Location`
does not. I find `count` to be difficult to attach to a more general structure.
In practice `edenapi::CommitLocation` is replaced by `CommitLocationToHashRequest`.
On top of the request we have the batch object: `CommitLocationToHashRequestBatch`.
Second, `edenapi::CommitLocation` did not have Wire types, the new structures do.
Reviewed By: quark-zju
Differential Revision: D26159865
fbshipit-source-id: f2508e123e11988726868c7f439a2ed186afce5c
Summary: Used when the IdMap is lazy to fetch missing locations.
Reviewed By: quark-zju
Differential Revision: D26131617
fbshipit-source-id: cde0232b16afb961d9c9a18899ca78bd644f1b6b
Summary:
I think that it makes sense to standardize on importing:
`dag_types::Id` rather than `dag::Id` now that this is a crate of its own.
I am not sure about re-exporting `minibytes::Bytes` but it makes sense to me.
Reviewed By: quark-zju
Differential Revision: D26131616
fbshipit-source-id: fefd0334cf188f247b1541be16421967e8340546
Summary:
Wire types has it's own meaning in Edenapi. I don't see it necessary to
add the wire qualifier to this crate and overload the term.
Reviewed By: quark-zju
Differential Revision: D26129827
fbshipit-source-id: eea66eef2db609611d8ffa215ba63ae4f0b669c8
Summary:
Introduce `FallbackStore`, a newstore combinator which implements the `ReadStore` trait and first checks a "preferred" store for the provided keys, before falling back to a "fallback" store. The combinator requires that both stores share the same `Key` type, but allows a user provided "value adapter function" to convert from the fallback store's value type to the preferred store's value type.
Currently `FallbackStore` does not support writing missing values to the preferred store - this functionality will be introduced in a future change.
Reviewed By: DurhamG
Differential Revision: D26203146
fbshipit-source-id: 0e99110f93130ff30c95cce15e3dc7873616519c
Summary: Introduce a new stream combinator, `select_drop`, based on `futures::stream::select`, which performs the same function except dropping the contained streams when they terminate, rather than when the combined stream is exhausted. This prevents a deadlock in cases where one stream will not terminate until the other terminates.
Reviewed By: DurhamG
Differential Revision: D26139816
fbshipit-source-id: d28da32244a215741476e1c3882154ea9e3116a5
Summary: Introduce a minimal version of an async, typed `ReadStore` trait and corresponding `EdenApi` implementation, along with a debug command, `debugnewstore` to exercise it.
Reviewed By: DurhamG
Differential Revision: D26050641
fbshipit-source-id: 2c14c3715e7067f9ecd1e649e6ca146a1ce249bf
