Summary:
Move the logic to a common method. A side effect is that
IndexedlogIdDag needs to calculate `last_span.high` again, and
InProcessIdDag needs to calculate `last_store_id` again. But it's
probably okay since the merge is rare and the InProcessIdDag operations should
be considered fast even if it's called frequently.
Reviewed By: sfilipco
Differential Revision: D26504915
fbshipit-source-id: e215ffe9fef7948b7b662dba3ed990e374be989a
Summary: Add a few more tests to the generic IdDag tests.
Reviewed By: sfilipco
Differential Revision: D26415514
fbshipit-source-id: 5ef2a1d4e03527b184a07d94de91c64a05427b90
Summary: It is a constant `true` now. Inline the logic and remove the field.
Reviewed By: sfilipco
Differential Revision: D26414838
fbshipit-source-id: ceace835ecfd263ba16c3dead41ce6ba95087e4f
Summary:
Similar to the previous diff. Implement the merge behavior for InProcess IdDag.
The feature is not turned on yet for easier review. It is turned on by the
next diff.
Reviewed By: sfilipco
Differential Revision: D26414839
fbshipit-source-id: 2a485e0dffcd9d8e2ad95380159d11342636f9aa
Summary:
Merge adjacent flat segments to reduce fragmentation.
The merging is done by a special entry that instructs:
- Remove the old segment from index.
- Insert the new segment.
Note: For the parent -> child index we record parent -> child id, not parent ->
segment. So that index does not need to be updated, because the merged segment
has the same "parent -> child id" information.
Note: The merging is a bit expensive (multiple lookups). But most segments are
not mergeable, because their parents are not "span.low - 1". So the slow path
is not hit often.
The feature is not enabled by default in this diff, because it will break IdDag
tests, which assume indexedlog and in-memory IdDags behave the same, and we
haven't updated in-memory IdDag.
Reviewed By: sfilipco
Differential Revision: D26414841
fbshipit-source-id: 6a268bef56b7a56c62bb8ad949f13e503a88b033
Summary:
With previous refactorings the `max_level` is now maintained by the underlying
IdDag stores. Remove the `max_level` on the wrapper IdDag to simplify the
logic.
Reviewed By: sfilipco
Differential Revision: D26414840
fbshipit-source-id: 084a7f6f75fd53103c45519190607607fef4e161
Summary: We're going to remove the fields. Those add noise to the serialized data.
Reviewed By: sfilipco
Differential Revision: D26504916
fbshipit-source-id: 7fd13bdab4511ceea50195595514fb89a4169419
Summary:
By default it spawns `nproc` threads, which is an overkill on hosts with many cores.
Data shows the max CPU busy % for tokio threads is 30%, avg is 0.1%.
Testing framework will spawn `nproc` tests, meaning it's `nproc ^ 2` threads, which
seems too many.
The number 8 is picked to match the checkout worker count (worker.numcpus), to
ensure checkout tasks won't have bottleneck on the count of tokio threads doing
work like parsing manifest etc.
Reviewed By: sfilipco
Differential Revision: D26437442
fbshipit-source-id: 4dccef83294e64c432ef5ab4be259d41dd890ece
Summary:
Fix for error:
can only concatenate str (not "NoneType") to str
Reviewed By: quark-zju
Differential Revision: D26523879
fbshipit-source-id: 3d15d72be3444df25fb705ce444d6c3621547551
Summary:
Remove selectivepullaccessedbookmarks feature because it contains bugs and causes many undesired issues.
This was added to migrate existing repos to selective pull and is not needed anymore.
Main effects are:
* if you enable selectivepull for an existing repo, it won't reduce number of subscribed bookmarks.
* some operations like `hg push` or `hg pull -r` in their underlying implementation update all subscribed bookmarks, not just accessed like before.
This drives changes to the tests. Reminder, the bookmark has been marked as "accessed" if the repo has been ever updated to that bookmark.
All tests fixed accordingly.
Reviewed By: markbt
Differential Revision: D26460435
fbshipit-source-id: f839b9f207bfc478a0336ec807b720d35a0bb12e
Summary:
This was added in D26387205 (ca1249c269) to make the treemanifest store usage look
more like the file store usage, but it appears to cause read-after-write
failures. I'm not sure exactly why, but I think it's because something takes a
reference to the old store and writes to it, then something else reads from the
new store. Or vice versa.
Either way, reverting these three lines fixes it.
Reviewed By: singhsrb
Differential Revision: D26497708
fbshipit-source-id: e9afabed8d5f800260e647d5ed1513eed0bc60cd
Summary: This diff introduces a Python checkoutplan, which will be used later for Python code to interact with the Rust checkout infrastructure
Reviewed By: DurhamG
Differential Revision: D26437080
fbshipit-source-id: fe3517f4e2f15244a7c2cd290a41394e904db51b
Summary: This diff exposes treemanifest for public use and introduce method to borrow underlyning manifest for usage in rust
Reviewed By: DurhamG
Differential Revision: D26437079
fbshipit-source-id: d2d6cd2a4d1f108881d985932a36d48fed4b238d
Summary: This is most universal/flexible interface
Reviewed By: quark-zju
Differential Revision: D26410641
fbshipit-source-id: cae74675f89c0c1c05c84331c3ce0e78fd0b0b2f
Summary: The latest libc has fixes for compiling on the M1 mac, let's update it.
Reviewed By: dtolnay
Differential Revision: D26476625
fbshipit-source-id: d9a997e69c428d53c51fc52353289a6510314c50
Summary:
Sort works differently on OSX. Let's set the locale in this case to get
consistent sorting across platforms.
Reviewed By: quark-zju
Differential Revision: D26438678
fbshipit-source-id: 80a5c758795fd9675b8b2b658521c0e9d6807366
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary:
Make the struct and method names in this crate more clearly reflective of what they do:
- `Auth` -> `AuthGroup`
- `Auth::try_from` -> `AuthGroup::new`
- `AuthConfig` -> `AuthSection`
- `AuthConfig::new` -> `AuthSection::from_config`
- `AuthConfig::auth_for_url` -> `AuthSection::best_match_for`
Reviewed By: singhsrb
Differential Revision: D26436095
fbshipit-source-id: a5ec5d9c48d3b75a0ee166b74e5340f9c529eeae
Summary:
In a non-segmented-chanelog repo, `last`, the revset function has suboptimal
path that makes the following query:
last(::b8f917d4c & public())
take a long time. Use `max` to fix it.
Reviewed By: singhsrb
Differential Revision: D26435439
fbshipit-source-id: dc854b4c6978d2925cab2cb8f217365ed9c7f146
Summary: Make `AuthConfig::auth_for_url` match the behavior of the Python `httpconnection.readauthforuri` function when an auth group has a URL prefix with a scheme. Previously, the Rust version did not allow schemes to be specified on the prefix, and instead required them to be specified in the "schemes" fields. The Python code allows a scheme in the prefix, which overrides the contents of the "schemes" field for that auth group if present.
Reviewed By: DurhamG
Differential Revision: D26419721
fbshipit-source-id: 909b52a2e37e2fc908d5eb56740dd41dda826033
Summary:
Move these conversion related function and trait out of `hg` module so EdenFS can use it too. Changes:
* Moved `get_opt`, `get_or` and `get_or_default` directly into `ConfigSet`.
* Moved `FromConfigValue` and `ByteCount` into `configparser::convert`.
Reviewed By: quark-zju
Differential Revision: D26355403
fbshipit-source-id: 9096b7b737bc4a0cccee1a3883e89a323f864fac
Summary:
The push-pull test was broken by my recent change to make indexedlog
the default. It wasn't caught because the test is disabled in Python 3. It now
passes with Python 3 so let's enable it.
The repack-remove fix is just some flakiness I've seen lately.
Reviewed By: singhsrb
Differential Revision: D26412998
fbshipit-source-id: be4b648f31bd6dfbf6a6e5d2e382acb084461974
Summary: Add Python bindings to the Rust `auth` crate, with the intention of replacing `httpconnection.readauthforuri`.
Reviewed By: quark-zju
Differential Revision: D26419447
fbshipit-source-id: dd13bea74961137790beb8c96120ebef99e3c313
Summary: The Python `readauthforuri` method will include *any* fields specified in the `[auth]` section for a given auth group, even if those fields aren't one of the expect ones (such as `cert`, `key`, etc). This is sometimes used in tests to attach additional information to an auth group. To support this in Rust, let's just collect all unknown fields into a `HashMap`.
Reviewed By: quark-zju
Differential Revision: D26416086
fbshipit-source-id: 0252e340e38850a54e24d54810e9abd77c566f63
Summary:
The auth crate is now able to check the presence and expiration of client certificates (D26009207 (9f7d4447fd)). When a problem is detected, it emits an `X509Error`, which specifies exactly what the problem is. Since this error always indicates a certificate issue, we can print out the message configured in `help.tlsauthhelp` (which is more specific than `help.tlshelp` from the previous diff).
Previously, Mercurial would attempt to use the certificate anyway, resulting in a difficult to understand error message. Although the previous diffs in this stack improved the error messages on any TLS failure, the `X509Error` messages are even more helpful.
Users can opt in to this certificate validation with `edenapi.validate-certs`. The functionality is gated on a config option to prevent Mercurial from crashing if certificates are misconfigured, but EdenAPI isn't being used.
Reviewed By: quark-zju
Differential Revision: D26385843
fbshipit-source-id: 9809f612f8aab3f2dd442d6dd8dc348f1af45296
Summary: Print out the help text configured in `help.tlshelp` upon hitting a `TlsError`. Note that in this case, we use `help.tlshelp` rather than `help.tlsauthhelp` since all we know in this case is that *some* kind of TLS error occurred.
Reviewed By: quark-zju
Differential Revision: D26385844
fbshipit-source-id: 1fb5280195de75107ecdfc9203ef8ddda2a04052
Summary: Add a new `TlsError` Python exception type corresponding to `HttpClientError::Tls`.
Reviewed By: quark-zju
Differential Revision: D26385846
fbshipit-source-id: c0df543032461de650a4d24c26c6b8aaab1abbb9
Summary:
Add a new `HttpClientError::Tls` variant specifically for TLS errors, separating them from other `curl::Error`s from libcurl.
To determine whether a particular `curl::Error` is a TLS error, we check both the error code and the contents of the error message.
Reviewed By: quark-zju
Differential Revision: D26385845
fbshipit-source-id: fd58f86a3a61fcfb845d19e262fdcb132dc7ec9f
Summary: This diff adds auto-generated test cases to checkout code. It generates partially overlapping trees and tests transitions between them
Reviewed By: quark-zju
Differential Revision: D26384962
fbshipit-source-id: 6140bbb7ff8b87843a2235f8325f57829cdd8cae
Summary: Currently PathComponentBuf::arbitrary generates any characters. Those characters are ok for unit tests on hg abstractions where they are currently used, but many of them do not work with real filesystems
Reviewed By: quark-zju
Differential Revision: D26384961
fbshipit-source-id: dde1e9276114b30262bc477a3e0f828645f1f32a
Summary:
Currently if VFS overwrites executable file with regular, it preserves exec bit[see added test].
This diff makes sure that file has correct permissions after overwrite
This diff also slightly optimizes write_executable, by calling set_mode on the file handle, instead of path
TODO - we can check if calling stats() before set_permissions will save some time
Reviewed By: quark-zju
Differential Revision: D26379824
fbshipit-source-id: 42d0b2fb79ed860ac37b2de077388002ade69449
Summary:
Before this diff VFS::write_regular did not handle correctly use case when file already existed as as symlink - it would write into symlink location, instead of replacing symlink with a regular file (see updated test_symlink_overwrite that is failing on old implementation)
This diff adds O_NOFOLLOW option on unix when overwriting the file. When destination is a symlink, attempt to write fails with E_LOOP and triggers clear_conflict that removes symlink and allows retry write to succeed.
This also allows one of test cases in checkout that previously did not work
Reviewed By: quark-zju
Differential Revision: D26378893
fbshipit-source-id: 28bcdaba78db283ac7a25bb232c198d3d8f73e5d
Summary: This diff contains basic test setup for checkout tests - we compare transition between two trees without dirty changes
Reviewed By: quark-zju
Differential Revision: D26359502
fbshipit-source-id: ef670c944200bae1652863c91ada92c6fecce4ac
Summary:
Update the dirsync code to allow mirror and exclude rules to match individual
files rather than just directory prefixes.
This simply appends `/` to all filenames when looking for rule matches. This
allows us to efficiently match individual filenames in addition to directory
prefixes.
Reviewed By: quark-zju
Differential Revision: D26294583
fbshipit-source-id: 83b283f344f6e0bc0fe53b9068e7e0170f53504b
Summary:
One of the primary use cases for hash_to_location is translating user provided
hashes. It is then perfectly valid for the hashes that are provided to not
exist. Where we would previously return an error for the full request if a
hash was invalid, we now omit the hash from the response.
Reviewed By: quark-zju
Differential Revision: D26389472
fbshipit-source-id: c59529d43f44bed7cdb2af0e9babc96160e0c4a7
Summary:
The approach is very similar to what commitrevlogdata does. You could say
that it's cargo culted.
I am not sure how appropriate it is to return CommitLocationToHashResponse
but I think that it's fine for now.
Reviewed By: quark-zju
Differential Revision: D26374219
fbshipit-source-id: 61d851d5a4fc4223c65078ef434a0c67314a90cd
Summary: This is defined else where in Python 3, so let's get rid of it here.
Reviewed By: quark-zju
Differential Revision: D26381054
fbshipit-source-id: 9746d2c53f83209d9c795ffdd5841d58ef1153ef
Summary: These are just a few one-liners to fix Python 3 on Windows.
Reviewed By: sfilipco
Differential Revision: D26381055
fbshipit-source-id: d9257f2cf35c05f931d74b7d26bdc79f5bf34185
Summary:
We've rolled both of these out to 100%. Let's make this the default so
we can delete those configs.
Reviewed By: quark-zju
Differential Revision: D26233645
fbshipit-source-id: cd7a08c404483f78ab714763870f5bf0fa801e7a
Summary:
We're seeing cases where Mercurial is creating unused .tmp pack files
and leaving them around. It looks like there are two places this can happen, 1)
in the treemanifest python code we manually instantiate some mutable packs, and
2) when doing a read from the rust data store, when it does a read against the
mutable pack store it will unnecessarily create the MutableDataPackInner struct,
which creates the temp file.
Let's fix those.
Reviewed By: quark-zju
Differential Revision: D26387205
fbshipit-source-id: 5a567c886849084bcc8121949dd2fb0f9e66d570
Summary:
In our upcoming migration away from chef/static rc files, we'll be
marking certain files as "allowed". Our hope is that that list only includes
things like .hg/hgrc, ~/.hgrc, etc.
There are cases however where it's convienent to continue to use chef, for
instance when we condition on machine type. To support this, let's add an
allowed_config option, which will allow configs from non-supported locations.
This will also be useful when remediating issues that come up when we start
enforcing allow_location, without rolling back the entire thing.
Reviewed By: quark-zju
Differential Revision: D26233451
fbshipit-source-id: 71789e0361923a6f80de4aef7f012afc0269440d
Summary:
Copied from D17312417 (e1f4dbeb3d), because that did what I needed done, but incremented by 1.
I would like to change the length of the displayed hash in scm-prompt
to 9. Why such an impactful diff? Because hg sl shows 9 characters, and I
always get confused when the hash in my prompt doesn't match hg sl
Reviewed By: kulshrax
Differential Revision: D25934253
fbshipit-source-id: 15f2bc8bc7d666de1a077d2bafd74ab3c9753341
