Summary:
The future has a "map" clause that takes GIL too. If they run in different
threads it could deadlock. For example, the following code might hang
forever:
In [1]: s,f=api.commitdata('fbsource',list(repo.nodes('master')))
In [2]: f.wait()
Reviewed By: sfilipco
Differential Revision: D26582806
fbshipit-source-id: e3259850c68b8d48a7a69ed9d47ef75f26179382
Summary:
Previously, one has to set both EDENSCM_LOG=trace and EDENSCM_TRACE_LEVEL=trace
to enable all logging. That is because the filtering is global - one subscriber
(or layer) filtering out a span or event, then the span or event is gone
forever.
For now, let's just set the TracingCollector Subscriber's filter level
(EDENSCM_TRACE_LEVEL) to TRACE so it solely depends on the EnvFilter
(EDENSCM_LOG) if EDENSCM_LOG is set. That's more friendly.
See `impl<L: Layer<S>, S: Subscriber> Subscriber for Layered<L, S>`:
fn enabled(&self, metadata: &Metadata<'_>) -> bool {
if self.layer.enabled(metadata, self.ctx()) {
// if the outer layer enables the callsite metadata, ask the subscriber.
self.inner.enabled(metadata)
} else {
// otherwise, the callsite is disabled by the layer
false
}
}
See also: https://github.com/tokio-rs/tracing/issues/302
Reviewed By: sfilipco
Differential Revision: D26518017
fbshipit-source-id: 9016b940621fc9a883e6ca3f00eaaeccc051ae74
Summary:
Now we use tracing_subscriber for equivalent features (by setting EDENSCM_LOG).
It's no longer necessary to forward to log eco-system.
Reviewed By: sfilipco
Differential Revision: D26518018
fbshipit-source-id: 7cc080faa734ca40eed1ce250df3135ca2626c1f
Summary: Now IO has internal locking, `mut` is no longer needed.
Reviewed By: sfilipco
Differential Revision: D26518025
fbshipit-source-id: d71e213f6fed24fdefb4c730d576f6cf2fb09d82
Summary:
Cloning IO would make it harder to run the clean-up logic in Drop (ex. flush,
stop the pager). Forbid it.
The weak reference output and error streams can still be cloned. They sastify
the main "clone" use-cases.
Reviewed By: sfilipco
Differential Revision: D26538451
fbshipit-source-id: 6cc59214ec5fe1340cb27c7ee1a09658bb83190a
Summary:
Similar to D26518021 (3aba8d89b0), provide a way to obtain the "output" stream
that implements `std::io::Write`.
The "output" stream is then used to replace the `std::io::Write`
implementation on the `IO` itself.
This has 2 benefits:
- Removes the `&mut` methods on `IO`. So all `&mut IO` can be replaced by `&IO` without exceptions.
- Make it possible to drop `Clone` on `IO` (strong ref is not clonable, but weak ref can). So we can control the lifetime of the streams more predicatably.
Reviewed By: sfilipco
Differential Revision: D26538452
fbshipit-source-id: b00c65ae0ef5ef4a609104111706365028f47ef7
Summary:
Use the global IO state instead of keeping a separate IO state.
This makes the Python pager logic able to affect the tracing logger's error
output. For example, the following command will write the tracing logs to
the streampager's stderr channel as expected:
EDENSCM_LOG=trace lhg log -r '.~1000::.' --config pager.pager=internal:streampager
Traditional pager also works:
EDENSCM_LOG=trace lhg log -r '.~1000::.' --config pager.pager=less
Reviewed By: sfilipco
Differential Revision: D26518024
fbshipit-source-id: c7020b7467f9392af8c2a58d064339d05bce35ce
Summary:
This makes the IOError stream used by tracing loggers use weak reference
so the main IO can still be dropped without being affected by potentially
lingering IOError values.
Reviewed By: sfilipco
Differential Revision: D26518022
fbshipit-source-id: df4c847a7a59fff39b7832190a8723c78ffae9ad
Summary:
Set the global IO reference to the IO created by hgmain. This allows
other crates like `pypager` to use the `IO` without cloning or passing
the reference across layers (Rust references do not actually work for
across the Python boundary).
Reviewed By: sfilipco
Differential Revision: D26518027
fbshipit-source-id: 6144b06035d0ef9384cc1a37245c306071e35a9d
Summary:
This makes it easier to figure out the "main" IO, similar to APIs like
`std::io::stdout`, without passing the IO around. It will be used by
the `pypager` module.
The global reference is a weak reference so it won't affect IO drop
behavior.
Reviewed By: sfilipco
Differential Revision: D26518015
fbshipit-source-id: ab75ec31cbb8f156b72c94a6fe5b2de6fe9e0bff
Summary: This will be used by pypager.
Reviewed By: sfilipco
Differential Revision: D26518026
fbshipit-source-id: ac8290501ea572cc5ce2067a52d62f4933cbe87b
Summary:
I encountered the error but it's unclear what happened. Print type_name to
clarify.
Now I get:
not implemented: converting non-stdio Write (alloc::vec::Vec<u8>) from Rust to Python is not implemented
This turns out to be an error in "clidispatch: make IO clonable" - The `Drop`
should be implemented on `Inner`, instead of `IO`. The fix has been folded
into that commit.
Reviewed By: sfilipco
Differential Revision: D26518016
fbshipit-source-id: ddb5f828c59cbffa9767694558113167bc96db4b
Summary: Done some reordering of fields in Cargo.toml, added test and doctest = false, name of the target that generated the Cargo.toml file and sorted the cratemap.
Reviewed By: ahornby
Differential Revision: D26581275
fbshipit-source-id: 4c363369438c72d43d8ccf4799f103ff092457cc
Summary:
Allow color effects to be joined with `+`, e.g. `blue+bold`. Unlike effects
separated by spaces, these effects must all be available in order for the
effect to apply. If any of the effects are not available, then the combined
effect is not valid.
This is useful in fallback chains, where some of the fallbacks are combined
effects, for example: `brightred:red+bold`.
Use these to improve the fallback chains for sparse profiles, so that the
profile names are not shown in black on 8 color terminals.
Reviewed By: quark-zju
Differential Revision: D26580365
fbshipit-source-id: 8e204c901566ac03fb4af066ab1a09142750a9fb
Summary:
Some features in debushell, like `%time` uses `locals()`. They currently cannot
use common variables like `cl` like:
In [1]: %time len(cl)
---------------------------------------------------------------------------
NameError Traceback (most recent call last)
<timed eval> in <module>
NameError: name 'cl' is not defined
Fix it by updating `locals()`.
In [1]: %time len(cl)
CPU times: user 700 µs, sys: 130 µs, total: 830 µs
Wall time: 1.28 ms
Reviewed By: ikostia
Differential Revision: D26590943
fbshipit-source-id: 2b7fd3274f1ea41f7996edbbdae0af8e103a9c0a
Summary:
The EdenFS import helper's `_fetch_tree_impl` method works by either calling `repo._prefetchtrees` or `repo.prefetchtrees`, both of which are methods from `treemanifest` extension's `treerepository` class. Unfortunately, these methods are lower-level than their names would suggest, and will always perform gettreepack style fetching (see [1]).
If HTTP fetching is enabled, this means that EdenFS will always query EdenAPI's `/complete_trees` endpoint instead of `/trees`, which is needlessly expensive given that EdenFS just wants the exact set of trees specified. As a somewhat hacky workaround, this diff just checks whether HTTP fetching is enabled, and if so, directly calls the appropriate HTTP fetch method.
This solution isn't ideal; it would be better for the import helper to request the trees from Mercurial's `remotetreestore` instead of via methods from `treerepository`. Unfortunately, with the current structure of Mercurial's storage layer, the `remotetreestore` isn't readily accessible because it gets passed into the Rust API's storage hierarchy upon construction.
---
[1]: The various `*prefetchtrees` methods are usually called from Mercurial's `remotetreestore`, which is where the choice of tree fetching strategy is made (i.e., designated nodes vs gettreepack). The `remotetreestore` then calls `*prefetchtrees` for gettreepack-style fetching, or `*getdesignatednodes` for on-demand fetching. As such, a call to `*prefetchtrees` generally implies gettreepack-style fetching.
Reviewed By: quark-zju
Differential Revision: D26560451
fbshipit-source-id: 2eedf50a6e66fac78df77214b777544eb8049714
Summary: This code has been used to transfer an existing repo to remotenames. After D26460435 (84280e36c3) it doesn't work as designed and should be removed as well.
Reviewed By: quark-zju
Differential Revision: D26544739
fbshipit-source-id: 34c08d4b9997c0b1f298ee1ecd0e8af24f4d8a39
Summary:
Support for unsubscribing from a scratch remote bookmarks in `hg hide` command.
This is support if you hide via a revision. Hiding by its name will be another change.
Reviewed By: quark-zju
Differential Revision: D26544305
fbshipit-source-id: d10372513dda88903e2cc031ff16883a001c8e34
Summary:
The `commit` or `amend` command might not have been the first item in the
command list (e.g. there could have been a custom config option). Expand the
heuristics for detecting the `commit` and `amend` cases to account for this.
Reviewed By: quark-zju
Differential Revision: D26545339
fbshipit-source-id: a5b1fc8ccc87989e742fce1fa79273266892ed79
Summary: Dynamic config only gets applied once the local repo is instantiated, but source peer is created long before that. This causes it to have missing configuration when being instantiated. Apply dynamic config much earlier in case of a clone so that srcpeer can use dynamicconf as well.
Reviewed By: DurhamG
Differential Revision: D26543635
fbshipit-source-id: 88c16d345f5116bfaaf57c593eb09145df81b4fb
Summary:
This diff ensures no one can exploit mkscratch's scratch directory. In reality probably no one would do this but just to add it as a precaution.
And also the renaming, I didn't land the rename changes in my previous diff and this diff fix that so we don't roll the "nested" mode name out.
Reviewed By: xavierd
Differential Revision: D26526702
fbshipit-source-id: 04ec883e4d1c0c3e9b54b6274e5113c8b4845b3f
Summary:
This warning was printed by old checkout, keeping it in nativecheckout too
The warning tells user that current working directory was removed during checkout
This diff also makes test-update-names.t to use nativecheckout. Otherwise last test fails on remote repo, because pyworker does not emit the deleted cwd warning
Reviewed By: quark-zju
Differential Revision: D26558188
fbshipit-source-id: 1f6ea2ea1ac7358ce2f06fed25069656481b30e6
Summary: After this update, test-update-names.t tests are using native checkout when congfig is set
Reviewed By: quark-zju
Differential Revision: D26558186
fbshipit-source-id: f70f65344b5f2209f313e3edd5fd7f541318459a
Summary: Those shortcuts can be used to setup remote repo via single line
Reviewed By: quark-zju
Differential Revision: D26558187
fbshipit-source-id: c6fd48ed38cc4dbaad4db714c4dfd76ec26bf608
Summary: This is needed to correctly show status after checkout
Reviewed By: quark-zju
Differential Revision: D26551119
fbshipit-source-id: 3ab576df8132b1fb8cccf507717923ecf3084757
Summary:
This diff makes CheckoutPlan::apply take &self instead of self, so that CheckoutPlan can be preserved.
This result in cloning of some paths(but realistically, they probably get moved anyway in this code path, so this should not have perf impact)
This will simplify pycheckout as it will allow to keep plan and later use it to update dirstate
This diff also exposes accessors for dirstate update (e.g. removed_files / updated_content_files / updated_meta_files)
Reviewed By: quark-zju
Differential Revision: D26551118
fbshipit-source-id: 4e2d02da0a692345eae7d5e7124663c577e9029c
Summary:
Currently native checkout works only on simplest cases when there is no merge / rebase and working copy is clean
Additionally, all native checkout logic is gated with experimental.nativecheckout config
Reviewed By: quark-zju
Differential Revision: D26501903
fbshipit-source-id: 2c1b2fd303f1cb1f461d4854ea60a7274ae38732
Summary:
Make the status fast path print warnings about non-utf8 names and continue on
other valid utf-8 names. It matches the behavior of the Python status code
path.
Before this change, an invalid utf-8 name would just print:
abort: invalid utf-8 sequence of 1 bytes from index 7
without telling which file it is.
Reviewed By: markbt
Differential Revision: D26553514
fbshipit-source-id: 6e26a2a8d738e9001d878a80c0dc82ae6d18a97c
Summary:
Similar to D26142175 (324f47b1f0), initialize the logger that writes to the error stream of
IO.
Reviewed By: sfilipco
Differential Revision: D26518014
fbshipit-source-id: 2547f4a8f3dba1f624fd00798672f1148869e250
Summary:
Add an API to obtain an `io::Write` object for the error stream.
It will be used by next changes.
Reviewed By: sfilipco
Differential Revision: D26518021
fbshipit-source-id: bc39fa54f83be8ff2c8fbc803700c4fe57e814bd
Summary:
Make it possible for IO to be cloned. This allows, for example, tracing or
progress bar logic to keep a reference of IO.
Reviewed By: sfilipco
Differential Revision: D26518028
fbshipit-source-id: d5ca7f4347251c68f7491f784f647ee80c47d159
Summary:
We're going to tweak the internals of IO. Making its fields private to allow
that.
Reviewed By: sfilipco
Differential Revision: D26518023
fbshipit-source-id: a1eed46f4f87a3b4b63e56be733102058b4d72d6
Summary: Add parking_lot dep. It will be used later.
Reviewed By: sfilipco
Differential Revision: D26518019
fbshipit-source-id: 76124a2f70731b30017a6d7019eef9dc856fe54a
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary: Introduce `FetchError` type, with separates "not found" errors (which require a key), from other errors (which may or may not have an associated key). This allows us to easily fall back for only "not found" errors propagate other fetch errors to the caller. The current version of `FetchError` still does not eliminate redundantly storing the key for cases like EdenApi, where the key may be part of the error type itself. This optimization may not be worthwhile after we refactor these store implementations to use `HgId` instead of `Key`.
Reviewed By: kulshrax
Differential Revision: D26410215
fbshipit-source-id: e6198d54de64b41ff696cabd64affc8dbaa41cf9
Summary: Add a wrapper type, `LegacyDatastore`, along with a `ReadStore` implementation that reads via the `HgIdDataStore` trait, for any wrapped types which implement that trait.
Reviewed By: kulshrax
Differential Revision: D26381207
fbshipit-source-id: d28dc3095adf20403f8b993d5a939d1e41c77462
Summary:
Added a new `ReadStore` implementation for EdenApi `FileEntry`s. This is pretty much identical to the `TreeEntry` implementation, but calls the `files` method instead, which does not yet support attributes or per-entry errors (although they are both supported on the wire).
I re-use the type-agnostic indexedlog `Entry` adapter for file support for now, to keep things minimal. This unfortunately means the output type of the `Fallback` combinator is a bare `Entry` for both files and trees, losing us some type safety, but I think this is acceptable for now, until I eventually introduce type-safe constructors and value types. Making the file and tree value output types distinct will only be necessary with the introduction of attribute support, or when we'd like to move away from using an "opaque blob" representation for them.
Reviewed By: kulshrax
Differential Revision: D26348577
fbshipit-source-id: 1a930af6e4b5b4d8bacd266c3e51db96cee92dbd
Summary:
Added write support to the `FallbackStore` combinator, along with a flag to control writing to the write store. This enables the "preferred" store to act as a read-through cache for the "fallback" store, with an optional "read only" mode via the "write" flag.
Currently, errors in the write path are silently ignored.
Reviewed By: kulshrax
Differential Revision: D26233602
fbshipit-source-id: acce020a877b87bfee87763e984923c7c51b2b17
Summary: Introduces a minimal WriteStore trait, with a basic implementation for IndexedLog. Currently, there's no explicit key in the `WriteStream`. Instead, it is implicit that the value type to be written should contain a key. In the future, we may introduce an abstraction over value types which do and do not contain a key.
Reviewed By: kulshrax
Differential Revision: D26213028
fbshipit-source-id: 75867ff7229d594346e7e7624e8ca91f8205fed4
Summary: This helps with pipelining (once chunk is prefetched it will start rolling out to fs) and also helps with limiting number of keys prefetched at once
Reviewed By: quark-zju
Differential Revision: D26496789
fbshipit-source-id: 9aa6b08c1605ab2a06a02347897f5dcfa22297fd
Summary: Also removing todo: test from checkout plan as there are tests already
Reviewed By: singhsrb
Differential Revision: D26496790
fbshipit-source-id: eb99771cb0cda7a90477190875997027b25e2482
Summary:
Move the logic to a common method. A side effect is that
IndexedlogIdDag needs to calculate `last_span.high` again, and
InProcessIdDag needs to calculate `last_store_id` again. But it's
probably okay since the merge is rare and the InProcessIdDag operations should
be considered fast even if it's called frequently.
Reviewed By: sfilipco
Differential Revision: D26504915
fbshipit-source-id: e215ffe9fef7948b7b662dba3ed990e374be989a
Summary: Add a few more tests to the generic IdDag tests.
Reviewed By: sfilipco
Differential Revision: D26415514
fbshipit-source-id: 5ef2a1d4e03527b184a07d94de91c64a05427b90
Summary: It is a constant `true` now. Inline the logic and remove the field.
Reviewed By: sfilipco
Differential Revision: D26414838
fbshipit-source-id: ceace835ecfd263ba16c3dead41ce6ba95087e4f
Summary:
Similar to the previous diff. Implement the merge behavior for InProcess IdDag.
The feature is not turned on yet for easier review. It is turned on by the
next diff.
Reviewed By: sfilipco
Differential Revision: D26414839
fbshipit-source-id: 2a485e0dffcd9d8e2ad95380159d11342636f9aa
Summary:
Merge adjacent flat segments to reduce fragmentation.
The merging is done by a special entry that instructs:
- Remove the old segment from index.
- Insert the new segment.
Note: For the parent -> child index we record parent -> child id, not parent ->
segment. So that index does not need to be updated, because the merged segment
has the same "parent -> child id" information.
Note: The merging is a bit expensive (multiple lookups). But most segments are
not mergeable, because their parents are not "span.low - 1". So the slow path
is not hit often.
The feature is not enabled by default in this diff, because it will break IdDag
tests, which assume indexedlog and in-memory IdDags behave the same, and we
haven't updated in-memory IdDag.
Reviewed By: sfilipco
Differential Revision: D26414841
fbshipit-source-id: 6a268bef56b7a56c62bb8ad949f13e503a88b033
Summary:
With previous refactorings the `max_level` is now maintained by the underlying
IdDag stores. Remove the `max_level` on the wrapper IdDag to simplify the
logic.
Reviewed By: sfilipco
Differential Revision: D26414840
fbshipit-source-id: 084a7f6f75fd53103c45519190607607fef4e161
Summary: We're going to remove the fields. Those add noise to the serialized data.
Reviewed By: sfilipco
Differential Revision: D26504916
fbshipit-source-id: 7fd13bdab4511ceea50195595514fb89a4169419
Summary:
By default it spawns `nproc` threads, which is an overkill on hosts with many cores.
Data shows the max CPU busy % for tokio threads is 30%, avg is 0.1%.
Testing framework will spawn `nproc` tests, meaning it's `nproc ^ 2` threads, which
seems too many.
The number 8 is picked to match the checkout worker count (worker.numcpus), to
ensure checkout tasks won't have bottleneck on the count of tokio threads doing
work like parsing manifest etc.
Reviewed By: sfilipco
Differential Revision: D26437442
fbshipit-source-id: 4dccef83294e64c432ef5ab4be259d41dd890ece
Summary:
Fix for error:
can only concatenate str (not "NoneType") to str
Reviewed By: quark-zju
Differential Revision: D26523879
fbshipit-source-id: 3d15d72be3444df25fb705ce444d6c3621547551
Summary:
Remove selectivepullaccessedbookmarks feature because it contains bugs and causes many undesired issues.
This was added to migrate existing repos to selective pull and is not needed anymore.
Main effects are:
* if you enable selectivepull for an existing repo, it won't reduce number of subscribed bookmarks.
* some operations like `hg push` or `hg pull -r` in their underlying implementation update all subscribed bookmarks, not just accessed like before.
This drives changes to the tests. Reminder, the bookmark has been marked as "accessed" if the repo has been ever updated to that bookmark.
All tests fixed accordingly.
Reviewed By: markbt
Differential Revision: D26460435
fbshipit-source-id: f839b9f207bfc478a0336ec807b720d35a0bb12e
Summary:
This was added in D26387205 (ca1249c269) to make the treemanifest store usage look
more like the file store usage, but it appears to cause read-after-write
failures. I'm not sure exactly why, but I think it's because something takes a
reference to the old store and writes to it, then something else reads from the
new store. Or vice versa.
Either way, reverting these three lines fixes it.
Reviewed By: singhsrb
Differential Revision: D26497708
fbshipit-source-id: e9afabed8d5f800260e647d5ed1513eed0bc60cd
Summary: This diff introduces a Python checkoutplan, which will be used later for Python code to interact with the Rust checkout infrastructure
Reviewed By: DurhamG
Differential Revision: D26437080
fbshipit-source-id: fe3517f4e2f15244a7c2cd290a41394e904db51b
Summary: This diff exposes treemanifest for public use and introduce method to borrow underlyning manifest for usage in rust
Reviewed By: DurhamG
Differential Revision: D26437079
fbshipit-source-id: d2d6cd2a4d1f108881d985932a36d48fed4b238d
Summary: This is most universal/flexible interface
Reviewed By: quark-zju
Differential Revision: D26410641
fbshipit-source-id: cae74675f89c0c1c05c84331c3ce0e78fd0b0b2f
Summary: The latest libc has fixes for compiling on the M1 mac, let's update it.
Reviewed By: dtolnay
Differential Revision: D26476625
fbshipit-source-id: d9a997e69c428d53c51fc52353289a6510314c50
Summary:
Sort works differently on OSX. Let's set the locale in this case to get
consistent sorting across platforms.
Reviewed By: quark-zju
Differential Revision: D26438678
fbshipit-source-id: 80a5c758795fd9675b8b2b658521c0e9d6807366
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary:
Make the struct and method names in this crate more clearly reflective of what they do:
- `Auth` -> `AuthGroup`
- `Auth::try_from` -> `AuthGroup::new`
- `AuthConfig` -> `AuthSection`
- `AuthConfig::new` -> `AuthSection::from_config`
- `AuthConfig::auth_for_url` -> `AuthSection::best_match_for`
Reviewed By: singhsrb
Differential Revision: D26436095
fbshipit-source-id: a5ec5d9c48d3b75a0ee166b74e5340f9c529eeae
Summary:
In a non-segmented-chanelog repo, `last`, the revset function has suboptimal
path that makes the following query:
last(::b8f917d4c & public())
take a long time. Use `max` to fix it.
Reviewed By: singhsrb
Differential Revision: D26435439
fbshipit-source-id: dc854b4c6978d2925cab2cb8f217365ed9c7f146
Summary: Make `AuthConfig::auth_for_url` match the behavior of the Python `httpconnection.readauthforuri` function when an auth group has a URL prefix with a scheme. Previously, the Rust version did not allow schemes to be specified on the prefix, and instead required them to be specified in the "schemes" fields. The Python code allows a scheme in the prefix, which overrides the contents of the "schemes" field for that auth group if present.
Reviewed By: DurhamG
Differential Revision: D26419721
fbshipit-source-id: 909b52a2e37e2fc908d5eb56740dd41dda826033
Summary:
Move these conversion related function and trait out of `hg` module so EdenFS can use it too. Changes:
* Moved `get_opt`, `get_or` and `get_or_default` directly into `ConfigSet`.
* Moved `FromConfigValue` and `ByteCount` into `configparser::convert`.
Reviewed By: quark-zju
Differential Revision: D26355403
fbshipit-source-id: 9096b7b737bc4a0cccee1a3883e89a323f864fac
Summary:
The push-pull test was broken by my recent change to make indexedlog
the default. It wasn't caught because the test is disabled in Python 3. It now
passes with Python 3 so let's enable it.
The repack-remove fix is just some flakiness I've seen lately.
Reviewed By: singhsrb
Differential Revision: D26412998
fbshipit-source-id: be4b648f31bd6dfbf6a6e5d2e382acb084461974
Summary: Add Python bindings to the Rust `auth` crate, with the intention of replacing `httpconnection.readauthforuri`.
Reviewed By: quark-zju
Differential Revision: D26419447
fbshipit-source-id: dd13bea74961137790beb8c96120ebef99e3c313
Summary: The Python `readauthforuri` method will include *any* fields specified in the `[auth]` section for a given auth group, even if those fields aren't one of the expect ones (such as `cert`, `key`, etc). This is sometimes used in tests to attach additional information to an auth group. To support this in Rust, let's just collect all unknown fields into a `HashMap`.
Reviewed By: quark-zju
Differential Revision: D26416086
fbshipit-source-id: 0252e340e38850a54e24d54810e9abd77c566f63
Summary:
The auth crate is now able to check the presence and expiration of client certificates (D26009207 (9f7d4447fd)). When a problem is detected, it emits an `X509Error`, which specifies exactly what the problem is. Since this error always indicates a certificate issue, we can print out the message configured in `help.tlsauthhelp` (which is more specific than `help.tlshelp` from the previous diff).
Previously, Mercurial would attempt to use the certificate anyway, resulting in a difficult to understand error message. Although the previous diffs in this stack improved the error messages on any TLS failure, the `X509Error` messages are even more helpful.
Users can opt in to this certificate validation with `edenapi.validate-certs`. The functionality is gated on a config option to prevent Mercurial from crashing if certificates are misconfigured, but EdenAPI isn't being used.
Reviewed By: quark-zju
Differential Revision: D26385843
fbshipit-source-id: 9809f612f8aab3f2dd442d6dd8dc348f1af45296
Summary: Print out the help text configured in `help.tlshelp` upon hitting a `TlsError`. Note that in this case, we use `help.tlshelp` rather than `help.tlsauthhelp` since all we know in this case is that *some* kind of TLS error occurred.
Reviewed By: quark-zju
Differential Revision: D26385844
fbshipit-source-id: 1fb5280195de75107ecdfc9203ef8ddda2a04052
Summary: Add a new `TlsError` Python exception type corresponding to `HttpClientError::Tls`.
Reviewed By: quark-zju
Differential Revision: D26385846
fbshipit-source-id: c0df543032461de650a4d24c26c6b8aaab1abbb9
Summary:
Add a new `HttpClientError::Tls` variant specifically for TLS errors, separating them from other `curl::Error`s from libcurl.
To determine whether a particular `curl::Error` is a TLS error, we check both the error code and the contents of the error message.
Reviewed By: quark-zju
Differential Revision: D26385845
fbshipit-source-id: fd58f86a3a61fcfb845d19e262fdcb132dc7ec9f
Summary: This diff adds auto-generated test cases to checkout code. It generates partially overlapping trees and tests transitions between them
Reviewed By: quark-zju
Differential Revision: D26384962
fbshipit-source-id: 6140bbb7ff8b87843a2235f8325f57829cdd8cae
Summary: Currently PathComponentBuf::arbitrary generates any characters. Those characters are ok for unit tests on hg abstractions where they are currently used, but many of them do not work with real filesystems
Reviewed By: quark-zju
Differential Revision: D26384961
fbshipit-source-id: dde1e9276114b30262bc477a3e0f828645f1f32a
Summary:
Currently if VFS overwrites executable file with regular, it preserves exec bit[see added test].
This diff makes sure that file has correct permissions after overwrite
This diff also slightly optimizes write_executable, by calling set_mode on the file handle, instead of path
TODO - we can check if calling stats() before set_permissions will save some time
Reviewed By: quark-zju
Differential Revision: D26379824
fbshipit-source-id: 42d0b2fb79ed860ac37b2de077388002ade69449
Summary:
Before this diff VFS::write_regular did not handle correctly use case when file already existed as as symlink - it would write into symlink location, instead of replacing symlink with a regular file (see updated test_symlink_overwrite that is failing on old implementation)
This diff adds O_NOFOLLOW option on unix when overwriting the file. When destination is a symlink, attempt to write fails with E_LOOP and triggers clear_conflict that removes symlink and allows retry write to succeed.
This also allows one of test cases in checkout that previously did not work
Reviewed By: quark-zju
Differential Revision: D26378893
fbshipit-source-id: 28bcdaba78db283ac7a25bb232c198d3d8f73e5d
Summary: This diff contains basic test setup for checkout tests - we compare transition between two trees without dirty changes
Reviewed By: quark-zju
Differential Revision: D26359502
fbshipit-source-id: ef670c944200bae1652863c91ada92c6fecce4ac
Summary:
Update the dirsync code to allow mirror and exclude rules to match individual
files rather than just directory prefixes.
This simply appends `/` to all filenames when looking for rule matches. This
allows us to efficiently match individual filenames in addition to directory
prefixes.
Reviewed By: quark-zju
Differential Revision: D26294583
fbshipit-source-id: 83b283f344f6e0bc0fe53b9068e7e0170f53504b
Summary:
One of the primary use cases for hash_to_location is translating user provided
hashes. It is then perfectly valid for the hashes that are provided to not
exist. Where we would previously return an error for the full request if a
hash was invalid, we now omit the hash from the response.
Reviewed By: quark-zju
Differential Revision: D26389472
fbshipit-source-id: c59529d43f44bed7cdb2af0e9babc96160e0c4a7
Summary:
The approach is very similar to what commitrevlogdata does. You could say
that it's cargo culted.
I am not sure how appropriate it is to return CommitLocationToHashResponse
but I think that it's fine for now.
Reviewed By: quark-zju
Differential Revision: D26374219
fbshipit-source-id: 61d851d5a4fc4223c65078ef434a0c67314a90cd
Summary: This is defined else where in Python 3, so let's get rid of it here.
Reviewed By: quark-zju
Differential Revision: D26381054
fbshipit-source-id: 9746d2c53f83209d9c795ffdd5841d58ef1153ef
Summary: These are just a few one-liners to fix Python 3 on Windows.
Reviewed By: sfilipco
Differential Revision: D26381055
fbshipit-source-id: d9257f2cf35c05f931d74b7d26bdc79f5bf34185
Summary:
We've rolled both of these out to 100%. Let's make this the default so
we can delete those configs.
Reviewed By: quark-zju
Differential Revision: D26233645
fbshipit-source-id: cd7a08c404483f78ab714763870f5bf0fa801e7a
Summary:
We're seeing cases where Mercurial is creating unused .tmp pack files
and leaving them around. It looks like there are two places this can happen, 1)
in the treemanifest python code we manually instantiate some mutable packs, and
2) when doing a read from the rust data store, when it does a read against the
mutable pack store it will unnecessarily create the MutableDataPackInner struct,
which creates the temp file.
Let's fix those.
Reviewed By: quark-zju
Differential Revision: D26387205
fbshipit-source-id: 5a567c886849084bcc8121949dd2fb0f9e66d570
Summary:
In our upcoming migration away from chef/static rc files, we'll be
marking certain files as "allowed". Our hope is that that list only includes
things like .hg/hgrc, ~/.hgrc, etc.
There are cases however where it's convienent to continue to use chef, for
instance when we condition on machine type. To support this, let's add an
allowed_config option, which will allow configs from non-supported locations.
This will also be useful when remediating issues that come up when we start
enforcing allow_location, without rolling back the entire thing.
Reviewed By: quark-zju
Differential Revision: D26233451
fbshipit-source-id: 71789e0361923a6f80de4aef7f012afc0269440d
Summary:
Copied from D17312417 (e1f4dbeb3d), because that did what I needed done, but incremented by 1.
I would like to change the length of the displayed hash in scm-prompt
to 9. Why such an impactful diff? Because hg sl shows 9 characters, and I
always get confused when the hash in my prompt doesn't match hg sl
Reviewed By: kulshrax
Differential Revision: D25934253
fbshipit-source-id: 15f2bc8bc7d666de1a077d2bafd74ab3c9753341
Summary:
On macOS, EdenFS can be configured to be case insensitive, and will soon be
switching its default to be case insensitive. Mercurial however considers that
EdenFS is always case sensistive on unix (linux, macOS), so let's change it on
macOS so it manually checks if the working copy is.
Long term, EdenFS on macOS will always be case insensitive, at which point this
code will be changed again to indicate that and avoid the manual checks.
Reviewed By: chadaustin
Differential Revision: D26357816
fbshipit-source-id: 4022c097f2804da69cfcc176840683a6dca12ffd
Summary:
The test wasn't meant to be committed. The feature it wants to test was not
easily testable (requires streamclone and pull to have different master,
which is true on Mononoke but false on local hg servers).
Reviewed By: sfilipco
Differential Revision: D26380544
fbshipit-source-id: 6fa720058df8b88ace704d186caa4213d9cd62c9
Summary:
Previously we include tip in sample. But that is problematic for fast paths
like:
if set(commonsample).issuperset(set(localheads) - {nullid}):
ui.note(_("all local heads known remotely\n"))
return localheads, True, remoteheads
If `localheads` is empty, then the returned "common heads" are empty, causing
downloading the entire repo inefficiently.
Fix the issue by moving `tip` from `sample` to `localheads` so the common
heads will include tip in the fast path.
Reviewed By: DurhamG
Differential Revision: D26374303
fbshipit-source-id: 45a2a44e4db4c4ec2a341522a257d46a62b058d5
Summary:
The backingstore crate depends on mio which depends on ntdll. It's not entirely
clear to me why we need to do this manually and why cargo/cmake doesn't pick it
up automatically, but let's fix it this way for now.
Reviewed By: genevievehelsel
Differential Revision: D26376606
fbshipit-source-id: 26714b3f03aabefafdf48c7fb0f442cad501a058
Summary:
Previously the `IdDag` struct has max_level caching. With the previous diff the
cache was gone. Re-implement it on the indexedlog IdDag to maintain
performance. This has visible performance wins:
Before:
building segments 115.949 ms
ancestors 93.072 ms
children (spans) 495.732 ms
children (1 id) 10.384 ms
common_ancestors (spans) 3.567 s
descendants (small subset) 25.829 ms
gca_one (2 ids) 258.997 ms
gca_one (spans) 3.718 s
gca_all (2 ids) 440.764 ms
gca_all (spans) 3.732 s
heads 322.552 ms
heads_ancestors 67.567 ms
is_ancestor 165.046 ms
parents 304.392 ms
parent_ids 11.765 ms
range (2 ids) 21.466 ms
range (spans) 18.663 ms
roots 471.934 ms
After:
benchmarking dag::iddagstore::indexedlog_store::IndexedLogStore
building segments 103.177 ms
ancestors 68.485 ms
children (spans) 451.383 ms
children (1 id) 9.817 ms
common_ancestors (spans) 3.096 s
descendants (small subset) 24.845 ms
gca_one (2 ids) 201.458 ms
gca_one (spans) 3.185 s
gca_all (2 ids) 357.899 ms
gca_all (spans) 3.239 s
heads 295.462 ms
heads_ancestors 50.991 ms
is_ancestor 146.798 ms
parents 296.667 ms
parent_ids 12.305 ms
range (2 ids) 7.781 ms
range (spans) 17.630 ms
roots 478.574 ms
Reviewed By: sfilipco
Differential Revision: D26360564
fbshipit-source-id: 51f55a5bb4e69321515e02f45545618320c1bce5
Summary:
Previously, algorithms are defined on `IdDag<Store>`, which requires a type
parameter to use. Move them to be defined directly on `Store` so using the
algorithms no longer require a type parameter. This will make it easier
to write code that calls algorithms on different IdDagStores.
Reviewed By: sfilipco
Differential Revision: D26360561
fbshipit-source-id: 8e0faf741019c4ed4119ad8e754aea9057b31866
Summary: It is not used. The definitions were moved to dag-types.
Reviewed By: sfilipco
Differential Revision: D26360562
fbshipit-source-id: 35e672194918e3f35294d69cad9e6990d8921900
Summary:
This allows us to "fork" the iterator so we can "peek ahead" multiple items,
without affecting the original iterator.
Reviewed By: sfilipco
Differential Revision: D26360566
fbshipit-source-id: 4cba280e64338b20fb3e1584609be8fda9b3d616
Summary:
Implement Iterator::{size_hint,count,last} for potential fast paths (ex.
Collect into a Vec).
Reviewed By: sfilipco
Differential Revision: D26360565
fbshipit-source-id: 227d9c5e615c2a0a624ba88d6d4c3f81b10d7795
Summary:
Benchmark code is out of sync. Fix it.
Example run:
benchmarking dag::iddag::IdDag<dag::iddagstore::in_process_store::InProcessStore> serde
serializing inprocess iddag with mincode 0.664 ms
mincode serialized blob has 707565 bytes
deserializing inprocess iddag with mincode 42.477 ms
Reviewed By: sfilipco
Differential Revision: D26360563
fbshipit-source-id: f87e7ad53e6b6dadecaa0976e1c61f0399814104
Summary:
Benchmark code is out of sync. This is an important benchmark.
Example run:
benchmarking dag::iddagstore::indexedlog_store::IndexedLogStore
building segments 111.814 ms
ancestors 67.953 ms
children (spans) 484.954 ms
children (1 id) 12.599 ms
common_ancestors (spans) 3.337 s
descendants (small subset) 27.979 ms
gca_one (2 ids) 216.430 ms
gca_one (spans) 3.297 s
gca_all (2 ids) 371.049 ms
gca_all (spans) 3.348 s
heads 303.232 ms
heads_ancestors 52.821 ms
is_ancestor 149.525 ms
parents 294.633 ms
parent_ids 12.173 ms
range (2 ids) 7.612 ms
range (spans) 16.991 ms
roots 459.869 ms
benchmarking dag::iddagstore::in_process_store::InProcessStore
building segments 68.869 ms
ancestors 6.683 ms
children (spans) 175.711 ms
children (1 id) 2.061 ms
common_ancestors (spans) 408.220 ms
descendants (small subset) 6.990 ms
gca_one (2 ids) 16.983 ms
gca_one (spans) 411.237 ms
gca_all (2 ids) 27.921 ms
gca_all (spans) 415.704 ms
heads 110.486 ms
heads_ancestors 5.228 ms
is_ancestor 11.223 ms
parents 108.636 ms
parent_ids 0.746 ms
range (2 ids) 1.539 ms
range (spans) 5.885 ms
roots 172.910 ms
benchmarking NameDag with many heads
range (master::draft) 55.400 ms
range (recent_draft::drafts) 12.439 ms
Reviewed By: sfilipco
Differential Revision: D26360567
fbshipit-source-id: 6d3244e3f4655634c239f84a7304540860a7d34a
Summary:
This endpoint is used by the lazy Segmented Changelog to understand the
location of commit described by hashes. For example users may say that they
want to check out an older commit by hash. The client would then call this
endpoint to understand the relationship of the destination commit relative to
the graph.
Reviewed By: quark-zju
Differential Revision: D26289622
fbshipit-source-id: 4bbfd4bd4f91c984384fff5a6480b8d9d77cf8d3
Summary:
Get the graph location of a given commit identifier.
The client using segmented changelog will have only a set of identifiers for
the commits in the graph. The client needs a way to translate user input to
data that it has locally. For example, when checking out an older commit by
hash the client will have to retrieve a location to understand the place in the
graph of the commit.
Reviewed By: quark-zju
Differential Revision: D26289623
fbshipit-source-id: 4192d91a4cce707419fb52168c5fdff53ac3a9d0
Summary: If a URL's path does not end in a trailing slash (e.g., https://example.com/foo/bar.html), `Url::join` will strip off the final path component under the assumption that it is a filename. This is problematic in the case of EdenAPI base URLs, since the endpoints might be under some base URL (e.g., https://example.com/edenapi/v1). This change ensures the final path component will not be stripped off in such cases.
Reviewed By: quark-zju
Differential Revision: D26353232
fbshipit-source-id: 2ec3dc56fd6d5a0ccaf522fcfc34e6741ec95c68
Summary:
The biggrep revision string has changed again. Let's update our code to
handle the old and new format. Filed T84566856 to track why it changed and how
to prevent breakages.
Reviewed By: quark-zju
Differential Revision: D26351188
fbshipit-source-id: aeb53f54e6e23af47c5eedfa32268c043c02088d
Summary: Allow different type of storages for CheckoutPlan::apply
Reviewed By: quark-zju
Differential Revision: D26291577
fbshipit-source-id: a1c9ba4dbef09e844727ae32eac25c37dd01358e
Summary:
Previously, `write` can fail because the destination file exists as a
directory, or the parent directory is missing. pyworker handles those cases
by calling `clear_conflicts` to remove conflicted directories and create
missing parent directories and retry `write`. Practically, for all `write`
usecases (including checkout) we always want the `clear_conflicts` behavior.
Therefore, move `clear_conflicts` to vfs `write` and make it private.
Reviewed By: quark-zju
Differential Revision: D26257829
fbshipit-source-id: 03d1da0767202edba61c47ae5654847c0ea3b33e
Summary: Currently we pick simple approach and just delete existing file before re-creating it if there is some non-trivial flag change
Reviewed By: quark-zju
Differential Revision: D26239977
fbshipit-source-id: 167efa1bf6018e7f967ef3a9e3c8c62781486ec9
Summary:
This is removing `edenapi::CommitLocation` in order to use
`dag_types::Location`.
First, `edenapi::CommitLocation` has a `count` field and `dag_types::Location`
does not. I find `count` to be difficult to attach to a more general structure.
In practice `edenapi::CommitLocation` is replaced by `CommitLocationToHashRequest`.
On top of the request we have the batch object: `CommitLocationToHashRequestBatch`.
Second, `edenapi::CommitLocation` did not have Wire types, the new structures do.
Reviewed By: quark-zju
Differential Revision: D26159865
fbshipit-source-id: f2508e123e11988726868c7f439a2ed186afce5c
Summary: Used when the IdMap is lazy to fetch missing locations.
Reviewed By: quark-zju
Differential Revision: D26131617
fbshipit-source-id: cde0232b16afb961d9c9a18899ca78bd644f1b6b
Summary:
I think that it makes sense to standardize on importing:
`dag_types::Id` rather than `dag::Id` now that this is a crate of its own.
I am not sure about re-exporting `minibytes::Bytes` but it makes sense to me.
Reviewed By: quark-zju
Differential Revision: D26131616
fbshipit-source-id: fefd0334cf188f247b1541be16421967e8340546
Summary:
Wire types has it's own meaning in Edenapi. I don't see it necessary to
add the wire qualifier to this crate and overload the term.
Reviewed By: quark-zju
Differential Revision: D26129827
fbshipit-source-id: eea66eef2db609611d8ffa215ba63ae4f0b669c8
Summary: It will help users better understand latencies - they will know the region they connect to.
Reviewed By: ahornby
Differential Revision: D26249531
fbshipit-source-id: b77c17c19efb93e5c7e83926f964cf42ec3326fe
Summary:
The test is broken in master. I might be using a stale binary running that
test in D26245424 (1392673a95). Revert the test to before D26245424 (1392673a95) fixes it.
Reviewed By: sfilipco
Differential Revision: D26321855
fbshipit-source-id: 4ada61211d3b354ae6f94f7fc8364f4550e9aeb8
Summary:
Introduce `FallbackStore`, a newstore combinator which implements the `ReadStore` trait and first checks a "preferred" store for the provided keys, before falling back to a "fallback" store. The combinator requires that both stores share the same `Key` type, but allows a user provided "value adapter function" to convert from the fallback store's value type to the preferred store's value type.
Currently `FallbackStore` does not support writing missing values to the preferred store - this functionality will be introduced in a future change.
Reviewed By: DurhamG
Differential Revision: D26203146
fbshipit-source-id: 0e99110f93130ff30c95cce15e3dc7873616519c
Summary: Introduce a new stream combinator, `select_drop`, based on `futures::stream::select`, which performs the same function except dropping the contained streams when they terminate, rather than when the combined stream is exhausted. This prevents a deadlock in cases where one stream will not terminate until the other terminates.
Reviewed By: DurhamG
Differential Revision: D26139816
fbshipit-source-id: d28da32244a215741476e1c3882154ea9e3116a5
Summary: Introduce a minimal version of an async, typed `ReadStore` trait and corresponding `EdenApi` implementation, along with a debug command, `debugnewstore` to exercise it.
Reviewed By: DurhamG
Differential Revision: D26050641
fbshipit-source-id: 2c14c3715e7067f9ecd1e649e6ca146a1ce249bf
Summary:
The warning is noisy if the wait is short and can make users dislike commit cloud.
Make sure we don't print the warning earlier than defaultlockwaitwarntimeout.
This is a follow up on D25587459 (18b8c66439) that doesn't fully work if warntimeout is passed equal to zero.
The default threshold has been introduced earlier in D25587459 (18b8c66439). This is just a fix.
A new test has been added. Also, the api should allow to pass value 0 meaning to show the warning always.
Reviewed By: quark-zju
Differential Revision: D26251321
fbshipit-source-id: c3beb5fec6a65f1816f667df70c1a39dd65ef083
Summary:
This matches the Rust behavior and is useful for filtering
because the env logger syntax applies to target:
% RUST_LOG=edenscm.hgext.debugshell=info lhg dbsh
In [1]: from edenscm import tracing
In [2]: tracing.info('foo')
[2021-02-05T19:21:41.082Z INFO edenscm.hgext.debugshell] message="foo"
Reviewed By: kulshrax
Differential Revision: D26282053
fbshipit-source-id: 8ee9e82b955835b24c49f9bf81c7a3aec7a65a33
Summary:
Make it possible to convert Bytes to `Vec<u8>` in a zero-copy way if possible.
This will make it possible to convert `minibytes::Bytes` to `bytes::Bytes` in a
zero-copy way if possible. Practically, it might be useful for some
revisionstore -> edenapi/types usecases.
Reviewed By: kulshrax
Differential Revision: D26237922
fbshipit-source-id: 28d620f303511099df77f79256d98abb1010f665
Summary: This API will be used in the next diff.
Reviewed By: DurhamG
Differential Revision: D26237923
fbshipit-source-id: 69438072c2edef1ce28ceef3b8b723f015f54ff5
Summary:
This affects `first`, `last`, `limit` revset functions.
Previously, they just iterate through the set naively. Now they have Rust fast
paths. For example:
In [1]: time repo.revs('last(parents(:1000000),100)')
CPU times: user 5.08 ms, sys: 1.02 ms, total: 6.1 ms
Wall time: 4.96 ms
Out[1]: <nameset+ <spans [0a087e42b29ba5c9ceb3588477d78f7f09ce2663:af5e72c2a0c2e78de462bba8bde63f2499aeb9b5+999900:999999]>>
In [2]: time repo.revs('first(parents(:1000000),100)')
CPU times: user 2.21 ms, sys: 40 µs, total: 2.25 ms
Wall time: 1.83 ms
Out[2]: <nameset+ <spans [06b96ec2a8b60d984606f36c30d3dbc899d804df:4cab7b68c0bbdc13eb2eded8fc8c4c8d520a7189+0:99]>>
In [5]: time repo.revs('first(reverse(parents(:1000000)),100)')
CPU times: user 2.2 ms, sys: 185 µs, total: 2.39 ms
Wall time: 1.67 ms
Out[5]: <nameset- <spans [0a087e42b29ba5c9ceb3588477d78f7f09ce2663:af5e72c2a0c2e78de462bba8bde63f2499aeb9b5+999900:999999]>>
In [6]: time repo.revs('last(reverse(parents(:1000000)),100)')
CPU times: user 2.01 ms, sys: 12 µs, total: 2.02 ms
Wall time: 1.68 ms
Out[6]: <nameset- <spans [06b96ec2a8b60d984606f36c30d3dbc899d804df:4cab7b68c0bbdc13eb2eded8fc8c4c8d520a7189+0:99]>>
In [7]: time repo.revs('limit(reverse(parents(:1000000)),100,10000)')
CPU times: user 1.48 ms, sys: 887 µs, total: 2.37 ms
Wall time: 1.89 ms
Out[7]: <nameset- <spans [5e23b6f07f1512a8991de5a0e883ff4d598ac1d7:f5f68207be4a458e99d4a9200296977aecf44ea2+989900:989999]>>
This, together with fast `_firstancestors`, could potentially answer
globalrev-like queries on the master branch without recording the globalrev
information server side (which has complexities like locking, etc). For
example, to convert a global rev `g` to commit:
c = repo.revs('limit(_firstancestors(master), 1, %s)', g).first()
To convert a commit `c` to global rev:
g = len(repo.revs('_firstancestors(%s)', c)) - 1
Reviewed By: sfilipco
Differential Revision: D26203558
fbshipit-source-id: 14d9247bbb07260f783e05b3fb1034406de48121
Summary: Provide a way to slice a set.
Reviewed By: sfilipco
Differential Revision: D26203562
fbshipit-source-id: 97a4349833a7a1c9664189d4737e2ad418369f22
Summary:
The SliceSet provides slicing support (skip n items, and then take m items)
for a general NameSet. The complexities come from caching and fast paths.
Basically, we cache the skipped items as a way to answer "contains" quickly.
We also cache the "taken" items if it's bounded to answer "iter_rev",
"contains".
Reviewed By: sfilipco
Differential Revision: D26203559
fbshipit-source-id: 6078b6178aff878e2169e87d446f0b254432aa80
Summary:
In the future we'd like to test "contains" but only use the "contains"
code path if it's better than O(N). Currently there is no way to know
whether "contains" is O(N) or not. Add an explicit API for that.
Reviewed By: sfilipco
Differential Revision: D26203554
fbshipit-source-id: 5d4c6014694c45b666a0ecd83fce33157cc15779
Summary: Enhance the test so it checks hints set by the `filter` function.
Reviewed By: sfilipco
Differential Revision: D26203553
fbshipit-source-id: b9bf5baa3bf51434835341e95e72073bd8c4256a
Summary:
It is incorrect with >= 2 hints.
For example,
iter: [hints1, hints2, hints1]
fold acc: hints1, None, hints1
The `None` should be permanent if there are two versions that do not have an
order.
Reviewed By: sfilipco
Differential Revision: D26203555
fbshipit-source-id: 96ff30ba45d439220519cd1505e3264118ffd9b2
Summary:
Previously, `LazySet` was constructed with default `Hints`. That disables fast
paths. Revise the API so LazySet requires an explicit `Hints` to address the
issue.
Reviewed By: sfilipco
Differential Revision: D26203561
fbshipit-source-id: c92cd1f7eb7b40ffaaf53abcf05e64f3d41b906d
Summary:
Previously, `MetaSet` was constructed with default `Hints`. That disables fast
paths. Revise the API so MetaSet requires an explicit `Hints` to address the
issue.
Reviewed By: sfilipco
Differential Revision: D26203557
fbshipit-source-id: 9e7658af8723b06d0efdcad1ab4671c79e907326
Summary: Those methods will be used for fast paths slicing a NameSet.
Reviewed By: sfilipco
Differential Revision: D26203556
fbshipit-source-id: aef18f60633653e19571e3fdeeb6b258e4dd32c7
Summary:
This just renames types so `IdSet` is the recommended name and `SpanSet`
remains an implementation detail.
Reviewed By: sfilipco
Differential Revision: D26203560
fbshipit-source-id: 7ca0262f3ad6d874363c73445f40f8c5bf3dc40e
Summary:
Before this change, nameset's min, max use the base class implementation, which
can be undesirably slow sometimes, while `first` and `last` remain fast
since they call into Rust logic which understands better about fast paths.
Optimize `min`, `max` by converting them to `first` or `last` if possible.
Before. Note `min()` is slow:
In [3]: s=repo.revs('parents(1:10000)')
warning: ':' is deprecated; use '::' instead. https://fburl.com/hgcolon
In [4]: s
Out[4]: <nameset+ <spans [06b96ec2a8b60d984606f36c30d3dbc899d804df:d6867d22c1ad3cf6e384332093e1d0de1fa86cb5+0:9999]>>
In [5]: %time s.min()
Out[5]: CPU times: user 51.5 ms, sys: 8.25 ms, total: 59.7 ms
Wall time: 57.8 ms
0
In [6]: %time s.max()
Out[6]: CPU times: user 64 µs, sys: 15 µs, total: 79 µs
Wall time: 84.6 µs
9999
In [7]: %time s.first()
Out[7]: CPU times: user 39 µs, sys: 9 µs, total: 48 µs
Wall time: 50.8 µs
0
In [8]: %time s.last()
Out[8]: CPU times: user 62 µs, sys: 0 ns, total: 62 µs
Wall time: 66.5 µs
9999
After:
In [2]: %time s.min()
CPU times: user 0 ns, sys: 902 µs, total: 902 µs
Wall time: 907 µs
Out[2]: 0
In [3]: %time s.max()
CPU times: user 551 µs, sys: 0 ns, total: 551 µs
Wall time: 557 µs
Out[3]: 9999
In [4]: %time s.first()
CPU times: user 50 µs, sys: 9 µs, total: 59 µs
Wall time: 62 µs
Out[4]: 0
In [5]: %time s.last()
CPU times: user 49 µs, sys: 9 µs, total: 58 µs
Wall time: 62 µs
Out[5]: 9999
Reviewed By: sfilipco
Differential Revision: D26182241
fbshipit-source-id: dedf3788c52d22c6b63ae60847cc0667616f11d2
Summary:
Optimize the `_firstancestors` revset function using Rust.
When calculating `len(_firstancestors(master))`, the new code took 2ms while
the old code needs 76s, a ~40000x improvement.
Reviewed By: sfilipco
Differential Revision: D26182242
fbshipit-source-id: 55f17b014e727d8e8e3099b7c287f8bf8479279b
Summary: This exposes the segments version of the algorithm.
Reviewed By: sfilipco
Differential Revision: D26182244
fbshipit-source-id: 716e6d5254c9962618040e7549c2804184230a97
Summary: This will be used by NameDag.
Reviewed By: sfilipco
Differential Revision: D26182243
fbshipit-source-id: 9db2ecde98281dc45fcfd0d7cf30d6c7bf2be81d
Summary:
This will be useful to optimize `_firstancestors` revset, which is useful to
calculate a linear branch to draw growth graphs. Without a fast path, the pure
Python `_firstancestors` implementation would have id <-> name translation
overhead that makes Rust changelog O(20) slower than the Python revlog.
Reviewed By: sfilipco
Differential Revision: D26182240
fbshipit-source-id: d44f5ca5dc8c38df74281832931d87868791209e
Summary:
Optimize the `x~n` revset function using Rust.
Note: This changes the behavior a bit, `x~n` no longer returns `null`.
Reviewed By: sfilipco
Differential Revision: D26142683
fbshipit-source-id: d6a45b7e67352d74986274e52002a769bbae772e
Summary:
When `n` is too large, return None. This matches the "parents()" behavior -
not error out but returns empty set.
Reviewed By: sfilipco
Differential Revision: D26142684
fbshipit-source-id: e45fca69e39c2968dc7abc5a4a155e6b7c280836
Summary: Optimize the "merge()" revset function using the merges() from Rust.
Reviewed By: sfilipco
Differential Revision: D26142169
fbshipit-source-id: 47f426625869b7889b28bb1a18544d4abae36cae
Summary:
The "merges" algorithm on the IdDag. Basically scan through flat segments
and conditionally pick up their "low"s.
Reviewed By: sfilipco
Differential Revision: D26142172
fbshipit-source-id: 305fe619a65ed4034423f303bee8d57be0424963
Summary:
The newly added API returns parent count wihtout actual parents.
Useful for the "merges" algorithm.
Reviewed By: sfilipco
Differential Revision: D26142176
fbshipit-source-id: 4f301b8de88f2af637f52bf62b24ddb12e65b6a7
Summary:
The function calculates all merges within a graph. It is useful to calculate
the "universally known" set, or to answer the "merge()" revset function.
This diff only adds a default impl. Upcoming diffs will add more efficient
versions on the segments graph.
Reviewed By: sfilipco
Differential Revision: D26142173
fbshipit-source-id: 02de180f6e444bcac63a1cc46dd23faeb8e08e14
Summary: The `filter` API filters a set by a function.
Reviewed By: sfilipco
Differential Revision: D26142177
fbshipit-source-id: f24cbeeaf1c85264706c933c98e364d7937790fe
Summary:
For `nameset OP baseset` or `baseset OP nameset`, convert baseset to nameset
automatically for fast paths. This is motivated by a slow query in pushrebase:
# pushrebase/__init__.py:validaterevset
if onto != donotrebasemarker and onto in repo:
rebased = list(repo.set("(successors(%r) & ::%s) - %r", revset, onto, revset))
if rebased:
raise error.Abort(
_("commits already rebased to destination as %s")
% ", ".join(str(c) for c in rebased)
)
Depending on the complexity of `revset`, `successors(revset) & ::onto` might
generate a filterset that is very inefficient - iterating through a large set `::%s`.
Optimize them by using the Rust nameset for calculations.
Before:
In [4]: repo.revs("(successors(1001) & ::1000) - 1000")
Out[4]: <filteredset <filteredset <baseset+ [1001]>, <nameset- <spans [06b96ec2a8b60d984606f36c30d3dbc899d804df:e890940eb1c34a06967bb9e38a0317ad7a0eb518+0:1000]>>>, <not <baseset [1000]>>>
After:
In [1]: repo.revs("(successors(1001) & ::1000) - 1000")
Out[1]: <nameset+ <spans []>>
This can change the ordering of sets. Therefore some test changes.
Reviewed By: DurhamG
Differential Revision: D26245424
fbshipit-source-id: 2e3ab891c586bb80cf947fff4bbdcd453c01ae70
Summary:
the revset cloudremote is no longer used anywhere in the codebase
also it is related to obsmarkers that are not longer used either
Reviewed By: quark-zju
Differential Revision: D26250792
fbshipit-source-id: b55b8d52c44869f50d5b5f5d8ef2e6c2fac07597
Summary: The function is used in many places and I noticed there are some issues with commit cloud due to the bug that visible heads can contain public commit.
Reviewed By: quark-zju
Differential Revision: D26250556
fbshipit-source-id: e57e447dee803719fcf38cf376ad5af569d8020d
Summary: I have noticed that server-side status code returned by speedtest's upload changed to 200. It's on master but it's not yet out in the wild.
Reviewed By: johansglock
Differential Revision: D26249530
fbshipit-source-id: 0f6e77ad4f9daf7f7a3bbc216f20435252101078
Summary:
Reduce local heads from unfiltered raw heads to visible heads. Reduce remote
heads from all heads to selected heads, plus those explicitly specified via
`-r`, `-B`, or via `repo.pull`.
This should speed up both pull and push for repos with lots of heads (ex.
fbsource), and make fastdiscovery less necessary.
Reviewed By: DurhamG
Differential Revision: D26207588
fbshipit-source-id: b64485566e0651ad47a5d1ee47e68301ba371e57
Summary:
With selectivepull, at the end of push, there is no need to fetch all bookmarks
from the server.
Reviewed By: DurhamG
Differential Revision: D26213970
fbshipit-source-id: c5fbb972f31bff8159be26517c6432af8b80225f
Summary:
Previously, remotenames lists all server-side bookmarks to check flags
like --delete, --create, --non-forward-move. That is inefficient. This
diff makes it use the listkeyspatterns API to reduce overhead.
Reviewed By: DurhamG
Differential Revision: D26213969
fbshipit-source-id: 2e51433829e80ebe685755049339c2dc03158717
Summary:
The "unknown server heads" should be "remoteheads - commonheads", not
"commonheads - remoeheads".
Reviewed By: DurhamG
Differential Revision: D26228573
fbshipit-source-id: c9336d135ea3628da47024083fd638896576d106
Summary: This makes it easier to reason about changes in reducing heads exchanged.
Reviewed By: DurhamG
Differential Revision: D26207589
fbshipit-source-id: 49c0c0dc25355a321c1aa4c9edfb5c43d2f23fd8
Summary:
During pull, especially `repo.pull`, we might already know that certain heads
exist on the server-side. Pass them to the discovery logic so discovery can
treat them as "remote known" unconditionally.
Reviewed By: DurhamG
Differential Revision: D26207587
fbshipit-source-id: 7f77b5a6ffcb15dd5ab0317f9deae7f84892fb00
Summary:
The `bytes` crate still does not support zero-copy on mmaped buffer.
Switch to `minibytes::Bytes` so bytes returned from our main storage
backend indexedlog is a zero-copy slice backed by a mmap buffer.
Migrate vfs, revisionstore, pyworker to minibytes so they can preserve
zero-copy mmap buffers from indexedlog.
The edenapi/types is unchanged, since it's also used in Mononoke which uses
`bytes::Bytes` all the places. The conversion to `minibytes::Bytes` is cheap
so it's probably not a performance issue.
Reviewed By: kulshrax
Differential Revision: D26218289
fbshipit-source-id: e4f1c631143b7676c6b48d3b4f97055299bfd334
Summary:
Make it possible to construct minibytes::Bytes via bytes::Bytes.
This will be used later.
Reviewed By: kulshrax
Differential Revision: D26232990
fbshipit-source-id: 36af6f28fd08eb457de8b9223235ec038ac3ef14
Summary:
Previously they were `Bytes`, which relies on `Bytes::from_static` being
a const_fn. Going to migrate from bytes to minibytes, which has a type
parameter on `Bytes` that makes it harder to have const_fn from_static.
Switch the constants to plain `&[u8]` to avoid the const_fn issue.
Reviewed By: kulshrax
Differential Revision: D26218290
fbshipit-source-id: 728c9b3831e551f41fb42ec257ca5fe75b7e93a3
Summary: Going to migrate from bytes to minibytes. Avoid using bytes' specific API.
Reviewed By: kulshrax
Differential Revision: D26218826
fbshipit-source-id: ba41697eab8fc5fb7bf73bc565c05a7c1b29464c
Summary: Going to migrate from bytes to minibytes. Avoid using bytes' specific API.
Reviewed By: kulshrax
Differential Revision: D26218292
fbshipit-source-id: c69c2a1e0d1bde37f49e7ad542bf5e952deebb7a
Summary:
The original migration strategy with dynamicconfig was to fix configs
one by one until the dynamicconfig values matched the chef/static ones, then we
can turn off chef/static configs. This looks to be too much work, so we're going
to try a different strategy of just turning off all chef/static configs on a
small number of hosts and seeing what breaks.
The legacylist and disallowlist configs were part of the old strategy, and they
make it more complicated to fix dynamicconfig mismatches, so let's get rid of
them.
Reviewed By: quark-zju
Differential Revision: D26208548
fbshipit-source-id: 63171f1f16aa0498c0eefa994dffaeb8e0cc0d72
Summary: This file uses new async syntax that is not valid on Python 2.
Reviewed By: singhsrb
Differential Revision: D26223827
fbshipit-source-id: 98ecd60d6f21eb91f5e3781539be7da33327667e
Summary: set_executable is a pub function of VFS that set exec permissions on simple file
Reviewed By: quark-zju
Differential Revision: D26212713
fbshipit-source-id: 4c3ef477fc8d61362285654dda0b006342e046ee
Summary:
Basic implementation only process file updates for now.
See todos on CheckoutPlan::apply for more detail
Reviewed By: quark-zju
Differential Revision: D26209984
fbshipit-source-id: fcfbf568359d553a51ea02ea194634048d093d0e
Summary: This is needed becase later during checkout list of updates that needs fetch will go into storage stream
Reviewed By: quark-zju
Differential Revision: D26209983
fbshipit-source-id: 9fb54d48c6f0afc4fb67320aafc2e981c96ab5a9
Summary:
Expose the async LazySet API via NameSet constructor so users won't need to
care about the LazySet type.
Reviewed By: sfilipco
Differential Revision: D26142170
fbshipit-source-id: 178383684981e81e43f2a5610c45a7ebbd354ab4
Summary:
Phabricator integration was broken in Python 3 on OSX because we didn't
supply the default context so it could verify against the ca_certs.
Reviewed By: quark-zju
Differential Revision: D26183713
fbshipit-source-id: ac395c002da0fb343d1d0b999a57d2059f66f0c7
Summary:
Add a way to capture tracing output in tests to verify certain logs are
emitted.
Reviewed By: sfilipco
Differential Revision: D26142174
fbshipit-source-id: 9267ffbe413973b8c54c54db75fe037c05614b1a
Summary:
`tracing_subscriber` has a fmt subscriber that satisfies the "log to stderr in
tests" need. It does not depend on `env_logger` or `TracingData` forwarding to
`env_logger`.
Reviewed By: sfilipco
Differential Revision: D26142175
fbshipit-source-id: 6e7dcd1585cb8431855322493d93bc49a8d57b76
Summary: This would allow us to ignore bogus data and still get something useful for logging.
Reviewed By: DurhamG
Differential Revision: D20343844
fbshipit-source-id: 763d294bc44bb203c1f206ca80e0839396e8de6e
Summary:
Errors could occur with various reasons. Such as permissions.
Not having periodical sigtrace does not affect correctness, so let's just
ignore errors.
Reviewed By: singhsrb
Differential Revision: D26192199
fbshipit-source-id: 8d2dba56979efd121c9bdc757895e86aa23d6694
Summary:
Address yarn's `node_modules` problem on Winodws.
The design is debatable and probably not the best. Another approach could be simply special casing subdir path ends with `node_modules`. This won't require any user configuration but it would be a special case.
Reviewed By: chadaustin
Differential Revision: D26149393
fbshipit-source-id: b3e66cb2d4b70078bb25e7329988cd5ff8fdeadd
Summary:
ui.fout and ui.ferr are now refcells. If you do ui.fout = ui.ferr (as
done in hg grep's fallback path to redirect stdout) and then do
ui.fout.swap(ui.ferr) (as is done in the pullpaths hotfix) you can end up with
a refcell owning itself.
Let's avoid that by making hg grep use ui.fout.swap appropriately.
Reviewed By: quark-zju
Differential Revision: D26150647
fbshipit-source-id: eede8f00f9f9566157b0c17fe7049ca860ce715b
Summary:
If your disk1 is an external HFS-formatted disk, then
eden_apfs_mount_helper will fail to create apfs subvolumes on
it. Instead, use the disk backing the mount.
Reviewed By: fanzeyi
Differential Revision: D26096296
fbshipit-source-id: baa45181afb6610a095c864eb3183e5af76ec4e0
Summary:
It is already broken with segmented changelog (it assumes 0..len(repo) are
valid revs). It is super slow and cannot be optimized efficiently. The _only_
non-zero-exit-code usage in the past month is like:
hg log -r 'reverse(children(ancestors(remote/master) and branchpoint()) and draft() and age("<4d"))'
which takes 40 to 100s and can be rewritten using more efficient queries like `parents(roots(draft()))`.
Reviewed By: singhsrb
Differential Revision: D26158011
fbshipit-source-id: 7957710f27af8a83920021a228e4fa00439b6f3d
Summary:
Migrate most crates to tokio 1.0. The exception is edenfs-client, which has
some dependencies on `//common/rust/shed/fbthrift_ext` and seems non-trivial
to upgrade. It creates a separate tokio runtime so it shouldn't be affected
feature-wise.
Reviewed By: singhsrb
Differential Revision: D26152862
fbshipit-source-id: c84c43b1b1423eabe3543bccde34cc489b7805be
