Summary:
This is simple command mostly to be used by testing before we fully integrate with hg pull
This command does not perform discovery and requires from/to revision to be passed in cmd line
Reviewed By: quark-zju
Differential Revision: D29315647
fbshipit-source-id: 26d67031e566b7c99af1e2a5ab287f02b52f7db0
Summary: Update versions for several of the crates we depend on.
Reviewed By: danobi
Differential Revision: D29165283
fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
Summary:
For a while we had two methods in async runtime: block_on_future and block_on_exclusive, due to historic reasons
Recently those methods were calling same code, and now it is time to replace both of them and rename to block_on
Reviewed By: quark-zju
Differential Revision: D29121107
fbshipit-source-id: 5faa76ae181e491b55d799c23c9de1b4e80298f3
Summary:
The added command gives access to read (execute) dynamicconfig without
using an on-disk repo.
It can be used by the clone script to stage rollout lazy changelog, or just to
verify dynamic config changes without using a repo.
Reviewed By: DurhamG
Differential Revision: D29123072
fbshipit-source-id: e8856d816a636fa860bfcc9694306a4a37552523
Summary: Straightforward update of `debugscmstore` from old scmstore to new scmstore. I'll want to improve this command a bit more in the future, but this at least enables you to easily test arbitrary fetches.
Reviewed By: DurhamG
Differential Revision: D29047527
fbshipit-source-id: 4e10cb88cba4b572d3e413640ca3d800940d675d
Summary: revert the zstd crates back to previous version
Reviewed By: johansglock
Differential Revision: D29038514
fbshipit-source-id: 3cbc31203052034bca428441d5514557311b86ae
Summary: Update to latest version. This includes a patch to async-compression crate from [my PR updating it](https://github.com/Nemo157/async-compression/pull/125), I will remove once the crate is released.
Reviewed By: mitrandir77
Differential Revision: D28897019
fbshipit-source-id: 07c72f2880e7f8b85097837d084178c6625e77be
Summary: This will be used for rate limiting decisions. Also, could be logged to scuba tables to get more info about clients.
Reviewed By: quark-zju
Differential Revision: D28750197
fbshipit-source-id: 83f54e38f998c9dd824ef2d3834c777a44d0ffed
Summary:
Convert client certificates (which are expected to be supplied as PEM files) into an in-memory PKCS#12 archive to pass into libcurl. This is necessary on certain platforms (such as Windows) whose native crypto APIs do not support loading PEM files.
This was previously landed as D27637069 (5b759a2b52), which unconditionally converted the certificates under the assumption that all major TLS backends support PKCS#12. That assumption is still true, but it did not account for the fact that libcurl itself is dynamically linked on some platforms (such as MacOS), and the system libcurl may be too old to support support in-memory certs (via `CURLOPT_SSLCERT_BLOB` added in libcurl version 7.71.0). This diff gates this feature behind the `http.convert-cert` config option, which we can selectively set on platforms where it is needed.
Reviewed By: mzr
Differential Revision: D28524444
fbshipit-source-id: 4af9cdd60b8ef3977ad81abdb8e406c63795e628
Summary:
This makes it easier to see what builder functions were registered:
% EDENSCM_LOG=edenapi=debug lhg log -r .
May 06 16:40:29.355 DEBUG edenapi::builder: registered eagerepo::api::edenapi_from_config to edenapi Builder
Reviewed By: DurhamG
Differential Revision: D28271366
fbshipit-source-id: f6c7c3aa9f29c3e47c2449e3d5fc16474aa338b0
Summary:
eagerepo -> metalog -> git2 -> libgit2-sys -> libgit2 conflicts with edenfs'
non-Rust libgit2 dependency. Rust git2 crate does not seem to provide a way to
depend on specified libgit2.
Quote https://github.com/rust-lang/git2-rs/issues/263#issuecomment-450934287:
> It's expected that git2-rs builds its own copy of libgit2 and doesn't use the
> system version, as the system version is likely incompatible
It also seems non-trivial to make buck C++ use the libgit2 frm `libgit2-sys` crate.
Let's just avoid depending on eagerepo from edenapi directly for now to solve the
issue. This basically revives D27948369 and D27951632.
Reviewed By: xavierd
Differential Revision: D28243784
fbshipit-source-id: 0c38c20c2d3a80c550732129da572fe26a229799
Summary: This makes CloneData possible to represent an empty repo.
Reviewed By: sfilipco
Differential Revision: D27926246
fbshipit-source-id: 0bcead224ef5b89c66d07a34d8217edaef62177f
Summary:
Update the zstd crates.
This also patches async-compression crate to point at my fork until upstream PR https://github.com/Nemo157/async-compression/pull/117 to update to zstd 1.4.9 can land.
Reviewed By: jsgf, dtolnay
Differential Revision: D27942174
fbshipit-source-id: 26e604d71417e6910a02ec27142c3a16ea516c2b
Summary: This seems to have broken the EdenFS HgPrefetch test.
Reviewed By: xavierd
Differential Revision: D27795192
fbshipit-source-id: 80a748036961aa6a5750182bf65637fb76825341
Summary:
Previously we'd skip dynamicconfigs when there wasn't a repo available.
Now that dynamicconfig can represent the NoRepo state, let's load dynamicconfigs
in that situation.
This also supports the case where there is no user name.
Reviewed By: kulshrax
Differential Revision: D26801059
fbshipit-source-id: 377cfffb6695a7fbe31303868a88862259ebf8c4
Summary:
Extend debugscmstore command to fetch arbitrary files / trees by key.
Replace debugpyscmstore with a python fallback for debugscmstore (allowing you to test with the store as it is constructed for Python, with legacy fallback).
Refactor some functionality so it is shared between the rust and python versions of debugscmstore.
Currently the output is pretty ugly. It uses the `{:#?}` format for everything. In the next change, I propose modifying the `Debug` implementation for `minibytes::Bytes` to use ascii-escaped bytestrings rather than the default slice formatter to make things much nicer.
This new `debugscmstore` functionality should be useful in integration tests for testing scmstore under different repo configurations, and for test harnesses and performance testing (fetch a specific set of things easily, simulate delays in the key stream by delaying the input pipe, etc).
Reviewed By: andll
Differential Revision: D27351321
fbshipit-source-id: 8650480e3f5b045b279472643570309c48d7fe6b
Summary: Like `FileScmStoreBuilder`, but for trees. As LFS is not used for trees, `TreeScmStoreBuilder` defaults to `ExtStoredPolicy::Use` (pass along anything you find without LFS-specific checks).
Reviewed By: DurhamG
Differential Revision: D27641290
fbshipit-source-id: 637340a23cef058e7e37a41ae7f5b4fcc9481190
Summary: Introduce a new `FileScmStoreBuilder` structured much like `ContentStoreBuilder`, but supporting the features needed for the intermingling of `contentstore` and `filescmstore` construction (shared indexedlog, scmstore fallback to contentstore).
Reviewed By: DurhamG
Differential Revision: D27640702
fbshipit-source-id: e9771e6f61d80698a9dd761a0db66407b565c010
Summary: The lazy backend can now (partially) support the non-full IdMap segment clone.
Reviewed By: sfilipco
Differential Revision: D27581488
fbshipit-source-id: 51eded6acdbe82d22f5bb73eb4a715e2c22f4d75
Summary: This crate does not panic on Windows.
Reviewed By: DurhamG
Differential Revision: D27640362
fbshipit-source-id: f50f6b8e0bd31e5f80fa939bcfb6846bc8fd4a63
Summary:
Recently we saw some progress rendering issues. Add a command to attempt to
reproduce them.
Reviewed By: DurhamG
Differential Revision: D27669184
fbshipit-source-id: 62fcf82d8261fd27e91ba5a116c61f4df1919007
Summary: Add a new `http.verbose` config option that turns on verbose output for libcurl (similar to the output printed by `curl -v`). This can be very useful for debugging HTTP issues.
Reviewed By: DurhamG
Differential Revision: D27693304
fbshipit-source-id: 2ad7a08889f40ffbcd2f14ac9c21d70433629da4
Summary: Vendor updated zstd crates. Started on this as a prerequisite for vendoring updated async-compression crate, however dependency resolution means this actually updates async-compression as well.
Reviewed By: farnz
Differential Revision: D27467760
fbshipit-source-id: 74ca9e4da9f336cf609cf06c62559c9ef913c9a2
Summary:
Add a `set_global_config` function to `hg-http` that gets called prior to command execution. This function can be used to configure the global behavior of all HTTP requests by setting up a callback that modifies every `Request` prior to sending.
Right now, the use case for this is to support the `--insecure` flag, as can be seen in the code.
Reviewed By: quark-zju
Differential Revision: D27243638
fbshipit-source-id: 61a52c1f4b56fffd860c2a7e9c7b03e146b2240a
Summary:
Rename "newstore" to "scmstore"
The three main options I'm considering are "edenstore", "scmstore", and "hgstore". They all describe the project sufficiently well, in my opinion. "edenstore" might be a stretch, as I have no reals plans for Mononoke to use it, while "hgstore" might be a bit limited, as integration with EdenFS is a core goal, and it'll be used by EdenFS to fetch remote data via EdenApi, not just Mercurial's local storage. I feel like "scmstore" rolls off the tongue the easiest, particularly vs. "hgstore" (both "H.G. Store" and "Mercurial Store"), and is a bit easier to type than "edenstore" (which has "ede" all with the same finger). I don't have a strong opinion on this matter, so If you'd like a different name I'm open to suggestions.
Speak now or forever hold your peace.
Reviewed By: andll
Differential Revision: D27180332
fbshipit-source-id: 19e6972ea0f6527e671792845dcfd339cf1ab767
Summary: Adds `ExtStoredPolicy` support to the EdenApi and IndexedLog ReadStore implementations. This flag controls how LfsPointers found during a file query are handled (either returned as a result, or treated as "not found").
Reviewed By: kulshrax
Differential Revision: D27171814
fbshipit-source-id: 14dda47f32184c3ee703fbc77106885ca4d3ea27
Summary:
eden/scm/lib: default cpython_ext to python3
```
CI internally, try a PR on github once its landed.
Reviewed By: quark-zju
Differential Revision: D27237636
fbshipit-source-id: 1768f778d75b5d6c62dfd5641f911604a37d3163
Summary:
Remove use of `#[async_trait]` in the new storage API, and eliminate the outer Future on the `fetch_stream` and `write_stream` methods, which was not used. The methods are now normal trait methods, which accept and return a stream.
It's possible we'll want to make these methods `async` again in the future, but until this this is more ergonomic, faster (one less layer of indirection), and avoids some of the type system limitations of `#[async_trait]`.
Reviewed By: andll
Differential Revision: D26695517
fbshipit-source-id: 2db60a3f37d594b0b9d1e1a2708532ef0ddaf585
Summary:
Split `FallbackStore` (which supports writing fallback fetches) into `FallbackCache` (which behaves the same) and `Fallback` (which doesn't support writing fallback fetches). This eliminates the `From<>` bound on `Fallback` which is nonsensical in some cases where we don't want writing.
Replace `write_store` and `write` fields with a single `Option<>` `write_store` in `FallbackCache`, which cleans up the code a bit.
Separate the "fallback value type", "preferred value type", "write value type", and "output value type" separately control the output value type independent of the preferred store.
Reviewed By: kulshrax
Differential Revision: D26526149
fbshipit-source-id: aa9f25f5efb8a9ddab03a7d86a7a007f24d156ce
Summary:
When testing "disable_progress" with chg (next diff) I found it was not
effective because there are 2 separate IO structs. The one we disable
from Python is different from the one the Rust progress thread uses.
I traced it down here. Since the Python IOs are just wrappers of
Rust IOs in the chg use-case. There is no need to recreate an IO
struct.
The "creating IO" struct is still useful, for things like "-t.py" testing where
the output needs to be captured into different Python variables per different
commands.
Reviewed By: DurhamG
Differential Revision: D27149243
fbshipit-source-id: 6e27adcc9f48b21fc24fba120be8c4a8fef1f909
Summary:
The Rust progress bars now integrates with hg-http and all requests get
progress bars for free. There is no need for a separate bar.
Reviewed By: andll
Differential Revision: D26970747
fbshipit-source-id: c5e59c745e75a93e5da0541baab21c3f23dc1322
Summary:
Render the progress bars in a Rust thread.
Also make it compatible with traditional Python progress bars (not the most efficient way because the Python still have its own progress thread, but the code change is minimal).
Reviewed By: andll
Differential Revision: D26886282
fbshipit-source-id: ad1f862ced278cc1cb90c37f576eb05f52a50b13
Summary:
Johan Schuijt-Li found they are broken on some CI hosts.
(But other store paths seem okay, which might indicate the fsync list is effective)
Reviewed By: DurhamG
Differential Revision: D26950101
fbshipit-source-id: e3bcd3f77636325be9e9ce8dfded8b17ec68f436
Summary: This diff removes the split between manually managed and autocargo managed Cargo.toml files in `eden/scm/lib`, now all files are autogenerated.
Reviewed By: quark-zju
Differential Revision: D26830884
fbshipit-source-id: 3a5d8409a61347c7650cc7d8192fa426c03733dc
Summary:
There are some reports about truncated latest files on NFS hosts (where the
hgcache portion might be outside NFS).
Fsync files in hgcache too. With indexedlog the total number of files is
bounded so listing them is considered bounded time.
Reviewed By: kulshrax
Differential Revision: D26741889
fbshipit-source-id: a789396cc5802110eb55fa6b52b5124cd9a3dbf3
Summary: It seems there is no "tty" check. So let's disable colors for tests explicitly.
Reviewed By: DurhamG
Differential Revision: D26708620
fbshipit-source-id: 802537af0d3f580c39a746b6aa6a617c9fb4c9c1
Summary:
Recently there are reports about "remote/master" pointing to a non-existed
commit, missing tree data, and general hard-reboot issues. Attempt to reduce
the chance of losing data by adding the critical paths to the fsync list.
Reviewed By: kulshrax
Differential Revision: D26708225
fbshipit-source-id: ae52534fdbf8d9d35498d30c24bdce7f5d9a96a3
Summary:
The Rust IO handles progress and streampager stuff. Switch to it so we don't
need to changing the `fout`, `ferr` when handling streampager in Python.
The chgserver logic is updated to just set raw fd 0, 1, 2 to update stdio,
since `fileno` is no longer exposed from Rust.
Manually tested the following commands, both without chg and with chg:
- lhg log -r . (no pager)
- lhg log (with streampager)
- lhg log --config pager.pager=less (with less pager)
- lhg commit (spawns pager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=off (progress in stderr)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.interface=fullscreen (progress in streampager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.pager='LESS= less' (progress is disabled with external pager)
Reviewed By: sfilipco
Differential Revision: D26612487
fbshipit-source-id: 8b4e36b614a0c080b93e41474f9a8fc33f890083
Summary:
Previously, one has to set both EDENSCM_LOG=trace and EDENSCM_TRACE_LEVEL=trace
to enable all logging. That is because the filtering is global - one subscriber
(or layer) filtering out a span or event, then the span or event is gone
forever.
For now, let's just set the TracingCollector Subscriber's filter level
(EDENSCM_TRACE_LEVEL) to TRACE so it solely depends on the EnvFilter
(EDENSCM_LOG) if EDENSCM_LOG is set. That's more friendly.
See `impl<L: Layer<S>, S: Subscriber> Subscriber for Layered<L, S>`:
fn enabled(&self, metadata: &Metadata<'_>) -> bool {
if self.layer.enabled(metadata, self.ctx()) {
// if the outer layer enables the callsite metadata, ask the subscriber.
self.inner.enabled(metadata)
} else {
// otherwise, the callsite is disabled by the layer
false
}
}
See also: https://github.com/tokio-rs/tracing/issues/302
Reviewed By: sfilipco
Differential Revision: D26518017
fbshipit-source-id: 9016b940621fc9a883e6ca3f00eaaeccc051ae74
Summary: Now IO has internal locking, `mut` is no longer needed.
Reviewed By: sfilipco
Differential Revision: D26518025
fbshipit-source-id: d71e213f6fed24fdefb4c730d576f6cf2fb09d82
Summary:
Similar to D26518021 (3aba8d89b0), provide a way to obtain the "output" stream
that implements `std::io::Write`.
The "output" stream is then used to replace the `std::io::Write`
implementation on the `IO` itself.
This has 2 benefits:
- Removes the `&mut` methods on `IO`. So all `&mut IO` can be replaced by `&IO` without exceptions.
- Make it possible to drop `Clone` on `IO` (strong ref is not clonable, but weak ref can). So we can control the lifetime of the streams more predicatably.
Reviewed By: sfilipco
Differential Revision: D26538452
fbshipit-source-id: b00c65ae0ef5ef4a609104111706365028f47ef7
Summary:
I encountered the error but it's unclear what happened. Print type_name to
clarify.
Now I get:
not implemented: converting non-stdio Write (alloc::vec::Vec<u8>) from Rust to Python is not implemented
This turns out to be an error in "clidispatch: make IO clonable" - The `Drop`
should be implemented on `Inner`, instead of `IO`. The fix has been folded
into that commit.
Reviewed By: sfilipco
Differential Revision: D26518016
fbshipit-source-id: ddb5f828c59cbffa9767694558113167bc96db4b
Summary:
Similar to D26142175 (324f47b1f0), initialize the logger that writes to the error stream of
IO.
Reviewed By: sfilipco
Differential Revision: D26518014
fbshipit-source-id: 2547f4a8f3dba1f624fd00798672f1148869e250
Summary:
We're going to tweak the internals of IO. Making its fields private to allow
that.
Reviewed By: sfilipco
Differential Revision: D26518023
fbshipit-source-id: a1eed46f4f87a3b4b63e56be733102058b4d72d6
Summary:
Added a new `ReadStore` implementation for EdenApi `FileEntry`s. This is pretty much identical to the `TreeEntry` implementation, but calls the `files` method instead, which does not yet support attributes or per-entry errors (although they are both supported on the wire).
I re-use the type-agnostic indexedlog `Entry` adapter for file support for now, to keep things minimal. This unfortunately means the output type of the `Fallback` combinator is a bare `Entry` for both files and trees, losing us some type safety, but I think this is acceptable for now, until I eventually introduce type-safe constructors and value types. Making the file and tree value output types distinct will only be necessary with the introduction of attribute support, or when we'd like to move away from using an "opaque blob" representation for them.
Reviewed By: kulshrax
Differential Revision: D26348577
fbshipit-source-id: 1a930af6e4b5b4d8bacd266c3e51db96cee92dbd
Summary:
Added write support to the `FallbackStore` combinator, along with a flag to control writing to the write store. This enables the "preferred" store to act as a read-through cache for the "fallback" store, with an optional "read only" mode via the "write" flag.
Currently, errors in the write path are silently ignored.
Reviewed By: kulshrax
Differential Revision: D26233602
fbshipit-source-id: acce020a877b87bfee87763e984923c7c51b2b17
Summary:
Introduce `FallbackStore`, a newstore combinator which implements the `ReadStore` trait and first checks a "preferred" store for the provided keys, before falling back to a "fallback" store. The combinator requires that both stores share the same `Key` type, but allows a user provided "value adapter function" to convert from the fallback store's value type to the preferred store's value type.
Currently `FallbackStore` does not support writing missing values to the preferred store - this functionality will be introduced in a future change.
Reviewed By: DurhamG
Differential Revision: D26203146
fbshipit-source-id: 0e99110f93130ff30c95cce15e3dc7873616519c
