Summary:
Migrate most crates to tokio 1.0. The exception is edenfs-client, which has
some dependencies on `//common/rust/shed/fbthrift_ext` and seems non-trivial
to upgrade. It creates a separate tokio runtime so it shouldn't be affected
feature-wise.
Reviewed By: singhsrb
Differential Revision: D26152862
fbshipit-source-id: c84c43b1b1423eabe3543bccde34cc489b7805be
Summary: This implementation is used for all things that are cached in Monononoke.
Reviewed By: quark-zju
Differential Revision: D26121497
fbshipit-source-id: a0088b539f3c3656921ab9a7a25c6442996aed18
Summary:
For fsnodes and skeleton manifests it should be possible to allow gaps in the
commits that are backfilled. Any access to a commit in one of these gaps can
be quickly derived from the nearest ancestor that is derived. Since each
commit's derived data is independent, there is no loss from omitting them.
Add the `--gap-size` option to `backfill_derived_data`. This allows `fsnodes`,
`skeleton_manifests` and other derived data types that implement it in the
future to skip some of the commits during backfill. This will speed up
backfilling and reduced the amount of data that is stored for infrequently
accessed historical commits.
Reviewed By: StanislavGlebik
Differential Revision: D25997854
fbshipit-source-id: bf4df3f5c16a913c13f732f6837fc4c817664e29
Summary:
Practically there could be a lot of duplicated strings. For example, `module`
name could be the same for all functions in a file. `cat` could be a commonly
used field name, etc. Intern the strings so we don't waste too much space.
Reviewed By: sfilipco
Differential Revision: D26013748
fbshipit-source-id: 2740319019ce5058acfd7a9d2bb972c08a9386de
Summary:
To make Python code integrate deeper with the Rust tracing eco-system (ex. be
aware of layered subscribers, and filters), add APIs to create tracing Callsite
on the fly.
Unfortunately, the tracing APIs make it a bit too verbose.
Reviewed By: sfilipco
Differential Revision: D26003017
fbshipit-source-id: c3a36bcbaf4b7bbeafc3e2a8ddfdc7193c0972d3
Summary:
Limit the span or event count that a callsite can log so we can handle highly
repetitive calls. This is similar to D23307793 (d8e775f423) but applies to the Rust tracing
APIs.
Reviewed By: sfilipco
Differential Revision: D26090876
fbshipit-source-id: 394f52d697fc5e9ba5b900cebc9405a9f29220ce
Summary: Those APIs are internal details. Public APIs do not need them.
Reviewed By: sfilipco
Differential Revision: D26090877
fbshipit-source-id: b6e3ac76de719d1db71c2b57da3e8ef76015089e
Summary:
The name is being taken by stdlib:
warning: a method with this name may be added to the standard library in the future
--> eden/scm/lib/dag/src/spanset.rs:228:14
|
228 | .binary_search_by(|probe| span.low.cmp(&probe.low))
| ^^^^^^^^^^^^^^^^
|
= note: `#[warn(unstable_name_collisions)]` on by default
= warning: once this method is added to the standard library, the ambiguity may cause an error or change in behavior!
= note: for more information, see issue #48919 <https://github.com/rust-lang/rust/issues/48919>
= help: call with fully qualified syntax `BinarySearchBy::binary_search_by(...)` to keep using the current method
= help: add `#![feature(vecdeque_binary_search)]` to the crate attributes to enable `VecDeque::<T>::binary_search_by`
Reviewed By: sfilipco
Differential Revision: D26092424
fbshipit-source-id: d2cdf7d73d2f808f038817c9dc9f4c531ff643bd
Summary:
Bytecode generation entry point `compile_ffi::compile_from_text_cpp_ffi(...)` and a driver to exercise it: `hh_single_compile_cpp`.
What we get here is a C FFI for generating HHAS from source text. We can tweak the interface as we need to going forward but this will do for a first version and I think has roughly feature parity with the OCaml FFI:
- `compile_ffi.rs` defines the function;
- `compile_ffi.h` defines the C interface;
- `hh_single_compile_cpp.cpp` provides a C++ CLI that calls the function.
Reviewed By: shiqicao
Differential Revision: D25979967
fbshipit-source-id: 4b46f9af23c61150dda6c33f9fa14e2c455c54c2
Summary:
This diff revives D25454687 (f98273063a), which was backed out by D25792491 (b52168c4c8) because it was causing Mercurial to crash in certain environments where certificates are configured incorrectly.
I've modified the code so that by default, certificates are not validated (maintaining the old behavior), but users of the API can opt-in to validation. In the case of EdenAPI (which is the only user that opts in), this is controlled via a new `edenapi.validate-certs` config option, which defaults to false. This allows enforcing validation on platforms where the configs should be correct (such as devservers) while maintaining the old behavior on other platforms by default.
Reviewed By: DurhamG
Differential Revision: D26009207
fbshipit-source-id: 904dee61fd12fdee4a0031d14adef7fdb4801139
Summary:
Previously the LazySet only supports non-async Iterator. This makes it more
flexible useful. It will be used in upcoming changes.
Reviewed By: sfilipco
Differential Revision: D25858800
fbshipit-source-id: 8c8e874f05cfab721bc0fa55160a9337ed7c2c27
Summary:
In the future we'd like to allow building the dag crate without the indexedlog
portion. This diff adds support for that.
Reviewed By: DurhamG
Differential Revision: D25769054
fbshipit-source-id: eb5a200841f878836a9f68e65e7d50be7e6b9a79
Summary:
In the future we want to build dag without indexedlog dep for Mononoke
use-case. One of the problem is the ToWire trait implemented on dag::Id by
edenapi-types. Within buck, the dag crate will have 2 targets: dag and dag-lite
(no indexedlog). They are incompatible meaning that edenapi-types depending on
dag-lite will not provide Id::to_wire for crates using dag, or vice-versa.
To solve that, we move the Id and other types to a separate crate that only has
one buck target so edenapi-types, and segmented_changelog from Mononoke can
depend on it without issues. This also makes edenapi-types more lightweight.
Reviewed By: sfilipco
Differential Revision: D25857917
fbshipit-source-id: d3e15a2b6638cc6e15171a1e9bc37362e03df583
Summary: In upcoming changes, we're moving Id to a separate crate. This makes that easier.
Reviewed By: sfilipco
Differential Revision: D25857918
fbshipit-source-id: 6e2163f6fa171d4cd3be4fc0c4c248fd87ba739c
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.
This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).
The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.
Reviewed By: farnz
Differential Revision: D25952428
fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
Summary:
This new crate is part of the new telemetry / logging effort. Its input is
tracing data, and output is aggregated NoSQL table content.
This diff is only the start, setting up the direction.
Reviewed By: DurhamG
Differential Revision: D19797702
fbshipit-source-id: bdf34461c05b5eae5e59652bc82d8ee1857dbf1e
Summary:
This setup is more extendable than the TracingData focused approach. We can
more easily add new functionality using the Subscriber list.
The approach taken here to introduce the new collector tries to maintain
existing functionality. We can then move various logic to their own
Subscribers.
Reviewed By: quark-zju
Differential Revision: D25988580
fbshipit-source-id: 045cd355dbd499109e554a29a1439c2d490b7c40
Summary:
Dot `.` is the common separator for the metrics aggregator that we use.
This adds some form of consistency.
Reviewed By: DurhamG
Differential Revision: D25968398
fbshipit-source-id: 194d2f33fe477fe5d768a9cd8f9f46f56445e3e8
Summary:
Make local build is broken with
```
building 'hgmain' binary
error: failed to select a version for the requirement `indicatif = "^0.14"`
candidate versions found which didn't match: 0.15.0
location searched: directory source `/data/users/liubovd/fbsource/fbcode/eden/scm/../../../third-party/rust/vendor` (which is replacing registry `https://github.com/rust-lang/crates.io-index`)
required by package `hgcommands v0.1.0 (/data/users/liubovd/fbsource/fbcode/eden/scm/lib/hgcommands)`
... which is depended on by `hgmain v0.1.0 (/data/users/liubovd/fbsource/fbcode/eden/scm/exec/hgmain)`
perhaps a crate was updated and forgotten to be re-vendored?
error: compilation of Rust target 'hgmain' failed
make: *** [Makefile:68: local] Error 1
```
Reviewed By: krallin
Differential Revision: D25973253
fbshipit-source-id: 42c66dcfe26dc5c7713d4bdedc2b353aa5eb20ed
Summary:
This feature is useful for testing time-dependent stuff (e.g. it
allows you to stop/forward time). It's already included in the buck build.
Reviewed By: SkyterX
Differential Revision: D25946732
fbshipit-source-id: 5e7b69967a45e6deaddaac34ba78b42d2f2ad90e
Summary:
quark-zju made it much safer to rollout configs recently by enforcing
time-sharding, but one little downside is that it becomes harder to test
them locally (or at diff-time, which I'm working on adding) because the time
shard will usually not pass.
This diffs add an environment variable to force all shards to pass.
Reviewed By: DurhamG
Differential Revision: D25925207
fbshipit-source-id: 343b90165ad2b6bae64b6821eae95c58f7d79698
Summary:
Use the new API to clean up stale logs at open time. This hopefully helps
releasing disk space on Windows if the normal rotation fails to remove
old files being mmap-ed by other processes.
Reviewed By: xavierd
Differential Revision: D25894282
fbshipit-source-id: a3d8247b737dd451ee68b58cc5a38fdd2822c0c3
Summary:
Previously rotation only happens at flush time and file deletion is a best
effort (it might fail on Windows). For use-cases that are sensitive about
space usage that's suboptimal. This diff adds an API to remove old files
manually so high level logic can choose to clean up old files after open().
Reviewed By: xavierd
Differential Revision: D25894283
fbshipit-source-id: fbffff426544b39349ddf3537d46954d3cab5d12
Summary:
We used to get those in the old (Python) LFS extension, but didn't have them in
the new one. However, this is helpful to correlate requests to LFS with data in
hg logs. It's also convenient to be able to identify whether a set of requests
are part of the same session or not.
This diffs threads the client correlator through to the LFS store from the
Python, similarly to how it's done in EdenAPI.
Reviewed By: DurhamG
Differential Revision: D25804930
fbshipit-source-id: a5d5508617fa4184344834bbd8e3423816aa7668
Summary: The dependency that we want to share is 0.14 instead of 0.15.
Reviewed By: singhsrb
Differential Revision: D25871110
fbshipit-source-id: 16e9f8a858ee04a47867c2916909edfc996f8bc4
Summary:
Rough progress reporting. The progress bars are straight coming from the
`indicatif` crate. Integrating with the `IO` object is not trivial because
we only have a reference. It gets tricky. I think that it makes sense for
us to expand the IO object to something that is more than a `Box<dyn Write>`.
We have about 3 scenarios:
1. Write object that we need to implement interior mutability for to give out
clones.
2. Stdin/Stdout which have their own imlementation for interior mutability.
3. PyObject which has locking already implemented.
(Note: this ignores all push blocking failures!)
Reviewed By: quark-zju
Differential Revision: D25840469
fbshipit-source-id: 87f466f06f2c5d4c63ccb3bbc5c009fae41ed002
Summary:
There is a little bug here. We produce a stream of futures of futures, then we
buffer it, which gives us a stream of futures, and then we await the futures
one by one, here:
```
while let Some(next) = stream.next().await {
next.await?
}
```
This is not really correct, because it means we don't actually do fetches
concurrently at all (we just instantiate futures concurrently, but that's not
really async work).
This fixes that by removing one layer of future-ing.
Reviewed By: singhsrb
Differential Revision: D25825895
fbshipit-source-id: 3ad3367f1eb802ce5b9b5288f04fd3705e172537
Summary:
This allows adding progress bars tracking downloads from the server.
We could be smarter in this instance if we were to deserialize on the fly.
The first part of the payload contains the number of idmap entries that we need
but it needs more work to make it clear. The progress object right now is
designed for general bytes.
Reviewed By: quark-zju
Differential Revision: D25840470
fbshipit-source-id: c466c8d606b44981fe63c95352db2d8f14d6071b
Summary:
The most common scenario where we see matcher errors is when we iterate through
a manifest and the user sends SIGTERM to the process. The matcher may be both
Rust and Python code. The Python code handles the interrupt and prevents future
function calls. The iterating Rust code will continue to call matcher functions
through this time so we get matcher errors from the terminated Python stack.
As long as we have Python matcher code, errors are valid.
It is unclear to me whether the matcher trait should have `Result` return
values when all implementations are Rust. It is easy to imagine implementations
that can fail in different circumstances but the ones that we use if we just
port the Python code wouldn't fail.
All in all, I think that this is a reasonable step forward.
Reviewed By: quark-zju
Differential Revision: D25697099
fbshipit-source-id: f61c80bd0a8caa58040a447ed02d48a1ae84ad60
Summary:
The cert path isn't correctly set up on all platforms, so this can
cause Mercurial to throw an error complaining about missing certs, even when
edenapi isn't enabled.
Let's back this out for now until we can fix the cert paths or only hit this
path when we actually use edenapi.
Reviewed By: singhsrb
Differential Revision: D25792491
fbshipit-source-id: 022a89a089cabcc709a07934eb62b883082261c2
Summary:
Lots of things can look like CBOR data, such as ... strings representing
errors. Right now, if the data in our CBOR stream is actually an error message,
then we'll just ignore it (see details in T80406893).
This isn't how we normally handle invalid data on the stream (we'd raise an
error) — it only happens with trailing data. This fixes our decoding to raise
an error in this case.
Reviewed By: quark-zju
Differential Revision: D25759082
fbshipit-source-id: c3d8be5007112ec1d2e7f25a102d8caaf0dbba56
Summary:
In the previous diff I had to make the same change in two places, this change
deduplicates the code so we can reuse the change. This isn't 100% equivalent,
since now we have 2 layers of boxing on the stream in `Fetch`.
That being said, that seems quite unlikely to matter considering that this is
ultimately handling responses that came to us over HTTP, so one pointer
traversal seems to be reasonable overhead (also, similar experience in Mononoke
suggests it really does not matter).
Reviewed By: quark-zju
Differential Revision: D25758652
fbshipit-source-id: 399ead1b67ffbb241597615a29129411580cf194
Summary:
This updates the edenapi fetch mechanism to check status codes from the server.
If the server responds with an error, we propagate the error up to the caller.
This is equivalent to what we would do if e.g. the server had just crashed.
Reviewed By: quark-zju
Differential Revision: D25758653
fbshipit-source-id: f44f6384be7944dce670c3825ccbb60b5fa2090a
Summary:
Like it says in the title. Note that I did *not* retry stuff like resolving
hosts or connecting, so this should only really temporary blips in
connectivity. We probably shouldn't go much beyond that at a low level like
this.
Reviewed By: HarveyHunt
Differential Revision: D25615915
fbshipit-source-id: 78c33eff2e9ce380a260708e9fbeb929eede383c
Summary:
This is the goal of this stack: retry errors that occur when Curl detects that
the transfer speed is too low. This should let us eventually set a much higher
timeout on overall request completion, thus ensuring that we don't uploads
that make progress, all the while aborting connections early and retrying them
if they are legitimately stuck.
Reviewed By: farnz
Differential Revision: D25615790
fbshipit-source-id: fe294aee090758b1a3aef138788ac2926c741b79
Summary:
Right now, the error handling in LFS doesn't handle e.g. transfer timeouts. I'd
like us to support that, notably so that we can have curl require a minimum
transfer speed and retry if we fail.
To do so, I need to be able to capture the errors and figure out if they're
retryable. Right now, everything is either a `FetchError` that includes a HTTP
status and URL, or just an `Error` that aborts.
This diff introduces a `TransferError` that explains why a transfer failed and
can be used for retry decisions. We later add the request details if we decide
to not retry and turn it into a `FetchError`.
Reviewed By: xavierd
Differential Revision: D25615789
fbshipit-source-id: e4a2f4f16a34ca2f86bd61491bb26e7f328dec63
Summary:
Like it says in the title. This adds support for setting a min-transfer-speed
in Curl. My goal with this is to fix two problems we have:
- a) Uploads that timeout on slow connections. Right now we set a transfer
timeout on requests, but given files to upload can be arbitrarily large, that
can fail. This happened earlier this week to a user (T81365552).
- b) Transfer timeouts in LFS. Right now, we have a very high timeout on
requests and we can't lower it due to this problem with uploads. Besides,
the reason for lowering the timeout would be to retry thing, but right now
we don't support this anyway.
Reviewed By: xavierd
Differential Revision: D25615788
fbshipit-source-id: 57d75ee8f522cf8524f9d12103e34b0765b6846a
Summary:
I'd like to make it a little easier to add more options without having to
thread them all the way through to the HTTP transfer callsite.
Reviewed By: xavierd
Differential Revision: D25615787
fbshipit-source-id: 4c6274dc2e6b5ba878e0027aae9a08b04f974463
Summary:
The intention was to sort entries by Dag Id entry. This was instead sorted
lexicographically.
Reviewed By: quark-zju
Differential Revision: D25684784
fbshipit-source-id: 0a3db6398aec7d8df080bbb2366e41660483608c
Summary:
`PartialOrd` was suggested by sfilipco. Note `Option<std::cmp::Ordering>` is
similar to `Side` in terms of expressiveness. `PartialOrd` can be written
using shorter symbols (`<=`, etc) so it's easier to understand.
The `compatible` family APIs were replaced by `partial_cmp` APIs.
There are some minor differences:
- Bitwise or used by union set is no longer supported. `Hints::union` was
added as a replacement.
- `Option<T>` implements full order. `Some(T) > None`. This is different
from `compatible_dag` and `compatible_id_map` APIs. Additional `> None`
checks were added for correctness.
Reviewed By: sfilipco
Differential Revision: D25652784
fbshipit-source-id: 51d88948fa556300678050088c06e9dda09cbf98
