Summary: The first byte is the flag.
Reviewed By: sfilipco
Differential Revision: D26654640
fbshipit-source-id: 4d66f14c73fe40a154ca7c08f9a9dff3a54ae337
Summary:
Facebook
It looks like we have quite a few errors when people cloning fbsource on their
laptops (see https://fburl.com/odo8tii0 for more details).
"Couln't resolve host name" is one of the errors we see often, so let's retry
that. ssl errors seem to be retryable as well.
Reviewed By: ikostia
Differential Revision: D26635252
fbshipit-source-id: 2412c4f09aab3d5d45923c4b05fc350556bf9af6
Summary:
See D23966992 (2a2971a4c7) for context. This is a continuation of D23966995 (ab88771161)
which toggles the deserialization part.
Without this change, the default tuple serialization is inefficient
in CBOR and is a disaster in Python:
In [1]: s,f=api.commithashtolocation('fbsource', repo['master'].node(), [bin('375c7ec7442d2bab635f4ae399be6604c5386eb7')])
In [2]: list(s)
Out[2]:
[{'hgid': (55, 92, 126, 199, 68, 45, 43, 171, 99, 95, 74, 227, 153, 190, 102, 4, 197, 56, 110, 183),
'location': {'descendant': (37, 71, 244, 168, 243, 219, 161, 246, 36, 253, 0, 121, 134, 114, 241, 20, 136, 105, 226, 80),
'distance': 10000}}]
With this change, the Python serialization works as expected:
In [2]: list(s)
Out[2]:
[{'hgid': '7\\~\xc7D-+\xabc_J\xe3\x99\xbef\x04\xc58n\xb7',
'location': {'descendant': '%G\xf4\xa8\xf3\xdb\xa1\xf6$\xfd\x00y\x86r\xf1\x14\x88i\xe2P',
'distance': 10000}}]
Places using HgId serialization directly has opt-in explicit serialization
format: D23966986 (0bb45fcbc4) (zstore), D23966991 (f833f03ba2) (metalog), D24009039 (9c5d20904d) (revisionstore) so
they should not be affected.
EdenAPI protocols used in production should be using `WireHgId` types so they
are not affected.
Reviewed By: sfilipco
Differential Revision: D26562685
fbshipit-source-id: 819b794272ee23a135bbed5b61706944ed0acb9f
Summary:
The Rust IO handles progress and streampager stuff. Switch to it so we don't
need to changing the `fout`, `ferr` when handling streampager in Python.
The chgserver logic is updated to just set raw fd 0, 1, 2 to update stdio,
since `fileno` is no longer exposed from Rust.
Manually tested the following commands, both without chg and with chg:
- lhg log -r . (no pager)
- lhg log (with streampager)
- lhg log --config pager.pager=less (with less pager)
- lhg commit (spawns pager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=off (progress in stderr)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.interface=fullscreen (progress in streampager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.pager='LESS= less' (progress is disabled with external pager)
Reviewed By: sfilipco
Differential Revision: D26612487
fbshipit-source-id: 8b4e36b614a0c080b93e41474f9a8fc33f890083
Summary:
Add a new API to test if a stream is a tty. This is needed to replace the
Python `fin`, `fout` etc. to Rust objects, because the Python land requires the
`istty` API. It is also useful for properly implement color detection in the
pure Rust land.
Reviewed By: sfilipco
Differential Revision: D26612480
fbshipit-source-id: 5cf79447b1d74e0031a954788db342afd48dc288
Summary:
Move IO to a separate crate so it is easier to be used in other libraries.
Practically, I'm going to add `is_tty()` API, and make some types from
`cpython-ext` implement it. If `cpython-ext` depends on `clidispatch`,
that would be too heavyweight.
Reviewed By: sfilipco
Differential Revision: D26612488
fbshipit-source-id: 00b18dadce3157a357d189aaa7d985efa6b4c80a
Summary:
I'm going to expose more APIs from the Rust IO such as setting progress
content, etc. Rename the module to clarify.
Reviewed By: sfilipco
Differential Revision: D26612490
fbshipit-source-id: 136ea27a733b09557d02077e68ce51184125ade1
Summary:
Unlike stderr or stdout, writing to progress clears the previous progress text.
Update the function name to clarify.
Reviewed By: sfilipco
Differential Revision: D26612481
fbshipit-source-id: 382182fdcc713f92117d01f53b8da47d195b06cb
Summary:
Update `write_progress` to write the progress to stderr if streampager is not
used.
Reviewed By: sfilipco
Differential Revision: D26612482
fbshipit-source-id: 3034252765024df2a6bda7347eff35f07f3186cb
Summary:
streampager uses "\f" to indicate "the end of progress". Move that logic from
Python to Rust. This hides the streampager implementation detail and allows
us to implement `write_progress` for non-streampager cases.
Reviewed By: sfilipco
Differential Revision: D26612489
fbshipit-source-id: 7f41c3af78a0c098de510f86c86726c43931cb6d
Summary:
There seems no need for the progress bar text to be binary (it is not used to
redirect to binary files). Let's change the type to make future changes easier.
Reviewed By: sfilipco
Differential Revision: D26612479
fbshipit-source-id: 1da7950d5ab2b590963f1591e2f1dd5ae8e0116c
Summary:
The future has a "map" clause that takes GIL too. If they run in different
threads it could deadlock. For example, the following code might hang
forever:
In [1]: s,f=api.commitdata('fbsource',list(repo.nodes('master')))
In [2]: f.wait()
Reviewed By: sfilipco
Differential Revision: D26582806
fbshipit-source-id: e3259850c68b8d48a7a69ed9d47ef75f26179382
Summary:
Previously, one has to set both EDENSCM_LOG=trace and EDENSCM_TRACE_LEVEL=trace
to enable all logging. That is because the filtering is global - one subscriber
(or layer) filtering out a span or event, then the span or event is gone
forever.
For now, let's just set the TracingCollector Subscriber's filter level
(EDENSCM_TRACE_LEVEL) to TRACE so it solely depends on the EnvFilter
(EDENSCM_LOG) if EDENSCM_LOG is set. That's more friendly.
See `impl<L: Layer<S>, S: Subscriber> Subscriber for Layered<L, S>`:
fn enabled(&self, metadata: &Metadata<'_>) -> bool {
if self.layer.enabled(metadata, self.ctx()) {
// if the outer layer enables the callsite metadata, ask the subscriber.
self.inner.enabled(metadata)
} else {
// otherwise, the callsite is disabled by the layer
false
}
}
See also: https://github.com/tokio-rs/tracing/issues/302
Reviewed By: sfilipco
Differential Revision: D26518017
fbshipit-source-id: 9016b940621fc9a883e6ca3f00eaaeccc051ae74
Summary:
Now we use tracing_subscriber for equivalent features (by setting EDENSCM_LOG).
It's no longer necessary to forward to log eco-system.
Reviewed By: sfilipco
Differential Revision: D26518018
fbshipit-source-id: 7cc080faa734ca40eed1ce250df3135ca2626c1f
Summary: Now IO has internal locking, `mut` is no longer needed.
Reviewed By: sfilipco
Differential Revision: D26518025
fbshipit-source-id: d71e213f6fed24fdefb4c730d576f6cf2fb09d82
Summary:
Cloning IO would make it harder to run the clean-up logic in Drop (ex. flush,
stop the pager). Forbid it.
The weak reference output and error streams can still be cloned. They sastify
the main "clone" use-cases.
Reviewed By: sfilipco
Differential Revision: D26538451
fbshipit-source-id: 6cc59214ec5fe1340cb27c7ee1a09658bb83190a
Summary:
Similar to D26518021 (3aba8d89b0), provide a way to obtain the "output" stream
that implements `std::io::Write`.
The "output" stream is then used to replace the `std::io::Write`
implementation on the `IO` itself.
This has 2 benefits:
- Removes the `&mut` methods on `IO`. So all `&mut IO` can be replaced by `&IO` without exceptions.
- Make it possible to drop `Clone` on `IO` (strong ref is not clonable, but weak ref can). So we can control the lifetime of the streams more predicatably.
Reviewed By: sfilipco
Differential Revision: D26538452
fbshipit-source-id: b00c65ae0ef5ef4a609104111706365028f47ef7
Summary:
This makes the IOError stream used by tracing loggers use weak reference
so the main IO can still be dropped without being affected by potentially
lingering IOError values.
Reviewed By: sfilipco
Differential Revision: D26518022
fbshipit-source-id: df4c847a7a59fff39b7832190a8723c78ffae9ad
Summary:
This makes it easier to figure out the "main" IO, similar to APIs like
`std::io::stdout`, without passing the IO around. It will be used by
the `pypager` module.
The global reference is a weak reference so it won't affect IO drop
behavior.
Reviewed By: sfilipco
Differential Revision: D26518015
fbshipit-source-id: ab75ec31cbb8f156b72c94a6fe5b2de6fe9e0bff
Summary: This will be used by pypager.
Reviewed By: sfilipco
Differential Revision: D26518026
fbshipit-source-id: ac8290501ea572cc5ce2067a52d62f4933cbe87b
Summary:
I encountered the error but it's unclear what happened. Print type_name to
clarify.
Now I get:
not implemented: converting non-stdio Write (alloc::vec::Vec<u8>) from Rust to Python is not implemented
This turns out to be an error in "clidispatch: make IO clonable" - The `Drop`
should be implemented on `Inner`, instead of `IO`. The fix has been folded
into that commit.
Reviewed By: sfilipco
Differential Revision: D26518016
fbshipit-source-id: ddb5f828c59cbffa9767694558113167bc96db4b
Summary: Done some reordering of fields in Cargo.toml, added test and doctest = false, name of the target that generated the Cargo.toml file and sorted the cratemap.
Reviewed By: ahornby
Differential Revision: D26581275
fbshipit-source-id: 4c363369438c72d43d8ccf4799f103ff092457cc
Summary:
This diff makes CheckoutPlan::apply take &self instead of self, so that CheckoutPlan can be preserved.
This result in cloning of some paths(but realistically, they probably get moved anyway in this code path, so this should not have perf impact)
This will simplify pycheckout as it will allow to keep plan and later use it to update dirstate
This diff also exposes accessors for dirstate update (e.g. removed_files / updated_content_files / updated_meta_files)
Reviewed By: quark-zju
Differential Revision: D26551118
fbshipit-source-id: 4e2d02da0a692345eae7d5e7124663c577e9029c
Summary:
Make the status fast path print warnings about non-utf8 names and continue on
other valid utf-8 names. It matches the behavior of the Python status code
path.
Before this change, an invalid utf-8 name would just print:
abort: invalid utf-8 sequence of 1 bytes from index 7
without telling which file it is.
Reviewed By: markbt
Differential Revision: D26553514
fbshipit-source-id: 6e26a2a8d738e9001d878a80c0dc82ae6d18a97c
Summary:
Similar to D26142175 (324f47b1f0), initialize the logger that writes to the error stream of
IO.
Reviewed By: sfilipco
Differential Revision: D26518014
fbshipit-source-id: 2547f4a8f3dba1f624fd00798672f1148869e250
Summary:
Add an API to obtain an `io::Write` object for the error stream.
It will be used by next changes.
Reviewed By: sfilipco
Differential Revision: D26518021
fbshipit-source-id: bc39fa54f83be8ff2c8fbc803700c4fe57e814bd
Summary:
Make it possible for IO to be cloned. This allows, for example, tracing or
progress bar logic to keep a reference of IO.
Reviewed By: sfilipco
Differential Revision: D26518028
fbshipit-source-id: d5ca7f4347251c68f7491f784f647ee80c47d159
Summary:
We're going to tweak the internals of IO. Making its fields private to allow
that.
Reviewed By: sfilipco
Differential Revision: D26518023
fbshipit-source-id: a1eed46f4f87a3b4b63e56be733102058b4d72d6
Summary: Add parking_lot dep. It will be used later.
Reviewed By: sfilipco
Differential Revision: D26518019
fbshipit-source-id: 76124a2f70731b30017a6d7019eef9dc856fe54a
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary: Introduce `FetchError` type, with separates "not found" errors (which require a key), from other errors (which may or may not have an associated key). This allows us to easily fall back for only "not found" errors propagate other fetch errors to the caller. The current version of `FetchError` still does not eliminate redundantly storing the key for cases like EdenApi, where the key may be part of the error type itself. This optimization may not be worthwhile after we refactor these store implementations to use `HgId` instead of `Key`.
Reviewed By: kulshrax
Differential Revision: D26410215
fbshipit-source-id: e6198d54de64b41ff696cabd64affc8dbaa41cf9
Summary: Add a wrapper type, `LegacyDatastore`, along with a `ReadStore` implementation that reads via the `HgIdDataStore` trait, for any wrapped types which implement that trait.
Reviewed By: kulshrax
Differential Revision: D26381207
fbshipit-source-id: d28dc3095adf20403f8b993d5a939d1e41c77462
Summary:
Added a new `ReadStore` implementation for EdenApi `FileEntry`s. This is pretty much identical to the `TreeEntry` implementation, but calls the `files` method instead, which does not yet support attributes or per-entry errors (although they are both supported on the wire).
I re-use the type-agnostic indexedlog `Entry` adapter for file support for now, to keep things minimal. This unfortunately means the output type of the `Fallback` combinator is a bare `Entry` for both files and trees, losing us some type safety, but I think this is acceptable for now, until I eventually introduce type-safe constructors and value types. Making the file and tree value output types distinct will only be necessary with the introduction of attribute support, or when we'd like to move away from using an "opaque blob" representation for them.
Reviewed By: kulshrax
Differential Revision: D26348577
fbshipit-source-id: 1a930af6e4b5b4d8bacd266c3e51db96cee92dbd
Summary:
Added write support to the `FallbackStore` combinator, along with a flag to control writing to the write store. This enables the "preferred" store to act as a read-through cache for the "fallback" store, with an optional "read only" mode via the "write" flag.
Currently, errors in the write path are silently ignored.
Reviewed By: kulshrax
Differential Revision: D26233602
fbshipit-source-id: acce020a877b87bfee87763e984923c7c51b2b17
Summary: Introduces a minimal WriteStore trait, with a basic implementation for IndexedLog. Currently, there's no explicit key in the `WriteStream`. Instead, it is implicit that the value type to be written should contain a key. In the future, we may introduce an abstraction over value types which do and do not contain a key.
Reviewed By: kulshrax
Differential Revision: D26213028
fbshipit-source-id: 75867ff7229d594346e7e7624e8ca91f8205fed4
Summary: This helps with pipelining (once chunk is prefetched it will start rolling out to fs) and also helps with limiting number of keys prefetched at once
Reviewed By: quark-zju
Differential Revision: D26496789
fbshipit-source-id: 9aa6b08c1605ab2a06a02347897f5dcfa22297fd
Summary: Also removing todo: test from checkout plan as there are tests already
Reviewed By: singhsrb
Differential Revision: D26496790
fbshipit-source-id: eb99771cb0cda7a90477190875997027b25e2482
Summary:
Move the logic to a common method. A side effect is that
IndexedlogIdDag needs to calculate `last_span.high` again, and
InProcessIdDag needs to calculate `last_store_id` again. But it's
probably okay since the merge is rare and the InProcessIdDag operations should
be considered fast even if it's called frequently.
Reviewed By: sfilipco
Differential Revision: D26504915
fbshipit-source-id: e215ffe9fef7948b7b662dba3ed990e374be989a
Summary: Add a few more tests to the generic IdDag tests.
Reviewed By: sfilipco
Differential Revision: D26415514
fbshipit-source-id: 5ef2a1d4e03527b184a07d94de91c64a05427b90
Summary: It is a constant `true` now. Inline the logic and remove the field.
Reviewed By: sfilipco
Differential Revision: D26414838
fbshipit-source-id: ceace835ecfd263ba16c3dead41ce6ba95087e4f
Summary:
Similar to the previous diff. Implement the merge behavior for InProcess IdDag.
The feature is not turned on yet for easier review. It is turned on by the
next diff.
Reviewed By: sfilipco
Differential Revision: D26414839
fbshipit-source-id: 2a485e0dffcd9d8e2ad95380159d11342636f9aa
