Summary:
This diff adds megarepo service methods to SCS. Each of the methods just returns "not implemented" at the moment.
The main goal of this diff is to provide a reference against which a sample client can be built.
This diffs makes two decisions that came from the source control discussion presentation:
1. expensive calls are asynchronous. This means that they are separated into an initial "request placing" call, which returns a polling token; and a polling call, which takes said token and blocks until it either times out or until the request is satisfied
1. racy calls take target location and only advance target if this location does not change until the advancement time comes. This ensures that the clients are aware that changing sync config versions or re-merging sources should be done why `megarepo_sync_changeset` calls are paused
There are also some decisions that I made while writing this, which may be questionable:
1. The polling token types are just string aliases. This is fine atm, but may be a problem in the future if we want to implement `AddScubaResponse` or `AddScubaParams` for some but not others. Moreover, now I had to implement both traits for `String`, which I am also not too happy about. Maybe I should turn polling tokens into wrapper structs right now
1. all new methods start with `megarepo_` or `poll_megarepo`. Generally I dislike name-based namespacing, but in this case the alternative is to either keep the names like `sync_changeset` (not descriptive enough), or move these methods into a separate service definition (too much overhead, and probably not desirable from operational perspective anyways). So I went with prefixes.
1. **the most questionable decision IMO** is about a config design. ATM the `SyncTargetConfig` type contains the `Target` internally. At the same time, `MegarepoChangeTargetConfigParams` contains both the `Target` and the `SyncTargetConfig` and will error out if `SyncTargetConfig` refers to a different target. The API could be made simpler by just taking `SyncTargetConfig`, but it feels a bit wrong to not have the client give us the target they want to change the config for. I can be easily persuaded to remove `Target` from `MegarepoChangeTargetConfigParams`. It can also be done incrementally, until we have any real clients, so figuring this out now is not important.
Reviewed By: markbt
Differential Revision: D26549580
fbshipit-source-id: 2d38d6273fb5ec46b846a242f4f1abf61502cb1c
Summary:
The Rust IO handles progress and streampager stuff. Switch to it so we don't
need to changing the `fout`, `ferr` when handling streampager in Python.
The chgserver logic is updated to just set raw fd 0, 1, 2 to update stdio,
since `fileno` is no longer exposed from Rust.
Manually tested the following commands, both without chg and with chg:
- lhg log -r . (no pager)
- lhg log (with streampager)
- lhg log --config pager.pager=less (with less pager)
- lhg commit (spawns pager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=off (progress in stderr)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.interface=fullscreen (progress in streampager)
- lhg debugprogress -s 100 --sleep 100 --with-output --pager=on --config pager.pager='LESS= less' (progress is disabled with external pager)
Reviewed By: sfilipco
Differential Revision: D26612487
fbshipit-source-id: 8b4e36b614a0c080b93e41474f9a8fc33f890083
Summary:
Previously those fds were obtained via `fp.fileno()`. We're going to replace
the `fp` with something backed by Rust that does not have `fileno()`. Update
chg to replace the raw fds (0, 1, 2) directly. This is enough to affect both
Python and Rust's stdio so there is no need to replace `ui.fout`, `ui.ferr`
again.
Reviewed By: sfilipco
Differential Revision: D26612484
fbshipit-source-id: 5cd89e5955a1dcaad3d3132730354ee67c016bf0
Summary:
Write "\n" to flush the lines. This makes the lines visible instead of being
rewritten by the progress bar.
Reviewed By: sfilipco
Differential Revision: D26612483
fbshipit-source-id: 00cdc4af4dbed479532cc93fe183a19244c8c7c6
Summary:
Add a new API to test if a stream is a tty. This is needed to replace the
Python `fin`, `fout` etc. to Rust objects, because the Python land requires the
`istty` API. It is also useful for properly implement color detection in the
pure Rust land.
Reviewed By: sfilipco
Differential Revision: D26612480
fbshipit-source-id: 5cf79447b1d74e0031a954788db342afd48dc288
Summary:
Move IO to a separate crate so it is easier to be used in other libraries.
Practically, I'm going to add `is_tty()` API, and make some types from
`cpython-ext` implement it. If `cpython-ext` depends on `clidispatch`,
that would be too heavyweight.
Reviewed By: sfilipco
Differential Revision: D26612488
fbshipit-source-id: 00b18dadce3157a357d189aaa7d985efa6b4c80a
Summary:
Rename `pager` to `IO` matching the Rust IO APIs. This allows us to use the
Rust IO APIs from Python.
Reviewed By: sfilipco
Differential Revision: D26612485
fbshipit-source-id: d78f1c8966429e563fe7d190bbcb3bad1b0710fd
Summary:
I'm going to expose more APIs from the Rust IO such as setting progress
content, etc. Rename the module to clarify.
Reviewed By: sfilipco
Differential Revision: D26612490
fbshipit-source-id: 136ea27a733b09557d02077e68ce51184125ade1
Summary:
Unlike stderr or stdout, writing to progress clears the previous progress text.
Update the function name to clarify.
Reviewed By: sfilipco
Differential Revision: D26612481
fbshipit-source-id: 382182fdcc713f92117d01f53b8da47d195b06cb
Summary:
Update `write_progress` to write the progress to stderr if streampager is not
used.
Reviewed By: sfilipco
Differential Revision: D26612482
fbshipit-source-id: 3034252765024df2a6bda7347eff35f07f3186cb
Summary:
streampager uses "\f" to indicate "the end of progress". Move that logic from
Python to Rust. This hides the streampager implementation detail and allows
us to implement `write_progress` for non-streampager cases.
Reviewed By: sfilipco
Differential Revision: D26612489
fbshipit-source-id: 7f41c3af78a0c098de510f86c86726c43931cb6d
Summary:
There seems no need for the progress bar text to be binary (it is not used to
redirect to binary files). Let's change the type to make future changes easier.
Reviewed By: sfilipco
Differential Revision: D26612479
fbshipit-source-id: 1da7950d5ab2b590963f1591e2f1dd5ae8e0116c
Summary:
This makes existing ui.debug appear in tracing debug output with correct
callsite information. They can then be filtered by the `EDENSCM_LOG`. For
example:
$ EDENSCM_LOG=edenscm=debug lhg dbsh
In [1]: ui.debug('foo')
Feb 19 20:03:31.795 DEBUG edenscm.hgext.debugshell: foo
Reviewed By: sfilipco
Differential Revision: D26582807
fbshipit-source-id: b3d97d43d8a31c7d42e3b5c86a05142206e2c9f7
Summary:
It's silly to use `eden prefetch --no-prefetch` to efficiently glob
for filenames. Introduce an `eden glob` command which resolves a glob
relative to the current working directory.
Reviewed By: genevievehelsel
Differential Revision: D25450358
fbshipit-source-id: 45d6dc870d21510e51d5662c75e80385886899fc
Summary:
I want to use EdenError from some code outside of service/, so move
EdenError into utils.
Reviewed By: genevievehelsel
Differential Revision: D25447438
fbshipit-source-id: 2d1ddfa379238369679e84708518a9ba106f76b9
Summary:
We shouldn't delete from a dictionary while iterating over it, instead we should iterate over a copy and then delete from the original.
`.keys()` returns a view of the dict, while wrapping it in `list` makes a deep copy.
Reviewed By: StanislavGlebik
Differential Revision: D26618782
fbshipit-source-id: 234503aab0a0a2151ae0818282651b547f414016
Summary:
The future has a "map" clause that takes GIL too. If they run in different
threads it could deadlock. For example, the following code might hang
forever:
In [1]: s,f=api.commitdata('fbsource',list(repo.nodes('master')))
In [2]: f.wait()
Reviewed By: sfilipco
Differential Revision: D26582806
fbshipit-source-id: e3259850c68b8d48a7a69ed9d47ef75f26179382
Summary:
Previously, one has to set both EDENSCM_LOG=trace and EDENSCM_TRACE_LEVEL=trace
to enable all logging. That is because the filtering is global - one subscriber
(or layer) filtering out a span or event, then the span or event is gone
forever.
For now, let's just set the TracingCollector Subscriber's filter level
(EDENSCM_TRACE_LEVEL) to TRACE so it solely depends on the EnvFilter
(EDENSCM_LOG) if EDENSCM_LOG is set. That's more friendly.
See `impl<L: Layer<S>, S: Subscriber> Subscriber for Layered<L, S>`:
fn enabled(&self, metadata: &Metadata<'_>) -> bool {
if self.layer.enabled(metadata, self.ctx()) {
// if the outer layer enables the callsite metadata, ask the subscriber.
self.inner.enabled(metadata)
} else {
// otherwise, the callsite is disabled by the layer
false
}
}
See also: https://github.com/tokio-rs/tracing/issues/302
Reviewed By: sfilipco
Differential Revision: D26518017
fbshipit-source-id: 9016b940621fc9a883e6ca3f00eaaeccc051ae74
Summary:
Now we use tracing_subscriber for equivalent features (by setting EDENSCM_LOG).
It's no longer necessary to forward to log eco-system.
Reviewed By: sfilipco
Differential Revision: D26518018
fbshipit-source-id: 7cc080faa734ca40eed1ce250df3135ca2626c1f
Summary: Now IO has internal locking, `mut` is no longer needed.
Reviewed By: sfilipco
Differential Revision: D26518025
fbshipit-source-id: d71e213f6fed24fdefb4c730d576f6cf2fb09d82
Summary:
Cloning IO would make it harder to run the clean-up logic in Drop (ex. flush,
stop the pager). Forbid it.
The weak reference output and error streams can still be cloned. They sastify
the main "clone" use-cases.
Reviewed By: sfilipco
Differential Revision: D26538451
fbshipit-source-id: 6cc59214ec5fe1340cb27c7ee1a09658bb83190a
Summary:
Similar to D26518021 (3aba8d89b0), provide a way to obtain the "output" stream
that implements `std::io::Write`.
The "output" stream is then used to replace the `std::io::Write`
implementation on the `IO` itself.
This has 2 benefits:
- Removes the `&mut` methods on `IO`. So all `&mut IO` can be replaced by `&IO` without exceptions.
- Make it possible to drop `Clone` on `IO` (strong ref is not clonable, but weak ref can). So we can control the lifetime of the streams more predicatably.
Reviewed By: sfilipco
Differential Revision: D26538452
fbshipit-source-id: b00c65ae0ef5ef4a609104111706365028f47ef7
Summary:
Use the global IO state instead of keeping a separate IO state.
This makes the Python pager logic able to affect the tracing logger's error
output. For example, the following command will write the tracing logs to
the streampager's stderr channel as expected:
EDENSCM_LOG=trace lhg log -r '.~1000::.' --config pager.pager=internal:streampager
Traditional pager also works:
EDENSCM_LOG=trace lhg log -r '.~1000::.' --config pager.pager=less
Reviewed By: sfilipco
Differential Revision: D26518024
fbshipit-source-id: c7020b7467f9392af8c2a58d064339d05bce35ce
Summary:
This makes the IOError stream used by tracing loggers use weak reference
so the main IO can still be dropped without being affected by potentially
lingering IOError values.
Reviewed By: sfilipco
Differential Revision: D26518022
fbshipit-source-id: df4c847a7a59fff39b7832190a8723c78ffae9ad
Summary:
Set the global IO reference to the IO created by hgmain. This allows
other crates like `pypager` to use the `IO` without cloning or passing
the reference across layers (Rust references do not actually work for
across the Python boundary).
Reviewed By: sfilipco
Differential Revision: D26518027
fbshipit-source-id: 6144b06035d0ef9384cc1a37245c306071e35a9d
Summary:
This makes it easier to figure out the "main" IO, similar to APIs like
`std::io::stdout`, without passing the IO around. It will be used by
the `pypager` module.
The global reference is a weak reference so it won't affect IO drop
behavior.
Reviewed By: sfilipco
Differential Revision: D26518015
fbshipit-source-id: ab75ec31cbb8f156b72c94a6fe5b2de6fe9e0bff
Summary: This will be used by pypager.
Reviewed By: sfilipco
Differential Revision: D26518026
fbshipit-source-id: ac8290501ea572cc5ce2067a52d62f4933cbe87b
Summary:
I encountered the error but it's unclear what happened. Print type_name to
clarify.
Now I get:
not implemented: converting non-stdio Write (alloc::vec::Vec<u8>) from Rust to Python is not implemented
This turns out to be an error in "clidispatch: make IO clonable" - The `Drop`
should be implemented on `Inner`, instead of `IO`. The fix has been folded
into that commit.
Reviewed By: sfilipco
Differential Revision: D26518016
fbshipit-source-id: ddb5f828c59cbffa9767694558113167bc96db4b
Summary:
The `wait_for_bookmark_update` function only waits for `mononoke_scs_server` to
update to the new bookmark value. `mononoke` might still be refreshing the
warm bookmark cache at this point, and might not have the value for `newkilo`
yet. Flush its bookmarks cache to ensure we get an up-to-date view.
Differential Revision: D26609089
fbshipit-source-id: c7c063e4eb96303d4fbd597c4153335a3c9d4429
Summary:
As useful as `eden strace` can be, it's hard to correlate inode
numbers to their filenames via the mkdir/create/lookup request that
returned the inode. Add logging for result codes.
Reviewed By: kmancini
Differential Revision: D26287958
fbshipit-source-id: dae4b56513831185b038546f238938b0d21a6bad
Summary: Done some reordering of fields in Cargo.toml, added test and doctest = false, name of the target that generated the Cargo.toml file and sorted the cratemap.
Reviewed By: ahornby
Differential Revision: D26581275
fbshipit-source-id: 4c363369438c72d43d8ccf4799f103ff092457cc
Summary:
The world has moved on utf-8 as the default encoding for files and data, but
EdenFS still accepts non utf-8 filenames to be written to it. In fact, most of
the time when a non utf-8 file is written to the working copy, and even though
EdenFS handles it properly, Mercurial ends up freaking out and crash. In all of
these cases, non-utf8 files were not intentional, and thus refusing to create
them wouldn't be a loss of functionality.
Note that this diff makes the asumption that Mercurial's manifest only accept
utf8 path, and thus we only have to protect against files being created in the
working copy that aren't utf8.
The unfortunate part of this diff is that it makes importing trees a bit more
expensive as testing that a path is utf8 valid is not free.
Reviewed By: chadaustin
Differential Revision: D25442975
fbshipit-source-id: 89341a004272736a61639751da43c2e9c673d5b3
Summary:
This'll move getdeps dependency for katran to 0.3 (which is already the case
for travis build)
Reviewed By: avasylev
Differential Revision: D26129525
fbshipit-source-id: ec9d0615a3e02d75454a3f9dd974ac5c7589e40a
Summary:
Simple test that can give us an intuition for how the ConcurrentMemIdMap should
perform.
Reviewed By: krallin
Differential Revision: D26601378
fbshipit-source-id: ae8f2ada6fc08eef806f3ece72a6c1c2f011ac32
Summary:
Scrubbing a repo is highly concurrent as its mostly IO bound. As such we can end up waiting on sql connection pool for connections where it allows less than scheduled_max connections.
This change makes bounded_traversal_unique calls from the walker aware of the database tier and shard a Node may connect to, so that execution can be limited to the bounds of what the connection pool can support without waiting.
We still end up waiting for the connection, but now it's done in bounded_traversal_unique, rather than in connection pool code, and are thus a) able to process other Nodes while waiting and b) not subject to connection pool timeouts.
Differential Revision: D26524074
fbshipit-source-id: 19125388c730f5cef7e9de34b5b550efa8e6b825
Summary: Small clean up. Allows us to pass Logger by reference, removing the FIXME in blobrepo factory
Reviewed By: farnz
Differential Revision: D26551592
fbshipit-source-id: d6bb04b8bb3034ad056f071b67b5ae0ce3c6f224
Summary:
Allow color effects to be joined with `+`, e.g. `blue+bold`. Unlike effects
separated by spaces, these effects must all be available in order for the
effect to apply. If any of the effects are not available, then the combined
effect is not valid.
This is useful in fallback chains, where some of the fallbacks are combined
effects, for example: `brightred:red+bold`.
Use these to improve the fallback chains for sparse profiles, so that the
profile names are not shown in black on 8 color terminals.
Reviewed By: quark-zju
Differential Revision: D26580365
fbshipit-source-id: 8e204c901566ac03fb4af066ab1a09142750a9fb
Summary: I removed tokio-compat yesterday but this landed at the same time and uses it.
Reviewed By: mitrandir77, StanislavGlebik
Differential Revision: D26605246
fbshipit-source-id: 189f485bc8bc3018abb3e9290953eba14bd178de
Summary: It's useful to know the max number of connections to a shard that it is sensible to open. Used in next diff in stack.
Differential Revision: D26543419
fbshipit-source-id: 57e7c3295a5b5db1572f26954ae0dfb04c84b374
Summary:
The walker mostly checks for duplicates before emitting a new edge, at the same time recording the edge as visited to prevent duplicate edges.
However for derived data where the node may or may not be present, the node isn't considered visited until the node data is successfully loaded and seen in state.rs record_resolved_visit().
In such cases multiple copies of a node could be enqueued, and then we need to run each one.
With this change, where the walker can detect that such a step has completed previously, it will now short circuit the step and return None.
Differential Revision: D26369917
fbshipit-source-id: c2bdbbabfaa80dbb7cc7d2bc25a17230531ae111
Summary:
LOOKUP is the next RPC that EdenFS needs to implement, let's start by adding
the RPC types.
Reviewed By: kmancini
Differential Revision: D26562919
fbshipit-source-id: 3a4ce1a039b6405df9ff94757963aa46acf56d92
