Summary: Introduce a new stream combinator, `select_drop`, based on `futures::stream::select`, which performs the same function except dropping the contained streams when they terminate, rather than when the combined stream is exhausted. This prevents a deadlock in cases where one stream will not terminate until the other terminates.
Reviewed By: DurhamG
Differential Revision: D26139816
fbshipit-source-id: d28da32244a215741476e1c3882154ea9e3116a5
Summary: Introduce a minimal version of an async, typed `ReadStore` trait and corresponding `EdenApi` implementation, along with a debug command, `debugnewstore` to exercise it.
Reviewed By: DurhamG
Differential Revision: D26050641
fbshipit-source-id: 2c14c3715e7067f9ecd1e649e6ca146a1ce249bf
Summary: Add skeleton manifests to the derived data types that the warm bookmarks cache keeps warm.
Reviewed By: krallin
Differential Revision: D26312403
fbshipit-source-id: 26bafee69ac0c38c1d49b29cdeb4968c37243271
Summary: Limit durations in derived data logging to two decimal places.
Reviewed By: krallin
Differential Revision: D26311139
fbshipit-source-id: 0f317d9c15c8425ad0591cbe7f07063c74e081b7
Summary:
The warning is noisy if the wait is short and can make users dislike commit cloud.
Make sure we don't print the warning earlier than defaultlockwaitwarntimeout.
This is a follow up on D25587459 (18b8c66439) that doesn't fully work if warntimeout is passed equal to zero.
The default threshold has been introduced earlier in D25587459 (18b8c66439). This is just a fix.
A new test has been added. Also, the api should allow to pass value 0 meaning to show the warning always.
Reviewed By: quark-zju
Differential Revision: D26251321
fbshipit-source-id: c3beb5fec6a65f1816f667df70c1a39dd65ef083
Summary:
This is a preparation for potential necessity of IO being done by this trait and its implementors.
We think the IO might be needed if we move commit sync config storage from `Configerator` into xdb, or some place else. To be clear, I personally am not certain we'll *need* this, but in any case, asyncifying the trait does not seem like a risky thing here (because we usually have only 0-2 sync functions in the stack above `LiveCommitSyncConfig` accessors, so it does not require large-scale code flow changes or anything).
I intentionally did not touch the push-redirection accessors, as those I don't think will ever move away from configerator.
Reviewed By: StanislavGlebik
Differential Revision: D26275905
fbshipit-source-id: 1bfdca087434d475d50032dd47dd94f16be051f9
Summary:
Add support for directly throttling blobstores based on read and write bandwidth used.
Read bytes throttling can only approximate given we only know the size *after* the inner Blobstore::get().
Reviewed By: krallin
Differential Revision: D26022763
fbshipit-source-id: b8019f53bda326117511aa659dcdab2a958dbd27
Summary:
MSVC is broken as it doesn't understand the various macros used to generate
XdrTrait for specific types, and I can't figure out a way to make that work.
Even if I provide dummy version of these, the buck build breaks due to glog
trying to redefine the ERROR symbol that some Microsoft headers contain. For
now, let's just ifdef it out since it's unlikely that we're going to be using
NFS on Windows anytime soon.
Reviewed By: singhsrb
Differential Revision: D26293519
fbshipit-source-id: bbaf325c7d1f1688327708360244797a6d48179e
Summary:
This matches the Rust behavior and is useful for filtering
because the env logger syntax applies to target:
% RUST_LOG=edenscm.hgext.debugshell=info lhg dbsh
In [1]: from edenscm import tracing
In [2]: tracing.info('foo')
[2021-02-05T19:21:41.082Z INFO edenscm.hgext.debugshell] message="foo"
Reviewed By: kulshrax
Differential Revision: D26282053
fbshipit-source-id: 8ee9e82b955835b24c49f9bf81c7a3aec7a65a33
Summary:
Like it says in the title. Let's add some basic throttling control here, in
line with what we have in Mononoke Server. The numbers don't quite match up
since fetches in EdenAPI don't include linknodes or history, but this should be
better than nothing and sufficient for now, and makes sense to have with
EdenAPI & Mononoke Server running in the same process.
Reviewed By: HarveyHunt
Differential Revision: D26250746
fbshipit-source-id: 338eda4341a163d0d915f10bf45fc7f40c74fc69
Summary:
Like it says in the title. I'd like to access our load limiter from there and
this is a good way to do it.
Note that EdenAPI doesn't have client hostnames when we instantiate the load
limiter, but we also don't truly need it because our client identities include
the hostprefix (at least in cases where a reverse lookup stands a chance of
succeeding, i.e. in prod).
Reviewed By: HarveyHunt
Differential Revision: D26250743
fbshipit-source-id: d1ad062d2967bfe28b3ddc164cac977d297d1b6a
Summary:
Like it says in the title. We can't do this for identities extracted from SSH
(because it's not there), but for traffic that comes to us via TLS, we can! So,
let's use it.
Reviewed By: johansglock
Differential Revision: D26250745
fbshipit-source-id: 9d2315712e164c5e21dbea87652a1d88dfd9bcda
Summary: I'd like to reuse this in EdenAPI. This will allow for that!
Reviewed By: StanislavGlebik
Differential Revision: D26250744
fbshipit-source-id: 47e86cba4d9fafc35f26cb6431921a157a207925
Summary:
I'd like to use those in the load_limiter crate, but that'd create a cyclic
dependency, so I need to move them. Considering those methods operate on
identity sets anyway, it seems like a good idea to just move them there.
Reviewed By: StanislavGlebik
Differential Revision: D26250747
fbshipit-source-id: 4de89eae50bd5b574c2c08798e1fd753da9cab90
Summary:
Make it possible to convert Bytes to `Vec<u8>` in a zero-copy way if possible.
This will make it possible to convert `minibytes::Bytes` to `bytes::Bytes` in a
zero-copy way if possible. Practically, it might be useful for some
revisionstore -> edenapi/types usecases.
Reviewed By: kulshrax
Differential Revision: D26237922
fbshipit-source-id: 28d620f303511099df77f79256d98abb1010f665
Summary: This API will be used in the next diff.
Reviewed By: DurhamG
Differential Revision: D26237923
fbshipit-source-id: 69438072c2edef1ce28ceef3b8b723f015f54ff5
Summary:
This affects `first`, `last`, `limit` revset functions.
Previously, they just iterate through the set naively. Now they have Rust fast
paths. For example:
In [1]: time repo.revs('last(parents(:1000000),100)')
CPU times: user 5.08 ms, sys: 1.02 ms, total: 6.1 ms
Wall time: 4.96 ms
Out[1]: <nameset+ <spans [0a087e42b29ba5c9ceb3588477d78f7f09ce2663:af5e72c2a0c2e78de462bba8bde63f2499aeb9b5+999900:999999]>>
In [2]: time repo.revs('first(parents(:1000000),100)')
CPU times: user 2.21 ms, sys: 40 µs, total: 2.25 ms
Wall time: 1.83 ms
Out[2]: <nameset+ <spans [06b96ec2a8b60d984606f36c30d3dbc899d804df:4cab7b68c0bbdc13eb2eded8fc8c4c8d520a7189+0:99]>>
In [5]: time repo.revs('first(reverse(parents(:1000000)),100)')
CPU times: user 2.2 ms, sys: 185 µs, total: 2.39 ms
Wall time: 1.67 ms
Out[5]: <nameset- <spans [0a087e42b29ba5c9ceb3588477d78f7f09ce2663:af5e72c2a0c2e78de462bba8bde63f2499aeb9b5+999900:999999]>>
In [6]: time repo.revs('last(reverse(parents(:1000000)),100)')
CPU times: user 2.01 ms, sys: 12 µs, total: 2.02 ms
Wall time: 1.68 ms
Out[6]: <nameset- <spans [06b96ec2a8b60d984606f36c30d3dbc899d804df:4cab7b68c0bbdc13eb2eded8fc8c4c8d520a7189+0:99]>>
In [7]: time repo.revs('limit(reverse(parents(:1000000)),100,10000)')
CPU times: user 1.48 ms, sys: 887 µs, total: 2.37 ms
Wall time: 1.89 ms
Out[7]: <nameset- <spans [5e23b6f07f1512a8991de5a0e883ff4d598ac1d7:f5f68207be4a458e99d4a9200296977aecf44ea2+989900:989999]>>
This, together with fast `_firstancestors`, could potentially answer
globalrev-like queries on the master branch without recording the globalrev
information server side (which has complexities like locking, etc). For
example, to convert a global rev `g` to commit:
c = repo.revs('limit(_firstancestors(master), 1, %s)', g).first()
To convert a commit `c` to global rev:
g = len(repo.revs('_firstancestors(%s)', c)) - 1
Reviewed By: sfilipco
Differential Revision: D26203558
fbshipit-source-id: 14d9247bbb07260f783e05b3fb1034406de48121
Summary: Provide a way to slice a set.
Reviewed By: sfilipco
Differential Revision: D26203562
fbshipit-source-id: 97a4349833a7a1c9664189d4737e2ad418369f22
Summary:
The SliceSet provides slicing support (skip n items, and then take m items)
for a general NameSet. The complexities come from caching and fast paths.
Basically, we cache the skipped items as a way to answer "contains" quickly.
We also cache the "taken" items if it's bounded to answer "iter_rev",
"contains".
Reviewed By: sfilipco
Differential Revision: D26203559
fbshipit-source-id: 6078b6178aff878e2169e87d446f0b254432aa80
Summary:
In the future we'd like to test "contains" but only use the "contains"
code path if it's better than O(N). Currently there is no way to know
whether "contains" is O(N) or not. Add an explicit API for that.
Reviewed By: sfilipco
Differential Revision: D26203554
fbshipit-source-id: 5d4c6014694c45b666a0ecd83fce33157cc15779
Summary: Enhance the test so it checks hints set by the `filter` function.
Reviewed By: sfilipco
Differential Revision: D26203553
fbshipit-source-id: b9bf5baa3bf51434835341e95e72073bd8c4256a
Summary:
It is incorrect with >= 2 hints.
For example,
iter: [hints1, hints2, hints1]
fold acc: hints1, None, hints1
The `None` should be permanent if there are two versions that do not have an
order.
Reviewed By: sfilipco
Differential Revision: D26203555
fbshipit-source-id: 96ff30ba45d439220519cd1505e3264118ffd9b2
Summary:
Previously, `LazySet` was constructed with default `Hints`. That disables fast
paths. Revise the API so LazySet requires an explicit `Hints` to address the
issue.
Reviewed By: sfilipco
Differential Revision: D26203561
fbshipit-source-id: c92cd1f7eb7b40ffaaf53abcf05e64f3d41b906d
Summary:
Previously, `MetaSet` was constructed with default `Hints`. That disables fast
paths. Revise the API so MetaSet requires an explicit `Hints` to address the
issue.
Reviewed By: sfilipco
Differential Revision: D26203557
fbshipit-source-id: 9e7658af8723b06d0efdcad1ab4671c79e907326
Summary: Those methods will be used for fast paths slicing a NameSet.
Reviewed By: sfilipco
Differential Revision: D26203556
fbshipit-source-id: aef18f60633653e19571e3fdeeb6b258e4dd32c7
Summary:
This just renames types so `IdSet` is the recommended name and `SpanSet`
remains an implementation detail.
Reviewed By: sfilipco
Differential Revision: D26203560
fbshipit-source-id: 7ca0262f3ad6d874363c73445f40f8c5bf3dc40e
Summary:
Before this change, nameset's min, max use the base class implementation, which
can be undesirably slow sometimes, while `first` and `last` remain fast
since they call into Rust logic which understands better about fast paths.
Optimize `min`, `max` by converting them to `first` or `last` if possible.
Before. Note `min()` is slow:
In [3]: s=repo.revs('parents(1:10000)')
warning: ':' is deprecated; use '::' instead. https://fburl.com/hgcolon
In [4]: s
Out[4]: <nameset+ <spans [06b96ec2a8b60d984606f36c30d3dbc899d804df:d6867d22c1ad3cf6e384332093e1d0de1fa86cb5+0:9999]>>
In [5]: %time s.min()
Out[5]: CPU times: user 51.5 ms, sys: 8.25 ms, total: 59.7 ms
Wall time: 57.8 ms
0
In [6]: %time s.max()
Out[6]: CPU times: user 64 µs, sys: 15 µs, total: 79 µs
Wall time: 84.6 µs
9999
In [7]: %time s.first()
Out[7]: CPU times: user 39 µs, sys: 9 µs, total: 48 µs
Wall time: 50.8 µs
0
In [8]: %time s.last()
Out[8]: CPU times: user 62 µs, sys: 0 ns, total: 62 µs
Wall time: 66.5 µs
9999
After:
In [2]: %time s.min()
CPU times: user 0 ns, sys: 902 µs, total: 902 µs
Wall time: 907 µs
Out[2]: 0
In [3]: %time s.max()
CPU times: user 551 µs, sys: 0 ns, total: 551 µs
Wall time: 557 µs
Out[3]: 9999
In [4]: %time s.first()
CPU times: user 50 µs, sys: 9 µs, total: 59 µs
Wall time: 62 µs
Out[4]: 0
In [5]: %time s.last()
CPU times: user 49 µs, sys: 9 µs, total: 58 µs
Wall time: 62 µs
Out[5]: 9999
Reviewed By: sfilipco
Differential Revision: D26182241
fbshipit-source-id: dedf3788c52d22c6b63ae60847cc0667616f11d2
Summary:
Optimize the `_firstancestors` revset function using Rust.
When calculating `len(_firstancestors(master))`, the new code took 2ms while
the old code needs 76s, a ~40000x improvement.
Reviewed By: sfilipco
Differential Revision: D26182242
fbshipit-source-id: 55f17b014e727d8e8e3099b7c287f8bf8479279b
Summary: This exposes the segments version of the algorithm.
Reviewed By: sfilipco
Differential Revision: D26182244
fbshipit-source-id: 716e6d5254c9962618040e7549c2804184230a97
Summary: This will be used by NameDag.
Reviewed By: sfilipco
Differential Revision: D26182243
fbshipit-source-id: 9db2ecde98281dc45fcfd0d7cf30d6c7bf2be81d
Summary:
This will be useful to optimize `_firstancestors` revset, which is useful to
calculate a linear branch to draw growth graphs. Without a fast path, the pure
Python `_firstancestors` implementation would have id <-> name translation
overhead that makes Rust changelog O(20) slower than the Python revlog.
Reviewed By: sfilipco
Differential Revision: D26182240
fbshipit-source-id: d44f5ca5dc8c38df74281832931d87868791209e
Summary:
Optimize the `x~n` revset function using Rust.
Note: This changes the behavior a bit, `x~n` no longer returns `null`.
Reviewed By: sfilipco
Differential Revision: D26142683
fbshipit-source-id: d6a45b7e67352d74986274e52002a769bbae772e
Summary:
When `n` is too large, return None. This matches the "parents()" behavior -
not error out but returns empty set.
Reviewed By: sfilipco
Differential Revision: D26142684
fbshipit-source-id: e45fca69e39c2968dc7abc5a4a155e6b7c280836
Summary: Optimize the "merge()" revset function using the merges() from Rust.
Reviewed By: sfilipco
Differential Revision: D26142169
fbshipit-source-id: 47f426625869b7889b28bb1a18544d4abae36cae
Summary:
The "merges" algorithm on the IdDag. Basically scan through flat segments
and conditionally pick up their "low"s.
Reviewed By: sfilipco
Differential Revision: D26142172
fbshipit-source-id: 305fe619a65ed4034423f303bee8d57be0424963
Summary:
The newly added API returns parent count wihtout actual parents.
Useful for the "merges" algorithm.
Reviewed By: sfilipco
Differential Revision: D26142176
fbshipit-source-id: 4f301b8de88f2af637f52bf62b24ddb12e65b6a7
Summary:
The function calculates all merges within a graph. It is useful to calculate
the "universally known" set, or to answer the "merge()" revset function.
This diff only adds a default impl. Upcoming diffs will add more efficient
versions on the segments graph.
Reviewed By: sfilipco
Differential Revision: D26142173
fbshipit-source-id: 02de180f6e444bcac63a1cc46dd23faeb8e08e14
Summary: The `filter` API filters a set by a function.
Reviewed By: sfilipco
Differential Revision: D26142177
fbshipit-source-id: f24cbeeaf1c85264706c933c98e364d7937790fe
