Summary:
The Rust upstream took the "set F_CLOEXEC on every opened file" approach and
provided no support for closing fds at spawn time to make spawn lightweight [1].
However, that does not play well in our case:
- On Windows:
- stdin/stdout/stderr are not created by Rust, and inheritable by
default (other process like `cargo`, or `dotslash` might leak them too).
- a few other handles like "Null", "Afd" are inheritable. It's
unclear how they get created, though.
- Fortunately, files opened by Python or C in edenscm (ex. packfiles) seem to
be not inheritable and do not require special handling.
- On Linux:
- Files opened by Python or C are likely lack of F_CLOEXEC and need special
handling.
Implement logic to close file handlers (or set F_CLOEXEC) explicitly.
[1]: https://github.com/rust-lang/rust/issues/12148
Reviewed By: DurhamG
Differential Revision: D23124167
fbshipit-source-id: 32f3a1b9e3ae3a9475609df282151c9d6c4badd4
Summary:
It uses `sys.argv`, which might be rewritten by `debugshell`. Capture
`sys.argv` to make hgcmd more reliable.
Reviewed By: DurhamG
Differential Revision: D22993215
fbshipit-source-id: 5fa319e8023b656c6cdf96cb3229ea9f2c9b9b99
Summary: This allows us to run commands after changes were made to the repo.
Reviewed By: DurhamG
Differential Revision: D22993218
fbshipit-source-id: d9943dcda94da42970fb9107f48f4caa14b6a9d4
Summary:
Some code paths (ex. metalog.commit) use `util.timer()` as a way to get
seconds since epoch, and get 0 for tests. Other use-cases of `util.timer()`
are ad-hoc time measure for displaying speed / progress. They do not need high
precision or strong guarantee that the clock does not go backwards. Drop the
`time.perf_counter()` to meet the first use-case's expectation.
Reviewed By: singhsrb
Differential Revision: D23431253
fbshipit-source-id: 8bf2d1ed32e284e17285742e1d0fd7178f181fb3
Summary:
With segments backend, the revision numbers will be longer than commit hashes
and are confusing.
Reviewed By: DurhamG
Differential Revision: D23408971
fbshipit-source-id: e2057fa644fc7b6be4291f879eee3235bb4e687b
Summary:
Pulling from older repos (ex. years ago) could require GBs of commit text data.
Flush commit data if they exceed certain size.
This is for revlog compatibility.
In the future we probably just make commit text lazy to avoid this kind of issues.
Reviewed By: DurhamG
Differential Revision: D23408834
fbshipit-source-id: 273384f5a05be07877bb1c9871c17b53ba436233
Summary: This would be used to avoid excessive memory usage during pull.
Reviewed By: DurhamG
Differential Revision: D23408833
fbshipit-source-id: 8edd95ab8201697074f65cc118d14755a230567d
Summary:
`addcommits` is designed to be more efficiently if called with a batch of
commits. So let's buffer the commits to add then only call it once.
This avoids some N^2 behaviors, for example, the NameDag internally will
prepare "snapshot" of itself which involves coping the pending Rust vecs
about the segments and id <-> hash map.
The change makes `pull` usable from unusably slow:
Original Python Revlog backend:
```
In [1]: %trace repo.pull(bookmarknames=['master'],quiet=False)
5191 +466 | Apply Changegroup edenscm.mercurial.bundle2 line 516
| - Commits = 125 :
| - Range = a1d1b3ade136:2e3fe78af189 :
5191 +466 | changegroup.cg1unpacker.apply edenscm.mercurial.changegroup line 313
5192 +416 | Progress Bar: commits (progressbar)
5192 +415 | changelog.changelog.addgroup edenscm.mercurial.changelog line 536
5192 +409 | revlog.revlog.addgroup edenscm.mercurial.revlog line 2116
5215 +371 | changelog.changelog._addrevision (125 times) edenscm.mercurial.changelog line 558
```
DoubleWrite (Segments + Revlog) backend, Before:
```
In [2]: %trace repo.pull(bookmarknames=['master'],quiet=False)
2396 +154059 | Apply Changegroup edenscm.mercurial.bundle2 line 516
| - Commits = 323 :
| - Range = cb0b100180ba:5fb57c74f72e :
2396 +154059 | changegroup.cg1unpacker.apply edenscm.mercurial.changegroup line 313
2397 +151433 \ Progress Bar: commits (progressbar)
2397 +151433 | changelog2.changelog.addgroup edenscm.mercurial.changelog2 line 334
```
DoubleWrite (Segments + Revlog) backend, After:
```
In [2]: %trace repo.pull(bookmarknames=['master'],quiet=False)
4629 +512 | Apply Changegroup edenscm.mercurial.bundle2 line 516
| - Commits = 45 :
| - Range = cf23c6972934:1ff0c5f0e7ad :
4629 +512 | changegroup.cg1unpacker.apply edenscm.mercurial.changegroup line 313
4630 +494 | changelog2.changelog.addgroup edenscm.mercurial.changelog2 line 334
```
Reviewed By: DurhamG
Differential Revision: D23390435
fbshipit-source-id: dd97a5008dedd844d4134b87bfef190fa739a80b
Summary:
The users of addrevisoncb are gone.
This also removes the "alwayscache" parameter of "_addrevision".
Reviewed By: DurhamG
Differential Revision: D23390437
fbshipit-source-id: 7edd9dd0b93d4cb9d4f35d088a1aef719b450ec1
Summary: It is about legacy revlog formats that are no longer relevant.
Reviewed By: DurhamG
Differential Revision: D23390436
fbshipit-source-id: 58c2c432804181bcc6517d6c988777b843fc9ba4
Summary:
We have a few safeguards against creating full checkouts. However we have
sparse profiles that are not full, but that include very large directories
which normally should not be included.
This diff adds a logic that checks if a new sparse profile has any of the "marker"
files i.e. some files from a folder that should not be included. Operation
aborts if that the case, however there's always a way to workaround that.
Reviewed By: DurhamG
Differential Revision: D23414200
fbshipit-source-id: 626f392319eb1be8b35f39cadafb61f3c1dfefe3
Summary:
Scuba logging that tracks undesired file fetches has some blind spots i.e. a
lot of fetches have null pid and null cmd line. This diff tries to fix another
part of the problem.
TreeInode::getOrLoadChild() has TODO `pass a fetch context down through
getOrLoadChild to track this load`. This diff fixes this TODO, and also starts
to pass context from EdenDispatcher:lookup method.
Note that it adds quite a lot of new `ObjectFetchContext::getNullContext()`
calls, and potentially those might be responsible for blind spots in logging.
I'll try to address this problem in the next diffs.
Reviewed By: kmancini
Differential Revision: D23418218
fbshipit-source-id: 319d7436494d8dce3580289aae9963aa13bfc191
Summary:
Scuba logging that tracks undesired file fetches has some blind spots i.e. a
lot of fetches have null pid and null cmd line. This diff fixes at least part
of the problem.
TreePrefetchContext which is used from TreePrefetchLease didn't logged client
pid at all (in fact, it logged almost nothing). This diff fixes at least one blind spot, however it doesn't look like this is the only one.
Reviewed By: kmancini
Differential Revision: D23417451
fbshipit-source-id: 107884e94c6b40de999328ec2ef78fe22174c1ca
Summary: `legacy.py` depends on other files in this directory, so lets add them all to the link tree
Reviewed By: fanzeyi
Differential Revision: D23356917
fbshipit-source-id: e4bfd82ebbd9d143a5454a43bb47e8dd55b4485f
Summary: These tests fail locally since adding these checks in eden doctor, so we need to mock them.
Reviewed By: chadaustin
Differential Revision: D23326597
fbshipit-source-id: 87a0e6ab0472e3ae56f89503e928c5a00a16ab04
Summary:
"hg diff" has --sparse option which diffs only files inside a sparse checkout.
The problem is that it doesn't work on eden checkouts because eden repo doesn't
have sparsematch() function.
This diff makes it so that if sparsematch() function doesn't exist then
--sparse option is just ignored.
The motivation for this change is
https://fb.workplace.com/groups/corehg/?post_id=687768245151742. There are some
diff calls that are triggered by arc lint that race with "hg update" and might download
loads of data on people's laptops. This diff doesn't fix the race, but it:
1) Makes sure we don't download too much data that are not in sparse profiles.
2) arc lint doesn't care about files outside of sparse profiles anyway, so
running --sparse make sense.
Reviewed By: DurhamG
Differential Revision: D23396918
fbshipit-source-id: 2a386fdbeab85187e2c2acab69cb86b74124d46f
Summary:
Most of the fixes are pretty trivial as the code was using functions not
present on Windows, either work around them, or switch to ones that are
multi-platform.
Of note, it looks like `hg doctor` doesn't properly detect when Mercurial and
EdenFS are out of sync, disabling the tests until we figure out why.
Reviewed By: genevievehelsel, fanzeyi
Differential Revision: D23409708
fbshipit-source-id: 3314c197d43364dda13891a6874caab4c29e76ca
Summary:
This is practically just 0 in our production setup during `pull`s. In the
future when the commit data become lazy, it's no longer possible to read the
files locally. So let's just don't scan the commits.
Reviewed By: DurhamG
Differential Revision: D23390438
fbshipit-source-id: 4c54c4aac5fd840205296ab86955ec1b8ab76607
Summary:
Mergedrivers can call dirstate.add directly and are adding paths with
"." and "..". Let's block those paths.
Reviewed By: quark-zju
Differential Revision: D23375469
fbshipit-source-id: 64e9f20169cfd50325ecd8ebcc1dd3be7a5cb202
Summary:
extdiff uses shutil.rmtree which calls os.rmdir with new python 3
options. Since we pathc os.rmdir, we need to support those options.
Reviewed By: quark-zju
Differential Revision: D23350968
fbshipit-source-id: 081d179dcd67b51ffdeb6b85899adf4e574a8d0f
Summary: Similar to D18528858 so module names do not need to be spelled twice.
Reviewed By: markbt
Differential Revision: D23091380
fbshipit-source-id: a2a261abc9c78c8805cea62b38498ba65398796d
Summary: This crate would fail to build without the "fb" feature because `serde_json` was listed as an optional dependency (but is used in a way that isn't conditional on the `fb` feature). This diff makes the dependency non-optional, and also silences several dead code warnings that are emitted when building without the "fb" feature.
Reviewed By: quark-zju
Differential Revision: D23386786
fbshipit-source-id: b00a8b0b8b0b978c1cfab2838629fcb388a076e9
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.
The fsync logic is put in a separate crate to avoid slow compiles.
Reviewed By: DurhamG
Differential Revision: D23124169
fbshipit-source-id: 438296002eed14db599d6ec225183bf824096940
Summary:
A warning means that every tree fetched will be printed in the edenfs log,
which is way too much. Let's decrease this to a debug message.
Reviewed By: genevievehelsel
Differential Revision: D23385778
fbshipit-source-id: d77f1cac3efb945d4b95750822f2f12f48c75ffe
Summary: `len(repo)` can no longer predicate the next rev number. Use nodes instead.
Reviewed By: DurhamG
Differential Revision: D23307791
fbshipit-source-id: cc20e53f039eee2a714748352e8e98aab253095a
Summary:
Some functions might be called very frequently. For example,
`phases.phasecache.loadphaserevs` might be called 100k+ times.
That makes the tracing data harder to process.
Limit the count of spans to 1k by default so the data is cheaper to process,
and some highly repetitive cases can now be reasoned about. Note the limit
is only put on static Span Ids. If a span uses dynamic metadata or ask for
different Span Ids each time, they will not be limited.
In debugshell,
td = %trace repo.revs('smartlog()')
len(td.serialize())
dropped from 6MB to 0.87MB.
It's also possible to reason about:
td = %trace len(repo.revs('ancestors(.)'))
in debugshell (taking 30s, 98KB serialized, vs 21s without tracing), while
previously the result would be too large to show (`%trace` just hangs).
Reviewed By: DurhamG
Differential Revision: D23307793
fbshipit-source-id: 3c1e9885ce7a275c2abd8935a4e4539a4f14ce83
Summary: Set a default limit so the output won't be too long.
Reviewed By: DurhamG
Differential Revision: D23307792
fbshipit-source-id: 7e2ed99e96bbde06436a034e78f899fc2e3e03f8
Summary:
The debugshell command can be long running and contains uninteresting stuff.
Do not profile it.
Practically this hides showing the background statprof thread when using `%trace`.
Reviewed By: DurhamG
Differential Revision: D23278597
fbshipit-source-id: bad97de22e1be2be8b866bee705ea3a6755aa54b
Summary:
This allows entering ipdb for code like: `ipdb` or `ipdb()`. It can be handy to
debug something.
Reviewed By: DurhamG
Differential Revision: D23278599
fbshipit-source-id: 4355dd1944617aeb795450935789f01f66f094eb
Summary: This makes it possible to get tracing results, or run hg commands directly.
Reviewed By: DurhamG
Differential Revision: D23278601
fbshipit-source-id: e7dc92080d2881cb4155a481df5ca93f324828fc
Summary:
The `--trace` flag enables tracing Python modules.
For compatibility reasons, it also enables `--traceback`.
It can be used with debugshell to make `%trace` more useful.
Reviewed By: sfilipco
Differential Revision: D23278600
fbshipit-source-id: d6d0b34bd5c48111f8cd33d7df115f349b0e95b6
Summary:
I found this when I aborted an rebase Dxxx and trying rebasing again and it
complained about "nothing to rebase". It was caused by Dxxx resolving into
a hidden commit.
Reviewed By: sfilipco
Differential Revision: D23307794
fbshipit-source-id: f7a956b5300240089b6a4648f28cf4a152ee2433
Summary:
`PerfCounters` was the only application-specific type exposed as a parameter to the post-request callbacks, and it was only being used in one place. To facilitate making the post-request callback functionality more general, this diff makes the callback in question capture the `CoreContext` in its environment, thereby giving it access to the `PerfCounters` without requiring it to be passed as an argument.
This should not change the behavior since regardless of how the callback obtains a reference, it will still refer to the same underlying `PerfCounters` from the request's `CoreContext`.
Reviewed By: DurhamG
Differential Revision: D23298417
fbshipit-source-id: 898f14e5b35b827e98eaf1731db436261baa43bb
Summary:
We shouldn't delete from a dictionary while iterating over it, instead we should iterate over a copy and then delete from the original.
`.items()` returns a view of the dict, while wrapping it in `list` makes a deep copy.
Reviewed By: DurhamG
Differential Revision: D23283668
fbshipit-source-id: a168eef1ed2a1ce02fe71b3f6e3aed090965d2a4
Summary:
Mononoke throws an error if we request the nullid. In the long term we
want to get rid of the concept of the nullid entirely, so let's just add some
Python level blocks to prevent us from attempting to fetch it. This way we can
start to limit how much Rust has to know about these concepts.
Reviewed By: sfilipco
Differential Revision: D23332359
fbshipit-source-id: 8a67703ba1197ead00d4984411f7ae0325612605
Summary: I refactored this method to be a member fuction of `EdenFSProcess` and I thought this instance of the method was deleted, but I came across it while working in this area again.
Reviewed By: fanzeyi
Differential Revision: D23113075
fbshipit-source-id: 2c257cca2da3a4bfefb974753eb00c7580c5a104
Summary:
Enabling hg dynamicconfigs in D23309090 (d643f48c8c) changed the output of `hg
manifest --debug` and broke HgImportTest. Set TESTTMP to avoid
production configs.
Reviewed By: DurhamG
Differential Revision: D23335847
fbshipit-source-id: 7ffd0394aa7a8466b266000b18f8742ed4a6b53f
Summary:
Corp has a different concept of tier than prod. Let's load the corp
tier into our tier set as well.
Reviewed By: quark-zju
Differential Revision: D23354056
fbshipit-source-id: c9543b8253f042c7b1224578e0687b4bdf21738e
Summary:
The Python 3 email library internally stores the message as text, even
though our input and requested output is bytes. Let's make our own wrapper
around the parser to use ascii surrogateescape encoding so we can get the
actual bytes out later and not get universal newlines.
Based off the upstream 7b12a2d2eedc995405187cdf9a35736a14d60706,
which is basically a copy of the BytesParser implementation (https://github.com/python/cpython/blob/3.8/Lib/email/parser.py) with
newline=chr(10) added.
Reviewed By: quark-zju
Differential Revision: D23363965
fbshipit-source-id: 880f0642cce96edfdd22da5908c0b573887bed12
Summary:
`hg cloud rejoin` command is used in fbclone and it is supposed to print a
message on RegistrationError but this has been broken recently.
Reviewed By: markbt
Differential Revision: D23342773
fbshipit-source-id: 4f3318848953656dea65a2b5d4d832694f6b353c
Summary:
Once we discover that the (small) repo we import into push-redirects (D23158826 (d3f3cffe13)) to a large repo,
we want to import into the large repo first, then backsync into the small one (see previous diff summary).
The aim of this diff is to setup the variables (e.g. bookmarks) needed for importing into
the large repo first before backsyncing the commits into the small repo.
Next step: add functionalities to control how we backsync from large repo to the small repo
Reviewed By: StanislavGlebik
Differential Revision: D23294833
fbshipit-source-id: 019d84498fae4772051520754991cb59ea33dbf4
Summary:
Setting up, tearing down, and querying RequestContext has some
overhead that I would like to avoid in the inner FUSE loop, so replace
RequestData with a single class that's heap-allocated at the start of
a request and is guaranteed to survive until the request ends, and is
otherwise explicitly passed where it's needed.
Reviewed By: kmancini
Differential Revision: D22712310
fbshipit-source-id: fc30d0b0f7e22b39306b857194ea07a913110b0f
Summary:
There are users who prefer run `hg cloud leave` if they notice they are
connected to commit cloud sync.
Proving more information and add a prompt might help them to change their mind.
For some users who left new fbclone will connect them back. So on next leave they can learn more information about Commit Cloud Workspaces.
Reviewed By: markbt
Differential Revision: D23346091
fbshipit-source-id: 72f170f7133cd64b772ec75ae29a85dc8809e351
Summary:
When updating to the null commit, the logic that computes the update
distance was broken. The null commit is pre-resolved to -1, which when passed to
a revset raw gets resolved as the tip commit. In large repositories this can
take a long time and use a lot of memory, since it's computing the difference
between tip and null.
Let's fix it to not pass the raw rev number, and also to handle the case of a 0
distance update.
Reviewed By: quark-zju
Differential Revision: D23358402
fbshipit-source-id: 3b0a1fe1bbcb07effba4d0ab2c092e66bdc02e67
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/46
See https://github.com/facebookexperimental/eden/runs/1034006668:
error: unused import: `env::set_var`
--> src/lfs.rs:1539:15
|
1539 | use std::{env::set_var, str::FromStr};
| ^^^^^^^^^^^^
|
note: the lint level is defined here
--> src/lib.rs:125:9
|
125 | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(unused_imports)]` implied by `#[deny(warnings)]`
error: unnecessary braces around method argument
--> src/lfs.rs:2439:36
|
2439 | remote.batch_upload(&objs, { move |sha256| local_lfs.blobs.get(&sha256) })?;
| ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ help: remove these braces
|
note: the lint level is defined here
--> src/lib.rs:125:9
|
125 | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(unused_braces)]` implied by `#[deny(warnings)]`
error: aborting due to 2 previous errors
error: could not compile `revisionstore`.
I dropped `#![deny(warnings)]` as I don't think warnings like the above ones
should break the build. (denying specific warnings that we care about explicitly
might be a better approach)
Reviewed By: singhsrb
Differential Revision: D23362178
fbshipit-source-id: 02258f57727edfac9818cd29dda5e451c7ca80a7
Summary: Now that it is possible to control which features are enabled on manually-managed dependencies, we can reenable autocargo for `edenapi`. See D23216925, D23327844, and D23329351 (840e6dd6f6) for context.
Reviewed By: dtolnay
Differential Revision: D23335122
fbshipit-source-id: 8ce250c3a106d2a02f457f7ed531623dd866232f
Summary:
Without the `--noproxy localhost` flag curl will obey the `https_proxy` env
variable but will not respect the `no_proxy` env variable or `curlrc`.
This means that tests running in a shell with `https_proxy` will likely fail.
The failures may vary in aspect based on what logic is running at the time.
Reviewed By: kulshrax
Differential Revision: D23360744
fbshipit-source-id: 0383a141e848bd2257438697e699f727d79dd5d2
Summary: The command does not crash but `-` lines are ignored.
Reviewed By: DurhamG
Differential Revision: D23357655
fbshipit-source-id: f48568bc193f947503bc19f3e192b33346c317e1
Summary:
std::iterator is deprecated in C++17. Removing it fixes warnings in
the Clang/Windows build.
Reviewed By: genevievehelsel
Differential Revision: D23352927
fbshipit-source-id: 293e30909eaa8a7c4856a91930a3886ad0b19364
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/45
Fix referring to 'version' without proper codegen by making 'version' compile
without codegen. This fixes configparser test when version/src/lib.rs was not
generated.
Make unneeded deps without 'fb' feature optional.
This would hopefully fix the "EdenSCM Rust Libraries" GitHub workflow.
Reviewed By: DurhamG
Differential Revision: D23269864
fbshipit-source-id: f9e691fe0a75159c4530177b8a96dad47d2494a9
Summary: Now that `TimerMiddleware` no longer depends on `RequestContext`, it can be moved into `gotham_ext`.
Reviewed By: farnz
Differential Revision: D23298414
fbshipit-source-id: 058cb67c9294b28ec7aec03a45da9588e97facc5
Summary: Previously, the LFS server's `TimerMiddleware` needed to be used in conjunction with `RequestContext`, as its purpose was to simply call a method on the `RequestContext` to record the elapsed time. This diff moves tracking of the elapsed time into `TimerMiddleware` itself (via Gotham's `State`), allowing the middleware to be used on its own.
Reviewed By: farnz
Differential Revision: D23298418
fbshipit-source-id: 8077d40edec0936d95317ac11d86bbcd33a3bf04
Summary: This makes the code simpler.
Reviewed By: sfilipco
Differential Revision: D23269858
fbshipit-source-id: bb9ac0bd1696f7429ca1856e6c63e04fabc2757a
Summary: This makes the code simpler.
Reviewed By: sfilipco
Differential Revision: D23269866
fbshipit-source-id: 30c9e9d218378c0d6df8b822b2a81df2b38f5b01
Summary: Will be used to simplify code.
Reviewed By: sfilipco
Differential Revision: D23269859
fbshipit-source-id: bed0c4dca075ff60900025642af1d84bdd03452d
Summary:
`impl<T> Trait for T` in the current Rust makes it impossible to have
`impl<Q> Trait for Q`. Avoid using it for IdConvert and PrefixLookup.
Reviewed By: sfilipco
Differential Revision: D23269861
fbshipit-source-id: a837f3984ff4e1bd5a3983dd1642b9f064f51a36
Summary:
`impl<T> Trait for T` in the current Rust makes it impossible to have
`impl<Q> Trait for Q`. Avoid using it for DagAlgorithm.
Reviewed By: sfilipco
Differential Revision: D23269860
fbshipit-source-id: 031e75e9bf1f1eec2b9e8f36220ef8b817a143a5
Summary: LowLevelAccess is a subset of NameDagStorage. Use the latter instead.
Reviewed By: sfilipco
Differential Revision: D23269865
fbshipit-source-id: 81ebb1e986d8b02c968a9a237ad9a97d4afd54bf
Summary:
If there are too many heads, the current `descendants` algorithm would visit
all "old" heads. For example, with this graph:
head9999 (N9999)
/
Z (master)
:
: (many heads)
:/
: head2 (N2)
:/
C head1 (N1)
|/
B head0 (N0)
|/
A
`A::head9999` or `Z::head9999` will visit N0, N1, ..., N9999, because
`descendands_up_to` is provided with `max_id = N9999` and Z as a vertex in the
master group, is before N0 in non-master. The current algorithm also means
`descendands_up_to` gets linearly slower as the user uses the repo more, which
is quite undesirable.
This diff changes `descendands_up_to` to take an `ancestors` set, which is
`::head9999` in this case, and iterate non-master flat segments in it. So it
will skip N0 to N9998 directly by finding the N9999 flat segment and only use
it. The number of heads will have a smaller impact on performance.
Another slowness is `draft::draft_heads`, if there are too many `draft_heads`,
the internal calculation of `::draft_heads` can be slow. Optimize it by
limiting `draft_heads` to `draft:`. Practically this affects `y::` revset as
`y::` is translated to `y::visible_heads` and `visible_heads` can be large.
`cargo bench --bench dag_ops -- '::-master'` shows significant difference:
Before:
range (master::draft) 18.112 s
range (recent_draft::drafts) 2.594 s
After:
range (master::draft) 72.542 ms
range (recent_draft::drafts) 14.932 ms
In my fbsource checkout there were 20k+ heads. The improvement of
`master::recent_draft` (`x::y`) is pretty visible, and `y::` is also improved:
% lhg debugbenchmarkrevsets -m -x 'p1(min(7e8c86ae % master))' -Y 'draft() & 7e8c86ae' -e 'x::y' -e 'y::' --no-default
# x: 168f5228e570fb6b2ff7f851bd82413102748d84 (p1(min(7e8c86ae % master)))
# y: 7e8c86aec68ebc6e0b8254afcb381315991fd21c (draft() & 7e8c86ae)
# before
| revset \ backend | segments | revlog | revlog-cpy |
|------------------|----------|--------|------------|
| x::y | 17ms | 0.1ms | 0.5ms |
| y:: | 3.3ms | 0.7ms | 1.3ms |
# after
| revset \ backend | segments | revlog | revlog-cpy |
|------------------|----------|--------|------------|
| x::y | 0.2ms | 0.1ms | 0.6ms |
| y:: | 1.0ms | 0.7ms | 1.3ms |
Reviewed By: sfilipco
Differential Revision: D23214387
fbshipit-source-id: 4d11db84cd28f4e04e8b991cbc650c9d5781fd27
Summary:
Lots of non-master heads is not an exercised graph in the benchmarks.
Add it as it practically happens. This will be used by the next change.
Reviewed By: sfilipco
Differential Revision: D23259879
fbshipit-source-id: 7fe290d14403e42e6d135bde56e2d5c8519ae530
Summary:
Currently the fuzz test only uses the master group. Let it exercise non-master
group too.
Reviewed By: DurhamG
Differential Revision: D23214388
fbshipit-source-id: 7108a1055fbdda2b012f93c5948fb83ef3b9a96f
Summary:
The calculation can take tens of milliseconds. Cache it.
Invalidate the cache on transaction commit.
This will improve perf on revsets like `descendants` that will use
`head()`.
Reviewed By: DurhamG
Differential Revision: D23196412
fbshipit-source-id: 2913310ebb97e1c0346198c1e2738799799c740a
Summary: Provide a way to see segments.
Reviewed By: sfilipco
Differential Revision: D23196408
fbshipit-source-id: b1418f945a5a3364ac73b0f97466d973dd4b6300
Summary:
Provide a way to print out all segments with resolved names. This will be used
in a debug command.
Reviewed By: sfilipco
Differential Revision: D23196410
fbshipit-source-id: 1712bfda0271aa548699fe4a6b8603c5ec07af7f
Summary:
Use the parent-child index to answer children query quickly.
`cargo bench --bench dag_ops -- children`:
Before:
children (spans) 606.076 ms
children (1 id) 124.105 ms
After:
children (spans) 602.999 ms
children (1 id) 10.777 ms
Reviewed By: sfilipco
Differential Revision: D23196411
fbshipit-source-id: 37195d5ccaa582d35314e0000352ef477287d38c
Summary: This will be used to optimize "children(single vertex)" query.
Reviewed By: sfilipco
Differential Revision: D23196409
fbshipit-source-id: 050c0859faf83b909e3174bb7c7bd6e7725165c0
Summary:
Update the parent index to store non-master group too. To make
"remove_non_master" work, the index contains a "child group" prefix that
allows efficient range invalidation.
This will allow answering "children(single vertex)" query more efficiently.
This diff does not expose an API to query the index yet.
Reviewed By: sfilipco
Differential Revision: D23196406
fbshipit-source-id: 9137da5ffa8306bdafbcabc06b6f0d23f38dcf57
Summary:
Practically, the input of `children` is often one vertex instead of a large set.
Add a benchmark for it.
It looks like:
children (spans) 606.076 ms
children (1 id) 124.105 ms
Reviewed By: sfilipco
Differential Revision: D23196407
fbshipit-source-id: 0645b59ac846836fd061386384f6386a57661741
Summary: They can be figured out at Hints initialization time. So they don't need to be mutable.
Reviewed By: sfilipco
Differential Revision: D23182518
fbshipit-source-id: 133375fdf27a2546a50b63fb130534acdadc5938
Summary:
Both IdSet and IdLazy set require both Dag and IdMap to construct.
This is step 1 torwards making Dag and IdMap immutable in hints.
A misspeall of "lhs" vs "hints" in the union set is discovered by the change
and fixed.
Reviewed By: sfilipco
Differential Revision: D23182520
fbshipit-source-id: 3d052de4b8681d3672ebc45d953d1e784f64b2a4
Summary:
It will be used in places (ex. tests) where a Dag is required but constructing
a real Dag is troublesome.
Reviewed By: sfilipco
Differential Revision: D23182517
fbshipit-source-id: 736911365778e5071c1e0b9615090a4e960392a0
Summary: This is more consistent with `id_map_snapshot`.
Reviewed By: sfilipco
Differential Revision: D23182519
fbshipit-source-id: 62b7fc8bfdc9d6b3a4639a6518ea084c7f3807dd
Summary:
Similar to descendants, the new range algorithm avoids potentially expensive
checks about whether high-level segments can be used or not. Practically this
is overall an improvement.
`cargo bench --bench dag_ops -- range`:
Before:
range (2 ids) 115.380 ms
range (spans) 243.666 ms
After:
range (2 ids) 123.274 ms
range (spans) 23.101 ms
It is 100x faster with the range x::y benchmark added later on `git.git`.
Reviewed By: sfilipco
Differential Revision: D23106175
fbshipit-source-id: 691e0418ba2b7ad9f52ac15b5cd6088ec28d5f48
Summary:
The old algorithm tries to make use high-level segments.
However, the code to test whether a high-level segment can be used is
often too expensive for the benefit. Often, high-level segments cannot
be used most of the time and it's similar to O(flat segments).
This diff adds a simpler algorithm that just iterates through the flat
segments. It's faster in most practical cases.
`cargo bench --bench dag_ops -- descendants` shows improvements too:
Before:
descendants (small subset) 436.515 ms
After:
descendants (small subset) 33.460 ms
Reviewed By: sfilipco
Differential Revision: D23106174
fbshipit-source-id: e6101483d8539b2b1c881be2ccfd0071f122352f
Summary: This will be used by upcoming changes.
Reviewed By: sfilipco
Differential Revision: D23106177
fbshipit-source-id: 9bf183f7464c06b801be64fd938db0babd544756
Summary: This internal struct will be used by upcoming changes.
Reviewed By: sfilipco
Differential Revision: D23106172
fbshipit-source-id: 6d5b9bc1c810984814d0912100acca38a2565a63
Summary:
Our internal build infra creates a workspace and workspaces don't like
it when two crates have the same name. Eden scm had third-party rust crates that
were simple redirects to the internal location, but had the same name. This
caused breakages once these crates became part of the edenfs open source build.
Let's rename them to avoid this issue.
Reviewed By: kulshrax
Differential Revision: D23252539
fbshipit-source-id: 9ff2fa160a19c6bc54e015c71f9da7044ce659a7
Summary:
We might need to rebackfill blame for configerator (see
https://fburl.com/hfylxmag). It's good to have a command that shows how many
files with rejected blame we have.
Reviewed By: farnz
Differential Revision: D23267648
fbshipit-source-id: 33e658b53391285461890bda3a94b391e6063c12
Summary:
Windows doesn't seem to follow the same write permission rules, so
let's just skip that part of the test for now.
Reviewed By: xavierd
Differential Revision: D23348515
fbshipit-source-id: bfcfa4f8adf94413047b045987e570ba98f9078d
Summary:
We'd like to keep track of hg cache size on users' machines. It's generally
useful, but it will be even more important with megarepo rollout. The job will be triggered by cron.
This is a very simple implementation, most of it was copy-pasted from
https://fburl.com/diffusion/g8ysyxo1.
Note that I intentionally didn't call hg to get hgcache size.
Since this binary will be triggered by cron, it will run as
root without a direct access to user's repository.
A few notes for the future:
1) Currentlly I'm planning to get hgcache path value from opsfiles.
I'm not sure if we'd need to change this later when dynamic configs are fully
rolled out. CC DurhamG
2) We might want to track size of pack files (are they still used at all?) and
indexed log files separately
Reviewed By: ikostia
Differential Revision: D23341149
fbshipit-source-id: 2a600d7a8034ac887014788f1024fb9866c3ef76
Summary:
Some user may think that the progress reporting is spammy, others can like it.
Make it configurable without disabling status completely.
We could explain the new config 'enableprogress' in the wiki for those who would like to disable the messages.
Also add 'enablestatus' under commitcloud section of config.
Reviewed By: markbt
Differential Revision: D23304206
fbshipit-source-id: 7735ae3284d19b9f1ea9da38b705a8932b34f91b
Summary:
It will be clearer to provide both names because all other commands (like `cloud
sl`, `cloud join`) use short names and if you would like to use the long one you would need to provide
'--raw-workspace' option and this is an advanced option.
Reviewed By: markbt
Differential Revision: D23274288
fbshipit-source-id: d751346a2e49cab21d1054eb196f7977c72c8764
Summary:
We have a thread blocking application. We have async libraries. This crate
provides common utilities for communicating between the blocking world and the
async world. It is intended to be a guide so that not all developers have to
get in depth understanding of Tokio in order to use async functions.
Reviewed By: quark-zju, xavierd
Differential Revision: D23222876
fbshipit-source-id: b9a61795bc917bfc664c9d6da95c9e5e2d506c79
Summary: The default method implementations on this trait were causing unused variable warnings. Prefix them with underscores to silence the warning and add `#![deny(warnings)]` to `revisionstore` to prevent future warnings from slipping through the cracks.
Reviewed By: singhsrb
Differential Revision: D23334309
fbshipit-source-id: d17b27ca0dd462e1613eac918fb595faa8637741
Summary: Move `middleware.rs` to `middleware/mod.rs` for consistency with the prevailing module file structure in the Mononoke codebase.
Reviewed By: sfilipco
Differential Revision: D23298420
fbshipit-source-id: 4f88d046a2c6ca1be2e3e315c9eea17845c6b8b3
Summary: This function used to be longer before AclChecker was replaced with PermissionsChecker. Now the function is a one-liner, so it doesn't make sense to keep it as a separate function.
Reviewed By: sfilipco
Differential Revision: D23304899
fbshipit-source-id: 23e8c4b2334cdbff21ca336aecedf6ba6c466f99
Summary: This diff fixes the issue that mkscratch does not build on Rust < 1.45.0
Reviewed By: xavierd
Differential Revision: D23325087
fbshipit-source-id: 7e737eb5bc0d75255a84c8ac353bf3a17c269300
Summary:
This was just a dumb issue. I added a test that depended on the
previous logic which didn't run on OSX.
Reviewed By: kulshrax, singhsrb
Differential Revision: D23324601
fbshipit-source-id: 259175c91c9eb54e8350f196770ff967b53ee841
Summary:
If a service is configured with no permitted paths, ensure we deny any writes
that might affect any path. This is not hugely useful, and probably means a
configuration error, but it's the safe choice.
In a similar vein, if a service is permitted to modify any path, there's not
much point in checking all the commits, so skip the path checks to save some
time.
Reviewed By: StanislavGlebik
Differential Revision: D23316392
fbshipit-source-id: 3d9bf034ce496540ddc4468b7128657e446059c6
Summary:
This tests creating, moving and deleting bookmarks using the source control
service, making sure that hooks and service write restrictions are applied
appropriately.
Reviewed By: StanislavGlebik
Differential Revision: D23287999
fbshipit-source-id: bd7e66ec3668400a617f496611e4f24f33f8083e
Summary: Implement these new thrift methods by calling the corresponding mononoke_api method.
Reviewed By: StanislavGlebik
Differential Revision: D23288002
fbshipit-source-id: 2abf1144fe524f695984a7aa472308b8bf067d45
Summary:
Use `PrefixTrie` to ensure that all service writes are to paths that are permitted
for the service.
By default, no paths are permitted. The service can be configured to allow all
paths by configuring the empty path as a permitted prefix.
Reviewed By: StanislavGlebik
Differential Revision: D23287997
fbshipit-source-id: 2b7a0df655084385f73551602d6107411d6aad2f
Summary:
Add `PrefixTrie`, which is a collection of path prefixes for testing against.
The tree is initially populated with a set of path prefixes. Once populated,
it can be tested against using other paths. These tests will return true if
the trie contains a prefix of that path.
Reviewed By: StanislavGlebik
Differential Revision: D23288127
fbshipit-source-id: 6096a9abc8e3a1bf5a8309123a46d321d9795f77
Summary:
Move handling of service write bookmark restrictions into the `bookmarks_movement` crate.
This moves `check_bookmark_modification_permitted` from `mononoke_api` onto
`SourceControlServiceParams`, where it can be called from `bookmarks_movement`.
Reviewed By: StanislavGlebik
Differential Revision: D23288000
fbshipit-source-id: e346231b183ce1533ab03130fd2ddab709176fcd
Summary:
Bookmark movement for service write will use different restrictions than hooks.
Move hook running to be controlled by an enum in preparation for adding service
write restrictions.
Reviewed By: StanislavGlebik
Differential Revision: D23287998
fbshipit-source-id: 30670d4d6666c341885b57a3f41246e52db541a2
Summary: Use bookmarks_movement to implement the bookmark move in repo_move_bookmark.
Reviewed By: StanislavGlebik
Differential Revision: D23222562
fbshipit-source-id: 31249411d9521823f90248f459eb34ed4e2faea5
Summary:
The error message for fast-forward failure is wrong. The correct way to allow
non-fast-forward moves is with the NON_FAST_FORWARD pushvar.
Reviewed By: StanislavGlebik
Differential Revision: D23243542
fbshipit-source-id: 554cdee078cd712f17441bd10bd7968b0674bbfe
Summary:
When bookmarks are moved or created, work out what additional changesets
should have the hooks run on them. This may apply to plain pushes,
force pushrebases, or bookmark only pushrebases.
At first, this will run in logging-only mode where we will count how many
changesets would have hooks run on them (up to a tunable limit). We can
enable running of hooks with a tunable killswitch later on.
Reviewed By: StanislavGlebik
Differential Revision: D23194240
fbshipit-source-id: 8031fdc1634168308c7fe2ad3c22ae4389a04711
Summary:
Move the running of hooks from in `repo_client` to in `bookmarks_movement`.
For pushrebase and plain push we still only run hooks on the new commits the client has sent.
Bookmark-only pushrebases, or moves where some commits were already known, do not run
the hooks on the omitted changesets. That will be addressed next.
The push-redirector currently runs hooks in the large repo. Since hook running has now been moved
to later on, they will automatically be run on the large repo, and instead the push-redirector runs them on
the small repo, to ensure they are run on both.
There's some additional complication with translating hook rejections in the push-redirector. Since a
bookmark-only push can result in hook rejections for commits that are not translated, we fall back to
using the large-repo commit hash in those scenarios.
Reviewed By: StanislavGlebik
Differential Revision: D23077551
fbshipit-source-id: 07f66a96eaca4df08fc534e335e6d9f6b028730d
Summary: We will shortly need a `HookManager` in the write methods of the source control service. Add one to `mononoke_api::Repo`
Reviewed By: StanislavGlebik
Differential Revision: D23077552
fbshipit-source-id: e1eed3661fe26a839e50ac4d884f4fadf793dbbb
Summary:
We want to eventually get rid of system and repo configs, but for now
they should take precedence over the dynamicconfigs. Previously we relied on
validation to remove any entries from dynamicconfig that interferes with a
system rc config, but in some code paths we didn't run that validation (like if
we loaded configs purely from Rust).
Let's just make dynamicconfig load before system configs. If validation doesn't
run, we might miss the case where dynamicconfig sets a value and the system rc
doesn't. But that's probably fine.
Reviewed By: quark-zju
Differential Revision: D23305711
fbshipit-source-id: 77b5f49d348cfa116694a641ed17e6d1184a81ab
Summary:
Dynamicconfigs are now part of our critical path. Let's remove the
option to not load them. This also let's us get rid of a circularl dependency
where loading dynamicconfigs required having already loaded some configs. This
will let us move dynammicconfig loading to be before system rc loading in a
later diff.
Reviewed By: sfilipco
Differential Revision: D23309090
fbshipit-source-id: 5138059a8ed944c3616007e7c1289b6a57be0e65
Summary:
An earlier diff moved all dynamicconfig loadding into Rust, but it lost
some of the config validation along the way. This allowed dynamicconfig values
to take precedence over system rcs when they shouldn't yet. Most notably the
remotefilelog.cachepath value.
Let's ensure validation is run on all load paths. A future diff will go a step
farther and move dynamicconfigs to be loaded before system configs so we can
ensure system configs always take precedence, until we can remove them entirely.
Reviewed By: quark-zju
Differential Revision: D23305712
fbshipit-source-id: 33a6b4c56d97fa2e2e8f3acc343a8a8868b797ef
Summary:
Python 2's `socket.error` type provided a tuple of (errno, error
string) that could be used for identifying the specific error type. In Python 3
this was switched to an object that doesn't expose such an interface. As a
result, accessing `socket.error` using an index will cause an exception, such
as:
```
File "/opt/fb/mercurial/edenscm/mercurial/keepalive.py", line 608, in safesend
if v[0] == errno.EPIPE: # Broken pipe
TypeError: 'BrokenPipeError' object is not subscriptable
```
Replace the use of indexing with accessing `.errno` instead, as this works for
both python 2 and python 3.
Differential Revision: D23294090
fbshipit-source-id: 5f901493c2db270817949d93840c6d469b17a560
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999792
fbshipit-source-id: 56e5ec68469cb9c154a5c3045ded969253270b94
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999793
fbshipit-source-id: 94e53adf5458e0bc1ebceffb3b548b7fc021218a
Summary:
Dynamicconfig was throwing errors if hgrc.dynamic wasn't writable.
Let's eat those errors for normal read operations. We still treat it as an error
for straight hg debugdynamicconfig invocations.
Reviewed By: quark-zju
Differential Revision: D23301100
fbshipit-source-id: ed0bd1282d2c7ee747f0909c238a5fa07b7bc9bc
Summary:
We've seen user reports of this error. Let's add a test to demonstrate
it. The next diff will fix it.
Reviewed By: sfilipco
Differential Revision: D23309612
fbshipit-source-id: 6fb9e4e65d3351fa29812fc75095d054465cfe13
Summary: Use `Subprocess` in `win/utils` to call `edenfsctl redirection fixup` after mount is done.
Reviewed By: wez
Differential Revision: D22958764
fbshipit-source-id: a485994a3816169299e8514a5c355f3d37edad99
Summary: Some clean up to do. `Process` will crash the entire process if `Pipe` is ever `std::nullptr`. So let's not give it a default argument `std::nullptr`.
Reviewed By: xavierd
Differential Revision: D22958765
fbshipit-source-id: 0c35e805f24a0d572bbc08efc97e59a37d0cbf88
Summary: With D22956659, we can now use mkscratch on Windows with EdenFS.
Reviewed By: xavierd
Differential Revision: D22956983
fbshipit-source-id: 995073cbc89d5cb23dbb9c1a58926f8c51f0a896
Summary:
On Windows, Rust's `std::fs::canonicalize` [1] will generate extended-length path that will include a `\\?\` prefix [2]. This has subsequently cause `encode` to generate a path that contains a question mark, which is an invalid path on Windows.
This diff teaches `encode` to handle extended-length path on Windows. It essentially converts the path back so it no longer contains the prefix.
[1] http://doc.rust-lang.org/1.45.2/std/fs/fn.canonicalize.html
[2] https://docs.microsoft.com/en-us/windows/win32/fileio/naming-a-file#maximum-path-length-limitation
Reviewed By: wez
Differential Revision: D22956659
fbshipit-source-id: 54691e204d7cb481bdb40f62c6520c0f70c3f648
Summary:
In python 3 curses sometimes throws an error when weird keys are
pressed. I'm not certain exactly what key causes the problem, but let's just
prevent all such errors from crashing the process.
Reviewed By: quark-zju
Differential Revision: D23310301
fbshipit-source-id: a9684ce6f690d0753ff9956ef9f13c330eb0a77b
Summary:
By making the EdenDispatcher less Windows dependant, we can more easily move it
into a non-Windows specific location later.
Reviewed By: chadaustin
Differential Revision: D23298028
fbshipit-source-id: 21726677808a9b8ce3d3e211dd65d9e47caad569
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999141
fbshipit-source-id: e2e4177e56db85f65930b67a9e927a5c93b652df
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999142
fbshipit-source-id: 04cea361ea6270626e7ff77255e3dc75875ece97
Summary:
Rust doesn't have named arguments as with positional it's hard to keep track
of all of them if there're many. I'm planning to add one more so let's switc to
struct.
Reviewed By: krallin
Differential Revision: D22999143
fbshipit-source-id: 54dade05f860b41d18bebb52317586015a893919
Summary:
If the imported commit has manifest id with all zeros (empty commit). Blobimport job can't find it in blobstore and returns error D23266254.
Add an early return when the manifest_id is NULL_HASH.
Reviewed By: StanislavGlebik
Differential Revision: D23266254
fbshipit-source-id: b8a3c47edfdfdc9d8cc8ea032fb96e27a04ef911
Summary:
Based on [user report](https://fb.workplace.com/groups/scm/permalink/3128221090560823/).
Note that slices in rust behave differently and if index exceeds slice size this will always be panic. My fix was based on assumption that behavior should be similar to python.
Reviewed By: quark-zju
Differential Revision: D23263922
fbshipit-source-id: 3d2a1a1b59f14e43b1f1a2b7102982b11637c0b4
Summary:
Having the type of data fetched can help in debugging where these fetches are
comming from. In the currently logs figuring out if a data fetch is blob or
tree requires some manual work. When looking at a big bunch of fetches this is
not super practical.
So this includes this info in our logging.
Reviewed By: chadaustin
Differential Revision: D23243444
fbshipit-source-id: 9abe5180c5d2afc0d02b27ba6a6b76401e86556e
Summary:
This could help simplify the graph a lot for repos with lots of merges. For
example, logging tags on linux.git looks like:
o fb893de3 Yesterday at 17:28 master
├─┬─┬─┬─┬─┬─┬─┬─┬─┬─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o bcf87687 Aug 02 at 14:21 v5.8
╷ ╷ ╷ ╭─────┬─┬───┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o 92ed3019 Jul 26 at 14:14 v5.8-rc7
╷ ╷ ╷ ╭─────┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ba47d845 Jul 19 at 15:41 v5.8-rc6
╷ ╷ ╷ ╭─┬─┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ o 11ba4688 Jul 12 at 16:34 v5.8-rc5
╷ ╷ ╷ ╭─┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ o dcb7fd82 Jul 05 at 16:20 v5.8-rc4
╷ ╷ ╷ ╭─┬─┬─┤
╷ ╷ ╷ ╷ ╷ o ╷ 9ebcfadb Jun 28 at 15:00 v5.8-rc3
╷ ╷ ╭─┬─┬─╯ ╷
╷ ╷ ╷ ╷ o ╷ 48778464 Jun 21 at 15:45 v5.8-rc2
╷ ╷ ╷ ╭─╯ ╷
╷ ╷ ╷ o ╷ b3a9e3b9 Jun 14 at 12:45 v5.8-rc1
╭─┬─┬─┼─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─╮
╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 3d77e6a8 May 31 at 16:49 v5.7
╭─┬─┴───────┬───────────┬─┬───┬─╮
╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 9cb1fd0e May 24 at 15:32 v5.7-rc7
╷ ╰─────────┬─────────────┬─┬─┬─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o b9bbe6ed May 17 at 16:48 v5.7-rc6
╭───────────┬─────────────┬─┬─┬─╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o 2ef96a5b May 10 at 15:16 v5.7-rc5
╭───────────┬─────────────┬─┬─╯
╷ ╷ ╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 0e698dfa May 03 at 14:56 v5.7-rc4
╭───────────┴───────────┬─┬─╮
o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 6a8b55ed Apr 26 at 13:51 v5.7-rc3
╰─────────────────┬───────┬─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ae83d0b4 Apr 19 at 14:35 v5.7-rc2
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╭─┤
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ 8f3d9f35 Apr 12 at 12:35 v5.7-rc1
╭─┬─┬───────┬─────┬─┬─┬─┬─┼─╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ ╷ 7111951b Mar 29 at 15:25 v5.6
╷ ╭─────────┬─────┬─┬─┬─┴───╮
╷ ╷ ╷ ╷ ╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ 16fbf79b Mar 22 at 18:31 v5.6-rc7
╷ ╷ ╭───────┴─────┬─┬─┬─────╮
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o fb33c651 Mar 15 at 15:01 v5.6-rc6
╷ ╭─┬─────────────┬─┬─┬─────╯
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ 2c523b34 Mar 08 at 17:44 v5.6-rc5
╷ ╭─┬─────────────┬─┬─╯ ╷
╷ ╷ o ╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ 98d54f81 Mar 01 at 14:38 v5.6-rc4
╷ ╭─┴─────────────┬─╮ ╷
╷ ╷ ╷ ╷ ╷ ╷ ╷ ╷ o ╷ f8788d86 Feb 23 at 16:17 v5.6-rc3
....
And with simplification turned on, it looks like:
o fb893de3 Yesterday at 17:28 master
├─╮
o ╷ bcf87687 Aug 02 at 14:21 v5.8
╷ ╷
o ╷ 92ed3019 Jul 26 at 14:14 v5.8-rc7
╷ ╷
o ╷ ba47d845 Jul 19 at 15:41 v5.8-rc6
╷ ╷
o ╷ 11ba4688 Jul 12 at 16:34 v5.8-rc5
╷ ╷
o ╷ dcb7fd82 Jul 05 at 16:20 v5.8-rc4
╷ ╷
o ╷ 9ebcfadb Jun 28 at 15:00 v5.8-rc3
╷ ╷
o ╷ 48778464 Jun 21 at 15:45 v5.8-rc2
├─╯
o b3a9e3b9 Jun 14 at 12:45 v5.8-rc1
╷
o 3d77e6a8 May 31 at 16:49 v5.7
╷
o 9cb1fd0e May 24 at 15:32 v5.7-rc7
╷
o b9bbe6ed May 17 at 16:48 v5.7-rc6
╷
o 2ef96a5b May 10 at 15:16 v5.7-rc5
╷
o 0e698dfa May 03 at 14:56 v5.7-rc4
╷
o 6a8b55ed Apr 26 at 13:51 v5.7-rc3
╷
o ae83d0b4 Apr 19 at 14:35 v5.7-rc2
╷
o 8f3d9f35 Apr 12 at 12:35 v5.7-rc1
╷
o 7111951b Mar 29 at 15:25 v5.6
╷
o 16fbf79b Mar 22 at 18:31 v5.6-rc7
╷
o fb33c651 Mar 15 at 15:01 v5.6-rc6
╷
o 2c523b34 Mar 08 at 17:44 v5.6-rc5
╷
o 98d54f81 Mar 01 at 14:38 v5.6-rc4
╷
o f8788d86 Feb 23 at 16:17 v5.6-rc3
....
Under the hood, the difference is how `reachableroots` gets calculated.
See also D22657197 (a5c36fd0b1) and D22368827 (da42f2c17e).
Since the old behavior almost always seems confusing to human. The new
config is turned on by default (but only takes effect if the "segments"
backend is used).
Reviewed By: sfilipco
Differential Revision: D23095468
fbshipit-source-id: f0fc631d2d9a00e3b36744e4236b43d230d10687
Summary:
Previously pieces of the command line for a process were seperated by `\0`.
This makes them a bit hard to read and also makes running queries on them
harder. Converts these `\0` back to spaces to fix this.
see https://fb.workplace.com/groups/edenfs/permalink/1446711485499079/ for
more motivation.
Reviewed By: wez
Differential Revision: D23266909
fbshipit-source-id: e4a9284e04039fcd971bed0d6e21d220e946acdb
Summary:
The files in commit cloud `References` structures are optional. Handle them
not being present.
Reviewed By: quark-zju
Differential Revision: D23266786
fbshipit-source-id: ed7128bc7e6b762d3509d77b40a00b77885191b9
Summary: This makes it a bit easier to track down perf issues printed by RUST_LOGs.
Reviewed By: sfilipco
Differential Revision: D23095463
fbshipit-source-id: 78221a1992389f512fac6e6e633be6d19123e04a
Summary:
Use `git config core.autocrlf false` to silent warnings like:
```
$ git add alpha
+ warning: LF will be replaced by CRLF in alpha.
+ The file will have its original line endings in your working directory
```
Reviewed By: sfilipco
Differential Revision: D23270146
fbshipit-source-id: af3bf241edb9f615bcc285b51cc491385f208039
Summary: The command is needed to restore a deleted workspace
Reviewed By: markbt
Differential Revision: D23250376
fbshipit-source-id: e24a7cbc0aad004291853b4c34d7474789aa9c2b
Summary:
The fuzz tests need `TestContext::id_dag()`, which was removed by D20471712 (1fb5acf242).
Restore it so fuzz tests can run. This is mainly to check the new `range`
function.
The `range` fuzz test does find an issue caused by `>` written as `>=`
relatively quickly.
Reviewed By: sfilipco
Differential Revision: D23106176
fbshipit-source-id: e9540cc932503a9d54246d24c70bac829fcb13df
Summary: Ensure that the commit text is verified, but do not verify git hashes.
Reviewed By: DurhamG
Differential Revision: D23095464
fbshipit-source-id: e62341f6c7258c6f18b7cc75088c25dfc7040ab1
Summary:
The immediate goal is to run benchmarks on a commit graph provided by a git
repo without converting a whole (large) repo from git to hg. Note git repos can
be cloned in a shallow way so it only contains the commit graph. For example:
git clone https://github.com/torvalds/linux --filter=tree:0 -n
Note: The above command writes repositoryformat=1 in `.git/config`
which is not supported by libgit2. Manually editing it to repositoryformat=0
would enable libgit2 to read it for this crate's use-case.
In the longer term we might want to extend the support so refs/trees/files can
be read/written directly via the git repo based on this work. However that's
currently beyond scope.
Reviewed By: DurhamG
Differential Revision: D23095467
fbshipit-source-id: 868beb0c7de60453b47962639863eb8f7e3f5753
Summary: Migrate to concrete types so it can be typechecked.
Reviewed By: DurhamG
Differential Revision: D23095469
fbshipit-source-id: 27c6da30ca8a1329df544cd2ded7d9734593e48a
Summary:
Read git commit graph and migrate them to `dag::Dag`.
This allows using Rust dag abstractions on the git
commit graph.
Reviewed By: DurhamG
Differential Revision: D23095471
fbshipit-source-id: 2163701350ce82ce6e97074e56ad5877f3c9c158
Summary:
Add alternative paths will be faster if changelog2 is used, since they are
backed by native paths.
Add a config option to disable the fast paths if they cause issues.
Reviewed By: DurhamG
Differential Revision: D23036074
fbshipit-source-id: 489b6eac64148867c209d595623d0b9c21ad1d5a
Summary:
OSX doesn't support touch -d. Let's just skip that part of the test on
that platform. This fixes the OSX build.
Reviewed By: singhsrb
Differential Revision: D23253475
fbshipit-source-id: 0eccb884cbdd4bf0a4068fbf943ba7dac9df4e04
Summary:
Detect the "segments" backend and calculate the revset differently.
Practically, with collapse-obsolete disabled, the time of related revset
calculation drops from 0.14s to 0.03s in my fbsource repo.
The `obsolete()` set calculation is expensive (0.4-0.6s) and a bit more
expensive with the new DAG APIs, which will be addressed in upcoming
changes. EDIT: Addressed by D23036063.
Reviewed By: DurhamG
Differential Revision: D23036055
fbshipit-source-id: 71140a88599cc68bfa90d564c786da89b3ebd38b
Summary:
The `compact` template is rarely used and is coupled with rev numbers (ex. rev
number decides what "parents" to show). Use explicit templates. This makes the
test change easier to check.
Reviewed By: DurhamG
Differential Revision: D23036076
fbshipit-source-id: f2cc0f25191711fa7d846a8ad38aee8fb9171273
Summary:
The `notbackedup()` revset is used as part of `summary` that prints information
at the end of `smartlog`. It can take hundreds of milliseconds if there are
many heads. Detect segmented changelog and use a fast path for it.
Practically this reduces `summary` from 594ms to 91ms for me:
With segmented changelog (doublewrite backend) and new code path:
91 \ summary status.py:23
2 \ currentworkspace workspace.py:121
3 | _get (2 times) workspace.py:110
3 | read (2 times) config.py:195
3 | parse (2 times) config.py:116
2 | compile (14 times) util.py:1464
3 \ __init__ syncstate.py:44
82 \ revs localrepo.py:1203
With revlog and old code path:
594 \ summary status.py:23
2 \ currentworkspace workspace.py:121
4 | _get (2 times) workspace.py:110
3 | read (2 times) config.py:195
3 | parse (2 times) config.py:116
3 | compile (14 times) util.py:1464
3 \ __init__ syncstate.py:44
46 \ revs localrepo.py:1203
539 \ _iterfilter smartset.py:647
538 | <lambda> (1565 times) commitcloud/__init__.py:371
537 | __contains__ (1565 times) smartset.py:1039
533 | _consumegen (17355 times) smartset.py:1122
Reviewed By: markbt
Differential Revision: D23036075
fbshipit-source-id: 09dcc34f34a42814c6526e558d40b4d75ba9d75f
Summary: Expose the Rust API so `getdag` can choose to skip successors or predecessors.
Reviewed By: markbt
Differential Revision: D23036056
fbshipit-source-id: 30cd437c5420d2d10176e33ef9de98814046f4ce
Summary:
The new path does not calculate the complicated `successorssets`, and is
known to make wez's repo operations significantly faster (which, I suspect is
slowed by a very long chain).
The new code is about 3x faster on my repo too:
# before
In [1]: list(repo.nodes('draft()'))
In [2]: %time len(m.mutation.obsoletenodes(repo))
CPU times: user 246 ms, sys: 42.2 ms, total: 288 ms
Wall time: 316 ms
Out[2]: 1127
# after
In [1]: list(repo.nodes('draft()'))
In [2]: %time len(m.mutation.obsoletenodes(repo))
CPU times: user 74.3 ms, sys: 7.92 ms, total: 82.3 ms
Wall time: 82.3 ms
Out[2]: 1127
Reviewed By: markbt
Differential Revision: D23036063
fbshipit-source-id: afd6ac122bb5d8d513b5cdc033e04d2c377286eb
Summary:
Optimize get_dag:
- Avoid parsing mutation entries once they are parsed, by keeping an in-memory
`parent_map`.
- Pass `heads` to `add_heads` so the segments are less fragmented, cycle break
helper is more efficient.
The `heads` optimization is effective. Practically this makes `get_dag` about 2x faster.
This has a subtle change on cycle handling - full cycle without any non-cycle heads will
be ignored. Practically cycles are rare so it might be okay.
Together with improvements on the `dag` side, `get_dag` is about 4x faster.
Reviewed By: markbt
Differential Revision: D23036062
fbshipit-source-id: 3dc407b562f7ebf2543a87c5cd651ad6a2339d67
Summary:
If there is no new master segments, it's still possible to have new non-master
segments. Fix the loop condition so we don't skip building non-master segments.
Reviewed By: sfilipco
Differential Revision: D23095465
fbshipit-source-id: 46eb9d5b5f2b04241981558646e0bc090652abce
Summary:
I noticed that high-level segments are somehow not built for non-master vertexes.
Add a test to demonstrate the issue.
Reviewed By: DurhamG, sfilipco
Differential Revision: D23095466
fbshipit-source-id: c5a6da14bdfabcf7c432f6c6dfe096c71cc10ee9
Summary: This is useful to investigate internals of dag calculations.
Reviewed By: sfilipco
Differential Revision: D23095473
fbshipit-source-id: 4750c1b4ffad32b1317051d17db9659aaaed59c4
Summary:
Follow up of the previous change by actually using the flat segments to build
segments. This significantly improved the perf. `cargo bench --bench dag_ops`
shows:
building segments (old) 774.109 ms
building segments (new) 143.879 ms
Besides, a `O(N^2)` update to `head_ids` is changed. It improves performance
when the graph has many heads (ex. the mutation graph).
Reviewed By: sfilipco
Differential Revision: D23036080
fbshipit-source-id: 033565700f253c6f20e30a00adb6b579921d6679
Summary:
While testing the `obsolete()` set, I found an in-memory segmented DAG takes
10x time to build than a HashMap DAG.
Part of the inefficiency is to use a translated "parent_func" that round-trips
through Id and Vertex, used by segment building logic. This diff makes
`IdMap::assign_head` return flat segments, so we don't need a translated
"parent_func" to build flat segments.
This diff only adds checks to make sure the parent_func (Id version) matches
the segments. The next diff switches the segment building to not use the
translated parent_func.
Reviewed By: sfilipco
Differential Revision: D23036060
fbshipit-source-id: 99137f4b5be455cdf43218ba23eb3954b6d9e05a
Summary:
This affects the `tonodes` API in the Python world. Practically this will bind
the main commit graph to sets like draft, public.
The `ToSet` requirement on `DagAlgorithm` has to be removed to avoid stack
overflow of rustc resolving constraints.
Reviewed By: sfilipco
Differential Revision: D23036077
fbshipit-source-id: 912b924e29611680ab6b2ee4dbcd7ab39824409a
Summary: This will be useful for the `obsolete()` set.
Reviewed By: sfilipco
Differential Revision: D23036072
fbshipit-source-id: 2f944ef31cf19f902622d90545fa02b7dda89221
Summary:
If two sets have different IdMap, their Ids cannot be compared directly
for correctness.
Reviewed By: sfilipco
Differential Revision: D23036068
fbshipit-source-id: e800e8273b95c1f8174236e0f30445db7fd44556
Summary: This is similar to the previous change. This allows "binding" IdMaps to sets.
Reviewed By: sfilipco
Differential Revision: D23036058
fbshipit-source-id: ec1b1ec73e949ad4865aecf17bfcc5c1ca723e0d
Summary:
This trades a bit performance (calculating the snapshot) for correctness (no
pointer reuse issues) and convenience (set captures dag information with them
and enables use-cases like converting NameSet from another dag to the
current dag without requiring extra `dag` objects).
Reviewed By: sfilipco
Differential Revision: D23036067
fbshipit-source-id: 2e691f09ad401ba79dbc635e908d79e54dadca5e
Summary:
If `x` and `y` come from a same graph, `x & y` is more efficient than
`y & x` if `y` is larger. However, if `x` and `y` are from different
graphs, the `FULL` hint can no longer accurately predict which one
is larger. Therefore the swap should be avoided.
Reviewed By: sfilipco
Differential Revision: D23036081
fbshipit-source-id: fe3970fc38c853b36689bfd0ee1dec20643ace78
Summary:
For sets like `obsolete()`, `merge()`, they could have a fast "contains" path:
Just check the given commit without calculating a full set. It's also possible
to have a relatively efficient code path to return StaticSet (for obsolete()),
or IdStaticSet (for merge(), by checking flat segments). This diff adds a
`MetaSet` that allows defining two fast paths separately.
This will be used for the `obsolete()` set in upcoming changes.
Reviewed By: sfilipco
Differential Revision: D23036059
fbshipit-source-id: 06e6f90e7e9511626a12cfa729c306ff539256d2
Summary:
Before this change, `flush` with empty changes but `master` moves will cause an
error, because the `parents_func` only contains "pending changes", aka. new
vertexes. The `parents_func` does not know `master` and `master` is needed to
re-assign them from the non-master to the master group.
With the snapshot API, things become easier. We just take a snapshot before
reloading, and use the snapshot to answer parent_names.
Reviewed By: sfilipco
Differential Revision: D22970569
fbshipit-source-id: 99a25857ba98792edff69985c16df118a560ffb0
Summary:
This API allows the underlying Dag to provide a snapshot. The snapshot can then
be used in places that do not want a lifetime (ex. NameSet).
Reviewed By: sfilipco
Differential Revision: D22970579
fbshipit-source-id: ededff82009fd5b4583f871eef084ec907b45d33
Summary:
Make it possible to snapshot a Dag. This is useful for cases where another
struct wants access to the Dag without lifetimes. Namely, the LazySet can
might want to keep a snapshot of Dag.
Reviewed By: sfilipco
Differential Revision: D22970568
fbshipit-source-id: 508c38d3ffac2ffcd2e682578c3c5e5787ea3bcf
Summary:
The only intended use of the inverse DAG is to implement the Python dag
interface in `dagutil.py`. D22519589 (2d4d44cf3d) stack changed it so the Python dag
interface becomes optional. Therefore there is no need to keep the inverse DAG
interface, which is a bit tricky on sorting.
Reviewed By: sfilipco
Differential Revision: D22970581
fbshipit-source-id: 58a126b41d992e75beaf76ece25cb578ee84760b
Summary:
The nameset serves as a bridge for Rust NameSet sets. It's different from the
Rust IdSet in a way that it supports all kinds of Rust NameSet (lazy or
non-lazy).
Unlike the native Rust binding, the added nameset uses rev numbers and fit in
the Python smartset framework.
Reviewed By: sfilipco
Differential Revision: D23036066
fbshipit-source-id: 060b3927dda6cd2275af21b093729c7e0e88ee7c
Summary: The Rust "flush(masternodes)" API does not handle nullid. Filter it out from Python.
Reviewed By: sfilipco
Differential Revision: D22970578
fbshipit-source-id: 671fe950948067a0b3f97c5b65ff2b9b7ed4b631
Summary:
By default, `torevs` calls Python iteration for non-list, non-spans Python
objects. The `idset` object has the `spans` which can be used as a fast
path.
Reviewed By: DurhamG
Differential Revision: D22970580
fbshipit-source-id: f491404ba803c4468c17cd74daaea90f46b8b38b
Summary:
This allows certain code paths to use `dageval` with the idea that `dageval` is
going to be faster.
Reviewed By: sfilipco
Differential Revision: D22970576
fbshipit-source-id: ba4536a55691de63640e574c898320629c6d7b2f
Summary: This allows migrating between a few changelog formats we have.
Reviewed By: DurhamG
Differential Revision: D22970571
fbshipit-source-id: d6b577ae5beb72a43fff999c26c35fcdc33e8f83
Summary:
This will be used for migrating revlog DAG to segmented changelog. It does not
migrate commit text data (which can take 10+ minutes).
Reviewed By: DurhamG, sfilipco
Differential Revision: D22970582
fbshipit-source-id: 125a8726d48e15ceb06edb139d6d5b2fc132a32c
Summary: For now it just prints some details about the changelog backend.
Reviewed By: DurhamG, sfilipco
Differential Revision: D22970573
fbshipit-source-id: 719a5e5bb6f3856df3c9357e47daa9e7c8584952
Summary:
This option is needed to validate Mononoke Smartlog against the original
infinitepush Commit Cloud Smartlog. This option is advanced and can be removed
after full migration to the Mononoke backend.
Reviewed By: markbt
Differential Revision: D23241251
fbshipit-source-id: e550334b104d18bb58d39acb8540ebdc9e711c4e
Summary:
We've been using a hard coded list to determine whether
mononoke is available, and that list is falling behind the current
state of our backend migration.
This commit removes the hard coded list in favor of testing
`remotefile.reponame`. If that is configured then it holds
the mononoke reponame.
I'm making the assumption that it being set implies that mononoke
is available for that repo. That may not be 100% true, but
it appears to be effectively true for the intersection of repos
known to `fbclone` and the migrated set of repos.
If this code decides that mononoke is supported, the behavior
in EdenFS is to then attempt to use the SCS to fetch tree
data using the returned `repo.name`. That appears to be the
only way that this information is used today.
Reviewed By: quark-zju
Differential Revision: D23214471
fbshipit-source-id: 17b6475b891df5423dca0c18ddae0838795f713a
Summary:
Related commits: D23214677 (dcb565409d), D23213192
In the previous commits we added phabricator callsigns to the repo configs.
Since we can extract the callsigns from them, we don't need the callsign
flag for repo_import tool. This diff removes the flag and uses the config variable.
Reviewed By: StanislavGlebik
Differential Revision: D23240398
fbshipit-source-id: d8b853d37e21be97af42e9f50658b9f471f8fc48
Summary:
Dynamicconfigs compares the timestamp of config files with the current
timestamp to determine when to regenerate. If the timestamp of the config file
is newer than the current timestamp, Rust throws an exception. Let's handle that
case and treat it as if the file was just created instead of crashing.
Reviewed By: quark-zju
Differential Revision: D23230216
fbshipit-source-id: ca185de7dfca46953e04ec08c84668eda6d749bd
Summary: The `map_err` call can be done with the new future from `compat()`.
Reviewed By: StanislavGlebik
Differential Revision: D23239251
fbshipit-source-id: c80609ae0a975bc54253784e002a07a048651aa3
Summary:
Add tests for basic functionality of `resolve_bookmark` and `list_bookmarks`,
ensuring that they correctly go through the warm bookmarks cache.
`list_bookmarks` was still using old-style streams, so upgrade it to new streams.
Differential Revision: D23239250
fbshipit-source-id: f78abae2d382263be76c34f1488249677134a74d
Summary:
If a cloud sync requires both fetching from the cloud and uploading new state to the cloud,
commit the transaction between the two steps, so that a successful cloud fetch is not
rolled back by failure to send to the cloud.
While we're here, limit the number of sync attempts to 3 in one go.
Reviewed By: farnz
Differential Revision: D23211846
fbshipit-source-id: fa97165d94eee973284ff3d00466387b3041306c
Summary: Avoid trying to send local obsmarkers if evolution is not enabled.
Reviewed By: farnz
Differential Revision: D23210800
fbshipit-source-id: dc247a18e92f6f5454eeed520854dd6254f66257
Summary:
Large numbers of pending obsmarkers can cause commit cloud `update_references`
requests to fail.
Prevent this from happening by only syncing the most recent obsmarkers.
Obsmarkers are in the process of being deprecated and removed, so this
shouldn't be a problem for long.
Reviewed By: farnz
Differential Revision: D23211621
fbshipit-source-id: 56a2e103722c3c162eacdb62638ff8ff614d5815
Summary: This fixes the Windows build.
Reviewed By: farnz
Differential Revision: D23212195
fbshipit-source-id: 159f3ddebf6a97f52f9b6c80ef19315c8f4b0c85
Summary:
If the warm bookmarks cache doesn't contain the bookmark we are looking for,
this might just be because it's a scratch bookmark, which aren't included in
that cache.
Always request the bookmark from the backing db if the cache misses.
Reviewed By: StanislavGlebik
Differential Revision: D23238009
fbshipit-source-id: c8843f1974ba14f148e30ba78a38eb710e7383b6
Summary:
We already had a logic that prints if we are about to run an expensive
getbundle. However this logic prints a warning after we've fetched 1M commits
already, and user would have to wait for a long time to get this message.
However in some cases we can give this warning very quickly. For example, if
the lowest "heads" generation number is >1M commits away from highest "common"
generation number, then we can print the warning right away.
Differential Revision: D23213482
fbshipit-source-id: 67e2399ca958703129cf3c22d82ce48cbbdcd2d1
Summary: Update bindings to expose the DoubleWrite backend and the DescribeBackend API.
Reviewed By: sfilipco
Differential Revision: D22970574
fbshipit-source-id: bdb52ff21dd0b9ffa0be214b4a4824025f460092
Summary:
This allows importing from other DAGs. It will be used to import revlog DAG to
the new segmented format.
Reviewed By: sfilipco
Differential Revision: D22970572
fbshipit-source-id: 0a183e7b64831574cc9c60d4639124d02d19cf43
Summary:
This allows dag to use renderdag in tests to verify graph result. Previously
it was hard because dag <-> renderdag would form circular dependency.
It also make it possible to implement more efficient and integrated fast paths
for graph rendering.
Reviewed By: sfilipco
Differential Revision: D22970570
fbshipit-source-id: 526497339bd7aa8898d1af4aa9cf6d2a6797aae0
Summary: This will be used to describe what the commit graph backend is.
Reviewed By: sfilipco
Differential Revision: D22970577
fbshipit-source-id: 753efdbdd4466730ece758d9f4789fbd21e2801b
Summary:
This allows us to try segmented changelog while maintaining revlog
compatibility.
Reviewed By: sfilipco
Differential Revision: D22970583
fbshipit-source-id: 7c43cdadd76300e76e89f38aac5ed3ecc0cff728
Summary: The value of linkrev is invalid with segmented changelog. Do not resolve it.
Reviewed By: sfilipco
Differential Revision: D23036079
fbshipit-source-id: 8f8b097458bc694327db6ba4e2dc4107bdf44157
Summary:
Phabricator reports some repos as being backed by git, and the phrevset
code path in that case uses the python 2 hex encoding. Let's just use the hex
function directly.
Reviewed By: quark-zju
Differential Revision: D23207771
fbshipit-source-id: 88de3153e52a3db456c17ab4ca4b9c9dd6049855
Summary:
In the next diff I'd like to compute generation number first, and then call
DifferenceOfUnionsOfAncestorsNodeStream. To avoid refetching these numbers
again let's create a function that accepts a vector of (ChangesetId,
Generation) pairs.
While here I also made the order more consistent: now we have "hashes"
parameters always in front of "excludes"
Differential Revision: D23212883
fbshipit-source-id: 11e0a1494126f84b36e3e33e65071449db5840d2
Summary:
Previously for undesired fetches of lfs files we were logging 0. Let's log the
real path instead
Reviewed By: ikostia
Differential Revision: D23209754
fbshipit-source-id: 7a893b257a89332a5169ab2072ecf48ae94b91e0
Summary: Rather than using a Python program, simulate having no editor during tests using `internal:none`.
Reviewed By: quark-zju, ikostia
Differential Revision: D23152446
fbshipit-source-id: 5560f58885ee5959c62f0ac8bcf0483b9c3072f6
Summary:
The commandserver was setting args as bytes when they should be
strings. This was breaking arg parsing for users using hglib, which communicates
with hg via the commandserver.
Reviewed By: sfilipco, singhsrb
Differential Revision: D23202623
fbshipit-source-id: f71e4145211069b2f7ed6935fe86585061cbf8b3
Summary:
The old discovery logic is incompatible with the rust changelog because
the changelog is not based on revlog.
Reviewed By: sfilipco
Differential Revision: D23036065
fbshipit-source-id: 633fdf8726d40cb14e63c3df2f5573d35cb1640a
Summary:
They are faster if the new dag backend is used. For example, `headsancestors`
will be backed by a fast native path, where the `head(ancestors(x))` revset
would be unusably slow if the new dag backend is used.
Reviewed By: sfilipco
Differential Revision: D23036069
fbshipit-source-id: 75df1e2240520a6e560bc5d8414ba2f4fb7a4674
Summary:
It read the sharedpath manually but needed to read it as utf8, not
bytes.
Reviewed By: quark-zju
Differential Revision: D23194858
fbshipit-source-id: aa73c2cc782070ba6c4d2c441d6f5338842b137f
Summary:
sshpeers were being cleaned up twice. Once during the explicit close()
and once during the __del__ destruction. This caused it to double count the
reported _totalbytesreceived.
Reviewed By: quark-zju
Differential Revision: D23178435
fbshipit-source-id: da76158be561511f2f2a5e255fd03aa4a6b78da0
Summary:
We missed a Windows http client breakage because our LFS server integration
wasn't run on Windows. Let's enable the fb feature for all our cargo test runs.
Reviewed By: singhsrb
Differential Revision: D23140315
fbshipit-source-id: 46cc533c1e543ffc32d472b49a8f6daeee3b5009
Summary:
Aux data wire protocol part 1: field annotations & basic compatibility model.
Annotates fields in `file`, `tree`, and `complete_tree` wire structs with `#[serde(rename = "N", default, skip_serializing_if = "is_default")]`. I've avoided using `#[serde(default)]` on the container structs themselves because this can cause some confusion / incorrect behavior if not used carefully. Consider a wire struct `FooRequest` with a field of type `Option<Bar>`. `Option<Bar>` defaults to `None`. If `FooRequest`'s `Default` implenentation sets the field's default to `Some(bar)`, a `FooRequest` explicitly constructed with `None` for the field will be serialized with the field omitted (because it passes `is_default`) and will be deserialized on the server as `Some(bar)`, causing incorrect behavior. To address this, we'd need to change the `is_default` function used with `skip_serializing_if` to check against the field's default value as set by the container, which isn't trivially possible without some sort of reflection (please correct me if you know a good way to achieve this). This is unfortunate, as it'd be very desirable for the container to be able to set defaults different from the individual field type defaults, for cases where one boolean, for instance, should default to true. As-is, we'd need to address this with wrapper types instead, where we can fully control the `Default` implenentation.
We can, of course, address this by providing an alternate `skip_serializing_if` function to fields with default that doesn't match that set by the container. This will need to be done carefully, though, to avoid the issue I described above.
Currently the JSON module manually serializes and deserializes all the top-level request objects, so the rename annotation doesn't impact it. We can add `#[serde(alias = "rustfieldname")]` if we'd like the server and client to be able to accept manually-crafted requests and responses with explicit field names. This could also be useful to replace the manual parsing in the JSON module, but can't replace the manual serialization in a clean way. We'd need to introduce a second copy of the wire types, without the serde `rename` attribute, to allow serializing with the actual rust field name.
I've only modified the `tree`, `file`, and `complete_tree` modules. I intend to eventually update the rest of the edenapi protocol later on, when the implementation of `file` and `tree` are complete / stable. This will give us a chance to fix any mistakes before copying the design to more places.
Note: I do not intend to keep to proper wire protocol compatibility at this stage in the implementation. Expect field numbers to be re-used by non-compatible changes.
Reviewed By: kulshrax
Differential Revision: D23172756
fbshipit-source-id: 39976ed4bede892bd6981f9c3f23557a91f9028b
Summary:
hg `cloud join -w myworkspace --switch/--merge` should require --create option for all not default
workspaces if the workspace doesn't already exist.
I also enabled reusing of http connection, so it makes `hg cloud switch` much faster.
It is less than 2 sec now.
Reviewed By: markbt
Differential Revision: D23188491
fbshipit-source-id: 9a2ba2df4ba91e9dba9484cd49fc886f70b09880
Summary: In py3, a string pattern cannot be used to match against a bytestring text. Convert the pattern to a bytestring, which should be compatible with python2 without additional changes.
Reviewed By: DurhamG
Differential Revision: D23135472
fbshipit-source-id: fcc0b111c646011f5b556f032ef85ef326cb9511
Summary: Minor refactor: moved the tests from main to their own file, because main was getting too large and found it hard to navigate through the code.
Reviewed By: StanislavGlebik
Differential Revision: D23188766
fbshipit-source-id: a2b2e32c77587f95c07a0bb02a4957e3671dd2c6
Summary:
This largely moves connection accepting from old style bytes, futures and tokio
to updated versions, while keeping some parts at old bytes/futures in order to
remain compatible with the rest of the Mononoke codebase.
Division lies on `Stdio` which maintains old channels, stream and futures,
while the socket handling, connection acception and wire encoding is updated.
With the updated futures, we now wait for the forwarding stream to have
succeeded before considering a connection fully handled.
Other notable changes:
- futures_ext now a mini codec Decoder instead of relying on NetstringDecoder,
which has been updated to use bytes 0.5
- hgcli has been modified to use updated NetstringDecoder
- netstring now requires the updated bytes 0.5 crate
- the part in connection_acceptor was handling repo/security logic is now part of repo_handler (as it should have been), connection_acceptor now only handles networking and framing
- tests now verify that the shutdown handler is triggered
Reviewed By: krallin
Differential Revision: D22526867
fbshipit-source-id: 34e43af4a0c8b84de0000f2093d7fffd3fb0e20d
Summary:
As noted in the documentation for it, this can be removed once get and prefetch
return a continuation. This is now done, and thus we can remove it entirely.
Mis-use of it caused data to be fetched twice: once by memcache, and the second
one by getpackv2.
Reviewed By: singhsrb
Differential Revision: D23123344
fbshipit-source-id: 9ac0594faaba94ead04a8bb9035e14809a706641
Summary: The python code stripped new lines but the Rust code did not.
Reviewed By: singhsrb
Differential Revision: D23167515
fbshipit-source-id: add33ec6e4cfd9169e6fef8208490e0aeede38bd
Summary:
The tests weren't windows compatible. Let's get rid of the path
separators.
Reviewed By: quark-zju, singhsrb
Differential Revision: D23170573
fbshipit-source-id: f934691cd2891205442885a12debe3a28d275fc5
Summary:
Subscribers to the commit tailer categories would like to know when Mononoke
received the commit.
Reviewed By: StanislavGlebik
Differential Revision: D23162447
fbshipit-source-id: 747214f1964a643f59c491aa08cdbd5c8fe331c8
Summary:
Allow callers of `resolve_bookmark` to specify whether they'd like the most recent value of
the bookmark, rather than one which may be stale.
Use this in the repo_move_bookmark test to avoid flakiness caused by the test code racing against
the warm bookmark cache.
Reviewed By: StanislavGlebik
Differential Revision: D23151427
fbshipit-source-id: 4b8358be1cf103479ccc23a41b2505776543ee49
Summary:
Extract construction of the hook manager to its own crate, so that we can re-use it.
Eventually the hook manager will become a repo attribute and will be constructed by
the repo attribute factory, but for now it needs its own factory method.
Differential Revision: D23129407
fbshipit-source-id: 302fde4d1ae38c6f61032a32c880018ebf84dee2
Summary:
Convert hook rejections from a tuple to a named struct. This will be used in
the bookmarks_movement public interface.
Reviewed By: krallin
Differential Revision: D23077550
fbshipit-source-id: a35476817660c38b8df879ba603b927a7e39be21
Summary: Some repos are push-redirected repos: pushes go to another repos, but then synced into this repo. Because of this, when we import a repo into a smaller repo that push-redirects to a large repo, we need to make sure we don't break the large repo with the imported code, since merges, pushes, imports etc. are redirected to the large repo. For now, in order to avoid breaking the large repo, we added a simple check that returns error, if the small repo push-redirects to the large one.
Reviewed By: ikostia
Differential Revision: D23158826
fbshipit-source-id: f722790441d641f67293e78c5d1ea5d1102bbb9b
Summary:
This new disallowlist will let us specify config section.key's which
should not be accepted from old rc files. This will let us incrementally disable
loading of those configs from the old files, which will then let us delete them
from the old rc's and eventually delete the old rc's entirely.
This diff also removes hgrc.local and hgrc.od from the list of configs we
verify, since those are not on the list of configs that need to be removed in
this initiative.
Reviewed By: quark-zju
Differential Revision: D23065595
fbshipit-source-id: 5cd742d099efd651174cab5e87bb7cdc4bae8054
Summary:
Previously the backing store was loading configs manually. Now that
system, dynamic, user, and repo config loading are unified, let's go through
that approved path.
Reviewed By: kulshrax
Differential Revision: D22736338
fbshipit-source-id: 232023e660107a096691e9d99bf89c04c218dfbd
Summary:
The last few diffs prepared Rust for loading dynamic and repo configs.
This diff finally changes Python so that we're no longer doing any of this work
in Python.
Reviewed By: quark-zju
Differential Revision: D22712625
fbshipit-source-id: 0b71e81d79d10ea3ce7a1b31f315ada5728af9a9
Summary:
This threads the calls to load_dynamic and load_repo through the Rust
layer up to the Python bindings. This diff does 2 notable things:
1. It adds a reload API for reloading configs in place, versus creating a new
one. This will be used in localrepo.__init__ to construct a new config for the
repo while still maintaining the old pinned values from the copied ui.
2. It threads a repo path and readonly config list from Python down to the Rust
code. This allows load_dynamic and load_repo to operate on the repo path, and
allows the readonly filter to applied to all configs during reloading.
Reviewed By: quark-zju
Differential Revision: D22712623
fbshipit-source-id: a0f372f4971c5feac2f20e89a0fb3fe6d4a65d6f
Summary:
In a future diff we'll enable dynamic and repo config loading purely
from Rust. To do so we need load functions for both cases. A future diff will
call these.
The dynamicconfig loading is based off the Python equivalent in uiconfig.py
Reviewed By: quark-zju
Differential Revision: D22712624
fbshipit-source-id: ff46f6315fb80d4cd9e31d875ac60264563b12f2
Summary:
Previously load_system would skip loading if HGRCPATH was present and
then load_user would actually load the HGRCPATH. In an upcoming diff I add
load_dynamic, which happens after system but before user. The tests for
dynamicconfig depend on HGRCPATH being loaded when load_dynamic runs, so let's
move HGRCPATH loading up to load_system.
Reviewed By: quark-zju
Differential Revision: D22712627
fbshipit-source-id: 91175d9d7f85b9392ffea4af815a4facebbfe7c1
Summary:
In a future diff we'll allow an outside caller to pass an Options down
to configparsers::hg::load() so that filters can be applied during loading. Inside
hg::load() we need to use the options multiple times with different values, so
let's make Options clonable.
Reviewed By: quark-zju
Differential Revision: D22712626
fbshipit-source-id: 975145f38d35afe7d4a6c8e87071b0fb0ae74797
Summary:
A future diff will move all dynamic and repo config loading to be in
configparser. As part of this, let's simplify the repo.rs API to not pass
configs around everywhere.
Reviewed By: quark-zju
Differential Revision: D22712628
fbshipit-source-id: 79f23991aa826ce8b4f7430b45d7702efdc6b982
Summary:
Similar to the Python runbgcommand (extutil.py), this is a Rust utility that runs a
detached background process in a cross platform way.
This will be used in a later diff to run dynamicconfig generation in the
background.
Reviewed By: quark-zju
Differential Revision: D22712629
fbshipit-source-id: a317465bf03c96d977a203678e2bef13ce57cc12
Summary:
As part of moving all hg config loading and generation logic into Rust,
let's move the config generation logic from hgcommands and pyconfigparser to
configparser, unifying them at the same time.
Future diffs will move config loading in as well.
Reviewed By: quark-zju
Differential Revision: D22590208
fbshipit-source-id: d1760c404a6a5c57347df30713c20de55cfdb9a4
Summary:
A future diff will unify all config loading into configparser::hg, but
to do so we need dynamicconfig to live in configparser, so it can load
dynamicconfigs. Let's move everything in.
Reviewed By: quark-zju
Differential Revision: D22587237
fbshipit-source-id: 5613094175b6e1597aa113ee3e6d92ce7ec79f6d
Summary:
We had two spots that loaded system and user configs, one in the
pyconfigparser layer, and one in the pure rust config layer. In an upcoming diff
I'd like to move dynamicconfig loading down into the pure rust layer, so let's
unify these.
Reviewed By: quark-zju
Differential Revision: D22585554
fbshipit-source-id: 0cea7801ae1d5a3a3c12b80ee23b37f9e690e2bc
Summary:
In a future diff we'll increase the size of the rotatelog temporarily
during clones. To do so we need it to be configurable.
Reviewed By: quark-zju
Differential Revision: D23089539
fbshipit-source-id: ebfc3beaf3c0fe5b01b87d97c19455b0a24afa72
Summary:
In a future diff we'll increase the size of the rotatelog temporarily
during clones. To do so we need it to be configurable.
Reviewed By: quark-zju
Differential Revision: D23089541
fbshipit-source-id: 5010e417a83a2611283322f1dbb7023f4286f503
Summary:
from_path is an awkward constructor because it doesn't pass any other
information, like a config object. It also requires that the constructor be very
generic across all the stores. Right now it's only needed for pack files, so
let's move it to it's own trait that is limited to pack files.
This will allow us to make the indexedlog store constructors more versatile in a
later diff. Once we get rid of pack files we can delete the StoreFromPath trait
entirely.
Reviewed By: xavierd
Differential Revision: D23089542
fbshipit-source-id: ea6c50853e5d5390a029002ef5d15c74fe41fe69
Summary:
Similar to `changelog.dageval`, but provides extra functions like `public`,
`draft`, `obsolete`, etc.
Reviewed By: sfilipco
Differential Revision: D23036070
fbshipit-source-id: b985f2b338a3dce11bddf53c00c30e4887762676
Summary:
Make it work with closure and `lambda dag: dag.only(...)`.
This is useful when there is a name conflict with a local variable, like:
only = ...
cl.dageval(lambda: only(...)) # only refers to the local variable.
cl.dageval(lambda dag: dag.only(...)) # only refers to the dag operation.
Besides, drop `func_` so they are Py3 compatible.
Reviewed By: sfilipco
Differential Revision: D23036064
fbshipit-source-id: 8d32a34c51b6ba945cda5be313f9d71464f813b7
Summary: If parent_revs gets an out-of-bound rev, it should fail.
Reviewed By: sfilipco
Differential Revision: D23036071
fbshipit-source-id: 7fae0fd5adf07ac3c933a29d7d06289d8d740c60
Summary:
If the text starts with `\0`, the `\0` should be considered as part of the
uncompressed text instead of a separated header.
Reviewed By: sfilipco
Differential Revision: D22970575
fbshipit-source-id: 49e8a1a1ea42a3c4cf153b70f59fd0558dcfcede
Summary:
The parent handling is unsound when there are revs that are skipped. Fix it by
reasoning about commit hashes for parents.
Reviewed By: sfilipco
Differential Revision: D23036078
fbshipit-source-id: 8f710171471025cd48b3bd8f6ea57c68330eb8b8
Summary:
Up to now, Windows had to have its own version of folly::{readFile, writeFile,
writeFileAtomic} as these only operate on `char *` path, which can only
represent ascii paths on Windows. Since the Windows version is slightly
different from folly, this forced the code to either ifdef _WIN32, or use the
folly version pretending that it would be OK. The Windows version was also
behaving slightly differently from folly. For instance, where folly would
return a boolean to indicate success, on Windows we would throw an exception.
To simplify our code, add type safety and unify both, we can implement our own
wrappers on top of either folly or Windows APIs.
We still have some code that uses folly::readFile but these should only be
dealing with filedescriptors. As a following step, we may want to have our own
File class that wraps a file descriptor/HANDLE so we can completely remove all
uses of folly::readFile.
Reviewed By: wez
Differential Revision: D23037325
fbshipit-source-id: 2b9a026f3ee6220ef55097abe649b23e38d9fe91
Summary:
After enabling strace profiling for the dispatcher, it became very clear that
once files are populated on disk, most of the notification in edenfs come from
listing directories. While looking at the code to find any improvements, I
picked up a couple of low hanging fruits and cleanup that should help (not
measured).
Reviewed By: chadaustin
Differential Revision: D23121434
fbshipit-source-id: 8a8b369f3be6ffb6097e6e12ab0f07af0a6d56b7
Summary:
From a quick experiment, this greatly cuts down on the amount requests to
nonexistent files. For instance, the .hg directory in folders is now only
looked up once and no longer afterwards.
Reviewed By: wez
Differential Revision: D23112343
fbshipit-source-id: 223134ca591054ae9ac2e839033bbd1b714443da
