Summary:
This diff adds a new `Walker` struct, which can contain either a `SingleWalker` or a `MultiWalker`. The concrete Walker implementations are hidden and are chosen by the `num_threads` parameter -- single-threaded walker if the thread count is 0, and multi-threaded walker otherwise. This maintains the logic in `pyworkingcopy`.
This change has the downside of adding additional trait bounds (`Send + Sync + 'static`) on the `Matcher` used, even if you only need a single-threaded walker. However, the next diff in this stack would require that anyway in the only place that doesn't already have that bound. Going forward, it seems like future uses of Matcher would also like to dynamically select between single and multi-threaded walkers, and thus would also need those trait bounds anyway.
Reviewed By: DurhamG
Differential Revision: D34251645
fbshipit-source-id: 1273a3f2da59dfe3f9fd165b2e0ab35477c37180
Summary:
Use the `quickcheck_arbitrary_derive` crate to auto-derive the implementation of the `quickcheck::Arbitrary` trait for amenable types under `fbcode/eden`.
An implementation is amenable to auto-derive if the macro-generated code is functionally equivalent with the replaced code.
Reviewed By: yancouto
Differential Revision: D34191759
fbshipit-source-id: 994904cbd1970318a9bfc2462b801aa84eda6861
Summary:
Use the `quickcheck_arbitrary_derive` crate to derive the `Arbitrary` trait for all the types for which the auto-derived impl would match the existing impl.
note: I did not commit the updated snapshots from the `buck test ... -- --env INSTA_UPDATE=1` execution because I still don't know if they are correct or not. :) yelp! :D
Reviewed By: yancouto
Differential Revision: D34161995
fbshipit-source-id: 567c6c43b1fb666abd460c4a7ac62bf095220600
Summary:
checkunknown is a large source of random slowdowns for users, since if
they have a lot of untracked files in their working copy, it could result in
downloading a lot of data, reading the files off disk, and comparing it. This
causes a lot of pain, and in contrast the checkunknown functionality likely
saves users time very rarely (vs just rewriting the files on disk), so let's
remove it.
Reviewed By: quark-zju
Differential Revision: D34113766
fbshipit-source-id: 442b347e5484422c5b27154763bd878b2bb8803e
Summary:
Previously filesystem._compareondisk relied on ctx.cmp(). Which
inspected the size, read the content out of the store, and compared the store
contents with what's on disk.
This is expensive for a few reasons.
1) Fetching LFS file size from the store actually fetches the entire lfs blob,
due to some unfortunate API design. This means we fetch the entire lfs blob,
then look at the lfs pointer to get the size and completely ignore the blob.
Then, if the size matches we later fetch the lfs blob *again* to compare it with
what's on disk.
We can fix the API later. For now let's circumvent ctx.cmp() and go straight to
filelog.cmp() to skip the size check, since we already did a size check earlier
in the status computation.
2) remotefilelog.cmp() requires that we have the store contents, which ends up
requiring an expensive prefetch step.
In reality we can do the cmp by computing the p1+p2+text hash of the contents on
disk and comparing it with the filenode in the manifest. So let's change
remotefilelog.cmp to use this method instead. This does require a history
prefetch (so we have p1 and p2), but that's much cheaper than a content
prefetch.
Reviewed By: quark-zju
Differential Revision: D34093036
fbshipit-source-id: 4a4386e59bca178ae58b10897c5cf68d3aa15dda
Summary: When cloning large repo's, sometimes the tls connection fails or is flaky. Let's retry for that case.
Reviewed By: quark-zju
Differential Revision: D34257302
fbshipit-source-id: 540df4fcad6db5d2d9e11683cd3e91d4093f8681
Summary:
The `--no-write-fetch-head` flag was added in git 2.29. It does not exist in
all git used in production yet.
Reviewed By: DurhamG
Differential Revision: D34214948
fbshipit-source-id: 97ab132205f48c92213cf54b5d79cb8d83f95873
Summary:
See D34095156 (035e8b61a5) for context. `util.system` is easy to misuse.
Rename it to a more scary name and document that `ui.system` is preferred.
Reviewed By: sggutier
Differential Revision: D34095594
fbshipit-source-id: 61a77b552e51de729a579edb34a6470d26681c7b
Summary:
Monkey patching `repo.newreporequirements` won't be supported by Rust repo
initialization. Let's replace the mokey patch with an explicit requirements
update in the clone code path.
Reviewed By: sggutier
Differential Revision: D34254583
fbshipit-source-id: 903fb7925687d50eb9035f3a063ef98cf2551c31
Summary:
LFS batch fetching would just silently eat any server errors it
encountered. Let's route those errors up to scmstore so they get reported
appropriately.
Note, this doesn't apply to standard http connection errors, put does apply to
the server's response to batch request queries. These errors happen if the
server doesn't know about the blob being asked for, etc.
Differential Revision: D34191914
fbshipit-source-id: 432aefb9871ddc91e5e9dc98f64182e8ef4ea471
Summary:
I introduced this "fb" feature to try to fix conditional compilation of fbwhoami into the hostname shed crate, but that wasn't the right way to do it (and it didn't work anyway).
The correct way:
1. The OSS build should use the public_autocargo version of hostname which doesn't reference fbwhoami in Cargo.toml.
2. The OSS build has fbcode_build=false which the hostname source uses to not reference fbwhoami.
Reviewed By: quark-zju
Differential Revision: D34010737
fbshipit-source-id: 3ea3b12aaf5c89fb9af8f34c4a2a3192f4fd93d7
Summary:
Replace the Python drawdag parsing logic with the Rust implementation.
This allows drawdag to parse horizontal graphs, and ranges.
The `o` special case to support `log -G` output is now gone and related
tests are migrated to not use `o`.
Differential Revision: D34163476
fbshipit-source-id: c37e336b60aa3fc06f00dfcccd21ecba11e9e074
Summary:
See https://matklad.github.io/2021/09/04/fast-rust-builds.html
It seems all users of `parse` just passes in `&str`. So let's drop the `impl`.
The `commit` function is less used so it's unchanged.
Differential Revision: D34163477
fbshipit-source-id: 71cb6333a0261f216366d4e1bda8fcec55baef96
Summary:
The Python drawdag implementation allows revset expressions as names.
To replace the Python drawdag with the Rust drawdag, let's support
the special chars used in revset.
Differential Revision: D34163104
fbshipit-source-id: 4f140858cb760d40f80326af337884d1f77c6319
Summary:
We want to exercise "large-ish" commit graph, to check if certain operations
might cause too many network round-trips. It wasn't easy to do that using
drawdag.
This diff adds `:` and `.` special edges for ranges, so
one can create a linear range quickly, like:
A..Z
or:
A001..A999
or vertically:
A999
:
A001
Differential Revision: D34163018
fbshipit-source-id: a043b2b9b3dc73b20e4ee37598b9d182a6d65c29
Summary:
This will be used in the next change to create a range of linear commits
easily, like A..=Z, A01..=A20, etc.
The implementation is inspired by the `String::succ` method in the Ruby
programming language, except for the non-alphanumeric case, because that's
a case not intersted by drawdag, and finding the next valid utf-8 code point
in Rust seems non-trivial.
Differential Revision: D34163017
fbshipit-source-id: 9d1036cabab7a7f33e93a00c3e7c90e40534c994
Summary:
Failing to use the fast pull path is not fatal. We can still use the generic
pull path.
Reviewed By: sggutier
Differential Revision: D34099015
fbshipit-source-id: 45a6e94c4192bda7257e97c42f544c0a4654204a
Summary:
`ui.system` handles more things properly, including suspending progress bars,
and telling chg to use the original terminal so ctty is available.
Differential Revision: D34095156
fbshipit-source-id: 20909fbfe7925a18993f0db05a01fbe9ea8c6afc
Summary:
Status has two real phases 1) Gets the best effort results from
Watchman, and 2) Looking at the files on disk that may or may not have changes.
Today we show one "Querying watchman" progress bar for this entire operation,
even though most of the time is spent in phase 2, after watchman has returned.
Even worse, the current progress bar is just a spinner and gives no sense of
progress, even though we know how much work happens in phase 2.
This diff splits the two progress bars.
Reviewed By: quark-zju
Differential Revision: D34093035
fbshipit-source-id: 44fb1a7c705e0a8b9e1bf8451931aa345876962f
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary: Change read_from to use an iterative approach. We hit some stack overflows in production we think were caused by this recursion.
Reviewed By: quark-zju
Differential Revision: D34188307
fbshipit-source-id: d62d559096519b7c8dbce89f07a93fc8cf671595
Summary:
`status` should not show changes in submodules. That's already working with the
vanilla `status` implementation but the fsmonitor one needs some updates.
Watching nested roots seems to cause timeouts. Therefore fsmonitor is disabled
for submodule repos.
Differential Revision: D34025075
fbshipit-source-id: 45f97fa62f634e44f66dec75acad055e819b964c
Summary:
This fixes annotate on git repos, which do not have filelogs but can use
pathhistory for file traversal.
Renames are not yet supported.
Differential Revision: D34017426
fbshipit-source-id: 817b0d635a31e0f052c82defb4615c87909668f5
Summary:
The file history traversal is based on hg filelog, which is considered as a
techdebt and is not present in git. Move it to a separate function so it's
cleaner to implement other kinds of file history traversal.
Differential Revision: D34017424
fbshipit-source-id: 50b9e9a818250dbb56d89460533e8859a3abc36b
Summary:
The main algorithm of "annotate" is relatively abstracted.
Move it to a separate module.
Type annotations are added to demostrate that it is abstract.
Differential Revision: D34017428
fbshipit-source-id: cf9817de807ede05621ab40432b73f146463a7e7
Summary: Expose the `NameSet::to_parents` API so it can be used in Python.
Differential Revision: D34017427
fbshipit-source-id: 9a602031eb9f709f43d89a0cd23448d9b8285021
Summary:
A nameset is a "flat" set without graph (parents) informatin.
In certain cases (ex. graph log, annotate) the parents information
is needed. Add an API for that.
This might potentially replace some logic in Python `graphmod.py`.
Differential Revision: D34017425
fbshipit-source-id: 38d0bccb5c455cd9ec2d60113c18207a90645de9
Summary:
When doing a clone, the nativecheckout action plan included a remove
action for every file not in the sparse checkout. This is because it didn't
account for the file not existing in the old manifest and just blindly queued
the remove.
On Linux this is pretty fast, but on Windows this is extremely slow to do for
every path. It's probably slow because of NTFS, but also because we do some path
auditing logic for Windows and Mac.
This fixes it to not do remove for a file that wasn't in the old manifest.
Reviewed By: sggutier
Differential Revision: D34188271
fbshipit-source-id: 714b2dcf98cfa9ebec287f00d7cd5397126f693e
Summary:
The old tracing info was noisy and hard to follow. The new one is much
more concise and provides useful insight into what is being fetched, at what
stage, and what errors happen at each stage.
Reviewed By: quark-zju
Differential Revision: D34159138
fbshipit-source-id: e1de87de9f9f64deb7e33ef76e6441bb6721f9b9
Summary:
These produce a ton of hard to read trace output, with enter/exit's
everywhere. Let's delete them. In the next diff we'll add more human readable
explanations of what's going on in scmstore.
Reviewed By: quark-zju
Differential Revision: D34159139
fbshipit-source-id: 48d1175edd20d3de952f12dc8fcf7a6661469d64
Summary:
scmstore was writing lfs pointers via the contentstore. This meant that
when we disabled the contentstore fallback, this path failed. This path is used
during unbundle (like in 'jf get') when the bundle contains an lfs pointer.
Reviewed By: quark-zju
Differential Revision: D34127065
fbshipit-source-id: d50245b78dd87e6c71d8775123fe954084289e2b
Summary:
When initializing a new repo in the old new repo initialization, directories would be created recursively if the target path did not exist. This diff fixes the new repo initialization not doing that.
The unit test for directory creation was also reworked.
Reviewed By: quark-zju
Differential Revision: D34093210
fbshipit-source-id: 3aa0aade9bd3034448615678a005966a7da6ef5c
Summary: Storing errors instead of their string representation should be much more useful for showing errors. This diff does that for `InitError`.
Reviewed By: quark-zju
Differential Revision: D34161428
fbshipit-source-id: 4f77428ef379db6b42b492bb5424a8fdd04b954a
Summary:
Add some counters to track lfs transient/fatal error response codes:
- "lfs.transient_error.{method}.{code}" counts error codes we got but ended up
succeeding after retries
- "lfs.fatal_error.{method}.{code}" counts errors we got and then ended up not
succeeding
- "lfs.success.{method}" counts successful queries
Reviewed By: quark-zju
Differential Revision: D33962157
fbshipit-source-id: 901625eb72cac2eba57f4b2424a778f4f8ac1779
Summary:
This feature is only used for e-discovery, and we don't need to build a
new e-discovery package from eden/scm anymore. Let's remove this feature since
the monkeypatch interferes with updating the hg status paths.
Reviewed By: quark-zju
Differential Revision: D34124434
fbshipit-source-id: b41382bc1a3cfeda82e1dd2af511244ad16a9fb8
Summary:
Convert type comments to type annotations.
Produced by running:
```
python -m libcst.tool codemod convert_type_comments.ConvertTypeComment eden
```
from a parent directory
Reviewed By: shannonzhu
Differential Revision: D34110680
fbshipit-source-id: 23081a8867b93cd349271075428176af514efa2b
Summary:
Convert type comments to type annotations in eden/scm/edenscm/hgext
Produced by running:
```
python -m libcst.tool codemod convert_type_comments.ConvertTypeComment eden/scm/edenscm/hgext
```
from a parent directory.
Reviewed By: grievejia
Differential Revision: D34110682
fbshipit-source-id: acf8d32290411ca8673904e4f80ddd846ebceef0
Summary:
Convert type comments to type annotations in eden/scm/edenscm/mercurial
Produced by running:
```
python -m libcst.tool codemod convert_type_comments.ConvertTypeComment eden/scm/edenscm/mercurial
```
from a parent directory.
Reviewed By: grievejia
Differential Revision: D34110681
fbshipit-source-id: 3f21aa39de9384cdf7d5b09a232d9605eadc1481
Summary: Support the case of a snapshot modifying a symlink by removing the symlink destination before trying to create the symlink.
Reviewed By: yancouto
Differential Revision: D34097684
fbshipit-source-id: 48d4a05f61a69e686e5bd530a7ed150326a32765
Summary: Unlike the python repo lock files, the rust lock files are not deleted when the lock is released. That means if the locks are created as root, they will break future commands run as other users. Try to mitigate that eventuality by chmoding the lock files to 0o666 on unix.
Reviewed By: quark-zju
Differential Revision: D34016719
fbshipit-source-id: 2bd1ec9d5da0e27a12eb95da5c2a4cb3f6286137
Summary:
The published `terminfo` crate has a WTFPL license which is forbidden from being used by Meta.
{F687594009}
However the WTFPL allows forking and relicensing, so I've gone ahead and relicensed the same code as dual MIT OR Apache 2.0 for fbsource to use. We can patch the properly licensed code in place of all transitive dependencies on the `terminfo` crate, as they have the same public Rust API.
The relevant changes: 9b4e1c6aae
Reviewed By: metajack
Differential Revision: D33190944
fbshipit-source-id: 0a9002b8799b5ceeaca3c20a211f15beec5a407f
Summary: Sometimes we can get errors, debugging info, or advices in headers returned from services and proxies. We know some headers that we can parse, but we don't have a comprehensive list. To improve debugging we might as well just print all headers of a response on failure. As of now we don't handle that in one place (like D33918785), because technically HTTP 5xx is a valid response from the POV of http library. It's the caller's logic that need to decide if it's a failure or not. In the future we could perhaps have such debugging logic in a trait like `ResponseExt` that could have `fn debug_failed_req` to be called by multiple callers if they decide a particular request was indeed an unrecoverable failure.
Reviewed By: quark-zju
Differential Revision: D34035084
fbshipit-source-id: 8247462ed9875b6530f215fb161c311325ea83fe
