Summary:
This diff fixes some incorrect logic in the physicalfilesystem pendingchanges Rust code that leads to duplicate entries in the "lookup" list (files that need to have their contents checked to know whether they're dirty). We need to fix this, otherwise we'll do 2x the number of (potentially expensive) checks, and we'll need to deduplicate later anyway.
I believe this bug was originally introduced in D25697099 (02606da6c5) -- the change to this region of the code changes the behavior. I believe this was just an oversight when the code was refactored, since the change doesn't have anything to do with the purpose of the diff.
Reviewed By: DurhamG
Differential Revision: D34403900
fbshipit-source-id: 4c38e6a0ba46dfefc8f7ba327b943cb7ccee7fc5
Summary: Also check for new /files2 endpoint.
Reviewed By: quark-zju
Differential Revision: D34423002
fbshipit-source-id: 409435633e8f776729d25ebf05342020f423b828
Summary: Tweak "grep lfs" pattern to not match an "lfs" that shows up in tempdir name.
Reviewed By: quark-zju
Differential Revision: D34399625
fbshipit-source-id: 1122d0be4e11f32ed9cf33c046ccf34a99f5cdc3
Summary:
Right now, when we create a snapshot on top of a commit that's not on commit cloud, we output things like `queue 0 files for upload` even when we are on plain mode, in which we'd like to only output the hash of the snapshot.
So I made it so the ui is quiet when on plain mode, which makes it not print that.
Reviewed By: quark-zju
Differential Revision: D34378357
fbshipit-source-id: 69b17147b8e72c6d72176f6069d729689130fdc9
Summary:
This makes it so HashMaps are serialized in a consistent way in tests (on this crate), which is good because it prevents flakiness.
It also does so at no cost to non-test execution.
Reviewed By: markbt
Differential Revision: D34339601
fbshipit-source-id: e9ec1cb74308f83fb1a346c68f646dfa12021098
Summary: This fixes a bug when snapshots with symlinks don't restore well when doing the operation from outside of the repo root. This happened because we were careful about relative paths on the normal file create codepath, but not on the snapshot codepath.
Differential Revision: D34302718
fbshipit-source-id: a8b06fb8a52e53c99735840fdbf5f3c74512a0c9
Summary: This simplifies the code a bit and makes future changes easier.
Differential Revision: D33865779
fbshipit-source-id: 52c284c9f61a24210465104f6d96a9f0bcb8b513
Summary:
The `calculate_definitely_unassigned_vertexes` function is an important
optimization to avoid excessive remote lookups when adding new commits
to the lazy graph. It contains the a "discovery" algorithm that uses a
bisect-ish way to find out what vertexes are known or unknown remotely.
That "discovery" algorithm can also be useful in other places, including:
- Figure out what vertexes need to be "reassigned" from the non-master
group to the master group.
- With proper Python bindings, replace the Python `setdiscovery.py`
entirely.
This diff makes the above possible by moving the algorithm to a separate
function.
Differential Revision: D33865775
fbshipit-source-id: 3b54de8929dd69ea80bdd2d8667b1791fe60ed50
Summary:
Make it a bit clearer that the callsite needs extra normalization if they
depend on the order.
Differential Revision: D33831518
fbshipit-source-id: 382a62d31e58137844ae954d38af6574d5bd5fb1
Summary:
The indexedlog iddag is written to disk. In the future "remove_non_master" might
be replaced by "strip". Add a test to ensure "remove_non_master" entry is still
supported.
Differential Revision: D33827176
fbshipit-source-id: a2dc76ae2a99badf8186ddb4721653c032378c5e
Summary:
The method does locking and flushing. In a future diff we want the
"flush_cached_idmap" behavior but we alredy have obtained a lock.
Let's split the logic so we can use the "flushed_cached_idmap"
logic without its locking and reloading overhead.
Differential Revision: D33822899
fbshipit-source-id: 672d165b2136f0aa7bd616836080b080135bc3d8
Summary: Don't trigger a background cloud sync for empty transactions. We had a recent issue where errant "hg log" commands kept trying to autopull an invalid commit hash, each time triggering a cloud sync.
Reviewed By: DurhamG
Differential Revision: D34289175
fbshipit-source-id: 35a6c0ef2ef30e0a384b554a8984340cbe6c8c2c
Summary:
This can be used for stripping set and its descendants.
See D33779403 (a8c32280cc) for high level goals.
Differential Revision: D33821384
fbshipit-source-id: 5dd6a550a166bb3a4eee847894afa4aaa2baa55c
Summary:
Creating bookmarks is a frequent use-case. Let's make drawdag support it so it's
possible to reduce the number of `hg bookmark` commands.
Reviewed By: DurhamG
Differential Revision: D34259550
fbshipit-source-id: b0cc750aaa3bd35f72ae0b92e15ec7b9e700dec6
Summary:
This is the recommended practise because the `node` name is often used as local
variables.
Reviewed By: DurhamG
Differential Revision: D34259549
fbshipit-source-id: d0101eb5da813ede865ac0d6685938c9a39e85ae
Summary:
Previously there are 3 places doing lock and transaction for drawdag.
Unify them to just 1 place.
Reviewed By: DurhamG
Differential Revision: D34259552
fbshipit-source-id: bb5773e63537dcd2c8fa07081178ea3622b73f5f
Summary:
We don't use obsmarkers. Some comments are outdated or confusing.
Update them without changing the drawdag feature.
Reviewed By: DurhamG
Differential Revision: D34259551
fbshipit-source-id: da92801dccae825d32eb71dd8da70de2e0cb4ef6
Summary:
This makes use of the new Rust repo initialization added in previous commits into the `clone` command in Python by overriding the repo initialization done by creating a new `localrepo` object (and setting `create` to `True`). There are few things of note here:
- New repo initialization was only done by the `init` and `clone` commands
- This was the only way new repos were initialized when calling the `clone` command.
- Repo initialization via the `init` command is already being done by the Rust repo initialization and this override should not affect it.
Reviewed By: DurhamG
Differential Revision: D34094589
fbshipit-source-id: 079d641b8001b51854838a4089ba4ef37942fc69
Summary:
This diff passes the `workingcopy.rustwalkerthreads` config value along to the physical filesystem `pending_changes` function, allowing it to use a multi-threaded walker when `rustwalkerthreads > 0`.
We also need to use `extract_matcher` to get a Rust Matcher from a Python one for use in the multi-threaded walker. Using `UnsafePythonMatcher` causes the walker to deadlock (presumably due to the GIL -- see P480409358 for an example backtrace of a hung thread). This was the last usage of `UnsafePythonMatcher`, so I removed it as well.
Reviewed By: DurhamG
Differential Revision: D34251644
fbshipit-source-id: 1f11f2038ebcc6e098432132b9e5726b25c18911
Summary:
This diff adds a new `Walker` struct, which can contain either a `SingleWalker` or a `MultiWalker`. The concrete Walker implementations are hidden and are chosen by the `num_threads` parameter -- single-threaded walker if the thread count is 0, and multi-threaded walker otherwise. This maintains the logic in `pyworkingcopy`.
This change has the downside of adding additional trait bounds (`Send + Sync + 'static`) on the `Matcher` used, even if you only need a single-threaded walker. However, the next diff in this stack would require that anyway in the only place that doesn't already have that bound. Going forward, it seems like future uses of Matcher would also like to dynamically select between single and multi-threaded walkers, and thus would also need those trait bounds anyway.
Reviewed By: DurhamG
Differential Revision: D34251645
fbshipit-source-id: 1273a3f2da59dfe3f9fd165b2e0ab35477c37180
Summary:
Use the `quickcheck_arbitrary_derive` crate to auto-derive the implementation of the `quickcheck::Arbitrary` trait for amenable types under `fbcode/eden`.
An implementation is amenable to auto-derive if the macro-generated code is functionally equivalent with the replaced code.
Reviewed By: yancouto
Differential Revision: D34191759
fbshipit-source-id: 994904cbd1970318a9bfc2462b801aa84eda6861
Summary:
Use the `quickcheck_arbitrary_derive` crate to derive the `Arbitrary` trait for all the types for which the auto-derived impl would match the existing impl.
note: I did not commit the updated snapshots from the `buck test ... -- --env INSTA_UPDATE=1` execution because I still don't know if they are correct or not. :) yelp! :D
Reviewed By: yancouto
Differential Revision: D34161995
fbshipit-source-id: 567c6c43b1fb666abd460c4a7ac62bf095220600
Summary:
checkunknown is a large source of random slowdowns for users, since if
they have a lot of untracked files in their working copy, it could result in
downloading a lot of data, reading the files off disk, and comparing it. This
causes a lot of pain, and in contrast the checkunknown functionality likely
saves users time very rarely (vs just rewriting the files on disk), so let's
remove it.
Reviewed By: quark-zju
Differential Revision: D34113766
fbshipit-source-id: 442b347e5484422c5b27154763bd878b2bb8803e
Summary:
Previously filesystem._compareondisk relied on ctx.cmp(). Which
inspected the size, read the content out of the store, and compared the store
contents with what's on disk.
This is expensive for a few reasons.
1) Fetching LFS file size from the store actually fetches the entire lfs blob,
due to some unfortunate API design. This means we fetch the entire lfs blob,
then look at the lfs pointer to get the size and completely ignore the blob.
Then, if the size matches we later fetch the lfs blob *again* to compare it with
what's on disk.
We can fix the API later. For now let's circumvent ctx.cmp() and go straight to
filelog.cmp() to skip the size check, since we already did a size check earlier
in the status computation.
2) remotefilelog.cmp() requires that we have the store contents, which ends up
requiring an expensive prefetch step.
In reality we can do the cmp by computing the p1+p2+text hash of the contents on
disk and comparing it with the filenode in the manifest. So let's change
remotefilelog.cmp to use this method instead. This does require a history
prefetch (so we have p1 and p2), but that's much cheaper than a content
prefetch.
Reviewed By: quark-zju
Differential Revision: D34093036
fbshipit-source-id: 4a4386e59bca178ae58b10897c5cf68d3aa15dda
Summary: When cloning large repo's, sometimes the tls connection fails or is flaky. Let's retry for that case.
Reviewed By: quark-zju
Differential Revision: D34257302
fbshipit-source-id: 540df4fcad6db5d2d9e11683cd3e91d4093f8681
Summary:
The `--no-write-fetch-head` flag was added in git 2.29. It does not exist in
all git used in production yet.
Reviewed By: DurhamG
Differential Revision: D34214948
fbshipit-source-id: 97ab132205f48c92213cf54b5d79cb8d83f95873
Summary:
See D34095156 (035e8b61a5) for context. `util.system` is easy to misuse.
Rename it to a more scary name and document that `ui.system` is preferred.
Reviewed By: sggutier
Differential Revision: D34095594
fbshipit-source-id: 61a77b552e51de729a579edb34a6470d26681c7b
Summary:
Monkey patching `repo.newreporequirements` won't be supported by Rust repo
initialization. Let's replace the mokey patch with an explicit requirements
update in the clone code path.
Reviewed By: sggutier
Differential Revision: D34254583
fbshipit-source-id: 903fb7925687d50eb9035f3a063ef98cf2551c31
Summary:
LFS batch fetching would just silently eat any server errors it
encountered. Let's route those errors up to scmstore so they get reported
appropriately.
Note, this doesn't apply to standard http connection errors, put does apply to
the server's response to batch request queries. These errors happen if the
server doesn't know about the blob being asked for, etc.
Differential Revision: D34191914
fbshipit-source-id: 432aefb9871ddc91e5e9dc98f64182e8ef4ea471
Summary:
I introduced this "fb" feature to try to fix conditional compilation of fbwhoami into the hostname shed crate, but that wasn't the right way to do it (and it didn't work anyway).
The correct way:
1. The OSS build should use the public_autocargo version of hostname which doesn't reference fbwhoami in Cargo.toml.
2. The OSS build has fbcode_build=false which the hostname source uses to not reference fbwhoami.
Reviewed By: quark-zju
Differential Revision: D34010737
fbshipit-source-id: 3ea3b12aaf5c89fb9af8f34c4a2a3192f4fd93d7
Summary:
Replace the Python drawdag parsing logic with the Rust implementation.
This allows drawdag to parse horizontal graphs, and ranges.
The `o` special case to support `log -G` output is now gone and related
tests are migrated to not use `o`.
Differential Revision: D34163476
fbshipit-source-id: c37e336b60aa3fc06f00dfcccd21ecba11e9e074
Summary:
See https://matklad.github.io/2021/09/04/fast-rust-builds.html
It seems all users of `parse` just passes in `&str`. So let's drop the `impl`.
The `commit` function is less used so it's unchanged.
Differential Revision: D34163477
fbshipit-source-id: 71cb6333a0261f216366d4e1bda8fcec55baef96
Summary:
The Python drawdag implementation allows revset expressions as names.
To replace the Python drawdag with the Rust drawdag, let's support
the special chars used in revset.
Differential Revision: D34163104
fbshipit-source-id: 4f140858cb760d40f80326af337884d1f77c6319
Summary:
We want to exercise "large-ish" commit graph, to check if certain operations
might cause too many network round-trips. It wasn't easy to do that using
drawdag.
This diff adds `:` and `.` special edges for ranges, so
one can create a linear range quickly, like:
A..Z
or:
A001..A999
or vertically:
A999
:
A001
Differential Revision: D34163018
fbshipit-source-id: a043b2b9b3dc73b20e4ee37598b9d182a6d65c29
Summary:
This will be used in the next change to create a range of linear commits
easily, like A..=Z, A01..=A20, etc.
The implementation is inspired by the `String::succ` method in the Ruby
programming language, except for the non-alphanumeric case, because that's
a case not intersted by drawdag, and finding the next valid utf-8 code point
in Rust seems non-trivial.
Differential Revision: D34163017
fbshipit-source-id: 9d1036cabab7a7f33e93a00c3e7c90e40534c994
Summary:
Failing to use the fast pull path is not fatal. We can still use the generic
pull path.
Reviewed By: sggutier
Differential Revision: D34099015
fbshipit-source-id: 45a6e94c4192bda7257e97c42f544c0a4654204a
Summary:
`ui.system` handles more things properly, including suspending progress bars,
and telling chg to use the original terminal so ctty is available.
Differential Revision: D34095156
fbshipit-source-id: 20909fbfe7925a18993f0db05a01fbe9ea8c6afc
Summary:
Status has two real phases 1) Gets the best effort results from
Watchman, and 2) Looking at the files on disk that may or may not have changes.
Today we show one "Querying watchman" progress bar for this entire operation,
even though most of the time is spent in phase 2, after watchman has returned.
Even worse, the current progress bar is just a spinner and gives no sense of
progress, even though we know how much work happens in phase 2.
This diff splits the two progress bars.
Reviewed By: quark-zju
Differential Revision: D34093035
fbshipit-source-id: 44fb1a7c705e0a8b9e1bf8451931aa345876962f
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary: Change read_from to use an iterative approach. We hit some stack overflows in production we think were caused by this recursion.
Reviewed By: quark-zju
Differential Revision: D34188307
fbshipit-source-id: d62d559096519b7c8dbce89f07a93fc8cf671595
Summary:
`status` should not show changes in submodules. That's already working with the
vanilla `status` implementation but the fsmonitor one needs some updates.
Watching nested roots seems to cause timeouts. Therefore fsmonitor is disabled
for submodule repos.
Differential Revision: D34025075
fbshipit-source-id: 45f97fa62f634e44f66dec75acad055e819b964c
Summary:
This fixes annotate on git repos, which do not have filelogs but can use
pathhistory for file traversal.
Renames are not yet supported.
Differential Revision: D34017426
fbshipit-source-id: 817b0d635a31e0f052c82defb4615c87909668f5
Summary:
The file history traversal is based on hg filelog, which is considered as a
techdebt and is not present in git. Move it to a separate function so it's
cleaner to implement other kinds of file history traversal.
Differential Revision: D34017424
fbshipit-source-id: 50b9e9a818250dbb56d89460533e8859a3abc36b
Summary:
The main algorithm of "annotate" is relatively abstracted.
Move it to a separate module.
Type annotations are added to demostrate that it is abstract.
Differential Revision: D34017428
fbshipit-source-id: cf9817de807ede05621ab40432b73f146463a7e7
