Summary:
I made the Rust repo able to construct the scmstore objects for Python, sharing the underlying stores. This way we don't have to worry as much about invalidating the Rust repo's stores. We only construct the minimal contentstore (i.e. no edenapi, memcache, etc). We only need the contentstore fallback to support the legacy Python fetching (and write to local stores) - everything else is handled by scmstore.
In particular, test-visibility.t was failing w/ workingcopy.ruststatus=True because the Rust repo's stores got out of sync w/ Python during an "unshelve" triggered rebase. The Rust stores are invalidated on transaction completion, but in this case the "unshelve" transaction hadn't closed yet.
Note that I couldn't remove the scmstore construction logic from pyrevisionstore because the tree stuff is still used by the treemanifestserver extension, and the file stuff is still used by the "shared only" store used to force fetch commits to update linkrev data. This would be good to clean up, but I'm focusing on the status stuff.
Reviewed By: quark-zju
Differential Revision: D46646005
fbshipit-source-id: 99683521a1df2ef07f6d166b670c6e860e629f58
Summary:
With recent changes, the changelog backend info is logged via Rust tracing
directly. It no longer goes through blackbox.
Reviewed By: zzl0
Differential Revision: D47033778
fbshipit-source-id: a8a65002ece718d45e81c61687c473b0fa98b60c
Summary:
This very simple change makes the `-c` behavior equivalent to a `-b / -g`
Otherwise, when running big bisection automatically (with `-c`), having to manually assess how much work was left was annoying.
Before:
```
$ hg bisect --command 'sh script.sh and some params'
changeset e7fa0811edb0: good
changeset 03750880c6b5: good
changeset b53bea5e2fcb: bad
changeset 7874a09ea728: bad
changeset a3d5c6fdf0d3: good
The first good revision is:
commit: a3d5c6fdf0d3
user: test
```
After:
```
$ hg bisect --command 'sh script.sh and some params'
changeset e7fa0811edb0: good
Testing changeset 03750880c6b5 (15 changesets remaining, ~3 tests)
changeset 03750880c6b5: good
Testing changeset b53bea5e2fcb (7 changesets remaining, ~2 tests)
changeset b53bea5e2fcb: bad
Testing changeset 7874a09ea728 (4 changesets remaining, ~2 tests)
changeset 7874a09ea728: bad
Testing changeset a3d5c6fdf0d3 (2 changesets remaining, ~1 tests)
changeset a3d5c6fdf0d3: good
Testing changeset a3d5c6fdf0d3 (0 changesets remaining, ~0 tests)
The first good revision is:
commit: a3d5c6fdf0d3
user: test
```
Reviewed By: xunnanxu, eeggl, drdarshan
Differential Revision: D46991676
fbshipit-source-id: 40ddf0c86e56a5c3edc42d135c3cf40e51efd274
Summary: We needed a way of making EdenFS aware that some mount supports symlinks on Windows while at the same time make hg aware that that same mount / working copy supports symlinks.
Reviewed By: xavierd
Differential Revision: D45442009
fbshipit-source-id: 33640dbe1b6508092b622a977c357fb172c11d4c
Summary:
This makes the tests under `eden/integration` use the buck-built hg except under certain conditions, namely:
- Paths that are too long for batch files to be processed
- Interactive commands that for some reason are still failing
Also, for making some tests work, it changes the way hg commands are invoked from that test framework by escaping them to be more bat compatible
Reviewed By: muirdm
Differential Revision: D46123680
fbshipit-source-id: 690b61e3111b3b07deaa36a677edfa0d363ba987
Summary:
I've been at a loss for how to resolve the Logview Scribe schematization task we got, since all of our logging is performed for us via slog_logview called from Mononoke.
Luckily, it looks like oresk has already solved this problem for slog_logview in D44409191 and we just need to make use of that solution. Even better, it looks like a single update in MononokeMatches will fix the problem both for mega_grepo_sync, and also Mononoke itself.
Reviewed By: korest
Differential Revision: D46990440
fbshipit-source-id: 4c8f3953768072487bd87dd748af3307a9e3848f
Summary: Check if the file is in source commit. Currently, dagcopytrace assume the source file is in the source commit
Reviewed By: sggutier
Differential Revision: D47056956
fbshipit-source-id: db4de51be66d26b01d239c0d6023b154578cddf8
Summary: This lets the Rust status operate with an up-to-date file/tree store when called from Python w/ pending changes.
Reviewed By: sggutier
Differential Revision: D47014121
fbshipit-source-id: cbef3d32fe85f7d3d0e086b6c723d6ce1ba3ee89
Summary:
This way when we click on a dashboard link in a rage, it'll take us to the time
period where the user experienced their issues.
Reviewed By: xavierd
Differential Revision: D46917976
fbshipit-source-id: ef772a9dbcfea5d2944aaf6daf77127cff05c841
Summary:
Provided implementation for EdenMount::chown on NFS (MacOS). Utilized exisiting code to get all referencered Inodes and then loaded and invaliated them.
Ran into some issues compiling Ranges - something to learn more about - where certain iterator traits needed to be defined. In subsquent diff should be able to address (if desired), but for this diff just used a vector instead.
Reviewed By: MichaelCuevas
Differential Revision: D46946414
fbshipit-source-id: b405f16712d6d6315d43d6973e7615aced7e5c7d
Summary:
* On MacOS uid/gid are both defined as uint64_t, specifically `nobody` is 4294967294.
* The thrift contract uses int32_t for uid/gid parameters
* An overflow is likely occurring when passing values above INT_MAX to the service
* This change creates a new `chownV2` that uses int64_t values, "deprecates" the previous method
Reviewed By: MichaelCuevas
Differential Revision: D46946415
fbshipit-source-id: fcff8ab6c46018561b78f23c6460441333284832
Summary: These seem to be failing/flaky on CI. I'm not sure Rust status is responsible for test-amend-metaedit.t, but it seems likely.
Reviewed By: zzl0
Differential Revision: D47020104
fbshipit-source-id: 556ce066de07fa5f92439493dea2ba2fbc5c7040
Summary: The original copytrace was too slow, so it was disabled by default. This diff enables `dagcopytrace` (new fast copytrace) for duplicatecopies() to fix the empty commit issue.
Reviewed By: sggutier
Differential Revision: D46975341
fbshipit-source-id: 11219daee9e400a5ff1d8eda26fe17fbca5d5f12
Summary: Retain and expose scmstore file and tree stores. I'm going to use this to share store instances between Python and Rust.
Reviewed By: sggutier
Differential Revision: D46683379
fbshipit-source-id: ad5204c47eb5d4147d27e9901611037384615bf8
Summary:
This Clap update adds some new checks in `debug_asserts` that triggers new test failures in some existing CLIs (that I know of, we'll see if there's more red signals before landing). This diff fixes those along with the upgrade itself. Failure looks like this:
```
thread 'main' panicked at 'Command : Argument group 'multirepos' conflicts with non-existent 'repo' id', third-party/rust/vendor/clap_builder-4.2.7/src/builder/debug_asserts.rs:317:13
```
More context:
* https://github.com/clap-rs/clap/blob/master/clap_builder/src/builder/debug_asserts.rs#L317
* 185729a7dc
First added in Clap 4.2.5.
Reviewed By: zertosh
Differential Revision: D46934728
fbshipit-source-id: 2d17c2b02fb88af04ff65a0ea35a2be171acce31
Summary: I want to enable this in production to share changelog object between Rust and Python. Let's turn it on for tests, first.
Reviewed By: sggutier
Differential Revision: D46849672
fbshipit-source-id: 8a27c7310253edc767a333b3026a5f1b7dfd1f5f
Summary:
There were three things wrong, in order of discovery:
1. It wasn't testing anything because the "cl" changelog in-hand was empty. Fix by getting cl after importing commits. Didn't investigate further.
2. ancestors.py was broken due to a previous change in D43839498. Reverted that in preceding commit D46961670.
2. Revision numbers were too big for ancestors.py. Set devel.segmented-changelog-rev-compat=true to get smaller numbers.
Reviewed By: zzl0
Differential Revision: D46898400
fbshipit-source-id: 9b2bb25b457d1757851ed05d9a79da185593910d
Summary:
Previously in D43839498 I tried to do add naive copy trace support for eager repos, but that wasn't correct as evidenced by the commit hash changes in some subequent test changes. It also broke the algorithm in ancestors.py, but that code wasn't being invoked at all due to some test issues.
Anyway, revert some parts of D43839498, instead adding a naive, separate commonancestorsheads implementation for eagerfilelog. Seems to be an improvement because the test output reverts to the previous commit hashes.
Reviewed By: quark-zju
Differential Revision: D46961670
fbshipit-source-id: e3e6e4821bbdda18d3c844d00ac742f84544ed46
Summary: Adds a parameter to the lower level APIs methods in SqlCommitGraphStorage to allow querying from master, and adds a method for querying only the changeset ids in a range without the associated changeset edges.
Differential Revision: D46862197
fbshipit-source-id: 57193eefdadd07474402108f452b9b6caf93b1f8
Summary: Each call to disjoint_segments is completely independent, so let's try to run them in parallel.
Reviewed By: markbt
Differential Revision: D46801400
fbshipit-source-id: 979ddaed4ac81d395debbea89b7a2cd84af25a0b
Summary: We need to make sure that the changesets the commit graph segments methods are querying are cache warm.
Reviewed By: markbt
Differential Revision: D46796191
fbshipit-source-id: 2d2d069222e909a6351ab7d596a41db590ceb87b
Summary: Adds a new EdenAPI endpoint commit/graph_segments that returns a segmented representation of ancestors of one set of commits (heads) excluding ancestors of another set ot commits (common).
Reviewed By: markbt
Differential Revision: D46796194
fbshipit-source-id: 45513e97a1b42f3fbdbe14ed94961512011cb0d5
Summary: Implements a method called ancestors_difference_segments that returns a segmented representation of ancestors of one set of changesets (heads) excluding ancestors of another set of changesets (common).
Reviewed By: markbt
Differential Revision: D46796195
fbshipit-source-id: c355cb3fe59829ae26d15b6064cfdf35510825c0
Summary: This fixes compatibility if both the Python LFS extension and remotefilelog.lfs are enabled. In particular, if we find an LFS pointer in the non-LFS indexedlog storage (i.e. inserted by Python LFS extension when remotefilelog.lfs was not enabled), we now upgrade it to the the scmstore pointer store.
Reviewed By: quark-zju
Differential Revision: D46714084
fbshipit-source-id: 4761f1c05017897544b33b0c322d4c3bce81274d
Summary: Tweak things so scmstore is compatible with the python lfs extension. In particular, if remotefilelog lfs support isn't enabled, don't treat lfs pointers specially (i.e. just store them as regular file data). This is what contentstore does, and it is required for compatibility tests that use remotefilelog with the python extension. (This is not used in production.)
Reviewed By: zzl0
Differential Revision: D46674241
fbshipit-source-id: faa5b5b992a7e2cc61bf98ba2919753e9825bb0a
Summary:
Use a different strategy to avoid "undo" on nested runcomand calls.
Previously it would remove the runcommand wrapper, but that effectively disables the undo extension for subsequent commands under debugruntest. Instead, we now have some global state that we unset when a command finishes so we can tell the difference between nested commands in the same process, and serial commands in the same process.
Reviewed By: zzl0
Differential Revision: D46646009
fbshipit-source-id: 47387c412da246e13b2b7c947ca04f3e419f5a87
Summary:
Now we propagate invalid path errors (e.g. invalid utf-8, invalid path components) to Python, and warn about them.
This is a change in behavior in that we don't bail out on invalid paths.
Reviewed By: quark-zju
Differential Revision: D46646008
fbshipit-source-id: af7fabdfcb559b987cd8254e28429256baaf34da
Summary: No functional change here, just giving the variable a more descriptive name
Reviewed By: chadaustin
Differential Revision: D46325246
fbshipit-source-id: daa40b625ccec66224c632309105c814a931b26f
Summary: Tuning particular numbers (like page size) did not show performance improvement when benchmarking, but enabling these two modes did show slight improvement.
Reviewed By: chadaustin
Differential Revision: D46325226
fbshipit-source-id: 6e0cc424f4d29c76be3237178a8a73f03066f67b
Summary:
Serializing and deserialzing the OverlayDir has similar runtime to copying the structure directly but is more memory efficient. For the benchmark I was running on Linux
```
I0509 16:33:45.357141 2489454 BufferedSqliteInodeCatalog.cpp:303] odirSize: 900, serializedSize: 254
```
Reviewed By: mshroyer
Differential Revision: D46325184
fbshipit-source-id: 8d5c063002e4601f81592941ee8791f239c85682
Summary:
D45510982 enabled rust status for debugruntest, but some tests are not
ready yet. This diff disable rust status explicitly for those tests.
Reviewed By: quark-zju
Differential Revision: D46985910
fbshipit-source-id: 92e1c825d420f7897f165dce93297991901ad6db
Summary: Adding a possibility to configure the default blake3 version that could be used if config is not set.
Reviewed By: xavierd
Differential Revision: D46934813
fbshipit-source-id: 6d6032eec1204800f06d5794c5150db7e139a6fa
Summary:
avoid establishing wire proto connections for fully edenapi pulls (segmented changelog `hg pull`)
we can speed up the command by not sending clinettelemetry and hello for this
use case that will significantly reduce wireproto traffic: https://fburl.com/scuba/mononoke_test_perf/hlllh1id
we will aslo save time not establishing wire proto connection (this is always a
separate connection than edenapi protocol)
at the moment this only helps for master segmented changelog pulls but this is
our top used command
Reviewed By: quark-zju
Differential Revision: D46313581
fbshipit-source-id: 275de25ab751f8339afe28ea937d5e7c5eac8984
Summary:
Make the eden dirstate map use a treestate object under the hood instead of a plain Python dictionary. This way Python and Rust can stay in-sync wrt the dirstate.
I'm working on making everything use the Rust status internally (e.g. "hg commit" uses the Rust status). For this to work, pending dirstate changes from Python need to be visible to the Rust status. Non-EdenFS handles this by sharing the same treestate object between Rust and Python. This doesn't work for EdenFS since EdenFS uses its own dirstate format completely separate from the Rust workingcopy.
However, the Rust workingcopy already converts the EdenFS dirstate into an in-memory treestate object for compatibility. In this commit, I pass this treestate object along from Rust to the eden_dirstate_map to act as the underling _map. This way, mutations from Python will be available to the Rust status code.
This isn't as nice as completely migrating to treestate, but it is much easier and is a step in the right direction.
Reviewed By: quark-zju
Differential Revision: D46840436
fbshipit-source-id: 99904622507645001cc5a9f2096da9d83cda98d4
Summary:
On macOS, some of these types are u32. On glibc, usize. Add more
compatible casts.
This fixes the Watchman build on macOS.
Reviewed By: zzl0
Differential Revision: D46959090
fbshipit-source-id: a15635d10bb0ff687d2aab8fb608b6d841c157fd
Summary: Adds a link to a host dashboard to rage output.
Reviewed By: xavierd
Differential Revision: D46917455
fbshipit-source-id: 5acd4679b74ba87bb0e1e232d14cf04fe2a8313b
Summary:
This prevents a dependency cycle when using get_normalized_hostname in
telemetry code.
Reviewed By: xavierd
Differential Revision: D46910713
fbshipit-source-id: 35ebd7505512d418c6558f6f06b055b651a34ab1
Summary:
This flag allows the repo import to progress through all the steps
but bail before actually merging in the repos.
Currently the repo_import tool is so slow that all the pre-merge steps
can take hours. This makes it really hard to control when the actual merge
commit will happen. This flag will allow us to prepare all those steps
ahead-of-time and then resume with just merge in mind.
I don't think it's a proper long-term fix but I found it useful when working
wath whatsapp/biz and would use it again util we properly optimize repo_import
tool.
Differential Revision: D46802952
fbshipit-source-id: 2e8185482c4ba9c04fed20013efcc80d75e80bad
Summary:
D45510982 enabled rust status for debugruntest, but this test is not
ready yet. This diff disable rust status explicitly in the test.
Reviewed By: quark-zju
Differential Revision: D46938285
fbshipit-source-id: b9cb514d8721328a65ac88229aeff8f85dbaf922
Summary: This diff allows the new admin tool to double check if the key is really we wanted to delete, before doing the actual deletion.
Reviewed By: mitrandir77
Differential Revision: D46901306
fbshipit-source-id: 58904c5272d22b696dd22b3c83a9caf33fa3a0b2
