Summary: Update debug messages and remove the Rust specific parts of test since they are now covered naturally.
Reviewed By: quark-zju
Differential Revision: D46832706
fbshipit-source-id: fcdfd65ae5309523a22407b150c19b8112647b8e
Summary: Had to tweak things to work w/ modern repo.
Reviewed By: quark-zju
Differential Revision: D46683381
fbshipit-source-id: a26623a4a89bea2643b1109690c12ff46231c29a
Summary:
I had to fix a minor issue w/ in the non-watchman case where we were case normalizing an on-disk file name w/ a dirstate entry incorrectly. When the dirstate entry (e.g. "foo") is marked for removal, it shouldn't coallesce with a different case file (e.g. "FOO"). Otherwise, it is impossible to properly show the status:
```
$ hg st
R foo
? FOO
```
Reviewed By: quark-zju
Differential Revision: D46683382
fbshipit-source-id: 83f93209ab10098efeb65fba3ba7379107ded50d
Summary:
This is dealing w/ a legacy repo feature that allows specifying branch names as URL fragments.
I don't think it's worth trying to reconcile this w/ remotefilelog.
Reviewed By: quark-zju
Differential Revision: D46683380
fbshipit-source-id: 10249e734bba1624c5e9b9dd07c6c1c5d189654f
Summary:
I made the Rust repo able to construct the scmstore objects for Python, sharing the underlying stores. This way we don't have to worry as much about invalidating the Rust repo's stores. We only construct the minimal contentstore (i.e. no edenapi, memcache, etc). We only need the contentstore fallback to support the legacy Python fetching (and write to local stores) - everything else is handled by scmstore.
In particular, test-visibility.t was failing w/ workingcopy.ruststatus=True because the Rust repo's stores got out of sync w/ Python during an "unshelve" triggered rebase. The Rust stores are invalidated on transaction completion, but in this case the "unshelve" transaction hadn't closed yet.
Note that I couldn't remove the scmstore construction logic from pyrevisionstore because the tree stuff is still used by the treemanifestserver extension, and the file stuff is still used by the "shared only" store used to force fetch commits to update linkrev data. This would be good to clean up, but I'm focusing on the status stuff.
Reviewed By: quark-zju
Differential Revision: D46646005
fbshipit-source-id: 99683521a1df2ef07f6d166b670c6e860e629f58
Summary:
With recent changes, the changelog backend info is logged via Rust tracing
directly. It no longer goes through blackbox.
Reviewed By: zzl0
Differential Revision: D47033778
fbshipit-source-id: a8a65002ece718d45e81c61687c473b0fa98b60c
Summary:
This very simple change makes the `-c` behavior equivalent to a `-b / -g`
Otherwise, when running big bisection automatically (with `-c`), having to manually assess how much work was left was annoying.
Before:
```
$ hg bisect --command 'sh script.sh and some params'
changeset e7fa0811edb0: good
changeset 03750880c6b5: good
changeset b53bea5e2fcb: bad
changeset 7874a09ea728: bad
changeset a3d5c6fdf0d3: good
The first good revision is:
commit: a3d5c6fdf0d3
user: test
```
After:
```
$ hg bisect --command 'sh script.sh and some params'
changeset e7fa0811edb0: good
Testing changeset 03750880c6b5 (15 changesets remaining, ~3 tests)
changeset 03750880c6b5: good
Testing changeset b53bea5e2fcb (7 changesets remaining, ~2 tests)
changeset b53bea5e2fcb: bad
Testing changeset 7874a09ea728 (4 changesets remaining, ~2 tests)
changeset 7874a09ea728: bad
Testing changeset a3d5c6fdf0d3 (2 changesets remaining, ~1 tests)
changeset a3d5c6fdf0d3: good
Testing changeset a3d5c6fdf0d3 (0 changesets remaining, ~0 tests)
The first good revision is:
commit: a3d5c6fdf0d3
user: test
```
Reviewed By: xunnanxu, eeggl, drdarshan
Differential Revision: D46991676
fbshipit-source-id: 40ddf0c86e56a5c3edc42d135c3cf40e51efd274
Summary: We needed a way of making EdenFS aware that some mount supports symlinks on Windows while at the same time make hg aware that that same mount / working copy supports symlinks.
Reviewed By: xavierd
Differential Revision: D45442009
fbshipit-source-id: 33640dbe1b6508092b622a977c357fb172c11d4c
Summary:
This makes the tests under `eden/integration` use the buck-built hg except under certain conditions, namely:
- Paths that are too long for batch files to be processed
- Interactive commands that for some reason are still failing
Also, for making some tests work, it changes the way hg commands are invoked from that test framework by escaping them to be more bat compatible
Reviewed By: muirdm
Differential Revision: D46123680
fbshipit-source-id: 690b61e3111b3b07deaa36a677edfa0d363ba987
Summary:
I've been at a loss for how to resolve the Logview Scribe schematization task we got, since all of our logging is performed for us via slog_logview called from Mononoke.
Luckily, it looks like oresk has already solved this problem for slog_logview in D44409191 and we just need to make use of that solution. Even better, it looks like a single update in MononokeMatches will fix the problem both for mega_grepo_sync, and also Mononoke itself.
Reviewed By: korest
Differential Revision: D46990440
fbshipit-source-id: 4c8f3953768072487bd87dd748af3307a9e3848f
Summary: Check if the file is in source commit. Currently, dagcopytrace assume the source file is in the source commit
Reviewed By: sggutier
Differential Revision: D47056956
fbshipit-source-id: db4de51be66d26b01d239c0d6023b154578cddf8
Summary: This lets the Rust status operate with an up-to-date file/tree store when called from Python w/ pending changes.
Reviewed By: sggutier
Differential Revision: D47014121
fbshipit-source-id: cbef3d32fe85f7d3d0e086b6c723d6ce1ba3ee89
Summary:
This way when we click on a dashboard link in a rage, it'll take us to the time
period where the user experienced their issues.
Reviewed By: xavierd
Differential Revision: D46917976
fbshipit-source-id: ef772a9dbcfea5d2944aaf6daf77127cff05c841
Summary:
Provided implementation for EdenMount::chown on NFS (MacOS). Utilized exisiting code to get all referencered Inodes and then loaded and invaliated them.
Ran into some issues compiling Ranges - something to learn more about - where certain iterator traits needed to be defined. In subsquent diff should be able to address (if desired), but for this diff just used a vector instead.
Reviewed By: MichaelCuevas
Differential Revision: D46946414
fbshipit-source-id: b405f16712d6d6315d43d6973e7615aced7e5c7d
Summary:
* On MacOS uid/gid are both defined as uint64_t, specifically `nobody` is 4294967294.
* The thrift contract uses int32_t for uid/gid parameters
* An overflow is likely occurring when passing values above INT_MAX to the service
* This change creates a new `chownV2` that uses int64_t values, "deprecates" the previous method
Reviewed By: MichaelCuevas
Differential Revision: D46946415
fbshipit-source-id: fcff8ab6c46018561b78f23c6460441333284832
Summary: These seem to be failing/flaky on CI. I'm not sure Rust status is responsible for test-amend-metaedit.t, but it seems likely.
Reviewed By: zzl0
Differential Revision: D47020104
fbshipit-source-id: 556ce066de07fa5f92439493dea2ba2fbc5c7040
Summary: The original copytrace was too slow, so it was disabled by default. This diff enables `dagcopytrace` (new fast copytrace) for duplicatecopies() to fix the empty commit issue.
Reviewed By: sggutier
Differential Revision: D46975341
fbshipit-source-id: 11219daee9e400a5ff1d8eda26fe17fbca5d5f12
Summary: Retain and expose scmstore file and tree stores. I'm going to use this to share store instances between Python and Rust.
Reviewed By: sggutier
Differential Revision: D46683379
fbshipit-source-id: ad5204c47eb5d4147d27e9901611037384615bf8
Summary:
This Clap update adds some new checks in `debug_asserts` that triggers new test failures in some existing CLIs (that I know of, we'll see if there's more red signals before landing). This diff fixes those along with the upgrade itself. Failure looks like this:
```
thread 'main' panicked at 'Command : Argument group 'multirepos' conflicts with non-existent 'repo' id', third-party/rust/vendor/clap_builder-4.2.7/src/builder/debug_asserts.rs:317:13
```
More context:
* https://github.com/clap-rs/clap/blob/master/clap_builder/src/builder/debug_asserts.rs#L317
* 185729a7dc
First added in Clap 4.2.5.
Reviewed By: zertosh
Differential Revision: D46934728
fbshipit-source-id: 2d17c2b02fb88af04ff65a0ea35a2be171acce31
Summary: I want to enable this in production to share changelog object between Rust and Python. Let's turn it on for tests, first.
Reviewed By: sggutier
Differential Revision: D46849672
fbshipit-source-id: 8a27c7310253edc767a333b3026a5f1b7dfd1f5f
Summary:
There were three things wrong, in order of discovery:
1. It wasn't testing anything because the "cl" changelog in-hand was empty. Fix by getting cl after importing commits. Didn't investigate further.
2. ancestors.py was broken due to a previous change in D43839498. Reverted that in preceding commit D46961670.
2. Revision numbers were too big for ancestors.py. Set devel.segmented-changelog-rev-compat=true to get smaller numbers.
Reviewed By: zzl0
Differential Revision: D46898400
fbshipit-source-id: 9b2bb25b457d1757851ed05d9a79da185593910d
Summary:
Previously in D43839498 I tried to do add naive copy trace support for eager repos, but that wasn't correct as evidenced by the commit hash changes in some subequent test changes. It also broke the algorithm in ancestors.py, but that code wasn't being invoked at all due to some test issues.
Anyway, revert some parts of D43839498, instead adding a naive, separate commonancestorsheads implementation for eagerfilelog. Seems to be an improvement because the test output reverts to the previous commit hashes.
Reviewed By: quark-zju
Differential Revision: D46961670
fbshipit-source-id: e3e6e4821bbdda18d3c844d00ac742f84544ed46
Summary: Adds a parameter to the lower level APIs methods in SqlCommitGraphStorage to allow querying from master, and adds a method for querying only the changeset ids in a range without the associated changeset edges.
Differential Revision: D46862197
fbshipit-source-id: 57193eefdadd07474402108f452b9b6caf93b1f8
Summary: Each call to disjoint_segments is completely independent, so let's try to run them in parallel.
Reviewed By: markbt
Differential Revision: D46801400
fbshipit-source-id: 979ddaed4ac81d395debbea89b7a2cd84af25a0b
Summary: We need to make sure that the changesets the commit graph segments methods are querying are cache warm.
Reviewed By: markbt
Differential Revision: D46796191
fbshipit-source-id: 2d2d069222e909a6351ab7d596a41db590ceb87b
Summary: Adds a new EdenAPI endpoint commit/graph_segments that returns a segmented representation of ancestors of one set of commits (heads) excluding ancestors of another set ot commits (common).
Reviewed By: markbt
Differential Revision: D46796194
fbshipit-source-id: 45513e97a1b42f3fbdbe14ed94961512011cb0d5
Summary: Implements a method called ancestors_difference_segments that returns a segmented representation of ancestors of one set of changesets (heads) excluding ancestors of another set of changesets (common).
Reviewed By: markbt
Differential Revision: D46796195
fbshipit-source-id: c355cb3fe59829ae26d15b6064cfdf35510825c0
Summary: This fixes compatibility if both the Python LFS extension and remotefilelog.lfs are enabled. In particular, if we find an LFS pointer in the non-LFS indexedlog storage (i.e. inserted by Python LFS extension when remotefilelog.lfs was not enabled), we now upgrade it to the the scmstore pointer store.
Reviewed By: quark-zju
Differential Revision: D46714084
fbshipit-source-id: 4761f1c05017897544b33b0c322d4c3bce81274d
Summary: Tweak things so scmstore is compatible with the python lfs extension. In particular, if remotefilelog lfs support isn't enabled, don't treat lfs pointers specially (i.e. just store them as regular file data). This is what contentstore does, and it is required for compatibility tests that use remotefilelog with the python extension. (This is not used in production.)
Reviewed By: zzl0
Differential Revision: D46674241
fbshipit-source-id: faa5b5b992a7e2cc61bf98ba2919753e9825bb0a
Summary:
Use a different strategy to avoid "undo" on nested runcomand calls.
Previously it would remove the runcommand wrapper, but that effectively disables the undo extension for subsequent commands under debugruntest. Instead, we now have some global state that we unset when a command finishes so we can tell the difference between nested commands in the same process, and serial commands in the same process.
Reviewed By: zzl0
Differential Revision: D46646009
fbshipit-source-id: 47387c412da246e13b2b7c947ca04f3e419f5a87
Summary:
Now we propagate invalid path errors (e.g. invalid utf-8, invalid path components) to Python, and warn about them.
This is a change in behavior in that we don't bail out on invalid paths.
Reviewed By: quark-zju
Differential Revision: D46646008
fbshipit-source-id: af7fabdfcb559b987cd8254e28429256baaf34da
Summary: No functional change here, just giving the variable a more descriptive name
Reviewed By: chadaustin
Differential Revision: D46325246
fbshipit-source-id: daa40b625ccec66224c632309105c814a931b26f
Summary: Tuning particular numbers (like page size) did not show performance improvement when benchmarking, but enabling these two modes did show slight improvement.
Reviewed By: chadaustin
Differential Revision: D46325226
fbshipit-source-id: 6e0cc424f4d29c76be3237178a8a73f03066f67b
Summary:
Serializing and deserialzing the OverlayDir has similar runtime to copying the structure directly but is more memory efficient. For the benchmark I was running on Linux
```
I0509 16:33:45.357141 2489454 BufferedSqliteInodeCatalog.cpp:303] odirSize: 900, serializedSize: 254
```
Reviewed By: mshroyer
Differential Revision: D46325184
fbshipit-source-id: 8d5c063002e4601f81592941ee8791f239c85682
Summary:
D45510982 enabled rust status for debugruntest, but some tests are not
ready yet. This diff disable rust status explicitly for those tests.
Reviewed By: quark-zju
Differential Revision: D46985910
fbshipit-source-id: 92e1c825d420f7897f165dce93297991901ad6db
Summary: Adding a possibility to configure the default blake3 version that could be used if config is not set.
Reviewed By: xavierd
Differential Revision: D46934813
fbshipit-source-id: 6d6032eec1204800f06d5794c5150db7e139a6fa
Summary:
avoid establishing wire proto connections for fully edenapi pulls (segmented changelog `hg pull`)
we can speed up the command by not sending clinettelemetry and hello for this
use case that will significantly reduce wireproto traffic: https://fburl.com/scuba/mononoke_test_perf/hlllh1id
we will aslo save time not establishing wire proto connection (this is always a
separate connection than edenapi protocol)
at the moment this only helps for master segmented changelog pulls but this is
our top used command
Reviewed By: quark-zju
Differential Revision: D46313581
fbshipit-source-id: 275de25ab751f8339afe28ea937d5e7c5eac8984
Summary:
Make the eden dirstate map use a treestate object under the hood instead of a plain Python dictionary. This way Python and Rust can stay in-sync wrt the dirstate.
I'm working on making everything use the Rust status internally (e.g. "hg commit" uses the Rust status). For this to work, pending dirstate changes from Python need to be visible to the Rust status. Non-EdenFS handles this by sharing the same treestate object between Rust and Python. This doesn't work for EdenFS since EdenFS uses its own dirstate format completely separate from the Rust workingcopy.
However, the Rust workingcopy already converts the EdenFS dirstate into an in-memory treestate object for compatibility. In this commit, I pass this treestate object along from Rust to the eden_dirstate_map to act as the underling _map. This way, mutations from Python will be available to the Rust status code.
This isn't as nice as completely migrating to treestate, but it is much easier and is a step in the right direction.
Reviewed By: quark-zju
Differential Revision: D46840436
fbshipit-source-id: 99904622507645001cc5a9f2096da9d83cda98d4
Summary:
On macOS, some of these types are u32. On glibc, usize. Add more
compatible casts.
This fixes the Watchman build on macOS.
Reviewed By: zzl0
Differential Revision: D46959090
fbshipit-source-id: a15635d10bb0ff687d2aab8fb608b6d841c157fd
Summary: Adds a link to a host dashboard to rage output.
Reviewed By: xavierd
Differential Revision: D46917455
fbshipit-source-id: 5acd4679b74ba87bb0e1e232d14cf04fe2a8313b
Summary:
This prevents a dependency cycle when using get_normalized_hostname in
telemetry code.
Reviewed By: xavierd
Differential Revision: D46910713
fbshipit-source-id: 35ebd7505512d418c6558f6f06b055b651a34ab1
Summary:
This flag allows the repo import to progress through all the steps
but bail before actually merging in the repos.
Currently the repo_import tool is so slow that all the pre-merge steps
can take hours. This makes it really hard to control when the actual merge
commit will happen. This flag will allow us to prepare all those steps
ahead-of-time and then resume with just merge in mind.
I don't think it's a proper long-term fix but I found it useful when working
wath whatsapp/biz and would use it again util we properly optimize repo_import
tool.
Differential Revision: D46802952
fbshipit-source-id: 2e8185482c4ba9c04fed20013efcc80d75e80bad
Summary:
D45510982 enabled rust status for debugruntest, but this test is not
ready yet. This diff disable rust status explicitly in the test.
Reviewed By: quark-zju
Differential Revision: D46938285
fbshipit-source-id: b9cb514d8721328a65ac88229aeff8f85dbaf922
Summary: This diff allows the new admin tool to double check if the key is really we wanted to delete, before doing the actual deletion.
Reviewed By: mitrandir77
Differential Revision: D46901306
fbshipit-source-id: 58904c5272d22b696dd22b3c83a9caf33fa3a0b2
Summary:
Similarly named method in scmquery pushes commits onto some branch.
Let's make the description very clear.
Reviewed By: malmond77
Differential Revision: D46916234
fbshipit-source-id: 6137f42df97be80fff2775638f766f05c1113488
Summary:
Add "connection pool" logic that emulates a pool using a directory.
This can be useful on Windows where `fork()` is unavailable.
Reviewed By: zzl0
Differential Revision: D46885307
fbshipit-source-id: 26c85f6836fc7a9437ccaef0aaf4a7f0a6e8561f
Summary:
Provide APIs to obtain the `NodeIpc` instance instead of raw sockets.
This makes it possible to use `NodeIpc` features like sending the
file descriptors, node channel, etc.
Reviewed By: muirdm
Differential Revision: D46812533
fbshipit-source-id: 7b3d51a6027c9dbacffcf74476f9b8a4a3440313
Summary:
Add a crate to make it easier to use UDS (unix-domain-socket) communication.
For now, it only contains the low-level bind, and connect functions.
Reviewed By: muirdm
Differential Revision: D46812534
fbshipit-source-id: efbf31dbb1286ae64051b570c20ca6f41c736afa
Summary:
The `FileDescriptor` type will close the underlying fd on drop. In our
use-cases `r` and `w` refer to the same fd so we only need to close it
once. Note the reason we use separate `Mutex`es for r and w is that we
want `recv()` and `send()` running from different threads to not block
by each other.
Reviewed By: zzl0
Differential Revision: D46885308
fbshipit-source-id: b2a5644251ac4773581a995ae8c3d955b0b36861
Summary: The crate is no longer used.
Reviewed By: muirdm
Differential Revision: D46812535
fbshipit-source-id: 659b73a563bf14cc17cb1b09e9a57422d21a6377
Summary: `recv_stdio` seems a better place for the logic.
Reviewed By: muirdm
Differential Revision: D46811171
fbshipit-source-id: ddd91489610fcf6a148897bbd4f35b1e404cbd0b
Summary:
This makes it easier to use. In addition, the "singleton" IPC initially decided
by NODE_CHANNEL_FD is handled internally so the callsite is less error-prone,
since it does not need to care about maintaining the NODE_CHANNEL_FD env var,
or the initialization order of the singleton.
Reviewed By: muirdm
Differential Revision: D46811166
fbshipit-source-id: a2a8a7a36986422a8d7c967d6fb5b820bedce0c9
Summary:
Remove the side effect when constructing the NodeIpc. It does not seem
necessary. This allows us to construct NodeIpc without checking the underlying
fd until first send/recv.
Reviewed By: muirdm
Differential Revision: D46811162
fbshipit-source-id: 4fc7941a1e9d6b9a9c47daa699de4283bfcec5c3
Summary:
This allows us to update it dynamically, without going through the
`NODE_CHANNEL_FD` environment variable.
Reviewed By: muirdm
Differential Revision: D46811169
fbshipit-source-id: bf52b5597fcf69291da81675d2917dc0dc010904
Summary: This makes it more flexible so we can replace the singleton if we want.
Reviewed By: muirdm
Differential Revision: D46811160
fbshipit-source-id: 902b7ff05a4fb3f8a7d7218afd3d07e0f3a89a9f
Summary:
Make it possible to send fds on POSIX. This is similar to what chg
(hgclient.c:attachio and osutil.c:recvfds) does. Note the `sendfd`
crate seems to have a solid implementation but its public API is
on the `std::os::unix::net::UnixStream` type, which is incompatible
with our struct that uses the `FileDescriptor`.
Example:
% cargo run --example spawn_sendfd
Parent: spawning child
Parent: sending hello
Parent: sending stdio and a.txt file descriptors
Parent: waiting for child to exit
Child: started with IPC handle 4
Child: got message Some("hello")
Child: got sendfd payload SendFdPayload { raw_fds: [3, 5, 6, 7] }
Child: writing "something\n" to fd 3
something
Child: writing "something\n" to fd 5
something
Child: writing "something\n" to fd 6
something
Child: writing "something\n" to fd 7
% cat a.txt
something
Reviewed By: muirdm
Differential Revision: D46811163
fbshipit-source-id: 37686327f02a7fa0b0e8768d025094a7fcb0e2b5
Summary:
This avoids over reading which might drop special messages from sendmsg(),
on POSIX systems.
The behavior disables itself if we need libuv compatibility on POSIX.
Reviewed By: muirdm
Differential Revision: D46811164
fbshipit-source-id: 3aed4bb4d63ad10ecc6a7bf76007cc7263c1c580
Summary:
This avoids over reading which might drop special messages (like file descriptors)
from sendmsg(). Buffered reader might turn those special messages into regular
messages and will lose the file descriptors.
Reviewed By: muirdm
Differential Revision: D46811158
fbshipit-source-id: 2294c5c2b132c25c21950f26e9f0d44094743fe0
Summary:
We're going to use winapi to implement sending and receiving file descriptors
on Windows.
Reviewed By: muirdm
Differential Revision: D46811161
fbshipit-source-id: dca85722da9f96e15f1ec8bb4676058487c51872
Summary: They will be used by upcoming changes.
Reviewed By: muirdm
Differential Revision: D46811168
fbshipit-source-id: fdd60a2aa96b979dd2dc5bb2d016189435bbf695
Summary: This makes it a bit easier to use in upcoming changes.
Reviewed By: zzl0
Differential Revision: D46811157
fbshipit-source-id: 333e17415d4a67fcb58bec61153b15e4cad03d23
Summary:
Exposes the ability to construct `NodeIpc` from a Windows HANDLE, not a
msvcrt/ucrt's fd.
Reviewed By: zzl0
Differential Revision: D46811167
fbshipit-source-id: 94f48dc0d059bca109c1792dceb0b733e4737adf
Summary:
In the next change I'm adding "raw ..." concept. Let's rename `raw_fd` to
`libc_fd` to make it obvious that the raw something is different from the raw
"HANDLE" on Windows.
Reviewed By: zzl0
Differential Revision: D46811165
fbshipit-source-id: e0fe48633fd8d928ce7d7ae0652f041a90aec922
Summary:
While most PrjFS operations only communicate about the filesystem object they're modifying via the [`PRJ_CALLBACK_DATA->FilePathName`](https://learn.microsoft.com/en-us/windows/win32/api/projectedfslib/ns-projectedfslib-prj_callback_data) member, notifications via the [`PRJ_NOTIFICATION_CB`](https://learn.microsoft.com/en-us/windows/win32/api/projectedfslib/nc-projectedfslib-prj_notification_cb) have a `destinationFileName` argument as well.
This is documented as:
> If notification is PRJ_NOTIFICATION_PRE_RENAME or PRJ_NOTIFICATION_PRE_SET_HARDLINK, this points to a null-terminated Unicode string specifying the path, relative to the virtualization root, of the target of the rename or set-hardlink operation.
While the individual STRACE `NotificationArgRenderer` functions handled the extra path, `PrjfsLiveRequest::formatTraceEventString` does not.
This forwards the string to that object, allowing it to be rendered if needed, so that it shows up in the `eden trace fs` output.
Reviewed By: xavierd
Differential Revision: D46809287
fbshipit-source-id: 781db1d911a975902d0b27d9ba4f553cc3b85299
Summary: We were forbidding \n but not \r. A couple Python spots validate against "\r" and "\n" in filenames (scmutil.checkfilename and manifest._checkforbidden), so let's do it in Rust as well.
Reviewed By: zzl0
Differential Revision: D46646011
fbshipit-source-id: 41b5cbad057e2b966e3c00011d75cd6cd6e804fc
Summary:
This test needed significant changes to use remotefilelog (unrelated to Rust status).
In particular:
- "hg cat" doesn't report "no such file" for non-existing files. Previously that came from py manifest, but Rust manifest walk doesn't support.
- Kill tampered bundle check. I tried for a while to get this working, but everything fails really early with the Rust manifest since bad paths are immediately rejected by the RepoPath type.
- Various update-over-symlink tests now work since the Rust vfs will remove symlinks in order to write the new file (thus avoiding the "xxx traverses symbolic link yyy" audit error).
Also this test revealed the need to process directories in the Rust status to catch a file being replaced by a directory. I changed the (Watchman) code path to no longer skip directories returned from watchman.
Reviewed By: zzl0
Differential Revision: D46646010
fbshipit-source-id: a2f8a92e85a97611212d917b2a6118f0dd1f5eed
Summary:
The x2p auth advice header is handled/displayed by the network doctor in practice now.
I'm making tests support the Rust status, and opting to remove this one instead of update it.
Reviewed By: zzl0
Differential Revision: D46646012
fbshipit-source-id: 72bd074b0e8d76ecbc4206920f6c8867d6d794cd
Summary: I had to tweak the case collision auditor to exclude untracked files. The Rust status (using Watchman) seem so to add untracked files to the dirstate earlier than Python, which was causing extra "possible case-folding" warnings.
Reviewed By: quark-zju
Differential Revision: D46832704
fbshipit-source-id: fa0e30df757804804ca437df94a4fb4b56dcbeda
Summary:
Some Python commands depend on the "bad" matcher callback getting called by dirstate.status(). Tweak things so this happens when we use the Rust status as well. This also adds support for the "clean" option when using the Rust status since clean files and "bad" are somewhat coupled.
I moved invalid file type detection from filesystem.py into dirstate.py so it covers Rust status as well.
Reviewed By: quark-zju
Differential Revision: D46646006
fbshipit-source-id: 28278af15e02e0f794aefc7bde850ce58b319ac6
Summary: It aleady passes for these, so turn it on.
Reviewed By: quark-zju
Differential Revision: D45510980
fbshipit-source-id: f45a2dbe11186411adc259863b88c6ff5ee1914e
Summary: It's annoying to have to remember when this setting is and isn't enabled by default.
Reviewed By: quark-zju
Differential Revision: D45510982
fbshipit-source-id: 7d54c0c95d7edf61f358692b1a7c487e5889b5fa
Summary:
add option for consistent routing of single tree requests and single blob requests
Meta:
Background:
We are having a lot of traffic for single tree blob or single file blob:
https://fburl.com/scuba/mononoke_test_perf/0eainxd3https://fburl.com/scuba/mononoke_test_perf/0ye4eduq
Let's route them consistently!
Reviewed By: quark-zju
Differential Revision: D46860178
fbshipit-source-id: 50298c583d82585b3345ec4c4ca6b7ab4ab57ab1
Summary:
The casefolding pushrebase check is blocking the sync of www commits causing
problems. Let's exempt www/ dir from it.
We'll also have to modify commit hooks - but that's a separate thing.
Reviewed By: markbt
Differential Revision: D46860559
fbshipit-source-id: 87db959e0d025c0c1fc5c6cfecbdcf96af9e0f81
Summary:
This diff does the following refactoring:
* Remove redundent key checking when parsing blobstore key, and repo_id from it
* Replace all the `writeln!` with `println!`
* Print errors using `eprintln!`
* Remove misuse of verbose
Differential Revision: D46857681
fbshipit-source-id: 1b065756eb91d8b4b67422ef49ded2783660992f
Summary: This diffs allows the new admin tool to print out how long it takes to process each file during bulk blobstore key unlinking.
Differential Revision: D46838088
fbshipit-source-id: 92fa57edcce5b171a1d4f6be6d7a57571fdb618b
Summary:
This diff:
* Adds a new HashMap to store the blobstres, regarding a given repo_id, to avoid duplicated computation
* Removes some debugging output
Differential Revision: D46802649
fbshipit-source-id: 8ddf0243b192b5684af2d4a430cdb8873a78246d
Summary:
This diff allows the new admin tool to actually delete the content from the blobstore.
When the key is not present, there is nothing happening.
Differential Revision: D46802651
fbshipit-source-id: ac1d9aabcb4fd17455263c29b1348f5d1619868b
Summary:
[plan_deletion] Add subcommand to redact relevant blobs
To make it easy to use in the context of plan deletion, we take the provisional deletion plan issued from the `propose-deletion-plan` subcommand in and we output the redaction id to a file.
In the real world, one would use this redaction id to make a diff to the mononoke config in configerator and restart the mononoke servers to make sure that the redaction actually takes effect before moving on to the next step.
Reviewed By: markbt
Differential Revision: D46768075
fbshipit-source-id: 368dafd485a02d677f3a74c6ede59a80cbe931d4
Summary:
The current interface relies on the assumption that redacted content only relates to file contents.
`list`, which is the way to read what was redacted discovers the blobs for filecontents by walking the fsnodes manifests.
For a feature I'm working on, I need to be able to redact arbitrary blobs, so I need a way to identify what was redacted
without relying on some derived data blobs being available.
Reviewed By: markbt
Differential Revision: D46806525
fbshipit-source-id: 78b5470d4dd741538e3e85353c0b1634f3b83b1c
Summary: Instead of taking in the full context of a `MononokeApp`, only take the two blobstores that we need.
Reviewed By: markbt
Differential Revision: D46768087
fbshipit-source-id: 773f0f31e12c58ee0cc8341e638fcbcedde8bab9
Summary:
The crux of the redaction process is this `create_key_list` function which appends each key to the redaction blobstores and provides instruction to generate the correct redaction config.
I will need to re-use this mechanism, so extract it to an accessible crate.
Reviewed By: markbt
Differential Revision: D46763714
fbshipit-source-id: 8f06a12c6348bc49d0ca4fe65155818a216c5b88
Summary:
The blackbox was reporting a mode mismatch. The cause was the filesysytem walk yielding a mode with permission bits set (e.g. 0666), but os.lstat within dirstate.normal() yields a mode without permission bits set (e.g. 0000). Fix by only reporting a mode change when the executable bit changes (which seems like the intention of the code).
I don't think this is a recent change in behavior, so I'm not going to look further into it. Hopefully the Python filesystem code will go away soon with the Rust status.
Reviewed By: zzl0
Differential Revision: D46848722
fbshipit-source-id: cdd73137ae891bc8038d7d7ddcd09d8c95170051
Summary:
This diff allows the new admin tool to extract the blobstore key from a given key.
For example, `flat/repo2122.blame.fileunode.blake2.2e58d897760aa5927c7ca1b0755992c9109be6048a9dc1deab1c86a4619f5839` will give `repo2122.blame.fileunode.blake2.2e58d897760aa5927c7ca1b0755992c9109be6048a9dc1deab1c86a4619f5839`.
Differential Revision: D46802650
fbshipit-source-id: bec63353b1ca50eeeab73cd84b5e3d8fb967c694
Summary:
This diff adds
* a new function to extract repo id from a given key
* a new function to construct all the blobstores given a repo id
Differential Revision: D46763054
fbshipit-source-id: 64bb1ab4fd8bcae8ab414dd03e78f0a7c2dd5639
Summary: Some of the usages of the commit graph (e.g. in pushrebase) query very recent commits that might not have been fully replicated in xdb yet. This diff switches to using fetch_edges_required and fetch_many_edges_required everywhere except in exists, changeset_parents and changeset_generation, to avoid any errors caused by replication lag. This shouldn't cause any performance degredation as we always use the normal read connection first and only query master in the rare case that a commit is missing from replicas.
Reviewed By: mitrandir77
Differential Revision: D46842804
fbshipit-source-id: 3822d883b079447d0af90c9169cd3db7600f9b25
Summary: The SQL implementation of fetch_edges_required first tries fetching the changeset edges using the normal read connection, then tries to fetch any missing changesets using the master connection. This can sometimes be the desired behavior when working with very recent commits so let's expose it through changeset_generation_required and changeset_parents_required.
Reviewed By: markbt
Differential Revision: D46796193
fbshipit-source-id: 38b3d79dd88ca4b8b98f36705ca1a215379a87b3
Summary: Added the missing blobstore fetch encoding except RedactionKeyList as that require creating a separate config and already has its own newadmin command.
Differential Revision: D46491769
fbshipit-source-id: 82208ac9dcb1acb5370ff7904c285e843be4f18f
Summary:
This makes commits created by megarepo tools roundtrippable through bonsais (so far they were not).
D27852341 shows that it's not a problem for actual backup job but.. it's a
problem for example for mirror_hg_commits binary so let's make thos commits
nice and tidy.
Reviewed By: mzr
Differential Revision: D46489608
fbshipit-source-id: d15ba6f6622f51189a1ba9e76efd68faf1bf2b71
Summary:
Verification is used to verify if given megarepo mapping configuration can be cleanly applied to given pair of small repo and large repo commits. It's used when we want to change current config and we want to check if according to the new config the mapping is still sound.
The previous *fast path* verification worked by limiting the amount of visited entries to `O(small repo)`
The new fast path verification doesn't walk every file in the repository, instead it leverages FSNodes to compare hashes of entire directories. This was if the repository verifies OK the verification is very fast. The amount of entries visited is `O(differences to report + number of config entries)`. When the configuration is simple and the repository is intact (usual case) it's almost instant. In case of more complex configurations (like ovrsource one) it's still **much** faster than current one.
WARNING: The implementation is a bit hacky due to the path mover functions being orignally designed with moving file paths not, directory paths. The hack is mostly contained to wrap_mover_result functiton.
Differential Revision: D45864549
fbshipit-source-id: 2fad0ed29a5718e655fa4a69b28306ca3db31dda
Summary:
I want to test for some bug in megarepotooling and I'd like to use that
feature.
Differential Revision: D46440647
fbshipit-source-id: 1f4bb0d937f6e681a02f9843a7241fde4ae9b774
Summary:
Some of our commits have those fields set so let's allow setting them on test
commits.
Differential Revision: D46440648
fbshipit-source-id: 393d9fcb616ddee5481f239264ab87bf5c8315e9
Summary:
fbcode is migrating to LLVM-15 for safer and more up-to-date code and new compiler features. All contbuilds in your directory have passed our build test with LLVM-15, and your directory does not host any packages. This diff will migrate it to LLVM-15.
If you approve of this diff, please use the "Accept & Ship" button. If you have a reason for why it should not build with LLVM 15, please make a comment and send it back to author. Otherwise we will land this on Thursday 06/15/2023.
See the [FAQ post](https://fb.workplace.com/groups/llvm15platform010/posts/749154386769776/)! Please also direct any questions to [this group](https://fb.workplace.com/groups/llvm15platform010).
- If you approve of this diff, please use the "Accept & Ship" button :-)
Build directives:
Reviewed By: meyering, zzl0
Differential Revision: D46662743
fbshipit-source-id: 1af43eea7bb103d29f3b409634f1b7a60ae84c88
Summary:
We need that for current catchup to avoid confusing clients with backdated
commits.
Reviewed By: markbt
Differential Revision: D46743487
fbshipit-source-id: a71ee30eada63dab7c64cbe43a4495662fe04371
Summary: Walker previously specified a default for the deprecated `cachelib-only-blobstore` argument. Switch to `cache-mode`, allowing us to specify local caching for all types, not just the blobstore.
Differential Revision: D45089769
fbshipit-source-id: 978ac545590935be5a825669db9df58f0281c41a
Summary: To avoid showing a bunch of "?" files, we need to ignore things within submodules. For watchman, we do similar to Python and add submodules to the ignored dirs list. For the manual walker, we similarly skip directories in the ignore list. Note that the walker already skipped the submodules due to the presence of the "dot dir" (e.g. ".sl") in the submodules, but I piped through the ignored dirs skipping to be consistent.
Reviewed By: quark-zju
Differential Revision: D46547238
fbshipit-source-id: b001dc02a3c73f69d74fe36615a812f9ac983944
Summary: Python status for Git has submodule aware logic we don't have in Rust, so we need to defer to Python. However, Python will still be able to use the internal Rust status API.
Reviewed By: zzl0
Differential Revision: D46547239
fbshipit-source-id: 2caf1436819219cea007307f602627fd1a38ef51
Summary: It wasn't used from Rust (only MultiWalker(threads=8) was used), and Rust walker is not enabled from Python. It's annoying to have to make changes twice, so let's kill the non-parallel walker.
Reviewed By: zzl0
Differential Revision: D46547240
fbshipit-source-id: 4a57222ec705059acbc6865f8e1a9701ae584ccc
Summary: This wasn't enabled in production, and the parallel walker doesn't currently work from Python due to matcher GIL deadlock. Anyway, I'm working to make Python use the Rust status directly, so this intermediate optimization of using the Rust walker in the Python status won't buy us anything.
Reviewed By: zzl0
Differential Revision: D46646007
fbshipit-source-id: 645d88adb85700062c3edf2483fb5a5ffb76cca2
Summary: These make it more convenient to conditionalize logic based on storage format.
Reviewed By: zzl0
Differential Revision: D46547242
fbshipit-source-id: ef833c954090f5cb2b1c2c6e616d5d67a442886d
Summary: I'm going to use this to skip submodules in the Rust status implemenation.
Reviewed By: zzl0
Differential Revision: D46547241
fbshipit-source-id: e5cf5dc5ee77aa3e8927ab3cf5922f3e29a25ae5
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stdout)`.
This codemod removes all remaining instances of atty
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`. `arc autocargo` was also ran
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: dbxfb
Differential Revision: D46736264
fbshipit-source-id: a686b96b1fa0aa4389f65487ed426f226c86e8e9
Summary: www has lots of empty commits that are being synced to all small repos. Until we fix that problem let's refrain from backsyncing any empty commits. I'm making it a tunable for easy on-off
Reviewed By: liubov-dmitrieva
Differential Revision: D46727657
fbshipit-source-id: 5f204cc8231fec3e6a4eaf25b6575c0d637d67a3
Summary:
[plan_deletion] Implement `commit-deletion-plan-to-deletion-log-db` subcommand
This is the last necessary step before enacting the deletion.
As the outcome of this step, the deletion_log db is populated and we can start taking the necessary steps to safely proceed with the actual deletion of changesets and blobs.
We still need to document and potentially write tools to help with the whole process, including blobs redaction.
Differential Revision: D46724273
fbshipit-source-id: bc60209a28e2ee8a50ac0371f78c0b3be234781b
Summary: This diff moves all the bulb unlinking functions into a struct.
Reviewed By: mitrandir77
Differential Revision: D46759318
fbshipit-source-id: 143171706485ec6a56152bdfea6202221d88fe03
Summary: This diff makes the `get_blobstores` function to be public, so that we can use it to get blobstores. These blobstores are going to be used to unlink keys in a bulk.
Reviewed By: mitrandir77
Differential Revision: D46729000
fbshipit-source-id: 11d7ddc5f90527ebdb79141f61f1671997108ac5
Summary:
This diff allows the new admin tool to do the following for the new bulk unlking keys subcommand:
* Read the key list from the files
* Print all the keys extracted when running in dry-run mode
* Report the progress
* Fix the argument to take a boolean value
Reviewed By: mitrandir77
Differential Revision: D46725312
fbshipit-source-id: c039b89e00a9ffbbbee41e20617f45543ef147ce
Summary: This diff adds a new subcommand for our new admin tool, so that we can delete keys bulkly. The input is the directory that contains a list of files, each file contains a list of *Manifold* keys to remove.
Reviewed By: mitrandir77
Differential Revision: D46689957
fbshipit-source-id: 077b6a4c76872ef01bc7f0eab7096abeb87d7047
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stdout)`.
This codemod replaces all usages of `atty::is(Stdout)` with `stdout().is_terminal()`, importing the necessary paths.
It used this script (generated with assistance from Metamate):
```
#!/bin/bash
files=$(fbgs -sl "atty::is(atty::Stream::Stdout)" | arc linttool debugfilterpaths --take RUSTFMT)
for file in $files; do
sed -i 's/atty::is(atty::Stream::Stdout)/stdout().is_terminal()/g' $file
done
for file in $files; do
sed -i '1i use std::io::{stdout, IsTerminal};' $file
done
```
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`.
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: danielocfb
Differential Revision: D46601191
fbshipit-source-id: 3f2269c518e36241ab0e995b87976323d7da9cc6
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stderr)`.
This codemod replaces all usages of `atty::is(Stderr)` with `stderr().is_terminal()`, importing the necessary paths.
It used this script (generated with assistance from Metamate):
```
#!/bin/bash
files=$(fbgs -sl "atty::is(atty::Stream::Stderr)" | arc linttool debugfilterpaths --take RUSTFMT)
for file in $files; do
sed -i 's/atty::is(atty::Stream::Stderr)/stderr().is_terminal()/g' $file
done
for file in $files; do
sed -i '1i use std::io::{stderr, IsTerminal};' $file
done
```
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`.
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: Imxset21
Differential Revision: D46601192
fbshipit-source-id: 18ef53a9fa25cbd733ca81f16b2c13ce8c5ddb98
Summary: `gitimport` is typically used for mirror repos which by definition are locked. Creating or moving bookmarks on locked repos is disallowed by default. This behaviour can be overriden by passing the bypass flag. This diff adds argument support for bypassing the read-only check.
Differential Revision: D46722005
fbshipit-source-id: 9defcbc159a55d0e7ff197d8d732e3d5a208a998
Summary: Use the EDEN_TRY macro to reduce error-handling line noise.
Reviewed By: genevievehelsel
Differential Revision: D46668011
fbshipit-source-id: bb54ff710928b3a5c394b6e79474c25ef4775895
Summary: Adds a macro to simplify bailing out of Try-returning functions, sort of like with Rust's question mark operator.
Reviewed By: chadaustin
Differential Revision: D46567326
fbshipit-source-id: f13eec678a101e423a99ba469a20d6de5513d73c
Summary: optimize the check against skip revset by reusing the unskipped result.
Reviewed By: muirdm
Differential Revision: D46687070
fbshipit-source-id: 01e5be4499daaa9e24437ddb16e2cc04f0436211
Summary:
Currently, the 'skip' option is too slow when users want to use it to bisect on
a file or folder like below:
sl bisect --skip "!( file('path:foo') & file('path:bar') )"
Since it tries to eval the skip revset upfront, which takes a long time for
current `file` implementation. This diff make it lazy evaluated.
Reviewed By: muirdm
Differential Revision: D46643574
fbshipit-source-id: c063fbea2bc0703772e07d5167adc8fbf989a68f
Summary:
move the bisect kind check earlier, so that it will not do unnecessary
node lookup
Reviewed By: sggutier
Differential Revision: D46643576
fbshipit-source-id: 624c6410aaf95f82a993093df4e24add6c5e48ea
Summary: this will be needed for making 'skip' lazy evaluated
Reviewed By: muirdm
Differential Revision: D46643575
fbshipit-source-id: 53cfe2395d320d85582fbf707f62d69a5088bea2
Summary: we will update this function in the following diffs, abstract it to a separate function will avoid making the main function too complex
Reviewed By: muirdm
Differential Revision: D46643577
fbshipit-source-id: 227e0b60fe2efa1d6d00fa409ea95a30cc972747
Summary: We handle the error gracefully, when deleting a key from the underlying blobstores.
Reviewed By: RajivTS
Differential Revision: D46684795
fbshipit-source-id: 45349aab01e56b11fd1ab455aec2bb69d72c497b
Summary:
If there's corruption in the cache, it should be considered a miss,
rather than failing.
Reviewed By: chadaustin
Differential Revision: D46577606
fbshipit-source-id: 4cc99e743e869422a149a073b726722896d971d4
Summary:
The dagcopytrace was enabled for source missing file in D46315431.
It's too slow if there are many files deleted in the source commit. I will debug the root cause tomorrow, let's disable it for hg repo first. We are not disabling it for git repo, because there is not alternative for git repos.
Reviewed By: muirdm
Differential Revision: D46711460
fbshipit-source-id: 4e20bbb34be126c13af081475953b83df61872df
Summary: Adding to track possible corruption issues we've been getting user reports about
Reviewed By: chadaustin
Differential Revision: D46706164
fbshipit-source-id: 6a868d9f9ffd3f72303123486bbb8312f2002072
Summary: This diff allows the new admin tool to delete keys from all the related blobs (either the single blobstore specified by the id, or all the underlying blobstore given a repo)
Differential Revision: D46564415
fbshipit-source-id: d0299262a82acb4e1a9f4cedd5510418f270d527
Summary:
This diff adds a new function to construct multiple blobstores from a given repo.
The finaly step is to refactoring the existing `get_blobstore` function.
Reviewed By: RajivTS
Differential Revision: D46559402
fbshipit-source-id: 9b9901d00566556614264831f8643f7c6d4ee20f
Summary: Running a job in sharded setting requires it to support pre-emption in case ShardManager decides to move the repo around. This diff adds support for that pre-emption.
Differential Revision: D46514772
fbshipit-source-id: fb086e74a9c56e0ced5921877de49401bc426170
Summary: As in title. This tool is currently run as 5-6 different TW jobs that compute and publish data for specific repos. Instead of having multiple jobs, if we manage to shard this job across set of available tasks, it becomes much more easier to manage.
Differential Revision: D46483039
fbshipit-source-id: d7d879dee088c8f431c22915bd7ef9602b83aa00
Summary:
In the win32 implementation of `readFile`, the ret string wasn't being resized
to the number of bytes actually read, which may be smaller than given in
`num_bytes`. This change makes it match the behavior of folly's readFile.
Reviewed By: xavierd
Differential Revision: D46532796
fbshipit-source-id: 1da3e4ff4190673796731610ae8a798b953d0c96
Summary:
This diff adds a new function of constructing a single blobstore from config and inner_id.
Next step is to add a new function to construct multiple blobstores from a given repo.
The finaly step is to refactoring the existing `get_blobstore` function.
Reviewed By: RajivTS
Differential Revision: D46559565
fbshipit-source-id: b59e77e13fbe76ba013caa62bfeb9f56cbcf31d2
Summary: This diff added a function to retrieve all the inner blobstore ids from a given multiplexed blobstore.
Reviewed By: RajivTS
Differential Revision: D46523044
fbshipit-source-id: e4c3b02e1672bc2eea3208bac4f23d879c004515
Summary: Now that the `blob_repo` dependency has been removed from `statistics_collector`, this diff migrates the tool from old Mononoke App to new Mononoke App getting rid of the old clap dependency.
Differential Revision: D46478775
fbshipit-source-id: dd713eef2175fa01934508445866d64ef01657f0
Summary: In order to better understand what happens in cases described in https://fb.workplace.com/groups/edenfswindows/permalink/1315557692693439/ adding better logging and some data integrity checks.
Reviewed By: chadaustin
Differential Revision: D46605682
fbshipit-source-id: 485c5c83834cdde9ccfea9cfa94323cf03df7002
Summary:
eden doctor can discover when an inode is missing for a file, but can't
remediate the issue. restart usually remediates the issue, but it would be
better to have doctor remediate since restart can be very slow.
We could do something similar to our remediation for phantom inodes (performing
a filesystem operation). However, messing with the filesystem leaves us open to
races with concurrent modifications. The point of the filesystem io is to make
eden see a notification about a certain path and match it's state to the
filesystem. So we can directly do that instead.
We can more directly do this by introducing a thrift call to make eden match
it's internal state to the filesystem.
We could replace the other remediation with this thrift call. I'll leave that
for a follow up.
Reviewed By: xavierd
Differential Revision: D46243633
fbshipit-source-id: a1df5929428dc4f6c8fd71d826fe1e7371ebf283
Summary:
We've got a couple sets of prjfs tests now and they have some utility functions.
Let's move this all into a base class so it can easily be reused.
Reviewed By: xavierd
Differential Revision: D46379827
fbshipit-source-id: c4b708a1effebe5f11c97a63519295be625c8018
Summary:
eliminate completely unnecessary traffic to commit graph endpoint for
main bookmark (for segmented changelog supported repos, they use pull_fast_forward_master edenapi endpoint)
After the fast path pull being completed, the filtration of what is already
present wasn't working correctly. It could easily skip (not to be able to
filter out) what was just fetched via the fast path.
That resulted in completely unnecessary traffic to commit graph (which is
slower, especially for large bits of the master branch).
During the SEV that traffic was super expensive due to the www merge commit, but even without SEV this traffic leads to unnecessary load on our services especially if the repo is significantly behind master.
The solution is to invalidate caches.
Differential Revision: D46590498
fbshipit-source-id: c8d920f4ffd3411297ca7cd01368b731f3c2236e
Summary:
fix incorrect logging to scuba number of slow path heads
it makes more sense to log after the filter I think because in this case it
would mean how many we need to pull via commit graph (slower) path
Reviewed By: zzl0
Differential Revision: D46559984
fbshipit-source-id: 4059c189afbb6ecc1dea69f0726af7b00e576f74
Summary:
we have added repo._dagcopytrace property in a previous diff, let's
use it in the debugcopytrace command
Reviewed By: quark-zju
Differential Revision: D46570173
fbshipit-source-id: 729c12160e8f0ce89ae07a5c94ca98852bbaf68f
Summary:
update master bookmark on pull
update of the master bookmark normally uses fast path for most of the repos
pulling of bookmarks is the first thing the repo.pull implementation does, if
after that the commit is present in the repo, slow path pull won't be called
otherwise, we are calling slow path pull for fetching a chunk of main branch
if the clone is old, it is a long chunk, if there are merges, it gets even
more complicated for the slow path pull
the current version seems to have the issue with phases as well, then you restore snapshot from old clone, you got enormous smartlog.
Differential Revision: D46556754
fbshipit-source-id: a771a856813d513cd34b207ea76f1a410dc0b1c0
Summary:
After previous commit D46473305, ctrl-c is no longer handled by streampager in "direct" mode since streampager isn't polling for user input. This leaves the terminal in a funny state since streampager doesn't clean up.
Fix by waiting up to 10ms for streampager to shut down after we send it the "quit" action. I had to make the "direct" mode in streampager actually handle the "quit" action.
Reviewed By: quark-zju
Differential Revision: D46474536
fbshipit-source-id: 11df5849ea13ac530e169bdba33d5061a77647e0
Summary: Tweak the "direct" mode run loop to use recv_timeout on the event channel instead of a try_recv+sleep. This allows it to respond faster to events (in particular, I'm going to make it support "quit" in following commit).
Reviewed By: zzl0
Differential Revision: D46508080
fbshipit-source-id: efb1261324b2720baaa1960fae60a3b09f4d82ce
Summary:
Command line users commonly type their next command before the current command has finished (or, they paste in a series of commands all at once). Previously if one of the commands invoked the streampager, all remaining pending input to the terminal would get lost. This was because streampager would always put the terminal into raw mode, which loses pending input AFAICT. Raw mode also disables terminal echo, which is also undesirable for streampager "direct" mode.
Fix by deferring enablement of raw mode until it is needed. I also disabled the startup_poll_input option since that requires raw mode in order to poll for user input. This means a user can't type "q" or "h" anymore in direct mode, but I think that is a worthwhile tradeoff since users probably don't even know they are in a pager. This makes streampager behave more like "less -F".
Reviewed By: zzl0
Differential Revision: D46473305
fbshipit-source-id: 4523050bc6c8b96d0f37e2b6078555aa23ae2b28
Summary: Log repo name as well as commit hash and other commit info.
Differential Revision: D46432233
fbshipit-source-id: dff22dd00297d3d5694f09a2b64ef7d6c4b7c87e
Summary: This is a refactor aimed to retain the same behavior while consolidating the code and improving it's resilience to future changes.
Reviewed By: mitrandir77
Differential Revision: D46190582
fbshipit-source-id: a956f5df1044c52346a712e271c75baf298b481d
Summary:
In the case where PrjfsChannel::start failed to initialized, EdenFS would crash
due to the stopPromise_ not being fulfilled. That promise is being set in the
PrjfsChannel::stop method, but unfortunately, when start throws, the caller
wouldn't catch the exception.
To fix this, we can simply move all the code in the initialize method, which if
it throws, the caller already properly tears down the mount thus solving the
crash mentioned above.
Reviewed By: genevievehelsel
Differential Revision: D46210460
fbshipit-source-id: 1524e8e0fc74ce3119b1b6dcd35b5419efcebdf3
Summary: See the previous commit for context.
Reviewed By: muirdm
Differential Revision: D45959185
fbshipit-source-id: ff60352111d175fa9dfd9609e0a1bb2a15dd776e
Summary: See the added test for context.
Reviewed By: muirdm
Differential Revision: D45959186
fbshipit-source-id: e437d8dc51b3730504528ee836632ed0b5ebe088
Summary:
The Rust `NameSet` tracks O(1) `min`, `max` in its `Hints` structure.
Use it to implement `fastmin`, `fastmax`. This is used by the next change.
Reviewed By: muirdm
Differential Revision: D45959181
fbshipit-source-id: a0a506243610914c879978715c59e1277863508d
Summary:
Similar to `fastlen`, these methods return `None` if the information cannot be
computed quickly.
Reviewed By: muirdm
Differential Revision: D45959183
fbshipit-source-id: 4c693eab2122c12bad40d8c9e9ab228f6d1f5057
Summary:
Optimize fastlog follow() revset to avoid iterating on the slow `subset` for
both ASC and DESC order.
Reviewed By: muirdm
Differential Revision: D45936985
fbshipit-source-id: 951af6d82376d0256c25b03af531a49d45a01559
Summary:
For commands like:
hg log -r SOMETHING PATH
Use `follow(PATH,heads(SOMETHING))` instead of `_matchfiles(...)` as an optimization.
Reviewed By: muirdm
Differential Revision: D45936986
fbshipit-source-id: 0922fff697b38278f4c33a11f8c9ec66246ae0d9
Summary:
The behavior was added by D2634918. It sets `--removed` with the intention of
forcing the "slow path", in other words, "do not use filelog() path". It does
not really care about the `--removed` behavior.
Note: tweakdefaults might add the `--follow` flag, depending on whether
tweakdefaults or remotefilelog runs first, the `--removed` flag might be added
undesirably if tweakdefaults is going to add `--follow` next.
Anyway, with D45936982, the `filelog()` code path knows to disable itself
for remotefilelog repos. So there is no need for remotefilelog to use
`--removed` to disable the `filelog()` code path. Let's remove it so we can
get the true `--removed` flag decided by the user.
Reviewed By: muirdm
Differential Revision: D45936984
fbshipit-source-id: e800c09d0489e84e01a7d0b499a6d31963c73bf6
Summary:
The `_makelogrevset` feels cryptic due to the intertwined deeply nested
conditions about what strategy it uses to generate the revset.
This diff makes the strategies more explicit by using dedicated functions to
make the code (and future changes) easier to reason about.
Reviewed By: muirdm
Differential Revision: D45936982
fbshipit-source-id: 531464a1dc16e2e0b4aa61eda49a6c87488ae23b
Summary:
The `_makelogrevset` feels cryptic, partially because its use of `opt2revset`
and related logic for adhoc revset expression calculation.
This diff removes `opt2revset` and related logic, so:
- `opts` is no longer both input and output. It's only input. The output is a
different variable.
- Use `revsetlang` APIs to format revset strings directly, without going
through an indirection via `opts`. Avoid `repr` or `a % b` formatting.
Note: `%d` is formatted as `_intlist(v)` instead of `rev(v)`.
The new code generates slightly simpler revset expressions with less `()`s
(or "group" in revsetlang AST).
Reviewed By: zzl0
Differential Revision: D45936983
fbshipit-source-id: 3155016a85fa9ea61f9334a2d1fc82eac5450fec
Summary:
It's already the same as `descendants` in our current implementation.
It's non-trivial to implement efficiently and is rarely used. So let's just
remove it for now.
Note the order of children of a commit is undefined, unlike parents. If we want
to provide a "meaningful" order and implement this in the future we might want
to use commit date etc to sort children.
Reviewed By: zzl0
Differential Revision: D45936979
fbshipit-source-id: bff72a130a5ffb7c573f06374da8f9c02791049f
Summary: Useful to chain a list of revset together.
Reviewed By: zzl0
Differential Revision: D45936980
fbshipit-source-id: 1775cd472e239f56971d0b441aae3e416852ea3f
Summary: Named branches are removed in our implementation. Clean them up in the log command.
Reviewed By: zzl0
Differential Revision: D45936987
fbshipit-source-id: 98d3df01b1f3db53861f886289cb720dccf0535a
Summary: It's not used anywhere. This removes a dependency on the legacy filelog.
Reviewed By: zzl0
Differential Revision: D45936981
fbshipit-source-id: 1c3af8dc792454d1bf893e922c1c37c80fa0a79b
Summary: This can simplify logic that checks a bunch of repo requirements.
Reviewed By: zzl0
Differential Revision: D45936978
fbshipit-source-id: eec8ca0035642bf2324782b7fb7e97cb740cc02d
Summary: We need that for optimizing our storage.
Reviewed By: RajivTS
Differential Revision: D46362446
fbshipit-source-id: d81973cd05cffdc78ad84da867fa4791efea877c
Summary: As in title. Once `BlobRepo` has been removed from this tool, I will also migrate it to the new `Mononoke Admin`
Reviewed By: mitrandir77
Differential Revision: D46444744
fbshipit-source-id: 059623d32d608f852254f16f811bcaf4e6bf3b4a
Summary: This includes PR 7 so drops the patch.
Reviewed By: zertosh
Differential Revision: D46451758
fbshipit-source-id: 5df8ccbfa610eab470744e28b5bffe411a983996
Summary: Removing the previous default ctor to ensure that we always compute and pass blake3.
Reviewed By: chadaustin
Differential Revision: D46268716
fbshipit-source-id: d9bfbc7bfd07b61dbb2e915c3fe72d7526d919e1
Summary: Adding a method to retrieve blake3 hash for a list of files.
Reviewed By: chadaustin
Differential Revision: D46268718
fbshipit-source-id: 59cb3d25a1d059a7e9b6a4da784a820945ffbd32
Summary: Expose blake3 via file attributes and changing some integration tests accordingly. Added blake3_sum so that it could be used to verify blake3 hashes and updated the tests to work with blake3.
Reviewed By: chadaustin
Differential Revision: D46307686
fbshipit-source-id: 6f2a4e8e25757862ef17d56f92b90a95c7f5a474
Summary: This utility binary could be used from the integration tests to compute blake3 hashes as it is easier to have it than adding the python pypi package with native dependencies.
Reviewed By: chadaustin
Differential Revision: D46307685
fbshipit-source-id: 1c48c689312cd9a04a19a62ad02ae3e9185041e5
