Summary:
Use nameset instead of idset to underly the fullreposet. The idset does not prefetch nodes, so there was a serial location_to_hash call for each "repo[rev]" context lookup. The nameset prefetches nodes under the hood, so "repo[rev]" is fast and the next layer of commit text prefetching fills up its 10k batch quickly.
Note that anything hitting this code will still be extremely slow since it scans all commits in the repo.
Reviewed By: zzl0
Differential Revision: D44438396
fbshipit-source-id: e1275a7a45f54cae1978820eb36d99a212fc4236
Summary:
Before fbcode/common/stats/ codes are open sourced , implement a basic
working version of MonotonicCounter so that all hw tests using fb303 APIs like
fb303::fbData->getStatMap() should just work
Reviewed By: jasmeetbagga
Differential Revision: D44441336
fbshipit-source-id: f3e0fe0a59523bf90dd1e99bdca7bf5eca212c58
Summary: There was the classic same second race condition that left files NEED_CHECK after commit, and the tracing message had changed.
Reviewed By: quark-zju
Differential Revision: D44442614
fbshipit-source-id: 97486163301045825922f9b68ec00ff12e5432a7
Summary:
Use the "exists" field from watchman to detect when a file has been moved behind a symlink. Watchman reports it properly as not-exists, but if we naively vfs.metadata(path), it may appear to be unchanged.
I added another layer of Option<> to the file metadata so watchman fs can explicitly say "this file doesn't exist", as opposed to only "I don't have metadata".
The alternative approach is to "audit" paths before we lstat them, but that is slow and we want to avoid if possible.
Reviewed By: quark-zju
Differential Revision: D44187187
fbshipit-source-id: e24a43cbdb36d9aabf2b0b7ae0c4d2eab66394c4
Summary: Now the Rust status requests file mode, size and mtime from watchman and uses them for file comparison in lieu of lstatting the files. This way, if nothing has changed (think fresh instance), or if file size has changed, we don't need to hit the disk at all!
Reviewed By: quark-zju
Differential Revision: D44033614
fbshipit-source-id: 54cd8bd6dddd7fb88e9538d72174ae88b2118550
Summary: Add a test showing we fetch file metadata after watchmant tells us a file changed. I'm going to optimize this away buy querying and using watchman's file metadata.
Reviewed By: quark-zju
Differential Revision: D44033616
fbshipit-source-id: 8b098a5868e20283d2fc98b198c70c4b086b9657
Summary: Add a Metadata type to abstract across the various places we get file metadata from. This doesn't buy us much right now, but will allow us to easily inject file metadata from Watchman into our file comparison logic.
Reviewed By: quark-zju
Differential Revision: D44033617
fbshipit-source-id: ed08e44df9ff44bb23e8fff031d431c4a519eaa4
Summary: Use some #pragmas to ignore unreachable code warnings because sigemptyset on Mac can't return -1.
Reviewed By: quark-zju
Differential Revision: D44344187
fbshipit-source-id: bf650d20c38c950899aec3d8e7f91d18db4c031d
Summary: Going to be adding more "metadata" related stuff that depends on HgModifiedTime, so move this type first.
Reviewed By: quark-zju
Differential Revision: D44033613
fbshipit-source-id: 30c6173fc069fbb7553d6d9fffaea0fc774cb72f
Summary: This can take a few seconds if there are a ton of files (e.g. watchman fresh instance).
Reviewed By: quark-zju
Differential Revision: D44033611
fbshipit-source-id: 8e033eef68e2f4560b859d35f78da139f86abc97
Summary:
When using watchman, we show a progress bar as we compare the contents of files. Due to how the file detector works, this progress bar was staying at zero and suddenly jumping to the end. Fix by moving the progress bar into the ParallelDetector. I had to add a stupid method to the detector trait to set the total number of files.
I don't think the FileDetectorTrait is quite right. Ideally the user of the detector would own the progress bar. It would set "total" based on the work it has, and increment "position" as it receives results. However, currently you can't receive results until you have finished publishing work. We should tweak things to make this possible.
Reviewed By: quark-zju
Differential Revision: D44033612
fbshipit-source-id: a22b237eaf98f3bd55f5f79eccc3fe42e733a297
Summary:
While we are waiting for watchman to crawl the repo, call "debug-root-status" and pipe the file count into a progress bar.
The similar Python logic renders the "crawling" progress bar until the crawl is done and then proceeds with the query. In Rust I made it render the progress bar concurrently with the query. This should show the progress bar even if the repo has never been watched before (i.e. when there is no crawl until the query happens).
Reviewed By: quark-zju
Differential Revision: D44033615
fbshipit-source-id: 3c97fa7a63cd090b5ace82d7385f887e564eb9cf
Summary: The parallelized MutliWalker was skipping nested repo dot dirs (e.g. ".sl"), but wansn't skipping sibling files to the dot dir.
Reviewed By: quark-zju
Differential Revision: D43970586
fbshipit-source-id: 0cfd703a067f7db2087a6e692316eab192330b54
Summary:
Fix detection of a directory being replaced by a symlink. When the walker iterates tracked files not seen during the walk it now directly emits "deleted" events instead of going through the file detector. If tracked files had the same size and mtime "through" the symlink, the file detector's lstat() call would make it seem like there were no changes.
The alternative approach is to "audit" paths before we lstat them, but that is slow and we want to avoid if possible.
Reviewed By: quark-zju
Differential Revision: D44187186
fbshipit-source-id: 5678d6027b74283608bd0430e7acd3ae16cfb046
Summary: Show a warning about fifo etc. files instead of aborting the command.
Reviewed By: quark-zju
Differential Revision: D44187185
fbshipit-source-id: c4c01436ad21e4e15fcae9c5d04f1aaacc7e5c80
Summary:
The manifest stores a file as executable or symlink (mutually exclusive). A symlink on disk also has the executable bit(s) set, so the fs metadata (and by extension the mode in the dirstate) appears to be both symlink and executable. This is a recipe for confusion.
In Python it seems to work because the dirstate mode and fs mode both store the execute bit for symlinks, so they match when compared for executableness. (Although I wouldn't be surprised if they could get out of sync somehow.) The symlinkness check is achieved through the "flags" which does not include "x" if the file is "l".
Fix in Rust be applying the flags logic to the dirstate and fs metadata as well. In other words, if a file is a symlink, don't report it as executable. This makes the manifest file mode compare naturally with the dirstate and filesystem.
Reviewed By: quark-zju
Differential Revision: D43970590
fbshipit-source-id: b459e466dace5ee1c9cc7505d3c780be6ecd4685
Summary:
Supporting exporting the wdir() virtual commit could make it easier for ISL to
obtain file contents for interactive commit or amend.
Before this change it will error out in the Rust dag layer, since the Rust dag
layer does not know about the special wdir commit (yet). This diff works around
it by adding some special logic about it. Note in the long term we would like
to add the virtual commits directly to the Rust dag logic so they can be
calculated without special casing.
Reviewed By: evangrayk
Differential Revision: D44231011
fbshipit-source-id: 27fce76afd2399502b30e15cca6c8f8986e1bc34
Summary:
This is the reversed direction of `debugexportstack`, to create commits from
automation. It also supports `goto` and `reset` to support partial commit
(commit, reset), absorb (commit*, reset), amend (commit*, goto) cases.
Reviewed By: muirdm
Differential Revision: D44231010
fbshipit-source-id: 03d2e5a7d577a2c6becb5e7d9f162fea191ad9af
Summary: Check that the test cases actually cover various logic in debugexportstack.
Reviewed By: zzl0
Differential Revision: D44231012
fbshipit-source-id: dbb24c7f87ecd11d23aae1707b68f77486177ef7
Summary:
This makes it easier to verify that a complex Python function is fully
exercised.
Reviewed By: zzl0
Differential Revision: D44231013
fbshipit-source-id: 6269dae56acff4f29a6aa5b7ad0ab89c33601ee1
Summary: Add debug info for edenapi configs to verify if it load the static_system configs correctly
Reviewed By: quark-zju
Differential Revision: D44298145
fbshipit-source-id: 96efb86c4e96a945b7329ae1e9cba3fdf57a5128
Summary:
During cross-repo pushrebases, we need to log new commits both in large and small repos. But currently, due to the code organization, it is not too hard to forget logging in the small repo. Which is what happened in case of SCS land_stack method.
This land path was not covered, and only large repo commits were logged. For example, this led to most public opsfiles commits being missing from the category (sync to opsfiles_backup uses different code path and exposes the discrepancy):
{F912075962}
(https://fburl.com/scuba/mononoke_new_commit/8q0nbq8i)
Reviewed By: markbt
Differential Revision: D44187074
fbshipit-source-id: bba2bf112fc9326ae5fc177d421ec936a2bb5322
Summary:
The environment is passed into the async runtime via an `Arc`. If the last reference to the environment is dropped from within the runtime, then we end up destroying the runtime from within the runtime, which is an error.
Move the runtime out of the environment, and store it in the `MononokeApp` (new) or on the stack in `main` (old). This means the runtime will live on the stack outside of the async runtime, and so execution of all async code should be complete when they are dropped.
Reviewed By: YousefSalama
Differential Revision: D44299775
fbshipit-source-id: 9a93ba49a8bbf376defdd0b5ddb1d6546b5ff12b
Summary:
Curently, `debugcrdump` returns an empty string for `branch` if there are multiple downstream remote bookmarks. This diff returns the first bookmark as the best guess, which should be the bookmark (commit) that the current local commit is based on.
Caveat: for multiple downstream bookmarks, it's a best effort guess, the algorithm is like:
1. For a given commit, find the first public ancestor.
2. For that ancestor commit, find all descendant commits that have remote
bookmarks sorted in ascending order.
3. If the descendant remote bookmarks contain "master", return that.
4. Otherwise report the first remote bookmark as the current branch. For draft commit,
this should be (best guess) the remote bookmark on which the draft commit was based
if user didn't run pull from remote server.
For example, if your commit graph looks like this, it will return `bookmark1` (first one) as the branch name, this is probably what you are expecting.
```
Y: public bookmark2
|
| a: draft (your local commit)
|/
X: public bookmark1
```
But if you run a `hg pull`, then the graph might look like below, it's impossible to know for certain which remote bookmark the local commit was made against. For now we also pick `bookmark1` as a guess.
```
Z: public bookmark2
|
|
Y: public bookmark1
|
| a: draft (your local commit)
|/
X: public
```
Related diff: D17814487, this was the diff that first added `branch` field.
Reviewed By: sggutier
Differential Revision: D44355664
fbshipit-source-id: b5287578d82c756f2d7035ae5ffa9a542ab957a0
Summary: this is for reproducing the issue of T148764976
Reviewed By: sggutier
Differential Revision: D44360807
fbshipit-source-id: 14400f9abc3ef6ca8e72d1c4a1f20b6ada6361fa
Summary:
This test has been flaky practically since its creation, although below the threshold to be detected as such some of the times.
Looking at its journal, it failed its first stress run and more than half
of the stress runs thereafter (failed 1920 of them, passed 1519 of them).
The assumption that the test makes that we will always take the local path first
is not stable and has never been.
Match the test's expectations to the actual behaviour of mononoke for now.
If the behaviour is problematic, let's tackle it in its own time.
Note that this test was written here D34276252 in reaction to a change to
segmented changelog here D34041066 and the discussion on that diff is relevant
to establish the correctness of the code.
Reviewed By: RajivTS
Differential Revision: D44297278
fbshipit-source-id: 07f42fc11ca0aa637b59b5fc46263b65d624e0a4
Summary: After enabling remote derivation we've lost related to derivation logging in the parent scuba table (like SCS server). This diffs add log tags, which indicates when we requested remoted derivation and when it was finished. In case of the error we already had the logging in place.
Reviewed By: Croohand
Differential Revision: D44297304
fbshipit-source-id: 1e2f6e279e111ce01b2feaf950a11c7b69a314cf
Summary: useful for checking if an object is in the hgcache or not
Reviewed By: quark-zju
Differential Revision: D44316641
fbshipit-source-id: f16093ec1659a8f139b88e702dbd1d8646319c8a
Summary: In the next diff, we'll use this util function to acquire an exclusive handle to a file for a prjfs integration test.
Reviewed By: xavierd
Differential Revision: D44349708
fbshipit-source-id: ded7037266f539c681569f0c90f875717760fed5
Summary: We are running into fileNotification() failures way more often than we expect on Sandcastle (5% of certain jobs are failing). Let's add some logging here to see if the failures are also happening to users.
Reviewed By: xavierd
Differential Revision: D44350819
fbshipit-source-id: 3be7578dccc297071280f528b53270a63994a99d
Summary:
DurationScope has a pair of atomic reference count operations at the
beginning and end of every recorded duration. To avoid that overhead,
reference EdenStats with RefPtr and, in production EdenFS, store a
global, unowned EdenStats object.
This gives us the benefits of explicit dependencies and no global
state in tests, while avoiding atomic RMWs in the release build.
Reviewed By: xavierd
Differential Revision: D44323723
fbshipit-source-id: 1b3384d2e6a0a2959fd79774a8ba46afc4c176ca
Summary:
In Rust 1.68.0, the `derivable_impls` clippy lint [got smarter](https://github.com/rust-lang/rust-clippy/pull/10161).
Fix all instances of that lint in Mononoke, mute one that's impractical to address.
Reviewed By: markbt
Differential Revision: D44378068
fbshipit-source-id: 473a051a7001d9596db43f47b56cbad5f5db7efe
Summary:
In Rust 1.68.0, the `useless_conversion` clippy lint [got smarter](https://github.com/rust-lang/rust-clippy/pull/10020).
Fix all instances of that lint in Mononoke.
Reviewed By: markbt
Differential Revision: D44378069
fbshipit-source-id: acfd6c77c6a400830c378b4040661323e7232441
