Summary:
Biggrep recently moved to a sharded system where fbsource is spread
over three shards and can have three different corpus revs. Let's update hg's
biggrep integration to handle this new logic.
We basically just take the earliest corpus commit and diff that against the
users commit. There's room for some incorrectness here, like if the users
commit is based off a commit before the earliest corpus commit then it will
receive changes from the alter corpus commits. Long term biggrep will work on
unifying the various shards to use a consistent corpus rev, but for now this is
probably acceptable.
Reviewed By: quark-zju
Differential Revision: D23057055
fbshipit-source-id: 745cafe772cea44bbe2d8eb1eee70dd317f255fe
Summary:
We had an accidentally quadratic behaviour in our blame implementation.
blame_range_split_at copied the right part of the range over and over again.
This diff fixes it by using VecDeque instead
Reviewed By: aslpavel
Differential Revision: D23102690
fbshipit-source-id: 951dd6383c48206fdc92757a47690f8e826a737b
Summary:
On Windows, unmounting the repo isn't sufficient to remove all traces of the
repo as ProjectedFS uses it as a cache, and thus all the files that were
touched by the user are still present on disk after unmounting it.
Reviewed By: wez
Differential Revision: D22915349
fbshipit-source-id: 445379fef57160b5da39a298ca4518e2662d16e3
Summary:
On EdenFS on Windows (case insensitive), _dirs isn't set, but touching it will
lead to the rest of the dirstate code trying to use it. For instance, removing
a file in the dirstate would try to remove it from _dirs, which would fail as
_dirs is expected to be empty on EdenFS.
Reviewed By: fanzeyi
Differential Revision: D23067897
fbshipit-source-id: 0e0d4aa6457c31d572a2aebc87f0a2fa7215001f
Summary:
Now that the subprocess output is no longer piped, we can see a weird situation
where the command being run is displayed after the compilation step:
+ cd /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/build/eden && \
[1/13] rust_job_pool
Finished release [optimized] target(s) in 0.75s
[12/13] Install the project...
-- Install configuration: "RelWithDebInfo"
-- Installing: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfs
-- Set runtime path of "/data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfs" to ""
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfsctl
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/lib/libbackingstore_rs.a
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/lib/libbackingstore.a
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/include/eden/scm/lib/backingstore/c_api/HgNativeBackingStore.h
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/include/eden/scm/lib/backingstore/c_api/RustBackingStore.h
+ /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/cmake-Ncng4tsJb6gdOu40ggy14-YtgNQD43 (4cb1fa6379)k5ev0n-FXq99I/bin/cmake \
+ --build \
+ /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/build/eden \
+ --target \
+ install \
+ --config \
+ Release \
+ -j \
+ 24
This is a bit awkward. Flushing stdout's buffer allows for the ordering to be
correct.
Reviewed By: wez
Differential Revision: D23079405
fbshipit-source-id: e2bf25b098d6ab4a788a5ec07deb635a42cae18c
Summary:
After creating the merge commit (D23028163 (f267bec3f7)) from the imported commit head and the destination bookmark's head, we need to push the commit onto that bookmark. This diff adds the push functionality to repo_import tool.
Note: GlobalrevPushrebaseHook is a hook to assign globalrevs to commits to keep the order of the commits
Reviewed By: StanislavGlebik
Differential Revision: D23072966
fbshipit-source-id: ff815467ed0f96de86da3de9a628fd45743eb167
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.
The fsync logic is put in a separate crate to avoid slow compiles.
Reviewed By: DurhamG
Differential Revision: D22992103
fbshipit-source-id: b5503e498d5216d4ba19701ecd5582387e4f45f5
Summary: This allows callsites to get access to the storage.
Reviewed By: DurhamG
Differential Revision: D22992104
fbshipit-source-id: c72fa313be1468170c9728d3856f822bb6385dc8
Summary:
This makes the main command table cleaner.
I dropped the `indexedlogrepair` command as it cannot rebuild indexes. `hg
doctor` is a better replacement. Some debug commands are renamed so they
no longer have `-` in the command name.
Reviewed By: DurhamG
Differential Revision: D22992107
fbshipit-source-id: f65d74e36fb971e592ad0cc8be9a94e245c39662
Summary:
If every command lives in their module, then we can define the "module" interface:
- run(...): run the command
- doc(): the help text
- name(): command name, with aliases
Then the macro would make command registration look simpler.
This diff changes `status` to use the pattern as an example.
Reviewed By: DurhamG
Differential Revision: D22992109
fbshipit-source-id: eaf589863092ec2eb1f8c24c1c7e425492fe1e3a
Summary:
As the number of commands grows, it starts making sense to move them to
individual files. Let's create a directory for them.
Reviewed By: DurhamG
Differential Revision: D22992108
fbshipit-source-id: a0556be602b832579a8e027342d5b86d9d84d257
Summary:
EdenFS on Windows is a bit weird as ProjectedFS is implemented as a filter
driver that adds reparse point to all the files/directories to get notified of
filesystem operations on them. It then hides these reparse points from the
outside which means that the dwAttributes of a file in EdenFS will not claim
that a reparse point is attached to it. On top of this, newly created
files/directories won't have any reparse points attached to them, until they
start being tracked by EdenFS.
While the first issue can be solved by always querying the reparse tags, I'm
not entirely sure how to solve the second one. That second issue causes
Mercurial to always try to create hardlink in the .hg directory, while it shouldn't.
Reviewed By: DurhamG
Differential Revision: D22937788
fbshipit-source-id: 5d90cd37d40858ed60103ff2d17c2cef16472b38
Summary:
In a repository with files with large histories we run into a lot of SqlTimeout
errors while fetching file history to serve getpack calls. However fetching the
whole file history is not really necessary - client knows how to work with
partial history i.e. if client misses some portion of history then it would
just fetch it on demand.
This diff adds way to add a limit on how many entries were going to be fetched, and if more entries were fetched then we return FilenodeRangeResult::TooBig. The downside of this diff is that we'd have to do more sequential database
queries.
Reviewed By: krallin
Differential Revision: D23025249
fbshipit-source-id: ebed9d6df6f8f40e658bc4b83123c75f78e70d93
Summary:
See D23053788 for motivation. Let's add a new warmer that checks
mutable_counters to understand which commit has been imported already.
Reviewed By: krallin
Differential Revision: D23053991
fbshipit-source-id: 3651aed8836a791675dd8d7bcc145fd32e56a13f
Summary:
With Mercurial now supporting CMD_CAT_TREE for efficiently fetching and reading
trees, we can plumb this onto EdenFS. At startup time, we detect whether
Mercurial supports CMD_CAT_TREE and use that method, otherwise, we fallback to
the old CMD_FETCH_TREE.
Reviewed By: wez
Differential Revision: D23044953
fbshipit-source-id: 9aea5c5b82e97039a75ef18976a155dcb6e150bc
Summary:
Use sorted_vector_map when parsing hg manifest blob, as blobs are usually stored sorted, which can result in high cost of BTree insertion when traversing large repos.
Also uses the size_hint() from the parsing Split to save reallocations during insert.
Reviewed By: markbt
Differential Revision: D22975883
fbshipit-source-id: 1faff754f03d7b2c20ebb741fec4f97b310852f9
Summary:
When running `edenfsctl prefetch **/BUCK` with an empty hgcache, EdenFS ends up
asking mercurial for every manifest one by one. Unfortunately, every manifest
fetched also causes the packfile to be flushed to disk, which then leads EdenFS
to rescan the filesystem for the new packfile. Once too many packfiles are
present on disk, Mercurial triggers a repack. Effectively, that means we have a
quadratic complexity both on Mercurial, and on EdenFS's side.
While this has been a long standing issue, we've so far avoided falling into
this complexity for a number of reason. The main one being that the hgcache is
very rarely empty, and thus the quadratic complexity is usually on low number
of files. Users also rarely run a prefetch of all the files for the entire
repo. However, on repositories with long standing branches, the hgcache is
effectively cold and thus any prefetch would trigger the pathological behavior.
To solve this, we take the same approach taken for files: sending the raw
manifest to EdenFS, which will then take care of deserializing it properly.
Reviewed By: DurhamG
Differential Revision: D23035335
fbshipit-source-id: 855e6fb4fabf81c427fad6c9f17d05f95c47e9ae
Summary: There are no users waiting on manual scrub, so set it to use the background session mode.
Reviewed By: krallin
Differential Revision: D23054581
fbshipit-source-id: 985bcadbaf17d2a8c92fdec811ecb239cbca7b37
Summary:
On macOS, it appears that ssh has a ~1% chance of never being able to connect
too the server and just hang. This caused mactest to be completely unhealthy
for a couple of days and a similar hotfix was applied to mitigate the issue.
Since it proved to be working, let's now backport this hotfix in the actual
code.
Reviewed By: DurhamG
Differential Revision: D22953230
fbshipit-source-id: ead7662ea6d0a33efaa5c4044c9391b2835ee421
Summary: Client portion for the commit/revlog_data endpoint that was added to the server.
Reviewed By: kulshrax
Differential Revision: D23065989
fbshipit-source-id: 3115ad2b426daca22472e2106fcd293f3ccd70f3
Summary:
Pyre now has improved support for decorators and descriptors, which makes it
possible for us to add type annotations to `dirstate.py` without needing lots
of `pyre-ignore` comments everywhere. (Previously Pyre could not handle the
`propertycache` decorator, causing it to be confused about the type of
various dirstate members, like `_map`).
Reviewed By: mrkmndz
Differential Revision: D22969757
fbshipit-source-id: 1b54f1edfb56c20c237a34f14a47404d10605240
Summary: Begin adding some initial type annotations for the Rust Python bindings.
Reviewed By: quark-zju
Differential Revision: D22993222
fbshipit-source-id: 2073db93b22f6bb04e30b767594d435c36ddb17f
Summary:
Using os.kill on EdenFS would always fail and raise an exception. Use the
proc_utils code to detect if the process is running. Also using BUCKVERSION
always raises an error on Windows, so let's ignore that for now.
Reviewed By: fanzeyi
Differential Revision: D22915350
fbshipit-source-id: 806bfab12ae0e8fc97e83d5720481f2a47516129
Summary:
Let's split logic from WarmBookmarksCache into a separate builder. This builder
will configure which warmers we'd like to use.
This will make it easier to introduce a new warmer later in the stack
Reviewed By: krallin
Differential Revision: D23053785
fbshipit-source-id: 32acc9da98d32624ca0dc00277910443f3d86f66
Summary:
Previously we were unconditionally adding hg changesets, but that's a bit
strange and there's no reason to do it. Let's do the same check we do for other
derived data types. Note that there should be no change in behaviour - all our
repos have "hgchangesets" derived data type enabled.
Reviewed By: krallin
Differential Revision: D23053786
fbshipit-source-id: 0b3ea99f649bc89ea9b216f368fee11fa25e153f
Summary: I want to add a new warmer in the next diffs which won't do any deriving.
Reviewed By: krallin
Differential Revision: D23053787
fbshipit-source-id: 4c7febb60ab7e835302db746c670d656bd9d1989
Summary:
EdenFS may spawn several Mercurial process concurrently and they would all try
to take the wlock at startup time, more often than not, one of these process
would die early due to the tmplock not being present on disk. This is due to
the other Mercurial process removing it, let's have a 10s grace period where
temporary locks aren't removed to avoid this race.
Reviewed By: DurhamG
Differential Revision: D22954997
fbshipit-source-id: ce191265c03a7042d9c6e45db0dc44a688fa204c
