Summary: This diff has a unit test to make sure on the fly sparse profiles work. We also allow for more than one sparse profile, and for multiple exclusion sparse profiles.
Reviewed By: quark-zju
Differential Revision: D23004062
fbshipit-source-id: 15f00401084c89a77d8e7bdd8c7eb688e20d131b
Summary: When running manual scrub for a large repo with one empty store, we are doing one peek per key. For keys that have existed for some time this is unnecessary as we know the key should exist and slows down the scrub.
Reviewed By: farnz
Differential Revision: D23054582
fbshipit-source-id: d2222350157ca37aa31b7792214af4446129c692
Summary:
Extract the calls to bookmarks_movement to separate functions to avoid duplication and
make the post-resolve action functions easier to read.
Reviewed By: StanislavGlebik
Differential Revision: D23057045
fbshipit-source-id: c6b5a8cdb2399e89c174c3df844529d4b5309edf
Summary: Refactor control of movement of non-scratch bookmarks through pushrebase.
Reviewed By: krallin
Differential Revision: D22920694
fbshipit-source-id: 347777045b4995b69973118781511686cf34bdba
Summary:
Some parts of the `pushrebase` public interface will be re-exported from `bookmarks_movement`.
Clean these up in preparation:
* Remove `OntoBookmarkParams` as it is now a simple wrapper around `BookmarkName` that
prevents us from using a reference.
* Make the bundle replay data `Option<&T>` rather than `&Option<T>`, allowing us to
use the former when available. The latter can be readily converted with `.as_ref()`.
* Rename `SuccessResult` to `Outcome` and `ErrorKind` to `InternalError`.
Reviewed By: krallin
Differential Revision: D23055580
fbshipit-source-id: 1208a934f979a9d5eb73310fb8711b1291393ecf
Summary:
Refactor control of movement of non-scratch bookmarks through force-pushrebase
or bookmark-only pushrebase. These are equivalent to ordinary pushes, and so
can use the same code path for moving the bookmarks.
This has the side-effect of enabling some patterns that were previously not
possible, like populating git mappings with a force-pushrebase.
Reviewed By: ikostia
Differential Revision: D22844828
fbshipit-source-id: 4ef71fa4cef69cc2f1d124837631e8304644ca06
Summary: Refactor control of movement of non-scratch bookmarks through plain pushes.
Reviewed By: krallin
Differential Revision: D22844829
fbshipit-source-id: 2f1a89e1d0f69880f74b7bc135144bfb305a918e
Summary:
Refactor control of movement of scratch bookmarks to a new `bookmark_movement` crate
that will contain all bookmark movement controls.
Reviewed By: krallin
Differential Revision: D22844830
fbshipit-source-id: 56d25ad45a9328eaa079c13466b4b802f033d1dd
Summary: Update internment to point at its latest master branch commit. Upstream has merged my PR to use DashMap inside internment, but they haven't cut a new crates release yet.
Reviewed By: jsgf, krallin
Differential Revision: D23075070
fbshipit-source-id: 8f4ec0e3ddbefd672c3040fb174d1cf5f6c1a94a
Summary: Ctime is an Option<i64>, so rather than as_ctime()/into_ctime() use the fact that it's fairly small and Copy to just .ctime()
Reviewed By: krallin
Differential Revision: D23081739
fbshipit-source-id: be62912eca02e5c29d7473d6f386d98df11000dd
Summary:
Windows defaults to checking a revocation server for ssl certs. Inside
our datacenter it can't reach the server and fails. We don't have this on for
any other platforms, so let's disable it.
Reviewed By: sfilipco
Differential Revision: D23121739
fbshipit-source-id: 4d44d2a065bf340a8f74332553deb09a9c61be9b
Summary: Previously we use HgImporter prefetch request in `prefetchBlobs()`, but using `getBlob()` can give us more control over the prefetch process later. So now `getBlob()` is used in `prefetchBlobs()` when `useEdenNativePrefetch` is configured as true.
Reviewed By: kmancini
Differential Revision: D22984848
fbshipit-source-id: 0bd0b1c5b50bb16da36f188915904d0223827dc3
Summary: Let's use new flag to enable/disable short history for getpack request
Reviewed By: krallin
Differential Revision: D23080200
fbshipit-source-id: 7aa0be6ded0601fa4d31d4b9ff8792a4f8d91b19
Summary:
The primary change is in `eden/scm/lib/edenapi/types`:
* Split `DataEntry` into `FileEntry` and `TreeEntry`.
* Split `DataError` into `FileError` and `TreeError`. Remove `Redacted` error variant from `TreeError` and `MaybeHybridManifest` error variant from `FileError`.
* Split `DataRequest`, `DataResponse` into appropriate File and Tree types.
* Refactor `data.rs` into `file.rs` and `tree.rs`.
* Lift `InvalidHgId` error, used by both File and Tree, into `lib.rs`.
* Bugfix: change `MaybeHybridManifest` to be returned only for hash mismatches with empty paths, to match documented behavior.
Most of the remaining changes are straightforward fallout of this split. Notable changes include:
* `eden/scm/lib/edenapi/tools/read_res`: I've split the "data" commands into "file" and "tree", but I've left the identical arguments sharing the same argument structs. These can be refactored later if / when they diverge.
* `eden/scm/lib/types/src/hgid.rs`: Moved `compute_hgid` from `eden/scm/lib/edenapi/types/src/data.rs` to as a new `from_content` constructor on the `HgId` struct.
* `eden/scm/lib/revisionstore/src/datastore.rs`: Split `add_entry` method on `HgIdMutableDeltaStore` trait into `add_file` and `add_tree` methods.
* `eden/scm/lib/revisionstore/src/edenapi`
* `mod.rs`: Split `prefetch` method on `EdenApiStoreKind` into `prefetch_files` and `prefetch_trees`, which are given a default implementation that fails with `unimplemented!`.
* `data.rs`: Replace blanket trait implementations for `EdenApiDataStore<T>` with specific implementations for `EdenApiDataStore<File>` and `EdenApiDataStore<Tree>` which call the appropriate fetch and add functions.
* `data.rs` `test_get_*`: Replace dummy hashes with real hashes. These tests were only passing due to the hash mismatches (incorrectly) being considered `MaybeHybridManifest` errors, and allowed to pass.
Reviewed By: kulshrax
Differential Revision: D22958373
fbshipit-source-id: 788baaad4d9be20686d527f819a7342678740bc3
Summary:
Biggrep recently moved to a sharded system where fbsource is spread
over three shards and can have three different corpus revs. Let's update hg's
biggrep integration to handle this new logic.
We basically just take the earliest corpus commit and diff that against the
users commit. There's room for some incorrectness here, like if the users
commit is based off a commit before the earliest corpus commit then it will
receive changes from the alter corpus commits. Long term biggrep will work on
unifying the various shards to use a consistent corpus rev, but for now this is
probably acceptable.
Reviewed By: quark-zju
Differential Revision: D23057055
fbshipit-source-id: 745cafe772cea44bbe2d8eb1eee70dd317f255fe
Summary:
We had an accidentally quadratic behaviour in our blame implementation.
blame_range_split_at copied the right part of the range over and over again.
This diff fixes it by using VecDeque instead
Reviewed By: aslpavel
Differential Revision: D23102690
fbshipit-source-id: 951dd6383c48206fdc92757a47690f8e826a737b
Summary:
On Windows, unmounting the repo isn't sufficient to remove all traces of the
repo as ProjectedFS uses it as a cache, and thus all the files that were
touched by the user are still present on disk after unmounting it.
Reviewed By: wez
Differential Revision: D22915349
fbshipit-source-id: 445379fef57160b5da39a298ca4518e2662d16e3
Summary:
On EdenFS on Windows (case insensitive), _dirs isn't set, but touching it will
lead to the rest of the dirstate code trying to use it. For instance, removing
a file in the dirstate would try to remove it from _dirs, which would fail as
_dirs is expected to be empty on EdenFS.
Reviewed By: fanzeyi
Differential Revision: D23067897
fbshipit-source-id: 0e0d4aa6457c31d572a2aebc87f0a2fa7215001f
Summary:
After creating the merge commit (D23028163 (f267bec3f7)) from the imported commit head and the destination bookmark's head, we need to push the commit onto that bookmark. This diff adds the push functionality to repo_import tool.
Note: GlobalrevPushrebaseHook is a hook to assign globalrevs to commits to keep the order of the commits
Reviewed By: StanislavGlebik
Differential Revision: D23072966
fbshipit-source-id: ff815467ed0f96de86da3de9a628fd45743eb167
Summary:
The `debugfsync` command calls fsync on newly modified files in svfs.
Right now it only includes locations that we know have constant number
of files.
The fsync logic is put in a separate crate to avoid slow compiles.
Reviewed By: DurhamG
Differential Revision: D22992103
fbshipit-source-id: b5503e498d5216d4ba19701ecd5582387e4f45f5
Summary: This allows callsites to get access to the storage.
Reviewed By: DurhamG
Differential Revision: D22992104
fbshipit-source-id: c72fa313be1468170c9728d3856f822bb6385dc8
Summary:
This makes the main command table cleaner.
I dropped the `indexedlogrepair` command as it cannot rebuild indexes. `hg
doctor` is a better replacement. Some debug commands are renamed so they
no longer have `-` in the command name.
Reviewed By: DurhamG
Differential Revision: D22992107
fbshipit-source-id: f65d74e36fb971e592ad0cc8be9a94e245c39662
Summary:
If every command lives in their module, then we can define the "module" interface:
- run(...): run the command
- doc(): the help text
- name(): command name, with aliases
Then the macro would make command registration look simpler.
This diff changes `status` to use the pattern as an example.
Reviewed By: DurhamG
Differential Revision: D22992109
fbshipit-source-id: eaf589863092ec2eb1f8c24c1c7e425492fe1e3a
Summary:
As the number of commands grows, it starts making sense to move them to
individual files. Let's create a directory for them.
Reviewed By: DurhamG
Differential Revision: D22992108
fbshipit-source-id: a0556be602b832579a8e027342d5b86d9d84d257
Summary:
EdenFS on Windows is a bit weird as ProjectedFS is implemented as a filter
driver that adds reparse point to all the files/directories to get notified of
filesystem operations on them. It then hides these reparse points from the
outside which means that the dwAttributes of a file in EdenFS will not claim
that a reparse point is attached to it. On top of this, newly created
files/directories won't have any reparse points attached to them, until they
start being tracked by EdenFS.
While the first issue can be solved by always querying the reparse tags, I'm
not entirely sure how to solve the second one. That second issue causes
Mercurial to always try to create hardlink in the .hg directory, while it shouldn't.
Reviewed By: DurhamG
Differential Revision: D22937788
fbshipit-source-id: 5d90cd37d40858ed60103ff2d17c2cef16472b38
Summary:
In a repository with files with large histories we run into a lot of SqlTimeout
errors while fetching file history to serve getpack calls. However fetching the
whole file history is not really necessary - client knows how to work with
partial history i.e. if client misses some portion of history then it would
just fetch it on demand.
This diff adds way to add a limit on how many entries were going to be fetched, and if more entries were fetched then we return FilenodeRangeResult::TooBig. The downside of this diff is that we'd have to do more sequential database
queries.
Reviewed By: krallin
Differential Revision: D23025249
fbshipit-source-id: ebed9d6df6f8f40e658bc4b83123c75f78e70d93
Summary:
See D23053788 for motivation. Let's add a new warmer that checks
mutable_counters to understand which commit has been imported already.
Reviewed By: krallin
Differential Revision: D23053991
fbshipit-source-id: 3651aed8836a791675dd8d7bcc145fd32e56a13f
Summary:
With Mercurial now supporting CMD_CAT_TREE for efficiently fetching and reading
trees, we can plumb this onto EdenFS. At startup time, we detect whether
Mercurial supports CMD_CAT_TREE and use that method, otherwise, we fallback to
the old CMD_FETCH_TREE.
Reviewed By: wez
Differential Revision: D23044953
fbshipit-source-id: 9aea5c5b82e97039a75ef18976a155dcb6e150bc
Summary:
Use sorted_vector_map when parsing hg manifest blob, as blobs are usually stored sorted, which can result in high cost of BTree insertion when traversing large repos.
Also uses the size_hint() from the parsing Split to save reallocations during insert.
Reviewed By: markbt
Differential Revision: D22975883
fbshipit-source-id: 1faff754f03d7b2c20ebb741fec4f97b310852f9
Summary:
When running `edenfsctl prefetch **/BUCK` with an empty hgcache, EdenFS ends up
asking mercurial for every manifest one by one. Unfortunately, every manifest
fetched also causes the packfile to be flushed to disk, which then leads EdenFS
to rescan the filesystem for the new packfile. Once too many packfiles are
present on disk, Mercurial triggers a repack. Effectively, that means we have a
quadratic complexity both on Mercurial, and on EdenFS's side.
While this has been a long standing issue, we've so far avoided falling into
this complexity for a number of reason. The main one being that the hgcache is
very rarely empty, and thus the quadratic complexity is usually on low number
of files. Users also rarely run a prefetch of all the files for the entire
repo. However, on repositories with long standing branches, the hgcache is
effectively cold and thus any prefetch would trigger the pathological behavior.
To solve this, we take the same approach taken for files: sending the raw
manifest to EdenFS, which will then take care of deserializing it properly.
Reviewed By: DurhamG
Differential Revision: D23035335
fbshipit-source-id: 855e6fb4fabf81c427fad6c9f17d05f95c47e9ae
Summary: There are no users waiting on manual scrub, so set it to use the background session mode.
Reviewed By: krallin
Differential Revision: D23054581
fbshipit-source-id: 985bcadbaf17d2a8c92fdec811ecb239cbca7b37
Summary:
On macOS, it appears that ssh has a ~1% chance of never being able to connect
too the server and just hang. This caused mactest to be completely unhealthy
for a couple of days and a similar hotfix was applied to mitigate the issue.
Since it proved to be working, let's now backport this hotfix in the actual
code.
Reviewed By: DurhamG
Differential Revision: D22953230
fbshipit-source-id: ead7662ea6d0a33efaa5c4044c9391b2835ee421
Summary: Client portion for the commit/revlog_data endpoint that was added to the server.
Reviewed By: kulshrax
Differential Revision: D23065989
fbshipit-source-id: 3115ad2b426daca22472e2106fcd293f3ccd70f3
Summary:
Pyre now has improved support for decorators and descriptors, which makes it
possible for us to add type annotations to `dirstate.py` without needing lots
of `pyre-ignore` comments everywhere. (Previously Pyre could not handle the
`propertycache` decorator, causing it to be confused about the type of
various dirstate members, like `_map`).
Reviewed By: mrkmndz
Differential Revision: D22969757
fbshipit-source-id: 1b54f1edfb56c20c237a34f14a47404d10605240
Summary: Begin adding some initial type annotations for the Rust Python bindings.
Reviewed By: quark-zju
Differential Revision: D22993222
fbshipit-source-id: 2073db93b22f6bb04e30b767594d435c36ddb17f
Summary:
Using os.kill on EdenFS would always fail and raise an exception. Use the
proc_utils code to detect if the process is running. Also using BUCKVERSION
always raises an error on Windows, so let's ignore that for now.
Reviewed By: fanzeyi
Differential Revision: D22915350
fbshipit-source-id: 806bfab12ae0e8fc97e83d5720481f2a47516129
Summary:
Let's split logic from WarmBookmarksCache into a separate builder. This builder
will configure which warmers we'd like to use.
This will make it easier to introduce a new warmer later in the stack
Reviewed By: krallin
Differential Revision: D23053785
fbshipit-source-id: 32acc9da98d32624ca0dc00277910443f3d86f66
Summary:
Previously we were unconditionally adding hg changesets, but that's a bit
strange and there's no reason to do it. Let's do the same check we do for other
derived data types. Note that there should be no change in behaviour - all our
repos have "hgchangesets" derived data type enabled.
Reviewed By: krallin
Differential Revision: D23053786
fbshipit-source-id: 0b3ea99f649bc89ea9b216f368fee11fa25e153f
Summary: I want to add a new warmer in the next diffs which won't do any deriving.
Reviewed By: krallin
Differential Revision: D23053787
fbshipit-source-id: 4c7febb60ab7e835302db746c670d656bd9d1989
Summary:
EdenFS may spawn several Mercurial process concurrently and they would all try
to take the wlock at startup time, more often than not, one of these process
would die early due to the tmplock not being present on disk. This is due to
the other Mercurial process removing it, let's have a 10s grace period where
temporary locks aren't removed to avoid this race.
Reviewed By: DurhamG
Differential Revision: D22954997
fbshipit-source-id: ce191265c03a7042d9c6e45db0dc44a688fa204c
