Summary:
Move handling of service write bookmark restrictions into the `bookmarks_movement` crate.
This moves `check_bookmark_modification_permitted` from `mononoke_api` onto
`SourceControlServiceParams`, where it can be called from `bookmarks_movement`.
Reviewed By: StanislavGlebik
Differential Revision: D23288000
fbshipit-source-id: e346231b183ce1533ab03130fd2ddab709176fcd
Summary:
Bookmark movement for service write will use different restrictions than hooks.
Move hook running to be controlled by an enum in preparation for adding service
write restrictions.
Reviewed By: StanislavGlebik
Differential Revision: D23287998
fbshipit-source-id: 30670d4d6666c341885b57a3f41246e52db541a2
Summary: Use bookmarks_movement to implement the bookmark move in repo_move_bookmark.
Reviewed By: StanislavGlebik
Differential Revision: D23222562
fbshipit-source-id: 31249411d9521823f90248f459eb34ed4e2faea5
Summary:
The error message for fast-forward failure is wrong. The correct way to allow
non-fast-forward moves is with the NON_FAST_FORWARD pushvar.
Reviewed By: StanislavGlebik
Differential Revision: D23243542
fbshipit-source-id: 554cdee078cd712f17441bd10bd7968b0674bbfe
Summary:
When bookmarks are moved or created, work out what additional changesets
should have the hooks run on them. This may apply to plain pushes,
force pushrebases, or bookmark only pushrebases.
At first, this will run in logging-only mode where we will count how many
changesets would have hooks run on them (up to a tunable limit). We can
enable running of hooks with a tunable killswitch later on.
Reviewed By: StanislavGlebik
Differential Revision: D23194240
fbshipit-source-id: 8031fdc1634168308c7fe2ad3c22ae4389a04711
Summary:
Move the running of hooks from in `repo_client` to in `bookmarks_movement`.
For pushrebase and plain push we still only run hooks on the new commits the client has sent.
Bookmark-only pushrebases, or moves where some commits were already known, do not run
the hooks on the omitted changesets. That will be addressed next.
The push-redirector currently runs hooks in the large repo. Since hook running has now been moved
to later on, they will automatically be run on the large repo, and instead the push-redirector runs them on
the small repo, to ensure they are run on both.
There's some additional complication with translating hook rejections in the push-redirector. Since a
bookmark-only push can result in hook rejections for commits that are not translated, we fall back to
using the large-repo commit hash in those scenarios.
Reviewed By: StanislavGlebik
Differential Revision: D23077551
fbshipit-source-id: 07f66a96eaca4df08fc534e335e6d9f6b028730d
Summary: We will shortly need a `HookManager` in the write methods of the source control service. Add one to `mononoke_api::Repo`
Reviewed By: StanislavGlebik
Differential Revision: D23077552
fbshipit-source-id: e1eed3661fe26a839e50ac4d884f4fadf793dbbb
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999792
fbshipit-source-id: 56e5ec68469cb9c154a5c3045ded969253270b94
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999793
fbshipit-source-id: 94e53adf5458e0bc1ebceffb3b548b7fc021218a
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999141
fbshipit-source-id: e2e4177e56db85f65930b67a9e927a5c93b652df
Summary: We need this functionality for scmquery replacement.
Reviewed By: krallin
Differential Revision: D22999142
fbshipit-source-id: 04cea361ea6270626e7ff77255e3dc75875ece97
Summary:
Rust doesn't have named arguments as with positional it's hard to keep track
of all of them if there're many. I'm planning to add one more so let's switc to
struct.
Reviewed By: krallin
Differential Revision: D22999143
fbshipit-source-id: 54dade05f860b41d18bebb52317586015a893919
Summary:
If the imported commit has manifest id with all zeros (empty commit). Blobimport job can't find it in blobstore and returns error D23266254.
Add an early return when the manifest_id is NULL_HASH.
Reviewed By: StanislavGlebik
Differential Revision: D23266254
fbshipit-source-id: b8a3c47edfdfdc9d8cc8ea032fb96e27a04ef911
Summary:
Related commits: D23214677 (dcb565409d), D23213192
In the previous commits we added phabricator callsigns to the repo configs.
Since we can extract the callsigns from them, we don't need the callsign
flag for repo_import tool. This diff removes the flag and uses the config variable.
Reviewed By: StanislavGlebik
Differential Revision: D23240398
fbshipit-source-id: d8b853d37e21be97af42e9f50658b9f471f8fc48
Summary: The `map_err` call can be done with the new future from `compat()`.
Reviewed By: StanislavGlebik
Differential Revision: D23239251
fbshipit-source-id: c80609ae0a975bc54253784e002a07a048651aa3
Summary:
Add tests for basic functionality of `resolve_bookmark` and `list_bookmarks`,
ensuring that they correctly go through the warm bookmarks cache.
`list_bookmarks` was still using old-style streams, so upgrade it to new streams.
Differential Revision: D23239250
fbshipit-source-id: f78abae2d382263be76c34f1488249677134a74d
Summary:
If the warm bookmarks cache doesn't contain the bookmark we are looking for,
this might just be because it's a scratch bookmark, which aren't included in
that cache.
Always request the bookmark from the backing db if the cache misses.
Reviewed By: StanislavGlebik
Differential Revision: D23238009
fbshipit-source-id: c8843f1974ba14f148e30ba78a38eb710e7383b6
Summary:
We already had a logic that prints if we are about to run an expensive
getbundle. However this logic prints a warning after we've fetched 1M commits
already, and user would have to wait for a long time to get this message.
However in some cases we can give this warning very quickly. For example, if
the lowest "heads" generation number is >1M commits away from highest "common"
generation number, then we can print the warning right away.
Differential Revision: D23213482
fbshipit-source-id: 67e2399ca958703129cf3c22d82ce48cbbdcd2d1
Summary:
In the next diff I'd like to compute generation number first, and then call
DifferenceOfUnionsOfAncestorsNodeStream. To avoid refetching these numbers
again let's create a function that accepts a vector of (ChangesetId,
Generation) pairs.
While here I also made the order more consistent: now we have "hashes"
parameters always in front of "excludes"
Differential Revision: D23212883
fbshipit-source-id: 11e0a1494126f84b36e3e33e65071449db5840d2
Summary:
Previously for undesired fetches of lfs files we were logging 0. Let's log the
real path instead
Reviewed By: ikostia
Differential Revision: D23209754
fbshipit-source-id: 7a893b257a89332a5169ab2072ecf48ae94b91e0
Summary: Minor refactor: moved the tests from main to their own file, because main was getting too large and found it hard to navigate through the code.
Reviewed By: StanislavGlebik
Differential Revision: D23188766
fbshipit-source-id: a2b2e32c77587f95c07a0bb02a4957e3671dd2c6
Summary:
This largely moves connection accepting from old style bytes, futures and tokio
to updated versions, while keeping some parts at old bytes/futures in order to
remain compatible with the rest of the Mononoke codebase.
Division lies on `Stdio` which maintains old channels, stream and futures,
while the socket handling, connection acception and wire encoding is updated.
With the updated futures, we now wait for the forwarding stream to have
succeeded before considering a connection fully handled.
Other notable changes:
- futures_ext now a mini codec Decoder instead of relying on NetstringDecoder,
which has been updated to use bytes 0.5
- hgcli has been modified to use updated NetstringDecoder
- netstring now requires the updated bytes 0.5 crate
- the part in connection_acceptor was handling repo/security logic is now part of repo_handler (as it should have been), connection_acceptor now only handles networking and framing
- tests now verify that the shutdown handler is triggered
Reviewed By: krallin
Differential Revision: D22526867
fbshipit-source-id: 34e43af4a0c8b84de0000f2093d7fffd3fb0e20d
Summary:
Subscribers to the commit tailer categories would like to know when Mononoke
received the commit.
Reviewed By: StanislavGlebik
Differential Revision: D23162447
fbshipit-source-id: 747214f1964a643f59c491aa08cdbd5c8fe331c8
Summary:
Allow callers of `resolve_bookmark` to specify whether they'd like the most recent value of
the bookmark, rather than one which may be stale.
Use this in the repo_move_bookmark test to avoid flakiness caused by the test code racing against
the warm bookmark cache.
Reviewed By: StanislavGlebik
Differential Revision: D23151427
fbshipit-source-id: 4b8358be1cf103479ccc23a41b2505776543ee49
Summary:
Extract construction of the hook manager to its own crate, so that we can re-use it.
Eventually the hook manager will become a repo attribute and will be constructed by
the repo attribute factory, but for now it needs its own factory method.
Differential Revision: D23129407
fbshipit-source-id: 302fde4d1ae38c6f61032a32c880018ebf84dee2
Summary:
Convert hook rejections from a tuple to a named struct. This will be used in
the bookmarks_movement public interface.
Reviewed By: krallin
Differential Revision: D23077550
fbshipit-source-id: a35476817660c38b8df879ba603b927a7e39be21
Summary: Some repos are push-redirected repos: pushes go to another repos, but then synced into this repo. Because of this, when we import a repo into a smaller repo that push-redirects to a large repo, we need to make sure we don't break the large repo with the imported code, since merges, pushes, imports etc. are redirected to the large repo. For now, in order to avoid breaking the large repo, we added a simple check that returns error, if the small repo push-redirects to the large one.
Reviewed By: ikostia
Differential Revision: D23158826
fbshipit-source-id: f722790441d641f67293e78c5d1ea5d1102bbb9b
Summary: When running manual scrub for a large repo with one empty store, we are doing one peek per key. For keys that have existed for some time this is unnecessary as we know the key should exist and slows down the scrub.
Reviewed By: farnz
Differential Revision: D23054582
fbshipit-source-id: d2222350157ca37aa31b7792214af4446129c692
Summary:
Extract the calls to bookmarks_movement to separate functions to avoid duplication and
make the post-resolve action functions easier to read.
Reviewed By: StanislavGlebik
Differential Revision: D23057045
fbshipit-source-id: c6b5a8cdb2399e89c174c3df844529d4b5309edf
Summary: Refactor control of movement of non-scratch bookmarks through pushrebase.
Reviewed By: krallin
Differential Revision: D22920694
fbshipit-source-id: 347777045b4995b69973118781511686cf34bdba
Summary:
Some parts of the `pushrebase` public interface will be re-exported from `bookmarks_movement`.
Clean these up in preparation:
* Remove `OntoBookmarkParams` as it is now a simple wrapper around `BookmarkName` that
prevents us from using a reference.
* Make the bundle replay data `Option<&T>` rather than `&Option<T>`, allowing us to
use the former when available. The latter can be readily converted with `.as_ref()`.
* Rename `SuccessResult` to `Outcome` and `ErrorKind` to `InternalError`.
Reviewed By: krallin
Differential Revision: D23055580
fbshipit-source-id: 1208a934f979a9d5eb73310fb8711b1291393ecf
Summary:
Refactor control of movement of non-scratch bookmarks through force-pushrebase
or bookmark-only pushrebase. These are equivalent to ordinary pushes, and so
can use the same code path for moving the bookmarks.
This has the side-effect of enabling some patterns that were previously not
possible, like populating git mappings with a force-pushrebase.
Reviewed By: ikostia
Differential Revision: D22844828
fbshipit-source-id: 4ef71fa4cef69cc2f1d124837631e8304644ca06
Summary: Refactor control of movement of non-scratch bookmarks through plain pushes.
Reviewed By: krallin
Differential Revision: D22844829
fbshipit-source-id: 2f1a89e1d0f69880f74b7bc135144bfb305a918e
Summary:
Refactor control of movement of scratch bookmarks to a new `bookmark_movement` crate
that will contain all bookmark movement controls.
Reviewed By: krallin
Differential Revision: D22844830
fbshipit-source-id: 56d25ad45a9328eaa079c13466b4b802f033d1dd
Summary: Update internment to point at its latest master branch commit. Upstream has merged my PR to use DashMap inside internment, but they haven't cut a new crates release yet.
Reviewed By: jsgf, krallin
Differential Revision: D23075070
fbshipit-source-id: 8f4ec0e3ddbefd672c3040fb174d1cf5f6c1a94a
Summary: Ctime is an Option<i64>, so rather than as_ctime()/into_ctime() use the fact that it's fairly small and Copy to just .ctime()
Reviewed By: krallin
Differential Revision: D23081739
fbshipit-source-id: be62912eca02e5c29d7473d6f386d98df11000dd
Summary: Let's use new flag to enable/disable short history for getpack request
Reviewed By: krallin
Differential Revision: D23080200
fbshipit-source-id: 7aa0be6ded0601fa4d31d4b9ff8792a4f8d91b19
Summary:
The primary change is in `eden/scm/lib/edenapi/types`:
* Split `DataEntry` into `FileEntry` and `TreeEntry`.
* Split `DataError` into `FileError` and `TreeError`. Remove `Redacted` error variant from `TreeError` and `MaybeHybridManifest` error variant from `FileError`.
* Split `DataRequest`, `DataResponse` into appropriate File and Tree types.
* Refactor `data.rs` into `file.rs` and `tree.rs`.
* Lift `InvalidHgId` error, used by both File and Tree, into `lib.rs`.
* Bugfix: change `MaybeHybridManifest` to be returned only for hash mismatches with empty paths, to match documented behavior.
Most of the remaining changes are straightforward fallout of this split. Notable changes include:
* `eden/scm/lib/edenapi/tools/read_res`: I've split the "data" commands into "file" and "tree", but I've left the identical arguments sharing the same argument structs. These can be refactored later if / when they diverge.
* `eden/scm/lib/types/src/hgid.rs`: Moved `compute_hgid` from `eden/scm/lib/edenapi/types/src/data.rs` to as a new `from_content` constructor on the `HgId` struct.
* `eden/scm/lib/revisionstore/src/datastore.rs`: Split `add_entry` method on `HgIdMutableDeltaStore` trait into `add_file` and `add_tree` methods.
* `eden/scm/lib/revisionstore/src/edenapi`
* `mod.rs`: Split `prefetch` method on `EdenApiStoreKind` into `prefetch_files` and `prefetch_trees`, which are given a default implementation that fails with `unimplemented!`.
* `data.rs`: Replace blanket trait implementations for `EdenApiDataStore<T>` with specific implementations for `EdenApiDataStore<File>` and `EdenApiDataStore<Tree>` which call the appropriate fetch and add functions.
* `data.rs` `test_get_*`: Replace dummy hashes with real hashes. These tests were only passing due to the hash mismatches (incorrectly) being considered `MaybeHybridManifest` errors, and allowed to pass.
Reviewed By: kulshrax
Differential Revision: D22958373
fbshipit-source-id: 788baaad4d9be20686d527f819a7342678740bc3
Summary:
We had an accidentally quadratic behaviour in our blame implementation.
blame_range_split_at copied the right part of the range over and over again.
This diff fixes it by using VecDeque instead
Reviewed By: aslpavel
Differential Revision: D23102690
fbshipit-source-id: 951dd6383c48206fdc92757a47690f8e826a737b
Summary:
After creating the merge commit (D23028163 (f267bec3f7)) from the imported commit head and the destination bookmark's head, we need to push the commit onto that bookmark. This diff adds the push functionality to repo_import tool.
Note: GlobalrevPushrebaseHook is a hook to assign globalrevs to commits to keep the order of the commits
Reviewed By: StanislavGlebik
Differential Revision: D23072966
fbshipit-source-id: ff815467ed0f96de86da3de9a628fd45743eb167
Summary:
In a repository with files with large histories we run into a lot of SqlTimeout
errors while fetching file history to serve getpack calls. However fetching the
whole file history is not really necessary - client knows how to work with
partial history i.e. if client misses some portion of history then it would
just fetch it on demand.
This diff adds way to add a limit on how many entries were going to be fetched, and if more entries were fetched then we return FilenodeRangeResult::TooBig. The downside of this diff is that we'd have to do more sequential database
queries.
Reviewed By: krallin
Differential Revision: D23025249
fbshipit-source-id: ebed9d6df6f8f40e658bc4b83123c75f78e70d93
Summary:
See D23053788 for motivation. Let's add a new warmer that checks
mutable_counters to understand which commit has been imported already.
Reviewed By: krallin
Differential Revision: D23053991
fbshipit-source-id: 3651aed8836a791675dd8d7bcc145fd32e56a13f
Summary:
Use sorted_vector_map when parsing hg manifest blob, as blobs are usually stored sorted, which can result in high cost of BTree insertion when traversing large repos.
Also uses the size_hint() from the parsing Split to save reallocations during insert.
Reviewed By: markbt
Differential Revision: D22975883
fbshipit-source-id: 1faff754f03d7b2c20ebb741fec4f97b310852f9
