Summary:
The earlier diffs in this stack have removed all our dependencies on the Tokio
0.1 runtime environment (so, basically, `tokio-executor` and `tokio-timer`), so
we don't need this anymore.
We do still have some deps on `tokio-io`, but this is just traits + helpers,
so this doesn't actually prevent us from removing the 0.1 runtime!
Note that we still have a few transitive dependencies on Tokio 0.1:
- async-unit uses tokio-compat
- hg depends on tokio-compat too, and we depend on it in tests
This isn't the end of the world though, we can live with that :)
Reviewed By: ahornby
Differential Revision: D26544410
fbshipit-source-id: 24789be2402c3f48220dcaad110e8246ef02ecd8
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary:
This is a preparation for potential necessity of IO being done by this trait and its implementors.
We think the IO might be needed if we move commit sync config storage from `Configerator` into xdb, or some place else. To be clear, I personally am not certain we'll *need* this, but in any case, asyncifying the trait does not seem like a risky thing here (because we usually have only 0-2 sync functions in the stack above `LiveCommitSyncConfig` accessors, so it does not require large-scale code flow changes or anything).
I intentionally did not touch the push-redirection accessors, as those I don't think will ever move away from configerator.
Reviewed By: StanislavGlebik
Differential Revision: D26275905
fbshipit-source-id: 1bfdca087434d475d50032dd47dd94f16be051f9
Summary:
Like it says in the title, this updates our implementation of Globalrevs to
be a little more relaxed, and allows you to create and move bookmarks as long as
they are ancestors of the "main" Globalrevs bookmark (but NOT to pushrebase to
them later, because we only want to allow ONE globalrevs-publishing bookmark
per repo).
While in there, I also deduplicated how we instantiate pushrebase hooks a
little bit. If anything, this could be better in the pushrebase crate, but
that'd be a circular dependency between pushrebase & bookmarks movement.
Eventually, the callsites should probably be using bookmarks movement anyway,
so leaving pushrebase as the low-level crate and bookmarks movement as the high
level one seems reasonable.
Reviewed By: StanislavGlebik
Differential Revision: D26020274
fbshipit-source-id: 5ff6c1a852800b491a16d16f632462ce9453c89a
Summary:
Like it says in the title. This is prep work for allowing extra
bookmarks in a Gobalrevs repo later in this stack.
Reviewed By: ahornby
Differential Revision: D26076566
fbshipit-source-id: c775d50dfaa51e0f0f64e861b6c5b7ee16d62074
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.
This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).
The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.
Reviewed By: farnz
Differential Revision: D25952428
fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
Summary:
This feature is useful for testing time-dependent stuff (e.g. it
allows you to stop/forward time). It's already included in the buck build.
Reviewed By: SkyterX
Differential Revision: D25946732
fbshipit-source-id: 5e7b69967a45e6deaddaac34ba78b42d2f2ad90e
Summary: Allow us to return arg parsing errors rather than panicing
Reviewed By: krallin
Differential Revision: D25837626
fbshipit-source-id: 87e39de140b1dcd3b13a529602fdafc31233175d
Summary: Convert all BlobRepoHg methods to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25471540
fbshipit-source-id: c8e99509d39d0e081d082097cbd9dbfca431637e
Summary:
Change derived data config to have "enabled" config and "backfilling" config.
The `Mapping` object has the responsibility of encapsulating the configuration options
for the derived data type. Since it is only possible to obtain a `Mapping` from
appropriate configuration, ownership of a `Mapping` means derivation is permitted,
and so the `DeriveMode` enum is removed.
Most callers will use `BonsaiDerived::derive`, or a default `derived_data_utils` implementation
that requires the derived data to be enabled and configured on the repo.
Backfillers can additionally use `derived_data_utils_for_backfill` which will use the
`backfilling` configuration in preference to the default configuration.
Reviewed By: ahornby
Differential Revision: D25246317
fbshipit-source-id: 352fe6509572409bc3338dd43d157f34c73b9eac
Summary: Take the parameters to `derived_data_utils` and `derived_data_utils_unsafe` by reference.
Reviewed By: krallin
Differential Revision: D25371970
fbshipit-source-id: d260650c2398e33667e1bc5779fbabdff04f1f98
Summary:
The `BonsaiDerived` trait is split in two:
* The new `BonsaiDerivable` trait encapsulates the process of deriving the data, either
a single item from its parents, or a batch.
* The `BonsaiDerived` trait is used only as an entry point for deriving with the default
mapping and config.
This split will allow us to use `BonsaiDerivable` in batch backfilling with non-default
config, for example when backfilling a new version of a derived data type.
Reviewed By: krallin
Differential Revision: D25371964
fbshipit-source-id: 5874836bc06c18db306ada947a690658bf89723c
Summary:
Following features added to gitimport, both the library rutine and the command line version.
* Define your own parts of a git-library to import by implementing the `GitimportTarget` trait.
* Added `GitimportTarget` implementation
`ImportMissingForCommit` that will search for any missing reference in Mononoke for the specified git-commit and import them. Note that it will not import commits unreachable by the specified commit history.
* Added support to update the bonsai<->git commit mapping while importing commits.
* Commit import progress is now shown, making it a bit easier to estimate how big an import job is and how long it will take.
* Adding optional git-repo name. This is useful when using gitimport as a library to import missing commits from many repositories simultaneously.
* Email to author is now added in the author field.
* Committer information is now also exported.
* Optimized the blob-store import by checking if a blob already exists prior to importing it.
* Added brief functions to basic hash structs, this is to get only the first 4 bytes (8 hex chars) for easier human inspection and debugging.
* Added support to suppress the long ref->BonzaiID mapping (on by default to match old behavior).
Reviewed By: krallin
Differential Revision: D25445974
fbshipit-source-id: 6dc7f977b61ceec1a95b5f3c38548ac8eddbea27
Summary:
Like it says in the title. This is helpful to measure the number of SQL queries
we make. This required actually threading in a CoreContext, which we didn't
have before.
Reviewed By: StanislavGlebik
Differential Revision: D25336069
fbshipit-source-id: 35677c55550e95b5126de29c2a824b4eda32092c
Summary:
Previously needed to pass in cachelib settings once to MononokeAppBuilder and once to parse_and_init_cachelib.
This change adds a MononokeClapApp and MononokeMatches that preserve the settings, thus preventing the need to pass them in twice (and thus avoiding possible inconsistency)
MononokeMatches uses MaybeOwned to hold the inner ArgMatches, which allows us to hold both the usual reference case from get_matches and an owned case for get_matches_from which is used in test cases.
Reviewed By: krallin
Differential Revision: D24788450
fbshipit-source-id: aad5fff2edda305177dcefa4b3a98ab99bc2d811
Summary:
Move expected_item_size_byte into CachelibSettings, seems like it should be there.
To enable its use also exposes a parse_and_init_cachelib method for callers that have different defaults to default cachelibe settings.
Reviewed By: krallin
Differential Revision: D24761024
fbshipit-source-id: 440082ab77b5b9f879c99b8f764e71bec68d270e
Summary:
It has a build() method and later in stack it will build a mononoke
specific type rather than the clap::App
Differential Revision: D25216827
fbshipit-source-id: 24a531856405a702e7fecf54d60be1ea3d2aa6e7
Summary:
- convert save_bonsai_changesets to new type futures
- `blobrepo:blobrepo` is free from old futures deps
Reviewed By: StanislavGlebik
Differential Revision: D25197060
fbshipit-source-id: 910bd3f9674094b56e1133d7799cefea56c84123
Summary: convert `BlobRepo::get_bonsai_bookmark` to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25188577
fbshipit-source-id: fb6f2b592b9e9f76736bc1af5fa5a08d12744b5f
Summary: We don't need to declare a fake empty version number
Reviewed By: farnz
Differential Revision: D24757981
fbshipit-source-id: 594c97e225704d783bea723efcbb9dfc4d5d800b
Summary:
Previously we were choosing current version, and just as with backsyncer this
is not always correct. Let's instead choose not the current version but the
version of the bookmark you are importing to.
This diff also introduced an integration test for a repo import into a pushredirected repo, and turned out there were a few bugs in the repo_import code (open_source_sql was used instead of open_sql). This diff fixed them as well
Reviewed By: ikostia
Differential Revision: D24651849
fbshipit-source-id: bfe36e005170ae2f49fa3a6cb208bf6d2c351298
Summary:
This diff changes semantic of `sync_commit()` function to return an error when
trying to sync a commit with no parents. This is a small code change which has big change
in semantics, and because of that I had to change how backsyncer and
mononoke_x_repo_sync job.
Instead of using `unsafe_sync_commit()/sync_commit()` functions both backsyncer and
`x_repo_sync_job` now use `unsafe_sync_commit_with_expected_version()`
which forces them to specify which version to use for commit with no parents.
And in order to find this version I changed find_toposorted_unsynced_ancestors
to not only return unsynced ancestors but also return mapping versions of their
(i.e. of unsynced ancestors) parents. Given this mapping we can figure out what
version is supposed to be used in `unsafe_sync_commit_with_expected_version()`.
The question raises what to do when a commit doesn't have any synced ancestor and hence we can't decide
which version to use to remap it. At the moment we are using current version (i.e. preserving the existing behaviour).
However this behaviour is incorrect, and so it will be changed in the next diffs
Reviewed By: ikostia
Differential Revision: D24617936
fbshipit-source-id: 6de26c50e4dde0d054ed2cba3508e6e8568f9222
Summary: This is to be able to see in Scuba why a given sync function was called.
Reviewed By: StanislavGlebik
Differential Revision: D24689366
fbshipit-source-id: f868fc1b6fcbf6c692e1373cbe8da8cd4a230780
Summary: The more we remove, the easier it'll be to remove the last few problem cases.
Reviewed By: StanislavGlebik
Differential Revision: D24592052
fbshipit-source-id: de6371305991d54bf2802fd904850b49aeb11bbd
Summary: This adds a `ScubaSampleBuilder` field to the `CommitSyncer` and ensures this field is instantiated with correct values (real vs discarding sample) depending on circumstances.
Reviewed By: StanislavGlebik
Differential Revision: D24539732
fbshipit-source-id: 37aedcff9aefcfcd6b740a0491ab35f9e5ce7c77
Summary:
Fewer clones, better code.
Note that in some cases we would previously have a fn that takes `ctx` by
ownership and just passes it through to some other fn outside of the
`cross_repo_sync`. I triead to make all such functions borrow and clone instead
in order to push cloning to the leaf fns of `cross_repo_sync`.
Reviewed By: StanislavGlebik
Differential Revision: D24538028
fbshipit-source-id: 8a3e78d4076b34d8b1767cdee1db3fdbb7acb4f7
Summary: In D24447404, I provided some utility functions that allowed me to avoid constructing and/or passing around a ConfigStore. Remove those functions and fix up the code to run.
Reviewed By: krallin
Differential Revision: D24502692
fbshipit-source-id: 742dbc54fbcf735895d6829745b9317af14dfa0b
Summary: It's designed as a singleton store for normal use - rather than have lots of ways to get different config stores, let's use one global store
Reviewed By: krallin
Differential Revision: D24447404
fbshipit-source-id: 6ecc46351b14794471f654cec98527a11a93d3ef
Summary:
- convert ManifestOps to new style futures
- at this point `//eden/manifest:manifest` crate is completely free from old style futures
Reviewed By: krallin
Differential Revision: D24502214
fbshipit-source-id: f1cdb11bd8234f22af5c905243f71e1e9fca11f1
Summary: Convert derived data utils to use new style futures
Reviewed By: StanislavGlebik
Differential Revision: D24331068
fbshipit-source-id: ad658b278802afa1e4ecd44c5a24164135748790
Summary: `clippy` often complains about the use of `.len() != 0`, `.len() > 0` or `.len() == 0`and proposes to use `.is_empty()` instead. This diff does that across Mononoke.
Reviewed By: aslpavel
Differential Revision: D24099427
fbshipit-source-id: 1bba2f958485b7efb3f41bf3eae820879c92b0e5
Summary:
TestLiveCommitSyncConfig is supposed to be a test replacement of
CfgrLiveCommitSyncConfig, however it was quite a different semantic. In
particular, it wasn't even possible to have two versions of the mapping for the
single repo.
This diff changes that. Now we'll have a method to add commit sync config
version, and mark/remove a version as current
Reviewed By: krallin
Differential Revision: D23951202
fbshipit-source-id: 242b4f088f67dac504544987e484cc290ee4e400
Summary:
get_mover() and get_reverse_mover() functions return the mover for the
"current" version of the commit sync config, which means these are movers for the version
of the config that's used to create the latest commits on master branch.
So this function returns correct mover only for the latest master commit, but
for all other commits it returns an incorrect mover! This is wrong and it
happened to work just by change, and that's why these functions are marked as deprecated
now, and later we'll add functions 'get_mover_by_version()' which could be used to
replace deprecated functions.
Note that the story for get_bookmark_renamer()/get_reverse_bookmark_renamer()
functions seems to be different. If we can always figure out what's the correct
mover for a commit by e.g. look at its parent we can't really do the same for
bookmarks. Because of that I suggest to keep using the current version for
get_bookmark_renamer() function.
Reviewed By: ikostia
Differential Revision: D23929582
fbshipit-source-id: 3e5e9b46224aca0b75cf2d981ea21c4f9a378ba9
Summary: Finally remove version_name from CommitSyncRepos. Note that this diff adds a few TODOs that we'd need to fix later.
Reviewed By: ikostia
Differential Revision: D23929010
fbshipit-source-id: c72130af548ac7b26bc20ddaac9a59562cc75e0b
Summary: Just as in the previous diff, but this time remove bookmark_renamers from CommitSyncRepos
Reviewed By: ikostia
Differential Revision: D23910295
fbshipit-source-id: 0c2d147057c8d3e0749d5b31ef98ab5022255d95
Summary:
This diff starts to use CommitSyncDataProvider introduced in the previous diff
and removes Mover from CommitSyncRepos struct.
Reviewed By: ikostia
Differential Revision: D23878683
fbshipit-source-id: 0d54f889781aebe4726b3388343a87df783c17d4
Summary:
A higher-level goal is to provide an interface for the manual remediation of
various xrepo-sync blockages. The nature of a candidate selection is such that
it may fail if the hint is not sufficient to decide which remapping changeset
is the best candidate. This is especially true about the `Only` hint variant:
it is designed to fail when there's more than one candidate. But even with
bookmark ancestorship hints, there are corner cases when the algorithm cannot
make a decision (as well as there may just be bugs in the algorithm). These
cases should be **extremely** rare. Nevertheless, we want to be able to unblock
ourselves. To do so, it is proposed to acccept parent selection hints in the `xrepo-lookup` scs
method. By default, it will use `Only` as a hint and be semantically equivalent
to the current behavior. But we'll provide CLI options to select other hints.
In order to make this work, we need the `sync_commit` method of the
`CommitSyncer` to accept hints too.
Reviewed By: StanislavGlebik
Differential Revision: D23913216
fbshipit-source-id: 05e1ff99cd2c6522829a6e8569040b226600af60
Summary:
CommitSyncer is a struct that we use to remap a commit from one repo to another. It uses commit sync map to figure out which paths need to be changed. Commit sync mapping might change, and each commit sync mapping has a version associated with it.
At the moment CommitSyncer doesn't work correctly if a commit sync mapping is changed. Consider the following DAG
```
large repo
A' <- remapped with mapping V1
|
O B' <- remapped with mapping V1
| /
...
small repo
A
|
O B
| /
...
```
We have commit A and B from a small repo remapped into a large repo into commits A' and B'. They were remapped with commit sync mapping V1, which for example remaps files in "dir/" into "smallrepo/dir".
Now let's say we start to use a new mapping v2 which remaps "dir/" into "otherdir/". After this point every commit will be created with new mapping. But this is incorrect - if we create a commit on top of B in a small repo that touches "dir/file.txt" then it will be remapped into "otherdir/file.txt" in the large repo, even though every other file is still in "smallrepo/dir"!
The fix for this issue is to always use the same mapping as commit parent was using (there are a few tricky cases with merge commits and commits with no parents, but those will be dealt with separately).
This diff is the first step - it threads through LiveCommitSyncConfig all the way to the CommitSyncer object, so that CommitSyncer can always fetch whatever mapping it needs.
Reviewed By: ikostia
Differential Revision: D23845720
fbshipit-source-id: 555cc31fd4ce09f0a6fa2869bfcee2c7cdfbcc61
Summary:
When running the repo import tool, it's possible that we need to do additional setup steps before being able to run the tool, which otherwise would only come up when we run it.
Firstly, if the repo we import into doesn't have a callsign (e.g. FBS, WWW...), but we want to check Phabricator, our tool would hang when checking Phabricator, because we need the callsign for checking. Therefore, we need to inform the user to set the callsign for the repo.
Secondly, in case the repo push-redirects to a larger repo, we generate a bookmark for the commits imported into the large. However, we need to inform the Phabricator team to include the large repo's bookmark before we can import the commits, because this bookmark publishes the imported commits on Phabricator.
This diff adds a subcommand to check these additional steps, so we wouldn't find these out during the actual import run.
Reviewed By: StanislavGlebik
Differential Revision: D23783462
fbshipit-source-id: 3cdf4035548213d8cee9717fb985c22741a6749b
Summary: Previously, we used the call sign of the repo we import when checking any of the if the commits are parsed by Phabricator. However, we also use this callsign for other repos when checking Phabricator, which is an incorrect implementation. E.g. if fbsource back-syncs to ovsource, we would have used FBS callsign when checking Phabricator for both fbsource and ovrsource, but we should use OVRSOURCE callsign for repo ovrsource. This diff corrects this implementation by saving the callsigns of the small repos in their SmallRepoBackSyncVars.
Reviewed By: StanislavGlebik
Differential Revision: D23758355
fbshipit-source-id: b322acb2ec589eabed5362bfd6b963e2dd1d6ea9
Summary: Since repo_import tool is automated, we need a way to recover the process when the tool break without restarting the whole process. To do this, I defined a new struct (RecoveryFields) that allows us to keep track of the state. The most important fields are the import_stage (ImportStage) we need for keeping track of the process and to indicate the first stage of recovery, and the cs_ids we use throughout the process. For each stage in importing, we save the state after we have finished that part. This way we can also recover from interrupts. To do process recovery, we only need to use `recover-process <file_path>` subcommand, where file_path stores the saved state of importing. For normal run we will use `import [<args>]` subcommand.
Reviewed By: krallin
Differential Revision: D23678367
fbshipit-source-id: c0e0b270ea2ccc499368e54f37550cfa58c03970
Summary:
This is the function that was used in repo_import tool to wait until hg sync
has processed all of the entries in the queue. Let's move it to the hg sync
helper lib so that it can be used in other places. E.g. I'd like to use it in
the next diffs in mononoke_x_repo_sync_job.
Reviewed By: krallin
Differential Revision: D23708280
fbshipit-source-id: ea846081d89b55b0d2f5407c971e13869cedfd8b
