Summary:
This scenario appears when master moves backwards. Since the master group in
segmented changelog is append-only. Non-fast-forward master move will cause
multiple heads in the master group.
Since Segmented Changelog was updated to handle multiple master heads, we can
propagate the full list that we get from the client.
This diff makes the assumption that Mononoke will know to convert all client "master head"
hashes from HgChangesetId (Sha1) form to ChangesetId (Blake2). If any of the master
heads cannot be converted then it means the server might not be reliably answer the
client's question (in "ancestors(master_heads)", translate "this hash" to a path, or tell me
confidently that the "hash" is outside "ancestors(master_heads)"). That's an error case.
Reviewed By: quark-zju
Differential Revision: D27521779
fbshipit-source-id: 219e08a66aac17ac06d2cf02676a43c7f37e8e26
Summary:
This scenario appears when master moves backwards.
Since the IdDag can handle multiple master heads, the server can piggy-back on that
functionality and support multiple master heads when translating location to hash.
Reviewed By: quark-zju
Differential Revision: D27521780
fbshipit-source-id: c27541890d4fda13648857f010c11a25bf96ef67
Summary:
Switch from `blobrepo_factory` to the new `RepoFactory` to construct `BlobRepo`
in `mononoke_api`.
The factory is part of the `MononokeEnvironment`, and is used to build all of the repos.
Reviewed By: krallin
Differential Revision: D27363473
fbshipit-source-id: 81345969e5899467f01d285c232a510b8edddb17
Summary:
When creating a commit via scs api we need to validate a few things (e.g. that
the file that a commit is trying to delete existed in the parent), and in order
to do that we need to use a manifest. Previously we were using fsnodes
manifests, however fsnodes is the slowest manifest to derive, and that makes
the whole create_commit operation much slower. Let's try to use skeleton
manifest, which is the fastest to derive.
Reviewed By: markbt
Differential Revision: D27587664
fbshipit-source-id: a60cab4956063bf26c0f1ec8c9cfa05233bb3cc0
Summary:
Previously ChangesetPathContext was holding both fsnode_id and unode_id, however it made it easier to misuse api and trigger expensive fsnodes or unodes path traversal (see more info in the comments for D27587664).
This diff splits it in two separate types.
Also I noticed that ChangesetPathContext wasn't using the `unode_id` future that it stored, so I just deleted it.
Reviewed By: markbt
Differential Revision: D27590997
fbshipit-source-id: 08fc14d33c82357275413c4cf2698f97620503ea
Summary:
This warmer hasn't been used, and we aren't going to use it going further.
Let's remove it
Reviewed By: krallin
Differential Revision: D27533802
fbshipit-source-id: deaaf954ed535789ab6b5dc89b8da9967c40e84f
Summary:
The comment seem to be incorrect - it wasn't checking that all files from a
replaced directory were removed (in fact, we allow to not remove them, see
https://fburl.com/diffusion/wdhu5arg).
Instead it was checking whether a file that was replaced with a directory was indeed
removed.
Reviewed By: markbt
Differential Revision: D27534121
fbshipit-source-id: 4f0d53096c2665fec8bb575ee183411c8edf2ccb
Summary: Use the test factory for existing mononoke_api tests.
Reviewed By: StanislavGlebik
Differential Revision: D27169444
fbshipit-source-id: d940bfa494dfe89fdb891d5248b73ba764f74faf
Summary:
I'd like to add support for Range queries in LFS. In HTTP, the range header
isn't start + size, it's start / end, so I'll need a slightly lower API. This
makes that possible.
Note: Range should have a private constructor, because otherwise we'll get
errors if start > end. Hence the `Range` / `RangeInner` here.
Reviewed By: StanislavGlebik
Differential Revision: D27187817
fbshipit-source-id: 20183d8b200fef6c16e45a66886ce8886d92e4f6
Summary:
In preparation for making `BlobRepo` buildable by facet factories, restore
`BlobRepo` members that had been converted to `TypeMap` attributes back into
real members.
This re-introduces some dependencies that were previously removed, but this
will be cleaned up when crates no longer have to depend on BlobRepo directly,
just the traits they are interested in.
Reviewed By: ahornby
Differential Revision: D27169422
fbshipit-source-id: 14354e6d984dfdd2be5c169f527e5f998f00db1e
Summary:
Currently we can only limit which users are allowed to move a bookmark by a
regex. We also want to allow specifying a hipster group.
Reviewed By: krallin
Differential Revision: D27156690
fbshipit-source-id: 99a5678a82f4c34ed2e57625361ba7cdb08ed839
Summary: Add the EdenAPI endpoint for resolving bookmarks. This is a first pass that just takes a bookmark name as a path variable, to make sure that this is on the right track. We'll want to add a proper request type that includes a list of bookmarks and a response type that can indicate that no bookmark was found. Then the hg bookmark command will also need support for prefix listing capabilities.
Reviewed By: kulshrax
Differential Revision: D26920845
fbshipit-source-id: 067db6a636a75531ee5953392b734c038a58efb6
Summary:
BonsaiChangesets are rarely mutated, and their maps are stored in sorted order,
so we can use `SortedVectorMap` to load them more efficiently.
In the cases where mutable maps of filechanges are needed, we can use `BTreeMap`
during the mutation and then convert them to `SortedVectorMap` to store them.
Reviewed By: mitrandir77
Differential Revision: D25615279
fbshipit-source-id: 796219c1130df5cb025952bb61002e8d2ae898f4
Summary:
AsyncVfs provides async vfs interface.
It will be used in the native checkout instead of current use case that spawns blocking tokio tasks for VFS action
Reviewed By: quark-zju
Differential Revision: D26801250
fbshipit-source-id: bb26c4fc8acac82f4b55bb3f2f3964a6d0b64014
Summary:
This diffs add a layer of indirection between fbinit and tokio, thus allowing
us to use fbinit with tokio 0.2 or tokio 1.x.
The way this works is that you specify the Tokio you want by adding it as an
extra dependency alongside `fbinit` in your `TARGETS` (before this, you had to
always include `tokio-02`).
If you use `fbinit-tokio`, then `#[fbinit::main]` and `#[fbinit::test]` get you
a Tokio 1.x runtime, whereas if you use `fbinit-tokio-02`, you get a Tokio 0.2
runtime.
This diff is big, because it needs to change all the TARGETS that reference
this in the same diff that introduces the mechanism. I also didn't produce it
by hand.
Instead, I scripted the transformation using this script: P242773846
I then ran it using:
```
{ hg grep -l "fbinit::test"; hg grep -l "fbinit::main" } | \
sort | \
uniq | \
xargs ~/codemod/codemod.py \
&& yes | arc lint \
&& common/rust/cargo_from_buck/bin/autocargo
```
Finally, I grabbed the files returned by `hg grep`, then fed them to:
```
arc lint-rust --paths-from ~/files2 --apply-patches --take RUSTFIXDEPS
```
(I had to modify the file list a bit: notably I removed stuff from scripts/ because
some of that causes Buck to crash when running lint-rust, and I also had to add
fbcode/ as a prefix everywhere).
Reviewed By: mitrandir77
Differential Revision: D26754757
fbshipit-source-id: 326b1c4efc9a57ea89db9b1d390677bcd2ab985e
Summary:
For public bookmarks, we can avoid querying the database and instead serve
`list_bookmarks` from the warm bookmarks cache. The listed bookmarks might
be slightly old, but only because derived data is still in progress, and so
listing the older bookmark value is a better choice.
The tests now need a way to make sure that the warm bookmark cache is
up-to-date, so add a `sync` method that waits for a complete cycle of the warm
bookmark cache update thread.
Reviewed By: StanislavGlebik
Differential Revision: D26693444
fbshipit-source-id: 7145964bdb8c22d98ab6a2bb8c5091c19addd03e
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary: Small clean up. Allows us to pass Logger by reference, removing the FIXME in blobrepo factory
Reviewed By: farnz
Differential Revision: D26551592
fbshipit-source-id: d6bb04b8bb3034ad056f071b67b5ae0ce3c6f224
Summary:
The earlier diffs in this stack have removed all our dependencies on the Tokio
0.1 runtime environment (so, basically, `tokio-executor` and `tokio-timer`), so
we don't need this anymore.
We do still have some deps on `tokio-io`, but this is just traits + helpers,
so this doesn't actually prevent us from removing the 0.1 runtime!
Note that we still have a few transitive dependencies on Tokio 0.1:
- async-unit uses tokio-compat
- hg depends on tokio-compat too, and we depend on it in tests
This isn't the end of the world though, we can live with that :)
Reviewed By: ahornby
Differential Revision: D26544410
fbshipit-source-id: 24789be2402c3f48220dcaad110e8246ef02ecd8
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary: makes it a bit easier to work with
Reviewed By: farnz
Differential Revision: D26497840
fbshipit-source-id: bccd248aa9a4caa4e5274f7f9e166ef17196e12f
Summary: just some boilerplate to make the use from SCS code clean and easy
Reviewed By: StanislavGlebik
Differential Revision: D26453406
fbshipit-source-id: efc280e0be14c3e0f145a489a9fd77b5c0eefa58
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary:
One of the primary use cases for hash_to_location is translating user provided
hashes. It is then perfectly valid for the hashes that are provided to not
exist. Where we would previously return an error for the full request if a
hash was invalid, we now omit the hash from the response.
Reviewed By: quark-zju
Differential Revision: D26389472
fbshipit-source-id: c59529d43f44bed7cdb2af0e9babc96160e0c4a7
Summary:
Exposes Segmented Changelog changeset to location translation functionality.
This is going to be used by EdenApi endpoints. Those endpoints will be used
by "lazy" Segmented Changelogs on client repositories.
Reviewed By: quark-zju
Differential Revision: D26289620
fbshipit-source-id: 6fb78abab465efd34e2768c8648f438647e9b91c
Summary:
We had individual fields for location components. This change will make it
easier for people to read through the code. This completes the integration
of the Location struct.
Reviewed By: quark-zju
Differential Revision: D26162272
fbshipit-source-id: 76259578430bac88317afb1935f63e06b6e8284e
Summary:
This is a preparation for potential necessity of IO being done by this trait and its implementors.
We think the IO might be needed if we move commit sync config storage from `Configerator` into xdb, or some place else. To be clear, I personally am not certain we'll *need* this, but in any case, asyncifying the trait does not seem like a risky thing here (because we usually have only 0-2 sync functions in the stack above `LiveCommitSyncConfig` accessors, so it does not require large-scale code flow changes or anything).
I intentionally did not touch the push-redirection accessors, as those I don't think will ever move away from configerator.
Reviewed By: StanislavGlebik
Differential Revision: D26275905
fbshipit-source-id: 1bfdca087434d475d50032dd47dd94f16be051f9
Summary:
Like it says in the title. This adds a crate that provides a combinator that
lets us easily find stalls caused by futures that stay in `poll()` for too
long.
The goal is to make this minimal overhead for whoever is using it: all you need
is to import it + give it a logger. It automatically looks up the line where
it's called and gives it back to you in logs. This uses the `track_caller`
functionality to make this work.
Reviewed By: farnz
Differential Revision: D26250068
fbshipit-source-id: a1458e5adebac7eab6c2de458f679c7215147937
Summary:
Like it says in the title, this updates our repo construction to rely on
Mononoke API. My underlying goal here is to have a Mononoke instance around so
that I can start EdenAPI on it, but it also allows for a bunch of cleanup &
code deduplication.
There is still some stuff that isn't initialized in Mononoke API and probably
does not belong there, but at least the shared pieces now come from there. I
also did keep the `Arc<Repo>` around in Mononoke Server's `MononokeRepo`, so
this way we can start to migrate things to Mononoke API (instead of
de-constructing my `Repo` and getting the parts I need to stuff them into
`MononokeRepo`).
One part of this that might be a bit controversial is that I exposed some of
the internals of `Repo` via accessor methods. I know we've historically
wanted access via Mononoke API to not use the fields but instead use the
RepoContext, and I think that's a good goal, but (IMO) realistically the only
way we get there is by first making Mononoke API *available* to use in
repo_client (which is what this ends up doing), and then we can port things to
call Mononoke API instead of using blobrepo and such directly.
To make this work properly I also updated our tests to default to always
set up Configerator configs when starting Mononoke, since we need them to start
MononokeApi (for the CfgrLiveCommitSyncConfig, which right now has an ad-hoc
"ignore the failures in test mode" branch in Mononoke Server).
Reviewed By: markbt
Differential Revision: D26108443
fbshipit-source-id: b7cf5452e044828e73a0aa3ca3ddbc78e466fe57
Summary:
Right now, if you enable Mononoke API, you always get a WBC, for all derived
data kinds, and with a delay. This isn't great for a few reasons:
- Some places don't really care about bookmarks being warm to being with. This
is the case in the bookmarks filler (note that right now, it still does
polling to satisfy the WBC API, it's just not "warm").
- Some places don't want a delay, or don't want all kinds. This is the case for
Mononoke Server (which doesn't use Mononoke API right now, but that's what
I'm working towards), or EdenAPI, which uses a WBC sort-of-de-facto but
doesn't really care (but likely will in the future, and will want to follow
Mononoke Server's behavior).
As of this diff, we can now configure this when initializing `Mononoke`. I also
split out all those args into a `MononokeEnvironment` because the args list
was getting out of hand. One thing I did not do is make a way to instantiate
`MononokeEnvironment` from args (but we should do it at some point in the
future).
Reviewed By: StanislavGlebik
Differential Revision: D26100706
fbshipit-source-id: 1daa6335f3ce2b297929a84788bc5b4d9ad6432f
Summary:
I'd like to be able to use mononoke_api from repo_client, but the hg parts
depend on repo_client, so that's a circular dependency.
This splits out the hg parts of Mononoke API so that places that don't want
them don't have to import them. This is similar to what we did with blobrepo.
Reviewed By: StanislavGlebik
Differential Revision: D26099495
fbshipit-source-id: 73a9c7b5dc95feceb35b5eabccf697e9aa0a27de
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.
This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).
The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.
Reviewed By: farnz
Differential Revision: D25952428
fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
Summary: Convert `Changsets` trait and all its uses to new type futures
Reviewed By: krallin
Differential Revision: D25638875
fbshipit-source-id: 947423e2ee47a463861678b146641bcc6b899a4a
Summary: This is in preparation for all walker hg to hg steps (e.g HgChangeset to Parent HgChangeset) going via Bonsai, which without this would continually check if the filenodes are derived
Reviewed By: krallin
Differential Revision: D25394086
fbshipit-source-id: bb75e7ddf5b09f9d13a0f436627f4c3c95e24430
Summary:
In the next diff I'm going to add Mysql connection object to `MysqlOptions` in order to pass it down from `MononokeAppData` to the code that works with sql.
This change will make MysqlOptions un-copyable.
This diff fixed all issues produced by the change.
Reviewed By: ahornby
Differential Revision: D25590772
fbshipit-source-id: 440ae5cba3d49ee6ccd2ff39a93829bcd14bb3f1
Summary:
Like it says in the title. This is nice to do because we had old futures
wrapping new futures here, so this lets us get rid of a lot of cruft.
Reviewed By: ahornby
Differential Revision: D25502648
fbshipit-source-id: a34973b32880d859b25dcb6dc455c42eec4c2f94
Summary: Convert all BlobRepoHg methods to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25471540
fbshipit-source-id: c8e99509d39d0e081d082097cbd9dbfca431637e
Summary:
The goal of this stack is to start logging commits to scribe even if a commit was
introduced by scs create_commit/move_bookmark api. Currently we don't do that.
Initially I had bigger plans and I wanted to log to scribe only from bookmarks_movement and remove scribe logging from unbundle/processing.rs, but it turned out to be trickier to implement. In general, the approach we use right now where in order to log to scribe we need to put `log_commit_to_scribe` call in all the places that can possibly create commits/move bookmarks seems wrong, but changing it is a bit hard. So for now I decided to solve the main problem we have, which is the fact that we don't get scribe logs from repos where bookmarks is moved via scs methods.
To fix that I added an additional option to CreateBookmark/UpdateBookmark structs. If this option is set to true then before moving/creating the bookmark it finds all draft commits that are going to be made public by the bookmark move i.e. all draft ancestors of new bookmark destinationl. This is unfortunate that we have to do this traversal on the critical path of the move_bookmark call, but in practice I hope it won't be too bad since we do similar traversal to record bonsai<->git mappings. In case my hopes are wrong we have scuba logging which should make it clear that this is an expensive operation and also we have a tunable to disable this behavioiur.
Also note that we don't use PushParams::commit_scribe_category. This is intentional - PushParams::commit_scribe_category doesn't seem useful at all, and I plan to deprecate it later in the stack. Even later it would make sense to deprecate PushrebaseParams::commit_scribe_category as well, and put commit_scribe_category optoin in another place in the config.
Reviewed By: markbt
Differential Revision: D25558248
fbshipit-source-id: f7dedea8d6f72ad40c006693d4f1a265977f843f
Summary:
Change derived data config to have "enabled" config and "backfilling" config.
The `Mapping` object has the responsibility of encapsulating the configuration options
for the derived data type. Since it is only possible to obtain a `Mapping` from
appropriate configuration, ownership of a `Mapping` means derivation is permitted,
and so the `DeriveMode` enum is removed.
Most callers will use `BonsaiDerived::derive`, or a default `derived_data_utils` implementation
that requires the derived data to be enabled and configured on the repo.
Backfillers can additionally use `derived_data_utils_for_backfill` which will use the
`backfilling` configuration in preference to the default configuration.
Reviewed By: ahornby
Differential Revision: D25246317
fbshipit-source-id: 352fe6509572409bc3338dd43d157f34c73b9eac
Summary:
The `BonsaiDerived` trait is split in two:
* The new `BonsaiDerivable` trait encapsulates the process of deriving the data, either
a single item from its parents, or a batch.
* The `BonsaiDerived` trait is used only as an entry point for deriving with the default
mapping and config.
This split will allow us to use `BonsaiDerivable` in batch backfilling with non-default
config, for example when backfilling a new version of a derived data type.
Reviewed By: krallin
Differential Revision: D25371964
fbshipit-source-id: 5874836bc06c18db306ada947a690658bf89723c
Summary:
This is useful to e.g. write tests in things that use mononoke_api (such as
edenapi): the test mode isn't transitive across crates. This also requires
making Repo itself public, since callers might reasonably want to create one.
I've also updated a few of the accessor methods that were `pub(crate)` given
that what we had right now seemed like it was kinda random: some things were
`pub(crate)`, others were just `pub`.
Reviewed By: markbt
Differential Revision: D25467624
fbshipit-source-id: 2279d4196e8dc0e7e1729239710d900b351be816
Summary:
The end goal is to have clients using a sparse IdMap. There is still some work
to get there though. In the mean time we can test repositories that don't use
any revlogs. The current expections for those repositories are that they have
a full idmap locally.
Reviewed By: quark-zju
Differential Revision: D25075341
fbshipit-source-id: 52ab881fc9c64d0d13944e9619c087e0d4fb547c
