Summary:
AsyncVfs provides async vfs interface.
It will be used in the native checkout instead of current use case that spawns blocking tokio tasks for VFS action
Reviewed By: quark-zju
Differential Revision: D26801250
fbshipit-source-id: bb26c4fc8acac82f4b55bb3f2f3964a6d0b64014
Summary:
This diffs add a layer of indirection between fbinit and tokio, thus allowing
us to use fbinit with tokio 0.2 or tokio 1.x.
The way this works is that you specify the Tokio you want by adding it as an
extra dependency alongside `fbinit` in your `TARGETS` (before this, you had to
always include `tokio-02`).
If you use `fbinit-tokio`, then `#[fbinit::main]` and `#[fbinit::test]` get you
a Tokio 1.x runtime, whereas if you use `fbinit-tokio-02`, you get a Tokio 0.2
runtime.
This diff is big, because it needs to change all the TARGETS that reference
this in the same diff that introduces the mechanism. I also didn't produce it
by hand.
Instead, I scripted the transformation using this script: P242773846
I then ran it using:
```
{ hg grep -l "fbinit::test"; hg grep -l "fbinit::main" } | \
sort | \
uniq | \
xargs ~/codemod/codemod.py \
&& yes | arc lint \
&& common/rust/cargo_from_buck/bin/autocargo
```
Finally, I grabbed the files returned by `hg grep`, then fed them to:
```
arc lint-rust --paths-from ~/files2 --apply-patches --take RUSTFIXDEPS
```
(I had to modify the file list a bit: notably I removed stuff from scripts/ because
some of that causes Buck to crash when running lint-rust, and I also had to add
fbcode/ as a prefix everywhere).
Reviewed By: mitrandir77
Differential Revision: D26754757
fbshipit-source-id: 326b1c4efc9a57ea89db9b1d390677bcd2ab985e
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary:
The earlier diffs in this stack have removed all our dependencies on the Tokio
0.1 runtime environment (so, basically, `tokio-executor` and `tokio-timer`), so
we don't need this anymore.
We do still have some deps on `tokio-io`, but this is just traits + helpers,
so this doesn't actually prevent us from removing the 0.1 runtime!
Note that we still have a few transitive dependencies on Tokio 0.1:
- async-unit uses tokio-compat
- hg depends on tokio-compat too, and we depend on it in tests
This isn't the end of the world though, we can live with that :)
Reviewed By: ahornby
Differential Revision: D26544410
fbshipit-source-id: 24789be2402c3f48220dcaad110e8246ef02ecd8
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary:
On setup SCS initializes the repos concurrently, warming up derived data for each repo, warming bookmark cache and fetching skiplists.
Fetching skiplists is an expensive operation and includes two steps: async get a large blob from the Blobstore and then sync deserialization of the blob.
While running on the same task as warming the bookmark cache, it takes all CPU and the other futures have to wait and can't process results returned by MySQL queries or connect to the DB. Thus SCS eventually fail to acquire a new connection or to perform a query in a reasonable time and terminates.
Spawning skiplists in a separate task helps to unlock the thread where the warm is running.
This was first noticed in TW tasks because after the MySQL rollout some of the SCS tasks started to take an hour to start.
To debug this and localize the issue, we put debug output to see what exactly blocks the repo initialization and, turned out it, when skiplists fetching started the rest was blocked.
Reviewed By: StanislavGlebik
Differential Revision: D26128171
fbshipit-source-id: fe9e1882af898950cf16d8e939dc6bc6be56510e
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.
This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).
The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.
Reviewed By: farnz
Differential Revision: D25952428
fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
Summary:
This feature is useful for testing time-dependent stuff (e.g. it
allows you to stop/forward time). It's already included in the buck build.
Reviewed By: SkyterX
Differential Revision: D25946732
fbshipit-source-id: 5e7b69967a45e6deaddaac34ba78b42d2f2ad90e
Summary:
Using `backfill-all` on very large repositories is slow to get started and slow
to resume, as it must traverse the repository history all the way to the start
before it can even begin.
Make this more usable by using the skiplist index to slice the repository into
reasonably sized slices of heads with the same range of generation numbers.
Each slice is then derived in turn. If interrupted, derivation can continue
at the next slice more quickly.
Reviewed By: krallin
Differential Revision: D25371968
fbshipit-source-id: f150ea847f9fbbe84852587d620ae37ba2c58f28
Summary: Convert `ChangesetFetcher` to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25244213
fbshipit-source-id: 4207386d81397a930a566db008019bb8f31bf602
Summary: convert `BlobRepo::get_bonsai_bookmark` to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25188577
fbshipit-source-id: fb6f2b592b9e9f76736bc1af5fa5a08d12744b5f
Summary: Remove 'static requirement for async methods of Blobstore, propagate this change and fixup low hanging fruits where the code can become 'static free easily.
Reviewed By: ahornby, farnz
Differential Revision: D24839054
fbshipit-source-id: 5d5daa04c23c4c9ae902b669b0a71fe41ee6dee6
Summary: As part of the effort to deprecate futures 0.1 in favor of 0.3 I want to create a new futures_ext crate that will contain some of the extensions that are applicable from the futures_01_ext. But first I need to reclame this crate name by renaming the old futures_ext crate. This will also make it easier to track which parts of codebase still use the old futures.
Reviewed By: farnz
Differential Revision: D24725776
fbshipit-source-id: 3574d2a0790f8212f6fad4106655cd41836ff74d
Summary:
Generated by formatting with rustfmt 2.0.0-rc.2 and then a second time with fbsource's current rustfmt (1.4.14).
This results in formatting for which rustfmt 1.4 is idempotent but is closer to the style of rustfmt 2.0, reducing the amount of code that will need to change atomically in that upgrade.
---
*Why now?* **:** The 1.x branch is no longer being developed and fixes like https://github.com/rust-lang/rustfmt/issues/4159 (which we need in fbcode) only land to the 2.0 branch.
---
Reviewed By: StanislavGlebik
Differential Revision: D23568780
fbshipit-source-id: b4b4a0aa683d236e2fdeb5b96d723ac2d84b9faf
Summary:
Currently our skiplists store a skip edge for almost all public commits. This
is problematic for a few reasons:
1) It uses more memory
2) It increases the startup time
3) It makes startup flakier. We've noticed a few times that our backend storage
return errors more often when try to download large blobs.
Let's change the way we build skiplist. Let's not index every public changeset
we have, but rather index it smarter. See comments for more details.
Reviewed By: farnz
Differential Revision: D22500300
fbshipit-source-id: 7e9c887595ba11da80233767dad4ec177d933f72
Summary:
This new skiplist feature allows to find merges between any two related
commits.
Reviewed By: StanislavGlebik
Differential Revision: D22457894
fbshipit-source-id: 203d43588040759b89a895395058a21c9b5ca43d
Summary: I'm planning to use it in my next diff to power `find_merges` functionality.
Reviewed By: StanislavGlebik
Differential Revision: D22457898
fbshipit-source-id: 76c3f107fd8b5bbef96e978037be31efca0f9841
Summary:
The `process_frontier` function is THE function that powers skiplist traversal
and it's quite complex. To make the core more readable I'm moving part of the
code into separate functions.
Differential Revision: D22457896
fbshipit-source-id: e3521855ae7ab889c21d7aff0204e27dc23cf906
Summary:
The `process_frontier` function is THE function that powers skiplist traversal
and it's quite complex. To make the core more readable I'm moving parts of the
code into separate functions.
I'm also planning to use the single step function to simplify lowest common
ancestor algorithm later.
Differential Revision: D22457895
fbshipit-source-id: 1234118705ca6b1b61e09fdd7867ce4366045a28
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.
Reviewed By: dtolnay
Differential Revision: D22403809
fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
Summary:
Eventually, we want everything to be `async`/`await`; as a stepping stone in that direction, switch the remaining lobstore traits to new-style futures.
This just pushes the `.compat()` out to old-style futures, but it makes the move to non-'static lifetimes easier, as all the compile errors will relate to lifetime issues.
Reviewed By: krallin
Differential Revision: D22183228
fbshipit-source-id: 3fe3977f4469626f55cbf5636d17fff905039827
Summary:
During expensive getbundle request traversing skiplist uses a lot of cpu, and
in fact it's blocking the whole CPU. krallin suggested to yield since it
should avoid blocking cpus, and that's what this diff is doing.
Reviewed By: krallin
Differential Revision: D22160477
fbshipit-source-id: 5bd881d5c50f4d8e64f2cc90389abc8568ec1df6
Summary:
Tooling can't handle named_deps yet, but it can warn about them
P133451794
Reviewed By: StanislavGlebik
Differential Revision: D22083499
fbshipit-source-id: 46de533c19b13b2469e912165c1577ddb63d15cd
Summary:
Remove unused dependencies for Rust targets.
This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.
Manual edits (named_deps) and misc output in P133451794
Reviewed By: dtolnay
Differential Revision: D22083498
fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
Summary: this is the last diff to complete the `reachabilityindex` module migration
Reviewed By: farnz
Differential Revision: D22049853
fbshipit-source-id: d8510f2d1b1c12a1ba903c2afda37096aadd8347
Summary:
After being trimmed down to store only a single edge the skiplist update
algoritm was working incorrectly. This diff introduces a way of fixing things
up.
Reviewed By: StanislavGlebik
Differential Revision: D21976995
fbshipit-source-id: 3bc4678011b332419d9f6f7b983a33fa066674bf
Summary:
In production we trim skiplists to hold a single entry which causes
the update algorithm to behave incorrectly
Reviewed By: StanislavGlebik
Differential Revision: D21976996
fbshipit-source-id: 4946be11512ac9577bdd3320913fe092d513ecdd
Summary: incremental step on the path to modernization of this module
Reviewed By: StanislavGlebik
Differential Revision: D21937310
fbshipit-source-id: 6ae5f9eb43355efee35705cce96736bfcb19c8ca
Summary:
NOTE: this diff doesn't change the public API of the trait - that's why we do
`compat()` and `boxify()`. This change will follow later in the stack.
Reviewed By: StanislavGlebik
Differential Revision: D21937308
fbshipit-source-id: 11bef4c5087a9ac06e209685d3281483b804c394
Summary:
I'm not sure why but D14405696 added this `is_ancestor` method with a
duplicate ancestry check implementation which:
* has different behaviour when `ancestor == descendant`
* doesn't log to ODS
Let's just settle on using one implementation now.
Reviewed By: StanislavGlebik
Differential Revision: D21937312
fbshipit-source-id: 0f4e2390cb0b984f92cd4fe645d9313fa4bba70e
Summary: incremental step on the path to modernization of this module
Reviewed By: StanislavGlebik
Differential Revision: D21937311
fbshipit-source-id: 9db84c7cb166084edfc5a0638c1c884964f9984c
Summary: incremental step on the path to modernization of this module
Reviewed By: StanislavGlebik
Differential Revision: D21937309
fbshipit-source-id: 61b1b4302782cea35d3f65e6fe982f530293dd81
Summary:
We need a lowest common ancestor operation for `merge_base` call
implementation.
The `lca` function returns only one of many possible ancestors in some
situation (multiple merges of the same branches). This is done to:
simplify the implementation and hide the complexity from our API users
(people don't really expect to receive more than one node from such APIs).
Mercurial arbitrarily chooses the commit with smallest hash in those situations.
Reviewed By: StanislavGlebik
Differential Revision: D21840081
fbshipit-source-id: 2dfc95a4cf549d8941fc5166e878bfee4b6b2ece
Summary:
- Change get return value for `Blobstore` from `BlobstoreBytes` to `BlobstoreGetData` which include `ctime` metadata
- Update the call sites and tests broken due to this change
- Change `ScrubHandler::on_repair` to accept metadata and log ctime
- `Fileblob` and `Manifoldblob` attach the ctime metadata
- Tests for fileblob in `mononoke:blobstore-test` and integration test `test-walker-scrub-blobstore.t`
- Make cachelib based caching use `BlobstoreGetData`
Reviewed By: ahornby
Differential Revision: D21094023
fbshipit-source-id: dc597e888eac2098c0e50d06e80ee180b4f3e069
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/
This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.
This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.
Codemod performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```
Reviewed By: k21
Differential Revision: D20213432
fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
Summary:
In targets that depend on *both* 0.1 and 0.3 futures, this codemod renames the 0.1 dependency to be exposed as futures_old::. This is in preparation for flipping the 0.3 dependencies from futures_preview:: to plain futures::.
rs changes performed by:
```
rg \
--files-with-matches \
--type-add buck:TARGETS \
--type buck \
--glob '!/experimental' \
--regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
-x \
buck query "labels(srcs,
rdeps(%Ss, fbsource//third-party/rust:futures-old, 1)
intersect
rdeps(%Ss, //common/rust/renamed:futures-preview, 1)
)" \
| xargs sed -i 's/\bfutures::/futures_old::/'
```
Reviewed By: jsgf
Differential Revision: D20168958
fbshipit-source-id: d2c099f9170c427e542975bc22fd96138a7725b0