Summary:
This won't be very long lived, as I'll remove the 0.2 version once Mononoke is
updated (we are the only users), and stop using a fork as soon as Gotham
releases. However, for now, I need both versions in order to avoid to do it all
in 1 diff.
A couple things worth noting here:
- See the previous diff for why I am using 2 Git repos for this.
- I can't easily use 2 forks for this because there are some extra crates they
vendor, notably `borrow_bag`. I removed it from both forks and pulled it from
crates.io instead.
Reviewed By: mitrandir77
Differential Revision: D26781291
fbshipit-source-id: 0b9824c07b880dbd5b94cd6d62d2cb2a68e348e9
Summary:
There is a bug in Hyper with 101 Continue responses: it sends a Content-Length.
This makes Proxygen unhappy with websocket upgrades. We used to have this
patched in hyper-02, but since Mononoke is about to update to Tokio 1.x, we
also need in the matching Hyper.
One thing that's a bit awkward is you might notice I changed the fork where the
patch comes from. This is because `cargo vendor` cannot deal with 2 overriding
sources coming from the same upstream. I posted about my adventures in this
here: https://fb.workplace.com/groups/rust.language/permalink/5278514098863829/
Reviewed By: HarveyHunt
Differential Revision: D26780899
fbshipit-source-id: e775b7151427898d63d8767acaa53f5f68229db6
Summary:
This diffs add a layer of indirection between fbinit and tokio, thus allowing
us to use fbinit with tokio 0.2 or tokio 1.x.
The way this works is that you specify the Tokio you want by adding it as an
extra dependency alongside `fbinit` in your `TARGETS` (before this, you had to
always include `tokio-02`).
If you use `fbinit-tokio`, then `#[fbinit::main]` and `#[fbinit::test]` get you
a Tokio 1.x runtime, whereas if you use `fbinit-tokio-02`, you get a Tokio 0.2
runtime.
This diff is big, because it needs to change all the TARGETS that reference
this in the same diff that introduces the mechanism. I also didn't produce it
by hand.
Instead, I scripted the transformation using this script: P242773846
I then ran it using:
```
{ hg grep -l "fbinit::test"; hg grep -l "fbinit::main" } | \
sort | \
uniq | \
xargs ~/codemod/codemod.py \
&& yes | arc lint \
&& common/rust/cargo_from_buck/bin/autocargo
```
Finally, I grabbed the files returned by `hg grep`, then fed them to:
```
arc lint-rust --paths-from ~/files2 --apply-patches --take RUSTFIXDEPS
```
(I had to modify the file list a bit: notably I removed stuff from scripts/ because
some of that causes Buck to crash when running lint-rust, and I also had to add
fbcode/ as a prefix everywhere).
Reviewed By: mitrandir77
Differential Revision: D26754757
fbshipit-source-id: 326b1c4efc9a57ea89db9b1d390677bcd2ab985e
Summary:
This introduces a couple "trampoline" crates to use alongside fbinit in order
to let callers choose which version of Tokio they want by selecting one or the
other and adding it to their `TARGETS` alongside `fbinit` (right now, they
need to include `tokio` there).
The entrypoints here will be called by the expansion of the `#[fbinit::main]`
and `#[fbinit::test]` macros for `async fn`-s.
Right now, this isn't wired up: that happens in the next diff. In this diff,
I'm just adding the two entrypoints.
Reviewed By: mitrandir77
Differential Revision: D26754751
fbshipit-source-id: 1966dadf8bbe427ce4a1e90559a81790d8e56e7a
Summary:
This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here.
Arc lint support:
V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff:
- Adds running `autocargo` as part of `arc lint-rust`
Mergedriver update:
- Mergedriver used in resolving conflicts on commits is now pointing to V2
- It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble
Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for:
- Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section)
- "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string
- "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo
- "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added)
- Files generated from thrift_library rules that do not support "rust" language are removed
- Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored
Others:
- Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo`
- Updated following files to use V2:
- `common/rust/tools/reindeer/version-bump`
- `remote_execution/rust/setup.sh`
- Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing
Reviewed By: zertosh
Differential Revision: D26728789
fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
Summary:
Changes that could be done as a separate step from the rollout diff:
- Added README.md with documentation for developers
- Pointed `external_linter_tests/run_linter.sh` to MSDK/dotslash released autocargo
- Added `project_configs/ios_static_listing.toml` to reflect changes in `autocargo_projects.py` that were made in the meantime
- Added `scripts/run_tests` to run `arc external-linter-test` using cargo built autocargo binary
- Modified `third-party/rust/Cargo.toml` and run cargo-autocargo to match formatting of thirt party crates produced by V2
Reviewed By: ahornby
Differential Revision: D26728788
fbshipit-source-id: 361d9e2d41685f34eb4d20fc5ad9930f24b48f85
Summary: This seems to have been reverted by accident in D26618363 (f317302b0f).
Differential Revision: D26689734
fbshipit-source-id: e86451716cab3cc62f517c3f5fca7898d1a25095
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary:
Like our other "tokio ecosystem" crates, I'm renaming this one tokio-util-02
(as in tokio-02) since this is the last version it is compatible with. Note that this had a
breaking change as well, so I updated the callsites.
Reviewed By: HarveyHunt
Differential Revision: D26636350
fbshipit-source-id: 30f7da1036c861a97717c8d26648daaae33ecfbd
Summary:
We use a forked version of Gotham in Mononoke. This isn't great, because we
have to maintain this fork. Ideally, we'd upstream our changes, but as is
they're a bit intrusive and not generally useful, which makes this hard.
I've reworked how we do our Gotham changes, and now we only need to make 1 bit
of code public, which might be easier to get upstream. Concretely, Gotham has a
concept of "connected handler" that links a Hyper request and a socket address,
but in our case we want more things. This change lets us instantiate our own
Gotham state, and then add a few more things to it as necessary.
This diff updates our code accordingly to be compatible. This also lets us trim
down on some ceremony we had to do call into Gotham
from Mononoke Server.
Reviewed By: farnz
Differential Revision: D26634653
fbshipit-source-id: 024a48ebc3f323c165ac412ef422755e8cb1c650
Summary:
Simple test that can give us an intuition for how the ConcurrentMemIdMap should
perform.
Reviewed By: krallin
Differential Revision: D26601378
fbshipit-source-id: ae8f2ada6fc08eef806f3ece72a6c1c2f011ac32
Summary:
The on demand update code we have is the most basic logic that we could have.
The main problem is that it has long and redundant write locks. This change
reduces the write lock strictly to the section that has to update the in memory
IdDag.
Updating the Dag has 3 phases:
* loading the data that is required for the update;
* updating the IdMap;
* updating the IdDag;
The Dag can function well for serving requests as long as the commits involved
have been built so we want to have easy read access to both the IdMap and the
IdDag. The IdMap is a very simple structure and because it's described as an
Arc<dyn IdMap> we push the update locking logic to the storage. The IdDag is a
complicated structure that we ask to update itself. Those functions take
mutable references. Updating the storage of the iddag to hide the complexities
of locking is more difficult. We deal with the IdDag directly by wrapping it in
a RwLock. The RwLock allows for easy read access which we expect to be the
predominant access pattern.
Updates to the dag are not completely stable so racing updates can have
conflicting results. In case of conflics one of the update processes would have
to restart. It's easier to reason about the process if we just allow one
"thread" to start an update process. The update process is locked by a sync
mutex. The "threads" that fail the race to update are asked to wait until the
ongoing update is complete. The waiters will poll on a shared future that
tracks the ongoing dag update. After the update is complete the waiters will go
back to checking if the data they have is available in the dag. It is possible
that the dag is updated in between determining that the an update is needed and
acquiring the ongoing_update lock. This is fine because the update building
process checks the state of dag before the dag and updates only what is
necessary if necessary.
Reviewed By: krallin
Differential Revision: D26508430
fbshipit-source-id: cd3bceed7e0ffb00aee64433816b5a23c0508d3c
Summary:
The earlier diffs in this stack have removed all our dependencies on the Tokio
0.1 runtime environment (so, basically, `tokio-executor` and `tokio-timer`), so
we don't need this anymore.
We do still have some deps on `tokio-io`, but this is just traits + helpers,
so this doesn't actually prevent us from removing the 0.1 runtime!
Note that we still have a few transitive dependencies on Tokio 0.1:
- async-unit uses tokio-compat
- hg depends on tokio-compat too, and we depend on it in tests
This isn't the end of the world though, we can live with that :)
Reviewed By: ahornby
Differential Revision: D26544410
fbshipit-source-id: 24789be2402c3f48220dcaad110e8246ef02ecd8
Summary:
This was a thing that was only ever used in Mononoke, and we don't think it's
usable and haven't been using it. Let's get rid of it. As-is, it won't even work
for most people due to its (indirect) dependency on Tokio 0.1.
Reviewed By: StanislavGlebik
Differential Revision: D26512243
fbshipit-source-id: faa16683f2adb20dfba43c4768486b982bc02ff9
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary: backfill_git_mapping was great for backfilling git mappings, let's use it for svn
Reviewed By: StanislavGlebik
Differential Revision: D26453407
fbshipit-source-id: 3ee61722ac729af5a2fc4187fda3341069bb53f6
Summary:
Bonsai svnrev mapping will allow us to store the original svn revision number for repos that
were imported from svn at some point in time. This will allow scs_server to
easily respond to queries about commits identified by them.
The code is mostly copypasta from globalrevs
Reviewed By: krallin
Differential Revision: D26424833
fbshipit-source-id: 9825de52c23ab9386de1a9d0e0a9506c0a035a8a
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.
Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.
Reviewed By: ahornby
Differential Revision: D26339606
fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
Summary: The local patch has been upstreamed so switching back to upstream's new version of sshcerts with proper support for our SSH keys and certs
Reviewed By: Imxset21
Differential Revision: D26419743
fbshipit-source-id: c6e6d2a8f77623360636a68febebfc85ad6f5c26
Summary:
For some repositories (e.g. configerator-sitevars) running "git gc" is too
slow. At the same time after initial rsync it would be good to run at least
some kind of validation to check that repo is not completely corrupt.
This diff adds that. This code checks that HEAD and 100 of its ancestors have
the object that they reference are reachable.
Reviewed By: krallin
Differential Revision: D26423769
fbshipit-source-id: a3fcd9fc5c30e5bf0fdc1cd0fb9e03bdc2e1371d
Summary:
johansglock pointed out that Hyper is affected by CVE-2021-21299. Let's update
to a fixed version.
Reviewed By: farnz
Differential Revision: D26313854
fbshipit-source-id: 4db04d3044fb9f22a037bda0a88a5314f62f9dfc
Summary:
For whatever reason, the metadata and vendored sources do not match
the actual upstream sources. I deleted all of third-party/rust/vendor/** and
revendored.
Notable changes:
- libra-crypto and libra-canonical-serialization picked up changes from D25098952 (Nov 2020)
D25986058 updated the source location of z3, but didn't update the sources.
Unfortunately the updated location seems completely unbuildable, so for now
this doesn't update any of z3 until jagill can fix it.
Reviewed By: wangbj
Differential Revision: D26377549
fbshipit-source-id: d30dbaa7502c82c4b7cd64f8c80ab978a5646271
Summary:
Like it says in the title. This adds a crate that provides a combinator that
lets us easily find stalls caused by futures that stay in `poll()` for too
long.
The goal is to make this minimal overhead for whoever is using it: all you need
is to import it + give it a logger. It automatically looks up the line where
it's called and gives it back to you in logs. This uses the `track_caller`
functionality to make this work.
Reviewed By: farnz
Differential Revision: D26250068
fbshipit-source-id: a1458e5adebac7eab6c2de458f679c7215147937
Summary:
Like it says in the title, this splits edenapi_server into 2 crates. Most of
the logic here is pretty straightforward:
- edenapi_service creates a Gotham handler
- edenapi_server instantiates the handler and sets up a socket, tls, etc.
Reviewed By: ahornby
Differential Revision: D26108439
fbshipit-source-id: 6a79e9767ba891265bca11f78eb1a6d3a61ee21f
Summary: Vendor update to smallvec 1.6.1, which is a prerequisite for governor crate.
Reviewed By: jsgf
Differential Revision: D26172450
fbshipit-source-id: 9e87c1ada75b75ac54afd85435cd5422a1c385fa
Summary:
I'd like to be able to use mononoke_api from repo_client, but the hg parts
depend on repo_client, so that's a circular dependency.
This splits out the hg parts of Mononoke API so that places that don't want
them don't have to import them. This is similar to what we did with blobrepo.
Reviewed By: StanislavGlebik
Differential Revision: D26099495
fbshipit-source-id: 73a9c7b5dc95feceb35b5eabccf697e9aa0a27de
Summary:
Like it says in the title, this updates our implementation of Globalrevs to
be a little more relaxed, and allows you to create and move bookmarks as long as
they are ancestors of the "main" Globalrevs bookmark (but NOT to pushrebase to
them later, because we only want to allow ONE globalrevs-publishing bookmark
per repo).
While in there, I also deduplicated how we instantiate pushrebase hooks a
little bit. If anything, this could be better in the pushrebase crate, but
that'd be a circular dependency between pushrebase & bookmarks movement.
Eventually, the callsites should probably be using bookmarks movement anyway,
so leaving pushrebase as the low-level crate and bookmarks movement as the high
level one seems reasonable.
Reviewed By: StanislavGlebik
Differential Revision: D26020274
fbshipit-source-id: 5ff6c1a852800b491a16d16f632462ce9453c89a
Summary:
`blobstore_healer` works by healing blobs in turn, with some level of concurrency. Healing each blob consumes at least `O(blob_size)` or memory, so healing multiple blobs consumes their combined size of memory. Because blob sizes are not distributed uniformly, we cannot just calculate the desired concurrency level once and for all. Prior to this diff, this is what we did and whenever a few multi-gb blobs ended up in the same concurrently-healed batch, the healer OOMed. To help with this problem, this diff starts using dynamic concurrency - it assigns weight to each healed blob and only concurrently heals up to a certain total weight of blobs. This way, we can limit the total amount of memory consumed by our healer.
This solution is not perfect for a variety of reasons:
- if a single blob is larger than the total allowed weight, we'll still let it through. It's better than never healing it, but it means that OOMs are still possible in theory.
- we do not yet know the sizes of all the blobs in the queue. To mitigate that, I took a look at the known sizes distribution and saw that between 0 and 2KB is the most common size range. I defaulted to 1KB size of the unknown blob
Note 1: I had to make `heal_blob` consume it's args instead of borrowing them because `buffered_weight_limited` needs `'static` lifetime for the futures.
Note 2: When using `futures_ext`, I explicitly rename them to `futures_03_ext`, despite the fact that `blobstore_healer` does not depend on the older version. This is because `Cargo.toml` uses the same `[dependencies]` section for the combined dependencies of all the targets in the same `TARGETS` file. As there are other targets that claim the name of `futures_ext` for 0.1 version, I decided that it's easier to just use `_03_` name here than fix in other places. We can always change that of course.
Reviewed By: krallin
Differential Revision: D26106044
fbshipit-source-id: 4931d86d6e85d055ed0eefdd357b9ba6266a1c37
Summary:
Provide a command that measures per-generation sizes, to be run in
Chronos.
Reviewed By: ahornby
Differential Revision: D25981511
fbshipit-source-id: 7d9877dd5d04ac6881b5024e0ba132c234499d5a
Summary:
I plan to use these functions in the megarepotool, so let's move them to a
library that can be used in both.
Reviewed By: krallin
Differential Revision: D26015773
fbshipit-source-id: 0d2d28d86471c417508494883b69fb64e1bea328
Summary:
This diff adds some basic information. Note I have no idea what most of
these crates do so the descriptions are a bit light. I'd appreciate it if people
could tell me what the descriptions should be. Or better yet, just put up a diff
for it after this one goes in.
Reviewed By: lukaspiatkowski
Differential Revision: D26031870
fbshipit-source-id: 496f44882b6ed80a4f3834cf0fa3215ab318c930
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
Pull Request resolved: https://github.com/facebookincubator/resctl/pull/8074
1. Upgrade `Cursive` to v0.16.3
2. patch `cursive_buffered_backend` with a temp fork before v0.4.2 is released. This is needed to avoid Rust being confused by `Backend` trait from different versions of `cursive_core`
Reviewed By: dschatzberg
Differential Revision: D25987535
fbshipit-source-id: 6826653f9b6d4d6bdba485abf42eb517a853acc1
Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.
This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).
The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.
Reviewed By: farnz
Differential Revision: D25952428
fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
Summary:
I'd like to update our connection acceptor to support ALPN in order to
distinguish hgcli traffic from http traffic (the former will have ALPN and the
latter will not).
To do so, I need to parse ALPN, which is a length-prefixed bytestring format.
Let's add support for it in a library. This is what this does.
Reviewed By: johansglock
Differential Revision: D25954536
fbshipit-source-id: 3756dde90906a5f0f36630b866bcb17d03bc94c6
Summary:
This feature is useful for testing time-dependent stuff (e.g. it
allows you to stop/forward time). It's already included in the buck build.
Reviewed By: SkyterX
Differential Revision: D25946732
fbshipit-source-id: 5e7b69967a45e6deaddaac34ba78b42d2f2ad90e
Summary: Add a benchmark tool for bulkops that uses the criterion benchmarking framework. This is so we can measure effect of optimizations later in stack.
Differential Revision: D25804026
fbshipit-source-id: 71b8addf1145c0ecb69d6392b4602172f2b52080
Summary:
This updates Gotham. Under the hood I rebased our fork, you can see the diff
here: P161171514.
The stuff that is relevant is that Gotham got rid of its dependency on
`failure` and now uses `anyhow` instead, and I also added a little bit to our
existing socket data patch by making a few things public so that we can get
access to a few more internals.
Reviewed By: StanislavGlebik
Differential Revision: D25850262
fbshipit-source-id: 25ebf5d63be39e3e93208705d91abc5c61c90453
Summary: Convert all BlobRepoHg methods to new type futures
Reviewed By: StanislavGlebik
Differential Revision: D25471540
fbshipit-source-id: c8e99509d39d0e081d082097cbd9dbfca431637e
Summary:
Like it says in the title, this adds a caching layer around Globalrevs using
our existing `GetOrFillMultipleFromCacheLayers` abstraction.
Note: I've opted to not track dedicated metrics for this (compare to the hg
mapping to see them), since I don't believe we really ever look at them.
I'd like to do a little bit of refactoring of
`GetOrFillMultipleFromCacheLayers` to a) track them without having to ad-hoc
code it, b) convert it 0.3 futures, and c) require less ceremony to call it.
However, I'll do so in another diff.
Reviewed By: StanislavGlebik
Differential Revision: D25334478
fbshipit-source-id: 1385298b8fdf1cd081b6e509c6c3e03b3abbfa5b
Summary:
`ObservabilityContext` is a structure that helps logging facilities within Mononoke to make logging decisions. Specifically, the upcoming `DynamicLoggingDrain` and already existing `MononokeScubaSampleBuilder` will have this structure as a component and decide whether a particular logging statement (slog or scuba) should go ahead.
Internally, `ObservabilityContext` is a wrapper around data received from a [configerator endpoint](https://www.internalfb.com/intern/configerator/edit/?path=scm%2Fmononoke%2Fobservability%2Fobservability_config.cconf).
This diff makes a few unobvious decisions about how this is organized. My goals were:
1. to have production (i.e. reading from configerator), static (i.e. emulating current prod behavior) and test variants of `ObservabilityContext`
1. to avoid having consumers know which of these variants are used
1. to avoid making all consumers of `ObservabilityContext` necessarily generic
1. to avoid using dynamic dispatch of `ObservabilityContext`'s methods
Points 3 and 4 mean that `ObservabilityContext` cannot be a trait. `enum` is a common solution in such cases. However, if `ObservabilityContext` is an `enum`, consumers will know which variant they are using (because `enum` variants are public). So the solution is to use a private enum wrapped in a struct.
Reviewed By: mitrandir77
Differential Revision: D25287759
fbshipit-source-id: da034c71570137e8a8fb7749b1e4ad43be482f66
