Commit Graph

185 Commits

Author SHA1 Message Date
Ilia Medianikov
449fd2fd02 mononoke/pushrebase_hooks: add a hook that saves prepushrebase changeset id
Summary:
Knowing the prepushrebase changeset id is required for retroactive_review.
retroactive_review checks landed commits, but verify_integrity hook runs on a commit before landing. This way the landed commit has no straightforward connection with the original one and retroactive_review can't acknowledge if verify_integrity have seen it.

Reviewed By: krallin

Differential Revision: D27911317

fbshipit-source-id: f7bb0cfbd54fa6ad2ed27fb9d4d67b9f087879f1
2021-04-27 03:52:50 -07:00
Ilia Medianikov
93b8cf116b mononoke/pushrebase: split pushrebase crate
Summary:
Split pushrebase crate into pushrebase hook definition and pushrebase implementation.
Before this change it was impossible to store an attribute in BlobRepo that would depend on PushrebaseHook as it would create a circular dependency `pushrebase -> blobrepo -> pushrebase`.

Reviewed By: krallin

Differential Revision: D27968029

fbshipit-source-id: 030ef1b02216546cd3658de1f417bc52b8dd9843
2021-04-27 03:52:50 -07:00
Alex Hornby
bc85aade21 rust: update to zstd to 0.7.0+zstd.1.4.9
Summary:
Update the zstd crates.

This also patches async-compression crate to point at my fork until upstream PR https://github.com/Nemo157/async-compression/pull/117 to update to zstd 1.4.9 can land.

Reviewed By: jsgf, dtolnay

Differential Revision: D27942174

fbshipit-source-id: 26e604d71417e6910a02ec27142c3a16ea516c2b
2021-04-22 14:34:06 -07:00
Kostia Balytskyi
d48c87a95e megarepo: introduce write side of MononokeMegarepoConfigs
Summary:
`MononokeMegarepoConfig` is going to be a single point of access to
config storage system - provide both writes and reads. It is also a trait, to
allow for unit-test implementations later.

This diff introduces a trait, as well as implements the write side of the
configerator-based implementor. The read side/oss impl/test impl
is left `unimplemented`. Read side and test impl will be implemented in the future.

Things I had to consider while implementing this:
- I wanted to store each version of `SyncTargetConfig` in an individual
  `.cconf` in configerator
- at the same time, I did not want all of them to live in the same dir, to
  avoid having dirs with thousands of files in it
- dir sharding uses sha1 of the target repo + target bookmark + version name,
  then separates it into a dir name and a file name, like git does
- this means that these `.cconf` files are not "human-addressable" in the
  configerator repo
- to help this, each new config creation also creates an entry in one of the
  "index" files: human-readable maps from target + version name to a
  corresponding `.cconf`
- using a single index file is also impractical, so these are separated by
  ascification of the repo_id + bookmark name

Note: this design means that there's no automatic way to fetch the list of all
targets in use. This can be bypassed by maintaining an extra index layer, whihc
will list all the targets. I don't think this is very important atm.

Reviewed By: StanislavGlebik

Differential Revision: D27795663

fbshipit-source-id: 4d824ee4320c8be5187915b23e9c9d261c198fe1
2021-04-22 02:13:19 -07:00
Alex Hornby
4b8ed9a79a rust: remove gotham and hyper patches as referenced PRs have been released
Summary: Remove gotham and hyper patches as referenced PRs have been released

Reviewed By: krallin

Differential Revision: D27905248

fbshipit-source-id: a2b25562614b71b25536b29bb1657a3f3a5de83c
2021-04-21 05:19:37 -07:00
Kostia Balytskyi
81675b2d5b megarepo: introduce megarepo_error and make configo_client use it
Summary:
We want to distinguish user vs system errors in `configo_client` and its users
(`mononoke_config` for instance). The reason is to allow `scs_server`
distinguish the two types of errors. Normally user errors would only ever be
instantiated fairly "shallowly" in the `scs_server` itself, but `configo_client`
is a transactional client (by this I mean that it calls user-provided transformation functions on fetched data), so we need to allow for a user error to originate from these updater functions.

Reviewed By: StanislavGlebik

Differential Revision: D27880928

fbshipit-source-id: e00a5580a339fdabc4b858235da9cd7e9fc7a376
2021-04-21 05:02:00 -07:00
Simon Farnsworth
ecf7c4665b Implement a baseline dumb packer
Summary:
This is a baseline packer, to let us experiment with packing.

It chooses the dictionary that gives us the smallest size, but does nothing intelligent around key handling.

Note that at this point, packing is not always a win - e.g. the alias blobs are 432 bytes in total, but the resulting pack is 1528 bytes.

Reviewed By: ahornby

Differential Revision: D27795486

fbshipit-source-id: 05620aadbf7b680cbfcb0932778f5849eaab8a48
2021-04-20 12:45:08 -07:00
Kostia Balytskyi
815b5ad04a megarepo_api: introduce a basic configo client
Summary:
This is not used on its on, but in subsequent diffs I will add a use-case, by
the megarepo configs crate.

When built in non-fbcode mode, this crate does not export anything. I chose this approach as opposed to the approach of exporting no-op stubs to force the clients to pay attention and implement gating their side too. This seems reasonable for a rather generic configo client.

Reviewed By: StanislavGlebik

Differential Revision: D27790753

fbshipit-source-id: d6dcec884ed7aa88abe5796ef0e58be8525893e2
2021-04-19 08:34:12 -07:00
Alex Hornby
43f8fbab26 rust: remove smallvec fork
Summary: Now we're on rustc 1.51 the fork is no longer needed.

Reviewed By: dtolnay

Differential Revision: D27827632

fbshipit-source-id: 131841590d3987d53f5f8afb5ebc205cd36937fb
2021-04-19 01:32:20 -07:00
Thomas Orozco
db4c509b9e mononoke: use MononokeEnvironment in RepoFactory
Summary:
There is a very frustrating operation that happens often when working on the
Mononoke code base:

- You want to add a flag
- You want to consume it in the repo somewhere

Unfortunately, when we need to do this, we end up having to thread this from a
million places and parse it out in every single main() we have.

This is a mess, and it results in every single Mononoke binary starting with
heaps of useless boilerplate:

```
    let matches = app.get_matches();

    let (caching, logger, mut runtime) = matches.init_mononoke(fb)?;

    let config_store = args::init_config_store(fb, &logger, &matches)?;

    let mysql_options = args::parse_mysql_options(&matches);
    let blobstore_options = args::parse_blobstore_options(&matches)?;
    let readonly_storage = args::parse_readonly_storage(&matches);
```

So, this diff updates us to just use MononokeEnvironment directly in
RepoFactory, which means none of that has to happen: we can now add a flag,
parse it into MononokeEnvironment, and get going.

While we're at it, we can also remove blobstore options and all that jazz from
MononokeApiEnvironment since now it's there in the underlying RepoFactory.

Reviewed By: HarveyHunt

Differential Revision: D27767700

fbshipit-source-id: e1e359bf403b4d3d7b36e5f670aa1a7dd4f1d209
2021-04-16 10:27:43 -07:00
Jan Mazur
77a205db89 quiet certain connection errors when shutting down
Summary: Similar to D27155123 (1a56da1c6f).

Reviewed By: krallin

Differential Revision: D27805926

fbshipit-source-id: cf58a2e9b2ef92ca536f3b61b63fb42cfb1ec940
2021-04-16 02:26:04 -07:00
Thomas Orozco
e64012ad9e mononoke/timeseries: introduce a basic crate for tracking time series
Summary:
I'd like to be able to track time series for access within Mononoke. The
underlying use case here is that I want to be able to track the max count of
connections in our SQL connection pools over time (and possibly other things in
the future).

Now, the obvious question is: why am I rolling my own? Well, as it turns out,
there isn't really an implementation of this that I can reuse:

- You might expect to be able to track the max of a value via fb303, but you
  can't:

https://www.internalfb.com/intern/diffusion/FBS/browse/master/fbcode/fb303/ExportType.h?commit=0405521ec858e012c0692063209f3e13a2671043&lines=26-29

- You might go look in Folly, but you'll find that the time series there only
  supports tracking Sum & Average, but I want my timeseries to track Max (and
  in fact I'd like it to be sufficiently flexible to track anything I want):

https://www.internalfb.com/intern/diffusion/FBS/browse/master/fbcode/folly/stats/BucketedTimeSeries.h

It's not the first time I've ran into a need for something like this. I needed
it in RendezVous to track connections over the last 2 N millisecond intervals,
and we needed it in metagit for host draining as well (note that the
implementation here is somewhat inspired by the implementation there).

Reviewed By: mzr

Differential Revision: D27678388

fbshipit-source-id: ba6d244b8bb848d4e1a12f9c6f54e3aa729f6c9c
2021-04-12 05:22:33 -07:00
Arun Kulshreshtha
e6e2e61084 third-party/rust: patch curl and curl-sys
Summary: Update the `curl` and `curl-sys` crates to use a patched version that supports `CURLOPT_SSLCERT_BLOB` and similar config options that allow the use of in-memory TLS credentials. These options were added last year in libcurl version `7.71.0`, but the Rust bindings have not yet been updated to support them. I intend to upstream this patch, but until then, this will allow us to use these options in fbcode.

Reviewed By: quark-zju

Differential Revision: D27633208

fbshipit-source-id: 911e0b8809bc0144ad8b32749e71208bd08458fd
2021-04-08 11:50:38 -07:00
Mark Juggurnauth-Thomas
53550b9f10 blobrepo_factory: remove blobrepo_factory
Summary: This has been superseded by `RepoFactory`.

Reviewed By: krallin

Differential Revision: D27400617

fbshipit-source-id: e029df8be6cd2b7f3a5917050520b83bce5630e9
2021-04-07 14:01:49 -07:00
Mark Juggurnauth-Thomas
3b9817b5d8 benchmark_storage_config: remove dependency on blobrepo_factory
Summary: Use the equivalent function from `repo_factory`.

Reviewed By: krallin

Differential Revision: D27363470

fbshipit-source-id: dce3cf843174caa2f9ef7083409e7935749be4cd
2021-04-07 14:01:47 -07:00
Mark Juggurnauth-Thomas
f902acfcd1 repo_factory: add main repo factory
Summary:
Add a factory for building development and production repositories.

This factory can be re-used to build many repositories, and they will share
metadata database factories and blobstores if their configs match.

Similarly, the factory will only load redacted blobs once per metadata
database config, rather than once per repo.

Reviewed By: krallin

Differential Revision: D27323369

fbshipit-source-id: 951f7343af97f5e507b76fb822ad2e66f4d8f3bd
2021-04-07 14:01:46 -07:00
Alex Hornby
1e98362120 mononoke: add checkpoint for input to manual_scrub
Summary: Reduce amount of manual steps needed to restart a manual scrub by checkpointing  where it has got to to a file.

Differential Revision: D27588450

fbshipit-source-id: cb0eda7d6ff57f3bb18a6669d38f5114ca9196b0
2021-04-07 09:30:51 -07:00
Alex Hornby
e907f53bf8 mononoke: add progress to manual_scrub
Summary:
Some manual scrub runs can take a long time.  Provide progress feedback logging.

Includes  a --quiet option for when progress reporting not required.

Reviewed By: farnz

Differential Revision: D27588449

fbshipit-source-id: 00840cdf2022358bc10398f08b3bbf3eeec2b299
2021-04-06 08:45:29 -07:00
Alex Hornby
d2f915d4dd mononoke: compress manual_scrub output files if requested
Summary: Use async-compression to optionally zstd compress key and error files.

Reviewed By: farnz

Differential Revision: D27467761

fbshipit-source-id: e1ccb7dc32e4c41eaba82a3716cf4d13f64f71ea
2021-04-01 07:02:06 -07:00
generatedunixname89002005287564
fc6c12b9c7 Daily common/rust/cargo_from_buck/bin/autocargo
Reviewed By: farnz

Differential Revision: D27458795

fbshipit-source-id: 6d4f0e8ebcebe2f71e1152a61bf6511230c31af5
2021-03-31 02:34:06 -07:00
Mark Juggurnauth-Thomas
64461bb361 test_repo_factory: use test factory for remaining tests
Summary: Use the test factory for the remaining existing tests.

Reviewed By: StanislavGlebik

Differential Revision: D27169443

fbshipit-source-id: 00d62d7794b66f5d3b053e8079f09f2532d757e7
2021-03-25 07:34:51 -07:00
Mark Juggurnauth-Thomas
cc30bf6552 test_repo_factory: add factory for test repos
Summary:
Create a factory that can be used to build repositories in tests.

The test repo factory will be kept in a separate crate to the production repo factory, so that it can depend on a smaller set of dependencies: just those needed for in-memory test repos.  This should eventually help make compilation speeds faster for tests.

A notable difference between the test repos produced by this factory and the ones produced by `blobrepo_factory` is that the new repos share the in-memory metadata database.  This is closer to what we use in production, and in a couple of places it is relied upon and existing tests must use `dangerous_override` to make it happen.

Reviewed By: ahornby

Differential Revision: D27169441

fbshipit-source-id: 82541a2ae71746f5e3b1a2a8a19c46bf29dd261c
2021-03-25 07:34:49 -07:00
Jan Mazur
24d2aa1442 bump region2region QPS
Summary:
We need to bump SCS counters expressing Mononoke's QPS. They will look something like:
`requests:mononoke:oregon:carolina` for requests coming from proxygen in prn and mononoke in frc.

CSLB expects regions' full names.

We're getting src region from proxygen as a header.

Reviewed By: krallin

Differential Revision: D27082868

fbshipit-source-id: 12accb8a9df5cf6a80c2c281d2f61ac1e68176d1
2021-03-22 10:12:43 -07:00
Mark Juggurnauth-Thomas
46e3a31d10 repo_derived_data: add new repo attribute to encapsulate derived data
Summary:
Add `repo_derived_data`.  This is a new struct that encapsulates derived data
configuration and lease operations, and will be used in the facet-based
construction of repositories.

Reviewed By: ahornby

Differential Revision: D27169431

fbshipit-source-id: dee7c032deb93db8934736c111ba7238a6aaf935
2021-03-22 07:26:48 -07:00
Mark Juggurnauth-Thomas
3d6160f216 repo_identity: add new repo attribute to encapsulate identity
Summary:
Add `repo_identity`.  This is a new struct that encapsulates repository
identity and will be used in the facet-based construction of repositories.

Reviewed By: ahornby

Differential Revision: D27169445

fbshipit-source-id: 02a435bba54a633190c6d2e4316e86726aecfdf0
2021-03-22 07:26:48 -07:00
Mark Juggurnauth-Thomas
c83baeb00d segmented_changelog: split trait to separate crate
Summary:
Resolve a circular dependency whereby `BlobRepo` needs to depend on
`Arc<dyn SegmentedChangelog>`, but the segmented changelog implementation
depends on `BlobRepo`, by moving the trait definition to its own crate.

Reviewed By: sfilipco

Differential Revision: D27169423

fbshipit-source-id: 5bf7c632607dc8baba40d7a9d65e96e265d58496
2021-03-22 07:26:47 -07:00
Thomas Orozco
a3a0347639 mononoke/rendezvous: introduce query batching
Summary:
This introduces a basic building block for query batching. I called this
rendezvous, since it's about multiple queries meeting up in the same place :)

There are a few (somewhat conflicting) goals this tries to satisfy, so let's go
over them:

1), we'd like to reduce the total number of queries made by batch jobs. For
example, group hg bonsai lookups made by the walker. Those jobs are
characterized by the fact that they have a lot of queries to make, all the
time. Here's an example: https://fburl.com/ods/zuiep7yh.

2), we'd like to reduce the overall number of connections held to MySQL by
our tasks. The main way we achieve this is by reducing the maximum number of
concurrent queries. Indeed, a high total number of queries doesn't necessarily
result in a lot of connections as long as they're not concurrent, because we
can reuse connections. On the other hand, if you dispatch 100 concurrent
queries, that _does_ use 100 connections. This is something that applies to
batch jobs due to their query volume, but also to "interactive" jobs like
Mononoke Server or SCS, just not all the time. Here's an example:
https://fburl.com/ods/o6gp07qp (you can see the query count is overall low, but
sometimes spikes substantially).

2.1) It's also worth noting that concurrent queries are often the result of
many clients wanting the same data, so deduplication is also useful here.

3), we also don't want to impact the latency of interactive jobs when they
need to a little query here or there (i.e. it's largely fine if our jobs all
hold a few connections to MySQL and use them somewhat consistently).

4), we'd like this to make it easier to do batching right. For example, if
you have 100 Bonsais to map to hg, you should be able to just map and call
`future::try_join_all` and have that do the right thing.

5), we don't want "bad" queries to affect other queries negatively. One
example would be the occasional queries we make to Bonsai <-> Hg mapping in
`known` for thousands (if not more) of rows.

6), we want this to be easy to incorporate into the codebase.

So, how do we try to address all of this? Here's how:

- We ... do batching, and we deduplicate requests in a batch. This is the
  easier bit and should address #1, #2 and #2.1, #4.
- However, batching is conditional. We notably don't batch very large requests
  with the rest (addresses #5). We also don't batch small queries all the time:
  we only batch if we are observing a throughput of queries that suggests we
  can find some benefit in batching (this targets #3).
- Finally, we have some utilities for common cases like having to group by repo
  id (this is `MultiRendezVous`), and this is all configurable via tunables
  (and the default is to not do anything).

Reviewed By: StanislavGlebik

Differential Revision: D27010317

fbshipit-source-id: 4a2397255f9785c6722c02e4d419438fd0aafa07
2021-03-19 08:50:40 -07:00
Chengxiong Ruan
4fb5ba1152 Use released cursive_tab and cursive_buffered_backend version (#8078)
Summary:
Pull Request resolved: https://github.com/facebookincubator/resctl/pull/8078

Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/21

Pull Request resolved: https://github.com/facebookexperimental/eden/pull/76

Use released version to fix cursive_core version conflicts.

Reviewed By: boyuni

Differential Revision: D27032206

fbshipit-source-id: ba664b21cd55453dbc8124ff967a6f9d61fc4926
2021-03-16 10:00:37 -07:00
Stanislau Hlebik
480a0e9ef7 mononoke: start moving streaming changelog logic to rust
Summary:
Our current straming changelog updater logic is written in python, and it has a
few downsides:
1) It writes directly to manifold, which means it bypasses all the multiplexed
blobstore logic...
2) ...more importantly, we can't write to non-manifold blobstores at all.
3) There are no tests for the streaming changelogs

This diff moves the logic of initial creation of streaming changelog entry to
rust, which should fix the issues mentioned above. I want to highligh that
this implementation only works for the initial creation case i.e. when there are no
entries in the database. Next diffs will add incremental updates functionality.

Reviewed By: krallin

Differential Revision: D27008485

fbshipit-source-id: d9583bb1b98e5c4abea11c0a43c42bc673f8ed48
2021-03-12 14:46:30 -08:00
Mark Juggurnauth-Thomas
91358f3716 mononoke_types: use SortedVectorMap for BonsaiChangeset
Summary:
BonsaiChangesets are rarely mutated, and their maps are stored in sorted order,
so we can use `SortedVectorMap` to load them more efficiently.

In the cases where mutable maps of filechanges are needed, we can use `BTreeMap`
during the mutation and then convert them to `SortedVectorMap` to store them.

Reviewed By: mitrandir77

Differential Revision: D25615279

fbshipit-source-id: 796219c1130df5cb025952bb61002e8d2ae898f4
2021-03-11 04:28:43 -08:00
Mark Juggurnauth-Thomas
36f78eadb8 benchmarks: add benchmark_large_directory
Summary:
Add a microbenchmark for deriving data with large directories.

This benchmark creates a commit with 100k files in a single directory, and then
derives data for that commit and 10 descendant commits, each of which add,
modify and remove some files.

Reviewed By: ahornby

Differential Revision: D26947361

fbshipit-source-id: 4215f1ac806c53a112217ceb10e50cfad56f4f28
2021-03-11 04:28:42 -08:00
Mark Juggurnauth-Thomas
eb4d31cc82 benchmark: rename to benchmarks/simulated_repo
Summary: Rename this benchmark to a specific name so that we can add new benchmarks.

Differential Revision: D26947362

fbshipit-source-id: a1d060ee79781aa0ead51f284517471431418659
2021-03-11 04:28:42 -08:00
Andrey Chursin
0be8e8ce29 vfs: introduce AsyncVfs
Summary:
AsyncVfs provides async vfs interface.
It will be used in the native checkout instead of current use case that spawns blocking tokio tasks for VFS action

Reviewed By: quark-zju

Differential Revision: D26801250

fbshipit-source-id: bb26c4fc8acac82f4b55bb3f2f3964a6d0b64014
2021-03-05 21:47:51 -08:00
Thomas Orozco
598dbd76b6 third-party: import a Tokio 1.x compatible version of Gotham
Summary:
This won't be very long lived, as I'll remove the 0.2 version once Mononoke is
updated (we are the only users), and stop using a fork as soon as Gotham
releases. However, for now, I need both versions in order to avoid to do it all
in 1 diff.

A couple things worth noting here:

- See the previous diff for why I am using 2 Git repos for this.
- I can't easily use 2 forks for this because there are some extra crates they
  vendor, notably `borrow_bag`. I removed it from both forks and pulled it from
  crates.io instead.

Reviewed By: mitrandir77

Differential Revision: D26781291

fbshipit-source-id: 0b9824c07b880dbd5b94cd6d62d2cb2a68e348e9
2021-03-04 06:42:56 -08:00
Thomas Orozco
5871cd0a90 third-party/rust: patch Hyper 0.14 sending Content-Length on 101 Continue responses
Summary:
There is a bug in Hyper with 101 Continue responses: it sends a Content-Length.
This makes Proxygen unhappy with websocket upgrades. We used to have this
patched in hyper-02, but since Mononoke is about to update to Tokio 1.x, we
also need in the matching Hyper.

One thing that's a bit awkward is you might notice I changed the fork where the
patch comes from. This is because `cargo vendor` cannot deal with 2 overriding
sources coming from the same upstream. I posted about my adventures in this
here: https://fb.workplace.com/groups/rust.language/permalink/5278514098863829/

Reviewed By: HarveyHunt

Differential Revision: D26780899

fbshipit-source-id: e775b7151427898d63d8767acaa53f5f68229db6
2021-03-04 06:42:56 -08:00
Thomas Orozco
2a803fc10d third-party/rust: update futures
Summary:
Those newer versions of Futures have compatibility improvements with Tokio,
notably:

- https://github.com/rust-lang/futures-rs/pull/2333
- https://github.com/rust-lang/futures-rs/pull/2358

Reviewed By: farnz

Differential Revision: D26778794

fbshipit-source-id: 5a9dc002083e5edfa5c614d8d2242e586a93fcf6
2021-03-04 06:42:55 -08:00
Thomas Orozco
ef7045e818 common/rust: use fbinit-tokio
Summary:
This diffs add a layer of indirection between fbinit and tokio, thus allowing
us to use fbinit with tokio 0.2 or tokio 1.x.

The way this works is that you specify the Tokio you want by adding it as an
extra dependency alongside `fbinit` in your `TARGETS` (before this, you had to
always include `tokio-02`).

If you use `fbinit-tokio`, then `#[fbinit::main]` and `#[fbinit::test]` get you
a Tokio 1.x runtime, whereas if you use `fbinit-tokio-02`, you get a Tokio 0.2
runtime.

This diff is big, because it needs to change all the TARGETS that reference
this in the same diff that introduces the mechanism. I also didn't produce it
by hand.

Instead, I scripted the transformation using this script: P242773846

I then ran it using:

```
{ hg grep -l "fbinit::test"; hg grep -l "fbinit::main"  } | \
  sort | \
  uniq | \
  xargs ~/codemod/codemod.py \
&&  yes | arc lint \
&& common/rust/cargo_from_buck/bin/autocargo
```

Finally, I grabbed the files returned by `hg grep`, then fed them to:

```
arc lint-rust --paths-from ~/files2 --apply-patches --take RUSTFIXDEPS
```

(I had to modify the file list a bit: notably I removed stuff from scripts/ because
some of that causes Buck to crash when running lint-rust, and I also had to add
fbcode/ as a prefix everywhere).

Reviewed By: mitrandir77

Differential Revision: D26754757

fbshipit-source-id: 326b1c4efc9a57ea89db9b1d390677bcd2ab985e
2021-03-03 04:09:15 -08:00
Thomas Orozco
a17cd28156 common/rust: introduce fbinit-tokio
Summary:
This introduces a couple "trampoline" crates to use alongside fbinit in order
to let callers choose which version of Tokio they want by selecting one or the
other and adding it to their `TARGETS` alongside `fbinit` (right now, they
need to include `tokio` there).

The entrypoints here will be called by the expansion of the `#[fbinit::main]`
and `#[fbinit::test]` macros for `async fn`-s.

Right now, this isn't wired up: that happens in the next diff. In this diff,
I'm just adding the two entrypoints.

Reviewed By: mitrandir77

Differential Revision: D26754751

fbshipit-source-id: 1966dadf8bbe427ce4a1e90559a81790d8e56e7a
2021-03-03 04:09:15 -08:00
Lukas Piatkowski
edb679f785 autocargo v2: rollout and replace v1 in all use-cases
Summary:
This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here.

Arc lint support:

V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff:
- Adds running `autocargo` as part of `arc lint-rust`

Mergedriver update:

- Mergedriver used in resolving conflicts on commits is now pointing to V2
- It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble

Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for:

- Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section)
- "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string
- "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo
- "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added)
- Files generated from thrift_library rules that do not support "rust" language are removed
- Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored

Others:

- Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo`
- Updated following files to use V2:
  - `common/rust/tools/reindeer/version-bump`
  - `remote_execution/rust/setup.sh`
- Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing

Reviewed By: zertosh

Differential Revision: D26728789

fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
2021-03-02 06:43:29 -08:00
Lukas Piatkowski
61019d6361 autocargo v2: documentation and config update before rollout
Summary:
Changes that could be done as a separate step from the rollout diff:
- Added README.md with documentation for developers
- Pointed `external_linter_tests/run_linter.sh` to MSDK/dotslash released autocargo
- Added `project_configs/ios_static_listing.toml` to reflect changes in `autocargo_projects.py` that were made in the meantime
- Added `scripts/run_tests` to run `arc external-linter-test` using cargo built autocargo binary
- Modified `third-party/rust/Cargo.toml` and run cargo-autocargo to match formatting of thirt party crates produced by V2

Reviewed By: ahornby

Differential Revision: D26728788

fbshipit-source-id: 361d9e2d41685f34eb4d20fc5ad9930f24b48f85
2021-03-02 06:43:29 -08:00
Thomas Orozco
e0c9e687b0 mononoke: put back gotham rev
Summary: This seems to have been reverted by accident in D26618363 (f317302b0f).

Differential Revision: D26689734

fbshipit-source-id: e86451716cab3cc62f517c3f5fca7898d1a25095
2021-02-26 05:21:46 -08:00
Lukas Piatkowski
f317302b0f autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.

Reviewed By: ikostia

Differential Revision: D26618363

fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-25 15:10:56 -08:00
Thomas Orozco
52a1c515a8 third-party/rust: update tokio-util to 0.3 and rename it
Summary:
Like our other "tokio ecosystem" crates, I'm renaming this one tokio-util-02
(as in tokio-02) since this is the last version it is compatible with. Note that this had a
breaking change as well, so I updated the callsites.

Reviewed By: HarveyHunt

Differential Revision: D26636350

fbshipit-source-id: 30f7da1036c861a97717c8d26648daaae33ecfbd
2021-02-25 07:42:30 -08:00
Thomas Orozco
c064d70c02 mononoke: fork Gotham less
Summary:
We use a forked version of Gotham in Mononoke. This isn't great, because we
have to maintain this fork. Ideally, we'd upstream our changes, but as is
they're a bit intrusive and not generally useful, which makes this hard.

I've reworked how we do our Gotham changes, and now we only need to make 1 bit
of code public, which might be easier to get upstream. Concretely, Gotham has a
concept of "connected handler" that links a Hyper request and a socket address,
but in our case we want more things. This change lets us instantiate our own
Gotham state, and then add a few more things to it as necessary.

This diff updates our code accordingly to be compatible. This also lets us trim
down on some ceremony we had to do call into Gotham
from Mononoke Server.

Reviewed By: farnz

Differential Revision: D26634653

fbshipit-source-id: 024a48ebc3f323c165ac412ef422755e8cb1c650
2021-02-25 07:42:29 -08:00
Stefan Filip
bf85eb884f segmented_changelog: add benchmark for ConcurrentMemIdMap
Summary:
Simple test that can give us an intuition for how the ConcurrentMemIdMap should
perform.

Reviewed By: krallin

Differential Revision: D26601378

fbshipit-source-id: ae8f2ada6fc08eef806f3ece72a6c1c2f011ac32
2021-02-23 11:12:18 -08:00
Stefan Filip
84017abe21 segmented_changelog: update OnDemandUpdateDag to have smaller critical sections
Summary:
The on demand update code we have is the most basic logic that we could have.
The main problem is that it has long and redundant write locks. This change
reduces the write lock strictly to the section that has to update the in memory
IdDag.

Updating the Dag has 3 phases:
* loading the data that is required for the update;
* updating the IdMap;
* updating the IdDag;

The Dag can function well for serving requests as long as the commits involved
have been built so we want to have easy read access to both the IdMap and the
IdDag. The IdMap is a very simple structure and because it's described as an
Arc<dyn IdMap> we push the update locking logic to the storage.  The IdDag is a
complicated structure that we ask to update itself. Those functions take
mutable references. Updating the storage of the iddag to hide the complexities
of locking is more difficult. We deal with the IdDag directly by wrapping it in
a RwLock. The RwLock allows for easy read access which we expect to be the
predominant access pattern.

Updates to the dag are not completely stable so racing updates can have
conflicting results. In case of conflics one of the update processes would have
to restart. It's easier to reason about the process if we just allow one
"thread" to start an update process. The update process is locked by a sync
mutex. The "threads" that fail the race to update are asked to wait until the
ongoing update is complete. The waiters will poll on a shared future that
tracks the ongoing dag update. After the update is complete the waiters will go
back to checking if the data they have is available in the dag. It is possible
that the dag is updated in between determining that the an update is needed and
acquiring the ongoing_update lock. This is fine because the update building
process checks the state of dag before the dag and updates only what is
necessary if necessary.

Reviewed By: krallin

Differential Revision: D26508430

fbshipit-source-id: cd3bceed7e0ffb00aee64433816b5a23c0508d3c
2021-02-22 18:17:21 -08:00
Thomas Orozco
097e4ad00c mononoke: remove tokio-compat (i.e. use tokio 0.2 exclusively)
Summary:
The earlier diffs in this stack have removed all our dependencies on the Tokio
0.1 runtime environment (so, basically, `tokio-executor` and `tokio-timer`), so
we don't need this anymore.

We do still have some deps on `tokio-io`, but this is just traits + helpers,
so this doesn't actually prevent us from removing the 0.1 runtime!

Note that we still have a few transitive dependencies on Tokio 0.1:

- async-unit uses tokio-compat
- hg depends on tokio-compat too, and we depend on it in tests

This isn't the end of the world though, we can live with that :)

Reviewed By: ahornby

Differential Revision: D26544410

fbshipit-source-id: 24789be2402c3f48220dcaad110e8246ef02ecd8
2021-02-22 09:22:42 -08:00
Thomas Orozco
f7d5b3db46 mononoke: remove tokio-timer dependencies
Summary: Like it says in the title.

Reviewed By: ahornby

Differential Revision: D26513724

fbshipit-source-id: 5d1f986af17c948ad24e3d378a7623a0d97f5cf4
2021-02-22 09:22:41 -08:00
Thomas Orozco
0734a61cb1 common/rust: remove tracing
Summary:
This was a thing that was only ever used in Mononoke, and we don't think it's
usable and haven't been using it. Let's get rid of it. As-is, it won't even work
for most people due to its (indirect) dependency on Tokio 0.1.

Reviewed By: StanislavGlebik

Differential Revision: D26512243

fbshipit-source-id: faa16683f2adb20dfba43c4768486b982bc02ff9
2021-02-22 09:22:41 -08:00
Lukas Piatkowski
cd0b6d50e2 autocargo v1: changes to match autocargo v2 generation results.
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.

Reviewed By: zertosh

Differential Revision: D26544150

fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 11:03:55 -08:00