Commit Graph

243 Commits

Author SHA1 Message Date
Davide Cavalca
5ad11146fc Bump some dependencies for below
Summary: Bump in preparation for a new release

Reviewed By: brianc118

Differential Revision: D31657479

fbshipit-source-id: eeb2a08c54f6db75b6e515ca0966637f31b906a8
2021-10-14 17:36:29 -07:00
Thomas Orozco
977755e59e Update autocargo component on FBS:master
Summary:
Automated component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/125803836415945/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/10/11 06:06AM PDT
---------
New project source changes since last bump based on 9cc37ee9d47923093119b37d4f2d60de5a5e490f at 2021/09/21 01:03PM UTC:
| 2021/09/21 01:05PM -05 | generatedunixname89002005294178 | D31082067 | [MSDK] Update autocargo component on FBS:master |
| 2021/09/29 01:58PM PDT | vgao1996 | D31115820 (ae87b82eaf) | [Rust] update rand and quickcheck |
| 2021/09/30 02:55PM PDT | jkeljo | D31284743 (542e84d8fc) | [rust][third-party] Enable `unbounded_depth` feature for `serde_json` |
| 2021/10/08 11:43AM BST | krallin | D31471849 | autocargo: emit rerun-if-changed on the buildscript itself |
---------

build-break (bot commits are not reviewed by a human)

Reviewed By: farnz

Differential Revision: D31541798

fbshipit-source-id: 8e41a39d552c31a879c41b7d244e2caca047b7da
2021-10-12 04:20:53 -07:00
Jan Mazur
361b61676a sync RepoConfigs configerator thrift structure to fbcode
Summary: I just landed D30899990. I'm landing this to sync the config, still without using it.

Differential Revision: D31472892

fbshipit-source-id: 5e18c4c3529118ef81880c886f0d8b9428efcbf4
2021-10-08 05:33:38 -07:00
Jeremy Fitzhardinge
66e1fc5370 rust: rerun autocargo after thrift/rust changes
Summary: Need this for cargo check/rust-analyzer to work.

Reviewed By: guswynn

Differential Revision: D31319911

fbshipit-source-id: ebd3fa72d8fc3667391a2067f95cab9e5f53301f
2021-09-30 17:40:04 -07:00
Stanislau Hlebik
f951d8d84f mononoke: include additional bonsai changesets during segmented changelog seeder
Summary:
In some cases (e.g. when master bookmark moves backwards) there might be
commits in segmented changelog that are not ancestors of master. When reseeding
we still want to build segments for these chagnesets, and this is what this
diff does (see D30898955 for more details about why we want to build segments
for these changesets).

Reviewed By: quark-zju

Differential Revision: D30996484

fbshipit-source-id: 864aaaacfc04d6169afd3d04ebcb6096ae2514e5
2021-09-17 01:23:05 -07:00
Jeremy Fitzhardinge
113060beb1 third-party/rust: bump all the tracing packages
Summary:
This includes recent changes:
- `skip_all` in #[instrument] https://github.com/tokio-rs/tracing/pull/1548
- NoSubscriber no-op Subscriber https://github.com/tokio-rs/tracing/pull/1549
- Box/Arc Layer (https://github.com/tokio-rs/tracing/pull/1536 +
  https://github.com/tokio-rs/tracing/pull/1547)
- log filtering fixes https://github.com/tokio-rs/tracing/pull/1543

Local: force `log` and `log-always` features off. This removes the
`enable-trace-logging` flag from smartplatform/reconciler.

Reviewed By: guswynn

Differential Revision: D30906032

fbshipit-source-id: fdc3e4e29a39e9e181d9f4ff017e6b3e16acdcd8
2021-09-15 16:52:25 -07:00
Callum Ryan
65bd8a9bc9 Support thrift_library::rust_include_srcs
Summary:
`rust_include_srcs` is supported on `thrift_library` as a way of including other Rust code in the generated crate, generally used to implement other traits on the generated types.

Adding support for this in autocargo by copying these files into the output dir and making sure their option is specified to the thrift compiler

Reviewed By: ahornby

Differential Revision: D30789835

fbshipit-source-id: 325cb59fdf85324dccfff20a559802c11816769f
2021-09-10 00:12:44 -07:00
Jeremy Fitzhardinge
fd03bff2e2 third-party/rust: bump tracing versions in preparation for patching
Summary:
Bump all the versions on crates.io to highest to make migration to github
versions in next diff work.

Reviewed By: dtolnay

Differential Revision: D30829928

fbshipit-source-id: 09567c26f275b3b1806bf8fd05417e91f04ba2ef
2021-09-09 22:38:25 -07:00
David Tolnay
5e9b8cd4b2 third-party/rust: Update thiserror from 1.0.23 to 1.0.29
Summary:
Release notes:

- https://github.com/dtolnay/thiserror/releases/tag/1.0.24
- https://github.com/dtolnay/thiserror/releases/tag/1.0.25
- https://github.com/dtolnay/thiserror/releases/tag/1.0.26
- https://github.com/dtolnay/thiserror/releases/tag/1.0.27
- https://github.com/dtolnay/thiserror/releases/tag/1.0.28
- https://github.com/dtolnay/thiserror/releases/tag/1.0.29

The pertinent feature is 1.0.29 adding support for inferred trait bounds on error types that contain generic type parameters. I remember someone asking for this in fbcode but I forget what project it was for.

```
use thiserror::Error;

#[derive(Error, Debug)]
pub enum MyError<E, F, G> {
    #[error("thing {0} ({0:?})")]
    Variant(E),
    #[error("some error")]
    Delegate(#[source] SomeError<F>),
    #[error("err 0o{val:o}")]
    Octal { val: G },
}
```

```
// generated

impl<E, F, G> std::error::Error for MyError<E, F, G>
where
    SomeError<F>: std::error::Error + 'static,  //'
    Self: std::fmt::Debug + std::fmt::Display;

impl<E, F, G> std::fmt::Display for MyError<E, F, G>
where
    E: std::fmt::Debug + std::fmt::Display,
    G: std::fmt::Octal;
```

Reviewed By: zertosh

Differential Revision: D30758449

fbshipit-source-id: b3afe08fe8c8affa26693df9cbb63e04632ea1d3
2021-09-08 20:49:35 -07:00
Thomas Orozco
0d2bfbeccd Update autocargo component on FBS:master
Summary:
Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/181247287328949/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/09/01 09:14AM PDT
---------
New project source changes since last bump based on D30663071 (08e362a355e0a64a503f5073f57f927394696b8c at 2021/08/31 03:47AM -05):
| 2021/08/31 04:41AM -05 | generatedunixname89002005294178 | D30665384 | [MSDK] Update autocargo component on FBS:master |
| 2021/08/31 07:14PM PDT | kavoor | D30681642 | [autocargo] Make cxx-build match version of cxx |
| 2021/09/01 04:05PM BST | krallin | D30698095 | autocargo: include generated comment in OSS manifests |
---------

build-break (bot commits are not reviewed by a human)

Reviewed By: farnz

Differential Revision: D30717040

fbshipit-source-id: 2c1d09f0d51b6ff2e2636496cf22bcf781f22889
2021-09-02 02:33:56 -07:00
Gus Wynn
87a09132dc tokio -> 1.10
Reviewed By: dtolnay

Differential Revision: D30647831

fbshipit-source-id: 7094873ec5cfbf80cd7c3564fdd011268053b0d3
2021-08-30 15:55:16 -07:00
Harvey Hunt
14941cc830 mononoke: Remove lfs rollout tier
Summary:
This config option was used to slowly roll out LFS for a repo.
However, it is no longer used and can therefore be removed.

Reviewed By: StanislavGlebik

Differential Revision: D30511880

fbshipit-source-id: 59fe5925cc203aa609488fdf8ea29e9ff65ee862
2021-08-26 09:22:28 -07:00
Aida Getoeva
e2d57e9f02 mononoke/multiplex: add multiplex logging
Summary:
The current Mononoke Blobstore Trace scuba table is used with idea of having a record per blobstore and operation. This diff adds logging to the new scuba table of the combined multiplexed operations' outcome, like time spent on the `put` including sync-queue and blobstore write or tracking record of the "some failed others none" cases in `get/is_present`.

This helps to see the real time spent on writes and reads and to assess the impact of changes coming in `get` and `is_present`.

Reviewed By: ahornby

Differential Revision: D30248284

fbshipit-source-id: f79050ced32ba77bd2e220e242407bcd711a9b6d
2021-08-16 04:25:33 -07:00
David Tolnay
aa8152f1dd Make thrift-generated dyn async traits future compatible
Summary:
The use of dyn traits of the Thrift-generated server traits was emitting future compatibility warnings with recent versions of rustc, due to a fixed soundness hole in the trait object system:

```
error: the trait `x_account_aggregator_if::server::XAccountAggregator` cannot be made into an object
     |
     = this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
note: for a trait to be "object safe" it needs to allow building a vtable to allow the call to be resolvable dynamically; for more information visit <https://doc.rust-lang.org/reference/items/traits.html#object-safety>
```

This diff pulls in https://github.com/dtolnay/async-trait/releases/tag/0.1.51 which results in the Thrift-generated server traits no longer hitting the problematic pattern.

Reviewed By: zertosh

Differential Revision: D29979939

fbshipit-source-id: 3e6e976181bfcf35ed453ae681baeb76a634ddda
2021-07-29 16:25:33 -07:00
Stanislau Hlebik
8cd6278de9 mononoke: implement ensure_ancestors_of option for bookmarks
Summary:
This option would let us tell that a given bookmark (or bookmarks if they are
specified via a regex) is allowed to move only if it stays an ancestor of a
given bookmark.
Note - this is a sev followup, and we intend to use it for */stable bookmarks
(e.g. fbcode/stable, fbsource/stable etc). They are always intended to be an
ancestor of master

Reviewed By: krallin

Differential Revision: D29878144

fbshipit-source-id: a5ce08a09328e6a19af4d233c1a273a5e620b9ce
2021-07-27 12:47:22 -07:00
Michael Voznesensky
7ed41f2b36 Bump configerator, add support for config driven no parent commits
Summary: As discussed, extends Mononoke service to support commits w/o parents for the AI Infra usecase.

Reviewed By: markbt

Differential Revision: D29810303

fbshipit-source-id: f07fd7f1521ffe1cea85f1f54e71fe37fc39bb62
2021-07-23 13:40:18 -07:00
Harvey Hunt
b3a504d191 mononoke: lfs: Remove throttle limits
Summary:
Now that the new `rate_limiting` crate is being used by LFS server we
can remove the throttle limits code and config.

Differential Revision: D29396505

fbshipit-source-id: 19638bd93ad9dea2638e8501837c6c13e4dd48ff
2021-07-15 04:09:51 -07:00
Harvey Hunt
14ba455e60 mononoke: Use new rate limiting crate
Summary:
Update Mononoke server to use the new `rate_limiting` crate. This diff
also removes the old rate limiting library.

Differential Revision: D29396507

fbshipit-source-id: 05adb9322705b771a739c8bcaf2816c95218a42d
2021-07-05 10:18:51 -07:00
Harvey Hunt
a92eae78ae mononoke: lfs: Use new load shedding config
Summary:
Replace the LFS server's load shedding logic with that provided by the
`rate_limiting` crate.

Differential Revision: D29396503

fbshipit-source-id: a71812a55b9c9f111ee2861dc1b131ad20ca82d2
2021-07-05 10:18:51 -07:00
Harvey Hunt
7b40d3af0d mononoke: Add new rate limiting library
Summary:
Add a new rate limiting library that also supports load shedding when
an individual server is overloaded. This library provides a few benefits:

- The code can be shared between the LFS server and Mononoke server.
- The library supports more complex expressions of which clients to apply a
  rate limit to (e.g. 10% of sandcastle and mactest machines).
- The rate limiting `Target` can be expanded in the future as the client
  provides more information (e.g. client region).
- Mononoke server will be able to loadshed if an individual host is overloaded,
  as we can currently do with the LFS server.

I've added this library as a separate crate rather than rewriting
`load_limiter` to make it easier to review. The next diff will make use of the
new library and remove the old one.

Reviewed By: StanislavGlebik

Differential Revision: D29396509

fbshipit-source-id: 2fbc04e266b18392062e6f952075efd5e24e89ba
2021-07-05 10:18:51 -07:00
Yan Soares Couto
8e3c29e8d1 Create class to load redaction config from configerator
Summary:
Adds class `ConfigeratorRedactedBlobs` that reads redaction data from configerator, and reloads it when necessary.

The class does this:
- Reads `RedactionSets` from configerator.
- For each key there, read `RedactionConfigBlobstore` looking for a `RedactionKeyList` with that key (these were populated by D29033598).
- From the keys listed, builds the map of redacted blobs, with the same format as before when it was fetched from XDB.
- Periodically checks if config changed. If so, reload the map of redactions. (should only happen when we land a new config change to redaction, which should be very very rare)
- We use ArcSwap to keep the config, as a good way to provide read-only access with eventual reloading.

Not implemented on this diff:
- Creation of `ConfigeratorRedactedBlobs`, or adding it to `RedactedBlobs` enum.

It's not used in this diff, will be used in the future, I split it mostly to make it easier to review.

Reviewed By: StanislavGlebik

Differential Revision: D29033595

fbshipit-source-id: 36603685433b6dd153f2c23123907f7311c20a32
2021-06-30 08:57:30 -07:00
Yan Soares Couto
b4f3a36802 Sync changes from D29360425 to fbcode
Summary: Syncing configerator changes from D29360425, and fixing all tests. Not used yet.

Reviewed By: markbt

Differential Revision: D29363416

fbshipit-source-id: d2de13d32bcec2e7fbff20204be8d9a8d65c0efe
2021-06-30 08:57:30 -07:00
Yan Soares Couto
1bcae1ae65 Add redaction config to common config, don't use it yet
Summary:
This reads the config added on D29305462. It populates it into `CommonConfig` struct, and also adds it to `RepoFactory`, but doesn't yet use it anywhere, this will be done on the next diff.

There is a single behaviour change in this diff, which I believe should be harmless but is noted in the comments in case it isn't.

Reviewed By: markbt

Differential Revision: D29272581

fbshipit-source-id: 62cd7dc78478c1d8cb212eafdd789527ead50ef6
2021-06-30 08:57:30 -07:00
Mark Juggurnauth-Thomas
a533bd24ea sync configerator thrift
Reviewed By: StanislavGlebik

Differential Revision: D29453806

fbshipit-source-id: 4897f7feb6f09371cd3963d9ed30e37403b24ba2
2021-06-29 10:32:40 -07:00
Daniel Xu
431a4ed16b Fix autocargo skew
Summary: I think someone landed a dependency change or something and forgot to update autocargo

Reviewed By: dtolnay

Differential Revision: D29402335

fbshipit-source-id: e9a4906bf249470351c2984ef64dfba9daac8891
2021-06-25 17:23:33 -07:00
Mark Juggurnauth-Thomas
b001dd9ea1 sync repos.thrift from configerator
Reviewed By: StanislavGlebik

Differential Revision: D29067721

fbshipit-source-id: f7d792205d0dfa9ad6b2b1ce0845f978f5f3f008
2021-06-24 04:13:58 -07:00
Davide Cavalca
b82c5672fc Update several rust crate versions
Summary: Update versions for several of the crates we depend on.

Reviewed By: danobi

Differential Revision: D29165283

fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
2021-06-17 16:38:19 -07:00
Stanislau Hlebik
879d8f3596 mononoke: add support for by repo vec of strings tunables
Summary: It will be used in the next diff to disable only certain derived data types

Reviewed By: yancouto

Differential Revision: D29031082

fbshipit-source-id: 9621a5eec522e369fef2b78e94599a056eda5317
2021-06-11 05:53:24 -07:00
Yan Soares Couto
24081bcf9b Sync redaction thrift files
Summary:
Syncing thrift files from D28931534

Will use it in following diffs

Reviewed By: farnz

Differential Revision: D28937805

fbshipit-source-id: d3ea22ae2c402aa244f3972d9a70516776f77a11
2021-06-08 07:16:45 -07:00
Jan Mazur
49fafb8903 option to enforce authN
Summary: LFS can reload config dynamically and can do it per-region. Let's do this instead rolling this change on the VIP, because here a rollback could happen in no time.

Reviewed By: krallin

Differential Revision: D28534346

fbshipit-source-id: 131ef2d0d084b27719a9aaf4e29003b06c578a67
2021-05-20 01:06:04 -07:00
Harvey Hunt
a58a9e805e mononoke: lfs server: Disable compression dynamically
Summary:
During S231236 the LFS servers were using a lot of CPU and were
compressing blobs sent to clients.  Thomas hotfixed the servers to disable
compression in order to save some CPU. Instead of having to rebuild and
redeploy the LFS server, update it to be able to disable compression using
configerator.

The `disable_compression` option will disable compression of downloads
globally. The `disable_compression_identities` allows us to disable compression
for some group of identities.

I also refactored some of the shared code from `download` and `download_sha256`
into a new function, `download_inner`.

Reviewed By: krallin

Differential Revision: D28382405

fbshipit-source-id: 792f10a9e3bb32b56ef87aa8e0b2c4b098567579
2021-05-14 02:39:00 -07:00
CodemodService Bot
e6d4eca353 Daily common/rust/cargo_from_buck/bin/autocargo
Reviewed By: krallin

Differential Revision: D27936164

fbshipit-source-id: 863666219bda84e989cd6008e62b33db74fb81d4
2021-04-22 05:12:20 -07:00
Jan Mazur
c9205507d5 specify more counters to be bumped
Summary: More context in previous diff in the stack.

Reviewed By: krallin

Differential Revision: D27852299

fbshipit-source-id: dc06b29794d0c4e8ff6bf3f44507a64c06f01771
2021-04-19 05:49:05 -07:00
Kostia Balytskyi
59ee3033c9 megarepo: sync D27794245 to fbcode
Reviewed By: farnz

Differential Revision: D27794480

fbshipit-source-id: 5e3643ea0124e22e78744d9cc45e4ec9cf1da8b2
2021-04-15 09:39:09 -07:00
Kostia Balytskyi
bf801f70a0 megarepo: import all_config_versions.thrift
Summary: To sync D27741269.

Reviewed By: mitrandir77

Differential Revision: D27741277

fbshipit-source-id: 52ade8d19ae434759a9b889fb82816dbf1feecfe
2021-04-13 11:38:27 -07:00
Kostia Balytskyi
e60f163a14 megarepo: introduce a cfgr-synced copy of megarepo_configs.thrift
Summary:
This diff adds boilerplate needed to have a cfgr-sourced (D27623488)
`megarepo_configs.thrift` live in `fbcode` successfully. This diff does
introduce any new use-cases for the file.

Note that I suspect that it's better to sync this file to www from fbcode, not from configerator, because the import path for `megarepo_configs.thrfit` in `source_control.thrift` should be the same across `fbcode` and `www`, and files are imported differently into `www` from `fbcode` and `configerator` unfortunately. As another justification, I think keanu's [`traffic_types.thrift`](https://www.internalfb.com/intern/diffusion/WWW/browsefile/master/flib/intern/thrift/if/fbcode/configerator/structs/keanu/glb/traffic_types.thrift) is synced into `www` from `fbcode` also, not from its source of truth in `configerator`.

Reviewed By: StanislavGlebik

Differential Revision: D27647707

fbshipit-source-id: 2bb2fdf1d4a68e686db302783b68062f3d95fb88
2021-04-08 06:41:07 -07:00
Jan Mazur
24d2aa1442 bump region2region QPS
Summary:
We need to bump SCS counters expressing Mononoke's QPS. They will look something like:
`requests:mononoke:oregon:carolina` for requests coming from proxygen in prn and mononoke in frc.

CSLB expects regions' full names.

We're getting src region from proxygen as a header.

Reviewed By: krallin

Differential Revision: D27082868

fbshipit-source-id: 12accb8a9df5cf6a80c2c281d2f61ac1e68176d1
2021-03-22 10:12:43 -07:00
Stanislau Hlebik
847a91291b mononoke: make it possible to allow moving a bookmark for a hipster group
Summary:
Currently we can only limit which users are allowed to move a bookmark by a
regex. We also want to allow specifying a hipster group.

Reviewed By: krallin

Differential Revision: D27156690

fbshipit-source-id: 99a5678a82f4c34ed2e57625361ba7cdb08ed839
2021-03-18 13:05:11 -07:00
Alex Hornby
12a1c4512c mononoke: sync configerator repos thrift for packblob compression options
Summary: Bring across the thrift changes so can code against them.

Reviewed By: krallin

Differential Revision: D27116899

fbshipit-source-id: 27bf6f23bebbc43d6c4d6c668ff905b72b0eb0f9
2021-03-18 03:21:40 -07:00
Stanislau Hlebik
a80dece152 mononoke: add a way to limit s3 blobstore
Summary:
S3 blobstore is sensitive to the number of open connections and it causes perf
problems. Let's add a way to limit the number of simultaneous open connections.

Also see D26911520

Reviewed By: krallin

Differential Revision: D26911679

fbshipit-source-id: 1e3e4922b26d0a29c94c89b4aa76b9b5928effe6
2021-03-11 00:55:14 -08:00
Ran Yaniv
e5d163411a run autocargo
Summary: Run autocargo.

Reviewed By: c-ryan747

Differential Revision: D26878408

fbshipit-source-id: 14e2efb280c906fa7a79253da74d7518e1a89079
2021-03-08 12:40:50 -08:00
Stanislau Hlebik
cc0c8cd0ff mononoke: allow changing xrepo mapping extra for if config is set
Summary:
Normally we prohibit landing commits that might accidentally
change the x-repo mapping. However we do want to allow landing commits like
that to backup repos, because backup repos should have all commits as their
counterpart repositories.

This change also has another side-effect - we don't call `load_additional_changeset()`
which can be very expensive for backup repos because of the issues in configuration -
in particular, we don't have `hooks_ancestors_of` option set, and that caused all ancestors to be considered
as "additional changesets". It would make since more properly later.

Differential Revision: D26883910

fbshipit-source-id: 07ceb7b96bc6cae851ac6ff57071eff5cef387e4
2021-03-08 08:39:30 -08:00
Thomas Orozco
2a803fc10d third-party/rust: update futures
Summary:
Those newer versions of Futures have compatibility improvements with Tokio,
notably:

- https://github.com/rust-lang/futures-rs/pull/2333
- https://github.com/rust-lang/futures-rs/pull/2358

Reviewed By: farnz

Differential Revision: D26778794

fbshipit-source-id: 5a9dc002083e5edfa5c614d8d2242e586a93fcf6
2021-03-04 06:42:55 -08:00
Lukas Piatkowski
edb679f785 autocargo v2: rollout and replace v1 in all use-cases
Summary:
This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here.

Arc lint support:

V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff:
- Adds running `autocargo` as part of `arc lint-rust`

Mergedriver update:

- Mergedriver used in resolving conflicts on commits is now pointing to V2
- It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble

Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for:

- Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section)
- "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string
- "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo
- "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added)
- Files generated from thrift_library rules that do not support "rust" language are removed
- Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored

Others:

- Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo`
- Updated following files to use V2:
  - `common/rust/tools/reindeer/version-bump`
  - `remote_execution/rust/setup.sh`
- Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing

Reviewed By: zertosh

Differential Revision: D26728789

fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
2021-03-02 06:43:29 -08:00
Lukas Piatkowski
f317302b0f autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.

Reviewed By: ikostia

Differential Revision: D26618363

fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-25 15:10:56 -08:00
Stefan Filip
d45198d79d config: sync repo.thrift changes to fbcode
Summary:
```
configerator-thrift-updater scm/mononoke/repos/repos.thrift
```

(Note: this ignores all push blocking failures!)

Reviewed By: quark-zju

Differential Revision: D26628122

fbshipit-source-id: 69124458996393f71098f707b375e184b929e466
2021-02-24 16:51:50 -08:00
Lukas Piatkowski
279eb2b538 autocargo v1: changes to how thrift-related generation is done to match v2
Summary: Done some reordering of fields in Cargo.toml, added test and doctest = false, name of the target that generated the Cargo.toml file and sorted the cratemap.

Reviewed By: ahornby

Differential Revision: D26581275

fbshipit-source-id: 4c363369438c72d43d8ccf4799f103ff092457cc
2021-02-23 11:38:45 -08:00
Lukas Piatkowski
cd0b6d50e2 autocargo v1: changes to match autocargo v2 generation results.
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.

Reviewed By: zertosh

Differential Revision: D26544150

fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 11:03:55 -08:00
Kostia Balytskyi
ce88c33192 observability: import thrift changes
Summary: Just import D26423954

Reviewed By: farnz

Differential Revision: D26482499

fbshipit-source-id: 4a1b2354109bfdb84ad07a167ec206159751c203
2021-02-18 07:35:40 -08:00
Jan Mazur
f9376fce90 load_limiter: static, sliced rate limitting
Summary: We would like to consistently rate limit a percentage of hosts from a specific tier expressed as a subset of identities.

Reviewed By: krallin

Differential Revision: D26312370

fbshipit-source-id: d3fc9e892a8c9f62e22b079fa947a85078831687
2021-02-15 09:56:56 -08:00
Kostia Balytskyi
879f85631b thrift: import changes to limits.thrift
Summary: This imports D26374072. See that diff for details.

Reviewed By: farnz

Differential Revision: D26374352

fbshipit-source-id: 71f39b5c606915bfbf421b366812fd40ebb6b4f9
2021-02-12 01:31:55 -08:00
Alex Hornby
246937b6f3 mononoke: sync configerator repos thrift
Summary: Sync across the thrift schema

Differential Revision: D26276002

fbshipit-source-id: 24f4a8a459f2fcdb18a14627dc7bb5e8d835b829
2021-02-05 08:22:46 -08:00
Jan Mazur
27f37a6b5e mononoke/lfs_server: remove unused struct member
Summary:
Syncing thrift structure modified in D26221917 to fbcode.
I think it's safe to land D26221917 first and then land this diff.

Reviewed By: ahornby

Differential Revision: D26195881

fbshipit-source-id: 14d29a9374e71e55d93818a0362ee8514710295f
2021-02-03 04:49:08 -08:00
Thomas Orozco
90e4bbca0a mononoke/server: add a trivial control API and use it to fix some tests
Summary:
We have some tests that are a bit racy because they write bookmarks from
one process then look at them from another process, but that can fail because
we have a cache on bookmarks that holds them for 2 seconds.

This is normally not a huge issue because we don't access said bookmarks, but
now that as of my earlier diff we run a warm bookmarks cache, it *is* a
problem. This fixes that. We can expand it later to do things like reload
tunables, but for now this satisfies one basic use case.

Reviewed By: ahornby

Differential Revision: D26149371

fbshipit-source-id: 11c7f64b1ae45f6a0de142be25ab367cb25df567
2021-02-01 07:53:17 -08:00
Jan Mazur
c2fcf857bd identity set throttling
Summary:
Before this change we could throttle only based on one identity matching one of the identities from user's set of identities.

Now we'll be able to match a subset of user's identities.

Depends on D26125638.

Reviewed By: krallin

Differential Revision: D26125637

fbshipit-source-id: 534326264b9093e46fbdda846516fdaceb40c931
2021-01-29 07:43:56 -08:00
Thomas Orozco
d863816e0e mononoke: sync Configerator repos schema
Summary: See D26019054 for what this does.

Reviewed By: StanislavGlebik

Differential Revision: D26019125

fbshipit-source-id: 071f0115d5ee8435bbc7ff5208ace8ef6e948272
2021-01-27 08:32:38 -08:00
Evgenii Kazakov
e05104c8c2 mononoke/tunables: add by_repo tunables
Summary: Added new bool tunables type, depending on repos.

Reviewed By: ikostia

Differential Revision: D26025163

fbshipit-source-id: 5c2fa846b3c115bea683be249e92d495d28c038c
2021-01-27 01:50:05 -08:00
Daniel Xu
5715e58fce Add version specificiation to internal dependencies
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.

Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.

See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .

Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.

Reviewed By: lukaspiatkowski

Differential Revision: D26028982

fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
2021-01-25 22:10:24 -08:00
Evgenii Kazakov
42486b81bd mononoke/tunables: add by_repo maps for tunables
Reviewed By: ikostia

Differential Revision: D25955587

fbshipit-source-id: b8c534f711da3f5abbb0cdf128c6fdcf25b199aa
2021-01-20 04:20:25 -08:00
Mark Juggurnauth-Thomas
0afac29010 sync configerator thrift
Reviewed By: StanislavGlebik

Differential Revision: D25952897

fbshipit-source-id: 0fbd43fa36c2cb749a6eed103b88840681a9f706
2021-01-19 08:11:11 -08:00
Ava Spataru
591b6782f2 Mononoke LFS: allow sharding content across > 1 host
Summary:
To ensure that a specific content gets sharded to multiple tasks (by tasks_per_content config parameter) each content can have multiple routing keys. These are the SHA256 (as before) or the SHA256 suffixed with "-1", "-2" etc..

When the routing key is generated for an item, one of them is picked randomly.

Added config constraint to ensure that the number of tasks parameter is at least 1.

Reviewed By: krallin

Differential Revision: D25886402

fbshipit-source-id: fb8911dad07d2f0b6bbf57b4ede084428fe6c49d
2021-01-15 06:53:14 -08:00
Daniel Xu
f83fb64f3e Do full regen
Summary:
D25850262 (772680f2f7) updated a dep but only did a partial regen. This broke
other diffs.

This diff does a full regen to sync up again.

Differential Revision: D25872444

fbshipit-source-id: b5749b3d634ffe355a5a3cf6e94066ca5158001f
2021-01-11 16:14:36 -08:00
Egor Tkachenko
ccc9e4e39a Verification of backup repos
Summary:
This package will be used by chronos job on regular basis. It will spawn legocastle jobs for each of backup repos, pull them in together with origin repo from production tier and compare hashes of commits.

`fbpkg.builder` allows to create package and add it to contbuild together with creating a binary.
`https://www.internalfb.com/intern/wiki/Fbpkg/fbpkg.builder/#overview`

Reviewed By: HarveyHunt

Differential Revision: D25573914

fbshipit-source-id: 3f7ff3a6b2d7acefed46683122b72c5bc862e1aa
2020-12-17 14:36:11 -08:00
Egor Tkachenko
7ba932a332 Add field backup_source_name to repo_config
Summary:
Remove logic of striping `_backup` suffix and get source repo from configerator.
Has dependency on D25571595. And should be landed after it.

Reviewed By: krallin

Differential Revision: D25420829

fbshipit-source-id: 50e2fe2bede849ba622dcbde2a360b08e640901a
2020-12-17 04:30:03 -08:00
Egor Tkachenko
0109baf930 Remove optional backup source name string field
Summary: I'm changing this field to the Struct type in later diffs, so need to delete it first. It wasn't used anywhere yet, next diff in the stack was supposed to use it, so it's safe to delete it

Reviewed By: HarveyHunt

Differential Revision: D25584263

fbshipit-source-id: 1d0aab01b52dc575cd6d466857ca850cbf8cd973
2020-12-16 05:30:37 -08:00
Mark Juggurnauth-Thomas
01251de9bb sync configerator thrift
Reviewed By: StanislavGlebik

Differential Revision: D25246318

fbshipit-source-id: fd97296e3540b43c72af00ab045c81c496e06728
2020-12-14 09:24:57 -08:00
Thomas Orozco
5194707ccf mononoke/lfs_server: import config structs from Configerator
Summary: See D24360844.

Reviewed By: HarveyHunt

Differential Revision: D24361313

fbshipit-source-id: a32c5d5ed6039bba9d8fbb67dcb8b0e724b7d395
2020-12-03 06:17:06 -08:00
Kostia Balytskyi
fd2b114424 observability: import thrift
Summary: Imports D25231497

Reviewed By: ahornby

Differential Revision: D25231581

fbshipit-source-id: e8be84e0cc25d2ae6b952b4f856fb427c82ca12b
2020-12-02 12:10:22 -08:00
Kostia Balytskyi
8d4db7b334 rate_limits: import thrift changes
Summary: This just imports D24871255.

Reviewed By: krallin

Differential Revision: D24871790

fbshipit-source-id: e5367eef5dfa4ce261fe9474f53df5ebb3cf6c4d
2020-11-13 05:18:33 -08:00
Mateusz Kwapich
10bc4843d3 new mode to reverse-fill bookmarks
Summary:
Adds `--reversefill` mode to bookmarks filler that fetches bookmark updates
from the queue and syncs them to infinitepush database.

Reviewed By: farnz

Differential Revision: D24538317

fbshipit-source-id: 5ac7ef601f2ff120c4efd8df08a416e00df0ceb9
2020-11-10 02:58:00 -08:00
Mateusz Kwapich
89fd335b1e populate bookmark filler queue on scratch bookmark updates
Summary:
Summarry:
This is the first part of syncing new scratch bookmark pushes from Mononoke to
Mercurial: on each bookmark movement we log this bookmark movement to filler's
queue.

Reviewed By: liubov-dmitrieva

Differential Revision: D24480546

fbshipit-source-id: 27103b4b4f8c4600aaf485826db2936eaffcc4a9
2020-11-10 02:58:00 -08:00
Jan Mazur
8152776a35 bypassing hooks for members of scm
Summary: Members of `scm` hipster group will be able to push to mononoke bypassing hooks when `BYPASS_ALL_HOOKS` pushvar is passed.

Reviewed By: krallin

Differential Revision: D24477468

fbshipit-source-id: ac910bf27e5510e1975c4a7cd0bfeff5216da70e
2020-10-29 10:14:25 -07:00
Simon Farnsworth
d0110ce4a8 Support marking garbage in SQLBlob
Summary:
We want to be able to detect garbage blobs by looking at generation numbers.

Update generation numbers on put, and have a mark command exist to mark blobs as not garbage.

Reviewed By: ahornby

Differential Revision: D23989289

fbshipit-source-id: d96f38649151e3dbd5297cffc262776e74f6cc86
2020-10-28 10:23:55 -07:00
Thomas Orozco
24560ded2e thrift: call hooks from Rust codegen
Summary:
This updates Thrift codegen for Rust to call hooks on a context stack in a
similar fashion as the C++ codegen.

There is still a bit of room for improvement here. Notably, we don't know how
much data we actually read or sent to the client, so for now those are
hard-coded to zero in the codegen. That's better than not calling those hooks
at all (which is what happens right now), but it could stand to be improved.

Reviewed By: jsgf

Differential Revision: D24445298

fbshipit-source-id: 470daf03057424dc300b6a193668be835ae28452
2020-10-22 17:01:47 -07:00
Mateusz Kwapich
42a783999d enable single process to work on multiple repos.
Summary:
This change enables the filler job to work on all repos available instead of a
single one. We are still going to be able to dedicate the job to a certain repo
(by crafting a config with a single repo enabled) but we can put the entire
long tail for low-traffic repos under a single job.

This requires D24110335 to land in configerator to work.

Reviewed By: krallin

Differential Revision: D24136239

fbshipit-source-id: 4b77d1667c37cc55f11c3087b02a09dbae29db0f
2020-10-08 05:38:01 -07:00
Egor Tkachenko
4d0ae8ae41 Added S3 blobstore
Summary:
Implemented S3 blobstore
Isilon implements S3 as 1:1 mapping into filesystem, and it limits the maximum number of blobs in the single directory. To overcome it lets shard the keys using base64 encoding and making 2 level dir structure with 2 chars dir names.

Reviewed By: krallin

Differential Revision: D23562541

fbshipit-source-id: c87aca2410381a07babb191cbd8cf28233556e03
2020-09-22 04:15:34 -07:00
Stanislau Hlebik
0740f99f13 mononoke: allow logging censored scuba accesses to file
Summary:
In the next diff I'm going to add log-only mode to redaction, and it would be
good to have a way of testing it (i.e. testing that it actually logs accesses
to bad keys).

In this diff let's use a config option that allows logging censored scuba
accesses to file, and let's update redaction integration test to use it

Reviewed By: ikostia

Differential Revision: D23537797

fbshipit-source-id: 69af2f05b86bdc0ff6145979f211ddd4f43142d2
2020-09-04 07:37:14 -07:00
Lukas Piatkowski
eebdc0b896 mononoke/metaconfig: sync thrift changes from configerator for HookConfig
Summary: Use the new fields from RawHookConfig in HookConfig

Reviewed By: StanislavGlebik

Differential Revision: D23499766

fbshipit-source-id: 43e9d2dfdcfb0fa0dd4de6310ea0013db1b69474
2020-09-04 02:02:06 -07:00
David Tolnay
75c2118e01 Remove crate_root from Rust dependency info
Reviewed By: danobi

Differential Revision: D23430948

fbshipit-source-id: c4b374021325fc247121ceecd0e82a0291aa75d6
2020-08-31 14:43:24 -07:00
Stefan Filip
a86409361c thrift: sync repos.thrift to fbcode
Summary:
`fbcode` sync after adding the `update_algorithm` option to
`RawSegmentedChangelogConfig`.

Reviewed By: quark-zju

Differential Revision: D23376694

fbshipit-source-id: df8a378303ec00d5d7d7ac3fff7a4a8cffc99328
2020-08-28 16:37:57 -07:00
Mark Thomas
f15976637e bookmarks_movement: restrict bookmarks that are marked allow_only_external_sync
Reviewed By: StanislavGlebik

Differential Revision: D23294907

fbshipit-source-id: ed89e5fd841e7d516b5d259c1f5de4e9f8f40ee3
2020-08-25 09:14:09 -07:00
Mark Thomas
02e382b203 sync repos.thrift
Reviewed By: StanislavGlebik

Differential Revision: D23222561

fbshipit-source-id: 09c25e11b5e022547c1a1363a4cbe1926714be86
2020-08-25 09:14:08 -07:00
Viet Hung Nguyen
dcb565409d mononoke: Sync repo.thrift changes to fbcode
Summary:
Related commit: D23213192

Currently repo configs don't store their phabricator callsigns and because of this,
it's hard to generate them with a given repo name (e.g. fbsource) in Mononoke.
This diff synced the changes between configerator (check the other commit) and fbsource.

Reviewed By: farnz

Differential Revision: D23214677

fbshipit-source-id: cf102d69958c7dc32f8b72cecbfad017d0a31329
2020-08-19 10:48:41 -07:00
Stanislau Hlebik
f68531f81c mononoke: add a flag to disable short history fetching
Summary: Let's use new flag to enable/disable short history for getpack request

Reviewed By: krallin

Differential Revision: D23080200

fbshipit-source-id: 7aa0be6ded0601fa4d31d4b9ff8792a4f8d91b19
2020-08-13 10:37:40 -07:00
Stanislau Hlebik
97041ca4dc sync configerator thrift update
Reviewed By: farnz

Differential Revision: D23057328

fbshipit-source-id: b57fced1a91e47006ca5d44c5bbcebd1a516e7d0
2020-08-12 12:03:18 -07:00
Simon Farnsworth
a9b8793d2d Add a write-mostly blobstore mode for populating blobstores
Summary:
We're going to add an SQL blobstore to our existing multiplex, which won't have all the blobs initially.

In order to populate it safely, we want to have normal operations filling it with the latest data, and then backfill from Manifold; once we're confident all the data is in here, we can switch to normal mode, and never have an excessive number of reads of blobs that we know aren't in the new blobstore.

Reviewed By: krallin

Differential Revision: D22820501

fbshipit-source-id: 5f1c78ad94136b97ae3ac273a83792ab9ac591a9
2020-08-03 04:36:19 -07:00
Stanislau Hlebik
6557e2e364 sync D22817643 to fbsource
Reviewed By: farnz

Differential Revision: D22817699

fbshipit-source-id: 015eb428fad6ae7726e5072d302e66cdb3551b5e
2020-07-30 08:41:15 -07:00
Harvey Hunt
0e7738768f mononoke: Sync repo.thrift changes to fbcode
Summary: Sync the changes from configerator.

Reviewed By: krallin

Differential Revision: D22762452

fbshipit-source-id: 56f3e3bf70c58be38947b7d0fd4737c5ef895d3c
2020-07-27 06:21:20 -07:00
Arun Kulshreshtha
5f0181f48c Regenerate all Cargo.tomls after upgrade to futures 0.3.5
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.

Reviewed By: dtolnay

Differential Revision: D22403809

fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
2020-07-06 20:49:43 -07:00
Stanislau Hlebik
783e1fcc3b mononoke: sync configerator update
Reviewed By: krallin

Differential Revision: D22373947

fbshipit-source-id: b4aa9a4767ff135cd07ffccbbbf8dcaa5de65703
2020-07-03 09:58:46 -07:00
Simon Farnsworth
aa00466319 Sync hook_type change from D22282802
Summary: Also fix up the parser test that now fails with this change

Reviewed By: StanislavGlebik

Differential Revision: D22306340

fbshipit-source-id: 820aad48068471b03cbc1c42107c443bfa680607
2020-06-30 11:20:54 -07:00
Stanislau Hlebik
c3276b4c5e mononoke: sync configerator update
Reviewed By: krallin

Differential Revision: D22211347

fbshipit-source-id: 2bb2d02277d06462e4cda9347bfd8a2ae3fe7222
2020-06-25 05:51:29 -07:00
Lukas Piatkowski
14f7dd70e4 Re-sync with internal repository 2020-06-24 21:35:50 +02:00