Summary:
The current Mononoke Blobstore Trace scuba table is used with idea of having a record per blobstore and operation. This diff adds logging to the new scuba table of the combined multiplexed operations' outcome, like time spent on the `put` including sync-queue and blobstore write or tracking record of the "some failed others none" cases in `get/is_present`.
This helps to see the real time spent on writes and reads and to assess the impact of changes coming in `get` and `is_present`.
Reviewed By: ahornby
Differential Revision: D30248284
fbshipit-source-id: f79050ced32ba77bd2e220e242407bcd711a9b6d
Summary:
The use of dyn traits of the Thrift-generated server traits was emitting future compatibility warnings with recent versions of rustc, due to a fixed soundness hole in the trait object system:
```
error: the trait `x_account_aggregator_if::server::XAccountAggregator` cannot be made into an object
|
= this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
note: for a trait to be "object safe" it needs to allow building a vtable to allow the call to be resolvable dynamically; for more information visit <https://doc.rust-lang.org/reference/items/traits.html#object-safety>
```
This diff pulls in https://github.com/dtolnay/async-trait/releases/tag/0.1.51 which results in the Thrift-generated server traits no longer hitting the problematic pattern.
Reviewed By: zertosh
Differential Revision: D29979939
fbshipit-source-id: 3e6e976181bfcf35ed453ae681baeb76a634ddda
Summary:
This option would let us tell that a given bookmark (or bookmarks if they are
specified via a regex) is allowed to move only if it stays an ancestor of a
given bookmark.
Note - this is a sev followup, and we intend to use it for */stable bookmarks
(e.g. fbcode/stable, fbsource/stable etc). They are always intended to be an
ancestor of master
Reviewed By: krallin
Differential Revision: D29878144
fbshipit-source-id: a5ce08a09328e6a19af4d233c1a273a5e620b9ce
Summary: As discussed, extends Mononoke service to support commits w/o parents for the AI Infra usecase.
Reviewed By: markbt
Differential Revision: D29810303
fbshipit-source-id: f07fd7f1521ffe1cea85f1f54e71fe37fc39bb62
Summary:
Now that the new `rate_limiting` crate is being used by LFS server we
can remove the throttle limits code and config.
Differential Revision: D29396505
fbshipit-source-id: 19638bd93ad9dea2638e8501837c6c13e4dd48ff
Summary:
Update Mononoke server to use the new `rate_limiting` crate. This diff
also removes the old rate limiting library.
Differential Revision: D29396507
fbshipit-source-id: 05adb9322705b771a739c8bcaf2816c95218a42d
Summary:
Replace the LFS server's load shedding logic with that provided by the
`rate_limiting` crate.
Differential Revision: D29396503
fbshipit-source-id: a71812a55b9c9f111ee2861dc1b131ad20ca82d2
Summary:
Add a new rate limiting library that also supports load shedding when
an individual server is overloaded. This library provides a few benefits:
- The code can be shared between the LFS server and Mononoke server.
- The library supports more complex expressions of which clients to apply a
rate limit to (e.g. 10% of sandcastle and mactest machines).
- The rate limiting `Target` can be expanded in the future as the client
provides more information (e.g. client region).
- Mononoke server will be able to loadshed if an individual host is overloaded,
as we can currently do with the LFS server.
I've added this library as a separate crate rather than rewriting
`load_limiter` to make it easier to review. The next diff will make use of the
new library and remove the old one.
Reviewed By: StanislavGlebik
Differential Revision: D29396509
fbshipit-source-id: 2fbc04e266b18392062e6f952075efd5e24e89ba
Summary:
Adds class `ConfigeratorRedactedBlobs` that reads redaction data from configerator, and reloads it when necessary.
The class does this:
- Reads `RedactionSets` from configerator.
- For each key there, read `RedactionConfigBlobstore` looking for a `RedactionKeyList` with that key (these were populated by D29033598).
- From the keys listed, builds the map of redacted blobs, with the same format as before when it was fetched from XDB.
- Periodically checks if config changed. If so, reload the map of redactions. (should only happen when we land a new config change to redaction, which should be very very rare)
- We use ArcSwap to keep the config, as a good way to provide read-only access with eventual reloading.
Not implemented on this diff:
- Creation of `ConfigeratorRedactedBlobs`, or adding it to `RedactedBlobs` enum.
It's not used in this diff, will be used in the future, I split it mostly to make it easier to review.
Reviewed By: StanislavGlebik
Differential Revision: D29033595
fbshipit-source-id: 36603685433b6dd153f2c23123907f7311c20a32
Summary: Syncing configerator changes from D29360425, and fixing all tests. Not used yet.
Reviewed By: markbt
Differential Revision: D29363416
fbshipit-source-id: d2de13d32bcec2e7fbff20204be8d9a8d65c0efe
Summary:
This reads the config added on D29305462. It populates it into `CommonConfig` struct, and also adds it to `RepoFactory`, but doesn't yet use it anywhere, this will be done on the next diff.
There is a single behaviour change in this diff, which I believe should be harmless but is noted in the comments in case it isn't.
Reviewed By: markbt
Differential Revision: D29272581
fbshipit-source-id: 62cd7dc78478c1d8cb212eafdd789527ead50ef6
Summary: I think someone landed a dependency change or something and forgot to update autocargo
Reviewed By: dtolnay
Differential Revision: D29402335
fbshipit-source-id: e9a4906bf249470351c2984ef64dfba9daac8891
Summary: Update versions for several of the crates we depend on.
Reviewed By: danobi
Differential Revision: D29165283
fbshipit-source-id: baaa9fa106b7dad000f93d2eefa95867ac46e5a1
Summary: It will be used in the next diff to disable only certain derived data types
Reviewed By: yancouto
Differential Revision: D29031082
fbshipit-source-id: 9621a5eec522e369fef2b78e94599a056eda5317
Summary:
Syncing thrift files from D28931534
Will use it in following diffs
Reviewed By: farnz
Differential Revision: D28937805
fbshipit-source-id: d3ea22ae2c402aa244f3972d9a70516776f77a11
Summary: LFS can reload config dynamically and can do it per-region. Let's do this instead rolling this change on the VIP, because here a rollback could happen in no time.
Reviewed By: krallin
Differential Revision: D28534346
fbshipit-source-id: 131ef2d0d084b27719a9aaf4e29003b06c578a67
Summary:
During S231236 the LFS servers were using a lot of CPU and were
compressing blobs sent to clients. Thomas hotfixed the servers to disable
compression in order to save some CPU. Instead of having to rebuild and
redeploy the LFS server, update it to be able to disable compression using
configerator.
The `disable_compression` option will disable compression of downloads
globally. The `disable_compression_identities` allows us to disable compression
for some group of identities.
I also refactored some of the shared code from `download` and `download_sha256`
into a new function, `download_inner`.
Reviewed By: krallin
Differential Revision: D28382405
fbshipit-source-id: 792f10a9e3bb32b56ef87aa8e0b2c4b098567579
Summary: More context in previous diff in the stack.
Reviewed By: krallin
Differential Revision: D27852299
fbshipit-source-id: dc06b29794d0c4e8ff6bf3f44507a64c06f01771
Summary:
This diff adds boilerplate needed to have a cfgr-sourced (D27623488)
`megarepo_configs.thrift` live in `fbcode` successfully. This diff does
introduce any new use-cases for the file.
Note that I suspect that it's better to sync this file to www from fbcode, not from configerator, because the import path for `megarepo_configs.thrfit` in `source_control.thrift` should be the same across `fbcode` and `www`, and files are imported differently into `www` from `fbcode` and `configerator` unfortunately. As another justification, I think keanu's [`traffic_types.thrift`](https://www.internalfb.com/intern/diffusion/WWW/browsefile/master/flib/intern/thrift/if/fbcode/configerator/structs/keanu/glb/traffic_types.thrift) is synced into `www` from `fbcode` also, not from its source of truth in `configerator`.
Reviewed By: StanislavGlebik
Differential Revision: D27647707
fbshipit-source-id: 2bb2fdf1d4a68e686db302783b68062f3d95fb88
Summary:
We need to bump SCS counters expressing Mononoke's QPS. They will look something like:
`requests:mononoke:oregon:carolina` for requests coming from proxygen in prn and mononoke in frc.
CSLB expects regions' full names.
We're getting src region from proxygen as a header.
Reviewed By: krallin
Differential Revision: D27082868
fbshipit-source-id: 12accb8a9df5cf6a80c2c281d2f61ac1e68176d1
Summary:
Currently we can only limit which users are allowed to move a bookmark by a
regex. We also want to allow specifying a hipster group.
Reviewed By: krallin
Differential Revision: D27156690
fbshipit-source-id: 99a5678a82f4c34ed2e57625361ba7cdb08ed839
Summary: Bring across the thrift changes so can code against them.
Reviewed By: krallin
Differential Revision: D27116899
fbshipit-source-id: 27bf6f23bebbc43d6c4d6c668ff905b72b0eb0f9
Summary:
S3 blobstore is sensitive to the number of open connections and it causes perf
problems. Let's add a way to limit the number of simultaneous open connections.
Also see D26911520
Reviewed By: krallin
Differential Revision: D26911679
fbshipit-source-id: 1e3e4922b26d0a29c94c89b4aa76b9b5928effe6
Summary:
Normally we prohibit landing commits that might accidentally
change the x-repo mapping. However we do want to allow landing commits like
that to backup repos, because backup repos should have all commits as their
counterpart repositories.
This change also has another side-effect - we don't call `load_additional_changeset()`
which can be very expensive for backup repos because of the issues in configuration -
in particular, we don't have `hooks_ancestors_of` option set, and that caused all ancestors to be considered
as "additional changesets". It would make since more properly later.
Differential Revision: D26883910
fbshipit-source-id: 07ceb7b96bc6cae851ac6ff57071eff5cef387e4
Summary:
This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here.
Arc lint support:
V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff:
- Adds running `autocargo` as part of `arc lint-rust`
Mergedriver update:
- Mergedriver used in resolving conflicts on commits is now pointing to V2
- It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble
Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for:
- Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section)
- "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string
- "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo
- "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added)
- Files generated from thrift_library rules that do not support "rust" language are removed
- Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored
Others:
- Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo`
- Updated following files to use V2:
- `common/rust/tools/reindeer/version-bump`
- `remote_execution/rust/setup.sh`
- Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing
Reviewed By: zertosh
Differential Revision: D26728789
fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.
Reviewed By: ikostia
Differential Revision: D26618363
fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
Summary: Done some reordering of fields in Cargo.toml, added test and doctest = false, name of the target that generated the Cargo.toml file and sorted the cratemap.
Reviewed By: ahornby
Differential Revision: D26581275
fbshipit-source-id: 4c363369438c72d43d8ccf4799f103ff092457cc
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.
Reviewed By: zertosh
Differential Revision: D26544150
fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
Summary: We would like to consistently rate limit a percentage of hosts from a specific tier expressed as a subset of identities.
Reviewed By: krallin
Differential Revision: D26312370
fbshipit-source-id: d3fc9e892a8c9f62e22b079fa947a85078831687
Summary: This imports D26374072. See that diff for details.
Reviewed By: farnz
Differential Revision: D26374352
fbshipit-source-id: 71f39b5c606915bfbf421b366812fd40ebb6b4f9
Summary:
Syncing thrift structure modified in D26221917 to fbcode.
I think it's safe to land D26221917 first and then land this diff.
Reviewed By: ahornby
Differential Revision: D26195881
fbshipit-source-id: 14d29a9374e71e55d93818a0362ee8514710295f
Summary:
We have some tests that are a bit racy because they write bookmarks from
one process then look at them from another process, but that can fail because
we have a cache on bookmarks that holds them for 2 seconds.
This is normally not a huge issue because we don't access said bookmarks, but
now that as of my earlier diff we run a warm bookmarks cache, it *is* a
problem. This fixes that. We can expand it later to do things like reload
tunables, but for now this satisfies one basic use case.
Reviewed By: ahornby
Differential Revision: D26149371
fbshipit-source-id: 11c7f64b1ae45f6a0de142be25ab367cb25df567
Summary:
Before this change we could throttle only based on one identity matching one of the identities from user's set of identities.
Now we'll be able to match a subset of user's identities.
Depends on D26125638.
Reviewed By: krallin
Differential Revision: D26125637
fbshipit-source-id: 534326264b9093e46fbdda846516fdaceb40c931
Summary: See D26019054 for what this does.
Reviewed By: StanislavGlebik
Differential Revision: D26019125
fbshipit-source-id: 071f0115d5ee8435bbc7ff5208ace8ef6e948272
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.
Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.
See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .
Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.
Reviewed By: lukaspiatkowski
Differential Revision: D26028982
fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
Summary:
To ensure that a specific content gets sharded to multiple tasks (by tasks_per_content config parameter) each content can have multiple routing keys. These are the SHA256 (as before) or the SHA256 suffixed with "-1", "-2" etc..
When the routing key is generated for an item, one of them is picked randomly.
Added config constraint to ensure that the number of tasks parameter is at least 1.
Reviewed By: krallin
Differential Revision: D25886402
fbshipit-source-id: fb8911dad07d2f0b6bbf57b4ede084428fe6c49d
Summary:
D25850262 (772680f2f7) updated a dep but only did a partial regen. This broke
other diffs.
This diff does a full regen to sync up again.
Differential Revision: D25872444
fbshipit-source-id: b5749b3d634ffe355a5a3cf6e94066ca5158001f
