Summary:
- udpated thrift definition to support i64 ints;
- updated metaconfig loader type to parse and load the new 64bit properties;
Reviewed By: mzr
Differential Revision: D34962606
fbshipit-source-id: 95653139b2149bb9df2899b8b703fa58ee4473c3
Summary:
This diff does the following:
- Updates the tracing crates to the latest versions available. Changes include:
- Experimental support for [Valuable](https://tokio.rs/blog/2021-05-valuable), which makes recording values much more like how serde does. Valuable also makes it easier to write a tracing-slog crate due to some implementation details of slog where buffering of values is necessary.
- Combinators for combining filters in tracing-subscriber.
- A `MakeWriter` implementation for `RollingFileAppender` in `tracing_appender`, which allows the `RollingFileAppender` without a background thread to being spun up.
- Reduced the disabled span drop overhead by 50-70%—this might be most apparent in tests that don't set a subscriber.
- Move the tokio-console off git dependencies and to published versions on crates.io.
This diff also updates the following, non-exhaustive list of crates:
- parking_lot
- prost
- thread_local
- tonic
- hyper
- httparse
- h2
allow-large-files
Reviewed By: jsgf, fanzeyi
Differential Revision: D34801437
fbshipit-source-id: 2409de9194c8e00647ac5e4c514325fd2e23d247
Summary: Added a new configuration enum, `BubbleDeletionMode` that indicates if the bubble needs to undergo soft or hard deletion.
Reviewed By: farnz
Differential Revision: D34895758
fbshipit-source-id: ec096ff2e0d3580049e3ee24cad1f2f6f1e46d70
Summary: Adding a new field to the `RepoConfig`. Not used anywhere yet.
Reviewed By: markbt
Differential Revision: D34555579
fbshipit-source-id: cba8e3eeeb1ebbc2341f1bf55cc741bdfa7a1279
Summary:
Traffic replay is gone. Now we can delete code that dumped wireproto traffic.
The logging that's left could be still somewhat useful: https://fburl.com/scuba/mononoke_test_perf/uismnrv9
Reviewed By: HarveyHunt
Differential Revision: D33898167
fbshipit-source-id: f5f12f3626c578ef90db99a45e5749fe8a94049f
Summary:
The LFS server included separate fields in its config for object
popularity category and threshold. However, they have been replaced by a dedicated
`ObjectPopularity` struct that also includes a `window` field.
Remove the old configuration from the code, as it's unused and confusing.
Reviewed By: mzr
Differential Revision: D34214336
fbshipit-source-id: 7c3ecd5aa8d7e882783c8ff8630edf48999428e9
Summary: Run autocargo before doing any other changes, likely some or all of this will rebase away
Reviewed By: c-ryan747
Differential Revision: D33790131
fbshipit-source-id: fe6c761a09b300e482b5eb9c9c1465716d35dded
Summary:
Currently, blobstore get on multiplexed works like this:
1. If the blob is found on ANY blobstore, return it right away and don't care about others.
2. If the blob is not present on ALL blobstores, return it as not present.
3. If it is not present on some and the get errors on other blobstores, then the whole get errors.
This makes get requests much slower when the blob is not present (since they ALWAYS need to wait for ALL blobstores), while put requests do fine as we only need to guarantee it is written to SOME blobstores.
This can cause problems even when a single blobstore is borked, which is not ideal (see [1] for a real life case).
This diff keeps the behaviour of 1, but changes the behaviour of 2 and 3, so that mononoke only needs to check SOME blobstores to determine the blob doesn't exist.
If there are N blobstores, and put quorum is P (that is, we say a write is successful if P puts are successful), then if N - P + 1 blobstores do not contain a blob, then there's no way a successful put happened for that blob (by pigeonhole principle). D32700029 uses that value for quorums, though it can be tweaked if desired.
Reviewed By: StanislavGlebik
Differential Revision: D32699218
fbshipit-source-id: 0e1d1f9640acb3770d8830b5868c57e1748e3c63
Summary:
For invocations of `ensure!` where the argument is one of the comparison operators `==` `<=` `<` `!=` `>=` `>` and the lhs and rhs of the operator both have a `Debug` impl, we now render both sides as part of the error message.
### Example:
```
ensure!(flags.len() <= 40);
```
```
ensure!(kind == Kind::File);
```
Before:
```
Condition failed: `flags.len() <= 40`
Condition failed: `kind == Kind::File`
```
After:
```
Condition failed: `flags.len() <= 40` (99 vs 40)
Condition failed: `kind == Kind::File` (Symlink vs File)
```
Reviewed By: zertosh
Differential Revision: D32598267
fbshipit-source-id: 70fdab6d8f0e3ceceb98e651eab0a73795cadaeb
Summary:
This release helps catch places that the `anyhow!` or `bail!` macros are invoked with a missing fmt argument, such as:
```
- anyhow!("unsupported redirect operation {:?}")
+ anyhow!("unsupported redirect operation {:?}", op)
```
All instances of this in fbcode were fixed in D32266567 (2f440eac7e).
Reviewed By: wqfish
Differential Revision: D32564287
fbshipit-source-id: e9cf5a1376e6c83343de3f3c6ba2925d88334a30
Summary: Bring the config in line with the oss branch names
Reviewed By: mzr
Differential Revision: D32109171
fbshipit-source-id: f87a0070b0ef06a49c86cacdfdc7f5c8b02f3fbf
Summary:
Use parameters added in D32018280 to avoid using current version to generate
bookmark renamer
Reviewed By: mitrandir77
Differential Revision: D32023697
fbshipit-source-id: 5256bd625ee028325f03d485f59bed47ec390365
Summary:
We want to allow move some bookmarks without hooks if these conditions are met:
1) The affected commits are public
2) Bookmark doesn't add any new commits to the repo (i.e. it's just a bookmark
move, nothing else)
See T98731740 and D31961821 for motivation
Reviewed By: mitrandir77
Differential Revision: D31961877
fbshipit-source-id: 4966a3bdcf5e598194f54f7d8b10f8b780a25f30
Summary: Generated by `sed -i 's/edition = "2018"/edition = "2021"/' common/rust/cargo_from_buck/project_configs/*.toml` and `cargo autocargo` following the autocargo code change in D31830985 and its corresponding MSDK rebuild in D31832311.
Reviewed By: zertosh
Differential Revision: D31832698
fbshipit-source-id: db79024cad311467d5b7c13b7abedc6464f09366
Summary:
Mononoke's config thrift types should opt-in to the `rust.exhaustive`
annotation. This will mean we can statically check that all uses of these
types handle any additions we make in the future.
Reviewed By: farnz
Differential Revision: D31828315
fbshipit-source-id: 85fda33b86fabceef03644037e84e063bbcd6e9a
Summary:
Add configuration to prefix mapping keys with an additional prefix. This
will allow us to rederive data without changing the underlying format by
rederiving with new unique keys.
Reviewed By: StanislavGlebik
Differential Revision: D31759248
fbshipit-source-id: 7bf43cda82fa4c5c30e9a41cc09ba5d9f5efa8e9
Summary: Bump in preparation for a new release
Reviewed By: brianc118
Differential Revision: D31657479
fbshipit-source-id: eeb2a08c54f6db75b6e515ca0966637f31b906a8
Summary: I just landed D30899990. I'm landing this to sync the config, still without using it.
Differential Revision: D31472892
fbshipit-source-id: 5e18c4c3529118ef81880c886f0d8b9428efcbf4
Summary: Need this for cargo check/rust-analyzer to work.
Reviewed By: guswynn
Differential Revision: D31319911
fbshipit-source-id: ebd3fa72d8fc3667391a2067f95cab9e5f53301f
Summary:
In some cases (e.g. when master bookmark moves backwards) there might be
commits in segmented changelog that are not ancestors of master. When reseeding
we still want to build segments for these chagnesets, and this is what this
diff does (see D30898955 for more details about why we want to build segments
for these changesets).
Reviewed By: quark-zju
Differential Revision: D30996484
fbshipit-source-id: 864aaaacfc04d6169afd3d04ebcb6096ae2514e5
Summary:
`rust_include_srcs` is supported on `thrift_library` as a way of including other Rust code in the generated crate, generally used to implement other traits on the generated types.
Adding support for this in autocargo by copying these files into the output dir and making sure their option is specified to the thrift compiler
Reviewed By: ahornby
Differential Revision: D30789835
fbshipit-source-id: 325cb59fdf85324dccfff20a559802c11816769f
Summary:
Bump all the versions on crates.io to highest to make migration to github
versions in next diff work.
Reviewed By: dtolnay
Differential Revision: D30829928
fbshipit-source-id: 09567c26f275b3b1806bf8fd05417e91f04ba2ef
Summary:
Manual component version update
Bump Schedule: https://www.internalfb.com/intern/msdk/bump/?schedule_fbid=342556550408072
Package: https://www.internalfb.com/intern/msdk/package/181247287328949/
Oncall Team: rust_foundation
NOTE: This build is expected to expire at 2022/09/01 09:14AM PDT
---------
New project source changes since last bump based on D30663071 (08e362a355e0a64a503f5073f57f927394696b8c at 2021/08/31 03:47AM -05):
| 2021/08/31 04:41AM -05 | generatedunixname89002005294178 | D30665384 | [MSDK] Update autocargo component on FBS:master |
| 2021/08/31 07:14PM PDT | kavoor | D30681642 | [autocargo] Make cxx-build match version of cxx |
| 2021/09/01 04:05PM BST | krallin | D30698095 | autocargo: include generated comment in OSS manifests |
---------
build-break (bot commits are not reviewed by a human)
Reviewed By: farnz
Differential Revision: D30717040
fbshipit-source-id: 2c1d09f0d51b6ff2e2636496cf22bcf781f22889
Summary:
This config option was used to slowly roll out LFS for a repo.
However, it is no longer used and can therefore be removed.
Reviewed By: StanislavGlebik
Differential Revision: D30511880
fbshipit-source-id: 59fe5925cc203aa609488fdf8ea29e9ff65ee862
Summary:
The current Mononoke Blobstore Trace scuba table is used with idea of having a record per blobstore and operation. This diff adds logging to the new scuba table of the combined multiplexed operations' outcome, like time spent on the `put` including sync-queue and blobstore write or tracking record of the "some failed others none" cases in `get/is_present`.
This helps to see the real time spent on writes and reads and to assess the impact of changes coming in `get` and `is_present`.
Reviewed By: ahornby
Differential Revision: D30248284
fbshipit-source-id: f79050ced32ba77bd2e220e242407bcd711a9b6d
Summary:
The use of dyn traits of the Thrift-generated server traits was emitting future compatibility warnings with recent versions of rustc, due to a fixed soundness hole in the trait object system:
```
error: the trait `x_account_aggregator_if::server::XAccountAggregator` cannot be made into an object
|
= this was previously accepted by the compiler but is being phased out; it will become a hard error in a future release!
note: for a trait to be "object safe" it needs to allow building a vtable to allow the call to be resolvable dynamically; for more information visit <https://doc.rust-lang.org/reference/items/traits.html#object-safety>
```
This diff pulls in https://github.com/dtolnay/async-trait/releases/tag/0.1.51 which results in the Thrift-generated server traits no longer hitting the problematic pattern.
Reviewed By: zertosh
Differential Revision: D29979939
fbshipit-source-id: 3e6e976181bfcf35ed453ae681baeb76a634ddda
Summary:
This option would let us tell that a given bookmark (or bookmarks if they are
specified via a regex) is allowed to move only if it stays an ancestor of a
given bookmark.
Note - this is a sev followup, and we intend to use it for */stable bookmarks
(e.g. fbcode/stable, fbsource/stable etc). They are always intended to be an
ancestor of master
Reviewed By: krallin
Differential Revision: D29878144
fbshipit-source-id: a5ce08a09328e6a19af4d233c1a273a5e620b9ce
Summary: As discussed, extends Mononoke service to support commits w/o parents for the AI Infra usecase.
Reviewed By: markbt
Differential Revision: D29810303
fbshipit-source-id: f07fd7f1521ffe1cea85f1f54e71fe37fc39bb62
Summary:
Now that the new `rate_limiting` crate is being used by LFS server we
can remove the throttle limits code and config.
Differential Revision: D29396505
fbshipit-source-id: 19638bd93ad9dea2638e8501837c6c13e4dd48ff
Summary:
Update Mononoke server to use the new `rate_limiting` crate. This diff
also removes the old rate limiting library.
Differential Revision: D29396507
fbshipit-source-id: 05adb9322705b771a739c8bcaf2816c95218a42d
Summary:
Replace the LFS server's load shedding logic with that provided by the
`rate_limiting` crate.
Differential Revision: D29396503
fbshipit-source-id: a71812a55b9c9f111ee2861dc1b131ad20ca82d2
Summary:
Add a new rate limiting library that also supports load shedding when
an individual server is overloaded. This library provides a few benefits:
- The code can be shared between the LFS server and Mononoke server.
- The library supports more complex expressions of which clients to apply a
rate limit to (e.g. 10% of sandcastle and mactest machines).
- The rate limiting `Target` can be expanded in the future as the client
provides more information (e.g. client region).
- Mononoke server will be able to loadshed if an individual host is overloaded,
as we can currently do with the LFS server.
I've added this library as a separate crate rather than rewriting
`load_limiter` to make it easier to review. The next diff will make use of the
new library and remove the old one.
Reviewed By: StanislavGlebik
Differential Revision: D29396509
fbshipit-source-id: 2fbc04e266b18392062e6f952075efd5e24e89ba
Summary:
Adds class `ConfigeratorRedactedBlobs` that reads redaction data from configerator, and reloads it when necessary.
The class does this:
- Reads `RedactionSets` from configerator.
- For each key there, read `RedactionConfigBlobstore` looking for a `RedactionKeyList` with that key (these were populated by D29033598).
- From the keys listed, builds the map of redacted blobs, with the same format as before when it was fetched from XDB.
- Periodically checks if config changed. If so, reload the map of redactions. (should only happen when we land a new config change to redaction, which should be very very rare)
- We use ArcSwap to keep the config, as a good way to provide read-only access with eventual reloading.
Not implemented on this diff:
- Creation of `ConfigeratorRedactedBlobs`, or adding it to `RedactedBlobs` enum.
It's not used in this diff, will be used in the future, I split it mostly to make it easier to review.
Reviewed By: StanislavGlebik
Differential Revision: D29033595
fbshipit-source-id: 36603685433b6dd153f2c23123907f7311c20a32
Summary: Syncing configerator changes from D29360425, and fixing all tests. Not used yet.
Reviewed By: markbt
Differential Revision: D29363416
fbshipit-source-id: d2de13d32bcec2e7fbff20204be8d9a8d65c0efe
Summary:
This reads the config added on D29305462. It populates it into `CommonConfig` struct, and also adds it to `RepoFactory`, but doesn't yet use it anywhere, this will be done on the next diff.
There is a single behaviour change in this diff, which I believe should be harmless but is noted in the comments in case it isn't.
Reviewed By: markbt
Differential Revision: D29272581
fbshipit-source-id: 62cd7dc78478c1d8cb212eafdd789527ead50ef6
