Summary:
We recently had a sev S290626 where Isilon team rotated all S3 access keys for their clients, which broke our backup sync jobs.
Lets read the access keys from the source of thruth, which Isilon team may rotate themsleves.
And also lets read the keys periodically so we won't need to restart services if the rotation happens.
I also spotted a bug where instead of `region_name` the smc tier were passed, but since Isilon don't verify region it was working fine, anyway let's fix it too.
Reviewed By: HarveyHunt
Differential Revision: D39057471
fbshipit-source-id: 543a0a5be5903778899d31d04863226b831b5776
Summary:
* Use the logger framework to log data to scribe and scuba on bookmark operations (create, update, delete and pushrebase)
* To enable integration tests, also use a raw scribe call when scribe is setup to log to file
* To support building in oss mode, where logger is not available, fall back to raw scribe logging in that configuration
* When using logger, the scribe category is automatically propagated from `MononokeBookmark`
* When using raw scribe to log to file or to scribe in oss mode, the scribe category is propagated from the repo config
* Modify the `PushrebaseOutcome` struct to keep track of the `old_changeset`
The initial change (D38744849 (bc00e538b0)) was problematic as it broke backward compatibility over the thrift
boundary, so it was reverted in D38906045 (f089b4d53c).
There are three significant differences since the original change that was reverted:
* Don't deprecate `head` in `PushrebaseOutcome`: that name still makes sense in that context.
We could have kept both, and made sure to populate both `head` and `new_bookmark_value`.
However, the decision is to consistently keep using `head` in the context of `pushrebase` as
it makes sense in that context, and limit the `new_bookmark_value` naming to the new logging
around bookmarks as it makes sense over there.
* Don't fail if the `old_bookmark_value` on `PushrebaseOutcome` is not set
This allows compatibility between mononoke and SCS servers before and after the change
* Use the logger framework instead of raw scribe calls only, which allows to also populate scuba without extra logic
Reviewed By: mitrandir77
Differential Revision: D38910447
fbshipit-source-id: cc4820eee83352f7512ac2189018c9dd116790ce
Summary:
* The thrift struct: `RepoConfig` has a new field: `bookmark_scribe_category`
This file was automatically generated from configerator where that category is set to "mononoke_bookmark" in repo.cinc (see D38827214)
* Propagate this new field all the way to `bookmark_movements/src/update.rs`
* Modify the `PushrebaseOutcome` to keep track of the `old_changeset` and the `new_changeset`
* On bookmark operations (create, update, delete and pushrebase) log data to this new scribe category
Reviewed By: markbt
Differential Revision: D38744849
fbshipit-source-id: 3334e599a68d3e8bed90832f2c21e74a014cb20b
Summary:
The proposed sharding model is going to categorize all known repos into two categories: `shallow-sharded` and `deep-sharded` repos.
The repos which are `deep-sharded` will be sharded in the true sense, i.e. the requests for these repos will be sharded by the router and end up on specific servers that serve this repo. `shallow-sharded` repos on the other hand will have only their requests sharded but the repos themselves will be on every host. Sharding and routing the requests to specific hosts in this case will offer greater cache locality.
Another reason for having `shallow-sharded` repos is the possibility of `ShardManager` failure. In case `SM` goes down, all the `shallow-sharded` repos (which are all `hg` repos for now) will continue to be served without a problem.
Reviewed By: mitrandir77
Differential Revision: D38791867
fbshipit-source-id: 3d9a05ad35c44014db06ca0390c771ee19c4093a
Summary: Syncs the thrift changes and allows us to use string of vecs tunables.
Reviewed By: farnz
Differential Revision: D38040347
fbshipit-source-id: 3591ff1413059ae843bcab5f07d2e4ef769fc9f6
Summary:
Problem: Currently, when we log pushrebase metrics (for calculating saturation), we do so on all pushrebase operations, independent of in which bookmark they're pushrebasing into.
This is overcounting the saturation, as two pushrebase opeations on different bookmarks can happen in parallel.
This stack aims to fix that, by only logging for a specific bookmark (We could also log for all bookmarks, but the data for most bookmarks is not that useful, as usually only one of them happens a lot).
This diff only reads the field added to configerator and parses it to the correct struct, but it is still unused and will be used on the next diff.
Reviewed By: HarveyHunt
Differential Revision: D37787218
fbshipit-source-id: 0e4ab7b5f8828f94e4343446fa195d9e0b2cb37f
Summary:
Code in fbcode (such as `scmadmin`) would read from the hgserverconf
config stored in configerator to find out which DB shards the hg repos are
stored on. However, that's no longer needed.
Remove the synced thrift files from fbcode.
Reviewed By: mzr
Differential Revision: D37993014
fbshipit-source-id: 75aa5b189e877274d1877f14a89d30ab96a40bf2
Summary:
Now that these config options have been removed from prod they can be
removed from the metaconfig parser.
NOTE: This change is effectively a revert of D37787998 (f36dd36bdb)
Differential Revision: D37961020
fbshipit-source-id: 02a94a608fe8ca512114db37e51bd410254ea9a6
Summary: Calculation of the sparse profiles size is costly. Lets use DB to avoid recalculation. Later I'll add a commit tailer job which will calculate sizes for the public commits and populate db
Reviewed By: mzr
Differential Revision: D37750151
fbshipit-source-id: 2aa3eee6025c37e61eda4c95b05b0381fa4fe2ed
Summary: This adds parsing for the config added on D37683212
Reviewed By: mitrandir77
Differential Revision: D37684062
fbshipit-source-id: 9af7a354d351f889d31dac5c440999547d8ac4e6
Summary: D36129275 (2d0b0be078) and D37684480 (3283f09bf1) removed config options from the repo thrift file too early. Add them back again to fix the build.
Reviewed By: yancouto
Differential Revision: D37787998
fbshipit-source-id: ff6a20db6cebaa731fa63a45650b5c3a299b5f9b
Summary:
These config options are no longer used. Remove them from the source
code so that the configs can be cleaned up.
Reviewed By: mzr
Differential Revision: D37684480
fbshipit-source-id: 330436b2bd06e7dc1a9abe93a0a0693d9dae0d17
Summary:
Now that we have been double writing to `repo_lock` DBs for a while,
the old code can be removed.
This diff removes the code as well as the parsing for the
`write_lock_db_address` config option. I'll remove the config in a later diff.
As `RepoLock` is a facet, this diff also does some refactoring to make code a
bit clearer.
The diff also adds a new repo lock checker that can be used from within a
transaction. This is required as the pushrebase flow starts a transaction to
the metadata DB.
Reviewed By: markbt
Differential Revision: D36129275
fbshipit-source-id: dcfb9f06e76b21dbda74b5de4f9a359ebd32fff6
Summary: Syncing thrift changes from D37553148 and populating the new field in the config.
Reviewed By: markbt
Differential Revision: D37550694
fbshipit-source-id: d670300d66bdf1b7c81a709c47a57e3bea0246c6
Summary:
This diff includes changes to include the configerator per-repo hg-sync parameters in mononoke repo.config type
Configerator Diff: D37454155
Differential Revision: D37462675
fbshipit-source-id: 6d06b386b4a27536ab0c08169e6ad6badcb00d82
Summary:
This is a pretty in depth third party library being added. The main changes are...
**version bumping**:
dunce = "1.0.0" => "1.0.2"
once_cell = "1.8" => "1.12"
tracing = "0.1.32" => "0.1.35"
adding ethers-rs fork:
https://github.com/rlkelly/ethers-rs.git
this depends on forks of:
coins-bip32 = https://github.com/rlkelly/bitcoins-rs.git
coins-bip39 = https://github.com/rlkelly/bitcoins-rs.git
eth-keystore = https://github.com/rlkelly/eth-keystore-rs.git
this was necessary to remove the wasm target, which creates a cyclic dependency for "indexmap". This was ran into previously here:
https://fb.workplace.com/groups/rust.language/permalink/8603206789727860/
also, some fixups were created for a few packages to facilitate build. I'm not a fan of libraries creating a build step to generate documentation...
----
This still does not have the ethers-middleware crate or the ethers-solc crate because it has the external build dependency of solc, which would make this a bit more complex to add:
https://docs.soliditylang.org/en/latest/installing-solidity.html
adding the crates without solc makes the process of integrating this library much easier, but it would be beneficial in the future.
@public
update rust deps
Reviewed By: jsgf
Differential Revision: D37320246
fbshipit-source-id: 251bd1c3c0e51733347a534fa597dea33d1df74b
Summary:
In the next diffs I'm going to use that config for handling merges with squash commit. I needed to fix configs validators and some tests.
Configerator changes are in D37167271
Reviewed By: mitrandir77
Differential Revision: D37153447
fbshipit-source-id: 71762ec2a6e978aa7e7c509e5232a82570e86dcb
Summary:
Land after D36445153
Since D36444986 (c7c7392a0f) this is not used in fbcode anymore.
Reviewed By: farnz
Differential Revision: D36445954
fbshipit-source-id: 6d4981625c5104b42d99f1af10bcc8c8a6fc38c6
Summary: D36779955 adds config to let cross-repo commit validation skip changes to "boring" bookmarks. Add the code to make it work
Reviewed By: mitrandir77
Differential Revision: D36781750
fbshipit-source-id: 4f93bb856fd3e3d5375f1530fcdc98356b4ccd08
Summary:
This diff includes changes to include the configerator per-repo walker parameters in mononoke repo.config type.
Certain configuration parameters for `walker` jobs take different values based on the `repo`. Currently, the python logic in `walker.tw` decides which repo should get which value. However, with dynamic sharding, the repo to be executed by tasks will not be known before hand. The only available information would be `repo-name`. Moving repo-specific `walker` parameters to `repo-walker-config` allows these parameters to be fetched based just on `repo-name`.
Configerator diff: D36278073
Mononoke diff using these changes: D36377191
Reviewed By: yancouto
Differential Revision: D36315736
fbshipit-source-id: c0c9f78d52fd285b37d372e7190aa12e9890444b
Summary:
This imports the configerator changes from D35613093 and adds support on Mononoke to changing the deleted manifest version.
It changes a bit from the approach I was using previously, where we had `deleted_manifest` name for V1 and `deleted_manifest2` for V2. Now it uses the same approach blame used, which is having the same name for both, but always changing the underlying implementation based on the config.
Since there was not much usage of deleted manifests directly (only on 3 files), I went with checking the config and using static types on each of them, instead of adding a wrapper type around DMv1 or v2, as that would involve creating a lot of duplicated code.
Reviewed By: kris1319
Differential Revision: D35814371
fbshipit-source-id: 4edf87296e2ea9986f9af18cb2a7e6914ea5ffbe
Summary:
This diff adds a new way to specify the commits to be included in the
server-side semgented changelog. This is meant to replace the previous means of
configuration. Added features:
* it's now possible to tail multiple bookmarks
* it's now possible to have different set of commits tailed by the live
serving jobs vs offline tailer/seeder jobs.
Configerator diff: D35905607
Reviewed By: farnz
Differential Revision: D35905650
fbshipit-source-id: f0b7002d9c0f6bbc53dc776ed5564cde638be5ed
Summary:
- udpated thrift definition to support i64 ints;
- updated metaconfig loader type to parse and load the new 64bit properties;
Reviewed By: mzr
Differential Revision: D34962606
fbshipit-source-id: 95653139b2149bb9df2899b8b703fa58ee4473c3
Summary:
This diff does the following:
- Updates the tracing crates to the latest versions available. Changes include:
- Experimental support for [Valuable](https://tokio.rs/blog/2021-05-valuable), which makes recording values much more like how serde does. Valuable also makes it easier to write a tracing-slog crate due to some implementation details of slog where buffering of values is necessary.
- Combinators for combining filters in tracing-subscriber.
- A `MakeWriter` implementation for `RollingFileAppender` in `tracing_appender`, which allows the `RollingFileAppender` without a background thread to being spun up.
- Reduced the disabled span drop overhead by 50-70%—this might be most apparent in tests that don't set a subscriber.
- Move the tokio-console off git dependencies and to published versions on crates.io.
This diff also updates the following, non-exhaustive list of crates:
- parking_lot
- prost
- thread_local
- tonic
- hyper
- httparse
- h2
allow-large-files
Reviewed By: jsgf, fanzeyi
Differential Revision: D34801437
fbshipit-source-id: 2409de9194c8e00647ac5e4c514325fd2e23d247
Summary: Added a new configuration enum, `BubbleDeletionMode` that indicates if the bubble needs to undergo soft or hard deletion.
Reviewed By: farnz
Differential Revision: D34895758
fbshipit-source-id: ec096ff2e0d3580049e3ee24cad1f2f6f1e46d70
Summary: Adding a new field to the `RepoConfig`. Not used anywhere yet.
Reviewed By: markbt
Differential Revision: D34555579
fbshipit-source-id: cba8e3eeeb1ebbc2341f1bf55cc741bdfa7a1279
Summary:
Traffic replay is gone. Now we can delete code that dumped wireproto traffic.
The logging that's left could be still somewhat useful: https://fburl.com/scuba/mononoke_test_perf/uismnrv9
Reviewed By: HarveyHunt
Differential Revision: D33898167
fbshipit-source-id: f5f12f3626c578ef90db99a45e5749fe8a94049f
Summary:
The LFS server included separate fields in its config for object
popularity category and threshold. However, they have been replaced by a dedicated
`ObjectPopularity` struct that also includes a `window` field.
Remove the old configuration from the code, as it's unused and confusing.
Reviewed By: mzr
Differential Revision: D34214336
fbshipit-source-id: 7c3ecd5aa8d7e882783c8ff8630edf48999428e9
Summary: Run autocargo before doing any other changes, likely some or all of this will rebase away
Reviewed By: c-ryan747
Differential Revision: D33790131
fbshipit-source-id: fe6c761a09b300e482b5eb9c9c1465716d35dded
Summary:
Currently, blobstore get on multiplexed works like this:
1. If the blob is found on ANY blobstore, return it right away and don't care about others.
2. If the blob is not present on ALL blobstores, return it as not present.
3. If it is not present on some and the get errors on other blobstores, then the whole get errors.
This makes get requests much slower when the blob is not present (since they ALWAYS need to wait for ALL blobstores), while put requests do fine as we only need to guarantee it is written to SOME blobstores.
This can cause problems even when a single blobstore is borked, which is not ideal (see [1] for a real life case).
This diff keeps the behaviour of 1, but changes the behaviour of 2 and 3, so that mononoke only needs to check SOME blobstores to determine the blob doesn't exist.
If there are N blobstores, and put quorum is P (that is, we say a write is successful if P puts are successful), then if N - P + 1 blobstores do not contain a blob, then there's no way a successful put happened for that blob (by pigeonhole principle). D32700029 uses that value for quorums, though it can be tweaked if desired.
Reviewed By: StanislavGlebik
Differential Revision: D32699218
fbshipit-source-id: 0e1d1f9640acb3770d8830b5868c57e1748e3c63
Summary:
For invocations of `ensure!` where the argument is one of the comparison operators `==` `<=` `<` `!=` `>=` `>` and the lhs and rhs of the operator both have a `Debug` impl, we now render both sides as part of the error message.
### Example:
```
ensure!(flags.len() <= 40);
```
```
ensure!(kind == Kind::File);
```
Before:
```
Condition failed: `flags.len() <= 40`
Condition failed: `kind == Kind::File`
```
After:
```
Condition failed: `flags.len() <= 40` (99 vs 40)
Condition failed: `kind == Kind::File` (Symlink vs File)
```
Reviewed By: zertosh
Differential Revision: D32598267
fbshipit-source-id: 70fdab6d8f0e3ceceb98e651eab0a73795cadaeb
Summary:
This release helps catch places that the `anyhow!` or `bail!` macros are invoked with a missing fmt argument, such as:
```
- anyhow!("unsupported redirect operation {:?}")
+ anyhow!("unsupported redirect operation {:?}", op)
```
All instances of this in fbcode were fixed in D32266567 (2f440eac7e).
Reviewed By: wqfish
Differential Revision: D32564287
fbshipit-source-id: e9cf5a1376e6c83343de3f3c6ba2925d88334a30
Summary: Bring the config in line with the oss branch names
Reviewed By: mzr
Differential Revision: D32109171
fbshipit-source-id: f87a0070b0ef06a49c86cacdfdc7f5c8b02f3fbf
