Summary:
Now that `deep_sharding` is expressed via `ShardingModeConfig`, the repo-level `deep_sharded` boolean flag can be removed.
Corresponding configerator diff: D42676137
Reviewed By: markbt
Differential Revision: D42676143
fbshipit-source-id: 09b87aab1718f828c3015930645ba58ca58bb1fd
Summary:
In D38786500, I introduced `deep-sharded` flag that controls whether a repo needs to execute in deep-sharded mode or shallow-sharded mode. However, the flag was universal (i.e. it was applicable to all services). Since we currently have a situation where certain services need to execute repos in deep sharded (e.g. `SCS`) while others need to execute in shallow-sharded mode (e.g. `Eden API`). This diff introduces `ShardingModeConfig` which determines if a repo is deep sharded for a particular service.
For now, the `deep_sharded` flag is kept as is. Once the config + code changes to use this new property are landed, I will delete the old `deep_sharded` flag.
Corresponding configerator diff: D42501648
Differential Revision: D42530372
fbshipit-source-id: 1bd46533ad18939da4465239dab1340a0e8219d2
Summary:
**Context**: The write-mostly name is confusing and out of date. We never read from write-mostly blobstores (only on scrub). Let's rename it to write only to make it less confusing.
See D41520772 for migration plan.
This diff is the step (3). It will only be landed after previous diff rolls out.
Reviewed By: mitrandir77
Differential Revision: D41520763
fbshipit-source-id: f3775ad13439a9326b7fdc4cba5b691d59af4770
Summary: This makes WAL support unsharded DB, which makes MySQL integration tests work again.
Reviewed By: mzr
Differential Revision: D42430809
fbshipit-source-id: b6e2dce380a5c91700dc920fa0cdc993e00cb3e6
Summary: This syncs changes from D42145360 and deletes the category fields from thrift.
Reviewed By: mitrandir77
Differential Revision: D42145669
fbshipit-source-id: 1268308be11af86180709c2b46a407c41c731321
Summary:
**Context**: The write-mostly name is confusing and out of date. We never read from write-mostly blobstores (only on scrub). Let's rename it to write only to make it less confusing.
See D41520772 for migration plan.
This diff is the fbsource part of step (1): **Makes our code accept both configs as meaning write only** and does all the renames.
The most important change is file `eden/mononoke/metaconfig/parser/src/convert/storage.rs`, which starts accepting `write-only` config as well as `write-mostly`.
Reviewed By: mitrandir77
Differential Revision: D41520648
fbshipit-source-id: b0274c937ca7db10debb8c952a9a1867c0b431c1
Summary: This syncs the config change from D41469348, and fully removes logic to do pushrebases remotely through SCS. Now everything should go through land service (and already does so in production).
Reviewed By: RajivTS
Differential Revision: D41469510
fbshipit-source-id: 0e41624a84b8cbefaf3793a5f98533f9abd2257a
Summary:
**Context**: We want to roll out the new WAL multiplexed blobstore and need final tweaks before doing that. See https://fb.quip.com/vyTWAgGZSA2Z
This sync configerator changes from D40754273, and updates our internal configs to load two types of scuba tables. But it continues doing the same as before, only logging to scuba on internal blobstore queries.
I also renamed `scuba` to `inner_blobstores_scuba` to make it clear, because that's confusing on the current multiplexed blobstore.
Note: the thrift compat error is fine to bypass because we're not using this thrift schema anywhere yet.
Differential Revision: D40720597
fbshipit-source-id: fceb9d72b20e298d52744dfd79798c36f34b07da
Summary:
**Context**: We want to roll out the new WAL multiplexed blobstore and need final tweaks before doing that. See https://fb.quip.com/vyTWAgGZSA2Z
This syncs configerator changes from D40677044 and updates our code so that WAL config now takes a sharded DB instead of unsharded. The code for dealing with sharded connections was already added on D40673033, this is just the config to enable it.
For now this is not enabled anywhere. The plan is to:
- Test it locally with locally built binaries
- Move a small-ish repo
- Move everything
This is once we have full feature parity, which is still not done (things like Scuba logging and scrub blobstores are missing)
Differential Revision: D40675512
fbshipit-source-id: 70f74a53dbf10eadc73ce9e210ec7719f866e30a
Summary:
This diff declares a new config for the WAL multiplexed blobstore. It is different from the current multiplex in the way it defines write/read quorums.
The main reason for a separate config is ability to iteratively switch repos and different parts of the Mononoke tools from the old multiplex to the new one.
Reviewed By: yancouto
Differential Revision: D40108825
fbshipit-source-id: 9dae447557702474b6427ad247164a879da90861
Summary: This release makes async trait methods work better with rust-analyzer's "go to definition" and other IDE functionality.
Reviewed By: zertosh
Differential Revision: D40532047
fbshipit-source-id: 1871d1819ebe1aab0846b0135f051c45e6e8821b
Summary:
It (1) adds the repo in the Land Service, (2) includes Remote Land Service and remote land service with local fallback, (2) get the address from tier or host and calls the client in the unbundle processing
The Static analysis failed in the FBTHRIFTCOMPAT1 because i added a new field in the land_changesets method in the impl LandService (land_service_impl.rs). It does not break anything since we do not have the server in production. I mean, changing the id does not influence on previous versions of the server since there is no land service server in production.
Reviewed By: yancouto
Differential Revision: D39553570
fbshipit-source-id: bb1f09c2ab2c111734b9026109fc39e14c514516
Summary: I will use this to correctly infer scm_type in Hack's scmquery-proxy client library.
Differential Revision: D39492898
fbshipit-source-id: 4d9aafcb88f75b20260d4212bae70495efca22d5
Summary: Most of fast replay was removed in D34073952 (a9eaa380fe) but I found some more code, so delete that
Reviewed By: HarveyHunt
Differential Revision: D39547680
fbshipit-source-id: b35850621b3d37b939a1a44db7926f52d583326a
Summary: D39418421 (a1593af094) prematurely upgraded `thiserror`. Various projects in fbcode use nightly features including std::backtrace but the current fbcode Rust toolchain is 1.63.0, in which the backtrace API is different from the API of the latest nightlies.
Reviewed By: quark-zju
Differential Revision: D39562441
fbshipit-source-id: d1f8a993ec776cf82b1736b242bd4466def21230
Summary: This adds a wrapper that knows how to wait for replication of xdb blobstore and sync queue, and for that uses the configerator config added on D39387517, so that this can be hot-reloaded.
Reviewed By: RajivTS
Differential Revision: D39387650
fbshipit-source-id: 548f99bc1b0b3634c8b8a6fb9ea7815edb41d3e4
Summary:
We recently had a sev S290626 where Isilon team rotated all S3 access keys for their clients, which broke our backup sync jobs.
Lets read the access keys from the source of thruth, which Isilon team may rotate themsleves.
And also lets read the keys periodically so we won't need to restart services if the rotation happens.
I also spotted a bug where instead of `region_name` the smc tier were passed, but since Isilon don't verify region it was working fine, anyway let's fix it too.
Reviewed By: HarveyHunt
Differential Revision: D39057471
fbshipit-source-id: 543a0a5be5903778899d31d04863226b831b5776
Summary:
* Use the logger framework to log data to scribe and scuba on bookmark operations (create, update, delete and pushrebase)
* To enable integration tests, also use a raw scribe call when scribe is setup to log to file
* To support building in oss mode, where logger is not available, fall back to raw scribe logging in that configuration
* When using logger, the scribe category is automatically propagated from `MononokeBookmark`
* When using raw scribe to log to file or to scribe in oss mode, the scribe category is propagated from the repo config
* Modify the `PushrebaseOutcome` struct to keep track of the `old_changeset`
The initial change (D38744849 (bc00e538b0)) was problematic as it broke backward compatibility over the thrift
boundary, so it was reverted in D38906045 (f089b4d53c).
There are three significant differences since the original change that was reverted:
* Don't deprecate `head` in `PushrebaseOutcome`: that name still makes sense in that context.
We could have kept both, and made sure to populate both `head` and `new_bookmark_value`.
However, the decision is to consistently keep using `head` in the context of `pushrebase` as
it makes sense in that context, and limit the `new_bookmark_value` naming to the new logging
around bookmarks as it makes sense over there.
* Don't fail if the `old_bookmark_value` on `PushrebaseOutcome` is not set
This allows compatibility between mononoke and SCS servers before and after the change
* Use the logger framework instead of raw scribe calls only, which allows to also populate scuba without extra logic
Reviewed By: mitrandir77
Differential Revision: D38910447
fbshipit-source-id: cc4820eee83352f7512ac2189018c9dd116790ce
Summary:
* The thrift struct: `RepoConfig` has a new field: `bookmark_scribe_category`
This file was automatically generated from configerator where that category is set to "mononoke_bookmark" in repo.cinc (see D38827214)
* Propagate this new field all the way to `bookmark_movements/src/update.rs`
* Modify the `PushrebaseOutcome` to keep track of the `old_changeset` and the `new_changeset`
* On bookmark operations (create, update, delete and pushrebase) log data to this new scribe category
Reviewed By: markbt
Differential Revision: D38744849
fbshipit-source-id: 3334e599a68d3e8bed90832f2c21e74a014cb20b
Summary:
The proposed sharding model is going to categorize all known repos into two categories: `shallow-sharded` and `deep-sharded` repos.
The repos which are `deep-sharded` will be sharded in the true sense, i.e. the requests for these repos will be sharded by the router and end up on specific servers that serve this repo. `shallow-sharded` repos on the other hand will have only their requests sharded but the repos themselves will be on every host. Sharding and routing the requests to specific hosts in this case will offer greater cache locality.
Another reason for having `shallow-sharded` repos is the possibility of `ShardManager` failure. In case `SM` goes down, all the `shallow-sharded` repos (which are all `hg` repos for now) will continue to be served without a problem.
Reviewed By: mitrandir77
Differential Revision: D38791867
fbshipit-source-id: 3d9a05ad35c44014db06ca0390c771ee19c4093a
Summary: Syncs the thrift changes and allows us to use string of vecs tunables.
Reviewed By: farnz
Differential Revision: D38040347
fbshipit-source-id: 3591ff1413059ae843bcab5f07d2e4ef769fc9f6
Summary:
Problem: Currently, when we log pushrebase metrics (for calculating saturation), we do so on all pushrebase operations, independent of in which bookmark they're pushrebasing into.
This is overcounting the saturation, as two pushrebase opeations on different bookmarks can happen in parallel.
This stack aims to fix that, by only logging for a specific bookmark (We could also log for all bookmarks, but the data for most bookmarks is not that useful, as usually only one of them happens a lot).
This diff only reads the field added to configerator and parses it to the correct struct, but it is still unused and will be used on the next diff.
Reviewed By: HarveyHunt
Differential Revision: D37787218
fbshipit-source-id: 0e4ab7b5f8828f94e4343446fa195d9e0b2cb37f
Summary:
Code in fbcode (such as `scmadmin`) would read from the hgserverconf
config stored in configerator to find out which DB shards the hg repos are
stored on. However, that's no longer needed.
Remove the synced thrift files from fbcode.
Reviewed By: mzr
Differential Revision: D37993014
fbshipit-source-id: 75aa5b189e877274d1877f14a89d30ab96a40bf2
Summary:
Now that these config options have been removed from prod they can be
removed from the metaconfig parser.
NOTE: This change is effectively a revert of D37787998 (f36dd36bdb)
Differential Revision: D37961020
fbshipit-source-id: 02a94a608fe8ca512114db37e51bd410254ea9a6
Summary: Calculation of the sparse profiles size is costly. Lets use DB to avoid recalculation. Later I'll add a commit tailer job which will calculate sizes for the public commits and populate db
Reviewed By: mzr
Differential Revision: D37750151
fbshipit-source-id: 2aa3eee6025c37e61eda4c95b05b0381fa4fe2ed
Summary: This adds parsing for the config added on D37683212
Reviewed By: mitrandir77
Differential Revision: D37684062
fbshipit-source-id: 9af7a354d351f889d31dac5c440999547d8ac4e6
Summary: D36129275 (2d0b0be078) and D37684480 (3283f09bf1) removed config options from the repo thrift file too early. Add them back again to fix the build.
Reviewed By: yancouto
Differential Revision: D37787998
fbshipit-source-id: ff6a20db6cebaa731fa63a45650b5c3a299b5f9b
Summary:
These config options are no longer used. Remove them from the source
code so that the configs can be cleaned up.
Reviewed By: mzr
Differential Revision: D37684480
fbshipit-source-id: 330436b2bd06e7dc1a9abe93a0a0693d9dae0d17
Summary:
Now that we have been double writing to `repo_lock` DBs for a while,
the old code can be removed.
This diff removes the code as well as the parsing for the
`write_lock_db_address` config option. I'll remove the config in a later diff.
As `RepoLock` is a facet, this diff also does some refactoring to make code a
bit clearer.
The diff also adds a new repo lock checker that can be used from within a
transaction. This is required as the pushrebase flow starts a transaction to
the metadata DB.
Reviewed By: markbt
Differential Revision: D36129275
fbshipit-source-id: dcfb9f06e76b21dbda74b5de4f9a359ebd32fff6
Summary: Syncing thrift changes from D37553148 and populating the new field in the config.
Reviewed By: markbt
Differential Revision: D37550694
fbshipit-source-id: d670300d66bdf1b7c81a709c47a57e3bea0246c6
Summary:
This diff includes changes to include the configerator per-repo hg-sync parameters in mononoke repo.config type
Configerator Diff: D37454155
Differential Revision: D37462675
fbshipit-source-id: 6d06b386b4a27536ab0c08169e6ad6badcb00d82
Summary:
This is a pretty in depth third party library being added. The main changes are...
**version bumping**:
dunce = "1.0.0" => "1.0.2"
once_cell = "1.8" => "1.12"
tracing = "0.1.32" => "0.1.35"
adding ethers-rs fork:
https://github.com/rlkelly/ethers-rs.git
this depends on forks of:
coins-bip32 = https://github.com/rlkelly/bitcoins-rs.git
coins-bip39 = https://github.com/rlkelly/bitcoins-rs.git
eth-keystore = https://github.com/rlkelly/eth-keystore-rs.git
this was necessary to remove the wasm target, which creates a cyclic dependency for "indexmap". This was ran into previously here:
https://fb.workplace.com/groups/rust.language/permalink/8603206789727860/
also, some fixups were created for a few packages to facilitate build. I'm not a fan of libraries creating a build step to generate documentation...
----
This still does not have the ethers-middleware crate or the ethers-solc crate because it has the external build dependency of solc, which would make this a bit more complex to add:
https://docs.soliditylang.org/en/latest/installing-solidity.html
adding the crates without solc makes the process of integrating this library much easier, but it would be beneficial in the future.
@public
update rust deps
Reviewed By: jsgf
Differential Revision: D37320246
fbshipit-source-id: 251bd1c3c0e51733347a534fa597dea33d1df74b
Summary:
In the next diffs I'm going to use that config for handling merges with squash commit. I needed to fix configs validators and some tests.
Configerator changes are in D37167271
Reviewed By: mitrandir77
Differential Revision: D37153447
fbshipit-source-id: 71762ec2a6e978aa7e7c509e5232a82570e86dcb
