Commit Graph

57 Commits

Author SHA1 Message Date
Egor Tkachenko
2494f1db7e Fix cache pool sizes
Summary: My previous diff shrink the size of the cache so the pools can't fit there anymore. Adjust them as well. Plus made cache argument pareser understand float values for `cache-size-gb`

Reviewed By: johansglock

Differential Revision: D24448419

fbshipit-source-id: 2b73f789df10c5df7685ba96b7f19b8c9d04cc71
2020-10-21 10:16:45 -07:00
Alex Hornby
7a65b60841 mononoke: add --blobstore-put-behaviour argument
Summary: Add --blobstore-put-behaviour argument so we can try running some workloads with modified behaviour, firstly with logging on overwrites to gather metrics,  and then not able to overwrite at all.

Reviewed By: StanislavGlebik

Differential Revision: D24109292

fbshipit-source-id: bbea31eb40604fdedc3a0db7a84e99b5b1fe7a23
2020-10-13 03:09:38 -07:00
Alex Hornby
2abe862535 mononoke: add put behaviour to BlobstoreOptions
Summary:
Add put behaviour to BlobstoreOptions in preparation for passing in the put behaviour through blobstore_factory.

Later in the stack a command line option is added to set this non-None so that we can turn on overwrite logging for particular jobs.

Reviewed By: StanislavGlebik

Differential Revision: D24021169

fbshipit-source-id: 5692e2d3912ebde07b0d7bcce54b79df188a9f16
2020-10-07 12:11:10 -07:00
Kostia Balytskyi
2ea25308ab commit_rewriting: use is_empty() where possible
Summary: `clippy` often complains about the use of `.len() != 0`, `.len() > 0` or `.len() == 0`and proposes to use `.is_empty()` instead. This diff does that across Mononoke.

Reviewed By: aslpavel

Differential Revision: D24099427

fbshipit-source-id: 1bba2f958485b7efb3f41bf3eae820879c92b0e5
2020-10-04 10:03:42 -07:00
Aida Getoeva
40b8353d21 mononoke: integrate mysql client
Summary:
This diff introduces Mysql client for Rust to Mononoke as a one more backend in the same row with raw xdb connections and myrouter. So now Mononoke can use new Mysql client connections instead of Myrouter.

To run Mononoke with the new backend, pass `--use-mysql-client` options (conflicts with `--myrouter-port`).

I also added a new target for integration tests, which runs mysql tests using mysql client.
Now to run mysql tests using raw xdb connections, you can use `mononoke/tests/integration:integration-mysql-raw-xdb` and using mysql client `mononoke/tests/integration:integration-mysql`

Reviewed By: ahornby

Differential Revision: D23213228

fbshipit-source-id: c124ccb15747edb17ed94cdad2c6f7703d3bf1a2
2020-09-29 03:09:05 -07:00
Stanislau Hlebik
a47464f60d mononoke: remove create_commit_syncer_from_matches
Summary:
At the moment CommitSyncConfig can be set in two ways:
1) Set in the normal mononoke config. That means that config can be updated
only after a service is restarted. This is an outdated way, and won't be used
in prod.
2) Set in separate config which can be updated on demand. This is what we are
planning to use in production.

create_commit_syncer_from_matches was used to build a CommitSyncer object via
normal mononoke config (i.e. outdated option #1). According to the comments it
was done so that we can read configs from the disk instead of configerator, but
this doesn't make sense because you already can read configerator configs
from disk. So create_commit_syncer_from_matches doesn't look particularly
useful, and besides it also make further refactorings harder. Let's remove it.

Reviewed By: ikostia

Differential Revision: D23811090

fbshipit-source-id: 114d88d9d9207c831d98dfa1cbb9e8ede5adeb1d
2020-09-23 04:30:41 -07:00
Stanislau Hlebik
b5f1e53cd6 mononoke: use logical number of cpus in our runtime
Summary:
We are using older version of tokio which spawns as many threads as we have
physical cores instead of the number of logical cores. It was fixed in
https://github.com/tokio-rs/tokio/issues/2269 but we can't use it yet because
we are waiting for another fix to be released -
https://github.com/rust-lang/futures-rs/pull/2154.

For now let's hardcode it in mononoke

Reviewed By: krallin

Differential Revision: D23599140

fbshipit-source-id: 80685651a7a29ba8938d9aa59770f191f7c42b8b
2020-09-09 09:25:40 -07:00
Stanislau Hlebik
0740f99f13 mononoke: allow logging censored scuba accesses to file
Summary:
In the next diff I'm going to add log-only mode to redaction, and it would be
good to have a way of testing it (i.e. testing that it actually logs accesses
to bad keys).

In this diff let's use a config option that allows logging censored scuba
accesses to file, and let's update redaction integration test to use it

Reviewed By: ikostia

Differential Revision: D23537797

fbshipit-source-id: 69af2f05b86bdc0ff6145979f211ddd4f43142d2
2020-09-04 07:37:14 -07:00
Mark Thomas
8790a793b9 mononoke_api: add HookManager
Summary: We will shortly need a `HookManager` in the write methods of the source control service.  Add one to `mononoke_api::Repo`

Reviewed By: StanislavGlebik

Differential Revision: D23077552

fbshipit-source-id: e1eed3661fe26a839e50ac4d884f4fadf793dbbb
2020-08-25 09:14:07 -07:00
Arun Kulshreshtha
523013c808 cmdlib: remove extra comment slashes
Reviewed By: quark-zju

Differential Revision: D23111025

fbshipit-source-id: b8606c322439c41097d739df59b551a8432e7fe4
2020-08-14 18:04:26 -07:00
Alex Hornby
ecb58ff8d7 mononoke: add cmdlib argument to control cachelib zstd compression
Summary:
Add a cmdlib argument to control cachelib zstd compression. The default behaviour is unchanged, in that the CachelibBlobstore will attempted compression when putting to the cache if the object is larger than the cachelib max size.

To make the cache behaviour more testable, this change also adds an option to do an eager put to cache without the spawn. The default remains to do a lazy fire and forget put into the cache with tokio::spawn.

The motivation for the change is that when running the walker the compression putting to cachelib can dominate CPU usage for part of the walk, so it's best to turn it off and let those items be uncached as the walker is unlikely to visit them again (it only revisits items that were not fully derived).

Reviewed By: StanislavGlebik

Differential Revision: D22797872

fbshipit-source-id: d05f63811e78597bf3874d7fd0e139b9268cf35d
2020-07-31 01:12:02 -07:00
Mark Thomas
89bc34035b scuba: add log sequence numbers
Summary:
Add log sequence numbers to the scuba sample builder.  This provides an ordering
over the logs made by an individual instance of Mononoke, allowing them to be
sorted.

Reviewed By: krallin

Differential Revision: D22728880

fbshipit-source-id: 854bde51c7bfc469677ad08bb738e5097cb05ad5
2020-07-27 07:10:07 -07:00
Arun Kulshreshtha
5f0181f48c Regenerate all Cargo.tomls after upgrade to futures 0.3.5
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.

Reviewed By: dtolnay

Differential Revision: D22403809

fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
2020-07-06 20:49:43 -07:00
Thomas Orozco
be1bac6c06 mononoke/virtually_sharded_blobstore: expose this in cmdlib
Summary:
Eventually, I plan to make this the default, but for now I'd like to make it
something we can choose to turn on or off as a cmd argument (so we can start
with the experimental tier and Fastreplay).

Note that this mixes volatile vs. non-volatile pools when accessing the pools
for cacheblob. In practice, those pools are actually volatile, it's just that
things don't break if you access them as non-volatile.

Reviewed By: farnz

Differential Revision: D22356537

fbshipit-source-id: 53071b6b21ca5727d422e10f685061c709114ae7
2020-07-03 05:53:11 -07:00
Stanislau Hlebik
2a54f281f2 mononoke: add an option to perform a stack move in megarepotool
Summary:
We've recently added an option to perform a stack move in megarepolib. A "stack
move" it's a stack of commits that move a files according to a mover. Now let's
expose it in the megarepotool

Reviewed By: ikostia

Differential Revision: D22312486

fbshipit-source-id: 878d4b2575ed2930bbbf0b9b35e51bb41393e622
2020-07-03 00:18:41 -07:00
Stanislau Hlebik
8a137ae922 mononoke: add Scribe
Summary:
At the moment we can't test logging to scribe easily - we don't have a way to
mock it. Scribe are supposed to help with that.

They will let us to configure all scribe logs to go to a directory on a
filesystem similar to the way we configure scuba. The Scribe itself will
be stored in CoreContext

Reviewed By: farnz

Differential Revision: D22237730

fbshipit-source-id: 144340bcfb1babc3577026191428df48e30a0bb6
2020-06-29 12:15:22 -07:00
Harvey Hunt
026710c2cd mononoke: Remove --config_path from server arguments
Summary:
D21642461 (46d2b44c0e) converted Mononoke server to use the
`--mononoke-config-path` common argument style to select a config path.

Now that this change has been running for a while, remove the extra logic in
the server that allowed it to accept both the deprecated `--config_path / -P`
and the new arg.

Reviewed By: ikostia

Differential Revision: D22257386

fbshipit-source-id: 7da4ed4e0039d3659f8872693fa4940c58bae844
2020-06-29 07:28:36 -07:00
Kostia Balytskyi
9a00efc973 cmdlib: expose test-instance and local-configerator-path
Summary:
Before this diff only the main Mononoke server binary was able to use fs-based
`ConfigStore`, which is pretty useful in integration tests.

Reviewed By: farnz

Differential Revision: D22256618

fbshipit-source-id: 493a064a279250d01469c9ff7f747585581caf51
2020-06-26 06:51:07 -07:00
Simon Farnsworth
454de31134 Switch Loadable and Storable interfaces to new-style futures
Summary:
Eventually, we want everything to be `async`/`await`; as a stepping stone in that direction, switch some of the blobstore interfaces to new-style `BoxFuture` with a `'static` lifetime.

This does not enable any fixes at this point, but does mean that `.compat()` moves to the places that need old-style futures instead of new. It also means that the work needed to make the transition fully complete is changed from a full conversion to new futures, to simply changing the lifetimes involved and fixing the resulting compile failures.

Reviewed By: krallin

Differential Revision: D22164315

fbshipit-source-id: dc655c36db4711d84d42d1e81b76e5dddd16f59d
2020-06-25 08:45:37 -07:00
Kostia Balytskyi
016b101be9 xrepo: add CommitSyncerArgs
Summary:
Similarly to how we have `PushRedirectorArgs`, we need `CommitSyncerArgs`: a struct, which a long-living process can own and periodically create a real `CommitSyncer` out of it, by consuming freshly reloaded `CommitSyncConfig`.

It is a little unfortunate that I am introducing yet another struct to `commit_rewriting/cross_repo_sync`, as it's already pretty confusing with `CommitSyncer` and `CommitSyncRepos`, but hopefully `CommitSyncerArgs`'s purpose is simple enough that it can be inferred from the name. Note that this struct does have a few convenience methods, as we need to access things like `target_repo` and various repo ids before we even create a real `CommitSyncer`. This makes it's purpose a little less singular, but still fine IMO.

Reviewed By: StanislavGlebik

Differential Revision: D22197123

fbshipit-source-id: e2d993e186075e33acec00200d2aab10fb893ffd
2020-06-25 03:28:08 -07:00
Pavel Aslanov
a1f5e45a5a BlobRepoHg extension trait.
Summary: This diff introduces `BlobRepoHg` extension trait for `BlobRepo` object. Which contains mercurial specific methods that were previously part of `BlobRepo`. This diff also stars moving some of the methods from BlobRepo to BlobRepoHg.

Reviewed By: ikostia

Differential Revision: D21659867

fbshipit-source-id: 1af992915a776f6f6e49b03e4156151741b2fca2
2020-06-22 07:29:19 -07:00
Jeremy Fitzhardinge
35b292ce9d eden: manual dependency fixes
Summary:
Tooling can't handle named_deps yet, but it can warn about them

P133451794

Reviewed By: StanislavGlebik

Differential Revision: D22083499

fbshipit-source-id: 46de533c19b13b2469e912165c1577ddb63d15cd
2020-06-17 17:55:04 -07:00
Jeremy Fitzhardinge
1b4edb5567 eden: remove unused Rust dependencies
Summary:
Remove unused dependencies for Rust targets.

This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.

Manual edits (named_deps) and misc output in P133451794

Reviewed By: dtolnay

Differential Revision: D22083498

fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
2020-06-17 17:55:03 -07:00
Alex Hornby
9c53e07e46 mononoke: add optional compress to packblob put
Summary:
Add optional compress on put controlled by a command line option.

Other than costing some CPU time, this may be a good option when populating repos from existing uncompressed stores to new stores.

Reviewed By: farnz

Differential Revision: D22037756

fbshipit-source-id: e75190ddf9cfd4ed3ea9a18a0ec6d9342a90707b
2020-06-17 02:35:04 -07:00
Jeremy Fitzhardinge
c530d32056 rust: clean up some warnings
Summary: Prep for 1.44 but also general cleanups.

Reviewed By: dtolnay

Differential Revision: D22024428

fbshipit-source-id: 8e1d39a1e78289129b38554674d3dbf80681f4c3
2020-06-15 16:50:40 -07:00
Aida Getoeva
81e7b76475 mononoke/apiserver: remove content sha1 cache
Summary: It was used by APIServer only, now when it's decomissioned we can delete this.

Reviewed By: HarveyHunt

Differential Revision: D21954670

fbshipit-source-id: a16bbbbb9dfe17eda5ff1ee6267251708ee4d562
2020-06-10 19:29:48 -07:00
David Tolnay
cf412e0d6b rustfmt: Use use_try_shorthand default
Summary:
I observed that for whatever reason our setting of `use_try_shorthand = true` in rustfmt.toml was causing entire functions to not get processed by rustfmt. Even files that contain neither `try` nor `?`. Remove it and reformat fbsource.

Documentation of that config:

- https://github.com/rust-lang/rustfmt/blob/master/Configurations.md#use_try_shorthand

We don't particularly care about the value anymore because nobody writes `r#try!(...)` in 2018 edition code.

Minimized:

```
fn f() {
    g(
        )
        // ...
        .h
}
```

This function gets formatted only if use_try_shorthand is not set.

The bug is fixed in the rustfmt 2.0 release candidate.

Reviewed By: jsgf

Differential Revision: D21878162

fbshipit-source-id: b028673c0eb703984d24bf0d2983453fc2a8c212
2020-06-10 19:29:15 -07:00
Mark Thomas
6ad7e5c96e metaconfig/parser: use free functions for loading configuration
Summary:
Replace the use of `RepoConfigs::read*` associated functions with free
functions.  These didn't really need to be associated functions (and in the
case of the common and storage configs, really didn't belong there either).

Reviewed By: krallin

Differential Revision: D21837270

fbshipit-source-id: 2dc73a880ed66e11ea484b88b749582ebdf8a73f
2020-06-02 09:30:03 -07:00
Harvey Hunt
46d2b44c0e mononoke: Use MononokeApp for mononoke server
Summary:
Mononoke server is the only binary using plain `clap::App`, whilst all
other services use `MononokeApp`. The `MononokeApp` variant configures some
default arguments and provides consistent argument naming between binaries.

Update the server to use `MononokeApp` and add some logic to allow passing
either `--config_path` or `--mononoke-config-path` until we remove all uses of
`--config-path`.

Reviewed By: StanislavGlebik

Differential Revision: D21642461

fbshipit-source-id: f42e0b6625f3979ced0920db269bdb4528f99e49
2020-05-20 05:47:17 -07:00
Stefan Filip
60966c93e7 autocargo: regenerate
Summary: maintenance

Reviewed By: StanislavGlebik

Differential Revision: D21640322

fbshipit-source-id: d0b2ce604735c05d540d06835c8e4c8a940fbf5c
2020-05-19 16:08:40 -07:00
Igor Katson
246e3e142b Remove .compat() from edenapi_server/main.rs
Summary: This removes .compat() from edenapi_server/main.rs. The actual removal probably could be done with less code, but in addition to removing compat(), I made most of the blocking code async.

Reviewed By: kulshrax, farnz

Differential Revision: D21426641

fbshipit-source-id: 1b3de4dc0b24d06faeb73de2e8658f0629d9491d
2020-05-11 09:12:07 -07:00
Stanislau Hlebik
c71469dc1c mononoke: add tunable for warm bookmark config
Summary: Let's make it tunable, more context in D21300885

Reviewed By: ikostia, krallin

Differential Revision: D21347636

fbshipit-source-id: 4bc91effdd212a3f5c4a0c3d4952f52a1cf417d7
2020-05-01 04:04:51 -07:00
Harvey Hunt
3cd49f9d3c mononoke: Add tunables - a simple form of config hot reloading
Summary:
Currently, Mononoke's configs are loaded at startup and only refreshed
during restart. There are some exceptions to this, including throttling limits.
Other Mononoke services (such as the LFS server) have their own implementations
of hot reloadable configs, however there isn't a universally agreed upon method.

Static configs makes it hard to roll out features gradually and safely. If a
bad config option is enabled, it can't be rectified until the entire tier is
restarted. However, Mononoke's code is structured with static configs in mind
and doesn't support hot reloading. Changing this would require a lot of work
(imagine trying to swap out blobstore configs during run time) and wouldn't
necessarily provide much benefit.

Instead, add a subset of hot reloadable configs called tunables. Tunables are
accessible from anywhere in the code and are cheap to read as they only require
reading an atomic value. This means that they can be used even in hot code
paths.

Currently tunables support reloading boolean values and i64s. In the future,
I'd like to expand tunables to include more functionality, such as a rollout
percentage.

The `--tunables-config` flag points to a configerator spec that exports a
Tunables thrift struct. This allows differents tiers and Mononoke services to
have their own tunables. If this isn't provided, `MononokeTunables::default()`
will be used.

This diff adds a proc_macro that will generate the relevant `get` and `update`
methods for the fields added to a struct which derives `Tunables`. This struct is
then stored in a `once_cell` and can be accessed using `tunables::tunables()`.

To add a new tunable, add a field to the `MononokeTunables` struct that is of
type `AtomicBool` or `AtomicI64`. Update the relevant tunables configerator
config to include your new field, with the exact same name.

Removing a tunable from `MononokeTunables` is fine, as is removing a tunable
from configerator.

If the `--tunables-config` path isn't passed, then a default tunables config
located at `scm/mononoke/tunables/default` will be loaded. There is also the
`--disable-tunables` flag that won't load anything from configerator, it
will instead use the `Tunable` struct's `default()` method to initialise it.
This is useful in integration tests.

Reviewed By: StanislavGlebik

Differential Revision: D21177252

fbshipit-source-id: 02a93c1ceee99066019b23d81ea308e4c565d371
2020-04-30 16:08:30 -07:00
Harvey Hunt
54e01c713a mononoke: Add init_mononoke helper function and update call sites
Summary:
Add a new function that initialises common parts of Mononoke:
- cachelib
- logging
- runtime

In the next diff, I will also update `init_mononoke` to initialise tunables
too.

I left some users of `init_runtime` unchanged (such as `mononoke_admin`) as
they conditionally enable cachelib.

Reviewed By: krallin

Differential Revision: D21177253

fbshipit-source-id: 40b534d1b244fd8dcd05e1cff1c9f3edfb32a4b9
2020-04-28 09:03:09 -07:00
Harvey Hunt
b24074ac35 mononoke: Add get_config_path helper to cmdlib
Summary:
Multiple functions in cmdlib were looking for
`"mononoke-config-path"`. Make it into a constant and provide a helper function
to reduce duplication.

Further, update `read_common_config` to accept `impl AsRef<Path>` to make
calling the function easier.

Reviewed By: farnz

Differential Revision: D21202528

fbshipit-source-id: 96cad817ed47be0f207965ad2bc33af13ca8b5fd
2020-04-23 07:43:37 -07:00
Stanislau Hlebik
3193f4fab3 mononoke: fail with non-zero exit code in serve_forever if failure happened
Summary:
Previously our jobs would have exit with error code 0 even if there was an
actual error. The reason for this was because error was just ignored (or rather
just printed to stderr).
This is not a huge problem but it makes tw output confusing - it shows that
the task was "Completed" while in reality it "Failed"

Reviewed By: ahornby

Differential Revision: D20693297

fbshipit-source-id: 4f615e2ef11f2edbb9bdbcf49cb1635929fdae89
2020-04-17 14:57:56 -07:00
Gabriel Russo
03d4e52ab3 Bump tokio to 0.2.13
Summary:
This is needed because the tonic crate (see the diff stack) relies on tokio ^0.2.13

We can't go to a newer version because a bug that affects mononoke was introduced on 0.2.14 (discussion started on T65261126). The issue was reported upstream https://github.com/tokio-rs/tokio/issues/2390

This diff simply changed the version number on `fbsource/third-party/rust/Cargo.toml` and ran `fbsource/third-party/rust/reindeer/vendor`.

Also ran `buck run //common/rust/cargo_from_buck:cargo_from_buck` to fix the tokio version on generated cargo files

Reviewed By: krallin

Differential Revision: D21043344

fbshipit-source-id: e61797317a581aa87a8a54e9e2ae22655f22fb97
2020-04-15 12:18:00 -07:00
Simon Farnsworth
92fce3d518 Clean out unused deps from our TARGETS files
Summary:
We had accumulated lots of unused dependendencies, and had several test_deps in deps instead. Clean this all up to reduce build times and speed up autocargo processing.

Net removal is of around 500 unneeded dependency lines, which represented false dependencies; by removing them, we should get more parallelism in dev builds, and less overbuilding in CI.

Reviewed By: krallin, StanislavGlebik

Differential Revision: D20999762

fbshipit-source-id: 4db3772cbc3fb2af09a16601bc075ae8ed6f0c75
2020-04-14 03:38:11 -07:00
Jeremy Fitzhardinge
28830035dd rust: regenerate autocargo for tokio rollback
Reviewed By: dtolnay

Differential Revision: D20956714

fbshipit-source-id: f13256350cc7082543c7b69231a783b262f8a4d8
2020-04-10 01:12:57 -07:00
Lukas Piatkowski
2e7baa454b mononoke: cover more crates with OSS buildability that depend on cmdlib crate
Reviewed By: krallin

Differential Revision: D20735803

fbshipit-source-id: d4159d16384ff795717f6ccdd278b6b4af45d1ab
2020-04-08 03:09:07 -07:00
Lukas Piatkowski
8e9df760c5 mononoke: make cmdlib OSS buildable
Summary: Some methods that were unused or barely used outside of the cmdlib crate were made non-public (parse_caching, CachelibSettings, init_cachelib_from_settings).

Reviewed By: krallin

Differential Revision: D20671251

fbshipit-source-id: 232e786fa5af5af543239aca939cb15ca2d6bc10
2020-04-08 03:09:06 -07:00
Mark Thomas
640f272598 migrate from sql_ext::SqlConstructors to sql_construct
Summary:
Migrate the configuration of sql data managers from the old configuration using `sql_ext::SqlConstructors` to the new configuration using `sql_construct::SqlConstruct`.

In the old configuration, sharded filenodes were included in the configuration of remote databases, even when that made no sense:
```
[storage.db.remote]
db_address = "main_database"
sharded_filenodes = { shard_map = "sharded_database", shard_num = 100 }

[storage.blobstore.multiplexed]
queue_db = { remote = {
    db_address = "queue_database",
    sharded_filenodes = { shard_map = "valid_config_but_meaningless", shard_num = 100 }
}
```

This change separates out:
* **DatabaseConfig**, which describes a single local or remote connection to a database, used in configuration like the queue database.
* **MetadataDatabaseConfig**, which describes the multiple databases used for repo metadata.

**MetadataDatabaseConfig** is either:
* **Local**, which is a local sqlite database, the same as for **DatabaseConfig**; or
* **Remote**, which contains:
    * `primary`, the database used for main metadata.
    * `filenodes`, the database used for filenodes, which may be sharded or unsharded.

More fields can be added to **RemoteMetadataDatabaseConfig** when we want to add new databases.

New configuration looks like:
```
[storage.metadata.remote]
primary = { db_address = "main_database" }
filenodes = { sharded = { shard_map = "sharded_database", shard_num = 100 } }

[storage.blobstore.multiplexed]
queue_db = { remote = { db_address = "queue_database" } }
```

The `sql_construct` crate facilitates this by providing the following traits:

* **SqlConstruct** defines the basic rules for construction, and allows construction based on a local sqlite database.
* **SqlShardedConstruct** defines the basic rules for construction based on sharded databases.
* **FbSqlConstruct** and **FbShardedSqlConstruct** allow construction based on unsharded and sharded remote databases on Facebook infra.
* **SqlConstructFromDatabaseConfig** allows construction based on the database defined in **DatabaseConfig**.
* **SqlConstructFromMetadataDatabaseConfig** allows construction based on the appropriate database defined in **MetadataDatabaseConfig**.
* **SqlShardableConstructFromMetadataDatabaseConfig** allows construction based on the appropriate shardable databases defined in **MetadataDatabaseConfig**.

Sql database managers should implement:

* **SqlConstruct** in order to define how to construct an unsharded instance from a single set of `SqlConnections`.
* **SqlShardedConstruct**, if they are shardable, in order to define how to construct a sharded instance.
* If the database is part of the repository metadata database config, either of:
    * **SqlConstructFromMetadataDatabaseConfig** if they are not shardable.  By default they will use the primary metadata database, but this can be overridden by implementing `remote_database_config`.
    * **SqlShardableConstructFromMetadataDatabaseConfig** if they are shardable.  They must implement `remote_database_config` to specify where to get the sharded or unsharded configuration from.

Reviewed By: StanislavGlebik

Differential Revision: D20734883

fbshipit-source-id: bb2f4cb3806edad2bbd54a47558a164e3190c5d1
2020-04-02 05:27:16 -07:00
Lukas Piatkowski
7fa825d40c rust-shed: move cached_config to the shed
Reviewed By: ahornby

Differential Revision: D20650304

fbshipit-source-id: 5fc704ce2964b9595722c3cd9c6f1dbd395a52ee
2020-03-31 04:02:46 -07:00
Stanislau Hlebik
4241ca0272 mononoke: add repo name in BlobRepo
Summary:
A lot of callsites want to know repo name. Currently they need to pass it from
the place where repo was initialized, and that's quite awkward, and in some
places even impossible (i.e. in derived data, where I want to log reponame).

This diff adds reponame in BlobRepo

Reviewed By: krallin

Differential Revision: D20363065

fbshipit-source-id: 5e2eb611fb9d58f8f78638574fdcb32234e5ca0d
2020-03-10 08:32:33 -07:00
Stanislau Hlebik
b0cb300af8 mononoke: introduce BlobrepoBuilder
Summary:
The goal of the whole stack is quite simple (add reponame field to BlobRepo), but
this stack also tries to make it easier to initialize BlobRepo.

To do that BlobrepoBuilder was added. It now accepts RepoConfig instead of 6
different fields from RepoConfig - that makes it easier to pass a field from
config into BlobRepo. It also allows to customize BlobRepo. Currently it's used
just to add redaction override, but later we can extend it for other use cases
as well, with the hope that we'll be able to remove a bunch of repo-creation
functions from cmdlib.

Because of BlobrepoBuilder we no longer need open_blobrepo function. Later we
might consider removing open_blobrepo_given_datasources as well.

Note that this diff *adds* a few new clones. I don't consider it being a big
problem, though I'm curious to hear your thoughts folks.

Note that another option for the implementation would be to take a reference to objects
instead of taking them by value. I briefly looked into how they used, and lot of them are passed to the
objects that actually take ownership of what's inside these config fields. I.e. Blobstore essentially takes ownership
of BlobstoreOptions, because it needs to store manifold bucket name.
Same for scuba_censored_table, filestore_params, bookmarks_cache_ttl etc. So unless I'm missing anything, we can
either pass them as reference and then we'll have to copy them, or we can
just pass a value from BlobrepoBuilder directly.

Reviewed By: krallin

Differential Revision: D20312567

fbshipit-source-id: 14634f5e14f103b110482557254f084da1c725e1
2020-03-09 12:05:57 -07:00
Stanislau Hlebik
c1ea56c17c mononoke: asyncify blobrepo_factory
Summary:
Note that comparing to many other asyncifying efforts, this one actually adds
one more clone instead of removing them. This is the clone of a logger field.
That shouldn't matter much because it can be cleaned up later and because this
function will be called once per repo.

Reviewed By: krallin

Differential Revision: D20311122

fbshipit-source-id: ace2a108790b1423f8525d08bdea9dc3a2e3c37c
2020-03-09 05:02:05 -07:00
Lukas Piatkowski
7ddcdd818c mononoke: make sql_ext OSS buildable
Summary: separate out the Facebook-specific pieces of the sql_ext crate

Reviewed By: ahornby

Differential Revision: D20218219

fbshipit-source-id: e933c7402b31fcd5c4af78d5e70adafd67e91ecd
2020-03-06 01:33:38 -08:00
David Tolnay
754a755eee rust: Rename tokio_preview:: to tokio::
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/

This codemod replaces all dependencies on `//common/rust/renamed:tokio-preview` with `fbsource//third-party/rust:tokio-preview` and their uses in Rust code from `tokio_preview::` to `tokio::`.

This does not introduce any collisions with `tokio::` meaning 0.1 tokio because D20235404 previously renamed all of those to `tokio_old::` in crates that depend on both 0.1 and 0.2 tokio.

This is the tokio version of what D20213432 did for futures.

Codemod performed by:

```
rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
    -x \
    buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:tokio-preview, 1))" \
| xargs sed -i 's,\btokio_preview::,tokio::,'

rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:tokio-preview,fbsource//third-party/rust:tokio-preview,'
```

Reviewed By: k21

Differential Revision: D20236557

fbshipit-source-id: 15068b93a0a944d6249a1d9f63840a4c61c9c1ba
2020-03-05 14:25:10 -08:00
David Tolnay
e988a88be9 rust: Rename futures_preview:: to futures::
Summary:
Context: https://fb.workplace.com/groups/rust.language/permalink/3338940432821215/

This codemod replaces *all* dependencies on `//common/rust/renamed:futures-preview` with `fbsource//third-party/rust:futures-preview` and their uses in Rust code from `futures_preview::` to `futures::`.

This does not introduce any collisions with `futures::` meaning 0.1 futures because D20168958 previously renamed all of those to `futures_old::` in crates that depend on *both* 0.1 and 0.3 futures.

Codemod performed by:

```
rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
    -x \
    buck query "labels(srcs, rdeps(%Ss, //common/rust/renamed:futures-preview, 1))" \
| xargs sed -i 's,\bfutures_preview::,futures::,'

rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| xargs sed -i 's,//common/rust/renamed:futures-preview,fbsource//third-party/rust:futures-preview,'
```

Reviewed By: k21

Differential Revision: D20213432

fbshipit-source-id: 07ee643d350c5817cda1f43684d55084f8ac68a6
2020-03-03 11:01:20 -08:00
David Tolnay
fe65402e46 rust: Move futures-old rdeps to renamed futures-old
Summary:
In targets that depend on *both* 0.1 and 0.3 futures, this codemod renames the 0.1 dependency to be exposed as futures_old::. This is in preparation for flipping the 0.3 dependencies from futures_preview:: to plain futures::.

rs changes performed by:

```
rg \
    --files-with-matches \
    --type-add buck:TARGETS \
    --type buck \
    --glob '!/experimental' \
    --regexp '(_|\b)rust(_|\b)' \
| sed 's,TARGETS$,:,' \
| xargs \
    -x \
    buck query "labels(srcs,
        rdeps(%Ss, fbsource//third-party/rust:futures-old, 1)
        intersect
        rdeps(%Ss, //common/rust/renamed:futures-preview, 1)
    )" \
| xargs sed -i 's/\bfutures::/futures_old::/'
```

Reviewed By: jsgf

Differential Revision: D20168958

fbshipit-source-id: d2c099f9170c427e542975bc22fd96138a7725b0
2020-03-02 21:02:50 -08:00