facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-10 08:47:12 +03:00
Code Issues Projects Releases Wiki Activity
eab97b6123
sapling/eden/mononoke/Cargo.toml

359 lines
11 KiB
TOML
Raw Normal View History

cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
[package]
name = "eden_mononoke"
version = "0.1.0"
autocargo v1: changes to match autocargo v2 generation results. Summary: The changes (and fixes) needed were: - Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml) - Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl` - Rearrange some attributes, like features, authors, edition etc. - Authors to use " instead of ' - Features to be sorted - Sort all dependencies as one instead of grouping third party and fbcode dependencies together - Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is. Reviewed By: zertosh Differential Revision: D26544150 fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 22:02:06 +03:00
authors = ["Facebook"]
edition = "2018"
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
license = "GPLv2+"
mononoke/admin: make it OSS buildable Reviewed By: krallin Differential Revision: D22458187 fbshipit-source-id: 05d321bc1aded67fb2eca851b4b1ad4a8bd49d52
2020-07-11 08:08:09 +03:00
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
[[bin]]
make backfill_git_mapping more generic and supporing svn Summary: backfill_git_mapping was great for backfilling git mappings, let's use it for svn Reviewed By: StanislavGlebik Differential Revision: D26453407 fbshipit-source-id: 3ee61722ac729af5a2fc4187fda3341069bb53f6
2021-02-17 23:40:17 +03:00
name = "backfill_mapping"
path = "cmds/backfill_mapping.rs"
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
[[bin]]
name = "benchmark_storage_config"
path = "cmds/benchmark_storage_config/main.rs"
[[bin]]
name = "bonsai_verify"
path = "cmds/bonsai_verify/main.rs"
autocargo v2: rollout and replace v1 in all use-cases Summary: This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here. Arc lint support: V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff: - Adds running `autocargo` as part of `arc lint-rust` Mergedriver update: - Mergedriver used in resolving conflicts on commits is now pointing to V2 - It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for: - Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section) - "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string - "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo - "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added) - Files generated from thrift_library rules that do not support "rust" language are removed - Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored Others: - Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo` - Updated following files to use V2: - `common/rust/tools/reindeer/version-bump` - `remote_execution/rust/setup.sh` - Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing Reviewed By: zertosh Differential Revision: D26728789 fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
2021-03-02 17:41:22 +03:00
test = false
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
[[bin]]
name = "configlint"
path = "cmds/configlint.rs"
[[bin]]
name = "dumprev"
path = "cmds/dumprev.rs"
autocargo v2: rollout and replace v1 in all use-cases Summary: This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here. Arc lint support: V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff: - Adds running `autocargo` as part of `arc lint-rust` Mergedriver update: - Mergedriver used in resolving conflicts on commits is now pointing to V2 - It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for: - Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section) - "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string - "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo - "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added) - Files generated from thrift_library rules that do not support "rust" language are removed - Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored Others: - Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo` - Updated following files to use V2: - `common/rust/tools/reindeer/version-bump` - `remote_execution/rust/setup.sh` - Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing Reviewed By: zertosh Differential Revision: D26728789 fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
2021-03-02 17:41:22 +03:00
test = false
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
[[bin]]
name = "idxdump"
path = "cmds/idxdump.rs"
autocargo v2: rollout and replace v1 in all use-cases Summary: This diff rollouts V2 of autocargo in an atomic way so there are quite a few things done here. Arc lint support: V1 used to be part of the default fbsource `arc lint` engine, but since V2 calls buck it must live in a separate lint engine. So this diff: - Adds running `autocargo` as part of `arc lint-rust` Mergedriver update: - Mergedriver used in resolving conflicts on commits is now pointing to V2 - It handles files in `public_autocargo/` directories in addition to the ones containig generation preamble Including regeneration results of running `common/rust/cargo_from_buck/bin/autocargo`. All the differences are accounted for: - Some sections and attributes are removed as they can be autodiscovered by Cargo (like `lib.path = "src/lib.rs"` or empty [lib] section) - "readme" attribute is properly defined as relative to Cargo.toml location rather than as hardcoded string - "unittest = false" on a Buck rule propagates as "test = false; doctest = false" to Cargo - "rusqlite" is not special-cased anymore, so the "budled" feature will have to be enabled using custom configuration if required by the project (for rust-shed in order to not break windows builds a default feature section was added) - Files generated from thrift_library rules that do not support "rust" language are removed - Custom .bzl rules that create rust artifacts (like `rust_python_extension`) are no longer ignored Others: - Changed `bin/cargo-autocargo` to be a wrapper for calling V2 via `cargo autocargo` - Updated following files to use V2: - `common/rust/tools/reindeer/version-bump` - `remote_execution/rust/setup.sh` - Removed few files from V1 that would otherwise interfere with V2 automatic regeneration/linting/testing Reviewed By: zertosh Differential Revision: D26728789 fbshipit-source-id: d1454e7ce658a2d3194704f8d77b12d688ec3e64
2021-03-02 17:41:22 +03:00
test = false
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
Command to manually scrub keys supplied on stdin Summary: For populating the XDB blobstore, we'd like to copy data from Manifold - the easiest way to do that is to exploit MultiplexedBlobstore's scrub mode to copy data directly. Reviewed By: krallin Differential Revision: D22373838 fbshipit-source-id: 550a9c73e79059380337fa35ac94fe1134378196
2020-07-10 10:59:29 +03:00
[[bin]]
name = "manual_scrub"
path = "cmds/manual_scrub/main.rs"
Implement a baseline dumb packer Summary: This is a baseline packer, to let us experiment with packing. It chooses the dictionary that gives us the smallest size, but does nothing intelligent around key handling. Note that at this point, packing is not always a win - e.g. the alias blobs are 432 bytes in total, but the resulting pack is 1528 bytes. Reviewed By: ahornby Differential Revision: D27795486 fbshipit-source-id: 05620aadbf7b680cbfcb0932778f5849eaab8a48
2021-04-20 22:44:12 +03:00
[[bin]]
name = "packer"
path = "cmds/packer/main.rs"
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
[[bin]]
name = "rechunker"
path = "cmds/rechunker.rs"
[[bin]]
name = "revlogrepo"
path = "cmds/revlogrepo.rs"
cmds: add segmented_changelog seeder Summary: Mononoke command for running the SegmentedChangelogSeeder for an existing repository. The result is going to be a new IdMap version in the metadata store and a new IdDag stored in the the blobstore resulting in a brand new SegmentedChangelog bundle. Reviewed By: krallin Differential Revision: D24096963 fbshipit-source-id: 1eaf78392d66542d9674a99ad0a741f24bc2cb1b
2020-10-08 19:41:16 +03:00
[[bin]]
name = "segmented_changelog_seeder"
path = "cmds/segmented_changelog_seeder.rs"
Support marking garbage in SQLBlob Summary: We want to be able to detect garbage blobs by looking at generation numbers. Update generation numbers on put, and have a mark command exist to mark blobs as not garbage. Reviewed By: ahornby Differential Revision: D23989289 fbshipit-source-id: d96f38649151e3dbd5297cffc262776e74f6cc86
2020-10-28 20:22:18 +03:00
[[bin]]
segmented_changelog: add segmented changelog tailer command Summary: The command reads the last SegmentedChangelog that was saved for a repository and updates head to match a given bookmark (master). Right now this is just a command that works on one repository. Follow up changes will look at deployment options and handling multiple repositories. Reviewed By: krallin Differential Revision: D24516438 fbshipit-source-id: 8f04f9426c2f2d7748c5363d2dbdf9f3acb79ddd
2020-10-30 03:35:41 +03:00
name = "segmented_changelog_tailer"
path = "cmds/segmented_changelog_tailer.rs"
[[bin]]
Support marking garbage in SQLBlob Summary: We want to be able to detect garbage blobs by looking at generation numbers. Update generation numbers on put, and have a mark command exist to mark blobs as not garbage. Reviewed By: ahornby Differential Revision: D23989289 fbshipit-source-id: d96f38649151e3dbd5297cffc262776e74f6cc86
2020-10-28 20:22:18 +03:00
name = "sqlblob_gc"
path = "cmds/sqlblob_gc/main.rs"
utility to keep streaming clone data warm Summary: This is streaming clone warmup binary as per https://fb.quip.com/hfuBAdYnzr9M Reviewed By: StanislavGlebik Differential Revision: D23347029 fbshipit-source-id: f187a2f3529a7eae5998bab199228bfbe6057e6e
2020-09-01 17:12:10 +03:00
[[bin]]
name = "streaming_clone_warmup"
path = "cmds/streaming_clone_warmup/main.rs"
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
[dependencies]
autocargo v1: changes to match autocargo v2 generation results. Summary: The changes (and fixes) needed were: - Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml) - Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl` - Rearrange some attributes, like features, authors, edition etc. - Authors to use " instead of ' - Features to be sorted - Sort all dependencies as one instead of grouping third party and fbcode dependencies together - Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is. Reviewed By: zertosh Differential Revision: D26544150 fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 22:02:06 +03:00
anyhow = "1.0"
ascii = "1.0"
rust: remove patch for async-compression Summary: Upstream crate has landed my PR for zstd 1.4.9 support and made a release, so can remove this patch now. Reviewed By: ikostia Differential Revision: D28221163 fbshipit-source-id: b95a6bee4f0c8d11f495dc17b2737c9ac9142b36
2021-05-05 22:19:32 +03:00
async-compression = { version = "0.3.8", features = ["all"] }
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
blobrepo = { version = "0.1.0", path = "blobrepo" }
blobrepo_hg = { version = "0.1.0", path = "blobrepo/blobrepo_hg" }
blobrepo_utils = { version = "0.1.0", path = "blobrepo_utils" }
blobstore = { version = "0.1.0", path = "blobstore" }
blobstore_factory = { version = "0.1.0", path = "blobstore/factory" }
bookmarks = { version = "0.1.0", path = "bookmarks" }
autocargo v1: changes to match autocargo v2 generation results. Summary: The changes (and fixes) needed were: - Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml) - Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl` - Rearrange some attributes, like features, authors, edition etc. - Authors to use " instead of ' - Features to be sorted - Sort all dependencies as one instead of grouping third party and fbcode dependencies together - Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is. Reviewed By: zertosh Differential Revision: D26544150 fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 22:02:06 +03:00
bytesize = "1.0"
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
cacheblob = { version = "0.1.0", path = "blobstore/cacheblob" }
cached_config = { version = "0.1.0", git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
autocargo v1: changes to match autocargo v2 generation results. Summary: The changes (and fixes) needed were: - Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml) - Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl` - Rearrange some attributes, like features, authors, edition etc. - Authors to use " instead of ' - Features to be sorted - Sort all dependencies as one instead of grouping third party and fbcode dependencies together - Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is. Reviewed By: zertosh Differential Revision: D26544150 fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 22:02:06 +03:00
clap = "2.33"
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
cloned = { version = "0.1.0", git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
cmdlib = { version = "0.1.0", path = "cmdlib" }
context = { version = "0.1.0", path = "server/context" }
autocargo v1: changes to match autocargo v2 generation results. Summary: The changes (and fixes) needed were: - Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml) - Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl` - Rearrange some attributes, like features, authors, edition etc. - Authors to use " instead of ' - Features to be sorted - Sort all dependencies as one instead of grouping third party and fbcode dependencies together - Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is. Reviewed By: zertosh Differential Revision: D26544150 fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 22:02:06 +03:00
criterion = "=0.3.1"
mononoke: add progress to manual_scrub Summary: Some manual scrub runs can take a long time. Provide progress feedback logging. Includes a --quiet option for when progress reporting not required. Reviewed By: farnz Differential Revision: D27588449 fbshipit-source-id: 00840cdf2022358bc10398f08b3bbf3eeec2b299
2021-04-06 18:42:33 +03:00
derive_more = "0.99.3"
mononoke: use MononokeEnvironment in RepoFactory Summary: There is a very frustrating operation that happens often when working on the Mononoke code base: - You want to add a flag - You want to consume it in the repo somewhere Unfortunately, when we need to do this, we end up having to thread this from a million places and parse it out in every single main() we have. This is a mess, and it results in every single Mononoke binary starting with heaps of useless boilerplate: ``` let matches = app.get_matches(); let (caching, logger, mut runtime) = matches.init_mononoke(fb)?; let config_store = args::init_config_store(fb, &logger, &matches)?; let mysql_options = args::parse_mysql_options(&matches); let blobstore_options = args::parse_blobstore_options(&matches)?; let readonly_storage = args::parse_readonly_storage(&matches); ``` So, this diff updates us to just use MononokeEnvironment directly in RepoFactory, which means none of that has to happen: we can now add a flag, parse it into MononokeEnvironment, and get going. While we're at it, we can also remove blobstore options and all that jazz from MononokeApiEnvironment since now it's there in the underlying RepoFactory. Reviewed By: HarveyHunt Differential Revision: D27767700 fbshipit-source-id: e1e359bf403b4d3d7b36e5f670aa1a7dd4f1d209
2021-04-16 20:26:03 +03:00
environment = { version = "0.1.0", path = "cmdlib/environment" }
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
failure_ext = { version = "0.1.0", git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
fbinit = { version = "0.1.0", git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
filestore = { version = "0.1.0", path = "filestore" }
third-party/rust: update futures Summary: Those newer versions of Futures have compatibility improvements with Tokio, notably: - https://github.com/rust-lang/futures-rs/pull/2333 - https://github.com/rust-lang/futures-rs/pull/2358 Reviewed By: farnz Differential Revision: D26778794 fbshipit-source-id: 5a9dc002083e5edfa5c614d8d2242e586a93fcf6
2021-03-04 17:40:30 +03:00
futures = { version = "0.3.13", features = ["async-await", "compat"] }
futures-old = { package = "futures", version = "0.1.31" }
autocargo v1: changes to match autocargo v2 generation results. Summary: The changes (and fixes) needed were: - Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml) - Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl` - Rearrange some attributes, like features, authors, edition etc. - Authors to use " instead of ' - Features to be sorted - Sort all dependencies as one instead of grouping third party and fbcode dependencies together - Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is. Reviewed By: zertosh Differential Revision: D26544150 fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 22:02:06 +03:00
futures-util = "0.3.7"
itertools = "0.8"
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
lock_ext = { version = "0.1.0", git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
mercurial_derived_data = { version = "0.1.0", path = "derived_data/mercurial_derived_data" }
mercurial_revlog = { version = "0.1.0", path = "mercurial/revlog" }
mercurial_types = { version = "0.1.0", path = "mercurial/types" }
metaconfig_types = { version = "0.1.0", path = "metaconfig/types" }
mononoke_types = { version = "0.1.0", path = "mononoke_types" }
Implement a baseline dumb packer Summary: This is a baseline packer, to let us experiment with packing. It chooses the dictionary that gives us the smallest size, but does nothing intelligent around key handling. Note that at this point, packing is not always a win - e.g. the alias blobs are 432 bytes in total, but the resulting pack is 1528 bytes. Reviewed By: ahornby Differential Revision: D27795486 fbshipit-source-id: 05620aadbf7b680cbfcb0932778f5849eaab8a48
2021-04-20 22:44:12 +03:00
packblob = { version = "0.1.0", path = "blobstore/packblob" }
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
prefixblob = { version = "0.1.0", path = "blobstore/prefixblob" }
autocargo v1: changes to match autocargo v2 generation results. Summary: The changes (and fixes) needed were: - Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml) - Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl` - Rearrange some attributes, like features, authors, edition etc. - Authors to use " instead of ' - Features to be sorted - Sort all dependencies as one instead of grouping third party and fbcode dependencies together - Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is. Reviewed By: zertosh Differential Revision: D26544150 fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 22:02:06 +03:00
rand = { version = "0.7", features = ["small_rng"] }
benchmark_storage_config: remove dependency on blobrepo_factory Summary: Use the equivalent function from `repo_factory`. Reviewed By: krallin Differential Revision: D27363470 fbshipit-source-id: dce3cf843174caa2f9ef7083409e7935749be4cd
2021-04-07 23:59:27 +03:00
repo_factory = { version = "0.1.0", path = "repo_factory" }
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
revset = { version = "0.1.0", path = "revset" }
scuba_ext = { version = "0.1.0", path = "common/scuba_ext" }
segmented_changelog = { version = "0.1.0", path = "segmented_changelog" }
segmented_changelog: update OnDemandUpdateDag to have smaller critical sections Summary: The on demand update code we have is the most basic logic that we could have. The main problem is that it has long and redundant write locks. This change reduces the write lock strictly to the section that has to update the in memory IdDag. Updating the Dag has 3 phases: * loading the data that is required for the update; * updating the IdMap; * updating the IdDag; The Dag can function well for serving requests as long as the commits involved have been built so we want to have easy read access to both the IdMap and the IdDag. The IdMap is a very simple structure and because it's described as an Arc<dyn IdMap> we push the update locking logic to the storage. The IdDag is a complicated structure that we ask to update itself. Those functions take mutable references. Updating the storage of the iddag to hide the complexities of locking is more difficult. We deal with the IdDag directly by wrapping it in a RwLock. The RwLock allows for easy read access which we expect to be the predominant access pattern. Updates to the dag are not completely stable so racing updates can have conflicting results. In case of conflics one of the update processes would have to restart. It's easier to reason about the process if we just allow one "thread" to start an update process. The update process is locked by a sync mutex. The "threads" that fail the race to update are asked to wait until the ongoing update is complete. The waiters will poll on a shared future that tracks the ongoing dag update. After the update is complete the waiters will go back to checking if the data they have is available in the dag. It is possible that the dag is updated in between determining that the an update is needed and acquiring the ongoing_update lock. This is fine because the update building process checks the state of dag before the dag and updates only what is necessary if necessary. Reviewed By: krallin Differential Revision: D26508430 fbshipit-source-id: cd3bceed7e0ffb00aee64433816b5a23c0508d3c
2021-02-23 05:14:45 +03:00
serde = { version = "=1.0.118", features = ["derive", "rc"] }
autocargo v1: changes to match autocargo v2 generation results. Summary: The changes (and fixes) needed were: - Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml) - Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl` - Rearrange some attributes, like features, authors, edition etc. - Authors to use " instead of ' - Features to be sorted - Sort all dependencies as one instead of grouping third party and fbcode dependencies together - Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is. Reviewed By: zertosh Differential Revision: D26544150 fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 22:02:06 +03:00
serde_derive = "1.0"
slog max_level_debug -> trace Reviewed By: Imxset21 Differential Revision: D28097080 fbshipit-source-id: 7d417f8256922926cf379d9c2fb3249f6d2544ef
2021-05-03 20:29:23 +03:00
slog = { version = "2.5", features = ["max_level_trace"] }
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
sql_construct = { version = "0.1.0", path = "common/sql_construct" }
sql_ext = { version = "0.1.0", path = "common/rust/sql_ext" }
sqlblob = { version = "0.1.0", path = "blobstore/sqlblob" }
streaming_clone = { version = "0.1.0", path = "repo_client/streaming_clone" }
mononoke: add checkpoint for input to manual_scrub Summary: Reduce amount of manual steps needed to restart a manual scrub by checkpointing where it has got to to a file. Differential Revision: D27588450 fbshipit-source-id: cb0eda7d6ff57f3bb18a6669d38f5114ca9196b0
2021-04-07 19:29:57 +03:00
tempfile = "3.1"
mononoke: sync changeset implementation for megarepo Summary: First stab at implementing sync changeset functionality for megarepo. Reviewed By: ikostia Differential Revision: D28357210 fbshipit-source-id: 660e3f9914737929391ab1b29f891b3b5dd47638
2021-05-13 20:03:09 +03:00
tokio = { version = "1.5", features = ["full", "test-util"] }
tokio-stream = { version = "0.1.4", features = ["fs", "io-util", "net", "signal", "sync", "time"] }
convert BlobRepoHg to new type futures Summary: Convert all BlobRepoHg methods to new type futures Reviewed By: StanislavGlebik Differential Revision: D25471540 fbshipit-source-id: c8e99509d39d0e081d082097cbd9dbfca431637e
2020-12-17 18:43:34 +03:00
toml = "=0.5.7"
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
[patch.crates-io]
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
addr2line = { git = "https://github.com/gimli-rs/addr2line.git", rev = "0b6b6018b5b252a18e628fba03885f7d21844b3c" }
mononoke: do not report bookmark validation error if we get an infra error Summary: We got a few ubns because one bookmark validator in a single region wasn't able to connect to mysql and was reporting errors. This diff fixes by separating logical and infra errors Reviewed By: ikostia Differential Revision: D25092364 fbshipit-source-id: 93f4be1a7e0467051b7b8d927eef9b4f5cd6a983
2020-11-19 23:56:30 +03:00
bytecount = { git = "https://github.com/llogiq/bytecount", rev = "469eaf8395c99397cd64d059737a9054aa014088" }
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
chashmap = { git = "https://gitlab.redox-os.org/ahornby/chashmap", rev = "901ace2ca3cdbc2095adb1af111d211e254e2aae" }
const-random = { git = "https://github.com/fbsource/const-random", rev = "374c5b46427fe2ffbf6acbd9c1687e0f1a809f95" }
third-party/rust: patch curl and curl-sys Summary: Update the `curl` and `curl-sys` crates to use a patched version that supports `CURLOPT_SSLCERT_BLOB` and similar config options that allow the use of in-memory TLS credentials. These options were added last year in libcurl version `7.71.0`, but the Rust bindings have not yet been updated to support them. I intend to upstream this patch, but until then, this will allow us to use these options in fbcode. Reviewed By: quark-zju Differential Revision: D27633208 fbshipit-source-id: 911e0b8809bc0144ad8b32749e71208bd08458fd
2021-04-08 21:48:52 +03:00
curl = { git = "https://github.com/kulshrax/curl-rust", rev = "2a15bbd8dbbd54734313fa703a64db7ce6ddaff0" }
curl-sys = { git = "https://github.com/kulshrax/curl-rust", rev = "2a15bbd8dbbd54734313fa703a64db7ce6ddaff0" }
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
enumset = { git = "https://github.com/danobi/enumset", rev = "4c01c583c27a725948fededbfb3461c572a669a4" }
Work around lru-disk-cache UB in Rust 1.48 Summary: `lru-disk-cache` depends on an old version of `linked-hash-map` which contains UB in 1.48 (see https://github.com/mozilla/sccache/issues/813). They updated the deps in their repo months ago, but haven't pushed a new version. This diff makes us get `lru-disk-cache` directly from their GitHub instead. Reviewed By: dtolnay Differential Revision: D25134582 fbshipit-source-id: 05fd63a76b7095ebeea458645b92a83bbd8c4614
2020-11-21 03:58:47 +03:00
lru-disk-cache = { git = "https://github.com/mozilla/sccache", rev = "033ebaae69beeb0ac04e8c35d6ff1103487bd9a3" }
Imported a mysql_common patch to support FromRow for tuples up to arity 16 (#23) Summary: Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/23 Pull Request resolved: https://github.com/facebookincubator/resctl/pull/8081 Pull Request resolved: https://github.com/facebookexperimental/eden/pull/82 Imported a mysql_common patch to support FromRow for tuples upto arity 16 Context: https://fburl.com/zfnw7r86 Followed the guide: https://www.internalfb.com/intern/wiki/Rust-at-facebook/Managing_fbsource_third-party_with_Reindeer/#maintaining-local-change Reviewed By: marcelogomez Differential Revision: D28094262 fbshipit-source-id: fed48e3950e8a3ba3d7a15407522167e5ae41a98
2021-05-05 20:31:41 +03:00
mysql_common = { git = "https://github.com/iammxt/rust_mysql_common", rev = "0e4c86952f1e799960e736c0b2bb9d2a6d935bf1" }
common/rust: introduce fbinit-tokio Summary: This introduces a couple "trampoline" crates to use alongside fbinit in order to let callers choose which version of Tokio they want by selecting one or the other and adding it to their `TARGETS` alongside `fbinit` (right now, they need to include `tokio` there). The entrypoints here will be called by the expansion of the `#[fbinit::main]` and `#[fbinit::test]` macros for `async fn`-s. Right now, this isn't wired up: that happens in the next diff. In this diff, I'm just adding the two entrypoints. Reviewed By: mitrandir77 Differential Revision: D26754751 fbshipit-source-id: 1966dadf8bbe427ce4a1e90559a81790d8e56e7a
2021-03-03 15:06:49 +03:00
openssl = { git = "https://github.com/sfackler/rust-openssl", rev = "68fc8ba890d77986b06ca5ce29d1089285fbbcf9" }
openssl-sys = { git = "https://github.com/sfackler/rust-openssl", rev = "68fc8ba890d77986b06ca5ce29d1089285fbbcf9" }
Daily `common/rust/cargo_from_buck/bin/autocargo` Reviewed By: farnz Differential Revision: D27458795 fbshipit-source-id: 6d4f0e8ebcebe2f71e1152a61bf6511230c31af5
2021-03-31 12:32:58 +03:00
petgraph = { git = "https://github.com/jkeljo/petgraph", rev = "e3e9dd8632d23973fdc0b42c1117d5e5fc5fa384" }
cargo_from_buck: add the patch section to Cargo workspace and allow workspace appending to existing manifest Summary: Two changes here: 1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches. 2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file. Reviewed By: farnz Differential Revision: D22023144 fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
2020-06-17 16:26:05 +03:00
prost = { git = "https://github.com/gabrielrussoc/prost", branch = "protoc-runtime" }
prost-build = { git = "https://github.com/gabrielrussoc/prost", branch = "protoc-runtime" }
prost-derive = { git = "https://github.com/gabrielrussoc/prost", branch = "protoc-runtime" }
prost-types = { git = "https://github.com/gabrielrussoc/prost", branch = "protoc-runtime" }
r2d2_sqlite = { git = "https://github.com/jsgf/r2d2-sqlite.git", rev = "6d77a828ca0a3c507a3f58561532a1b6c66c7918" }
rustfilt = { git = "https://github.com/jsgf/rustfilt.git", rev = "8141fa7f1caee562ee8daffb2ddeca3d1f0d36e5" }
tokio-core = { git = "https://github.com/bolinfest/tokio-core", rev = "5f37aa3c627d56ee49154bc851d6930f5ab4398f" }
convert BlobRepoHg to new type futures Summary: Convert all BlobRepoHg methods to new type futures Reviewed By: StanislavGlebik Differential Revision: D25471540 fbshipit-source-id: c8e99509d39d0e081d082097cbd9dbfca431637e
2020-12-17 18:43:34 +03:00
toml = { git = "https://github.com/jsgf/toml-rs", branch = "dotted-table-0.5.7" }
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
[workspace]
members = [
".",
mononoke: update to tokio 1.x Summary: NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing. There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs). Here's a detailed list of what is going on: - I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace. - Lots of little Tokio changes: - Stream abstractions moving to `tokio-stream` - `tokio::time::delay_for` became `tokio::time::sleep` - `tokio::sync::watch::Sender::send` became `tokio::sync::watch::Sender::broadcast` - `tokio::sync::Semaphore::acquire` returns a `Result` now. - `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`). - `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case. - Runtime builders have changed quite a bit: - My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`. - Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out. - Tokio 1.x now uses Bytes 1.x: - At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore). - In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine. - In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x. - In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy. - tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to: - How we listen in Mononoke Server & LFS - How we connect in hgcli. - Note: all this stuff has test coverage. - The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this. - Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it. - I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x. Reviewed By: mitrandir77 Differential Revision: D26669620 fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
2021-04-28 17:35:21 +03:00
"admin",
"aliasverify",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"alpn",
mononoke: update to tokio 1.x Summary: NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing. There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs). Here's a detailed list of what is going on: - I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace. - Lots of little Tokio changes: - Stream abstractions moving to `tokio-stream` - `tokio::time::delay_for` became `tokio::time::sleep` - `tokio::sync::watch::Sender::send` became `tokio::sync::watch::Sender::broadcast` - `tokio::sync::Semaphore::acquire` returns a `Result` now. - `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`). - `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case. - Runtime builders have changed quite a bit: - My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`. - Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out. - Tokio 1.x now uses Bytes 1.x: - At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore). - In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine. - In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x. - In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy. - tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to: - How we listen in Mononoke Server & LFS - How we connect in hgcli. - Note: all this stuff has test coverage. - The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this. - Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it. - I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x. Reviewed By: mitrandir77 Differential Revision: D26669620 fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
2021-04-28 17:35:21 +03:00
"backfill_derived_data",
"benchmark_filestore",
benchmarks: add benchmark_large_directory Summary: Add a microbenchmark for deriving data with large directories. This benchmark creates a commit with 100k files in a single directory, and then derives data for that commit and 10 descendant commits, each of which add, modify and remove some files. Reviewed By: ahornby Differential Revision: D26947361 fbshipit-source-id: 4215f1ac806c53a112217ceb10e50cfad56f4f28
2021-03-11 15:25:00 +03:00
"benchmarks/derived_data",
benchmark: rename to benchmarks/simulated_repo Summary: Rename this benchmark to a specific name so that we can add new benchmarks. Differential Revision: D26947362 fbshipit-source-id: a1d060ee79781aa0ead51f284517471431418659
2021-03-11 15:25:00 +03:00
"benchmarks/simulated_repo",
mononoke: update to tokio 1.x Summary: NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing. There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs). Here's a detailed list of what is going on: - I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace. - Lots of little Tokio changes: - Stream abstractions moving to `tokio-stream` - `tokio::time::delay_for` became `tokio::time::sleep` - `tokio::sync::watch::Sender::send` became `tokio::sync::watch::Sender::broadcast` - `tokio::sync::Semaphore::acquire` returns a `Result` now. - `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`). - `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case. - Runtime builders have changed quite a bit: - My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`. - Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out. - Tokio 1.x now uses Bytes 1.x: - At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore). - In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine. - In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x. - In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy. - tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to: - How we listen in Mononoke Server & LFS - How we connect in hgcli. - Note: all this stuff has test coverage. - The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this. - Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it. - I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x. Reviewed By: mitrandir77 Differential Revision: D26669620 fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
2021-04-28 17:35:21 +03:00
"blobimport",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"blobimport_lib",
"blobimport_lib/consts",
"blobrepo",
"blobrepo/blobrepo_hg",
"blobrepo/blobsync",
"blobrepo/changeset_fetcher",
"blobrepo/common",
"blobrepo/errors",
"blobrepo/override",
"blobrepo/repo_blobstore",
"blobrepo_utils",
"blobstore",
"blobstore/blobstore_stats",
"blobstore/cacheblob",
"blobstore/chaosblob",
"blobstore/delayblob",
"blobstore/factory",
"blobstore/fileblob",
"blobstore/if",
"blobstore/logblob",
"blobstore/memblob",
"blobstore/multiplexedblob",
"blobstore/packblob",
"blobstore/packblob/if",
"blobstore/prefixblob",
"blobstore/readonlyblob",
"blobstore/redactedblobstore",
"blobstore/samplingblob",
"blobstore/sqlblob",
"blobstore/throttledblob",
"blobstore/virtually_sharded_blobstore",
mononoke: update to tokio 1.x Summary: NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing. There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs). Here's a detailed list of what is going on: - I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace. - Lots of little Tokio changes: - Stream abstractions moving to `tokio-stream` - `tokio::time::delay_for` became `tokio::time::sleep` - `tokio::sync::watch::Sender::send` became `tokio::sync::watch::Sender::broadcast` - `tokio::sync::Semaphore::acquire` returns a `Result` now. - `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`). - `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case. - Runtime builders have changed quite a bit: - My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`. - Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out. - Tokio 1.x now uses Bytes 1.x: - At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore). - In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine. - In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x. - In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy. - tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to: - How we listen in Mononoke Server & LFS - How we connect in hgcli. - Note: all this stuff has test coverage. - The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this. - Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it. - I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x. Reviewed By: mitrandir77 Differential Revision: D26669620 fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
2021-04-28 17:35:21 +03:00
"blobstore_healer",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"blobstore_sync_queue",
"bonsai_git_mapping",
"bonsai_git_mapping/git_mapping_pushrebase_hook",
"bonsai_globalrev_mapping",
"bonsai_globalrev_mapping/globalrev_pushrebase_hook",
"bonsai_globalrev_mapping/if",
"bonsai_hg_mapping",
"bonsai_hg_mapping/if",
"bonsai_svnrev_mapping",
"bonsai_svnrev_mapping/if",
"bookmarks",
"bookmarks/bookmarks_movement",
"bookmarks/bookmarks_types",
"bookmarks/dbbookmarks",
"bookmarks/warm_bookmarks_cache",
"bulkops",
"bulkops/bench",
"cache_warmup",
"changesets",
changesets: split implementation to a separate crate Summary: Keeping the `Changesets` trait as well as its implementations in the same crate means that users of `Changesets` also transitively depend on everything that is needed to implement it. Flatten the dependency graph a little by splitting it into two crates: most users of `Changesets` will only depend on the trait definition. Only the factories need depend on the implementations. Reviewed By: krallin Differential Revision: D27430612 fbshipit-source-id: 6b45fe4ae6b0fa1b95439be5ab491b1675c4b177
2021-04-29 16:10:17 +03:00
"changesets/changesets_impl",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"changesets/if",
"cmdlib",
mononoke: use MononokeEnvironment in RepoFactory Summary: There is a very frustrating operation that happens often when working on the Mononoke code base: - You want to add a flag - You want to consume it in the repo somewhere Unfortunately, when we need to do this, we end up having to thread this from a million places and parse it out in every single main() we have. This is a mess, and it results in every single Mononoke binary starting with heaps of useless boilerplate: ``` let matches = app.get_matches(); let (caching, logger, mut runtime) = matches.init_mononoke(fb)?; let config_store = args::init_config_store(fb, &logger, &matches)?; let mysql_options = args::parse_mysql_options(&matches); let blobstore_options = args::parse_blobstore_options(&matches)?; let readonly_storage = args::parse_readonly_storage(&matches); ``` So, this diff updates us to just use MononokeEnvironment directly in RepoFactory, which means none of that has to happen: we can now add a flag, parse it into MononokeEnvironment, and get going. While we're at it, we can also remove blobstore options and all that jazz from MononokeApiEnvironment since now it's there in the underlying RepoFactory. Reviewed By: HarveyHunt Differential Revision: D27767700 fbshipit-source-id: e1e359bf403b4d3d7b36e5f670aa1a7dd4f1d209
2021-04-16 20:26:03 +03:00
"cmdlib/environment",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"cmdlib/x_repo",
mononoke: add a tool to copy blobs from one repo to another Reviewed By: krallin Differential Revision: D28024373 fbshipit-source-id: 4954f6d322f924b9291326ef5a948a1c52230955
2021-04-27 18:53:40 +03:00
"cmds/copy_blobstore_keys",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"commit_rewriting/backsyncer",
"commit_rewriting/bookmark_renaming",
"commit_rewriting/bookmarks_validator",
"commit_rewriting/commit_validator",
"commit_rewriting/cross_repo_sync",
"commit_rewriting/cross_repo_sync/test_utils",
"commit_rewriting/live_commit_sync_config",
"commit_rewriting/megarepo",
"commit_rewriting/mononoke_x_repo_sync_job",
"commit_rewriting/movers",
"commit_rewriting/synced_commit_mapping",
"common/allocation_tracing",
"common/async_limiter",
"common/async_limiter/examples/tokio_v2",
"common/bounded_traversal",
"common/copy_utils",
"common/dedupmap",
"common/futures_watchdog",
"common/iterhelpers",
mononoke/rendezvous: introduce query batching Summary: This introduces a basic building block for query batching. I called this rendezvous, since it's about multiple queries meeting up in the same place :) There are a few (somewhat conflicting) goals this tries to satisfy, so let's go over them: 1), we'd like to reduce the total number of queries made by batch jobs. For example, group hg bonsai lookups made by the walker. Those jobs are characterized by the fact that they have a lot of queries to make, all the time. Here's an example: https://fburl.com/ods/zuiep7yh. 2), we'd like to reduce the overall number of connections held to MySQL by our tasks. The main way we achieve this is by reducing the maximum number of concurrent queries. Indeed, a high total number of queries doesn't necessarily result in a lot of connections as long as they're not concurrent, because we can reuse connections. On the other hand, if you dispatch 100 concurrent queries, that _does_ use 100 connections. This is something that applies to batch jobs due to their query volume, but also to "interactive" jobs like Mononoke Server or SCS, just not all the time. Here's an example: https://fburl.com/ods/o6gp07qp (you can see the query count is overall low, but sometimes spikes substantially). 2.1) It's also worth noting that concurrent queries are often the result of many clients wanting the same data, so deduplication is also useful here. 3), we also don't want to impact the latency of interactive jobs when they need to a little query here or there (i.e. it's largely fine if our jobs all hold a few connections to MySQL and use them somewhat consistently). 4), we'd like this to make it easier to do batching right. For example, if you have 100 Bonsais to map to hg, you should be able to just map and call `future::try_join_all` and have that do the right thing. 5), we don't want "bad" queries to affect other queries negatively. One example would be the occasional queries we make to Bonsai <-> Hg mapping in `known` for thousands (if not more) of rows. 6), we want this to be easy to incorporate into the codebase. So, how do we try to address all of this? Here's how: - We ... do batching, and we deduplicate requests in a batch. This is the easier bit and should address #1, #2 and #2.1, #4. - However, batching is conditional. We notably don't batch very large requests with the rest (addresses #5). We also don't batch small queries all the time: we only batch if we are observing a throughput of queries that suggests we can find some benefit in batching (this targets #3). - Finally, we have some utilities for common cases like having to group by repo id (this is `MultiRendezVous`), and this is all configurable via tunables (and the default is to not do anything). Reviewed By: StanislavGlebik Differential Revision: D27010317 fbshipit-source-id: 4a2397255f9785c6722c02e4d419438fd0aafa07
2021-03-19 18:49:00 +03:00
"common/rendezvous",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"common/rust/caching_ext",
"common/rust/slog_ext",
"common/rust/sql_ext",
"common/scribe_ext",
"common/scuba_ext",
"common/sql_construct",
mononoke/timeseries: introduce a basic crate for tracking time series Summary: I'd like to be able to track time series for access within Mononoke. The underlying use case here is that I want to be able to track the max count of connections in our SQL connection pools over time (and possibly other things in the future). Now, the obvious question is: why am I rolling my own? Well, as it turns out, there isn't really an implementation of this that I can reuse: - You might expect to be able to track the max of a value via fb303, but you can't: https://www.internalfb.com/intern/diffusion/FBS/browse/master/fbcode/fb303/ExportType.h?commit=0405521ec858e012c0692063209f3e13a2671043&lines=26-29 - You might go look in Folly, but you'll find that the time series there only supports tracking Sum & Average, but I want my timeseries to track Max (and in fact I'd like it to be sufficiently flexible to track anything I want): https://www.internalfb.com/intern/diffusion/FBS/browse/master/fbcode/folly/stats/BucketedTimeSeries.h It's not the first time I've ran into a need for something like this. I needed it in RendezVous to track connections over the last 2 N millisecond intervals, and we needed it in metagit for host draining as well (note that the implementation here is somewhat inspired by the implementation there). Reviewed By: mzr Differential Revision: D27678388 fbshipit-source-id: ba6d244b8bb848d4e1a12f9c6f54e3aa729f6c9c
2021-04-12 15:21:39 +03:00
"common/timeseries",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"common/topo_sort",
"common/type_map",
"common/uniqueheap",
"derived_data",
"derived_data/blame",
"derived_data/changeset_info",
"derived_data/changeset_info/if",
"derived_data/deleted_files_manifest",
"derived_data/fastlog",
"derived_data/filenodes",
"derived_data/fsnodes",
"derived_data/mercurial_derived_data",
"derived_data/skeleton_manifest",
"derived_data/test_utils",
"derived_data/unodes",
"derived_data/utils",
"edenapi_server",
"edenapi_service",
"fastreplay",
"filenodes",
"filenodes/if",
"filestore",
"git/git-pool",
"git/git_types",
"git/git_types/if",
"git/gitimport",
"git/import_tools",
"gotham_ext",
"hgcli",
"hgproto",
"hook_tailer",
"hooks",
"hooks/content-stores",
"hooks/hook_manager_factory",
mononoke: update to tokio 1.x Summary: NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing. There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs). Here's a detailed list of what is going on: - I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace. - Lots of little Tokio changes: - Stream abstractions moving to `tokio-stream` - `tokio::time::delay_for` became `tokio::time::sleep` - `tokio::sync::watch::Sender::send` became `tokio::sync::watch::Sender::broadcast` - `tokio::sync::Semaphore::acquire` returns a `Result` now. - `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`). - `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case. - Runtime builders have changed quite a bit: - My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`. - Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out. - Tokio 1.x now uses Bytes 1.x: - At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore). - In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine. - In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x. - In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy. - tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to: - How we listen in Mononoke Server & LFS - How we connect in hgcli. - Note: all this stuff has test coverage. - The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this. - Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it. - I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x. Reviewed By: mitrandir77 Differential Revision: D26669620 fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
2021-04-28 17:35:21 +03:00
"lfs_import",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"lfs_import_lib",
"lfs_protocol",
"lfs_server",
"load_limiter",
"manifest",
megarepo_api: introduce a MegarepoApi struct to rule them all Summary: This struct is intended to be a single entry-point to the megarepo logic. It is also intended to be owned directly by scs_server without the `Mononoke` struct (from `mononoke_api`) intermediary. In effect, this means that mononoke server won't be affected by `MegarepoApi` at all. Apart from adding this struct, this diff also adds instantiation of prod `AsyncMethodRequestQueue` and wires it up to the scs_server to enqueue and poll requests. Reviewed By: StanislavGlebik Differential Revision: D28356563 fbshipit-source-id: b67ee48387d794fd333d106d3ffd40124086c97e
2021-05-12 21:59:19 +03:00
"megarepo_api",
megarepo: basic version of async-requests crate Summary: This crate is a foundation for the async requests support in megarepo service. The idea is to be able to store serialized parameters in the blobstore upon request arrival, and to be able to query request results from the blobstore while polling. This diff manipulates the following classes of types: - param types for async methods: self-explanatory - response types: these contain only a resulting value of a completed successful execution - stored result types: these contain a result value of a completed execution. It may either be successful or failed. These types exist for the purpose of preserving execution result in the blobstore. - poll-response types: these contain and option of a response. If the optional value is empty, this means that the request is not yet ready - polling tokens: these are used by the client to ask about the processing status for a submitted request Apart from that, some of these types have both Rust and Thrift counterparts, mainly for the purposes of us being able to implement traits for Rust types. Relationships between these types are encoded in various traits and their associated types. The lifecycle of an async request is as follows therefore: 1. the request is submitted by the client, and enqueued 1. params are serialized and saved into a blobstore 1. an entry is created in the SQL table 1. the key from that table is used to create a polling token 1. some external system processes a request [completely absent form this diff] 1. it notices a new entry in the queue 1. it reads request's params from the blobstore 1. it processes the request 1. it preserves either a success of a failure of the request into the blobstore 1. it updates the SQL table to mention that the request is now ready to be polled 1. the client polls the request 1. queue struct receives a polling token 1. out of that token it constructs DB keys 1. it looks up the request row and checks if it is in the ready state 1. if that is the case, it reads the result_blobstore_key value and fetches serialized result object 1. now it has to turn this serialized result into a poll response: 1. if the result is absent, poll response is a success with an empty payload 1. if the result is present and successful, poll response is a success with the result's successful variant as a payload 1. if the result is present and is a failure, the polling call throws a thrift exception with that failure Note: Why is there yet another .thrift file introduced in this diff? I felt like these types aren't a part of the scs interface, so they don't belong in `source_control.thrift`. On the other hand, they wrap things defined in `source_control.thrift,` so I needed to include it. Reviewed By: StanislavGlebik Differential Revision: D27964822 fbshipit-source-id: fc1a33a799d01c908bbe18a5394eba034b780174
2021-05-10 16:50:29 +03:00
"megarepo_api/async_requests",
mononoke: add commit_rewriting logic to megarepo_api Summary: This is going to be use to rewrite (or transform) commits from source to target. This diff does a few tihngs: 1) adds a MultiMover type and a function that produces a mover given a config. This is similar to Mover type we used for fbsource<-> ovrsource megarepo sync, though this time it can produce a few target paths for a given source path. 2) Moves `rewrite_commit` function from cross_repo_sync to megarepo_api, and make it work with MultiMover. Reviewed By: ikostia Differential Revision: D28259214 fbshipit-source-id: 16ba106dc0c65cb606df10c1a210578621c62367
2021-05-10 21:47:19 +03:00
"megarepo_api/commit_transformation",
megarepo_api: introduce a basic configo client Summary: This is not used on its on, but in subsequent diffs I will add a use-case, by the megarepo configs crate. When built in non-fbcode mode, this crate does not export anything. I chose this approach as opposed to the approach of exporting no-op stubs to force the clients to pay attention and implement gating their side too. This seems reasonable for a rather generic configo client. Reviewed By: StanislavGlebik Differential Revision: D27790753 fbshipit-source-id: d6dcec884ed7aa88abe5796ef0e58be8525893e2
2021-04-19 18:33:00 +03:00
"megarepo_api/configo_client",
megarepo: basic version of async-requests crate Summary: This crate is a foundation for the async requests support in megarepo service. The idea is to be able to store serialized parameters in the blobstore upon request arrival, and to be able to query request results from the blobstore while polling. This diff manipulates the following classes of types: - param types for async methods: self-explanatory - response types: these contain only a resulting value of a completed successful execution - stored result types: these contain a result value of a completed execution. It may either be successful or failed. These types exist for the purpose of preserving execution result in the blobstore. - poll-response types: these contain and option of a response. If the optional value is empty, this means that the request is not yet ready - polling tokens: these are used by the client to ask about the processing status for a submitted request Apart from that, some of these types have both Rust and Thrift counterparts, mainly for the purposes of us being able to implement traits for Rust types. Relationships between these types are encoded in various traits and their associated types. The lifecycle of an async request is as follows therefore: 1. the request is submitted by the client, and enqueued 1. params are serialized and saved into a blobstore 1. an entry is created in the SQL table 1. the key from that table is used to create a polling token 1. some external system processes a request [completely absent form this diff] 1. it notices a new entry in the queue 1. it reads request's params from the blobstore 1. it processes the request 1. it preserves either a success of a failure of the request into the blobstore 1. it updates the SQL table to mention that the request is now ready to be polled 1. the client polls the request 1. queue struct receives a polling token 1. out of that token it constructs DB keys 1. it looks up the request row and checks if it is in the ready state 1. if that is the case, it reads the result_blobstore_key value and fetches serialized result object 1. now it has to turn this serialized result into a poll response: 1. if the result is absent, poll response is a success with an empty payload 1. if the result is present and successful, poll response is a success with the result's successful variant as a payload 1. if the result is present and is a failure, the polling call throws a thrift exception with that failure Note: Why is there yet another .thrift file introduced in this diff? I felt like these types aren't a part of the scs interface, so they don't belong in `source_control.thrift`. On the other hand, they wrap things defined in `source_control.thrift,` so I needed to include it. Reviewed By: StanislavGlebik Differential Revision: D27964822 fbshipit-source-id: fc1a33a799d01c908bbe18a5394eba034b780174
2021-05-10 16:50:29 +03:00
"megarepo_api/if",
mononoke: add mapping for megarepo Summary: Adding mappng to keep track of two things: 1) keep track of the latest source commit that was synced into a given target - this will be used during sync_changeset() method to validate if a parent changeset of a given changeset was already synced 2) which source commit maps to what target commit Reviewed By: ikostia Differential Revision: D28319908 fbshipit-source-id: f776d294d779695e99d644bf5f0a5a331272cc14
2021-05-11 12:53:14 +03:00
"megarepo_api/mapping",
megarepo: introduce write side of MononokeMegarepoConfigs Summary: `MononokeMegarepoConfig` is going to be a single point of access to config storage system - provide both writes and reads. It is also a trait, to allow for unit-test implementations later. This diff introduces a trait, as well as implements the write side of the configerator-based implementor. The read side/oss impl/test impl is left `unimplemented`. Read side and test impl will be implemented in the future. Things I had to consider while implementing this: - I wanted to store each version of `SyncTargetConfig` in an individual `.cconf` in configerator - at the same time, I did not want all of them to live in the same dir, to avoid having dirs with thousands of files in it - dir sharding uses sha1 of the target repo + target bookmark + version name, then separates it into a dir name and a file name, like git does - this means that these `.cconf` files are not "human-addressable" in the configerator repo - to help this, each new config creation also creates an entry in one of the "index" files: human-readable maps from target + version name to a corresponding `.cconf` - using a single index file is also impractical, so these are separated by ascification of the repo_id + bookmark name Note: this design means that there's no automatic way to fetch the list of all targets in use. This can be bypassed by maintaining an extra index layer, whihc will list all the targets. I don't think this is very important atm. Reviewed By: StanislavGlebik Differential Revision: D27795663 fbshipit-source-id: 4d824ee4320c8be5187915b23e9c9d261c198fe1
2021-04-22 12:12:40 +03:00
"megarepo_api/megarepo_config",
megarepo: introduce megarepo_error and make configo_client use it Summary: We want to distinguish user vs system errors in `configo_client` and its users (`mononoke_config` for instance). The reason is to allow `scs_server` distinguish the two types of errors. Normally user errors would only ever be instantiated fairly "shallowly" in the `scs_server` itself, but `configo_client` is a transactional client (by this I mean that it calls user-provided transformation functions on fetched data), so we need to allow for a user error to originate from these updater functions. Reviewed By: StanislavGlebik Differential Revision: D27880928 fbshipit-source-id: e00a5580a339fdabc4b858235da9cd7e9fc7a376
2021-04-21 15:00:58 +03:00
"megarepo_api/megarepo_error",
megarepo: intial impl of async requests table Summary: This is where async requests are logged to be processed, and from where they are polled later. It will acquire more functionality when the actual request processing business logic is implemented. Reviewed By: StanislavGlebik Differential Revision: D28092910 fbshipit-source-id: 00e45229aa2db73fa0ae5a1cf99b8f2d3a162006
2021-05-06 21:32:31 +03:00
"megarepo_api/requests_table",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"mercurial/bundle_replay_data",
"mercurial/bundles",
"mercurial/mutation",
"mercurial/revlog",
"mercurial/types",
"mercurial/types/if",
"mercurial/types/mocks",
"metaconfig/parser",
"metaconfig/types",
"microwave",
"microwave/builder",
"microwave/if",
"mononoke_api",
"mononoke_api_hg",
"mononoke_hg_sync_job",
"mononoke_types",
"mononoke_types/if",
"mononoke_types/mocks",
"mutable_counters",
"newfilenodes",
"observability",
"permission_checker",
"phases",
"pushrebase",
mononoke/pushrebase: split pushrebase crate Summary: Split pushrebase crate into pushrebase hook definition and pushrebase implementation. Before this change it was impossible to store an attribute in BlobRepo that would depend on PushrebaseHook as it would create a circular dependency `pushrebase -> blobrepo -> pushrebase`. Reviewed By: krallin Differential Revision: D27968029 fbshipit-source-id: 030ef1b02216546cd3658de1f417bc52b8dd9843
2021-04-27 13:51:52 +03:00
"pushrebase/pushrebase_hook",
mononoke/pushrebase_hooks: add a hook that saves prepushrebase changeset id Summary: Knowing the prepushrebase changeset id is required for retroactive_review. retroactive_review checks landed commits, but verify_integrity hook runs on a commit before landing. This way the landed commit has no straightforward connection with the original one and retroactive_review can't acknowledge if verify_integrity have seen it. Reviewed By: krallin Differential Revision: D27911317 fbshipit-source-id: f7bb0cfbd54fa6ad2ed27fb9d4d67b9f087879f1
2021-04-27 13:51:52 +03:00
"pushrebase_mutation_mapping",
quiet certain connection errors when shutting down Summary: Similar to D27155123 (https://github.com/facebookexperimental/eden/commit/1a56da1c6f6ea71b5e1c6d8f1768b44990a2bdb0). Reviewed By: krallin Differential Revision: D27805926 fbshipit-source-id: cf58a2e9b2ef92ca536f3b61b63fb42cfb1ec940
2021-04-16 12:25:01 +03:00
"quiet_stream",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"reachabilityindex",
"reachabilityindex/common",
"reachabilityindex/if",
"reachabilityindex/skiplist",
"reachabilityindex/test-helpers",
"regenerate_hg_filenodes",
repo_derived_data: add new repo attribute to encapsulate derived data Summary: Add `repo_derived_data`. This is a new struct that encapsulates derived data configuration and lease operations, and will be used in the facet-based construction of repositories. Reviewed By: ahornby Differential Revision: D27169431 fbshipit-source-id: dee7c032deb93db8934736c111ba7238a6aaf935
2021-03-22 17:23:06 +03:00
"repo_attributes/repo_derived_data",
repo_identity: add new repo attribute to encapsulate identity Summary: Add `repo_identity`. This is a new struct that encapsulates repository identity and will be used in the facet-based construction of repositories. Reviewed By: ahornby Differential Revision: D27169445 fbshipit-source-id: 02a435bba54a633190c6d2e4316e86726aecfdf0
2021-03-22 17:23:06 +03:00
"repo_attributes/repo_identity",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"repo_client",
"repo_client/getbundle_response",
"repo_client/mononoke_repo",
"repo_client/obsolete",
"repo_client/remotefilelog",
"repo_client/repo_read_write_status",
"repo_client/reverse_filler_queue",
"repo_client/scribe_commit_queue",
"repo_client/streaming_clone",
"repo_client/unbundle",
"repo_client/wirepack",
repo_factory: add main repo factory Summary: Add a factory for building development and production repositories. This factory can be re-used to build many repositories, and they will share metadata database factories and blobstores if their configs match. Similarly, the factory will only load redacted blobs once per metadata database config, rather than once per repo. Reviewed By: krallin Differential Revision: D27323369 fbshipit-source-id: 951f7343af97f5e507b76fb822ad2e66f4d8f3bd
2021-04-07 23:59:27 +03:00
"repo_factory",
test_repo_factory: add factory for test repos Summary: Create a factory that can be used to build repositories in tests. The test repo factory will be kept in a separate crate to the production repo factory, so that it can depend on a smaller set of dependencies: just those needed for in-memory test repos. This should eventually help make compilation speeds faster for tests. A notable difference between the test repos produced by this factory and the ones produced by `blobrepo_factory` is that the new repos share the in-memory metadata database. This is closer to what we use in production, and in a couple of places it is relied upon and existing tests must use `dangerous_override` to make it happen. Reviewed By: ahornby Differential Revision: D27169441 fbshipit-source-id: 82541a2ae71746f5e3b1a2a8a19c46bf29dd261c
2021-03-25 17:32:21 +03:00
"repo_factory/test_repo_factory",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"repo_import",
"revset",
"revset/revset-test-helper",
move scm/service into eden/mononoke/scs Reviewed By: ahornby Differential Revision: D28286267 fbshipit-source-id: 349a2d94eca9cf563ee2bb4076e268917aaa4fd6
2021-05-10 15:52:39 +03:00
"scs/if",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"segmented_changelog",
"segmented_changelog/bench/concurrent_idmap",
segmented_changelog: split trait to separate crate Summary: Resolve a circular dependency whereby `BlobRepo` needs to depend on `Arc<dyn SegmentedChangelog>`, but the segmented changelog implementation depends on `BlobRepo`, by moving the trait definition to its own crate. Reviewed By: sfilipco Differential Revision: D27169423 fbshipit-source-id: 5bf7c632607dc8baba40d7a9d65e96e265d58496
2021-03-22 17:23:06 +03:00
"segmented_changelog/types",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"server",
"server/context",
"server/monitoring",
bump region2region QPS Summary: We need to bump SCS counters expressing Mononoke's QPS. They will look something like: `requests:mononoke:oregon:carolina` for requests coming from proxygen in prn and mononoke in frc. CSLB expects regions' full names. We're getting src region from proxygen as a header. Reviewed By: krallin Differential Revision: D27082868 fbshipit-source-id: 12accb8a9df5cf6a80c2c281d2f61ac1e68176d1
2021-03-22 20:10:53 +03:00
"server/qps",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"server/repo_listener",
"server/session_id",
"sshrelay",
mononoke: update to tokio 1.x Summary: NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing. There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs). Here's a detailed list of what is going on: - I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace. - Lots of little Tokio changes: - Stream abstractions moving to `tokio-stream` - `tokio::time::delay_for` became `tokio::time::sleep` - `tokio::sync::watch::Sender::send` became `tokio::sync::watch::Sender::broadcast` - `tokio::sync::Semaphore::acquire` returns a `Result` now. - `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`). - `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case. - Runtime builders have changed quite a bit: - My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`. - Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out. - Tokio 1.x now uses Bytes 1.x: - At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore). - In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine. - In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x. - In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy. - tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to: - How we listen in Mononoke Server & LFS - How we connect in hgcli. - Note: all this stuff has test coverage. - The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this. - Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it. - I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x. Reviewed By: mitrandir77 Differential Revision: D26669620 fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
2021-04-28 17:35:21 +03:00
"statistics_collector",
mononoke: start moving streaming changelog logic to rust Summary: Our current straming changelog updater logic is written in python, and it has a few downsides: 1) It writes directly to manifold, which means it bypasses all the multiplexed blobstore logic... 2) ...more importantly, we can't write to non-manifold blobstores at all. 3) There are no tests for the streaming changelogs This diff moves the logic of initial creation of streaming changelog entry to rust, which should fix the issues mentioned above. I want to highligh that this implementation only works for the initial creation case i.e. when there are no entries in the database. Next diffs will add incremental updates functionality. Reviewed By: krallin Differential Revision: D27008485 fbshipit-source-id: d9583bb1b98e5c4abea11c0a43c42bc673f8ed48
2021-03-13 01:45:03 +03:00
"streaming_clone",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"tests/fixtures",
"tests/utils",
"tests/write_stub_log_entry",
"time_window_counter",
"tunables",
"tunables/tunables-derive",
"unbundle_replay",
mononoke: update to tokio 1.x Summary: NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing. There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs). Here's a detailed list of what is going on: - I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace. - Lots of little Tokio changes: - Stream abstractions moving to `tokio-stream` - `tokio::time::delay_for` became `tokio::time::sleep` - `tokio::sync::watch::Sender::send` became `tokio::sync::watch::Sender::broadcast` - `tokio::sync::Semaphore::acquire` returns a `Result` now. - `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`). - `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case. - Runtime builders have changed quite a bit: - My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`. - Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out. - Tokio 1.x now uses Bytes 1.x: - At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore). - In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine. - In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x. - In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy. - tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to: - How we listen in Mononoke Server & LFS - How we connect in hgcli. - Note: all this stuff has test coverage. - The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this. - Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it. - I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x. Reviewed By: mitrandir77 Differential Revision: D26669620 fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
2021-04-28 17:35:21 +03:00
"upload_globalrevs",
autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2 Summary: For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present. Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is. The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed. Reviewed By: ikostia Differential Revision: D26618363 fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-26 02:09:13 +03:00
"walker",
]
Reference in New Issue Copy Permalink