Summary:
Prior to {D54500949}, the generated `:foo-rust` target used to depend on `:foo-rust-services` and needed to refer to its contents.
After that diff, there is no longer any generated code that depends on the `-services` crate, so we no longer need to pass the Thrift compiler a crate name by which to refer to the contents of the `-services` crate. This diff cleans that up.
Reviewed By: shayne-fletcher
Differential Revision: D56726365
fbshipit-source-id: d0e1cc1d302a6e848699c0e2e78692c9c6f9af5f
Summary: We want to create a new version of `GitDeltaManifest` that uses git's diff algorithm to generate deltas, and we need to be able to use it alongside the current version. In order to do this, this diff creates traits that encapsulate what is needed out of `GitDeltaManifest` and makes the packfile generation code generic over them. This diff also implements a `fetch_git_delta_manifest` method to get whatever `GitDeltaManifest` version is configured in derived data types repo config.
Reviewed By: markbt
Differential Revision: D56578016
fbshipit-source-id: a6bf23836ebe81c421e99cb44b0b652aa74c5394
Summary: We want to be able to run `RepoMetadataLogger` in incremental mode where we can process only subset of the repo instead of processing the whole repo in every iteration. Since the incremental operation will go over a much smaller subset of file and directories, we want to run the job more frequently than a day but at the same time we don't want it running every 10-20 minutes. This diff allows us to configure how long we want to wait before every iteration for a repo.
Differential Revision: D56421891
fbshipit-source-id: 7ba9afec35f49ed876488cf5f3f83527964b6b2e
Summary: `RepoMetadataLogger` is going to be introduced as a new TW job that will be deep sharded. This diff adds the necessary sharding configuration parsing to make that happen. The corresponding configerator diff is D56354567
Differential Revision: D56354845
fbshipit-source-id: 1955215da1e19e51fe0bf7ee87fc130f78a67c69
Summary:
According to the analysis in D54709575, about 60% of the compile time of Rust Thrift client crates is the mocks. Mocks are rarely used compared to the rest of the contents of the clients crate, and almost always just in tests. Moving mocks out to a separate crate therefore makes clients build 2.5× faster.
This diff keeps the mock implementations in the `-clients` crate for now, but adds a new `-mocks` crate matching the API that we'll need to migrate to. According to `fbgs -frs$ '>::mock()' -l` there are 769 call sites that need to be updated, which is small compared to the Thrift codemods from earlier in the year. Once the codemod is done, we get to cash in on the build speedup in {D56451418}.
The dependency graph is:
```lang=mermaid
flowchart BT;
foo-clients --> foo["foo (#quot;types#quot;)"];
foo-services --> foo;
foo-mocks --> foo-clients;
```
Previously, mocks were constructed using an associated function on the client trait object: `<dyn FooClient>::mock()`. In the new API, the trait `FooClient` is defined in `foo-clients` whereas the mock client for the same Thrift service is defined in `foo-mocks`, so a different approach is needed because the mocks crate cannot add an inherent method onto a trait object defined in a different crate. We need to pick one of the following APIs to use instead:
1. An extension trait: `use foo_mocks::DynClient as _; <dyn FooService>::mock()`
2. `<dyn FooService as foo_mocks::DynClient>::mock()`
3. `foo_mocks:🆕:<dyn FooService>()`
4. `foo_mocks::FooService::new()`
5. `foo_mocks::FooServiceMock::new()` (append `Mock` suffix to service name)
Number 3 or 5 seems the most promising to me. 4 doesn't seem pleasant because you'd almost always want to have the client `trait FooService` imported in order to be able to make trait method calls on your mock, which gives rise to a name collision in the type namespace.
This diff implements option 3 but I am interested in feedback.
Reviewed By: Imxset21, shayne-fletcher
Differential Revision: D56451421
fbshipit-source-id: c23fd9a28f8dd5bd75306a913819bc0ed66fb4e5
Summary:
This diff is the fbsource counterpart of D56242448. For more context, refer that diff.
NOTE: The config change made here is backwards compatible. The new value is optional so clients without this value can still continue to function.
Reviewed By: singhsrb
Differential Revision: D56243061
fbshipit-source-id: 41997d4354048fd8f695c00fc84f290cdd065cc9
Summary:
We switched to justknobs in past hack-a-thons.
The configuration-side of this were already removed a while back: D54004256
This is now fully unused.
Reviewed By: mitrandir77
Differential Revision: D56241092
fbshipit-source-id: f0d7c677d71f2cefcd287e393f54d1fa67c9690d
Summary: After landing the configurator changes and creating a "test_db" for integration tests to use, this allows Commit Cloud to use xdb.commit_cloud (which will be added in the next diff
Reviewed By: mitrandir77
Differential Revision: D55679989
fbshipit-source-id: ddbef0c2c6aa6431ac9529bfe1b708303597da39
Summary:
Context: repo_metadata_logger (https://www.internalfb.com/code/fbsource/[47c9910c86bc]/fbcode/scm/repo_metadata_logger/main.py) is a python script that requests metadata about all files and directories in repos from SCS and then logs it. As this script powers the dim_scm_repo_metadata hive table which is used for tracking developer productivity, we would like to make it faster and more reliable, and a good step towards this is to turn it into a Mononoke app.
Currently which bookmarks to log metadata for is hardcoded for each repo in the python script. Let's make it a repo config instead
Reviewed By: RajivTS
Differential Revision: D55426907
fbshipit-source-id: 3b7f53a3f1f6ce7634377bfb3af82eb538e8b6cf
Summary:
When the request supplies a revision, and the `nearest_revision`
feature flag is enabled, and `exact_revision = False`, Glass will
choose DBs that are nearest to the supplied revision using a
site-specific `getGeneration` API to map revisions to a linear scale.
Reviewed By: pepeiborra
Differential Revision: D55519942
fbshipit-source-id: 56bf0e52a137b238fd839ab92ff082305f830cbf
Summary: They support no-std with a bunch of features turned off.
Reviewed By: dtolnay
Differential Revision: D55333115
fbshipit-source-id: e66ee699cb3bc3e74f6c7b4d9a81e488e9768e40
Summary: The fbcode counterpart of D55313321. This diff needs to land before the configerator one due to config hot reloading
Reviewed By: clara-9
Differential Revision: D55314515
fbshipit-source-id: 78b3398b60cdbc00a8ed89cc966143533778aa11
Summary: This diff adds the rust parsing code for the configerator changes in `RepoConfig` for `GitConcurrencyParams`. These params are used for controlling the concurrency during git protocol packfile stream generation.
Reviewed By: clara-9
Differential Revision: D55137452
fbshipit-source-id: 3d7b7f0d5dbe1d234ca68910b58d425bed74a06a
Summary:
Just syncs the changes from D54812808.
I'll only ship it once D54812808 is accepted and pushed.
Reviewed By: mitrandir77
Differential Revision: D54815705
fbshipit-source-id: fe148ebbbe15362e1ea95d26e96e9fff3ce75ed8
Summary:
This suffix was necessary prior to D54710352 because there used to be 2 distinct Thrift-generated packages and they needed to have different Cargo package names (`foo`, `foo_types`) in order to appear in the same Cargo workspace.
kuecks noticed failures like the following in `oss-eden-linux-getdeps-nosystem` since the above diff:
```lang=text,counterexample
error: failed to resolve patches for `https://github.com/facebook/fb303.git`
Caused by:
patch for `fb303_core` in `https://github.com/facebook/fb303.git` failed to resolve
Caused by:
The patch location `/data/sandcastle/temp/skycastle/fbcode_builder_getdeps/shipit/fb303/fb303/thrift` does not appear to contain any packages matching the name `fb303_core`.
```
I suspect this is related to:
https://www.internalfb.com/code/fbsource/[07765f7b56438645403907f8f314b21961d1e522]/fbcode/opensource/fbcode_builder/manifests/fb303?lines=1-2%2C13-14
using the crate's name (`fb303_core`) instead of the Cargo package name (`fb303_core_types`).
The `_types` suffix is no longer needed. This diff removes them and hopefully resolves that failure.
Reviewed By: zertosh
Differential Revision: D54859971
fbshipit-source-id: 538e64122bc7b57158d47ab4c2685b7bbbe2e3a1
Summary:
Refer to the summary of D54709575. This diff implements Step 1: `:foo-rust-types` becomes an alias for `:foo-rust`, as opposed to `:foo-rust` being a separate crate containing re-exports from `:foo-rust-types`.
With this, a lot of targets now have a significantly smaller dependency graph. For example: `buck2 cquery 'kind("rust_library", deps(fbcode//zippydb/tools/zu_rs:))' | wc -l`
- **Before:** 7235
- **After:** 5608
Reviewed By: zertosh
Differential Revision: D54710352
fbshipit-source-id: d7dbcdf1fa80d3dc2fd50cc1a75d9ea98146ed3e
Summary:
And with that, a lot of targets now have a significantly smaller dependency graph. For example: `buck2 cquery 'kind("rust_library", deps(fbcode//zippydb/tools/zu_rs:))' | wc -l`
- **Before:** 9360
- **After:** 7148
After this diff, `:foo-rust` is almost completely vestigial. The summary of D54468159 describes how it will be eliminated next, leaving even less dependencies.
Context: https://fb.workplace.com/groups/rust.language/posts/25780668961555042
Reviewed By: zertosh
Differential Revision: D54500949
fbshipit-source-id: a7e4ea36f652d1820809945eab1c7fa9b15929ab
Summary:
This is the start of the codemod for https://fb.workplace.com/groups/rust.language/posts/25780668961555042.
It follows the same approach as {D53374511} by adding a comment above the re-exports to identify the correct Buck target to use instead. This way I can codemod all uses by locally changing the comment into a real `#[deprecated = "..."]` attribute, compiling all Rust targets with `-Awarnings -Wdeprecated` in rustflags, parsing the deprecation warnings, and programmatically applying the correct fix.
Note that the deprecation needs to be on a `pub mod` rather than a `pub use` because deprecation on re-exports is not supported by Rust.
Reviewed By: zertosh
Differential Revision: D54332814
fbshipit-source-id: 16358f211032a7b1a701c08d909078b86b53d65f
Summary:
the intent is to test conversion to rust structured annotations for a subset of fbcode (have chosen 'fbcode/configerator/structs/scm/mononoke')
this diff relies on D53674463 (should land in conjunction with this one) where .thrift files are updated so that rust unstructured annotations are replaced by their structured equivalents.
Reviewed By: dtolnay
Differential Revision: D53676407
fbshipit-source-id: a3f71bd94354f2cf4f6400cf773c71f08b73966e
Summary: As title. Will update `SmallRepoCommitSyncConfig` and `GitSubmodulesChangesAction` structs in the next diff, along with the conversion layer.
Differential Revision: D53517922
fbshipit-source-id: 1575362448dbd6eb9e2f5770726e4e8a2453825e
Summary:
Changes resulting from {D53374511}. I think this was supposed to be in that diff.
It doesn't necessarily make sense to have these fbcode target names in Autocargo-generated code that gets shipped to GitHub. I can look into getting autocargo to omit these.
Reviewed By: zertosh, shayne-fletcher
Differential Revision: D53379521
fbshipit-source-id: 6377e6181b0b1e0a86f6144e208773959fda0ac0
Summary:
I came across this failure in `oss-mononoke-linux-getdeps` likely caused by {D53302058}:
> error: document output filename collision
> The lib `bonsai_globalrev_mapping_thrift` in package `bonsai_globalrev_mapping_thrift_types v0.1.0 (/data/sandcastle/temp/skycastle/fbcode_builder_getdeps/build/mononoke/source/eden/mononoke/bonsai_globalrev_mapping/if/types)` has the same name as the lib `bonsai_globalrev_mapping_thrift` in package `bonsai_globalrev_mapping_thrift v0.1.0 (/data/sandcastle/temp/skycastle/fbcode_builder_getdeps/build/mononoke/source/eden/mononoke/bonsai_globalrev_mapping/if)`.
> Only one may be documented at once since they output to the same path.
> Consider documenting only one, renaming one, or marking one with `doc = false` in Cargo.toml.
We don't need Cargo-generated docs of Thrift targets.
Reviewed By: zertosh
Differential Revision: D53339579
fbshipit-source-id: 3dc0d404777ca436e0f6de7ae7491925d709e331
Summary:
The Rust Thrift implementation presently compiles to 4 crates:
```lang=mermaid
flowchart TD;
foo --> foo-clients;
foo --> foo-types;
foo --> foo-services;
foo-clients --> foo-types;
foo-services --> foo-types;
```
Sadly, the one with the best name is the most useless. That top-level foo crate just contains the following:
```
pub use ::foo_types as types;
pub use ::foo_clients as client;
pub use ::foo_services as server;
pub use ::foo_types::consts;
pub use ::foo_types::errors;
pub use ::foo_types::services;
pub use ::foo_clients::mock;
pub use self::consts::*;
pub use self::errors::*;
pub use self::types::*;
```
Furthermore, it is extremely uncommon for anything to want to be both a client and a server of the *same* Thrift interface. It arises occasionally for proxies, but otherwise you are either implementing the server for a particular service, or accessing it as a client, not both.
I am interested in eliminating the useless crate, and renaming the types subcrate to the nice name.
```lang=mermaid
flowchart TD;
foo-clients --> foo;
foo-services --> foo;
```
### Implementation plan
1. This diff
2. Migrate everything that uses `:foo-rust` and needs a client to `:foo-rust-clients`, and everything that needs a server to `:foo-rust-services` -- after this point, everything still depending on `:foo-rust` only uses types
3. In the macros, atomically delete `:foo-rust` and rename `:foo-rust-types` into its place (this will be a small diff)
Reviewed By: zertosh, shayne-fletcher
Differential Revision: D53302058
fbshipit-source-id: 8d28c90f3a13e71e0cedd3ce3c1cb6fe40cffe27
Summary:
# Before:
If a Thrift library A depends on Thrift library B, all of the dependencies on B are bottlenecked through B's main library. That means building A's clients requires building B's servers, and building A's servers requires building B's clients, neither of which should be necessary.
```lang=mermaid
flowchart TD;
A --> A-clients;
A --> A-types;
A --> A-services;
A-clients --> A-types;
A-services --> A-types;
B --> B-clients;
B --> B-types;
B --> B-services;
B-clients --> B-types;
B-services --> B-types;
A --> B;
A-clients --> B;
A-services --> B;
A-types --> B;
```
# After:
A tidy lattice. Projects that implement a Thrift service can build only the servers for A and B, without also building all their clients. Projects that need Thrift clients for various services can build just the clients without also building a bunch of unused infrastructure for writing a server implementation.
```lang=mermaid
flowchart TD;
A --> A-clients;
A --> A-types;
A --> A-services;
A-clients --> A-types;
A-services --> A-types;
B --> B-clients;
B --> B-types;
B --> B-services;
B-clients --> B-types;
B-services --> B-types;
A --> B;
A-clients --> B-clients;
A-services --> B-services;
A-types --> B-types;
```
Reviewed By: shayne-fletcher
Differential Revision: D53246158
fbshipit-source-id: 37cc5cb111c39c567c69e8fb2eaf23fd940082b3
Summary:
This diff adds oncall to Buck TARGETS files in fbcode repository based on configerator acl information.
This diff was generated by this command:
./fbcode/ownership/coverage/submit_diff_with_file_oncall_mapping.sh /tmp/configerator_sorted_missing_oncalls_4_oncall_finder_fixed.csv
see more context here: https://fb.workplace.com/groups/fbcode/posts/6551897871513663 and https://fb.workplace.com/groups/fbcode.devx/permalink/3149549092006627/
drop_conflicts
Reviewed By: bayarmunkh
Differential Revision: D52751199
fbshipit-source-id: 9ae64a554edbc3639a3c0d8c0bc443c2277858d1
Summary: I've changed the config in configerator in D52786201, let's sync the changes to fbsource.
Reviewed By: liubov-dmitrieva
Differential Revision: D52786217
fbshipit-source-id: 9e8cdd96915aee6624403dd1a288443b9b946426
Summary:
Release notes: https://blog.rust-lang.org/2023/10/05/Rust-1.73.0.html
This release is coupled with an update of the `anyhow` and `thiserror` crates because the unstable standard library API for backtraces has changed.
Fbcode changes:
- `feature(default_free_fn)` deleted (D50300881)
- `noop_method_call` lint becomes warn-by-default (D50486032, D50516201)
- stronger `invalid_reference_casting` detection (D50488164)
- `feature(unix_chown)` has been stabilized
- `feature(provide_any)` and `std::provider` deleted
- `clippy::unwrap_or_else_default` renamed to `clippy::unwrap_or_default`
- type inference ambiguities (D51780425)
- `nu-command` build error (D51779062)
Reviewed By: AndreasBackx, shayne-fletcher
Differential Revision: D50294321
fbshipit-source-id: 0fac87f6ba072ad029f9ce41ce94ed813e855b20
Summary: In this diff I sync the Thrift file from D51347853 and parse it into the actual struct that will create the s3 client to write to the blobstore
Differential Revision: D51350175
fbshipit-source-id: 0979582727499fdc13e97909949cbd7be3b4b642
Summary: This is simply a sync of the changes made in to D51067227
Reviewed By: mitrandir77
Differential Revision: D51260726
fbshipit-source-id: 9be63518b2cb05ee9e850206b17cc9a87e9167b3
Summary:
## This stack
See summary of D50967891.
## This diff
As title.
See comments in D51107527 where the field is implemented in the Rust data types along with the necessary conversion.
Reviewed By: markbt
Differential Revision: D51202229
fbshipit-source-id: 1ccfc119ec6044596c0b096db9d0a956d6de7c22
Summary:
Currently, the repo service throws if any of the repo service root folders don't match the provided ServiceID. This logic is incorrect; we should instead check if _any_ of the root folders _match_, and return the first one that does.
Context: https://fb.workplace.com/groups/826961712023862/posts/1169514477768582
Differential Revision: D50755264
fbshipit-source-id: 71e1f5e63a1c35268ca57d09a5064607eb2fa5e4
The internal and external repositories are out of sync. This Pull Request attempts to brings them back in sync by patching the GitHub repository. Please carefully review this patch. You must disable ShipIt for your project in order to merge this pull request. DO NOT IMPORT this pull request. Instead, merge it directly on GitHub using the MERGE BUTTON. Re-enable ShipIt after merging.
Summary:
anyhow-1.0.73 [uses][1] the new `Error::provide` API. This API is
available starting in Rust 1.73. This means that if you want backtraces,
you need one:
- anyhow-1.0.72 & Rust 1.72
- anyhow-1.0.73 & Rust 1.73
Since we're still on 1.72, pin the version to avoid accidentally losing
backtraces.
There are no real changes between anyhow-1.0.71 and anyhow-1.0.72. But
update to anyhow-1.0.72 so that we're right up until the point where we
get backtrace support.
[1]: https://github.com/dtolnay/anyhow/pull/319
Reviewed By: shayne-fletcher
Differential Revision: D49970958
fbshipit-source-id: 1193611e6d16bc840e63b689e932ea3d33562152
Summary: Sync definitions from configerator and add local definitions for new hook config.
Reviewed By: RajivTS
Differential Revision: D49094388
fbshipit-source-id: aeba2d69d1be6ebd2f52fe61fdbd4ef5415c9681
