Summary: Migrating from using the Changesets facet to the CommitGraph facet would otherwise require deblobrepoifying too much stuff.
Differential Revision: D45319163
fbshipit-source-id: f93c091befc5b3419a7d1364107cc717c9d19303
Summary:
The previous diff introduced `RepoShard` as the struct containing the parsed information from the passed in string shard-id. This diff adds relevant test cases to ensure that the logic for parsing the string based ID is accurate.
This diff also modifies the original parsing logic in response to the errors discovered during unit testing.
Reviewed By: mitrandir77
Differential Revision: D45355079
fbshipit-source-id: 78f28fa80bf4a3186ab66b08b29f20d39230094d
Summary:
streamimg support for public pulls
If all heads in the request are public, use the streaming API, so that we are not
limited to any keep alive timeouts on HTTP side, since we can return things more gradually.
This is to be used for hg pulls in not segmented changelog repos, pulling of
expensive branches, etc
Commit Cloud would still use the API where we calculate the whole graph before returning it but this is fine because:
* we need to derive filenodes and calculate phases in parallel
* that function is a little bit more optimised since better batching of hg <=> bonsai conversions, for example.
Reviewed By: mitrandir77
Differential Revision: D45352368
fbshipit-source-id: 7549396d249a50adc163b5de1586d8789776b0df
Summary:
eliminate lifetimes from ancestors_difference_stream method since they
create unnecessary complexity and some consumers require a static stream.
self (arc under the hood) and ctx is cheap to clone there
Reviewed By: markbt
Differential Revision: D45352367
fbshipit-source-id: 6eca62838d1e31ff046ce987c52afced84227a6c
Summary: This diff introduces the `RepoShard` type which will serve as the holder of shard-level state information. Currently, we keep the information encoded in the shard_id (e.g. `fbsource_TO_ovrsource_CHUNK_1-11_SIZE_1000`) and expose it directly to the clients who then have to parse it themselves to get the relevant pieces of information. To avoid this duplication and to centralize the parsing and error handling logic, we will use the `RepoShard` struct. I have added it in `sharding_ext` crate so the type itself can be used independently of the sharding framework (for lightweight dependencies).
Reviewed By: markbt
Differential Revision: D45277195
fbshipit-source-id: 6ae9717fdcf312f0546e7b75d530e8dec8f69d32
Summary:
The `sharding_lib_ext` crate was created to contain the add-on logic that is used by `ShardedProcessManager` but also by other components (e.g. encoding, decoding repo-name for shard-aware routing). I plan to use this crate further for adding chunking support. Before I do that, I decided to clean things up a bit. Particularly,
- Move `sharding_lib_ext` crate under `cmdlib` instead of keeping it under `mononoke` since its a library
- Renamed the crate to `sharding_ext` removing the `lib` since all crates under `cmdlib` are libraries anyway
Reviewed By: markbt
Differential Revision: D45272976
fbshipit-source-id: 90be4f1f4659a03a591a8acbb04fdcbf06234a74
Summary: Similar to direct `gitimport`, this diff adds support in `remote-gitimport` to upload raw git annotated tags when pushed to the remote git server. Note that this only uploads annotated tags since basic tags get directly converted into branches while annotated tags need a corresponding changeset to be represented in the Mononoke data model. This diff doesn't delete existing tags when deletions are pushed to the server and modifications of the raw git tag are prevented by the no-update model in Mononoke blobstores.
Reviewed By: markbt
Differential Revision: D45271703
fbshipit-source-id: f7cdbed7fb665c9d32b08e26d7e9f404175b6eb9
Summary:
Add a new commit resolution step that checks if unknown hashes are from another repo, and if so translates to corresponding hash of current repo.
This behavior is enabled by configuring the other repo name under the megarepo.transparent-lookup config.
Reviewed By: sggutier
Differential Revision: D45092192
fbshipit-source-id: 9434a21d258cdfda153eb81e2d1eb0149f8a063e
Summary:
Respect the optional "from" and/or "to" repo names to convert the commit ids from and/or to different repos. Only applies for repos that are megarepo'd together.
I added a get_repo method to the EdenApiContext object to facilitate the xrepo behavior.
I thought about disallowing requests where both "from" and "to" are different from the "subject" repo in the path, but it seems harmless.
Reviewed By: quark-zju
Differential Revision: D45019173
fbshipit-source-id: 8a25f5506521765296142553963b19f47570e865
Summary: Previously any outer error returned by an EdenApiHandler got turned into a 500. With this change, we no longer allow the generic `Into<anyhow::Error>` propagation. This allows for specialized HttpError propagation that retains response code.
Reviewed By: quark-zju
Differential Revision: D45296090
fbshipit-source-id: bb21518e5f8c77f578eec6d42a2582c474b5d1e9
Summary:
Use the CommitId directly as HashMap key to get rid of some per-type bookkeeping.
This minor change allows my next diff for xrepo translation to be significantly simpler.
Reviewed By: markbt
Differential Revision: D45019175
fbshipit-source-id: eb369b9e2d51345482590bbbbd9a1758b140b309
Summary:
I want to enable EdenApiHandlers to fetch other repo objects, but that requires various other objects not currently available. I don't want to complicate the handler signature with multiple additional args, so introduce a context object that can contain the dependent objects (i.e. RequestContext and ServerContext).
I tucked the path and query extractors into the context since they are barely used by any handlers (in fact, the path extractor is not used at all).
Reviewed By: markbt
Differential Revision: D45019176
fbshipit-source-id: ec4427d2f134277e89a8c48018f29c5f56cdc60d
Summary: Implements a stream that yields all changesets that are simulatenously a descendant of one changeset and a descendant of another, in topological order.
Reviewed By: markbt
Differential Revision: D45156176
fbshipit-source-id: 6c70d40252b5e2402ec1718e6cbe4c35ec8e9c1e
Summary: Make this help text more helpful. Also record that `--bookmark-info` and `--path` are mutually exclusive.
Reviewed By: MichaelCuevas
Differential Revision: D45315761
fbshipit-source-id: 150c80e1e1369c3f0c76d6708e9453fb64e3e7e0
Summary:
The `ContentMetadataV2` provides the seeded blake3 hash for each created file. This information needs to be passed to `buck` which interacts with `EdenAPI` through `EdenFS`. The relevant endpoints in `EdenAPI` are `trees` and `files` which were modified in the previous diff to support the new hash and are used here.
NOTE: This diff was already landed (D44010967) but then had to be reverted (D45156142) since it was causing build issues for EdenFS on ARM64 MacOS. More context [here](https://fb.workplace.com/groups/learningrust/permalink/3405570553032635/)
Reviewed By: quark-zju
Differential Revision: D45226440
fbshipit-source-id: 051dee9898d2441d6a03c10809988c6b99f36ab4
Summary:
prefetch expensive bookmarks separately
this is to avoid timeouts until we have streaming support and ublock rollout
for AOSP repo
also, this provides better ux since we notify users that slow fetch is being executed
I also noticed that the commitcloud.remotebookmarkssync config option wasn't mentioned in the help. So, add it to avoid confusion.
Reviewed By: quark-zju, zzl0
Differential Revision: D44998087
fbshipit-source-id: f24bd959f02380ff0d7ff4f30dbe7e1647403329
Summary:
This data may be useful for analysing the blame history of the file.
* `approx_commit_count` is the total number of commits in the blame history of the file. It is approximate as some merged in commits may be omitted.
* `distinct_range_count` is the number of distinct ranges in the blame, which may indicate the file's complexity.
Reviewed By: RajivTS
Differential Revision: D45218574
fbshipit-source-id: c0c399f525f8c047a0415cf9d6f6e3483c033700
Summary: bunch of unit tests that relied on the specific error formatting weren't caught by CI and are failing. this fixes them by changing their error formatting to use `{:#}`.
Reviewed By: markbt
Differential Revision: D45245281
fbshipit-source-id: 6b0c9c5f3a3ecb594292d2a658ecce457f007e7e
Summary: Directs DifferenceOfUnionsOfAncestorsNodeStream callsites to use commit_graph().ancestors_difference_stream instead conditional on a tunable (enable_new_commit_graph_ancestors_difference_stream) being set to true.
Reviewed By: markbt
Differential Revision: D45042733
fbshipit-source-id: ca508d9b1a2e9efd7b45d1ee834f7587a7d886c3
Summary: Implements a streamable variant of ancestors_difference to be used as a replacement for DifferenceOfUnionsOfAncestorsStream
Reviewed By: Croohand
Differential Revision: D45042732
fbshipit-source-id: aa7377512866038172899545ee9c43a2d6db39f7
Summary: Use ChangesetsCommitGraphCompat when constructing Changesets in TestRepoFactory to enable double writing to the new commit graph
Reviewed By: mitrandir77, singhsrb
Differential Revision: D45227317
fbshipit-source-id: 6ce1809a2e30d930f8edef1efda0f9bef53b2a7c
Summary: Replaced by using TestRepoFactory directly and adding the extra config to the default config.
Reviewed By: RajivTS
Differential Revision: D45227320
fbshipit-source-id: bd93d1a2fd3a10398ca1cfd4b647a32627b83179
Summary: Has the same behaviour as the contruction of warm bookmarks cache in Repo::new_test_common. To keep the same behaviour this required making TestRepoFactory into an async builder and modifying all of its callsites.
Reviewed By: RajivTS
Differential Revision: D45227318
fbshipit-source-id: 51303d48bebc0c4b915a359a7d0942bf485e8db5
Summary: Adds a sync method that awaits the completion of all ongoing updates. Mostly useful for tests to avoid adding artificial sleeps.
Reviewed By: RajivTS
Differential Revision: D45227319
fbshipit-source-id: 9f3a9c7d063ad065b913db521a27624041b2400a
Summary:
When creating a stack, make it possible to request that a derived data type is prepared as part of creating the stack. Since the data for the stack is in local cache, this should be faster than it would otherwise be.
This is a simple initial implementation. It requires that the data is already derived for the parents of the stack, and does not support parallel derivation of the different types, nor types that include interdependencies. In fact, we only support fsnodes and skeleton manifests right now.
Differential Revision: D45216982
fbshipit-source-id: c6ca7b265b1e6219a1193c51d0a984794aa34965
Summary:
Add support for skeleton manifests as a type of derived data that can be prepared.
The current enum for derived data types does not follow best practices and uses zero. Fix this now by adding a new variant for `FSNODE`. Once server support is available, we will remove `FSNODE_NEW` and switch to that value for `FSNODE`.
Differential Revision: D45216983
fbshipit-source-id: b0714ef6f9121139cc8d4602a5a8d1beaac00a3f
Summary: Move the preparation of derived data into a `mononoke_api` method in preparation for re-use.
Differential Revision: D45216984
fbshipit-source-id: ee99379685a202018b4c32b4b56e2930755d723a
Summary: When tests are run in parallel, it can take more than a couple of seconds for the server update to become visible. Increase the wait to up to to 10 seconds.
Reviewed By: RajivTS
Differential Revision: D45229687
fbshipit-source-id: 24bc9607c285a4098c35a4105b8deac7916a7591
Summary: Like in the previous diffs, this diff uploads the raw git tags during repo import through `gitimport`.
Differential Revision: D45118571
fbshipit-source-id: f5a6bb9b5805235e9f66f97ab3a58bd165ef475c
Summary: The previous diff uploaded raw git commit data while the commit gets imported in Mononoke through gitimport or remote-gitimport. This diff ensures that the git `tree` associated with the commit also gets uploaded in Mononoke.
Differential Revision: D45085959
fbshipit-source-id: cf4f4a5ebd0d077972f75fe9c6a8d1dfc5067180
Summary:
With the `upload_object` method in place, this diff utilizes the new uploader method to upload the raw git commit in Mononoke store before creating its corresponding Changeset at Mononoke end.
This diff has the following changes:
- Modified the `get_object` method in `GitReader` to return the raw git content (i.e. header + body bytes of git object) along with the parsed `GitObject`
- Added the `read_raw_commit` method in the `gitimport_objects` library to read the raw commit bytes
- Modified the `ExtractedCommit` type to contain the raw bytes of the commit as well
- Modified all modes of `gitimport` execution to upload the raw git commit before creating the corresponding changeset at Mononoke end
- Made changes in integration test to validate the commit upload behavior
Differential Revision: D45082135
fbshipit-source-id: f339b2851646996be7f43f02458ea5afabb9ca48
Summary:
This diff includes the following changes:
- Introduces the `upload_object` method in the `GitUploader` trait for uploading raw git-objects (e.g. commit, tree, tag) in the mercurial mirror of the repo
- Implements the `upload_object` method in `Direct` and `Remote` implementations of the `GitUploader` trait allowing the objects to uploaded from both `gitimport` and `remote-gitimport`
This diff just introduces this method, follow-up diffs will use it for uploading different commit objects.
Differential Revision: D45046811
fbshipit-source-id: 9b4e6a295ec22e06294e410735e67877e6d0ef10
Summary:
The existing `mononoke_api` git methods were defined under `RepoContext` which relies on a specific type of `Repo` facet container as defined in `mononoke_api` crate. These methods can easily be used through SCS which serves as an external entry point into the `mononoke_api` crate. `remote-gitimport` uses SCS and thus can leverage the necessary endpoints.
However, `gitimport` (or `direct-gitimport`) directly references the logic behind these methods. To ensure that we don't repeat ourselves, there are two alternatives.
- Make `gitimport` use the same `Repo` facet container as the one in `mononoke_api` crate
- Decouple the git methods from `RepoContext` and define them with explicit `facet` dependencies
I chose the later since using the same facet container would have made `gitimport` use a much bulkier `Repo` definition than is necessary. I also made some changes in the `create_annotated_tag` method which are as follows:
- Removed the `start_write()` statement which was essentially an authorization check to validate if writing to the repo is allowed. This check makes sense when validating an external caller (e.g. SCS) but direct references of the library should be allowed
- Removed `CreateChangeset` permission check since we are not actually creating a changeset that corresponds to a commit. Additionally, any auth checks need to be performed at the call-site (e.g. SCS) and not in the API
- Removed the `allowed_no_parents` check since any changeset created for representing a `Tag` object will not have a parent. Infact the concept of parent is invalid for a tag.
- Replaced the returned `MononokeError` into `GitError` to maintain consistency with other methods
- Replaced the instance `save_changeset` method with the struct-level `save_changeset` method. The new method doesn't log to the scribe category thus preventing users tailing it from receiving an invalid commit creation notification. Additionally, the new method doesn't consider any bubble changes but they are anyway irrelevant in the context of these git methods.
Differential Revision: D45046639
fbshipit-source-id: 7ff02cde3fd356eb7a18f5ca12e6c478e23ee315
Summary: In preparation for deleting blame v1, make blame v2 the default.
Differential Revision: D45044338
fbshipit-source-id: c9c71a148b42f4a3301d6c4fd0de5ad7ed321a3e
Summary:
Allow the walker to walk the blame v2 graph.
Since the walker is the main user of blame in the integration tests, we can make v2 the default for the integration tests, too.
Differential Revision: D45044294
fbshipit-source-id: 52c991dd81be78053ffc0a6953caf8a068ef59e3
Summary: This is a benchmark, so put it with the other benchmarks.
Differential Revision: D45114045
fbshipit-source-id: c524c76671d4183f6baa71ca9cd4c3cdfc402dac
Summary:
Original commit changeset: a755f0750ed3
Original Phabricator Diff: D44010967
need to revert to fix our release
https://www.internalfb.com/intern/sandcastle/job/36028797946731051/insights
Reviewed By: xavierd
Differential Revision: D45156142
fbshipit-source-id: 6e5362cf609f69cdbfeb7da30981227f93d42b65
Summary: This tool was used for the original import of globalrevs. We are now the source of truth for globalrevs, so it is no longer needed.
Differential Revision: D45112829
fbshipit-source-id: 8ff106c66de74a79e6cd963ca48bd1875d6ee44c
Summary: This tool was used to rechunk the filestore. The filestore is always chunked now, so this tool is no longer needed.
Differential Revision: D45112830
fbshipit-source-id: cd03dd06f7dc55a6b47b8fc338accf9d300977d2
Summary: This tool was used for the initial population of the XDB blobstore. It has been superseded by the walker, and can now be deleted.
Differential Revision: D45112828
fbshipit-source-id: 134e2c2b761fd27a9b096df4d64404bc14c89c43
Summary: This tool lints toml-format configs which we don't use in production any more, and as such the tool is unused.
Differential Revision: D45112827
fbshipit-source-id: a4f7783144ab782adba5f0a0fb346e8cb542ad97
Summary: This tool was used early on to compute commit stats, but is no longer needed as it has been superseded by other data.
Differential Revision: D45112824
fbshipit-source-id: e88282647e8e3ed85a95ff2a9b15d9212851b8ff
Summary: This tool was introduced to fix up the problem with linknodes pointing at commit cloud commits, however that has long since been resolved, so the tool is no longer needed.
Differential Revision: D45112822
fbshipit-source-id: 350b770e9a3499a8bad8aca29badc3e20a02f168
Summary: General-purpose running of hooks has been superseded by the source control service `run_hooks` method.
Differential Revision: D45112823
fbshipit-source-id: 0093953da8fbf4289260058e7db076c2537bf262
Summary: This was used for an experiment with very large repos which has now concluded. Remove the code so that it doesn't need to be maintained.
Differential Revision: D45112825
fbshipit-source-id: 2591ab469ddf40ab8155c9d3713208f9bc3e5463
Summary: Switch from the deprecated cmdlib to `mononoke_app` as a dependency. The type originates in `mononoke_app` and is simply re-exported from cmdlib, so this is a no-op change.
Reviewed By: RajivTS
Differential Revision: D45092104
fbshipit-source-id: ff8999a68c9707131a5e499b67f6d96492066b28
Summary: This library doesn't seem to be used anymore, so let's delete it.
Reviewed By: RajivTS
Differential Revision: D45092106
fbshipit-source-id: 46d477e6847b876ee2ece74d5e7ca69c787ce32a
Summary: Look at at all the lines of code we could delete!
Reviewed By: YousefSalama
Differential Revision: D45057185
fbshipit-source-id: c454a4bd70527460bf4a727aa3ebe57b3becab4d
Summary:
I didn't want to repeat myself too much so I'm introducing macro that will do
that for me.
Reviewed By: YousefSalama
Differential Revision: D45057186
fbshipit-source-id: 6e13a5a72c6ad463982611ce11a16f937dcd9386
Summary:
The add_recursive would try to access the parents from changesets table
even if they're not there yet (in some places we're racing to inster them to
both tables at the same time).
For that we need to:
* process changesets in reverse topological order
* check edges_map before trying to fetch parrents from legacy changesets table
* skip fetching parents if the parent will be processed in the same function call
Reviewed By: YousefSalama
Differential Revision: D45057189
fbshipit-source-id: 7906c610c7a4f8473647796e0c56962e074a035f
Summary:
`add_recursive` has problem when adding many changesets not present in the
legacy changesets table. This test showcases the problem, the next diffs will
attempt to fix it.
Reviewed By: YousefSalama
Differential Revision: D45057188
fbshipit-source-id: 56297b7e2d47c6ec65023e9782ac0e18012a4319
Summary: Those helped me to localize the source of errors. I might as well commmit them.
Reviewed By: YousefSalama
Differential Revision: D45057187
fbshipit-source-id: 694d9596dec030e65b0d29590068103fb1b70ca9
Summary:
**Context**
D37136735 introduced a new type called `ContentMetadataV2`, to include more information about a file. For example, if a file is binary.
We want to modify the file_info endpoint in SCS, so that users can directly query the metadata for a file.
**This diff**
Implement the modified version of file_info endpoint using the ContentMetadataV2
Reviewed By: markbt
Differential Revision: D43537343
fbshipit-source-id: 98554366a725636d4cd6989fb7e2d9b5fe29d5e4
Summary:
**Context**
D37136735 introduced a new type called `ContentMetadataV2`, to include more information about a file. For example, if a file is binary.
We want to modify the file_info endpoint in SCS, so that users can directly query the metadata for a file.
**This diff**
Modify Thrift data type, and the function signature for the endpoint
Reviewed By: markbt
Differential Revision: D43537344
fbshipit-source-id: 6076703f67923e0fa95676b0db812a75fb6c3650
Summary:
These have broken production as the behaviour was changed so that `--foo` would no longer be valid when it used to be.
The behaviour changed inadvertently when migrating these CLIs to clap 4.
Differential Revision: D45081836
fbshipit-source-id: ffa7a2a2b1c04c860082bb2770eaa2cde1e7519f
Summary: The `ContentMetadataV2` provides the seeded blake3 hash for each created file. This information needs to be passed to `buck` which interacts with `EdenAPI` through `EdenFS`. The relevant endpoints in `EdenAPI` are `trees` and `files` which were modified in the previous diff to support the new hash and are used here.
Reviewed By: muirdm, markbt
Differential Revision: D44010967
fbshipit-source-id: a755f0750ed39719b1f0e6895b7ca81a405fcfed
Summary: The `ContentMetadataV2` provides the seeded blake3 hash for each created file. This information needs to be passed to `buck` which interacts with `EdenAPI` through `EdenFS`. The relevant endpoints in `EdenAPI` are `trees` and `files` which are modified in this diff to support the new hash.
Reviewed By: quark-zju
Differential Revision: D43977390
fbshipit-source-id: bd9a264dc4483574a7a46e7ce4f9109910f18f2d
Summary: The clash between `-h` for both `--help` and `--host` was inadvertent. Switch to `-H` for `--host` to disambiguate. It seems clap4 handles this clash less well than clap3 did.
Reviewed By: clara-9
Differential Revision: D44994925
fbshipit-source-id: c4195500cecd2c796e4765c2fbe6047df2f57593
Summary: The walker was migrated to the new cmdlib, but still depended on the old one for a couple of structs. Remove these dependencies.
Differential Revision: D44886561
fbshipit-source-id: 122d8aa01269b49651b95bd431b52045f91b88bc
Summary: LFS server uses the new command library, so no need to import the old one just for this struct.
Differential Revision: D44886560
fbshipit-source-id: 8b8869c9cf717aa622f374468ab061dff0ef349f
Summary: This is a benchmark so put it with the benchmarks.
Reviewed By: mitrandir77
Differential Revision: D44712484
fbshipit-source-id: e81a383ca1b89b87dc64577bbc42c56740599bf4
Summary: This is a benchmark so put it with the benchmarks.
Reviewed By: mitrandir77
Differential Revision: D44711493
fbshipit-source-id: adb9ce812423a11fae4f6c6a368d755780606944
Summary:
Canonical repo names can/will have slashes in them. We shouldn't convert slashes to directories on disk; just a bad idea.
When choosing the clone destination, use the basename. It could make sense to clone "foo/bar" into "foo_bar" instead of just "bar", but let's maintain the existing/expected behavior for now.
For internal uses, we percent encode the reponame. For example, the hgcache directory for "foo/bar" will be "foo%2Fbar". This is to minimize chance of collision. If we named the cache dir "foo_bar", that would collide unpleasantly with a repo named "foo_bar".
I relaxed reponame inference a bit to allow slashes in URLs like "test:foo/bar". This was purely so I could test with a reponame containing slashes.
A better long term approach is to use the hash of a canonicalized URL, but that is a bigger change.
Reviewed By: quark-zju
Differential Revision: D44857514
fbshipit-source-id: 4fa0b83a0660b81e89fc96b50ecbeb93869dd2b2
Summary: This benchmark was used during development of Mercurial data derivation, however it is very synthetic and we haven't used it in a long time. Remove it to ease the burden of maintenance.
Reviewed By: mitrandir77
Differential Revision: D44707860
fbshipit-source-id: 6dd2ac893b4020a0c77847c4f10848ba20d96e38
Summary: Replace the usage of `benchmark/simulated_repo`'s repository with an ordinary test repository with instrumented blobstore (via `DelayedBlobstore`) and database (via `SqliteCallbacks`). This removes this dependency on this code.
Reviewed By: mitrandir77
Differential Revision: D44707861
fbshipit-source-id: 49d4c2b43aa4dc64b10d94e21d8d242c97e05415
Summary:
We want the ability to observe and change the behaviour for sqlite connections. Add the ability to register callbacks that are called whenever the multithreaded sqlite connection is acquired.
No change in behaviour yet. We will use these in a future diff.
Reviewed By: mitrandir77
Differential Revision: D44707859
fbshipit-source-id: f8156e294bbc51b612a6e38604804b4f815ed919
Summary: The blobrepo test is actually a test of mercurial derivation, so move it there.
Differential Revision: D44676336
fbshipit-source-id: 2f7f002a91171a7081ceacb3c6f10b9c62824bfd
Summary: Move the library down into its own test directory with a `src` directory.
Differential Revision: D44676334
fbshipit-source-id: cd65adaf2b4a445f13b05d700be7a30492cadb97
Summary: Reorganise filenodes derivation so that it matches mercurial derivation.
Differential Revision: D44676333
fbshipit-source-id: bb5e11cb5a8344f3f7e812e1e53360355b1ce36b
Summary: Rename to `mercurial_derivation` and create a proper `src` directory. I plan to make this change consistently across all derived data types, but will start with Mercurial derived data so that I can add a test.
Differential Revision: D44676335
fbshipit-source-id: aae67df807575e532636dd46c4655f135d2d57b0
Summary:
follow up clean up dead code
forgotten in earlier diff
Reviewed By: YousefSalama
Differential Revision: D44931222
fbshipit-source-id: 6ec965ae5c7851693432fadc9b71df6a967259bb
Summary:
deprecate skip list based version
this is part of the effort to deprecate skip lists infra known as **Skip Lists Deprecation**
this endpoint is not called, it is safe to remove it server side first
we don't need to keep both for checking correctness, we passed that phase!
Reviewed By: YousefSalama
Differential Revision: D44916005
fbshipit-source-id: 662cfa20ad44346011126e0ae3aae69deb719dbc
Summary:
log duration for the origin request
so that we will be able configure replay in a more flexible way
Reviewed By: clara-9
Differential Revision: D44910760
fbshipit-source-id: 8d716376702283e24bb4151eb0a68b8bc2e57e6f
Summary:
An invalid bonsai changeset or mutable rename could lead to a copy source that does not exist.
For Mercurial derived data, we already ignore this. Do the same for general diffing.
Reviewed By: mitrandir77
Differential Revision: D44909673
fbshipit-source-id: ac754e0e483e75976afb07078e57bc1f33008a17
Summary:
In the repo factory, we do different things for sql connection creation in tests and prod. In prod we cache `MetadataSqlFactory` instances for each metadata database configuration, and request a new connection from the cached factory each time. In tests, this doesn't work, as requesting a new sqlite connection from the factory is the same as opening the file again, which can be slow for large numbers of connections, so instead we cache connections directly in the repo factory, which isn't really the right place.
We can fix this by making the `MetadataSqlFactory` handle the caching of sqlite connections for us. This makes the repo factory consistent between prod and test. It also means the `MetadataSqlFactory` can hold on to the original connection to the sqlite database and perform the creation queries directly, rather than having to hold it separately in the wrapper around `SqlConnections`, and we can deduplicate the code for opening the database, too.
Reviewed By: mitrandir77
Differential Revision: D44665118
fbshipit-source-id: 1c7d45ac1900d5f2b36285e01159924d0cdb09eb
Summary: These connections are no longer used, so they and their setup code can all be removed.
Differential Revision: D44635160
fbshipit-source-id: 4407a9318086b61942523cf551988e394c6df2e1
Summary: This is already used by a couple of other tests, so it makes sense to move it out of the benchmark.
Differential Revision: D44635161
fbshipit-source-id: 41414b66b380ea1d5635c6131df4fa3f3084fcfc
Summary: The previous diffs added support in SCS and remote-gitimport for including the `git-extra-headers` as part of the mirrored commits in Mononoke. This diff adds similar support for gitimport.
Differential Revision: D44869271
fbshipit-source-id: 5941b0c8729cd972188cc758837c6c1f1b0f72a1
Summary: The current healing logs just mention that a CMv2 blob was healed, but there is no mention of the actual blobstore key that it healed. This diff updates that.
Reviewed By: mitrandir77
Differential Revision: D44164682
fbshipit-source-id: c156a930716499fb1b5c7b248ce11d6bf88f8281
Summary: To maintain backwards compatibility for a while, we will be writing both the new `ContentMetadataV2` and the old `ContentMetadata` blobs. This will be behind a tunable which will be removed once we can verify that the code works as expected with `ContentMetadataV2`
Reviewed By: markbt
Differential Revision: D44221076
fbshipit-source-id: 95c01a96198a99e53ab3ea4f1b04b1ba78f4b298
Summary: D44136799 was the backout diff for backing out all the ContentMetadataV2 changes made so far. This was due to the reason that the changes were landed before the backfilling was completed. This diff is the revert of the backout, to be landed when the backfilling is properly completed.
Reviewed By: YousefSalama
Differential Revision: D44195175
fbshipit-source-id: 2872c699cc7c41cf436a1c7123e01a5190db7ece
Summary: Sometimes it's desirable to set the value for a by-repo tunable for all repos (e.g. to roll out a change everywhere after testing on some of the largest repos, or to quickly roll back) or to specify a default value, and right now the only two ways of doing this is either to specify the value for every single repo separately, or to have another non-by-repo tuanble that overrides it. Instead let's have two special keys: ":override:" and ":default:", so whenever we try to lookup a by-repo tunable we first check if it's set for ":override:" and use its value if so, otherwise we lookup the repo specific value, otherwise we lookup the value for ":default:".
Reviewed By: markbt, RajivTS
Differential Revision: D44750394
fbshipit-source-id: d8fe074ec700f69e52129dfa7de87b7972d11012
Summary:
There is a new `Target::Pipe` that could be used to redirect logs to a file. So updating to the latest version which doesn't add any breaking changes.
Also, updating the deprecated call sites:
```
xbgs -sl 'env_logger::from_env' \
| arc linttool debugfilterpaths --take RUSTFMT \
| xargs sed -i 's/env_logger::from_env/env_logger::Builder::from_env/'
arc f
```
Reviewed By: dtolnay
Differential Revision: D44671457
fbshipit-source-id: 6734f2839731b189564d3b7ceffcd95410fe02e8
Summary:
If experimental.edenapi-blame is configured and the repo has an edenapi client, the annotate command will fetch blame data from the new blame endpoint.
Currently this doesn't support any of the whitespace options since the server only provides whitespace sensitive blame data. We can probably emulate the whitespace insensitive behaviors locally with more work.
Reviewed By: mitrandir77
Differential Revision: D44596604
fbshipit-source-id: 73d7462d0ded2d6941b704dfb0294e52f462879a
Summary:
This handler simply exposes the existing blame derived data over EdenAPI.
The main work is re-indexing the commit index so it can be a simple list instead of a VecMap.
Any error is propagated in-band in the results. I was a little surprised to learn that errors within the HandlerResult stream are ignored (i.e. not propagated to client). It seems like the stream items should have an envelope type to allow automatic Result propagation from server to client.
Reviewed By: quark-zju
Differential Revision: D44596606
fbshipit-source-id: 0ae9d501a35e3e930fd4cbe710fd3e76f493312c
Summary:
The one existing user has MPath in hand (and so does my upcoming code).
Also, defer the error context to avoid copying bytes (again) when there is no error. Note that the ParseError returned by RepoPathBuf::from_utf8 itself contains the bad bytes, but there doesn't seem to be a good way to share those bytes without losing context.
Reviewed By: mitrandir77
Differential Revision: D44596605
fbshipit-source-id: d9b1f635a7e5b90828f934707845a2efcb5c08a5
Summary:
Currently WBC on each bookmark change will try to find all underived cs_ids and will request derivation for them one-by-one. This is redundand because both old path (using derivation manager) and new path (DDS) doing that discovery internally and will derive all dependency.
This diff just removing the part when WBC asks for derivation of each commits and just requests the latest underived commit, however it will still doing discovery of underived commits.
The next step would be to add an API for DDS to answer queries about derived bookmarks locations. This way we could remove discovery part completely.
Reviewed By: RajivTS
Differential Revision: D44536860
fbshipit-source-id: 3867abdf095141b89da44242a35327da02ab05f8
Summary: Now that the dependent logic is in place, this diff parses the git commits and extracts the git headers. It then passes these git headers while creating the corresponding changeset at Mononoke end using SCS's `repo_create_commit` method.
Differential Revision: D44705976
fbshipit-source-id: 464695e273908ef7642514b08158be4affc32ac2
Summary:
Introduce **h2** support into mononoke server.
Remove alpn crate that repeats the standard "select_next_proto" method
Server now offers the following protocols: hgcli, h2, http/1.1 in this order in protocol negotiation.
We will select the first protocol supported by the server which is also supported by the client.
This will add h2 support server side but won't change protocols currently used (because h2 is not offered client side right now, while http/1.1 is explicitly set).
We then later will switch **only** Eden Api Traffic to H2 by enabling the following config client side via slow rollout:
```
[edenapi]
http-version=2
```
We might need to tune some settings on proxygen side before that.
Wireproto traffic will continue to be obliged to set http/1.1 client side explicitly, which is totally fine because it needs web socket upgrade.
### At a high level, HTTP/2:
* is binary, instead of textual
* is fully multiplexed, instead of ordered and blocking
can therefore use one connection for parallelism
* uses header compression to reduce overhead
* allows servers to “push” responses proactively into client caches
Reviewed By: mzr
Differential Revision: D44540283
fbshipit-source-id: 560d80d68d5d364adee122881d12cf61ba9f1695
Summary:
Double writing is enabled for all repos for a long time, so it's safe to delete enable_writing_to_new_commit_graph.
enable_reading_from_new_commit_graph was never used since we needed more granular tunables.
Reviewed By: Croohand
Differential Revision: D44624559
fbshipit-source-id: a06a0f2079bb735e93d72d0d24bbc7ae684600a7
Summary:
The latest "main" contains all the patches that we were carrying on my
fork so far. Bonus point, hyperx is not a depedency anymore so we can also get
rid of this.
Reviewed By: zertosh
Differential Revision: D44682428
fbshipit-source-id: cbb1da18edce7478a5454d1a3ba70bd8767e089c
Summary:
Note: this is a re-land of D44623815, which was was reverted because of a land race with D44626072
In particular this fixes `FlattenUnordered` having a nasty deadlock, but also
some other bugs:
```
# 0.3.28 - 2023-03-30
* Update to syn 2. This raises MSRV of utility crates to 1.56. (#2730, #2733)
* Fix bug in `FlattenUnordered` (#2726, #2728)
# 0.3.27 - 2023-03-11
* Add `TryFlattenUnordered` (#2577, #2590, #2606, #2607)
* Add `AbortHandle::is_aborted` (#2710)
* Add `AbortRegistration::handle` (#2712)
* Make `BiLock` strict-provenance compatible (#2716)
# 0.3.26 - 2023-01-30
* Add `Either::as_pin_mut` and `Either::as_pin_ref` (#2691)
* Add `Shared::ptr_eq` and `Shared::ptr_hash` (#2691)
* Implement `FusedStream` for `Buffered` (#2676)
* Implement `FusedStream` for all streams in `ReadyChunks` (#2693)
* Fix bug in `FuturesOrdered::push_front` (#2664)
* Remove `Fut::Output: Clone` bounds from some `Shared` methods (#2662)
* Remove `T: Debug` bounds from `Debug` implementations of `mpsc` and `oneshot` types (#2666, #2667)
# 0.3.25 - 2022-10-20
* Fix soundness issue in `join!` and `try_join!` macros (#2649)
* Implement `Clone` for `sink::Drain` (#2650)
# 0.3.24 - 2022-08-29
* Fix incorrect termination of `select_with_strategy` streams (#2635)
# 0.3.23 - 2022-08-14
* Work around MSRV increase due to a cargo bug.
```
Reviewed By: zertosh
Differential Revision: D44632588
fbshipit-source-id: bdd87cb02b3aef63a65b1f9b852579225adfedbd
Summary: When calling `fill` methods on a no-op memcache, we do all the work to serialize the data and then throw it away. Add a quick check at the start of the fill method to exit early.
Reviewed By: YousefSalama
Differential Revision: D44503801
fbshipit-source-id: fd490bb5cce70ee1bca931e55e1671faec09e894
Summary: The `Prefetch::Hint` variant indicates that prefetching is possible, but should not be acted upon unless an intermediate caching layer determines that the prefetch should be included. That means when we ask for the target, we shouldn't return a hinted target.
Reviewed By: liubov-dmitrieva
Differential Revision: D44498446
fbshipit-source-id: 56a0fb3b09de985cca9989335375daf972e43276
Summary: If we have missed in cachelib, and been asked to prefetch, then memcache fetches are actually slower than the prefetch would be, even if memcache is warm. Instead, go straight to the database and prefetch from there.
Reviewed By: mitrandir77
Differential Revision: D44498103
fbshipit-source-id: 944c26d553a770e054f38f478cf2bd475afa413f
Summary: When computing ancestors difference, we need all of the commits on the `head` branches, not just the skew binary steps. Prefetch all of them.
Reviewed By: YousefSalama
Differential Revision: D44498105
fbshipit-source-id: acf08930361f03c76685a94fff2c5a41a99089b3
Summary: Make the number of steps taken a parameter of prefetching, so that we can vary it based on prefetch type.
Reviewed By: YousefSalama
Differential Revision: D44498104
fbshipit-source-id: 9d24af68ea620fd11a18d9dbaa822fc9848fdd3c
Summary:
This test used to be detected as flaky in mode/dev-rust-oss and flaky, but not detected as such in mode/opt.
This was due to a race condition between fresh and warm bookmarks, which take a while to update due to WBC.
Using the helper functions to wait for bookmarks to get warm, or directly querying fresh ids, fixes the issue.
Reviewed By: RajivTS
Differential Revision: D44543648
fbshipit-source-id: f70b16609fe995341b84a77e590efdc3d71b4fdf
Summary:
D44297278 was an attempt at fixing this test, but the regex for lines with DEBUG
wasn't actually correct as it's not actually the first pattern in the strings being grepped.
It may be preceded by some spaces.
Yesterday, we saw a contbuild breakage caused by this.
https://www.internalfb.com/intern/sandcastle/job/9007200167370789/insights
Relax the grep step to achieve the intent of the previous diff.
Reviewed By: RajivTS
Differential Revision: D44536438
fbshipit-source-id: a40719f4669b55b2442715bc9350b8dcce7a9747
Summary:
Running tests with network access disabled is much slower. Let's separate those
test targets out so on CI we run with and without network access disabled. For
iterating during developement the tests with network access enabled are
usually enough.
In this diff I:
* add `disable_all_network_access_target` option to `dott_test` rule that adds the extra test target disabling network access
* convert existing opt-out list into options on test target - this required separating some targets as the config is now on per-target not per-test level
Reviewed By: clara-9
Differential Revision: D44474156
fbshipit-source-id: 780fc81c24b22e2dcae98901cd604dc50f16c70c
Summary:
Implement prefetching of commit graph edges via recursive common table expressions.
Prefetching can be performed either by the first parent edge, or by the skip tree skew ancestor edge. We can add more edges later on, however the nature of the `mononoke_queries!` macro means each one needs its own, only slightly customized, version of the SQL query that performs the operation.
Reviewed By: liubov-dmitrieva
Differential Revision: D43455724
fbshipit-source-id: 75f19533f1f4e9d118a098469bbe3f8afb5f7f92
Summary:
The `native_push_only_deny_patterns` contains the list of paths that should be
changed only via pushrebase operations.
I think the `native_push_only_deny_patterns` should be moved into its own hook
but let's start from fixing current behaviour and refactor later.
Reviewed By: liubov-dmitrieva
Differential Revision: D44433734
fbshipit-source-id: 2da93e84763cdfb7ee997cb148d4eac288fba27f
Summary:
During cross-repo pushrebases, we need to log new commits both in large and small repos. But currently, due to the code organization, it is not too hard to forget logging in the small repo. Which is what happened in case of SCS land_stack method.
This land path was not covered, and only large repo commits were logged. For example, this led to most public opsfiles commits being missing from the category (sync to opsfiles_backup uses different code path and exposes the discrepancy):
{F912075962}
(https://fburl.com/scuba/mononoke_new_commit/8q0nbq8i)
Reviewed By: markbt
Differential Revision: D44187074
fbshipit-source-id: bba2bf112fc9326ae5fc177d421ec936a2bb5322
Summary:
The environment is passed into the async runtime via an `Arc`. If the last reference to the environment is dropped from within the runtime, then we end up destroying the runtime from within the runtime, which is an error.
Move the runtime out of the environment, and store it in the `MononokeApp` (new) or on the stack in `main` (old). This means the runtime will live on the stack outside of the async runtime, and so execution of all async code should be complete when they are dropped.
Reviewed By: YousefSalama
Differential Revision: D44299775
fbshipit-source-id: 9a93ba49a8bbf376defdd0b5ddb1d6546b5ff12b
Summary:
This test has been flaky practically since its creation, although below the threshold to be detected as such some of the times.
Looking at its journal, it failed its first stress run and more than half
of the stress runs thereafter (failed 1920 of them, passed 1519 of them).
The assumption that the test makes that we will always take the local path first
is not stable and has never been.
Match the test's expectations to the actual behaviour of mononoke for now.
If the behaviour is problematic, let's tackle it in its own time.
Note that this test was written here D34276252 in reaction to a change to
segmented changelog here D34041066 and the discussion on that diff is relevant
to establish the correctness of the code.
Reviewed By: RajivTS
Differential Revision: D44297278
fbshipit-source-id: 07f42fc11ca0aa637b59b5fc46263b65d624e0a4
Summary: After enabling remote derivation we've lost related to derivation logging in the parent scuba table (like SCS server). This diffs add log tags, which indicates when we requested remoted derivation and when it was finished. In case of the error we already had the logging in place.
Reviewed By: Croohand
Differential Revision: D44297304
fbshipit-source-id: 1e2f6e279e111ce01b2feaf950a11c7b69a314cf
Summary:
In Rust 1.68.0, the `derivable_impls` clippy lint [got smarter](https://github.com/rust-lang/rust-clippy/pull/10161).
Fix all instances of that lint in Mononoke, mute one that's impractical to address.
Reviewed By: markbt
Differential Revision: D44378068
fbshipit-source-id: 473a051a7001d9596db43f47b56cbad5f5db7efe
Summary:
In Rust 1.68.0, the `useless_conversion` clippy lint [got smarter](https://github.com/rust-lang/rust-clippy/pull/10020).
Fix all instances of that lint in Mononoke.
Reviewed By: markbt
Differential Revision: D44378069
fbshipit-source-id: acfd6c77c6a400830c378b4040661323e7232441
Summary: If an EdenAPI request is taking too long, we currently have no signal into this until it either completes or is cancelled. As for wireproto, add logs for long-running requests.
Reviewed By: liubov-dmitrieva
Differential Revision: D44340299
fbshipit-source-id: e24bed96505964c519ea85748207670ded3baacb
Summary:
When streams are cancelled we lose all of their stats, as they only get logged after the stream is polled to completion.
Make the stream stats callback synchronous (we don't actually use its async nature), which means we can call it from a `Drop` implementation, allowing us to log the stats even when the stream is not completed.
To distinguish the two cases, we add a "completed" flag, which is true if the stream was polled to completion.
This also makes `completion_time` optional, as we don't have a value for this if the stream was never polled at all when it got dropped.
`TimedStream` had an implicit fuse-like nature (once completed the stream would never yield anything again), which we rely upon, so make sure this property is maintained.
Reviewed By: liubov-dmitrieva
Differential Revision: D44331959
fbshipit-source-id: 5a7d9875a4d7fd4276c304b6c30ff45c6b990b38
Summary:
As Croohand pointed out on D44183928, if the shared future is cancelled, the `WeakShared` will linger in the collection until the next read of that key. If the read of that key never comes, it will never be cleaned up.
Switch to a scopeguard, that we move into the future. This will execute the code to remove the shared future whether the future succeeds, fails, or is dropped.
Reviewed By: Croohand
Differential Revision: D44342772
fbshipit-source-id: 37fb15d126d8289d61ab33f64f6e1cf9f7a8550d
Summary:
In production we use Logger to log to Scribe (e.g. to "mononoke_new_commit" category). It has WhenceScribeLogged enum field which can be "default", "prod", "sandbox", etc.
Without override, Logger sets "default" value that usually resolves to "prod", **but** it resolves to "sandbox" if code is run from devserver. "sandbox" means we log data to "/sandbox" category instead of usual place.
This is confusing/harmful for us since we only configure to use Logger while working with prod data. For example, if we run SCS locally and land to prod using "land_stack", the commit will go to prod while logging will go to sandbox category. On the other hand, if we run Mononoke Bootstrap to work with a test repo, we don't use Logger at all so the "sandbox" category has no benefit.
We can add more complexity to the Mononoke logging configuration, so we are able to override WhenceScribeLogged in any way we want. But for now, I suggest overriding it in place with "prod" value since we only use Logger in prod. Later we can even simplify the config by removing the old way of logging directly to Scribe and leave just two options: test (log to file) and prod (log via Logger using the "prod" override).
Reviewed By: markbt
Differential Revision: D44118890
fbshipit-source-id: 8d4b9faa6f1f2da1faa96db35dd567102ccde34d
Summary: Basename suffix skeleton manifests are not been warmed by the warm bookmark cache. This means requests that need these are slowed down while they are derived. Add a warmer for them.
Reviewed By: yancouto
Differential Revision: D43906545
fbshipit-source-id: e1f77b5fe2a7a4512bfc49bf8047afef9b441fa9
Summary:
do not sample out slow requests in edenapi replay
we already don't sample them in mononoke test perf
if we also don't sample here, we will have more insight in those and also we will be able to replay
Reviewed By: clara-9
Differential Revision: D44258777
fbshipit-source-id: 597fc475062b23eb24b0c61024eea9046b808420
Summary: Deduplicate reads of large blobs by using a shared future for any in-flight reads. If another read for the same key comes in after then we will read it again, but there should be at most one read in flight for a particular key at a time.
Reviewed By: Croohand
Differential Revision: D44183928
fbshipit-source-id: d6911422a6e16b3dd246dd48cbce4390e6856940
Summary: Make it possible for tickets to become `'static` by using `std::borrow::Cow` for the borrowed values, and converting to the owned variant if needed. We will use this to shared futures that hold a ticket.
Reviewed By: Croohand
Differential Revision: D44262842
fbshipit-source-id: 2c549b2f2f3724c6485b41463b973273211dfe77
Summary:
The `awaited` boolean on the `Ticket` structure and corresponding `Drop` makes it impossible to move out of the fields of `Ticket`. This is inconvenient, especially since it only enables a unit test check.
Move the field to a structure that's internal to the `Ticket`. We still check for failure to await, as dropping the `Ticket` with a `CheckAwait` that hasn't been marked as awaited will still panic during tests when the inner `CheckAwait` gets dropped.
Reviewed By: Croohand
Differential Revision: D44270321
fbshipit-source-id: 29429c40243b31dcc1a5046969b85fb7a15fd0aa
Summary: This log was useful while we were testing the new commit graph, but now it is too noisy and should be removed.
Reviewed By: liubov-dmitrieva
Differential Revision: D44297423
fbshipit-source-id: c9969d4a0b4709ea2065f19508ae6b51a7ee6baf
Summary:
Sometimes the tests run so fast that we're being served stale version of the
bookmarks.
This may cause the test to occasionally take the fast path where it expects to take the slow path, and has caused this test to be flaky since its creation (although not always flaky enough to be marked as such).
Differential Revision: D44298928
fbshipit-source-id: 046ce5bc4e005f3d443794202352bfdc5c80895a
Summary: Youssef accidentally discovered that config hot reloading was not working for SCS server when working on an oncall task. He then went ahead and fixed that issue in D44206206. However, we did not get any notification or alert about this problem and it would have stayed hidden much longer had we not accidentally detected it. This diff adds a liveness metric for the `ConfigUpdater` which will keep publishing every 5 mins as long as the `MononokeConfigs` object is live.
Reviewed By: YousefSalama
Differential Revision: D44293235
fbshipit-source-id: 6f54831c8b1f2154a23c021a7d2d8f01615a7d70
Summary:
A gotham request consists of two parts. The first part is a future, in which the handler resolves the request and produces the response headers. The second part is the body which may optionally be a stream. We added collection of future stats for the first part, but the majority of work may actually be done when computing the stream for the body.
In this diff we add stats collection for that body stream, if it is used. Since this is separate from the future stats we already collected, we prefix the scuba columns with "stream_" to make it clear this is the stream.
Reviewed By: liubov-dmitrieva
Differential Revision: D44260233
fbshipit-source-id: 9820d3f30fb147740a95d8afc2fc1bf1fcc16aab
Summary:
Removes need for patch.
It had already been updated to 3.4 but this just formalizes it.
Reviewed By: jasonwhite
Differential Revision: D44274378
fbshipit-source-id: d17ecbbee06c7df30f689eb859fb31fdf07d8d44
Summary:
bump request_id len
Currently there are conflicts even within the same date https://fburl.com/scuba/mononoke_test_perf/ti2v0ka8
This is confusing. Let's bump.
Differential Revision: D44255577
fbshipit-source-id: 12d4f0ff8a7589da2057d1557fb651583551e329
Summary:
We need to copy redacted commits as well, to maintain consistency, otherwise we can't progress the sync.
This is stopping the www import to fbsource.
Reviewed By: RajivTS
Differential Revision: D44217272
fbshipit-source-id: c6c03df1ebc6310688a134b1bfdcb3e19fc3daa7
Summary:
When MononokeApp is dropped, the ConfigHandle inside of it gets dropped, and the config updater thread stops receiving updates causing config hot reloading to stop working. This was happening in scs, the land service, and the derived data service (not sure if this is also happening in Mononoke server).
This diff adds an explicit drop in the server closure in wait_until_terminated to make sure MononokeApp is not dropped before receiving a termination signal.
Reviewed By: Croohand
Differential Revision: D44206206
fbshipit-source-id: 7322f516c0dcbf4668cba811bdb582f0e5e2f11c
Summary:
remove dead code
left over from earlier clean up
Reviewed By: YousefSalama
Differential Revision: D44131387
fbshipit-source-id: 20601b5e32777d8b2952120e95a24930ef4d2159
Summary:
vmagro deleted his forked and now broke Reindeer importing. Please be
more careful.
https://github.com/graphql-rust/graphql-parser/pull/66 has been merged
but not released.
Reviewed By: dtolnay
Differential Revision: D44198593
fbshipit-source-id: 287f0f3f6860cdf16deb7be80448c2962a9f10a6
Summary: Backing out ContentMetadataV2 stuff since the backfilling for that data has not yet been done and live traffic would see quite laggy performance if we did it directly in Prod Mononoke. So backing out for now until backfilling is fully complete.
Reviewed By: YousefSalama
Differential Revision: D44136799
fbshipit-source-id: ca403f89b8215f2dfd5582dda7499f1618944f2a
Summary: Due to backfilling `ContentMetadataV2` in phases, there are some cases where the metadata exists but the `Blake3` alias doesn't. When copying the alias, if the alias is not present, let's regenerate it.
Reviewed By: markbt
Differential Revision: D44132513
fbshipit-source-id: 4185ece38db350b0069dff485a40166bb0cb1c4d
Summary:
Create changeset uses `FuturesUnordered` and `FuturesOrdered`. These are unbounded, so with very large changesets, we may try to do too much work at once. Switch to `buffered` and `try_for_each_concurrent` so that we can add bounds.
As always, the numbers are somewhat arbitrary: 10 commits at a time, and up to 1000 files at a time.
Reviewed By: mitrandir77
Differential Revision: D44056862
fbshipit-source-id: 6284c207e3768ec0b16371dcd24db53afc0d98d6
Summary: Remove another dependency on the blobrepo crate by migrating to the underlying feature code.
Reviewed By: yancouto
Differential Revision: D44054973
fbshipit-source-id: 612de351beb2d7bffcd98cff629eb787c0184890
Summary: Extract the code that converts create commit parameters to the internal types. These methods will be used for creating stacks as well as for creating single commits.
Reviewed By: yancouto
Differential Revision: D44032677
fbshipit-source-id: 25aaeeb129d4523d51bfc27d375180f09c282555
Summary:
Add `create_changeset_stack` to `RepoContext` to allow creation of a stack of commits in one operation.
In addition to previous checks, we must build a stack of the accumulated changes for each of the earlier commits in the stack so that we can check that the stack is internally consistent. For example, a file can only be deleted once in the stack, or it can be created and then deleted in the same stack.
Reviewed By: mitrandir77
Differential Revision: D44032676
fbshipit-source-id: f08a7df39c52c9de9d6efc728321112ac5ee60f8
Summary:
Let's see if we have long poll problem anywhere which could *greatly*
contribute to any latency spikes.
Reviewed By: markbt
Differential Revision: D44059679
fbshipit-source-id: c8cb37d65e084de1500bee14773ca8229fb07a0c
Summary:
So far we didn't log poll_count and poll_time for edenAPI requests which limits
our introspection into their problems.
This adds logging to both ways of defining handlers: old and new one - this is a bit unfortunate but I don't think I can dig into finishing the migration now.
Reviewed By: liubov-dmitrieva
Differential Revision: D44059681
fbshipit-source-id: c9cdba1fe8e9657bcffa19fd149bb20a1951bfde
Summary:
I want to improve EdenAPI scuba logigng but we had no proper test for logging
since D34346264 changed this test not test anything!
Reviewed By: liubov-dmitrieva
Differential Revision: D44059680
fbshipit-source-id: 44754d433fbc62d66f726cf88757bca496b0dd0d
Summary:
The create commit method performs a number of checks to ensure that the requested commit is valid. This is in addition to the bonsai changeset checks that ensure that a bonsai changeset is internally consistent.
Extend these methods so that they can operate when changes are stacked. The checks use a path tree of stacked changes to validate that file operations earlier in the stack are consistent with file operations of each commit.
Reviewed By: mitrandir77
Differential Revision: D44032675
fbshipit-source-id: e245dd997d57dee1aebe76605c80bff9e3586be9
Summary: Reorganize the create changeset method so all non-async work is done up-front.
Reviewed By: yancouto
Differential Revision: D44032678
fbshipit-source-id: 73cb051a354b47df69a334fe6fb60fb2f1a65e7d
Summary: We will want to pass vectors of these around for creating stacks, so collect them together in a struct.
Reviewed By: yancouto
Differential Revision: D44032680
fbshipit-source-id: f507dea3c9344b90619152c534e53b425e79d2a1
Summary: Implement the `repo_upload_file_content` method, which stores file content ahead of a call to `repo_create_commit` or `repo_create_stack`.
Reviewed By: yancouto
Differential Revision: D44032679
fbshipit-source-id: 5d25202e84da831e27cd5f107027446029e1c8dc
Summary:
Add new source control thrift methods for:
* Uploading file contents ahead of creating new commits. This will allow thrift clients to upload in parallel and before attempting to create commits.
* Creating a stack of commits in one go. Currently this is a sequential operation as the commit hash of the first commit is needed to make it the parent of the second commit. The new method allows the stack to be created in a single operation.
Reviewed By: yancouto
Differential Revision: D44032681
fbshipit-source-id: b8f8cf4cfbce1d13cb4e5e5e0d3e6975ee338273
Summary:
increase insert rate on 20%
As discuss this is a simple change that may or may not help but also safe, so
we proceed.
Reviewed By: mitrandir77
Differential Revision: D44089651
fbshipit-source-id: 96b46c1415d49a762fa929ddba3da3dae741961d
Summary:
These two tests:
```
eden/mononoke/tests/integration/facebook:remote-gitimport - test-remote-gitimport-merge-partially-imported.t (run-tests)
eden/mononoke/tests/integration/facebook:remote-gitimport - test-remote-gitimport.t (run-tests)
```
are marked as flaky.
Running locally, I could see occasionally:
* timeouts to connect to scs
* timeouts to receive answers from scs
from `scsc` and `remote_gitimport`
Increase the connection timeout to 5s and the receive timeout to 30s for all of these.
Reviewed By: markbt
Differential Revision: D44066722
fbshipit-source-id: 37e24c653bb097b5f6be1b892e18bb249fc9553e
Summary: The implementation of `new_memcache_blobstore` and `new_memcache_blobstore_no_lease` is exactly the same, I'm removing `new_memcache_blobstore_no_lease` to simplify the code as leases in Memcache aren't used anymore
Reviewed By: genevievehelsel
Differential Revision: D44029035
fbshipit-source-id: 602ea8a92056f885e70b861f940d505b49d12a2b
Summary:
bump len of the unique id
len 5 gives 1048576 ids that not enough since usage of Eden Api is growing
Reviewed By: clara-9
Differential Revision: D44024880
fbshipit-source-id: 92b525e5f408791fc066974d38f4c7186a985f15
Summary:
log the request in human readable format
if the request is rather small, log its content in human readable format for
debugging purposes
Reviewed By: clara-9
Differential Revision: D44027385
fbshipit-source-id: 9194462be4415aedb26be90b858bbae2803f8dfa
Summary:
add logging of request_id, so that we could match with perf trace
table
this is very nice for debugging, because we can find a slow request in the perf
table and then find it in the replay table as well to learn the request's content if
logged (we don't log uploads requests but most fetch type of requests are
logged) and try to repro the issue using hg debugedenapi command.
Reviewed By: clara-9
Differential Revision: D44024786
fbshipit-source-id: 0884f7479a5bfc068106f14536edc4ea2ab80ee3
Summary:
This diff fixes the flakiness of `test-multiplexed-wal.t`. The root cause is the eventual consistency model we used for writing data into blobstore (3 places).
We now manually flush everything, to eliminate the flakiness.
Reviewed By: yancouto
Differential Revision: D44027781
fbshipit-source-id: ee4fed562908fc7a200e41883540c9c4ef52af22
Summary:
We found 2 failure tests listed here, for commitgraph and commitgraph_v2 respectively:
https://www.internalfb.com/intern/testinfra/testconsole/testrun/6755399583562643/
This diff is going to fix the itegration tests to include the new debug messages introduced by D42453084
Differential Revision: D44024623
fbshipit-source-id: ffe9c9e71018f5bb3a4de6096e33f1b8c710d358
Summary:
Before this change,
```
arc rust-clippy //eden/mononoke/blobstore:multiplexedblob_wal-unittest
```
failed ungraciously with:
```
File changed: fbcode//eden/mononoke/blobstore/multiplexedblob_wal/src/multiplex.rs
File changed: fbcode//eden/fs/store/git/GitBackingStore.cpp
Action failed: fbcode//eden/mononoke/blobstore:multiplexedblob_wal-unittest (rustc bin-pic-shared-metadata/multiplexedblob_wal_unittest-metadata bin,pic,metadata [clippy])
Remote command returned non-zero exit code 101
Reproduce locally: `frecli cas download-action e2c5c9e58703ce3c526ab116fd1cce8c0302c307:94`
stdout:
stderr:
thread 'rustc' panicked at 'called `Option::unwrap()` on a `None` value', src/tools/clippy/clippy_lints/src/ptr.rs:234:49
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
error: internal compiler error: unexpected panic
note: the compiler unexpectedly panicked. this is a bug.
note: we would appreciate a bug report: https://github.com/rust-lang/rust-clippy/issues/new
note: Clippy version: clippy 0.1.67
query stack during panic:
#0 [analysis] running analysis passes on this crate
end of query stack
Buck UI: https://www.internalfb.com/buck2/8802c9ab-ae8f-4b18-83a4-2b4992bc8f5b
RE: reSessionID-26b8008f-93cc-402a-bdde-fff3596ed795 Up: 1.0 KiB Down: 0 B
Jobs completed: 5. Time elapsed: 50.7s. Cache hits: 0%. Commands: 1 (cached: 0, remote: 1, local: 0)
BXL SUCCEEDED
```
I found out the issue was happening in the `cloned` macro with the `key` by progressively isolating the code that was causing issues.
This code is equivalent, but happens not to hit this clippy bug.
Reviewed By: yancouto
Differential Revision: D44027145
fbshipit-source-id: 678e8d870754162665aa7d50c5839e116b5af186
Summary:
Until now, we could use `BookmarkCategory` to create bookmarks referring to branches, tags or notes; but we didn't have a mechanism to create a tag annotation, which makes the difference between a lightweight tag and an annotated tag (notes should also always be annotated).
With this change, we have all the mechanics to create, delete or move any bookmark that can be imported from git.
Differential Revision: D43775948
fbshipit-source-id: 5241f94f190fcfa8c64747fd862cf70a3ed8909d
Summary:
This fixes the remaining clippy errors.
The only thing l haven't fixed is that clippy crashes when trying to parse `eden/mononoke/blobstore:multiplexedblob_wal`. This doesn't generate warnings and the command still succeeds, but it's annoying. I haven't been able to track down why, so I'm leaving it be.
Reviewed By: YousefSalama
Differential Revision: D43980977
fbshipit-source-id: 28ddcce5894903d6e0f980cd04b586dc1afc00c6
Summary: This fixes a lot of clippy lints. In particular, I didn't yet fix the mutable_key ones because we might want to disable that rule instead.
Differential Revision: D43979960
fbshipit-source-id: b5d0da42480f9a30bba38d31dca60a2fb810a0b3
Summary:
Removes dependency to futures_old. It still needs some compats because it still uses old futures from other libraries.
This makes us clone a lot less stuff and makes the code clearer.
Differential Revision: D43979027
fbshipit-source-id: d2e065df7626aa9e3cf4b3606caf999a21c56c20
Summary: While moving tags is not a common operation, it can happen so we should support it.
Differential Revision: D43909899
fbshipit-source-id: 19e9c4bb65aa496c91736bd2aff36ef8c7318e98
Summary: Prior to this change, we could only delete bookmarks of category branch, which is the default value.
Differential Revision: D43909898
fbshipit-source-id: 4f7b2f9df3167397d3a69542382961858444d8f4
Summary:
This allows us to refer explicitely to a specific bookmark whether it's a tag, a note or a branch.
Prior to this change, we could only refer to bookmarks of type `Branch` (which is the default value and the case for any bookmark that was not imported from a git repository).
Differential Revision: D43909897
fbshipit-source-id: f767a3bcf236652aae48b63a4724355187282a46
Summary:
This allows user of this API to create bookmarks that can represent git tags or git notes in addition to the default bookmarks that map to git branches.
Note that annotated tags are not covered in this diff. What differentiates a lightweight tag from an annotated tag is the changeset it points at. Creating the changeset is independent from creating the bookmark.
Differential Revision: D43775947
fbshipit-source-id: 6865c091a29917528fccfdca6db1af26a6f1fcb4
Summary:
This was intentionally omitted when the `BookmarkCategory` was introduced in D43444397.
The reasoning behind that omission was that this is used during `move_bookmark` and tags are not expected to move.
While tag moves are more rare than branch moves, they can still happen:
git provides a mechanism for moving them (`git tag -f`) and it can also occur indirectly if a tag is removed and re-added somewhere else in git, followed by a push, where live git sync will need to move the bookmark.
Don't special-case the category in the bookmark_update_log: tags and notes should behave like any other bookmarks.
Note: D43908929 is the configerator counterpart to this change and should land and rollout first.
Differential Revision: D43908709
fbshipit-source-id: 4423dc6dc9a5e7f2ffe9d4e8f62741f7ee38a831
