Summary: We need that for optimizing our storage.
Reviewed By: RajivTS
Differential Revision: D46362446
fbshipit-source-id: d81973cd05cffdc78ad84da867fa4791efea877c
Summary: As in title. Once `BlobRepo` has been removed from this tool, I will also migrate it to the new `Mononoke Admin`
Reviewed By: mitrandir77
Differential Revision: D46444744
fbshipit-source-id: 059623d32d608f852254f16f811bcaf4e6bf3b4a
Summary: This includes PR 7 so drops the patch.
Reviewed By: zertosh
Differential Revision: D46451758
fbshipit-source-id: 5df8ccbfa610eab470744e28b5bffe411a983996
Summary:
saintelmosfire
Will only land this after D45356044 is landed and tested.
Differential Revision: D45356043
fbshipit-source-id: 068aa45e1edd954b8ba5dc3355f9392ab06419b5
Summary: upgrade parking_lot from version 0.11.2 to version 0.12.1
Reviewed By: zertosh, diliop, dtolnay
Differential Revision: D46398501
fbshipit-source-id: ffae0ea188abad5253f1524216dd18ce9a53a74c
Summary:
In D41769964, we decided not to stop `gitimport` simply because `create-bookmark` fails cause that could be due to the fact that the bookmark being created already exists. If that is indeed the case, we try to resolve the bookmark and move it to the new value. However, there could be other unrelated reasons for `create_bookmark` to fail in which case we would still try to resolve the bookmark and end up in unexpected situations (e.g. Luibov trying to import `paws` repo faced this problem). This diff does two things:
- Performs an early check to determine if the bookmark exists by resolving it. If it does exist, it executes the move bookmark branch of code
- If the bookmark doesn't exist, then it creates it.
If there are any errors during bookmark creation or movement, they get logged for the user.
Reviewed By: mitrandir77
Differential Revision: D46071703
fbshipit-source-id: 446b0bc720d13b9e9314feccdb147b2bab0b7cd9
Summary: Added `fill_from_function` which makes it much more useful
Reviewed By: fanzeyi
Differential Revision: D46342378
fbshipit-source-id: 9bc8c01dcb10806a03f1eab628f9763c4e5edc81
Summary:
add new ods logging for bookmark fetching from phases crate
This would help with investigation sevs like S344849, also having this info we
can setup alerts.
Reviewed By: YousefSalama
Differential Revision: D46314640
fbshipit-source-id: 66e72da19ad4eb528c78d0783c297746f57cd4f1
Summary:
This offers a new interface for Shard Manager managed repos to be able to load redacted content.
The new API entry point: `open_managed_repos_unredacted` is analogous to `open_repo_unredacted`, but supports a Shard Manager based workflow.
Reviewed By: RajivTS
Differential Revision: D46280863
fbshipit-source-id: e2e4aebcb08d01f3be97d11d6fe8da2822ba6f29
Summary:
This diff modifies the behavior of `gitimport` and `remote-gitimport` when importing `annotated-tags`. In addition to:
- storing the raw tag
- converting the tag to a bookmark
- storing the metadata of the tag as changeset
this diff maintains a mapping between the tag-name and the changeset representing the metadata of the tag. This mapping would be useful to answer queries about the tag metadata when queried through Git and also enable going back to Git from Mononoke without losing data.
Differential Revision: D46067809
fbshipit-source-id: e2463160cdf647b50fa4ecfe4674d079016b4adc
Summary: Implement an alternative to ChangesetPathHistory::history using the new commit graph. This is almost identical to the skiplist implementation except that an (incorrect) optimization was removed that was trying to minimize the number of is_ancestor queries which should no longer be needed as they are a lot faster with the new commit graph.
Differential Revision: D46109911
fbshipit-source-id: 3e8f0c36a48084116673dda3741de87e0741f1a2
Summary: Implement an alternative to ChangesetContext::history using the new commit graph by using AncestorsStreamBuilder. Also made the tests not depend on the exact output history order but instead accept any topological order.
Differential Revision: D46109910
fbshipit-source-id: 31f3e219e9addf04b49fc500ad245d541f110887
Summary:
Factors out the logic of creating an ancestors stream to a builder to allow for more customizability (required for ChangesetContext::history) and reduce code duplication.
Additionally this optimizes range_stream to only traverse changesets that are descendants of start_id. Ancestry checks are kept to a minimum (worst case O(merges)) by only checking on parents of changesets that have a skip tree parent with generation lower than the generation of start_id.
Reviewed By: liubov-dmitrieva
Differential Revision: D46276429
fbshipit-source-id: 65c4f2892c92baaab0fef4f5041f06095ec49e05
Summary: We didn't have a way to do this before so let's add it.
Reviewed By: mitrandir77
Differential Revision: D46190481
fbshipit-source-id: 84544f0226ef3a4ef4d8ac9b4bf66551e4671793
Summary: It's not ideal that we pull mononoke_app and even instantiate MononokeApp here. This is an attempt to decouple Mononoke from Metagit.
Reviewed By: mitrandir77
Differential Revision: D45862434
fbshipit-source-id: 6eab3577743810461f22535d89cf9a00dc1cbb65
Summary: Most of the useful ways to use these methods requires providing an async closure so let's asyncify them.
Reviewed By: RajivTS
Differential Revision: D45607981
fbshipit-source-id: ee7b471cc5d60a793a79928f6c15c48c0da1d08e
Summary: In the case of a multiplexed blobstore, the put operation can exit after it succeeds in one inner blobstore before finishing in all, and leave the rest running in the background. This sleep tries to prevent exiting early before they all finish.
Reviewed By: mitrandir77
Differential Revision: D46190037
fbshipit-source-id: ac2ae0d1453ebdc923170b9da7ca98df7519bf52
Summary: Implements a public method changeset_children in CommitGraph that fetches the changeset ids of the children of a given changeset.
Differential Revision: D46185434
fbshipit-source-id: f0b2e2ee53c7a34fe94ee5ef1f74f6e31429d056
Summary: used cached version of API
Reviewed By: YousefSalama
Differential Revision: D46184380
fbshipit-source-id: 5deb8c455ad682afcf1f0cad822d34fdd632b88b
Summary: The admin tool currently doesn't register the `fb303` extension with `MononokeApp` during its creation. This makes the tool incapable of registering for health check and responding to thrift health check requests by TW. In absence of ALIVE response from the job, TW ends up killing it. This diff adds the extension during `MononokeApp` construction. This will not impact admin tool's usage through CLI but the error messages in the integration tests will need to be updated.
Differential Revision: D46147128
fbshipit-source-id: f06b8bcdce938346726b9ff520a3d809d9fda6fc
Summary: Makes it easier to rollback
Reviewed By: mitrandir77
Differential Revision: D46156526
fbshipit-source-id: eae70801069d09ab03b3b683712d9353fb1e5a43
Summary: The bookmark `get` command simply displays the changeset that is pointed to by the bookmark. This diff modifies the command to also handle git annotated tags converted into bookmarks. If the bookmark is a tag bookmark, then the command also outputs the metadata changeset associated with the bookmark. This will be useful for validating that imported annotated tags retain their metadata through metadata changesets.
Differential Revision: D46022692
fbshipit-source-id: b3980f4799066913268ef7acc1f4956d3feb5b23
Summary: `BonsaiTagMapping` exists as a facet so let's add the appropriate builder methods in `RepoFactory` and `TestRepoFactory`. Follow-up diffs will add this new facet in the `Repo` facet container for `SCS` (for `remote-gitimport`) and in `gitimport`.
Differential Revision: D46020528
fbshipit-source-id: 88e65a498355bf21edc8c8d8aa3755a5507fe991
Summary:
The `metadata-diff` endpoint in SCS is used by Phabricator to get information about the difference between files of two commits. Currently, this endpoint works using raw file content. This diff makes the following optimizations:
- Avoids loading file content in memory if the file is `binary` or `non-utf8`
- Avoids parsing `generated` and `partially-generated` tags if the file is known to be neither
- Avoid parsing raw file bytes into UTF-8 string if the file is non-generated
These optimizations will be most helpful for the case of `binary` or `non-utf8` files but should improve performance of the endpoint in general.
Reviewed By: YousefSalama
Differential Revision: D46109336
fbshipit-source-id: b3c915005686f77defceeb53f1587049bc8b6bd8
Summary: Implement an alternative to slice_repository using the new commit graph that doesn't depend on the skiplist. Useful when doing any kind of processing that requires ancestors to be processed first (backfilling derived data, mirroring commits, ...)
Reviewed By: mitrandir77
Differential Revision: D45607347
fbshipit-source-id: b2012972472e718522e4f33ff01fc9b253b76fff
Summary: Makes slightly more sense
Reviewed By: mitrandir77
Differential Revision: D45607346
fbshipit-source-id: 6b505796da96ff24cd7d555290dcf341f93c797f
Summary: My fingers are tired from scrolling
Reviewed By: mitrandir77
Differential Revision: D45607349
fbshipit-source-id: fe366797a37fbd1682ca917d6bdd39c88195c8e9
Summary: common_base returns all of the highest generation changesets that are common ancestors of two changesets. Implementation is optimized for the most common case were there are no merges between the two changesets and their lowest common ancestor (taking O(log(height) iterations in that case), but is sound for all cases.
Reviewed By: mitrandir77
Differential Revision: D45480232
fbshipit-source-id: 147b6feaa57cdbeb5efb4b77335728f38105fb87
Summary:
The bookmark update log is sequential and we can't write two things to it at
once (for a single repo). The aosp megarepo infra sometimes moves two bookmarks
concurrently and hits this ugly race: P743467889. Let's add retries to failed
bookmark moves.
Reviewed By: YousefSalama
Differential Revision: D46105621
fbshipit-source-id: 5c822301fc3201b070c1fe29eda6c395d265c33f
Summary: This diff adds unit tests to validate that the `BonsaiTagMapping` implementation works as expected.
Reviewed By: mitrandir77
Differential Revision: D46020108
fbshipit-source-id: 12e8268363f0917645a5eadd37eff321b510b755
Summary: Constructs a preloaded commit graph when preloaded_commit_graph_blobstore_key is set in repo config.
Reviewed By: RajivTS
Differential Revision: D45993556
fbshipit-source-id: 3dd22617cbabcc007d02965e7392eb2f5283b1b8
Summary: We need a way to specify which repos will use a preloaded commit graph and the blobstore key of the commit graph blob in that case.
Reviewed By: RajivTS
Differential Revision: D45993558
fbshipit-source-id: 4cf2ea84b72e3a50187fab3ece86e623a754d571
Summary: Adds a new command update-preloaded that tries to load a blob of the preloaded commit graph, update it with any newly added changeset edges, and re-upload it to the blobstore.
Reviewed By: mitrandir77
Differential Revision: D45993560
fbshipit-source-id: 4d14cbf091d86f165ed58fbba9305f0fa6021632
Summary: Implements a commit graph storage for preloading commit graph edges in memory. Useful for repos with complicated commit graph structures but relatively small overall size. It uses Reloader to periodically update the preloaded edges. CompactChangesetEdges are introduced to lower the memory usage by referencing other changesets using interned ids.
Reviewed By: mitrandir77
Differential Revision: D45992604
fbshipit-source-id: 5852079d10a0ba8c491e7b239d8cf4c58813dd45
Summary: These are not specific to caching so let's move them
Reviewed By: RajivTS
Differential Revision: D45993559
fbshipit-source-id: b5f127bf2997c4bf5381013df10af28f1c42c1c9
