Summary:
We need that for current catchup to avoid confusing clients with backdated
commits.
Reviewed By: markbt
Differential Revision: D46743487
fbshipit-source-id: a71ee30eada63dab7c64cbe43a4495662fe04371
Summary: Walker previously specified a default for the deprecated `cachelib-only-blobstore` argument. Switch to `cache-mode`, allowing us to specify local caching for all types, not just the blobstore.
Differential Revision: D45089769
fbshipit-source-id: 978ac545590935be5a825669db9df58f0281c41a
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stdout)`.
This codemod removes all remaining instances of atty
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`. `arc autocargo` was also ran
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: dbxfb
Differential Revision: D46736264
fbshipit-source-id: a686b96b1fa0aa4389f65487ed426f226c86e8e9
Summary: www has lots of empty commits that are being synced to all small repos. Until we fix that problem let's refrain from backsyncing any empty commits. I'm making it a tunable for easy on-off
Reviewed By: liubov-dmitrieva
Differential Revision: D46727657
fbshipit-source-id: 5f204cc8231fec3e6a4eaf25b6575c0d637d67a3
Summary:
[plan_deletion] Implement `commit-deletion-plan-to-deletion-log-db` subcommand
This is the last necessary step before enacting the deletion.
As the outcome of this step, the deletion_log db is populated and we can start taking the necessary steps to safely proceed with the actual deletion of changesets and blobs.
We still need to document and potentially write tools to help with the whole process, including blobs redaction.
Differential Revision: D46724273
fbshipit-source-id: bc60209a28e2ee8a50ac0371f78c0b3be234781b
Summary: This diff moves all the bulb unlinking functions into a struct.
Reviewed By: mitrandir77
Differential Revision: D46759318
fbshipit-source-id: 143171706485ec6a56152bdfea6202221d88fe03
Summary: This diff makes the `get_blobstores` function to be public, so that we can use it to get blobstores. These blobstores are going to be used to unlink keys in a bulk.
Reviewed By: mitrandir77
Differential Revision: D46729000
fbshipit-source-id: 11d7ddc5f90527ebdb79141f61f1671997108ac5
Summary:
This diff allows the new admin tool to do the following for the new bulk unlking keys subcommand:
* Read the key list from the files
* Print all the keys extracted when running in dry-run mode
* Report the progress
* Fix the argument to take a boolean value
Reviewed By: mitrandir77
Differential Revision: D46725312
fbshipit-source-id: c039b89e00a9ffbbbee41e20617f45543ef147ce
Summary: This diff adds a new subcommand for our new admin tool, so that we can delete keys bulkly. The input is the directory that contains a list of files, each file contains a list of *Manifold* keys to remove.
Reviewed By: mitrandir77
Differential Revision: D46689957
fbshipit-source-id: 077b6a4c76872ef01bc7f0eab7096abeb87d7047
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stdout)`.
This codemod replaces all usages of `atty::is(Stdout)` with `stdout().is_terminal()`, importing the necessary paths.
It used this script (generated with assistance from Metamate):
```
#!/bin/bash
files=$(fbgs -sl "atty::is(atty::Stream::Stdout)" | arc linttool debugfilterpaths --take RUSTFMT)
for file in $files; do
sed -i 's/atty::is(atty::Stream::Stdout)/stdout().is_terminal()/g' $file
done
for file in $files; do
sed -i '1i use std::io::{stdout, IsTerminal};' $file
done
```
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`.
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: danielocfb
Differential Revision: D46601191
fbshipit-source-id: 3f2269c518e36241ab0e995b87976323d7da9cc6
Summary: `gitimport` is typically used for mirror repos which by definition are locked. Creating or moving bookmarks on locked repos is disallowed by default. This behaviour can be overriden by passing the bypass flag. This diff adds argument support for bypassing the read-only check.
Differential Revision: D46722005
fbshipit-source-id: 9defcbc159a55d0e7ff197d8d732e3d5a208a998
Summary: We handle the error gracefully, when deleting a key from the underlying blobstores.
Reviewed By: RajivTS
Differential Revision: D46684795
fbshipit-source-id: 45349aab01e56b11fd1ab455aec2bb69d72c497b
Summary: This diff allows the new admin tool to delete keys from all the related blobs (either the single blobstore specified by the id, or all the underlying blobstore given a repo)
Differential Revision: D46564415
fbshipit-source-id: d0299262a82acb4e1a9f4cedd5510418f270d527
Summary:
This diff adds a new function to construct multiple blobstores from a given repo.
The finaly step is to refactoring the existing `get_blobstore` function.
Reviewed By: RajivTS
Differential Revision: D46559402
fbshipit-source-id: 9b9901d00566556614264831f8643f7c6d4ee20f
Summary: Running a job in sharded setting requires it to support pre-emption in case ShardManager decides to move the repo around. This diff adds support for that pre-emption.
Differential Revision: D46514772
fbshipit-source-id: fb086e74a9c56e0ced5921877de49401bc426170
Summary: As in title. This tool is currently run as 5-6 different TW jobs that compute and publish data for specific repos. Instead of having multiple jobs, if we manage to shard this job across set of available tasks, it becomes much more easier to manage.
Differential Revision: D46483039
fbshipit-source-id: d7d879dee088c8f431c22915bd7ef9602b83aa00
Summary:
This diff adds a new function of constructing a single blobstore from config and inner_id.
Next step is to add a new function to construct multiple blobstores from a given repo.
The finaly step is to refactoring the existing `get_blobstore` function.
Reviewed By: RajivTS
Differential Revision: D46559565
fbshipit-source-id: b59e77e13fbe76ba013caa62bfeb9f56cbcf31d2
Summary: This diff added a function to retrieve all the inner blobstore ids from a given multiplexed blobstore.
Reviewed By: RajivTS
Differential Revision: D46523044
fbshipit-source-id: e4c3b02e1672bc2eea3208bac4f23d879c004515
Summary: Now that the `blob_repo` dependency has been removed from `statistics_collector`, this diff migrates the tool from old Mononoke App to new Mononoke App getting rid of the old clap dependency.
Differential Revision: D46478775
fbshipit-source-id: dd713eef2175fa01934508445866d64ef01657f0
Summary: This is a refactor aimed to retain the same behavior while consolidating the code and improving it's resilience to future changes.
Reviewed By: mitrandir77
Differential Revision: D46190582
fbshipit-source-id: a956f5df1044c52346a712e271c75baf298b481d
Summary: We need that for optimizing our storage.
Reviewed By: RajivTS
Differential Revision: D46362446
fbshipit-source-id: d81973cd05cffdc78ad84da867fa4791efea877c
Summary: As in title. Once `BlobRepo` has been removed from this tool, I will also migrate it to the new `Mononoke Admin`
Reviewed By: mitrandir77
Differential Revision: D46444744
fbshipit-source-id: 059623d32d608f852254f16f811bcaf4e6bf3b4a
Summary: This includes PR 7 so drops the patch.
Reviewed By: zertosh
Differential Revision: D46451758
fbshipit-source-id: 5df8ccbfa610eab470744e28b5bffe411a983996
Summary:
saintelmosfire
Will only land this after D45356044 is landed and tested.
Differential Revision: D45356043
fbshipit-source-id: 068aa45e1edd954b8ba5dc3355f9392ab06419b5
Summary: upgrade parking_lot from version 0.11.2 to version 0.12.1
Reviewed By: zertosh, diliop, dtolnay
Differential Revision: D46398501
fbshipit-source-id: ffae0ea188abad5253f1524216dd18ce9a53a74c
Summary:
In D41769964, we decided not to stop `gitimport` simply because `create-bookmark` fails cause that could be due to the fact that the bookmark being created already exists. If that is indeed the case, we try to resolve the bookmark and move it to the new value. However, there could be other unrelated reasons for `create_bookmark` to fail in which case we would still try to resolve the bookmark and end up in unexpected situations (e.g. Luibov trying to import `paws` repo faced this problem). This diff does two things:
- Performs an early check to determine if the bookmark exists by resolving it. If it does exist, it executes the move bookmark branch of code
- If the bookmark doesn't exist, then it creates it.
If there are any errors during bookmark creation or movement, they get logged for the user.
Reviewed By: mitrandir77
Differential Revision: D46071703
fbshipit-source-id: 446b0bc720d13b9e9314feccdb147b2bab0b7cd9
Summary: Added `fill_from_function` which makes it much more useful
Reviewed By: fanzeyi
Differential Revision: D46342378
fbshipit-source-id: 9bc8c01dcb10806a03f1eab628f9763c4e5edc81
Summary:
add new ods logging for bookmark fetching from phases crate
This would help with investigation sevs like S344849, also having this info we
can setup alerts.
Reviewed By: YousefSalama
Differential Revision: D46314640
fbshipit-source-id: 66e72da19ad4eb528c78d0783c297746f57cd4f1
Summary:
This offers a new interface for Shard Manager managed repos to be able to load redacted content.
The new API entry point: `open_managed_repos_unredacted` is analogous to `open_repo_unredacted`, but supports a Shard Manager based workflow.
Reviewed By: RajivTS
Differential Revision: D46280863
fbshipit-source-id: e2e4aebcb08d01f3be97d11d6fe8da2822ba6f29
Summary:
This diff modifies the behavior of `gitimport` and `remote-gitimport` when importing `annotated-tags`. In addition to:
- storing the raw tag
- converting the tag to a bookmark
- storing the metadata of the tag as changeset
this diff maintains a mapping between the tag-name and the changeset representing the metadata of the tag. This mapping would be useful to answer queries about the tag metadata when queried through Git and also enable going back to Git from Mononoke without losing data.
Differential Revision: D46067809
fbshipit-source-id: e2463160cdf647b50fa4ecfe4674d079016b4adc
Summary: Implement an alternative to ChangesetPathHistory::history using the new commit graph. This is almost identical to the skiplist implementation except that an (incorrect) optimization was removed that was trying to minimize the number of is_ancestor queries which should no longer be needed as they are a lot faster with the new commit graph.
Differential Revision: D46109911
fbshipit-source-id: 3e8f0c36a48084116673dda3741de87e0741f1a2
Summary: Implement an alternative to ChangesetContext::history using the new commit graph by using AncestorsStreamBuilder. Also made the tests not depend on the exact output history order but instead accept any topological order.
Differential Revision: D46109910
fbshipit-source-id: 31f3e219e9addf04b49fc500ad245d541f110887
Summary:
Factors out the logic of creating an ancestors stream to a builder to allow for more customizability (required for ChangesetContext::history) and reduce code duplication.
Additionally this optimizes range_stream to only traverse changesets that are descendants of start_id. Ancestry checks are kept to a minimum (worst case O(merges)) by only checking on parents of changesets that have a skip tree parent with generation lower than the generation of start_id.
Reviewed By: liubov-dmitrieva
Differential Revision: D46276429
fbshipit-source-id: 65c4f2892c92baaab0fef4f5041f06095ec49e05
Summary: We didn't have a way to do this before so let's add it.
Reviewed By: mitrandir77
Differential Revision: D46190481
fbshipit-source-id: 84544f0226ef3a4ef4d8ac9b4bf66551e4671793
Summary: It's not ideal that we pull mononoke_app and even instantiate MononokeApp here. This is an attempt to decouple Mononoke from Metagit.
Reviewed By: mitrandir77
Differential Revision: D45862434
fbshipit-source-id: 6eab3577743810461f22535d89cf9a00dc1cbb65
