Summary:
We need that for current catchup to avoid confusing clients with backdated
commits.
Reviewed By: markbt
Differential Revision: D46743487
fbshipit-source-id: a71ee30eada63dab7c64cbe43a4495662fe04371
Summary: Walker previously specified a default for the deprecated `cachelib-only-blobstore` argument. Switch to `cache-mode`, allowing us to specify local caching for all types, not just the blobstore.
Differential Revision: D45089769
fbshipit-source-id: 978ac545590935be5a825669db9df58f0281c41a
Summary: To avoid showing a bunch of "?" files, we need to ignore things within submodules. For watchman, we do similar to Python and add submodules to the ignored dirs list. For the manual walker, we similarly skip directories in the ignore list. Note that the walker already skipped the submodules due to the presence of the "dot dir" (e.g. ".sl") in the submodules, but I piped through the ignored dirs skipping to be consistent.
Reviewed By: quark-zju
Differential Revision: D46547238
fbshipit-source-id: b001dc02a3c73f69d74fe36615a812f9ac983944
Summary: Python status for Git has submodule aware logic we don't have in Rust, so we need to defer to Python. However, Python will still be able to use the internal Rust status API.
Reviewed By: zzl0
Differential Revision: D46547239
fbshipit-source-id: 2caf1436819219cea007307f602627fd1a38ef51
Summary: It wasn't used from Rust (only MultiWalker(threads=8) was used), and Rust walker is not enabled from Python. It's annoying to have to make changes twice, so let's kill the non-parallel walker.
Reviewed By: zzl0
Differential Revision: D46547240
fbshipit-source-id: 4a57222ec705059acbc6865f8e1a9701ae584ccc
Summary: This wasn't enabled in production, and the parallel walker doesn't currently work from Python due to matcher GIL deadlock. Anyway, I'm working to make Python use the Rust status directly, so this intermediate optimization of using the Rust walker in the Python status won't buy us anything.
Reviewed By: zzl0
Differential Revision: D46646007
fbshipit-source-id: 645d88adb85700062c3edf2483fb5a5ffb76cca2
Summary: These make it more convenient to conditionalize logic based on storage format.
Reviewed By: zzl0
Differential Revision: D46547242
fbshipit-source-id: ef833c954090f5cb2b1c2c6e616d5d67a442886d
Summary: I'm going to use this to skip submodules in the Rust status implemenation.
Reviewed By: zzl0
Differential Revision: D46547241
fbshipit-source-id: e5cf5dc5ee77aa3e8927ab3cf5922f3e29a25ae5
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stdout)`.
This codemod removes all remaining instances of atty
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`. `arc autocargo` was also ran
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: dbxfb
Differential Revision: D46736264
fbshipit-source-id: a686b96b1fa0aa4389f65487ed426f226c86e8e9
Summary: www has lots of empty commits that are being synced to all small repos. Until we fix that problem let's refrain from backsyncing any empty commits. I'm making it a tunable for easy on-off
Reviewed By: liubov-dmitrieva
Differential Revision: D46727657
fbshipit-source-id: 5f204cc8231fec3e6a4eaf25b6575c0d637d67a3
Summary:
[plan_deletion] Implement `commit-deletion-plan-to-deletion-log-db` subcommand
This is the last necessary step before enacting the deletion.
As the outcome of this step, the deletion_log db is populated and we can start taking the necessary steps to safely proceed with the actual deletion of changesets and blobs.
We still need to document and potentially write tools to help with the whole process, including blobs redaction.
Differential Revision: D46724273
fbshipit-source-id: bc60209a28e2ee8a50ac0371f78c0b3be234781b
Summary: This diff moves all the bulb unlinking functions into a struct.
Reviewed By: mitrandir77
Differential Revision: D46759318
fbshipit-source-id: 143171706485ec6a56152bdfea6202221d88fe03
Summary: This diff makes the `get_blobstores` function to be public, so that we can use it to get blobstores. These blobstores are going to be used to unlink keys in a bulk.
Reviewed By: mitrandir77
Differential Revision: D46729000
fbshipit-source-id: 11d7ddc5f90527ebdb79141f61f1671997108ac5
Summary:
This diff allows the new admin tool to do the following for the new bulk unlking keys subcommand:
* Read the key list from the files
* Print all the keys extracted when running in dry-run mode
* Report the progress
* Fix the argument to take a boolean value
Reviewed By: mitrandir77
Differential Revision: D46725312
fbshipit-source-id: c039b89e00a9ffbbbee41e20617f45543ef147ce
Summary: This diff adds a new subcommand for our new admin tool, so that we can delete keys bulkly. The input is the directory that contains a list of files, each file contains a list of *Manifold* keys to remove.
Reviewed By: mitrandir77
Differential Revision: D46689957
fbshipit-source-id: 077b6a4c76872ef01bc7f0eab7096abeb87d7047
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stdout)`.
This codemod replaces all usages of `atty::is(Stdout)` with `stdout().is_terminal()`, importing the necessary paths.
It used this script (generated with assistance from Metamate):
```
#!/bin/bash
files=$(fbgs -sl "atty::is(atty::Stream::Stdout)" | arc linttool debugfilterpaths --take RUSTFMT)
for file in $files; do
sed -i 's/atty::is(atty::Stream::Stdout)/stdout().is_terminal()/g' $file
done
for file in $files; do
sed -i '1i use std::io::{stdout, IsTerminal};' $file
done
```
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`.
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: danielocfb
Differential Revision: D46601191
fbshipit-source-id: 3f2269c518e36241ab0e995b87976323d7da9cc6
Summary:
`std::io::IsTerminal` was stabilized in 1.70. It is intended to be a drop in replacement for `atty::is(Stderr)`.
This codemod replaces all usages of `atty::is(Stderr)` with `stderr().is_terminal()`, importing the necessary paths.
It used this script (generated with assistance from Metamate):
```
#!/bin/bash
files=$(fbgs -sl "atty::is(atty::Stream::Stderr)" | arc linttool debugfilterpaths --take RUSTFMT)
for file in $files; do
sed -i 's/atty::is(atty::Stream::Stderr)/stderr().is_terminal()/g' $file
done
for file in $files; do
sed -i '1i use std::io::{stderr, IsTerminal};' $file
done
```
`arc f` was ran twice, followed by two invocations of `arc lint -e extra --take RUSTFIXDEPS`.
This script was a little naive though, and as a result, I had to manually move the newly added imports to the correct line. I ran lints again afterwards.
Reviewed By: Imxset21
Differential Revision: D46601192
fbshipit-source-id: 18ef53a9fa25cbd733ca81f16b2c13ce8c5ddb98
Summary: `gitimport` is typically used for mirror repos which by definition are locked. Creating or moving bookmarks on locked repos is disallowed by default. This behaviour can be overriden by passing the bypass flag. This diff adds argument support for bypassing the read-only check.
Differential Revision: D46722005
fbshipit-source-id: 9defcbc159a55d0e7ff197d8d732e3d5a208a998
Summary: Use the EDEN_TRY macro to reduce error-handling line noise.
Reviewed By: genevievehelsel
Differential Revision: D46668011
fbshipit-source-id: bb54ff710928b3a5c394b6e79474c25ef4775895
Summary: Adds a macro to simplify bailing out of Try-returning functions, sort of like with Rust's question mark operator.
Reviewed By: chadaustin
Differential Revision: D46567326
fbshipit-source-id: f13eec678a101e423a99ba469a20d6de5513d73c
Summary: optimize the check against skip revset by reusing the unskipped result.
Reviewed By: muirdm
Differential Revision: D46687070
fbshipit-source-id: 01e5be4499daaa9e24437ddb16e2cc04f0436211
Summary:
Currently, the 'skip' option is too slow when users want to use it to bisect on
a file or folder like below:
sl bisect --skip "!( file('path:foo') & file('path:bar') )"
Since it tries to eval the skip revset upfront, which takes a long time for
current `file` implementation. This diff make it lazy evaluated.
Reviewed By: muirdm
Differential Revision: D46643574
fbshipit-source-id: c063fbea2bc0703772e07d5167adc8fbf989a68f
Summary:
move the bisect kind check earlier, so that it will not do unnecessary
node lookup
Reviewed By: sggutier
Differential Revision: D46643576
fbshipit-source-id: 624c6410aaf95f82a993093df4e24add6c5e48ea
Summary: this will be needed for making 'skip' lazy evaluated
Reviewed By: muirdm
Differential Revision: D46643575
fbshipit-source-id: 53cfe2395d320d85582fbf707f62d69a5088bea2
Summary: we will update this function in the following diffs, abstract it to a separate function will avoid making the main function too complex
Reviewed By: muirdm
Differential Revision: D46643577
fbshipit-source-id: 227e0b60fe2efa1d6d00fa409ea95a30cc972747
Summary: We handle the error gracefully, when deleting a key from the underlying blobstores.
Reviewed By: RajivTS
Differential Revision: D46684795
fbshipit-source-id: 45349aab01e56b11fd1ab455aec2bb69d72c497b
Summary:
If there's corruption in the cache, it should be considered a miss,
rather than failing.
Reviewed By: chadaustin
Differential Revision: D46577606
fbshipit-source-id: 4cc99e743e869422a149a073b726722896d971d4
Summary:
The dagcopytrace was enabled for source missing file in D46315431.
It's too slow if there are many files deleted in the source commit. I will debug the root cause tomorrow, let's disable it for hg repo first. We are not disabling it for git repo, because there is not alternative for git repos.
Reviewed By: muirdm
Differential Revision: D46711460
fbshipit-source-id: 4e20bbb34be126c13af081475953b83df61872df
Summary: Adding to track possible corruption issues we've been getting user reports about
Reviewed By: chadaustin
Differential Revision: D46706164
fbshipit-source-id: 6a868d9f9ffd3f72303123486bbb8312f2002072
Summary: This diff allows the new admin tool to delete keys from all the related blobs (either the single blobstore specified by the id, or all the underlying blobstore given a repo)
Differential Revision: D46564415
fbshipit-source-id: d0299262a82acb4e1a9f4cedd5510418f270d527
Summary:
This diff adds a new function to construct multiple blobstores from a given repo.
The finaly step is to refactoring the existing `get_blobstore` function.
Reviewed By: RajivTS
Differential Revision: D46559402
fbshipit-source-id: 9b9901d00566556614264831f8643f7c6d4ee20f
Summary: Running a job in sharded setting requires it to support pre-emption in case ShardManager decides to move the repo around. This diff adds support for that pre-emption.
Differential Revision: D46514772
fbshipit-source-id: fb086e74a9c56e0ced5921877de49401bc426170
Summary: As in title. This tool is currently run as 5-6 different TW jobs that compute and publish data for specific repos. Instead of having multiple jobs, if we manage to shard this job across set of available tasks, it becomes much more easier to manage.
Differential Revision: D46483039
fbshipit-source-id: d7d879dee088c8f431c22915bd7ef9602b83aa00
Summary:
In the win32 implementation of `readFile`, the ret string wasn't being resized
to the number of bytes actually read, which may be smaller than given in
`num_bytes`. This change makes it match the behavior of folly's readFile.
Reviewed By: xavierd
Differential Revision: D46532796
fbshipit-source-id: 1da3e4ff4190673796731610ae8a798b953d0c96
Summary:
This diff adds a new function of constructing a single blobstore from config and inner_id.
Next step is to add a new function to construct multiple blobstores from a given repo.
The finaly step is to refactoring the existing `get_blobstore` function.
Reviewed By: RajivTS
Differential Revision: D46559565
fbshipit-source-id: b59e77e13fbe76ba013caa62bfeb9f56cbcf31d2
Summary: This diff added a function to retrieve all the inner blobstore ids from a given multiplexed blobstore.
Reviewed By: RajivTS
Differential Revision: D46523044
fbshipit-source-id: e4c3b02e1672bc2eea3208bac4f23d879c004515
Summary: Now that the `blob_repo` dependency has been removed from `statistics_collector`, this diff migrates the tool from old Mononoke App to new Mononoke App getting rid of the old clap dependency.
Differential Revision: D46478775
fbshipit-source-id: dd713eef2175fa01934508445866d64ef01657f0
Summary: In order to better understand what happens in cases described in https://fb.workplace.com/groups/edenfswindows/permalink/1315557692693439/ adding better logging and some data integrity checks.
Reviewed By: chadaustin
Differential Revision: D46605682
fbshipit-source-id: 485c5c83834cdde9ccfea9cfa94323cf03df7002
Summary:
eden doctor can discover when an inode is missing for a file, but can't
remediate the issue. restart usually remediates the issue, but it would be
better to have doctor remediate since restart can be very slow.
We could do something similar to our remediation for phantom inodes (performing
a filesystem operation). However, messing with the filesystem leaves us open to
races with concurrent modifications. The point of the filesystem io is to make
eden see a notification about a certain path and match it's state to the
filesystem. So we can directly do that instead.
We can more directly do this by introducing a thrift call to make eden match
it's internal state to the filesystem.
We could replace the other remediation with this thrift call. I'll leave that
for a follow up.
Reviewed By: xavierd
Differential Revision: D46243633
fbshipit-source-id: a1df5929428dc4f6c8fd71d826fe1e7371ebf283
Summary:
We've got a couple sets of prjfs tests now and they have some utility functions.
Let's move this all into a base class so it can easily be reused.
Reviewed By: xavierd
Differential Revision: D46379827
fbshipit-source-id: c4b708a1effebe5f11c97a63519295be625c8018
