Commit Graph

4 Commits

Author SHA1 Message Date
Simon Farnsworth
9b9607b02e Have manual_scrub continue on errors, writing out files to let you retry instead
Summary:
There are blobs that fail to scrub and terminate the process early for a variety of reasons; when this is running as a background task, it'd be nice to get the remaining keys scrubbed, so that you don't have a large number of keys to fix up later.

Instead of simply outputting to stdout, write keys to one of three files in the format accepted on stdin:

1. Success; you can use `sort` and `comm -3` to remove these keys from the input dat, thus ensuring that you can continue scrubbing.
2. Missing; you can look at these keys to determine which blobs are genuinely lost from all blobstores, and fix up.
3. Error; these will need running through scrub again to determine what's broken.

Reviewed By: krallin

Differential Revision: D23574855

fbshipit-source-id: a613e93a38dc7c3465550963c3b1c757b7371a3b
2020-09-09 07:25:13 -07:00
Alex Hornby
7766cac6a1 mononoke: add task spawning to manual_scrub
Summary: Add task spawning to manual_scrub to increase throughput

Reviewed By: farnz

Differential Revision: D23055811

fbshipit-source-id: 1e3d1f0e5b5fc2f2935aa367ae2e749c867d2d62
2020-08-12 01:31:03 -07:00
Alex Hornby
e18cf0210f mononoke: use the background session mode from manual scrub
Summary: There are no users waiting on manual scrub, so set it to use the background session mode.

Reviewed By: krallin

Differential Revision: D23054581

fbshipit-source-id: 985bcadbaf17d2a8c92fdec811ecb239cbca7b37
2020-08-12 01:31:03 -07:00
Simon Farnsworth
65e7404eba Command to manually scrub keys supplied on stdin
Summary: For populating the XDB blobstore, we'd like to copy data from Manifold - the easiest way to do that is to exploit MultiplexedBlobstore's scrub mode to copy data directly.

Reviewed By: krallin

Differential Revision: D22373838

fbshipit-source-id: 550a9c73e79059380337fa35ac94fe1134378196
2020-07-10 01:01:05 -07:00