Summary: Add --drain-only option to blobstore_healer so we can drain a queue without healing when required.
Reviewed By: ikostia
Differential Revision: D17111436
fbshipit-source-id: e2fba4914e9699a4836933d0f35def015cd3dadc
Summary: Log the region with max replication lag as well as the lag size
Reviewed By: StanislavGlebik
Differential Revision: D17097266
fbshipit-source-id: c449167b09dbd2cdce7d65bfc956e7ccf59b2415
Summary:
blobstore_healer performance improvements
* speed up delete by using bulk delete
* speed up rep lag check by only doing it once if lag within bounds
* log some stats from each healer chunk run to help diagnose issues
* dont log the full storage config on each log line ( this was making it hard to read the logs )
Reviewed By: StanislavGlebik
Differential Revision: D17093809
fbshipit-source-id: 34299ad1835c5548b9502ad9d5a3e18eff1ab95b
Summary:
Utility to backfill derived data:
- currently only support `unodes`
Reviewed By: krallin
Differential Revision: D16827897
fbshipit-source-id: aa0bb8ae4fe895233e3f11775b3432184afb37f9
Summary:
This is a part of an effort to move mercurial related stuff into mercurial crate:
- `BlobManifest|HgBlobEntry` -> `mercurial/types/blobs`
- helper `Id` type is no longer required and `BlobManifeset` can be loaded with `HgManifesetId::load`
- `BlobManifest` implements `HgManifest` and typed `Manifest` traits
Reviewed By: StanislavGlebik
Differential Revision: D16984745
fbshipit-source-id: 33d4eb9c33137258956917328efa1f5ec6737ee9
Summary:
Add two helpful commands:
1) Fetch the phase of a commit
2) List all public commits
Reviewed By: ikostia
Differential Revision: D16830576
fbshipit-source-id: 03f503cb30a7f150ea383d62fb71913dd2b93e6e
Summary:
This updates the `mononoke_admin` filenodes command to allow showing filenodes
by path + ID in addition to revision + path. This makes it possible to locate
the linknode given a filenode ID but no revision.
This updates the filenodes CLI command a little bit (so if you want the old
command you need to add `by-revision`), but this should be fine since this is
a command that a human will normally be calling.
Reviewed By: farnz
Differential Revision: D16936628
fbshipit-source-id: 1079ad580b235076739c92d0c1f3c9bfc80db4c1
Summary: This adds a command that allows importing a set of LFS blobs into Mononoke by streaming them out of another storage location (e.g. Dewey).
Reviewed By: HarveyHunt
Differential Revision: D16917806
fbshipit-source-id: 4917d56e11a187c89e00c23a32c6e791b351f8ef
Summary:
We need this to accomodate `blobimport`'s needs of using (opening(D16914875)
and creating(this)) unredacted repos.
Note: this is nearly identical to D16914875.
Reviewed By: StanislavGlebik
Differential Revision: D16915557
fbshipit-source-id: 6b522e6fcee4582870df27cf66e8ad55e91235fe
Summary:
Currently blobimport fails if it tries to import a redacted blob. We should
only enforce redaction in the scenarios where end users could see the data.
Internal imports should be fine to proceed.
As a separate note, problems like this would probably go away when redaction
would be moved to the `FileStore` level.
Reviewed By: krallin
Differential Revision: D16914875
fbshipit-source-id: e531a6248f66b47de87c7aedcbc928bb7ad4ebc9
Summary:
This is a mechanical part of rename, does not change any commit messages in
tests, does not change the scuba table name/config setting. Those are more
complex.
Reviewed By: krallin
Differential Revision: D16890120
fbshipit-source-id: 966c0066f5e959631995a1abcc7123549f7495b6
Summary:
mononoke_types doesn't look like a good crate for it. Let's move to blobstore
crate instead
Reviewed By: krallin
Differential Revision: D16914142
fbshipit-source-id: 57602c5a644adc87beea0cbc92beeb0d901d5873
Summary:
We now have two Manifest and two Entry traits and it makes it very confusing.
The old Manifest trait (and related EmptyManifest struct and Entry trait) are
mercurial-specific, while new Manifest trait is generic over all kinds of
manifests (unodes, mercurial etc).
In ideal state we should have only new Manifest trait, but it'd require quite a
big refactoring. In short term let's at least rename Manifest to HgManifest and
Entry to HgEntry to make distinction clearer.
Reviewed By: krallin
Differential Revision: D16886032
fbshipit-source-id: 80e3999fae3e89347448579caf7aafec56705500
Summary:
This is the first diff in the stack that removes ugly hack `struct Id` that we
used in derived data (see more details about that later in the stack).
This diff reverses the dependency between blobstore crate and mononoke_types
crate. Previously blobstore depended on mononoke_types, and blobstore also had
Loadable/Storable traits. That means that we can't implement Loadable/Storable
traits for types from mononoke_types - we can't even import them without
creating cyclic dependency between crates! And we'll use Loadable in
mononoke_types later in the stack.
This diff reverses the dependency and moves Loadable/Storable in mononoke_types
Reviewed By: krallin
Differential Revision: D16884649
fbshipit-source-id: 20753f3e62980be7bb6a04ab9d19980bc459b40e
Summary:
There are a couple of functions that take a reference to a blobstore and then
immediately clone it, in order to pass ownership to another function.
Modify these functions to take ownership.
Reviewed By: farnz
Differential Revision: D16859228
fbshipit-source-id: fd7b0ae5af2262601db2201aa510d3099f37ced5
Summary:
Create a new binary that can be used to rechunk files content using the filestore.
The binary accepts multiple filenodes, that it will then go and rechunk using the filestore
config provided to it.
Reviewed By: krallin
Differential Revision: D16802701
fbshipit-source-id: d7c05729f5072ff2925bbc90cdd89fcfed56bba2
Summary:
We have a FetchKey object in the filestore that serves as both the thing you use to ask for reads and to write aliases. One downside of this is that the same enum is used for reads and writes, and that means that you could potentially accidentally try to write an alias for the "canonical" fetch key, which would effectively overwrite your content.
That feels bad, and the underlying reason is that the typing doens't really reflect the reality of what we are trying to do here, which is to support a set of aliases for pieces of content, and expose an enum that allows clients to access content by alias or canonical ID.
This patch updates the type hierarchy to be more in line with what we are trying to do (note that you can no longer store a canonical fetch key). One downside is that it makes the API a bit more verbose, since now you need to say e.g. `FetchKey::Aliased(Alias::Sha256(...))` instead of just `FetchKey::Sha256`. I think the tradeoff is reasonable, but I'm curious to hear other folk's thoughts.
Reviewed By: farnz
Differential Revision: D16764582
fbshipit-source-id: 00c596d1370b52ef2aea04ac44cdcd19b7a2c92a
Summary:
We may need to inspect the list of blacklisted files for a given commit.
This command is called to do so.
Reviewed By: krallin
Differential Revision: D16643467
fbshipit-source-id: ad389c4b9262ffa9ee6ea9b792b4866a23bebaa5
Summary: Clean up non-test usage of CoreContext::test_mock left over from T37478150
Reviewed By: farnz
Differential Revision: D16804838
fbshipit-source-id: f420b8186557a42e9b6c78437c0fb76c9a343b31
Summary:
Start moving mercurial related stuff to `mercurial` directory:
- rename `mercurial` to `mercurial_revlog` and moved to `/mercurial/revlog`
- move `mercurial_types` to `/mercurial/types`
- move `mercurial_bundles` to `/mercurial/bundels`
Reviewed By: farnz
Differential Revision: D16783728
fbshipit-source-id: 79cf1757bb7cc84a6273a4a3c486242b1ef4cd00
Summary:
This resolves the list of files at a given commit to a list of `ContentId`s
and unblacklists all of those `ContentId`s.
Note 1: This means that these particular versions of files will be
unblacklisted in all commits which reference them.
Note 2: There's no auditing of this admin command invokation, if the
need for one arises, we should add it externally.
Reviewed By: krallin
Differential Revision: D16643436
fbshipit-source-id: 3af488e3e4d748027605587ab1e08c0abd0da109
Summary:
This is done to make `remove` and `list` be implementable in terms of the same
boilerplate with some different closures.
NB: I don't particularly like how this refactoring looks, but I do think it
looks better than copy/pasting the code and I think it is of no major
importance, given that this is an admin command.
Reviewed By: krallin
Differential Revision: D16643437
fbshipit-source-id: 7f64b112eda1ce963dffed897bf2c9fa42234d79
Summary: In the following diffs, I will implement the actual logic.
Reviewed By: krallin
Differential Revision: D16639720
fbshipit-source-id: f42b91476035343a6f00b11e4921e26836b1a574
Summary: This adds support for running benchmarks using sqlblob. I've refactored the command a little bit so that destinations are subcommands now.
Reviewed By: HarveyHunt
Differential Revision: D16733238
fbshipit-source-id: 963365ed3492cddf1c7302f18fc44148d9c1ec16
Summary: add support for fb303 and ods stats to cmdlib. use for blobstore_healer to monitor healer status
Reviewed By: krallin
Differential Revision: D16671526
fbshipit-source-id: 8f95fc097133f2fe697ea3cf70fd9d0fa3b3649c
Summary:
This command allows a file to be uploaded to the blobstore using
the filestore - which allows for chunking if required.
Additionally, port the other filestore commands to use logger
rather than println!().
Reviewed By: krallin
Differential Revision: D16712167
fbshipit-source-id: 0fad13c29035a550f8caee7a02d37071eb87cd73
Summary: This updates blobimport to a newer Rust-2018-style. Rust 2018 is nicer.
Reviewed By: StanislavGlebik
Differential Revision: D16667043
fbshipit-source-id: a217cbef0337e7d7cf288f972e9b4af0340a8e0b
Summary:
This updates blobimport to avoid using a per-changeset limit for the number of blob uploads (which used to be 100), and instead to use a global blob upload limit, and a global LFS import limit.
This allows for finer-grained control over the resource utilization (notably in terms of Manifold quota) of blobimport.
While I was in there, I also eliminated an `Arc<BlobRepo>` that was laying around, sicne we don't use those anymore.
Reviewed By: HarveyHunt
Differential Revision: D16667044
fbshipit-source-id: 9fc2f347969c7ca9472ce8dd3d4e2f1acb175b66
Summary: This updates blobimport to allow specifying a different commit limit than the default of 100. This makes it a little more convenient when tuning limits.
Reviewed By: HarveyHunt
Differential Revision: D16648558
fbshipit-source-id: 2e8c8496487a79bec84ec964302b1d6e871caf2a
Summary:
Sometimes, our Mononoke CLI tests report errors in mode/opt where they don't
actually output error messages at all when returning an error. Those error
messages actually get buffered by our logger, and are normally emitted when
said logger is dropped.
However, I have a suspicion that by calling process::exit(), we're not giving
Rust a chance to drop the logger (because that eventually just calls the exit
syscall), and as a result our buffer isn't getting flushed.
This patch attempts to fix this by removing all calls to process::exit in the
admin CLI, and instead returning an exit code from main, which I hope will
allow main to drop everything before it returns.
More broadly speaking, I don't think it's very good practice to exit from all
over the place in the admin CLI, so this fixes that :)
Reviewed By: HarveyHunt
Differential Revision: D16687747
fbshipit-source-id: 38e987463363e239d4f9166050a8dd26a4bef985
Summary:
Address nits from D16599270:
* use map instead of and_then
* use &[T] instead of &Vec[T]
Reviewed By: StanislavGlebik
Differential Revision: D16666838
fbshipit-source-id: ad0afa2d44d7a713409ac75bab599a35f5cf1a87
Summary: This adds debug subcommands metadata and verify in the Filestore. Those respectively output the metadata for a file verify that the file is reachable through all aliases.
Reviewed By: ahornby
Differential Revision: D16621789
fbshipit-source-id: 4a2156bfffb9d9641ce58f6d5f691364ba9dc145
Summary:
Update blobstore_healer handling of unknown stores to re-queue and delete original entries.
Make sure it we still make progress in the case where there are lots of unknown blobstore entries on the queue.
Previous diff in stack took the approach of not deleting, which could keep loading and logging same entries if there were more than blobstore_sync_queue_limit of them. Better to reinsert with new timestamp and delete old entries.
Reviewed By: krallin
Differential Revision: D16599270
fbshipit-source-id: efa3e5602f0ab3a037d0534e1fe8e3d42fbb52e6
Summary: make blob store healer preserve queue entries for unknown blobstores rather than erroring
Reviewed By: ikostia
Differential Revision: D16586816
fbshipit-source-id: 3d4987a95adcddd0329b9ededdf95887aa11286e
Summary:
Add healer logic to fetch from all source blobstores on the queue
Add tests for the healer queue state including put failures
Reviewed By: krallin
Differential Revision: D16549013
fbshipit-source-id: 6aa55b3cb2ed7fa9a1630edd5bc5b2ad2c6f5011
Summary:
Fixes for the handling of blobstores after heal:
1. If all blobstores are successfully healed for a key, no need to requeue it
2. Where all heal puts fail, make sure we requeue with at least the original source blobstore we loaded the blob from
3. When we do write to the queue, write with all blobstore ids where we know we have good data, so that when it is read later it is not considered missing.
Reviewed By: krallin
Differential Revision: D15911853
fbshipit-source-id: 1c81ce4ec5f975e5230b27934662e02ec515cb8f
Summary: make blobstore_healer auto-heal source blobstores found to be missing data so long as at least one other source blobstore from the queue has the data for the missing key
Reviewed By: krallin
Differential Revision: D16464895
fbshipit-source-id: 32549e58933f39bb20c173caf02a35c91123fe8d
Summary: Since we're only running a single healer in the process for a single blobstore, its easy to bound the concurrency by limiting it to the number of entries we deal with at once. As a result, we don't need a separate mechanism to do overall control.
Reviewed By: StanislavGlebik
Differential Revision: D15912818
fbshipit-source-id: 3087b88cfdfed2490664cd0df10bd6f126267b83
Summary: Basically notes I took for myself to truely understand the code.
Reviewed By: StanislavGlebik
Differential Revision: D15908406
fbshipit-source-id: 3f21f7a1ddce8e15ceeeffdb5518fd7f5b1749c4
Summary:
Allow blobstore_healer to be directly configured to operate on a blobstore.
This makes two changes:
- Define which blobstore to operate on defined in storage.toml (doesn't
currently support server.toml-local storage configs)
- Only heal one blobstore at a time. We can run multiple separate instances of the
healer to heal multiple blobstores.
Reviewed By: HarveyHunt
Differential Revision: D15065422
fbshipit-source-id: 5bc9f1a16fc83ca5966d804b5715b09d359a3832
Summary: Update populate_healer to act directly on a blobstore config rather than indirectly via a repo config.
Reviewed By: StanislavGlebik
Differential Revision: D15065424
fbshipit-source-id: 638778a61283dc9ed991c49936a21d02b8d2e3f3
Summary:
The healer is a blobstore-level operation, which is orthogonal to the concept of a repo; therefore, there should be no mention of repoid in any of the healer's structures or tables.
For now this leaves the schema unmodified, and fills the repoid with a dummy value (0). We can clean that up later.
Reviewed By: lukaspiatkowski, HarveyHunt
Differential Revision: D15051896
fbshipit-source-id: 438b4c6885f18934228f43d85cdb8bf2f0e542f1
Summary: RepositoryId shouldn't leak into the blobstore layer. This leaves repoid in the schema, but just populates it with a dummy value (0). We can clean up the schema and this code in a later diff.
Reviewed By: StanislavGlebik
Differential Revision: D15021285
fbshipit-source-id: 3ecb04a76ce74409ed0cced3d2a0217eacd0e2fb
