Summary: Add --drain-only option to blobstore_healer so we can drain a queue without healing when required.
Reviewed By: ikostia
Differential Revision: D17111436
fbshipit-source-id: e2fba4914e9699a4836933d0f35def015cd3dadc
Summary: Log the region with max replication lag as well as the lag size
Reviewed By: StanislavGlebik
Differential Revision: D17097266
fbshipit-source-id: c449167b09dbd2cdce7d65bfc956e7ccf59b2415
Summary:
blobstore_healer performance improvements
* speed up delete by using bulk delete
* speed up rep lag check by only doing it once if lag within bounds
* log some stats from each healer chunk run to help diagnose issues
* dont log the full storage config on each log line ( this was making it hard to read the logs )
Reviewed By: StanislavGlebik
Differential Revision: D17093809
fbshipit-source-id: 34299ad1835c5548b9502ad9d5a3e18eff1ab95b
Summary:
Utility to backfill derived data:
- currently only support `unodes`
Reviewed By: krallin
Differential Revision: D16827897
fbshipit-source-id: aa0bb8ae4fe895233e3f11775b3432184afb37f9
Summary:
This is a part of an effort to move mercurial related stuff into mercurial crate:
- `BlobManifest|HgBlobEntry` -> `mercurial/types/blobs`
- helper `Id` type is no longer required and `BlobManifeset` can be loaded with `HgManifesetId::load`
- `BlobManifest` implements `HgManifest` and typed `Manifest` traits
Reviewed By: StanislavGlebik
Differential Revision: D16984745
fbshipit-source-id: 33d4eb9c33137258956917328efa1f5ec6737ee9
Summary:
Add two helpful commands:
1) Fetch the phase of a commit
2) List all public commits
Reviewed By: ikostia
Differential Revision: D16830576
fbshipit-source-id: 03f503cb30a7f150ea383d62fb71913dd2b93e6e
Summary:
This updates the `mononoke_admin` filenodes command to allow showing filenodes
by path + ID in addition to revision + path. This makes it possible to locate
the linknode given a filenode ID but no revision.
This updates the filenodes CLI command a little bit (so if you want the old
command you need to add `by-revision`), but this should be fine since this is
a command that a human will normally be calling.
Reviewed By: farnz
Differential Revision: D16936628
fbshipit-source-id: 1079ad580b235076739c92d0c1f3c9bfc80db4c1
Summary: This adds a command that allows importing a set of LFS blobs into Mononoke by streaming them out of another storage location (e.g. Dewey).
Reviewed By: HarveyHunt
Differential Revision: D16917806
fbshipit-source-id: 4917d56e11a187c89e00c23a32c6e791b351f8ef
Summary:
We need this to accomodate `blobimport`'s needs of using (opening(D16914875)
and creating(this)) unredacted repos.
Note: this is nearly identical to D16914875.
Reviewed By: StanislavGlebik
Differential Revision: D16915557
fbshipit-source-id: 6b522e6fcee4582870df27cf66e8ad55e91235fe
Summary:
Currently blobimport fails if it tries to import a redacted blob. We should
only enforce redaction in the scenarios where end users could see the data.
Internal imports should be fine to proceed.
As a separate note, problems like this would probably go away when redaction
would be moved to the `FileStore` level.
Reviewed By: krallin
Differential Revision: D16914875
fbshipit-source-id: e531a6248f66b47de87c7aedcbc928bb7ad4ebc9
Summary:
This is a mechanical part of rename, does not change any commit messages in
tests, does not change the scuba table name/config setting. Those are more
complex.
Reviewed By: krallin
Differential Revision: D16890120
fbshipit-source-id: 966c0066f5e959631995a1abcc7123549f7495b6
Summary:
mononoke_types doesn't look like a good crate for it. Let's move to blobstore
crate instead
Reviewed By: krallin
Differential Revision: D16914142
fbshipit-source-id: 57602c5a644adc87beea0cbc92beeb0d901d5873
Summary:
We now have two Manifest and two Entry traits and it makes it very confusing.
The old Manifest trait (and related EmptyManifest struct and Entry trait) are
mercurial-specific, while new Manifest trait is generic over all kinds of
manifests (unodes, mercurial etc).
In ideal state we should have only new Manifest trait, but it'd require quite a
big refactoring. In short term let's at least rename Manifest to HgManifest and
Entry to HgEntry to make distinction clearer.
Reviewed By: krallin
Differential Revision: D16886032
fbshipit-source-id: 80e3999fae3e89347448579caf7aafec56705500
Summary:
This is the first diff in the stack that removes ugly hack `struct Id` that we
used in derived data (see more details about that later in the stack).
This diff reverses the dependency between blobstore crate and mononoke_types
crate. Previously blobstore depended on mononoke_types, and blobstore also had
Loadable/Storable traits. That means that we can't implement Loadable/Storable
traits for types from mononoke_types - we can't even import them without
creating cyclic dependency between crates! And we'll use Loadable in
mononoke_types later in the stack.
This diff reverses the dependency and moves Loadable/Storable in mononoke_types
Reviewed By: krallin
Differential Revision: D16884649
fbshipit-source-id: 20753f3e62980be7bb6a04ab9d19980bc459b40e
Summary:
There are a couple of functions that take a reference to a blobstore and then
immediately clone it, in order to pass ownership to another function.
Modify these functions to take ownership.
Reviewed By: farnz
Differential Revision: D16859228
fbshipit-source-id: fd7b0ae5af2262601db2201aa510d3099f37ced5
Summary:
Create a new binary that can be used to rechunk files content using the filestore.
The binary accepts multiple filenodes, that it will then go and rechunk using the filestore
config provided to it.
Reviewed By: krallin
Differential Revision: D16802701
fbshipit-source-id: d7c05729f5072ff2925bbc90cdd89fcfed56bba2
Summary:
We have a FetchKey object in the filestore that serves as both the thing you use to ask for reads and to write aliases. One downside of this is that the same enum is used for reads and writes, and that means that you could potentially accidentally try to write an alias for the "canonical" fetch key, which would effectively overwrite your content.
That feels bad, and the underlying reason is that the typing doens't really reflect the reality of what we are trying to do here, which is to support a set of aliases for pieces of content, and expose an enum that allows clients to access content by alias or canonical ID.
This patch updates the type hierarchy to be more in line with what we are trying to do (note that you can no longer store a canonical fetch key). One downside is that it makes the API a bit more verbose, since now you need to say e.g. `FetchKey::Aliased(Alias::Sha256(...))` instead of just `FetchKey::Sha256`. I think the tradeoff is reasonable, but I'm curious to hear other folk's thoughts.
Reviewed By: farnz
Differential Revision: D16764582
fbshipit-source-id: 00c596d1370b52ef2aea04ac44cdcd19b7a2c92a
Summary:
We may need to inspect the list of blacklisted files for a given commit.
This command is called to do so.
Reviewed By: krallin
Differential Revision: D16643467
fbshipit-source-id: ad389c4b9262ffa9ee6ea9b792b4866a23bebaa5
Summary: Clean up non-test usage of CoreContext::test_mock left over from T37478150
Reviewed By: farnz
Differential Revision: D16804838
fbshipit-source-id: f420b8186557a42e9b6c78437c0fb76c9a343b31
Summary:
Start moving mercurial related stuff to `mercurial` directory:
- rename `mercurial` to `mercurial_revlog` and moved to `/mercurial/revlog`
- move `mercurial_types` to `/mercurial/types`
- move `mercurial_bundles` to `/mercurial/bundels`
Reviewed By: farnz
Differential Revision: D16783728
fbshipit-source-id: 79cf1757bb7cc84a6273a4a3c486242b1ef4cd00
Summary:
This resolves the list of files at a given commit to a list of `ContentId`s
and unblacklists all of those `ContentId`s.
Note 1: This means that these particular versions of files will be
unblacklisted in all commits which reference them.
Note 2: There's no auditing of this admin command invokation, if the
need for one arises, we should add it externally.
Reviewed By: krallin
Differential Revision: D16643436
fbshipit-source-id: 3af488e3e4d748027605587ab1e08c0abd0da109
Summary:
This is done to make `remove` and `list` be implementable in terms of the same
boilerplate with some different closures.
NB: I don't particularly like how this refactoring looks, but I do think it
looks better than copy/pasting the code and I think it is of no major
importance, given that this is an admin command.
Reviewed By: krallin
Differential Revision: D16643437
fbshipit-source-id: 7f64b112eda1ce963dffed897bf2c9fa42234d79
Summary: In the following diffs, I will implement the actual logic.
Reviewed By: krallin
Differential Revision: D16639720
fbshipit-source-id: f42b91476035343a6f00b11e4921e26836b1a574
Summary: This adds support for running benchmarks using sqlblob. I've refactored the command a little bit so that destinations are subcommands now.
Reviewed By: HarveyHunt
Differential Revision: D16733238
fbshipit-source-id: 963365ed3492cddf1c7302f18fc44148d9c1ec16
Summary: add support for fb303 and ods stats to cmdlib. use for blobstore_healer to monitor healer status
Reviewed By: krallin
Differential Revision: D16671526
fbshipit-source-id: 8f95fc097133f2fe697ea3cf70fd9d0fa3b3649c
Summary:
This command allows a file to be uploaded to the blobstore using
the filestore - which allows for chunking if required.
Additionally, port the other filestore commands to use logger
rather than println!().
Reviewed By: krallin
Differential Revision: D16712167
fbshipit-source-id: 0fad13c29035a550f8caee7a02d37071eb87cd73
Summary: This updates blobimport to a newer Rust-2018-style. Rust 2018 is nicer.
Reviewed By: StanislavGlebik
Differential Revision: D16667043
fbshipit-source-id: a217cbef0337e7d7cf288f972e9b4af0340a8e0b
Summary:
This updates blobimport to avoid using a per-changeset limit for the number of blob uploads (which used to be 100), and instead to use a global blob upload limit, and a global LFS import limit.
This allows for finer-grained control over the resource utilization (notably in terms of Manifold quota) of blobimport.
While I was in there, I also eliminated an `Arc<BlobRepo>` that was laying around, sicne we don't use those anymore.
Reviewed By: HarveyHunt
Differential Revision: D16667044
fbshipit-source-id: 9fc2f347969c7ca9472ce8dd3d4e2f1acb175b66
Summary: This updates blobimport to allow specifying a different commit limit than the default of 100. This makes it a little more convenient when tuning limits.
Reviewed By: HarveyHunt
Differential Revision: D16648558
fbshipit-source-id: 2e8c8496487a79bec84ec964302b1d6e871caf2a
Summary:
Sometimes, our Mononoke CLI tests report errors in mode/opt where they don't
actually output error messages at all when returning an error. Those error
messages actually get buffered by our logger, and are normally emitted when
said logger is dropped.
However, I have a suspicion that by calling process::exit(), we're not giving
Rust a chance to drop the logger (because that eventually just calls the exit
syscall), and as a result our buffer isn't getting flushed.
This patch attempts to fix this by removing all calls to process::exit in the
admin CLI, and instead returning an exit code from main, which I hope will
allow main to drop everything before it returns.
More broadly speaking, I don't think it's very good practice to exit from all
over the place in the admin CLI, so this fixes that :)
Reviewed By: HarveyHunt
Differential Revision: D16687747
fbshipit-source-id: 38e987463363e239d4f9166050a8dd26a4bef985
Summary:
Address nits from D16599270:
* use map instead of and_then
* use &[T] instead of &Vec[T]
Reviewed By: StanislavGlebik
Differential Revision: D16666838
fbshipit-source-id: ad0afa2d44d7a713409ac75bab599a35f5cf1a87
Summary: This adds debug subcommands metadata and verify in the Filestore. Those respectively output the metadata for a file verify that the file is reachable through all aliases.
Reviewed By: ahornby
Differential Revision: D16621789
fbshipit-source-id: 4a2156bfffb9d9641ce58f6d5f691364ba9dc145
Summary:
Update blobstore_healer handling of unknown stores to re-queue and delete original entries.
Make sure it we still make progress in the case where there are lots of unknown blobstore entries on the queue.
Previous diff in stack took the approach of not deleting, which could keep loading and logging same entries if there were more than blobstore_sync_queue_limit of them. Better to reinsert with new timestamp and delete old entries.
Reviewed By: krallin
Differential Revision: D16599270
fbshipit-source-id: efa3e5602f0ab3a037d0534e1fe8e3d42fbb52e6
Summary: make blob store healer preserve queue entries for unknown blobstores rather than erroring
Reviewed By: ikostia
Differential Revision: D16586816
fbshipit-source-id: 3d4987a95adcddd0329b9ededdf95887aa11286e
Summary:
Add healer logic to fetch from all source blobstores on the queue
Add tests for the healer queue state including put failures
Reviewed By: krallin
Differential Revision: D16549013
fbshipit-source-id: 6aa55b3cb2ed7fa9a1630edd5bc5b2ad2c6f5011
Summary:
Fixes for the handling of blobstores after heal:
1. If all blobstores are successfully healed for a key, no need to requeue it
2. Where all heal puts fail, make sure we requeue with at least the original source blobstore we loaded the blob from
3. When we do write to the queue, write with all blobstore ids where we know we have good data, so that when it is read later it is not considered missing.
Reviewed By: krallin
Differential Revision: D15911853
fbshipit-source-id: 1c81ce4ec5f975e5230b27934662e02ec515cb8f
Summary: make blobstore_healer auto-heal source blobstores found to be missing data so long as at least one other source blobstore from the queue has the data for the missing key
Reviewed By: krallin
Differential Revision: D16464895
fbshipit-source-id: 32549e58933f39bb20c173caf02a35c91123fe8d
Summary: Since we're only running a single healer in the process for a single blobstore, its easy to bound the concurrency by limiting it to the number of entries we deal with at once. As a result, we don't need a separate mechanism to do overall control.
Reviewed By: StanislavGlebik
Differential Revision: D15912818
fbshipit-source-id: 3087b88cfdfed2490664cd0df10bd6f126267b83
Summary: Basically notes I took for myself to truely understand the code.
Reviewed By: StanislavGlebik
Differential Revision: D15908406
fbshipit-source-id: 3f21f7a1ddce8e15ceeeffdb5518fd7f5b1749c4
Summary:
Allow blobstore_healer to be directly configured to operate on a blobstore.
This makes two changes:
- Define which blobstore to operate on defined in storage.toml (doesn't
currently support server.toml-local storage configs)
- Only heal one blobstore at a time. We can run multiple separate instances of the
healer to heal multiple blobstores.
Reviewed By: HarveyHunt
Differential Revision: D15065422
fbshipit-source-id: 5bc9f1a16fc83ca5966d804b5715b09d359a3832
Summary: Update populate_healer to act directly on a blobstore config rather than indirectly via a repo config.
Reviewed By: StanislavGlebik
Differential Revision: D15065424
fbshipit-source-id: 638778a61283dc9ed991c49936a21d02b8d2e3f3
Summary:
The healer is a blobstore-level operation, which is orthogonal to the concept of a repo; therefore, there should be no mention of repoid in any of the healer's structures or tables.
For now this leaves the schema unmodified, and fills the repoid with a dummy value (0). We can clean that up later.
Reviewed By: lukaspiatkowski, HarveyHunt
Differential Revision: D15051896
fbshipit-source-id: 438b4c6885f18934228f43d85cdb8bf2f0e542f1
Summary: RepositoryId shouldn't leak into the blobstore layer. This leaves repoid in the schema, but just populates it with a dummy value (0). We can clean up the schema and this code in a later diff.
Reviewed By: StanislavGlebik
Differential Revision: D15021285
fbshipit-source-id: 3ecb04a76ce74409ed0cced3d2a0217eacd0e2fb
Summary:
This is useful to inspect the Mercurial filenodes in Mononoke, like in S183272.
For example, I intend to use this subcommand to verify how well the future linknode healing works.
Reviewed By: krallin
Differential Revision: D16621516
fbshipit-source-id: 4266f85bce29b59072bf9c4f3e63777dae09a4f1
Summary: This is needed in the following diff.
Reviewed By: krallin
Differential Revision: D16621517
fbshipit-source-id: 5a50cae7c8b761d7578bcbe5caf302a5ee2578a3
Summary: This updates benchmark_filestore to allow testing with caches (notably, Memcache & Cachelib). It also reads twice now, which is nice for caches that aren't filled by us (e.g. Manifold CDN).
Reviewed By: ahornby
Differential Revision: D16584952
fbshipit-source-id: 48ceaa9f2ea393626ac0e5f3988672df020fbc28
Summary:
NOTE: This changes our file storage format. It's fine to do it now since we haven't started enabling chunking yet (see: D16560844).
This updates the Filestore's chunking to store chunks as their own entity in Thrift, as opposed to have them be just FileContents.
The upside of this approach is that this we can't have an entity that's both a File and a Chunk, which means:
- We don't have to deal with recursive chunks (since, unlike Files, Chunks can't contain be pointers to other chunks).
- We don't have to produce metadata (forward mappings and backmappings) for chunks (the reason we had to produce it was to make sure we wouldn't accidentally produce inconsitent data if the upload for one of our chunks happened to have been tried as a file earlier and failed).
Note that this also updates the return value from the Filestore to `ContentMetadata`. We were using `Chunk` before there because it was sufficient and convenient, but now that `Chunk` no longer contains a `ContentId`, it no longer is convenient, so it's worth changing :)
Reviewed By: HarveyHunt
Differential Revision: D16598906
fbshipit-source-id: f6bec75d921f1dea3a9ea3441f57213f13aeb395
Summary:
This allows running blobimport multiple times over the same path locally (with a blob files storage, for example), which is how we use it in prod (but there we don't use the file blobstore so it works).
This is helpful when playing around with local changes to blobimport.
Reviewed By: HarveyHunt
Differential Revision: D16580697
fbshipit-source-id: 4a62ff89542f67ce6396948c666244ef40ffe5e7
Summary:
This is a very trivial patch simply intended to make it easier to safely roll
out the Filestore. With this, we can roll out the Filestore with chunking fully
disabled and make sure all hosts know how to read chunked data before we turn
it on.
Reviewed By: aslpavel, HarveyHunt
Differential Revision: D16560844
fbshipit-source-id: 30c49c27e839dfb06b417050c0fcde13296ddade
Summary: This adds a filestore benchmark that allows for playing around with Filestore parameters and makes it easier to measure performance improvements.
Reviewed By: aslpavel
Differential Revision: D16559941
fbshipit-source-id: 50a4e91ad07bf6f9fc1efab14aa1ea6c81b9ca27
Summary:
This adds support for LFS filenodes in blobimport. This works by passing a `--lfs-helper` argument, which should be an executable that can provide a LFS blob's contents on stdout when called with the OID and size of a LFS blob. My thinking is to `curl` directly from Dewey when running this in prod.
Note that, as of this change, we can blobimport LFS files, but doing so might be somewhat inefficient, since we'll roundtrip the blobs to our filestore, then generate filenodes. For now, however, I'd like to start with this so we can get a sense of whether this is acceptable performance-wise.
Reviewed By: farnz
Differential Revision: D16494241
fbshipit-source-id: 2ae032feb1530c558edf2cfbe967444a9a7c0d0f
Summary:
NOTE: This isn't 100% complete yet. I have a little more work to do around the aliasverify binary, but I think it'll make sense to rework this a little bit with the Filestore anyway.
This patch incorporates the Filestore throughout Mononoke. At this time, what this means is:
- Blobrepo methods return streams of `FileBytes`.
- Various callsites that need access to `FileBytes` call `concat2` on those streams.
This also eliminates the Sha256 aliasing code that we had written for LFS and replaces it with a Filestore-based implementation.
However, note that this does _not_ change how files submitted through `unbundle` are written to blobstores right now. Indeed, those contents are passed into the Filestore through `store_bytes`, which doesn't do chunking. This is intentional since it lets us use LFS uploads as a testbed for chunked storage before turning it on for everything else (also, chunking those requires further refactoring of content uploads, since right now they don't expect the `ContentId` to come back through a Future).
The goal of doing it this way is to make the transition simpler. In other words, this diff doesn't change anything functionally — it just updates the underlying API we use to access files. This is also important to get a smooth release: it we had new servers that started chunking things while old servers tried to read them, things would be bad. Doing it this way ensures that doesn't happen.
This means that streaming is there, but it's not being leveraged just yet. I'm planning to do so in a separate diff, starting with the LFS read and write endpoints in
Reviewed By: farnz
Differential Revision: D16440671
fbshipit-source-id: 02ae23783f38da895ee3052252fa6023b4a51979
Summary:
NOTE: This diff updates Thrift serialization for chunked files. Normally, this wouldn't be OK, but we never stored anything in this format, so that's fine.
This updates the Filestore to rebuild backmappings on read. The rationale for this is that since we store backmappings *after* writing FileContents, it is possible to have FileContents that logically exist but don't have backmappings. This should be a fairly exceptional case, but we can handle it by recomputing the missing backmappings on the fly.
As part of this change, I'm updating our Thrift serialization for chunked files and backmappings (as mentioned earlier, this is normally not a good idea, but it should be fine here):
- Chunked files now contain chunk lengths, which lets us derive offsets as well as total size for a chunked file.
- Backmappings don't contain the size anymore (since FileContents contains it now).
This was necessary because we need to have a file's size in order to recompute backmappings.
In this change, I also updated GitSha1 to stop embedding its blob type and length. This lets us reconstruct a strongly-typed GitSha1 object from just the hash (and was therefore necessary to avoid duplicating file size into backmappings), which seems sufficient at this stage (and if we do need the size, we can obtain it through the Filestore by querying the file).
Reviewed By: aslpavel
Differential Revision: D16440676
fbshipit-source-id: 23b66caf40fde2a2f756fef89af9fe0bb8bdadef
Summary:
This adds support for chunking in the Filestore. To do so, this reworks writes to be a 2 stage process:
- First, you prepare your write. This puts everything into place, but doesn't upload the aliases, backmappings, and the logical FileContents blob representing your file. If you're uploading a file that fits in a single chunk, preparing your writes effectively makes no blobstore changes. If you're uploading chunks, then you upload the individual chunks (those are uploaded essentially as if they were small files).
- Then, you commit your write. At that point, prepared gave you `FileContents` that represent your write, and a set of aliases. To commit your write, you write the aliases, then the file contents, and then a backmapping.
Note that we never create hierarchies when writing to the Filestore (i.e. chunked files always reference concrete chunks that contain file data), but that's not a guarantee we can rely on when reading. Indeed, since chunks and files are identical as far as blobstore storage is concerned, we have to handle the case where a chunked file references chunks that are themselves chunked (as I mentioned, we won't write it that way, but it could happen if we uploaded a file, then later on reduced the chunk size and wrote an identical file).
Note that this diff also updates the FileContents enum and adds unimplemented methods there. These are all going away later in this stack (in the diff where I incorporate the Filestore into the rest of Mononoke).
Reviewed By: StanislavGlebik
Differential Revision: D16440678
fbshipit-source-id: 07187aa154f4fdcbd3b3faab7c0cbcb1f8a91217
Summary:
Currently, we require in the MononokeId trait that the Id know how to recompute itself from a value (and this has to return `Self, not `Result<Self>`, so it also can't fail).
This isn't ideal for a few reasons:
- It means we cannot support the MononokeId trait (and therefore typed blobstore fetches) for things that aren't purely a hash of their contents
- One workaround (which jsgf had implemented for ContentAliasBackmappingId) is to peek at the contents and embed an ID in there. A downside of this approach is that we end up parsing the content twice when loading a value from a blobstore (once to get the ID, and once to get the contents later).
- It's a little inefficient in general, since it means we recompute hashes of things we just fetched just to know what their hash should be (which we often proceed to immediately discard afterwards). This could be worth doing if we verified that the ID we got is the ID wee wanted, but we don't actually do this right now.
Reviewed By: StanislavGlebik
Differential Revision: D16486775
fbshipit-source-id: a75864eed3efa7e07b8bf642dbac3ada00cadc7c
Summary:
It's not cool to include the `flat/` prefix into the queue. Not cool at all.
It's not part of the actual blobstore key.
Let's make the populator cool again.
Reviewed By: ahornby
Differential Revision: D16494119
fbshipit-source-id: 8d3a34271d19ab0531cbb59e56b7a1ac04f66b07
Summary:
Before this change, we would always include the shard id in our mysql-related fb303 counters. This is not perfect for two reasons:
- the the xdb blobstore we have 4K shards and 24 counters, so we were reporting 96K counters in total
- we rarely care about per-counter metrics anyway, since in most cases all queries are uniformly distributed
Therefore, let's change this approach to not use per-shard counters and use per-shardmap ones (when sharding is involved).
Reviewed By: krallin
Differential Revision: D16360591
fbshipit-source-id: b2df94a3ca9cacbf5c1f328b48e87b48cd18287e
Summary:
Improve memory usage on mononoke startup and reduce number of small allocations. Done via:
* Pre-size CHashMap used by SkiplistEdgeMapping, working around the 4x multiplier and lack of load factor awareness in CHashMap::with_capacity
* Add a SingleEdge optimization to SkiplistNodeType so as to save vector overhead in the case of one edge ( this is the common case )
* Size the HashMap in rust thrift deserialization with HashMap::with_capacity
Reviewed By: krallin
Differential Revision: D16265993
fbshipit-source-id: 99c3a7149493d824a3c00540bc5557410d0273fc
Summary:
My previous diff (D16327788), despite claiming a 1000x improvement, was merely a ~220x
improvement. This diff is some tweaking with numbers, whihc brings about
another 1.3x improvement, bringing the total one to be 290x.
Reviewed By: krallin
Differential Revision: D16356932
fbshipit-source-id: 3d3f0c844eec9866217cf5a57f285fe8a56152de
Summary:
In earlier diffs in this stack, I updated the callsites that reference XDB tiers to use concrete &str types (which is what they were receiving until now ... but it wasn't spelled out as-is).
In this diff, I'm updating them to use owned `String` instead, which lets us hoist up `to_string()` and `clone()` calls in the stack, rather than pass down reference only to copy them later on.
This allows us to skip some unnecessary copies. Tt turns out we were doing quite a few "turn this String into a reference, pass it down the stack, then turn it back into a String".
Reviewed By: farnz
Differential Revision: D16260372
fbshipit-source-id: faec402a575833f6555130cccdc04e79ddb8cfef
Summary:
This implements and uses the `add_many` method of the blob healer queue. This method allows us to do batched adds, which in turn allows us to use `chunks` on Manifold iteration.
NB 1: I deliberately removed control symbols form progress print message. If we only print it on the same line, we loose it when the job crashes.
NB 2: I deliberately use range of `entries[0]`, as I want to pessimistically restart from the earliest in case of a failure.
Reviewed By: krallin
Differential Revision: D16327788
fbshipit-source-id: 8d9f3cf85ee7cbca657a8003a787b5ea84a1b9b0
Summary:
Instantiating a new DB connection may require remote calls to be made to e.g. Hipster to allocate a new certificate (this is only the case when connecting to MySQL).
Currently, our bindings to our underlying DB locator make a blocking call to pretend that this operaiton is synchronous: https://fburl.com/ytmljxkb
This isn't ideal, because this call might actually take time, and we might also occasionally want to retry it (we've had issues in our MySQL tests with acquiring certificates that retrying should resolve). Running this synchronously makes doing so inefficient.
This patch doesn't update that, but it fixes everything on the Rust side of things to stop expecting connections to return a `Result` (and to start expecting a Future instead).
In a follow up diff, I'll work on making the changes in common/rust/sql to start returning a Future here.
Reviewed By: StanislavGlebik
Differential Revision: D16221857
fbshipit-source-id: 263f9237ff9394477c65e455de91b19a9de24a20
Summary: This helps us investigate the workings of the (upcoming) xdb blobstore.
Reviewed By: StanislavGlebik
Differential Revision: D16208770
fbshipit-source-id: 33542f3d34a5c8b4287bb14b0aa97d3802b0e0d6
Summary: Tool that can run generation of hg manifests from bonsai, and compare them with already recorded one.
Reviewed By: StanislavGlebik
Differential Revision: D16131666
fbshipit-source-id: 8c59ec0fdf03431af85c886c22ee269cecc8e8c4
Summary:
This means that we can later destructure and rebuild the `CensoredBlob` struct. If it is only
possible to construct it from a talbe name, there's no way to get this name
back from a destructured struct.
Note: I do this to eventually rebase D16163225 on top of this one, then be able implement `in_memory_writes_READ_DOC_COMMENT` in terms of destructuring of the existing `CensoredBlob`.
Reviewed By: krallin
Differential Revision: D16180745
fbshipit-source-id: e587675d444bf53f0bd25f01602503abad9d7d97
Summary:
Report to Scuba whenever someone tries to access a blobstore which is blacklisted. Scuba reporting is done for any `get` or `put` method call.
Because of the possible overload - given the high number of requests mononoke receives and that CensoredBlobstore make the verification before we add the caching layer for blobstores - I considered reporting at most one bad request per second. If multiple requests to blacklisted blobstores are made in less than one second, only the first request should be reported. Again, this is not the best approach (to not report all of them), but performance wise is the best solution.
NOTE: I also wrote an implementation using `RwLock` (instead of the current `AtomicI64`), but atomic variables should be faster than using lockers so I gave up on that idea.
Reviewed By: ikostia, StanislavGlebik
Differential Revision: D16108456
fbshipit-source-id: 9e5338c50a1c7d15f823a2b8af177ffdb99e399f
Summary: With these fixes running the healer for `repo_id` 502 seems to work.
Reviewed By: aslpavel
Differential Revision: D16140872
fbshipit-source-id: bf2caf7428a74cf14cd61ea8b52b7e5822432646
Summary: Log when a `censored file` is accessed via calling `get` or `put` method.
Reviewed By: StanislavGlebik
Differential Revision: D16089138
fbshipit-source-id: 0d2f0e21e7afcad8783be7587e6b676af20ba029
Summary:
`local_instances` option was used to create fileblobstore or sqlite blobstore.
Now we use mononoke config for this purpose. Since this option is no longer
useful let's delete it
Reviewed By: krallin
Differential Revision: D16120065
fbshipit-source-id: 375a168b27e7f2cf1a6a77f487c5e013f9004546
Summary:
`MultiplexedBlobstore` can hide errors up until we suddenly lose availabilty of the wrong blobstore. Introduce an opt-in `ScrubBlobstore`, which functions as a `MultiplexedBlobstore` but checks that the combination of blobstores and healer queue should result in no data loss.
Use this new blobstore in the blobrepo checker, so that we can be confident that data is safe.
Later, this blobstore should trigger the healer to fix "obvious" problems.
Reviewed By: krallin
Differential Revision: D15353422
fbshipit-source-id: 83bb73261f8ae291285890324473f5fc078a4a87
Summary:
Added an option to control for which repositories should censoring be
enabled or disabled. The option is added in `server.toml` as `censoring` and it
is set to true or false. If `censoring` is not specified, then the default
option is set to true ( censoring is enabled).
By disabling `censoring` the verification if the key is blacklisted or not is
omitted, therefor all the files are fetchable.
Reviewed By: ikostia
Differential Revision: D16029509
fbshipit-source-id: e9822c917fbcec3b3683d0e3619d0ef340a44926
Summary:
Subcommand `blacklist` to censor multiple files. It stores the censored keys of the blacklisted files in table censored_contents.
The subcommand takes as arguments
- list of files
- commit hash
- string, representing the task ID for censoring the files (the task should contain the reason why the files is censored)
Reviewed By: ikostia
Differential Revision: D15939450
fbshipit-source-id: 7261ab2358cc4905d61a14f354de2949a2a94e7c
Summary:
CensoredBlob was placed between Blobstore and PrefixBlobstore. I moved CensoredBlob, so that now it is a wrapper for PrefixBlobstore. This means the key is compared before appending the `repoid` to the key.
By moving CensoredBlob on top of PrefixBlobstore, it provides better isolation for the existing blobstores. This way CensoredBlob does not interact with the underlaying layers and future changes of those layers will, most probably, not impact CensoredBlob's implementation.
Reviewed By: ikostia
Differential Revision: D15900610
fbshipit-source-id: 391594355d766f43638f3152b56d4e9acf49af32
Summary: We are copying `MPath` a lot, this change should reduce amount of allocations and memory reuse for `MPath`
Reviewed By: farnz
Differential Revision: D15939495
fbshipit-source-id: 8da8f2c38f7b46f27d0df661210c9964aed52101
Summary: This adds a command in Mononoke Admin to list all bookmarks.
Reviewed By: ikostia
Differential Revision: D15737660
fbshipit-source-id: 5d4e649adf09c5f601b149a87515707f3d88b203
Summary: We have the same pattern 3 times, and I have at least two more instances of this pattern to add. Factor it out.
Reviewed By: StanislavGlebik
Differential Revision: D15202663
fbshipit-source-id: 9c0de8ccef71964e65389e0bf0f2b0fc383f7c1d
Summary: This help text was wrong. This fixes it.
Reviewed By: HarveyHunt
Differential Revision: D15555767
fbshipit-source-id: 5202d46a8527c9e44c5588d52f7425d3efb38898
Summary:
As part of adding support for infinitepush in Mononoke, we'll include additional server-side metadata on Bookmarks (specifically, whether they are publishing and pull-default).
However, we do use the name `Bookmark` right now to just reference a Bookmark name. This patch updates all reference to `Bookmark` to `BookmarkName` in order to free up `Bookmark`.
Reviewed By: StanislavGlebik
Differential Revision: D15364674
fbshipit-source-id: 126142e24e4361c19d1a6e20daa28bc793fb8686
Summary:
`Phases` currently have very ugly API, which is constant source of confusion. I've made following changes
- only return/cache public phases
- do not require `public_heads` and always request them from `BlobRepo::get_bonsai_heads_maybe_stale`
- consolidated `HintPhases` and `SqlPhases`
- removed `Hint` from types which does not carry any meaning
- fixed all effected callsites
Reviewed By: StanislavGlebik
Differential Revision: D15344092
fbshipit-source-id: 848245a58a4e34e481706dbcea23450f3c43810b
Summary: Let's make it a bit more manageable.
Reviewed By: StanislavGlebik
Differential Revision: D15407978
fbshipit-source-id: 490e4d48c36349cf65171f38df0ef0372883281d
Summary: Exploit the fact that database parents are faster than blobstore reads to get more concurrency.
Reviewed By: aslpavel
Differential Revision: D15202668
fbshipit-source-id: e2f35f80531e4a558b40b969cb6d7b9434919a9f
Summary: I have three similar chunks of boilerplate code; there's two more that I plan to write. Factor it out.
Reviewed By: aslpavel
Differential Revision: D15202667
fbshipit-source-id: 16b7b2d44852679d33b44da57c003e7833539f00
Summary: We can't recreate HgChangesets from Bonsai cleanly - we have imported with broken hashes created by Mercurial. Make sure that they're not lost in the checker.
Reviewed By: StanislavGlebik
Differential Revision: D15120894
fbshipit-source-id: 0eab8ad3924e961f3603dd465d87702f456be36f
Summary: When you ask a repo for parents, it goes to a database in preference to the blobstore. Compare the blobstore and database parents, to ensure they match
Reviewed By: aslpavel
Differential Revision: D15120890
fbshipit-source-id: 1123fba8c97865affe86c06965c78d70d61e3d7a
Summary:
When underlying storage layers give us trouble, we want to be able to validate that nothing is broken.
Use `BlobRepo` to fetch all the data for history of a given bookmark, so that we can be confident that we have sufficient data to not lose history.
Later diffs will add support for checking the saved Mercurial changesets, too
Reviewed By: StanislavGlebik
Differential Revision: D15084862
fbshipit-source-id: 8c7184cf1cd0d52ca2ca9c8ce1d7e97b1e51db6d
Summary: Practice commit. The behaviour will be implemented in the future
Reviewed By: ikostia
Differential Revision: D15372554
fbshipit-source-id: c9b3de81fcd2d79d34b8f8c898e28de2564cd887
Summary: I'm going to be doing some work on this file, but it's not up-to-date with rustfmt. To minimize merge conflicts and simplify diff reviews, I ran that earlier.
Reviewed By: StanislavGlebik
Differential Revision: D15364673
fbshipit-source-id: 32658629cc7ee29cdfc21be8dd526766d2dcbb0e
Summary: I'm going to be doing some work on this file, but it's not up-to-date with rustfmt. To minimize merge conflicts and simplify diff reviews, I ran that earlier.
Reviewed By: StanislavGlebik
Differential Revision: D15364672
fbshipit-source-id: dd420ed8fc625142b686d08f6459148c73891a20
Summary:
Printint id will make it easier to debug sync job problems (for example,
rewiding the latest replayed id counter)
Reviewed By: krallin
Differential Revision: D15322624
fbshipit-source-id: 5c94be9cc0dcced9df51162adb598b6498f1c749
Summary:
This change has two goals:
- Put storage configuration that's common to multiple repos in a common place,
rather than replicating it in each server.toml
- Allow tools that only operate on the blobstore level - like blobstore healing
- to be configured directly in terms of the blobstore, rather than indirectly
by using a representative repo config.
This change makes several changes to repo configuration:
1. There's a separate common/storage.toml which defines named storage
configurations (ie, a combination of a blobstore and metadata DB)
2. server.toml files can also define local storage configurations (mostly
useful for testing)
3. server.toml files now reference which storage they're using with
`storage_config = "name"`.
4. Configuration of multiplex blobstores is now explicit. Previously if a
server.toml defined multiple blobstores, it was assumed that it was a
multiplex. Now storage configuration only accepts a single blobstore config,
but that config can be explicitly a multiplexed blobstore, which has the
sub-blobstores defined within it, in the `components` field. (This is
recursive, so it could be nested, but I'm not sure if this has much value in
practice.)
5. Makes configuration parsing more strict - unknown fields will be treated as
an error rather than ignored. This helps flag problems in refactoring/updating
configs.
I've updated all the configs to the new format, both production and in
integration tests. Please review to make sure I haven't broken anything.
Reviewed By: StanislavGlebik
Differential Revision: D15065423
fbshipit-source-id: b7ce58e46e91877f4e15518c014496fb826fe03c
