sapling/cmds
Thomas Orozco ff10fff199 mononoke/filestore: introduce chunking support
Summary:
This adds support for chunking in the Filestore. To do so, this reworks writes to be a 2 stage process:

- First, you prepare your write. This puts everything into place, but doesn't upload the aliases, backmappings, and the logical FileContents blob representing your file. If you're uploading a file that fits in a single chunk, preparing your writes effectively makes no blobstore changes. If you're uploading chunks, then you upload the individual chunks (those are uploaded essentially as if they were small files).
- Then, you commit your write. At that point, prepared gave you `FileContents` that represent your write, and a set of aliases. To commit your write, you write the aliases, then the file contents, and then a backmapping.

Note that we never create hierarchies when writing to the Filestore (i.e. chunked files always reference concrete chunks that contain file data), but that's not a guarantee we can rely on when reading. Indeed, since chunks and files are identical as far as blobstore storage is concerned, we have to handle the case where a chunked file references chunks that are themselves chunked (as I mentioned, we won't write it that way, but it could happen if we uploaded a file, then later on reduced the chunk size and wrote an identical file).

Note that this diff also updates the FileContents enum and adds unimplemented methods there. These are all going away later in this stack (in the diff where I incorporate the Filestore into the rest of Mononoke).

Reviewed By: StanislavGlebik

Differential Revision: D16440678

fbshipit-source-id: 07187aa154f4fdcbd3b3faab7c0cbcb1f8a91217
2019-07-31 05:19:37 -07:00
..
admin mononoke/filestore: introduce chunking support 2019-07-31 05:19:37 -07:00
blobrepo_checker/src mononoke: remove local_instances option from cmdlib 2019-07-04 08:03:51 -07:00
blobstore_healer sql: do not use shard_id in fb303 counter names 2019-07-19 06:30:40 -07:00
bonsai_verify added new admin subcommand to verify manifest generation logic 2019-07-12 07:31:02 -07:00
tests mononoke/configlint: check locality 2019-05-21 12:25:45 -07:00
aliasverify.rs mononoke: convert DB connection instantiation to future 2019-07-16 03:17:58 -07:00
blobimport.rs mononoke: convert DB connection instantiation to future 2019-07-16 03:17:58 -07:00
configlint.rs mononoke: remove local_instances option from cmdlib 2019-07-04 08:03:51 -07:00
dumprev.rs Convert scm/mononoke to Rust 2018 2019-05-23 11:14:19 -07:00
idxdump.rs Convert scm/mononoke to Rust 2018 2019-05-23 11:14:19 -07:00
populate_healer.rs mononoke: fix populate_healer to strip the namespace prefix 2019-07-25 07:57:59 -07:00
revlogrepo.rs Apply rustfmt to all rust files 2019-07-10 19:36:32 -07:00