sapling/cmds/admin
Thomas Orozco ff10fff199 mononoke/filestore: introduce chunking support
Summary:
This adds support for chunking in the Filestore. To do so, this reworks writes to be a 2 stage process:

- First, you prepare your write. This puts everything into place, but doesn't upload the aliases, backmappings, and the logical FileContents blob representing your file. If you're uploading a file that fits in a single chunk, preparing your writes effectively makes no blobstore changes. If you're uploading chunks, then you upload the individual chunks (those are uploaded essentially as if they were small files).
- Then, you commit your write. At that point, prepared gave you `FileContents` that represent your write, and a set of aliases. To commit your write, you write the aliases, then the file contents, and then a backmapping.

Note that we never create hierarchies when writing to the Filestore (i.e. chunked files always reference concrete chunks that contain file data), but that's not a guarantee we can rely on when reading. Indeed, since chunks and files are identical as far as blobstore storage is concerned, we have to handle the case where a chunked file references chunks that are themselves chunked (as I mentioned, we won't write it that way, but it could happen if we uploaded a file, then later on reduced the chunk size and wrote an identical file).

Note that this diff also updates the FileContents enum and adds unimplemented methods there. These are all going away later in this stack (in the diff where I incorporate the Filestore into the rest of Mononoke).

Reviewed By: StanislavGlebik

Differential Revision: D16440678

fbshipit-source-id: 07187aa154f4fdcbd3b3faab7c0cbcb1f8a91217
2019-07-31 05:19:37 -07:00
..
blacklist.rs mononoke: convert DB connection instantiation to future 2019-07-16 03:17:58 -07:00
blobstore_fetch.rs mononoke: don't require MononokeId impls to know how to recompute themselves from a value 2019-07-31 05:19:36 -07:00
bonsai_fetch.rs mononoke: refactor the admin tool into components 2019-05-21 12:25:59 -07:00
bookmarks_manager.rs mononoke: make admin tool support bookmark deletion 2019-06-21 02:35:27 -07:00
cmdargs.rs mononoke: refactor the admin tool into components 2019-05-21 12:25:59 -07:00
common.rs mononoke: Rename Bookmark to BookmarkName 2019-05-21 12:26:02 -07:00
content_fetch.rs mononoke/filestore: introduce chunking support 2019-07-31 05:19:37 -07:00
hash_convert.rs mononoke: refactor the admin tool into components 2019-05-21 12:25:59 -07:00
hg_changeset.rs mononoke: refactor the admin tool into components 2019-05-21 12:25:59 -07:00
hg_sync.rs mononoke: convert DB connection instantiation to future 2019-07-16 03:17:58 -07:00
main.rs mononoke: make admin tool capable of fetching non-manifold blobstores 2019-07-12 10:50:50 -07:00
public_phases.rs mononoke: convert DB connection instantiation to future 2019-07-16 03:17:58 -07:00
skiplist_subcommand.rs mononoke: improve memory usage on mononoke startup and reduce number of small allocations 2019-07-19 04:30:26 -07:00