sapling/repo_client
Thomas Orozco 895aa3c27a mononoke: getpackv2 LFS support (+ getfiles / getpack / eden_get_data refactor)
Summary:
This updates Mononoke to support LFS metadata when serving data over getpackv2.

However, in doing so, I've also refactored the various ways in which we currently access file data to serve it to clients or to process client uploads (when we need to compute deltas). The motivation to do that is that we've had several issues recently where some protocols knew about some functionality, and others didn't. Notably, redaction and LFS were supported in getfiles, but neither of them were supported in getpack or eden_get_data.

This patch refactors all those callsites away from blobrepo and instead through repo_client/remotefilelog, which provides an internal common method to fetch a filenode and return its metadata and bytes (prepare_blob), and separate protocol specific implementations for getpackv1 (includes metadata + file content -- this is basically the existing fetch_raw_filenode_bytes function), getpackv2 (includes metadata + file contents + getpackv2 metadata), getfiles (includes just file content, and ties file history into its response) and eden_get_data (which uses getpackv1).

Here are a few notable changes here that are worth noting as you review this:

- The getfiles method used to get its filenode from get_maybe_draft_filenode, but all it needed was the copy info. However, the updated method gets its filenode from the envelope (which also has this data). This should be equivalent.
- I haven't been able to remove fetch_raw_filenode_bytes yet because there's a callsite that still uses it and it's not entirely clear to me whether this is used and why. I'll look into it, but for now I left it unchanged.
- I've used the Mercurial implementation of getpack metadata here. This feels like the better approach so we can reuse some of the code, but historically I don't think we've depended on many Mercurial crates. Let me know if there's a reason not to do that.

Finally, there are a couple things to be aware of as you review:

- I removed some more `Arc<BlobRepo>` in places where it made it more difficult to call the new remotefilelog methods.
- I updated the implementation to get copy metadata out of a file envelope to not require copying the metadata into a mercurial::file::File only to immediately discard it.
- I cleaned up an LFS integration test a little bit. There are few functional changes there, but it makes tests a little easier to work with.

Reviewed By: farnz

Differential Revision: D16784413

fbshipit-source-id: 5c045d001472fb338a009044ede1e22ccd34dc55
2019-08-14 08:48:35 -07:00
..
remotefilelog/src mononoke: getpackv2 LFS support (+ getfiles / getpack / eden_get_data refactor) 2019-08-14 08:48:35 -07:00
repo_read_write_status/src mononoke: lock repo from sync job itself, not from failure handler 2019-07-10 02:38:10 -07:00
schemas mononoke: Allow dynamic repo read_only mode 2019-03-18 08:53:53 -07:00
src mononoke: getpackv2 LFS support (+ getfiles / getpack / eden_get_data refactor) 2019-08-14 08:48:35 -07:00
streaming_clone/src mononoke: allow running mononoke server without myrouter 2019-06-18 07:52:22 -07:00