Summary:
This imports the async-compression crate. We have an equivalent-ish in
common/rust, but it targets Tokio 0.1, whereas this community-supported crate
targets Tokio 0.2 (it offers a richer API, notably in the sense that we
can use it for Streams, whereas the async-compression crate we have is only for
AsyncWrite).
In the immediate term, I'd like to use this for transfer compression in
Mononoke's LFS Server. In the future, we might also use it in Mononoke where we
currently use our own async compression crate when all that stuff moves to
Tokio 0.2.
Finally, this also updates zstd: the version we link to from tp2 is actually
zstd 1.4.5, so it's a good idea to just get the same version of the zstd crate.
The zstd crate doesn't keep a great changelog, so it's hard to tell what has changed.
At a glance, it looks like the answer is not much, but I'm going to look to Sandcastle
to root out potential issues here.
Reviewed By: StanislavGlebik
Differential Revision: D23652335
fbshipit-source-id: e250cef7a52d640bbbcccd72448fd2d4f548a48a
Summary:
Generated by formatting with rustfmt 2.0.0-rc.2 and then a second time with fbsource's current rustfmt (1.4.14).
This results in formatting for which rustfmt 1.4 is idempotent but is closer to the style of rustfmt 2.0, reducing the amount of code that will need to change atomically in that upgrade.
---
*Why now?* **:** The 1.x branch is no longer being developed and fixes like https://github.com/rust-lang/rustfmt/issues/4159 (which we need in fbcode) only land to the 2.0 branch.
---
Reviewed By: StanislavGlebik
Differential Revision: D23568780
fbshipit-source-id: b4b4a0aa683d236e2fdeb5b96d723ac2d84b9faf
Summary: Add a non-thrift header to packblob so we can vary thrift protocol in future.
Reviewed By: farnz
Differential Revision: D22953758
fbshipit-source-id: a114a350105e75cbe57f6c824295d863c723f32f
Summary: D22381744 updated the version of `futures` in third-party/rust to 0.3.5, but did not regenerate the autocargo-managed Cargo.toml files in the repo. Although this is a semver-compatible change (and therefore should not break anything), it means that affected projects would see changes to all of their Cargo.toml files the next time they ran `cargo autocargo`.
Reviewed By: dtolnay
Differential Revision: D22403809
fbshipit-source-id: eb1fdbaf69c99549309da0f67c9bebcb69c1131b
Summary:
Eventually, we want everything to be `async`/`await`; as a stepping stone in that direction, switch the remaining lobstore traits to new-style futures.
This just pushes the `.compat()` out to old-style futures, but it makes the move to non-'static lifetimes easier, as all the compile errors will relate to lifetime issues.
Reviewed By: krallin
Differential Revision: D22183228
fbshipit-source-id: 3fe3977f4469626f55cbf5636d17fff905039827
Summary:
Remove unused dependencies for Rust targets.
This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.
Manual edits (named_deps) and misc output in P133451794
Reviewed By: dtolnay
Differential Revision: D22083498
fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
Summary:
Add optional compress on put controlled by a command line option.
Other than costing some CPU time, this may be a good option when populating repos from existing uncompressed stores to new stores.
Reviewed By: farnz
Differential Revision: D22037756
fbshipit-source-id: e75190ddf9cfd4ed3ea9a18a0ec6d9342a90707b
Summary: Add zstd decoding support to packblob so that if store contains individually zstd compressed blobs we can load them on get()
Reviewed By: farnz
Differential Revision: D22037755
fbshipit-source-id: 41d85be6fcccf14fb198f6ea33a7ca26c4527a46
Summary:
Add a regex for repo prefix detection, and use it in prefixblob for removal of repo prefix from blob-embedded keys.
This is important to keep blobs copyable between repos, and to allow dedupe between same blob in two repos.
I looked at alternate approaches of passing down the prefix from PrefixBlob::new(), but that was fragile and didn't cover the use cases from things like blobstore_healer where no prefixblob is configured at all but packblob will be in the blobstore stack. Using a pattern is the only real option in such "all repo" cases.
The aim of binding the pattern and its tests closely to the prefix generation is to make it hard for someone to get them out of sync, and provide a clear local test failure if they do.
Reviewed By: farnz
Differential Revision: D22034638
fbshipit-source-id: 95a1c2e1ef81432cba606c22afa16b026f59fd5f
Summary:
Add pack and unpack logic to packblob.
Loading a packed form is possible from a regular get(), as the store may contain packed data.
Storing a packed form is only possible from a new put_packed() method, intended for use from the packer (and tests).
NB As noted in the TODO in get, this does not yet handle prefix removal on get, will address that in a separate diff.
Reviewed By: StanislavGlebik
Differential Revision: D21980498
fbshipit-source-id: f534b0e754aa29c42bf00bb7e764f93f1446c14c
Summary: Add basic packblob store which wraps blobs in the thrift StorageEnvelope type on put() and unwraps them on get()
Reviewed By: farnz
Differential Revision: D21924405
fbshipit-source-id: 81eccf5da6b5b50bdb9aae13622301a20dca0eac
Summary:
Thift definitions for a new blobstore layer known as packblob, which is an enabling part of compression at the mononoke blobstore layer. Its storage formats will support raw data, individual blob compression and blob delta compression.
The included README.md has more detail on the expected use of these types.
Reviewed By: farnz
Differential Revision: D21761723
fbshipit-source-id: a277800f4b4aed5972c4a5bb277e984ff12660f8
