Summary:
See the comment near "DecompressionType::OverreadingZstd" to see what it does.
Why OverreadingZstd works for Mononoke's use case? Answer:
Because we use it in bundle2 parsing, which is already chunked by the outer Reader. This means that when we have a stream of bytes:
```
uncompressed -> compressed bundle2 -> uncompressed
```
thanks to chunking we extract the compressed part:
```
do_stuff(uncompressed)
ZstdDecoder(compressed bundle2)
do_stuff(uncompressed)
```
rather than
```
do_stuff(uncompressed)
ZstdDecoder(compressed bundle2 -> uncompressed)
```
So overreading doesn't hurt us here
Reviewed By: StanislavGlebik
Differential Revision: D9700778
fbshipit-source-id: 70dd6f405ffa00fb981791aff25c60f60831ea6b
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
- Added test for checking if decompressing isn't overreading it's input
- As a result removed ZSTD decompression which is overreading input
- Changes in decompression code propagated usage of BufRead in bundle2 parsing code
- As a result a bug in OuterDecoder was found where the buf was not consumed if the header value is "0"
Reviewed By: jsgf
Differential Revision: D6557440
fbshipit-source-id: 89a9f4c8790017c5b86d28d467e45f687d7323f6
Summary:
The no-compression decompressor cann't provide framing as other decompressors, so a safe approach would be not to have it at all.
I replaced occurances of no-compression with `Either` reader that seem to be a pattern present in the community (f.e. `futures::future::Either` or `itertools::Either`)
Reviewed By: jsgf
Differential Revision: D6555922
fbshipit-source-id: 998dafab8d9b2f00d058ce2f9e0aced76cf15b4e
Summary:
When using forward/send_all on a Sink it is being flushed and closed, this means that if we call forward multiple times on a Sink it might fail if the underlying IO is already closed.
For ZSTD compression it didn't mattered, because it doesn't use the close/finish, but for other compressions it does.
Reviewed By: jsgf
Differential Revision: D6545343
fbshipit-source-id: 079d950f34716d621dd1fabda4f97d58d0e1d02d
Summary:
Lifetime doesn't need to be static - it can be something smaller.
Propagate this into the bundle2 parser.
Reviewed By: farnz
Differential Revision: D6203918
fbshipit-source-id: f559b92029c975fee6e5c64d9d34ff9708364dc9
