sapling/lib/lz4-pyframe/Cargo.toml

[package]
name = "lz4-pyframe"
version = "0.1.0"
authors = ["Facebook Source Control Team <sourcecontrol-dev@fb.com>"]

[dependencies]
byteorder = "1"
failure = "0"
libc = "0.2"
lz4-sys = "1"

[dev-dependencies]
minibench = { path = "../minibench" }
quickcheck = "0"
rand_chacha = "0.1.0"
rand_core = "0.3.0"

[[bench]]
name = "bench"
harness = false
Back out "Reuse pylz4 encoding between hg and Mononoke into a separate library" Summary: Backout D9124508. This is actually more complex than it seems. It breaks non-buck build everywhere: - hgbuild on all platforms. POSIX platforms break because `hg archive` will miss `scm/common`. Windows build breaks because of symlink. - `make local` on GitHub repo because `failure_ext` is not public. The `pylz4` Cargo.toml has missing dependencies. Fixing them correctly seems non-trivial. Therefore let's backout the change to unblock builds quickly. The linter change is kept in case we'd like to try again in the future. Reviewed By: simpkins Differential Revision: D9225955 fbshipit-source-id: 4170a5f7664ac0f6aa78f3b32f61a09d65e19f63 2018-08-08 22:15:27 +03:00			`[package]`
			`name = "lz4-pyframe"`
			`version = "0.1.0"`
			`authors = ["Facebook Source Control Team <sourcecontrol-dev@fb.com>"]`

			`[dependencies]`
			`byteorder = "1"`
			`failure = "0"`
lz4-pyframe: add compresshc Summary: Unfortunately required symbols are not exposed by lz4-sys. So we just declare them ourselves. Make sure it compresses better: In [1]: c=open('/bin/bash').read(); In [2]: from mercurial.rust import lz4 In [3]: len(lz4.compress(c)) Out[3]: 762906 In [4]: len(lz4.compresshc(c)) Out[4]: 626970 While it's much slower for larger data (and compresshc is slower than pylz4): Benchmarking (easy to compress data, 20MB)... pylz4.compress: 10328.03 MB/s rustlz4.compress_py: 9373.84 MB/s pylz4.compressHC: 1666.80 MB/s rustlz4.compresshc_py: 8298.57 MB/s pylz4.decompress: 3953.03 MB/s rustlz4.decompress_py: 3935.57 MB/s Benchmarking (hard to compress data, 0.2MB)... pylz4.compress: 4357.88 MB/s rustlz4.compress_py: 4193.34 MB/s pylz4.compressHC: 3740.40 MB/s rustlz4.compresshc_py: 2730.71 MB/s pylz4.decompress: 5600.94 MB/s rustlz4.decompress_py: 5362.96 MB/s Benchmarking (hard to compress data, 20MB)... pylz4.compress: 5156.72 MB/s rustlz4.compress_py: 5447.00 MB/s pylz4.compressHC: 33.70 MB/s rustlz4.compresshc_py: 22.25 MB/s pylz4.decompress: 2375.42 MB/s rustlz4.decompress_py: 5755.46 MB/s Note python-lz4 was using an ancient version of lz4. So there could be differences. Reviewed By: DurhamG Differential Revision: D13528200 fbshipit-source-id: 6be1c1dd71f57d40dcffcc8d212d40a853583254 2018-12-21 04:48:27 +03:00			`libc = "0.2"`
Back out "Reuse pylz4 encoding between hg and Mononoke into a separate library" Summary: Backout D9124508. This is actually more complex than it seems. It breaks non-buck build everywhere: - hgbuild on all platforms. POSIX platforms break because `hg archive` will miss `scm/common`. Windows build breaks because of symlink. - `make local` on GitHub repo because `failure_ext` is not public. The `pylz4` Cargo.toml has missing dependencies. Fixing them correctly seems non-trivial. Therefore let's backout the change to unblock builds quickly. The linter change is kept in case we'd like to try again in the future. Reviewed By: simpkins Differential Revision: D9225955 fbshipit-source-id: 4170a5f7664ac0f6aa78f3b32f61a09d65e19f63 2018-08-08 22:15:27 +03:00			`lz4-sys = "1"`

			`[dev-dependencies]`
lz4-pyframe: add a benchmark Summary: This gives some sense about how fast it is. Background: I was trying to get rid of python-lz4, by exposing this to Python. However, I noticed it's 10x slower than python-lz4. Therefore I added some benchmark here to test if it's the wrapper or the Rust lz4 code. It does not seem to be this crate: ``` # Pure Rust compress (100M) 77.170 ms decompress (~100M) 67.043 ms # python-lz4 In [1]: import lz4, os In [2]: b=os.urandom(100000000); In [3]: %timeit lz4.compress(b) 10 loops, best of 3: 87.4 ms per loop ``` Reviewed By: DurhamG Differential Revision: D13516205 fbshipit-source-id: f55f94bbecc3b49667ed12174f7000b1aa29e7c4 2018-12-21 04:48:25 +03:00			`minibench = { path = "../minibench" }`
Back out "Reuse pylz4 encoding between hg and Mononoke into a separate library" Summary: Backout D9124508. This is actually more complex than it seems. It breaks non-buck build everywhere: - hgbuild on all platforms. POSIX platforms break because `hg archive` will miss `scm/common`. Windows build breaks because of symlink. - `make local` on GitHub repo because `failure_ext` is not public. The `pylz4` Cargo.toml has missing dependencies. Fixing them correctly seems non-trivial. Therefore let's backout the change to unblock builds quickly. The linter change is kept in case we'd like to try again in the future. Reviewed By: simpkins Differential Revision: D9225955 fbshipit-source-id: 4170a5f7664ac0f6aa78f3b32f61a09d65e19f63 2018-08-08 22:15:27 +03:00			`quickcheck = "0"`
lz4-pyframe: add a benchmark Summary: This gives some sense about how fast it is. Background: I was trying to get rid of python-lz4, by exposing this to Python. However, I noticed it's 10x slower than python-lz4. Therefore I added some benchmark here to test if it's the wrapper or the Rust lz4 code. It does not seem to be this crate: ``` # Pure Rust compress (100M) 77.170 ms decompress (~100M) 67.043 ms # python-lz4 In [1]: import lz4, os In [2]: b=os.urandom(100000000); In [3]: %timeit lz4.compress(b) 10 loops, best of 3: 87.4 ms per loop ``` Reviewed By: DurhamG Differential Revision: D13516205 fbshipit-source-id: f55f94bbecc3b49667ed12174f7000b1aa29e7c4 2018-12-21 04:48:25 +03:00			`rand_chacha = "0.1.0"`
			`rand_core = "0.3.0"`

			`[[bench]]`
			`name = "bench"`
			`harness = false`