facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-10 08:47:12 +03:00
Code Issues Projects Releases Wiki Activity
7747032bdf
sapling/eden/scm/lib/zstore/Cargo.toml

23 lines
547 B
TOML
Raw Normal View History

hgcommits: add hybrid backend Summary: The hybrid backend is similar to the doublewrite backend, except that it does not use revlog to read commit data, but uses EdenAPI instead. Note: - The non-stream API will not fetch commit data from EdenAPI. - The commit hashes are not lazy yet. Reviewed By: sfilipco Differential Revision: D23924147 fbshipit-source-id: eb2cf8d3a7e1704b4efb13ad3ad86f8b6a1b31d0
2020-10-07 05:10:56 +03:00
# @generated by autocargo from //eden/scm/lib/zstore:zstore
zstore: blob store backed by zstdelta Summary: This blob store is the first usage of indexedlog. It uses zstd dictionary compression provided by the zstdelta crate and aims to replace lz4 + bdiff revlog for storing file revisions. Blobs are enforced SHA1 addressed. A separate mapping from filenode to SHA1 is needed if we want to integrate it into Mercurial file storage. Practically, compare with lz4+bdiff, this setup compresses better (which could be improved by tweaking parameters in zstdelta - it uses a variant of zstd level 1 currently) when using same delta-chains: | Repo | File revisions | Zstore size (log + index) | Revlog du size | Revlog raw size | | hg | 74.9K | 28 + 2.1MB | 40MB | 33.3MB | | cpython | 232K | 179 + 6.5MB | 221MB | 199.9MB | It should also compress better for binary files, since there is no line-based diff algorithm. Performance wise, my previous experiments on fbsource showed compression is faster with zstdelta. Decompression is slower. But in reality we might have many files to decompress, in which case, multi-thread should make things better. The delta base decision logic is designed to automatically reshape a linear chain to a tree to reduce chain length. The next diff adds dedicated tests for the algorithm. The name `Zstore` was chosen because the prefix `zst` is the name of the compression algorithm. Reviewed By: kulshrax Differential Revision: D10251983 fbshipit-source-id: 3ba308d2799bb759a36f69a88f7c0144830fc41e
2019-11-14 01:41:18 +03:00
[package]
name = "zstore"
edition = "2018"
hgcommits: add hybrid backend Summary: The hybrid backend is similar to the doublewrite backend, except that it does not use revlog to read commit data, but uses EdenAPI instead. Note: - The non-stream API will not fetch commit data from EdenAPI. - The commit hashes are not lazy yet. Reviewed By: sfilipco Differential Revision: D23924147 fbshipit-source-id: eb2cf8d3a7e1704b4efb13ad3ad86f8b6a1b31d0
2020-10-07 05:10:56 +03:00
version = "0.1.0"
zstore: blob store backed by zstdelta Summary: This blob store is the first usage of indexedlog. It uses zstd dictionary compression provided by the zstdelta crate and aims to replace lz4 + bdiff revlog for storing file revisions. Blobs are enforced SHA1 addressed. A separate mapping from filenode to SHA1 is needed if we want to integrate it into Mercurial file storage. Practically, compare with lz4+bdiff, this setup compresses better (which could be improved by tweaking parameters in zstdelta - it uses a variant of zstd level 1 currently) when using same delta-chains: | Repo | File revisions | Zstore size (log + index) | Revlog du size | Revlog raw size | | hg | 74.9K | 28 + 2.1MB | 40MB | 33.3MB | | cpython | 232K | 179 + 6.5MB | 221MB | 199.9MB | It should also compress better for binary files, since there is no line-based diff algorithm. Performance wise, my previous experiments on fbsource showed compression is faster with zstdelta. Decompression is slower. But in reality we might have many files to decompress, in which case, multi-thread should make things better. The delta base decision logic is designed to automatically reshape a linear chain to a tree to reduce chain length. The next diff adds dedicated tests for the algorithm. The name `Zstore` was chosen because the prefix `zst` is the name of the compression algorithm. Reviewed By: kulshrax Differential Revision: D10251983 fbshipit-source-id: 3ba308d2799bb759a36f69a88f7c0144830fc41e
2019-11-14 01:41:18 +03:00
[dependencies]
indexedlog = { path = "../indexedlog" }
mincode = { path = "../mincode" }
lib: bytes::Bytes -> minibytes::Bytes Summary: The `bytes` crate still does not support zero-copy on mmaped buffer. Switch to `minibytes::Bytes` so bytes returned from our main storage backend indexedlog is a zero-copy slice backed by a mmap buffer. Migrate vfs, revisionstore, pyworker to minibytes so they can preserve zero-copy mmap buffers from indexedlog. The edenapi/types is unchanged, since it's also used in Mononoke which uses `bytes::Bytes` all the places. The conversion to `minibytes::Bytes` is cheap so it's probably not a performance issue. Reviewed By: kulshrax Differential Revision: D26218289 fbshipit-source-id: e4f1c631143b7676c6b48d3b4f97055299bfd334
2021-02-04 07:19:27 +03:00
minibytes = { path = "../minibytes", features = ["frombytes"] }
zstore: blob store backed by zstdelta Summary: This blob store is the first usage of indexedlog. It uses zstd dictionary compression provided by the zstdelta crate and aims to replace lz4 + bdiff revlog for storing file revisions. Blobs are enforced SHA1 addressed. A separate mapping from filenode to SHA1 is needed if we want to integrate it into Mercurial file storage. Practically, compare with lz4+bdiff, this setup compresses better (which could be improved by tweaking parameters in zstdelta - it uses a variant of zstd level 1 currently) when using same delta-chains: | Repo | File revisions | Zstore size (log + index) | Revlog du size | Revlog raw size | | hg | 74.9K | 28 + 2.1MB | 40MB | 33.3MB | | cpython | 232K | 179 + 6.5MB | 221MB | 199.9MB | It should also compress better for binary files, since there is no line-based diff algorithm. Performance wise, my previous experiments on fbsource showed compression is faster with zstdelta. Decompression is slower. But in reality we might have many files to decompress, in which case, multi-thread should make things better. The delta base decision logic is designed to automatically reshape a linear chain to a tree to reduce chain length. The next diff adds dedicated tests for the algorithm. The name `Zstore` was chosen because the prefix `zst` is the name of the compression algorithm. Reviewed By: kulshrax Differential Revision: D10251983 fbshipit-source-id: 3ba308d2799bb759a36f69a88f7c0144830fc41e
2019-11-14 01:41:18 +03:00
types = { path = "../types" }
zstdelta = { path = "../zstdelta" }
hgcommits: add hybrid backend Summary: The hybrid backend is similar to the doublewrite backend, except that it does not use revlog to read commit data, but uses EdenAPI instead. Note: - The non-stream API will not fetch commit data from EdenAPI. - The commit hashes are not lazy yet. Reviewed By: sfilipco Differential Revision: D23924147 fbshipit-source-id: eb2cf8d3a7e1704b4efb13ad3ad86f8b6a1b31d0
2020-10-07 05:10:56 +03:00
lazy_static = "1.0"
lru-cache = "0.1.2"
parking_lot = "0.10.2"
serde = { version = "1.0", features = ["derive", "rc"] }
sha-1 = "0.8"
tracing = "0.1"
zstore: blob store backed by zstdelta Summary: This blob store is the first usage of indexedlog. It uses zstd dictionary compression provided by the zstdelta crate and aims to replace lz4 + bdiff revlog for storing file revisions. Blobs are enforced SHA1 addressed. A separate mapping from filenode to SHA1 is needed if we want to integrate it into Mercurial file storage. Practically, compare with lz4+bdiff, this setup compresses better (which could be improved by tweaking parameters in zstdelta - it uses a variant of zstd level 1 currently) when using same delta-chains: | Repo | File revisions | Zstore size (log + index) | Revlog du size | Revlog raw size | | hg | 74.9K | 28 + 2.1MB | 40MB | 33.3MB | | cpython | 232K | 179 + 6.5MB | 221MB | 199.9MB | It should also compress better for binary files, since there is no line-based diff algorithm. Performance wise, my previous experiments on fbsource showed compression is faster with zstdelta. Decompression is slower. But in reality we might have many files to decompress, in which case, multi-thread should make things better. The delta base decision logic is designed to automatically reshape a linear chain to a tree to reduce chain length. The next diff adds dedicated tests for the algorithm. The name `Zstore` was chosen because the prefix `zst` is the name of the compression algorithm. Reviewed By: kulshrax Differential Revision: D10251983 fbshipit-source-id: 3ba308d2799bb759a36f69a88f7c0144830fc41e
2019-11-14 01:41:18 +03:00
[dev-dependencies]
quickcheck = "0.9"
hgcommits: add hybrid backend Summary: The hybrid backend is similar to the doublewrite backend, except that it does not use revlog to read commit data, but uses EdenAPI instead. Note: - The non-stream API will not fetch commit data from EdenAPI. - The commit hashes are not lazy yet. Reviewed By: sfilipco Differential Revision: D23924147 fbshipit-source-id: eb2cf8d3a7e1704b4efb13ad3ad86f8b6a1b31d0
2020-10-07 05:10:56 +03:00
tempfile = "3.1"
Reference in New Issue Copy Permalink