facebook/sapling
1
1
Fork 0
mirror of https://github.com/facebook/sapling.git synced 2024-10-12 01:39:21 +03:00
Code Issues Projects Releases Wiki Activity
6153dab328
sapling/eden/mononoke/blobstore/virtually_sharded_blobstore/Cargo.toml

30 lines
1.2 KiB
TOML
Raw Normal View History

mononoke/virtually_sharded_blobstore: a caching blobstore that deduplicates Summary: This introduces a caching blobstore that deduplicates reads and writes. The underlying motivation is to improve performance for processes that might find themsleves inadvertently reading the same data concurrently from a bunch of independent callsites (most of Mononoke), or writing the same bit of data over and over again. The latter is particularly useful for things like commit cloud backfilling in WWW, where some logger commits include the same blob being written hundreds or thousands of times, and cause us to overload the underlying Zippy shard in Manifold. This is however a problem we've also encountered in the past in e.g. the deleted files manifest and had to solve there. This blobstore is a little different in the sense that it solves that problem for all writers. This comes at the cost of writes being dropped if they're known to be redundant, which prevents updates through this blobstore. This is desirable for most of Mononoke, but not all (notably, for skiplist updates it's not great). For now, I'm going to add this behind an opt-in flag, and later on I'm planning to make it opt-out and turn it off there (I'm thinking to use the CoreContext for this). Reviewed By: farnz Differential Revision: D22285270 fbshipit-source-id: 4e3502ab2da52a3a0e0e471cd9bc4c10b84a3cc5
2020-07-03 15:51:08 +03:00
[package]
name = "virtually_sharded_blobstore"
edition = "2018"
version = "0.1.0"
authors = ['Facebook']
license = "GPLv2+"
include = ["src/*.rs"]
[dependencies]
blobstore = { path = ".." }
context = { path = "../../server/context" }
mononoke_types = { path = "../../mononoke_types" }
mononoke/virtually_sharded_blobstore: log deduplicated puts Summary: If anything were to go wrong, we'd be happy to know which puts we ignored. So, let's log them. Reviewed By: farnz Differential Revision: D22356714 fbshipit-source-id: 5687bf0fc426421c5f28b99a9004d87c97106695
2020-07-03 15:51:08 +03:00
scuba_ext = { path = "../../common/scuba_ext" }
mononoke/virtually_sharded_blobstore: a caching blobstore that deduplicates Summary: This introduces a caching blobstore that deduplicates reads and writes. The underlying motivation is to improve performance for processes that might find themsleves inadvertently reading the same data concurrently from a bunch of independent callsites (most of Mononoke), or writing the same bit of data over and over again. The latter is particularly useful for things like commit cloud backfilling in WWW, where some logger commits include the same blob being written hundreds or thousands of times, and cause us to overload the underlying Zippy shard in Manifold. This is however a problem we've also encountered in the past in e.g. the deleted files manifest and had to solve there. This blobstore is a little different in the sense that it solves that problem for all writers. This comes at the cost of writes being dropped if they're known to be redundant, which prevents updates through this blobstore. This is desirable for most of Mononoke, but not all (notably, for skiplist updates it's not great). For now, I'm going to add this behind an opt-in flag, and later on I'm planning to make it opt-out and turn it off there (I'm thinking to use the CoreContext for this). Reviewed By: farnz Differential Revision: D22285270 fbshipit-source-id: 4e3502ab2da52a3a0e0e471cd9bc4c10b84a3cc5
2020-07-03 15:51:08 +03:00
cachelib = { git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
cloned = { git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
futures_stats = { git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
mononoke/virtually_sharded_blobstore: add ODS metrics Summary: Those are useful to track. Reviewed By: farnz Differential Revision: D22332480 fbshipit-source-id: 43f5cd7121c4aa497d961015e7c16973615798d1
2020-07-03 15:51:08 +03:00
stats = { git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
mononoke/virtually_sharded_blobstore: a caching blobstore that deduplicates Summary: This introduces a caching blobstore that deduplicates reads and writes. The underlying motivation is to improve performance for processes that might find themsleves inadvertently reading the same data concurrently from a bunch of independent callsites (most of Mononoke), or writing the same bit of data over and over again. The latter is particularly useful for things like commit cloud backfilling in WWW, where some logger commits include the same blob being written hundreds or thousands of times, and cause us to overload the underlying Zippy shard in Manifold. This is however a problem we've also encountered in the past in e.g. the deleted files manifest and had to solve there. This blobstore is a little different in the sense that it solves that problem for all writers. This comes at the cost of writes being dropped if they're known to be redundant, which prevents updates through this blobstore. This is desirable for most of Mononoke, but not all (notably, for skiplist updates it's not great). For now, I'm going to add this behind an opt-in flag, and later on I'm planning to make it opt-out and turn it off there (I'm thinking to use the CoreContext for this). Reviewed By: farnz Differential Revision: D22285270 fbshipit-source-id: 4e3502ab2da52a3a0e0e471cd9bc4c10b84a3cc5
2020-07-03 15:51:08 +03:00
time_ext = { git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
anyhow = "1.0"
bytes = { version = "0.5", features = ["serde"] }
futures = { version = "0.3", features = ["async-await", "compat"] }
scopeguard = "1.0.0"
tokio = { version = "=0.2.13", features = ["full"] }
[dev-dependencies]
fbinit = { git = "https://github.com/facebookexperimental/rust-shed.git", branch = "master" }
nonzero_ext = "0.2"
once_cell = "1.4"
tokio-compat = "0.1"
Reference in New Issue Copy Permalink