Summary:
cachelib can throw exceptions from any method. Make all our bindings
exception-safe
Reviewed By: jsgf
Differential Revision: D9158062
fbshipit-source-id: 57948aa6ead304fac6e037621a1f6641826345c5
Summary:
I had enough seeing compiler giving me error messages with `blobrepo::Error` whereas it should be `failure::Error`.
`pub type` is exporting a type alias whereas `pub use` is re-exporting. The later apparently will fool the compiler to think that they own the thing they are re-exporting which in this case, failure::Error. If it is for the purpose of declaring the error type the crate is using, type alias is obviously more suitable as it won't take the ownership of the type.
Reviewed By: lukaspiatkowski
Differential Revision: D8926275
fbshipit-source-id: 6b1346adf52bec6b666a3f8b23e7a4ea4e0ab070
Summary:
Back out "[mononoke] Switch to cachelib for blob caching"
Original commit changeset: 2549d85dfcba
Back out "[mononoke] Remove unused asyncmemo imports"
Original commit changeset: e34f8c34a3f6
Back out "mononoke: fix blobimport"
Original commit changeset: b540201b93f1
Reviewed By: StanislavGlebik
Differential Revision: D8989404
fbshipit-source-id: e4e7c629cb4dcf196aa56eb07a53a45f6008eb4e
Summary: These files import the asyncmemo crate but do not use it. Remove it.
Reviewed By: StanislavGlebik
Differential Revision: D8951887
fbshipit-source-id: e34f8c34a3f652156b63d795023d67260242a58e
Summary:
Start deprecating AsyncMemo for caching purposes, by removing its use
as a blobstore cache.
Reviewed By: StanislavGlebik
Differential Revision: D8840496
fbshipit-source-id: 2549d85dfcba6647e9b0824ab55ab76165a17564
Summary:
This proves to be extremely useful for the upcoming bonsai
verification code.
The in-memory stuff is more complicated for the database backends, so punt on
that for now with some warnings.
Reviewed By: farnz
Differential Revision: D8909426
fbshipit-source-id: 1d66d877cfa48ef06bbe614f37c66cf6c2f0446c
Summary:
This is generally quite useful while trying to figure out what chain
of blobstores this actually is.
Reviewed By: farnz
Differential Revision: D8909424
fbshipit-source-id: 49ba471c71c3da109bd219ec5f78cc0fdc2bc857
Summary:
StanislavGlebik noticed that the lease release on a successful store didn't
have the correct hint - fix it.
Reviewed By: StanislavGlebik
Differential Revision: D8893519
fbshipit-source-id: d509aa72575533ad161b96599c80f8d9c3b4f997
Summary:
It's wasteful to send the same bunch of bytes to a backing blobstore
twice; introduce an in-memory LeaseOps that, when combined with a presence
cache, ensures that we'll never do that.
Reviewed By: StanislavGlebik
Differential Revision: D8820987
fbshipit-source-id: 406a388cb1f08a35daa49151dadd023fde260e07
Summary:
We want to remove AsyncMemo from caching duties - abusing a memoizer
to provide caching introduces issues of its own. Provide a replacement cache we
can use instead of AsyncMemo
Reviewed By: StanislavGlebik
Differential Revision: D8810965
fbshipit-source-id: 0469c5ff5f16cf40ae1a1652f0af0d301611a39a
Summary:
This blobstore gets cloned a lot, and an inline string is much cheaper
to clone.
Reviewed By: StanislavGlebik
Differential Revision: D8843939
fbshipit-source-id: 879c2e43b27ec7a410ef64eb65d3ac78d21fa1db
Summary:
The CachingBlobstore can use a LeaseOps implementation to prevent
multiple parallel writes to the same backing store. Provide a dummy implementation
that does not block parallel writes, for performance testing and to use where
there's no way to implement a useful lease operation using a given cache.
Reviewed By: StanislavGlebik
Differential Revision: D8809077
fbshipit-source-id: 54cf67882a42b43dde1c6ac68611fb28144509bc
Summary:
The memcache implementations of CacheOps and LeaseOps are fairly complex, with lots of edge cases.
Add some stats, so we can see which edge cases are being hit, and optimize accordingly
Reviewed By: lukaspiatkowski
Differential Revision: D8752599
fbshipit-source-id: fe1376956beceff702c2b6a020598556016033b5
Summary:
When we add cachelib bindings to Rust, we're going to want to implement a
cachelib blobstore that's more or less the same as the memcache version, but
backed by a cachelib pool instead of a memcache instance.
Split this code up so that we don't duplicate functionality
Reviewed By: StanislavGlebik
Differential Revision: D8523713
fbshipit-source-id: 882298abab8c208103f6d8c74fee60a768c877f6
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
Manifold can complain when we expect it to succeed if we write the
same thing twice. As this is inefficient, use memcache to avoid writing to the
backing blobstore if we know the object is already present
Reviewed By: StanislavGlebik
Differential Revision: D8424296
fbshipit-source-id: 3b97683b80a1ea5245998d8446c8f540975343df
Summary:
It would be suboptimal if memcache could be abused to examine objects
you should not have access to. Change the key used so that (a) it's a regional
cluster for wider caching, and (b) it's got the blobstore details in it so that
you cannot extract objects not from your blob store.
Reviewed By: StanislavGlebik
Differential Revision: D8424298
fbshipit-source-id: 78f9a1a7302b4a60575f257bda665f719dc1a7b6
Summary:
So that both prefixed and non-prefixed blobstores can be used with the
same code.
Reviewed By: farnz
Differential Revision: D8402466
fbshipit-source-id: 3a4f7882ce697d0582eb7f7908b74716322806cf
Summary:
We'd like to prefix each key with a fixed string, and this wrapper
easily lets us achieve that.
Reviewed By: farnz
Differential Revision: D8374708
fbshipit-source-id: f02845d207c8015ea0241789f696884d440e3db6
Summary:
When I implement a cachemutex to keep the herd from all trying to
write the same blob multiple times, we'll be able to accelerate this. In the
short term, move the implementation up to here so that the change will be
obvious when it's put up for review.
Reviewed By: jsgf
Differential Revision: D8318152
fbshipit-source-id: f117c5fd834cab55ace10ac46bffa881457c840a
Summary:
Memcache is going to provide a faster implementation of `is_present`
that relies on the put cachemutex. We lose all the benefits if `is_present` up
here ends up always calling `get` from lower layers - teach it not to do that.
Reviewed By: jsgf
Differential Revision: D8318151
fbshipit-source-id: 130d4cb17c530c8677bdd563806989c393b4c6cf
Summary: If we're going to use memcache to cache things, we need to be able to confirm that the blobstore contains what we expect. Add an admin option that lets us check what's in memcache and/or in the blobstore when using memcache
Reviewed By: StanislavGlebik
Differential Revision: D8124624
fbshipit-source-id: 763786930cf9f4be6d2d8e346ae1ef94dc8e492c
Summary:
Memcache provides us with a low-latency out-of-process cache, shared
between multiple machines. If we use this to provide a write-through cache,
then we get a couple of benefits:
1. Mononoke servers accessing the same memcache backend get low latency access
to commonly requested blobs.
2. Recently written blobs are available at low latency to the server that just
wrote them, hopefully speeding up new_blobimport.
Reviewed By: StanislavGlebik
Differential Revision: D8124623
fbshipit-source-id: 5af085aa8bb63c1366740edfda9020d72b8a9015
Summary:
I'm about to introduce a MemoizedBlobstore - let's ensure that
ambiguity is kept to a minimum by renaming CachingBlobstore
Reviewed By: jsgf
Differential Revision: D8124625
fbshipit-source-id: 301fbb31c5a772c11e84566889b8a6ac86cdae19
Summary:
In practice, caching blobstores are always wrapped around a known
concrete blob store. Make it generic in the type of the Blobstore, so that
rustc can reach in and inline appropriately if it sees an optimization
opportunity.
This is a micro-optimization, hence nice to have
Reviewed By: StanislavGlebik
Differential Revision: D8124627
fbshipit-source-id: 0d5d9b11fdede062ceaa949ace574b3d1c75e6b5
Summary:
I'm about to add a memcache caching blobstore, and the blobstore
lib.rs is getting large. Shrink it by removing the outdated comment, replacing
it with documentation comments, and moving the CachingBlobstore to its own file
Reviewed By: jsgf
Differential Revision: D8124626
fbshipit-source-id: ac081a162c2f15a26452ff708d16a108699c279b
Summary:
Rust 1.26 adds many new language features. In particular `impl Trait` is now
stable, so we no longer need `conservative_impl_trait`.
There also seems to have been changed in the (unstable) TryFrom with respect to
usize, and the behaviour of the never type `!`.
There are still a few deprecation warnings, but they don't cause the build to
fail.
Path remapping is now stable, so the buck config needs to change to use it
rather than the unstable command line option.
TODO:
- get aarch64 rust-crates-io build (can defer to a later update)
Reviewed By: Imxset21
Differential Revision: D7966091
fbshipit-source-id: 2e61e262c21eb01c852a36f49c6a6369cdaddcdb
Summary:
We don't need to explicitly create timers since the environment has
one set up by default.
Reviewed By: StanislavGlebik
Differential Revision: D7873576
fbshipit-source-id: bfcdc27a46397bff0730f64ad4f3de3865c7cfa1
Summary:
Currently, any sort of `Bytes` can be stored in the blobstore. That
caused me to make several mistakes while writing the code to store bonsai
changesets, because I'd just end up storing the wrong set of `Bytes`.
Introduce stronger typing so that only types that explicitly implement
`BlobStorable` can be stored in the blobstore.
Currently, these sorts of blobs can be stored in the blob store:
* `ChangesetBlob` and `ContentBlob` in `mononoke-types` (these are Thrift-serialized structures)
* The envelope `RawNodeBlob` and `RawCSBlob` types in `blobrepo`, once converted to `EnvelopeBlob` instances
* `HgBlob`, which contains revlog data (manifests or files) exactly as serialized by Mercurial
Reviewed By: StanislavGlebik
Differential Revision: D7627290
fbshipit-source-id: d1bcbde8881e365dec99618556e7b054985bccf7
Summary:
Add CachingBlobstore that uses Asyncmemo as a cache.
Normal Blobstore returns Option<Bytes>. CachingBlobstore doesn't cache `None` response, because this blob may appear later in the blobstore, and we want clients to be able to access it.
Reviewed By: farnz
Differential Revision: D6579385
fbshipit-source-id: 746a5c222dd00a35c3353e76c2792fc8e2323381
Summary:
We don't want to depend on Manifold being consistent and reliable when
we're trying to fix our performance issues. Introduce a DelayBlob store that
wraps another blobstore and allows you to pass it a function that generates
delay times (e.g. a random number generator with a log-normal distribution).
The idea is that we can use DelayBlob wrapped around RocksBlob to approximate
expected Manifold behaviour, and can improve if needed.
Reviewed By: StanislavGlebik
Differential Revision: D7353228
fbshipit-source-id: fee977d2754f3134af752efc2dbfc77fedc7f2bf
Summary:
The current Memblob store is eager; this is great for finding certain
classes of bugs (those that assume an ordering that is not guaranteed), but not
so good for exposing other classes of bugs (those that assume that the future
has done its work before it resolves).
Add a lazy variant that functions in the same way as Memblob, but that waits
until it's polled to return.
Reviewed By: StanislavGlebik
Differential Revision: D7033792
fbshipit-source-id: 4c2d8a8150d908bcb26347757f96f99e20d74fc2
Summary:
The default implementation of `assert_present` only asserted that no
errors were found accessing the blobstore. Update it to fail if the blob is
absent.
Reviewed By: StanislavGlebik
Differential Revision: D7033794
fbshipit-source-id: 596d3b0d3d55d05a5b96760c311defe5a05af634
Summary:
The commit API would like to avoid uploading and downloading data when
it doesn't need to - extend the blobstore API so that blobstores can avoid
wasted work
Reviewed By: StanislavGlebik
Differential Revision: D6966729
fbshipit-source-id: 17783e2123d47e43824af724da0b4358a163fa5b
Summary:
Remove the last associated types from BlobStore - this means that
BlobStore now has an associated trait object type.
Reviewed By: jsgf
Differential Revision: D6425414
fbshipit-source-id: 7186dab9b56593dd1d70be732d4ad56d1e7b3c63
Summary:
Don't use failure's bail!() and ensure!() macros.
Instead, failure_ext provides:
- bail_err!(err) - Converts its single parameter to the expected error and returns; ie `return Err(From::from(err));`
- bail_msg!(fmt, ...) - takes format string parameters and returns a `failure::err_msg()` error
- ensure_err!(), ensure_msg!() - corresponding changes
Also:
- remove all stray references to error-chain
- remove direct references to failure_derive (it's reexported via failure and failure_ext)
- replace uses of `Err(foo)?;` with `bail_err!()` (since `bail_err` unconditionally returns, but `Err(x)?` does not in principle, which can affect type inference)
Reviewed By: kulshrax
Differential Revision: D6507717
fbshipit-source-id: 635fb6f8c96d185b195dff171ea9c8db9e83af10
Summary:
Convert scm/mononoke to use failure, and update common/rust crates it depends on as well.
What it looks like is a lot of deleted code...
General strategy:
- common/rust/failure_ext adds some things that are in git failure that aren't yet in crates.io (`bail!` and `ensure!`, `Result<T, Error>`)
- everything returns `Result<T, failure::Error>`
- crates with real error get an error type, with a derived Fail implementation
- replicate error-chain by defining an `enum ErrorKind` where the fields match the declared errors in the error! macro
- crates with dummy error-chain (no local errors) lose it
- `.chain_err()` -> `.context()` or `.with_context()`
So far the only place I've needed to extract an error is in a unit test.
Having a single unified error type has simplified a lot of things, and removed a lot of error type parameters, error conversion, etc, etc.
Reviewed By: sid0
Differential Revision: D6446584
fbshipit-source-id: 744640ca2997d4a85513c4519017f2e2e78a73f5
Summary:
BlobStore is entirely generic, and puts no limits on its
implementations. Remove ValueIn and ValueOut type parameters, and insist that
all blobs are Bytes (as per production setups)
Reviewed By: StanislavGlebik
Differential Revision: D6425413
fbshipit-source-id: 455e526d8baebd0d0f1906941648acca89be4881
Summary:
BlobStore is entirely generic, and puts no limits on its
implementations. Remove the "Key" type parameter, and insist that all keys are
String (as per production setups)
Reviewed By: StanislavGlebik
Differential Revision: D6425412
fbshipit-source-id: 1f1229bf8e001bf780964e883c6beb071e9ef1d8
Summary:
Putting retries on this layer is not very good, because it requires every
client to add RetryingBlobstore.
Reviewed By: kulshrax
Differential Revision: D6298254
fbshipit-source-id: dbdce7fe141f9e1511322e74a1258d3819a68eb5
Summary:
Removes uses of deprecated Tokio API by doing the following:
1. Removing the ignore deprecation header
2. Including `futures_ext` in TARGETS
3. Porting all FutureBox uses to `futures_ext`
4. Porting `boxed()` to `boxify()`
The guide I followed:
https://our.intern.facebook.com/intern/wiki/Rust/Tokio_cleanup/
Reviewed By: sid0
Differential Revision: D6083953
fbshipit-source-id: 93b3ce0c70affbbf2b1158abf66f6f3a4d60a420
Summary:
Similar to heads/bookmarks, share blobstore tests across
implementations.
The trait bounds got a bit tricky, but I think I managed to get the ones we
need and no more.
Reviewed By: StanislavGlebik
Differential Revision: D6082649
fbshipit-source-id: 3a80adc95d6c1059dab7ec89e5ec5724370cc268
Summary:
Realized that we were missing a few crates from the Tokio cleanup because those crates
didn't have `#![deny(warnings)]`.
This also caused a bunch of files to be rustfmted, which is fine.
Reviewed By: kulshrax
Differential Revision: D6024628
fbshipit-source-id: 55032d20f3676c92ef124d861e1edcd34126ab55
Summary: Removes usage of BoxFuture and BoxStream from futures crate in blobstore/fileblob, following instructions in https://our.intern.facebook.com/intern/wiki/Rust/Tokio_cleanup/
Reviewed By: jsgf
Differential Revision: D5884656
fbshipit-source-id: dcf979daf939cd193f4d6361eb36d03977380a22
