Summary:
Right now this is the same as our Filestore threshold so we always attempt to
compress Filestore chunks even though they were designed to fit in cache.
Whoops.
Reviewed By: ikostia
Differential Revision: D25274080
fbshipit-source-id: f17b54710fc36ca7c11c74247d038bf73777f7f9
Summary:
Create `BlobstoreMapping` as a trait with the common implementations for
derived data mappings that are stored in the blobstore.
Reviewed By: StanislavGlebik
Differential Revision: D25099915
fbshipit-source-id: 8a62fbb809918045336944c8cd3584b109811012
Summary:
This diff prepares the Mononoke codebase for composition-based extendability of
`ScubaSampleBuilder`. Specifically, in the near future I will add:
- new methods for verbose scuba logging
- new data field (`ObservabilityContext`) to check if verbose logging should
be enabled or disabled
The higher-level goal here is to be able to enable/disable verbose Scuba
logging (either overall or for certain slices of logs, like for a certain
session id) in real time, without restarting Mononoke. To do so, I plan to
expose the aforementioned verbose logging methods, which will run a check
against the stored `ObservabilityContext` and make a decision of whether the
logging is enabled or not. `ObservabilityContext` will of course hide
implementation details from the renamed `ScubaSampleBuilderExt`, and just provide a yes/no
answer based on the current config and sample fields.
At the moment this should be a completely harmless change.
Reviewed By: krallin
Differential Revision: D25211089
fbshipit-source-id: ea03dda82fadb7fc91a2433e12e220582ede5fb8
Summary: Remove 'static requirement for async methods of Blobstore, propagate this change and fixup low hanging fruits where the code can become 'static free easily.
Reviewed By: ahornby, farnz
Differential Revision: D24839054
fbshipit-source-id: 5d5daa04c23c4c9ae902b669b0a71fe41ee6dee6
Summary:
Apparently shards for some DBs start from 1 (production_filenodes) and for some - from 0 (production_blobstore).
This diff fixed the issue for mysql connections.
Long term we might want to query SMC for the list of shards instead of hardcoding different values in the different places.
Reviewed By: farnz
Differential Revision: D25057136
fbshipit-source-id: 9201a2ec8afe0b66a246a2ee91cc9389630f5ddf
Summary:
To ensure other parts of Mononoke can fully read new blobs as soon as they've
been written, ensure their buffers are flushed and they've been synced to disk
before returning from the blob put.
Reviewed By: krallin
Differential Revision: D24921657
fbshipit-source-id: df401470aaeeebcdc9d237271b40a399115ba25f
Summary: Update to strum 0.19 as it has improved EnumCount derivation
Reviewed By: mohanz
Differential Revision: D24680442
fbshipit-source-id: 2d3d2a84e994f09bf3b1c7ea748a80a67d100c13
Summary: As part of the effort to deprecate futures 0.1 in favor of 0.3 I want to create a new futures_ext crate that will contain some of the extensions that are applicable from the futures_01_ext. But first I need to reclame this crate name by renaming the old futures_ext crate. This will also make it easier to track which parts of codebase still use the old futures.
Reviewed By: farnz
Differential Revision: D24725776
fbshipit-source-id: 3574d2a0790f8212f6fad4106655cd41836ff74d
Summary:
In Mononoke for a sharded DB we historically used connection pool size 1 per shard. With the Mysql FFI client now it doesn't make sense, as the client's conn pool is smart enough and designed to works with sharded DBs, so currently we don't even benefit from having a pool.
In this diff I added an API to create sharded connections: a single pool is shared between all the shards.
Reviewed By: farnz
Differential Revision: D24475317
fbshipit-source-id: b7142c030a10ccfde1d5a44943b38cfa70332c6a
Summary: Change the default blobstore put behaviour to IfAbsent, so that all binaries apart from admin tools using MononokeApp::special_put_behaviour() pick up the change.
Reviewed By: farnz
Differential Revision: D24619663
fbshipit-source-id: 98439513b985d2cde88ef99e5eb177974e9db5c9
Summary:
Extend MononokeApp so admin apps can have a special default put behaviour (typically
overwrite) vs the soon to be new default of IfAbsent
Use it from the admin tools.
Reviewed By: farnz
Differential Revision: D24623094
fbshipit-source-id: 5709c68429f8e1de0535eec132998d20411fc0e6
Summary:
We want to be able to detect garbage blobs by looking at generation numbers.
Update generation numbers on put, and have a mark command exist to mark blobs as not garbage.
Reviewed By: ahornby
Differential Revision: D23989289
fbshipit-source-id: d96f38649151e3dbd5297cffc262776e74f6cc86
Summary: SQLBlob GC (next diff in stack) will need a ConfigStore in SQLBlob. Make one available to blobstore creation
Reviewed By: krallin
Differential Revision: D24460586
fbshipit-source-id: ea2d5149e0c548844f1fd2a0d241ed0647e137ae
Summary:
This updates Thrift codegen for Rust to call hooks on a context stack in a
similar fashion as the C++ codegen.
There is still a bit of room for improvement here. Notably, we don't know how
much data we actually read or sent to the client, so for now those are
hard-coded to zero in the codegen. That's better than not calling those hooks
at all (which is what happens right now), but it could stand to be improved.
Reviewed By: jsgf
Differential Revision: D24445298
fbshipit-source-id: 470daf03057424dc300b6a193668be835ae28452
Summary:
This updates multiplexedblob and logblob to capture perf counters for the
operations we run, and log them to Scuba. Along with the previous diffs in this
stack, this gives us the number of Manifold retries, total delay, and conflicts
all logged to blobstore trace on a per-operation basis.
Reviewed By: HarveyHunt
Differential Revision: D24333039
fbshipit-source-id: 9c7d0a467f8df08dcb2a0d3bb6b543cdb3ea1d90
Summary:
This updates ManifoldBlob to log the aformentioned data points to perf
counters. There's a bit of refactoring that also had to go into this to make
`ctx` available everywhere it's needed.
Reviewed By: aslpavel
Differential Revision: D24333040
fbshipit-source-id: 1b63bcd1e1ee36bae4dbbc1da053c7f1bdf96675
Summary:
Delegate chaosblob put to put_impl to remove some duplication.
This is possible as all chaosblob construction has an inner BlobstorePutOps.
Reviewed By: StanislavGlebik
Differential Revision: D24258401
fbshipit-source-id: dee9fce888e5ef9c2f34865c97921b9cc87ac3bf
Summary: Factor out common async limit code in throttledblob to throttled_access function.
Reviewed By: StanislavGlebik
Differential Revision: D24254357
fbshipit-source-id: 8f3277ccc065e3eb93f089edb2f3bd12e80ef352
Summary:
Remove BlobstorePutOps::put_behaviour and the default implementation of BlobstorePutOps::put_with_status as they did not make sense for multiplexed stores
This resolves the corresponding TODO in multiplexblob/base.rs
Reviewed By: StanislavGlebik
Differential Revision: D24162629
fbshipit-source-id: aa175bbedac473093dd1862226e910dea15c1299
Summary:
Implement BlobstorePutOps for multiplex blobstores, so restoring the ability to have multiplexes of multiplexes.
Note that this makes BlobstorePutOps::put_behaviour accessor problematic as the inner stores could have different put_behaviour, will remove it in next diff so as to keep this diff reasonable size.
Reviewed By: StanislavGlebik
Differential Revision: D24162235
fbshipit-source-id: 2ace3af5f60607996e449451316c5c0720351f82
Summary:
Add overwrite status logging to multiplexblob puts now that the inner types held all implement BlobstorePutOps
This has the effect of making any configuration that specifies a multiplex of multiplexes invalid, which is addressed in next diff in stack.
Reviewed By: StanislavGlebik
Differential Revision: D24159958
fbshipit-source-id: dd1f70a636dfb36686d796af7afd8d5da8797a23
Summary: Delegate put to put_with_status for logblob now that the blobstore factory can provide right inner type.
Reviewed By: StanislavGlebik
Differential Revision: D24109296
fbshipit-source-id: 1493421f632708d8ec24bd9f7c5cd5745f17ae68
Summary: Delegate put to put_with_status for packblob now that the blobstore factory can provide right inner type
Reviewed By: StanislavGlebik
Differential Revision: D24109288
fbshipit-source-id: 745fe3f6398833c9ac51e70f4d1c2959fceba0d5
Summary:
Make blobstore_factory PutBehaviour aware by layering all except the final multiplex as BlobstorePutOps
This makes it so all the components that go into a multiplex are BlobstorePutOps, which is a prerequisite for making the multiplex logging include the Overwrite status.
Reviewed By: StanislavGlebik
Differential Revision: D24109289
fbshipit-source-id: 23f4cedbaebadae194e41cfbff9ef46b651e3fd4
Summary:
log whether puts overwrite keys by implementing BlobstoreWithPutBehaviour for Logblob.
It logs a count of type of overwrite per put so we can sum them up.
Reviewed By: StanislavGlebik
Differential Revision: D24079272
fbshipit-source-id: 81944d92a56b0d3349ef390eb83f9e5bf4ee3d39
Summary: Added sqlblob to the common blobstore tests to make sure it doesn't diverge from expectations for other stores.
Reviewed By: StanislavGlebik
Differential Revision: D24079254
fbshipit-source-id: 49ef1c372171a1594ba298c66d3473ef682d24cc
Summary:
Add CountedBlobstoreOps so that blobstore layers that need BlobstoreOps can still use counting
This unblocks adding sqlblob to blobstore-test in next diff in stack
Reviewed By: farnz
Differential Revision: D24079256
fbshipit-source-id: 6a6505aff8c8405353a1f10d79f6e6e08911228a
Summary: Add BlobstorePutOps so that blobstore layers that need BlobstorePutOps can still use PrefixBlob as a wrapper.
Reviewed By: farnz
Differential Revision: D24109298
fbshipit-source-id: 710571e6c30fa8a432d463eedfab5fcc0389baa3
Summary:
Add predicate based PutBehaviour logic to manifoldblob.
This will prevent overwrites of keys when in IfAbsent mode, and will generate useful logging in OverwriteAndLog and IsAbsent mode.
This change factors our part of the put logic to put_check_conflict, so that it can use re-used from each of the PutBehaviour cases.
Reviewed By: StanislavGlebik
Differential Revision: D24021170
fbshipit-source-id: d2e71afadada3d5e661634449108e6c9f8dc5907
Summary:
Implement BlobstorePutOps for S3Blob. This uses is_present to check the various put behaviours
While implementing this I noticed get_sharded_key could be updated to take a reference, so I did that as well.
Differential Revision: D24079253
fbshipit-source-id: 16e194076dbdb4da8a7a9b779e0bd5fb60f550a6
Summary: Now that fileblob and memblob support put behaviour logic, update the overwrite test to check the overwrite result.
Differential Revision: D24021167
fbshipit-source-id: d9578630205cf5d79999a459cc29481968d5717d
Summary: Update memblob to be PutBehaviour aware by changing implementation from Blobstore to BlobstoreOps
Differential Revision: D24021166
fbshipit-source-id: 04dd25c5535769ea507120c1886592b808a7bbc6
Summary: Update Memblob::new callsites to ::default() in preparation for adding arguments to ::new() to specify the put behaviour desired
Differential Revision: D24021173
fbshipit-source-id: 07bf4e6c576ba85c9fa0374d5aac57a533132448
Summary: Add put behaviour handling to fileblob so that it can prevent overwrites if requested.
Differential Revision: D23933228
fbshipit-source-id: 8e74ac96b232be841174f6ad2bd2fccf92aaa90d
Summary:
Add put behaviour to BlobstoreOptions in preparation for passing in the put behaviour through blobstore_factory.
Later in the stack a command line option is added to set this non-None so that we can turn on overwrite logging for particular jobs.
Reviewed By: StanislavGlebik
Differential Revision: D24021169
fbshipit-source-id: 5692e2d3912ebde07b0d7bcce54b79df188a9f16
Summary:
This is the first part of allowing us to update mononoke blobstore put behaviour to optionally a) log when it is overwriting keys, and b) not overwrite existing keys.
Introduce BlobstorePutOps for blobstore implementations so we can track overwrite status of a put, and force an explicit PutBehaviour if required. Its intended that only blobstore implementation code and special admin tooling will need to access BlobstorePutOps methods.
Reviewed By: farnz
Differential Revision: D24021168
fbshipit-source-id: 56ae34f9995a93cf1e47fbcfa2565f236c28ae12
Summary: Remove unnecessary clone in packblob along with the Clone constraint on the inner blobstore.
Reviewed By: krallin
Differential Revision: D24109293
fbshipit-source-id: b47e68e63b6ffda95d28d974ed6883e4ae31b3a1
Summary:
We want to end up with two `put` behaviours - overwrite and do not overwrite.
Currently, SQLBlob only implements the latter, but some users assume that `put` always overwrites. Change to match Manifold
Reviewed By: aslpavel
Differential Revision: D24079501
fbshipit-source-id: f75cac81acf874337c38f82597aae645c41a319b
Summary:
Remove assert_present from Blobstore trait as it had only one callsite other than the various blobstore layers/impls.
Replaced that one last call in repo_commit.rs/assert_in_blobstore() with an equivalent call to is_present.
Reviewed By: farnz
Differential Revision: D24016927
fbshipit-source-id: 764fddbebeb4b1192d196078b8824cf8a08e9691
Summary:
This diff introduces Mysql client for Rust to Mononoke as a one more backend in the same row with raw xdb connections and myrouter. So now Mononoke can use new Mysql client connections instead of Myrouter.
To run Mononoke with the new backend, pass `--use-mysql-client` options (conflicts with `--myrouter-port`).
I also added a new target for integration tests, which runs mysql tests using mysql client.
Now to run mysql tests using raw xdb connections, you can use `mononoke/tests/integration:integration-mysql-raw-xdb` and using mysql client `mononoke/tests/integration:integration-mysql`
Reviewed By: ahornby
Differential Revision: D23213228
fbshipit-source-id: c124ccb15747edb17ed94cdad2c6f7703d3bf1a2