Summary:
Wrapped Blobstore in a CensoredBlob, with the purpose of checking for
censored commits.
Reviewed By: StanislavGlebik
Differential Revision: D15432210
fbshipit-source-id: 7136196d6ac8e03faabe7c9568c47f77cfe6e801
Summary:
Implemented a no-op censoreblob struct. The functionality is to be
implemented
Reviewed By: farnz
Differential Revision: D15393101
fbshipit-source-id: 9b1989474d71029c5aa65521a07c0b474e48cfe3
Summary: `rustfmt` wants to reformat. Let it.
Reviewed By: StanislavGlebik
Differential Revision: D15353606
fbshipit-source-id: ac79274982c2c9a32288bcbc447dcc1966774144
Summary:
In the case of mononoke's admin tool it's annoying for users to be required to run myrouter in the background and provide myrouter port to every command.
Thanks to this change it is no longer necessary to run admin commands through myrouter - the tool will simply use a direct connection to XDB using the sql crate.
It is important to note that the raw XDB connection via sql crate doesn't have connection pooling and doesn't handle XDB failover so it is crucial that it is never used for long-lived or request heavy use cases like running mononoke server or blobimport
Reviewed By: jsgf
Differential Revision: D15174538
fbshipit-source-id: 299d3d7941ae6aec31961149f926c2a4965ed970
Summary:
Convert to `2018` version and remove `#[macro_use]` and `extern crate` from following folders:
- blobrepo
- blobstore
- mercurial_types
- mononoke_types
Reviewed By: StanislavGlebik
Differential Revision: D14827294
fbshipit-source-id: 4b8a2edb5d9ad1fe75083e172cd0da359fa6dc2e
Summary:
Add a blobstore type which always returns an error for
administratively disabled blobstores.
Reviewed By: StanislavGlebik
Differential Revision: D14682588
fbshipit-source-id: 66d06a6066051371ad5fcd5934ac4824885d0237
Summary: Update Rust toolchain to 1.33.0 with fixes to make our code compatible with 1.33.0.
Reviewed By: Imxset21, kulshrax
Differential Revision: D14608312
fbshipit-source-id: 2d9cf7d01692abaed32f9adffa0e5eb51cfacb4f
Summary: Slim down the blobstore trait crate as much as possible.
Reviewed By: aslpavel
Differential Revision: D14542675
fbshipit-source-id: faf09255f7fe2236a491742cd836226474f5967c
Summary:
Apparently Gluster really can't copy with renames, so write to a
unique name, then symlink the canonical key to it. If the symlink already
exists, then we'll assume that the file already does, and remove the unique
name.
Reviewed By: StanislavGlebik
Differential Revision: D14014167
fbshipit-source-id: 1e5e2ce989652232d67d2aaac776e35127f58fb0
Summary: Keep track of error causes so we can print better errors.
Reviewed By: StanislavGlebik
Differential Revision: D14014165
fbshipit-source-id: e9a7846256bbfbfd689e0d78f01b1ac50cf64c1b
Summary:
Reduce directory fan out levels to 2 wide ones (xxx/xxx) rather than 4 narrow
(xx/xx/xx/xx). This saves write latency cost: with 4 levels, until we get
~billions of blobs, every write will also be a mkdir, and we'll have many
single-entry directories. With 2 wide levels this will be true up to ~millions of
blobs, but then start to fill out the leaf directories.
The downside is that the leaf directory loading will be higher once we do have
billions of objects, but I think it will be managable.
This is NOT INTEROPERABLE with 4-level fanout, so invalidates existing stores.
Reviewed By: aslpavel
Differential Revision: D13923991
fbshipit-source-id: b93ddc49305c921f8a1b34606796b6d75273fb75
Summary:
Adding `.data` to the end breaks the "rsync hack" to keep files colocated on
the same node.
Reviewed By: aslpavel
Differential Revision: D14014284
fbshipit-source-id: c65b237ed03c9c8ea3a6ac239135820379b7e38d
Summary:
Previously, CountedBlobstore used many dynamic_timeseries in its
stats structure to allow for a dynamic key based upon blobstore name. Replace
this with a stats struct that allows a dynamic prefix. This simplfies the
stats code for CountedBlobstore and also allows for a String to be used
as a prefix.
Differential Revision: D14084622
fbshipit-source-id: 1c6bdbe63e5c4f6e9f0428c9b3cdfdbed475b877
Summary:
Add a timeout to the get() and put() operations so that we don't leave
connections to blobstores open for too long in the case of a pathological
blobstore that sends or receives data extremely slowly.
Reviewed By: StanislavGlebik
Differential Revision: D14022738
fbshipit-source-id: 5a6ea53bcdbffdf2befdc8f9912ec3f46b3dd466
Summary:
Previously we didn't log any errors for reads as we were cancelling
outstanding futures for read requests as soon as one completed.
Modify this behaviour so that every time we decide to sample a request, we
allow all blobstore gets() to complete and record the data. As a result
of this, we will also record the errors.
Reviewed By: StanislavGlebik
Differential Revision: D14022739
fbshipit-source-id: 62b878a08b5c4f6a2ed9efe71b2f97cf66c58256
Summary:
We don't want to log too much, we're only interested in a general
trend (i.e. is this blobstore returning loads of errors) rather than specific
blobstore operations.
Only log 1/100 requests, although we can tweak this in the future if required.
Also, fix up a typo in an error message.
Reviewed By: StanislavGlebik
Differential Revision: D14022740
fbshipit-source-id: e62e97fb4e4dc551a3249318092ee55937e7b4e2
Summary: Tool that is used to populate blobstore synchronization queue from manifold bucket
Reviewed By: StanislavGlebik
Differential Revision: D13880216
fbshipit-source-id: 6527e4f7027dc8ce8e73f9949e080e53eda1c541
Summary:
Rather than just talking to a single host, make connections to as many machines
as possible, and randomly fan out operations to them.
Reviewed By: lukaspiatkowski
Differential Revision: D13915510
fbshipit-source-id: fb8a8c2c0a0abaa9ba0a39d4702c0d81fbe590f6
Summary:
This avoids making blobstore depend on mononoke-types, which itself
has complex dependencies.
Reviewed By: StanislavGlebik
Differential Revision: D13944481
fbshipit-source-id: ade338a03efee2107256a91f6b622df98246a11f
Summary: Move complex things out of `blobstore` to thin out the dep graph. This didn't work as well as I'd hoped because `blobstore`->`mononoke-types`->`thrift`/`sql`.
Reviewed By: StanislavGlebik
Differential Revision: D13915511
fbshipit-source-id: c210dda23fa7102168c0ca14a035ed5c03a6993c
Summary:
There are two warnings introduced because of the upgrade. Fixing in these.
```
warning: use of deprecated item 'failure::Error::cause': please use 'as_fail()' method instead
--> tools/xar/clean_xar_mounts/src/main.rs:582:38
|
582 | writeln!(stderr, "{}, {}", e.cause(), e.backtrace()).expect(errmsg);
| ^^^^^
|
= note: #[warn(deprecated)] on by default
warning: the feature `int_to_from_bytes` has been stable since 1.32.0 and no longer requires an attribute to enable
--> scm/mononoke/blobstore/glusterblob/src/lib.rs:8:12
|
8 | #![feature(int_to_from_bytes)] // stable 1.32
| ^^^^^^^^^^^^^^^^^
|
= note: #[warn(stable_features)] on by default
```
Differential Revision: D13951745
fbshipit-source-id: c86ca37d0a3344a4f8c492dd9d8703fe2a3e1647
Summary:
Currently if a crate depends even on a single type from metaconfig then in
order to compile this trait buck first compiles metaconfig crate with all the
logic of parsing the configs.
This diff split metaconfig into two crates. The first one just holds the types for
"external consumption" by other crates. The second holds the parsing logic.
That makes builds faster
Reviewed By: jsgf, lukaspiatkowski
Differential Revision: D13877592
fbshipit-source-id: f353fb2d1737845bf1fa0de515ff8ef131020063
Summary:
The test is ignored by default because it would probably be
flakey/slowThe test is ignored by default because it would probably be
flakey/slow.
Reviewed By: aslpavel
Differential Revision: D13795070
fbshipit-source-id: 8459d34126917e0fce3151ec2f4e68e639d2e7e6
Summary: Added tw_handle to be able to distinguish source of samples
Reviewed By: ikostia
Differential Revision: D13800968
fbshipit-source-id: 6c8528fc69302b2d5c5fbd40ccdf729c9379a101
Summary: Simple stats for put/get/is_present so we can monitor basic functionality.
Reviewed By: StanislavGlebik
Differential Revision: D13736062
fbshipit-source-id: 1fd6def0d98d9d29dfbbf11430006b91936d35dc
Summary:
Use rustfix and some hand-editing to convert to Rust 2018 - main
benefit is the elimination of a lot of `extern crate` lines.
Reviewed By: HarveyHunt
Differential Revision: D13736087
fbshipit-source-id: 50e4edfdff3e8ceea94b2beed36399681871a436
Summary: as opposed to other blobstores, this one is using memcache directly, because it even stores chunks in the cache
Reviewed By: aslpavel
Differential Revision: D13487613
fbshipit-source-id: bf9eeaef4d795e4f2322f128fb8501ace619d8f1
Summary: MyRouter and cache support will come in next diffs
Reviewed By: StanislavGlebik
Differential Revision: D13465146
fbshipit-source-id: 0ede0e875d4a49794ff71173bd0d1563abb3ff08
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary:
Now that Rust macros can be `use`d like normal symbols, `stats` can
simply import the `lazy_static!` macro without requiring its users to do it.
Reviewed By: Imxset21
Differential Revision: D13281897
fbshipit-source-id: a6780fbace07dd784308e642d4a384322a17c367
Summary:
Prepare for gluster blobstore by making the test framework properly
async. It needs a full tokio environment (IO reactor, etc) for async operation, rather than
just `Future::wait`.
Reviewed By: StanislavGlebik
Differential Revision: D13055656
fbshipit-source-id: 6263ff706b3a374f0868f05989b6880f3c5a16ed
Summary: Add `with_smc` to allow making a connection using an SMC tier name.
Reviewed By: kulshrax
Differential Revision: D12989046
fbshipit-source-id: bdf9ef4729ac8f90a6a0ac16fb3ce9669d0bcf52
Summary:
This implements the Blobstore trait using an underlying Gluster volume (or
really any NFS server, but some aspects are tailored to Gluster).
Reviewed By: lukaspiatkowski
Differential Revision: D12989045
fbshipit-source-id: 76b4fe87ed1f392342c31be12115eee972221ca0
Summary:
cachelib can throw exceptions from any method. Make all our bindings
exception-safe
Reviewed By: jsgf
Differential Revision: D9158062
fbshipit-source-id: 57948aa6ead304fac6e037621a1f6641826345c5
Summary:
I had enough seeing compiler giving me error messages with `blobrepo::Error` whereas it should be `failure::Error`.
`pub type` is exporting a type alias whereas `pub use` is re-exporting. The later apparently will fool the compiler to think that they own the thing they are re-exporting which in this case, failure::Error. If it is for the purpose of declaring the error type the crate is using, type alias is obviously more suitable as it won't take the ownership of the type.
Reviewed By: lukaspiatkowski
Differential Revision: D8926275
fbshipit-source-id: 6b1346adf52bec6b666a3f8b23e7a4ea4e0ab070
