Summary:
It's going to be useful to share certain structures between client and server.
Looking ahead, the plan is to share the segment graph along with all the
algorithms implemented for it.
Reviewed By: StanislavGlebik
Differential Revision: D20550951
fbshipit-source-id: f498a6b0cba1bcdd35fc9720125b223d7e891a44
Summary:
`iter_segments_with_parent` has a few more conditions attached to it than the
name would imply. We are renaming it to give a better sense of its true
behavior.
Reviewed By: quark-zju
Differential Revision: D20547631
fbshipit-source-id: 406f46b9de5efc9e8e6a8c4bc22ab18fa5bc54bb
Summary:
The main question I had while writing the tests was whether we expect a
specific order for Segments for `iter_segments_with_parent`. `InProcessStore`
will return the segments in the order that they were inserted.
Reviewed By: quark-zju
Differential Revision: D20501401
fbshipit-source-id: 48ceb78f3191c7425c1488a3392cf3167f7e7268
Summary:
First 6 methods implemented from the IdDagStore trait for the InProcessStore.
Any suggestions welcome.
Reviewed By: quark-zju
Differential Revision: D20499228
fbshipit-source-id: cb536a3a0136077ada78934d82a25d079a5bc809
Summary:
If `HGPLAIN` is set, and the non-legacy graph renderer is in use, use the ASCII
renderer to ensure compatible output on all platforms and configurations.
Reviewed By: farnz
Differential Revision: D20622425
fbshipit-source-id: b25a8d0526652bab07059492f7adbb684b5cbee7
Summary:
At this point all future-returning `fn`s in `benchmark_filestore`
are async.
Reviewed By: krallin
Differential Revision: D20615922
fbshipit-source-id: 8b9998e4d8933e46f0c2612270faefcb061eae37
Summary:
This diff is a combination of an asyncification and a refactor:
- isolatedthe logic for setting `blob` (which gets overwritten
5 times, often blocking on futures in the process) into it's own
function `get_blob`, which I think makes it a lot easier to see
which variables depend on one another.
- push all the rest of the logic into an async function
`run_benchmark_filestore`
Reviewed By: krallin
Differential Revision: D20615743
fbshipit-source-id: 9d0918e4078b8193c59758701c5dc5e744298d1d
Summary: Fix test that was broken by D20557896
Reviewed By: krallin
Differential Revision: D20619387
fbshipit-source-id: 3fd7ee501144528b18b7162f73dcf3d251fd5f2f
Summary:
From time to time we're experiencing the blobstore healer to crash
because its SQL queries timing out. The rootcause of the problem
is that the same blob_key may show up on the queue many times repeatedly
and the query is trying to select all occurences.
But, the original intention of blobstore healer is to act on a single
put operation across all blobstores. To be able to identify which
puts in the healer queue are part of the same operation we need
some unique id that we'll use per such operation, let's call it OperationKey.
corresponding configerator change to create db column: D20557659
NOTE: This diff has to be landed and rolled out first, before D20557700 is rolled out. I'm assuming that after some time since rolling out this diff all the rows in the production db will have proper `operation_key` value set.
Reviewed By: krallin
Differential Revision: D20557702
fbshipit-source-id: 404d9fdea6796b38193292d1bbd4b8cd4b5b3eb8
Summary:
In the previous diffs, I asyncified the api of all `fn`s in
`aliasverify`, but I still had old stream logic, which was hard to
read and required many clones of parameters to handle static lifetime
requirements.
This diff switches all the internal code over to new-style streams.
In the process of doing this, I was able to remove almost all of the
clones, and also switch `get_bounded` from needing to explicitly return
a `Future` via and `async` block to being a standard `async fn` returning
a `Result`.
Reviewed By: krallin
Differential Revision: D20582402
fbshipit-source-id: a98bb31075a41e8d6897b6893cd51ea0c5a94767
Summary:
This diff asyncifies the APIs in `aliasverify` so that all
functions are either `async fns` or (in one case where there were
challenges dealing with lifetimes) functions returning new-style
`Future`.
This diff does not asyncify the old stream API uses inside of functions,
and the remaining issues (complicated lifetime handling and extra
`clone` calls) are all related to using the old stream api, but the
function signatures should all be forward-compatible with new futures.
Reviewed By: krallin
Differential Revision: D20577861
fbshipit-source-id: 79ca2ef8f91ee9a1be9e27d8bf6f083dbc0bb7cd
Summary:
This diff pushes `compat` down one layer in `aliasverify`, by:
- Moving the future-based portion of main into a new acync fn
`run_aliasverify`
- Using new-style futures for `run_aliasverify`
Reviewed By: krallin
Differential Revision: D20572805
fbshipit-source-id: f0363793715944963a02380e861b851905975653
Summary:
This diff moves all of the Stream operations in `statistics_collector` over
from the old futures api to the new futures-preview api, and in the process
removes all `clone` calls (except where we interface with lower-level
old-futures-based apis).
For reasons that aren't entirely clear to me, rust seems to be unable to
infer the lifetimes of some parameters in `update_statistics`, so I had to
anotate it explicitly.
Reviewed By: krallin
Differential Revision: D20583198
fbshipit-source-id: 1ea247f905649b05ebaabcf27d504d903d5e6365
Summary:
Asyncify all remaining helper functions in `statistics_collector`, in the
sense that they all are now either `async fn` or return a new `Future`
explicitly.
The remaining work is to convert the guts of a couple of functions (particularly
`update_statistics` and `get_statistics_from_changeset` are still using
old-style futures. I haven't figured out how to convert from the old stream
api to the new one, so I didn't touch those spots.
Because of a lifetime issue resolving `ctx` and `repo` - which need
to be cloned before the async scope when we know the references are
still good - it was necessary to explicitly return a `NewFuture`
in `update_statistics`. The lifteime issues also forced me to do
two levels of clones of the parameters in this function, but at least the
signature is now forward-compatible.
Reviewed By: krallin
Differential Revision: D20569773
fbshipit-source-id: c2162a1fbde6d226028b3a66d2f6526c0a268232
Summary:
Instead of explicitly running results on the Thrift thread manager
executor, use the semifuture implementation of the Thrift method which
does the same thing automatically.
Reviewed By: fanzeyi
Differential Revision: D20481540
fbshipit-source-id: 5f23c0193514fce20d80e3292bfa7402f4448e38
Summary:
Mononoke has performance counters that allow to log number of Blobstore gets and puts, Memcache hits and misses, etc. These metrics are very useful for debugging issues in production.
APIServer logs perf counters, but SCS apparently doesn't, this diff adds logging.
Reviewed By: mitrandir77
Differential Revision: D20598544
fbshipit-source-id: 5071434a90ae6bf356326a97f1d0593901912ef7
Summary: Actually use vendored Rust crates when building with CMake. The `--offline` option make sure cargo do not go to the Internet for missing crates.
Reviewed By: simpkins
Differential Revision: D20542672
fbshipit-source-id: ab4af40150c6af8b531a75f503a4fa848b3914da
Summary:
Replace `rust-crypto` with `hex`, `sha-1`, `sha2`.
- `crypto::sha1::Sha1` with `sha1::Sha1`
- `crypto::sha2::Sha2` with `sha2::Sha2`
- `crypto::digest::Digest` with `sha1::Digest` and `sha2::Digest`
- `.result_str()` with `hex::encode` and `.result()`
Reviewed By: jsgf
Differential Revision: D20588313
fbshipit-source-id: 75c4342e8b6285f0f960f864c21457a1a0808f64
Summary:
The markforrefresh only exist in the legacy Python contentstore, commiting the
current store will have the same effect for the Rust store.
Reviewed By: DurhamG
Differential Revision: D20562032
fbshipit-source-id: de12cfd65d992395c9b9894d1f05cf50d86b0191
Summary:
In a strongly typed langage, using strings should be avoided whenever possible
as they do not provide the safety guarantees that types provide.
I took the liberty of removing all the filesystems that are not relevant for
Mercurial for simplification reasons. If needs arise, we can always add a new
FsType to the enum.
Reviewed By: DurhamG
Differential Revision: D20517138
fbshipit-source-id: 0a38b53c6a87f05f4b2d664038e10c4293de96ae
Summary:
This makes it so `hg status A` won't complain about race conditions about file
`B`. As we're here, the detector logic was also moved to the filesystem layer.
Reviewed By: DurhamG
Differential Revision: D20568151
fbshipit-source-id: 713e3775a14674fd170df391977d42d9dee0f718
Summary:
If you have built the `eden/scm` subdirectory with `make local` in your
repository it will have left a bunch of build output artifacts in the source
tree. Update the getdeps shipit path map for eden to explicitly exclude many
of these build artifacts.
In particular:
- Exclude `eden/scm/build/` since this directory can contains a lot of files.
- Exclude all `*.pyc` files since there may be a reasonable number of these.
- Exclude several Cargo-related files since these will cause problems when
trying to build Rust code.
Reviewed By: chadaustin
Differential Revision: D20570720
fbshipit-source-id: a60dec4854ae470fdb58e9651fd8a3b910c76004
Summary:
Replace `rust-crypto` with `sha-1`:
- `crypto::digest::Digest` with `sha1::Digest`
- `crypto::Sha1` with `sha1::Sha1`
The interface changes slightly - no need to pass a mutable byte array when
getting the result.
Reviewed By: jsgf
Differential Revision: D20587638
fbshipit-source-id: c6c737f3f8eba94b98c728e198eb4fac12c5c80b
Summary:
Swap out `rust-crypto` for `sha-1`
- `crypto::sha1::Sha1` is replaced by `sha1::Sha1`
- `crypto::digest::Digest` is replaced by `digest::Digest`
Reviewed By: jsgf
Differential Revision: D20587685
fbshipit-source-id: 971fdaa8ce5b3e9e60db219131f6c36dcbc213d9
Summary:
Switched out the `sha` package for the `rust-crypto` package. The
apis aren't an exact match, so I had to insert a clone in place of
a modification to a mutable reference.
Reviewed By: jsgf
Differential Revision: D20585336
fbshipit-source-id: 22245157aea1115ae6f225b17b0346f0696653f7
Summary:
`test-scs.t` is very big and takes too much time to run.
I'm moving integration tests for `scsc log` to the `test-scs-log.t` file.
Reviewed By: krallin
Differential Revision: D20537505
fbshipit-source-id: 8f4a06ad4b48f34eb131d095ec21bd2d08cfe9d9
Summary: Add WalkVisitor::start_step entry point, so we have opportunity to take action before Nodes are loaded.
Reviewed By: krallin
Differential Revision: D20534842
fbshipit-source-id: 93c3e0974c99ee68b7c2e930f4b8fdea87b44bc3
Summary:
Add sampling key to LoggingContainer so that we have a way to sample low level blobstore actions for a given high level blobrepo action with the
This is will be used to track the blobs loaded when walking a repo and associate them with paths.
Walker will assign a new sampling_key to each step, and the SamplingHandler attached to SamplingBlobstore will use it correlate the get() with the Node or Path we want to track accesses for. E.g. to build a corpus of blobs per path for compression analysis
Reviewed By: mitrandir77
Differential Revision: D20534844
fbshipit-source-id: 22662b66a57ad4fef044a1108648f4ad8f2dae78
Summary:
Add a sampling blobstore for use by the walker.
Combined with the SamplingKey in next in stack it allows samplinglow level blobstore operations from hight level actions on things like BlobRepo.
Reviewed By: mitrandir77
Differential Revision: D20534843
fbshipit-source-id: 8c7378572a4f78534f66b09adfccf08b3aaa103d
Summary: use derive_more to remove some boilerplate for trivial Add/Div/Mul/Sum implementations.
Reviewed By: StanislavGlebik
Differential Revision: D20560288
fbshipit-source-id: ddab11134b133d8f29a2384e4858a35e9774e5cc
Summary:
This is a small refactoring that does two things:
1) Move StoreLoadable into the Blobstore crate. This might be a bit
controversial since StoreLoadable doesn't have anything to do with blobstore.
But I put it there for a few reasons:
* In the next diffs I want to use StoreLoadable in mononoke_types. However I
can't create dependency from `mononoke_types` crate to `manifest` crate because
`manifest` depends on `mononoke_types`
* `manifest` crate doesn't look like a good place for this struct either
* I didn't find a better place for it - don't want to put it in crates like
`common`, and it creating a separate crate just for this trait looks like an
overkill.
2) Changes error type to LoadableError to match Loadable trait. This add one
FIXME for git tree which doesn't differentiate between these errors.
Reviewed By: krallin
Differential Revision: D20561442
fbshipit-source-id: a0dca0ff8daa5d7fa166f4527c2f7cc7f541a4c0
