Summary:
This is because these Mercurial entries are (at least currently) going
to be stored as they come in, and this data structure is entirely in the
Mercurial domain.
Reviewed By: lukaspiatkowski
Differential Revision: D7664972
fbshipit-source-id: 9de5475eed0d7ab7085c29fd0282f205043cfe5a
Summary:
The list of arguments is becoming too long, and I need to add even
more here.
Reviewed By: StanislavGlebik, farnz
Differential Revision: D7652096
fbshipit-source-id: 62a4631e163e95cf5c950a949e72facab629ea54
Summary:
The base type is better because it can represent dates from before
1970 as well.
Reviewed By: StanislavGlebik
Differential Revision: D7652095
fbshipit-source-id: 6d66a06e18ba28e13e70b9f0e921acbd3d55baaf
Summary:
Now that `BlobNode` no longer returns `None`:
* don't expose the `BlobNode` API outside the crate because it turns out to not be very useful (it should probably go away eventually?)
* make the `File` API not return `Option` types
* Add a new `file_contents` that returns a brand-new `FileContents` (this is the first time we're tying together Mercurial and Mononoke data structures!)
Also remove a `Symlink` API that isn't really correct honestly.
Reviewed By: StanislavGlebik
Differential Revision: D7624729
fbshipit-source-id: 38443093b8bfea91384c959f3425cf355fac9f65
Summary:
This is not only the newer, more specific type -- it also makes a couple
of upcoming diffs more straightforward.
Reviewed By: StanislavGlebik
Differential Revision: D7622906
fbshipit-source-id: 4e453b827512c538f4f9777ae4d24627f3b124cf
Summary: mercurial_types::DEntryId should be replaced by types from mononoke_types or mercurial in most cases. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619571
fbshipit-source-id: bf8d81ec9ffe6a5525d923d7ee67d8e92498aa4d
Summary: mercurial_types::DManifestId should be replaced by types from mononoke_types in most cases and by mercurial::HgManifestId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619062
fbshipit-source-id: 447224194c6555334b64dc29ebabe3ef0d0cb87e
Summary: mercurial_types::DChangesetId should be replaced by types from mononoke_types in most cases and by mercurial::HgChangesetId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7618897
fbshipit-source-id: 78904f57376606be99b56662164e0c110e632c64
Summary: mercurial_types::NodeHash should be replaced by types from mononoke_types in most cases and by mercurial::NodeHash in others. This rename should help with tracking this fact.
Reviewed By: sid0
Differential Revision: D7618389
fbshipit-source-id: a876e723d911df626c7851fba56a056843b4e049
Summary: They are replaced by filenodes
Reviewed By: farnz
Differential Revision: D7443320
fbshipit-source-id: 13c7d07bc00dcbaa991663c8da8a07fcb0de1332
Summary:
This will probably go away soon, but for now I want to be able to
disambiguate the new Thrift-encoded blobs in Mononoke from these.
Reviewed By: StanislavGlebik
Differential Revision: D7565808
fbshipit-source-id: d61f3096fa368b934a923dee54a0ea1e3469ae0d
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary:
They do not provide a lot of value, so let's not have them at all. It will make
adding filenodes easier.
Reviewed By: farnz
Differential Revision: D7428601
fbshipit-source-id: 647fa36d962cb6a8996f92246e4d900751040a52
Summary:
Run changeset db operations in worker threads to make them async as
far as the rest of the system is concerned.
Reviewed By: farnz
Differential Revision: D7350002
fbshipit-source-id: 66fadf9ad2f16929e0c07a6907aa9d5f5a7075a8
Summary: Remove usage of deprecated `time` crate in `futures-stats`, and fix all callsites using the new `time-ext` crate.
Reviewed By: farnz
Differential Revision: D7349956
fbshipit-source-id: 10ef86c4942b8533a734c7daadfa895f5ef92f23
Summary:
The `Option<&MPathElement>` type is more general -- it's easy to
convert from `&Option<MPathElement>` to it, but the other way around can
require a clone.
Reviewed By: farnz
Differential Revision: D7339161
fbshipit-source-id: 0c8ab57a19bc330245c612e3e0e3651e368ab8cb
Summary:
To diagnose slow changeset creation, we need to know which chunk was
slow. As with upload_blob and process_entries in past diffs, measure key
phases.
The idea is that we can combine all these times (5 per changeset, 2 per blob)
to work out what needs tracing when we're looking for slowness
Reviewed By: StanislavGlebik
Differential Revision: D7195013
fbshipit-source-id: 9769ead981f02d6ecf0258190763e90762936387
Summary:
Processing entries does two things:
1. Polls all the upload futures until they're all complete.
2. Works out what entries are required to be present for this changeset.
This is the bulk of the Blobstore operations in creating a changeset, so we
know that if this is slow, then we either have a slow Blobstore, *or* we have a
bug. Further, we can combine this with the metrics for upload_blob to know
whether we're doing uploads nicely in parallel, or whether there's
serialization we didn't intended (time taken here greater than sum of time
taken in blob uploads)
Reviewed By: StanislavGlebik
Differential Revision: D7182445
fbshipit-source-id: 205b43700f28ff7e5461235e16a6955b8c597a52
Summary:
While writing Thrift deserialization code I realized there was nothing
that actually checked that MPathElement instances don't have embedded nulls or
slashes.
Reviewed By: farnz
Differential Revision: D7296838
fbshipit-source-id: 6a23d559da11e5e935e23d7b9a13f58894efaf62
Summary:
Mononoke will introduce its own ChangesetId, ManifestId and BlobHash, and it
would be good to rename these before that lands.
Reviewed By: farnz
Differential Revision: D7293334
fbshipit-source-id: 7d9d5ddf1f1f45ad45f04194e4811b0f6decb3b0
Summary:
I'm going to reuse this for unit-testing changeset timings. Make it a
macro so that I don't keep repeating myself.
Reviewed By: StanislavGlebik
Differential Revision: D7182442
fbshipit-source-id: de40e0f10892b2268c4d39cf771b7a8be6e1cf76
Summary:
We want to be able to measure the time it takes to upload individual
blobs, to confirm that we don't have a concurrency issue to chase down (e.g.
blobs accidentally uploaded in series).
Measure content upload time separately, so that we know not to dive down a
rabbit hole if the measured slowness is just the time spent uploading content
Reviewed By: StanislavGlebik
Differential Revision: D7172154
fbshipit-source-id: 08729a8ffaa69a364a64f6277edfa591a8712592
Summary:
I'm going to be adding more to test the timing features, so split this
up to make code sharing easier
Reviewed By: StanislavGlebik
Differential Revision: D7172156
fbshipit-source-id: 056be70268dd1c8a37aff8e8d53342b8cea4a355
Summary: I'm going to need a logger to log future-stats output to (and later trace output). Thread one through to BlobRepo
Reviewed By: StanislavGlebik
Differential Revision: D7167450
fbshipit-source-id: 4ed729e4d448b66e491cefa19380d3be9bc99091
Summary: Replace the generic types if `Blob` and `BlobNode` with `Bytes`.
Reviewed By: lukaspiatkowski
Differential Revision: D7115361
fbshipit-source-id: 924d347377569c6d1b3b4aed14d584510598da7b
Summary: This starts porting uses of Vec<u8> for file contents to the Bytes type.
Reviewed By: jsgf
Differential Revision: D7106766
fbshipit-source-id: 15d531836132317cede7a6f9d6b047a423deb5bb
Summary:
Provide an API to ask BlobRepo to create changesets for you from
pieces that you either have to hand, or have created via upload_entry().
Parallelism is maintained in as far as possible - if you commit N changesets,
they should all upload blobs in parallel, but the final completion future
depends on the parents, so that completion order can be maintained.
The ultimate goal of this API is to ensure that only valid commits are added to the `BlobRepo` - this means that, once the future returned by `create_changeset` resolves, you have a repo with commits and blobs in place. Until then, all the pieces can be uploaded, but are not guaranteed to be accessible to clients.
Still TODO is teaching this to use the complete changesets infra so that we
simply know which changesets are fully uploaded.
Reviewed By: StanislavGlebik
Differential Revision: D6743004
fbshipit-source-id: 813329058d85c022d75388890181b48b78d2acf3
Summary: Changests store requires it in it's api methods. Let's pass repoid from configs
Reviewed By: farnz
Differential Revision: D7043830
fbshipit-source-id: e4e4d5852d0ca8488cabe2140555508c143ab8df
Summary:
For now it does nothing. In the next diffs it will be used to tell if commit
exists in the repo or not and to speed up revsets
Reviewed By: farnz
Differential Revision: D7043828
fbshipit-source-id: 9fcc668e68ba238123a89f18ff67828848ba0cec
Summary:
The current Memblob store is eager; this is great for finding certain
classes of bugs (those that assume an ordering that is not guaranteed), but not
so good for exposing other classes of bugs (those that assume that the future
has done its work before it resolves).
Add a lazy variant that functions in the same way as Memblob, but that waits
until it's polled to return.
Reviewed By: StanislavGlebik
Differential Revision: D7033792
fbshipit-source-id: 4c2d8a8150d908bcb26347757f96f99e20d74fc2
Summary:
We want to be able to upload raw entries to the backing store, so that
we can assemble files + manifests into a commit. Provide the API to just upload
raw entries
Reviewed By: lukaspiatkowski
Differential Revision: D6966727
fbshipit-source-id: 8eb0dce210f2646f29b38d216c54bb60a2b1ff60