Summary:
This is a cleanup of NodeHash API. There were few unused methods and few ways to convert between mercurial and mercurial_types hashes. With this diff it is very easy to identify the places where this converstion happens.
A followup of this diff will be to use this new API to easily replace the NodeHash convertions in places where it requires remapping.
Reviewed By: sid0
Differential Revision: D7592876
fbshipit-source-id: 6875aa6df1a3708ce54ca5724f6eb960d179192b
Summary: They are replaced by filenodes
Reviewed By: farnz
Differential Revision: D7443320
fbshipit-source-id: 13c7d07bc00dcbaa991663c8da8a07fcb0de1332
Summary:
This will probably go away soon, but for now I want to be able to
disambiguate the new Thrift-encoded blobs in Mononoke from these.
Reviewed By: StanislavGlebik
Differential Revision: D7565808
fbshipit-source-id: d61f3096fa368b934a923dee54a0ea1e3469ae0d
Summary:
`left` and `right` are too vague in this context, since there's
already `Added` and `Removed` which imply an explicit direction.
Reviewed By: StanislavGlebik
Differential Revision: D7560975
fbshipit-source-id: 847f600508434d62857ff60a6e6678a4426ffa88
Summary:
There's no guarantee that a symlink is a valid `MPath` -- in fact, it
frequently will be an absolute symlink which is not a valid `MPath`.
Reviewed By: lukaspiatkowski
Differential Revision: D7541672
fbshipit-source-id: c7eca54cd06e811cf54932ab8f671c13a62b5f71
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary: Previously, the `.timed()` combinator exported by the `futures-stats` would accept a synchronous callback to log the recorded stats. In practice, logging the stats may require blocking IO, so this combinator should be modified to accept a closure that returns a Future instead.
Reviewed By: farnz
Differential Revision: D7492957
fbshipit-source-id: c634b511e5a93a18787ab3e11ab8e1c19de7566e
Summary:
filenodes are faster than blobstore in answering `get_parents` and
`get_copy_info` queries. So let's use them instead of going to blobstore.
Reviewed By: jsgf
Differential Revision: D7485151
fbshipit-source-id: f9e2ce9dd42b3e9b18140cc3de2c9718f20ab6d4
Summary:
Use filenodes in BlobRepo instead of linknodes.
That involves changes in commit API because we need to write filenodes not linknodes.
Also it involves changes to memblob test repos.
Note that in this diff they are basically "broken" - linknodes were replaced with empty filenodes. However looks like our unittests haven't exercised this functionality, so no unit-tests has failed. In the next diffs the correct filenodes will be added.
Reviewed By: farnz
Differential Revision: D7413481
fbshipit-source-id: 7f994ea55887b96a0eacf8e6ffdebd1f8c5f025d
Summary:
They do not provide a lot of value, so let's not have them at all. It will make
adding filenodes easier.
Reviewed By: farnz
Differential Revision: D7428601
fbshipit-source-id: 647fa36d962cb6a8996f92246e4d900751040a52
Summary: We are not using it, so there is no point in keeping it around
Reviewed By: farnz
Differential Revision: D7400428
fbshipit-source-id: 481ef3ec8ef1f188e01add36e81da789f186548e
Summary:
For historical reasons (back when we thought Mononoke might need to handle both flat and tree manifests natively), we decided to make the entry type in a Manifest be `MPath`, not `MPathElement`.
In the tree manifest only world we're now in, it's a bug if a Manifest contains a path instead of a path element - enforce this through the type system, so that we can't backslide accidentally.
Reviewed By: StanislavGlebik
Differential Revision: D7413870
fbshipit-source-id: fa13b676475ea8a3265fc962910542f2f555d7de
Summary: Make it possible to use a simulated remote backend as a blobstore. This allows me to look at the test results and be happy that Mononoke is at fault for any slowness.
Reviewed By: kulshrax
Differential Revision: D7353229
fbshipit-source-id: 57528af704b517a70570bb2b9b140caa9120a956
Summary:
Run changeset db operations in worker threads to make them async as
far as the rest of the system is concerned.
Reviewed By: farnz
Differential Revision: D7350002
fbshipit-source-id: 66fadf9ad2f16929e0c07a6907aa9d5f5a7075a8
Summary:
There's no point passing it by reference since callers don't need to
retain it, and the async implementation needs to move it into another context.
Reviewed By: farnz
Differential Revision: D7350001
fbshipit-source-id: 5947557a84621afae801dc20e3994496244e3a10
Summary:
This codemod tries not to change the existing behavior of system, only introduce new types specific to Mercurial Revlogs.
It introduces a lot of copypasta intentionally and it will be cleaned in following diffs.
Reviewed By: farnz
Differential Revision: D7367191
fbshipit-source-id: 0a915f427dff431065e903b5f6fbd3cba6bc22a7
Summary: Remove usage of deprecated `time` crate in `futures-stats`, and fix all callsites using the new `time-ext` crate.
Reviewed By: farnz
Differential Revision: D7349956
fbshipit-source-id: 10ef86c4942b8533a734c7daadfa895f5ef92f23
Summary:
A lot of Mononoke expects MPath instances to be non-empty, and some
Mononoke code wants empty MPaths. Have formerly-empty `MPath` instances be
represented as `None` instead.
There's a bit of a wart where we have to check for some instances being empty
even though they should never be. This is not a regression -- this is actually
a deficiency in our current type hierarchy being exposed. `Entry` can sometimes
represent a root manifest, but a root path and a root manifest will never be
joined together in practice. We may want to figure out a better way to
represent that in the future.
Also clean up a bunch of code that checks for empty `MPath` instances. That
just isn't possible any more -- yay.
Reviewed By: StanislavGlebik
Differential Revision: D7359048
fbshipit-source-id: 684f908dabaa06cb25b1dd33f4fd7bb021cb5e1b
Summary:
Lots of places specifically want a non-empty `MPath`, and earlier those cases
weren't type-checked. Now `Option<MPath>` stands for an empty `MPath`. In the
next diff `MPath::empty()` will go away and the only way to represent an
`MPath` that doesn't exist will be with `Option<MPath>`.
Reviewed By: farnz
Differential Revision: D7350970
fbshipit-source-id: 1612aec67134e7a0ebad15dbaa93b5ea972f8ddf
Summary:
Actually update the bookmarks using pushkey part. Note that bookmark deletion
doesn't currently work. It will be fixed in the next diffs
Reviewed By: farnz
Differential Revision: D7271600
fbshipit-source-id: fd13d1adfd3ea490174c31e23289e8560dc2b737
Summary:
The `Option<&MPathElement>` type is more general -- it's easy to
convert from `&Option<MPathElement>` to it, but the other way around can
require a clone.
Reviewed By: farnz
Differential Revision: D7339161
fbshipit-source-id: 0c8ab57a19bc330245c612e3e0e3651e368ab8cb
Summary: just to excersise our RocksDb blobstore which might come in handy more often than file based as it should be more efficient
Reviewed By: StanislavGlebik
Differential Revision: D7290069
fbshipit-source-id: ce776cfa14e43dc45cca796ef187655ba665d177
Summary:
To diagnose slow changeset creation, we need to know which chunk was
slow. As with upload_blob and process_entries in past diffs, measure key
phases.
The idea is that we can combine all these times (5 per changeset, 2 per blob)
to work out what needs tracing when we're looking for slowness
Reviewed By: StanislavGlebik
Differential Revision: D7195013
fbshipit-source-id: 9769ead981f02d6ecf0258190763e90762936387
Summary:
Processing entries does two things:
1. Polls all the upload futures until they're all complete.
2. Works out what entries are required to be present for this changeset.
This is the bulk of the Blobstore operations in creating a changeset, so we
know that if this is slow, then we either have a slow Blobstore, *or* we have a
bug. Further, we can combine this with the metrics for upload_blob to know
whether we're doing uploads nicely in parallel, or whether there's
serialization we didn't intended (time taken here greater than sum of time
taken in blob uploads)
Reviewed By: StanislavGlebik
Differential Revision: D7182445
fbshipit-source-id: 205b43700f28ff7e5461235e16a6955b8c597a52
Summary:
While writing Thrift deserialization code I realized there was nothing
that actually checked that MPathElement instances don't have embedded nulls or
slashes.
Reviewed By: farnz
Differential Revision: D7296838
fbshipit-source-id: 6a23d559da11e5e935e23d7b9a13f58894efaf62
Summary:
Mononoke will introduce its own ChangesetId, ManifestId and BlobHash, and it
would be good to rename these before that lands.
Reviewed By: farnz
Differential Revision: D7293334
fbshipit-source-id: 7d9d5ddf1f1f45ad45f04194e4811b0f6decb3b0
Summary:
I'm going to reuse this for unit-testing changeset timings. Make it a
macro so that I don't keep repeating myself.
Reviewed By: StanislavGlebik
Differential Revision: D7182442
fbshipit-source-id: de40e0f10892b2268c4d39cf771b7a8be6e1cf76
Summary:
I want to be able to work in a private Manifold namespace when testing
that my hunches about Manifold/Mononoke interactions are good, and when testing
changes to the ManifoldBlob that should make life better. Make that possible.
In the future, once I have a Blobstore that simulates Manifold locally, this
also makes it possible to compare the simulation to Manifold without tripping
over other users.
Reviewed By: StanislavGlebik
Differential Revision: D7194754
fbshipit-source-id: 601bf1c2ded1af5db6a123fdd05600bc3eb5d7cf
Summary: SharedError has a default implementation for std Error, but the parameter must implement std Error, that is why I wrapped failure::Error in failure::Compat. Thanks to that we can use SharedError as any other normal std Error (including the fact that it implements failure::Fail)
Reviewed By: farnz
Differential Revision: D7213459
fbshipit-source-id: 54899c64c2627dfdba276630d986a3d6007ea59a
Summary:
We want to be able to measure the time it takes to upload individual
blobs, to confirm that we don't have a concurrency issue to chase down (e.g.
blobs accidentally uploaded in series).
Measure content upload time separately, so that we know not to dive down a
rabbit hole if the measured slowness is just the time spent uploading content
Reviewed By: StanislavGlebik
Differential Revision: D7172154
fbshipit-source-id: 08729a8ffaa69a364a64f6277edfa591a8712592
Summary:
I'm going to be adding more to test the timing features, so split this
up to make code sharing easier
Reviewed By: StanislavGlebik
Differential Revision: D7172156
fbshipit-source-id: 056be70268dd1c8a37aff8e8d53342b8cea4a355
Summary: I'm going to need a logger to log future-stats output to (and later trace output). Thread one through to BlobRepo
Reviewed By: StanislavGlebik
Differential Revision: D7167450
fbshipit-source-id: 4ed729e4d448b66e491cefa19380d3be9bc99091
Summary:
We had two separate issues in blob size, which were breaking `test-eden-server.t`:
1. The value documented as "blob size without metadata" was calculated with metadata included (because we didn't strip the metadata from the content blob).
2. If the BlobNode was clean (i.e. had both data and a hash), the size was the size of the hash, not the data.
Reviewed By: StanislavGlebik
Differential Revision: D7167581
fbshipit-source-id: 965778c1d053af36c760a988f0d34130052e8a1a
Summary: Replace the generic types if `Blob` and `BlobNode` with `Bytes`.
Reviewed By: lukaspiatkowski
Differential Revision: D7115361
fbshipit-source-id: 924d347377569c6d1b3b4aed14d584510598da7b
Summary: This starts porting uses of Vec<u8> for file contents to the Bytes type.
Reviewed By: jsgf
Differential Revision: D7106766
fbshipit-source-id: 15d531836132317cede7a6f9d6b047a423deb5bb
Summary:
This is a hacky way of getting the push/pull working. We should instead remove the commits that are no longer heads from HeadStore and add those that are the new heads.
In this diff though we add all the commits as heads, just because this does not break the client
Reviewed By: farnz
Differential Revision: D7112279
fbshipit-source-id: 036f0fd230de52e96cbf4168c2cda7c2a1c5bd89
Summary: The assumption that deltas can be decoded using only bundle2 was wrong, we need to be able to fetch content of other files from the BlobRepo for delta application.
Reviewed By: StanislavGlebik
Differential Revision: D7085957
fbshipit-source-id: 2f6803d7f61389c5ba38b1207ede42579b9cf2e6
Summary: Update to include num_cpu and blake2. Also update bincode to 1.0.0
Reviewed By: StanislavGlebik
Differential Revision: D7098292
fbshipit-source-id: 67793a6f458d50fc049781f34abaf313c8ff7a79
Summary:
We want to use the Changeset table as source of truth for changesets.
Record completed changesets in the database.
Additionally, now that completion is a database matter, increase the available
parallelism marginally, by allowing the changeset to upload while its parents
are still uploading
Reviewed By: jsgf
Differential Revision: D7071700
fbshipit-source-id: 405bf84eba4a06e3d6c6299ade2fbea99edf9c19
Summary:
Provide an API to ask BlobRepo to create changesets for you from
pieces that you either have to hand, or have created via upload_entry().
Parallelism is maintained in as far as possible - if you commit N changesets,
they should all upload blobs in parallel, but the final completion future
depends on the parents, so that completion order can be maintained.
The ultimate goal of this API is to ensure that only valid commits are added to the `BlobRepo` - this means that, once the future returned by `create_changeset` resolves, you have a repo with commits and blobs in place. Until then, all the pieces can be uploaded, but are not guaranteed to be accessible to clients.
Still TODO is teaching this to use the complete changesets infra so that we
simply know which changesets are fully uploaded.
Reviewed By: StanislavGlebik
Differential Revision: D6743004
fbshipit-source-id: 813329058d85c022d75388890181b48b78d2acf3
Summary:
Changesets table gives us a relatively fast way to get generation number. Let's
use it inside repoinfo crate.
Reviewed By: sid0
Differential Revision: D7043829
fbshipit-source-id: 82557340a7f2fd6361f4e29c1a11e38e473eeb15
Summary:
Iff all the inserts finished successfully, then it's safe to mark changesets as complete.
This diff fills up changesets store after blobimport successfully finishes.
For simplicity if --commit-limit or --skip is set then we skip filling up the changeset store.
Reviewed By: sid0
Differential Revision: D7043831
fbshipit-source-id: 8ae864b45222d52281c885a49c2dca44ba577137
Summary: Changests store requires it in it's api methods. Let's pass repoid from configs
Reviewed By: farnz
Differential Revision: D7043830
fbshipit-source-id: e4e4d5852d0ca8488cabe2140555508c143ab8df
Summary:
For now it does nothing. In the next diffs it will be used to tell if commit
exists in the repo or not and to speed up revsets
Reviewed By: farnz
Differential Revision: D7043828
fbshipit-source-id: 9fcc668e68ba238123a89f18ff67828848ba0cec
Summary: This is required in further diffs where a single blob can be referenced by multiple Changesets and thus must be Sharable
Reviewed By: farnz
Differential Revision: D7056227
fbshipit-source-id: 81de95fbd933d7888347ffdfbab392b1f0398e89
Summary: BlobChangeset has some methods that are useful also outside of blobrepo crate, like getting it's node.
Reviewed By: farnz
Differential Revision: D7056035
fbshipit-source-id: 197d261fd21ab9332950d3fda401e7dab3730cb5
Summary: This is needed for the next diff where the ManifestContent is used in TreemanifestBundle2Parser
Reviewed By: StanislavGlebik
Differential Revision: D7030689
fbshipit-source-id: cf199e3f1317d8950446782de9899eb2ef97a149
Summary:
This change removes get_changeset_by_nodeid and replaces it with
get_changeset_by_changesetid, and propagates the changes to callers.
A few places still have ChangesetId::new() because I'm not sure where
the original NodeHash comes from. If you have any pointers, I would be
happy to fix them before landing.
Reviewed By: lukaspiatkowski
Differential Revision: D7031923
fbshipit-source-id: cd00ea1d2b955538e26d7b5735aed33fe0ae0330
Summary:
The current Memblob store is eager; this is great for finding certain
classes of bugs (those that assume an ordering that is not guaranteed), but not
so good for exposing other classes of bugs (those that assume that the future
has done its work before it resolves).
Add a lazy variant that functions in the same way as Memblob, but that waits
until it's polled to return.
Reviewed By: StanislavGlebik
Differential Revision: D7033792
fbshipit-source-id: 4c2d8a8150d908bcb26347757f96f99e20d74fc2
Summary:
Quick recap: gettreepack is a treemanifest wireproto method.
It's used to send treemanifest data to the client. Client sends list of
manifest nodes it's interested in (mfnodes), and list of nodes it already has
(basemfnodes). Server should find the difference and send it back. Client
usually call this wireproto method when it checks out new revision (`hg
update`), but it uses in other cases too (for example, in `hg prefetch`).
Before we supported exactly one mfnode and exactly one basemfnode. This is
usually fine for `hg update` use-case, but `hg prefetch` can request many
mfnodes and can send empty basemfnode. So let's support this too.
Note that current implementation isn't efficient. It uses at most one
basemfnode, meaning that it can send data that client already has. Also for
each mfnode we generate separate stream of changed entries. That means that the
same entries can be fetched many times.
Reviewed By: lukaspiatkowski
Differential Revision: D6923197
fbshipit-source-id: d25f9a01ca568c84811ee1a13181e70eb217eb53
Summary:
We want to be able to upload raw entries to the backing store, so that
we can assemble files + manifests into a commit. Provide the API to just upload
raw entries
Reviewed By: lukaspiatkowski
Differential Revision: D6966727
fbshipit-source-id: 8eb0dce210f2646f29b38d216c54bb60a2b1ff60
Summary:
These methods are
1) get_file_copy(node) -> returns copy info for a file (copyfrom revision and copyfrom filename). Can be None.
2) get_parents(node) -> returns parents of the node
It will be used in remotefilelog getfiles method to construct file history.
Reviewed By: jsgf
Differential Revision: D6913176
fbshipit-source-id: ba74a71926c5ca80249f8cc0f300a0aa20a65bbc
Summary:
As we discussed before, let's add get_name() method that returns MPathElement,
and remove get_path() and get_mpath().
Except for renaming, diff also make repoconfig work with tree manifest, and
fixes linknodes creation in blobimport - previously basename was used instead
of the whole path.
Reviewed By: jsgf
Differential Revision: D6857097
fbshipit-source-id: c09f3ff40d38643bd44aee8b4488277d658cf4f6
Summary:
Give them correct names, because this functions actually fetch node from
blobstore and strip copy metadata if it exists.
Reviewed By: farnz
Differential Revision: D6871271
fbshipit-source-id: 4d8c83786b223f5cd4f55188fbcb8f438713d3b7
Summary:
Currently there is no easy way to create Entry that represents root manifest.
More importantly, `get_mpath()` fails if internal RepoPath is root. This diff
fixes this issue and adds method to create root entry.
It will be used in `gettreepack` wireproto method implementation
Reviewed By: jsgf
Differential Revision: D6845033
fbshipit-source-id: 0999db89441e9ab783b8bca8047b01e4c6832187
Summary: It will be used by many things, in particular by gettreepack method
Reviewed By: jsgf
Differential Revision: D6845036
fbshipit-source-id: 3174dfafeeab830ededb70ddff3dd62f6a608d54
Summary: Change BlobChangeset and callers to use ChangesetId instead of NodeId
Reviewed By: lukaspiatkowski
Differential Revision: D6835450
fbshipit-source-id: 7b20359837632aef4803e40965380c38f54c9d0a
Summary: Update the bookmarks module to use ChangesetId to represent bookmarks, rather than NodeHash.
Reviewed By: lukaspiatkowski
Differential Revision: D6774650
fbshipit-source-id: 1742e4e78798ad68a7f17ebd345eef14a7de2cec
Summary:
This will (in practice) always be `Arc<Blobstore>`, so push down and
make it the concrete type.
Reviewed By: jsgf
Differential Revision: D6761690
fbshipit-source-id: c279751d98a8882c209b79faa7cbfd3e635beda5
Summary:
We're never going to serve RevlogRepo in production, and we're down to
a single BlobRepo type that will have different backing stores. Remove the
unused trait, and use BlobRepo everywhere bar blobimport and repo_config
(because we previously hardcoded revlog here - we want to change to a BlobRepo
once blobimport is full-fidelity).
Reviewed By: jsgf
Differential Revision: D6596164
fbshipit-source-id: ba6e76e78c495720792cbe77ae6037f7802ec126
Summary:
Remove the last associated types from BlobStore - this means that
BlobStore now has an associated trait object type.
Reviewed By: jsgf
Differential Revision: D6425414
fbshipit-source-id: 7186dab9b56593dd1d70be732d4ad56d1e7b3c63
Summary:
Don't use failure's bail!() and ensure!() macros.
Instead, failure_ext provides:
- bail_err!(err) - Converts its single parameter to the expected error and returns; ie `return Err(From::from(err));`
- bail_msg!(fmt, ...) - takes format string parameters and returns a `failure::err_msg()` error
- ensure_err!(), ensure_msg!() - corresponding changes
Also:
- remove all stray references to error-chain
- remove direct references to failure_derive (it's reexported via failure and failure_ext)
- replace uses of `Err(foo)?;` with `bail_err!()` (since `bail_err` unconditionally returns, but `Err(x)?` does not in principle, which can affect type inference)
Reviewed By: kulshrax
Differential Revision: D6507717
fbshipit-source-id: 635fb6f8c96d185b195dff171ea9c8db9e83af10
Summary:
Convert scm/mononoke to use failure, and update common/rust crates it depends on as well.
What it looks like is a lot of deleted code...
General strategy:
- common/rust/failure_ext adds some things that are in git failure that aren't yet in crates.io (`bail!` and `ensure!`, `Result<T, Error>`)
- everything returns `Result<T, failure::Error>`
- crates with real error get an error type, with a derived Fail implementation
- replicate error-chain by defining an `enum ErrorKind` where the fields match the declared errors in the error! macro
- crates with dummy error-chain (no local errors) lose it
- `.chain_err()` -> `.context()` or `.with_context()`
So far the only place I've needed to extract an error is in a unit test.
Having a single unified error type has simplified a lot of things, and removed a lot of error type parameters, error conversion, etc, etc.
Reviewed By: sid0
Differential Revision: D6446584
fbshipit-source-id: 744640ca2997d4a85513c4519017f2e2e78a73f5
Summary:
BlobStore is entirely generic, and puts no limits on its
implementations. Remove ValueIn and ValueOut type parameters, and insist that
all blobs are Bytes (as per production setups)
Reviewed By: StanislavGlebik
Differential Revision: D6425413
fbshipit-source-id: 455e526d8baebd0d0f1906941648acca89be4881
Summary:
BlobStore is entirely generic, and puts no limits on its
implementations. Remove the "Key" type parameter, and insist that all keys are
String (as per production setups)
Reviewed By: StanislavGlebik
Differential Revision: D6425412
fbshipit-source-id: 1f1229bf8e001bf780964e883c6beb071e9ef1d8
Summary:
As part of removing excess genericism, make Bookmarks a trait with no
associated types or type parameters. I will revisit error types here once I
have removed the Repo trait - these are not ideal, but enough to get things
going
Reviewed By: StanislavGlebik
Differential Revision: D6405315
fbshipit-source-id: 814c106612a061e1766f1ea9a9428a13a73bd007
Summary:
Returning the bookmarks object gets in the way of degenericising
bookmarks. It's also not in line with other methods on the Repo trait - the
repo handles querying the underlying storage, not the user.
Switch to providing pass-through interfaces for bookmarks.
Reviewed By: StanislavGlebik
Differential Revision: D6408644
fbshipit-source-id: 2808850a070b7bcc478cd40d824bdc8d3acb8b0f
Summary:
As part of removing excess genericism, make Heads a trait with no
associated types or type parameters.
Reviewed By: StanislavGlebik
Differential Revision: D6352727
fbshipit-source-id: df9ef87e0e0abe43c30e7318da38d7f930c37c6e
Summary: We are going to use them to fetch not only file blobs, but also tree blobs.
Reviewed By: kulshrax
Differential Revision: D6336010
fbshipit-source-id: 77fcc45698a447c10d180baf929465f1d7e4ee72
Summary: Also support them in `generate_memblob_repo.py`.
Reviewed By: jsgf
Differential Revision: D6215721
fbshipit-source-id: c8b855bbc74b619bc093b3aac449a283a1ad33ec
Summary:
Bookmarks and heads are not used in eden server, but user still have to specify
--blobrepo-folder. Let's use MemHeads and MemBookmarks instead to avoid
specifying this parameter
Reviewed By: sid0
Differential Revision: D6099149
fbshipit-source-id: 26927bdd5c5dba2164688e72d238587f5535ccfc
Summary:
`RepoPath` represents any absolute path -- root, directory or file. There's a
lot of code that manually switches between directory and file entries --
abstract all of that away.
Reviewed By: farnz
Differential Revision: D6201383
fbshipit-source-id: 0047023a67a5484ddbdd00bb57bca3bfb7d4dd3f
Summary:
I spent too much time debugging something that was made a lot easier
once I used `impl Future`.
Reviewed By: StanislavGlebik
Differential Revision: D6094810
fbshipit-source-id: 6bdcf6159a87df56eb61d85a51ade31397ff6994
Summary:
Manifold requires a separate thread to do the IO, so we create a separate
detached thread.
Reviewed By: sid0
Differential Revision: D6063729
fbshipit-source-id: a67d1da955aa2cd34e923a0762bbb76c858728b1
Summary:
Blobrepo state with file-based bookmarks and heads and manifold blobstore.
It will be used for eden server testing.
Reviewed By: sid0
Differential Revision: D6063728
fbshipit-source-id: f0542265af015d5c20db225ed6bf85cae954a3ab
Summary:
Mercurial filelog entries may have metadata fields in the beginning, usually used to track copies/renames. Previously mononoke eden server returned this metadata as part of the file blob.
This diff changes it. Now `get_content()` method returns file content without metadata, and to make it consistent, both `get_content()` of the blobrepo and revlog repo do the same.
This decision certainly has it's tradeoffs, because now it's more difficult to get metadata (`get_raw_content` needs to be used).
But we'll probably change how metadata is stored anyway, that's why I think changing `get_content` method is fine.
This diff also cleans up server/src/main.rs file, because previously it had to strip metadata itself.
Also diff fixes problem in metadata parsing - it previously failed if file is less than 2 bytes
Reviewed By: farnz
Differential Revision: D5901476
fbshipit-source-id: f3ade0179710352590068c238e6a733aab68a512
Summary:
`Path` has the potential to be confused with `std::path::Path`.
`MPath` is nice, concise, and clearly different from `Path`.
Reviewed By: jsgf
Differential Revision: D5895665
fbshipit-source-id: dc5ed5c3866b227d753c6d904d3c6d213c882cd7
Summary: file blobstore was read data from incorrect folders. This diff fixes it.
Reviewed By: farnz
Differential Revision: D5852736
fbshipit-source-id: 1bc08788023ff2cc9d631985d70329475afc589f
Summary:
We're going to add more stores like obsmarkers and linknodes very
soon. Stuffing all of them into type parameters is going to get ugly very soon.
Instead, add a trait which represents all the state, and make `BlobRepo`
generic on that trait. Add a few implementations for common use cases like
files and RocksDB-based stores.
This allows the dependency lists for the Mononoke and Eden servers to be more
straightforward.
Reviewed By: jsgf
Differential Revision: D5850372
fbshipit-source-id: 37a0687611687b9616ebbddce70f53e1e5d3267b
Summary:
Just like the previous diff with bookmarks, every user needs this and it makes
sense anyway.
Reviewed By: jsgf
Differential Revision: D5847877
fbshipit-source-id: b91a49e94da2d7e207061b6c52b78c55a2229dec
Summary: All its users need this, and it makes sense.
Reviewed By: jsgf
Differential Revision: D5847753
fbshipit-source-id: 042c7b5637e5f83918fa9bc5d131c8745d7bbc3e
Summary: Also ensure that `blobimport` doesn't use its own copy.
Reviewed By: jsgf
Differential Revision: D5847604
fbshipit-source-id: 5390848cd5fab8abd967ef9701720491d703c0f1
Summary:
Removes usage of BoxFuture and BoxStream from futures crate and instead use
it from internal futures_ext lib.
Reviewed By: sid0
Differential Revision: D5847771
fbshipit-source-id: 01823513176add37caa73c0ea2810e80d7fae955
Summary:
Finally got an update working by removing the `mysql_async` crate.
Some notes:
* The `mysql_async` crate was responsible in this case: see https://github.com/rust-lang/cargo/issues/4066#issuecomment-328210204 for why.
* tokio/futures deprecated a bunch of stuff. I've filed a TODO for now.
* We finally pulled in error-chain 0.11, which has a bunch of nice improvements.
Reviewed By: kulshrax
Differential Revision: D5798282
fbshipit-source-id: a38a7b17ee0205428e2ea63334722aa408582493
Summary:
`:` is a reserved character for Windows paths, so Mercurial rejects
them from being committed. Use `-` instead, so that we can commit file blob
repo test fixtures.
Reviewed By: kulshrax
Differential Revision: D5731525
fbshipit-source-id: 8d14fc03f1b135cbc4d42aeaf2f3a0ae6d13f956
Summary: Would be nice to be able to return future that references a boxed Entry
Reviewed By: jsgf
Differential Revision: D5688859
fbshipit-source-id: acb7bcc461f886bca08b84f31dbebe4f692187f1
Summary: Main part is `get_stream_of_manifest_entries` that creates a stream of all tree manifest entries by recursively going through all of them.
Reviewed By: jsgf
Differential Revision: D5622490
fbshipit-source-id: 4a8b2707df0300a37931c465bafb1ed54d6d4d25
Summary:
Adds method similar to https://doc.rust-lang.org/std/path/struct.Path.html#method.join.
Will be used in the next diff to prepend path prefix to the tree manifest entries
Reviewed By: jsgf
Differential Revision: D5563808
fbshipit-source-id: 3637275093c301bf159083cb9bfedaa0e490a75e
Summary:
A preparation step before blob importing of tree manifest repos to blobrepo.
`get_parents()` method of BlobEntry reads parents from the blobstore. It works fine for file entries because file entries can stores its parents in the blobstore. With tree manifests BlobEntry can contain also tree manifest entries, and that means that tree manifest entries parents should also be stored somewhere in the blobstore.
I suggest to use the same logic for the tree manifest entries as for the file entries. File and manifest entries have two blobstore entries - one stores hash of the content and parents, another stores the actual content.
To do this I moved `RawNodeBlob` and `get_node()` to the separate module and made fields public.
Reviewed By: jsgf
Differential Revision: D5622342
fbshipit-source-id: c9f0c446107d4697b042544ff8b37a159064f061
Summary:
Let's create a separate function `fetch_file_blob_from_blobstore` that will be
used to get content of the blob without creating file::BlobEntry.
file::BlobEntry requires Path and type parameters that won't be passed to eden
server.
Reviewed By: jsgf
Differential Revision: D5537647
fbshipit-source-id: 13fb2fbba7d3068b88aac2d842cc92b4da6ca80c
Summary: Will be used by eden server in the next diff
Reviewed By: jsgf
Differential Revision: D5553955
fbshipit-source-id: 196e2da597d0456b4337a6d9ada8fe188075e87e
Summary: Add helpers to allow an implementation of a Repo object to be boxed up into a type-erased trait object.
Reviewed By: sid0
Differential Revision: D5540673
fbshipit-source-id: c16332f6a548d053125dd3376e7e06d0ddffadf9
Summary: Add a `get_bookmarks()` method to the Repo trait and implement it for RevlogRepo and BlobRepo.
Reviewed By: sid0
Differential Revision: D5540667
fbshipit-source-id: 710849e50b1d358be1c4f95a1cbd61efb7c2cf6b
Summary: Blobrepo can take any Heads or Blobstore implementation, each of which may have their own error type. Error chain doesn't support type parameters, and can only have foreign links for specific concrete types. To handle this we have a couple of placeholder errors in Blobrepo's errors which are chained onto the real underlying error.
Reviewed By: sid0
Differential Revision: D5540676
fbshipit-source-id: 9543e204754f22dfd68a04aec5a484fa85d53ff8
Summary:
blobimport appends ".bincode" in the end of the file.
Let's do the same in blobrepo
Reviewed By: sid0
Differential Revision: D5527661
fbshipit-source-id: d54fe52c1d282f038a47ac2f2cd0aaf917a92227
