Summary:
Profiling showed that since we are inserting objects into blobstore
sequentially it takes a lot of time for long stacks of commit. Let's do it in
parallel.
Note that we are still inserting sequentially into changesets table
Reviewed By: farnz
Differential Revision: D9683037
fbshipit-source-id: 8f9496b97eaf265d9991b94243f0f14133f463da
Summary: - Make `Bookmakrs` work with `ChangsetId` instead of `HgChangesetId`
Reviewed By: StanislavGlebik, farnz
Differential Revision: D9297139
fbshipit-source-id: e18661793d144669354e509271044410caa3502a
Summary:
Alas, the diff is huge. One thing is changing Changesets to use ChangesetId.
This is actually quite straightforward. But in order to do this we need to
adapt our test fixtures to also use bonsai changesets. Modifying existing test
fixtures to work with bonsai changesets is very tricky. Besides, existing test
fixtures is a big pile of tech debt anyway, so I used this chance to get rid of
them.
Now test fixtures use `generate_new_fixtures` binary to generate an actual Rust
code that creates a BlobRepo. This Rust code creates a bonsai changeset, that
is converted to hg changeset later.
In many cases it results in the same hg hashes as in old test fixtures.
However, there are a couple of cases where the hashes are different:
1) In the case of merge we are generating different hashes because of different
changed file list (lukaspiatkowski, aslpavel, is it expected?). this is the case for test
fixtures like merge_even, merge_uneven and so on.
2) Old test fixtures used flat manifest hashes while new test fixtures are tree
manifest only.
Reviewed By: jsgf
Differential Revision: D9132296
fbshipit-source-id: 5c4effd8d56dfc0bca13c924683c19665e7bed31
Summary:
So the fixtures were broken -- there was apparently a bug that would
cause p2 of root tree manifests to always be null. Fix them by regenerating
trees.
Also the directory name the tarball got extracted to had hyphens in it instead
of underscores. Fix that to make it all underscores.
I used the following commands to do this -- apologies for the hacky zsh:
```
$ for tar in *.tar.gz; do (x=${tar//.tar.gz/}; cp $x.tar.gz ~/local/tmp &&
pushd ~/local/tmp && rm -rf $x ${x//_/-} && tar xf $x.tar.gz && mv ${x//_/-} $x
&& pushd $x/.hg/store && rm 00manifesttree.i && hg backfilltree && popd && tar
cf $x.tar.gz $x && popd && cp ~/local/tmp/$x.tar.gz .); done
$ buck run scm/mononoke/tests/fixtures:regenerate -- *.tar.gz --overwrite
```
Reviewed By: farnz
Differential Revision: D8854625
fbshipit-source-id: 08be7a1e89150b70a308343b43b299a0030708b9
Summary: This will be useful for the bonsai verification tool.
Reviewed By: StanislavGlebik
Differential Revision: D8792562
fbshipit-source-id: f409d0fa042528b04462a1539fd3c2a8064a4f6e
Summary:
Manifests are always able to return entries immediately, and never
fail.
Reviewed By: lukaspiatkowski, farnz
Differential Revision: D8556499
fbshipit-source-id: e21a2522f1219e47db9b55b24b6ac6c0c463933e
Summary:
This will also allow file blob sharing between the Mercurial and Mononoke
data models.
Reviewed By: farnz
Differential Revision: D8440330
fbshipit-source-id: a29cd07dcecf0959dffb74b7428f3cb11fbd3db6
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.
Reviewed By: farnz
Differential Revision: D8416238
fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
Summary:
Pretty straightforward. Also using this opportunity to add per-repo
prefixes, since all the hashes are going to break anyway.
Note for reviewers: almost all the change is regenerated test fixtures (unfortunately necessary to make atomic). The actual substantive changes are all in the first few files.
Reviewed By: farnz
Differential Revision: D8392234
fbshipit-source-id: c93fc8c6388cb00fe5cff95646ad8c853581cb8c
Summary:
Mostly this was about adding support for file stores to
`new_blobimport`.
Reviewed By: StanislavGlebik
Differential Revision: D8372063
fbshipit-source-id: 2e3791c6222ec430015008f038e1df0464d3f0ba
Summary:
Mononoke hasn't written out heads in a while, so use the topology file
instead.
Reviewed By: StanislavGlebik
Differential Revision: D8371751
fbshipit-source-id: f9af9dddc3349a41c5e9f3a67033bcb3c739f511
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary:
This will make it easier to change the "real" bookmark type from AsciiString to
String if we decide to do that.
BookmarkPrefix is a separate type because we may want to change it from
AsciiString to String. Also we don't want to confuse a bookmark prefix with a
bookmark name.
Reviewed By: jsgf
Differential Revision: D7909992
fbshipit-source-id: 3d4d075c204ed5ef1114a743430982c2836bac04
Summary: the idea of Mercurial heads in Mononoke will be represented by bookmarks, so there is no need to have them around
Reviewed By: StanislavGlebik
Differential Revision: D7775032
fbshipit-source-id: 1618a1e51862d7c115b2955082f40ee890a045f1
Summary: Although this change is breaking with the traditional definition of heads in Mercurial, we decided that using bookmarks as heads is sufficient for fbsource use case as well as it matches the git model where every unnamed branch is garbage collected.
Reviewed By: jsgf
Differential Revision: D7745541
fbshipit-source-id: 2ec716db7799bf7bab8b2b77a85d1521a880f3df
Summary:
Currently, any sort of `Bytes` can be stored in the blobstore. That
caused me to make several mistakes while writing the code to store bonsai
changesets, because I'd just end up storing the wrong set of `Bytes`.
Introduce stronger typing so that only types that explicitly implement
`BlobStorable` can be stored in the blobstore.
Currently, these sorts of blobs can be stored in the blob store:
* `ChangesetBlob` and `ContentBlob` in `mononoke-types` (these are Thrift-serialized structures)
* The envelope `RawNodeBlob` and `RawCSBlob` types in `blobrepo`, once converted to `EnvelopeBlob` instances
* `HgBlob`, which contains revlog data (manifests or files) exactly as serialized by Mercurial
Reviewed By: StanislavGlebik
Differential Revision: D7627290
fbshipit-source-id: d1bcbde8881e365dec99618556e7b054985bccf7
Summary:
This is not only the newer, more specific type -- it also makes a couple
of upcoming diffs more straightforward.
Reviewed By: StanislavGlebik
Differential Revision: D7622906
fbshipit-source-id: 4e453b827512c538f4f9777ae4d24627f3b124cf
Summary: mercurial_types::DChangesetId should be replaced by types from mononoke_types in most cases and by mercurial::HgChangesetId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7618897
fbshipit-source-id: 78904f57376606be99b56662164e0c110e632c64
Summary: mercurial_types::NodeHash should be replaced by types from mononoke_types in most cases and by mercurial::NodeHash in others. This rename should help with tracking this fact.
Reviewed By: sid0
Differential Revision: D7618389
fbshipit-source-id: a876e723d911df626c7851fba56a056843b4e049
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary:
Use filenodes in BlobRepo instead of linknodes.
That involves changes in commit API because we need to write filenodes not linknodes.
Also it involves changes to memblob test repos.
Note that in this diff they are basically "broken" - linknodes were replaced with empty filenodes. However looks like our unittests haven't exercised this functionality, so no unit-tests has failed. In the next diffs the correct filenodes will be added.
Reviewed By: farnz
Differential Revision: D7413481
fbshipit-source-id: 7f994ea55887b96a0eacf8e6ffdebd1f8c5f025d
Summary:
They do not provide a lot of value, so let's not have them at all. It will make
adding filenodes easier.
Reviewed By: farnz
Differential Revision: D7428601
fbshipit-source-id: 647fa36d962cb6a8996f92246e4d900751040a52
Summary:
Our test repos had flatmanifests. Now blobimport doesn't work with flat manifests.
Generate tree manifests with `hg backfilltree`
Also run `backfilltree` as a part of regenerate script
Reviewed By: farnz
Differential Revision: D7427344
fbshipit-source-id: 134bd0caa7e2b6724664d07598718000f41c69db
Summary:
For historical reasons (back when we thought Mononoke might need to handle both flat and tree manifests natively), we decided to make the entry type in a Manifest be `MPath`, not `MPathElement`.
In the tree manifest only world we're now in, it's a bug if a Manifest contains a path instead of a path element - enforce this through the type system, so that we can't backslide accidentally.
Reviewed By: StanislavGlebik
Differential Revision: D7413870
fbshipit-source-id: fa13b676475ea8a3265fc962910542f2f555d7de
Summary:
Run changeset db operations in worker threads to make them async as
far as the rest of the system is concerned.
Reviewed By: farnz
Differential Revision: D7350002
fbshipit-source-id: 66fadf9ad2f16929e0c07a6907aa9d5f5a7075a8
Summary:
There's no point passing it by reference since callers don't need to
retain it, and the async implementation needs to move it into another context.
Reviewed By: farnz
Differential Revision: D7350001
fbshipit-source-id: 5947557a84621afae801dc20e3994496244e3a10
Summary:
Mononoke will introduce its own ChangesetId, ManifestId and BlobHash, and it
would be good to rename these before that lands.
Reviewed By: farnz
Differential Revision: D7293334
fbshipit-source-id: 7d9d5ddf1f1f45ad45f04194e4811b0f6decb3b0
Summary:
These are types that are going to be used throughout Mononoke -- I'm hoping to
avoid references to current Mercurial data structures in here.
For now, the only module I've moved is part of the `path` module. The
Mercurial-specific `fsencode` bits have been kept in mercurial-types (though
maybe they should move to `mercurial`?)
In the future, this module will also include definitions for unodes, etc.
Reviewed By: jsgf
Differential Revision: D7188722
fbshipit-source-id: fc097ca12c38a787f83e35af9b8dd308f2b910ea
Summary: I'm going to need a logger to log future-stats output to (and later trace output). Thread one through to BlobRepo
Reviewed By: StanislavGlebik
Differential Revision: D7167450
fbshipit-source-id: 4ed729e4d448b66e491cefa19380d3be9bc99091
Summary: Replace the generic types if `Blob` and `BlobNode` with `Bytes`.
Reviewed By: lukaspiatkowski
Differential Revision: D7115361
fbshipit-source-id: 924d347377569c6d1b3b4aed14d584510598da7b
Summary:
Iff all the inserts finished successfully, then it's safe to mark changesets as complete.
This diff fills up changesets store after blobimport successfully finishes.
For simplicity if --commit-limit or --skip is set then we skip filling up the changeset store.
Reviewed By: sid0
Differential Revision: D7043831
fbshipit-source-id: 8ae864b45222d52281c885a49c2dca44ba577137
Summary: Changests store requires it in it's api methods. Let's pass repoid from configs
Reviewed By: farnz
Differential Revision: D7043830
fbshipit-source-id: e4e4d5852d0ca8488cabe2140555508c143ab8df
Summary:
For now it does nothing. In the next diffs it will be used to tell if commit
exists in the repo or not and to speed up revsets
Reviewed By: farnz
Differential Revision: D7043828
fbshipit-source-id: 9fcc668e68ba238123a89f18ff67828848ba0cec
Summary: BlobChangeset has some methods that are useful also outside of blobrepo crate, like getting it's node.
Reviewed By: farnz
Differential Revision: D7056035
fbshipit-source-id: 197d261fd21ab9332950d3fda401e7dab3730cb5
Summary:
This change removes get_changeset_by_nodeid and replaces it with
get_changeset_by_changesetid, and propagates the changes to callers.
A few places still have ChangesetId::new() because I'm not sure where
the original NodeHash comes from. If you have any pointers, I would be
happy to fix them before landing.
Reviewed By: lukaspiatkowski
Differential Revision: D7031923
fbshipit-source-id: cd00ea1d2b955538e26d7b5735aed33fe0ae0330
Summary:
The current Memblob store is eager; this is great for finding certain
classes of bugs (those that assume an ordering that is not guaranteed), but not
so good for exposing other classes of bugs (those that assume that the future
has done its work before it resolves).
Add a lazy variant that functions in the same way as Memblob, but that waits
until it's polled to return.
Reviewed By: StanislavGlebik
Differential Revision: D7033792
fbshipit-source-id: 4c2d8a8150d908bcb26347757f96f99e20d74fc2
Summary:
Add a fixture where there are many files and directories.
History is linear for now, but later we may add merge commtis.
It will be used to test gettreepack method.
Reviewed By: lukaspiatkowski
Differential Revision: D6818544
fbshipit-source-id: 798a1bea66eb49491f98fe5e997b172f3c927c53
Summary:
We're never going to serve RevlogRepo in production, and we're down to
a single BlobRepo type that will have different backing stores. Remove the
unused trait, and use BlobRepo everywhere bar blobimport and repo_config
(because we previously hardcoded revlog here - we want to change to a BlobRepo
once blobimport is full-fidelity).
Reviewed By: jsgf
Differential Revision: D6596164
fbshipit-source-id: ba6e76e78c495720792cbe77ae6037f7802ec126
Summary:
BlobStore is entirely generic, and puts no limits on its
implementations. Remove ValueIn and ValueOut type parameters, and insist that
all blobs are Bytes (as per production setups)
Reviewed By: StanislavGlebik
Differential Revision: D6425413
fbshipit-source-id: 455e526d8baebd0d0f1906941648acca89be4881
Summary:
As part of removing excess genericism, make Bookmarks a trait with no
associated types or type parameters. I will revisit error types here once I
have removed the Repo trait - these are not ideal, but enough to get things
going
Reviewed By: StanislavGlebik
Differential Revision: D6405315
fbshipit-source-id: 814c106612a061e1766f1ea9a9428a13a73bd007
