Summary:
This is a cleanup of NodeHash API. There were few unused methods and few ways to convert between mercurial and mercurial_types hashes. With this diff it is very easy to identify the places where this converstion happens.
A followup of this diff will be to use this new API to easily replace the NodeHash convertions in places where it requires remapping.
Reviewed By: sid0
Differential Revision: D7592876
fbshipit-source-id: 6875aa6df1a3708ce54ca5724f6eb960d179192b
Summary:
Let's fill cs table even if we import only part of the repo. This let's us
import new changesets incrementally.
That can be dangerous since we don't check if parent commits are present.
However this blobimport is a temporary measure until we get a full-fidelity
blobimport that uses a commit API.
Reviewed By: jsgf
Differential Revision: D7485495
fbshipit-source-id: 63ba91bad4eb1c1662db73293c76a506f48a4753
Summary: They are no longer used in Mononoke and they were replaced with filenodes
Reviewed By: farnz
Differential Revision: D7443319
fbshipit-source-id: 91769de0d4db9f3adf9290070d459c13aa4ed8cc
Summary: They are replaced by filenodes
Reviewed By: farnz
Differential Revision: D7443320
fbshipit-source-id: 13c7d07bc00dcbaa991663c8da8a07fcb0de1332
Summary:
This is used in a bunch of places throughout bundle2-resolver, and is
the default key that should be used throughout Mercurial.
Reviewed By: farnz
Differential Revision: D7578081
fbshipit-source-id: 8f44b54189b22b0aba8f08853bbd2bbbaa3efc5c
Summary:
This will probably go away soon, but for now I want to be able to
disambiguate the new Thrift-encoded blobs in Mononoke from these.
Reviewed By: StanislavGlebik
Differential Revision: D7565808
fbshipit-source-id: d61f3096fa368b934a923dee54a0ea1e3469ae0d
Summary:
`left` and `right` are too vague in this context, since there's
already `Added` and `Removed` which imply an explicit direction.
Reviewed By: StanislavGlebik
Differential Revision: D7560975
fbshipit-source-id: 847f600508434d62857ff60a6e6678a4426ffa88
Summary: Need this for bonsai changeset tests
Reviewed By: StanislavGlebik
Differential Revision: D7560093
fbshipit-source-id: caa0f92fa7ef1c1adb7775835615cb1cb3c7b6aa
Summary:
This is going to be useful to test out some of the work I'm doing with
generating bonsai changesets.
Reviewed By: StanislavGlebik
Differential Revision: D7555735
fbshipit-source-id: 26175244ed59ba080e1740d004d299d4d15ff593
Summary: Found while writing a test against this.
Reviewed By: StanislavGlebik
Differential Revision: D7555159
fbshipit-source-id: c6f279f41974e08ab690befcc84718c2bb22ddc0
Summary:
It increases parallelism, hence speeds up gettreepack request.
Note that currently buffer size is hardcoded. We may change it later, but
right now let's not add too many config knobs.
Reviewed By: farnz
Differential Revision: D7568349
fbshipit-source-id: d1833f1f64438da96c619746d205733af406c921
Summary:
Buffering many file requests increases parallelism.
Note that currently buffer size is hardcoded. We may change it later, but right
now let's not add too many config knobs.
Reviewed By: farnz
Differential Revision: D7568348
fbshipit-source-id: 80456d189cc21d050fcaf7d17cabf8cd59d61d63
Summary:
Trace parent fetching and linknode fetching for gettreepack
And trace blobstore calls. Note that we are interested in tracing both single
blobstore call and in the retrying manifold calls. Because there can be short
single blobstore calls that fail, but overall retrying manifold call can take a few
seconds.
Reviewed By: farnz
Differential Revision: D7550958
fbshipit-source-id: d607812f558900ee55ddd61b529b7b43e9827f49
Summary:
tokio::run either hangs forever or reports no error at all if test fails. So
let's use async_unit
Reviewed By: farnz
Differential Revision: D7550921
fbshipit-source-id: 5114700bb932fdcd9229da760007703c3b73e11e
Summary:
In previous diff upstream select_all() was added. However it had a bug.
In
```
match self.inner.poll() {
...
Async::Ready(_) => {
return Async::Ready(None);
}
}
```
Async::Ready(None) was returned for the case when inner poll returned `Async::Ready(None)`
and for the case when inner poll returned `Async::Ready(Some(None, remaining))`.
The former was correct, however the latter wasn't. `Async::Ready(Some(None, remaining))`
happens when one of the internal streams was exhausted, however there can be
many more internal streams that are not exhausted yet. Before this diff these
non-exhausted elements would be just lost, so this diff fixes it.
Reviewed By: farnz
Differential Revision: D7550918
fbshipit-source-id: 908af9fed17744b884aa40afdccfc4654520048b
Summary:
This is a simple but useful combinator that queries lots of streams in parallel
and merges the results.
This is code is taken from futures-0.2.0, but it was modified so that it works
with futures-0.1.*. It was re-formatted with our linter.
The code had a bug that will be fixed in the next diffs
Reviewed By: farnz
Differential Revision: D7550919
fbshipit-source-id: c5b394065c0184a89dfab6a9de699725bc2bd6c2
Summary:
There's no guarantee that a symlink is a valid `MPath` -- in fact, it
frequently will be an absolute symlink which is not a valid `MPath`.
Reviewed By: lukaspiatkowski
Differential Revision: D7541672
fbshipit-source-id: c7eca54cd06e811cf54932ab8f671c13a62b5f71
Summary:
This will help in case we ever need to abandon this format in the
future.
Reviewed By: jsgf
Differential Revision: D7483425
fbshipit-source-id: 0cca098ca861cca0994c06709849ececffb38fc5
Summary:
Bump bookmark size limit - we use 512 for infinitepush, let's use the same
here. Also add a primary key
Reviewed By: farnz
Differential Revision: D7534967
fbshipit-source-id: aeef926de910a3a9934fb1588778f8f503821071
Summary: Since we have changesets, we can answer known queries more efficiently.
Reviewed By: farnz
Differential Revision: D7534911
fbshipit-source-id: 72f75c9d9a6cd7f1464660eb0ec16f94ca24f33c
Summary:
The current structure makes defining new manifests really verbose and
doesn't easily support nested manifests. Make it really simple: either accept a
path map or a description (a csv).
Also switch to a `BTreeMap` to allow lookups in the next diff.
This is unfortunately hard to separate out because all the bits are
interconnected.
Reviewed By: StanislavGlebik
Differential Revision: D7531115
fbshipit-source-id: ca0fa35d4e0ac6a4e23adb2a7d2b5679ce52b643
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary: Update the Scuba logging code in Mononoke server to also upload traces to Everstore
Reviewed By: jsgf, farnz
Differential Revision: D7492955
fbshipit-source-id: 6f0f788af33210064977894b97798bb4143caf1b
Summary: Previously, the `.timed()` combinator exported by the `futures-stats` would accept a synchronous callback to log the recorded stats. In practice, logging the stats may require blocking IO, so this combinator should be modified to accept a closure that returns a Future instead.
Reviewed By: farnz
Differential Revision: D7492957
fbshipit-source-id: c634b511e5a93a18787ab3e11ab8e1c19de7566e
Summary:
filenodes are faster than blobstore in answering `get_parents` and
`get_copy_info` queries. So let's use them instead of going to blobstore.
Reviewed By: jsgf
Differential Revision: D7485151
fbshipit-source-id: f9e2ce9dd42b3e9b18140cc3de2c9718f20ab6d4
Summary:
Previously we were able to create just sqlite filenodes, now let's make it
possible to create a mysql filenodes.
Reviewed By: jsgf
Differential Revision: D7485098
fbshipit-source-id: b9156e51d41a570f9e6aaf9eaa9e476222257bca
Summary:
The current "run_as_bundle" mode is suboptimal because it causes all
the tests to be run within the same integration runner run. This means that
testpilot can't do things like limit the number of tests run in parallel.
With this change, each test will be run as a separate `integration_runner`
instance.
Thanks to DragonMinded for help figuring this out.
Reviewed By: DragonMinded
Differential Revision: D7493887
fbshipit-source-id: 8d55c2a64573b6bc213dac3daf0d78aa3432d430
Summary:
We don't need parent_id in the index, because we'll search only using cs_id and
seq
Reviewed By: jsgf
Differential Revision: D7487275
fbshipit-source-id: 93ceb238b7d5f42e5d37c1a85eadb8e7aab5c1f4
Summary:
Not clear why this is happening, and I can't repro on my devserver, so
this debug code it is.
Reviewed By: swolchok
Differential Revision: D7480116
fbshipit-source-id: 0c9d54e22ed8ca51b27796792b0cc2045fe88f45
Summary:
Use filenodes in BlobRepo instead of linknodes.
That involves changes in commit API because we need to write filenodes not linknodes.
Also it involves changes to memblob test repos.
Note that in this diff they are basically "broken" - linknodes were replaced with empty filenodes. However looks like our unittests haven't exercised this functionality, so no unit-tests has failed. In the next diffs the correct filenodes will be added.
Reviewed By: farnz
Differential Revision: D7413481
fbshipit-source-id: 7f994ea55887b96a0eacf8e6ffdebd1f8c5f025d
Summary:
We'll need filenodes in blobimport when we'll add filenodes to the BlobRepo.
The implementation is not great - it creates a separate thread for the
filenodes (see "filenodeinserts"). New filenodes are sent via UnboundedSender
from the parsing cpupool
However it doesn't worth the effort to clean up the code that we are
going to deprecate in a couple of weeks.
Reviewed By: farnz
Differential Revision: D7429440
fbshipit-source-id: 4a9220915bd27f5c1c2028ec604afd700bb8a509
Summary:
open_or_create is useful for testing.
insert_chunk_size sets the mysql batch size
Reviewed By: farnz
Differential Revision: D7429043
fbshipit-source-id: 9d651a2f71abb9022c5390d059e4298efb49f38c
Summary: It will be used in BlobRepo, and it requires it to be Send + Sync
Reviewed By: farnz
Differential Revision: D7429058
fbshipit-source-id: 610051ba3ed7ce66dd8058e1ba9fabb3ffd12c4e
Summary:
They do not provide a lot of value, so let's not have them at all. It will make
adding filenodes easier.
Reviewed By: farnz
Differential Revision: D7428601
fbshipit-source-id: 647fa36d962cb6a8996f92246e4d900751040a52
Summary:
Our test repos had flatmanifests. Now blobimport doesn't work with flat manifests.
Generate tree manifests with `hg backfilltree`
Also run `backfilltree` as a part of regenerate script
Reviewed By: farnz
Differential Revision: D7427344
fbshipit-source-id: 134bd0caa7e2b6724664d07598718000f41c69db
Summary: There can be a lot of changesets - let's not limit ourselves to 2B
Reviewed By: farnz
Differential Revision: D7429645
fbshipit-source-id: 9501214f6f10b87b8a37cb5b9503a9d231e241be
Summary: To query all the parents of the filenode. This emulates getfiles request
Reviewed By: farnz
Differential Revision: D7427133
fbshipit-source-id: 54d054fc7a30db027aa327d3be3b93c86dd3a223
Summary: We are not using it, so there is no point in keeping it around
Reviewed By: farnz
Differential Revision: D7400428
fbshipit-source-id: 481ef3ec8ef1f188e01add36e81da789f186548e
Summary:
Previously all the copy information was ignored. In this diff we are using it.
Note that `paths` table is also used now - it is used to convert from hash of
the path to the real path.
Reviewed By: farnz
Differential Revision: D7381437
fbshipit-source-id: 13f8bf0f269cf40d06693dd0cdca15da40c1ab64
