Summary:
tokio::run either hangs forever or reports no error at all if test fails. So
let's use async_unit
Reviewed By: farnz
Differential Revision: D7550921
fbshipit-source-id: 5114700bb932fdcd9229da760007703c3b73e11e
Summary:
In previous diff upstream select_all() was added. However it had a bug.
In
```
match self.inner.poll() {
...
Async::Ready(_) => {
return Async::Ready(None);
}
}
```
Async::Ready(None) was returned for the case when inner poll returned `Async::Ready(None)`
and for the case when inner poll returned `Async::Ready(Some(None, remaining))`.
The former was correct, however the latter wasn't. `Async::Ready(Some(None, remaining))`
happens when one of the internal streams was exhausted, however there can be
many more internal streams that are not exhausted yet. Before this diff these
non-exhausted elements would be just lost, so this diff fixes it.
Reviewed By: farnz
Differential Revision: D7550918
fbshipit-source-id: 908af9fed17744b884aa40afdccfc4654520048b
Summary:
This is a simple but useful combinator that queries lots of streams in parallel
and merges the results.
This is code is taken from futures-0.2.0, but it was modified so that it works
with futures-0.1.*. It was re-formatted with our linter.
The code had a bug that will be fixed in the next diffs
Reviewed By: farnz
Differential Revision: D7550919
fbshipit-source-id: c5b394065c0184a89dfab6a9de699725bc2bd6c2
Summary:
There's no guarantee that a symlink is a valid `MPath` -- in fact, it
frequently will be an absolute symlink which is not a valid `MPath`.
Reviewed By: lukaspiatkowski
Differential Revision: D7541672
fbshipit-source-id: c7eca54cd06e811cf54932ab8f671c13a62b5f71
Summary:
This will help in case we ever need to abandon this format in the
future.
Reviewed By: jsgf
Differential Revision: D7483425
fbshipit-source-id: 0cca098ca861cca0994c06709849ececffb38fc5
Summary:
Bump bookmark size limit - we use 512 for infinitepush, let's use the same
here. Also add a primary key
Reviewed By: farnz
Differential Revision: D7534967
fbshipit-source-id: aeef926de910a3a9934fb1588778f8f503821071
Summary: Since we have changesets, we can answer known queries more efficiently.
Reviewed By: farnz
Differential Revision: D7534911
fbshipit-source-id: 72f75c9d9a6cd7f1464660eb0ec16f94ca24f33c
Summary:
The current structure makes defining new manifests really verbose and
doesn't easily support nested manifests. Make it really simple: either accept a
path map or a description (a csv).
Also switch to a `BTreeMap` to allow lookups in the next diff.
This is unfortunately hard to separate out because all the bits are
interconnected.
Reviewed By: StanislavGlebik
Differential Revision: D7531115
fbshipit-source-id: ca0fa35d4e0ac6a4e23adb2a7d2b5679ce52b643
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary: Update the Scuba logging code in Mononoke server to also upload traces to Everstore
Reviewed By: jsgf, farnz
Differential Revision: D7492955
fbshipit-source-id: 6f0f788af33210064977894b97798bb4143caf1b
Summary: Previously, the `.timed()` combinator exported by the `futures-stats` would accept a synchronous callback to log the recorded stats. In practice, logging the stats may require blocking IO, so this combinator should be modified to accept a closure that returns a Future instead.
Reviewed By: farnz
Differential Revision: D7492957
fbshipit-source-id: c634b511e5a93a18787ab3e11ab8e1c19de7566e
Summary:
filenodes are faster than blobstore in answering `get_parents` and
`get_copy_info` queries. So let's use them instead of going to blobstore.
Reviewed By: jsgf
Differential Revision: D7485151
fbshipit-source-id: f9e2ce9dd42b3e9b18140cc3de2c9718f20ab6d4
Summary:
Previously we were able to create just sqlite filenodes, now let's make it
possible to create a mysql filenodes.
Reviewed By: jsgf
Differential Revision: D7485098
fbshipit-source-id: b9156e51d41a570f9e6aaf9eaa9e476222257bca
Summary:
The current "run_as_bundle" mode is suboptimal because it causes all
the tests to be run within the same integration runner run. This means that
testpilot can't do things like limit the number of tests run in parallel.
With this change, each test will be run as a separate `integration_runner`
instance.
Thanks to DragonMinded for help figuring this out.
Reviewed By: DragonMinded
Differential Revision: D7493887
fbshipit-source-id: 8d55c2a64573b6bc213dac3daf0d78aa3432d430
Summary:
We don't need parent_id in the index, because we'll search only using cs_id and
seq
Reviewed By: jsgf
Differential Revision: D7487275
fbshipit-source-id: 93ceb238b7d5f42e5d37c1a85eadb8e7aab5c1f4
Summary:
Not clear why this is happening, and I can't repro on my devserver, so
this debug code it is.
Reviewed By: swolchok
Differential Revision: D7480116
fbshipit-source-id: 0c9d54e22ed8ca51b27796792b0cc2045fe88f45
Summary:
Use filenodes in BlobRepo instead of linknodes.
That involves changes in commit API because we need to write filenodes not linknodes.
Also it involves changes to memblob test repos.
Note that in this diff they are basically "broken" - linknodes were replaced with empty filenodes. However looks like our unittests haven't exercised this functionality, so no unit-tests has failed. In the next diffs the correct filenodes will be added.
Reviewed By: farnz
Differential Revision: D7413481
fbshipit-source-id: 7f994ea55887b96a0eacf8e6ffdebd1f8c5f025d
Summary:
We'll need filenodes in blobimport when we'll add filenodes to the BlobRepo.
The implementation is not great - it creates a separate thread for the
filenodes (see "filenodeinserts"). New filenodes are sent via UnboundedSender
from the parsing cpupool
However it doesn't worth the effort to clean up the code that we are
going to deprecate in a couple of weeks.
Reviewed By: farnz
Differential Revision: D7429440
fbshipit-source-id: 4a9220915bd27f5c1c2028ec604afd700bb8a509
Summary:
open_or_create is useful for testing.
insert_chunk_size sets the mysql batch size
Reviewed By: farnz
Differential Revision: D7429043
fbshipit-source-id: 9d651a2f71abb9022c5390d059e4298efb49f38c
Summary: It will be used in BlobRepo, and it requires it to be Send + Sync
Reviewed By: farnz
Differential Revision: D7429058
fbshipit-source-id: 610051ba3ed7ce66dd8058e1ba9fabb3ffd12c4e
Summary:
They do not provide a lot of value, so let's not have them at all. It will make
adding filenodes easier.
Reviewed By: farnz
Differential Revision: D7428601
fbshipit-source-id: 647fa36d962cb6a8996f92246e4d900751040a52
Summary:
Our test repos had flatmanifests. Now blobimport doesn't work with flat manifests.
Generate tree manifests with `hg backfilltree`
Also run `backfilltree` as a part of regenerate script
Reviewed By: farnz
Differential Revision: D7427344
fbshipit-source-id: 134bd0caa7e2b6724664d07598718000f41c69db
Summary: There can be a lot of changesets - let's not limit ourselves to 2B
Reviewed By: farnz
Differential Revision: D7429645
fbshipit-source-id: 9501214f6f10b87b8a37cb5b9503a9d231e241be
Summary: To query all the parents of the filenode. This emulates getfiles request
Reviewed By: farnz
Differential Revision: D7427133
fbshipit-source-id: 54d054fc7a30db027aa327d3be3b93c86dd3a223
Summary: We are not using it, so there is no point in keeping it around
Reviewed By: farnz
Differential Revision: D7400428
fbshipit-source-id: 481ef3ec8ef1f188e01add36e81da789f186548e
Summary:
Previously all the copy information was ignored. In this diff we are using it.
Note that `paths` table is also used now - it is used to convert from hash of
the path to the real path.
Reviewed By: farnz
Differential Revision: D7381437
fbshipit-source-id: 13f8bf0f269cf40d06693dd0cdca15da40c1ab64
Summary:
Initial support for filenodes using diesel.
These filenodes implementation is very close to what hg stores in revlog files.
Currently this implementation doesn't support copies.
This issue will be fixed in the next diffs.
Reviewed By: farnz
Differential Revision: D7305910
fbshipit-source-id: bbb0196dbee24c3a99d58055de11d433a518e40b
Summary:
To avoid using raw NodeHash, let's add a type for a filenode.
For example, it will be used in Filenodes trait.
Reviewed By: farnz
Differential Revision: D7415873
fbshipit-source-id: 695dbe3f845d82849ea4ef7e51748be9446be772
Summary:
They would hang on any failure rather than reporting properly.
Convert to using async_unit::tokio_unit_test() to make sure unwinds are caught.
Reviewed By: StanislavGlebik
Differential Revision: D7408174
fbshipit-source-id: 39b41bd66557e0d455f87ad29a2df73768cfb9c6
Summary:
For historical reasons (back when we thought Mononoke might need to handle both flat and tree manifests natively), we decided to make the entry type in a Manifest be `MPath`, not `MPathElement`.
In the tree manifest only world we're now in, it's a bug if a Manifest contains a path instead of a path element - enforce this through the type system, so that we can't backslide accidentally.
Reviewed By: StanislavGlebik
Differential Revision: D7413870
fbshipit-source-id: fa13b676475ea8a3265fc962910542f2f555d7de
Summary:
One spot was missed when updating to allow for config repos now being
blobrepos
Reviewed By: StanislavGlebik
Differential Revision: D7414012
fbshipit-source-id: f29e499a5863e02d6d2a2412e8cdf26814950ca7
Summary: We had a fun bit of debugging because an out-of-order `oneshot::recv()` gave a `oneshot::Canceled` error. Give it an enum for errors, so that we can distinguish dropping the rx channel from calling `oneshot::recv()` before `oneshot::send()`
Reviewed By: StanislavGlebik
Differential Revision: D7382354
fbshipit-source-id: c96f4ac40449a5864b7ba79f43f9af402de7735b
Summary:
It is explictly not permitted to ask for filelogs before you've handled changesets, as they are after changesets in stream order.
Force changeset evaluation to finish before filelog evaluation begins
Reviewed By: StanislavGlebik
Differential Revision: D7382355
fbshipit-source-id: 24decd7fe223f8acbc30d42b6c85d89d5ab9edcc
Summary: Make it possible to use a simulated remote backend as a blobstore. This allows me to look at the test results and be happy that Mononoke is at fault for any slowness.
Reviewed By: kulshrax
Differential Revision: D7353229
fbshipit-source-id: 57528af704b517a70570bb2b9b140caa9120a956
Summary: Port more uses of context to the new failure extensions library.
Reviewed By: lukaspiatkowski
Differential Revision: D7350923
fbshipit-source-id: 0a91f5dcc8ce6ad00de70535c5976c78107fe22f
Summary:
The diff adds an extension of streams to the `failures_ext` crate, allowing streams with error type `failure::Error` or `failure::Fail` to store a context.
As a proof-of-concept, the resulting `context()` function is applied to a stream in use in mononoke.
Reviewed By: lukaspiatkowski
Differential Revision: D7336012
fbshipit-source-id: 822c9dcd5b6c0a60470e8fd98fecd569928be7d1
Summary:
Run changeset db operations in worker threads to make them async as
far as the rest of the system is concerned.
Reviewed By: farnz
Differential Revision: D7350002
fbshipit-source-id: 66fadf9ad2f16929e0c07a6907aa9d5f5a7075a8
Summary:
There's no point passing it by reference since callers don't need to
retain it, and the async implementation needs to move it into another context.
Reviewed By: farnz
Differential Revision: D7350001
fbshipit-source-id: 5947557a84621afae801dc20e3994496244e3a10
Summary:
Use connection pooling for mysql. We can't use it for sqlite because each
"connection" ends up being to a separate db.
Reviewed By: sid0
Differential Revision: D7350000
fbshipit-source-id: 37be71b863810c48ae69da579cf74358058f7d2f
Summary: We only need stockbookmarks now
Reviewed By: farnz
Differential Revision: D7382258
fbshipit-source-id: d8f5f5fa1424d509c9e0364529f8aeb9b4627585
Summary:
For testing, I want to be able to persist state in SQLite (so that
when we have in-process caching, I can restart and see cold cache behaviour).
This means that I want to open a database, and create the tables ignoring
failures. Make it possible
Reviewed By: StanislavGlebik
Differential Revision: D7353231
fbshipit-source-id: 0d0926c501e7951dfaf82998ed58d152e80ace7e
Summary: Very similar to serialize and deserialize for bonsai changesets.
Reviewed By: farnz
Differential Revision: D7380859
fbshipit-source-id: 42567ae3c2e52597b3708871358165ec54a8c04a
Summary: This name is less misleading. Thanks markbt for suggesting it.
Reviewed By: farnz
Differential Revision: D7380821
fbshipit-source-id: 96abb846fd8979cc322e1f176d15caf24bb00874
Summary:
Seems to flap around between 152 and 156 bytes. Not sure why, but it
isn't very relevant for the test anyway.
Reviewed By: farnz
Differential Revision: D7380504
fbshipit-source-id: 91cacb39ed0d67f1df6ac4138f4250c757b5aec3
