Summary:
Use filenodes in BlobRepo instead of linknodes.
That involves changes in commit API because we need to write filenodes not linknodes.
Also it involves changes to memblob test repos.
Note that in this diff they are basically "broken" - linknodes were replaced with empty filenodes. However looks like our unittests haven't exercised this functionality, so no unit-tests has failed. In the next diffs the correct filenodes will be added.
Reviewed By: farnz
Differential Revision: D7413481
fbshipit-source-id: 7f994ea55887b96a0eacf8e6ffdebd1f8c5f025d
Summary:
We'll need filenodes in blobimport when we'll add filenodes to the BlobRepo.
The implementation is not great - it creates a separate thread for the
filenodes (see "filenodeinserts"). New filenodes are sent via UnboundedSender
from the parsing cpupool
However it doesn't worth the effort to clean up the code that we are
going to deprecate in a couple of weeks.
Reviewed By: farnz
Differential Revision: D7429440
fbshipit-source-id: 4a9220915bd27f5c1c2028ec604afd700bb8a509
Summary:
open_or_create is useful for testing.
insert_chunk_size sets the mysql batch size
Reviewed By: farnz
Differential Revision: D7429043
fbshipit-source-id: 9d651a2f71abb9022c5390d059e4298efb49f38c
Summary: It will be used in BlobRepo, and it requires it to be Send + Sync
Reviewed By: farnz
Differential Revision: D7429058
fbshipit-source-id: 610051ba3ed7ce66dd8058e1ba9fabb3ffd12c4e
Summary:
They do not provide a lot of value, so let's not have them at all. It will make
adding filenodes easier.
Reviewed By: farnz
Differential Revision: D7428601
fbshipit-source-id: 647fa36d962cb6a8996f92246e4d900751040a52
Summary:
Our test repos had flatmanifests. Now blobimport doesn't work with flat manifests.
Generate tree manifests with `hg backfilltree`
Also run `backfilltree` as a part of regenerate script
Reviewed By: farnz
Differential Revision: D7427344
fbshipit-source-id: 134bd0caa7e2b6724664d07598718000f41c69db
Summary: There can be a lot of changesets - let's not limit ourselves to 2B
Reviewed By: farnz
Differential Revision: D7429645
fbshipit-source-id: 9501214f6f10b87b8a37cb5b9503a9d231e241be
Summary: To query all the parents of the filenode. This emulates getfiles request
Reviewed By: farnz
Differential Revision: D7427133
fbshipit-source-id: 54d054fc7a30db027aa327d3be3b93c86dd3a223
Summary: We are not using it, so there is no point in keeping it around
Reviewed By: farnz
Differential Revision: D7400428
fbshipit-source-id: 481ef3ec8ef1f188e01add36e81da789f186548e
Summary:
Previously all the copy information was ignored. In this diff we are using it.
Note that `paths` table is also used now - it is used to convert from hash of
the path to the real path.
Reviewed By: farnz
Differential Revision: D7381437
fbshipit-source-id: 13f8bf0f269cf40d06693dd0cdca15da40c1ab64
Summary:
Initial support for filenodes using diesel.
These filenodes implementation is very close to what hg stores in revlog files.
Currently this implementation doesn't support copies.
This issue will be fixed in the next diffs.
Reviewed By: farnz
Differential Revision: D7305910
fbshipit-source-id: bbb0196dbee24c3a99d58055de11d433a518e40b
Summary:
To avoid using raw NodeHash, let's add a type for a filenode.
For example, it will be used in Filenodes trait.
Reviewed By: farnz
Differential Revision: D7415873
fbshipit-source-id: 695dbe3f845d82849ea4ef7e51748be9446be772
Summary:
They would hang on any failure rather than reporting properly.
Convert to using async_unit::tokio_unit_test() to make sure unwinds are caught.
Reviewed By: StanislavGlebik
Differential Revision: D7408174
fbshipit-source-id: 39b41bd66557e0d455f87ad29a2df73768cfb9c6
Summary:
For historical reasons (back when we thought Mononoke might need to handle both flat and tree manifests natively), we decided to make the entry type in a Manifest be `MPath`, not `MPathElement`.
In the tree manifest only world we're now in, it's a bug if a Manifest contains a path instead of a path element - enforce this through the type system, so that we can't backslide accidentally.
Reviewed By: StanislavGlebik
Differential Revision: D7413870
fbshipit-source-id: fa13b676475ea8a3265fc962910542f2f555d7de
Summary:
One spot was missed when updating to allow for config repos now being
blobrepos
Reviewed By: StanislavGlebik
Differential Revision: D7414012
fbshipit-source-id: f29e499a5863e02d6d2a2412e8cdf26814950ca7
Summary: We had a fun bit of debugging because an out-of-order `oneshot::recv()` gave a `oneshot::Canceled` error. Give it an enum for errors, so that we can distinguish dropping the rx channel from calling `oneshot::recv()` before `oneshot::send()`
Reviewed By: StanislavGlebik
Differential Revision: D7382354
fbshipit-source-id: c96f4ac40449a5864b7ba79f43f9af402de7735b
Summary:
It is explictly not permitted to ask for filelogs before you've handled changesets, as they are after changesets in stream order.
Force changeset evaluation to finish before filelog evaluation begins
Reviewed By: StanislavGlebik
Differential Revision: D7382355
fbshipit-source-id: 24decd7fe223f8acbc30d42b6c85d89d5ab9edcc
Summary: Make it possible to use a simulated remote backend as a blobstore. This allows me to look at the test results and be happy that Mononoke is at fault for any slowness.
Reviewed By: kulshrax
Differential Revision: D7353229
fbshipit-source-id: 57528af704b517a70570bb2b9b140caa9120a956
Summary: Port more uses of context to the new failure extensions library.
Reviewed By: lukaspiatkowski
Differential Revision: D7350923
fbshipit-source-id: 0a91f5dcc8ce6ad00de70535c5976c78107fe22f
Summary:
The diff adds an extension of streams to the `failures_ext` crate, allowing streams with error type `failure::Error` or `failure::Fail` to store a context.
As a proof-of-concept, the resulting `context()` function is applied to a stream in use in mononoke.
Reviewed By: lukaspiatkowski
Differential Revision: D7336012
fbshipit-source-id: 822c9dcd5b6c0a60470e8fd98fecd569928be7d1
Summary:
Run changeset db operations in worker threads to make them async as
far as the rest of the system is concerned.
Reviewed By: farnz
Differential Revision: D7350002
fbshipit-source-id: 66fadf9ad2f16929e0c07a6907aa9d5f5a7075a8
Summary:
There's no point passing it by reference since callers don't need to
retain it, and the async implementation needs to move it into another context.
Reviewed By: farnz
Differential Revision: D7350001
fbshipit-source-id: 5947557a84621afae801dc20e3994496244e3a10
Summary:
Use connection pooling for mysql. We can't use it for sqlite because each
"connection" ends up being to a separate db.
Reviewed By: sid0
Differential Revision: D7350000
fbshipit-source-id: 37be71b863810c48ae69da579cf74358058f7d2f
Summary: We only need stockbookmarks now
Reviewed By: farnz
Differential Revision: D7382258
fbshipit-source-id: d8f5f5fa1424d509c9e0364529f8aeb9b4627585
Summary:
For testing, I want to be able to persist state in SQLite (so that
when we have in-process caching, I can restart and see cold cache behaviour).
This means that I want to open a database, and create the tables ignoring
failures. Make it possible
Reviewed By: StanislavGlebik
Differential Revision: D7353231
fbshipit-source-id: 0d0926c501e7951dfaf82998ed58d152e80ace7e
Summary: Very similar to serialize and deserialize for bonsai changesets.
Reviewed By: farnz
Differential Revision: D7380859
fbshipit-source-id: 42567ae3c2e52597b3708871358165ec54a8c04a
Summary: This name is less misleading. Thanks markbt for suggesting it.
Reviewed By: farnz
Differential Revision: D7380821
fbshipit-source-id: 96abb846fd8979cc322e1f176d15caf24bb00874
Summary:
Seems to flap around between 152 and 156 bytes. Not sure why, but it
isn't very relevant for the test anyway.
Reviewed By: farnz
Differential Revision: D7380504
fbshipit-source-id: 91cacb39ed0d67f1df6ac4138f4250c757b5aec3
Summary: The compact_protocol code is from jsgf.
Reviewed By: farnz
Differential Revision: D7365901
fbshipit-source-id: 4cef925ad60c579188ab016fed3f027b3944a3a1
Summary:
This also means that a direct conversion from a byte slice to a Blake2
is no longer desirable. Forcing people to go through the typed hashes seems
quite desirable.
Reviewed By: farnz
Differential Revision: D7365900
fbshipit-source-id: 36f4afd5a53f857c8952c636b0db0b4b509a6f44
Summary:
This codemod tries not to change the existing behavior of system, only introduce new types specific to Mercurial Revlogs.
It introduces a lot of copypasta intentionally and it will be cleaned in following diffs.
Reviewed By: farnz
Differential Revision: D7367191
fbshipit-source-id: 0a915f427dff431065e903b5f6fbd3cba6bc22a7
Summary:
Since none of the typed hashes use it, NULL hashes aren't going to be
used anywhere in the system.
Reviewed By: farnz
Differential Revision: D7365068
fbshipit-source-id: d67c021a6c04ca1a120264ec88943a5b35d9b8c4
Summary: We are going to use blake 2 instead of sha1.
Reviewed By: farnz
Differential Revision: D7349914
fbshipit-source-id: 2c3556cd745b39c6f86e6db430b93ac50fc273bd
Summary:
We don't want to depend on Manifold being consistent and reliable when
we're trying to fix our performance issues. Introduce a DelayBlob store that
wraps another blobstore and allows you to pass it a function that generates
delay times (e.g. a random number generator with a log-normal distribution).
The idea is that we can use DelayBlob wrapped around RocksBlob to approximate
expected Manifold behaviour, and can improve if needed.
Reviewed By: StanislavGlebik
Differential Revision: D7353228
fbshipit-source-id: fee977d2754f3134af752efc2dbfc77fedc7f2bf
Summary: The `time` crate has been deprecated in favor of `std::time` and `chrono`.
Reviewed By: jsgf
Differential Revision: D7350099
fbshipit-source-id: f150b606843d49358a17bff1c9a873c76631e23b
Summary: Scuba is able to nicely format times in microseconds but not nanoseconds. Since all of the poll times we were recording in the various stats scuba tables were all >1000ns anyway, switch to using microseconds instead.
Reviewed By: farnz
Differential Revision: D7350980
fbshipit-source-id: 0fad1ffcf21a3ee3ecfcd7062efb42fb41e0809f
Summary: Remove usage of deprecated `time` crate in `futures-stats`, and fix all callsites using the new `time-ext` crate.
Reviewed By: farnz
Differential Revision: D7349956
fbshipit-source-id: 10ef86c4942b8533a734c7daadfa895f5ef92f23
Summary:
I'm still filling this in, but thought I'd put this up for review
right now.
I haven't done the Thrift conversions yet -- those will happen soon.
Reviewed By: farnz
Differential Revision: D7300834
fbshipit-source-id: 96f5f67dbd3d6bafffa127ad93acd178292ec788
Summary:
A lot of Mononoke expects MPath instances to be non-empty, and some
Mononoke code wants empty MPaths. Have formerly-empty `MPath` instances be
represented as `None` instead.
There's a bit of a wart where we have to check for some instances being empty
even though they should never be. This is not a regression -- this is actually
a deficiency in our current type hierarchy being exposed. `Entry` can sometimes
represent a root manifest, but a root path and a root manifest will never be
joined together in practice. We may want to figure out a better way to
represent that in the future.
Also clean up a bunch of code that checks for empty `MPath` instances. That
just isn't possible any more -- yay.
Reviewed By: StanislavGlebik
Differential Revision: D7359048
fbshipit-source-id: 684f908dabaa06cb25b1dd33f4fd7bb021cb5e1b
Summary:
In this case it's super convenient to just use `RepoPath` instances,
so just do that.
The next diff will remove the ability to create `MPath` instances entirely.
Reviewed By: StanislavGlebik
Differential Revision: D7358368
fbshipit-source-id: 8915e765d7e879ec35abac9dfe3ea9653520395d
Summary:
We're going to get rid of empty MPaths very soon, so stop using them
here.
Reviewed By: StanislavGlebik, farnz
Differential Revision: D7358023
fbshipit-source-id: 2d5fff40eae03cc63ef5514faee1a8505b3b2bd0
Summary: Represent the root tree as None.
Reviewed By: farnz
Differential Revision: D7354168
fbshipit-source-id: 5d71a3bd43c19e86ecf7d53a3f721547acabe080
Summary:
Lots of places specifically want a non-empty `MPath`, and earlier those cases
weren't type-checked. Now `Option<MPath>` stands for an empty `MPath`. In the
next diff `MPath::empty()` will go away and the only way to represent an
`MPath` that doesn't exist will be with `Option<MPath>`.
Reviewed By: farnz
Differential Revision: D7350970
fbshipit-source-id: 1612aec67134e7a0ebad15dbaa93b5ea972f8ddf
Summary:
To do this we have to make changegroup part optional, because `hg push --delete
BOOK` doesn't actually send changegroup at all.
This also fixes the issue with `hg pushbackup` that sends only infinitepush
bookmarks part (see test for details)
Reviewed By: farnz
Differential Revision: D7271596
fbshipit-source-id: 77fa26e176fd4e31f62e0412249a934ee1176ee0
Summary:
Actually update the bookmarks using pushkey part. Note that bookmark deletion
doesn't currently work. It will be fixed in the next diffs
Reviewed By: farnz
Differential Revision: D7271600
fbshipit-source-id: fd13d1adfd3ea490174c31e23289e8560dc2b737
Summary:
This wireproto method is used by remotenames to update the list of
remotebookmarks. Implementation is the same as for listkeys bundle2 part.
Reviewed By: farnz
Differential Revision: D7271597
fbshipit-source-id: 8a75a93cae0e571d86d657e1c1d718a7fa0ab4ea
Summary:
Implementation of the bookmarks trait using diesel. Currently it's quite slow
because it uses mutex to make the struct sync. Also it's not really async, even
though it returns future. The plan to make it async is to move the computation
on a separate thread pool.
Reviewed By: kulshrax
Differential Revision: D7255935
fbshipit-source-id: ce622a14af777ae470aecc039cef8c6cab9db41e
Summary:
I want to see if two commits hits unexpected serialization. Test that
case, too.
Reviewed By: kulshrax
Differential Revision: D7330451
fbshipit-source-id: da80382f46bc4c1d928fa885e03460626db109c6
Summary:
I want to be able to get perf numbers for the commit API, and to avoid
human error contaminating the numbers I get. Script the test, to be run
manually.
Reviewed By: kulshrax
Differential Revision: D7261639
fbshipit-source-id: d8a39f79fd2362e4ca5e3aac652a4bf2f3cb45c9
