Summary:
Support b2x:infinitepush part. It contains changegroup v2, so just reuse the
normal push path for it.
Note that pushbackup still fails because we don't support
b2x:infinitepushscratchbookmarks part. Also all wireproto params are ignored
for b2x:infinitepush.
Reviewed By: farnz
Differential Revision: D7086120
fbshipit-source-id: 2f98e5d59059ca3c2b82842c98e6dc771c70c6f0
Summary:
Asyncmemo re-implementation using Shared future. The reasons for doing it are:
1 Now it's safe to drop MemoFuture
2 Now there will be no deadlocks where one future evict another future from the
cache, and then the second future evicts the first futures and so on.
Since Shared returns SharedError, I had to do a hack to work around that. More
details in the code comments.
Reviewed By: farnz
Differential Revision: D7099852
fbshipit-source-id: e08cb859ef06f28339ac9314edb8e53e92480874
Summary: This starts porting uses of Vec<u8> for file contents to the Bytes type.
Reviewed By: jsgf
Differential Revision: D7106766
fbshipit-source-id: 15d531836132317cede7a6f9d6b047a423deb5bb
Summary:
This is a hacky way of getting the push/pull working. We should instead remove the commits that are no longer heads from HeadStore and add those that are the new heads.
In this diff though we add all the commits as heads, just because this does not break the client
Reviewed By: farnz
Differential Revision: D7112279
fbshipit-source-id: 036f0fd230de52e96cbf4168c2cda7c2a1c5bd89
Summary: Now both RepoPath and MPath are showing human readable paths when printed
Reviewed By: farnz
Differential Revision: D7112280
fbshipit-source-id: ba7698d2ffa90a125176d70d665cdef669d63b93
Summary:
From upload_entry perspective NodeHash is not unique for uploaded entry, but when combined with RepoPath it is. An example are two files with the same content, but different paths.
Additionally in this diff the requirement for uploaded blobs to be unique is loosened for Manifests, because the b2xtreegroup might contain duplicates of Manifests
Reviewed By: farnz
Differential Revision: D7087863
fbshipit-source-id: 7e9c2438db037fa171f1e65b6882b445e8c09f7a
Summary: It seems that my assumtion that Filelog End section follow every Filelog Chunk section was wrong, because there might be multiple Filelog Chunk sections being different version of the same file from different commits.
Reviewed By: farnz
Differential Revision: D7073007
fbshipit-source-id: 3baee3fd29c77aabf4173a618509de7ff88e4de6
Summary: The assumption that deltas can be decoded using only bundle2 was wrong, we need to be able to fetch content of other files from the BlobRepo for delta application.
Reviewed By: StanislavGlebik
Differential Revision: D7085957
fbshipit-source-id: 2f6803d7f61389c5ba38b1207ede42579b9cf2e6
Summary: This will make it easy to track down an error without looking at stacktraces
Reviewed By: StanislavGlebik
Differential Revision: D7087301
fbshipit-source-id: c5460dae9c5c9ab43713e5db1457f5d9155b5e8e
Summary: Update to include num_cpu and blake2. Also update bincode to 1.0.0
Reviewed By: StanislavGlebik
Differential Revision: D7098292
fbshipit-source-id: 67793a6f458d50fc049781f34abaf313c8ff7a79
Summary:
We want to use the Changeset table as source of truth for changesets.
Record completed changesets in the database.
Additionally, now that completion is a database matter, increase the available
parallelism marginally, by allowing the changeset to upload while its parents
are still uploading
Reviewed By: jsgf
Differential Revision: D7071700
fbshipit-source-id: 405bf84eba4a06e3d6c6299ade2fbea99edf9c19
Summary:
Provide an API to ask BlobRepo to create changesets for you from
pieces that you either have to hand, or have created via upload_entry().
Parallelism is maintained in as far as possible - if you commit N changesets,
they should all upload blobs in parallel, but the final completion future
depends on the parents, so that completion order can be maintained.
The ultimate goal of this API is to ensure that only valid commits are added to the `BlobRepo` - this means that, once the future returned by `create_changeset` resolves, you have a repo with commits and blobs in place. Until then, all the pieces can be uploaded, but are not guaranteed to be accessible to clients.
Still TODO is teaching this to use the complete changesets infra so that we
simply know which changesets are fully uploaded.
Reviewed By: StanislavGlebik
Differential Revision: D6743004
fbshipit-source-id: 813329058d85c022d75388890181b48b78d2acf3
Summary:
Changesets table gives us a relatively fast way to get generation number. Let's
use it inside repoinfo crate.
Reviewed By: sid0
Differential Revision: D7043829
fbshipit-source-id: 82557340a7f2fd6361f4e29c1a11e38e473eeb15
Summary:
Iff all the inserts finished successfully, then it's safe to mark changesets as complete.
This diff fills up changesets store after blobimport successfully finishes.
For simplicity if --commit-limit or --skip is set then we skip filling up the changeset store.
Reviewed By: sid0
Differential Revision: D7043831
fbshipit-source-id: 8ae864b45222d52281c885a49c2dca44ba577137
Summary: Changests store requires it in it's api methods. Let's pass repoid from configs
Reviewed By: farnz
Differential Revision: D7043830
fbshipit-source-id: e4e4d5852d0ca8488cabe2140555508c143ab8df
Summary:
For now it does nothing. In the next diffs it will be used to tell if commit
exists in the repo or not and to speed up revsets
Reviewed By: farnz
Differential Revision: D7043828
fbshipit-source-id: 9fcc668e68ba238123a89f18ff67828848ba0cec
Summary: The introduced commits feature things like adding blob that should already be present, adding blob twice and having non-linear history
Reviewed By: farnz
Differential Revision: D7071447
fbshipit-source-id: e26b7792808351e2380d68b9eb3a4d7e6e859b0e
Summary: This is required in further diffs where a single blob can be referenced by multiple Changesets and thus must be Sharable
Reviewed By: farnz
Differential Revision: D7056227
fbshipit-source-id: 81de95fbd933d7888347ffdfbab392b1f0398e89
Summary: BlobChangeset has some methods that are useful also outside of blobrepo crate, like getting it's node.
Reviewed By: farnz
Differential Revision: D7056035
fbshipit-source-id: 197d261fd21ab9332950d3fda401e7dab3730cb5
Summary:
Diesel looks quite promising overall, as long as we can ensure async operations
work.
Some basic notes:
* This just looks async but isn't really -- doing that would need a way to send
queries over to a thread pool. (cc kulshrax)
* It is unfortunate that we must use a macro for the exact same `add` code across
MySQL and SQLite. There doesn't appear to be a way out though -- there is no
trait bound that we can apply to the `add` method here.
This is currently pretend-async -- it would be good to make this really async, but that should wait for a future diff IMO
Reviewed By: StanislavGlebik
Differential Revision: D6932028
fbshipit-source-id: ac1cfde3b96bb790f00b73d49c6d90ed128f0255
Summary:
These functions don't need ownership, and the commit API wants to be
able to pass the same manifest in multiple times. Use a reference instead of a
Box.
Reviewed By: lukaspiatkowski
Differential Revision: D7043872
fbshipit-source-id: 5ea1b87962b07f8ebecc2f64989339a133e52fa5
Summary: this parsed ManifestContent will be used in later diffs to determine the graph of dependencies between BlobEntries and Changesets
Reviewed By: StanislavGlebik
Differential Revision: D7030688
fbshipit-source-id: 243d28f7964e8247ac989a1736210457dd3eab96
Summary: This is needed for the next diff where the ManifestContent is used in TreemanifestBundle2Parser
Reviewed By: StanislavGlebik
Differential Revision: D7030689
fbshipit-source-id: cf199e3f1317d8950446782de9899eb2ef97a149
Summary:
The DeltaCache structure will hold onto the full content of all filelogs during a push.
This might potentially lead to memory spikes during large pushes or pushes of large files.
The basic telemetry added in this diff will track how does this structure memory usage changes.
Reviewed By: farnz
Differential Revision: D7025207
fbshipit-source-id: 61b8359e4a9a05b6ddb5ec6a7bc4ce9e58752f1b
Summary: This code handles deltaed filelogs and resolves them into proper Filelogs
Reviewed By: farnz
Differential Revision: D7011702
fbshipit-source-id: e8dc4844657011bc1085463eedd1790b87d317dc
Summary:
Return a reply to a client so that it doesn't fail.
Reply consists of just one replychangegroup that tells a client that the push
has succeeded (even though currently it wasn't).
resolve() function returns future of Bytes instead of a stream. It may be
suboptimal, but should be fine for now, and it's already used by a few
wireproto methods.
Reviewed By: lukaspiatkowski
Differential Revision: D7010578
fbshipit-source-id: 9b5425b912c640d4e2bac957a02e9881813b8871
Summary:
This change removes get_changeset_by_nodeid and replaces it with
get_changeset_by_changesetid, and propagates the changes to callers.
A few places still have ChangesetId::new() because I'm not sure where
the original NodeHash comes from. If you have any pointers, I would be
happy to fix them before landing.
Reviewed By: lukaspiatkowski
Differential Revision: D7031923
fbshipit-source-id: cd00ea1d2b955538e26d7b5735aed33fe0ae0330
Summary:
The current Memblob store is eager; this is great for finding certain
classes of bugs (those that assume an ordering that is not guaranteed), but not
so good for exposing other classes of bugs (those that assume that the future
has done its work before it resolves).
Add a lazy variant that functions in the same way as Memblob, but that waits
until it's polled to return.
Reviewed By: StanislavGlebik
Differential Revision: D7033792
fbshipit-source-id: 4c2d8a8150d908bcb26347757f96f99e20d74fc2
Summary:
The default implementation of `assert_present` only asserted that no
errors were found accessing the blobstore. Update it to fail if the blob is
absent.
Reviewed By: StanislavGlebik
Differential Revision: D7033794
fbshipit-source-id: 596d3b0d3d55d05a5b96760c311defe5a05af634
Summary:
We prefer to use Option<NodeHash> to represent a reference which may
be missing, whereas Mercurial file formats and wire protocols tend to use
NULL_HASH. Add some converters to make this simple.
libstd already has a global `impl From<T> for Option<T>` so we can't specialize
that here. Add a specific `into_option` method instead.
Reviewed By: StanislavGlebik
Differential Revision: D7016479
fbshipit-source-id: 1a65f8c2b8c252692a443aa7c5dd5976ad1e24b1
Summary:
Now resolve() function accepts stream of Bundle2Item and it doesn't return
remainder. Remainder is parsed in hgproto. This makes unbundle() similar to
other hgproto methods.
Note that resolve() returns future of () - this is temporary and it'll be
changed in the next diff.
Reviewed By: jsgf
Differential Revision: D7010577
fbshipit-source-id: f4bde495d0ab8785476559f65b0165dbd09e4ba5
Summary: I commited the sin of overengineering here and in the next diff I would remove some of these stuff, so doing it here to make the next diff easier to read
Reviewed By: StanislavGlebik
Differential Revision: D7002559
fbshipit-source-id: 46d0776c55c6b2a76a0a816e1168347fea763d43
Summary:
Use hgclone_treemanifest instead of hgcloneshallow.
Note that it changes the behaviour of test-infinitepush.t - now it sends
treegroup part too. Since we actually want to send treegroup part during
infinitepush push, that means that the test was broken and now it fixed.
Reviewed By: lukaspiatkowski
Differential Revision: D7000046
fbshipit-source-id: 61b4517f23081faab1738f44d27220ae8063e077
Summary:
In D6988301 luk noticed that test output is incorrect. Turned out that client
asked server to fetch revision that exists only client-side.
To fix the test let's set treeonly=True and do shallow clone.
Figuring out why exactly it won't work without these flags is very
time-consuming. But that's the usual setup we expect our clients to have.
Besides test-init.t have used these flags and worked fine, so does
test-push-protocol.t
Reviewed By: lukaspiatkowski
Differential Revision: D7000045
fbshipit-source-id: 5e3a6e9fc9c0a1ed0a1e2074f5ac86533d42fa09
Summary:
1) Do not use system hg in tests.
Let's use current version of hg in the tests. It will simplify debugging
because we can now easily tweak both hg client and server. It will also result
in fewer test breakages, because previously our tests were broken when system
hg was updated.
2) Run integration tests in parallel with -j option
Reviewed By: quark-zju
Differential Revision: D6998535
fbshipit-source-id: dac4eb6d33978161ffe42d9d6a5a39a3ddf3a4e3
Summary: The changesets and filelogs can later be handled separately and applied on BlobRepo
Reviewed By: jsgf
Differential Revision: D6988952
fbshipit-source-id: 9959b6acb7913db208c2b4eeea8a1c9913d13678
Summary:
The previous Bundle2Stream is a stream of enum that can represent any part of bundle2 - being it a header, inner payload or end marker.
This commit changes the Bundle2Stream to be a stream of more structured elements, where each element is a header with corresponding payload attached.
This new approach is easier to handle, because the way a bundle2 stream is supposed to be dealt with is running an operation per header (where the type of operation depends on type of header). With the old Bundle2Stream the "handling" of bundle2 required basically reparsing it's content to the structures that the new design returns directly.
Reviewed By: jsgf
Differential Revision: D6965529
fbshipit-source-id: 03f7cf57fec5c38addf32bdef3ca56d3239288b9
