Summary: this parsed ManifestContent will be used in later diffs to determine the graph of dependencies between BlobEntries and Changesets
Reviewed By: StanislavGlebik
Differential Revision: D7030688
fbshipit-source-id: 243d28f7964e8247ac989a1736210457dd3eab96
Summary: This is needed for the next diff where the ManifestContent is used in TreemanifestBundle2Parser
Reviewed By: StanislavGlebik
Differential Revision: D7030689
fbshipit-source-id: cf199e3f1317d8950446782de9899eb2ef97a149
Summary:
The DeltaCache structure will hold onto the full content of all filelogs during a push.
This might potentially lead to memory spikes during large pushes or pushes of large files.
The basic telemetry added in this diff will track how does this structure memory usage changes.
Reviewed By: farnz
Differential Revision: D7025207
fbshipit-source-id: 61b8359e4a9a05b6ddb5ec6a7bc4ce9e58752f1b
Summary: This code handles deltaed filelogs and resolves them into proper Filelogs
Reviewed By: farnz
Differential Revision: D7011702
fbshipit-source-id: e8dc4844657011bc1085463eedd1790b87d317dc
Summary:
Return a reply to a client so that it doesn't fail.
Reply consists of just one replychangegroup that tells a client that the push
has succeeded (even though currently it wasn't).
resolve() function returns future of Bytes instead of a stream. It may be
suboptimal, but should be fine for now, and it's already used by a few
wireproto methods.
Reviewed By: lukaspiatkowski
Differential Revision: D7010578
fbshipit-source-id: 9b5425b912c640d4e2bac957a02e9881813b8871
Summary:
This change removes get_changeset_by_nodeid and replaces it with
get_changeset_by_changesetid, and propagates the changes to callers.
A few places still have ChangesetId::new() because I'm not sure where
the original NodeHash comes from. If you have any pointers, I would be
happy to fix them before landing.
Reviewed By: lukaspiatkowski
Differential Revision: D7031923
fbshipit-source-id: cd00ea1d2b955538e26d7b5735aed33fe0ae0330
Summary:
The current Memblob store is eager; this is great for finding certain
classes of bugs (those that assume an ordering that is not guaranteed), but not
so good for exposing other classes of bugs (those that assume that the future
has done its work before it resolves).
Add a lazy variant that functions in the same way as Memblob, but that waits
until it's polled to return.
Reviewed By: StanislavGlebik
Differential Revision: D7033792
fbshipit-source-id: 4c2d8a8150d908bcb26347757f96f99e20d74fc2
Summary:
The default implementation of `assert_present` only asserted that no
errors were found accessing the blobstore. Update it to fail if the blob is
absent.
Reviewed By: StanislavGlebik
Differential Revision: D7033794
fbshipit-source-id: 596d3b0d3d55d05a5b96760c311defe5a05af634
Summary:
We prefer to use Option<NodeHash> to represent a reference which may
be missing, whereas Mercurial file formats and wire protocols tend to use
NULL_HASH. Add some converters to make this simple.
libstd already has a global `impl From<T> for Option<T>` so we can't specialize
that here. Add a specific `into_option` method instead.
Reviewed By: StanislavGlebik
Differential Revision: D7016479
fbshipit-source-id: 1a65f8c2b8c252692a443aa7c5dd5976ad1e24b1
Summary:
Now resolve() function accepts stream of Bundle2Item and it doesn't return
remainder. Remainder is parsed in hgproto. This makes unbundle() similar to
other hgproto methods.
Note that resolve() returns future of () - this is temporary and it'll be
changed in the next diff.
Reviewed By: jsgf
Differential Revision: D7010577
fbshipit-source-id: f4bde495d0ab8785476559f65b0165dbd09e4ba5
Summary: I commited the sin of overengineering here and in the next diff I would remove some of these stuff, so doing it here to make the next diff easier to read
Reviewed By: StanislavGlebik
Differential Revision: D7002559
fbshipit-source-id: 46d0776c55c6b2a76a0a816e1168347fea763d43
Summary:
Use hgclone_treemanifest instead of hgcloneshallow.
Note that it changes the behaviour of test-infinitepush.t - now it sends
treegroup part too. Since we actually want to send treegroup part during
infinitepush push, that means that the test was broken and now it fixed.
Reviewed By: lukaspiatkowski
Differential Revision: D7000046
fbshipit-source-id: 61b4517f23081faab1738f44d27220ae8063e077
Summary:
In D6988301 luk noticed that test output is incorrect. Turned out that client
asked server to fetch revision that exists only client-side.
To fix the test let's set treeonly=True and do shallow clone.
Figuring out why exactly it won't work without these flags is very
time-consuming. But that's the usual setup we expect our clients to have.
Besides test-init.t have used these flags and worked fine, so does
test-push-protocol.t
Reviewed By: lukaspiatkowski
Differential Revision: D7000045
fbshipit-source-id: 5e3a6e9fc9c0a1ed0a1e2074f5ac86533d42fa09
Summary:
1) Do not use system hg in tests.
Let's use current version of hg in the tests. It will simplify debugging
because we can now easily tweak both hg client and server. It will also result
in fewer test breakages, because previously our tests were broken when system
hg was updated.
2) Run integration tests in parallel with -j option
Reviewed By: quark-zju
Differential Revision: D6998535
fbshipit-source-id: dac4eb6d33978161ffe42d9d6a5a39a3ddf3a4e3
Summary: The changesets and filelogs can later be handled separately and applied on BlobRepo
Reviewed By: jsgf
Differential Revision: D6988952
fbshipit-source-id: 9959b6acb7913db208c2b4eeea8a1c9913d13678
Summary:
The previous Bundle2Stream is a stream of enum that can represent any part of bundle2 - being it a header, inner payload or end marker.
This commit changes the Bundle2Stream to be a stream of more structured elements, where each element is a header with corresponding payload attached.
This new approach is easier to handle, because the way a bundle2 stream is supposed to be dealt with is running an operation per header (where the type of operation depends on type of header). With the old Bundle2Stream the "handling" of bundle2 required basically reparsing it's content to the structures that the new design returns directly.
Reviewed By: jsgf
Differential Revision: D6965529
fbshipit-source-id: 03f7cf57fec5c38addf32bdef3ca56d3239288b9
Summary: The only real use case of Bundle2Stream requires it to be 'static, so the is no real point of preserving 'a.
Reviewed By: StanislavGlebik
Differential Revision: D6965528
fbshipit-source-id: 3d5c9c74c1d17980befd3742c54cdde9279aa62c
Summary:
In D6923197 we've made it so that nullid always exists. This is necessary for
pull, because client can send nullids (for example, if client has an empty
repo). That has broken revset test.
Also gettreepack capability was added, that has broken test-push-protocol.t
Reviewed By: quark-zju
Differential Revision: D6988301
fbshipit-source-id: 2deafc3489de9f50126de7212656ffc075ef3537
Summary:
This code will be used to convert incoming b2xtreegroup in wirepack
format into TreemanifestEntry, then later can be used in Commit Api.
Reviewed By: lukaspiatkowski
Differential Revision: D6965828
fbshipit-source-id: aff798ea52343d6f64564c9eff34dc5bc2510b38
Summary:
Wirepack packer code has an interesting logic that can be generalized.
For example, it can be used on the push side to convert b2xtreegroup wirepack
stream into stream on entries that can be used by Commit Api. This logic will
be implemented in the next diffs.
In this diff we separate wirepack logic into
two parts: converter stream, that converts wirepack stream into another stream,
and a trait that is used by converter.
Reviewed By: lukaspiatkowski
Differential Revision: D6965830
fbshipit-source-id: 726a84ba44c7f85b5aac48e702755dc904c9c6ab
Summary:
Quick recap: gettreepack is a treemanifest wireproto method.
It's used to send treemanifest data to the client. Client sends list of
manifest nodes it's interested in (mfnodes), and list of nodes it already has
(basemfnodes). Server should find the difference and send it back. Client
usually call this wireproto method when it checks out new revision (`hg
update`), but it uses in other cases too (for example, in `hg prefetch`).
Before we supported exactly one mfnode and exactly one basemfnode. This is
usually fine for `hg update` use-case, but `hg prefetch` can request many
mfnodes and can send empty basemfnode. So let's support this too.
Note that current implementation isn't efficient. It uses at most one
basemfnode, meaning that it can send data that client already has. Also for
each mfnode we generate separate stream of changed entries. That means that the
same entries can be fetched many times.
Reviewed By: lukaspiatkowski
Differential Revision: D6923197
fbshipit-source-id: d25f9a01ca568c84811ee1a13181e70eb217eb53
Summary: This new method is useful in my future work where I rely on this easy to use and readable API to be able to get over the fact that futures combinators take Stream by value. With this one can chain on the end of consuming Stream and by using the "remainder" Future get the consumed Stream.
Reviewed By: StanislavGlebik
Differential Revision: D6965532
fbshipit-source-id: 3ab19851b3d48c43c8d7e3a96ae5c03a7d242960
Summary:
Add an integration infinitepush test. Currently infinitepush is not implemented
in mononoke, so the test fails. We'll use it later to test our functionality.
Also add infinitepush wireproto capability, otherwise the test is useless - it
sends no infinitepush parts at all.
Reviewed By: quark-zju
Differential Revision: D6948967
fbshipit-source-id: 3b479c2c7f036c2b9424de0f160f742bb8ff7412
Summary:
Wireproto method to lookup if a revision/bookmark exists.
Currently supports only full hashes, later we'll add support for bookmarks and
node prefixes.
Reviewed By: farnz
Differential Revision: D6948358
fbshipit-source-id: af92a892eae8f787053447e601ebf95dfd638fa4
Summary:
We want to be able to upload raw entries to the backing store, so that
we can assemble files + manifests into a commit. Provide the API to just upload
raw entries
Reviewed By: lukaspiatkowski
Differential Revision: D6966727
fbshipit-source-id: 8eb0dce210f2646f29b38d216c54bb60a2b1ff60
Summary:
remotefilelog getfiles method requires fetching the whole history. Current
implementation is naive and it's going to be slow on big repos.
But later we're going to replace it with mysql
Reviewed By: quark-zju
Differential Revision: D6913180
fbshipit-source-id: 9244272a0339a2a83cf5b5615963b65ad2fcdc07
Summary:
Currently missing a few things:
1) Fetching the whole history for a file
2) Dealing with renames
That will be fixed in the next diffs
Reviewed By: quark-zju
Differential Revision: D6913179
fbshipit-source-id: f847c61791488bb2842a245ae00de7420b97145e
Summary:
These methods are
1) get_file_copy(node) -> returns copy info for a file (copyfrom revision and copyfrom filename). Can be None.
2) get_parents(node) -> returns parents of the node
It will be used in remotefilelog getfiles method to construct file history.
Reviewed By: jsgf
Differential Revision: D6913176
fbshipit-source-id: ba74a71926c5ca80249f8cc0f300a0aa20a65bbc
Summary:
getfiles is a remotefilelog method to fetch file content. Mercurial client
calls it on-demand, usually during `hg update` command. So normal `hg pull`
with remotefilelog doesn't send filelogs back to the client.
This diff implements getfiles arguments parsing. The code is tricky, so I've
added lots of comments to explain what's going on.
Reviewed By: farnz
Differential Revision: D6913178
fbshipit-source-id: 0248993c3ac487956e0ff547996d51b75cdaee96
Summary: making this test work is a 2018 m1 milestone
Reviewed By: farnz
Differential Revision: D6950112
fbshipit-source-id: 798a1b8e722e96b27f0cb8267b82232d3fe0496d
Summary:
The commit API would like to avoid uploading and downloading data when
it doesn't need to - extend the blobstore API so that blobstores can avoid
wasted work
Reviewed By: StanislavGlebik
Differential Revision: D6966729
fbshipit-source-id: 17783e2123d47e43824af724da0b4358a163fa5b
Summary:
In hg file is maybe_copied if first parent is null
(see https://www.mercurial-scm.org/repo/hg/file/tip/mercurial/filelog.py#l61)
We need to use the same logic for mononoke.
Reviewed By: farnz
Differential Revision: D6948175
fbshipit-source-id: 28c46ecd5ad72a56077d08a8f3756e2500f61ec9
Summary:
As we discussed before, let's add get_name() method that returns MPathElement,
and remove get_path() and get_mpath().
Except for renaming, diff also make repoconfig work with tree manifest, and
fixes linknodes creation in blobimport - previously basename was used instead
of the whole path.
Reviewed By: jsgf
Differential Revision: D6857097
fbshipit-source-id: c09f3ff40d38643bd44aee8b4488277d658cf4f6
Summary:
Simple gettreepack implementation. Current limitations:
1) only one mfnode
2) only one basemfnode
3) only empty rootdir
There is nothing in particular that doesn't allow us to lift this limitations,
I left them for the simplicity of the diff
Reviewed By: jsgf
Differential Revision: D6845032
fbshipit-source-id: 6ecc1f4ab7de4000d29923215592f95685b1ad87
Summary:
Everything needed for gettreepack method to work except for the gettreepack
implementation
Reviewed By: lukaspiatkowski
Differential Revision: D6845034
fbshipit-source-id: b675a2c38c7f6204fac5a6a0885caef246ba7987
Summary: Will be used in remotefilelog getfiles method.
Reviewed By: jsgf
Differential Revision: D6884919
fbshipit-source-id: e8037123a4843322c29b37c6b5749444781f4fa7
