Summary:
1) Do not use system hg in tests.
Let's use current version of hg in the tests. It will simplify debugging
because we can now easily tweak both hg client and server. It will also result
in fewer test breakages, because previously our tests were broken when system
hg was updated.
2) Run integration tests in parallel with -j option
Reviewed By: quark-zju
Differential Revision: D6998535
fbshipit-source-id: dac4eb6d33978161ffe42d9d6a5a39a3ddf3a4e3
Summary: The changesets and filelogs can later be handled separately and applied on BlobRepo
Reviewed By: jsgf
Differential Revision: D6988952
fbshipit-source-id: 9959b6acb7913db208c2b4eeea8a1c9913d13678
Summary:
The previous Bundle2Stream is a stream of enum that can represent any part of bundle2 - being it a header, inner payload or end marker.
This commit changes the Bundle2Stream to be a stream of more structured elements, where each element is a header with corresponding payload attached.
This new approach is easier to handle, because the way a bundle2 stream is supposed to be dealt with is running an operation per header (where the type of operation depends on type of header). With the old Bundle2Stream the "handling" of bundle2 required basically reparsing it's content to the structures that the new design returns directly.
Reviewed By: jsgf
Differential Revision: D6965529
fbshipit-source-id: 03f7cf57fec5c38addf32bdef3ca56d3239288b9
Summary: The only real use case of Bundle2Stream requires it to be 'static, so the is no real point of preserving 'a.
Reviewed By: StanislavGlebik
Differential Revision: D6965528
fbshipit-source-id: 3d5c9c74c1d17980befd3742c54cdde9279aa62c
Summary:
In D6923197 we've made it so that nullid always exists. This is necessary for
pull, because client can send nullids (for example, if client has an empty
repo). That has broken revset test.
Also gettreepack capability was added, that has broken test-push-protocol.t
Reviewed By: quark-zju
Differential Revision: D6988301
fbshipit-source-id: 2deafc3489de9f50126de7212656ffc075ef3537
Summary:
This code will be used to convert incoming b2xtreegroup in wirepack
format into TreemanifestEntry, then later can be used in Commit Api.
Reviewed By: lukaspiatkowski
Differential Revision: D6965828
fbshipit-source-id: aff798ea52343d6f64564c9eff34dc5bc2510b38
Summary:
Wirepack packer code has an interesting logic that can be generalized.
For example, it can be used on the push side to convert b2xtreegroup wirepack
stream into stream on entries that can be used by Commit Api. This logic will
be implemented in the next diffs.
In this diff we separate wirepack logic into
two parts: converter stream, that converts wirepack stream into another stream,
and a trait that is used by converter.
Reviewed By: lukaspiatkowski
Differential Revision: D6965830
fbshipit-source-id: 726a84ba44c7f85b5aac48e702755dc904c9c6ab
Summary:
Quick recap: gettreepack is a treemanifest wireproto method.
It's used to send treemanifest data to the client. Client sends list of
manifest nodes it's interested in (mfnodes), and list of nodes it already has
(basemfnodes). Server should find the difference and send it back. Client
usually call this wireproto method when it checks out new revision (`hg
update`), but it uses in other cases too (for example, in `hg prefetch`).
Before we supported exactly one mfnode and exactly one basemfnode. This is
usually fine for `hg update` use-case, but `hg prefetch` can request many
mfnodes and can send empty basemfnode. So let's support this too.
Note that current implementation isn't efficient. It uses at most one
basemfnode, meaning that it can send data that client already has. Also for
each mfnode we generate separate stream of changed entries. That means that the
same entries can be fetched many times.
Reviewed By: lukaspiatkowski
Differential Revision: D6923197
fbshipit-source-id: d25f9a01ca568c84811ee1a13181e70eb217eb53
Summary: This new method is useful in my future work where I rely on this easy to use and readable API to be able to get over the fact that futures combinators take Stream by value. With this one can chain on the end of consuming Stream and by using the "remainder" Future get the consumed Stream.
Reviewed By: StanislavGlebik
Differential Revision: D6965532
fbshipit-source-id: 3ab19851b3d48c43c8d7e3a96ae5c03a7d242960
Summary:
Add an integration infinitepush test. Currently infinitepush is not implemented
in mononoke, so the test fails. We'll use it later to test our functionality.
Also add infinitepush wireproto capability, otherwise the test is useless - it
sends no infinitepush parts at all.
Reviewed By: quark-zju
Differential Revision: D6948967
fbshipit-source-id: 3b479c2c7f036c2b9424de0f160f742bb8ff7412
Summary:
Wireproto method to lookup if a revision/bookmark exists.
Currently supports only full hashes, later we'll add support for bookmarks and
node prefixes.
Reviewed By: farnz
Differential Revision: D6948358
fbshipit-source-id: af92a892eae8f787053447e601ebf95dfd638fa4
Summary:
We want to be able to upload raw entries to the backing store, so that
we can assemble files + manifests into a commit. Provide the API to just upload
raw entries
Reviewed By: lukaspiatkowski
Differential Revision: D6966727
fbshipit-source-id: 8eb0dce210f2646f29b38d216c54bb60a2b1ff60
Summary:
remotefilelog getfiles method requires fetching the whole history. Current
implementation is naive and it's going to be slow on big repos.
But later we're going to replace it with mysql
Reviewed By: quark-zju
Differential Revision: D6913180
fbshipit-source-id: 9244272a0339a2a83cf5b5615963b65ad2fcdc07
Summary:
Currently missing a few things:
1) Fetching the whole history for a file
2) Dealing with renames
That will be fixed in the next diffs
Reviewed By: quark-zju
Differential Revision: D6913179
fbshipit-source-id: f847c61791488bb2842a245ae00de7420b97145e
Summary:
These methods are
1) get_file_copy(node) -> returns copy info for a file (copyfrom revision and copyfrom filename). Can be None.
2) get_parents(node) -> returns parents of the node
It will be used in remotefilelog getfiles method to construct file history.
Reviewed By: jsgf
Differential Revision: D6913176
fbshipit-source-id: ba74a71926c5ca80249f8cc0f300a0aa20a65bbc
Summary:
getfiles is a remotefilelog method to fetch file content. Mercurial client
calls it on-demand, usually during `hg update` command. So normal `hg pull`
with remotefilelog doesn't send filelogs back to the client.
This diff implements getfiles arguments parsing. The code is tricky, so I've
added lots of comments to explain what's going on.
Reviewed By: farnz
Differential Revision: D6913178
fbshipit-source-id: 0248993c3ac487956e0ff547996d51b75cdaee96
Summary: making this test work is a 2018 m1 milestone
Reviewed By: farnz
Differential Revision: D6950112
fbshipit-source-id: 798a1b8e722e96b27f0cb8267b82232d3fe0496d
Summary:
The commit API would like to avoid uploading and downloading data when
it doesn't need to - extend the blobstore API so that blobstores can avoid
wasted work
Reviewed By: StanislavGlebik
Differential Revision: D6966729
fbshipit-source-id: 17783e2123d47e43824af724da0b4358a163fa5b
Summary:
In hg file is maybe_copied if first parent is null
(see https://www.mercurial-scm.org/repo/hg/file/tip/mercurial/filelog.py#l61)
We need to use the same logic for mononoke.
Reviewed By: farnz
Differential Revision: D6948175
fbshipit-source-id: 28c46ecd5ad72a56077d08a8f3756e2500f61ec9
Summary:
As we discussed before, let's add get_name() method that returns MPathElement,
and remove get_path() and get_mpath().
Except for renaming, diff also make repoconfig work with tree manifest, and
fixes linknodes creation in blobimport - previously basename was used instead
of the whole path.
Reviewed By: jsgf
Differential Revision: D6857097
fbshipit-source-id: c09f3ff40d38643bd44aee8b4488277d658cf4f6
Summary:
Simple gettreepack implementation. Current limitations:
1) only one mfnode
2) only one basemfnode
3) only empty rootdir
There is nothing in particular that doesn't allow us to lift this limitations,
I left them for the simplicity of the diff
Reviewed By: jsgf
Differential Revision: D6845032
fbshipit-source-id: 6ecc1f4ab7de4000d29923215592f95685b1ad87
Summary:
Everything needed for gettreepack method to work except for the gettreepack
implementation
Reviewed By: lukaspiatkowski
Differential Revision: D6845034
fbshipit-source-id: b675a2c38c7f6204fac5a6a0885caef246ba7987
Summary: Will be used in remotefilelog getfiles method.
Reviewed By: jsgf
Differential Revision: D6884919
fbshipit-source-id: e8037123a4843322c29b37c6b5749444781f4fa7
Summary:
Add a separate crate that uses lz4 in the same way as python lz4 library. The
main difference is that first 4 bytes are length of the raw data in le32
format. The reason for moving it in a separate crate is to use pylz4 for
remotefilelog getfiles method.
Also removed one panic and replaced it with error.
Reviewed By: jsgf
Differential Revision: D6884918
fbshipit-source-id: 1b05381c045a1f138ab28820175289233b07a91d
Summary: Initialize the RevlogRepo using the ChangesetId instead of the NodeHash.
Reviewed By: lukaspiatkowski
Differential Revision: D6867296
fbshipit-source-id: 12f0847324d138466421acc3b1f159a03655333b
Summary: If part header send by the client is non-mandatory and unknown then we silently ignore it. Let's add some debug logs to be able to investigate them
Reviewed By: farnz
Differential Revision: D6871679
fbshipit-source-id: c37f9913dd455e47d75356ad0723650528674aab
Summary: This change let's the handler return multiple responses to the client that is required for commands like `unbundle` to work
Reviewed By: jsgf
Differential Revision: D6820845
fbshipit-source-id: 5f17c2a6100aa989f35d468c188322da4a921059
Summary: Dechunker parses a hgproto stream of bytes by taking the chunks and letting the code to read them via BufRead API
Reviewed By: jsgf
Differential Revision: D6619521
fbshipit-source-id: 26dfab554a8e4c79079f4c4666c272ec2924c7d2
Summary:
Give them correct names, because this functions actually fetch node from
blobstore and strip copy metadata if it exists.
Reviewed By: farnz
Differential Revision: D6871271
fbshipit-source-id: 4d8c83786b223f5cd4f55188fbcb8f438713d3b7
Summary:
Pulling more than one node doesn't work, because mononoke revsets return
children before the parents. In that case mercurial client fails because new
child points to non-existent parents. To fix it let's reverse the output.
It's bad, because we lose the advantages of streaming, but we do it now anyway
because `getbundle()` method returns Bytes. Besides changelog data should be
fairly small.
Reviewed By: farnz
Differential Revision: D6899239
fbshipit-source-id: 67b3954bfbed8e0aaa69e0da453dae443cd24912
Summary: This is what the client expexts to receive before streaming can happen
Reviewed By: farnz
Differential Revision: D6833721
fbshipit-source-id: 04a66415638187108366109b0d9062bd8481a628
Summary:
The client sends us a request that we parse and return response for it. As simple as that.
When the client sends `unbundle` request we need to return an acknowledgment that we are ready to read a bundle2 stream from it.
This means that we first parse the request, then send the response, then parse the stream.
This will be changed in future diffs so that we might:
Parse request from client -> [send acknowledgments -> parse data from client]* -> send response;
Reviewed By: farnz
Differential Revision: D6833723
fbshipit-source-id: 0c13b1e3253bd664bc565eacb5b58a9fc53d4efa
Summary:
Currently there is no easy way to create Entry that represents root manifest.
More importantly, `get_mpath()` fails if internal RepoPath is root. This diff
fixes this issue and adds method to create root entry.
It will be used in `gettreepack` wireproto method implementation
Reviewed By: jsgf
Differential Revision: D6845033
fbshipit-source-id: 0999db89441e9ab783b8bca8047b01e4c6832187
Summary: It will be used by many things, in particular by gettreepack method
Reviewed By: jsgf
Differential Revision: D6845036
fbshipit-source-id: 3174dfafeeab830ededb70ddff3dd62f6a608d54
Summary:
It will be used in other places, such as repoconfig.
Unfortunately I had to split tests and implementation. The reason is that tests
depend on blobrepo. Blobrepo crate depends on mercurial_types and that causes
circular dependency. It shouldn't be the case, because
mercurial_types_unittests should be a separate crate, so there may be a bug in
the build system. For now I've split the tests and implementation.
Reviewed By: jsgf
Differential Revision: D6857098
fbshipit-source-id: 1c0af234564dd5628ba6d63d28cf54d7a2dc9217
Summary:
Per the title, enable the mysql and sqlite backends for diesel, so
that we can experiment with using diesel with those two DBMSs.
Reviewed By: jsgf
Differential Revision: D6854679
fbshipit-source-id: 66581db1bcafbcc422ababb969500e5d4dd5fafc
Summary:
Add some doc comments. Also added tasks for things that at least need
to be looked at.
Reviewed By: StanislavGlebik
Differential Revision: D6836570
fbshipit-source-id: 107f1f4799a369f11fdec7dbe051764a287e3a24
Summary: Change BlobChangeset and callers to use ChangesetId instead of NodeId
Reviewed By: lukaspiatkowski
Differential Revision: D6835450
fbshipit-source-id: 7b20359837632aef4803e40965380c38f54c9d0a
Summary:
This is a stream of diffs between two manfiests. It will be used in
`gettreepack` wireproto method of treemanifest.
This implementation is slow and simple, because it fetches from blobstore sequentially. We'll make it faster later if needed
Reviewed By: lukaspiatkowski
Differential Revision: D6808892
fbshipit-source-id: 3edcd02a287276b6015ac73fbe977a5b6d8573b5
