Summary:
Pushkey part is used to send bookmark updates from hg client to the server.
This diff does all the wireproto parsing, but doesn't actually apply bookmark
updates on the server.
Also this diff "implements" branchmap method. We have no plans to support it,
but currently remotenames extension calls it. So this diff adds a fake
implementation that always returns empty response.
Reviewed By: farnz
Differential Revision: D7150973
fbshipit-source-id: 6889c02a1105127b1805ef1fafa6fbe9c2d57e7d
Summary:
I've spent quite some time but wasn't able to get to the bottom of why
test-infinitepush.t has been flaky. For some reason error message
```
Expected Bundle2 Changegroup
```
hasn't made it's way to the hg client. Adding a multisecond delay before
sending the error "fixes" the problem, but I haven't found the actual reason
behind the problem. So since that's not a #1 priority now I suggest to postpone
fixing it for now.
Reviewed By: farnz
Differential Revision: D7246511
fbshipit-source-id: a385130e6bdc978765e04f44c6a536405ee12e02
Summary:
The reason for flakiness was in unpredictable order of the tree items from
Mononoke. In the first gettreepack call we request two different revisions, and
Mononoke can return them in any order. hg handles it just fine (see updated
test with `hgmn up ...`), but hashes of the tree packs may change.
To remove flakiness let's not rely on the treepack files hash
Reviewed By: farnz
Differential Revision: D7237559
fbshipit-source-id: c04e9d45c41f1d288a90706d0ecc27ede36f8008
Summary:
It was flaky because server may not yet be ready to accept connections when we
send first request. Sometimes delay needs to be > 1 sec.
Let's query eden server in a loop until it responds.
Reviewed By: farnz
Differential Revision: D7233069
fbshipit-source-id: 8bcb5b2b8ebdc52d2447b33e18580e50c1e27031
Summary:
This is the part that sends client bookmarks to the server. Quite likely we are
not going to process it, so we are just reading the part and ignoring it.
Reviewed By: farnz
Differential Revision: D7123758
fbshipit-source-id: 7ad39bde77b6f77cf6e440e726ac3bdb9f340cea
Summary:
Support b2x:infinitepush part. It contains changegroup v2, so just reuse the
normal push path for it.
Note that pushbackup still fails because we don't support
b2x:infinitepushscratchbookmarks part. Also all wireproto params are ignored
for b2x:infinitepush.
Reviewed By: farnz
Differential Revision: D7086120
fbshipit-source-id: 2f98e5d59059ca3c2b82842c98e6dc771c70c6f0
Summary:
This is a hacky way of getting the push/pull working. We should instead remove the commits that are no longer heads from HeadStore and add those that are the new heads.
In this diff though we add all the commits as heads, just because this does not break the client
Reviewed By: farnz
Differential Revision: D7112279
fbshipit-source-id: 036f0fd230de52e96cbf4168c2cda7c2a1c5bd89
Summary:
From upload_entry perspective NodeHash is not unique for uploaded entry, but when combined with RepoPath it is. An example are two files with the same content, but different paths.
Additionally in this diff the requirement for uploaded blobs to be unique is loosened for Manifests, because the b2xtreegroup might contain duplicates of Manifests
Reviewed By: farnz
Differential Revision: D7087863
fbshipit-source-id: 7e9c2438db037fa171f1e65b6882b445e8c09f7a
Summary: It seems that my assumtion that Filelog End section follow every Filelog Chunk section was wrong, because there might be multiple Filelog Chunk sections being different version of the same file from different commits.
Reviewed By: farnz
Differential Revision: D7073007
fbshipit-source-id: 3baee3fd29c77aabf4173a618509de7ff88e4de6
Summary: The assumption that deltas can be decoded using only bundle2 was wrong, we need to be able to fetch content of other files from the BlobRepo for delta application.
Reviewed By: StanislavGlebik
Differential Revision: D7085957
fbshipit-source-id: 2f6803d7f61389c5ba38b1207ede42579b9cf2e6
Summary: This will make it easy to track down an error without looking at stacktraces
Reviewed By: StanislavGlebik
Differential Revision: D7087301
fbshipit-source-id: c5460dae9c5c9ab43713e5db1457f5d9155b5e8e
Summary:
Changesets table gives us a relatively fast way to get generation number. Let's
use it inside repoinfo crate.
Reviewed By: sid0
Differential Revision: D7043829
fbshipit-source-id: 82557340a7f2fd6361f4e29c1a11e38e473eeb15
Summary: Changests store requires it in it's api methods. Let's pass repoid from configs
Reviewed By: farnz
Differential Revision: D7043830
fbshipit-source-id: e4e4d5852d0ca8488cabe2140555508c143ab8df
Summary: The introduced commits feature things like adding blob that should already be present, adding blob twice and having non-linear history
Reviewed By: farnz
Differential Revision: D7071447
fbshipit-source-id: e26b7792808351e2380d68b9eb3a4d7e6e859b0e
Summary: This code handles deltaed filelogs and resolves them into proper Filelogs
Reviewed By: farnz
Differential Revision: D7011702
fbshipit-source-id: e8dc4844657011bc1085463eedd1790b87d317dc
Summary:
Return a reply to a client so that it doesn't fail.
Reply consists of just one replychangegroup that tells a client that the push
has succeeded (even though currently it wasn't).
resolve() function returns future of Bytes instead of a stream. It may be
suboptimal, but should be fine for now, and it's already used by a few
wireproto methods.
Reviewed By: lukaspiatkowski
Differential Revision: D7010578
fbshipit-source-id: 9b5425b912c640d4e2bac957a02e9881813b8871
Summary:
This change removes get_changeset_by_nodeid and replaces it with
get_changeset_by_changesetid, and propagates the changes to callers.
A few places still have ChangesetId::new() because I'm not sure where
the original NodeHash comes from. If you have any pointers, I would be
happy to fix them before landing.
Reviewed By: lukaspiatkowski
Differential Revision: D7031923
fbshipit-source-id: cd00ea1d2b955538e26d7b5735aed33fe0ae0330
Summary:
Use hgclone_treemanifest instead of hgcloneshallow.
Note that it changes the behaviour of test-infinitepush.t - now it sends
treegroup part too. Since we actually want to send treegroup part during
infinitepush push, that means that the test was broken and now it fixed.
Reviewed By: lukaspiatkowski
Differential Revision: D7000046
fbshipit-source-id: 61b4517f23081faab1738f44d27220ae8063e077
Summary:
In D6988301 luk noticed that test output is incorrect. Turned out that client
asked server to fetch revision that exists only client-side.
To fix the test let's set treeonly=True and do shallow clone.
Figuring out why exactly it won't work without these flags is very
time-consuming. But that's the usual setup we expect our clients to have.
Besides test-init.t have used these flags and worked fine, so does
test-push-protocol.t
Reviewed By: lukaspiatkowski
Differential Revision: D7000045
fbshipit-source-id: 5e3a6e9fc9c0a1ed0a1e2074f5ac86533d42fa09
Summary:
1) Do not use system hg in tests.
Let's use current version of hg in the tests. It will simplify debugging
because we can now easily tweak both hg client and server. It will also result
in fewer test breakages, because previously our tests were broken when system
hg was updated.
2) Run integration tests in parallel with -j option
Reviewed By: quark-zju
Differential Revision: D6998535
fbshipit-source-id: dac4eb6d33978161ffe42d9d6a5a39a3ddf3a4e3
Summary:
In D6923197 we've made it so that nullid always exists. This is necessary for
pull, because client can send nullids (for example, if client has an empty
repo). That has broken revset test.
Also gettreepack capability was added, that has broken test-push-protocol.t
Reviewed By: quark-zju
Differential Revision: D6988301
fbshipit-source-id: 2deafc3489de9f50126de7212656ffc075ef3537
Summary:
Quick recap: gettreepack is a treemanifest wireproto method.
It's used to send treemanifest data to the client. Client sends list of
manifest nodes it's interested in (mfnodes), and list of nodes it already has
(basemfnodes). Server should find the difference and send it back. Client
usually call this wireproto method when it checks out new revision (`hg
update`), but it uses in other cases too (for example, in `hg prefetch`).
Before we supported exactly one mfnode and exactly one basemfnode. This is
usually fine for `hg update` use-case, but `hg prefetch` can request many
mfnodes and can send empty basemfnode. So let's support this too.
Note that current implementation isn't efficient. It uses at most one
basemfnode, meaning that it can send data that client already has. Also for
each mfnode we generate separate stream of changed entries. That means that the
same entries can be fetched many times.
Reviewed By: lukaspiatkowski
Differential Revision: D6923197
fbshipit-source-id: d25f9a01ca568c84811ee1a13181e70eb217eb53
Summary:
Add an integration infinitepush test. Currently infinitepush is not implemented
in mononoke, so the test fails. We'll use it later to test our functionality.
Also add infinitepush wireproto capability, otherwise the test is useless - it
sends no infinitepush parts at all.
Reviewed By: quark-zju
Differential Revision: D6948967
fbshipit-source-id: 3b479c2c7f036c2b9424de0f160f742bb8ff7412
Summary:
Wireproto method to lookup if a revision/bookmark exists.
Currently supports only full hashes, later we'll add support for bookmarks and
node prefixes.
Reviewed By: farnz
Differential Revision: D6948358
fbshipit-source-id: af92a892eae8f787053447e601ebf95dfd638fa4
Summary:
remotefilelog getfiles method requires fetching the whole history. Current
implementation is naive and it's going to be slow on big repos.
But later we're going to replace it with mysql
Reviewed By: quark-zju
Differential Revision: D6913180
fbshipit-source-id: 9244272a0339a2a83cf5b5615963b65ad2fcdc07
Summary:
Currently missing a few things:
1) Fetching the whole history for a file
2) Dealing with renames
That will be fixed in the next diffs
Reviewed By: quark-zju
Differential Revision: D6913179
fbshipit-source-id: f847c61791488bb2842a245ae00de7420b97145e
Summary:
getfiles is a remotefilelog method to fetch file content. Mercurial client
calls it on-demand, usually during `hg update` command. So normal `hg pull`
with remotefilelog doesn't send filelogs back to the client.
This diff implements getfiles arguments parsing. The code is tricky, so I've
added lots of comments to explain what's going on.
Reviewed By: farnz
Differential Revision: D6913178
fbshipit-source-id: 0248993c3ac487956e0ff547996d51b75cdaee96
Summary: making this test work is a 2018 m1 milestone
Reviewed By: farnz
Differential Revision: D6950112
fbshipit-source-id: 798a1b8e722e96b27f0cb8267b82232d3fe0496d
Summary:
As we discussed before, let's add get_name() method that returns MPathElement,
and remove get_path() and get_mpath().
Except for renaming, diff also make repoconfig work with tree manifest, and
fixes linknodes creation in blobimport - previously basename was used instead
of the whole path.
Reviewed By: jsgf
Differential Revision: D6857097
fbshipit-source-id: c09f3ff40d38643bd44aee8b4488277d658cf4f6
Summary:
Pulling more than one node doesn't work, because mononoke revsets return
children before the parents. In that case mercurial client fails because new
child points to non-existent parents. To fix it let's reverse the output.
It's bad, because we lose the advantages of streaming, but we do it now anyway
because `getbundle()` method returns Bytes. Besides changelog data should be
fairly small.
Reviewed By: farnz
Differential Revision: D6899239
fbshipit-source-id: 67b3954bfbed8e0aaa69e0da453dae443cd24912
Summary:
Super-basic implementation of generating changegroup part in getbundle. All it does is sending changelog
entries - so no manifest and no filelog entries. While it's very simple, it may
actually be enough for a real production use-case. With treemanifest and
remotefilelog we don't need to pull anything except for changelog entries,
everything else will be downloaded on demand.
Reviewed By: jsgf
Differential Revision: D6748070
fbshipit-source-id: 1dcd802a4b3b111f935e713e7696d58f05861b77
Summary:
We're never going to serve RevlogRepo in production, and we're down to
a single BlobRepo type that will have different backing stores. Remove the
unused trait, and use BlobRepo everywhere bar blobimport and repo_config
(because we previously hardcoded revlog here - we want to change to a BlobRepo
once blobimport is full-fidelity).
Reviewed By: jsgf
Differential Revision: D6596164
fbshipit-source-id: ba6e76e78c495720792cbe77ae6037f7802ec126
Summary: It removes a bit of copy-pastes code
Reviewed By: sid0
Differential Revision: D6627250
fbshipit-source-id: b110dcad998a240b6da60bc3347c7c5c0370aae0
Summary:
It won't be used in prod anyway, and for local use rocksdb blob state is still
preferable
Reviewed By: sid0
Differential Revision: D6627197
fbshipit-source-id: d9ca55b0221c050e8e8e35914aff22906198874a
Summary:
The test wasn't waiting for the Mononoke server to be available. Also
improve the error message slightly.
Reviewed By: jsgf
Differential Revision: D6606102
fbshipit-source-id: d0fde39aef8b3423fd1a8996a01f12e8a9661597
Summary:
Fetching file sizes is very expensive - it's one more round-trip to the
blobstore.
Add method that returns tree content, but doesn't fetch blob sizes.
Reviewed By: jsgf
Differential Revision: D6545108
fbshipit-source-id: 3d335eafa9dc367fb9e607f5c147b7e5c37cf133
Summary:
Two tests can't be run simultaneously because they use the same port. This diff
fixes it.
Reviewed By: farnz
Differential Revision: D6457855
fbshipit-source-id: 3e5f82211bcd87abc608bdeab239fe680976316d
Summary: `changeset_exists` only checks that the changeset is in the blob store. Once we start accepting pushes, we'll also want to check that the changeset is reachable from a head. Use revsets to check this. Once we have a notion of "completeness" for changesets, we'll switch `known` to use that instead - this code is still useful as a way to go from `changeset_exists` to `changeset_complete` in bulk.
Reviewed By: jsgf
Differential Revision: D6205695
fbshipit-source-id: 7e3b5c30bc5e459feb95a20913d8a04f3fda7469
Summary:
It maybe not installed on the system. Let's use simple python script instead.
Note: unfortunately we can't use 'python -m json.tool' because we need to sort
entries in the list
Reviewed By: jsgf
Differential Revision: D6437136
fbshipit-source-id: 0703f45c46d220046b9ef484bdf406a7f936557b
Summary:
Config file needs to have ca file in PEM format.
It is used to authenticate client.
See code comments for more details.
Reviewed By: jsgf
Differential Revision: D6323439
fbshipit-source-id: 6b6f1fd68605b263dcb33b051843e10d3f5cb38e
Summary:
Avoid using plain HTTP and use HTTPS instead.
To do this config needs to provide paths to server certificate and private key files in PEM format.
Then they will be converted to Pkcs12 archive.
This diff adds authentication of server i.e. client can check that it talks to a real server. Next diff adds authentication of a client.
Lower-level `hyper::server::Http::bind_connection()` is used instead of `hyper::server::Http::bind()` method in order to add TLS support.
See code comments for more details.
Implementation is more complicated than I expected it to be. I need to use 3 more new crates. Lmk if there is a better way to do this.
Reviewed By: jsgf
Differential Revision: D6323440
fbshipit-source-id: 544f27e6ec210ddf840212b0c0c94145980e8be3
Summary:
We have too many parameters, and cmd-line args are no longer good enough. Let's
use toml based config, with a format that's similar to mononoke server
repoconfig.
Reviewed By: farnz
Differential Revision: D6313512
fbshipit-source-id: 61379e93707a4fcbb42aa8a0392ce526dab19f2e
