Summary: This code handles deltaed filelogs and resolves them into proper Filelogs
Reviewed By: farnz
Differential Revision: D7011702
fbshipit-source-id: e8dc4844657011bc1085463eedd1790b87d317dc
Summary:
Return a reply to a client so that it doesn't fail.
Reply consists of just one replychangegroup that tells a client that the push
has succeeded (even though currently it wasn't).
resolve() function returns future of Bytes instead of a stream. It may be
suboptimal, but should be fine for now, and it's already used by a few
wireproto methods.
Reviewed By: lukaspiatkowski
Differential Revision: D7010578
fbshipit-source-id: 9b5425b912c640d4e2bac957a02e9881813b8871
Summary:
This change removes get_changeset_by_nodeid and replaces it with
get_changeset_by_changesetid, and propagates the changes to callers.
A few places still have ChangesetId::new() because I'm not sure where
the original NodeHash comes from. If you have any pointers, I would be
happy to fix them before landing.
Reviewed By: lukaspiatkowski
Differential Revision: D7031923
fbshipit-source-id: cd00ea1d2b955538e26d7b5735aed33fe0ae0330
Summary:
The current Memblob store is eager; this is great for finding certain
classes of bugs (those that assume an ordering that is not guaranteed), but not
so good for exposing other classes of bugs (those that assume that the future
has done its work before it resolves).
Add a lazy variant that functions in the same way as Memblob, but that waits
until it's polled to return.
Reviewed By: StanislavGlebik
Differential Revision: D7033792
fbshipit-source-id: 4c2d8a8150d908bcb26347757f96f99e20d74fc2
Summary:
Use hgclone_treemanifest instead of hgcloneshallow.
Note that it changes the behaviour of test-infinitepush.t - now it sends
treegroup part too. Since we actually want to send treegroup part during
infinitepush push, that means that the test was broken and now it fixed.
Reviewed By: lukaspiatkowski
Differential Revision: D7000046
fbshipit-source-id: 61b4517f23081faab1738f44d27220ae8063e077
Summary:
In D6988301 luk noticed that test output is incorrect. Turned out that client
asked server to fetch revision that exists only client-side.
To fix the test let's set treeonly=True and do shallow clone.
Figuring out why exactly it won't work without these flags is very
time-consuming. But that's the usual setup we expect our clients to have.
Besides test-init.t have used these flags and worked fine, so does
test-push-protocol.t
Reviewed By: lukaspiatkowski
Differential Revision: D7000045
fbshipit-source-id: 5e3a6e9fc9c0a1ed0a1e2074f5ac86533d42fa09
Summary:
1) Do not use system hg in tests.
Let's use current version of hg in the tests. It will simplify debugging
because we can now easily tweak both hg client and server. It will also result
in fewer test breakages, because previously our tests were broken when system
hg was updated.
2) Run integration tests in parallel with -j option
Reviewed By: quark-zju
Differential Revision: D6998535
fbshipit-source-id: dac4eb6d33978161ffe42d9d6a5a39a3ddf3a4e3
Summary:
In D6923197 we've made it so that nullid always exists. This is necessary for
pull, because client can send nullids (for example, if client has an empty
repo). That has broken revset test.
Also gettreepack capability was added, that has broken test-push-protocol.t
Reviewed By: quark-zju
Differential Revision: D6988301
fbshipit-source-id: 2deafc3489de9f50126de7212656ffc075ef3537
Summary:
Quick recap: gettreepack is a treemanifest wireproto method.
It's used to send treemanifest data to the client. Client sends list of
manifest nodes it's interested in (mfnodes), and list of nodes it already has
(basemfnodes). Server should find the difference and send it back. Client
usually call this wireproto method when it checks out new revision (`hg
update`), but it uses in other cases too (for example, in `hg prefetch`).
Before we supported exactly one mfnode and exactly one basemfnode. This is
usually fine for `hg update` use-case, but `hg prefetch` can request many
mfnodes and can send empty basemfnode. So let's support this too.
Note that current implementation isn't efficient. It uses at most one
basemfnode, meaning that it can send data that client already has. Also for
each mfnode we generate separate stream of changed entries. That means that the
same entries can be fetched many times.
Reviewed By: lukaspiatkowski
Differential Revision: D6923197
fbshipit-source-id: d25f9a01ca568c84811ee1a13181e70eb217eb53
Summary:
Add an integration infinitepush test. Currently infinitepush is not implemented
in mononoke, so the test fails. We'll use it later to test our functionality.
Also add infinitepush wireproto capability, otherwise the test is useless - it
sends no infinitepush parts at all.
Reviewed By: quark-zju
Differential Revision: D6948967
fbshipit-source-id: 3b479c2c7f036c2b9424de0f160f742bb8ff7412
Summary:
Wireproto method to lookup if a revision/bookmark exists.
Currently supports only full hashes, later we'll add support for bookmarks and
node prefixes.
Reviewed By: farnz
Differential Revision: D6948358
fbshipit-source-id: af92a892eae8f787053447e601ebf95dfd638fa4
Summary:
remotefilelog getfiles method requires fetching the whole history. Current
implementation is naive and it's going to be slow on big repos.
But later we're going to replace it with mysql
Reviewed By: quark-zju
Differential Revision: D6913180
fbshipit-source-id: 9244272a0339a2a83cf5b5615963b65ad2fcdc07
Summary:
Currently missing a few things:
1) Fetching the whole history for a file
2) Dealing with renames
That will be fixed in the next diffs
Reviewed By: quark-zju
Differential Revision: D6913179
fbshipit-source-id: f847c61791488bb2842a245ae00de7420b97145e
Summary:
getfiles is a remotefilelog method to fetch file content. Mercurial client
calls it on-demand, usually during `hg update` command. So normal `hg pull`
with remotefilelog doesn't send filelogs back to the client.
This diff implements getfiles arguments parsing. The code is tricky, so I've
added lots of comments to explain what's going on.
Reviewed By: farnz
Differential Revision: D6913178
fbshipit-source-id: 0248993c3ac487956e0ff547996d51b75cdaee96
Summary: making this test work is a 2018 m1 milestone
Reviewed By: farnz
Differential Revision: D6950112
fbshipit-source-id: 798a1b8e722e96b27f0cb8267b82232d3fe0496d
Summary:
As we discussed before, let's add get_name() method that returns MPathElement,
and remove get_path() and get_mpath().
Except for renaming, diff also make repoconfig work with tree manifest, and
fixes linknodes creation in blobimport - previously basename was used instead
of the whole path.
Reviewed By: jsgf
Differential Revision: D6857097
fbshipit-source-id: c09f3ff40d38643bd44aee8b4488277d658cf4f6
Summary:
Pulling more than one node doesn't work, because mononoke revsets return
children before the parents. In that case mercurial client fails because new
child points to non-existent parents. To fix it let's reverse the output.
It's bad, because we lose the advantages of streaming, but we do it now anyway
because `getbundle()` method returns Bytes. Besides changelog data should be
fairly small.
Reviewed By: farnz
Differential Revision: D6899239
fbshipit-source-id: 67b3954bfbed8e0aaa69e0da453dae443cd24912
Summary:
Add a fixture where there are many files and directories.
History is linear for now, but later we may add merge commtis.
It will be used to test gettreepack method.
Reviewed By: lukaspiatkowski
Differential Revision: D6818544
fbshipit-source-id: 798a1bea66eb49491f98fe5e997b172f3c927c53
Summary:
Super-basic implementation of generating changegroup part in getbundle. All it does is sending changelog
entries - so no manifest and no filelog entries. While it's very simple, it may
actually be enough for a real production use-case. With treemanifest and
remotefilelog we don't need to pull anything except for changelog entries,
everything else will be downloaded on demand.
Reviewed By: jsgf
Differential Revision: D6748070
fbshipit-source-id: 1dcd802a4b3b111f935e713e7696d58f05861b77
Summary:
We're never going to serve RevlogRepo in production, and we're down to
a single BlobRepo type that will have different backing stores. Remove the
unused trait, and use BlobRepo everywhere bar blobimport and repo_config
(because we previously hardcoded revlog here - we want to change to a BlobRepo
once blobimport is full-fidelity).
Reviewed By: jsgf
Differential Revision: D6596164
fbshipit-source-id: ba6e76e78c495720792cbe77ae6037f7802ec126
Summary: It removes a bit of copy-pastes code
Reviewed By: sid0
Differential Revision: D6627250
fbshipit-source-id: b110dcad998a240b6da60bc3347c7c5c0370aae0
Summary:
It won't be used in prod anyway, and for local use rocksdb blob state is still
preferable
Reviewed By: sid0
Differential Revision: D6627197
fbshipit-source-id: d9ca55b0221c050e8e8e35914aff22906198874a
Summary:
The test wasn't waiting for the Mononoke server to be available. Also
improve the error message slightly.
Reviewed By: jsgf
Differential Revision: D6606102
fbshipit-source-id: d0fde39aef8b3423fd1a8996a01f12e8a9661597
Summary:
Fetching file sizes is very expensive - it's one more round-trip to the
blobstore.
Add method that returns tree content, but doesn't fetch blob sizes.
Reviewed By: jsgf
Differential Revision: D6545108
fbshipit-source-id: 3d335eafa9dc367fb9e607f5c147b7e5c37cf133
Summary:
Two tests can't be run simultaneously because they use the same port. This diff
fixes it.
Reviewed By: farnz
Differential Revision: D6457855
fbshipit-source-id: 3e5f82211bcd87abc608bdeab239fe680976316d
Summary: `changeset_exists` only checks that the changeset is in the blob store. Once we start accepting pushes, we'll also want to check that the changeset is reachable from a head. Use revsets to check this. Once we have a notion of "completeness" for changesets, we'll switch `known` to use that instead - this code is still useful as a way to go from `changeset_exists` to `changeset_complete` in bulk.
Reviewed By: jsgf
Differential Revision: D6205695
fbshipit-source-id: 7e3b5c30bc5e459feb95a20913d8a04f3fda7469
Summary:
BlobStore is entirely generic, and puts no limits on its
implementations. Remove ValueIn and ValueOut type parameters, and insist that
all blobs are Bytes (as per production setups)
Reviewed By: StanislavGlebik
Differential Revision: D6425413
fbshipit-source-id: 455e526d8baebd0d0f1906941648acca89be4881
Summary:
It maybe not installed on the system. Let's use simple python script instead.
Note: unfortunately we can't use 'python -m json.tool' because we need to sort
entries in the list
Reviewed By: jsgf
Differential Revision: D6437136
fbshipit-source-id: 0703f45c46d220046b9ef484bdf406a7f936557b
Summary:
As part of removing excess genericism, make Bookmarks a trait with no
associated types or type parameters. I will revisit error types here once I
have removed the Repo trait - these are not ideal, but enough to get things
going
Reviewed By: StanislavGlebik
Differential Revision: D6405315
fbshipit-source-id: 814c106612a061e1766f1ea9a9428a13a73bd007
Summary:
As part of removing excess genericism, make Heads a trait with no
associated types or type parameters.
Reviewed By: StanislavGlebik
Differential Revision: D6352727
fbshipit-source-id: df9ef87e0e0abe43c30e7318da38d7f930c37c6e
Summary:
Config file needs to have ca file in PEM format.
It is used to authenticate client.
See code comments for more details.
Reviewed By: jsgf
Differential Revision: D6323439
fbshipit-source-id: 6b6f1fd68605b263dcb33b051843e10d3f5cb38e
Summary:
Avoid using plain HTTP and use HTTPS instead.
To do this config needs to provide paths to server certificate and private key files in PEM format.
Then they will be converted to Pkcs12 archive.
This diff adds authentication of server i.e. client can check that it talks to a real server. Next diff adds authentication of a client.
Lower-level `hyper::server::Http::bind_connection()` is used instead of `hyper::server::Http::bind()` method in order to add TLS support.
See code comments for more details.
Implementation is more complicated than I expected it to be. I need to use 3 more new crates. Lmk if there is a better way to do this.
Reviewed By: jsgf
Differential Revision: D6323440
fbshipit-source-id: 544f27e6ec210ddf840212b0c0c94145980e8be3
Summary:
We have too many parameters, and cmd-line args are no longer good enough. Let's
use toml based config, with a format that's similar to mononoke server
repoconfig.
Reviewed By: farnz
Differential Revision: D6313512
fbshipit-source-id: 61379e93707a4fcbb42aa8a0392ce526dab19f2e
Summary: Also support them in `generate_memblob_repo.py`.
Reviewed By: jsgf
Differential Revision: D6215721
fbshipit-source-id: c8b855bbc74b619bc093b3aac449a283a1ad33ec
Summary:
This makes it quite easy to write out linknodes.
Also regenerate linknodes for our test fixtures -- the next commit will bring
them in.
Reviewed By: jsgf
Differential Revision: D6214033
fbshipit-source-id: 3b930fe9eda45a1b7bc6f0b3f81dd8af102061fc
Summary: Let's not be too restrictive about urls
Reviewed By: markbt
Differential Revision: D6099460
fbshipit-source-id: 228a5b3a3cc0f09d68307a3695952543a2f5f05a
Summary: Use a more easily discoverable name.
Reviewed By: farnz
Differential Revision: D6191999
fbshipit-source-id: 140372bb714af7b8c2fff481f257264acb719735
Summary: To make folder name consistent with the buck target
Reviewed By: farnz
Differential Revision: D6098739
fbshipit-source-id: 7d876a0fa352d98a5d4164093d07ebe7388680e7
Summary: the glog drain does not swallow f.e. backtrace of error_chain errors, so it is a bit easier to debug the tool
Reviewed By: farnz
Differential Revision: D6021671
fbshipit-source-id: 32bfe01bfd77d85c37a2a446cb3e5d000763c689
Summary: Make sure we use treemanifest in the tests, and add commit with a directory
Reviewed By: farnz
Differential Revision: D5974524
fbshipit-source-id: 5ef9f110243001e1916509f53741cccb42dc8deb
Summary:
In Eden server we want to return list of file sizes for each file in the directory.
For now we are doing a separate call to the blobrepo. Since there can be lots of such calls, let's make them in parallel using cpupool
Reviewed By: lukaspiatkowski
Differential Revision: D5950452
fbshipit-source-id: 6b852a49c201ccc14ceb8f950dd61fa5cdebad0a
Summary:
There was an upstream fix 573baab2a797 that fixed run-tests --interactive for
test with many test cases. This diff grafts the change
Reviewed By: lukaspiatkowski
Differential Revision: D5902793
fbshipit-source-id: 0b90a0105d0bf0e2d9f5f788503dc40009441c40
Summary:
Mercurial filelog entries may have metadata fields in the beginning, usually used to track copies/renames. Previously mononoke eden server returned this metadata as part of the file blob.
This diff changes it. Now `get_content()` method returns file content without metadata, and to make it consistent, both `get_content()` of the blobrepo and revlog repo do the same.
This decision certainly has it's tradeoffs, because now it's more difficult to get metadata (`get_raw_content` needs to be used).
But we'll probably change how metadata is stored anyway, that's why I think changing `get_content` method is fine.
This diff also cleans up server/src/main.rs file, because previously it had to strip metadata itself.
Also diff fixes problem in metadata parsing - it previously failed if file is less than 2 bytes
Reviewed By: farnz
Differential Revision: D5901476
fbshipit-source-id: f3ade0179710352590068c238e6a733aab68a512
Summary:
Mercurial test suite have #testcases features. It allows to use the same test
file for two different tests while tweaking just small parts of it.
I used testcases in test-eden-server.t, but I used it incorrectly - #files test
case wasn't actually used at all. This is because hghave is not present in the
repo.
This diff adds hghave. It's a copy-paste from upstream mercurial
Reviewed By: farnz
Differential Revision: D5901439
fbshipit-source-id: 2131733d8c523b7f7ce9e6a087c77fed6a427e6d
Summary:
`Path` has the potential to be confused with `std::path::Path`.
`MPath` is nice, concise, and clearly different from `Path`.
Reviewed By: jsgf
Differential Revision: D5895665
fbshipit-source-id: dc5ed5c3866b227d753c6d904d3c6d213c882cd7
Summary: Compaction can slow down blobimporting a lot. Let's add an option to postpone it till the end
Reviewed By: farnz
Differential Revision: D5882003
fbshipit-source-id: 0611a8e94b3d7331bdacf909d820526f547414a0
Summary:
Manually adding all the dependencies in the TARGETS file is quite annoying. This simple tool does it for you!
It queries buck to get list of files for the target, and also queries all the rust_library mononoke dependencies.
Then it parses source files and generates TARGETS list.
Tool is super-simple and that's why it's not always correct (for example, it fails to find a deps if hasn't been used in mononoke before), but it works good enough for many use-cases (see test plan).
Reviewed By: lukaspiatkowski
Differential Revision: D5861828
fbshipit-source-id: ab8cc96d8bc394af172ee09da3aceb0ffb7493e8
Summary:
We're going to add more stores like obsmarkers and linknodes very
soon. Stuffing all of them into type parameters is going to get ugly very soon.
Instead, add a trait which represents all the state, and make `BlobRepo`
generic on that trait. Add a few implementations for common use cases like
files and RocksDB-based stores.
This allows the dependency lists for the Mononoke and Eden servers to be more
straightforward.
Reviewed By: jsgf
Differential Revision: D5850372
fbshipit-source-id: 37a0687611687b9616ebbddce70f53e1e5d3267b
Summary: I'm going to move this to a common location very soon.
Reviewed By: jsgf
Differential Revision: D5850319
fbshipit-source-id: 09937659330de93cf74bf827f225418dd413865d
Summary:
We want to be able to test against repo fixtures. Turn repo fixtures
that have been created by blobimport into in-memory BlobRepos for testing.
Reviewed By: jsgf
Differential Revision: D5834109
fbshipit-source-id: d6fc01f19986b5a6951f25c7c8bda8aa7c9dee56
Summary: Makes a formerly manual process automatic.
Reviewed By: jsgf
Differential Revision: D5832456
fbshipit-source-id: 0bfa01aaf08f0794a3b98ab7290d163a65650a7e
Summary: Simple linear history with 10 changesets as a file blob repo.
Reviewed By: sid0
Differential Revision: D5731223
fbshipit-source-id: bc479304d13f7a2299d3ecc382e052ad5600c46c
Summary: Method that returns content of the manifest entry in json format.
Reviewed By: sid0
Differential Revision: D5527659
fbshipit-source-id: 1832b645f69da40cbd620a6bff318e25594c5148
Summary:
Server can construct boxed Repo instances, and performs all client protocol operations in terms of trait methods. This allows the repo to be set dynamically.
This currently makes the repo type a single setting which applies to all repos; this can be generalized to a per-repo setting later.
Reviewed By: sid0
Differential Revision: D5540670
fbshipit-source-id: ac450b078849d9bd65d6e3dc91b73d652ca86ce7
Summary:
Soon I'm going to add more different url handlers. And current url handling is
neither robust nor extendable. Let's use regexes to do url handling
Reviewed By: jsgf
Differential Revision: D5489445
fbshipit-source-id: f418c3f40892903b573a33e57899b47e5719a5ea
