Summary:
Mononoke hasn't written out heads in a while, so use the topology file
instead.
Reviewed By: StanislavGlebik
Differential Revision: D8371751
fbshipit-source-id: f9af9dddc3349a41c5e9f3a67033bcb3c739f511
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary:
`hg pushbackup` got moved to the `infinitepushbackup` extension.
Also fix some minor output changes.
Reviewed By: StanislavGlebik
Differential Revision: D8310862
fbshipit-source-id: ae8d5579dd4082bd91e80beef4a4b675af9a8986
Summary: Unfortunately even 10 seems to be too little for stress runs.
Reviewed By: StanislavGlebik
Differential Revision: D8310639
fbshipit-source-id: 280824a7c2ec2283d2ef3acfb968e336ca13c93b
Summary: See the comment for a full explanation.
Reviewed By: jsgf
Differential Revision: D8274912
fbshipit-source-id: 936baa60925fff5ddf1e903c8408410c0a429e77
Summary: See previous diff for the movitvation
Reviewed By: jsgf
Differential Revision: D8207273
fbshipit-source-id: 527c1d97546afedf36f84059ccb7c740bd412907
Summary:
Instead of writing changesets one-by-one run mutiple of them at once.
The size `100` for the buffer is arbitrary, but it shouldn't matter much since we already have backpressure on the database writes.
Reviewed By: farnz
Differential Revision: D8057268
fbshipit-source-id: ca3766505395dcb6be6684323462f1bb23222435
Summary: printing every CS is too verbose, but we still want to see progress in non-debug mode
Reviewed By: kulshrax
Differential Revision: D7925747
fbshipit-source-id: c3ed92ef8c8fbf7714779a2bf011d31c94aefa37
Summary: This allows us to put hgcli on the different host
Reviewed By: jsgf
Differential Revision: D7910980
fbshipit-source-id: 45bdd1ee42b54f8f1a425ed6950e785cdf85bfc3
Summary:
Curently hgcli can connect to Mononoke only if they are on the same host,
because unix domain socket are used. Also Mononoke has to use separate unix
domain socket for different repos.
The goal of this stack of diffs is to remove these limitations:
1 Make it possible to have hgcli and Mononoke server on different hosts
2 Make it possible to use one port/unix domain socket to connect to many repos.
This diff adds a separate thread that parses the Preamble, extracts the
reponame and sends request to a thread responsible for this repo.
Also hgcli now has a new cmd line option that specifies path to connect to.
Reviewed By: jsgf
Differential Revision: D7845156
fbshipit-source-id: a48bcfeec4755b2d5b3dfcf8e0383e6945178018
Summary:
This will make it easier to change the "real" bookmark type from AsciiString to
String if we decide to do that.
BookmarkPrefix is a separate type because we may want to change it from
AsciiString to String. Also we don't want to confuse a bookmark prefix with a
bookmark name.
Reviewed By: jsgf
Differential Revision: D7909992
fbshipit-source-id: 3d4d075c204ed5ef1114a743430982c2836bac04
Summary:
These have bitrotted, but I need a test case for a Scuba change. Fix
them.
Reviewed By: StanislavGlebik
Differential Revision: D7813907
fbshipit-source-id: e4e9b01a8a3c1de27f59c6d5ea695152df99d4ff
Summary: The deltas are meant to be applied on raw filenode content that has incorporated copy/move information. Make it so
Reviewed By: jsgf
Differential Revision: D7777329
fbshipit-source-id: 23ce90269cdf5dbb8a16aab026c031c2af68fccb
Summary: the idea of Mercurial heads in Mononoke will be represented by bookmarks, so there is no need to have them around
Reviewed By: StanislavGlebik
Differential Revision: D7775032
fbshipit-source-id: 1618a1e51862d7c115b2955082f40ee890a045f1
Summary: Although this change is breaking with the traditional definition of heads in Mercurial, we decided that using bookmarks as heads is sufficient for fbsource use case as well as it matches the git model where every unnamed branch is garbage collected.
Reviewed By: jsgf
Differential Revision: D7745541
fbshipit-source-id: 2ec716db7799bf7bab8b2b77a85d1521a880f3df
Summary: Being able to push multiple bookmarks in a single hg push is required for using hg push as tailing of fbsource which contains few remote bookmarks
Reviewed By: StanislavGlebik
Differential Revision: D7743737
fbshipit-source-id: ba24445762baafbaa5b3295dc8995fe871f97872
Summary: new blobimport has taken over integration tests
Reviewed By: StanislavGlebik
Differential Revision: D7710186
fbshipit-source-id: 7d61f3239d8edbb1f458dfde0abf59914a912aeb
Summary: The eden integration test contains a commit with no content which new_blobimport couldn't import. With this changes the commit API is capable of handling such commits.
Reviewed By: jsgf
Differential Revision: D7709243
fbshipit-source-id: 7d55eb2ec421820d189ab05b0f8cb4411f850a7b
Summary: The problem in compute_changed_files is that for 2 parents given instead of producing a list of files that were changed in the merge commit from perspective of both parents it produced the opposite of that. This should fix it
Reviewed By: jsgf
Differential Revision: D7708215
fbshipit-source-id: d58457054625beb853f61cb060d25b09279ecceb
Summary: After ceasing recomputation of NodeHashes the newblobimport is working in many of our tests as a replacement of blobimport
Reviewed By: sid0
Differential Revision: D7707684
fbshipit-source-id: e7b4391916cd4a37968afd828f456a7b49ecabf9
Summary:
5 seconds appears to still not be enough for stress test runs,
unfortunately. Try bumping it up a bit more.
Reviewed By: StanislavGlebik
Differential Revision: D7663231
fbshipit-source-id: 8dc82afd64185c12fd98138b96b981b7f0e9173e
Summary:
Currently, any sort of `Bytes` can be stored in the blobstore. That
caused me to make several mistakes while writing the code to store bonsai
changesets, because I'd just end up storing the wrong set of `Bytes`.
Introduce stronger typing so that only types that explicitly implement
`BlobStorable` can be stored in the blobstore.
Currently, these sorts of blobs can be stored in the blob store:
* `ChangesetBlob` and `ContentBlob` in `mononoke-types` (these are Thrift-serialized structures)
* The envelope `RawNodeBlob` and `RawCSBlob` types in `blobrepo`, once converted to `EnvelopeBlob` instances
* `HgBlob`, which contains revlog data (manifests or files) exactly as serialized by Mercurial
Reviewed By: StanislavGlebik
Differential Revision: D7627290
fbshipit-source-id: d1bcbde8881e365dec99618556e7b054985bccf7
Summary:
This is not only the newer, more specific type -- it also makes a couple
of upcoming diffs more straightforward.
Reviewed By: StanislavGlebik
Differential Revision: D7622906
fbshipit-source-id: 4e453b827512c538f4f9777ae4d24627f3b124cf
Summary: mercurial_types::DChangesetId should be replaced by types from mononoke_types in most cases and by mercurial::HgChangesetId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7618897
fbshipit-source-id: 78904f57376606be99b56662164e0c110e632c64
Summary: mercurial_types::NodeHash should be replaced by types from mononoke_types in most cases and by mercurial::NodeHash in others. This rename should help with tracking this fact.
Reviewed By: sid0
Differential Revision: D7618389
fbshipit-source-id: a876e723d911df626c7851fba56a056843b4e049
Summary: They are replaced by filenodes
Reviewed By: farnz
Differential Revision: D7443320
fbshipit-source-id: 13c7d07bc00dcbaa991663c8da8a07fcb0de1332
Summary:
Since `FileType` now exists, the `Type` enum can use it instead of
defining its own stuff.
Reviewed By: farnz
Differential Revision: D7526046
fbshipit-source-id: 3b8eb5502bee9bc410ced811dc019c1ce757633f
Summary:
The current "run_as_bundle" mode is suboptimal because it causes all
the tests to be run within the same integration runner run. This means that
testpilot can't do things like limit the number of tests run in parallel.
With this change, each test will be run as a separate `integration_runner`
instance.
Thanks to DragonMinded for help figuring this out.
Reviewed By: DragonMinded
Differential Revision: D7493887
fbshipit-source-id: 8d55c2a64573b6bc213dac3daf0d78aa3432d430
Summary:
Use filenodes in BlobRepo instead of linknodes.
That involves changes in commit API because we need to write filenodes not linknodes.
Also it involves changes to memblob test repos.
Note that in this diff they are basically "broken" - linknodes were replaced with empty filenodes. However looks like our unittests haven't exercised this functionality, so no unit-tests has failed. In the next diffs the correct filenodes will be added.
Reviewed By: farnz
Differential Revision: D7413481
fbshipit-source-id: 7f994ea55887b96a0eacf8e6ffdebd1f8c5f025d
Summary:
They do not provide a lot of value, so let's not have them at all. It will make
adding filenodes easier.
Reviewed By: farnz
Differential Revision: D7428601
fbshipit-source-id: 647fa36d962cb6a8996f92246e4d900751040a52
Summary:
Our test repos had flatmanifests. Now blobimport doesn't work with flat manifests.
Generate tree manifests with `hg backfilltree`
Also run `backfilltree` as a part of regenerate script
Reviewed By: farnz
Differential Revision: D7427344
fbshipit-source-id: 134bd0caa7e2b6724664d07598718000f41c69db
Summary:
For historical reasons (back when we thought Mononoke might need to handle both flat and tree manifests natively), we decided to make the entry type in a Manifest be `MPath`, not `MPathElement`.
In the tree manifest only world we're now in, it's a bug if a Manifest contains a path instead of a path element - enforce this through the type system, so that we can't backslide accidentally.
Reviewed By: StanislavGlebik
Differential Revision: D7413870
fbshipit-source-id: fa13b676475ea8a3265fc962910542f2f555d7de
Summary:
One spot was missed when updating to allow for config repos now being
blobrepos
Reviewed By: StanislavGlebik
Differential Revision: D7414012
fbshipit-source-id: f29e499a5863e02d6d2a2412e8cdf26814950ca7
Summary: Make it possible to use a simulated remote backend as a blobstore. This allows me to look at the test results and be happy that Mononoke is at fault for any slowness.
Reviewed By: kulshrax
Differential Revision: D7353229
fbshipit-source-id: 57528af704b517a70570bb2b9b140caa9120a956
Summary:
Run changeset db operations in worker threads to make them async as
far as the rest of the system is concerned.
Reviewed By: farnz
Differential Revision: D7350002
fbshipit-source-id: 66fadf9ad2f16929e0c07a6907aa9d5f5a7075a8
Summary:
There's no point passing it by reference since callers don't need to
retain it, and the async implementation needs to move it into another context.
Reviewed By: farnz
Differential Revision: D7350001
fbshipit-source-id: 5947557a84621afae801dc20e3994496244e3a10
Summary:
Seems to flap around between 152 and 156 bytes. Not sure why, but it
isn't very relevant for the test anyway.
Reviewed By: farnz
Differential Revision: D7380504
fbshipit-source-id: 91cacb39ed0d67f1df6ac4138f4250c757b5aec3
Summary:
This codemod tries not to change the existing behavior of system, only introduce new types specific to Mercurial Revlogs.
It introduces a lot of copypasta intentionally and it will be cleaned in following diffs.
Reviewed By: farnz
Differential Revision: D7367191
fbshipit-source-id: 0a915f427dff431065e903b5f6fbd3cba6bc22a7
Summary:
To do this we have to make changegroup part optional, because `hg push --delete
BOOK` doesn't actually send changegroup at all.
This also fixes the issue with `hg pushbackup` that sends only infinitepush
bookmarks part (see test for details)
Reviewed By: farnz
Differential Revision: D7271596
fbshipit-source-id: 77fa26e176fd4e31f62e0412249a934ee1176ee0
Summary:
Actually update the bookmarks using pushkey part. Note that bookmark deletion
doesn't currently work. It will be fixed in the next diffs
Reviewed By: farnz
Differential Revision: D7271600
fbshipit-source-id: fd13d1adfd3ea490174c31e23289e8560dc2b737
Summary:
This wireproto method is used by remotenames to update the list of
remotebookmarks. Implementation is the same as for listkeys bundle2 part.
Reviewed By: farnz
Differential Revision: D7271597
fbshipit-source-id: 8a75a93cae0e571d86d657e1c1d718a7fa0ab4ea
Summary:
I want to see if two commits hits unexpected serialization. Test that
case, too.
Reviewed By: kulshrax
Differential Revision: D7330451
fbshipit-source-id: da80382f46bc4c1d928fa885e03460626db109c6
Summary:
I want to be able to get perf numbers for the commit API, and to avoid
human error contaminating the numbers I get. Script the test, to be run
manually.
Reviewed By: kulshrax
Differential Revision: D7261639
fbshipit-source-id: d8a39f79fd2362e4ca5e3aac652a4bf2f3cb45c9
Summary:
The new_blobimport as opposed to the old one do two things differently:
1. It uses a better structured API of RevlogRepo. The old one reads the Revlogs directly and does not verify if the data it has read is correct or it does not let us fix it into canonical form easily (once we have a canonical form different from Revlog's).
2. It uses BlobRepo's Commit API instead of writing directly to storage. This ensures consistency in our code and let's us leverage the validation that is incorporated in Commit API.
Reviewed By: farnz
Differential Revision: D7041976
fbshipit-source-id: fe592524533955f364f1b037109b3b5b5bab6b02
Summary:
Codemoding imports from libfb.py of the format "from libfb import X".
This is part of a larger codemod to remove the mapping from libfb/py to libfb,
in the interest of enabling static typechecking in fbcode.
Reviewed By: dark
Differential Revision: D7335696
fbshipit-source-id: f1a1c3b11bec15610ef2bb24cd1c2941f9409b49
Summary: just to excersise our RocksDb blobstore which might come in handy more often than file based as it should be more efficient
Reviewed By: StanislavGlebik
Differential Revision: D7290069
fbshipit-source-id: ce776cfa14e43dc45cca796ef187655ba665d177
Summary: This test should excercise the situation when a path to a file is hashed when stored in revlog and for a large file the index revlog is split into index and data revlog
Reviewed By: StanislavGlebik
Differential Revision: D7299332
fbshipit-source-id: 6b0f0385e391c8396d5a3702ced5feb1aba2163c
Summary:
Pushkey part is used to send bookmark updates from hg client to the server.
This diff does all the wireproto parsing, but doesn't actually apply bookmark
updates on the server.
Also this diff "implements" branchmap method. We have no plans to support it,
but currently remotenames extension calls it. So this diff adds a fake
implementation that always returns empty response.
Reviewed By: farnz
Differential Revision: D7150973
fbshipit-source-id: 6889c02a1105127b1805ef1fafa6fbe9c2d57e7d
Summary:
Mononoke will introduce its own ChangesetId, ManifestId and BlobHash, and it
would be good to rename these before that lands.
Reviewed By: farnz
Differential Revision: D7293334
fbshipit-source-id: 7d9d5ddf1f1f45ad45f04194e4811b0f6decb3b0
Summary:
I've spent quite some time but wasn't able to get to the bottom of why
test-infinitepush.t has been flaky. For some reason error message
```
Expected Bundle2 Changegroup
```
hasn't made it's way to the hg client. Adding a multisecond delay before
sending the error "fixes" the problem, but I haven't found the actual reason
behind the problem. So since that's not a #1 priority now I suggest to postpone
fixing it for now.
Reviewed By: farnz
Differential Revision: D7246511
fbshipit-source-id: a385130e6bdc978765e04f44c6a536405ee12e02
Summary:
The reason for flakiness was in unpredictable order of the tree items from
Mononoke. In the first gettreepack call we request two different revisions, and
Mononoke can return them in any order. hg handles it just fine (see updated
test with `hgmn up ...`), but hashes of the tree packs may change.
To remove flakiness let's not rely on the treepack files hash
Reviewed By: farnz
Differential Revision: D7237559
fbshipit-source-id: c04e9d45c41f1d288a90706d0ecc27ede36f8008
Summary:
It was flaky because server may not yet be ready to accept connections when we
send first request. Sometimes delay needs to be > 1 sec.
Let's query eden server in a loop until it responds.
Reviewed By: farnz
Differential Revision: D7233069
fbshipit-source-id: 8bcb5b2b8ebdc52d2447b33e18580e50c1e27031
Summary:
These are types that are going to be used throughout Mononoke -- I'm hoping to
avoid references to current Mercurial data structures in here.
For now, the only module I've moved is part of the `path` module. The
Mercurial-specific `fsencode` bits have been kept in mercurial-types (though
maybe they should move to `mercurial`?)
In the future, this module will also include definitions for unodes, etc.
Reviewed By: jsgf
Differential Revision: D7188722
fbshipit-source-id: fc097ca12c38a787f83e35af9b8dd308f2b910ea
Summary: I'm going to need a logger to log future-stats output to (and later trace output). Thread one through to BlobRepo
Reviewed By: StanislavGlebik
Differential Revision: D7167450
fbshipit-source-id: 4ed729e4d448b66e491cefa19380d3be9bc99091
Summary: Replace the generic types if `Blob` and `BlobNode` with `Bytes`.
Reviewed By: lukaspiatkowski
Differential Revision: D7115361
fbshipit-source-id: 924d347377569c6d1b3b4aed14d584510598da7b
Summary:
This is the part that sends client bookmarks to the server. Quite likely we are
not going to process it, so we are just reading the part and ignoring it.
Reviewed By: farnz
Differential Revision: D7123758
fbshipit-source-id: 7ad39bde77b6f77cf6e440e726ac3bdb9f340cea
Summary:
Support b2x:infinitepush part. It contains changegroup v2, so just reuse the
normal push path for it.
Note that pushbackup still fails because we don't support
b2x:infinitepushscratchbookmarks part. Also all wireproto params are ignored
for b2x:infinitepush.
Reviewed By: farnz
Differential Revision: D7086120
fbshipit-source-id: 2f98e5d59059ca3c2b82842c98e6dc771c70c6f0
Summary:
This is a hacky way of getting the push/pull working. We should instead remove the commits that are no longer heads from HeadStore and add those that are the new heads.
In this diff though we add all the commits as heads, just because this does not break the client
Reviewed By: farnz
Differential Revision: D7112279
fbshipit-source-id: 036f0fd230de52e96cbf4168c2cda7c2a1c5bd89
Summary:
From upload_entry perspective NodeHash is not unique for uploaded entry, but when combined with RepoPath it is. An example are two files with the same content, but different paths.
Additionally in this diff the requirement for uploaded blobs to be unique is loosened for Manifests, because the b2xtreegroup might contain duplicates of Manifests
Reviewed By: farnz
Differential Revision: D7087863
fbshipit-source-id: 7e9c2438db037fa171f1e65b6882b445e8c09f7a
Summary: It seems that my assumtion that Filelog End section follow every Filelog Chunk section was wrong, because there might be multiple Filelog Chunk sections being different version of the same file from different commits.
Reviewed By: farnz
Differential Revision: D7073007
fbshipit-source-id: 3baee3fd29c77aabf4173a618509de7ff88e4de6
Summary: The assumption that deltas can be decoded using only bundle2 was wrong, we need to be able to fetch content of other files from the BlobRepo for delta application.
Reviewed By: StanislavGlebik
Differential Revision: D7085957
fbshipit-source-id: 2f6803d7f61389c5ba38b1207ede42579b9cf2e6
Summary: This will make it easy to track down an error without looking at stacktraces
Reviewed By: StanislavGlebik
Differential Revision: D7087301
fbshipit-source-id: c5460dae9c5c9ab43713e5db1457f5d9155b5e8e
Summary:
Changesets table gives us a relatively fast way to get generation number. Let's
use it inside repoinfo crate.
Reviewed By: sid0
Differential Revision: D7043829
fbshipit-source-id: 82557340a7f2fd6361f4e29c1a11e38e473eeb15
Summary:
Iff all the inserts finished successfully, then it's safe to mark changesets as complete.
This diff fills up changesets store after blobimport successfully finishes.
For simplicity if --commit-limit or --skip is set then we skip filling up the changeset store.
Reviewed By: sid0
Differential Revision: D7043831
fbshipit-source-id: 8ae864b45222d52281c885a49c2dca44ba577137
Summary: Changests store requires it in it's api methods. Let's pass repoid from configs
Reviewed By: farnz
Differential Revision: D7043830
fbshipit-source-id: e4e4d5852d0ca8488cabe2140555508c143ab8df
Summary:
For now it does nothing. In the next diffs it will be used to tell if commit
exists in the repo or not and to speed up revsets
Reviewed By: farnz
Differential Revision: D7043828
fbshipit-source-id: 9fcc668e68ba238123a89f18ff67828848ba0cec
Summary: The introduced commits feature things like adding blob that should already be present, adding blob twice and having non-linear history
Reviewed By: farnz
Differential Revision: D7071447
fbshipit-source-id: e26b7792808351e2380d68b9eb3a4d7e6e859b0e
Summary: BlobChangeset has some methods that are useful also outside of blobrepo crate, like getting it's node.
Reviewed By: farnz
Differential Revision: D7056035
fbshipit-source-id: 197d261fd21ab9332950d3fda401e7dab3730cb5
Summary: This code handles deltaed filelogs and resolves them into proper Filelogs
Reviewed By: farnz
Differential Revision: D7011702
fbshipit-source-id: e8dc4844657011bc1085463eedd1790b87d317dc
Summary:
Return a reply to a client so that it doesn't fail.
Reply consists of just one replychangegroup that tells a client that the push
has succeeded (even though currently it wasn't).
resolve() function returns future of Bytes instead of a stream. It may be
suboptimal, but should be fine for now, and it's already used by a few
wireproto methods.
Reviewed By: lukaspiatkowski
Differential Revision: D7010578
fbshipit-source-id: 9b5425b912c640d4e2bac957a02e9881813b8871
Summary:
This change removes get_changeset_by_nodeid and replaces it with
get_changeset_by_changesetid, and propagates the changes to callers.
A few places still have ChangesetId::new() because I'm not sure where
the original NodeHash comes from. If you have any pointers, I would be
happy to fix them before landing.
Reviewed By: lukaspiatkowski
Differential Revision: D7031923
fbshipit-source-id: cd00ea1d2b955538e26d7b5735aed33fe0ae0330
Summary:
The current Memblob store is eager; this is great for finding certain
classes of bugs (those that assume an ordering that is not guaranteed), but not
so good for exposing other classes of bugs (those that assume that the future
has done its work before it resolves).
Add a lazy variant that functions in the same way as Memblob, but that waits
until it's polled to return.
Reviewed By: StanislavGlebik
Differential Revision: D7033792
fbshipit-source-id: 4c2d8a8150d908bcb26347757f96f99e20d74fc2
Summary:
Use hgclone_treemanifest instead of hgcloneshallow.
Note that it changes the behaviour of test-infinitepush.t - now it sends
treegroup part too. Since we actually want to send treegroup part during
infinitepush push, that means that the test was broken and now it fixed.
Reviewed By: lukaspiatkowski
Differential Revision: D7000046
fbshipit-source-id: 61b4517f23081faab1738f44d27220ae8063e077
Summary:
In D6988301 luk noticed that test output is incorrect. Turned out that client
asked server to fetch revision that exists only client-side.
To fix the test let's set treeonly=True and do shallow clone.
Figuring out why exactly it won't work without these flags is very
time-consuming. But that's the usual setup we expect our clients to have.
Besides test-init.t have used these flags and worked fine, so does
test-push-protocol.t
Reviewed By: lukaspiatkowski
Differential Revision: D7000045
fbshipit-source-id: 5e3a6e9fc9c0a1ed0a1e2074f5ac86533d42fa09
Summary:
1) Do not use system hg in tests.
Let's use current version of hg in the tests. It will simplify debugging
because we can now easily tweak both hg client and server. It will also result
in fewer test breakages, because previously our tests were broken when system
hg was updated.
2) Run integration tests in parallel with -j option
Reviewed By: quark-zju
Differential Revision: D6998535
fbshipit-source-id: dac4eb6d33978161ffe42d9d6a5a39a3ddf3a4e3
Summary:
In D6923197 we've made it so that nullid always exists. This is necessary for
pull, because client can send nullids (for example, if client has an empty
repo). That has broken revset test.
Also gettreepack capability was added, that has broken test-push-protocol.t
Reviewed By: quark-zju
Differential Revision: D6988301
fbshipit-source-id: 2deafc3489de9f50126de7212656ffc075ef3537
Summary:
Quick recap: gettreepack is a treemanifest wireproto method.
It's used to send treemanifest data to the client. Client sends list of
manifest nodes it's interested in (mfnodes), and list of nodes it already has
(basemfnodes). Server should find the difference and send it back. Client
usually call this wireproto method when it checks out new revision (`hg
update`), but it uses in other cases too (for example, in `hg prefetch`).
Before we supported exactly one mfnode and exactly one basemfnode. This is
usually fine for `hg update` use-case, but `hg prefetch` can request many
mfnodes and can send empty basemfnode. So let's support this too.
Note that current implementation isn't efficient. It uses at most one
basemfnode, meaning that it can send data that client already has. Also for
each mfnode we generate separate stream of changed entries. That means that the
same entries can be fetched many times.
Reviewed By: lukaspiatkowski
Differential Revision: D6923197
fbshipit-source-id: d25f9a01ca568c84811ee1a13181e70eb217eb53
Summary:
Add an integration infinitepush test. Currently infinitepush is not implemented
in mononoke, so the test fails. We'll use it later to test our functionality.
Also add infinitepush wireproto capability, otherwise the test is useless - it
sends no infinitepush parts at all.
Reviewed By: quark-zju
Differential Revision: D6948967
fbshipit-source-id: 3b479c2c7f036c2b9424de0f160f742bb8ff7412
Summary:
Wireproto method to lookup if a revision/bookmark exists.
Currently supports only full hashes, later we'll add support for bookmarks and
node prefixes.
Reviewed By: farnz
Differential Revision: D6948358
fbshipit-source-id: af92a892eae8f787053447e601ebf95dfd638fa4
Summary:
remotefilelog getfiles method requires fetching the whole history. Current
implementation is naive and it's going to be slow on big repos.
But later we're going to replace it with mysql
Reviewed By: quark-zju
Differential Revision: D6913180
fbshipit-source-id: 9244272a0339a2a83cf5b5615963b65ad2fcdc07
Summary:
Currently missing a few things:
1) Fetching the whole history for a file
2) Dealing with renames
That will be fixed in the next diffs
Reviewed By: quark-zju
Differential Revision: D6913179
fbshipit-source-id: f847c61791488bb2842a245ae00de7420b97145e
Summary:
getfiles is a remotefilelog method to fetch file content. Mercurial client
calls it on-demand, usually during `hg update` command. So normal `hg pull`
with remotefilelog doesn't send filelogs back to the client.
This diff implements getfiles arguments parsing. The code is tricky, so I've
added lots of comments to explain what's going on.
Reviewed By: farnz
Differential Revision: D6913178
fbshipit-source-id: 0248993c3ac487956e0ff547996d51b75cdaee96
Summary: making this test work is a 2018 m1 milestone
Reviewed By: farnz
Differential Revision: D6950112
fbshipit-source-id: 798a1b8e722e96b27f0cb8267b82232d3fe0496d
Summary:
As we discussed before, let's add get_name() method that returns MPathElement,
and remove get_path() and get_mpath().
Except for renaming, diff also make repoconfig work with tree manifest, and
fixes linknodes creation in blobimport - previously basename was used instead
of the whole path.
Reviewed By: jsgf
Differential Revision: D6857097
fbshipit-source-id: c09f3ff40d38643bd44aee8b4488277d658cf4f6
Summary:
Pulling more than one node doesn't work, because mononoke revsets return
children before the parents. In that case mercurial client fails because new
child points to non-existent parents. To fix it let's reverse the output.
It's bad, because we lose the advantages of streaming, but we do it now anyway
because `getbundle()` method returns Bytes. Besides changelog data should be
fairly small.
Reviewed By: farnz
Differential Revision: D6899239
fbshipit-source-id: 67b3954bfbed8e0aaa69e0da453dae443cd24912
Summary:
Add a fixture where there are many files and directories.
History is linear for now, but later we may add merge commtis.
It will be used to test gettreepack method.
Reviewed By: lukaspiatkowski
Differential Revision: D6818544
fbshipit-source-id: 798a1bea66eb49491f98fe5e997b172f3c927c53
Summary:
Super-basic implementation of generating changegroup part in getbundle. All it does is sending changelog
entries - so no manifest and no filelog entries. While it's very simple, it may
actually be enough for a real production use-case. With treemanifest and
remotefilelog we don't need to pull anything except for changelog entries,
everything else will be downloaded on demand.
Reviewed By: jsgf
Differential Revision: D6748070
fbshipit-source-id: 1dcd802a4b3b111f935e713e7696d58f05861b77
Summary:
We're never going to serve RevlogRepo in production, and we're down to
a single BlobRepo type that will have different backing stores. Remove the
unused trait, and use BlobRepo everywhere bar blobimport and repo_config
(because we previously hardcoded revlog here - we want to change to a BlobRepo
once blobimport is full-fidelity).
Reviewed By: jsgf
Differential Revision: D6596164
fbshipit-source-id: ba6e76e78c495720792cbe77ae6037f7802ec126
Summary: It removes a bit of copy-pastes code
Reviewed By: sid0
Differential Revision: D6627250
fbshipit-source-id: b110dcad998a240b6da60bc3347c7c5c0370aae0
Summary:
It won't be used in prod anyway, and for local use rocksdb blob state is still
preferable
Reviewed By: sid0
Differential Revision: D6627197
fbshipit-source-id: d9ca55b0221c050e8e8e35914aff22906198874a
Summary:
The test wasn't waiting for the Mononoke server to be available. Also
improve the error message slightly.
Reviewed By: jsgf
Differential Revision: D6606102
fbshipit-source-id: d0fde39aef8b3423fd1a8996a01f12e8a9661597
Summary:
Fetching file sizes is very expensive - it's one more round-trip to the
blobstore.
Add method that returns tree content, but doesn't fetch blob sizes.
Reviewed By: jsgf
Differential Revision: D6545108
fbshipit-source-id: 3d335eafa9dc367fb9e607f5c147b7e5c37cf133
Summary:
Two tests can't be run simultaneously because they use the same port. This diff
fixes it.
Reviewed By: farnz
Differential Revision: D6457855
fbshipit-source-id: 3e5f82211bcd87abc608bdeab239fe680976316d
Summary: `changeset_exists` only checks that the changeset is in the blob store. Once we start accepting pushes, we'll also want to check that the changeset is reachable from a head. Use revsets to check this. Once we have a notion of "completeness" for changesets, we'll switch `known` to use that instead - this code is still useful as a way to go from `changeset_exists` to `changeset_complete` in bulk.
Reviewed By: jsgf
Differential Revision: D6205695
fbshipit-source-id: 7e3b5c30bc5e459feb95a20913d8a04f3fda7469
Summary:
BlobStore is entirely generic, and puts no limits on its
implementations. Remove ValueIn and ValueOut type parameters, and insist that
all blobs are Bytes (as per production setups)
Reviewed By: StanislavGlebik
Differential Revision: D6425413
fbshipit-source-id: 455e526d8baebd0d0f1906941648acca89be4881
Summary:
It maybe not installed on the system. Let's use simple python script instead.
Note: unfortunately we can't use 'python -m json.tool' because we need to sort
entries in the list
Reviewed By: jsgf
Differential Revision: D6437136
fbshipit-source-id: 0703f45c46d220046b9ef484bdf406a7f936557b
Summary:
As part of removing excess genericism, make Bookmarks a trait with no
associated types or type parameters. I will revisit error types here once I
have removed the Repo trait - these are not ideal, but enough to get things
going
Reviewed By: StanislavGlebik
Differential Revision: D6405315
fbshipit-source-id: 814c106612a061e1766f1ea9a9428a13a73bd007
Summary:
As part of removing excess genericism, make Heads a trait with no
associated types or type parameters.
Reviewed By: StanislavGlebik
Differential Revision: D6352727
fbshipit-source-id: df9ef87e0e0abe43c30e7318da38d7f930c37c6e
Summary:
Config file needs to have ca file in PEM format.
It is used to authenticate client.
See code comments for more details.
Reviewed By: jsgf
Differential Revision: D6323439
fbshipit-source-id: 6b6f1fd68605b263dcb33b051843e10d3f5cb38e
Summary:
Avoid using plain HTTP and use HTTPS instead.
To do this config needs to provide paths to server certificate and private key files in PEM format.
Then they will be converted to Pkcs12 archive.
This diff adds authentication of server i.e. client can check that it talks to a real server. Next diff adds authentication of a client.
Lower-level `hyper::server::Http::bind_connection()` is used instead of `hyper::server::Http::bind()` method in order to add TLS support.
See code comments for more details.
Implementation is more complicated than I expected it to be. I need to use 3 more new crates. Lmk if there is a better way to do this.
Reviewed By: jsgf
Differential Revision: D6323440
fbshipit-source-id: 544f27e6ec210ddf840212b0c0c94145980e8be3
Summary:
We have too many parameters, and cmd-line args are no longer good enough. Let's
use toml based config, with a format that's similar to mononoke server
repoconfig.
Reviewed By: farnz
Differential Revision: D6313512
fbshipit-source-id: 61379e93707a4fcbb42aa8a0392ce526dab19f2e
Summary: Also support them in `generate_memblob_repo.py`.
Reviewed By: jsgf
Differential Revision: D6215721
fbshipit-source-id: c8b855bbc74b619bc093b3aac449a283a1ad33ec
Summary:
This makes it quite easy to write out linknodes.
Also regenerate linknodes for our test fixtures -- the next commit will bring
them in.
Reviewed By: jsgf
Differential Revision: D6214033
fbshipit-source-id: 3b930fe9eda45a1b7bc6f0b3f81dd8af102061fc
Summary: Let's not be too restrictive about urls
Reviewed By: markbt
Differential Revision: D6099460
fbshipit-source-id: 228a5b3a3cc0f09d68307a3695952543a2f5f05a
Summary: Use a more easily discoverable name.
Reviewed By: farnz
Differential Revision: D6191999
fbshipit-source-id: 140372bb714af7b8c2fff481f257264acb719735
Summary: To make folder name consistent with the buck target
Reviewed By: farnz
Differential Revision: D6098739
fbshipit-source-id: 7d876a0fa352d98a5d4164093d07ebe7388680e7
Summary: the glog drain does not swallow f.e. backtrace of error_chain errors, so it is a bit easier to debug the tool
Reviewed By: farnz
Differential Revision: D6021671
fbshipit-source-id: 32bfe01bfd77d85c37a2a446cb3e5d000763c689
Summary: Make sure we use treemanifest in the tests, and add commit with a directory
Reviewed By: farnz
Differential Revision: D5974524
fbshipit-source-id: 5ef9f110243001e1916509f53741cccb42dc8deb
Summary:
In Eden server we want to return list of file sizes for each file in the directory.
For now we are doing a separate call to the blobrepo. Since there can be lots of such calls, let's make them in parallel using cpupool
Reviewed By: lukaspiatkowski
Differential Revision: D5950452
fbshipit-source-id: 6b852a49c201ccc14ceb8f950dd61fa5cdebad0a
Summary:
There was an upstream fix 573baab2a797 that fixed run-tests --interactive for
test with many test cases. This diff grafts the change
Reviewed By: lukaspiatkowski
Differential Revision: D5902793
fbshipit-source-id: 0b90a0105d0bf0e2d9f5f788503dc40009441c40
Summary:
Mercurial filelog entries may have metadata fields in the beginning, usually used to track copies/renames. Previously mononoke eden server returned this metadata as part of the file blob.
This diff changes it. Now `get_content()` method returns file content without metadata, and to make it consistent, both `get_content()` of the blobrepo and revlog repo do the same.
This decision certainly has it's tradeoffs, because now it's more difficult to get metadata (`get_raw_content` needs to be used).
But we'll probably change how metadata is stored anyway, that's why I think changing `get_content` method is fine.
This diff also cleans up server/src/main.rs file, because previously it had to strip metadata itself.
Also diff fixes problem in metadata parsing - it previously failed if file is less than 2 bytes
Reviewed By: farnz
Differential Revision: D5901476
fbshipit-source-id: f3ade0179710352590068c238e6a733aab68a512
Summary:
Mercurial test suite have #testcases features. It allows to use the same test
file for two different tests while tweaking just small parts of it.
I used testcases in test-eden-server.t, but I used it incorrectly - #files test
case wasn't actually used at all. This is because hghave is not present in the
repo.
This diff adds hghave. It's a copy-paste from upstream mercurial
Reviewed By: farnz
Differential Revision: D5901439
fbshipit-source-id: 2131733d8c523b7f7ce9e6a087c77fed6a427e6d
Summary:
`Path` has the potential to be confused with `std::path::Path`.
`MPath` is nice, concise, and clearly different from `Path`.
Reviewed By: jsgf
Differential Revision: D5895665
fbshipit-source-id: dc5ed5c3866b227d753c6d904d3c6d213c882cd7
Summary: Compaction can slow down blobimporting a lot. Let's add an option to postpone it till the end
Reviewed By: farnz
Differential Revision: D5882003
fbshipit-source-id: 0611a8e94b3d7331bdacf909d820526f547414a0
Summary:
Manually adding all the dependencies in the TARGETS file is quite annoying. This simple tool does it for you!
It queries buck to get list of files for the target, and also queries all the rust_library mononoke dependencies.
Then it parses source files and generates TARGETS list.
Tool is super-simple and that's why it's not always correct (for example, it fails to find a deps if hasn't been used in mononoke before), but it works good enough for many use-cases (see test plan).
Reviewed By: lukaspiatkowski
Differential Revision: D5861828
fbshipit-source-id: ab8cc96d8bc394af172ee09da3aceb0ffb7493e8
Summary:
We're going to add more stores like obsmarkers and linknodes very
soon. Stuffing all of them into type parameters is going to get ugly very soon.
Instead, add a trait which represents all the state, and make `BlobRepo`
generic on that trait. Add a few implementations for common use cases like
files and RocksDB-based stores.
This allows the dependency lists for the Mononoke and Eden servers to be more
straightforward.
Reviewed By: jsgf
Differential Revision: D5850372
fbshipit-source-id: 37a0687611687b9616ebbddce70f53e1e5d3267b
Summary: I'm going to move this to a common location very soon.
Reviewed By: jsgf
Differential Revision: D5850319
fbshipit-source-id: 09937659330de93cf74bf827f225418dd413865d
Summary:
We want to be able to test against repo fixtures. Turn repo fixtures
that have been created by blobimport into in-memory BlobRepos for testing.
Reviewed By: jsgf
Differential Revision: D5834109
fbshipit-source-id: d6fc01f19986b5a6951f25c7c8bda8aa7c9dee56
Summary: Makes a formerly manual process automatic.
Reviewed By: jsgf
Differential Revision: D5832456
fbshipit-source-id: 0bfa01aaf08f0794a3b98ab7290d163a65650a7e
Summary: Simple linear history with 10 changesets as a file blob repo.
Reviewed By: sid0
Differential Revision: D5731223
fbshipit-source-id: bc479304d13f7a2299d3ecc382e052ad5600c46c
Summary: Method that returns content of the manifest entry in json format.
Reviewed By: sid0
Differential Revision: D5527659
fbshipit-source-id: 1832b645f69da40cbd620a6bff318e25594c5148
Summary:
Server can construct boxed Repo instances, and performs all client protocol operations in terms of trait methods. This allows the repo to be set dynamically.
This currently makes the repo type a single setting which applies to all repos; this can be generalized to a per-repo setting later.
Reviewed By: sid0
Differential Revision: D5540670
fbshipit-source-id: ac450b078849d9bd65d6e3dc91b73d652ca86ce7
Summary:
Soon I'm going to add more different url handlers. And current url handling is
neither robust nor extendable. Let's use regexes to do url handling
Reviewed By: jsgf
Differential Revision: D5489445
fbshipit-source-id: f418c3f40892903b573a33e57899b47e5719a5ea
