Commit Graph

171 Commits

Author SHA1 Message Date
Lukas Piatkowski
38b66202cb server: print the session_uuid on the client side
Summary: Session UUID will help identify the issues on Mononoke side whenever the client encounters problems

Reviewed By: StanislavGlebik

Differential Revision: D8732396

fbshipit-source-id: 35d04b0d56be0cfc2c608f08287a2b1d236a96e3
2018-07-04 11:49:54 -07:00
Tim Fox
02ce67d40e Refactor repoconfig to support loading of hooks, and directory per repo
Summary:
This diff refactors the server config repository to support storing and loading of hooks. In the new structure each repo lives in its own directory and the config file for the server is called "server.toml".
Hooks can be referenced by relative or absolute paths allowing either local or common hooks to be loaded.

Reviewed By: StanislavGlebik

Differential Revision: D8625178

fbshipit-source-id: 62c8c515a0fbbf7a38cfc68317300d8f42eb4d7a
2018-07-04 07:07:02 -07:00
Stanislau Hlebik
18d3345c71 mononoke: fix test-gettreepack.t
Summary:
New parameter depth was added. Mononoke just ignores, which if correct, but can
be less efficient. However test-gettreepack.t was failing because it didn't
supply the parameter. This diff fixes it by sending a big depth to the server

P. S.
Attached task tracks the support of depth parameter on Mononoke server-side

Reviewed By: lukaspiatkowski

Differential Revision: D8712906

fbshipit-source-id: 8246344b3fa39f00eb39f2262b4aa5aa74834be2
2018-07-02 05:07:06 -07:00
Stanislau Hlebik
680c1b4fbd mononoke: fix some integration tests
Summary:
They were broken since 2646d41a64917d3e50f662b0b4b628ccfdbb05a8.
It added enabled_repo repository to config, however, no enabled_repo has ever been
blobimported. Since we don't really need enabled_repo, let's remove it

Reviewed By: farnz

Differential Revision: D8712891

fbshipit-source-id: 360f2c9564087d3a8411871335aa40fa81e38c0c
2018-07-02 04:06:06 -07:00
Jeremy Fitzhardinge
1a7ae3ed36 mononoke: add per-repo "enabled" flag
Summary:
Add a per-repo config flag to repos to be configed without being
enabled. Setting "enabled = false" will make Mononoke completely ignore the
repo config. If not present, "enabled" is assumed to be true.

Reviewed By: farnz

Differential Revision: D8647161

fbshipit-source-id: 2646d41a64917d3e50f662b0b4b628ccfdbb05a8
2018-06-29 20:21:36 -07:00
Stanislau Hlebik
d787921c35 mononoke: tls for hgcli and Mononoke server
Summary:
Use tls for connection between hgcli and Mononoke server always, even for
localhost connections[1]

The setup is similar to tls setup of Eden server.

[1] This is not necessary of course, but adding an option to bypass tls
connection may result in accidental use of it in prod. However if it turns out
to be too unusable, we can add such option in the future

Reviewed By: jsgf

Differential Revision: D8644299

fbshipit-source-id: 0898e30e33b718e13a766763479f3adf9323ffe7
2018-06-29 14:37:44 -07:00
Zeyi Fan
8f4e63cc50 Upgrade openssl, enable alpn for actix-web, add tokio-codec
Summary: This commit upgraded openssl, enabled alpn for actix-web and added tokio-codec with fixes due to the upgrade.

Reviewed By: StanislavGlebik

Differential Revision: D8682673

fbshipit-source-id: 8c7cadfd6c0c7b016202f6cb038eb4951d0f9333
2018-06-29 10:57:36 -07:00
Zeyi Fan
e924f7f105 Change file content retrieval API to /<changeset>/<path>
Summary: This commit changes the API from directly asking for NodeHash to changeset id and path of the file.

Reviewed By: StanislavGlebik

Differential Revision: D8628826

fbshipit-source-id: 1fa37cf36db0ca00530f3a60de78da1d1d232398
2018-06-27 17:52:35 -07:00
Zeyi Fan
6ef23f3646 Add /status for Tupperware health check
Summary: This commit adds /status to return ok for Tupperware health check.

Reviewed By: kulshrax

Differential Revision: D8629282

fbshipit-source-id: 8f36bf73d4d9399721c68649e6b475362770889b
2018-06-27 16:47:50 -07:00
Zeyi Fan
528b2bf36f Read filecontent from blobstore
Summary: This commit implements the retrieving logic for the api server.

Reviewed By: StanislavGlebik

Differential Revision: D8507784

fbshipit-source-id: 109b7fad35c5fefca593d04ac63d57534f9bd12b
2018-06-27 13:05:55 -07:00
Stanislau Hlebik
6545e0ee34 mononoke: rename test self-signed certs
Summary:
In the next diffs I'm going to use them in other tests, so the name is no
longer correct. Let's rename it

Reviewed By: jsgf

Differential Revision: D8644295

fbshipit-source-id: 455133b2bfcdfb98f69ae737e1d5f23e9e898c92
2018-06-27 07:52:53 -07:00
Stanislau Hlebik
51ab84156e mononoke: add a simple test for cache warmup
Summary:
Previously this code was completely untested. This diff adds a simple test that
checks that warmup has at least finished

Reviewed By: farnz

Differential Revision: D8611295

fbshipit-source-id: bf55d3b8953660040b4f9a934094fb688683c7ba
2018-06-25 07:36:01 -07:00
Zeyi Fan
eba437040d let apiserver read mononoke config
Summary: This commit lets apiserver reads Mononoke's config repository and creates MononokeRepo based on it.

Reviewed By: kulshrax

Differential Revision: D8465574

fbshipit-source-id: 0af1ad5f62499f83261e21bac605725156fc22d0
2018-06-22 15:36:44 -07:00
Lukas Piatkowski
8cc5514d41 server: refactor the code that reports success of failure of entire request
Summary: With this changes we will always get a sample representing result of handling a request written in scuba, regardless if it was success, failure or timeout.

Reviewed By: StanislavGlebik

Differential Revision: D8579277

fbshipit-source-id: 6c7943bb26c8880e4f3df988dcebd12fd57ed062
2018-06-21 16:53:11 -07:00
Rain ⁣
93c181c35d mercurial-types: simplify Manifest API significantly
Summary:
Manifests are always able to return entries immediately, and never
fail.

Reviewed By: lukaspiatkowski, farnz

Differential Revision: D8556499

fbshipit-source-id: e21a2522f1219e47db9b55b24b6ac6c0c463933e
2018-06-20 18:22:55 -07:00
Rain ⁣
4eca2ec2d9 upload file blobs as Thrift data structures
Summary:
This will also allow file blob sharing between the Mercurial and Mononoke
data models.

Reviewed By: farnz

Differential Revision: D8440330

fbshipit-source-id: a29cd07dcecf0959dffb74b7428f3cb11fbd3db6
2018-06-20 13:24:56 -07:00
Rain ⁣
8b05fd5c16 blobrepo: use thrift blobs for manifests
Summary:
Store manifests as Thrift blobs instead. Required fixing up a lot of
different places, but they should all be pretty clear now.

Reviewed By: farnz

Differential Revision: D8416238

fbshipit-source-id: 523e3054e467e54d180df5ba78445c9b1ccc3b5c
2018-06-20 13:24:56 -07:00
Rain ⁣
9ede6636a7 BREAKING CHANGE: switch to Thrift encoding for changesets
Summary:
Pretty straightforward. Also using this opportunity to add per-repo
prefixes, since all the hashes are going to break anyway.

Note for reviewers: almost all the change is regenerated test fixtures (unfortunately necessary to make atomic). The actual substantive changes are all in the first few files.

Reviewed By: farnz

Differential Revision: D8392234

fbshipit-source-id: c93fc8c6388cb00fe5cff95646ad8c853581cb8c
2018-06-20 13:24:56 -07:00
Zeyi Fan
1c4af3bda8 Create initial binary for Mononoke API Server
Summary: This revision adds an initial binary target for Mononoke API Server as well as a few basic options and logging.

Reviewed By: kulshrax

Differential Revision: D8413745

fbshipit-source-id: 65523433284e970348efcafd724ae28102d85671
2018-06-18 10:09:43 -07:00
Stanislau Hlebik
ca15bc472a mononoke: use SinkAsyncWrite in gettreepack
Summary:
Use SinkAsyncWrite that was added in the previous diff.
The main motivation is to make sure that we do not buffer bundle2 response in
memory. For more details look at the summary from the previous diff.

Note that this diff still buffers results in memory, but it prepares the code
for the next diffs, in which we'll finally remove the buffering.

Reviewed By: farnz

Differential Revision: D8379586

fbshipit-source-id: 5985ddb157f7066732ef47a04919eb1fba68c74e
2018-06-14 02:50:18 -07:00
Rain ⁣
e3ff3de80c blobstore: move memblob into main blobstore
Summary: It's really straightforward and useful for internal blobstore testing.

Reviewed By: farnz

Differential Revision: D8374496

fbshipit-source-id: 8501e31218ecb5d80530a114e0be5e5d0e8f6c91
2018-06-12 15:40:10 -07:00
Rain ⁣
388474e1bf rename new_blobimport to blobimport
Summary: The old blobimport is dead.

Reviewed By: farnz

Differential Revision: D8372593

fbshipit-source-id: 63595dbcd3ac6da1cad3c5ae6761ef01be6619ed
2018-06-12 15:40:10 -07:00
Rain ⁣
72ba438c95 switch fixture generation to new_blobimport
Summary:
Mostly this was about adding support for file stores to
`new_blobimport`.

Reviewed By: StanislavGlebik

Differential Revision: D8372063

fbshipit-source-id: 2e3791c6222ec430015008f038e1df0464d3f0ba
2018-06-12 15:40:10 -07:00
Rain ⁣
022879f49e generate_memblob_repo: don't read from heads file any more
Summary:
Mononoke hasn't written out heads in a while, so use the topology file
instead.

Reviewed By: StanislavGlebik

Differential Revision: D8371751

fbshipit-source-id: f9af9dddc3349a41c5e9f3a67033bcb3c739f511
2018-06-12 15:40:10 -07:00
Rain ⁣
cf102a1d53 simplify memblob generation code slightly
Summary: Makes upcoming code simpler.

Reviewed By: StanislavGlebik

Differential Revision: D8371506

fbshipit-source-id: 95480058aa78d3a21998ed42a1b52254136146e8
2018-06-12 15:40:10 -07:00
Lukas Piatkowski
4f44c3f130 mercurial_types: remove D* types and move mercurial types around
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge

Reviewed By: farnz

Differential Revision: D8319906

fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
2018-06-07 13:19:16 -07:00
Rain ⁣
02c2187851 tests: fix a couple of integration tests
Summary:
`hg pushbackup` got moved to the `infinitepushbackup` extension.

Also fix some minor output changes.

Reviewed By: StanislavGlebik

Differential Revision: D8310862

fbshipit-source-id: ae8d5579dd4082bd91e80beef4a4b675af9a8986
2018-06-07 09:51:31 -07:00
Rain ⁣
26356e4e3d increase the wait timeout in integration tests to 15 seconds
Summary: Unfortunately even 10 seems to be too little for stress runs.

Reviewed By: StanislavGlebik

Differential Revision: D8310639

fbshipit-source-id: 280824a7c2ec2283d2ef3acfb968e336ca13c93b
2018-06-07 09:51:31 -07:00
Rain ⁣
c18bbfa396 fix race with bookmark updates during pulls
Summary: See the comment for a full explanation.

Reviewed By: jsgf

Differential Revision: D8274912

fbshipit-source-id: 936baa60925fff5ddf1e903c8408410c0a429e77
2018-06-05 13:03:04 -07:00
Stanislau Hlebik
171a456691 mononoke: use visited_pruner
Summary: See previous diff for the movitvation

Reviewed By: jsgf

Differential Revision: D8207273

fbshipit-source-id: 527c1d97546afedf36f84059ccb7c740bd412907
2018-06-04 10:28:47 -07:00
Lukas Piatkowski
d0b695406a new_blobimport: buffer_unordered the changesets in main.rs
Summary:
Instead of writing changesets one-by-one run mutiple of them at once.
The size `100` for the buffer is arbitrary, but it shouldn't matter much since we already have backpressure on the database writes.

Reviewed By: farnz

Differential Revision: D8057268

fbshipit-source-id: ca3766505395dcb6be6684323462f1bb23222435
2018-05-21 06:24:12 -07:00
Lukas Piatkowski
1f58bd7060 new_blobimport: print number of uploaded CS every 5000
Summary: printing every CS is too verbose, but we still want to see progress in non-debug mode

Reviewed By: kulshrax

Differential Revision: D7925747

fbshipit-source-id: c3ed92ef8c8fbf7714779a2bf011d31c94aefa37
2018-05-11 12:54:46 -07:00
Stanislau Hlebik
7e540c6dbc mononoke: use tcp instead of unix socket for hgcli
Summary: This allows us to put hgcli on the different host

Reviewed By: jsgf

Differential Revision: D7910980

fbshipit-source-id: 45bdd1ee42b54f8f1a425ed6950e785cdf85bfc3
2018-05-10 02:02:40 -07:00
Stanislau Hlebik
a4e7962ffc mononoke: add a separate connection acceptor thread
Summary:
Curently hgcli can connect to Mononoke only if they are on the same host,
because unix domain socket are used. Also Mononoke has to use separate unix
domain socket for different repos.
The goal of this stack of diffs is to remove these limitations:
1 Make it possible to have hgcli and Mononoke server on different hosts
2 Make it possible to use one port/unix domain socket to connect to many repos.

This diff adds a separate thread that parses the Preamble, extracts the
reponame and sends request to a thread responsible for this repo.

Also hgcli now has a new cmd line option that specifies path to connect to.

Reviewed By: jsgf

Differential Revision: D7845156

fbshipit-source-id: a48bcfeec4755b2d5b3dfcf8e0383e6945178018
2018-05-10 02:02:38 -07:00
Stanislau Hlebik
e99fce6bd4 mononoke: add separate types for Bookmark and BookmarkPrefix
Summary:
This will make it easier to change the "real" bookmark type from AsciiString to
String if we decide to do that.

BookmarkPrefix is a separate type because we may want to change it from
AsciiString to String. Also we don't want to confuse a bookmark prefix with a
bookmark name.

Reviewed By: jsgf

Differential Revision: D7909992

fbshipit-source-id: 3d4d075c204ed5ef1114a743430982c2836bac04
2018-05-09 02:11:17 -07:00
Stanislau Hlebik
12968b1b9e mononoke: make it possible to fetch config repo by bookmark
Summary: It will make deploying mononoke considerably easier

Reviewed By: farnz

Differential Revision: D7831620

fbshipit-source-id: 77db867f5ce03975eadea7a8693bc88c1d1122cd
2018-05-03 10:31:32 -07:00
Simon Farnsworth
e3adaf7f50 Fix performance tests
Summary:
These have bitrotted, but I need a test case for a Scuba change. Fix
them.

Reviewed By: StanislavGlebik

Differential Revision: D7813907

fbshipit-source-id: e4e9b01a8a3c1de27f59c6d5ea695152df99d4ff
2018-04-30 10:44:04 -07:00
Lukas Piatkowski
346183fa8c blobrepo: fix delta application by applying it on raw content of filenode
Summary: The deltas are meant to be applied on raw filenode content that has incorporated copy/move information. Make it so

Reviewed By: jsgf

Differential Revision: D7777329

fbshipit-source-id: 23ce90269cdf5dbb8a16aab026c031c2af68fccb
2018-04-28 16:33:03 -07:00
Lukas Piatkowski
18800e00ea heads: remove related crates
Summary: the idea of Mercurial heads in Mononoke will be represented by bookmarks, so there is no need to have them around

Reviewed By: StanislavGlebik

Differential Revision: D7775032

fbshipit-source-id: 1618a1e51862d7c115b2955082f40ee890a045f1
2018-04-27 04:35:21 -07:00
Lukas Piatkowski
35656a1e7d blobrepo: use all server side bookmarks as heads for the sake of discovery
Summary: Although this change is breaking with the traditional definition of heads in Mercurial, we decided that using bookmarks as heads is sufficient for fbsource use case as well as it matches the git model where every unnamed branch is garbage collected.

Reviewed By: jsgf

Differential Revision: D7745541

fbshipit-source-id: 2ec716db7799bf7bab8b2b77a85d1521a880f3df
2018-04-25 04:22:19 -07:00
Lukas Piatkowski
800b605a35 bundle2-resolver: multiple pushkey parts for pushing multiple bookmarks
Summary: Being able to push multiple bookmarks in a single hg push is required for using hg push as tailing of fbsource which contains few remote bookmarks

Reviewed By: StanislavGlebik

Differential Revision: D7743737

fbshipit-source-id: ba24445762baafbaa5b3295dc8995fe871f97872
2018-04-24 11:17:06 -07:00
Lukas Piatkowski
15e35579fa performance tests: use new blobimport for importing config repo
Reviewed By: farnz

Differential Revision: D7710591

fbshipit-source-id: 0b4a58665749eb3e2af45f9a55ef6fa207d60c21
2018-04-24 11:17:06 -07:00
Lukas Piatkowski
5f43055821 integration tests: remove blobimport from integration tests
Summary: new blobimport has taken over integration tests

Reviewed By: StanislavGlebik

Differential Revision: D7710186

fbshipit-source-id: 7d61f3239d8edbb1f458dfde0abf59914a912aeb
2018-04-24 11:17:06 -07:00
Lukas Piatkowski
f28ce26504 blobrepo: handle changesets with null root manifests
Summary: The eden integration test contains a commit with no content which new_blobimport couldn't import. With this changes the commit API is capable of handling such commits.

Reviewed By: jsgf

Differential Revision: D7709243

fbshipit-source-id: 7d55eb2ec421820d189ab05b0f8cb4411f850a7b
2018-04-24 11:17:06 -07:00
Lukas Piatkowski
2187aa38ef blobrepo: fix compute_changed_files bug - use intersection instead of symmetric difference
Summary: The problem in compute_changed_files is that for 2 parents given instead of producing a list of files that were changed in the merge commit from perspective of both parents it produced the opposite of that. This should fix it

Reviewed By: jsgf

Differential Revision: D7708215

fbshipit-source-id: d58457054625beb853f61cb060d25b09279ecceb
2018-04-24 11:17:06 -07:00
Lukas Piatkowski
928356bfc8 integration test: use new blobimport for importing config repo
Reviewed By: StanislavGlebik

Differential Revision: D7708480

fbshipit-source-id: c658ab59356f2e76c4f84a87ddc55c453e8a6413
2018-04-20 12:17:49 -07:00
Lukas Piatkowski
040c1a2b4d integration tests: use newblobimport in integration tests where possible
Summary: After ceasing recomputation of NodeHashes the newblobimport is working in many of our tests as a replacement of blobimport

Reviewed By: sid0

Differential Revision: D7707684

fbshipit-source-id: e7b4391916cd4a37968afd828f456a7b49ecabf9
2018-04-20 12:17:40 -07:00
Siddharth Agarwal
e9854e9d3c tests: increase server start timeout to 10 seconds
Summary:
5 seconds appears to still not be enough for stress test runs,
unfortunately. Try bumping it up a bit more.

Reviewed By: StanislavGlebik

Differential Revision: D7663231

fbshipit-source-id: 8dc82afd64185c12fd98138b96b981b7f0e9173e
2018-04-18 19:53:34 -07:00
Siddharth Agarwal
4e9f826728 blobstore: restrict the types that can be stored
Summary:
Currently, any sort of `Bytes` can be stored in the blobstore. That
caused me to make several mistakes while writing the code to store bonsai
changesets, because I'd just end up storing the wrong set of `Bytes`.

Introduce stronger typing so that only types that explicitly implement
`BlobStorable` can be stored in the blobstore.

Currently, these sorts of blobs can be stored in the blob store:

* `ChangesetBlob` and `ContentBlob` in `mononoke-types` (these are Thrift-serialized structures)
* The envelope `RawNodeBlob` and `RawCSBlob` types in `blobrepo`, once converted to `EnvelopeBlob` instances
* `HgBlob`, which contains revlog data (manifests or files) exactly as serialized by Mercurial

Reviewed By: StanislavGlebik

Differential Revision: D7627290

fbshipit-source-id: d1bcbde8881e365dec99618556e7b054985bccf7
2018-04-18 19:53:32 -07:00
Siddharth Agarwal
cc57855b90 mercurial: use FileContents for file contents
Summary:
This is not only the newer, more specific type -- it also makes a couple
of upcoming diffs more straightforward.

Reviewed By: StanislavGlebik

Differential Revision: D7622906

fbshipit-source-id: 4e453b827512c538f4f9777ae4d24627f3b124cf
2018-04-17 11:37:29 -07:00