Commit Graph

1144 Commits

Author SHA1 Message Date
Lukas Piatkowski
81e872828a stats: replace usage of tokio_core with tokio
Reviewed By: farnz

Differential Revision: D8862007

fbshipit-source-id: 3e296adf2efdf77a853a5d8817dc3c8a40a7cafe
2018-07-17 04:54:58 -07:00
Lukas Piatkowski
1d7b5ec88f mercurial-bundles tests: replace tokio_core with tokio::runtime
Summary: this changes only the tests, there were to tokio_core in non-test code

Reviewed By: StanislavGlebik

Differential Revision: D8861362

fbshipit-source-id: 059e32789516009a2a747016be2fa53ef96a6f7f
2018-07-17 04:54:58 -07:00
Lukas Piatkowski
62c77dec99 hgcli: replace tokio_core usage with tokio::runtime
Reviewed By: StanislavGlebik

Differential Revision: D8861231

fbshipit-source-id: 434f187ecc510f2e20c0d1b398eb3eb42764be70
2018-07-17 04:54:58 -07:00
Lukas Piatkowski
24dc56c8a3 manifoldblob: remove usages of tokio_core
Summary: tokio::runtime can handle multithreading, no need for all this bolierplate

Reviewed By: StanislavGlebik

Differential Revision: D8861170

fbshipit-source-id: 2c489068a55f8cba1854f8a748df1e6efe8b47b7
2018-07-17 04:54:58 -07:00
Lukas Piatkowski
69d791a81f server: split server binary crate into 4 separate crates
Summary: Iterating over the code on server is a bit painful and it has grown a lot, splitting it should speed up future refactories and make it more maintainable

Reviewed By: jsgf, StanislavGlebik

Differential Revision: D8859811

fbshipit-source-id: 7c56f9f835f45eca322955cb3b9eadd87fbb30a1
2018-07-17 04:54:58 -07:00
Zeyi Fan
07f201f93b implement TLS & HTTP2 support
Summary:
This commit added three options to specify locations to SSL certificates so the API server will accept encrypted traffic.

Currently this only works in HTTP/1.1 due to a bug in HTTP/2 parsing in actix-web. Once they fixed the bug upstream we will be able to serve HTTP/2 traffic as well.

Reviewed By: jsgf

Differential Revision: D8703861

fbshipit-source-id: 0d4e68276013a8aeb6ee006e5175b8caeba767cb
2018-07-16 17:07:46 -07:00
Zeyi Fan
7aed398af3 move SslConfig & build_tls_acceptor to secure_utils
Summary: Moving to the crate allows apiserver to reuse the function.

Reviewed By: jsgf

Differential Revision: D8843178

fbshipit-source-id: 9d110c7f2683ff58654187222e7820240bfda98e
2018-07-16 17:07:46 -07:00
Zeyi Fan
b9ca1809b2 add scuba to apiserver.tw & filter out /status request from scuba
Summary:
This commit:

* add `--with-scuba` to apiserver.tw so we collect scuba metrics in tw deployed jobs.
* disable scuba for `/status` as it adds too much noise in apiserver's scuba table.

Reviewed By: kulshrax

Differential Revision: D8848271

fbshipit-source-id: 2f9638bd7ccb44f8ae02172097ccc38b3818b0e4
2018-07-16 10:08:53 -07:00
Lukas Piatkowski
a274c622b8 blobimport: replace usage of tokio_core with tokio::run
Summary:
There seems to be a deadlock in the internals of the now outdated tokio_core.
After applying the modern tokio::run the deadlock is not being triggered.

Reviewed By: farnz

Differential Revision: D8783183

fbshipit-source-id: 47a7d1d8e2756ea4d40812d0b8a6c850d7f7e9f8
2018-07-16 07:36:54 -07:00
Rain ⁣
c463be6e11 blobstore: use inlinable string for prefix
Summary:
This blobstore gets cloned a lot, and an inline string is much cheaper
to clone.

Reviewed By: StanislavGlebik

Differential Revision: D8843939

fbshipit-source-id: 879c2e43b27ec7a410ef64eb65d3ac78d21fa1db
2018-07-16 00:21:34 -07:00
Rain ⁣
80db1ae90c add inlinable_string and chashmap
Summary:
Add optionally and necessarily inlinable strings, and another
concurrent hashmap impl.

The asyncmemo changes are because of futures 0.1.23, which fixed an [unsoundness bug](15dbc97f30).

Reviewed By: kulshrax

Differential Revision: D8840579

fbshipit-source-id: 77d3e724310b727aeacc0c86a0353ea01e8fc679
2018-07-13 14:37:46 -07:00
Pavel Aslanov
6a77477d4d added BonsaiHgMapping to BlobRepo
Summary: Makes `BonsaiHgMaping` accessible from `BlobRepo` object

Reviewed By: StanislavGlebik

Differential Revision: D8821709

fbshipit-source-id: 03070ac4f9f390b76740c5d493f5a86b14024158
2018-07-13 08:03:13 -07:00
Matthew Dippel
6cdd85d661 Removed RepoGenCache structure, and repoinfo crate.
Summary:
Deleted the RepoGenCache structure, associated file, and public exports.
Also deleted the containing repoinfo crate, as nothing else was using it now.

Deleted some existing references to it in experimental code which weren't caught in the test plan but were blocking this from landing.

Reviewed By: StanislavGlebik

Differential Revision: D8787103

fbshipit-source-id: 0b90c758ea8175cb0f3ec74c371592b9ca5b192e
2018-07-12 14:22:17 -07:00
Matthew Dippel
0d76a20f31 Removed final usage of RepoGenCache from RangeNodeStream
Summary: Removed RepoGenCache from RangeNodeStream structure, methods, and tests. Last step is to delete RepoGenCache structure.

Reviewed By: StanislavGlebik

Differential Revision: D8786943

fbshipit-source-id: 70fa5b82da548d11126a289eb2cb11aaace18463
2018-07-12 10:52:02 -07:00
Matthew Dippel
95f8c4c6e3 Removed RepoGenCache from assert_node_sequence test helper parameters
Summary: Completely removed references to RepoGenCache in the test helper function assert_node_sequence, and updated all usages of it in the revset tests.

Reviewed By: StanislavGlebik

Differential Revision: D8786191

fbshipit-source-id: f1504e7a95e5555c86a2f02cb98ce1f28e374eab
2018-07-12 10:52:02 -07:00
Matthew Dippel
0cc6839f28 Removed usage of RepoGenCache in revset test helper
Summary: Replaced usage of a RepoGenCache in a revset test helper with repo.get_generation() calls. The RepoGenCache parameter is still passed, but is now unused and marked as such. Next step is to remove it from all of the tests and as a parameter altogether.

Reviewed By: StanislavGlebik

Differential Revision: D8785579

fbshipit-source-id: 95ec575117a7dfea5d683a683d54d19de2835a3b
2018-07-12 10:52:01 -07:00
Matthew Dippel
42e809a2fc Fixed performance bug in AncestorNodeStream by changing how parents were fetched
Summary: Fixed a bug in which AncestorNodeStream was excessively slow if it were used in cache_warmup. Changing how a node hash was mapped to its parents made this a lot faster.

Reviewed By: farnz

Differential Revision: D8754990

fbshipit-source-id: bbd0cd97b516b7d3a5f9e048c7bdea0f24d6f7f3
2018-07-12 10:38:29 -07:00
Matthew Dippel
c6fd393842 Removed RepoGenCache from the public API of revset
Summary:
Removed all references to RepoGenCache from publically callable functions in the revset package. This involved:
- Modifying blobrepo so that its get_generation_number method returned a Generation wrapper instead of a raw usize, to allow it to be used in a cleaner manner in the revset code.
- Simultaneously changing the constructors of all the structures in revset. This seems like a big change, but many of them call each other, passing a RepoGenCache object down the line, so eliminating them all at once made for the cleanest update.
- Modifying helper functions in the revset structures which would create streams of nodes by taking ownership of a RepoGenCache object within a closure. Instead they now take ownership of a clone of the repo. This strategy was already done earlier in the same helper functions, so I am assuming the cost of cloning a repo into a closure is small.
- Modifying the only external usage of revset within the mononoke server code.

This is part of a several step process to completely remove RepoGenCache from the code base. The next steps should be:
- Remove all references to RepoGenCache in the testing macros for revset.
- Delete RepoGenCache and clean up any dangling references to it.

Reviewed By: StanislavGlebik

Differential Revision: D8743560

fbshipit-source-id: 125f851075d836d40224d339e1daee912a39f7e4
2018-07-12 09:24:32 -07:00
Simon Farnsworth
4933b3f8e2 Introduce DummyLease, to nop out Blobstore leases for testing
Summary:
The CachingBlobstore can use a LeaseOps implementation to prevent
multiple parallel writes to the same backing store. Provide a dummy implementation
that does not block parallel writes, for performance testing and to use where
there's no way to implement a useful lease operation using a given cache.

Reviewed By: StanislavGlebik

Differential Revision: D8809077

fbshipit-source-id: 54cf67882a42b43dde1c6ac68611fb28144509bc
2018-07-12 09:07:00 -07:00
Pavel Aslanov
0733c5340f id for unique (repo_id, hg_cs_id) in mysql schema
Summary:
- [moononoke][bonsai_hg_mapping] id for unique `(repo_id, hg_cs_id)` in  mysql schema
- NOTE: sqlite does not understand this syntax so it is left unchanged

Reviewed By: StanislavGlebik

Differential Revision: D8820198

fbshipit-source-id: f0e2478900dc68d619aef0ed16602b76371c85f8
2018-07-12 07:21:17 -07:00
Lukas Piatkowski
0e3eba1955 rust: update Facebook to use hyper 0.12.x and openssl 0.10.x
Summary:
Those are the changes made to crates in tp2: P59806629

Several changes that were hard to split are here:
- update manifold client to use hyper 0.12.x
- update scribe client to use hyper 0.12.x
- update mononoke manifoldblob to use the updated manifold client
- update mononoke hgcli to use openssl 0.10.x and tokio-openssl
- update mononoke server to use openssl 0.10.x and tokio-openssl
- remove sendwrapper

Reviewed By: jsgf

Differential Revision: D8806931

fbshipit-source-id: 65412d483f77d8c0a0d5692c41c6516bb8f86046
2018-07-12 07:09:22 -07:00
Rain ⁣
5a29953c59 cmds: move creating blobrepo to cmdlib
Summary: This will be useful for the bonsai verification tool.

Reviewed By: StanislavGlebik

Differential Revision: D8792562

fbshipit-source-id: f409d0fa042528b04462a1539fd3c2a8064a4f6e
2018-07-11 18:36:50 -07:00
Rain ⁣
3f476a9b16 cmds: move getting the repo id to cmdlib
Summary: This code can easily be shared.

Reviewed By: StanislavGlebik

Differential Revision: D8777307

fbshipit-source-id: f11314f6a63bb191dc38d07cec181a4b05b158d9
2018-07-11 18:36:50 -07:00
Matthew Dippel
d2cc91b79f Initial package for reachabilityindex
Summary:
Creating the package for reachabilityindex, where methods that support efficient reachability queries on BlobRepo objects will live.

Includes the initial definition of the trait ReachabilityIndex which methods supporting reachability queries on BlobRepo objects will implement.

Reviewed By: StanislavGlebik

Differential Revision: D8792977

fbshipit-source-id: 293a810518e0a1fa260ccbc38902484e56ef2038
2018-07-11 09:55:43 -07:00
Pavel Aslanov
fcb58cf972 sql mapping between {Bonsai|Hg} changesets
Summary:
We need to store relation between Hg changesets and Bonsai changesets.
- `BonsaiHgMapping` is exactly this mapping which establishes injective relation between `{Hg|Bonsai}Changeset`

Reviewed By: StanislavGlebik

Differential Revision: D8801254

fbshipit-source-id: c7df14172e6c2d67c039a24e1bb821e6d92860af
2018-07-11 09:07:13 -07:00
Pavel Aslanov
2d3f584e65 fix integration tests if proxy is set
Summary: - integration test of were failing on my devserver as `{http|https}_proxy` environment variables

Reviewed By: StanislavGlebik

Differential Revision: D8804505

fbshipit-source-id: fafea55ed80897b24ad816bc26b241e64cd8b913
2018-07-11 08:52:10 -07:00
Simon Farnsworth
2f35a8dd24 Add stats to memcache operations to distinguish the causes of each success/failure
Summary:
The memcache implementations of CacheOps and LeaseOps are fairly complex, with lots of edge cases.

Add some stats, so we can see which edge cases are being hit, and optimize accordingly

Reviewed By: lukaspiatkowski

Differential Revision: D8752599

fbshipit-source-id: fe1376956beceff702c2b6a020598556016033b5
2018-07-10 12:52:46 -07:00
Simon Farnsworth
d40a725f7b Refactor MemcacheBlobstore to split into common and memcache parts
Summary:
When we add cachelib bindings to Rust, we're going to want to implement a
cachelib blobstore that's more or less the same as the memcache version, but
backed by a cachelib pool instead of a memcache instance.

Split this code up so that we don't duplicate functionality

Reviewed By: StanislavGlebik

Differential Revision: D8523713

fbshipit-source-id: 882298abab8c208103f6d8c74fee60a768c877f6
2018-07-10 10:10:18 -07:00
Pulkit Goyal
fc880f518b Add Cargo.toml files to crates. (#7)
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.

Following are the reasons why the crates don't build:

  * failure_ext and netstring crates which are internal
  * error related to tokio_io, there might be an patched version of tokio_io internally
  * actix-web depends on httparse which uses nightly features

All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7

Differential Revision: D8778746

Pulled By: jsgf

fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
2018-07-09 19:52:27 -07:00
Lukas Piatkowski
4ed747d0da hgcli: measure the time it takes to serve all requests
Reviewed By: farnz

Differential Revision: D8750668

fbshipit-source-id: 443c6a8c426ff01112f878afc6fc809c3cc0ec8f
2018-07-09 15:39:09 -07:00
Lukas Piatkowski
b7cfc6ba3e hgcli: move tokio_core usage to the main.rs file leaving Futures behind
Summary: by using futures rather than reactor::run calls we are able to add f.e. tracing and timeouts easily - in the next diff

Reviewed By: farnz

Differential Revision: D8750662

fbshipit-source-id: d928d903cf5631a275f5b319d075db9f1611d76b
2018-07-09 15:39:09 -07:00
Lukas Piatkowski
221d51465b hgcli/server: move session_uuid creation to hgcli and log more information
Summary:
hgcli will start logging stuff as well and it will use the same session_uuid as the server.
This also includes logging the user and source hostname.

Reviewed By: farnz

Differential Revision: D8750663

fbshipit-source-id: 7ebc8b6c10b7560d985fd23e9e3f2645f3bd0a1c
2018-07-09 15:39:09 -07:00
Lukas Piatkowski
567051928c sshrelay: move Stdio and SenderBytesWrite from mononoke server to sshrelay
Summary: those structures are sshrelay specific, move them there so it's easier to share them in future diffs

Reviewed By: farnz

Differential Revision: D8750666

fbshipit-source-id: b58596e63787d221a3970d5f1648e11d81949925
2018-07-09 15:39:08 -07:00
Christopher Kuklewicz
6cc2cd49f8 Unifying sql code for Mononoke/Diesel
Summary:
Unify all uses of Sqlite and of Mysql

This superceded D8712926

Reviewed By: farnz

Differential Revision: D8732579

fbshipit-source-id: a02cd04055a915e5f97b540d6d98e2ff2d707875
2018-07-08 09:06:22 -07:00
Lukas Piatkowski
31bd6f65b1 scuba_ext: move Mononoke specific scuba builder initialization to scuba_ext
Summary: This is a preparation to using Preamble more heavily

Reviewed By: StanislavGlebik

Differential Revision: D8750665

fbshipit-source-id: 44d4bcedbe95fe05679faeedf4479ebad4d9359c
2018-07-06 13:52:08 -07:00
Zeyi Fan
cc02bf8371 bridge stdlog to slog
Summary: This commit bridges messages logged via `log` crate to be forwarded to `slog` using `slog-stdlog` crate. This allows us to see debug messages printed from third party crates that are using `log` instead of `slog`.

Reviewed By: StanislavGlebik

Differential Revision: D8703996

fbshipit-source-id: f847b76a52c262eded4f576a5d9065574ff7e4dd
2018-07-05 11:07:08 -07:00
Zeyi Fan
c53f98dece add scuba for mononoke apiserver
Summary: This commit adds a scuba middleware for actix. It will automatically collects metrics from the request and the response.

Reviewed By: StanislavGlebik

Differential Revision: D8706387

fbshipit-source-id: 4dddad0d977d42718ce53e1e4fd7b771a1eda7df
2018-07-05 10:07:38 -07:00
Zeyi Fan
dfd3b3c9de split middlewares
Summary: This commit splits middleware.rs into mod middleware, and move logic of computing responsive to a separate trait so it can be shared.

Reviewed By: StanislavGlebik

Differential Revision: D8715657

fbshipit-source-id: cf06249d41b9fbe1a4a7500ed5e8024bbf376d3f
2018-07-05 10:07:38 -07:00
Tim Fox
7f55aad570 Use the Bookmark struct to represent bookmarks instead of string
Summary: This diff refactors the hook code to use the Bookmark struct instead of strings

Reviewed By: farnz

Differential Revision: D8724197

fbshipit-source-id: 920aa1266ca94b2bd8683a995e4fd781159bd5b1
2018-07-05 03:22:52 -07:00
Tim Fox
ccf594d858 Update repoconfig and hookloader to support per file hooks
Summary: This diff updates rep config to support per file hooks. We also extend the hookloader to support per file hooks.

Reviewed By: lukaspiatkowski

Differential Revision: D8713464

fbshipit-source-id: 65cf20957506adc620bf06d84707669d125366ed
2018-07-05 03:22:52 -07:00
Tim Fox
c6009ab467 Implement Lua per file hooks
Summary:
This diff extends lua hooks to support per file hooks.

We also now call the hook function via a Lua wrapper function which allows us to have a better API in the hook as we can construct lua table to pass into the hook and we can also return multiple return values from the hook, which allows us to support rich hook failure reasons properly. Both of these are hard to do when calling hooks directly using hlua.

Reviewed By: StanislavGlebik

Differential Revision: D8711280

fbshipit-source-id: b91f9e47a1f8eab302775a5bbfd61590a8635282
2018-07-05 03:22:52 -07:00
Matthew Dippel
bff17009e1 Added UniqueHeap to AncestorsNodeStream for faster max generation number extraction.
Summary: Replaced the old behavior of getting the 'next_generation' via a max call on a current set of generation numbers, with a heap which stores the currently viewed set. If the currently viewed set gets large in practice, then this should have a significant speed up.

Reviewed By: farnz

Differential Revision: D8725928

fbshipit-source-id: e03f4573bb14e7170d23dbbb0fd62118761bec62
2018-07-04 13:51:23 -07:00
Matthew Dippel
d5541e465d Move Generation struct to mononoke-types
Summary: Moved the Generation struct, which is a u64 wrapper, from repogen to mononoke-types, and updated the according usages. This should make it easier to phase out RepoGenCache.

Reviewed By: farnz

Differential Revision: D8725538

fbshipit-source-id: cfd39be03bae56d2288053b7b5e212e6d547c833
2018-07-04 13:51:23 -07:00
Lukas Piatkowski
38b66202cb server: print the session_uuid on the client side
Summary: Session UUID will help identify the issues on Mononoke side whenever the client encounters problems

Reviewed By: StanislavGlebik

Differential Revision: D8732396

fbshipit-source-id: 35d04b0d56be0cfc2c608f08287a2b1d236a96e3
2018-07-04 11:49:54 -07:00
Stanislau Hlebik
bac5f5dcca mononoke apiserver: fix integration test
Summary:
We've recently added a flag in a config repo to enable or disable the repo.
Let's use it in apiserver as well.

P. S.
As I think about it now, it maybe worth filtering enabled/disabled flag in
RepoConfig

Reviewed By: lukaspiatkowski

Differential Revision: D8732450

fbshipit-source-id: 1cd59f5a91dec28efd1e2c122f2f1ea58ded6ac1
2018-07-04 08:53:34 -07:00
Tim Fox
5adf5c109b Implement per file hooks
Summary:
This diff implements hooks which act upon individual files. The results of the hooks are cached using Asyncmemo.

The cache is currently keyed on (changeset_id, hook_name, file path) but this will change to file content hash once we move to Bonsai changesets.

Reviewed By: StanislavGlebik

Differential Revision: D8707726

fbshipit-source-id: ceaf94abd09e1dd7f6b2d8f9c87a9a221439a252
2018-07-04 07:07:02 -07:00
Tim Fox
5c3404f3a1 Use lua.into_get method not ffi
Summary: This diff removes the unnecessary ffi in lua_hook and instead uses lua.into_get to get the hook function without borrowing.

Reviewed By: farnz

Differential Revision: D8525921

fbshipit-source-id: b1d43c919c2aac113ee58fde28043639e2bd2362
2018-07-04 07:07:02 -07:00
Tim Fox
8809de14f0 Implement hook loader
Summary: This diff introduces a hook loader, which has the responsibility of loading Lua hooks into the hook manager given a repo config.

Reviewed By: StanislavGlebik

Differential Revision: D8625177

fbshipit-source-id: 51948d9a70c02b859ab3a9b45c8487a9da6e7f6a
2018-07-04 07:07:02 -07:00
Tim Fox
f0bbb31967 Implement running of hooks by bookmark
Summary: This diff implements tracking of hooks by bookmark in the hook manager and functionality to run hooks per bookmark.

Reviewed By: StanislavGlebik

Differential Revision: D8525598

fbshipit-source-id: 7987d1f8d90a77667f120f4940f12aa3cb5aa86e
2018-07-04 07:07:02 -07:00
Tim Fox
02ce67d40e Refactor repoconfig to support loading of hooks, and directory per repo
Summary:
This diff refactors the server config repository to support storing and loading of hooks. In the new structure each repo lives in its own directory and the config file for the server is called "server.toml".
Hooks can be referenced by relative or absolute paths allowing either local or common hooks to be loaded.

Reviewed By: StanislavGlebik

Differential Revision: D8625178

fbshipit-source-id: 62c8c515a0fbbf7a38cfc68317300d8f42eb4d7a
2018-07-04 07:07:02 -07:00