Summary: use common funcions to parse --myrouter-port, this is a simple clean up
Reviewed By: StanislavGlebik
Differential Revision: D14084003
fbshipit-source-id: 63d6c8301e977faead62cb1c705bac372d56594e
Summary: New commits should be logged to scribe, these will be used to trigger the update for the hg clone streamfile.
Reviewed By: lukaspiatkowski
Differential Revision: D14022599
fbshipit-source-id: a8a68f12a8dc1e65663d1ccf1a5eafa54ca2daf0
Summary: This is adds a metaconfig option to preserve push/pushrebase bundles in the blobstore.
Reviewed By: StanislavGlebik
Differential Revision: D14020299
fbshipit-source-id: 94304d69e0ac5d81232f058c6d94eec61eb0020a
Summary:
Repo name is used only be verify_integrity hook and even there the name that Mononoke provides is incorrect. Instead of Mononoke's `repo-RepositoryId(1234)` name the hook is interested in Mercurial's `fbsource` name.
HookConfig is a perfect way to pass such an arbitrary parameter so use it.
Reviewed By: StanislavGlebik
Differential Revision: D13964486
fbshipit-source-id: 94090e409d5206828364202ae62a37abc16e4a27
Summary: Replace Arc<BlobRepo> with direct usage of BlobRepo because BlobRepo is cheaply clonable.
Reviewed By: StanislavGlebik
Differential Revision: D13944990
fbshipit-source-id: b6e049edb04bd19343ced1f15857f6602ea9d9c2
Summary:
`blobrepo_factory` is a crate that knows how to create blobrepo given
a configuration i.e. it creates blobstores, filenodes, changesets etc and
initializes blobrepo with them.
`post_commit` is a small part of blobrepo which can also be extracted from
blobrepo crate.
There are a few upsides with this approach
1) Less dependencies on Blobrepo, meaning we have to rebuild it fewer times
2) BlobRepo compilation is faster
Reviewed By: jsgf
Differential Revision: D13896334
fbshipit-source-id: 1f5701341f01fcefff4e5f9430ddc914b9496064
Summary:
Let's split reachability index crate. The main goal is to reduce compilation
time. Now crates like revsets will only depend on traits definition but not on
the actual implementation (skiplist of genbfs).
Reviewed By: lukaspiatkowski
Differential Revision: D13878403
fbshipit-source-id: 022eca50ac4bc7416e9fe5f3104f0a9a65195b26
Summary:
The main reason for doing it is to remove dependency on `BlobRepo` from hooks. Most of the `hooks` crate code needs from `BlobRepo` just a HgBlobChangeset type, which was moved to a separate crate in one of the previous diffs. There is just a small piece of code that depends on blobrepo, and it was moved in the separate crate.
Because of that changing anything in BlobRepo won't trigger rebuilding of most of the hooks crate.
Reviewed By: lukaspiatkowski
Differential Revision: D13878208
fbshipit-source-id: d74336e959282c176258c653d4c408854e1f1849
Summary:
Currently if a crate depends even on a single type from metaconfig then in
order to compile this trait buck first compiles metaconfig crate with all the
logic of parsing the configs.
This diff split metaconfig into two crates. The first one just holds the types for
"external consumption" by other crates. The second holds the parsing logic.
That makes builds faster
Reviewed By: jsgf, lukaspiatkowski
Differential Revision: D13877592
fbshipit-source-id: f353fb2d1737845bf1fa0de515ff8ef131020063
Summary:
The main reason to do it is to remove dependency from cmdlib to repo_client.
repo_client depends on a lot of other crates like bundle2-resolver, hooks etc.
And it means that in order to compile mononoke_admin we need to compile these
crates too. By moving open_blobrepo into blobrepo crate we are removing
unnecessary dependencies.
Also let's remove unused blobrepo type
Reviewed By: aslpavel
Differential Revision: D13848878
fbshipit-source-id: cd3d04354649cdb5b2947f08762051318725c781
Summary: This hint is passed to many places, so it reduces the code.
Reviewed By: StanislavGlebik
Differential Revision: D13802159
fbshipit-source-id: 891eef00c236b2241571e24c50dc82b9862872cc
Summary:
Previously we had a timeout per session i.e. multiple wireproto command will
share the same timeout. It had a few disadvantages:
1) The main disadvantage was that if connection had timed out we didn't log
stats such as number of files, response size etc and we didn't log parameters
to scribe. The latter is even a bigger problem, because we usually want to
replay requests that were slow and timed out and not the requests that finished
quickly.
2) The less important disadvantage is that we have clients that do small
request from the server and then keep the connection open for a long time.
Eventually we kill the connection and log it as an error. With this change
the connection will be open until client closes it. That might potentially be
a problem, and if that's the case then we can reintroduce perconnection
timeout.
Initially I was planning to use tokio::util::timer to implement all the
timeouts, but it has different behaviour for stream - it only allows to set
per-item timeout, while we want timeout for the whole stream.
(https://docs.rs/tokio/0.1/tokio/timer/struct.Timeout.html#futures-and-streams)
To overcome it I implemented simple combinator StreamWithTimeout which does
exactly what I want.
Reviewed By: HarveyHunt
Differential Revision: D13731966
fbshipit-source-id: 211240267c7568cedd18af08155d94bf9246ecc3
Summary:
PerfCounters is a small wrapper around a concurrent hashmap
that can be used to store performance metrics in. Include it in CoreContext
so that it can be used throughout the codebase.
Reviewed By: aslpavel
Differential Revision: D13528647
fbshipit-source-id: 7c3f26ab8c0c7ba5ee619e85a069af7e7721037f
Summary: MyRouter and cache support will come in next diffs
Reviewed By: StanislavGlebik
Differential Revision: D13465146
fbshipit-source-id: 0ede0e875d4a49794ff71173bd0d1563abb3ff08
Summary: Format files effected by next commit in a stack
Reviewed By: StanislavGlebik
Differential Revision: D13650639
fbshipit-source-id: d4e37acd2bcd29b291968a529543c202f6944e1a
Summary:
use the correct skip index
sorry for some rustfmt.
Reviewed By: StanislavGlebik
Differential Revision: D13636059
fbshipit-source-id: 2815d82b63b86bda053f5a3a9a1b8a3b72abbf82
Summary: There's nothing Mercurial-specific about identifying a repo. This also outright removes some dependencies on mercurial-types.
Reviewed By: StanislavGlebik
Differential Revision: D13512616
fbshipit-source-id: 4496a93a8d4e56cd6ca319dfd8effc71e694ff3e
Summary:
As discussed RepoClient should own phases_hit. Initialize it correspondingly.
Some rustfmt is unrelated.
Reviewed By: lukaspiatkowski
Differential Revision: D13488572
fbshipit-source-id: 3d2c0169c06220bcd4b3f8845d6db7f357052e0a
Summary:
As we no longer use a config repo, rename the argument
to --config_path.
Reviewed By: rlangst, StanislavGlebik
Differential Revision: D13415205
fbshipit-source-id: 8c7d335373c0ba33131d4476ee72fcd914097385
Summary: passing CoreContext around instead of Logger gives us more flexibility
Reviewed By: StanislavGlebik
Differential Revision: D13340855
fbshipit-source-id: 43bc4def1532ce675a4ec70903fd6c04332c967e
Summary: Restructure the configs so that we can specify more than one blobstore
Reviewed By: lukaspiatkowski
Differential Revision: D13234286
fbshipit-source-id: a98ede17921ed6148add570288ac23636b086398
Summary:
Config repo proved to be tricky to understand and hard to use. Let's just use
toml files.
Reviewed By: farnz
Differential Revision: D13179926
fbshipit-source-id: 3a44ee08c37284cc4c189c74b5c369ce82651cc6
Summary: Previous if repository was disabled or not found, `hgcli` did not return any error.
Reviewed By: lukaspiatkowski
Differential Revision: D12975351
fbshipit-source-id: 1390b7ddaa4d80d10bcb54208b437cc9c41ec43a
Summary:
As part of our read path rollout, we want to block user error from
creating new commits that will confuse blobimport. Make it possible to
configure a read-only repo
Reviewed By: StanislavGlebik
Differential Revision: D12945024
fbshipit-source-id: 4265bf57f8adac7965117b710b8285bac483b8ee
Summary: This will help traffic replay to distinguish between different repos
Reviewed By: farnz
Differential Revision: D12922287
fbshipit-source-id: 6eed2a0eebceca0636512baa3ee885f5d9c95ccb
Summary:
This diffs add a signal handler for SIGTERM signal. When it's received then a
terminate process flag is set to true. When this flag is set then no new client
connections will be accepted, and server waits until open connections finish.
The connections can take a long time, so ideally an external process should
sent SIGKILL after a timeout.
Note that this change also makes thrift server thread detached. The reason is
because making it gracefully stop is non-trivial, so for making it detached
should be fine
Reviewed By: farnz
Differential Revision: D12857453
fbshipit-source-id: 6a8f890ff529d74c21fc0c62e16951dd95a3f101
Summary:
Let's add an option to validate the getfiles content that we return to users on
some percent of requests.
It'll increase the latency so let's not enabling that by default
Reviewed By: HarveyHunt
Differential Revision: D10558180
fbshipit-source-id: 2d7ec4dfe7b37b7b5541013278006278d1df68fa
Summary:
Using multiple Runtimes might be a cause of problems in future and even if it isn't it will be a cause for investigating whether it is a problem or not.
The issue I have in mind is that if someone runs a future on one runtime that calls `tokio::spawn` on it (f.e. schedule a job that works for ever) but then uses a different runtime to drive another future to complection one might not suspect that the previous spawn is already lost with the previous Runtime.
Reviewed By: farnz
Differential Revision: D10446122
fbshipit-source-id: 4bfd2a04487a70355a26f821e6348f5223901c0d
Summary:
getfiles implementation for lfs
The implementation is the following:
- get file size from file envelope (retrieve from manifold by HgNodeId)
- if file size > threshold from lfs config
- fetch file to memory, get sha256 of the file, will be fixed later, as this approach consumes a lot of memory, but we don't have any mapping from sha256 - blake2 [T35239107](https://our.intern.facebook.com/intern/tasks/?t=35239107)
- generate lfs metadata file according to [LfsPlan](https://www.mercurial-scm.org/wiki/LfsPlan)
- set metakeyflag (REVID_STORED_EXT) in the file header
- if file size < threshold, process usual way
Reviewed By: StanislavGlebik
Differential Revision: D10335988
fbshipit-source-id: 6a1ba671bae46159bcc16613f99a0e21cf3b5e3a
Summary:
As per the comments added - MyRouter setup is such that it starts inside a tupperware container together with the binary that will be using it. This means that by the time the binary wants to use the MyRouter connection the MyRouter instance might not be ready yet. In order to mitigate this effect the myrouter::Builder will attempt to make a "Select 1" query and retry it with a backoff for a max of 2 min or until the connection is actually established.
Unfortunately the `queries!` macro had to be moved inside the `macro` module in order to make it usable from inside `myrouter` module, see this: https://stackoverflow.com/questions/31103213/import-macro-from-parent-module
Reviewed By: farnz
Differential Revision: D10270464
fbshipit-source-id: 9cf6ad936a0cabd72967fb96796d4af3bab25822
Summary:
The idea for rollout is to:
- first make sure that Mononoke doesn't crash when a --myrouter-port is provided
- then tupperware configs will be modified to include myrouter as a collocated proces on every host and the port of that myrouter instance will be provided via command line
- lastly land the change that actually talks to myrouter
Reviewed By: StanislavGlebik
Differential Revision: D10258251
fbshipit-source-id: ea9d461b401d41ef624304084014c2227968d33f
Summary:
We now have a way for a MySQL database to tell us how to send
streaming clones to the client. Hook it all up, so that (with any luck), once
we have data in MySQL and the blobstore, we'll see working streaming clones.
Reviewed By: StanislavGlebik
Differential Revision: D10130774
fbshipit-source-id: b22ffb642d0a54b09545889779f79e7a0f81acd7
Summary:
Previously cachelib cmdline args were added only to cmd line binaries, but not
to Mononoke this diff fixes it.
Reviewed By: farnz
Differential Revision: D10083899
fbshipit-source-id: 8febba96561c5ab9a61f60fafc7a7e56985dc038
Summary:
The "path" in manifold blobrepo is used for logging, but it has been quite confusing with "fbsource" and "fbsource-pushrebase" to be logged in an identical way - both are "fbsource", because of the "path" config. Lets not use the "path" for logging, instead use the "reponame" from metaconfig repo.
In case we ever want to have two repos that are named the same (please don't) or have logging under a different name than "reponame" from config then we can add a proper optional "name" parameter, but for now we don't require this confusing feature.
Reviewed By: StanislavGlebik
Differential Revision: D9769514
fbshipit-source-id: 89f2291df90a7396749e127d8985dc12e61f4af4
