Summary: Restructure the configs so that we can specify more than one blobstore
Reviewed By: lukaspiatkowski
Differential Revision: D13234286
fbshipit-source-id: a98ede17921ed6148add570288ac23636b086398
Summary:
Refactored UnionNodeStream to use changesetid
Modified all other classes using it
Modified tests.
Depends on: D13275956
Reviewed By: StanislavGlebik
Differential Revision: D13326684
fbshipit-source-id: fd34f52739cf3e2876aa7a30c5060b3b3410f413
Summary:
Refactored intersect nodestream to use changesetid
Modified all other classes using it
Modified tests (some still need to be modified, see `TODO Hrenic`)
Reviewed By: StanislavGlebik
Differential Revision: D13275956
fbshipit-source-id: 90d3c191e161be8a9ce1841de0e04ce60438764b
Summary: Set the proper context for sessions.
Reviewed By: liubov-dmitrieva
Differential Revision: D13258641
fbshipit-source-id: edd18d4abc8f5475e0d2ac8395dfc877b2dd5958
Summary:
Previously we manually specified blobstore type and all the necessary
parameters. That was error-prone but worked because we had only one blobstore.
Since we are going to add secondary blobstores soon configuring binaries like
blobimport will be harder because we'll need to specify parameters for all
blobstores. Let's make it so that
blobimport, mononoke admin and other binaries read the configuration the same
way as Mononoke does it i.e. via toml files.
Reviewed By: lukaspiatkowski
Differential Revision: D13183244
fbshipit-source-id: 99caa6348133acec11dd04ae44e1f9f0a8ebb197
Summary:
Previously metaconfig depended on BlobRepo, and so ManifoldArgs had to be
definided in BlobRepo. That was an weird dependency, but a necessary one
because of Mononoke config repo. In the previous diffs we got rid of the
Mononoke config repo, so now we can reverse the dependencies.
Reviewed By: lukaspiatkowski
Differential Revision: D13180160
fbshipit-source-id: efe713ce3b160c98d56fc13559c57a920146841f
Summary:
Config repo proved to be tricky to understand and hard to use. Let's just use
toml files.
Reviewed By: farnz
Differential Revision: D13179926
fbshipit-source-id: 3a44ee08c37284cc4c189c74b5c369ce82651cc6
Summary:
Add an integration test that talks to the API server from remotefilelog.
Since this involves running a local instance of Mononoke and the API server, I've added it to Mononoke's integration tests rather than Mercurial's integration tests in order to re-use the setup code from Mononoke's tests. Since the server is running locally, and would use a different SSL setup that the VIP that hg would normally access it through, the local API server is run without SSL enabled.
Reviewed By: Anastasiya-Zhyrkevich, farnz
Differential Revision: D13089196
fbshipit-source-id: 01f415d8ee7173f7f2ab3a234565fd79d618126e
Summary: test-init.t and test-lfs-to-mononoke.t were failing. This diff fixes them
Differential Revision: D13301143
fbshipit-source-id: 1f8060d4c6b641c555ba8a5cdcfe4cb14ac89d0a
Summary:
Now that Rust macros can be `use`d like normal symbols, `stats` can
simply import the `lazy_static!` macro without requiring its users to do it.
Reviewed By: Imxset21
Differential Revision: D13281897
fbshipit-source-id: a6780fbace07dd784308e642d4a384322a17c367
Summary:
We currently log information for wireproto commands, such as duration
and occasionally the args.
As a first step to increasing the amount of information we log, record the
number of known and unknown nodes for the known() wireproto command to the new
scuba column "extra_context".
Reviewed By: StanislavGlebik
Differential Revision: D13254857
fbshipit-source-id: 891d127c0a8efe741a6e7bfa35d3a56b9d79bf5c
Summary:
Previously `max_gen()` function did a linear scan through all the keys, and it
was linear. Let's use `UniqueHeap` datastructure to track maximum generation
number.
Reviewed By: lukaspiatkowski
Differential Revision: D13275471
fbshipit-source-id: 21b026c54d4bc08b26a96102d2b77c58a981930f
Summary:
Recursion can easily become too deep and overflow the stack. Let's use
`loop_fn` instead
Reviewed By: lukaspiatkowski
Differential Revision: D13169015
fbshipit-source-id: bf5cf151e83fd4bd785ff4b81a93858e7e2dcfde
Summary:
Let's add a command that builds and reads a skiplist indexes. This indexes will
be used by getbundle wireproto request to decrease the latency and cpu usage.
Note that we are saving only the longest "jump" from the skiplist. This is done
in order to save space.
Reviewed By: jsgf
Differential Revision: D13169018
fbshipit-source-id: 4d654284b0c0d8a579444816781419ba6ad86baa
Summary:
Previously if a commit didn't have a skiplist index then it'd be built lazily.
So if a getbundle requests a very old commit then skiplist index entries for
too many commits might be built and stored in memory.
Instead of lazily building the index let's add a constructor that accepts a
skiplist graph. That'd allow us to skiplist graph persistently and create it on
demand
Reviewed By: jsgf
Differential Revision: D13169019
fbshipit-source-id: 813bdd7e6cc90312d97275a7db5e266d97b545e0
Summary:
NodeFrontier is a hashset that stores commits and their generation numbers.
We'll use it to figure out what nodes client already has (`exclude` nodes).
Before this change we used `GroupedByGenenerationStream`, but it doesn't allow
us to skip some commits. We can skip commits with NodeFrontier though.
Reviewed By: lukaspiatkowski
Differential Revision: D13122521
fbshipit-source-id: 08eddb71e49b16b879f65bc9b8b177dc5dbcc034
Summary:
We'd like to save skiplist data in blobstore so that we won't have to recompute
it. Let's use thrift serialization for it
Reviewed By: lukaspiatkowski
Differential Revision: D13122386
fbshipit-source-id: f44cdcf38fa6a219df9217906a872e60c319e391
Summary:
Let's make it use the same ChangesetFethcer as getbundle already does. It will
be used in the next diffs
Reviewed By: lukaspiatkowski
Differential Revision: D13122344
fbshipit-source-id: 37eba612935a209098a245f4be0af3bc18c5787e
Summary:
Most of our revsets are already migrated, let's migrate skiplists as well since
we want to use them in getbundle requests.
Reviewed By: lukaspiatkowski
Differential Revision: D13083910
fbshipit-source-id: 4c3bc40ccff95c3231c76b9e920af5db31b80d01
Summary:
We've recently found that `known()` wireproto request gets much slower when we
send more traffic to Mononoke jobs. Other wireproto methods looked fine, cpu
and memory usage was fine as well.
Background: `known()` request takes a list of hg commit hashes and returns
which of them Mononoke knows about.
One thing that we've noticed is that `known()` handler sends db requests sequentially.
Experiments with sending `known()` requests with commit hashes that Mononoke
didn't know about confirmed that it's latency got higher the more parallel
requests we sent. We suspect this is because Mononoke has to send a requests to
db master, and we limit the number of master connections.
A thing that should help is batching the requests i.e. do not send many
requests asking if a single hg commit exists, but sending the same request for
many commits at once.
That change also required doing changes to the bonsai-mapping caching layer to
do batch cache requests.
Reviewed By: lukaspiatkowski
Differential Revision: D13194775
fbshipit-source-id: 47c035959c7ee12ab92e89e8e85b723cb72738ae
Summary:
Currently we read all bookmarks from primary replica a few times during `hg
pull`. First time when we do listkeys, second time when we get heads.
That might create a high load on primary replica.
However the delay between primary and secondary replicas is fairly small, and so it
*should* be fine to read bookmarks from secondary local replica as long as there is only
one replica per region (because if we have a few replicas per region, then
heads and listkeys response might be inconsistent).
Reviewed By: lukaspiatkowski
Differential Revision: D13039779
fbshipit-source-id: e1b8050f63a3a05dc6cf837e17a448c3b346b723
Summary:
Mononoke blobimports filenodes from Mercurial Host Machines to its blobstore.
It parses revlogs to retrieve flags used for parsing.
Revlogs Contains the flags for Marking Ext Stored FIles. This flag means that the file is not stored at the Mercurial Host machine, but somewhere remotely.
This diff provides the api to get the extstored flag from revlog.
Reviewed By: farnz
Differential Revision: D13081950
fbshipit-source-id: ba5bc04ad3659d4880960995d1cc46594d89e220
Summary:
According to [Git-LFS Plan](https://www.mercurial-scm.org/wiki/LfsPlan), `getfiles` instead of file content should return file in the [following format](https://www.mercurial-scm.org/wiki/LfsPlan#Metadata_format)
```
oid: sha256.SHA256HASH
size: size_int
```
Hg client requests files using sha1 hgfilenode hash. To calculate sha256 of the content, Mononoke is fetching the file from blobstore to memory, and calculate sha256.
It does not give any profit in time and memory consumptions, comparing to non-LFS transfer of Mononoke.
*Solution:*
To put a `key-value` to blobstore, after first request of the file. This means, that after hg client requested sha256 of the file for the first time, after calculation, put it to the blobstore.
Next request of the sha256 of the file content avoid recalcualtion of sha256 in Mononoke. It return sha256 saved in the blob.
Reviewed By: StanislavGlebik
Differential Revision: D13021826
fbshipit-source-id: 692e01e212e7d716bd822fa968e87abed5103aa7
Summary:
Add a helper function for hooks that can be used to match a string to a regex.
Regex matching can either be called from ctx.regex_match(s, r) or file.path_regex_match(r)
This will later be used in the no_bad_filenames hook, but will be helpful for other
hooks too.
Reviewed By: purplefox
Differential Revision: D12941544
fbshipit-source-id: f8f73a767da5ae77f486c925b6adc1333a198c3f
Summary:
The file of some revision is the initial file content with applied deltas
Delta is a vector of Fragments.
Fragment is a sequential change of the file (old part of the content -> new content)
This diff is representing the implementation of optimization of the process of getting a file content of some revision.
Reviewed By: lukaspiatkowski
Differential Revision: D12928138
fbshipit-source-id: fcc28e2d0e0acf83e17887092f6593e155431c1b
Summary:
Mononoke requires several references to the same blob in the blobstore.
Sha256 aliases are good example. [post](https://fb.facebook.com/groups/scm.mononoke/permalink/739273266435251/)
Short description of alias mechanism:
- we have `key: value` blob in blobstore.
- put a `key1: key` blob in blobstore to have 2-step access from `key1` to `value`.
All the keys in Mononoke are of the form `type_prefix.hash_name.hash`
I expanded MononokeId interface to have an access to the prefix `type_prefix.hash_name` for verification `key` content (see alias mechanism description).
Reviewed By: farnz
Differential Revision: D13084145
fbshipit-source-id: 5b8a4e80869481414a7356ccd7c9aab6e24a5138
Summary:
Prepare for gluster blobstore by making the test framework properly
async. It needs a full tokio environment (IO reactor, etc) for async operation, rather than
just `Future::wait`.
Reviewed By: StanislavGlebik
Differential Revision: D13055656
fbshipit-source-id: 6263ff706b3a374f0868f05989b6880f3c5a16ed
Summary: Add `with_smc` to allow making a connection using an SMC tier name.
Reviewed By: kulshrax
Differential Revision: D12989046
fbshipit-source-id: bdf9ef4729ac8f90a6a0ac16fb3ce9669d0bcf52
Summary:
This implements the Blobstore trait using an underlying Gluster volume (or
really any NFS server, but some aspects are tailored to Gluster).
Reviewed By: lukaspiatkowski
Differential Revision: D12989045
fbshipit-source-id: 76b4fe87ed1f392342c31be12115eee972221ca0
Summary:
Update test certificates, because they expired. Set next expiration date in 10
years.
```
stash@devvm3292
/data/users/stash/fbsource/fbcode/scm/mononoke/tests/integration
(eab3173|remote/master) $ openssl req -x509 -nodes -days 3650 -newkey rsa:2
048 -keyout testcert.key -out testcert.crt
Generating a 2048 bit RSA private key
.............+++
.....................................................................................................................+++
writing new private key to 'testcert.key'
-----
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
-----
Country Name (2 letter code) [XX]:uk
State or Province Name (full name) []:
Locality Name (eg, city) [Default City]:
Organization Name (eg, company) [Default Company Ltd]:
Organizational Unit Name (eg, section) []:
Common Name (eg, your name or your server's hostname) []:localhost
Email Address []:
```
Also update the warning in some tests
Reviewed By: farnz
Differential Revision: D13082351
fbshipit-source-id: 8636e711e82113e692ecd146154cbe4796b33a59
Summary:
New debug line started to show up. Hard to track where it comes from, but let's
add it to the test.
Reviewed By: farnz
Differential Revision: D13042463
fbshipit-source-id: 0901920d127099b4f27c242df4f6c3814d701a08
Summary: It causes test failures. REvert for now until they fixed
Reviewed By: farnz
Differential Revision: D13040073
fbshipit-source-id: fc05373c882baf42f7bd2a3a1c1173e8ba26a952
Summary:
Previously sql query that was sent to the server had `name like CONCAT('', %)`
for empty prefixes. That's inefficient, and since empty prefix requests are
common it's worth fixing them.
Reviewed By: lukaspiatkowski
Differential Revision: D13021876
fbshipit-source-id: 2fd9b2361e9be57cb15251e37f7988ee048468ec
