Summary: This allows us to run pushbackup and cloud sync commands for Read Only Mononoke repos.
Reviewed By: ikostia
Differential Revision: D13804545
fbshipit-source-id: 8026fc4668afc8bb5c2c0a9587ca024e3c6920da
Summary:
The test is ignored by default because it would probably be
flakey/slowThe test is ignored by default because it would probably be
flakey/slow.
Reviewed By: aslpavel
Differential Revision: D13795070
fbshipit-source-id: 8459d34126917e0fce3151ec2f4e68e639d2e7e6
Summary:
Next step is to change infinitepush / cloud sync to pass this pushvar to
Mononoke
Reviewed By: StanislavGlebik
Differential Revision: D13802402
fbshipit-source-id: 25e3d699bd934e3e015b9784040bd2dc4b43188d
Summary: This hint is passed to many places, so it reduces the code.
Reviewed By: StanislavGlebik
Differential Revision: D13802159
fbshipit-source-id: 891eef00c236b2241571e24c50dc82b9862872cc
Summary: Added tw_handle to be able to distinguish source of samples
Reviewed By: ikostia
Differential Revision: D13800968
fbshipit-source-id: 6c8528fc69302b2d5c5fbd40ccdf729c9379a101
Summary: Simple stats for put/get/is_present so we can monitor basic functionality.
Reviewed By: StanislavGlebik
Differential Revision: D13736062
fbshipit-source-id: 1fd6def0d98d9d29dfbbf11430006b91936d35dc
Summary:
Use rustfix and some hand-editing to convert to Rust 2018 - main
benefit is the elimination of a lot of `extern crate` lines.
Reviewed By: HarveyHunt
Differential Revision: D13736087
fbshipit-source-id: 50e4edfdff3e8ceea94b2beed36399681871a436
Summary: This is the cloud sync test with local (on disk) commit cloud backend and mononoke backend.
Reviewed By: markbt
Differential Revision: D13762527
fbshipit-source-id: c454dbf67999333e37a54e5e6ac84c3cebcf8c3b
Summary: This test covers corner case (partially public stacks)
Reviewed By: StanislavGlebik
Differential Revision: D13750852
fbshipit-source-id: cd1a5a84dfb62951cb37f1fbdd6c510d825adb41
Summary:
this is required to cover corner cases when client has some stacks and part of those became public
calculation for public roots happen for draft heads only, it doesn't change performance of hg pull
Reviewed By: StanislavGlebik
Differential Revision: D13742685
fbshipit-source-id: d8c8bc357628b9b513bbfad4a82a7220d143f364
Summary: Create the corresponding endpoint for thrift calls to get_commit_v2. For now this only supports the non optional fields.
Reviewed By: StanislavGlebik
Differential Revision: D13730602
fbshipit-source-id: fd9d845620c864bf7dade13d810e98270425ea00
Summary:
Previously we rely on CachingChangesetFetcher to quickly fetch all commits into
memory, but CachingChangesetFetcher was deleted in D13695201. Instead let's use
`get_many()` method from Changesets trait to quickly fetch many changesets at
once into memory.
Reviewed By: lukaspiatkowski
Differential Revision: D13712783
fbshipit-source-id: 12e8fa148f7989028547ac8d374438e23b44b6d1
Summary:
The primary motivation for this method is to quickly build skiplist indexes
(see next diff). Building skiplists is much faster if all the data is in
memory.
Reviewed By: lukaspiatkowski
Differential Revision: D13712784
fbshipit-source-id: 716dc020a49cbffac273eb466e474ed86887927d
Summary:
We decided to populate phases table in 2 places: blobimport and push-rebase
(alredy done).
This diff is for blobimport. We know the commits are public.
Reviewed By: lukaspiatkowski
Differential Revision: D13731900
fbshipit-source-id: b64e5643e7cffd9e8fb842e9929f4c1ee7a66197
Summary: Implemented advanced parser to parse bundlecaps json object into more suitable to work datastructure.
Reviewed By: aslpavel
Differential Revision: D13602738
fbshipit-source-id: 9a2f8e78d55a21e80229aae23e5a38f6cc14c7e8
Summary: Log all thrift calls to scuba. Since the params are in thrift format we log them both in json and human readable format (e.g. path is binary thus not really readable in json format). In addition both http and thrift now log the request type.
Reviewed By: StanislavGlebik
Differential Revision: D13730245
fbshipit-source-id: 3419ec067a9066e181210d184195fbd02980c1e0
Summary: `BlobRepo` is cheaply clonable and doesn't need to be `Arc`'d.
Reviewed By: StanislavGlebik
Differential Revision: D13726727
fbshipit-source-id: 0b983c7c4625f47df8a11da5781b7777cc38d72f
Summary:
Previously we had a timeout per session i.e. multiple wireproto command will
share the same timeout. It had a few disadvantages:
1) The main disadvantage was that if connection had timed out we didn't log
stats such as number of files, response size etc and we didn't log parameters
to scribe. The latter is even a bigger problem, because we usually want to
replay requests that were slow and timed out and not the requests that finished
quickly.
2) The less important disadvantage is that we have clients that do small
request from the server and then keep the connection open for a long time.
Eventually we kill the connection and log it as an error. With this change
the connection will be open until client closes it. That might potentially be
a problem, and if that's the case then we can reintroduce perconnection
timeout.
Initially I was planning to use tokio::util::timer to implement all the
timeouts, but it has different behaviour for stream - it only allows to set
per-item timeout, while we want timeout for the whole stream.
(https://docs.rs/tokio/0.1/tokio/timer/struct.Timeout.html#futures-and-streams)
To overcome it I implemented simple combinator StreamWithTimeout which does
exactly what I want.
Reviewed By: HarveyHunt
Differential Revision: D13731966
fbshipit-source-id: 211240267c7568cedd18af08155d94bf9246ecc3
Summary:
the extension was not enabled for repo pull (the second repo)
hg pull -r was still working but other things like hg up <commit cloud hash> were not working.
it caused a bit of confusion
It is cleaner to enable the extension for both sides.
Reviewed By: StanislavGlebik
Differential Revision: D13710518
fbshipit-source-id: 231aec1a71a5c13d707c2b361ce77158573b93f0
Summary:
There is no much point in keeping since we have skiplist which should solve the
same problems in a better way.
The only way where CachingChangesets maybe useful is when many users fetch a
lot of commits simultaneously. It may happen when we merge a new big repository.
However current implementation of CachingChangesets won't help with it since we
do not update its indexes.
Reviewed By: lukaspiatkowski
Differential Revision: D13695201
fbshipit-source-id: 2a4600eccf8224453ca13047e5a2ef3a0af650e3
Summary:
Previously to get copy/move source we had to join `paths` and `fixedcopyinfo`
table. That worked fine when we had just one shard. However now we have many
shards, and join no longer works. The reason is that move source path is in a
different shard compared to move destination path, and join returns no data.
Consider this situation. shardA contains all the data for pathA, shardB
contains all the data for pathB. That means that sharded `paths` table will
have pathA in shardA and pathB in shardB. Then if file pathA was copied form
pathB, then `fixedcopyinfo` table in shardA contains a path_hash of pathB.
However joining shardA's `fixedcopyinfo` with shardA's `paths` to convert
path_hash to path fails because pathB is in shardB.
The only possible fix is to split fetching path_hash from `fixedcopyinfo` and
converting path_hash to path.
I don't think we'll be able to keep the logic with join that we have at the
moment. It would require us to have all paths on all shards which is
unfeasible because it'll make writes much slower.
Reviewed By: aslpavel
Differential Revision: D13690141
fbshipit-source-id: 16b5cae6f23c162bb502b65c208f3ca9e443fb04
Summary:
Going to change these files in the next diff. To make next diff smaller
splitting format changes to this diff.
Reviewed By: aslpavel
Differential Revision: D13690143
fbshipit-source-id: 124232b832d8c67ee7fe931ef174230cb09ff564
Summary:
File content blobs are thrift encoded in Mononoke. This is done so
that we can change the encoding of content blobs easily. For example, we can
add compression or we can add split the blobs in chunks.
However there is a problem. At the moment file content blob key is a hash of
the actual data that's written to blobstore i.e. of a thrift encoded data. That
means that if we add compression or change thrift encoding in any way, then the
file content blob key changes and it changes the commit hashes.
This is wrong. To fix it let's use hash of the actual file content as the key.
Reviewed By: farnz
Differential Revision: D12884898
fbshipit-source-id: e60a7b326c39dad86e2b26c6f637defcb0acc8e8
Summary:
Mercurial has a hack to determine if a file was renamed. If p1 is None then
copy metadata is checked. Note that this hack is purely to make finding renames
faster and we don't need it in Mononoke. So let's just read copy metadata.
This diff also removes `maybe_copied()` method and unused code like `Symlink`
Reviewed By: farnz
Differential Revision: D12826409
fbshipit-source-id: 53792218cb61fcba96144765790278d17eecdbb1
Summary:
as you can test the query like this:
```
select * from demo WHERE `name` IN ()
```
is fine for sqlite but **invalid syntax** in MySql (empty list of value)
the error will be similar to this:
```
You have an error in your SQL syntax; check the manual that corresponds to your MySQL server version for the right syntax to use near ') LIMIT 10000' at line 1; 'select * from phases WHERE repo_id IN () LIMIT 10000'
```
So, such errors are usually shoot in production.
It is better to have the empty check right before calling queries with lists
Reviewed By: lukaspiatkowski
Differential Revision: D13704726
fbshipit-source-id: a9fb3a2e21e88b3af14f57917c2004454eb42531
Summary:
Reformat several rust files with the current rustfmt, to make the linter
happy.
Reviewed By: yfeldblum
Differential Revision: D13683205
fbshipit-source-id: f7a02dae0fbe095b6acde4de380aca2acfedf39d
Summary: This diff is created to separate the lint formatting work from the rest of the code changes in D13632296
Reviewed By: lukaspiatkowski
Differential Revision: D13691680
fbshipit-source-id: 8e12016534d2e6066d803b51b5f12cbf6e89a822
Summary:
It showed up in our profiles because it does unnecessary copies. And for big
request like getfiles it matters a bit. So let's fix it.
Reviewed By: aslpavel
Differential Revision: D13634952
fbshipit-source-id: 98be8bf7236eb12a4009b4b174ffac258f46e0f4
Summary:
Add data to the extra_context scuba field that includes the number of commits in the bundle
as well as certain stats from the changesetfetcher (such as cache misses).
Reviewed By: aslpavel
Differential Revision: D13528646
fbshipit-source-id: 4603d7e95182f4e36b5ef325651ec80997742ea0
Summary:
Update the wireproto command gettreepack to log the total size of the returned
treepacks, as well as the number that are returned.
Reviewed By: StanislavGlebik
Differential Revision: D13278254
fbshipit-source-id: aab9b6f42b11240a7b84bfda07bf99f15508043d
Summary:
Update the wireproto logging to log a summary of the getfile requests, rather than
logging every individual request. This should reduce our logging to scuba.
This diff includes logging of:
- Number of returned files
- Maximum file size
- Total size of files
- Maximum file request latency
Reviewed By: aslpavel
Differential Revision: D13278256
fbshipit-source-id: 069318a718fe915995c7bbe25aa8ccb02c2372f8
Summary:
Some wireproto commands use WireProtoLogger to record information to
both scuba and scribe (for replay). Modify this struct to also allow
a PerfCounter struct to be logged to scuba but _not_ scribe.
This allows for logging of command specific information to scuba, such as
number of files requested.
Reviewed By: StanislavGlebik
Differential Revision: D13278255
fbshipit-source-id: 0ed364c8264ba3ae439746387126a7778712b860
Summary:
PerfCounters is a small wrapper around a concurrent hashmap
that can be used to store performance metrics in. Include it in CoreContext
so that it can be used throughout the codebase.
Reviewed By: aslpavel
Differential Revision: D13528647
fbshipit-source-id: 7c3f26ab8c0c7ba5ee619e85a069af7e7721037f
Summary:
The bulk api makes less queries to mysql and therefore is more efficient.
This is especially important for `hg pull` requests where the list of heads is very large.
Reviewed By: lukaspiatkowski
Differential Revision: D13677298
fbshipit-source-id: 3dec1b3462c520c11481325e82523ef7a6ae6516
