Summary: this breaks the prod since we are not able to parse clienttelemetry being send by hg
Reviewed By: jsgf
Differential Revision: D7967361
fbshipit-source-id: e10eb52852ccb8c2878db01cd63b05cd7f901ae4
Summary:
@[1636530097:mung] reported that there was a bug wherein Diesel was incorrectly using MySQL's API by incorrectly calling `mysql_store_result` after `mysql_next_result` indicates that no more results are available. This behavior [appears to be intentional](705cdbe1fe/diesel/src/mysql/connection/raw.rs (L160)), but directly contradicts what the [documentation says is correct](https://dev.mysql.com/doc/refman/8.0/en/mysql-store-result.html). In MySQL 8.0, this results in the client getting an error for incorrect usage of the API.
I've added a local patch to [our fork of Diesel](https://github.com/kulshrax/diesel) on Github to [remove the extra call to `mysql_store_result`](7ae79c193e), and rebuilt the `rust-crates-io` tp2 project to use this version of Diesel. This diff updates the symlinks in fbcode to point to the updated tp2 project.
Reviewed By: lth
Differential Revision: D7961290
fbshipit-source-id: 804c599a5688cd54fb0fcc33688c608d068d7a7b
Summary: This allows us to put hgcli on the different host
Reviewed By: jsgf
Differential Revision: D7910980
fbshipit-source-id: 45bdd1ee42b54f8f1a425ed6950e785cdf85bfc3
Summary:
This is a (hopefully) short term hack to overcome the problem of overloading
Manifold.
Ideally manifold client has to adjust dynamically to the load. However
implementing it is
not trivial, so for now let's configure via config option.
Reviewed By: jsgf
Differential Revision: D7910979
fbshipit-source-id: c2dc32b592747732e7e6574e0fecf2d0aaef447e
Summary:
Curently hgcli can connect to Mononoke only if they are on the same host,
because unix domain socket are used. Also Mononoke has to use separate unix
domain socket for different repos.
The goal of this stack of diffs is to remove these limitations:
1 Make it possible to have hgcli and Mononoke server on different hosts
2 Make it possible to use one port/unix domain socket to connect to many repos.
This diff adds a separate thread that parses the Preamble, extracts the
reponame and sends request to a thread responsible for this repo.
Also hgcli now has a new cmd line option that specifies path to connect to.
Reviewed By: jsgf
Differential Revision: D7845156
fbshipit-source-id: a48bcfeec4755b2d5b3dfcf8e0383e6945178018
Summary:
Simple precaching. Reads all the manifests for a bookmark and up to
`commit_warmup_limit` of ancestors.
Warming up file content can be slow, so we don't do it now.
Reviewed By: jsgf
Differential Revision: D7863728
fbshipit-source-id: bed1508b01e4e002a399d00ea45faf8a8e228d0a
Summary:
This will make it easier to change the "real" bookmark type from AsciiString to
String if we decide to do that.
BookmarkPrefix is a separate type because we may want to change it from
AsciiString to String. Also we don't want to confuse a bookmark prefix with a
bookmark name.
Reviewed By: jsgf
Differential Revision: D7909992
fbshipit-source-id: 3d4d075c204ed5ef1114a743430982c2836bac04
Summary:
There are already many parameters, and I want to add a few more. Let's pass
RepoConfig directly
Reviewed By: jsgf
Differential Revision: D7861092
fbshipit-source-id: 76a6e4d2fbb06792fea50cee544f6fd62f670887
Summary:
We don't need to explicitly create timers since the environment has
one set up by default.
Reviewed By: StanislavGlebik
Differential Revision: D7873576
fbshipit-source-id: bfcdc27a46397bff0730f64ad4f3de3865c7cfa1
Summary:
Previously we wouldn't start processing next request until we are done with
this one.
It's easy to fix with asynchronize.
Reviewed By: jsgf
Differential Revision: D7835828
fbshipit-source-id: a9efef896f09815ca2f3e3659c814ad65edbb715
Summary:
Let's use the new feature in SendWrapper to use many io threads. That will help
us mitigate the high cpu usage issues we were having with blobstore requests.
Manifold blobstore now creates the io threads itself.
Reviewed By: kulshrax
Differential Revision: D7831420
fbshipit-source-id: ec9f3327347ca6bfbd23c482e69a6fee663b1da5
Summary: As with changesets and blobs, let's cache filenodes data
Reviewed By: jsgf
Differential Revision: D7831105
fbshipit-source-id: 334cb474f5cc3ef8dba0945d11273b2b3875e8ad
Summary:
Curently hgcli can connect to Mononoke only if they are on the same host, because unix domain socket are used. Also Mononoke has to use separate unix domain socket for different repos.
The goal of this stack of diffs is to remove these limitations:
1 Make it possible to have hgcli and Mononoke server on different hosts
2 Make it possible to use one port/unix domain socket to connect to many repos.
Check that that reponame from Preamble is the same as Mononoke reponame, and raise an error if it isn't.
Reviewed By: jsgf
Differential Revision: D7845157
fbshipit-source-id: 6567d5417ce58c4fba0364108592eed5bf0223c1
Summary:
Curently hgcli can connect to Mononoke only if they are on the same host, because unix domain socket are used. Also Mononoke has to use separate unix domain socket for different repos.
The goal of this stack of diffs is to remove these limitations:
1 Make it possible to have hgcli and Mononoke server on different hosts
2 Make it possible to use one port/unix domain socket to connect to many repos.
In this diff let's start sending Preamble as the first message from hgcli to Mononoke server.
For now Mononoke server will just ignore it. Later we'll use it to connect to the correct repo.
Reviewed By: jsgf
Differential Revision: D7845159
fbshipit-source-id: 3d2232028d931bfa7309e826b38084a87be1f85c
Summary:
Specialized revsets to make pull faster.
Previous Union + Intersect combination was extremely slow because it fetched a
lot of stuff that wasn't used.
Reviewed By: farnz
Differential Revision: D7829394
fbshipit-source-id: c038f184c305e48e18b6fcb0f83bab9e9a42b098
Summary:
We want to be able to identify "interesting" sessions - add a Scuba
sample that tells us what wireprotocol commands were sent, and how long the
session lasted.
Reviewed By: jsgf
Differential Revision: D7813906
fbshipit-source-id: a9bd48996a60b41098243f6c815465cd33d1429c
Summary:
We want to be able to tie up our logs with a session identifier, so
that we can see if a given slow operation is expected (a complex session) or a
surprise. Give the client a logger that has a session ID tied in.
Reviewed By: jsgf
Differential Revision: D7813908
fbshipit-source-id: bc8a59046c19a0939fd044ca8b5b7ccc2956c5f0
Summary:
Use asyncmemo to cache Changesets.
Unfortunately currently we are using separate asyncmemo cache, so we have to
specify the size for the caches separately. Later we'll have a single cache for
everything, and the number of config knobs will go down.
Reviewed By: lukaspiatkowski
Differential Revision: D7685376
fbshipit-source-id: efe8a3a95fcc72fab4f4af93564e706cd1540c2f
Summary:
Let's use it! Pass config option that set's the cache max memory usage (don't
put a limit on the number of entries, it's useless in that case).
Currently we'll set a separate size for each of the caches that we use
(blobstore, changesets, filenodes, etc). Later we'll have just one single option that
sets the cache size for all of them.
Reviewed By: lukaspiatkowski
Differential Revision: D7671814
fbshipit-source-id: f9571078e6faaa80ea4c31c76a9eebcc24d8a68a
Summary: The Hg prefix suggests that a thing is a pure mercurial object, but in the case of HgRepo and hgrepo it suppose to be a Mononoke blobrepo. Renaming to reduce the confusion
Reviewed By: StanislavGlebik
Differential Revision: D7667260
fbshipit-source-id: 72c172c035212ba260f9bc4dc02469400e1d0813
Summary:
Pass mysql tier name to the BlobRepo, so that we can use it to connect to mysql
based storages like mysql changeset storage, filenodes storage etc.
Note that currently Filenodes storage only connects to master region. This will
be fixed in the later diffs
Reviewed By: lukaspiatkowski
Differential Revision: D7585191
fbshipit-source-id: 168082abfeb7ccba549c7a49e6269cc01c490c14
Summary:
As with other methods like gettreepack and getfiles, let's use parallelism.
Later we may make the buffer size configurable if we need to.
Reviewed By: lukaspiatkowski
Differential Revision: D7587674
fbshipit-source-id: 51be6d0e338fb062a116fe6c723f272c66f9da2a
Summary:
Now that `BlobNode` no longer returns `None`:
* don't expose the `BlobNode` API outside the crate because it turns out to not be very useful (it should probably go away eventually?)
* make the `File` API not return `Option` types
* Add a new `file_contents` that returns a brand-new `FileContents` (this is the first time we're tying together Mercurial and Mononoke data structures!)
Also remove a `Symlink` API that isn't really correct honestly.
Reviewed By: StanislavGlebik
Differential Revision: D7624729
fbshipit-source-id: 38443093b8bfea91384c959f3425cf355fac9f65
Summary: The Hg prefix is unique now so let's not use verbose mercurial::
Reviewed By: sid0
Differential Revision: D7620112
fbshipit-source-id: 0aece310ed817445fef4c94b32f78fda3a3b1c49
Summary: mercurial_types::DParent should be replaced by types from mononoke_types or mercurial in most cases. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619686
fbshipit-source-id: 5ad105113779387f1408c806860483e06ed5fb3d
Summary: mercurial_types::DManifestId should be replaced by types from mononoke_types in most cases and by mercurial::HgManifestId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7619062
fbshipit-source-id: 447224194c6555334b64dc29ebabe3ef0d0cb87e
Summary: mercurial_types::DChangesetId should be replaced by types from mononoke_types in most cases and by mercurial::HgChangesetId in others. This rename should help with tracking this
Reviewed By: sid0
Differential Revision: D7618897
fbshipit-source-id: 78904f57376606be99b56662164e0c110e632c64
Summary: mercurial_types::NodeHash should be replaced by types from mononoke_types in most cases and by mercurial::NodeHash in others. This rename should help with tracking this fact.
Reviewed By: sid0
Differential Revision: D7618389
fbshipit-source-id: a876e723d911df626c7851fba56a056843b4e049
Summary:
This should be mostly a no-op change, only the types change so that it prepares this code to be able to remap Root Manifest ids.
There is one small logical change that I am also shipping here, which is that fetching of linknode is happening in parallel with fetching of parents and content of the entry.
Reviewed By: StanislavGlebik
Differential Revision: D7600960
fbshipit-source-id: 2233f1c7d6f14e38cb075b5d7f40339d61cbf574
Summary: getbundle returns changelogs back to Mercurial, so it's IDs should be retrieved from the remapping tables. This diff prepares it for that change
Reviewed By: StanislavGlebik
Differential Revision: D7600959
fbshipit-source-id: e586a92b272e604a1de2619feb9838e982dd802c
Summary:
mercurial::NodeHash is the representation of Mercurial's Sha1 hash, so it should be used to parse and generate hgproto.
This diff doesn't not change two crucial places that are required for full coverage of NodeHash remapping in future:
- Bundle2's tree manifest generation code
- Bundle2's resolver code
Those will be added in dependent diffs
Reviewed By: sid0
Differential Revision: D7600471
fbshipit-source-id: 40e05d5cce6c454200169f6f0049e57d427e9403
Summary:
This is a cleanup of NodeHash API. There were few unused methods and few ways to convert between mercurial and mercurial_types hashes. With this diff it is very easy to identify the places where this converstion happens.
A followup of this diff will be to use this new API to easily replace the NodeHash convertions in places where it requires remapping.
Reviewed By: sid0
Differential Revision: D7592876
fbshipit-source-id: 6875aa6df1a3708ce54ca5724f6eb960d179192b
Summary:
`left` and `right` are too vague in this context, since there's
already `Added` and `Removed` which imply an explicit direction.
Reviewed By: StanislavGlebik
Differential Revision: D7560975
fbshipit-source-id: 847f600508434d62857ff60a6e6678a4426ffa88
Summary:
Buffering many file requests increases parallelism.
Note that currently buffer size is hardcoded. We may change it later, but right
now let's not add too many config knobs.
Reviewed By: farnz
Differential Revision: D7568348
fbshipit-source-id: 80456d189cc21d050fcaf7d17cabf8cd59d61d63
Summary: Since we have changesets, we can answer known queries more efficiently.
Reviewed By: farnz
Differential Revision: D7534911
fbshipit-source-id: 72f75c9d9a6cd7f1464660eb0ec16f94ca24f33c
Summary: Update the Scuba logging code in Mononoke server to also upload traces to Everstore
Reviewed By: jsgf, farnz
Differential Revision: D7492955
fbshipit-source-id: 6f0f788af33210064977894b97798bb4143caf1b
Summary: Previously, the `.timed()` combinator exported by the `futures-stats` would accept a synchronous callback to log the recorded stats. In practice, logging the stats may require blocking IO, so this combinator should be modified to accept a closure that returns a Future instead.
Reviewed By: farnz
Differential Revision: D7492957
fbshipit-source-id: c634b511e5a93a18787ab3e11ab8e1c19de7566e
Summary:
filenodes are faster than blobstore in answering `get_parents` and
`get_copy_info` queries. So let's use them instead of going to blobstore.
Reviewed By: jsgf
Differential Revision: D7485151
fbshipit-source-id: f9e2ce9dd42b3e9b18140cc3de2c9718f20ab6d4
Summary: We are not using it, so there is no point in keeping it around
Reviewed By: farnz
Differential Revision: D7400428
fbshipit-source-id: 481ef3ec8ef1f188e01add36e81da789f186548e
