Summary:
It's currently challenging to compare client logging to server logging, because we just don't have enough information to correlate logs.
Add in two pieces of data from the client, if available:
1. Client correlator, which lets us compare to client performance logging.
2. Short hg command (`pull`, `update`, `rebase`, but not `update --clean master` or `rebase --source foo --destination master`). This lets us quickly look for problematic commands with a suitable query
Reviewed By: StanislavGlebik
Differential Revision: D16499809
fbshipit-source-id: c17a348c6f950bd207df5296531e5d39124e3a79
Summary:
This diff sets two Rust lints to warn in fbcode:
```
[rust]
warn_lints = bare_trait_objects, ellipsis_inclusive_range_patterns
```
and fixes occurrences of those warnings within common/rust, hg, and mononoke.
Both of these lints are set to warn by default starting with rustc 1.37. Enabling them early avoids writing even more new code that needs to be fixed when we pull in 1.37 in six weeks.
Upstream tracking issue: https://github.com/rust-lang/rust/issues/54910
Reviewed By: Imxset21
Differential Revision: D16200291
fbshipit-source-id: aca11a7a944e9fa95f94e226b52f6f053b97ec74
Summary: Gettreepack requests can be slow if there's no base manifest. Instead of trying to track every manifest we've passed down to the client (which is CPU and memory expensive, as we end up tracking every `(path, manifesthash)` pair in the output), simply use the first manifest we were asked for as the base manifest in this case
Reviewed By: StanislavGlebik
Differential Revision: D16120603
fbshipit-source-id: 19119e481dfad37c2539f6f02f6ffccb13bc9895
Summary:
The getpackv1 protocol doesn't unfortunately support LFS blobs, which is
therefore blocking deploying remotefilelog.fetchpacks on ovrsource on the
clients.
The easiest way to get there was to simply add a getpackv2 API that is similar
in every way to getpackv1, but with the addition of a metadata field. While
full support for this was added to Mercurial, the Mononoke support is the
absolute minimum required as Mononoke doesn't support LFS.
I'm expecting that EdenAPI will completely remove the need for getpackv2 and
therefore for this code should be fairly short-lived.
Reviewed By: farnz
Differential Revision: D15954031
fbshipit-source-id: 465ac13ed8987191ccf9a7cec198d913143aaf13
Summary:
Side-effect changes:
- Doc comments on macro invocations generate a warning that they don't document what the macro expands to => make them non-doc comments
- New warnings from some bad borrowing patterns which were not previously diagnosed (T45010740)
Reviewed By: Imxset21
Differential Revision: D15515133
fbshipit-source-id: a71db51433598e338b660cbf9d2079f4bd51cfba
Summary: Convert from the type Mercurial uses to represent the arguments to `gettreepack` to Mononoke's representation of that type.
Reviewed By: quark-zju
Differential Revision: D15492733
fbshipit-source-id: 15097cb564f7b5edb024013caea2e4d444263947
Summary:
Removed references to HgNodeHash in repo_client in the specified functions. In
addition, updated other files due to type related dependencies.
Reviewed By: StanislavGlebik
Differential Revision: D14543934
fbshipit-source-id: b0d860fe7085ed4b91a62ab1e27fb2907a642a1d
Summary:
This is the test to cover tricky case in the discovery logic.
Previously Mononoke's known() wireproto method returned `true` for both public
and draft commits. The problem was in that it affects pushrebase.
There are a few problems with the current setup. A push command like `hg push
-r HASH --to BOOK` may actually do two things - it can either move a bookmark
on the server or do a pushrebase. What it does depends on how discovery phase
of the push finishes.
Each `hg push` starts with a discovery algorithm that tries to figure out what commits
to send to the server. If client decides that server already has all the
commits then it'll just move the bookmark, otherwise it'll run the pushrebase.
During discovery client sends wireproto `known()` method to the server
with a list of commit hashes, and server returns a list of booleans telling if
a server knows the commit or not. Before this diff Mononoke returned true for
both draft commits and public commits, while Mercurial returned true it only
for public commits.
So if Mononoke already has a draft commit (it might have it because the commit was created
via `hg pushbackup` or was created in the previous unsuccessful push attempt),
then hg client discovery will decide to move a bookmark instead of
pushrebasing, which in the case of master bookmark might have disastrous
consequences.
To fix it let's return false for draft commits, and also implement `knownnodes` which return true for draft commits (a better name for these methods would be `knownpublic` and `known`).
Note though that in order to trigger the problem the position of the bookmark on the server
should be different from the position of the bookmark on the client. This is
because of short-circuting in the hg client discovery logic (see
https://fburl.com/s5r76yle).
The potential downside of the change is that we'll fetch bookmarks more often,
but we'll add bookmark cache later if necessary.
Reviewed By: ikostia
Differential Revision: D14560355
fbshipit-source-id: b943714199576e14a32e87f325ae8059d95cb8ed
Summary:
This stack is about adding getpackv1 wireproto request support (see task for
more details).
This diff adds a Decoder trait implementation. It parses all parameters for a
single file (i.e. filename + list of nodes). This Decoder is combined with
the previous diff to fully parse getpackv1 request
Reviewed By: lukaspiatkowski
Differential Revision: D14401516
fbshipit-source-id: 9b5fa1f48de338e58a288eb0653bd734cd8d1623
Summary:
The stack is about adding support for getpackv1 wireproto request.
This diff make decode_getfiles_arg_stream function generic over Decoder type.
At the moment decode_getfiles_arg_stream is used for unpacking `getfiles` wireproto
request. However getpackv1 packs request in a similar way to getfiles (which
is different from any other wireproto requests). Let's re-use the same
functionality for getpackv1
Reviewed By: lukaspiatkowski
Differential Revision: D14401517
fbshipit-source-id: ef0e9abeecec61c7c3d25b4a9e36da3f05c870cb
Summary:
Client telemetry is used to send which commands are being run on the client,
which makes debugging slow sessions easier. For the client perspective, the
server hostname is send back so that the user knows to which physical host it's
connected, which is also useful debug info.
Reviewed By: StanislavGlebik
Differential Revision: D14261097
fbshipit-source-id: f4dc752671b76483f9dcb38aa4e3d16680087850
Summary:
It breaks mononoke traffic replay - https://fburl.com/scuba/1rxovt8t.
It changed how mononoke logs requets to replay and now bundlecaps is a dict
instead of a string, so this line in traffic replay fails -
https://fburl.com/g8gwhzrq.
Reviewed By: lukaspiatkowski
Differential Revision: D13972033
fbshipit-source-id: a6e8258da8e7a5b6f90b869781f448a2202a07a1
Summary: This is adds a metaconfig option to preserve push/pushrebase bundles in the blobstore.
Reviewed By: StanislavGlebik
Differential Revision: D14020299
fbshipit-source-id: 94304d69e0ac5d81232f058c6d94eec61eb0020a
Summary:
This diff does not change anything on it's own, but rather adds the not
reachable (but already somewhat tested) code to preserve bundles when doing
pushes and pushrebases.
I want to land it now so that conflict resolution is easier.
Reviewed By: StanislavGlebik
Differential Revision: D14001738
fbshipit-source-id: e3279bc34946400210d8d013910e28f8d519a5f8
Summary:
Repo name is used only be verify_integrity hook and even there the name that Mononoke provides is incorrect. Instead of Mononoke's `repo-RepositoryId(1234)` name the hook is interested in Mercurial's `fbsource` name.
HookConfig is a perfect way to pass such an arbitrary parameter so use it.
Reviewed By: StanislavGlebik
Differential Revision: D13964486
fbshipit-source-id: 94090e409d5206828364202ae62a37abc16e4a27
Summary:
Implemented advanced parser to parse bundlecaps json object into more suitable to work datastructure.
Patched version of D13602738
Reviewed By: ikostia
Differential Revision: D13751782
fbshipit-source-id: 977b70c121d0df587082b8db615cbea7a00e3aa4
Summary: Implemented advanced parser to parse bundlecaps json object into more suitable to work datastructure.
Reviewed By: aslpavel
Differential Revision: D13602738
fbshipit-source-id: 9a2f8e78d55a21e80229aae23e5a38f6cc14c7e8
Summary:
It showed up in our profiles because it does unnecessary copies. And for big
request like getfiles it matters a bit. So let's fix it.
Reviewed By: aslpavel
Differential Revision: D13634952
fbshipit-source-id: 98be8bf7236eb12a4009b4b174ffac258f46e0f4
Summary:
See representation from Mercurial.
```
gboptsmap = {
"heads": "nodes",
"bookmarks": "boolean",
"common": "nodes",
"obsmarkers": "boolean",
"phases": "boolean",
"bundlecaps": "scsv",
"listkeys": "csv",
"cg": "boolean",
"cbattempted": "boolean",
}
```
Some logic might need to check what caps client supports. So convenient lookup is needed.
For example for phases, when we choose to transport phases via a separate bundle2 section called "phase-heads", we need to check that it is a supported on the client side.
Reviewed By: StanislavGlebik
Differential Revision: D13519888
fbshipit-source-id: 6471f2a5057cc3bb2dd72d9318a6f253a447b758
Summary:
Some params in getbundle requests were ignored in Mononoke
we are going to use "phases" param to check if we need provide phases or not
Mercurial uses 1/0:
```
elif keytype == "boolean":
value = "%i" % bool(value)
```
Reviewed By: aslpavel
Differential Revision: D13517508
fbshipit-source-id: b066b335d1b972e9845be9fa3862bbf4d11817cd
Summary: passing CoreContext around instead of Logger gives us more flexibility
Reviewed By: StanislavGlebik
Differential Revision: D13340855
fbshipit-source-id: 43bc4def1532ce675a4ec70903fd6c04332c967e
Summary:
Add libnfs, libffi and starlark.
Also nom now has "verbose-errors" feature (via bindgen -> cexpr -> nom), so make some tweaks to cope.
Reviewed By: farnz
Differential Revision: D10371391
fbshipit-source-id: ba889ad16a7b662c5eddafcb1e705b068ccc9af7
Summary:
Streaming clones are a neat hack; we get to send files down to the
Mercurial client, which it then writes out as-is. Usefully, we can send no
files, and the protocol still works.
Set up the capabilities etc needed so that we send streaming clones to
Mercurial clients, even if they're rather useless so far.
Reviewed By: StanislavGlebik
Differential Revision: D9926967
fbshipit-source-id: b543e802adac38c8bc318081eec364aed87b0231
Summary: Should reduce memory usage and make getbundle a bit faster
Reviewed By: farnz
Differential Revision: D9861578
fbshipit-source-id: 57bca3700e3a38aeb70f267e6dc90d6b8a9d2955
Summary: It avoids a heap of copies
Reviewed By: StanislavGlebik
Differential Revision: D9595689
fbshipit-source-id: a64f0a383acd517830d08cf0be9fc0a1b6903382
Summary:
I was debugging pushrebase bugs, and the only error I got was
'oneshot::Cancelled'. Let's add a bit more context around it
Reviewed By: farnz
Differential Revision: D9554384
fbshipit-source-id: b3111ef1b5c743d65740f7fa3fd1a92eef9ab784
Summary: all that places logged all list of HgNodeHash
Reviewed By: StanislavGlebik
Differential Revision: D9242904
fbshipit-source-id: 42a98b04986f2ed432a8956828356bd5b9bcaa88
Summary:
This is a series of patches which adds Cargo.toml files to all the crates and tries to build them. There is individual patch for each crate which tells whether that crate build successfully right now using cargo or not, and if not, reason behind that.
Following are the reasons why the crates don't build:
* failure_ext and netstring crates which are internal
* error related to tokio_io, there might be an patched version of tokio_io internally
* actix-web depends on httparse which uses nightly features
All the build is done using rustc version `rustc 1.27.0-dev`.
Pull Request resolved: https://github.com/facebookexperimental/mononoke/pull/7
Differential Revision: D8778746
Pulled By: jsgf
fbshipit-source-id: 927a7a20b1d5c9643869b26c0eab09e90048443e
Summary:
Unfortunately I have to remove tracing. The reason is because tracing doesn't
work with Streams. For now it should be fine because enabling tracing in prod
is still not possible because of the memory and cpu overhead.
Reviewed By: farnz
Differential Revision: D8381855
fbshipit-source-id: e28b4396c81527bdf30fa1703c634688cf645ada
Summary:
Now it is as it should be: mercurial_types have the types, mercurial has revlog related structures
burnbridge
Reviewed By: farnz
Differential Revision: D8319906
fbshipit-source-id: 256e73cdd1b1a304c957b812b227abfc142fd725
Summary:
Some wireproto methods may have huge argument lists. They just print too much
data to stderr, and make stderr basically useless. Let's limit it
Reviewed By: jsgf
Differential Revision: D8076949
fbshipit-source-id: 58e760f25a7d4fdc9fc7bd95635f916a08e15ed2
Summary:
Rust 1.26 adds many new language features. In particular `impl Trait` is now
stable, so we no longer need `conservative_impl_trait`.
There also seems to have been changed in the (unstable) TryFrom with respect to
usize, and the behaviour of the never type `!`.
There are still a few deprecation warnings, but they don't cause the build to
fail.
Path remapping is now stable, so the buck config needs to change to use it
rather than the unstable command line option.
TODO:
- get aarch64 rust-crates-io build (can defer to a later update)
Reviewed By: Imxset21
Differential Revision: D7966091
fbshipit-source-id: 2e61e262c21eb01c852a36f49c6a6369cdaddcdb
Summary:
We want to be able to identify "interesting" sessions - add a Scuba
sample that tells us what wireprotocol commands were sent, and how long the
session lasted.
Reviewed By: jsgf
Differential Revision: D7813906
fbshipit-source-id: a9bd48996a60b41098243f6c815465cd33d1429c
Summary:
mercurial::NodeHash is the representation of Mercurial's Sha1 hash, so it should be used to parse and generate hgproto.
This diff doesn't not change two crucial places that are required for full coverage of NodeHash remapping in future:
- Bundle2's tree manifest generation code
- Bundle2's resolver code
Those will be added in dependent diffs
Reviewed By: sid0
Differential Revision: D7600471
fbshipit-source-id: 40e05d5cce6c454200169f6f0049e57d427e9403
Summary:
This wireproto method is used by remotenames to update the list of
remotebookmarks. Implementation is the same as for listkeys bundle2 part.
Reviewed By: farnz
Differential Revision: D7271597
fbshipit-source-id: 8a75a93cae0e571d86d657e1c1d718a7fa0ab4ea
Summary:
Pushkey part is used to send bookmark updates from hg client to the server.
This diff does all the wireproto parsing, but doesn't actually apply bookmark
updates on the server.
Also this diff "implements" branchmap method. We have no plans to support it,
but currently remotenames extension calls it. So this diff adds a fake
implementation that always returns empty response.
Reviewed By: farnz
Differential Revision: D7150973
fbshipit-source-id: 6889c02a1105127b1805ef1fafa6fbe9c2d57e7d
