Commit Graph

110 Commits

Author SHA1 Message Date
Lukas Piatkowski
f317302b0f autocargo v1: reformating of oss-dependencies, workspace and patch sections and thrift files to match v2
Summary:
For dependencies V2 puts "version" as the first attribute of dependency or just after "package" if present.
Workspace section is after patch section in V2 and since V2 autoformats patch section then the third-party/rust/Cargo.toml manual entries had to be formatted manually since V1 takes it as it is.
The thrift files are to have "generated by autocargo" and not only "generated" on their first line. This diff also removes some previously generated thrift files that have been incorrectly left when the corresponding Cargo.toml was removed.

Reviewed By: ikostia

Differential Revision: D26618363

fbshipit-source-id: c45d296074f5b0319bba975f3cb0240119729c92
2021-02-25 15:10:56 -08:00
Thomas Orozco
c064d70c02 mononoke: fork Gotham less
Summary:
We use a forked version of Gotham in Mononoke. This isn't great, because we
have to maintain this fork. Ideally, we'd upstream our changes, but as is
they're a bit intrusive and not generally useful, which makes this hard.

I've reworked how we do our Gotham changes, and now we only need to make 1 bit
of code public, which might be easier to get upstream. Concretely, Gotham has a
concept of "connected handler" that links a Hyper request and a socket address,
but in our case we want more things. This change lets us instantiate our own
Gotham state, and then add a few more things to it as necessary.

This diff updates our code accordingly to be compatible. This also lets us trim
down on some ceremony we had to do call into Gotham
from Mononoke Server.

Reviewed By: farnz

Differential Revision: D26634653

fbshipit-source-id: 024a48ebc3f323c165ac412ef422755e8cb1c650
2021-02-25 07:42:29 -08:00
Thomas Orozco
097e4ad00c mononoke: remove tokio-compat (i.e. use tokio 0.2 exclusively)
Summary:
The earlier diffs in this stack have removed all our dependencies on the Tokio
0.1 runtime environment (so, basically, `tokio-executor` and `tokio-timer`), so
we don't need this anymore.

We do still have some deps on `tokio-io`, but this is just traits + helpers,
so this doesn't actually prevent us from removing the 0.1 runtime!

Note that we still have a few transitive dependencies on Tokio 0.1:

- async-unit uses tokio-compat
- hg depends on tokio-compat too, and we depend on it in tests

This isn't the end of the world though, we can live with that :)

Reviewed By: ahornby

Differential Revision: D26544410

fbshipit-source-id: 24789be2402c3f48220dcaad110e8246ef02ecd8
2021-02-22 09:22:42 -08:00
Lukas Piatkowski
cd0b6d50e2 autocargo v1: changes to match autocargo v2 generation results.
Summary:
The changes (and fixes) needed were:
- Ignore rules that are not rust_library or thrift_library (previously only ignore rust_bindgen_library, so that binary and test dependencies were incorrectly added to Cargo.toml)
- Thrift package name to match escaping logic of `tools/build_defs/fbcode_macros/build_defs/lib/thrift/rust.bzl`
- Rearrange some attributes, like features, authors, edition etc.
- Authors to use " instead of '
- Features to be sorted
- Sort all dependencies as one instead of grouping third party and fbcode dependencies together
- Manually format certain entries from third-party/rust/Cargo.toml, since V2 formats third party dependency entries and V1 just takes them as is.

Reviewed By: zertosh

Differential Revision: D26544150

fbshipit-source-id: 19d98985bd6c3ac901ad40cff38ee1ced547e8eb
2021-02-19 11:03:55 -08:00
Lukas Piatkowski
87ddbe2f74 autocargo v1: update autocargo field format to allow transition to autocargo v2
Summary:
Autocargo V2 will use a more structured format for autocargo field
with the help of `cargo_toml` crate it will be easy to deserialize and handle
it.

Also the "include" field is apparently obsolete as it is used for cargo-publish (see https://doc.rust-lang.org/cargo/reference/manifest.html#the-exclude-and-include-fields). From what I know this might be often wrong, especially if someone tries to publish a package from fbcode, then the private facebook folders might be shipped. Lets just not set it and in the new system one will be able to set it explicitly via autocargo parameter on a rule.

Reviewed By: ahornby

Differential Revision: D26339606

fbshipit-source-id: 510a01a4dd80b3efe58a14553b752009d516d651
2021-02-12 23:28:25 -08:00
Thomas Orozco
2a21e4fb17 third-party/rust: update Tokio to 0.2.25 + add a patch to disable coop scheduling
Summary:
See the patch & motivation here:

818f943db3

Reviewed By: StanislavGlebik

Differential Revision: D26399890

fbshipit-source-id: e184a3f6c1dd03cb4cdb7ea18073c3392d7ce355
2021-02-12 04:56:23 -08:00
Thomas Orozco
67125980bb mononoke: add load limiter to EdenAPI sessions
Summary:
Like it says in the title. I'd like to access our load limiter from there and
this is a good way to do it.

Note that EdenAPI doesn't have client hostnames when we instantiate the load
limiter, but we also don't truly need it because our client identities include
the hostprefix (at least in cases where a reverse lookup stands a chance of
succeeding, i.e. in prod).

Reviewed By: HarveyHunt

Differential Revision: D26250743

fbshipit-source-id: d1ad062d2967bfe28b3ddc164cac977d297d1b6a
2021-02-05 15:16:09 -08:00
Thomas Orozco
a062348079 mononoke/edenapi_server: split edenapi into 2 crates
Summary:
Like it says in the title, this splits edenapi_server into 2 crates. Most of
the logic here is pretty straightforward:

- edenapi_service creates a Gotham handler
- edenapi_server instantiates the handler and sets up a socket, tls, etc.

Reviewed By: ahornby

Differential Revision: D26108439

fbshipit-source-id: 6a79e9767ba891265bca11f78eb1a6d3a61ee21f
2021-02-04 10:40:03 -08:00
Thomas Orozco
632c6e2280 mononoke/gotham_ext: move trusted-proxy check out of middleware
Summary:
I'd like to support bridging to EdenAPI Server in Mononoke Server. Mononoke
Server already performs an ACL check for trusted proxies when the client
connects, I'd like to pass this information to EdenAPI to avoid re-doing
a check we've already done.

This allows that.

Reviewed By: ahornby

Differential Revision: D26108441

fbshipit-source-id: f0a294e340f38d039b3ba30a4c262c4a8ccbb318
2021-02-04 10:40:03 -08:00
Alex Hornby
960740c37d mononoke: return Result from cmdlib::args::parse_blobstore_options
Summary: Use normal ? style rather than panic with except.

Reviewed By: krallin

Differential Revision: D26176912

fbshipit-source-id: d04ebc4b6c04dd1f8f34b49bee350b52feb11ec1
2021-02-04 00:26:33 -08:00
Thomas Orozco
e1dca89dca mononoke/mononoke_api: make WBC configurable + take fewer parameters
Summary:
Right now, if you enable Mononoke API, you always get a WBC, for all derived
data kinds, and with a delay. This isn't great for a few reasons:

- Some places don't really care about bookmarks being warm to being with. This
  is the case in the bookmarks filler (note that right now, it still does
  polling to satisfy the WBC API, it's just not "warm").
- Some places don't want a delay, or don't want all kinds. This is the case for
  Mononoke Server (which doesn't use Mononoke API right now, but that's what
  I'm working towards), or EdenAPI, which uses a WBC sort-of-de-facto but
  doesn't really care (but likely will in the future, and will want to follow
  Mononoke Server's behavior).

As of this diff, we can now configure this when initializing `Mononoke`. I also
split out all those args into a `MononokeEnvironment` because the args list
was getting out of hand. One thing I did not do is make a way to instantiate
`MononokeEnvironment` from args (but we should do it at some point in the
future).

Reviewed By: StanislavGlebik

Differential Revision: D26100706

fbshipit-source-id: 1daa6335f3ce2b297929a84788bc5b4d9ad6432f
2021-02-01 07:53:17 -08:00
Thomas Orozco
05d54fcb46 mononoke/mononoke_api: split hg parts into mononoke_api_hg
Summary:
I'd like to be able to use mononoke_api from repo_client, but the hg parts
depend on repo_client, so that's a circular dependency.

This splits out the hg parts of Mononoke API so that places that don't want
them don't have to import them. This is similar to what we did with blobrepo.

Reviewed By: StanislavGlebik

Differential Revision: D26099495

fbshipit-source-id: 73a9c7b5dc95feceb35b5eabccf697e9aa0a27de
2021-02-01 07:53:16 -08:00
Kostia Balytskyi
43d406a808 make get_scuba_sample_builder use observability_context
Summary:
Prior to this diff, only mononoke server initialized
`MononokeScubaSampleBuilder` in a way that used observability context, and
therefore respected vebosity settings.

Let's make a generic sample initializing function use this config too.

Reviewed By: ahornby

Differential Revision: D26156986

fbshipit-source-id: 632bda279e7f3905367b82db5b36f81264156de3
2021-02-01 03:57:06 -08:00
Thomas Orozco
ad5dc2d59d third-party/rust: hyper-02: update to 0.13.9
Summary:
Just a minor version update. I'd like to add a patch on top of this for a PR
that hasn't been merged yet, but updating to the underlying released version
first will make the diff clearer.

Reviewed By: ahornby

Differential Revision: D26047997

fbshipit-source-id: 91856f645ec3aaaf4fbf256a23c7e8d4db0f6b37
2021-01-26 07:21:28 -08:00
Daniel Xu
5715e58fce Add version specificiation to internal dependencies
Summary:
Lots of generated code in this diff. Only code change was in
`common/rust/cargo_from_buck/lib/cargo_generator.py`.

Path/git-only dependencies (ie `mydep = { path = "../foo/bar" }`) are not
publishable to crates.io. However, we are allowed to specify both a path/git
_and_ a version. When building locally, the path/git is chosen. When publishing,
the version on crates.io is chosen.

See https://doc.rust-lang.org/cargo/reference/specifying-dependencies.html#multiple-locations .

Note that I understand that not all autocargo projects are published on crates.io (yet).
The point of this diff is to allow projects to slowly start getting uploaded.
The end goal is autocargo generated `Cargo.toml`s that can be `cargo publish`ed
without further modification.

Reviewed By: lukaspiatkowski

Differential Revision: D26028982

fbshipit-source-id: f7b4c9d4f4dd004727202bd98ab10e201a21e88c
2021-01-25 22:10:24 -08:00
Thomas Orozco
4dd3461824 third-party/rust: update Tokio 0.2.x to 0.2.24 & futures 1.x to 1.30
Summary:
When we tried to update to Tokio 0.2.14, we hit lots of hangs. Those were due
to incompatibilities between Tokio 0.2.14 and Futures 1.29. We fixed some of
the bugs (and others had been fixed and were pending a release), and Futures
1.30 have now been released, which unblocks our update.

This diff updates Tokio accordingly (the previous diff in the stack fixes an
incompatibility).

The underlying motivation here is to ease the transition to Tokio 1.0.
Ultimately we'll be pulling in those changes one or way or another, so let's
get started on this incremental first step.

Reviewed By: farnz

Differential Revision: D25952428

fbshipit-source-id: b753195a1ffb404e0b0975eb7002d6d67ba100c2
2021-01-25 08:06:55 -08:00
Radu Szasz
5fb5d23ec8 Make tokio-0.2 include test-util feature
Summary:
This feature is useful for testing time-dependent stuff (e.g. it
allows you to stop/forward time). It's already included in the buck build.

Reviewed By: SkyterX

Differential Revision: D25946732

fbshipit-source-id: 5e7b69967a45e6deaddaac34ba78b42d2f2ad90e
2021-01-18 10:38:08 -08:00
Thomas Orozco
772680f2f7 third-party: rust: update Gotham to 0.5
Summary:
This updates Gotham. Under the hood I rebased our fork, you can see the diff
here: P161171514.

The stuff that is relevant is that Gotham got rid of its dependency on
`failure` and now uses `anyhow` instead, and I also added a little bit to our
existing socket data patch by making a few things public so that we can get
access to a few more internals.

Reviewed By: StanislavGlebik

Differential Revision: D25850262

fbshipit-source-id: 25ebf5d63be39e3e93208705d91abc5c61c90453
2021-01-11 03:32:25 -08:00
Thomas Orozco
81d6aae51d mononoke/edenapi_server: capture handler errors
Summary:
Right now we don't log handler errors to Scuba. This can make debugging a
little tricky: if a client is sending an invalid request, we'll see that we
sent them a 400, but we won't know what was invalid about the request.

This diff updates our logging to actually log that.

Reviewed By: sfilipco

Differential Revision: D25826522

fbshipit-source-id: 89486014e0eeaac5c9b149224601db54a26080d9
2021-01-08 08:45:01 -08:00
Stefan Filip
5f6d1a2c61 edenapi: add full_idmap_clone endpoint
Summary:
The end goal is to have clients using a sparse IdMap. There is still some work
to get there though. In the mean time we can test repositories that don't use
any revlogs. The current expections for those repositories are that they have
a full idmap locally.

Reviewed By: quark-zju

Differential Revision: D25075341

fbshipit-source-id: 52ab881fc9c64d0d13944e9619c087e0d4fb547c
2020-12-08 18:30:24 -08:00
Stefan Filip
2193b84b43 autocargo: regen
Summary: Regen autocargo

Reviewed By: quark-zju

Differential Revision: D25409963

fbshipit-source-id: 7dbbe420aeb30248bf43d3a96a9aa6a47bb2b0be
2020-12-08 18:30:24 -08:00
Alex Hornby
935a7ddfc8 mononoke: remove the need to pass in cachelib settings twice
Summary:
Previously needed to pass in cachelib settings once to MononokeAppBuilder and once to parse_and_init_cachelib.

This change adds a MononokeClapApp and MononokeMatches that preserve the settings, thus preventing the need to pass them in twice (and thus avoiding possible inconsistency)

MononokeMatches uses MaybeOwned to hold the inner ArgMatches, which allows us to hold both the usual reference case from get_matches and an owned case for get_matches_from which is used in test cases.

Reviewed By: krallin

Differential Revision: D24788450

fbshipit-source-id: aad5fff2edda305177dcefa4b3a98ab99bc2d811
2020-12-02 07:27:23 -08:00
Alex Hornby
caa684450f mononoke: show cachelib cache_size in --help usage
Summary:
Show cachelib cache_size default in --help usage so its clear what you'll get if no command line args passed

Because we need to convert from bytes to GiB, the lifetime of the help string isn't long enough for clap's reference recieving default_value, so use OnceCell to be able to pass a static reference.

Reviewed By: krallin

Differential Revision: D24761026

fbshipit-source-id: 81b5e7ceb832d5cb55cc9faef59c5e6432f7c4b0
2020-12-02 07:27:23 -08:00
Meyer Jacobs
293053b774 edenapi: expose "attributes" parameter in tree request API
Summary:
Introduce a new API type, `TreeAttributes`, corresponding to the existing type `WireTreeAttributesRequest`, which exposes which optional attributes are available for fetching. An `Option<TreeAttributes>` parameter is added to the `trees` API, and if set to `None`, the client will make a request with TreeAttributes::default().

The Python bindings accept a dictionary for the attributes parameter, and any fields present will overwrite the default settings from TreeAttributes::default(). Unrecognized attributes will be silently ignored.

Reviewed By: kulshrax

Differential Revision: D25041255

fbshipit-source-id: 5c581c20aac06eeb0428fff42bfd93f6aecbb629
2020-12-01 19:07:25 -08:00
Alex Hornby
a2247dc41c mononoke: rename MononokeApp to MononokeAppBuilder
Summary:
It has a build() method and later in stack it will build a mononoke
specific type rather than the clap::App

Differential Revision: D25216827

fbshipit-source-id: 24a531856405a702e7fecf54d60be1ea3d2aa6e7
2020-12-01 11:45:00 -08:00
Kostia Balytskyi
e4dab84619 scuba: turn ScubaSampleBuilderExt into a wrapper struct
Summary:
This diff prepares the Mononoke codebase for composition-based extendability of
`ScubaSampleBuilder`. Specifically, in the near future I will add:
- new methods for verbose scuba logging
- new data field (`ObservabilityContext`) to check if verbose logging should
  be enabled or disabled

The higher-level goal here is to be able to enable/disable verbose Scuba
logging (either overall or for certain slices of logs, like for a certain
session id) in real time, without restarting Mononoke. To do so, I plan to
expose the aforementioned verbose logging methods, which will run a check
against the stored `ObservabilityContext` and make a decision of whether the
logging is enabled or not. `ObservabilityContext` will of course hide
implementation details from the renamed `ScubaSampleBuilderExt`, and just provide a yes/no
answer based on the current config and sample fields.

At the moment this should be a completely harmless change.

Reviewed By: krallin

Differential Revision: D25211089

fbshipit-source-id: ea03dda82fadb7fc91a2433e12e220582ede5fb8
2020-11-30 21:26:24 -08:00
Thomas Orozco
8840f92d07 edenapi server: properly deserialize history requests
Summary:
While trying to repro a user report (https://fburl.com/jqvm320o), I ran into a
new hg error: P151186623, i.e.:

```
KeyError: 'Key not found HgId(Key { path: RepoPathBuf("fbcode/thrift/facebook/test/py/TARGETS"), hgid: HgId("55713728544d5955703d604299d77bb1ed50c62d") })'
```

After some investigation (and adding a lot of prints), I noticed that this
was trying to query the EdenAPI server for this filenode. That request should
succeed, given Mononoke knows about this filenode:

```
[torozco@devbig051]~/fbcode % mononoke_exec mononoke_admin fbsource --use-mysql-client filenodes by-id fbcode/thrift/facebook/test/py/TARGETS 55713728544d5955703d604299d77bb1ed50c62d
mononoke_exec: Using config stage prod (set MONONOKE_EXEC_STAGE= to override)
I1126 08:10:02.089614 3697890 [main] eden/mononoke/cmdlib/src/args/mod.rs:1097] using repo "fbsource" repoid RepositoryId(2100)
I1126 08:10:02.995172 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:137] Filenode: HgFileNodeId(HgNodeHash(Sha1(55713728544d5955703d604299d77bb1ed50c62d)))
I1126 08:10:02.995282 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:138] -- path: FilePath(MPath("fbcode/thrift/facebook/test/py/TARGETS"))
I1126 08:10:02.995341 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:139] -- p1: Some(HgFileNodeId(HgNodeHash(Sha1(ccb76adc7db0fc4a395be066fe5464873cdf57e7))))
I1126 08:10:02.995405 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:140] -- p2: None
I1126 08:10:02.995449 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:141] -- copyfrom: None
I1126 08:10:02.995486 3697890 [main] eden/mononoke/cmds/admin/filenodes.rs:142] -- linknode: HgChangesetId(HgNodeHash(Sha1(dfe46f7d6cd8bc9b03af8870aca521b1801126f0)))
```

Turns out, the success rate for this method is actually 0% — out of thousands of requests,
not a single one succeeded :(

https://fburl.com/scuba/edenapi_server/cma3c3j0

The root cause here is that the server side is not properly deserializing
requests (actually finding that was a problem of its own, I filed T80406893 for this).

If you manage to get it to print the errors, it says:

```
{"message":"Deserialization failed: missing field `path`","request_id":"f97e2c7c-a432-4696-9a4e-538ed0db0418"}
```

The reason for this is that the server side tries to deserialize the request
as if it was a `WireHistoryRequest`, but actually it's a `HistoryRequest`, so
all the fields have different names (we use numbers in `WireHistoryRequest`).

This diff fixes that. I also introduced a helper method to make this a little
less footgun-y and double-checked the other callsites. There is one callsite
right now that looks like it might be broken (the commit one), but I couldn't
find the client side interface for this (I'm guessing it's not implemented
yet), so for now I left it as-is.

Reviewed By: StanislavGlebik

Differential Revision: D25187639

fbshipit-source-id: fa993579666dda762c0d71ccb56a646c20aee606
2020-11-27 02:48:58 -08:00
Stefan Filip
451d6758ec edenapi: add segmented changelog clone endpoints
Summary:
Transfers iddag flat segments along with the head_id that should be use to
rebuild a full fledged IdDag. It also transfers idmap details. In the current
version it only transfers universal commit mappings.

Reviewed By: krallin

Differential Revision: D24808329

fbshipit-source-id: 4de9edcab56b54b901df1ca4be5985af2539ae05
2020-11-11 22:53:38 -08:00
Meyer Jacobs
5a6ef9c5f7 history: introduce separate wire types for history API
Summary: This change adds wire types for the history API in the most straightforward way possible. In a future change, I'll move the `WireHistoryEntry` / `HistoryResponseChunk` conversion logic into the `ToApi` implementation. This implementation also doesn't add per-item errors, or standardize `history` to match the protocol evolution standards used by `trees`.

Reviewed By: kulshrax

Differential Revision: D24342046

fbshipit-source-id: d46403a823f2a1e89ad9d6d2074241d8bfe4810e
2020-10-25 18:39:34 -07:00
Meyer Jacobs
02b7fcc90a trees: use separate type for child metadata entries
Summary:
Removes top-level metadata from `TreeEntry` and add a new, specialized type for carrying child entry metadata, `TreeChildEntry`. This change also fully removes the `revisionstore_types::Metadata` from EdenAPI trees, which is only used on files.

In a follow up change I'll optimize handlings of paths / path segments in `TreeChildEntry` keys. Right now they need to be joined manually.

Reviewed By: kulshrax

Differential Revision: D24434716

fbshipit-source-id: d0739471b1f6cef58b435e10b5fb774bfb08f7f6
2020-10-25 18:39:34 -07:00
Meyer Jacobs
0ccae1cef9 errors: introduce per-item errors to EdenAPI protocol
Summary: Rather than silently dropping entries which cannot be fetched, this change has the `WireTreeEntry` type carry optional error information, allowing it to be (de)serialized to / from `Result<TreeEntry, EdenApiServerError>` instead of a bare `TreeEntry`. Currently, handling of these failures is up to the individual application code, but it might be useful to introduce utility functions to drop failed entries and log errors.

Reviewed By: kulshrax

Differential Revision: D24315399

fbshipit-source-id: 94e4593b77cf2dc12d0dcc93d174c8a4eda95344
2020-10-25 18:39:34 -07:00
Simon Farnsworth
7e06175e61 Make config store always explicit
Summary: In D24447404, I provided some utility functions that allowed me to avoid constructing and/or passing around a ConfigStore. Remove those functions and fix up the code to run.

Reviewed By: krallin

Differential Revision: D24502692

fbshipit-source-id: 742dbc54fbcf735895d6829745b9317af14dfa0b
2020-10-24 06:23:49 -07:00
Simon Farnsworth
00871310a7 Ensure we have only one ConfigStore for the application
Summary: It's designed as a singleton store for normal use - rather than have lots of ways to get different config stores, let's use one global store

Reviewed By: krallin

Differential Revision: D24447404

fbshipit-source-id: 6ecc46351b14794471f654cec98527a11a93d3ef
2020-10-24 06:23:49 -07:00
Meyer Jacobs
120fbd3280 trees: port SCS aux data request method to edenapi
Summary: Introduces fetching of child entry IDs, and child file metadata for a specified tree manifest ID. The aux data lookup will only be performed if `with_file_metadata` is set, which is actually kind of wrong. Instead `with_children` from the wire type should be exposed in the API request type, and `with_*_metadata` should be hidden or used for data other than the child entry `Key`s.

Reviewed By: kulshrax

Differential Revision: D23886678

fbshipit-source-id: 0cba72cea7be47ae3348a406d407a19b60976c0c
2020-10-14 11:12:59 -07:00
Arun Kulshreshtha
5b8a25acd7 edenapi_server: log client username to scuba
Summary: Add a new "user" column to the EdenAPI server's Scuba data.

Reviewed By: krallin

Differential Revision: D24153765

fbshipit-source-id: 95a3177d9283e5b0e3f47e7f42a1de5142049b99
2020-10-07 08:17:59 -07:00
Meyer Jacobs
75105421ce edenapi: Hide edenapi wire types from externally visible API
Summary:
Introduce separate wire types to allow protocol evolution and client API changes to happen independently.

* Duplicate `*Request`, `*Entry`, `Key`, `Parents`, `RepoPathBuf`, `HgId`, and `revisionstore_types::Metadata` types into the `wire` module. The versions in the `wire` module are required to have proper `serde` annotations, `Serialize` / `Deserialize` implementations, etc. These have been removed from the original structs.
* Introduce infallible conversions from "API types" to "wire types" with the `ToWire` trait and fallible conversions from "wire types" to "API types" with the `ToApi`. API -> wire conversions should never fail in a binary that builds succesfully, but wire -> API conversions can fail in the case that the server and client are using different versions of the library. This will cause, for instance, a newly-introduced enum variant used by the client to be deserialized into the catch-all `Unknown` variant on the server, which won't generally have a corresponding representation in the API type.
* Cleanup: remove `*Response` types, which are no longer used anywhere.
* Introduce a `map` method on `Fetch` struct which allows a fallible conversion function to be used to convert a `Fetch<T>` to a `Fetch<U>`. This function is used in the edenapi client implementation to convert from wire types to API types.
* Modify `edenapi_server` to convert from API types to wire types.
* Modify `edenapi_cli` to convert back to wire types before serializing responses to disk.
* Modify `make_req` to use `ToWire` for converting API structs from the `json` module to wire structs.
* Modify `read_res` to use `ToApi` to convert deserialized wire types to API types with the necessary methods for investigating the contents (`.data()`, primarily). It will print an error message to stderr if it encounters a wire type which cannot be converted into the corresponding API type.
* Add some documentation about protocol conventions to the root of the `wire` module.

Reviewed By: kulshrax

Differential Revision: D23224705

fbshipit-source-id: 88f8addc403f3a8da3cde2aeee765899a826446d
2020-09-23 17:27:08 -07:00
Arun Kulshreshtha
3078f346f8 gotham_ext: make StreamBody take an infallible byte stream
Summary: Make `StreamBody` accept a `Stream` of `Bytes` instead of a `TryStream` of `Bytes`. This means that applications returning streaming responses will be forced to deal with errors prior to returning the response.

Reviewed By: krallin

Differential Revision: D23780216

fbshipit-source-id: dbad61947ef23bbfc4edf3d286ad0218c1859d87
2020-09-18 22:49:55 -07:00
Arun Kulshreshtha
71674d87c5 edenapi_server: log errors during streaming responses
Summary: Use the new `ForwardErr` stream combinator to log errors that occur during a streaming response. Right now, they are just printed to stderr, but in the future we should also do other things such as logging them to Scuba. This diff supersedes the approach from D22720957.

Reviewed By: krallin

Differential Revision: D23780215

fbshipit-source-id: 8d2267f1166e665a62a167a6d95bb0b1797b5767
2020-09-18 22:37:42 -07:00
Arun Kulshreshtha
6e5f95067a gotham_ext: move content streams into separate module
Summary:
The `gotham_ext::response` module was getting a bit large, so this diff moves `ContentMeta`, `ContentStream`, and `CompressedContentStream` into a new submodule, alongside the contents of the old `content_encoding` module. This way, the `response` module remains entirely centered around the `TryIntoResponse` trait (and the various body structs that implement that trait).

Later diffs in this stack will be adding an additional layer between the content streams and the body structs, at which point it probably doesn't make sense to have these right next to each other. Splitting them out now will allow for better code organization going forward.

Reviewed By: krallin

Differential Revision: D23777492

fbshipit-source-id: 86e598dcb37578d3b22217a2a65f1bde84d72215
2020-09-18 01:34:04 -07:00
Johan Schuijt-Li
deb57a25ed mononoke: deprecate preamble in favor of metadata
Summary:
In preparation of moving away from SSH as an intermediate entry point for
Mononoke, let Mononoke work with newly introduced Metadata. This removes any
assumptions we now make about how certain data is presented to us, making the
current "ssh preamble" no longer central.

Metadata is primarily based around identities and provides some
backwards-compatible entry points to make sure we can satisfy downstream
consumers of commits like hooks and logs.

Simarly we now do our own reverse DNS resolving instead of relying on what's
been provided by the client. This is done in an async matter and we don't rely
on the result, so Mononoke can keep functioning in case DNS is offline.

Reviewed By: farnz

Differential Revision: D23596262

fbshipit-source-id: 3a4e97a429b13bae76ae1cdf428de0246e684a27
2020-09-15 10:28:38 -07:00
Thomas Orozco
acc0d428cc mononoke/lfs_server: add support for response compression
Summary:
This adds support for compressing responses in the LFS Server, based on what
the client sent in `Accept-Encoding`. The compression changes are fairly
simple. Most of the codes changes are around the fact that when we compress,
we don't send a Content-Length (because we don't know how long the content will
be).

Note that this is largely implemented in StreamBody. This means it can be used
for free by the EdenAPI server as well. The reason it's in there is because we
need to avoid setting the Content-Length when compression is going to be used
(`StreamBody` is what takes charge for doing this). This also exposes a
callback to get access to the stream post-compression, which also needs to be
exposed in `StreamBody`, since that's where compression happens.

Reviewed By: aslpavel

Differential Revision: D23652334

fbshipit-source-id: 8f462d69139991c3e1d37f392d448904206ec0d2
2020-09-15 07:59:53 -07:00
Arun Kulshreshtha
7904099f13 edenapi_server: Add EdenAPI-specific columns to Scuba samples
Summary: Add a Scuba handler for the EdenAPI server, which will allow the server to log custom columns to Scuba via `ScubaMiddleware`. For now, the only application specific columns are `repo` and `method`.

Reviewed By: krallin

Differential Revision: D23619437

fbshipit-source-id: f08aaf9c84657b4d92f1a1cfe5f0cb5ccd408e5e
2020-09-10 20:57:12 -07:00
Arun Kulshreshtha
cec871cc9f edenapi_server: log stats to ODS
Summary: Add `OdsMiddleware` to the EdenAPI server to log various aggregate request stats to ODS. This middleware is directly based on the `OdsMiddleware` from the LFS server (which unfortunately can't be easily generalized due to the way in which the `stats` crate works).

Reviewed By: krallin

Differential Revision: D23619075

fbshipit-source-id: d361c73d18e0d1cb57347fd24c43bdb68fb7819d
2020-09-10 20:57:12 -07:00
Arun Kulshreshtha
25af6e81d7 edenapi_server: add HandlerInfo
Summary:
Add a new `HandlerInfo` struct that can be inserted into the request's `State` by each handler to provide information about the handler to post-request middleware. (This includes things like which handler ran, which repo was queried, etc.)

A note on design: I did consider adding these fields to `RequestContext` (similar to how it's done in the LFS server), but that proved somewhat problematic in that it would require using interior mutability to avoid forcing each handler to borrow the `RequestContext` mutably (and therefore prevent it from borrowing other state data). The current approach seems preferable to adding a `Mutex` inside of `RequestContext`.

Reviewed By: krallin

Differential Revision: D23619077

fbshipit-source-id: 911806b60126d41e2132d1ca62ba863c929d4dc9
2020-09-10 20:57:12 -07:00
Arun Kulshreshtha
54641e8d1b edenapi_server: remove extraneous fields/methods from RequestContext
Summary: Remove unused fields and superfluous methods from `RequestContext`.

Reviewed By: singhsrb

Differential Revision: D23619076

fbshipit-source-id: 0fc42d6c29a8bb5c197d3559baa497a9e6e9c825
2020-09-09 22:43:31 -07:00
Arun Kulshreshtha
8a26c3c960 edenapi_server: add Scuba logging
Summary: Add Scuba logging using `ScubaMiddleware` from `gotham_ext`. Each request will be logged to the Scuba dataset specified by the `--scuba-dataset` flag, as well as optionally to the log file specified by `--scuba-log-file`.

Reviewed By: sfilipco

Differential Revision: D23547668

fbshipit-source-id: e6cd88ad729a40cf45b63538f7481ee098ea12dc
2020-09-07 17:24:45 -07:00
Arun Kulshreshtha
5556a447d1 edenapi_server: use LogMiddleware
Summary: Add `LogMiddleware` to the EdenAPI server, which will print a log message whenever a request is received or has completed.

Reviewed By: DurhamG

Differential Revision: D23299902

fbshipit-source-id: f44ef1b01692f0e4f9b109917fcee89a84ca4208
2020-09-03 11:59:32 -07:00
Arun Kulshreshtha
96a6a3fcfb edenapi_server: use LoadMiddleware
Summary: Use `LoadMiddleware` to track the number of outstanding requests in the server.

Reviewed By: DurhamG

Differential Revision: D23298415

fbshipit-source-id: bdcdb0f657d8deac593d356c87ac0d8d3f39e322
2020-09-03 11:59:32 -07:00
Stefan Filip
932450fb15 handlers: update location-to-hash endpoint with count parameter
Summary:
To reduce the size over the wire on cases where we would be traversing the
changelog on the client, we want to allow the endpoint to return a whole parent
chain with their hashes.

Reviewed By: kulshrax

Differential Revision: D23456216

fbshipit-source-id: d048462fa8415d0466dd8e814144347df7a3452a
2020-09-02 17:20:42 -07:00
Stefan Filip
7122cdded7 types: rename Location to CommitLocation
Summary:
Renaming all the LocationToHash related structures to CommitLocationToHash.
This is done for consistency. I realized the issue when the command for reading
the request from cbor was not what I was expecting it to be. The reason was that
the commit prefix was used inconsistently for LocationToHash.

Reviewed By: kulshrax

Differential Revision: D23456221

fbshipit-source-id: 0181dcaf81368b978902d8ca79c5405838e4b184
2020-09-02 17:20:42 -07:00