Summary:
Matches the `getcommitdata` SSH endpoint.
This is going to be used to remove the requirement that client repostories
need to have all commits locally.
Reviewed By: krallin
Differential Revision: D22979458
fbshipit-source-id: 75d7265daf4e51d3b32d76aeac12207f553f8f61
Summary:
Eden api endpoint for segmented changelog. It translates a path in the
graph to the hash corresponding to that commit that the path lands on.
It is expected that paths point to unique commits.
This change looks to go through the plumbing of getting the request from
the edenapi side through mononoke internals and to the segmented changelog
crate. The request used is an example. Follow up changes will look more at
what shape the request and reponse should have.
Reviewed By: kulshrax
Differential Revision: D22702016
fbshipit-source-id: 9615a0571f31a8819acd2b4dc548f49e36f44ab2
Summary: Allow the user to specify extra HTTP headers that should be sent with each EdenAPI request in the `edenapi.headers` config option. The field is expected to be a JSON object whose key-value pairs are used as header keys and values.
Reviewed By: quark-zju
Differential Revision: D22591870
fbshipit-source-id: ac1bb669270d667895554dcc5f7176d18736375c
Summary: Include the name of bad config fields in the error message so the user can more easily fix the problem.
Reviewed By: quark-zju
Differential Revision: D22591871
fbshipit-source-id: e23e2c71e49e0458e7ea5c13e7feac3a990ead0c
Summary: Update `edenapi::Builder` to use the CA certificate bundle specified in the `[auth]` section of the user's config.
Reviewed By: quark-zju
Differential Revision: D22591034
fbshipit-source-id: 3a417adbf50ef7d2c538f4a032e54a038cbd282e
Summary:
Add a metadata field to `read_res` containing a `revisionstore::Metadata` struct (which contains the object size and flags). The main purpose of this is to support LFS, which is indicated via a metadata flag.
Although this change affects the `DataEntry` struct which is serialized over the wire, version skew between the client and server should not break things since the field will automatically be populated with a default value if it is missing in the serialized response, and ignored if the client was built with an earlier version of the code without this field.
In practice, version skew isn't really a concern since this isn't used in production yet.
Reviewed By: quark-zju
Differential Revision: D22544195
fbshipit-source-id: 0af5c0565c17bdd61be5d346df008c92c5854e08
Summary: Instead of restricting the allowed characters in a repo name, allow any UTF-8 string. The string will be percent-encoded before being used in URLs.
Reviewed By: quark-zju
Differential Revision: D22559830
fbshipit-source-id: f9caa51d263e06d424531e0947766f4fd37b035f
Summary: We have several repos whose names contain various non-alphanumeric/underscore/hyphen characters, so we need to be more permissive about accepting repo names.
Reviewed By: quark-zju
Differential Revision: D22554846
fbshipit-source-id: e7bb030e0b8fb6aa275c119ba0aa540405b29186
Summary: When working with large CBOR responses, it is sometimes useful to limit processing to the first N entries to prevent the operation from taking a long time. This diff adds an option to the `read_res` tool to only look at the first N entries in a data or history response.
Reviewed By: quark-zju
Differential Revision: D22544451
fbshipit-source-id: 5e8e2c7212aa3b315a25bd4cf9273009a5e43f72
Summary: Ensure that all of the components of an EdenAPI response use the same error type.
Reviewed By: quark-zju
Differential Revision: D22443029
fbshipit-source-id: 3e00a8b83677beb5ef2d90630fe9b85760874186
Summary:
D22396026 made it so that `HttpClient::send_async` no longer consumes `self`. This means that instead of creating a new HTTP client for each request, we can reuse the same one.
This has the benefit of allowing for connection reuse (which was the point of D22396026), resulting in lower latency for serial fetches.
Reviewed By: quark-zju
Differential Revision: D22397768
fbshipit-source-id: 9d066c1ec64a6aa1b36ec674ef294030c1f90b41
Summary: Allow passing multiple JSON requests to the EdenAPI CLI. The requests will be performed serially, which allows for testing the performance of serial EdenAPI calls.
Reviewed By: quark-zju
Differential Revision: D22397769
fbshipit-source-id: c59e5abf53eee9c2014010672183e202b6f180fc
Summary: Update the `revisionstore` and `backingstore` crates to use the new EdenAPI crate.
Reviewed By: quark-zju
Differential Revision: D22378330
fbshipit-source-id: 989f34827b744ff4b4ac0aa10d004f03dbe9058f
Summary: Add a new `EdenApiBlocking` trait that exposes blocking versions of the `EdenApi` trait's methods, for use in non-async code.
Reviewed By: quark-zju
Differential Revision: D22305396
fbshipit-source-id: d0f3a73cad1a23a4f0892a17f18267374e63108e
Summary:
This diff adds an EdenAPI CLI program that allows manually sending requests to the server.
Requests are read from stdin in a JSON format (the same format used by the `make_req` tool and the EdenAPI server integration tests). This makes it easy to create and edit requests during debugging.
Responses are re-serialized as CBOR and written to stdout. (The program will refuse to write output if stdout is a TTY.) These responses can then be analyzed using the `read_res` tool (also used by the EdenAPI server integration tests).
The program prints real-time download statistics during data fetching, allow the user to debug performance in addition to correctness.
The program uses standard `hgrc` files to configure the EdenAPI client, which means that one can simulate production settings by specifying a production `hgrc`. By default, it will read from `~/.hgrc.edenapi` rather than `~/.hgrc` since the user will most likely want to configure this program independently of Mercurial.
Reviewed By: quark-zju
Differential Revision: D22370163
fbshipit-source-id: 5d9974bc05fa960d26cd2c87810f4646e2bc55b4
Summary:
This diff is a complete, ground-up rewrite of the EdenAPI client. Rather than attempting to use `libcurl` directly, it relies on the new `http_client` crate, which makes the code considerably simpler and allows for a proper async interface.
The most notable change is that `EdenApi` is now an async trait. A blocking API is added later in the stack for use in non-async contexts.
Reviewed By: quark-zju
Differential Revision: D22305397
fbshipit-source-id: 4c1e5d3091d6dd04cf13291e7b7a4217dfdd249f
Summary: Use `thiserror` to provide a more ergonomic API for `DataEntry` hash checking. The `.data()` method now simply returns a `Result` rather than a tuple with an ad-hoc enum.
Reviewed By: quark-zju
Differential Revision: D22376164
fbshipit-source-id: fc39cb212ec1ee5830292db4aa5eca18f2c16a2b
Summary: Tidy up some comments in this file.
Reviewed By: ikostia
Differential Revision: D22376165
fbshipit-source-id: ce4760776048aa8e72809b4f828d0ea426fcf878
Summary: Add a `ToJson` trait as a counterpart to the `FromJson` trait introduced in the last diff. The primary use case for this will be to allow recording and replaying the data fetches that occur during Mercurial operations. A JSON representation was chosen so that the format will be directly compatible with tools like `make_req` used in EdenAPI integration tests.
Reviewed By: markbt
Differential Revision: D22344599
fbshipit-source-id: 52c888bde93a8e86b6dd76cb862337f716b007eb
Summary:
Add a `FromJson` trait to `edenapi_types` and use it instead of the parsing functions when parsing requests from JSON.
Some design commentary:
I've created a custom trait rather than using `TryFrom<serde_json::Value>` for two reasons:
- From a design standpoint, I'd like users to have to explicitly opt-in to this functionality by importing this `FromJson` trait, since this is different from the usual way of deserializing with serde, and it might cause confusion.
- From an implementation standpoint, it turns out that using `TryFrom` in a trait bound causes difficulties with type inference in some situations (in particular, around the associated `Error` type), which necessitates type annotations, making the API less ergonomic to use.
Why not just use `serde::Deserialize` directly?
- The representation here doesn't actually directly match the structure of the underlying type. Instead, the JSON representation has been designed to be easy for humans to edit when writing tests. (For example, keys are replaced with simply arrays, and hashes are represented as hex rather than as byte arrays.)
- In this case, we'd like to work with `serde_json::Value` rather than going directly between bytes and structs, since this allows for a level of dynamism that will be useful later in the stack.
Reviewed By: markbt
Differential Revision: D22320576
fbshipit-source-id: 64b6bed42e1ec599a0da61ae5d55feb7c90101a4
Summary: Add `Arbitrary` implementations for `DataRequest`, `HistoryRequest`, and `CompleteTreeRequest` so that these types can be used in quickcheck tests.
Reviewed By: quark-zju
Differential Revision: D22344600
fbshipit-source-id: c7fcbcd4648ab45f8dde00cc4bb3c1c4d203c4e1
Summary: The `#[quickcheck]` attribute is a more modern way of defining quickcheck predicates (as opposed to the older `quickcheck!` macro). Update all existing usages in the crate.
Reviewed By: quark-zju
Differential Revision: D22344601
fbshipit-source-id: b7ee4d317a64bed5fcd8b35f90f2544f6b024410
Summary: Replace use of `.ok_or_else(|| anyhow!(...))?` with `.context(...)?`, which is a cleaner way to express the same thing.
Reviewed By: quark-zju
Differential Revision: D22323298
fbshipit-source-id: 9fc1a8183a54ee0c4f30f7497b98005a18a06468
Summary:
Move JSON request parsing code out of `make_req` into `edenapi_types` so it can be reused in the EdenAPI test CLI (added later in the stack).
Note to reviewer: This diff look bigger than it actually is; it is mostly just cut-and-paste (recorded as a copy to preserve history).
Reviewed By: quark-zju
Differential Revision: D22305693
fbshipit-source-id: 13248fb29b2fbb705203f889f3d2fb3c1c760bed
Summary:
EdenAPI's `make_req` tools allows developers to create ad-hoc CBOR request payloads for debugging purposes (e.g., for use with `curl`). The tool generates requests from human-created JSON, which are particularly useful in Mercurial and Mononoke's integration tests.
Later in this stack, the use of this JSON format will be extended beyond just this one tool. As such, it is important that the representation be sufficiently extensible so accommodate future changes to the request structs. In the case of the JSON representation of `DataRequest`, this means changing from an array to a single-attribute object, so that additional fields can potentially be added in the future.
Reviewed By: quark-zju
Differential Revision: D22319314
fbshipit-source-id: 5931bc7ab01ca48ceab5ffd1c9177dd3035b643c
Summary: Use autocargo for all EdenAPI code.
Reviewed By: quark-zju
Differential Revision: D22344400
fbshipit-source-id: 522e82fcc76792ed01ca2cdbfa169c23be5bf38f
Summary: Move old EdenAPI crate to `scm/lib/edenapi/old` to make room for the new crate. This old code will eventually been deleted once all references to it are removed from the codebase.
Reviewed By: quark-zju
Differential Revision: D22305173
fbshipit-source-id: 45d211340900192d0488543ba13d9bf84909ce53
Summary: Replace the term 'blacklisted' with 'redacted'.
Reviewed By: quark-zju
Differential Revision: D22142842
fbshipit-source-id: 461faa09a5d3958a4575c48f03e2286eec917681
Summary: Add a crate-level doc comment clarifying the purpose of this crate. In particular, the motivation behind moving some of EdenAPI's types into this crate was to make it easier to share types between the server and client. As such, types that are not shared (which may pull in more complex dependencies and therefore cause build issues for either the server or client) should not be put in this crate.
Reviewed By: quark-zju
Differential Revision: D22142180
fbshipit-source-id: 7258ef33b73a87acf72d4f6bcbe8b27cbc361735
Summary:
Remove unused dependencies for Rust targets.
This failed to remove the dependencies in eden/scm/edenscmnative/bindings
because of the extra macro layer.
Manual edits (named_deps) and misc output in P133451794
Reviewed By: dtolnay
Differential Revision: D22083498
fbshipit-source-id: 170bbaf3c6d767e52e86152d0f34bf6daa198283
Summary:
Rename the `subtree` endpoint on the EdenAPI server to `complete_trees` to better express what it does (namely, fetching complete trees, in contrast to the lighter weight `/trees` endpoint that serves individual tree nodes). This endpoint is not used by anything yet, so there isn't much risk in renaming it at this stage.
In addition to renaming the endpoint, the relevant request struct has been renamed to `CompleteTreeRequest` to better evoke its purpose, and the relevant client and test code has been updated accordingly. Notably, now that the API server is gone, we can remove the usage of this type from Mononoke's `hgproto` crate, thereby cleaning up our dependency graph a bit.
Reviewed By: krallin
Differential Revision: D22033356
fbshipit-source-id: 87bf6afbeb5e0054896a39577bf701f67a3edfec
Summary: Make most of the handlers on the EdenAPI server return streaming responses.
Reviewed By: krallin
Differential Revision: D22014652
fbshipit-source-id: 177e540e1372e7dfcba73c594614f0225da3a10f
Summary:
Note: Although this diff looks big, it's all just cutting and pasting code.
Restructure the crate so that the types relevant to each EdenAPI endpoint are grouped in their own module. This means that authors of new endpoints can just add a new module and put their types there, rather than having to make edits throughout the crate.
Reviewed By: quark-zju
Differential Revision: D21983470
fbshipit-source-id: 4901662dc6b5f0a1feca1ab8d914f394114ef233
Summary: Change `HistoryResponse` so that instead of repeatedly sending the file path alongside every history entry, we group the history entries by file and send the path once per group. This will prevent the server from transmitting redundant information.
Reviewed By: quark-zju
Differential Revision: D21982558
fbshipit-source-id: f1c5d2573c97940c7bf1645ed7fef6e1887c0d42
Summary:
These particular unit tests essentially just verify that these types (which are trivial wrapper structs around `Vec`s) implement `IntoIterator`. As such, they don't really provide value (the `IntoIterator` implementations are trivial*) but require updates whenever the format of the items produced by the iterators changes. This adds a maintenance burden with little value to show for it, so let's just get rid of these.
*The only thing non-trivial is that for `HistoryResponse`, the iterator does convert the entries from `WireHistory` to `HistoryEntry`, but the actual logic of this conversion is tested separately in `api.rs`, so testing it here is redundant.
Reviewed By: quark-zju
Differential Revision: D21983471
fbshipit-source-id: beebad53abb9ce5801b1738af92f250405c50990
Summary: D21880220 renamed the `depth` parameter in Mononoke's history fetching functions to be `length`. This diff makes the same change for EdenAPI's `HistoryRequest` struct.
Reviewed By: StanislavGlebik
Differential Revision: D21948599
fbshipit-source-id: fe8649a5789f07d8262ad3d5e2f477a8b50f2c6f
Summary:
Several of the structs used by EdenAPI were previously defined in Mercurial's
`types` crate. This is not ideal since these structs are used for data interchange
between the client and server, and changes to them can break Mononoke, Mercurial,
or both. As such, they are more fragile than other types and their use should be
limited to the EdenAPI server and client to avoid the need for extensive breaking
changes or costly refactors down the line.
I'm about to make a series of breaking changes to these types as part of the
migration to the new EdenAPI server, so this seemed like an ideal time to split
these types into their own crate.
Reviewed By: krallin
Differential Revision: D21857425
fbshipit-source-id: 82dedc4a2d597532e58072267d8a3368c3d5c9e7
Summary: Previously, `read_res` was called `data_util` and only dealt with EdenAPI data responses. Support for history responses was added later as a `history` subcommand. For consistency, let's move the top-level commands for data responses underneath a new `data` subcommand. When support for addition response types is added in the future, those can also go under their own subcommands.
Reviewed By: quark-zju
Differential Revision: D21825197
fbshipit-source-id: f5cb759a68324e7d0f98e3448bd5d1cba6417bad
Summary: Give this tool a more descriptive name. (It reads EdenAPI responses, so `read_res` seemed fitting.)
Reviewed By: quark-zju
Differential Revision: D21796964
fbshipit-source-id: 8a4ee365aa3bcf115fc7a3452406ed96b4a25edc
Summary: In line with other crates that contain utility binaries alongside the crate, rename the `edenapi/utils` directory to `edenapi/tools`.
Reviewed By: quark-zju
Differential Revision: D21796899
fbshipit-source-id: 058319e2756b1d596f06d6e57d17a6c07a7f1c9c
Summary: Added comments showing the expected JSON input format for each kind of EdenAPI request.
Reviewed By: xavierd
Differential Revision: D21782765
fbshipit-source-id: bf08dd4b36a33aca506eb0fa0341e40d0150d7cb
Summary: Update the JSON format for history requests to use an array rather than an object to represent keys, for the same reason as D21412989. (Namely, that it's possible for two keys to share the same path, making the path unsuitable for use as a field name in a JSON object.)
Reviewed By: xavierd
Differential Revision: D21782763
fbshipit-source-id: eb04013795d1279ecbf00a8a0be106318695bd05
Summary: Add the ability to parse EdenAPI history responses to `data_util`.
Reviewed By: quark-zju
Differential Revision: D21489228
fbshipit-source-id: 42dda64273673431a6f3e4d7bd430689c76c387f
Summary: Change `make_req` to take a JSON array as input when constructing `DataRequest`s instead of a JSON object. This is more correct because DataRequests can include multiple `Key`s with the same path; this cannot be represented as an object since an object is effectively a hash map wherein we would have duplicate keys.
Reviewed By: quark-zju
Differential Revision: D21412989
fbshipit-source-id: 07a092a15372d86f3198bea2aa07b973b1a8449d
Summary:
Add two utility programs for ad-hoc debugging of EdenAPI. EdenAPI requests and responses are encoded as CBOR, which is not easy to work with manually on the command line. In order to allow debugging the HTTP API using tools like `curl`, we need tools that can generate raw request payloads and interpret CBOR responses.
The utility programs included in this diff are:
- `make_req` - Can construct EdenAPI request payloads from a human-editable JSON representation.
- `data_util` - Can list, validate, and extract the contents of an EdenAPI data response.
These tools can be used by themselves or as part of a pipeline. See test plan for examples.
Reviewed By: xavierd
Differential Revision: D21136575
fbshipit-source-id: d1ac8d92964614005078a6ac76dd0835c29a80a5
