Summary:
create end to end intergation for the lookup API on the client
Start prototyping of `hg cloud upload` command.
Currently, it just performs lookup for existing heads.
This way we can end to end test the new APIs.
Reviewed By: markbt
Differential Revision: D28848205
fbshipit-source-id: 730c1ed4a21c1559d5d9b54d533b0cf551c41b9c
Summary:
Files upload will be executed in 2 stages:
* check if content is already present
* upload missing files
The check api is generic, it could be used for any id type. Called 'lookup' API.
Reviewed By: markbt
Differential Revision: D28708934
fbshipit-source-id: 654c73b054790d5a4c6e76f7dac6c97091a4311f
Summary:
Time 0.2 is current, and 0.1 is long obsolete. Unfortunately there's a
large 0.1 -> 0.2 API change, so I preserved 0.1 and updated the targets of its
users. Also unfortunate that `chrono` has `oldtime` as a default feature, which
makes it use `time-0.1`'s `Duration` type. Excluding it from the features
doesn't help because every other user is specifying it by default.
Reviewed By: dtolnay
Differential Revision: D28854148
fbshipit-source-id: 0c41ac6b998dfbdcddc85a22178aadb05e2b2f2b
Summary:
Like it says in the title. The API between Bytes 1.x has changed a little bit,
but the concepts are basically the same, so we just need to change the
callsites that were calling `bytes()` and have them ask for `chunk()` instead.
This diff attempts to be as small as it can (and it's already quite big). I
didn't attempt to update *everything*: I only updated whatever was needed to
keep `common/rust/tools/scripts/check_all.sh` passing.
However, there are a few changes that fall out of this. I'll outline them here:
## `BufExt`
One little caveat is the `copy_to_bytes` we had on `BufExt`. This was
introduced into Bytes 1.x (under that name), but we can't use it here directly.
The reason we can't is because the instance we have is a `Cursor<Bytes>`, which
receives an implementation of `copy_from_bytes` via:
```
impl<T: AsRef<[u8]>> Buf for std::io::Cursor<T>
```
This means that implementation isn't capable of using the optimized
`Bytes::copy_from_bytes` which doesn't do a copy at all. So, instead, we need
to use a dedicated method on `Cursor<Bytes>`: `copy_or_reuse_bytes`.
## Calls to `Buf::to_bytes()`
This method is gone in Bytes 1.x, and replaced by the idiom
`x.copy_to_bytes(x.remaining())`, so I updated callsites of `to_bytes()`
accordingly.
## `fbthrift_ext`
This set of crates provides transports for Thrift calls that rely on Tokio 0.2
for I/O. Unfortunately, Tokio 0.2 uses Bytes 0.5, so that doesn't work well.
For now, I included a copy here (there was only one required, when reading from
the socket). This can be removed if we update the whole `fbthrift_ext` stack to
Bytes 1.x. fanzeyi had been wanting to update this to Tokio 1.x, but was blocked on `thrift/lib/rust` using Bytes 0.5, and confirmed that the overhead of a copy here is fine (besides, this code can now be updated to Tokio 1.x to remove the copy).
## Crates using both Bytes 0.5 & Bytes 1.x
This was mostly the case in Mononoke. That's no coincidence: this is why I'm
working on this. There, I had to make changes that consist of removing Bytes
0.5 to Bytes 1.x copies.
## Misuse of `Buf::bytes()`
Some places use `bytes()` when they probably mean to use `copy_to_bytes()`. For
now, I updated those to use `chunk()`, which keeps the behavior the same but
keeps the code buggy. I filed T91156115 to track fixing those (in all
likelihood I will file tasks for the relevant teams).
Reviewed By: dtolnay
Differential Revision: D28537964
fbshipit-source-id: ca42a614036bc3cb08b21a572166c4add72520ad
Summary:
This makes it easier to see what builder functions were registered:
% EDENSCM_LOG=edenapi=debug lhg log -r .
May 06 16:40:29.355 DEBUG edenapi::builder: registered eagerepo::api::edenapi_from_config to edenapi Builder
Reviewed By: DurhamG
Differential Revision: D28271366
fbshipit-source-id: f6c7c3aa9f29c3e47c2449e3d5fc16474aa338b0
Summary:
The trees endpoint is another example where we try to send errors to the
client. As it was done previously we would fail to log any errors on the
server side. This diff corrects that by using custom_cbor_stream.
Reviewed By: kulshrax
Differential Revision: D28111102
fbshipit-source-id: 468095d024647f472b8ad9a9e17ca8364605ff98
Summary:
eagerepo -> metalog -> git2 -> libgit2-sys -> libgit2 conflicts with edenfs'
non-Rust libgit2 dependency. Rust git2 crate does not seem to provide a way to
depend on specified libgit2.
Quote https://github.com/rust-lang/git2-rs/issues/263#issuecomment-450934287:
> It's expected that git2-rs builds its own copy of libgit2 and doesn't use the
> system version, as the system version is likely incompatible
It also seems non-trivial to make buck C++ use the libgit2 frm `libgit2-sys` crate.
Let's just avoid depending on eagerepo from edenapi directly for now to solve the
issue. This basically revives D27948369 and D27951632.
Reviewed By: xavierd
Differential Revision: D28243784
fbshipit-source-id: 0c38c20c2d3a80c550732129da572fe26a229799
Summary: Add a way to fetch tree content without going through store.
Reviewed By: liubov-dmitrieva
Differential Revision: D28200387
fbshipit-source-id: 8f5b2214aafba39c7674f0f6b27af0c985f0ea72
Summary:
Now EdenApi trait is moved to a separate crate, we can inline the EdenApi
backed by EagerRepo without using dynamic registration functions.
Reviewed By: andll
Differential Revision: D28006553
fbshipit-source-id: 427513da94db228745b1a7e90af0e62296056128
Summary:
This will be used to migrate "pull" from bundle2 to EdenApi.
This diff just defines the interface in a minimal way.
Actual implementation is TBD.
Differential Revision: D27951634
fbshipit-source-id: 66210a833d8fc87452a19d1935e9d208a1d31b14
Summary:
This will be used to migrate "pull" from bundle2 to EdenApi.
This diff just defines the interface in a minimal way. Actual implementation is TBD.
I dropped the progress callback parameter since the native Rust progress bars
make it unnecessary. I also avoided the blocking API interface, since we would
like pure Rust code to not call blocking APIs which is likely a mistake, and
only use `async_runtime::block_on` in Python bindings - in that case blocking
API is not needed.
Reviewed By: andll
Differential Revision: D27951624
fbshipit-source-id: 3844dd96df265cc6e61d7cf5e79f39c891e8117d
Summary: This makes CloneData possible to represent an empty repo.
Reviewed By: sfilipco
Differential Revision: D27926246
fbshipit-source-id: 0bcead224ef5b89c66d07a34d8217edaef62177f
Summary:
This allows an external crate C that implements `EdenApi` to depend on a more
lightweight library just providing `EdenApi` without things like `hg_http`.
Then the `edenapi` crate can depend on C too.
Didn't move it to `edenapi_types` because it would add extra dependencies
(http, http_client, auth, etc.)
Reviewed By: kulshrax
Differential Revision: D28006548
fbshipit-source-id: 6e828974fd3f78fec70d4a04ae7be85abc459b36
Summary:
The `Builder` API is the main API used by external users to obtain an `EdenApi`
client. In the future we want to support different kinds of `EdenApi`, like a
local repo serving it, if `paths.default` is set to something like
`myrepotype:path`. Make `Builder` more flexible to support non-HTTP `EdenApi`s,
by returning `EdenApi` trait objects.
The old builder that is coupled with HTTP is renamed to HttpClientBuilder.
Reviewed By: kulshrax
Differential Revision: D28018586
fbshipit-source-id: 1eff7bbb8f0e5521a9bcf5a225ac361ddf7c310f
Summary:
This ensures the User-Agent is always set. It also makes the `header` less
unnecessary.
Reviewed By: DurhamG
Differential Revision: D28018587
fbshipit-source-id: 1125d2122431579f127e81c4713de45135b1f972
Summary:
Make it easier to use.
This makes it easier for other crates to use `edenapi::Result<T>`, which is
a bit shorter than `Result<T, EdenApiError>`. Also re-export `Metadata`
from revisionstore-types so callsite does not need to depend on
revisionstore-types explicitly.
Reviewed By: kulshrax
Differential Revision: D27926250
fbshipit-source-id: c85198b5c151e10a2d4d2567e23e32605a3e7c36
Summary:
New endpoint. This endpoint can be used for prefix lookup and the contains
check.
Reviewed By: quark-zju
Differential Revision: D28034533
fbshipit-source-id: d724b85c3816414475b142215e3052d0b555cf59
Summary:
These structures are going to be used to implement the `commit/hash_lookup`
endpoint in EdenApi.
Reviewed By: quark-zju
Differential Revision: D28034532
fbshipit-source-id: 7b00d0d97dd0593dfa43834cda9fc9e9ab9021c5
Summary:
Generic container for a bunch of uniform objects. This is primarily intended
for requests and responses which can be difficult to evolve when the top level
object is an array. For cases where evolution is required we would
probably replace the Batch wrapper with a specialized type. For example,
starting from `Batch<MyRequest>` we would change to:
struct MyRequestBatch {
pub batch: Vec<T>,
pub evolution: Evolution,
}
Reviewed By: quark-zju
Differential Revision: D28034534
fbshipit-source-id: d231c063eeacf3500b75ae76bcc101ccbcda8881
Summary: Add bookmark method to Edenapi client for use in Mercurial client.
Reviewed By: kulshrax
Differential Revision: D27174441
fbshipit-source-id: cdc324e9115e87eab2e078209bbbc266e4e1dcdc
Summary:
It is considered an anti-pattern (https://rust-unofficial.github.io/patterns/anti_patterns/deny-warnings.html)
and is causing Github CI breakage unnecessarily (https://github.com/facebookexperimental/eden/runs/2402094456):
error: function is never used: `example_blob`
--> src/lfs.rs:1860:8
|
1860 | fn example_blob() -> (Sha256, usize, Bytes) {
| ^^^^^^^^^^^^
|
note: the lint level is defined here
--> src/lib.rs:125:9
|
125 | #![deny(warnings)]
| ^^^^^^^^
= note: `#[deny(dead_code)]` implied by `#[deny(warnings)]`
Reviewed By: andll
Differential Revision: D27911477
fbshipit-source-id: df8d642fe74fe311eb0f329d977b9b8270c27bf4
Summary:
Integration test showing that MismatchedHeads errors are sent over the wire to
the client.
Reviewed By: quark-zju
Differential Revision: D27798555
fbshipit-source-id: b14a213e9055486bf07ecbb4b5453385df701b48
Summary:
This error occurs when the client sends us a set of heads that we cannot match.
For example when the client forces a commit in the master group; this was
possible with revlogs but should be a bug with Segmented Changelog. This can
also happen when master moves backwards, clients and server have multiple heads
then the server reseeds.
Clients that get this error should reclone.
Reviewed By: quark-zju
Differential Revision: D27786602
fbshipit-source-id: 9854ccee929ae0a845236ebd83ed68158c93fc7a
Summary:
We want to distinguish between no location and failure to compute location.
It is useful to know on the client side if we failed to compute the location of
some hash or we are not sending it because we don't have a location for it.
We have different options for hash-to-location in particular but the problem is
a general one. Does it make sense for us to skip sending error at the EdenApi
handler layer? I don't think so, I think that's an old hask. We should be
evolving the EdenApi handler layer to handle application errors.
This is still at the exploration phase. I haven't fixed the client side to
handle the errors.
Reviewed By: quark-zju, kulshrax
Differential Revision: D27549926
fbshipit-source-id: 7094160fe997af0e69c8d006b9731ea7dfb3e3f8
Summary:
The goal here is to add an easy way of propagating application errors from
server to the client.
The most convenient form for an error message is a message string so we just
start with that. I think the most important thing to add is some way of
communicating whether the error is retryable or not.
With conversions for anyhow::Error to WireError and from WireError to
ServerError it should be trivial for application code to pass application
errors down to the client.
The format here is driven by the fact that we have streams in the response
object. A batch oriented format for responses has more options. For example
with batches it is common to have a response object that has 3 categories:
1. found: HashMap, 2. not_found: Vec, 3. errors: Vec/HashMap
Reviewed By: kulshrax
Differential Revision: D27549923
fbshipit-source-id: 33b790253adc4761ea9ac7caced4148f4026e620
Summary: The Python `[auth]` matching code does not take cert validity into account when performing certificate matching, whereas the Rust version of the code does. In practice, the existing call sites for the Rust code disable match-time validation, and instead validate the certificate at time-of-use. This diff makes the Rust code's behavior match Python so we can remove the latter entirely.
Reviewed By: DurhamG
Differential Revision: D27837343
fbshipit-source-id: 0bfb5ebc3a36c8fa748cb289e2d8d1495ba8b296
Summary: Add a new `edenapi.maxrequests` config option to allow controlling the number of parallel in-flight requests. This can be used to bound resource usage of the client when requesting large amounts of data.
Reviewed By: sfilipco
Differential Revision: D27724817
fbshipit-source-id: 8d607efa83d8b0b94074d1a6e06f6c536cc0c791
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641665
fbshipit-source-id: d46f62f9067eb9cb4c8517a62efa6f663d4b6732
Summary: The latter is more lightweight.
Reviewed By: DurhamG
Differential Revision: D27641669
fbshipit-source-id: d907407f5a6e868862fe37f1f67fbe99ee378156
Summary: Provides a way to read config with lighter dependencies.
Reviewed By: DurhamG
Differential Revision: D27641668
fbshipit-source-id: fc99a78f5f51e63f61d1b049af74f61f5d1916a3
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the client code to thread the list of master commits to the
EdenApi client.
Reviewed By: quark-zju
Differential Revision: D27523868
fbshipit-source-id: db4148a3f1d0e8b0b162e0ecc934e07f041c5511
Summary:
We want to handle the case where the client has multiple heads for master. For
example when master is moved backwards (or when it get moved on the client by
force). Updating the request object for HashToLocation to send over all the
master heads.
When the server builds non-master commits, we will want to send over non-master
heads too. We may consider having one big list of heads but I think that we
would still want to distinguish the non-master commit case in order to optimize
that use-case.
Reviewed By: quark-zju
Differential Revision: D27521778
fbshipit-source-id: cc83119b47ee90f902c186528186ad57bf023804
Summary:
Only used by one test that can define the constaint itself.
The problem with having it on the trait is that it's a bit noisy when
things operate on ToApi at the generic level. It adds to the list of
constaints they these users of the ToApi trait need to add.
Reviewed By: kulshrax
Differential Revision: D27549922
fbshipit-source-id: fff9e513eb4c06862111ce6eecc84ab981eea893
Summary:
This is only used in one utility which can define the constaint itself.
I am looking to simplify the Requirements for ToWire so that we can more
easily iterate on them. Debug as a requirement produces too much noise.
There is the risk of ending up in a scenario where we want to print the Wire
type but it's more practical to annotate structs with derive Debug when that
happens than to add the constaint in the trait.
Reviewed By: kulshrax
Differential Revision: D27549925
fbshipit-source-id: aacf7c1c465c94414be02aa143187897c7084980
Summary:
There is no use for it outside of one test which can describe that constraint
itself.
I think that requiring ToWire and ToApi to use the same objects is too much
for the general use case. We regularly convert between different object types
that are the same logical thing but have different representations. A good
example for that is the HgId. It makes sense to implement ToWire for all HgId
variations.
Reviewed By: kulshrax
Differential Revision: D27549924
fbshipit-source-id: d76d7a4beb528634bed46ae93dbd634d850547e5
Summary: This makes it easier to use.
Reviewed By: kulshrax
Differential Revision: D27406589
fbshipit-source-id: 11bef407ab620859381c6ee952e6ef00494551e1
Summary: Use Edenapi book request and response type in bookmarks edenapi endpoint. Serialize the response as cbor.
Reviewed By: kulshrax
Differential Revision: D27174122
fbshipit-source-id: 6bc7295c25bd355db4625da3c1f8c68349e7b0b7
Summary: Add edenapi types for the bookmarks endpoint. Now the endpoint can handle a request for a batch of bookmarks instead of a single bookmark. The request type still needs to be modified at some point to allow for bookmark prefix listing.
Reviewed By: kulshrax
Differential Revision: D27133284
fbshipit-source-id: c3960629cad76504e222f726a151eb3390850276
Summary:
Introduce a new `StoreFile` type for the `revisionstore` crate. This is an enum of `File`, `LfsPointer`, and `RedactedFile`, which represent the different cases the `Entry` type's `content` might actually represent. The `File` variant also handles stripping the copy header, if present, and stores both the `copied_from`, raw unstripped `content`, and stripped `content`, though only the latter can be accessed through the public API right now. Ideally, we'll move copy information into the history API and never actually need to make it public here.
Conversions are provided from `Entry` and EdenApi's `FileEntry` type (which also supports redaction and LFS pointers, but as errors instead of first-class values).
Modify the output type used in the Python bindings to use this new `StoreFile` type.
Currently, the `LfsPointer` variant is unused. This will be used later when we add first-class LFS support.
Reviewed By: kulshrax
Differential Revision: D26862710
fbshipit-source-id: 8326921f3ee43bf2e253847d5735c61f5a50bfa6
Summary:
The `Request` builder's methods currently take `self` by value, which was intended to make it easy to create new `Request`s in a fluent style without assignment. Unfortunately, it turns out that this complicates situations where we need to modify a `Request` in-place.
The ideal solution would be to change `Request`'s builder methods to take `&mut self` instead [1]. I tried doing this, but unfortunately there are too many other parts of the current design that rely on the existing behavior in ways that are difficult to change (particularly around the creation of streaming and async requests).
As a workaround, this diff simply adds matching `fn set_X(&mut self, ...) -> &mut Self` methods for each builder method on `Request`. The existing consuming methods have been rewritten in terms of those methods to prevent duplication of the actual method contents.
Given that all of the consuming builders now contain essentially the same body, it seems like we could reduce the verbosity here by using a macro. Unfortunately, I'm not sufficiently experienced with writing nontrivial macros to come up with something quickly, but I do think that it would be a good idea to eventually use a macro here.
---
[1]: In fact, it turns out that [this is considered a better practice](https://github.com/rust-lang/api-guidelines/discussions/81) when designing builders in Rust -- as long as the terminal method of the builder chain returns the built struct by value, it is still possible to create a new instance without assignment. This does mean, however, that we cannot easily move things from the builder to the final struct without using tricks like `Option::take` and `mem::swap`, since the signature of the terminal method would be `(&mut self) -> Self`).
Reviewed By: andll
Differential Revision: D27256048
fbshipit-source-id: 14f770a87abc839d358e5ba211a096226d3e0dc6
Summary:
Introduce newfilestore class in pyrevisionstore, which constructs a newfilestore (BoxedReadStore for files) and a corresponding `ContentStore` which share the underlying IndexedLog object and EdenApi client.
Modify remotefilelog to construct ContentStore via this new class.
Currently, no methods are provided for the newfilestore - it is meant to be passed back into Rust code, where you may call it's Rust methods as shown in the `test_newstore` method (which will be removed in the future).
Currently the `util` module is made public for access from pyrevisionstore. In the future, this will be replaced in favor of a `NewFileStoreBuilder` which handles these concerns internally.
Reviewed By: DurhamG
Differential Revision: D26526331
fbshipit-source-id: c0f439fbee4c303db4a82171c866a3f3a5fc2324
Summary:
Add an `edenapi.logdir` config option, which, when set, will cause the EdenAPI client to write a JSON version of every request it sends to the specified directory. This is intended for use when reproducing user issues (so that the developer can inspect and replay the requests that were sent).
The JSON request files can be directly given to the `edenapi_cli` or converted to CBOR using the `make_req` tool for manual testing with `curl`.
The approach right now is a bit simplistic, in that we just write a JSON file whenever the EdenAPI client makes a request. I'm open to suggestions on how to improve this. (For example, if there were a way for us to always have a record of the last N requests, that would be helpful for debugging user issues.)
Reviewed By: DurhamG
Differential Revision: D27145093
fbshipit-source-id: 3834c2052b0c5efa05d1d209962953b29f545a3f
Summary:
`CommitHashToLocationRequestBatch` and `CommitRevlogDataRequest` implemented `FromJson` but not `ToJson`. The latter is required so that the EdenAPI client can log JSON representations of all request types.
In the process of adding tests for those two types, I discovered a bug in `CommitLocationToHashRequestBatch`'s `ToJson` implementation which prevented it from passing JSON roundtrip tests (which I added because this type didn't have test coverage here). The bug is now fixed and the roundtrip test passes.
Reviewed By: krallin
Differential Revision: D27105414
fbshipit-source-id: d2e23e21f996ac8062bca948ed9abe3779e4669d
Summary: Use a macro to define quickcheck tests that perform roundtrip conversions for EdenAPI types that implement `ToJson` and `FromJson`. This makes it easier to add additional roundtrip tests for new types.
Reviewed By: krallin
Differential Revision: D27105416
fbshipit-source-id: c6059f9527d57b88cd6e3d1647fa89d8f826b1e3
Summary: This line just collects a vector into a vector. Probably a remnant of a refactor. Delete it.
Reviewed By: quark-zju
Differential Revision: D27091643
fbshipit-source-id: fb611aabea375b8495476401b2d9cdf7ba12fde1
