Summary: Implement the endpoint. Most of the complexity is type conversion.
Reviewed By: andll
Differential Revision: D27929201
fbshipit-source-id: b96f65a0173a31f716272c8e0dd47ce8a90759cd
Summary: This makes CloneData possible to represent an empty repo.
Reviewed By: sfilipco
Differential Revision: D27926246
fbshipit-source-id: 0bcead224ef5b89c66d07a34d8217edaef62177f
Summary: Implement the files API. It's just reading content from the zstore.
Reviewed By: kulshrax
Differential Revision: D27926251
fbshipit-source-id: 54d04caa63e01b6ce5b9c785990c14043f7f22ad
Summary: The will be useful for "push" logic.
Reviewed By: kulshrax
Differential Revision: D27951633
fbshipit-source-id: 38bbdc554f017d5776df0577b82fbb0c78d18a83
Summary:
This will be useful for "push" related logic.
The name "eager" is to make it explicit that the repo is not lazy.
Reviewed By: kulshrax
Differential Revision: D27951618
fbshipit-source-id: 8039059beba68d269c752bc8ed3e72bde0c55790
Summary:
Currently it's hard to test EdenApi related features in hg tests. The Mononoke
test suite can do it but it's too heavyweight. Looking at the API surface of
EdenApi it's actually quite small. So let's add a minimal Rust struct that can
serve as an EdenApi server.
This diff just adds a few minimal features. EdenApi related features and
push/pull support will be added later.
The name "eager" is to make it explicit that the repo is not lazy. I thought
about names like "testrepo" or "serverrepo", but the implementation is
somewhat "sound" to be used as a client, non-test repo. It can potentially
be used as starting point for a real "repo" in pure Rust. So I didn't choose
those names.
(I'm not entirely happy with the name but it's more like a placeholder
that makes it look different from other names for now).
Reviewed By: kulshrax
Differential Revision: D27926255
fbshipit-source-id: ad7a023de5e77605a553509de82ff13ae8112439
Summary:
This allows an external crate C that implements `EdenApi` to depend on a more
lightweight library just providing `EdenApi` without things like `hg_http`.
Then the `edenapi` crate can depend on C too.
Didn't move it to `edenapi_types` because it would add extra dependencies
(http, http_client, auth, etc.)
Reviewed By: kulshrax
Differential Revision: D28006548
fbshipit-source-id: 6e828974fd3f78fec70d4a04ae7be85abc459b36
Summary:
The `Builder` API is the main API used by external users to obtain an `EdenApi`
client. In the future we want to support different kinds of `EdenApi`, like a
local repo serving it, if `paths.default` is set to something like
`myrepotype:path`. Make `Builder` more flexible to support non-HTTP `EdenApi`s,
by returning `EdenApi` trait objects.
The old builder that is coupled with HTTP is renamed to HttpClientBuilder.
Reviewed By: kulshrax
Differential Revision: D28018586
fbshipit-source-id: 1eff7bbb8f0e5521a9bcf5a225ac361ddf7c310f
Summary:
This ensures the User-Agent is always set. It also makes the `header` less
unnecessary.
Reviewed By: DurhamG
Differential Revision: D28018587
fbshipit-source-id: 1125d2122431579f127e81c4713de45135b1f972
Summary:
Make it easier to use.
This makes it easier for other crates to use `edenapi::Result<T>`, which is
a bit shorter than `Result<T, EdenApiError>`. Also re-export `Metadata`
from revisionstore-types so callsite does not need to depend on
revisionstore-types explicitly.
Reviewed By: kulshrax
Differential Revision: D27926250
fbshipit-source-id: c85198b5c151e10a2d4d2567e23e32605a3e7c36
Summary: Ensure these settings are available in mononokepeer.
Reviewed By: mitrandir77
Differential Revision: D28061520
fbshipit-source-id: 68cbe9f427d4a1528a4c9968b3f1f9dcd2541004
Summary:
New endpoint. This endpoint can be used for prefix lookup and the contains
check.
Reviewed By: quark-zju
Differential Revision: D28034533
fbshipit-source-id: d724b85c3816414475b142215e3052d0b555cf59
Summary:
These structures are going to be used to implement the `commit/hash_lookup`
endpoint in EdenApi.
Reviewed By: quark-zju
Differential Revision: D28034532
fbshipit-source-id: 7b00d0d97dd0593dfa43834cda9fc9e9ab9021c5
Summary:
Generic container for a bunch of uniform objects. This is primarily intended
for requests and responses which can be difficult to evolve when the top level
object is an array. For cases where evolution is required we would
probably replace the Batch wrapper with a specialized type. For example,
starting from `Batch<MyRequest>` we would change to:
struct MyRequestBatch {
pub batch: Vec<T>,
pub evolution: Evolution,
}
Reviewed By: quark-zju
Differential Revision: D28034534
fbshipit-source-id: d231c063eeacf3500b75ae76bcc101ccbcda8881
Summary:
I want to add prefix lookup functionality to EdenApi.
I considered adding some sort of prefix abstraction to Hg or our wire types.
so that I could use the method that was already implemented. I found the
serialization and conversion logic complicated to reason about. It is easier
to transform the prefix query in a range query and add a range query endpoint.
No need to invent new fancy types and deal conversions.
This diff updates the prefix query implementation to rely on the range query.
The range query is functionality that is more general than prefix lookup.
Reviewed By: quark-zju
Differential Revision: D28034531
fbshipit-source-id: 491db2354e3804c4cea6db16fe7d056a962515f7
Summary:
NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing.
There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs).
Here's a detailed list of what is going on:
- I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace.
- Lots of little Tokio changes:
- Stream abstractions moving to `tokio-stream`
- `tokio::time::delay_for` became `tokio::time::sleep`
- `tokio::sync:⌚:Sender::send` became `tokio::sync:⌚:Sender::broadcast`
- `tokio::sync::Semaphore::acquire` returns a `Result` now.
- `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`).
- `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case.
- Runtime builders have changed quite a bit:
- My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`.
- Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out.
- Tokio 1.x now uses Bytes 1.x:
- At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore).
- In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine.
- In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x.
- In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy.
- tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to:
- How we listen in Mononoke Server & LFS
- How we connect in hgcli.
- Note: all this stuff has test coverage.
- The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this.
- Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it.
- I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x.
Reviewed By: mitrandir77
Differential Revision: D26669620
fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
Summary:
MyRouter is no longer used by Mononoke services, it is deprecated and will stop working when we upgrade the tokio.
This diff removes MyRouter support from Mononoke and simplifies the Mysql connection type struct.
Before we had `MysqlOptions` and `MysqlConnectionType` enum to represent what kind of a client we want to use. Now we use only MySQL FFI so I removed `MysqlConnectionType` completely and put everything into the options struct.
As setting up the connections (aka conn pool) is not an async operation, some of the methods don't need to be async anymore. Because this diff is already enormous, I'm refactoring this in the next one.
Reviewed By: StanislavGlebik
Differential Revision: D28007850
fbshipit-source-id: 32c3740f4bb132f06e1e256b0530ace755446cdd
Summary:
This command executes native checkout dry run code for updating between specificed revisions
Command tests network and storage perf, without actually affecting working copy
Examples of usages:
1) User reports that when updating between revisions multiple times, they see unexpected fetches, even though they expected that all data should already be in cache. Running this command shows that update between those revs fetches more data then a cache size, so repating fetches are not a surprise
2) People working on storage/network layer optimizations can utilize this command to test performance without messing with their working copy
This command understands nativecheckout.usescmstore config, same way it is used in actual native checkout
Differential Revision: D28040641
fbshipit-source-id: e5454f3110ade11f3227d6adc804a22176f530b9
Summary: This fn can be use to format time using human readable units.
Differential Revision: D28042138
fbshipit-source-id: 8b1eb6fa892754ee1008b529477fd555bd41c692
Summary: Those methods only access store/network to fetch content but does not write to disk
Differential Revision: D28040640
fbshipit-source-id: e45dd08e12d128d54b3446e1137465981cde8f13
Summary:
HgManifestFileNode is one of the last remaining types we don't walk ( other known one is the git derived data).
Its added as a separate NodeType from HgFileNode as HgManifestFileNode is used much less and users may want to see only the HgFileNodes. Server side the manifest file node is only used to build the bundles returned to the client.
Differential Revision: D28010248
fbshipit-source-id: ce4c773b0f1996df308f1b271890f29947c2c304
Summary:
This uses code that depends on Tokio 0.x so let's get this removed. I think
I could have updated this to just use manifold-thrift, but while I'm at it,
might as well update to the new shiny client.
Reviewed By: farnz
Differential Revision: D28025165
fbshipit-source-id: 4b79c8a4bd0b8789e6827d2135d36245db4447d5
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
We collect some cache statistics already, lets expose them through thrift with
the rest of the stats.
Reviewed By: chadaustin
Differential Revision: D27052265
fbshipit-source-id: d7fdf70260599b8df43824e2442471e332c1b0cf
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
Here we actually start to use the cache!
Reviewed By: chadaustin
Differential Revision: D27050310
fbshipit-source-id: e35db193fea0af7f387b6f44c49b5bcc2a902858
Summary:
This introduces some basic unit tests to ensure correctness of the cache.
We are adding tests to cover the simple methods of the object cache since we
are using that code path here. And adding a few sanity check tests to make sure
the cache works with trees.
Reviewed By: chadaustin
Differential Revision: D27050296
fbshipit-source-id: b5f0577c1662483f732bb962c5b40bca8e1dcb40
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
This introduces the class for the in memory cache and is based on the existing
BlobCache. note that we keep the minimum number of entries functionality from
the blob cache. This is unlikely to be needed as trees are much less likely
than blobs to exceed a reasonable cache size limit, but kept since we already
have it.
Reviewed By: chadaustin
Differential Revision: D27050285
fbshipit-source-id: 9dd46419761d32387b6f55ff508b60105edae3af
Summary:
On all the code paths that matter we always acquire the write lock. Since we are
basically just using a simple lock, distributed mutex is a more efficient implementation
that does fancy tricks with cachelines. This improves performance from testing
globs which cause many concurrent reads from the cache.
Reviewed By: chadaustin
Differential Revision: D27810990
fbshipit-source-id: d22470f3f39e2cd3895f5ea772955b62030d154a
Summary:
Now that Object Cache actually does most the work, I am moving the
BlobCache tests to be ObjectCache tests. I am leaving a few blob cache
tests to sanity check that the cache works with blobs.
Reviewed By: chadaustin
Differential Revision: D27776113
fbshipit-source-id: ef58279d93035588beb162ee19173a42e3ca4e5b
Summary:
We would like to use a limited size LRU cache fore trees as well as blobs,
so I am templatizing this to allow us to use this cache for trees.
Trees will not need to use Interest handles, but in the future we could use
this cache for blob metadata, which might want to use interest handles.
Additionally if we at somepoint change the inode tree scheme that would remove
the tree content from the inodes itself, interest handle might be useful for
trees. We could also use this cache proxy hashes which may or may not use
interest handles. Since some caches may want interest handles and others will
not I am creating get/insert functions that work with and without interest
handles.
Reviewed By: chadaustin
Differential Revision: D27797025
fbshipit-source-id: 6db3e6ade56a9f65f851c01eeea5de734371d8f0
Summary:
Thrift setter API is deprecated since it doesn't bring any value over direct assignment. Removing it can reduce build-time and make our codebase more consistent.
If result of `s.set_foo(bar)` is unused, this diff replaces
s.set_foo(bar);
with
s.foo_ref() = bar;
Otherwise, it replaces
s.set_foo(bar)
with
s.foo_ref().emplace(bar)
Reviewed By: xavierd
Differential Revision: D27986185
fbshipit-source-id: d90aaf27f25f2ecfcbbbe7886e0c0d784f607a87
Summary:
This and following diff will refactor CheckoutPlan creation.
Right now we create CheckoutPlan from manifest diff and then manipulate it with methods like `with_sparse_profile` to adjust plan for different cases.
Those 'adjustment' do not work great with the structure of CheckoutPlan, for example `with_sparse_profile` has to create temporary HashSet just to index files in CheckoutPlan
We are going to add more adjustments in the future (for example, checkout --clean), and will run into same issues with current structure of CheckoutPlan
To avoid those issues, we are going to refactor this code, so that instead of Diff -> CheckoutPlan -> adjustments, we are going to have Diff -> ActionMap -> adjustments -> CheckoutPlan
The structure of CheckoutPlan is still good for it's direct purpose (execution), but all the 'changes' to the plan will be done in ActionMap instead.
Reviewed By: DurhamG
Differential Revision: D27980390
fbshipit-source-id: 403f371fd2fe7760984925a38429e1bfb88d8e3f
Summary: For now this does not work with --clean flag(fallback to regular checkout in that case)
Reviewed By: quark-zju
Differential Revision: D27953967
fbshipit-source-id: 71c097cf1e395ff2cba2f4ee528145d3b2c83c23
Summary: This crate does not calculate status, but allows to convert python status into rust status, that can later be used by python code
Reviewed By: quark-zju
Differential Revision: D27953962
fbshipit-source-id: ab91d9d035140e43d8b17988b24bd030af77c96d
