Summary:
The `Builder` API is the main API used by external users to obtain an `EdenApi`
client. In the future we want to support different kinds of `EdenApi`, like a
local repo serving it, if `paths.default` is set to something like
`myrepotype:path`. Make `Builder` more flexible to support non-HTTP `EdenApi`s,
by returning `EdenApi` trait objects.
The old builder that is coupled with HTTP is renamed to HttpClientBuilder.
Reviewed By: kulshrax
Differential Revision: D28018586
fbshipit-source-id: 1eff7bbb8f0e5521a9bcf5a225ac361ddf7c310f
Summary:
This ensures the User-Agent is always set. It also makes the `header` less
unnecessary.
Reviewed By: DurhamG
Differential Revision: D28018587
fbshipit-source-id: 1125d2122431579f127e81c4713de45135b1f972
Summary:
Make it easier to use.
This makes it easier for other crates to use `edenapi::Result<T>`, which is
a bit shorter than `Result<T, EdenApiError>`. Also re-export `Metadata`
from revisionstore-types so callsite does not need to depend on
revisionstore-types explicitly.
Reviewed By: kulshrax
Differential Revision: D27926250
fbshipit-source-id: c85198b5c151e10a2d4d2567e23e32605a3e7c36
Summary: Ensure these settings are available in mononokepeer.
Reviewed By: mitrandir77
Differential Revision: D28061520
fbshipit-source-id: 68cbe9f427d4a1528a4c9968b3f1f9dcd2541004
Summary:
New endpoint. This endpoint can be used for prefix lookup and the contains
check.
Reviewed By: quark-zju
Differential Revision: D28034533
fbshipit-source-id: d724b85c3816414475b142215e3052d0b555cf59
Summary:
These structures are going to be used to implement the `commit/hash_lookup`
endpoint in EdenApi.
Reviewed By: quark-zju
Differential Revision: D28034532
fbshipit-source-id: 7b00d0d97dd0593dfa43834cda9fc9e9ab9021c5
Summary:
Generic container for a bunch of uniform objects. This is primarily intended
for requests and responses which can be difficult to evolve when the top level
object is an array. For cases where evolution is required we would
probably replace the Batch wrapper with a specialized type. For example,
starting from `Batch<MyRequest>` we would change to:
struct MyRequestBatch {
pub batch: Vec<T>,
pub evolution: Evolution,
}
Reviewed By: quark-zju
Differential Revision: D28034534
fbshipit-source-id: d231c063eeacf3500b75ae76bcc101ccbcda8881
Summary:
I want to add prefix lookup functionality to EdenApi.
I considered adding some sort of prefix abstraction to Hg or our wire types.
so that I could use the method that was already implemented. I found the
serialization and conversion logic complicated to reason about. It is easier
to transform the prefix query in a range query and add a range query endpoint.
No need to invent new fancy types and deal conversions.
This diff updates the prefix query implementation to rely on the range query.
The range query is functionality that is more general than prefix lookup.
Reviewed By: quark-zju
Differential Revision: D28034531
fbshipit-source-id: 491db2354e3804c4cea6db16fe7d056a962515f7
Summary:
NOTE: there is one final pre-requisite here, which is that we should default all Mononoke binaries to `--use-mysql-client` because the other SQL client implementations will break once this lands. That said, this is probably the right time to start reviewing.
There's a lot going on here, but Tokio updates being what they are, it has to happen as just one diff (though I did try to minimize churn by modernizing a bunch of stuff in earlier diffs).
Here's a detailed list of what is going on:
- I had to add a number `cargo_toml_dir` for binaries in `eden/mononoke/TARGETS`, because we have to use 2 versions of Bytes concurrently at this time, and the two cannot co-exist in the same Cargo workspace.
- Lots of little Tokio changes:
- Stream abstractions moving to `tokio-stream`
- `tokio::time::delay_for` became `tokio::time::sleep`
- `tokio::sync:⌚:Sender::send` became `tokio::sync:⌚:Sender::broadcast`
- `tokio::sync::Semaphore::acquire` returns a `Result` now.
- `tokio::runtime::Runtime::block_on` no longer takes a `&mut self` (just a `&self`).
- `Notify` grew a few more methods with different semantics. We only use this in tests, I used what seemed logical given the use case.
- Runtime builders have changed quite a bit:
- My `no_coop` patch is gone in Tokio 1.x, but it has a new `tokio::task::unconstrained` wrapper (also from me), which I included on `MononokeApi::new`.
- Tokio now detects your logical CPUs, not physical CPUs, so we no longer need to use `num_cpus::get()` to figure it out.
- Tokio 1.x now uses Bytes 1.x:
- At the edges (i.e. streams returned to Hyper or emitted by RepoClient), we need to return Bytes 1.x. However, internally we still use Bytes 0.5 in some places (notably: Filestore).
- In LFS, this means we make a copy. We used to do that a while ago anyway (in the other direction) and it was never a meaningful CPU cost, so I think this is fine.
- In Mononoke Server it doesn't really matter because that still generates ... Bytes 0.1 anyway so there was a copy before from 0.1 to 0.5 and it's from 0.1 to 1.x.
- In the very few places where we read stuff using Tokio from the outside world (historical import tools for LFS), we copy.
- tokio-tls changed a lot, they removed all the convenience methods around connecting. This resulted in updates to:
- How we listen in Mononoke Server & LFS
- How we connect in hgcli.
- Note: all this stuff has test coverage.
- The child process API changed a little bit. We used to have a ChildWrapper around the hg sync job to make a Tokio 0.2.x child look more like a Tokio 1.x Child, so now we can just remove this.
- Hyper changed their Websocket upgrade mechanism (you now need the whole `Request` to upgrade, whereas before that you needed just the `Body`, so I changed up our code a little bit in Mononoke's HTTP acceptor to defer splitting up the `Request` into parts until after we know whether we plan to upgrade it.
- I removed the MySQL tests that didn't use mysql client, because we're leaving that behind and don't intend to support it on Tokio 1.x.
Reviewed By: mitrandir77
Differential Revision: D26669620
fbshipit-source-id: acb6aff92e7f70a7a43f32cf758f252f330e60c9
Summary:
MyRouter is no longer used by Mononoke services, it is deprecated and will stop working when we upgrade the tokio.
This diff removes MyRouter support from Mononoke and simplifies the Mysql connection type struct.
Before we had `MysqlOptions` and `MysqlConnectionType` enum to represent what kind of a client we want to use. Now we use only MySQL FFI so I removed `MysqlConnectionType` completely and put everything into the options struct.
As setting up the connections (aka conn pool) is not an async operation, some of the methods don't need to be async anymore. Because this diff is already enormous, I'm refactoring this in the next one.
Reviewed By: StanislavGlebik
Differential Revision: D28007850
fbshipit-source-id: 32c3740f4bb132f06e1e256b0530ace755446cdd
Summary:
This command executes native checkout dry run code for updating between specificed revisions
Command tests network and storage perf, without actually affecting working copy
Examples of usages:
1) User reports that when updating between revisions multiple times, they see unexpected fetches, even though they expected that all data should already be in cache. Running this command shows that update between those revs fetches more data then a cache size, so repating fetches are not a surprise
2) People working on storage/network layer optimizations can utilize this command to test performance without messing with their working copy
This command understands nativecheckout.usescmstore config, same way it is used in actual native checkout
Differential Revision: D28040641
fbshipit-source-id: e5454f3110ade11f3227d6adc804a22176f530b9
Summary: This fn can be use to format time using human readable units.
Differential Revision: D28042138
fbshipit-source-id: 8b1eb6fa892754ee1008b529477fd555bd41c692
Summary: Those methods only access store/network to fetch content but does not write to disk
Differential Revision: D28040640
fbshipit-source-id: e45dd08e12d128d54b3446e1137465981cde8f13
Summary:
HgManifestFileNode is one of the last remaining types we don't walk ( other known one is the git derived data).
Its added as a separate NodeType from HgFileNode as HgManifestFileNode is used much less and users may want to see only the HgFileNodes. Server side the manifest file node is only used to build the bundles returned to the client.
Differential Revision: D28010248
fbshipit-source-id: ce4c773b0f1996df308f1b271890f29947c2c304
Summary:
This uses code that depends on Tokio 0.x so let's get this removed. I think
I could have updated this to just use manifold-thrift, but while I'm at it,
might as well update to the new shiny client.
Reviewed By: farnz
Differential Revision: D28025165
fbshipit-source-id: 4b79c8a4bd0b8789e6827d2135d36245db4447d5
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
We collect some cache statistics already, lets expose them through thrift with
the rest of the stats.
Reviewed By: chadaustin
Differential Revision: D27052265
fbshipit-source-id: d7fdf70260599b8df43824e2442471e332c1b0cf
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
Here we actually start to use the cache!
Reviewed By: chadaustin
Differential Revision: D27050310
fbshipit-source-id: e35db193fea0af7f387b6f44c49b5bcc2a902858
Summary:
This introduces some basic unit tests to ensure correctness of the cache.
We are adding tests to cover the simple methods of the object cache since we
are using that code path here. And adding a few sanity check tests to make sure
the cache works with trees.
Reviewed By: chadaustin
Differential Revision: D27050296
fbshipit-source-id: b5f0577c1662483f732bb962c5b40bca8e1dcb40
Summary:
Chad first noted that deserializing trees from the local store can be expensive.
From the thrift side EdenFS does not have a copy of trees in memory. This
means for glob files each of the trees that have not been materialized will be
read from the local store. Since reading an deserializing trees from the local
store can be expensive lets add an in memory cache so that some of these
reads can be satisfied from here instead.
This introduces the class for the in memory cache and is based on the existing
BlobCache. note that we keep the minimum number of entries functionality from
the blob cache. This is unlikely to be needed as trees are much less likely
than blobs to exceed a reasonable cache size limit, but kept since we already
have it.
Reviewed By: chadaustin
Differential Revision: D27050285
fbshipit-source-id: 9dd46419761d32387b6f55ff508b60105edae3af
Summary:
On all the code paths that matter we always acquire the write lock. Since we are
basically just using a simple lock, distributed mutex is a more efficient implementation
that does fancy tricks with cachelines. This improves performance from testing
globs which cause many concurrent reads from the cache.
Reviewed By: chadaustin
Differential Revision: D27810990
fbshipit-source-id: d22470f3f39e2cd3895f5ea772955b62030d154a
Summary:
Now that Object Cache actually does most the work, I am moving the
BlobCache tests to be ObjectCache tests. I am leaving a few blob cache
tests to sanity check that the cache works with blobs.
Reviewed By: chadaustin
Differential Revision: D27776113
fbshipit-source-id: ef58279d93035588beb162ee19173a42e3ca4e5b
Summary:
We would like to use a limited size LRU cache fore trees as well as blobs,
so I am templatizing this to allow us to use this cache for trees.
Trees will not need to use Interest handles, but in the future we could use
this cache for blob metadata, which might want to use interest handles.
Additionally if we at somepoint change the inode tree scheme that would remove
the tree content from the inodes itself, interest handle might be useful for
trees. We could also use this cache proxy hashes which may or may not use
interest handles. Since some caches may want interest handles and others will
not I am creating get/insert functions that work with and without interest
handles.
Reviewed By: chadaustin
Differential Revision: D27797025
fbshipit-source-id: 6db3e6ade56a9f65f851c01eeea5de734371d8f0
Summary:
Thrift setter API is deprecated since it doesn't bring any value over direct assignment. Removing it can reduce build-time and make our codebase more consistent.
If result of `s.set_foo(bar)` is unused, this diff replaces
s.set_foo(bar);
with
s.foo_ref() = bar;
Otherwise, it replaces
s.set_foo(bar)
with
s.foo_ref().emplace(bar)
Reviewed By: xavierd
Differential Revision: D27986185
fbshipit-source-id: d90aaf27f25f2ecfcbbbe7886e0c0d784f607a87
Summary:
This and following diff will refactor CheckoutPlan creation.
Right now we create CheckoutPlan from manifest diff and then manipulate it with methods like `with_sparse_profile` to adjust plan for different cases.
Those 'adjustment' do not work great with the structure of CheckoutPlan, for example `with_sparse_profile` has to create temporary HashSet just to index files in CheckoutPlan
We are going to add more adjustments in the future (for example, checkout --clean), and will run into same issues with current structure of CheckoutPlan
To avoid those issues, we are going to refactor this code, so that instead of Diff -> CheckoutPlan -> adjustments, we are going to have Diff -> ActionMap -> adjustments -> CheckoutPlan
The structure of CheckoutPlan is still good for it's direct purpose (execution), but all the 'changes' to the plan will be done in ActionMap instead.
Reviewed By: DurhamG
Differential Revision: D27980390
fbshipit-source-id: 403f371fd2fe7760984925a38429e1bfb88d8e3f
Summary: For now this does not work with --clean flag(fallback to regular checkout in that case)
Reviewed By: quark-zju
Differential Revision: D27953967
fbshipit-source-id: 71c097cf1e395ff2cba2f4ee528145d3b2c83c23
Summary: This crate does not calculate status, but allows to convert python status into rust status, that can later be used by python code
Reviewed By: quark-zju
Differential Revision: D27953962
fbshipit-source-id: ab91d9d035140e43d8b17988b24bd030af77c96d
Summary: When checking out on dirty copy without --clean this function can be used to check if checkout operation conflicts with currently modified files
Reviewed By: quark-zju
Differential Revision: D27953965
fbshipit-source-id: 4096506e4cbf8b102e0afa1a929c066dfa474825
Summary:
This crate introduces consumer API for status in rust
Currently the implementation will just take status from Python and convert it into this struct
But in the future we can get pure Rust implementation to get status
Reviewed By: quark-zju
Differential Revision: D27953963
fbshipit-source-id: 29c876400c82056eaf81fffa4adc814473853c1e
Summary: This method can be used to 'normalize' path for case insentive use cases
Reviewed By: quark-zju
Differential Revision: D27953964
fbshipit-source-id: 421832af22af9a3b56eec0d045b9f983592ed192
Summary:
Fix missing stats for the bookmarks endpoint because we have the wrong name in
code.
Reviewed By: quark-zju
Differential Revision: D28008091
fbshipit-source-id: 128fe00e00a06ebe9b65fb11512cd03a042d55fe
Summary:
This doesn't link the errors into Scuba, which makes it not very useful for
debugging, since we're not routinely looking at stderr on our tasks, and makes
it impossible to e.g. look anywhere and find a count of failed requests.
Instead, update this to use `capture_first_err`, which will report both the
first error and the total error count to Scuba.
Reviewed By: sfilipco
Differential Revision: D27998037
fbshipit-source-id: b941d44a2ac21bbf640b9bc977de749207f12d9a
Summary:
In EdenAPI, most endpoints don't raise errors when they fail, and instead just take items out of the response stream (though there are some exceptions: D24315399 (0ccae1cef9).
Right now, this just gets logged to stderr, which isn't great. This diff introduces a CaptureFirstError wrapper we can use to capture those errors and expose them in post-response callbacks (i.e. Scuba).
Reviewed By: sfilipco
Differential Revision: D27998036
fbshipit-source-id: 960d0e09a82ca79adfafe22e2eeef2e0294d27dc
Summary:
`lld` is faster than `ld.gold`.
To compare, `make local3`, then add a blank line in `hgmain`, then run `make
local3` again.
Using `lld`:
building 'hgmain' binary
Compiling hgmain v0.1.0 (/home/quark/fbsource-lazy/fbcode/eden/scm/exec/hgmain)
Finished release [optimized + debuginfo] target(s) in 3.73s
building 'mkscratch' binary
Finished release [optimized] target(s) in 0.37s
building 'scm_daemon' binary
Finished release [optimized] target(s) in 0.42s
Using `ld.gold`:
building 'hgmain' binary
Compiling hgmain v0.1.0 (/home/quark/fbsource-lazy/fbcode/eden/scm/exec/hgmain)
Finished release [optimized + debuginfo] target(s) in 10.15s
building 'mkscratch' binary
Finished release [optimized] target(s) in 0.39s
building 'scm_daemon' binary
Finished release [optimized] target(s) in 0.46s
Reviewed By: ikostia
Differential Revision: D28006551
fbshipit-source-id: 55b19be93d8152634d79ed92c9cd53237f91b820
Summary:
This raw XDB connections API is not used in Mononoke although there are projects depending on it.
The API creates connection objects based on shed/sql crate.
This diff moves `SqlConnections` into `shed/sql:sql_common`, closer to the `Connection` definition, and moves `create_raw_xdb_connections` API into the `shed/sql:sql_facebook::raw`.
Reviewed By: krallin
Differential Revision: D28003638
fbshipit-source-id: ea4a29b6e239a89c97237877e2dfde4c7c7ff89b
Summary:
It turns out during the initial clone, we're not using selectivepull for some
tiers that do not have the non-repo selectivepull config.
We've been using selectivepull for devservers and corp (and it's effective
during clone) for a long time. Let's just enable it by default so even if
dynamicconfig does not set it properly, we can still use selectivepull clone
to avoid pulling 10k+ remote bookmarks (which triggers auto bookmark cleanups
as logged in hgfeatures).
There are too many incompatible tests so I'm not migrating them for now.
Reviewed By: DurhamG
Differential Revision: D28006488
fbshipit-source-id: f0dc000156abde530fd8881bd26b4642a36167be
Summary:
As I split the pushrebase crate in D27968029 (93b8cf116b) it makes sense to stop re-exporting hook definitions from it.
This change will also make dependencies more accurate.
Reviewed By: krallin
Differential Revision: D28028045
fbshipit-source-id: 622ef5d7db737e19153109f4d2dcefe54fba2bb4
