Summary: Add appropriate EdenAPI calls to allow for HTTP tree fetching in `treemanifest`. Enabling `remotefilelog.http` essentially reroutes `_prefetch` and `getdesignatednodes` to their HTTP equivalents.
Reviewed By: DurhamG
Differential Revision: D23877319
fbshipit-source-id: 8a71934b47d07d2655fa46c103a14fb99e2f7b1f
Summary: Use the functionality from D23910534 (721f5af278) to set a timeout for EdenAPI requests, configured via the `edenapi.timeout` option.
Reviewed By: DurhamG
Differential Revision: D23911552
fbshipit-source-id: 4a6e3de1094d0faa1daaf6fe4b9b7aafb37a25a8
Summary:
Cast `names` and `nodes` to lists. The reason that they are sets is deduplication, but otherwise the code doesn't rely on them being sets (and in fact casts them to lists at multiple points later).
The main motivation for this is to allow these to be passed to Rust code later. The Rust bindings make a distinction between sequence types and unordered types, so passing a set in place of a list would result in a type error.
Reviewed By: DurhamG
Differential Revision: D23893108
fbshipit-source-id: 9ce2addb824867bcb2d24ba14c589b8791a156e8
Summary: Add the ability to set a timeout on HTTP requests. Equivalent to [`CURLOPT_TIMEOUT_MS`](https://curl.haxx.se/libcurl/c/CURLOPT_TIMEOUT_MS.html).
Reviewed By: DurhamG
Differential Revision: D23910534
fbshipit-source-id: a7aec792ec3c122a01aa44fcfe2e2df6e3a111fc
Summary:
This diff adds the use of candidate selection hints to `cross_repo_sync` code sights, which need to query `CommitSyncOutcome` in the small-to-large direction. Specifically: `unsafe_sync_commit` and `unsafe_sync_commit_pushrebase` are the two main functions.
One will now get `CandidateSelectionHint` from the callsight (most notably: `push_redirector`), the other one will build a bookmark-based hint itself.
Reviewed By: StanislavGlebik
Differential Revision: D23715259
fbshipit-source-id: 3f4924f1337b09f3762cc050c4017c5d2bd6cab6
Summary: These concatenated strings and bytes and needed to be fixed.
Reviewed By: kulshrax
Differential Revision: D23907301
fbshipit-source-id: 0008d7d54469266ecbae8ddaaa7625820f62cb7e
Summary:
There are several places in the HTTP client where we log and discard errors. (Typically, these are "this should never happen" type situations.)
Previously, these were logged at the `trace` log level, meaning that in practice no one would ever know if we did hit these errors.
Let's upgrade them to `error` so that they'll be printed out. (In theory, users should never see these error messages unless something has gone horribly wrong.)
Reviewed By: DurhamG
Differential Revision: D23888268
fbshipit-source-id: 9007205f946ebb0127238c76812cf62524878047
Summary:
This test relies on precise timing of racing processes. This is flakey
in our automated tests. Since this is mainly about revlog based repos, and we
only have that on the servers now, and we're going to delete them soon, let's
delete this test to make our tests more stable.
Reviewed By: kulshrax
Differential Revision: D23908316
fbshipit-source-id: 3fd110a8267d3bc16bbcb4545b9ff921274f7588
Summary:
Prefetch profile can now be used in automation to register profiles and to fetch
them. Since it will be used out side of the team it seems like this should be
moved out from under debug.
Reviewed By: genevievehelsel
Differential Revision: D23773337
fbshipit-source-id: b7f646ec202653233ed82528380d91f583a8341e
Summary:
This will allow us (and maybe users) to remove a users active profiles
easily when investigating issues. (And also will be useful for testing.)
Reviewed By: genevievehelsel
Differential Revision: D23517409
fbshipit-source-id: d57e595770b5a60726e2efe9cbf21c89dd694cc0
Summary:
This will allow tools to activate prefetch profiles relavant to their workflow.
Eventually this should also cause eden to prefetch the profile for the current
commit, but for now it just activates the profile.
Reviewed By: genevievehelsel
Differential Revision: D23517412
fbshipit-source-id: 172a844ca00bf3bcf14f4b94c25e8256e516b737
Summary:
This will allow us (and maybe users) to see a users active profiles
easily when investigating issues. (And also will be useful for testing.)
Reviewed By: genevievehelsel
Differential Revision: D23517410
fbshipit-source-id: 7a592f7b39c887600307a4f08668ca1ee9b982bf
Summary:
Assuming the format for active prefetch profiles as discussed here:
https://fb.quip.com/v8yFAzeGx8TU
This allows the cli to read in and write out active profiles, allowing the cli
to manipulate them in the following changes. This will allow tools to activate
profiles useful for that workflow. Additionally, it will allow us (and maybe
users) to see/remove a users active profiles easily when investigating issues.
Reviewed By: genevievehelsel
Differential Revision: D23517411
fbshipit-source-id: 8ab18a1aeccf8dcdd04aca826d61070fa0aedd33
Summary: Linkrevs are on their way out. Let's switch to linknodes.
Reviewed By: quark-zju
Differential Revision: D23765176
fbshipit-source-id: 0dc1e0db11d732ce1edd24d863f32f08a5a5ce42
Summary:
The rust contentstore doesn't allow runtime manipulation of the list of
stores, which is required in order to insert the bundle store into the store.
Let's continue using the old python union store in these cases. This still let's
us delete the python pack code later, but we'll have to keep around the python
union store until we come up with a better solution.
Reviewed By: quark-zju
Differential Revision: D23689630
fbshipit-source-id: 0c60e6f268e81804149afa24409f64b5303e1e34
Summary:
The remaining test failures are mostly around bundle support, which
I'll fix in a later diff.
Reviewed By: quark-zju
Differential Revision: D23664037
fbshipit-source-id: 2bdde3cb4fcded6e0cf3afdc23269662544821df
Summary:
The high level prefetch trees API had a depth parameter, but the Rust
prefetch path doesn't support that. In the long run we probably want to get rid
of the depth parameter (or make it more useful), so for now let's get rid of it
from the function signature. You can still set the depth via config, and the few
places that needed depth are changed to use the config.
Reviewed By: quark-zju
Differential Revision: D23772384
fbshipit-source-id: a037d7207d4076a47368366ef7fd2dc1cfbf5cfb
Summary:
Treemanifest needs to be able to write to the shared stores from paths
other than just prefetch (like when it receives certain trees via a standard
pull). To make this possible we need to expose the Rust shared mutable stores.
This will also make just general integration with Python cleaner.
In the future we can get rid of the non-prefetch download paths and remove this.
Reviewed By: quark-zju
Differential Revision: D23772385
fbshipit-source-id: c1e67e3d21b354b85895dba8d82a7a9f0ffc5d73
Summary:
ConnectionSecurityChecker now supports per repository ACL checking.
PermissionCheckers are created in constructor for each repo.
Later when there is a need to check permissions, they're retrieved using a hash map.
Reviewed By: HarveyHunt
Differential Revision: D23678515
fbshipit-source-id: 3d2880fc9df137872ea64a47636f1142d0b36fc1
Summary:
Previously we had just an empty TestLiveCommitSyncConfig in tests. Since we are
not using it at all right now, it was fine, but we are planning to start using
it later. To do that let's configure TestLiveCommitSyncConfig so that it's not
empty but actually stores a real content.
Reviewed By: ikostia
Differential Revision: D23903579
fbshipit-source-id: af05a377f730c1824b03327749e6f824361e23e2
Summary:
At the moment we have a weird setup where cross repo sync configuration is
stored in both live commit sync configuration and in normal mononoke config.
The latter is deprecated, however there are still a few parts of the codebase
that rely on that. This diff fixes one place
Reviewed By: ikostia
Differential Revision: D23903578
fbshipit-source-id: 2bf4b3d17c34fe2eb6330cd862f7b0f5cd6ffa40
Summary:
In D23845720 (5de500bb99) I described what changes we need to make in our commit syncer. One
part of it is that we should remove get_mover() method, as this method always
uses current version of commit sync map even, and that's incorrect.
This diff removes it from commit validator
Reviewed By: ikostia
Differential Revision: D23864350
fbshipit-source-id: 3f650a32835dda9f82949002d63b52cc36cf04e0
Summary:
D23599866 (54d43b7f95) added an optimization for getbundle that reduces cpu usage when a new
commit with log generation number is added. I.e. the case like this
```
O
|
O
..
O <- new commit, low generation number
|
...
```
Unfortunately this optimization doesn't help with the case where a new repo is
merged into master
```
O <- also new commit, but generationo number is high!
| \
.. O <- new commit, low generation number, but it's not in "heads" parameter
|
|
O
...
```
The merge commit actually has a high generation number, but it's p2 has a low
generation number, so it causes the same issue with high cpu usage.
This diff adds a second optimization ( :( ) that should help with the shortcoming of the first one. See comments for more details.
Reviewed By: ikostia
Differential Revision: D23824204
fbshipit-source-id: 8f647f1813d2662e41325829d05def633372c140
Summary:
This takes johansglock's D23757705 one step further, and gets rid of the
`Wait<...>` wrapper we use to synchronously write to stderr in our logging on
the Mononoke Server side.
This should be fine because We send very little logs to the client, so just
buffering them seems like it won't really hurt, and even if we were writing a
log, it certainly would hurt less than blocking our runtime threads into an
interruptible wait.
A problem is that we actually use this in hgcli, where we want to read from our
stdin and write to our stdout / stderr. Rather than port all this stuff, this
diff updates hgcli to just use Tokio's abstractions for stdink, stdout, and
stderr. I ported the various buffer sizes we use to use there in here (I think
we should buffer less from the server though — 50000 buffers is a lot).
I did however update this to write to `std::io::stderr()` instead of an async
stream for this. I think it's fine considering:
- Internally, Tokio also uses `std:io::stderr()` which has a lock on writing.
- We hardly write anything anyway
Reviewed By: StanislavGlebik
Differential Revision: D23762062
fbshipit-source-id: c8d5330b0735d47b6de00e1a54aee4fed97db6b0
Summary:
There were a few instances of timed out getpack requests on ovrsource.
Example: https://fburl.com/sandcastle/yylr1w3v
Let's bump the timeout to unblock them.
Reviewed By: krallin
Differential Revision: D23900374
fbshipit-source-id: 3ee6e2d4f6b6ed12cd0c1516c686a03c87fa7cb4
Summary:
CommitSyncer is a struct that we use to remap a commit from one repo to another. It uses commit sync map to figure out which paths need to be changed. Commit sync mapping might change, and each commit sync mapping has a version associated with it.
At the moment CommitSyncer doesn't work correctly if a commit sync mapping is changed. Consider the following DAG
```
large repo
A' <- remapped with mapping V1
|
O B' <- remapped with mapping V1
| /
...
small repo
A
|
O B
| /
...
```
We have commit A and B from a small repo remapped into a large repo into commits A' and B'. They were remapped with commit sync mapping V1, which for example remaps files in "dir/" into "smallrepo/dir".
Now let's say we start to use a new mapping v2 which remaps "dir/" into "otherdir/". After this point every commit will be created with new mapping. But this is incorrect - if we create a commit on top of B in a small repo that touches "dir/file.txt" then it will be remapped into "otherdir/file.txt" in the large repo, even though every other file is still in "smallrepo/dir"!
The fix for this issue is to always use the same mapping as commit parent was using (there are a few tricky cases with merge commits and commits with no parents, but those will be dealt with separately).
This diff is the first step - it threads through LiveCommitSyncConfig all the way to the CommitSyncer object, so that CommitSyncer can always fetch whatever mapping it needs.
Reviewed By: ikostia
Differential Revision: D23845720
fbshipit-source-id: 555cc31fd4ce09f0a6fa2869bfcee2c7cdfbcc61
Summary:
Our current megarepo configuration is in a bit of a mess:
1) We have LiveCommitSyncConfig, which fetches the latest version of configs
from configerator and should be used in all cases
2) However we still have an old commit that's stored in mononoke config. It
shouldn't really be used at all.
Unfortunately there are a few places where #2 is still used. This diff removes
one of them.
Reviewed By: ikostia
Differential Revision: D23845297
fbshipit-source-id: aa2d591223cc4b8fe5ef264147457fcb3d1faad7
Summary:
It looks like clang in mode/win choke on the code otherwise: P143369452. It's
definitively not clear to me why. Since fmt::format should pretty much
similarly to folly::to<std::string>, just use that instead.
Reviewed By: chadaustin, fanzeyi
Differential Revision: D23884978
fbshipit-source-id: f4a7e47169c8b78011340ec0a41871c9fa4b7181
Summary:
This stack updates eden to be able to check all of the locations that able
users certificate may reside.
There can be multiple places where a cert may reside (we cant always
definitively choose one place to look based on the platform). Thus we
need to be able to configure multiple locations for certs in our eden
config.
This makes the switch over in eden from using one place for the client
cert to use and using the first available client cert from a list.
NOTE: most of this diff is fixing unit tests take a look at `EdenConfig.h` and `EdenConfig.cpp` first
Reviewed By: wez
Differential Revision: D23359939
fbshipit-source-id: 44beecce3ef098a734dbd7c5eb3fa5f0aad6b50b
Summary:
Introduce separate wire types to allow protocol evolution and client API changes to happen independently.
* Duplicate `*Request`, `*Entry`, `Key`, `Parents`, `RepoPathBuf`, `HgId`, and `revisionstore_types::Metadata` types into the `wire` module. The versions in the `wire` module are required to have proper `serde` annotations, `Serialize` / `Deserialize` implementations, etc. These have been removed from the original structs.
* Introduce infallible conversions from "API types" to "wire types" with the `ToWire` trait and fallible conversions from "wire types" to "API types" with the `ToApi`. API -> wire conversions should never fail in a binary that builds succesfully, but wire -> API conversions can fail in the case that the server and client are using different versions of the library. This will cause, for instance, a newly-introduced enum variant used by the client to be deserialized into the catch-all `Unknown` variant on the server, which won't generally have a corresponding representation in the API type.
* Cleanup: remove `*Response` types, which are no longer used anywhere.
* Introduce a `map` method on `Fetch` struct which allows a fallible conversion function to be used to convert a `Fetch<T>` to a `Fetch<U>`. This function is used in the edenapi client implementation to convert from wire types to API types.
* Modify `edenapi_server` to convert from API types to wire types.
* Modify `edenapi_cli` to convert back to wire types before serializing responses to disk.
* Modify `make_req` to use `ToWire` for converting API structs from the `json` module to wire structs.
* Modify `read_res` to use `ToApi` to convert deserialized wire types to API types with the necessary methods for investigating the contents (`.data()`, primarily). It will print an error message to stderr if it encounters a wire type which cannot be converted into the corresponding API type.
* Add some documentation about protocol conventions to the root of the `wire` module.
Reviewed By: kulshrax
Differential Revision: D23224705
fbshipit-source-id: 88f8addc403f3a8da3cde2aeee765899a826446d
Summary: Add log messages for debugging using the `tracing` crate, which allows them to be enabled via `env_logger`.
Reviewed By: quark-zju
Differential Revision: D23858076
fbshipit-source-id: a8ef1afac6c9ecbfb5d6d78232aa0d03a2fe2054
Summary: Log HTTP stats to stderr to assist with ad-hoc debugging. Will not be printed unless `RUST_LOG` is set appropriately.
Reviewed By: quark-zju
Differential Revision: D23858077
fbshipit-source-id: 39acf3de3fd0ca4403a986eb5373a6a79f1d004a
Summary: Similar to D23819023 (c96de76ac0) but works on Python 2, too.
Reviewed By: DurhamG
Differential Revision: D23858273
fbshipit-source-id: b15be07c8657bc8cb37960b631f2b31e4a78892b
Summary:
These shows up on Windows when building with mode/win, silencing them is easy,
so let's do it.
Reviewed By: wez
Differential Revision: D23871727
fbshipit-source-id: 7d7ea9504c397b72903e98967188a5295f2f1040
Summary:
These don't compile on Windows, and in order to get mode/win to compile, we
need to avoid compiling their contents. Ideally, we could do that in the
TARGETS files with the select statement, but that's not available in fbcode.
Thus, we do the next best thing: ifdef the file entirely.
Reviewed By: wez
Differential Revision: D23871728
fbshipit-source-id: b4d9df6503eaa008e649afd7bdc665cd37a9585d
