Summary:
Introduce separate wire types to allow protocol evolution and client API changes to happen independently.
* Duplicate `*Request`, `*Entry`, `Key`, `Parents`, `RepoPathBuf`, `HgId`, and `revisionstore_types::Metadata` types into the `wire` module. The versions in the `wire` module are required to have proper `serde` annotations, `Serialize` / `Deserialize` implementations, etc. These have been removed from the original structs.
* Introduce infallible conversions from "API types" to "wire types" with the `ToWire` trait and fallible conversions from "wire types" to "API types" with the `ToApi`. API -> wire conversions should never fail in a binary that builds succesfully, but wire -> API conversions can fail in the case that the server and client are using different versions of the library. This will cause, for instance, a newly-introduced enum variant used by the client to be deserialized into the catch-all `Unknown` variant on the server, which won't generally have a corresponding representation in the API type.
* Cleanup: remove `*Response` types, which are no longer used anywhere.
* Introduce a `map` method on `Fetch` struct which allows a fallible conversion function to be used to convert a `Fetch<T>` to a `Fetch<U>`. This function is used in the edenapi client implementation to convert from wire types to API types.
* Modify `edenapi_server` to convert from API types to wire types.
* Modify `edenapi_cli` to convert back to wire types before serializing responses to disk.
* Modify `make_req` to use `ToWire` for converting API structs from the `json` module to wire structs.
* Modify `read_res` to use `ToApi` to convert deserialized wire types to API types with the necessary methods for investigating the contents (`.data()`, primarily). It will print an error message to stderr if it encounters a wire type which cannot be converted into the corresponding API type.
* Add some documentation about protocol conventions to the root of the `wire` module.
Reviewed By: kulshrax
Differential Revision: D23224705
fbshipit-source-id: 88f8addc403f3a8da3cde2aeee765899a826446d
Summary: Add log messages for debugging using the `tracing` crate, which allows them to be enabled via `env_logger`.
Reviewed By: quark-zju
Differential Revision: D23858076
fbshipit-source-id: a8ef1afac6c9ecbfb5d6d78232aa0d03a2fe2054
Summary: Log HTTP stats to stderr to assist with ad-hoc debugging. Will not be printed unless `RUST_LOG` is set appropriately.
Reviewed By: quark-zju
Differential Revision: D23858077
fbshipit-source-id: 39acf3de3fd0ca4403a986eb5373a6a79f1d004a
Summary: Similar to D23819023 (c96de76ac0) but works on Python 2, too.
Reviewed By: DurhamG
Differential Revision: D23858273
fbshipit-source-id: b15be07c8657bc8cb37960b631f2b31e4a78892b
Summary:
These shows up on Windows when building with mode/win, silencing them is easy,
so let's do it.
Reviewed By: wez
Differential Revision: D23871727
fbshipit-source-id: 7d7ea9504c397b72903e98967188a5295f2f1040
Summary:
These don't compile on Windows, and in order to get mode/win to compile, we
need to avoid compiling their contents. Ideally, we could do that in the
TARGETS files with the select statement, but that's not available in fbcode.
Thus, we do the next best thing: ifdef the file entirely.
Reviewed By: wez
Differential Revision: D23871728
fbshipit-source-id: b4d9df6503eaa008e649afd7bdc665cd37a9585d
Summary:
test-commitcloud-sync.t is a new change and just needs to be made cross
platform.
I have no idea how test-common-commands-fb.t ever worked. When HGRCPATH is set,
I expect the system hgrc to not be loaded, and therefore we can't run hg-clone.
Let's just unset it, since this is meant to test if the new Mercurial can
execute a clone. Ideally we'd redirect the system hgrc to the in-repo
staticfiles, but that's more effort.
Reviewed By: singhsrb
Differential Revision: D23869645
fbshipit-source-id: 66669d9fd9c3a23b01bc43b365723185b7b2ed33
Summary:
*shakes fist at C++ copy-constructors*
We weren't guaranteeing that case-insensitive status was being propagated on
copies or moves, which meant that eg: `lookup("workspace")` would be treated as
case-sensitive when mounted case insensitively.
Reviewed By: xavierd
Differential Revision: D23857218
fbshipit-source-id: 67e33a8455a0a85e5885389b5bb38b20ef043894
Summary: This can be included in all the supported platforms, thus don't #ifdef it.
Reviewed By: wez
Differential Revision: D23858664
fbshipit-source-id: e61da33a97d87cbfab5bb96d2bdaa865d2c01801
Summary:
This enables autodeps, and brings us one step closer to building EdenFS with
Buck on Windows.
Reviewed By: fanzeyi
Differential Revision: D23857794
fbshipit-source-id: c8587a6f7b9e4d9575a62f592c1d0737dff2a8f0
Summary:
Now that prjfs/EdenDispatcher is no longer directly tied to ProjectedFS (sort
of, this is still a bit WIP), we can move it out of the prjfs directory onto
the inodes one. This allows breaking the circular dependency cycle mentioned in
the previous diff where prjfs and inodes depend on each other.
For now, this is merely a copy/paste of the code enclosed in big #ifdef _WIN32,
we might be able to do better later, but for now this is properly good enough.
Reviewed By: wez
Differential Revision: D23857539
fbshipit-source-id: 77c620bac1656d01d7daee4dbf8b10694a589751
Summary:
If we are to look at the dependency graph for EdenFS on Windows, we would
notice that the eden_prjfs target depends on eden_inodes, and vice versa,
causing a cycle. While CMake is perfectly happy with that, Buck doesn't like
that. The solution to removing this cycle is to move the code that needs the
dependency to eden_inodes into the eden_inodes target, and that's the
EdenDispatcher. However, since PrjfsChannel needs to hold a dispatcher to call
into it, it needs to know the methods exposed by the dispatcher. To achieve
this, a simple abstract class is added, this is the same as what is done for
FUSE.
Reviewed By: wez
Differential Revision: D23857540
fbshipit-source-id: c495c67d43724f648e5ffa17776e4d5d4513698a
Summary: The Windows code is now free of sformat, and only depends on fmt::format.
Reviewed By: fanzeyi
Differential Revision: D23786940
fbshipit-source-id: 4c9d4a8c30e5a3d5eb7971cdc3139539c5893a8d
Summary:
Now that all the logic is in PrjfsChannel.cpp, this header no longer needs to
be included. This helps in making this file as free as possible from Windows
specific bits which will allow it to be moved into the inodes top level
directory.
Reviewed By: fanzeyi
Differential Revision: D23786941
fbshipit-source-id: a4a3a47ea00808ca7a839709068efab06436167f
Summary:
Most of the non-tests callers have an ObjectFetchContext available, let's use
it instead of a static null context. This will help in understanding why some
files/trees are being fetched.
Reviewed By: chadaustin
Differential Revision: D23745752
fbshipit-source-id: b2e145d9e559bde0542adbc5b20ff56ccfc59ece
Summary:
This allows removal of a NullContext which will provide more details as to why
fetches are triggered.
Reviewed By: chadaustin
Differential Revision: D23745755
fbshipit-source-id: c26f648b8695b86caf8fe15c4bc6c4128d345aa1
Summary:
Similarly to the other callbacks, this makes the main function return to
ProjectedFS as soon as the future is created which will allow for it to be
interrupted in a subsequent diff.
Reviewed By: fanzeyi
Differential Revision: D23745754
fbshipit-source-id: 2d77d0eacfe0d37eb9075bf9f0660e4f4af77e8f
Summary:
This merely moves Windows bits outside of the EdenDispatcher and into the
PrjfsChannel, making the former less dependant on Windows, and paving the way
to handling this callback fully asynchronous.
One caveat currently is that while the callback supports specifying an offset
and length, the underlying backing store only allows reading the entire file at
once, thus these arguments are simply ignored in the dispatcher.
Reviewed By: fanzeyi
Differential Revision: D23745753
fbshipit-source-id: 266e1f448f9db536d746da1462a2a590ffad19a6
Summary:
Move some commit cloud operations under infinitepush read path:
those are:
* `hg cloud check` command
* `hg cloud sync` command when the local repo is clean
* `hg cloud switch` command will normally use the read path for the dest workspace because we clean up the repo before performing the switch
* `hg cloud rejoin` command we use in fbclone will normally go through the read path as it runs in a fresh repo
If something is broken, there is always a way to rerun any of these command with '--dest' flag pointing it to the write path.
```
./hg cloud check -r 0c9596fd1 --remote --dest infinitepush-write
./hg cloud sync --dest infinitepush-write
./hg cloud switch -w other --dest infinitepush-write
```
Those use cases are limited and the lag of forward filler shouldn't be noticeable for them but we will be able to collect more signal how Mononoke performs with Commit Cloud.
Sitevar to control the routing of read traffic:
https://www.internalfb.com/intern/sv/HG_SSH_WRAPPER_MONONOKE_ROLLOUT/#revisions_list
Reviewed By: mitrandir77
Differential Revision: D23840914
fbshipit-source-id: 40fbe2e72756e7a4cf8bc5be6a0b94f6cf4906b4
Summary:
At the moment CommitSyncConfig can be set in two ways:
1) Set in the normal mononoke config. That means that config can be updated
only after a service is restarted. This is an outdated way, and won't be used
in prod.
2) Set in separate config which can be updated on demand. This is what we are
planning to use in production.
create_commit_syncer_from_matches was used to build a CommitSyncer object via
normal mononoke config (i.e. outdated option #1). According to the comments it
was done so that we can read configs from the disk instead of configerator, but
this doesn't make sense because you already can read configerator configs
from disk. So create_commit_syncer_from_matches doesn't look particularly
useful, and besides it also make further refactorings harder. Let's remove it.
Reviewed By: ikostia
Differential Revision: D23811090
fbshipit-source-id: 114d88d9d9207c831d98dfa1cbb9e8ede5adeb1d
Summary:
With segmented changelog backend, the revs can be changed, even if len(repo)
didn't change. Caching revs might not get invalidated properly. Let's cache
head nodes instead.
Reviewed By: DurhamG
Differential Revision: D23856176
fbshipit-source-id: c5154c536298c348b847a12de8c4f582f877f96e
Summary:
On Ubuntu the output is a bit different:
```
$ hg cloud sync --use-bgssh
commitcloud: synchronizing 'server' with 'user/test/default'
- remote: /bin/sh: trashssh: command not found
- abort: no suitable response from remote hg!
+ remote: /bin/sh: 1: trashssh: not found
+ abort: no suitable response from remote hg: '[Errno 32] Broken pipe'!
```
Glob them out to make the test pass.
Reviewed By: DurhamG
Differential Revision: D23824735
fbshipit-source-id: 7f96149ee16daff31fd0a1c68975b5edfa27cc46
Summary:
It seems OSX python2 has SIGINT handler set to SIG_IGN by default when running
inside tests. Detect that and reset SIGINT handler to raise KeyboardInterrupt.
This fixes test-ctrl-c.t on OSX.
As we're here, improve test-ctrl-c.t so it checks a bit more things and run
quicker.
Reviewed By: DurhamG
Differential Revision: D23853455
fbshipit-source-id: 05c47650bc80f9880f724828d307c32786265e2c
Summary:
There's a [flaw](https://fb.workplace.com/groups/scm.mononoke/permalink/1220069065022333) in the current `synced_commit_mapping` data model. In a nutshell, the flaw is in the assumption that the `RewrtittenAs` relationship is `1:1`, while in fact it is `1:n` with `n` on the large repo side.
To address this flaw I propose to:
- relax the DB constraints to represent the semantically correct model
- select all the synced candidates from the DB
- for places in code, which require a single mapping for a candidate, use the provided hint to resolve any ambiguity
More concretely:
- instead of a single `CommitSyncOutcome` struct, I propose to have the "canonical" `PluralCommitSyncOutcome` and the "resolved" `CommitSyncOutcome`
- every variant of `PluralCommitSyncOutcome` that is not `RewrittenAs` just maps to an identical variant of `CommitSyncOutcome`
- have a `CandidateSelectionHint` passed from the clients, which would help resolve `PluralCommitSyncOutcome::RewrittenAs` into a `CommitSyncOutcome::RewrittenAs`
- if the hint does not help to resolve `PluralCommitSyncOutcome::RewrittenAs` into an unambiguous `CommitSyncOutcome::RewrittenAs`, just fail the request and require human intervention to deal with things
- within the hint, have for the following variants for the resolution algorithm:
- `Only` which fails the resolution if there's more than one candidate
- `Exact` behaves like `Only` if there's one candidate, otherwise selects a provided candidate
- `OnlyOr(Ancestor|Descendant)Of(Commit|Bookmark)` behave like `Only` if there's one candidate, otherwise select a candidate in the expected topological relationship
Note some important decisions, that may be surprising at first:
- if there's just one candidate, resolutions with all types of hints succeed, even if this candidate does not fit the hint otherwise (for example, if the hint is `Exact(A)`, and the list of candidates is `[B]`, the resolution succeeds.
- for bookmark-related hints, if the bookmark does not exist at the time of resolution, the hint just "downgrades" itself to be `Only`
Both of these emphasize the fact that if the mapping has only one `RewrittenAs` candidate for a given changeset, the behavior does not change.
Reviewed By: StanislavGlebik
Differential Revision: D23670180
fbshipit-source-id: 1cee1f65fc8020e0ae8a7da789b2532d2e436b77
Summary: SomeFailedOthersNone should not consider write mostly blobstores None if all other stores Error
Reviewed By: farnz
Differential Revision: D23840334
fbshipit-source-id: 9838bead6fec0d5f920e4a788387025d0dacf80b
Summary: Add a test for SomeFailedOthersNone when write mostly blobstore is None
Reviewed By: farnz
Differential Revision: D23840685
fbshipit-source-id: 81834663169b3a522b9c08e0a36f0b91354916c7
Summary:
Now that the win directory only contains the mount directory, we can rename it
to be more faithful to its intent. Since this is about ProjectedFS, let's
rename it "prjfs".
Reviewed By: chadaustin
Differential Revision: D23828561
fbshipit-source-id: cb31fe4652fd4356dc2579028d3ae2c7935371a7
Summary: This will make it easier to build with Buck.
Reviewed By: fanzeyi
Differential Revision: D23827754
fbshipit-source-id: bf3bf4d607a08b9831f9dfea172b2e923a219561
Summary:
Now that sqliteoverlay has a TARGETS file, we can remove the manual, and use
autodeps to have the right dependencies in the TARGETS files. This will help in
getting EdenFS to build on Windows with buck.
Reviewed By: chadaustin
Differential Revision: D23820312
fbshipit-source-id: 34bfd13d2ae6d11a404a9b913562c7d45a4b3de7
Summary:
Phabstatus for smartlog uses `PeekeaheadList` rather than `PeekaheadRevsetIterator` as
all of the commits are known ahead of time, and we don't need to collect together
batches as we iterate across the revset.
However, we should still batch up requests to Phabricator, as users with very high
numbers of commits in their smartlog may hit timeouts.
Add a batching mechanism to `PeekaheadList` that splits the list into chunks to
return with each peekahead.
Reviewed By: liubov-dmitrieva
Differential Revision: D23840071
fbshipit-source-id: 68596c7eb4f7404ce6109e69914f328565e34582
Summary:
This provides a way to fix the local cache of backed up heads if it is in an
invalid state.
The most important, it will allow early dogfooding of write traffic from Mononoke
without the reverse filler in place for developers or for the team.
You could just run `hg cloud backup -f` assuming the repo is backfilled to fix
any inconsistency when switch between the two backends
Reviewed By: markbt
Differential Revision: D23840162
fbshipit-source-id: bbd331162d65ba193c4774e37324f15ed0635f82
Summary:
Let's add a command that validates that the created catchup commit is correct.
For now it validates that unodes are the same between catchup commit and commit
that we are merging in.
Later we can add more invariants that we want to check.
Reviewed By: krallin
Differential Revision: D23782369
fbshipit-source-id: 61d19aa73777d5fbb3e1b127bdcf39f5e6309b52
Summary: Add error context to file content scrub so that we can tell if an Error has propagated via the scrub stream loading.
Reviewed By: StanislavGlebik
Differential Revision: D23838144
fbshipit-source-id: 40a8a090510959cab1020182c19076b8a3317b1b
Summary:
Implemented S3 blobstore
Isilon implements S3 as 1:1 mapping into filesystem, and it limits the maximum number of blobs in the single directory. To overcome it lets shard the keys using base64 encoding and making 2 level dir structure with 2 chars dir names.
Reviewed By: krallin
Differential Revision: D23562541
fbshipit-source-id: c87aca2410381a07babb191cbd8cf28233556e03
Summary:
For Python 3 we must ensure that the displayer messages have all been converted
to unicode before providing them to the Rust graph renderer.
The is because the Python 3 version of `encoding.unifromlocal` is a no-op, so
the result may still be `bytes` that need to be converted to `str`.
Reviewed By: quark-zju
Differential Revision: D23827233
fbshipit-source-id: 8f2b707ceceb210c0a2b5b589b99d4016452c61c
Summary:
D23759711 (be51116cf4) changed the way signal handlers work, which apparently causes
this test to fail. The SIGCHLD signal of the child changing state is received
during os.waitpid, which apparently counts as a signal during a system call,
which throws an OSError.
I'm not sure what the real fix should be. Sleeping gets us past the issue, since
presumably the signal is handled before the system call.
Reviewed By: quark-zju
Differential Revision: D23832606
fbshipit-source-id: 70fca19e419da55bbf546b8530406c9b3a9a6d77
