Summary:
Sometimes the `AsyncIntoResponse` trait needs additional data (e.g. the set of commit
identity schemes the client is interested in) to convert the item into the response
type.
Currently we use a tuple of `(item, &additional_data)` to work around this, however
this will become less readable as we add new items with more additional data.
Split this use-case out into a new trait: `AsyncIntoResponseWith`. This defines
an associated type which is the type of the additional data needed, and provides a
new method `into_response_with`, where a reference to the additional data can be
provided.
Note that conversions for tuple types that are logical `(name, value)` or `(id,
value)` pairs are still ok. It is specifically the case where we have `(item,
&additional_data)` that we are converting here (i.e. the additional data merely
informs the conversion, it is not part of the resulting response value).
Reviewed By: aslpavel
Differential Revision: D23813371
fbshipit-source-id: c0dcfe826288ad53ad572ae4dd956540605998f5
Summary: Make it clear which error is which, and what the number of expected and actual items are.
Reviewed By: StanislavGlebik
Differential Revision: D23813369
fbshipit-source-id: 5b94c5a67438c475235876669ec2be3fd1866700
Summary: Intern ids to reduce space used in the walk state. This is significant on large repos.
Reviewed By: farnz
Differential Revision: D23691524
fbshipit-source-id: b42f926d88083d06ffc44508db44747f9a14e0a5
Summary:
Passing option is not necessary since live_commit_sync_config is always
available.
Reviewed By: ahornby
Differential Revision: D23811021
fbshipit-source-id: ee11f88d57814d9abac8650e52febd9e431770da
Summary:
Automigration gets messed up with `hg cloud rejoin` command in fbclone code because it triggered by the pull command.
As a result fbclone ends up to join a hostname workspace instead of the default for some cases.
* make sure that the migration never runs if background commit cloud operations are disabled
* also, add skip the migration in the pull command in fbclone
Once of those would be enough to fix the issue but I prefer to make both
changes.
Reviewed By: markbt
Differential Revision: D23813184
fbshipit-source-id: 3b49a3f079e889634e3c4f98b51557ca0679090b
Summary:
I've re-backfilled some of blame values for configerator. But old values might
still be in memcache. To make sure that's not the case let's bump the memcache
key.
Reviewed By: krallin
Differential Revision: D23810971
fbshipit-source-id: c333a51ffb2babf7da808b276f9cfa31baaa105c
Summary:
The children revset iterated over everything in the subset, which in
many cases was the entire repo. This can take hundreds of milliseconds. Let's
use the new _makerangeset to only iterate over descendants of the parentset.
Reviewed By: quark-zju
Differential Revision: D23794344
fbshipit-source-id: 9ac9bc014d56a95b5ac65534769389167b0f4508
Summary:
Now that Mercurial itself can properly handle SIGINT, there isn't a need for a Python wrapper around the Rust EdenAPI client (since the main purpose of the wrapper was to ensure proper SIGINT handling--something that could only be done in Python).
Note that while this does remove some code that prints out certificate warnings, that code was actually broken after the big refactor of the Rust bindings. (The exception types referenced no longer exist, so the code would simple result in a `NameError` if it actually tried to catch an exception from the Rust client.)
Reviewed By: singhsrb
Differential Revision: D23801363
fbshipit-source-id: 3359c181fd05dbec24d77fa1b7d9c8bd821b49a6
Summary: Small change to make it more readable and reduce likelihood of allocation (although the collect might be optimized away anyway)
Reviewed By: farnz
Differential Revision: D23760762
fbshipit-source-id: 5c47352386de128b65052d63b3f3ff1081a462e3
Summary: Make `StreamBody` accept a `Stream` of `Bytes` instead of a `TryStream` of `Bytes`. This means that applications returning streaming responses will be forced to deal with errors prior to returning the response.
Reviewed By: krallin
Differential Revision: D23780216
fbshipit-source-id: dbad61947ef23bbfc4edf3d286ad0218c1859d87
Summary:
Using the `EndOnErr` combinator introduced in the previous diff, log any errors that occur during a streaming response to stderr.
Note that **the intent of this diff is to implement the most basic possible example of doing something with these errors**, with the goal of allowing us to modify `StreamBody` to only accept infallible `Stream`s.
I'd imagine that in all likelihood we'd want to do something smarter with the errors than just print them, but I figure that can be added in later diffs since it seems like doing something else (like logging the error to Scuba or adding to the RequestContext) might require additional changes that are beyond the scope of this diff.
At the very least, this seems like an improvement from before, where these errors would just get passed straight through to Hyper.
Reviewed By: krallin
Differential Revision: D23780217
fbshipit-source-id: 2f885f9fdc6af3dd28d95be1daa1d82c732453fa
Summary: Add a new `EndOnErr` combinator for `TryStream`s (exposed via the `GothamTryStreamExt::end_on_err` method) which fuses the underlying `TryStream` upon hitting an error, and passes the error to the user-provided callback. This is useful for contexts like the LFS server, where mid-stream errors are considered unrecoverable and must result in termination of the response.
Reviewed By: krallin
Differential Revision: D23778490
fbshipit-source-id: 05caa52ca62d085cc63cc8feb4619188fe0fac61
Summary: Use the new `ForwardErr` stream combinator to log errors that occur during a streaming response. Right now, they are just printed to stderr, but in the future we should also do other things such as logging them to Scuba. This diff supersedes the approach from D22720957.
Reviewed By: krallin
Differential Revision: D23780215
fbshipit-source-id: 8d2267f1166e665a62a167a6d95bb0b1797b5767
Summary: Implement `ContentMeta` for streams wrapped with the `ForwardErr` combinator, so that they may be used as input to `StreamBody`. (Presently, this won't actually work since `StreamBody` expects a `TryStream`, but this will change later in the stack.)
Reviewed By: krallin
Differential Revision: D23777842
fbshipit-source-id: 234bcdf104afbf2c9832fbe54d85744bfb470363
Summary:
This diff adds a new `ForwardErr` combinator that allows redirecting the errors from a `TryStream` into a channel, allowing them to be processed asynchronously without disrupting the stream itself. This effectively splits the `TryStream` into two `Stream`s containing the successful items and errors respectively.
To make it easy to use the combinator, I've added a new `GothamTryStreamExt` extension trait (in the vein of the old `futures_ext` crate) that allows the user to simply call `forward_err` on any `TryStream`. The trait name is a bit of a misnomer (suggestions welcome!) in that there isn't anything Gotham-specific about it, but the name `TryStreamExt` is taken and I didn't want to set up a successor to `futures_ext` just for the sake of one combinator. (Though we will likely expand the trait in the future.)
Reviewed By: krallin
Differential Revision: D23777501
fbshipit-source-id: 8628cdc2e104cd9b972afda745858f9cb9e85245
Summary:
This extends the Ctrl+C special handling from edenapi to the entire Python
command so Ctrl+C should be able to exit the program even if it's running
some blocking Rust functions.
`edenapi` no longer needs to spawn threads for fetching.
Reviewed By: singhsrb
Differential Revision: D23759710
fbshipit-source-id: cbaaa8e5f93d8d74a8692117a00d9de20646d232
Summary:
on macOS we cannot safely use `fork`.
This commit replaces the use of `fork` in the startup logger subsystem.
This was a little tricky to untangle; originally (prior to any of
the `fork` removal efforts in this diff stack), the startup flow was
to spawn a set of processes via fork:
```
edenfs (setuid)
\-----edenfs (privhelper, as root)
\------edenfs (daemonized)
```
The forked children take advantage of being able to implicitly pass state to
the child processes from the parent. That data flow needs to become explicit
when removing the fork which makes some things a little awkward.
With fork removed:
* `edenfs` unconditionally spawns `edenfs_privhelper` while it has
root privs and before most of the process has been initialized.
* That same `edenfs` process will then spawn a child `edenfs`
process which starts from scratch, but that which needs to
run as the real server instance
* The original `edenfs` instance needs to linger for a while
to remain connected to the controlling tty to pass back the
startup state to the user, before terminating.
This commit deletes the check that `edenfs` is started originally
as root; previously the logic relied on the forked startup logger
continuing past the `daemonizeIfRequested` call and simply deferring
the check until after folly::init. With these changes we can't
easily perform such a check without adding some extra gymnastics
to pass the state around; the place where that is checked is in
the spawned child of the original edenfs, which is not a privileged
process and doesn't know the original euid. I don't believe this
to be a great loss as we tuck `edenfs` away under the libexec dir.
Reviewed By: chadaustin
Differential Revision: D23696569
fbshipit-source-id: 55b95daf022601a4699274d696af419f0a11f6f2
Summary:
On macOS we cannot safely use `fork` to spawn processes while other threads may initialize objc classes.
This commit replaces the use of `fork` in the privhelper startup with
`SpawnedProcess` instead. We need to take care with this as we are generally
installed setuid root and we'd like to avoid being tricked into running an
arbitrary child process as root.
This commit defines a separate executable called `edenfs_privhelper` that
contains just the privhelper server code.
We need to be careful about locating this executable; to avoid invoking an
arbitrary process while we have root privileges we require that the privhelper
be a sibling to the edenfs executable and carry out some additional ownership
verification so that we can tell that the owner of edenfs also controls
edenfs_privhelper.
To facilitate this, I've added an `executablePath` function to PathFuncs; it
returns the path to the current executable image.
To make the integration test scenario simpler, I've added the edenfs_executable
binary definition alongside that of the edenfs binary in the buck and cmake
build systems. This causes the binaries to be siblings in-situ in the build
tree and avoids the need to move things into place in the test harness.
Reviewed By: chadaustin
Differential Revision: D23653343
fbshipit-source-id: 3c2539a5e0e11cee88960db49c885ce0366d314e
Summary: This diff fixes the eden redirection tests so it runs on Windows.
Reviewed By: xavierd
Differential Revision: D22958766
fbshipit-source-id: 45d26587831ed74d6bd7912b22c7c955b077f571
Summary:
Move bunch of code into a separate file (scm daemon related options). Move them
out of cloud sync.
Also introduce additional check that the `hg cloud sync` command scm daemon
runs is intended for the current connected workspace
In theory when we switch a subscription, the SCM daemon gets notified but races possible and it is better to have this additional check, so SCM daemon triggers cloud sync where it is supposed to.
Reviewed By: markbt
Differential Revision: D23783616
fbshipit-source-id: b91a8b79189b7810538c15f8e61080b41abde386
Summary:
The config is not actually used any more (with rust-commits, it is forced on, without rust-commits,
there is no point to keep it on). Therefore removed.
Reviewed By: singhsrb
Differential Revision: D23771570
fbshipit-source-id: ad3e89619ac5e193ef552c25fc064ca9eddba0c6
Summary:
See the previous diff for context. This allows the code to run from non-main
thread.
Reviewed By: singhsrb
Differential Revision: D23759712
fbshipit-source-id: 044193a9d7193488c700d769da9ad68987356d69
Summary:
The idea is to extend D22703916 (61712e381c)'s way of calling functions from just edenapi to
the entire command for better Ctrl+C handling. Some code paths (ex. pager,
crecord) use `signal.signal` and `signal.signal` does not work from non-main
thread.
To workaround the `signal.signal` limitation, we pre-register all signals we care
about in the main thread to a special handler. The special handler reads a
global variable to decide what to do. Other threads can modify that global
variable to affect what the special signal handler does, therefore indirectly
"register" their handles.
Reviewed By: kulshrax
Differential Revision: D23759711
fbshipit-source-id: 8ba389072433e68a36360db6a1b17638e40faefa
Summary:
Before this change, for a long-running function wrapped by 'threaded',
it might:
background thread> start
main thread> receive SIGINT, raise KeyboardInterrupt
main thread> raise at 'thread.join(1)'
main thread> exiting, but wait for threads to complete (Py_Finalize)
background thread> did not receive KeyboardInterrupt, continue running
main thread> continue waiting for background thread
Teach `thread.join(1)` to forward the `KeyboardInterrupt` (or its subclass
`error.SignalInterrupt`) to the background thread, so the background thread
_might_ stop. Besides, label the background thread as daemon so it won't
be waited upon exit.
Reviewed By: kulshrax
Differential Revision: D23759713
fbshipit-source-id: 91893d034f1ad256007ab09b7a8b974325157ea5
Summary:
Move the wrapper to util.py. It'll be used in dispatch.py to make the entire
command Ctrl+C friendly.
Reviewed By: singhsrb
Differential Revision: D23759715
fbshipit-source-id: fa2098362413dcfd0b68e05455aad543a6980907
Summary: This will be used to test Ctrl+C handling with native code.
Reviewed By: kulshrax
Differential Revision: D23759714
fbshipit-source-id: 50da40d475b80da26b7dbc654e010d77cb0ad2d1
Summary: This makes it easier to test the API via debugshell.
Reviewed By: kulshrax
Differential Revision: D23750677
fbshipit-source-id: e29284395f03c9848cf90dd2df187e437890c56e
Summary: It is handy to test edenapi methods directly.
Reviewed By: kulshrax
Differential Revision: D23750709
fbshipit-source-id: 33c15cecaa0372ba9e4688502e7d8f3fdda7c3b8
Summary:
Add a command to rebuild the changelog without recloning other parts of the
repo. This can be used as a way to recover from corrupted changelog. It
currently uses revlog because revlog is still the only supported format during
streamclone.
In the future this can be used for defragmentation.
Reviewed By: DurhamG
Differential Revision: D23720215
fbshipit-source-id: 6db0453d18dbf553660d55d528f990a4029d9da4
Summary:
This commit moves a compile-time template parameter
to be a runtime boolean parameter.
There's a bit of fan-out that, while I don't think it is
super awesome, isn't super terrible either.
The case sensitivity value is read from the checkout config
added in the prior diff in this stack.
Reviewed By: xavierd
Differential Revision: D23751192
fbshipit-source-id: 46f6fe25bfa6666305096ad9c416b510cd3aac8f
Summary:
This diff teaches the CheckoutConfig how to determine
whether a given checkout should be case-sensitive (the default)
or case-insensitive-case-preserving.
This option is passed through to the fuse channel initialization,
so that the kernel will respect it, however, our DirEntry layer
doesn't yet know that it should respect this.
There's currently no UI to set this option. My game plan
is to suggest the following steps to folks that want to try
this out:
```
$ eden stop
$ vim ~/local/.eden/clients/ovrsource/config.toml
```
and then add this line to the `[repository]` section:
```
case-sensitive = false
```
and finally:
```
$ eden start
```
Reviewed By: xavierd
Differential Revision: D23751184
fbshipit-source-id: 6facb23c460cfff6e37d0091b51b97ab06f62c91
Summary:
Improve help to reflect that the system is also meant for managing backups
add missing commands
reshuffle a bit
Reviewed By: markbt
Differential Revision: D23782794
fbshipit-source-id: d7fd3fa06ca7acd649cef557f3fe020295259e3d
