Summary:
We've seen what appears to be phantom calls to shutdown() so we'd like
to add some degree of auditing. This diff adds a new method with some
context; this will allow us to distinguish between `eden stop`, `eden restart`
and eden server internal calls to the `shutdown` method. It may still
be possible that something else is calling our shutdown method, but it
seems unlikely as we're only accessible to our own code via a unix domain
socket.
Reviewed By: chadaustin
Differential Revision: D8341595
fbshipit-source-id: 50d58ea0b56e5f42cd37c404048d710bde0d13a3
Summary:
I've noticed that we don't shut down aux processes when
we remove a mount point.
This can cause `eden rm` to hang if the buck daemon is still running in the background.
Reviewed By: strager
Differential Revision: D8499623
fbshipit-source-id: 9a7f515820a3ffaec86c8c81110283bc0eec3750
Summary:
D7946842 moved some of the daemon management code from util.py to daemon.py
but accidentally broke handling of TimeoutError exceptions. The code in
daemon.py was trying to catch python's built-in `TimeoutError` type, while the
code in util.py throws its own `util.TimeoutError` type.
This removes the custom `TimeoutError` type in util.py and just uses the
built-in python type to be consistent everywhere.
Reviewed By: wez
Differential Revision: D8373739
fbshipit-source-id: 8523c444fef74b0758c38b4fe2644d4f68d38442
Summary:
Update the `SnapshotDirstateConsistencyCheck` to catch exceptions thrown when
trying to read the `.hg/dirstate` file.
Reviewed By: wez
Differential Revision: D8373862
fbshipit-source-id: 875330c5fc4c8cbb7ba0393e02d7fbe7722d69fd
Summary:
Add a command line option to start edenfs only if there is at least one
Eden checkout already configured.
This makes it easier to automatically start edenfs on system boot only if the
user needs edenfs.
Reviewed By: wez
Differential Revision: D8174819
fbshipit-source-id: bf585b8c5aa719d3b6424567f2e21c1c63850d66
Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.
Reviewed By: bolinfest
Differential Revision: D8108649
fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: cooperlees
Differential Revision: D8198776
fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
Summary:
This removes the main point of contention for eden prefetch
in two ways:
1. We batch up the complete list of blobs so that they can be processed
in bulk rather than stalling the tree walk
2. We can ask remotefilelog to check and fetch that list to the local
hgcache, again as a batch, rather than by forcing the data to be
loaded through into the local store
The goal of this prefetch is to bulk load data from the mercurial server
so that a subsequent file access doesn't have to make a one-off ssh session
for each one, rather than making sure that all the data is loaded into
the local store.
Reviewed By: chadaustin
Differential Revision: D7965818
fbshipit-source-id: 753400460d633b5467c5110e3f5608ce06106e00
Summary:
This is a first pass at a prefetcher. The idea is simple,
but the execution is impeded by some unfortunate slowness in different
parts of mercurial.
The idea is that you pass a list of glob patterns and we'll do something
to make accessing files that match those patterns ideally faster than
if you didn't give us the prefetch hint.
In theory we could run `hg prefetch -I PATTERN` for this, but prefetch
takes several minutes materializing and walking the whole manifest to
find matches, checking outgoing revs and various other overheads.
There is a revision flag that can be specified to try to reduce this
effort, but it still takes more than a minute.
This diff:
* Removes a `Future::get()` call in the GlobNode code
* Makes `globFiles` use Futures directly rather than `Future::get()`
* Adds a `prefetchFiles` parameter to `globFiles`
* Adds `eden prefetch` to the CLI and makes it call `globFiles` with
`prefetchFiles=true`
* Adds the abillity to glob over `Tree` as well as the existing `TreeInode`.
This means that we can avoid allocating inodes for portions of the
tree that have not yet been loaded.
When `prefetchFiles` is set we'll ask ObjectStore to load the blob for
matching files. I'm not currently doing this in the `TreeInode` case
on the assumption that we already did this earlier when its `TreeInode::prefetch`
method was called.
The glob executor joins the blob prefetches at each GlobNode level. It may
be possible to observe higher throughput if we join the complete set at the
end.
Reviewed By: chadaustin
Differential Revision: D7825423
fbshipit-source-id: d2ae03d0f62f00090537198095661475056e968d
Summary:
When calling mercurial from inside the Eden CLI, use `$EDEN_HG_BINARY` as the
path to hg if this variable is set. If it is not set, continue the current
behavior of using `hg` from the user's `$PATH`. This is primarily used during
the `eden clone` command.
This makes sure the Eden integration tests use `hg` built from the local
repository rather than the version of `hg` installed on the system. Most
locations in the integration tests were already doing so, but `eden clone` was
one place that still ended up using the system hg binary.
Reviewed By: wez
Differential Revision: D7839850
fbshipit-source-id: da801fad0767a111b3e3dfd393d82da8e2147e22
Summary: Notice this while checking something for a new hire yesterday.
Reviewed By: bolinfest
Differential Revision: D8119130
fbshipit-source-id: 8a27797061b2316c62f72e34c9f20130d97dc2b1
Summary:
This is used to dump the raw `JournalDelta` entries in the journal.
Hopefully this will help us figure out what's happening in T28686395, or more
generally, why we see get those merge errors that appear in the logs of the
form:
```
Journal for .hg/rebasestate holds invalid Created, Created sequence
```
from `eden/fs/journal/JournalDelta.cpp`.
(Note: this ignores all push blocking failures!)
Reviewed By: wez
Differential Revision: D7855071
fbshipit-source-id: f195813695bec7426329a9aacd84a9b1613feec9
Summary:
Add a `remove` command to the Eden CLI. This behaves like
`eden unmount --destroy`, but calling this "remove" is hopefully a more
intuitive UI. If stdin is a TTY this command also prompts the user for
confirmation before removing the checkout.
I plan to deprecate the `eden unmount --destroy` command in a subsequent
diff.
Reviewed By: wez
Differential Revision: D8086823
fbshipit-source-id: 562cf0f998eea416b80589b188eee255a10b9699
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
This moves some things around in order to facilitate adding the migration
command in a separate file.
Reviewed By: bolinfest
Differential Revision: D7946842
fbshipit-source-id: 54a554fb02e83a12f1d626b81377bc042fac41aa
Summary:
Before I fixed an issue with `nuclide-connections` in D7833410, it could throw
an error. Although there was a try/except around the `check_output` call in
`eden doctor`, it caught `OSError` rather than `CalledProcessError`, which
seems like a mistake (or maybe an inadvertent evolution of the code).
While there, I also extended it to catch `ValueError` in case the stdout of the
subprocess is not valid JSON, in which case `json.loads()` will raise a
`ValueError`.
Reviewed By: chadaustin
Differential Revision: D7890571
fbshipit-source-id: 184f6f669e9d62a5fb04db29bcbab450defc226e
Summary:
For sure, there is still more we can do to improve the output of `eden doctor`
to make it easier to scan, but I thought some color to classify the status
would be helpful. With this change, `eden doctor`:
* Prints green when no issues are encountered.
* Prints red when there are issues that could not be fixed.
* Prints yellow when there were issues that were fixed.
* Prints yellow when there were issues that were not fixed because it was a dry run.
In making this change, I took `StdoutPrinter` from `eden/cli/debug.py` and moved
it into its own file, `eden/cli/stdout_printer.py`. While there, I introduced an
`AnsiEscapeCodes` class that can be passed to the constructor of `StdoutPrinter`
so the client can specify its own ANSI escape codes. The unit test uses these to
ensure the test output will be the same independent of the value of
`sys.stdout.isatty()` when the test is run.
Reviewed By: chadaustin
Differential Revision: D7890525
fbshipit-source-id: a95ff8c1685b48c2d239923cf08456ec6de757fe
Summary:
This diff works a little harder to be able to successfully
stop buck in a repo. It does so by performing a single level glob
to find the main buckconfig files and then invoking buck kill in
each of those locations.
The output from buck is suppressed as we've had reports that it
was confusing.
I've removed the code that shutdown chg; it's been causing us
problems in our integration tests, and the problematic behavior
will soon be addressed in chg itself.
Reviewed By: chadaustin
Differential Revision: D7874975
fbshipit-source-id: e9755099b1d22f2b4e3684280eb95cb9c9d11a41
Summary:
I have an upcoming change that touches this file and I don't want to pull in
unrelated edits. (I thought we had already autoformatted all of this code?)
Reviewed By: chadaustin
Differential Revision: D7855072
fbshipit-source-id: 2d2ab2ce19d438af73c30471199d15db98fa4e3a
Summary:
While testing the fbclone --eden changes I found that
we were failing to pick up the bind mounts. The root cause of
this is that the default head rev for hg repos is `.`, so if
the source repo is bare then we won't find an `.arcconfig` file
and thus won't be able to set up the bind mounts.
Reviewed By: simpkins
Differential Revision: D7782489
fbshipit-source-id: f41d3a7daf39ecd0946707cb2c3211d70c36eea2
Summary:
These daemon processes tend to keep file handles open
and block shutdown in various annoying ways, so we terminate
them prior to stopping the eden server.
Note that the chg shutdown is linux specific. We need a supported
command that does the right thing because this is fragile and won't
work on macos. cc: quark-zju.
We'd talked about prompting the user about this; I can add prompting
as a follow on diff.
Reviewed By: chadaustin
Differential Revision: D7784903
fbshipit-source-id: 47a2f6395f99b2bfde3dbd049b6ca4804d011373
Summary:
We've been talking about this for a little while;
this changes the terminology to refer to checkouts rather
than clients because that concept feels more intuitive.
Reviewed By: simpkins
Differential Revision: D7784902
fbshipit-source-id: 5e1e9be9438eed7915a95ea4141c4452e86adbfc
Summary: We set `metavar=''` to suppress the default and make things look nicer.
Reviewed By: simpkins
Differential Revision: D7783417
fbshipit-source-id: 05f970e8f964cb38e153c509cd5c71dcaa3143d0
Summary:
I found this while testing the fbclone script; clone was
not performing realpath on the destination path, so we ended up
creating and registering a mount under a non-canonical name.
That doesn't sound so bad except that unmount always canonicalizes
the path to be unmounted and this made it impossible to unmount
such a repo.
Reviewed By: chadaustin
Differential Revision: D7766366
fbshipit-source-id: cbfd484a1481d5082969fc97eaf211c18c0d30be
Summary:
Only treat edenfs mount points as stale if attempting to access them fails with
ENOTCONN. If we can successfully access the mount point then it cannot be
stale and does not need to be unmounted.
This prevents `eden doctor` from incorrectly unmounting mount points owned by
another edenfs process running on this system.
Reviewed By: bolinfest
Differential Revision: D7683503
fbshipit-source-id: 412cc17442fac1e8030dd40de0e0f642c15d1253
Summary:
This command directly invoked Eden's `checkOutRevision()` thrift call. This
bypasses mercurial, so mercurial would not know about the working directory
parent change.
If we need to we could add this back as a debug command in the future.
However, this isn't something that normal users should ever invoke directly.
Reviewed By: chadaustin
Differential Revision: D7754529
fbshipit-source-id: 8d6aad70da37e56b3b8ced1ade4c8537141ab239
Summary:
Update the Eden CLI command names to more closely match the command names used
by other Unix tools like systemctl and runit.
* `eden daemon` --> `eden start`
* `eden shutdown` --> `eden stop`
* `eden health` --> `eden status`
I have added aliases for the old command names so that they still work.
Reviewed By: chadaustin
Differential Revision: D7753780
fbshipit-source-id: af2c95dd5bcad09c395496f1cd5def082bccdc5c
Summary:
Try treating the argument as a repository path first, rather than a well-known
repository name. If we can find a mercurial or git repository at this path,
look for a .arcconfig file in the repository, and try to identify the
repository project type from this file. Use this project name to load our
default bind-mount and hooks configuration for the repository.
If the argument is not a valid repository path, fall back to treating it as a
well-known project name in our repository configs.
This also updates `eden clone` to print a few more diagnostic messages as it
works. (For instance, printing when it is starting the edenfs daemon, and
reporting the commit ID that was checked out.)
Reviewed By: wez
Differential Revision: D7739917
fbshipit-source-id: ac927b9e93039e4d1b8afa80466c2eee3a8829e9
Summary:
Define an `add_mount()` method to simplify the process of defining mounts in
the FakeMountTable. Also simplify the `lstat()` code to only try lookups using
strings. The keys in the dictionary are always strings, so there is no need to
ever try looking up entries using bytes.
Reviewed By: chadaustin
Differential Revision: D7704335
fbshipit-source-id: 9c81b43c0a196287254bcd749d096f9b148af8ac
Summary:
Add a pretty basic `restart` command to the Eden CLI.
This is relatively simple at the moment. It has `--graceful` and `--full`
options, and defaults to a full restart at the moment if neither of these
arguments is specified. We probably could add most of the arguments from the
`daemon` command to `restart`, but I'm not sure if we really want to support
that complexity in this command.
In the future we can make this command smarter and have it do a better job of
reporting the status as Eden restarts.
Reviewed By: chadaustin
Differential Revision: D7682838
fbshipit-source-id: 80c8d6604c3ca120fb71d6b8a388309218b5500c
Summary:
Refactor the Eden CLI command so that all subcommands are implemented as
subclasses. This helps keep the command line argument definitions together
with the logic for the command.
This is primarily just a code refactoring change, but I did include a few minor
behavioral changes to the help output:
- The command list is now always sorted alphabetically in the help output.
- The "help" subcommand can now show help for more than just one subcommand
deep. (e.g., `eden help stats io` now works correctly)
- I made some minor improvements to a few of the help strings.
Reviewed By: chadaustin
Differential Revision: D7673021
fbshipit-source-id: dc4c6db20a0fe7452d38bdafc6273e234dba8e4e
Summary:
Temporary fix to bump the takeover timeout to five minutes in the cli
code. I messed up and forgot to ship these changes with D7660642.
Reviewed By: simpkins
Differential Revision: D7676402
fbshipit-source-id: a22963fbf89663e44537e7946b3be5eb870f930c
Summary:
In practice, we have received a number of user reports where Nuclide had lost
track of its Watchman subscriptions and `eden doctor` failed to report it.
When the `NuclideHasExpectedWatchmanSubscriptions` check was first added to
`eden doctor`, in the repro case we had, it seemed that Nuclide would lose its
`filewatcher-<root-folder>` subscription, but would retain its
`hg-repository-watchman-subscription-primary` subscription. As such, we encoded
a heuristic where the presence of the `hg-repository-watchman-subscription-primary`
subscription without the accompanying `filewatcher-<root-folder>` subscription
was taken as a sign that (1) the user was connected to the directory via
Nuclide and (2) Nuclide's Watchman subscriptions had gotten corrupted.
Unfortunately, this was not a sufficient heuristic because it would not detect
the error case where Nuclide had lost //all// of its Watchman subscriptions!
Because `eden doctor` did not flag this, users kept using Nuclide even though
things were in a bad state, sometimes losing data as a result.
To fix that, we introduced a new tool, `nuclide-connections`, in D7598900. This
executable prints the list of local folders used as root folders in Nuclide
clients connected to the local Nuclide server. Now we no longer have to rely on
the presence of the `hg-repository-watchman-subscription-primary` Watchman
subscription as a proxy for a Nuclide connection.
This commit updates the `NuclideHasExpectedWatchmanSubscriptions` check in
`eden doctor` to use `nuclide-connections`. For every Nuclide connection that is
also an Eden mount, it verifies that the expected Watchman subscriptions exist.
As noted in the TODO in `NuclideHasExpectedWatchmanSubscriptions`, this
check is a bit brittle, but it is intended as a stopgap until we find the root cause
of the missing Watchman subscriptions and fix it.
Reviewed By: simpkins
Differential Revision: D7663214
fbshipit-source-id: 200b78ce25010222e59f3296f2412161086aa7ce
Summary: Materialized entries are stored with an unset hash. Fix the CLI.
Reviewed By: simpkins
Differential Revision: D7465192
fbshipit-source-id: 41e50c7214feb8afb1f3bf47252e4538945a5737
Summary:
My muscle memory doesn't quite know where the edenfs log file
is, so add a shortcut command to view it.
Reviewed By: simpkins
Differential Revision: D7418743
fbshipit-source-id: 290b51b63fc3573663ad53f44e50dfd073c211c8
Summary:
Update the Eden CLI to preserve the `ASAN_OPTIONS` environment variable when
starting edenfs.
Reviewed By: wez
Differential Revision: D7440148
fbshipit-source-id: c9932c5c14b5a2e040309268f95229c04715bb28
Summary:
Update the `eden clone` command to accept extra command line flags telling it
where to find the edenfs executable, as well as additional arguments to pass
to edenfs.
This enables the `test_clone_should_start_daemon()` integration test to pass in
flags to tell it how to start edenfs correctly.
Reviewed By: chadaustin
Differential Revision: D7433367
fbshipit-source-id: 0eddd93d1332e113fb85fa4b8fc87ba51d7eab2c
Summary:
Update most of the Eden CLI code to include python type annotations.
I believe the stats.py and stats_print.py are the only CLI files that do not
have complete typing information now.
Reviewed By: chadaustin
Differential Revision: D7433368
fbshipit-source-id: dfd6a064cacffeeed9147739da7064f3303de789
Summary:
Request data was coming back as null because of a race between ctx->getContextData() in FuseChannel::getOutstandingRequests() and the it initialization.
Also, added a check to verify rdata is valid before dereferencing it.
Plus, fixed debug.py where output gave an error.
Reviewed By: chadaustin
Differential Revision: D7377993
fbshipit-source-id: 8343119983c74185fd5d8cc05c2f5af63dcff99e
Summary:
Added a thrift call to return the outstanding FUSE requests.
Cli will call the thrift and print the output.
Added a unit test to test getOutstandingRequests().
Reviewed By: simpkins
Differential Revision: D7314584
fbshipit-source-id: 420790405babdb734f598e19719b487096ec53ca
Summary:
If you run `eden daemon` on a machine where sudo needs input, sudo's
output would get redirected to edenfs.log and eden daemon would appear
to hang. Worse, if you pressed ctrl-C, sudo would remain in the
background and continue to swallow keypresses from the tty, appearing
to somewhat break your shell until it gave up. The fix is to stop
redirecting stdout and stderr from Python and instead have edenfs
redirect itself into the log if given.
Reviewed By: simpkins
Differential Revision: D7223399
fbshipit-source-id: bae17b150b6594a3dd87bb88741c8dcefd1c5213
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
We would immediately satisfy the health check and
tell the user that the system is healthy and show the pid of
the prior incarnation rather than the one for the instance
that we just launched.
This diff refactors the health checking code so that we can
share the implementation between the cli and the integration
tests; the integration tests already had code to do the right
thing for this.
Reviewed By: simpkins
Differential Revision: D6944989
fbshipit-source-id: 7c0f02c875b1b81f8f1b7521add67928200b27ed
Summary:
If we fail to lstat a mount because the FUSE daemon is
unconnected, go ahead and unmount it.
Reviewed By: simpkins
Differential Revision: D6848115
fbshipit-source-id: df797b15956db24c4ac87dc6d4cd0a7177dd20ef
Summary:
This is a fairly minimal attempt at asking sandcastle to
run our tests as root. This is inspired by D6608746 but is for
the moment pretty coarse; I just want to see how successful this
is before finishing this off nicely, which means splitting our
contbuild config into two pieces along similar lines as D6608746;
one piece would run the tests as usual without root privileges
whilst the other would run just the integration tests with root
privs.
Reviewed By: chadaustin
Differential Revision: D6612669
fbshipit-source-id: 9a730b02424e939dd3509310b6265d50d1102e77
Summary:
in our sandcastle environment, eden is not currently
installed and the tests that look at the version are throwing
an error when the rpm query fails.
Don't do that!
Reviewed By: simpkins
Differential Revision: D6853305
fbshipit-source-id: d17cb1fb0253ef5ae5000695b6ec5bcda4a6a448
Summary: A minor convenience for mass mounting and unmounting.
Reviewed By: simpkins
Differential Revision: D6803003
fbshipit-source-id: 16c0d6982ba0ce2dba9900ee15013fcec8bc5ad5
