Summary:
If --config-dir was not specified when running 'eden stop', and edenfs is unresponsive, StopCmd.run calls `check_health_using_lockfile(None)`. This violates check_health_using_lockfile's contract because its argument must be a non-null `str`. The result is that 'eden stop' fails:
```
$ eden stop
warning: edenfs is not responding: Socket read failed: timed out
Traceback (most recent call last):
File "[snip]/thrift/transport/TSocket.py", line 298, in read
buff = self.handle.recv(sz)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "[snip]/eden/cli/main.py", line 1221, in run
pid = client.getPid()
[snip]
File "[snip]/thrift/transport/TSocket.py", line 305, in read
message='Socket read failed: {}'.format(str(e))
thrift.transport.TTransport.TTransportException: Socket read failed: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
[snip]
File "[snip]/eden/cli/main.py", line 1231, in run
pid = check_health_using_lockfile(args.config_dir).pid
File "[snip]/eden/cli/util.py", line 104, in check_health_using_lockfile
lockfile = os.path.join(config_dir, LOCK_FILE)
File "[snip]/posixpath.py", line 78, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```
Make 'eden stop' use the default config directory if --config-dir was not specified.
This issue was not caught by automated tests because they specify --config-dir explicitly.
This issue was not caught by type checkers because the type of argparse.Namespace's attrs unify with any type (including `str`).
Reviewed By: simpkins
Differential Revision: D13568405
fbshipit-source-id: 395aad7e3a80b91fec05969b648a51939281ffec
Summary: 'eden clone' starts the EdenFS daemon if it's not already running. If the user opted into systemd integration, make sure the daemon is started via systemd.
Reviewed By: wez
Differential Revision: D13498650
fbshipit-source-id: 8c5da579f9b79363e2d825ea7c85d423cbcc6509
Summary:
Rename `ClientConfig` to `CheckoutConfig` to move slightly closer towards
standardizing on the term "checkout" rather than "client" to refer to an Eden
mount point directory.
Reviewed By: chadaustin
Differential Revision: D13437952
fbshipit-source-id: 5086b4187eb975c860d48cadf0ccd4e5dce78e9a
Summary:
I want to allow opting into systemd using a setting in ~/.edenrc. Since should_use_experimental_systemd_mode is a global function, it can't read any configs. Make the config file visible to should_use_experimental_systemd_mode by moving it into EdenInstance.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13370823
fbshipit-source-id: b604db66954d0a08973030daae38bf6b1433e821
Summary:
If the user opted into systemd integration, make 'eden restart' restart the systemd service.
Don't support --graceful yet. --graceful will be implemented in a later diff. Mark it as explicitly unsupported for now.
Reviewed By: chadaustin
Differential Revision: D13271438
fbshipit-source-id: e505f00cbc337a2bf4da77bdea4b8faba063607c
Summary: Several tests for 'eden start', 'eden stop', and 'eden status' need to pass command-line arguments to fake_edenfs. With systemd support enabled, make 'eden start' forward daemon arguments to fake_edenfs, making these tests pass.
Reviewed By: wez
Differential Revision: D13249891
fbshipit-source-id: 9008a361fce7a5629535cc9d245b86073ee70826
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.
Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.
'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).
Reviewed By: simpkins
Differential Revision: D10849390
fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
Summary:
Add basic high level logic to fsck to begin fixing problems that are found.
This adds basic checks to decide if we should fix errors or not.
If errors are found and need to be fixed, this creates a new directory inside
the checkout state directory in `.eden` to record the actions taken by this
fsck run. This directory will contain a log file that records the actions
taken. In the future the fsck logic will also use this directory to store
copies of the corrupted inode data, and can store extracted orphan inode data
here as well.
Reviewed By: wez
Differential Revision: D12955044
fbshipit-source-id: 06c1e17a0a51fa5e2c0f2aab83b367b9358fd004
Summary: Update the CLI code to make the pyre type checker happy.
Reviewed By: wez
Differential Revision: D13035413
fbshipit-source-id: d201f2e65667e0ce1bf4a73fbb05878e8711ad16
Summary:
Move the error printing logic up into the CLI-specific `FsckCmd` class and out
of the lower-level `fsck.FilesystemChecker` class.
Reviewed By: wez
Differential Revision: D12955043
fbshipit-source-id: 4b5a1fcb791e784c736672484990b5406ee29b65
Summary:
Improve the way `eden fsck` processes its arguments. Previously it required a
single checkout path as an argument. If an `--overlay` argument was specified
it used this instead of the checkout path to find the directory to check, but
the checkout path argument was still required (but was ignored).
This changes the code to accept one or more paths as arguments. These paths
can either be the path to an Eden checkout (which does not have be currently
mounted), or the path to a checkout state directory inside the `.eden` state
directory. The `fsck` code automatically figures out what type of directory
the argument refers to and processes it correctly.
If no paths are specified `eden fsck` now automatically checks all configured
checkouts that are not currently mounted.
Reviewed By: wez
Differential Revision: D12955041
fbshipit-source-id: c37bc6752746d8ecd0f4a672d0d3b25d1c3a4fa1
Summary:
Sandcastle has several cases where we chown the entire
repository which performs terribly on Eden. As a workaround we have a
command to do this in eden without loading all the files.
Reviewed By: chadaustin
Differential Revision: D12857956
fbshipit-source-id: 36cebcc710fbcf4e1eb265df901513cf50a227b9
Summary:
This adds methods to enable and disable traces as well a
command to dump traces to an internal format.
Reviewed By: strager
Differential Revision: D10384073
fbshipit-source-id: 93122c6484f93cd1e1f8ae3ca7475c4ebcd5e06d
Summary: Doctor now detects if multiple edenfs are running for the same edenDir and recommends to kill the rogue one.
Reviewed By: strager
Differential Revision: D10098157
fbshipit-source-id: 8a785a9fee0c768696f897a4950552d141012b2a
Summary:
If 'eden stop' tries to tell edenfs to cleanly shut down, and edenfs never responds, 'eden stop' just hangs. Make 'eden stop' time out CLI -> edenfs Thrift calls. After a timeout, 'eden stop' should by stop edenfs via SIGKILL.
Note: This diff does not affect 'eden restart'.
Reviewed By: chadaustin
Differential Revision: D9982372
fbshipit-source-id: 42e498fb93fd97a2ec963e97c25f4b4101c7d524
Summary: If edenfs is unresponsive, 'eden status' hangs forever. Add a timeout to turn the hang into a user-friendly error.
Reviewed By: chadaustin
Differential Revision: D10156229
fbshipit-source-id: 9186826ae6b131a193b1499c8baac616d131357f
Summary:
wait_for_showdown doesn't use its instance parameter. Delete it.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D9980227
fbshipit-source-id: bfc2dd6d84854b20e8c04fc0391edff50d6d0a49
Summary:
Add a curses-based `eden top` command that displays the top process
names currently accessing an Eden mount over FUSE.
In the future, I'd like to extend this to count and record Thrift
calls too.
Reviewed By: strager
Differential Revision: D9477936
fbshipit-source-id: f878f2e67f4ea24c036880eb4b1162597dc04185
Summary:
Add an --overlay flag to the `eden fsck` command to allow running it on an
alternative overlay directory.
This is similar to the `--overlay` flag added to the `eden debug overlay`
command in D9445560.
At the moment a checkout path is still required when running
with `--overlay`. This is used to find the socket to talk to edenfs. This is
not actually used at the moment, but in the future we may need this in order
to help fix some of the filesystem state in some cases. (For instance to get
the current blob or tree ID for a path in the current commit.)
Reviewed By: wez
Differential Revision: D9615334
fbshipit-source-id: 5a0da55f00429f596be5d86e5303e1fcdeff9ea7
Summary:
Update all of the `eden debug` commands to use the newer `find_checkout()`
code rather than older `get_mount_path()` function.
The `find_checkout()` code makes sure that the EdenInstance actually points to
the correct edenfs instance for this checkout, and also works with checkouts
that are not currently mounted. In particular this allows
`eden debug overlay` to examine the overlay state even when the checkout is
not currently mounted.
Reviewed By: wez
Differential Revision: D9389865
fbshipit-source-id: 00578519d4805157a30c9b39abee9838925e8e76
Summary:
This adds a new `find_eden()` method to the Eden CLI code which can look up
information about the correct EdenCheckout and EdenInstance given a path.
In the future most CLI commands should switch to use this function over the
current `get_eden_instance()` and `get_mount_path()` methods. These older
APIs only work to find currently mounted checkouts, which is inadequate for
commands like `fsck` and some other debug commands that want to be able to
operate on unmounted checkouts.
This new logic is able to correctly find checkout information even for
unmounted checkouts. If the checkout is managed by an edenfs instance running
out of a non-default location it also correctly finds the state information
for Eden, rather than potentially using the wrong Eden state directory.
Reviewed By: wez
Differential Revision: D9385821
fbshipit-source-id: a6638f3c3817a595a7b7979c5cd218a2d7400f51
Summary: Fix a few minor issues that `mypy --strict` complains about.
Reviewed By: wez
Differential Revision: D9355653
fbshipit-source-id: af63825721fc964b7713df68e8618b595c91561d
Summary:
In the CLI, rename the "Config" class to "EdenInstance". This class
represents all state about a particular edenfs instance. It provides APIs for
making thrift calls to edenfs, for creating and destroying checkouts, and
generally does much more than just managing configuration.
Renaming it to "EdenInstance" also helps clarify that it is distinct from the
configuration tracked in the user's ~/.edenrc and the /etc/eden directory.
This change purely renames internal class and variable names and should not
affect any user-visible functionality.
Reviewed By: wez
Differential Revision: D9355515
fbshipit-source-id: ba5d4c3b753c6eb12a3783306dcd29e85fea3f52
Summary:
This prevents `hg status` from blowing up with a UTF-8 decode
error inside the generated thrift code.
Push safety concerns:
* This doesn't change the wire representation of the data
* Existing clients that believe it to be a string will continue to have
the same behavior
* Buck has its own copy of an older version of the thrift spec, so it will
continue to work "OK".
* When buck resyncs with our thrift file, some changes will likely be needed
to convert the byte arrays to strings or paths or whatever is appropriate
for bucks internal API
Work "OK" above means that clients that currently believe that `string` is
utf-8 encoded will have a runtime error if we ever send them a path that
is not utf-8. This is the behavior prior to this diff and will continue
to be the behavior for clients (like buck) that have an older version
of the thrift file.
Reviewed By: simpkins
Differential Revision: D9270843
fbshipit-source-id: b01135aec9152aaf5199e1c654ddd7f61c03717e
Summary:
Add the beginnings of an eden gc command. Today it's equivalent to
`eden debug clear_local_caches` followed by `eden
debug_compact_local_storage`, except that it compacts each column as
they're cleared to minimize peak disk consumption.
Eventually, it will also unload in-memory inodes, flush data from the
overlay, and clear the kernel's VFS cache too.
Reviewed By: wez
Differential Revision: D9138305
fbshipit-source-id: b303a63f601014cf38ca94c9e6f7c04394159ea8
Summary:
This replaces the `StdoutPrinter` class with a different `Output` class
interface that combines both the `StdoutPrinter` functionality and the
underlying I/O object being printed to.
Previously the doctor code had to pass around two separate `out` and
`printer` objects. This simplifies things so that only one object has to be
passed around.
This also adds some additional error checking around the curses
initialization, and falls back to a plain output object if anything goes wrong
during setup. Previously we had seen some reports of `eden doctor` failing
due to errors in the terminal setup code.
Reviewed By: strager
Differential Revision: D8870494
fbshipit-source-id: a6dbafdde9aa5e1a0336fa7d723d83fa7d2426ca
Summary:
When users run into "Transport endpoint not connected" errors they sometimes
attempt to run `eden status` or `eden doctor` to fix the problem. However
previously the CLI did not deal with being run from an invalid working
directory very well.
This updates the CLI to explicitly check if the current working directory is
valid before doing anything else. If it gets an ENOTCONN error it prints a
message telling the user how to update their shell's working directory. If
the `$PWD` environment variable points to a valid directory it attempts to
continue using that as the working directory. It records the fact that the
original working directory was invalid so that `eden doctor` can report this
error if that is the command being run.
Reviewed By: wez, strager
Differential Revision: D8731005
fbshipit-source-id: 4be55ccb43e22ed878114268e4dcc4c90c20e8e2
Summary:
This adds initial code for an `eden debug fsck` command.
At the moment this simply scans the overlay and reports errors, but does not
correct any issues. In subsequent diffs I'll update it to support correcting
some problems.
There are also several other issues that I'll improve in subsequent diffs as
well. At the moment this code uses the same functions as other CLI commands
to find the client directory, and these only work if the checkout is already
mounted. For `fsck` we generally want the checkout to be unmounted before
checking it. I also plan to clean up some of the output handling as well.
Reviewed By: strager
Differential Revision: D8813714
fbshipit-source-id: 90238e94540db1cc06c17eabf8ce3a1bb808d4b3
Summary:
Update `eden restart` to explain that the restart will be disruptive when
performing a full restart, and prompt the user for confirmation. This prompt
can be overridden with the `--force` option. Also print more messaging after
the restart completes telling users what to do if they see "Transport endpoint
not connected" errors from programs still using the old mount points.
This also updates `eden restart` to more carefully handle cases where there is
an existing edenfs daemon running but it is not fully healthy.
Reviewed By: wez
Differential Revision: D8731004
fbshipit-source-id: 05762187b47057b2930d0a6b71b0a6fdbd4aa7e5
Summary:
Update "eden clone" to create a README file inside the mount point directory
before the checkout is mounted. When Eden is running normally this file will
not be visible because the Eden checkout has been mounted on top of this
directory. However if the checkout is not currently mounted users will see
this file instead.
This should hopefully make it easier for users to figure out how to remount
their checkout on their own, and should prevent them from worrying that their
files have been deleted when the mount point simply is not mounted.
Reviewed By: chadaustin, wez
Differential Revision: D8631281
fbshipit-source-id: 7610450983687a839bdbc3558ff35e0422edc3be
Summary:
Update the python CLI code to let the C++ edenfs code handle daemonization
rather than daemonizing in the python code.
This means that we now daemonize after the `sudo` invocation rather than
before it. This allow's sudo's wrapper process to exit once edenfs starts,
rather than remaining around for the lifetime of the edenfs daemon. This
means that the original process' stdout and stderr file descriptors are
closed properly rather than being held open by the sudo wrapper.
This also removes the logic in the CLI that waits for edenfs startup and can
potentially time out and give up.
Reviewed By: strager
Differential Revision: D8508489
fbshipit-source-id: 6a38439d5596049d6f40cb6589b6865c229f736e
Summary:
Previously `eden rm` tried to look at `<path>/.eden/root` to find the root of
the mount point. This failed if the checkout was not currently mounted.
This updates the code to try looking for the exact input path in the config
file in this case. It also avoids sending an `unmount()` thrift call to
edenfs in this case as well.
Reviewed By: strager
Differential Revision: D8558765
fbshipit-source-id: a5d9db7eee702817713b089a457f25710438661e
Summary:
Previously `eden list` reported normally running active checkouts with an
`(active)` suffix, and non-mounted checkouts with no suffix.
The non-mounted situation is rare. This diff changes the command to report
non-mounted checkouts with a `(not mounted)` suffix. I removed the `(active)`
suffix from normally-running checkouts now, but let me know if people think it
is still valuable to keep this annotation in the output.
Reviewed By: chadaustin
Differential Revision: D8558770
fbshipit-source-id: ccce0f7c2031e50f2847089e49007274c31a69a7
Summary:
Update `eden unmount --destroy` to print a message recommending `eden rm`
instead. It still functions as before, but just prints this additional
message.
Also suppress the `--destroy` option from the `eden unmount --help` output.
Reviewed By: chadaustin
Differential Revision: D8558769
fbshipit-source-id: 78215a61fafeae2f7c5cfe4bc55ab0ea6682d2bc
Summary:
The rage command creates a paste by default, so tweak the wording to state
that it gathers information rather than printing it.
Reviewed By: chadaustin
Differential Revision: D8558771
fbshipit-source-id: 819a3c56a9ead173f6aadf639ac954e800c4a046
Summary:
Eden doctor checks to that configured bind mounts are present for each
Eden repository. It reports errors for missing directories and the device type is incorrect. The device type must differ from the device type of the repository's Eden path.
Eden doctor can fix (by creating) missing bind point directories and then calling the eden thrift server to mount.
Reviewed By: chadaustin
Differential Revision: D8244287
fbshipit-source-id: c78e5ecce63002761a266c5925f2d6618e648e4a
Summary:
We've seen what appears to be phantom calls to shutdown() so we'd like
to add some degree of auditing. This diff adds a new method with some
context; this will allow us to distinguish between `eden stop`, `eden restart`
and eden server internal calls to the `shutdown` method. It may still
be possible that something else is calling our shutdown method, but it
seems unlikely as we're only accessible to our own code via a unix domain
socket.
Reviewed By: chadaustin
Differential Revision: D8341595
fbshipit-source-id: 50d58ea0b56e5f42cd37c404048d710bde0d13a3
Summary:
I've noticed that we don't shut down aux processes when
we remove a mount point.
This can cause `eden rm` to hang if the buck daemon is still running in the background.
Reviewed By: strager
Differential Revision: D8499623
fbshipit-source-id: 9a7f515820a3ffaec86c8c81110283bc0eec3750
Summary:
Add a command line option to start edenfs only if there is at least one
Eden checkout already configured.
This makes it easier to automatically start edenfs on system boot only if the
user needs edenfs.
Reviewed By: wez
Differential Revision: D8174819
fbshipit-source-id: bf585b8c5aa719d3b6424567f2e21c1c63850d66
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: cooperlees
Differential Revision: D8198776
fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
Summary:
This removes the main point of contention for eden prefetch
in two ways:
1. We batch up the complete list of blobs so that they can be processed
in bulk rather than stalling the tree walk
2. We can ask remotefilelog to check and fetch that list to the local
hgcache, again as a batch, rather than by forcing the data to be
loaded through into the local store
The goal of this prefetch is to bulk load data from the mercurial server
so that a subsequent file access doesn't have to make a one-off ssh session
for each one, rather than making sure that all the data is loaded into
the local store.
Reviewed By: chadaustin
Differential Revision: D7965818
fbshipit-source-id: 753400460d633b5467c5110e3f5608ce06106e00
Summary:
This is a first pass at a prefetcher. The idea is simple,
but the execution is impeded by some unfortunate slowness in different
parts of mercurial.
The idea is that you pass a list of glob patterns and we'll do something
to make accessing files that match those patterns ideally faster than
if you didn't give us the prefetch hint.
In theory we could run `hg prefetch -I PATTERN` for this, but prefetch
takes several minutes materializing and walking the whole manifest to
find matches, checking outgoing revs and various other overheads.
There is a revision flag that can be specified to try to reduce this
effort, but it still takes more than a minute.
This diff:
* Removes a `Future::get()` call in the GlobNode code
* Makes `globFiles` use Futures directly rather than `Future::get()`
* Adds a `prefetchFiles` parameter to `globFiles`
* Adds `eden prefetch` to the CLI and makes it call `globFiles` with
`prefetchFiles=true`
* Adds the abillity to glob over `Tree` as well as the existing `TreeInode`.
This means that we can avoid allocating inodes for portions of the
tree that have not yet been loaded.
When `prefetchFiles` is set we'll ask ObjectStore to load the blob for
matching files. I'm not currently doing this in the `TreeInode` case
on the assumption that we already did this earlier when its `TreeInode::prefetch`
method was called.
The glob executor joins the blob prefetches at each GlobNode level. It may
be possible to observe higher throughput if we join the complete set at the
end.
Reviewed By: chadaustin
Differential Revision: D7825423
fbshipit-source-id: d2ae03d0f62f00090537198095661475056e968d
Summary:
Add a `remove` command to the Eden CLI. This behaves like
`eden unmount --destroy`, but calling this "remove" is hopefully a more
intuitive UI. If stdin is a TTY this command also prompts the user for
confirmation before removing the checkout.
I plan to deprecate the `eden unmount --destroy` command in a subsequent
diff.
Reviewed By: wez
Differential Revision: D8086823
fbshipit-source-id: 562cf0f998eea416b80589b188eee255a10b9699
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
This moves some things around in order to facilitate adding the migration
command in a separate file.
Reviewed By: bolinfest
Differential Revision: D7946842
fbshipit-source-id: 54a554fb02e83a12f1d626b81377bc042fac41aa
Summary:
This diff works a little harder to be able to successfully
stop buck in a repo. It does so by performing a single level glob
to find the main buckconfig files and then invoking buck kill in
each of those locations.
The output from buck is suppressed as we've had reports that it
was confusing.
I've removed the code that shutdown chg; it's been causing us
problems in our integration tests, and the problematic behavior
will soon be addressed in chg itself.
Reviewed By: chadaustin
Differential Revision: D7874975
fbshipit-source-id: e9755099b1d22f2b4e3684280eb95cb9c9d11a41
Summary:
While testing the fbclone --eden changes I found that
we were failing to pick up the bind mounts. The root cause of
this is that the default head rev for hg repos is `.`, so if
the source repo is bare then we won't find an `.arcconfig` file
and thus won't be able to set up the bind mounts.
Reviewed By: simpkins
Differential Revision: D7782489
fbshipit-source-id: f41d3a7daf39ecd0946707cb2c3211d70c36eea2
