Summary: Update type information so that `mypy --strict` no longer has any warnings.
Reviewed By: strager
Differential Revision: D8940414
fbshipit-source-id: e55533ad5c7c67bd598fae2ae57e6a8d84b47b49
Summary:
Fix the arguments to `doctor.cure_what_ails_you()`, which were updated in
D8870494.
Reviewed By: chadaustin
Differential Revision: D8940413
fbshipit-source-id: 21d45036f668c9bae9a2002c360532918eb157a5
Summary:
If the `nuclide-connections` command is not installed
`subprocess.check_output()` throws a `FileNotFound` exception. Previously the
code only caught `CalledProcessError` and `ValueError` exception. This
updates the code to catch any Exception type, just in case some other
unexpected exception type can also be thrown too.
Reviewed By: chadaustin
Differential Revision: D8945774
fbshipit-source-id: 5e6f63981c3ed6695a687b4bfbaac97ea440b537
Summary:
Add a new `eden debug logging` command to show and update the current edenfs
logging settings.
I have deprecated the `eden debug set_logging_command`, but not removed it
entirely yet. It now longer shows up in the `eden debug --help` output, and
if run it prints the `eden debug logging` command that the user should run
instead.
Reviewed By: chadaustin
Differential Revision: D8907310
fbshipit-source-id: 7ed056637b236db5a50c619023bd8025b787e227
Summary:
This replaces the `StdoutPrinter` class with a different `Output` class
interface that combines both the `StdoutPrinter` functionality and the
underlying I/O object being printed to.
Previously the doctor code had to pass around two separate `out` and
`printer` objects. This simplifies things so that only one object has to be
passed around.
This also adds some additional error checking around the curses
initialization, and falls back to a plain output object if anything goes wrong
during setup. Previously we had seen some reports of `eden doctor` failing
due to errors in the terminal setup code.
Reviewed By: strager
Differential Revision: D8870494
fbshipit-source-id: a6dbafdde9aa5e1a0336fa7d723d83fa7d2426ca
Summary:
When users run into "Transport endpoint not connected" errors they sometimes
attempt to run `eden status` or `eden doctor` to fix the problem. However
previously the CLI did not deal with being run from an invalid working
directory very well.
This updates the CLI to explicitly check if the current working directory is
valid before doing anything else. If it gets an ENOTCONN error it prints a
message telling the user how to update their shell's working directory. If
the `$PWD` environment variable points to a valid directory it attempts to
continue using that as the working directory. It records the fact that the
original working directory was invalid so that `eden doctor` can report this
error if that is the command being run.
Reviewed By: wez, strager
Differential Revision: D8731005
fbshipit-source-id: 4be55ccb43e22ed878114268e4dcc4c90c20e8e2
Summary:
This adds initial code for an `eden debug fsck` command.
At the moment this simply scans the overlay and reports errors, but does not
correct any issues. In subsequent diffs I'll update it to support correcting
some problems.
There are also several other issues that I'll improve in subsequent diffs as
well. At the moment this code uses the same functions as other CLI commands
to find the client directory, and these only work if the checkout is already
mounted. For `fsck` we generally want the checkout to be unmounted before
checking it. I also plan to clean up some of the output handling as well.
Reviewed By: strager
Differential Revision: D8813714
fbshipit-source-id: 90238e94540db1cc06c17eabf8ce3a1bb808d4b3
Summary:
The `eden debug overlay` code has some logic in python for parsing overlay
files. This moves that logic into a new module so I can re-use it for an
`fsck` command in a subsequent diff.
Reviewed By: wez
Differential Revision: D8813715
fbshipit-source-id: 6c32561db95f5da112cfd3e06b9e4653f431b94a
Summary:
This class name looks like it was copy-and-pasted incorrectly, probably from
the `eden debug set_log_level` code. Update the name to specify "Journal"
rather than "Level"
Reviewed By: strager
Differential Revision: D8813713
fbshipit-source-id: 2a5edd7f2de8279ee70dcc72b200c2b86b60f376
Summary:
Update `eden restart` to explain that the restart will be disruptive when
performing a full restart, and prompt the user for confirmation. This prompt
can be overridden with the `--force` option. Also print more messaging after
the restart completes telling users what to do if they see "Transport endpoint
not connected" errors from programs still using the old mount points.
This also updates `eden restart` to more carefully handle cases where there is
an existing edenfs daemon running but it is not fully healthy.
Reviewed By: wez
Differential Revision: D8731004
fbshipit-source-id: 05762187b47057b2930d0a6b71b0a6fdbd4aa7e5
Summary:
Fix two debug subcommands that were calling `os.getcwd()` while setting up the
argument parser. This caused the eden CLI to throw an exception of the
current working directory was invalid (for instance, if it was inside a stale
Eden mount point).
Reviewed By: strager
Differential Revision: D8731006
fbshipit-source-id: 8b7a67815d61b3476109aad646d21af8282ac68c
Summary:
Update "eden clone" to create a README file inside the mount point directory
before the checkout is mounted. When Eden is running normally this file will
not be visible because the Eden checkout has been mounted on top of this
directory. However if the checkout is not currently mounted users will see
this file instead.
This should hopefully make it easier for users to figure out how to remount
their checkout on their own, and should prevent them from worrying that their
files have been deleted when the mount point simply is not mounted.
Reviewed By: chadaustin, wez
Differential Revision: D8631281
fbshipit-source-id: 7610450983687a839bdbc3558ff35e0422edc3be
Summary:
Update the python CLI code to let the C++ edenfs code handle daemonization
rather than daemonizing in the python code.
This means that we now daemonize after the `sudo` invocation rather than
before it. This allow's sudo's wrapper process to exit once edenfs starts,
rather than remaining around for the lifetime of the edenfs daemon. This
means that the original process' stdout and stderr file descriptors are
closed properly rather than being held open by the sudo wrapper.
This also removes the logic in the CLI that waits for edenfs startup and can
potentially time out and give up.
Reviewed By: strager
Differential Revision: D8508489
fbshipit-source-id: 6a38439d5596049d6f40cb6589b6865c229f736e
Summary:
Update the C++ edenfs code to ensure that the .eden and
.eden/storage/rocks-db directories exist, rather than requiring the python CLI
code create these directories as part of `eden start`
Reviewed By: strager
Differential Revision: D8508488
fbshipit-source-id: 358521b4f5eed1d19bf37903900ca50718e2c35c
Summary:
This updates edenfs to be able to daemonize itself, and moves the
daemonization logic from the python CLI code into C++.
The main benefit of this is that we can now do a better job of reporting
messages to the user during start-up. We can log around potentially slow
operations (opening the RocksDB local store), and we can print messages
directly to the user if startup fails. Previously most failure messages would
go only to the eden log and would not be printed to the user's terminal.
This also fixes some issues where stdin and stdout were not closed properly
when daemonization was performed by the CLI. sudo needed access to these file
descriptors in case it needed to prompt for a password, and it would then hold
the descriptors open until edenfs exited.
Reviewed By: wez
Differential Revision: D8373672
fbshipit-source-id: 3272bff2208596f41d26e479c82c700d6c1efe11
Summary:
This updates the `eden doctor` code to define a `Problem` class, and to update
all of the checks to generate `Problem` objects. Problems that can be
automatically fixed derive from the `FixableProblem` subclass and implement a
`perform_fix()` method.
This provides a more consistent API to report which problems were found and
which ones can be automatically fixed. This also simplifies handling of the
`--dry-run` option, as each check no longer needs to individually check if
they are running in dry-run mode or not.
I also removed the `Check` API, since it only had a single `do_check()`
method. The code now simply uses a few separate helper functions that perform
various checks and generate `Problem` objects.
Reviewed By: wez
Differential Revision: D8617702
fbshipit-source-id: f7da2b7c17799b2844ae9416e827fd59b99d93c5
Summary:
Add a helper function for defining mount points during the test, rather than
requiring each test function to manually set up all of the individual
configurations for each mount point.
Reviewed By: wez
Differential Revision: D8581496
fbshipit-source-id: 41c8f1099292f7ebdbad30278ed530c01d083543
Summary:
Add a helper function that uses `self.addCleanup()` to destroy the temporary
directory rather than having to use `try...finally` blocks.
Reviewed By: strager
Differential Revision: D8581494
fbshipit-source-id: 3e1368c876b305c1528c5ed1de948a65a3da8199
Summary:
This updates the CLI code to put a timeout on `unmount()` thrift calls to
edenfs. We've had a few reports of situations where the unmount thrift call
has hung while edenfs was waiting for all inodes to be released.
Eventually we should fix the C++ code to put a hard timeout on the operation,
(and forcibly remove the mount point from its list in EdenServer even if it is
still waiting for some inodes to be released). However as an interim solution
timing out on the CLI is better than hanging forever. In particular the old
behavior made `eden rm` unable to clean up some mount points.
Reviewed By: pkaush
Differential Revision: D8558767
fbshipit-source-id: 942e5c7bcc147229f040f377ce59fcc2565276e0
Summary:
Previously `eden rm` tried to look at `<path>/.eden/root` to find the root of
the mount point. This failed if the checkout was not currently mounted.
This updates the code to try looking for the exact input path in the config
file in this case. It also avoids sending an `unmount()` thrift call to
edenfs in this case as well.
Reviewed By: strager
Differential Revision: D8558765
fbshipit-source-id: a5d9db7eee702817713b089a457f25710438661e
Summary:
Update `eden doctor` to automatically remount any checkouts that have been
unmounted.
In general we don't expect users to need to unmount checkouts. If they have
gotten into this state it is most likely unexpected, so have `eden doctor` fix
it for them. This is also consistent with how `edenfs` automatically remounts
all configured mount points on startup.
Reviewed By: chadaustin
Differential Revision: D8558766
fbshipit-source-id: 2e22cccead69d01530c6f646a958fc435ccce6c4
Summary:
We provide average values for vmRSS and memory (dirty bytes) on linux systems
by parsing the proc file system. This change allows the current values
(non-average) to be queried and calculated from the thrift API. The cli has
been updated to present these values (and no longer consumes the parsed smaps
files from EdenServer).
Reviewed By: chadaustin
Differential Revision: D8549305
fbshipit-source-id: 77f6838f39784e7ebeda11d8c66dba1fa9f10591
Summary:
Previously `eden list` reported normally running active checkouts with an
`(active)` suffix, and non-mounted checkouts with no suffix.
The non-mounted situation is rare. This diff changes the command to report
non-mounted checkouts with a `(not mounted)` suffix. I removed the `(active)`
suffix from normally-running checkouts now, but let me know if people think it
is still valuable to keep this annotation in the output.
Reviewed By: chadaustin
Differential Revision: D8558770
fbshipit-source-id: ccce0f7c2031e50f2847089e49007274c31a69a7
Summary:
Update `eden unmount --destroy` to print a message recommending `eden rm`
instead. It still functions as before, but just prints this additional
message.
Also suppress the `--destroy` option from the `eden unmount --help` output.
Reviewed By: chadaustin
Differential Revision: D8558769
fbshipit-source-id: 78215a61fafeae2f7c5cfe4bc55ab0ea6682d2bc
Summary:
The rage command creates a paste by default, so tweak the wording to state
that it gathers information rather than printing it.
Reviewed By: chadaustin
Differential Revision: D8558771
fbshipit-source-id: 819a3c56a9ead173f6aadf639ac954e800c4a046
Summary:
The `debug` command was previously hidden in the `eden --help` output.
There doesn't really seem to be a strong reason to hide this subcommand. It
seems useful to expose it to users who do want to poke around at their current
edenfs instance.
Reviewed By: chadaustin
Differential Revision: D8558768
fbshipit-source-id: f23fd29b473607d83f4f0aec064e72a06dd3ae45
Summary:
Remove python code to parse /proc/self/smaps and associated tests. This is now
done in the EdenServer c++ code.
Reviewed By: simpkins
Differential Revision: D8550273
fbshipit-source-id: 6a2a5cbd25beff412376b0ada683f9dc30e15aff
Summary:
If someone's having problems with Eden and they run `eden doctor` it
sometimes says 'All is well.' even though the person is still having
issues. 'No issues detected.' is a slightly humbler (and accurate)
statement.
Reviewed By: eamonnkent
Differential Revision: D8553069
fbshipit-source-id: 5dd02dc304c6aa6f75b2b2e11757ceb333f8f506
Summary:
Eden doctor checks to that configured bind mounts are present for each
Eden repository. It reports errors for missing directories and the device type is incorrect. The device type must differ from the device type of the repository's Eden path.
Eden doctor can fix (by creating) missing bind point directories and then calling the eden thrift server to mount.
Reviewed By: chadaustin
Differential Revision: D8244287
fbshipit-source-id: c78e5ecce63002761a266c5925f2d6618e648e4a
Summary:
We've seen what appears to be phantom calls to shutdown() so we'd like
to add some degree of auditing. This diff adds a new method with some
context; this will allow us to distinguish between `eden stop`, `eden restart`
and eden server internal calls to the `shutdown` method. It may still
be possible that something else is calling our shutdown method, but it
seems unlikely as we're only accessible to our own code via a unix domain
socket.
Reviewed By: chadaustin
Differential Revision: D8341595
fbshipit-source-id: 50d58ea0b56e5f42cd37c404048d710bde0d13a3
Summary:
I've noticed that we don't shut down aux processes when
we remove a mount point.
This can cause `eden rm` to hang if the buck daemon is still running in the background.
Reviewed By: strager
Differential Revision: D8499623
fbshipit-source-id: 9a7f515820a3ffaec86c8c81110283bc0eec3750
Summary:
D7946842 moved some of the daemon management code from util.py to daemon.py
but accidentally broke handling of TimeoutError exceptions. The code in
daemon.py was trying to catch python's built-in `TimeoutError` type, while the
code in util.py throws its own `util.TimeoutError` type.
This removes the custom `TimeoutError` type in util.py and just uses the
built-in python type to be consistent everywhere.
Reviewed By: wez
Differential Revision: D8373739
fbshipit-source-id: 8523c444fef74b0758c38b4fe2644d4f68d38442
Summary:
Update the `SnapshotDirstateConsistencyCheck` to catch exceptions thrown when
trying to read the `.hg/dirstate` file.
Reviewed By: wez
Differential Revision: D8373862
fbshipit-source-id: 875330c5fc4c8cbb7ba0393e02d7fbe7722d69fd
Summary:
Add a command line option to start edenfs only if there is at least one
Eden checkout already configured.
This makes it easier to automatically start edenfs on system boot only if the
user needs edenfs.
Reviewed By: wez
Differential Revision: D8174819
fbshipit-source-id: bf585b8c5aa719d3b6424567f2e21c1c63850d66
Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.
Reviewed By: bolinfest
Differential Revision: D8108649
fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: cooperlees
Differential Revision: D8198776
fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
Summary:
This removes the main point of contention for eden prefetch
in two ways:
1. We batch up the complete list of blobs so that they can be processed
in bulk rather than stalling the tree walk
2. We can ask remotefilelog to check and fetch that list to the local
hgcache, again as a batch, rather than by forcing the data to be
loaded through into the local store
The goal of this prefetch is to bulk load data from the mercurial server
so that a subsequent file access doesn't have to make a one-off ssh session
for each one, rather than making sure that all the data is loaded into
the local store.
Reviewed By: chadaustin
Differential Revision: D7965818
fbshipit-source-id: 753400460d633b5467c5110e3f5608ce06106e00
Summary:
This is a first pass at a prefetcher. The idea is simple,
but the execution is impeded by some unfortunate slowness in different
parts of mercurial.
The idea is that you pass a list of glob patterns and we'll do something
to make accessing files that match those patterns ideally faster than
if you didn't give us the prefetch hint.
In theory we could run `hg prefetch -I PATTERN` for this, but prefetch
takes several minutes materializing and walking the whole manifest to
find matches, checking outgoing revs and various other overheads.
There is a revision flag that can be specified to try to reduce this
effort, but it still takes more than a minute.
This diff:
* Removes a `Future::get()` call in the GlobNode code
* Makes `globFiles` use Futures directly rather than `Future::get()`
* Adds a `prefetchFiles` parameter to `globFiles`
* Adds `eden prefetch` to the CLI and makes it call `globFiles` with
`prefetchFiles=true`
* Adds the abillity to glob over `Tree` as well as the existing `TreeInode`.
This means that we can avoid allocating inodes for portions of the
tree that have not yet been loaded.
When `prefetchFiles` is set we'll ask ObjectStore to load the blob for
matching files. I'm not currently doing this in the `TreeInode` case
on the assumption that we already did this earlier when its `TreeInode::prefetch`
method was called.
The glob executor joins the blob prefetches at each GlobNode level. It may
be possible to observe higher throughput if we join the complete set at the
end.
Reviewed By: chadaustin
Differential Revision: D7825423
fbshipit-source-id: d2ae03d0f62f00090537198095661475056e968d
Summary:
When calling mercurial from inside the Eden CLI, use `$EDEN_HG_BINARY` as the
path to hg if this variable is set. If it is not set, continue the current
behavior of using `hg` from the user's `$PATH`. This is primarily used during
the `eden clone` command.
This makes sure the Eden integration tests use `hg` built from the local
repository rather than the version of `hg` installed on the system. Most
locations in the integration tests were already doing so, but `eden clone` was
one place that still ended up using the system hg binary.
Reviewed By: wez
Differential Revision: D7839850
fbshipit-source-id: da801fad0767a111b3e3dfd393d82da8e2147e22
Summary: Notice this while checking something for a new hire yesterday.
Reviewed By: bolinfest
Differential Revision: D8119130
fbshipit-source-id: 8a27797061b2316c62f72e34c9f20130d97dc2b1
Summary:
This is used to dump the raw `JournalDelta` entries in the journal.
Hopefully this will help us figure out what's happening in T28686395, or more
generally, why we see get those merge errors that appear in the logs of the
form:
```
Journal for .hg/rebasestate holds invalid Created, Created sequence
```
from `eden/fs/journal/JournalDelta.cpp`.
(Note: this ignores all push blocking failures!)
Reviewed By: wez
Differential Revision: D7855071
fbshipit-source-id: f195813695bec7426329a9aacd84a9b1613feec9
Summary:
Add a `remove` command to the Eden CLI. This behaves like
`eden unmount --destroy`, but calling this "remove" is hopefully a more
intuitive UI. If stdin is a TTY this command also prompts the user for
confirmation before removing the checkout.
I plan to deprecate the `eden unmount --destroy` command in a subsequent
diff.
Reviewed By: wez
Differential Revision: D8086823
fbshipit-source-id: 562cf0f998eea416b80589b188eee255a10b9699
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
This moves some things around in order to facilitate adding the migration
command in a separate file.
Reviewed By: bolinfest
Differential Revision: D7946842
fbshipit-source-id: 54a554fb02e83a12f1d626b81377bc042fac41aa
Summary:
Before I fixed an issue with `nuclide-connections` in D7833410, it could throw
an error. Although there was a try/except around the `check_output` call in
`eden doctor`, it caught `OSError` rather than `CalledProcessError`, which
seems like a mistake (or maybe an inadvertent evolution of the code).
While there, I also extended it to catch `ValueError` in case the stdout of the
subprocess is not valid JSON, in which case `json.loads()` will raise a
`ValueError`.
Reviewed By: chadaustin
Differential Revision: D7890571
fbshipit-source-id: 184f6f669e9d62a5fb04db29bcbab450defc226e
Summary:
For sure, there is still more we can do to improve the output of `eden doctor`
to make it easier to scan, but I thought some color to classify the status
would be helpful. With this change, `eden doctor`:
* Prints green when no issues are encountered.
* Prints red when there are issues that could not be fixed.
* Prints yellow when there were issues that were fixed.
* Prints yellow when there were issues that were not fixed because it was a dry run.
In making this change, I took `StdoutPrinter` from `eden/cli/debug.py` and moved
it into its own file, `eden/cli/stdout_printer.py`. While there, I introduced an
`AnsiEscapeCodes` class that can be passed to the constructor of `StdoutPrinter`
so the client can specify its own ANSI escape codes. The unit test uses these to
ensure the test output will be the same independent of the value of
`sys.stdout.isatty()` when the test is run.
Reviewed By: chadaustin
Differential Revision: D7890525
fbshipit-source-id: a95ff8c1685b48c2d239923cf08456ec6de757fe
Summary:
This diff works a little harder to be able to successfully
stop buck in a repo. It does so by performing a single level glob
to find the main buckconfig files and then invoking buck kill in
each of those locations.
The output from buck is suppressed as we've had reports that it
was confusing.
I've removed the code that shutdown chg; it's been causing us
problems in our integration tests, and the problematic behavior
will soon be addressed in chg itself.
Reviewed By: chadaustin
Differential Revision: D7874975
fbshipit-source-id: e9755099b1d22f2b4e3684280eb95cb9c9d11a41
Summary:
I have an upcoming change that touches this file and I don't want to pull in
unrelated edits. (I thought we had already autoformatted all of this code?)
Reviewed By: chadaustin
Differential Revision: D7855072
fbshipit-source-id: 2d2ab2ce19d438af73c30471199d15db98fa4e3a
