Summary:
Eden should load its configuration from toml files. This change includes:
- ConfigParser is the class that parses and provides access to the configuration. We load the configuration from toml files and pass to ConfigParser in order to preserve existing behavior.
- The toml configuration format differs from the Config format. Thus, we use toml.load() and toml.dump() to load/save toml format configuration.
- Tests
- configuration is properly loaded from a group of config files
- toml configuration (if present) takes precedence
- missing configuration files (/home/user/.edenrc) are ignored
- invalid configuration files are fatal
- update of configuration works (adding repo)
- toml parsing handles over-rides of properties in the same way as cfg files
Deployment Strategy:
- This code determines whether to use toml configuration based on the presence
of atleast 1 '.toml' file in the etc-eden-dir/config.d.
Reviewed By: chadaustin
Differential Revision: D9023232
fbshipit-source-id: 6734a9d91eca92b05872a758c764546451dd2d51
Summary:
This replaces the `StdoutPrinter` class with a different `Output` class
interface that combines both the `StdoutPrinter` functionality and the
underlying I/O object being printed to.
Previously the doctor code had to pass around two separate `out` and
`printer` objects. This simplifies things so that only one object has to be
passed around.
This also adds some additional error checking around the curses
initialization, and falls back to a plain output object if anything goes wrong
during setup. Previously we had seen some reports of `eden doctor` failing
due to errors in the terminal setup code.
Reviewed By: strager
Differential Revision: D8870494
fbshipit-source-id: a6dbafdde9aa5e1a0336fa7d723d83fa7d2426ca
Summary:
This updates the `eden doctor` code to define a `Problem` class, and to update
all of the checks to generate `Problem` objects. Problems that can be
automatically fixed derive from the `FixableProblem` subclass and implement a
`perform_fix()` method.
This provides a more consistent API to report which problems were found and
which ones can be automatically fixed. This also simplifies handling of the
`--dry-run` option, as each check no longer needs to individually check if
they are running in dry-run mode or not.
I also removed the `Check` API, since it only had a single `do_check()`
method. The code now simply uses a few separate helper functions that perform
various checks and generate `Problem` objects.
Reviewed By: wez
Differential Revision: D8617702
fbshipit-source-id: f7da2b7c17799b2844ae9416e827fd59b99d93c5
Summary:
Add a helper function for defining mount points during the test, rather than
requiring each test function to manually set up all of the individual
configurations for each mount point.
Reviewed By: wez
Differential Revision: D8581496
fbshipit-source-id: 41c8f1099292f7ebdbad30278ed530c01d083543
Summary:
Add a helper function that uses `self.addCleanup()` to destroy the temporary
directory rather than having to use `try...finally` blocks.
Reviewed By: strager
Differential Revision: D8581494
fbshipit-source-id: 3e1368c876b305c1528c5ed1de948a65a3da8199
Summary:
Update `eden doctor` to automatically remount any checkouts that have been
unmounted.
In general we don't expect users to need to unmount checkouts. If they have
gotten into this state it is most likely unexpected, so have `eden doctor` fix
it for them. This is also consistent with how `edenfs` automatically remounts
all configured mount points on startup.
Reviewed By: chadaustin
Differential Revision: D8558766
fbshipit-source-id: 2e22cccead69d01530c6f646a958fc435ccce6c4
Summary:
Remove python code to parse /proc/self/smaps and associated tests. This is now
done in the EdenServer c++ code.
Reviewed By: simpkins
Differential Revision: D8550273
fbshipit-source-id: 6a2a5cbd25beff412376b0ada683f9dc30e15aff
Summary:
If someone's having problems with Eden and they run `eden doctor` it
sometimes says 'All is well.' even though the person is still having
issues. 'No issues detected.' is a slightly humbler (and accurate)
statement.
Reviewed By: eamonnkent
Differential Revision: D8553069
fbshipit-source-id: 5dd02dc304c6aa6f75b2b2e11757ceb333f8f506
Summary:
Eden doctor checks to that configured bind mounts are present for each
Eden repository. It reports errors for missing directories and the device type is incorrect. The device type must differ from the device type of the repository's Eden path.
Eden doctor can fix (by creating) missing bind point directories and then calling the eden thrift server to mount.
Reviewed By: chadaustin
Differential Revision: D8244287
fbshipit-source-id: c78e5ecce63002761a266c5925f2d6618e648e4a
Summary:
Update the `SnapshotDirstateConsistencyCheck` to catch exceptions thrown when
trying to read the `.hg/dirstate` file.
Reviewed By: wez
Differential Revision: D8373862
fbshipit-source-id: 875330c5fc4c8cbb7ba0393e02d7fbe7722d69fd
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: cooperlees
Differential Revision: D8198776
fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
Summary: Notice this while checking something for a new hire yesterday.
Reviewed By: bolinfest
Differential Revision: D8119130
fbshipit-source-id: 8a27797061b2316c62f72e34c9f20130d97dc2b1
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
For sure, there is still more we can do to improve the output of `eden doctor`
to make it easier to scan, but I thought some color to classify the status
would be helpful. With this change, `eden doctor`:
* Prints green when no issues are encountered.
* Prints red when there are issues that could not be fixed.
* Prints yellow when there were issues that were fixed.
* Prints yellow when there were issues that were not fixed because it was a dry run.
In making this change, I took `StdoutPrinter` from `eden/cli/debug.py` and moved
it into its own file, `eden/cli/stdout_printer.py`. While there, I introduced an
`AnsiEscapeCodes` class that can be passed to the constructor of `StdoutPrinter`
so the client can specify its own ANSI escape codes. The unit test uses these to
ensure the test output will be the same independent of the value of
`sys.stdout.isatty()` when the test is run.
Reviewed By: chadaustin
Differential Revision: D7890525
fbshipit-source-id: a95ff8c1685b48c2d239923cf08456ec6de757fe
Summary:
Only treat edenfs mount points as stale if attempting to access them fails with
ENOTCONN. If we can successfully access the mount point then it cannot be
stale and does not need to be unmounted.
This prevents `eden doctor` from incorrectly unmounting mount points owned by
another edenfs process running on this system.
Reviewed By: bolinfest
Differential Revision: D7683503
fbshipit-source-id: 412cc17442fac1e8030dd40de0e0f642c15d1253
Summary:
Define an `add_mount()` method to simplify the process of defining mounts in
the FakeMountTable. Also simplify the `lstat()` code to only try lookups using
strings. The keys in the dictionary are always strings, so there is no need to
ever try looking up entries using bytes.
Reviewed By: chadaustin
Differential Revision: D7704335
fbshipit-source-id: 9c81b43c0a196287254bcd749d096f9b148af8ac
Summary:
In practice, we have received a number of user reports where Nuclide had lost
track of its Watchman subscriptions and `eden doctor` failed to report it.
When the `NuclideHasExpectedWatchmanSubscriptions` check was first added to
`eden doctor`, in the repro case we had, it seemed that Nuclide would lose its
`filewatcher-<root-folder>` subscription, but would retain its
`hg-repository-watchman-subscription-primary` subscription. As such, we encoded
a heuristic where the presence of the `hg-repository-watchman-subscription-primary`
subscription without the accompanying `filewatcher-<root-folder>` subscription
was taken as a sign that (1) the user was connected to the directory via
Nuclide and (2) Nuclide's Watchman subscriptions had gotten corrupted.
Unfortunately, this was not a sufficient heuristic because it would not detect
the error case where Nuclide had lost //all// of its Watchman subscriptions!
Because `eden doctor` did not flag this, users kept using Nuclide even though
things were in a bad state, sometimes losing data as a result.
To fix that, we introduced a new tool, `nuclide-connections`, in D7598900. This
executable prints the list of local folders used as root folders in Nuclide
clients connected to the local Nuclide server. Now we no longer have to rely on
the presence of the `hg-repository-watchman-subscription-primary` Watchman
subscription as a proxy for a Nuclide connection.
This commit updates the `NuclideHasExpectedWatchmanSubscriptions` check in
`eden doctor` to use `nuclide-connections`. For every Nuclide connection that is
also an Eden mount, it verifies that the expected Watchman subscriptions exist.
As noted in the TODO in `NuclideHasExpectedWatchmanSubscriptions`, this
check is a bit brittle, but it is intended as a stopgap until we find the root cause
of the missing Watchman subscriptions and fix it.
Reviewed By: simpkins
Differential Revision: D7663214
fbshipit-source-id: 200b78ce25010222e59f3296f2412161086aa7ce
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
If we fail to lstat a mount because the FUSE daemon is
unconnected, go ahead and unmount it.
Reviewed By: simpkins
Differential Revision: D6848115
fbshipit-source-id: df797b15956db24c4ac87dc6d4cd0a7177dd20ef
Summary:
My stale mounts check regressed eden doctor's handling of
unconfigured mounts. For now, just skip them, and add a test that
eden doctor doesn't crash.
Reviewed By: simpkins
Differential Revision: D6801445
fbshipit-source-id: 65d2f9028af651ef487a7ce6c334f387b541492d
Summary:
The prior implementation of StaleMountsCheck filtered by path
and did not correctly handle seeing the same FUSE mount multiple times
in the mount table. This occurred when an Eden mount was created
underneath a bind mount.
Now it only unmounts mounts where st_dev does not match the st_dev of
any active mounts, and where st_uid matches the current user.
Reviewed By: simpkins
Differential Revision: D6787618
fbshipit-source-id: 24e0f156cb74822500d91205349c0e6638c0340c
Summary:
If the Eden process dies or malfunctions, it's possible to end up with stale
edenfs mounts on the system. Change `eden doctor` to correct them.
Reviewed By: simpkins
Differential Revision: D6659193
fbshipit-source-id: d9fcf2e68663295e4f43b2c11fd4503a1dfac038
Summary:
Added --version to main.py, including -v and version.
It prints both installed and running version (as per Wez's suggestion), and accounts for the possibilities that eden may not be running at all, or dev version may be running.
Reviewed By: wez
Differential Revision: D6724204
fbshipit-source-id: 5085f53a00a557f759a23fe41fb57189c9ad6a7e
Summary:
Changed `eden clone` to check out master commit in both git and hg.
Previously, it checked out the current commit for the repo.
Reviewed By: simpkins
Differential Revision: D6663754
fbshipit-source-id: 92b185ccca5d082dc2bde9c8b191c82a2a4f06b4
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary: Previously, this would cause a crash when running `eden doctor`. Now fixed.
Reviewed By: simpkins
Differential Revision: D6607211
fbshipit-source-id: a0e077beaf6b7031d57efd72b947e90884369852
Summary:
`eden stats` used to show system memory usage which was not very
interesting (and can be gleaned from top). Instead read the contents
of /proc/self/smaps and sum the Private_Dirty fields to get a number
that more accurately reflects impact on the rest of the system.
Reviewed By: wez
Differential Revision: D6575595
fbshipit-source-id: 9badc5cd5a1b56d3ccb27edd1a2d20ee74ec34ae
Summary:
I forgot that `subprocess.check_output()` returns a `bytes` instead of a `str`.
This caused `eden doctor` to return false positives because the installed and
running versions of `edenfs` would be the same, but the check would fail because
it was comparing a `bytes` to a `str`.
Reviewed By: simpkins
Differential Revision: D6577031
fbshipit-source-id: 681ea22ef79604a3dfb278d713e5c68c54d8ecd5
Summary:
In the initial implementation of `eden doctor`, if you had a mount point like:
/data/users/mbolin/eden-fbsource
then it looked for a pair of Watchman subscriptions for
`/data/users/mbolin/eden-fbsource` with the names:
hg-repository-watchman-subscription-primary
filewatcher-/data/users/mbolin/eden-fbsource
as a heuristic that Nuclide was being used to edit files in that Eden mount.
However, if the user is using Nuclide to edit files in a subdirectory of that
Eden mount (as determined by the Nuclide connection profile), then the
subscriptions look like this:
hg-repository-watchman-subscription-primary
filewatcher-/data/users/mbolin/eden-fbsource/<path-to-subdirectory>
In this case, `eden doctor` was returning a false positive because it thought
that the `filewatcher-/data/users/mbolin/eden-fbsource` subscription was missing
when in fact everything was just fine.
This revision changes the logic to look for a subscription to the root or one of
its subdirectories.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6467113
fbshipit-source-id: bfe5fafaa388405f2e59e61315c5a2084c8adc56
Summary:
This makes the `expected_output` in `stats_test.py` cleaner to declare.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6465282
fbshipit-source-id: 80b697c84f28a0fece77eb1050c4364b14685c0b
Summary:
Presumably for historical reasons, we still had some tests hanging around with
`from __future__ import absolute_import` in the header even though all of our
tests should be Python 3. I converted these stragglers and added a smattering of
type annotations to enforce their Python 3-ness.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6464805
fbshipit-source-id: 6177d7663ab428a0dbbecb287f340d770679551d
Summary: This is not material to the test, so it's distracting.
Reviewed By: wez
Differential Revision: D6462338
fbshipit-source-id: f289516c7e67891f1fed74df721fa1eb24e5f368
Summary:
`eden doctor` is a new subcommand that attempts to diagnose
issues with Eden and autofix them, as appropriate. There is also
a `--dry-run` flag that will tell the user about issues without attempting
to fix them.
Initially, `eden doctor` checks for the following:
- If Watchman has an `inotify` watcher, `eden doctor` identifies this and
will attempt to replace it with an `eden` watcher.
- If there are some active Watchman subscriptions that appear to come
from Nuclide, warn the user if the mission-critical `filewatcher-` subscription
that watches the entire repo is missing.
- If p1 in `SNAPSHOT` and `.hg/dirstate` do not match, warn the user.
The code for `eden doctor` is organized so that it should be relatively
easy to add new conditions to check for going forward. Admittedly, the
UX could be better by formatting the output (color, boldness, etc.) to
make certain information stand out, but we can improve that in
subsequent revisions.
Note that I had to do a bit of cleanup in `eden/cli/TARGETS` as part of
this revision and I created `eden/cli/test/TARGETS` so the tests have
their own build file.
Reviewed By: chadaustin
Differential Revision: D6446057
fbshipit-source-id: ae23c5996dba4f7f70f118179e5556efc29c31c3
Summary: Adds an `eden stats thrift` command that prints call counts for each Thrift call, similar to `eden stats io` printing FUSE counts.
Reviewed By: bolinfest
Differential Revision: D6189543
fbshipit-source-id: 59c29a4036687e1bf75b1c0ca2a7d311b9d1399f
Summary:
Added new tool to report stat information of EdenFs like fuse counters, Memory counters, latencies, Inode status for all the mount points etc.
eden stat : Prints the general information about eden like list of mount points, loaded unloaded and materialized inodes in each mount point. Also this reports how well periodic unload job is doing by reporting the number of unloaded inodes by periodic job.
eden stat io : Prints how many number of calls made to a system call in Edenfs.
eden stat memory : returns the memory stat for edenfs.
eden stat latency : reports the latencies of system calls in Edenfs.
Reviewed By: bolinfest
Differential Revision: D5660345
fbshipit-source-id: 97a1c2b83a6d8df0cd1b82c4d54b52d7ebd126bd
Summary:
Move the code for the command-line tool up one directory, out of eden/fs.
This better separates the code so that eden/fs contains code for the edenfs
daemon, while eden/cli contains code for the command line tool.
Reviewed By: bolinfest
Differential Revision: D4888633
fbshipit-source-id: 5041e292c5353d05122eefe5db3257289e31239a
