Summary:
The test was failing because it was trying to run pgrep which was failing. This happened even before the point where the code to be tested was executed.
It is fixed by mocking that part of code as well where the pgrep command was executed as that code is irrelevant to the subject of the test.
A similar other test is also fixed to avoid repeating this kind of failure there.
Reviewed By: pkaush
Differential Revision: D10356357
fbshipit-source-id: 0336e047329d7bfe55e74fc33a468492c319c7ae
Summary: Doctor now detects if multiple edenfs are running for the same edenDir and recommends to kill the rogue one.
Reviewed By: strager
Differential Revision: D10098157
fbshipit-source-id: 8a785a9fee0c768696f897a4950552d141012b2a
Summary: Update the eden doctor to detect if the commit hash in the SNAPSHOT and dirstate file is valid or not, if invalid, try to fix it
Reviewed By: strager
Differential Revision: D10156841
fbshipit-source-id: 447a508604b87203ee8f1b0cf0b04f9d49752c9d
Summary:
We've been shipping our RPMs with toml config support turned on for a
while now. Remove support for the old config file format. Continue to
ship the old format configurations. We'll remove those in a later
diff.
Reviewed By: strager
Differential Revision: D10020958
fbshipit-source-id: 11c2ca3b5da086b142042496a2814699880c4f81
Summary:
If 'eden stop' tries to tell edenfs to cleanly shut down, and edenfs never responds, 'eden stop' just hangs. Make 'eden stop' time out CLI -> edenfs Thrift calls. After a timeout, 'eden stop' should by stop edenfs via SIGKILL.
Note: This diff does not affect 'eden restart'.
Reviewed By: chadaustin
Differential Revision: D9982372
fbshipit-source-id: 42e498fb93fd97a2ec963e97c25f4b4101c7d524
Summary:
Added checks for kernel version and system type (eg. Linux, MacOS etc).
A check for running on linux has been added to ensure we're not comparing kernel versions from other unixy systems (eg. my mac kernel is currently version 17.7.0). OS tests are not run for non-linux kernels.
The configuration parameters (in doctor.py) MIN_KERNEL_VERSION should be set to the lowest supported kernel version. For bad kernel versions, add them to KNOWN_BAD_KERNEL_VERSIONS as strings or regexes.
Reviewed By: chadaustin
Differential Revision: D10126747
fbshipit-source-id: a0c60351377a9b85e9d76376c7c61fbf63f74a35
Summary: If edenfs is unresponsive, 'eden status' hangs forever. Add a timeout to turn the hang into a user-friendly error.
Reviewed By: chadaustin
Differential Revision: D10156229
fbshipit-source-id: 9186826ae6b131a193b1499c8baac616d131357f
Summary: Reformat all opted-in python code with version `18.9b0` of Black.
Reviewed By: ambv
Differential Revision: D10121017
fbshipit-source-id: 08404dba959bc63bcd7eee7eafe1753c9cfb58ee
Summary:
I noticed that, when Eden hangs, an importer is usually stuck for a
long time too. Add information about edenfs's child processes in
`eden rage`.
Reviewed By: strager
Differential Revision: D10020317
fbshipit-source-id: afe82f559ea0905f10c757fc0b05c3ff64f5ab41
Summary:
'eden stop' has poor test coverage. Write tests for the common case and some edge cases which 'eden stop' already handles.
This diff should not change behavior.
Reviewed By: wez
Differential Revision: D10017819
fbshipit-source-id: 4d9f5db52187c34c62a9379a6b3dd62f62894233
Summary:
wait_for_shutdown waits for the process to no longer be alive using kill. Unfortunately, kill considers a zombie process (i.e. a process whose parent process hasn't reaped its children) to be alive. Teach wait_for_shutdown to consider zombie processes as dead.
Here's how I discovered this bug:
1. Launch edenfs via 'eden start'
2. Send SIGSTOP to edenfs
3. Call wait_for_shutdown on edenfs
edenfs' parent process (sudo) noticed edenfs' SIGSTOP and sent SIGSTOP to itself. sudo became stuck and never reaped its child edenfs process. Thus, edenfs was a zombie process, and wait_for_shutdown complained.
Reviewed By: chadaustin
Differential Revision: D9980225
fbshipit-source-id: c2663a850225775571b02553ccf9e2d460241b6d
Summary: I plan on fixing a bug in wait_for_showdown. wait_for_showdown lacks test coverage, so my change could easily break wait_for_showdown. Improve wait_for_showdown's test coverage to prevent bugs.
Reviewed By: chadaustin
Differential Revision: D9980226
fbshipit-source-id: 3f5019a2c5be32b75ca3daa25e799e956c93dab4
Summary:
wait_for_showdown doesn't use its instance parameter. Delete it.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D9980227
fbshipit-source-id: bfc2dd6d84854b20e8c04fc0391edff50d6d0a49
Summary:
Update `eden doctor` to automatically fix inconsistencies in Eden and
Mercurial's view of the current parent commit by updating Eden to point to
Mercurial's current p1 commit.
The previous instructions were to use `hg reset --keep` to try and reset
Mercurial's data to point at Eden's parent commit. However, the most common
case where Eden and Mercurial can get out of sync is after a Mercurial command
has been interrupted, and Mercurial rolled back a transaction without telling
Eden. In this case the commit that Eden points at does not exist at all, so
trying to use it fails.
Reviewed By: strager
Differential Revision: D9893097
fbshipit-source-id: 3817b83805c7a9959037941204a95ad2bbff2890
Summary:
To diagnose why a process is hitting Eden so hard, show the top pids
in eden top's output too.
Reviewed By: strager
Differential Revision: D9486293
fbshipit-source-id: 0714c354cafef4dae60b2615d20f7e940e24daf0
Summary:
Add a curses-based `eden top` command that displays the top process
names currently accessing an Eden mount over FUSE.
In the future, I'd like to extend this to count and record Thrift
calls too.
Reviewed By: strager
Differential Revision: D9477936
fbshipit-source-id: f878f2e67f4ea24c036880eb4b1162597dc04185
Summary:
Change how fsck reports orphan inodes so that it only creates a single Error
object tracking all orphan inodes.
This will make it easier to write code to fix the orphan inodes as we will
want to process all of the orphan inodes together.
Reviewed By: wez
Differential Revision: D9615337
fbshipit-source-id: 452d0e67f357b8b2ac3b24d89e23fffb5bd816fd
Summary:
Wait to print errors until after we have finished scanning all of the overlay
files. Previously we printed errors as they were detected. However we would
not be able to print the path names correctly if we had not processed all of
the inode's parents yet. This resulted in error messages incorrectly
displaying paths as unlinked in some cases.
Also improve the error strings shown for MissingMaterializedInode and
InvalidMaterializedInode to also include the inode number. Also correct the
inode type display for InvalidMaterializedInode.
Reviewed By: wez
Differential Revision: D9615336
fbshipit-source-id: eb273d51c937e76ffed0e021da848f5fb940145d
Summary:
Add an --overlay flag to the `eden fsck` command to allow running it on an
alternative overlay directory.
This is similar to the `--overlay` flag added to the `eden debug overlay`
command in D9445560.
At the moment a checkout path is still required when running
with `--overlay`. This is used to find the socket to talk to edenfs. This is
not actually used at the moment, but in the future we may need this in order
to help fix some of the filesystem state in some cases. (For instance to get
the current blob or tree ID for a path in the current commit.)
Reviewed By: wez
Differential Revision: D9615334
fbshipit-source-id: 5a0da55f00429f596be5d86e5303e1fcdeff9ea7
Summary:
Refactored the existing fuse latency method for reuse and
added logic to pull out the thrift request latency stats.
Reviewed By: simpkins
Differential Revision: D9502587
fbshipit-source-id: b93d496c33a17efea7539f149c5db749285d75c4
Summary:
D9389865 changed the rel_path variable from a string to a pathlib.Path object.
However it missed a couple places where the path object needed to be converted
to bytes.
Reviewed By: chadaustin
Differential Revision: D9476525
fbshipit-source-id: 2dc69f5a0378ffbc02b2724a097604df288683c9
Summary:
Update the `_get_watch_roots_for_watchman()` check so that it does not fail if
there is no `roots` entry in the output from `watchman watch-list`
Reviewed By: wez
Differential Revision: D9445138
fbshipit-source-id: e066375db3e963d08a91aa2e257d6c4fdd18da74
Summary:
Add an `--extract-to` flag to `eden debug overlay` to ask it to copy inode
data out of the overlay. When used with a file inode this copies the file
contents. When used with a directory inode this copies the directory
structure and file contents for all materialized files under that directory.
This makes it easier to extract user's untracked and modified files out of the
overlay when recovering state after an unclean filesystem shutdown.
Reviewed By: wez
Differential Revision: D9445559
fbshipit-source-id: 9abcf82695014d79f248ba5fdb09727a49f3098a
Summary:
Allow callers to explicitly pass in the location of an overlay storage
directory, rather than requiring that this be found from a running Eden
instance.
This makes it easier to make copies of an overlay for later offline debugging.
Reviewed By: wez
Differential Revision: D9445560
fbshipit-source-id: b1f2ee0dc8bfca80dcc02c57c348f65599ab86e2
Summary:
Update all of the `eden debug` commands to use the newer `find_checkout()`
code rather than older `get_mount_path()` function.
The `find_checkout()` code makes sure that the EdenInstance actually points to
the correct edenfs instance for this checkout, and also works with checkouts
that are not currently mounted. In particular this allows
`eden debug overlay` to examine the overlay state even when the checkout is
not currently mounted.
Reviewed By: wez
Differential Revision: D9389865
fbshipit-source-id: 00578519d4805157a30c9b39abee9838925e8e76
Summary:
This adds a new `find_eden()` method to the Eden CLI code which can look up
information about the correct EdenCheckout and EdenInstance given a path.
In the future most CLI commands should switch to use this function over the
current `get_eden_instance()` and `get_mount_path()` methods. These older
APIs only work to find currently mounted checkouts, which is inadequate for
commands like `fsck` and some other debug commands that want to be able to
operate on unmounted checkouts.
This new logic is able to correctly find checkout information even for
unmounted checkouts. If the checkout is managed by an edenfs instance running
out of a non-default location it also correctly finds the state information
for Eden, rather than potentially using the wrong Eden state directory.
Reviewed By: wez
Differential Revision: D9385821
fbshipit-source-id: a6638f3c3817a595a7b7979c5cd218a2d7400f51
Summary:
Fix the output of `eden debug journal` after changing the Thrift API
to pass paths in bytes instead of strings.
Reviewed By: simpkins
Differential Revision: D9403416
fbshipit-source-id: acd9d0f8b241d11295a48fa11f9cc59bd86381f2
Summary: This commit let `HgImporter` pick up configuration values from `EdenConfig`.
Reviewed By: chadaustin
Differential Revision: D9346293
fbshipit-source-id: cb63f7d13a86058e9bf076eddb52212560a64cb1
Summary:
The Eden CLI allowed setting the `EDEN_CONFIG_DIR` environment variable to
control where to find the `.eden` state directory, as an alternative to the
`--config-dir` command line argument. However, nothing currently appears to
use this variable. Therefore remove this functionality for now to help
simplify the code.
Reviewed By: wez
Differential Revision: D9355514
fbshipit-source-id: c64afb54f599924945573b07bc6d91b346978ea8
Summary: Fix a few minor issues that `mypy --strict` complains about.
Reviewed By: wez
Differential Revision: D9355653
fbshipit-source-id: af63825721fc964b7713df68e8618b595c91561d
Summary:
In the CLI, rename the "Config" class to "EdenInstance". This class
represents all state about a particular edenfs instance. It provides APIs for
making thrift calls to edenfs, for creating and destroying checkouts, and
generally does much more than just managing configuration.
Renaming it to "EdenInstance" also helps clarify that it is distinct from the
configuration tracked in the user's ~/.edenrc and the /etc/eden directory.
This change purely renames internal class and variable names and should not
affect any user-visible functionality.
Reviewed By: wez
Differential Revision: D9355515
fbshipit-source-id: ba5d4c3b753c6eb12a3783306dcd29e85fea3f52
Summary:
Watchman's Eden integration has a bug where the combination of
Watchman querying Eden for overlapping delta ranges ("give me changes
between X and Y, now changes between X+1 and Y+1") and Eden eliding
redundant change events ("add-modify-remove" -> []) results in
Watchman sometimes reporting that a file exists in its final
subscription update when it no longer does.
The fix is to never elide events, even for files that were added and
removed in the same sequence. To continue to support Watchman's `new`
flag, track whether a file existed at the beginning and end of a
journal delta.
Reviewed By: wez
Differential Revision: D9304964
fbshipit-source-id: f34c12b25f2b24e3a0d46fc94aa428528f4c5098
Summary:
Most of the time, the age cutoff for unloads is extraneous, so default
it to 0 in order to unload everything.
Reviewed By: wez
Differential Revision: D9329161
fbshipit-source-id: a241630ea5c069b6f9dafc9f021e39365b96b3bf
Summary:
This prevents `hg status` from blowing up with a UTF-8 decode
error inside the generated thrift code.
Push safety concerns:
* This doesn't change the wire representation of the data
* Existing clients that believe it to be a string will continue to have
the same behavior
* Buck has its own copy of an older version of the thrift spec, so it will
continue to work "OK".
* When buck resyncs with our thrift file, some changes will likely be needed
to convert the byte arrays to strings or paths or whatever is appropriate
for bucks internal API
Work "OK" above means that clients that currently believe that `string` is
utf-8 encoded will have a runtime error if we ever send them a path that
is not utf-8. This is the behavior prior to this diff and will continue
to be the behavior for clients (like buck) that have an older version
of the thrift file.
Reviewed By: simpkins
Differential Revision: D9270843
fbshipit-source-id: b01135aec9152aaf5199e1c654ddd7f61c03717e
Summary:
This is a recurring issue for folks that have unstable systems
that either OOM or are hard rebooted.
Provide advice on how to resolve the issue.
Reviewed By: simpkins
Differential Revision: D9275963
fbshipit-source-id: d746b2cce56716d738304430e484784baf49eb29
Summary:
eden doctor misreports the case that there are duplicate filewatcher
subscriptions for a Nuclide mount. Correctly report those cases as
duplicates instead of missing.
Reviewed By: wez
Differential Revision: D9228450
fbshipit-source-id: cd6fb3c3aa0c3c12f370c084ebc413eeff16ef02
Summary:
Add the beginnings of an eden gc command. Today it's equivalent to
`eden debug clear_local_caches` followed by `eden
debug_compact_local_storage`, except that it compacts each column as
they're cleared to minimize peak disk consumption.
Eventually, it will also unload in-memory inodes, flush data from the
overlay, and clear the kernel's VFS cache too.
Reviewed By: wez
Differential Revision: D9138305
fbshipit-source-id: b303a63f601014cf38ca94c9e6f7c04394159ea8
Summary:
I noticed while making some changes to eden doctor that our
typechecker wasn't working. check_types in python_library became a
no-op, so set typing instead and fix recently-introduced type-checking
failures.
Reviewed By: simpkins
Differential Revision: D9225840
fbshipit-source-id: 32edd60be862b70d97ca48dc9e1929b9add9f166
Summary:
This commit integrates Mononoke API Server with eden:
* Added two new command line options: `--client_certificate` and `--use_mononoke`.
* Added two new config values: `ssl.client-certificate` and `mononoke.use-mononoke`.
* Made `HgImporter` return the repo name along with treemanifest options.
* Make `HgImporter` ask Mononoke API Server for tree data when the desired tree is not present in local Mercurial store.
Reviewed By: chadaustin
Differential Revision: D9183035
fbshipit-source-id: e328fb3237d10c545c8af71f856007ad6c487061
Summary:
Eden should load its configuration from toml files. This change includes:
- ConfigParser is the class that parses and provides access to the configuration. We load the configuration from toml files and pass to ConfigParser in order to preserve existing behavior.
- The toml configuration format differs from the Config format. Thus, we use toml.load() and toml.dump() to load/save toml format configuration.
- Tests
- configuration is properly loaded from a group of config files
- toml configuration (if present) takes precedence
- missing configuration files (/home/user/.edenrc) are ignored
- invalid configuration files are fatal
- update of configuration works (adding repo)
- toml parsing handles over-rides of properties in the same way as cfg files
Deployment Strategy:
- This code determines whether to use toml configuration based on the presence
of atleast 1 '.toml' file in the etc-eden-dir/config.d.
Reviewed By: chadaustin
Differential Revision: D9023232
fbshipit-source-id: 6734a9d91eca92b05872a758c764546451dd2d51
Summary: Update type information so that `mypy --strict` no longer has any warnings.
Reviewed By: strager
Differential Revision: D8940414
fbshipit-source-id: e55533ad5c7c67bd598fae2ae57e6a8d84b47b49
Summary:
Fix the arguments to `doctor.cure_what_ails_you()`, which were updated in
D8870494.
Reviewed By: chadaustin
Differential Revision: D8940413
fbshipit-source-id: 21d45036f668c9bae9a2002c360532918eb157a5
Summary:
If the `nuclide-connections` command is not installed
`subprocess.check_output()` throws a `FileNotFound` exception. Previously the
code only caught `CalledProcessError` and `ValueError` exception. This
updates the code to catch any Exception type, just in case some other
unexpected exception type can also be thrown too.
Reviewed By: chadaustin
Differential Revision: D8945774
fbshipit-source-id: 5e6f63981c3ed6695a687b4bfbaac97ea440b537
Summary:
Add a new `eden debug logging` command to show and update the current edenfs
logging settings.
I have deprecated the `eden debug set_logging_command`, but not removed it
entirely yet. It now longer shows up in the `eden debug --help` output, and
if run it prints the `eden debug logging` command that the user should run
instead.
Reviewed By: chadaustin
Differential Revision: D8907310
fbshipit-source-id: 7ed056637b236db5a50c619023bd8025b787e227
Summary:
This replaces the `StdoutPrinter` class with a different `Output` class
interface that combines both the `StdoutPrinter` functionality and the
underlying I/O object being printed to.
Previously the doctor code had to pass around two separate `out` and
`printer` objects. This simplifies things so that only one object has to be
passed around.
This also adds some additional error checking around the curses
initialization, and falls back to a plain output object if anything goes wrong
during setup. Previously we had seen some reports of `eden doctor` failing
due to errors in the terminal setup code.
Reviewed By: strager
Differential Revision: D8870494
fbshipit-source-id: a6dbafdde9aa5e1a0336fa7d723d83fa7d2426ca
Summary:
When users run into "Transport endpoint not connected" errors they sometimes
attempt to run `eden status` or `eden doctor` to fix the problem. However
previously the CLI did not deal with being run from an invalid working
directory very well.
This updates the CLI to explicitly check if the current working directory is
valid before doing anything else. If it gets an ENOTCONN error it prints a
message telling the user how to update their shell's working directory. If
the `$PWD` environment variable points to a valid directory it attempts to
continue using that as the working directory. It records the fact that the
original working directory was invalid so that `eden doctor` can report this
error if that is the command being run.
Reviewed By: wez, strager
Differential Revision: D8731005
fbshipit-source-id: 4be55ccb43e22ed878114268e4dcc4c90c20e8e2
Summary:
This adds initial code for an `eden debug fsck` command.
At the moment this simply scans the overlay and reports errors, but does not
correct any issues. In subsequent diffs I'll update it to support correcting
some problems.
There are also several other issues that I'll improve in subsequent diffs as
well. At the moment this code uses the same functions as other CLI commands
to find the client directory, and these only work if the checkout is already
mounted. For `fsck` we generally want the checkout to be unmounted before
checking it. I also plan to clean up some of the output handling as well.
Reviewed By: strager
Differential Revision: D8813714
fbshipit-source-id: 90238e94540db1cc06c17eabf8ce3a1bb808d4b3
Summary:
The `eden debug overlay` code has some logic in python for parsing overlay
files. This moves that logic into a new module so I can re-use it for an
`fsck` command in a subsequent diff.
Reviewed By: wez
Differential Revision: D8813715
fbshipit-source-id: 6c32561db95f5da112cfd3e06b9e4653f431b94a
Summary:
This class name looks like it was copy-and-pasted incorrectly, probably from
the `eden debug set_log_level` code. Update the name to specify "Journal"
rather than "Level"
Reviewed By: strager
Differential Revision: D8813713
fbshipit-source-id: 2a5edd7f2de8279ee70dcc72b200c2b86b60f376
