Summary:
Previously the `eden doctor` tests were all in a single `doctor_test.py` file
and it was getting quite large and unwieldy (3000+ lines).
This moves the doctor tests from `eden/cli/test/` to `eden/cli/doctor/test/`
and splits up the various test case classes into their own modules, and the
various helper classes into modules in `eden/cli/doctor/test/lib/`
This doesn't have any functional code changes other than moving things around.
Reviewed By: chadaustin
Differential Revision: D14338814
fbshipit-source-id: 5373c7454567b4417241b4c85ec53d3ded29a8c9
Summary:
Update the "eden doctor" code to directly examine `/proc` when searching for
edenfs processes rather than calling `pgrep`.
This is a bit more efficient, it lets us accurately detect the command line
arguments. In the deadlock scenario in T38417129 this likely would have
prevented `eden doctor` from hanging since it no longer attempts to read
`/proc/<pid>/cmdline` for any process other than `edenfs` itself.
This also fixes a minor bug where the code incorrectly warns about `pgrep`
failing when there are no `edenfs` processes running.
Reviewed By: strager
Differential Revision: D14255535
fbshipit-source-id: f5f40566b46b86201aa59f4554e6acf058337c5b
Summary: Internal Facebook infrastructure is nagging me about some files not having a Facebook copyright notice. Add a notice to these files to make the nagging stop.
Reviewed By: simpkins
Differential Revision: D14173944
fbshipit-source-id: 7234431224fcf4f86ea56ca2f9108f47ef959d87
Summary:
I was thinking about teaching EdenFS' `globFiles` to follow symlinks. To help me think, I documented my understanding of how EdenFS currently handles glob patterns. Publish my notes, even if I decide to not change `globFiles`' behavior.
This document isn't meant to be thorough, so many definitions and nuances are omitted.
Reviewed By: simpkins
Differential Revision: D14288417
fbshipit-source-id: e78749e114224926c2673868854d068672e38cef
Summary:
Add a locale translation file that we can install to modify the error message
that most programs print for `ENOTCONN` errors.
Any I/O operations inside an unmounted FUSE mount will fail with `ENOTCONN`.
Many users don't really know what to do when they encounter this error. This
alternate error message should hopefully help point them in the right
direction.
Reviewed By: wez, pixelb
Differential Revision: D14360858
fbshipit-source-id: 9d538ee675815f017160c3382e2919136367464f
Summary:
This updates `edenfs` to automatically create the mount point directory
if it does not exist.
Previously the `eden mount` CLI command would automatically create the mount
directory in the Python logic. This adds similar logic to the C++ code, which
handles more situations. In particular this makes it so that `eden start`
will now automatically create missing mount point directories.
Note that the C++ code does not create the `README_EDEN.txt` symlink inside
the mount point if it is missing. We could move that functionality into the
C++ code in the future if needed.
Reviewed By: strager
Differential Revision: D14254699
fbshipit-source-id: bad5634f57fba6e7af3b6a3830eb51ac099b435e
Summary:
Update `Overlay::initialization()` to perform the bulk of the initialization
logic in the GC thread. We re-use the GC thread simply for convenience, since
it already exists and won't have any work to do until initialization has
completed anyway.
After an unclean shutdown this allows edenfs to start and perform the overlay
scans for all of its checkouts in parallel rather than serially.
Reviewed By: wez
Differential Revision: D14154868
fbshipit-source-id: c177cb9f950a6317fd7ea06888bd5b326a55ace7
Summary:
Add an Overlay::initialize() function, and consolidate all Overlay
initializtion logic in this function. Previously some initialization was done
by the Overlay constructor, and some was done in `EdenMount::initialize()`
Overlay::initialize() returns a folly::SemiFuture as it may perform a
non-trivial amount of disk I/O. However, at the moment it currently performs
all I/O in the current thread before it returns. I plan to move this work to
a separate thread in a subsequent diff.
Reviewed By: strager
Differential Revision: D13981140
fbshipit-source-id: b59eaef88012a8e74fcb770a9c93ca3f9bde32a0
Summary:
This updates the `EdenServer` class so that the existing `getMount()` and
`getMountPoints()` APIs only return mounts that have finished initializing.
These APIs are primarily used by the thrift interfaces. In most cases the
callers did not intend to operate on mounts that were still initializing, and
doing so was unsafe. The code could potentially dereference a null pointer if
it tried to access the mount's root inode before the root inode object had
been created.
New `getMountUnsafe()` and `getAllMountPoints()` APIs have been added for call
sites that explicitly want to be able to access mounts that may still be
initializing. Currently the `listMounts()` thrift API is the only location
that needs this.
Reviewed By: strager
Differential Revision: D13981139
fbshipit-source-id: e6168d7a15694c79ca2bcc129dda46f82382e8e9
Summary:
Add a flag to tell edenfs to report successful start-up as soon as the thrift
server is running, without waiting for all mount points to finish being
remounted.
In the future I plan to have edenfs automatically perform an fsck scan of the
overlay for checkouts that were not shut down cleanly. This may cause the
remount to take a significant amount of extra start-up time in some cases.
(This is already true today in some cases even with the simpler scan we do to
re-compute the max inode number.)
I think we will probably want to have systemd invoke edenfs with this option,
so that we do not time out during system start up if some mount points need to
be rescanned.
Reviewed By: strager
Differential Revision: D13522040
fbshipit-source-id: 6f183770c25efee34c4805c9bad42a9cce51039e
Summary:
In the future, I want some tests to mock PrivHelper::fuseUnmount. The existing classes for mocking PrivHelper::fuseMount, MountPromiseDelegate and FailingMountDelegate, work poorly for fuseUnmount.
Combine MountPromiseDelegate and FailingMountDelegate to create MockMountDelegate, which is more flexible and allows a mocking fuseUnmount to be implemented easily.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14319712
fbshipit-source-id: 6eecf121e3b44f39cd5dffbeafaf0f3aab75cb76
Summary: Switch to using cpptoml from third-party on all platforms.
Reviewed By: igorsugak
Differential Revision: D14351179
fbshipit-source-id: 183bedc7d27e9c0f9216f3d0cca58ada75ac74e7
Summary: Ensure that an EdenMount is RUNNING after it takes over an EdenMount from another process.
Reviewed By: simpkins
Differential Revision: D14178500
fbshipit-source-id: 2813b024b6815dc5d31f3e3bf89cffa98ad0f1c3
Summary:
I want manipulate a FakeFuseMountDelegate in a test, but FakeFuseMountDelegate is private to FakePrivHelper.cpp. Make FakeFuseMountDelegate usable outside FakePrivHelper.cpp by putting its declaration in FakePrivHelper.h.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14319711
fbshipit-source-id: d7af18c316f63febe1d60f6e5aadcd4f4827cca5
Summary:
Pyre complains about this cast being redundant. It was needed at one point
(in D9144139) to make either pyre or mypy happy, but now neither seem to need
it and pyre actually seems to complain about it.
Reviewed By: chadaustin
Differential Revision: D14319772
fbshipit-source-id: ed252763acfe9c3c6326c6712e30d2840080454c
Summary:
Update `edenfs` to automatically create the bind mount source directories if
they are missing. Previously Eden would report an error and would not be able
to mount the checkout if some of the bind mount source directories were
missing.
Reviewed By: strager
Differential Revision: D14253771
fbshipit-source-id: 87ad091ccf2c0f0f72aebb50437fd7680ddbfd1c
Summary:
Update the `eden rage` code to catch any unexpected exceptions from the
`eden doctor` logic so that it can still generate the final rage report.
Having a report that is missing the doctor information is better than failing
to generate a rage report at all. The traceback from the `eden doctor`
exception may also help diagnose whatever issue is causing problems.
Reviewed By: chadaustin
Differential Revision: D14255648
fbshipit-source-id: 365826f22ed933f80a3cd43ddd1d66009d7032fc
Summary:
Make "eden doctor" check if the process's current working directory matches
the "$PWD" environment variable, and suggest that the user run "cd / && cd -"
if they do not match.
The most common cause of this is users that cd'ed into an Eden checkout
directory before Eden was running, and then started Eden but haven't updated
their working directory since. In this shell they will still see the contents
of the underlying mount point directory itself and not the Eden checkout
contents that have been mounted on top of it.
Reviewed By: chadaustin
Differential Revision: D14249909
fbshipit-source-id: ecfc2a56180e3a6161940e9e1ec6d94663bddbc0
Summary:
[Folly] Stop checking `EventBase::runInEventBaseThread` result, as the function will soon be changed not to return any result.
It returned `false` when failing to enqueue a task. But it cannot really fail anyway besides allocation failure, unless in the `EventBase` destructor and while draining and the `AlwaysEnqueue` variant is called.
Reviewed By: andriigrynenko
Differential Revision: D14254969
fbshipit-source-id: a6a9199cbafa18b61488a240e4318ce946953f51
Summary:
We're not that far from building privhelper on mode/mac but it does
require figuring out how to depend on osxfuse from the Buck build, so
bypass that by breaking the inodes target's dependency on privhelper's
<fuse_ioctl.h> include.
Reviewed By: strager
Differential Revision: D14218709
fbshipit-source-id: edbb2a21df06d6f2a4f860ef13718ad05d445e98
Summary: Fix some missing includes that showed up in my mode/mac builds of parts of Eden.
Reviewed By: simpkins, strager
Differential Revision: D14213572
fbshipit-source-id: 54e070e89afdfb8479abdaa122ba76ca1d2d9ba9
Summary:
Throw a slightly nicer error message if the hg import helper exits before
returning a `CMD_STARTED` response or an error response.
When switching from the `hg_import_helper.py` script to the
`hg debugedenimporthelper` subcommand we are slightly more likely to encounter
errors during startup. For instance, if the repository path was not a valid
repository the `hg_import_helper.py` code would send an error message back
that the `HgImporter.cpp` code could parse. However, we unfortunately can't
catch errors loading the repository the same way when using an `hg`
subcommand. If it fails to load the repository it will simply print an error
message to stderr and then bail out before it invokes the
`debugedenimporthelper` code.
Reviewed By: chadaustin
Differential Revision: D14222266
fbshipit-source-id: cd60e5a61725d45a816b74f63b9969b29ade2160
Summary:
EdenMount calls fuseMount, but EdenServer calls fuseUnmount. I think only one of EdenMount or EdenServer should call fuseMount/fuseUnmount. Splitting the responsibility is confusing.
Move the call to fuseUnmount into EdenMount by creating a member function called unmount.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14115385
fbshipit-source-id: 845c4a87cdba9f270c0eacaf01916ad91c3b7c0e
Summary:
Update `EdenMount::initialize()` to perform a fault injection check. This
allows test code to inject delays and errors into the mount initialization
flow.
Reviewed By: strager
Differential Revision: D14079491
fbshipit-source-id: be80135b0833c8f0300104524473cc3e949fec34
Summary:
This reverts the change in D14188312 that moved `hg_import_helper.py` from
`bin/` to `libexec/eden/`, and instead updates edenfs to look for
`hg_import_helper.py` at `../../bin/hg_import_helper.py`
We cannot remove `hg_import_helper.py` from `/usr/local/bin` without breaking
existing running `edenfs` daemons as they will continue to look for
`hg_import_helper.py` in this location.
Rather than keeping a copy in both `bin/` and `libexec/eden` it seems better
to simply update edenfs so that it can correctly find `hg_import_helper.py`
even when edenfs itself has been installed in `libexec/eden/`
Reviewed By: strager
Differential Revision: D14197390
fbshipit-source-id: 2845bd79343bc29af9d3acbacecac4501cdb9032
Summary: We should only attempt to include selinux on Linux, not macOS.
Reviewed By: strager
Differential Revision: D14181109
fbshipit-source-id: be47b7bdadc3409577fa114559e905214848ebd8
Summary: This diff fixes Folly's xlog when building with CMake.
Reviewed By: strager
Differential Revision: D14155982
fbshipit-source-id: cb88500ca1fa9f5f748129e1d11471cecd71ce22
Summary:
The Eden CLI tool is really a control program for `edenfs`. Rename it to
`edenfsctl` to free up the `eden` name for future use.
The Eden daemon shouldn't really be on the user's path, and instead belongs in
`libexec`.
For transition compatibility, `eden` is symlinked to `edenfsctl`.
Reviewed By: simpkins
Differential Revision: D13888875
fbshipit-source-id: 435cc63e92b85b1f28b8691e4846fbcb05bc450e
.pyre_configuration.local is a file specific to Facebook's
infrastructure. Committing this file to the public Eden repo was
unintentional. Delete this file to reduce confusion.
Summary:
futures::sleep returning a Future leads to continuations easily being run on
the Timekeeper's callback. The goal is to change the return type so that
futures::sleep returns a folly::SemiFuture.
This codemod is part of the first stage:
* Migrate all call sites off of futures::sleep and onto futures::sleepUnsafe.
This will be followed by:
* Changing the return type of futures::sleep to folly::SemiFuture.
* Migrating callers, where clearly safe such as when they follow with .via or
.semi() from sleepUnsafe to sleep.
* Migrating remaining callers.
Reviewed By: yfeldblum
Differential Revision: D14152623
fbshipit-source-id: bc6874e4320e4a7352ac61b20d9750458e2cbbff
Summary:
If EdenMount::shutdown is in progress and EdenMount::destroy is called [1], the EdenMount object won't ever be deleted. A comment in EdenMount::destroy claims that EdenMount::shutdown will delete the EdenMount object, but the comment is wrong:
case State::SHUTTING_DOWN:
// Nothing else to do. shutdown() will destroy us when it completes.
return;
This leak is probably benign, but the leak checker can cause tests to fail spuriously. I think the leak currently can't happen for EdenServer's mounts, but in the future the leak might occur and cause problems.
Make EdenMount::shutdown delete the EdenMount if EdenMount::destroy was called. This fixes the leak.
[1] EdenMount::destroy is called by EdenMountDeleter when the last std::shared_ptr<EdenMount> is destroyed.
Reviewed By: simpkins
Differential Revision: D14015024
fbshipit-source-id: 39a0ba78bb00afb5f0363fe5c74dbec00f9d68e3
Summary:
If EdenMount::shutdownImpl and EdenMount::startFuse execute concurrently, the mount's `state_` might end up inconsistent. For example, for the following sequence of events, EdenMount::shutdownImpl is erroneously called twice:
1. EdenMount::initialize transitions the mount's state to INITIALIZED.
2. EdenMount::startFuse begins and transitions the mount's state to STARTING.
3. EdenMount::shutdown begins and transitions the mount's state to SHUTTING_DOWN.
4. EdenMount::shutdown finishes and transitions the mount's state to SHUT_DOWN.
5. EdenMount::startFuse finishes due to FuseChannel::initialize failing with an error and transitions the mount's state to FUSE_ERROR.
6. EdenMountDeleter calls EdenMount::destroy, which observes that the mount's state is FUSE_ERROR. EdenMount::destroy proceeds by calling EdenMount::shutdownImpl.
This specific scenario can't happen in production right now because step 3 requires allowFuseNotStarted to be true (and it's never true in production).
A similar scenario can happen with EdenMount::startFuse+EdenMount::destroy running currently, but the effects are benign. However, in a future diff (D14015024), I am changing how EdenMount::destroy works, and the SHUT_DOWN -> FUSE_ERROR transition causes problems.
When FuseChannel::initialize returns an error, transition the mount to the FUSE_ERROR state only if we are not already shutting down (instead of unconditionally). This fixes the [impossible] scenario above, and also fixes EdenMount::destroy in a future diff.
Reviewed By: simpkins
Differential Revision: D14030898
fbshipit-source-id: fe5a749b8798749734e0de91eb3b71b601ebb64b
Summary:
as above; this makes the build faster and makes us
slightly less sensitive to problems with building its tests.
Reviewed By: pkaush
Differential Revision: D14137089
fbshipit-source-id: 7edf4ac4a24bd90db55410363821f30635010627
Summary:
This diff adds the dtype field to the glob results;
this will help to reduce the cost of some watchman queries by avoiding a
getFileInformation call that instantiates inodes.
As part of this, I added a bunch of unit test coverage.
Reviewed By: strager
Differential Revision: D8779149
fbshipit-source-id: 3064a3e42be55ec576fed9e0f7112edef426f32d
Summary:
This diff updates our internal eden packaging to correctly declare
a dep on python3, and to specify that python3 interpreter when building
the CLI.
It also allows importing of `readline` to fail, which is important because
the python3 on the system may not have a functioning readline extension.
Reviewed By: strager
Differential Revision: D14098276
fbshipit-source-id: ad1174e46b9b1c0ec1e602ecaeb59c1f0835472a
Summary: SDL checks generate errors for using negatives with unsigned ints (for ex x = -uint_var), which we have at few places in Folly and Thrift.
Reviewed By: wez
Differential Revision: D14125841
fbshipit-source-id: 3a9dd0fc36ecde4cdd75473c893c5f7adb450740
Summary: This would use all the available CPUs and speed up the build process.
Reviewed By: strager
Differential Revision: D13563433
fbshipit-source-id: 18b3862ae0c56ae3865c56864b58cf749c844bd4
Summary:
Add a new class for injecting faults into Eden. This will make it easier to
build tests that exercise specific error conditions or specifically control
the ordering of operations that might otherwise race.
To minimize the performance impact, the fault injection framework is disabled
by default and must be explicitly enabled at process startup using a command
line flag. If this command line flag was not specified all fault injection
checks short-circuit into a no-op. The `checkAsync()` version does still add
some extra overhead due to the addition of a new `folly::SemiFuture` into the
code path.
Reviewed By: wez
Differential Revision: D14079489
fbshipit-source-id: 3e8725d2e51e5c829199a602e90574424ea52636