Summary:
This updates the privhelper code to use the UnixSocket class for performing
I/O. This reduces the number of separate implementations of code we have for
sending file descriptors across Unix domain sockets, and also makes the
privhelper APIs non-blocking.
This will make it easier to clean up some of the initialization ordering in
the future. It will also make it easier to send file descriptors to the
privhelper server, instead of just receiving them. This may be helpful for
passing a file descriptor to use for logging to the privhelper process, which
will make it easier to fork the privhelper before logging redirection has
occurred.
Reviewed By: bolinfest
Differential Revision: D8053422
fbshipit-source-id: 1f8fdf22afc797eead0213be1352ea530762140d
Summary:
Up until now all of the privhelper APIs have been blocking calls. This
changes the privhelper functions to return Futures, and updates all users of
these APIs to be able to handle the results using Futures.
One benefit of this change is that all existing mount points are remounted in
parallel now during startup, rather than being mounted serially. The old code
performed a blocking `get()` call on the future returned by
`EdenServer::mount()`.
The privhelper calls themselves are still blocking for now--they block until
complete and always return completed Future objects. I will update the
privhelper code in a subsequent diff to actually make it asynchronous.
Reviewed By: bolinfest
Differential Revision: D8053421
fbshipit-source-id: 342d38697f67518f6ca96a37c12dd9812ddb151d
Summary:
1. Enabled a number of additional C++ compiler warnings in Eden.
2. Fixed warnings-turned-errors that resulted from this change.
Reviewed By: simpkins
Differential Revision: D8132543
fbshipit-source-id: 2290ffaaab55024d582e29201a1bcaa1152e6b3e
Summary:
Fix the code in FuseChannel::processSession() to look at the errno value
instead of the return value from `read()` when checking for ENODEV.
I accidentally broke this in D7436867. This wasn't causing any major issues,
since we would still break out of the loop, but it caused us to incorrectly
print warning messages about receiving an unexpected error. We would also
print this error message once per FUSE thread rather than just once for the
mount point.
Reviewed By: bolinfest
Differential Revision: D8109889
fbshipit-source-id: 9a53ca47b436ccf6731144ee2b829131339b6445
Summary: Now that permissions on directories work, verify umask works as intended.
Reviewed By: simpkins
Differential Revision: D7783743
fbshipit-source-id: 635221cd3255cc20e9ffa26b6838922c4a4110f3
Summary:
This is not at all clear from cppreference.com, but per
https://www.youtube.com/watch?v=dTeKf5Oek2c, it sounds to me like
recommended practice is to either:
`using namespace std::chrono_literals` (or string_literals or
whatever) to pull in a focused set of literals.
or
`using namespace std::literals` to pull in all standard literals.
or
`using namespace std` to pull in everything.
`using namespace std::literals::chrono_literals` is unnecessarily
verbose.
Adopt those standards in Eden.
Reviewed By: simpkins
Differential Revision: D8060944
fbshipit-source-id: 4d9dd4329698b7ff5e5c81b5b28780ca4d81a2a1
Summary:
I'd misunderstood the point of SharedMutex's upgrade locks -
unless they're used in rare paths, they don't allow for increased
concurrency. This diff and D7885245 remove all of Eden's ulocks,
replacing them with a helper which checks once with an rlock held, and
if the check fails, switches to a wlock (and checks again).
Reviewed By: yfeldblum
Differential Revision: D7886046
fbshipit-source-id: 545bb0dbb4898cbb71412efc6222ef12e4ee374e
Summary:
Update the folly::Init code to define a `--logging` command line flag, and call
`folly::initLoggingOrDie()` with the value of this command line during
initialization.
This is similar to the existing code that initializes the glog library.
(Programs can use both glog and folly logging together in the same program, and
I expect that many programs will do so as parts get converted to folly::logging
and parts remain using glog.)
Reviewed By: yfeldblum
Differential Revision: D7827344
fbshipit-source-id: 8aa239fbad43bc0b551cbe40cad7b92fa97fcdde
Summary:
Per yfeldblum's comment in D7886046, we can use folly::unit instead
of folly::Unit{}. We weren't using folly::unit anywhere, so this diff
replaces folly::Unit{} elsewhere in the Eden code.
Reviewed By: yfeldblum
Differential Revision: D7913462
fbshipit-source-id: fa6ab44ceb406d38713e0f4649224a74e6e51abd
Summary:
Promote the folly logging code out of the experimental subdirectory.
We have been using this for several months in a few projects and are pretty
happy with it so far.
After moving it out of the experimental/ subdirectory I plan to update
folly::Init() to automatically support configuring it via a `--logging` command
line flag (similar to the initialization it already does today for glog).
Reviewed By: yfeldblum, chadaustin
Differential Revision: D7755455
fbshipit-source-id: 052db34c97f7516728f7cbb1a5ad959def2f6efb
Summary:
To avoid bugs similar to the ones fixed in D7781691, mark a bunch of
folly::Future<folly::Unit> functions with FOLLY_NODISCARD.
Reviewed By: simpkins
Differential Revision: D7782224
fbshipit-source-id: 23ba42aa63011cc33e5a6e18d5bc6d00403a78d3
Summary:
Remove the EDEN_HAS_COMMON_STATS checks now that the common/stats stubs have
the required APIs needed by Eden.
Reviewed By: wez
Differential Revision: D7479593
fbshipit-source-id: cc3db50288bfea7aefd6c91391ab800628b7978f
Summary:
Add a macro to help users define the `getBaseLoggingConfig()` function.
While I would prefer to avoid macros if possible, this seems worthwhile. This
saves 4 or 5 lines of boilerplate code in each program that sets a custom base
logger setting. It also reduces the likelihood of a developer accidentally
having a typo in the function name, which would still build successfully but
not have the desired results.
Reviewed By: chadaustin
Differential Revision: D7457652
fbshipit-source-id: 1c316c7ea6949c16bd7b61c0440cc1ee69ecb83e
Summary:
Fix FuseChannel::processSession() to always process all FUSE requests that it
reads. Previously it checked to see if it should stop immediately after
reading FUSE request. It was possible for the old process to successfully read
a FUSE request, see that it was supposed to stop, and then exit this worker
thread without ever processing this FUSE request. This would cause the client
that sent the request to hang indefinitely, since no response would ever be
sent.
Reviewed By: wez
Differential Revision: D7436867
fbshipit-source-id: c58c2f6c49102fa6b66ac83fc1639595a5277ce0
Summary:
Update FuseChannel to send all invalidation requests in a separate thread.
This eliminates a deadlock that could previously occur during checkout
operations. The invalidation requests would block until they could acquire the
kernel's inode lock on the inode in question. However, the inode lock may
already be held by another thread attempting to perform an unlink() or rename()
call. These FUSE unlink or rename operations would be blocked waiting on
Eden's mount point rename lock, which was acquired by the checkout operation.
Checkout operations now let the invalidations complete asynchronously, but we
wait for all invalidation operations to complete before indicating to our
caller that the checkout has succeeded.
Reviewed By: chadaustin, wez
Differential Revision: D7404971
fbshipit-source-id: 6fa20c00d054e210eb0258d247d083010557f210
Summary:
Request data was coming back as null because of a race between ctx->getContextData() in FuseChannel::getOutstandingRequests() and the it initialization.
Also, added a check to verify rdata is valid before dereferencing it.
Plus, fixed debug.py where output gave an error.
Reviewed By: chadaustin
Differential Revision: D7377993
fbshipit-source-id: 8343119983c74185fd5d8cc05c2f5af63dcff99e
Summary:
The changes in this diff comments out unused parameters. All changes are automated using clang-tidy.
This will allow us to enable `-Wunused-parameter` as error.
Reviewed By: simpkins
Differential Revision: D7371610
fbshipit-source-id: 0134e2f0b916313d690c073a46d747c52399a226
Summary:
Set the default logging settings by overriding folly::getBaseLoggingConfig()
rather than by setting a default value for the `--logging` command line
argument.
This has two advantages:
- Custom logging settings supplied in the `--logging` argument will now
combined with the base logging settings rather than completely replacing
them. Previously users had to always add "eden=DBG2" to this argument if
they did not want to drop this setting.
- This will make it possible to move the definion of the `--logging` flag to
folly::Init and not lose our custom settings for Eden.
Reviewed By: yfeldblum
Differential Revision: D7348519
fbshipit-source-id: bd9012f387b0460da8bcc9c7c31aef46b1abec5c
Summary:
Fix the code to generate exceptions based on an errno error using
std::generic_category rather than std::system_category.
Reviewed By: yfeldblum
Differential Revision: D7329997
fbshipit-source-id: 3fe257bbbc7a631c801f31120592c8bdbc25c8bf
Summary:
Add a new utils/SystemError.h header with helper functions to check if a
`std::system_error` contains an errno value.
Most of the code in Eden previously only checked for `std::system_category`
when looking for errno values. `std::generic_category` is the correct category
to use for errno exceptions, but folly/Exception.h incorrectly throws them as
`std::system_category` today. This change makes Eden treat either error type
as errno values for now.
Reviewed By: yfeldblum
Differential Revision: D7329999
fbshipit-source-id: 67a3c3ea10371c53a2e34236b7575deac4cbd53a
Summary:
Added a thrift call to return the outstanding FUSE requests.
Cli will call the thrift and print the output.
Added a unit test to test getOutstandingRequests().
Reviewed By: simpkins
Differential Revision: D7314584
fbshipit-source-id: 420790405babdb734f598e19719b487096ec53ca
Summary:
This test exercises code under eden/fs/fuse, so put it with that directory's
tests.
Reviewed By: chadaustin
Differential Revision: D7314457
fbshipit-source-id: b664aaa3086b2e65f7ae8b76ae71878f39fd9d17
Summary:
Previously the FuseChannel code unconditionally set `connInfo_.major` and
`connInfo_.minor` to the FUSE version that it supports, rather than the FUSE
version supported by the kernel it is talking to.
This does not appear to be the intended behavior, given that in several other
location it checks the value of `connInfo_->minor` to see if particular
features are supported.
Reviewed By: wez
Differential Revision: D7300845
fbshipit-source-id: 1f5946134037b2cbe03a8b86241acaa401c21ffa
Summary:
Remove the `FuseChannel::stealFuseDevice()` method, and instead change the
session completion future to return the FUSE device FD if it is still valid.
This ensures we only extract the FUSE device when it is still valid: if an
unmount event is being processed by a FUSE worker thread simultaneously with a
call to `FuseChannel::takeoverStop()` the session completion future will return
an empty `folly::File` since the FUSE device has been shut down.
This also helps clarify the synchronization semantics around modification to
the `fuseDevice_` member variable.
Reviewed By: wez
Differential Revision: D7300847
fbshipit-source-id: 02ced8fc9d24e7cd526d911782949d0bfbf0e5a7
Summary:
Eliminate a few unnecessary includes from Dispatcher.h and one from
RequestData.h
Reviewed By: wez
Differential Revision: D7300846
fbshipit-source-id: 0cde1f70f605fcb5c95ed13950f4fa8d353fd72f
Summary:
FuseChannel::processSession() was missing a return statement after we received
EOF from the FUSE device. This would cause us to fall through and try
processing the empty buffer as a request. This would fail, and we would try to
send a reply, which would in turn fail since the FUSE device was closed.
This was only an issue with the unit tests: on real FUSE devices we get an
ENODEV error when reading from the device. In unit tests we get an EOF
instead.
Reviewed By: wez
Differential Revision: D7299336
fbshipit-source-id: c275e03edbf594c0d120257b1cfc89d7c75f60f6
Summary:
The FuseChannel destructor previously could be invoked while there are still
outstanding requests, which would cause problems when responses were generated
for these requests.
This makes the FuseChannel destructor private, requiring users to always
allocate FuseChannel objects on the heap and destroy them using a destroy()
method. The destroy() method still blocks waiting for the FUSE worker threads
to exit, but if there are still outstanding FUSE requests it defers the
deletion of the FuseChannel until they are complete.
The EdenMount class currently couldn't ever destroy FuseChannel objects with
outstanding requests, since it waits for the completion future to fire before
invoking the destructor. Fixing this behavior is mostly for correctness of the
FuseChannel API and for the benefit of our unit tests.
Reviewed By: chadaustin
Differential Revision: D7297829
fbshipit-source-id: 643d1332be84a1f25ee30ba2a2ea3515806f00ef
Summary:
Add a simple test program that starts a FUSE mount. At the moment the only
call it actually responds to is getattr() for the root inode. However this is
sufficient for exercising the FuseChannel startup and shutdown code.
Reviewed By: chadaustin
Differential Revision: D7297827
fbshipit-source-id: 1bae6015def37913a5bf57541a14c3c8c050eff5
Summary:
Remove the `FuseChannel::getSessionCompleteFuture()` method and instead have
`initialize()` and `initializeFromTakeover()` return the `Future` that can be
used to wait on session completion.
This makes it explicit from an API perspective that this session completion
future will only be invoked if initialization is successful. It also
eliminates the possibility of anyone calling getSessionCompleteFuture() more
than once.
I also changed the session completion future to use a folly::SemiFuture rather
than a folly::Future, to make it explicit that callers generally should not
execute their callback inline in the same thread where the SemiFuture is
fulfilled.
Reviewed By: chadaustin
Differential Revision: D7297835
fbshipit-source-id: 3a1157951f0738f1692833ed5e875c3e9c6d6d69
Summary: Testing convenience. The NO_CHECK constructor is no longer necessary now that simpkins made the original constructor constexpr.
Reviewed By: simpkins
Differential Revision: D7295539
fbshipit-source-id: 717f5a7a8d3d9790966fad441da7b3308495e2c3
Summary:
The FuseChannel unit tests would sometimes hang because a worker thread would
still be stuck in a blocking read() call.
The issue appears to be that the kernel would often automatically restart the
interrupted read(), so the worker would never wake up. This switches the code
from using SIGPIPE to wake up, and instead uses SIGUSR2, and installs a handler
for SIGUSR2 with the SA_RESTART flag unset.
It's possible that this was an issue only for the unit tests, where the FUSE
device is a socket. I'm not sure if the kernel would automatically restart
read operations on a real FUSE device.
There are still theoretically some race conditions here, since
`requestSessionExit()` could be called right after a worker thread checks
`(runState_` and before it enters the blocking `read()` call. Fixing that
would likely require switching to something like `folly::EventBase` or manually
using epoll. Fortunately I haven't seen that be an issue in practice so far.
Reviewed By: chadaustin, wez
Differential Revision: D7282002
fbshipit-source-id: 99d29de13a7858dd25f258fdc94d8f8c71ee84d9
Summary:
This changes FuseChannel to fulfill the session complete future immediately in
the thread where it finishes. This may occur inside a FUSE worker thread.
It is now up to the caller to use `via()` if they want their callback to run in
a separate thread.
This simplifies the FuseChannel code and also addresses a crash that occurs in
the unit tests sometimes: If the FuseChannel destructor is invoked without
explicitly invoking the FuseChannel, it would schedule the session complete
promise to be fulfilled in a separate EventBase thread. However, the promise
may then have already been destroyed by the time it can run in the EventBase
thread.
There are still additional synchronization issues to fix in the FuseChannel
tests, but this resolves one problem.
Reviewed By: chadaustin
Differential Revision: D7282001
fbshipit-source-id: be64d3ef6a0e664ed7a2cf93a549acc140184095
Summary:
If you run `eden daemon` on a machine where sudo needs input, sudo's
output would get redirected to edenfs.log and eden daemon would appear
to hang. Worse, if you pressed ctrl-C, sudo would remain in the
background and continue to swallow keypresses from the tty, appearing
to somewhat break your shell until it gave up. The fix is to stop
redirecting stdout and stderr from Python and instead have edenfs
redirect itself into the log if given.
Reviewed By: simpkins
Differential Revision: D7223399
fbshipit-source-id: bae17b150b6594a3dd87bb88741c8dcefd1c5213
Summary:
Update FuseChannel::startWorkerThreads() to return without doing anything if
runState_ indicates that the FuseChannel is currently stopping.
This addresses a race betwen startWorkerThreads() and the destructor. If the
destructor is invoked in parallel with startWorkerThreads() the destructor can
grab the state_ lock and clear out the workerThreads list first. This then
causes problems when startWorkerThreads() runs and adds new threads to the
workerThreads list, which then never get joined.
Reviewed By: wez
Differential Revision: D7253574
fbshipit-source-id: f2cac11f1e71e1a14e3f020368152e0f2948c625
Summary:
Update FuseChannel::readInitPacket() to honor `runState_` and stop if it is
changed to indicate that the channel is stopping. Previously the init worker
thread would always wait until it received an INIT packet or an error on the
FUSE channel.
This also refactors the `readInitPacket()` code slightly mainly to to reduce
the level of indentation.
Reviewed By: wez
Differential Revision: D7253575
fbshipit-source-id: 7f27698947b629daecd9cd4ddffb6e6a921ce41e
Summary:
Update the FuseChannel code to keep track of why it is stopping, and to return
this data in the session complete future.
This is mainly just available for diagnostic reasons, and at the moment nothing
is using this outside of the unit tests.
Reviewed By: wez
Differential Revision: D7253573
fbshipit-source-id: 0c7e5b8bd9c3c1de3788918c2257c06d4fe0a90c
Summary:
Catch exceptions thrown by FuseChannel::processSession() and terminate the
FuseChannel if any errors occur.
Also explicitly mark `fuseWorkerThread()` and `initWorkerThread()` as
`noexcept` so we will crash with useful exception backtraces if an unhandled
exception does ever attempt to propagate out of these functions. (Prior to
gcc 8.x, `std::thread()` ends up swallowing the exception backtrace information
if a thread function throws an exception.)
Reviewed By: chadaustin
Differential Revision: D7243909
fbshipit-source-id: fbe7173a2167532d7e42901f810c17bb173d8d63
Summary:
- Update FuseChannel::initializate() to not require an Executor. Rather than
performing initialization using the supplied Executor, it simply starts one
worker thread first and performs initialization there.
- Change the API semantics slightly so that the session complete future is
invoked only if initialization succeeds. Previously the session complete
future could be invoked if initialization failed as well.
- Replace the activeThreads counter with a stoppedThreads counter to determine
when the session complete future should be invoked. The previous
activeThreads behavior was somewhat racy, since the activeThreads counter was
incremented inside each worker thread, but most places that checked it did
not wait to ensure that all worker threads had successfully incremented it
yet.
Reviewed By: chadaustin
Differential Revision: D7243910
fbshipit-source-id: 93b3465509bd9bf6fa90ea097e70dac3193172f9
Summary:
Split up the FuseChannel and EdenMount APIs for starting a brand new FUSE
channel vs taking over an existing channel.
These operations are somewhat different as initializing a new FUSE channel
requires performing a negotiation with the kernel that will not complete
immediately. Therefore these APIs return a Future object. Taking over an
existing FUSE channel is always an immediate operation, and therefor does not
need to return a Future object.
Reviewed By: chadaustin
Differential Revision: D7241997
fbshipit-source-id: 22780f2df76b2d554ab2a4043b6425fa7a4e9c94
Summary:
Update FuseChannel to only join its worker threads in its destructor, rather
scheduling a function in the main EventBase thread to join each thread as soon
as it completes.
This makes FuseChannel a bit easier to use: the destructor can now be called at
any point in time (as long as it is not called inside one of the worker threads
themselves). If it is called while the FuseChannel is still running this will
automatically stop the worker threads.
Reviewed By: wez
Differential Revision: D7224369
fbshipit-source-id: 4de0e7c0c677e870fd629eaaaa5ab40d134ee870
Summary:
Ensure that we ignore SIGPIPE in unit tests. The
`FuseChannel::requestSessionExit()` function sends SIGPIPE to its worker
threads, and expects this to wake up the worker threads if they are stuck
inside a read() call.
In normal program builds folly/Subprocess.cpp causes SIGPIPE to be ignored on
startup. However, Subprocess.cpp is not necessarily linked into all of our
unit tests. I haven't tracked down exactly why, but SIGPIPE in the unit tests
seems to kill the entire process only in opt-ubsan builds.
In the future we probably should clean up how requestSessionExit() triggers its
worker threads to exit: the current code is still potentially racy, since the
worker threads could receive the SIGPIPE just after checking `sessionFinished_`
and before calling `read()`, in which case they may go back to sleep rather
than exiting. However for now this should be a simple fix to get the tests
passing in all build modes.
Reviewed By: wez
Differential Revision: D7224370
fbshipit-source-id: a56fe3331fc5aa6a49ccbe6b0678b479a44ffc07
Summary:
Only the FuseChannel code should invoke the handler methods. This makes them
all private.
This did require making handlerMap be a private static member variable of
FuseChannel so that it can access these methods during its initialization.
Reviewed By: wez
Differential Revision: D7126002
fbshipit-source-id: 28c421173cb6cf8b7a8e1ca085ad78ec9ea76b8f
Summary: This method is never removed anywhere, so delete it for now.
Reviewed By: wez
Differential Revision: D7126003
fbshipit-source-id: 974645c8063fc17c4e69e5e46c870e52fecfcd65
Summary:
We were curious why we needed `sudo gdb` to debug Eden even
though it drops privileges. The process dumpable bit is the trick.
Reviewed By: simpkins
Differential Revision: D7182742
fbshipit-source-id: 48e3dd778bf07ec1bfadace4edeb64668b936b9b
Summary:
Today, attaching gdb to edenfs requires sudo. Our hypothesis is that,
even though we drop uid and gid, the SELinux context remains elevated,
preventing ptrace from attaching.
Reviewed By: simpkins
Differential Revision: D7128692
fbshipit-source-id: 0f9b9a2ebef5a438bf0f4dcdc00a7b703a7aea02
Summary:
`DCHECK_NE()` unfortunately is not constexpr, so it results in a build error
when used in a constexpr function. Fortunately `assert()` is allowed in
constexpr functions as of C++14, so use it instead.
Reviewed By: wez
Differential Revision: D7108887
fbshipit-source-id: 2be9181929a357c23f48c09173646683060aad28
Summary:
This adds a FakeFuse class to simulate a FUSE connection to the kernel, but
that is actually communicating with our userspace test harness code. This also
adds a FakePrivHelper class which implements the PrivHelper interface but
returns FakeFuse connections rather than real FUSE connections to the kernel.
This will enable us to write unit tests that exercise more of the FUSE and
EdenMount logic, and will also enable us to test error conditions and ordering
behaviors that are difficult to reliably reproduce with a real FUSE mount.
This also includes some very basic tests using this new code. The code in
fuse/test/FuseChannelTest.cpp creates a FuseChannel using a FakeFuse object,
and the code in inodes/test/FuseTest.cpp creates a full EdenMount object using
FakePrivHelper and FakeFuse. The tests are pretty similar for now, and only
exercise the FUSE initialization code. I will expand these tests in subsequent
diffs.
Reviewed By: wez
Differential Revision: D7050826
fbshipit-source-id: 4f82375d65664ca3a5a228a577caa4d1d47454fe
