Summary: Now we can write unit tests for takeover. :)
Reviewed By: simpkins
Differential Revision: D7345419
fbshipit-source-id: 05741dab65016c59109bf190c83be2f676a6141d
Summary:
This replaces the `ensureDataLoaded()`, `materializeForWrite()`, and
`materializeAndTruncate()` methods in `FileInode` with new safer alternatives:
`runWhileDataLoaded()`, `runWhileMaterialized()`, and `truncateAndRun()`
These new methods take a function to run, and run it while holding the state
lock with the FileInode guaranteed to be in the desired state. They also
require the caller move in the state lock as an argument, to help ensure that
the caller gets the locking behavior correct.
The old methods required that the caller not already hold the state lock while
calling them. This pre-condition was violated in `FileInode::write()` and
possibly other places. They also required releasing the lock between they
confirmed the file state and when the caller's function ran, requiring
additional lock and unlock operations, and also making the code harder to
reason about since other threads could change the state during the time when
the lock was released.
Reviewed By: chadaustin
Differential Revision: D7363180
fbshipit-source-id: d8e667d0bc7006c519252a8d0682af97517997eb
Summary:
The changes in this diff comments out unused parameters. All changes are automated using clang-tidy.
This will allow us to enable `-Wunused-parameter` as error.
Reviewed By: simpkins
Differential Revision: D7371610
fbshipit-source-id: 0134e2f0b916313d690c073a46d747c52399a226
Summary:
FileInode::read() did not handle being called in the BLOB_LOADING or NOT_LOADED
states. This method is only called from EdenFileHandle::read(), which does no
checking that it is fully loaded first.
This fixes read() to always call ensureDataLoaded() first.
Reviewed By: chadaustin
Differential Revision: D7359878
fbshipit-source-id: f5a1c8a28db3267da3180b67f970430e3ea291da
Summary:
Update edenfs to enable the `async` setting for the default log handler.
This will avoid ever blocking due to logging, and will cause edenfs drop
messages if they are being logged faster than they can be written to stderr.
Reviewed By: chadaustin
Differential Revision: D7348517
fbshipit-source-id: 7cdf5772742e4e92bb15dd4de6315103cd42301e
Summary:
Set the default logging settings by overriding folly::getBaseLoggingConfig()
rather than by setting a default value for the `--logging` command line
argument.
This has two advantages:
- Custom logging settings supplied in the `--logging` argument will now
combined with the base logging settings rather than completely replacing
them. Previously users had to always add "eden=DBG2" to this argument if
they did not want to drop this setting.
- This will make it possible to move the definion of the `--logging` flag to
folly::Init and not lose our custom settings for Eden.
Reviewed By: yfeldblum
Differential Revision: D7348519
fbshipit-source-id: bd9012f387b0460da8bcc9c7c31aef46b1abec5c
Summary:
While debugging a takeover crash (task #25590050) we wanted to help confirm
that inode objects weren't ever getting copied somehow. That turned out not to
be the issue, but it still seems worth explicitly deleting the copy and move
constructors and assignment operators for InodeBase.
Reviewed By: wez
Differential Revision: D7342050
fbshipit-source-id: ae07214072bfb8353584a07066aae2dc1adf2797
Summary:
Add a new integration test that performs a graceful restart after invoking the
getScmStatusBetweenRevisions() thrift call.
Prior to D7341609 this would cause edenfs to crash on shutdown with the error
"!!BUG!! After InodeMap::shutdown() finished, 2 inodes still loaded; they must
all (except the root) have been unloaded for this to succeed!"
Before D7341609 the `EdenMount::diffRevisions()` created a new temporary inode
tree solely to perform a diff. This resulted in multiple root inodes that all
pointed to the same EdenMount, but the EdenMount didn't know about any of these
alternate root inodes. These temporary inode trees never got destroyed,
causing this error on shutdown.
Reviewed By: chadaustin, wez
Differential Revision: D7333005
fbshipit-source-id: 8406d2e2ceb00264050b0aceec583baae2da69ec
Summary:
Update the getScmStatusBetweenRevisions() thrift call to use the new
diffCommits() function that diffs source control Tree objects without creating
TreeInode objects to perform the diff.
This addresses two bugs:
- Each call to EdenMount::diffRevisions() constructed a new root inode
associated with the mount, and this would never get destroyed. It was not
destroyed at the end of the diffRevisions() call since inodes are normally
not destroyed immediately when they are unreferenced. It was not destroyed
during EdenMount::shutdown() since EdenMount didn't have any references to
these additional root inode structures and their children.
- EdenMount::diffRevisions() incorrectly swapped ADDED and REMOVED statuses in
the result.
Reviewed By: wez
Differential Revision: D7341609
fbshipit-source-id: 16e755a0ff685f51c977c3b27d6af96908f33494
Summary:
Add a function for diffing two source control commits without needing to
instantiate TreeInode objects.
Reviewed By: wez
Differential Revision: D7341604
fbshipit-source-id: 557eef87faa2785ab96d51b09569a46f892a71f6
Summary:
Update Eden's thrift service handler code to accept BinaryHash arguments either
as 20-byte binary values or as 40-byte hexadecimal values.
This will make it easier to transition APIs like getScmStatusBetweenRevisions()
to use 20-byte binary hash arguments without breaking existing clients.
Reviewed By: wez
Differential Revision: D7341607
fbshipit-source-id: 3e952211900d3ec4b9c2073cf3afd55ae7e253ea
Summary:
Add an integration test for the getScmStatusBetweenRevisions() thrift call.
This call apparently gets the ADDED and REMOVED states backwards. For now the
test checks for the current (incorrect) behavior.
This also fixes the thrift definition for this function to stop using the
BinaryHash typedef. Unlike most of our other thrift functions this method
appears to require the arguments as 40-byte hexadecimal strings.
Reviewed By: wez
Differential Revision: D7341606
fbshipit-source-id: 73cbd0ecf4445da6b1f0ef9cf6d9dce47e6fb593
Summary:
This makes gitrepo.commit() return the new commit ID as a hexadecimal string,
just like hgrepo.commit() currently does.
Reviewed By: wez
Differential Revision: D7341605
fbshipit-source-id: 83ebddb8c23d5e4650432bea6f8dcb8d18c0ff38
Summary:
Update the hgrepo and gitrepo helper classes with a new `remove_file()` method,
and also improve the git code to include the process stderr in the exception
message if the command fails.
Reviewed By: wez
Differential Revision: D7341610
fbshipit-source-id: 28cca89520923d92bba4833a4dcfab6d21357cfb
Summary:
Update repobase.py, hgrepo.py, and gitrepo.py to include python type hints for
all functions and to pass most of mypy's strict type checking.
There are still a couple minor errors around the fact that hgrepo.hg() and
gitrepo.git() return an `Optional[str]` rather than a plain `str`. Many callers
know that they cannot return None unless the stdout argument is None, but mypy
can't figure this out. In the long run maybe we should split these into two
separate methods, one that always returns `str` and one that always returns
`None`.
Reviewed By: wez
Differential Revision: D7341608
fbshipit-source-id: c62da578fb32edb9272363fadabbdc11b1d5d2c2
Summary:
Fix the code to generate exceptions based on an errno error using
std::generic_category rather than std::system_category.
Reviewed By: yfeldblum
Differential Revision: D7329997
fbshipit-source-id: 3fe257bbbc7a631c801f31120592c8bdbc25c8bf
Summary:
Add a new utils/SystemError.h header with helper functions to check if a
`std::system_error` contains an errno value.
Most of the code in Eden previously only checked for `std::system_category`
when looking for errno values. `std::generic_category` is the correct category
to use for errno exceptions, but folly/Exception.h incorrectly throws them as
`std::system_category` today. This change makes Eden treat either error type
as errno values for now.
Reviewed By: yfeldblum
Differential Revision: D7329999
fbshipit-source-id: 67a3c3ea10371c53a2e34236b7575deac4cbd53a
Summary:
Added a thrift call to return the outstanding FUSE requests.
Cli will call the thrift and print the output.
Added a unit test to test getOutstandingRequests().
Reviewed By: simpkins
Differential Revision: D7314584
fbshipit-source-id: 420790405babdb734f598e19719b487096ec53ca
Summary: Fix a simple TODO in EdenServiceHandler.cpp
Reviewed By: chadaustin
Differential Revision: D7329635
fbshipit-source-id: b42b2cc13be3ad5b18a629ed15f6c51cea52fbda
Summary:
This helps distinguish `facebook::eden::FileHandle` from
`facebook::eden::fusell::FileHandle`, and will make it possible to eliminate
the separate `facebook::eden::fusell` namespace in the future.
Reviewed By: chadaustin
Differential Revision: D7314456
fbshipit-source-id: 0869427766fe666d119a59c7df1c97825a8ad36e
Summary:
This test exercises code under eden/fs/fuse, so put it with that directory's
tests.
Reviewed By: chadaustin
Differential Revision: D7314457
fbshipit-source-id: b664aaa3086b2e65f7ae8b76ae71878f39fd9d17
Summary:
There's some code that uses Entry::hasInodeNumber to decide whether an
inode is in the process of being loaded. This change is a
prerequisite for D7052470.
Reviewed By: simpkins
Differential Revision: D7311182
fbshipit-source-id: e04a89c814950e6ce33e73b963b3ffb0f0e353b8
Summary:
As of D7300847 the FuseChannel code will now return the FUSE FD as a closed
File object if the mount was unmounted, even if a takeover stop was requested.
This updates the EdenServer code to deal with the fact that it might receive a
closed FD even during a takeover. The correct behavior is to simply skip this
mount when building the takeover state to pass to the new process.
This is a somewhat minimal change to fix up this bit of the code. It would
probably be nice to continue simplifying this logic in the future.
Reviewed By: wez
Differential Revision: D7300877
fbshipit-source-id: c2f617449e155e4b42784ac42ec3aaa378e3288b
Summary:
Previously the FuseChannel code unconditionally set `connInfo_.major` and
`connInfo_.minor` to the FUSE version that it supports, rather than the FUSE
version supported by the kernel it is talking to.
This does not appear to be the intended behavior, given that in several other
location it checks the value of `connInfo_->minor` to see if particular
features are supported.
Reviewed By: wez
Differential Revision: D7300845
fbshipit-source-id: 1f5946134037b2cbe03a8b86241acaa401c21ffa
Summary:
Remove the `FuseChannel::stealFuseDevice()` method, and instead change the
session completion future to return the FUSE device FD if it is still valid.
This ensures we only extract the FUSE device when it is still valid: if an
unmount event is being processed by a FUSE worker thread simultaneously with a
call to `FuseChannel::takeoverStop()` the session completion future will return
an empty `folly::File` since the FUSE device has been shut down.
This also helps clarify the synchronization semantics around modification to
the `fuseDevice_` member variable.
Reviewed By: wez
Differential Revision: D7300847
fbshipit-source-id: 02ced8fc9d24e7cd526d911782949d0bfbf0e5a7
Summary:
I noticed from the logs that this was funky:
```
C0131 01:55:28.870926 24027 Bug.cpp:55] EDEN_BUG at eden/fs/inodes/InodeMap.cpp425: !!BUG!! InodeMap::save() called with 104957 inodes still loaded; they must all (except the root) have been unloaded for this to succeed!
```
Reviewed By: chadaustin
Differential Revision: D7310995
fbshipit-source-id: 8c7840bbd60e3e5e815c8cef0d50bf4021e89baf
Summary:
Eliminate a few unnecessary includes from Dispatcher.h and one from
RequestData.h
Reviewed By: wez
Differential Revision: D7300846
fbshipit-source-id: 0cde1f70f605fcb5c95ed13950f4fa8d353fd72f
Summary:
FuseChannel::processSession() was missing a return statement after we received
EOF from the FUSE device. This would cause us to fall through and try
processing the empty buffer as a request. This would fail, and we would try to
send a reply, which would in turn fail since the FUSE device was closed.
This was only an issue with the unit tests: on real FUSE devices we get an
ENODEV error when reading from the device. In unit tests we get an EOF
instead.
Reviewed By: wez
Differential Revision: D7299336
fbshipit-source-id: c275e03edbf594c0d120257b1cfc89d7c75f60f6
Summary:
The FuseChannel destructor previously could be invoked while there are still
outstanding requests, which would cause problems when responses were generated
for these requests.
This makes the FuseChannel destructor private, requiring users to always
allocate FuseChannel objects on the heap and destroy them using a destroy()
method. The destroy() method still blocks waiting for the FUSE worker threads
to exit, but if there are still outstanding FUSE requests it defers the
deletion of the FuseChannel until they are complete.
The EdenMount class currently couldn't ever destroy FuseChannel objects with
outstanding requests, since it waits for the completion future to fire before
invoking the destructor. Fixing this behavior is mostly for correctness of the
FuseChannel API and for the benefit of our unit tests.
Reviewed By: chadaustin
Differential Revision: D7297829
fbshipit-source-id: 643d1332be84a1f25ee30ba2a2ea3515806f00ef
Summary:
Make ServerState also store the clock as a member variable. This moves more
server-wide state into the ServerState class, and allows us to eliminate
another argument to the EdenMount constructor.
Reviewed By: chadaustin
Differential Revision: D7297832
fbshipit-source-id: 2a063d67752f46686987390b3faefb304455568a
Summary:
Move the EdenCPUThreadPool from EdenServer to ServerState.
This ensures that all EdenMount objects always have a thread pool. Previously
most unit tests were creating an EdenMount without a thread pool set, but the
FileInode code and TreeInode code could end up wanting to use the thread pool
even if no FuseChannel is running.
Reviewed By: chadaustin
Differential Revision: D7297828
fbshipit-source-id: 9861b3439cd091afe31fa13f449aa14656983318
Summary:
Add a simple test program that starts a FUSE mount. At the moment the only
call it actually responds to is getattr() for the root inode. However this is
sufficient for exercising the FuseChannel startup and shutdown code.
Reviewed By: chadaustin
Differential Revision: D7297827
fbshipit-source-id: 1bae6015def37913a5bf57541a14c3c8c050eff5
Summary:
Remove the `FuseChannel::getSessionCompleteFuture()` method and instead have
`initialize()` and `initializeFromTakeover()` return the `Future` that can be
used to wait on session completion.
This makes it explicit from an API perspective that this session completion
future will only be invoked if initialization is successful. It also
eliminates the possibility of anyone calling getSessionCompleteFuture() more
than once.
I also changed the session completion future to use a folly::SemiFuture rather
than a folly::Future, to make it explicit that callers generally should not
execute their callback inline in the same thread where the SemiFuture is
fulfilled.
Reviewed By: chadaustin
Differential Revision: D7297835
fbshipit-source-id: 3a1157951f0738f1692833ed5e875c3e9c6d6d69
Summary:
The TakeoverServer::ConnHandler code uses a folly::EventBase to drive the
socket I/O, and therefore must perform all I/O from EventBase thread.
This changes ensures that we always call `sendTakeoverData()` from the correct
thread.
Up until now this has happened to work since the future returned by
`EdenServer::startTakeoverShutdown()` happened to always be completed in the
EdenServer main EventBase thread, and this is the same thread used for the
TakeoverServer socket.
This change makes it explicit that this particular code must be run from the
correct EventBase thread.
Reviewed By: chadaustin
Differential Revision: D7297834
fbshipit-source-id: cbace5f63ea13aa62dd53be0b89d59bea2779cf4
Summary: Testing convenience. The NO_CHECK constructor is no longer necessary now that simpkins made the original constructor constexpr.
Reviewed By: simpkins
Differential Revision: D7295539
fbshipit-source-id: 717f5a7a8d3d9790966fad441da7b3308495e2c3
Summary: Cleaning up the takeover initialization code for EdenMount and InodeMap.
Reviewed By: simpkins
Differential Revision: D7294419
fbshipit-source-id: 58506f04259bb1017e6cd2e80e40e5820de6200f
Summary:
Instead of having a rule that save() must be called after
InodeMap::shutdown, just have InodeMap::shutdown return the
SerializedInodeMap if it's desired.
Reviewed By: simpkins
Differential Revision: D7292773
fbshipit-source-id: 2ba35fc729e19af122fe5d6c5a3287ad6b8af946
Summary:
The FuseChannel unit tests would sometimes hang because a worker thread would
still be stuck in a blocking read() call.
The issue appears to be that the kernel would often automatically restart the
interrupted read(), so the worker would never wake up. This switches the code
from using SIGPIPE to wake up, and instead uses SIGUSR2, and installs a handler
for SIGUSR2 with the SA_RESTART flag unset.
It's possible that this was an issue only for the unit tests, where the FUSE
device is a socket. I'm not sure if the kernel would automatically restart
read operations on a real FUSE device.
There are still theoretically some race conditions here, since
`requestSessionExit()` could be called right after a worker thread checks
`(runState_` and before it enters the blocking `read()` call. Fixing that
would likely require switching to something like `folly::EventBase` or manually
using epoll. Fortunately I haven't seen that be an issue in practice so far.
Reviewed By: chadaustin, wez
Differential Revision: D7282002
fbshipit-source-id: 99d29de13a7858dd25f258fdc94d8f8c71ee84d9
Summary:
This changes FuseChannel to fulfill the session complete future immediately in
the thread where it finishes. This may occur inside a FUSE worker thread.
It is now up to the caller to use `via()` if they want their callback to run in
a separate thread.
This simplifies the FuseChannel code and also addresses a crash that occurs in
the unit tests sometimes: If the FuseChannel destructor is invoked without
explicitly invoking the FuseChannel, it would schedule the session complete
promise to be fulfilled in a separate EventBase thread. However, the promise
may then have already been destroyed by the time it can run in the EventBase
thread.
There are still additional synchronization issues to fix in the FuseChannel
tests, but this resolves one problem.
Reviewed By: chadaustin
Differential Revision: D7282001
fbshipit-source-id: be64d3ef6a0e664ed7a2cf93a549acc140184095
Summary:
If you run `eden daemon` on a machine where sudo needs input, sudo's
output would get redirected to edenfs.log and eden daemon would appear
to hang. Worse, if you pressed ctrl-C, sudo would remain in the
background and continue to swallow keypresses from the tty, appearing
to somewhat break your shell until it gave up. The fix is to stop
redirecting stdout and stderr from Python and instead have edenfs
redirect itself into the log if given.
Reviewed By: simpkins
Differential Revision: D7223399
fbshipit-source-id: bae17b150b6594a3dd87bb88741c8dcefd1c5213
Summary:
Update FuseChannel::startWorkerThreads() to return without doing anything if
runState_ indicates that the FuseChannel is currently stopping.
This addresses a race betwen startWorkerThreads() and the destructor. If the
destructor is invoked in parallel with startWorkerThreads() the destructor can
grab the state_ lock and clear out the workerThreads list first. This then
causes problems when startWorkerThreads() runs and adds new threads to the
workerThreads list, which then never get joined.
Reviewed By: wez
Differential Revision: D7253574
fbshipit-source-id: f2cac11f1e71e1a14e3f020368152e0f2948c625
Summary:
Update FuseChannel::readInitPacket() to honor `runState_` and stop if it is
changed to indicate that the channel is stopping. Previously the init worker
thread would always wait until it received an INIT packet or an error on the
FUSE channel.
This also refactors the `readInitPacket()` code slightly mainly to to reduce
the level of indentation.
Reviewed By: wez
Differential Revision: D7253575
fbshipit-source-id: 7f27698947b629daecd9cd4ddffb6e6a921ce41e
Summary:
Update the FuseChannel code to keep track of why it is stopping, and to return
this data in the session complete future.
This is mainly just available for diagnostic reasons, and at the moment nothing
is using this outside of the unit tests.
Reviewed By: wez
Differential Revision: D7253573
fbshipit-source-id: 0c7e5b8bd9c3c1de3788918c2257c06d4fe0a90c
Summary:
Empty .gitignore files should be processed normally, and treated as no new
ignore rules. The logic in D6659654 had accidentally introduced a bug where we
would completely skip processing any directories that contained empty
.gitignore files.
Reviewed By: wez
Differential Revision: D7261276
fbshipit-source-id: 033e199f15d7763bff5f9a080a226c50e481aa9d
Summary:
Catch exceptions thrown by FuseChannel::processSession() and terminate the
FuseChannel if any errors occur.
Also explicitly mark `fuseWorkerThread()` and `initWorkerThread()` as
`noexcept` so we will crash with useful exception backtraces if an unhandled
exception does ever attempt to propagate out of these functions. (Prior to
gcc 8.x, `std::thread()` ends up swallowing the exception backtrace information
if a thread function throws an exception.)
Reviewed By: chadaustin
Differential Revision: D7243909
fbshipit-source-id: fbe7173a2167532d7e42901f810c17bb173d8d63
Summary:
- Update FuseChannel::initializate() to not require an Executor. Rather than
performing initialization using the supplied Executor, it simply starts one
worker thread first and performs initialization there.
- Change the API semantics slightly so that the session complete future is
invoked only if initialization succeeds. Previously the session complete
future could be invoked if initialization failed as well.
- Replace the activeThreads counter with a stoppedThreads counter to determine
when the session complete future should be invoked. The previous
activeThreads behavior was somewhat racy, since the activeThreads counter was
incremented inside each worker thread, but most places that checked it did
not wait to ensure that all worker threads had successfully incremented it
yet.
Reviewed By: chadaustin
Differential Revision: D7243910
fbshipit-source-id: 93b3465509bd9bf6fa90ea097e70dac3193172f9
Summary: It's a little surprising to me the clang build passed without these explicit template instantiations!
Reviewed By: simpkins
Differential Revision: D7253536
fbshipit-source-id: 2f48d5571777f4e978b6947183eefb03158d5014
Summary:
Split up the FuseChannel and EdenMount APIs for starting a brand new FUSE
channel vs taking over an existing channel.
These operations are somewhat different as initializing a new FUSE channel
requires performing a negotiation with the kernel that will not complete
immediately. Therefore these APIs return a Future object. Taking over an
existing FUSE channel is always an immediate operation, and therefor does not
need to return a Future object.
Reviewed By: chadaustin
Differential Revision: D7241997
fbshipit-source-id: 22780f2df76b2d554ab2a4043b6425fa7a4e9c94
