Summary:
Part of the larger project to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
future<T>.then(callable with operator()(folly::Try<T>)) to future<T>.thenTry(callable)
Reviewed By: Orvid
Differential Revision: D10227086
fbshipit-source-id: 1bb31c91cc65e28291e39302627f97801bfde15c
Summary:
Aborting the process with self-access is overlay aggressive. We
already respond with EIO, and the calling syscall should handle the
error. Continue logging at CRITICAL.
Reviewed By: wez
Differential Revision: D10134987
fbshipit-source-id: a4c4eed5e5de20698e95f442b3e063a09db311e6
Summary: We've diverged in a few places from clang-format, so run it across the entirety of Eden.
Reviewed By: wez
Differential Revision: D10137785
fbshipit-source-id: 9603c2eeddc7472c33041ae60e3e280065095eb7
Summary:
If an edenfs thread accesses an Eden mount, it trips SIGABRT from the
DFATAL, but the SIGABRT can't end the process because one thread is
stuck in uninterruptible sleep. So respond to the FUSE request before
aborting the process.
Reviewed By: strager
Differential Revision: D10118625
fbshipit-source-id: b84063e0b186d6a464531284b70c25b0b6a710ce
Summary:
When running the watchman integration tests, I saw that some of them were
fataling here:
diffusion/FBS/browse/master/fbcode/thrift/lib/cpp2/async/StreamPublisher-inl.h$71
In this situation we're tearing down all the subscribers via
`Journal::cancelAllSubscribers` and this causes the stream be destroyed and
trigger this FATAL check.
We need to explicitly complete it before we can proceeed, so we do the gross
thing and call this in the destructor of a little wrapper.
This is made a little more complex because the disconnection handler has to
inform the Journal that it was torn down in the case where the client has
gone away. The result is that we need to be careful to avoid the destructor
on Journal callback from running while we hold the Journal subscriber lock.
Reviewed By: strager
Differential Revision: D10024271
fbshipit-source-id: 06a9d40f7f6e46fe35ffcedba2669e27e6624427
Summary: Reformat all opted-in python code with version `18.9b0` of Black.
Reviewed By: ambv
Differential Revision: D10121017
fbshipit-source-id: 08404dba959bc63bcd7eee7eafe1753c9cfb58ee
Summary:
In D6766900, we turned on automatic background inode
unloading. However, there's a bug somewhere in InodeMap or the loading
sequence that causes occasional crashes for users, so turn it back off
for now.
Reviewed By: pkaush
Differential Revision: D10114443
fbshipit-source-id: fd35fd31b5731aa9bf26004fca8aa73930bb2344
Summary:
Restructured the Windows code to align with the eden code layout. Plus changed the build location to eden/win/build directory, which is generated by the Windows build script.
eden
\_fs
\_ ...
\_ win
\_fs
\_service
\_utils
\_build (generated by the build script)
Reviewed By: strager
Differential Revision: D10081143
fbshipit-source-id: db9fb25f963d1a9cccb8a8f83646e7e45c87d409
Summary: Taking (#ifdef'ing) out some of the mononoke code from the Windows builds. Mononoke support will come once we compile the proxygen on Windows.
Reviewed By: strager
Differential Revision: D10079379
fbshipit-source-id: df431f9c57832d43811af41d4512674f1108cacf
Summary:
I noticed that, when Eden hangs, an importer is usually stuck for a
long time too. Add information about edenfs's child processes in
`eden rage`.
Reviewed By: strager
Differential Revision: D10020317
fbshipit-source-id: afe82f559ea0905f10c757fc0b05c3ff64f5ab41
Summary: HypothesisSimpleTest.test_create is timing out on Sandcastle (Facebook's CI). It looks like the hypothesis tests run indefinitely on Sandcastle. Set a time limit so the tests complete and provide signal.
Reviewed By: wez
Differential Revision: D10053452
fbshipit-source-id: 8c8cb2c375a73b7727e827ea2a9e7d56ab390e54
Summary:
'eden stop' has poor test coverage. Write tests for the common case and some edge cases which 'eden stop' already handles.
This diff should not change behavior.
Reviewed By: wez
Differential Revision: D10017819
fbshipit-source-id: 4d9f5db52187c34c62a9379a6b3dd62f62894233
Summary:
When a file's contents are not cached by the kernel, open(), read(),
and close() are all piped into the FUSE daemon. But when a file's
contents ARE cached by the kernel, only open() and close() are. Thus,
in the common case, the kernel will notify us that a file is being
opened, but read() will be served out of cache.
In that case, prefetching the blob is not beneficial, because it will
be dropped anyway upon close().
In situations where the VFS cache is hot but eden's own caches are
cold, this is probably a win.
Reviewed By: strager
Differential Revision: D10044546
fbshipit-source-id: eeb0854dbff021b2c73f1a42f31a94dd9fcf0837
Summary:
Previously, Eden would log `UnloadInodeScheduler Unloading Free
Inodes` every 10 minutes. Now it only logs if it actually unloads an
inode for a mount.
Reviewed By: strager
Differential Revision: D9999029
fbshipit-source-id: f01d0e7b1341bef230bcb5a327c31ffc8e3aeb2c
Summary:
We're looking into doing some performance optimizations in Eden to the
open() and close() calls, since they are never handled by the kernel
and always pass through to the underlying user-space FUSE daemon. But
first, let's capture some baseline numbers on Eden and non-Eden
checkouts.
Reviewed By: strager
Differential Revision: D10036941
fbshipit-source-id: 64f3414a4572fd963017491db37d70e6b5ae4f24
Summary:
My eden crashed with an invariant violation in InodeMap. It looks
related to background unloading, so I attempted to reproduce the
sequence in a unit test.
These tests pass but they're worth checking in I think.
Reviewed By: strager
Differential Revision: D9998970
fbshipit-source-id: d182fc8d5b185c286082320adda3ea4c862f2f18
Summary:
There's a lot of logic here that need not be offloaded to the
importer threads
Reviewed By: chadaustin
Differential Revision: D9539811
fbshipit-source-id: 1379eef390df6400291a61263d003b493a474b81
Summary: This function is never used.
Reviewed By: strager
Differential Revision: D9998991
fbshipit-source-id: 27a8f5180d7516c3bf61b11192672142f77abccc
Summary: D6612669 introduced a workaround for edenfs not running as a child of sudo if you run Eden's integration tests as root. D9980225 fixed the underlying bug in 'eden stop' which necessitated the workaround. Remove the now-unnecessary workaround.
Reviewed By: wez
Differential Revision: D9984524
fbshipit-source-id: b9ed7114cd7385a13899d16b4da0d40e6e9fd704
Summary:
wait_for_shutdown waits for the process to no longer be alive using kill. Unfortunately, kill considers a zombie process (i.e. a process whose parent process hasn't reaped its children) to be alive. Teach wait_for_shutdown to consider zombie processes as dead.
Here's how I discovered this bug:
1. Launch edenfs via 'eden start'
2. Send SIGSTOP to edenfs
3. Call wait_for_shutdown on edenfs
edenfs' parent process (sudo) noticed edenfs' SIGSTOP and sent SIGSTOP to itself. sudo became stuck and never reaped its child edenfs process. Thus, edenfs was a zombie process, and wait_for_shutdown complained.
Reviewed By: chadaustin
Differential Revision: D9980225
fbshipit-source-id: c2663a850225775571b02553ccf9e2d460241b6d
Summary: I plan on fixing a bug in wait_for_showdown. wait_for_showdown lacks test coverage, so my change could easily break wait_for_showdown. Improve wait_for_showdown's test coverage to prevent bugs.
Reviewed By: chadaustin
Differential Revision: D9980226
fbshipit-source-id: 3f5019a2c5be32b75ca3daa25e799e956c93dab4
Summary:
wait_for_showdown doesn't use its instance parameter. Delete it.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D9980227
fbshipit-source-id: bfc2dd6d84854b20e8c04fc0391edff50d6d0a49
Summary:
While looking into the ESTALE issue, I was tracing through
the kernel code. I found that the root inode is set up with a generation
of 0 rather than 1 in this portion of the kernel code:
diffusion/LK/browse/v4.11-fb/fs/fuse/inode.c;abf7d7755299ee009f0c3cba40fcdcd038a212ac$650
The consequences of the generation being different are that the kernel
will report an ESTALE result when returning the dentry:
diffusion/LK/browse/v4.11-fb/fs/fuse/inode.c;abf7d7755299ee009f0c3cba40fcdcd038a212ac$690
This diff zero-initializes the entry result struct and explicitly sets the generation field to 0.
Sadly, this did not impact the manifestation of the ESTALE behavior that I've been running down.
It's also worth noting that the source of the `1` was from reading this comment in libfuse:
0a519c9772/include/fuse_lowlevel.h (L78-L79)
> The generation must be non-zero, otherwise FUSE will treat it as an error.
That isn't true of the kernel component.
Reviewed By: chadaustin
Differential Revision: D9944646
fbshipit-source-id: 9c1e2f4faec40a3aa446a4646d4518a854a1d73c
Summary: I wanted to see more details from the fuse requests, so log them.
Reviewed By: strager
Differential Revision: D9944649
fbshipit-source-id: 143703528fa029ed51e6cb42a5f6d8b3b0230ca3
Summary:
In heap profiles we observed a lot of objects associated
with the journal related to the subscription path. Those objects
appear to be alive for the duration of the subscription session,
so it gives us good reason to move forwards with updating to the
newer thrift streaming API.
This diff introduces a new endpoint `subscribeStream` that uses
the new RSocket based subscription channel.
Both the Eden server and the watchman client are able to deal
with connections from/to old versions of the server which checks
off the box around push safety.
Once this is widely deployed we can remove `StreamingSubscriber.{cpp|h}`
from the eden code base, and can remove the `legacySubscribe` code
from the watchman code base.
Reviewed By: strager
Differential Revision: D9595967
fbshipit-source-id: 0843e56315f83f1e5fb9bc827b7ee6cf1bf507a6
Summary:
This begins adding a framework for generating snapshots of Eden state for use
in tests. I plan to add a few kinds of tests based on these snapshots:
- Tests that ensure Eden can successfully load snapshots created by older
versions of the code.
- Tests that corrupt the snapshot data in various ways and then confirm that
it can be repaired by `eden fsck` and/or still loaded successfully by
`edenfs` even if it was not repaired with `fsck` first.
This code still needs some additional functionality, but I figured it was
worth checking in what I have so far. The main functionality that remains to
be added is unpacking the snapshots and updating absolute paths inside the
config files so that they work in the new path where they were unpacked.
Reviewed By: strager
Differential Revision: D9690267
fbshipit-source-id: a2660e49b84d7833e6778108d9abe081ab7e2cbd
Summary:
Update `eden doctor` to automatically fix inconsistencies in Eden and
Mercurial's view of the current parent commit by updating Eden to point to
Mercurial's current p1 commit.
The previous instructions were to use `hg reset --keep` to try and reset
Mercurial's data to point at Eden's parent commit. However, the most common
case where Eden and Mercurial can get out of sync is after a Mercurial command
has been interrupted, and Mercurial rolled back a transaction without telling
Eden. In this case the commit that Eden points at does not exist at all, so
trying to use it fails.
Reviewed By: strager
Differential Revision: D9893097
fbshipit-source-id: 3817b83805c7a9959037941204a95ad2bbff2890
Summary:
Move the `_cleanup_tmp_dir()` function out of the integration lib's testcase
module, and make it a public function in the integration lib util module.
This will make it easier to re-use from some other python utilities that need
to be able to clean up Eden state directories.
Reviewed By: strager
Differential Revision: D9690266
fbshipit-source-id: 157c5221f81f90a3e9612315681459d16693cdbc
Summary:
Update the Eden unit tests and integration tests to set the `NOSCMLOG`
environment variable when running `hg` commands. This ensures that our
mercurial telemetry wrapper does not log events from Eden test runs.
Reviewed By: quark-zju, strager
Differential Revision: D9893096
fbshipit-source-id: c0dd4b5eb042dcb5e9493c89aaee10a513022bae
Summary:
If the overlay file for a directory is corrupted (e.g. empty), Overlay::scanForNextInodeNumber throws. This causes Eden to crash on start [1]. Fix the crash by ignoring corrupted directories.
[1] `test_mount_possible_after_corrupt_directory_and_cached_next_inode_number` reproduces this crash.
Reviewed By: chadaustin
Differential Revision: D9806105
fbshipit-source-id: 1b95083b6a6aa253a2296d6f754edbf4b9f64734
Summary: Override `txnutil.mayhavesharedpending` in the same way as `txnutil.mayhavepending`.
Reviewed By: strager
Differential Revision: D9834375
fbshipit-source-id: 7cd9d35121957343e8b15728485457072de047b5
Summary:
Part of the larger project to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
future<T>.then(callable with operator()(folly::Try<T>)) to future<T>.thenTry(callable)
Reviewed By: Orvid
Differential Revision: D9819578
fbshipit-source-id: f9e31f47354c041ecbf0a90953cbe50ebfda6adc
Summary: When a RawOverlayTest test fails, it's sometimes difficult to diagnose the failure because assertions use inode numbers instead of file paths. When a test fails, include inode data (with file paths) in the failure message.
Reviewed By: chadaustin
Differential Revision: D9806106
fbshipit-source-id: 6160632bf8c64ceeb84e9d4709347e9268747ca4
Summary:
The C++ code disallows use of "." in RelativePaths, but it's
reasonable for users of the Thrift API to pass "." to indicate the
root of the mount. Handle that in the EdenServiceHandler.
Reviewed By: strager
Differential Revision: D9647776
fbshipit-source-id: b61c2d1c0dcd69ccfa38bf27379281d10cdf1ceb
Summary:
Reduce some code duplication in RawOverlayTest:
* Factor testDir_-to-AbsolutePath conversion into a getLocalDir function.
* Factor Overlay construction into a loadOverlay function.
* Use getLocalDir to construct the path to next-inode-number.
This refactor will let me use getLocalDir for more things in future diffs.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D9813443
fbshipit-source-id: ab0dc88d36e91fc04e0aeb48468060b93b48f0ec
Summary:
Eden keeps track of the the highest-allocated inode number using the next-inode-number file. The TODO comment in Overlay::scanForNextInodeNumber is thus out of date. Delete the outdated comment since the feature has already been implemented.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D9806107
fbshipit-source-id: f546eb179b86a11a6c4bf0d5f945dc7dbbf29160
Summary:
The checkout operation is a bit hamstrung, performance-wise, when
inodes are loaded. If it's possible the kernel has a reference to an
inode, we can't short-circuit the checkout operation and instead must
precisely add and remove children from the source control tree and
persist the new children and inode numbers into the overlay.
It's much faster if we treat checkout as an inode-number-invalidating
change and unload all inodes that aren't referenced by the kernel.
Reviewed By: strager
Differential Revision: D9781532
fbshipit-source-id: 50912cf89dbe9ae1bd7ef91fdf7bf8ee5eda8667
Summary: Add yet another unloading code path... This one is used for fast checkouts.
Reviewed By: strager
Differential Revision: D9781956
fbshipit-source-id: ae89377aea823f94e2ec1bcc2fa209c8f9bc821c
Summary:
unloadChildrenNow would only unload files or trees at the leaf. Apply
the fixes from D8302998 to unloadChildrenNow, which is only called
during takeover.
Reviewed By: strager
Differential Revision: D9774970
fbshipit-source-id: c2f4d1e6b838cc3b9e99eb8786114e643128a519
Summary: This hardcoded list is blocking using eden + mononoke with wwww
Reviewed By: chadaustin
Differential Revision: D9762557
fbshipit-source-id: 40991b26b702aa87a86a66934a069995d70f9f34
Summary:
To diagnose why a process is hitting Eden so hard, show the top pids
in eden top's output too.
Reviewed By: strager
Differential Revision: D9486293
fbshipit-source-id: 0714c354cafef4dae60b2615d20f7e940e24daf0
Summary:
Add a curses-based `eden top` command that displays the top process
names currently accessing an Eden mount over FUSE.
In the future, I'd like to extend this to count and record Thrift
calls too.
Reviewed By: strager
Differential Revision: D9477936
fbshipit-source-id: f878f2e67f4ea24c036880eb4b1162597dc04185
Summary:
Add a Thrift API for reading the pid access logs from each
EdenMount/FuseChannel. Used in a future diff.
Reviewed By: strager
Differential Revision: D9477867
fbshipit-source-id: 0897a915ca654bca952aecc123ea40105830a75b
Summary: Begin tracking pids passed into FUSE in the ProcessAccessLog.
Reviewed By: strager
Differential Revision: D9595795
fbshipit-source-id: 02e5fefebcd0de860274409ba6258f14fa050b55
Summary:
Part of the larger project to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
Reviewed By: Orvid
Differential Revision: D9696716
fbshipit-source-id: d71433c75af8422b2f16733c0b18a417d5a4cf2e
Summary:
It looks like D9477181 conflicted with D9635507, causing a compile error:
```
eden/fs/utils/ProcessNameCache.cpp:74:15: error: no member named 'names' in 'folly::LockedPtr<folly::Synchronized<facebook::eden::ProcessNameCache::State, folly::SharedMutexImpl<false> >, folly::LockPolicyExclusive>'
state.names.emplace(pid, ProcessName{detail::readPidName(pid), now});
~~~~~ ^
eden/fs/utils/Synchronized.h:48:10: note: in instantiation of function template specialization 'facebook::eden::ProcessNameCache::add(pid_t)::(anonymous class)::operator()<folly::LockedPtr<folly::Synchronized<facebook::eden::ProcessNameCache::State, folly::SharedMutexImpl<false> >, folly::LockPolicyExclusive> >' requested here
return update(wlock);
^
eden/fs/utils/ProcessNameCache.cpp:58:3: note: in instantiation of function template specialization 'facebook::eden::tryRlockCheckBeforeUpdate<folly::Unit, facebook::eden::ProcessNameCache::State, (lambda), (lambda)>' requested here
tryRlockCheckBeforeUpdate<folly::Unit>(
^
eden/fs/utils/ProcessNameCache.cpp:81:15: error: no member named 'waterLevel' in 'folly::LockedPtr<folly::Synchronized<facebook::eden::ProcessNameCache::State, folly::SharedMutexImpl<false> >, folly::LockPolicyExclusive>'
state.waterLevel += 2;
~~~~~ ^
eden/fs/utils/ProcessNameCache.cpp:82:19: error: no member named 'waterLevel' in 'folly::LockedPtr<folly::Synchronized<facebook::eden::ProcessNameCache::State, folly::SharedMutexImpl<false> >, folly::LockPolicyExclusive>'
if (state.waterLevel > state.names.size()) {
~~~~~ ^
3 errors generated.
```
Dereference `state` to fix the error.
Reviewed By: chadaustin
Differential Revision: D9673324
fbshipit-source-id: 5143cd22baf66307e5aacb0a04669de69dde99e8
Summary:
Add a ProcessAccessLog that supports cheaply tracking pids passed
into FUSE. It's a write-many, read-maybe access pattern, so the code
is careful to add as little latency as possible on the hot path.
Reviewed By: strager
Differential Revision: D9477839
fbshipit-source-id: 6928c1d09d55c2b0c0958ac2cb0dc91ec21b370c
Summary: While looking at FuseChannel I noticed an unnecessary Future.
Reviewed By: strager
Differential Revision: D9595672
fbshipit-source-id: 5c84822c4f2c4c3c78b88456e44728e463d5a1e8
