Summary:
These don't compile on Windows, and in order to get mode/win to compile, we
need to avoid compiling their contents. Ideally, we could do that in the
TARGETS files with the select statement, but that's not available in fbcode.
Thus, we do the next best thing: ifdef the file entirely.
Reviewed By: wez
Differential Revision: D23871728
fbshipit-source-id: b4d9df6503eaa008e649afd7bdc665cd37a9585d
Summary:
Most of the RequestData code is platform generic, but bits of it are currently
strongly tied to FUSE. By splitting these 2 parts, we will be able to use the
RequestContext class in Windows too and not having to re-implement all the
logic there.
Reviewed By: chadaustin
Differential Revision: D23482072
fbshipit-source-id: 857fd9ca4264d0f308ec10cc487e9ff3eeb5ee16
Summary:
Setting up, tearing down, and querying RequestContext has some
overhead that I would like to avoid in the inner FUSE loop, so replace
RequestData with a single class that's heap-allocated at the start of
a request and is guaranteed to survive until the request ends, and is
otherwise explicitly passed where it's needed.
Reviewed By: kmancini
Differential Revision: D22712310
fbshipit-source-id: fc30d0b0f7e22b39306b857194ea07a913110b0f
Summary: This commit makes `readdir()` to correct send its `RequestContext` to `EdenDispatcher` method so underlying code can know the current request being processed is from FUSE.
Reviewed By: xavierd
Differential Revision: D21821949
fbshipit-source-id: f41ba912fedbfc040e3c9267aad25e7f33f8e912
Summary:
This diff makes `lookup` to use `RequestData` as `ObjectFetchContext` in `getattr` calls.
This will make sure we correctly record backing store fetches in `eden top`
Reviewed By: chadaustin
Differential Revision: D21780969
fbshipit-source-id: 468e2fadcebf4a00477bc5de434e6c658b99d1ce
Summary: This diff starts to use `RequestData` as `ObjectFetchContext` in our `read()` methods. This ensures EdenFS could track backing store fetches happened in FUSE requests.
Reviewed By: chadaustin
Differential Revision: D21792503
fbshipit-source-id: 9509a1bc8f28100a0dfe196e312c4785c7842345
Summary:
When we implemented FUSE_NO_OPEN_SUPPORT and FUSE_NO_OPENDIR_SUPPORT,
we forgot to remove the FileHandle and FileHandleBase code.
Reviewed By: pkaush
Differential Revision: D17991710
fbshipit-source-id: dfeb26d512f017cef7710929ccff1f6940cf8641
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary:
I'm confused by the naming of EdenThreadStats and ThreadLocalEdenStats. For example, when I see "ThreadLocalEdenStats", I think that such objects can only be accessed by one thread. That's definitely not what "ThreadLocalEdenStats" means!
I think the following naming scheme makes more sense:
* **EdenThreadStats**: Statistics which are updated by one thread. Currently called EdenThreadStats.
* **EdenStats**: Statistics for all threads. Provides access to EdenThreadStats. Currently called ThreadLocalEdenStats.
Implement my preferred scheme: rename ThreadLocalEdenStats to EdenStats.
This diff should not change behavior.
Note: Prior to D14822274, EdenThreadStats was called EdenStats.
Reviewed By: simpkins
Differential Revision: D14822271
fbshipit-source-id: bd20179b1010588e3fc16dc9ed0657d458606f16
Summary:
the osxfuse implementation includes some additional
fields in the `fuse_attr` struct. One of those fields is
`flags`. We were not initializing this field when converting
the stat data to fuse attributes which resulted in it holding
"random" data. In debug builds this seemed to usually end up
zeroed out but in release builds it was usually a bit pattern
that caused the kernel to respond with EPERM when attempting
to access the files. I didn't capture exactly what that
bit pattern was, just that initializing the struct to zeroes
reliably fixed up the EPERM issues in the Release build.
Reviewed By: chadaustin
Differential Revision: D13680004
fbshipit-source-id: 6b2ce6c10ef8f7db4a8a50bd3f2ddcfdddc3bb45
Summary:
Avoid this compilation error:
```
[ 23%] Building CXX object eden/fs/fuse/CMakeFiles/eden_fuse.dir/Dispatcher.cpp.o
In file included from /Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.cpp:10:
In file included from /Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.h:12:
In file included from /Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/Portability.h:19:
In file included from /Library/Developer/CommandLineTools/usr/include/c++/v1/cstddef:110:
/Library/Developer/CommandLineTools/usr/include/c++/v1/type_traits:3141:38: error: incomplete type 'facebook::eden::DirList' used in type trait expression
: public integral_constant<bool, __is_constructible(_Tp, _Args...)>
^
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/futures/Future.h:146:36: note: in instantiation of template class 'std::__1::is_constructible<facebook::eden::DirList>' requested here
typename std::enable_if<std::is_constructible<T, Args&&...>::value, int>::
^
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/futures/Future.h:148:12: note: while substituting prior template arguments into non-type template parameter [with Args = <>]
explicit FutureBase(in_place_t, Args&&... args);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/futures/Future.h:115:7: note: while substituting deduced template arguments into function template 'FutureBase' [with Args = <>, $1 = (no value)]
class FutureBase {
^
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.cpp:175:13: note: in instantiation of template class 'folly::Future<facebook::eden::DirList>' requested here
Dispatcher::readdir(InodeNumber, DirList&&, off_t, uint64_t) {
^
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.h:35:7: note: forward declaration of 'facebook::eden::DirList'
class DirList;
^
1 error generated.
```
Reviewed By: chadaustin
Differential Revision: D13475720
fbshipit-source-id: 2da1692010a82b73cbed71d996993cf5fc13af0e
Summary:
ported forward from D4209167, add a couple of helpers
to access these fields on mac and linux, centralizing/minimizing ifdefs.
Simplify some of the logic in FileChangeMonitor.
Reviewed By: chadaustin
Differential Revision: D13475717
fbshipit-source-id: d7b39999808bc41a6dc17a87189501cb34e68b30
Summary:
Eden no longer tracks any state in file handles, and has no plans to in the future.
Therefore, remove all related code.
Reviewed By: strager
Differential Revision: D13354307
fbshipit-source-id: 341d081f64c6c8fb2b4b1b5a5ff42f2cc7d38039
Summary:
Previously, a file handle must have been held for the entirety of a write operation. That is no
longer true. Stop looking up file handles on write.
Reviewed By: strager
Differential Revision: D13325662
fbshipit-source-id: 9ae31b467d17d633c388917d18098e6e5a620b89
Summary:
Send readdir requests to TreeInode. This may not sound like a good
idea: the FUSE documentation suggests that stateful directory handles
are required to implement correct readdir semantics under concurrent
deletes and renames. However, the 63-bit offset value is treated as a
cookie that is passed from one readdir call into the next, and 63 bits
should be sufficient to implement readdir concurrent with
rename/unlink. So move readdir's implementation into TreeInode in
preparation for the complete removal of TreeInodeDirHandle.
Reviewed By: strager
Differential Revision: D13287664
fbshipit-source-id: c0d615675edd9b83353534468a69b89068bba923
Summary:
Send fsyncdir straight through the inode rather than going through
DirHandle. This is the better design anyway, since the DirHandle does
not receive directory-mutating requests like mkdir.
Reviewed By: strager
Differential Revision: D13287610
fbshipit-source-id: 154fa32a3877c89a204a2d10b4e2b637410d9486
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary: Always send write requests straight to the inode rather than going through FileHandle.
Reviewed By: wez
Differential Revision: D10220619
fbshipit-source-id: 9ce328583cf0fa9d7d8850d92d9e15ddc382d6a3
Summary:
Always send read requests straight to the inode rather than going
through the FileHandle.
Reviewed By: wez
Differential Revision: D10220604
fbshipit-source-id: 6aa5d20f3ce09696a29bd5c1cb95d0b987ab213c
Summary:
Promote the folly logging code out of the experimental subdirectory.
We have been using this for several months in a few projects and are pretty
happy with it so far.
After moving it out of the experimental/ subdirectory I plan to update
folly::Init() to automatically support configuring it via a `--logging` command
line flag (similar to the initialization it already does today for glog).
Reviewed By: yfeldblum, chadaustin
Differential Revision: D7755455
fbshipit-source-id: 052db34c97f7516728f7cbb1a5ad959def2f6efb
Summary:
To avoid bugs similar to the ones fixed in D7781691, mark a bunch of
folly::Future<folly::Unit> functions with FOLLY_NODISCARD.
Reviewed By: simpkins
Differential Revision: D7782224
fbshipit-source-id: 23ba42aa63011cc33e5a6e18d5bc6d00403a78d3
Summary:
Eliminate a few unnecessary includes from Dispatcher.h and one from
RequestData.h
Reviewed By: wez
Differential Revision: D7300846
fbshipit-source-id: 0cde1f70f605fcb5c95ed13950f4fa8d353fd72f
Summary:
This diff moves the mount-time initialization handling
out of the main loop. This rationale for this is:
* We don't (and shouldn't!) need to process FUSE_INIT for takeover
processing, and this structure allows us to make stronger assertions
about our state.
* we can avoid spinning up multiple threads in the (rare!) case that
the FUSE_INIT fails
* It is now a little harder for exceptions during initialization to
escape our notice.
In rearranging this stuff, I found a race condition in the worker thread
shutdown; we could erroneously emit a completion event before all of
the threads had been torn down and this resulted in sporadic integration
test failures hitting the assertion for the number of joined threads
in the destructor.
Reviewed By: simpkins
Differential Revision: D6766330
fbshipit-source-id: 32afb5a7c739c75aebfdb0a8f896eec5f41ad33f
Summary: Replaced memset used to 0-init structs with POD fields with = {} init.
Reviewed By: simpkins
Differential Revision: D6770868
fbshipit-source-id: 190205a277e3c73711813bc515b860b107d41f83
Summary:
Update the statfs() code to return non-zero values for namelen and frsize.
Returning 0 for namelen was causing problems for programs that checked
`pathconf(path, _PC_NAME_MAX)` and tried to honor this value. For instance,
GNU patch would try to generate files with a 0-length name since we indicated
the maximum name length was 0.
I haven't investigated too closely, but this behavior might have broken only
recently when we stopped using libfuse. libfuse may have been setting this to
255 for us. I didn't see code in libfuse that would do this, but I'm fairly
sure GNU patch was working correctly very recently.
Reviewed By: wez
Differential Revision: D6710370
fbshipit-source-id: fc9a0320cd7c1eb2545219a3ec123c2f0644fb5d
Summary:
This serves a few purposes:
1. We can avoid some conditional code inside eden if we know that
we have a specific fuse_kernel.h header implementation.
2. We don't have to figure out a way to propagate the kernel
capabilities through the graceful restart process.
3. libfuse3 removed the channel/session hooks that we've been
using thus far to interject ourselves for mounting and
graceful restarting, so we were already effectively the
walking dead here.
4. We're now able to take advtange of the latest aspects of
the fuse kernel interface without being tied to the implementation
of libfuse2 or libfuse3. We're interested in the readdirplus
functionality and will look at enabling that in a future diff.
This may make some things slightly harder for the more immediate
macOS port but I belive that we're in a much better place overall.
This diff is relatively mechanical and sadly is (unavoidably) large.
The main aspects of this diff are:
1. The `fuse_ino_t` type was provided by libfuse so we needed to
replace it with our own definition. This has decent penetration
throughout the codebase.
2. The confusing `fuse_file_info` type that was multi-purpose and
had fields that were sometimes *in* parameters and sometimes *out*
parameters has been removed and replaced with a simpler *flags*
parameter that corresponds to the `open(2)` flags parameter.
The *out* portions are subsumed by existing file handle metadata
methods.
3. The fuse parameters returned from variations of the `LOOKUP` opcode
now return the fuse kernel type for this directly. I suspect
that we may need to introduce a compatibility type when we revisit
the macOS port, but this at least makes this diff slightly simpler.
You'll notice that some field and symbol name prefixes vary as
a result of this.
4. Similarly for `setattr`, libfuse separated the kernel data into
two parameters that were a little awkward to use; we're now just
passing the kernel data through and this, IMO, makes the interface
slightly more understandable.
5. The bulk of the code from `Dispatcher.cpp` that shimmed the
libfuse callbacks into the C++ virtual methods has been removed
and replaced by a `switch` statement based dispatcher in
`FuseChannel`. I'm not married to this being `switch` based
and may revise this to be driven by an `unordered_map` of
opcode -> dispatcher method defined in `FuseChannel`. Regardless,
`Dispatcher.cpp` is now much slimmer and should be easier to
replace by rolling it together into `EdenDispatcher.cpp` in
the future should we desire to do so.
6. This diff disables dispatching `poll` and `ioctl` calls. We
didn't make use of them and their interfaces are a bit fiddly.
7. `INTERRUPT` is also disabled here. I will re-enable it in
a follow-up diff where I can also revise how we track outstanding
requests for graceful shutdown.
8. I've imported `fuse_kernel.h` from libfuse. This is included
under the permissive 2-clause BSD license that it allows for
exactly this integration purpose.
Reviewed By: simpkins
Differential Revision: D6576472
fbshipit-source-id: 7cb088af5e06fe27bf22a1bed295c18c17d8006c
Summary:
There's a bug in some combination of Eden and FUSE where open(O_TRUNC)
followed by a sequence of writes over an existing file does not flush
the kernel's VFS page cache, which manifests as an mmap larger than
the file's size not zeroing the data beyond the file's size. These
tests attempt capture that use case, but they are fiddly.
Disabling ATOMIC_O_TRUNC seems to resolve the issue.
Reviewed By: wez
Differential Revision: D6430152
fbshipit-source-id: f7626e268e778ebab60c66322e0ce42bce746ae1
Summary:
Per discussion with bolinfest, this brings Eden in line with clang-format.
This diff was generated with `find . \( -iname '*.cpp' -o -iname '*.h' \) -exec bash -c "yes | arc lint {}" \;`
Reviewed By: bolinfest
Differential Revision: D6232695
fbshipit-source-id: d54942bf1c69b5b0dcd4df629f1f2d5538c9e28c
Summary:
CodeMod: Change `#include`'s of `wangle/concurrent/GlobalExecutor.h` to use `folly`.
The file in `wangle/` is just a shim around the same file in `folly/executors/GlobalExecutor.h`. Just codemod all the `#include` sites.
Reviewed By: Orvid
Differential Revision: D5981467
fbshipit-source-id: ad7f0dce959e2760d3977b04925945e0447abc1d
Summary:
This is a mechanical and dumb move of the code from MountPoint
and into the EdenMount class.
Of note, it doesn't merge together the two different state/status fields
into a unified thing; that will be tackled in a follow on diff.
Reviewed By: bolinfest
Differential Revision: D5778212
fbshipit-source-id: 6e91a90a5cc760429d87a475ec12f81b93f87be0
Summary:
This is leading up to folding the MountPoint code into
the EdenMount class.
There's still a mention of the MountPoint in Dispatcher.h; that is
being dealt with in the following diff.
Reviewed By: bolinfest
Differential Revision: D5778215
fbshipit-source-id: 996640b3773988a4738ad55bb13de45e1ffe1880
Summary:
The higher level goal is to make it easier to deal
with the graceful restart scenario.
This diff removes the SessionDeleter class and effectively renames
the Channel class to FuseChannel. The FuseChannel represents
the channel to the kernel; it can be constructed from a fuse
device descriptor that has been obtained either from the privhelper
at mount time, or from the graceful restart procedure. Importantly
for graceful restart, it is possible to move the fuse device
descriptor out of the FuseChannel so that it can be transferred
to a new eden process.
The graceful restart procedure requires a bit more control over
the lifecycle of the fuse event loop so this diff also takes over
managing the thread pool for the worker threads. The threads
are owned by the MountPoint class which continues to be responsible
for starting and stopping the fuse session and notifying EdenServer
when it has finished. A nice side effect of this change is that
we can remove a couple of inelegant aspects of the integration;
the stack size env var stuff and the redundant extra thread
to wait for the loop to finish.
I opted to expose the dispatcher ops struct via an `extern` to
simplify the code in the MountPoint class and avoid adding special
interfaces for passing the ops around; they're constant anyway
so this doesn't feel especially egregious.
Reviewed By: bolinfest
Differential Revision: D5751521
fbshipit-source-id: 5ba4fff48f3efb31a809adfc7787555711f649c9
Summary:
Update all of the code using ThreadLocal<EdenStats> to pass in a non-default
Tag parameter to the ThreadLocal template.
A non-default tag parameter is required to use the accessAllThreads() method on
the ThreadLocal object. We need to use accessAllThreads() to perform stats
aggregation correctly. Currently the EdenServer code is only performing
aggregation for stats in the FunctionScheduler.
This diff only updates the ThreadLocal<EdenStats> type, but does not contain
any behavior changes. I will fix the stats aggregation in a subsequent diff.
Reviewed By: bolinfest
Differential Revision: D5657268
fbshipit-source-id: bc4b6f56324eb8d3052c023fd3f6f64f99b1d4e0
Summary: This uses `clang-tidy` to comment out unused parameters (in functions, methods and lambdas) in fbcode. Cases that the tool failed to handle are fixed manually.
Reviewed By: igorsugak
Differential Revision: D5454343
fbshipit-source-id: 5dee339b4334e25e963891b519a5aa81fbf627b2
Summary:
Update eden to log via the new folly logging APIs rather than with glog.
This adds a new --logging flag that takes a logging configuration string.
By default we set the log level to INFO for all eden logs, and WARNING for
everything else. (I suspect we may eventually want to run with some
high-priority debug logs enabled for some or all of eden, but this seems like a
reasonable default to start with.)
Reviewed By: wez
Differential Revision: D5290783
fbshipit-source-id: 14183489c48c96613e2aca0f513bfa82fd9798c7
Summary:
This change makes it so that all of the C++ code related to the edenfs daemon
is now contained in the eden/fs subdirectory.
Reviewed By: bolinfest, wez
Differential Revision: D4889053
fbshipit-source-id: d0bd4774cc0bdb5d1d6b6f47d716ecae52391f37
