Summary:
This is much better than having `ObjectFetchContext` itself owns a copy of `ImportPriority`. We can actually customize how different fetch context manages these priority.
We set all FUSE requests to a higher priority and prefetch requests to a lower priority
Reviewed By: xavierd
Differential Revision: D22342802
fbshipit-source-id: b9c1d0f2ddbc7a5e5d619bc2c2222e5df0e702af
Summary: This commit makes `readdir()` to correct send its `RequestContext` to `EdenDispatcher` method so underlying code can know the current request being processed is from FUSE.
Reviewed By: xavierd
Differential Revision: D21821949
fbshipit-source-id: f41ba912fedbfc040e3c9267aad25e7f33f8e912
Summary:
Recently the server team added an un-used directory to test that eden would not
fetch these as a test for the upcoming repo merge. They saw that these files
were fetched a non trivial number of times. We want to know why eden is causing
these fetches.
This adds the pid and interface through which the client is interacting with eden to
ObjectFetchContext for this purpose. This information will be logged in later
changes.
note: currently this is only for fetches through Fuse (thrift interface to follow).
Reviewed By: chadaustin
Differential Revision: D22050919
fbshipit-source-id: 49b93257a0e6d910f48b1e8ec6471527e056d22a
Summary:
In following changes I will be threading ObjectFetchContext into the backing
store importing process, since this will start to be used more outside of the
ObjectStore, I am moving this class into its own files.
Reviewed By: chadaustin
Differential Revision: D22022488
fbshipit-source-id: 1a291fea6e0fd56855936962363dfc9f6de8533d
Summary: This diff adds a PID-fetchCounts map to `ObjectStore` and makes `ObjectStore` update that map after every `didFetch`
Reviewed By: kmancini
Differential Revision: D22100413
fbshipit-source-id: 740342c7b4a453fe482344c2db9542381c3772e4
Summary:
This diff makes `lookup` to use `RequestData` as `ObjectFetchContext` in `getattr` calls.
This will make sure we correctly record backing store fetches in `eden top`
Reviewed By: chadaustin
Differential Revision: D21780969
fbshipit-source-id: 468e2fadcebf4a00477bc5de434e6c658b99d1ce
Summary: This diff starts to use `RequestData` as `ObjectFetchContext` in our `read()` methods. This ensures EdenFS could track backing store fetches happened in FUSE requests.
Reviewed By: chadaustin
Differential Revision: D21792503
fbshipit-source-id: 9509a1bc8f28100a0dfe196e312c4785c7842345
Summary:
Previously we check if a request is a fuse request when we fetch anything from backing store, so we can collect number of fetches happened for each process in eden top.
This is creating a dependency from store to fuse, which is a little awkward. Instead, we could make `RequestData` a `ObjectFetchContext` and record the fetches when that happens.
Similarly in the future we should also have something equivalent in our Thrift layer.
Reviewed By: kmancini
Differential Revision: D21775919
fbshipit-source-id: 95056830ddbe7c999051c43e0d8eca9a67350904
Summary: Fix some RVO warnings and a compilation error when compiling with gcc.
Reviewed By: xavierd
Differential Revision: D21481738
fbshipit-source-id: 0621f5886df40c24ef1a6a68ccd957e38f2f4122
Summary: EdenFS doesnt daemonize correctly due to the privhelper not closing fd 0 (see http://www.faqs.org/faqs/unix-faq/programmer/faq/). This redirects stdin to /dev/null/ in order to do so.
Reviewed By: xavierd
Differential Revision: D21602545
fbshipit-source-id: 0aeb589efbf214ef22c0db039fbb6a436a71e360
Summary:
In glibc, pthread cancellation support adds two atomic CAS operations
to each "cancellation point" syscall (see pthreads(7)). This includes
read() and write(). We can avoid that overhead by disabling pthread
cancellation at the start of the FUSE worker threads.
This saves two CAS operations (~40 ns) in the critical FUSE request
processing loop.
Reviewed By: simpkins
Differential Revision: D21469690
fbshipit-source-id: 7f28a2a8e831006351657981e901dc572c58cf48
Summary:
See the comment in FuseChannel.cpp, but it's currently not easy to
avoid the "security.capability" getxattr request for every
write. Since we can't avoid the request, the fastest thing we can do
is branch, strcmp, and fast-path a result on the same thread.
This appears to save three or four microseconds in the 4k random write
benchmark.
Reviewed By: wez
Differential Revision: D21341973
fbshipit-source-id: a23620767f4bdec4daf02ecfe3acb924dd57857a
Summary: These third-party includes are edenfs-specific, so move them into eden/fs/
Reviewed By: simpkins
Differential Revision: D21314642
fbshipit-source-id: c52b0a00d5080934e1f07e4cd55373602f2f6b0a
Summary:
While EdenFS does not use a separate privhelper process on Windows, it still
defines a stub PrivHelper class. However, this class was previously defined
in a separate win/utils/Stub.h header file, which led to awkward `#ifdef`s to
include the correct platform-specific header file.
This diff moves the definition of the dummy PrivHelper class in Windows into
the same `PrivHelper.h` header file used on POSIX platforms. This results in
a few more `ifdef`s in the PrivHelper files, but fewer `ifdef`s in the calling
code, and will make it easier to start unifying more of the `EdenMain` logic
on Windows and non-Windows platforms.
Reviewed By: xavierd
Differential Revision: D21332568
fbshipit-source-id: c63bf2b4a8b7e767d7db7dcda28675f735c23bf8
Summary:
The kernel has some legacy logic to attempt to clear the setuid and
setgid mode bits upon writes, but it is racy and disabled by default
in libfuse3. Now that we don't support setgid and setuid bits at all,
opt out of the legacy kernel behavior.
Reviewed By: wez
Differential Revision: D21334075
fbshipit-source-id: bdef12c1958a5e9bd2649c2bcb54975b0b4e78d6
Summary:
Instead of multiple linear array scans to determine the type of a FUSE
request, use a single lookup table. This is a small optimization, but
it makes the code a bit nicer too.
Reviewed By: wez
Differential Revision: D21314720
fbshipit-source-id: 5b6700ad5bb8e353da1e457de1533e6a626e8f68
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.
This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper. However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.
Reviewed By: wez
Differential Revision: D21296594
fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
Summary:
Tl;DR Reduces costs of fuse request mertics by reducing lock contention.
D20922194 adds tracking for FUSE request metrics, this makes tracking those
metrics more efficient. Since every user request goes through the FUSE channel,
we want to reduce the cost of these metrics as much as possible (originally
mentioned in a comment D20922194).
The synchronization used to track the metrics is costly especially
when the lock is contended.
To reduce the cost, each FuseChannel will have a thread local copy of metrics.
Each will still be synchronized to allow for reading the metrics and for
Requests moved to other threads that will need to access the metrics. However,
the lock should be contended less often since only requests from a single fuse
channel thread will access it.
Reviewed By: chadaustin
Differential Revision: D21043792
fbshipit-source-id: ce58a0cbce334095976233bfac7578d39c81bb55
Summary:
This exposes metrics for the live FUSE requests (the duration
of the longest outstanding request and the number of outstanding
requests).
Because FUSE is the interface through which the user mostly interacts
with the file system they provide good metrics to judge if the perfomance
of eden is normal, or there may be an issue.
Exposing these counters this way will send them to ods, so it will not only
allow for debuging current issues, but can be used to look back at previous
problems. This data could also be used for alerting or more proactive
remediation.
Metrics are exposed per checkout to allow seeing which checkout was
having issues. This data will aggregated in `eden top` to be used as
an overall health indicator, but should more information be needed it
will be logged in ods.
Reviewed By: chadaustin
Differential Revision: D20922194
fbshipit-source-id: 16208883417acb77b62bf712cfdd9068c5420303
Summary:
Build and run the fuse privhelper unit tests, as well as the `drop_privs`
helper program used by some of the integration tests.
Reviewed By: wez
Differential Revision: D21004425
fbshipit-source-id: 650e0729909f4753095e19fba4f01c02d516713b
Summary:
The `eden_fuse_privhelper` library depends on CoreFoundation and IOKit.
Previously this dependency was specified for the `eden_service` library, which
doesn't use these frameworks directly, only indirectly through the
`eden_fuse_privhelper` library.
The fact that `eden_fuse_privhelper` was missing this dependency causes
problems when trying to build unit tests and tools that depend on
`eden_fuse_privhelper` without using the `eden_service` library.
Reviewed By: genevievehelsel
Differential Revision: D21057925
fbshipit-source-id: b846ebde0de158b70be462067c4412a655ad8036
Summary:
D20846826 added a dependency from the fuse library to the telemetry code,
so the CMakeLists.txt file needs to be updated.
Reviewed By: genevievehelsel
Differential Revision: D21083715
fbshipit-source-id: 823cc2eb128808d0e807c6b91cacc9fd91cdad48
Summary:
This sets up the counters that will allow us to expose metrics for
live FUSE requests in Eden top.
This will allow us to track the number of live FUSE requests and the duration
of the longest FUSE request. If the duration of the longest FUSE request has
become very long, the number of FUSE requests is stuck at some value or
increasingly growing these can indicate there is a problem with the Fuse Channel
or something effecting this.
This provides more insight beyond the metrics for imports since many of the
FUSE requests will not cause imports, so this monitors more functionality.
Further because all user interaction with the filesystem goes through FUSE,
these metrics will give a good overall indication weather Eden is performing
normally.
Reviewed By: chadaustin
Differential Revision: D20846826
fbshipit-source-id: 34d834d193833a3c91d95fbb87361ddd863f64ca
Summary:
Some of our types were vulnerable to the issue described in
http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2018/p1008r0.pdf so
make all deleted default constructors explicit.
Reviewed By: simpkins
Differential Revision: D21008976
fbshipit-source-id: 5b21923f25121dabf4bb0ea55f94536fb3532e6b
Summary:
The bind-mounts configuration has been ignored by EdenFS since D17236366.
This removes all CLI code for dealing with this config section.
Reviewed By: wez
Differential Revision: D20876460
fbshipit-source-id: 6b3f3552de25ee28fc0418a6aaec14446520203c
Summary:
Plumb the read-only mount flag into FUSE on macOS. This fixes a build
warning and enables the --read-only mount flag.
Reviewed By: wez
Differential Revision: D20634119
fbshipit-source-id: a5e68cd163e36ceb6d86fd753844718ee9a5727f
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier diffs added SemiFuture and Unsafe versions. This codemod migrates
collect versions to the Unsafe versions to allow the basic collect versions to
be made safe.
Reviewed By: simpkins
Differential Revision: D20331206
fbshipit-source-id: efc8dff487d45f7d53ee55e8c4696bd3eed0e6da
Summary:
Call `folly::setThreadName()` in the privhelper process when it starts. This
changes the command name reported in `/proc/PID/comm` and in `ps`
The process name is limited to 15 bytes, so this shows up as `edenfs_privhelp`
Reviewed By: fanzeyi
Differential Revision: D20199409
fbshipit-source-id: a5349bfab9230174aaa99c87f0db73fe31659186
Summary:
This commit causes a desktop notification to be shown if we generate
EIO or ETIMEDOUT responses via fuse; the prompt is intended to make it obvious
to the user that they need to connect to the VPN.
The commit by itself doesn't show a notification, it allows configuring a
command that can be run to do something to show a notification.
The test plan includes one such configuration for our corp environment.
* It doesn't trigger for thrift-originated downloads (eg: prefetch), only for
VFS operations through FUSE.
* Ideally we'd know exactly when we have a network related error in the store
code and use that to trigger the notification. However, we have a rather
convoluted set of importers and fallbacks today, one of which is interpreting
a generic response returned from a pipe, so it is not especially clear
exactly where we should locate the logic
Reviewed By: chadaustin
Differential Revision: D17513364
fbshipit-source-id: 45134f3672679cb5580cb0c1bc12a0d6e38525ca
Summary:
1fb027d759
changed the kernel behavior to reject reads smaller than 8KB,
even for requests that would never need to be that large.
That causes eden to fail to start up on eg: Fedora 31 with a 5.4 kernel.
This commit adds some padding to satisfy this new check.
Reviewed By: chadaustin
Differential Revision: D19736893
fbshipit-source-id: 926456d72124b186976ee9a8a21242e93c26f790
Summary:
Reduce the log level for a few messages to help reduce some less important
clutter when running with an elevated debug log level.
Reviewed By: chadaustin
Differential Revision: D19188773
fbshipit-source-id: 396bb15e119fc12765ecbc707e7a1dadbdd02422
Summary: Fix some warnings that only show up in the macOS build.
Reviewed By: fanzeyi
Differential Revision: D19053236
fbshipit-source-id: 81f7187b263e0db6a57582677088519f9b97f1d7
Summary:
Two bugs conspired to cause edenfs after a graceful restart to think
the kernel supported FUSE_NO_OPENDIR_SUPPORT when it didn't: the
connection info struct wasn't zeroed, and FUSE connection capabilities
weren't properly mirrored into the Dispatcher upon graceful
restart. Fix both and add an integration test.
Reviewed By: simpkins
Differential Revision: D18903761
fbshipit-source-id: 23f4db3e240ee7d035f707820072c606a45f1138
Summary:
Now that fmt is available in Folly builds (D14813810), use it to reduce binary code size in Folly Logger. This is done by moving most of the formatting logic behind the type-erased `vformat` API. Previously it was instantiated for all combinations of formatting argument types used in calls to `FB_LOGF` and `XLOGF` in a program.
The effect of this change can be illustrated by looking at symbol sizes as given by `nm -S -td` for the following test function:
```
void test_log() {
FB_LOGF(logger, WARN, "num events: {:06d}, duration: {:6.3f}", 1234, 5.6789);
}
```
compiled in `opt` mode.
`nm` before:
```
0000000004236736 0000000000000231 T test_log()
0000000004236992 0000000000001002 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
```
`nm` after:
```
0000000004237536 0000000000000231 T test_log()
0000000004237792 0000000000000251 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
0000000004238048 0000000000000740 W folly::LogStreamProcessor::vformatLogString[abi:cxx11](folly::Range<char const*>, fmt::v5::format_args, bool&)
```
Before we had one 1002 byte instantiation of `formatLogString<int, double>`. With this change it was reduced 4x to 251 bytes and non-template function `vformatLogString` was added which is shared among all logging calls. The size of `test_log` remained unchanged. There are even bigger savings from Folly Formatter instantiations which are no longer needed, e.g.
```
0000000004238032 0000000000001363 W _ZNK5folly13BaseFormatterINS_9FormatterILb0EJRKiRKdEEELb0EJS3_S5_EEclIZNKS7_8appendToINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr12IsSomeStringIT_EE5valueEvE4typeERSH_EUlNS_5RangeIPKcEEE_EEvSK_
```
So in total this change results in ~5x per-call/instantiation binary size. It is possible to reduce binary size even further but it is not done in the current diff to keep it manageable.
In addition to binary size improvements, switching to fmt will potentially
* allow catching errors in format strings at compile time,
* simplify future migration to C++20 [`std::format`](http://eel.is/c++draft/format).
Reviewed By: simpkins
Differential Revision: D15485589
fbshipit-source-id: 06db4436839f11c2c3dbed7b36658e2193343411
Summary:
We noticed that some operations were surprisingly slow and
the hypothesis was that caching in the kext wasn't working well.
I traced into this with a simple repeated `realpath` of the same file
while running the server with debug logging turned up. That showed
that we were getting two requests from the kernel for each inode in
the path resolution for each invocation of the realpath tool.
I switched to looking at the kext code and found this section to be
a potential issue:
https://github.com/osxfuse/kext/blob/support/osxfuse-3/osxfuse/fuse_internal.h#L711
```
/* XXX: truncation; user space sends us a 64-bit tv_sec */ \
VTOFUD(vp)->attr_valid.tv_sec = (time_t)struct_name ## _get_attr_valid(fuse_out); \
VTOFUD(vp)->attr_valid.tv_nsec = struct_name ## _get_attr_valid_nsec(fuse_out); \
nanouptime(&uptsp_ ## __funct__); \
```
Switching our default TTL to be the maximum possible value that fits
in a signed 32-bit time_t cuts down on the requests from the kernel to
a single lookup for the symlink:
```
V1105 15:56:28.900839 28132386 FuseChannel.cpp:1141] fuse request opcode=1 FUSE_LOOKUP unique=2 len=47 nodeid=14910747 uid=2048904527 gid=1876110778 pid=24546
V1105 15:56:28.900885 28132386 FuseChannel.cpp:1437] FUSE_LOOKUP parent=14910747 name=README
V1105 15:56:28.901111 28143988 FuseChannel.cpp:421] sendRawReply: unique=2 header->len=160 wrote=160
```
and no requests at all for a plain file in the repo.
Reviewed By: simpkins
Differential Revision: D18340767
fbshipit-source-id: caebf051a543c54f7a03852fd2e0abab68448ded
Summary:
The Windows build spews a great many warnings. Address some of them by
enabling the unused-exception-parameter warning on Clang/Linux too.
Reviewed By: yfeldblum
Differential Revision: D18178930
fbshipit-source-id: efecb605b84d4f06c8c8411a23d17904bbdff746
Summary:
When we implemented FUSE_NO_OPEN_SUPPORT and FUSE_NO_OPENDIR_SUPPORT,
we forgot to remove the FileHandle and FileHandleBase code.
Reviewed By: pkaush
Differential Revision: D17991710
fbshipit-source-id: dfeb26d512f017cef7710929ccff1f6940cf8641
Summary: Tracing was not an accurate name for what this directory had become. So rename it to telemetry.
Reviewed By: wez
Differential Revision: D17923303
fbshipit-source-id: fca07e8447d9b9b3ea5d860809a2d377e3c4f9f2
Summary:
This diff removes the logic that consumes the legacy bind
mount list and mounts them on startup. That functionality has been
replaced with the eden redirect command.
Instead of performing the bind mounts in the server, the server will
now run `eden redirect fixup` to apply that configuration.
This diff also changes the behavior of performBindMounts: previously, if the
bind mount setup failed, we would tear down the entire repo mount. Since we're
now spawning an external process, it is much more likely that something might
fail and result in a bad experience, so we no longer bail out in that case:
we'll continue and leave the bind mounts as-is. The user can then use `eden
doctor` or `eden redirect fixup` to sort things out.
Reviewed By: simpkins
Differential Revision: D17236366
fbshipit-source-id: 8b004551a076216f0e5448942f00b5195ee18803
Summary:
Increase the log level if a FUSE request timeout occurs, so we can tell if
this has occurred in the logs when debugging user problems. Rate limit the
messages to at most once per second to avoid lots of log spam when a problem
occurs that causes lots of FUSE requests to time out.
Reviewed By: chadaustin
Differential Revision: D17506837
fbshipit-source-id: b84bd094034f1b31ef74038a9fd3828d4db9acf2
Summary:
The Catalina beta reports `10.15` and we were throwing an
exception when we didn't match three version number components.
Allow for 2 or more components to match when scanning the string.
Reviewed By: fanzeyi
Differential Revision: D17490838
fbshipit-source-id: 240513f21c0f57e629a300a35a815e7a572e93ff
Summary:
The macos fuse kext will block umount(2) until it gets
a reply to the FUSE_DESTROY opcode, causing `eden stop` to always
end up SIGKILL'ing the process, and thus causing an fsck on the next
startup.
So: let's respond to it after dispatching to the destroy method.
Reviewed By: chadaustin
Differential Revision: D17251386
fbshipit-source-id: 947d2aeb65edadbfbbac8b2673933ade283215b0
Summary:
This diff adds a fuse request timeout configuration setting
and threads it through to both the FuseChannel for our internal processing
and to the privhelper so that we can set an appropriate (slightly larger)
value for the kernel level daemon_timeout setting.
Reviewed By: chadaustin
Differential Revision: D16957552
fbshipit-source-id: a0fecc691d72914b5aebaed8a006dc0d6b0d7d12
