Summary:
When unmount and checkout are both in progress at the same time, the
FuseChannel may be unmounted prior to the checkout operation completing in that
case, the invalidation thread would be terminated prior to checkout completing.
This would also mean that checkout wouldn't be able to complete due to waiting
on the invalidation thread to flush all the invalidations, but since that
thread is gone, the flush would never complete.
This doesn't affect NFS due to the NFS code relying on a folly Executor and not
a separate thread.
Reviewed By: chadaustin
Differential Revision: D34905149
fbshipit-source-id: e0358f28e6a38dd2b0a14d9a6b5071389ad68b62
Summary:
Refactor the existing notification class so it uses the new Notifier interface. The notifier interface will be used to set up notifications/notification trays on different platforms.
The notifier interface is not fully completed yet. More methods will be added in a future diff.
As of now, we only have 1 Notifier, the command notifier. I hope to remove this notifier once we implement notifiers for all platforms that we support.
Reviewed By: chadaustin
Differential Revision: D34291809
fbshipit-source-id: a2a67af2683f1f88781428e8d88191f49e100e96
Summary:
When ran in the destructor, the class inheriting from RequestContext would have
been destroyed, and thus the vtable for the RequestContext would no longer from
the child class. In practice that meant that the PID of all requests was always
a nullopt, making `eden minitop` and others completely useless.
To solve this, we can simply create a shared_ptr with a custom deleter that
calls finishRequest first before calling the destructor.
Reviewed By: genevievehelsel
Differential Revision: D34160296
fbshipit-source-id: 7f6d9bfa36bc3db256cfe0f61478c22a44028001
Summary:
Fix a bunch of issues that a blocking migration to fmt 8.x:
* Missing includes
* Use of deprecated APIs
* Ambiguous calls to unqualified `format` (can be fmt or folly)
* Unused result of `fmt::format`
* Missing path formatter
* Passing runtime format string
Reviewed By: xavierd
Differential Revision: D33767880
fbshipit-source-id: 3976c4cc0551f3fb42c05a04c29e1c9723245749
Summary:
In the case where the shared_ptr holding a RequestContext is copied, some stats
may still be updated after the request completes. Thus it's best to call
finishRequest in the destructor to make sure the stats are properly updated.
Reviewed By: chadaustin
Differential Revision: D33724775
fbshipit-source-id: 5e4b247d44bdbeb322f25316dc98798e551cc43b
Summary:
With Facebook having been renamed Meta Platforms, we need to change the license
headers.
Reviewed By: fanzeyi
Differential Revision: D33407812
fbshipit-source-id: b11bfbbf13a48873f0cea75f212cc7b07a68fb2e
Summary: Pass use_readdirplus config option to the priv helper. If the option is enabled, we should use the appropriate options when mounting.
Reviewed By: kmancini, xavierd
Differential Revision: D32777711
fbshipit-source-id: b4717fdb52e7a36f9ec1b5ba4158f34a855f3a9b
Summary:
static_assert sizeof doesn't show the actual size in clang or gcc, so
implement a CheckSize template.
Reviewed By: xavierd
Differential Revision: D32998926
fbshipit-source-id: 432fbf19eae5b4572e5a4908f9007b7242a3ae69
Summary:
I ran into a rough situation this morning where EdenFS was running but
all of my mounts were unmounted. The sequence of events was that,
overnight, the system attempted a graceful restart. EdenFS crashed
during the restart, and left dangling mounts in the mount table. Then
a new EdenFS process started up and attempted to remount, but failed
because the `statfs` in filesystem validation returned ENOTCONN.
I grepped for ENOTCONN in linux/fs/ and it looks like the only
filesystems where this matters are already in the valid filesystem
type list, so allow mounting over an ENOTCONN filesystem.
Reviewed By: xavierd
Differential Revision: D32891776
fbshipit-source-id: 073134326822f417d9148e79bcd5590867884e18
Summary:
The privhelper can be tricked to mount or umount in places of the FS that the
user doesn't actually have access. This can be misused to mount or unmount to
privileged paths.
When accepting mount points from EdenFS, let's verify that the user actually
have access to them first.
Reviewed By: chadaustin
Differential Revision: D31875209
fbshipit-source-id: 939bf8f519adb63ce54f91c7b0757a7486cabf51
Summary:
The privhelper has "fuseTakover" methods but there is really nothing fuse about
it. This commit just renames the "fuseTakover" things to "takover" things to
sound generic. We will use these for NFS mounts in a later commit.
Reviewed By: xavierd
Differential Revision: D31802785
fbshipit-source-id: 77263d5e4a55541db1aa9d295b5ad93912d8488b
Summary:
Takeover data needs to include fuse data for fuse
mounts and NFS data for NFS mounts.
Fuse related data is already wrapped up in
FuseChannelData. we are making an NfsChannelData to
mirror this for NFS. These structs will be used in
TakeoverData in a later commit.
Reviewed By: xavierd
Differential Revision: D31740475
fbshipit-source-id: f64367a5881175b9e20d261e43ec7570c498f380
Summary:
To avoid any risk of SSLKEYLOGFILE being used to unexpectedly write
files as root, ensure the privhelper does not even link against
OpenSSL.
Reviewed By: fanzeyi
Differential Revision: D31944839
fbshipit-source-id: 3280657a10f4a468ddb63a874d38198bd333e01c
Summary:
folly:format is deprecated in lieu of fmt and std::format. Migrate
most of EdenFS to fmt instead.
Differential Revision: D31025948
fbshipit-source-id: 82ed674d5e255ac129995b56bc8b9731a5fbf82e
Summary:
EdenFS would never log anything when mounting via NFS, let's make it more
visible and easier to grep.
Reviewed By: chadaustin
Differential Revision: D31022158
fbshipit-source-id: 99fd3a04c90526eedf9951ac7c2bcd9e18ef8953
Summary:
As title, sampling group determines the sampling rate at which an FS event is logged. The higher the sampling group the more heavily its events are dropped, thus, more frequent events are assigned to the higher sampling groups.
I ran activity recorders on a few workflows, buck build, getdepts, and vscode editing and came up with the following assignment. Note that only a subset of events are assigned to a sampling group (so events not included will not be logged) as we just start to tune the sampling rates and these events should be good for a start.
```
Group1 (1/10)
FUSE_MKDIR
FUSE_RMDIR
FUSE_CREATE
FUSE_RENAME
Group2 (1/100)
FUSE_WRITE
FUSE_LISTXATTR
FUSE_SETATTR
Group3 (1/1000)
FUSE_GETXATTR
FUSE_GETATTR
FUSE_READ
FUSE_READDIR
Group4 (1/10000)
FUSE_LOOKUP
```
For reference, here are the counts of FS events of a cold buck build. The frequencies of other workflows are similar.
```
FUSE_LOOKUP 60.09 98733
FUSE_READ 12.80 21037
FUSE_GETXATTR 8.91 14645
FUSE_FORGET 8.01 13162
FUSE_GETATTR 5.55 9116
FUSE_READDIR 3.21 5270
FUSE_LISTXATTR 0.59 969
FUSE_READLINK 0.54 892
FUSE_STATFS 0.21 338
FUSE_WRITE 0.04 64
FUSE_CREATE 0.02 28
FUSE_RENAME 0.01 23
FUSE_SETATTR 0.01 13
FUSE_UNLINK 0.00 6
FUSE_RMDIR 0.00 1
FUSE_MKDIR 0.00 1
FUSE_MKNOD 0.00 1
```
Reviewed By: xavierd
Differential Revision: D30770533
fbshipit-source-id: 90be881ddbeba2113bbb190bdb1e300a68f500a0
Summary:
The current fb303 counters only report aggregated latency while we want to track Eden performance under different version, os, channel, and configs. So I am setting up a new logging mechanism for this purpose.
This diff introduces the class `FsEventLogger` for sampling and logging. There are 3 configs introduced by this diff. The configs are reloaded every 30 minutes.
1. `telemetry:request-sampling-config-allowlist`
A list of config keys that we want to attach to scuba events.
2. `telemetry:request-samples-per-minute`
Max number of events logged to scuba per minute per mount.
3. `telemetry:request-sampling-group-denominators`
* Each type of operation has a "sampling group" (defaulted to 0, which is dropping all).
* We use this sampling group as index to look up its denominator in this config.
* The denominator is then used for sampling. e.g. `1/x` of the events are send to scuba, if we haven't reached the cap specified by #2.
Example workflow:
1. receive tracing event
2. look up denominator of the sampling group of the operation type
3. sample based on the denominator
4. check that we have not exceeded the logging cap per min
5. create sample and send to scribe
Reviewed By: xavierd
Differential Revision: D30288054
fbshipit-source-id: 8f2b95c11c718550a8162f4d1259a25628f499ff
Summary:
* `FuseRequestContext` currently uses `fuseHeader_.opcode` to get the cause detail.
* However, `fuseHeader_.opcode` is set to 0 by `stealReq` to indicate that the request has been released.
* This lead to `getCauseDetail` returning `<unknown>` if we call it after the request is released (e.g. in the clean up future).
This adds the `opcode_` member so that `getCauseDetail` always return the correct cause detail regardless of whether the request has been released. This also removes `examineReq` as it's not used.
Reviewed By: xavierd
Differential Revision: D30050518
fbshipit-source-id: 0c3e4d31e59a5b5fd87967d6c976b573692a2af9
Summary: Extended eden doctor to check if the PrivHelper is accessible and report when it is not.
Reviewed By: genevievehelsel
Differential Revision: D29593250
fbshipit-source-id: 2390e75b91c9d6f713db4b6084868af91a0b6623
Summary:
When running tests, we need a way for Eden to find its privhelper. Normally, it
finds it by looking next to the EdenFS binary itself, but while that works in
Buck 1, it does not work in Buck 2, where the binaries are in separate
directories
Compared to the previous diffs, this requires some extra care because on a real
system, EdenFS is running as a setuid binary, and we don't want to let people
just run whatever binary they want the EdenFS privhelper!
To that end, we reject this argument entirely if the binary is setuid.
Reviewed By: xavierd
Differential Revision: D29061439
fbshipit-source-id: bf9427211d4209cf0bea805b9ea3a53270ec455b
Summary:
## This diff
* Implement `getCauseDetail` for Fuse fetch context so that we log which Fuse call triggers the fetch.
Reviewed By: chadaustin, kmancini
Differential Revision: D29092690
fbshipit-source-id: 230ef193332a90a01ca9b89f5b8d8f20cdf4f321
Summary: The new name is `LockedPtr::as_lock`.
Reviewed By: aary
Differential Revision: D28987868
fbshipit-source-id: 8abd6a69a1b9c884adf137f06c24fe0df9ddd089
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
A few final write calls need to be threaded as well.
Reviewed By: xavierd
Differential Revision: D28844011
fbshipit-source-id: 214853eea7fa65263fe6415e1ae8b76ca21512d6
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
This change is for symlink
Reviewed By: genevievehelsel
Differential Revision: D28841453
fbshipit-source-id: 080eb62f0b562f8e0995c34e9a8302238fc59ed8
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
this change is for rename
Reviewed By: genevievehelsel
Differential Revision: D23467437
fbshipit-source-id: e9d79c65fb5c4d686f0597550e43a0e87c4792cb
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally, we generally should not be using these singletons
in our production code.
This change is for mkdir
Reviewed By: genevievehelsel
Differential Revision: D23458622
fbshipit-source-id: f3914a4f692490434882143664a5d5f1701e93ba
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
This change is for create
Reviewed By: genevievehelsel
Differential Revision: D23457862
fbshipit-source-id: d4c9cc658c26b3119b2b2a1da061e299eaf510c9
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for mknod
Reviewed By: chadaustin
Differential Revision: D23452153
fbshipit-source-id: 7b9bc6b624fbe81b91770bc65a0d27bc9d397032
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for getxattr
Reviewed By: chadaustin
Differential Revision: D23451954
fbshipit-source-id: bae73878754d59661cddf7c0b001e506bbc88d13
Summary:
There are a few remaining holes where we are not passing a full fetch context.
We will need a full fetch context to do all data fetch logging needed for the
intern project. Additionally we generally should not be using these singletons
in our production code.
this change is for readlink
Reviewed By: chadaustin
Differential Revision: D23451821
fbshipit-source-id: 1f8ee369a992ab3489a9366f9a972f67461970de
Summary:
Similarly to the NFS change, this moves all the folly::Future handling in the
FuseChannel itself instead of inside the dispatcher. This should allow the
inode code to be converted to using ImmediateFuture instead of folly::Future.
Reviewed By: genevievehelsel
Differential Revision: D28372252
fbshipit-source-id: 7aae29d4a32500513c0062dfa42b2c7a3be038db
Summary:
DurhamG discovered that setting max_background in fuse_init_out allows
a larger number of concurrent FUSE requests. The documentation
indicates it's intended to affect background requests like readahead,
but empirically you can observe increased live FUSE requests with `rg
-j200` and `eden top`. Default to 1000 because EdenFS can handle a
large amount of concurrency and we want to avoid blob and tree fetches
to block FUSE IO when not necessary.
Reviewed By: xavierd
Differential Revision: D27526032
fbshipit-source-id: 0d3fa383772f719524a9be84b73fa2eb599580d7
Summary:
gtest includes some windows headers that will have conflicts with the
folly portability versions. This caused some issues in my in-memory tree
cache diffs (D27050310 (8a1a529fcc)).
We should probably generally be using the folly portable gtests so we can
avoid such issues in the future.
see here for more details: bd600cd4e8/folly/portability/GTest.h (L19)
I ran this with codemod yes to all
- convert all the includes with quotes:
`codemod -d eden/fs --extensions cpp,h '\#include\ "gtest/gtest\.h"' '#include <folly/portability/GTest.h>'`
- convert all the includes with brackets
`codemod -d eden/fs --extensions cpp,h '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
- convert the test template
`codemod -d eden/facebook --extensions template '\#include\ <gtest/gtest\.h>' '#include <folly/portability/GTest.h>'`
then used `arc lint` to clean up all the targets files
Reviewed By: genevievehelsel, xavierd
Differential Revision: D28035146
fbshipit-source-id: c3b88df5d4e7cdf4d1e51d9689987ce039f47fde
Summary:
In order to get Watchman to recognize EdenFS mounted as NFS, we previously set
the f_mntfromname to be "edenfs". Unfortunately, Apple decided that when the
NFS server is accessed via a unix socket it would overwrite that and instead
use the path to the socket.
Digging in the source code, I did find that the f_fstypename can be overwritten
by calling fsctl with the right set of arguments. Thus, let's do just that.
Reviewed By: kmancini
Differential Revision: D28270605
fbshipit-source-id: f6be4e394d814806aa03ec3e82b8bc2faf364ea7
Summary:
macOS supports NFS servers that can be reached via a unix socket as a way to
improve performance by reducing the TCP cost. To support this, let's first
allow the socket to bind to to be passed to the RpcServer, and then pass it
through to the privhelper code.
Reviewed By: kmancini
Differential Revision: D28261423
fbshipit-source-id: 78c60aac26353d1da76a67897429b964332df8b3
Summary: This is the same change as D27137328 (a9a1b73418) but for macFUSE.
Reviewed By: kmancini
Differential Revision: D28328029
fbshipit-source-id: c58e146dba2e7e3bdb320f2b5e80946e4a7b3afe
Summary: Looks like these aren't needed since these files are owned by a TARGETS file.
Reviewed By: genevievehelsel
Differential Revision: D28101197
fbshipit-source-id: d790530227641bf25e48bd96c8a95dd31f08a954
Summary: `CaseSensitivity::Sensitive` is better than a mere `true` out of nowhere.
Reviewed By: kmancini
Differential Revision: D27867180
fbshipit-source-id: 39d21d3cc3b70c78c6984c3ddbd65b53520770be
Summary:
While soft mount are nice as they allow the server (edenfs) to die and the
client applications to not end up in D state, this also force a maximum
(non-configuerable) 60s timeout for all IOs, after which application receive a
ETIMEDOUT. Thus, we need to not make the mount hard, thankfully, since the
mount is INTR, applications should not stay in D state if EdenFS dies.
Reviewed By: genevievehelsel
Differential Revision: D27808311
fbshipit-source-id: 17c30e88e5b236418064d8c309d85fdc6f1ca3e9
Summary:
MacOS does not have the device field like linux that we can use to mark edenfs
nfs mounts. But there is the `f_mntfromname` field. This field more typically
might have the path which this nfs mount is mirrored from, but it should be fine
to hyjack as the edenfs indicator field.
Reviewed By: xavierd
Differential Revision: D27717945
fbshipit-source-id: 056fb39dc3273b68d79c26487fd045f4e7f93b7b
Summary:
With fuse we report "edenfs:" as the device, let's do the same thing with nfs
so watchman can recognize edenfs nfs mounts similarly.
I think its fine to use the standard "edenfs" as the server name in the mount
call rather than the address, from looking at:
https://www.systutorials.com/docs/linux/man/8-mount.nfs/
Reviewed By: xavierd
Differential Revision: D27630764
fbshipit-source-id: 9e476c90ece90e758b98d140c6bf4067dbca3661
Summary:
On macOS, the mount syscall for NFS expects the arguments to be XDR encoded.
This set of argument roughly match with its Linux counterpart and appears to
start the mount process. It fails early on when trying to access the .hg
directory but this is probably an issue with the NFS server code, not of the
mounting code.
Reviewed By: kmancini
Differential Revision: D27306769
fbshipit-source-id: 697fadfddc4048ef56c3a23f75dd5bdbcc92af1b
Summary:
In the case where EdenFS crashes, any client performing IO to EdenFS would be
stuck in the D state waiting for the server to be back up and running. Since on
restart EdenFS will chose different ports, this makes it hard/impossible to
actually recover and will keep some process unkillable, often time requiring
rebooting a host.
To avoid this, we can make the mount "soft" which aborts the IO after some time
has elapsed and the server isn't back up. This should enable us to kill these
processes and recover more gracefully.
Reviewed By: chadaustin
Differential Revision: D27402817
fbshipit-source-id: ff81a4360900c4c94665174e3c0cf63402f1533e
Summary: On macOS, the mount argument are encoded with XDR, let's add them first before using them.
Reviewed By: genevievehelsel
Differential Revision: D27306770
fbshipit-source-id: 727824f05d3874119858af60c263267adfb3e61e
