Summary:
FUSE_ACCESS was incorrectly categorized as an 'other' FUSE request
type. Categorize it as 'read'.
Reviewed By: wez
Differential Revision: D24437458
fbshipit-source-id: 5ea0a7d5a0fd57b9230f37674fd2568a4917bab5
Summary:
I noticed while playing with `eden strace` that, on my Linux machine,
running `ls -lA` would cause a pile of FUSE_GETXATTR requests for
`system.posix_acl_access` and `system.posix_acl_default`.
FUSE does not cache getxattr yet, but the kernel can cache ACLs if we
pretend to support them. Thus, advertise FUSE_POSIX_ACL support, but
continue returning ENOSYS for all setxattr calls.
Reviewed By: wez
Differential Revision: D24039130
fbshipit-source-id: a2e69c43b728fb51fb1d1b41ca9d31eba8efaa19
Summary:
Include opcode name and process name in eden.FuseCall Thrift structs
so we can use eden.FuseCall in a later diff in the stack.
Reviewed By: kmancini
Differential Revision: D24036420
fbshipit-source-id: fc6d8f3d174b85e07fac299a6f86b2b2d24f301d
Summary:
Provide an API that allows a FuseChannel TraceBus subscriber to
request population of rendered argument strings. This is not on by
default to avoid the cost in the common case that there are no
subscribers that use the field.
Reviewed By: kmancini
Differential Revision: D24036018
fbshipit-source-id: ff12252de3777606360efc75814149931637aa9b
Summary:
Provide a standard argument renderer per HandlerEntry so it can be
used outside of the strace logger.
Reviewed By: kmancini
Differential Revision: D24035882
fbshipit-source-id: f3f1d918b1a7c937777e2c5fc0de981b5ca3286c
Summary:
Instead of logging in the Dispatcher, move strace logging to
FuseChannel where it can be standardized for all FUSE request types.
Reviewed By: wez
Differential Revision: D24035838
fbshipit-source-id: c84d8c27b62f9944e2d26a35a7ed7bbbeeb5bf0e
Summary:
GNU `df` (and any other coreutil that relies on gnulib's ME_REMOTE
flag) detects remote filesystems with some heuristics. One of which is
whether the device type contains a colon. Since edenfs is a remote
filesystem, include a colon, so it's properly detected as such.
Reviewed By: genevievehelsel
Differential Revision: D24174015
fbshipit-source-id: 4b1f2f49c6eee6e690a6f570924274060a66eee7
Summary:
These don't compile on Windows, and in order to get mode/win to compile, we
need to avoid compiling their contents. Ideally, we could do that in the
TARGETS files with the select statement, but that's not available in fbcode.
Thus, we do the next best thing: ifdef the file entirely.
Reviewed By: wez
Differential Revision: D23871728
fbshipit-source-id: b4d9df6503eaa008e649afd7bdc665cd37a9585d
Summary:
on macOS we cannot safely use `fork`.
This commit replaces the use of `fork` in the startup logger subsystem.
This was a little tricky to untangle; originally (prior to any of
the `fork` removal efforts in this diff stack), the startup flow was
to spawn a set of processes via fork:
```
edenfs (setuid)
\-----edenfs (privhelper, as root)
\------edenfs (daemonized)
```
The forked children take advantage of being able to implicitly pass state to
the child processes from the parent. That data flow needs to become explicit
when removing the fork which makes some things a little awkward.
With fork removed:
* `edenfs` unconditionally spawns `edenfs_privhelper` while it has
root privs and before most of the process has been initialized.
* That same `edenfs` process will then spawn a child `edenfs`
process which starts from scratch, but that which needs to
run as the real server instance
* The original `edenfs` instance needs to linger for a while
to remain connected to the controlling tty to pass back the
startup state to the user, before terminating.
This commit deletes the check that `edenfs` is started originally
as root; previously the logic relied on the forked startup logger
continuing past the `daemonizeIfRequested` call and simply deferring
the check until after folly::init. With these changes we can't
easily perform such a check without adding some extra gymnastics
to pass the state around; the place where that is checked is in
the spawned child of the original edenfs, which is not a privileged
process and doesn't know the original euid. I don't believe this
to be a great loss as we tuck `edenfs` away under the libexec dir.
Reviewed By: chadaustin
Differential Revision: D23696569
fbshipit-source-id: 55b95daf022601a4699274d696af419f0a11f6f2
Summary:
On macOS we cannot safely use `fork` to spawn processes while other threads may initialize objc classes.
This commit replaces the use of `fork` in the privhelper startup with
`SpawnedProcess` instead. We need to take care with this as we are generally
installed setuid root and we'd like to avoid being tricked into running an
arbitrary child process as root.
This commit defines a separate executable called `edenfs_privhelper` that
contains just the privhelper server code.
We need to be careful about locating this executable; to avoid invoking an
arbitrary process while we have root privileges we require that the privhelper
be a sibling to the edenfs executable and carry out some additional ownership
verification so that we can tell that the owner of edenfs also controls
edenfs_privhelper.
To facilitate this, I've added an `executablePath` function to PathFuncs; it
returns the path to the current executable image.
To make the integration test scenario simpler, I've added the edenfs_executable
binary definition alongside that of the edenfs binary in the buck and cmake
build systems. This causes the binaries to be siblings in-situ in the build
tree and avoids the need to move things into place in the test harness.
Reviewed By: chadaustin
Differential Revision: D23653343
fbshipit-source-id: 3c2539a5e0e11cee88960db49c885ce0366d314e
Summary:
This diff teaches the CheckoutConfig how to determine
whether a given checkout should be case-sensitive (the default)
or case-insensitive-case-preserving.
This option is passed through to the fuse channel initialization,
so that the kernel will respect it, however, our DirEntry layer
doesn't yet know that it should respect this.
There's currently no UI to set this option. My game plan
is to suggest the following steps to folks that want to try
this out:
```
$ eden stop
$ vim ~/local/.eden/clients/ovrsource/config.toml
```
and then add this line to the `[repository]` section:
```
case-sensitive = false
```
and finally:
```
$ eden start
```
Reviewed By: xavierd
Differential Revision: D23751184
fbshipit-source-id: 6facb23c460cfff6e37d0091b51b97ab06f62c91
Summary:
Original commit changeset: f4816b303e19
The newer version of Watchman that understands the new mount type name
won't release in time, so back this out for now.
Reviewed By: fanzeyi
Differential Revision: D23720167
fbshipit-source-id: 588541e1d9093533611d1f32b319d2562318506a
Summary:
GNU `df` (and any other coreutil that relies on gnulib's ME_REMOTE
flag) detects remote filesystems with some heuristics. One of which is
whether the device type contains a colon. Since edenfs is a remote
filesystem, include a colon, so it's properly detected as such.
Reviewed By: genevievehelsel
Differential Revision: D23520233
fbshipit-source-id: f4816b303e198d4e2a446efdcc5b49a593e09a05
Summary:
Most of the RequestData code is platform generic, but bits of it are currently
strongly tied to FUSE. By splitting these 2 parts, we will be able to use the
RequestContext class in Windows too and not having to re-implement all the
logic there.
Reviewed By: chadaustin
Differential Revision: D23482072
fbshipit-source-id: 857fd9ca4264d0f308ec10cc487e9ff3eeb5ee16
Summary:
Removing Fuse from the enum name makes it non tied to Fuse and thus makes it
more portable. This also eliminates the last platform specific bit from
RequestData.
Reviewed By: chadaustin
Differential Revision: D23467773
fbshipit-source-id: 52515522c8ac51d0c4b56dc5e42d4b6593df6623
Summary:
This helps make RequestData slightly more generic by depending less on Fuse
specific types/code.
Reviewed By: chadaustin
Differential Revision: D23467487
fbshipit-source-id: 830f8269e2114c2968dcc49d3b5574e687191e4d
Summary:
This is unused and is sadly Fuse specific, making it harder to move to inodes/,
thus, let's remove it.
Reviewed By: chadaustin
Differential Revision: D23467486
fbshipit-source-id: c34e1abe245cfb79b9414002fd1055b8c0a5f1d3
Summary:
None of these were used, let's remove them.
ps: I thought we had a system to detect unused headers and lint about them?
Reviewed By: chadaustin
Differential Revision: D23465783
fbshipit-source-id: c21a34c9838db29f4fd0057d3be4e0fcb527cd6d
Summary:
This is not per-se fuse related, thus move it to a common location and remove
the duplicated define in FileInode.h
Reviewed By: chadaustin
Differential Revision: D23465192
fbshipit-source-id: 5fa7709f127c2d3372ee5ea3aeb89e793ea5b9f7
Summary: This file is not fuse specific, therefore, let's move it to a non-fuse folder.
Reviewed By: chadaustin
Differential Revision: D23464460
fbshipit-source-id: f70e94bb0ecc37bd74798fd230dee2058484f31b
Summary:
In the code base, "channel" is used to denote the OS mechanism that sends
EdenFS notifications. In macOS and Linux, that's Fuse, on Windows, that's
ProjectedFS. To avoid platform specific naming in ObjectedFetchContext, let's
rename the fetch cause enum.
Reviewed By: kmancini
Differential Revision: D23462460
fbshipit-source-id: 3ac68cdf4999e6a3b4ff4ee266f94e1f9736df39
Summary:
This commit introduces a new process spawning class derived
from the ChildProcess class in the watchman codebase.
`SpawnedProcess` is similar to folly::Subprocess but is designed around the
idea that we will use a system provided spawning API to start a process, rather
than assuming the use of `fork`.
`fork` is to be avoided because it can be expensive for processes with large
address spaces and also because it interacts poorly with threads on macOS. In
particular, we see the objC runtime terminating our process in some scenarios
where fork and threads are mixed.
There are some important differences from `folly::Subprocess` and that means
that some assumptions and uses need to be altered slightly from their prior
workings. For example, detaching a SpawnedProcess moves the responsibility of
waiting on the child to a periodic task as there is no way to detach via
posix_spawn without also using fork.
On the plus side, this commit allows unifying spawning between posix and
windows systems, which simplifies the code!
Reviewed By: xavierd
Differential Revision: D23287763
fbshipit-source-id: b662af1d7eaaa9ed445c42f6c5765ae9af975eea
Summary:
I'm not sure if that's the right thing to do, so I'd like feedback on that.
Check for opcode was added in D22050919 (fdb1af8bc9), but there was also a comment from
chadaustin
```
Why do you think fuseHeader isn't staying valid? It's a struct that's copied
into the RequestData. It does look like stealReq() sets the opcode to 0, but I
don't see anything that would affect the pid field.
```
So I wonder if this code can be responsible for some of the blind spots in
logging?
Reviewed By: chadaustin
Differential Revision: D23422635
fbshipit-source-id: 4d9a7171d685eb3f6f69da7b8a191df2f65ad897
Summary:
Setting up, tearing down, and querying RequestContext has some
overhead that I would like to avoid in the inner FUSE loop, so replace
RequestData with a single class that's heap-allocated at the start of
a request and is guaranteed to survive until the request ends, and is
otherwise explicitly passed where it's needed.
Reviewed By: kmancini
Differential Revision: D22712310
fbshipit-source-id: fc30d0b0f7e22b39306b857194ea07a913110b0f
Summary:
Our error handling looked pretty, but allocating all of these futures
is expensive. Each future is an allocation and some atomics. This diff
buys back some performance which I will soon spend on a new async
event queue.
Reviewed By: xavierd
Differential Revision: D22799737
fbshipit-source-id: 91dcfe974cf8f461109dfaa9dbf75c054ed84f59
Summary:
Avoid some overhead and complexity by storing BufVec as a
unique_ptr<IOBuf>. The complexity can be reintroduced if we ever find
FUSE splice support to be a performance win for us.
Reviewed By: kmancini
Differential Revision: D22710795
fbshipit-source-id: e58eedc0fb5cea9e9743ccd20d3e4e2b7cc5d198
Summary:
The tilde got dropped as part of the changes in D22672240 (be3683b1d4)
(an easy mistake to make!) and that renders this function less
useful.
Thankfully the caps display isn't a critical function; just for
some diagnostic printing.
Reviewed By: chadaustin
Differential Revision: D22847590
fbshipit-source-id: 716d7c7bd674260687fbc09e3dc94538359f98b3
Summary:
As opposed to FUSE, ProjectedFS sends notifications for file/directory creation
after the fact, and for directory that means these will be visible on disk before
EdenFS may be aware of it. While EdenFS usually process it quickly, a heavily
multi-threaded application that tries to concurrently create a directory
hierarchy may end up sending notifications to EdenFS in a somewhat out of order
fashion.
Since this should be a very rare occurence, we make this a very slow path by
being optimistic and calling `getInode` first, and then only if that fails, we
aggressively create all the parent directories. During a buck build of ~1k
jobs, this happened only 3 times.
If we fully think this through, this change doesn't fully fix the race, as a
similar race can now happen when a create and remove/rename operations are
concurrent. However, a client performing these operations concurrently is
either aware that this is racy and should handle these properly, or is most
likely buggy. Both of these should significantly reduce the likelyhod of this
happening, thus, I'm leaving this unfixed for now.
To better understand how frequently this happens, I've added a stat counter.
For now, these aren't published to ODS, but this will be tackled later.
Reviewed By: wez
Differential Revision: D22783484
fbshipit-source-id: ea3aafc2f77b65d3967f697f68114921d5909137
Summary:
To avoid mistakes when refactoring, static_assert that the template
variant of sendReply and make_iovec are only used with POD types.
Reviewed By: genevievehelsel
Differential Revision: D22710742
fbshipit-source-id: 4557761f3946fe8969ce31a42502f64cc3298e1d
Summary:
Previously, we kept a list of weak pointers to RequestData instances
to track live FUSE calls, but the only thing we need is the FUSE
request header, so directly track that instead.
Reviewed By: kmancini
Differential Revision: D22710716
fbshipit-source-id: 3820acb314ac2db85b86de128fd082bc4871d9c6
Summary:
Now that we are explicit about whether the kernel caches must be
invalidated, we can remove a use of folly::RequestContext.
Reviewed By: kmancini
Differential Revision: D22710518
fbshipit-source-id: 4bd5267bf5dd3135adf33e4f4fa1ea2649816564
Summary:
Avoid the cost of dynamically querying whether we are in a FUSE
request handler or not by passing a flag.
Reviewed By: genevievehelsel
Differential Revision: D22710397
fbshipit-source-id: 7c62f45dfc227416c91070842a349b9d0c626cba
Summary:
We're experimenting with enabling NO_OPEN support in our internal
build of the osxfuse kext. This commit includes the relevant capability bits
for the kernel interface (which are compatible with the linux FUSE
implementation) as well as adjusts our FUSE client code to detect and use
those bits on macOS.
Reviewed By: xavierd
Differential Revision: D22744378
fbshipit-source-id: 21376439a85b0b0f5a71916dd1af618d9627695e
Summary:
check fetch count before `getPriority()` is used. If fetch count has exceeded `fetchThreshold_`, lower the priority by 1.
Note: this diff only guarantees that the `getPriority()` function is returning the lowered priority. How the returned value is used for scheduling is handled by `HgQueuedBackingStore`
Reviewed By: kmancini
Differential Revision: D22550640
fbshipit-source-id: c032f8f72ca658618ac118dfb3ad3dcae61e9735
Summary: Previously `getPriority()` always returned a fixed priority. Now that we want `ObjectFetchContext` to lower its priority, it is better to make it have its own copy of `ImportPriority`.
Reviewed By: kmancini
Differential Revision: D22550587
fbshipit-source-id: 029c797def477ae4533f66cfba146a3972cfb65d
Summary:
This is much better than having `ObjectFetchContext` itself owns a copy of `ImportPriority`. We can actually customize how different fetch context manages these priority.
We set all FUSE requests to a higher priority and prefetch requests to a lower priority
Reviewed By: xavierd
Differential Revision: D22342802
fbshipit-source-id: b9c1d0f2ddbc7a5e5d619bc2c2222e5df0e702af
Summary: This commit makes `readdir()` to correct send its `RequestContext` to `EdenDispatcher` method so underlying code can know the current request being processed is from FUSE.
Reviewed By: xavierd
Differential Revision: D21821949
fbshipit-source-id: f41ba912fedbfc040e3c9267aad25e7f33f8e912
Summary:
Recently the server team added an un-used directory to test that eden would not
fetch these as a test for the upcoming repo merge. They saw that these files
were fetched a non trivial number of times. We want to know why eden is causing
these fetches.
This adds the pid and interface through which the client is interacting with eden to
ObjectFetchContext for this purpose. This information will be logged in later
changes.
note: currently this is only for fetches through Fuse (thrift interface to follow).
Reviewed By: chadaustin
Differential Revision: D22050919
fbshipit-source-id: 49b93257a0e6d910f48b1e8ec6471527e056d22a
Summary:
In following changes I will be threading ObjectFetchContext into the backing
store importing process, since this will start to be used more outside of the
ObjectStore, I am moving this class into its own files.
Reviewed By: chadaustin
Differential Revision: D22022488
fbshipit-source-id: 1a291fea6e0fd56855936962363dfc9f6de8533d
Summary: This diff adds a PID-fetchCounts map to `ObjectStore` and makes `ObjectStore` update that map after every `didFetch`
Reviewed By: kmancini
Differential Revision: D22100413
fbshipit-source-id: 740342c7b4a453fe482344c2db9542381c3772e4
Summary:
This diff makes `lookup` to use `RequestData` as `ObjectFetchContext` in `getattr` calls.
This will make sure we correctly record backing store fetches in `eden top`
Reviewed By: chadaustin
Differential Revision: D21780969
fbshipit-source-id: 468e2fadcebf4a00477bc5de434e6c658b99d1ce
Summary: This diff starts to use `RequestData` as `ObjectFetchContext` in our `read()` methods. This ensures EdenFS could track backing store fetches happened in FUSE requests.
Reviewed By: chadaustin
Differential Revision: D21792503
fbshipit-source-id: 9509a1bc8f28100a0dfe196e312c4785c7842345
Summary:
Previously we check if a request is a fuse request when we fetch anything from backing store, so we can collect number of fetches happened for each process in eden top.
This is creating a dependency from store to fuse, which is a little awkward. Instead, we could make `RequestData` a `ObjectFetchContext` and record the fetches when that happens.
Similarly in the future we should also have something equivalent in our Thrift layer.
Reviewed By: kmancini
Differential Revision: D21775919
fbshipit-source-id: 95056830ddbe7c999051c43e0d8eca9a67350904
Summary: Fix some RVO warnings and a compilation error when compiling with gcc.
Reviewed By: xavierd
Differential Revision: D21481738
fbshipit-source-id: 0621f5886df40c24ef1a6a68ccd957e38f2f4122
