Summary:
This changes prefetch so that it loads all of the direct children of
the tree. This improves `time ls -l bigdir` performance by 2x.
Reviewed By: wez
Differential Revision: D12888690
fbshipit-source-id: eb8c8274bd9c5b5edc94d7092a5feb492fad6d66
Summary:
We think that it shouldn't really be needed to perform
the prefetch call during lookup; for file inodes it doesn't buy
us much, and it should only really help for readdir.
This removes the prefetch call from lookup, instead prefetching
upon the first readdir() of a loaded TreeInode.
Reviewed By: simpkins
Differential Revision: D12896022
fbshipit-source-id: 0209eb64bd522daf5f7461dffccd1312d32a1554
Summary: For each mounted system, checked to see if the client directory ore the relevant shared path is on a NFS filesystem and complained if so.
Reviewed By: strager
Differential Revision: D13575375
fbshipit-source-id: 0f7a7910ea690b9dfd55e35d7b5d9572d4e0d8a2
Summary:
The regex pattern needs to be built with a `bytes` object rather than `str` so
it can be used to match `bytes` data.
Reviewed By: chadaustin
Differential Revision: D13579206
fbshipit-source-id: d963a15ef6b6187f161a493f8e43920b9ebc1a9f
Summary:
Update the `FuseTest.destroyWithInitRace()` test to succeed even if the future
returned by `startFuse()` completes with an exception.
Even though the test waits to see the `FUSE_INIT` response sent back to the
kernel, there is more initialization work performed by EdenMount after the
`FUSE_RESPONSE` is sent back. This initialization code can potentially fail.
At the moment the initialization code generally succeeds even if the
`EdenMount` has already transitioned to the `SHUT_DOWN` state. However, I
plan to change the `EdenMount` code soon to error out in this case. This
currently will cause this test to fail with its existing behavior.
Reviewed By: strager
Differential Revision: D13503048
fbshipit-source-id: 6ff147d8679559f0520f5e6091291c3a07bba3ed
Summary:
Update the `listMounts()` thrift API to also report the current mount point
state. This will allow us to do a better job of reporting mount points that
are in the process of initializing or shutting down.
This change splits the `MountInfo` thrift type into two distinct types for
the `listMounts()` vs `mount()` APIs. However this change should be
completely backwards compatible at the wire protocol level for older client
and server code.
Reviewed By: strager
Differential Revision: D13503049
fbshipit-source-id: 68e7ca708b956991c8fd93bbf8973d90650aced9
Summary:
Add another suppression for T38220626 that appears to have been missed in
D13502225 when it was rebased before landing.
Reviewed By: strager
Differential Revision: D13526440
fbshipit-source-id: 60f5f6eff36b5f8462286c229836ffcb88f3afc1
Summary:
Some people encounter system-wide hangs on their Linux machines. Debugging points to a deadlock related to EdenFS' process name lookup code. Disable the process name cache during FUSE dispatch to avoid the deadlock.
Effects:
* Hopefully, the deadlock no longer happens.
* 'eden top' will no longer report process names. (It should work otherwise, though.)
Reviewed By: simpkins
Differential Revision: D13540947
fbshipit-source-id: 595c36150a5f8ff1b8e7cd81d8f61ee1463d96eb
Summary:
To mitigate a deadlock, I want to make ProcessAccessLog not access /proc/. Allow this by making ProcessNameCache optional.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13540948
fbshipit-source-id: 4c5d68c972c04122de1d2414084debfec078dd4c
Summary:
Change `eden clone` so that it no longer makes the underlying mount point
directory read-only.
The original intention of making the directory read-only was to prevent users
from accidentally removing the `README_EDEN.txt` file that tells them how to
recover if edenfs is not running. However in practice this ends up confusing
users that are legitimately trying to remove checkouts. (At the moment
`eden rm` can sometimes time out trying to unmount the checkout, in which case
it can leave this directory behind.
Reviewed By: strager
Differential Revision: D13515854
fbshipit-source-id: c2e86632c1ab8a92d54179abb58839c68a8b23c6
Summary:
I saw an integration test failure because `eden shutdown` encountered a
`ProcessLookupError`. This was thrown by the `proc_stat.read()` call.
Update this function to also expect this exception type if the process has
exited after we opened the file but before we could read it.
Reviewed By: strager
Differential Revision: D13521580
fbshipit-source-id: 027ad562024b5f05684e82cd940987eea2349077
Summary:
Update the `eden clone` command to automatically create the `.hg` directory
when creating a checkout for a Mercurial repository.
Previously this logic was performed by a separate post-clone hook that was
invoked by `eden clone`. Having this logic in a separate script made the code
slightly more complicated, and meant that configuring Eden was also more
complicated, as the hook also needed to be installed and configured. Moving
the logic into the Eden CLI will make it easier to re-use this code in
`eden doctor` if the `.hg` directory needs to be repaired.
Reviewed By: wez
Differential Revision: D13447272
fbshipit-source-id: 11c4f8e389aead151dd235eff95c860a326967af
Summary:
Eden can handle parallel readdir and lookup so don't require the
kernel to acquire a mutex just to serialize the requests.
5c672ab3f0
Reviewed By: strager
Differential Revision: D13386133
fbshipit-source-id: aa935af941ff2901b07b63751c97052c295f7076
Summary:
The fuse opcodes are defined as an enum so we have to use
the relatively coarse and indirect apple vs linux preprocessor
checks in the maps for the opcode names.
The osxfuse implementation branched off from the 7.19 fuse
implementation, so add a light dusting of some preprocessor
checks around enabling the performance optimization features
we desire on Linux.
We also need to relax the compile time check for the min
fuse version; I've constrained this to be apple specific,
although I suppose it wouldn't hurt to make it more broadly
applicable.
Reviewed By: chadaustin
Differential Revision: D13480145
fbshipit-source-id: 010ac114e22ea942dfcebf1105cb1f01b766f297
Summary:
There's nothing nice about this; the full set of kernel headers are
not installed with the binary distribution, and since the kernel distribution
has to be signed to be loaded on osx, there's no benefit to us building it
for ourselves.
This diff adds a nop builder and tweaks the cmake to point into the osxfuse
repo.
The osxfuse repo aggregates a couple of related repos using the git
submodule feature, so trigger that from getdeps.py too.
Reviewed By: strager
Differential Revision: D13480148
fbshipit-source-id: 84e09a86f6a83f83ffd1e3fe113dc7b15b3ea208
Summary:
a couple of parts of the rust, datapack and mononoke code
didn't have appropriate definitions.
Allow `useDatapackGetBlob_` to be default initialized per its
definition in the header file rather than replicating the relatively
complex ifdef logic in the implementation.
Reviewed By: strager
Differential Revision: D13475710
fbshipit-source-id: d2955c9b22f1186f5897aa8bdbd9046b8f6b5f7a
Summary:
handle this best-effort by setting this bit on each fd after
allocating them.
Reviewed By: strager
Differential Revision: D13475712
fbshipit-source-id: 46be80f025b21967f75822f983bc327c5e2d20af
Summary:
ScopedHTTPServer allows callers to specify a simple request handler function
instead of having to define their own handler factory. This updates the
MononokeBackingStore tests to change the `Handler` class in to a simpler
functor object, and delete the `HandlerFactory` class.
Reviewed By: wez
Differential Revision: D13476411
fbshipit-source-id: 0ede232ff9570c95e877b689272ea8eb26d97d83
Summary:
Change the code to use ScopedHTTPServer instead of its own custom logic that
starts a server and runs all of the test logic inside the `onSuccess()`
callback.
Doing all of the test logic inside `onSuccess()` is generally bad, as most of
the tests perform blocking `get()` calls on `folly::Future` objects. Blocking
work generally should not be done from inside EventBase threads.
When built with ASAN these tests occasionally crash with heap-use-after-free
errors during the HTTP server shutdown. I didn't track down the exact bad
behavior that was causing this, but attempting to stop the server from inside
the `onSuccess()` callback does seem a little fishy. Hopefully simply
switching to `ScopedHTTPServer` will fix this issue.
Reviewed By: wez
Differential Revision: D13476413
fbshipit-source-id: ab92cc16a5bf99a5e7b52529012a03786495c319
Summary:
Switch to ScopedEventBaseThread for the standalone EventBase thread create in
the unit tests. This way the test does not need its own custom logic for
starting and joining this thread. `ScopedEventBaseThread` also ensures that
the thread is actually running and ready to process events before its
constructor returns.
Reviewed By: wez
Differential Revision: D13476412
fbshipit-source-id: 260a40d93050e2e9b92ef9efd1549633679f36f7
Summary:
Update the Hash constructors that accept a `ByteRange` and a `StringPiece` to
be `constexpr` so that all Hash constructors are now `constexpr`.
This probably shouldn't really make a big difference in practice. I added
this since I wanted to define some static `Hash` constants in some tests, and
didn't want to worry about SIOF issues.
Reviewed By: chadaustin
Differential Revision: D13475781
fbshipit-source-id: fc1ce91c998f1badadbd6becd525458c25dd30de
Summary: If edenfs crashes when starting, we don't want systemd to keep trying to restart the service forever. systemd already behaves as we want, but add a test to make sure this feature doesn't regress.
Reviewed By: wez
Differential Revision: D13327803
fbshipit-source-id: df4fb0e5b2d9874fda58bad903087e411efeeefc
Summary:
Avoid this compilation error:
```
[ 23%] Building CXX object eden/fs/fuse/CMakeFiles/eden_fuse.dir/Dispatcher.cpp.o
In file included from /Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.cpp:10:
In file included from /Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.h:12:
In file included from /Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/Portability.h:19:
In file included from /Library/Developer/CommandLineTools/usr/include/c++/v1/cstddef:110:
/Library/Developer/CommandLineTools/usr/include/c++/v1/type_traits:3141:38: error: incomplete type 'facebook::eden::DirList' used in type trait expression
: public integral_constant<bool, __is_constructible(_Tp, _Args...)>
^
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/futures/Future.h:146:36: note: in instantiation of template class 'std::__1::is_constructible<facebook::eden::DirList>' requested here
typename std::enable_if<std::is_constructible<T, Args&&...>::value, int>::
^
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/futures/Future.h:148:12: note: while substituting prior template arguments into non-type template parameter [with Args = <>]
explicit FutureBase(in_place_t, Args&&... args);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/futures/Future.h:115:7: note: while substituting deduced template arguments into function template 'FutureBase' [with Args = <>, $1 = (no value)]
class FutureBase {
^
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.cpp:175:13: note: in instantiation of template class 'folly::Future<facebook::eden::DirList>' requested here
Dispatcher::readdir(InodeNumber, DirList&&, off_t, uint64_t) {
^
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.h:35:7: note: forward declaration of 'facebook::eden::DirList'
class DirList;
^
1 error generated.
```
Reviewed By: chadaustin
Differential Revision: D13475720
fbshipit-source-id: 2da1692010a82b73cbed71d996993cf5fc13af0e
Summary:
It appears as though osxfuse is a bit eager to re-use `header->unique`
values, as this DCHECK to ensure that we erased our request at the end of a
request is frequently triggered. It's not for 100% of requests, but I did
notice that the `header->unique` values tend to be in the range 1-6.
We don't check for collisions when we emplace requests in the dispatching
logic, and this particular check happens after we've acked the kernel, so it
doesn't seem like a safe thing to check and abort on really, so let's remove
it.
Reviewed By: chadaustin
Differential Revision: D13479516
fbshipit-source-id: a01e6c3e47b78b651f65fc3f5138168c71968030
Summary:
When run inside the systemd service (fb-edenfs@.service), edenfs' logs are written to `/var/log/messages` (on Facebook dev servers). This is undesirable, since those logs have a bunch of noise.
Make systemd-managed edenfs log to `~/local/.eden/logs/edenfs.log` instead, matching the behavior of custom-managed edenfs.
---
I considered using systemd's StandardOutput= and StandardError= directives [1], but they have limitations:
* **StandardOutput=file:%f/logs/edenfs.log**: When the `logs` directory is missing, systemd does not create it. In this case, systemd fails when it opens the log file, so systemd refuses to start the service.
* **StandardOutput=journal** [2]: journald and journalctl are broken for user services. Logging to journald only works with persistent journal storage [3][4], but Facebook uses volatile journal storage.
* **StandardOutput=syslog** [5]: rsyslog seems designed for system administrators, not users. I didn't investigate much, but I suspect it's impossible to make rsyslog write to a user-controlled path such as `~/local/.eden/logs/edenfs.log`.
* **LogsDirectory=%f/logs and StandardOutput=file:%L/edenfs.log** [6][7]: LogsDirectory= does exactly what we need, except it only supports paths relative to `/var/log` or `~/.config/log/`. `LogsDirectory=%f/logs` does not work, and systemd will ignore such a directive.
* **StandardOutput=file:%f/logs/edenfs.log and a `mkdir` service**: If we create a service which just creates the `logs` directory, and make fb-edenfs@.service depend upon that service, systemd can successfully open the log file [8]. In theory, using StandardOutput= would cause errors like "could not set resource limits" to be logged to `edenfs.log`. In practice, systemd does not respect the service's logging configuration when reporting such errors [9]. Therefore, this solution is no better than the manual redirect.
[1] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardOutput=
[2] https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html#
[3] https://www.freedesktop.org/software/systemd/man/journald.conf.html#SplitMode=
[4] https://lists.freedesktop.org/archives/systemd-devel/2016-October/037554.html
[5] https://www.rsyslog.com/
[6] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=
[7] https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers
[8]
```name=fb-edenfs-logs@.service
[Service]
Environment=EDENFS_CONFIG_DIR=%f
ExecStart=/bin/sh -c ' \
set -e; \
set -u; \
\
/bin/mkdir -p -- "$${EDENFS_CONFIG_DIR}/logs""; \
'
```
```name=fb-edenfs@.service
[Unit]
After=fb-edenfs-logs@%i.service
Requires=fb-edenfs-logs@%i.service
```
[9] fd0ec39d38/src/basic/log.c (L560-L639)
Reviewed By: simpkins
Differential Revision: D13422459
fbshipit-source-id: 57c575a6f377812caa2a79168778576c6ccff33e
Summary:
ported forward from D4209167, add a couple of helpers
to access these fields on mac and linux, centralizing/minimizing ifdefs.
Simplify some of the logic in FileChangeMonitor.
Reviewed By: chadaustin
Differential Revision: D13475717
fbshipit-source-id: d7b39999808bc41a6dc17a87189501cb34e68b30
Summary:
Thankfully, we can simply remove it; it is really just a performance
optimization that we can enable for linux.
Reviewed By: simpkins
Differential Revision: D13475719
fbshipit-source-id: 09b60dcf995c2c5390b91aa316c62ca1b4d3f944
Summary:
It is only 16 bits wide on this system, so we don't need
to borrow bits and there may not be enough bits to borrow from
anyway.
Reviewed By: simpkins
Differential Revision: D13475714
fbshipit-source-id: 1d342c89a3037abd744d97fef21ad421b5e60356
Summary: only include and use it on linux
Reviewed By: simpkins
Differential Revision: D13475715
fbshipit-source-id: 6b0b9da1b32088e01cbb932f9b3ed62532dfe00f
Summary: only add the selinux deps if we found selinux
Reviewed By: simpkins
Differential Revision: D13475711
fbshipit-source-id: c3375282b61881317f9a6c4c8e321ce717d1f9ab
Summary: overlooked because there is no CI exercising this today
Reviewed By: simpkins
Differential Revision: D13475721
fbshipit-source-id: 3e8fe280ab73d249da374129b37d32cd7e17f472
Summary:
this causes getdeps.py to successfully finish building the deps for
eden on a mac.
I'll follow-up to try to get eden building on top of this.
Reviewed By: simpkins
Differential Revision: D4676073
fbshipit-source-id: 303aa44137b56d1884093c26f8f4836c968aed69
Summary:
on macOS the LHS is `unsigned long` and the RHS is `unsigned long long`.
Introduce a cast for consistency.
Reviewed By: chadaustin
Differential Revision: D13470036
fbshipit-source-id: f726507013e4ed9f123ced474299bb2d6818732f
Summary: Even in C++17 mode, this isn't available, so work without it
Reviewed By: chadaustin
Differential Revision: D13470034
fbshipit-source-id: c917cc011aaabc2cfcf79e801bb6870482302fd8
Summary: this is required for the build on macos
Reviewed By: chadaustin
Differential Revision: D13470035
fbshipit-source-id: 066cb5b2ea86ffddb9c8cf6f7ae50da90b62a5bc
Summary:
Change master configuration while holding off on push blocking projects so we can work through upgrade errors.
Waiting for client upgrade....
Reviewed By: dkgi
Differential Revision: D13402132
fbshipit-source-id: 588dd87f8642c119a0b9251d8060459c93a64af0
Summary:
Don't let any of the `eden doctor` tests try to actually search for rogue
edenfs processes on the current machine. We want the test behavior to be
independent of any other processes running on the current system.
Reviewed By: strager
Differential Revision: D13373059
fbshipit-source-id: 7ab26e4bba3b59b75450f223b56baa377b77319d
Summary:
Fix the `eden doctor` check that searches for rogue Eden processes to be more
conservative, and stop reporting processes if it can't confirm if they are
legitimate or not.
This check is inherently racy, and cannot do a good job of identifying rogue
processes if other edenfs processes are in the middle of starting or shutting
down. In practice this happens frequently when running the Eden tests,
especially in stress test modes where many tests are run repeatedly in
parallel.
This changes the code to only report rogue processes when it actually
identifies multiple edenfs processes trying to use the same state directory.
Reviewed By: strager
Differential Revision: D13373058
fbshipit-source-id: 0f0e9fdb59db6b9721afc4171d77b27239c4de94
