Summary:
the osxfuse implementation includes some additional
fields in the `fuse_attr` struct. One of those fields is
`flags`. We were not initializing this field when converting
the stat data to fuse attributes which resulted in it holding
"random" data. In debug builds this seemed to usually end up
zeroed out but in release builds it was usually a bit pattern
that caused the kernel to respond with EPERM when attempting
to access the files. I didn't capture exactly what that
bit pattern was, just that initializing the struct to zeroes
reliably fixed up the EPERM issues in the Release build.
Reviewed By: chadaustin
Differential Revision: D13680004
fbshipit-source-id: 6b2ce6c10ef8f7db4a8a50bd3f2ddcfdddc3bb45
Summary:
The `make-client.py` script assembles an executable zip file
that holds all of the deps needed to run the `eden` cli on a posix
system.
Reviewed By: simpkins
Differential Revision: D13480144
fbshipit-source-id: de8cb093427c793a40e8bf81727f879216c9b41a
Summary:
This is necessary because the folly installed cmake config
advertises that it requires glog::glog but doesn't specify how to
find it.
See also D13482288
Reviewed By: simpkins
Differential Revision: D13486767
fbshipit-source-id: 0713ae70a1863fc23a5e86c21e8f72e3ba9e4ed2
Summary: Small things I noticed while working on other stuff.
Reviewed By: strager
Differential Revision: D10055671
fbshipit-source-id: de8c3b04928567a821172e6fa7ee0e056958e1e7
Summary:
If edenfs is not running or is unhealthy, 'eden rage' does not run 'eden doctor'. This means 'eden rage' does not include helpful output such as ~/local/.eden/clients/* being on NFS, or the Linux kernel version being unsupported.
Make 'eden rage' run 'eden doctor' regardless of the health of edenfs.
Reviewed By: simpkins
Differential Revision: D13633381
fbshipit-source-id: 2439057ba7a7bbe5041991ddc4ede256e86634f3
Summary:
Address this error with clang:
```
In file included from /Users/wez/fbsource/fbcode/eden/oss/eden/fs/service/main.cpp:25:
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/privhelper/PrivHelper.h:22:1: warning: class 'Unit' was previously declared as a struct [-Wmismatched-tags]
class Unit;
^
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/Unit.h:36:8: note: previous use is here
struct Unit {
^
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/privhelper/PrivHelper.h:22:1: note: did you mean struct here?
class Unit;
^~~~~
struct
```
Reviewed By: strager
Differential Revision: D13602383
fbshipit-source-id: 6e69716498680660181ab441c3c007b074ec1d40
Summary: The create_no_such_repository_exception method is not referenced anywhere. Delete it.
Reviewed By: chadaustin
Differential Revision: D13646638
fbshipit-source-id: 631c9230e6242bd858c30eec56e000ce99fbf2d8
Summary:
If a file in the overlay is truncated unexpectedly, reads on it would
fail with EIO and a log message of "std::out_of_range: string
underflow" which wasn't helpful. Instead, use the same truncated file
detection as directories use.
Reviewed By: strager
Differential Revision: D13627565
fbshipit-source-id: 246f2659ba139e8f7adb7d556719e5ead9d84ebd
Summary: If .hg/dirstate is incomplete or is otherwise unparsable, 'eden doctor' crashes. Turn the crash into a reported problem so 'eden doctor' can run its other checks.
Reviewed By: chadaustin
Differential Revision: D13630116
fbshipit-source-id: f3b27dc4e68b80da71912934c537d23eed88489f
Summary: If .hg/sharedpath is inaccessible for some reason (e.g. EdenFS reports EIO due to overlay corruption), 'eden doctor' crashes. Turn the crash into a reported problem so 'eden doctor' can run its other checks.
Reviewed By: chadaustin
Differential Revision: D13629972
fbshipit-source-id: 257ffa04358777a3bc7d0aabdde79b16c46118b6
Summary:
Add `ensureStateTransition()` and `unconditionalStateTransition()` helper
methods to EdenMount, for performing state transitions where we want to verify
the current state and where we do not care about the current state,
respectively. Update most call sites that change the state to use these
methods.
Also change the code to use acq_rel semantics for updating the state,
rather than seq_cst, which is not necessary in this case. We do not care
about having a consistent visible ordering of change to the state variable
relative to other atomic variables.
Reviewed By: strager
Differential Revision: D13579333
fbshipit-source-id: 5fd62e740a7ea2777f79f722bbde7f5b65255cb6
Summary:
Update the Eden CLI to use `os.path.realpath()` to resolve symlinks in the
Eden config directory before using it.
In most situations today it is common for the default path
(`$HOME/local/.eden`) to traverse as symlink in the user's home directory.
For users that are still using NFS home directories we can sometimes read this
symlink when running as the user, but we cannot read the symlink as root (for
instance, from inside the privhelper process).
Resolving symlinks in the CLI code ensures that the `edenfs` daemon will see
the final resolved path and will not need to traverse symlinks in the user's
home directory in this situation.
Reviewed By: strager
Differential Revision: D13515871
fbshipit-source-id: 0602389492afc0b542e089bb002534f3d714882e
Summary:
When running `eden start` in systemd mode, the code needs to make sure the
eden state directory exists before trying to write the systemd unit
configuration file. It may not exist yet if this is the first time that
`edenfs` is being started.
Reviewed By: strager
Differential Revision: D13526717
fbshipit-source-id: 2a0698ebca2ef75eddd404f60704b762ef21e61c
Summary:
Update the ServiceTestCaseBase code so that each test case keeps its state in
a single top-level temporary directory. This makes it a little easier to
figure out which directory is which when debugging a test. I also plan to
write a new test soon that creates some additional directories, and having one
top-level temporary directory avoids needing to create new TemporaryDirectory
objects.
Reviewed By: strager
Differential Revision: D13522026
fbshipit-source-id: 95a3d268d267a107bbf5e405839d64afd6afdb03
Summary:
Change some of the integration tests to read back the original command line
arguments from fake_edenfs over thrift rather than by writing it out to a file
on disk.
This shouldn't really change much, it just seemed slightly simpler.
Reviewed By: strager
Differential Revision: D13515855
fbshipit-source-id: 386207c00f28626e2125958895387a870ca87b82
Summary:
To determine the precise behavior of unmounting an Eden mount with
MNT_FORCE, I needed to see whether ENODEV was being returned from the
FUSE socket.
This helps us clarify the documentation in libfuse
https://github.com/libfuse/libfuse/issues/333
Reviewed By: strager
Differential Revision: D13630289
fbshipit-source-id: 90e6f0afc927c042a24cd6c82deac644c15ed066
Summary:
'eden start' polls for systemctl to exit, which is pretty gross. Use asyncio to avoid this explicit polling. (asyncio may still poll internally, though.)
We still poll the log file. We could use inotify in the future to avoid polling.
This refactor makes it easier to replace systemctl with direct D-Bus access in a future diff.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13526245
fbshipit-source-id: a29b1e062c489d8f2c70759e61765f8dd8df6664
Summary:
Indent all of the file entries by two spaces. Also avoid printing a blank
line when no file entries are reported (which normally occurs when the delta
only changed the commit ID and no other matching file paths).
Reviewed By: chadaustin
Differential Revision: D13627558
fbshipit-source-id: 333f5f48ded84dfaca5dce3165c033d0c1448ccb
Summary:
In the future, we will likely coalesce redundant fetches at the
BlobAccess/ObjectStore layer. To measure what we actually want,
populate a normal ObjectStore with a NullLocalStore and add counters
to FakeBackingStore.
Reviewed By: wez
Differential Revision: D13454331
fbshipit-source-id: 2fbf393d159f94e84c24ac53ccc207162fa754b7
Summary:
Update the `eden debug journal` command to report the commit IDs from the
journal entries. Also always report deltas that modify the commit.
Previously it would always skip these deltas since they do not normally report
any modified files.
Reviewed By: strager
Differential Revision: D13579205
fbshipit-source-id: 6bdbf319b571574a4b154efaa8b6cdc5a51591f9
Summary:
If --config-dir was not specified when running 'eden stop', and edenfs is unresponsive, StopCmd.run calls `check_health_using_lockfile(None)`. This violates check_health_using_lockfile's contract because its argument must be a non-null `str`. The result is that 'eden stop' fails:
```
$ eden stop
warning: edenfs is not responding: Socket read failed: timed out
Traceback (most recent call last):
File "[snip]/thrift/transport/TSocket.py", line 298, in read
buff = self.handle.recv(sz)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "[snip]/eden/cli/main.py", line 1221, in run
pid = client.getPid()
[snip]
File "[snip]/thrift/transport/TSocket.py", line 305, in read
message='Socket read failed: {}'.format(str(e))
thrift.transport.TTransport.TTransportException: Socket read failed: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
[snip]
File "[snip]/eden/cli/main.py", line 1231, in run
pid = check_health_using_lockfile(args.config_dir).pid
File "[snip]/eden/cli/util.py", line 104, in check_health_using_lockfile
lockfile = os.path.join(config_dir, LOCK_FILE)
File "[snip]/posixpath.py", line 78, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```
Make 'eden stop' use the default config directory if --config-dir was not specified.
This issue was not caught by automated tests because they specify --config-dir explicitly.
This issue was not caught by type checkers because the type of argparse.Namespace's attrs unify with any type (including `str`).
Reviewed By: simpkins
Differential Revision: D13568405
fbshipit-source-id: 395aad7e3a80b91fec05969b648a51939281ffec
Summary: 'eden clone' starts the EdenFS daemon if it's not already running. If the user opted into systemd integration, make sure the daemon is started via systemd.
Reviewed By: wez
Differential Revision: D13498650
fbshipit-source-id: 8c5da579f9b79363e2d825ea7c85d423cbcc6509
Summary:
This changes prefetch so that it loads all of the direct children of
the tree. This improves `time ls -l bigdir` performance by 2x.
Reviewed By: wez
Differential Revision: D12888690
fbshipit-source-id: eb8c8274bd9c5b5edc94d7092a5feb492fad6d66
Summary:
We think that it shouldn't really be needed to perform
the prefetch call during lookup; for file inodes it doesn't buy
us much, and it should only really help for readdir.
This removes the prefetch call from lookup, instead prefetching
upon the first readdir() of a loaded TreeInode.
Reviewed By: simpkins
Differential Revision: D12896022
fbshipit-source-id: 0209eb64bd522daf5f7461dffccd1312d32a1554
Summary: For each mounted system, checked to see if the client directory ore the relevant shared path is on a NFS filesystem and complained if so.
Reviewed By: strager
Differential Revision: D13575375
fbshipit-source-id: 0f7a7910ea690b9dfd55e35d7b5d9572d4e0d8a2
Summary:
The regex pattern needs to be built with a `bytes` object rather than `str` so
it can be used to match `bytes` data.
Reviewed By: chadaustin
Differential Revision: D13579206
fbshipit-source-id: d963a15ef6b6187f161a493f8e43920b9ebc1a9f
Summary:
Update the `FuseTest.destroyWithInitRace()` test to succeed even if the future
returned by `startFuse()` completes with an exception.
Even though the test waits to see the `FUSE_INIT` response sent back to the
kernel, there is more initialization work performed by EdenMount after the
`FUSE_RESPONSE` is sent back. This initialization code can potentially fail.
At the moment the initialization code generally succeeds even if the
`EdenMount` has already transitioned to the `SHUT_DOWN` state. However, I
plan to change the `EdenMount` code soon to error out in this case. This
currently will cause this test to fail with its existing behavior.
Reviewed By: strager
Differential Revision: D13503048
fbshipit-source-id: 6ff147d8679559f0520f5e6091291c3a07bba3ed
Summary:
Update the `listMounts()` thrift API to also report the current mount point
state. This will allow us to do a better job of reporting mount points that
are in the process of initializing or shutting down.
This change splits the `MountInfo` thrift type into two distinct types for
the `listMounts()` vs `mount()` APIs. However this change should be
completely backwards compatible at the wire protocol level for older client
and server code.
Reviewed By: strager
Differential Revision: D13503049
fbshipit-source-id: 68e7ca708b956991c8fd93bbf8973d90650aced9
Summary:
Add another suppression for T38220626 that appears to have been missed in
D13502225 when it was rebased before landing.
Reviewed By: strager
Differential Revision: D13526440
fbshipit-source-id: 60f5f6eff36b5f8462286c229836ffcb88f3afc1
Summary:
Some people encounter system-wide hangs on their Linux machines. Debugging points to a deadlock related to EdenFS' process name lookup code. Disable the process name cache during FUSE dispatch to avoid the deadlock.
Effects:
* Hopefully, the deadlock no longer happens.
* 'eden top' will no longer report process names. (It should work otherwise, though.)
Reviewed By: simpkins
Differential Revision: D13540947
fbshipit-source-id: 595c36150a5f8ff1b8e7cd81d8f61ee1463d96eb
Summary:
To mitigate a deadlock, I want to make ProcessAccessLog not access /proc/. Allow this by making ProcessNameCache optional.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13540948
fbshipit-source-id: 4c5d68c972c04122de1d2414084debfec078dd4c
Summary:
Change `eden clone` so that it no longer makes the underlying mount point
directory read-only.
The original intention of making the directory read-only was to prevent users
from accidentally removing the `README_EDEN.txt` file that tells them how to
recover if edenfs is not running. However in practice this ends up confusing
users that are legitimately trying to remove checkouts. (At the moment
`eden rm` can sometimes time out trying to unmount the checkout, in which case
it can leave this directory behind.
Reviewed By: strager
Differential Revision: D13515854
fbshipit-source-id: c2e86632c1ab8a92d54179abb58839c68a8b23c6
Summary:
I saw an integration test failure because `eden shutdown` encountered a
`ProcessLookupError`. This was thrown by the `proc_stat.read()` call.
Update this function to also expect this exception type if the process has
exited after we opened the file but before we could read it.
Reviewed By: strager
Differential Revision: D13521580
fbshipit-source-id: 027ad562024b5f05684e82cd940987eea2349077
Summary:
Update the `eden clone` command to automatically create the `.hg` directory
when creating a checkout for a Mercurial repository.
Previously this logic was performed by a separate post-clone hook that was
invoked by `eden clone`. Having this logic in a separate script made the code
slightly more complicated, and meant that configuring Eden was also more
complicated, as the hook also needed to be installed and configured. Moving
the logic into the Eden CLI will make it easier to re-use this code in
`eden doctor` if the `.hg` directory needs to be repaired.
Reviewed By: wez
Differential Revision: D13447272
fbshipit-source-id: 11c4f8e389aead151dd235eff95c860a326967af
Summary:
Eden can handle parallel readdir and lookup so don't require the
kernel to acquire a mutex just to serialize the requests.
5c672ab3f0
Reviewed By: strager
Differential Revision: D13386133
fbshipit-source-id: aa935af941ff2901b07b63751c97052c295f7076
Summary:
The fuse opcodes are defined as an enum so we have to use
the relatively coarse and indirect apple vs linux preprocessor
checks in the maps for the opcode names.
The osxfuse implementation branched off from the 7.19 fuse
implementation, so add a light dusting of some preprocessor
checks around enabling the performance optimization features
we desire on Linux.
We also need to relax the compile time check for the min
fuse version; I've constrained this to be apple specific,
although I suppose it wouldn't hurt to make it more broadly
applicable.
Reviewed By: chadaustin
Differential Revision: D13480145
fbshipit-source-id: 010ac114e22ea942dfcebf1105cb1f01b766f297
Summary:
There's nothing nice about this; the full set of kernel headers are
not installed with the binary distribution, and since the kernel distribution
has to be signed to be loaded on osx, there's no benefit to us building it
for ourselves.
This diff adds a nop builder and tweaks the cmake to point into the osxfuse
repo.
The osxfuse repo aggregates a couple of related repos using the git
submodule feature, so trigger that from getdeps.py too.
Reviewed By: strager
Differential Revision: D13480148
fbshipit-source-id: 84e09a86f6a83f83ffd1e3fe113dc7b15b3ea208
Summary:
a couple of parts of the rust, datapack and mononoke code
didn't have appropriate definitions.
Allow `useDatapackGetBlob_` to be default initialized per its
definition in the header file rather than replicating the relatively
complex ifdef logic in the implementation.
Reviewed By: strager
Differential Revision: D13475710
fbshipit-source-id: d2955c9b22f1186f5897aa8bdbd9046b8f6b5f7a
Summary:
handle this best-effort by setting this bit on each fd after
allocating them.
Reviewed By: strager
Differential Revision: D13475712
fbshipit-source-id: 46be80f025b21967f75822f983bc327c5e2d20af
Summary:
ScopedHTTPServer allows callers to specify a simple request handler function
instead of having to define their own handler factory. This updates the
MononokeBackingStore tests to change the `Handler` class in to a simpler
functor object, and delete the `HandlerFactory` class.
Reviewed By: wez
Differential Revision: D13476411
fbshipit-source-id: 0ede232ff9570c95e877b689272ea8eb26d97d83
Summary:
Change the code to use ScopedHTTPServer instead of its own custom logic that
starts a server and runs all of the test logic inside the `onSuccess()`
callback.
Doing all of the test logic inside `onSuccess()` is generally bad, as most of
the tests perform blocking `get()` calls on `folly::Future` objects. Blocking
work generally should not be done from inside EventBase threads.
When built with ASAN these tests occasionally crash with heap-use-after-free
errors during the HTTP server shutdown. I didn't track down the exact bad
behavior that was causing this, but attempting to stop the server from inside
the `onSuccess()` callback does seem a little fishy. Hopefully simply
switching to `ScopedHTTPServer` will fix this issue.
Reviewed By: wez
Differential Revision: D13476413
fbshipit-source-id: ab92cc16a5bf99a5e7b52529012a03786495c319
Summary:
Switch to ScopedEventBaseThread for the standalone EventBase thread create in
the unit tests. This way the test does not need its own custom logic for
starting and joining this thread. `ScopedEventBaseThread` also ensures that
the thread is actually running and ready to process events before its
constructor returns.
Reviewed By: wez
Differential Revision: D13476412
fbshipit-source-id: 260a40d93050e2e9b92ef9efd1549633679f36f7
Summary:
Update the Hash constructors that accept a `ByteRange` and a `StringPiece` to
be `constexpr` so that all Hash constructors are now `constexpr`.
This probably shouldn't really make a big difference in practice. I added
this since I wanted to define some static `Hash` constants in some tests, and
didn't want to worry about SIOF issues.
Reviewed By: chadaustin
Differential Revision: D13475781
fbshipit-source-id: fc1ce91c998f1badadbd6becd525458c25dd30de
Summary: If edenfs crashes when starting, we don't want systemd to keep trying to restart the service forever. systemd already behaves as we want, but add a test to make sure this feature doesn't regress.
Reviewed By: wez
Differential Revision: D13327803
fbshipit-source-id: df4fb0e5b2d9874fda58bad903087e411efeeefc
