Summary:
On a systemd-managed system, the `XDG_RUNTIME_DIR` environment variable is set on login. `systemctl` uses this variable to know how to talk to the systemd user manager. If `XDG_RUNTIME_DIR` is not set in the environment, `systemctl` (and thus `eden start`) fails with an unhelpful message:
Failed to connect to bus: No such file or directory
Improve this message by explicitly checking for the absence of `XDG_RUNTIME_DIR`.
Reviewed By: simpkins
Differential Revision: D13728111
fbshipit-source-id: a7f60fc29561acd05fbc1bf52d7968ae0e64d0c2
Summary:
The failure messages printed by 'eden start' are kinda crappy with systemd integration enabled. Add some tests for these messages so we can easily iterate on them.
In particular, test the following cases:
* The systemd user manager is no longer running
* The XDG_RUNTIME_DIR environment variable, needed to talk to systemd, is not set
* edenfs fails to start
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13723440
fbshipit-source-id: abae5c0e4a9f0bc6b8d0d606e8f5f36760aad5fa
Summary:
A bug in Pyre causes the properties of FindEXE to have an incorrect type. We currently work around this bug by silencing type errors. Unfortunately, this might silence legitimate errors too.
Instead of silencing type errors, using `typing.cast` to tell Pyre the correct type. This should expose legitimate errors if they exist.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13709138
fbshipit-source-id: 55f47f47062a35911c6bbe03ffd7b02a90a5107f
Summary:
The 'eden config' command prints output naively. This leads to confusing output when using the hg.extra_hgrc option. For example, in the following output, `mode = off` is part of hg.extra_hgrc's value, but it looks like it's a separate option in Eden's config:
```
$ eden config
[clone]
default-revision=master
[rage]
reporter=pastry --title "eden rage from $(hostname)"
[hg]
extra_hgrc=[fsmonitor]
mode = off
%include /etc/mercurial/repo-specific/eden.rc
[service]
experimental_systemd=True
```
Fix this issue by making 'eden config' output valid TOML:
```
[clone]
default-revision = "master"
[rage]
reporter = "pastry --title \"eden rage from $(hostname)\""
[hg]
extra_hgrc = "[fsmonitor]\nmode = off\n\n%include /etc/mercurial/repo-specific/eden.rc\n"
[service]
experimental_systemd = "True"
```
Reviewed By: simpkins
Differential Revision: D13661229
fbshipit-source-id: 76e4fa83ad186d04451623e3d8d87a78e4b821d8
Summary:
This is something that is not needed on linux because
the kernel module is typically already loaded (at least on the systems
on which we run).
On macos, since fuse is not part of the kernel, libfuse has some code
that loads it when needed.
This diff performs the equivalent actions for eden.
Reviewed By: simpkins
Differential Revision: D13721489
fbshipit-source-id: 627bc90681141d0e7da3d5b5e06756a36839958c
Summary:
This requires our mercurial repo to be available during
the build; I symlink it in alongside `common` in the `oss` dir,
and point it up to `scm/hg`.
This has partial support for mononoke too, but will need to add
logic to getdeps to pull down the proxygen repo and build that.
Reviewed By: simpkins
Differential Revision: D13480146
fbshipit-source-id: 54874245015af83a259f56944d2e5f87615baee7
Summary:
Note that the concept of bind mounts doesn't exist on macos, so that
portion of the server just throws.
Reviewed By: simpkins
Differential Revision: D13480147
fbshipit-source-id: 92225188c0af42574d090004490f3926d393747b
Summary:
In our linux deployments it was relatively straightforward
to import the mercurial runtime from a python process running the
system python executable. Our macOS deployments are a lot more
complex because they do not use the system python and do not install
the mercurial python packages in the python path of the target
python executable.
It is simpler to move the import helper functional into a mercurial
command that we can invoke instead of our own helper program.
This diff moves the script to be a debug command and adjusts its
argument parsing to match the mercurial dispatcher requirements.
There are some stylistic mismatches between this code and the
rest of mercurial; I'm suggesting that we ignore those as the
medium term solution is that this command is replaced by eden
directly consuming the rust config parsing code and by native
rust code to perform the data fetching that we need.
Reviewed By: pkaush
Differential Revision: D13522225
fbshipit-source-id: 28d751c5de4228491924df4df88ab382cfbf146a
Summary:
This updates the logic in EdenServer to add the EdenMount to the mountPoints_
map as soon as it is created, so that we track mount points as they are
initializing.
I don't expect this change to have any major impact in functionality yet. In
a subsequent diff I also plan have EdenServer keep mount points in the
mountPoints_ map longer while they are shutting down. I expect that change to
matter a bit more, as that will allow us to do a better job reporting and
debugging when mount points are taking a non-trivial amount of time to become
unreferenced and fully shut down.
Reviewed By: strager
Differential Revision: D13503050
fbshipit-source-id: 2e0e8dfde64c6a005efd6dcf503ad7577f314356
Summary:
There are some features of folly futures that are
currently being deprecated. Until that codemod lands, deprecation
warnings have been disabled in the buck build. To avoid
swamping the build output in the oss build, let's also turn
them off for cmake.
Reviewed By: strager
Differential Revision: D13686585
fbshipit-source-id: 14609a882bc78b7b31beb7ae02d762b9318e1312
Summary:
This is really a continuation of D13479516; the issue is that
the osxfuse kernel module is very eager to recycle `unique` request
id values, recycling them before our code has had a chance to update
internal state.
This diff re-keys the requests map so that we generate our own sequence
of identifiers to use as the key rather than the fuse protocol `unique`
value.
Because we cannot reliably track by `unique` value we also cannot
reliably implement interrupt support. We've never really tested
interrupt support, and it relies on functionality in folly futures
that hasn't really been tested or proven either, so I've removed
that functionality as part of this diff.
That allows simplifying some code in RequestData and FuseChannel;
we're now able to simply tack an `.ensure` on the end of the
future chain to ensure that we remove the entry from the map
once the future is resolved, successfully or otherwise.
Reviewed By: chadaustin
Differential Revision: D13679964
fbshipit-source-id: c1081a868c4061de2a725589ec1614959a8e9316
Summary:
Migrate the code from accessing optional Thrift fields directly to a safer
`optional_field_ref` API. See https://fburl.com/safe for more details.
The output of this codemod has been reviewed in D13259011.
To preserve semantics, each unchecked access is replaced with an explicit call
to `value_unchecked()`. If you are sure that accessing a field is safe (the
field is marked as set), you can later replace `value_unchecked()` with
`value()` or dereferencing (`operator *`):
```
ThriftStruct s = ...
- auto foo = s.foo_ref().value_unchecked();
+ auto foo = *s.foo_ref(); // will throw if s.foo is unset
```
Reviewed By: chadaustin
Differential Revision: D13684410
fbshipit-source-id: 919de4ddf89e7f0463f2614baba4bfbac1c8255c
Summary: the build breaks when making clean unless we declare this dep
Reviewed By: simpkins
Differential Revision: D13679633
fbshipit-source-id: f23a533eab9e37fdeab839e4f5e1b6b312ea10b0
Summary: This just makes the debug a little easier to follow.
Reviewed By: chadaustin
Differential Revision: D13680020
fbshipit-source-id: e4045822e56ba42a831ccb0ceaa9baaba5b79a10
Summary:
the osxfuse implementation includes some additional
fields in the `fuse_attr` struct. One of those fields is
`flags`. We were not initializing this field when converting
the stat data to fuse attributes which resulted in it holding
"random" data. In debug builds this seemed to usually end up
zeroed out but in release builds it was usually a bit pattern
that caused the kernel to respond with EPERM when attempting
to access the files. I didn't capture exactly what that
bit pattern was, just that initializing the struct to zeroes
reliably fixed up the EPERM issues in the Release build.
Reviewed By: chadaustin
Differential Revision: D13680004
fbshipit-source-id: 6b2ce6c10ef8f7db4a8a50bd3f2ddcfdddc3bb45
Summary:
The `make-client.py` script assembles an executable zip file
that holds all of the deps needed to run the `eden` cli on a posix
system.
Reviewed By: simpkins
Differential Revision: D13480144
fbshipit-source-id: de8cb093427c793a40e8bf81727f879216c9b41a
Summary:
This is necessary because the folly installed cmake config
advertises that it requires glog::glog but doesn't specify how to
find it.
See also D13482288
Reviewed By: simpkins
Differential Revision: D13486767
fbshipit-source-id: 0713ae70a1863fc23a5e86c21e8f72e3ba9e4ed2
Summary: Small things I noticed while working on other stuff.
Reviewed By: strager
Differential Revision: D10055671
fbshipit-source-id: de8c3b04928567a821172e6fa7ee0e056958e1e7
Summary:
If edenfs is not running or is unhealthy, 'eden rage' does not run 'eden doctor'. This means 'eden rage' does not include helpful output such as ~/local/.eden/clients/* being on NFS, or the Linux kernel version being unsupported.
Make 'eden rage' run 'eden doctor' regardless of the health of edenfs.
Reviewed By: simpkins
Differential Revision: D13633381
fbshipit-source-id: 2439057ba7a7bbe5041991ddc4ede256e86634f3
Summary:
Address this error with clang:
```
In file included from /Users/wez/fbsource/fbcode/eden/oss/eden/fs/service/main.cpp:25:
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/privhelper/PrivHelper.h:22:1: warning: class 'Unit' was previously declared as a struct [-Wmismatched-tags]
class Unit;
^
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/Unit.h:36:8: note: previous use is here
struct Unit {
^
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/privhelper/PrivHelper.h:22:1: note: did you mean struct here?
class Unit;
^~~~~
struct
```
Reviewed By: strager
Differential Revision: D13602383
fbshipit-source-id: 6e69716498680660181ab441c3c007b074ec1d40
Summary: The create_no_such_repository_exception method is not referenced anywhere. Delete it.
Reviewed By: chadaustin
Differential Revision: D13646638
fbshipit-source-id: 631c9230e6242bd858c30eec56e000ce99fbf2d8
Summary:
If a file in the overlay is truncated unexpectedly, reads on it would
fail with EIO and a log message of "std::out_of_range: string
underflow" which wasn't helpful. Instead, use the same truncated file
detection as directories use.
Reviewed By: strager
Differential Revision: D13627565
fbshipit-source-id: 246f2659ba139e8f7adb7d556719e5ead9d84ebd
Summary: If .hg/dirstate is incomplete or is otherwise unparsable, 'eden doctor' crashes. Turn the crash into a reported problem so 'eden doctor' can run its other checks.
Reviewed By: chadaustin
Differential Revision: D13630116
fbshipit-source-id: f3b27dc4e68b80da71912934c537d23eed88489f
Summary: If .hg/sharedpath is inaccessible for some reason (e.g. EdenFS reports EIO due to overlay corruption), 'eden doctor' crashes. Turn the crash into a reported problem so 'eden doctor' can run its other checks.
Reviewed By: chadaustin
Differential Revision: D13629972
fbshipit-source-id: 257ffa04358777a3bc7d0aabdde79b16c46118b6
Summary:
Add `ensureStateTransition()` and `unconditionalStateTransition()` helper
methods to EdenMount, for performing state transitions where we want to verify
the current state and where we do not care about the current state,
respectively. Update most call sites that change the state to use these
methods.
Also change the code to use acq_rel semantics for updating the state,
rather than seq_cst, which is not necessary in this case. We do not care
about having a consistent visible ordering of change to the state variable
relative to other atomic variables.
Reviewed By: strager
Differential Revision: D13579333
fbshipit-source-id: 5fd62e740a7ea2777f79f722bbde7f5b65255cb6
Summary:
Update the Eden CLI to use `os.path.realpath()` to resolve symlinks in the
Eden config directory before using it.
In most situations today it is common for the default path
(`$HOME/local/.eden`) to traverse as symlink in the user's home directory.
For users that are still using NFS home directories we can sometimes read this
symlink when running as the user, but we cannot read the symlink as root (for
instance, from inside the privhelper process).
Resolving symlinks in the CLI code ensures that the `edenfs` daemon will see
the final resolved path and will not need to traverse symlinks in the user's
home directory in this situation.
Reviewed By: strager
Differential Revision: D13515871
fbshipit-source-id: 0602389492afc0b542e089bb002534f3d714882e
Summary:
When running `eden start` in systemd mode, the code needs to make sure the
eden state directory exists before trying to write the systemd unit
configuration file. It may not exist yet if this is the first time that
`edenfs` is being started.
Reviewed By: strager
Differential Revision: D13526717
fbshipit-source-id: 2a0698ebca2ef75eddd404f60704b762ef21e61c
Summary:
Update the ServiceTestCaseBase code so that each test case keeps its state in
a single top-level temporary directory. This makes it a little easier to
figure out which directory is which when debugging a test. I also plan to
write a new test soon that creates some additional directories, and having one
top-level temporary directory avoids needing to create new TemporaryDirectory
objects.
Reviewed By: strager
Differential Revision: D13522026
fbshipit-source-id: 95a3d268d267a107bbf5e405839d64afd6afdb03
Summary:
Change some of the integration tests to read back the original command line
arguments from fake_edenfs over thrift rather than by writing it out to a file
on disk.
This shouldn't really change much, it just seemed slightly simpler.
Reviewed By: strager
Differential Revision: D13515855
fbshipit-source-id: 386207c00f28626e2125958895387a870ca87b82
Summary:
To determine the precise behavior of unmounting an Eden mount with
MNT_FORCE, I needed to see whether ENODEV was being returned from the
FUSE socket.
This helps us clarify the documentation in libfuse
https://github.com/libfuse/libfuse/issues/333
Reviewed By: strager
Differential Revision: D13630289
fbshipit-source-id: 90e6f0afc927c042a24cd6c82deac644c15ed066
Summary:
'eden start' polls for systemctl to exit, which is pretty gross. Use asyncio to avoid this explicit polling. (asyncio may still poll internally, though.)
We still poll the log file. We could use inotify in the future to avoid polling.
This refactor makes it easier to replace systemctl with direct D-Bus access in a future diff.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13526245
fbshipit-source-id: a29b1e062c489d8f2c70759e61765f8dd8df6664
Summary:
Indent all of the file entries by two spaces. Also avoid printing a blank
line when no file entries are reported (which normally occurs when the delta
only changed the commit ID and no other matching file paths).
Reviewed By: chadaustin
Differential Revision: D13627558
fbshipit-source-id: 333f5f48ded84dfaca5dce3165c033d0c1448ccb
Summary:
In the future, we will likely coalesce redundant fetches at the
BlobAccess/ObjectStore layer. To measure what we actually want,
populate a normal ObjectStore with a NullLocalStore and add counters
to FakeBackingStore.
Reviewed By: wez
Differential Revision: D13454331
fbshipit-source-id: 2fbf393d159f94e84c24ac53ccc207162fa754b7
Summary:
Update the `eden debug journal` command to report the commit IDs from the
journal entries. Also always report deltas that modify the commit.
Previously it would always skip these deltas since they do not normally report
any modified files.
Reviewed By: strager
Differential Revision: D13579205
fbshipit-source-id: 6bdbf319b571574a4b154efaa8b6cdc5a51591f9
Summary:
If --config-dir was not specified when running 'eden stop', and edenfs is unresponsive, StopCmd.run calls `check_health_using_lockfile(None)`. This violates check_health_using_lockfile's contract because its argument must be a non-null `str`. The result is that 'eden stop' fails:
```
$ eden stop
warning: edenfs is not responding: Socket read failed: timed out
Traceback (most recent call last):
File "[snip]/thrift/transport/TSocket.py", line 298, in read
buff = self.handle.recv(sz)
socket.timeout: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "[snip]/eden/cli/main.py", line 1221, in run
pid = client.getPid()
[snip]
File "[snip]/thrift/transport/TSocket.py", line 305, in read
message='Socket read failed: {}'.format(str(e))
thrift.transport.TTransport.TTransportException: Socket read failed: timed out
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
[snip]
File "[snip]/eden/cli/main.py", line 1231, in run
pid = check_health_using_lockfile(args.config_dir).pid
File "[snip]/eden/cli/util.py", line 104, in check_health_using_lockfile
lockfile = os.path.join(config_dir, LOCK_FILE)
File "[snip]/posixpath.py", line 78, in join
a = os.fspath(a)
TypeError: expected str, bytes or os.PathLike object, not NoneType
```
Make 'eden stop' use the default config directory if --config-dir was not specified.
This issue was not caught by automated tests because they specify --config-dir explicitly.
This issue was not caught by type checkers because the type of argparse.Namespace's attrs unify with any type (including `str`).
Reviewed By: simpkins
Differential Revision: D13568405
fbshipit-source-id: 395aad7e3a80b91fec05969b648a51939281ffec
Summary: 'eden clone' starts the EdenFS daemon if it's not already running. If the user opted into systemd integration, make sure the daemon is started via systemd.
Reviewed By: wez
Differential Revision: D13498650
fbshipit-source-id: 8c5da579f9b79363e2d825ea7c85d423cbcc6509
Summary:
This changes prefetch so that it loads all of the direct children of
the tree. This improves `time ls -l bigdir` performance by 2x.
Reviewed By: wez
Differential Revision: D12888690
fbshipit-source-id: eb8c8274bd9c5b5edc94d7092a5feb492fad6d66
Summary:
We think that it shouldn't really be needed to perform
the prefetch call during lookup; for file inodes it doesn't buy
us much, and it should only really help for readdir.
This removes the prefetch call from lookup, instead prefetching
upon the first readdir() of a loaded TreeInode.
Reviewed By: simpkins
Differential Revision: D12896022
fbshipit-source-id: 0209eb64bd522daf5f7461dffccd1312d32a1554
Summary: For each mounted system, checked to see if the client directory ore the relevant shared path is on a NFS filesystem and complained if so.
Reviewed By: strager
Differential Revision: D13575375
fbshipit-source-id: 0f7a7910ea690b9dfd55e35d7b5d9572d4e0d8a2
Summary:
The regex pattern needs to be built with a `bytes` object rather than `str` so
it can be used to match `bytes` data.
Reviewed By: chadaustin
Differential Revision: D13579206
fbshipit-source-id: d963a15ef6b6187f161a493f8e43920b9ebc1a9f
Summary:
Update the `FuseTest.destroyWithInitRace()` test to succeed even if the future
returned by `startFuse()` completes with an exception.
Even though the test waits to see the `FUSE_INIT` response sent back to the
kernel, there is more initialization work performed by EdenMount after the
`FUSE_RESPONSE` is sent back. This initialization code can potentially fail.
At the moment the initialization code generally succeeds even if the
`EdenMount` has already transitioned to the `SHUT_DOWN` state. However, I
plan to change the `EdenMount` code soon to error out in this case. This
currently will cause this test to fail with its existing behavior.
Reviewed By: strager
Differential Revision: D13503048
fbshipit-source-id: 6ff147d8679559f0520f5e6091291c3a07bba3ed
Summary:
Update the `listMounts()` thrift API to also report the current mount point
state. This will allow us to do a better job of reporting mount points that
are in the process of initializing or shutting down.
This change splits the `MountInfo` thrift type into two distinct types for
the `listMounts()` vs `mount()` APIs. However this change should be
completely backwards compatible at the wire protocol level for older client
and server code.
Reviewed By: strager
Differential Revision: D13503049
fbshipit-source-id: 68e7ca708b956991c8fd93bbf8973d90650aced9
Summary:
Add another suppression for T38220626 that appears to have been missed in
D13502225 when it was rebased before landing.
Reviewed By: strager
Differential Revision: D13526440
fbshipit-source-id: 60f5f6eff36b5f8462286c229836ffcb88f3afc1
Summary:
Some people encounter system-wide hangs on their Linux machines. Debugging points to a deadlock related to EdenFS' process name lookup code. Disable the process name cache during FUSE dispatch to avoid the deadlock.
Effects:
* Hopefully, the deadlock no longer happens.
* 'eden top' will no longer report process names. (It should work otherwise, though.)
Reviewed By: simpkins
Differential Revision: D13540947
fbshipit-source-id: 595c36150a5f8ff1b8e7cd81d8f61ee1463d96eb
Summary:
To mitigate a deadlock, I want to make ProcessAccessLog not access /proc/. Allow this by making ProcessNameCache optional.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13540948
fbshipit-source-id: 4c5d68c972c04122de1d2414084debfec078dd4c
Summary:
Change `eden clone` so that it no longer makes the underlying mount point
directory read-only.
The original intention of making the directory read-only was to prevent users
from accidentally removing the `README_EDEN.txt` file that tells them how to
recover if edenfs is not running. However in practice this ends up confusing
users that are legitimately trying to remove checkouts. (At the moment
`eden rm` can sometimes time out trying to unmount the checkout, in which case
it can leave this directory behind.
Reviewed By: strager
Differential Revision: D13515854
fbshipit-source-id: c2e86632c1ab8a92d54179abb58839c68a8b23c6
Summary:
I saw an integration test failure because `eden shutdown` encountered a
`ProcessLookupError`. This was thrown by the `proc_stat.read()` call.
Update this function to also expect this exception type if the process has
exited after we opened the file but before we could read it.
Reviewed By: strager
Differential Revision: D13521580
fbshipit-source-id: 027ad562024b5f05684e82cd940987eea2349077
Summary:
Update the `eden clone` command to automatically create the `.hg` directory
when creating a checkout for a Mercurial repository.
Previously this logic was performed by a separate post-clone hook that was
invoked by `eden clone`. Having this logic in a separate script made the code
slightly more complicated, and meant that configuring Eden was also more
complicated, as the hook also needed to be installed and configured. Moving
the logic into the Eden CLI will make it easier to re-use this code in
`eden doctor` if the `.hg` directory needs to be repaired.
Reviewed By: wez
Differential Revision: D13447272
fbshipit-source-id: 11c4f8e389aead151dd235eff95c860a326967af
