Summary: There are a lot of places in user visible text such as command line help where EdenFS is mentioned as eden/Eden/edenfs/EdenFS. This will make it consistent to 'EdenFS' in most cases. In the places where it is referring to the process/daemon, 'edenfs' will be used.
Reviewed By: chadaustin
Differential Revision: D29419151
fbshipit-source-id: 7b8296f0a0c84fdcb566ff811f7fcedbe7079189
Summary:
Like in many of the other cases, this needs to be told where the Eden binary
is instead of assuming it's right next to the edenfsctl binary, because on Buck
2 it's not.
Reviewed By: xavierd
Differential Revision: D29265845
fbshipit-source-id: 756bd863dc7d18eaf25a9ee209a9fd59345e6b5d
Summary:
Like it says in the title. This fails because it assumes it's next to the
EdenFS binary but that's not true in Buck v2.
Reviewed By: xavierd
Differential Revision: D29200150
fbshipit-source-id: 1be5b6c647047a50dd553031691e9bbec5f9caaf
Summary:
Similarly to the inodes and utils change, let's also split the integration
test target. This should allow Buck to run all of this concurrently while we
would run all the tests sequentially previously.
Reviewed By: fanzeyi
Differential Revision: D28524894
fbshipit-source-id: f11d30caa35d92b0b3516ebb17c5d2c16adf8446
Summary:
This applies the formatting changes from black v21.4b2 to all covered
projects in fbsource. Most changes are to single line docstrings, as black
will now remove leading and trailing whitespace to match PEP8. Any other
formatting changes are likely due to files that landed without formatting,
or files that previously triggered errors in black.
Any changes to code should be AST identical. Any test failures are likely
due to bad tests, or testing against the output of pyfmt.
Reviewed By: thatch
Differential Revision: D28204910
fbshipit-source-id: 804725bcd14f763e90c5ddff1d0418117c15809a
Summary:
on macOS we cannot safely use `fork`.
This commit replaces the use of `fork` in the startup logger subsystem.
This was a little tricky to untangle; originally (prior to any of
the `fork` removal efforts in this diff stack), the startup flow was
to spawn a set of processes via fork:
```
edenfs (setuid)
\-----edenfs (privhelper, as root)
\------edenfs (daemonized)
```
The forked children take advantage of being able to implicitly pass state to
the child processes from the parent. That data flow needs to become explicit
when removing the fork which makes some things a little awkward.
With fork removed:
* `edenfs` unconditionally spawns `edenfs_privhelper` while it has
root privs and before most of the process has been initialized.
* That same `edenfs` process will then spawn a child `edenfs`
process which starts from scratch, but that which needs to
run as the real server instance
* The original `edenfs` instance needs to linger for a while
to remain connected to the controlling tty to pass back the
startup state to the user, before terminating.
This commit deletes the check that `edenfs` is started originally
as root; previously the logic relied on the forked startup logger
continuing past the `daemonizeIfRequested` call and simply deferring
the check until after folly::init. With these changes we can't
easily perform such a check without adding some extra gymnastics
to pass the state around; the place where that is checked is in
the spawned child of the original edenfs, which is not a privileged
process and doesn't know the original euid. I don't believe this
to be a great loss as we tuck `edenfs` away under the libexec dir.
Reviewed By: chadaustin
Differential Revision: D23696569
fbshipit-source-id: 55b95daf022601a4699274d696af419f0a11f6f2
Summary:
Update EdenFS on Windows to use EdenMain.cpp, the same as on POSIX.
This reduces the amount of code divergence, and also brings several
improvements to Windows: exposing version information over fb303, support for
dynamically changing logging configuration over fb303, etc.
Reviewed By: simpkins
Differential Revision: D21332569
fbshipit-source-id: dd2da7c385e96f65fc3927511c9f84b96bec9e2b
Summary:
Add a new IntegrationTestCase base class that checks the test blacklist during
`setUp()`, and update a few remaining test classes that did not derive from
`EdenTestCase` to derive from this.
Also drop the method name argument from the `skip_test_if_blacklisted()`
function, since we can get this from the existing test case argument.
Reviewed By: genevievehelsel
Differential Revision: D21340539
fbshipit-source-id: d4fc125f119d74ab923c2cc3c9070b86c582c87e
Summary:
Remove the requirement that the `--edenfs` argument flag must be passed in
when invoking the `edenfs` daemon.
This flag was added in D12927803 to help provide a better error message if
users accidentally ran `edenfs` when they really mean to run `edenfsctl`.
However, this shouldn't really be a major problem any more: on Linux and Mac
we now install `edenfs` in a `libexec` subdirectory instead of `bin`, so it
should not be in most user's `$PATH` by default any more. Additionally we
also verify that no other unexpected arguments have been supplied. Prior to
D12927803 the `edenfs` binary would ignore unexpected arguments, making the
error messages more confusing when users tried to run it with `edenfsctl`
subcommand arguments.
One motivation for changing this now is that the Windows version of `edenfs`
does not require this flag, and it is desirable to remove this behavior
discrepancy. Rather than making this flag required on Windows it seems fine
to just drop the requirement on Mac & Linux.
Reviewed By: wez
Differential Revision: D21297159
fbshipit-source-id: e24bd694dadc036cd31dead287ae2c1184747822
Summary:
Update the tests that derive from `ServiceTestCaseBase` to honor the test
blacklist, and to also skip if we cannot run the `fake_edenfs` utility. This
latter check primarily fails when we cannot run `sudo` non-interactively
without a password prompt.
Most of the other tests were already honoring blacklist through the
`EdenTestCase` base class, but `ServiceTestCaseBase` does not derive from
`EdenTestCase`.
Reviewed By: wez
Differential Revision: D21297160
fbshipit-source-id: 5044e9939bbe487c09aa96021166c95f02fb376e
Summary:
systemd is Linux-specific, so it does not make sense to try and run these
tests on Linux and Mac. Additionally, the pystemd module currently used by
the edenfsctl CLI is not currently open source, and therefore only works in
our internal Linux builds.
This updates the integration test code to only include systemd-related tests
if we are actually in an environment where we support systemd.
Reviewed By: wez
Differential Revision: D21084095
fbshipit-source-id: fb6c9a066d9dcb741bddf3cdcf9decdb763a1d05
Summary:
Several of the tests started edenfs of fake_edenfs processes and didn't kill
them at the end of the test. This adds proper cleanup functions to try and
kill these processes when the test exits.
This also eliminates using pexpect in these tests, since it isn't actually
necessary here.
Reviewed By: wez
Differential Revision: D21084096
fbshipit-source-id: 4e92a99a5c398d4a78830ac51507ba34d7a6c0b1
Summary:
This updates the various systemd-related tests to use EdenTestCaseBase,
and allows us to delete the EnvironmentVariableMixin and
SystemdUserServiceManagerMixin classes.
The main goal of this clean-up is to make it easier to consolidate most of the
systemd-related code into just a few locations, so that we can more easily
disable the systemd-related tests in build environments where systemd is not
supported.
Reviewed By: wez
Differential Revision: D21084098
fbshipit-source-id: d5e05254c689c28751fe48d2dc38d722c7e77ed3
Summary:
Replace explicit casts working around T38947910 (Pyre not understanding
decorators) with `pyre-ignore` comments. This will hopefully help ensure that
these are tracked more visibly, and are removed when this task is fixed.
Reviewed By: wez
Differential Revision: D20434082
fbshipit-source-id: da4e4d11e4e029a11984c0efcad2aecd0d3094bf
Summary:
Fix `EdenFS.shutdown()` to call `edenfsctl stop` with a timeout of 0 seconds,
telling it not to wait for EdenFS to exit. This code then performs its own
wait with a timeout.
Previously the code called `edenfsctl stop` asking it to wait for EdenFS to
exit with a 30 second timeout. However, since the integration test could be
the immediate parent process of EdenFS the `edenfs` process may not actually
go away until the test called `wait()` on this process, which wouldn't happen
until `edenfsctl stop` returned. This only caused problems for cases where
the test could run `edenfs` directly without needing to run it through `sudo`:
when run through `sudo` the edenfs process would get cleaned up since `sudo`
was the immediate parent and it would wait on the process.
Reviewed By: genevievehelsel
Differential Revision: D20434081
fbshipit-source-id: 513fd2ebb5fc24a54c546a76e94827c81a4ab754
Summary:
Add a flag to cause `eden start` to exit successfully without doing anything
if EdenFS is already running. This flag makes it slightly easier for
automation to ensure that EdenFS is running, without logging warnings if
EdenFS was already running.
I also cleaned up the error message slightly when `eden start` is used
without this flag and fails if EdenFS was already running. Previously the
exception thrown was unhandled so it also printed a python backtrace. Now the
code throws an exception that is caught by the higher level command line code,
so it is printed in a more user-friendly way.
Reviewed By: wez
Differential Revision: D17440486
fbshipit-source-id: d7661ef7be7159bf5542b20e99a0b5495690e5a2
Summary: Add a dependency from the eden open source build to the fb303 open source build and switch EdenServiceHandler to BaseService.
Reviewed By: simpkins
Differential Revision: D15528156
fbshipit-source-id: 2ca5c31dd9fcc9bac43fd399b27f33b6f2c5ebfc
Summary:
Update the copyright & license headers in Python files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487088
fbshipit-source-id: 9f2138dff41048d2c35f15e09a04ae5a9c9c80dd
Summary:
With systemd integration enabled, if edenfs is running, or if edenfs' systemd service is active, `edenfsctl start` does nothing. This behavior differs from `edenfsctl start` with systemd integration disabled, and can cause `edenfsctl restart` to think that it successfully started edenfs.
Make `edenfsctl start` fail if edenfs is running and healthy, or if edenfs' systemd service is active (yet edenfs is unhealthy).
Reviewed By: chadaustin
Differential Revision: D15703310
fbshipit-source-id: ce0a13780ee03de1f896a938d002901023e5bdd3
Summary:
Update the `eden list` command to also report the current state for each
checkout if it is not running normally. Also added a `--json` flag to
print information as JSON so it can be consumed programmatically.
Reviewed By: strager
Differential Revision: D13503053
fbshipit-source-id: 4ef366f5bf4a1157036fdfd7ff1056079588e802
Summary:
A bug in Pyre causes the properties of FindEXE to have an incorrect type. We currently work around this bug by silencing type errors. Unfortunately, this might silence legitimate errors too.
Instead of silencing type errors, using `typing.cast` to tell Pyre the correct type. This should expose legitimate errors if they exist.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13709138
fbshipit-source-id: 55f47f47062a35911c6bbe03ffd7b02a90a5107f
Summary:
Update the Eden CLI to use `os.path.realpath()` to resolve symlinks in the
Eden config directory before using it.
In most situations today it is common for the default path
(`$HOME/local/.eden`) to traverse as symlink in the user's home directory.
For users that are still using NFS home directories we can sometimes read this
symlink when running as the user, but we cannot read the symlink as root (for
instance, from inside the privhelper process).
Resolving symlinks in the CLI code ensures that the `edenfs` daemon will see
the final resolved path and will not need to traverse symlinks in the user's
home directory in this situation.
Reviewed By: strager
Differential Revision: D13515871
fbshipit-source-id: 0602389492afc0b542e089bb002534f3d714882e
Summary:
Change some of the integration tests to read back the original command line
arguments from fake_edenfs over thrift rather than by writing it out to a file
on disk.
This shouldn't really change much, it just seemed slightly simpler.
Reviewed By: strager
Differential Revision: D13515855
fbshipit-source-id: 386207c00f28626e2125958895387a870ca87b82
Summary:
I want to use fake_edenfs to test logging via EdenFS' systemd service. Make fake_edenfs and the real edenfs use similar logic to determine the log file path.
This diff should not change behavior for the real edenfs.
Reviewed By: simpkins
Differential Revision: D13424470
fbshipit-source-id: d0c2e035fdb5884dbd2d9704c7e0244d35e052f2
Summary:
D13422460 made TemporaryDirectoryMixin required for classes deriving from ServiceTestCaseBase, but ServiceTestCaseBase expressed this requirement poorly. Make the dependency explicit by making ServiceTestCaseBase derive from TemporaryDirectoryMixin.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13454141
fbshipit-source-id: e07745cfd2a364da5011371fe8671d94d32c6798
Summary:
I noticed some Managed service tests behaving strangely on my dev server. They behave as if they are being managed by systemd.
I noticed that `systemctl --user status 'fb-edenfs@*.service'` showed a bunch of tmp-eden_test services. Since the Managed tests don't create an isolated systemd instance, they are starting and stopping services on my real systemd!
This behavior is caused by me setting service.experimental_systemd=true in my ~/.edenrc (D13371186).
Fix the odd behavior of these tests by preventing them from reading ~/.edenrc. Also do the same for /etc/eden.
Reviewed By: simpkins
Differential Revision: D13422460
fbshipit-source-id: b8a4cbabe55b75b34729d4122ba804cd7d3297a2
Summary:
The code for invoking 'eden start' is duplicated by test_daemon_command_arguments_should_forward_to_edenfs_without_leading_dashdash. I am going to add more required flags for 'eden start' tests, so the duplication is a problem. Make this test reuse spawn_start.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13453511
fbshipit-source-id: 72293ed56ca90fce1cc1fa33d996edd08ca15767
Summary:
I want to allow opting into systemd using a setting in ~/.edenrc. Since should_use_experimental_systemd_mode is a global function, it can't read any configs. Make the config file visible to should_use_experimental_systemd_mode by moving it into EdenInstance.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13370823
fbshipit-source-id: b604db66954d0a08973030daae38bf6b1433e821
Summary:
When you run 'eden start' without systemd integration, edenfs writes startup logs to the terminal to let users know that stuff is happening:
```
$ eden start
Starting edenfs (dev build), pid 2792025
Opening local RocksDB store...
Opened RocksDB store in 0.95 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2792025)
Logs available at /data/users/strager/.eden-dev/logs/edenfs.log
```
These startup logs are also used by various tests (especially 'eden restart's tests).
Make the same thing happen when running 'eden start' with systemd integration, improving the user experience and making some tests work:
```
$ EDEN_EXPERIMENTAL_SYSTEMD=1 \
./buck-out/gen/eden/cli/eden.par start \
--daemon-binary "${PWD}/buck-out/gen/eden/fs/service/edenfs"
Starting edenfs (dev build), pid 2800760
Opening local RocksDB store...
Opened RocksDB store in 0.693 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2800760)
```
Reviewed By: wez
Differential Revision: D13241979
fbshipit-source-id: de79b714e42b690fdab7c21d9add46bc2da35328
Summary: Several tests for 'eden start', 'eden stop', and 'eden status' need to pass command-line arguments to fake_edenfs. With systemd support enabled, make 'eden start' forward daemon arguments to fake_edenfs, making these tests pass.
Reviewed By: wez
Differential Revision: D13249891
fbshipit-source-id: 9008a361fce7a5629535cc9d245b86073ee70826
Summary:
edenfs's privhelper process needs the CAP_SYS_ADMIN capability [1] in order to manage mount points. Give it this capability by invoking edenfs as the `root` user using `sudo`.
Known issues:
* `sudo` must be passwordless in order for fb-edenfs@.service to start.
* systemd can't kill all of fb-edenfs@.service's processes, so `systemctl stop` is unreliable for example.
[1] https://manpage.me/index.cgi?q=capabilities&apropos=0&sektion=0&manpath=CentOS+7.1&arch=default&format=html
Reviewed By: chadaustin
Differential Revision: D13113450
fbshipit-source-id: 01b89521cab371b5017fab6fbd38d55eea599c46
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.
Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.
'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).
Reviewed By: simpkins
Differential Revision: D10849390
fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
Summary: CLI tests use shutil.rmtree to clean up temporary directories. Reuse TemporaryDirectoryMixin which is more robust against errors and supports EDEN_TEST_NO_CLEANUP.
Reviewed By: chadaustin
Differential Revision: D13268108
fbshipit-source-id: d77e95a2def0dceb34cf14e19c0c0c0e3aeef3f2
Summary:
This updates the integration tests to add type annotations to most functions
that were missing annotations.
In particular this is needed to make pyre happy, as it complains if subclasses
override methods from their parent class and do not specify type annotations
if the parent class did have annotations.
This diff also contains some minor changes to hg_extension_test_base.py to
explicitly declare some abstract methods that it uses. This was also
necessary to make pyre happy about this ocde.
Reviewed By: strager
Differential Revision: D13051097
fbshipit-source-id: 77567ed2f4d3050f93acefb52e688932d276d587
Summary:
Sometimes users accidentally run `edenfs start` or `edenfs restart` instead of
`eden start` or `eden restart`. This adds a new `--edenfs` flag to the
`edenfs` binary, and asks users if they meant to run `eden` instead if they do
not pass in this flag.
This used to be less of a problem since `edenfs` required users to also
explicitly specify several other configuration flags (like `--edenDir`).
However `edenfs` can not automatically figure out these settings, so these
flags are no longer required. Therefore `edenfs` would still try to start
normally when invoked with `edenfs restart`, since it did not require these
flags and it did not complain about unhandled command line arguments.
Reviewed By: wez
Differential Revision: D12927803
fbshipit-source-id: dbf7ce2449c391ca218652439eb68ff43c2ebd46
Summary:
When giving arguments to the edenfs program, 'eden start' only strips `--` if it is the first positional argument. The position of `--` shouldn't matter as long (as it's before any options).
In other words, given the following command invocation:
$ eden start --opt-a arg-b -- --opt-c arg-d
Current behavior:
* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b -- --opt-c arg-d` is passed to edenfs as four arguments.
Desired behavior:
* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b --opt-c arg-d` is passed to edenfs as three arguments.
* The `--` argument is not passed to edenfs.
Fix 'eden start' by stripping `--` regardless of its position.
Reviewed By: chadaustin
Differential Revision: D10863987
fbshipit-source-id: 094da3f3674e775fe3e7eb8441ec95c37c34ff05
Summary: While creating a systemd service unit for EdenFS, I noticed a few features of the current implementation of 'eden start', 'eden stop', and 'eden restart' which are not covered by tests. Write tests for these features to ensure they don't regress with systemd integration.
Reviewed By: wez
Differential Revision: D10434379
fbshipit-source-id: 6c828a85d59179bbc4beda87e1bf2534543b60b2
Summary:
Logging some messages to stdout and some messages to stderr has a few problems:
* stdout and stderr can get interleaved. For pipes, interleaving is avoidable for messages smaller than PIPE_BUF (512 bytes), but there's no guarantee that stdout and stderr are pipes or that the reader of the pipe reads PIPE_BUF bytes at a time.
* For the systemd integration I am developing, I need one file for stderr logs and another file for stdout logs. This is more complicated than having a single file for all logs.
* The distinction between stdout and stderr is unclear. No messages from StartupLogger are meant to be piped to another program.
Address these problems by only logging to stderr.
From a user's point of view, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10370931
fbshipit-source-id: b063ed3a66cefc846d30ce1486759cbd3f3ea29e
Summary:
Update the python CLI code to let the C++ edenfs code handle daemonization
rather than daemonizing in the python code.
This means that we now daemonize after the `sudo` invocation rather than
before it. This allow's sudo's wrapper process to exit once edenfs starts,
rather than remaining around for the lifetime of the edenfs daemon. This
means that the original process' stdout and stderr file descriptors are
closed properly rather than being held open by the sudo wrapper.
This also removes the logic in the CLI that waits for edenfs startup and can
potentially time out and give up.
Reviewed By: strager
Differential Revision: D8508489
fbshipit-source-id: 6a38439d5596049d6f40cb6589b6865c229f736e
