Summary: Formatting had diverged in a few places. Fix that up.
Reviewed By: fanzeyi
Differential Revision: D18123219
fbshipit-source-id: 832cdd70789642f665a029196998928a9173be81
Summary:
Instead of having accessors for every config setting in EdenConfig,
just expose the ConfigSettings directly.
Reviewed By: fanzeyi
Differential Revision: D17847805
fbshipit-source-id: 8c6c1010c010113cf859677449797ea916f2a2a5
Summary:
Force a reference to the edenfsctlPath flag, otherwise the
linker will discard it and a large number of tests will fail.
Reviewed By: simpkins
Differential Revision: D17683222
fbshipit-source-id: b7cb29e74af85b544f45a228770ad2613c8e6efc
Summary:
Integration test helpers relied on an implicit gflags include. Make
that explicit so they compile against open source gflags and glog.
Reviewed By: wez
Differential Revision: D17264335
fbshipit-source-id: e336423b71c0f15e29b0e4ad604328b7624080a8
Summary: Add a dependency from the eden open source build to the fb303 open source build and switch EdenServiceHandler to BaseService.
Reviewed By: simpkins
Differential Revision: D15528156
fbshipit-source-id: 2ca5c31dd9fcc9bac43fd399b27f33b6f2c5ebfc
Summary:
Open source fb303 will not have getPid() or getCommandLine(), so
introduce a new method for Eden's tests.
Reviewed By: fanzeyi
Differential Revision: D16292993
fbshipit-source-id: 5cdc006ec0ee15f50a3e1cebe9b46a3ea275ff78
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary:
Move some of the argument parsing and config setup code out into a new
EdenInit.h header file. This makes it possible to re-use this logic for other
standalone utilities that want to be able to find the Eden state directory and
config information.
For now I have updated the `fake_edenfs` helper tool used by the integration
tests to use this. This may also be useful for writing standalone tools that
can perform garbage collection of the LocalStore or checking of the overlay
state.
Reviewed By: chadaustin
Differential Revision: D14889616
fbshipit-source-id: b0b193a42cb2f52177d0c44592426b42e27242aa
Summary:
If TreeInode::startLoadingInode() is in progress, and EdenServer::startTakeoverShutdown() is called, edenfs can deadlock:
1. Thread A: A FUSE request calls TreeInode::readdir() -> TreeInode::prefetch() -> TreeInode::startLoadingInode() on the children TreeInode-s -> RocksDbLocalStore::getFuture().
2. Thread B: A takeover request calls EdenServer::performTakeoverShutdown() -> InodeMap::shutdown().
3. Thread C: RocksDbLocalStore::getFuture() (called in step 1) completes -> TreeInode::inodeLoadComplete(). (The inodeLoadComplete continuation was registered by TreeInode::registerInodeLoadComplete().)
4. Thread C: After TreeInode::inodeLoadComplete() returns, the TreeInode's InodePtr is destructed, dropping the reference count to 0.
5. Thread C: InodeMap::onInodeUnreferenced() -> InodeMap::shutdownComplete() -> EdenMount::shutdown() (called in step 2) completes -> EdenServer::performTakeoverShutdown().
6. Thread C: EdenServer::performTakeoverShutdown() -> localStore_.reset() -> RocksDbLocalStore::~RocksDbLocalStore().
7. Thread C: RocksDbLocalStore::~RocksDbLocalStore() signals the thread pool to exit and waits for the pool's threads to exit. Because thread C is one of the threads managed by RocksDbLocalStore's thread pool, the signal is never handled and RocksDbLocalStore::~RocksDbLocalStore() never finishes.
Fix this deadlock by executing EdenServer::shutdown()'s callback (in EdenServer::performTakeoverShutdown()) on a different thread.
Reviewed By: simpkins
Differential Revision: D14337058
fbshipit-source-id: 1d63b4e7d8f5103a2dde31e329150bf763be3db7
Summary:
Change some of the integration tests to read back the original command line
arguments from fake_edenfs over thrift rather than by writing it out to a file
on disk.
This shouldn't really change much, it just seemed slightly simpler.
Reviewed By: strager
Differential Revision: D13515855
fbshipit-source-id: 386207c00f28626e2125958895387a870ca87b82
Summary:
I want to use fake_edenfs to test logging via EdenFS' systemd service. Make fake_edenfs and the real edenfs use similar logic to determine the log file path.
This diff should not change behavior for the real edenfs.
Reviewed By: simpkins
Differential Revision: D13424470
fbshipit-source-id: d0c2e035fdb5884dbd2d9704c7e0244d35e052f2
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.
Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.
'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).
Reviewed By: simpkins
Differential Revision: D10849390
fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
Summary:
eden/benchmarks/ is a good place to write external benchmarks for FUSE
and Thrift APIs.
(Note: this ignores all push blocking failures!)
Reviewed By: wez, strager
Differential Revision: D12970347
fbshipit-source-id: 81fa35897fc6e626254eea6e1ee44a8d35052261
Summary:
As we start to build out both FUSE and Thrift benchmarks, we'll want a
standard library. Introduce a benchharness and have both the thrift
sha-1 and parallel_open_close benchmarks use it.
(Note: this ignores all push blocking failures!)
Differential Revision: D12969306
fbshipit-source-id: 89c8bbcc37d53560decffb9281af4aba20345787
Summary:
My changes to command line argument parsing in D12927803 conflicted with one
of the new tests added in D10863987.
My changes made `edenfs` and `fake_edenfs` reject any trailing non-option
arguments. The new test runs `fake_edenfs` with an extra
`--commandArgumentsLogFile` argument asking it to log all non-option arguments
to a file.
This diff updates `fake_edenfs` to allow non-option arguments when
`--commandArgumentsLogFile` is specified.
Reviewed By: strager
Differential Revision: D13014079
fbshipit-source-id: 82ec5fd758716c2b66ac0738b7aacdf884f31233
Summary:
Sometimes users accidentally run `edenfs start` or `edenfs restart` instead of
`eden start` or `eden restart`. This adds a new `--edenfs` flag to the
`edenfs` binary, and asks users if they meant to run `eden` instead if they do
not pass in this flag.
This used to be less of a problem since `edenfs` required users to also
explicitly specify several other configuration flags (like `--edenDir`).
However `edenfs` can not automatically figure out these settings, so these
flags are no longer required. Therefore `edenfs` would still try to start
normally when invoked with `edenfs restart`, since it did not require these
flags and it did not complain about unhandled command line arguments.
Reviewed By: wez
Differential Revision: D12927803
fbshipit-source-id: dbf7ce2449c391ca218652439eb68ff43c2ebd46
Summary:
When giving arguments to the edenfs program, 'eden start' only strips `--` if it is the first positional argument. The position of `--` shouldn't matter as long (as it's before any options).
In other words, given the following command invocation:
$ eden start --opt-a arg-b -- --opt-c arg-d
Current behavior:
* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b -- --opt-c arg-d` is passed to edenfs as four arguments.
Desired behavior:
* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b --opt-c arg-d` is passed to edenfs as three arguments.
* The `--` argument is not passed to edenfs.
Fix 'eden start' by stripping `--` regardless of its position.
Reviewed By: chadaustin
Differential Revision: D10863987
fbshipit-source-id: 094da3f3674e775fe3e7eb8441ec95c37c34ff05
Summary: While creating a systemd service unit for EdenFS, I noticed a few features of the current implementation of 'eden start', 'eden stop', and 'eden restart' which are not covered by tests. Write tests for these features to ensure they don't regress with systemd integration.
Reviewed By: wez
Differential Revision: D10434379
fbshipit-source-id: 6c828a85d59179bbc4beda87e1bf2534543b60b2
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Unfortunately, we can't create a new instance of systemd if a machine is not managed by systemd. However, we can trick systemd into thinking that the machine *is* managed by systemd. When we do that, the systemd instance seems to work without any issues.
Create a library which, when injected into systemd using LD_PRELOAD, makes systemd thinks the machine is managed by systemd.
The library will be used in a subsequent diff (D10286940). This diff is separate so it's easier to review.
Reviewed By: simpkins
Differential Revision: D10371831
fbshipit-source-id: b54b704dc631ebdc5091320dd6b680ca255f3ba2
Summary:
When running fake_edenfs in daemonize mode (i.e. when --foreground is not specified), the daemon process inherits stdout and stderr from the parent and never closes them. This has two consequences:
* The daemon process holds a reference to stdout/stderr even after the parent exits. This causes tests to hang if they slurp stdout/stderr after waiting for the parent process to exit. (I discovered this issue when adding tests for 'eden start' in D10434379.)
* fake_edenfs logs to stdout/stderr after 'eden start' has returned. I think this behavior is surprising, but it doesn't really hurt anyone because fake_edenfs is just a developer tool.
Promptly close stdout and stderr after startup finishes by redirecting further output to a log file. This mimics the behavior of the real edenfs.
Note: stdin is still leaked. stdin will be addressed in D10442346.
Reviewed By: chadaustin
Differential Revision: D10439299
fbshipit-source-id: a7a423aa901982f3d35e81940ffc27db3a985d76
Summary:
No one uses --logPath with fake_edenfs. Remove the dead flag, inlining its default value.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439298
fbshipit-source-id: 068e5677db1bec81a8da174b136ef9f5b54c7e34
Summary:
The daemonizeIfRequested is implemented in the main edenfs process as well as in fake_edenfs. Consolidate the almost-identical implementations.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439300
fbshipit-source-id: 6253668896de0f8393b92f4aa5ee8a8a2fcbfb31
Summary:
StartupLogger handles two independent scenarios: edenfs in daemon mode, and edenfs in foreground mode. I need to add a third for systemd support: edenfs in foreground mode but with startup logs written to a file.
Refactor the existing StartupLogger into an abstract base class with two concrete derived classes: DaemonStartupLogger and ForegroundStartupLogger. This will make adding the systemd feature easy without confusing the logic for the existing scenarios.
Aside from dropping the setsid() call for ForegroundStartupLogger, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D10228156
fbshipit-source-id: fbc2c771c6fbb675779ef05ca3b7395b70b67485
Summary:
To ease the migration from the current custom daemon setup to a systemd-managed daemon, add a way to enable the to-be-implemented systemd behavior.
* To the edenfs daemon, support the --experimentalSystemd option.
* To the Eden CLI, support the EDEN_EXPERIMENTAL_SYSTEMD environment variable. When set, invoke the edenfs daemon with --experimentalSystemd.
Aside from adding a log message, this diff does not change any behavior. This diff just sets up scaffolding for future diffs.
Reviewed By: simpkins
Differential Revision: D10248055
fbshipit-source-id: 30ca5a4bfde00ff43fd6f2a5d0282ced4f177fed
Summary:
If 'eden stop' tries to tell edenfs to cleanly shut down, and edenfs never responds, 'eden stop' just hangs. Make 'eden stop' time out CLI -> edenfs Thrift calls. After a timeout, 'eden stop' should by stop edenfs via SIGKILL.
Note: This diff does not affect 'eden restart'.
Reviewed By: chadaustin
Differential Revision: D9982372
fbshipit-source-id: 42e498fb93fd97a2ec963e97c25f4b4101c7d524
Summary: If edenfs is unresponsive, 'eden status' hangs forever. Add a timeout to turn the hang into a user-friendly error.
Reviewed By: chadaustin
Differential Revision: D10156229
fbshipit-source-id: 9186826ae6b131a193b1499c8baac616d131357f
Summary:
I want to test more conditions for 'eden stop', such as the edenfs process taking a long time to respond to the Thrift stop request. Refactor how fake_edenfs decides to handle stop requests to make it easy to add new behavior.
Aside from changing the log messages, this diff should not change behavior.
Reviewed By: wez
Differential Revision: D10019459
fbshipit-source-id: f6d5f514a67435a446d8d672bf7d21af77d76c56
Summary:
We're looking into doing some performance optimizations in Eden to the
open() and close() calls, since they are never handled by the kernel
and always pass through to the underlying user-space FUSE daemon. But
first, let's capture some baseline numbers on Eden and non-Eden
checkouts.
Reviewed By: strager
Differential Revision: D10036941
fbshipit-source-id: 64f3414a4572fd963017491db37d70e6b5ae4f24
Summary:
When we load an empty blob from the LocalStore double check with the
BackingStore to confirm that it should actually be empty.
We have seen multiple instances of files being incorrectly imported as empty.
So far this error has always been fixed by a re-import. We still haven't
tracked down the root cause, but this change should help workaround the issue
by ensuring that we double check the file contents before returning the data.
Reviewed By: chadaustin
Differential Revision: D9476522
fbshipit-source-id: 6d57cf15c42736ecbcb106a731259b77db06d8f1
Summary:
Add some integration tests that invoked "eden doctor" and "eden list" inside a
stale mount point, and confirms that they work correctly. These commands do
fail in dev builds, since python itself does not handle this error.
Fortunately XAR-based python archives avoids this error, so these commands
work in opt mode builds.
Reviewed By: strager
Differential Revision: D8905489
fbshipit-source-id: 0666f8cb0715d61697253bee9fe0bde2f4a3fa15
Summary:
Update `eden restart` to explain that the restart will be disruptive when
performing a full restart, and prompt the user for confirmation. This prompt
can be overridden with the `--force` option. Also print more messaging after
the restart completes telling users what to do if they see "Transport endpoint
not connected" errors from programs still using the old mount points.
This also updates `eden restart` to more carefully handle cases where there is
an existing edenfs daemon running but it is not fully healthy.
Reviewed By: wez
Differential Revision: D8731004
fbshipit-source-id: 05762187b47057b2930d0a6b71b0a6fdbd4aa7e5
Summary:
This moves the fsattr helper program into the helpers/ directory
created in D8373672.
Reviewed By: chadaustin
Differential Revision: D8705391
fbshipit-source-id: 43e04d8b06b5d17caefafdfc750e3e60cad1b0cc
Summary:
This updates edenfs to be able to daemonize itself, and moves the
daemonization logic from the python CLI code into C++.
The main benefit of this is that we can now do a better job of reporting
messages to the user during start-up. We can log around potentially slow
operations (opening the RocksDB local store), and we can print messages
directly to the user if startup fails. Previously most failure messages would
go only to the eden log and would not be printed to the user's terminal.
This also fixes some issues where stdin and stdout were not closed properly
when daemonization was performed by the CLI. sudo needed access to these file
descriptors in case it needed to prompt for a password, and it would then hold
the descriptors open until edenfs exited.
Reviewed By: wez
Differential Revision: D8373672
fbshipit-source-id: 3272bff2208596f41d26e479c82c700d6c1efe11
Summary:
We've seen what appears to be phantom calls to shutdown() so we'd like
to add some degree of auditing. This diff adds a new method with some
context; this will allow us to distinguish between `eden stop`, `eden restart`
and eden server internal calls to the `shutdown` method. It may still
be possible that something else is calling our shutdown method, but it
seems unlikely as we're only accessible to our own code via a unix domain
socket.
Reviewed By: chadaustin
Differential Revision: D8341595
fbshipit-source-id: 50d58ea0b56e5f42cd37c404048d710bde0d13a3
Summary:
D7946842 moved some of the daemon management code from util.py to daemon.py
but accidentally broke handling of TimeoutError exceptions. The code in
daemon.py was trying to catch python's built-in `TimeoutError` type, while the
code in util.py throws its own `util.TimeoutError` type.
This removes the custom `TimeoutError` type in util.py and just uses the
built-in python type to be consistent everywhere.
Reviewed By: wez
Differential Revision: D8373739
fbshipit-source-id: 8523c444fef74b0758c38b4fe2644d4f68d38442