Summary:
When you run 'eden start' without systemd integration, edenfs writes startup logs to the terminal to let users know that stuff is happening:
```
$ eden start
Starting edenfs (dev build), pid 2792025
Opening local RocksDB store...
Opened RocksDB store in 0.95 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2792025)
Logs available at /data/users/strager/.eden-dev/logs/edenfs.log
```
These startup logs are also used by various tests (especially 'eden restart's tests).
Make the same thing happen when running 'eden start' with systemd integration, improving the user experience and making some tests work:
```
$ EDEN_EXPERIMENTAL_SYSTEMD=1 \
./buck-out/gen/eden/cli/eden.par start \
--daemon-binary "${PWD}/buck-out/gen/eden/fs/service/edenfs"
Starting edenfs (dev build), pid 2800760
Opening local RocksDB store...
Opened RocksDB store in 0.693 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2800760)
```
Reviewed By: wez
Differential Revision: D13241979
fbshipit-source-id: de79b714e42b690fdab7c21d9add46bc2da35328
Summary: Several tests for 'eden start', 'eden stop', and 'eden status' need to pass command-line arguments to fake_edenfs. With systemd support enabled, make 'eden start' forward daemon arguments to fake_edenfs, making these tests pass.
Reviewed By: wez
Differential Revision: D13249891
fbshipit-source-id: 9008a361fce7a5629535cc9d245b86073ee70826
Summary:
getSHA1 is only handling std::system_error. If another kind of exception is thrown, it's never caught and edenfs crashes.
Calling EdenMount::getInode with a path such as "./hello" will cause a std::domain_error to be thrown. Since std::domain_error is not derived from std::system_error, `getSHA1ForPathDefensively(["./hello"])` crashes edenfs.
Fix the crash by forwarding all exceptions over Thrift, not just std::system_error-s.
Reviewed By: simpkins
Differential Revision: D13386450
fbshipit-source-id: 06262dad30a5508ed482c9e8979b61aa9643280a
Summary:
Have `eden stats` print the size of the blob cache if the running
edenfs has information about it.
Reviewed By: strager
Differential Revision: D13349220
fbshipit-source-id: 9f59f4399f2d4283aa80bcb54ba73c51d555d502
Summary:
edenfs's privhelper process needs the CAP_SYS_ADMIN capability [1] in order to manage mount points. Give it this capability by invoking edenfs as the `root` user using `sudo`.
Known issues:
* `sudo` must be passwordless in order for fb-edenfs@.service to start.
* systemd can't kill all of fb-edenfs@.service's processes, so `systemctl stop` is unreliable for example.
[1] https://manpage.me/index.cgi?q=capabilities&apropos=0&sektion=0&manpath=CentOS+7.1&arch=default&format=html
Reviewed By: chadaustin
Differential Revision: D13113450
fbshipit-source-id: 01b89521cab371b5017fab6fbd38d55eea599c46
Summary:
I want to add some non-trivial systemd-specific logic to fb-edenfs@.service's ExecStart command. Refactor the existing command to use sh, giving us access to `if` and other useful utilities.
Also, the shell language is likely more familiar (e.g. `${foo}` and `$foo` behave the same in sh, but behave differently in systemd), so using sh should lead to fewer headaches.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13165377
fbshipit-source-id: 2e500abf75c6f79a8b174a848f7eeb1cfaebb96c
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.
Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.
'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).
Reviewed By: simpkins
Differential Revision: D10849390
fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
Summary: This diff also includes build script changes to EdenDispatcher, FsChannel and EdenMount.
Reviewed By: strager
Differential Revision: D13091786
fbshipit-source-id: cecc8d849fcb9ebc8fa718e1011ef8931bebc279
Summary: Instantiate a blob cache in the EdenServer with configurable settings.
Reviewed By: strager
Differential Revision: D12813880
fbshipit-source-id: 8ccc89826f04aca78964230374dea48abf05e05e
Summary: Add type annotations to all functions in eden/fs/service/client.py
Reviewed By: strager
Differential Revision: D13051091
fbshipit-source-id: 23b93008352664336ad155a7f5cc281bd5529702
Summary: Update the CLI code to make the pyre type checker happy.
Reviewed By: wez
Differential Revision: D13035413
fbshipit-source-id: d201f2e65667e0ce1bf4a73fbb05878e8711ad16
Summary:
This makes it possible to change configuration options
for the LocalStore while the server is running.
As you'll see in the next diff, our current layering makes using
the config a bit more awkward, but at least this diff doesn't
look gross :-p
This diff doesn't introduce any new functionality or configuration.
Reviewed By: strager
Differential Revision: D12949577
fbshipit-source-id: cf897ba676b9359f92865170faa42ff17329b85f
Summary:
Sometimes users accidentally run `edenfs start` or `edenfs restart` instead of
`eden start` or `eden restart`. This adds a new `--edenfs` flag to the
`edenfs` binary, and asks users if they meant to run `eden` instead if they do
not pass in this flag.
This used to be less of a problem since `edenfs` required users to also
explicitly specify several other configuration flags (like `--edenDir`).
However `edenfs` can not automatically figure out these settings, so these
flags are no longer required. Therefore `edenfs` would still try to start
normally when invoked with `edenfs restart`, since it did not require these
flags and it did not complain about unhandled command line arguments.
Reviewed By: wez
Differential Revision: D12927803
fbshipit-source-id: dbf7ce2449c391ca218652439eb68ff43c2ebd46
Summary: folly/Subprocess is not compatible with Windows and has broken the Windows build. It's not used so removing it.
Reviewed By: wez
Differential Revision: D12967451
fbshipit-source-id: 54d33bf6fe2ec3ede9d68eccd99e53c5eb6ed53d
Summary:
Sandcastle has several cases where we chown the entire
repository which performs terribly on Eden. As a workaround we have a
command to do this in eden without loading all the files.
Reviewed By: chadaustin
Differential Revision: D12857956
fbshipit-source-id: 36cebcc710fbcf4e1eb265df901513cf50a227b9
Summary:
With this the eden cli can dump tracepoints and translate
them to various formats or perform any processing
Reviewed By: chadaustin
Differential Revision: D10384072
fbshipit-source-id: 8b38e7f6b551a2bd98b3e748ba1cceafeceeec8c
Summary:
This uses the tracing infrastructure to create blocks around
the getSHA1 invocation as well as the getOrLoadChild (called for each
element of the path-walk) method.
Reviewed By: chadaustin
Differential Revision: D10384074
fbshipit-source-id: a06fbe38e8d3f35fcb248e6bd724e5572724d27d
Summary:
Remove the handful of (but growing) parameters that
are threaded down to the HgBackingStore in favor of the ReloadableConfig
object and use that to configure mononoke support.
This doesn't notice changes to the config on the fly today, but
does simplify the amount of stuff we need to pass around.
Reviewed By: chadaustin
Differential Revision: D12813053
fbshipit-source-id: b8309328fab4980889624257cfd387c8c1335001
Summary:
This allows detecting changes to the edenrc configuration
and adjusting how we read data accordingly.
To avoid a huge gnarly diff with a lot of dependencies, this diff
introduces a little interface class that defines the method to obtain
a possibly reloaded EdenConfig instance.
This diff shouldn't change any functionality.
Reviewed By: chadaustin
Differential Revision: D12813051
fbshipit-source-id: 35390dc28671ba46a03d098c4f280e2f567dbdc9
Summary: If an error happens when opening the log file, edenfs should stop immediately and report the error. Write a test to verify that this happens with DaemonStartupLogger.
Reviewed By: chadaustin
Differential Revision: D10436926
fbshipit-source-id: 3f33ed276a668b7ecda8176a55b114b5878304e3
Summary:
When running edenfs in daemonize mode (i.e. when --foreground is not specified), the daemon process inherits stdin from the parent and never closes it. This has two consequences:
* The daemon process holds a reference to stdin even after the parent exits. This causes tests to hang if they wait for EPIPE on stdin after waiting for the parent process to exit. (I discovered this issue when adding tests for 'eden start' in D10434379.)
* edenfs could potentially read from the terminal after 'eden start' has returned. I think this behavior is surprising and could potentially cause problems.
Promptly close stdin after forking the daemon process, since nothing inside the daemon needs stdin.
Reviewed By: simpkins
Differential Revision: D10442346
fbshipit-source-id: f3d287e8b48e121807f845b73a1a550e63efae25
Summary:
getLogPath in eden/fs/service/main.cpp ensures that a non-empty log path is given to DaemonStartupLogger::daemonize. An empty log path is not possible. DaemonStartupLogger behaves differently with an empty log path, but that behavior has caused problems (see D10439299).
Make DaemonStartupLogger require a (non-empty) log path, simplifying its implementation.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10227597
fbshipit-source-id: 37931528add51ab3e6e1b941478000297cb8b693
Summary:
The daemonizeIfRequested is implemented in the main edenfs process as well as in fake_edenfs. Consolidate the almost-identical implementations.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439300
fbshipit-source-id: 6253668896de0f8393b92f4aa5ee8a8a2fcbfb31
Summary:
Our CMake-based build had gotten somewhat out of date. This fixes several
issues that were preventing it from building:
- wangle now depends on fizz, which in turn depends on libsodium
- fbthrift now depends on yarpl for the streaming functionality
- Eden now depends on libsqlite3
- The eden/fs/inode library now depends on eden/fs/config
Reviewed By: strager
Differential Revision: D10436531
fbshipit-source-id: f41065da2edb909e8eb94e03d0e62ae19ad3a84f
Summary:
Because Mercurial blob IDs change without the contents changing, and
because files get unloaded upon checkout, rebasing across a large
distance in history can result in status fetching a lot of
metadata. Keep a smallish LRU cache for SHA-1 and size by blob ID.
Reviewed By: strager
Differential Revision: D10419965
fbshipit-source-id: 81499573814775471913db05f924767c3bab300e
Summary: When debugging a failure in DaemonStartupLoggerTest.closePipeWhileStillRunning, I was confused because the pipe which is closed in the test (readPipe/writePipe) is unrelated to the DaemonStartupLogger. Eventually I learned that the pipe referred to by the test's name is a pipe internal to DaemonStartupLogger. Rename the test to express the high-level scenario instead of mentioning pipes (which are an implementation detail).
Reviewed By: simpkins
Differential Revision: D10247254
fbshipit-source-id: f5f5b358adad0b656db27ea75fec65fe0331317b
Summary:
StartupLogger handles two independent scenarios: edenfs in daemon mode, and edenfs in foreground mode. I need to add a third for systemd support: edenfs in foreground mode but with startup logs written to a file.
Refactor the existing StartupLogger into an abstract base class with two concrete derived classes: DaemonStartupLogger and ForegroundStartupLogger. This will make adding the systemd feature easy without confusing the logic for the existing scenarios.
Aside from dropping the setsid() call for ForegroundStartupLogger, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D10228156
fbshipit-source-id: fbc2c771c6fbb675779ef05ca3b7395b70b67485
Summary: Convert then to thenValue or thenTry as appropriate.
Reviewed By: Orvid
Differential Revision: D10413847
fbshipit-source-id: 834756c2bcb4722fbe3729ce54aece0482b0a7ff
Summary:
To ease the migration from the current custom daemon setup to a systemd-managed daemon, add a way to enable the to-be-implemented systemd behavior.
* To the edenfs daemon, support the --experimentalSystemd option.
* To the Eden CLI, support the EDEN_EXPERIMENTAL_SYSTEMD environment variable. When set, invoke the edenfs daemon with --experimentalSystemd.
Aside from adding a log message, this diff does not change any behavior. This diff just sets up scaffolding for future diffs.
Reviewed By: simpkins
Differential Revision: D10248055
fbshipit-source-id: 30ca5a4bfde00ff43fd6f2a5d0282ced4f177fed
Summary:
D10370931 made all logs from StartupLogger use origStderr_ and never origStdout_. origStdout_ is thus unused. Delete origStdout_.
This diff should not change behavior.
Reviewed By: pkaush
Differential Revision: D10371294
fbshipit-source-id: 5dc1049192fea68a064a81143ea140c07452ecef
Summary:
Logging some messages to stdout and some messages to stderr has a few problems:
* stdout and stderr can get interleaved. For pipes, interleaving is avoidable for messages smaller than PIPE_BUF (512 bytes), but there's no guarantee that stdout and stderr are pipes or that the reader of the pipe reads PIPE_BUF bytes at a time.
* For the systemd integration I am developing, I need one file for stderr logs and another file for stdout logs. This is more complicated than having a single file for all logs.
* The distinction between stdout and stderr is unclear. No messages from StartupLogger are meant to be piped to another program.
Address these problems by only logging to stderr.
From a user's point of view, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10370931
fbshipit-source-id: b063ed3a66cefc846d30ce1486759cbd3f3ea29e
Summary:
I want to split the StartupLogger class (see D10228156). The split will make logPath_ inaccessible from the success() function. I see two solutions:
* Make successImpl() (called by success()) write the entire message. This means each implementation of successImpl() needs to log the same "Started edenfs" message.
* Make success() write just the "Started edenfs" message, and have successImpl() write the "Logs available at" message (if possible). This means each implementation of successImpl() which has a log file needs to log the same "Logs available at" message, but in the short term there will be only one such implementation.
Implement the latter solution to minimize source code size.
Reviewed By: simpkins
Differential Revision: D10228164
fbshipit-source-id: aaa03e9d8c1d08c00a388a7fcaa76aa3f629decf
Summary: Ensure the "Started edenfs" log message is shown to the parent process when running in daemon mode. Also ensure the log message is shown at all in foreground mode.
Reviewed By: simpkins
Differential Revision: D10227185
fbshipit-source-id: c13987100c4b591218bbf1aa80828a8490188434
Summary: Existing tests for StartupLogger only test the daemon (not --foreground) case. Test the --foreground case as well.
Reviewed By: simpkins
Differential Revision: D10225293
fbshipit-source-id: b99d3ed9457d2d57cefd648d5e9513d12189d0b7
Summary: This was previously hard coded and making it configurable is operationally desirable.
Reviewed By: chadaustin
Differential Revision: D10298279
fbshipit-source-id: 124c3fa319c44aacb3d17ef6e380e9dec49db8be
Summary:
This explains why some people ended up with broken bind mounts. If
bind mount creation ever failed, eden start would silently ignore the
error, leaving buck-out as a mutable directory in the Eden overlay
itself.
Reviewed By: wez, strager
Differential Revision: D10118612
fbshipit-source-id: ce99df4216f6815c24dc8283e55a9a30d58d29a1
Summary: If edenfs is unresponsive, 'eden status' hangs forever. Add a timeout to turn the hang into a user-friendly error.
Reviewed By: chadaustin
Differential Revision: D10156229
fbshipit-source-id: 9186826ae6b131a193b1499c8baac616d131357f
Summary:
The eden server library used to rely on a symbol being defined by the
executable that linked it. Pass runServer as a parameter to
EdenServer::run to allow us to leave undefined_symbols=False in the
library rule.
Reviewed By: wez
Differential Revision: D10246972
fbshipit-source-id: d4c47e58947a2d57c4163134db2455fe9c815344
Summary: Eliminate the need to look up an InodeNumber from a FileHandle. Instead, simply preserve the mapping when it's created.
Reviewed By: wez
Differential Revision: D10187120
fbshipit-source-id: dc47f7776294871ff2398f33c31bd85d240ead50
Summary:
When running the watchman integration tests, I saw that some of them were
fataling here:
diffusion/FBS/browse/master/fbcode/thrift/lib/cpp2/async/StreamPublisher-inl.h$71
In this situation we're tearing down all the subscribers via
`Journal::cancelAllSubscribers` and this causes the stream be destroyed and
trigger this FATAL check.
We need to explicitly complete it before we can proceeed, so we do the gross
thing and call this in the destructor of a little wrapper.
This is made a little more complex because the disconnection handler has to
inform the Journal that it was torn down in the case where the client has
gone away. The result is that we need to be careful to avoid the destructor
on Journal callback from running while we hold the Journal subscriber lock.
Reviewed By: strager
Differential Revision: D10024271
fbshipit-source-id: 06a9d40f7f6e46fe35ffcedba2669e27e6624427
Summary:
In D6766900, we turned on automatic background inode
unloading. However, there's a bug somewhere in InodeMap or the loading
sequence that causes occasional crashes for users, so turn it back off
for now.
Reviewed By: pkaush
Differential Revision: D10114443
fbshipit-source-id: fd35fd31b5731aa9bf26004fca8aa73930bb2344
Summary:
Previously, Eden would log `UnloadInodeScheduler Unloading Free
Inodes` every 10 minutes. Now it only logs if it actually unloads an
inode for a mount.
Reviewed By: strager
Differential Revision: D9999029
fbshipit-source-id: f01d0e7b1341bef230bcb5a327c31ffc8e3aeb2c
