Commit Graph

1936 Commits

Author SHA1 Message Date
Adam Simpkins
ebe2fec3bf use a FakeProcessFinder in all of the eden doctor tests
Summary:
Don't let any of the `eden doctor` tests try to actually search for rogue
edenfs processes on the current machine.  We want the test behavior to be
independent of any other processes running on the current system.

Reviewed By: strager

Differential Revision: D13373059

fbshipit-source-id: 7ab26e4bba3b59b75450f223b56baa377b77319d
2018-12-13 19:57:54 -08:00
Adam Simpkins
e9ffe24065 reduce false positives from eden doctor's rogue process check
Summary:
Fix the `eden doctor` check that searches for rogue Eden processes to be more
conservative, and stop reporting processes if it can't confirm if they are
legitimate or not.

This check is inherently racy, and cannot do a good job of identifying rogue
processes if other edenfs processes are in the middle of starting or shutting
down.  In practice this happens frequently when running the Eden tests,
especially in stress test modes where many tests are run repeatedly in
parallel.

This changes the code to only report rogue processes when it actually
identifies multiple edenfs processes trying to use the same state directory.

Reviewed By: strager

Differential Revision: D13373058

fbshipit-source-id: 0f0e9fdb59db6b9721afc4171d77b27239c4de94
2018-12-13 19:57:54 -08:00
Adam Simpkins
9670ac9189 fix some typos in the eden doctor output
Summary: mispelling

Reviewed By: strager

Differential Revision: D13373060

fbshipit-source-id: c6734ba51caae9b8aa461cc3b5029217fa4eeab8
2018-12-13 19:57:54 -08:00
Matt Glazar
7f97ecc7bd Make fake_edenfs use same log path as real edenfs
Summary:
I want to use fake_edenfs to test logging via EdenFS' systemd service. Make fake_edenfs and the real edenfs use similar logic to determine the log file path.

This diff should not change behavior for the real edenfs.

Reviewed By: simpkins

Differential Revision: D13424470

fbshipit-source-id: d0c2e035fdb5884dbd2d9704c7e0244d35e052f2
2018-12-13 19:47:47 -08:00
Adam Simpkins
158fc65813 convert some of the clone test code to use pathlib
Summary:
Update some of the test code to use `Path.read_text()`, so we can eliminate
`eden.cli.util.read_all()`

Reviewed By: strager

Differential Revision: D13374245

fbshipit-source-id: 3399923b60ae78a4f7ea57367d097697c8b9c1cb
2018-12-13 19:09:15 -08:00
Adam Simpkins
e13ad7baea change how the doctor tests override pgrep
Summary:
Change the FakeProcessFinder class to derive from LinuxProcessFinder and
override its code that calls pgrep and checks lock files.  This means the test
code no longer needs to globally monkey-patch `subprocess.check_output()` and
`util.read_all()`

Reviewed By: strager

Differential Revision: D13373061

fbshipit-source-id: 82b4d28940941d1b7906c7797fe597154469ab24
2018-12-13 19:09:15 -08:00
Adam Simpkins
8c00fa3dd7 start splitting up some of the eden doctor code
Summary:
This turns the `doctor.py` module into a package, and starts splitting some of
the code out into different modules.  This should make it easier to maintain
as we continue adding more functionality.

For now I split the base `Problem` classes out into their own module, and I
split the watchman-related checks in their own module.  I will probably
continue splitting additional checks out into their own module.  (I didn't
split them out in this diff just to avoid making the diff too large.)

Reviewed By: strager

Differential Revision: D13372771

fbshipit-source-id: 015dea501ce2898ce7303a419f763f2fc962092f
2018-12-13 19:09:15 -08:00
Matt Glazar
9262202c9e Refactor TemporaryDirectoryMixin into ServiceTestCaseBase
Summary:
D13422460 made TemporaryDirectoryMixin required for classes deriving from ServiceTestCaseBase, but ServiceTestCaseBase expressed this requirement poorly. Make the dependency explicit by making ServiceTestCaseBase derive from TemporaryDirectoryMixin.

This diff should not change behavior.

Reviewed By: simpkins

Differential Revision: D13454141

fbshipit-source-id: e07745cfd2a364da5011371fe8671d94d32c6798
2018-12-13 13:52:17 -08:00
Matt Glazar
efd2d6b397 Isolate fake_edenfs tests from ~/.edenrc
Summary:
I noticed some Managed service tests behaving strangely on my dev server. They behave as if they are being managed by systemd.

I noticed that `systemctl --user status 'fb-edenfs@*.service'` showed a bunch of tmp-eden_test services. Since the Managed tests don't create an isolated systemd instance, they are starting and stopping services on my real systemd!

This behavior is caused by me setting service.experimental_systemd=true in my ~/.edenrc (D13371186).

Fix the odd behavior of these tests by preventing them from reading ~/.edenrc. Also do the same for /etc/eden.

Reviewed By: simpkins

Differential Revision: D13422460

fbshipit-source-id: b8a4cbabe55b75b34729d4122ba804cd7d3297a2
2018-12-13 13:52:17 -08:00
Matt Glazar
911fc21f2b Refactor some duplicated code in start_test.py
Summary:
The code for invoking 'eden start' is duplicated by test_daemon_command_arguments_should_forward_to_edenfs_without_leading_dashdash. I am going to add more required flags for 'eden start' tests, so the duplication is a problem. Make this test reuse spawn_start.

This diff should not change behavior.

Reviewed By: simpkins

Differential Revision: D13453511

fbshipit-source-id: 72293ed56ca90fce1cc1fa33d996edd08ca15767
2018-12-13 13:52:17 -08:00
Matt Glazar
909c2e7396 Sanitize environment vars for systemd in tests
Summary:
While trying to reproduce a failing test on Sandcastle (Facebook's CI), I encountered a bug: when run as root, StartWithRepoTestGit test_eden_start_with_systemd_mounts_checkouts failed with permission denied errors:

```
$ sudo -Hi
root$ SANDCASTLE=1 EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE=unmanaged ./buck-out/gen/eden/integration/integration#binary.par -r StartWithRepoTestGit.test_eden_start_with_systemd_mounts_checkouts
[snip]
fb-edenfs@tmp-eden_test.j62w_nfx-homedir-local-.eden.service: Main process exited, code=exited, status=70/SOFTWARE
[snip]

root$ less /var/log/messages
[snip]
Dec 10 20:05:11 devvm3761.prn2.facebook.com sh[3668152]: I1210 20:05:11.781113 3668152 main.cpp:280] edenfs exiting successfully
Dec 10 20:05:36 devvm3761.prn2.facebook.com sh[3669439]: I1210 20:05:36.908677 3669439 main.cpp:153] Running in experimental systemd mode
Dec 10 20:05:36 devvm3761.prn2.facebook.com sh[3669439]: W1210 20:05:36.910221 3669439 EdenConfig.cpp:362] error accessing config file /tmp/eden_test.j62w_nfx/homedir/.edenrc: Permission denied
Dec 10 20:05:36 devvm3761.prn2.facebook.com sh[3669439]: error creating /tmp/eden_test.j62w_nfx/homedir/local/.eden: boost::filesystem::filesystem_error: boost::filesystem::create_directory: Permission denied: "/tmp/eden_test.j62w_nfx/homedir/local/.eden"
```

edenfs is dropping its permission to my regular user. SUDO_ variables [1] propagate to edenfs, and edenfs calls `setuid($SUDO_UID)`. Clearing SUDO_UID manually fixes the test:

```
$ sudo -Hi
root$ env --unset SUDO_UID SANDCASTLE=1 EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE=unmanaged ./buck-out/gen/eden/integration/integration#binary.par -r StartWithRepoTestGit.test_eden_start_with_systemd_mounts_checkouts
[snip]
Ran 1 test in 14.720s

OK
```

According to systemd's documentation, services should have a mostly-empty environment [2]. It looks like the systemd user manager relies on this (because it's normally run via user@.service) and doesn't sanitize its environment before forking service processes.

Fix the bug by cleansing the environment when running systemd manually. This prevents SUDO_UID and other environment variables from propagating to services.

 ---

By coincidence, this change fixes the original bug I was trying to reproduce. On Sandcastle, `SUDO_COMMAND` is set to a long string with plenty of "special" characters (spaces, quotes, backslashes, equal signs, etc.). systemd barfs when shuffling the environment block around:

```
Reloading.
Failed to parse environment entry: "env=SUDO_COMMAND=/bin/bash -c SANDCASTLE_INSTANCE_ID=1488557959 [snip]\'\\\'\' \'\\\'\'\\--collection\'\\\'\' \'\\\'
Unknown serialization item '' \'\\\'\'\\--no-stress-run-sub-run\'\\\'\' \'\\\'\'\\--extended-tests\'\\\'\'\''
```

By not setting SUDO_COMMAND, we avoid this systemd bug.

 ---

[1]
```
root$ env | grep SUDO
SUDO_GID=100
SUDO_COMMAND=/bin/bash
SUDO_USER=strager
SUDO_UID=6986
```

[2] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment%20variables%20in%20spawned%20processes

Reviewed By: simpkins

Differential Revision: D13413110

fbshipit-source-id: a91d70f33c93e034bdef5573451d528a255e5fc1
2018-12-13 13:10:52 -08:00
Chad Austin
008497c69a remove SerializedFileHandleMap
Summary: SerializedFileHandleMap is dead code now.

Reviewed By: strager

Differential Revision: D13381629

fbshipit-source-id: ba872aaf8335d2be68d6af0465bd04e4ca59d578
2018-12-13 12:29:13 -08:00
Chad Austin
1e83ec3df0 remove FileHandleMap
Summary:
Eden no longer tracks any state in file handles, and has no plans to in the future.
Therefore, remove all related code.

Reviewed By: strager

Differential Revision: D13354307

fbshipit-source-id: 341d081f64c6c8fb2b4b1b5a5ff42f2cc7d38039
2018-12-13 12:29:13 -08:00
Chad Austin
6e0ce0ace0 stop handling FUSE_OPEN
Summary:
Now that all file access in Eden is stateless, we no longer need to handle open() or release().
If the kernel advertises FUSE_NO_OPEN_SUPPORT, return ENOSYS from open().

Reviewed By: simpkins

Differential Revision: D13325759

fbshipit-source-id: 38486848f27ffeb005f74407888e94d891496f98
2018-12-12 21:49:07 -08:00
Adam Simpkins
4ddda70198 unbreak open()
Summary:
D13325746 changed `EdenDispatcher::open()` to no longer return a file handle.
However the code in `FuseChannel::fuseOpen()` throws an exception if the
dispatcher does not create a file handle.  This breaks most file operations in
Eden.

The Dispatcher check is removed in D13354307, but that hasn't landed yet.
That change probably just needed to be part of D13325746.

Reviewed By: chadaustin

Differential Revision: D13445570

fbshipit-source-id: 70d639142057740766bcbe02a0df50b14f7c9937
2018-12-12 20:32:32 -08:00
Adam Simpkins
5525ce9663 remove some more of the leftover empty file checks
Summary:
Remove some more of the double-checking code left over from debugging
T24406969.  We fixed the root cause of that task and haven't seen any errors
reported by this code.  The higher log levels used for reporting information
about empty files just causes extra noise in the logs.

Reviewed By: strager

Differential Revision: D13403531

fbshipit-source-id: bcb7b29a1d687ffcefd6ecf586130b96f87b4820
2018-12-12 18:48:06 -08:00
Chad Austin
5222e339e6 remove EdenFileHandle and FileInode::open
Summary: Title says it all.

Reviewed By: strager

Differential Revision: D13325746

fbshipit-source-id: 22f1b12ba0bf47eba62c2312e5069c45b1c28ef3
2018-12-12 17:10:29 -08:00
Chad Austin
a978af2c62 stop looking up file handles in FuseChannel
Summary:
Previously, a file handle must have been held for the entirety of a write operation. That is no
longer true. Stop looking up file handles on write.

Reviewed By: strager

Differential Revision: D13325662

fbshipit-source-id: 9ae31b467d17d633c388917d18098e6e5a620b89
2018-12-12 17:10:29 -08:00
Chad Austin
cc1ede2ffa stop counting open FileInode handles
Summary:
Now that FileInode read and write operations are stateless via BlobAccess and OverlayFileAccess,
EdenFileHandle no longer provides any value. Remove it. This also fixes eden's shutdown timeout
when a file handle is open and paves the way for FUSE_NO_OPEN_SUPPORT.

Reviewed By: strager

Differential Revision: D13325137

fbshipit-source-id: 71ed47a7c997f5035b4394ccb311f94332ecd8c2
2018-12-12 17:10:29 -08:00
Chad Austin
7c5c5e75d0 route overlay file access through OverlayFileAccess
Summary:
Have FileInode use OverlayFileAccess instead of using the Overlay directly.
This allows IO on materialized files to be stateless and pave the way for
eliminating EdenFileHandle. It also paves the way for performance improvements
such as nicer SHA-1 caching.

Reviewed By: strager

Differential Revision: D13325079

fbshipit-source-id: fb27d48b5dc9196dc6e36557596f601194a56aa9
2018-12-12 17:10:29 -08:00
Chad Austin
2e53003e4a introduce an OverlayFileAccess class with a handle cache
Summary:
Move the overlay IO logic out of FileInode into a centralized OverlayFileAccess class. It keeps the last 100 overlay file handles open.

FileInode's changes are coming in D13325079.

Reviewed By: strager

Differential Revision: D13324995

fbshipit-source-id: 04fb3fe50114b0f19b78acd17a9684c92f8e8072
2018-12-12 17:10:29 -08:00
Adam Simpkins
fbc7306e50 rename ClientConfig to CheckoutConfig in the CLI code
Summary:
Rename `ClientConfig` to `CheckoutConfig` to move slightly closer towards
standardizing on the term "checkout" rather than "client" to refer to an Eden
mount point directory.

Reviewed By: chadaustin

Differential Revision: D13437952

fbshipit-source-id: 5086b4187eb975c860d48cadf0ccd4e5dce78e9a
2018-12-12 16:42:56 -08:00
Adam Simpkins
aff543f3f5 remove the non-toml config files
Summary:
Support for reading non-toml configs was removed in D10020958.
This updates the code to stop deploying these files.

The files in `/etc/eden/config.d` are only read by the python CLI code,
not by the edenfs daemon.  Therefore it should be fine to delete these files
even if some users still happen to still be running an older build of edenfs.

Reviewed By: chadaustin

Differential Revision: D13437953

fbshipit-source-id: 9001a49a697f80fc7a25343e3c22dff322924de8
2018-12-12 16:42:56 -08:00
Chad Austin
ad044f2da0 add fuzz test for modifications concurrent with readdir
Summary:
Add a fuzz test that randomly modifies a directory during a series of
readdir calls and verifies each unmodified entry is returned once.

Reviewed By: simpkins

Differential Revision: D13371162

fbshipit-source-id: 252b03ab0288e82b56a33c347955de129e61ae42
2018-12-11 12:22:12 -08:00
Matt Glazar
a333717ef2 Add default edenfs paths for systemd service
Summary: With systemd integration enabled, --daemon-binary is required for 'eden start'. Make 'eden start' infer the location if one is not given explicitly. This matches 'eden start' with systemd integration disabled.

Reviewed By: chadaustin

Differential Revision: D13271450

fbshipit-source-id: c9ace1f271a5d8b6b4515c591f16ed770f584a6b
2018-12-10 21:53:13 -08:00
Matt Glazar
b0dc135133 Refactor _get_daemon_args to throw on error
Summary:
_get_daemon_args's return type is awkward. On success, it returns the command and required environment variables. On failure, it returns a non-zero `int`.

Instead of returning errors via `int`, raise an exception. Exceptions are a natural way to report errors, and make reusing this code easier in a future diff.

This diff should not change behavior.

Reviewed By: chadaustin

Differential Revision: D13404920

fbshipit-source-id: 8ae24559ddf1a6adc582fa5418e9b6eb40c68181
2018-12-10 21:53:13 -08:00
Chad Austin
0253a3aa5c fix crash after blob is evicted
Summary:
If a blob was partially read, evicted from cache, and then read again,
the readByteRanges coverage set was not being cleared. Always clear it
in startLoadingData.

Reviewed By: strager

Differential Revision: D13405267

fbshipit-source-id: 6f60e6e80662fd470fe4ddbc833fc8efd8850686
2018-12-10 19:30:28 -08:00
Chad Austin
fe557ace7c only evict when interest handle is dropped if blob wasn't reloaded
Summary:
There was a bug in BlobCache where, if you had an interest handle to a
blob, but that blob was evicted anyway and then something else caused
it to be reloaded, dropping your interest handle would cause the blob
to be incorrectly evicted since the reference counts were no longer
compatible. Add a version to cache items and only decrement the
reference count on an item if the interest handle and item agree.

Reviewed By: strager

Differential Revision: D13405144

fbshipit-source-id: aee052bf777e7225551c3ae2b8b69a99f4f77691
2018-12-10 19:30:28 -08:00
Matt Glazar
f0003e3735 Make tests work with LSAN logging
Summary:
D13366696 made some tests fail. These tests assume stderr doesn't print extra messages.

Make these tests tolerate extra messages on stderr, so extra logging doesn't make them fail.

Reviewed By: simpkins

Differential Revision: D13409510

fbshipit-source-id: 7c80766dd3978bb78c9a6fc8ed23310c2ee744e3
2018-12-10 19:06:32 -08:00
Matt Glazar
c91b552ef5 Allow systemd opt-in via config
Summary:
Currently, users can opt into systemd for edenfs by setting EDEN_EXPERIMENTAL_SYSTEMD=1 every time they use Eden's CLI. Make it easier to opt in: use systemd if service.experimental_systemd is set in ~/.edenrc. Also allow opting out for a single command via EDEN_EXPERIMENTAL_SYSTEMD=0.

For users who don't set service.experimental_systemd or use EDEN_EXPERIMENTAL_SYSTEMD, this diff should not change behavior. (The EDEN_EXPERIMENTAL_SYSTEMD environment variable is still respected.)

Reviewed By: simpkins

Differential Revision: D13371186

fbshipit-source-id: d7b533ea3583f4b6c8a8f63c6a74eace2be7d783
2018-12-10 16:30:06 -08:00
Matt Glazar
7d5de37259 Factor duplicated code in config_test.py
Reviewed By: simpkins

Differential Revision: D13371095

fbshipit-source-id: 281442ab379e9ee3a02d779c59049e433862c378
2018-12-07 18:50:35 -08:00
Matt Glazar
4af60a78e0 Refactor should_use_experimental_systemd_mode into EdenInstance
Summary:
I want to allow opting into systemd using a setting in ~/.edenrc. Since should_use_experimental_systemd_mode is a global function, it can't read any configs. Make the config file visible to should_use_experimental_systemd_mode by moving it into EdenInstance.

This diff should not change behavior.

Reviewed By: simpkins

Differential Revision: D13370823

fbshipit-source-id: b604db66954d0a08973030daae38bf6b1433e821
2018-12-07 18:50:35 -08:00
Matt Glazar
36e609d909 Integrate 'eden restart' with systemd
Summary:
If the user opted into systemd integration, make 'eden restart' restart the systemd service.

Don't support --graceful yet. --graceful will be implemented in a later diff. Mark it as explicitly unsupported for now.

Reviewed By: chadaustin

Differential Revision: D13271438

fbshipit-source-id: e505f00cbc337a2bf4da77bdea4b8faba063607c
2018-12-07 18:50:35 -08:00
Chad Austin
1402e9a0d6 fix rage output test
Summary:
This made it through landcastle because of our current ASAN flakiness
issue on sandcastle marking almost all integration tests as disabled.

Reviewed By: strager

Differential Revision: D13385128

fbshipit-source-id: 70dee16fc5c99819fe121f4b96f7307d05ac24e6
2018-12-07 17:23:32 -08:00
Matt Glazar
f86b6fc746 Forward systemd startup logs to 'eden start'
Summary:
When you run 'eden start' without systemd integration, edenfs writes startup logs to the terminal to let users know that stuff is happening:

```
$ eden start
Starting edenfs (dev build), pid 2792025
Opening local RocksDB store...
Opened RocksDB store in 0.95 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2792025)
Logs available at /data/users/strager/.eden-dev/logs/edenfs.log
```

These startup logs are also used by various tests (especially 'eden restart's tests).

Make the same thing happen when running 'eden start' with systemd integration, improving the user experience and making some tests work:

```
$ EDEN_EXPERIMENTAL_SYSTEMD=1 \
  ./buck-out/gen/eden/cli/eden.par start \
  --daemon-binary "${PWD}/buck-out/gen/eden/fs/service/edenfs"
Starting edenfs (dev build), pid 2800760
Opening local RocksDB store...
Opened RocksDB store in 0.693 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2800760)
```

Reviewed By: wez

Differential Revision: D13241979

fbshipit-source-id: de79b714e42b690fdab7c21d9add46bc2da35328
2018-12-07 17:07:08 -08:00
Matt Glazar
75d7a135b5 Pass extra daemon args from 'eden start' to systemd
Summary: Several tests for 'eden start', 'eden stop', and 'eden status' need to pass command-line arguments to fake_edenfs. With systemd support enabled, make 'eden start' forward daemon arguments to fake_edenfs, making these tests pass.

Reviewed By: wez

Differential Revision: D13249891

fbshipit-source-id: 9008a361fce7a5629535cc9d245b86073ee70826
2018-12-07 17:07:08 -08:00
Matt Glazar
f33c394da5 Fix crash for getSHA1(./hello)
Summary:
getSHA1 is only handling std::system_error. If another kind of exception is thrown, it's never caught and edenfs crashes.

Calling EdenMount::getInode with a path such as "./hello" will cause a std::domain_error to be thrown. Since std::domain_error is not derived from std::system_error, `getSHA1ForPathDefensively(["./hello"])` crashes edenfs.

Fix the crash by forwarding all exceptions over Thrift, not just std::system_error-s.

Reviewed By: simpkins

Differential Revision: D13386450

fbshipit-source-id: 06262dad30a5508ed482c9e8979b61aa9643280a
2018-12-07 16:34:42 -08:00
Matt Glazar
4ca24ee2b3 Disable LSAN to fix Sandcastle flakiness
Summary:
On Sandcastle's continuous builds, many EdenFS tests are flaky. D13366696 added logging, but the logs don't tell us much:

```
==24060==AddressSanitizer: libc interceptors initialized
|| `[0x10007fff8000, 0x7fffffffffff]` || HighMem    ||
|| `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
|| `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap  ||
|| `[0x00007fff8000, 0x00008fff6fff]` || LowShadow  ||
|| `[0x000000000000, 0x00007fff7fff]` || LowMem     ||
MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
redzone=16
max_redzone=2048
quarantine_size_mb=256M
thread_local_quarantine_size_kb=1024K
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 0x7fff8000
==24060==Installed the sigaction for signal 11
==24060==Installed the sigaction for signal 7
==24060==Installed the sigaction for signal 8
==24060==T0: FakeStack created: 0x7f2a640b4000 -- 0x7f2a64bbd000 stack_size_log: 20; mmapped 11300K, noreserve=0
==24060==T0: stack [0x7fff12e4e000,0x7fff1364e000) size 0x800000; local=0x7fff1364d298
==24060==AddressSanitizer Init done
AddressSanitizer ignores mlock/mlockall/munlock/munlockall
==24208==Processing thread 24060.
==24208==Stack at 0x7fff12e4e000-0x7fff1364e000 (SP = 0x7fff1364c3f8).
==24208==TLS at 0x7f2a6a7e3600-0x7f2a6a7e46c0.
==24208==DTLS 18 at 0x4f80004920000010-0x5280004820000012.
Tracer caught signal 11: addr=0x0 pc=0x7f2a69e830e0 sp=0x7f2a4ae85cd0
==24060==LeakSanitizer has encountered a fatal error.
==24060==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==24060==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
```

Facebook's test infra is disabling many of EdenFS' tests. This has already caused regressions to land [1]. We need to get these tests passing and enabled ASAP.

The current hypothesis is that LSAN's leak detection is crashing. Disable the leak detection, hopefully making tests pass again. (We still want to find and fix the root cause, of course!)

[1] T37784916: test_rage_output is failing: AssertionError: 'General EdenFS Statistics' not found

Reviewed By: simpkins

Differential Revision: D13385022

fbshipit-source-id: c442146c39ce84c19fc53916aef421cece6d8b40
2018-12-07 15:45:32 -08:00
Chad Austin
78c1829324 make readdir POSIX-compliant
Summary:
As I'm working on ripping out file handle support, I wanted to ensure
that we had a path towards a correct and stateless readdir()
implementation. Stateful file handles require extra care during
graceful restart, and it's nice that we can avoid them. In fact, this
work paves the way towards a possible FUSE_NO_OPENDIR_SUPPORT feature.

This diff fixes concurrent rm -rf in the same directory.

Reviewed By: simpkins

Differential Revision: D13370898

fbshipit-source-id: eba650e673a7cb9559e04ba28417980f6d0c65cb
2018-12-07 13:01:04 -08:00
Chad Austin
3dbad3ac01 show blob cache sizes in eden stats
Summary:
Have `eden stats` print the size of the blob cache if the running
edenfs has information about it.

Reviewed By: strager

Differential Revision: D13349220

fbshipit-source-id: 9f59f4399f2d4283aa80bcb54ba73c51d555d502
2018-12-06 19:43:52 -08:00
Matt Glazar
50c0030d8f Give edenfs mount permissions in systemd
Summary:
edenfs's privhelper process needs the CAP_SYS_ADMIN capability [1] in order to manage mount points. Give it this capability by invoking edenfs as the `root` user using `sudo`.

Known issues:

* `sudo` must be passwordless in order for fb-edenfs@.service to start.
* systemd can't kill all of fb-edenfs@.service's processes, so `systemctl stop` is unreliable for example.

[1] https://manpage.me/index.cgi?q=capabilities&apropos=0&sektion=0&manpath=CentOS+7.1&arch=default&format=html

Reviewed By: chadaustin

Differential Revision: D13113450

fbshipit-source-id: 01b89521cab371b5017fab6fbd38d55eea599c46
2018-12-06 19:43:52 -08:00
Matt Glazar
7e7c15976e Refactor systemd status assertions into EdenFSSystemdMixin
Summary:
I want to reuse assert_systemd_service_is_active in another test. Move it (and the related assert_systemd_service_is_stopped function) so it can be reused.

This diff should not change behavior.

Reviewed By: chadaustin

Differential Revision: D13327802

fbshipit-source-id: 022c3ed3b9e8f04ef1156c2bb4b3deda662439e4
2018-12-06 19:13:17 -08:00
Matt Glazar
9f84aba270 Add test: 'eden stop' stops systemd service
Summary: 'eden stop' should behave like 'systemctl stop'; the service should be stopped after 'eden stop' returns. Add a test which verifies this.

Reviewed By: chadaustin

Differential Revision: D13288819

fbshipit-source-id: 5b836c8ac7c5eb97c484195496f38c7cf70c84dc
2018-12-06 19:13:17 -08:00
Matt Glazar
956f7f714e Add logging to debug flaky tests
Summary:
On Sandcastle's continuous builds, many EdenFS tests are flaky:

```
EdenCommandError: eden command [/mnt/btrfs/trunk-hg-fbcode-fbsource-303-1544100804/fbcode/buck-out/dev/gen/eden/cli/eden.par --config-dir /tmp/eden_test.udfpngq0/homedir/local/.eden --etc-eden-dir /tmp/eden_test.udfpngq0/etc-eden --home-dir /tmp/eden_test.udfpngq0/homedir repository main /tmp/eden_test.udfpngq0/repos/main] returned non-zero exit status 1
stderr=b'Tracer caught signal 11: addr=0x0 pc=0x7f65cea990e0 sp=0x7f65afb21cd0\n==1698412==LeakSanitizer has encountered a fatal error.\n==1698412==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1\n==1698412==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)\n'
```

I failed to reproduce this issue on my own dev server, and I failed to reproduce this issue by running 'buck test' on a Sandcastle worker.

As a last resort, make LSAN report more stuff in an effort to track down the cause of this failure.

Aside from extra logging, this diff should not change behavior.

Reviewed By: chadaustin

Differential Revision: D13366696

fbshipit-source-id: e052700dbc4ed9d30f864b8d2dc5ccad27e1a281
2018-12-06 18:33:12 -08:00
Chad Austin
f96cae3138 simplify eden stats output
Summary:
`eden stats` had ugly output - it was verbose and indented oddly. Make
a pass at improving it.

Reviewed By: strager

Differential Revision: D13349143

fbshipit-source-id: b7853cfdb8ce31bb46f28a426907eb6d4c33d5ee
2018-12-06 18:18:21 -08:00
Matt Glazar
2890b1f6b2 Delete dead code in config_test.py
Summary: Some functions in config_test.py are not called anywhere. Delete them.

Reviewed By: chadaustin

Differential Revision: D13370691

fbshipit-source-id: 89a2e3956df31a24773dec87868bf4612665af82
2018-12-06 18:11:27 -08:00
Matt Glazar
b8911889f0 Refactor systemd ExecStart: enable sh logic
Summary:
I want to add some non-trivial systemd-specific logic to fb-edenfs@.service's ExecStart command. Refactor the existing command to use sh, giving us access to `if` and other useful utilities.

Also, the shell language is likely more familiar (e.g. `${foo}` and `$foo` behave the same in sh, but behave differently in systemd), so using sh should lead to fewer headaches.

This diff should not change behavior.

Reviewed By: chadaustin

Differential Revision: D13165377

fbshipit-source-id: 2e500abf75c6f79a8b174a848f7eeb1cfaebb96c
2018-12-06 17:51:25 -08:00
Matt Glazar
e53e3c4cad Enable some 'eden stop' tests with systemd
Summary:
The following tests in stop_test.py pass with systemd enabled:

* test_killing_hung_daemon_during_stop_makes_stop_finish
* test_stop_not_running
* test_stop_stops_running_daemon
* test_stopping_daemon_stopped_by_sigstop_kills_daemon
* test_stopping_killed_daemon_reports_not_running

Enable these tests to prevent regressions.

Reviewed By: chadaustin

Differential Revision: D13229634

fbshipit-source-id: 1c632260afdc7286bd8b1465258b129f20f02f18
2018-12-06 17:48:15 -08:00
Matt Glazar
c20d275a6b Make 'eden start' run a bare minimum systemd service (opt-in)
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.

Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.

'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).

Reviewed By: simpkins

Differential Revision: D10849390

fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
2018-12-06 16:01:43 -08:00
Chad Austin
4745a38c15 add a caching strategy document
Summary:
Describe the various caches in Eden and the decisions made while
shaping them.

Reviewed By: wez, strager

Differential Revision: D13149018

fbshipit-source-id: b7f25a3a25ec7ab3ff364fb0c4f4e755acec9a4e
2018-12-06 12:33:31 -08:00