Summary:
When you run 'eden start' without systemd integration, edenfs writes startup logs to the terminal to let users know that stuff is happening:
```
$ eden start
Starting edenfs (dev build), pid 2792025
Opening local RocksDB store...
Opened RocksDB store in 0.95 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2792025)
Logs available at /data/users/strager/.eden-dev/logs/edenfs.log
```
These startup logs are also used by various tests (especially 'eden restart's tests).
Make the same thing happen when running 'eden start' with systemd integration, improving the user experience and making some tests work:
```
$ EDEN_EXPERIMENTAL_SYSTEMD=1 \
./buck-out/gen/eden/cli/eden.par start \
--daemon-binary "${PWD}/buck-out/gen/eden/fs/service/edenfs"
Starting edenfs (dev build), pid 2800760
Opening local RocksDB store...
Opened RocksDB store in 0.693 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2800760)
```
Reviewed By: wez
Differential Revision: D13241979
fbshipit-source-id: de79b714e42b690fdab7c21d9add46bc2da35328
Summary: Several tests for 'eden start', 'eden stop', and 'eden status' need to pass command-line arguments to fake_edenfs. With systemd support enabled, make 'eden start' forward daemon arguments to fake_edenfs, making these tests pass.
Reviewed By: wez
Differential Revision: D13249891
fbshipit-source-id: 9008a361fce7a5629535cc9d245b86073ee70826
Summary:
getSHA1 is only handling std::system_error. If another kind of exception is thrown, it's never caught and edenfs crashes.
Calling EdenMount::getInode with a path such as "./hello" will cause a std::domain_error to be thrown. Since std::domain_error is not derived from std::system_error, `getSHA1ForPathDefensively(["./hello"])` crashes edenfs.
Fix the crash by forwarding all exceptions over Thrift, not just std::system_error-s.
Reviewed By: simpkins
Differential Revision: D13386450
fbshipit-source-id: 06262dad30a5508ed482c9e8979b61aa9643280a
Summary:
On Sandcastle's continuous builds, many EdenFS tests are flaky. D13366696 added logging, but the logs don't tell us much:
```
==24060==AddressSanitizer: libc interceptors initialized
|| `[0x10007fff8000, 0x7fffffffffff]` || HighMem ||
|| `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
|| `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap ||
|| `[0x00007fff8000, 0x00008fff6fff]` || LowShadow ||
|| `[0x000000000000, 0x00007fff7fff]` || LowMem ||
MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
redzone=16
max_redzone=2048
quarantine_size_mb=256M
thread_local_quarantine_size_kb=1024K
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 0x7fff8000
==24060==Installed the sigaction for signal 11
==24060==Installed the sigaction for signal 7
==24060==Installed the sigaction for signal 8
==24060==T0: FakeStack created: 0x7f2a640b4000 -- 0x7f2a64bbd000 stack_size_log: 20; mmapped 11300K, noreserve=0
==24060==T0: stack [0x7fff12e4e000,0x7fff1364e000) size 0x800000; local=0x7fff1364d298
==24060==AddressSanitizer Init done
AddressSanitizer ignores mlock/mlockall/munlock/munlockall
==24208==Processing thread 24060.
==24208==Stack at 0x7fff12e4e000-0x7fff1364e000 (SP = 0x7fff1364c3f8).
==24208==TLS at 0x7f2a6a7e3600-0x7f2a6a7e46c0.
==24208==DTLS 18 at 0x4f80004920000010-0x5280004820000012.
Tracer caught signal 11: addr=0x0 pc=0x7f2a69e830e0 sp=0x7f2a4ae85cd0
==24060==LeakSanitizer has encountered a fatal error.
==24060==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==24060==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
```
Facebook's test infra is disabling many of EdenFS' tests. This has already caused regressions to land [1]. We need to get these tests passing and enabled ASAP.
The current hypothesis is that LSAN's leak detection is crashing. Disable the leak detection, hopefully making tests pass again. (We still want to find and fix the root cause, of course!)
[1] T37784916: test_rage_output is failing: AssertionError: 'General EdenFS Statistics' not found
Reviewed By: simpkins
Differential Revision: D13385022
fbshipit-source-id: c442146c39ce84c19fc53916aef421cece6d8b40
Summary:
edenfs's privhelper process needs the CAP_SYS_ADMIN capability [1] in order to manage mount points. Give it this capability by invoking edenfs as the `root` user using `sudo`.
Known issues:
* `sudo` must be passwordless in order for fb-edenfs@.service to start.
* systemd can't kill all of fb-edenfs@.service's processes, so `systemctl stop` is unreliable for example.
[1] https://manpage.me/index.cgi?q=capabilities&apropos=0&sektion=0&manpath=CentOS+7.1&arch=default&format=html
Reviewed By: chadaustin
Differential Revision: D13113450
fbshipit-source-id: 01b89521cab371b5017fab6fbd38d55eea599c46
Summary:
I want to reuse assert_systemd_service_is_active in another test. Move it (and the related assert_systemd_service_is_stopped function) so it can be reused.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13327802
fbshipit-source-id: 022c3ed3b9e8f04ef1156c2bb4b3deda662439e4
Summary: 'eden stop' should behave like 'systemctl stop'; the service should be stopped after 'eden stop' returns. Add a test which verifies this.
Reviewed By: chadaustin
Differential Revision: D13288819
fbshipit-source-id: 5b836c8ac7c5eb97c484195496f38c7cf70c84dc
Summary:
On Sandcastle's continuous builds, many EdenFS tests are flaky:
```
EdenCommandError: eden command [/mnt/btrfs/trunk-hg-fbcode-fbsource-303-1544100804/fbcode/buck-out/dev/gen/eden/cli/eden.par --config-dir /tmp/eden_test.udfpngq0/homedir/local/.eden --etc-eden-dir /tmp/eden_test.udfpngq0/etc-eden --home-dir /tmp/eden_test.udfpngq0/homedir repository main /tmp/eden_test.udfpngq0/repos/main] returned non-zero exit status 1
stderr=b'Tracer caught signal 11: addr=0x0 pc=0x7f65cea990e0 sp=0x7f65afb21cd0\n==1698412==LeakSanitizer has encountered a fatal error.\n==1698412==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1\n==1698412==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)\n'
```
I failed to reproduce this issue on my own dev server, and I failed to reproduce this issue by running 'buck test' on a Sandcastle worker.
As a last resort, make LSAN report more stuff in an effort to track down the cause of this failure.
Aside from extra logging, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13366696
fbshipit-source-id: e052700dbc4ed9d30f864b8d2dc5ccad27e1a281
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.
Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.
'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).
Reviewed By: simpkins
Differential Revision: D10849390
fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
Summary:
Stop holding a reference count to the TreeInode while a directory
handle is open. This allows eden to shut down while a directory handle
is open.
Reviewed By: strager
Differential Revision: D13287701
fbshipit-source-id: a24f32a1ac40b6c19bc5864aa5f5785f3016361b
Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary:
Add some skipped tests that illustrate that Eden cannot shut down when
file handles are open. They will be enabled later in this stack as the
root cause is fixed..
Reviewed By: strager
Differential Revision: D13287587
fbshipit-source-id: cd8d79896127676853c183bb4b86c0d586ce511e
Summary: I'm about to add some more integration tests for unmounting.
Reviewed By: strager
Differential Revision: D13279068
fbshipit-source-id: e84580497f22b9dc6d5a04835dc4beede52a07fd
Summary: CLI tests use shutil.rmtree to clean up temporary directories. Reuse TemporaryDirectoryMixin which is more robust against errors and supports EDEN_TEST_NO_CLEANUP.
Reviewed By: chadaustin
Differential Revision: D13268108
fbshipit-source-id: d77e95a2def0dceb34cf14e19c0c0c0e3aeef3f2
Summary:
systemd uses cgroups to contain a service's processes.
When EdenFS' tests run systemd on Sandcastle, the systemd process creates its cgroups as children of Sandcastle's cgroup. For example, the test-SystemdServiceTest.service service created in test_running_simple_service_is_active has the following cgroup on Sandcastle:
/sandcastle-job/command/test-SystemdServiceTest.service
Unfortunately, systemd uses predictable cgroup names based on the service name. A service managed by one systemd instance shares the same cgroup as a same-named service managed by a different systemd instance. For example, if multiple systemd tests run concurrently on Sandcastle (e.g. during stress-testing), `systemctl --user stop foo.service` from one run will kill the processes of a different run's active foo.service. This causes systemd tests to be flaky on Sandcastle.
Fix the conflict by creating a unique cgroup per systemd instance. For example, separate runs of test_running_simple_service_is_active might use the following cgroups for the test service:
/sandcastle-job/command/edenfs_test.1_qat2xv/test-SystemdServiceTest.service
/sandcastle-job/command/edenfs_test.bbd3gj_o/test-SystemdServiceTest.service
Reviewed By: simpkins
Differential Revision: D13016626
fbshipit-source-id: 8535dc14a06bdb403c926b111cad4aed6c8ec3e3
Summary:
Check that the next-inode-number file exists and that the inode number it
contains is actually larger than all existing inode numbers. Replace it with
correct data if the file does not exist, is corrupt, or contains an incorrect
inode number.
Reviewed By: chadaustin
Differential Revision: D12955093
fbshipit-source-id: 3d26fb475535577d9a2697bbd575fba350766d01
Summary:
Update fsck to extract data for orphan inodes to a lost+found directory in the
fsck log directory, and remove them from the overlay. This will allow users
to recover the orphan file data if they want, and remove it otherwise.
Reviewed By: chadaustin
Differential Revision: D12955094
fbshipit-source-id: 9783452fce4060b9c5c48b3d48dd1f70294211c6
Summary:
Add several more files to the basic snapshot, so we can test more cases
in the fsck tests:
- Materialized, new, and unmodified symlinks
- A deeper directory tree of directory inodes that are not materialized (still
have a source control tree hash) but have children inodes allocated and are
therefore present in the overlay.
- A socket in a slightly deeper directory so we can test behavior of sockets
inside directories that have been corrupted..
As before I have replaced the older basic snapshot instead of adding a new
one, since the Eden data storage formats have not changed since the last
snapshot was created.
Reviewed By: chadaustin
Differential Revision: D13164658
fbshipit-source-id: d117c9cc336709044de212637c03140dfadd9a96
Summary:
This updates the basic snapshot code to include a couple slightly deeper
directories. I plan to use this in the fsck tests to verify handling of
orphan directories that contain subdirectories.
Normally it would be preferable to keep the old `basic-20181030` snapshot, and
simply add this new snapshot without than replacing the old one. However, I
don't think we have made any meaningful changes to our on-disk storage formats
since the previous snapshot was generated, so it seems okay to just delete the
old snapshot.
Reviewed By: strager
Differential Revision: D13151861
fbshipit-source-id: e6b7583beecb5d9cc55271ad2dea8d36980542d1
Summary:
Add a helper class for maintaining the list of expected files.
I plan to use this for the fsck tests, so I can more easily modify the
snapshot's expected contents based on how we expect fsck to repair various
types of overlay corruption. This also helps slightly simplify the code that
constructs the expected file list.
Reviewed By: chadaustin
Differential Revision: D13095918
fbshipit-source-id: 57686e82d1bf7f23a92eda0ed4d66623a3f58840
Summary:
Add basic high level logic to fsck to begin fixing problems that are found.
This adds basic checks to decide if we should fix errors or not.
If errors are found and need to be fixed, this creates a new directory inside
the checkout state directory in `.eden` to record the actions taken by this
fsck run. This directory will contain a log file that records the actions
taken. In the future the fsck logic will also use this directory to store
copies of the corrupted inode data, and can store extracted orphan inode data
here as well.
Reviewed By: wez
Differential Revision: D12955044
fbshipit-source-id: 06c1e17a0a51fa5e2c0f2aab83b367b9358fd004
Summary:
After sending SIGTERM to systemd, increase the wait timeout from 3 seconds to
15 seconds. The previous 3 second timeout was easy to hit in practice on our
continuous build infrastructure.
If systemd does not exit within 15 seconds after we send SIGTERM, send SIGKILL
and then try waiting on it again for up to 3 seconds. Forcefully killing it
seems preferable to leaving the process hanging around after the tests exit.
Reviewed By: strager
Differential Revision: D13159491
fbshipit-source-id: debce21f2f202fb7cfa4a53120dcb2b2b35ccbe3
Summary:
Replace some member variables in
`_TransientUnmanagedSystemdUserServiceManager` with locals, to make pyre
happy.
Previously pyre complained that these member variables are not initialized in
`__init__()`. These variables are only needed temporarily during
`__enter__()` (and in some clean up closures), so just use them as local
variables instead.
Reviewed By: strager
Differential Revision: D13135257
fbshipit-source-id: 76f2bdc4b7b36d2102ad8dab4a60722a03197fab
Summary:
If invoked with `--log-target=console` systemd will log to stderr even if it
is not a tty.
This changes the tests to pass in `--log-target=console` rather than creating
a pty and forwarding I/O from it in a separate background thread.
Reviewed By: strager
Differential Revision: D13135258
fbshipit-source-id: 11dfe0711adaa62cedba2882045d8088e0df5499
Summary: test_processes_of_forking_service_includes_all_child_processes is failing on some machines. The test assumes /sys/fs/cgroup/ is a cgroups v2 mount point, so it fails if /sys/fs/cgroup/ is a directory containing cgroups v1 mount points. Disable the test on machines where cgroups v1 is detected.
Reviewed By: simpkins
Differential Revision: D13112836
fbshipit-source-id: 7921604707a0c1fe81a82c87e767a6a99cdd6206
Summary: Fix the remaining set of errors reported by pyre and mypy.
Reviewed By: strager
Differential Revision: D13086855
fbshipit-source-id: 4c2b21352f94ef225a5555aef0f6b95b92e56f6d
Summary:
Our code in testcase.py mucks around with test case classes in several ways
that mypy doesn't like. In particular, mypy does not currently support
replacing methods on classes, and it also does not understand dynamic base
classes. pyre also trips up on some of these changes, although in different
ways.
This updates testcase.py so that mypy and pyre no longer report errors on it,
often just by suppressing the errors. Also fix similar errors in
service_test_case.py around replicating test classes.
Reviewed By: strager
Differential Revision: D13086856
fbshipit-source-id: af446dd13791f5da50b09657012db95c2bcf0e39
Summary:
This updates the integration tests to add type annotations to most functions
that were missing annotations.
In particular this is needed to make pyre happy, as it complains if subclasses
override methods from their parent class and do not specify type annotations
if the parent class did have annotations.
This diff also contains some minor changes to hg_extension_test_base.py to
explicitly declare some abstract methods that it uses. This was also
necessary to make pyre happy about this ocde.
Reviewed By: strager
Differential Revision: D13051097
fbshipit-source-id: 77567ed2f4d3050f93acefb52e688932d276d587
Summary:
Drop the `stdout` and `stderr` arguments, so that this method always return a
string. Change callers that were previously calling this method with
`stderr=None` to use the `run_hg()` method instead of `hg()`. `run_hg()`
returns a `subprocess.CompletedProcess` object.
This change simplifies the python type checking, and fixes several existing
type checking errors in the code. Even though most call sites could be
guaranteed that this function would return a `str`, the type checker wasn't
smart enough to tell that the return type would be fixed based on the argument
values, and so it assumed the result always needed to be checked for `None`.
This also updates the `GitRepository.git()` method in a similar fashion.
However, that was a simpler change since it already returned a `str` in all
cases.
Reviewed By: strager
Differential Revision: D13078095
fbshipit-source-id: a8def2a33edc865ac40279bbcb3ada4dade68374
Summary:
Restructure storage_engine_test so that the base class derives from
EdenTestCase.
Reviewed By: strager
Differential Revision: D13051096
fbshipit-source-id: e89c4b56e361460b2457d1c2e6a22727a25d7646
Summary:
I want to use Hypothesis to fuzz some CLI code. Move EdenFS' Hypothesis configuration out of eden.integration.lib.testcase and into a place where both CLI tests and integration tests can use it.
This diff should not change behavior.
Reviewed By: wez
Differential Revision: D12813285
fbshipit-source-id: 3a1badd1e18b0e070295ea03dcb24be166cd42c1
Summary:
_transient_unmanaged_systemd_user_service_manager has a few inner functions and a few variables shared between these inner functions. The function is pretty long and hard to follow.
I think a class is more familiar than closures. Refactor _transient_unmanaged_systemd_user_service_manager into a class: turn inner functions into methods, and shared variables into instance attrs.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13017642
fbshipit-source-id: d20f7476142fa5d7ba0ae09291228ec63e127338
Summary:
If you enable type checking of util.py, mypy complains about _remove_readonly having the wrong type:
eden/integration/lib/util.py:87: error: Argument "onerror" to "rmtree" has incompatible type "Callable[[Callable[[str], Any], str, Tuple[Type[Any], BaseException, TracebackType]], None]"; expected "Optional[Callable[[Any, _PathLike[str], Any], Any]]"
Fix the error by making _remove_readonly compatible with onerror's signature in typeshed [1]:
overload
def rmtree(path: _AnyPath, ignore_errors: bool = ...,
onerror: Optional[Callable[[Any, _AnyPath, Any], Any]] = ...) -> None: ...
This diff should not change behavior.
[1] 4dc21f04dd/stdlib/2and3/shutil.pyi (L90-L92)
Reviewed By: simpkins
Differential Revision: D13086137
fbshipit-source-id: 222e5fa2e06a26464483a0f09545089a7ecc5234
Summary:
While debugging flakiness in test_running_simple_service_is_active, I noticed that sometimes ActiveState=active despite the service process being dead. This can happen if the service process is killed outside systemd. When this happens, SubState=exited instead of the expected SubState=running.
Prevent false positives in SystemdServiceTest tests by checking SubState in addition to ActiveState.
Reviewed By: simpkins
Differential Revision: D13032619
fbshipit-source-id: 2d5754291a19290d29a817115923e9cc5efc90ab
Summary:
A few times, I've needed to manually make _is_system_booted_with_systemd return False in order to emulate how Sandcastle behaves. Make this easier by introducing an environment variable, EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE, which allows choosing how to start `systemd --user` when running tests.
When EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE is unset, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13031404
fbshipit-source-id: ecbd5f90ff55f4dffa47ba797686db5c25a7198c
Summary:
Add type annotations for class member variables. The pyre type checker has
some limited automatic type detection for member variables set in
`__init__()`, but in general it expects member variables to be explicitly
declared at the top-level of the class.
Reviewed By: strager
Differential Revision: D13051092
fbshipit-source-id: 080259ab3f422ffae2b908ed610062237105ccbe
Summary:
Update `EdenRepoTest` to explicitly declare its own `get_thrift_client()` that
just calls through to its parent class' implementation.
I'm not sure why, but mypy can't seem to figure out the correct return type of
this method without this extra declaration. It gets confused when this method
is called from `EdenRepoTest` subclasses, and either complains that it doesn't
know the type at all and needs an explicit type declaration, or in some cases
it thinks the type is `Optional[EdenClient]` rather than `EdenClient`.
Currently mypy doesn't complain about these errors by default, but it seems to
complain when I start adding more type information for other class members.
Adding this extra declaration makes it possible to start adding type
annotations in more places in the code without triggering new errors from
mypy.
Reviewed By: strager
Differential Revision: D13051089
fbshipit-source-id: 69776642949d293c645686ba00d5d6b71c0cd31c
Summary:
The thrift APIs accept path names and commit IDs as binary data (python bytes)
rather than unicode strings. Our python code got this wrong in several
locations. It looks like mypy didn't previously flag this since mypy doesn't
actually figure out the correct type for the thrift `client` object, and
seemed to just be largely ignoring it. I plan to update the code so that mypy
can figure out out the client type correctly. Fixing these type errors is
required to make sure we won't get type errors once that is changed.
This just simply uses `encode("utf-8")` for now. In the future the path
arguments should be converted to `pathlib.Path`, which will do a slightly
smarter conversion, and avoid errors on non-UTF-8 binary data. In the
meantime, I believe that just using `encode("utf-8")` preserves what the
thrift code was doing implicitly before, and does not make handling of
non-UTF-8 data any worse than it was before.
Reviewed By: strager
Differential Revision: D13051094
fbshipit-source-id: 94cb62f3dd78b8e854a72a392fe8fdfad5ffd4cb
Summary:
eden/benchmarks/ is a good place to write external benchmarks for FUSE
and Thrift APIs.
(Note: this ignores all push blocking failures!)
Reviewed By: wez, strager
Differential Revision: D12970347
fbshipit-source-id: 81fa35897fc6e626254eea6e1ee44a8d35052261
Summary:
As we start to build out both FUSE and Thrift benchmarks, we'll want a
standard library. Introduce a benchharness and have both the thrift
sha-1 and parallel_open_close benchmarks use it.
(Note: this ignores all push blocking failures!)
Differential Revision: D12969306
fbshipit-source-id: 89c8bbcc37d53560decffb9281af4aba20345787
Summary:
Use `Path` objects and its associated APIs in most locations rather than using
plain strings.
Reviewed By: wez
Differential Revision: D13035415
fbshipit-source-id: ef3fb72e884b1f5dbc2a9bcd9f36ba1397a7c21a
Summary:
This removes the logic that would completely avoid reporting most of the
integration tests if we could not run Eden.
This used to be an issue in the past before we could run Eden in our CI build
environment. To avoid noise in the test reports with hundreds of tests
reported as skipped we would simply hide these tests entirely. Our CI
environment does run all the Eden tests now, so this is no longer an issue.
I'm removing this logic now since it confuses the pyre type checker. Rather
than modifying the code to work around the problem, it seems preferable to
just remove this logic entirely. Now that the CI environment should be
running all the tests it seems like we probably would want to be notified if
something goes wrong and the tests think they cannot successfully run Eden for
some reason.
Reviewed By: wez
Differential Revision: D13035412
fbshipit-source-id: 73c0ce7fa759a1fbb9aff08ccad475b36f332646
Summary:
My changes to command line argument parsing in D12927803 conflicted with one
of the new tests added in D10863987.
My changes made `edenfs` and `fake_edenfs` reject any trailing non-option
arguments. The new test runs `fake_edenfs` with an extra
`--commandArgumentsLogFile` argument asking it to log all non-option arguments
to a file.
This diff updates `fake_edenfs` to allow non-option arguments when
`--commandArgumentsLogFile` is specified.
Reviewed By: strager
Differential Revision: D13014079
fbshipit-source-id: 82ec5fd758716c2b66ac0738b7aacdf884f31233
Summary:
Improve the way `eden fsck` processes its arguments. Previously it required a
single checkout path as an argument. If an `--overlay` argument was specified
it used this instead of the checkout path to find the directory to check, but
the checkout path argument was still required (but was ignored).
This changes the code to accept one or more paths as arguments. These paths
can either be the path to an Eden checkout (which does not have be currently
mounted), or the path to a checkout state directory inside the `.eden` state
directory. The `fsck` code automatically figures out what type of directory
the argument refers to and processes it correctly.
If no paths are specified `eden fsck` now automatically checks all configured
checkouts that are not currently mounted.
Reviewed By: wez
Differential Revision: D12955041
fbshipit-source-id: c37bc6752746d8ecd0f4a672d0d3b25d1c3a4fa1
Summary:
This is the basic framework for some new tests for `eden fsck` that work by
unpacking one of the existing saved snapshot files, breaking it in various
ways, and then running `eden fsck` to fix it.
Using an existing snapshot file rather than creating a new mount point on the
fly in the test has a few advantages:
- This lets us confirm that newer versions of Eden can still correctly repair
and mount old file formats, even if we update the mount point data formats
in the future.
- Unpacking a snapshot is much faster than starting Eden, creating an hg
repository, cloning a new Eden checkout from the repo, and then unmounting
the checkout.
- The inode number allocations for the snapshot are fixed, which makes it
easier for the test code to manipulate specific inode numbers and always
know which path this refers to. If we created a mount point on the fly we
can't guarantee ahead of time which inode numbers would map to each file,
and we would need to do more work to look up this information after creating
the checkout.
These tests are pretty basic at the moment, but I plan to expand them as I
check in more of the fsck logic to repair errors.
Reviewed By: wez
Differential Revision: D12955045
fbshipit-source-id: 5d5a96cec812f8e72caf93e57bf0f1311e28aab8
Summary:
Sometimes users accidentally run `edenfs start` or `edenfs restart` instead of
`eden start` or `eden restart`. This adds a new `--edenfs` flag to the
`edenfs` binary, and asks users if they meant to run `eden` instead if they do
not pass in this flag.
This used to be less of a problem since `edenfs` required users to also
explicitly specify several other configuration flags (like `--edenDir`).
However `edenfs` can not automatically figure out these settings, so these
flags are no longer required. Therefore `edenfs` would still try to start
normally when invoked with `edenfs restart`, since it did not require these
flags and it did not complain about unhandled command line arguments.
Reviewed By: wez
Differential Revision: D12927803
fbshipit-source-id: dbf7ce2449c391ca218652439eb68ff43c2ebd46
Summary:
When unpacking a snapshot, rewrite Eden's inode metadata table to change the
UID and GID values to the current user.
This is needed so that the current user can access files inside the mounted
checkout correctly.
Reviewed By: wez
Differential Revision: D12966640
fbshipit-source-id: eec4aba690117bf7b8f944221b31b7c7cc66fc0c
Summary:
Add a script for unpacking saved snapshot files.
This unpacks the tarball and then fixes up absolute paths inside the snapshot
so it will work at the unpacked location. It also emits a small helper script
to make it easier to invoke eden for the snapshot.
Reviewed By: wez
Differential Revision: D12927842
fbshipit-source-id: f46c07a79894877842f2a78aa0924fd66e4c8969
Summary:
Thanks to some bpf tracing by strager, we traced the ESTALE response to
`d_splice_alias` and noted this comment above the implementation in the kernel:
> If a non-IS_ROOT directory is found, the filesystem is corrupt, and
> we should error out: directories can't have multiple aliases.
Well, our magic `.eden` directory is a directory with aliases and we were
seeing the error trigger on that dir. So, this diff replaces hardlinking
directories into each tree with a hardlink to a symlink in each tree!
At mount time we create `.eden/this-dir` as a symlink to `/abs/path/to/mount/.eden`
so that `readlink("/abs/path/to/mount/sub/dir/.eden/socket")` still
resolves as it did prior to this diff.
Reviewed By: strager
Differential Revision: D12954819
fbshipit-source-id: 7f3b1b53f2bd5b9c51e64055fc34110657a19110
Summary:
When giving arguments to the edenfs program, 'eden start' only strips `--` if it is the first positional argument. The position of `--` shouldn't matter as long (as it's before any options).
In other words, given the following command invocation:
$ eden start --opt-a arg-b -- --opt-c arg-d
Current behavior:
* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b -- --opt-c arg-d` is passed to edenfs as four arguments.
Desired behavior:
* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b --opt-c arg-d` is passed to edenfs as three arguments.
* The `--` argument is not passed to edenfs.
Fix 'eden start' by stripping `--` regardless of its position.
Reviewed By: chadaustin
Differential Revision: D10863987
fbshipit-source-id: 094da3f3674e775fe3e7eb8441ec95c37c34ff05
Summary:
Sandcastle has several cases where we chown the entire
repository which performs terribly on Eden. As a workaround we have a
command to do this in eden without loading all the files.
Reviewed By: chadaustin
Differential Revision: D12857956
fbshipit-source-id: 36cebcc710fbcf4e1eb265df901513cf50a227b9
Summary:
The `test_hg_clone_non_eden_repo_within_eden_repo()` test had been disabled
when running with treemanifest enabled since the treemanifest code did not
correctly detect which repositories should actually use treemanifest.
This issue appears to have since been fixed in the treemanifest code, so this
test passes now.
Reviewed By: quark-zju
Differential Revision: D12927801
fbshipit-source-id: b9e2e041f7eab5e24007888e2dba142e1f0b2251
Summary:
- Update mypy in fbcode to upstream version 0.641.
- Add/fix some type stubs
- Disabled some pieces that break mypy (T35976411 and T35973384)
This diff will likely cause some unit test failures, due to incorrect type annotations that should be fixed by the team owning the relevant code.
mathchecksout
Reviewed By: jeffmo
Differential Revision: D10414250
fbshipit-source-id: 282ea62aef6793a2f4baef78afd58ddd4ddf60f5
Summary: This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D12873644
fbshipit-source-id: c8eaac8bee56268b8a6479f7ea27e496a6b12d96
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Create utilities for tests to enable units within a temporary instance instance, start services, and inspect the processes inside a running service.
These utilities are currently unused, but will be used in future diffs as 'eden start', etc. grows systemd support.
Reviewed By: simpkins
Differential Revision: D10371824
fbshipit-source-id: c1e62eebb8480309ee17c49c08af73c3f5cf75d6
Summary: While creating a systemd service unit for EdenFS, I noticed a few features of the current implementation of 'eden start', 'eden stop', and 'eden restart' which are not covered by tests. Write tests for these features to ensure they don't regress with systemd integration.
Reviewed By: wez
Differential Revision: D10434379
fbshipit-source-id: 6c828a85d59179bbc4beda87e1bf2534543b60b2
Summary:
Update the snapshot framework code to add tests that mount saved snapshots
with the current edenfs build and verifies that they behave as expected.
This also changes the snapshot generation code a fair amount as well, in
order to be able to verify the snapshot data:
- We now more cleanly split out data that should be saved when the snapshot is
first generated (such as the `.eden` state directory) and data that should
be re-generated each time the snapshot is unpacked (such as the `/etc/eden`
config directory and the system configuration for `hg`).
- The code is now capable of rewriting absolute paths in the Eden state files
so it is usable in a new location after it has been unpacked.
This also updates the "basic" snapshot type to create a wider variety of types
of files and file changes, and to be able to verify the contents of a resumed
snapshot.
Reviewed By: strager
Differential Revision: D9955411
fbshipit-source-id: 76012c01016cf4d89dc611c5596a05f5e7f013ed
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Create a utility for tests to create a temporary systemd instance. It should work regardless of whether the host uses systemd to manage its services. (systemd's tooling must still be installed in order to use the utility, though.)
This utility is currently unused, but will be used in future diffs as 'eden start', etc. grows systemd support.
Reviewed By: simpkins
Differential Revision: D10286940
fbshipit-source-id: 4fbaa695bf36ac4ae44b5c12b6255514bd7143b3
Summary: spawn_status already sets the log file to stderr. Setting it again is redundant. Delete the dead store.
Reviewed By: simpkins
Differential Revision: D10434381
fbshipit-source-id: d0e4cfe9dc63ae80fc8a9e621b02a3ac31f8244a
Summary:
Since we can't seem to find a way to prevent ESTALE
from being returned by the kernel, and that only seems to occur
with readlink, add a retry to our readlink calls.
This feels gross, but not sure how we can do better :/
Reviewed By: strager
Differential Revision: D12239143
fbshipit-source-id: 04ecbdeac3b3ea3969e779da4f50cf91ccc5cd00
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Unfortunately, we can't create a new instance of systemd if a machine is not managed by systemd. However, we can trick systemd into thinking that the machine *is* managed by systemd. When we do that, the systemd instance seems to work without any issues.
Create a library which, when injected into systemd using LD_PRELOAD, makes systemd thinks the machine is managed by systemd.
The library will be used in a subsequent diff (D10286940). This diff is separate so it's easier to review.
Reviewed By: simpkins
Differential Revision: D10371831
fbshipit-source-id: b54b704dc631ebdc5091320dd6b680ca255f3ba2
Summary:
When running fake_edenfs in daemonize mode (i.e. when --foreground is not specified), the daemon process inherits stdout and stderr from the parent and never closes them. This has two consequences:
* The daemon process holds a reference to stdout/stderr even after the parent exits. This causes tests to hang if they slurp stdout/stderr after waiting for the parent process to exit. (I discovered this issue when adding tests for 'eden start' in D10434379.)
* fake_edenfs logs to stdout/stderr after 'eden start' has returned. I think this behavior is surprising, but it doesn't really hurt anyone because fake_edenfs is just a developer tool.
Promptly close stdout and stderr after startup finishes by redirecting further output to a log file. This mimics the behavior of the real edenfs.
Note: stdin is still leaked. stdin will be addressed in D10442346.
Reviewed By: chadaustin
Differential Revision: D10439299
fbshipit-source-id: a7a423aa901982f3d35e81940ffc27db3a985d76
Summary:
No one uses --logPath with fake_edenfs. Remove the dead flag, inlining its default value.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439298
fbshipit-source-id: 068e5677db1bec81a8da174b136ef9f5b54c7e34
Summary:
The daemonizeIfRequested is implemented in the main edenfs process as well as in fake_edenfs. Consolidate the almost-identical implementations.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439300
fbshipit-source-id: 6253668896de0f8393b92f4aa5ee8a8a2fcbfb31
Summary:
Right now, 'eden start' spawns edenfs directly (using `os.exec`). In the future, 'eden start' will spawn edenfs using systemd. 'eden restart', 'eden stop', and 'eden status' might behave differently depending on how the edenfs process was spawned.
Parameterize some tests on how fake_edenfs was originally started. This flexibility lets us easily spawn fake_edenfs with 'systemctl start' (systemd) in the future. For now, spawn fake_edenfs in ad-hoc mode (i.e. directly without 'eden start') or in managed mode (i.e. indirectly with 'eden start').
Reviewed By: chadaustin
Differential Revision: D10414995
fbshipit-source-id: 2ee0d5df8a605ca9d7da8f6eeca1fc171a8342e8
Summary:
RestartTest uses 'eden restart' during setup to start the fake_edenfs process. There are two issues with doing this:
* I want to extend the tests in RestartTest so they also tests with an ad-hoc fake_edenfs process (D10414995). Restarting edenfs during setup makes no sense in that context.
* I think using 'eden restart' during setup is clunky and unintuitive. 'eden start' is more appropriate.
Refactor RestartTest's setup so it uses 'eden start' instead. Also add a test which explicitly verifies that 'eden restart' starts fake_edenfs if no daemon is already running.
Reviewed By: chadaustin
Differential Revision: D10417304
fbshipit-source-id: 20f8190026cd153dd9b539067f6f63b6bd27abed
Summary:
I want a function like `fake_eden_daemon` which instead spawns fake_edenfs using `eden start`. Refactor existing code to make such a function easier to write:
* Replace `contextlib.contextmanager` with a class called `FakeEdenFS` so its `__exit__` method can be reused.
* Rename `fake_eden_daemon` to `FakeEdenFS.spawn` so it's clearer that the function creates a process.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10415123
fbshipit-source-id: 81e6a0618214a527dc7f15cc18d4fe97dd1a957e
Summary:
Update the HgRepository.status() function in the integration tests to return
the status information as a dictionary instead of a string. Only one test was
still using the old string API.
Reviewed By: chadaustin
Differential Revision: D10503168
fbshipit-source-id: 574e4438d23bf6612a70ae5ae3174db3d464d198
Summary:
Update the eden+hg integration test code so that it uses find_executables.py
to find the post clone hook. Previously it was still using its own one-off
helper function for this.
Moving this code into find_executables.py will make it possible have the
snapshot tool also find the post clone hook.
Reviewed By: chadaustin
Differential Revision: D10503169
fbshipit-source-id: d59e3e517d19e80e0d0c997be09865519b80d34e
Summary:
Now that the import bug has been fixed for some time, it's likely few
people have cached empty files. And if they do, `eden gc; eden
restart` is a fine workaround.
On the other hand, reimporting empty files gums up the importer, and
I've seen several people recently complaining about performance that
could be partially attributed to the fact that their Eden needed to
verify empty files. simpkins has talked about adding a bit to the
cached blob to determine whether it needs reimporting or not, but it's
probably going to take a while for anyone to implement that, and this
reverification logic is hurting people today.
Reviewed By: strager
Differential Revision: D10456519
fbshipit-source-id: 657bc377ee16ce93494075bde4388aed59dceecf
Summary:
When you get an error, let's print the number of conflicts in each file. This will give the user some sense of how much work they have to do.
The code change is entirely in `filemerge.py`, and `tests/test-merge-conflict-count.t` adds a new test.
Reviewed By: quark-zju
Differential Revision: D9815243
fbshipit-source-id: 1b73a1db293902ac7242997a7d6ae09478344068
Summary:
Add a new sparse show command, which shows the current sparse configuration.
Use common templating, including colors, between sparse show and sparse list.
Reviewed By: phillco
Differential Revision: D10446705
fbshipit-source-id: acb539f80f625945716758c785703b46a81aa6f1
Summary: Some commands in extensions have incorrectly defined usage. Correct these.
Reviewed By: phillco
Differential Revision: D10446707
fbshipit-source-id: e606314892a31fa09f4a655042d66d87929fc2ec
Summary: D10416965 accidentally removed the import for `typing` in health_test.py. This causes tests to fail. Re-add the import to fix the tests.
Reviewed By: singhsrb
Differential Revision: D10452517
fbshipit-source-id: af58d076b7bac375c20630113cbb3972e24ffc12
Summary:
Eden supports reading the SHA-1 of a file via getxattr. Unfortunately,
it returned ENOSYS if you called getxattr on a directory inode. This
caused FUSE to fail all future getxattr requests with EOPNOTSUPP.
In addition to fixing that, this diff makes our xattr handling a
little more consistent across inodes:
- setxattr() always fails with ENOSYS
- removexattr() always fails with ENOSYS
- listxattr() is always handled by the corresponding inode class
- getxattr() is always handled by the corresponding inode class
Differential Revision: D10437723
fbshipit-source-id: a1ea1e92d3412abb15e91057becea3160a17f1e2
Summary: Instead of using tempfile.TemporaryDirectory and calling unittest.TestCase.addCleanup manually, use TemporaryDirectoryMixin which does a better job.
Reviewed By: chadaustin
Differential Revision: D10416965
fbshipit-source-id: aed9d80788d54692bc7f024a5a62523ca168bbf5
Summary:
StartupLogger handles two independent scenarios: edenfs in daemon mode, and edenfs in foreground mode. I need to add a third for systemd support: edenfs in foreground mode but with startup logs written to a file.
Refactor the existing StartupLogger into an abstract base class with two concrete derived classes: DaemonStartupLogger and ForegroundStartupLogger. This will make adding the systemd feature easy without confusing the logic for the existing scenarios.
Aside from dropping the setsid() call for ForegroundStartupLogger, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D10228156
fbshipit-source-id: fbc2c771c6fbb675779ef05ca3b7395b70b67485
Summary: D10335083 renamed `fbsparse` into `sparse` and I forgot to udpate this test.
Reviewed By: simpkins
Differential Revision: D10423192
fbshipit-source-id: a547c403a9461d6923ecfecc931df82bb7887597
Summary:
Make StopTest and HealthOfFakeEdenFSTest similar to RestartTest by putting the code to invoke 'eden --config-dir ... stop-or-status' into its own function.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10414421
fbshipit-source-id: e42cdee64ae282fa0575cf875b53ac91fa61d490
Summary:
To ease the migration from the current custom daemon setup to a systemd-managed daemon, add a way to enable the to-be-implemented systemd behavior.
* To the edenfs daemon, support the --experimentalSystemd option.
* To the Eden CLI, support the EDEN_EXPERIMENTAL_SYSTEMD environment variable. When set, invoke the edenfs daemon with --experimentalSystemd.
Aside from adding a log message, this diff does not change any behavior. This diff just sets up scaffolding for future diffs.
Reviewed By: simpkins
Differential Revision: D10248055
fbshipit-source-id: 30ca5a4bfde00ff43fd6f2a5d0282ced4f177fed
Summary:
Logging some messages to stdout and some messages to stderr has a few problems:
* stdout and stderr can get interleaved. For pipes, interleaving is avoidable for messages smaller than PIPE_BUF (512 bytes), but there's no guarantee that stdout and stderr are pipes or that the reader of the pipe reads PIPE_BUF bytes at a time.
* For the systemd integration I am developing, I need one file for stderr logs and another file for stdout logs. This is more complicated than having a single file for all logs.
* The distinction between stdout and stderr is unclear. No messages from StartupLogger are meant to be piped to another program.
Address these problems by only logging to stderr.
From a user's point of view, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10370931
fbshipit-source-id: b063ed3a66cefc846d30ce1486759cbd3f3ea29e
Summary:
Note that this is more aggressive than the tweakdefaults version; it doesn't allow named branches to be created or listed at all with the config set. As we want to delete named branches entirely, this seems reasonable.
I ported the `--new` flag, but made it optional, to make it easier to support both the core and tweakdefaults callers.
Saurabh Singh and I chatted with the releng team (Craig).
**tldr:** we can remove named branches from the UI but should keep the innards for now
Today the releng team doesn't make any branches (#continuous), but in the near-term future, they might either:
- create remote bookmarks in hg (similar to fbsource)
- create branches in SVN (which are then synced as hg branches by hgsubversion)
- We'll leave all the hgsubversion code to do this alone, just remove user-facing entry points
- They'll also need to be able to query for a commit's branch (`hg log -T '{branch}'` will still work)
- Once www is on hg, this possibility goes away and we can kill named branches for good
Reviewed By: ikostia
Differential Revision: D10401485
fbshipit-source-id: 6f2f3ae28af249bae1fdf782eb14fe7bfc472bb7
Summary:
In another diff, I want to set an environment variable within a test and restore its original value later. Code to do this exists already inside setup_eden_test, but it's hard-coded to HOME. Generalize the code and factor it into its own class so it can be used in my other diff.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D10282151
fbshipit-source-id: 063ed980889441e7b693cf7f1c84f5039b2cbd64
Summary:
In Python 3, `pexpect.spawn` objects come in two flavors: bytes and unicode. The flavor is selected using `pexpect.spawn.__init__`'s `encoding` parameter.
For the bytes flavor, `pexpect.spawn`'s instance attributes generally use `bytes`:
class spawn:
args: Sequence[bytes]
before: bytes
command: str
For the unicode flavor, `pexpect.spawn`'s instance attributes generally use `str`:
class spawn:
args: Sequence[str]
before: str
command: str
Teach the type checker (mypy) about these two flavors to reduce false positives in calling code.
Reviewed By: simpkins
Differential Revision: D10280499
fbshipit-source-id: b4475226423ef37ff5bf2bbae0eb9deac6a1099a
Summary:
RestartTest._spawn_restart dynamically types its kwargs parameter, hiding potential type errors. The kwargs parameter is unused anyway, so delete it and tidy up the related code.
This diff should not change behavior.
Reviewed By: pkaush
Differential Revision: D10281769
fbshipit-source-id: 3e7f7078edf9e47013a9eb9ba42cf51fa8dd0a16
Summary:
Update hg absorb to use the templater and display the first line of the
description of the affected commits.
Also change the default behaviour to display the changes and prompt the user to
confirm that absorb is going to do what they want. Use the new `-a` option to
get the old behaviour.
Reviewed By: quark-zju
Differential Revision: D10366510
fbshipit-source-id: 23df2e5ab2d21a0805a16e7118686a29634d2918
Summary:
Update `EdenFS.cleanup()` to only delete its `eden` directory if it was
created as temporary directory in `__init__`. If the caller specified an
explicit directory avoid deleting it.
All of the users that currently create `EdenFS` with an explicit directory
argument already perform their own clean-up of this directory if needed.
Therefore this diff does not meaningfully change the behavior of existing
programs, it simply make the API behavior less surprising.
Reviewed By: strager
Differential Revision: D10351822
fbshipit-source-id: e9ba7f5869bd2cc221a84be45df233acc7a1a660
Summary:
We go through a lot of effort to set EDENFS_FAKE_EDENFS for tests and verify its path. However, we never actually use the path! We always fall through to searching candidate paths, ignoring the value of the environment variable.
Add an early `return` so the environment variable is respected.
Reviewed By: simpkins
Differential Revision: D10369153
fbshipit-source-id: f1767a240361c6c7e1c95f93c6284a4ac95c2645
Summary: This seems to yield about 18% space savings compared to bzip2.
Reviewed By: strager
Differential Revision: D10351855
fbshipit-source-id: 58b969a8df4a9eb075526d5cc919c99ff6e8f4f3
Summary: Doctor now detects if multiple edenfs are running for the same edenDir and recommends to kill the rogue one.
Reviewed By: strager
Differential Revision: D10098157
fbshipit-source-id: 8a785a9fee0c768696f897a4950552d141012b2a
Summary:
The functionality we care about is provided by the `fbamend`
extension. Therefore, lets replace the `amend` extension with the `fbamend`
extension.
Reviewed By: farnz
Differential Revision: D10320739
fbshipit-source-id: 5700d39f488777fcc4033f60ce0a51cda15ef2ad
Summary: Several tests create a temporary directory and destroy it. Consolidate the temporary directories and their cleanup code to reduce code duplication.
Reviewed By: simpkins
Differential Revision: D10251110
fbshipit-source-id: caa5b5ad49fcb7925f63094594af4a20009d790d
Summary:
We've been shipping our RPMs with toml config support turned on for a
while now. Remove support for the old config file format. Continue to
ship the old format configurations. We'll remove those in a later
diff.
Reviewed By: strager
Differential Revision: D10020958
fbshipit-source-id: 11c2ca3b5da086b142042496a2814699880c4f81
Summary:
The logic we care about has been moved to core so we can delete the
`inhibit` extension.
Reviewed By: markbt
Differential Revision: D10276448
fbshipit-source-id: 44cabe5d561344cf8f196127ce52491e4654b598
Summary:
If 'eden stop' tries to tell edenfs to cleanly shut down, and edenfs never responds, 'eden stop' just hangs. Make 'eden stop' time out CLI -> edenfs Thrift calls. After a timeout, 'eden stop' should by stop edenfs via SIGKILL.
Note: This diff does not affect 'eden restart'.
Reviewed By: chadaustin
Differential Revision: D9982372
fbshipit-source-id: 42e498fb93fd97a2ec963e97c25f4b4101c7d524
Summary:
pexpect prints program output while the program is running, which is very useful when debugging. Make the tests in stop_test.py use this feature.
Also, add some assertions to some of the tests.
Reviewed By: chadaustin
Differential Revision: D10157354
fbshipit-source-id: 210fcde79d5d21c6ee34fad3813e7c56eb298b9b
Summary: If edenfs is unresponsive, 'eden status' hangs forever. Add a timeout to turn the hang into a user-friendly error.
Reviewed By: chadaustin
Differential Revision: D10156229
fbshipit-source-id: 9186826ae6b131a193b1499c8baac616d131357f
Summary:
I want to test more conditions for 'eden stop', such as the edenfs process taking a long time to respond to the Thrift stop request. Refactor how fake_edenfs decides to handle stop requests to make it easy to add new behavior.
Aside from changing the log messages, this diff should not change behavior.
Reviewed By: wez
Differential Revision: D10019459
fbshipit-source-id: f6d5f514a67435a446d8d672bf7d21af77d76c56
Summary: HypothesisSimpleTest.test_create is timing out on Sandcastle (Facebook's CI). It looks like the hypothesis tests run indefinitely on Sandcastle. Set a time limit so the tests complete and provide signal.
Reviewed By: wez
Differential Revision: D10053452
fbshipit-source-id: 8c8cb2c375a73b7727e827ea2a9e7d56ab390e54
Summary:
'eden stop' has poor test coverage. Write tests for the common case and some edge cases which 'eden stop' already handles.
This diff should not change behavior.
Reviewed By: wez
Differential Revision: D10017819
fbshipit-source-id: 4d9f5db52187c34c62a9379a6b3dd62f62894233
Summary:
We're looking into doing some performance optimizations in Eden to the
open() and close() calls, since they are never handled by the kernel
and always pass through to the underlying user-space FUSE daemon. But
first, let's capture some baseline numbers on Eden and non-Eden
checkouts.
Reviewed By: strager
Differential Revision: D10036941
fbshipit-source-id: 64f3414a4572fd963017491db37d70e6b5ae4f24
Summary: D6612669 introduced a workaround for edenfs not running as a child of sudo if you run Eden's integration tests as root. D9980225 fixed the underlying bug in 'eden stop' which necessitated the workaround. Remove the now-unnecessary workaround.
Reviewed By: wez
Differential Revision: D9984524
fbshipit-source-id: b9ed7114cd7385a13899d16b4da0d40e6e9fd704
Summary:
This begins adding a framework for generating snapshots of Eden state for use
in tests. I plan to add a few kinds of tests based on these snapshots:
- Tests that ensure Eden can successfully load snapshots created by older
versions of the code.
- Tests that corrupt the snapshot data in various ways and then confirm that
it can be repaired by `eden fsck` and/or still loaded successfully by
`edenfs` even if it was not repaired with `fsck` first.
This code still needs some additional functionality, but I figured it was
worth checking in what I have so far. The main functionality that remains to
be added is unpacking the snapshots and updating absolute paths inside the
config files so that they work in the new path where they were unpacked.
Reviewed By: strager
Differential Revision: D9690267
fbshipit-source-id: a2660e49b84d7833e6778108d9abe081ab7e2cbd
Summary:
Move the `_cleanup_tmp_dir()` function out of the integration lib's testcase
module, and make it a public function in the integration lib util module.
This will make it easier to re-use from some other python utilities that need
to be able to clean up Eden state directories.
Reviewed By: strager
Differential Revision: D9690266
fbshipit-source-id: 157c5221f81f90a3e9612315681459d16693cdbc
Summary:
Update the Eden unit tests and integration tests to set the `NOSCMLOG`
environment variable when running `hg` commands. This ensures that our
mercurial telemetry wrapper does not log events from Eden test runs.
Reviewed By: quark-zju, strager
Differential Revision: D9893096
fbshipit-source-id: c0dd4b5eb042dcb5e9493c89aaee10a513022bae
Summary:
If the overlay file for a directory is corrupted (e.g. empty), Overlay::scanForNextInodeNumber throws. This causes Eden to crash on start [1]. Fix the crash by ignoring corrupted directories.
[1] `test_mount_possible_after_corrupt_directory_and_cached_next_inode_number` reproduces this crash.
Reviewed By: chadaustin
Differential Revision: D9806105
fbshipit-source-id: 1b95083b6a6aa253a2296d6f754edbf4b9f64734
Summary:
The C++ code disallows use of "." in RelativePaths, but it's
reasonable for users of the Thrift API to pass "." to indicate the
root of the mount. Handle that in the EdenServiceHandler.
Reviewed By: strager
Differential Revision: D9647776
fbshipit-source-id: b61c2d1c0dcd69ccfa38bf27379281d10cdf1ceb
Summary:
Fix the code in `eden_dirstate._call_match_callbacks()` to correctly match
ignored symlinks that are explicitly listed in the matcher's file list.
It looks like this was just a mistake in the original code on my part: I
intended to check both `S_ISREG()` and `S_ISLNK()` but instead incorrectly put
the check for `S_ISREG()` twice.
Reviewed By: wez
Differential Revision: D9476524
fbshipit-source-id: 67e0fa7c2fbaac97598a8e2d028c9ef0999ed88a
Summary:
When we load an empty blob from the LocalStore double check with the
BackingStore to confirm that it should actually be empty.
We have seen multiple instances of files being incorrectly imported as empty.
So far this error has always been fixed by a re-import. We still haven't
tracked down the root cause, but this change should help workaround the issue
by ensuring that we double check the file contents before returning the data.
Reviewed By: chadaustin
Differential Revision: D9476522
fbshipit-source-id: 6d57cf15c42736ecbcb106a731259b77db06d8f1
Summary:
Update all of the `eden debug` commands to use the newer `find_checkout()`
code rather than older `get_mount_path()` function.
The `find_checkout()` code makes sure that the EdenInstance actually points to
the correct edenfs instance for this checkout, and also works with checkouts
that are not currently mounted. In particular this allows
`eden debug overlay` to examine the overlay state even when the checkout is
not currently mounted.
Reviewed By: wez
Differential Revision: D9389865
fbshipit-source-id: 00578519d4805157a30c9b39abee9838925e8e76
Summary:
Watchman's Eden integration has a bug where the combination of
Watchman querying Eden for overlapping delta ranges ("give me changes
between X and Y, now changes between X+1 and Y+1") and Eden eliding
redundant change events ("add-modify-remove" -> []) results in
Watchman sometimes reporting that a file exists in its final
subscription update when it no longer does.
The fix is to never elide events, even for files that were added and
removed in the same sequence. To continue to support Watchman's `new`
flag, track whether a file existed at the beginning and end of a
journal delta.
Reviewed By: wez
Differential Revision: D9304964
fbshipit-source-id: f34c12b25f2b24e3a0d46fc94aa428528f4c5098
Summary:
To prevent the regression fixed by D8323051, make sure we run at least a couple
integration tests with the RocksDB LocalStore implementation.
Reviewed By: wez
Differential Revision: D8408390
fbshipit-source-id: 8fab4041ae39915d8be80e42814aab375c4acdda
Summary:
This prevents `hg status` from blowing up with a UTF-8 decode
error inside the generated thrift code.
Push safety concerns:
* This doesn't change the wire representation of the data
* Existing clients that believe it to be a string will continue to have
the same behavior
* Buck has its own copy of an older version of the thrift spec, so it will
continue to work "OK".
* When buck resyncs with our thrift file, some changes will likely be needed
to convert the byte arrays to strings or paths or whatever is appropriate
for bucks internal API
Work "OK" above means that clients that currently believe that `string` is
utf-8 encoded will have a runtime error if we ever send them a path that
is not utf-8. This is the behavior prior to this diff and will continue
to be the behavior for clients (like buck) that have an older version
of the thrift file.
Reviewed By: simpkins
Differential Revision: D9270843
fbshipit-source-id: b01135aec9152aaf5199e1c654ddd7f61c03717e
Summary:
Add some integration tests that invoked "eden doctor" and "eden list" inside a
stale mount point, and confirms that they work correctly. These commands do
fail in dev builds, since python itself does not handle this error.
Fortunately XAR-based python archives avoids this error, so these commands
work in opt mode builds.
Reviewed By: strager
Differential Revision: D8905489
fbshipit-source-id: 0666f8cb0715d61697253bee9fe0bde2f4a3fa15
Summary:
Update `InodeMap::updateOverlayForUnload()` to catch exceptions that occur
trying to save state to the overlay. If something goes wrong when saving
state there is not much we can do other than log an error. We still want to
unload the inode, and code that is unloading inodes generally cannot deal with
exceptions at this point. In particular if this error occurred while trying
to shut down an EdenMount the code would crash.
Reviewed By: chadaustin
Differential Revision: D8884795
fbshipit-source-id: c2f850f13d775be4b0a0a10f9df3948c7b2c8f4a
Summary:
Encountering a truncated overlay file doesn't necessarily indicate a software
bug in Eden. Depending on the underlying filesystem this often happens after
a hard system reboot since we write the overlay files without an `fdatasync()`
call.
Change the code to simply log an error and throw an exception rather than
using `EDEN_BUG()`. This makes it possible to exercise this code path in
tests without having it crash in debug builds.
Reviewed By: chadaustin
Differential Revision: D8988209
fbshipit-source-id: 8c0fe1dae692f4c493413d3939d2e4c21e0da596
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
Summary:
Fix the arguments to `doctor.cure_what_ails_you()`, which were updated in
D8870494.
Reviewed By: chadaustin
Differential Revision: D8940413
fbshipit-source-id: 21d45036f668c9bae9a2002c360532918eb157a5
Summary:
This returns a `Repository` object and not a `Repository` subclass type.
This type information has been broken since D7512320 replaced the
`get_repo_class()` method with `create_repo()`.
Reviewed By: strager
Differential Revision: D8940415
fbshipit-source-id: 7711a2a33bc12fc361dfd0ff3f47eba1e587e6e7
Summary:
Update `EdenFS.run_unchecked()` to return the full
`subprocess.CompletedProcess` argument rather than just the return code.
This will make it possible for future tests to check the output of the command
in addition to just its return code.
Reviewed By: strager
Differential Revision: D8905491
fbshipit-source-id: 7f17f5d103a06aeb56a3be0618683bd1c017fdf0
Summary:
D8508489 updated the integration test code so that the `capture_output`
argument to `EdenFS.run_cmd()` was no longer necessary and was not used.
However I forgot to actually remove the argument in that diff. This deletes
the argument.
Reviewed By: strager
Differential Revision: D8905492
fbshipit-source-id: d7a691f7d628a9fd5daa6723575e4454480f6aa1
Summary:
As of D8631281 `eden clone` now makes the mount point directory read-only and
puts a readme file inside of it. Since the directory is read-only the
integration tests are unable to clean it up using a normal `shutil.rmtree()`
call.
This fixes the integration tests to try making the directory writable first
if it encounters an error during cleanup.
Reviewed By: strager
Differential Revision: D8905488
fbshipit-source-id: c6b9d0b12c2b1dcb4a0839757cbcc3aad179bfc1
Summary:
This adds initial code for an `eden debug fsck` command.
At the moment this simply scans the overlay and reports errors, but does not
correct any issues. In subsequent diffs I'll update it to support correcting
some problems.
There are also several other issues that I'll improve in subsequent diffs as
well. At the moment this code uses the same functions as other CLI commands
to find the client directory, and these only work if the checkout is already
mounted. For `fsck` we generally want the checkout to be unmounted before
checking it. I also plan to clean up some of the output handling as well.
Reviewed By: strager
Differential Revision: D8813714
fbshipit-source-id: 90238e94540db1cc06c17eabf8ce3a1bb808d4b3
Summary:
This makes it easier to add some test coverage.
There's no real functional change in this diff; the only code change is to
throw a system_error instead of a thrift eden error wrapper class from the core
globbing code. There's a little bit of code to restore this exception type in
the callers in EdenServiceHandler; this is covered by existing integration
tests, but I've also expanded that coverage to cover both variants of the glob
thrift calls.
Reviewed By: strager
Differential Revision: D8776767
fbshipit-source-id: 3ea4ea642ae5108aa4b0153541bd3604f010b54c
Summary:
Update `eden restart` to explain that the restart will be disruptive when
performing a full restart, and prompt the user for confirmation. This prompt
can be overridden with the `--force` option. Also print more messaging after
the restart completes telling users what to do if they see "Transport endpoint
not connected" errors from programs still using the old mount points.
This also updates `eden restart` to more carefully handle cases where there is
an existing edenfs daemon running but it is not fully healthy.
Reviewed By: wez
Differential Revision: D8731004
fbshipit-source-id: 05762187b47057b2930d0a6b71b0a6fdbd4aa7e5
Summary:
This moves the fsattr helper program into the helpers/ directory
created in D8373672.
Reviewed By: chadaustin
Differential Revision: D8705391
fbshipit-source-id: 43e04d8b06b5d17caefafdfc750e3e60cad1b0cc
Summary:
Update "eden clone" to create a README file inside the mount point directory
before the checkout is mounted. When Eden is running normally this file will
not be visible because the Eden checkout has been mounted on top of this
directory. However if the checkout is not currently mounted users will see
this file instead.
This should hopefully make it easier for users to figure out how to remount
their checkout on their own, and should prevent them from worrying that their
files have been deleted when the mount point simply is not mounted.
Reviewed By: chadaustin, wez
Differential Revision: D8631281
fbshipit-source-id: 7610450983687a839bdbc3558ff35e0422edc3be
Summary:
The eden integration tests currently run hg through the telemetry wrapper.
This wrapper script adds profiling arguments to a percentage of the commands
it runs. The extra arguments that it adds are not entirely transparent to the
user, and are unfortunately reported in the output of "hg journal".
This was causing occasional test failures in the Eden integration tests that
check the output of "hg journal".
This diff changes the Eden code that checks the journal output to try to strip
off these flags before checking the command output.
Reviewed By: wez
Differential Revision: D8631180
fbshipit-source-id: 004d3bcce291ffc2208f85a29cb34a921f16fdde
Summary:
Update the python CLI code to let the C++ edenfs code handle daemonization
rather than daemonizing in the python code.
This means that we now daemonize after the `sudo` invocation rather than
before it. This allow's sudo's wrapper process to exit once edenfs starts,
rather than remaining around for the lifetime of the edenfs daemon. This
means that the original process' stdout and stderr file descriptors are
closed properly rather than being held open by the sudo wrapper.
This also removes the logic in the CLI that waits for edenfs startup and can
potentially time out and give up.
Reviewed By: strager
Differential Revision: D8508489
fbshipit-source-id: 6a38439d5596049d6f40cb6589b6865c229f736e
Summary:
This updates edenfs to be able to daemonize itself, and moves the
daemonization logic from the python CLI code into C++.
The main benefit of this is that we can now do a better job of reporting
messages to the user during start-up. We can log around potentially slow
operations (opening the RocksDB local store), and we can print messages
directly to the user if startup fails. Previously most failure messages would
go only to the eden log and would not be printed to the user's terminal.
This also fixes some issues where stdin and stdout were not closed properly
when daemonization was performed by the CLI. sudo needed access to these file
descriptors in case it needed to prompt for a password, and it would then hold
the descriptors open until edenfs exited.
Reviewed By: wez
Differential Revision: D8373672
fbshipit-source-id: 3272bff2208596f41d26e479c82c700d6c1efe11
Summary:
Previously `eden rm` tried to look at `<path>/.eden/root` to find the root of
the mount point. This failed if the checkout was not currently mounted.
This updates the code to try looking for the exact input path in the config
file in this case. It also avoids sending an `unmount()` thrift call to
edenfs in this case as well.
Reviewed By: strager
Differential Revision: D8558765
fbshipit-source-id: a5d9db7eee702817713b089a457f25710438661e
Summary:
When running the integration tests in parallel in a dev mode build I
occasionally see timeouts in the shutdown code in some tests. (Usually this
seems to happen in some of the remount tests that create multiple mount points
and shutdown and restart edenfs multiple times.)
Usually when I see this error edenfs does exit successfully on its own just
after this 15 second timeout has expired.
This bumps up the timeout to 30 seconds to avoid spurious failures.
Reviewed By: pkaush
Differential Revision: D8559052
fbshipit-source-id: 567062b08b34eced8f05538607ae9f4773c4aa9c
Summary:
Previously `eden list` reported normally running active checkouts with an
`(active)` suffix, and non-mounted checkouts with no suffix.
The non-mounted situation is rare. This diff changes the command to report
non-mounted checkouts with a `(not mounted)` suffix. I removed the `(active)`
suffix from normally-running checkouts now, but let me know if people think it
is still valuable to keep this annotation in the output.
Reviewed By: chadaustin
Differential Revision: D8558770
fbshipit-source-id: ccce0f7c2031e50f2847089e49007274c31a69a7
Summary:
I saw RemountTest.test_restart_twice() fail once when running locally, but I
haven't been able to reproduce the failure yet. This updates the checks in
this code to hopefully provide better information about what went wrong in
case we see this failure again in the future. This now checks the `eden list`
output first so if something goes wrong we will hopefully have more
information about which checkouts were or weren't remounted.
Reviewed By: strager
Differential Revision: D8528544
fbshipit-source-id: 4f50145aa97956f45f50813070fb73836bc7acde
Summary:
We've seen what appears to be phantom calls to shutdown() so we'd like
to add some degree of auditing. This diff adds a new method with some
context; this will allow us to distinguish between `eden stop`, `eden restart`
and eden server internal calls to the `shutdown` method. It may still
be possible that something else is calling our shutdown method, but it
seems unlikely as we're only accessible to our own code via a unix domain
socket.
Reviewed By: chadaustin
Differential Revision: D8341595
fbshipit-source-id: 50d58ea0b56e5f42cd37c404048d710bde0d13a3
Summary:
D7946842 moved some of the daemon management code from util.py to daemon.py
but accidentally broke handling of TimeoutError exceptions. The code in
daemon.py was trying to catch python's built-in `TimeoutError` type, while the
code in util.py throws its own `util.TimeoutError` type.
This removes the custom `TimeoutError` type in util.py and just uses the
built-in python type to be consistent everywhere.
Reviewed By: wez
Differential Revision: D8373739
fbshipit-source-id: 8523c444fef74b0758c38b4fe2644d4f68d38442
Summary:
mkdtemp deliberately removes these bits and doesn't respect
the umask. We perhaps should respect the umask ourselves here.
Reviewed By: chadaustin
Differential Revision: D8335539
fbshipit-source-id: 98b8f3e2db17c65904b12627e52966d5f6b729ef
Summary:
Some folks might attempt to load this extension despite it
not being useful with eden. The extension knows not to hook in to
the repo when eden is in used, but the various commands to manipulate
the sparse config do not.
This diff adds a little helper function to show a more reasonable
error message than an ugly stack trace when attempting to use them.
Reviewed By: simpkins
Differential Revision: D8328229
fbshipit-source-id: d7cc4c7047b11f8da96e26ab70bd522b52546151
Summary:
Fix the integration tests to set CHG_BIN correctly when run manually rather
than via `buck test`.
Reviewed By: wez, quark-zju
Differential Revision: D8329593
fbshipit-source-id: 79fa3c9e9538e4b30a98ef777515527562e1ac1b
Summary:
This fixes a bug simpkins pointed out in D6891479 - we weren't
updating mtime and ctime on renames.
Reviewed By: simpkins
Differential Revision: D7937303
fbshipit-source-id: 08fd8f4fe5d99d33e9f7629965d6146330c8f35b
Summary:
I kept running into issues trying to get graceful restart and
flush_cache to work together in the hg integration suite, so add a
test to ensure flush_cache succeeds after a graceful restart in the
main integration suite.
Also, to make the test's output easier to follow, add logging when
invalidating inodes.
Reviewed By: simpkins
Differential Revision: D8215961
fbshipit-source-id: 33db4292af3969ae23940c3027ba513ed20c53fb
Summary:
Add a command line option to start edenfs only if there is at least one
Eden checkout already configured.
This makes it easier to automatically start edenfs on system boot only if the
user needs edenfs.
Reviewed By: wez
Differential Revision: D8174819
fbshipit-source-id: bf585b8c5aa719d3b6424567f2e21c1c63850d66
Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.
Reviewed By: bolinfest
Differential Revision: D8108649
fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
Summary:
The Overlay is the natural home for nextInodeNumber_ now that every
directory allocates inode numbers for its children right away. This
also simplifies serializing nextInodeNumber_ to disk in the following
diff.
Reviewed By: simpkins
Differential Revision: D8192442
fbshipit-source-id: 9b776a73c8d7653002b55985d592b1746e52f878
Summary:
Add a new integration test that modifies a file, commits it, then restores it
to its original contents and creates a new commit with the revert.
This test isn't really exercising any Eden-specific behavior. The only reason
it is is interesting is that this happened to trigger a bug in the underlying
mercurial cdatapack code that caused the test to fail when run in treemanifest
mode. The mercurial cdatapack bug is fixed in D8131020
Reviewed By: quark-zju
Differential Revision: D7826284
fbshipit-source-id: 32da2db04452d5df0527f6be00b0c4c1a56c900b
Summary:
Update find_executables.py to be able to find the HG wrapper and binary using
buck rules. This is necessary in order to be able to run the integration
tests manually rather than via `buck test`. Without this the tests try to use
`/bin/hg.real` which tests the wrong version of the mercurial code and the
eden extension.
Reviewed By: bolinfest
Differential Revision: D8200168
fbshipit-source-id: d2a68c765c7143bbe613f32eef1268ed567790a7
