Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary:
Now that the Overlay no longer serializes timestamps, remove all of
the special-case migration logic.
Reviewed By: strager
Differential Revision: D13144764
fbshipit-source-id: 713a4bfcde9003a8d5a28837cb530b05a9017c22
Summary:
systemd uses cgroups to contain a service's processes.
When EdenFS' tests run systemd on Sandcastle, the systemd process creates its cgroups as children of Sandcastle's cgroup. For example, the test-SystemdServiceTest.service service created in test_running_simple_service_is_active has the following cgroup on Sandcastle:
/sandcastle-job/command/test-SystemdServiceTest.service
Unfortunately, systemd uses predictable cgroup names based on the service name. A service managed by one systemd instance shares the same cgroup as a same-named service managed by a different systemd instance. For example, if multiple systemd tests run concurrently on Sandcastle (e.g. during stress-testing), `systemctl --user stop foo.service` from one run will kill the processes of a different run's active foo.service. This causes systemd tests to be flaky on Sandcastle.
Fix the conflict by creating a unique cgroup per systemd instance. For example, separate runs of test_running_simple_service_is_active might use the following cgroups for the test service:
/sandcastle-job/command/edenfs_test.1_qat2xv/test-SystemdServiceTest.service
/sandcastle-job/command/edenfs_test.bbd3gj_o/test-SystemdServiceTest.service
Reviewed By: simpkins
Differential Revision: D13016626
fbshipit-source-id: 8535dc14a06bdb403c926b111cad4aed6c8ec3e3
Summary:
Check that the next-inode-number file exists and that the inode number it
contains is actually larger than all existing inode numbers. Replace it with
correct data if the file does not exist, is corrupt, or contains an incorrect
inode number.
Reviewed By: chadaustin
Differential Revision: D12955093
fbshipit-source-id: 3d26fb475535577d9a2697bbd575fba350766d01
Summary:
Update fsck to extract data for orphan inodes to a lost+found directory in the
fsck log directory, and remove them from the overlay. This will allow users
to recover the orphan file data if they want, and remove it otherwise.
Reviewed By: chadaustin
Differential Revision: D12955094
fbshipit-source-id: 9783452fce4060b9c5c48b3d48dd1f70294211c6
Summary:
The linter tripped on D12813838 even though the warning was unrelated
to my changes.
Reviewed By: simpkins
Differential Revision: D13167184
fbshipit-source-id: 555691f00d113ed2bff9488b61392cc92f4395e3
Summary:
Add several more files to the basic snapshot, so we can test more cases
in the fsck tests:
- Materialized, new, and unmodified symlinks
- A deeper directory tree of directory inodes that are not materialized (still
have a source control tree hash) but have children inodes allocated and are
therefore present in the overlay.
- A socket in a slightly deeper directory so we can test behavior of sockets
inside directories that have been corrupted..
As before I have replaced the older basic snapshot instead of adding a new
one, since the Eden data storage formats have not changed since the last
snapshot was created.
Reviewed By: chadaustin
Differential Revision: D13164658
fbshipit-source-id: d117c9cc336709044de212637c03140dfadd9a96
Summary:
This updates the basic snapshot code to include a couple slightly deeper
directories. I plan to use this in the fsck tests to verify handling of
orphan directories that contain subdirectories.
Normally it would be preferable to keep the old `basic-20181030` snapshot, and
simply add this new snapshot without than replacing the old one. However, I
don't think we have made any meaningful changes to our on-disk storage formats
since the previous snapshot was generated, so it seems okay to just delete the
old snapshot.
Reviewed By: strager
Differential Revision: D13151861
fbshipit-source-id: e6b7583beecb5d9cc55271ad2dea8d36980542d1
Summary:
Add a helper class for maintaining the list of expected files.
I plan to use this for the fsck tests, so I can more easily modify the
snapshot's expected contents based on how we expect fsck to repair various
types of overlay corruption. This also helps slightly simplify the code that
constructs the expected file list.
Reviewed By: chadaustin
Differential Revision: D13095918
fbshipit-source-id: 57686e82d1bf7f23a92eda0ed4d66623a3f58840
Summary:
Add basic high level logic to fsck to begin fixing problems that are found.
This adds basic checks to decide if we should fix errors or not.
If errors are found and need to be fixed, this creates a new directory inside
the checkout state directory in `.eden` to record the actions taken by this
fsck run. This directory will contain a log file that records the actions
taken. In the future the fsck logic will also use this directory to store
copies of the corrupted inode data, and can store extracted orphan inode data
here as well.
Reviewed By: wez
Differential Revision: D12955044
fbshipit-source-id: 06c1e17a0a51fa5e2c0f2aab83b367b9358fd004
Summary:
Eden has used the InodeMetadataTable as the authoritative source for
timestamp metadata for over six months. I think we can safely assume
that anyone at this point who has old inodes in the overlay that don't
have corresponding entries in the inode metadata table are fine with
those timestamps being reset to the last checkout time.
Reviewed By: strager
Differential Revision: D13144735
fbshipit-source-id: 06a9a8835ea83c98fb6a165e4c8d5c3c6b28ad84
Summary:
Eden has used the InodeMetadataTable as the primary source of
timestamp data for more than six months. Stop writing timestamps into
the overlay, since they will never be used.
Reviewed By: strager
Differential Revision: D13144696
fbshipit-source-id: e36423036228e89dd2a986e6bacfa74553c17a92
Summary:
After sending SIGTERM to systemd, increase the wait timeout from 3 seconds to
15 seconds. The previous 3 second timeout was easy to hit in practice on our
continuous build infrastructure.
If systemd does not exit within 15 seconds after we send SIGTERM, send SIGKILL
and then try waiting on it again for up to 3 seconds. Forcefully killing it
seems preferable to leaving the process hanging around after the tests exit.
Reviewed By: strager
Differential Revision: D13159491
fbshipit-source-id: debce21f2f202fb7cfa4a53120dcb2b2b35ccbe3
Summary: Instantiate a blob cache in the EdenServer with configurable settings.
Reviewed By: strager
Differential Revision: D12813880
fbshipit-source-id: 8ccc89826f04aca78964230374dea48abf05e05e
Summary: In support of making Eden's file access stateless, add a facade that connects the BlobCache and ObjectStore, allowing FileInode to fetch blobs, minimizing reloads and memory usage.
Reviewed By: strager
Differential Revision: D10850143
fbshipit-source-id: 4307f7c1143aecad1284ea3cadf3e4efca9a3925
Summary: Change the Windows pipe read and write functions to read/write in a loop. Plus changed the functions prototype to match the POSIX version.
Reviewed By: strager
Differential Revision: D13091785
fbshipit-source-id: 375b22bd9e62d371a78d410f42068945b966a743
Summary:
Backing store works with eden strings(UTF8 + Unix path separator). The path strings we receive on Windows from FS and cli are Windows paths
(Win path separator and/or UTF16). Adding the functions to convert one to another.
Reviewed By: strager
Differential Revision: D13091788
fbshipit-source-id: f7fc8a79e360e964cf4619dfa540b57f1f18d283
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary: Add a BlobCache with a maximum cache size and a minimum entry count and interest-based eviction.
Reviewed By: strager
Differential Revision: D12972062
fbshipit-source-id: 1958f7f500c051a5bc0b39b5b89a6f0fc1774b0f
Summary: Create a platform independent function to compare the stats and use that to check if the file contents have changed.
Reviewed By: strager
Differential Revision: D13046269
fbshipit-source-id: c4f5bc88cec3df5cb6555d13cea2f23fd4eeb7ce
Summary: Add a pyre configuration file to allow it to check all Eden python files.
Reviewed By: strager
Differential Revision: D13087241
fbshipit-source-id: efd3f05dc465f8e58271ce46a97aaaf6d5f0a12a
Summary:
Replace some member variables in
`_TransientUnmanagedSystemdUserServiceManager` with locals, to make pyre
happy.
Previously pyre complained that these member variables are not initialized in
`__init__()`. These variables are only needed temporarily during
`__enter__()` (and in some clean up closures), so just use them as local
variables instead.
Reviewed By: strager
Differential Revision: D13135257
fbshipit-source-id: 76f2bdc4b7b36d2102ad8dab4a60722a03197fab
Summary:
If invoked with `--log-target=console` systemd will log to stderr even if it
is not a tty.
This changes the tests to pass in `--log-target=console` rather than creating
a pty and forwarding I/O from it in a separate background thread.
Reviewed By: strager
Differential Revision: D13135258
fbshipit-source-id: 11dfe0711adaa62cedba2882045d8088e0df5499
Summary:
Update the CLI's overlay parsing code to make sure the `entries` field of
returned `OverlayDir` objects is always set, even if the serialized thrift
data on disk does not have data for this field.
Apparently "default" thrift fields (neither optional nor required) behave
differently in python than in C++. In C++ these fields are always populated
by deserialization: if they are not present in the serialized data a default
value is used. In Python it looks like the thrift code simply leaves the
value unmodified (which usually means `None`). This changes the code to
ensure that this value will default to an empty dictionary if it is not
present.
Reviewed By: chadaustin
Differential Revision: D13135378
fbshipit-source-id: ce5254922e6fc88d6f3d8c58778ad9d45f02741f
Summary: test_processes_of_forking_service_includes_all_child_processes is failing on some machines. The test assumes /sys/fs/cgroup/ is a cgroups v2 mount point, so it fails if /sys/fs/cgroup/ is a directory containing cgroups v1 mount points. Disable the test on machines where cgroups v1 is detected.
Reviewed By: simpkins
Differential Revision: D13112836
fbshipit-source-id: 7921604707a0c1fe81a82c87e767a6a99cdd6206
Summary:
[Thrift] Cut most of the remaining Thrift SASL integration. Legacy Thrift SASL is replaced by TLS.
(Note: this ignores all push blocking failures!)
Reviewed By: vitaut
Differential Revision: D12954177
fbshipit-source-id: 42909632f4232301cdeeda465848d7e9700d3074
Summary: Fix the remaining set of errors reported by pyre and mypy.
Reviewed By: strager
Differential Revision: D13086855
fbshipit-source-id: 4c2b21352f94ef225a5555aef0f6b95b92e56f6d
Summary:
Our code in testcase.py mucks around with test case classes in several ways
that mypy doesn't like. In particular, mypy does not currently support
replacing methods on classes, and it also does not understand dynamic base
classes. pyre also trips up on some of these changes, although in different
ways.
This updates testcase.py so that mypy and pyre no longer report errors on it,
often just by suppressing the errors. Also fix similar errors in
service_test_case.py around replicating test classes.
Reviewed By: strager
Differential Revision: D13086856
fbshipit-source-id: af446dd13791f5da50b09657012db95c2bcf0e39
Summary:
This updates the integration tests to add type annotations to most functions
that were missing annotations.
In particular this is needed to make pyre happy, as it complains if subclasses
override methods from their parent class and do not specify type annotations
if the parent class did have annotations.
This diff also contains some minor changes to hg_extension_test_base.py to
explicitly declare some abstract methods that it uses. This was also
necessary to make pyre happy about this ocde.
Reviewed By: strager
Differential Revision: D13051097
fbshipit-source-id: 77567ed2f4d3050f93acefb52e688932d276d587
Summary:
Drop the `stdout` and `stderr` arguments, so that this method always return a
string. Change callers that were previously calling this method with
`stderr=None` to use the `run_hg()` method instead of `hg()`. `run_hg()`
returns a `subprocess.CompletedProcess` object.
This change simplifies the python type checking, and fixes several existing
type checking errors in the code. Even though most call sites could be
guaranteed that this function would return a `str`, the type checker wasn't
smart enough to tell that the return type would be fixed based on the argument
values, and so it assumed the result always needed to be checked for `None`.
This also updates the `GitRepository.git()` method in a similar fashion.
However, that was a simpler change since it already returned a `str` in all
cases.
Reviewed By: strager
Differential Revision: D13078095
fbshipit-source-id: a8def2a33edc865ac40279bbcb3ada4dade68374
Summary:
Restructure storage_engine_test so that the base class derives from
EdenTestCase.
Reviewed By: strager
Differential Revision: D13051096
fbshipit-source-id: e89c4b56e361460b2457d1c2e6a22727a25d7646
Summary:
EdenFS' systemd service will be configurable using config files. The Eden CLI will communicate information, such as extra daemon arguments, to systemd using a file specified using EnvironmentFile= [1]. Write code to serialize and deserialize these files.
The code introduced by this diff isn't used yet, but it will be used in future diffs.
[1] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#EnvironmentFile=
Reviewed By: chadaustin
Differential Revision: D12066543
fbshipit-source-id: 28ba38d823b92f7a3a7ad97c416120a7f95be122
Summary:
I want to use Hypothesis to fuzz some CLI code. Move EdenFS' Hypothesis configuration out of eden.integration.lib.testcase and into a place where both CLI tests and integration tests can use it.
This diff should not change behavior.
Reviewed By: wez
Differential Revision: D12813285
fbshipit-source-id: 3a1badd1e18b0e070295ea03dcb24be166cd42c1
Summary:
_transient_unmanaged_systemd_user_service_manager has a few inner functions and a few variables shared between these inner functions. The function is pretty long and hard to follow.
I think a class is more familiar than closures. Refactor _transient_unmanaged_systemd_user_service_manager into a class: turn inner functions into methods, and shared variables into instance attrs.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13017642
fbshipit-source-id: d20f7476142fa5d7ba0ae09291228ec63e127338
Summary:
If you enable type checking of util.py, mypy complains about _remove_readonly having the wrong type:
eden/integration/lib/util.py:87: error: Argument "onerror" to "rmtree" has incompatible type "Callable[[Callable[[str], Any], str, Tuple[Type[Any], BaseException, TracebackType]], None]"; expected "Optional[Callable[[Any, _PathLike[str], Any], Any]]"
Fix the error by making _remove_readonly compatible with onerror's signature in typeshed [1]:
overload
def rmtree(path: _AnyPath, ignore_errors: bool = ...,
onerror: Optional[Callable[[Any, _AnyPath, Any], Any]] = ...) -> None: ...
This diff should not change behavior.
[1] 4dc21f04dd/stdlib/2and3/shutil.pyi (L90-L92)
Reviewed By: simpkins
Differential Revision: D13086137
fbshipit-source-id: 222e5fa2e06a26464483a0f09545089a7ecc5234
Summary:
While debugging flakiness in test_running_simple_service_is_active, I noticed that sometimes ActiveState=active despite the service process being dead. This can happen if the service process is killed outside systemd. When this happens, SubState=exited instead of the expected SubState=running.
Prevent false positives in SystemdServiceTest tests by checking SubState in addition to ActiveState.
Reviewed By: simpkins
Differential Revision: D13032619
fbshipit-source-id: 2d5754291a19290d29a817115923e9cc5efc90ab
Summary:
A few times, I've needed to manually make _is_system_booted_with_systemd return False in order to emulate how Sandcastle behaves. Make this easier by introducing an environment variable, EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE, which allows choosing how to start `systemd --user` when running tests.
When EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE is unset, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13031404
fbshipit-source-id: ecbd5f90ff55f4dffa47ba797686db5c25a7198c
Summary: Add type annotations to all functions in eden/fs/service/client.py
Reviewed By: strager
Differential Revision: D13051091
fbshipit-source-id: 23b93008352664336ad155a7f5cc281bd5529702
Summary:
Add type annotations for class member variables. The pyre type checker has
some limited automatic type detection for member variables set in
`__init__()`, but in general it expects member variables to be explicitly
declared at the top-level of the class.
Reviewed By: strager
Differential Revision: D13051092
fbshipit-source-id: 080259ab3f422ffae2b908ed610062237105ccbe
Summary:
Update `EdenRepoTest` to explicitly declare its own `get_thrift_client()` that
just calls through to its parent class' implementation.
I'm not sure why, but mypy can't seem to figure out the correct return type of
this method without this extra declaration. It gets confused when this method
is called from `EdenRepoTest` subclasses, and either complains that it doesn't
know the type at all and needs an explicit type declaration, or in some cases
it thinks the type is `Optional[EdenClient]` rather than `EdenClient`.
Currently mypy doesn't complain about these errors by default, but it seems to
complain when I start adding more type information for other class members.
Adding this extra declaration makes it possible to start adding type
annotations in more places in the code without triggering new errors from
mypy.
Reviewed By: strager
Differential Revision: D13051089
fbshipit-source-id: 69776642949d293c645686ba00d5d6b71c0cd31c
Summary:
The fb303 setOption() call does not return a value. Delete the code that was
assigning the result to a variable. This variable was not used, and was
always overridden below with the result from a `getOption()` call.
Reviewed By: strager
Differential Revision: D13051093
fbshipit-source-id: e86c47d56424abaa6b89fb2335b868b0eddc2c43
Summary:
The thrift APIs accept path names and commit IDs as binary data (python bytes)
rather than unicode strings. Our python code got this wrong in several
locations. It looks like mypy didn't previously flag this since mypy doesn't
actually figure out the correct type for the thrift `client` object, and
seemed to just be largely ignoring it. I plan to update the code so that mypy
can figure out out the client type correctly. Fixing these type errors is
required to make sure we won't get type errors once that is changed.
This just simply uses `encode("utf-8")` for now. In the future the path
arguments should be converted to `pathlib.Path`, which will do a slightly
smarter conversion, and avoid errors on non-UTF-8 binary data. In the
meantime, I believe that just using `encode("utf-8")` preserves what the
thrift code was doing implicitly before, and does not make handling of
non-UTF-8 data any worse than it was before.
Reviewed By: strager
Differential Revision: D13051094
fbshipit-source-id: 94cb62f3dd78b8e854a72a392fe8fdfad5ffd4cb
Summary:
Now that we have a standardized StartingGate for benchmarks, use it
everywhere we used to use folly::Baton.
(Note: this ignores all push blocking failures!)
Differential Revision: D13012244
fbshipit-source-id: 5841ab74cfa408e87d021fe5591557e79e677e5c
Summary:
eden/benchmarks/ is a good place to write external benchmarks for FUSE
and Thrift APIs.
(Note: this ignores all push blocking failures!)
Reviewed By: wez, strager
Differential Revision: D12970347
fbshipit-source-id: 81fa35897fc6e626254eea6e1ee44a8d35052261
Summary:
As we start to build out both FUSE and Thrift benchmarks, we'll want a
standard library. Introduce a benchharness and have both the thrift
sha-1 and parallel_open_close benchmarks use it.
(Note: this ignores all push blocking failures!)
Differential Revision: D12969306
fbshipit-source-id: 89c8bbcc37d53560decffb9281af4aba20345787
Summary:
Use `Path` objects and its associated APIs in most locations rather than using
plain strings.
Reviewed By: wez
Differential Revision: D13035415
fbshipit-source-id: ef3fb72e884b1f5dbc2a9bcd9f36ba1397a7c21a
