Summary:
Write tests for readdir's semantics (we really do want to return . and
..) and simplify the logic. It's still quadratic in large directories,
but there aren't any allocations anymore.
Reviewed By: strager
Differential Revision: D13287764
fbshipit-source-id: 5e0d4b86eb16dbd7a16cdeb324e4b43363512e25
Summary: On my machine, post-clone is installed into `/etc/eden/hooks/post-clone`. There is no `/etc/eden/hooks/hg/` directory. Update README.md to refer to the right path.
Reviewed By: chadaustin
Differential Revision: D13314084
fbshipit-source-id: 8fe8110487e706bc30d1a43501bdc2784a46b5eb
Summary: This diff also includes build script changes to EdenDispatcher, FsChannel and EdenMount.
Reviewed By: strager
Differential Revision: D13091786
fbshipit-source-id: cecc8d849fcb9ebc8fa718e1011ef8931bebc279
Summary:
Stop holding a reference count to the TreeInode while a directory
handle is open. This allows eden to shut down while a directory handle
is open.
Reviewed By: strager
Differential Revision: D13287701
fbshipit-source-id: a24f32a1ac40b6c19bc5864aa5f5785f3016361b
Summary:
Send readdir requests to TreeInode. This may not sound like a good
idea: the FUSE documentation suggests that stateful directory handles
are required to implement correct readdir semantics under concurrent
deletes and renames. However, the 63-bit offset value is treated as a
cookie that is passed from one readdir call into the next, and 63 bits
should be sufficient to implement readdir concurrent with
rename/unlink. So move readdir's implementation into TreeInode in
preparation for the complete removal of TreeInodeDirHandle.
Reviewed By: strager
Differential Revision: D13287664
fbshipit-source-id: c0d615675edd9b83353534468a69b89068bba923
Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary:
Send fsyncdir straight through the inode rather than going through
DirHandle. This is the better design anyway, since the DirHandle does
not receive directory-mutating requests like mkdir.
Reviewed By: strager
Differential Revision: D13287610
fbshipit-source-id: 154fa32a3877c89a204a2d10b4e2b637410d9486
Summary:
Add some skipped tests that illustrate that Eden cannot shut down when
file handles are open. They will be enabled later in this stack as the
root cause is fixed..
Reviewed By: strager
Differential Revision: D13287587
fbshipit-source-id: cd8d79896127676853c183bb4b86c0d586ce511e
Summary: I'm about to add some more integration tests for unmounting.
Reviewed By: strager
Differential Revision: D13279068
fbshipit-source-id: e84580497f22b9dc6d5a04835dc4beede52a07fd
Summary: CLI tests use shutil.rmtree to clean up temporary directories. Reuse TemporaryDirectoryMixin which is more robust against errors and supports EDEN_TEST_NO_CLEANUP.
Reviewed By: chadaustin
Differential Revision: D13268108
fbshipit-source-id: d77e95a2def0dceb34cf14e19c0c0c0e3aeef3f2
Summary:
Add a `__str__()` method to the `HardLinkedInode` error class, so it prints
useful information in the fsck logs. Previously it just printed the default
python `repr()` information, which includes the class name but not the inode
number and path details.
Reviewed By: strager
Differential Revision: D13165857
fbshipit-source-id: 68c7484ae65127653a48eb71ec0cd1be3fe8b85f
Summary: No need for String to import fbvector.
Reviewed By: yfeldblum
Differential Revision: D13007715
fbshipit-source-id: c61639a04273f14dd3daf230ee6ed9ade93d058e
Summary:
FUSE_NO_OPEN_SUPPORT is better than ATOMIC_O_TRUNC for Eden's use
case. Remove the code that pretended we might support ATOMIC_O_TRUNC
again someday.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13163382
fbshipit-source-id: 948d701571a8d2977da3d2532fdc9538c5011636
Summary:
It's not clear that this code is a win and either way it will be a
no-op when FUSE_NO_OPEN_SUPPORT is enabled so just remove the prefetch
in open().
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13162205
fbshipit-source-id: a3161c0d042e13bd092fc9589e851be78552fa7a
Summary:
FileInode no longer has a strong reference to a blob. Instead, all accesses go through the blob cache. This diff changes the caching behavior for blobs.
The previous behavior was:
When a file's contents are needed in any way, the blob is loaded and referenced by the inode. When the number of open file handles goes to zero, the blob is dropped. The blob is also dropped when the inode is unloaded. Future inode loads or open requests, in that situation, require the blob to be reloaded from the LocalStore.
The new behavior is:
When a file's contents are needed, the blob is loaded and stored into the BlobCache, evicting any if necessary. Future blob requests are satisfied from the BlobCache, pushing it to the back of the eviction queue. When the inode is materialized or unloaded, the blob will be evicted from cache if no other blob has interest in it.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D12813912
fbshipit-source-id: 20d20807d2e4a6c37cddab38562fdb7456316aac
Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary:
Now that the Overlay no longer serializes timestamps, remove all of
the special-case migration logic.
Reviewed By: strager
Differential Revision: D13144764
fbshipit-source-id: 713a4bfcde9003a8d5a28837cb530b05a9017c22
Summary:
systemd uses cgroups to contain a service's processes.
When EdenFS' tests run systemd on Sandcastle, the systemd process creates its cgroups as children of Sandcastle's cgroup. For example, the test-SystemdServiceTest.service service created in test_running_simple_service_is_active has the following cgroup on Sandcastle:
/sandcastle-job/command/test-SystemdServiceTest.service
Unfortunately, systemd uses predictable cgroup names based on the service name. A service managed by one systemd instance shares the same cgroup as a same-named service managed by a different systemd instance. For example, if multiple systemd tests run concurrently on Sandcastle (e.g. during stress-testing), `systemctl --user stop foo.service` from one run will kill the processes of a different run's active foo.service. This causes systemd tests to be flaky on Sandcastle.
Fix the conflict by creating a unique cgroup per systemd instance. For example, separate runs of test_running_simple_service_is_active might use the following cgroups for the test service:
/sandcastle-job/command/edenfs_test.1_qat2xv/test-SystemdServiceTest.service
/sandcastle-job/command/edenfs_test.bbd3gj_o/test-SystemdServiceTest.service
Reviewed By: simpkins
Differential Revision: D13016626
fbshipit-source-id: 8535dc14a06bdb403c926b111cad4aed6c8ec3e3
Summary:
Check that the next-inode-number file exists and that the inode number it
contains is actually larger than all existing inode numbers. Replace it with
correct data if the file does not exist, is corrupt, or contains an incorrect
inode number.
Reviewed By: chadaustin
Differential Revision: D12955093
fbshipit-source-id: 3d26fb475535577d9a2697bbd575fba350766d01
Summary:
Update fsck to extract data for orphan inodes to a lost+found directory in the
fsck log directory, and remove them from the overlay. This will allow users
to recover the orphan file data if they want, and remove it otherwise.
Reviewed By: chadaustin
Differential Revision: D12955094
fbshipit-source-id: 9783452fce4060b9c5c48b3d48dd1f70294211c6
Summary:
The linter tripped on D12813838 even though the warning was unrelated
to my changes.
Reviewed By: simpkins
Differential Revision: D13167184
fbshipit-source-id: 555691f00d113ed2bff9488b61392cc92f4395e3
Summary:
Add several more files to the basic snapshot, so we can test more cases
in the fsck tests:
- Materialized, new, and unmodified symlinks
- A deeper directory tree of directory inodes that are not materialized (still
have a source control tree hash) but have children inodes allocated and are
therefore present in the overlay.
- A socket in a slightly deeper directory so we can test behavior of sockets
inside directories that have been corrupted..
As before I have replaced the older basic snapshot instead of adding a new
one, since the Eden data storage formats have not changed since the last
snapshot was created.
Reviewed By: chadaustin
Differential Revision: D13164658
fbshipit-source-id: d117c9cc336709044de212637c03140dfadd9a96
Summary:
This updates the basic snapshot code to include a couple slightly deeper
directories. I plan to use this in the fsck tests to verify handling of
orphan directories that contain subdirectories.
Normally it would be preferable to keep the old `basic-20181030` snapshot, and
simply add this new snapshot without than replacing the old one. However, I
don't think we have made any meaningful changes to our on-disk storage formats
since the previous snapshot was generated, so it seems okay to just delete the
old snapshot.
Reviewed By: strager
Differential Revision: D13151861
fbshipit-source-id: e6b7583beecb5d9cc55271ad2dea8d36980542d1
Summary:
Add a helper class for maintaining the list of expected files.
I plan to use this for the fsck tests, so I can more easily modify the
snapshot's expected contents based on how we expect fsck to repair various
types of overlay corruption. This also helps slightly simplify the code that
constructs the expected file list.
Reviewed By: chadaustin
Differential Revision: D13095918
fbshipit-source-id: 57686e82d1bf7f23a92eda0ed4d66623a3f58840
Summary:
Add basic high level logic to fsck to begin fixing problems that are found.
This adds basic checks to decide if we should fix errors or not.
If errors are found and need to be fixed, this creates a new directory inside
the checkout state directory in `.eden` to record the actions taken by this
fsck run. This directory will contain a log file that records the actions
taken. In the future the fsck logic will also use this directory to store
copies of the corrupted inode data, and can store extracted orphan inode data
here as well.
Reviewed By: wez
Differential Revision: D12955044
fbshipit-source-id: 06c1e17a0a51fa5e2c0f2aab83b367b9358fd004
Summary:
Eden has used the InodeMetadataTable as the authoritative source for
timestamp metadata for over six months. I think we can safely assume
that anyone at this point who has old inodes in the overlay that don't
have corresponding entries in the inode metadata table are fine with
those timestamps being reset to the last checkout time.
Reviewed By: strager
Differential Revision: D13144735
fbshipit-source-id: 06a9a8835ea83c98fb6a165e4c8d5c3c6b28ad84
Summary:
Eden has used the InodeMetadataTable as the primary source of
timestamp data for more than six months. Stop writing timestamps into
the overlay, since they will never be used.
Reviewed By: strager
Differential Revision: D13144696
fbshipit-source-id: e36423036228e89dd2a986e6bacfa74553c17a92
Summary:
After sending SIGTERM to systemd, increase the wait timeout from 3 seconds to
15 seconds. The previous 3 second timeout was easy to hit in practice on our
continuous build infrastructure.
If systemd does not exit within 15 seconds after we send SIGTERM, send SIGKILL
and then try waiting on it again for up to 3 seconds. Forcefully killing it
seems preferable to leaving the process hanging around after the tests exit.
Reviewed By: strager
Differential Revision: D13159491
fbshipit-source-id: debce21f2f202fb7cfa4a53120dcb2b2b35ccbe3
Summary: Instantiate a blob cache in the EdenServer with configurable settings.
Reviewed By: strager
Differential Revision: D12813880
fbshipit-source-id: 8ccc89826f04aca78964230374dea48abf05e05e
Summary: In support of making Eden's file access stateless, add a facade that connects the BlobCache and ObjectStore, allowing FileInode to fetch blobs, minimizing reloads and memory usage.
Reviewed By: strager
Differential Revision: D10850143
fbshipit-source-id: 4307f7c1143aecad1284ea3cadf3e4efca9a3925
Summary: Change the Windows pipe read and write functions to read/write in a loop. Plus changed the functions prototype to match the POSIX version.
Reviewed By: strager
Differential Revision: D13091785
fbshipit-source-id: 375b22bd9e62d371a78d410f42068945b966a743
Summary:
Backing store works with eden strings(UTF8 + Unix path separator). The path strings we receive on Windows from FS and cli are Windows paths
(Win path separator and/or UTF16). Adding the functions to convert one to another.
Reviewed By: strager
Differential Revision: D13091788
fbshipit-source-id: f7fc8a79e360e964cf4619dfa540b57f1f18d283
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary: Add a BlobCache with a maximum cache size and a minimum entry count and interest-based eviction.
Reviewed By: strager
Differential Revision: D12972062
fbshipit-source-id: 1958f7f500c051a5bc0b39b5b89a6f0fc1774b0f
Summary: Create a platform independent function to compare the stats and use that to check if the file contents have changed.
Reviewed By: strager
Differential Revision: D13046269
fbshipit-source-id: c4f5bc88cec3df5cb6555d13cea2f23fd4eeb7ce
Summary: Add a pyre configuration file to allow it to check all Eden python files.
Reviewed By: strager
Differential Revision: D13087241
fbshipit-source-id: efd3f05dc465f8e58271ce46a97aaaf6d5f0a12a
Summary:
Replace some member variables in
`_TransientUnmanagedSystemdUserServiceManager` with locals, to make pyre
happy.
Previously pyre complained that these member variables are not initialized in
`__init__()`. These variables are only needed temporarily during
`__enter__()` (and in some clean up closures), so just use them as local
variables instead.
Reviewed By: strager
Differential Revision: D13135257
fbshipit-source-id: 76f2bdc4b7b36d2102ad8dab4a60722a03197fab
Summary:
If invoked with `--log-target=console` systemd will log to stderr even if it
is not a tty.
This changes the tests to pass in `--log-target=console` rather than creating
a pty and forwarding I/O from it in a separate background thread.
Reviewed By: strager
Differential Revision: D13135258
fbshipit-source-id: 11dfe0711adaa62cedba2882045d8088e0df5499
Summary:
Update the CLI's overlay parsing code to make sure the `entries` field of
returned `OverlayDir` objects is always set, even if the serialized thrift
data on disk does not have data for this field.
Apparently "default" thrift fields (neither optional nor required) behave
differently in python than in C++. In C++ these fields are always populated
by deserialization: if they are not present in the serialized data a default
value is used. In Python it looks like the thrift code simply leaves the
value unmodified (which usually means `None`). This changes the code to
ensure that this value will default to an empty dictionary if it is not
present.
Reviewed By: chadaustin
Differential Revision: D13135378
fbshipit-source-id: ce5254922e6fc88d6f3d8c58778ad9d45f02741f
Summary: test_processes_of_forking_service_includes_all_child_processes is failing on some machines. The test assumes /sys/fs/cgroup/ is a cgroups v2 mount point, so it fails if /sys/fs/cgroup/ is a directory containing cgroups v1 mount points. Disable the test on machines where cgroups v1 is detected.
Reviewed By: simpkins
Differential Revision: D13112836
fbshipit-source-id: 7921604707a0c1fe81a82c87e767a6a99cdd6206
Summary:
[Thrift] Cut most of the remaining Thrift SASL integration. Legacy Thrift SASL is replaced by TLS.
(Note: this ignores all push blocking failures!)
Reviewed By: vitaut
Differential Revision: D12954177
fbshipit-source-id: 42909632f4232301cdeeda465848d7e9700d3074
Summary: Fix the remaining set of errors reported by pyre and mypy.
Reviewed By: strager
Differential Revision: D13086855
fbshipit-source-id: 4c2b21352f94ef225a5555aef0f6b95b92e56f6d
Summary:
Our code in testcase.py mucks around with test case classes in several ways
that mypy doesn't like. In particular, mypy does not currently support
replacing methods on classes, and it also does not understand dynamic base
classes. pyre also trips up on some of these changes, although in different
ways.
This updates testcase.py so that mypy and pyre no longer report errors on it,
often just by suppressing the errors. Also fix similar errors in
service_test_case.py around replicating test classes.
Reviewed By: strager
Differential Revision: D13086856
fbshipit-source-id: af446dd13791f5da50b09657012db95c2bcf0e39
