Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
#!/usr/bin/env python3
|
2019-06-20 02:58:25 +03:00
|
|
|
# Copyright (c) Facebook, Inc. and its affiliates.
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
#
|
2019-06-20 02:58:25 +03:00
|
|
|
# This software may be used and distributed according to the terms of the
|
|
|
|
# GNU General Public License version 2.
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
|
2018-07-26 06:53:00 +03:00
|
|
|
import logging
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
import os
|
|
|
|
import pathlib
|
2018-07-26 06:53:00 +03:00
|
|
|
import stat
|
2018-07-27 03:02:07 +03:00
|
|
|
from typing import List
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
|
|
|
|
import eden.integration.lib.overlay as overlay_mod
|
|
|
|
from eden.integration.lib import testcase
|
|
|
|
|
|
|
|
|
2018-07-26 06:53:00 +03:00
|
|
|
class CorruptOverlayTest(testcase.HgRepoTestMixin, testcase.EdenRepoTest):
|
|
|
|
"""Test file operations when Eden's overlay is corrupted."""
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
|
|
|
|
def setUp(self) -> None:
|
2018-11-17 06:39:20 +03:00
|
|
|
super().setUp() # type: ignore (pyre does not follow python MRO properly)
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
self.overlay = overlay_mod.OverlayStore(self.eden, self.mount_path)
|
|
|
|
|
|
|
|
def populate_repo(self) -> None:
|
2018-07-26 06:53:00 +03:00
|
|
|
self.repo.write_file("src/committed_file", "committed_file content")
|
|
|
|
self.repo.write_file("readme.txt", "readme content")
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
self.repo.commit("Initial commit.")
|
|
|
|
|
2018-07-27 03:02:07 +03:00
|
|
|
def _corrupt_files(self) -> List[pathlib.Path]:
|
|
|
|
"""Corrupt some files inside the mount.
|
|
|
|
Returns relative paths to these files inside the mount.
|
|
|
|
"""
|
|
|
|
# Corrupt 3 separate files. 2 are tracked by mercurial, one is not.
|
|
|
|
# We will corrupt 2 of them by truncating the overlay file, and one by
|
|
|
|
# completely removing the overlay file. (In practice an unclean reboot often
|
|
|
|
# leaves overlay files that exist but have 0 length.)
|
2018-07-26 06:53:00 +03:00
|
|
|
tracked_path = pathlib.Path("src/committed_file")
|
|
|
|
untracked_path = pathlib.Path("src/new_file")
|
|
|
|
readme_path = pathlib.Path("readme.txt")
|
|
|
|
|
|
|
|
tracked_overlay_file_path = self.overlay.materialize_file(tracked_path)
|
|
|
|
untracked_overlay_file_path = self.overlay.materialize_file(untracked_path)
|
|
|
|
readme_overlay_file_path = self.overlay.materialize_file(readme_path)
|
|
|
|
|
|
|
|
self.eden.unmount(self.mount_path)
|
|
|
|
os.truncate(tracked_overlay_file_path, 0)
|
|
|
|
os.unlink(untracked_overlay_file_path)
|
|
|
|
os.truncate(readme_overlay_file_path, 0)
|
|
|
|
self.eden.mount(self.mount_path)
|
|
|
|
|
2018-07-27 03:02:07 +03:00
|
|
|
return [tracked_path, untracked_path, readme_path]
|
|
|
|
|
|
|
|
def test_unmount_succeeds(self) -> None:
|
|
|
|
corrupted_paths = self._corrupt_files()
|
|
|
|
|
|
|
|
# Access the files to make sure that edenfs loads them.
|
|
|
|
# The stat calls should succeed, but reading them would fail.
|
|
|
|
for path in corrupted_paths:
|
|
|
|
os.lstat(str(self.mount_path / path))
|
|
|
|
|
|
|
|
# Make sure that eden can successfully unmount the mount point
|
|
|
|
# Previously we had a bug where the inode unloading code would throw an
|
|
|
|
# exception if it failed to update the overlay state for some inodes.
|
|
|
|
self.eden.unmount(self.mount_path)
|
|
|
|
|
|
|
|
def test_unlink_deletes_corrupted_files(self) -> None:
|
|
|
|
corrupted_paths = self._corrupt_files()
|
|
|
|
for path in corrupted_paths:
|
2018-07-26 06:53:00 +03:00
|
|
|
logging.info(f"stat()ing and unlinking {path}")
|
|
|
|
full_path = self.mount_path / path
|
|
|
|
s = os.lstat(str(full_path))
|
|
|
|
self.assertTrue(stat.S_ISREG, s.st_mode)
|
|
|
|
self.assertEqual(0, s.st_mode & 0o7777)
|
|
|
|
self.assertEqual(0, s.st_size)
|
|
|
|
full_path.unlink()
|
|
|
|
self.assertFalse(
|
|
|
|
full_path.exists(), f"{full_path} should not exist after being deleted"
|
|
|
|
)
|
2018-09-18 00:13:21 +03:00
|
|
|
|
|
|
|
def test_mount_possible_after_corrupt_directory_and_cached_next_inode_number(
|
2020-10-04 14:49:10 +03:00
|
|
|
self,
|
2018-09-18 00:13:21 +03:00
|
|
|
) -> None:
|
|
|
|
test_dir_path = self.mount_path / "test_dir"
|
|
|
|
test_dir_path.mkdir()
|
|
|
|
test_dir_overlay_file_path = self.overlay.materialize_dir(test_dir_path)
|
|
|
|
|
|
|
|
self.eden.unmount(self.mount_path)
|
|
|
|
os.truncate(test_dir_overlay_file_path, 0)
|
|
|
|
self.overlay.delete_cached_next_inode_number()
|
|
|
|
|
|
|
|
self.eden.mount(self.mount_path)
|
2019-08-01 06:06:21 +03:00
|
|
|
|
|
|
|
def test_eden_list_does_not_return_corrupt_mounts(self) -> None:
|
|
|
|
self.eden.shutdown()
|
|
|
|
|
|
|
|
# Truncate the overlay info file so edenfs will
|
|
|
|
# not be able to open the overlay
|
|
|
|
os.truncate(self.overlay.get_info_path(), 0)
|
|
|
|
|
|
|
|
self.eden.start()
|
|
|
|
self.assertEqual({str(self.mount): "NOT_RUNNING"}, self.eden.list_cmd_simple())
|