Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
#!/usr/bin/env python3
|
2019-06-20 02:58:25 +03:00
|
|
|
# Copyright (c) Facebook, Inc. and its affiliates.
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
#
|
2019-06-20 02:58:25 +03:00
|
|
|
# This software may be used and distributed according to the terms of the
|
|
|
|
# GNU General Public License version 2.
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
|
|
|
|
import os
|
|
|
|
import pathlib
|
|
|
|
import stat
|
|
|
|
import tempfile
|
|
|
|
|
|
|
|
import eden.integration.lib.edenclient as edenclient
|
|
|
|
|
|
|
|
|
|
|
|
class OverlayStore:
|
|
|
|
def __init__(self, eden: edenclient.EdenFS, mount: pathlib.Path) -> None:
|
|
|
|
self.eden = eden
|
|
|
|
self.mount = mount
|
|
|
|
self.overlay_dir = eden.overlay_dir_for_mount(mount)
|
|
|
|
|
|
|
|
def materialize_file(self, path: pathlib.Path) -> pathlib.Path:
|
|
|
|
"""Force the file inode at the specified path to be materialized and recorded in
|
|
|
|
the overlay. Returns the path to the overlay file that stores the data for this
|
|
|
|
inode in the overlay.
|
|
|
|
"""
|
|
|
|
path_in_mount = self.mount / path
|
|
|
|
# Opening the file in write mode will materialize it
|
|
|
|
with path_in_mount.open("w+b") as f:
|
|
|
|
s = os.fstat(f.fileno())
|
|
|
|
|
|
|
|
return self._get_overlay_path(s.st_ino)
|
|
|
|
|
|
|
|
def materialize_dir(self, path: pathlib.Path) -> pathlib.Path:
|
|
|
|
"""Force the directory inode at the specified path to be materialized and
|
|
|
|
recorded in the overlay. Returns the path to the overlay file that stores the
|
|
|
|
data for this inode in the overlay.
|
|
|
|
"""
|
2018-09-18 00:13:21 +03:00
|
|
|
path_in_mount = self.mount / path
|
|
|
|
s = os.lstat(path_in_mount)
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
assert stat.S_ISDIR(s.st_mode)
|
|
|
|
# Creating and then removing a file inside the directory will materialize it
|
2018-09-18 00:13:21 +03:00
|
|
|
with tempfile.NamedTemporaryFile(dir=str(path_in_mount)):
|
Allow rm of files with corrupt overlay
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
|
|
|
pass
|
|
|
|
|
|
|
|
return self._get_overlay_path(s.st_ino)
|
|
|
|
|
|
|
|
def _get_overlay_path(self, inode_number: int) -> pathlib.Path:
|
|
|
|
subdir = "{:02x}".format(inode_number % 256)
|
|
|
|
return self.overlay_dir / subdir / str(inode_number)
|
|
|
|
|
2018-09-18 00:13:21 +03:00
|
|
|
def delete_cached_next_inode_number(self) -> None:
|
|
|
|
(self.overlay_dir / "next-inode-number").unlink()
|
2019-08-01 06:06:21 +03:00
|
|
|
|
|
|
|
def get_info_path(self) -> pathlib.Path:
|
|
|
|
"""Get the path to the overlay "info" file that contains the top-level overlay
|
|
|
|
metadata and also serves as the overlay lock file.
|
|
|
|
|
|
|
|
Corrupting this file will make it impossible for Eden to read or repair the
|
|
|
|
overlay data. This can be used to make the overlay unusable in tests.
|
|
|
|
"""
|
|
|
|
return self.overlay_dir / "info"
|