sapling/eden/integration/lib/overlay.py

62 lines
2.3 KiB
Python
Raw Normal View History

Allow rm of files with corrupt overlay Summary: Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In particular, sometimes overlay files can be truncated or missing after a hard reboot where the underlying filesystem state was not flushed to disk. For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet readdir() on the containing directory shows that the file does exist. In other words, the problematic file is undeletable: ``` $ ls -la dir/ /bin/ls: cannot access dir/corrupt_file: No such file or directory total 0 drwxr-xr-x. 3 strager 0 Jul 10 21:41 . drwxr-xr-x. 48 strager 0 Jul 10 21:41 .. -?????????? ? ? ? ? corrupt_file $ rm dir/corrupt_file rm: cannot remove ‘dir/corrupt_file’: No such file or directory ``` Allow users to delete these problematic files (if the file was a regular file and not a directory) by doing the following: * Allow corrupt regular files to be unlink()d successfully. * Allow corrupt regular files to be stat()d. Making stat() succeed is a requirement by FUSE: * For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for corrupt files. * For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we must make FUSE_LOOKUP succeed (for unlink()), it's natural to make FUSE_GETATTR succeed too. A future diff will fix corrupted directories. Reviewed By: chadaustin Differential Revision: D8884793 fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
#!/usr/bin/env python3
# Copyright (c) Facebook, Inc. and its affiliates.
Allow rm of files with corrupt overlay Summary: Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In particular, sometimes overlay files can be truncated or missing after a hard reboot where the underlying filesystem state was not flushed to disk. For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet readdir() on the containing directory shows that the file does exist. In other words, the problematic file is undeletable: ``` $ ls -la dir/ /bin/ls: cannot access dir/corrupt_file: No such file or directory total 0 drwxr-xr-x. 3 strager 0 Jul 10 21:41 . drwxr-xr-x. 48 strager 0 Jul 10 21:41 .. -?????????? ? ? ? ? corrupt_file $ rm dir/corrupt_file rm: cannot remove ‘dir/corrupt_file’: No such file or directory ``` Allow users to delete these problematic files (if the file was a regular file and not a directory) by doing the following: * Allow corrupt regular files to be unlink()d successfully. * Allow corrupt regular files to be stat()d. Making stat() succeed is a requirement by FUSE: * For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for corrupt files. * For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we must make FUSE_LOOKUP succeed (for unlink()), it's natural to make FUSE_GETATTR succeed too. A future diff will fix corrupted directories. Reviewed By: chadaustin Differential Revision: D8884793 fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
#
# This software may be used and distributed according to the terms of the
# GNU General Public License version 2.
Allow rm of files with corrupt overlay Summary: Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In particular, sometimes overlay files can be truncated or missing after a hard reboot where the underlying filesystem state was not flushed to disk. For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet readdir() on the containing directory shows that the file does exist. In other words, the problematic file is undeletable: ``` $ ls -la dir/ /bin/ls: cannot access dir/corrupt_file: No such file or directory total 0 drwxr-xr-x. 3 strager 0 Jul 10 21:41 . drwxr-xr-x. 48 strager 0 Jul 10 21:41 .. -?????????? ? ? ? ? corrupt_file $ rm dir/corrupt_file rm: cannot remove ‘dir/corrupt_file’: No such file or directory ``` Allow users to delete these problematic files (if the file was a regular file and not a directory) by doing the following: * Allow corrupt regular files to be unlink()d successfully. * Allow corrupt regular files to be stat()d. Making stat() succeed is a requirement by FUSE: * For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for corrupt files. * For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we must make FUSE_LOOKUP succeed (for unlink()), it's natural to make FUSE_GETATTR succeed too. A future diff will fix corrupted directories. Reviewed By: chadaustin Differential Revision: D8884793 fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
import os
import pathlib
import stat
import tempfile
import eden.integration.lib.edenclient as edenclient
class OverlayStore:
def __init__(self, eden: edenclient.EdenFS, mount: pathlib.Path) -> None:
self.eden = eden
self.mount = mount
self.overlay_dir = eden.overlay_dir_for_mount(mount)
def materialize_file(self, path: pathlib.Path) -> pathlib.Path:
"""Force the file inode at the specified path to be materialized and recorded in
the overlay. Returns the path to the overlay file that stores the data for this
inode in the overlay.
"""
path_in_mount = self.mount / path
# Opening the file in write mode will materialize it
with path_in_mount.open("w+b") as f:
s = os.fstat(f.fileno())
return self._get_overlay_path(s.st_ino)
def materialize_dir(self, path: pathlib.Path) -> pathlib.Path:
"""Force the directory inode at the specified path to be materialized and
recorded in the overlay. Returns the path to the overlay file that stores the
data for this inode in the overlay.
"""
path_in_mount = self.mount / path
s = os.lstat(path_in_mount)
Allow rm of files with corrupt overlay Summary: Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In particular, sometimes overlay files can be truncated or missing after a hard reboot where the underlying filesystem state was not flushed to disk. For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet readdir() on the containing directory shows that the file does exist. In other words, the problematic file is undeletable: ``` $ ls -la dir/ /bin/ls: cannot access dir/corrupt_file: No such file or directory total 0 drwxr-xr-x. 3 strager 0 Jul 10 21:41 . drwxr-xr-x. 48 strager 0 Jul 10 21:41 .. -?????????? ? ? ? ? corrupt_file $ rm dir/corrupt_file rm: cannot remove ‘dir/corrupt_file’: No such file or directory ``` Allow users to delete these problematic files (if the file was a regular file and not a directory) by doing the following: * Allow corrupt regular files to be unlink()d successfully. * Allow corrupt regular files to be stat()d. Making stat() succeed is a requirement by FUSE: * For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for corrupt files. * For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we must make FUSE_LOOKUP succeed (for unlink()), it's natural to make FUSE_GETATTR succeed too. A future diff will fix corrupted directories. Reviewed By: chadaustin Differential Revision: D8884793 fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
assert stat.S_ISDIR(s.st_mode)
# Creating and then removing a file inside the directory will materialize it
with tempfile.NamedTemporaryFile(dir=str(path_in_mount)):
Allow rm of files with corrupt overlay Summary: Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In particular, sometimes overlay files can be truncated or missing after a hard reboot where the underlying filesystem state was not flushed to disk. For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet readdir() on the containing directory shows that the file does exist. In other words, the problematic file is undeletable: ``` $ ls -la dir/ /bin/ls: cannot access dir/corrupt_file: No such file or directory total 0 drwxr-xr-x. 3 strager 0 Jul 10 21:41 . drwxr-xr-x. 48 strager 0 Jul 10 21:41 .. -?????????? ? ? ? ? corrupt_file $ rm dir/corrupt_file rm: cannot remove ‘dir/corrupt_file’: No such file or directory ``` Allow users to delete these problematic files (if the file was a regular file and not a directory) by doing the following: * Allow corrupt regular files to be unlink()d successfully. * Allow corrupt regular files to be stat()d. Making stat() succeed is a requirement by FUSE: * For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for corrupt files. * For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we must make FUSE_LOOKUP succeed (for unlink()), it's natural to make FUSE_GETATTR succeed too. A future diff will fix corrupted directories. Reviewed By: chadaustin Differential Revision: D8884793 fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
2018-07-26 06:52:58 +03:00
pass
return self._get_overlay_path(s.st_ino)
def _get_overlay_path(self, inode_number: int) -> pathlib.Path:
subdir = "{:02x}".format(inode_number % 256)
return self.overlay_dir / subdir / str(inode_number)
def delete_cached_next_inode_number(self) -> None:
(self.overlay_dir / "next-inode-number").unlink()
def get_info_path(self) -> pathlib.Path:
"""Get the path to the overlay "info" file that contains the top-level overlay
metadata and also serves as the overlay lock file.
Corrupting this file will make it impossible for Eden to read or repair the
overlay data. This can be used to make the overlay unusable in tests.
"""
return self.overlay_dir / "info"