Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary:
We think that it shouldn't really be needed to perform
the prefetch call during lookup; for file inodes it doesn't buy
us much, and it should only really help for readdir.
This removes the prefetch call from lookup, instead prefetching
upon the first readdir() of a loaded TreeInode.
Reviewed By: simpkins
Differential Revision: D12896022
fbshipit-source-id: 0209eb64bd522daf5f7461dffccd1312d32a1554
Summary:
Now that FileInode read and write operations are stateless via BlobAccess and OverlayFileAccess,
EdenFileHandle no longer provides any value. Remove it. This also fixes eden's shutdown timeout
when a file handle is open and paves the way for FUSE_NO_OPEN_SUPPORT.
Reviewed By: strager
Differential Revision: D13325137
fbshipit-source-id: 71ed47a7c997f5035b4394ccb311f94332ecd8c2
Summary:
Have FileInode use OverlayFileAccess instead of using the Overlay directly.
This allows IO on materialized files to be stateless and pave the way for
eliminating EdenFileHandle. It also paves the way for performance improvements
such as nicer SHA-1 caching.
Reviewed By: strager
Differential Revision: D13325079
fbshipit-source-id: fb27d48b5dc9196dc6e36557596f601194a56aa9
Summary:
Move the overlay IO logic out of FileInode into a centralized OverlayFileAccess class. It keeps the last 100 overlay file handles open.
FileInode's changes are coming in D13325079.
Reviewed By: strager
Differential Revision: D13324995
fbshipit-source-id: 04fb3fe50114b0f19b78acd17a9684c92f8e8072
Summary:
Drop interest in cached blobs at various points in the FileInode
lifecycle.
Reviewed By: strager
Differential Revision: D12991762
fbshipit-source-id: 19fd94938c96485160d547ecbd259ffeb39b2341
Summary:
FUSE_NO_OPEN_SUPPORT is better than ATOMIC_O_TRUNC for Eden's use
case. Remove the code that pretended we might support ATOMIC_O_TRUNC
again someday.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13163382
fbshipit-source-id: 948d701571a8d2977da3d2532fdc9538c5011636
Summary:
FileInode no longer has a strong reference to a blob. Instead, all accesses go through the blob cache. This diff changes the caching behavior for blobs.
The previous behavior was:
When a file's contents are needed in any way, the blob is loaded and referenced by the inode. When the number of open file handles goes to zero, the blob is dropped. The blob is also dropped when the inode is unloaded. Future inode loads or open requests, in that situation, require the blob to be reloaded from the LocalStore.
The new behavior is:
When a file's contents are needed, the blob is loaded and stored into the BlobCache, evicting any if necessary. Future blob requests are satisfied from the BlobCache, pushing it to the back of the eviction queue. When the inode is materialized or unloaded, the blob will be evicted from cache if no other blob has interest in it.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D12813912
fbshipit-source-id: 20d20807d2e4a6c37cddab38562fdb7456316aac
Summary:
Now that the Overlay no longer serializes timestamps, remove all of
the special-case migration logic.
Reviewed By: strager
Differential Revision: D13144764
fbshipit-source-id: 713a4bfcde9003a8d5a28837cb530b05a9017c22
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary:
Eden supports reading the SHA-1 of a file via getxattr. Unfortunately,
it returned ENOSYS if you called getxattr on a directory inode. This
caused FUSE to fail all future getxattr requests with EOPNOTSUPP.
In addition to fixing that, this diff makes our xattr handling a
little more consistent across inodes:
- setxattr() always fails with ENOSYS
- removexattr() always fails with ENOSYS
- listxattr() is always handled by the corresponding inode class
- getxattr() is always handled by the corresponding inode class
Differential Revision: D10437723
fbshipit-source-id: a1ea1e92d3412abb15e91057becea3160a17f1e2
Summary:
We weren't doing anything that interesting in flush()
anyway. Precomputing the SHA-1 for materialized files optimizes for a
relatively rare situation that penalizes the writer of large files for
the possibility that Buck might read the files later.
Differential Revision: D10435552
fbshipit-source-id: 24aa8f7d9ec5094b084ebd02964840b4b01ad48b
Summary: Always send write requests straight to the inode rather than going through FileHandle.
Reviewed By: wez
Differential Revision: D10220619
fbshipit-source-id: 9ce328583cf0fa9d7d8850d92d9e15ddc382d6a3
Summary:
Always send read requests straight to the inode rather than going
through the FileHandle.
Reviewed By: wez
Differential Revision: D10220604
fbshipit-source-id: 6aa5d20f3ce09696a29bd5c1cb95d0b987ab213c
Summary:
Prior to this diff, to read the inode metadata from an InodeBase, you
had to cast it to a FileInode or TreeInode. Add a public getMetadata
call that acquires the appropriate lock depending on whether it's a
FileInode or TreeInode.
Reviewed By: simpkins
Differential Revision: D9327180
fbshipit-source-id: 66594fb639f1ea72a0e48dcb79234e14d87ec4a2
Summary:
Now that timestamps are read from the inode metadata table, and users
aren't likely to run a pre-metadata-table version, the timestamp data
in the overlay header's no longer needs to be written. So remove that
code which has the bonus of making unloading faster.
Reviewed By: wez
Differential Revision: D9318044
fbshipit-source-id: 27a9a9ee954003940209819466932237a81f8929
Summary:
The `updateOverlayHeader()` only updates the overlay data if the inode is
materialized. This updates the name to clarify that.
(This function name change was previously part of D8884795, and I'm just
splitting it into its own separate diff.)
Reviewed By: bolinfest
Differential Revision: D9011358
fbshipit-source-id: 6024d64a1dee0b5d741bec32ed88f6c8f8dd8a9a
Summary:
The two FileInode::isSameAs() methods called FileInode::getSha1(), which
returns a folly::Future object, and immediately called `value()` on the
resulting future without waiting for it to complete first.
This bug dates back to D5430128. However, I suspect it was D7888344 that
actually exposed this issue recently. D7888344 updates the
`RocksDbLocalStore` code to perform work in an I/O thread pool. Before this
change the SHA1 futures were likely always immediately ready, so this bug was
not causing problems.
Reviewed By: wez
Differential Revision: D8393671
fbshipit-source-id: ec2116751ddda31a119bfe85eab5612b622f83cf
Summary:
Per code review feedback from D6891479, this diff enforces that
metadata writes and reads are done while the corresponding inode's
state lock is held.
Reviewed By: simpkins
Differential Revision: D7884463
fbshipit-source-id: d0e7a95415c280441276452ece7233d4cbf5e942
Summary:
FileInode and TreeInode's State classes are complicated enough that
they deserve to be lifted out. In addition, this is necessary for
using the type system to enforce that contents locks are held in
InodeBase's metadata accessors.
Reviewed By: simpkins
Differential Revision: D7867399
fbshipit-source-id: 6ce082149ba02099487e8caed33a7bd8510dfebb
Summary:
Relax the restriction on changing uid/gid on inodes. We'll see what
cans of worms this opens I guess. (Landing this is low priority, but
might be important for making some of the existing tooling in fbsource
and www work.)
Reviewed By: simpkins
Differential Revision: D7768655
fbshipit-source-id: 95fe02fe7ddc001335dbdb34e16a989a85820240
Summary:
This moves most inode metadata management into InodeBase and
persists permission bits (and eventually uid/gid) across Eden runs.
Reviewed By: simpkins
Differential Revision: D7035163
fbshipit-source-id: 50145449b56aad1662d53156e6e4960c5f7b6166
Summary: Store tree and file timestamps in the InodeTable so they persist across runs.
Reviewed By: simpkins
Differential Revision: D6891479
fbshipit-source-id: 1c9e6266375aceeaf293a81e73cf7f5334dbc32d
Summary: The InodeTable work will homogenize mode_t storage, access, and modification across file and tree inodes. In preparation, have InodeBase keep track of the initial mode bits instead of a dtype_t.
Reviewed By: simpkins
Differential Revision: D7031924
fbshipit-source-id: f2e6e4467cecfc0ca06ad998cce0af18a99cc251
Summary: Also print the inode path if the assertions in EXPECT_FILE_INODE fail.
Reviewed By: simpkins
Differential Revision: D7035517
fbshipit-source-id: 6c50acb588d1c985c7e7a1586c06f04a657698f9
Summary:
Update all Overlay APIs to accept InodeNumbers and compute the path to the
overlay file internally, rather than requiring the caller to pass in the path
to the file inside the overlay.
Reviewed By: chadaustin
Differential Revision: D7411497
fbshipit-source-id: d057babb0b1b8d6c3b9adf1a21b84eed363dffa1
Summary:
This adds a new LockedState helper class in FileInode to also keep track of
whether or not we are currently holding an refcount on the open file/blob. The
LockedState object automatically decrements this refcount on destruction.
Alternatively, this refcount can be transferred to a new EdenFileHandle object
when unlocking the state.
Previously most of the internal FileInode code simply used EdenFileHandle
objects to manage this refcount. However, this is prone to deadlock, since you
have to ensure that EdenFileHandle objects are never destroyed while the state
is already locked. This new LockedState API makes it harder to have code paths
that may accidentally destroy an EdenFileHandle object while holding the state
lock.
Reviewed By: chadaustin
Differential Revision: D7407423
fbshipit-source-id: 610fcda3220a9f49b734910b7a13e8d68a81a779
Summary:
Update InodeMap::onInodeUnreferenced() to take a pointer to a non-const
InodeBase. This allows the methods it calls, including
InodeBase::updateOverlayHeader() to be non-const.
Using non-const InodeBase objects seems to make sense here since the inode is
in the process of being destroyed.
In a future diff I plan to update FileInode::updateOverlayHeader() to share the
same code as normal file methods to ensure that the overlay file is open. This
modifies the FileInode state's open refcount, so it is useful to have this
method be non-const for that purpose.
Reviewed By: chadaustin
Differential Revision: D7407424
fbshipit-source-id: 541656c7b9b283c5e5650445de5bbdbaae3fc57f
Summary:
This replaces the `ensureDataLoaded()`, `materializeForWrite()`, and
`materializeAndTruncate()` methods in `FileInode` with new safer alternatives:
`runWhileDataLoaded()`, `runWhileMaterialized()`, and `truncateAndRun()`
These new methods take a function to run, and run it while holding the state
lock with the FileInode guaranteed to be in the desired state. They also
require the caller move in the state lock as an argument, to help ensure that
the caller gets the locking behavior correct.
The old methods required that the caller not already hold the state lock while
calling them. This pre-condition was violated in `FileInode::write()` and
possibly other places. They also required releasing the lock between they
confirmed the file state and when the caller's function ran, requiring
additional lock and unlock operations, and also making the code harder to
reason about since other threads could change the state during the time when
the lock was released.
Reviewed By: chadaustin
Differential Revision: D7363180
fbshipit-source-id: d8e667d0bc7006c519252a8d0682af97517997eb
Summary:
FileInode::read() did not handle being called in the BLOB_LOADING or NOT_LOADED
states. This method is only called from EdenFileHandle::read(), which does no
checking that it is fully loaded first.
This fixes read() to always call ensureDataLoaded() first.
Reviewed By: chadaustin
Differential Revision: D7359878
fbshipit-source-id: f5a1c8a28db3267da3180b67f970430e3ea291da
Summary:
This helps distinguish `facebook::eden::FileHandle` from
`facebook::eden::fusell::FileHandle`, and will make it possible to eliminate
the separate `facebook::eden::fusell` namespace in the future.
Reviewed By: chadaustin
Differential Revision: D7314456
fbshipit-source-id: 0869427766fe666d119a59c7df1c97825a8ad36e
Summary: Per code review comments on D6983198, this simplifies the way we check if mode bits have changed in a meaningful-to-source-control way.
Reviewed By: simpkins
Differential Revision: D7015339
fbshipit-source-id: 548ead337fbea1c1dcb72b880921671e9b6188ac
Summary:
Update FileInode so that getattr() and setattr() both return a reasonable value
in st_blocks.
Previously we always returned 0 in st_blocks, which caused applications like
`du` to always report files as using 0 space in Eden mounts. Now we compute
st_blocks based on st_size, so that `du` will report reasonable estimates for
when scanning the size of subdirectories inside an Eden mount.
Reviewed By: chadaustin
Differential Revision: D6932098
fbshipit-source-id: bd29e46821176e510f420e6e2b6ce480b80d50ff
Summary:
isSameAs calls getSha1 which was failing on symlinks. The
original concern was that asking for the SHA-1 of a symlink is
ambiguous: do you want the hash of the symlink or the target? But we
already check for whether you are requesting the SHA-1 of a symlink in
EdenServiceHandler, so it's redundant and incorrect to check in
FileInode too.
Reviewed By: simpkins
Differential Revision: D6847489
fbshipit-source-id: 13966da06bcde75c5c568e09fef14e735de47cfb
Summary:
rdev doesn't add any value yet. We can add it back if we want
to implement support.
Reviewed By: simpkins
Differential Revision: D6792346
fbshipit-source-id: ce16317074f1daa456737c55804da8fb7f2b7a94
Summary:
This came up when I was auditing the rules about when it's
safe to read from a FileInode. read() must only be called while openCount > 0.
Reviewed By: wez
Differential Revision: D6604898
fbshipit-source-id: 829ddc335bd58201c2b456ee544cdc6253ebf66c
Summary:
In a follow-on diff, the constraint will be that state->blob
will only be guaranteed valid after ensureDataLoaded() while a
FileHandle is alive. Thus, ensureDataLoaded() must return a
FileHandle.
Reviewed By: wez
Differential Revision: D6586237
fbshipit-source-id: ccc269d322b8c725c93145df5de2add9a2b90207
Summary:
This serves a few purposes:
1. We can avoid some conditional code inside eden if we know that
we have a specific fuse_kernel.h header implementation.
2. We don't have to figure out a way to propagate the kernel
capabilities through the graceful restart process.
3. libfuse3 removed the channel/session hooks that we've been
using thus far to interject ourselves for mounting and
graceful restarting, so we were already effectively the
walking dead here.
4. We're now able to take advtange of the latest aspects of
the fuse kernel interface without being tied to the implementation
of libfuse2 or libfuse3. We're interested in the readdirplus
functionality and will look at enabling that in a future diff.
This may make some things slightly harder for the more immediate
macOS port but I belive that we're in a much better place overall.
This diff is relatively mechanical and sadly is (unavoidably) large.
The main aspects of this diff are:
1. The `fuse_ino_t` type was provided by libfuse so we needed to
replace it with our own definition. This has decent penetration
throughout the codebase.
2. The confusing `fuse_file_info` type that was multi-purpose and
had fields that were sometimes *in* parameters and sometimes *out*
parameters has been removed and replaced with a simpler *flags*
parameter that corresponds to the `open(2)` flags parameter.
The *out* portions are subsumed by existing file handle metadata
methods.
3. The fuse parameters returned from variations of the `LOOKUP` opcode
now return the fuse kernel type for this directly. I suspect
that we may need to introduce a compatibility type when we revisit
the macOS port, but this at least makes this diff slightly simpler.
You'll notice that some field and symbol name prefixes vary as
a result of this.
4. Similarly for `setattr`, libfuse separated the kernel data into
two parameters that were a little awkward to use; we're now just
passing the kernel data through and this, IMO, makes the interface
slightly more understandable.
5. The bulk of the code from `Dispatcher.cpp` that shimmed the
libfuse callbacks into the C++ virtual methods has been removed
and replaced by a `switch` statement based dispatcher in
`FuseChannel`. I'm not married to this being `switch` based
and may revise this to be driven by an `unordered_map` of
opcode -> dispatcher method defined in `FuseChannel`. Regardless,
`Dispatcher.cpp` is now much slimmer and should be easier to
replace by rolling it together into `EdenDispatcher.cpp` in
the future should we desire to do so.
6. This diff disables dispatching `poll` and `ioctl` calls. We
didn't make use of them and their interfaces are a bit fiddly.
7. `INTERRUPT` is also disabled here. I will re-enable it in
a follow-up diff where I can also revise how we track outstanding
requests for graceful shutdown.
8. I've imported `fuse_kernel.h` from libfuse. This is included
under the permissive 2-clause BSD license that it allows for
exactly this integration purpose.
Reviewed By: simpkins
Differential Revision: D6576472
fbshipit-source-id: 7cb088af5e06fe27bf22a1bed295c18c17d8006c
Summary:
There's no technical reason to block an open() request until the data
load / materialization returns. This change returns immediately from
open() and then waits if necessary in a subsequent write() call.
Reviewed By: wez
Differential Revision: D6391486
fbshipit-source-id: 862f87e3c3a0d760bacb0f8ca7acc479037fec2f
Summary:
To make the materialization code paths a bit clearer, this decouples
materialization from a blob and truncation. It also caches opened
files if openCount > 0 in both the truncation and materialization from
blob paths.
Reviewed By: wez
Differential Revision: D6388318
fbshipit-source-id: c95a85f5bdaa405130f2f7260143592cdc90d45e
Summary:
This corrects a bug where the in-memory timestamp for new files would
be set to the last checkout time instead of when the file was created.
Reviewed By: simpkins
Differential Revision: D6366189
fbshipit-source-id: c5fa8c779726d3a75c2d57b2a161293297eb9271
Summary:
Per a discussion with wez, let's make it explicit in the comments that the kernel handles
O_EXCL for us.
Reviewed By: simpkins
Differential Revision: D6365157
fbshipit-source-id: 638f67cd89a9450ff084e0ef77c731ef738bf518
Summary:
In some workloads we're seeing folks run out of file descriptors.
We forgot that we'd taken out the code that closes the underlying fds.
This diff takes a run at adding a simple counter of the open file handle
objects that is incremented when they are constructed and decremented
when they are destroyed.
When the count falls to zero we release the file handle.
Note that we unconditionally open files when we first load the inodes
from the overlay. I tried to defer that open attempt and it broke
the timestamp overlay test. I think we can revisit that aspect in
a follow on diff; for now we should be more resilient to transiently
opened files from things like ripgrep or similar.
Reviewed By: simpkins
Differential Revision: D6097090
fbshipit-source-id: 9a48220002e760fb1ffb8d7e2a68fa7036558b78
Summary:
In preparation for bringing D6097090 across the finish line, we need a
more explicit mechanism for determining which state the inode is in.
The issue is that whether an inode is materialized was checked in a
couple ways: the nonexistence of the hash field or the definedness of
the file field. This diff introduces an explicit enum indicating the
state.
Reviewed By: simpkins
Differential Revision: D6325955
fbshipit-source-id: 3682a4ebc9330193baadbb33a4dd9845f26e59a6
