Summary:
We think that it shouldn't really be needed to perform
the prefetch call during lookup; for file inodes it doesn't buy
us much, and it should only really help for readdir.
This removes the prefetch call from lookup, instead prefetching
upon the first readdir() of a loaded TreeInode.
Reviewed By: simpkins
Differential Revision: D12896022
fbshipit-source-id: 0209eb64bd522daf5f7461dffccd1312d32a1554
Summary:
Now that FileInode read and write operations are stateless via BlobAccess and OverlayFileAccess,
EdenFileHandle no longer provides any value. Remove it. This also fixes eden's shutdown timeout
when a file handle is open and paves the way for FUSE_NO_OPEN_SUPPORT.
Reviewed By: strager
Differential Revision: D13325137
fbshipit-source-id: 71ed47a7c997f5035b4394ccb311f94332ecd8c2
Summary:
Have FileInode use OverlayFileAccess instead of using the Overlay directly.
This allows IO on materialized files to be stateless and pave the way for
eliminating EdenFileHandle. It also paves the way for performance improvements
such as nicer SHA-1 caching.
Reviewed By: strager
Differential Revision: D13325079
fbshipit-source-id: fb27d48b5dc9196dc6e36557596f601194a56aa9
Summary:
If a blob was partially read, evicted from cache, and then read again,
the readByteRanges coverage set was not being cleared. Always clear it
in startLoadingData.
Reviewed By: strager
Differential Revision: D13405267
fbshipit-source-id: 6f60e6e80662fd470fe4ddbc833fc8efd8850686
Summary:
Drop interest in cached blobs at various points in the FileInode
lifecycle.
Reviewed By: strager
Differential Revision: D12991762
fbshipit-source-id: 19fd94938c96485160d547ecbd259ffeb39b2341
Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary:
FUSE_NO_OPEN_SUPPORT is better than ATOMIC_O_TRUNC for Eden's use
case. Remove the code that pretended we might support ATOMIC_O_TRUNC
again someday.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13163382
fbshipit-source-id: 948d701571a8d2977da3d2532fdc9538c5011636
Summary:
It's not clear that this code is a win and either way it will be a
no-op when FUSE_NO_OPEN_SUPPORT is enabled so just remove the prefetch
in open().
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13162205
fbshipit-source-id: a3161c0d042e13bd092fc9589e851be78552fa7a
Summary:
FileInode no longer has a strong reference to a blob. Instead, all accesses go through the blob cache. This diff changes the caching behavior for blobs.
The previous behavior was:
When a file's contents are needed in any way, the blob is loaded and referenced by the inode. When the number of open file handles goes to zero, the blob is dropped. The blob is also dropped when the inode is unloaded. Future inode loads or open requests, in that situation, require the blob to be reloaded from the LocalStore.
The new behavior is:
When a file's contents are needed, the blob is loaded and stored into the BlobCache, evicting any if necessary. Future blob requests are satisfied from the BlobCache, pushing it to the back of the eviction queue. When the inode is materialized or unloaded, the blob will be evicted from cache if no other blob has interest in it.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D12813912
fbshipit-source-id: 20d20807d2e4a6c37cddab38562fdb7456316aac
Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary:
Now that the Overlay no longer serializes timestamps, remove all of
the special-case migration logic.
Reviewed By: strager
Differential Revision: D13144764
fbshipit-source-id: 713a4bfcde9003a8d5a28837cb530b05a9017c22
Summary:
Eden has used the InodeMetadataTable as the primary source of
timestamp data for more than six months. Stop writing timestamps into
the overlay, since they will never be used.
Reviewed By: strager
Differential Revision: D13144696
fbshipit-source-id: e36423036228e89dd2a986e6bacfa74553c17a92
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary:
FileInode::prefetch was entirely redundant - it queried for metadata
upon inode lookup after getattr() was called already (which requires
the blob metadata to be loaded).
Reviewed By: wez
Differential Revision: D12896473
fbshipit-source-id: 9ba5104a43860e1f22b88726b9e3e977d0b50e89
Summary:
FileInode::stat is called very often, even by FUSE operations
as common as lookup(). stat() requires the size, which we don't in
general know until the blob's been imported. That said, if the blob
has been imported once, we don't actually need to decode the entire
blob out of RocksDB - we can much more cheaply read the cached blob
metadata to get the size.
Differential Revision: D10441161
fbshipit-source-id: aafc52b54aca9ba30248420fbc4f2ccf1ec0bed8
Summary:
This code to recompute the SHA-1 in fsync is probably unnecessary. It
shifts some work to the writer of the file from Buck's getSHA1 query,
which may not even occur and also overlaps with a lot of other work
Buck is doing. In addition, computing SHA-1 during fsync is an O(file)
operation, so a series of writes and fsyncs would result in quadratic
behavior.
Differential Revision: D10436219
fbshipit-source-id: 9ea9b027e7676181478c4ffc9d791fed8033255c
Summary:
We weren't doing anything that interesting in flush()
anyway. Precomputing the SHA-1 for materialized files optimizes for a
relatively rare situation that penalizes the writer of large files for
the possibility that Buck might read the files later.
Differential Revision: D10435552
fbshipit-source-id: 24aa8f7d9ec5094b084ebd02964840b4b01ad48b
Summary:
Add a direct getSize() accessor to Blob. The thinking here is that all
of this information is known and in cache when the Blob is
constructed, so there's no need to walk a list later on.
Reviewed By: simpkins
Differential Revision: D10245695
fbshipit-source-id: f6d5abbae75d468085dcc02bbbac8aa6239a7c70
Summary:
Instead of calling getBlobMetadata in multiple places and only using
the .sha1 field, add a getSha1 function directly to ObjectStore. This
gives ObjectStore the latitude to fetch it and store it in different ways.
Reviewed By: wez
Differential Revision: D10227935
fbshipit-source-id: 180830534db3c42c07f04216599e496406af5ced
Summary:
EdenFileHandle is just a thin wrapper around a FileInodePtr, so remove
the last vestiges of interesting logic from it.
Reviewed By: wez
Differential Revision: D10187221
fbshipit-source-id: 327e99ae0d860bcc010e31753e7226f2a6f953fd
Summary: We've diverged in a few places from clang-format, so run it across the entirety of Eden.
Reviewed By: wez
Differential Revision: D10137785
fbshipit-source-id: 9603c2eeddc7472c33041ae60e3e280065095eb7
Summary:
When a file's contents are not cached by the kernel, open(), read(),
and close() are all piped into the FUSE daemon. But when a file's
contents ARE cached by the kernel, only open() and close() are. Thus,
in the common case, the kernel will notify us that a file is being
opened, but read() will be served out of cache.
In that case, prefetching the blob is not beneficial, because it will
be dropped anyway upon close().
In situations where the VFS cache is hot but eden's own caches are
cold, this is probably a win.
Reviewed By: strager
Differential Revision: D10044546
fbshipit-source-id: eeb0854dbff021b2c73f1a42f31a94dd9fcf0837
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
* future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
* future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
* future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
Reviewed By: chadaustin
Differential Revision: D9443286
fbshipit-source-id: be712b58b92dc7422f128713deaf6f46b29b36ce
Summary:
Now that timestamps are read from the inode metadata table, and users
aren't likely to run a pre-metadata-table version, the timestamp data
in the overlay header's no longer needs to be written. So remove that
code which has the bonus of making unloading faster.
Reviewed By: wez
Differential Revision: D9318044
fbshipit-source-id: 27a9a9ee954003940209819466932237a81f8929
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
6/n: Codemod rvalue-future<T>.then(...) to rvalue-future<T>.then(...).
Reviewed By: yfeldblum
Differential Revision: D9152002
fbshipit-source-id: 166475c1dcafb29a11154cbfbdf7e2e1feaf745b
Summary:
To improve the determinism of our C++ tests, I am planning to switch
TestMount to a ManualExecutor. This adds a ManualExecutor constructor
to UnboundedQueueExecutor.
In Rust, I'd use a trait, but a simple class with two constructors works fine.
Reviewed By: strager
Differential Revision: D8846553
fbshipit-source-id: c52752105255503d26f1e65494c32b3f62882e44
Summary:
The `updateOverlayHeader()` only updates the overlay data if the inode is
materialized. This updates the name to clarify that.
(This function name change was previously part of D8884795, and I'm just
splitting it into its own separate diff.)
Reviewed By: bolinfest
Differential Revision: D9011358
fbshipit-source-id: 6024d64a1dee0b5d741bec32ed88f6c8f8dd8a9a
Summary:
Encountering a truncated overlay file doesn't necessarily indicate a software
bug in Eden. Depending on the underlying filesystem this often happens after
a hard system reboot since we write the overlay files without an `fdatasync()`
call.
Change the code to simply log an error and throw an exception rather than
using `EDEN_BUG()`. This makes it possible to exercise this code path in
tests without having it crash in debug builds.
Reviewed By: chadaustin
Differential Revision: D8988209
fbshipit-source-id: 8c0fe1dae692f4c493413d3939d2e4c21e0da596
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
3/n: Codemod rvalue-future<T>.then(callable with operator()(not-a-try)) to rvalue-future<T>.thenValue(callable with operator()(not-a-try)).
Reviewed By: yfeldblum
Differential Revision: D8986716
fbshipit-source-id: 906339d9ffb90b3c38a24ce8bf0cef7be318d946
Summary:
The two FileInode::isSameAs() methods called FileInode::getSha1(), which
returns a folly::Future object, and immediately called `value()` on the
resulting future without waiting for it to complete first.
This bug dates back to D5430128. However, I suspect it was D7888344 that
actually exposed this issue recently. D7888344 updates the
`RocksDbLocalStore` code to perform work in an I/O thread pool. Before this
change the SHA1 futures were likely always immediately ready, so this bug was
not causing problems.
Reviewed By: wez
Differential Revision: D8393671
fbshipit-source-id: ec2116751ddda31a119bfe85eab5612b622f83cf
Summary:
Per code review feedback from D6891479, this diff enforces that
metadata writes and reads are done while the corresponding inode's
state lock is held.
Reviewed By: simpkins
Differential Revision: D7884463
fbshipit-source-id: d0e7a95415c280441276452ece7233d4cbf5e942
Summary:
1. Enabled a number of additional C++ compiler warnings in Eden.
2. Fixed warnings-turned-errors that resulted from this change.
Reviewed By: simpkins
Differential Revision: D8132543
fbshipit-source-id: 2290ffaaab55024d582e29201a1bcaa1152e6b3e
Summary:
FileInode and TreeInode's State classes are complicated enough that
they deserve to be lifted out. In addition, this is necessary for
using the type system to enforce that contents locks are held in
InodeBase's metadata accessors.
Reviewed By: simpkins
Differential Revision: D7867399
fbshipit-source-id: 6ce082149ba02099487e8caed33a7bd8510dfebb
Summary:
Relax the restriction on changing uid/gid on inodes. We'll see what
cans of worms this opens I guess. (Landing this is low priority, but
might be important for making some of the existing tooling in fbsource
and www work.)
Reviewed By: simpkins
Differential Revision: D7768655
fbshipit-source-id: 95fe02fe7ddc001335dbdb34e16a989a85820240
Summary:
Unify how inode metadata is modified across inode types. This allows
changing the permission bits on a directory.
Reviewed By: simpkins
Differential Revision: D7767254
fbshipit-source-id: 35e9cf652c84c7d8680cc22dec7942e94e9f5af1
Summary:
This moves most inode metadata management into InodeBase and
persists permission bits (and eventually uid/gid) across Eden runs.
Reviewed By: simpkins
Differential Revision: D7035163
fbshipit-source-id: 50145449b56aad1662d53156e6e4960c5f7b6166
Summary: Store tree and file timestamps in the InodeTable so they persist across runs.
Reviewed By: simpkins
Differential Revision: D6891479
fbshipit-source-id: 1c9e6266375aceeaf293a81e73cf7f5334dbc32d
Summary:
When comparing two source control blob hashes, identical hashes can be assumed
to mean that the file contents are equal. However, differing hashes does not
necessarily mean that the file contents differ. In particular, mercurial
hashes history metadata in addition to the file contents when computing the
blob hash.
This updates Eden to always compare the file contents when the source control
blob hashes differ, rather than assuming that the file contents are different.
Reviewed By: wez
Differential Revision: D7825900
fbshipit-source-id: e611124a66cdd5c44589f20d1d4665a603286530
Summary:
Promote the folly logging code out of the experimental subdirectory.
We have been using this for several months in a few projects and are pretty
happy with it so far.
After moving it out of the experimental/ subdirectory I plan to update
folly::Init() to automatically support configuring it via a `--logging` command
line flag (similar to the initialization it already does today for glog).
Reviewed By: yfeldblum, chadaustin
Differential Revision: D7755455
fbshipit-source-id: 052db34c97f7516728f7cbb1a5ad959def2f6efb
Summary:
Disallow any kind of mutation operation inside of the .eden directory. We had some
code in place to prevent some of this already, but errors (including EPERM) weren't
passed out from unlink and rename out to FUSE.
Reviewed By: simpkins
Differential Revision: D7781691
fbshipit-source-id: aaecf13779eca75d6ee8765fc8bb3727ce9341de
Summary: The InodeTable work will homogenize mode_t storage, access, and modification across file and tree inodes. In preparation, have InodeBase keep track of the initial mode bits instead of a dtype_t.
Reviewed By: simpkins
Differential Revision: D7031924
fbshipit-source-id: f2e6e4467cecfc0ca06ad998cce0af18a99cc251
Summary: Also print the inode path if the assertions in EXPECT_FILE_INODE fail.
Reviewed By: simpkins
Differential Revision: D7035517
fbshipit-source-id: 6c50acb588d1c985c7e7a1586c06f04a657698f9
Summary: This gets rid of those pesky #ifdefs in FileInode.cpp and TreeInode.cpp.
Reviewed By: wez
Differential Revision: D7735914
fbshipit-source-id: d810461984e21f72670f43ca2d1b4f5aacbf376e
Summary:
Most of the logic in TreeInode::create(), symlink() and mknod() was very
similar. This adds a TreeInode::createImpl() helper method that these three
functions can share.
This also updates TreeInode::create() to throw EEXIST if the file in question
does exist, rather than using EDEN_BUG(). I believe it is possible (but
unlikely) for this to occur in practice since inode invalidation operations
triggered by a checkout are now processed asynchronously.
Reviewed By: chadaustin
Differential Revision: D7411500
fbshipit-source-id: 85d97995139eee6bff96381561fc28e76d7a2b7c
Summary:
Update all Overlay APIs to accept InodeNumbers and compute the path to the
overlay file internally, rather than requiring the caller to pass in the path
to the file inside the overlay.
Reviewed By: chadaustin
Differential Revision: D7411497
fbshipit-source-id: d057babb0b1b8d6c3b9adf1a21b84eed363dffa1
Summary:
This adds a new LockedState helper class in FileInode to also keep track of
whether or not we are currently holding an refcount on the open file/blob. The
LockedState object automatically decrements this refcount on destruction.
Alternatively, this refcount can be transferred to a new EdenFileHandle object
when unlocking the state.
Previously most of the internal FileInode code simply used EdenFileHandle
objects to manage this refcount. However, this is prone to deadlock, since you
have to ensure that EdenFileHandle objects are never destroyed while the state
is already locked. This new LockedState API makes it harder to have code paths
that may accidentally destroy an EdenFileHandle object while holding the state
lock.
Reviewed By: chadaustin
Differential Revision: D7407423
fbshipit-source-id: 610fcda3220a9f49b734910b7a13e8d68a81a779
Summary:
Update InodeMap::onInodeUnreferenced() to take a pointer to a non-const
InodeBase. This allows the methods it calls, including
InodeBase::updateOverlayHeader() to be non-const.
Using non-const InodeBase objects seems to make sense here since the inode is
in the process of being destroyed.
In a future diff I plan to update FileInode::updateOverlayHeader() to share the
same code as normal file methods to ensure that the overlay file is open. This
modifies the FileInode state's open refcount, so it is useful to have this
method be non-const for that purpose.
Reviewed By: chadaustin
Differential Revision: D7407424
fbshipit-source-id: 541656c7b9b283c5e5650445de5bbdbaae3fc57f
