Summary: This diff ports TreeInode, FileInode, InodeMap and related classes to Windows. We don't build or test it here, there are more dependcies we need to port. The built script and the test are part of other diffs in this stack.
Reviewed By: simpkins
Differential Revision: D19956266
fbshipit-source-id: 9eb754233bca3d5a336f465c2400512a8593ca4f
Summary:
This is a rough pass that resolves a linker issue on MSVC by
switching to inline static member functions.
Reviewed By: chadaustin
Differential Revision: D20529163
fbshipit-source-id: 578ed440758c685091d3e039e261638e027db17a
Summary: This diff bumps the open call from FUSE to High priority (which is higher than any other blob open request atm). This has shown improvement on the user experience of EdenFS when it's importing many other things from other channels (thrift, etc.)
Reviewed By: chadaustin
Differential Revision: D20287389
fbshipit-source-id: 319bc44ef8be5c904d7cf0db7cc2f8be28b4760a
Summary:
Migration from Future-returning executor-erasing collectX forms to
SemiFuture-returning forms, that are less risky in particular with coroutines.
Earlier diffs added SemiFuture and Unsafe versions. This codemod migrates
collect versions to the Unsafe versions to allow the basic collect versions to
be made safe.
Reviewed By: simpkins
Differential Revision: D20331206
fbshipit-source-id: efc8dff487d45f7d53ee55e8c4696bd3eed0e6da
Summary: Pass a valid ObjectFetchContext down into certain untracked requests.
Reviewed By: chadaustin
Differential Revision: D20243575
fbshipit-source-id: e7112c3bab1265803a26130c4d72905c25f2e729
Summary:
Add a fetch context interface to ObjectStore that allows tracing cache
hits, backing store fetches, and fetch durations in the context of a
diff or checkout operation.
Reviewed By: simpkins
Differential Revision: D19135625
fbshipit-source-id: d0d8f134b1c89f7ba4971a404a46a69a1704ba5c
Summary:
This splits `EDEN_BUG()` into three separate version. All three crash in
debug mode builds, but in release builds they behave differently:
- `EDEN_BUG()` throws an exception
- `EDEN_BUG_FUTURE(Type)` returns a `folly::Future<Type>` that has been
fulfilled with an exception.
- `EDEN_BUG_EXCEPTION()` returns a `folly::exception_wrapper`.
The main advantage of this is that this allows the compiler to detect that
`EDEN_BUG()` can never return. Previously `EDEN_BUG()` was used for all 3 of
these different cases, and its behavior depended on whether `toException()`
was ever called. As a result we could not easily get the compiler to identify
code paths where we know at compile time that it will never return.
Reviewed By: chadaustin
Differential Revision: D18652103
fbshipit-source-id: 070107c7520f51b05696905fa243de5f8df15958
Summary:
This moves error handling with `OverlayFile` closer to the source of the error with the use of `folly::Expected`.
This is for future IOReference counting but also lets us transform all of these generic `std::system_error`s into InodeErrors (which is a subclass of `std::system_error` that can include inode path in the error message).
Another option for error handling would be to throw folly::checkUnixError inside the `OverlayFile` instead of returning a `folly::Expected`, but I am always hesitant to assume that the user of this class wants the error to be thrown, since they may want to do some other work (cleanup etc) before throwing an error.
Reviewed By: chadaustin
Differential Revision: D18215242
fbshipit-source-id: 5e38fdc7e6efe7d646dd3895932c9b2f26674b36
Summary:
This silences a compiler warning about reaching the end of a non-void function
without returning a value.
Reviewed By: chadaustin
Differential Revision: D17308290
fbshipit-source-id: 95cdb3353364a36dcd2295b19bf745a941e5e3cf
Summary:
This was causing problems on macos where various tools
would enumerate and helpfully try to preserve attributes across
copies. On macos this would result in appledouble metadata files
being created to track the metadata in the destination file,
which clutters up the repo and has surprising secondary effects
such as being picked up by glob operations in cmake build rules.
This diff simply stops enumerating the extended attribute.
Reviewed By: fanzeyi
Differential Revision: D17140414
fbshipit-source-id: 2924657dc75b900baf70595edfa72e5d0521a697
Summary:
Implements size-only local storage, as opposed to storing metadata. This is useful when the blob's SHA-1 is not needed. This diff prevents SHA-1 computations, which can be especially expensive for large blobs.
From D15934535, operations such as `ls -l` and `stat` will get the size of a blob in two ways:
1) The blob's size is already stored locally, so it will be deserialized and returned.
2) The blob is fetched from the backing store, stored, and its size is returned.
This diff optimizes the second case, because SHA-1 is no longer computed.
Reviewed By: strager
Differential Revision: D15723239
fbshipit-source-id: a868f3bf6b68a83ddafb057dc3e4e65f0a2dd989
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary: Making addDelta private and giving users a more user-friendly way of appending entries to the journal.
Reviewed By: chadaustin, strager
Differential Revision: D15868089
fbshipit-source-id: 00c8a3066f0e4483e3c792651ade5f6a7ea05eed
Summary: `getSize` and `getSha1` were misleading function names, since the functions refer to the size and hash of a given blob and not the object store itself. These functions have been renamed to `getBlobSize` and `getBlobSha1`.
Reviewed By: chadaustin
Differential Revision: D15696510
fbshipit-source-id: 4dd31659f60969fa90d8e2b39f43c46a2b7dff7c
Summary:
We think that it shouldn't really be needed to perform
the prefetch call during lookup; for file inodes it doesn't buy
us much, and it should only really help for readdir.
This removes the prefetch call from lookup, instead prefetching
upon the first readdir() of a loaded TreeInode.
Reviewed By: simpkins
Differential Revision: D12896022
fbshipit-source-id: 0209eb64bd522daf5f7461dffccd1312d32a1554
Summary:
Now that FileInode read and write operations are stateless via BlobAccess and OverlayFileAccess,
EdenFileHandle no longer provides any value. Remove it. This also fixes eden's shutdown timeout
when a file handle is open and paves the way for FUSE_NO_OPEN_SUPPORT.
Reviewed By: strager
Differential Revision: D13325137
fbshipit-source-id: 71ed47a7c997f5035b4394ccb311f94332ecd8c2
Summary:
Have FileInode use OverlayFileAccess instead of using the Overlay directly.
This allows IO on materialized files to be stateless and pave the way for
eliminating EdenFileHandle. It also paves the way for performance improvements
such as nicer SHA-1 caching.
Reviewed By: strager
Differential Revision: D13325079
fbshipit-source-id: fb27d48b5dc9196dc6e36557596f601194a56aa9
Summary:
If a blob was partially read, evicted from cache, and then read again,
the readByteRanges coverage set was not being cleared. Always clear it
in startLoadingData.
Reviewed By: strager
Differential Revision: D13405267
fbshipit-source-id: 6f60e6e80662fd470fe4ddbc833fc8efd8850686
Summary:
Drop interest in cached blobs at various points in the FileInode
lifecycle.
Reviewed By: strager
Differential Revision: D12991762
fbshipit-source-id: 19fd94938c96485160d547ecbd259ffeb39b2341
Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary:
FUSE_NO_OPEN_SUPPORT is better than ATOMIC_O_TRUNC for Eden's use
case. Remove the code that pretended we might support ATOMIC_O_TRUNC
again someday.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13163382
fbshipit-source-id: 948d701571a8d2977da3d2532fdc9538c5011636
Summary:
It's not clear that this code is a win and either way it will be a
no-op when FUSE_NO_OPEN_SUPPORT is enabled so just remove the prefetch
in open().
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13162205
fbshipit-source-id: a3161c0d042e13bd092fc9589e851be78552fa7a
Summary:
FileInode no longer has a strong reference to a blob. Instead, all accesses go through the blob cache. This diff changes the caching behavior for blobs.
The previous behavior was:
When a file's contents are needed in any way, the blob is loaded and referenced by the inode. When the number of open file handles goes to zero, the blob is dropped. The blob is also dropped when the inode is unloaded. Future inode loads or open requests, in that situation, require the blob to be reloaded from the LocalStore.
The new behavior is:
When a file's contents are needed, the blob is loaded and stored into the BlobCache, evicting any if necessary. Future blob requests are satisfied from the BlobCache, pushing it to the back of the eviction queue. When the inode is materialized or unloaded, the blob will be evicted from cache if no other blob has interest in it.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D12813912
fbshipit-source-id: 20d20807d2e4a6c37cddab38562fdb7456316aac
Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary:
Now that the Overlay no longer serializes timestamps, remove all of
the special-case migration logic.
Reviewed By: strager
Differential Revision: D13144764
fbshipit-source-id: 713a4bfcde9003a8d5a28837cb530b05a9017c22
Summary:
Eden has used the InodeMetadataTable as the primary source of
timestamp data for more than six months. Stop writing timestamps into
the overlay, since they will never be used.
Reviewed By: strager
Differential Revision: D13144696
fbshipit-source-id: e36423036228e89dd2a986e6bacfa74553c17a92
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary:
FileInode::prefetch was entirely redundant - it queried for metadata
upon inode lookup after getattr() was called already (which requires
the blob metadata to be loaded).
Reviewed By: wez
Differential Revision: D12896473
fbshipit-source-id: 9ba5104a43860e1f22b88726b9e3e977d0b50e89
Summary:
FileInode::stat is called very often, even by FUSE operations
as common as lookup(). stat() requires the size, which we don't in
general know until the blob's been imported. That said, if the blob
has been imported once, we don't actually need to decode the entire
blob out of RocksDB - we can much more cheaply read the cached blob
metadata to get the size.
Differential Revision: D10441161
fbshipit-source-id: aafc52b54aca9ba30248420fbc4f2ccf1ec0bed8
Summary:
This code to recompute the SHA-1 in fsync is probably unnecessary. It
shifts some work to the writer of the file from Buck's getSHA1 query,
which may not even occur and also overlaps with a lot of other work
Buck is doing. In addition, computing SHA-1 during fsync is an O(file)
operation, so a series of writes and fsyncs would result in quadratic
behavior.
Differential Revision: D10436219
fbshipit-source-id: 9ea9b027e7676181478c4ffc9d791fed8033255c
Summary:
We weren't doing anything that interesting in flush()
anyway. Precomputing the SHA-1 for materialized files optimizes for a
relatively rare situation that penalizes the writer of large files for
the possibility that Buck might read the files later.
Differential Revision: D10435552
fbshipit-source-id: 24aa8f7d9ec5094b084ebd02964840b4b01ad48b
Summary:
Add a direct getSize() accessor to Blob. The thinking here is that all
of this information is known and in cache when the Blob is
constructed, so there's no need to walk a list later on.
Reviewed By: simpkins
Differential Revision: D10245695
fbshipit-source-id: f6d5abbae75d468085dcc02bbbac8aa6239a7c70
Summary:
Instead of calling getBlobMetadata in multiple places and only using
the .sha1 field, add a getSha1 function directly to ObjectStore. This
gives ObjectStore the latitude to fetch it and store it in different ways.
Reviewed By: wez
Differential Revision: D10227935
fbshipit-source-id: 180830534db3c42c07f04216599e496406af5ced
Summary:
EdenFileHandle is just a thin wrapper around a FileInodePtr, so remove
the last vestiges of interesting logic from it.
Reviewed By: wez
Differential Revision: D10187221
fbshipit-source-id: 327e99ae0d860bcc010e31753e7226f2a6f953fd
Summary: We've diverged in a few places from clang-format, so run it across the entirety of Eden.
Reviewed By: wez
Differential Revision: D10137785
fbshipit-source-id: 9603c2eeddc7472c33041ae60e3e280065095eb7
Summary:
When a file's contents are not cached by the kernel, open(), read(),
and close() are all piped into the FUSE daemon. But when a file's
contents ARE cached by the kernel, only open() and close() are. Thus,
in the common case, the kernel will notify us that a file is being
opened, but read() will be served out of cache.
In that case, prefetching the blob is not beneficial, because it will
be dropped anyway upon close().
In situations where the VFS cache is hot but eden's own caches are
cold, this is probably a win.
Reviewed By: strager
Differential Revision: D10044546
fbshipit-source-id: eeb0854dbff021b2c73f1a42f31a94dd9fcf0837
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
* future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
* future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
* future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
Reviewed By: chadaustin
Differential Revision: D9443286
fbshipit-source-id: be712b58b92dc7422f128713deaf6f46b29b36ce
Summary:
Now that timestamps are read from the inode metadata table, and users
aren't likely to run a pre-metadata-table version, the timestamp data
in the overlay header's no longer needs to be written. So remove that
code which has the bonus of making unloading faster.
Reviewed By: wez
Differential Revision: D9318044
fbshipit-source-id: 27a9a9ee954003940209819466932237a81f8929
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
6/n: Codemod rvalue-future<T>.then(...) to rvalue-future<T>.then(...).
Reviewed By: yfeldblum
Differential Revision: D9152002
fbshipit-source-id: 166475c1dcafb29a11154cbfbdf7e2e1feaf745b
Summary:
To improve the determinism of our C++ tests, I am planning to switch
TestMount to a ManualExecutor. This adds a ManualExecutor constructor
to UnboundedQueueExecutor.
In Rust, I'd use a trait, but a simple class with two constructors works fine.
Reviewed By: strager
Differential Revision: D8846553
fbshipit-source-id: c52752105255503d26f1e65494c32b3f62882e44
Summary:
The `updateOverlayHeader()` only updates the overlay data if the inode is
materialized. This updates the name to clarify that.
(This function name change was previously part of D8884795, and I'm just
splitting it into its own separate diff.)
Reviewed By: bolinfest
Differential Revision: D9011358
fbshipit-source-id: 6024d64a1dee0b5d741bec32ed88f6c8f8dd8a9a
Summary:
Encountering a truncated overlay file doesn't necessarily indicate a software
bug in Eden. Depending on the underlying filesystem this often happens after
a hard system reboot since we write the overlay files without an `fdatasync()`
call.
Change the code to simply log an error and throw an exception rather than
using `EDEN_BUG()`. This makes it possible to exercise this code path in
tests without having it crash in debug builds.
Reviewed By: chadaustin
Differential Revision: D8988209
fbshipit-source-id: 8c0fe1dae692f4c493413d3939d2e4c21e0da596
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
3/n: Codemod rvalue-future<T>.then(callable with operator()(not-a-try)) to rvalue-future<T>.thenValue(callable with operator()(not-a-try)).
Reviewed By: yfeldblum
Differential Revision: D8986716
fbshipit-source-id: 906339d9ffb90b3c38a24ce8bf0cef7be318d946
Summary:
The two FileInode::isSameAs() methods called FileInode::getSha1(), which
returns a folly::Future object, and immediately called `value()` on the
resulting future without waiting for it to complete first.
This bug dates back to D5430128. However, I suspect it was D7888344 that
actually exposed this issue recently. D7888344 updates the
`RocksDbLocalStore` code to perform work in an I/O thread pool. Before this
change the SHA1 futures were likely always immediately ready, so this bug was
not causing problems.
Reviewed By: wez
Differential Revision: D8393671
fbshipit-source-id: ec2116751ddda31a119bfe85eab5612b622f83cf
Summary:
Per code review feedback from D6891479, this diff enforces that
metadata writes and reads are done while the corresponding inode's
state lock is held.
Reviewed By: simpkins
Differential Revision: D7884463
fbshipit-source-id: d0e7a95415c280441276452ece7233d4cbf5e942
