Summary:
We think that it shouldn't really be needed to perform
the prefetch call during lookup; for file inodes it doesn't buy
us much, and it should only really help for readdir.
This removes the prefetch call from lookup, instead prefetching
upon the first readdir() of a loaded TreeInode.
Reviewed By: simpkins
Differential Revision: D12896022
fbshipit-source-id: 0209eb64bd522daf5f7461dffccd1312d32a1554
Summary:
Update the `FuseTest.destroyWithInitRace()` test to succeed even if the future
returned by `startFuse()` completes with an exception.
Even though the test waits to see the `FUSE_INIT` response sent back to the
kernel, there is more initialization work performed by EdenMount after the
`FUSE_RESPONSE` is sent back. This initialization code can potentially fail.
At the moment the initialization code generally succeeds even if the
`EdenMount` has already transitioned to the `SHUT_DOWN` state. However, I
plan to change the `EdenMount` code soon to error out in this case. This
currently will cause this test to fail with its existing behavior.
Reviewed By: strager
Differential Revision: D13503048
fbshipit-source-id: 6ff147d8679559f0520f5e6091291c3a07bba3ed
Summary:
Update the `listMounts()` thrift API to also report the current mount point
state. This will allow us to do a better job of reporting mount points that
are in the process of initializing or shutting down.
This change splits the `MountInfo` thrift type into two distinct types for
the `listMounts()` vs `mount()` APIs. However this change should be
completely backwards compatible at the wire protocol level for older client
and server code.
Reviewed By: strager
Differential Revision: D13503049
fbshipit-source-id: 68e7ca708b956991c8fd93bbf8973d90650aced9
Summary:
Some people encounter system-wide hangs on their Linux machines. Debugging points to a deadlock related to EdenFS' process name lookup code. Disable the process name cache during FUSE dispatch to avoid the deadlock.
Effects:
* Hopefully, the deadlock no longer happens.
* 'eden top' will no longer report process names. (It should work otherwise, though.)
Reviewed By: simpkins
Differential Revision: D13540947
fbshipit-source-id: 595c36150a5f8ff1b8e7cd81d8f61ee1463d96eb
Summary:
The fuse opcodes are defined as an enum so we have to use
the relatively coarse and indirect apple vs linux preprocessor
checks in the maps for the opcode names.
The osxfuse implementation branched off from the 7.19 fuse
implementation, so add a light dusting of some preprocessor
checks around enabling the performance optimization features
we desire on Linux.
We also need to relax the compile time check for the min
fuse version; I've constrained this to be apple specific,
although I suppose it wouldn't hurt to make it more broadly
applicable.
Reviewed By: chadaustin
Differential Revision: D13480145
fbshipit-source-id: 010ac114e22ea942dfcebf1105cb1f01b766f297
Summary:
ported forward from D4209167, add a couple of helpers
to access these fields on mac and linux, centralizing/minimizing ifdefs.
Simplify some of the logic in FileChangeMonitor.
Reviewed By: chadaustin
Differential Revision: D13475717
fbshipit-source-id: d7b39999808bc41a6dc17a87189501cb34e68b30
Summary:
Thankfully, we can simply remove it; it is really just a performance
optimization that we can enable for linux.
Reviewed By: simpkins
Differential Revision: D13475719
fbshipit-source-id: 09b60dcf995c2c5390b91aa316c62ca1b4d3f944
Summary:
It is only 16 bits wide on this system, so we don't need
to borrow bits and there may not be enough bits to borrow from
anyway.
Reviewed By: simpkins
Differential Revision: D13475714
fbshipit-source-id: 1d342c89a3037abd744d97fef21ad421b5e60356
Summary:
Eden no longer tracks any state in file handles, and has no plans to in the future.
Therefore, remove all related code.
Reviewed By: strager
Differential Revision: D13354307
fbshipit-source-id: 341d081f64c6c8fb2b4b1b5a5ff42f2cc7d38039
Summary:
Now that all file access in Eden is stateless, we no longer need to handle open() or release().
If the kernel advertises FUSE_NO_OPEN_SUPPORT, return ENOSYS from open().
Reviewed By: simpkins
Differential Revision: D13325759
fbshipit-source-id: 38486848f27ffeb005f74407888e94d891496f98
Summary:
Previously, a file handle must have been held for the entirety of a write operation. That is no
longer true. Stop looking up file handles on write.
Reviewed By: strager
Differential Revision: D13325662
fbshipit-source-id: 9ae31b467d17d633c388917d18098e6e5a620b89
Summary:
Now that FileInode read and write operations are stateless via BlobAccess and OverlayFileAccess,
EdenFileHandle no longer provides any value. Remove it. This also fixes eden's shutdown timeout
when a file handle is open and paves the way for FUSE_NO_OPEN_SUPPORT.
Reviewed By: strager
Differential Revision: D13325137
fbshipit-source-id: 71ed47a7c997f5035b4394ccb311f94332ecd8c2
Summary:
Have FileInode use OverlayFileAccess instead of using the Overlay directly.
This allows IO on materialized files to be stateless and pave the way for
eliminating EdenFileHandle. It also paves the way for performance improvements
such as nicer SHA-1 caching.
Reviewed By: strager
Differential Revision: D13325079
fbshipit-source-id: fb27d48b5dc9196dc6e36557596f601194a56aa9
Summary:
Move the overlay IO logic out of FileInode into a centralized OverlayFileAccess class. It keeps the last 100 overlay file handles open.
FileInode's changes are coming in D13325079.
Reviewed By: strager
Differential Revision: D13324995
fbshipit-source-id: 04fb3fe50114b0f19b78acd17a9684c92f8e8072
Summary:
Add a fuzz test that randomly modifies a directory during a series of
readdir calls and verifies each unmodified entry is returned once.
Reviewed By: simpkins
Differential Revision: D13371162
fbshipit-source-id: 252b03ab0288e82b56a33c347955de129e61ae42
Summary:
If a blob was partially read, evicted from cache, and then read again,
the readByteRanges coverage set was not being cleared. Always clear it
in startLoadingData.
Reviewed By: strager
Differential Revision: D13405267
fbshipit-source-id: 6f60e6e80662fd470fe4ddbc833fc8efd8850686
Summary:
As I'm working on ripping out file handle support, I wanted to ensure
that we had a path towards a correct and stateless readdir()
implementation. Stateful file handles require extra care during
graceful restart, and it's nice that we can avoid them. In fact, this
work paves the way towards a possible FUSE_NO_OPENDIR_SUPPORT feature.
This diff fixes concurrent rm -rf in the same directory.
Reviewed By: simpkins
Differential Revision: D13370898
fbshipit-source-id: eba650e673a7cb9559e04ba28417980f6d0c65cb
Summary:
Drop interest in cached blobs at various points in the FileInode
lifecycle.
Reviewed By: strager
Differential Revision: D12991762
fbshipit-source-id: 19fd94938c96485160d547ecbd259ffeb39b2341
Summary:
Write tests for readdir's semantics (we really do want to return . and
..) and simplify the logic. It's still quadratic in large directories,
but there aren't any allocations anymore.
Reviewed By: strager
Differential Revision: D13287764
fbshipit-source-id: 5e0d4b86eb16dbd7a16cdeb324e4b43363512e25
Summary:
Stop holding a reference count to the TreeInode while a directory
handle is open. This allows eden to shut down while a directory handle
is open.
Reviewed By: strager
Differential Revision: D13287701
fbshipit-source-id: a24f32a1ac40b6c19bc5864aa5f5785f3016361b
Summary:
Send readdir requests to TreeInode. This may not sound like a good
idea: the FUSE documentation suggests that stateful directory handles
are required to implement correct readdir semantics under concurrent
deletes and renames. However, the 63-bit offset value is treated as a
cookie that is passed from one readdir call into the next, and 63 bits
should be sufficient to implement readdir concurrent with
rename/unlink. So move readdir's implementation into TreeInode in
preparation for the complete removal of TreeInodeDirHandle.
Reviewed By: strager
Differential Revision: D13287664
fbshipit-source-id: c0d615675edd9b83353534468a69b89068bba923
Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary:
Send fsyncdir straight through the inode rather than going through
DirHandle. This is the better design anyway, since the DirHandle does
not receive directory-mutating requests like mkdir.
Reviewed By: strager
Differential Revision: D13287610
fbshipit-source-id: 154fa32a3877c89a204a2d10b4e2b637410d9486
Summary:
FUSE_NO_OPEN_SUPPORT is better than ATOMIC_O_TRUNC for Eden's use
case. Remove the code that pretended we might support ATOMIC_O_TRUNC
again someday.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13163382
fbshipit-source-id: 948d701571a8d2977da3d2532fdc9538c5011636
Summary:
It's not clear that this code is a win and either way it will be a
no-op when FUSE_NO_OPEN_SUPPORT is enabled so just remove the prefetch
in open().
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13162205
fbshipit-source-id: a3161c0d042e13bd092fc9589e851be78552fa7a
Summary:
FileInode no longer has a strong reference to a blob. Instead, all accesses go through the blob cache. This diff changes the caching behavior for blobs.
The previous behavior was:
When a file's contents are needed in any way, the blob is loaded and referenced by the inode. When the number of open file handles goes to zero, the blob is dropped. The blob is also dropped when the inode is unloaded. Future inode loads or open requests, in that situation, require the blob to be reloaded from the LocalStore.
The new behavior is:
When a file's contents are needed, the blob is loaded and stored into the BlobCache, evicting any if necessary. Future blob requests are satisfied from the BlobCache, pushing it to the back of the eviction queue. When the inode is materialized or unloaded, the blob will be evicted from cache if no other blob has interest in it.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D12813912
fbshipit-source-id: 20d20807d2e4a6c37cddab38562fdb7456316aac
Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary:
Now that the Overlay no longer serializes timestamps, remove all of
the special-case migration logic.
Reviewed By: strager
Differential Revision: D13144764
fbshipit-source-id: 713a4bfcde9003a8d5a28837cb530b05a9017c22
Summary:
The linter tripped on D12813838 even though the warning was unrelated
to my changes.
Reviewed By: simpkins
Differential Revision: D13167184
fbshipit-source-id: 555691f00d113ed2bff9488b61392cc92f4395e3
Summary:
Eden has used the InodeMetadataTable as the authoritative source for
timestamp metadata for over six months. I think we can safely assume
that anyone at this point who has old inodes in the overlay that don't
have corresponding entries in the inode metadata table are fine with
those timestamps being reset to the last checkout time.
Reviewed By: strager
Differential Revision: D13144735
fbshipit-source-id: 06a9a8835ea83c98fb6a165e4c8d5c3c6b28ad84
Summary:
Eden has used the InodeMetadataTable as the primary source of
timestamp data for more than six months. Stop writing timestamps into
the overlay, since they will never be used.
Reviewed By: strager
Differential Revision: D13144696
fbshipit-source-id: e36423036228e89dd2a986e6bacfa74553c17a92
Summary: Instantiate a blob cache in the EdenServer with configurable settings.
Reviewed By: strager
Differential Revision: D12813880
fbshipit-source-id: 8ccc89826f04aca78964230374dea48abf05e05e
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary:
chadaustin noticed this as part of fixing up the ESTALE
handling. The issue is that we were using `inodeMap->lookupInode` and
assuming that it will always return one of our magical inodes. This
isn't guaranteed so it is better to resolve the inode by name from
the root, so that's what this diff does.
Reviewed By: chadaustin
Differential Revision: D12970034
fbshipit-source-id: 8207660cbc71577b276cb092d1ef19e1076b4946
Summary:
Update all of the C++ unit tests that create temporary files and directories
to use the new `facebook::eden::makeTempFile()` and
`facebook::eden::makeTempDir()` functions.
Note that this likely changes the behavior of some code paths in meaningful
ways: `/dev/shm` normally does not support `getxattr()`, and Eden's overlay
code attempts to store the SHA-1 for materialized files as using extended
attributes. This means that the tests will now typically hit the fallback
code path and will not store SHA-1 data in the overlay.
Reviewed By: chadaustin, strager
Differential Revision: D12971162
fbshipit-source-id: 6cc5eba2e04be7e9a13a30e90883feadfb78f9ce
Summary:
Thanks to some bpf tracing by strager, we traced the ESTALE response to
`d_splice_alias` and noted this comment above the implementation in the kernel:
> If a non-IS_ROOT directory is found, the filesystem is corrupt, and
> we should error out: directories can't have multiple aliases.
Well, our magic `.eden` directory is a directory with aliases and we were
seeing the error trigger on that dir. So, this diff replaces hardlinking
directories into each tree with a hardlink to a symlink in each tree!
At mount time we create `.eden/this-dir` as a symlink to `/abs/path/to/mount/.eden`
so that `readlink("/abs/path/to/mount/sub/dir/.eden/socket")` still
resolves as it did prior to this diff.
Reviewed By: strager
Differential Revision: D12954819
fbshipit-source-id: 7f3b1b53f2bd5b9c51e64055fc34110657a19110
Summary:
Sandcastle has several cases where we chown the entire
repository which performs terribly on Eden. As a workaround we have a
command to do this in eden without loading all the files.
Reviewed By: chadaustin
Differential Revision: D12857956
fbshipit-source-id: 36cebcc710fbcf4e1eb265df901513cf50a227b9
Summary:
FileInode::prefetch was entirely redundant - it queried for metadata
upon inode lookup after getattr() was called already (which requires
the blob metadata to be loaded).
Reviewed By: wez
Differential Revision: D12896473
fbshipit-source-id: 9ba5104a43860e1f22b88726b9e3e977d0b50e89
Summary:
Eden used to load materialized entries at startup. It no longer does
and this code is dead.
Reviewed By: wez
Differential Revision: D12896210
fbshipit-source-id: 398363724a661f87208cf05313e61755a451edb7
Summary:
This uses the tracing infrastructure to create blocks around
the getSHA1 invocation as well as the getOrLoadChild (called for each
element of the path-walk) method.
Reviewed By: chadaustin
Differential Revision: D10384074
fbshipit-source-id: a06fbe38e8d3f35fcb248e6bd724e5572724d27d
Summary:
This allows detecting changes to the edenrc configuration
and adjusting how we read data accordingly.
To avoid a huge gnarly diff with a lot of dependencies, this diff
introduces a little interface class that defines the method to obtain
a possibly reloaded EdenConfig instance.
This diff shouldn't change any functionality.
Reviewed By: chadaustin
Differential Revision: D12813051
fbshipit-source-id: 35390dc28671ba46a03d098c4f280e2f567dbdc9
Summary:
Checkout.modifyFile is the long pole when running the Eden unit
tests. Splitting it into three runs it across multiple threads. It
could be split further if needed.
Reviewed By: strager
Differential Revision: D12813684
fbshipit-source-id: 49e847d46bc54f16fcb2b273ec81e92eaf756dbb
Summary:
Our CMake-based build had gotten somewhat out of date. This fixes several
issues that were preventing it from building:
- wangle now depends on fizz, which in turn depends on libsodium
- fbthrift now depends on yarpl for the streaming functionality
- Eden now depends on libsqlite3
- The eden/fs/inode library now depends on eden/fs/config
Reviewed By: strager
Differential Revision: D10436531
fbshipit-source-id: f41065da2edb909e8eb94e03d0e62ae19ad3a84f
Summary:
Because Mercurial blob IDs change without the contents changing, and
because files get unloaded upon checkout, rebasing across a large
distance in history can result in status fetching a lot of
metadata. Keep a smallish LRU cache for SHA-1 and size by blob ID.
Reviewed By: strager
Differential Revision: D10419965
fbshipit-source-id: 81499573814775471913db05f924767c3bab300e
