Summary: Use callback to save ScmStatus instead of storing status inside a `TreeDiffer` object. Involes a bit of restructuring of some code to avoid circular dependencies and library creation. Mostly renames and file moves, some funtion moves as well.
Reviewed By: simpkins
Differential Revision: D17400466
fbshipit-source-id: fcd194a4c20204dffd3d11cd4a083564dc0ea938
Summary: Use callback to save ScmStatus instead of storing status inside a `TreeDiffer` object.
Reviewed By: simpkins
Differential Revision: D17379318
fbshipit-source-id: b5f56c937509f0861f5dc06c742ebc3e1f85c695
Summary:
Use the correct inode pointer when creating an `InodeError`. Previously this
passed in an inode pointer that was always null.
Reviewed By: wez
Differential Revision: D17438864
fbshipit-source-id: 37b7e40723afb9335b7616b585eed3b6cf5b3c81
Summary:
Fix `GlobNode::evaluate()` so that it only generates a result once all
processing is complete. The glob evaluation walks down the inode tree in
parallel. Previously if an error occurred while processing one TreeInode it
would return that exception to the caller immediately, even if work was still
being done to evaluate the glob on other parts of the inode tree. This
behavior is problematic, since the caller may destroy the GlobNode immediately
once a result is generated, and the caller has no way to know that other
children GlobNodes may still be using some of the data.
This fixes the code to wait until all evaluation sub-futures have completed
processing even on error, so that we know it is safe to destroy the GlobNode
before we generate a result.
Reviewed By: wez
Differential Revision: D17434327
fbshipit-source-id: ef70dc82281eeb3540840202ec5e704afe2d29eb
Summary:
Make sure the contents of the special `.eden/` subdirectory are correct each
time we mount a checkout. Before we would generally only set up the contents
of this directory if it didn't previously exist.
Now the code verifies that the contents of each symlink in this directory are
correct, and recreates the symlink if needed.
This allows EdenFS to automatically repair the contents of this directory even
if the checkout or its `clients` directory has been manually moved.
Reviewed By: wez
Differential Revision: D17279413
fbshipit-source-id: e24e7530f44fff94ebb6f67174aaf78c9b498d6b
Summary:
Update the CMakeLists.txt to also build the Python-based `edenfsctl` command
line tool.
This requires switching most of the thrift rules to generate both C++ and
Python sources.
Note that one missing feature at this point is that this does not package
external dependencies into the binary. Currently `edenfsctl` depends on both
`six` and `toml` as external dependencies. For now these must be available in
your `PYTHONPATH` in order to run the generated `edenfsctl` binary.
Reviewed By: chadaustin
Differential Revision: D17127615
fbshipit-source-id: fc138ab39e75c6a5bbd39e3f527d4e9f7f420e46
Summary:
This silences a compiler warning about reaching the end of a non-void function
without returning a value.
Reviewed By: chadaustin
Differential Revision: D17308290
fbshipit-source-id: 95cdb3353364a36dcd2295b19bf745a941e5e3cf
Summary:
This diff passes the free space information through from
the overlay lock file descriptor back up to the kernel when the
filesystem stats are requested.
This makes the user experience with eg: Finder on macOS nicer.
Reviewed By: chadaustin
Differential Revision: D17255859
fbshipit-source-id: ed9f3b9fd386c5706539879513854a9cd1550d8a
Summary:
Update the fsck code to save any orphaned symlink inodes that it finds as
symlinks in the repair archive directory, rather than saving the contents as a
regular file.
Reviewed By: wez
Differential Revision: D17170346
fbshipit-source-id: 4cba8b27233b728114a80a327ab519b039297aea
Summary:
Use the new `OverlayChecker` class to automatically scan for errors and
attempt to repair them if the overlay was not shutdown cleanly the previous
time it was used.
Reviewed By: wez
Differential Revision: D16596601
fbshipit-source-id: 9923565b101ba953e92909e502be6ef5895c5cbd
Summary:
Update the OverlayChecker's `inodes_` map to store `InodeInfo` objects
directly, rather than pointers to `InodeInfo` objects allocated on the heap.
This tweaks some of the error handling code to look up `InodeInfo` objects in
the map in a few places, to avoid storing raw pointers to `InodeInfo` objects.
The `InodeInfo` objects generally shouldn't move once we start performing
error checking, but it seems better to be conservative here and avoid using
raw pointers that are only loosely coupled with the original object's
lifetime.
This probably shouldn't really make much of a performance difference in
practice, since this code is likely to be disk I/O bound anyway. (It does
appear to make a small difference in performance for ASAN-enabled builds when
the inode files are warmed up in the kernel cache.)
Reviewed By: wez
Differential Revision: D16750736
fbshipit-source-id: b089a259cc83ffd2fda6f83b617d95ce4ac467f9
Summary:
We are probably going to add more inodemap counters, so add a prefix
now.
Reviewed By: fanzeyi
Differential Revision: D17142015
fbshipit-source-id: 4bd3cd4fd9234d8766864f364fef0b0d963f03b6
Summary:
This was causing problems on macos where various tools
would enumerate and helpfully try to preserve attributes across
copies. On macos this would result in appledouble metadata files
being created to track the metadata in the destination file,
which clutters up the repo and has surprising secondary effects
such as being picked up by glob operations in cmake build rules.
This diff simply stops enumerating the extended attribute.
Reviewed By: fanzeyi
Differential Revision: D17140414
fbshipit-source-id: 2924657dc75b900baf70595edfa72e5d0521a697
Summary:
This fixes a few issues with the library dependencies:
- The `eden_utils` library depends on `eden_service_thrift`, not
`eden_service`. By incorrectly depending on `eden_service` this introduced
a circular dependency which would cause a build failure, depending on which
order CMake chose to try and emit the link line.
- The `eden_config` library depends on code from `eden_model` (for `Hash` and
`ParentCommits`)
- The `eden_inodes` library depends on `eden_model_git` for the `GitIgnore`
logic. I also alphabetized the dependency list.
Reviewed By: wez
Differential Revision: D17124930
fbshipit-source-id: 70cbe81081fc1dc807cca13a93edc25ba270b01f
Summary:
Rename the `ThriftCppLibrary.cmake` file to `FBThriftCppLibrary.cmake`, and
also rename `add_thrift_cpp2_library()` to `add_fbthrift_cpp_library()`.
Explicitly calling this `fbthrift` helps clearly distinguish that this is
intended for use with fbthrift (https://github.com/facebook/fbthrift/), as
opposed to Apache thrift.
Reviewed By: wez
Differential Revision: D16738440
fbshipit-source-id: 9b255e06b71c98ad74a34989f564a211958dcdd5
Summary:
Implement the repair() logic for InodeDataError, MissingMaterializedInode, and
OrphanInode.
The most common errors we see after an unclean system reboot is an inode file
not being present at all (`MissingMaterializedInode`), or it being present but
empty (`InodeDataError`). When either of these errors occurs for a directory
it also produces `OrphanInode` errors, as all of the children entries in the
directory are no longer part of the directory hierarchy.
This implements repair logic for these three error types. The repair behavior
is largely similar to the Python version in `eden/cli/fsck.py`, with some
minor changes to the output paths in the `lost+found/` repair directory. This
also includes more unit test to better exercise handling when `InodeDataError`
and `MissingMaterializedInode` errors occur inside an orphan subdirectory that
needs to be extracted to `lost+found`
Reviewed By: strager
Differential Revision: D16577696
fbshipit-source-id: 948158c5a0c32b31574d93011281c42051645ad9
Summary:
This diff adds a fuse request timeout configuration setting
and threads it through to both the FuseChannel for our internal processing
and to the privhelper so that we can set an appropriate (slightly larger)
value for the kernel level daemon_timeout setting.
Reviewed By: chadaustin
Differential Revision: D16957552
fbshipit-source-id: a0fecc691d72914b5aebaed8a006dc0d6b0d7d12
Summary:
The definition of RLockedPtr should use a non-const Subclass type, and not a
const one.
Reviewed By: ot
Differential Revision: D15356827
fbshipit-source-id: b8ad41e263f0e15ffa25b0698aa85eab8ca2ccb8
Summary:
The Java environment on Sandcastle macOS does not match corp laptops,
so allow disabling javadeprecated Thrift bindings on macOS for
now. Long-term, the better fix is to standardize the Java runtime and
JDK deployments across both environments so fbcode scripts can rely on
/usr/local/java-runtimes. See discussion in D16656570.
Reviewed By: strager
Differential Revision: D16676813
fbshipit-source-id: c21420bad554402c4c1e0881cabb962856719f5f
Summary:
Implement OverlayChecker::computePath().
This also updates it to return a structure which can better represent
information if the inode is part of an orphaned subtree, which prevents us
from figuring out the full path to the root of the checkout.
Reviewed By: chadaustin
Differential Revision: D16577697
fbshipit-source-id: b543e2210fe8bc561d78194852962bf57ad9644a
Summary:
This ports much of the Python logic in `eden/cli/fsck.py` to C++, so that
the edenfs daemon can invoke it directly during mount initialization when
necessary.
This initial diff just contains the code to scan for errors, without any
repair logic.
This also includes some improvements to the logic for efficiency. For
instance, we compute less state during the initial scan phase. This makes the
scan phase faster, at the expense of making some of the repair steps slightly
more expensive if errors are found. This newer C++ scan logic is
significantly faster than the old Python version: scanning 260,000 inode files
took around 1m20 second for the Python code, and runs in under 3 seconds with
the new C++ code. (On an SSD with the files in question already warm in the
kernel cache.)
Reviewed By: strager
Differential Revision: D16577699
fbshipit-source-id: 2f8713a219ad882848cee0c57dca34f2c7808340
Summary:
Now that we have unsafe access to the contents of a
folly::Synchronized, InodeMap::updateOverlayForUnload can just access
the storage because it knows nobody else has access.
Reviewed By: simpkins
Differential Revision: D16632778
fbshipit-source-id: 4b2c1d6679e2655671d43899407bac1a19f2cab9
Summary:
Update `EdenServer::mount()` to correctly handle errors that occur during the
mount `INITIALIZING` phase. Previously the code did not add error callbacks
to the `Future` result to handle errors during initialization. As a result we
would propagate the exception back to the thrift caller, but the `EdenMount`
object would remain in our mount point list, stuck forever in the
`INITIALIZING` state.
Reviewed By: strager
Differential Revision: D16590032
fbshipit-source-id: 9adbdf05441dad815096b195ece36f3d958c96a9
Summary:
Update `FsOverlay::validateHeader()` to throw `EdenError` exceptions rather
than `std::system_error`. These exceptions are generated when we find invalid
data, rather than from system calls that return errno values. Previously the
code was using `folly::throwSystemErrorExplicit()` and
`folly:throwSystemError()`, and passing in a made-up `EIO` value.
However, the latter two calls were incorrectly using
`folly::throwSystemError()` rather than `throwSystemErrorExplicit()`. As a
result the `EIO` parameter was being treated as part of the error message, and
the code was appending the error description for whatever value `errno`
happened to be set to at the moment.
This fixes the code to just throw `EdenError` exception types, since these
errors aren't caused by a real `errno` value.
Reviewed By: strager
Differential Revision: D16577698
fbshipit-source-id: 9487c34f04da99d397611f005f00f02114b12094
Summary:
Fix `FsOverlay::initOverlay()` to also create the `tmp` subdirectory, which is
required for the `FsOverlay` code to function properly. Previously this
directory was created by the `Overlay` class. However, `FsOverlay` should be
responsible for creating it so it can be used from other locations besides
just `Overlay`.
I also removed the code that checks if the `tmp` directory needs to be created
on restart. The code to create `tmp` has been present for over a year
(D8330070), and we no longer have anyone using EdenFS versions older than
this.
Reviewed By: strager
Differential Revision: D16577700
fbshipit-source-id: 3eaa28003159a03084c4f367c7b96d69fc1702d0
Summary:
There is no point to call `readBE<uint64_t>()` here. The only way these calls
can fail is if the buffer is too short for the data, but we already checked
the buffer length at the top of the function.
Reviewed By: strager
Differential Revision: D16577701
fbshipit-source-id: 8b57cec05453b9b7224470671e032e3aefe41033
Summary: Keep track of the longest query we had to resolve in accumulateRange. This stat will be helpful in determining how large the memory limit of the journal should be since we will know how far back people usually need to go.
Reviewed By: strager
Differential Revision: D16227920
fbshipit-source-id: a41c3b9f16b701cd8165e20409888983b8899dab
Summary: Set up the infrastructure to add in Timeseries to the journal so that we can add in stats for the journal that relate to the whole process. For example, allow us to add in a truncatedRead TimeSeries easily as done in D16017968
Reviewed By: chadaustin
Differential Revision: D16461081
fbshipit-source-id: 964ff32e62aed0369da434793491b857c136b074
Summary: Create a counter for the end-to-end duration of the journal.
Reviewed By: chadaustin
Differential Revision: D15993675
fbshipit-source-id: 260a58a7b37dfa16b82b3df187ad7f7263ff67cf
Summary: Change getLatest such that it just returns Info about the latest delta and not the delta itself.
Reviewed By: strager
Differential Revision: D15931214
fbshipit-source-id: c7a1cf4d62cdd4f9396fab46354eabcbb31f32c0
Summary:
Implements size-only local storage, as opposed to storing metadata. This is useful when the blob's SHA-1 is not needed. This diff prevents SHA-1 computations, which can be especially expensive for large blobs.
From D15934535, operations such as `ls -l` and `stat` will get the size of a blob in two ways:
1) The blob's size is already stored locally, so it will be deserialized and returned.
2) The blob is fetched from the backing store, stored, and its size is returned.
This diff optimizes the second case, because SHA-1 is no longer computed.
Reviewed By: strager
Differential Revision: D15723239
fbshipit-source-id: a868f3bf6b68a83ddafb057dc3e4e65f0a2dd989
Summary:
These allow the cli to setup and tear down mounts and
have the eden server keep track of them.
Reviewed By: strager
Differential Revision: D15707318
fbshipit-source-id: abdb8eaa28c8c67c8211a8af1647efe3a083e998
Summary: We need --edenDir support to run muliple instance of Edenfs, which is required to run the integration tests.
Reviewed By: simpkins
Differential Revision: D15951483
fbshipit-source-id: a516159cdeb5f046f795fc28399a2af5fe8a9f95
Summary: EdenMount.cpp includes folly/Subprocess.h, but uses none of the included symbols. Drop the unnecessary #include.
Reviewed By: simpkins
Differential Revision: D15773136
fbshipit-source-id: 4f30af9a1afcc3258aaa00d519d2aceeda72824f
Summary:
Update the copyright & license headers in thrift files to reflect the
relicensing to GPLv2+
Reviewed By: wez
Differential Revision: D15487082
fbshipit-source-id: 33f68617037f36c07075fb962a16a4d8f55bd6a6
Summary:
Update the copyright & license headers in CMake files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487079
fbshipit-source-id: 715e559464c19a0070d6e55a095b3fc7d61ad2f8
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary: Merge was a function on a JournalDelta that created a new JournalDelta and optionally left it connected to old JournalDeltas. AccumulateRange is a new function on the Journal itself (acting on the latest delta) that creates a JournalDeltaSum (which can't be connected to older Deltas)
Reviewed By: chadaustin
Differential Revision: D15881452
fbshipit-source-id: 573505c1171f78d46afc98f1db9b5b9ee2fff60f
Summary: Making addDelta private and giving users a more user-friendly way of appending entries to the journal.
Reviewed By: chadaustin, strager
Differential Revision: D15868089
fbshipit-source-id: 00c8a3066f0e4483e3c792651ade5f6a7ea05eed
Summary:
Some newer versions of `clang` (such as Apple's version 11) will warn/error out if a constructor or assignment operator
marked `default` is implicitly deleted (e.g., if the object contains a non-moveable/non-copyable member). This diff
removes all such defaulted constructors/assignment operators, which I ran into while building `edenfs` on my Macbook Pro.
Reviewed By: chadaustin, strager
Differential Revision: D15901794
fbshipit-source-id: 794ed8377693a6735bb567635dc919bc678751a4
Summary: `getSize` and `getSha1` were misleading function names, since the functions refer to the size and hash of a given blob and not the object store itself. These functions have been renamed to `getBlobSize` and `getBlobSha1`.
Reviewed By: chadaustin
Differential Revision: D15696510
fbshipit-source-id: 4dd31659f60969fa90d8e2b39f43c46a2b7dff7c
Summary:
Change the `ReloadableConfig::getEdenConfig()` method to accept an enum to
control whether it should reload the configuration, rather than the simpler
bool that it used previously.
The primary motivation for this is to allow us to force a reload to occur in
unit tests and integration tests, even if this operation would otherwise be
within the normal reload throttling time limit.
Reviewed By: strager
Differential Revision: D15572122
fbshipit-source-id: 9c638a415a451f26e7f5923f544724594ed4e0f1
Summary: Setting up structure to get the stats from Eden's Journal to ODS
Reviewed By: chadaustin
Differential Revision: D15684380
fbshipit-source-id: 90debc3f2a09d497201e5e73f85400a994b7afba
Summary: These functions are part of the blocking API, which is being replaced by the Future-based API. They are not being called, so they are unneeded.
Reviewed By: chadaustin, strager
Differential Revision: D15582641
fbshipit-source-id: 11f693040124846515aef7c7ee6ad750e1b5c164
Summary: Developers should be encouraged to use Future-based functions instead of blocking functions in production code. Removing this blocking function makes it less convenient for developers to write blocking code in the future.
Reviewed By: strager
Differential Revision: D15564952
fbshipit-source-id: 3f62db472a59a6487ffe12e48c04178cad84ca61
Summary: Small parts of our code have diverged from our clang-format rules.
Reviewed By: strager
Differential Revision: D15380260
fbshipit-source-id: f668ac22d6c0c5f2468549f2a94dd1c9bb22ce3d
Summary:
Now Eden on Windows could be build using
python.exe .\opensource\fbcode_builder\getdeps.py build eden
Reviewed By: strager
Differential Revision: D15251103
fbshipit-source-id: 8ac1097b234bb27ce478101bfbc9591251e2e359
Summary: tar is in /usr/bin on macOS.
Reviewed By: simpkins
Differential Revision: D15052024
fbshipit-source-id: 5388b2d53f876717310d0f9726b74e8a283d1a89
Summary:
Rename the `ClientConfig` class to `CheckoutConfig`, to help further progress
on updating our terminology from "client" to "checkout".
Reviewed By: chadaustin
Differential Revision: D15162814
fbshipit-source-id: 3575958f1161d5842c6f0ee9e2d2d20ab20b7372
Summary:
Move the `ReloadableConfig` implementation out of `ServerState` and into the
`ReloadableConfig` class itself. This also updates `ServerState` to simply
contain a `ReloadableConfig` member variable rather than inheriting from
`ReloadableConfig`.
This makes it easier to build other utilities that need a `ReloadableConfig`
object but do not have a full `ServerState` object, such as the
`eden_store_util` class and the `zero_blob` helper tool used by the
integration tests.
Reviewed By: chadaustin
Differential Revision: D15162813
fbshipit-source-id: 1ebc0ac6c475f4aa1090dd933ebce16bc222674c
Summary:
If EdenFS' globFiles API is given two patterns, and one pattern is a prefix of the other, EdenFS effectively ignores the longer pattern. Given the patterns `project/src/*` and `project/src/*/*`, when EdenFS encounters `project/src/dir/`, it generates a result (because `project/src/dir/` matches the first pattern) but does not recurse into its children.
The problem caused by an incorrect understanding of the GlobNode::isLeaf_ flag. isLeaf_ means "this GlobNode should generate results", but GlobNode::evaluateImpl understands it to mean "this GlobNode should generate results, and this GlobNode has no children". Given the patterns `project/src/*` and `project/src/*/*`, the GlobNode representing `project/src/*` has isLeaf_=true but also has children.
Fix the bug by not using isLeaf_ for determining whether to recurse, and instead relying on the presence of child GlobNode-s.
Reviewed By: chadaustin
Differential Revision: D15078089
fbshipit-source-id: 1c480d11361f89193b35965266e6873c57181113
Summary:
If globbing returns a directory, that directory doesn't need to be loaded. GlobNode::evaluateImpl is careful to only load directories/trees if necessary. When fixing a bug in GlobNode::evaluateImpl, I accidentally broke this optimization. No test failed, though.
Ensure this optimization works and doesn't regress by writing a simple test.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D15090386
fbshipit-source-id: fe932e4a7ad517ff4c4523d58668f05fe8b0cafb
Summary:
In GlobNode::evaluateImpl, the logic for recursing into child GlobNode-s is duplicated token-for-token for hasSpecials_ and !hasSpecials_. While fixing a bug in GlobNode::evaluateImpl, I was forced to update both copies, which is prone to mistakes.
Fix the duplication by factoring the code into a function.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D15090688
fbshipit-source-id: 65eec62214074139cb8af75026758b631eb89bb2
Summary:
gdb and lldb have a hard time printing recursive structures involving std::vector and std::unique_ptr. This makes debugging GlobNode a pain.
Add a function to print a GlobNode recursively to stderr. This makes it easier to debug GlobNode using printf.
Reviewed By: simpkins
Differential Revision: D15078111
fbshipit-source-id: 3ba211026c7c894676435ade278ae859e1b5d1f6
Summary:
On macOS, the atime, mtime, and ctime fields are named st_atimespec,
etc. Add some portability functions so the inode tests compile on
macOS.
Reviewed By: simpkins
Differential Revision: D15048758
fbshipit-source-id: 6f85b8eb7f7da2f9bd21c9034296cde96b68f740
Summary: Add some includes that are required on macOS.
Reviewed By: simpkins
Differential Revision: D15048681
fbshipit-source-id: d8642dc0977dc8cd8aadc83fc750400a8f917b7a
Summary: Fallout from another diff that never got landed.
Reviewed By: simpkins
Differential Revision: D14938191
fbshipit-source-id: 980728ca6a1d4c4f90fea46aac9525a03d9809e5
Summary:
I'm confused by the naming of EdenThreadStats and ThreadLocalEdenStats. For example, when I see "ThreadLocalEdenStats", I think that such objects can only be accessed by one thread. That's definitely not what "ThreadLocalEdenStats" means!
I think the following naming scheme makes more sense:
* **EdenThreadStats**: Statistics which are updated by one thread. Currently called EdenThreadStats.
* **EdenStats**: Statistics for all threads. Provides access to EdenThreadStats. Currently called ThreadLocalEdenStats.
Implement my preferred scheme: rename ThreadLocalEdenStats to EdenStats.
This diff should not change behavior.
Note: Prior to D14822274, EdenThreadStats was called EdenStats.
Reviewed By: simpkins
Differential Revision: D14822271
fbshipit-source-id: bd20179b1010588e3fc16dc9ed0657d458606f16
Summary:
Ensure that the `TreeInode::createImpl()` code always releases the contents
lock at the end of the scope that was intended to signify the critical
section. Previously this was only unlocked when returning from this scope
successfully.
In particular, the old behavior could deadlock:
- We first created the new child inode.
- If saveOverlayDir() failed, we would throw an exception, causing us to not
release the directory lock yet.
- As we returned in the exception case, the local `inode` variable would be
destroyed before the `contents` argument, causing us to release the child
InodePtr's refcount before releasing the parent directory's lock. Because
we were the only user of the child inode we would end up needing to acquire
the parent inode's content's lock to check if it was unlinked.
This fixes the code to ensure that we always release the tree's contents lock
before the child inode becomes unreferenced.
Reviewed By: chadaustin
Differential Revision: D14848807
fbshipit-source-id: 28dcde842072b9ee1e7c63d54456d849e31af8fe
Summary:
I want to use EdenStats in eden/fs/store/. EdenStats currently lives in eden/fs/fuse/, and making eden/fs/store/ depend upon eden/fs/fuse/ is confusing. (It's also confusing that some code in eden/fs/fuse/ is used on Windows.)
Reorganize the code: move EdenStats into eden/fs/tracing/.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14677337
fbshipit-source-id: af26d214bcc3a9919920fbd4e59e6098fe4e3834
Summary:
While testing out the new getdeps code I found that none
of the include directories from the probed libraries were being used.
The new getdeps installs each dep into its own prefix, whereas the
existing getdeps script installed them into the installation
prefix for eden itself. That meant that they were being implicitly
found from a single include directory.
In addition to this, I encountered linker failures for the pretty
printers; the solution to those was to add appropriate deps for
the modules that depend upon the pretty printers.
Reviewed By: pkaush
Differential Revision: D14638758
fbshipit-source-id: a4c2b4c79603c268e1b1c707a05c3cb0e3f2757b
Summary:
Eden requires no state in its directory handles, so tell the kernel it
doesn't need to send opendir() and releasedir() requests, provided it
has FUSE_NO_OPENDIR_SUPPORT.
Reviewed By: strager
Differential Revision: D13594734
fbshipit-source-id: ebd4b69f4efcd1428a69024c4bdffb1ae455fa40
Summary:
After the kernel added readdir caching, my testing uncovered that Eden
was invalidating TreeInode entries incorrectly when new entries were
added. Change TreeInode to distinguish between directory entry changes
and removals (FUSE_NOTIFY_INVAL_ENTRY) and additions
(FUSE_NOTIFY_INVAL_INODE).
Reviewed By: strager
Differential Revision: D13870422
fbshipit-source-id: 2a6f25bfd9e77436a5aae639fedbfd8a445b2e05
Summary:
Prior to D14398507, two calls to EdenMount::unmount would result in two calls to fuseMountPromise->getFuture(). D14398507 made two calls to EdenMount::unmount result in only one call to fuseMountPromise->getFuture().
Since fuseMountPromise->getFuture() is only called at most once, turn fuseMountPromise into a Promise instead of a SharedPromise.
Reviewed By: simpkins
Differential Revision: D14515070
fbshipit-source-id: 3ecaf9b55f3274046bc840bd5a65481cc552df72
Summary:
I want to make EdenMount robust if mounting and unmounting are requested concurrently. This will make it safer to call EdenMount::destroy in different situations.
Take a step toward that goal by making EdenMount::unmount idempotent; calling EdenMount::unmount multiple times should return the same future.
Changed behaviors:
* If EdenMount::unmount is called multiple times, EdenMount::fuseMount is called at most once, and each call to EdenMount::unmount waits for the EdenMount::fuseMount call to finish.
* **Old behavior**: One of the calls to unmount succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Unchanged behaviors:
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse fails with EdenMountCancelled.
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount waits for fuseMount to finish before calling PrivHelper::fuseUnmount.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::takeoverFuse is in progress, EdenMount::unmount does not cause EdenMount::takeoverFuse to fail.
* If EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If neither EdenMount::startFuse nor EdenMount::takeoverFuse was never called, EdenMount::unmount succeeds and does not call PrivHelper::fuseUnmount.
Reviewed By: simpkins
Differential Revision: D14398507
fbshipit-source-id: 1da5424abc47341e4db3da1c50d425711e177366
Summary:
I want to make EdenMount robust if mounting and unmounting are requested concurrently. This will make it safer to call EdenMount::destroy in different situations.
Take a step toward that goal by making EdenMount::unmount succeed if EdenMount::startFuse was never called.
Changed behaviors:
* If neither EdenMount::startFuse nor EdenMount::takeoverFuse was never called, EdenMount::unmount succeeds and does not call PrivHelper::fuseUnmount.
* **Old behavior**: EdenMount::unmount fails with an exception from PrivHelper.
Unchanged behaviors:
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse fails with EdenMountCancelled.
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount waits for fuseMount to finish before calling PrivHelper::fuseUnmount.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::takeoverFuse is in progress, EdenMount::unmount does not cause EdenMount::takeoverFuse to fail.
* if EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::unmount is called multiple times, one of the calls succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Reviewed By: simpkins
Differential Revision: D14398537
fbshipit-source-id: 9209aad8b2980045fd0f89a5ba149a833579c2cf
Summary:
I want to make EdenMount robust if mounting and unmounting are requested concurrently. This will make it safer to call EdenMount::destroy in different situations.
Take a step toward that goal by making EdenMount::unmount complete only after a concurrent PrivHelper::fuseMount call finishes.
Changed behaviors:
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount waits for fuseMount to finish before calling PrivHelper::fuseUnmount.
* In other words, if EdenMount::startFuse and EdenMount::unmount execute concurrently, either:
* mount(2) is never called (if EdenMount::unmount won the race), or
* mount(2) is called then umount(2) is called (if EdenMount::startFuse won the race).
* **Old behavior**: EdenMount::unmount calls PrivHelper::fuseUnmount immediately and either fails or succeeds.
Unchanged behaviors:
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse fails with EdenMountCancelled.
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount causes the fuseMount to be rolled back (by calling PrivHelper::fuseUnmount) after fuseMount completes.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* if EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::unmount is called multiple times, one of the calls succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Reviewed By: simpkins
Differential Revision: D14398508
fbshipit-source-id: 6f01ce9e36b5802045acf54afe756b80fe102b1e
Summary:
D14398423 made EdenMount::startFuse fail immediately if EdenMount::unmount was ever called. EdenMount::takeoverFuse behaves differently; any prior EdenMount::unmount calls don't cause EdenMount::takeoverFuse to fail.
For consistency, make EdenMount::takeoverFuse fail in the same situations that make EdenMount::startFuse fail.
Changed behaviors:
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse fails with EdenMountCancelled.
* **Old behavior**: takeoverFuse succeeds.
* **Note**: EdenMount::unmount still fails with an exception from PrivHelper.
Unchanged behaviors:
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount causes the fuseMount to be rolled back (by calling PrivHelper::fuseUnmount) after fuseMount completes.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::takeoverFuse is in progress, EdenMount::unmount does not cause EdenMount::takeoverFuse to fail.
* if EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::unmount is called multiple times, one of the calls succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Reviewed By: simpkins
Differential Revision: D14511630
fbshipit-source-id: b77ca5ee39c888e2d50547f76977db7aa5a0331a
Summary:
I want to make EdenMount robust if mounting and unmounting are requested concurrently. This will make it safer to call EdenMount::destroy in different situations.
Take a step toward that goal by making EdenMount::unmount cancel a concurrent EdenMount::startFuse call.
Changed behaviors:
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* **Old behavior**: startFuse might succeed or might fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount causes the fuseMount to be rolled back (by calling PrivHelper::fuseUnmount) after fuseMount completes.
* **Old behavior**: EdenMount::unmount does not change the behavior of EdenMount::startFuse.
* **Note**: EdenMount::unmount still fails with an exception from PrivHelper, despite having an effect on EdenMount::startFuse. This quirk will be addressed by a future diff.
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* **Old behavior**: startFuse succeeds.
* **Note**: EdenMount::unmount still fails with an exception from PrivHelper, despite having an effect on EdenMount::startFuse. This quirk will be addressed by a future diff.
Unchanged behaviors:
* If neither EdenMount::startFuse nor EdenMount::takeoverFuse was never called, EdenMount::unmount fails with an exception from PrivHelper.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount calls PrivHelper::fuseUnmount immediately and either fails or succeeds.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::takeoverFuse is in progress, EdenMount::unmount does not cause EdenMount::takeoverFuse to fail.
* If EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse succeeds.
* If EdenMount::unmount is called multiple times, one of the calls succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Reviewed By: simpkins
Differential Revision: D14398423
fbshipit-source-id: eb86a237859ad2d992de5c901e63dda80d5ea6d5
Summary:
In future diffs, I will introduce some EdenMount tests which need to mock PrivHelper in more ways than are currently supported. For example, I want to mock PrivHelper::fuseUnmount and check how many times fuseUmount is called. Extend the existing MockMountDelegate and FakeFuseMountDelegate classes to support these to-be-written tests.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14418206
fbshipit-source-id: a9b1257bb62d9a46fae0eef89c3fa42718bec142
Summary:
If EdenMount::takeoverFuse is called and createFuseChannel fails, the EdenMount reports that it's in the STARTING state. This is misleading and doesn't match how EdenMount::startFuse handles failures in createFuseChannel.
Extend the `try` block in EdenMount::takeoverFuse to handle exceptions raised from createFuseChannel.
Reviewed By: simpkins
Differential Revision: D14511564
fbshipit-source-id: 8f791efe1221bcfc1042f95054d181c303564086
Summary:
Implement `flush()` and `fsyncdir()` in `EdenDispatcher` and explicitly return
ENOSYS. We intentionally do not need to do any work for these methods.
Providing our an implementation that returns ENOSYS avoids the error log
message in the default `Dispatcher` implementation about the these calls being
unimplemented.
Reviewed By: chadaustin
Differential Revision: D14453245
fbshipit-source-id: 71efe6de6af73a5d705dace0f3439ba2466a50a8
Summary:
The future returned by EdenMount::shutdown() can continue on an arbitrary thread. This has caused at least one deadlock in the past (D14337058).
Prevent users of EdenMount::shutdown() from continuing on the wrong thread by making shutdown return a SemiFuture instead of a Future.
This diff should not change behavior for production code, since all users of EdenMount::shutdown() already use Future<>::via().
Reviewed By: simpkins
Differential Revision: D14389607
fbshipit-source-id: 821d206d01ea3dcb0261cd0c7ca2d591b2c84e7a
Summary:
In this change, we separate the low-level code that manipulates the overlay
into the FsOverlay class. The Overlay class makes use of the FsOverlay and
InodeMetaData table to support its Overlay interfaces. The FsOverlay class
is decoupled from the Overlay class, allowing other classes to manipulate
the overlay independently. We have a need for this in order to add
fsck to the c++ code base : described in T40728883.
Reviewed By: simpkins
Differential Revision: D14218281
fbshipit-source-id: 66c587f2b341579b8075ca5e5eeb4da6ffadf6f5
Summary:
Not sure why gflags is automatically included on Linux and not in
mode/mac, but add the includes nonetheless!
Reviewed By: strager
Differential Revision: D14389458
fbshipit-source-id: 27811ec4bb65b03e73b15bb51de1264dbfe053dc
Summary: Internal Facebook infrastructure is nagging me about some files not having a Facebook copyright notice. Add a notice to these files to make the nagging stop.
Reviewed By: simpkins
Differential Revision: D14173944
fbshipit-source-id: 7234431224fcf4f86ea56ca2f9108f47ef959d87
Summary:
This updates `edenfs` to automatically create the mount point directory
if it does not exist.
Previously the `eden mount` CLI command would automatically create the mount
directory in the Python logic. This adds similar logic to the C++ code, which
handles more situations. In particular this makes it so that `eden start`
will now automatically create missing mount point directories.
Note that the C++ code does not create the `README_EDEN.txt` symlink inside
the mount point if it is missing. We could move that functionality into the
C++ code in the future if needed.
Reviewed By: strager
Differential Revision: D14254699
fbshipit-source-id: bad5634f57fba6e7af3b6a3830eb51ac099b435e
Summary:
Update `Overlay::initialization()` to perform the bulk of the initialization
logic in the GC thread. We re-use the GC thread simply for convenience, since
it already exists and won't have any work to do until initialization has
completed anyway.
After an unclean shutdown this allows edenfs to start and perform the overlay
scans for all of its checkouts in parallel rather than serially.
Reviewed By: wez
Differential Revision: D14154868
fbshipit-source-id: c177cb9f950a6317fd7ea06888bd5b326a55ace7
Summary:
Add an Overlay::initialize() function, and consolidate all Overlay
initializtion logic in this function. Previously some initialization was done
by the Overlay constructor, and some was done in `EdenMount::initialize()`
Overlay::initialize() returns a folly::SemiFuture as it may perform a
non-trivial amount of disk I/O. However, at the moment it currently performs
all I/O in the current thread before it returns. I plan to move this work to
a separate thread in a subsequent diff.
Reviewed By: strager
Differential Revision: D13981140
fbshipit-source-id: b59eaef88012a8e74fcb770a9c93ca3f9bde32a0
Summary:
This updates the `EdenServer` class so that the existing `getMount()` and
`getMountPoints()` APIs only return mounts that have finished initializing.
These APIs are primarily used by the thrift interfaces. In most cases the
callers did not intend to operate on mounts that were still initializing, and
doing so was unsafe. The code could potentially dereference a null pointer if
it tried to access the mount's root inode before the root inode object had
been created.
New `getMountUnsafe()` and `getAllMountPoints()` APIs have been added for call
sites that explicitly want to be able to access mounts that may still be
initializing. Currently the `listMounts()` thrift API is the only location
that needs this.
Reviewed By: strager
Differential Revision: D13981139
fbshipit-source-id: e6168d7a15694c79ca2bcc129dda46f82382e8e9
Summary:
Add a flag to tell edenfs to report successful start-up as soon as the thrift
server is running, without waiting for all mount points to finish being
remounted.
In the future I plan to have edenfs automatically perform an fsck scan of the
overlay for checkouts that were not shut down cleanly. This may cause the
remount to take a significant amount of extra start-up time in some cases.
(This is already true today in some cases even with the simpler scan we do to
re-compute the max inode number.)
I think we will probably want to have systemd invoke edenfs with this option,
so that we do not time out during system start up if some mount points need to
be rescanned.
Reviewed By: strager
Differential Revision: D13522040
fbshipit-source-id: 6f183770c25efee34c4805c9bad42a9cce51039e
Summary:
In the future, I want some tests to mock PrivHelper::fuseUnmount. The existing classes for mocking PrivHelper::fuseMount, MountPromiseDelegate and FailingMountDelegate, work poorly for fuseUnmount.
Combine MountPromiseDelegate and FailingMountDelegate to create MockMountDelegate, which is more flexible and allows a mocking fuseUnmount to be implemented easily.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14319712
fbshipit-source-id: 6eecf121e3b44f39cd5dffbeafaf0f3aab75cb76
Summary: Ensure that an EdenMount is RUNNING after it takes over an EdenMount from another process.
Reviewed By: simpkins
Differential Revision: D14178500
fbshipit-source-id: 2813b024b6815dc5d31f3e3bf89cffa98ad0f1c3
Summary:
Update `edenfs` to automatically create the bind mount source directories if
they are missing. Previously Eden would report an error and would not be able
to mount the checkout if some of the bind mount source directories were
missing.
Reviewed By: strager
Differential Revision: D14253771
fbshipit-source-id: 87ad091ccf2c0f0f72aebb50437fd7680ddbfd1c
Summary: Fix some missing includes that showed up in my mode/mac builds of parts of Eden.
Reviewed By: simpkins, strager
Differential Revision: D14213572
fbshipit-source-id: 54e070e89afdfb8479abdaa122ba76ca1d2d9ba9
Summary:
EdenMount calls fuseMount, but EdenServer calls fuseUnmount. I think only one of EdenMount or EdenServer should call fuseMount/fuseUnmount. Splitting the responsibility is confusing.
Move the call to fuseUnmount into EdenMount by creating a member function called unmount.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14115385
fbshipit-source-id: 845c4a87cdba9f270c0eacaf01916ad91c3b7c0e
Summary:
Update `EdenMount::initialize()` to perform a fault injection check. This
allows test code to inject delays and errors into the mount initialization
flow.
Reviewed By: strager
Differential Revision: D14079491
fbshipit-source-id: be80135b0833c8f0300104524473cc3e949fec34
Summary:
If EdenMount::shutdown is in progress and EdenMount::destroy is called [1], the EdenMount object won't ever be deleted. A comment in EdenMount::destroy claims that EdenMount::shutdown will delete the EdenMount object, but the comment is wrong:
case State::SHUTTING_DOWN:
// Nothing else to do. shutdown() will destroy us when it completes.
return;
This leak is probably benign, but the leak checker can cause tests to fail spuriously. I think the leak currently can't happen for EdenServer's mounts, but in the future the leak might occur and cause problems.
Make EdenMount::shutdown delete the EdenMount if EdenMount::destroy was called. This fixes the leak.
[1] EdenMount::destroy is called by EdenMountDeleter when the last std::shared_ptr<EdenMount> is destroyed.
Reviewed By: simpkins
Differential Revision: D14015024
fbshipit-source-id: 39a0ba78bb00afb5f0363fe5c74dbec00f9d68e3
Summary:
If EdenMount::shutdownImpl and EdenMount::startFuse execute concurrently, the mount's `state_` might end up inconsistent. For example, for the following sequence of events, EdenMount::shutdownImpl is erroneously called twice:
1. EdenMount::initialize transitions the mount's state to INITIALIZED.
2. EdenMount::startFuse begins and transitions the mount's state to STARTING.
3. EdenMount::shutdown begins and transitions the mount's state to SHUTTING_DOWN.
4. EdenMount::shutdown finishes and transitions the mount's state to SHUT_DOWN.
5. EdenMount::startFuse finishes due to FuseChannel::initialize failing with an error and transitions the mount's state to FUSE_ERROR.
6. EdenMountDeleter calls EdenMount::destroy, which observes that the mount's state is FUSE_ERROR. EdenMount::destroy proceeds by calling EdenMount::shutdownImpl.
This specific scenario can't happen in production right now because step 3 requires allowFuseNotStarted to be true (and it's never true in production).
A similar scenario can happen with EdenMount::startFuse+EdenMount::destroy running currently, but the effects are benign. However, in a future diff (D14015024), I am changing how EdenMount::destroy works, and the SHUT_DOWN -> FUSE_ERROR transition causes problems.
When FuseChannel::initialize returns an error, transition the mount to the FUSE_ERROR state only if we are not already shutting down (instead of unconditionally). This fixes the [impossible] scenario above, and also fixes EdenMount::destroy in a future diff.
Reviewed By: simpkins
Differential Revision: D14030898
fbshipit-source-id: fe5a749b8798749734e0de91eb3b71b601ebb64b
Summary:
This diff adds the dtype field to the glob results;
this will help to reduce the cost of some watchman queries by avoiding a
getFileInformation call that instantiates inodes.
As part of this, I added a bunch of unit test coverage.
Reviewed By: strager
Differential Revision: D8779149
fbshipit-source-id: 3064a3e42be55ec576fed9e0f7112edef426f32d
Summary:
Add a new class for injecting faults into Eden. This will make it easier to
build tests that exercise specific error conditions or specifically control
the ordering of operations that might otherwise race.
To minimize the performance impact, the fault injection framework is disabled
by default and must be explicitly enabled at process startup using a command
line flag. If this command line flag was not specified all fault injection
checks short-circuit into a no-op. The `checkAsync()` version does still add
some extra overhead due to the addition of a new `folly::SemiFuture` into the
code path.
Reviewed By: wez
Differential Revision: D14079489
fbshipit-source-id: 3e8725d2e51e5c829199a602e90574424ea52636
Summary:
If PrivHelper::fuseMount fails with an error, EdenMount::getState returns STARTING. This is confusing. Make EdenMount::getState instead return FUSE_ERROR in this case.
Aside from changing the output of 'eden list' in some cases, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14060982
fbshipit-source-id: ae9c2c532b1654cce36b806971053d89231058c3
Summary:
While moving code around in EdenMount::startFuse, EdenMount::shutdown, and EdenMount::destroy, I accidentally changed the result of EdenMount::getState in some circumstances. I noticed that we don't have any tests for EdenMount::getState, so it could have been broken even before my changes!
Write some tests for EdenMount::getState to verify the existing behavior and to prevent future regressions. Only add tests which pass with the current implementation; future diffs will fixe cases where EdenMount::getState returns the wrong value.
Reviewed By: chadaustin
Differential Revision: D14058334
fbshipit-source-id: 68b60ac92f94d336fcfb8f11e571379573bb3f42
Summary:
Some tests want to create a TestMount but don't care what files are in the mount. The following code is intuitively correct. However, TestMount's constructor takes a FakeTreeBuilder by non-const reference, so it fails to compile:
auto testMount = TestMount{FakeTreeBuilder{}};
The following code works, but more verbose:
auto builder = FakeTreeBuilder{};
auto testMount = TestMount{builder};
Overload TestMount's constructor to allow the former form in addition to the latter form. This neatly shrinks some test code.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14060110
fbshipit-source-id: c6a5d4b2c5859812efff279dedbc1fe690c8f8ad
Summary:
Make it easier to write tests involving a FakeFuse-mounted EdenMount by creating the TestMount::startFuseAndWait function which initializes EdenMount.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14059808
fbshipit-source-id: ce63bbce7f2c6e710910ad530aa1b4efefe80272
Summary:
Differ.h was including an overly-broad header when all it needed was
the type definitions. This makes building the inodes target with
mode/mac easier.
Reviewed By: simpkins, strager
Differential Revision: D14036477
fbshipit-source-id: 99f3a55d0523c179f6cf2aa4210119ae6f8e51f8
Summary:
Some tests set up fuse_init_in manually then call FakeFuse::sendRequest. The specific fuse_init_in setup is benign and already exists in FakeFuse::sendInitRequest. Shorten these tests by making them call FakeFuse::sendInitRequest instead.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14059700
fbshipit-source-id: 210dcfdc4ae1c74e10c50dbc3f3044c3b908da0a
Summary:
getInitialMode and getModeUnsafe were not inherently different, so
remove getModeUnsafe and write an accurate comment in Overlay's
serialization code about why the initial mode is saved and how we
could remove that logic in the future.
Reviewed By: strager
Differential Revision: D14007488
fbshipit-source-id: db42f45f00dcd213fabd9575360da1261931778b
Summary:
It's common for code to use InodeNumber without needing to include the
main FUSE headers or vice versa. Split them into two separate headers.
Reviewed By: strager
Differential Revision: D13979868
fbshipit-source-id: c5eeb6a3697bb538729a403434dc4f0f7408cda0
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: yfeldblum
Differential Revision: D13908257
fbshipit-source-id: 8b5315b019290f1c60087ca5716c31ebbf1f1be5
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: yfeldblum
Differential Revision: D13784772
fbshipit-source-id: 1d3ede848b7d31c7a197a21b4ff2b31e840040a5
Summary:
Now that the deadlock in ProcessNameCache has been fixed, bring it
back.
Reviewed By: strager
Differential Revision: D13742965
fbshipit-source-id: f407105e06b9954766bdb48ef1303e2003c07284
Summary:
This updates the logic in EdenServer to add the EdenMount to the mountPoints_
map as soon as it is created, so that we track mount points as they are
initializing.
I don't expect this change to have any major impact in functionality yet. In
a subsequent diff I also plan have EdenServer keep mount points in the
mountPoints_ map longer while they are shutting down. I expect that change to
matter a bit more, as that will allow us to do a better job reporting and
debugging when mount points are taking a non-trivial amount of time to become
unreferenced and fully shut down.
Reviewed By: strager
Differential Revision: D13503050
fbshipit-source-id: 2e0e8dfde64c6a005efd6dcf503ad7577f314356
Summary:
Migrate the code from accessing optional Thrift fields directly to a safer
`optional_field_ref` API. See https://fburl.com/safe for more details.
The output of this codemod has been reviewed in D13259011.
To preserve semantics, each unchecked access is replaced with an explicit call
to `value_unchecked()`. If you are sure that accessing a field is safe (the
field is marked as set), you can later replace `value_unchecked()` with
`value()` or dereferencing (`operator *`):
```
ThriftStruct s = ...
- auto foo = s.foo_ref().value_unchecked();
+ auto foo = *s.foo_ref(); // will throw if s.foo is unset
```
Reviewed By: chadaustin
Differential Revision: D13684410
fbshipit-source-id: 919de4ddf89e7f0463f2614baba4bfbac1c8255c
Summary: the build breaks when making clean unless we declare this dep
Reviewed By: simpkins
Differential Revision: D13679633
fbshipit-source-id: f23a533eab9e37fdeab839e4f5e1b6b312ea10b0
Summary: Small things I noticed while working on other stuff.
Reviewed By: strager
Differential Revision: D10055671
fbshipit-source-id: de8c3b04928567a821172e6fa7ee0e056958e1e7
Summary:
If a file in the overlay is truncated unexpectedly, reads on it would
fail with EIO and a log message of "std::out_of_range: string
underflow" which wasn't helpful. Instead, use the same truncated file
detection as directories use.
Reviewed By: strager
Differential Revision: D13627565
fbshipit-source-id: 246f2659ba139e8f7adb7d556719e5ead9d84ebd
Summary:
Add `ensureStateTransition()` and `unconditionalStateTransition()` helper
methods to EdenMount, for performing state transitions where we want to verify
the current state and where we do not care about the current state,
respectively. Update most call sites that change the state to use these
methods.
Also change the code to use acq_rel semantics for updating the state,
rather than seq_cst, which is not necessary in this case. We do not care
about having a consistent visible ordering of change to the state variable
relative to other atomic variables.
Reviewed By: strager
Differential Revision: D13579333
fbshipit-source-id: 5fd62e740a7ea2777f79f722bbde7f5b65255cb6
Summary:
This changes prefetch so that it loads all of the direct children of
the tree. This improves `time ls -l bigdir` performance by 2x.
Reviewed By: wez
Differential Revision: D12888690
fbshipit-source-id: eb8c8274bd9c5b5edc94d7092a5feb492fad6d66
Summary:
We think that it shouldn't really be needed to perform
the prefetch call during lookup; for file inodes it doesn't buy
us much, and it should only really help for readdir.
This removes the prefetch call from lookup, instead prefetching
upon the first readdir() of a loaded TreeInode.
Reviewed By: simpkins
Differential Revision: D12896022
fbshipit-source-id: 0209eb64bd522daf5f7461dffccd1312d32a1554
Summary:
Update the `FuseTest.destroyWithInitRace()` test to succeed even if the future
returned by `startFuse()` completes with an exception.
Even though the test waits to see the `FUSE_INIT` response sent back to the
kernel, there is more initialization work performed by EdenMount after the
`FUSE_RESPONSE` is sent back. This initialization code can potentially fail.
At the moment the initialization code generally succeeds even if the
`EdenMount` has already transitioned to the `SHUT_DOWN` state. However, I
plan to change the `EdenMount` code soon to error out in this case. This
currently will cause this test to fail with its existing behavior.
Reviewed By: strager
Differential Revision: D13503048
fbshipit-source-id: 6ff147d8679559f0520f5e6091291c3a07bba3ed
Summary:
Update the `listMounts()` thrift API to also report the current mount point
state. This will allow us to do a better job of reporting mount points that
are in the process of initializing or shutting down.
This change splits the `MountInfo` thrift type into two distinct types for
the `listMounts()` vs `mount()` APIs. However this change should be
completely backwards compatible at the wire protocol level for older client
and server code.
Reviewed By: strager
Differential Revision: D13503049
fbshipit-source-id: 68e7ca708b956991c8fd93bbf8973d90650aced9
Summary:
Some people encounter system-wide hangs on their Linux machines. Debugging points to a deadlock related to EdenFS' process name lookup code. Disable the process name cache during FUSE dispatch to avoid the deadlock.
Effects:
* Hopefully, the deadlock no longer happens.
* 'eden top' will no longer report process names. (It should work otherwise, though.)
Reviewed By: simpkins
Differential Revision: D13540947
fbshipit-source-id: 595c36150a5f8ff1b8e7cd81d8f61ee1463d96eb
Summary:
The fuse opcodes are defined as an enum so we have to use
the relatively coarse and indirect apple vs linux preprocessor
checks in the maps for the opcode names.
The osxfuse implementation branched off from the 7.19 fuse
implementation, so add a light dusting of some preprocessor
checks around enabling the performance optimization features
we desire on Linux.
We also need to relax the compile time check for the min
fuse version; I've constrained this to be apple specific,
although I suppose it wouldn't hurt to make it more broadly
applicable.
Reviewed By: chadaustin
Differential Revision: D13480145
fbshipit-source-id: 010ac114e22ea942dfcebf1105cb1f01b766f297
Summary:
ported forward from D4209167, add a couple of helpers
to access these fields on mac and linux, centralizing/minimizing ifdefs.
Simplify some of the logic in FileChangeMonitor.
Reviewed By: chadaustin
Differential Revision: D13475717
fbshipit-source-id: d7b39999808bc41a6dc17a87189501cb34e68b30
Summary:
Thankfully, we can simply remove it; it is really just a performance
optimization that we can enable for linux.
Reviewed By: simpkins
Differential Revision: D13475719
fbshipit-source-id: 09b60dcf995c2c5390b91aa316c62ca1b4d3f944
Summary:
It is only 16 bits wide on this system, so we don't need
to borrow bits and there may not be enough bits to borrow from
anyway.
Reviewed By: simpkins
Differential Revision: D13475714
fbshipit-source-id: 1d342c89a3037abd744d97fef21ad421b5e60356
Summary:
Eden no longer tracks any state in file handles, and has no plans to in the future.
Therefore, remove all related code.
Reviewed By: strager
Differential Revision: D13354307
fbshipit-source-id: 341d081f64c6c8fb2b4b1b5a5ff42f2cc7d38039
Summary:
Now that all file access in Eden is stateless, we no longer need to handle open() or release().
If the kernel advertises FUSE_NO_OPEN_SUPPORT, return ENOSYS from open().
Reviewed By: simpkins
Differential Revision: D13325759
fbshipit-source-id: 38486848f27ffeb005f74407888e94d891496f98
Summary:
Previously, a file handle must have been held for the entirety of a write operation. That is no
longer true. Stop looking up file handles on write.
Reviewed By: strager
Differential Revision: D13325662
fbshipit-source-id: 9ae31b467d17d633c388917d18098e6e5a620b89
Summary:
Now that FileInode read and write operations are stateless via BlobAccess and OverlayFileAccess,
EdenFileHandle no longer provides any value. Remove it. This also fixes eden's shutdown timeout
when a file handle is open and paves the way for FUSE_NO_OPEN_SUPPORT.
Reviewed By: strager
Differential Revision: D13325137
fbshipit-source-id: 71ed47a7c997f5035b4394ccb311f94332ecd8c2
Summary:
Have FileInode use OverlayFileAccess instead of using the Overlay directly.
This allows IO on materialized files to be stateless and pave the way for
eliminating EdenFileHandle. It also paves the way for performance improvements
such as nicer SHA-1 caching.
Reviewed By: strager
Differential Revision: D13325079
fbshipit-source-id: fb27d48b5dc9196dc6e36557596f601194a56aa9
Summary:
Move the overlay IO logic out of FileInode into a centralized OverlayFileAccess class. It keeps the last 100 overlay file handles open.
FileInode's changes are coming in D13325079.
Reviewed By: strager
Differential Revision: D13324995
fbshipit-source-id: 04fb3fe50114b0f19b78acd17a9684c92f8e8072
Summary:
Add a fuzz test that randomly modifies a directory during a series of
readdir calls and verifies each unmodified entry is returned once.
Reviewed By: simpkins
Differential Revision: D13371162
fbshipit-source-id: 252b03ab0288e82b56a33c347955de129e61ae42
Summary:
If a blob was partially read, evicted from cache, and then read again,
the readByteRanges coverage set was not being cleared. Always clear it
in startLoadingData.
Reviewed By: strager
Differential Revision: D13405267
fbshipit-source-id: 6f60e6e80662fd470fe4ddbc833fc8efd8850686
Summary:
As I'm working on ripping out file handle support, I wanted to ensure
that we had a path towards a correct and stateless readdir()
implementation. Stateful file handles require extra care during
graceful restart, and it's nice that we can avoid them. In fact, this
work paves the way towards a possible FUSE_NO_OPENDIR_SUPPORT feature.
This diff fixes concurrent rm -rf in the same directory.
Reviewed By: simpkins
Differential Revision: D13370898
fbshipit-source-id: eba650e673a7cb9559e04ba28417980f6d0c65cb
Summary:
Drop interest in cached blobs at various points in the FileInode
lifecycle.
Reviewed By: strager
Differential Revision: D12991762
fbshipit-source-id: 19fd94938c96485160d547ecbd259ffeb39b2341
Summary:
Write tests for readdir's semantics (we really do want to return . and
..) and simplify the logic. It's still quadratic in large directories,
but there aren't any allocations anymore.
Reviewed By: strager
Differential Revision: D13287764
fbshipit-source-id: 5e0d4b86eb16dbd7a16cdeb324e4b43363512e25
Summary:
Stop holding a reference count to the TreeInode while a directory
handle is open. This allows eden to shut down while a directory handle
is open.
Reviewed By: strager
Differential Revision: D13287701
fbshipit-source-id: a24f32a1ac40b6c19bc5864aa5f5785f3016361b
Summary:
Send readdir requests to TreeInode. This may not sound like a good
idea: the FUSE documentation suggests that stateful directory handles
are required to implement correct readdir semantics under concurrent
deletes and renames. However, the 63-bit offset value is treated as a
cookie that is passed from one readdir call into the next, and 63 bits
should be sufficient to implement readdir concurrent with
rename/unlink. So move readdir's implementation into TreeInode in
preparation for the complete removal of TreeInodeDirHandle.
Reviewed By: strager
Differential Revision: D13287664
fbshipit-source-id: c0d615675edd9b83353534468a69b89068bba923
Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary:
Send fsyncdir straight through the inode rather than going through
DirHandle. This is the better design anyway, since the DirHandle does
not receive directory-mutating requests like mkdir.
Reviewed By: strager
Differential Revision: D13287610
fbshipit-source-id: 154fa32a3877c89a204a2d10b4e2b637410d9486
Summary:
FUSE_NO_OPEN_SUPPORT is better than ATOMIC_O_TRUNC for Eden's use
case. Remove the code that pretended we might support ATOMIC_O_TRUNC
again someday.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13163382
fbshipit-source-id: 948d701571a8d2977da3d2532fdc9538c5011636
Summary:
It's not clear that this code is a win and either way it will be a
no-op when FUSE_NO_OPEN_SUPPORT is enabled so just remove the prefetch
in open().
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13162205
fbshipit-source-id: a3161c0d042e13bd092fc9589e851be78552fa7a
Summary:
FileInode no longer has a strong reference to a blob. Instead, all accesses go through the blob cache. This diff changes the caching behavior for blobs.
The previous behavior was:
When a file's contents are needed in any way, the blob is loaded and referenced by the inode. When the number of open file handles goes to zero, the blob is dropped. The blob is also dropped when the inode is unloaded. Future inode loads or open requests, in that situation, require the blob to be reloaded from the LocalStore.
The new behavior is:
When a file's contents are needed, the blob is loaded and stored into the BlobCache, evicting any if necessary. Future blob requests are satisfied from the BlobCache, pushing it to the back of the eviction queue. When the inode is materialized or unloaded, the blob will be evicted from cache if no other blob has interest in it.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D12813912
fbshipit-source-id: 20d20807d2e4a6c37cddab38562fdb7456316aac
Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary:
Now that the Overlay no longer serializes timestamps, remove all of
the special-case migration logic.
Reviewed By: strager
Differential Revision: D13144764
fbshipit-source-id: 713a4bfcde9003a8d5a28837cb530b05a9017c22
Summary:
The linter tripped on D12813838 even though the warning was unrelated
to my changes.
Reviewed By: simpkins
Differential Revision: D13167184
fbshipit-source-id: 555691f00d113ed2bff9488b61392cc92f4395e3
Summary:
Eden has used the InodeMetadataTable as the authoritative source for
timestamp metadata for over six months. I think we can safely assume
that anyone at this point who has old inodes in the overlay that don't
have corresponding entries in the inode metadata table are fine with
those timestamps being reset to the last checkout time.
Reviewed By: strager
Differential Revision: D13144735
fbshipit-source-id: 06a9a8835ea83c98fb6a165e4c8d5c3c6b28ad84
Summary:
Eden has used the InodeMetadataTable as the primary source of
timestamp data for more than six months. Stop writing timestamps into
the overlay, since they will never be used.
Reviewed By: strager
Differential Revision: D13144696
fbshipit-source-id: e36423036228e89dd2a986e6bacfa74553c17a92
Summary: Instantiate a blob cache in the EdenServer with configurable settings.
Reviewed By: strager
Differential Revision: D12813880
fbshipit-source-id: 8ccc89826f04aca78964230374dea48abf05e05e
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary:
chadaustin noticed this as part of fixing up the ESTALE
handling. The issue is that we were using `inodeMap->lookupInode` and
assuming that it will always return one of our magical inodes. This
isn't guaranteed so it is better to resolve the inode by name from
the root, so that's what this diff does.
Reviewed By: chadaustin
Differential Revision: D12970034
fbshipit-source-id: 8207660cbc71577b276cb092d1ef19e1076b4946
Summary:
Update all of the C++ unit tests that create temporary files and directories
to use the new `facebook::eden::makeTempFile()` and
`facebook::eden::makeTempDir()` functions.
Note that this likely changes the behavior of some code paths in meaningful
ways: `/dev/shm` normally does not support `getxattr()`, and Eden's overlay
code attempts to store the SHA-1 for materialized files as using extended
attributes. This means that the tests will now typically hit the fallback
code path and will not store SHA-1 data in the overlay.
Reviewed By: chadaustin, strager
Differential Revision: D12971162
fbshipit-source-id: 6cc5eba2e04be7e9a13a30e90883feadfb78f9ce
Summary:
Thanks to some bpf tracing by strager, we traced the ESTALE response to
`d_splice_alias` and noted this comment above the implementation in the kernel:
> If a non-IS_ROOT directory is found, the filesystem is corrupt, and
> we should error out: directories can't have multiple aliases.
Well, our magic `.eden` directory is a directory with aliases and we were
seeing the error trigger on that dir. So, this diff replaces hardlinking
directories into each tree with a hardlink to a symlink in each tree!
At mount time we create `.eden/this-dir` as a symlink to `/abs/path/to/mount/.eden`
so that `readlink("/abs/path/to/mount/sub/dir/.eden/socket")` still
resolves as it did prior to this diff.
Reviewed By: strager
Differential Revision: D12954819
fbshipit-source-id: 7f3b1b53f2bd5b9c51e64055fc34110657a19110
Summary:
Sandcastle has several cases where we chown the entire
repository which performs terribly on Eden. As a workaround we have a
command to do this in eden without loading all the files.
Reviewed By: chadaustin
Differential Revision: D12857956
fbshipit-source-id: 36cebcc710fbcf4e1eb265df901513cf50a227b9
Summary:
FileInode::prefetch was entirely redundant - it queried for metadata
upon inode lookup after getattr() was called already (which requires
the blob metadata to be loaded).
Reviewed By: wez
Differential Revision: D12896473
fbshipit-source-id: 9ba5104a43860e1f22b88726b9e3e977d0b50e89
Summary:
Eden used to load materialized entries at startup. It no longer does
and this code is dead.
Reviewed By: wez
Differential Revision: D12896210
fbshipit-source-id: 398363724a661f87208cf05313e61755a451edb7
Summary:
This uses the tracing infrastructure to create blocks around
the getSHA1 invocation as well as the getOrLoadChild (called for each
element of the path-walk) method.
Reviewed By: chadaustin
Differential Revision: D10384074
fbshipit-source-id: a06fbe38e8d3f35fcb248e6bd724e5572724d27d
Summary:
This allows detecting changes to the edenrc configuration
and adjusting how we read data accordingly.
To avoid a huge gnarly diff with a lot of dependencies, this diff
introduces a little interface class that defines the method to obtain
a possibly reloaded EdenConfig instance.
This diff shouldn't change any functionality.
Reviewed By: chadaustin
Differential Revision: D12813051
fbshipit-source-id: 35390dc28671ba46a03d098c4f280e2f567dbdc9
Summary:
Checkout.modifyFile is the long pole when running the Eden unit
tests. Splitting it into three runs it across multiple threads. It
could be split further if needed.
Reviewed By: strager
Differential Revision: D12813684
fbshipit-source-id: 49e847d46bc54f16fcb2b273ec81e92eaf756dbb
Summary:
Our CMake-based build had gotten somewhat out of date. This fixes several
issues that were preventing it from building:
- wangle now depends on fizz, which in turn depends on libsodium
- fbthrift now depends on yarpl for the streaming functionality
- Eden now depends on libsqlite3
- The eden/fs/inode library now depends on eden/fs/config
Reviewed By: strager
Differential Revision: D10436531
fbshipit-source-id: f41065da2edb909e8eb94e03d0e62ae19ad3a84f
Summary:
Because Mercurial blob IDs change without the contents changing, and
because files get unloaded upon checkout, rebasing across a large
distance in history can result in status fetching a lot of
metadata. Keep a smallish LRU cache for SHA-1 and size by blob ID.
Reviewed By: strager
Differential Revision: D10419965
fbshipit-source-id: 81499573814775471913db05f924767c3bab300e
Summary:
FileInode::stat is called very often, even by FUSE operations
as common as lookup(). stat() requires the size, which we don't in
general know until the blob's been imported. That said, if the blob
has been imported once, we don't actually need to decode the entire
blob out of RocksDB - we can much more cheaply read the cached blob
metadata to get the size.
Differential Revision: D10441161
fbshipit-source-id: aafc52b54aca9ba30248420fbc4f2ccf1ec0bed8
Summary:
Eden supports reading the SHA-1 of a file via getxattr. Unfortunately,
it returned ENOSYS if you called getxattr on a directory inode. This
caused FUSE to fail all future getxattr requests with EOPNOTSUPP.
In addition to fixing that, this diff makes our xattr handling a
little more consistent across inodes:
- setxattr() always fails with ENOSYS
- removexattr() always fails with ENOSYS
- listxattr() is always handled by the corresponding inode class
- getxattr() is always handled by the corresponding inode class
Differential Revision: D10437723
fbshipit-source-id: a1ea1e92d3412abb15e91057becea3160a17f1e2
Summary:
This code to recompute the SHA-1 in fsync is probably unnecessary. It
shifts some work to the writer of the file from Buck's getSHA1 query,
which may not even occur and also overlaps with a lot of other work
Buck is doing. In addition, computing SHA-1 during fsync is an O(file)
operation, so a series of writes and fsyncs would result in quadratic
behavior.
Differential Revision: D10436219
fbshipit-source-id: 9ea9b027e7676181478c4ffc9d791fed8033255c
Summary:
We weren't doing anything that interesting in flush()
anyway. Precomputing the SHA-1 for materialized files optimizes for a
relatively rare situation that penalizes the writer of large files for
the possibility that Buck might read the files later.
Differential Revision: D10435552
fbshipit-source-id: 24aa8f7d9ec5094b084ebd02964840b4b01ad48b
Summary: Update some deprecated `Future::then()` calls to the new methods.
Reviewed By: chadaustin
Differential Revision: D10436529
fbshipit-source-id: 357ebd21606af6273ff0d622d6badfca6d8fb903
Summary:
The next version of LLVM that will be released to fbcode will include two new
warnings: `-Wreturn-std-move` and `-Wreturn-std-move-in-c++11`. Although these
are disabled globally in fbcode (see T34353622), `eden/BUILD_MODE.bzl`
re-enables these due to its `-Wextra` option.
Fix the `-Wreturn-std-move` violations.
Alternatively, I considered having `eden/BUILD_MODE.bzl` explicitly disable
these warnings, but simply fixing the warnings seemed like the path of least
resistance.
Reviewed By: taewookoh
Differential Revision: D10376364
fbshipit-source-id: 376dc3a216e2c7eb71934bb0015e852dda37bdfd
Summary:
Using boost::filesystem to create the bind mount directories causes
edenfs to communicate with itself via FUSE, which causes edenfs to die
at startup (sometimes?). Instead, create the directories by traversing
the TreeInodes.
Reviewed By: simpkins
Differential Revision: D10154921
fbshipit-source-id: 4d8be71911e8a988b6fc5796904856f58a9a5153
Summary:
This explains why some people ended up with broken bind mounts. If
bind mount creation ever failed, eden start would silently ignore the
error, leaving buck-out as a mutable directory in the Eden overlay
itself.
Reviewed By: wez, strager
Differential Revision: D10118612
fbshipit-source-id: ce99df4216f6815c24dc8283e55a9a30d58d29a1
Summary: Small change that makes the use case more explicit.
Reviewed By: wez
Differential Revision: D10245746
fbshipit-source-id: cb8312e5f5eddf2dfe0a4c8b98748932201ece18
Summary:
Add a direct getSize() accessor to Blob. The thinking here is that all
of this information is known and in cache when the Blob is
constructed, so there's no need to walk a list later on.
Reviewed By: simpkins
Differential Revision: D10245695
fbshipit-source-id: f6d5abbae75d468085dcc02bbbac8aa6239a7c70
Summary:
Instead of calling getBlobMetadata in multiple places and only using
the .sha1 field, add a getSha1 function directly to ObjectStore. This
gives ObjectStore the latitude to fetch it and store it in different ways.
Reviewed By: wez
Differential Revision: D10227935
fbshipit-source-id: 180830534db3c42c07f04216599e496406af5ced
Summary: Always send write requests straight to the inode rather than going through FileHandle.
Reviewed By: wez
Differential Revision: D10220619
fbshipit-source-id: 9ce328583cf0fa9d7d8850d92d9e15ddc382d6a3
Summary:
Always send read requests straight to the inode rather than going
through the FileHandle.
Reviewed By: wez
Differential Revision: D10220604
fbshipit-source-id: 6aa5d20f3ce09696a29bd5c1cb95d0b987ab213c
Summary:
Always send setattr and getattr straight to the inode rather than
going through the FileHandle.
Reviewed By: wez
Differential Revision: D10187876
fbshipit-source-id: 4c3aaa977cd568d5f9cc4b28583e164119c07c1b
Summary:
EdenFileHandle is just a thin wrapper around a FileInodePtr, so remove
the last vestiges of interesting logic from it.
Reviewed By: wez
Differential Revision: D10187221
fbshipit-source-id: 327e99ae0d860bcc010e31753e7226f2a6f953fd
Summary: Eliminate the need to look up an InodeNumber from a FileHandle. Instead, simply preserve the mapping when it's created.
Reviewed By: wez
Differential Revision: D10187120
fbshipit-source-id: dc47f7776294871ff2398f33c31bd85d240ead50
Summary:
In order to implement FUSE_NO_OPEN_SUPPORT, we must eliminate
FileHandle and FileHandleBase. They didn't add any value anyway. Start
clipping.
Reviewed By: wez
Differential Revision: D10187103
fbshipit-source-id: 81e226f9c12486e0bbbde99b798b169fa31740c2
Summary:
Part of the larger project to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
future<T>.then(callable with operator()(folly::Try<T>)) to future<T>.thenTry(callable)
Reviewed By: Orvid
Differential Revision: D10227086
fbshipit-source-id: 1bb31c91cc65e28291e39302627f97801bfde15c
Summary: We've diverged in a few places from clang-format, so run it across the entirety of Eden.
Reviewed By: wez
Differential Revision: D10137785
fbshipit-source-id: 9603c2eeddc7472c33041ae60e3e280065095eb7
Summary:
When a file's contents are not cached by the kernel, open(), read(),
and close() are all piped into the FUSE daemon. But when a file's
contents ARE cached by the kernel, only open() and close() are. Thus,
in the common case, the kernel will notify us that a file is being
opened, but read() will be served out of cache.
In that case, prefetching the blob is not beneficial, because it will
be dropped anyway upon close().
In situations where the VFS cache is hot but eden's own caches are
cold, this is probably a win.
Reviewed By: strager
Differential Revision: D10044546
fbshipit-source-id: eeb0854dbff021b2c73f1a42f31a94dd9fcf0837
Summary:
My eden crashed with an invariant violation in InodeMap. It looks
related to background unloading, so I attempted to reproduce the
sequence in a unit test.
These tests pass but they're worth checking in I think.
Reviewed By: strager
Differential Revision: D9998970
fbshipit-source-id: d182fc8d5b185c286082320adda3ea4c862f2f18
Summary: This function is never used.
Reviewed By: strager
Differential Revision: D9998991
fbshipit-source-id: 27a8f5180d7516c3bf61b11192672142f77abccc
Summary:
While looking into the ESTALE issue, I was tracing through
the kernel code. I found that the root inode is set up with a generation
of 0 rather than 1 in this portion of the kernel code:
diffusion/LK/browse/v4.11-fb/fs/fuse/inode.c;abf7d7755299ee009f0c3cba40fcdcd038a212ac$650
The consequences of the generation being different are that the kernel
will report an ESTALE result when returning the dentry:
diffusion/LK/browse/v4.11-fb/fs/fuse/inode.c;abf7d7755299ee009f0c3cba40fcdcd038a212ac$690
This diff zero-initializes the entry result struct and explicitly sets the generation field to 0.
Sadly, this did not impact the manifestation of the ESTALE behavior that I've been running down.
It's also worth noting that the source of the `1` was from reading this comment in libfuse:
0a519c9772/include/fuse_lowlevel.h (L78-L79)
> The generation must be non-zero, otherwise FUSE will treat it as an error.
That isn't true of the kernel component.
Reviewed By: chadaustin
Differential Revision: D9944646
fbshipit-source-id: 9c1e2f4faec40a3aa446a4646d4518a854a1d73c
Summary:
If the overlay file for a directory is corrupted (e.g. empty), Overlay::scanForNextInodeNumber throws. This causes Eden to crash on start [1]. Fix the crash by ignoring corrupted directories.
[1] `test_mount_possible_after_corrupt_directory_and_cached_next_inode_number` reproduces this crash.
Reviewed By: chadaustin
Differential Revision: D9806105
fbshipit-source-id: 1b95083b6a6aa253a2296d6f754edbf4b9f64734
Summary:
Part of the larger project to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
future<T>.then(callable with operator()(folly::Try<T>)) to future<T>.thenTry(callable)
Reviewed By: Orvid
Differential Revision: D9819578
fbshipit-source-id: f9e31f47354c041ecbf0a90953cbe50ebfda6adc
Summary: When a RawOverlayTest test fails, it's sometimes difficult to diagnose the failure because assertions use inode numbers instead of file paths. When a test fails, include inode data (with file paths) in the failure message.
Reviewed By: chadaustin
Differential Revision: D9806106
fbshipit-source-id: 6160632bf8c64ceeb84e9d4709347e9268747ca4
Summary:
Reduce some code duplication in RawOverlayTest:
* Factor testDir_-to-AbsolutePath conversion into a getLocalDir function.
* Factor Overlay construction into a loadOverlay function.
* Use getLocalDir to construct the path to next-inode-number.
This refactor will let me use getLocalDir for more things in future diffs.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D9813443
fbshipit-source-id: ab0dc88d36e91fc04e0aeb48468060b93b48f0ec
Summary:
Eden keeps track of the the highest-allocated inode number using the next-inode-number file. The TODO comment in Overlay::scanForNextInodeNumber is thus out of date. Delete the outdated comment since the feature has already been implemented.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D9806107
fbshipit-source-id: f546eb179b86a11a6c4bf0d5f945dc7dbbf29160
Summary:
The checkout operation is a bit hamstrung, performance-wise, when
inodes are loaded. If it's possible the kernel has a reference to an
inode, we can't short-circuit the checkout operation and instead must
precisely add and remove children from the source control tree and
persist the new children and inode numbers into the overlay.
It's much faster if we treat checkout as an inode-number-invalidating
change and unload all inodes that aren't referenced by the kernel.
Reviewed By: strager
Differential Revision: D9781532
fbshipit-source-id: 50912cf89dbe9ae1bd7ef91fdf7bf8ee5eda8667
Summary: Add yet another unloading code path... This one is used for fast checkouts.
Reviewed By: strager
Differential Revision: D9781956
fbshipit-source-id: ae89377aea823f94e2ec1bcc2fa209c8f9bc821c
Summary:
unloadChildrenNow would only unload files or trees at the leaf. Apply
the fixes from D8302998 to unloadChildrenNow, which is only called
during takeover.
Reviewed By: strager
Differential Revision: D9774970
fbshipit-source-id: c2f4d1e6b838cc3b9e99eb8786114e643128a519
Summary: Begin tracking pids passed into FUSE in the ProcessAccessLog.
Reviewed By: strager
Differential Revision: D9595795
fbshipit-source-id: 02e5fefebcd0de860274409ba6258f14fa050b55
Summary:
This method is called on every lookup and getSha1 (anything
that seeks a path) and can be a pretty contended lock, so let's make
it optimistically attempt a read lock.
Reviewed By: chadaustin
Differential Revision: D9635505
fbshipit-source-id: c84751efb7a952b2cf458e06ffc8ee670638f9bb
Summary:
The desired end state for how next inode number gets passed across
takeover. This should not land as is - the prior diffs need to land
and wait for a while until everyone transitions.
Reviewed By: simpkins
Differential Revision: D8195550
fbshipit-source-id: 2fc40f881cc1a331df95ef99f7e9e4f2e69c8643
Summary:
Prior to this diff, to read the inode metadata from an InodeBase, you
had to cast it to a FileInode or TreeInode. Add a public getMetadata
call that acquires the appropriate lock depending on whether it's a
FileInode or TreeInode.
Reviewed By: simpkins
Differential Revision: D9327180
fbshipit-source-id: 66594fb639f1ea72a0e48dcb79234e14d87ec4a2
Summary: Modifies a few callsites in Eden that indirect through a separately allocated lambda to replace Future::then with Future::thenValue.
Reviewed By: yfeldblum
Differential Revision: D9485137
fbshipit-source-id: 84fa03a736a0faa162372e221625d82813a68620
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
* future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
* future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
* future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
Reviewed By: chadaustin
Differential Revision: D9443286
fbshipit-source-id: be712b58b92dc7422f128713deaf6f46b29b36ce
Summary:
This diff is first in the series to make Eden work on Windows. It includes:
1. HG backing store and Object store, which provides the capability to talk to mercurial and fetch the file and folder contents on Windows.
2. Subprocess and Pipe definition for Windows.
3. The Visual studio solution and projects files to compile Eden and scm datapack.
Few Important points:
1. Most of the changes to existing code is done under a macro EDEN_WIN so that it doesn't impact on other platform.
2. Sqlite is used for caching the fetched contents. We are not using Rocksdb on Windows.
3. The main function only calls some test code and exit after printing the output.
4. The initializeMononoke code is disabled for Windows because it needs Proxygen to talk HTTP. Will enable this once I get Proxygen and other dependencies working.
5. HgImporter pass Windows handles to hg_import_helper as command line args. The code to convert these handles into fds is in a separate diff.
Reviewed By: wez
Differential Revision: D8653992
fbshipit-source-id: 52a3c3750425fb92c2a7158c2c214a9372661e13
Summary:
As a prerequisite to running inode unloading code in a background
queue, use a deterministic executor in TestMount to make sequencing in
unit tests reliable.
Reviewed By: simpkins
Differential Revision: D9323878
fbshipit-source-id: 0b85632c1637a8cf83d6f238675e5b6bbb6923c7
Summary:
For O(1) stack space JournalDelta destruction, we need to check
whether a reference-counted pointer has a unique refcount. It looks at
first glance like shared_ptr<T>::unique() gives us that, but it uses a
relaxed load and is now deprecated.
Sadly, the easiest thing is to make a new smart pointer class. (It
happens to be slightly more efficient, requiring one fewer heap
allocation per entry in the chain.)
Reviewed By: simpkins
Differential Revision: D9355314
fbshipit-source-id: 8c782ba9e0ec27fae90325079c199e1b82df88fa
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Several of folly::Future's methods are lvalue-qualified even though they act as though they are rvalue-qualified, that is, they provide a postcondition that says, in effect, callers should act as though the method invalidated its `this` object (regardless of whether that invalidation was actual or logical).
* This violates the C++ principle to "Express ideas directly in code" (see Core Guidelines), and generally makes it more confusing for callers as well as hiding the actual semantics from tools (linters, compilers, etc.).
* This dichotomy and confusion has manifested itself by some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* The goal of rvalueification is to make sure methods that are logically rvalue-qualified are actually rvalue-qualified, which forces callsites to acknowledge that rvalueification, e.g., `std::move(f).ensure(...)` instead of `f.ensure(...)`. This syntactic change in the callsites forces callers to acknowledge the method's rvalue semantics.
This diff started as a Codemod, then required manual fixes. Here were the codemod steps:
* expr.ensure(...) ==> std::move(expr).ensure(...) // if expr is not already an xvalue
* expr->ensure(...) ==> std::move(*expr).ensure(...)
Note: operator precedence of that last step is safe - no need to parenthesize `expr`. Reason: `->` binds more tightly than unary `*`.
Reviewed By: yfeldblum
Differential Revision: D9332070
fbshipit-source-id: 882121fe82c05fdb196ce676db686b6bc254974b
Summary:
Now that timestamps are read from the inode metadata table, and users
aren't likely to run a pre-metadata-table version, the timestamp data
in the overlay header's no longer needs to be written. So remove that
code which has the bonus of making unloading faster.
Reviewed By: wez
Differential Revision: D9318044
fbshipit-source-id: 27a9a9ee954003940209819466932237a81f8929
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Several of folly::Future's methods are lvalue-qualified even though they act as though they are rvalue-qualified, that is, they provide a postcondition that says, in effect, callers should act as though the method invalidated its `this` object (regardless of whether that invalidation was actual or logical).
* This violates the C++ principle to "Express ideas directly in code" (see Core Guidelines), and generally makes it more confusing for callers as well as hiding the actual semantics from tools (linters, compilers, etc.).
* This dichotomy and confusion has manifested itself by some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* The goal of rvalueification is to make sure methods that are logically rvalue-qualified are actually rvalue-qualified, which forces callsites to acknowledge that rvalueification, e.g., `std::move(f).unit(...)` instead of `f.unit(...)`. This syntactic change in the callsites forces callers to acknowledge the method's rvalue semantics.
Codemod changes:
* expr.unit(...) ==> std::move(expr).unit(...) // if expr is not already an xvalue
* expr->unit(...) ==> std::move(*expr).unit(...)
Note: operator precedence of that last step is safe - no need to parenthesize `expr`. Reason: `->` binds more tightly than unary `*`.
Reviewed By: LeeHowes
Differential Revision: D9347419
fbshipit-source-id: 2773365f3793d977f2bad1c0a85ef394633e7d2c
Summary:
Watchman's Eden integration has a bug where the combination of
Watchman querying Eden for overlapping delta ranges ("give me changes
between X and Y, now changes between X+1 and Y+1") and Eden eliding
redundant change events ("add-modify-remove" -> []) results in
Watchman sometimes reporting that a file exists in its final
subscription update when it no longer does.
The fix is to never elide events, even for files that were added and
removed in the same sequence. To continue to support Watchman's `new`
flag, track whether a file existed at the beginning and end of a
journal delta.
Reviewed By: wez
Differential Revision: D9304964
fbshipit-source-id: f34c12b25f2b24e3a0d46fc94aa428528f4c5098
Summary:
The root inode is particularly important, so always call `fdatasync()` on its
changes in the overlay before committing them.
This will hopefully reduce the number of cases we see where users have empty
or corrupt data for the root inode after hard rebooting their server. My
guess is that the root directory is being modified by hg or other tools
creating and removing temporary files in the root directory. If a change like
this is in progress when a hard reboot has been performed we risk data loss
without the `fdatasync()` call.
While Eden can still mostly serve the checkout data if other files or
directories are corrupt/missing in the overlay it currently is completely
unable to mount the checkout if the root overlay is corrupt. Therefore it
seems worth being more cautious about making sure that the root overlay data
is updated atomically.
Reviewed By: chadaustin, wez
Differential Revision: D9275852
fbshipit-source-id: b1e3eeb94ba670d0e2b52da4af7143d3ddbc919b
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
6/n: Codemod rvalue-future<T>.then(...) to rvalue-future<T>.then(...).
Reviewed By: yfeldblum
Differential Revision: D9152002
fbshipit-source-id: 166475c1dcafb29a11154cbfbdf7e2e1feaf745b
Summary:
To improve the determinism of our C++ tests, I am planning to switch
TestMount to a ManualExecutor. This adds a ManualExecutor constructor
to UnboundedQueueExecutor.
In Rust, I'd use a trait, but a simple class with two constructors works fine.
Reviewed By: strager
Differential Revision: D8846553
fbshipit-source-id: c52752105255503d26f1e65494c32b3f62882e44
Summary:
This code is no longer necessary since introduction of FileChangeMonitor and
CachedParsedFileMonitor.
Reviewed By: simpkins
Differential Revision: D8915599
fbshipit-source-id: 7871dac1f7985968bd4d04a4dfb42684892a2e72
Summary:
[Eden] EdenMount should make use of EdenConfig in order to determine the user and system ignore files to load.
This commit does the following:
- adds EdenConfig to ServerState that gets passed to the MountPoint.
- removes hard-coded system ignore settings.
- allows the ignore files to be updated in the user and system config.
- has the diff context loaded with actual ignore file contents.
- adds the FileChangeMonitor class to efficiently (throttled) identify when file changes have occured and reload as necessary.
Reviewed By: simpkins
Differential Revision: D8876214
fbshipit-source-id: d2697c130d3d4960c7f645ace226e5ce6b772048
Summary:
The `updateOverlayHeader()` only updates the overlay data if the inode is
materialized. This updates the name to clarify that.
(This function name change was previously part of D8884795, and I'm just
splitting it into its own separate diff.)
Reviewed By: bolinfest
Differential Revision: D9011358
fbshipit-source-id: 6024d64a1dee0b5d741bec32ed88f6c8f8dd8a9a
Summary:
Update `InodeMap::updateOverlayForUnload()` to catch exceptions that occur
trying to save state to the overlay. If something goes wrong when saving
state there is not much we can do other than log an error. We still want to
unload the inode, and code that is unloading inodes generally cannot deal with
exceptions at this point. In particular if this error occurred while trying
to shut down an EdenMount the code would crash.
Reviewed By: chadaustin
Differential Revision: D8884795
fbshipit-source-id: c2f850f13d775be4b0a0a10f9df3948c7b2c8f4a
Summary:
Encountering a truncated overlay file doesn't necessarily indicate a software
bug in Eden. Depending on the underlying filesystem this often happens after
a hard system reboot since we write the overlay files without an `fdatasync()`
call.
Change the code to simply log an error and throw an exception rather than
using `EDEN_BUG()`. This makes it possible to exercise this code path in
tests without having it crash in debug builds.
Reviewed By: chadaustin
Differential Revision: D8988209
fbshipit-source-id: 8c0fe1dae692f4c493413d3939d2e4c21e0da596
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
3/n: Codemod rvalue-future<T>.then(callable with operator()(not-a-try)) to rvalue-future<T>.thenValue(callable with operator()(not-a-try)).
Reviewed By: yfeldblum
Differential Revision: D8986716
fbshipit-source-id: 906339d9ffb90b3c38a24ce8bf0cef7be318d946
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use
Future<T>::thenTry or Future<T>::thenValue.
1/n: Codemod rvalue-future<T>.then(callable with operator()(Try<T>)) to rvalue-future<T>.thenTry(callable with operator()(Try<T>)).
Reviewed By: simpkins
Differential Revision: D8961903
fbshipit-source-id: ff17b7833d240c221197cdf0bf914b8a39f80b07
Summary:
InodePath is difficult to work with because it's just an alias to std::array. Make InodePath look more like std::string so using InodePath instances is more intuitive.
This diff should not change behaviour.
Reviewed By: chadaustin
Differential Revision: D8923073
fbshipit-source-id: 85a2548f0cfa61e50b6590048084076b9bece3da
Summary:
Make use of the TopLevelIgnores class to hold the system and user
GitIgnoreStack details. This is a cleaner implementation making ownership
semantics more intuitive. In later commits we will provide acess to the
TopLevelIgnores as part of the ServerState. It will be dynamically loaded.
Reviewed By: simpkins
Differential Revision: D8906226
fbshipit-source-id: d955436582498861ac4b4113a47f357432c8f32e
Summary:
The raw Inode pointer in a DirEntry is more of an association than
ownership, so add comments and have clearInode return the old value.
Reviewed By: strager
Differential Revision: D8842315
fbshipit-source-id: d401dcdf4955ea335b39c2a57b0bedb1f83fdf9b
Summary:
Add a class to encapsulate details of top-level system and user ignore files.
This approach has the following benefits:
- simplifies memory management of the GitIgnore;
- separate structure that can be created and provided as part of ServerState
interface;
We will make use of it later in this commit stack.
Reviewed By: simpkins
Differential Revision: D8876064
fbshipit-source-id: 35c82b918c09e58068370401883edd8474dd3fbf
Summary:
While trying to make destroyWithInitRace non-flaky, it uncovered an
ASAN violation in the case that fuseCompleteFuture finishes during
EdenMount destruction. In that case, path_ gets destroyed prior to the
executor, so it's illegal to construct TakeoverData::MountInfo from
path_. This diff removes path_ entirely and reads it from
ClientConfig.
Reviewed By: simpkins, strager
Differential Revision: D8848663
fbshipit-source-id: f9368aa9eec7dfa8f2897cce55fad6d19723e30c
Summary:
We should only kick off prefetching for the files
that matched the glob. We were prefetching files that
didn't match the glob.
facepalms
Reviewed By: strager
Differential Revision: D8846994
fbshipit-source-id: 593e85d843ffa1cc0707ed1dc86f1385262821f5
Summary:
This makes it easier to add some test coverage.
There's no real functional change in this diff; the only code change is to
throw a system_error instead of a thrift eden error wrapper class from the core
globbing code. There's a little bit of code to restore this exception type in
the callers in EdenServiceHandler; this is covered by existing integration
tests, but I've also expanded that coverage to cover both variants of the glob
thrift calls.
Reviewed By: strager
Differential Revision: D8776767
fbshipit-source-id: 3ea4ea642ae5108aa4b0153541bd3604f010b54c
Summary:
Some of the heavier watchman->eden queries in www today
trigger calls to `getFileInformation` with very large numbers
of paths.
For a set of paths with a lot of common prefixes there is a lot
of repeated work to load same inodes over and over again.
eg: `a/b/c/d/A` and `a/b/c/d/B` both have to load `a -> b -> c -> d` before
they can load the leaf node.
This diff pre-processes the list of paths and builds a tree-shaped plan
so that we won't need to load any inode referenced by any of the paths
more than once.
The `getSHA1` method could benefit from a similar change; I'll do that in
a follow-up diff.
Reviewed By: strager
Differential Revision: D8578261
fbshipit-source-id: e899640a2ef6dcdb6b903d2a2735e9240496d74b
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Context: `Future::get(...)` means both `Future::get()` and `Future::get(Duration)`
* Using lvalue-qualified `Future::get(...)` has caused some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* Problems with `Future::get(...) &`: it moves-out the result but doesn't invalidate the Future - the Future remains (technically) valid even though it actually is partially moved-out. Callers can subsequently access that moved-out result via things like `future.get(...)`, `future.result()`, `future.value()`, etc. - these access an already-moved-out result which is/can be surprising.
* Reasons `Future::get(...) &&` is better: its semantics are more obvious and user-testable. It moves-out the Future, leaving it with `future.valid() == false`.
Reviewed By: LeeHowes
Differential Revision: D8756764
fbshipit-source-id: 5c75c79cebcec77658a3a4605087716e969a376d
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Context: `Future::get(...)` means both `Future::get()` and `Future::get(Duration)`
* Using lvalue-qualified `Future::get(...)` has caused some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* Problems with `Future::get(...) &`: it moves-out the result but doesn't invalidate the Future - the Future remains (technically) valid even though it actually is partially moved-out. Callers can subsequently access that moved-out result via things like `future.get(...)`, `future.result()`, `future.value()`, etc. - these access an already-moved-out result which is/can be surprising.
* Reasons `Future::get(...) &&` is better: its semantics are more obvious and user-testable. It moves-out the Future, leaving it with `future.valid() == false`.
Reviewed By: yfeldblum
Differential Revision: D8711368
fbshipit-source-id: fbfcb731097cdf9d8d98583956bc7fe614157a6b
Summary:
Update the C++ edenfs code to ensure that the .eden and
.eden/storage/rocks-db directories exist, rather than requiring the python CLI
code create these directories as part of `eden start`
Reviewed By: strager
Differential Revision: D8508488
fbshipit-source-id: 358521b4f5eed1d19bf37903900ca50718e2c35c
Summary:
The OverlayGoldMasterTest.can_load_overlay_v2 unit test is failing stress test
runs because it keeps overlay data checked into the source repository and runs
the tests directly against this directory. This causes the tests to fail if
multiple test are run in parallel since they are all trying to use this same
directory simultaneously.
We need to make a copy of this directory rather than having the tests run
directly against the source tree to avoid locking issues. The overlay code
can potentially also try to upgrade the overlay format when it opens the
directory. This seems like another reason why it should not run directly
against the directory in the source repository.
This changes the test to make a copy of the overlay directory in a temporary
location, and run the tests against that directory. As part of this change I
also bundled the original input into a tar file.
Reviewed By: chadaustin
Differential Revision: D8555716
fbshipit-source-id: bf24bd96a0a31c097d9cf8e0fbe8b0bfaf009943
Summary:
There was a conflict between the DirContents refactoring and some of
the Overlay diffs.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D8427680
fbshipit-source-id: a01a2a8456f7a35249d8b65f2b5fcd55825438ee
Summary:
As code review follow-up from D8321501 and D8330070, have getFilePath
return InodePath rather than the index to the null terminator.
Also move getFileTmpPath and ensureTmpDirectoryCreated into their own
functions
Reviewed By: strager
Differential Revision: D8363811
fbshipit-source-id: 42718ed4af07fc7a6600564cc7d934f85fe333ed
Summary:
Creating new Overlay entries in tmp/ and then moving to their home is
even faster than creating them in the root of the overlay.
Reviewed By: strager
Differential Revision: D8330070
fbshipit-source-id: 620dde8fb6ccf9bfdc10872f5911d02fea28fdb0
Summary:
Writing to the overlay is excessively expensive on XFS and, to a
lesser extent, Btrfs. Placing temporary files in the root of the
overlay is dramatically faster than placing them next to their
destination.
Reviewed By: simpkins, strager
Differential Revision: D8321501
fbshipit-source-id: f6c9d5dff9d3eb1d393ff6c760518bc65c361aa2
Summary:
To get a read on whether there's anything egregious about the
overlay on various filesystem types and whether we can make any quick
improvements, I wrote up this quick benchmark.
Reviewed By: strager
Differential Revision: D8310783
fbshipit-source-id: 15fa1ea4bcbb7e4896e52b4acf0f8802133dbf3c
Summary:
Per a conversation with simpkins when code reviewing D7882648, this
diff removes the inheritance relationship between TreeInodeState and
DirContents. It doesn't change the binary layout of anything, but
defines DirContents as a typedef of PathMap<DirEntry>.
Reviewed By: strager
Differential Revision: D8232052
fbshipit-source-id: a2166f3ca2ab90fabbded0e48307b8a92a2b0250
Summary:
Several places in edenfs need to represent empty future objects, and were
written before the Future::makeEmpty() method was added. These locations
used Optional<Future> as a workaround.
This updates the code to simply use empty Futures instead of Optional<Future>
now.
Reviewed By: wez
Differential Revision: D8393712
fbshipit-source-id: eeb9e347d0973a4ab602500ee24fba77277d01ea
Summary:
The two FileInode::isSameAs() methods called FileInode::getSha1(), which
returns a folly::Future object, and immediately called `value()` on the
resulting future without waiting for it to complete first.
This bug dates back to D5430128. However, I suspect it was D7888344 that
actually exposed this issue recently. D7888344 updates the
`RocksDbLocalStore` code to perform work in an I/O thread pool. Before this
change the SHA1 futures were likely always immediately ready, so this bug was
not causing problems.
Reviewed By: wez
Differential Revision: D8393671
fbshipit-source-id: ec2116751ddda31a119bfe85eab5612b622f83cf
Summary:
Up until now all of the privhelper APIs have been blocking calls. This
changes the privhelper functions to return Futures, and updates all users of
these APIs to be able to handle the results using Futures.
One benefit of this change is that all existing mount points are remounted in
parallel now during startup, rather than being mounted serially. The old code
performed a blocking `get()` call on the future returned by
`EdenServer::mount()`.
The privhelper calls themselves are still blocking for now--they block until
complete and always return completed Future objects. I will update the
privhelper code in a subsequent diff to actually make it asynchronous.
Reviewed By: bolinfest
Differential Revision: D8053421
fbshipit-source-id: 342d38697f67518f6ca96a37c12dd9812ddb151d
Summary:
Per
35ba669307,
if the return value of DCHECK_NOT_NULL is expected to be unused,
DCHECK should be used instead.
Reviewed By: strager
Differential Revision: D8336319
fbshipit-source-id: 9ea758502baead8941b274dc0ed38ce59b1cc136
