Summary: Use callback to save ScmStatus instead of storing status inside a `TreeDiffer` object.
Reviewed By: simpkins
Differential Revision: D17379318
fbshipit-source-id: b5f56c937509f0861f5dc06c742ebc3e1f85c695
Summary:
Use the correct inode pointer when creating an `InodeError`. Previously this
passed in an inode pointer that was always null.
Reviewed By: wez
Differential Revision: D17438864
fbshipit-source-id: 37b7e40723afb9335b7616b585eed3b6cf5b3c81
Summary:
Fix `GlobNode::evaluate()` so that it only generates a result once all
processing is complete. The glob evaluation walks down the inode tree in
parallel. Previously if an error occurred while processing one TreeInode it
would return that exception to the caller immediately, even if work was still
being done to evaluate the glob on other parts of the inode tree. This
behavior is problematic, since the caller may destroy the GlobNode immediately
once a result is generated, and the caller has no way to know that other
children GlobNodes may still be using some of the data.
This fixes the code to wait until all evaluation sub-futures have completed
processing even on error, so that we know it is safe to destroy the GlobNode
before we generate a result.
Reviewed By: wez
Differential Revision: D17434327
fbshipit-source-id: ef70dc82281eeb3540840202ec5e704afe2d29eb
Summary:
Make sure the contents of the special `.eden/` subdirectory are correct each
time we mount a checkout. Before we would generally only set up the contents
of this directory if it didn't previously exist.
Now the code verifies that the contents of each symlink in this directory are
correct, and recreates the symlink if needed.
This allows EdenFS to automatically repair the contents of this directory even
if the checkout or its `clients` directory has been manually moved.
Reviewed By: wez
Differential Revision: D17279413
fbshipit-source-id: e24e7530f44fff94ebb6f67174aaf78c9b498d6b
Summary:
Update the CMakeLists.txt to also build the Python-based `edenfsctl` command
line tool.
This requires switching most of the thrift rules to generate both C++ and
Python sources.
Note that one missing feature at this point is that this does not package
external dependencies into the binary. Currently `edenfsctl` depends on both
`six` and `toml` as external dependencies. For now these must be available in
your `PYTHONPATH` in order to run the generated `edenfsctl` binary.
Reviewed By: chadaustin
Differential Revision: D17127615
fbshipit-source-id: fc138ab39e75c6a5bbd39e3f527d4e9f7f420e46
Summary:
This silences a compiler warning about reaching the end of a non-void function
without returning a value.
Reviewed By: chadaustin
Differential Revision: D17308290
fbshipit-source-id: 95cdb3353364a36dcd2295b19bf745a941e5e3cf
Summary:
This diff passes the free space information through from
the overlay lock file descriptor back up to the kernel when the
filesystem stats are requested.
This makes the user experience with eg: Finder on macOS nicer.
Reviewed By: chadaustin
Differential Revision: D17255859
fbshipit-source-id: ed9f3b9fd386c5706539879513854a9cd1550d8a
Summary:
Update the fsck code to save any orphaned symlink inodes that it finds as
symlinks in the repair archive directory, rather than saving the contents as a
regular file.
Reviewed By: wez
Differential Revision: D17170346
fbshipit-source-id: 4cba8b27233b728114a80a327ab519b039297aea
Summary:
Use the new `OverlayChecker` class to automatically scan for errors and
attempt to repair them if the overlay was not shutdown cleanly the previous
time it was used.
Reviewed By: wez
Differential Revision: D16596601
fbshipit-source-id: 9923565b101ba953e92909e502be6ef5895c5cbd
Summary:
Update the OverlayChecker's `inodes_` map to store `InodeInfo` objects
directly, rather than pointers to `InodeInfo` objects allocated on the heap.
This tweaks some of the error handling code to look up `InodeInfo` objects in
the map in a few places, to avoid storing raw pointers to `InodeInfo` objects.
The `InodeInfo` objects generally shouldn't move once we start performing
error checking, but it seems better to be conservative here and avoid using
raw pointers that are only loosely coupled with the original object's
lifetime.
This probably shouldn't really make much of a performance difference in
practice, since this code is likely to be disk I/O bound anyway. (It does
appear to make a small difference in performance for ASAN-enabled builds when
the inode files are warmed up in the kernel cache.)
Reviewed By: wez
Differential Revision: D16750736
fbshipit-source-id: b089a259cc83ffd2fda6f83b617d95ce4ac467f9
Summary:
We are probably going to add more inodemap counters, so add a prefix
now.
Reviewed By: fanzeyi
Differential Revision: D17142015
fbshipit-source-id: 4bd3cd4fd9234d8766864f364fef0b0d963f03b6
Summary:
This was causing problems on macos where various tools
would enumerate and helpfully try to preserve attributes across
copies. On macos this would result in appledouble metadata files
being created to track the metadata in the destination file,
which clutters up the repo and has surprising secondary effects
such as being picked up by glob operations in cmake build rules.
This diff simply stops enumerating the extended attribute.
Reviewed By: fanzeyi
Differential Revision: D17140414
fbshipit-source-id: 2924657dc75b900baf70595edfa72e5d0521a697
Summary:
This fixes a few issues with the library dependencies:
- The `eden_utils` library depends on `eden_service_thrift`, not
`eden_service`. By incorrectly depending on `eden_service` this introduced
a circular dependency which would cause a build failure, depending on which
order CMake chose to try and emit the link line.
- The `eden_config` library depends on code from `eden_model` (for `Hash` and
`ParentCommits`)
- The `eden_inodes` library depends on `eden_model_git` for the `GitIgnore`
logic. I also alphabetized the dependency list.
Reviewed By: wez
Differential Revision: D17124930
fbshipit-source-id: 70cbe81081fc1dc807cca13a93edc25ba270b01f
Summary:
Rename the `ThriftCppLibrary.cmake` file to `FBThriftCppLibrary.cmake`, and
also rename `add_thrift_cpp2_library()` to `add_fbthrift_cpp_library()`.
Explicitly calling this `fbthrift` helps clearly distinguish that this is
intended for use with fbthrift (https://github.com/facebook/fbthrift/), as
opposed to Apache thrift.
Reviewed By: wez
Differential Revision: D16738440
fbshipit-source-id: 9b255e06b71c98ad74a34989f564a211958dcdd5
Summary:
Implement the repair() logic for InodeDataError, MissingMaterializedInode, and
OrphanInode.
The most common errors we see after an unclean system reboot is an inode file
not being present at all (`MissingMaterializedInode`), or it being present but
empty (`InodeDataError`). When either of these errors occurs for a directory
it also produces `OrphanInode` errors, as all of the children entries in the
directory are no longer part of the directory hierarchy.
This implements repair logic for these three error types. The repair behavior
is largely similar to the Python version in `eden/cli/fsck.py`, with some
minor changes to the output paths in the `lost+found/` repair directory. This
also includes more unit test to better exercise handling when `InodeDataError`
and `MissingMaterializedInode` errors occur inside an orphan subdirectory that
needs to be extracted to `lost+found`
Reviewed By: strager
Differential Revision: D16577696
fbshipit-source-id: 948158c5a0c32b31574d93011281c42051645ad9
Summary:
This diff adds a fuse request timeout configuration setting
and threads it through to both the FuseChannel for our internal processing
and to the privhelper so that we can set an appropriate (slightly larger)
value for the kernel level daemon_timeout setting.
Reviewed By: chadaustin
Differential Revision: D16957552
fbshipit-source-id: a0fecc691d72914b5aebaed8a006dc0d6b0d7d12
Summary:
The definition of RLockedPtr should use a non-const Subclass type, and not a
const one.
Reviewed By: ot
Differential Revision: D15356827
fbshipit-source-id: b8ad41e263f0e15ffa25b0698aa85eab8ca2ccb8
Summary:
The Java environment on Sandcastle macOS does not match corp laptops,
so allow disabling javadeprecated Thrift bindings on macOS for
now. Long-term, the better fix is to standardize the Java runtime and
JDK deployments across both environments so fbcode scripts can rely on
/usr/local/java-runtimes. See discussion in D16656570.
Reviewed By: strager
Differential Revision: D16676813
fbshipit-source-id: c21420bad554402c4c1e0881cabb962856719f5f
Summary:
Implement OverlayChecker::computePath().
This also updates it to return a structure which can better represent
information if the inode is part of an orphaned subtree, which prevents us
from figuring out the full path to the root of the checkout.
Reviewed By: chadaustin
Differential Revision: D16577697
fbshipit-source-id: b543e2210fe8bc561d78194852962bf57ad9644a
Summary:
This ports much of the Python logic in `eden/cli/fsck.py` to C++, so that
the edenfs daemon can invoke it directly during mount initialization when
necessary.
This initial diff just contains the code to scan for errors, without any
repair logic.
This also includes some improvements to the logic for efficiency. For
instance, we compute less state during the initial scan phase. This makes the
scan phase faster, at the expense of making some of the repair steps slightly
more expensive if errors are found. This newer C++ scan logic is
significantly faster than the old Python version: scanning 260,000 inode files
took around 1m20 second for the Python code, and runs in under 3 seconds with
the new C++ code. (On an SSD with the files in question already warm in the
kernel cache.)
Reviewed By: strager
Differential Revision: D16577699
fbshipit-source-id: 2f8713a219ad882848cee0c57dca34f2c7808340
Summary:
Now that we have unsafe access to the contents of a
folly::Synchronized, InodeMap::updateOverlayForUnload can just access
the storage because it knows nobody else has access.
Reviewed By: simpkins
Differential Revision: D16632778
fbshipit-source-id: 4b2c1d6679e2655671d43899407bac1a19f2cab9
Summary:
Update `EdenServer::mount()` to correctly handle errors that occur during the
mount `INITIALIZING` phase. Previously the code did not add error callbacks
to the `Future` result to handle errors during initialization. As a result we
would propagate the exception back to the thrift caller, but the `EdenMount`
object would remain in our mount point list, stuck forever in the
`INITIALIZING` state.
Reviewed By: strager
Differential Revision: D16590032
fbshipit-source-id: 9adbdf05441dad815096b195ece36f3d958c96a9
Summary:
Update `FsOverlay::validateHeader()` to throw `EdenError` exceptions rather
than `std::system_error`. These exceptions are generated when we find invalid
data, rather than from system calls that return errno values. Previously the
code was using `folly::throwSystemErrorExplicit()` and
`folly:throwSystemError()`, and passing in a made-up `EIO` value.
However, the latter two calls were incorrectly using
`folly::throwSystemError()` rather than `throwSystemErrorExplicit()`. As a
result the `EIO` parameter was being treated as part of the error message, and
the code was appending the error description for whatever value `errno`
happened to be set to at the moment.
This fixes the code to just throw `EdenError` exception types, since these
errors aren't caused by a real `errno` value.
Reviewed By: strager
Differential Revision: D16577698
fbshipit-source-id: 9487c34f04da99d397611f005f00f02114b12094
Summary:
Fix `FsOverlay::initOverlay()` to also create the `tmp` subdirectory, which is
required for the `FsOverlay` code to function properly. Previously this
directory was created by the `Overlay` class. However, `FsOverlay` should be
responsible for creating it so it can be used from other locations besides
just `Overlay`.
I also removed the code that checks if the `tmp` directory needs to be created
on restart. The code to create `tmp` has been present for over a year
(D8330070), and we no longer have anyone using EdenFS versions older than
this.
Reviewed By: strager
Differential Revision: D16577700
fbshipit-source-id: 3eaa28003159a03084c4f367c7b96d69fc1702d0
Summary:
There is no point to call `readBE<uint64_t>()` here. The only way these calls
can fail is if the buffer is too short for the data, but we already checked
the buffer length at the top of the function.
Reviewed By: strager
Differential Revision: D16577701
fbshipit-source-id: 8b57cec05453b9b7224470671e032e3aefe41033
Summary: Keep track of the longest query we had to resolve in accumulateRange. This stat will be helpful in determining how large the memory limit of the journal should be since we will know how far back people usually need to go.
Reviewed By: strager
Differential Revision: D16227920
fbshipit-source-id: a41c3b9f16b701cd8165e20409888983b8899dab
Summary: Set up the infrastructure to add in Timeseries to the journal so that we can add in stats for the journal that relate to the whole process. For example, allow us to add in a truncatedRead TimeSeries easily as done in D16017968
Reviewed By: chadaustin
Differential Revision: D16461081
fbshipit-source-id: 964ff32e62aed0369da434793491b857c136b074
Summary: Create a counter for the end-to-end duration of the journal.
Reviewed By: chadaustin
Differential Revision: D15993675
fbshipit-source-id: 260a58a7b37dfa16b82b3df187ad7f7263ff67cf
Summary: Change getLatest such that it just returns Info about the latest delta and not the delta itself.
Reviewed By: strager
Differential Revision: D15931214
fbshipit-source-id: c7a1cf4d62cdd4f9396fab46354eabcbb31f32c0
Summary:
Implements size-only local storage, as opposed to storing metadata. This is useful when the blob's SHA-1 is not needed. This diff prevents SHA-1 computations, which can be especially expensive for large blobs.
From D15934535, operations such as `ls -l` and `stat` will get the size of a blob in two ways:
1) The blob's size is already stored locally, so it will be deserialized and returned.
2) The blob is fetched from the backing store, stored, and its size is returned.
This diff optimizes the second case, because SHA-1 is no longer computed.
Reviewed By: strager
Differential Revision: D15723239
fbshipit-source-id: a868f3bf6b68a83ddafb057dc3e4e65f0a2dd989
Summary:
These allow the cli to setup and tear down mounts and
have the eden server keep track of them.
Reviewed By: strager
Differential Revision: D15707318
fbshipit-source-id: abdb8eaa28c8c67c8211a8af1647efe3a083e998
Summary: We need --edenDir support to run muliple instance of Edenfs, which is required to run the integration tests.
Reviewed By: simpkins
Differential Revision: D15951483
fbshipit-source-id: a516159cdeb5f046f795fc28399a2af5fe8a9f95
Summary: EdenMount.cpp includes folly/Subprocess.h, but uses none of the included symbols. Drop the unnecessary #include.
Reviewed By: simpkins
Differential Revision: D15773136
fbshipit-source-id: 4f30af9a1afcc3258aaa00d519d2aceeda72824f
Summary:
Update the copyright & license headers in thrift files to reflect the
relicensing to GPLv2+
Reviewed By: wez
Differential Revision: D15487082
fbshipit-source-id: 33f68617037f36c07075fb962a16a4d8f55bd6a6
Summary:
Update the copyright & license headers in CMake files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487079
fbshipit-source-id: 715e559464c19a0070d6e55a095b3fc7d61ad2f8
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary: Merge was a function on a JournalDelta that created a new JournalDelta and optionally left it connected to old JournalDeltas. AccumulateRange is a new function on the Journal itself (acting on the latest delta) that creates a JournalDeltaSum (which can't be connected to older Deltas)
Reviewed By: chadaustin
Differential Revision: D15881452
fbshipit-source-id: 573505c1171f78d46afc98f1db9b5b9ee2fff60f
Summary: Making addDelta private and giving users a more user-friendly way of appending entries to the journal.
Reviewed By: chadaustin, strager
Differential Revision: D15868089
fbshipit-source-id: 00c8a3066f0e4483e3c792651ade5f6a7ea05eed
Summary:
Some newer versions of `clang` (such as Apple's version 11) will warn/error out if a constructor or assignment operator
marked `default` is implicitly deleted (e.g., if the object contains a non-moveable/non-copyable member). This diff
removes all such defaulted constructors/assignment operators, which I ran into while building `edenfs` on my Macbook Pro.
Reviewed By: chadaustin, strager
Differential Revision: D15901794
fbshipit-source-id: 794ed8377693a6735bb567635dc919bc678751a4
Summary: `getSize` and `getSha1` were misleading function names, since the functions refer to the size and hash of a given blob and not the object store itself. These functions have been renamed to `getBlobSize` and `getBlobSha1`.
Reviewed By: chadaustin
Differential Revision: D15696510
fbshipit-source-id: 4dd31659f60969fa90d8e2b39f43c46a2b7dff7c
Summary:
Change the `ReloadableConfig::getEdenConfig()` method to accept an enum to
control whether it should reload the configuration, rather than the simpler
bool that it used previously.
The primary motivation for this is to allow us to force a reload to occur in
unit tests and integration tests, even if this operation would otherwise be
within the normal reload throttling time limit.
Reviewed By: strager
Differential Revision: D15572122
fbshipit-source-id: 9c638a415a451f26e7f5923f544724594ed4e0f1
Summary: Setting up structure to get the stats from Eden's Journal to ODS
Reviewed By: chadaustin
Differential Revision: D15684380
fbshipit-source-id: 90debc3f2a09d497201e5e73f85400a994b7afba
Summary: These functions are part of the blocking API, which is being replaced by the Future-based API. They are not being called, so they are unneeded.
Reviewed By: chadaustin, strager
Differential Revision: D15582641
fbshipit-source-id: 11f693040124846515aef7c7ee6ad750e1b5c164
Summary: Developers should be encouraged to use Future-based functions instead of blocking functions in production code. Removing this blocking function makes it less convenient for developers to write blocking code in the future.
Reviewed By: strager
Differential Revision: D15564952
fbshipit-source-id: 3f62db472a59a6487ffe12e48c04178cad84ca61
Summary: Small parts of our code have diverged from our clang-format rules.
Reviewed By: strager
Differential Revision: D15380260
fbshipit-source-id: f668ac22d6c0c5f2468549f2a94dd1c9bb22ce3d
Summary:
Now Eden on Windows could be build using
python.exe .\opensource\fbcode_builder\getdeps.py build eden
Reviewed By: strager
Differential Revision: D15251103
fbshipit-source-id: 8ac1097b234bb27ce478101bfbc9591251e2e359
