Summary:
Build the hg store on macOS with Buck. This primarily involves putting
ServiceRouter and Rust datapack support behind build switches.
Reviewed By: fanzeyi
Differential Revision: D17106241
fbshipit-source-id: da4643a11d6e5d0a6df23bff1d5e1454ab351abe
Summary:
The Eden macOS build has regressed in a few ways since it's been
broken due to RocksDB changes. Fix those small issues.
Reviewed By: fanzeyi
Differential Revision: D17107209
fbshipit-source-id: ef2e97a84e2403e7b7f36a4a01e2ceecab69106e
Summary:
Update the fsck code to save any orphaned symlink inodes that it finds as
symlinks in the repair archive directory, rather than saving the contents as a
regular file.
Reviewed By: wez
Differential Revision: D17170346
fbshipit-source-id: 4cba8b27233b728114a80a327ab519b039297aea
Summary:
Use the new `OverlayChecker` class to automatically scan for errors and
attempt to repair them if the overlay was not shutdown cleanly the previous
time it was used.
Reviewed By: wez
Differential Revision: D16596601
fbshipit-source-id: 9923565b101ba953e92909e502be6ef5895c5cbd
Summary:
Update the OverlayChecker's `inodes_` map to store `InodeInfo` objects
directly, rather than pointers to `InodeInfo` objects allocated on the heap.
This tweaks some of the error handling code to look up `InodeInfo` objects in
the map in a few places, to avoid storing raw pointers to `InodeInfo` objects.
The `InodeInfo` objects generally shouldn't move once we start performing
error checking, but it seems better to be conservative here and avoid using
raw pointers that are only loosely coupled with the original object's
lifetime.
This probably shouldn't really make much of a performance difference in
practice, since this code is likely to be disk I/O bound anyway. (It does
appear to make a small difference in performance for ASAN-enabled builds when
the inode files are warmed up in the kernel cache.)
Reviewed By: wez
Differential Revision: D16750736
fbshipit-source-id: b089a259cc83ffd2fda6f83b617d95ce4ac467f9
Summary:
We are probably going to add more inodemap counters, so add a prefix
now.
Reviewed By: fanzeyi
Differential Revision: D17142015
fbshipit-source-id: 4bd3cd4fd9234d8766864f364fef0b0d963f03b6
Summary:
This was causing problems on macos where various tools
would enumerate and helpfully try to preserve attributes across
copies. On macos this would result in appledouble metadata files
being created to track the metadata in the destination file,
which clutters up the repo and has surprising secondary effects
such as being picked up by glob operations in cmake build rules.
This diff simply stops enumerating the extended attribute.
Reviewed By: fanzeyi
Differential Revision: D17140414
fbshipit-source-id: 2924657dc75b900baf70595edfa72e5d0521a697
Summary:
This fixes a few issues with the library dependencies:
- The `eden_utils` library depends on `eden_service_thrift`, not
`eden_service`. By incorrectly depending on `eden_service` this introduced
a circular dependency which would cause a build failure, depending on which
order CMake chose to try and emit the link line.
- The `eden_config` library depends on code from `eden_model` (for `Hash` and
`ParentCommits`)
- The `eden_inodes` library depends on `eden_model_git` for the `GitIgnore`
logic. I also alphabetized the dependency list.
Reviewed By: wez
Differential Revision: D17124930
fbshipit-source-id: 70cbe81081fc1dc807cca13a93edc25ba270b01f
Summary:
Rename the `ThriftCppLibrary.cmake` file to `FBThriftCppLibrary.cmake`, and
also rename `add_thrift_cpp2_library()` to `add_fbthrift_cpp_library()`.
Explicitly calling this `fbthrift` helps clearly distinguish that this is
intended for use with fbthrift (https://github.com/facebook/fbthrift/), as
opposed to Apache thrift.
Reviewed By: wez
Differential Revision: D16738440
fbshipit-source-id: 9b255e06b71c98ad74a34989f564a211958dcdd5
Summary:
Implement the repair() logic for InodeDataError, MissingMaterializedInode, and
OrphanInode.
The most common errors we see after an unclean system reboot is an inode file
not being present at all (`MissingMaterializedInode`), or it being present but
empty (`InodeDataError`). When either of these errors occurs for a directory
it also produces `OrphanInode` errors, as all of the children entries in the
directory are no longer part of the directory hierarchy.
This implements repair logic for these three error types. The repair behavior
is largely similar to the Python version in `eden/cli/fsck.py`, with some
minor changes to the output paths in the `lost+found/` repair directory. This
also includes more unit test to better exercise handling when `InodeDataError`
and `MissingMaterializedInode` errors occur inside an orphan subdirectory that
needs to be extracted to `lost+found`
Reviewed By: strager
Differential Revision: D16577696
fbshipit-source-id: 948158c5a0c32b31574d93011281c42051645ad9
Summary:
I added a None check to make redirect fixup work on macos
where we don't return running build information, but this broke the
integration tests (which don't run on mac!) where the build information
is actually the empty string.
Only check the version date if it is truthy, otherwise it will always
evaluate as less than the cutoff date and short-circuit the fixup
command.
Reviewed By: fanzeyi
Differential Revision: D17113336
fbshipit-source-id: 38dff9157a760b747958722e70567871dccd8cc0
Summary:
If the hg_import_helper (really `hg debugedenimporthelper`) command
claims to not support treemanifest, then immediately fail.
Reviewed By: strager
Differential Revision: D15302513
fbshipit-source-id: 044a9cba877d65b6dab0ede6b431146a361d2801
Summary:
The 2018-11 and 2019-03 snapshots were created with flatmanifest
repositories. Flatmanifest support is going away, so update them with
treemanifest repositories.
I untarred them and added an hgrc containing the following contents:
```
[extensions]
treemanifest =
[treemanifest]
treeonly = True
[remotefilelog]
reponame = snapshot
```
It's a bit janky to update a repo in place like this, but it means the
snapshot doesn't have to be recreated from scratch.
Reviewed By: simpkins
Differential Revision: D17103193
fbshipit-source-id: 62da8c2fe579d90281af83d4d8912cc64b09897b
Summary: Create config.toml in first Eden start up and store initial local store flag.
Reviewed By: chadaustin
Differential Revision: D16993074
fbshipit-source-id: dca8dbe60a892b7402ebf2f6c17df715dd2e8792
Summary:
The snapshot generation code and many integration tests create
repositories. By default, they were creating flatmanifest
repositories, which are on their way out. Instead, create tree-only
repos.
Reviewed By: strager
Differential Revision: D17066151
fbshipit-source-id: f99a9543440da6fd7cce0065c3cd7f91a59a02d5
Summary:
We saw some weird cases where we'd get error spew when
folks were using `eden redirect fixup` against really old servers,
despite us having fallback logic for those.
For now let's do a simple manual pre-check; we use july first as
the min server version for this; it was technically available earlier
than that, but this seems like a safe date to use in case of delays
in pushing out that code.
Reviewed By: strager
Differential Revision: D17094330
fbshipit-source-id: 239378906d382cb249a74e1534d0cd363988eaed
Summary:
This diff adds a fuse request timeout configuration setting
and threads it through to both the FuseChannel for our internal processing
and to the privhelper so that we can set an appropriate (slightly larger)
value for the kernel level daemon_timeout setting.
Reviewed By: chadaustin
Differential Revision: D16957552
fbshipit-source-id: a0fecc691d72914b5aebaed8a006dc0d6b0d7d12
Summary:
The macOS FUSE implementation has the concept of a daemon timeout,
which is used to set an upper bound on the length of time that the kernel will
wait for a request to be satisfied. If a request takes too long, the kernel
will shutdown the fuse device and the user is left with a relatively broken
experience in their repo until they restart the eden server. Critically, when
it is left in the broken state it is hard for the user to realize that they
need to restart the server because the kernel will still respond with EIO to
file accesses within the mount; even though it has been stopped, it isn't
fully unmounted until the fuse process stops.
This diff introduces a self imposed timeout in the fuse processing flow
and emits an ETIMEDOUT error when it is reached.
The intent is that we'll configure this timeout to be smaller than the
macOS daemon_timeout (which will be adjusted in a separate diff) so
that we don't trigger the problematic behavior when the kernel decides
that we've timedout.
This change is made for all fuse implementations, not just macOS, for
the sake of consistency: there's value to doing this on Linux as well,
to avoid some deadlock like scenarios: this should put an upper bound
on blocking in certain situations.
I've made the timeout default to 60 seconds, but haven't added any
configuration code for this yet; will do that in a follow on diff.
Reviewed By: chadaustin
Differential Revision: D16917954
fbshipit-source-id: 675539c43cf7f0009fd65d138081b9126464b7e0
Summary:
on linux kernel 5.1 we noticed that some caps were requested but not
known to the diagnostic code. This little diff adds a missing entry.
There is still one more that is unknown, but I think that we'll need to update
our copy of the fuse header to get that handled here.
Reviewed By: chadaustin
Differential Revision: D16953397
fbshipit-source-id: 30d19843a2cf1ec22fd469654773afaafa8e3b9b
Summary:
StartingGate was similar to std::barrier except that it guaranteed the
coordinating thread woke the worker threads. While this symmetry is
nice in concept, given enough threads, I doubt it pays for the
duplication.
Just adopt folly::test::Barrier (which is basically std::barrier)
instead.
Reviewed By: strager
Differential Revision: D16942136
fbshipit-source-id: 1691d8ef72b88c8867df74c8ed938f6c8d6ee094
Summary: Using a positive meaning rather than a double negative makes the build a tad simpler.
Reviewed By: wez
Differential Revision: D17000782
fbshipit-source-id: ef6c7b64708aa9b1f50c7ad4086c492a90c944f4
Summary: Using a positive meaning rather than a double negative makes the build a tad simpler.
Reviewed By: strager
Differential Revision: D17000620
fbshipit-source-id: ff27eb8098786b8ed6ed1ba81166b51e29e62d47
Summary:
The definition of RLockedPtr should use a non-const Subclass type, and not a
const one.
Reviewed By: ot
Differential Revision: D15356827
fbshipit-source-id: b8ad41e263f0e15ffa25b0698aa85eab8ca2ccb8
Summary:
The Java environment on Sandcastle macOS does not match corp laptops,
so allow disabling javadeprecated Thrift bindings on macOS for
now. Long-term, the better fix is to standardize the Java runtime and
JDK deployments across both environments so fbcode scripts can rely on
/usr/local/java-runtimes. See discussion in D16656570.
Reviewed By: strager
Differential Revision: D16676813
fbshipit-source-id: c21420bad554402c4c1e0881cabb962856719f5f
Summary: Allow building fuse_tester with open source gflags and glog.
Reviewed By: strager
Differential Revision: D16636221
fbshipit-source-id: 5128e936d5a6b6d0ed33bf34b770ee661728af3f
Summary:
Update the `eden prefetch` code to use the `require_checkout()` function,
which correctly finds the correct EdenFS instance that is serving this
checkout. Previously the prefetch logic found the default EdenFS instance,
which may be different from the one that was serving the specified checkout.
Reviewed By: wez
Differential Revision: D16878489
fbshipit-source-id: fed366df61960b5b87399bcd1d2f3f79aefc8dba
Summary: Now that StartingGate is in a more accessible dependency, use it in Eden's benchmarks.
Reviewed By: strager
Differential Revision: D16705853
fbshipit-source-id: 79ab9e9b32ab93f5b50a06a4889edb5e7b0d7e8b
Summary:
eden debug file_stat - add --sizes arg to tool
Currently using naive shell call and temprary file to store all of the paths to
get the sizes. This is significantly slower than running without sizes and will
be replaced with proper sizes provided by eden thrif call in the future.
This is sufficient to establish the structure of the output of this command and
start integrating it with sandcastle
Reviewed By: strager
Differential Revision: D16836053
fbshipit-source-id: ad47416a0036d1046768fa9b9b6c9dfefbb8edd4
Summary:
Formatting the output of the eden debug file_stats as json
This will make it easier to use with further tooling to analyze and build
actionable statistics.
Reviewed By: strager
Differential Revision: D16832739
fbshipit-source-id: fb90ff59f121eeac3bf4865956ed5314ce243844
Summary:
our argparse stuff was configured to require 1+ patterns
which meant that an invocation like:
```
echo 'fbcode/eden/fs/TARGETS' | eden prefetch --pattern-file /dev/stdin
```
would emit:
```
eden prefetch: error: the following arguments are required: PATTERN
```
the workaround was to specify a dummy pattern argument.
This diff fixes up the argparse config to make the pattern argument be 0+ patterns.
Reviewed By: pkaush
Differential Revision: D16831732
fbshipit-source-id: 93cb4c4677b1af8b9d4824bcad60708c40867375
Summary:
This was an unanticipated casualty of some unicode vs bytes
work in the past, and we hadn't gotten around to fixing this... until now!
Reviewed By: pkaush
Differential Revision: D16831603
fbshipit-source-id: dd38751e6d0b1c242937a7dbb150f3aa15d1c8ae
Summary: The type checker doesn't understand `windll`. Sanest path is to leave the kernel of this function untyped.
Reviewed By: strager
Differential Revision: D16820160
fbshipit-source-id: 47c1bb5f29b790b48dd28d3b1b1e21b50722eb12
Summary: On POSIX system we use InodeTree structure to implement prefetch. These structures are missing on the Eden Windows. We will instead use mercurial's prefetch to prefetch the contents for us.
Reviewed By: strager
Differential Revision: D16742336
fbshipit-source-id: db0a389503cb98469707c07b6da412056f4dd72e
Summary: On Windows, we create a .eden on the root of the repo as compared to in each directory. Updating get_eden_mount_name() to look for .eden in the current parent path.
Reviewed By: strager
Differential Revision: D16742335
fbshipit-source-id: 2d27c54895917524d3d9c233ccc723b51f33f032
Summary: On Linux and Mac, we provision .eden inside every directory, which contains symlinks to other locations. On Windows this won't work as well. First, Projected FS will cache all the .eden and will bump up the number of files on the disk. Second, it would be hard to simulate symlinks from Projected FS.
Reviewed By: strager
Differential Revision: D16724354
fbshipit-source-id: a560698fae8c7f14e475f1c1fb9ecc53e60c189f
Summary: Until now we were running Edenfs on the console. Now we could use edenfsctl start.
Reviewed By: chadaustin
Differential Revision: D16565624
fbshipit-source-id: 9a12e52234d36d7b8ce4e3741ffb62655ae334b2
Summary:
Basic tool to collect read/written files on eden
This is v0. The next steps are:
* Add collection of sizes for all paths
* Add as an optional step in duplo/jobs
* Add some smarter analysis options as we discover need
Reviewed By: strager
Differential Revision: D16663820
fbshipit-source-id: 1c25289c9840fc23318e0653bc632b097256a5bd
Summary: Peer-to-peer DBus sockets do not have an org.freedesktop.DBus object on the bus, nor anything (or any reason) to accept the Hello message. Enable users of DBusAddress to explicitly use this behavior if desired.
Reviewed By: zeroxoneb, simpkins, aleivag
Differential Revision: D16780064
fbshipit-source-id: e10e51aef7754f5b022b8d7455b51baee3db8db8
Summary:
Implement OverlayChecker::computePath().
This also updates it to return a structure which can better represent
information if the inode is part of an orphaned subtree, which prevents us
from figuring out the full path to the root of the checkout.
Reviewed By: chadaustin
Differential Revision: D16577697
fbshipit-source-id: b543e2210fe8bc561d78194852962bf57ad9644a
Summary:
This ports much of the Python logic in `eden/cli/fsck.py` to C++, so that
the edenfs daemon can invoke it directly during mount initialization when
necessary.
This initial diff just contains the code to scan for errors, without any
repair logic.
This also includes some improvements to the logic for efficiency. For
instance, we compute less state during the initial scan phase. This makes the
scan phase faster, at the expense of making some of the repair steps slightly
more expensive if errors are found. This newer C++ scan logic is
significantly faster than the old Python version: scanning 260,000 inode files
took around 1m20 second for the Python code, and runs in under 3 seconds with
the new C++ code. (On an SSD with the files in question already warm in the
kernel cache.)
Reviewed By: strager
Differential Revision: D16577699
fbshipit-source-id: 2f8713a219ad882848cee0c57dca34f2c7808340
Summary: Redirecting the stdout and stderr to the eden log file. The log categories are not working as expected on Windows and will need to be fixed. At this point this doesn't log the "eden=DBG2" on Windows. One workaround for now is to remove "FOLLY_INIT_LOGGING_CONFIG()" to get the default logs.
Reviewed By: chadaustin
Differential Revision: D16653632
fbshipit-source-id: 028cdf1d698798ff3fb0284b3f0fa2df5570eaf0
Summary: Moving these functions to EdenInit so we could use them on Windows. This diff doesn't change the behavior.
Reviewed By: chadaustin
Differential Revision: D16653498
fbshipit-source-id: 5cc8c2f4b12f54035fb86ae20f3274245642d032
Summary:
Moves the CMD column to the right, to prevent long commands from being truncated.
https://pxl.cl/GG37
Reviewed By: chadaustin
Differential Revision: D16626771
fbshipit-source-id: 291e862d2fcff6c252ae6263536a799428577137
Summary:
Drop the `2` from the `fb303_thrift_cpp2` rule name.
The old `cpp` version of fbthrift is gone, `cpp2` is the only supported C++
version, so lets drop the `2` from this build rule name.
I plan to change some of the CMake thrift rule generation code soon, to
support multiple languages (namely Python). As part of this I plan to
automatically add a language-specific suffix(`_py` or `_cpp`) to the build
rules. It seems better to use `_cpp` rather than `_cpp2` for C++ libraries.
Changing the name for this fb303 rule will make it fit this planned naming
scheme.
Reviewed By: chadaustin
Differential Revision: D16653264
fbshipit-source-id: e4da70c79ff14e9981717ab909d982770d104512
