Summary:
The `LocalStore` constructor was updated to accept a `ReloadableConfig`
argument in D12949577, but this was never used anywhere. Remove it for now to
help simplify the code. If we do want to add it back in the future I think we
should make it required, rather than allowing a null config to be specified.
Reviewed By: wez
Differential Revision: D15350217
fbshipit-source-id: 8571b48dff8c8d079ba6b25821dd0b1d77ffe791
Summary:
Implement the `show_sizes` function which reports the approximate size on disk
used by each column family.
The initial skeleton for this command was accidentally included in D15162813.
Reviewed By: wez
Differential Revision: D15307391
fbshipit-source-id: 03efee939d65500e17b48047552dec1dbb7aff7e
Summary:
Add a `name` field to KeySpaceRecord. This allows the `RocksDbLocalStore` and
`SqliteLocalStore` code to use the name from the `kKeySpaceRecords` data
structure rather than keeping their own array of names that must be maintained
in sync with the `LocalStore` code.
Reviewed By: wez
Differential Revision: D15307390
fbshipit-source-id: b53bca7d53d630f9527a810f145380288c54198e
Summary:
Update the KeySpace enum values to start at 0 instead of 1. This simplifies
the code to avoid having to skip over 0 in a few places.
This also makes the `kKeySpaceRecords` array slightly less confusing. Unlike
the `columns` array used by `RocksDbLocalStore` and the `tableNames` array
used by `SqliteLocalStore`, the `kKeySpaceRecords` array was not previously
indexed by the `KeySpace` enum values.
Reviewed By: wez, strager
Differential Revision: D15307393
fbshipit-source-id: ae8392d02396b4dc3c18e9ee94b198fcbb9b1a34
Summary:
Move the KeySpace record definitions out of LocalStore.cpp and into a header
file. This will make it possible to access the key space information from
other files in the future.
I would like to have access to this data directly in the various LocalStore
implementations, as well as in eden_store_util.cpp. Currently both
RocksDbLocalStore.cpp and SqliteLocalStore.cpp have their own copies of key
space information that should really be shared.
Reviewed By: chadaustin, strager
Differential Revision: D15307392
fbshipit-source-id: ad7cdc705d3dc9834592fe0f14b38d82f91a3b77
Summary:
Now Eden on Windows could be build using
python.exe .\opensource\fbcode_builder\getdeps.py build eden
Reviewed By: strager
Differential Revision: D15251103
fbshipit-source-id: 8ac1097b234bb27ce478101bfbc9591251e2e359
Summary: MSVC doesn't like the ImporterOptions declared as class and defined as struct.
Reviewed By: wez
Differential Revision: D15251102
fbshipit-source-id: e7725df5db8ee39a20a1940a8d045369800aa4c3
Summary: tar is in /usr/bin on macOS.
Reviewed By: simpkins
Differential Revision: D15052024
fbshipit-source-id: 5388b2d53f876717310d0f9726b74e8a283d1a89
Summary:
This solves the client certificate mystery. We creates a new `EdenConfig` object
whenever there is a change detected and these two fields are lost during the
copying. Causing the variable substitution not working as expected.
Reviewed By: wez
Differential Revision: D15221180
fbshipit-source-id: 3fba36c8f261cb3dbe25e2e363d65661240be8cf
Summary: Change `ConfigSource` from a plain `enum` to an `enum class`.
Reviewed By: chadaustin
Differential Revision: D15162811
fbshipit-source-id: 680ec1e785de1a54b623d99477914afa5d0931ca
Summary:
Move the `ConfigSetting` template and related classes from `EdenConfig.h` to a
new `ConfigSetting.h` header file.
Reviewed By: chadaustin
Differential Revision: D15162815
fbshipit-source-id: 34fba0b8b0e8e50714f622bae642d20d77050673
Summary:
Update the `isValidAbsolutePath()` implementation to use `kDirSeparator`, and
update the TODO comment.
Reviewed By: chadaustin
Differential Revision: D15162812
fbshipit-source-id: f52f0e85c5eef342723947c6efaa52992da1660b
Summary:
Move the `FieldConverter` logic out of EdenConfig.h/cpp and into its own
module.
Reviewed By: chadaustin
Differential Revision: D15162816
fbshipit-source-id: f30d81329139e5827b81a609e83baddf71b6af5e
Summary:
Rename the `ClientConfig` class to `CheckoutConfig`, to help further progress
on updating our terminology from "client" to "checkout".
Reviewed By: chadaustin
Differential Revision: D15162814
fbshipit-source-id: 3575958f1161d5842c6f0ee9e2d2d20ab20b7372
Summary:
Move the `ReloadableConfig` implementation out of `ServerState` and into the
`ReloadableConfig` class itself. This also updates `ServerState` to simply
contain a `ReloadableConfig` member variable rather than inheriting from
`ReloadableConfig`.
This makes it easier to build other utilities that need a `ReloadableConfig`
object but do not have a full `ServerState` object, such as the
`eden_store_util` class and the `zero_blob` helper tool used by the
integration tests.
Reviewed By: chadaustin
Differential Revision: D15162813
fbshipit-source-id: 1ebc0ac6c475f4aa1090dd933ebce16bc222674c
Summary:
This timeout appears to be too short for the last step of the test that sends
32MB of data in multiple send calls of no more than 1000 bytes each. On
stress test runs this timeout has occasionally been firing and causing the
test to fail.
Reviewed By: chadaustin
Differential Revision: D15221306
fbshipit-source-id: d1f7ed2d58b47512a8f31f8324c61372394a6709
Summary:
beholdunittests
This enables some plumbing for running some of the
tests using the gtest/gmock machinery in cmake.
Part of this diff is removing the FindGMock.cmake file from the
eden repo; we now pull this in from the shared cmake library
that is populated by shipit.
Reviewed By: simpkins
Differential Revision: D14993344
fbshipit-source-id: 51caf9518c7f3a083a3b90cda10324c3a8170359
Summary:
This argument is unnecessary since D14222323, which should be deployed
everywhere by now. Passing in the extra repo argument forces hg to open the
repository object twice, which is unnecessary.
We already set the current working directory to the repository when invoking
the import helper, so hg will find the repository correctly from that.
Reviewed By: pkaush
Differential Revision: D15179667
fbshipit-source-id: 838cbee91748c41c713731187608a9823b971a53
Summary:
Update HgImporter to use `hg debugedenimporthelper` instead of the separate
`hg_import_helper.py` script by default on all platforms now. The
`debugedenimporthelper` subcommand has existed in our deployed versions of hg
for a while now.
Reviewed By: pkaush
Differential Revision: D15179668
fbshipit-source-id: 2fb8c4c9f92aed54c84899d6643f746baac73327
Summary:
This diff tries to improve the Thrift Mononoke backing store implementation by:
* Re-creating thrift client for every request since thrift has builtin connection pooling
* Using Eden's CPU thread pool for processing data fetched from remote
* Calling thrift methods within correct executor
Reviewed By: wez
Differential Revision: D15170818
fbshipit-source-id: c8be70755a851f63fb62e4d4f36833703283565e
Summary: We intend to break support for flatmanifest hg, so require that treemanifest is available in the build.
Reviewed By: wez
Differential Revision: D15057150
fbshipit-source-id: 449399cfb9d018f3b722598091eead1bd5d7c77d
Summary: Flatmanifest is on its way out. Remove support for falling back to it if a tree import fails.
Reviewed By: pkaush
Differential Revision: D15056459
fbshipit-source-id: a4df820322ee354d77f50a0ec92e9705d0f152ec
Summary:
Interestingly, the request doesn't fail, but does leave
our stack garbage intact. Let's be sure to zero it out to make
it more obvious what is happening.
While we're in here, let's make sure that we get the same
results from both ends of the socket pair.
Reviewed By: simpkins
Differential Revision: D14994593
fbshipit-source-id: 9aec957dfcd80d88c3d8fbce6bf45480502ea812
Summary: this fixes linking the tests when building with cmake in D14993344
Reviewed By: chadaustin
Differential Revision: D15167223
fbshipit-source-id: 1dda3beed3b6ff3788f3992783c58b4b92f697f2
Summary:
D15043209 missed fixing up the name of the variable
containing the list of sources.
The result is that we compile the `main` from `eden_store_util`
in to the tests and break gtest discovery in D14993344
Reviewed By: chadaustin
Differential Revision: D15167219
fbshipit-source-id: 95e9dc8ba74965f41bfebed1c1776605d8c9443a
Summary:
UnixSocketTest would block forever on macos if the message
size was larger than approx 1kb. It turns out that MSG_DONTWAIT
isn't documented in the `sendmsg` man pages and apparently has
no effect at all. Instead, the socket must be placed in non-blocking
mode. This appears to be the case for the client side of the sockets
but I need to follow up for the server side.
Reviewed By: simpkins, strager
Differential Revision: D14994588
fbshipit-source-id: 2a0a1c26a7b45ece82f8f79c15fb7756844bc86f
Summary:
on macOS `/var/tmp` is a symlink to `/private/var/tmp`
which means that our tests for path canonicalization end up
trying to compare the non-canonical `/var/tmp` prefix of the
temporary paths generated by the tests with `/private/var/tmp`
and failing.
Reviewed By: simpkins
Differential Revision: D14994584
fbshipit-source-id: e69addca8e315855d3ca87feb576ca7c54567c31
Summary:
D14677339 added tracing to all HgImporter requests. Each trace loads a thread-local variable through EdenStats::getStatsForCurrentThread(). Each of these loads has a small but non-trivial cost.
Because each instance of HgImporter is used only on one thread, each getStatsForCurrentThread() call returns the same EdenThreadStats object. Avoid the thread-local-variable lookups by caching the result of getStatsForCurrentThread() in HgImporter.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14794284
fbshipit-source-id: d1609a1720d44c680dc0ebaa2536779def2a6f37
Summary:
If EdenFS' globFiles API is given two patterns, and one pattern is a prefix of the other, EdenFS effectively ignores the longer pattern. Given the patterns `project/src/*` and `project/src/*/*`, when EdenFS encounters `project/src/dir/`, it generates a result (because `project/src/dir/` matches the first pattern) but does not recurse into its children.
The problem caused by an incorrect understanding of the GlobNode::isLeaf_ flag. isLeaf_ means "this GlobNode should generate results", but GlobNode::evaluateImpl understands it to mean "this GlobNode should generate results, and this GlobNode has no children". Given the patterns `project/src/*` and `project/src/*/*`, the GlobNode representing `project/src/*` has isLeaf_=true but also has children.
Fix the bug by not using isLeaf_ for determining whether to recurse, and instead relying on the presence of child GlobNode-s.
Reviewed By: chadaustin
Differential Revision: D15078089
fbshipit-source-id: 1c480d11361f89193b35965266e6873c57181113
Summary:
If globbing returns a directory, that directory doesn't need to be loaded. GlobNode::evaluateImpl is careful to only load directories/trees if necessary. When fixing a bug in GlobNode::evaluateImpl, I accidentally broke this optimization. No test failed, though.
Ensure this optimization works and doesn't regress by writing a simple test.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D15090386
fbshipit-source-id: fe932e4a7ad517ff4c4523d58668f05fe8b0cafb
Summary:
In GlobNode::evaluateImpl, the logic for recursing into child GlobNode-s is duplicated token-for-token for hasSpecials_ and !hasSpecials_. While fixing a bug in GlobNode::evaluateImpl, I was forced to update both copies, which is prone to mistakes.
Fix the duplication by factoring the code into a function.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D15090688
fbshipit-source-id: 65eec62214074139cb8af75026758b631eb89bb2
Summary:
gdb and lldb have a hard time printing recursive structures involving std::vector and std::unique_ptr. This makes debugging GlobNode a pain.
Add a function to print a GlobNode recursively to stderr. This makes it easier to debug GlobNode using printf.
Reviewed By: simpkins
Differential Revision: D15078111
fbshipit-source-id: 3ba211026c7c894676435ade278ae859e1b5d1f6
Summary:
Sandcastle's [1] code coverage builds compile EdenFS and Hg with Clang's -fprofile-generate flag (or a related flag). By default, running a program with with that flag creates a `default.profraw` file in the current directory. This causes some tests to fail, such as HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import, because `default.profraw` is unintentionally committed to the test repo:
```
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from HgBackingStoreTest
[ RUN ] HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import
eden/fs/store/hg/test/HgBackingStoreTest.cpp:64: Failure
Value of: tree1->getEntryNames()
Expected: has 2 elements where
element #0 is equal to foo,
element #1 is equal to src
Actual: { default.profraw, foo, src }, which has 3 elements
[ FAILED ] HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import (1563 ms)
[----------] 1 test from HgBackingStoreTest (1563 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1563 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import
```
When Sandcastle runs the tests, it sets the `LLVM_PROFILE_FILE` environment variable, which configures the path to `default.profraw`. EdenFS' HgRepo class is stripping this variable when invoking hg, so hg uses the default path and not the configured path.
Make HgRepo pass `LLVM_PROFILE_FILE` through to hg. This fixes some of EdenFS' tests when run by Sandcastle for code coverage purposes. (This does *not* fix running the tests manually (without setting `LLVM_PROFILE_FILE`), though.)
[1] Sandcastle is Facebook's continuous integration system.
Reviewed By: simpkins
Differential Revision: D15104832
fbshipit-source-id: 3929b9b0ab0904f2552ace7d8e4e4f4a4221192c
Summary:
Migrate the code from accessing optional Thrift fields directly to a safer
`optional_field_ref` API. See https://fburl.com/safe for more details.
The output of this codemod has been reviewed in D13259011.
To preserve semantics, each unchecked access is replaced with an explicit call
to `value_unchecked()`. If you are sure that accessing a field is safe (the
field is marked as set), you can later replace `value_unchecked()` with
`value()` or dereferencing (`operator *`):
```
ThriftStruct s = ...
- auto foo = s.foo_ref().value_unchecked();
+ auto foo = *s.foo_ref(); // will throw if s.foo is unset
```
Reviewed By: fanzeyi
Differential Revision: D15078040
fbshipit-source-id: 8ef270e8c687535e9d4aa2208beff4517e121814
Summary:
Add a command to force a call to RepairDB() on the local store.
This is similar to using `ldb repair`, but invokes `RepairDB()` with the same
set of column family options as normally used by edenfs.
Reviewed By: chadaustin
Differential Revision: D15043210
fbshipit-source-id: 2c4c0e2d3410a50cb1e523611f569f1701604ae6
Summary:
Change eden_store_util to take a subcommand argument, and define 3 separate
subcommands: "gc", "clear", and "compact"
The "gc" was the old default behavior. The "clear" and "compact" subcommands
simply clear cached entries and compact the storage, respectively.
This also adds a `--keySpace` argument. If this argument is specified the
command will process only this single key space, instead of all key spaces.
Reviewed By: chadaustin
Differential Revision: D15043207
fbshipit-source-id: 7e76bc89862d9647edde631ed3d82c9afda09f41
Summary:
This moves some logic from the RocksHandles class up to RocksDbLocalStore.
The main thing moved here is the logic to automatically try and repair the DB
if opening fails. This will make it easier in a subsequent diff to make the
repair logic a bit smarter and more aware of our column family semantics.
This keeps RocksHandles a pretty dumb wrapper around the RocksDB object and
column family handles, whose only purpose is to manage destroying these two
things in the correct order.
Reviewed By: chadaustin
Differential Revision: D15043208
fbshipit-source-id: ee2d5619ac7781a892e1ba151712eee9e3ebfb14
Summary:
Move this utility from eden/fs/service to eden/fs/store, and rename it to
`eden_store_util`.
I originally put it in the `service` subdirectory since it depends on library
code from `service/`, and most of the other code in `store/` shouldn't depend
on code in `service/`. However logically it makes sense to be in the `store/`
directory.
I plan to update in the future to be able to perform some additionally utility
operations besides just garbage collection, so rename it to the more generic
`eden_store_util` name.
Reviewed By: chadaustin
Differential Revision: D15043209
fbshipit-source-id: e6c4a874ddd05bd1576ea2e8a6c6156ccdf4628f
Summary:
On macOS, the atime, mtime, and ctime fields are named st_atimespec,
etc. Add some portability functions so the inode tests compile on
macOS.
Reviewed By: simpkins
Differential Revision: D15048758
fbshipit-source-id: 6f85b8eb7f7da2f9bd21c9034296cde96b68f740
Summary: Add some includes that are required on macOS.
Reviewed By: simpkins
Differential Revision: D15048681
fbshipit-source-id: d8642dc0977dc8cd8aadc83fc750400a8f917b7a
Summary:
Add a standalone program that can be used to garbage collect the local store
data when edenfs is not currently running.
Reviewed By: wez
Differential Revision: D14910392
fbshipit-source-id: 1cf8f959e5967cfbb797752653cc72223c87134f
Summary:
Let's catch more Xcode and upstream regressions by enabling more
builds and tests on macOS.
Reviewed By: simpkins
Differential Revision: D15003839
fbshipit-source-id: 257086d578f28198c7718906e1ec06395a55896e
Summary:
Build the store tests on mode/mac. RocksDB doesn't build yet (and we
will likely move away from it anyway) so split it into its own target.
Reviewed By: simpkins
Differential Revision: D15002840
fbshipit-source-id: fa0fea1d52df06526032a6fd585e2da7273910ef
Summary:
The hg store implementation depends on Rust code that doesn't build on
mode/mac yet, but TestMount's dependency on hg was false anyway.
Reviewed By: simpkins
Differential Revision: D15001077
fbshipit-source-id: 840fdf565093ad7a6eb95e001a74ae95eecbf1db
Summary: These allow more tests to compile
Reviewed By: chadaustin
Differential Revision: D14994582
fbshipit-source-id: 6b2a0b276fda64c7f27e28ea9e6d548aaaa1db7c
Summary: This diff add four metrics to the stats we are collecting in Eden. Both time cost on calling `get_tree` and `get_blob` from hg and mononoke backing store.
Reviewed By: chadaustin
Differential Revision: D14780349
fbshipit-source-id: f0e902c7f7523fac2ae576a67f0c85555064e26c
Summary:
The Finder shows the volume name instead of the directory
name when it renders the directory containing a mount, and the default
name of `OSXFuse Volume X` makes it hard to distinguish between multiple
mounts in the same folder.
This changes the volume name to be the basename of the mounted path so
that the finder renders things more reasonably.
Reviewed By: chadaustin
Differential Revision: D14969807
fbshipit-source-id: de9d8469360712828598fed124ed02ee5d6dd44a
Summary:
Unfortunately, some node modules packages include `.DS_Store` files
and since we disable apple double, the kext refuses to allow them to be
unpacked in the repo. Ideally they wouldn't be in the repo, but we need to
unblock that workflow.
Reviewed By: chadaustin
Differential Revision: D14969808
fbshipit-source-id: f9262294a223fd5cc7929a2b324f2401e1fed083
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: Orvid
Differential Revision: D14975808
fbshipit-source-id: b7790bf0e8965c9ffb3e63901bcdaf108480d4ca
Summary:
Add a helper function to recursively remove a directory from the local
filesystem, given an `AbsolutePath`.
Reviewed By: wez
Differential Revision: D14954737
fbshipit-source-id: a3f6b60e610c18eabea2a7c507661a43502a2554
Summary:
Add a new class to manage access to the .eden state directory, and move some
logic for managing this director from the `EdenServer` class and into this new
class.
This is largely just a first step. In the future I think we should move
additional functionality into this class. For instance, logic to determine
the correct local store type and logic to load the checkout map from
config.json. In the future it probably also make sense to move this class
from the `service` library into a slightly lower-level library (perhaps
`config` or `store`).
Reviewed By: chadaustin
Differential Revision: D14889617
fbshipit-source-id: 15cb7daa9b97500124152d016bfdd686d3039771
Summary:
Move some of the argument parsing and config setup code out into a new
EdenInit.h header file. This makes it possible to re-use this logic for other
standalone utilities that want to be able to find the Eden state directory and
config information.
For now I have updated the `fake_edenfs` helper tool used by the integration
tests to use this. This may also be useful for writing standalone tools that
can perform garbage collection of the LocalStore or checking of the overlay
state.
Reviewed By: chadaustin
Differential Revision: D14889616
fbshipit-source-id: b0b193a42cb2f52177d0c44592426b42e27242aa
Summary:
This effectively reverts D14452214, which caused Eden to write an `id` entry
to each RocksDB column family and then flush the column family each time
edenfs started.
There was relatively little benefit to this in practice. It only matters in
cases where the RocksDB column families never had enough data written to them
to get flushed automatically and then a repair is required.
On the other hand it does have some material downsides: it flushing the column
families can be fairly expensive, and can require a substantial amount of free
disk space. This flush caused some users to not be able to start up edenfs
when they did not have enough free disk space.
Reviewed By: chadaustin
Differential Revision: D14947235
fbshipit-source-id: a29f98163fa87185b028bb47945b6fab75700fd6
Summary: Fallout from another diff that never got landed.
Reviewed By: simpkins
Differential Revision: D14938191
fbshipit-source-id: 980728ca6a1d4c4f90fea46aac9525a03d9809e5
Summary:
Change the `eden gc` implementation to use RocksDB's `DeleteFilesInRange()`
function, as well as its new-ish `DeleteRange()` method.
This makes the garbage collection much faster, and also require much less free
disk space than previously.
`DeleteFilesInRange()` asks RocksDB to simply delete all SST files from disk
if they only contain keys in the specified range. Since the range we specify
should include all keys in the DB this should simply drop all SST files for
this column family.
We also call `DeleteRange()` after this, just in case. This API is relatively
recent, and writes a single tombstone saying that the specified range has been
deleted.
Reviewed By: wez
Differential Revision: D14910345
fbshipit-source-id: c76bdc1c8e07cb2def66673ea892e7f455c9dc7a
Summary:
EdenStats is currently an alias for ThreadLocal<EdenThreadStats>. I want to split EdenThreadStats into two structs which are allocated independently, but EdenStats's interface makes this impossible.
Refactor EdenStats from an alias to a class and encapsulate the underlying ThreadLocal<EdenThreadStats> member.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14822272
fbshipit-source-id: 691f4731aa22ecbdcd3535ee0bb0b99c816ffc3d
Summary:
I'm confused by the naming of EdenThreadStats and ThreadLocalEdenStats. For example, when I see "ThreadLocalEdenStats", I think that such objects can only be accessed by one thread. That's definitely not what "ThreadLocalEdenStats" means!
I think the following naming scheme makes more sense:
* **EdenThreadStats**: Statistics which are updated by one thread. Currently called EdenThreadStats.
* **EdenStats**: Statistics for all threads. Provides access to EdenThreadStats. Currently called ThreadLocalEdenStats.
Implement my preferred scheme: rename ThreadLocalEdenStats to EdenStats.
This diff should not change behavior.
Note: Prior to D14822274, EdenThreadStats was called EdenStats.
Reviewed By: simpkins
Differential Revision: D14822271
fbshipit-source-id: bd20179b1010588e3fc16dc9ed0657d458606f16
Summary:
I'm confused by the naming of EdenStats and ThreadLocalEdenStats. For example, when I see "ThreadLocalEdenStats", I think that such objects can only be accessed by one thread. That's definitely not what "ThreadLocalEdenStats" means!
I think the following naming scheme makes more sense:
* **EdenThreadStats**: Statistics which are updated by one thread. Currently called EdenStats.
* **EdenStats**: Statistics for all threads. Provides access to EdenThreadStats. Currently called ThreadLocalEdenStats.
Implement half of my preferred scheme: rename EdenStats to EdenThreadStats. (Make this diff easier to read by renaming ThreadLocalEdenStats to EdenStats in a separate diff.) In effect, implement the following naming scheme:
* **EdenThreadStats**: Statistics which are updated by one thread. (was EdenStats)
* **ThreadLocalEdenStats**: Statistics for all threads. Provides access to EdenThreadStats. (no change)
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14822274
fbshipit-source-id: 236cd9878cd249a06d14e124050ee01329667a18
Summary: This diff makes `MononokeHttpBackingStore` uses `ServiceAddress` to manage remote address of the Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D14855661
fbshipit-source-id: 1c102f7b2c547dc6a5b5c2cba65a513313d90bdc
Summary: This diff adds `ServiceAddress` to utils. This is a class that encapsulates an address of a service that could be a traditional hostname & port pair as well as an SMC tier name. This eliminates the needs of the manually resolving network address for remote services.
Reviewed By: chadaustin
Differential Revision: D14845257
fbshipit-source-id: 9fe9847cca4bba0170be94b9c209247342708574
Summary:
Sometimes, EdenFS goes bonkers and talks to 'hg debugedenimporthelper' a lot for seemingly no reason.
Make these situations easier to debug by counting how many requests EdenFS makes to 'hg debugedenimporthelper'. These counters let us answer questions such as the following:
* When performance sucks, is EdenFS is making a lot of requests or only a few requests? I.e. is Hg slow at responding or is EdenFS very demanding?
* Does a recent performance issue correlate with EdenFS communicating with 'hg debugedenimporthelper'?
* Which engineers are outliers having orders of magnitude more 'hg debugedenimporthelper' requests than p50 engineers?
We could get fancier with these counters and include the number of bytes received, the duration of the request, etc. For now, just having a request count is useful.
Reviewed By: simpkins
Differential Revision: D14677339
fbshipit-source-id: 7f8f394fb0096aef65d6a8a45d7da5936db539a0
Summary:
D8559702 changed `folly::IOBuf::computeChainDataLength()` to return a `size_t`
Update our format specifier to match to avoid compiler warnings on Mac.
Reviewed By: chadaustin
Differential Revision: D14878220
fbshipit-source-id: 19e96bea07c57bb542a848b3688d65143db51d13
Summary:
The file descriptor API here needs to go away, so switch this API to NetworkSocket
It is expected that this commit will cause a number of Open Source projects to temporarily show up as broken. This is due to the fact that not all projects get synced to Github at the exact same time, so the builds may temporarily be fetching an older version of it's dependencies than it needs to :) It should fix itself quickly.
Reviewed By: yfeldblum
Differential Revision: D14673328
fbshipit-source-id: c5842fa5dc383d50043e0d8228e35d03b10a1c6b
Summary:
Ensure that the `TreeInode::createImpl()` code always releases the contents
lock at the end of the scope that was intended to signify the critical
section. Previously this was only unlocked when returning from this scope
successfully.
In particular, the old behavior could deadlock:
- We first created the new child inode.
- If saveOverlayDir() failed, we would throw an exception, causing us to not
release the directory lock yet.
- As we returned in the exception case, the local `inode` variable would be
destroyed before the `contents` argument, causing us to release the child
InodePtr's refcount before releasing the parent directory's lock. Because
we were the only user of the child inode we would end up needing to acquire
the parent inode's content's lock to check if it was unlinked.
This fixes the code to ensure that we always release the tree's contents lock
before the child inode becomes unreferenced.
Reviewed By: chadaustin
Differential Revision: D14848807
fbshipit-source-id: 28dcde842072b9ee1e7c63d54456d849e31af8fe
Summary: Move Mononoke backing store initialization process to every request so we can adjust the configuration value without restarting the process.
Reviewed By: strager
Differential Revision: D14583357
fbshipit-source-id: 49ce2736229ce3062d34337757ebda6bb6eae16a
Summary: This is a helper function to help lazily initializing Mononoke backing stores only when necessary and release it when it is not used any more. See next diff in the stack for detail usage.
Reviewed By: chadaustin
Differential Revision: D14791528
fbshipit-source-id: c26811bc5c7aebcd02f704f10ad19bc35f8b9a21
Summary:
We updating the encoding that mercurial uses for storing paths to UTF-8.
The current step is preventing files that have invalid names from being
committed. To that end we are updating the HgPrefetch test to submit
valid file names (valid UTF-8). This allows us to start using UTF-8
encoded string for the internal representation of file paths.
Doing encoding transformation between internal storage and system locale will
be implemented later. Normalization should also be handled at that point.
Reviewed By: simpkins
Differential Revision: D14812780
fbshipit-source-id: 1689910500391eed941df185ba92aa2d2f3f0960
Summary:
I want to use EdenStats in eden/fs/store/. EdenStats currently lives in eden/fs/fuse/, and making eden/fs/store/ depend upon eden/fs/fuse/ is confusing. (It's also confusing that some code in eden/fs/fuse/ is used on Windows.)
Reorganize the code: move EdenStats into eden/fs/tracing/.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14677337
fbshipit-source-id: af26d214bcc3a9919920fbd4e59e6098fe4e3834
Summary:
This message was added in D14337058. It is logged at the `INFO` level, which
is enabled by default, but doesn't seem to add much value to normal production
logs.
Reviewed By: chadaustin
Differential Revision: D14712654
fbshipit-source-id: 5a86d883ace30e22d299046e33a6cd6247432857
Summary:
we need to be explicit about pulling in the dep for the
case where folly is not installed into a default installation
prefix.
Reviewed By: strager, pkaush
Differential Revision: D14683955
fbshipit-source-id: 0f302877674fb744ef8076641cd3fa72de74efe4
Summary:
This isn't a complete install but it is sufficient
to generate an `install` target for the getdeps build to run.
Reviewed By: strager
Differential Revision: D14680670
fbshipit-source-id: 9de1caa24c25702795842fe5b1b1f4d82aef24d8
Summary:
While testing out the new getdeps code I found that none
of the include directories from the probed libraries were being used.
The new getdeps installs each dep into its own prefix, whereas the
existing getdeps script installed them into the installation
prefix for eden itself. That meant that they were being implicitly
found from a single include directory.
In addition to this, I encountered linker failures for the pretty
printers; the solution to those was to add appropriate deps for
the modules that depend upon the pretty printers.
Reviewed By: pkaush
Differential Revision: D14638758
fbshipit-source-id: a4c2b4c79603c268e1b1c707a05c3cb0e3f2757b
Summary:
This function depends on things that are not available
when EDEN_HAVE_HG_TREEMANIFEST is not enabled.
Ensure that the entire function is disabled if that is the case.
Reviewed By: chadaustin
Differential Revision: D14638759
fbshipit-source-id: 3fe83b7b42357c2b818469214b016ff591058e35
Summary:
The issue is that the compiler needs an `else` to see
that we can only reach the throw if none of the other paths are
taken; with that satisfied it believes that we are legitimately
constexpr.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67371
Reviewed By: chadaustin
Differential Revision: D14638234
fbshipit-source-id: f9524d2816580f41842a40e30118b03998c3660a
Summary: This is a largely automated codemod that shifts `folly::AsyncServerSocket::AcceptCallback::connectionAccepted` to taking a `NetworkSocket` rather than a file descriptor. This needs to be done as a single atomic change to avoid breaking things.
Reviewed By: yfeldblum
Differential Revision: D13966702
fbshipit-source-id: 415622dc347de53368c404dfbe9a2deae5b75e18
Summary: This diff enables thrift as a type of connection when talks to Mononoke API Server.
Reviewed By: strager, pkaush
Differential Revision: D14508956
fbshipit-source-id: 32e130c714be164f19452dec554976ea1cb594d3
Summary: Implement a backing store that uses thrift protocol to communicate with Mononoke API Server.
Reviewed By: strager
Differential Revision: D13966575
fbshipit-source-id: 66f66dda6b17aecd6c6b4475ab6b004c608f457f
Summary:
When Eden runs on kernel 4.20 (has readdir caching but not
FUSE_NO_OPENDIR_SUPPORT) indicate to the kernel that Eden wants
readdir results to be cached.
Reviewed By: wez
Differential Revision: D13893922
fbshipit-source-id: 3092adb16dabc4273bdba1e6e9f15e9e3bd7bb87
Summary:
Eden requires no state in its directory handles, so tell the kernel it
doesn't need to send opendir() and releasedir() requests, provided it
has FUSE_NO_OPENDIR_SUPPORT.
Reviewed By: strager
Differential Revision: D13594734
fbshipit-source-id: ebd4b69f4efcd1428a69024c4bdffb1ae455fa40
Summary:
After the kernel added readdir caching, my testing uncovered that Eden
was invalidating TreeInode entries incorrectly when new entries were
added. Change TreeInode to distinguish between directory entry changes
and removals (FUSE_NOTIFY_INVAL_ENTRY) and additions
(FUSE_NOTIFY_INVAL_INODE).
Reviewed By: strager
Differential Revision: D13870422
fbshipit-source-id: 2a6f25bfd9e77436a5aae639fedbfd8a445b2e05
Summary:
Prior to D14398507, two calls to EdenMount::unmount would result in two calls to fuseMountPromise->getFuture(). D14398507 made two calls to EdenMount::unmount result in only one call to fuseMountPromise->getFuture().
Since fuseMountPromise->getFuture() is only called at most once, turn fuseMountPromise into a Promise instead of a SharedPromise.
Reviewed By: simpkins
Differential Revision: D14515070
fbshipit-source-id: 3ecaf9b55f3274046bc840bd5a65481cc552df72
Summary:
I want to make EdenMount robust if mounting and unmounting are requested concurrently. This will make it safer to call EdenMount::destroy in different situations.
Take a step toward that goal by making EdenMount::unmount idempotent; calling EdenMount::unmount multiple times should return the same future.
Changed behaviors:
* If EdenMount::unmount is called multiple times, EdenMount::fuseMount is called at most once, and each call to EdenMount::unmount waits for the EdenMount::fuseMount call to finish.
* **Old behavior**: One of the calls to unmount succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Unchanged behaviors:
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse fails with EdenMountCancelled.
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount waits for fuseMount to finish before calling PrivHelper::fuseUnmount.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::takeoverFuse is in progress, EdenMount::unmount does not cause EdenMount::takeoverFuse to fail.
* If EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If neither EdenMount::startFuse nor EdenMount::takeoverFuse was never called, EdenMount::unmount succeeds and does not call PrivHelper::fuseUnmount.
Reviewed By: simpkins
Differential Revision: D14398507
fbshipit-source-id: 1da5424abc47341e4db3da1c50d425711e177366
Summary:
I want to make EdenMount robust if mounting and unmounting are requested concurrently. This will make it safer to call EdenMount::destroy in different situations.
Take a step toward that goal by making EdenMount::unmount succeed if EdenMount::startFuse was never called.
Changed behaviors:
* If neither EdenMount::startFuse nor EdenMount::takeoverFuse was never called, EdenMount::unmount succeeds and does not call PrivHelper::fuseUnmount.
* **Old behavior**: EdenMount::unmount fails with an exception from PrivHelper.
Unchanged behaviors:
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse fails with EdenMountCancelled.
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount waits for fuseMount to finish before calling PrivHelper::fuseUnmount.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::takeoverFuse is in progress, EdenMount::unmount does not cause EdenMount::takeoverFuse to fail.
* if EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::unmount is called multiple times, one of the calls succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Reviewed By: simpkins
Differential Revision: D14398537
fbshipit-source-id: 9209aad8b2980045fd0f89a5ba149a833579c2cf
Summary:
I want to make EdenMount robust if mounting and unmounting are requested concurrently. This will make it safer to call EdenMount::destroy in different situations.
Take a step toward that goal by making EdenMount::unmount complete only after a concurrent PrivHelper::fuseMount call finishes.
Changed behaviors:
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount waits for fuseMount to finish before calling PrivHelper::fuseUnmount.
* In other words, if EdenMount::startFuse and EdenMount::unmount execute concurrently, either:
* mount(2) is never called (if EdenMount::unmount won the race), or
* mount(2) is called then umount(2) is called (if EdenMount::startFuse won the race).
* **Old behavior**: EdenMount::unmount calls PrivHelper::fuseUnmount immediately and either fails or succeeds.
Unchanged behaviors:
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse fails with EdenMountCancelled.
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount causes the fuseMount to be rolled back (by calling PrivHelper::fuseUnmount) after fuseMount completes.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* if EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::unmount is called multiple times, one of the calls succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Reviewed By: simpkins
Differential Revision: D14398508
fbshipit-source-id: 6f01ce9e36b5802045acf54afe756b80fe102b1e
Summary:
D14398423 made EdenMount::startFuse fail immediately if EdenMount::unmount was ever called. EdenMount::takeoverFuse behaves differently; any prior EdenMount::unmount calls don't cause EdenMount::takeoverFuse to fail.
For consistency, make EdenMount::takeoverFuse fail in the same situations that make EdenMount::startFuse fail.
Changed behaviors:
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse fails with EdenMountCancelled.
* **Old behavior**: takeoverFuse succeeds.
* **Note**: EdenMount::unmount still fails with an exception from PrivHelper.
Unchanged behaviors:
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount causes the fuseMount to be rolled back (by calling PrivHelper::fuseUnmount) after fuseMount completes.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::takeoverFuse is in progress, EdenMount::unmount does not cause EdenMount::takeoverFuse to fail.
* if EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::unmount is called multiple times, one of the calls succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Reviewed By: simpkins
Differential Revision: D14511630
fbshipit-source-id: b77ca5ee39c888e2d50547f76977db7aa5a0331a
Summary:
I want to make EdenMount robust if mounting and unmounting are requested concurrently. This will make it safer to call EdenMount::destroy in different situations.
Take a step toward that goal by making EdenMount::unmount cancel a concurrent EdenMount::startFuse call.
Changed behaviors:
* If EdenMount::startFuse is in progress, EdenMount::unmount causes the concurrent startFuse call to fail with FuseDeviceUnmountedDuringInitialization.
* **Old behavior**: startFuse might succeed or might fail with FuseDeviceUnmountedDuringInitialization.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount causes the fuseMount to be rolled back (by calling PrivHelper::fuseUnmount) after fuseMount completes.
* **Old behavior**: EdenMount::unmount does not change the behavior of EdenMount::startFuse.
* **Note**: EdenMount::unmount still fails with an exception from PrivHelper, despite having an effect on EdenMount::startFuse. This quirk will be addressed by a future diff.
* If EdenMount::unmount was ever called, EdenMount::startFuse fails with EdenMountCancelled.
* **Old behavior**: startFuse succeeds.
* **Note**: EdenMount::unmount still fails with an exception from PrivHelper, despite having an effect on EdenMount::startFuse. This quirk will be addressed by a future diff.
Unchanged behaviors:
* If neither EdenMount::startFuse nor EdenMount::takeoverFuse was never called, EdenMount::unmount fails with an exception from PrivHelper.
* If EdenMount::startFuse -> PrivHelper::fuseMount is in progress, EdenMount::unmount calls PrivHelper::fuseUnmount immediately and either fails or succeeds.
* If EdenMount::startFuse -> PrivHelper::fuseMount is complete, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::takeoverFuse is in progress, EdenMount::unmount does not cause EdenMount::takeoverFuse to fail.
* If EdenMount::takeoverFuse was called, EdenMount::unmount calls PrivHelper::fuseUnmount immediately.
* If EdenMount::unmount was ever called, EdenMount::takeoverFuse succeeds.
* If EdenMount::unmount is called multiple times, one of the calls succeeds, and the remaining calls to unmount fail with an exception from PrivHelper.
Reviewed By: simpkins
Differential Revision: D14398423
fbshipit-source-id: eb86a237859ad2d992de5c901e63dda80d5ea6d5
Summary:
In future diffs, I will introduce some EdenMount tests which need to mock PrivHelper in more ways than are currently supported. For example, I want to mock PrivHelper::fuseUnmount and check how many times fuseUmount is called. Extend the existing MockMountDelegate and FakeFuseMountDelegate classes to support these to-be-written tests.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14418206
fbshipit-source-id: a9b1257bb62d9a46fae0eef89c3fa42718bec142
Summary:
Eden needs the osxfuse version of the FUSE protocol on mac. The open
source cmake build pulls in osxfuse with getdeps.py, but that doesn't
work in the Buck build, and all we need is this one (old) copy of the
Linux kernel headers.
Reviewed By: strager
Differential Revision: D14506747
fbshipit-source-id: 028ddbaf80be9cad412462f3338c30b8f2a70087
Summary:
If EdenMount::takeoverFuse is called and createFuseChannel fails, the EdenMount reports that it's in the STARTING state. This is misleading and doesn't match how EdenMount::startFuse handles failures in createFuseChannel.
Extend the `try` block in EdenMount::takeoverFuse to handle exceptions raised from createFuseChannel.
Reviewed By: simpkins
Differential Revision: D14511564
fbshipit-source-id: 8f791efe1221bcfc1042f95054d181c303564086
Summary:
Update `RocksHandles` to call `RepairDB()` if we get an error opening the
database.
We have seen errors opening the DB in some cases after hard server reboots.
In every case so far `ldb repair` has been able to repair it with no adverse
effects. This simply makes `edenfs` automatically attempt to perform this DB
repair step.
Reviewed By: chadaustin, strager
Differential Revision: D14452216
fbshipit-source-id: 10c0cb0ff9cea3c3bbe485a4e489e4a6df640803
Summary:
When opening the RocksDB, write one entry to each column family and then
explicitly flush that column family. This ensures that the column family
information has actually been flushed to an SST file. Without this some
column families may only have been written out to the write-ahead log files.
(Even calling `db->Flush()` does not appear to be sufficient; each column
family has to be explicitly flushed.)
The RocksDB' `RepairDB()` function (used by `ldb repair`) currently ends up
deleting column families that do not have any data defined in an SST file.
The repair tool ends up deleting column families that only have data in log
files.
The fact that we haven't been doing these explicit flushes previously probably
isn't too much of a concern in practice: once we write out enough data RocksDB
will automatically trigger a flush. This only matters in cases where we have
not yet written out enough data to trigger an automatic flush.
Note that with this change we re-write these `id` keys each time we open the
RocksDB store, even if they were already present.
Reviewed By: chadaustin, strager
Differential Revision: D14452214
fbshipit-source-id: 3f1b17e240cc89fe00e3d31105d16452795e754d
Summary:
folly::readFile() doesn't work correctly on Windows. It was returning the incorrect size. I didn't check if the read contents were correct, because I was testing it on a binary data file.
Moreover we would need wide char API which won't be a part of Folly for sometime.
Reviewed By: strager
Differential Revision: D14205306
fbshipit-source-id: 55859dca1a6399d5b5f106b4c6757e582ba1375d
Summary:
Rename `main.cpp` to `EdenMain.cpp` now that this contains the `EdenMain`
class rather than the top-level `main()` function, and rename `RunServer.cpp`
to `main.cpp`
Reviewed By: pkaush
Differential Revision: D14435306
fbshipit-source-id: e1528a773e0724c6bd50e31f6b33a1762d7bd49e
Summary:
This restructures `main.cpp` to turn our current top-level `main()` into a
method of a virtual class.
This allows us to eliminate the slightly awkward way that `main.cpp` depends
on `RunServer.cpp`. This required us to list `main.cpp` in the sources
parameter of multiple buck targets rules, which confuses some other tools like
autodeps. With this new change, `RunServer.cpp` depends on `main.cpp` instead
of the other way around. The real top-level `main()` function now lives in
`RunServer.cpp`
Reviewed By: pkaush
Differential Revision: D14435309
fbshipit-source-id: 402d00db0d8aa8d473d51a4f0e9d9d80c97a0134
Summary:
The systemctl command requires XDG_RUNTIME_DIR to be configured. If it's not configured, 'eden start' should pick a sane default. The sane default includes the user's UID (e.g. /run/user/6986). I want this default to be configurable via Eden's config files.
Expose the ${USER_ID} token to Eden configs. This will let administrators can customize XDG_RUNTIME_DIR's fallback value in the future.
Reviewed By: wez
Differential Revision: D13811732
fbshipit-source-id: 7933e078dd5f2b3bbbb0299730220a129c257256
Summary:
Implement `flush()` and `fsyncdir()` in `EdenDispatcher` and explicitly return
ENOSYS. We intentionally do not need to do any work for these methods.
Providing our an implementation that returns ENOSYS avoids the error log
message in the default `Dispatcher` implementation about the these calls being
unimplemented.
Reviewed By: chadaustin
Differential Revision: D14453245
fbshipit-source-id: 71efe6de6af73a5d705dace0f3439ba2466a50a8
Summary:
Update the FuseChannel code to explicitly handle `FUSE_LSEEK` and `FUSE_POLL`
to avoid emitting error log messages about these calls not being implemented.
We intentionally do not implement these operations at the moment.
Also drop the log about other unknown opcodes from an error log to a warning
message.
Reviewed By: chadaustin
Differential Revision: D14453246
fbshipit-source-id: 2605cf7e5c160cda92460a80187438ac3549ade5
Summary:
I want to change some of the StartupLogger behavior to automatically show all
INFO and higher level log messages to the user during start-up. These two
messages from the FUSE code are currently logged at the INFO level, but don't
really seem important enough to show to the user.
This drops their level to DBG1. These messages will therefore still be shown
in the normal `edenfs.log` file, since it includes everything from DBG2 and
up.
Reviewed By: chadaustin
Differential Revision: D14453247
fbshipit-source-id: 5d79766f87e658b807029d82ac035cb94ae68832
Summary:
The Windows headers and source files currently aren't mentioned in any buck
build rule, which confuses autodeps. Mark them all `manual` to tell
autodeps to simply ignore them.
Reviewed By: strager
Differential Revision: D14435307
fbshipit-source-id: a10b9be779f232cc5ae74356d7f54e77ca5414cb
Summary:
- Wait to call `curl_global_init()` until after we have dropped root
privileges. `curl_global_init()` performs a non-trivial amount of work,
including loading and processing OpenSSL configuration files from disk. To
guard against any possible security issues in this code we should wait until
after we have dropped root privileges to do this.
- Move the call to `curl_global_cleanup()` to after the main `EdenServer`
object has been destroyed. Since the EdenServer object will likely contain
backing store objects that use curl it seems like we want to make sure we
wait to clean up the curl library until after the `EdenServer` has been
destroyed. This change uses `SCOPE_EXIT` to perform the cleanup as one of
the last steps of `main()`, and this also reduces the number of `#ifdef`
blocks that we need.
Reviewed By: strager, fanzeyi
Differential Revision: D14435308
fbshipit-source-id: 6c277e4471f0f93decebd4fc741639c6a047d62a
Summary: We don't run this binary anymore, no reason to build and ship it.
Reviewed By: quark-zju
Differential Revision: D14437317
fbshipit-source-id: dd6da521783f18a2a518a7aa042be98950894e89
Summary:
The future returned by EdenMount::shutdown() can continue on an arbitrary thread. This has caused at least one deadlock in the past (D14337058).
Prevent users of EdenMount::shutdown() from continuing on the wrong thread by making shutdown return a SemiFuture instead of a Future.
This diff should not change behavior for production code, since all users of EdenMount::shutdown() already use Future<>::via().
Reviewed By: simpkins
Differential Revision: D14389607
fbshipit-source-id: 821d206d01ea3dcb0261cd0c7ca2d591b2c84e7a
Summary:
If TreeInode::startLoadingInode() is in progress, and EdenServer::startTakeoverShutdown() is called, edenfs can deadlock:
1. Thread A: A FUSE request calls TreeInode::readdir() -> TreeInode::prefetch() -> TreeInode::startLoadingInode() on the children TreeInode-s -> RocksDbLocalStore::getFuture().
2. Thread B: A takeover request calls EdenServer::performTakeoverShutdown() -> InodeMap::shutdown().
3. Thread C: RocksDbLocalStore::getFuture() (called in step 1) completes -> TreeInode::inodeLoadComplete(). (The inodeLoadComplete continuation was registered by TreeInode::registerInodeLoadComplete().)
4. Thread C: After TreeInode::inodeLoadComplete() returns, the TreeInode's InodePtr is destructed, dropping the reference count to 0.
5. Thread C: InodeMap::onInodeUnreferenced() -> InodeMap::shutdownComplete() -> EdenMount::shutdown() (called in step 2) completes -> EdenServer::performTakeoverShutdown().
6. Thread C: EdenServer::performTakeoverShutdown() -> localStore_.reset() -> RocksDbLocalStore::~RocksDbLocalStore().
7. Thread C: RocksDbLocalStore::~RocksDbLocalStore() signals the thread pool to exit and waits for the pool's threads to exit. Because thread C is one of the threads managed by RocksDbLocalStore's thread pool, the signal is never handled and RocksDbLocalStore::~RocksDbLocalStore() never finishes.
Fix this deadlock by executing EdenServer::shutdown()'s callback (in EdenServer::performTakeoverShutdown()) on a different thread.
Reviewed By: simpkins
Differential Revision: D14337058
fbshipit-source-id: 1d63b4e7d8f5103a2dde31e329150bf763be3db7
Summary:
FuseChannel::initialize throws runtime_error when the FUSE connection is interrupted. I want EdenMount::startFuse to handle unexpected-unmount errors from FuseChannel::initialize, but catching runtime_error would catch unrelated errors too.
When the FUSE connection is interrupted during initialization, make FuseChannel throw FuseDeviceUnmountedDuringInitialization instead of runtime_error.
Reviewed By: chadaustin
Differential Revision: D14077848
fbshipit-source-id: ed7b7d370a83ed1a9c36a443d8bb06ba940dc032
Summary:
In this change, we separate the low-level code that manipulates the overlay
into the FsOverlay class. The Overlay class makes use of the FsOverlay and
InodeMetaData table to support its Overlay interfaces. The FsOverlay class
is decoupled from the Overlay class, allowing other classes to manipulate
the overlay independently. We have a need for this in order to add
fsck to the c++ code base : described in T40728883.
Reviewed By: simpkins
Differential Revision: D14218281
fbshipit-source-id: 66c587f2b341579b8075ca5e5eeb4da6ffadf6f5
Summary: Since `CurlClient` is going to be used in multiple threads and each thread has its own instance of CurlClient, we are now able to assign one curl_easy handle for each of the client. So curl would be able to reuse the existing connections to Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D14334486
fbshipit-source-id: aacd210773c44dea6f1d9dd7f9b09d765a9c7a0a
Summary: Put `CurlClient` into its own thread pool (like how `HgBackingStore` uses `HgImporter`)
Reviewed By: chadaustin
Differential Revision: D14334483
fbshipit-source-id: 3ce2a8245761c11bfa0f66691d10a66004b46fe6
Summary: This diff enables `MononokeCurlBackingStore` in HgBackingStore. It also adds a new config value `mononoke:connection-type` (values: `http`, `curl` and `thrift`).
Reviewed By: chadaustin
Differential Revision: D14135028
fbshipit-source-id: cc6fb95d94b3d98996d872d22ad1a95a97d562ab
Summary: This diff add a MononokeBackingStore implemented based on libcurl so we could add Mononoke support on macOS.
Reviewed By: chadaustin
Differential Revision: D14096313
fbshipit-source-id: 2bcb0c49002dcbb0a0190ba505a6e814e74db747
Summary:
This diff added a configurable value "mononoke:connector" to EdenConfig, so that we can specify a particular type of connection we need when using Mononoke as backing store.
This diff also moved the logic of initializing the existing `MononokeHttpBackingStore` to `initializeHttpMononokeBackingStore`.
Reviewed By: wez, strager
Differential Revision: D14141725
fbshipit-source-id: c7208295d7b3853740d7b0e5166f8854457fcf8e
Summary:
Not sure why gflags is automatically included on Linux and not in
mode/mac, but add the includes nonetheless!
Reviewed By: strager
Differential Revision: D14389458
fbshipit-source-id: 27811ec4bb65b03e73b15bb51de1264dbfe053dc
Summary: Internal Facebook infrastructure is nagging me about some files not having a Facebook copyright notice. Add a notice to these files to make the nagging stop.
Reviewed By: simpkins
Differential Revision: D14173944
fbshipit-source-id: 7234431224fcf4f86ea56ca2f9108f47ef959d87
Summary:
This updates `edenfs` to automatically create the mount point directory
if it does not exist.
Previously the `eden mount` CLI command would automatically create the mount
directory in the Python logic. This adds similar logic to the C++ code, which
handles more situations. In particular this makes it so that `eden start`
will now automatically create missing mount point directories.
Note that the C++ code does not create the `README_EDEN.txt` symlink inside
the mount point if it is missing. We could move that functionality into the
C++ code in the future if needed.
Reviewed By: strager
Differential Revision: D14254699
fbshipit-source-id: bad5634f57fba6e7af3b6a3830eb51ac099b435e
Summary:
Update `Overlay::initialization()` to perform the bulk of the initialization
logic in the GC thread. We re-use the GC thread simply for convenience, since
it already exists and won't have any work to do until initialization has
completed anyway.
After an unclean shutdown this allows edenfs to start and perform the overlay
scans for all of its checkouts in parallel rather than serially.
Reviewed By: wez
Differential Revision: D14154868
fbshipit-source-id: c177cb9f950a6317fd7ea06888bd5b326a55ace7
Summary:
Add an Overlay::initialize() function, and consolidate all Overlay
initializtion logic in this function. Previously some initialization was done
by the Overlay constructor, and some was done in `EdenMount::initialize()`
Overlay::initialize() returns a folly::SemiFuture as it may perform a
non-trivial amount of disk I/O. However, at the moment it currently performs
all I/O in the current thread before it returns. I plan to move this work to
a separate thread in a subsequent diff.
Reviewed By: strager
Differential Revision: D13981140
fbshipit-source-id: b59eaef88012a8e74fcb770a9c93ca3f9bde32a0
Summary:
This updates the `EdenServer` class so that the existing `getMount()` and
`getMountPoints()` APIs only return mounts that have finished initializing.
These APIs are primarily used by the thrift interfaces. In most cases the
callers did not intend to operate on mounts that were still initializing, and
doing so was unsafe. The code could potentially dereference a null pointer if
it tried to access the mount's root inode before the root inode object had
been created.
New `getMountUnsafe()` and `getAllMountPoints()` APIs have been added for call
sites that explicitly want to be able to access mounts that may still be
initializing. Currently the `listMounts()` thrift API is the only location
that needs this.
Reviewed By: strager
Differential Revision: D13981139
fbshipit-source-id: e6168d7a15694c79ca2bcc129dda46f82382e8e9
Summary:
Add a flag to tell edenfs to report successful start-up as soon as the thrift
server is running, without waiting for all mount points to finish being
remounted.
In the future I plan to have edenfs automatically perform an fsck scan of the
overlay for checkouts that were not shut down cleanly. This may cause the
remount to take a significant amount of extra start-up time in some cases.
(This is already true today in some cases even with the simpler scan we do to
re-compute the max inode number.)
I think we will probably want to have systemd invoke edenfs with this option,
so that we do not time out during system start up if some mount points need to
be rescanned.
Reviewed By: strager
Differential Revision: D13522040
fbshipit-source-id: 6f183770c25efee34c4805c9bad42a9cce51039e
Summary:
In the future, I want some tests to mock PrivHelper::fuseUnmount. The existing classes for mocking PrivHelper::fuseMount, MountPromiseDelegate and FailingMountDelegate, work poorly for fuseUnmount.
Combine MountPromiseDelegate and FailingMountDelegate to create MockMountDelegate, which is more flexible and allows a mocking fuseUnmount to be implemented easily.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14319712
fbshipit-source-id: 6eecf121e3b44f39cd5dffbeafaf0f3aab75cb76
Summary: Switch to using cpptoml from third-party on all platforms.
Reviewed By: igorsugak
Differential Revision: D14351179
fbshipit-source-id: 183bedc7d27e9c0f9216f3d0cca58ada75ac74e7
Summary: Ensure that an EdenMount is RUNNING after it takes over an EdenMount from another process.
Reviewed By: simpkins
Differential Revision: D14178500
fbshipit-source-id: 2813b024b6815dc5d31f3e3bf89cffa98ad0f1c3
Summary:
I want manipulate a FakeFuseMountDelegate in a test, but FakeFuseMountDelegate is private to FakePrivHelper.cpp. Make FakeFuseMountDelegate usable outside FakePrivHelper.cpp by putting its declaration in FakePrivHelper.h.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14319711
fbshipit-source-id: d7af18c316f63febe1d60f6e5aadcd4f4827cca5
Summary:
Update `edenfs` to automatically create the bind mount source directories if
they are missing. Previously Eden would report an error and would not be able
to mount the checkout if some of the bind mount source directories were
missing.
Reviewed By: strager
Differential Revision: D14253771
fbshipit-source-id: 87ad091ccf2c0f0f72aebb50437fd7680ddbfd1c
Summary:
[Folly] Stop checking `EventBase::runInEventBaseThread` result, as the function will soon be changed not to return any result.
It returned `false` when failing to enqueue a task. But it cannot really fail anyway besides allocation failure, unless in the `EventBase` destructor and while draining and the `AlwaysEnqueue` variant is called.
Reviewed By: andriigrynenko
Differential Revision: D14254969
fbshipit-source-id: a6a9199cbafa18b61488a240e4318ce946953f51
Summary:
We're not that far from building privhelper on mode/mac but it does
require figuring out how to depend on osxfuse from the Buck build, so
bypass that by breaking the inodes target's dependency on privhelper's
<fuse_ioctl.h> include.
Reviewed By: strager
Differential Revision: D14218709
fbshipit-source-id: edbb2a21df06d6f2a4f860ef13718ad05d445e98
Summary: Fix some missing includes that showed up in my mode/mac builds of parts of Eden.
Reviewed By: simpkins, strager
Differential Revision: D14213572
fbshipit-source-id: 54e070e89afdfb8479abdaa122ba76ca1d2d9ba9
Summary:
Throw a slightly nicer error message if the hg import helper exits before
returning a `CMD_STARTED` response or an error response.
When switching from the `hg_import_helper.py` script to the
`hg debugedenimporthelper` subcommand we are slightly more likely to encounter
errors during startup. For instance, if the repository path was not a valid
repository the `hg_import_helper.py` code would send an error message back
that the `HgImporter.cpp` code could parse. However, we unfortunately can't
catch errors loading the repository the same way when using an `hg`
subcommand. If it fails to load the repository it will simply print an error
message to stderr and then bail out before it invokes the
`debugedenimporthelper` code.
Reviewed By: chadaustin
Differential Revision: D14222266
fbshipit-source-id: cd60e5a61725d45a816b74f63b9969b29ade2160
Summary:
EdenMount calls fuseMount, but EdenServer calls fuseUnmount. I think only one of EdenMount or EdenServer should call fuseMount/fuseUnmount. Splitting the responsibility is confusing.
Move the call to fuseUnmount into EdenMount by creating a member function called unmount.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14115385
fbshipit-source-id: 845c4a87cdba9f270c0eacaf01916ad91c3b7c0e
Summary:
Update `EdenMount::initialize()` to perform a fault injection check. This
allows test code to inject delays and errors into the mount initialization
flow.
Reviewed By: strager
Differential Revision: D14079491
fbshipit-source-id: be80135b0833c8f0300104524473cc3e949fec34
Summary:
This reverts the change in D14188312 that moved `hg_import_helper.py` from
`bin/` to `libexec/eden/`, and instead updates edenfs to look for
`hg_import_helper.py` at `../../bin/hg_import_helper.py`
We cannot remove `hg_import_helper.py` from `/usr/local/bin` without breaking
existing running `edenfs` daemons as they will continue to look for
`hg_import_helper.py` in this location.
Rather than keeping a copy in both `bin/` and `libexec/eden` it seems better
to simply update edenfs so that it can correctly find `hg_import_helper.py`
even when edenfs itself has been installed in `libexec/eden/`
Reviewed By: strager
Differential Revision: D14197390
fbshipit-source-id: 2845bd79343bc29af9d3acbacecac4501cdb9032
Summary: We should only attempt to include selinux on Linux, not macOS.
Reviewed By: strager
Differential Revision: D14181109
fbshipit-source-id: be47b7bdadc3409577fa114559e905214848ebd8
Summary:
The Eden CLI tool is really a control program for `edenfs`. Rename it to
`edenfsctl` to free up the `eden` name for future use.
The Eden daemon shouldn't really be on the user's path, and instead belongs in
`libexec`.
For transition compatibility, `eden` is symlinked to `edenfsctl`.
Reviewed By: simpkins
Differential Revision: D13888875
fbshipit-source-id: 435cc63e92b85b1f28b8691e4846fbcb05bc450e
Summary:
futures::sleep returning a Future leads to continuations easily being run on
the Timekeeper's callback. The goal is to change the return type so that
futures::sleep returns a folly::SemiFuture.
This codemod is part of the first stage:
* Migrate all call sites off of futures::sleep and onto futures::sleepUnsafe.
This will be followed by:
* Changing the return type of futures::sleep to folly::SemiFuture.
* Migrating callers, where clearly safe such as when they follow with .via or
.semi() from sleepUnsafe to sleep.
* Migrating remaining callers.
Reviewed By: yfeldblum
Differential Revision: D14152623
fbshipit-source-id: bc6874e4320e4a7352ac61b20d9750458e2cbbff
Summary:
If EdenMount::shutdown is in progress and EdenMount::destroy is called [1], the EdenMount object won't ever be deleted. A comment in EdenMount::destroy claims that EdenMount::shutdown will delete the EdenMount object, but the comment is wrong:
case State::SHUTTING_DOWN:
// Nothing else to do. shutdown() will destroy us when it completes.
return;
This leak is probably benign, but the leak checker can cause tests to fail spuriously. I think the leak currently can't happen for EdenServer's mounts, but in the future the leak might occur and cause problems.
Make EdenMount::shutdown delete the EdenMount if EdenMount::destroy was called. This fixes the leak.
[1] EdenMount::destroy is called by EdenMountDeleter when the last std::shared_ptr<EdenMount> is destroyed.
Reviewed By: simpkins
Differential Revision: D14015024
fbshipit-source-id: 39a0ba78bb00afb5f0363fe5c74dbec00f9d68e3
Summary:
If EdenMount::shutdownImpl and EdenMount::startFuse execute concurrently, the mount's `state_` might end up inconsistent. For example, for the following sequence of events, EdenMount::shutdownImpl is erroneously called twice:
1. EdenMount::initialize transitions the mount's state to INITIALIZED.
2. EdenMount::startFuse begins and transitions the mount's state to STARTING.
3. EdenMount::shutdown begins and transitions the mount's state to SHUTTING_DOWN.
4. EdenMount::shutdown finishes and transitions the mount's state to SHUT_DOWN.
5. EdenMount::startFuse finishes due to FuseChannel::initialize failing with an error and transitions the mount's state to FUSE_ERROR.
6. EdenMountDeleter calls EdenMount::destroy, which observes that the mount's state is FUSE_ERROR. EdenMount::destroy proceeds by calling EdenMount::shutdownImpl.
This specific scenario can't happen in production right now because step 3 requires allowFuseNotStarted to be true (and it's never true in production).
A similar scenario can happen with EdenMount::startFuse+EdenMount::destroy running currently, but the effects are benign. However, in a future diff (D14015024), I am changing how EdenMount::destroy works, and the SHUT_DOWN -> FUSE_ERROR transition causes problems.
When FuseChannel::initialize returns an error, transition the mount to the FUSE_ERROR state only if we are not already shutting down (instead of unconditionally). This fixes the [impossible] scenario above, and also fixes EdenMount::destroy in a future diff.
Reviewed By: simpkins
Differential Revision: D14030898
fbshipit-source-id: fe5a749b8798749734e0de91eb3b71b601ebb64b
Summary:
This diff adds the dtype field to the glob results;
this will help to reduce the cost of some watchman queries by avoiding a
getFileInformation call that instantiates inodes.
As part of this, I added a bunch of unit test coverage.
Reviewed By: strager
Differential Revision: D8779149
fbshipit-source-id: 3064a3e42be55ec576fed9e0f7112edef426f32d
Summary:
Add a new class for injecting faults into Eden. This will make it easier to
build tests that exercise specific error conditions or specifically control
the ordering of operations that might otherwise race.
To minimize the performance impact, the fault injection framework is disabled
by default and must be explicitly enabled at process startup using a command
line flag. If this command line flag was not specified all fault injection
checks short-circuit into a no-op. The `checkAsync()` version does still add
some extra overhead due to the addition of a new `folly::SemiFuture` into the
code path.
Reviewed By: wez
Differential Revision: D14079489
fbshipit-source-id: 3e8725d2e51e5c829199a602e90574424ea52636
Summary:
If PrivHelper::fuseMount fails with an error, EdenMount::getState returns STARTING. This is confusing. Make EdenMount::getState instead return FUSE_ERROR in this case.
Aside from changing the output of 'eden list' in some cases, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14060982
fbshipit-source-id: ae9c2c532b1654cce36b806971053d89231058c3
Summary:
While moving code around in EdenMount::startFuse, EdenMount::shutdown, and EdenMount::destroy, I accidentally changed the result of EdenMount::getState in some circumstances. I noticed that we don't have any tests for EdenMount::getState, so it could have been broken even before my changes!
Write some tests for EdenMount::getState to verify the existing behavior and to prevent future regressions. Only add tests which pass with the current implementation; future diffs will fixe cases where EdenMount::getState returns the wrong value.
Reviewed By: chadaustin
Differential Revision: D14058334
fbshipit-source-id: 68b60ac92f94d336fcfb8f11e571379573bb3f42
Summary:
Some tests want to create a TestMount but don't care what files are in the mount. The following code is intuitively correct. However, TestMount's constructor takes a FakeTreeBuilder by non-const reference, so it fails to compile:
auto testMount = TestMount{FakeTreeBuilder{}};
The following code works, but more verbose:
auto builder = FakeTreeBuilder{};
auto testMount = TestMount{builder};
Overload TestMount's constructor to allow the former form in addition to the latter form. This neatly shrinks some test code.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14060110
fbshipit-source-id: c6a5d4b2c5859812efff279dedbc1fe690c8f8ad
Summary:
Make it easier to write tests involving a FakeFuse-mounted EdenMount by creating the TestMount::startFuseAndWait function which initializes EdenMount.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14059808
fbshipit-source-id: ce63bbce7f2c6e710910ad530aa1b4efefe80272
Summary:
I want to test EdenMount::startFuse when PrivHelper::fuseMount fails. FakePrivHelper makes this inconvenient. Extend FakePrivHelper so a test can execute arbitrary code for fuseMount calls.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14058663
fbshipit-source-id: 27ff32a75adde8aff952ac50b3f1977cf0b59e93
Summary:
Now that `hg debugedenimporthelper` has been released for
a little while, we can remove the bundled implementation of it from
the eden release.
However, we cannot remove the script itself as there are users
with long running edenfs instances that pre-date the knowledge
of `hg debugedenimporthelper`. So, as a compatibility shim,
this diff redirects `hg_import_helper.py` and has it exec the
command in mercurial.
For extra fun, our own integration tests rely on being able
to import `hg_import_helper.py` and override portions of it,
so we cannot remove its implementation from the codebase just
yet either.
So this diff:
* Introduces `proxy_import_helper.py` which execs `hg debugedenimporthelper`
* Installs `proxy_import_helper.py` as `hg_import_helper.py` in the rpm
* Leave `hg_import_helper.py` as-is in the tree for now.
Reviewed By: simpkins
Differential Revision: D13970332
fbshipit-source-id: 717dc86a880fbbbe4a7e801a8b748abd053c7f7c
Summary:
in D13970332 we want to allow using `hg.real` to run the
import helper that is embedded into mercurial, but we don't know
the full path, so we need to resolve it from the environment.
`folly::Subprocess` has `usePath` option, but it is not compatible
with also overriding the environment for a child process.
The easiest short term thing to do is to resolve the path ourselves
using this simple function call.
I've also added the path to `hg.real` that is used on our mac systems
to the PATH we pass down to `edenfs` when starting eden.
Longer term we want to better encapsulate how we resolve and start
the import helper, but for now this unblocks starting eden on the
mac.
Reviewed By: strager
Differential Revision: D14078612
fbshipit-source-id: bc6787bbe99cd87224b783d2fb9d3255b825b976
Summary:
Differ.h was including an overly-broad header when all it needed was
the type definitions. This makes building the inodes target with
mode/mac easier.
Reviewed By: simpkins, strager
Differential Revision: D14036477
fbshipit-source-id: 99f3a55d0523c179f6cf2aa4210119ae6f8e51f8
Summary:
I don't have RocksDB building in mode/mac yet, and the inode code
only depends on the store interface, not all of the specific
implementations.
Reviewed By: simpkins, strager
Differential Revision: D14036390
fbshipit-source-id: d0b8d5c4d4f2f38dbc9bc2476c24ac99398d37b7
Summary:
Some tests set up fuse_init_in manually then call FakeFuse::sendRequest. The specific fuse_init_in setup is benign and already exists in FakeFuse::sendInitRequest. Shorten these tests by making them call FakeFuse::sendInitRequest instead.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14059700
fbshipit-source-id: 210dcfdc4ae1c74e10c50dbc3f3044c3b908da0a
Summary:
Calls to ObjectStore::create and EdenMount::create are duplicated in the various TestMount::initialize overloads. Fix the duplication by moving the two calls into their own function.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14058972
fbshipit-source-id: b3e79756793292b59f09f53c05649a637c8af07e
Summary:
getInitialMode and getModeUnsafe were not inherently different, so
remove getModeUnsafe and write an accurate comment in Overlay's
serialization code about why the initial mode is saved and how we
could remove that logic in the future.
Reviewed By: strager
Differential Revision: D14007488
fbshipit-source-id: db42f45f00dcd213fabd9575360da1261931778b
Summary:
It's common for code to use InodeNumber without needing to include the
main FUSE headers or vice versa. Split them into two separate headers.
Reviewed By: strager
Differential Revision: D13979868
fbshipit-source-id: c5eeb6a3697bb538729a403434dc4f0f7408cda0
Summary:
Previously the default ignore path was `~/ignore`. This doesn't really seem
like a great path choice, and I suspect that no one is actually using an
ignore file at this location. This default was set in D8594297, but I it
looks like this was mostly accidental from just not separating out data for
the system ignore path (`/etc/eden/ignore`) vs the user ignore path.
Using `~/.edenignore` seems reasonable for now, and unlikely to conflict with
other paths. We used to use `~/.gitignore` as the default before D8906226,
but this did cause problems for a handful of users that treated their entire
home directory as a git repository, and had a `~/.gitignore` file that was
supposed to apply only to their home directory.
Reviewed By: wez
Differential Revision: D13984153
fbshipit-source-id: 887528372b9be789317933f7026dfcbde8cd4539
Summary:
Update FileChangeMonitor to stop logging messages about files not being
present. We normally use FileChangeMonitor to monitor the user's `edenrc`
configuration files and their personal and system-wide ignore files. It is
normal for these files to not exist in many cases, so there is no need to log
a warning message in this case.
Reviewed By: chadaustin
Differential Revision: D13981278
fbshipit-source-id: fa5f8d42980fd8683f55d3d51f5375a72f511dfe
Summary:
This diff enables building the eden watcher by linking in thrift and
its various dependencies.
To support building in-fbsource and in the github repo, a `maybe_shipit_dir`
function is used to setup a symlink to the `eden` and `fboss/common` dirs (this
mirrors the shipit configuration for this project: we cannot simply run shipit
because we have to build on mac and windows and shipit requires Hack, and that
does not support those platforms).
I tried to persuade cmake to let me build this without the use of a symlink but
found it too difficult to teach everything about the path mapping. The
symlinks aren't terrible, but are the reason why this diff also updates some
`.gitinore` files that are seemingly unrelated to this diff.
This diff changes a couple of build/link options: without them the end product
fails to link either due to implicit/unilateral enablement of UBSAN in some of
the deps, or because warning->error promotion is turned on.
This diff includes a copy of the `ThriftCppLibrary.cmake` file from the fboss
repo. This should get centralized and shipit'ed out into the places that
consume it. That can be done when someone gets around to doing the same for
the `FindGlog.cmake` file and doesn't need to hold up this diff.
Reviewed By: simpkins
Differential Revision: D13486486
fbshipit-source-id: 3bb5b011771b2a87618147ca019b4e50a8e0aaf2
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: yfeldblum
Differential Revision: D13908257
fbshipit-source-id: 8b5315b019290f1c60087ca5716c31ebbf1f1be5
Summary: FuseTypes.h defines the InodeNumber struct, but also has a comment which implies that InodeNumber is an alias of uint64_t. Delete this misleading commented-out definition of InodeNumber.
Reviewed By: chadaustin
Differential Revision: D13892907
fbshipit-source-id: 8e9710db29d8e0d708ee1785956f6953a0b2a49f
Summary:
When debugging Eden's behavior under a kernel that caches readdir
calls, I observed that directories were not returning the correct
entries after a checkout. The issue is that Eden calls INVAL_ENTRY
when new entries are added, but INVAL_ENTRY is a no-op and returns
ENOENT if the kernel doesn't know about that entry.
Reviewed By: strager
Differential Revision: D13864281
fbshipit-source-id: 56b3a67d0bc6f8ed1733acaf5e889414b753cbc7
Summary: `hgext` was moved to `edenscm.hgext`.
Reviewed By: simpkins
Differential Revision: D13884629
fbshipit-source-id: 4bf69720cccbd43665cd350ac8a3ba4d46b10b93
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: yfeldblum
Differential Revision: D13784772
fbshipit-source-id: 1d3ede848b7d31c7a197a21b4ff2b31e840040a5
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
Summary: using upgrade script to clear out all remaining version-set configs
Reviewed By: dark
Differential Revision: D13832474
fbshipit-source-id: 52c280cbd79b1410821ed829465b1c0907b50a86
Summary: PrivHelperServer::initLogging is declared in the .h file, but it's never defined. Remove the useless declaration.
Reviewed By: simpkins
Differential Revision: D13814416
fbshipit-source-id: 25cb47442a19947da08d13d9bed9b4631a1c9739
Summary: Currently, `runOnDestruction` aims to be thread-safe; new callbacks are added to the `onDestructionCallbacks_` list while the associated mutex is held. However, the caller may own the `LoopCallback` and wish to destroy/cancel it before the `EventBase` destructor runs, and this callback cancellation is not thread-safe, since unlinking does not happen under the lock protecting `onDestructionCallbacks_`. The primary motivation of this diff is to make on-destruction callback cancellation thread-safe; in particular, it is safe to cancel an on-destruction callback concurrently with `~EventBase()`.
Reviewed By: spalamarchuk
Differential Revision: D13440552
fbshipit-source-id: 65cee1e361d37647920baaad4490dd26b791315d
Summary:
Now that the deadlock in ProcessNameCache has been fixed, bring it
back.
Reviewed By: strager
Differential Revision: D13742965
fbshipit-source-id: f407105e06b9954766bdb48ef1303e2003c07284
Summary:
This fixes a deadlock where the kernel held the memory manager lock while satisfying a page fault resulting in a call into Eden, which tried to read /proc/pid/cmdline, acquiring the memory manager lock again.
The fix is to never read /proc/pid/cmdline while handling a request. That work is now done on a background thread.
Reviewed By: strager
Differential Revision: D13685318
fbshipit-source-id: c4e17de3358b668638db0c2dcfba8536d05e5b7c
Summary:
Previously edenfs always returned `ALIVE` in response to the fb303
`getStatus()` call. This fixes the code to return `STARTING` while Eden is
still starting and `STOPPING` while Eden is shutting down.
Reviewed By: strager
Differential Revision: D13515856
fbshipit-source-id: 91fe52e5f7f0a9e2c48ca1dd7663c99319afa8ad
Summary:
When graceful restart was first implemented we forgot to update the
lock file with the new pid, resulting in occasional unexpected output
from tools like eden doctor.
Reviewed By: simpkins
Differential Revision: D13744411
fbshipit-source-id: cdc758ed6ac1201fd2ff3e9d7805bb5ab6f83e8a
Summary:
This is something that is not needed on linux because
the kernel module is typically already loaded (at least on the systems
on which we run).
On macos, since fuse is not part of the kernel, libfuse has some code
that loads it when needed.
This diff performs the equivalent actions for eden.
Reviewed By: simpkins
Differential Revision: D13721489
fbshipit-source-id: 627bc90681141d0e7da3d5b5e06756a36839958c
Summary:
This requires our mercurial repo to be available during
the build; I symlink it in alongside `common` in the `oss` dir,
and point it up to `scm/hg`.
This has partial support for mononoke too, but will need to add
logic to getdeps to pull down the proxygen repo and build that.
Reviewed By: simpkins
Differential Revision: D13480146
fbshipit-source-id: 54874245015af83a259f56944d2e5f87615baee7
Summary:
Note that the concept of bind mounts doesn't exist on macos, so that
portion of the server just throws.
Reviewed By: simpkins
Differential Revision: D13480147
fbshipit-source-id: 92225188c0af42574d090004490f3926d393747b
Summary:
In our linux deployments it was relatively straightforward
to import the mercurial runtime from a python process running the
system python executable. Our macOS deployments are a lot more
complex because they do not use the system python and do not install
the mercurial python packages in the python path of the target
python executable.
It is simpler to move the import helper functional into a mercurial
command that we can invoke instead of our own helper program.
This diff moves the script to be a debug command and adjusts its
argument parsing to match the mercurial dispatcher requirements.
There are some stylistic mismatches between this code and the
rest of mercurial; I'm suggesting that we ignore those as the
medium term solution is that this command is replaced by eden
directly consuming the rust config parsing code and by native
rust code to perform the data fetching that we need.
Reviewed By: pkaush
Differential Revision: D13522225
fbshipit-source-id: 28d751c5de4228491924df4df88ab382cfbf146a
Summary:
This updates the logic in EdenServer to add the EdenMount to the mountPoints_
map as soon as it is created, so that we track mount points as they are
initializing.
I don't expect this change to have any major impact in functionality yet. In
a subsequent diff I also plan have EdenServer keep mount points in the
mountPoints_ map longer while they are shutting down. I expect that change to
matter a bit more, as that will allow us to do a better job reporting and
debugging when mount points are taking a non-trivial amount of time to become
unreferenced and fully shut down.
Reviewed By: strager
Differential Revision: D13503050
fbshipit-source-id: 2e0e8dfde64c6a005efd6dcf503ad7577f314356
Summary:
This is really a continuation of D13479516; the issue is that
the osxfuse kernel module is very eager to recycle `unique` request
id values, recycling them before our code has had a chance to update
internal state.
This diff re-keys the requests map so that we generate our own sequence
of identifiers to use as the key rather than the fuse protocol `unique`
value.
Because we cannot reliably track by `unique` value we also cannot
reliably implement interrupt support. We've never really tested
interrupt support, and it relies on functionality in folly futures
that hasn't really been tested or proven either, so I've removed
that functionality as part of this diff.
That allows simplifying some code in RequestData and FuseChannel;
we're now able to simply tack an `.ensure` on the end of the
future chain to ensure that we remove the entry from the map
once the future is resolved, successfully or otherwise.
Reviewed By: chadaustin
Differential Revision: D13679964
fbshipit-source-id: c1081a868c4061de2a725589ec1614959a8e9316
Summary:
Migrate the code from accessing optional Thrift fields directly to a safer
`optional_field_ref` API. See https://fburl.com/safe for more details.
The output of this codemod has been reviewed in D13259011.
To preserve semantics, each unchecked access is replaced with an explicit call
to `value_unchecked()`. If you are sure that accessing a field is safe (the
field is marked as set), you can later replace `value_unchecked()` with
`value()` or dereferencing (`operator *`):
```
ThriftStruct s = ...
- auto foo = s.foo_ref().value_unchecked();
+ auto foo = *s.foo_ref(); // will throw if s.foo is unset
```
Reviewed By: chadaustin
Differential Revision: D13684410
fbshipit-source-id: 919de4ddf89e7f0463f2614baba4bfbac1c8255c
Summary: the build breaks when making clean unless we declare this dep
Reviewed By: simpkins
Differential Revision: D13679633
fbshipit-source-id: f23a533eab9e37fdeab839e4f5e1b6b312ea10b0
Summary: This just makes the debug a little easier to follow.
Reviewed By: chadaustin
Differential Revision: D13680020
fbshipit-source-id: e4045822e56ba42a831ccb0ceaa9baaba5b79a10
Summary:
the osxfuse implementation includes some additional
fields in the `fuse_attr` struct. One of those fields is
`flags`. We were not initializing this field when converting
the stat data to fuse attributes which resulted in it holding
"random" data. In debug builds this seemed to usually end up
zeroed out but in release builds it was usually a bit pattern
that caused the kernel to respond with EPERM when attempting
to access the files. I didn't capture exactly what that
bit pattern was, just that initializing the struct to zeroes
reliably fixed up the EPERM issues in the Release build.
Reviewed By: chadaustin
Differential Revision: D13680004
fbshipit-source-id: 6b2ce6c10ef8f7db4a8a50bd3f2ddcfdddc3bb45
Summary: Small things I noticed while working on other stuff.
Reviewed By: strager
Differential Revision: D10055671
fbshipit-source-id: de8c3b04928567a821172e6fa7ee0e056958e1e7
Summary:
Address this error with clang:
```
In file included from /Users/wez/fbsource/fbcode/eden/oss/eden/fs/service/main.cpp:25:
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/privhelper/PrivHelper.h:22:1: warning: class 'Unit' was previously declared as a struct [-Wmismatched-tags]
class Unit;
^
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/Unit.h:36:8: note: previous use is here
struct Unit {
^
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/privhelper/PrivHelper.h:22:1: note: did you mean struct here?
class Unit;
^~~~~
struct
```
Reviewed By: strager
Differential Revision: D13602383
fbshipit-source-id: 6e69716498680660181ab441c3c007b074ec1d40
Summary:
If a file in the overlay is truncated unexpectedly, reads on it would
fail with EIO and a log message of "std::out_of_range: string
underflow" which wasn't helpful. Instead, use the same truncated file
detection as directories use.
Reviewed By: strager
Differential Revision: D13627565
fbshipit-source-id: 246f2659ba139e8f7adb7d556719e5ead9d84ebd
Summary:
Add `ensureStateTransition()` and `unconditionalStateTransition()` helper
methods to EdenMount, for performing state transitions where we want to verify
the current state and where we do not care about the current state,
respectively. Update most call sites that change the state to use these
methods.
Also change the code to use acq_rel semantics for updating the state,
rather than seq_cst, which is not necessary in this case. We do not care
about having a consistent visible ordering of change to the state variable
relative to other atomic variables.
Reviewed By: strager
Differential Revision: D13579333
fbshipit-source-id: 5fd62e740a7ea2777f79f722bbde7f5b65255cb6
Summary:
To determine the precise behavior of unmounting an Eden mount with
MNT_FORCE, I needed to see whether ENODEV was being returned from the
FUSE socket.
This helps us clarify the documentation in libfuse
https://github.com/libfuse/libfuse/issues/333
Reviewed By: strager
Differential Revision: D13630289
fbshipit-source-id: 90e6f0afc927c042a24cd6c82deac644c15ed066
Summary:
In the future, we will likely coalesce redundant fetches at the
BlobAccess/ObjectStore layer. To measure what we actually want,
populate a normal ObjectStore with a NullLocalStore and add counters
to FakeBackingStore.
Reviewed By: wez
Differential Revision: D13454331
fbshipit-source-id: 2fbf393d159f94e84c24ac53ccc207162fa754b7
Summary:
This changes prefetch so that it loads all of the direct children of
the tree. This improves `time ls -l bigdir` performance by 2x.
Reviewed By: wez
Differential Revision: D12888690
fbshipit-source-id: eb8c8274bd9c5b5edc94d7092a5feb492fad6d66
Summary:
We think that it shouldn't really be needed to perform
the prefetch call during lookup; for file inodes it doesn't buy
us much, and it should only really help for readdir.
This removes the prefetch call from lookup, instead prefetching
upon the first readdir() of a loaded TreeInode.
Reviewed By: simpkins
Differential Revision: D12896022
fbshipit-source-id: 0209eb64bd522daf5f7461dffccd1312d32a1554
Summary:
Update the `FuseTest.destroyWithInitRace()` test to succeed even if the future
returned by `startFuse()` completes with an exception.
Even though the test waits to see the `FUSE_INIT` response sent back to the
kernel, there is more initialization work performed by EdenMount after the
`FUSE_RESPONSE` is sent back. This initialization code can potentially fail.
At the moment the initialization code generally succeeds even if the
`EdenMount` has already transitioned to the `SHUT_DOWN` state. However, I
plan to change the `EdenMount` code soon to error out in this case. This
currently will cause this test to fail with its existing behavior.
Reviewed By: strager
Differential Revision: D13503048
fbshipit-source-id: 6ff147d8679559f0520f5e6091291c3a07bba3ed
Summary:
Update the `listMounts()` thrift API to also report the current mount point
state. This will allow us to do a better job of reporting mount points that
are in the process of initializing or shutting down.
This change splits the `MountInfo` thrift type into two distinct types for
the `listMounts()` vs `mount()` APIs. However this change should be
completely backwards compatible at the wire protocol level for older client
and server code.
Reviewed By: strager
Differential Revision: D13503049
fbshipit-source-id: 68e7ca708b956991c8fd93bbf8973d90650aced9
Summary:
Some people encounter system-wide hangs on their Linux machines. Debugging points to a deadlock related to EdenFS' process name lookup code. Disable the process name cache during FUSE dispatch to avoid the deadlock.
Effects:
* Hopefully, the deadlock no longer happens.
* 'eden top' will no longer report process names. (It should work otherwise, though.)
Reviewed By: simpkins
Differential Revision: D13540947
fbshipit-source-id: 595c36150a5f8ff1b8e7cd81d8f61ee1463d96eb
Summary:
To mitigate a deadlock, I want to make ProcessAccessLog not access /proc/. Allow this by making ProcessNameCache optional.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13540948
fbshipit-source-id: 4c5d68c972c04122de1d2414084debfec078dd4c
Summary:
Eden can handle parallel readdir and lookup so don't require the
kernel to acquire a mutex just to serialize the requests.
5c672ab3f0
Reviewed By: strager
Differential Revision: D13386133
fbshipit-source-id: aa935af941ff2901b07b63751c97052c295f7076
Summary:
The fuse opcodes are defined as an enum so we have to use
the relatively coarse and indirect apple vs linux preprocessor
checks in the maps for the opcode names.
The osxfuse implementation branched off from the 7.19 fuse
implementation, so add a light dusting of some preprocessor
checks around enabling the performance optimization features
we desire on Linux.
We also need to relax the compile time check for the min
fuse version; I've constrained this to be apple specific,
although I suppose it wouldn't hurt to make it more broadly
applicable.
Reviewed By: chadaustin
Differential Revision: D13480145
fbshipit-source-id: 010ac114e22ea942dfcebf1105cb1f01b766f297
Summary:
There's nothing nice about this; the full set of kernel headers are
not installed with the binary distribution, and since the kernel distribution
has to be signed to be loaded on osx, there's no benefit to us building it
for ourselves.
This diff adds a nop builder and tweaks the cmake to point into the osxfuse
repo.
The osxfuse repo aggregates a couple of related repos using the git
submodule feature, so trigger that from getdeps.py too.
Reviewed By: strager
Differential Revision: D13480148
fbshipit-source-id: 84e09a86f6a83f83ffd1e3fe113dc7b15b3ea208
Summary:
a couple of parts of the rust, datapack and mononoke code
didn't have appropriate definitions.
Allow `useDatapackGetBlob_` to be default initialized per its
definition in the header file rather than replicating the relatively
complex ifdef logic in the implementation.
Reviewed By: strager
Differential Revision: D13475710
fbshipit-source-id: d2955c9b22f1186f5897aa8bdbd9046b8f6b5f7a
Summary:
handle this best-effort by setting this bit on each fd after
allocating them.
Reviewed By: strager
Differential Revision: D13475712
fbshipit-source-id: 46be80f025b21967f75822f983bc327c5e2d20af
Summary:
ScopedHTTPServer allows callers to specify a simple request handler function
instead of having to define their own handler factory. This updates the
MononokeBackingStore tests to change the `Handler` class in to a simpler
functor object, and delete the `HandlerFactory` class.
Reviewed By: wez
Differential Revision: D13476411
fbshipit-source-id: 0ede232ff9570c95e877b689272ea8eb26d97d83
Summary:
Change the code to use ScopedHTTPServer instead of its own custom logic that
starts a server and runs all of the test logic inside the `onSuccess()`
callback.
Doing all of the test logic inside `onSuccess()` is generally bad, as most of
the tests perform blocking `get()` calls on `folly::Future` objects. Blocking
work generally should not be done from inside EventBase threads.
When built with ASAN these tests occasionally crash with heap-use-after-free
errors during the HTTP server shutdown. I didn't track down the exact bad
behavior that was causing this, but attempting to stop the server from inside
the `onSuccess()` callback does seem a little fishy. Hopefully simply
switching to `ScopedHTTPServer` will fix this issue.
Reviewed By: wez
Differential Revision: D13476413
fbshipit-source-id: ab92cc16a5bf99a5e7b52529012a03786495c319
Summary:
Switch to ScopedEventBaseThread for the standalone EventBase thread create in
the unit tests. This way the test does not need its own custom logic for
starting and joining this thread. `ScopedEventBaseThread` also ensures that
the thread is actually running and ready to process events before its
constructor returns.
Reviewed By: wez
Differential Revision: D13476412
fbshipit-source-id: 260a40d93050e2e9b92ef9efd1549633679f36f7
Summary:
Update the Hash constructors that accept a `ByteRange` and a `StringPiece` to
be `constexpr` so that all Hash constructors are now `constexpr`.
This probably shouldn't really make a big difference in practice. I added
this since I wanted to define some static `Hash` constants in some tests, and
didn't want to worry about SIOF issues.
Reviewed By: chadaustin
Differential Revision: D13475781
fbshipit-source-id: fc1ce91c998f1badadbd6becd525458c25dd30de
Summary:
Avoid this compilation error:
```
[ 23%] Building CXX object eden/fs/fuse/CMakeFiles/eden_fuse.dir/Dispatcher.cpp.o
In file included from /Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.cpp:10:
In file included from /Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.h:12:
In file included from /Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/Portability.h:19:
In file included from /Library/Developer/CommandLineTools/usr/include/c++/v1/cstddef:110:
/Library/Developer/CommandLineTools/usr/include/c++/v1/type_traits:3141:38: error: incomplete type 'facebook::eden::DirList' used in type trait expression
: public integral_constant<bool, __is_constructible(_Tp, _Args...)>
^
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/futures/Future.h:146:36: note: in instantiation of template class 'std::__1::is_constructible<facebook::eden::DirList>' requested here
typename std::enable_if<std::is_constructible<T, Args&&...>::value, int>::
^
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/futures/Future.h:148:12: note: while substituting prior template arguments into non-type template parameter [with Args = <>]
explicit FutureBase(in_place_t, Args&&... args);
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
/Users/wez/fbsource/fbcode/eden/oss/external/install/include/folly/futures/Future.h:115:7: note: while substituting deduced template arguments into function template 'FutureBase' [with Args = <>, $1 = (no value)]
class FutureBase {
^
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.cpp:175:13: note: in instantiation of template class 'folly::Future<facebook::eden::DirList>' requested here
Dispatcher::readdir(InodeNumber, DirList&&, off_t, uint64_t) {
^
/Users/wez/fbsource/fbcode/eden/oss/eden/fs/fuse/Dispatcher.h:35:7: note: forward declaration of 'facebook::eden::DirList'
class DirList;
^
1 error generated.
```
Reviewed By: chadaustin
Differential Revision: D13475720
fbshipit-source-id: 2da1692010a82b73cbed71d996993cf5fc13af0e
Summary:
It appears as though osxfuse is a bit eager to re-use `header->unique`
values, as this DCHECK to ensure that we erased our request at the end of a
request is frequently triggered. It's not for 100% of requests, but I did
notice that the `header->unique` values tend to be in the range 1-6.
We don't check for collisions when we emplace requests in the dispatching
logic, and this particular check happens after we've acked the kernel, so it
doesn't seem like a safe thing to check and abort on really, so let's remove
it.
Reviewed By: chadaustin
Differential Revision: D13479516
fbshipit-source-id: a01e6c3e47b78b651f65fc3f5138168c71968030
Summary:
When run inside the systemd service (fb-edenfs@.service), edenfs' logs are written to `/var/log/messages` (on Facebook dev servers). This is undesirable, since those logs have a bunch of noise.
Make systemd-managed edenfs log to `~/local/.eden/logs/edenfs.log` instead, matching the behavior of custom-managed edenfs.
---
I considered using systemd's StandardOutput= and StandardError= directives [1], but they have limitations:
* **StandardOutput=file:%f/logs/edenfs.log**: When the `logs` directory is missing, systemd does not create it. In this case, systemd fails when it opens the log file, so systemd refuses to start the service.
* **StandardOutput=journal** [2]: journald and journalctl are broken for user services. Logging to journald only works with persistent journal storage [3][4], but Facebook uses volatile journal storage.
* **StandardOutput=syslog** [5]: rsyslog seems designed for system administrators, not users. I didn't investigate much, but I suspect it's impossible to make rsyslog write to a user-controlled path such as `~/local/.eden/logs/edenfs.log`.
* **LogsDirectory=%f/logs and StandardOutput=file:%L/edenfs.log** [6][7]: LogsDirectory= does exactly what we need, except it only supports paths relative to `/var/log` or `~/.config/log/`. `LogsDirectory=%f/logs` does not work, and systemd will ignore such a directive.
* **StandardOutput=file:%f/logs/edenfs.log and a `mkdir` service**: If we create a service which just creates the `logs` directory, and make fb-edenfs@.service depend upon that service, systemd can successfully open the log file [8]. In theory, using StandardOutput= would cause errors like "could not set resource limits" to be logged to `edenfs.log`. In practice, systemd does not respect the service's logging configuration when reporting such errors [9]. Therefore, this solution is no better than the manual redirect.
[1] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardOutput=
[2] https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html#
[3] https://www.freedesktop.org/software/systemd/man/journald.conf.html#SplitMode=
[4] https://lists.freedesktop.org/archives/systemd-devel/2016-October/037554.html
[5] https://www.rsyslog.com/
[6] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=
[7] https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers
[8]
```name=fb-edenfs-logs@.service
[Service]
Environment=EDENFS_CONFIG_DIR=%f
ExecStart=/bin/sh -c ' \
set -e; \
set -u; \
\
/bin/mkdir -p -- "$${EDENFS_CONFIG_DIR}/logs""; \
'
```
```name=fb-edenfs@.service
[Unit]
After=fb-edenfs-logs@%i.service
Requires=fb-edenfs-logs@%i.service
```
[9] fd0ec39d38/src/basic/log.c (L560-L639)
Reviewed By: simpkins
Differential Revision: D13422459
fbshipit-source-id: 57c575a6f377812caa2a79168778576c6ccff33e
Summary:
ported forward from D4209167, add a couple of helpers
to access these fields on mac and linux, centralizing/minimizing ifdefs.
Simplify some of the logic in FileChangeMonitor.
Reviewed By: chadaustin
Differential Revision: D13475717
fbshipit-source-id: d7b39999808bc41a6dc17a87189501cb34e68b30
Summary:
Thankfully, we can simply remove it; it is really just a performance
optimization that we can enable for linux.
Reviewed By: simpkins
Differential Revision: D13475719
fbshipit-source-id: 09b60dcf995c2c5390b91aa316c62ca1b4d3f944
Summary:
It is only 16 bits wide on this system, so we don't need
to borrow bits and there may not be enough bits to borrow from
anyway.
Reviewed By: simpkins
Differential Revision: D13475714
fbshipit-source-id: 1d342c89a3037abd744d97fef21ad421b5e60356
Summary: only include and use it on linux
Reviewed By: simpkins
Differential Revision: D13475715
fbshipit-source-id: 6b0b9da1b32088e01cbb932f9b3ed62532dfe00f
Summary: only add the selinux deps if we found selinux
Reviewed By: simpkins
Differential Revision: D13475711
fbshipit-source-id: c3375282b61881317f9a6c4c8e321ce717d1f9ab
Summary: overlooked because there is no CI exercising this today
Reviewed By: simpkins
Differential Revision: D13475721
fbshipit-source-id: 3e8fe280ab73d249da374129b37d32cd7e17f472
Summary:
on macOS the LHS is `unsigned long` and the RHS is `unsigned long long`.
Introduce a cast for consistency.
Reviewed By: chadaustin
Differential Revision: D13470036
fbshipit-source-id: f726507013e4ed9f123ced474299bb2d6818732f
Summary: Even in C++17 mode, this isn't available, so work without it
Reviewed By: chadaustin
Differential Revision: D13470034
fbshipit-source-id: c917cc011aaabc2cfcf79e801bb6870482302fd8
Summary: this is required for the build on macos
Reviewed By: chadaustin
Differential Revision: D13470035
fbshipit-source-id: 066cb5b2ea86ffddb9c8cf6f7ae50da90b62a5bc
Summary:
I want to use fake_edenfs to test logging via EdenFS' systemd service. Make fake_edenfs and the real edenfs use similar logic to determine the log file path.
This diff should not change behavior for the real edenfs.
Reviewed By: simpkins
Differential Revision: D13424470
fbshipit-source-id: d0c2e035fdb5884dbd2d9704c7e0244d35e052f2
Summary:
Eden no longer tracks any state in file handles, and has no plans to in the future.
Therefore, remove all related code.
Reviewed By: strager
Differential Revision: D13354307
fbshipit-source-id: 341d081f64c6c8fb2b4b1b5a5ff42f2cc7d38039
Summary:
Now that all file access in Eden is stateless, we no longer need to handle open() or release().
If the kernel advertises FUSE_NO_OPEN_SUPPORT, return ENOSYS from open().
Reviewed By: simpkins
Differential Revision: D13325759
fbshipit-source-id: 38486848f27ffeb005f74407888e94d891496f98
Summary:
D13325746 changed `EdenDispatcher::open()` to no longer return a file handle.
However the code in `FuseChannel::fuseOpen()` throws an exception if the
dispatcher does not create a file handle. This breaks most file operations in
Eden.
The Dispatcher check is removed in D13354307, but that hasn't landed yet.
That change probably just needed to be part of D13325746.
Reviewed By: chadaustin
Differential Revision: D13445570
fbshipit-source-id: 70d639142057740766bcbe02a0df50b14f7c9937
Summary:
Remove some more of the double-checking code left over from debugging
T24406969. We fixed the root cause of that task and haven't seen any errors
reported by this code. The higher log levels used for reporting information
about empty files just causes extra noise in the logs.
Reviewed By: strager
Differential Revision: D13403531
fbshipit-source-id: bcb7b29a1d687ffcefd6ecf586130b96f87b4820
Summary:
Previously, a file handle must have been held for the entirety of a write operation. That is no
longer true. Stop looking up file handles on write.
Reviewed By: strager
Differential Revision: D13325662
fbshipit-source-id: 9ae31b467d17d633c388917d18098e6e5a620b89
Summary:
Now that FileInode read and write operations are stateless via BlobAccess and OverlayFileAccess,
EdenFileHandle no longer provides any value. Remove it. This also fixes eden's shutdown timeout
when a file handle is open and paves the way for FUSE_NO_OPEN_SUPPORT.
Reviewed By: strager
Differential Revision: D13325137
fbshipit-source-id: 71ed47a7c997f5035b4394ccb311f94332ecd8c2
Summary:
Have FileInode use OverlayFileAccess instead of using the Overlay directly.
This allows IO on materialized files to be stateless and pave the way for
eliminating EdenFileHandle. It also paves the way for performance improvements
such as nicer SHA-1 caching.
Reviewed By: strager
Differential Revision: D13325079
fbshipit-source-id: fb27d48b5dc9196dc6e36557596f601194a56aa9
Summary:
Move the overlay IO logic out of FileInode into a centralized OverlayFileAccess class. It keeps the last 100 overlay file handles open.
FileInode's changes are coming in D13325079.
Reviewed By: strager
Differential Revision: D13324995
fbshipit-source-id: 04fb3fe50114b0f19b78acd17a9684c92f8e8072
Summary:
Add a fuzz test that randomly modifies a directory during a series of
readdir calls and verifies each unmodified entry is returned once.
Reviewed By: simpkins
Differential Revision: D13371162
fbshipit-source-id: 252b03ab0288e82b56a33c347955de129e61ae42
Summary:
If a blob was partially read, evicted from cache, and then read again,
the readByteRanges coverage set was not being cleared. Always clear it
in startLoadingData.
Reviewed By: strager
Differential Revision: D13405267
fbshipit-source-id: 6f60e6e80662fd470fe4ddbc833fc8efd8850686
Summary:
There was a bug in BlobCache where, if you had an interest handle to a
blob, but that blob was evicted anyway and then something else caused
it to be reloaded, dropping your interest handle would cause the blob
to be incorrectly evicted since the reference counts were no longer
compatible. Add a version to cache items and only decrement the
reference count on an item if the interest handle and item agree.
Reviewed By: strager
Differential Revision: D13405144
fbshipit-source-id: aee052bf777e7225551c3ae2b8b69a99f4f77691
Summary:
When you run 'eden start' without systemd integration, edenfs writes startup logs to the terminal to let users know that stuff is happening:
```
$ eden start
Starting edenfs (dev build), pid 2792025
Opening local RocksDB store...
Opened RocksDB store in 0.95 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2792025)
Logs available at /data/users/strager/.eden-dev/logs/edenfs.log
```
These startup logs are also used by various tests (especially 'eden restart's tests).
Make the same thing happen when running 'eden start' with systemd integration, improving the user experience and making some tests work:
```
$ EDEN_EXPERIMENTAL_SYSTEMD=1 \
./buck-out/gen/eden/cli/eden.par start \
--daemon-binary "${PWD}/buck-out/gen/eden/fs/service/edenfs"
Starting edenfs (dev build), pid 2800760
Opening local RocksDB store...
Opened RocksDB store in 0.693 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2800760)
```
Reviewed By: wez
Differential Revision: D13241979
fbshipit-source-id: de79b714e42b690fdab7c21d9add46bc2da35328
Summary: Several tests for 'eden start', 'eden stop', and 'eden status' need to pass command-line arguments to fake_edenfs. With systemd support enabled, make 'eden start' forward daemon arguments to fake_edenfs, making these tests pass.
Reviewed By: wez
Differential Revision: D13249891
fbshipit-source-id: 9008a361fce7a5629535cc9d245b86073ee70826
Summary:
getSHA1 is only handling std::system_error. If another kind of exception is thrown, it's never caught and edenfs crashes.
Calling EdenMount::getInode with a path such as "./hello" will cause a std::domain_error to be thrown. Since std::domain_error is not derived from std::system_error, `getSHA1ForPathDefensively(["./hello"])` crashes edenfs.
Fix the crash by forwarding all exceptions over Thrift, not just std::system_error-s.
Reviewed By: simpkins
Differential Revision: D13386450
fbshipit-source-id: 06262dad30a5508ed482c9e8979b61aa9643280a
Summary:
As I'm working on ripping out file handle support, I wanted to ensure
that we had a path towards a correct and stateless readdir()
implementation. Stateful file handles require extra care during
graceful restart, and it's nice that we can avoid them. In fact, this
work paves the way towards a possible FUSE_NO_OPENDIR_SUPPORT feature.
This diff fixes concurrent rm -rf in the same directory.
Reviewed By: simpkins
Differential Revision: D13370898
fbshipit-source-id: eba650e673a7cb9559e04ba28417980f6d0c65cb
Summary:
Have `eden stats` print the size of the blob cache if the running
edenfs has information about it.
Reviewed By: strager
Differential Revision: D13349220
fbshipit-source-id: 9f59f4399f2d4283aa80bcb54ba73c51d555d502
Summary:
edenfs's privhelper process needs the CAP_SYS_ADMIN capability [1] in order to manage mount points. Give it this capability by invoking edenfs as the `root` user using `sudo`.
Known issues:
* `sudo` must be passwordless in order for fb-edenfs@.service to start.
* systemd can't kill all of fb-edenfs@.service's processes, so `systemctl stop` is unreliable for example.
[1] https://manpage.me/index.cgi?q=capabilities&apropos=0&sektion=0&manpath=CentOS+7.1&arch=default&format=html
Reviewed By: chadaustin
Differential Revision: D13113450
fbshipit-source-id: 01b89521cab371b5017fab6fbd38d55eea599c46
Summary:
I want to add some non-trivial systemd-specific logic to fb-edenfs@.service's ExecStart command. Refactor the existing command to use sh, giving us access to `if` and other useful utilities.
Also, the shell language is likely more familiar (e.g. `${foo}` and `$foo` behave the same in sh, but behave differently in systemd), so using sh should lead to fewer headaches.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13165377
fbshipit-source-id: 2e500abf75c6f79a8b174a848f7eeb1cfaebb96c
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.
Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.
'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).
Reviewed By: simpkins
Differential Revision: D10849390
fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
Summary:
Drop interest in cached blobs at various points in the FileInode
lifecycle.
Reviewed By: strager
Differential Revision: D12991762
fbshipit-source-id: 19fd94938c96485160d547ecbd259ffeb39b2341
Summary:
Frequently, non-eden-owner-owned processes show up as `err:13` in
the `eden top` output which is a bit of a PITA. This diff pulls the data
from `/proc/PID/cmdline` instead of `/proc/PID/exe`; the former is world
readable always while the latter is restricted to process owner readable.
Reviewed By: chadaustin
Differential Revision: D13342789
fbshipit-source-id: d4e395b318107e873189a1e2039329015c4c38f8
Summary: Add a CoverageSet type that tracks non-overlapping intervals so FileInode will be able to tell when the entire blob has been read by FUSE.
Reviewed By: strager
Differential Revision: D12947146
fbshipit-source-id: fbcfd9c19b09d4a7b5364671dcdbd39b53e6d186
Summary:
Write tests for readdir's semantics (we really do want to return . and
..) and simplify the logic. It's still quadratic in large directories,
but there aren't any allocations anymore.
Reviewed By: strager
Differential Revision: D13287764
fbshipit-source-id: 5e0d4b86eb16dbd7a16cdeb324e4b43363512e25
Summary: This diff also includes build script changes to EdenDispatcher, FsChannel and EdenMount.
Reviewed By: strager
Differential Revision: D13091786
fbshipit-source-id: cecc8d849fcb9ebc8fa718e1011ef8931bebc279
Summary:
Stop holding a reference count to the TreeInode while a directory
handle is open. This allows eden to shut down while a directory handle
is open.
Reviewed By: strager
Differential Revision: D13287701
fbshipit-source-id: a24f32a1ac40b6c19bc5864aa5f5785f3016361b
Summary:
Send readdir requests to TreeInode. This may not sound like a good
idea: the FUSE documentation suggests that stateful directory handles
are required to implement correct readdir semantics under concurrent
deletes and renames. However, the 63-bit offset value is treated as a
cookie that is passed from one readdir call into the next, and 63 bits
should be sufficient to implement readdir concurrent with
rename/unlink. So move readdir's implementation into TreeInode in
preparation for the complete removal of TreeInodeDirHandle.
Reviewed By: strager
Differential Revision: D13287664
fbshipit-source-id: c0d615675edd9b83353534468a69b89068bba923
Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary:
Send fsyncdir straight through the inode rather than going through
DirHandle. This is the better design anyway, since the DirHandle does
not receive directory-mutating requests like mkdir.
Reviewed By: strager
Differential Revision: D13287610
fbshipit-source-id: 154fa32a3877c89a204a2d10b4e2b637410d9486
Summary: No need for String to import fbvector.
Reviewed By: yfeldblum
Differential Revision: D13007715
fbshipit-source-id: c61639a04273f14dd3daf230ee6ed9ade93d058e
Summary:
FUSE_NO_OPEN_SUPPORT is better than ATOMIC_O_TRUNC for Eden's use
case. Remove the code that pretended we might support ATOMIC_O_TRUNC
again someday.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13163382
fbshipit-source-id: 948d701571a8d2977da3d2532fdc9538c5011636
Summary:
It's not clear that this code is a win and either way it will be a
no-op when FUSE_NO_OPEN_SUPPORT is enabled so just remove the prefetch
in open().
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D13162205
fbshipit-source-id: a3161c0d042e13bd092fc9589e851be78552fa7a
Summary:
FileInode no longer has a strong reference to a blob. Instead, all accesses go through the blob cache. This diff changes the caching behavior for blobs.
The previous behavior was:
When a file's contents are needed in any way, the blob is loaded and referenced by the inode. When the number of open file handles goes to zero, the blob is dropped. The blob is also dropped when the inode is unloaded. Future inode loads or open requests, in that situation, require the blob to be reloaded from the LocalStore.
The new behavior is:
When a file's contents are needed, the blob is loaded and stored into the BlobCache, evicting any if necessary. Future blob requests are satisfied from the BlobCache, pushing it to the back of the eviction queue. When the inode is materialized or unloaded, the blob will be evicted from cache if no other blob has interest in it.
(Note: this ignores all push blocking failures!)
Reviewed By: strager
Differential Revision: D12813912
fbshipit-source-id: 20d20807d2e4a6c37cddab38562fdb7456316aac
Summary:
A later diff needed a constant for the SHA-1 of an empty buffer. While
I'm at it, I made Hash a little bit nicer to use.
Reviewed By: strager
Differential Revision: D13224195
fbshipit-source-id: b2fb1437be042215b5b398a8c7fc9fc5dd115e9e
Summary:
Now that the Overlay no longer serializes timestamps, remove all of
the special-case migration logic.
Reviewed By: strager
Differential Revision: D13144764
fbshipit-source-id: 713a4bfcde9003a8d5a28837cb530b05a9017c22
Summary:
The linter tripped on D12813838 even though the warning was unrelated
to my changes.
Reviewed By: simpkins
Differential Revision: D13167184
fbshipit-source-id: 555691f00d113ed2bff9488b61392cc92f4395e3
Summary:
Eden has used the InodeMetadataTable as the authoritative source for
timestamp metadata for over six months. I think we can safely assume
that anyone at this point who has old inodes in the overlay that don't
have corresponding entries in the inode metadata table are fine with
those timestamps being reset to the last checkout time.
Reviewed By: strager
Differential Revision: D13144735
fbshipit-source-id: 06a9a8835ea83c98fb6a165e4c8d5c3c6b28ad84
Summary:
Eden has used the InodeMetadataTable as the primary source of
timestamp data for more than six months. Stop writing timestamps into
the overlay, since they will never be used.
Reviewed By: strager
Differential Revision: D13144696
fbshipit-source-id: e36423036228e89dd2a986e6bacfa74553c17a92
Summary: Instantiate a blob cache in the EdenServer with configurable settings.
Reviewed By: strager
Differential Revision: D12813880
fbshipit-source-id: 8ccc89826f04aca78964230374dea48abf05e05e
Summary: In support of making Eden's file access stateless, add a facade that connects the BlobCache and ObjectStore, allowing FileInode to fetch blobs, minimizing reloads and memory usage.
Reviewed By: strager
Differential Revision: D10850143
fbshipit-source-id: 4307f7c1143aecad1284ea3cadf3e4efca9a3925
Summary: Change the Windows pipe read and write functions to read/write in a loop. Plus changed the functions prototype to match the POSIX version.
Reviewed By: strager
Differential Revision: D13091785
fbshipit-source-id: 375b22bd9e62d371a78d410f42068945b966a743
Summary:
Backing store works with eden strings(UTF8 + Unix path separator). The path strings we receive on Windows from FS and cli are Windows paths
(Win path separator and/or UTF16). Adding the functions to convert one to another.
Reviewed By: strager
Differential Revision: D13091788
fbshipit-source-id: f7fc8a79e360e964cf4619dfa540b57f1f18d283
Summary:
The new blob cache wants to know, given a request, whether the blob is
expected to be needed or not. The answer, in general, is yes if the
request came from Thrift and no if it came from FUSE, because the kernel
will cache the result of the request in its own page and dentry caches.
Propagate this information through FileInode.
Reviewed By: strager
Differential Revision: D12813838
fbshipit-source-id: 7a359686149cd4daff41630c94085b680c448c4f
Summary: Add a BlobCache with a maximum cache size and a minimum entry count and interest-based eviction.
Reviewed By: strager
Differential Revision: D12972062
fbshipit-source-id: 1958f7f500c051a5bc0b39b5b89a6f0fc1774b0f
Summary: Create a platform independent function to compare the stats and use that to check if the file contents have changed.
Reviewed By: strager
Differential Revision: D13046269
fbshipit-source-id: c4f5bc88cec3df5cb6555d13cea2f23fd4eeb7ce
Summary: Add type annotations to all functions in eden/fs/service/client.py
Reviewed By: strager
Differential Revision: D13051091
fbshipit-source-id: 23b93008352664336ad155a7f5cc281bd5529702
Summary:
Now that we have a standardized StartingGate for benchmarks, use it
everywhere we used to use folly::Baton.
(Note: this ignores all push blocking failures!)
Differential Revision: D13012244
fbshipit-source-id: 5841ab74cfa408e87d021fe5591557e79e677e5c
Summary:
As we start to build out both FUSE and Thrift benchmarks, we'll want a
standard library. Introduce a benchharness and have both the thrift
sha-1 and parallel_open_close benchmarks use it.
(Note: this ignores all push blocking failures!)
Differential Revision: D12969306
fbshipit-source-id: 89c8bbcc37d53560decffb9281af4aba20345787
Summary: Update the CLI code to make the pyre type checker happy.
Reviewed By: wez
Differential Revision: D13035413
fbshipit-source-id: d201f2e65667e0ce1bf4a73fbb05878e8711ad16
Summary:
chadaustin noticed this as part of fixing up the ESTALE
handling. The issue is that we were using `inodeMap->lookupInode` and
assuming that it will always return one of our magical inodes. This
isn't guaranteed so it is better to resolve the inode by name from
the root, so that's what this diff does.
Reviewed By: chadaustin
Differential Revision: D12970034
fbshipit-source-id: 8207660cbc71577b276cb092d1ef19e1076b4946
Summary:
yfeldblum were talking about whether this code might make sense in
folly. That led to polishing it a bit more. The hot path is only six
instructions now. It's not any faster in a tight generateUniqueID loop
but uses only one thread_local slot and a bit less global storage.
Reviewed By: strager
Differential Revision: D12927275
fbshipit-source-id: 94a5872c61dfe9dd441f1f34fab65f93c12997d8
Summary:
In a later diff, I needed generateUniqueID to be
noexcept. folly::ThreadLocal does not guarantee that (and it allocates
the first time a thread calls get()), so use C++ thread_local
instead. Bonus: it's about half a nanosecond faster now.
Reviewed By: strager
Differential Revision: D12914625
fbshipit-source-id: 9ddbe65d0ba1d317907f821c03dea5a207a73a68
Summary:
Update the HgUI object used by hg_import_helper to always return true from the
`plain()` function, regardless of whether HGPLAIN is set in the environment or
not.
This should help ensure that this script is never affected by user-defined
configuration settings. This should also help ensure that mercurial won't try
to print progress bars or other strange things.
Reviewed By: quark-zju
Differential Revision: D13008639
fbshipit-source-id: afe581958470c4c4b89825a259c460ece4e20fe7
Summary:
It is desirable to be able to reference the same variable
multiple times in the RHS of a config setting. This diff makes that
possible.
Reviewed By: strager
Differential Revision: D12906500
fbshipit-source-id: 4277f12105d0a0fb3dca880d3dad6b0238acedc0
Summary:
With the mononoke service being configured with proxygen
in front of the servers we need to ensure that we're correctly
setting up our http request if we want them not to fail with a
400 bad request error.
This diff allows setting mononoke.hostname to a DNS resolvable
name rather than an IP address and passing that hostname through
so that we can set it in the HTTP request.
Reviewed By: strager
Differential Revision: D12906498
fbshipit-source-id: b5aaabfd6f2f4c48d45128eaad8406e648477f75
Summary:
Update all of the C++ unit tests that create temporary files and directories
to use the new `facebook::eden::makeTempFile()` and
`facebook::eden::makeTempDir()` functions.
Note that this likely changes the behavior of some code paths in meaningful
ways: `/dev/shm` normally does not support `getxattr()`, and Eden's overlay
code attempts to store the SHA-1 for materialized files as using extended
attributes. This means that the tests will now typically hit the fallback
code path and will not store SHA-1 data in the overlay.
Reviewed By: chadaustin, strager
Differential Revision: D12971162
fbshipit-source-id: 6cc5eba2e04be7e9a13a30e90883feadfb78f9ce
Summary:
Add new helper files for creating temporary files and directories.
The main advantage of these functions is that they prefer creating files in
`/dev/shm` by default instead of `/tmp`. Some of the eden unit tests are
fairly I/O intensive (particularly the checkout tests, which create and
destroy many test mounts). This can be quite slow on hosts where `/tmp` is a
a spinning hard disk. Putting the temporary files in a ramdisk greatly speeds
up the tests in this case.
These test functions also default to using the prefix `eden_test.` for the
temporary file names.
This does not yet change any of the test code to use these functions. I will
do that in a subsequent diff.
Reviewed By: chadaustin, strager
Differential Revision: D12971161
fbshipit-source-id: 3f74be7a467e8080185d4d97d114288b4528755a
Summary:
This function behaves similarly to the python `os.path.expanduser`
function, except that it is restricted to expanding only information for the
current user.
Use this new function to process the hgcache path returned via the
rust config parsing code.
In the longer term I want to centralize and add accessors for
the rust config and move this stuff into EdenConfig, but for
now this is the one place that does this and it is OK to
process it this way here.
Reviewed By: strager
Differential Revision: D12988902
fbshipit-source-id: 96b10640359c3b8c0adac1e14cd42dd592023c3d
Summary:
This makes it possible to change configuration options
for the LocalStore while the server is running.
As you'll see in the next diff, our current layering makes using
the config a bit more awkward, but at least this diff doesn't
look gross :-p
This diff doesn't introduce any new functionality or configuration.
Reviewed By: strager
Differential Revision: D12949577
fbshipit-source-id: cf897ba676b9359f92865170faa42ff17329b85f
Summary:
Sometimes users accidentally run `edenfs start` or `edenfs restart` instead of
`eden start` or `eden restart`. This adds a new `--edenfs` flag to the
`edenfs` binary, and asks users if they meant to run `eden` instead if they do
not pass in this flag.
This used to be less of a problem since `edenfs` required users to also
explicitly specify several other configuration flags (like `--edenDir`).
However `edenfs` can not automatically figure out these settings, so these
flags are no longer required. Therefore `edenfs` would still try to start
normally when invoked with `edenfs restart`, since it did not require these
flags and it did not complain about unhandled command line arguments.
Reviewed By: wez
Differential Revision: D12927803
fbshipit-source-id: dbf7ce2449c391ca218652439eb68ff43c2ebd46
Summary: folly/Subprocess is not compatible with Windows and has broken the Windows build. It's not used so removing it.
Reviewed By: wez
Differential Revision: D12967451
fbshipit-source-id: 54d33bf6fe2ec3ede9d68eccd99e53c5eb6ed53d
Summary:
In a later diff, I needed to make generateUniqueID() noexcept. This
made me think that C++'s thread_local might be a little faster than
folly::ThreadLocal. So I added this benchmark.
Reviewed By: yfeldblum
Differential Revision: D12914249
fbshipit-source-id: 2c4acfff2162f66d13f456439d91df2ecb4167e3
Summary:
Thanks to some bpf tracing by strager, we traced the ESTALE response to
`d_splice_alias` and noted this comment above the implementation in the kernel:
> If a non-IS_ROOT directory is found, the filesystem is corrupt, and
> we should error out: directories can't have multiple aliases.
Well, our magic `.eden` directory is a directory with aliases and we were
seeing the error trigger on that dir. So, this diff replaces hardlinking
directories into each tree with a hardlink to a symlink in each tree!
At mount time we create `.eden/this-dir` as a symlink to `/abs/path/to/mount/.eden`
so that `readlink("/abs/path/to/mount/sub/dir/.eden/socket")` still
resolves as it did prior to this diff.
Reviewed By: strager
Differential Revision: D12954819
fbshipit-source-id: 7f3b1b53f2bd5b9c51e64055fc34110657a19110
Summary:
Sandcastle has several cases where we chown the entire
repository which performs terribly on Eden. As a workaround we have a
command to do this in eden without loading all the files.
Reviewed By: chadaustin
Differential Revision: D12857956
fbshipit-source-id: 36cebcc710fbcf4e1eb265df901513cf50a227b9
Summary:
I've included a missing `.via` for the mononoke tree import path. It's hard to
measure the difference here, but it is more technically correct than letting
that second stage run in an event base thread. desired.
Reviewed By: chadaustin
Differential Revision: D12896023
fbshipit-source-id: 0af0f7f94c3274a3e4b570fed0af4d4d3c783bc3
Summary:
FileInode::prefetch was entirely redundant - it queried for metadata
upon inode lookup after getattr() was called already (which requires
the blob metadata to be loaded).
Reviewed By: wez
Differential Revision: D12896473
fbshipit-source-id: 9ba5104a43860e1f22b88726b9e3e977d0b50e89
Summary:
Eden used to load materialized entries at startup. It no longer does
and this code is dead.
Reviewed By: wez
Differential Revision: D12896210
fbshipit-source-id: 398363724a661f87208cf05313e61755a451edb7
Summary:
With this the eden cli can dump tracepoints and translate
them to various formats or perform any processing
Reviewed By: chadaustin
Differential Revision: D10384072
fbshipit-source-id: 8b38e7f6b551a2bd98b3e748ba1cceafeceeec8c
Summary:
This uses the tracing infrastructure to create blocks around
the getSHA1 invocation as well as the getOrLoadChild (called for each
element of the path-walk) method.
Reviewed By: chadaustin
Differential Revision: D10384074
fbshipit-source-id: a06fbe38e8d3f35fcb248e6bd724e5572724d27d
Summary:
In order to get better insights into where time is spent on
various requests, some tracing infrastructure is helpful. This will
record tracepoints into a per-thread ring-buffer which can then be
dumped out-of-band.
The trace format is quite simple, a trace is composed of blocks with
begin and end points and each block may have a parent block. We can
then translate this format to many other formats as we desire.
Reviewed By: chadaustin, strager
Differential Revision: D10384071
fbshipit-source-id: f9e0f11521c68b96ab40d517c7a83cf89375b101
Summary:
This diff implements getBlob on top of the mercurial rust
datapack code. It adds a C++ binding on top of the rust code to
make it easier to use and hooks it up in the hg backing store.
Need to figure this out for our opensource and windows builds:
* Need to teach them how to build and link the rust code
* need to add a windows version of the methods that accept paths;
this is just a matter of adding a WCHAR version of the functions.
Reviewed By: strager
Differential Revision: D10433450
fbshipit-source-id: 45ce34fb9c383ea6018a0ca858581e0fe11ef3b5
Summary:
Nothing populates these fields yet today, but will in a later diff.
If present in the json data for an entry, populate them in our TreeEntry.
When processing trees from mononoke, check for these fields being set
and emit them into the current writeBatch so that we set the BlobMetadata.
Reviewed By: chadaustin
Differential Revision: D12814793
fbshipit-source-id: cd19d3f553c22462adc58c024a90cfeb5b8da224
Summary:
We'll start populating these in the mononoke client.
They are currently unused.
Reviewed By: chadaustin
Differential Revision: D12814791
fbshipit-source-id: f5407de5cdb9f1f3ad6ee2befed50e2a7562ec97
Summary:
I want to reuse this outside the LocalStore implementation
in a later diff. This has no functional changes.
Reviewed By: chadaustin
Differential Revision: D12814792
fbshipit-source-id: 2cc34b449a93afb2e71bd49ed6587009ebf3e419
Summary:
While this doesn't deliver any gains immediately, it is desirable
because the shortest path to getting trees with pre-populated size and
content sha1 information is via the mononoke api server fetch.
If we use the local trees we will populate the basic metadata slightly
faster but will then pay more to fetch and hash the data.
The calculus is a little more complex really, as having the contents of
a dir locally in a local datapack is probably faster, but we don't have
a good way to know if that is the case right now.
Reviewed By: strager
Differential Revision: D12813490
fbshipit-source-id: 545b90bf20940dab193bef8120712dd9227033e1
Summary:
This partially wires up handling mononoke related config
changes if they are made after starting the server. We don't currently
have logic to enable mononoke after starting the server, but can disable/enable
it if it was configured at process start.
Reviewed By: strager
Differential Revision: D12813346
fbshipit-source-id: ee29e91fe714a65a3b61c4e23929b67d6b7cb521
Summary:
To facilitate testing against a test instance rather than selecting
from a tier, add a config option to return the host override.
The semantics are that a hostname string will override the tier name setting.
An empty host name string is treated as missing and will not replace the
tier name.
Reviewed By: chadaustin
Differential Revision: D12813050
fbshipit-source-id: 67773d772d78d3f7882825ce54f4313c97ab0a33
Summary:
Remove the handful of (but growing) parameters that
are threaded down to the HgBackingStore in favor of the ReloadableConfig
object and use that to configure mononoke support.
This doesn't notice changes to the config on the fly today, but
does simplify the amount of stuff we need to pass around.
Reviewed By: chadaustin
Differential Revision: D12813053
fbshipit-source-id: b8309328fab4980889624257cfd387c8c1335001
Summary:
This allows detecting changes to the edenrc configuration
and adjusting how we read data accordingly.
To avoid a huge gnarly diff with a lot of dependencies, this diff
introduces a little interface class that defines the method to obtain
a possibly reloaded EdenConfig instance.
This diff shouldn't change any functionality.
Reviewed By: chadaustin
Differential Revision: D12813051
fbshipit-source-id: 35390dc28671ba46a03d098c4f280e2f567dbdc9
Summary: This option was added upstream and to our internal 4.16 kernel series
Reviewed By: chadaustin
Differential Revision: D12823476
fbshipit-source-id: 54d241b77eff92f4083c0f1d4c98b47a6a098e3f
Summary:
Checkout.modifyFile is the long pole when running the Eden unit
tests. Splitting it into three runs it across multiple threads. It
could be split further if needed.
Reviewed By: strager
Differential Revision: D12813684
fbshipit-source-id: 49e847d46bc54f16fcb2b273ec81e92eaf756dbb
Summary: If an error happens when opening the log file, edenfs should stop immediately and report the error. Write a test to verify that this happens with DaemonStartupLogger.
Reviewed By: chadaustin
Differential Revision: D10436926
fbshipit-source-id: 3f33ed276a668b7ecda8176a55b114b5878304e3
Summary:
When running edenfs in daemonize mode (i.e. when --foreground is not specified), the daemon process inherits stdin from the parent and never closes it. This has two consequences:
* The daemon process holds a reference to stdin even after the parent exits. This causes tests to hang if they wait for EPIPE on stdin after waiting for the parent process to exit. (I discovered this issue when adding tests for 'eden start' in D10434379.)
* edenfs could potentially read from the terminal after 'eden start' has returned. I think this behavior is surprising and could potentially cause problems.
Promptly close stdin after forking the daemon process, since nothing inside the daemon needs stdin.
Reviewed By: simpkins
Differential Revision: D10442346
fbshipit-source-id: f3d287e8b48e121807f845b73a1a550e63efae25
Summary:
getLogPath in eden/fs/service/main.cpp ensures that a non-empty log path is given to DaemonStartupLogger::daemonize. An empty log path is not possible. DaemonStartupLogger behaves differently with an empty log path, but that behavior has caused problems (see D10439299).
Make DaemonStartupLogger require a (non-empty) log path, simplifying its implementation.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10227597
fbshipit-source-id: 37931528add51ab3e6e1b941478000297cb8b693
Summary:
There may be a metadata header in the contents and we currently
do not have any code to process this, so we want to disable this.
Add a constructor boolean to disable it and default to off while we
work up a proper solution to this.
Reviewed By: chadaustin
Differential Revision: D10523393
fbshipit-source-id: ec96b63b6897841ee5a6b848834b5a0ada145223
Summary:
The daemonizeIfRequested is implemented in the main edenfs process as well as in fake_edenfs. Consolidate the almost-identical implementations.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439300
fbshipit-source-id: 6253668896de0f8393b92f4aa5ee8a8a2fcbfb31
Summary:
D9029272 (github commit fae2056037) accidentally broke the CMake-based
build of Eden when SELinux is available: it defined `PRIVHELPER_LIBS` but
later used `PRIVHELPER_LIBRARIES` (LIBS vs LIBRARIES).
This fixes that error by simply removing this intermediate variable. We
can just use SELINUX_INCLUDE_DIR and SELINUX_LIBRARIES directly, as they
will simply be empty if SELinux is not available.
Reviewed By: quark-zju
Differential Revision: D10503905
fbshipit-source-id: 1c12bb1cad0351e4e0a77d0c7e8a83086209efee
Summary:
Our CMake-based build had gotten somewhat out of date. This fixes several
issues that were preventing it from building:
- wangle now depends on fizz, which in turn depends on libsodium
- fbthrift now depends on yarpl for the streaming functionality
- Eden now depends on libsqlite3
- The eden/fs/inode library now depends on eden/fs/config
Reviewed By: strager
Differential Revision: D10436531
fbshipit-source-id: f41065da2edb909e8eb94e03d0e62ae19ad3a84f
Summary:
Add a very cheap unique ID allocator designed for extremely
low-latency uses like D10384071.
Differential Revision: D10501712
fbshipit-source-id: 15a0b7a2d344c0f6100082850dd967c585f3e2da
Summary:
Because Mercurial blob IDs change without the contents changing, and
because files get unloaded upon checkout, rebasing across a large
distance in history can result in status fetching a lot of
metadata. Keep a smallish LRU cache for SHA-1 and size by blob ID.
Reviewed By: strager
Differential Revision: D10419965
fbshipit-source-id: 81499573814775471913db05f924767c3bab300e
Summary:
Now that the import bug has been fixed for some time, it's likely few
people have cached empty files. And if they do, `eden gc; eden
restart` is a fine workaround.
On the other hand, reimporting empty files gums up the importer, and
I've seen several people recently complaining about performance that
could be partially attributed to the fact that their Eden needed to
verify empty files. simpkins has talked about adding a bit to the
cached blob to determine whether it needs reimporting or not, but it's
probably going to take a while for anyone to implement that, and this
reverification logic is hurting people today.
Reviewed By: strager
Differential Revision: D10456519
fbshipit-source-id: 657bc377ee16ce93494075bde4388aed59dceecf
Summary:
Noticed that the filenames in the logging in the error
case were sometimes garbled; capture a copy of the filename
rather than a reference to it.
Reviewed By: bolinfest
Differential Revision: D10473798
fbshipit-source-id: ee218a2ad9d6edfad6ba7246107c152e79a9efd8
Summary: When debugging a failure in DaemonStartupLoggerTest.closePipeWhileStillRunning, I was confused because the pipe which is closed in the test (readPipe/writePipe) is unrelated to the DaemonStartupLogger. Eventually I learned that the pipe referred to by the test's name is a pipe internal to DaemonStartupLogger. Rename the test to express the high-level scenario instead of mentioning pipes (which are an implementation detail).
Reviewed By: simpkins
Differential Revision: D10247254
fbshipit-source-id: f5f5b358adad0b656db27ea75fec65fe0331317b
Summary:
This uses the existing unionstore to attempt to load blobs directly from
the hgcache. This is a pretty simplistic implementation:
1. We unconditionally add the filepack path into the list of treepack paths
in the import helper startup. This is not 100% good, but should be good
enough in practice. This does mean that we'll only successfully find
a filepack if we have enabled treemanifests.
2. The load is performed in context of the calling thread, which means that
we may block. Not sure if that is critical at this point.
In the longer run, we want to migrate this to the newer rust code; this is
just early stage proof of concept.
Reviewed By: strager
Differential Revision: D10277985
fbshipit-source-id: 9b953bbf3983bdb777635c6ff0b78efc101a17cc
Summary:
FileInode::stat is called very often, even by FUSE operations
as common as lookup(). stat() requires the size, which we don't in
general know until the blob's been imported. That said, if the blob
has been imported once, we don't actually need to decode the entire
blob out of RocksDB - we can much more cheaply read the cached blob
metadata to get the size.
Differential Revision: D10441161
fbshipit-source-id: aafc52b54aca9ba30248420fbc4f2ccf1ec0bed8
Summary:
Eden supports reading the SHA-1 of a file via getxattr. Unfortunately,
it returned ENOSYS if you called getxattr on a directory inode. This
caused FUSE to fail all future getxattr requests with EOPNOTSUPP.
In addition to fixing that, this diff makes our xattr handling a
little more consistent across inodes:
- setxattr() always fails with ENOSYS
- removexattr() always fails with ENOSYS
- listxattr() is always handled by the corresponding inode class
- getxattr() is always handled by the corresponding inode class
Differential Revision: D10437723
fbshipit-source-id: a1ea1e92d3412abb15e91057becea3160a17f1e2
