Summary:
Some of our tests create Mercurial repositories with default settings
which means they don't support treemanifest. Flip treemanifest on for
them.
Reviewed By: wez
Differential Revision: D15302347
fbshipit-source-id: 34f98b88976b5de1bf2ad07c5272e217e3fe3b0a
Summary:
Add a mechanism to open the RocksDbLocalStore in read-only mode. This is
primarily helpful to allow the `eden_store_util` process to examine a DB
without making many modifications, including automatic compactions that might
happen otherwise.
Note that this doesn't appear to be entirely read-only from a file contents
perspective: RocksDB does still rewrite the MANIFEST and OPTIONS files when
opening the DB in read-only mode, and does rotate the logs and write a new LOG
file.
Reviewed By: wez
Differential Revision: D15350216
fbshipit-source-id: 797498f9575a05334b741322654843d6545ca35c
Summary: Use `folly::prettyPrint()` when printing the sizes.
Reviewed By: chadaustin
Differential Revision: D15398827
fbshipit-source-id: 17b1bee7b78f63c5b766aad8a65f1a8f776bf6b4
Summary:
Increase the default send timeout from 100ms to 250ms.
Increse the timeout in the `UnixSocket.sendDataAndFiles` test to 10s.
We have seen some failures in the `UnixSocket.sendDataAndFiles()` test on
our continuous build infrastructure when testing debug builds on Mac OS.
These failed due to hitting the send timeout on the 4MB and 32MB sends. Bump
up the timeout for this test to 10s. (I initially tried 250ms, and this was
still not sufficient.)
I increased the default send timeout to 250ms just to help ensure that we
don't hit it unexpectedly during real production code when running on Mac OS.
Reviewed By: wez
Differential Revision: D15401154
fbshipit-source-id: c8be68a77231e90d37c9b5b4f349379bdf2b15ed
Summary: getEntryPtr() does a case sensitive lookup because of which few Ovrsource builds were failing. Ovrsource code is including header files with the wrong case.
Reviewed By: strager
Differential Revision: D15344850
fbshipit-source-id: 3d5d658a49cdafc07dc9a18a2f3d2073306e8f40
Summary: Small parts of our code have diverged from our clang-format rules.
Reviewed By: strager
Differential Revision: D15380260
fbshipit-source-id: f668ac22d6c0c5f2468549f2a94dd1c9bb22ce3d
Summary: 2 seconds is apparently too short for some of the big directory and files we have in our repository.
Reviewed By: wez
Differential Revision: D15387381
fbshipit-source-id: 6ebf850c5fbad8d60c2e8c9f8cb19b1c28297d47
Summary:
The `LocalStore` constructor was updated to accept a `ReloadableConfig`
argument in D12949577, but this was never used anywhere. Remove it for now to
help simplify the code. If we do want to add it back in the future I think we
should make it required, rather than allowing a null config to be specified.
Reviewed By: wez
Differential Revision: D15350217
fbshipit-source-id: 8571b48dff8c8d079ba6b25821dd0b1d77ffe791
Summary:
Implement the `show_sizes` function which reports the approximate size on disk
used by each column family.
The initial skeleton for this command was accidentally included in D15162813.
Reviewed By: wez
Differential Revision: D15307391
fbshipit-source-id: 03efee939d65500e17b48047552dec1dbb7aff7e
Summary:
Add a `name` field to KeySpaceRecord. This allows the `RocksDbLocalStore` and
`SqliteLocalStore` code to use the name from the `kKeySpaceRecords` data
structure rather than keeping their own array of names that must be maintained
in sync with the `LocalStore` code.
Reviewed By: wez
Differential Revision: D15307390
fbshipit-source-id: b53bca7d53d630f9527a810f145380288c54198e
Summary:
Update the KeySpace enum values to start at 0 instead of 1. This simplifies
the code to avoid having to skip over 0 in a few places.
This also makes the `kKeySpaceRecords` array slightly less confusing. Unlike
the `columns` array used by `RocksDbLocalStore` and the `tableNames` array
used by `SqliteLocalStore`, the `kKeySpaceRecords` array was not previously
indexed by the `KeySpace` enum values.
Reviewed By: wez, strager
Differential Revision: D15307393
fbshipit-source-id: ae8392d02396b4dc3c18e9ee94b198fcbb9b1a34
Summary:
Move the KeySpace record definitions out of LocalStore.cpp and into a header
file. This will make it possible to access the key space information from
other files in the future.
I would like to have access to this data directly in the various LocalStore
implementations, as well as in eden_store_util.cpp. Currently both
RocksDbLocalStore.cpp and SqliteLocalStore.cpp have their own copies of key
space information that should really be shared.
Reviewed By: chadaustin, strager
Differential Revision: D15307392
fbshipit-source-id: ad7cdc705d3dc9834592fe0f14b38d82f91a3b77
Summary:
Now Eden on Windows could be build using
python.exe .\opensource\fbcode_builder\getdeps.py build eden
Reviewed By: strager
Differential Revision: D15251103
fbshipit-source-id: 8ac1097b234bb27ce478101bfbc9591251e2e359
Summary: MSVC doesn't like the ImporterOptions declared as class and defined as struct.
Reviewed By: wez
Differential Revision: D15251102
fbshipit-source-id: e7725df5db8ee39a20a1940a8d045369800aa4c3
Summary: tar is in /usr/bin on macOS.
Reviewed By: simpkins
Differential Revision: D15052024
fbshipit-source-id: 5388b2d53f876717310d0f9726b74e8a283d1a89
Summary:
This solves the client certificate mystery. We creates a new `EdenConfig` object
whenever there is a change detected and these two fields are lost during the
copying. Causing the variable substitution not working as expected.
Reviewed By: wez
Differential Revision: D15221180
fbshipit-source-id: 3fba36c8f261cb3dbe25e2e363d65661240be8cf
Summary: Change `ConfigSource` from a plain `enum` to an `enum class`.
Reviewed By: chadaustin
Differential Revision: D15162811
fbshipit-source-id: 680ec1e785de1a54b623d99477914afa5d0931ca
Summary:
Move the `ConfigSetting` template and related classes from `EdenConfig.h` to a
new `ConfigSetting.h` header file.
Reviewed By: chadaustin
Differential Revision: D15162815
fbshipit-source-id: 34fba0b8b0e8e50714f622bae642d20d77050673
Summary:
Update the `isValidAbsolutePath()` implementation to use `kDirSeparator`, and
update the TODO comment.
Reviewed By: chadaustin
Differential Revision: D15162812
fbshipit-source-id: f52f0e85c5eef342723947c6efaa52992da1660b
Summary:
Move the `FieldConverter` logic out of EdenConfig.h/cpp and into its own
module.
Reviewed By: chadaustin
Differential Revision: D15162816
fbshipit-source-id: f30d81329139e5827b81a609e83baddf71b6af5e
Summary:
Rename the `ClientConfig` class to `CheckoutConfig`, to help further progress
on updating our terminology from "client" to "checkout".
Reviewed By: chadaustin
Differential Revision: D15162814
fbshipit-source-id: 3575958f1161d5842c6f0ee9e2d2d20ab20b7372
Summary:
Move the `ReloadableConfig` implementation out of `ServerState` and into the
`ReloadableConfig` class itself. This also updates `ServerState` to simply
contain a `ReloadableConfig` member variable rather than inheriting from
`ReloadableConfig`.
This makes it easier to build other utilities that need a `ReloadableConfig`
object but do not have a full `ServerState` object, such as the
`eden_store_util` class and the `zero_blob` helper tool used by the
integration tests.
Reviewed By: chadaustin
Differential Revision: D15162813
fbshipit-source-id: 1ebc0ac6c475f4aa1090dd933ebce16bc222674c
Summary:
This timeout appears to be too short for the last step of the test that sends
32MB of data in multiple send calls of no more than 1000 bytes each. On
stress test runs this timeout has occasionally been firing and causing the
test to fail.
Reviewed By: chadaustin
Differential Revision: D15221306
fbshipit-source-id: d1f7ed2d58b47512a8f31f8324c61372394a6709
Summary:
beholdunittests
This enables some plumbing for running some of the
tests using the gtest/gmock machinery in cmake.
Part of this diff is removing the FindGMock.cmake file from the
eden repo; we now pull this in from the shared cmake library
that is populated by shipit.
Reviewed By: simpkins
Differential Revision: D14993344
fbshipit-source-id: 51caf9518c7f3a083a3b90cda10324c3a8170359
Summary:
This argument is unnecessary since D14222323, which should be deployed
everywhere by now. Passing in the extra repo argument forces hg to open the
repository object twice, which is unnecessary.
We already set the current working directory to the repository when invoking
the import helper, so hg will find the repository correctly from that.
Reviewed By: pkaush
Differential Revision: D15179667
fbshipit-source-id: 838cbee91748c41c713731187608a9823b971a53
Summary:
Update HgImporter to use `hg debugedenimporthelper` instead of the separate
`hg_import_helper.py` script by default on all platforms now. The
`debugedenimporthelper` subcommand has existed in our deployed versions of hg
for a while now.
Reviewed By: pkaush
Differential Revision: D15179668
fbshipit-source-id: 2fb8c4c9f92aed54c84899d6643f746baac73327
Summary:
This diff tries to improve the Thrift Mononoke backing store implementation by:
* Re-creating thrift client for every request since thrift has builtin connection pooling
* Using Eden's CPU thread pool for processing data fetched from remote
* Calling thrift methods within correct executor
Reviewed By: wez
Differential Revision: D15170818
fbshipit-source-id: c8be70755a851f63fb62e4d4f36833703283565e
Summary: We intend to break support for flatmanifest hg, so require that treemanifest is available in the build.
Reviewed By: wez
Differential Revision: D15057150
fbshipit-source-id: 449399cfb9d018f3b722598091eead1bd5d7c77d
Summary: Flatmanifest is on its way out. Remove support for falling back to it if a tree import fails.
Reviewed By: pkaush
Differential Revision: D15056459
fbshipit-source-id: a4df820322ee354d77f50a0ec92e9705d0f152ec
Summary:
Interestingly, the request doesn't fail, but does leave
our stack garbage intact. Let's be sure to zero it out to make
it more obvious what is happening.
While we're in here, let's make sure that we get the same
results from both ends of the socket pair.
Reviewed By: simpkins
Differential Revision: D14994593
fbshipit-source-id: 9aec957dfcd80d88c3d8fbce6bf45480502ea812
Summary: this fixes linking the tests when building with cmake in D14993344
Reviewed By: chadaustin
Differential Revision: D15167223
fbshipit-source-id: 1dda3beed3b6ff3788f3992783c58b4b92f697f2
Summary:
D15043209 missed fixing up the name of the variable
containing the list of sources.
The result is that we compile the `main` from `eden_store_util`
in to the tests and break gtest discovery in D14993344
Reviewed By: chadaustin
Differential Revision: D15167219
fbshipit-source-id: 95e9dc8ba74965f41bfebed1c1776605d8c9443a
Summary:
UnixSocketTest would block forever on macos if the message
size was larger than approx 1kb. It turns out that MSG_DONTWAIT
isn't documented in the `sendmsg` man pages and apparently has
no effect at all. Instead, the socket must be placed in non-blocking
mode. This appears to be the case for the client side of the sockets
but I need to follow up for the server side.
Reviewed By: simpkins, strager
Differential Revision: D14994588
fbshipit-source-id: 2a0a1c26a7b45ece82f8f79c15fb7756844bc86f
Summary:
on macOS `/var/tmp` is a symlink to `/private/var/tmp`
which means that our tests for path canonicalization end up
trying to compare the non-canonical `/var/tmp` prefix of the
temporary paths generated by the tests with `/private/var/tmp`
and failing.
Reviewed By: simpkins
Differential Revision: D14994584
fbshipit-source-id: e69addca8e315855d3ca87feb576ca7c54567c31
Summary:
D14677339 added tracing to all HgImporter requests. Each trace loads a thread-local variable through EdenStats::getStatsForCurrentThread(). Each of these loads has a small but non-trivial cost.
Because each instance of HgImporter is used only on one thread, each getStatsForCurrentThread() call returns the same EdenThreadStats object. Avoid the thread-local-variable lookups by caching the result of getStatsForCurrentThread() in HgImporter.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14794284
fbshipit-source-id: d1609a1720d44c680dc0ebaa2536779def2a6f37
Summary:
If EdenFS' globFiles API is given two patterns, and one pattern is a prefix of the other, EdenFS effectively ignores the longer pattern. Given the patterns `project/src/*` and `project/src/*/*`, when EdenFS encounters `project/src/dir/`, it generates a result (because `project/src/dir/` matches the first pattern) but does not recurse into its children.
The problem caused by an incorrect understanding of the GlobNode::isLeaf_ flag. isLeaf_ means "this GlobNode should generate results", but GlobNode::evaluateImpl understands it to mean "this GlobNode should generate results, and this GlobNode has no children". Given the patterns `project/src/*` and `project/src/*/*`, the GlobNode representing `project/src/*` has isLeaf_=true but also has children.
Fix the bug by not using isLeaf_ for determining whether to recurse, and instead relying on the presence of child GlobNode-s.
Reviewed By: chadaustin
Differential Revision: D15078089
fbshipit-source-id: 1c480d11361f89193b35965266e6873c57181113
Summary:
If globbing returns a directory, that directory doesn't need to be loaded. GlobNode::evaluateImpl is careful to only load directories/trees if necessary. When fixing a bug in GlobNode::evaluateImpl, I accidentally broke this optimization. No test failed, though.
Ensure this optimization works and doesn't regress by writing a simple test.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D15090386
fbshipit-source-id: fe932e4a7ad517ff4c4523d58668f05fe8b0cafb
Summary:
In GlobNode::evaluateImpl, the logic for recursing into child GlobNode-s is duplicated token-for-token for hasSpecials_ and !hasSpecials_. While fixing a bug in GlobNode::evaluateImpl, I was forced to update both copies, which is prone to mistakes.
Fix the duplication by factoring the code into a function.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D15090688
fbshipit-source-id: 65eec62214074139cb8af75026758b631eb89bb2
Summary:
gdb and lldb have a hard time printing recursive structures involving std::vector and std::unique_ptr. This makes debugging GlobNode a pain.
Add a function to print a GlobNode recursively to stderr. This makes it easier to debug GlobNode using printf.
Reviewed By: simpkins
Differential Revision: D15078111
fbshipit-source-id: 3ba211026c7c894676435ade278ae859e1b5d1f6
Summary:
Sandcastle's [1] code coverage builds compile EdenFS and Hg with Clang's -fprofile-generate flag (or a related flag). By default, running a program with with that flag creates a `default.profraw` file in the current directory. This causes some tests to fail, such as HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import, because `default.profraw` is unintentionally committed to the test repo:
```
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from HgBackingStoreTest
[ RUN ] HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import
eden/fs/store/hg/test/HgBackingStoreTest.cpp:64: Failure
Value of: tree1->getEntryNames()
Expected: has 2 elements where
element #0 is equal to foo,
element #1 is equal to src
Actual: { default.profraw, foo, src }, which has 3 elements
[ FAILED ] HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import (1563 ms)
[----------] 1 test from HgBackingStoreTest (1563 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1563 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import
```
When Sandcastle runs the tests, it sets the `LLVM_PROFILE_FILE` environment variable, which configures the path to `default.profraw`. EdenFS' HgRepo class is stripping this variable when invoking hg, so hg uses the default path and not the configured path.
Make HgRepo pass `LLVM_PROFILE_FILE` through to hg. This fixes some of EdenFS' tests when run by Sandcastle for code coverage purposes. (This does *not* fix running the tests manually (without setting `LLVM_PROFILE_FILE`), though.)
[1] Sandcastle is Facebook's continuous integration system.
Reviewed By: simpkins
Differential Revision: D15104832
fbshipit-source-id: 3929b9b0ab0904f2552ace7d8e4e4f4a4221192c
Summary:
Migrate the code from accessing optional Thrift fields directly to a safer
`optional_field_ref` API. See https://fburl.com/safe for more details.
The output of this codemod has been reviewed in D13259011.
To preserve semantics, each unchecked access is replaced with an explicit call
to `value_unchecked()`. If you are sure that accessing a field is safe (the
field is marked as set), you can later replace `value_unchecked()` with
`value()` or dereferencing (`operator *`):
```
ThriftStruct s = ...
- auto foo = s.foo_ref().value_unchecked();
+ auto foo = *s.foo_ref(); // will throw if s.foo is unset
```
Reviewed By: fanzeyi
Differential Revision: D15078040
fbshipit-source-id: 8ef270e8c687535e9d4aa2208beff4517e121814
Summary:
Add a command to force a call to RepairDB() on the local store.
This is similar to using `ldb repair`, but invokes `RepairDB()` with the same
set of column family options as normally used by edenfs.
Reviewed By: chadaustin
Differential Revision: D15043210
fbshipit-source-id: 2c4c0e2d3410a50cb1e523611f569f1701604ae6
Summary:
Change eden_store_util to take a subcommand argument, and define 3 separate
subcommands: "gc", "clear", and "compact"
The "gc" was the old default behavior. The "clear" and "compact" subcommands
simply clear cached entries and compact the storage, respectively.
This also adds a `--keySpace` argument. If this argument is specified the
command will process only this single key space, instead of all key spaces.
Reviewed By: chadaustin
Differential Revision: D15043207
fbshipit-source-id: 7e76bc89862d9647edde631ed3d82c9afda09f41
Summary:
This moves some logic from the RocksHandles class up to RocksDbLocalStore.
The main thing moved here is the logic to automatically try and repair the DB
if opening fails. This will make it easier in a subsequent diff to make the
repair logic a bit smarter and more aware of our column family semantics.
This keeps RocksHandles a pretty dumb wrapper around the RocksDB object and
column family handles, whose only purpose is to manage destroying these two
things in the correct order.
Reviewed By: chadaustin
Differential Revision: D15043208
fbshipit-source-id: ee2d5619ac7781a892e1ba151712eee9e3ebfb14
Summary:
Move this utility from eden/fs/service to eden/fs/store, and rename it to
`eden_store_util`.
I originally put it in the `service` subdirectory since it depends on library
code from `service/`, and most of the other code in `store/` shouldn't depend
on code in `service/`. However logically it makes sense to be in the `store/`
directory.
I plan to update in the future to be able to perform some additionally utility
operations besides just garbage collection, so rename it to the more generic
`eden_store_util` name.
Reviewed By: chadaustin
Differential Revision: D15043209
fbshipit-source-id: e6c4a874ddd05bd1576ea2e8a6c6156ccdf4628f
Summary:
On macOS, the atime, mtime, and ctime fields are named st_atimespec,
etc. Add some portability functions so the inode tests compile on
macOS.
Reviewed By: simpkins
Differential Revision: D15048758
fbshipit-source-id: 6f85b8eb7f7da2f9bd21c9034296cde96b68f740
Summary: Add some includes that are required on macOS.
Reviewed By: simpkins
Differential Revision: D15048681
fbshipit-source-id: d8642dc0977dc8cd8aadc83fc750400a8f917b7a
Summary:
Add a standalone program that can be used to garbage collect the local store
data when edenfs is not currently running.
Reviewed By: wez
Differential Revision: D14910392
fbshipit-source-id: 1cf8f959e5967cfbb797752653cc72223c87134f
Summary:
Let's catch more Xcode and upstream regressions by enabling more
builds and tests on macOS.
Reviewed By: simpkins
Differential Revision: D15003839
fbshipit-source-id: 257086d578f28198c7718906e1ec06395a55896e
Summary:
Build the store tests on mode/mac. RocksDB doesn't build yet (and we
will likely move away from it anyway) so split it into its own target.
Reviewed By: simpkins
Differential Revision: D15002840
fbshipit-source-id: fa0fea1d52df06526032a6fd585e2da7273910ef
Summary:
The hg store implementation depends on Rust code that doesn't build on
mode/mac yet, but TestMount's dependency on hg was false anyway.
Reviewed By: simpkins
Differential Revision: D15001077
fbshipit-source-id: 840fdf565093ad7a6eb95e001a74ae95eecbf1db
Summary: These allow more tests to compile
Reviewed By: chadaustin
Differential Revision: D14994582
fbshipit-source-id: 6b2a0b276fda64c7f27e28ea9e6d548aaaa1db7c
Summary: This diff add four metrics to the stats we are collecting in Eden. Both time cost on calling `get_tree` and `get_blob` from hg and mononoke backing store.
Reviewed By: chadaustin
Differential Revision: D14780349
fbshipit-source-id: f0e902c7f7523fac2ae576a67f0c85555064e26c
Summary:
The Finder shows the volume name instead of the directory
name when it renders the directory containing a mount, and the default
name of `OSXFuse Volume X` makes it hard to distinguish between multiple
mounts in the same folder.
This changes the volume name to be the basename of the mounted path so
that the finder renders things more reasonably.
Reviewed By: chadaustin
Differential Revision: D14969807
fbshipit-source-id: de9d8469360712828598fed124ed02ee5d6dd44a
Summary:
Unfortunately, some node modules packages include `.DS_Store` files
and since we disable apple double, the kext refuses to allow them to be
unpacked in the repo. Ideally they wouldn't be in the repo, but we need to
unblock that workflow.
Reviewed By: chadaustin
Differential Revision: D14969808
fbshipit-source-id: f9262294a223fd5cc7929a2b324f2401e1fed083
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: Orvid
Differential Revision: D14975808
fbshipit-source-id: b7790bf0e8965c9ffb3e63901bcdaf108480d4ca
Summary:
Add a helper function to recursively remove a directory from the local
filesystem, given an `AbsolutePath`.
Reviewed By: wez
Differential Revision: D14954737
fbshipit-source-id: a3f6b60e610c18eabea2a7c507661a43502a2554
Summary:
Add a new class to manage access to the .eden state directory, and move some
logic for managing this director from the `EdenServer` class and into this new
class.
This is largely just a first step. In the future I think we should move
additional functionality into this class. For instance, logic to determine
the correct local store type and logic to load the checkout map from
config.json. In the future it probably also make sense to move this class
from the `service` library into a slightly lower-level library (perhaps
`config` or `store`).
Reviewed By: chadaustin
Differential Revision: D14889617
fbshipit-source-id: 15cb7daa9b97500124152d016bfdd686d3039771
Summary:
Move some of the argument parsing and config setup code out into a new
EdenInit.h header file. This makes it possible to re-use this logic for other
standalone utilities that want to be able to find the Eden state directory and
config information.
For now I have updated the `fake_edenfs` helper tool used by the integration
tests to use this. This may also be useful for writing standalone tools that
can perform garbage collection of the LocalStore or checking of the overlay
state.
Reviewed By: chadaustin
Differential Revision: D14889616
fbshipit-source-id: b0b193a42cb2f52177d0c44592426b42e27242aa
Summary:
This effectively reverts D14452214, which caused Eden to write an `id` entry
to each RocksDB column family and then flush the column family each time
edenfs started.
There was relatively little benefit to this in practice. It only matters in
cases where the RocksDB column families never had enough data written to them
to get flushed automatically and then a repair is required.
On the other hand it does have some material downsides: it flushing the column
families can be fairly expensive, and can require a substantial amount of free
disk space. This flush caused some users to not be able to start up edenfs
when they did not have enough free disk space.
Reviewed By: chadaustin
Differential Revision: D14947235
fbshipit-source-id: a29f98163fa87185b028bb47945b6fab75700fd6
Summary: Fallout from another diff that never got landed.
Reviewed By: simpkins
Differential Revision: D14938191
fbshipit-source-id: 980728ca6a1d4c4f90fea46aac9525a03d9809e5
Summary:
Change the `eden gc` implementation to use RocksDB's `DeleteFilesInRange()`
function, as well as its new-ish `DeleteRange()` method.
This makes the garbage collection much faster, and also require much less free
disk space than previously.
`DeleteFilesInRange()` asks RocksDB to simply delete all SST files from disk
if they only contain keys in the specified range. Since the range we specify
should include all keys in the DB this should simply drop all SST files for
this column family.
We also call `DeleteRange()` after this, just in case. This API is relatively
recent, and writes a single tombstone saying that the specified range has been
deleted.
Reviewed By: wez
Differential Revision: D14910345
fbshipit-source-id: c76bdc1c8e07cb2def66673ea892e7f455c9dc7a
Summary:
EdenStats is currently an alias for ThreadLocal<EdenThreadStats>. I want to split EdenThreadStats into two structs which are allocated independently, but EdenStats's interface makes this impossible.
Refactor EdenStats from an alias to a class and encapsulate the underlying ThreadLocal<EdenThreadStats> member.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14822272
fbshipit-source-id: 691f4731aa22ecbdcd3535ee0bb0b99c816ffc3d
Summary:
I'm confused by the naming of EdenThreadStats and ThreadLocalEdenStats. For example, when I see "ThreadLocalEdenStats", I think that such objects can only be accessed by one thread. That's definitely not what "ThreadLocalEdenStats" means!
I think the following naming scheme makes more sense:
* **EdenThreadStats**: Statistics which are updated by one thread. Currently called EdenThreadStats.
* **EdenStats**: Statistics for all threads. Provides access to EdenThreadStats. Currently called ThreadLocalEdenStats.
Implement my preferred scheme: rename ThreadLocalEdenStats to EdenStats.
This diff should not change behavior.
Note: Prior to D14822274, EdenThreadStats was called EdenStats.
Reviewed By: simpkins
Differential Revision: D14822271
fbshipit-source-id: bd20179b1010588e3fc16dc9ed0657d458606f16
Summary:
I'm confused by the naming of EdenStats and ThreadLocalEdenStats. For example, when I see "ThreadLocalEdenStats", I think that such objects can only be accessed by one thread. That's definitely not what "ThreadLocalEdenStats" means!
I think the following naming scheme makes more sense:
* **EdenThreadStats**: Statistics which are updated by one thread. Currently called EdenStats.
* **EdenStats**: Statistics for all threads. Provides access to EdenThreadStats. Currently called ThreadLocalEdenStats.
Implement half of my preferred scheme: rename EdenStats to EdenThreadStats. (Make this diff easier to read by renaming ThreadLocalEdenStats to EdenStats in a separate diff.) In effect, implement the following naming scheme:
* **EdenThreadStats**: Statistics which are updated by one thread. (was EdenStats)
* **ThreadLocalEdenStats**: Statistics for all threads. Provides access to EdenThreadStats. (no change)
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14822274
fbshipit-source-id: 236cd9878cd249a06d14e124050ee01329667a18
Summary: This diff makes `MononokeHttpBackingStore` uses `ServiceAddress` to manage remote address of the Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D14855661
fbshipit-source-id: 1c102f7b2c547dc6a5b5c2cba65a513313d90bdc
Summary: This diff adds `ServiceAddress` to utils. This is a class that encapsulates an address of a service that could be a traditional hostname & port pair as well as an SMC tier name. This eliminates the needs of the manually resolving network address for remote services.
Reviewed By: chadaustin
Differential Revision: D14845257
fbshipit-source-id: 9fe9847cca4bba0170be94b9c209247342708574
Summary:
Sometimes, EdenFS goes bonkers and talks to 'hg debugedenimporthelper' a lot for seemingly no reason.
Make these situations easier to debug by counting how many requests EdenFS makes to 'hg debugedenimporthelper'. These counters let us answer questions such as the following:
* When performance sucks, is EdenFS is making a lot of requests or only a few requests? I.e. is Hg slow at responding or is EdenFS very demanding?
* Does a recent performance issue correlate with EdenFS communicating with 'hg debugedenimporthelper'?
* Which engineers are outliers having orders of magnitude more 'hg debugedenimporthelper' requests than p50 engineers?
We could get fancier with these counters and include the number of bytes received, the duration of the request, etc. For now, just having a request count is useful.
Reviewed By: simpkins
Differential Revision: D14677339
fbshipit-source-id: 7f8f394fb0096aef65d6a8a45d7da5936db539a0
Summary:
D8559702 changed `folly::IOBuf::computeChainDataLength()` to return a `size_t`
Update our format specifier to match to avoid compiler warnings on Mac.
Reviewed By: chadaustin
Differential Revision: D14878220
fbshipit-source-id: 19e96bea07c57bb542a848b3688d65143db51d13
Summary:
The file descriptor API here needs to go away, so switch this API to NetworkSocket
It is expected that this commit will cause a number of Open Source projects to temporarily show up as broken. This is due to the fact that not all projects get synced to Github at the exact same time, so the builds may temporarily be fetching an older version of it's dependencies than it needs to :) It should fix itself quickly.
Reviewed By: yfeldblum
Differential Revision: D14673328
fbshipit-source-id: c5842fa5dc383d50043e0d8228e35d03b10a1c6b
Summary:
Ensure that the `TreeInode::createImpl()` code always releases the contents
lock at the end of the scope that was intended to signify the critical
section. Previously this was only unlocked when returning from this scope
successfully.
In particular, the old behavior could deadlock:
- We first created the new child inode.
- If saveOverlayDir() failed, we would throw an exception, causing us to not
release the directory lock yet.
- As we returned in the exception case, the local `inode` variable would be
destroyed before the `contents` argument, causing us to release the child
InodePtr's refcount before releasing the parent directory's lock. Because
we were the only user of the child inode we would end up needing to acquire
the parent inode's content's lock to check if it was unlinked.
This fixes the code to ensure that we always release the tree's contents lock
before the child inode becomes unreferenced.
Reviewed By: chadaustin
Differential Revision: D14848807
fbshipit-source-id: 28dcde842072b9ee1e7c63d54456d849e31af8fe
Summary: Move Mononoke backing store initialization process to every request so we can adjust the configuration value without restarting the process.
Reviewed By: strager
Differential Revision: D14583357
fbshipit-source-id: 49ce2736229ce3062d34337757ebda6bb6eae16a
Summary: This is a helper function to help lazily initializing Mononoke backing stores only when necessary and release it when it is not used any more. See next diff in the stack for detail usage.
Reviewed By: chadaustin
Differential Revision: D14791528
fbshipit-source-id: c26811bc5c7aebcd02f704f10ad19bc35f8b9a21
Summary:
We updating the encoding that mercurial uses for storing paths to UTF-8.
The current step is preventing files that have invalid names from being
committed. To that end we are updating the HgPrefetch test to submit
valid file names (valid UTF-8). This allows us to start using UTF-8
encoded string for the internal representation of file paths.
Doing encoding transformation between internal storage and system locale will
be implemented later. Normalization should also be handled at that point.
Reviewed By: simpkins
Differential Revision: D14812780
fbshipit-source-id: 1689910500391eed941df185ba92aa2d2f3f0960
Summary:
I want to use EdenStats in eden/fs/store/. EdenStats currently lives in eden/fs/fuse/, and making eden/fs/store/ depend upon eden/fs/fuse/ is confusing. (It's also confusing that some code in eden/fs/fuse/ is used on Windows.)
Reorganize the code: move EdenStats into eden/fs/tracing/.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14677337
fbshipit-source-id: af26d214bcc3a9919920fbd4e59e6098fe4e3834
Summary:
This message was added in D14337058. It is logged at the `INFO` level, which
is enabled by default, but doesn't seem to add much value to normal production
logs.
Reviewed By: chadaustin
Differential Revision: D14712654
fbshipit-source-id: 5a86d883ace30e22d299046e33a6cd6247432857
Summary:
we need to be explicit about pulling in the dep for the
case where folly is not installed into a default installation
prefix.
Reviewed By: strager, pkaush
Differential Revision: D14683955
fbshipit-source-id: 0f302877674fb744ef8076641cd3fa72de74efe4
Summary:
This isn't a complete install but it is sufficient
to generate an `install` target for the getdeps build to run.
Reviewed By: strager
Differential Revision: D14680670
fbshipit-source-id: 9de1caa24c25702795842fe5b1b1f4d82aef24d8
Summary:
While testing out the new getdeps code I found that none
of the include directories from the probed libraries were being used.
The new getdeps installs each dep into its own prefix, whereas the
existing getdeps script installed them into the installation
prefix for eden itself. That meant that they were being implicitly
found from a single include directory.
In addition to this, I encountered linker failures for the pretty
printers; the solution to those was to add appropriate deps for
the modules that depend upon the pretty printers.
Reviewed By: pkaush
Differential Revision: D14638758
fbshipit-source-id: a4c2b4c79603c268e1b1c707a05c3cb0e3f2757b
Summary:
This function depends on things that are not available
when EDEN_HAVE_HG_TREEMANIFEST is not enabled.
Ensure that the entire function is disabled if that is the case.
Reviewed By: chadaustin
Differential Revision: D14638759
fbshipit-source-id: 3fe83b7b42357c2b818469214b016ff591058e35
Summary:
The issue is that the compiler needs an `else` to see
that we can only reach the throw if none of the other paths are
taken; with that satisfied it believes that we are legitimately
constexpr.
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=67371
Reviewed By: chadaustin
Differential Revision: D14638234
fbshipit-source-id: f9524d2816580f41842a40e30118b03998c3660a
Summary: This is a largely automated codemod that shifts `folly::AsyncServerSocket::AcceptCallback::connectionAccepted` to taking a `NetworkSocket` rather than a file descriptor. This needs to be done as a single atomic change to avoid breaking things.
Reviewed By: yfeldblum
Differential Revision: D13966702
fbshipit-source-id: 415622dc347de53368c404dfbe9a2deae5b75e18
Summary: This diff enables thrift as a type of connection when talks to Mononoke API Server.
Reviewed By: strager, pkaush
Differential Revision: D14508956
fbshipit-source-id: 32e130c714be164f19452dec554976ea1cb594d3
Summary: Implement a backing store that uses thrift protocol to communicate with Mononoke API Server.
Reviewed By: strager
Differential Revision: D13966575
fbshipit-source-id: 66f66dda6b17aecd6c6b4475ab6b004c608f457f
Summary:
When Eden runs on kernel 4.20 (has readdir caching but not
FUSE_NO_OPENDIR_SUPPORT) indicate to the kernel that Eden wants
readdir results to be cached.
Reviewed By: wez
Differential Revision: D13893922
fbshipit-source-id: 3092adb16dabc4273bdba1e6e9f15e9e3bd7bb87
Summary:
Eden requires no state in its directory handles, so tell the kernel it
doesn't need to send opendir() and releasedir() requests, provided it
has FUSE_NO_OPENDIR_SUPPORT.
Reviewed By: strager
Differential Revision: D13594734
fbshipit-source-id: ebd4b69f4efcd1428a69024c4bdffb1ae455fa40
Summary:
After the kernel added readdir caching, my testing uncovered that Eden
was invalidating TreeInode entries incorrectly when new entries were
added. Change TreeInode to distinguish between directory entry changes
and removals (FUSE_NOTIFY_INVAL_ENTRY) and additions
(FUSE_NOTIFY_INVAL_INODE).
Reviewed By: strager
Differential Revision: D13870422
fbshipit-source-id: 2a6f25bfd9e77436a5aae639fedbfd8a445b2e05
Summary:
Prior to D14398507, two calls to EdenMount::unmount would result in two calls to fuseMountPromise->getFuture(). D14398507 made two calls to EdenMount::unmount result in only one call to fuseMountPromise->getFuture().
Since fuseMountPromise->getFuture() is only called at most once, turn fuseMountPromise into a Promise instead of a SharedPromise.
Reviewed By: simpkins
Differential Revision: D14515070
fbshipit-source-id: 3ecaf9b55f3274046bc840bd5a65481cc552df72
