Summary:
This utility was initially used for testing flatmanifest and treemanifest
import. It's less important now that we are planning to drop flatmanifest
import.
The recent changes to this code have mostly just been keeping up with changes
to Eden's initialization code. I plan to continue making some additional
changes to Eden's config and local store initialization. Rather than
continuing to keep this code up-to-date it seems simpler to just delete it.
We can always resurrect this later if we find a need for it again.
Reviewed By: chadaustin, strager
Differential Revision: D15456687
fbshipit-source-id: dd65f93012530392b399a353f5b092192666f102
Summary:
Some of our tests create Mercurial repositories with default settings
which means they don't support treemanifest. Flip treemanifest on for
them.
Reviewed By: wez
Differential Revision: D15302347
fbshipit-source-id: 34f98b88976b5de1bf2ad07c5272e217e3fe3b0a
Summary:
Add a mechanism to open the RocksDbLocalStore in read-only mode. This is
primarily helpful to allow the `eden_store_util` process to examine a DB
without making many modifications, including automatic compactions that might
happen otherwise.
Note that this doesn't appear to be entirely read-only from a file contents
perspective: RocksDB does still rewrite the MANIFEST and OPTIONS files when
opening the DB in read-only mode, and does rotate the logs and write a new LOG
file.
Reviewed By: wez
Differential Revision: D15350216
fbshipit-source-id: 797498f9575a05334b741322654843d6545ca35c
Summary: Use `folly::prettyPrint()` when printing the sizes.
Reviewed By: chadaustin
Differential Revision: D15398827
fbshipit-source-id: 17b1bee7b78f63c5b766aad8a65f1a8f776bf6b4
Summary:
Increase the default send timeout from 100ms to 250ms.
Increse the timeout in the `UnixSocket.sendDataAndFiles` test to 10s.
We have seen some failures in the `UnixSocket.sendDataAndFiles()` test on
our continuous build infrastructure when testing debug builds on Mac OS.
These failed due to hitting the send timeout on the 4MB and 32MB sends. Bump
up the timeout for this test to 10s. (I initially tried 250ms, and this was
still not sufficient.)
I increased the default send timeout to 250ms just to help ensure that we
don't hit it unexpectedly during real production code when running on Mac OS.
Reviewed By: wez
Differential Revision: D15401154
fbshipit-source-id: c8be68a77231e90d37c9b5b4f349379bdf2b15ed
Summary: getEntryPtr() does a case sensitive lookup because of which few Ovrsource builds were failing. Ovrsource code is including header files with the wrong case.
Reviewed By: strager
Differential Revision: D15344850
fbshipit-source-id: 3d5d658a49cdafc07dc9a18a2f3d2073306e8f40
Summary: Small parts of our code have diverged from our clang-format rules.
Reviewed By: strager
Differential Revision: D15380260
fbshipit-source-id: f668ac22d6c0c5f2468549f2a94dd1c9bb22ce3d
Summary: 2 seconds is apparently too short for some of the big directory and files we have in our repository.
Reviewed By: wez
Differential Revision: D15387381
fbshipit-source-id: 6ebf850c5fbad8d60c2e8c9f8cb19b1c28297d47
Summary:
The `LocalStore` constructor was updated to accept a `ReloadableConfig`
argument in D12949577, but this was never used anywhere. Remove it for now to
help simplify the code. If we do want to add it back in the future I think we
should make it required, rather than allowing a null config to be specified.
Reviewed By: wez
Differential Revision: D15350217
fbshipit-source-id: 8571b48dff8c8d079ba6b25821dd0b1d77ffe791
Summary:
Implement the `show_sizes` function which reports the approximate size on disk
used by each column family.
The initial skeleton for this command was accidentally included in D15162813.
Reviewed By: wez
Differential Revision: D15307391
fbshipit-source-id: 03efee939d65500e17b48047552dec1dbb7aff7e
Summary:
Add a `name` field to KeySpaceRecord. This allows the `RocksDbLocalStore` and
`SqliteLocalStore` code to use the name from the `kKeySpaceRecords` data
structure rather than keeping their own array of names that must be maintained
in sync with the `LocalStore` code.
Reviewed By: wez
Differential Revision: D15307390
fbshipit-source-id: b53bca7d53d630f9527a810f145380288c54198e
Summary:
Update the KeySpace enum values to start at 0 instead of 1. This simplifies
the code to avoid having to skip over 0 in a few places.
This also makes the `kKeySpaceRecords` array slightly less confusing. Unlike
the `columns` array used by `RocksDbLocalStore` and the `tableNames` array
used by `SqliteLocalStore`, the `kKeySpaceRecords` array was not previously
indexed by the `KeySpace` enum values.
Reviewed By: wez, strager
Differential Revision: D15307393
fbshipit-source-id: ae8392d02396b4dc3c18e9ee94b198fcbb9b1a34
Summary:
Move the KeySpace record definitions out of LocalStore.cpp and into a header
file. This will make it possible to access the key space information from
other files in the future.
I would like to have access to this data directly in the various LocalStore
implementations, as well as in eden_store_util.cpp. Currently both
RocksDbLocalStore.cpp and SqliteLocalStore.cpp have their own copies of key
space information that should really be shared.
Reviewed By: chadaustin, strager
Differential Revision: D15307392
fbshipit-source-id: ad7cdc705d3dc9834592fe0f14b38d82f91a3b77
Summary:
Now Eden on Windows could be build using
python.exe .\opensource\fbcode_builder\getdeps.py build eden
Reviewed By: strager
Differential Revision: D15251103
fbshipit-source-id: 8ac1097b234bb27ce478101bfbc9591251e2e359
Summary: MSVC doesn't like the ImporterOptions declared as class and defined as struct.
Reviewed By: wez
Differential Revision: D15251102
fbshipit-source-id: e7725df5db8ee39a20a1940a8d045369800aa4c3
Summary: tar is in /usr/bin on macOS.
Reviewed By: simpkins
Differential Revision: D15052024
fbshipit-source-id: 5388b2d53f876717310d0f9726b74e8a283d1a89
Summary:
This solves the client certificate mystery. We creates a new `EdenConfig` object
whenever there is a change detected and these two fields are lost during the
copying. Causing the variable substitution not working as expected.
Reviewed By: wez
Differential Revision: D15221180
fbshipit-source-id: 3fba36c8f261cb3dbe25e2e363d65661240be8cf
Summary: Change `ConfigSource` from a plain `enum` to an `enum class`.
Reviewed By: chadaustin
Differential Revision: D15162811
fbshipit-source-id: 680ec1e785de1a54b623d99477914afa5d0931ca
Summary:
Move the `ConfigSetting` template and related classes from `EdenConfig.h` to a
new `ConfigSetting.h` header file.
Reviewed By: chadaustin
Differential Revision: D15162815
fbshipit-source-id: 34fba0b8b0e8e50714f622bae642d20d77050673
Summary:
Update the `isValidAbsolutePath()` implementation to use `kDirSeparator`, and
update the TODO comment.
Reviewed By: chadaustin
Differential Revision: D15162812
fbshipit-source-id: f52f0e85c5eef342723947c6efaa52992da1660b
Summary:
Move the `FieldConverter` logic out of EdenConfig.h/cpp and into its own
module.
Reviewed By: chadaustin
Differential Revision: D15162816
fbshipit-source-id: f30d81329139e5827b81a609e83baddf71b6af5e
Summary:
Rename the `ClientConfig` class to `CheckoutConfig`, to help further progress
on updating our terminology from "client" to "checkout".
Reviewed By: chadaustin
Differential Revision: D15162814
fbshipit-source-id: 3575958f1161d5842c6f0ee9e2d2d20ab20b7372
Summary:
Move the `ReloadableConfig` implementation out of `ServerState` and into the
`ReloadableConfig` class itself. This also updates `ServerState` to simply
contain a `ReloadableConfig` member variable rather than inheriting from
`ReloadableConfig`.
This makes it easier to build other utilities that need a `ReloadableConfig`
object but do not have a full `ServerState` object, such as the
`eden_store_util` class and the `zero_blob` helper tool used by the
integration tests.
Reviewed By: chadaustin
Differential Revision: D15162813
fbshipit-source-id: 1ebc0ac6c475f4aa1090dd933ebce16bc222674c
Summary:
This timeout appears to be too short for the last step of the test that sends
32MB of data in multiple send calls of no more than 1000 bytes each. On
stress test runs this timeout has occasionally been firing and causing the
test to fail.
Reviewed By: chadaustin
Differential Revision: D15221306
fbshipit-source-id: d1f7ed2d58b47512a8f31f8324c61372394a6709
Summary:
beholdunittests
This enables some plumbing for running some of the
tests using the gtest/gmock machinery in cmake.
Part of this diff is removing the FindGMock.cmake file from the
eden repo; we now pull this in from the shared cmake library
that is populated by shipit.
Reviewed By: simpkins
Differential Revision: D14993344
fbshipit-source-id: 51caf9518c7f3a083a3b90cda10324c3a8170359
Summary:
This argument is unnecessary since D14222323, which should be deployed
everywhere by now. Passing in the extra repo argument forces hg to open the
repository object twice, which is unnecessary.
We already set the current working directory to the repository when invoking
the import helper, so hg will find the repository correctly from that.
Reviewed By: pkaush
Differential Revision: D15179667
fbshipit-source-id: 838cbee91748c41c713731187608a9823b971a53
Summary:
Update HgImporter to use `hg debugedenimporthelper` instead of the separate
`hg_import_helper.py` script by default on all platforms now. The
`debugedenimporthelper` subcommand has existed in our deployed versions of hg
for a while now.
Reviewed By: pkaush
Differential Revision: D15179668
fbshipit-source-id: 2fb8c4c9f92aed54c84899d6643f746baac73327
Summary:
This diff tries to improve the Thrift Mononoke backing store implementation by:
* Re-creating thrift client for every request since thrift has builtin connection pooling
* Using Eden's CPU thread pool for processing data fetched from remote
* Calling thrift methods within correct executor
Reviewed By: wez
Differential Revision: D15170818
fbshipit-source-id: c8be70755a851f63fb62e4d4f36833703283565e
Summary: We intend to break support for flatmanifest hg, so require that treemanifest is available in the build.
Reviewed By: wez
Differential Revision: D15057150
fbshipit-source-id: 449399cfb9d018f3b722598091eead1bd5d7c77d
Summary: Flatmanifest is on its way out. Remove support for falling back to it if a tree import fails.
Reviewed By: pkaush
Differential Revision: D15056459
fbshipit-source-id: a4df820322ee354d77f50a0ec92e9705d0f152ec
Summary:
Interestingly, the request doesn't fail, but does leave
our stack garbage intact. Let's be sure to zero it out to make
it more obvious what is happening.
While we're in here, let's make sure that we get the same
results from both ends of the socket pair.
Reviewed By: simpkins
Differential Revision: D14994593
fbshipit-source-id: 9aec957dfcd80d88c3d8fbce6bf45480502ea812
Summary: this fixes linking the tests when building with cmake in D14993344
Reviewed By: chadaustin
Differential Revision: D15167223
fbshipit-source-id: 1dda3beed3b6ff3788f3992783c58b4b92f697f2
Summary:
D15043209 missed fixing up the name of the variable
containing the list of sources.
The result is that we compile the `main` from `eden_store_util`
in to the tests and break gtest discovery in D14993344
Reviewed By: chadaustin
Differential Revision: D15167219
fbshipit-source-id: 95e9dc8ba74965f41bfebed1c1776605d8c9443a
Summary:
UnixSocketTest would block forever on macos if the message
size was larger than approx 1kb. It turns out that MSG_DONTWAIT
isn't documented in the `sendmsg` man pages and apparently has
no effect at all. Instead, the socket must be placed in non-blocking
mode. This appears to be the case for the client side of the sockets
but I need to follow up for the server side.
Reviewed By: simpkins, strager
Differential Revision: D14994588
fbshipit-source-id: 2a0a1c26a7b45ece82f8f79c15fb7756844bc86f
Summary:
on macOS `/var/tmp` is a symlink to `/private/var/tmp`
which means that our tests for path canonicalization end up
trying to compare the non-canonical `/var/tmp` prefix of the
temporary paths generated by the tests with `/private/var/tmp`
and failing.
Reviewed By: simpkins
Differential Revision: D14994584
fbshipit-source-id: e69addca8e315855d3ca87feb576ca7c54567c31
Summary:
D14677339 added tracing to all HgImporter requests. Each trace loads a thread-local variable through EdenStats::getStatsForCurrentThread(). Each of these loads has a small but non-trivial cost.
Because each instance of HgImporter is used only on one thread, each getStatsForCurrentThread() call returns the same EdenThreadStats object. Avoid the thread-local-variable lookups by caching the result of getStatsForCurrentThread() in HgImporter.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D14794284
fbshipit-source-id: d1609a1720d44c680dc0ebaa2536779def2a6f37
Summary:
If EdenFS' globFiles API is given two patterns, and one pattern is a prefix of the other, EdenFS effectively ignores the longer pattern. Given the patterns `project/src/*` and `project/src/*/*`, when EdenFS encounters `project/src/dir/`, it generates a result (because `project/src/dir/` matches the first pattern) but does not recurse into its children.
The problem caused by an incorrect understanding of the GlobNode::isLeaf_ flag. isLeaf_ means "this GlobNode should generate results", but GlobNode::evaluateImpl understands it to mean "this GlobNode should generate results, and this GlobNode has no children". Given the patterns `project/src/*` and `project/src/*/*`, the GlobNode representing `project/src/*` has isLeaf_=true but also has children.
Fix the bug by not using isLeaf_ for determining whether to recurse, and instead relying on the presence of child GlobNode-s.
Reviewed By: chadaustin
Differential Revision: D15078089
fbshipit-source-id: 1c480d11361f89193b35965266e6873c57181113
Summary:
If globbing returns a directory, that directory doesn't need to be loaded. GlobNode::evaluateImpl is careful to only load directories/trees if necessary. When fixing a bug in GlobNode::evaluateImpl, I accidentally broke this optimization. No test failed, though.
Ensure this optimization works and doesn't regress by writing a simple test.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D15090386
fbshipit-source-id: fe932e4a7ad517ff4c4523d58668f05fe8b0cafb
Summary:
In GlobNode::evaluateImpl, the logic for recursing into child GlobNode-s is duplicated token-for-token for hasSpecials_ and !hasSpecials_. While fixing a bug in GlobNode::evaluateImpl, I was forced to update both copies, which is prone to mistakes.
Fix the duplication by factoring the code into a function.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D15090688
fbshipit-source-id: 65eec62214074139cb8af75026758b631eb89bb2
Summary:
gdb and lldb have a hard time printing recursive structures involving std::vector and std::unique_ptr. This makes debugging GlobNode a pain.
Add a function to print a GlobNode recursively to stderr. This makes it easier to debug GlobNode using printf.
Reviewed By: simpkins
Differential Revision: D15078111
fbshipit-source-id: 3ba211026c7c894676435ade278ae859e1b5d1f6
Summary:
Sandcastle's [1] code coverage builds compile EdenFS and Hg with Clang's -fprofile-generate flag (or a related flag). By default, running a program with with that flag creates a `default.profraw` file in the current directory. This causes some tests to fail, such as HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import, because `default.profraw` is unintentionally committed to the test repo:
```
[==========] Running 1 test from 1 test case.
[----------] Global test environment set-up.
[----------] 1 test from HgBackingStoreTest
[ RUN ] HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import
eden/fs/store/hg/test/HgBackingStoreTest.cpp:64: Failure
Value of: tree1->getEntryNames()
Expected: has 2 elements where
element #0 is equal to foo,
element #1 is equal to src
Actual: { default.profraw, foo, src }, which has 3 elements
[ FAILED ] HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import (1563 ms)
[----------] 1 test from HgBackingStoreTest (1563 ms total)
[----------] Global test environment tear-down
[==========] 1 test from 1 test case ran. (1563 ms total)
[ PASSED ] 0 tests.
[ FAILED ] 1 test, listed below:
[ FAILED ] HgBackingStoreTest.getTreeForCommit_reimports_tree_if_it_was_deleted_after_import
```
When Sandcastle runs the tests, it sets the `LLVM_PROFILE_FILE` environment variable, which configures the path to `default.profraw`. EdenFS' HgRepo class is stripping this variable when invoking hg, so hg uses the default path and not the configured path.
Make HgRepo pass `LLVM_PROFILE_FILE` through to hg. This fixes some of EdenFS' tests when run by Sandcastle for code coverage purposes. (This does *not* fix running the tests manually (without setting `LLVM_PROFILE_FILE`), though.)
[1] Sandcastle is Facebook's continuous integration system.
Reviewed By: simpkins
Differential Revision: D15104832
fbshipit-source-id: 3929b9b0ab0904f2552ace7d8e4e4f4a4221192c
Summary:
Migrate the code from accessing optional Thrift fields directly to a safer
`optional_field_ref` API. See https://fburl.com/safe for more details.
The output of this codemod has been reviewed in D13259011.
To preserve semantics, each unchecked access is replaced with an explicit call
to `value_unchecked()`. If you are sure that accessing a field is safe (the
field is marked as set), you can later replace `value_unchecked()` with
`value()` or dereferencing (`operator *`):
```
ThriftStruct s = ...
- auto foo = s.foo_ref().value_unchecked();
+ auto foo = *s.foo_ref(); // will throw if s.foo is unset
```
Reviewed By: fanzeyi
Differential Revision: D15078040
fbshipit-source-id: 8ef270e8c687535e9d4aa2208beff4517e121814
Summary:
Add a command to force a call to RepairDB() on the local store.
This is similar to using `ldb repair`, but invokes `RepairDB()` with the same
set of column family options as normally used by edenfs.
Reviewed By: chadaustin
Differential Revision: D15043210
fbshipit-source-id: 2c4c0e2d3410a50cb1e523611f569f1701604ae6
Summary:
Change eden_store_util to take a subcommand argument, and define 3 separate
subcommands: "gc", "clear", and "compact"
The "gc" was the old default behavior. The "clear" and "compact" subcommands
simply clear cached entries and compact the storage, respectively.
This also adds a `--keySpace` argument. If this argument is specified the
command will process only this single key space, instead of all key spaces.
Reviewed By: chadaustin
Differential Revision: D15043207
fbshipit-source-id: 7e76bc89862d9647edde631ed3d82c9afda09f41
Summary:
This moves some logic from the RocksHandles class up to RocksDbLocalStore.
The main thing moved here is the logic to automatically try and repair the DB
if opening fails. This will make it easier in a subsequent diff to make the
repair logic a bit smarter and more aware of our column family semantics.
This keeps RocksHandles a pretty dumb wrapper around the RocksDB object and
column family handles, whose only purpose is to manage destroying these two
things in the correct order.
Reviewed By: chadaustin
Differential Revision: D15043208
fbshipit-source-id: ee2d5619ac7781a892e1ba151712eee9e3ebfb14
