Summary: Wrote tests for retrieving hashes from the object store and refactored existing tests. Includes positive and negative tests.
Reviewed By: strager, pkaush
Differential Revision: D15697488
fbshipit-source-id: 44aa8a36fbc1d1e56dcbbb6bcb665d3784cbb476
Summary: `getSize` and `getSha1` were misleading function names, since the functions refer to the size and hash of a given blob and not the object store itself. These functions have been renamed to `getBlobSize` and `getBlobSha1`.
Reviewed By: chadaustin
Differential Revision: D15696510
fbshipit-source-id: 4dd31659f60969fa90d8e2b39f43c46a2b7dff7c
Summary: I decoupled the getSize() function from the getMetadata() function, using a naive implementation for now. This was necessary because I want to add support for fetching only the size of a blob during a request like `ls -l`. Right now, the size and hash of a blob are coupled in a Metadata object, so if the size is requested, the whole file must be downloaded to calculate the hash, which is expensive for large files.
Reviewed By: chadaustin, strager
Differential Revision: D15678216
fbshipit-source-id: 8f68692768faaae0e65373ffe608d09ae49bbc42
Summary: HgCommand.h and related code isn't used anywhere. Delete it.
Reviewed By: simpkins
Differential Revision: D15773223
fbshipit-source-id: c4c116150bb74e29e4f220d3b0cbf988d4a1b1c5
Summary:
The eden/fs/store library depends on eden/fs/config. The changes in
D15428932 exposed a build failure due to this missing dependency.
Reviewed By: strager
Differential Revision: D15768197
fbshipit-source-id: 4482e3a9027fd49163af39d6887da709be810b0a
Summary:
The `FieldConverter<bool>` code previously relied on `folly::to<string>()` for
all arithmetic types, including `bool`. Unfortunately `folly::to<string>()`
returns `"1"` and `"0"` for booleans, which isn't terribly human-friendly.
This updates the code to explicitly return `"true"` and `"false"` as the
output for converting boolean values to strings.
Reviewed By: strager
Differential Revision: D15550014
fbshipit-source-id: 69a3385e529e2940a8be20dbcbeda92e5836d969
Summary:
Add a `FieldConvertor` implementation allowing a config field to be parsed
into a `std::chrono::nanoseconds` value.
Reviewed By: strager
Differential Revision: D15428932
fbshipit-source-id: d3ace45a2986717c798897534e7b3015eda35fa9
Summary:
The `ConfigSetting` code previously used `folly::to<string>()` to convert
field values back to strings for debugging. However, this only works for
values that actually are supported by `folly::to()`
This changes the code to use the `FieldConverter` to perform the conversion
back to string. The `FieldConverter` class now contains the logic for doing
conversion both from strings and back to strings.
Reviewed By: strager
Differential Revision: D15428933
fbshipit-source-id: a37a978d151bf0ba79ca2a2b9107401c847b6591
Summary:
Add a `durationToString()` function. This converts a duration value to a
string that can be parsed back with `stringToDuration()`.
Reviewed By: wez, strager
Differential Revision: D15428935
fbshipit-source-id: 2840663385434bc35f04c16d2570fd86e234cb3f
Summary:
Add a `stringToDuration()` function to parse a string into a
`std::chrono::nanoseconds` value.
Strings are parsed as a sequence of `[number] [unit]` pairs.
For instance:
`1m30s` --> 90 seconds
`5s10ms` --> 5010 milliseconds
`1d3h10m` --> 1 day, 3 hours, 10 minutes (1630 minutes total)
Reviewed By: strager
Differential Revision: D15428934
fbshipit-source-id: 63f6614595c1aaa08c0d03633c6d1b53ca5bf3d5
Summary:
Update the `FieldConverter` code to support parsing any arithmetic type
(integers and floating point) using `folly::to<T>()`. Also change the `bool`
conversion implementation to use the same code.
This does change the `bool` implementation to accept any string accepted by
`folly::to<bool>(string)`. The old code only allowed `true` and `false`. The
new code allows other strings: `yes`, `no`, `y`, `n`, `0`, `1`, `on`, `off`
This also makes the implementation more efficient by eliminating some
unnecessary string copies and avoiding throwing exceptions internally on
error.
Reviewed By: wez
Differential Revision: D15428700
fbshipit-source-id: feb73a1f53775fbf3d3acb8585fa689f79c0db81
Summary:
Add a thrift call to get the current config settings.
My primary use case for this method at the moment is to make it possible to
build integration tests that check the config behavior. However in the future
this will probably also be useful for building CLI commands to report the
current config values to allow debugging if there are ever issues. This API
can also be used to force EdenFS to immediately reload the config from disk.
Reviewed By: strager
Differential Revision: D15572124
fbshipit-source-id: da3bc982f9c419b3314a8b0560c9bd327760d429
Summary:
Change the `ReloadableConfig::getEdenConfig()` method to accept an enum to
control whether it should reload the configuration, rather than the simpler
bool that it used previously.
The primary motivation for this is to allow us to force a reload to occur in
unit tests and integration tests, even if this operation would otherwise be
within the normal reload throttling time limit.
Reviewed By: strager
Differential Revision: D15572122
fbshipit-source-id: 9c638a415a451f26e7f5923f544724594ed4e0f1
Summary: Previously functions were defined per string type but since SSO detection is now templatized so can path memory estimation.
Reviewed By: chadaustin
Differential Revision: D15723098
fbshipit-source-id: df8dcd359a0f4f704eba337ebf5e382d87ca3abe
Summary: Previously hard-coded values for how long a string has to be for SSO, now just check if data pointer is inside string stuct
Reviewed By: chadaustin
Differential Revision: D15720268
fbshipit-source-id: 988e45648e8b96332587f8d2f021642407d3dac3
Summary:
Update the EdenServer::unmountAll() code to make sure to keep a shared_ptr
reference to the EdenMount while the EdenMount::unmount() call is running.
Previously this could cause use-after-free errors due to the EdenMount
object being destroyed while the `unmount()` logic was still running. In
particular, as soon as we call `privhelper->fuseUnmount()` the
fuseCompletionFuture could complete in a different thread. The
fuseCompletionFuture's callback (normally the lambda inside
`EdenServer::mount()` which triggers the call to
`EdenServer::mountFinished()`) may have the last outstanding shared_ptr
refcount to the `EdenMount` object. When it finishes in this other thread
it will then destroy the EdenMount, possibly before the
`EdenMount::unmount()` logic has completed.
Reviewed By: strager
Differential Revision: D15688674
fbshipit-source-id: 47b8b4fe022d2bca112558f9ef32bcdd169fefb0
Summary: JournalStats is currently O(# of deltas), updating it to be O(1)
Reviewed By: chadaustin
Differential Revision: D15718255
fbshipit-source-id: 1fb3f0b76d736bfa22195231c21d5f8b742fa1f7
Summary: Setting up structure to get the stats from Eden's Journal to ODS
Reviewed By: chadaustin
Differential Revision: D15684380
fbshipit-source-id: 90debc3f2a09d497201e5e73f85400a994b7afba
Summary: Replace inclusions of `folly/futures/helpers.h` with `folly/futures/Future.h` to avoid the cyclic include trap.
Differential Revision: D15600549
fbshipit-source-id: 19950be24a7437fb1fbec293e24058adf17343ca
Summary:
The `testSplayOn()` and `testSplayOff()` tests have been failing occasionally
in continuous integration tests. In every test failure I have looked at, the
only reported failures were for iteration 0: the very first time the task was
scheduled after the server started. I believe the server start is simply
taking a non-zero amount of time, delaying how long it takes for the first
iteration of the tasks to be run.
This updates the tests to wait for 3 iterations of the main event base loop
before recording the test start time and scheduling the tasks.
Reviewed By: strager
Differential Revision: D15618328
fbshipit-source-id: 2b119700662457c0fcdf38cc231c6c7d26ade248
Summary:
Move the ConfigSource enum definition to a thrift file. This will let us
return ConfigSource values over thrift APIs in the future. This also allows
us to use thrift's `TEnumTraits` functionality to determine the maximum enum
value, rather than having to maintain a separate `kConfigSourceLastIndex`
variable.
As part of this change I also renamed the enum values to be CamelCase to match
our current C++ style recommendations and to avoid possibly conflicting with
macros defined in other headers (`DEFAULT` seemed particularly susceptible to
collision).
Reviewed By: strager
Differential Revision: D15572120
fbshipit-source-id: 8fbd03da221a9f75ef670dee1eb250eb198a5bd0
Summary:
Update the thrift definition for `initiateShutdown()` to indicate that it may
throw `EdenError` on failure. Without this all application-level errors that
occur will be translated to generic thrift `TApplicationException` objects on
the wire.
Also move this method declaration out of the "debugging APIs" section of the
file.
Reviewed By: strager
Differential Revision: D15572121
fbshipit-source-id: 7e621a24abd4347cedbb1bcce1ae9c2b70f991fd
Summary:
Don't honor the older (and deprecated) fb303 shutdown() call. The
`initiateShutdown()` call should always be used instead now, which includes a
bit more diagnostic information that we log to describe why the shutdown was
requested.
Reviewed By: wez, strager
Differential Revision: D15572123
fbshipit-source-id: 413be611313c9ae490673a2b34742e512cf8e5ea
Summary:
This was accidentally left behind from my original version of this code that
did not use `TEST_F()` but instead had a local `TestServer` variable in each
test function.
Reviewed By: wez
Differential Revision: D15618327
fbshipit-source-id: 5dd1252c77c286d22a53dd5cb4708a0b890f887c
Summary: This turned out to be a lot simpler than I thought it would!
Reviewed By: chadaustin
Differential Revision: D15630515
fbshipit-source-id: 51aeb8b6739cb886c3bca23ab441874ea9ac819c
Summary: These functions are part of the blocking API, which is being replaced by the Future-based API. They are not being called, so they are unneeded.
Reviewed By: chadaustin, strager
Differential Revision: D15582641
fbshipit-source-id: 11f693040124846515aef7c7ee6ad750e1b5c164
Summary: Developers should be encouraged to use Future-based functions instead of blocking functions in production code. Removing this blocking function makes it less convenient for developers to write blocking code in the future.
Reviewed By: strager
Differential Revision: D15564952
fbshipit-source-id: 3f62db472a59a6487ffe12e48c04178cad84ca61
Summary: The new Windows CMake targets were failing to include the generated thrift headers on Sandcastle machines. Making eden_service_thrift a dependecy will make sure that the thrift generated files are created before we compile the code.
Reviewed By: simpkins
Differential Revision: D15536743
fbshipit-source-id: 4bedb8f33cddf5f7eb8f5b5ce52b2013728b38f7
Summary: This is a stop gap solution to get the Windows build working with the getdeps, until we fix the SDK.
Reviewed By: chadaustin
Differential Revision: D15536740
fbshipit-source-id: 77cc6ea80c304a6cfcd0180bb28f63ce4dac2988
Summary:
Update EdenServer to use the new PeriodicTask class for flushing thread local
stats and collecting memory usage stats.
Reviewed By: strager
Differential Revision: D15492169
fbshipit-source-id: 41c23630ffe2f8365a0234a252821d07d48c8b29
Summary:
Add a new class for running periodic tasks on the main EventBase thread.
We already have several tasks that we run periodically:
- Flushing thread local stats (every 1s)
- Reporting memory stats (every 30s)
- Unloading unused inodes (configurable, but disabled by default)
I plan to add some new periodic tasks soon:
- Report LocalStore stats and perform garbage collection if needed
- Check if the config files have been updated and need to be re-read
This new class should make it easier to add periodic tasks with configurable
intervals, and make the intervals at which they run controllable from the
configuration file.
Reviewed By: strager
Differential Revision: D15492165
fbshipit-source-id: c1d2663da3c08851b60ba64d7a1e38ccca709b8a
Summary:
This diff teach Eden's Python client to use Windows' Unix Domain Socket support to connect to Eden.
#delayed_publish
Reviewed By: chadaustin
Differential Revision: D15454514
fbshipit-source-id: 86aa39f8aa13ae789efabda3fa88635f471a1a7a
Summary:
Some threads update FUSE counters, some threads update HgBackingStore counters, and some threads update HgImporter counters. No thread updates all of FUSE counters, HgBackingStore counters, and HgImporter counters. FUSE threads and HgImporter threads allocate the full set of counters (EdenThreadStats), even though many of the allocated counters can never be used during the thread's lifetime.
Reduce redundant allocation (and overhead during counter aggregation): split EdenThreadStats into three classes (FuseThreadStats, HgBackingStoreThreadStats, and HgImporterThreadStats) so different threads can allocate different sets of counters.
This diff should not change observable behavior.
Reviewed By: simpkins
Differential Revision: D14822273
fbshipit-source-id: cfd238187d20a0b8d3959673401ecad894e2095b
Summary: This diff pushes service resolution down to the `CurlClient` so that when a host is gone it can avoid sending more requests to a dead server.
Reviewed By: strager
Differential Revision: D15385384
fbshipit-source-id: a47e80f796dcd93d662750e3426abcb7cfb79607
Summary: Enable rocket server for services that switch to rocket/rsocket transport *in the future*. This diff should be a no-op for existing services that take rocket/rsocket traffic.
Reviewed By: yfeldblum
Differential Revision: D15511722
fbshipit-source-id: e2ba1ed9a32c18eca68b5d06722edbf029244f05
Summary:
Add a helper class for creating an EdenServer in a unit test.
Most of our existing unit tests only create EdenMount objects, without a full
EdenServer. This new class will make it easier to write tests for
functionality that does require a full EdenServer object.
Reviewed By: chadaustin
Differential Revision: D15492166
fbshipit-source-id: f8b1ce3b78a1160a5d55d305e6bf4b5305cca509
Summary:
In dev builds I have seen the `DaemonStartupLoggerTest.crashWithNoResult()`
test sometimes fail with the message
`edenfs is still running but did not report its initialization status`
instead of the expected error `edenfs crashed with signal 9`.
It appears that when killing the child process with `SIGKILL` we can see the
pipe close before the child is actually waitable. Update the `StartupLogger`
code to sleep briefly and retry the wait a few times before giving up in this
case. This allows us to determine the child's exit status correctly in this
situation.
Reviewed By: wez
Differential Revision: D15507309
fbshipit-source-id: 1f169c3061d1b2bd4154849c5dc7249e8f8c8e0c
Summary:
Read information from the machine-readable `/proc/self/statm` rather than
attempting to parse the human-readable `/proc/self/status` file.
This simplifies the parsing logic overall.
Reviewed By: wez
Differential Revision: D15466907
fbshipit-source-id: c0184b0faa0a18cf0101828c3822b546ca7a677e
Summary:
HgBackingStore has two constructors. The constructor used in production code accepts an EdenStats object, but the constructor used in test code does not. This means that `HgBackingStore::stats_` could be null, thus tracking statistics in HgBackingStore is more work than it should be.
Make HgBackingStore require an EdenStats object so `HgBackingStore::stats_` is never null.
In production code, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D15256266
fbshipit-source-id: f74137bed3e83e1b82d68b840fa55f3a3f0bf0cc
Summary:
`HgBackingStore::getBlob` calls `HgImporter::importFileContents` in two different branches. The calls are identical, and have an identical comment explaining the use of `folly::SemiFuture::via`. This duplication makes it annoying to add instrumentation and other changes.
Factor each call into a new `HgBackingStore::getBlobFromHgImporter` function, deduplicating the code.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D15256267
fbshipit-source-id: cad0566e9dab15775ee0d711e452f3e60fa6e645
Summary:
Remove calls to Future::then(executor, callback).
This form of Future::then is ambiguous, and does not yet implement the stronger
typing of thenValue and thenTry. It is also tempting to use instead of via,
where it is not obvious that it has the behaviour of wrapping a via call in a
push and pop of the current executor:
.pushCurrentExecutor().via(executor).then(callback).popCurrentExecutor().
With the addition of inline continuations, we can instead make the nesting
explicit at low cost by making it an inline continuation that launches an
asynchronous task on the passed executor.
Reviewed By: yfeldblum
Differential Revision: D15487604
fbshipit-source-id: 24b83f56c4f3e8843a6ce8339c70ff4794500d5b
Summary:
D15456687 didn't catch this due a problem with the CI
not showing the build failure in the code review tool.
Reviewed By: chadaustin
Differential Revision: D15472747
fbshipit-source-id: 5ad739926c0c01faaddd0d5c9631cdc8aafd33bd
Summary:
Computing the private memory usage from /proc/self/smaps is quite expensive.
On my system this takes around 90ms on opt mode builds, and 2s on
dev mode builds. (I believe dev mode builds are much worse because they use
lots of small shared libraries.)
In practice the RSS memory usage tracks relatively closely to the private
bytes count, and should good enough for most performance investigation and
monitoring.
We do still report the private memory usage in the `getStatInfo()` thrift call
results.
Reviewed By: chadaustin
Differential Revision: D15466906
fbshipit-source-id: 80ebce25046dfb705e37f50d1ec0270698718576
Summary:
This utility was initially used for testing flatmanifest and treemanifest
import. It's less important now that we are planning to drop flatmanifest
import.
The recent changes to this code have mostly just been keeping up with changes
to Eden's initialization code. I plan to continue making some additional
changes to Eden's config and local store initialization. Rather than
continuing to keep this code up-to-date it seems simpler to just delete it.
We can always resurrect this later if we find a need for it again.
Reviewed By: chadaustin, strager
Differential Revision: D15456687
fbshipit-source-id: dd65f93012530392b399a353f5b092192666f102
