Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
Codemod:
* future<T>.then(callable with operator()(not-a-try)) to future<T>.thenValue(callable with operator()(not-a-try)).
* future<T>.then(callable with operator()()) to future<T>.thenValue(callable with operator()(auto&&)).
* future<T>.then(callable with operator()(auto)) to future<T>.thenValue(callable with operator()(auto)).
Reviewed By: chadaustin
Differential Revision: D9443286
fbshipit-source-id: be712b58b92dc7422f128713deaf6f46b29b36ce
Summary:
Update the `_get_watch_roots_for_watchman()` check so that it does not fail if
there is no `roots` entry in the output from `watchman watch-list`
Reviewed By: wez
Differential Revision: D9445138
fbshipit-source-id: e066375db3e963d08a91aa2e257d6c4fdd18da74
Summary:
Existing timer precision (milliseconds) is too coarse to
debug many operations, it will often round down to 0ms. I didn't see
any rationale for not using something more precise here.
Reviewed By: chadaustin
Differential Revision: D9437556
fbshipit-source-id: 6a79431c5a976e933ecda228aa5386c07b8fb668
Summary:
Add an `--extract-to` flag to `eden debug overlay` to ask it to copy inode
data out of the overlay. When used with a file inode this copies the file
contents. When used with a directory inode this copies the directory
structure and file contents for all materialized files under that directory.
This makes it easier to extract user's untracked and modified files out of the
overlay when recovering state after an unclean filesystem shutdown.
Reviewed By: wez
Differential Revision: D9445559
fbshipit-source-id: 9abcf82695014d79f248ba5fdb09727a49f3098a
Summary:
Allow callers to explicitly pass in the location of an overlay storage
directory, rather than requiring that this be found from a running Eden
instance.
This makes it easier to make copies of an overlay for later offline debugging.
Reviewed By: wez
Differential Revision: D9445560
fbshipit-source-id: b1f2ee0dc8bfca80dcc02c57c348f65599ab86e2
Summary:
Update all of the `eden debug` commands to use the newer `find_checkout()`
code rather than older `get_mount_path()` function.
The `find_checkout()` code makes sure that the EdenInstance actually points to
the correct edenfs instance for this checkout, and also works with checkouts
that are not currently mounted. In particular this allows
`eden debug overlay` to examine the overlay state even when the checkout is
not currently mounted.
Reviewed By: wez
Differential Revision: D9389865
fbshipit-source-id: 00578519d4805157a30c9b39abee9838925e8e76
Summary:
This adds a new `find_eden()` method to the Eden CLI code which can look up
information about the correct EdenCheckout and EdenInstance given a path.
In the future most CLI commands should switch to use this function over the
current `get_eden_instance()` and `get_mount_path()` methods. These older
APIs only work to find currently mounted checkouts, which is inadequate for
commands like `fsck` and some other debug commands that want to be able to
operate on unmounted checkouts.
This new logic is able to correctly find checkout information even for
unmounted checkouts. If the checkout is managed by an edenfs instance running
out of a non-default location it also correctly finds the state information
for Eden, rather than potentially using the wrong Eden state directory.
Reviewed By: wez
Differential Revision: D9385821
fbshipit-source-id: a6638f3c3817a595a7b7979c5cd218a2d7400f51
Summary: If the OS is Windows it will expect the in-fd and out-fd as Windows handles and convert them to CRT fds. This will not change the behavior if it's not Windows.
Reviewed By: wez
Differential Revision: D9344414
fbshipit-source-id: 8b30b57d979dc45af06676979e75e00b02d2b89b
Summary:
This diff is first in the series to make Eden work on Windows. It includes:
1. HG backing store and Object store, which provides the capability to talk to mercurial and fetch the file and folder contents on Windows.
2. Subprocess and Pipe definition for Windows.
3. The Visual studio solution and projects files to compile Eden and scm datapack.
Few Important points:
1. Most of the changes to existing code is done under a macro EDEN_WIN so that it doesn't impact on other platform.
2. Sqlite is used for caching the fetched contents. We are not using Rocksdb on Windows.
3. The main function only calls some test code and exit after printing the output.
4. The initializeMononoke code is disabled for Windows because it needs Proxygen to talk HTTP. Will enable this once I get Proxygen and other dependencies working.
5. HgImporter pass Windows handles to hg_import_helper as command line args. The code to convert these handles into fds is in a separate diff.
Reviewed By: wez
Differential Revision: D8653992
fbshipit-source-id: 52a3c3750425fb92c2a7158c2c214a9372661e13
Summary:
As a prerequisite to running inode unloading code in a background
queue, use a deterministic executor in TestMount to make sequencing in
unit tests reliable.
Reviewed By: simpkins
Differential Revision: D9323878
fbshipit-source-id: 0b85632c1637a8cf83d6f238675e5b6bbb6923c7
Summary:
Update the hg_import_helper.py code to catch exceptions from `repo.close()`
when we are re-opening the repository after an error. Also set `self.repo` to
`None` before we start, so that it will be left as None if anything goes wrong
either closing or re-opening the repository.
I ran into a situation where `repo.close()` threw an error, and previously
this would leave `hg_import_helper.py` stuck in a bad state since it would
still be pointing to the old repo object. The next time it received a command
it would fail and try to call `self.repo.close()` again, which would still
fail since the repository was already in a halfway closed state.
Now the code will always forget about the old repository object and create a
new object.
Reviewed By: wez
Differential Revision: D9419109
fbshipit-source-id: 15bb296ba19d9d3d2a2b90169bf25b0e8e197c1f
Summary:
Fix the output of `eden debug journal` after changing the Thrift API
to pass paths in bytes instead of strings.
Reviewed By: simpkins
Differential Revision: D9403416
fbshipit-source-id: acd9d0f8b241d11295a48fa11f9cc59bd86381f2
Summary: This commit let `HgImporter` pick up configuration values from `EdenConfig`.
Reviewed By: chadaustin
Differential Revision: D9346293
fbshipit-source-id: cb63f7d13a86058e9bf076eddb52212560a64cb1
Summary:
While destroying JournalDelta entries, unzip the chain as we go and
destroy each entry one by one. This prevents the stack from
overflowing on long JournalDelta chains and should fix some flaky
tests.
Reviewed By: wez
Differential Revision: D9355365
fbshipit-source-id: 31af124d318ca5d7a84314b707e1b3c71b2ccaa9
Summary:
For O(1) stack space JournalDelta destruction, we need to check
whether a reference-counted pointer has a unique refcount. It looks at
first glance like shared_ptr<T>::unique() gives us that, but it uses a
relaxed load and is now deprecated.
Sadly, the easiest thing is to make a new smart pointer class. (It
happens to be slightly more efficient, requiring one fewer heap
allocation per entry in the chain.)
Reviewed By: simpkins
Differential Revision: D9355314
fbshipit-source-id: 8c782ba9e0ec27fae90325079c199e1b82df88fa
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Several of folly::Future's methods are lvalue-qualified even though they act as though they are rvalue-qualified, that is, they provide a postcondition that says, in effect, callers should act as though the method invalidated its `this` object (regardless of whether that invalidation was actual or logical).
* This violates the C++ principle to "Express ideas directly in code" (see Core Guidelines), and generally makes it more confusing for callers as well as hiding the actual semantics from tools (linters, compilers, etc.).
* This dichotomy and confusion has manifested itself by some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* The goal of rvalueification is to make sure methods that are logically rvalue-qualified are actually rvalue-qualified, which forces callsites to acknowledge that rvalueification, e.g., `std::move(f).ensure(...)` instead of `f.ensure(...)`. This syntactic change in the callsites forces callers to acknowledge the method's rvalue semantics.
This diff started as a Codemod, then required manual fixes. Here were the codemod steps:
* expr.ensure(...) ==> std::move(expr).ensure(...) // if expr is not already an xvalue
* expr->ensure(...) ==> std::move(*expr).ensure(...)
Note: operator precedence of that last step is safe - no need to parenthesize `expr`. Reason: `->` binds more tightly than unary `*`.
Reviewed By: yfeldblum
Differential Revision: D9332070
fbshipit-source-id: 882121fe82c05fdb196ce676db686b6bc254974b
Summary:
Now that timestamps are read from the inode metadata table, and users
aren't likely to run a pre-metadata-table version, the timestamp data
in the overlay header's no longer needs to be written. So remove that
code which has the bonus of making unloading faster.
Reviewed By: wez
Differential Revision: D9318044
fbshipit-source-id: 27a9a9ee954003940209819466932237a81f8929
Summary:
The Eden CLI allowed setting the `EDEN_CONFIG_DIR` environment variable to
control where to find the `.eden` state directory, as an alternative to the
`--config-dir` command line argument. However, nothing currently appears to
use this variable. Therefore remove this functionality for now to help
simplify the code.
Reviewed By: wez
Differential Revision: D9355514
fbshipit-source-id: c64afb54f599924945573b07bc6d91b346978ea8
Summary: Fix a few minor issues that `mypy --strict` complains about.
Reviewed By: wez
Differential Revision: D9355653
fbshipit-source-id: af63825721fc964b7713df68e8618b595c91561d
Summary:
In the CLI, rename the "Config" class to "EdenInstance". This class
represents all state about a particular edenfs instance. It provides APIs for
making thrift calls to edenfs, for creating and destroying checkouts, and
generally does much more than just managing configuration.
Renaming it to "EdenInstance" also helps clarify that it is distinct from the
configuration tracked in the user's ~/.edenrc and the /etc/eden directory.
This change purely renames internal class and variable names and should not
affect any user-visible functionality.
Reviewed By: wez
Differential Revision: D9355515
fbshipit-source-id: ba5d4c3b753c6eb12a3783306dcd29e85fea3f52
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Several of folly::Future's methods are lvalue-qualified even though they act as though they are rvalue-qualified, that is, they provide a postcondition that says, in effect, callers should act as though the method invalidated its `this` object (regardless of whether that invalidation was actual or logical).
* This violates the C++ principle to "Express ideas directly in code" (see Core Guidelines), and generally makes it more confusing for callers as well as hiding the actual semantics from tools (linters, compilers, etc.).
* This dichotomy and confusion has manifested itself by some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* The goal of rvalueification is to make sure methods that are logically rvalue-qualified are actually rvalue-qualified, which forces callsites to acknowledge that rvalueification, e.g., `std::move(f).unit(...)` instead of `f.unit(...)`. This syntactic change in the callsites forces callers to acknowledge the method's rvalue semantics.
Codemod changes:
* expr.unit(...) ==> std::move(expr).unit(...) // if expr is not already an xvalue
* expr->unit(...) ==> std::move(*expr).unit(...)
Note: operator precedence of that last step is safe - no need to parenthesize `expr`. Reason: `->` binds more tightly than unary `*`.
Reviewed By: LeeHowes
Differential Revision: D9347419
fbshipit-source-id: 2773365f3793d977f2bad1c0a85ef394633e7d2c
Summary:
Watchman's Eden integration has a bug where the combination of
Watchman querying Eden for overlapping delta ranges ("give me changes
between X and Y, now changes between X+1 and Y+1") and Eden eliding
redundant change events ("add-modify-remove" -> []) results in
Watchman sometimes reporting that a file exists in its final
subscription update when it no longer does.
The fix is to never elide events, even for files that were added and
removed in the same sequence. To continue to support Watchman's `new`
flag, track whether a file existed at the beginning and end of a
journal delta.
Reviewed By: wez
Differential Revision: D9304964
fbshipit-source-id: f34c12b25f2b24e3a0d46fc94aa428528f4c5098
Summary:
The setcon() failure is not actionable or interesting, so don't log it
to stdout at Eden startup.
Reviewed By: wez
Differential Revision: D9344467
fbshipit-source-id: 68435c8f22c228f2fbb86f37c2b1874723934169
Summary:
Most of the time, the age cutoff for unloads is extraneous, so default
it to 0 in order to unload everything.
Reviewed By: wez
Differential Revision: D9329161
fbshipit-source-id: a241630ea5c069b6f9dafc9f021e39365b96b3bf
Summary: This commit adds two configurable value for setting path to client certificate and a flag to control Mononoke integration.
Reviewed By: chadaustin
Differential Revision: D9303157
fbshipit-source-id: 2f44d55d17b567655157a5f4b6f52e9468dda234
Summary:
To prevent the regression fixed by D8323051, make sure we run at least a couple
integration tests with the RocksDB LocalStore implementation.
Reviewed By: wez
Differential Revision: D8408390
fbshipit-source-id: 8fab4041ae39915d8be80e42814aab375c4acdda
Summary:
This prevents `hg status` from blowing up with a UTF-8 decode
error inside the generated thrift code.
Push safety concerns:
* This doesn't change the wire representation of the data
* Existing clients that believe it to be a string will continue to have
the same behavior
* Buck has its own copy of an older version of the thrift spec, so it will
continue to work "OK".
* When buck resyncs with our thrift file, some changes will likely be needed
to convert the byte arrays to strings or paths or whatever is appropriate
for bucks internal API
Work "OK" above means that clients that currently believe that `string` is
utf-8 encoded will have a runtime error if we ever send them a path that
is not utf-8. This is the behavior prior to this diff and will continue
to be the behavior for clients (like buck) that have an older version
of the thrift file.
Reviewed By: simpkins
Differential Revision: D9270843
fbshipit-source-id: b01135aec9152aaf5199e1c654ddd7f61c03717e
Summary:
This is a recurring issue for folks that have unstable systems
that either OOM or are hard rebooted.
Provide advice on how to resolve the issue.
Reviewed By: simpkins
Differential Revision: D9275963
fbshipit-source-id: d746b2cce56716d738304430e484784baf49eb29
Summary:
The root inode is particularly important, so always call `fdatasync()` on its
changes in the overlay before committing them.
This will hopefully reduce the number of cases we see where users have empty
or corrupt data for the root inode after hard rebooting their server. My
guess is that the root directory is being modified by hg or other tools
creating and removing temporary files in the root directory. If a change like
this is in progress when a hard reboot has been performed we risk data loss
without the `fdatasync()` call.
While Eden can still mostly serve the checkout data if other files or
directories are corrupt/missing in the overlay it currently is completely
unable to mount the checkout if the root overlay is corrupt. Therefore it
seems worth being more cautious about making sure that the root overlay data
is updated atomically.
Reviewed By: chadaustin, wez
Differential Revision: D9275852
fbshipit-source-id: b1e3eeb94ba670d0e2b52da4af7143d3ddbc919b
Summary:
eden doctor misreports the case that there are duplicate filewatcher
subscriptions for a Nuclide mount. Correctly report those cases as
duplicates instead of missing.
Reviewed By: wez
Differential Revision: D9228450
fbshipit-source-id: cd6fb3c3aa0c3c12f370c084ebc413eeff16ef02
Summary:
We have seen issues with Eden occasionally getting empty file contents back
from Mercurial. We have not been able to reproduce this issue directly
yet--the overall incidence rate relative to the number of users is fairly
small, but the problem is pretty problematic when it does occur.
This updates the hg_import_helper.py script to perform additional checking
when it finds a file that is empty.
Empty files that have never been modified or renamed all have the same file
revlog hash. If the rev hash in question is this known hash, we know that the
file is in fact empty. Otherwise, if remotefilelog is in use we check the
remotefilelog metadata to confirm the size, and we log an error if it is
non-zero. We then try re-opening the repository and re-importing the file.
These additional checks and log messages should hopefully help narrow down the
problem the next time we see a report of the issue. This should help us
identify if the remotefilelog metadata has the correct file size or not.
Reviewed By: wez
Differential Revision: D9260788
fbshipit-source-id: 29615b32632946cd319aa837bec3c68b757d3ee0
Summary:
Add the beginnings of an eden gc command. Today it's equivalent to
`eden debug clear_local_caches` followed by `eden
debug_compact_local_storage`, except that it compacts each column as
they're cleared to minimize peak disk consumption.
Eventually, it will also unload in-memory inodes, flush data from the
overlay, and clear the kernel's VFS cache too.
Reviewed By: wez
Differential Revision: D9138305
fbshipit-source-id: b303a63f601014cf38ca94c9e6f7c04394159ea8
Summary:
Facebook's open source tooling no longer publishes *.bzl files to
github by default, since these are primarily used only for our internal
builds.
Remove eden/defs.bzl to bring the github repository contents back in
sync with what the new open source tooling logic thinks it should be.
Test Plan:
Will confirm the open source tooling sync succeeds.
Summary:
I noticed while making some changes to eden doctor that our
typechecker wasn't working. check_types in python_library became a
no-op, so set typing instead and fix recently-introduced type-checking
failures.
Reviewed By: simpkins
Differential Revision: D9225840
fbshipit-source-id: 32edd60be862b70d97ca48dc9e1929b9add9f166
Summary: [Thrift] Rename generated client source file for consistency with the recently-added header. The new generated client source file is `{Service}AsyncClient.cpp`.
Reviewed By: stevegury
Differential Revision: D9174822
fbshipit-source-id: 49a050a96d3f583fdb5ad56545ca4a16d31605d7
Summary:
This commit integrates Mononoke API Server with eden:
* Added two new command line options: `--client_certificate` and `--use_mononoke`.
* Added two new config values: `ssl.client-certificate` and `mononoke.use-mononoke`.
* Made `HgImporter` return the repo name along with treemanifest options.
* Make `HgImporter` ask Mononoke API Server for tree data when the desired tree is not present in local Mercurial store.
Reviewed By: chadaustin
Differential Revision: D9183035
fbshipit-source-id: e328fb3237d10c545c8af71f856007ad6c487061
Summary: This update adds the name of the function which called the LOG() to the `LogMessage` object so that a custom formatter could use it to show method name with logs.
Reviewed By: simpkins
Differential Revision: D9071926
fbshipit-source-id: 9d5359d84cf2544371c6f1826f99d8b47a272dd1
Summary:
The python code supports expansion of ${HOME} and ${USER} in the config files.
Some of the config files use these values. This adds support for their
interpolation.
Reviewed By: chadaustin
Differential Revision: D9160960
fbshipit-source-id: 9133ee247c17937c9d8d548b9bfd26559a1459e9
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
6/n: Codemod rvalue-future<T>.then(...) to rvalue-future<T>.then(...).
Reviewed By: yfeldblum
Differential Revision: D9152002
fbshipit-source-id: 166475c1dcafb29a11154cbfbdf7e2e1feaf745b
Summary:
Eden should load its configuration from toml files. This change includes:
- ConfigParser is the class that parses and provides access to the configuration. We load the configuration from toml files and pass to ConfigParser in order to preserve existing behavior.
- The toml configuration format differs from the Config format. Thus, we use toml.load() and toml.dump() to load/save toml format configuration.
- Tests
- configuration is properly loaded from a group of config files
- toml configuration (if present) takes precedence
- missing configuration files (/home/user/.edenrc) are ignored
- invalid configuration files are fatal
- update of configuration works (adding repo)
- toml parsing handles over-rides of properties in the same way as cfg files
Deployment Strategy:
- This code determines whether to use toml configuration based on the presence
of atleast 1 '.toml' file in the etc-eden-dir/config.d.
Reviewed By: chadaustin
Differential Revision: D9023232
fbshipit-source-id: 6734a9d91eca92b05872a758c764546451dd2d51
Summary:
To improve the determinism of our C++ tests, I am planning to switch
TestMount to a ManualExecutor. This adds a ManualExecutor constructor
to UnboundedQueueExecutor.
In Rust, I'd use a trait, but a simple class with two constructors works fine.
Reviewed By: strager
Differential Revision: D8846553
fbshipit-source-id: c52752105255503d26f1e65494c32b3f62882e44
Summary:
This code is no longer necessary since introduction of FileChangeMonitor and
CachedParsedFileMonitor.
Reviewed By: simpkins
Differential Revision: D8915599
fbshipit-source-id: 7871dac1f7985968bd4d04a4dfb42684892a2e72
Summary:
[Eden] EdenMount should make use of EdenConfig in order to determine the user and system ignore files to load.
This commit does the following:
- adds EdenConfig to ServerState that gets passed to the MountPoint.
- removes hard-coded system ignore settings.
- allows the ignore files to be updated in the user and system config.
- has the diff context loaded with actual ignore file contents.
- adds the FileChangeMonitor class to efficiently (throttled) identify when file changes have occured and reload as necessary.
Reviewed By: simpkins
Differential Revision: D8876214
fbshipit-source-id: d2697c130d3d4960c7f645ace226e5ce6b772048
Summary:
This diff includes the following changes:
- Typically, we stat the monitored file to check for changes. Then, if changes
are detected, we open/fstat the file. If open fails, on subsequent
operations, we can skip the stat check.
- We only update the stat struct of the FileChangeMonitor member when the file
is successfully opened/processed. This means that all file change checks are
done relative to the processed file (rather than the results of the previous
stat).
- We still skip open/fstat if the isChanged() check indicates a file change
because of a different stat error code. This will still result in the
call-back being called.
Reviewed By: simpkins
Differential Revision: D9030118
fbshipit-source-id: def735fffe913f84b210cbf646f2849ab130bda4
Summary:
Add some integration tests that invoked "eden doctor" and "eden list" inside a
stale mount point, and confirms that they work correctly. These commands do
fail in dev builds, since python itself does not handle this error.
Fortunately XAR-based python archives avoids this error, so these commands
work in opt mode builds.
Reviewed By: strager
Differential Revision: D8905489
fbshipit-source-id: 0666f8cb0715d61697253bee9fe0bde2f4a3fa15
Summary:
Rocksdb default build does not include snappy support.
Eden needs it. Therefore enable it when building rocksdb.
Reviewed By: bolinfest
Differential Revision: D9029273
fbshipit-source-id: e2b702f1f2f46984d80afe79f609a2c32ee8dd5e
Summary:
Solves an issue building eden GitHub version:
eden/fs/service/EdenServer.cpp:1044:34: error: ‘class facebook::stats::ServiceData’ has no member named ‘addStatValue’
stats::ServiceData::get()->addStatValue(
^~~~~~~~~~~~
Reviewed By: bolinfest, simpkins
Differential Revision: D9029274
fbshipit-source-id: c0e5ea492c1c38f16bfae4b08d41538a2443450f
Summary:
Fixed a data race in Eden's `FuseChannel` implementation that could cause a crash if a `FUSE_INTERRUPT` request was concurrently on a different thread while a first thread was still launching the original request.
Modified `FakeFuse` to use `SOCK_SEQPACKET` instead of `SOCK_STREAM` so that tests can submit several requests in a pipelined fashion without having to first wait until the previous request was received. This requires the `recvResponse()` function to first read the header with `MSG_PEEK` to determine the response size and then subsequently read the entire message atomically to avoid reading a truncated message.
Added a new unit-test that exercises the `FUSE_INTERRUPT` race condition by sending a series of alternating `FUSE_LOOKUP`/`FUSE_INTERRUPT` requests.
Reviewed By: simpkins
Differential Revision: D9023654
fbshipit-source-id: 0eb44669ea8c4f58cf4313adf6ceb11098115a70
