Summary: Fix a few minor issues that `mypy --strict` complains about.
Reviewed By: wez
Differential Revision: D9355653
fbshipit-source-id: af63825721fc964b7713df68e8618b595c91561d
Summary:
In the CLI, rename the "Config" class to "EdenInstance". This class
represents all state about a particular edenfs instance. It provides APIs for
making thrift calls to edenfs, for creating and destroying checkouts, and
generally does much more than just managing configuration.
Renaming it to "EdenInstance" also helps clarify that it is distinct from the
configuration tracked in the user's ~/.edenrc and the /etc/eden directory.
This change purely renames internal class and variable names and should not
affect any user-visible functionality.
Reviewed By: wez
Differential Revision: D9355515
fbshipit-source-id: ba5d4c3b753c6eb12a3783306dcd29e85fea3f52
Summary:
This is part of "the great r-valuification of folly::Future":
* This is something we should do for safety in general.
* Several of folly::Future's methods are lvalue-qualified even though they act as though they are rvalue-qualified, that is, they provide a postcondition that says, in effect, callers should act as though the method invalidated its `this` object (regardless of whether that invalidation was actual or logical).
* This violates the C++ principle to "Express ideas directly in code" (see Core Guidelines), and generally makes it more confusing for callers as well as hiding the actual semantics from tools (linters, compilers, etc.).
* This dichotomy and confusion has manifested itself by some failures around D7840699 since lvalue-qualification hides that operation's move-out semantics - leads to some use of future operations that are really not correct, but are not obviously incorrect.
* The goal of rvalueification is to make sure methods that are logically rvalue-qualified are actually rvalue-qualified, which forces callsites to acknowledge that rvalueification, e.g., `std::move(f).unit(...)` instead of `f.unit(...)`. This syntactic change in the callsites forces callers to acknowledge the method's rvalue semantics.
Codemod changes:
* expr.unit(...) ==> std::move(expr).unit(...) // if expr is not already an xvalue
* expr->unit(...) ==> std::move(*expr).unit(...)
Note: operator precedence of that last step is safe - no need to parenthesize `expr`. Reason: `->` binds more tightly than unary `*`.
Reviewed By: LeeHowes
Differential Revision: D9347419
fbshipit-source-id: 2773365f3793d977f2bad1c0a85ef394633e7d2c
Summary:
Watchman's Eden integration has a bug where the combination of
Watchman querying Eden for overlapping delta ranges ("give me changes
between X and Y, now changes between X+1 and Y+1") and Eden eliding
redundant change events ("add-modify-remove" -> []) results in
Watchman sometimes reporting that a file exists in its final
subscription update when it no longer does.
The fix is to never elide events, even for files that were added and
removed in the same sequence. To continue to support Watchman's `new`
flag, track whether a file existed at the beginning and end of a
journal delta.
Reviewed By: wez
Differential Revision: D9304964
fbshipit-source-id: f34c12b25f2b24e3a0d46fc94aa428528f4c5098
Summary:
The setcon() failure is not actionable or interesting, so don't log it
to stdout at Eden startup.
Reviewed By: wez
Differential Revision: D9344467
fbshipit-source-id: 68435c8f22c228f2fbb86f37c2b1874723934169
Summary:
Most of the time, the age cutoff for unloads is extraneous, so default
it to 0 in order to unload everything.
Reviewed By: wez
Differential Revision: D9329161
fbshipit-source-id: a241630ea5c069b6f9dafc9f021e39365b96b3bf
Summary: This commit adds two configurable value for setting path to client certificate and a flag to control Mononoke integration.
Reviewed By: chadaustin
Differential Revision: D9303157
fbshipit-source-id: 2f44d55d17b567655157a5f4b6f52e9468dda234
Summary:
To prevent the regression fixed by D8323051, make sure we run at least a couple
integration tests with the RocksDB LocalStore implementation.
Reviewed By: wez
Differential Revision: D8408390
fbshipit-source-id: 8fab4041ae39915d8be80e42814aab375c4acdda
Summary:
This prevents `hg status` from blowing up with a UTF-8 decode
error inside the generated thrift code.
Push safety concerns:
* This doesn't change the wire representation of the data
* Existing clients that believe it to be a string will continue to have
the same behavior
* Buck has its own copy of an older version of the thrift spec, so it will
continue to work "OK".
* When buck resyncs with our thrift file, some changes will likely be needed
to convert the byte arrays to strings or paths or whatever is appropriate
for bucks internal API
Work "OK" above means that clients that currently believe that `string` is
utf-8 encoded will have a runtime error if we ever send them a path that
is not utf-8. This is the behavior prior to this diff and will continue
to be the behavior for clients (like buck) that have an older version
of the thrift file.
Reviewed By: simpkins
Differential Revision: D9270843
fbshipit-source-id: b01135aec9152aaf5199e1c654ddd7f61c03717e
Summary:
This is a recurring issue for folks that have unstable systems
that either OOM or are hard rebooted.
Provide advice on how to resolve the issue.
Reviewed By: simpkins
Differential Revision: D9275963
fbshipit-source-id: d746b2cce56716d738304430e484784baf49eb29
Summary:
The root inode is particularly important, so always call `fdatasync()` on its
changes in the overlay before committing them.
This will hopefully reduce the number of cases we see where users have empty
or corrupt data for the root inode after hard rebooting their server. My
guess is that the root directory is being modified by hg or other tools
creating and removing temporary files in the root directory. If a change like
this is in progress when a hard reboot has been performed we risk data loss
without the `fdatasync()` call.
While Eden can still mostly serve the checkout data if other files or
directories are corrupt/missing in the overlay it currently is completely
unable to mount the checkout if the root overlay is corrupt. Therefore it
seems worth being more cautious about making sure that the root overlay data
is updated atomically.
Reviewed By: chadaustin, wez
Differential Revision: D9275852
fbshipit-source-id: b1e3eeb94ba670d0e2b52da4af7143d3ddbc919b
Summary:
eden doctor misreports the case that there are duplicate filewatcher
subscriptions for a Nuclide mount. Correctly report those cases as
duplicates instead of missing.
Reviewed By: wez
Differential Revision: D9228450
fbshipit-source-id: cd6fb3c3aa0c3c12f370c084ebc413eeff16ef02
Summary:
We have seen issues with Eden occasionally getting empty file contents back
from Mercurial. We have not been able to reproduce this issue directly
yet--the overall incidence rate relative to the number of users is fairly
small, but the problem is pretty problematic when it does occur.
This updates the hg_import_helper.py script to perform additional checking
when it finds a file that is empty.
Empty files that have never been modified or renamed all have the same file
revlog hash. If the rev hash in question is this known hash, we know that the
file is in fact empty. Otherwise, if remotefilelog is in use we check the
remotefilelog metadata to confirm the size, and we log an error if it is
non-zero. We then try re-opening the repository and re-importing the file.
These additional checks and log messages should hopefully help narrow down the
problem the next time we see a report of the issue. This should help us
identify if the remotefilelog metadata has the correct file size or not.
Reviewed By: wez
Differential Revision: D9260788
fbshipit-source-id: 29615b32632946cd319aa837bec3c68b757d3ee0
Summary:
Add the beginnings of an eden gc command. Today it's equivalent to
`eden debug clear_local_caches` followed by `eden
debug_compact_local_storage`, except that it compacts each column as
they're cleared to minimize peak disk consumption.
Eventually, it will also unload in-memory inodes, flush data from the
overlay, and clear the kernel's VFS cache too.
Reviewed By: wez
Differential Revision: D9138305
fbshipit-source-id: b303a63f601014cf38ca94c9e6f7c04394159ea8
Summary:
Facebook's open source tooling no longer publishes *.bzl files to
github by default, since these are primarily used only for our internal
builds.
Remove eden/defs.bzl to bring the github repository contents back in
sync with what the new open source tooling logic thinks it should be.
Test Plan:
Will confirm the open source tooling sync succeeds.
Summary:
I noticed while making some changes to eden doctor that our
typechecker wasn't working. check_types in python_library became a
no-op, so set typing instead and fix recently-introduced type-checking
failures.
Reviewed By: simpkins
Differential Revision: D9225840
fbshipit-source-id: 32edd60be862b70d97ca48dc9e1929b9add9f166
Summary: [Thrift] Rename generated client source file for consistency with the recently-added header. The new generated client source file is `{Service}AsyncClient.cpp`.
Reviewed By: stevegury
Differential Revision: D9174822
fbshipit-source-id: 49a050a96d3f583fdb5ad56545ca4a16d31605d7
Summary:
This commit integrates Mononoke API Server with eden:
* Added two new command line options: `--client_certificate` and `--use_mononoke`.
* Added two new config values: `ssl.client-certificate` and `mononoke.use-mononoke`.
* Made `HgImporter` return the repo name along with treemanifest options.
* Make `HgImporter` ask Mononoke API Server for tree data when the desired tree is not present in local Mercurial store.
Reviewed By: chadaustin
Differential Revision: D9183035
fbshipit-source-id: e328fb3237d10c545c8af71f856007ad6c487061
Summary: This update adds the name of the function which called the LOG() to the `LogMessage` object so that a custom formatter could use it to show method name with logs.
Reviewed By: simpkins
Differential Revision: D9071926
fbshipit-source-id: 9d5359d84cf2544371c6f1826f99d8b47a272dd1
Summary:
The python code supports expansion of ${HOME} and ${USER} in the config files.
Some of the config files use these values. This adds support for their
interpolation.
Reviewed By: chadaustin
Differential Revision: D9160960
fbshipit-source-id: 9133ee247c17937c9d8d548b9bfd26559a1459e9
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
6/n: Codemod rvalue-future<T>.then(...) to rvalue-future<T>.then(...).
Reviewed By: yfeldblum
Differential Revision: D9152002
fbshipit-source-id: 166475c1dcafb29a11154cbfbdf7e2e1feaf745b
Summary:
Eden should load its configuration from toml files. This change includes:
- ConfigParser is the class that parses and provides access to the configuration. We load the configuration from toml files and pass to ConfigParser in order to preserve existing behavior.
- The toml configuration format differs from the Config format. Thus, we use toml.load() and toml.dump() to load/save toml format configuration.
- Tests
- configuration is properly loaded from a group of config files
- toml configuration (if present) takes precedence
- missing configuration files (/home/user/.edenrc) are ignored
- invalid configuration files are fatal
- update of configuration works (adding repo)
- toml parsing handles over-rides of properties in the same way as cfg files
Deployment Strategy:
- This code determines whether to use toml configuration based on the presence
of atleast 1 '.toml' file in the etc-eden-dir/config.d.
Reviewed By: chadaustin
Differential Revision: D9023232
fbshipit-source-id: 6734a9d91eca92b05872a758c764546451dd2d51
Summary:
To improve the determinism of our C++ tests, I am planning to switch
TestMount to a ManualExecutor. This adds a ManualExecutor constructor
to UnboundedQueueExecutor.
In Rust, I'd use a trait, but a simple class with two constructors works fine.
Reviewed By: strager
Differential Revision: D8846553
fbshipit-source-id: c52752105255503d26f1e65494c32b3f62882e44
Summary:
This code is no longer necessary since introduction of FileChangeMonitor and
CachedParsedFileMonitor.
Reviewed By: simpkins
Differential Revision: D8915599
fbshipit-source-id: 7871dac1f7985968bd4d04a4dfb42684892a2e72
Summary:
[Eden] EdenMount should make use of EdenConfig in order to determine the user and system ignore files to load.
This commit does the following:
- adds EdenConfig to ServerState that gets passed to the MountPoint.
- removes hard-coded system ignore settings.
- allows the ignore files to be updated in the user and system config.
- has the diff context loaded with actual ignore file contents.
- adds the FileChangeMonitor class to efficiently (throttled) identify when file changes have occured and reload as necessary.
Reviewed By: simpkins
Differential Revision: D8876214
fbshipit-source-id: d2697c130d3d4960c7f645ace226e5ce6b772048
Summary:
This diff includes the following changes:
- Typically, we stat the monitored file to check for changes. Then, if changes
are detected, we open/fstat the file. If open fails, on subsequent
operations, we can skip the stat check.
- We only update the stat struct of the FileChangeMonitor member when the file
is successfully opened/processed. This means that all file change checks are
done relative to the processed file (rather than the results of the previous
stat).
- We still skip open/fstat if the isChanged() check indicates a file change
because of a different stat error code. This will still result in the
call-back being called.
Reviewed By: simpkins
Differential Revision: D9030118
fbshipit-source-id: def735fffe913f84b210cbf646f2849ab130bda4
Summary:
Add some integration tests that invoked "eden doctor" and "eden list" inside a
stale mount point, and confirms that they work correctly. These commands do
fail in dev builds, since python itself does not handle this error.
Fortunately XAR-based python archives avoids this error, so these commands
work in opt mode builds.
Reviewed By: strager
Differential Revision: D8905489
fbshipit-source-id: 0666f8cb0715d61697253bee9fe0bde2f4a3fa15
Summary:
Rocksdb default build does not include snappy support.
Eden needs it. Therefore enable it when building rocksdb.
Reviewed By: bolinfest
Differential Revision: D9029273
fbshipit-source-id: e2b702f1f2f46984d80afe79f609a2c32ee8dd5e
Summary:
Solves an issue building eden GitHub version:
eden/fs/service/EdenServer.cpp:1044:34: error: ‘class facebook::stats::ServiceData’ has no member named ‘addStatValue’
stats::ServiceData::get()->addStatValue(
^~~~~~~~~~~~
Reviewed By: bolinfest, simpkins
Differential Revision: D9029274
fbshipit-source-id: c0e5ea492c1c38f16bfae4b08d41538a2443450f
Summary:
Fixed a data race in Eden's `FuseChannel` implementation that could cause a crash if a `FUSE_INTERRUPT` request was concurrently on a different thread while a first thread was still launching the original request.
Modified `FakeFuse` to use `SOCK_SEQPACKET` instead of `SOCK_STREAM` so that tests can submit several requests in a pipelined fashion without having to first wait until the previous request was received. This requires the `recvResponse()` function to first read the header with `MSG_PEEK` to determine the response size and then subsequently read the entire message atomically to avoid reading a truncated message.
Added a new unit-test that exercises the `FUSE_INTERRUPT` race condition by sending a series of alternating `FUSE_LOOKUP`/`FUSE_INTERRUPT` requests.
Reviewed By: simpkins
Differential Revision: D9023654
fbshipit-source-id: 0eb44669ea8c4f58cf4313adf6ceb11098115a70
Summary:
The `updateOverlayHeader()` only updates the overlay data if the inode is
materialized. This updates the name to clarify that.
(This function name change was previously part of D8884795, and I'm just
splitting it into its own separate diff.)
Reviewed By: bolinfest
Differential Revision: D9011358
fbshipit-source-id: 6024d64a1dee0b5d741bec32ed88f6c8f8dd8a9a
Summary:
I tried to build eden using GitHub version and ran into SELinux issues.
This patch fixed related issues, namely:
- `set(X Y)` sets X to string literal Y. Change it to `set(X ${Y})`.
- `SELINUX_INCLUDE_DIR` could be undefined. Check before use it.
- `./eden/fs/eden-config.h` and `./build/eden/fs/eden-config.h` both exist and
the former is used. Set include dirs so the latter gets used.
Reviewed By: bolinfest
Differential Revision: D9029272
fbshipit-source-id: 0c94bbe2f9e3fa90973802ddde16ad4d9ddfc0e8
Summary:
Many of the tests in FuseChannelTest.cpp perform blocking waits on Futures.
Previously they used a 100ms timeout. This could sometimes result in flaky
test failures if the system was under heavy CPU load. Bumping up the timeout
to 1 second appears to avoid this issue.
Reviewed By: lewissbaker
Differential Revision: D9024730
fbshipit-source-id: 7af78c61008c6b4c1e5e130b3d37b2f3ac787a01
Summary:
Update `InodeMap::updateOverlayForUnload()` to catch exceptions that occur
trying to save state to the overlay. If something goes wrong when saving
state there is not much we can do other than log an error. We still want to
unload the inode, and code that is unloading inodes generally cannot deal with
exceptions at this point. In particular if this error occurred while trying
to shut down an EdenMount the code would crash.
Reviewed By: chadaustin
Differential Revision: D8884795
fbshipit-source-id: c2f850f13d775be4b0a0a10f9df3948c7b2c8f4a
Summary: This commit changes MononokeBackingStore to use new APIs provided by Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D8882789
fbshipit-source-id: 0f06ca5f850af9fb52f1d593b9abd715a541488a
Summary:
Encountering a truncated overlay file doesn't necessarily indicate a software
bug in Eden. Depending on the underlying filesystem this often happens after
a hard system reboot since we write the overlay files without an `fdatasync()`
call.
Change the code to simply log an error and throw an exception rather than
using `EDEN_BUG()`. This makes it possible to exercise this code path in
tests without having it crash in debug builds.
Reviewed By: chadaustin
Differential Revision: D8988209
fbshipit-source-id: 8c0fe1dae692f4c493413d3939d2e4c21e0da596
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
Summary:
CachedParsedFileMonitor provides cached access to an object created by parsing
a data file. The object can be accessed through "getFileContents().
"getFileContents()" will check if the file has changed and reload/parse if
necessary. A throttle is applied to limit change checks to at most to 1
per throttleMilliSeconds.
Reviewed By: simpkins
Differential Revision: D8903257
fbshipit-source-id: 7ed28610a3e9b40b98b0a9ffaff7b307bbcd32a8
Summary:
This commit improves the FileChangeMonitor interfaces. Changes are:
- on-demand check initiates a call-back
- FileChangeMonitor does not rely on user to call-back to update stat
- updated tests for new api
- fixed issue for determining if file changed (on error)
Reviewed By: simpkins
Differential Revision: D8989042
fbshipit-source-id: cf7a03b2bc8679891c8b9c22a24eed980eeb4060
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
3/n: Codemod rvalue-future<T>.then(callable with operator()(not-a-try)) to rvalue-future<T>.thenValue(callable with operator()(not-a-try)).
Reviewed By: yfeldblum
Differential Revision: D8986716
fbshipit-source-id: 906339d9ffb90b3c38a24ce8bf0cef7be318d946
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use
Future<T>::thenTry or Future<T>::thenValue.
1/n: Codemod rvalue-future<T>.then(callable with operator()(Try<T>)) to rvalue-future<T>.thenTry(callable with operator()(Try<T>)).
Reviewed By: simpkins
Differential Revision: D8961903
fbshipit-source-id: ff17b7833d240c221197cdf0bf914b8a39f80b07
Summary: Update type information so that `mypy --strict` no longer has any warnings.
Reviewed By: strager
Differential Revision: D8940414
fbshipit-source-id: e55533ad5c7c67bd598fae2ae57e6a8d84b47b49
Summary:
Fix the arguments to `doctor.cure_what_ails_you()`, which were updated in
D8870494.
Reviewed By: chadaustin
Differential Revision: D8940413
fbshipit-source-id: 21d45036f668c9bae9a2002c360532918eb157a5
Summary:
If the `nuclide-connections` command is not installed
`subprocess.check_output()` throws a `FileNotFound` exception. Previously the
code only caught `CalledProcessError` and `ValueError` exception. This
updates the code to catch any Exception type, just in case some other
unexpected exception type can also be thrown too.
Reviewed By: chadaustin
Differential Revision: D8945774
fbshipit-source-id: 5e6f63981c3ed6695a687b4bfbaac97ea440b537
Summary:
Add a new `eden debug logging` command to show and update the current edenfs
logging settings.
I have deprecated the `eden debug set_logging_command`, but not removed it
entirely yet. It now longer shows up in the `eden debug --help` output, and
if run it prints the `eden debug logging` command that the user should run
instead.
Reviewed By: chadaustin
Differential Revision: D8907310
fbshipit-source-id: 7ed056637b236db5a50c619023bd8025b787e227
Summary:
This returns a `Repository` object and not a `Repository` subclass type.
This type information has been broken since D7512320 replaced the
`get_repo_class()` method with `create_repo()`.
Reviewed By: strager
Differential Revision: D8940415
fbshipit-source-id: 7711a2a33bc12fc361dfd0ff3f47eba1e587e6e7
Summary:
Add the FileChangeMonitor class to monitor for a file for changes. It provides
an interface "isChanged()" to check if the file has changed. It limits checks
by doing them on demand and by throttling. The throtte limits checks to 1 per
throttleMilliSeconds. It uses Stat information to determine if the file has
changed.
Reviewed By: simpkins
Differential Revision: D8898783
fbshipit-source-id: 067781489a9a91e5fe77cb613fba9d35b8348548
Summary:
InodePath is difficult to work with because it's just an alias to std::array. Make InodePath look more like std::string so using InodePath instances is more intuitive.
This diff should not change behaviour.
Reviewed By: chadaustin
Differential Revision: D8923073
fbshipit-source-id: 85a2548f0cfa61e50b6590048084076b9bece3da
