Summary:
I noticed while making some changes to eden doctor that our
typechecker wasn't working. check_types in python_library became a
no-op, so set typing instead and fix recently-introduced type-checking
failures.
Reviewed By: simpkins
Differential Revision: D9225840
fbshipit-source-id: 32edd60be862b70d97ca48dc9e1929b9add9f166
Summary: [Thrift] Rename generated client source file for consistency with the recently-added header. The new generated client source file is `{Service}AsyncClient.cpp`.
Reviewed By: stevegury
Differential Revision: D9174822
fbshipit-source-id: 49a050a96d3f583fdb5ad56545ca4a16d31605d7
Summary:
This commit integrates Mononoke API Server with eden:
* Added two new command line options: `--client_certificate` and `--use_mononoke`.
* Added two new config values: `ssl.client-certificate` and `mononoke.use-mononoke`.
* Made `HgImporter` return the repo name along with treemanifest options.
* Make `HgImporter` ask Mononoke API Server for tree data when the desired tree is not present in local Mercurial store.
Reviewed By: chadaustin
Differential Revision: D9183035
fbshipit-source-id: e328fb3237d10c545c8af71f856007ad6c487061
Summary: This update adds the name of the function which called the LOG() to the `LogMessage` object so that a custom formatter could use it to show method name with logs.
Reviewed By: simpkins
Differential Revision: D9071926
fbshipit-source-id: 9d5359d84cf2544371c6f1826f99d8b47a272dd1
Summary:
The python code supports expansion of ${HOME} and ${USER} in the config files.
Some of the config files use these values. This adds support for their
interpolation.
Reviewed By: chadaustin
Differential Revision: D9160960
fbshipit-source-id: 9133ee247c17937c9d8d548b9bfd26559a1459e9
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
6/n: Codemod rvalue-future<T>.then(...) to rvalue-future<T>.then(...).
Reviewed By: yfeldblum
Differential Revision: D9152002
fbshipit-source-id: 166475c1dcafb29a11154cbfbdf7e2e1feaf745b
Summary:
Eden should load its configuration from toml files. This change includes:
- ConfigParser is the class that parses and provides access to the configuration. We load the configuration from toml files and pass to ConfigParser in order to preserve existing behavior.
- The toml configuration format differs from the Config format. Thus, we use toml.load() and toml.dump() to load/save toml format configuration.
- Tests
- configuration is properly loaded from a group of config files
- toml configuration (if present) takes precedence
- missing configuration files (/home/user/.edenrc) are ignored
- invalid configuration files are fatal
- update of configuration works (adding repo)
- toml parsing handles over-rides of properties in the same way as cfg files
Deployment Strategy:
- This code determines whether to use toml configuration based on the presence
of atleast 1 '.toml' file in the etc-eden-dir/config.d.
Reviewed By: chadaustin
Differential Revision: D9023232
fbshipit-source-id: 6734a9d91eca92b05872a758c764546451dd2d51
Summary:
To improve the determinism of our C++ tests, I am planning to switch
TestMount to a ManualExecutor. This adds a ManualExecutor constructor
to UnboundedQueueExecutor.
In Rust, I'd use a trait, but a simple class with two constructors works fine.
Reviewed By: strager
Differential Revision: D8846553
fbshipit-source-id: c52752105255503d26f1e65494c32b3f62882e44
Summary:
This code is no longer necessary since introduction of FileChangeMonitor and
CachedParsedFileMonitor.
Reviewed By: simpkins
Differential Revision: D8915599
fbshipit-source-id: 7871dac1f7985968bd4d04a4dfb42684892a2e72
Summary:
[Eden] EdenMount should make use of EdenConfig in order to determine the user and system ignore files to load.
This commit does the following:
- adds EdenConfig to ServerState that gets passed to the MountPoint.
- removes hard-coded system ignore settings.
- allows the ignore files to be updated in the user and system config.
- has the diff context loaded with actual ignore file contents.
- adds the FileChangeMonitor class to efficiently (throttled) identify when file changes have occured and reload as necessary.
Reviewed By: simpkins
Differential Revision: D8876214
fbshipit-source-id: d2697c130d3d4960c7f645ace226e5ce6b772048
Summary:
This diff includes the following changes:
- Typically, we stat the monitored file to check for changes. Then, if changes
are detected, we open/fstat the file. If open fails, on subsequent
operations, we can skip the stat check.
- We only update the stat struct of the FileChangeMonitor member when the file
is successfully opened/processed. This means that all file change checks are
done relative to the processed file (rather than the results of the previous
stat).
- We still skip open/fstat if the isChanged() check indicates a file change
because of a different stat error code. This will still result in the
call-back being called.
Reviewed By: simpkins
Differential Revision: D9030118
fbshipit-source-id: def735fffe913f84b210cbf646f2849ab130bda4
Summary:
Add some integration tests that invoked "eden doctor" and "eden list" inside a
stale mount point, and confirms that they work correctly. These commands do
fail in dev builds, since python itself does not handle this error.
Fortunately XAR-based python archives avoids this error, so these commands
work in opt mode builds.
Reviewed By: strager
Differential Revision: D8905489
fbshipit-source-id: 0666f8cb0715d61697253bee9fe0bde2f4a3fa15
Summary:
Rocksdb default build does not include snappy support.
Eden needs it. Therefore enable it when building rocksdb.
Reviewed By: bolinfest
Differential Revision: D9029273
fbshipit-source-id: e2b702f1f2f46984d80afe79f609a2c32ee8dd5e
Summary:
Solves an issue building eden GitHub version:
eden/fs/service/EdenServer.cpp:1044:34: error: ‘class facebook::stats::ServiceData’ has no member named ‘addStatValue’
stats::ServiceData::get()->addStatValue(
^~~~~~~~~~~~
Reviewed By: bolinfest, simpkins
Differential Revision: D9029274
fbshipit-source-id: c0e5ea492c1c38f16bfae4b08d41538a2443450f
Summary:
Fixed a data race in Eden's `FuseChannel` implementation that could cause a crash if a `FUSE_INTERRUPT` request was concurrently on a different thread while a first thread was still launching the original request.
Modified `FakeFuse` to use `SOCK_SEQPACKET` instead of `SOCK_STREAM` so that tests can submit several requests in a pipelined fashion without having to first wait until the previous request was received. This requires the `recvResponse()` function to first read the header with `MSG_PEEK` to determine the response size and then subsequently read the entire message atomically to avoid reading a truncated message.
Added a new unit-test that exercises the `FUSE_INTERRUPT` race condition by sending a series of alternating `FUSE_LOOKUP`/`FUSE_INTERRUPT` requests.
Reviewed By: simpkins
Differential Revision: D9023654
fbshipit-source-id: 0eb44669ea8c4f58cf4313adf6ceb11098115a70
Summary:
The `updateOverlayHeader()` only updates the overlay data if the inode is
materialized. This updates the name to clarify that.
(This function name change was previously part of D8884795, and I'm just
splitting it into its own separate diff.)
Reviewed By: bolinfest
Differential Revision: D9011358
fbshipit-source-id: 6024d64a1dee0b5d741bec32ed88f6c8f8dd8a9a
Summary:
I tried to build eden using GitHub version and ran into SELinux issues.
This patch fixed related issues, namely:
- `set(X Y)` sets X to string literal Y. Change it to `set(X ${Y})`.
- `SELINUX_INCLUDE_DIR` could be undefined. Check before use it.
- `./eden/fs/eden-config.h` and `./build/eden/fs/eden-config.h` both exist and
the former is used. Set include dirs so the latter gets used.
Reviewed By: bolinfest
Differential Revision: D9029272
fbshipit-source-id: 0c94bbe2f9e3fa90973802ddde16ad4d9ddfc0e8
Summary:
Many of the tests in FuseChannelTest.cpp perform blocking waits on Futures.
Previously they used a 100ms timeout. This could sometimes result in flaky
test failures if the system was under heavy CPU load. Bumping up the timeout
to 1 second appears to avoid this issue.
Reviewed By: lewissbaker
Differential Revision: D9024730
fbshipit-source-id: 7af78c61008c6b4c1e5e130b3d37b2f3ac787a01
Summary:
Update `InodeMap::updateOverlayForUnload()` to catch exceptions that occur
trying to save state to the overlay. If something goes wrong when saving
state there is not much we can do other than log an error. We still want to
unload the inode, and code that is unloading inodes generally cannot deal with
exceptions at this point. In particular if this error occurred while trying
to shut down an EdenMount the code would crash.
Reviewed By: chadaustin
Differential Revision: D8884795
fbshipit-source-id: c2f850f13d775be4b0a0a10f9df3948c7b2c8f4a
Summary: This commit changes MononokeBackingStore to use new APIs provided by Mononoke API Server.
Reviewed By: chadaustin
Differential Revision: D8882789
fbshipit-source-id: 0f06ca5f850af9fb52f1d593b9abd715a541488a
Summary:
Encountering a truncated overlay file doesn't necessarily indicate a software
bug in Eden. Depending on the underlying filesystem this often happens after
a hard system reboot since we write the overlay files without an `fdatasync()`
call.
Change the code to simply log an error and throw an exception rather than
using `EDEN_BUG()`. This makes it possible to exercise this code path in
tests without having it crash in debug builds.
Reviewed By: chadaustin
Differential Revision: D8988209
fbshipit-source-id: 8c0fe1dae692f4c493413d3939d2e4c21e0da596
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
Summary:
CachedParsedFileMonitor provides cached access to an object created by parsing
a data file. The object can be accessed through "getFileContents().
"getFileContents()" will check if the file has changed and reload/parse if
necessary. A throttle is applied to limit change checks to at most to 1
per throttleMilliSeconds.
Reviewed By: simpkins
Differential Revision: D8903257
fbshipit-source-id: 7ed28610a3e9b40b98b0a9ffaff7b307bbcd32a8
Summary:
This commit improves the FileChangeMonitor interfaces. Changes are:
- on-demand check initiates a call-back
- FileChangeMonitor does not rely on user to call-back to update stat
- updated tests for new api
- fixed issue for determining if file changed (on error)
Reviewed By: simpkins
Differential Revision: D8989042
fbshipit-source-id: cf7a03b2bc8679891c8b9c22a24eed980eeb4060
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use Future<T>::thenTry or Future<T>::thenValue.
The goal is to disambiguate folly::Future and to improve type and lifetime safety of Future and its methods.
3/n: Codemod rvalue-future<T>.then(callable with operator()(not-a-try)) to rvalue-future<T>.thenValue(callable with operator()(not-a-try)).
Reviewed By: yfeldblum
Differential Revision: D8986716
fbshipit-source-id: 906339d9ffb90b3c38a24ce8bf0cef7be318d946
Summary:
Overall plan to modify Future<T>::then to be r-value qualified and use
Future<T>::thenTry or Future<T>::thenValue.
1/n: Codemod rvalue-future<T>.then(callable with operator()(Try<T>)) to rvalue-future<T>.thenTry(callable with operator()(Try<T>)).
Reviewed By: simpkins
Differential Revision: D8961903
fbshipit-source-id: ff17b7833d240c221197cdf0bf914b8a39f80b07
Summary: Update type information so that `mypy --strict` no longer has any warnings.
Reviewed By: strager
Differential Revision: D8940414
fbshipit-source-id: e55533ad5c7c67bd598fae2ae57e6a8d84b47b49
Summary:
Fix the arguments to `doctor.cure_what_ails_you()`, which were updated in
D8870494.
Reviewed By: chadaustin
Differential Revision: D8940413
fbshipit-source-id: 21d45036f668c9bae9a2002c360532918eb157a5
Summary:
If the `nuclide-connections` command is not installed
`subprocess.check_output()` throws a `FileNotFound` exception. Previously the
code only caught `CalledProcessError` and `ValueError` exception. This
updates the code to catch any Exception type, just in case some other
unexpected exception type can also be thrown too.
Reviewed By: chadaustin
Differential Revision: D8945774
fbshipit-source-id: 5e6f63981c3ed6695a687b4bfbaac97ea440b537
Summary:
Add a new `eden debug logging` command to show and update the current edenfs
logging settings.
I have deprecated the `eden debug set_logging_command`, but not removed it
entirely yet. It now longer shows up in the `eden debug --help` output, and
if run it prints the `eden debug logging` command that the user should run
instead.
Reviewed By: chadaustin
Differential Revision: D8907310
fbshipit-source-id: 7ed056637b236db5a50c619023bd8025b787e227
Summary:
This returns a `Repository` object and not a `Repository` subclass type.
This type information has been broken since D7512320 replaced the
`get_repo_class()` method with `create_repo()`.
Reviewed By: strager
Differential Revision: D8940415
fbshipit-source-id: 7711a2a33bc12fc361dfd0ff3f47eba1e587e6e7
Summary:
Add the FileChangeMonitor class to monitor for a file for changes. It provides
an interface "isChanged()" to check if the file has changed. It limits checks
by doing them on demand and by throttling. The throtte limits checks to 1 per
throttleMilliSeconds. It uses Stat information to determine if the file has
changed.
Reviewed By: simpkins
Differential Revision: D8898783
fbshipit-source-id: 067781489a9a91e5fe77cb613fba9d35b8348548
Summary:
InodePath is difficult to work with because it's just an alias to std::array. Make InodePath look more like std::string so using InodePath instances is more intuitive.
This diff should not change behaviour.
Reviewed By: chadaustin
Differential Revision: D8923073
fbshipit-source-id: 85a2548f0cfa61e50b6590048084076b9bece3da
Summary:
Make use of the TopLevelIgnores class to hold the system and user
GitIgnoreStack details. This is a cleaner implementation making ownership
semantics more intuitive. In later commits we will provide acess to the
TopLevelIgnores as part of the ServerState. It will be dynamically loaded.
Reviewed By: simpkins
Differential Revision: D8906226
fbshipit-source-id: d955436582498861ac4b4113a47f357432c8f32e
Summary:
Update `EdenFS.run_unchecked()` to return the full
`subprocess.CompletedProcess` argument rather than just the return code.
This will make it possible for future tests to check the output of the command
in addition to just its return code.
Reviewed By: strager
Differential Revision: D8905491
fbshipit-source-id: 7f17f5d103a06aeb56a3be0618683bd1c017fdf0
Summary:
D8508489 updated the integration test code so that the `capture_output`
argument to `EdenFS.run_cmd()` was no longer necessary and was not used.
However I forgot to actually remove the argument in that diff. This deletes
the argument.
Reviewed By: strager
Differential Revision: D8905492
fbshipit-source-id: d7a691f7d628a9fd5daa6723575e4454480f6aa1
Summary:
As of D8631281 `eden clone` now makes the mount point directory read-only and
puts a readme file inside of it. Since the directory is read-only the
integration tests are unable to clean it up using a normal `shutil.rmtree()`
call.
This fixes the integration tests to try making the directory writable first
if it encounters an error during cleanup.
Reviewed By: strager
Differential Revision: D8905488
fbshipit-source-id: c6b9d0b12c2b1dcb4a0839757cbcc3aad179bfc1
Summary:
This replaces the `StdoutPrinter` class with a different `Output` class
interface that combines both the `StdoutPrinter` functionality and the
underlying I/O object being printed to.
Previously the doctor code had to pass around two separate `out` and
`printer` objects. This simplifies things so that only one object has to be
passed around.
This also adds some additional error checking around the curses
initialization, and falls back to a plain output object if anything goes wrong
during setup. Previously we had seen some reports of `eden doctor` failing
due to errors in the terminal setup code.
Reviewed By: strager
Differential Revision: D8870494
fbshipit-source-id: a6dbafdde9aa5e1a0336fa7d723d83fa7d2426ca
Summary:
The raw Inode pointer in a DirEntry is more of an association than
ownership, so add comments and have clearInode return the old value.
Reviewed By: strager
Differential Revision: D8842315
fbshipit-source-id: d401dcdf4955ea335b39c2a57b0bedb1f83fdf9b
Summary:
When users run into "Transport endpoint not connected" errors they sometimes
attempt to run `eden status` or `eden doctor` to fix the problem. However
previously the CLI did not deal with being run from an invalid working
directory very well.
This updates the CLI to explicitly check if the current working directory is
valid before doing anything else. If it gets an ENOTCONN error it prints a
message telling the user how to update their shell's working directory. If
the `$PWD` environment variable points to a valid directory it attempts to
continue using that as the working directory. It records the fact that the
original working directory was invalid so that `eden doctor` can report this
error if that is the command being run.
Reviewed By: wez, strager
Differential Revision: D8731005
fbshipit-source-id: 4be55ccb43e22ed878114268e4dcc4c90c20e8e2
Summary:
Add a class to encapsulate details of top-level system and user ignore files.
This approach has the following benefits:
- simplifies memory management of the GitIgnore;
- separate structure that can be created and provided as part of ServerState
interface;
We will make use of it later in this commit stack.
Reviewed By: simpkins
Differential Revision: D8876064
fbshipit-source-id: 35c82b918c09e58068370401883edd8474dd3fbf
Summary: GitIgnore and GitIgnoreStack encapsulate details of ignore files. Further changes (as part of this commit stack) require copy constructors and assignment operators. Also, compiler wants destructors in .cpp file (since otherwise, it cannot tell the object size).Keeping this as a separate commit to simplify review process.
Reviewed By: simpkins
Differential Revision: D8730299
fbshipit-source-id: 7cb443906143c80209278b85fc5ad4dc5ea0bf56
Summary:
This adds initial code for an `eden debug fsck` command.
At the moment this simply scans the overlay and reports errors, but does not
correct any issues. In subsequent diffs I'll update it to support correcting
some problems.
There are also several other issues that I'll improve in subsequent diffs as
well. At the moment this code uses the same functions as other CLI commands
to find the client directory, and these only work if the checkout is already
mounted. For `fsck` we generally want the checkout to be unmounted before
checking it. I also plan to clean up some of the output handling as well.
Reviewed By: strager
Differential Revision: D8813714
fbshipit-source-id: 90238e94540db1cc06c17eabf8ce3a1bb808d4b3
Summary:
The methods were marked const when they were actually mutating the inner
objects, this was working because the pointed to objects were referenced via
`shared_ptr` handles
Reviewed By: yfeldblum
Differential Revision: D8851578
fbshipit-source-id: 4ca49390daf46f5c8681149da4d1e9eeab5aacb5
Summary:
While trying to make destroyWithInitRace non-flaky, it uncovered an
ASAN violation in the case that fuseCompleteFuture finishes during
EdenMount destruction. In that case, path_ gets destroyed prior to the
executor, so it's illegal to construct TakeoverData::MountInfo from
path_. This diff removes path_ entirely and reads it from
ClientConfig.
Reviewed By: simpkins, strager
Differential Revision: D8848663
fbshipit-source-id: f9368aa9eec7dfa8f2897cce55fad6d19723e30c
Summary:
We should only kick off prefetching for the files
that matched the glob. We were prefetching files that
didn't match the glob.
facepalms
Reviewed By: strager
Differential Revision: D8846994
fbshipit-source-id: 593e85d843ffa1cc0707ed1dc86f1385262821f5
Summary:
This makes it easier to add some test coverage.
There's no real functional change in this diff; the only code change is to
throw a system_error instead of a thrift eden error wrapper class from the core
globbing code. There's a little bit of code to restore this exception type in
the callers in EdenServiceHandler; this is covered by existing integration
tests, but I've also expanded that coverage to cover both variants of the glob
thrift calls.
Reviewed By: strager
Differential Revision: D8776767
fbshipit-source-id: 3ea4ea642ae5108aa4b0153541bd3604f010b54c
Summary:
Let's use consistent naming for the user ignore file. We will go with
'userIgnoreFile' as recomended. There is now symetry since we have
'systemIgnoreFile'.
I have another diff (later) that already uses the 'userIgnoreFile'.
Reviewed By: chadaustin
Differential Revision: D8823906
fbshipit-source-id: 47594e8971fa4db809821fc819da05e9eb2c5277
