Summary:
Add a new class for injecting faults into Eden. This will make it easier to
build tests that exercise specific error conditions or specifically control
the ordering of operations that might otherwise race.
To minimize the performance impact, the fault injection framework is disabled
by default and must be explicitly enabled at process startup using a command
line flag. If this command line flag was not specified all fault injection
checks short-circuit into a no-op. The `checkAsync()` version does still add
some extra overhead due to the addition of a new `folly::SemiFuture` into the
code path.
Reviewed By: wez
Differential Revision: D14079489
fbshipit-source-id: 3e8725d2e51e5c829199a602e90574424ea52636
Summary:
If PrivHelper::fuseMount fails with an error, EdenMount::getState returns STARTING. This is confusing. Make EdenMount::getState instead return FUSE_ERROR in this case.
Aside from changing the output of 'eden list' in some cases, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14060982
fbshipit-source-id: ae9c2c532b1654cce36b806971053d89231058c3
Summary:
While moving code around in EdenMount::startFuse, EdenMount::shutdown, and EdenMount::destroy, I accidentally changed the result of EdenMount::getState in some circumstances. I noticed that we don't have any tests for EdenMount::getState, so it could have been broken even before my changes!
Write some tests for EdenMount::getState to verify the existing behavior and to prevent future regressions. Only add tests which pass with the current implementation; future diffs will fixe cases where EdenMount::getState returns the wrong value.
Reviewed By: chadaustin
Differential Revision: D14058334
fbshipit-source-id: 68b60ac92f94d336fcfb8f11e571379573bb3f42
Summary:
Some tests want to create a TestMount but don't care what files are in the mount. The following code is intuitively correct. However, TestMount's constructor takes a FakeTreeBuilder by non-const reference, so it fails to compile:
auto testMount = TestMount{FakeTreeBuilder{}};
The following code works, but more verbose:
auto builder = FakeTreeBuilder{};
auto testMount = TestMount{builder};
Overload TestMount's constructor to allow the former form in addition to the latter form. This neatly shrinks some test code.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14060110
fbshipit-source-id: c6a5d4b2c5859812efff279dedbc1fe690c8f8ad
Summary:
Make it easier to write tests involving a FakeFuse-mounted EdenMount by creating the TestMount::startFuseAndWait function which initializes EdenMount.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14059808
fbshipit-source-id: ce63bbce7f2c6e710910ad530aa1b4efefe80272
Summary:
I want to test EdenMount::startFuse when PrivHelper::fuseMount fails. FakePrivHelper makes this inconvenient. Extend FakePrivHelper so a test can execute arbitrary code for fuseMount calls.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14058663
fbshipit-source-id: 27ff32a75adde8aff952ac50b3f1977cf0b59e93
Summary: It's been a few weeks since we dropped support for `eden config --get` (D13737807), and we've received no reports of this being a problem. Delete the code which handles `eden config --get`.
Reviewed By: chadaustin
Differential Revision: D13772873
fbshipit-source-id: 04e2aa7f193806ee26c2f0ab1fe4246815c581a6
Summary:
This got lost partly because the symlink/shipit bit
was a manual step when I did this first time around, and partly
because the mercurial code got refactored.
Restoring treemanifest restores `hg up` performance to saner
levels.
This is fairly straightforward, but now that we have !wez building this we've
found that my machine had lz4 installed in the default include path and that we
need to probe and find the correct one, so that is part of this diff also.
Reviewed By: simpkins
Differential Revision: D14078886
fbshipit-source-id: 971cd49757bbb7dcc484439e6d896698598a583b
Summary:
Now that `hg debugedenimporthelper` has been released for
a little while, we can remove the bundled implementation of it from
the eden release.
However, we cannot remove the script itself as there are users
with long running edenfs instances that pre-date the knowledge
of `hg debugedenimporthelper`. So, as a compatibility shim,
this diff redirects `hg_import_helper.py` and has it exec the
command in mercurial.
For extra fun, our own integration tests rely on being able
to import `hg_import_helper.py` and override portions of it,
so we cannot remove its implementation from the codebase just
yet either.
So this diff:
* Introduces `proxy_import_helper.py` which execs `hg debugedenimporthelper`
* Installs `proxy_import_helper.py` as `hg_import_helper.py` in the rpm
* Leave `hg_import_helper.py` as-is in the tree for now.
Reviewed By: simpkins
Differential Revision: D13970332
fbshipit-source-id: 717dc86a880fbbbe4a7e801a8b748abd053c7f7c
Summary:
in D13970332 we want to allow using `hg.real` to run the
import helper that is embedded into mercurial, but we don't know
the full path, so we need to resolve it from the environment.
`folly::Subprocess` has `usePath` option, but it is not compatible
with also overriding the environment for a child process.
The easiest short term thing to do is to resolve the path ourselves
using this simple function call.
I've also added the path to `hg.real` that is used on our mac systems
to the PATH we pass down to `edenfs` when starting eden.
Longer term we want to better encapsulate how we resolve and start
the import helper, but for now this unblocks starting eden on the
mac.
Reviewed By: strager
Differential Revision: D14078612
fbshipit-source-id: bc6787bbe99cd87224b783d2fb9d3255b825b976
Summary:
Differ.h was including an overly-broad header when all it needed was
the type definitions. This makes building the inodes target with
mode/mac easier.
Reviewed By: simpkins, strager
Differential Revision: D14036477
fbshipit-source-id: 99f3a55d0523c179f6cf2aa4210119ae6f8e51f8
Summary:
I don't have RocksDB building in mode/mac yet, and the inode code
only depends on the store interface, not all of the specific
implementations.
Reviewed By: simpkins, strager
Differential Revision: D14036390
fbshipit-source-id: d0b8d5c4d4f2f38dbc9bc2476c24ac99398d37b7
Summary:
Some tests set up fuse_init_in manually then call FakeFuse::sendRequest. The specific fuse_init_in setup is benign and already exists in FakeFuse::sendInitRequest. Shorten these tests by making them call FakeFuse::sendInitRequest instead.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14059700
fbshipit-source-id: 210dcfdc4ae1c74e10c50dbc3f3044c3b908da0a
Summary:
Calls to ObjectStore::create and EdenMount::create are duplicated in the various TestMount::initialize overloads. Fix the duplication by moving the two calls into their own function.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D14058972
fbshipit-source-id: b3e79756793292b59f09f53c05649a637c8af07e
Summary:
getInitialMode and getModeUnsafe were not inherently different, so
remove getModeUnsafe and write an accurate comment in Overlay's
serialization code about why the initial mode is saved and how we
could remove that logic in the future.
Reviewed By: strager
Differential Revision: D14007488
fbshipit-source-id: db42f45f00dcd213fabd9575360da1261931778b
Summary: Branches are going away. Remove the use of them.
Reviewed By: strager
Differential Revision: D14062107
fbshipit-source-id: 00f6d3666eb3cb6900cd570fa3fcf12ba75c2ae0
Summary:
This fixes a bug where "eden doctor" would incorrectly report that some
checkouts were not mounted when the version of `edenfs` was too old. It would
think it needed to remount the checkouts even though they were in fact working
correctly.
Older versions of `edenfs` did not return a `state` field in the
`listMounts()` reply. This caused the newer `eden doctor` logic in D13526754
to incorrectly think the mounts were not running. This fixes the code to
assume the state is `RUNNING` in this case.
Reviewed By: wez
Differential Revision: D14017363
fbshipit-source-id: 020135f386aad84b1a92f2699a2706ba73141ec7
Summary:
It's common for code to use InodeNumber without needing to include the
main FUSE headers or vice versa. Split them into two separate headers.
Reviewed By: strager
Differential Revision: D13979868
fbshipit-source-id: c5eeb6a3697bb538729a403434dc4f0f7408cda0
Summary:
Previously the default ignore path was `~/ignore`. This doesn't really seem
like a great path choice, and I suspect that no one is actually using an
ignore file at this location. This default was set in D8594297, but I it
looks like this was mostly accidental from just not separating out data for
the system ignore path (`/etc/eden/ignore`) vs the user ignore path.
Using `~/.edenignore` seems reasonable for now, and unlikely to conflict with
other paths. We used to use `~/.gitignore` as the default before D8906226,
but this did cause problems for a handful of users that treated their entire
home directory as a git repository, and had a `~/.gitignore` file that was
supposed to apply only to their home directory.
Reviewed By: wez
Differential Revision: D13984153
fbshipit-source-id: 887528372b9be789317933f7026dfcbde8cd4539
Summary:
Update FileChangeMonitor to stop logging messages about files not being
present. We normally use FileChangeMonitor to monitor the user's `edenrc`
configuration files and their personal and system-wide ignore files. It is
normal for these files to not exist in many cases, so there is no need to log
a warning message in this case.
Reviewed By: chadaustin
Differential Revision: D13981278
fbshipit-source-id: fa5f8d42980fd8683f55d3d51f5375a72f511dfe
Summary:
Update the `check_hg()` code to verify all required files in the `.hg/`
directory, and recreate them if necessary.
Reviewed By: chadaustin
Differential Revision: D13935005
fbshipit-source-id: aceb1aebcc54d039cb0a6a19a44e867424f15bf0
Summary:
This renames the CheckoutConfig `path` member variable to `backing_repo`.
The value in this variable is actually the path to the source control backing
repository, and not the path to the Eden checkout, which was confusing. (The
CLI tests were actually setting this up incorrectly in `create_test_mount()`,
but until now nothing in the tests had been using it, so we hadn't noticed
this problem earlier.)
I also changed `CheckoutConfig` to be a `typing.NamedTuple` as part of this
change.
Reviewed By: strager
Differential Revision: D13935006
fbshipit-source-id: 43b6ba4efa34f8c6fb8264de8bb31a33a6d73036
Summary:
Update the code in `hg_util` to take advantage of the `EdenCheckout` methods
for getting config data rather than passing around an `EdenInstance` and a
config object separately.
Reviewed By: strager
Differential Revision: D13935007
fbshipit-source-id: 7c4079fa9b6c1a56f15006f44da994bf69b0f050
Summary:
Make `build-oss.sh` actually run the cmake build.
At the moment this is only run from nbtd for mac platforms,
so that is all that I've tested with it.
This is similar to the the equivalent scripts for watchman.
In a later diff we can centralize and clean up the duplication.
Reviewed By: simpkins
Differential Revision: D13910031
fbshipit-source-id: 6a7250c22033e913d999cf928fed27dada0647ef
Summary:
This diff enables building the eden watcher by linking in thrift and
its various dependencies.
To support building in-fbsource and in the github repo, a `maybe_shipit_dir`
function is used to setup a symlink to the `eden` and `fboss/common` dirs (this
mirrors the shipit configuration for this project: we cannot simply run shipit
because we have to build on mac and windows and shipit requires Hack, and that
does not support those platforms).
I tried to persuade cmake to let me build this without the use of a symlink but
found it too difficult to teach everything about the path mapping. The
symlinks aren't terrible, but are the reason why this diff also updates some
`.gitinore` files that are seemingly unrelated to this diff.
This diff changes a couple of build/link options: without them the end product
fails to link either due to implicit/unilateral enablement of UBSAN in some of
the deps, or because warning->error promotion is turned on.
This diff includes a copy of the `ThriftCppLibrary.cmake` file from the fboss
repo. This should get centralized and shipit'ed out into the places that
consume it. That can be done when someone gets around to doing the same for
the `FindGlog.cmake` file and doesn't need to hold up this diff.
Reviewed By: simpkins
Differential Revision: D13486486
fbshipit-source-id: 3bb5b011771b2a87618147ca019b4e50a8e0aaf2
Summary:
Move code for loading and saving checkout configs and snapshot files from
EdenInstance to EdenCheckout.
Reviewed By: chadaustin
Differential Revision: D13932141
fbshipit-source-id: aa5cf56ed6f2ac7d3e5529eee353e1dd0ea12a21
Summary:
Refactor the checks that verify the contents of `.hg/dirstate`, in preparation
for teaching `eden doctor` how to repair the other required files in the
`.hg/` directory.
This creates a new `HgChecker` class, which I plan to use in the future for
adding checks for each specific file that needs to be checked. This also
consolidates all issues with the .hg directory into a new `HgDirectoryError`
class. This is mainly to keep the error reporting simpler: if the entire .hg
directory is missing we will report this as a single problem rather than a
separate problem for each file in the .hg directory that we will recreate.
Reviewed By: chadaustin
Differential Revision: D13928667
fbshipit-source-id: fb78f014d44e85b9ac07048cbe641e9e8be4ed4c
Summary:
Update the `create_test_mount()` helper function to return an `EdenCheckout`
object instead of simply the mount path as a string. This will make it easier
to write some upcoming tests that need a full `EdenCheckout` object.
Since `EdenCheckout` stores the mount path as a `pathlib.Path` instead of as a
plain `str`, this also updates a few more locations in the test code to use
`Path`.
Reviewed By: chadaustin
Differential Revision: D13928666
fbshipit-source-id: 585f27c344e2d0283bf5f8562e17ac3fd356c539
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: yfeldblum
Differential Revision: D13908257
fbshipit-source-id: 8b5315b019290f1c60087ca5716c31ebbf1f1be5
Summary: FuseTypes.h defines the InodeNumber struct, but also has a comment which implies that InodeNumber is an alias of uint64_t. Delete this misleading commented-out definition of InodeNumber.
Reviewed By: chadaustin
Differential Revision: D13892907
fbshipit-source-id: 8e9710db29d8e0d708ee1785956f6953a0b2a49f
Summary:
In the help text, it is clearer that a subcommand is required if you set the
metavar placeholder to `COMMAND`.
Reviewed By: chadaustin, strager
Differential Revision: D13888877
fbshipit-source-id: 1a5b9ef4a382dd202b6279e99ca837fdd39e47af
Summary:
When debugging Eden's behavior under a kernel that caches readdir
calls, I observed that directories were not returning the correct
entries after a checkout. The issue is that Eden calls INVAL_ENTRY
when new entries are added, but INVAL_ENTRY is a no-op and returns
ENOENT if the kernel doesn't know about that entry.
Reviewed By: strager
Differential Revision: D13864281
fbshipit-source-id: 56b3a67d0bc6f8ed1733acaf5e889414b753cbc7
Summary:
Grab the latest kernel headers from an upcoming kernel (fuse/for-next,
5a86a6974d5289763af7be79e3fcfc697f034e21). The specific header
is from fuse/for-next so we can opt Eden into the
FUSE_NO_OPENDIR_SUPPORT flag.
Reviewed By: strager
Differential Revision: D13853062
fbshipit-source-id: 1bacb17c63a170975deb61d5dba01e2aded0703a
Summary: `hgext` was moved to `edenscm.hgext`.
Reviewed By: simpkins
Differential Revision: D13884629
fbshipit-source-id: 4bf69720cccbd43665cd350ac8a3ba4d46b10b93
Summary:
Replace Future::onError with Future::thenError:
* to remove ambiguous typing
* to ensure that the executor is not lost and the returned Future is still bound to an executor
See:
https://fb.workplace.com/groups/fbcode/permalink/2002251863144976/
for details.
Reviewed By: yfeldblum
Differential Revision: D13784772
fbshipit-source-id: 1d3ede848b7d31c7a197a21b4ff2b31e840040a5
Summary:
D13853115 adds `edenscm/` to `sys.path` and code still uses `import mercurial`.
That has nasty problems if both `import mercurial` and
`import edenscm.mercurial` are used, because Python would think `mercurial.foo`
and `edenscm.mercurial.foo` are different modules so code like
`try: ... except mercurial.error.Foo: ...`, or `isinstance(x, mercurial.foo.Bar)`
would fail to handle the `edenscm.mercurial` version. There are also some
module-level states (ex. `extensions._extensions`) that would cause trouble if
they have multiple versions in a single process.
Change imports to use the `edenscm` so ideally the `mercurial` is no longer
imported at all. Add checks in extensions.py to catch unexpected extensions
importing modules from the old (wrong) locations when running tests.
Reviewed By: phillco
Differential Revision: D13868981
fbshipit-source-id: f4e2513766957fd81d85407994f7521a08e4de48
Summary:
Extend EdenConfigParser to read string arrays from config files. TOML already supports arrays; this diff just makes the arrays accessible to CLI code as Python sequences.
Currently, no code needs to read string arrays. This feature will be used in the future for the service.start_systemd_manager_command config option:
[service]
start_systemd_manager_command = ['sudo', 'systemctl', 'start', 'user@${USER}.service']
Reviewed By: simpkins
Differential Revision: D13708911
fbshipit-source-id: 74e9e03ded9896a84a28611dc1df589d8fcb7597
Summary:
Added functions to Eden Doctor to check for insufficient disk space available on the Eden mount points used by all checkouts.
Relevant Eden directories are retrieved automatically and the mount point is checked for each folder. Thus, it only reports excessive disk usage at most once per mount point.
All this functionality has also been factored out into the check_filesystems.py file (former check_using_nfs.p) to keep similar functionality in the same module.
The 90% threshold is hardcoded for now but could also be made customizable via Eden's config (e.g. EdenInstance.get_config_value and /etc/eden/config.d/doctor.toml) and triggers an advice problem, unless the total free disk space is below 1GB on any mount point of non-zero size.
Also added state_dir to FakeEdenInstance to fix bug in test cases which accesses state_dir and attached test cases for the problem generation logic due to disk usage.
Reviewed By: strager
Differential Revision: D13732678
fbshipit-source-id: a1bbb8ffa0bfdc816611dfd628bed62485e54294
Summary:
The service.experimental_systemd config option is implemented hackily: the value is parsed as a boolean, then converted to a string (with `str`), then compared with `"True"`. Remove this ugliness by not converting all config options to strings in EdenConfigParser, and adding a type-safe `get_bool` accessor.
Aside from disallowing `experimental_systemd = "True"` in configs and changing the output of `eden config`, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13666331
fbshipit-source-id: 65b7d233c43db08bc7e0288f248041ca4a18a5fd
Summary:
Move top-level Python packages `mercurial`, `hgext` and `hgdemandimport` to
a new top-level package `edenscm`. This allows the Python packages provided by
the upstream Mercurial to be installed side-by-side.
To maintain compatibility, `edenscm/` gets added to `sys.path` in
`mercurial/__init__.py`.
Reviewed By: phillco, ikostia
Differential Revision: D13853115
fbshipit-source-id: b296b0673dc54c61ef6a591ebc687057ff53b22e
Summary:
This adds a new `CheckoutInfo` class in the `eden doctor` code to consolidates
some information about a particular checkout. This should make it easier to
pass around this single object to various check functions, instead of several
separate arguments.
I did make a few functional changes as part of this refactoring:
- All checkouts are now checked in order sorted by their mount path.
Previously non-mounted checkouts were checked before running checkouts.
- The code now reports additional errors about checkouts if they are running
but not listed in the on-disk configuration, or if their running state
differs from the on-disk configuration in some cases.
Reviewed By: strager
Differential Revision: D13526754
fbshipit-source-id: 81802f96a3a3c9f2aaad9a4468fb666ed8b2e47e
Summary:
Update "eden doctor" to just check if its main state directory is on NFS
rather than checking if the state directory for each checkout is on NFS.
It's quite unlikely that Eden's state directory spans multiple devices on both
NFS and non-NFS, given that Eden itself generally creates this directory
structure. Therefore it seems unnecessary to individually check different
checkout subdirectories inside this structure. We also should report a
problem if the state directory is on NFS even if there currently aren't any
checkouts configured.
This also provides remediation text telling users the most likely cause for
having their state directory on NFS, and how they can probably fix it.
Reviewed By: wez
Differential Revision: D13831086
fbshipit-source-id: f9f243898fbfe3dcb581678a6ec8a63a29855c5e
Summary:
In preparation for proper support for booleans and arrays in configs, refactor the interface of `EdenConfigParser`:
* Rename `get` to `get_str` so callers explicitly mention the expected type. This change will make sense in the future when `EdenConfigParser` supports boolean and array options in addition to string options. Also, rename the `fallback` parameter to `default`.
* Move `config_to_raw_dict` into `EdenConfigParser` and remove the `raw` boolean parameter from `items`.
* Rename `items` to `get_section_str_to_str`, and change its return type to `Mapping` (instead of `List[Tuple]`).
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13661228
fbshipit-source-id: fc93560f95b75ade78044d8146e92c6f65dfc404
Summary:
Python's `configparser` module requires values to be strings. Values in TOML can have other types, such as boolean and array, and `configparser` gets in the way of parsing such values.
Allow fixing `configparser`'s limitations by rewriting parts of it. Instead of using `configparser.ConfigParser`, create an `EdenConfigParser` class (which implements a subset of `configparser.ConfigParser`'s API) and use that instead.
This diff should not change behavior. A future diff will enable non-string config values.
Reviewed By: simpkins
Differential Revision: D13646873
fbshipit-source-id: e26a886789cf00dc363f3a97fe0febe6f5010f8e
Summary: using upgrade script to clear out all remaining version-set configs
Reviewed By: dark
Differential Revision: D13832474
fbshipit-source-id: 52c280cbd79b1410821ed829465b1c0907b50a86
Summary: PrivHelperServer::initLogging is declared in the .h file, but it's never defined. Remove the useless declaration.
Reviewed By: simpkins
Differential Revision: D13814416
fbshipit-source-id: 25cb47442a19947da08d13d9bed9b4631a1c9739
Summary:
`EdenInstance.get_config_value` raises a KeyError if the option or section is not set in any config files. Every caller of `EdenInstance.get_config_value` handles KeyError manually. Simplify code by letting callers specify a default value.
Aside from treating some options with an empty value as if they were unset, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13737806
fbshipit-source-id: fdb7fa75d601de4644704813db38b671b27f73d7
Summary:
Update the `eden list` command to also report the current state for each
checkout if it is not running normally. Also added a `--json` flag to
print information as JSON so it can be consumed programmatically.
Reviewed By: strager
Differential Revision: D13503053
fbshipit-source-id: 4ef366f5bf4a1157036fdfd7ff1056079588e802
Summary:
The code that parses the kernel version requires a fairly strict
structure and not all kernels match that structure. If the number
doesn't parse, skip the checks.
Reviewed By: wez
Differential Revision: D13824190
fbshipit-source-id: 8eb2ea2778e1e470d7f7708fc256dd5fae5a02b4
