Summary: adds a cli debug command to inspect the working copy parent. by default just returns eden's snapshot contents, but adds optional --hg flag to print Mercurial's dirstate information
Reviewed By: chadaustin
Differential Revision: D19167518
fbshipit-source-id: b65e112df6abe4e0e7a8a528a90b2e3d17297e66
Summary:
Remove some half-baked, unnecessary logic for caching sizes separately
from SHA-1. Eden's backing stores do not support chunking large files
yet, so there's no value in caching content SHA-1 and size
separately. This fixes a scenario where fetching blob size and then
SHA-1 would result in two backing store imports.
Reviewed By: fanzeyi
Differential Revision: D19169096
fbshipit-source-id: dc32f3313e5f4230c06a5bbaa67da7bf0febaba8
Summary: There is one instace of `getScmStatusBetweenRevisions` in use - it is used in the eden cli in a hacky way to check if a commit hash is valid. Since this is not used anywhere else in a meaningful way, this replaces that use case with a hg call and depreciates `getScmStatusBetweenRevisions`
Reviewed By: simpkins
Differential Revision: D18690026
fbshipit-source-id: 02bd2c20a0f631ec41116f9fd4e18d14369298ef
Summary:
A spike in automatic GCs usually implies something has gone wrong. Log
an event for each one, recording the cache size prior to the GC and
the cache size after.
Reviewed By: simpkins
Differential Revision: D18902580
fbshipit-source-id: 158b2635733a415a9fcc7c412b2c0f44ed04aa01
Summary:
Two bugs conspired to cause edenfs after a graceful restart to think
the kernel supported FUSE_NO_OPENDIR_SUPPORT when it didn't: the
connection info struct wasn't zeroed, and FUSE connection capabilities
weren't properly mirrored into the Dispatcher upon graceful
restart. Fix both and add an integration test.
Reviewed By: simpkins
Differential Revision: D18903761
fbshipit-source-id: 23f4db3e240ee7d035f707820072c606a45f1138
Summary: This updates the hg and telemetry wrapper callsites of getScmStatus to first try running getScmStatusV2() with fallback option. This does not retry `hg status` while a checkout is in progress.
Reviewed By: simpkins
Differential Revision: D18209899
fbshipit-source-id: e7a77b902f5a0ee624e4ea3185a1901bdac090e6
Summary:
Now that we've transitioned to the newer redirections
configuration we can remove the older bind mounts configuration
parsing, and that is what this diff does.
This is helpful because in situations where the user has run
`hg co null` as part of some troubleshooting advice, the legacy
bind mounts would remain mounted and obstruct some of the
steps that would have fixed up the users repo.
Reviewed By: pkaush
Differential Revision: D18337246
fbshipit-source-id: 23f27787d609e1c38a9c98b8b6596bb40743b9ca
Summary:
The purpose of this command is to unmount/unlink any configured
redirections without removing their configuration.
The intent is to call this for a mount when we are unmounting; I'll do
that bit in a follow on diff.
Reviewed By: pkaush
Differential Revision: D18801872
fbshipit-source-id: 096d9595091da72aa85f4259cbab022a1fe0c01f
Summary: Even if blobs have different hashes, they could have the same contents. For example, if between the two revisions being compared, if a file was changed and then later reverted. In that case, the contents would be the same but the blobs would have different hashes. Currently, `getScmStatusBetweenRevisions()` would report false positives in this case. This is also needed so we do not report false positives in `getScmStatus()` when hit this code path
Reviewed By: simpkins
Differential Revision: D18647086
fbshipit-source-id: 66e12648a24fd7e5612eee5e599a5b81c7c5f2d1
Summary: This reads `enforceParents` from a config instead of always assuming true for `getScmStatusV2()`. This will allow a easy kill switch in case throwing errors from this thrift call causes issues with something that calls hg status
Reviewed By: simpkins
Differential Revision: D18258164
fbshipit-source-id: 1ae421a941c01a678d25d5453c771262b03558d0
Summary: make the error message returned in the case of out of date parents during a new status call more user friendly and provide possible remediation instructions
Reviewed By: simpkins
Differential Revision: D18328835
fbshipit-source-id: b214f45bb055d008db8b233ddd2a1843332db838
Summary:
Merge the fb-mercurial code into the Eden repository, under the
`eden/scm` subdirectory.
Reviewed By: quark-zju
Differential Revision: D18445774
fbshipit-source-id: fc3307f9937e0c7e1c8f7d03c5102c4fe5dedb10
Summary:
Add a new thrift API for computing the difference between the working
directory and a given source control commit.
This has the following differences from the old getScmStatus() commit:
- The parameters are accepted in a GetScmStatusParams structure now.
This makes it easier for the server-side C++ implementation to tell which
parameters have actually been specified by the caller. This will make it
easier to extend this API in the future without having to replace it with a
new function call again.
- The return value is a GetScmStatusResult, which includes both the ScmStatus
and the EdenFS version number. This will allow code like `hg status` to get
both the status results and the EdenFS version in a single call, without
needing to make multiple separate thrift calls.
- This new call will return an error if the caller requests the status against
a commit that disagrees with EdenFS's view of the current commit. Because
the individual `hg` command line processes do not perform any
synchronization of their own when reading the working directory parent,
they can often call EdenFS with stale parent information, or while a
checkout is currently in progress. This new behavior will reject the
request with an error, rather than having EdenFS perform a potentially very
expensive status computation when the results probably aren't actually
useful to the caller anyway.
Reviewed By: chadaustin
Differential Revision: D15110218
fbshipit-source-id: ebc2f74dafc090d4fd245de8e4d62e2b086500dd
Summary: Adds a non-optional EdenErrorType struct for EdenError, this can be used in case of special error case handling of errors without error message parsing. Currently this is just passed along and not consumed anywhere in the client, but later in the stack is used for specific retry of checkout if "CHECKOUT_IN_PROGRESS" on the consuming side.
Reviewed By: chadaustin
Differential Revision: D18139917
fbshipit-source-id: b3f2ec4c480fc5246ff2f46d09c436021bad8b61
Summary: Fastmanifest is going away, remove it from the test.
Reviewed By: chadaustin
Differential Revision: D18145000
fbshipit-source-id: ee75ebe4eda19caca92fd0a84bf0ae9f48112167
Summary: Formatting had diverged in a few places. Fix that up.
Reviewed By: fanzeyi
Differential Revision: D18123219
fbshipit-source-id: 832cdd70789642f665a029196998928a9173be81
Summary:
A recent change removed the revision number from the rebase output, fix the
EdenFS test to not have it too.
Reviewed By: genevievehelsel
Differential Revision: D17954310
fbshipit-source-id: 6c1db48086af4b7b138e6c3f4ef0bb362d2256f8
Summary:
Instead of having accessors for every config setting in EdenConfig,
just expose the ConfigSettings directly.
Reviewed By: fanzeyi
Differential Revision: D17847805
fbshipit-source-id: 8c6c1010c010113cf859677449797ea916f2a2a5
Summary:
Some Unix applications (notably, nfsd) create regular files using vfs_create, which ends up invoking the `mknod` system call rather than `open`, which for historical reasons only supported socket creation with Eden. However, since Eden supports regular files, we can broaden the FUSE mknod handler to support regular files as well.
For context, see https://github.com/GoogleCloudPlatform/gcsfuse/issues/137#issuecomment-155273363
Reviewed By: chadaustin
Differential Revision: D17792424
fbshipit-source-id: 466fcbcb3bcb587e731bc8b2a3e0f1508ff1f4e4
Summary: D17766371 added an `update --merge` state. Teach eden tests about it.
Reviewed By: wez
Differential Revision: D17837836
fbshipit-source-id: a95ed326bf435f7340d7910307c8c5c761812514
Summary:
## Backstory
Pyre was throwing errors in my diff (D17747558) regarding an extraneous fixme. Turns out PyreBot has been adding and removing these fixmes during version updates (see D17135557 and D16183608), so I suspect it's something to do with the Pyre version. Anyways, I figured it'd be easier to do the annotation than to remove the fixme and risk Pyre throwing the same error in a later diff.
## What I did
I added the Key type (Path) and Value type(ExpectedFileBase). mypy then started throwing an error regarding __iter__ returning the wrong type because it wanted an iterator over keys rather than values. Fixed that and added .values() to the for loop.
Reviewed By: genevievehelsel
Differential Revision: D17806135
fbshipit-source-id: c07feee33db78a9bff19ba9856a7047657b8c63e
Summary:
Update the CMakeLists.txt files to support building the hg integration tests.
At the moment this only includes one of the test files (`status_test.py`).
I have not verified if tests from the other modules pass yet or if they need
any additional tweaks to work in CMake-based builds.
Reviewed By: pkaush, fanzeyi
Differential Revision: D17678991
fbshipit-source-id: 4a5ee5a8d6039d9d2a635c7027897bbeed14f8c0
Summary:
Add initial support for building and running some of the integration tests
with CMake. For now this just runs the tests from basic_test.py, just to
confirm that most of the framework code works in CMake-based builds.
Many of the other tests should also work as well, but a few of them we may
want to disable for CMake-based builds. e.g., a couple of the tests depend on
hypothesis, and we would need to include hypothesis as a dependency. Some of
the tests that use systemd might also require a little more work to get
working.
Reviewed By: fanzeyi
Differential Revision: D17659026
fbshipit-source-id: 67420fda9e1021a0cddee2d385fd21e34fb2fd70
Summary:
Force a reference to the edenfsctlPath flag, otherwise the
linker will discard it and a large number of tests will fail.
Reviewed By: simpkins
Differential Revision: D17683222
fbshipit-source-id: b7cb29e74af85b544f45a228770ad2613c8e6efc
Summary:
This diff removes the logic that consumes the legacy bind
mount list and mounts them on startup. That functionality has been
replaced with the eden redirect command.
Instead of performing the bind mounts in the server, the server will
now run `eden redirect fixup` to apply that configuration.
This diff also changes the behavior of performBindMounts: previously, if the
bind mount setup failed, we would tear down the entire repo mount. Since we're
now spawning an external process, it is much more likely that something might
fail and result in a bad experience, so we no longer bail out in that case:
we'll continue and leave the bind mounts as-is. The user can then use `eden
doctor` or `eden redirect fixup` to sort things out.
Reviewed By: simpkins
Differential Revision: D17236366
fbshipit-source-id: 8b004551a076216f0e5448942f00b5195ee18803
Summary:
Change the `//scm/hg:hg` target to use an `sh_binary()` rule that invokes the
`:hg_rust` binary with the proper environment so it can find its dependencies,
rather than copying the binary and all of its dependencies into a new
subdirectory.
In dev mode builds the `hg_rust` binary isn't guaranteed to work anywhere
other than its original location, due to the way that dev mode builds use
`$ORIGIN` in the binary's `RPATH` setting. This happened to work up until now
as the hg_rust binary did not have any separate libraries, but I plan to add
one on the `chg` library.
Reviewed By: quark-zju
Differential Revision: D17109104
fbshipit-source-id: ae8bb1126969f012d1d2fb7d04e80867a310b9a8
Summary:
Add a flag to cause `eden start` to exit successfully without doing anything
if EdenFS is already running. This flag makes it slightly easier for
automation to ensure that EdenFS is running, without logging warnings if
EdenFS was already running.
I also cleaned up the error message slightly when `eden start` is used
without this flag and fails if EdenFS was already running. Previously the
exception thrown was unhandled so it also printed a python backtrace. Now the
code throws an exception that is caught by the higher level command line code,
so it is printed in a more user-friendly way.
Reviewed By: wez
Differential Revision: D17440486
fbshipit-source-id: d7661ef7be7159bf5542b20e99a0b5495690e5a2
Summary: This makes it a bit more human friendly
Reviewed By: chadaustin
Differential Revision: D17249465
fbshipit-source-id: 40d5afc77ded34237e1860d5b91e9257a732e480
Summary:
D17236366 will disable the getBindsMount thrift call and
remove the internal source of data about bind mounts. We instead
have a more current set of data from the `redirect` command, so
tech `eden chown` to use that data.
Reviewed By: chadaustin
Differential Revision: D17249433
fbshipit-source-id: 853f24e729814c501768e9834765e1be283d6aac
Summary:
Integration test helpers relied on an implicit gflags include. Make
that explicit so they compile against open source gflags and glog.
Reviewed By: wez
Differential Revision: D17264335
fbshipit-source-id: e336423b71c0f15e29b0e4ad604328b7624080a8
Summary:
Make sure the contents of the special `.eden/` subdirectory are correct each
time we mount a checkout. Before we would generally only set up the contents
of this directory if it didn't previously exist.
Now the code verifies that the contents of each symlink in this directory are
correct, and recreates the symlink if needed.
This allows EdenFS to automatically repair the contents of this directory even
if the checkout or its `clients` directory has been manually moved.
Reviewed By: wez
Differential Revision: D17279413
fbshipit-source-id: e24e7530f44fff94ebb6f67174aaf78c9b498d6b
Summary:
Update the fsck code to save any orphaned symlink inodes that it finds as
symlinks in the repair archive directory, rather than saving the contents as a
regular file.
Reviewed By: wez
Differential Revision: D17170346
fbshipit-source-id: 4cba8b27233b728114a80a327ab519b039297aea
Summary:
Use the new `OverlayChecker` class to automatically scan for errors and
attempt to repair them if the overlay was not shutdown cleanly the previous
time it was used.
Reviewed By: wez
Differential Revision: D16596601
fbshipit-source-id: 9923565b101ba953e92909e502be6ef5895c5cbd
Summary:
This was causing problems on macos where various tools
would enumerate and helpfully try to preserve attributes across
copies. On macos this would result in appledouble metadata files
being created to track the metadata in the destination file,
which clutters up the repo and has surprising secondary effects
such as being picked up by glob operations in cmake build rules.
This diff simply stops enumerating the extended attribute.
Reviewed By: fanzeyi
Differential Revision: D17140414
fbshipit-source-id: 2924657dc75b900baf70595edfa72e5d0521a697
Summary:
The snapshot generation code and many integration tests create
repositories. By default, they were creating flatmanifest
repositories, which are on their way out. Instead, create tree-only
repos.
Reviewed By: strager
Differential Revision: D17066151
fbshipit-source-id: f99a9543440da6fd7cce0065c3cd7f91a59a02d5
Summary:
Update `EdenServer::mount()` to correctly handle errors that occur during the
mount `INITIALIZING` phase. Previously the code did not add error callbacks
to the `Future` result to handle errors during initialization. As a result we
would propagate the exception back to the thrift caller, but the `EdenMount`
object would remain in our mount point list, stuck forever in the
`INITIALIZING` state.
Reviewed By: strager
Differential Revision: D16590032
fbshipit-source-id: 9adbdf05441dad815096b195ece36f3d958c96a9
Summary: Add a dependency from the eden open source build to the fb303 open source build and switch EdenServiceHandler to BaseService.
Reviewed By: simpkins
Differential Revision: D15528156
fbshipit-source-id: 2ca5c31dd9fcc9bac43fd399b27f33b6f2c5ebfc
Summary:
Open source fb303 will not have getPid() or getCommandLine(), so
introduce a new method for Eden's tests.
Reviewed By: fanzeyi
Differential Revision: D16292993
fbshipit-source-id: 5cdc006ec0ee15f50a3e1cebe9b46a3ea275ff78
Summary: The CountersTest would previously fail if by chance the counters prefixed by "thrift" and "thrift_client" were accounted for between getting "counters" and "counters2", since these counters should not be modified when mounting/unmounting mounts we will just filter them out.
Reviewed By: chadaustin
Differential Revision: D16265511
fbshipit-source-id: 21af0dff345977692785136ca0333d23d5c77e0d
Summary: I found out that the journal stats callbacks that were getting registered were not getting unregistered, this diff fixes that.
Reviewed By: chadaustin
Differential Revision: D16187569
fbshipit-source-id: 8c84e1515e376ccd7036a22c06e2e6b98dc62342
Summary: Turn on logview collecting by default. Also disable logview for Eden integration tests.
Reviewed By: chadaustin
Differential Revision: D15978960
fbshipit-source-id: 623c3be7a461e0bb9bc44924fccfdb006565fad6
Summary: Added the cli command `eden stats object-store` for querying the counts on what part of the object store was responsible for finding the blob or blob size (local store or backing store). This will tell us how well local and in-memory caching works for different workflows.
Reviewed By: chadaustin
Differential Revision: D15934535
fbshipit-source-id: 70345f11a51c3c6996dc001d4101744395a3d182
Summary:
This diff adds a single repo-wide `.eden-redirections` file that is used to
record the redirection configuration for the repo.
The redirection configuration code knows to look at this file and fold it in to
the effective configuration; the legacy bind mounts are applied first, then the
repo redirection configuration and then the user specified redirections.
The intent is that a `post-update` hook will invoke `eden redirect fixup` to
apply the configuration from the repo, and to have the eden daemon also
trigger that after (re-)mounting the repo.
In the future we can extend this to allow different profiles to be enabled
or disabled, but for now this is the minimum viable product.
Reviewed By: strager
Differential Revision: D15867225
fbshipit-source-id: b0a95936dd28283de6c7439ca8e503caef4e7247
Summary:
This is part of the effort to make our bind-mount configuration more
visible and easier to change.
The idea is to generalize the concept of redirection and add a command to help
manage it.
The `eden redirect add` subcommand allows creating one of two different kinds
of redirection:
* `bind` - allocate some space using `mkscratch` and mount it into the repo
* `symlink` - allocate some space using `mkscratch` and create a symlink
that points to it from the repo
On Linux we use bind mounts to implement `bind` but on macOS, which doesn't
have a bind mount concept, we create a sparse disk image file that can grow
to match the size of the disk on which it is created (in practice these are a
7-15MB in size to start and grow as the user stores data into them).
The `eden redirect del` subcommand allows removing a redirection, including
the legacy `bind-mounts` configuration from `.eden/client/config.toml`.
The `eden redirect list` subcommand lists the effective set of redirections,
both from the new redirections configuration and the legacy `bind-mounts`
configuration, along with their state.
The `eden redirect fixup` subcommand iterates over the effective set of
redirections and can remove and reinstate any that are in a broken state.
Reviewed By: strager
Differential Revision: D15707319
fbshipit-source-id: a5dd8c44c9f748482d7b48855b1305d44267885c
Summary: This diff takes care of importing blob from Mononoke and Mercurial at the same time, also improves the name situation in the statistics counters.
Reviewed By: strager
Differential Revision: D15768557
fbshipit-source-id: 10cf831b1ae6dc9e6b91f1e96508c4fa92583743
Summary:
If edenfs was started using `sudo`, the `$USER` environment variable will be
set to `root` rather than the actual user. When we drop privileges make sure
we restore the value of `$USER` as well.
The `$USER` variable isn't checked anywhere else in edenfs itself, but it
matters for subprocesses we spawn, like `hg debugedenimporthelper`.
I also changed the code to clear the `SUDO_*` variables as well, mostly
just for good measure.
Reviewed By: kulshrax
Differential Revision: D15929539
fbshipit-source-id: e022c7ae762e2a5e86d0227058bb476aff17cf55
Summary:
Add a periodic task for performing LocalStore management tasks. For now only
the RocksDBLocalStore class implements this management task.
When this periodic task runs the RocksDBLocalStore object computes how much
space each of the column families are using and publishes this as fb303
counters. If the total size of the ephemeral column families exceeds a
configurable limit it then triggers a background garbage collection task.
I also added a new `edenfsctl stats local_store` command that reports the new
counters added by this diff.
Reviewed By: chadaustin, strager
Differential Revision: D15798505
fbshipit-source-id: 25ca4ba80f5a9c4a1a09dc08633c7b3af363d7ff
Summary:
Update the copyright & license headers in Python files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487088
fbshipit-source-id: 9f2138dff41048d2c35f15e09a04ae5a9c9c80dd
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary:
Add a periodic task to reload the configuration file from disk. By default
this runs once every 5 minutes, but this interval can be controlled from the
config file.
At the moment reloading the config file does not do much other than update the
interval for how frequently the config file is reloaded. However, I plan to
add additional periodic tasks shortly that are controlled by this config
setting.
This will also make it possible for other parts of the code to
access the config settings in the `ServerState` and use them as-is without
checking to see if they reloaded. Currently all of the code that accesses
config values performs a check to see if the config needs to be reloaded. If
we want to switch to Mercurial-style configs in the future that check will be
substantially more expensive.
This diff also includes a new thrift call to force the config file to be
reloaded immediately. This can be used to restart automatic config reloading
if it is ever disabled in the config file.
Reviewed By: wez
Differential Revision: D15756357
fbshipit-source-id: 1999f4730903633ce838842932a6ae6a65eda4e6
Summary: Fix a few more issues raised by our Python lint checks.
Reviewed By: wez
Differential Revision: D15776717
fbshipit-source-id: 621960579c4567c4fb9395ae14cd7a8666726c1c
Summary: Remove a number of unused imports detected by the linter.
Reviewed By: wez
Differential Revision: D15776268
fbshipit-source-id: 221f45d275664d037bbabcac9858b40266b4833e
Summary:
If the systemd user manager is not running (e.g. it crashed or was manually stopped), 'eden stop' fails with an unhelpful error message:
> pystemd.dbusexc.DBusConnectionRefusedError: [err -111]: Could not open a bus to DBus
or
> pystemd.dbusexc.DBusBaseError: [err -2]: Could not open a bus to DBus
Provide a better experience: tell the user the most likely cause, and suggest a solution:
> error: The systemd user manager is not running. Run the following command to start it, then try again:
> sudo systemctl start user@strager.service
Reviewed By: wez
Differential Revision: D13791023
fbshipit-source-id: 5172df0a52d21c311b27b8a527cad934f9882154
Summary:
Summary
Change `ConfigTest` to derive directly from `EdenTestCase` rather than using
the `eden_repo_test` decorator. The configuration test code doesn't really
need a repository, and so we don't need to run it twice (for both Mercurial
and Git repositories).
Reviewed By: wez
Differential Revision: D15756359
fbshipit-source-id: 90d5011ae1ff7d2a251c9e7bb776045fbe2fdfe1
Summary:
Clean up the `EdenFS` class construction.
Previously it accepted the `eden_dir`, `etc_eden_dir`, and `home_dir`
arguments as separate parameters. If `etc_eden_dir` or `home_dir` were not
specified it would not pass these arguments to `edenfs`, allowing the default
values to be used. This is undesirable for most tests.
Now it accepts a `base_dir` argument. Explicit values for the `eden_dir`,
`etc_eden_dir`, and `home_dir` parameters can still be specified (this is used
for the snapshot tests), but if they aren't specified, default locations
inside the `base_dir` will be used instead.
This also cleans up some of the code to use `pathlib.Path` values instead of
plain `str` objects in more places.
Reviewed By: strager
Differential Revision: D15756358
fbshipit-source-id: 3e87ddc98d15fcb7f60c6c3116d4fcc8e49432ea
Summary:
Add a thrift call to get the current config settings.
My primary use case for this method at the moment is to make it possible to
build integration tests that check the config behavior. However in the future
this will probably also be useful for building CLI commands to report the
current config values to allow debugging if there are ever issues. This API
can also be used to force EdenFS to immediately reload the config from disk.
Reviewed By: strager
Differential Revision: D15572124
fbshipit-source-id: da3bc982f9c419b3314a8b0560c9bd327760d429
Summary:
With systemd integration enabled, if edenfs is running, or if edenfs' systemd service is active, `edenfsctl start` does nothing. This behavior differs from `edenfsctl start` with systemd integration disabled, and can cause `edenfsctl restart` to think that it successfully started edenfs.
Make `edenfsctl start` fail if edenfs is running and healthy, or if edenfs' systemd service is active (yet edenfs is unhealthy).
Reviewed By: chadaustin
Differential Revision: D15703310
fbshipit-source-id: ce0a13780ee03de1f896a938d002901023e5bdd3
Summary:
systemctl has some problems for Eden. For example:
* With Restart=on-failure, 'systemctl start' reports that the job failed when the first failure occurs. 'systemctl start' does not wait for retries to finish. This means 'eden start' can fail despite edenfs starting successfully.
* If the service fails to start, 'systemctl start' prints a suggestion to use journalctl, even though journalctl is broken and is not even used by fb-edenfs@.service.
* If 'systemctl' can't connect to systemd, it prints a generic message such as "Failed to connect to bus: Connection refused" which the Eden CLI can't easily detect and customize.
For 'eden start', instead of using systemctl, talk to systemd using its D-Bus API (via pystemd [1]). This automatically solves the journalctl message problem, makes it trivial to customize certain errors, and will let us solve the Restart=on-failure problem in the future.
Aside from changing some error messages, this diff should not change behavior.
[1] https://github.com/facebookincubator/pystemd
Reviewed By: simpkins
Differential Revision: D13533184
fbshipit-source-id: 7fedc8ad4a094a2d04b14c2f6e82b51a0ed348a6
Summary: We want to have just one entry point to Mercurial, namely the Rust binary. Getting rid of the `hg` Python script means that we finally can do things that only exist in Rust and not in Python.
Reviewed By: simpkins
Differential Revision: D13186374
fbshipit-source-id: f3c8cfe4beb7bf764172a8af04fd25202eca9af2
Summary:
test_reading_committed_file_bumps_read_counter is flaky with optimized builds of edenfs. I think it's flaky because FuseChannel bumps counters *after* responding to the kernel, so the test can call get_counters before the counters are bumped.
Fix the flakiness by making the test wait a while for the counters to change.
Reviewed By: chadaustin
Differential Revision: D15550972
fbshipit-source-id: 891e5d0a9748b43eb0ef1089ef6bc0a547c47d4d
Summary:
In the past few months, test_no_units_are_active started failing. It looks like 'systemctl list-units' is now listing devices. For example:
```
$ systemctl list-units --all --full --no-pager
UNIT LOAD ACTIVE SUB JOB DESCRIPTION
● boot.automount not-found inactive dead boot.automount
proc-sys-fs-binfmt_misc.automount loaded active running Arbitrary Executable File Formats File System Automount Point
dev-disk-by\x2dlabel-\x5cx2f.device loaded active plugged /dev/disk/by-label/\x2f
[snip]
dev-getty.device loaded inactive dead /dev/getty
dev-loop0.device loaded active plugged /dev/loop0
dev-ram0.device loaded active plugged /dev/ram0
dev-ram1.device loaded active plugged /dev/ram1
[snip]
```
I don't know if systemctl changed or if systemd changed or if my machine's configuration changed. Either way, the test is failing now due to these systemd units.
Teach test_no_units_are_active to ignore these unimportant device units, since they don't represent running services or timers. This causes the test to pass on my machine.
Reviewed By: wez
Differential Revision: D15548072
fbshipit-source-id: 4f49c72d88b836aba37ec5ea7b5ee5b7cb8172f6
Summary:
EdenFS' systemd tests detect if the running system is managed by systemd, and chooses different strategies for creating a sandboxed systemd user manager depending on this detection. Sandcastle (Facebook's continuous integration system) recently started running systemd as the container's init system.
EdenFS' systemd tests correctly detect that Sandcastle's container is managed by systemd. Unfortunately, the tests cannot communicate with systemd. For example, `XDG_RUNTIME_DIR=/run systemctl --user status` in a Sandcastle job reports the following error:
> Failed to connect to bus: No data available
This error happens because systemd and the EdenFS' tests are running in a different process namespace. The filesystem shows that the system is managed by systemd, but the process table shows otherwise! This causes all of EdenFS' systemd tests to fail.
Work around this issue making Sandcastle use the old "unmanaged" code path. (The unmanaged code path is run if the system is not managed by systemd. Before Sandcastle started running systemd in its container, Sandcastle used this code path when running EdenFS's tests.)
(Ideally we would figure out why we need both the "managed" code path in the first place. This diff is just meant to fix tests on Sandcastle for now, not implement a long-term solution.)
Possibly related upstream systemd issue: https://github.com/systemd/systemd/issues/11300
Reviewed By: wez
Differential Revision: D15530685
fbshipit-source-id: b65b568e660310c50a4e25e0aa143f9388f1ad45
Summary:
I am refactoring edenfs' EdenStats class. In doing so, I accidentally removed a call to `aggregate`, causing `flushStatsNow` to not expose accurate counters. This wasn't caught by any existing tests, only by manual testing.
Add some tests to prevent FUSE and HgBackingStore statistics collection from completely regressing.
Reviewed By: simpkins
Differential Revision: D15274275
fbshipit-source-id: c8a9c9848dd60aee7f252a93f10ddce6d7560799
Summary:
In another diff, Adam Simpkins noticed that HgImporterStatsTest.get_counters was clunky and that the logic belongs in the EdenTestCase base class.
Hoist HgImporterStatsTest.get_counters into EdenTestCase. Also avoid reusing the Thrift client between calls to get_counters because edenfs might restart between calls (in which case the old Thrift client won't work).
Reviewed By: wez
Differential Revision: D15514537
fbshipit-source-id: 0ae25106baa0e5b2d857b0bb2552d884b9b270ef
Summary: Some pyre-fixme directives are on the wrong line. Move them to the line to fix "Unused ignore" errors from Pyre.
Reviewed By: wez
Differential Revision: D15507418
fbshipit-source-id: b8d1163080b1c64868c37e7581411be31f495141
Summary: This function isn't used anywhere. Delete it.
Reviewed By: pkaush
Differential Revision: D9695388
fbshipit-source-id: 1ac702c98ee63d09c15c8a7b8a9c8d44fcec630d
Summary:
In EdenFS' tests, when systemd-run fails, subprocess.check_output raises an exception and causes the test to fail. The exception's message does not include any output of systemd-run, so systemd-run's error messages are hidden. This makes it hard to debug systemd-run failures.
When systemd-run fails, print the captured output. This surfaces error messages printed by systemd-run, improving the debugging experience.
In the normal case where systemd-run succeeds, this diff should not change behavior.
Reviewed By: wez
Differential Revision: D15466610
fbshipit-source-id: 2b2db80b989308967e13499fcfadd37b44ca878f
Summary:
Update the integration tests to avoid explicitly using the old
`hg_import_helper.py` script.
Reviewed By: pkaush
Differential Revision: D15223982
fbshipit-source-id: 6e2310d5a9e6e0b95690a07d61d295cc3b1bca92
Summary: Flatmanifest is on its way out. Remove support for falling back to it if a tree import fails.
Reviewed By: pkaush
Differential Revision: D15056459
fbshipit-source-id: a4df820322ee354d77f50a0ec92e9705d0f152ec
Summary:
Move some of the argument parsing and config setup code out into a new
EdenInit.h header file. This makes it possible to re-use this logic for other
standalone utilities that want to be able to find the Eden state directory and
config information.
For now I have updated the `fake_edenfs` helper tool used by the integration
tests to use this. This may also be useful for writing standalone tools that
can perform garbage collection of the LocalStore or checking of the overlay
state.
Reviewed By: chadaustin
Differential Revision: D14889616
fbshipit-source-id: b0b193a42cb2f52177d0c44592426b42e27242aa
Summary:
On my system this test was failing because the `systemctl start exit.target`
command was exiting with an error. It looks like this command exits because
systemd closes its D-Bus connection as it is shutting down. This modifies the
test to ignore the return code from this systemctl command.
I also updated the log message when shutting down systemd to no longer include
the exception backtrace. This message is always logged when running this test
(since systemd is not actually running), and the backtrace information simply
makes the test logs harder to read.
Reviewed By: wez
Differential Revision: D14886191
fbshipit-source-id: 87c996b2579a9920a72ee5b57608c263ca080d6e
Summary:
Sometimes, EdenFS goes bonkers and talks to 'hg debugedenimporthelper' a lot for seemingly no reason.
Make these situations easier to debug by counting how many requests EdenFS makes to 'hg debugedenimporthelper'. These counters let us answer questions such as the following:
* When performance sucks, is EdenFS is making a lot of requests or only a few requests? I.e. is Hg slow at responding or is EdenFS very demanding?
* Does a recent performance issue correlate with EdenFS communicating with 'hg debugedenimporthelper'?
* Which engineers are outliers having orders of magnitude more 'hg debugedenimporthelper' requests than p50 engineers?
We could get fancier with these counters and include the number of bytes received, the duration of the request, etc. For now, just having a request count is useful.
Reviewed By: simpkins
Differential Revision: D14677339
fbshipit-source-id: 7f8f394fb0096aef65d6a8a45d7da5936db539a0
Summary:
After the kernel added readdir caching, my testing uncovered that Eden
was invalidating TreeInode entries incorrectly when new entries were
added. Change TreeInode to distinguish between directory entry changes
and removals (FUSE_NOTIFY_INVAL_ENTRY) and additions
(FUSE_NOTIFY_INVAL_INODE).
Reviewed By: strager
Differential Revision: D13870422
fbshipit-source-id: 2a6f25bfd9e77436a5aae639fedbfd8a445b2e05
Summary:
Update `RocksHandles` to call `RepairDB()` if we get an error opening the
database.
We have seen errors opening the DB in some cases after hard server reboots.
In every case so far `ldb repair` has been able to repair it with no adverse
effects. This simply makes `edenfs` automatically attempt to perform this DB
repair step.
Reviewed By: chadaustin, strager
Differential Revision: D14452216
fbshipit-source-id: 10c0cb0ff9cea3c3bbe485a4e489e4a6df640803
Summary:
Sometimes, the XDG_RUNTIME_DIR environment variable isn't set. If this happens, 'eden start' fails because systemctl uses XDG_RUNTIME_DIR to talk to systemd. We still want 'eden start' to work in these cases, so guess what XDG_RUNTIME_DIR should be and use that guess if the variable isn't set.
If XDG_RUNTIME_DIR is set in the environment, its value should still be used.
Reviewed By: chadaustin
Differential Revision: D13811813
fbshipit-source-id: bb44d99e585bbe7a4341087c5cb4644c606fc441
Summary:
Sometimes, Facebook's CI servers might not have a /run/systemd directory. This causes EdenFS' systemd tests to fail, because daemon-respawn can't access that directory [1].
Fix the tests on CI by creating /run/systemd.
Why did the tests only start failing recently? I'm not sure. I think we were just lucky in the past; tests in other projects seem to create /run/systemd (e.g. using the systemd-nspawn command), and it looks like this state persists across CI jobs.
[1] https://github.com/systemd/systemd/blob/v239/src/core/dbus-manager.c#L1277
Reviewed By: simpkins
Differential Revision: D14436098
fbshipit-source-id: eb48abeb1ce38ea4ae760192db37bb1910efff99
Summary:
If sanity_check_enabled_unit_fragment fails, the error message is unhelpful:
```
Exception: Enabled unit's FragmentPath does not match unit file
Expected: {repr(expected_unit_file)}
Actual: {repr(actual_unit_file)}
```
Use format strings as was originally intended, and drop repr to make the paths easier to read:
```
Exception: Enabled unit's FragmentPath does not match unit file
Expected: /data/users/strager/fbsource/fbcode/eden/fs/service/fb-edenfs@.service
Actual: /usr/lib/systemd/user/fb-edenfs@.service
```
Reviewed By: simpkins
Differential Revision: D13372758
fbshipit-source-id: 0f12cc7a6f63fc53d72ce92b265e0ccbcc26d394
Summary: We don't run this binary anymore, no reason to build and ship it.
Reviewed By: quark-zju
Differential Revision: D14437317
fbshipit-source-id: dd6da521783f18a2a518a7aa042be98950894e89
Summary:
If TreeInode::startLoadingInode() is in progress, and EdenServer::startTakeoverShutdown() is called, edenfs can deadlock:
1. Thread A: A FUSE request calls TreeInode::readdir() -> TreeInode::prefetch() -> TreeInode::startLoadingInode() on the children TreeInode-s -> RocksDbLocalStore::getFuture().
2. Thread B: A takeover request calls EdenServer::performTakeoverShutdown() -> InodeMap::shutdown().
3. Thread C: RocksDbLocalStore::getFuture() (called in step 1) completes -> TreeInode::inodeLoadComplete(). (The inodeLoadComplete continuation was registered by TreeInode::registerInodeLoadComplete().)
4. Thread C: After TreeInode::inodeLoadComplete() returns, the TreeInode's InodePtr is destructed, dropping the reference count to 0.
5. Thread C: InodeMap::onInodeUnreferenced() -> InodeMap::shutdownComplete() -> EdenMount::shutdown() (called in step 2) completes -> EdenServer::performTakeoverShutdown().
6. Thread C: EdenServer::performTakeoverShutdown() -> localStore_.reset() -> RocksDbLocalStore::~RocksDbLocalStore().
7. Thread C: RocksDbLocalStore::~RocksDbLocalStore() signals the thread pool to exit and waits for the pool's threads to exit. Because thread C is one of the threads managed by RocksDbLocalStore's thread pool, the signal is never handled and RocksDbLocalStore::~RocksDbLocalStore() never finishes.
Fix this deadlock by executing EdenServer::shutdown()'s callback (in EdenServer::performTakeoverShutdown()) on a different thread.
Reviewed By: simpkins
Differential Revision: D14337058
fbshipit-source-id: 1d63b4e7d8f5103a2dde31e329150bf763be3db7
Summary:
The feature was completed by Phil in D9816270. It's handy and can probably
reduce user support burden like: https://fb.intern.facebook.com/groups/scm/permalink/2039619916087618/
Therefore let's enable it.
Reviewed By: DurhamG
Differential Revision: D14293405
fbshipit-source-id: 54e934e0bf495c090109462e4f743d427df39380
Summary:
Update most of the `eden/cli/config.py` to use `Path` instead of `str` where
appropriate. This also updates several of the APIs in `util.py` that were
affected as well.
Reviewed By: chadaustin
Differential Revision: D14356543
fbshipit-source-id: a8f6d15b8870bf689eeb78f9fc0e9a0c65c97218
Summary:
If no mounts are configured `eden fsck` previously threw an exception when
trying to compute the return value. It called `max(return_codes)` on an empty
return codes list, which would fail. This changes the code to handle that
code specially and report a warning that there was nothing to check.
Reviewed By: chadaustin
Differential Revision: D14352112
fbshipit-source-id: 3815ef34a12834d642f3eee867dda6dc1117c2ef
Summary:
This updates `edenfs` to automatically create the mount point directory
if it does not exist.
Previously the `eden mount` CLI command would automatically create the mount
directory in the Python logic. This adds similar logic to the C++ code, which
handles more situations. In particular this makes it so that `eden start`
will now automatically create missing mount point directories.
Note that the C++ code does not create the `README_EDEN.txt` symlink inside
the mount point if it is missing. We could move that functionality into the
C++ code in the future if needed.
Reviewed By: strager
Differential Revision: D14254699
fbshipit-source-id: bad5634f57fba6e7af3b6a3830eb51ac099b435e
Summary:
This updates the `EdenServer` class so that the existing `getMount()` and
`getMountPoints()` APIs only return mounts that have finished initializing.
These APIs are primarily used by the thrift interfaces. In most cases the
callers did not intend to operate on mounts that were still initializing, and
doing so was unsafe. The code could potentially dereference a null pointer if
it tried to access the mount's root inode before the root inode object had
been created.
New `getMountUnsafe()` and `getAllMountPoints()` APIs have been added for call
sites that explicitly want to be able to access mounts that may still be
initializing. Currently the `listMounts()` thrift API is the only location
that needs this.
Reviewed By: strager
Differential Revision: D13981139
fbshipit-source-id: e6168d7a15694c79ca2bcc129dda46f82382e8e9
Summary:
Add a flag to tell edenfs to report successful start-up as soon as the thrift
server is running, without waiting for all mount points to finish being
remounted.
In the future I plan to have edenfs automatically perform an fsck scan of the
overlay for checkouts that were not shut down cleanly. This may cause the
remount to take a significant amount of extra start-up time in some cases.
(This is already true today in some cases even with the simpler scan we do to
re-compute the max inode number.)
I think we will probably want to have systemd invoke edenfs with this option,
so that we do not time out during system start up if some mount points need to
be rescanned.
Reviewed By: strager
Differential Revision: D13522040
fbshipit-source-id: 6f183770c25efee34c4805c9bad42a9cce51039e
Summary:
Update `edenfs` to automatically create the bind mount source directories if
they are missing. Previously Eden would report an error and would not be able
to mount the checkout if some of the bind mount source directories were
missing.
Reviewed By: strager
Differential Revision: D14253771
fbshipit-source-id: 87ad091ccf2c0f0f72aebb50437fd7680ddbfd1c
Summary:
Update `EdenMount::initialize()` to perform a fault injection check. This
allows test code to inject delays and errors into the mount initialization
flow.
Reviewed By: strager
Differential Revision: D14079491
fbshipit-source-id: be80135b0833c8f0300104524473cc3e949fec34
Summary:
The Eden CLI tool is really a control program for `edenfs`. Rename it to
`edenfsctl` to free up the `eden` name for future use.
The Eden daemon shouldn't really be on the user's path, and instead belongs in
`libexec`.
For transition compatibility, `eden` is symlinked to `edenfsctl`.
Reviewed By: simpkins
Differential Revision: D13888875
fbshipit-source-id: 435cc63e92b85b1f28b8691e4846fbcb05bc450e
Summary:
Now that `hg debugedenimporthelper` has been released for
a little while, we can remove the bundled implementation of it from
the eden release.
However, we cannot remove the script itself as there are users
with long running edenfs instances that pre-date the knowledge
of `hg debugedenimporthelper`. So, as a compatibility shim,
this diff redirects `hg_import_helper.py` and has it exec the
command in mercurial.
For extra fun, our own integration tests rely on being able
to import `hg_import_helper.py` and override portions of it,
so we cannot remove its implementation from the codebase just
yet either.
So this diff:
* Introduces `proxy_import_helper.py` which execs `hg debugedenimporthelper`
* Installs `proxy_import_helper.py` as `hg_import_helper.py` in the rpm
* Leave `hg_import_helper.py` as-is in the tree for now.
Reviewed By: simpkins
Differential Revision: D13970332
fbshipit-source-id: 717dc86a880fbbbe4a7e801a8b748abd053c7f7c
Summary: Branches are going away. Remove the use of them.
Reviewed By: strager
Differential Revision: D14062107
fbshipit-source-id: 00f6d3666eb3cb6900cd570fa3fcf12ba75c2ae0
Summary: using upgrade script to clear out all remaining version-set configs
Reviewed By: dark
Differential Revision: D13832474
fbshipit-source-id: 52c280cbd79b1410821ed829465b1c0907b50a86
Summary:
Update the `eden list` command to also report the current state for each
checkout if it is not running normally. Also added a `--json` flag to
print information as JSON so it can be consumed programmatically.
Reviewed By: strager
Differential Revision: D13503053
fbshipit-source-id: 4ef366f5bf4a1157036fdfd7ff1056079588e802
Summary:
TemporarySystemdUserServiceManagerTest.test_exit_kills_manager is flaky due to a race condition. SystemdUserServiceManager.exit does not wait for the systemd process to exit; I think it only waits for systemd to close its socket. This means the process can still be alive, and `did_process_exit` can return true.
Fix the race condition by making SystemdUserServiceManager.exit block until the systemd process exits.
Reviewed By: chadaustin
Differential Revision: D13791407
fbshipit-source-id: 8422e0101eaea8b4da285dcb0fcf564435b30065
Summary:
Add an option to forcibly kill `edenfs` with SIGKILL without ever attempting
to query it over thrift.
This should provide a way for users to reliably kill edenfs even if it is
hung. This shouldn't be necessary in most cases, but it lets us tell users to
run this command as a last resort if something is wrong.
Reviewed By: chadaustin, strager
Differential Revision: D13744188
fbshipit-source-id: 13378d04b3398e72ed3733d4ebb68b39868007bd
Summary:
When graceful restart was first implemented we forgot to update the
lock file with the new pid, resulting in occasional unexpected output
from tools like eden doctor.
Reviewed By: simpkins
Differential Revision: D13744411
fbshipit-source-id: cdc758ed6ac1201fd2ff3e9d7805bb5ab6f83e8a
Summary:
On a systemd-managed system, the `XDG_RUNTIME_DIR` environment variable is set on login. `systemctl` uses this variable to know how to talk to the systemd user manager. If `XDG_RUNTIME_DIR` is not set in the environment, `systemctl` (and thus `eden start`) fails with an unhelpful message:
Failed to connect to bus: No such file or directory
Improve this message by explicitly checking for the absence of `XDG_RUNTIME_DIR`.
Reviewed By: simpkins
Differential Revision: D13728111
fbshipit-source-id: a7f60fc29561acd05fbc1bf52d7968ae0e64d0c2
Summary:
The failure messages printed by 'eden start' are kinda crappy with systemd integration enabled. Add some tests for these messages so we can easily iterate on them.
In particular, test the following cases:
* The systemd user manager is no longer running
* The XDG_RUNTIME_DIR environment variable, needed to talk to systemd, is not set
* edenfs fails to start
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13723440
fbshipit-source-id: abae5c0e4a9f0bc6b8d0d606e8f5f36760aad5fa
Summary:
A bug in Pyre causes the properties of FindEXE to have an incorrect type. We currently work around this bug by silencing type errors. Unfortunately, this might silence legitimate errors too.
Instead of silencing type errors, using `typing.cast` to tell Pyre the correct type. This should expose legitimate errors if they exist.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13709138
fbshipit-source-id: 55f47f47062a35911c6bbe03ffd7b02a90a5107f
Summary:
In our linux deployments it was relatively straightforward
to import the mercurial runtime from a python process running the
system python executable. Our macOS deployments are a lot more
complex because they do not use the system python and do not install
the mercurial python packages in the python path of the target
python executable.
It is simpler to move the import helper functional into a mercurial
command that we can invoke instead of our own helper program.
This diff moves the script to be a debug command and adjusts its
argument parsing to match the mercurial dispatcher requirements.
There are some stylistic mismatches between this code and the
rest of mercurial; I'm suggesting that we ignore those as the
medium term solution is that this command is replaced by eden
directly consuming the rust config parsing code and by native
rust code to perform the data fetching that we need.
Reviewed By: pkaush
Differential Revision: D13522225
fbshipit-source-id: 28d751c5de4228491924df4df88ab382cfbf146a
Summary:
If edenfs is not running or is unhealthy, 'eden rage' does not run 'eden doctor'. This means 'eden rage' does not include helpful output such as ~/local/.eden/clients/* being on NFS, or the Linux kernel version being unsupported.
Make 'eden rage' run 'eden doctor' regardless of the health of edenfs.
Reviewed By: simpkins
Differential Revision: D13633381
fbshipit-source-id: 2439057ba7a7bbe5041991ddc4ede256e86634f3
Summary:
Update the Eden CLI to use `os.path.realpath()` to resolve symlinks in the
Eden config directory before using it.
In most situations today it is common for the default path
(`$HOME/local/.eden`) to traverse as symlink in the user's home directory.
For users that are still using NFS home directories we can sometimes read this
symlink when running as the user, but we cannot read the symlink as root (for
instance, from inside the privhelper process).
Resolving symlinks in the CLI code ensures that the `edenfs` daemon will see
the final resolved path and will not need to traverse symlinks in the user's
home directory in this situation.
Reviewed By: strager
Differential Revision: D13515871
fbshipit-source-id: 0602389492afc0b542e089bb002534f3d714882e
Summary:
Update the ServiceTestCaseBase code so that each test case keeps its state in
a single top-level temporary directory. This makes it a little easier to
figure out which directory is which when debugging a test. I also plan to
write a new test soon that creates some additional directories, and having one
top-level temporary directory avoids needing to create new TemporaryDirectory
objects.
Reviewed By: strager
Differential Revision: D13522026
fbshipit-source-id: 95a3d268d267a107bbf5e405839d64afd6afdb03
Summary:
Change some of the integration tests to read back the original command line
arguments from fake_edenfs over thrift rather than by writing it out to a file
on disk.
This shouldn't really change much, it just seemed slightly simpler.
Reviewed By: strager
Differential Revision: D13515855
fbshipit-source-id: 386207c00f28626e2125958895387a870ca87b82
Summary: 'eden clone' starts the EdenFS daemon if it's not already running. If the user opted into systemd integration, make sure the daemon is started via systemd.
Reviewed By: wez
Differential Revision: D13498650
fbshipit-source-id: 8c5da579f9b79363e2d825ea7c85d423cbcc6509
Summary:
Add another suppression for T38220626 that appears to have been missed in
D13502225 when it was rebased before landing.
Reviewed By: strager
Differential Revision: D13526440
fbshipit-source-id: 60f5f6eff36b5f8462286c229836ffcb88f3afc1
Summary:
Update the `eden clone` command to automatically create the `.hg` directory
when creating a checkout for a Mercurial repository.
Previously this logic was performed by a separate post-clone hook that was
invoked by `eden clone`. Having this logic in a separate script made the code
slightly more complicated, and meant that configuring Eden was also more
complicated, as the hook also needed to be installed and configured. Moving
the logic into the Eden CLI will make it easier to re-use this code in
`eden doctor` if the `.hg` directory needs to be repaired.
Reviewed By: wez
Differential Revision: D13447272
fbshipit-source-id: 11c4f8e389aead151dd235eff95c860a326967af
Summary: If edenfs crashes when starting, we don't want systemd to keep trying to restart the service forever. systemd already behaves as we want, but add a test to make sure this feature doesn't regress.
Reviewed By: wez
Differential Revision: D13327803
fbshipit-source-id: df4fb0e5b2d9874fda58bad903087e411efeeefc
Summary:
When run inside the systemd service (fb-edenfs@.service), edenfs' logs are written to `/var/log/messages` (on Facebook dev servers). This is undesirable, since those logs have a bunch of noise.
Make systemd-managed edenfs log to `~/local/.eden/logs/edenfs.log` instead, matching the behavior of custom-managed edenfs.
---
I considered using systemd's StandardOutput= and StandardError= directives [1], but they have limitations:
* **StandardOutput=file:%f/logs/edenfs.log**: When the `logs` directory is missing, systemd does not create it. In this case, systemd fails when it opens the log file, so systemd refuses to start the service.
* **StandardOutput=journal** [2]: journald and journalctl are broken for user services. Logging to journald only works with persistent journal storage [3][4], but Facebook uses volatile journal storage.
* **StandardOutput=syslog** [5]: rsyslog seems designed for system administrators, not users. I didn't investigate much, but I suspect it's impossible to make rsyslog write to a user-controlled path such as `~/local/.eden/logs/edenfs.log`.
* **LogsDirectory=%f/logs and StandardOutput=file:%L/edenfs.log** [6][7]: LogsDirectory= does exactly what we need, except it only supports paths relative to `/var/log` or `~/.config/log/`. `LogsDirectory=%f/logs` does not work, and systemd will ignore such a directive.
* **StandardOutput=file:%f/logs/edenfs.log and a `mkdir` service**: If we create a service which just creates the `logs` directory, and make fb-edenfs@.service depend upon that service, systemd can successfully open the log file [8]. In theory, using StandardOutput= would cause errors like "could not set resource limits" to be logged to `edenfs.log`. In practice, systemd does not respect the service's logging configuration when reporting such errors [9]. Therefore, this solution is no better than the manual redirect.
[1] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#StandardOutput=
[2] https://www.freedesktop.org/software/systemd/man/systemd-journald.service.html#
[3] https://www.freedesktop.org/software/systemd/man/journald.conf.html#SplitMode=
[4] https://lists.freedesktop.org/archives/systemd-devel/2016-October/037554.html
[5] https://www.rsyslog.com/
[6] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#RuntimeDirectory=
[7] https://www.freedesktop.org/software/systemd/man/systemd.unit.html#Specifiers
[8]
```name=fb-edenfs-logs@.service
[Service]
Environment=EDENFS_CONFIG_DIR=%f
ExecStart=/bin/sh -c ' \
set -e; \
set -u; \
\
/bin/mkdir -p -- "$${EDENFS_CONFIG_DIR}/logs""; \
'
```
```name=fb-edenfs@.service
[Unit]
After=fb-edenfs-logs@%i.service
Requires=fb-edenfs-logs@%i.service
```
[9] fd0ec39d38/src/basic/log.c (L560-L639)
Reviewed By: simpkins
Differential Revision: D13422459
fbshipit-source-id: 57c575a6f377812caa2a79168778576c6ccff33e
Summary:
I want to use fake_edenfs to test logging via EdenFS' systemd service. Make fake_edenfs and the real edenfs use similar logic to determine the log file path.
This diff should not change behavior for the real edenfs.
Reviewed By: simpkins
Differential Revision: D13424470
fbshipit-source-id: d0c2e035fdb5884dbd2d9704c7e0244d35e052f2
Summary:
Update some of the test code to use `Path.read_text()`, so we can eliminate
`eden.cli.util.read_all()`
Reviewed By: strager
Differential Revision: D13374245
fbshipit-source-id: 3399923b60ae78a4f7ea57367d097697c8b9c1cb
Summary:
D13422460 made TemporaryDirectoryMixin required for classes deriving from ServiceTestCaseBase, but ServiceTestCaseBase expressed this requirement poorly. Make the dependency explicit by making ServiceTestCaseBase derive from TemporaryDirectoryMixin.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13454141
fbshipit-source-id: e07745cfd2a364da5011371fe8671d94d32c6798
Summary:
I noticed some Managed service tests behaving strangely on my dev server. They behave as if they are being managed by systemd.
I noticed that `systemctl --user status 'fb-edenfs@*.service'` showed a bunch of tmp-eden_test services. Since the Managed tests don't create an isolated systemd instance, they are starting and stopping services on my real systemd!
This behavior is caused by me setting service.experimental_systemd=true in my ~/.edenrc (D13371186).
Fix the odd behavior of these tests by preventing them from reading ~/.edenrc. Also do the same for /etc/eden.
Reviewed By: simpkins
Differential Revision: D13422460
fbshipit-source-id: b8a4cbabe55b75b34729d4122ba804cd7d3297a2
Summary:
The code for invoking 'eden start' is duplicated by test_daemon_command_arguments_should_forward_to_edenfs_without_leading_dashdash. I am going to add more required flags for 'eden start' tests, so the duplication is a problem. Make this test reuse spawn_start.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13453511
fbshipit-source-id: 72293ed56ca90fce1cc1fa33d996edd08ca15767
Summary:
While trying to reproduce a failing test on Sandcastle (Facebook's CI), I encountered a bug: when run as root, StartWithRepoTestGit test_eden_start_with_systemd_mounts_checkouts failed with permission denied errors:
```
$ sudo -Hi
root$ SANDCASTLE=1 EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE=unmanaged ./buck-out/gen/eden/integration/integration#binary.par -r StartWithRepoTestGit.test_eden_start_with_systemd_mounts_checkouts
[snip]
fb-edenfs@tmp-eden_test.j62w_nfx-homedir-local-.eden.service: Main process exited, code=exited, status=70/SOFTWARE
[snip]
root$ less /var/log/messages
[snip]
Dec 10 20:05:11 devvm3761.prn2.facebook.com sh[3668152]: I1210 20:05:11.781113 3668152 main.cpp:280] edenfs exiting successfully
Dec 10 20:05:36 devvm3761.prn2.facebook.com sh[3669439]: I1210 20:05:36.908677 3669439 main.cpp:153] Running in experimental systemd mode
Dec 10 20:05:36 devvm3761.prn2.facebook.com sh[3669439]: W1210 20:05:36.910221 3669439 EdenConfig.cpp:362] error accessing config file /tmp/eden_test.j62w_nfx/homedir/.edenrc: Permission denied
Dec 10 20:05:36 devvm3761.prn2.facebook.com sh[3669439]: error creating /tmp/eden_test.j62w_nfx/homedir/local/.eden: boost::filesystem::filesystem_error: boost::filesystem::create_directory: Permission denied: "/tmp/eden_test.j62w_nfx/homedir/local/.eden"
```
edenfs is dropping its permission to my regular user. SUDO_ variables [1] propagate to edenfs, and edenfs calls `setuid($SUDO_UID)`. Clearing SUDO_UID manually fixes the test:
```
$ sudo -Hi
root$ env --unset SUDO_UID SANDCASTLE=1 EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE=unmanaged ./buck-out/gen/eden/integration/integration#binary.par -r StartWithRepoTestGit.test_eden_start_with_systemd_mounts_checkouts
[snip]
Ran 1 test in 14.720s
OK
```
According to systemd's documentation, services should have a mostly-empty environment [2]. It looks like the systemd user manager relies on this (because it's normally run via user@.service) and doesn't sanitize its environment before forking service processes.
Fix the bug by cleansing the environment when running systemd manually. This prevents SUDO_UID and other environment variables from propagating to services.
---
By coincidence, this change fixes the original bug I was trying to reproduce. On Sandcastle, `SUDO_COMMAND` is set to a long string with plenty of "special" characters (spaces, quotes, backslashes, equal signs, etc.). systemd barfs when shuffling the environment block around:
```
Reloading.
Failed to parse environment entry: "env=SUDO_COMMAND=/bin/bash -c SANDCASTLE_INSTANCE_ID=1488557959 [snip]\'\\\'\' \'\\\'\'\\--collection\'\\\'\' \'\\\'
Unknown serialization item '' \'\\\'\'\\--no-stress-run-sub-run\'\\\'\' \'\\\'\'\\--extended-tests\'\\\'\'\''
```
By not setting SUDO_COMMAND, we avoid this systemd bug.
---
[1]
```
root$ env | grep SUDO
SUDO_GID=100
SUDO_COMMAND=/bin/bash
SUDO_USER=strager
SUDO_UID=6986
```
[2] https://www.freedesktop.org/software/systemd/man/systemd.exec.html#Environment%20variables%20in%20spawned%20processes
Reviewed By: simpkins
Differential Revision: D13413110
fbshipit-source-id: a91d70f33c93e034bdef5573451d528a255e5fc1
Summary:
Now that FileInode read and write operations are stateless via BlobAccess and OverlayFileAccess,
EdenFileHandle no longer provides any value. Remove it. This also fixes eden's shutdown timeout
when a file handle is open and paves the way for FUSE_NO_OPEN_SUPPORT.
Reviewed By: strager
Differential Revision: D13325137
fbshipit-source-id: 71ed47a7c997f5035b4394ccb311f94332ecd8c2
Summary:
D13366696 made some tests fail. These tests assume stderr doesn't print extra messages.
Make these tests tolerate extra messages on stderr, so extra logging doesn't make them fail.
Reviewed By: simpkins
Differential Revision: D13409510
fbshipit-source-id: 7c80766dd3978bb78c9a6fc8ed23310c2ee744e3
Summary:
I want to allow opting into systemd using a setting in ~/.edenrc. Since should_use_experimental_systemd_mode is a global function, it can't read any configs. Make the config file visible to should_use_experimental_systemd_mode by moving it into EdenInstance.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13370823
fbshipit-source-id: b604db66954d0a08973030daae38bf6b1433e821
Summary:
If the user opted into systemd integration, make 'eden restart' restart the systemd service.
Don't support --graceful yet. --graceful will be implemented in a later diff. Mark it as explicitly unsupported for now.
Reviewed By: chadaustin
Differential Revision: D13271438
fbshipit-source-id: e505f00cbc337a2bf4da77bdea4b8faba063607c
Summary:
This made it through landcastle because of our current ASAN flakiness
issue on sandcastle marking almost all integration tests as disabled.
Reviewed By: strager
Differential Revision: D13385128
fbshipit-source-id: 70dee16fc5c99819fe121f4b96f7307d05ac24e6
Summary:
When you run 'eden start' without systemd integration, edenfs writes startup logs to the terminal to let users know that stuff is happening:
```
$ eden start
Starting edenfs (dev build), pid 2792025
Opening local RocksDB store...
Opened RocksDB store in 0.95 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2792025)
Logs available at /data/users/strager/.eden-dev/logs/edenfs.log
```
These startup logs are also used by various tests (especially 'eden restart's tests).
Make the same thing happen when running 'eden start' with systemd integration, improving the user experience and making some tests work:
```
$ EDEN_EXPERIMENTAL_SYSTEMD=1 \
./buck-out/gen/eden/cli/eden.par start \
--daemon-binary "${PWD}/buck-out/gen/eden/fs/service/edenfs"
Starting edenfs (dev build), pid 2800760
Opening local RocksDB store...
Opened RocksDB store in 0.693 seconds.
Remounting 1 mount points...
Successfully remounted /data/users/strager/fbsource-dev
Started edenfs (pid 2800760)
```
Reviewed By: wez
Differential Revision: D13241979
fbshipit-source-id: de79b714e42b690fdab7c21d9add46bc2da35328
Summary: Several tests for 'eden start', 'eden stop', and 'eden status' need to pass command-line arguments to fake_edenfs. With systemd support enabled, make 'eden start' forward daemon arguments to fake_edenfs, making these tests pass.
Reviewed By: wez
Differential Revision: D13249891
fbshipit-source-id: 9008a361fce7a5629535cc9d245b86073ee70826
Summary:
getSHA1 is only handling std::system_error. If another kind of exception is thrown, it's never caught and edenfs crashes.
Calling EdenMount::getInode with a path such as "./hello" will cause a std::domain_error to be thrown. Since std::domain_error is not derived from std::system_error, `getSHA1ForPathDefensively(["./hello"])` crashes edenfs.
Fix the crash by forwarding all exceptions over Thrift, not just std::system_error-s.
Reviewed By: simpkins
Differential Revision: D13386450
fbshipit-source-id: 06262dad30a5508ed482c9e8979b61aa9643280a
Summary:
On Sandcastle's continuous builds, many EdenFS tests are flaky. D13366696 added logging, but the logs don't tell us much:
```
==24060==AddressSanitizer: libc interceptors initialized
|| `[0x10007fff8000, 0x7fffffffffff]` || HighMem ||
|| `[0x02008fff7000, 0x10007fff7fff]` || HighShadow ||
|| `[0x00008fff7000, 0x02008fff6fff]` || ShadowGap ||
|| `[0x00007fff8000, 0x00008fff6fff]` || LowShadow ||
|| `[0x000000000000, 0x00007fff7fff]` || LowMem ||
MemToShadow(shadow): 0x00008fff7000 0x000091ff6dff 0x004091ff6e00 0x02008fff6fff
redzone=16
max_redzone=2048
quarantine_size_mb=256M
thread_local_quarantine_size_kb=1024K
malloc_context_size=30
SHADOW_SCALE: 3
SHADOW_GRANULARITY: 8
SHADOW_OFFSET: 0x7fff8000
==24060==Installed the sigaction for signal 11
==24060==Installed the sigaction for signal 7
==24060==Installed the sigaction for signal 8
==24060==T0: FakeStack created: 0x7f2a640b4000 -- 0x7f2a64bbd000 stack_size_log: 20; mmapped 11300K, noreserve=0
==24060==T0: stack [0x7fff12e4e000,0x7fff1364e000) size 0x800000; local=0x7fff1364d298
==24060==AddressSanitizer Init done
AddressSanitizer ignores mlock/mlockall/munlock/munlockall
==24208==Processing thread 24060.
==24208==Stack at 0x7fff12e4e000-0x7fff1364e000 (SP = 0x7fff1364c3f8).
==24208==TLS at 0x7f2a6a7e3600-0x7f2a6a7e46c0.
==24208==DTLS 18 at 0x4f80004920000010-0x5280004820000012.
Tracer caught signal 11: addr=0x0 pc=0x7f2a69e830e0 sp=0x7f2a4ae85cd0
==24060==LeakSanitizer has encountered a fatal error.
==24060==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1
==24060==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)
```
Facebook's test infra is disabling many of EdenFS' tests. This has already caused regressions to land [1]. We need to get these tests passing and enabled ASAP.
The current hypothesis is that LSAN's leak detection is crashing. Disable the leak detection, hopefully making tests pass again. (We still want to find and fix the root cause, of course!)
[1] T37784916: test_rage_output is failing: AssertionError: 'General EdenFS Statistics' not found
Reviewed By: simpkins
Differential Revision: D13385022
fbshipit-source-id: c442146c39ce84c19fc53916aef421cece6d8b40
Summary:
edenfs's privhelper process needs the CAP_SYS_ADMIN capability [1] in order to manage mount points. Give it this capability by invoking edenfs as the `root` user using `sudo`.
Known issues:
* `sudo` must be passwordless in order for fb-edenfs@.service to start.
* systemd can't kill all of fb-edenfs@.service's processes, so `systemctl stop` is unreliable for example.
[1] https://manpage.me/index.cgi?q=capabilities&apropos=0&sektion=0&manpath=CentOS+7.1&arch=default&format=html
Reviewed By: chadaustin
Differential Revision: D13113450
fbshipit-source-id: 01b89521cab371b5017fab6fbd38d55eea599c46
Summary:
I want to reuse assert_systemd_service_is_active in another test. Move it (and the related assert_systemd_service_is_stopped function) so it can be reused.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13327802
fbshipit-source-id: 022c3ed3b9e8f04ef1156c2bb4b3deda662439e4
Summary: 'eden stop' should behave like 'systemctl stop'; the service should be stopped after 'eden stop' returns. Add a test which verifies this.
Reviewed By: chadaustin
Differential Revision: D13288819
fbshipit-source-id: 5b836c8ac7c5eb97c484195496f38c7cf70c84dc
Summary:
On Sandcastle's continuous builds, many EdenFS tests are flaky:
```
EdenCommandError: eden command [/mnt/btrfs/trunk-hg-fbcode-fbsource-303-1544100804/fbcode/buck-out/dev/gen/eden/cli/eden.par --config-dir /tmp/eden_test.udfpngq0/homedir/local/.eden --etc-eden-dir /tmp/eden_test.udfpngq0/etc-eden --home-dir /tmp/eden_test.udfpngq0/homedir repository main /tmp/eden_test.udfpngq0/repos/main] returned non-zero exit status 1
stderr=b'Tracer caught signal 11: addr=0x0 pc=0x7f65cea990e0 sp=0x7f65afb21cd0\n==1698412==LeakSanitizer has encountered a fatal error.\n==1698412==HINT: For debugging, try setting environment variable LSAN_OPTIONS=verbosity=1:log_threads=1\n==1698412==HINT: LeakSanitizer does not work under ptrace (strace, gdb, etc)\n'
```
I failed to reproduce this issue on my own dev server, and I failed to reproduce this issue by running 'buck test' on a Sandcastle worker.
As a last resort, make LSAN report more stuff in an effort to track down the cause of this failure.
Aside from extra logging, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D13366696
fbshipit-source-id: e052700dbc4ed9d30f864b8d2dc5ccad27e1a281
Summary:
Add the plumbing necessary to make 'eden start' start a systemd user service. This is only enabled if you opt in using EDEN_EXPERIMENTAL_SYSTEMD.
Currently, only fake_edenfs works. The real edenfs doesn't work yet because it needs root access to configure mount points.
'eden restart', 'eden stop', etc. are not affected by this diff (and are probably broken with EDEN_EXPERIMENTAL_SYSTEMD enabled).
Reviewed By: simpkins
Differential Revision: D10849390
fbshipit-source-id: c087a6498951ff100e5c80bd07ad869b2709e1b3
Summary:
Stop holding a reference count to the TreeInode while a directory
handle is open. This allows eden to shut down while a directory handle
is open.
Reviewed By: strager
Differential Revision: D13287701
fbshipit-source-id: a24f32a1ac40b6c19bc5864aa5f5785f3016361b
Summary:
If a file was partially truncated, it would not always be marked as
materialized. During materialization, the SHA-1 would be cached,
but not invalidated after the truncation.
Write tests that ensure that both ftruncate and O_TRUNC mark files as
modified.
Reviewed By: simpkins
Differential Revision: D13329102
fbshipit-source-id: f09fdc5f11f1da25e1b4453de1b29d1390b3dc71
Summary:
Add some skipped tests that illustrate that Eden cannot shut down when
file handles are open. They will be enabled later in this stack as the
root cause is fixed..
Reviewed By: strager
Differential Revision: D13287587
fbshipit-source-id: cd8d79896127676853c183bb4b86c0d586ce511e
Summary: I'm about to add some more integration tests for unmounting.
Reviewed By: strager
Differential Revision: D13279068
fbshipit-source-id: e84580497f22b9dc6d5a04835dc4beede52a07fd
Summary: CLI tests use shutil.rmtree to clean up temporary directories. Reuse TemporaryDirectoryMixin which is more robust against errors and supports EDEN_TEST_NO_CLEANUP.
Reviewed By: chadaustin
Differential Revision: D13268108
fbshipit-source-id: d77e95a2def0dceb34cf14e19c0c0c0e3aeef3f2
Summary:
systemd uses cgroups to contain a service's processes.
When EdenFS' tests run systemd on Sandcastle, the systemd process creates its cgroups as children of Sandcastle's cgroup. For example, the test-SystemdServiceTest.service service created in test_running_simple_service_is_active has the following cgroup on Sandcastle:
/sandcastle-job/command/test-SystemdServiceTest.service
Unfortunately, systemd uses predictable cgroup names based on the service name. A service managed by one systemd instance shares the same cgroup as a same-named service managed by a different systemd instance. For example, if multiple systemd tests run concurrently on Sandcastle (e.g. during stress-testing), `systemctl --user stop foo.service` from one run will kill the processes of a different run's active foo.service. This causes systemd tests to be flaky on Sandcastle.
Fix the conflict by creating a unique cgroup per systemd instance. For example, separate runs of test_running_simple_service_is_active might use the following cgroups for the test service:
/sandcastle-job/command/edenfs_test.1_qat2xv/test-SystemdServiceTest.service
/sandcastle-job/command/edenfs_test.bbd3gj_o/test-SystemdServiceTest.service
Reviewed By: simpkins
Differential Revision: D13016626
fbshipit-source-id: 8535dc14a06bdb403c926b111cad4aed6c8ec3e3
Summary:
Check that the next-inode-number file exists and that the inode number it
contains is actually larger than all existing inode numbers. Replace it with
correct data if the file does not exist, is corrupt, or contains an incorrect
inode number.
Reviewed By: chadaustin
Differential Revision: D12955093
fbshipit-source-id: 3d26fb475535577d9a2697bbd575fba350766d01
Summary:
Update fsck to extract data for orphan inodes to a lost+found directory in the
fsck log directory, and remove them from the overlay. This will allow users
to recover the orphan file data if they want, and remove it otherwise.
Reviewed By: chadaustin
Differential Revision: D12955094
fbshipit-source-id: 9783452fce4060b9c5c48b3d48dd1f70294211c6
Summary:
Add several more files to the basic snapshot, so we can test more cases
in the fsck tests:
- Materialized, new, and unmodified symlinks
- A deeper directory tree of directory inodes that are not materialized (still
have a source control tree hash) but have children inodes allocated and are
therefore present in the overlay.
- A socket in a slightly deeper directory so we can test behavior of sockets
inside directories that have been corrupted..
As before I have replaced the older basic snapshot instead of adding a new
one, since the Eden data storage formats have not changed since the last
snapshot was created.
Reviewed By: chadaustin
Differential Revision: D13164658
fbshipit-source-id: d117c9cc336709044de212637c03140dfadd9a96
Summary:
This updates the basic snapshot code to include a couple slightly deeper
directories. I plan to use this in the fsck tests to verify handling of
orphan directories that contain subdirectories.
Normally it would be preferable to keep the old `basic-20181030` snapshot, and
simply add this new snapshot without than replacing the old one. However, I
don't think we have made any meaningful changes to our on-disk storage formats
since the previous snapshot was generated, so it seems okay to just delete the
old snapshot.
Reviewed By: strager
Differential Revision: D13151861
fbshipit-source-id: e6b7583beecb5d9cc55271ad2dea8d36980542d1
Summary:
Add a helper class for maintaining the list of expected files.
I plan to use this for the fsck tests, so I can more easily modify the
snapshot's expected contents based on how we expect fsck to repair various
types of overlay corruption. This also helps slightly simplify the code that
constructs the expected file list.
Reviewed By: chadaustin
Differential Revision: D13095918
fbshipit-source-id: 57686e82d1bf7f23a92eda0ed4d66623a3f58840
Summary:
Add basic high level logic to fsck to begin fixing problems that are found.
This adds basic checks to decide if we should fix errors or not.
If errors are found and need to be fixed, this creates a new directory inside
the checkout state directory in `.eden` to record the actions taken by this
fsck run. This directory will contain a log file that records the actions
taken. In the future the fsck logic will also use this directory to store
copies of the corrupted inode data, and can store extracted orphan inode data
here as well.
Reviewed By: wez
Differential Revision: D12955044
fbshipit-source-id: 06c1e17a0a51fa5e2c0f2aab83b367b9358fd004
Summary:
After sending SIGTERM to systemd, increase the wait timeout from 3 seconds to
15 seconds. The previous 3 second timeout was easy to hit in practice on our
continuous build infrastructure.
If systemd does not exit within 15 seconds after we send SIGTERM, send SIGKILL
and then try waiting on it again for up to 3 seconds. Forcefully killing it
seems preferable to leaving the process hanging around after the tests exit.
Reviewed By: strager
Differential Revision: D13159491
fbshipit-source-id: debce21f2f202fb7cfa4a53120dcb2b2b35ccbe3
Summary:
Replace some member variables in
`_TransientUnmanagedSystemdUserServiceManager` with locals, to make pyre
happy.
Previously pyre complained that these member variables are not initialized in
`__init__()`. These variables are only needed temporarily during
`__enter__()` (and in some clean up closures), so just use them as local
variables instead.
Reviewed By: strager
Differential Revision: D13135257
fbshipit-source-id: 76f2bdc4b7b36d2102ad8dab4a60722a03197fab
Summary:
If invoked with `--log-target=console` systemd will log to stderr even if it
is not a tty.
This changes the tests to pass in `--log-target=console` rather than creating
a pty and forwarding I/O from it in a separate background thread.
Reviewed By: strager
Differential Revision: D13135258
fbshipit-source-id: 11dfe0711adaa62cedba2882045d8088e0df5499
Summary: test_processes_of_forking_service_includes_all_child_processes is failing on some machines. The test assumes /sys/fs/cgroup/ is a cgroups v2 mount point, so it fails if /sys/fs/cgroup/ is a directory containing cgroups v1 mount points. Disable the test on machines where cgroups v1 is detected.
Reviewed By: simpkins
Differential Revision: D13112836
fbshipit-source-id: 7921604707a0c1fe81a82c87e767a6a99cdd6206
Summary: Fix the remaining set of errors reported by pyre and mypy.
Reviewed By: strager
Differential Revision: D13086855
fbshipit-source-id: 4c2b21352f94ef225a5555aef0f6b95b92e56f6d
Summary:
Our code in testcase.py mucks around with test case classes in several ways
that mypy doesn't like. In particular, mypy does not currently support
replacing methods on classes, and it also does not understand dynamic base
classes. pyre also trips up on some of these changes, although in different
ways.
This updates testcase.py so that mypy and pyre no longer report errors on it,
often just by suppressing the errors. Also fix similar errors in
service_test_case.py around replicating test classes.
Reviewed By: strager
Differential Revision: D13086856
fbshipit-source-id: af446dd13791f5da50b09657012db95c2bcf0e39
Summary:
This updates the integration tests to add type annotations to most functions
that were missing annotations.
In particular this is needed to make pyre happy, as it complains if subclasses
override methods from their parent class and do not specify type annotations
if the parent class did have annotations.
This diff also contains some minor changes to hg_extension_test_base.py to
explicitly declare some abstract methods that it uses. This was also
necessary to make pyre happy about this ocde.
Reviewed By: strager
Differential Revision: D13051097
fbshipit-source-id: 77567ed2f4d3050f93acefb52e688932d276d587
Summary:
Drop the `stdout` and `stderr` arguments, so that this method always return a
string. Change callers that were previously calling this method with
`stderr=None` to use the `run_hg()` method instead of `hg()`. `run_hg()`
returns a `subprocess.CompletedProcess` object.
This change simplifies the python type checking, and fixes several existing
type checking errors in the code. Even though most call sites could be
guaranteed that this function would return a `str`, the type checker wasn't
smart enough to tell that the return type would be fixed based on the argument
values, and so it assumed the result always needed to be checked for `None`.
This also updates the `GitRepository.git()` method in a similar fashion.
However, that was a simpler change since it already returned a `str` in all
cases.
Reviewed By: strager
Differential Revision: D13078095
fbshipit-source-id: a8def2a33edc865ac40279bbcb3ada4dade68374
Summary:
Restructure storage_engine_test so that the base class derives from
EdenTestCase.
Reviewed By: strager
Differential Revision: D13051096
fbshipit-source-id: e89c4b56e361460b2457d1c2e6a22727a25d7646
Summary:
I want to use Hypothesis to fuzz some CLI code. Move EdenFS' Hypothesis configuration out of eden.integration.lib.testcase and into a place where both CLI tests and integration tests can use it.
This diff should not change behavior.
Reviewed By: wez
Differential Revision: D12813285
fbshipit-source-id: 3a1badd1e18b0e070295ea03dcb24be166cd42c1
Summary:
_transient_unmanaged_systemd_user_service_manager has a few inner functions and a few variables shared between these inner functions. The function is pretty long and hard to follow.
I think a class is more familiar than closures. Refactor _transient_unmanaged_systemd_user_service_manager into a class: turn inner functions into methods, and shared variables into instance attrs.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13017642
fbshipit-source-id: d20f7476142fa5d7ba0ae09291228ec63e127338
Summary:
If you enable type checking of util.py, mypy complains about _remove_readonly having the wrong type:
eden/integration/lib/util.py:87: error: Argument "onerror" to "rmtree" has incompatible type "Callable[[Callable[[str], Any], str, Tuple[Type[Any], BaseException, TracebackType]], None]"; expected "Optional[Callable[[Any, _PathLike[str], Any], Any]]"
Fix the error by making _remove_readonly compatible with onerror's signature in typeshed [1]:
overload
def rmtree(path: _AnyPath, ignore_errors: bool = ...,
onerror: Optional[Callable[[Any, _AnyPath, Any], Any]] = ...) -> None: ...
This diff should not change behavior.
[1] 4dc21f04dd/stdlib/2and3/shutil.pyi (L90-L92)
Reviewed By: simpkins
Differential Revision: D13086137
fbshipit-source-id: 222e5fa2e06a26464483a0f09545089a7ecc5234
Summary:
While debugging flakiness in test_running_simple_service_is_active, I noticed that sometimes ActiveState=active despite the service process being dead. This can happen if the service process is killed outside systemd. When this happens, SubState=exited instead of the expected SubState=running.
Prevent false positives in SystemdServiceTest tests by checking SubState in addition to ActiveState.
Reviewed By: simpkins
Differential Revision: D13032619
fbshipit-source-id: 2d5754291a19290d29a817115923e9cc5efc90ab
Summary:
A few times, I've needed to manually make _is_system_booted_with_systemd return False in order to emulate how Sandcastle behaves. Make this easier by introducing an environment variable, EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE, which allows choosing how to start `systemd --user` when running tests.
When EDEN_TEST_FORCE_SYSTEMD_USER_SERVICE_MANAGER_TYPE is unset, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D13031404
fbshipit-source-id: ecbd5f90ff55f4dffa47ba797686db5c25a7198c
Summary:
Add type annotations for class member variables. The pyre type checker has
some limited automatic type detection for member variables set in
`__init__()`, but in general it expects member variables to be explicitly
declared at the top-level of the class.
Reviewed By: strager
Differential Revision: D13051092
fbshipit-source-id: 080259ab3f422ffae2b908ed610062237105ccbe
Summary:
Update `EdenRepoTest` to explicitly declare its own `get_thrift_client()` that
just calls through to its parent class' implementation.
I'm not sure why, but mypy can't seem to figure out the correct return type of
this method without this extra declaration. It gets confused when this method
is called from `EdenRepoTest` subclasses, and either complains that it doesn't
know the type at all and needs an explicit type declaration, or in some cases
it thinks the type is `Optional[EdenClient]` rather than `EdenClient`.
Currently mypy doesn't complain about these errors by default, but it seems to
complain when I start adding more type information for other class members.
Adding this extra declaration makes it possible to start adding type
annotations in more places in the code without triggering new errors from
mypy.
Reviewed By: strager
Differential Revision: D13051089
fbshipit-source-id: 69776642949d293c645686ba00d5d6b71c0cd31c
Summary:
The thrift APIs accept path names and commit IDs as binary data (python bytes)
rather than unicode strings. Our python code got this wrong in several
locations. It looks like mypy didn't previously flag this since mypy doesn't
actually figure out the correct type for the thrift `client` object, and
seemed to just be largely ignoring it. I plan to update the code so that mypy
can figure out out the client type correctly. Fixing these type errors is
required to make sure we won't get type errors once that is changed.
This just simply uses `encode("utf-8")` for now. In the future the path
arguments should be converted to `pathlib.Path`, which will do a slightly
smarter conversion, and avoid errors on non-UTF-8 binary data. In the
meantime, I believe that just using `encode("utf-8")` preserves what the
thrift code was doing implicitly before, and does not make handling of
non-UTF-8 data any worse than it was before.
Reviewed By: strager
Differential Revision: D13051094
fbshipit-source-id: 94cb62f3dd78b8e854a72a392fe8fdfad5ffd4cb
Summary:
eden/benchmarks/ is a good place to write external benchmarks for FUSE
and Thrift APIs.
(Note: this ignores all push blocking failures!)
Reviewed By: wez, strager
Differential Revision: D12970347
fbshipit-source-id: 81fa35897fc6e626254eea6e1ee44a8d35052261
Summary:
As we start to build out both FUSE and Thrift benchmarks, we'll want a
standard library. Introduce a benchharness and have both the thrift
sha-1 and parallel_open_close benchmarks use it.
(Note: this ignores all push blocking failures!)
Differential Revision: D12969306
fbshipit-source-id: 89c8bbcc37d53560decffb9281af4aba20345787
Summary:
Use `Path` objects and its associated APIs in most locations rather than using
plain strings.
Reviewed By: wez
Differential Revision: D13035415
fbshipit-source-id: ef3fb72e884b1f5dbc2a9bcd9f36ba1397a7c21a
Summary:
This removes the logic that would completely avoid reporting most of the
integration tests if we could not run Eden.
This used to be an issue in the past before we could run Eden in our CI build
environment. To avoid noise in the test reports with hundreds of tests
reported as skipped we would simply hide these tests entirely. Our CI
environment does run all the Eden tests now, so this is no longer an issue.
I'm removing this logic now since it confuses the pyre type checker. Rather
than modifying the code to work around the problem, it seems preferable to
just remove this logic entirely. Now that the CI environment should be
running all the tests it seems like we probably would want to be notified if
something goes wrong and the tests think they cannot successfully run Eden for
some reason.
Reviewed By: wez
Differential Revision: D13035412
fbshipit-source-id: 73c0ce7fa759a1fbb9aff08ccad475b36f332646
Summary:
My changes to command line argument parsing in D12927803 conflicted with one
of the new tests added in D10863987.
My changes made `edenfs` and `fake_edenfs` reject any trailing non-option
arguments. The new test runs `fake_edenfs` with an extra
`--commandArgumentsLogFile` argument asking it to log all non-option arguments
to a file.
This diff updates `fake_edenfs` to allow non-option arguments when
`--commandArgumentsLogFile` is specified.
Reviewed By: strager
Differential Revision: D13014079
fbshipit-source-id: 82ec5fd758716c2b66ac0738b7aacdf884f31233
Summary:
Improve the way `eden fsck` processes its arguments. Previously it required a
single checkout path as an argument. If an `--overlay` argument was specified
it used this instead of the checkout path to find the directory to check, but
the checkout path argument was still required (but was ignored).
This changes the code to accept one or more paths as arguments. These paths
can either be the path to an Eden checkout (which does not have be currently
mounted), or the path to a checkout state directory inside the `.eden` state
directory. The `fsck` code automatically figures out what type of directory
the argument refers to and processes it correctly.
If no paths are specified `eden fsck` now automatically checks all configured
checkouts that are not currently mounted.
Reviewed By: wez
Differential Revision: D12955041
fbshipit-source-id: c37bc6752746d8ecd0f4a672d0d3b25d1c3a4fa1
Summary:
This is the basic framework for some new tests for `eden fsck` that work by
unpacking one of the existing saved snapshot files, breaking it in various
ways, and then running `eden fsck` to fix it.
Using an existing snapshot file rather than creating a new mount point on the
fly in the test has a few advantages:
- This lets us confirm that newer versions of Eden can still correctly repair
and mount old file formats, even if we update the mount point data formats
in the future.
- Unpacking a snapshot is much faster than starting Eden, creating an hg
repository, cloning a new Eden checkout from the repo, and then unmounting
the checkout.
- The inode number allocations for the snapshot are fixed, which makes it
easier for the test code to manipulate specific inode numbers and always
know which path this refers to. If we created a mount point on the fly we
can't guarantee ahead of time which inode numbers would map to each file,
and we would need to do more work to look up this information after creating
the checkout.
These tests are pretty basic at the moment, but I plan to expand them as I
check in more of the fsck logic to repair errors.
Reviewed By: wez
Differential Revision: D12955045
fbshipit-source-id: 5d5a96cec812f8e72caf93e57bf0f1311e28aab8
Summary:
Sometimes users accidentally run `edenfs start` or `edenfs restart` instead of
`eden start` or `eden restart`. This adds a new `--edenfs` flag to the
`edenfs` binary, and asks users if they meant to run `eden` instead if they do
not pass in this flag.
This used to be less of a problem since `edenfs` required users to also
explicitly specify several other configuration flags (like `--edenDir`).
However `edenfs` can not automatically figure out these settings, so these
flags are no longer required. Therefore `edenfs` would still try to start
normally when invoked with `edenfs restart`, since it did not require these
flags and it did not complain about unhandled command line arguments.
Reviewed By: wez
Differential Revision: D12927803
fbshipit-source-id: dbf7ce2449c391ca218652439eb68ff43c2ebd46
Summary:
When unpacking a snapshot, rewrite Eden's inode metadata table to change the
UID and GID values to the current user.
This is needed so that the current user can access files inside the mounted
checkout correctly.
Reviewed By: wez
Differential Revision: D12966640
fbshipit-source-id: eec4aba690117bf7b8f944221b31b7c7cc66fc0c
Summary:
Add a script for unpacking saved snapshot files.
This unpacks the tarball and then fixes up absolute paths inside the snapshot
so it will work at the unpacked location. It also emits a small helper script
to make it easier to invoke eden for the snapshot.
Reviewed By: wez
Differential Revision: D12927842
fbshipit-source-id: f46c07a79894877842f2a78aa0924fd66e4c8969
Summary:
Thanks to some bpf tracing by strager, we traced the ESTALE response to
`d_splice_alias` and noted this comment above the implementation in the kernel:
> If a non-IS_ROOT directory is found, the filesystem is corrupt, and
> we should error out: directories can't have multiple aliases.
Well, our magic `.eden` directory is a directory with aliases and we were
seeing the error trigger on that dir. So, this diff replaces hardlinking
directories into each tree with a hardlink to a symlink in each tree!
At mount time we create `.eden/this-dir` as a symlink to `/abs/path/to/mount/.eden`
so that `readlink("/abs/path/to/mount/sub/dir/.eden/socket")` still
resolves as it did prior to this diff.
Reviewed By: strager
Differential Revision: D12954819
fbshipit-source-id: 7f3b1b53f2bd5b9c51e64055fc34110657a19110
Summary:
When giving arguments to the edenfs program, 'eden start' only strips `--` if it is the first positional argument. The position of `--` shouldn't matter as long (as it's before any options).
In other words, given the following command invocation:
$ eden start --opt-a arg-b -- --opt-c arg-d
Current behavior:
* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b -- --opt-c arg-d` is passed to edenfs as four arguments.
Desired behavior:
* `--opt-a` is interpreted as an option for the 'eden start' command.
* `arg-b --opt-c arg-d` is passed to edenfs as three arguments.
* The `--` argument is not passed to edenfs.
Fix 'eden start' by stripping `--` regardless of its position.
Reviewed By: chadaustin
Differential Revision: D10863987
fbshipit-source-id: 094da3f3674e775fe3e7eb8441ec95c37c34ff05
Summary:
Sandcastle has several cases where we chown the entire
repository which performs terribly on Eden. As a workaround we have a
command to do this in eden without loading all the files.
Reviewed By: chadaustin
Differential Revision: D12857956
fbshipit-source-id: 36cebcc710fbcf4e1eb265df901513cf50a227b9
Summary:
The `test_hg_clone_non_eden_repo_within_eden_repo()` test had been disabled
when running with treemanifest enabled since the treemanifest code did not
correctly detect which repositories should actually use treemanifest.
This issue appears to have since been fixed in the treemanifest code, so this
test passes now.
Reviewed By: quark-zju
Differential Revision: D12927801
fbshipit-source-id: b9e2e041f7eab5e24007888e2dba142e1f0b2251
Summary:
- Update mypy in fbcode to upstream version 0.641.
- Add/fix some type stubs
- Disabled some pieces that break mypy (T35976411 and T35973384)
This diff will likely cause some unit test failures, due to incorrect type annotations that should be fixed by the team owning the relevant code.
mathchecksout
Reviewed By: jeffmo
Differential Revision: D10414250
fbshipit-source-id: 282ea62aef6793a2f4baef78afd58ddd4ddf60f5
Summary: This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D12873644
fbshipit-source-id: c8eaac8bee56268b8a6479f7ea27e496a6b12d96
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Create utilities for tests to enable units within a temporary instance instance, start services, and inspect the processes inside a running service.
These utilities are currently unused, but will be used in future diffs as 'eden start', etc. grows systemd support.
Reviewed By: simpkins
Differential Revision: D10371824
fbshipit-source-id: c1e62eebb8480309ee17c49c08af73c3f5cf75d6
Summary: While creating a systemd service unit for EdenFS, I noticed a few features of the current implementation of 'eden start', 'eden stop', and 'eden restart' which are not covered by tests. Write tests for these features to ensure they don't regress with systemd integration.
Reviewed By: wez
Differential Revision: D10434379
fbshipit-source-id: 6c828a85d59179bbc4beda87e1bf2534543b60b2
Summary:
Update the snapshot framework code to add tests that mount saved snapshots
with the current edenfs build and verifies that they behave as expected.
This also changes the snapshot generation code a fair amount as well, in
order to be able to verify the snapshot data:
- We now more cleanly split out data that should be saved when the snapshot is
first generated (such as the `.eden` state directory) and data that should
be re-generated each time the snapshot is unpacked (such as the `/etc/eden`
config directory and the system configuration for `hg`).
- The code is now capable of rewriting absolute paths in the Eden state files
so it is usable in a new location after it has been unpacked.
This also updates the "basic" snapshot type to create a wider variety of types
of files and file changes, and to be able to verify the contents of a resumed
snapshot.
Reviewed By: strager
Differential Revision: D9955411
fbshipit-source-id: 76012c01016cf4d89dc611c5596a05f5e7f013ed
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Create a utility for tests to create a temporary systemd instance. It should work regardless of whether the host uses systemd to manage its services. (systemd's tooling must still be installed in order to use the utility, though.)
This utility is currently unused, but will be used in future diffs as 'eden start', etc. grows systemd support.
Reviewed By: simpkins
Differential Revision: D10286940
fbshipit-source-id: 4fbaa695bf36ac4ae44b5c12b6255514bd7143b3
Summary: spawn_status already sets the log file to stderr. Setting it again is redundant. Delete the dead store.
Reviewed By: simpkins
Differential Revision: D10434381
fbshipit-source-id: d0e4cfe9dc63ae80fc8a9e621b02a3ac31f8244a
Summary:
Since we can't seem to find a way to prevent ESTALE
from being returned by the kernel, and that only seems to occur
with readlink, add a retry to our readlink calls.
This feels gross, but not sure how we can do better :/
Reviewed By: strager
Differential Revision: D12239143
fbshipit-source-id: 04ecbdeac3b3ea3969e779da4f50cf91ccc5cd00
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Unfortunately, we can't create a new instance of systemd if a machine is not managed by systemd. However, we can trick systemd into thinking that the machine *is* managed by systemd. When we do that, the systemd instance seems to work without any issues.
Create a library which, when injected into systemd using LD_PRELOAD, makes systemd thinks the machine is managed by systemd.
The library will be used in a subsequent diff (D10286940). This diff is separate so it's easier to review.
Reviewed By: simpkins
Differential Revision: D10371831
fbshipit-source-id: b54b704dc631ebdc5091320dd6b680ca255f3ba2
Summary:
When running fake_edenfs in daemonize mode (i.e. when --foreground is not specified), the daemon process inherits stdout and stderr from the parent and never closes them. This has two consequences:
* The daemon process holds a reference to stdout/stderr even after the parent exits. This causes tests to hang if they slurp stdout/stderr after waiting for the parent process to exit. (I discovered this issue when adding tests for 'eden start' in D10434379.)
* fake_edenfs logs to stdout/stderr after 'eden start' has returned. I think this behavior is surprising, but it doesn't really hurt anyone because fake_edenfs is just a developer tool.
Promptly close stdout and stderr after startup finishes by redirecting further output to a log file. This mimics the behavior of the real edenfs.
Note: stdin is still leaked. stdin will be addressed in D10442346.
Reviewed By: chadaustin
Differential Revision: D10439299
fbshipit-source-id: a7a423aa901982f3d35e81940ffc27db3a985d76
Summary:
No one uses --logPath with fake_edenfs. Remove the dead flag, inlining its default value.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439298
fbshipit-source-id: 068e5677db1bec81a8da174b136ef9f5b54c7e34
Summary:
The daemonizeIfRequested is implemented in the main edenfs process as well as in fake_edenfs. Consolidate the almost-identical implementations.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439300
fbshipit-source-id: 6253668896de0f8393b92f4aa5ee8a8a2fcbfb31
Summary:
Right now, 'eden start' spawns edenfs directly (using `os.exec`). In the future, 'eden start' will spawn edenfs using systemd. 'eden restart', 'eden stop', and 'eden status' might behave differently depending on how the edenfs process was spawned.
Parameterize some tests on how fake_edenfs was originally started. This flexibility lets us easily spawn fake_edenfs with 'systemctl start' (systemd) in the future. For now, spawn fake_edenfs in ad-hoc mode (i.e. directly without 'eden start') or in managed mode (i.e. indirectly with 'eden start').
Reviewed By: chadaustin
Differential Revision: D10414995
fbshipit-source-id: 2ee0d5df8a605ca9d7da8f6eeca1fc171a8342e8
Summary:
RestartTest uses 'eden restart' during setup to start the fake_edenfs process. There are two issues with doing this:
* I want to extend the tests in RestartTest so they also tests with an ad-hoc fake_edenfs process (D10414995). Restarting edenfs during setup makes no sense in that context.
* I think using 'eden restart' during setup is clunky and unintuitive. 'eden start' is more appropriate.
Refactor RestartTest's setup so it uses 'eden start' instead. Also add a test which explicitly verifies that 'eden restart' starts fake_edenfs if no daemon is already running.
Reviewed By: chadaustin
Differential Revision: D10417304
fbshipit-source-id: 20f8190026cd153dd9b539067f6f63b6bd27abed
Summary:
I want a function like `fake_eden_daemon` which instead spawns fake_edenfs using `eden start`. Refactor existing code to make such a function easier to write:
* Replace `contextlib.contextmanager` with a class called `FakeEdenFS` so its `__exit__` method can be reused.
* Rename `fake_eden_daemon` to `FakeEdenFS.spawn` so it's clearer that the function creates a process.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10415123
fbshipit-source-id: 81e6a0618214a527dc7f15cc18d4fe97dd1a957e
Summary:
Update the HgRepository.status() function in the integration tests to return
the status information as a dictionary instead of a string. Only one test was
still using the old string API.
Reviewed By: chadaustin
Differential Revision: D10503168
fbshipit-source-id: 574e4438d23bf6612a70ae5ae3174db3d464d198
Summary:
Update the eden+hg integration test code so that it uses find_executables.py
to find the post clone hook. Previously it was still using its own one-off
helper function for this.
Moving this code into find_executables.py will make it possible have the
snapshot tool also find the post clone hook.
Reviewed By: chadaustin
Differential Revision: D10503169
fbshipit-source-id: d59e3e517d19e80e0d0c997be09865519b80d34e
Summary:
Now that the import bug has been fixed for some time, it's likely few
people have cached empty files. And if they do, `eden gc; eden
restart` is a fine workaround.
On the other hand, reimporting empty files gums up the importer, and
I've seen several people recently complaining about performance that
could be partially attributed to the fact that their Eden needed to
verify empty files. simpkins has talked about adding a bit to the
cached blob to determine whether it needs reimporting or not, but it's
probably going to take a while for anyone to implement that, and this
reverification logic is hurting people today.
Reviewed By: strager
Differential Revision: D10456519
fbshipit-source-id: 657bc377ee16ce93494075bde4388aed59dceecf
Summary:
When you get an error, let's print the number of conflicts in each file. This will give the user some sense of how much work they have to do.
The code change is entirely in `filemerge.py`, and `tests/test-merge-conflict-count.t` adds a new test.
Reviewed By: quark-zju
Differential Revision: D9815243
fbshipit-source-id: 1b73a1db293902ac7242997a7d6ae09478344068
Summary:
Add a new sparse show command, which shows the current sparse configuration.
Use common templating, including colors, between sparse show and sparse list.
Reviewed By: phillco
Differential Revision: D10446705
fbshipit-source-id: acb539f80f625945716758c785703b46a81aa6f1
Summary: Some commands in extensions have incorrectly defined usage. Correct these.
Reviewed By: phillco
Differential Revision: D10446707
fbshipit-source-id: e606314892a31fa09f4a655042d66d87929fc2ec
Summary: D10416965 accidentally removed the import for `typing` in health_test.py. This causes tests to fail. Re-add the import to fix the tests.
Reviewed By: singhsrb
Differential Revision: D10452517
fbshipit-source-id: af58d076b7bac375c20630113cbb3972e24ffc12
Summary:
Eden supports reading the SHA-1 of a file via getxattr. Unfortunately,
it returned ENOSYS if you called getxattr on a directory inode. This
caused FUSE to fail all future getxattr requests with EOPNOTSUPP.
In addition to fixing that, this diff makes our xattr handling a
little more consistent across inodes:
- setxattr() always fails with ENOSYS
- removexattr() always fails with ENOSYS
- listxattr() is always handled by the corresponding inode class
- getxattr() is always handled by the corresponding inode class
Differential Revision: D10437723
fbshipit-source-id: a1ea1e92d3412abb15e91057becea3160a17f1e2
Summary: Instead of using tempfile.TemporaryDirectory and calling unittest.TestCase.addCleanup manually, use TemporaryDirectoryMixin which does a better job.
Reviewed By: chadaustin
Differential Revision: D10416965
fbshipit-source-id: aed9d80788d54692bc7f024a5a62523ca168bbf5
Summary:
StartupLogger handles two independent scenarios: edenfs in daemon mode, and edenfs in foreground mode. I need to add a third for systemd support: edenfs in foreground mode but with startup logs written to a file.
Refactor the existing StartupLogger into an abstract base class with two concrete derived classes: DaemonStartupLogger and ForegroundStartupLogger. This will make adding the systemd feature easy without confusing the logic for the existing scenarios.
Aside from dropping the setsid() call for ForegroundStartupLogger, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D10228156
fbshipit-source-id: fbc2c771c6fbb675779ef05ca3b7395b70b67485
Summary: D10335083 renamed `fbsparse` into `sparse` and I forgot to udpate this test.
Reviewed By: simpkins
Differential Revision: D10423192
fbshipit-source-id: a547c403a9461d6923ecfecc931df82bb7887597
Summary:
Make StopTest and HealthOfFakeEdenFSTest similar to RestartTest by putting the code to invoke 'eden --config-dir ... stop-or-status' into its own function.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10414421
fbshipit-source-id: e42cdee64ae282fa0575cf875b53ac91fa61d490
Summary:
To ease the migration from the current custom daemon setup to a systemd-managed daemon, add a way to enable the to-be-implemented systemd behavior.
* To the edenfs daemon, support the --experimentalSystemd option.
* To the Eden CLI, support the EDEN_EXPERIMENTAL_SYSTEMD environment variable. When set, invoke the edenfs daemon with --experimentalSystemd.
Aside from adding a log message, this diff does not change any behavior. This diff just sets up scaffolding for future diffs.
Reviewed By: simpkins
Differential Revision: D10248055
fbshipit-source-id: 30ca5a4bfde00ff43fd6f2a5d0282ced4f177fed
Summary:
Logging some messages to stdout and some messages to stderr has a few problems:
* stdout and stderr can get interleaved. For pipes, interleaving is avoidable for messages smaller than PIPE_BUF (512 bytes), but there's no guarantee that stdout and stderr are pipes or that the reader of the pipe reads PIPE_BUF bytes at a time.
* For the systemd integration I am developing, I need one file for stderr logs and another file for stdout logs. This is more complicated than having a single file for all logs.
* The distinction between stdout and stderr is unclear. No messages from StartupLogger are meant to be piped to another program.
Address these problems by only logging to stderr.
From a user's point of view, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10370931
fbshipit-source-id: b063ed3a66cefc846d30ce1486759cbd3f3ea29e
Summary:
Note that this is more aggressive than the tweakdefaults version; it doesn't allow named branches to be created or listed at all with the config set. As we want to delete named branches entirely, this seems reasonable.
I ported the `--new` flag, but made it optional, to make it easier to support both the core and tweakdefaults callers.
Saurabh Singh and I chatted with the releng team (Craig).
**tldr:** we can remove named branches from the UI but should keep the innards for now
Today the releng team doesn't make any branches (#continuous), but in the near-term future, they might either:
- create remote bookmarks in hg (similar to fbsource)
- create branches in SVN (which are then synced as hg branches by hgsubversion)
- We'll leave all the hgsubversion code to do this alone, just remove user-facing entry points
- They'll also need to be able to query for a commit's branch (`hg log -T '{branch}'` will still work)
- Once www is on hg, this possibility goes away and we can kill named branches for good
Reviewed By: ikostia
Differential Revision: D10401485
fbshipit-source-id: 6f2f3ae28af249bae1fdf782eb14fe7bfc472bb7
Summary:
In another diff, I want to set an environment variable within a test and restore its original value later. Code to do this exists already inside setup_eden_test, but it's hard-coded to HOME. Generalize the code and factor it into its own class so it can be used in my other diff.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D10282151
fbshipit-source-id: 063ed980889441e7b693cf7f1c84f5039b2cbd64
Summary:
In Python 3, `pexpect.spawn` objects come in two flavors: bytes and unicode. The flavor is selected using `pexpect.spawn.__init__`'s `encoding` parameter.
For the bytes flavor, `pexpect.spawn`'s instance attributes generally use `bytes`:
class spawn:
args: Sequence[bytes]
before: bytes
command: str
For the unicode flavor, `pexpect.spawn`'s instance attributes generally use `str`:
class spawn:
args: Sequence[str]
before: str
command: str
Teach the type checker (mypy) about these two flavors to reduce false positives in calling code.
Reviewed By: simpkins
Differential Revision: D10280499
fbshipit-source-id: b4475226423ef37ff5bf2bbae0eb9deac6a1099a
Summary:
RestartTest._spawn_restart dynamically types its kwargs parameter, hiding potential type errors. The kwargs parameter is unused anyway, so delete it and tidy up the related code.
This diff should not change behavior.
Reviewed By: pkaush
Differential Revision: D10281769
fbshipit-source-id: 3e7f7078edf9e47013a9eb9ba42cf51fa8dd0a16
Summary:
Update hg absorb to use the templater and display the first line of the
description of the affected commits.
Also change the default behaviour to display the changes and prompt the user to
confirm that absorb is going to do what they want. Use the new `-a` option to
get the old behaviour.
Reviewed By: quark-zju
Differential Revision: D10366510
fbshipit-source-id: 23df2e5ab2d21a0805a16e7118686a29634d2918
Summary:
Update `EdenFS.cleanup()` to only delete its `eden` directory if it was
created as temporary directory in `__init__`. If the caller specified an
explicit directory avoid deleting it.
All of the users that currently create `EdenFS` with an explicit directory
argument already perform their own clean-up of this directory if needed.
Therefore this diff does not meaningfully change the behavior of existing
programs, it simply make the API behavior less surprising.
Reviewed By: strager
Differential Revision: D10351822
fbshipit-source-id: e9ba7f5869bd2cc221a84be45df233acc7a1a660
Summary:
We go through a lot of effort to set EDENFS_FAKE_EDENFS for tests and verify its path. However, we never actually use the path! We always fall through to searching candidate paths, ignoring the value of the environment variable.
Add an early `return` so the environment variable is respected.
Reviewed By: simpkins
Differential Revision: D10369153
fbshipit-source-id: f1767a240361c6c7e1c95f93c6284a4ac95c2645
Summary: This seems to yield about 18% space savings compared to bzip2.
Reviewed By: strager
Differential Revision: D10351855
fbshipit-source-id: 58b969a8df4a9eb075526d5cc919c99ff6e8f4f3
Summary: Doctor now detects if multiple edenfs are running for the same edenDir and recommends to kill the rogue one.
Reviewed By: strager
Differential Revision: D10098157
fbshipit-source-id: 8a785a9fee0c768696f897a4950552d141012b2a
Summary:
The functionality we care about is provided by the `fbamend`
extension. Therefore, lets replace the `amend` extension with the `fbamend`
extension.
Reviewed By: farnz
Differential Revision: D10320739
fbshipit-source-id: 5700d39f488777fcc4033f60ce0a51cda15ef2ad
Summary: Several tests create a temporary directory and destroy it. Consolidate the temporary directories and their cleanup code to reduce code duplication.
Reviewed By: simpkins
Differential Revision: D10251110
fbshipit-source-id: caa5b5ad49fcb7925f63094594af4a20009d790d
Summary:
We've been shipping our RPMs with toml config support turned on for a
while now. Remove support for the old config file format. Continue to
ship the old format configurations. We'll remove those in a later
diff.
Reviewed By: strager
Differential Revision: D10020958
fbshipit-source-id: 11c2ca3b5da086b142042496a2814699880c4f81
Summary:
The logic we care about has been moved to core so we can delete the
`inhibit` extension.
Reviewed By: markbt
Differential Revision: D10276448
fbshipit-source-id: 44cabe5d561344cf8f196127ce52491e4654b598
Summary:
If 'eden stop' tries to tell edenfs to cleanly shut down, and edenfs never responds, 'eden stop' just hangs. Make 'eden stop' time out CLI -> edenfs Thrift calls. After a timeout, 'eden stop' should by stop edenfs via SIGKILL.
Note: This diff does not affect 'eden restart'.
Reviewed By: chadaustin
Differential Revision: D9982372
fbshipit-source-id: 42e498fb93fd97a2ec963e97c25f4b4101c7d524
Summary:
pexpect prints program output while the program is running, which is very useful when debugging. Make the tests in stop_test.py use this feature.
Also, add some assertions to some of the tests.
Reviewed By: chadaustin
Differential Revision: D10157354
fbshipit-source-id: 210fcde79d5d21c6ee34fad3813e7c56eb298b9b
Summary: If edenfs is unresponsive, 'eden status' hangs forever. Add a timeout to turn the hang into a user-friendly error.
Reviewed By: chadaustin
Differential Revision: D10156229
fbshipit-source-id: 9186826ae6b131a193b1499c8baac616d131357f
Summary:
I want to test more conditions for 'eden stop', such as the edenfs process taking a long time to respond to the Thrift stop request. Refactor how fake_edenfs decides to handle stop requests to make it easy to add new behavior.
Aside from changing the log messages, this diff should not change behavior.
Reviewed By: wez
Differential Revision: D10019459
fbshipit-source-id: f6d5f514a67435a446d8d672bf7d21af77d76c56
Summary: HypothesisSimpleTest.test_create is timing out on Sandcastle (Facebook's CI). It looks like the hypothesis tests run indefinitely on Sandcastle. Set a time limit so the tests complete and provide signal.
Reviewed By: wez
Differential Revision: D10053452
fbshipit-source-id: 8c8cb2c375a73b7727e827ea2a9e7d56ab390e54
Summary:
'eden stop' has poor test coverage. Write tests for the common case and some edge cases which 'eden stop' already handles.
This diff should not change behavior.
Reviewed By: wez
Differential Revision: D10017819
fbshipit-source-id: 4d9f5db52187c34c62a9379a6b3dd62f62894233
Summary:
We're looking into doing some performance optimizations in Eden to the
open() and close() calls, since they are never handled by the kernel
and always pass through to the underlying user-space FUSE daemon. But
first, let's capture some baseline numbers on Eden and non-Eden
checkouts.
Reviewed By: strager
Differential Revision: D10036941
fbshipit-source-id: 64f3414a4572fd963017491db37d70e6b5ae4f24
Summary: D6612669 introduced a workaround for edenfs not running as a child of sudo if you run Eden's integration tests as root. D9980225 fixed the underlying bug in 'eden stop' which necessitated the workaround. Remove the now-unnecessary workaround.
Reviewed By: wez
Differential Revision: D9984524
fbshipit-source-id: b9ed7114cd7385a13899d16b4da0d40e6e9fd704
Summary:
This begins adding a framework for generating snapshots of Eden state for use
in tests. I plan to add a few kinds of tests based on these snapshots:
- Tests that ensure Eden can successfully load snapshots created by older
versions of the code.
- Tests that corrupt the snapshot data in various ways and then confirm that
it can be repaired by `eden fsck` and/or still loaded successfully by
`edenfs` even if it was not repaired with `fsck` first.
This code still needs some additional functionality, but I figured it was
worth checking in what I have so far. The main functionality that remains to
be added is unpacking the snapshots and updating absolute paths inside the
config files so that they work in the new path where they were unpacked.
Reviewed By: strager
Differential Revision: D9690267
fbshipit-source-id: a2660e49b84d7833e6778108d9abe081ab7e2cbd
Summary:
Move the `_cleanup_tmp_dir()` function out of the integration lib's testcase
module, and make it a public function in the integration lib util module.
This will make it easier to re-use from some other python utilities that need
to be able to clean up Eden state directories.
Reviewed By: strager
Differential Revision: D9690266
fbshipit-source-id: 157c5221f81f90a3e9612315681459d16693cdbc
Summary:
Update the Eden unit tests and integration tests to set the `NOSCMLOG`
environment variable when running `hg` commands. This ensures that our
mercurial telemetry wrapper does not log events from Eden test runs.
Reviewed By: quark-zju, strager
Differential Revision: D9893096
fbshipit-source-id: c0dd4b5eb042dcb5e9493c89aaee10a513022bae
Summary:
If the overlay file for a directory is corrupted (e.g. empty), Overlay::scanForNextInodeNumber throws. This causes Eden to crash on start [1]. Fix the crash by ignoring corrupted directories.
[1] `test_mount_possible_after_corrupt_directory_and_cached_next_inode_number` reproduces this crash.
Reviewed By: chadaustin
Differential Revision: D9806105
fbshipit-source-id: 1b95083b6a6aa253a2296d6f754edbf4b9f64734
Summary:
The C++ code disallows use of "." in RelativePaths, but it's
reasonable for users of the Thrift API to pass "." to indicate the
root of the mount. Handle that in the EdenServiceHandler.
Reviewed By: strager
Differential Revision: D9647776
fbshipit-source-id: b61c2d1c0dcd69ccfa38bf27379281d10cdf1ceb
Summary:
Fix the code in `eden_dirstate._call_match_callbacks()` to correctly match
ignored symlinks that are explicitly listed in the matcher's file list.
It looks like this was just a mistake in the original code on my part: I
intended to check both `S_ISREG()` and `S_ISLNK()` but instead incorrectly put
the check for `S_ISREG()` twice.
Reviewed By: wez
Differential Revision: D9476524
fbshipit-source-id: 67e0fa7c2fbaac97598a8e2d028c9ef0999ed88a
Summary:
When we load an empty blob from the LocalStore double check with the
BackingStore to confirm that it should actually be empty.
We have seen multiple instances of files being incorrectly imported as empty.
So far this error has always been fixed by a re-import. We still haven't
tracked down the root cause, but this change should help workaround the issue
by ensuring that we double check the file contents before returning the data.
Reviewed By: chadaustin
Differential Revision: D9476522
fbshipit-source-id: 6d57cf15c42736ecbcb106a731259b77db06d8f1
Summary:
Update all of the `eden debug` commands to use the newer `find_checkout()`
code rather than older `get_mount_path()` function.
The `find_checkout()` code makes sure that the EdenInstance actually points to
the correct edenfs instance for this checkout, and also works with checkouts
that are not currently mounted. In particular this allows
`eden debug overlay` to examine the overlay state even when the checkout is
not currently mounted.
Reviewed By: wez
Differential Revision: D9389865
fbshipit-source-id: 00578519d4805157a30c9b39abee9838925e8e76
Summary:
Watchman's Eden integration has a bug where the combination of
Watchman querying Eden for overlapping delta ranges ("give me changes
between X and Y, now changes between X+1 and Y+1") and Eden eliding
redundant change events ("add-modify-remove" -> []) results in
Watchman sometimes reporting that a file exists in its final
subscription update when it no longer does.
The fix is to never elide events, even for files that were added and
removed in the same sequence. To continue to support Watchman's `new`
flag, track whether a file existed at the beginning and end of a
journal delta.
Reviewed By: wez
Differential Revision: D9304964
fbshipit-source-id: f34c12b25f2b24e3a0d46fc94aa428528f4c5098
Summary:
To prevent the regression fixed by D8323051, make sure we run at least a couple
integration tests with the RocksDB LocalStore implementation.
Reviewed By: wez
Differential Revision: D8408390
fbshipit-source-id: 8fab4041ae39915d8be80e42814aab375c4acdda
Summary:
This prevents `hg status` from blowing up with a UTF-8 decode
error inside the generated thrift code.
Push safety concerns:
* This doesn't change the wire representation of the data
* Existing clients that believe it to be a string will continue to have
the same behavior
* Buck has its own copy of an older version of the thrift spec, so it will
continue to work "OK".
* When buck resyncs with our thrift file, some changes will likely be needed
to convert the byte arrays to strings or paths or whatever is appropriate
for bucks internal API
Work "OK" above means that clients that currently believe that `string` is
utf-8 encoded will have a runtime error if we ever send them a path that
is not utf-8. This is the behavior prior to this diff and will continue
to be the behavior for clients (like buck) that have an older version
of the thrift file.
Reviewed By: simpkins
Differential Revision: D9270843
fbshipit-source-id: b01135aec9152aaf5199e1c654ddd7f61c03717e
Summary:
Add some integration tests that invoked "eden doctor" and "eden list" inside a
stale mount point, and confirms that they work correctly. These commands do
fail in dev builds, since python itself does not handle this error.
Fortunately XAR-based python archives avoids this error, so these commands
work in opt mode builds.
Reviewed By: strager
Differential Revision: D8905489
fbshipit-source-id: 0666f8cb0715d61697253bee9fe0bde2f4a3fa15
Summary:
Update `InodeMap::updateOverlayForUnload()` to catch exceptions that occur
trying to save state to the overlay. If something goes wrong when saving
state there is not much we can do other than log an error. We still want to
unload the inode, and code that is unloading inodes generally cannot deal with
exceptions at this point. In particular if this error occurred while trying
to shut down an EdenMount the code would crash.
Reviewed By: chadaustin
Differential Revision: D8884795
fbshipit-source-id: c2f850f13d775be4b0a0a10f9df3948c7b2c8f4a
Summary:
Encountering a truncated overlay file doesn't necessarily indicate a software
bug in Eden. Depending on the underlying filesystem this often happens after
a hard system reboot since we write the overlay files without an `fdatasync()`
call.
Change the code to simply log an error and throw an exception rather than
using `EDEN_BUG()`. This makes it possible to exercise this code path in
tests without having it crash in debug builds.
Reviewed By: chadaustin
Differential Revision: D8988209
fbshipit-source-id: 8c0fe1dae692f4c493413d3939d2e4c21e0da596
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
Summary:
Fix the arguments to `doctor.cure_what_ails_you()`, which were updated in
D8870494.
Reviewed By: chadaustin
Differential Revision: D8940413
fbshipit-source-id: 21d45036f668c9bae9a2002c360532918eb157a5
Summary:
This returns a `Repository` object and not a `Repository` subclass type.
This type information has been broken since D7512320 replaced the
`get_repo_class()` method with `create_repo()`.
Reviewed By: strager
Differential Revision: D8940415
fbshipit-source-id: 7711a2a33bc12fc361dfd0ff3f47eba1e587e6e7
Summary:
Update `EdenFS.run_unchecked()` to return the full
`subprocess.CompletedProcess` argument rather than just the return code.
This will make it possible for future tests to check the output of the command
in addition to just its return code.
Reviewed By: strager
Differential Revision: D8905491
fbshipit-source-id: 7f17f5d103a06aeb56a3be0618683bd1c017fdf0
Summary:
D8508489 updated the integration test code so that the `capture_output`
argument to `EdenFS.run_cmd()` was no longer necessary and was not used.
However I forgot to actually remove the argument in that diff. This deletes
the argument.
Reviewed By: strager
Differential Revision: D8905492
fbshipit-source-id: d7a691f7d628a9fd5daa6723575e4454480f6aa1
Summary:
As of D8631281 `eden clone` now makes the mount point directory read-only and
puts a readme file inside of it. Since the directory is read-only the
integration tests are unable to clean it up using a normal `shutil.rmtree()`
call.
This fixes the integration tests to try making the directory writable first
if it encounters an error during cleanup.
Reviewed By: strager
Differential Revision: D8905488
fbshipit-source-id: c6b9d0b12c2b1dcb4a0839757cbcc3aad179bfc1
Summary:
This adds initial code for an `eden debug fsck` command.
At the moment this simply scans the overlay and reports errors, but does not
correct any issues. In subsequent diffs I'll update it to support correcting
some problems.
There are also several other issues that I'll improve in subsequent diffs as
well. At the moment this code uses the same functions as other CLI commands
to find the client directory, and these only work if the checkout is already
mounted. For `fsck` we generally want the checkout to be unmounted before
checking it. I also plan to clean up some of the output handling as well.
Reviewed By: strager
Differential Revision: D8813714
fbshipit-source-id: 90238e94540db1cc06c17eabf8ce3a1bb808d4b3
Summary:
This makes it easier to add some test coverage.
There's no real functional change in this diff; the only code change is to
throw a system_error instead of a thrift eden error wrapper class from the core
globbing code. There's a little bit of code to restore this exception type in
the callers in EdenServiceHandler; this is covered by existing integration
tests, but I've also expanded that coverage to cover both variants of the glob
thrift calls.
Reviewed By: strager
Differential Revision: D8776767
fbshipit-source-id: 3ea4ea642ae5108aa4b0153541bd3604f010b54c
Summary:
Update `eden restart` to explain that the restart will be disruptive when
performing a full restart, and prompt the user for confirmation. This prompt
can be overridden with the `--force` option. Also print more messaging after
the restart completes telling users what to do if they see "Transport endpoint
not connected" errors from programs still using the old mount points.
This also updates `eden restart` to more carefully handle cases where there is
an existing edenfs daemon running but it is not fully healthy.
Reviewed By: wez
Differential Revision: D8731004
fbshipit-source-id: 05762187b47057b2930d0a6b71b0a6fdbd4aa7e5
Summary:
This moves the fsattr helper program into the helpers/ directory
created in D8373672.
Reviewed By: chadaustin
Differential Revision: D8705391
fbshipit-source-id: 43e04d8b06b5d17caefafdfc750e3e60cad1b0cc
Summary:
Update "eden clone" to create a README file inside the mount point directory
before the checkout is mounted. When Eden is running normally this file will
not be visible because the Eden checkout has been mounted on top of this
directory. However if the checkout is not currently mounted users will see
this file instead.
This should hopefully make it easier for users to figure out how to remount
their checkout on their own, and should prevent them from worrying that their
files have been deleted when the mount point simply is not mounted.
Reviewed By: chadaustin, wez
Differential Revision: D8631281
fbshipit-source-id: 7610450983687a839bdbc3558ff35e0422edc3be
Summary:
The eden integration tests currently run hg through the telemetry wrapper.
This wrapper script adds profiling arguments to a percentage of the commands
it runs. The extra arguments that it adds are not entirely transparent to the
user, and are unfortunately reported in the output of "hg journal".
This was causing occasional test failures in the Eden integration tests that
check the output of "hg journal".
This diff changes the Eden code that checks the journal output to try to strip
off these flags before checking the command output.
Reviewed By: wez
Differential Revision: D8631180
fbshipit-source-id: 004d3bcce291ffc2208f85a29cb34a921f16fdde
Summary:
Update the python CLI code to let the C++ edenfs code handle daemonization
rather than daemonizing in the python code.
This means that we now daemonize after the `sudo` invocation rather than
before it. This allow's sudo's wrapper process to exit once edenfs starts,
rather than remaining around for the lifetime of the edenfs daemon. This
means that the original process' stdout and stderr file descriptors are
closed properly rather than being held open by the sudo wrapper.
This also removes the logic in the CLI that waits for edenfs startup and can
potentially time out and give up.
Reviewed By: strager
Differential Revision: D8508489
fbshipit-source-id: 6a38439d5596049d6f40cb6589b6865c229f736e
Summary:
This updates edenfs to be able to daemonize itself, and moves the
daemonization logic from the python CLI code into C++.
The main benefit of this is that we can now do a better job of reporting
messages to the user during start-up. We can log around potentially slow
operations (opening the RocksDB local store), and we can print messages
directly to the user if startup fails. Previously most failure messages would
go only to the eden log and would not be printed to the user's terminal.
This also fixes some issues where stdin and stdout were not closed properly
when daemonization was performed by the CLI. sudo needed access to these file
descriptors in case it needed to prompt for a password, and it would then hold
the descriptors open until edenfs exited.
Reviewed By: wez
Differential Revision: D8373672
fbshipit-source-id: 3272bff2208596f41d26e479c82c700d6c1efe11
Summary:
Previously `eden rm` tried to look at `<path>/.eden/root` to find the root of
the mount point. This failed if the checkout was not currently mounted.
This updates the code to try looking for the exact input path in the config
file in this case. It also avoids sending an `unmount()` thrift call to
edenfs in this case as well.
Reviewed By: strager
Differential Revision: D8558765
fbshipit-source-id: a5d9db7eee702817713b089a457f25710438661e
Summary:
When running the integration tests in parallel in a dev mode build I
occasionally see timeouts in the shutdown code in some tests. (Usually this
seems to happen in some of the remount tests that create multiple mount points
and shutdown and restart edenfs multiple times.)
Usually when I see this error edenfs does exit successfully on its own just
after this 15 second timeout has expired.
This bumps up the timeout to 30 seconds to avoid spurious failures.
Reviewed By: pkaush
Differential Revision: D8559052
fbshipit-source-id: 567062b08b34eced8f05538607ae9f4773c4aa9c
Summary:
Previously `eden list` reported normally running active checkouts with an
`(active)` suffix, and non-mounted checkouts with no suffix.
The non-mounted situation is rare. This diff changes the command to report
non-mounted checkouts with a `(not mounted)` suffix. I removed the `(active)`
suffix from normally-running checkouts now, but let me know if people think it
is still valuable to keep this annotation in the output.
Reviewed By: chadaustin
Differential Revision: D8558770
fbshipit-source-id: ccce0f7c2031e50f2847089e49007274c31a69a7
Summary:
I saw RemountTest.test_restart_twice() fail once when running locally, but I
haven't been able to reproduce the failure yet. This updates the checks in
this code to hopefully provide better information about what went wrong in
case we see this failure again in the future. This now checks the `eden list`
output first so if something goes wrong we will hopefully have more
information about which checkouts were or weren't remounted.
Reviewed By: strager
Differential Revision: D8528544
fbshipit-source-id: 4f50145aa97956f45f50813070fb73836bc7acde
Summary:
We've seen what appears to be phantom calls to shutdown() so we'd like
to add some degree of auditing. This diff adds a new method with some
context; this will allow us to distinguish between `eden stop`, `eden restart`
and eden server internal calls to the `shutdown` method. It may still
be possible that something else is calling our shutdown method, but it
seems unlikely as we're only accessible to our own code via a unix domain
socket.
Reviewed By: chadaustin
Differential Revision: D8341595
fbshipit-source-id: 50d58ea0b56e5f42cd37c404048d710bde0d13a3
Summary:
D7946842 moved some of the daemon management code from util.py to daemon.py
but accidentally broke handling of TimeoutError exceptions. The code in
daemon.py was trying to catch python's built-in `TimeoutError` type, while the
code in util.py throws its own `util.TimeoutError` type.
This removes the custom `TimeoutError` type in util.py and just uses the
built-in python type to be consistent everywhere.
Reviewed By: wez
Differential Revision: D8373739
fbshipit-source-id: 8523c444fef74b0758c38b4fe2644d4f68d38442
Summary:
mkdtemp deliberately removes these bits and doesn't respect
the umask. We perhaps should respect the umask ourselves here.
Reviewed By: chadaustin
Differential Revision: D8335539
fbshipit-source-id: 98b8f3e2db17c65904b12627e52966d5f6b729ef
Summary:
Some folks might attempt to load this extension despite it
not being useful with eden. The extension knows not to hook in to
the repo when eden is in used, but the various commands to manipulate
the sparse config do not.
This diff adds a little helper function to show a more reasonable
error message than an ugly stack trace when attempting to use them.
Reviewed By: simpkins
Differential Revision: D8328229
fbshipit-source-id: d7cc4c7047b11f8da96e26ab70bd522b52546151
Summary:
Fix the integration tests to set CHG_BIN correctly when run manually rather
than via `buck test`.
Reviewed By: wez, quark-zju
Differential Revision: D8329593
fbshipit-source-id: 79fa3c9e9538e4b30a98ef777515527562e1ac1b
Summary:
This fixes a bug simpkins pointed out in D6891479 - we weren't
updating mtime and ctime on renames.
Reviewed By: simpkins
Differential Revision: D7937303
fbshipit-source-id: 08fd8f4fe5d99d33e9f7629965d6146330c8f35b
Summary:
I kept running into issues trying to get graceful restart and
flush_cache to work together in the hg integration suite, so add a
test to ensure flush_cache succeeds after a graceful restart in the
main integration suite.
Also, to make the test's output easier to follow, add logging when
invalidating inodes.
Reviewed By: simpkins
Differential Revision: D8215961
fbshipit-source-id: 33db4292af3969ae23940c3027ba513ed20c53fb
Summary:
Add a command line option to start edenfs only if there is at least one
Eden checkout already configured.
This makes it easier to automatically start edenfs on system boot only if the
user needs edenfs.
Reviewed By: wez
Differential Revision: D8174819
fbshipit-source-id: bf585b8c5aa719d3b6424567f2e21c1c63850d66
Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.
Reviewed By: bolinfest
Differential Revision: D8108649
fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
Summary:
The Overlay is the natural home for nextInodeNumber_ now that every
directory allocates inode numbers for its children right away. This
also simplifies serializing nextInodeNumber_ to disk in the following
diff.
Reviewed By: simpkins
Differential Revision: D8192442
fbshipit-source-id: 9b776a73c8d7653002b55985d592b1746e52f878
Summary:
Add a new integration test that modifies a file, commits it, then restores it
to its original contents and creates a new commit with the revert.
This test isn't really exercising any Eden-specific behavior. The only reason
it is is interesting is that this happened to trigger a bug in the underlying
mercurial cdatapack code that caused the test to fail when run in treemanifest
mode. The mercurial cdatapack bug is fixed in D8131020
Reviewed By: quark-zju
Differential Revision: D7826284
fbshipit-source-id: 32da2db04452d5df0527f6be00b0c4c1a56c900b
Summary:
Update find_executables.py to be able to find the HG wrapper and binary using
buck rules. This is necessary in order to be able to run the integration
tests manually rather than via `buck test`. Without this the tests try to use
`/bin/hg.real` which tests the wrong version of the mercurial code and the
eden extension.
Reviewed By: bolinfest
Differential Revision: D8200168
fbshipit-source-id: d2a68c765c7143bbe613f32eef1268ed567790a7
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: cooperlees
Differential Revision: D8198776
fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
Summary:
Relax the restriction on changing uid/gid on inodes. We'll see what
cans of worms this opens I guess. (Landing this is low priority, but
might be important for making some of the existing tooling in fbsource
and www work.)
Reviewed By: simpkins
Differential Revision: D7768655
fbshipit-source-id: 95fe02fe7ddc001335dbdb34e16a989a85820240
Summary:
Add a `remove` command to the Eden CLI. This behaves like
`eden unmount --destroy`, but calling this "remove" is hopefully a more
intuitive UI. If stdin is a TTY this command also prompts the user for
confirmation before removing the checkout.
I plan to deprecate the `eden unmount --destroy` command in a subsequent
diff.
Reviewed By: wez
Differential Revision: D8086823
fbshipit-source-id: 562cf0f998eea416b80589b188eee255a10b9699
Summary: Now that permissions on directories work, verify umask works as intended.
Reviewed By: simpkins
Differential Revision: D7783743
fbshipit-source-id: 635221cd3255cc20e9ffa26b6838922c4a4110f3
Summary:
This moves most inode metadata management into InodeBase and
persists permission bits (and eventually uid/gid) across Eden runs.
Reviewed By: simpkins
Differential Revision: D7035163
fbshipit-source-id: 50145449b56aad1662d53156e6e4960c5f7b6166
Summary: Store tree and file timestamps in the InodeTable so they persist across runs.
Reviewed By: simpkins
Differential Revision: D6891479
fbshipit-source-id: 1c9e6266375aceeaf293a81e73cf7f5334dbc32d
