Summary:
This was causing problems on macos where various tools
would enumerate and helpfully try to preserve attributes across
copies. On macos this would result in appledouble metadata files
being created to track the metadata in the destination file,
which clutters up the repo and has surprising secondary effects
such as being picked up by glob operations in cmake build rules.
This diff simply stops enumerating the extended attribute.
Reviewed By: fanzeyi
Differential Revision: D17140414
fbshipit-source-id: 2924657dc75b900baf70595edfa72e5d0521a697
Summary:
The snapshot generation code and many integration tests create
repositories. By default, they were creating flatmanifest
repositories, which are on their way out. Instead, create tree-only
repos.
Reviewed By: strager
Differential Revision: D17066151
fbshipit-source-id: f99a9543440da6fd7cce0065c3cd7f91a59a02d5
Summary:
Update `EdenServer::mount()` to correctly handle errors that occur during the
mount `INITIALIZING` phase. Previously the code did not add error callbacks
to the `Future` result to handle errors during initialization. As a result we
would propagate the exception back to the thrift caller, but the `EdenMount`
object would remain in our mount point list, stuck forever in the
`INITIALIZING` state.
Reviewed By: strager
Differential Revision: D16590032
fbshipit-source-id: 9adbdf05441dad815096b195ece36f3d958c96a9
Summary: Add a dependency from the eden open source build to the fb303 open source build and switch EdenServiceHandler to BaseService.
Reviewed By: simpkins
Differential Revision: D15528156
fbshipit-source-id: 2ca5c31dd9fcc9bac43fd399b27f33b6f2c5ebfc
Summary:
Open source fb303 will not have getPid() or getCommandLine(), so
introduce a new method for Eden's tests.
Reviewed By: fanzeyi
Differential Revision: D16292993
fbshipit-source-id: 5cdc006ec0ee15f50a3e1cebe9b46a3ea275ff78
Summary: The CountersTest would previously fail if by chance the counters prefixed by "thrift" and "thrift_client" were accounted for between getting "counters" and "counters2", since these counters should not be modified when mounting/unmounting mounts we will just filter them out.
Reviewed By: chadaustin
Differential Revision: D16265511
fbshipit-source-id: 21af0dff345977692785136ca0333d23d5c77e0d
Summary: I found out that the journal stats callbacks that were getting registered were not getting unregistered, this diff fixes that.
Reviewed By: chadaustin
Differential Revision: D16187569
fbshipit-source-id: 8c84e1515e376ccd7036a22c06e2e6b98dc62342
Summary: Turn on logview collecting by default. Also disable logview for Eden integration tests.
Reviewed By: chadaustin
Differential Revision: D15978960
fbshipit-source-id: 623c3be7a461e0bb9bc44924fccfdb006565fad6
Summary: Added the cli command `eden stats object-store` for querying the counts on what part of the object store was responsible for finding the blob or blob size (local store or backing store). This will tell us how well local and in-memory caching works for different workflows.
Reviewed By: chadaustin
Differential Revision: D15934535
fbshipit-source-id: 70345f11a51c3c6996dc001d4101744395a3d182
Summary:
This diff adds a single repo-wide `.eden-redirections` file that is used to
record the redirection configuration for the repo.
The redirection configuration code knows to look at this file and fold it in to
the effective configuration; the legacy bind mounts are applied first, then the
repo redirection configuration and then the user specified redirections.
The intent is that a `post-update` hook will invoke `eden redirect fixup` to
apply the configuration from the repo, and to have the eden daemon also
trigger that after (re-)mounting the repo.
In the future we can extend this to allow different profiles to be enabled
or disabled, but for now this is the minimum viable product.
Reviewed By: strager
Differential Revision: D15867225
fbshipit-source-id: b0a95936dd28283de6c7439ca8e503caef4e7247
Summary:
This is part of the effort to make our bind-mount configuration more
visible and easier to change.
The idea is to generalize the concept of redirection and add a command to help
manage it.
The `eden redirect add` subcommand allows creating one of two different kinds
of redirection:
* `bind` - allocate some space using `mkscratch` and mount it into the repo
* `symlink` - allocate some space using `mkscratch` and create a symlink
that points to it from the repo
On Linux we use bind mounts to implement `bind` but on macOS, which doesn't
have a bind mount concept, we create a sparse disk image file that can grow
to match the size of the disk on which it is created (in practice these are a
7-15MB in size to start and grow as the user stores data into them).
The `eden redirect del` subcommand allows removing a redirection, including
the legacy `bind-mounts` configuration from `.eden/client/config.toml`.
The `eden redirect list` subcommand lists the effective set of redirections,
both from the new redirections configuration and the legacy `bind-mounts`
configuration, along with their state.
The `eden redirect fixup` subcommand iterates over the effective set of
redirections and can remove and reinstate any that are in a broken state.
Reviewed By: strager
Differential Revision: D15707319
fbshipit-source-id: a5dd8c44c9f748482d7b48855b1305d44267885c
Summary: This diff takes care of importing blob from Mononoke and Mercurial at the same time, also improves the name situation in the statistics counters.
Reviewed By: strager
Differential Revision: D15768557
fbshipit-source-id: 10cf831b1ae6dc9e6b91f1e96508c4fa92583743
Summary:
If edenfs was started using `sudo`, the `$USER` environment variable will be
set to `root` rather than the actual user. When we drop privileges make sure
we restore the value of `$USER` as well.
The `$USER` variable isn't checked anywhere else in edenfs itself, but it
matters for subprocesses we spawn, like `hg debugedenimporthelper`.
I also changed the code to clear the `SUDO_*` variables as well, mostly
just for good measure.
Reviewed By: kulshrax
Differential Revision: D15929539
fbshipit-source-id: e022c7ae762e2a5e86d0227058bb476aff17cf55
Summary:
Add a periodic task for performing LocalStore management tasks. For now only
the RocksDBLocalStore class implements this management task.
When this periodic task runs the RocksDBLocalStore object computes how much
space each of the column families are using and publishes this as fb303
counters. If the total size of the ephemeral column families exceeds a
configurable limit it then triggers a background garbage collection task.
I also added a new `edenfsctl stats local_store` command that reports the new
counters added by this diff.
Reviewed By: chadaustin, strager
Differential Revision: D15798505
fbshipit-source-id: 25ca4ba80f5a9c4a1a09dc08633c7b3af363d7ff
Summary:
Update the copyright & license headers in Python files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487088
fbshipit-source-id: 9f2138dff41048d2c35f15e09a04ae5a9c9c80dd
Summary:
Update the copyright & license headers in C++ files to reflect the
relicensing to GPLv2
Reviewed By: wez
Differential Revision: D15487078
fbshipit-source-id: 19f24c933a64ecad0d3a692d0f8d2a38b4194b1d
Summary:
Add a periodic task to reload the configuration file from disk. By default
this runs once every 5 minutes, but this interval can be controlled from the
config file.
At the moment reloading the config file does not do much other than update the
interval for how frequently the config file is reloaded. However, I plan to
add additional periodic tasks shortly that are controlled by this config
setting.
This will also make it possible for other parts of the code to
access the config settings in the `ServerState` and use them as-is without
checking to see if they reloaded. Currently all of the code that accesses
config values performs a check to see if the config needs to be reloaded. If
we want to switch to Mercurial-style configs in the future that check will be
substantially more expensive.
This diff also includes a new thrift call to force the config file to be
reloaded immediately. This can be used to restart automatic config reloading
if it is ever disabled in the config file.
Reviewed By: wez
Differential Revision: D15756357
fbshipit-source-id: 1999f4730903633ce838842932a6ae6a65eda4e6
Summary: Fix a few more issues raised by our Python lint checks.
Reviewed By: wez
Differential Revision: D15776717
fbshipit-source-id: 621960579c4567c4fb9395ae14cd7a8666726c1c
Summary: Remove a number of unused imports detected by the linter.
Reviewed By: wez
Differential Revision: D15776268
fbshipit-source-id: 221f45d275664d037bbabcac9858b40266b4833e
Summary:
If the systemd user manager is not running (e.g. it crashed or was manually stopped), 'eden stop' fails with an unhelpful error message:
> pystemd.dbusexc.DBusConnectionRefusedError: [err -111]: Could not open a bus to DBus
or
> pystemd.dbusexc.DBusBaseError: [err -2]: Could not open a bus to DBus
Provide a better experience: tell the user the most likely cause, and suggest a solution:
> error: The systemd user manager is not running. Run the following command to start it, then try again:
> sudo systemctl start user@strager.service
Reviewed By: wez
Differential Revision: D13791023
fbshipit-source-id: 5172df0a52d21c311b27b8a527cad934f9882154
Summary:
Summary
Change `ConfigTest` to derive directly from `EdenTestCase` rather than using
the `eden_repo_test` decorator. The configuration test code doesn't really
need a repository, and so we don't need to run it twice (for both Mercurial
and Git repositories).
Reviewed By: wez
Differential Revision: D15756359
fbshipit-source-id: 90d5011ae1ff7d2a251c9e7bb776045fbe2fdfe1
Summary:
Clean up the `EdenFS` class construction.
Previously it accepted the `eden_dir`, `etc_eden_dir`, and `home_dir`
arguments as separate parameters. If `etc_eden_dir` or `home_dir` were not
specified it would not pass these arguments to `edenfs`, allowing the default
values to be used. This is undesirable for most tests.
Now it accepts a `base_dir` argument. Explicit values for the `eden_dir`,
`etc_eden_dir`, and `home_dir` parameters can still be specified (this is used
for the snapshot tests), but if they aren't specified, default locations
inside the `base_dir` will be used instead.
This also cleans up some of the code to use `pathlib.Path` values instead of
plain `str` objects in more places.
Reviewed By: strager
Differential Revision: D15756358
fbshipit-source-id: 3e87ddc98d15fcb7f60c6c3116d4fcc8e49432ea
Summary:
Add a thrift call to get the current config settings.
My primary use case for this method at the moment is to make it possible to
build integration tests that check the config behavior. However in the future
this will probably also be useful for building CLI commands to report the
current config values to allow debugging if there are ever issues. This API
can also be used to force EdenFS to immediately reload the config from disk.
Reviewed By: strager
Differential Revision: D15572124
fbshipit-source-id: da3bc982f9c419b3314a8b0560c9bd327760d429
Summary:
With systemd integration enabled, if edenfs is running, or if edenfs' systemd service is active, `edenfsctl start` does nothing. This behavior differs from `edenfsctl start` with systemd integration disabled, and can cause `edenfsctl restart` to think that it successfully started edenfs.
Make `edenfsctl start` fail if edenfs is running and healthy, or if edenfs' systemd service is active (yet edenfs is unhealthy).
Reviewed By: chadaustin
Differential Revision: D15703310
fbshipit-source-id: ce0a13780ee03de1f896a938d002901023e5bdd3
Summary:
systemctl has some problems for Eden. For example:
* With Restart=on-failure, 'systemctl start' reports that the job failed when the first failure occurs. 'systemctl start' does not wait for retries to finish. This means 'eden start' can fail despite edenfs starting successfully.
* If the service fails to start, 'systemctl start' prints a suggestion to use journalctl, even though journalctl is broken and is not even used by fb-edenfs@.service.
* If 'systemctl' can't connect to systemd, it prints a generic message such as "Failed to connect to bus: Connection refused" which the Eden CLI can't easily detect and customize.
For 'eden start', instead of using systemctl, talk to systemd using its D-Bus API (via pystemd [1]). This automatically solves the journalctl message problem, makes it trivial to customize certain errors, and will let us solve the Restart=on-failure problem in the future.
Aside from changing some error messages, this diff should not change behavior.
[1] https://github.com/facebookincubator/pystemd
Reviewed By: simpkins
Differential Revision: D13533184
fbshipit-source-id: 7fedc8ad4a094a2d04b14c2f6e82b51a0ed348a6
Summary: We want to have just one entry point to Mercurial, namely the Rust binary. Getting rid of the `hg` Python script means that we finally can do things that only exist in Rust and not in Python.
Reviewed By: simpkins
Differential Revision: D13186374
fbshipit-source-id: f3c8cfe4beb7bf764172a8af04fd25202eca9af2
Summary:
test_reading_committed_file_bumps_read_counter is flaky with optimized builds of edenfs. I think it's flaky because FuseChannel bumps counters *after* responding to the kernel, so the test can call get_counters before the counters are bumped.
Fix the flakiness by making the test wait a while for the counters to change.
Reviewed By: chadaustin
Differential Revision: D15550972
fbshipit-source-id: 891e5d0a9748b43eb0ef1089ef6bc0a547c47d4d
Summary:
In the past few months, test_no_units_are_active started failing. It looks like 'systemctl list-units' is now listing devices. For example:
```
$ systemctl list-units --all --full --no-pager
UNIT LOAD ACTIVE SUB JOB DESCRIPTION
● boot.automount not-found inactive dead boot.automount
proc-sys-fs-binfmt_misc.automount loaded active running Arbitrary Executable File Formats File System Automount Point
dev-disk-by\x2dlabel-\x5cx2f.device loaded active plugged /dev/disk/by-label/\x2f
[snip]
dev-getty.device loaded inactive dead /dev/getty
dev-loop0.device loaded active plugged /dev/loop0
dev-ram0.device loaded active plugged /dev/ram0
dev-ram1.device loaded active plugged /dev/ram1
[snip]
```
I don't know if systemctl changed or if systemd changed or if my machine's configuration changed. Either way, the test is failing now due to these systemd units.
Teach test_no_units_are_active to ignore these unimportant device units, since they don't represent running services or timers. This causes the test to pass on my machine.
Reviewed By: wez
Differential Revision: D15548072
fbshipit-source-id: 4f49c72d88b836aba37ec5ea7b5ee5b7cb8172f6
Summary:
EdenFS' systemd tests detect if the running system is managed by systemd, and chooses different strategies for creating a sandboxed systemd user manager depending on this detection. Sandcastle (Facebook's continuous integration system) recently started running systemd as the container's init system.
EdenFS' systemd tests correctly detect that Sandcastle's container is managed by systemd. Unfortunately, the tests cannot communicate with systemd. For example, `XDG_RUNTIME_DIR=/run systemctl --user status` in a Sandcastle job reports the following error:
> Failed to connect to bus: No data available
This error happens because systemd and the EdenFS' tests are running in a different process namespace. The filesystem shows that the system is managed by systemd, but the process table shows otherwise! This causes all of EdenFS' systemd tests to fail.
Work around this issue making Sandcastle use the old "unmanaged" code path. (The unmanaged code path is run if the system is not managed by systemd. Before Sandcastle started running systemd in its container, Sandcastle used this code path when running EdenFS's tests.)
(Ideally we would figure out why we need both the "managed" code path in the first place. This diff is just meant to fix tests on Sandcastle for now, not implement a long-term solution.)
Possibly related upstream systemd issue: https://github.com/systemd/systemd/issues/11300
Reviewed By: wez
Differential Revision: D15530685
fbshipit-source-id: b65b568e660310c50a4e25e0aa143f9388f1ad45
Summary:
I am refactoring edenfs' EdenStats class. In doing so, I accidentally removed a call to `aggregate`, causing `flushStatsNow` to not expose accurate counters. This wasn't caught by any existing tests, only by manual testing.
Add some tests to prevent FUSE and HgBackingStore statistics collection from completely regressing.
Reviewed By: simpkins
Differential Revision: D15274275
fbshipit-source-id: c8a9c9848dd60aee7f252a93f10ddce6d7560799
Summary:
In another diff, Adam Simpkins noticed that HgImporterStatsTest.get_counters was clunky and that the logic belongs in the EdenTestCase base class.
Hoist HgImporterStatsTest.get_counters into EdenTestCase. Also avoid reusing the Thrift client between calls to get_counters because edenfs might restart between calls (in which case the old Thrift client won't work).
Reviewed By: wez
Differential Revision: D15514537
fbshipit-source-id: 0ae25106baa0e5b2d857b0bb2552d884b9b270ef
Summary: Some pyre-fixme directives are on the wrong line. Move them to the line to fix "Unused ignore" errors from Pyre.
Reviewed By: wez
Differential Revision: D15507418
fbshipit-source-id: b8d1163080b1c64868c37e7581411be31f495141
Summary: This function isn't used anywhere. Delete it.
Reviewed By: pkaush
Differential Revision: D9695388
fbshipit-source-id: 1ac702c98ee63d09c15c8a7b8a9c8d44fcec630d
Summary:
In EdenFS' tests, when systemd-run fails, subprocess.check_output raises an exception and causes the test to fail. The exception's message does not include any output of systemd-run, so systemd-run's error messages are hidden. This makes it hard to debug systemd-run failures.
When systemd-run fails, print the captured output. This surfaces error messages printed by systemd-run, improving the debugging experience.
In the normal case where systemd-run succeeds, this diff should not change behavior.
Reviewed By: wez
Differential Revision: D15466610
fbshipit-source-id: 2b2db80b989308967e13499fcfadd37b44ca878f
Summary:
Update the integration tests to avoid explicitly using the old
`hg_import_helper.py` script.
Reviewed By: pkaush
Differential Revision: D15223982
fbshipit-source-id: 6e2310d5a9e6e0b95690a07d61d295cc3b1bca92
Summary: Flatmanifest is on its way out. Remove support for falling back to it if a tree import fails.
Reviewed By: pkaush
Differential Revision: D15056459
fbshipit-source-id: a4df820322ee354d77f50a0ec92e9705d0f152ec
Summary:
Move some of the argument parsing and config setup code out into a new
EdenInit.h header file. This makes it possible to re-use this logic for other
standalone utilities that want to be able to find the Eden state directory and
config information.
For now I have updated the `fake_edenfs` helper tool used by the integration
tests to use this. This may also be useful for writing standalone tools that
can perform garbage collection of the LocalStore or checking of the overlay
state.
Reviewed By: chadaustin
Differential Revision: D14889616
fbshipit-source-id: b0b193a42cb2f52177d0c44592426b42e27242aa
Summary:
On my system this test was failing because the `systemctl start exit.target`
command was exiting with an error. It looks like this command exits because
systemd closes its D-Bus connection as it is shutting down. This modifies the
test to ignore the return code from this systemctl command.
I also updated the log message when shutting down systemd to no longer include
the exception backtrace. This message is always logged when running this test
(since systemd is not actually running), and the backtrace information simply
makes the test logs harder to read.
Reviewed By: wez
Differential Revision: D14886191
fbshipit-source-id: 87c996b2579a9920a72ee5b57608c263ca080d6e
Summary:
Sometimes, EdenFS goes bonkers and talks to 'hg debugedenimporthelper' a lot for seemingly no reason.
Make these situations easier to debug by counting how many requests EdenFS makes to 'hg debugedenimporthelper'. These counters let us answer questions such as the following:
* When performance sucks, is EdenFS is making a lot of requests or only a few requests? I.e. is Hg slow at responding or is EdenFS very demanding?
* Does a recent performance issue correlate with EdenFS communicating with 'hg debugedenimporthelper'?
* Which engineers are outliers having orders of magnitude more 'hg debugedenimporthelper' requests than p50 engineers?
We could get fancier with these counters and include the number of bytes received, the duration of the request, etc. For now, just having a request count is useful.
Reviewed By: simpkins
Differential Revision: D14677339
fbshipit-source-id: 7f8f394fb0096aef65d6a8a45d7da5936db539a0
Summary:
After the kernel added readdir caching, my testing uncovered that Eden
was invalidating TreeInode entries incorrectly when new entries were
added. Change TreeInode to distinguish between directory entry changes
and removals (FUSE_NOTIFY_INVAL_ENTRY) and additions
(FUSE_NOTIFY_INVAL_INODE).
Reviewed By: strager
Differential Revision: D13870422
fbshipit-source-id: 2a6f25bfd9e77436a5aae639fedbfd8a445b2e05
