Summary:
The `test_hg_clone_non_eden_repo_within_eden_repo()` test had been disabled
when running with treemanifest enabled since the treemanifest code did not
correctly detect which repositories should actually use treemanifest.
This issue appears to have since been fixed in the treemanifest code, so this
test passes now.
Reviewed By: quark-zju
Differential Revision: D12927801
fbshipit-source-id: b9e2e041f7eab5e24007888e2dba142e1f0b2251
Summary:
- Update mypy in fbcode to upstream version 0.641.
- Add/fix some type stubs
- Disabled some pieces that break mypy (T35976411 and T35973384)
This diff will likely cause some unit test failures, due to incorrect type annotations that should be fixed by the team owning the relevant code.
mathchecksout
Reviewed By: jeffmo
Differential Revision: D10414250
fbshipit-source-id: 282ea62aef6793a2f4baef78afd58ddd4ddf60f5
Summary: This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D12873644
fbshipit-source-id: c8eaac8bee56268b8a6479f7ea27e496a6b12d96
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Create utilities for tests to enable units within a temporary instance instance, start services, and inspect the processes inside a running service.
These utilities are currently unused, but will be used in future diffs as 'eden start', etc. grows systemd support.
Reviewed By: simpkins
Differential Revision: D10371824
fbshipit-source-id: c1e62eebb8480309ee17c49c08af73c3f5cf75d6
Summary: While creating a systemd service unit for EdenFS, I noticed a few features of the current implementation of 'eden start', 'eden stop', and 'eden restart' which are not covered by tests. Write tests for these features to ensure they don't regress with systemd integration.
Reviewed By: wez
Differential Revision: D10434379
fbshipit-source-id: 6c828a85d59179bbc4beda87e1bf2534543b60b2
Summary:
Update the snapshot framework code to add tests that mount saved snapshots
with the current edenfs build and verifies that they behave as expected.
This also changes the snapshot generation code a fair amount as well, in
order to be able to verify the snapshot data:
- We now more cleanly split out data that should be saved when the snapshot is
first generated (such as the `.eden` state directory) and data that should
be re-generated each time the snapshot is unpacked (such as the `/etc/eden`
config directory and the system configuration for `hg`).
- The code is now capable of rewriting absolute paths in the Eden state files
so it is usable in a new location after it has been unpacked.
This also updates the "basic" snapshot type to create a wider variety of types
of files and file changes, and to be able to verify the contents of a resumed
snapshot.
Reviewed By: strager
Differential Revision: D9955411
fbshipit-source-id: 76012c01016cf4d89dc611c5596a05f5e7f013ed
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Create a utility for tests to create a temporary systemd instance. It should work regardless of whether the host uses systemd to manage its services. (systemd's tooling must still be installed in order to use the utility, though.)
This utility is currently unused, but will be used in future diffs as 'eden start', etc. grows systemd support.
Reviewed By: simpkins
Differential Revision: D10286940
fbshipit-source-id: 4fbaa695bf36ac4ae44b5c12b6255514bd7143b3
Summary: spawn_status already sets the log file to stderr. Setting it again is redundant. Delete the dead store.
Reviewed By: simpkins
Differential Revision: D10434381
fbshipit-source-id: d0e4cfe9dc63ae80fc8a9e621b02a3ac31f8244a
Summary:
Since we can't seem to find a way to prevent ESTALE
from being returned by the kernel, and that only seems to occur
with readlink, add a retry to our readlink calls.
This feels gross, but not sure how we can do better :/
Reviewed By: strager
Differential Revision: D12239143
fbshipit-source-id: 04ecbdeac3b3ea3969e779da4f50cf91ccc5cd00
Summary:
In order to test 'eden start', etc. when edenfs is managed by systemd, we need to install the systemd service and let the Eden CLI start and stop the service. To avoid interfering with the user's running edenfs systemd service, and to avoid interference between unrelated tests, tests need to create a new instance of systemd.
Unfortunately, we can't create a new instance of systemd if a machine is not managed by systemd. However, we can trick systemd into thinking that the machine *is* managed by systemd. When we do that, the systemd instance seems to work without any issues.
Create a library which, when injected into systemd using LD_PRELOAD, makes systemd thinks the machine is managed by systemd.
The library will be used in a subsequent diff (D10286940). This diff is separate so it's easier to review.
Reviewed By: simpkins
Differential Revision: D10371831
fbshipit-source-id: b54b704dc631ebdc5091320dd6b680ca255f3ba2
Summary:
When running fake_edenfs in daemonize mode (i.e. when --foreground is not specified), the daemon process inherits stdout and stderr from the parent and never closes them. This has two consequences:
* The daemon process holds a reference to stdout/stderr even after the parent exits. This causes tests to hang if they slurp stdout/stderr after waiting for the parent process to exit. (I discovered this issue when adding tests for 'eden start' in D10434379.)
* fake_edenfs logs to stdout/stderr after 'eden start' has returned. I think this behavior is surprising, but it doesn't really hurt anyone because fake_edenfs is just a developer tool.
Promptly close stdout and stderr after startup finishes by redirecting further output to a log file. This mimics the behavior of the real edenfs.
Note: stdin is still leaked. stdin will be addressed in D10442346.
Reviewed By: chadaustin
Differential Revision: D10439299
fbshipit-source-id: a7a423aa901982f3d35e81940ffc27db3a985d76
Summary:
No one uses --logPath with fake_edenfs. Remove the dead flag, inlining its default value.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439298
fbshipit-source-id: 068e5677db1bec81a8da174b136ef9f5b54c7e34
Summary:
The daemonizeIfRequested is implemented in the main edenfs process as well as in fake_edenfs. Consolidate the almost-identical implementations.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10439300
fbshipit-source-id: 6253668896de0f8393b92f4aa5ee8a8a2fcbfb31
Summary:
Right now, 'eden start' spawns edenfs directly (using `os.exec`). In the future, 'eden start' will spawn edenfs using systemd. 'eden restart', 'eden stop', and 'eden status' might behave differently depending on how the edenfs process was spawned.
Parameterize some tests on how fake_edenfs was originally started. This flexibility lets us easily spawn fake_edenfs with 'systemctl start' (systemd) in the future. For now, spawn fake_edenfs in ad-hoc mode (i.e. directly without 'eden start') or in managed mode (i.e. indirectly with 'eden start').
Reviewed By: chadaustin
Differential Revision: D10414995
fbshipit-source-id: 2ee0d5df8a605ca9d7da8f6eeca1fc171a8342e8
Summary:
RestartTest uses 'eden restart' during setup to start the fake_edenfs process. There are two issues with doing this:
* I want to extend the tests in RestartTest so they also tests with an ad-hoc fake_edenfs process (D10414995). Restarting edenfs during setup makes no sense in that context.
* I think using 'eden restart' during setup is clunky and unintuitive. 'eden start' is more appropriate.
Refactor RestartTest's setup so it uses 'eden start' instead. Also add a test which explicitly verifies that 'eden restart' starts fake_edenfs if no daemon is already running.
Reviewed By: chadaustin
Differential Revision: D10417304
fbshipit-source-id: 20f8190026cd153dd9b539067f6f63b6bd27abed
Summary:
I want a function like `fake_eden_daemon` which instead spawns fake_edenfs using `eden start`. Refactor existing code to make such a function easier to write:
* Replace `contextlib.contextmanager` with a class called `FakeEdenFS` so its `__exit__` method can be reused.
* Rename `fake_eden_daemon` to `FakeEdenFS.spawn` so it's clearer that the function creates a process.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10415123
fbshipit-source-id: 81e6a0618214a527dc7f15cc18d4fe97dd1a957e
Summary:
Update the HgRepository.status() function in the integration tests to return
the status information as a dictionary instead of a string. Only one test was
still using the old string API.
Reviewed By: chadaustin
Differential Revision: D10503168
fbshipit-source-id: 574e4438d23bf6612a70ae5ae3174db3d464d198
Summary:
Update the eden+hg integration test code so that it uses find_executables.py
to find the post clone hook. Previously it was still using its own one-off
helper function for this.
Moving this code into find_executables.py will make it possible have the
snapshot tool also find the post clone hook.
Reviewed By: chadaustin
Differential Revision: D10503169
fbshipit-source-id: d59e3e517d19e80e0d0c997be09865519b80d34e
Summary:
Now that the import bug has been fixed for some time, it's likely few
people have cached empty files. And if they do, `eden gc; eden
restart` is a fine workaround.
On the other hand, reimporting empty files gums up the importer, and
I've seen several people recently complaining about performance that
could be partially attributed to the fact that their Eden needed to
verify empty files. simpkins has talked about adding a bit to the
cached blob to determine whether it needs reimporting or not, but it's
probably going to take a while for anyone to implement that, and this
reverification logic is hurting people today.
Reviewed By: strager
Differential Revision: D10456519
fbshipit-source-id: 657bc377ee16ce93494075bde4388aed59dceecf
Summary:
When you get an error, let's print the number of conflicts in each file. This will give the user some sense of how much work they have to do.
The code change is entirely in `filemerge.py`, and `tests/test-merge-conflict-count.t` adds a new test.
Reviewed By: quark-zju
Differential Revision: D9815243
fbshipit-source-id: 1b73a1db293902ac7242997a7d6ae09478344068
Summary:
Add a new sparse show command, which shows the current sparse configuration.
Use common templating, including colors, between sparse show and sparse list.
Reviewed By: phillco
Differential Revision: D10446705
fbshipit-source-id: acb539f80f625945716758c785703b46a81aa6f1
Summary: Some commands in extensions have incorrectly defined usage. Correct these.
Reviewed By: phillco
Differential Revision: D10446707
fbshipit-source-id: e606314892a31fa09f4a655042d66d87929fc2ec
Summary: D10416965 accidentally removed the import for `typing` in health_test.py. This causes tests to fail. Re-add the import to fix the tests.
Reviewed By: singhsrb
Differential Revision: D10452517
fbshipit-source-id: af58d076b7bac375c20630113cbb3972e24ffc12
Summary:
Eden supports reading the SHA-1 of a file via getxattr. Unfortunately,
it returned ENOSYS if you called getxattr on a directory inode. This
caused FUSE to fail all future getxattr requests with EOPNOTSUPP.
In addition to fixing that, this diff makes our xattr handling a
little more consistent across inodes:
- setxattr() always fails with ENOSYS
- removexattr() always fails with ENOSYS
- listxattr() is always handled by the corresponding inode class
- getxattr() is always handled by the corresponding inode class
Differential Revision: D10437723
fbshipit-source-id: a1ea1e92d3412abb15e91057becea3160a17f1e2
Summary: Instead of using tempfile.TemporaryDirectory and calling unittest.TestCase.addCleanup manually, use TemporaryDirectoryMixin which does a better job.
Reviewed By: chadaustin
Differential Revision: D10416965
fbshipit-source-id: aed9d80788d54692bc7f024a5a62523ca168bbf5
Summary:
StartupLogger handles two independent scenarios: edenfs in daemon mode, and edenfs in foreground mode. I need to add a third for systemd support: edenfs in foreground mode but with startup logs written to a file.
Refactor the existing StartupLogger into an abstract base class with two concrete derived classes: DaemonStartupLogger and ForegroundStartupLogger. This will make adding the systemd feature easy without confusing the logic for the existing scenarios.
Aside from dropping the setsid() call for ForegroundStartupLogger, this diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D10228156
fbshipit-source-id: fbc2c771c6fbb675779ef05ca3b7395b70b67485
Summary: D10335083 renamed `fbsparse` into `sparse` and I forgot to udpate this test.
Reviewed By: simpkins
Differential Revision: D10423192
fbshipit-source-id: a547c403a9461d6923ecfecc931df82bb7887597
Summary:
Make StopTest and HealthOfFakeEdenFSTest similar to RestartTest by putting the code to invoke 'eden --config-dir ... stop-or-status' into its own function.
This diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10414421
fbshipit-source-id: e42cdee64ae282fa0575cf875b53ac91fa61d490
Summary:
To ease the migration from the current custom daemon setup to a systemd-managed daemon, add a way to enable the to-be-implemented systemd behavior.
* To the edenfs daemon, support the --experimentalSystemd option.
* To the Eden CLI, support the EDEN_EXPERIMENTAL_SYSTEMD environment variable. When set, invoke the edenfs daemon with --experimentalSystemd.
Aside from adding a log message, this diff does not change any behavior. This diff just sets up scaffolding for future diffs.
Reviewed By: simpkins
Differential Revision: D10248055
fbshipit-source-id: 30ca5a4bfde00ff43fd6f2a5d0282ced4f177fed
Summary:
Logging some messages to stdout and some messages to stderr has a few problems:
* stdout and stderr can get interleaved. For pipes, interleaving is avoidable for messages smaller than PIPE_BUF (512 bytes), but there's no guarantee that stdout and stderr are pipes or that the reader of the pipe reads PIPE_BUF bytes at a time.
* For the systemd integration I am developing, I need one file for stderr logs and another file for stdout logs. This is more complicated than having a single file for all logs.
* The distinction between stdout and stderr is unclear. No messages from StartupLogger are meant to be piped to another program.
Address these problems by only logging to stderr.
From a user's point of view, this diff should not change behavior.
Reviewed By: chadaustin
Differential Revision: D10370931
fbshipit-source-id: b063ed3a66cefc846d30ce1486759cbd3f3ea29e
Summary:
Note that this is more aggressive than the tweakdefaults version; it doesn't allow named branches to be created or listed at all with the config set. As we want to delete named branches entirely, this seems reasonable.
I ported the `--new` flag, but made it optional, to make it easier to support both the core and tweakdefaults callers.
Saurabh Singh and I chatted with the releng team (Craig).
**tldr:** we can remove named branches from the UI but should keep the innards for now
Today the releng team doesn't make any branches (#continuous), but in the near-term future, they might either:
- create remote bookmarks in hg (similar to fbsource)
- create branches in SVN (which are then synced as hg branches by hgsubversion)
- We'll leave all the hgsubversion code to do this alone, just remove user-facing entry points
- They'll also need to be able to query for a commit's branch (`hg log -T '{branch}'` will still work)
- Once www is on hg, this possibility goes away and we can kill named branches for good
Reviewed By: ikostia
Differential Revision: D10401485
fbshipit-source-id: 6f2f3ae28af249bae1fdf782eb14fe7bfc472bb7
Summary:
In another diff, I want to set an environment variable within a test and restore its original value later. Code to do this exists already inside setup_eden_test, but it's hard-coded to HOME. Generalize the code and factor it into its own class so it can be used in my other diff.
This diff should not change behavior.
Reviewed By: simpkins
Differential Revision: D10282151
fbshipit-source-id: 063ed980889441e7b693cf7f1c84f5039b2cbd64
Summary:
In Python 3, `pexpect.spawn` objects come in two flavors: bytes and unicode. The flavor is selected using `pexpect.spawn.__init__`'s `encoding` parameter.
For the bytes flavor, `pexpect.spawn`'s instance attributes generally use `bytes`:
class spawn:
args: Sequence[bytes]
before: bytes
command: str
For the unicode flavor, `pexpect.spawn`'s instance attributes generally use `str`:
class spawn:
args: Sequence[str]
before: str
command: str
Teach the type checker (mypy) about these two flavors to reduce false positives in calling code.
Reviewed By: simpkins
Differential Revision: D10280499
fbshipit-source-id: b4475226423ef37ff5bf2bbae0eb9deac6a1099a
Summary:
RestartTest._spawn_restart dynamically types its kwargs parameter, hiding potential type errors. The kwargs parameter is unused anyway, so delete it and tidy up the related code.
This diff should not change behavior.
Reviewed By: pkaush
Differential Revision: D10281769
fbshipit-source-id: 3e7f7078edf9e47013a9eb9ba42cf51fa8dd0a16
Summary:
Update hg absorb to use the templater and display the first line of the
description of the affected commits.
Also change the default behaviour to display the changes and prompt the user to
confirm that absorb is going to do what they want. Use the new `-a` option to
get the old behaviour.
Reviewed By: quark-zju
Differential Revision: D10366510
fbshipit-source-id: 23df2e5ab2d21a0805a16e7118686a29634d2918
Summary:
Update `EdenFS.cleanup()` to only delete its `eden` directory if it was
created as temporary directory in `__init__`. If the caller specified an
explicit directory avoid deleting it.
All of the users that currently create `EdenFS` with an explicit directory
argument already perform their own clean-up of this directory if needed.
Therefore this diff does not meaningfully change the behavior of existing
programs, it simply make the API behavior less surprising.
Reviewed By: strager
Differential Revision: D10351822
fbshipit-source-id: e9ba7f5869bd2cc221a84be45df233acc7a1a660
Summary:
We go through a lot of effort to set EDENFS_FAKE_EDENFS for tests and verify its path. However, we never actually use the path! We always fall through to searching candidate paths, ignoring the value of the environment variable.
Add an early `return` so the environment variable is respected.
Reviewed By: simpkins
Differential Revision: D10369153
fbshipit-source-id: f1767a240361c6c7e1c95f93c6284a4ac95c2645
Summary: This seems to yield about 18% space savings compared to bzip2.
Reviewed By: strager
Differential Revision: D10351855
fbshipit-source-id: 58b969a8df4a9eb075526d5cc919c99ff6e8f4f3
Summary: Doctor now detects if multiple edenfs are running for the same edenDir and recommends to kill the rogue one.
Reviewed By: strager
Differential Revision: D10098157
fbshipit-source-id: 8a785a9fee0c768696f897a4950552d141012b2a
Summary:
The functionality we care about is provided by the `fbamend`
extension. Therefore, lets replace the `amend` extension with the `fbamend`
extension.
Reviewed By: farnz
Differential Revision: D10320739
fbshipit-source-id: 5700d39f488777fcc4033f60ce0a51cda15ef2ad
Summary: Several tests create a temporary directory and destroy it. Consolidate the temporary directories and their cleanup code to reduce code duplication.
Reviewed By: simpkins
Differential Revision: D10251110
fbshipit-source-id: caa5b5ad49fcb7925f63094594af4a20009d790d
Summary:
We've been shipping our RPMs with toml config support turned on for a
while now. Remove support for the old config file format. Continue to
ship the old format configurations. We'll remove those in a later
diff.
Reviewed By: strager
Differential Revision: D10020958
fbshipit-source-id: 11c2ca3b5da086b142042496a2814699880c4f81
Summary:
The logic we care about has been moved to core so we can delete the
`inhibit` extension.
Reviewed By: markbt
Differential Revision: D10276448
fbshipit-source-id: 44cabe5d561344cf8f196127ce52491e4654b598
Summary:
If 'eden stop' tries to tell edenfs to cleanly shut down, and edenfs never responds, 'eden stop' just hangs. Make 'eden stop' time out CLI -> edenfs Thrift calls. After a timeout, 'eden stop' should by stop edenfs via SIGKILL.
Note: This diff does not affect 'eden restart'.
Reviewed By: chadaustin
Differential Revision: D9982372
fbshipit-source-id: 42e498fb93fd97a2ec963e97c25f4b4101c7d524
Summary:
pexpect prints program output while the program is running, which is very useful when debugging. Make the tests in stop_test.py use this feature.
Also, add some assertions to some of the tests.
Reviewed By: chadaustin
Differential Revision: D10157354
fbshipit-source-id: 210fcde79d5d21c6ee34fad3813e7c56eb298b9b
Summary: If edenfs is unresponsive, 'eden status' hangs forever. Add a timeout to turn the hang into a user-friendly error.
Reviewed By: chadaustin
Differential Revision: D10156229
fbshipit-source-id: 9186826ae6b131a193b1499c8baac616d131357f
Summary:
I want to test more conditions for 'eden stop', such as the edenfs process taking a long time to respond to the Thrift stop request. Refactor how fake_edenfs decides to handle stop requests to make it easy to add new behavior.
Aside from changing the log messages, this diff should not change behavior.
Reviewed By: wez
Differential Revision: D10019459
fbshipit-source-id: f6d5f514a67435a446d8d672bf7d21af77d76c56
Summary: HypothesisSimpleTest.test_create is timing out on Sandcastle (Facebook's CI). It looks like the hypothesis tests run indefinitely on Sandcastle. Set a time limit so the tests complete and provide signal.
Reviewed By: wez
Differential Revision: D10053452
fbshipit-source-id: 8c8cb2c375a73b7727e827ea2a9e7d56ab390e54
Summary:
'eden stop' has poor test coverage. Write tests for the common case and some edge cases which 'eden stop' already handles.
This diff should not change behavior.
Reviewed By: wez
Differential Revision: D10017819
fbshipit-source-id: 4d9f5db52187c34c62a9379a6b3dd62f62894233
Summary:
We're looking into doing some performance optimizations in Eden to the
open() and close() calls, since they are never handled by the kernel
and always pass through to the underlying user-space FUSE daemon. But
first, let's capture some baseline numbers on Eden and non-Eden
checkouts.
Reviewed By: strager
Differential Revision: D10036941
fbshipit-source-id: 64f3414a4572fd963017491db37d70e6b5ae4f24
Summary: D6612669 introduced a workaround for edenfs not running as a child of sudo if you run Eden's integration tests as root. D9980225 fixed the underlying bug in 'eden stop' which necessitated the workaround. Remove the now-unnecessary workaround.
Reviewed By: wez
Differential Revision: D9984524
fbshipit-source-id: b9ed7114cd7385a13899d16b4da0d40e6e9fd704
Summary:
This begins adding a framework for generating snapshots of Eden state for use
in tests. I plan to add a few kinds of tests based on these snapshots:
- Tests that ensure Eden can successfully load snapshots created by older
versions of the code.
- Tests that corrupt the snapshot data in various ways and then confirm that
it can be repaired by `eden fsck` and/or still loaded successfully by
`edenfs` even if it was not repaired with `fsck` first.
This code still needs some additional functionality, but I figured it was
worth checking in what I have so far. The main functionality that remains to
be added is unpacking the snapshots and updating absolute paths inside the
config files so that they work in the new path where they were unpacked.
Reviewed By: strager
Differential Revision: D9690267
fbshipit-source-id: a2660e49b84d7833e6778108d9abe081ab7e2cbd
Summary:
Move the `_cleanup_tmp_dir()` function out of the integration lib's testcase
module, and make it a public function in the integration lib util module.
This will make it easier to re-use from some other python utilities that need
to be able to clean up Eden state directories.
Reviewed By: strager
Differential Revision: D9690266
fbshipit-source-id: 157c5221f81f90a3e9612315681459d16693cdbc
Summary:
Update the Eden unit tests and integration tests to set the `NOSCMLOG`
environment variable when running `hg` commands. This ensures that our
mercurial telemetry wrapper does not log events from Eden test runs.
Reviewed By: quark-zju, strager
Differential Revision: D9893096
fbshipit-source-id: c0dd4b5eb042dcb5e9493c89aaee10a513022bae
Summary:
If the overlay file for a directory is corrupted (e.g. empty), Overlay::scanForNextInodeNumber throws. This causes Eden to crash on start [1]. Fix the crash by ignoring corrupted directories.
[1] `test_mount_possible_after_corrupt_directory_and_cached_next_inode_number` reproduces this crash.
Reviewed By: chadaustin
Differential Revision: D9806105
fbshipit-source-id: 1b95083b6a6aa253a2296d6f754edbf4b9f64734
Summary:
The C++ code disallows use of "." in RelativePaths, but it's
reasonable for users of the Thrift API to pass "." to indicate the
root of the mount. Handle that in the EdenServiceHandler.
Reviewed By: strager
Differential Revision: D9647776
fbshipit-source-id: b61c2d1c0dcd69ccfa38bf27379281d10cdf1ceb
Summary:
Fix the code in `eden_dirstate._call_match_callbacks()` to correctly match
ignored symlinks that are explicitly listed in the matcher's file list.
It looks like this was just a mistake in the original code on my part: I
intended to check both `S_ISREG()` and `S_ISLNK()` but instead incorrectly put
the check for `S_ISREG()` twice.
Reviewed By: wez
Differential Revision: D9476524
fbshipit-source-id: 67e0fa7c2fbaac97598a8e2d028c9ef0999ed88a
Summary:
When we load an empty blob from the LocalStore double check with the
BackingStore to confirm that it should actually be empty.
We have seen multiple instances of files being incorrectly imported as empty.
So far this error has always been fixed by a re-import. We still haven't
tracked down the root cause, but this change should help workaround the issue
by ensuring that we double check the file contents before returning the data.
Reviewed By: chadaustin
Differential Revision: D9476522
fbshipit-source-id: 6d57cf15c42736ecbcb106a731259b77db06d8f1
Summary:
Update all of the `eden debug` commands to use the newer `find_checkout()`
code rather than older `get_mount_path()` function.
The `find_checkout()` code makes sure that the EdenInstance actually points to
the correct edenfs instance for this checkout, and also works with checkouts
that are not currently mounted. In particular this allows
`eden debug overlay` to examine the overlay state even when the checkout is
not currently mounted.
Reviewed By: wez
Differential Revision: D9389865
fbshipit-source-id: 00578519d4805157a30c9b39abee9838925e8e76
Summary:
Watchman's Eden integration has a bug where the combination of
Watchman querying Eden for overlapping delta ranges ("give me changes
between X and Y, now changes between X+1 and Y+1") and Eden eliding
redundant change events ("add-modify-remove" -> []) results in
Watchman sometimes reporting that a file exists in its final
subscription update when it no longer does.
The fix is to never elide events, even for files that were added and
removed in the same sequence. To continue to support Watchman's `new`
flag, track whether a file existed at the beginning and end of a
journal delta.
Reviewed By: wez
Differential Revision: D9304964
fbshipit-source-id: f34c12b25f2b24e3a0d46fc94aa428528f4c5098
Summary:
To prevent the regression fixed by D8323051, make sure we run at least a couple
integration tests with the RocksDB LocalStore implementation.
Reviewed By: wez
Differential Revision: D8408390
fbshipit-source-id: 8fab4041ae39915d8be80e42814aab375c4acdda
Summary:
This prevents `hg status` from blowing up with a UTF-8 decode
error inside the generated thrift code.
Push safety concerns:
* This doesn't change the wire representation of the data
* Existing clients that believe it to be a string will continue to have
the same behavior
* Buck has its own copy of an older version of the thrift spec, so it will
continue to work "OK".
* When buck resyncs with our thrift file, some changes will likely be needed
to convert the byte arrays to strings or paths or whatever is appropriate
for bucks internal API
Work "OK" above means that clients that currently believe that `string` is
utf-8 encoded will have a runtime error if we ever send them a path that
is not utf-8. This is the behavior prior to this diff and will continue
to be the behavior for clients (like buck) that have an older version
of the thrift file.
Reviewed By: simpkins
Differential Revision: D9270843
fbshipit-source-id: b01135aec9152aaf5199e1c654ddd7f61c03717e
Summary:
Add some integration tests that invoked "eden doctor" and "eden list" inside a
stale mount point, and confirms that they work correctly. These commands do
fail in dev builds, since python itself does not handle this error.
Fortunately XAR-based python archives avoids this error, so these commands
work in opt mode builds.
Reviewed By: strager
Differential Revision: D8905489
fbshipit-source-id: 0666f8cb0715d61697253bee9fe0bde2f4a3fa15
Summary:
Update `InodeMap::updateOverlayForUnload()` to catch exceptions that occur
trying to save state to the overlay. If something goes wrong when saving
state there is not much we can do other than log an error. We still want to
unload the inode, and code that is unloading inodes generally cannot deal with
exceptions at this point. In particular if this error occurred while trying
to shut down an EdenMount the code would crash.
Reviewed By: chadaustin
Differential Revision: D8884795
fbshipit-source-id: c2f850f13d775be4b0a0a10f9df3948c7b2c8f4a
Summary:
Encountering a truncated overlay file doesn't necessarily indicate a software
bug in Eden. Depending on the underlying filesystem this often happens after
a hard system reboot since we write the overlay files without an `fdatasync()`
call.
Change the code to simply log an error and throw an exception rather than
using `EDEN_BUG()`. This makes it possible to exercise this code path in
tests without having it crash in debug builds.
Reviewed By: chadaustin
Differential Revision: D8988209
fbshipit-source-id: 8c0fe1dae692f4c493413d3939d2e4c21e0da596
Summary:
Sometimes, Eden's overlay (in `$client_dir/local/`) gets corrupt. In
particular, sometimes overlay files can be truncated or missing after a hard
reboot where the underlying filesystem state was not flushed to disk.
For such files, open(), stat(), unlink(), etc. from Eden report ENOENT, yet
readdir() on the containing directory shows that the file does exist.
In other words, the problematic file is undeletable:
```
$ ls -la dir/
/bin/ls: cannot access dir/corrupt_file: No such file or directory
total 0
drwxr-xr-x. 3 strager 0 Jul 10 21:41 .
drwxr-xr-x. 48 strager 0 Jul 10 21:41 ..
-?????????? ? ? ? ? corrupt_file
$ rm dir/corrupt_file
rm: cannot remove ‘dir/corrupt_file’: No such file or directory
```
Allow users to delete these problematic files (if the file was a regular file
and not a directory) by doing the following:
* Allow corrupt regular files to be unlink()d successfully.
* Allow corrupt regular files to be stat()d.
Making stat() succeed is a requirement by FUSE:
* For unlink(), FUSE performs FUSE_LOOKUP before FUSE_UNLINK. If FUSE_LOOKUP
fails, unlink() fails. Therefore, we must make FUSE_LOOKUP succeed for
corrupt files.
* For stat(), FUSE performs FUSE_LOOKUP and sometimes FUSE_GETATTR. Since we
must make FUSE_LOOKUP succeed (for unlink()), it's natural to make
FUSE_GETATTR succeed too.
A future diff will fix corrupted directories.
Reviewed By: chadaustin
Differential Revision: D8884793
fbshipit-source-id: 1100037bf52475fcca66f39946b917ce604f12dc
Summary:
Fix the arguments to `doctor.cure_what_ails_you()`, which were updated in
D8870494.
Reviewed By: chadaustin
Differential Revision: D8940413
fbshipit-source-id: 21d45036f668c9bae9a2002c360532918eb157a5
Summary:
This returns a `Repository` object and not a `Repository` subclass type.
This type information has been broken since D7512320 replaced the
`get_repo_class()` method with `create_repo()`.
Reviewed By: strager
Differential Revision: D8940415
fbshipit-source-id: 7711a2a33bc12fc361dfd0ff3f47eba1e587e6e7
Summary:
Update `EdenFS.run_unchecked()` to return the full
`subprocess.CompletedProcess` argument rather than just the return code.
This will make it possible for future tests to check the output of the command
in addition to just its return code.
Reviewed By: strager
Differential Revision: D8905491
fbshipit-source-id: 7f17f5d103a06aeb56a3be0618683bd1c017fdf0
Summary:
D8508489 updated the integration test code so that the `capture_output`
argument to `EdenFS.run_cmd()` was no longer necessary and was not used.
However I forgot to actually remove the argument in that diff. This deletes
the argument.
Reviewed By: strager
Differential Revision: D8905492
fbshipit-source-id: d7a691f7d628a9fd5daa6723575e4454480f6aa1
Summary:
As of D8631281 `eden clone` now makes the mount point directory read-only and
puts a readme file inside of it. Since the directory is read-only the
integration tests are unable to clean it up using a normal `shutil.rmtree()`
call.
This fixes the integration tests to try making the directory writable first
if it encounters an error during cleanup.
Reviewed By: strager
Differential Revision: D8905488
fbshipit-source-id: c6b9d0b12c2b1dcb4a0839757cbcc3aad179bfc1
Summary:
This adds initial code for an `eden debug fsck` command.
At the moment this simply scans the overlay and reports errors, but does not
correct any issues. In subsequent diffs I'll update it to support correcting
some problems.
There are also several other issues that I'll improve in subsequent diffs as
well. At the moment this code uses the same functions as other CLI commands
to find the client directory, and these only work if the checkout is already
mounted. For `fsck` we generally want the checkout to be unmounted before
checking it. I also plan to clean up some of the output handling as well.
Reviewed By: strager
Differential Revision: D8813714
fbshipit-source-id: 90238e94540db1cc06c17eabf8ce3a1bb808d4b3
Summary:
This makes it easier to add some test coverage.
There's no real functional change in this diff; the only code change is to
throw a system_error instead of a thrift eden error wrapper class from the core
globbing code. There's a little bit of code to restore this exception type in
the callers in EdenServiceHandler; this is covered by existing integration
tests, but I've also expanded that coverage to cover both variants of the glob
thrift calls.
Reviewed By: strager
Differential Revision: D8776767
fbshipit-source-id: 3ea4ea642ae5108aa4b0153541bd3604f010b54c
Summary:
Update `eden restart` to explain that the restart will be disruptive when
performing a full restart, and prompt the user for confirmation. This prompt
can be overridden with the `--force` option. Also print more messaging after
the restart completes telling users what to do if they see "Transport endpoint
not connected" errors from programs still using the old mount points.
This also updates `eden restart` to more carefully handle cases where there is
an existing edenfs daemon running but it is not fully healthy.
Reviewed By: wez
Differential Revision: D8731004
fbshipit-source-id: 05762187b47057b2930d0a6b71b0a6fdbd4aa7e5
Summary:
This moves the fsattr helper program into the helpers/ directory
created in D8373672.
Reviewed By: chadaustin
Differential Revision: D8705391
fbshipit-source-id: 43e04d8b06b5d17caefafdfc750e3e60cad1b0cc
Summary:
Update "eden clone" to create a README file inside the mount point directory
before the checkout is mounted. When Eden is running normally this file will
not be visible because the Eden checkout has been mounted on top of this
directory. However if the checkout is not currently mounted users will see
this file instead.
This should hopefully make it easier for users to figure out how to remount
their checkout on their own, and should prevent them from worrying that their
files have been deleted when the mount point simply is not mounted.
Reviewed By: chadaustin, wez
Differential Revision: D8631281
fbshipit-source-id: 7610450983687a839bdbc3558ff35e0422edc3be
Summary:
The eden integration tests currently run hg through the telemetry wrapper.
This wrapper script adds profiling arguments to a percentage of the commands
it runs. The extra arguments that it adds are not entirely transparent to the
user, and are unfortunately reported in the output of "hg journal".
This was causing occasional test failures in the Eden integration tests that
check the output of "hg journal".
This diff changes the Eden code that checks the journal output to try to strip
off these flags before checking the command output.
Reviewed By: wez
Differential Revision: D8631180
fbshipit-source-id: 004d3bcce291ffc2208f85a29cb34a921f16fdde
Summary:
Update the python CLI code to let the C++ edenfs code handle daemonization
rather than daemonizing in the python code.
This means that we now daemonize after the `sudo` invocation rather than
before it. This allow's sudo's wrapper process to exit once edenfs starts,
rather than remaining around for the lifetime of the edenfs daemon. This
means that the original process' stdout and stderr file descriptors are
closed properly rather than being held open by the sudo wrapper.
This also removes the logic in the CLI that waits for edenfs startup and can
potentially time out and give up.
Reviewed By: strager
Differential Revision: D8508489
fbshipit-source-id: 6a38439d5596049d6f40cb6589b6865c229f736e
Summary:
This updates edenfs to be able to daemonize itself, and moves the
daemonization logic from the python CLI code into C++.
The main benefit of this is that we can now do a better job of reporting
messages to the user during start-up. We can log around potentially slow
operations (opening the RocksDB local store), and we can print messages
directly to the user if startup fails. Previously most failure messages would
go only to the eden log and would not be printed to the user's terminal.
This also fixes some issues where stdin and stdout were not closed properly
when daemonization was performed by the CLI. sudo needed access to these file
descriptors in case it needed to prompt for a password, and it would then hold
the descriptors open until edenfs exited.
Reviewed By: wez
Differential Revision: D8373672
fbshipit-source-id: 3272bff2208596f41d26e479c82c700d6c1efe11
Summary:
Previously `eden rm` tried to look at `<path>/.eden/root` to find the root of
the mount point. This failed if the checkout was not currently mounted.
This updates the code to try looking for the exact input path in the config
file in this case. It also avoids sending an `unmount()` thrift call to
edenfs in this case as well.
Reviewed By: strager
Differential Revision: D8558765
fbshipit-source-id: a5d9db7eee702817713b089a457f25710438661e
Summary:
When running the integration tests in parallel in a dev mode build I
occasionally see timeouts in the shutdown code in some tests. (Usually this
seems to happen in some of the remount tests that create multiple mount points
and shutdown and restart edenfs multiple times.)
Usually when I see this error edenfs does exit successfully on its own just
after this 15 second timeout has expired.
This bumps up the timeout to 30 seconds to avoid spurious failures.
Reviewed By: pkaush
Differential Revision: D8559052
fbshipit-source-id: 567062b08b34eced8f05538607ae9f4773c4aa9c
Summary:
Previously `eden list` reported normally running active checkouts with an
`(active)` suffix, and non-mounted checkouts with no suffix.
The non-mounted situation is rare. This diff changes the command to report
non-mounted checkouts with a `(not mounted)` suffix. I removed the `(active)`
suffix from normally-running checkouts now, but let me know if people think it
is still valuable to keep this annotation in the output.
Reviewed By: chadaustin
Differential Revision: D8558770
fbshipit-source-id: ccce0f7c2031e50f2847089e49007274c31a69a7
Summary:
I saw RemountTest.test_restart_twice() fail once when running locally, but I
haven't been able to reproduce the failure yet. This updates the checks in
this code to hopefully provide better information about what went wrong in
case we see this failure again in the future. This now checks the `eden list`
output first so if something goes wrong we will hopefully have more
information about which checkouts were or weren't remounted.
Reviewed By: strager
Differential Revision: D8528544
fbshipit-source-id: 4f50145aa97956f45f50813070fb73836bc7acde
Summary:
We've seen what appears to be phantom calls to shutdown() so we'd like
to add some degree of auditing. This diff adds a new method with some
context; this will allow us to distinguish between `eden stop`, `eden restart`
and eden server internal calls to the `shutdown` method. It may still
be possible that something else is calling our shutdown method, but it
seems unlikely as we're only accessible to our own code via a unix domain
socket.
Reviewed By: chadaustin
Differential Revision: D8341595
fbshipit-source-id: 50d58ea0b56e5f42cd37c404048d710bde0d13a3
Summary:
D7946842 moved some of the daemon management code from util.py to daemon.py
but accidentally broke handling of TimeoutError exceptions. The code in
daemon.py was trying to catch python's built-in `TimeoutError` type, while the
code in util.py throws its own `util.TimeoutError` type.
This removes the custom `TimeoutError` type in util.py and just uses the
built-in python type to be consistent everywhere.
Reviewed By: wez
Differential Revision: D8373739
fbshipit-source-id: 8523c444fef74b0758c38b4fe2644d4f68d38442
Summary:
mkdtemp deliberately removes these bits and doesn't respect
the umask. We perhaps should respect the umask ourselves here.
Reviewed By: chadaustin
Differential Revision: D8335539
fbshipit-source-id: 98b8f3e2db17c65904b12627e52966d5f6b729ef
Summary:
Some folks might attempt to load this extension despite it
not being useful with eden. The extension knows not to hook in to
the repo when eden is in used, but the various commands to manipulate
the sparse config do not.
This diff adds a little helper function to show a more reasonable
error message than an ugly stack trace when attempting to use them.
Reviewed By: simpkins
Differential Revision: D8328229
fbshipit-source-id: d7cc4c7047b11f8da96e26ab70bd522b52546151
Summary:
Fix the integration tests to set CHG_BIN correctly when run manually rather
than via `buck test`.
Reviewed By: wez, quark-zju
Differential Revision: D8329593
fbshipit-source-id: 79fa3c9e9538e4b30a98ef777515527562e1ac1b
Summary:
This fixes a bug simpkins pointed out in D6891479 - we weren't
updating mtime and ctime on renames.
Reviewed By: simpkins
Differential Revision: D7937303
fbshipit-source-id: 08fd8f4fe5d99d33e9f7629965d6146330c8f35b
Summary:
I kept running into issues trying to get graceful restart and
flush_cache to work together in the hg integration suite, so add a
test to ensure flush_cache succeeds after a graceful restart in the
main integration suite.
Also, to make the test's output easier to follow, add logging when
invalidating inodes.
Reviewed By: simpkins
Differential Revision: D8215961
fbshipit-source-id: 33db4292af3969ae23940c3027ba513ed20c53fb
Summary:
Add a command line option to start edenfs only if there is at least one
Eden checkout already configured.
This makes it easier to automatically start edenfs on system boot only if the
user needs edenfs.
Reviewed By: wez
Differential Revision: D8174819
fbshipit-source-id: bf585b8c5aa719d3b6424567f2e21c1c63850d66
Summary:
This adds a debug command to blow away all RocksDB information that
can be reproduced from Mercurial. We will use it to help an Eden user
recover from a corrupted blob.
Reviewed By: bolinfest
Differential Revision: D8108649
fbshipit-source-id: 056dec19d51b9e430b3c2a249747b26830cfc875
Summary:
The Overlay is the natural home for nextInodeNumber_ now that every
directory allocates inode numbers for its children right away. This
also simplifies serializing nextInodeNumber_ to disk in the following
diff.
Reviewed By: simpkins
Differential Revision: D8192442
fbshipit-source-id: 9b776a73c8d7653002b55985d592b1746e52f878
Summary:
Add a new integration test that modifies a file, commits it, then restores it
to its original contents and creates a new commit with the revert.
This test isn't really exercising any Eden-specific behavior. The only reason
it is is interesting is that this happened to trigger a bug in the underlying
mercurial cdatapack code that caused the test to fail when run in treemanifest
mode. The mercurial cdatapack bug is fixed in D8131020
Reviewed By: quark-zju
Differential Revision: D7826284
fbshipit-source-id: 32da2db04452d5df0527f6be00b0c4c1a56c900b
Summary:
Update find_executables.py to be able to find the HG wrapper and binary using
buck rules. This is necessary in order to be able to run the integration
tests manually rather than via `buck test`. Without this the tests try to use
`/bin/hg.real` which tests the wrong version of the mercurial code and the
eden extension.
Reviewed By: bolinfest
Differential Revision: D8200168
fbshipit-source-id: d2a68c765c7143bbe613f32eef1268ed567790a7
Summary: Mostly empty lines removed and added. A few bugfixes on excessive line splitting.
Reviewed By: cooperlees
Differential Revision: D8198776
fbshipit-source-id: 4361faf4a2b9347d57fb6e1342c494575f2beb67
Summary:
Relax the restriction on changing uid/gid on inodes. We'll see what
cans of worms this opens I guess. (Landing this is low priority, but
might be important for making some of the existing tooling in fbsource
and www work.)
Reviewed By: simpkins
Differential Revision: D7768655
fbshipit-source-id: 95fe02fe7ddc001335dbdb34e16a989a85820240
Summary:
Add a `remove` command to the Eden CLI. This behaves like
`eden unmount --destroy`, but calling this "remove" is hopefully a more
intuitive UI. If stdin is a TTY this command also prompts the user for
confirmation before removing the checkout.
I plan to deprecate the `eden unmount --destroy` command in a subsequent
diff.
Reviewed By: wez
Differential Revision: D8086823
fbshipit-source-id: 562cf0f998eea416b80589b188eee255a10b9699
Summary: Now that permissions on directories work, verify umask works as intended.
Reviewed By: simpkins
Differential Revision: D7783743
fbshipit-source-id: 635221cd3255cc20e9ffa26b6838922c4a4110f3
Summary:
This moves most inode metadata management into InodeBase and
persists permission bits (and eventually uid/gid) across Eden runs.
Reviewed By: simpkins
Differential Revision: D7035163
fbshipit-source-id: 50145449b56aad1662d53156e6e4960c5f7b6166
Summary: Store tree and file timestamps in the InodeTable so they persist across runs.
Reviewed By: simpkins
Differential Revision: D6891479
fbshipit-source-id: 1c9e6266375aceeaf293a81e73cf7f5334dbc32d
Summary:
This is stacked on top of Black 18.5b0.
allow-large-files
Reviewed By: carljm
Differential Revision: D8061834
fbshipit-source-id: 92e3645e159b60d77cf7e0bec64a8262ca4e88c2
Summary:
This updates the Eden mercurial extension to no longer invoke the Eden
`resetParentCommits()` thrift call when `setparents()` is called on the
dirstate map. Instead we now defer the call to `resetParentCommits()` until
`write()` is called to write the dirstate data to disk.
Informing edenfs of the parent change as soon as `setparents()` was called was
problematic, as this made edenfs reflect the change before the transaction was
committed. Some mercurial commands, notably `hg status` also call
`setparents()` on the dirstate but never write this back to disk at all. This
is problematic since `hg status` calls `setparents()` without holding any
mercurial locks. As a result it may call `setparents()` with the "wrong"
parent if another mercurial process is running and is in the middle of a
transaction.
Reviewed By: bolinfest, chadaustin
Differential Revision: D7980375
fbshipit-source-id: 4f5e4391fd291d4ea5fc93bb9d49ed0380fc1721
Summary:
While running the secfs filesystem validation tests against Eden, I
discovered a test that caused the eden process to abort. I bisected
and found that D7451330 regressed renaming a directory onto an empty
one. This fixes that case.
Reviewed By: simpkins
Differential Revision: D7945727
fbshipit-source-id: 592ede1b391528c02cd12b2b6ebbf3733fe8f503
Summary:
We need to introduce a new `includeDotfiles` option to `glob()`. [As we have
done for all of our Thrift API, to date], rather than define `glob()` so that it
takes a single struct, we specified the parameters individually, so we can no
longer add new params to `glob()`.
In particular, we need to support `includeDotfiles` because we often configure
Buck to use Watchman to implement `glob()` in `BUCK` files, and when Watchman is
used in Eden, it leverages Eden's Thrift API to implement `glob()`. Because
Buck's `glob()` has an `include_dotfiles` option, we must be able to honor it
and pass it all the way through to Eden's `glob()` implementation.
Rather than name the new API `glob2()`, I'm electing to go with `globFiles()`.
(Perhaps once we eliminate all known users of `glob()` in the wild, which
requires turning over the current version of Watchman we have deployed, we can
redefine `glob()` in `eden.thrift` to be the same as `globFiles()` and then
update everyone to use `glob()` again so it has the more intuitive name.)
Reviewed By: wez
Differential Revision: D7748870
fbshipit-source-id: 92438f9c41e4fbdbd6cdccca5fce0e41cc3e9b07
Summary:
When comparing two source control blob hashes, identical hashes can be assumed
to mean that the file contents are equal. However, differing hashes does not
necessarily mean that the file contents differ. In particular, mercurial
hashes history metadata in addition to the file contents when computing the
blob hash.
This updates Eden to always compare the file contents when the source control
blob hashes differ, rather than assuming that the file contents are different.
Reviewed By: wez
Differential Revision: D7825900
fbshipit-source-id: e611124a66cdd5c44589f20d1d4665a603286530
Summary:
While testing the fbclone --eden changes I found that
we were failing to pick up the bind mounts. The root cause of
this is that the default head rev for hg repos is `.`, so if
the source repo is bare then we won't find an `.arcconfig` file
and thus won't be able to set up the bind mounts.
Reviewed By: simpkins
Differential Revision: D7782489
fbshipit-source-id: f41d3a7daf39ecd0946707cb2c3211d70c36eea2
Summary:
Disallow any kind of mutation operation inside of the .eden directory. We had some
code in place to prevent some of this already, but errors (including EPERM) weren't
passed out from unlink and rename out to FUSE.
Reviewed By: simpkins
Differential Revision: D7781691
fbshipit-source-id: aaecf13779eca75d6ee8765fc8bb3727ce9341de
Summary:
By introducing `_assert_glob()`, it eliminates a bit of the boilerplate in the
individual test cases. I think it also makes things easier to read because now
the glob patterns appear before the result of the glob.
Though most importantly, this will help with a subsequent change where I am
going to change the contract of the Thrift API, as now it can be done within
`_assert_glob()`.
Reviewed By: wez
Differential Revision: D7748871
fbshipit-source-id: 9609cde104979e892f0858e7a0c7e53e976ff8e2
Summary:
In the spirit of TDD, I am writing the tests first and annotating them with
`unittest.skip()` so the build still succeeds.
Reviewed By: chadaustin
Differential Revision: D7741507
fbshipit-source-id: 4ede0b933c75f9be4016e399936ff2e47eb4e538
Summary:
I think this makes it easier to reason about the coverage of the space of
inputs to `glob()`.
Reviewed By: chadaustin
Differential Revision: D7741509
fbshipit-source-id: 5882d859df95279189512716004263dd5320ff3f
Summary:
I would like to to test more inputs for `glob()` with different characteristics.
I think this would be more logically organized when divided across a number of
test methods in a single `GlobTest` class.
This revision does a straight move of the eixsting `test_glob()` method without
introducing any new test cases.
Reviewed By: chadaustin
Differential Revision: D7741506
fbshipit-source-id: 141341d74265f3949ed7523f40a56f98d95ee13e
Summary:
I found this while testing the fbclone script; clone was
not performing realpath on the destination path, so we ended up
creating and registering a mount under a non-canonical name.
That doesn't sound so bad except that unmount always canonicalizes
the path to be unmounted and this made it impossible to unmount
such a repo.
Reviewed By: chadaustin
Differential Revision: D7766366
fbshipit-source-id: cbfd484a1481d5082969fc97eaf211c18c0d30be
Summary:
Try treating the argument as a repository path first, rather than a well-known
repository name. If we can find a mercurial or git repository at this path,
look for a .arcconfig file in the repository, and try to identify the
repository project type from this file. Use this project name to load our
default bind-mount and hooks configuration for the repository.
If the argument is not a valid repository path, fall back to treating it as a
well-known project name in our repository configs.
This also updates `eden clone` to print a few more diagnostic messages as it
works. (For instance, printing when it is starting the edenfs daemon, and
reporting the commit ID that was checked out.)
Reviewed By: wez
Differential Revision: D7739917
fbshipit-source-id: ac927b9e93039e4d1b8afa80466c2eee3a8829e9
Summary:
Update the Eden mercurial extension to read the `.eden/root` symlink to
determine what Eden thinks the mount path is. This might be different from
what directory mercurial thinks it is in if a parent directory of the Eden
mount has been bind-mounted to an alternate location.
Maybe in the future we should update thrift clients to pass in the client ID
(currently readable via `.eden/client`) rather than the mount path. That would
make it less likely for clients to accidentally forget to read `.eden/root` and
pass in the wrong mount path.
Reviewed By: wez
Differential Revision: D7705655
fbshipit-source-id: 7bd1e8013b99a52ff06dd45f63d6669b66bdf577
Summary: I forgot to remove this comment when I fixed the test in D7565503.
Reviewed By: chadaustin
Differential Revision: D7716106
fbshipit-source-id: 58acd71ac96560f2cc454b3a7da75bb61e6e71c7
Summary:
Rename the rule for the Eden CLI script from `cli` to `eden`. This way the
generated par file will be named `eden.par` rather than `cli.par`. This seems
like it will be slightly less confusing to new developers starting to work on
Eden--currently it doesn't seem particularly intuitive that `cli.par` gets
installed as `eden`.
Reviewed By: bolinfest
Differential Revision: D7658023
fbshipit-source-id: 7afc5e0b703d02751b509efe42f8b8be1f56bbc1
Summary:
These were making some noise in the Nuclide diagnostics pane.
In the course of making these changes, I ended up reformatting
some files, as well. Perhaps we should flag flake8 and autoformat
violations more aggressively in CI now?
Reviewed By: chadaustin
Differential Revision: D7658030
fbshipit-source-id: b52a29b1cc242967f868dcc8ee46ec1bb9bdfbee
Summary:
Fix a couple issues when when performing an `hg update` with updatecheck set to
`noconflict` and when conflicts are detected:
- Report the files with conflicts. The conflicts may be non-obvious to the
user if the conflicts are in ignored files that are not tracked in the
current commit but are present in the destination commit.
- Do not invoke the preupdate hook or create the .hg/updatestate file until
after we have performed the conflict check. Otherwise the repository will be
marked as in the middle of an unfinished update even though we never
attempted the update itself.
Reviewed By: quark-zju
Differential Revision: D7581782
fbshipit-source-id: e290ebd6a27f228a805bea40aabb3f3cc633cc7d
Summary:
Update the assert_status() function in the integration tests so that it also
checks for unfinished update/rebase/graft/etc operations. We unfortunately
have to manually check for the presence of these files ourselves, since
`hg status` provides no mechanism to report this data when `HGPLAIN` is set.
Reviewed By: quark-zju
Differential Revision: D7581781
fbshipit-source-id: 230234e5b8ce28cd3569cdacced686fed2a9dd32
Summary:
This rewrites the `walk()` and `status()` methods in the `eden_dirstate` class.
The changes to the `status()` function should not have any major behavior
changes: this primarily moves logic out of `EdenThriftClient.getStatus()` and
into `dirstate.status()`
The changes to the `walk()` function do fix a number of bugs. We now implement
walk by combining Eden's `getScmStatus()` results with the dirstate's
non-normal files and the current parent commit's manifest. All of the glob
matching is done purely in python now. This fixes some cases where Eden's glob
handling behavior did not support some glob patterns allowed by mercurial. The
Eden glob() call also always returned ignored files and hidden files (such as
everything inside the `.hg` directory). I believe this behavior is desired by
watchman, but it caused problems for the `walk()` code.
Reviewed By: chadaustin
Differential Revision: D7565503
fbshipit-source-id: deb91b4772501e8fbdba56e5c099a72d55fb61ee
Summary:
Update `eden_dirstate.status()` to invoke the `match.bad()` and
`match.explicitdir()` appropriately when any of the explicitly specified match
paths either don't exist or refer to directories.
Reviewed By: chadaustin
Differential Revision: D7556850
fbshipit-source-id: 2e8d88ff0e3c09cea2f3ef5b18c9a214231815ab
Summary:
Implement `dirstate.matches()` in Eden by calling `matches()` on the parent
commit's context, and then updating that result with the list of non-normal
files from Eden's dirstate.
Reviewed By: chadaustin
Differential Revision: D7556847
fbshipit-source-id: 34e78c0c125cd1b184a7773a149edafdca34fee1
Summary:
Add integration tests to check the behavior of the `hg journal` command in an
Eden checkout.
Reviewed By: wez
Differential Revision: D7512339
fbshipit-source-id: 9a83a1752835bc6ded9f0c3b2cb29ec24a6db631
Summary:
When running the integration tests, set HGRCPATH to point to the configs found
at `scm/hg/fb/staticfiles/etc/mercurial`
This does affect mercurial's behavior in some cases, and therefore required
changes to some of our tests:
- `hg diff` output now uses an output style similar to `git diff`
- merge conflict markers include some extra commit metadata
Reviewed By: wez
Differential Revision: D7512321
fbshipit-source-id: 6141605147797ea8b13fdb5a06f67beaeff6afe7
Summary:
This slightly refactors the way that EdenTestCase and EdenRepoTest initialize
repositories. This removes the `create_repo()` method from EdenTestCase and
replaces it with separate `create_hg_repo()` and `create_git_repo()` methods.
The `EdenRepoTest` subclasses now provide alternate `create_repo()`
implementations instead of `get_repo_class()`.
This cleans up the code a bit, since `create_repo()` no longer takes different
arguments based on what type of repository is being created. This also will
make it easier in upcoming diffs to further customize the logic that occurs in
`create_hg_repo()`.
Reviewed By: chadaustin
Differential Revision: D7512320
fbshipit-source-id: d268b0ac0ffb33f3dfd34f2dd1917d57033c81aa
Summary:
Once a mount point has been unmounted we no longer need to care about
outstanding FUSE reference counts--we can treat them as if they are all zero.
This updates EdenMount to tell the InodeMap when the mount point is unloaded,
and changes InodeMap::unloadInode() to make use of this information when
deciding if it needs to remember the inode information.
Previously InodeMap would save information for inodes with outstanding FUSE
reference counts. Writing all of this state to the overlay could take a
non-trivial amount of time.
Reviewed By: chadaustin
Differential Revision: D7555998
fbshipit-source-id: 0896f867ce850ab3e61c262776d536de003685ff
Summary:
Update the GitRepository and HgRepository classes in the integration test code
to use the find_executable module to find the location of git and hg.
Reviewed By: ryanmce
Differential Revision: D7512037
fbshipit-source-id: 139072c3493f70abd0dc92acab295d707bee8ad6
Summary:
Refactor the find_executables module to only look up executables when they are
needed, instead of performing all look-ups immediately even if the test in
question may not need all of the binaries. Also add a _find_exe() helper
function to eliminate some code duplication.
Reviewed By: ryanmce
Differential Revision: D7512038
fbshipit-source-id: fdfb8ec70b3f6292603826b3fb22c01dbd1f0d72
Summary:
Add an integration test that nonmaterialized files have the same inode
numbers after restarting the Eden mount.
Reviewed By: simpkins
Differential Revision: D7328311
fbshipit-source-id: 989abbed38626839e0badffeac2d86dc46c8dd72
Summary:
We already had type annotations on most of the `hg` integration tests. This
adds them for the top-level (non-source-control-specific) tests.
typeseverywhere
Reviewed By: wez
Differential Revision: D7459281
fbshipit-source-id: 41266b232ded510d6b63dd3e62c272a0cd6a0e1a
Summary:
Implement the tear-down actions using addCleanup() rather than through a
tearDown() function. This simplifies the logic, and also makes it easier to
write the code that can be type-checked more easily. (We no longer need to
initialize some variables to `None` just to help `tearDown()` identify if they
have been set up yet or not.)
Reviewed By: wez
Differential Revision: D7459278
fbshipit-source-id: de32ebfcb5474a9583f2a51cc9277102189cb77e
Summary:
Update the eden_repo_test decorator so that it no longer automatically adds
`EdenRepoTestBase` as a parent class. Individual test classes still specify
`EdenRepoTest` as their parent now.
This enables `mypy` to correctly figure out that the individual test classes
derive from `unittest.TestCase`.
This basically does the same thing as D6268258 for the top-level integration
tests.
Reviewed By: wez
Differential Revision: D7459280
fbshipit-source-id: 5d18bd241dad77d55541ac3fa1d169496ffe7003
Summary:
This integration tests appears to be frequently hanging on sandcastle, and
leaving behind unkillable zombie processes. I'm disabling it for now until we
can figure out why it hangs and fix it and/or make sandcastle smart enough to
be able to successfully tear down the FUSE mounts.
These stuck processes are likely stuck waiting on a kernel lock that can only
be released by using `/sys/fs/fuse/connections/DEVICE_ID/abort`
Reviewed By: wez
Differential Revision: D7440282
fbshipit-source-id: 70034fd65267eff18bf5a052cde8d11e0a66b46b
Summary:
Fix FuseChannel::processSession() to always process all FUSE requests that it
reads. Previously it checked to see if it should stop immediately after
reading FUSE request. It was possible for the old process to successfully read
a FUSE request, see that it was supposed to stop, and then exit this worker
thread without ever processing this FUSE request. This would cause the client
that sent the request to hang indefinitely, since no response would ever be
sent.
Reviewed By: wez
Differential Revision: D7436867
fbshipit-source-id: c58c2f6c49102fa6b66ac83fc1639595a5277ce0
Summary:
Perform cleanup with an `addCleanup()` function rather than in `tearDown()`.
This is slightly simpler, but also avoids throwing an exception in `tearDown()`
if `setUp()` failed before it created the thrift client.
Reviewed By: wez
Differential Revision: D7436636
fbshipit-source-id: fe525c1fac86bb866c560cca0ad940ea87d96f07
Summary:
Update the `eden clone` command to accept extra command line flags telling it
where to find the edenfs executable, as well as additional arguments to pass
to edenfs.
This enables the `test_clone_should_start_daemon()` integration test to pass in
flags to tell it how to start edenfs correctly.
Reviewed By: chadaustin
Differential Revision: D7433367
fbshipit-source-id: 0eddd93d1332e113fb85fa4b8fc87ba51d7eab2c
Summary:
Update test_version() so that it can succeed even if the RPM is not installed
on the local system.
Reviewed By: wez
Differential Revision: D7433366
fbshipit-source-id: 5502cca02f9b481cc136beb665048525bae72807
Summary:
Update FuseChannel to send all invalidation requests in a separate thread.
This eliminates a deadlock that could previously occur during checkout
operations. The invalidation requests would block until they could acquire the
kernel's inode lock on the inode in question. However, the inode lock may
already be held by another thread attempting to perform an unlink() or rename()
call. These FUSE unlink or rename operations would be blocked waiting on
Eden's mount point rename lock, which was acquired by the checkout operation.
Checkout operations now let the invalidations complete asynchronously, but we
wait for all invalidation operations to complete before indicating to our
caller that the checkout has succeeded.
Reviewed By: chadaustin, wez
Differential Revision: D7404971
fbshipit-source-id: 6fa20c00d054e210eb0258d247d083010557f210
Summary:
The status_deadlock_test code has a small helper function used to create
directory trees in the test. This moves that function into a helper module so
we can re-use it in other tests in the future.
Reviewed By: chadaustin
Differential Revision: D7407492
fbshipit-source-id: 257e5a2ce7543bb6cd218b412d165f0fac852970
Summary:
Decouple inode number assignment from materialization status.
The idea is that we will always assign entries an inode number and
track whether an entry is materialized otherwise. This is necessary
to give consistent inode values across remounts.
Reviewed By: simpkins
Differential Revision: D7052470
fbshipit-source-id: 80d3f2a2938463198a3132182537e6223c79d509
Summary:
Add a new integration test that performs a graceful restart after invoking the
getScmStatusBetweenRevisions() thrift call.
Prior to D7341609 this would cause edenfs to crash on shutdown with the error
"!!BUG!! After InodeMap::shutdown() finished, 2 inodes still loaded; they must
all (except the root) have been unloaded for this to succeed!"
Before D7341609 the `EdenMount::diffRevisions()` created a new temporary inode
tree solely to perform a diff. This resulted in multiple root inodes that all
pointed to the same EdenMount, but the EdenMount didn't know about any of these
alternate root inodes. These temporary inode trees never got destroyed,
causing this error on shutdown.
Reviewed By: chadaustin, wez
Differential Revision: D7333005
fbshipit-source-id: 8406d2e2ceb00264050b0aceec583baae2da69ec
Summary:
Update the getScmStatusBetweenRevisions() thrift call to use the new
diffCommits() function that diffs source control Tree objects without creating
TreeInode objects to perform the diff.
This addresses two bugs:
- Each call to EdenMount::diffRevisions() constructed a new root inode
associated with the mount, and this would never get destroyed. It was not
destroyed at the end of the diffRevisions() call since inodes are normally
not destroyed immediately when they are unreferenced. It was not destroyed
during EdenMount::shutdown() since EdenMount didn't have any references to
these additional root inode structures and their children.
- EdenMount::diffRevisions() incorrectly swapped ADDED and REMOVED statuses in
the result.
Reviewed By: wez
Differential Revision: D7341609
fbshipit-source-id: 16e755a0ff685f51c977c3b27d6af96908f33494
Summary:
Update Eden's thrift service handler code to accept BinaryHash arguments either
as 20-byte binary values or as 40-byte hexadecimal values.
This will make it easier to transition APIs like getScmStatusBetweenRevisions()
to use 20-byte binary hash arguments without breaking existing clients.
Reviewed By: wez
Differential Revision: D7341607
fbshipit-source-id: 3e952211900d3ec4b9c2073cf3afd55ae7e253ea
Summary:
Add an integration test for the getScmStatusBetweenRevisions() thrift call.
This call apparently gets the ADDED and REMOVED states backwards. For now the
test checks for the current (incorrect) behavior.
This also fixes the thrift definition for this function to stop using the
BinaryHash typedef. Unlike most of our other thrift functions this method
appears to require the arguments as 40-byte hexadecimal strings.
Reviewed By: wez
Differential Revision: D7341606
fbshipit-source-id: 73cbd0ecf4445da6b1f0ef9cf6d9dce47e6fb593
Summary:
This makes gitrepo.commit() return the new commit ID as a hexadecimal string,
just like hgrepo.commit() currently does.
Reviewed By: wez
Differential Revision: D7341605
fbshipit-source-id: 83ebddb8c23d5e4650432bea6f8dcb8d18c0ff38
Summary:
Update the hgrepo and gitrepo helper classes with a new `remove_file()` method,
and also improve the git code to include the process stderr in the exception
message if the command fails.
Reviewed By: wez
Differential Revision: D7341610
fbshipit-source-id: 28cca89520923d92bba4833a4dcfab6d21357cfb
Summary:
Update repobase.py, hgrepo.py, and gitrepo.py to include python type hints for
all functions and to pass most of mypy's strict type checking.
There are still a couple minor errors around the fact that hgrepo.hg() and
gitrepo.git() return an `Optional[str]` rather than a plain `str`. Many callers
know that they cannot return None unless the stdout argument is None, but mypy
can't figure this out. In the long run maybe we should split these into two
separate methods, one that always returns `str` and one that always returns
`None`.
Reviewed By: wez
Differential Revision: D7341608
fbshipit-source-id: c62da578fb32edb9272363fadabbdc11b1d5d2c2
Summary: Fix a simple TODO in EdenServiceHandler.cpp
Reviewed By: chadaustin
Differential Revision: D7329635
fbshipit-source-id: b42b2cc13be3ad5b18a629ed15f6c51cea52fbda
Summary:
If you run `eden daemon` on a machine where sudo needs input, sudo's
output would get redirected to edenfs.log and eden daemon would appear
to hang. Worse, if you pressed ctrl-C, sudo would remain in the
background and continue to swallow keypresses from the tty, appearing
to somewhat break your shell until it gave up. The fix is to stop
redirecting stdout and stderr from Python and instead have edenfs
redirect itself into the log if given.
Reviewed By: simpkins
Differential Revision: D7223399
fbshipit-source-id: bae17b150b6594a3dd87bb88741c8dcefd1c5213
Summary:
This integration test verifies that observed timestamps should persist
across runs. D6891479 makes it pass.
Reviewed By: wez
Differential Revision: D6930208
fbshipit-source-id: b8c95bce00933b9ae0de101a0bd8b6abfbfa1177
Summary:
The getpath_unloaded_inode tests failed on my machine quite
regularly. The two possible races here I can imagine are racing the
system clock and FUSE not having released its refcount on an inode by
the time unload is called.
Reviewed By: wez
Differential Revision: D7118883
fbshipit-source-id: c3708f14a860f5ad04ddec988fc67a683b7dcfde
Summary:
Previously we returned an ENOENT error in response to a FUSE lookup() call for
a name that does not exist. However, this does not allow FUSE to cache the
result, so we will continue to receive lookup() calls for this path in the
future.
This changes EdenDispatcher to return a successful response with an inode
number of 0 instead. This tells the kernel that the name does not exist, but
allows the kernel to cache this negative lookup result (for as long as
specified in the entry_valid field in the response).
Reviewed By: wez
Differential Revision: D7076811
fbshipit-source-id: a2b9977e58d6b6eecb584699b9d93b5ad29ad5ad
Summary:
Add a test that exercises `hg pull`. This confirms that eden can see new
commits created on the server after Eden and its hg_import_helper processes
have started. This test gets run in flatmanifest, hybrid treemanifest, and
treeonly mode.
This currently performs pulls using a local peer repository rather than over
SSH. This does exercise a different code path in mercurial than what typically
occurs in production. In the future we should perhaps also add a test that
uses a fake SSH helper program to exercise mercurial's sshpeer code paths as
well.
Reviewed By: chadaustin
Differential Revision: D6993788
fbshipit-source-id: 40628c0b3faac0dc8622b605a29b084979b8c089
Summary:
Whenever we tell Eden to change the working directory
parents, we need to make sure the appropriate objects are written to
disk. This fixes hg fold in Eden.
Reviewed By: simpkins
Differential Revision: D7045299
fbshipit-source-id: cbd51be59cf943a843b77c2abe66a84b745bce22
Summary:
Update the integration tests to avoid specifying an explicit path to the eden
extension. This way they use the version that we now package into hg.par
during the build.
This avoids issues with hg not being able to find and load native .so libraries
from the eden extension. Mercurial is able to find these libraries correctly
when they are packaged into hg.par (since the par start-up script sets
LD_LIBRARY_PATH to point to the par unpack directory). When using eden from an
external directory mercurial was not able to find these libraries.
Reviewed By: chadaustin
Differential Revision: D7047245
fbshipit-source-id: d56bffa953c178949c866efec507298a1f40da8b
Summary:
This removes the TARGETS files from the eden github repository. The
open source buck build has been failing for several months, since buck
removed support for the thrift_library() rule.
I will potentially take a stab at adding CMake build support for Eden
at some point in the future.
Reviewed By: chadaustin
Differential Revision: D6893233
fbshipit-source-id: e6023094a807cf481ac49998c6f21b213be6c288
Summary:
Add a command line flag to control whether or not Eden should ever try falling
back to import tree data using flatmanifest if an error occurs trying to import
it directly via treemanifest, in repositories that support treemanifest.
This is particularly useful for tests, where we usually do not want to fall
back to flatmanifest import if an error occurs during treemanifest import. The
fallback can otherwise mask real issues that should trigger test failures.
This is probably also a good thing to have in general. Supporting
flatmanifest+treemanifest data in a single Eden repository has some unfortunate
problems today: we compute hashes differently for flatmanifest trees vs
treemanifest trees. As a result, we can end up with identical trees that have
different hashes. This can result in unfortunate performance consequences in
some cases where Eden assumes it must scan a directory for differences if the
hashes are different.
I have left flatmanifest import enabled by default for now, but we may want to
disable it by default in the future. I would be more inclined to disable it by
default if we did added a thrift method to explicitly re-enable it (or to
import a single commit using flatmanifest), so that users could work around
this setting if necessary without having to fully restart edenfs.
Reviewed By: wez
Differential Revision: D6993791
fbshipit-source-id: 6e091a426cf1e7c973df5a641d2f8a1101011346
Summary:
Update the integration tests to build `hg_import_helper` into a python archive
that includes the current mercurial sources from the local repository. This
way hg_import_helper will use the local mercurial code rather than whatever
mercurial modules are installed on the system. This will help ensure that we
detect any breakages caused by changes in the mercurial source when the
mercurial changes are made rather than when they are deployed.
Reviewed By: wez
Differential Revision: D6993790
fbshipit-source-id: f3ad404583cadcf07156bac1ce6bc869bd1160e1
Summary:
Update the `hg_test` decorator to accept additional parameters specifying the
list of test modes. e.g., `hg_test('Treemanifest')` asks to only run the test
in the Treemanifest configuration. With no arguments tests are still run with
both the Flatmanifest and Treemanifest configurations by default.
This also enables the TreeOnly configuration mode, which appears to work now.
(It was previously disabled since `hg init` would fail in treeonly
repositories.)
This new changes allows tests to explicitly opt-in to running in `TreeOnly`
mode.
Reviewed By: wez
Differential Revision: D6993789
fbshipit-source-id: 9ee51318d0f661038fe29f246b2b14eebbb1c3d9
Summary:
Update the integration tests so that they do not delete the temporary test
directories if the environment variable `EDEN_TEST_NO_CLEANUP` is set.
This makes it easier to manually examine the repository state after a test
fails.
Reviewed By: chadaustin
Differential Revision: D6986217
fbshipit-source-id: 727321c2c3da4d19d9edf8ed20b2aca3449779de
Summary:
Update the logic for how the Eden integration tests find the hg binary:
- Use the contents of the EDEN_HG_BINARY environment variable if set. When
running tests via `buck test` buck will pass the hg.par output location in
this variable.
- If EDEN_HG_BINARY is not set, use libfb.py.pathutils to find the location of
the //scm/hg:hg rule output. This makes sure the integration tests still
prefer this par path even when run manually without EDEN_HG_BINARY set. This
is convenient when running individual tests not through buck.
If for some reason the hg python_binary() output cannot be found then we still
search through $PATH for hg.real or hg as usual. For internal fbsource builds
we generally shouldn't hit this fallback case, though.
Reviewed By: wez, quark-zju
Differential Revision: D6986221
fbshipit-source-id: 982cb99112405a674dbc45df4ada73a990536489
Summary:
We would immediately satisfy the health check and
tell the user that the system is healthy and show the pid of
the prior incarnation rather than the one for the instance
that we just launched.
This diff refactors the health checking code so that we can
share the implementation between the cli and the integration
tests; the integration tests already had code to do the right
thing for this.
Reviewed By: simpkins
Differential Revision: D6944989
fbshipit-source-id: 7c0f02c875b1b81f8f1b7521add67928200b27ed
Summary:
chadaustin is going to think about how to test this
in unit tests intestead :-p
Reviewed By: chadaustin
Differential Revision: D6951788
fbshipit-source-id: 137f285f3a1f080ce43392a621c73640ce3a9bf7
Summary:
This makes the default `memory` for speed and minimal
flakiness, but allows a test to select a different engine where
appropriate (eg: restart and remount tests).
Reviewed By: chadaustin
Differential Revision: D6944207
fbshipit-source-id: 1fb11387beda02d059a796dad5a42d56ddcf6e88
Summary:
Adds a SQLite storage implementation and makes it the
default engine for integration tests; this requires fewer resources
to run and the integration tests thus run faster and more reliably.
In the future we may add a configuration option to remember the
storage engine that was used as it is currently not "safe" to switch
from one to the other because the hgproxyhash data cannot be
recreated without re-importing a revision.
Reviewed By: simpkins
Differential Revision: D6919456
fbshipit-source-id: 3afbfafb190cca0e3c797cd9b7cd051768575a8c
Summary:
To me, restart implied shutting down the daemon and restarting it
again. Perhaps instead of `eden daemon --takeover` we should have
`eden restart`. But if people typed `eden restart` I imagine they're
trying to debug a problem, so that's probably not the right verb.
Reviewed By: wez
Differential Revision: D6929166
fbshipit-source-id: d568a1940d67f755e4c3656098c58fc81e0a3156
Summary: This should make it less flaky
Reviewed By: simpkins
Differential Revision: D6908454
fbshipit-source-id: 07622b469ec635972290c3a4e5d1ed4b05ddd51a
Summary:
This was causing flakiness in the form of premature timeouts
generating failure status. Looking at the docs for hypothesis, this
option is deprecated and slated to be replaced with an infinite timeout
in the future:
http://hypothesis.readthedocs.io/en/latest/settings.html#hypothesis.settings.timeout
Those docs suggest using `hypothesis.unlimited` instead, but I couldn't import
that symbol (maybe our version is too old?) so I'm just using 0 which means
that no timeout will be applied.
Reviewed By: simpkins
Differential Revision: D6908200
fbshipit-source-id: 1143d4745fc84d9c14b29ff5a2906a4e776d14fe
Summary:
A little bit of plumbing to enable the watchman<->eden integration tests on sandcastle.
I noticed that we were still referencing the lame thrift par that I removed a little while back,
so let's clean that up.
Reviewed By: simpkins
Differential Revision: D6863221
fbshipit-source-id: 026f3d316d9ac52fe359cedc4e3c88394ec5c7c4
Summary:
This is a fairly minimal attempt at asking sandcastle to
run our tests as root. This is inspired by D6608746 but is for
the moment pretty coarse; I just want to see how successful this
is before finishing this off nicely, which means splitting our
contbuild config into two pieces along similar lines as D6608746;
one piece would run the tests as usual without root privileges
whilst the other would run just the integration tests with root
privs.
Reviewed By: chadaustin
Differential Revision: D6612669
fbshipit-source-id: 9a730b02424e939dd3509310b6265d50d1102e77
Summary: Rather than EPERM we get EACCESS when running as root
Reviewed By: chadaustin
Differential Revision: D6853303
fbshipit-source-id: c6743c8d036a24255d2d31c560034c972a0253c7
Summary:
in our sandcastle environment, eden is not currently
installed and the tests that look at the version are throwing
an error when the rpm query fails.
Don't do that!
Reviewed By: simpkins
Differential Revision: D6853305
fbshipit-source-id: d17cb1fb0253ef5ae5000695b6ec5bcda4a6a448
Summary:
This is the spiritual successor to D3302706 which originally
wanted to solve this by adding a python extension. That would prove
to be too painful for the opensource build so it was shelved.
We now need to be able to run our tests in an environment that doesn't
have the `attr` rpm installed so this is a good time to fix this
in a more portable way.
This diff adds a little wrapper around the functions that we already
have for consuming extended attribute information and augments them
with another to list attributes.
The utility emits output in json format and is intended to be fed
directly into the helper functions we have in `fs.py`.
Reviewed By: chadaustin
Differential Revision: D6851182
fbshipit-source-id: 3d1d1a351f2e01405645d45658d1c8bc61a659a4
Summary:
isSameAs calls getSha1 which was failing on symlinks. The
original concern was that asking for the SHA-1 of a symlink is
ambiguous: do you want the hash of the symlink or the target? But we
already check for whether you are requesting the SHA-1 of a symlink in
EdenServiceHandler, so it's redundant and incorrect to check in
FileInode too.
Reviewed By: simpkins
Differential Revision: D6847489
fbshipit-source-id: 13966da06bcde75c5c568e09fef14e735de47cfb
Summary:
This adds support for the following flags for `hg status` in the Rust wrapper:
* `--all`
* `--modified`
* `--added`
* `--removed`
* `--deleted`
* `--clean`
* `--unknown`
* `--ignored`
* `--no-status`
* `--copies`
There is now a `create_print_config()` function that takes the parsed arguments
and distills them into a `PrintConfig`, which describes the net configuration
as a result of all the flags. (Note that some flags can interfere with each other,
e.g., `--no-status` forces `--copies` to be disabled.)
While here, I cleaned up a bunch of duplication in the code that prints the
status to stdout. Much of the duplication in the old implementation has been
encapsulated in a closure within the new `print_status()` function.
This also made it easier to implement `--copies` for all of the different status
types.
Incidentally, this helps us specify `listIgnored` correctly when calling the
`getScmStatus()` Thrift method as we now keep track of whether the user
specified `--ignored`. This is particularly important for the Eden integration
test suite because its `assert_status()` helper method calls
`hg status --print0 -mardui` by default, so `buck test eden/...` makes much
greater use of the Rust wrapper than it did before.
Reviewed By: wez
Differential Revision: D6770916
fbshipit-source-id: fea1a159addca6d16caa1b7019dc11dc2dcb3e1b
Summary: Add type annotations to the test functions in this file.
Reviewed By: chadaustin
Differential Revision: D6789863
fbshipit-source-id: d298a388b5ea03fcf25047937e747051c8fa682b
Summary:
Test running `hg commit` with a path argument. This currently fails if the
path refers to a directory rather than a file.
Reviewed By: chadaustin
Differential Revision: D6789862
fbshipit-source-id: 054ee26ea0fdb843cc12d5e8ac3c3d34b9308017
Summary: removed check for target dir of eden clone in edenclient
Reviewed By: simpkins
Differential Revision: D6770615
fbshipit-source-id: 4f480c2a494673179d92c0571082b9be895cf51b
Summary:
Update the `hg update --merge` to test marking the change resolved and
committing the result.
Reviewed By: wez
Differential Revision: D6757299
fbshipit-source-id: c147e6ff8b63e364f96d2ce8a2094dd809d8da7d
Summary:
Fix `dirstate.normallookup()` to behave like `dirstate.normal()`, since our
`dirstate.status()` function does not handle "unsure" files that need to be
checked on disk.
This was previously causing problems after running `hg commit -i`.
We ideally should probably have some additional changes here to avoid making an
filesystem `lstat()` call even in the `dirstate.normal()`, but that can wait
until later.
Reviewed By: bolinfest
Differential Revision: D6746814
fbshipit-source-id: c1623415fc3a923b0b7878be651db3ad8fbb2aff
Summary:
The storage format for dirstate data in Eden changed substantially in D6179950.
Although `status.rs` was modified as part of that change, it was only updated to
make it so the rust wrapper still compiled, but it did not maintain the
correctness of the overall function.
This revision is a port of the logic in `eden/py/dirstate.py` and
`eden/hg/eden/EdenThriftClient.py` so that the Python code in the Eden/Hg
extension behaves the same as that of the Rust wrapper.
The current implementation of `hg status` in Rust does not relativize paths to `$PWD`, so
it always behaves as if `--root-relative` was specified. This should be fixed in a
follow-up revision.
Reviewed By: jsgf
Differential Revision: D6717211
fbshipit-source-id: ca900e251d392a77c07cb7305c4d4a62def1c7ab
Summary: Added integration test validating umask is properly set on file creation by touch.
Reviewed By: wez
Differential Revision: D6749557
fbshipit-source-id: adb89ec996148ede95bdb23399745b39504ae8db
Summary:
Rename the `stdout_charset` parameter to `encoding` now that we also use this
for encoding the stdin contents.
Reviewed By: wez
Differential Revision: D6757300
fbshipit-source-id: f79fc760e0f9fdcd6af559fcdad34976aaf16412
Summary:
Update the `hg split` tests to pass input to the split command using
the `input` argument to `subprocess.run()` rather than running the command
through the shell and using a shell "here document".
Reviewed By: bolinfest, chadaustin
Differential Revision: D6746815
fbshipit-source-id: b607c554a249f702758dcad2ac4d863d1c1c7bad
Summary:
Added --version to main.py, including -v and version.
It prints both installed and running version (as per Wez's suggestion), and accounts for the possibilities that eden may not be running at all, or dev version may be running.
Reviewed By: wez
Differential Revision: D6724204
fbshipit-source-id: 5085f53a00a557f759a23fe41fb57189c9ad6a7e
Summary:
Changed `eden clone` to check out master commit in both git and hg.
Previously, it checked out the current commit for the repo.
Reviewed By: simpkins
Differential Revision: D6663754
fbshipit-source-id: 92b185ccca5d082dc2bde9c8b191c82a2a4f06b4
Summary:
This implements a TODO/FATAL that is important for
graceful restarts to be useful in my "acid test" scenario,
which is to perform a graceful restart while buck build is
running.
Reviewed By: simpkins
Differential Revision: D6700189
fbshipit-source-id: dec1b818ebc9e907841bc127ee08c953b59d6487
Summary:
Fix lines longer than 80 characters, and also reduce the number of times the
initial directory listing is repeated throughout this file.
Reviewed By: wez
Differential Revision: D6710372
fbshipit-source-id: bdb02cbebabeff7d7c6c88aebee7ebab1865535b
Summary:
Update the statfs() code to return non-zero values for namelen and frsize.
Returning 0 for namelen was causing problems for programs that checked
`pathconf(path, _PC_NAME_MAX)` and tried to honor this value. For instance,
GNU patch would try to generate files with a 0-length name since we indicated
the maximum name length was 0.
I haven't investigated too closely, but this behavior might have broken only
recently when we stopped using libfuse. libfuse may have been setting this to
255 for us. I didn't see code in libfuse that would do this, but I'm fairly
sure GNU patch was working correctly very recently.
Reviewed By: wez
Differential Revision: D6710370
fbshipit-source-id: fc9a0320cd7c1eb2545219a3ec123c2f0644fb5d
Summary:
I'm so-so on a bit of the implementation here, but it works!
I had to change the `takeoverPromise` from the `pair<fuseDevice, connInfo>`
to a new helper struct because we now have three distinct pieces of data
to pass out of EdenMount to build up the overall TakeoverData.
The key change in this diff is that we have to release all of the file handles
we're maintaining in the `FileHandleMap` prior to shutting down the `InodeMap`,
otherwise the `InodeMap` will never complete (it waits for all inodes to be
unreferenced, and that cannot happen while there are open file handles). I've
made the `FileHandleMap` serialization and clearing contingent on performing a
takeover shutdown because that feels like the safest thing to do wrt. not
losing any pending writes.
Reviewed By: simpkins
Differential Revision: D6672437
fbshipit-source-id: 7b1f0f8e7ff09dbed850c7737383ecdf1e5ff0c7
Summary:
This is the key portion that makes the graceful restart
function. This diff connects almost all of the moving pieces together;
it informs the priv helper about the takeover mount and transfers
the InodeMap information into the new generation of the eden server.
It doesn't yet load the fileHandleMap (will tackle that in a follow up diff)
Reviewed By: simpkins
Differential Revision: D6670903
fbshipit-source-id: 1770d99eb1477440a6c1deed83b0da55b9c1bbe4
Summary:
The unix `patch` command attempts to issue an `FS_IOC_GETFLAGS`
ioctl and blows up when we return `ENOSYS`. The man page for ioctl says:
```
ENOTTY The specified request does not apply to the kind of object that
the file descriptor fd references.
```
so let's return that error code to ioctl.
In addition, the integration test I added for this trips up when it calls
`llistxattr` on the file; turns out we don't need to insert the
`fuse_getxattr_out` when we're returning the attribute list and that it
is only needed when measuring up the required length, so let's move
things around to resolve this and make it clearer.
Reviewed By: chadaustin
Differential Revision: D6685568
fbshipit-source-id: 81963ffe9af30db5634e5e96b7a8aa1485859d65
Summary:
Update the integration tests to understand the new `eden list` output format
changes introduced in D6661303.
Reviewed By: chadaustin
Differential Revision: D6665272
fbshipit-source-id: 13f65a7a66f997e6b8135436811fe3fc396eb38f
Summary:
We're unconditionally emitting a `--` above, so the `--`
down here doesn't get stripped out by the logic in `do_daemon`
in the CLI and that renders it invisible to gflags:
Here's our `eden cli` invocation:
```
['/data/users/wez/eden-fbsource/fbcode/buck-out/gen/eden/cli/cli.par', '--config-dir', '/var/tmp/eden_test.15oico01/homedir/local/.eden', '--etc-eden-dir', '/var/tmp/eden_test.15oico01/etc-ed
en', '--home-dir', '/var/tmp/eden_test.15oico01/homedir', 'daemon', '--daemon-binary', '/data/users/wez/eden-fbsource/fbcode/buck-out/gen/eden/fs/service/edenfs', '--foreground', '--', '--num
_hg_import_threads', '2', '--', '--logging=eden.fs.store.hg=DBG9,eden.strace=DBG7']
```
This is what `eden cli` passes to `edenfs`:
```
edenfs_args ['--num_hg_import_threads', '2', '--', '--logging=eden.fs.store.hg=DBG9,eden.strace=DBG7']
```
Reviewed By: bolinfest
Differential Revision: D6628253
fbshipit-source-id: 2c6806e69baff52d14ca64194f1bf7d916833844
Summary:
This is a codemod to change from using @/ to // in basic cases.
- TARGETS files with lines starting with @/ (but excluding @/third-party:
- autodeps lines in source and TARGETS files ( (dep|manual)=@/ ), excluding @/third-party
- Targets in string macros
The only thing left of the old format should be @/third-party:foo:bar
drop-conflicts
Reviewed By: ttsugriy
Differential Revision: D6605465
fbshipit-source-id: ae50de2e1edb3f97c0b839d4021f38d77b7ab64c
Summary:
Encountered a funny situation where running `hg clone src dest` from inside an
Eden mount where `src` was a directory that contained a non-Eden Hg repo would
fail with a stacktrace that ended with:
```
File "/usr/local/fb-mercurial/eden/hgext3rd/eden/__init__.py", line 195, in merge_update
conflicts = repo.dirstate.eden_client.checkout(
AttributeError: 'dirstate' object has no attribute 'eden_client'
```
This was very confusing because we had this check at the top of the function:
```
if not util.safehasattr(repo.dirstate, 'eden_client'):
why_not_eden = 'This is not an eden repository.'
```
So it seemed that we already verified that `repo.dirstate.eden_client` must be a
valid attribute. However, we followed this check with a new set of checks, the
final one being:
```
else:
why_not_eden = None
```
This one had the unintended effect of resetting the value of `why_not_eden` that
we set in the first `if`. Changing the `if` to an `elif` introduces the proper
decision tree.
Reviewed By: simpkins
Differential Revision: D6608867
fbshipit-source-id: 320e69925737135d84f9d6a46a7fb43437cc37e0
Summary:
My devvm was running out of memory when running `buck test eden/...`.
Because Buck farms out tests across cores and each Eden integration
test launches 8 hg import daemons, there were over a hundred importer
processes live on the system.
Reviewed By: bolinfest, wez
Differential Revision: D6598276
fbshipit-source-id: d26916af79c24aa73abfa4c3ef9be3178657b6e7
Summary:
This attempts to repro an internal bug report, though note the integration test
passes without any changes to Eden. It is possible that the person who reported
the bug was stuck on an old version of the Eden daemon while using a newer
version of the Mercurial extension, which could account for the unexpected
behavior.
Reviewed By: simpkins
Differential Revision: D6536375
fbshipit-source-id: 1bc4c50ee5f616502dc06f8ed0167817c566e179
Summary:
I am seeing occasional timeouts on my smaller devvm when
running `buck test eden/integration/...`.
Reviewed By: bolinfest
Differential Revision: D6541864
fbshipit-source-id: 401deb8b44adae8cc362bbba8b638fe08abb9b1e
Summary:
In D6446057, I added a new entry to the dict returned by
`config.get_client_info()`. I only ran the cli tests while working on D6446057,
but I should have ran all of the tests because there was an integration test
(`InfoTest`) that verified the return value of this method, so it broke due to
my change.
(Note: this ignores all push blocking failures!)
Reviewed By: simpkins
Differential Revision: D6464806
fbshipit-source-id: 1b0ac0853301ba33e5e948353e4c89c0d97c0d83
Summary:
Make sure `hg update --clean` clears the merge state data. In non-clean
updates this is performed in `mercurial.merge.applyupdates()`. However, we
never call `applyupdates()` on clean updates in eden.
Reviewed By: bolinfest
Differential Revision: D6456720
fbshipit-source-id: b40d02ca0fb677bcde82822a8eafd5fcf926dae6
Summary:
Previously, we used the Mercurial code `g` when faced with an `UNTRACKED_ADDED`
file conflict, but that was allowing merges to silently succeed that should not
have. This revision changes our logic to use the code `m` for merge, which
unearthed that we were not honoring the user's `update.check` setting properly.
Because we use `update.check=noconflict` internally at Facebook, we changed the
Eden integration tests to default to verifying Hg running with this setting. To
support it properly, we had to port this code from `update.py` in Mercurial to
our own `_determine_actions_for_conflicts()` function:
```
if updatecheck == 'noconflict':
for f, (m, args, msg) in actionbyfile.iteritems():
if m not in ('g', 'k', 'e', 'r', 'pr'):
msg = _("conflicting changes")
hint = _("commit or update --clean to discard changes")
raise error.Abort(msg, hint=hint)
```
However, this introduced an interesting issue where the `checkOutRevision()`
Thrift call from Hg would update the `SNAPSHOT` file on the server, but
`.hg/dirstate` would not get updated with the new parents until the update
completed on the client. With the new call to `raise error.Abort` on the client,
we could get in a state where the `SNAPSHOT` file had the hash of the commit
assuming the update succeeded, but `.hg/dirstate` reflected the reality where it
failed.
To that end, we changed `checkOutRevision()` to take a new parameter,
`checkoutMode`, which can take on one of three values: `NORMAL`, `DRY_RUN`, and
`FORCE`. Now if the user tries to do an ordinary `hg update` with
`update.check=noconflict`, we first do a `DRY_RUN` and examine the potential
conflicts. Only if the conflicts should not block the update do we proceed with
a call to `checkOutRevision()` in `NORMAL` mode.
To make this work, we had to make a number of changes to `CheckoutAction`,
`CheckoutContext`, `EdenMount`, and `TreeInode` to keep track of the
`checkoutMode` and ensure that no changes are made to the working copy when a
`DRY_RUN` is in effect.
One minor issue (for which there is a `TODO`) is that a `DRY_RUN` will not
report any `DIRECTORY_NOT_EMPTY` conflicts that may exist. As `TreeInode` is
implemented today, it is a bit messy to report this type of conflict without
modifying the working copy along the way.
Finally, any `UNTRACKED_ADDED` conflict should cause an update to
abort to match the behavior in stock Mercurial if the user has the following
config setting:
```
[commands]
update.check = noconflict
```
Though the original name for this setting was:
```
[experimental]
updatecheck = noconflict
```
Although I am on Mercurial 4.4.1, the `update.check` setting does not seem to
take effect when I run the integration tests, but the `updatecheck` setting
does, so for now, I set both in `hg_extension_test_base.py` with a `TODO` to
remove `updatecheck` once I can get `update.check` to do its job.
Reviewed By: simpkins
Differential Revision: D6366007
fbshipit-source-id: bb3ecb1270e77d59d7d9e7baa36ada61971bbc49
Summary:
I accidentally broke the flatmanifest fallback code in D6333613 by changing the
exception type thrown for errors received from hg_import_helper.py but not
updating the catch statement in HgImporter::importTreeImpl().
This updates importTreeImpl() to catch the new HgImportPyError type correctly.
I have dropped the check on the error message entirely, since the mercurial
python code can throw a variety of errors that all mean this tree data isn't
available.
Reviewed By: bolinfest
Differential Revision: D6434359
fbshipit-source-id: c62d3c1667681712293873de2b9bf6d9220da767
Summary: Annotate more integration test functions with type information.
Reviewed By: bolinfest
Differential Revision: D6434358
fbshipit-source-id: b88351eebee58561465752378c6771b7b1f9554e
Summary:
There's a bug in some combination of Eden and FUSE where open(O_TRUNC)
followed by a sequence of writes over an existing file does not flush
the kernel's VFS page cache, which manifests as an mmap larger than
the file's size not zeroing the data beyond the file's size. These
tests attempt capture that use case, but they are fiddly.
Disabling ATOMIC_O_TRUNC seems to resolve the issue.
Reviewed By: wez
Differential Revision: D6430152
fbshipit-source-id: f7626e268e778ebab60c66322e0ce42bce746ae1
Summary:
In onboarding users, we usually tell them to run `eden clone fbsource`,
but that fails because users generally have not run `eden daemon` yet.
The simplest thing is to do it for them when they run `eden clone` when
the daemon is not running.
Reviewed By: wez
Differential Revision: D6357249
fbshipit-source-id: dc112c1efe214485e3c5c8e06522d299a100d3a0
Summary:
We now run two versions of this test: one where the file that exists in the
destination commit is untracked before the update and one in which it is added
before the update.
Reviewed By: simpkins
Differential Revision: D6334002
fbshipit-source-id: ef6bffa27bc18171b5e21dc284c7a21aa6e35da4
Summary: Addresses outstanding TODO now that D6322052 has landed.
Reviewed By: simpkins
Differential Revision: D6368884
fbshipit-source-id: 497c42466e05af0f1690bc6401b1d271de691e58
Summary:
This begins implementing the "client-side" portion of graceful takeover in
edenfs. When the --takeover flag is specified, if edenfs finds that another
edenfs process is already running it will attempt to gracefully take over its
mount points rather than exiting with an error.
This does not yet actually take over the mount points themselves--it still
sends dummy mount information during shutdown, and does not use this data
during startup. However, we do perform takeover of the eden lock file and the
thrift server socket.
Reviewed By: bolinfest
Differential Revision: D6038944
fbshipit-source-id: 42406a0559367cec79af088b4ca84c22de3f3ef3
Summary:
When cloning an existing Eden mount, we should be smart and inherit its
underlying config so that we inherit properties such as its bind mounts.
Reviewed By: wez
Differential Revision: D6322002
fbshipit-source-id: 3f5ba135b12ad7dcecef6676d27495cfbf0ce97b
Summary:
Previously, a user had to define a config for a repo in a file like `~/.edenrc`
in order to create a new Eden mount via `eden clone`. In practice, the
information that is hardcoded in the config can generally be inferred from an
existing repo, so this expands `eden clone` to support both modes of operation.
Note this made it possible to finally unify the `RepoConfig` and `ClientConfig`
types. This revision removes `RepoConfig`, so I dutifully renamed every
local variable named `repo_config` to `client_config`.
Reviewed By: wez
Differential Revision: D6314115
fbshipit-source-id: 9625a5fbe35b30f76b6099180580c64435a4cf72
Summary:
The headline changes of this revision are:
- Changes the format of the config file from INI to TOML
(the `edenrc` file under `~/local/.eden` has been replaced
with `config.toml`). This revision includes logic for automatically
performing the migration when Eden is restarted.
- Inlines data from `/etc/eden/config.d` into the TOML file.
Historically, the `edenrc` file for a client would contain the
name of the "configuration alias" defined in a config file like
`~/.edenrc` or `/etc/eden/config.d/00-defaults`. When Eden
loaded a client, it would have to first read the `edenrc` and
then reconstitute the rest of the client configuration by
looking up the alias in the set of config files that were used to
create the client in the first place.
This changes things so that all of the data that was being
cross-referenced is now inlined in the client's config file.
This makes loading a config considerably simpler at the cost
of no longer being able to change the config for multiple clients
that were cloned from the same configuration alias in one place.
It was questionable whether being able to modify a client from
a foreign config after it was created was a safe thing to do, anyway.
Eliminating the need for a historic link to the configuration alias
will make it easier to support running `eden clone` on an arbitrary
local Hg or Git repo. So long as `eden clone` can extract enough
information from the local repo to create an appropriate config file
for the new Eden client, there is no need for a configuration alias
to exist a priori.
Since we were already changing the data in the config file, this
seemed like an appropriate time to make the switch from INI to
TOML, as this was something we wanted to do, anyway.
In testing, I discovered a discrepancy between how boost's
`boost::property_tree::ptree` and Python's `ConfigParser` handled
the following section heading:
```
[repository ZtmpZsillyZeden-clone.LIkh32]
```
Apparently `hasSection("repository ZtmpZsillyZeden-clone.LIkh32")`
in boost would fail to find this section. Because
[[https://stackoverflow.com/questions/13109506/are-hyphens-allowed-in-section-definitions-in-ini-files | there is no spec for INI]],
it is not that surprising that boost and `ConfigParser` do not 100% agree
on what they accept. Moving to TOML means we have a configuration
language with the following desirable properties:
- It has a formal spec, unlike INI. This is important because there are parsers
in a wide range of programming languages that, in theory, accept a consistent
input language.
- It is reasonable for humans to write, as it supports comments, unlike JSON.
- It supports nested structures, like maps and arrays, without going crazy
on the input language it supports, unlike YAML.
Eden now depends on the following third-party TOML parsers:
* C++ https://github.com/skystrife/cpptoml
* Python https://github.com/uiri/toml
This revision also changes the organization of `~/local/.eden` slightly. For now,
there is still a `config.json` file, but the values are no longer hashes of the realpath
of the mount. Instead, we take the basename of the realpath and use that as the
name of the directory under `~/local/.eden/clients`. If there is a naming collision, we
add the first available integral suffix. Using the basename makes it easier to
navigate the `~/local/.eden/clients` directory.
Although the `edenrc` file under `~/local/.eden/clients` has been switched from INI
to TOML, the other Eden config files (`~/.edenrc` and `/etc/eden/config.d/*`) still use
INI. Migrating those to TOML will be done in a future revision.
Note this revision allowed us to eliminate `facebook::eden::InterpolatedPropertyTree`
as well as a number of uses of boost due to the elimination of
`ClientConfig::loadConfigData()` in the C++ code. Because `ClientConfig`
no longer does interpolation, a bit of `ClientConfigTest` was deleted as part of
this revision because it is no longer relevant.
Reviewed By: wez
Differential Revision: D6310325
fbshipit-source-id: 2548149c064cdf8e78a3b3ce6fe667ff70f94f84
Summary:
This flips the sense of the unmount command; previously
we would default to destroying the mount and associated state,
but this was a bit of a massive sharp edge to our UX.
Now the default is non-destructive and you have to explicitly
pass in `--destroy` to enable destructive mode.
Reviewed By: chadaustin
Differential Revision: D6346013
fbshipit-source-id: ce612e7d8a70540d63217a97f96bc5760f3951af
Summary:
Users often run `hg update --clean .` to get out of a bad state, but this was
not clearing the "added" or "removed" state in the dirstate as it should in
Eden.
Reviewed By: wez
Differential Revision: D6331858
fbshipit-source-id: 616f187930587a1af40a1f151e3a424d50dd8da3
Summary:
Override dirstate.rebuild() so that it drops all file changes without trying to
mark every file in the commit manifest as normal. We don't want to track
normal files in the eden dirstate.
Reviewed By: bolinfest
Differential Revision: D6322227
fbshipit-source-id: d81ade1cdafb5fa03c642239b0cff91308c7fc35
Summary:
Previously calling `hg debugdirstate` inside an eden repository crashed, since
it would try to iterate over the dirstate and we do not allow this.
This re-implements the `debugdirstate' command inside eden repositories to
print out the data stored in eden's dirstate file.
Reviewed By: wez
Differential Revision: D6322052
fbshipit-source-id: 92f230438a545fe83134b0d67545ebb89877d2f0
Summary:
In the course of verifying a fix for `hg update --merge` in D6270272, I
discovered a new bug in our merge logic in the Python code. As expained in the
test plan, there was a case where a file was listed as "untracked" instead of
"added" after a merge with `--tool :local`.
I traced through what happens in stock Mercurial. After the call to
`applyupdates()` in `update()` in `merge.py`, there is this code:
```
stats = applyupdates(repo, actions, wc, p2, overwrite, labels=labels)
wc.flushall()
if not partial:
with repo.dirstate.parentchange():
repo.setparents(fp1, fp2)
recordupdates(repo, actions, branchmerge)
# update completed, clear state
util.unlink(repo.vfs.join('updatestate'))
if not branchmerge:
repo.dirstate.setbranch(p2.branch())
```
It turns out that `applyupdates()` can have the side-effect of adding new
entries to the `actions` dict. In this case, we have a `'cd'` action for which
an `'am'` action is generated. Our `merge_update()` function in
`eden/hg/eden/__init__.py` did not have the `recordupdates()` call that the
stock implementation of Mercurial does, so the `'am'` (for "add/merge") was not
getting applied.
It seems likely that introducing this `recordupdates()` call may fix other
subtle bugs in Eden's Mercurial extension for which we do not yet have
integration tests.
Reviewed By: wez
Differential Revision: D6279971
fbshipit-source-id: 901c1bc563a7a3910dde18cf2f0d8b8ff9cd6fbe
Summary:
The underlying issue is that we were reporting a `MODIFIED_REMOVED`
conflict as a `MODIFIED_MODIFIED` conflict. This put us in a state where
Mercurial expected to find a file in the new manifest, but failed because the
file was not present in that revision, so no such file could be found.
Somewhat surprisingly, the appropriate handler for a `MODIFIED_REMOVED`
conflict already existed in our Mercurial extension, but there was no logic on
the server that would generate a `MODIFIED_REMOVED` conflict previous to
this change.
Like D6204916, this was an issue I ran into when trying to create a repro case
for the issue that was fixed in D6199215.
Reviewed By: wez
Differential Revision: D6270272
fbshipit-source-id: 6604eea00b0794cd44b01d2ba6b9ea10db32d556
Summary:
If a file is explicitly added via `hg add`, then it should be considered added
even if it matches a pattern in `.gitignore`. Further, if it is deleted without
running `hg forget`, it should be considered missing rather than ignored.
To make this work, I had to update `eden_dirstate.walk()`, which already
had a special case when used as part of `hg add`. The new logic ensures
that files that are specified explicitly are still considered even if they are
matched via `.gitignore`.
I also had to address a TODO in `EdenThriftClient.py` related to the
handling of ignored files that was introduced as part of the major
`eden_dirstate` changes in D6179950. It was expected that it would
be easier to handle ignored files properly after D6179950 landed.
Reviewed By: wez
Differential Revision: D6242223
fbshipit-source-id: cf1cfe97a8d2ec57bce1d524074c43978a78e4ef
Summary:
Python 3 type checking currently complains about most of our integration
testing since the tests use an `hg_test` decorator to inherit from the base
test class. This prevents the type checker from being able to figure out this
inheritance.
This updates all of the test cases to explicitly derive from the test case base
class, rather than using the decorator to do so. I also renamed the base test
case class to `EdenHgTestCase` to be slightly more succinct and to follow the
common pattern of calling `unittest.TestCase` subclasses `FooTestCase`
Reviewed By: bolinfest
Differential Revision: D6268258
fbshipit-source-id: 09eef2f8217932a6516f78d17dddcd35c83b73da
Summary:
This adds an integration test that exercises a deadlock we could encounter in
the past. An "hg status" operation could trigger many trees and files to be
imported. Unfortunately the file import code currently blocks waiting for file
import futures to complete. This could result in a state where all threads in
the pool were waiting for a file import to complete, and the file import was
waiting for a free thread to complete.
Reviewed By: bolinfest
Differential Revision: D6216871
fbshipit-source-id: e1795a543a71fccbed035febb159e126e27d1950
Summary:
This fixes an issue where the `DIRECTORY_NOT_EMPTY` conflict type reported by
the server was not handled by the client. Somewhat ironically, the fix appears
to be to explicitly "do nothing," though the important part of this revision is
the new integration test.
As this is only one test, I'm not convinced this covers all possible corner
cases, but it's certainly better than blowing up, which is what we did before.
Reviewed By: wez
Differential Revision: D6264069
fbshipit-source-id: a7c45a43776a903a4d6b6cdfb0ce75db9549c380
Summary:
A few fixes:
* Fix a bug where `date_str` would not get set when `date` was specified.
* Remove `from __future__ import` stuff since this code is Python 3.
* Add type annotations to the `commit()` method.
Reviewed By: simpkins
Differential Revision: D6261874
fbshipit-source-id: 5f942d01c107cd0265c2d6ec6e1f46295bb3ec24
Summary:
If you have an untracked file and you `hg update` to a commit that has
that file in the tracked state, then the contents of the untracked version
should be ignored, as they are replaced with the contents of the file in the
commit you are updating to. The untracked version should be backed up
as specified by `ui.origbackuppath`.
Previously, our code in `eden/hg/eden/__init__.py` mapped this to a merge action
named `c`, but we did not include that in our set of `actions`, so we were
getting a `KeyError` if you exercised this code path.
I discovered this while trying to reproduce the issue that I fixed in D6199215.
Reviewed By: simpkins
Differential Revision: D6204916
fbshipit-source-id: b70153428291bda9a8853a37c0955ad7cb3bd89d
Summary:
Since Eden's integration tests do not run in CI yet, this adds a
test that verifies the Eden CLI can start without Python errors.
Reviewed By: wez
Differential Revision: D6250515
fbshipit-source-id: 907bffaff122c9929a7623d97f665de5b2a6f2d3
Summary:
This is a major change to how we manage the dirstate in Eden's Hg extension.
Previously, the dirstate information was stored under `$EDEN_CONFIG_DIR`,
which is Eden's private storage. Any time the Mercurial extension wanted to
read or write the dirstate, it had to make a Thrift request to Eden to do so on
its behalf. The upside is that Eden could answer dirstate-related questions
independently of the Python code.
This was sufficiently different than how Mercurial's default dirstate worked
that our subclass, `eden_dirstate`, had to override quite a bit of behavior.
Failing to manage the `.hg/dirstate` file in a way similar to the way Mercurial
does has exposed some "unofficial contracts" that Mercurial has. For example,
tools like Nuclide rely on changes to the `.hg/dirstate` file as a heuristic to
determine when to invalidate its internal caches for Mercurial data.
Today, Mercurial has a well-factored `dirstatemap` abstraction that is primarily
responsible for the transactions with the dirstate's data. With this split, we can
focus on putting most of our customizations in our `eden_dirstate_map` subclass
while our `eden_dirstate` class has to override fewer methods. Because the
data is managed through the `.hg/dirstate` file, transaction logic in Mercurial that
relies on renaming/copying that file will work out-of-the-box. This change
also reduces the number of Thrift calls the Mercurial extension has to make
for operations like `hg status` or `hg add`.
In this revision, we introduce our own binary format for the `.hg/dirstate` file.
The logic to read and write this file is in `eden/py/dirstate.py`. After the first
40 bytes, which are used for the parent hashes, the next four bytes are
reserved for a version number for the file format so we can manage file format
changes going forward.
Admittedly one downside of this change is that it is a breaking change.
Ideally, users should commit all of their local changes in their existing mounts,
shutdown Eden, delete the old mounts, restart Eden, and re-clone.
In the end, this change deletes a number of Mercurial-specific code and Thrift
APIs from Eden. This is a better separation of concerns that makes Eden more
SCM-agnostic. For example, this change removes `Dirstate.cpp` and
`DirstatePersistance.cpp`, replacing them with the much simpler and more
general `Differ.cpp`. The Mercurial-specific logic from `Dirstate.cpp` that turned
a diff into an `hg status` now lives in the Mercurial extension in
`EdenThriftClient.getStatus()`, which is much more appropriate.
Note that this reverts the changes that were recently introduced in D6116105:
we now need to intercept `localrepo.localrepository.dirstate` once again.
Reviewed By: simpkins
Differential Revision: D6179950
fbshipit-source-id: 5b78904909b669c9cc606e2fe1fd118ef6eaab95
Summary:
There is logic in `eden_dirstate.walk()` that looks to see if any of the files
that are reported as "removed" by `hg status` are still on disk, and if so,
should be considered for a walk. Because the files are likely removed, we were
catching `ENOENT` for a failed `os.stat()`, but we also needed to be catching
`ENOTDIR`. This turned out to be the reason `hg add` was failing in a specific
case, for which we already had an integration test, but it wasn't passing until
now.
Reviewed By: simpkins
Differential Revision: D6207233
fbshipit-source-id: 44e5252bb0130ca279160f0a64286053fa5509d5
Summary:
Relying on the toString of an Exception in Python seems a little gross,
especially when the `stderr` field is available directly. Cleaned up two
instances of this so it doesn't get copypasta'd further.
Reviewed By: simpkins
Differential Revision: D6195633
fbshipit-source-id: 9ae77796c287a454cb169ebf6de2953909a1e6c3
Summary:
This was an error that an end-user ran into. Previously, we did not fail
gracefully and the user was faced with an intimidating stacktrace.
Reviewed By: simpkins
Differential Revision: D6195529
fbshipit-source-id: bde3c2a3e6f49457a4c6ac5c87103cf52cd227c2
Summary:
I ran into this issue while manually testing Eden.
Currently, this integration test fails, so it is tagged with `unittest.skip`.
There are substantial changes to our distate logic coming in D6179950, so I
will attempt to make the test pass as part of that revision.
Reviewed By: simpkins
Differential Revision: D6199789
fbshipit-source-id: cd7ce48b72bf0b54e13547b23823f4d496fa5b0b
Summary:
Upstream, some new merge actions were added:
* `p` https://phab.mercurial-scm.org/D776
* `pr` https://phab.mercurial-scm.org/D777
We must include entries for these in the list of `actions` that we build up in
`eden/hg/eden/__init__.py` because the `actions` dict gets passed through to
Mercurial's own `applyupdates()` function in `merge.py` that contains this line:
```
for f, args, msg in actions['p']:
```
Therefore, without an entry for `p` in `actions` here, we get a `KeyError`.
Reviewed By: markbt
Differential Revision: D6199215
fbshipit-source-id: a7408e5ef84a659f37e7771a7c15f6a4b14ae0f9
Summary:
In practice, if the `hg graft` succeeds in a weird way, `assert_status_empty()`
tells a lot more about what went wrong than the number of commits not matching up.
While here, I also added the following entry to the default `.hgrc` used in integration tests:
```
[ui]
origbackuppath=.hg/origbackups
```
I needed this for the change to `graft_test.py`. As we were already setting this option in
the `histedit_command.py` utility as a one-off and this is the default value of this setting
for our internal Mercurial use at Facebook, it seemed best to make it the default for all
of our integration tests. As such, I removed the one-off setting in `histedit_command.py`.
Reviewed By: simpkins
Differential Revision: D6180342
fbshipit-source-id: 6f0487624a1824459403126997ea52d1a7921feb
Summary:
Previously we flushed the pending transaction data in
eden_dirstate.setparents(). However, some dirstate code paths (particularly
dirstate.rebuild()) can directly call eden_dirstate_map.setparents().
We need to make sure the transaction data is flushed in this case.
Reviewed By: bolinfest
Differential Revision: D6175410
fbshipit-source-id: 256cb07f57ada02d6c1f118ec5075fb8ac93506c
Summary:
Previous to this change, the user got an inscrutable error message. It turns out
that it is easy to make this mistake, typing `eden clone fbsource/` instead of
`eden clone fbsource` if you accidentally use tab completion.
Reviewed By: simpkins
Differential Revision: D6153889
fbshipit-source-id: 3642fdd207d6abf896d6a12891d5eb68ad984acc
Summary:
Previously, the `savebackup()` and `restorebackup()` methods in `eden_dirstate`
only retained the parent commit hashes. With this change, now the dirstate tuples
and entries in the copymap for the dirstate are also included as part of the saved
state.
Failing to restore all of the state caused issues when doing things like aborting
an `hg split`, as observed by one of our users. Although this fix works, we ultimately
plan to move the responsibility for persisting dirstate data out of Eden and into the
Hg extension. Then the data will live in `.hg/dirstate` like it would for the default
dirstate implementation.
Reviewed By: simpkins
Differential Revision: D6145420
fbshipit-source-id: baa077dee73847a47cc171cd980cdd272b3a3a99
Summary:
Add an `eden debug hg_dirstate` command to dump the contents of the Hg dirstate.
This data is stored in a binary format, so we need a custom command to view it
easily.
Reviewed By: simpkins
Differential Revision: D6139172
fbshipit-source-id: 622c0b7bcaa471a88483c6c4ddef7e0be95a3dfa
Summary:
When performing an source control checkout operation, we attempt to remove
directories that are empty after the checkout. However, this code path was
missing a call to flush the kernel cache for these directories.
As a result, even though eden thought the directory not longer existed, and
would not report it in `readdir()` results, the kernel would return stale
information from its cache when explicitly accessing this path.
Reviewed By: bolinfest
Differential Revision: D6151543
fbshipit-source-id: 6031feb268ff6f980c885efb26c3d43243dec3f4
Summary:
This ports some logic from Mercurial's `dirstate.py` to our `eden_dirstate.py`.
It was known that Eden was missing this behavior (D5686636), but we did not have
an integration test that demonstrated the importance of this logic until now.
Admittedly, the current implementation ports the logic verbatim from
`dirstate.py`, though this will yield quite a number of Thrift calls in the Eden
implementation. We will address this in a subsequent revision.
Reviewed By: simpkins
Differential Revision: D6046273
fbshipit-source-id: f7a27ba6dca36cddac898f19637f29f3bc79a0cb
Summary:
Running Mercurial's own integration tests revealed that we had a bug here:
https://www.mercurial-scm.org/repo/hg/file/tip/tests/test-histedit-arguments.t
Somewhat unsurprisingly, it was time to finally address a longstanding `TODO`
in `Dirstate.cpp`. The issue was that, after running `hg merge --tool :local`,
`hg status` was not including a merged file in the list of modified files. Because
the information from `hg status` is used to create a commit context, that meant
that when a commit was made after running `hg merge`, the commit did not
include the merged file in the list of files for the commit, which differs from
Mercurial's behavior.
Most of the implementation of `hg status` on the Eden side is done by
`EdenMount.diff()`. However, in this case, `diff()` does not categorize the
merged file by invoking one of the methods of `InodeDiffCallback` because
as far as `EdenMount` is concerned, the file has not changed because `EdenMount`
is unaware of the `Dirstate`. We already have some existing cases where we have
to do some post-processing on the result of `EdenMount.diff()` using information
in the `Dirstate` (e.g., files that are marked for addition or removal), so the fix was
to add a check for the case when the file is flagged as "needs merging" and
then including it as modified in the `hg status` output, as appropriate.
Reviewed By: wez
Differential Revision: D6005603
fbshipit-source-id: 7d4dd80e1a2e9f4b98243da80989e0e9119a566d
Summary:
Note that the original motivation for this test was to verify
`savebackup()` and `restorebackup()` in `eden_dirstate`: D5485950.
As singhsrb recently updated Mercurial to remove a redundant commit when doing
`hg amend` in upstream Mercurial (https://phab.mercurial-scm.org/D636), I
suspect that is responsible for the change in behavior that is necessitates the
change in our test.
We now use a precommit hook failure to trigger the rollback rather than an editor
with a non-zero exit code. As you can see, `transaction abort!\nrollback completed\n`
still appears in the error message, so we are still verifying the behavior of interest.
Differential Revision: D5826751
fbshipit-source-id: bcbf00042c3f26b6e9aa1a980060a0561725a56c
