Commit Graph

590 Commits

Author SHA1 Message Date
Shannon Zhu
096b35c70b Convert type check targets in eden to use configuration
Summary:
Migrating buck integration to use configurations, which is more performant and is the primary supported way to run Pyre.
Mypy is no longer supported at Facebook, so mypy buck integration is covered in this migration as well.

Configuration target automatically expanded to include all subtargets, expanding type coverage while introducing no more than 2 fixmes per file.

Reviewed By: dkgi

Differential Revision: D21474723

fbshipit-source-id: 2501af8250b4998fe331350b80c9dc8b4ba2dd1a
2020-05-08 15:24:57 -07:00
Chad Austin
c4db2f39a9 remove some internal uses of the term 'client'
Summary:
Our use of the term "client" to refer to a checkout is
deprecated. Rename some internal functions that use the term client.

Reviewed By: simpkins

Differential Revision: D21395159

fbshipit-source-id: fa96ba593f53b493e5ae816fa686f333a132c232
2020-05-07 22:08:10 -07:00
Adam Simpkins
13a8f17325 de-blacklist most of the hg status related tests on Windows
Summary: All of the tests in hg/status_test.py pass except for these 3.

Reviewed By: genevievehelsel

Differential Revision: D21410570

fbshipit-source-id: cf9a87ce0f025860c1e95927a780757fda6801ad
2020-05-07 16:41:30 -07:00
Adam Simpkins
ccdbc2279b update most of the integration/basic_test.py tests to work on Windows
Summary:
- If we fail to create symlinks ignore this error on Windows
- Using a file as a directory produces an ENOENT error instead of ENOTDIR
  on Windows.
- Deal with some issues around CR -> CRLF translation when writing files in
  non-binary mode on Windows.
- Move several POSIX-specific tests out into a separate test class.

Reviewed By: wez

Differential Revision: D21214598

fbshipit-source-id: 587a755bd06990bf88d26017254a69ff8869edbc
2020-05-07 16:41:30 -07:00
Adam Simpkins
54af4a5770 update can_run_eden() to check for ProjectedFSLib.dll on Windows
Summary:
On Windows we can only run EdenFS if ProjectedFSLib.dll is available.  It
turns out that some Windows systems have ProjectedFSLib.lib available, so we
can build `edenfs.exe`, but are missing ProjectedFSLib.dll, so we cannot
actually run it.

Reviewed By: wez

Differential Revision: D21443255

fbshipit-source-id: 6396fce8bd71adc0695468a2df3fd5f584909138
2020-05-07 16:41:30 -07:00
Shannon Zhu
83dc7b9129 Custom pyre upgrade for eden
Summary:
Addressing issues simpkins brought up on D21207287 when we upgraded and introduced some pyre bugs.

Temporarily upgrading just this project, once we resolve some sandcastle capacity issues we'll release this via another global upgrade in fbcode.

Reviewed By: simpkins

Differential Revision: D21316793

fbshipit-source-id: f0c79f53d97f7182e7d8fe6e081c58ef53ce0c9a
2020-05-04 21:19:20 -07:00
Adam Simpkins
c0fb9cea2d enable all integration test sources in CMake builds
Summary:
Previously we only included `basic_test.py` and `hg/status_test.py` in the
integration tests during CMake-based builds.  This updates the code to now
include all of the test files, with just a few exclusions based on platform
type and what dependencies were available at build time.

Reviewed By: wez

Differential Revision: D21239912

fbshipit-source-id: b8826d249a6323ac3bcc555c9ceba54a4cbcfde9
2020-05-04 11:46:09 -07:00
Adam Simpkins
a755d9e979 blacklist some integration tests on non-RedHat Linux distributions
Summary:
10 of the integration tests fail on Ubuntu.  For now simply blacklist them
from running on non-RedHat-based distributions.

Reviewed By: wez

Differential Revision: D21332629

fbshipit-source-id: 3fadb74bb31e89092177afaa01ddc7f6bcd0f9de
2020-05-04 11:46:09 -07:00
Adam Simpkins
ea7460f2d3 apply the integration test case blacklist to some more tests
Summary:
Add a new IntegrationTestCase base class that checks the test blacklist during
`setUp()`, and update a few remaining test classes that did not derive from
`EdenTestCase` to derive from this.

Also drop the method name argument from the `skip_test_if_blacklisted()`
function, since we can get this from the existing test case argument.

Reviewed By: genevievehelsel

Differential Revision: D21340539

fbshipit-source-id: d4fc125f119d74ab923c2cc3c9070b86c582c87e
2020-05-04 11:46:09 -07:00
Adam Simpkins
f390948707 fix the UserInfoTest on Ubuntu
Summary:
Run `env` as `/usr/bin/env` instead of `/bin/env`
This command is typically installed in `/usr/bin`.  Recent RedHat releases
replaced `/bin/ with a symlink to `/usr/bin`, allowing this command to work
when invoked as `/bin/env`, but this isn't true on all distributions, such as
Ubuntu.

Reviewed By: chadaustin

Differential Revision: D21340540

fbshipit-source-id: d7588f8c90e9a86a0cb31fd3ab3a9067aa6e79ea
2020-05-04 11:46:08 -07:00
Shannon Zhu
0b93429dc1 Convert type check targets in eden to use configuration
Summary:
D21316793 is blocked from landing because there are a few targets in eden that are running pyre via buck targets integration.

We can't do custom version overrides for projects that are using a mix of local configurations and buck integration, because buck doesn't provide an interface for setting the equivalent pyre version override.

We're moving away from buck targets integration for pyre across the board, and I've run a codemod over the project to clean up all of the buck typing integration (including some residual mypy) as well as updated type ignores / fixmes accordingly.

Let me know if you have any concerns; upon skimming it looks like most changes are either converting `type: ignore`s into fixmes, or removing `type: ignores`.

Reviewed By: dkgi

Differential Revision: D21343093

fbshipit-source-id: 5ee1436377eb526c0a679fb821c42e07cbca52a5
2020-05-02 21:16:55 -07:00
Chad Austin
ec00e91309 disallow setting suid, sgid, and sticky bits
Summary:
FUSE_HANDLE_KILLPRIV expects that any write() call is handled by
clearing the setuid and setgid bits in the userspace. To avoid
implementing that behavior, disallow setting setuid or setgid in the
first place.

Reviewed By: xavierd

Differential Revision: D21333703

fbshipit-source-id: eb084ee8b00afe74c0da26e41c32c2cb742723da
2020-04-30 15:18:35 -07:00
Adam Simpkins
8392001e38 update the systemd-related integration tests to work on Ubuntu
Summary:
The systemd binary is installed in a different location on Debian-based
distributions (e.g., Ubuntu) from RedHat-based distributions.  Update the
integration tests to look in both places.

Reviewed By: wez

Differential Revision: D21297591

fbshipit-source-id: 67e5a698080205b8624c1f38aa49e75c20b6a28c
2020-04-29 23:12:37 -07:00
Adam Simpkins
0400e99d09 update the integration tests to guess at CMAKE_SOURCE_DIR
Summary:
Our CMakeLists.txt file requests `ctest` to pass in a `CMAKE_SOURCE_DIR`
environment variable when running our integration tests via `ctest`.  However
if a developer manually invokes the test binary this environment variable may
not be set.

For convenience, this updates the code to try looking at the default location
where getdeps will have put the source relative to the build directory (at
least for internal builds using shipit-converted fbsource sources).

Reviewed By: wez

Differential Revision: D21307807

fbshipit-source-id: d747c50d79d2d378721b68b18c256d5363e41f26
2020-04-29 18:50:34 -07:00
Adam Simpkins
ea59a68521 improve finding the system hgrc during CMake-based integration tests
Summary:
Update the integration tests to attempt to find the Mercurial config files
from the source directory even in CMake-based builds.  This directory will be
available when doing an internal build from fbsource via getdeps.py

If this directory isn't present (e.g., on a build from the github repository)
we will still fall back to using the Mercurial configuration files installed
on the system.  In that case I did update the code to use the correct system
configuration path on Windows.

Reviewed By: wez

Differential Revision: D21307808

fbshipit-source-id: 917ba5e8548fed999fb06a29324be125cec83cac
2020-04-29 18:50:34 -07:00
Adam Simpkins
1c3eb54e21 remove the requirement that edenfs must be invoked with --edenfs
Summary:
Remove the requirement that the `--edenfs` argument flag must be passed in
when invoking the `edenfs` daemon.

This flag was added in D12927803 to help provide a better error message if
users accidentally ran `edenfs` when they really mean to run `edenfsctl`.
However, this shouldn't really be a major problem any more: on Linux and Mac
we now install `edenfs` in a `libexec` subdirectory instead of `bin`, so it
should not be in most user's `$PATH` by default any more.  Additionally we
also verify that no other unexpected arguments have been supplied.  Prior to
D12927803 the `edenfs` binary would ignore unexpected arguments, making the
error messages more confusing when users tried to run it with `edenfsctl`
subcommand arguments.

One motivation for changing this now is that the Windows version of `edenfs`
does not require this flag, and it is desirable to remove this behavior
discrepancy.  Rather than making this flag required on Windows it seems fine
to just drop the requirement on Mac & Linux.

Reviewed By: wez

Differential Revision: D21297159

fbshipit-source-id: e24bd694dadc036cd31dead287ae2c1184747822
2020-04-29 17:55:34 -07:00
Adam Simpkins
f48674b1df update the service test cases to skip when appropriate
Summary:
Update the tests that derive from `ServiceTestCaseBase` to honor the test
blacklist, and to also skip if we cannot run the `fake_edenfs` utility.  This
latter check primarily fails when we cannot run `sudo` non-interactively
without a password prompt.

Most of the other tests were already honoring blacklist through the
`EdenTestCase` base class, but `ServiceTestCaseBase` does not derive from
`EdenTestCase`.

Reviewed By: wez

Differential Revision: D21297160

fbshipit-source-id: 5044e9939bbe487c09aa96021166c95f02fb376e
2020-04-29 17:55:34 -07:00
Adam Simpkins
c55781c666 move UserInfo to eden/fs/utils/
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.

This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper.  However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.

Reviewed By: wez

Differential Revision: D21296594

fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
2020-04-29 17:21:12 -07:00
Katie Mancini
48804631f7 expose number pending fuse requests
Summary:
This sets up the counters that will allow us to expose the number
of pending FUSE requests in Eden top.

As D20846826 mentions adding metrics for FUSE request gives
visibility into fuse requests and overall health of eden.

This provides more insight beyond the metrics for live FUSE requests
since it shows the kernels view of FUSE requests. Looking at the difference
between the number of pending and live request, can identify issues
that arise at the interface between eden and FUSE and monitor how
quickly fuse workers are processing requests.

**note**: this is only for linux since macos has no equivalent to
`/sys/fs/fuse/connections`

Reviewed By: chadaustin

Differential Revision: D21074489

fbshipit-source-id: c0951f0dfd4fa764be28d8686d08cd0dd807db37
2020-04-28 13:28:01 -07:00
Genevieve Helsel
cfb64a0a2d clean up EdenMount destruction process
Summary:
In the past, if an EdenMount got stuck while shutting down (inodes still referenced), the EdenMount would already be removed from the server's `mounPoint's` map while stuck. This resulted in `eden list` showing that the mount was not mounted, and then if we tried to remount the mount, we'd get a confusing error message about the not being able to grab the overlay lock. This also made it so if we tried to attach with gdb, or if we core dumped and tried to look at the data, we wouldn't be able to examine the object.

Instead, we should only remove the mount from the `mounPoints_` map after it has completely been shutdown. This will allow us to get better signal on the state of the mount (it will be in the `SHUTTING_DOWN` state in `eden list` instead of `(not mounted)`) and it will allow us to better debug these situations (where a mount is stuck shutting down) since we can get more information about the mount that is in this state.

Reviewed By: simpkins

Differential Revision: D21211544

fbshipit-source-id: 97ff11df67d0a80f47cf6b9f563876e17bd76622
2020-04-28 12:44:12 -07:00
Adam Simpkins
e193085594 some ctypes-related portability improvements in the integration tests
Summary:
Update `integration/lib/linux.py` to use `ctypes` to call `statfs`, instead of
running `/bin/stat` and parsing the output.  This allows this test code to
work even on systems that do not have `/bin/stat` (e.g., on Ubuntu this is
installed at `/usr/bin/stat`).

Also update the usage of ctypes in the `mmap.py` test file to allow this file
to be imported successfully on Mac OS X.

Reviewed By: xavierd

Differential Revision: D21273385

fbshipit-source-id: c2173758f8ea67cde2a68e57827acc5f73b63179
2020-04-28 12:11:51 -07:00
Adam Simpkins
5470190bf3 update one of the rebase tests to explicitly use non-in-memory update
Summary:
Update the `test_rebase_commit_with_independent_folder()` test to explicitly
disable in-memory updates.  The in-memory update code ends up changing the
number of update operations performed (it skips the initial and final update),
causing one of the assertions in this test to fail.

Reviewed By: xavierd

Differential Revision: D21271902

fbshipit-source-id: 3e40adce5fac1c851c8f277d2209d48a19deed55
2020-04-28 12:11:51 -07:00
Adam Simpkins
c13f0ac5f7 update the usage of pexpect to work on Windows
Summary:
Update the integration test code that uses pexpect to use the more modern
pexpect APIs.  The top-level `pexpect.spawn()` function is considered legacy
now, and is only provided for backwards compatibility on Unix platforms.

Reviewed By: wez

Differential Revision: D21214640

fbshipit-source-id: 941da5435c4f8afaf22e8053f4c344175d7b1a7f
2020-04-24 14:48:05 -07:00
Adam Simpkins
cadcd0ab39 enable building the integration helper tools in the CMake build
Summary:
Enable building fake_edenfs.cpp and TakeoverTool.cpp in the CMake build.

This includes a few changes to get `fake_edenfs.cpp` closer to building on
Windows, but at the moment it still does not fully build since the
`StartupLogger` is currently not compiled on Windows.

TakeoverTool.cpp is specific to the graceful restart tests, and doesn't make
sense to ever build on Windows.

Reviewed By: wez

Differential Revision: D21214614

fbshipit-source-id: a7a8306ea3f69579fa02bd3753a66d3095ea004a
2020-04-24 14:48:05 -07:00
Adam Simpkins
cb5b9fefd7 tweaks to update the integration tests to allow running on Windows
Summary:
Previously none of the integration tests ran on Windows, as they checked for
the presence of `/dev/fuse` first, and would skip the test if it was not
present.

This updates the tests to allow running on Windows, and updates the
`find_executables.py` code to find the executables properly on Windows.

Reviewed By: wez

Differential Revision: D17733621

fbshipit-source-id: a8bf4e5138f9ea6c9b040814dd68fc64228961de
2020-04-24 14:14:15 -07:00
Adam Simpkins
6500f94127 avoid running systemd-related tests when systemd is not supported
Summary:
systemd is Linux-specific, so it does not make sense to try and run these
tests on Linux and Mac.  Additionally, the pystemd module currently used by
the edenfsctl CLI is not currently open source, and therefore only works in
our internal Linux builds.

This updates the integration test code to only include systemd-related tests
if we are actually in an environment where we support systemd.

Reviewed By: wez

Differential Revision: D21084095

fbshipit-source-id: fb6c9a066d9dcb741bddf3cdcf9decdb763a1d05
2020-04-23 15:06:32 -07:00
Shannon Zhu
6ec7546cab Update pyre version for eden
Summary: Automatic upgrade to remove `version` override and silence errors.

Reviewed By: grievejia

Differential Revision: D21207287

fbshipit-source-id: ab82cf81690cb847bd130ff3448345f5a0ea19af
2020-04-23 12:09:39 -07:00
Adam Simpkins
1e8e8b05de clean up parts of the start and clone integration tests
Summary:
Several of the tests started edenfs of fake_edenfs processes and didn't kill
them at the end of the test.  This adds proper cleanup functions to try and
kill these processes when the test exits.

This also eliminates using pexpect in these tests, since it isn't actually
necessary here.

Reviewed By: wez

Differential Revision: D21084096

fbshipit-source-id: 4e92a99a5c398d4a78830ac51507ba34d7a6c0b1
2020-04-22 15:02:42 -07:00
Adam Simpkins
c763e922a2 kill off some of the integration test mixin classes
Summary:
This updates the various systemd-related tests to use EdenTestCaseBase,
and allows us to delete the EnvironmentVariableMixin and
SystemdUserServiceManagerMixin classes.

The main goal of this clean-up is to make it easier to consolidate most of the
systemd-related code into just a few locations, so that we can more easily
disable the systemd-related tests in build environments where systemd is not
supported.

Reviewed By: wez

Differential Revision: D21084098

fbshipit-source-id: d5e05254c689c28751fe48d2dc38d722c7e77ed3
2020-04-22 15:02:42 -07:00
Adam Simpkins
72f6ada7c2 add a new EdenTestCaseBase class
Summary:
This adds a new `EdenTestCaseBase` class to serve as the base class across a
number of our integration tests and some of our CLI tests.

The main goal of this is to allow eliminating many of the annoying `*Mixin`
classes used in a lot of our integration tests.  These mixin classes are
annoying since they result in complicated multiple inheritance, and it can be
tricky to ensure that the method resolution order behaves the way you want.
The systemd tests in particular use a lot of mixins, which gets complicated.
These mixin classes are also awkward from a Python typing perspective, and the
systemd tests end up resorting to just declaring different APIs in several
places when `typing.TYPE_CHECKING` is True.

The fact that `EdenTestCaseBase` has a `contextlib.ExitStack` member variable
should make it easier for us to eliminate these mixins moving forward: rather
than using mixins that use inheritance and assume a `self.cleanUp()` method
exists, we can transition this code to standalone functions or context
managers, and they can take the `ExitStack` variable as an argument if
necessary.

Reviewed By: wez

Differential Revision: D21084097

fbshipit-source-id: 77ee457b7debe6f584f630e3e30f79fe634a2026
2020-04-22 15:02:41 -07:00
Adam Simpkins
62a463ea76 only run the git integration tests when built with git support
Summary:
Update the integration tests to only run git integration tests if EdenFS was
built with git support.

Reviewed By: genevievehelsel

Differential Revision: D21000162

fbshipit-source-id: 0a015ae93a9420f040be702096e0b378fe138e4c
2020-04-22 12:48:49 -07:00
Adam Simpkins
5917c8e097 address some pyre-fixme issues in find_executables.py
Summary:
Update the code to work around the fact that Pyre refuses to unwrap Optional
member variables.  Store the data in local variables so that Pyre will allow
unwrapping these objects after we check against None.

Reviewed By: genevievehelsel

Differential Revision: D21000165

fbshipit-source-id: 59e8430de30ec35213a53b06670b16c34be85593
2020-04-22 12:48:48 -07:00
Adam Simpkins
ba949fa2cf update some of the integration test code to use the eden.config module
Summary:
Update some of the eden integration tests to use the `eden.config` module
instead of trying to import the `__manifest__` module, which is only present
in Buck-based builds.

Reviewed By: wez

Differential Revision: D21000167

fbshipit-source-id: 347082a32e842752e5539a8e6c8addde97a16eee
2020-04-22 12:48:48 -07:00
Adam Simpkins
a26b8e9930 update edenfsctl to report the compile-time version
Summary:
Update most locations in edenfsctl to report the version number that was built
into the edenfsctl binary at build time, rather than querying the RPM database
for the installed RPM version.  The RPM behavior only works on to RedHat-based
Linux systems, and the currently running process doesn't necessarily have to
have come from the RPM.

The one place where we do still attempt to print the RPM version is in the
`edenfsctl rage` report, when running on Linux.

Reviewed By: wez

Differential Revision: D21000168

fbshipit-source-id: 0fb747e71b6950d74f22c458efa0dfcbd45270bd
2020-04-22 12:48:48 -07:00
Katie Mancini
d850667e19 expose live FUSE requests
Summary:
This exposes metrics for the live FUSE requests (the duration
of the longest outstanding request and the number of outstanding
requests).

Because FUSE is the interface through which the user mostly interacts
with the file system they provide good metrics to judge if the perfomance
of eden is normal, or there may be an issue.

Exposing these counters this way will send them to ods, so it will not only
allow for debuging current issues, but can be used to look back at previous
problems. This data could also be used for alerting or more proactive
remediation.

Metrics are exposed per checkout to allow seeing which checkout was
having issues. This data will aggregated in `eden top` to be used as
an overall health indicator, but should more information be needed it
will be logged in ods.

Reviewed By: chadaustin

Differential Revision: D20922194

fbshipit-source-id: 16208883417acb77b62bf712cfdd9068c5420303
2020-04-22 12:33:34 -07:00
Adam Simpkins
15dd55fb80 add a framework for per-platform blacklists of integration tests
Summary:
Many of our integration tests do not yet work on Windows.  In order to allow
us to start running some integration tests on Windows while not getting lots
of failure reports, this adds a framework for blacklisting specific tests on a
per-platform basis.

Reviewed By: genevievehelsel

Differential Revision: D21084150

fbshipit-source-id: 141cd11f7e8113fcbbb2416d518864fca9179554
2020-04-21 20:37:20 -07:00
Genevieve Helsel
87bd70f0b0 restructure ThriftServer stop code in takeover case
Summary:
This restructures the `ThriftServer server_` stopping logic in the graceful restart case. Instead of stopping the server, we stop listening then explicitly stop the server. This refactor should exhibit the same behavior as today since we block on `stopListening()`, but this will allow for simpler refactoring of thrift call queueing logic in the future (by removing the `stopListening()` call and replacing it with a `startQueueingAndWaitForOutstandingCallsToFinish()` type of call).

In terms of layout, this consolidates all the stopping code into one function `startTakeoverShutdown()`. This function now returns the `TakeoverData` itself instead returning a future that is fulfilled after stopping the Thrift server. The TakeoverServer still communicates via `takeoverComplete`, but now that future is stored during `startTakeoverShutdown()` instead of being the return value of `performTakeoverShutdown()`. This also generally eliminates `TakeoverPromise`.

Reviewed By: simpkins

Differential Revision: D20744151

fbshipit-source-id: 60f0c273b4f3889b53586d79efd95bfb27256e1b
2020-04-20 12:28:36 -07:00
Adam Simpkins
80a150fb53 improve a failure message in the rebase tests
Summary:
Improve some failure messages to include the full command output that we are
checking.

Reviewed By: genevievehelsel

Differential Revision: D21084135

fbshipit-source-id: 65be3adce7bf54bab6beccf5d0c463a749d585d2
2020-04-20 12:16:53 -07:00
Adam Simpkins
94c52753d1 relax checking of C++ type formatting in an integration tests
Summary:
One of the thrift tests was checking an error string that was produced from
`folly::exceptionStr()`, which includes the exception type name.  However, in
some build environments this name does not always get demangled (the folly
demangling code may not have been included in the build, or there may be some
other issue).  For instance, in some environments using the CMake-based build
I was seeing the exception type reported as `St12domain_error` instead of
`std::domain_error`.

Reviewed By: genevievehelsel

Differential Revision: D21004345

fbshipit-source-id: 8e6a4ee4d93b1aabbe9c9816c43db803e6e12798
2020-04-15 17:58:12 -07:00
Adam Simpkins
d18ce1082e split edenfsctl start into foreground vs service logic
Summary:
This refactors the `edenfsctl start` command so that we more clearly split the
functionality into two pieces:
* Starting EdenFS as a service
* Running EdenFS in the foreground

In most normal situations in production the `start`, `restart`, and `stop`
commands are used to manage running EdenFS as a service.  In the future I
believe our service management logic will start to diverge a bit more on Linux
vs Mac vs Windows, and this should help isolate the service-management code a
bit more cleanly.

The foreground behavior is mainly only used by developers during testing and
during the integration tests.  Several options like `--gdb` and `--strace` are
only allowed in foreground mode, and this refactoring makes that clearer.  In
the future we may also want to further restrict this, to allow only
specifying additional custom arguments and a custom binary path when running
in foreground mode.  However, for now I have not updated that as I believe
some of our integration tests may be exercising this behavior today.

This change also cleans up some of the platform-specific code, and lets them
share more of the logic to construct arguments for edenfs.  With this change
`edenfsctl start --foreground` now works on Windows.

Reviewed By: pkaush

Differential Revision: D20833244

fbshipit-source-id: 0f09d59702d8b64ca8f4fedccbc30da1c858afb4
2020-04-15 15:46:25 -07:00
Adam Simpkins
772369731c implement our own temporary file management during integration tests
Summary:
Unfortunately the Python standard library's `tempfile.NamedTemporaryFile`
class does not work very well on Windows: even though the temporary files are
named, they cannot actually be opened by other processes.  Attempting to open
the file fails with a permission denied error.

This code extends our existing temporary directory management logic to also
add APIs for creating individual temporary files.  This has the advantage of
better grouping all temporary files for a given test in the same directory.  I
also updated the code to include the test function name in the temporary
directory prefix.  This should make it a little easier to identify which test
is at fault when there are temporary files left behind after a test.  (This
can happen if the test gets killed or if it leaves behind files or mount
points that cannot be removed normally).

Reviewed By: wez

Differential Revision: D20885160

fbshipit-source-id: 7267b2352e51214354eab7ead839bf166d690974
2020-04-13 13:12:04 -07:00
Adam Simpkins
c5e6e40888 remove support for repo configs from edenfsctl clone
Summary:
Update the `edenfsctl clone` command to require that an explicit repository
path be passed in.  Using the name of a repository listed in an EdenFS config
file is no longer supported.

Reviewed By: wez

Differential Revision: D20876459

fbshipit-source-id: 5e9fef11a2afa4cc48cb8a9bb5b874d2e7923f25
2020-04-10 13:57:52 -07:00
Adam Simpkins
ef8c3435c4 remove the "edenfsctl repository" subcommand
Summary:
We no longer use repository configs, so remove the `repository` subcommand
that supported adding and listing these configurations.

The main information that used to be included in the repository configuration
was the bind mount settings.  This has since been replaced with the
`.eden-redirections` file that is placed directly in each repository.

Reviewed By: wez

Differential Revision: D20876462

fbshipit-source-id: cc7d8e6f0a6a2e04fbf3159417af41a44908b3a8
2020-04-10 13:57:52 -07:00
Adam Simpkins
4910c924a8 fix some failures in the systemd integration tests on devservers
Summary:
When running the systemd integration tests on devservers, the edenfs process
will be started as root, but the test will be running as non-root.  Therefore
the test will not have permission to send SIGINT to the edenfs process to stop
it.  Update the tests to call `edenfsctl stop` instead of directly trying to
send `SIGINT` to the edenfs process.

Reviewed By: genevievehelsel

Differential Revision: D20939015

fbshipit-source-id: 5bcf0e58fdf532084fa1273254b3d81fd20de23e
2020-04-10 13:48:11 -07:00
Adam Simpkins
baeac18ef3 add a --current-edenfs-only flag to edenfsctl doctor
Summary:
Add a command line argument to `edenfsctl doctor` to cause it to only report
problems with the current EdenFS instance, and to skip checks for system-wide
problems or other running EdenFS processes.

This does refactor a fair amount of the top-level doctor logic to encapsulate
most of the state into an `EdenDoctor` class.

This also updates the integration tests to use this flag when invoking
`edenfsctl doctor`.  Previously the integration tests could spuriously fail
due to other pre-existing problems on the system, or due to other EdenFS
instances that are currently being started or shut down by other tests running
in parallel.

Reviewed By: wez

Differential Revision: D20357521

fbshipit-source-id: 36640cc21e7bd79fbd300c4d2c7dbba127ec9170
2020-04-10 13:48:11 -07:00
Adam Simpkins
34275429fe remove all CLI code dealing with the old legacy bind-mounts config
Summary:
The bind-mounts configuration has been ignored by EdenFS since D17236366.
This removes all CLI code for dealing with this config section.

Reviewed By: wez

Differential Revision: D20876460

fbshipit-source-id: 6b3f3552de25ee28fc0418a6aaec14446520203c
2020-04-09 17:31:03 -07:00
Katie Mancini
e299b71988 Explicitly track current/pending imports -- refactor and expose current metrics
Summary:
As mentioned in D20629833, adding metrics for live
imports in `eden top` gives more transparency to the
imports process and makes identifying import related
issues easier. This is set up to expose metrics for live
imports like those for pending imports in `eden top`.

Similar to D20611728 exposing this via these counters
will log this data. Having this data persisted will allow
tracking the performance of imports, and does the set
up for more pro-active fixing of issues. Further we can
look back to see issues that are no longer occurring, but
still of interest.

This also refactors the registration code so that it requires
no copy pasting to add a new counter. Avoiding copy paste
errors when adding more counters and making it easier to
maintain.

Reviewed By: chadaustin

Differential Revision: D20630813

fbshipit-source-id: 8a7a2a0135c7b7a5cde960b84dcb434c6c99eaeb
2020-04-09 12:35:22 -07:00
Genevieve Helsel
9944a5dff5 add EdenServer recovery step and recover after failed takeover data send handshake
Summary:
* This adds a `EdenServer::recover()` method to start back up on unsuccessful takeover data send.
    * On an unsuccessful ping, filfill the `shutdownPromise` with a `TakeoverSendError` continaing the constructed `TakeoverData`. After this `recover` function is called, `takeoverPromise_` is reset, `takeoverShutdown` is set to `false`, and the `runningState_` is set to `RUNNING`.
With taking over from the returned `TakeoverData`, the user will not encounter `Transport not connected` errors on recovery.

* This adds a `EdenServer::closeStorage()` method to defer closing the `backingStore_` and `localStore_` until after our ready handshake is successful.
* This defers the shutdown of the `PrivHelper` until a successful ready handshake.

I also update the takeover documentation here with the new logic (and fix some formatting issues)

Reviewed By: simpkins

Differential Revision: D20433433

fbshipit-source-id: f59e660922674d281957e80aee5049735b901a2c
2020-04-07 09:52:21 -07:00
Genevieve Helsel
003bdfe7a5 add additional takeover "ready" handshake
Summary:
For graceful restart takeovers, we would like to implement an additional handshake. This handshake will occur right after the takeover data is ready to be sent to the client, but before actually sending it. This is to make sure the old daemon can recover in case of the client not being responsive (the client replies back to the server, and if no response is recieved in 5 seconds, the server will recover).

There are a few cases here:
* **Server sends ping (two cases discussed below)**
I introduced a new ProtocolVersion. Daemons with this change will now have ProtocolVersion4. The Server checks the max version of the client, and if this version is ProtocolVersion4, we know the client can listen for pings. So we will send the ping. Otherwise, we don't send a ping. With this, we will only send pings if we know the client will be listening for one. The case in which a client isn't listening is if we adopt this change and we downgrade past the change.

* **Server does not send ping and Client knows to listen for ping**
This will be a common case immediately after this change. The client will parse the sent data and check if it matches the "ready" ping, and if it doesn't, the client assumes the server simply sent the Takeover Data.

* **Server does not sends ping and Client doesn't know to listen for ping**
This is the case before this change.

Reviewed By: simpkins

Differential Revision: D20290271

fbshipit-source-id: b68e4df6264fb071d770671a80e28c90ddb0d3f2
2020-04-07 09:52:21 -07:00
Adam Simpkins
c0d6deaae1 remove the deprecated debugSetLogLevel() thrift API
Summary:
Using `setOption("logging", config)` is the preferred way to update the
logging configuration.  This older API is no longer needed, and I don't
believe anything should be using it any more.

Reviewed By: genevievehelsel

Differential Revision: D20876083

fbshipit-source-id: c0b40c61535bee2363f273345cbce818389de6b7
2020-04-06 18:53:00 -07:00