Summary:
we've seen flaky tests here on stress test runs, lets subtract time.time() by an epsilon of one to ensure that the comparison is only made within a second precision (time.time() worst case precision)
an example error: `AssertionError: 1595414015.5022793 not greater than or equal to 1595414015.5025299`
`AssertionError: 1595759609.439305 not greater than or equal to 1595759609.441682`
Reviewed By: chadaustin
Differential Revision: D22687496
fbshipit-source-id: 8e87148d620577e3198d2845d785a87a909cd1d3
Summary:
Disregard two default systemd services on Fedora 32 that cause our
systemd tests to fail.
I believe genevievehelsel is planning on removing this code soon anyway.
Reviewed By: genevievehelsel
Differential Revision: D22713393
fbshipit-source-id: b703b23a3158cb007dc2e1cb53fae36be7282719
Summary:
We are unifying C++ APIs for accessing optional and unqualified fields:
https://fb.workplace.com/groups/1730279463893632/permalink/2541675446087359/.
This diff migrates code from accessing data members generated from unqualified
Thrift fields directly to the `field_ref` API, i.e. replacing
```
thrift_obj.field
```
with
```
*thrift_obj.field_ref()
```
The `_ref` suffixes will be removed in the future once data members are private
and names can be reclaimed.
The output of this codemod has been reviewed in D20039637.
The new API is documented in
https://our.intern.facebook.com/intern/wiki/Thrift/FieldAccess/.
drop-conflicts
Reviewed By: yfeldblum
Differential Revision: D22631599
fbshipit-source-id: 9bfcaeb636f34a32fd871c7cd6a2db4a7ace30bf
Summary:
Bypass truncation-based transaction if narrow-heads is on.
The transaction abort still works logically because commit references stay
unchanged on abort.
Related EdenFS and Mononoke tests are updated. Mononoke tests probably
shouldn't rely on revlog / fncache implementation details in hg.
Reviewed By: DurhamG
Differential Revision: D22240186
fbshipit-source-id: f97efd60855467b52c9fb83e7c794ded269e9617
Summary:
An assertion error is raised if `eden doctor` is in the middle of a merge. This is because we enter a specific "if" condition in the case that mercurial has two parent commits, and EdenFS only ever tracks `p0`, so EdenFS simply sets `p1` to the null commit in `_select_new_parents()`. Specifically, this is in the case in which both `_old_dirstate_parents` and `_old_snapshot` are not None.
Because `_old_dirstate_parents` has `p1` set to nonnull, and Eden thinks it is null , the check `self._new_parents != self._old_dirstate_parents` would be `True` even though there was actually no error.
Reviewed By: chadaustin
Differential Revision: D22048525
fbshipit-source-id: 9a19cc092e2bd80db0e01fb38533a1007640bee6
Summary:
On Windows, paths are case insensitive (but the filesystem is case preserving),
and thus `open("FILE.TXT")` and `open("file.txt")` refer to the same file. When
that file is not materialized and its parent directory isn't yet enumerated,
PrjFS will call the PRJ_GET_PLACEHOLDER_INFO_CB with the file name passed in to
the `open` call. In this callback, if the passed in name refers to a valid
file, it needs to call PrjWritePlaceholderInfo to populate the directory entry.
Here is what the documentation for that function states:
"For example, if the PRJ_GET_PLACEHOLDER_INFO_CB callback specifies
dir1\dir1\FILE.TXT in callbackData->FilePathName, and the provider's backing
store contains a file called File.txt in the dir1\dir2 directory, and
PrjFileNameCompare returns 0 when comparing the names FILE.TXT and
File.txt, then the provider specifies dir1\dir2\File.txt as the value of
this parameter."
While the documentation doesn't state how that name is used internally, we can
infer (and test) that the returned case will be used as the canonical
representation of that file, ie: the one that a directory listing will see.
Since the PathMap code already does a case insensitive search, we just need to
make sure to use what it returns instead of re-using the name used for the search.
The only caveat to all of this is the original comment that describe that
`metadata.name` can't be used as it causes crashes. From what I can tell, this
was written in later 2018, and I believe is no longer relevant: the
`metadata.name` field was simply not populated.
Reviewed By: wez
Differential Revision: D21799627
fbshipit-source-id: aee877cc2d5f057944fcd39b1d59f0e97de6315c
Summary:
updated `eden top` to:
- obtain PID-fetchCounts data from the updated -`getAccessCounts` thrift call in the previous diff
- display that data in a new column `FUSE FETCH`
Reviewed By: kmancini
Differential Revision: D22101430
fbshipit-source-id: 6584e71ce3a4629c73469607ca0a4c6ffd63e46f
Summary: The word "disabled" better express the intent behind it.
Reviewed By: fanzeyi
Differential Revision: D21945736
fbshipit-source-id: a61a3df4f1b704224e7f4d4a7f1a43f4d2b5b2aa
Summary:
Removing a directory with unlink raises different errors on Windows: EACCES
instead of EISDIR.
Reviewed By: fanzeyi
Differential Revision: D21944455
fbshipit-source-id: 251ceb305a46c01ece78f45b0fa8000778c6c650
Summary:
test_no_units_are_active was failing on CentOS 8. Presumably systemd
has some new default services. I believe genevievehelsel is planning on
replacing all of this code, so, rather than understand the failure,
just ignore the additional service names.
Reviewed By: genevievehelsel
Differential Revision: D22055992
fbshipit-source-id: b6f313350d0e1b107fe6ea3d7ed7f5b4eb025ef3
Summary:
In text mode, Python will use '\r\n' as line endings instead of '\n' on
Windows. This has the negative side effect of making some tests fails due to
the file content not being exactly what is being written.
Reviewed By: fanzeyi
Differential Revision: D21944456
fbshipit-source-id: f7f271e2ef9e2f21bd78c76233e644ce411b63df
Summary: In its current state, FORCE is a restart mode that cannot be combined with other modes (specifically `graceful`). Removing this coupling from `force` will allow us to at first attempt a graceful restart, but if the daemon is not healthy, then do a force restart, all within one call.
Reviewed By: wez
Differential Revision: D21873166
fbshipit-source-id: 079a777ff6e7b219198cbef15335244b9865f5f3
Summary:
Unawaited coroutines are pernicious and not errors by default. In
advance of enabling asynchronous tests, fail our tests when Python
warnings are shown.
We record warnings and report them after the test body instead of
enabling an "error" warning filter because exceptions thrown by
finalizers are silently swallowed, and most important warnings occur
during finalization.
Reviewed By: genevievehelsel
Differential Revision: D21955210
fbshipit-source-id: b1fc52ddfa9f9befbde6800f85f090862af0ef15
Summary:
Add a disabled test that illustrates that binary data fails to
serialize between edenfs and the CLI with the current Thrift
implementation.
Reviewed By: fanzeyi
Differential Revision: D21890379
fbshipit-source-id: 725e99751e1d62c39f4059ec0f1197857c47e3b4
Summary:
The Python 2-and-3 Thrift API is sort of deprecated and does not
handle binary data in `binary` fields. In advance of migrating to the
modern Python 3 API, remane eden.thrift to eden.thrift.legacy.
Reviewed By: fanzeyi
Differential Revision: D21889697
fbshipit-source-id: a745ee8977999acbfb383a4edebe81d8deb1734e
Summary:
I am planning to start migrating Eden's CLI to the new Python 3 Thrift
implementation. In preparation, slightly clean up the interface and
implementation of our Python 2 Thrift wrapper.
Reviewed By: genevievehelsel
Differential Revision: D21854539
fbshipit-source-id: d398dd3f324c12288871cf0c9db41e64ed4cf7ed
Summary:
Under heavy parallelism or system load, our tests could trigger
short-ish timeouts and cause tests to flake. The stats test in
particular often failed in continuous integration. It looks like
opening a unix domain Thrift socket early and holding onto it can
cause it to sometimes hit ThriftServer's default idle timeout of 60
seconds, which results in the test failing with BrokenPipeError
(EPIPE).
Reviewed By: simpkins
Differential Revision: D21780023
fbshipit-source-id: 7e8838429475c2a322d836b9a497411199948cce
Summary:
Update EdenFS on Windows to use EdenMain.cpp, the same as on POSIX.
This reduces the amount of code divergence, and also brings several
improvements to Windows: exposing version information over fb303, support for
dynamically changing logging configuration over fb303, etc.
Reviewed By: simpkins
Differential Revision: D21332569
fbshipit-source-id: dd2da7c385e96f65fc3927511c9f84b96bec9e2b
Summary:
This diff is auto-generated to upgrade the Pyre version and suppress errors in eden. The upgrade will affect Pyre local configurations in the following directories:
```
eden
```
Differential Revision: D21687853
fbshipit-source-id: baf0d9bc33f86da63ea289690faca6cf4d566588
Summary:
Now that regular users can create symlinks, a handful of tests were failing
due to the symlink call no longer raising an exception. For now, let's always
consider that symlinks aren't supported on Windows.
Reviewed By: fanzeyi
Differential Revision: D21664213
fbshipit-source-id: c55a99d1cb92e68b9861701b5517b1d5db2d40c6
Summary:
Migrating buck integration to use configurations, which is more performant and is the primary supported way to run Pyre.
Mypy is no longer supported at Facebook, so mypy buck integration is covered in this migration as well.
Configuration target automatically expanded to include all subtargets, expanding type coverage while introducing no more than 2 fixmes per file.
Reviewed By: dkgi
Differential Revision: D21474723
fbshipit-source-id: 2501af8250b4998fe331350b80c9dc8b4ba2dd1a
Summary:
Our use of the term "client" to refer to a checkout is
deprecated. Rename some internal functions that use the term client.
Reviewed By: simpkins
Differential Revision: D21395159
fbshipit-source-id: fa96ba593f53b493e5ae816fa686f333a132c232
Summary: All of the tests in hg/status_test.py pass except for these 3.
Reviewed By: genevievehelsel
Differential Revision: D21410570
fbshipit-source-id: cf9a87ce0f025860c1e95927a780757fda6801ad
Summary:
- If we fail to create symlinks ignore this error on Windows
- Using a file as a directory produces an ENOENT error instead of ENOTDIR
on Windows.
- Deal with some issues around CR -> CRLF translation when writing files in
non-binary mode on Windows.
- Move several POSIX-specific tests out into a separate test class.
Reviewed By: wez
Differential Revision: D21214598
fbshipit-source-id: 587a755bd06990bf88d26017254a69ff8869edbc
Summary:
On Windows we can only run EdenFS if ProjectedFSLib.dll is available. It
turns out that some Windows systems have ProjectedFSLib.lib available, so we
can build `edenfs.exe`, but are missing ProjectedFSLib.dll, so we cannot
actually run it.
Reviewed By: wez
Differential Revision: D21443255
fbshipit-source-id: 6396fce8bd71adc0695468a2df3fd5f584909138
Summary:
Addressing issues simpkins brought up on D21207287 when we upgraded and introduced some pyre bugs.
Temporarily upgrading just this project, once we resolve some sandcastle capacity issues we'll release this via another global upgrade in fbcode.
Reviewed By: simpkins
Differential Revision: D21316793
fbshipit-source-id: f0c79f53d97f7182e7d8fe6e081c58ef53ce0c9a
Summary:
Previously we only included `basic_test.py` and `hg/status_test.py` in the
integration tests during CMake-based builds. This updates the code to now
include all of the test files, with just a few exclusions based on platform
type and what dependencies were available at build time.
Reviewed By: wez
Differential Revision: D21239912
fbshipit-source-id: b8826d249a6323ac3bcc555c9ceba54a4cbcfde9
Summary:
10 of the integration tests fail on Ubuntu. For now simply blacklist them
from running on non-RedHat-based distributions.
Reviewed By: wez
Differential Revision: D21332629
fbshipit-source-id: 3fadb74bb31e89092177afaa01ddc7f6bcd0f9de
Summary:
Add a new IntegrationTestCase base class that checks the test blacklist during
`setUp()`, and update a few remaining test classes that did not derive from
`EdenTestCase` to derive from this.
Also drop the method name argument from the `skip_test_if_blacklisted()`
function, since we can get this from the existing test case argument.
Reviewed By: genevievehelsel
Differential Revision: D21340539
fbshipit-source-id: d4fc125f119d74ab923c2cc3c9070b86c582c87e
Summary:
Run `env` as `/usr/bin/env` instead of `/bin/env`
This command is typically installed in `/usr/bin`. Recent RedHat releases
replaced `/bin/ with a symlink to `/usr/bin`, allowing this command to work
when invoked as `/bin/env`, but this isn't true on all distributions, such as
Ubuntu.
Reviewed By: chadaustin
Differential Revision: D21340540
fbshipit-source-id: d7588f8c90e9a86a0cb31fd3ab3a9067aa6e79ea
Summary:
D21316793 is blocked from landing because there are a few targets in eden that are running pyre via buck targets integration.
We can't do custom version overrides for projects that are using a mix of local configurations and buck integration, because buck doesn't provide an interface for setting the equivalent pyre version override.
We're moving away from buck targets integration for pyre across the board, and I've run a codemod over the project to clean up all of the buck typing integration (including some residual mypy) as well as updated type ignores / fixmes accordingly.
Let me know if you have any concerns; upon skimming it looks like most changes are either converting `type: ignore`s into fixmes, or removing `type: ignores`.
Reviewed By: dkgi
Differential Revision: D21343093
fbshipit-source-id: 5ee1436377eb526c0a679fb821c42e07cbca52a5
Summary:
FUSE_HANDLE_KILLPRIV expects that any write() call is handled by
clearing the setuid and setgid bits in the userspace. To avoid
implementing that behavior, disallow setting setuid or setgid in the
first place.
Reviewed By: xavierd
Differential Revision: D21333703
fbshipit-source-id: eb084ee8b00afe74c0da26e41c32c2cb742723da
Summary:
The systemd binary is installed in a different location on Debian-based
distributions (e.g., Ubuntu) from RedHat-based distributions. Update the
integration tests to look in both places.
Reviewed By: wez
Differential Revision: D21297591
fbshipit-source-id: 67e5a698080205b8624c1f38aa49e75c20b6a28c
Summary:
Our CMakeLists.txt file requests `ctest` to pass in a `CMAKE_SOURCE_DIR`
environment variable when running our integration tests via `ctest`. However
if a developer manually invokes the test binary this environment variable may
not be set.
For convenience, this updates the code to try looking at the default location
where getdeps will have put the source relative to the build directory (at
least for internal builds using shipit-converted fbsource sources).
Reviewed By: wez
Differential Revision: D21307807
fbshipit-source-id: d747c50d79d2d378721b68b18c256d5363e41f26
Summary:
Update the integration tests to attempt to find the Mercurial config files
from the source directory even in CMake-based builds. This directory will be
available when doing an internal build from fbsource via getdeps.py
If this directory isn't present (e.g., on a build from the github repository)
we will still fall back to using the Mercurial configuration files installed
on the system. In that case I did update the code to use the correct system
configuration path on Windows.
Reviewed By: wez
Differential Revision: D21307808
fbshipit-source-id: 917ba5e8548fed999fb06a29324be125cec83cac
Summary:
Remove the requirement that the `--edenfs` argument flag must be passed in
when invoking the `edenfs` daemon.
This flag was added in D12927803 to help provide a better error message if
users accidentally ran `edenfs` when they really mean to run `edenfsctl`.
However, this shouldn't really be a major problem any more: on Linux and Mac
we now install `edenfs` in a `libexec` subdirectory instead of `bin`, so it
should not be in most user's `$PATH` by default any more. Additionally we
also verify that no other unexpected arguments have been supplied. Prior to
D12927803 the `edenfs` binary would ignore unexpected arguments, making the
error messages more confusing when users tried to run it with `edenfsctl`
subcommand arguments.
One motivation for changing this now is that the Windows version of `edenfs`
does not require this flag, and it is desirable to remove this behavior
discrepancy. Rather than making this flag required on Windows it seems fine
to just drop the requirement on Mac & Linux.
Reviewed By: wez
Differential Revision: D21297159
fbshipit-source-id: e24bd694dadc036cd31dead287ae2c1184747822
Summary:
Update the tests that derive from `ServiceTestCaseBase` to honor the test
blacklist, and to also skip if we cannot run the `fake_edenfs` utility. This
latter check primarily fails when we cannot run `sudo` non-interactively
without a password prompt.
Most of the other tests were already honoring blacklist through the
`EdenTestCase` base class, but `ServiceTestCaseBase` does not derive from
`EdenTestCase`.
Reviewed By: wez
Differential Revision: D21297160
fbshipit-source-id: 5044e9939bbe487c09aa96021166c95f02fb376e
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.
This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper. However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.
Reviewed By: wez
Differential Revision: D21296594
fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
Summary:
This sets up the counters that will allow us to expose the number
of pending FUSE requests in Eden top.
As D20846826 mentions adding metrics for FUSE request gives
visibility into fuse requests and overall health of eden.
This provides more insight beyond the metrics for live FUSE requests
since it shows the kernels view of FUSE requests. Looking at the difference
between the number of pending and live request, can identify issues
that arise at the interface between eden and FUSE and monitor how
quickly fuse workers are processing requests.
**note**: this is only for linux since macos has no equivalent to
`/sys/fs/fuse/connections`
Reviewed By: chadaustin
Differential Revision: D21074489
fbshipit-source-id: c0951f0dfd4fa764be28d8686d08cd0dd807db37
Summary:
In the past, if an EdenMount got stuck while shutting down (inodes still referenced), the EdenMount would already be removed from the server's `mounPoint's` map while stuck. This resulted in `eden list` showing that the mount was not mounted, and then if we tried to remount the mount, we'd get a confusing error message about the not being able to grab the overlay lock. This also made it so if we tried to attach with gdb, or if we core dumped and tried to look at the data, we wouldn't be able to examine the object.
Instead, we should only remove the mount from the `mounPoints_` map after it has completely been shutdown. This will allow us to get better signal on the state of the mount (it will be in the `SHUTTING_DOWN` state in `eden list` instead of `(not mounted)`) and it will allow us to better debug these situations (where a mount is stuck shutting down) since we can get more information about the mount that is in this state.
Reviewed By: simpkins
Differential Revision: D21211544
fbshipit-source-id: 97ff11df67d0a80f47cf6b9f563876e17bd76622
Summary:
Update `integration/lib/linux.py` to use `ctypes` to call `statfs`, instead of
running `/bin/stat` and parsing the output. This allows this test code to
work even on systems that do not have `/bin/stat` (e.g., on Ubuntu this is
installed at `/usr/bin/stat`).
Also update the usage of ctypes in the `mmap.py` test file to allow this file
to be imported successfully on Mac OS X.
Reviewed By: xavierd
Differential Revision: D21273385
fbshipit-source-id: c2173758f8ea67cde2a68e57827acc5f73b63179
Summary:
Update the `test_rebase_commit_with_independent_folder()` test to explicitly
disable in-memory updates. The in-memory update code ends up changing the
number of update operations performed (it skips the initial and final update),
causing one of the assertions in this test to fail.
Reviewed By: xavierd
Differential Revision: D21271902
fbshipit-source-id: 3e40adce5fac1c851c8f277d2209d48a19deed55
Summary:
Update the integration test code that uses pexpect to use the more modern
pexpect APIs. The top-level `pexpect.spawn()` function is considered legacy
now, and is only provided for backwards compatibility on Unix platforms.
Reviewed By: wez
Differential Revision: D21214640
fbshipit-source-id: 941da5435c4f8afaf22e8053f4c344175d7b1a7f
Summary:
Enable building fake_edenfs.cpp and TakeoverTool.cpp in the CMake build.
This includes a few changes to get `fake_edenfs.cpp` closer to building on
Windows, but at the moment it still does not fully build since the
`StartupLogger` is currently not compiled on Windows.
TakeoverTool.cpp is specific to the graceful restart tests, and doesn't make
sense to ever build on Windows.
Reviewed By: wez
Differential Revision: D21214614
fbshipit-source-id: a7a8306ea3f69579fa02bd3753a66d3095ea004a
Summary:
Previously none of the integration tests ran on Windows, as they checked for
the presence of `/dev/fuse` first, and would skip the test if it was not
present.
This updates the tests to allow running on Windows, and updates the
`find_executables.py` code to find the executables properly on Windows.
Reviewed By: wez
Differential Revision: D17733621
fbshipit-source-id: a8bf4e5138f9ea6c9b040814dd68fc64228961de
Summary:
systemd is Linux-specific, so it does not make sense to try and run these
tests on Linux and Mac. Additionally, the pystemd module currently used by
the edenfsctl CLI is not currently open source, and therefore only works in
our internal Linux builds.
This updates the integration test code to only include systemd-related tests
if we are actually in an environment where we support systemd.
Reviewed By: wez
Differential Revision: D21084095
fbshipit-source-id: fb6c9a066d9dcb741bddf3cdcf9decdb763a1d05
Summary:
Several of the tests started edenfs of fake_edenfs processes and didn't kill
them at the end of the test. This adds proper cleanup functions to try and
kill these processes when the test exits.
This also eliminates using pexpect in these tests, since it isn't actually
necessary here.
Reviewed By: wez
Differential Revision: D21084096
fbshipit-source-id: 4e92a99a5c398d4a78830ac51507ba34d7a6c0b1