Summary:
The systemd binary is installed in a different location on Debian-based
distributions (e.g., Ubuntu) from RedHat-based distributions. Update the
integration tests to look in both places.
Reviewed By: wez
Differential Revision: D21297591
fbshipit-source-id: 67e5a698080205b8624c1f38aa49e75c20b6a28c
Summary:
On Linux the debug info sections in projects downstream from folly and
thrift are huge due to a lot of C++ template usage.
We build in RelWithDebInfo mode as that is most useful when debugging
locally and because the build times are long, but most casual consumers
of the binaries via GH actions really don't care about debug info,
and this should save ~180MB of size in (for example) the watchman
executables.
Pull Request resolved: https://github.com/facebook/watchman/pull/809
Refs: https://github.com/facebook/watchman/issues/804
Test Plan:
Review the artifact size on the linux build in this PR (see
https://github.com/facebook/watchman/actions/runs/91688952) and confirm that it
is a bit smaller than 180MB; now under 3 MB.
Reviewed By: simpkins
Differential Revision: D21318302
Pulled By: wez
fbshipit-source-id: f78bc5e735dd78771e0604abae64c0a23cf9158d
Summary:
Our CMakeLists.txt file requests `ctest` to pass in a `CMAKE_SOURCE_DIR`
environment variable when running our integration tests via `ctest`. However
if a developer manually invokes the test binary this environment variable may
not be set.
For convenience, this updates the code to try looking at the default location
where getdeps will have put the source relative to the build directory (at
least for internal builds using shipit-converted fbsource sources).
Reviewed By: wez
Differential Revision: D21307807
fbshipit-source-id: d747c50d79d2d378721b68b18c256d5363e41f26
Summary:
Update the integration tests to attempt to find the Mercurial config files
from the source directory even in CMake-based builds. This directory will be
available when doing an internal build from fbsource via getdeps.py
If this directory isn't present (e.g., on a build from the github repository)
we will still fall back to using the Mercurial configuration files installed
on the system. In that case I did update the code to use the correct system
configuration path on Windows.
Reviewed By: wez
Differential Revision: D21307808
fbshipit-source-id: 917ba5e8548fed999fb06a29324be125cec83cac
Summary:
Remove the requirement that the `--edenfs` argument flag must be passed in
when invoking the `edenfs` daemon.
This flag was added in D12927803 to help provide a better error message if
users accidentally ran `edenfs` when they really mean to run `edenfsctl`.
However, this shouldn't really be a major problem any more: on Linux and Mac
we now install `edenfs` in a `libexec` subdirectory instead of `bin`, so it
should not be in most user's `$PATH` by default any more. Additionally we
also verify that no other unexpected arguments have been supplied. Prior to
D12927803 the `edenfs` binary would ignore unexpected arguments, making the
error messages more confusing when users tried to run it with `edenfsctl`
subcommand arguments.
One motivation for changing this now is that the Windows version of `edenfs`
does not require this flag, and it is desirable to remove this behavior
discrepancy. Rather than making this flag required on Windows it seems fine
to just drop the requirement on Mac & Linux.
Reviewed By: wez
Differential Revision: D21297159
fbshipit-source-id: e24bd694dadc036cd31dead287ae2c1184747822
Summary:
Update the tests that derive from `ServiceTestCaseBase` to honor the test
blacklist, and to also skip if we cannot run the `fake_edenfs` utility. This
latter check primarily fails when we cannot run `sudo` non-interactively
without a password prompt.
Most of the other tests were already honoring blacklist through the
`EdenTestCase` base class, but `ServiceTestCaseBase` does not derive from
`EdenTestCase`.
Reviewed By: wez
Differential Revision: D21297160
fbshipit-source-id: 5044e9939bbe487c09aa96021166c95f02fb376e
Summary:
Move the `UserInfo` code from `fuse/privhelper` to `utils`, and also unify the
POSIX and Windows implementations of this class.
This code was originally put in the `fuse/privhelper` directory since it was
written at the same time as the privhelper. However, it's really a
lower-level library that doesn't depend on FUSE or any of the other code in
the `fuse/` subdirectory.
Reviewed By: wez
Differential Revision: D21296594
fbshipit-source-id: f58682f6ce86bba0328472c491bb4c0dc3370319
Summary:
Rather than have a single main.yml file containing all off the different
builds, split that up so that we have one file per build environment
(linux, mac, windows).
This has a couple of advantages:
* It is quicker to see the status of just one of the platforms
* Artifact collection for one platform is not blocked pending completion
of the builds for all systems
* It's a little easier to understand what is happening for a single platform
To support having multiple files I've changed the output-file option to
be an output-dir.
I've included the rm of main.yml in this commit. Once this gets
imported back to the FB internal system I'll amend in an update to
the helper script that updates all of our opensource builds and run
and amend that.
Pull Request resolved: https://github.com/facebook/folly/pull/1360
Test Plan:
the GH action status on this PR should show three different
actions running, one for each platform.
I updated and ran
`fbcode/opensource/fbcode_builder/getdeps/facebook/update-all-github-actions.sh`
to regenerate all the actions files for FB.
Reviewed By: yfeldblum
Differential Revision: D21310991
Pulled By: wez
fbshipit-source-id: 604ef652c8f746781a4b410c6b996cdee4524e0d
Summary:
Merge react and userprofile_shell into master, rewrite profile to use react and revert changes to p page
```
Reviewed By: markbt
Differential Revision: D21285297
fbshipit-source-id: f60006738af068982a25fa368fd260ba5b7f3781
Summary:
This commit resolves an issue with our zipapp executables
on Windows that meant that the only reliable way to start them was
to use the fully path to the executable.
The root cause is that the __wargv array is produced by parsing the
process command line into an array, and if you ran `watchman-wait -h`
__wargv[0] would have `watchman-wait` rather than the fully qualified
path to the executable that the zipapp plumbing requires.
The fix is to ask the system for the fully qualified path and ensure that
that gets set as both argv[0] AND argv[1].
Reviewed By: xavierd
Differential Revision: D21190350
fbshipit-source-id: eeb95084592d30a028a93b2b03877f8cc6c72729
Summary:
The environment changed since I tested D20740410 and now it appears
that we'll need to re-export a versioned variable in order for cmake
to detect boost on the GH actions hosts.
Pull Request resolved: https://github.com/facebook/folly/pull/1359
Test Plan:
the GH actions status of this diff.
The workflow was updated via:
```
python3 build/fbcode_builder/getdeps.py generate-github-actions folly --output-file .github/workflows/main.yml
```
Reviewed By: yfeldblum
Differential Revision: D21307640
Pulled By: yfeldblum
fbshipit-source-id: 1555cbcade822775379cd9054be37fdbc17b4d93
Summary: The changes to server/context, gotham_ext and the code that depends on them are the only reminding places where aclchecker is used directly and it is not easy to split this diff to convert them separately.
Reviewed By: krallin
Differential Revision: D21067809
fbshipit-source-id: a041ab141caa6fe6871e1fda6013e33f1f09bc56
Summary:
From the outset, we wanted to be sure that getdeps was able
to source and build the dependencies so that we knew that we'd have
a repeatable build. This came at the cost of build times: having
to build boost on each CI run is a bit of a chore.
This commit adds three new elements to the manifest files:
* `rpms` - a list of RPM names that are all required to be present
in order to consider the dependency satisfied
* `debs` - like `rpms` above, but scoped to debian package names
* `preinstalled.env` - a list of environment variables that if they
are all set and non-empty will satisfy the dependency.
A new `--allow-system-packages` option to getdeps enables the new
logic that looks for system packages; it is off by default, but
enabled in the generated GitHub Actions workflows.
A new `install-system-deps` subcommand is provided that will attempt
to install the system packages needed to satisfy the build. This
typically needs to be run via sudo and is thus broken out separately
from the main getdeps build flow.
I made a pass over the manifest files and added package names that
satisfy the build on ubuntu-18 and fedora-31.
shri-khare: I renamed the `Python3.7.6` manifest to just `python` as
part of this change; the version of python that it pulls in through
the normal build is the same and I believe that an equal or newer
version of python3 is available in the GH actions builder.
The `preinstalled.env` is used only by the boost manifest: it references
the name of an environment variable that is set by the github
windows hosts and that points to a pre-built and pre-installed
copy of boost. Since there is no package manager that we can
easily query for this sort of thing, probing from the environment
seems like a reasonable and fast way to check for this. We
may need to evolve this over time to become more feature rich,
but this seems like a good starting point.
This commit has the potential to save 20 minutes of build time
from each public CI build just due to the boost dependency alone!
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: yfeldblum
Differential Revision: D20740410
fbshipit-source-id: 6c38019449c54465127656c3d18a6ff1f30adaea
Summary:
Enable the unit tests under eden/fs/utils on Windows.
This does comment out a few tests related to `AbsolutePath` that are broken on
Windows. The AbsolutePath constructor does not enforce that the input path is
actually absolute. Additionally the `canonicalPath()` function ends up doing
weird things and returning paths that start with `/` followed by a drive
letter (e.g., `/C:/foo`). These issues should ideally be addressed in
subsequent diffs.
Reviewed By: xavierd
Differential Revision: D21239886
fbshipit-source-id: ef08d62353ba83b96d9fd79bd4636f4a0f961373
Summary:
When connecting to Mononoke, it's possible for the connection to hang
for a while before ultimately failing. This is a poor experience for users.
Add a timeout, such that hgcli will only wait for 15 seconds before reporting a
failure back to the user.
Reviewed By: krallin
Differential Revision: D21281472
fbshipit-source-id: 6329a108a1ae3ec337d74529a80e0fcd74e02231
Summary: Add a README to provide an overview of the walker for reviewers and maintainers
Reviewed By: ikostia
Differential Revision: D21250280
fbshipit-source-id: 9fd7cd3a076a276de904b8ae5372fc1c0c28458b
Summary:
The operation originally wanted to operate on the fuse `Attr`
structure which we don't have on Windows, so I repurposed the
`InodeBase::getattr` into `InodeBase::stat` and moved the conversion
of `struct stat` to `Dispatcher::Attr` to the `EdenDispatcher::getattr`
method (and a couple of other adhoc places that were doing a similar
conversion).
Reviewed By: chadaustin
Differential Revision: D20562459
fbshipit-source-id: 6b538110038352e9b5590fcb5ff5c33fe84ac1d8
Summary:
The only change I had to make was due to the fact that MSVC wasn't smart
enough to realize that the shift value couldn't be negative, so a manual
folly::to_unsigned was added to silence the warnings.
Reviewed By: simpkins
Differential Revision: D21268634
fbshipit-source-id: e65f15d58d5ea23bfa6796bab23cf1f5c2e7c12c
Summary:
On Windows, the expected refcount is one less than what it is on Linux, due
to the .eden directory not being present.
Reviewed By: simpkins
Differential Revision: D21268203
fbshipit-source-id: 91cfe742fa4d576917d552964d9541dc68ad2c75
Summary: Fix permission issues we are seeing with the latest Mercurial release.
Reviewed By: xavierd
Differential Revision: D21294499
fbshipit-source-id: bcfb13dd005258b2e3b74fa281dbd8df36133ef6
Summary: I thought it would be helpful to introduce and `eden uptime` command, especially with automated graceful restart on the way. This prints it in human readable format, later on if for some reason automation would like to use this, a flag could be added that allows for custom formatting. Also, this can be added to `eden rage` output later.
Reviewed By: chadaustin
Differential Revision: D21260800
fbshipit-source-id: 3f9a4f8d6264dfc38bd15c024a0209f7eeb912fa
Summary:
Previously, for logging the value of unexpected enum values, we wrote
`static_cast<int>`. Given enumerations can have any integral type
backing them, this was somewhat inaccurate. Instead, introduce an
explicit enumValue function which returns a value of the appropriate
underlying type.
Reviewed By: genevievehelsel
Differential Revision: D20975176
fbshipit-source-id: 0bb5b0d2f68f8fe9d68e4c6a847d59ae0997d0df
Summary:
D21074489 adds metrics for pending FUSE requests, this cleans up the display for
pending requests.
This removes the max duration of pending requests for FUSE requests since this
data is not available (it is not measured by the FUSE library).
Reviewed By: chadaustin
Differential Revision: D21074746
fbshipit-source-id: e5585ec091aa5fd5499deee2d8be89f47f769a6a
Summary:
This adds the number and duration of FUSE requests to eden top.
The purpose of this metric is to provide a highlevel indicator of
the health of eden and the fuse interface. Because FUSE is the
interface through which the user mostly interacts with the file
system it these metrics are reflective of edens overall
responsiveness.
The metrics are collected per mount, but we only display the
aggregate since this serves as an overall summary metric.
These metrics are displayed similarly to the import metrics,
for consistency. There is not a metric for the max duration
of pending FUSE requests D21074746 cleans this up so that
the NA is removed from the display.
Reviewed By: chadaustin
Differential Revision: D20928603
fbshipit-source-id: ce78f4274203c2302e7595ae2ef601e18893db38
Summary:
This sets up the counters that will allow us to expose the number
of pending FUSE requests in Eden top.
As D20846826 mentions adding metrics for FUSE request gives
visibility into fuse requests and overall health of eden.
This provides more insight beyond the metrics for live FUSE requests
since it shows the kernels view of FUSE requests. Looking at the difference
between the number of pending and live request, can identify issues
that arise at the interface between eden and FUSE and monitor how
quickly fuse workers are processing requests.
**note**: this is only for linux since macos has no equivalent to
`/sys/fs/fuse/connections`
Reviewed By: chadaustin
Differential Revision: D21074489
fbshipit-source-id: c0951f0dfd4fa764be28d8686d08cd0dd807db37
Summary:
It turns out that macOS, FreeBSD, and Linux all agree on the values
for dtype. If we assume dtype is just the high nibble of mode_t
shifted right by 12, then the Windows CRT agrees too. So hardcode the
values in eden.thrift and add appropriate static_asserts. This opens
the possibility of simply static_cast'ing the numeric values in
Watchman.
Reviewed By: simpkins
Differential Revision: D20975101
fbshipit-source-id: 354ffcbdf3f1d5f8b1715abf1026eaea429d16cf
Summary:
This makes it a bit easier to write integration tests
that run an isolated EdenFS instance complete with its own
unique scratch path environment.
Reviewed By: genevievehelsel
Differential Revision: D21258455
fbshipit-source-id: 3ea13b6337bb8bb58a237e67488b116c423bbe6c
Summary:
In the past, if an EdenMount got stuck while shutting down (inodes still referenced), the EdenMount would already be removed from the server's `mounPoint's` map while stuck. This resulted in `eden list` showing that the mount was not mounted, and then if we tried to remount the mount, we'd get a confusing error message about the not being able to grab the overlay lock. This also made it so if we tried to attach with gdb, or if we core dumped and tried to look at the data, we wouldn't be able to examine the object.
Instead, we should only remove the mount from the `mounPoints_` map after it has completely been shutdown. This will allow us to get better signal on the state of the mount (it will be in the `SHUTTING_DOWN` state in `eden list` instead of `(not mounted)`) and it will allow us to better debug these situations (where a mount is stuck shutting down) since we can get more information about the mount that is in this state.
Reviewed By: simpkins
Differential Revision: D21211544
fbshipit-source-id: 97ff11df67d0a80f47cf6b9f563876e17bd76622
Summary:
Update `integration/lib/linux.py` to use `ctypes` to call `statfs`, instead of
running `/bin/stat` and parsing the output. This allows this test code to
work even on systems that do not have `/bin/stat` (e.g., on Ubuntu this is
installed at `/usr/bin/stat`).
Also update the usage of ctypes in the `mmap.py` test file to allow this file
to be imported successfully on Mac OS X.
Reviewed By: xavierd
Differential Revision: D21273385
fbshipit-source-id: c2173758f8ea67cde2a68e57827acc5f73b63179
Summary:
Update the `test_rebase_commit_with_independent_folder()` test to explicitly
disable in-memory updates. The in-memory update code ends up changing the
number of update operations performed (it skips the initial and final update),
causing one of the assertions in this test to fail.
Reviewed By: xavierd
Differential Revision: D21271902
fbshipit-source-id: 3e40adce5fac1c851c8f277d2209d48a19deed55
Summary:
The new temporary file code added in D20885160 was missing a call to close the
underlying file in the `__exit__()` function.
Reviewed By: xavierd
Differential Revision: D21271901
fbshipit-source-id: 653258d471421edd08b473654ae3d0503f3c6829
Summary:
enums and i16 are not serialization-compatible in Thrift, but we want
an enumeration with defined values for dtypes. Rename the existing
Dtype type to OsDtype in preparation for introducing the Dtype enum.
Reviewed By: simpkins
Differential Revision: D20974559
fbshipit-source-id: 5d9087a79e7fd59b99b257a99351a8287ba00b5f
Summary: `termios` is not avialable on Windows. Do not import it.
Reviewed By: DurhamG
Differential Revision: D21258999
fbshipit-source-id: f4390b69fe9abceea8b1959e7506c1558778f980
Summary:
From looking at the code, it appears that the FUSE refcount is always a
uint32_t, except when serialized. Let's do the signedness conversion and
narrowing then.
Reviewed By: simpkins
Differential Revision: D21240161
fbshipit-source-id: 877c6cb6881cb36346c64cf92d99b1e588aed580
Summary:
Add a new function that initialises common parts of Mononoke:
- cachelib
- logging
- runtime
In the next diff, I will also update `init_mononoke` to initialise tunables
too.
I left some users of `init_runtime` unchanged (such as `mononoke_admin`) as
they conditionally enable cachelib.
Reviewed By: krallin
Differential Revision: D21177253
fbshipit-source-id: 40b534d1b244fd8dcd05e1cff1c9f3edfb32a4b9
Summary:
For a while now, `arc lint` wouldn't properly format the code on Windows, the reason is due to the fact that the default sparse profile doesn't include fbcode/.clang-format.
However, adding this file to the sparse profile wouldn't help as it is a symbolink link to fbsource/tools/clang-format/clang-format.ini, and as we all know, Mercurial will not create symbolic links on Windows, instead a plain file is created and its content is the link destination. The main issue with this is that clang-format simply doesn't understand what this means, and thus fails to format any code :(.
Now, it turns out with recent enough version of Windows (every Windows version we support at FB do support it) being an administrator is not a requirement to create symlinks (D21258136), once this lands. I believe that once this lands, and Mercurial is taught to properly create symlinks on Windows, this change will no longer be necessary and we can revert it. In the short term, let's make our life slightly easier by copying the fbsource clang-format configuration.
Reviewed By: simpkins
Differential Revision: D21258015
fbshipit-source-id: 21b4fda3a28bd397244b0fc9ca730ced775e96f0
Summary:
The disabled test uses chmod and symlinks that aren't present in the WIN32
build. I'm not exactly sure why the .eden couldn't be found though.
Reviewed By: simpkins
Differential Revision: D21264070
fbshipit-source-id: da7385d9de4002c99fcbe79f37bd09acd41fbec8
Summary: The new API is more generic, use it in favour of the fbcode only aclchecker
Reviewed By: StanislavGlebik
Differential Revision: D21067812
fbshipit-source-id: fdbd61697f65bd34c114e6d37108f395c77c0b17
