Summary:
## Backstory
Pyre was throwing errors in my diff (D17747558) regarding an extraneous fixme. Turns out PyreBot has been adding and removing these fixmes during version updates (see D17135557 and D16183608), so I suspect it's something to do with the Pyre version. Anyways, I figured it'd be easier to do the annotation than to remove the fixme and risk Pyre throwing the same error in a later diff.
## What I did
I added the Key type (Path) and Value type(ExpectedFileBase). mypy then started throwing an error regarding __iter__ returning the wrong type because it wanted an iterator over keys rather than values. Fixed that and added .values() to the for loop.
Reviewed By: genevievehelsel
Differential Revision: D17806135
fbshipit-source-id: c07feee33db78a9bff19ba9856a7047657b8c63e
Summary:
We make use of the KERN_PROCARGS2 MIB data that we can
retrieve via `sysctl`.
If we can't retrieve that data then we fall back to libproc as
we were doing previously. From my testing so far it seems like
the main reason for failure is that the target process is a
protected system process.
Reviewed By: chadaustin
Differential Revision: D17724101
fbshipit-source-id: 8de1a978e6f89612bfe247e0fd540d9078f50746
Summary: This will help us identify the calling process for the FS request. This was helpful while testing watchman crawl in Eden repo.
Reviewed By: chadaustin
Differential Revision: D17719028
fbshipit-source-id: 0d646d7949a6e26268b71287b26d63473a9a87f8
Summary:
We are seeing random segment fault originating from OpenSSL on macOS when
Mononoke fetching is enabled.
The cause is that on macOS we are actually linking against libcurl shipped with
the system instead of ours. That copy of libcurl is linked with macOS's
libcrypto instead of the one we compiles during Eden's build, and it seems that
version of libcrypto does not provide concurrency safety.
The solution is to build curl on macOS and make sure it is linked to our
OpenSSL that has the concurrency callbacks registered.
Reviewed By: wez
Differential Revision: D17657822
fbshipit-source-id: 85abdf3be10b3903a5efc6b3a91624c7258de790
Summary:
Update the CMakeLists.txt files to support building the hg integration tests.
At the moment this only includes one of the test files (`status_test.py`).
I have not verified if tests from the other modules pass yet or if they need
any additional tweaks to work in CMake-based builds.
Reviewed By: pkaush, fanzeyi
Differential Revision: D17678991
fbshipit-source-id: 4a5ee5a8d6039d9d2a635c7027897bbeed14f8c0
Summary:
Add initial support for building and running some of the integration tests
with CMake. For now this just runs the tests from basic_test.py, just to
confirm that most of the framework code works in CMake-based builds.
Many of the other tests should also work as well, but a few of them we may
want to disable for CMake-based builds. e.g., a couple of the tests depend on
hypothesis, and we would need to include hypothesis as a dependency. Some of
the tests that use systemd might also require a little more work to get
working.
Reviewed By: fanzeyi
Differential Revision: D17659026
fbshipit-source-id: 67420fda9e1021a0cddee2d385fd21e34fb2fd70
Summary:
One of our diagnostics attempts to run `nuclide-connections`,
and prints a warning if that tool is not found.
In our macOS environment that tool is never present so it always emits
a warning that is not actionable.
This diff suppresses that particular warning.
Reviewed By: fanzeyi
Differential Revision: D17745753
fbshipit-source-id: 3394e88d443383880b632fca04a1ec074e46a9bd
Summary: Moved the code from vcproj testutils folder to test folder and enabled the Cmake build on it.
Reviewed By: wez
Differential Revision: D17738227
fbshipit-source-id: bcba41cb29b7e24782c36899c0977c5592888ff5
Summary:
Make it possible for any user on the system to connect to Eden's unix
domain socket. Our own permission checking is implemented in D17459971.
Reviewed By: wez
Differential Revision: D17460113
fbshipit-source-id: 29eba62010a58837df0376c66e252f53314a25dd
Summary:
To allow other user accounts to access fb303 counters, we plan to open
our unix domain socket to the world. But that requires implementing
our own permission checks, so do that first.
Reviewed By: wez
Differential Revision: D17459971
fbshipit-source-id: 3cc512da90f725d1c93f25f2a7a5ea7c72fb2b96
Summary:
Previously, readdir-triggered prefetches would merely look up and
cache the inodes for entries. Given a lookup also needs to return stat
information, including file sizes, prefetch blob sizes too.
Reviewed By: wez
Differential Revision: D17728752
fbshipit-source-id: 1c555c123fb7405f82885b427b32cd007fdb16ef
Summary:
Converting the const ptr in EdenDispatcher because we need to call non-const function (like update Journal). Also storing the reference in EdenDispatcher and WinStore instead of ptr.
This diff should not change the behavior of the Eden.
Reviewed By: chadaustin
Differential Revision: D17719606
fbshipit-source-id: 6c371c38045c760837130bb607163c2bc469b17a
Summary: This allows us to test the datapack code easier without rebuilding Eden.
Reviewed By: wez
Differential Revision: D17468473
fbshipit-source-id: a6807b4d6e747ae8557ae51fdf798de2a54fd4f1
Summary:
When Eden is competing with hundreds of build processes using about
the same amount of RAM, it's less painful for the OOM killer to kill
the compilers, not the edenfs daemon. Bias its OOM score a little.
Reviewed By: wez
Differential Revision: D17701071
fbshipit-source-id: 5ee1078e78e5aa6251692944ca12d9a892f86a6f
Summary: Now that Eden has dropped support for flatmanifest, allow GC'ing trees in the local storage.
Reviewed By: simpkins
Differential Revision: D17236345
fbshipit-source-id: d40ec0c2e61c9971ba1d976572d002c48c7bfc20
Summary:
The current debug file_stat tool returns a list of read and written files with their corresponding sizes. This is difficult to read and understand.
We add a summary section with:
- file count
- total bytes read/written
- total large files (> 10MB) read/written
- list of large files
The existing list of read/written files are moved into a `details` section.
Reviewed By: fanzeyi
Differential Revision: D17633963
fbshipit-source-id: a0b14d900e13590214b64fd9bbd7db2a0c69b300
Summary:
D16836053 -> Diff where --sizes arg was added. Back then, sizes were computed locally and added significant overhead, so it made sense to put this feature behind a flag.
D17080186 -> Diff where function was updated to fetch sizes from thrift call instead of computing it locally
Eden CLI seems to be deployed on Sandcastle on a weekly cadence, so I'm holding off on removing the --sizes flag for now. We can wait till this change is deployed, then make the change to the callsite [here](https://our.intern.facebook.com/intern/diffusion/WWW/browse/master/flib/intern/sandcastle/util/SandcastleEdenUtil.php?commit=1001229135&lines=102) before coming back and removing the flag completely.
Reviewed By: fanzeyi
Differential Revision: D17633671
fbshipit-source-id: c640e5803d1c221f6912c3b4f995a2716c5e520b
Summary: The file_size attribute for an inode is not always defined because it can refer to a directory. However, since `LoadedNode`s are always regular files, we can enforce file_size to be always an `int`.
Reviewed By: chadaustin
Differential Revision: D17683020
fbshipit-source-id: b79275c2146e5304bdef02c3228cc0bf35456095
Summary:
it turns out that buck was using this to avoid making a thrift
call to get the sha1 of files.
Let's restore the method but have it return an empty list until buck
has migrated away from that.
asp2insp is looking at fixing up that aspect of buck
Reviewed By: chadaustin
Differential Revision: D17694987
fbshipit-source-id: 9ab0771c00fc64ed3c3395948650189d50d99180
Summary: Keep track of `ResponseChannelRequest` in diff() in order to short circut function if client request is no longer active.
Reviewed By: wez, simpkins
Differential Revision: D17531102
fbshipit-source-id: 662333e5fbc2ed9b3b32b619b8d15db618ad06ec
Summary:
The Python 3 zipapp module always exits with return code 0 if the main
function returns. Therefore we need to add our own wrapper function that
handles calling `sys.exit()` if we want to exit with a non-zero status.
Reviewed By: wez
Differential Revision: D17678021
fbshipit-source-id: 5944c75d8e765ab3e3fa038fa6f2eee95ab0e780
Summary:
The doctor code can't see into the newer redirection
config, which makes it only partially valid. In addition,
we run `eden redirect fixup` post-update; that supersedes
and replaces the doctor functionality.
I've also removed the associated tests: it's a shame that
they can't be re-purposed to work with the redirection code,
but their model is too awkward to use in that way--those tests
fake up aspects of the mount table and mount commands in layers
that are difficult to holistically fake up.
We have integration tests for the redirect functionality, so
we're not leaving things totally uncovered.
Reviewed By: simpkins
Differential Revision: D17252083
fbshipit-source-id: 8a576fb457bf0a4082bde1770085e809484ee73a
Summary:
Force a reference to the edenfsctlPath flag, otherwise the
linker will discard it and a large number of tests will fail.
Reviewed By: simpkins
Differential Revision: D17683222
fbshipit-source-id: b7cb29e74af85b544f45a228770ad2613c8e6efc
Summary:
This diff removes the logic that consumes the legacy bind
mount list and mounts them on startup. That functionality has been
replaced with the eden redirect command.
Instead of performing the bind mounts in the server, the server will
now run `eden redirect fixup` to apply that configuration.
This diff also changes the behavior of performBindMounts: previously, if the
bind mount setup failed, we would tear down the entire repo mount. Since we're
now spawning an external process, it is much more likely that something might
fail and result in a bad experience, so we no longer bail out in that case:
we'll continue and leave the bind mounts as-is. The user can then use `eden
doctor` or `eden redirect fixup` to sort things out.
Reviewed By: simpkins
Differential Revision: D17236366
fbshipit-source-id: 8b004551a076216f0e5448942f00b5195ee18803
Summary:
mostafaeweda and otawfik observed that repeating the same `eden redirect
add` call with the same parameters was tearing down and re-mounting the bind
mount. That shouldn't matter but it isn't as fast as not doing anything, and
sometimes hdiutil on mac can get into a weird state.
This commit compares the existing configuration against the new redirection
entry that we intend to add before applying the config; if they match then
it prints a note about not doing anything and then returns.
Reviewed By: mostafaeweda
Differential Revision: D17639592
fbshipit-source-id: 48d70d11e14a1a191ae655ad7a990c1c90c14953
Summary:
The RocksDbLocalStore.cpp code updates counters, so it depends on the fb303
library.
Reviewed By: fanzeyi
Differential Revision: D17669619
fbshipit-source-id: d05bee82b05743c2ba292808ce3d77812413d2eb
Summary: `folly::ssl::init` will initialize some of the random number generator and callbacks needed for concurrency.
Reviewed By: wez
Differential Revision: D17644835
fbshipit-source-id: 8b0b4ec98b119298053d7362339ebb8bfbd30586
Summary: In the case of a hard reboot, ensure the per-checkout config.toml isn't corrupted.
Reviewed By: wez
Differential Revision: D17672198
fbshipit-source-id: 98141b411e70ed8b30052f989be8161c3ffcc354
Summary:
Update the `Future` based version of the `ThriftLogHelper` class to log how
long the call took, even if it failed.
The code already logged on error for non-Future thrift handler functions, but
for `future_*()` calls we only logged when producing a successful result.
This switches the code from `thenValue()` to `thenTry()` so that we log on
both successful and failed results.
Reviewed By: wez
Differential Revision: D17636471
fbshipit-source-id: 859a326e5bc54183c80f16bf463ba6d451f624e5
Summary:
D17623550 deleted `ServiceCacheIf::invalidateSelection()` but did not remove
this override, which broke the build for EdenFS's tests.
Reviewed By: wez
Differential Revision: D17638351
fbshipit-source-id: 200ead0f79a2252dc7ac2318745cd98e520737a4
Summary:
Update the logging for `getFileInformation()`, `glob()`, `globFiles()`, and
`getSHA1()` to log at most 5 paths, plus the number of remaining paths.
Watchman can sometimes call `getFileInformation()` with tens of thousands of
paths. We don't enable these log messages by default, but if someone has
turned up the log levels to help debug, this can easily fill up the log files
with spam.
Reviewed By: wez
Differential Revision: D17623234
fbshipit-source-id: 3ea8ace294fb8f042f1c53382c76148ba4af95f6
Summary:
hdiutil is horribly verbose, which in turn causes `eden redirect fixup`
to be horribly verbose on macos, so let's supress its output when successful.
Reviewed By: simpkins
Differential Revision: D17254318
fbshipit-source-id: ed2508e554d70bdcde0f0546e24dd6155e05d258
Summary:
The sparse images that we mount in place of bind mounts on macos
can grow over time, and don't automatically shrink if their contents are
purged.
This commit teaches `eden gc` to run `hdiutil compact` on these images
to reclaim space.
The output from `hdiutil compact` is really verbose and noisy, so we
suppress it unless there is an error.
Reviewed By: simpkins
Differential Revision: D17254235
fbshipit-source-id: 944b3824717c8b46d8170ec729c04a93323ab7f0
Summary:
Add a CMake option to control whether or not we should build support for
fb-mercurial (aka Eden SCM). If this is disabled we avoid building anything
under eden/fs/store/hg and drop support for the "hg" backing store.
Reviewed By: wez
Differential Revision: D15980320
fbshipit-source-id: 23a49d3e5cf89199666ff4a0bf46626502c12171
Summary:
Change the `//scm/hg:hg` target to use an `sh_binary()` rule that invokes the
`:hg_rust` binary with the proper environment so it can find its dependencies,
rather than copying the binary and all of its dependencies into a new
subdirectory.
In dev mode builds the `hg_rust` binary isn't guaranteed to work anywhere
other than its original location, due to the way that dev mode builds use
`$ORIGIN` in the binary's `RPATH` setting. This happened to work up until now
as the hg_rust binary did not have any separate libraries, but I plan to add
one on the `chg` library.
Reviewed By: quark-zju
Differential Revision: D17109104
fbshipit-source-id: ae8bb1126969f012d1d2fb7d04e80867a310b9a8
Summary:
We're seeing an issue where for an Eden instance started
as root a subsequent chown doesn't chown the root of the bind mount
and thus makes it impossible to write into the directory.
Reviewed By: simpkins
Differential Revision: D17620135
fbshipit-source-id: 4ee511234c551bbfc7330a60ee85fc5efd329d73
Summary:
It's convenient to manipulate Thrift output with jq sometimes, so add
an option to try to encode the resulting Thrift object as JSON.
Reviewed By: simpkins
Differential Revision: D17468559
fbshipit-source-id: b1f4639b6ea0d366400c8283cd7095bb4e0728de
Summary:
On a machine with one core, it's quite possible for the process to
freeze for a second, resulting in a flaky test.
Reviewed By: wez
Differential Revision: D17579535
fbshipit-source-id: a3e158b5895c06bdf560ebecd849a74e4efa591e
Summary:
Use fdatasync (or fsync on macOS and Windows) and os.replace to write
Eden's config.json.
Reviewed By: wez
Differential Revision: D17465682
fbshipit-source-id: 0e92a58653befc84d45eb815bc0a02b8c2df3d39
Summary: Use callback to save ScmStatus instead of storing status inside a `TreeDiffer` object. Involes a bit of restructuring of some code to avoid circular dependencies and library creation. Mostly renames and file moves, some funtion moves as well.
Reviewed By: simpkins
Differential Revision: D17400466
fbshipit-source-id: fcd194a4c20204dffd3d11cd4a083564dc0ea938
Summary:
Increase the log level if a FUSE request timeout occurs, so we can tell if
this has occurred in the logs when debugging user problems. Rate limit the
messages to at most once per second to avoid lots of log spam when a problem
occurs that causes lots of FUSE requests to time out.
Reviewed By: chadaustin
Differential Revision: D17506837
fbshipit-source-id: b84bd094034f1b31ef74038a9fd3828d4db9acf2
Summary: Use callback to save ScmStatus instead of storing status inside a `TreeDiffer` object.
Reviewed By: simpkins
Differential Revision: D17379318
fbshipit-source-id: b5f56c937509f0861f5dc06c742ebc3e1f85c695
Summary:
Add a CMake option to control whether or not we should build support for
Git. If this is disabled we avoid building anything under eden/fs/store/git
and drop support for the "git" backing store.
Reviewed By: wez
Differential Revision: D15980321
fbshipit-source-id: 434364d81b44935ce86fdf4d66697ee21ff2992f
Summary:
The folly's version of fileutil's throw's "permission denied" error on Windows while creating the config file. Updating the code to call the Win32 version from eden/fs/win/utils.
We do eventually plan to fix Folly by moving the Eden Windows's FileUtils code to it. Until then we will call it from Eden's utils.
Reviewed By: fanzeyi
Differential Revision: D17490623
fbshipit-source-id: 948c488f1dfddd7b12eb8623e056a7680fd5aa73
Summary:
Update the getdeps manifest and Eden's CMake files to indicate that the
edenfsctl program depends on python-toml.
Reviewed By: chadaustin
Differential Revision: D17401215
fbshipit-source-id: f512678d8bca9c7b2b4d25bf9c3ecd7eed825de9
