Summary:
In order to optimize jobs using eden, we want to be able to see which directories we write/read a lot of files to/from. This information allows us to see if we can improve performance by adding certain directories to prefetch, or by symlinking directories that we write to often.
This diff adds to the summary a list of directories with at least 1000 files written to/read from them.
Reviewed By: fanzeyi
Differential Revision: D17747558
fbshipit-source-id: cf59beeee7c2eda9dea0dfa99f7ab92be9b33e1e
Summary:
Since it has become cheap to fetch file sizes, we made a change in D17633671 to ignore the --sizes flag and always fetch sizes.
Now that we have removed the use of --sizes flag on the Sandcastle side in D17729705, we can remove the flag from the cli as well.
Reviewed By: fanzeyi
Differential Revision: D17746946
fbshipit-source-id: 1a943e32db6a6f6754a20d8e8fa3ac25f241f778
Summary:
## Backstory
Pyre was throwing errors in my diff (D17747558) regarding an extraneous fixme. Turns out PyreBot has been adding and removing these fixmes during version updates (see D17135557 and D16183608), so I suspect it's something to do with the Pyre version. Anyways, I figured it'd be easier to do the annotation than to remove the fixme and risk Pyre throwing the same error in a later diff.
## What I did
I added the Key type (Path) and Value type(ExpectedFileBase). mypy then started throwing an error regarding __iter__ returning the wrong type because it wanted an iterator over keys rather than values. Fixed that and added .values() to the for loop.
Reviewed By: genevievehelsel
Differential Revision: D17806135
fbshipit-source-id: c07feee33db78a9bff19ba9856a7047657b8c63e
Summary:
We make use of the KERN_PROCARGS2 MIB data that we can
retrieve via `sysctl`.
If we can't retrieve that data then we fall back to libproc as
we were doing previously. From my testing so far it seems like
the main reason for failure is that the target process is a
protected system process.
Reviewed By: chadaustin
Differential Revision: D17724101
fbshipit-source-id: 8de1a978e6f89612bfe247e0fd540d9078f50746
Summary: This will help us identify the calling process for the FS request. This was helpful while testing watchman crawl in Eden repo.
Reviewed By: chadaustin
Differential Revision: D17719028
fbshipit-source-id: 0d646d7949a6e26268b71287b26d63473a9a87f8
Summary:
We are seeing random segment fault originating from OpenSSL on macOS when
Mononoke fetching is enabled.
The cause is that on macOS we are actually linking against libcurl shipped with
the system instead of ours. That copy of libcurl is linked with macOS's
libcrypto instead of the one we compiles during Eden's build, and it seems that
version of libcrypto does not provide concurrency safety.
The solution is to build curl on macOS and make sure it is linked to our
OpenSSL that has the concurrency callbacks registered.
Reviewed By: wez
Differential Revision: D17657822
fbshipit-source-id: 85abdf3be10b3903a5efc6b3a91624c7258de790
Summary:
Update the CMakeLists.txt files to support building the hg integration tests.
At the moment this only includes one of the test files (`status_test.py`).
I have not verified if tests from the other modules pass yet or if they need
any additional tweaks to work in CMake-based builds.
Reviewed By: pkaush, fanzeyi
Differential Revision: D17678991
fbshipit-source-id: 4a5ee5a8d6039d9d2a635c7027897bbeed14f8c0
Summary:
Add initial support for building and running some of the integration tests
with CMake. For now this just runs the tests from basic_test.py, just to
confirm that most of the framework code works in CMake-based builds.
Many of the other tests should also work as well, but a few of them we may
want to disable for CMake-based builds. e.g., a couple of the tests depend on
hypothesis, and we would need to include hypothesis as a dependency. Some of
the tests that use systemd might also require a little more work to get
working.
Reviewed By: fanzeyi
Differential Revision: D17659026
fbshipit-source-id: 67420fda9e1021a0cddee2d385fd21e34fb2fd70
Summary:
One of our diagnostics attempts to run `nuclide-connections`,
and prints a warning if that tool is not found.
In our macOS environment that tool is never present so it always emits
a warning that is not actionable.
This diff suppresses that particular warning.
Reviewed By: fanzeyi
Differential Revision: D17745753
fbshipit-source-id: 3394e88d443383880b632fca04a1ec074e46a9bd
Summary: Moved the code from vcproj testutils folder to test folder and enabled the Cmake build on it.
Reviewed By: wez
Differential Revision: D17738227
fbshipit-source-id: bcba41cb29b7e24782c36899c0977c5592888ff5
Summary:
We were troubleshooting an issue with the eden tests on windows
where the boost dlls where not being found during gtest discovery.
When we compute the environment, we were only including INST/bin in the
PATH on windows. On Windows, the dlls are searched for in the PATH, and
since boost installs those into its `lib` dir we were missing those.
This diff causes `lib` dirs to get added to PATH on windows in the same
manner that we would add them to `LD_LIBRARY_PATH` on linux.
Reviewed By: pkaush
Differential Revision: D17694542
fbshipit-source-id: 143a907e6d30d8c12360caa43c8d9c26ff8c88c6
Summary:
Make it possible for any user on the system to connect to Eden's unix
domain socket. Our own permission checking is implemented in D17459971.
Reviewed By: wez
Differential Revision: D17460113
fbshipit-source-id: 29eba62010a58837df0376c66e252f53314a25dd
Summary:
To allow other user accounts to access fb303 counters, we plan to open
our unix domain socket to the world. But that requires implementing
our own permission checks, so do that first.
Reviewed By: wez
Differential Revision: D17459971
fbshipit-source-id: 3cc512da90f725d1c93f25f2a7a5ea7c72fb2b96
Summary:
Previously, readdir-triggered prefetches would merely look up and
cache the inodes for entries. Given a lookup also needs to return stat
information, including file sizes, prefetch blob sizes too.
Reviewed By: wez
Differential Revision: D17728752
fbshipit-source-id: 1c555c123fb7405f82885b427b32cd007fdb16ef
Summary:
On linux we use `patchelf` to manipulate dynamic deps but it
isn't guaranteed to be installed everywhere. We have a manifest file
that describes how to build it, but so far nothing has told getdeps
that it should build it.
This diff updates the ELF dep munging code to literally run
`getdeps.py build patchelf` and then use that patchelf binary to
manipulate the object files.
Refs: https://github.com/facebook/watchman/pull/750
Reviewed By: pkaush
Differential Revision: D17705351
Pulled By: wez
fbshipit-source-id: 358ef239edb389fbd51fa023ff553963aa80b6c7
Summary:
Converting the const ptr in EdenDispatcher because we need to call non-const function (like update Journal). Also storing the reference in EdenDispatcher and WinStore instead of ptr.
This diff should not change the behavior of the Eden.
Reviewed By: chadaustin
Differential Revision: D17719606
fbshipit-source-id: 6c371c38045c760837130bb607163c2bc469b17a
Summary: This allows us to test the datapack code easier without rebuilding Eden.
Reviewed By: wez
Differential Revision: D17468473
fbshipit-source-id: a6807b4d6e747ae8557ae51fdf798de2a54fd4f1
Summary:
When Eden is competing with hundreds of build processes using about
the same amount of RAM, it's less painful for the OOM killer to kill
the compilers, not the edenfs daemon. Bias its OOM score a little.
Reviewed By: wez
Differential Revision: D17701071
fbshipit-source-id: 5ee1078e78e5aa6251692944ca12d9a892f86a6f
Summary: Now that Eden has dropped support for flatmanifest, allow GC'ing trees in the local storage.
Reviewed By: simpkins
Differential Revision: D17236345
fbshipit-source-id: d40ec0c2e61c9971ba1d976572d002c48c7bfc20
Summary:
The current debug file_stat tool returns a list of read and written files with their corresponding sizes. This is difficult to read and understand.
We add a summary section with:
- file count
- total bytes read/written
- total large files (> 10MB) read/written
- list of large files
The existing list of read/written files are moved into a `details` section.
Reviewed By: fanzeyi
Differential Revision: D17633963
fbshipit-source-id: a0b14d900e13590214b64fd9bbd7db2a0c69b300
Summary:
D16836053 -> Diff where --sizes arg was added. Back then, sizes were computed locally and added significant overhead, so it made sense to put this feature behind a flag.
D17080186 -> Diff where function was updated to fetch sizes from thrift call instead of computing it locally
Eden CLI seems to be deployed on Sandcastle on a weekly cadence, so I'm holding off on removing the --sizes flag for now. We can wait till this change is deployed, then make the change to the callsite [here](https://our.intern.facebook.com/intern/diffusion/WWW/browse/master/flib/intern/sandcastle/util/SandcastleEdenUtil.php?commit=1001229135&lines=102) before coming back and removing the flag completely.
Reviewed By: fanzeyi
Differential Revision: D17633671
fbshipit-source-id: c640e5803d1c221f6912c3b4f995a2716c5e520b
Summary: The file_size attribute for an inode is not always defined because it can refer to a directory. However, since `LoadedNode`s are always regular files, we can enforce file_size to be always an `int`.
Reviewed By: chadaustin
Differential Revision: D17683020
fbshipit-source-id: b79275c2146e5304bdef02c3228cc0bf35456095
Summary:
it turns out that buck was using this to avoid making a thrift
call to get the sha1 of files.
Let's restore the method but have it return an empty list until buck
has migrated away from that.
asp2insp is looking at fixing up that aspect of buck
Reviewed By: chadaustin
Differential Revision: D17694987
fbshipit-source-id: 9ab0771c00fc64ed3c3395948650189d50d99180
