Summary:
Our CI environment cannot directly connect to the internet,
and even if it could, doing so is undesirable to fetch javascript deps.
We maintain an offline mirror of packages used by the build(s) so that
we don't have to go out to the internet.
When running in fbsource, configure the environment to use that offline
mirror.
Reviewed By: chadaustin
Differential Revision: D18061773
fbshipit-source-id: 1a5e112f23c1baaedfb3dff0c4c2a1641f6bb9a1
Summary: Ask testpilot to include more output in test failures.
Reviewed By: fanzeyi
Differential Revision: D18061772
fbshipit-source-id: 0c14092557c21396c877d3b1776c5707437a117c
Summary:
currently, the implementation of `eden prefetch` calls into
a mercurial function that is overly eager in making network connections,
which results in what should be a fast NOP second prefetch call taking
more time than is desirable.
This diff adds a little cache to avoid repeatedly calling prefetch
for the same directory more than once for the life of the getdeps
process.
Given the usage pattern of getdeps it is OK that we don't provide
a way to invalidate this cache.
Reviewed By: fanzeyi
Differential Revision: D18005408
fbshipit-source-id: 0ec3f477da1043a5a715704b512c81fcfaa0acde
Summary:
This avoids invalidating the entire build in response
to just running `hg amend`, which is frustrating and slow.
Reviewed By: chadaustin
Differential Revision: D18005409
fbshipit-source-id: ef93313859919298be78204046eb08bcadc5398e
Summary:
This diff allows passing a watchman version number override
via the environment as well as via the cmake `WATCHMAN_VERSION_OVERRIDE`
option.
To help invalidate the build I've added a new section to the manifest
files that allows listing out additional env vars that the project
hashes should be sensitive to. The effect of this is that we'll
re-run the cmake configure step if the listed env vars are changed.
Reviewed By: Ben0mega
Differential Revision: D17865896
fbshipit-source-id: 8ea5572b0b9b7af95ec5c310e494cb17a139ced4
Summary: testpilot's defaults assume a bigger machine than some of our laptops
Reviewed By: fanzeyi
Differential Revision: D17878120
fbshipit-source-id: e01f1f9c77a4f5f011051c9c642dbe934c66bc0b
Summary: This is the first step towards removing `watchman/thirdparty/tap.{cpp,h}`
Reviewed By: chadaustin
Differential Revision: D17775680
fbshipit-source-id: d6ac32c3b2489e1713fb132b0bb46d848c56811f
Summary: This helps to squash out some flakiness
Reviewed By: pkaush
Differential Revision: D17804696
fbshipit-source-id: decd8e5dd37d802c62cae1168c4f4d72c0fc5c83
Summary: As it turns out, several of the `fizz` dependencies require it to have been built with tests enabled, so it's just easier to build them always, IIRC they only waste 1-2 minutes of time.
Reviewed By: lnicco
Differential Revision: D17837758
fbshipit-source-id: dd0c73b3aaf72831ce702dbcecd4e3ff627a4901
Summary:
Proxygen no longer uses `fbcode_builder` to run its tests, so whatever the purpose of D17158685, these `fbcode_builder` configs no longer affect Proxygen, and can be reverted to their original state.
Since the general design pattern for `fbcode_builder` has been to link everything as `.so`s, let's return to this (which helps fix Bistro's build).
Also, let's not waste time building & linking tests for libraries that are not the library under test. That is:
- Before: The Bistro build also builds tests for wangle, proxygen, etc. This is a result of some accidental changes in D17158685.
- After: We explicitly don't build test for any of the 4 dependencies here. This is OK because each project also has its own `fbcode_builder_config.py`, which **does** build tests.
This latter part should result in a build-time reduction.
Reviewed By: lnicco
Differential Revision: D17819858
fbshipit-source-id: 7cad1bed86b2f0c3934b0fc5d6fb33e6a2ee2695
Summary:
We are seeing random segment fault originating from OpenSSL on macOS when
Mononoke fetching is enabled.
The cause is that on macOS we are actually linking against libcurl shipped with
the system instead of ours. That copy of libcurl is linked with macOS's
libcrypto instead of the one we compiles during Eden's build, and it seems that
version of libcrypto does not provide concurrency safety.
The solution is to build curl on macOS and make sure it is linked to our
OpenSSL that has the concurrency callbacks registered.
Reviewed By: wez
Differential Revision: D17657822
fbshipit-source-id: 85abdf3be10b3903a5efc6b3a91624c7258de790
Summary:
We were troubleshooting an issue with the eden tests on windows
where the boost dlls where not being found during gtest discovery.
When we compute the environment, we were only including INST/bin in the
PATH on windows. On Windows, the dlls are searched for in the PATH, and
since boost installs those into its `lib` dir we were missing those.
This diff causes `lib` dirs to get added to PATH on windows in the same
manner that we would add them to `LD_LIBRARY_PATH` on linux.
Reviewed By: pkaush
Differential Revision: D17694542
fbshipit-source-id: 143a907e6d30d8c12360caa43c8d9c26ff8c88c6
Summary:
On linux we use `patchelf` to manipulate dynamic deps but it
isn't guaranteed to be installed everywhere. We have a manifest file
that describes how to build it, but so far nothing has told getdeps
that it should build it.
This diff updates the ELF dep munging code to literally run
`getdeps.py build patchelf` and then use that patchelf binary to
manipulate the object files.
Refs: https://github.com/facebook/watchman/pull/750
Reviewed By: pkaush
Differential Revision: D17705351
Pulled By: wez
fbshipit-source-id: 358ef239edb389fbd51fa023ff553963aa80b6c7
Summary:
Add a manifest to download pexpect-4.7.0 from PyPI, as well as its ptyprocess
dependency.
Reviewed By: wez
Differential Revision: D17669618
fbshipit-source-id: 13395ec07f503f39adb3dc5aa8d0c2d8d0f1d927
Summary:
Correctly emit dependency information when one Python package depends on
another.
Reviewed By: wez
Differential Revision: D17669620
fbshipit-source-id: f51c7851470fe50dc0c17263c94c4d858d6e0921
Summary:
Add a function for defining Python unit tests. This creates the test
executable, and also emits logic to perform test discovery for ctest.
Reviewed By: simpkins
Differential Revision: D17610034
fbshipit-source-id: cdf15b0b04acc1d3e906a1e2a95eb327951176ba
Summary:
Export a property that indicates the path to the test executable. This is
useful for callers that want to install the binary or run it from other CMake
rules.
Reviewed By: simpkins
Differential Revision: D17647146
fbshipit-source-id: b32e2694e44a07d7c234e53a7a5c8443cb144487
Summary:
this adds `oss-openr-linux-getdeps` to diffs affecting files under openr. With soma going away and the old fbcode_builder job disabled, this will give us the signal we need to keep the cmake build healthy.
[Some Info on Getdeps](https://our.intern.facebook.com/intern/wiki/Test_your_Open_Source_build_with_getdeps.py/)
Michael, this change may require you to bump up some of the dependent libraries and build them with cmake if not already. The main changes to the cmake script are around using package configs instead of `find_library`
Also, for those with more CMake experience: since there are some big changes in the `CmakeLists`, feel free to pour on more suggestions on how I could make it better and more aligned with other facebook OSS
Reviewed By: saifhhasan
Differential Revision: D16010068
fbshipit-source-id: 66f914f1971f826e0868c4130839380639a7e44b
Summary:
This updates fbcode_builder to try and automatically detect the current
repository's project name by looking for a `.projectid` file in the repository
root. If the project name can be detected:
- The current repository will automatically be used as the source directory
for this project (instead of fetching the sources into the scratch
directory).
- If an explicit project name was not specified on the command line, the
current project will be built by default.
This also changes the repository detection logic to use the current working
directory, rather than the directory where the fbcode_builder code lives.
This will allow this logic to work even if we move fbcode_builder into its own
repository in the future, and have projects depend on it using git submodules.
This does mean that callers need to invoke fbcode_builder.py from inside the
repository.
Reviewed By: wez
Differential Revision: D17088938
fbshipit-source-id: f14d28fdcfaa330ff837ea52b8bdd4e358b81c61
Summary:
Update the generated `run_cmake.py` script to use `subprocess.run()` instead
of `os.execve()`. The `os.execve()` call doesn't really do what we want on
Windows: this causes the script to exit while CMake is still running,
resulting in confusing output. During the build step it also did not work
correctly with `vcvarsall.bat`, and using `subprocess` also solves this.
Reviewed By: wez
Differential Revision: D17493897
fbshipit-source-id: e0477627fc1824b0efcb1fa5a782d207853bcae8
Summary:
Never cache first-party projects that use ShipIt. Previously the code checked
the `shipit_fbcode_builder` property, which controlled whether or not
the `fbcode_builder` sources should be included in the project's ShipIt
mapping. This setting is enabled for most but not all projects that use
ShipIt.
This resulted in projects that use ShipIt but that do not include the fbcode
builder sources being incorrectly cached. This caused getdeps.py to not
run the SimpleShipitTransformerFetcher properly when their sources changed.
Reviewed By: wez
Differential Revision: D17493522
fbshipit-source-id: 57be5ac94ae44f56ccb3ce60ba23fac5d68bce0f
Summary:
Update the getdeps manifest and Eden's CMake files to indicate that the
edenfsctl program depends on python-toml.
Reviewed By: chadaustin
Differential Revision: D17401215
fbshipit-source-id: f512678d8bca9c7b2b4d25bf9c3ecd7eed825de9
Summary:
Throws an exception when:
* The name specified in the manifest mismatches the filename
* Duplicated manifest -- with the sub-directory support it is now able to have multiple manifest files with the same name
Reviewed By: chadaustin
Differential Revision: D17438460
fbshipit-source-id: ac7ad0b701beb15f0e91bb05cd1ec8fe378ad5b6
Summary: Make getdeps to look for subdirectories for manifest files.
Reviewed By: simpkins
Differential Revision: D17222388
fbshipit-source-id: e13503beccd9edf6d80f78fbc3238b2a8d2053dd
Summary:
The libraries in thrift/lib/py support both Python 2 and Python 3, and rely on
the Python six module for some of this compatibility support.
Update the getdeps manifest for fbthrift to indicate this dependency, and
update fbthrift's CMakeLists.txt file to find and reference python-six
properly. This will ensure that the python-six code is built into any python
executable that uses the thrift/lib/py libraries.
Reviewed By: yfeldblum
Differential Revision: D17401218
fbshipit-source-id: 0007dda8974ae9bd87e4d7e256c74908c9a30d8f
Summary:
Add a new builder that can extract Python wheel files, and re-package them
for consumption by our add_fb_python_library() and add_fb_python_executable()
CMake functions. This is useful for dependencies on packages from PyPI.
At the moment this code only handles architecture-independent pure-Python
packages. It shouldn't be too hard to extend this to handle more complex
wheels, but for now I only need to use it for some pure-Python wheels and so I
haven't tested with more complex wheel files.
This also includes two new manifests for python-six and python-toml that take
use this new builder.
Reviewed By: wez
Differential Revision: D17401216
fbshipit-source-id: d6f74565887c3f004e1c06503dc9ec81599dd697
Summary:
Update the code to use `os.replace()` rather than `os.rename()` so that it
won't fail on Windows if the destination path already exists.
Reviewed By: chadaustin
Differential Revision: D17462716
fbshipit-source-id: cbc06319ccb2d73868f80ab1874890ebec5a621b
Summary: This enable test targets to be built and ran
Reviewed By: lnicco
Differential Revision: D17408942
fbshipit-source-id: 144d223bc3830d07a0420e9569d3166a8646cd9a
Summary: changes the way it pulls its dependencies
Reviewed By: wez
Differential Revision: D17363235
fbshipit-source-id: 37e509c7e413f763e3553e9f01ac23951045089c
Summary:
Update the thrift C++ and Python CMake rules to indicate that the output also
depends on the thrift compiler itself.
Previously the C++ rule indicated that the output depended on the thrift
template files, which caught most cases when the thrift compiler was updated,
but wasn't fully correct. The thrift templates were also removed and baked
into the thrift compiler binary in D16356056.
Reviewed By: yfeldblum, chadaustin
Differential Revision: D17401217
fbshipit-source-id: ae5cde7a7e5e07a74406a1b6f4469124187bc12f
Summary:
This is useful when working on changes to some of the builder functions,
to skip ever trying to use cached results.
Reviewed By: chadaustin
Differential Revision: D17401219
fbshipit-source-id: fb7d5ea45618653957b9bd6a62eed91d8334824d
Summary:
This commit teaches fixup-dyn-deps how to generate correct
absolute paths in the context of the ultimate install path (specified
via the `--final-install-prefix` option)
Absolute paths are desirable if you have, for example, an executable
that you wish to install with the setuid bit set.
Reviewed By: simpkins
Differential Revision: D17361491
fbshipit-source-id: 4c4f3f15208266300622f84dc9cd1ba83883dfb7
Summary:
Add a license header to satisfy the open source linter. Use the same
header the other .cmake files have.
Reviewed By: mhlakhani
Differential Revision: D17404782
fbshipit-source-id: 66679d72c9e680f8bb8b27869e981a046b3520cf
Summary:
This commit adds a getdeps command that is able to generate
a workflow file for the GitHub Actions CI environment.
The workflow file could be expressed more simply using the matrix
syntax and with three steps (checkout, build, test), but I chose to
break out the steps for each of the dependencies because the UX
while waiting on the build is much nicer that way: the steps show
during and live log tailing for the section of the build that is
underway. If they were all lumped into a single build step then
the logs from the boost section of the build dominate and make
the github UI work very hard.
Pull Request resolved: https://github.com/facebook/watchman/pull/743
Test Plan:
https://github.com/facebook/watchman/pull/743 successfully
executes the github actions CI flow.
```
$ opensource/fbcode_builder/getdeps.py generate-github-actions --output-file watchman/.github/workflows/main.yml watchman
```
Reviewed By: simpkins
Differential Revision: D17384915
Pulled By: wez
fbshipit-source-id: 9a9e5a3e806c18f6cc38ba1cb7059740cda01ad4
Summary:
GitHub Actions CI `windows-latest` environment has only VS 2019
installed, so we need to expand our logic to be able to locate it.
Note that Boost 1.69 doesn't know how to locate VS 2019 so we are effectively
tied to VS 2017 at the moment; the search order in this diff reflects that.
(That means that we can't target `windows-latest` on GitHub Actions, but that
is really a concern for a later diff in this stack)
Reviewed By: simpkins
Differential Revision: D17385052
fbshipit-source-id: 9bb0612154f42d425a625406488f39bb4ec3d8ae
Summary:
while testing https://github.com/facebook/watchman/pull/743 I
noticed that the cmake builds were picking up the installed mingw GCC
compiler rather than the MSVC compiler. That would be fine except that
boost is built with MSVC and its generated libraries cannot be subsequently
found by a cmake gcc build that uses FindBoost.
This commit forces cmake to pick cl.exe rather than gcc. This is probably
fine to do unconditionally on windows, but since I've only observed this
particular problem with GitHub Actions I'm keeping it constrained to that
environment for now.
Reviewed By: simpkins
Differential Revision: D17385050
fbshipit-source-id: 90bef898b138e5d4bbd28a155ed1bd468acee9de
Summary:
We've been squeaking by with assuming that flex is installed already
on posix systems, but that isn't the case on the github actions default
configuration.
Adjust the bison recipe: on windows it deploys both flex and bison. We use the
same source for both flex and bison but install flex to a separate install
prefix to make it easier to consume the flex dependency distinct from the bison
dependency.
The latest flex release segfaults during compilation on linux unless we
force -DGNU_SOURCE, so the manifest does that on linux.
Reviewed By: simpkins
Differential Revision: D17385051
fbshipit-source-id: 9f31b07849af9de50099d1b20bedba517bbbdf2f
Summary:
while testing https://github.com/facebook/watchman/pull/743 on macOS
I noticed that the libevent build failed to find openssl.
openssl is special on macos because apple do not ship the headers.
We already build and depend on openssl for the folly manifest on
macos, so this is really just adding a missing edge.
Reviewed By: simpkins
Differential Revision: D17385053
fbshipit-source-id: 1b688537fef422d81a959fc5749c871b9e868baa
Summary:
We would like to build a version of proxygen that has minimal
dependencies on dynamic libraries.
Reviewed By: yfeldblum
Differential Revision: D17228181
fbshipit-source-id: cfd61afdfa978c49a536184f426502196241fb8a
Summary:
On Windows we have to explicitly invoke `make_fbpy_archive.py` with python.
Therefore use CMake's built-in `FindPythonInterp` module to find the python
executable and use that when invoking `make_fbpy_archive.py`
This is slightly complicated by the effort required to find python with older
versions of CMake. We ideally still want to support versions of CMake back to
at least 3.8, which means we need to still support finding Python with the
older `FindPythonInterp.cmake` module
Reviewed By: wez
Differential Revision: D17128606
fbshipit-source-id: 3f4beff76848b8a362ebdf21198e7a8bf1b0537f
