Summary:
Next OpenNSA release with increased stack size (needed for FBOSS) is available.
Start linking with it.
Differential Revision: D20062997
fbshipit-source-id: 9938270c322087ac3990861aa1ddd3b9ea1148ac
Summary:
Ideally we'd be using mkscratch, but this still isn't shipped
to our Windows systems.
Pick a path that is more friendly to our corp windows environment by default.
Reviewed By: pkaush
Differential Revision: D20342277
fbshipit-source-id: c85bccee6701adc03b26c92ba217b18bd684257a
Summary:
Replace the methods to get CPU and memory usage statistics:
- For the memory: use `VmRSS` of `/proc/[pid]/status`: http://man7.org/linux/man-pages/man5/proc.5.html
- For the CPU%: calculate the process is occupied how much percentage of the CPU time, use `getrusage()`: http://man7.org/linux/man-pages/man2/getrusage.2.html
- Implemented like the sigar: https://our.intern.facebook.com/intern/diffusion/FBS/browse/master/third-party/sigar/src/sigar.c?commit=4f945812675131ea64cb3d143350b1414f34a351&lines=111-169
- Formula:
- CPU% = `process used time` during the period / `time period` * 100
- `time period` = current query timestamp - last query timestamp
- `process used time` = current `process total time` - last query `process total time`
- `process total time` = CPU time used in user mode + CPU time used in system mode // get from the API `ru_utime` and `ru_stime`
Remove the `fbzmq::ResourceMonitor` and `sigar`:
- Change and rename the UT
- `ResourceMonitorTest.cpp` -> `SystemMetricsTest.cpp`
- `ResourceMonitor` -> `SystemMetricsTest` in `openr/tests/OpenrSystemTest.cpp`
- Remove `ResourceMonitor` code and dependency for `Watchdog` and `ZmqMonitor`
- Remove `sigar` dependency used in building
Reviewed By: saifhhasan
Differential Revision: D20049944
fbshipit-source-id: 00b90c8558dc5f0fb18cc31a09b9666a47b096fe
Summary:
This diff adds fb303 dependency to Open/R. fb303 is FB's open-sourced project
which provides basic infrastructure for thrift service e.g. base functions,
counters supports etc.
Using this in Open/R will allow both internal & external users leverage counters
via thrift APIs
Reviewed By: xiangxu1121
Differential Revision: D20134243
fbshipit-source-id: 23782ec99dc749dd27dc19f2b2b023f1c229608b
Summary:
The SAI diag shell implementation requires Python3.7.6.
Previous diff in the stack added Python3.7.6 manifest.
Make use of it to build diag shell and link with Python lib.
Differential Revision: D20064315
fbshipit-source-id: c855f8a9f554066e8150ce1d867828a84161961b
Summary:
As titled.
FBOSS depends on Python3.7.6, subsequent diff in the stack adds Python3.7.6
depedency for FBOSS.
Differential Revision: D20064316
fbshipit-source-id: 27b328dc25e326f3927ea6cb003fb7bb45732d61
Summary:
We need to add a separate manifest for SAI 1.4 and build that too, but that is
for later diffs, for now, we need 1.5, so change it.
Differential Revision: D20019016
fbshipit-source-id: 8630340ac7c8b7d94e6b768a2efa96266b2cc3e0
Summary:
The Buck build of RocksDB supports LZ4, so if an Eden instance's
object cache ends up compressed with LZ4 compression, the opensource
getdeps build cannot open it. Enable LZ4 in the open source getdeps
build of rocksdb.
Reviewed By: pkaush
Differential Revision: D19961878
fbshipit-source-id: 9a4f37e12bd2b40a0c5dcf2dd1684876886fae26
Summary: There are downstream oss builds showing build failure caused by introduction of metadata. We want to make them proceed to build so we won't get errors in CI.
Reviewed By: iahs
Differential Revision: D19900007
fbshipit-source-id: 4201448f7980b53e407fd2bc1c74ed4ffa8e18c1
Summary:
Take the README.md from
7ead0e29e4/README.md
and apply it on Eden repo.
Re-add the Cargo.toml file that declares Cargo workspace.
Re-add fbcode_builder/getdeps manifest for Mononoke
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/13
Test Plan:
./build/fbcode_builder/getdeps.py build mononoke
./build/fbcode_builder/getdeps.py test mononoke
Reviewed By: ahornby
Differential Revision: D19833059
Pulled By: lukaspiatkowski
fbshipit-source-id: fb37e13306c0b9969a7c4e52b05e1a66a577022f
Summary:
Instead of fetching a tarball for the vendored Rust crates, let's just use
what's present in third-party/rust.
Reviewed By: wez, fanzeyi
Differential Revision: D19770580
fbshipit-source-id: ca6615a5f5c672d9a5527ad47feab6bdd33a39db
Summary: Mononoke is becoming part of Eden repository, remove it's getdeps manifest until we finalize the move.
Reviewed By: krallin
Differential Revision: D19769090
fbshipit-source-id: 9b471686728e9ff28317f9157f5f8a1834c5f5e4
Summary: Remove the shared CheckAtomic cmake module out of the shared dir and back to the projects that need it.
Reviewed By: lukaspiatkowski
Differential Revision: D19553656
fbshipit-source-id: 5e89b5b9448ef6d6c57ef904a652e9f9a1d5dbb3
Summary:
When building with clang, the build fails with:
```
-- Looking for __atomic_is_lock_free in atomic
-- Looking for __atomic_is_lock_free in atomic - not found
CMake Error at cmake/CheckAtomic.cmake:90 (message):
Host compiler appears to require libatomic, but cannot find it.
Call Stack (most recent call first):
CMakeLists.txt:75 (include)
```
And the error is:
```
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: error: conflicting types for '__atomic_is_lock_free'
CHECK_FUNCTION_EXISTS(void);
^
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: note: '__atomic_is_lock_free' is a builtin with type 'int (unsigned long, const volatile void *)'
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:17:25: error: too few arguments to function call, expected 2, have 0
CHECK_FUNCTION_EXISTS();
~~~~~~~~~~~~~~~~~~~~~ ^
```
LLVM's CheckAtomic (https://fburl.com/bk14shjt) uses `__atomic_fetch_add_4` so I'm modifying the configs to use it as well to check for the existence of the library.
Reviewed By: yfeldblum
Differential Revision: D19497168
fbshipit-source-id: 64f77487efd16dba49055f6c4cb1cdd0fc4ae6da
Summary:
With this change the getdeps' CargoBuilder will support depencies between cargo builds.
The way how it works is documented in the code and required few assumptions about how a cargo project has to be defined in order to support this.
This change also adds the "mononoke" manifest and few Cargo.toml files to the mononoke project to prove that this new feature works.
Reviewed By: farnz
Differential Revision: D19468912
fbshipit-source-id: f299733cbbc2ec7bca399c898ec8d710334d0fa9
Summary:
FBOSS needs to use header files from more directories in OpenNSA,
so make those available in include directory.
Differential Revision: D19461874
fbshipit-source-id: 1fe3fbbc39477baecf9cd4f7c7a964be40cbbb12
Summary:
This is an interim solution that unblocks failing FBOSS OSS builds.
Broadcom's OpenNSA is available on github and contains large precompiled
libraries. For large files (> 100Mb), github uses git-lfs.
git clones to OpenNSA started failing yesterday with:
Error downloading object: lib/x86-64/libopennsa.a (7f21d94): Smudge error:
Error downloading lib/x86-64/libopennsa.a
(7f21d941ac32ccae38adb05b3386be739bdccd502a878364d386e7a5cb172f35): batch
response: This repository is over its data quota. Account responsible for LFS
bandwidth should purchase more data packs to restore access.
Github documents (refer:
https://help.github.com/en/github/managing-large-files/about-storage-and-bandwidth-usage)
“When you download a file tracked with Git LFS, the total file size is counted
against the repository owner’s bandwidth limit”. It appears we are hitting that
limit.
Thus, github is not a good choice here. Even if we create a 'github release
tarball', the tarball continues to carry lfs files and thus downloading those
counts towards the quota (which is limited).
Broadcom paid some amount to bump up the quota, but it is matter of time before
we hit it esp given that these builds run on-diff.
A better solution is to download a tarball (that contains real files, not lfs),
and then allow Facebook lfs to cache it.
While github does not support that, it seems bitbucket does.
Thus, we forked OpenNSA from github into a bitbucket repo and this patch
modifies the manifest to download a tarball from bitbucket that will be cached.
Broadcom is working on a cleaner solution but this will keep our FBOSS OSS build
clean in the interim.
Reviewed By: wez
Differential Revision: D19437386
fbshipit-source-id: b1213a186a0af57b9fd71b30e8899e80affd4cc0
Summary:
Broadcom has formally replaced OpenNSL with newer OpenNSA:
https://github.com/Broadcom-Network-Switching-Software/OpenNSA/blob/master/README.md
With D16401189, FBOSS no longer depends on OpenNSL, but uses the new OpenNSA
instead. Remove this manifest.
Reviewed By: wez
Differential Revision: D19413708
fbshipit-source-id: f3662f1101d59a5e0e59cb7238797d652564ff90
Summary:
With the release of OpenNSA (https://github.com/Broadcom-Network-Switching-Software/OpenNSA), we can stop using opennsal altogether!
This diff was created using following steps:
cd fboss/agent
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl_spl_//g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl/bcm/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OPENNSL/BCM/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OpenNSL/Bcm/g' {} \; # for symbols like snmpBcmTransmittedPkts2048to4095Octets
Furthermore:
- In all TARGETS under fboss/agent replace wrapped_opennsl_symbols with wrapped_bcm_symbols.
- hg mv agent/facebook/test/OpenNSLInterface.h agent/facebook/test/BcmInterface.h
- BcmEgress.cpp remove operator== overload for opennsl API altogether, and remove operator== BCM API version from inside unnamed namespace, and put it in fboss namespace.
- BcmCinter.h and BcmCinter.cpp now have some duplicate definitions as opennsl_ got renamed to bcm_. Delete one of the definitions "as appropriate".
- Similarly, remove duplicate definitions in FakeSdk.h and FakeSdk.cpp
- Similarly, remove duplicate definitions in SdkTracer.cpp (remove definitions calling CALL_WRAPPERS_RV_NO_CINTER). And lastly, remove CALL_WRAPPERS_RV_NO_CINTER itself.
- arc lint
- Remove OpennslCompatTests.cpp
- Fix ./agent/hw/bcm/tests/facebook/BcmEgressTest.cpp to have #include
"fboss/agent/facebook/test/MockOpenNSL.h" instead of MockBcm.h
TODO:
- This patch retains the directory structure (facebook/ subdirectory), which will be fixed in subsequent patches.
- Remove opennsl references from TARGETS file, additional cleanup etc.
- Remove all oss directories...
Differential Revision: D16401189
fbshipit-source-id: 607b6c0f97ffcacf3707f6b7a4c9454cc0b24476
Summary:
Broadcom provides this library and they decided to rename it from OpenBCM to
OpenNSA. Thus, rename corresponding fbcode_builder code.
Reviewed By: wez
Differential Revision: D19396687
fbshipit-source-id: 8233dbf4de9342b5a0e54ae275d6c73d43abe6d0
Summary: Enable `INSTALL_PYTHON_LIB=ON` for internal builds of Eden on Windows.
Reviewed By: pkaush
Differential Revision: D19060630
fbshipit-source-id: 40d4f8784958fbeaad4f0d332c55fa0cb810cb5e
Summary:
Don't error out if we can't read the ELF/MACH-O header; just treat
it is not an object.
Reviewed By: chadaustin, simpkins
Differential Revision: D19253434
fbshipit-source-id: c5ecc7f0bc7a20e2611b7e2ff754355155f095da
Summary:
This should fix the Travis CI builds. It adds rust toolchain support inside docker and sets the required THRIFT env variable.
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/3
Reviewed By: krallin
Differential Revision: D18905608
Pulled By: lukaspiatkowski
fbshipit-source-id: 5db1eff6f215a6617d8acaa0c99a62d45225956b
Summary:
Update the getdeps builders to save all command output to a log file in the
build directory. This makes it easier to go back after the fact and look at
the config options selected by the build run.
Reviewed By: wez
Differential Revision: D18777059
fbshipit-source-id: 86d3ffa48ec992fe90c59a3835e18f08310ed88c
Summary: The projects have been moved to a "shed/" subdirectory, so the root Cargo.toml with workspace has to be adjusted to that move.
Reviewed By: farnz
Differential Revision: D18807189
fbshipit-source-id: 0fd66fa7edd38ab4fdf905872f38fac57ae0230e
Summary:
Add a `install_fb_python_executable()` function to `FBPythonBinary.cmake` for
helping to install python executables generated with
`add_fb_python_executable()`. This primarily helps by automatically looking
up the correct output file to install from the generated targets.
Reviewed By: wez
Differential Revision: D18774539
fbshipit-source-id: 4b397580d72ac448f21d1db6d2cdd653cf3635df
Summary:
It's impossible for a consumer of a released version of
rocksdb to do anything about this except not use it, and this particular
version of rocksdb ships with a number of shadow warnings.
Disable warning to error promotion.
See also: https://twitter.com/pcwalton/status/1201679307552083968
Reviewed By: chadaustin
Differential Revision: D18785637
fbshipit-source-id: 1db2b00b3c397d6c0b8f05b9d1c658877685c961
Summary:
OpenBCM libraries are stored with git LFS. As a result, fetcher fetches LFS pointers and not the contents. Use git-lfs to pull the real contents before copying to install dir using NoopBuilder.
In future, if more builders require git-lfs, we would consider installing
git-lfs as part of the sandcastle infra as against repeating similar
logic for each builder that requires git-lfs.
Reviewed By: wez
Differential Revision: D18759806
fbshipit-source-id: f988a0460107bc0685e7aba107daba9ed88f71e7
Summary:
As titled. This is needed by OpenBCM, and in future, possibly by other
manifests as well.
Reviewed By: wez
Differential Revision: D18759807
fbshipit-source-id: d445dfa382cea4bf96443ab9889926a4abbf0757
Summary:
This is an update to the fbcode_builder codebase to allow setting up the python virtualenv with python dependencies installed. I've included wheel and cython (with a pinned version to 0.28.6 which is the only version that works with thriftpy3 at the moment, due to https://github.com/cython/cython/issues/2985) as standard packages since these are required by some of our top-level dependencies (folly and thrift)
As far as I know, there are no other projects that use PYTHON_VENV at the moment except LogDevice so the impact should be minimal.
Reviewed By: lucaspmelo
Differential Revision: D18758383
fbshipit-source-id: 264941311c5e3a19dc4ef2bb78c9a1baa34dfd8c
Summary: The cargo builder will be used to verify if an opensource Rust project passes Cargo build, test and (optionally) documentation build.
Reviewed By: markbt
Differential Revision: D18636934
fbshipit-source-id: e982e6a017eb32913e2994e7457c8add2e9d6b95
Summary:
This diff fixes `CMakeLists.txt` to enable building `openr` tests using CMake:
1. It adds `add_openr_test` CMake function that adds executable target, registers it as test, links it with bunch of libraries like GTest and GMock, etc...
2. There is no `openr/tests/OpenrModuleTestBase.cpp` anywhere in the source tree, so this commit replaces it with `openr/common/Flags.cpp`.
Reviewed By: jstrizich
Differential Revision: D18584028
fbshipit-source-id: 07d854ef98d0d2509889a08ad042a371101a2825
Summary:
Remove the Facebook-specific dependencies from the Eden manifest: these
dependencies are now available directly in the Eden repository.
Reviewed By: chadaustin
Differential Revision: D18588009
fbshipit-source-id: 590c74e38e4e05939e6955839af3ebb959f9251b
Summary:
Remove the standalone fb-mercurial-rust target that was an internal-only
dependency for the Eden build. This build step is now done entirely in the
Eden build.
Reviewed By: fanzeyi
Differential Revision: D18623943
fbshipit-source-id: c62a1155ddd1c0a6b2270c472176ba25194c6145
Summary:
- Added sai to getdeps
- added SAI to FBOSS as a dependency
- Included path of SAI headers to FBOSS's cmake file.
Reviewed By: shri-khare
Differential Revision: D18629127
fbshipit-source-id: 119a6b7f2b64bd84414d9e16a903bc8df48a35e7
Summary:
Include tools/lfs in the shipit path map for now, since it is needed to
download vendored Rust crates on internal builds. Also correctly indicate
that the fb-mercurial* dependencies also apply only to internal builds.
Reviewed By: chadaustin
Differential Revision: D18621421
fbshipit-source-id: 74f00bf318ff9aefdca8696a61a8f33587749e66
Summary:
Automatically detect the `--facebook-internal` flag based on the current
repository project name.
Reviewed By: wez
Differential Revision: D18621358
fbshipit-source-id: f2b3018169b151811eec455863a8bfc17667d4d8
Summary:
On Windows, compile a small C executable to prepend to the zip file, to allow
the resulting executable files to be run directly on Windows, without needing
to explicitly invoke the Python interpreter to run the output file.
Reviewed By: wez
Differential Revision: D17733616
fbshipit-source-id: 989a93851412d0bbe1e7857aa9111db082f67a4c
Summary:
Now that fmt is available in Folly builds (D14813810), use it to reduce binary code size in Folly Logger. This is done by moving most of the formatting logic behind the type-erased `vformat` API. Previously it was instantiated for all combinations of formatting argument types used in calls to `FB_LOGF` and `XLOGF` in a program.
The effect of this change can be illustrated by looking at symbol sizes as given by `nm -S -td` for the following test function:
```
void test_log() {
FB_LOGF(logger, WARN, "num events: {:06d}, duration: {:6.3f}", 1234, 5.6789);
}
```
compiled in `opt` mode.
`nm` before:
```
0000000004236736 0000000000000231 T test_log()
0000000004236992 0000000000001002 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
```
`nm` after:
```
0000000004237536 0000000000000231 T test_log()
0000000004237792 0000000000000251 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
0000000004238048 0000000000000740 W folly::LogStreamProcessor::vformatLogString[abi:cxx11](folly::Range<char const*>, fmt::v5::format_args, bool&)
```
Before we had one 1002 byte instantiation of `formatLogString<int, double>`. With this change it was reduced 4x to 251 bytes and non-template function `vformatLogString` was added which is shared among all logging calls. The size of `test_log` remained unchanged. There are even bigger savings from Folly Formatter instantiations which are no longer needed, e.g.
```
0000000004238032 0000000000001363 W _ZNK5folly13BaseFormatterINS_9FormatterILb0EJRKiRKdEEELb0EJS3_S5_EEclIZNKS7_8appendToINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr12IsSomeStringIT_EE5valueEvE4typeERSH_EUlNS_5RangeIPKcEEE_EEvSK_
```
So in total this change results in ~5x per-call/instantiation binary size. It is possible to reduce binary size even further but it is not done in the current diff to keep it manageable.
In addition to binary size improvements, switching to fmt will potentially
* allow catching errors in format strings at compile time,
* simplify future migration to C++20 [`std::format`](http://eel.is/c++draft/format).
Reviewed By: simpkins
Differential Revision: D15485589
fbshipit-source-id: 06db4436839f11c2c3dbed7b36658e2193343411
Summary:
When we detect that we are building out of the fbsource
monorepo internal to FB, pre-set some environment variables to help
projects detect our internally pinned versions of node and yarn.
Previously, the detection logic was deciding that the `yarn` shell
script was the executable to invoke, but since that is a bash script
it would fail to spawn.
Reviewed By: simpkins
Differential Revision: D18523871
fbshipit-source-id: b932d2b0ccd7b79d6f9cd74d363bc426c288e38f
Summary: We removed this dependency in the code a while ago.
Reviewed By: saifhhasan
Differential Revision: D18366401
fbshipit-source-id: daf1c7865fe0c312694bdf395f9d7ace99f70395
Summary:
Our CI environment cannot directly connect to the internet,
and even if it could, doing so is undesirable to fetch javascript deps.
We maintain an offline mirror of packages used by the build(s) so that
we don't have to go out to the internet.
When running in fbsource, configure the environment to use that offline
mirror.
Reviewed By: chadaustin
Differential Revision: D18061773
fbshipit-source-id: 1a5e112f23c1baaedfb3dff0c4c2a1641f6bb9a1
Summary: Ask testpilot to include more output in test failures.
Reviewed By: fanzeyi
Differential Revision: D18061772
fbshipit-source-id: 0c14092557c21396c877d3b1776c5707437a117c
Summary:
currently, the implementation of `eden prefetch` calls into
a mercurial function that is overly eager in making network connections,
which results in what should be a fast NOP second prefetch call taking
more time than is desirable.
This diff adds a little cache to avoid repeatedly calling prefetch
for the same directory more than once for the life of the getdeps
process.
Given the usage pattern of getdeps it is OK that we don't provide
a way to invalidate this cache.
Reviewed By: fanzeyi
Differential Revision: D18005408
fbshipit-source-id: 0ec3f477da1043a5a715704b512c81fcfaa0acde
Summary:
This avoids invalidating the entire build in response
to just running `hg amend`, which is frustrating and slow.
Reviewed By: chadaustin
Differential Revision: D18005409
fbshipit-source-id: ef93313859919298be78204046eb08bcadc5398e
Summary:
This diff allows passing a watchman version number override
via the environment as well as via the cmake `WATCHMAN_VERSION_OVERRIDE`
option.
To help invalidate the build I've added a new section to the manifest
files that allows listing out additional env vars that the project
hashes should be sensitive to. The effect of this is that we'll
re-run the cmake configure step if the listed env vars are changed.
Reviewed By: Ben0mega
Differential Revision: D17865896
fbshipit-source-id: 8ea5572b0b9b7af95ec5c310e494cb17a139ced4
Summary: testpilot's defaults assume a bigger machine than some of our laptops
Reviewed By: fanzeyi
Differential Revision: D17878120
fbshipit-source-id: e01f1f9c77a4f5f011051c9c642dbe934c66bc0b
Summary: This is the first step towards removing `watchman/thirdparty/tap.{cpp,h}`
Reviewed By: chadaustin
Differential Revision: D17775680
fbshipit-source-id: d6ac32c3b2489e1713fb132b0bb46d848c56811f
Summary: This helps to squash out some flakiness
Reviewed By: pkaush
Differential Revision: D17804696
fbshipit-source-id: decd8e5dd37d802c62cae1168c4f4d72c0fc5c83
Summary: As it turns out, several of the `fizz` dependencies require it to have been built with tests enabled, so it's just easier to build them always, IIRC they only waste 1-2 minutes of time.
Reviewed By: lnicco
Differential Revision: D17837758
fbshipit-source-id: dd0c73b3aaf72831ce702dbcecd4e3ff627a4901
Summary:
Proxygen no longer uses `fbcode_builder` to run its tests, so whatever the purpose of D17158685, these `fbcode_builder` configs no longer affect Proxygen, and can be reverted to their original state.
Since the general design pattern for `fbcode_builder` has been to link everything as `.so`s, let's return to this (which helps fix Bistro's build).
Also, let's not waste time building & linking tests for libraries that are not the library under test. That is:
- Before: The Bistro build also builds tests for wangle, proxygen, etc. This is a result of some accidental changes in D17158685.
- After: We explicitly don't build test for any of the 4 dependencies here. This is OK because each project also has its own `fbcode_builder_config.py`, which **does** build tests.
This latter part should result in a build-time reduction.
Reviewed By: lnicco
Differential Revision: D17819858
fbshipit-source-id: 7cad1bed86b2f0c3934b0fc5d6fb33e6a2ee2695
Summary:
We are seeing random segment fault originating from OpenSSL on macOS when
Mononoke fetching is enabled.
The cause is that on macOS we are actually linking against libcurl shipped with
the system instead of ours. That copy of libcurl is linked with macOS's
libcrypto instead of the one we compiles during Eden's build, and it seems that
version of libcrypto does not provide concurrency safety.
The solution is to build curl on macOS and make sure it is linked to our
OpenSSL that has the concurrency callbacks registered.
Reviewed By: wez
Differential Revision: D17657822
fbshipit-source-id: 85abdf3be10b3903a5efc6b3a91624c7258de790
Summary:
We were troubleshooting an issue with the eden tests on windows
where the boost dlls where not being found during gtest discovery.
When we compute the environment, we were only including INST/bin in the
PATH on windows. On Windows, the dlls are searched for in the PATH, and
since boost installs those into its `lib` dir we were missing those.
This diff causes `lib` dirs to get added to PATH on windows in the same
manner that we would add them to `LD_LIBRARY_PATH` on linux.
Reviewed By: pkaush
Differential Revision: D17694542
fbshipit-source-id: 143a907e6d30d8c12360caa43c8d9c26ff8c88c6
Summary:
On linux we use `patchelf` to manipulate dynamic deps but it
isn't guaranteed to be installed everywhere. We have a manifest file
that describes how to build it, but so far nothing has told getdeps
that it should build it.
This diff updates the ELF dep munging code to literally run
`getdeps.py build patchelf` and then use that patchelf binary to
manipulate the object files.
Refs: https://github.com/facebook/watchman/pull/750
Reviewed By: pkaush
Differential Revision: D17705351
Pulled By: wez
fbshipit-source-id: 358ef239edb389fbd51fa023ff553963aa80b6c7
Summary:
Add a manifest to download pexpect-4.7.0 from PyPI, as well as its ptyprocess
dependency.
Reviewed By: wez
Differential Revision: D17669618
fbshipit-source-id: 13395ec07f503f39adb3dc5aa8d0c2d8d0f1d927
Summary:
Correctly emit dependency information when one Python package depends on
another.
Reviewed By: wez
Differential Revision: D17669620
fbshipit-source-id: f51c7851470fe50dc0c17263c94c4d858d6e0921
Summary:
Add a function for defining Python unit tests. This creates the test
executable, and also emits logic to perform test discovery for ctest.
Reviewed By: simpkins
Differential Revision: D17610034
fbshipit-source-id: cdf15b0b04acc1d3e906a1e2a95eb327951176ba
Summary:
Export a property that indicates the path to the test executable. This is
useful for callers that want to install the binary or run it from other CMake
rules.
Reviewed By: simpkins
Differential Revision: D17647146
fbshipit-source-id: b32e2694e44a07d7c234e53a7a5c8443cb144487
Summary:
this adds `oss-openr-linux-getdeps` to diffs affecting files under openr. With soma going away and the old fbcode_builder job disabled, this will give us the signal we need to keep the cmake build healthy.
[Some Info on Getdeps](https://our.intern.facebook.com/intern/wiki/Test_your_Open_Source_build_with_getdeps.py/)
Michael, this change may require you to bump up some of the dependent libraries and build them with cmake if not already. The main changes to the cmake script are around using package configs instead of `find_library`
Also, for those with more CMake experience: since there are some big changes in the `CmakeLists`, feel free to pour on more suggestions on how I could make it better and more aligned with other facebook OSS
Reviewed By: saifhhasan
Differential Revision: D16010068
fbshipit-source-id: 66f914f1971f826e0868c4130839380639a7e44b
Summary:
This updates fbcode_builder to try and automatically detect the current
repository's project name by looking for a `.projectid` file in the repository
root. If the project name can be detected:
- The current repository will automatically be used as the source directory
for this project (instead of fetching the sources into the scratch
directory).
- If an explicit project name was not specified on the command line, the
current project will be built by default.
This also changes the repository detection logic to use the current working
directory, rather than the directory where the fbcode_builder code lives.
This will allow this logic to work even if we move fbcode_builder into its own
repository in the future, and have projects depend on it using git submodules.
This does mean that callers need to invoke fbcode_builder.py from inside the
repository.
Reviewed By: wez
Differential Revision: D17088938
fbshipit-source-id: f14d28fdcfaa330ff837ea52b8bdd4e358b81c61
Summary:
Update the generated `run_cmake.py` script to use `subprocess.run()` instead
of `os.execve()`. The `os.execve()` call doesn't really do what we want on
Windows: this causes the script to exit while CMake is still running,
resulting in confusing output. During the build step it also did not work
correctly with `vcvarsall.bat`, and using `subprocess` also solves this.
Reviewed By: wez
Differential Revision: D17493897
fbshipit-source-id: e0477627fc1824b0efcb1fa5a782d207853bcae8
Summary:
Never cache first-party projects that use ShipIt. Previously the code checked
the `shipit_fbcode_builder` property, which controlled whether or not
the `fbcode_builder` sources should be included in the project's ShipIt
mapping. This setting is enabled for most but not all projects that use
ShipIt.
This resulted in projects that use ShipIt but that do not include the fbcode
builder sources being incorrectly cached. This caused getdeps.py to not
run the SimpleShipitTransformerFetcher properly when their sources changed.
Reviewed By: wez
Differential Revision: D17493522
fbshipit-source-id: 57be5ac94ae44f56ccb3ce60ba23fac5d68bce0f
Summary:
Update the getdeps manifest and Eden's CMake files to indicate that the
edenfsctl program depends on python-toml.
Reviewed By: chadaustin
Differential Revision: D17401215
fbshipit-source-id: f512678d8bca9c7b2b4d25bf9c3ecd7eed825de9
Summary:
Throws an exception when:
* The name specified in the manifest mismatches the filename
* Duplicated manifest -- with the sub-directory support it is now able to have multiple manifest files with the same name
Reviewed By: chadaustin
Differential Revision: D17438460
fbshipit-source-id: ac7ad0b701beb15f0e91bb05cd1ec8fe378ad5b6
Summary: Make getdeps to look for subdirectories for manifest files.
Reviewed By: simpkins
Differential Revision: D17222388
fbshipit-source-id: e13503beccd9edf6d80f78fbc3238b2a8d2053dd
Summary:
The libraries in thrift/lib/py support both Python 2 and Python 3, and rely on
the Python six module for some of this compatibility support.
Update the getdeps manifest for fbthrift to indicate this dependency, and
update fbthrift's CMakeLists.txt file to find and reference python-six
properly. This will ensure that the python-six code is built into any python
executable that uses the thrift/lib/py libraries.
Reviewed By: yfeldblum
Differential Revision: D17401218
fbshipit-source-id: 0007dda8974ae9bd87e4d7e256c74908c9a30d8f
Summary:
Add a new builder that can extract Python wheel files, and re-package them
for consumption by our add_fb_python_library() and add_fb_python_executable()
CMake functions. This is useful for dependencies on packages from PyPI.
At the moment this code only handles architecture-independent pure-Python
packages. It shouldn't be too hard to extend this to handle more complex
wheels, but for now I only need to use it for some pure-Python wheels and so I
haven't tested with more complex wheel files.
This also includes two new manifests for python-six and python-toml that take
use this new builder.
Reviewed By: wez
Differential Revision: D17401216
fbshipit-source-id: d6f74565887c3f004e1c06503dc9ec81599dd697
Summary:
Update the code to use `os.replace()` rather than `os.rename()` so that it
won't fail on Windows if the destination path already exists.
Reviewed By: chadaustin
Differential Revision: D17462716
fbshipit-source-id: cbc06319ccb2d73868f80ab1874890ebec5a621b
Summary: This enable test targets to be built and ran
Reviewed By: lnicco
Differential Revision: D17408942
fbshipit-source-id: 144d223bc3830d07a0420e9569d3166a8646cd9a
Summary: changes the way it pulls its dependencies
Reviewed By: wez
Differential Revision: D17363235
fbshipit-source-id: 37e509c7e413f763e3553e9f01ac23951045089c
Summary:
Update the thrift C++ and Python CMake rules to indicate that the output also
depends on the thrift compiler itself.
Previously the C++ rule indicated that the output depended on the thrift
template files, which caught most cases when the thrift compiler was updated,
but wasn't fully correct. The thrift templates were also removed and baked
into the thrift compiler binary in D16356056.
Reviewed By: yfeldblum, chadaustin
Differential Revision: D17401217
fbshipit-source-id: ae5cde7a7e5e07a74406a1b6f4469124187bc12f
Summary:
This is useful when working on changes to some of the builder functions,
to skip ever trying to use cached results.
Reviewed By: chadaustin
Differential Revision: D17401219
fbshipit-source-id: fb7d5ea45618653957b9bd6a62eed91d8334824d
Summary:
This commit teaches fixup-dyn-deps how to generate correct
absolute paths in the context of the ultimate install path (specified
via the `--final-install-prefix` option)
Absolute paths are desirable if you have, for example, an executable
that you wish to install with the setuid bit set.
Reviewed By: simpkins
Differential Revision: D17361491
fbshipit-source-id: 4c4f3f15208266300622f84dc9cd1ba83883dfb7
Summary:
Add a license header to satisfy the open source linter. Use the same
header the other .cmake files have.
Reviewed By: mhlakhani
Differential Revision: D17404782
fbshipit-source-id: 66679d72c9e680f8bb8b27869e981a046b3520cf
Summary:
This commit adds a getdeps command that is able to generate
a workflow file for the GitHub Actions CI environment.
The workflow file could be expressed more simply using the matrix
syntax and with three steps (checkout, build, test), but I chose to
break out the steps for each of the dependencies because the UX
while waiting on the build is much nicer that way: the steps show
during and live log tailing for the section of the build that is
underway. If they were all lumped into a single build step then
the logs from the boost section of the build dominate and make
the github UI work very hard.
Pull Request resolved: https://github.com/facebook/watchman/pull/743
Test Plan:
https://github.com/facebook/watchman/pull/743 successfully
executes the github actions CI flow.
```
$ opensource/fbcode_builder/getdeps.py generate-github-actions --output-file watchman/.github/workflows/main.yml watchman
```
Reviewed By: simpkins
Differential Revision: D17384915
Pulled By: wez
fbshipit-source-id: 9a9e5a3e806c18f6cc38ba1cb7059740cda01ad4
Summary:
GitHub Actions CI `windows-latest` environment has only VS 2019
installed, so we need to expand our logic to be able to locate it.
Note that Boost 1.69 doesn't know how to locate VS 2019 so we are effectively
tied to VS 2017 at the moment; the search order in this diff reflects that.
(That means that we can't target `windows-latest` on GitHub Actions, but that
is really a concern for a later diff in this stack)
Reviewed By: simpkins
Differential Revision: D17385052
fbshipit-source-id: 9bb0612154f42d425a625406488f39bb4ec3d8ae
Summary:
while testing https://github.com/facebook/watchman/pull/743 I
noticed that the cmake builds were picking up the installed mingw GCC
compiler rather than the MSVC compiler. That would be fine except that
boost is built with MSVC and its generated libraries cannot be subsequently
found by a cmake gcc build that uses FindBoost.
This commit forces cmake to pick cl.exe rather than gcc. This is probably
fine to do unconditionally on windows, but since I've only observed this
particular problem with GitHub Actions I'm keeping it constrained to that
environment for now.
Reviewed By: simpkins
Differential Revision: D17385050
fbshipit-source-id: 90bef898b138e5d4bbd28a155ed1bd468acee9de
Summary:
We've been squeaking by with assuming that flex is installed already
on posix systems, but that isn't the case on the github actions default
configuration.
Adjust the bison recipe: on windows it deploys both flex and bison. We use the
same source for both flex and bison but install flex to a separate install
prefix to make it easier to consume the flex dependency distinct from the bison
dependency.
The latest flex release segfaults during compilation on linux unless we
force -DGNU_SOURCE, so the manifest does that on linux.
Reviewed By: simpkins
Differential Revision: D17385051
fbshipit-source-id: 9f31b07849af9de50099d1b20bedba517bbbdf2f
Summary:
while testing https://github.com/facebook/watchman/pull/743 on macOS
I noticed that the libevent build failed to find openssl.
openssl is special on macos because apple do not ship the headers.
We already build and depend on openssl for the folly manifest on
macos, so this is really just adding a missing edge.
Reviewed By: simpkins
Differential Revision: D17385053
fbshipit-source-id: 1b688537fef422d81a959fc5749c871b9e868baa
Summary:
We would like to build a version of proxygen that has minimal
dependencies on dynamic libraries.
Reviewed By: yfeldblum
Differential Revision: D17228181
fbshipit-source-id: cfd61afdfa978c49a536184f426502196241fb8a
Summary:
On Windows we have to explicitly invoke `make_fbpy_archive.py` with python.
Therefore use CMake's built-in `FindPythonInterp` module to find the python
executable and use that when invoking `make_fbpy_archive.py`
This is slightly complicated by the effort required to find python with older
versions of CMake. We ideally still want to support versions of CMake back to
at least 3.8, which means we need to still support finding Python with the
older `FindPythonInterp.cmake` module
Reviewed By: wez
Differential Revision: D17128606
fbshipit-source-id: 3f4beff76848b8a362ebdf21198e7a8bf1b0537f
