Summary:
When building with clang, the build fails with:
```
-- Looking for __atomic_is_lock_free in atomic
-- Looking for __atomic_is_lock_free in atomic - not found
CMake Error at cmake/CheckAtomic.cmake:90 (message):
Host compiler appears to require libatomic, but cannot find it.
Call Stack (most recent call first):
CMakeLists.txt:75 (include)
```
And the error is:
```
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: error: conflicting types for '__atomic_is_lock_free'
CHECK_FUNCTION_EXISTS(void);
^
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: note: '__atomic_is_lock_free' is a builtin with type 'int (unsigned long, const volatile void *)'
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:17:25: error: too few arguments to function call, expected 2, have 0
CHECK_FUNCTION_EXISTS();
~~~~~~~~~~~~~~~~~~~~~ ^
```
LLVM's CheckAtomic (https://fburl.com/bk14shjt) uses `__atomic_fetch_add_4` so I'm modifying the configs to use it as well to check for the existence of the library.
Reviewed By: yfeldblum
Differential Revision: D19497168
fbshipit-source-id: 64f77487efd16dba49055f6c4cb1cdd0fc4ae6da
Summary:
With this change the getdeps' CargoBuilder will support depencies between cargo builds.
The way how it works is documented in the code and required few assumptions about how a cargo project has to be defined in order to support this.
This change also adds the "mononoke" manifest and few Cargo.toml files to the mononoke project to prove that this new feature works.
Reviewed By: farnz
Differential Revision: D19468912
fbshipit-source-id: f299733cbbc2ec7bca399c898ec8d710334d0fa9
Summary:
FBOSS needs to use header files from more directories in OpenNSA,
so make those available in include directory.
Differential Revision: D19461874
fbshipit-source-id: 1fe3fbbc39477baecf9cd4f7c7a964be40cbbb12
Summary:
This is an interim solution that unblocks failing FBOSS OSS builds.
Broadcom's OpenNSA is available on github and contains large precompiled
libraries. For large files (> 100Mb), github uses git-lfs.
git clones to OpenNSA started failing yesterday with:
Error downloading object: lib/x86-64/libopennsa.a (7f21d94): Smudge error:
Error downloading lib/x86-64/libopennsa.a
(7f21d941ac32ccae38adb05b3386be739bdccd502a878364d386e7a5cb172f35): batch
response: This repository is over its data quota. Account responsible for LFS
bandwidth should purchase more data packs to restore access.
Github documents (refer:
https://help.github.com/en/github/managing-large-files/about-storage-and-bandwidth-usage)
“When you download a file tracked with Git LFS, the total file size is counted
against the repository owner’s bandwidth limit”. It appears we are hitting that
limit.
Thus, github is not a good choice here. Even if we create a 'github release
tarball', the tarball continues to carry lfs files and thus downloading those
counts towards the quota (which is limited).
Broadcom paid some amount to bump up the quota, but it is matter of time before
we hit it esp given that these builds run on-diff.
A better solution is to download a tarball (that contains real files, not lfs),
and then allow Facebook lfs to cache it.
While github does not support that, it seems bitbucket does.
Thus, we forked OpenNSA from github into a bitbucket repo and this patch
modifies the manifest to download a tarball from bitbucket that will be cached.
Broadcom is working on a cleaner solution but this will keep our FBOSS OSS build
clean in the interim.
Reviewed By: wez
Differential Revision: D19437386
fbshipit-source-id: b1213a186a0af57b9fd71b30e8899e80affd4cc0
Summary:
Broadcom has formally replaced OpenNSL with newer OpenNSA:
https://github.com/Broadcom-Network-Switching-Software/OpenNSA/blob/master/README.md
With D16401189, FBOSS no longer depends on OpenNSL, but uses the new OpenNSA
instead. Remove this manifest.
Reviewed By: wez
Differential Revision: D19413708
fbshipit-source-id: f3662f1101d59a5e0e59cb7238797d652564ff90
Summary:
With the release of OpenNSA (https://github.com/Broadcom-Network-Switching-Software/OpenNSA), we can stop using opennsal altogether!
This diff was created using following steps:
cd fboss/agent
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl_spl_//g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl/bcm/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OPENNSL/BCM/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OpenNSL/Bcm/g' {} \; # for symbols like snmpBcmTransmittedPkts2048to4095Octets
Furthermore:
- In all TARGETS under fboss/agent replace wrapped_opennsl_symbols with wrapped_bcm_symbols.
- hg mv agent/facebook/test/OpenNSLInterface.h agent/facebook/test/BcmInterface.h
- BcmEgress.cpp remove operator== overload for opennsl API altogether, and remove operator== BCM API version from inside unnamed namespace, and put it in fboss namespace.
- BcmCinter.h and BcmCinter.cpp now have some duplicate definitions as opennsl_ got renamed to bcm_. Delete one of the definitions "as appropriate".
- Similarly, remove duplicate definitions in FakeSdk.h and FakeSdk.cpp
- Similarly, remove duplicate definitions in SdkTracer.cpp (remove definitions calling CALL_WRAPPERS_RV_NO_CINTER). And lastly, remove CALL_WRAPPERS_RV_NO_CINTER itself.
- arc lint
- Remove OpennslCompatTests.cpp
- Fix ./agent/hw/bcm/tests/facebook/BcmEgressTest.cpp to have #include
"fboss/agent/facebook/test/MockOpenNSL.h" instead of MockBcm.h
TODO:
- This patch retains the directory structure (facebook/ subdirectory), which will be fixed in subsequent patches.
- Remove opennsl references from TARGETS file, additional cleanup etc.
- Remove all oss directories...
Differential Revision: D16401189
fbshipit-source-id: 607b6c0f97ffcacf3707f6b7a4c9454cc0b24476
Summary:
Broadcom provides this library and they decided to rename it from OpenBCM to
OpenNSA. Thus, rename corresponding fbcode_builder code.
Reviewed By: wez
Differential Revision: D19396687
fbshipit-source-id: 8233dbf4de9342b5a0e54ae275d6c73d43abe6d0
Summary: Enable `INSTALL_PYTHON_LIB=ON` for internal builds of Eden on Windows.
Reviewed By: pkaush
Differential Revision: D19060630
fbshipit-source-id: 40d4f8784958fbeaad4f0d332c55fa0cb810cb5e
Summary:
Don't error out if we can't read the ELF/MACH-O header; just treat
it is not an object.
Reviewed By: chadaustin, simpkins
Differential Revision: D19253434
fbshipit-source-id: c5ecc7f0bc7a20e2611b7e2ff754355155f095da
Summary:
This should fix the Travis CI builds. It adds rust toolchain support inside docker and sets the required THRIFT env variable.
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/3
Reviewed By: krallin
Differential Revision: D18905608
Pulled By: lukaspiatkowski
fbshipit-source-id: 5db1eff6f215a6617d8acaa0c99a62d45225956b
Summary:
Update the getdeps builders to save all command output to a log file in the
build directory. This makes it easier to go back after the fact and look at
the config options selected by the build run.
Reviewed By: wez
Differential Revision: D18777059
fbshipit-source-id: 86d3ffa48ec992fe90c59a3835e18f08310ed88c
Summary: The projects have been moved to a "shed/" subdirectory, so the root Cargo.toml with workspace has to be adjusted to that move.
Reviewed By: farnz
Differential Revision: D18807189
fbshipit-source-id: 0fd66fa7edd38ab4fdf905872f38fac57ae0230e
Summary:
Add a `install_fb_python_executable()` function to `FBPythonBinary.cmake` for
helping to install python executables generated with
`add_fb_python_executable()`. This primarily helps by automatically looking
up the correct output file to install from the generated targets.
Reviewed By: wez
Differential Revision: D18774539
fbshipit-source-id: 4b397580d72ac448f21d1db6d2cdd653cf3635df
Summary:
It's impossible for a consumer of a released version of
rocksdb to do anything about this except not use it, and this particular
version of rocksdb ships with a number of shadow warnings.
Disable warning to error promotion.
See also: https://twitter.com/pcwalton/status/1201679307552083968
Reviewed By: chadaustin
Differential Revision: D18785637
fbshipit-source-id: 1db2b00b3c397d6c0b8f05b9d1c658877685c961
Summary:
OpenBCM libraries are stored with git LFS. As a result, fetcher fetches LFS pointers and not the contents. Use git-lfs to pull the real contents before copying to install dir using NoopBuilder.
In future, if more builders require git-lfs, we would consider installing
git-lfs as part of the sandcastle infra as against repeating similar
logic for each builder that requires git-lfs.
Reviewed By: wez
Differential Revision: D18759806
fbshipit-source-id: f988a0460107bc0685e7aba107daba9ed88f71e7
Summary:
As titled. This is needed by OpenBCM, and in future, possibly by other
manifests as well.
Reviewed By: wez
Differential Revision: D18759807
fbshipit-source-id: d445dfa382cea4bf96443ab9889926a4abbf0757
Summary:
This is an update to the fbcode_builder codebase to allow setting up the python virtualenv with python dependencies installed. I've included wheel and cython (with a pinned version to 0.28.6 which is the only version that works with thriftpy3 at the moment, due to https://github.com/cython/cython/issues/2985) as standard packages since these are required by some of our top-level dependencies (folly and thrift)
As far as I know, there are no other projects that use PYTHON_VENV at the moment except LogDevice so the impact should be minimal.
Reviewed By: lucaspmelo
Differential Revision: D18758383
fbshipit-source-id: 264941311c5e3a19dc4ef2bb78c9a1baa34dfd8c
Summary: The cargo builder will be used to verify if an opensource Rust project passes Cargo build, test and (optionally) documentation build.
Reviewed By: markbt
Differential Revision: D18636934
fbshipit-source-id: e982e6a017eb32913e2994e7457c8add2e9d6b95
Summary:
This diff fixes `CMakeLists.txt` to enable building `openr` tests using CMake:
1. It adds `add_openr_test` CMake function that adds executable target, registers it as test, links it with bunch of libraries like GTest and GMock, etc...
2. There is no `openr/tests/OpenrModuleTestBase.cpp` anywhere in the source tree, so this commit replaces it with `openr/common/Flags.cpp`.
Reviewed By: jstrizich
Differential Revision: D18584028
fbshipit-source-id: 07d854ef98d0d2509889a08ad042a371101a2825
Summary:
Remove the Facebook-specific dependencies from the Eden manifest: these
dependencies are now available directly in the Eden repository.
Reviewed By: chadaustin
Differential Revision: D18588009
fbshipit-source-id: 590c74e38e4e05939e6955839af3ebb959f9251b
Summary:
Remove the standalone fb-mercurial-rust target that was an internal-only
dependency for the Eden build. This build step is now done entirely in the
Eden build.
Reviewed By: fanzeyi
Differential Revision: D18623943
fbshipit-source-id: c62a1155ddd1c0a6b2270c472176ba25194c6145
Summary:
- Added sai to getdeps
- added SAI to FBOSS as a dependency
- Included path of SAI headers to FBOSS's cmake file.
Reviewed By: shri-khare
Differential Revision: D18629127
fbshipit-source-id: 119a6b7f2b64bd84414d9e16a903bc8df48a35e7
Summary:
Include tools/lfs in the shipit path map for now, since it is needed to
download vendored Rust crates on internal builds. Also correctly indicate
that the fb-mercurial* dependencies also apply only to internal builds.
Reviewed By: chadaustin
Differential Revision: D18621421
fbshipit-source-id: 74f00bf318ff9aefdca8696a61a8f33587749e66
Summary:
Automatically detect the `--facebook-internal` flag based on the current
repository project name.
Reviewed By: wez
Differential Revision: D18621358
fbshipit-source-id: f2b3018169b151811eec455863a8bfc17667d4d8
Summary:
On Windows, compile a small C executable to prepend to the zip file, to allow
the resulting executable files to be run directly on Windows, without needing
to explicitly invoke the Python interpreter to run the output file.
Reviewed By: wez
Differential Revision: D17733616
fbshipit-source-id: 989a93851412d0bbe1e7857aa9111db082f67a4c
Summary:
Now that fmt is available in Folly builds (D14813810), use it to reduce binary code size in Folly Logger. This is done by moving most of the formatting logic behind the type-erased `vformat` API. Previously it was instantiated for all combinations of formatting argument types used in calls to `FB_LOGF` and `XLOGF` in a program.
The effect of this change can be illustrated by looking at symbol sizes as given by `nm -S -td` for the following test function:
```
void test_log() {
FB_LOGF(logger, WARN, "num events: {:06d}, duration: {:6.3f}", 1234, 5.6789);
}
```
compiled in `opt` mode.
`nm` before:
```
0000000004236736 0000000000000231 T test_log()
0000000004236992 0000000000001002 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
```
`nm` after:
```
0000000004237536 0000000000000231 T test_log()
0000000004237792 0000000000000251 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
0000000004238048 0000000000000740 W folly::LogStreamProcessor::vformatLogString[abi:cxx11](folly::Range<char const*>, fmt::v5::format_args, bool&)
```
Before we had one 1002 byte instantiation of `formatLogString<int, double>`. With this change it was reduced 4x to 251 bytes and non-template function `vformatLogString` was added which is shared among all logging calls. The size of `test_log` remained unchanged. There are even bigger savings from Folly Formatter instantiations which are no longer needed, e.g.
```
0000000004238032 0000000000001363 W _ZNK5folly13BaseFormatterINS_9FormatterILb0EJRKiRKdEEELb0EJS3_S5_EEclIZNKS7_8appendToINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr12IsSomeStringIT_EE5valueEvE4typeERSH_EUlNS_5RangeIPKcEEE_EEvSK_
```
So in total this change results in ~5x per-call/instantiation binary size. It is possible to reduce binary size even further but it is not done in the current diff to keep it manageable.
In addition to binary size improvements, switching to fmt will potentially
* allow catching errors in format strings at compile time,
* simplify future migration to C++20 [`std::format`](http://eel.is/c++draft/format).
Reviewed By: simpkins
Differential Revision: D15485589
fbshipit-source-id: 06db4436839f11c2c3dbed7b36658e2193343411
Summary:
When we detect that we are building out of the fbsource
monorepo internal to FB, pre-set some environment variables to help
projects detect our internally pinned versions of node and yarn.
Previously, the detection logic was deciding that the `yarn` shell
script was the executable to invoke, but since that is a bash script
it would fail to spawn.
Reviewed By: simpkins
Differential Revision: D18523871
fbshipit-source-id: b932d2b0ccd7b79d6f9cd74d363bc426c288e38f
Summary: We removed this dependency in the code a while ago.
Reviewed By: saifhhasan
Differential Revision: D18366401
fbshipit-source-id: daf1c7865fe0c312694bdf395f9d7ace99f70395
Summary:
Our CI environment cannot directly connect to the internet,
and even if it could, doing so is undesirable to fetch javascript deps.
We maintain an offline mirror of packages used by the build(s) so that
we don't have to go out to the internet.
When running in fbsource, configure the environment to use that offline
mirror.
Reviewed By: chadaustin
Differential Revision: D18061773
fbshipit-source-id: 1a5e112f23c1baaedfb3dff0c4c2a1641f6bb9a1
Summary: Ask testpilot to include more output in test failures.
Reviewed By: fanzeyi
Differential Revision: D18061772
fbshipit-source-id: 0c14092557c21396c877d3b1776c5707437a117c
Summary:
currently, the implementation of `eden prefetch` calls into
a mercurial function that is overly eager in making network connections,
which results in what should be a fast NOP second prefetch call taking
more time than is desirable.
This diff adds a little cache to avoid repeatedly calling prefetch
for the same directory more than once for the life of the getdeps
process.
Given the usage pattern of getdeps it is OK that we don't provide
a way to invalidate this cache.
Reviewed By: fanzeyi
Differential Revision: D18005408
fbshipit-source-id: 0ec3f477da1043a5a715704b512c81fcfaa0acde
Summary:
This avoids invalidating the entire build in response
to just running `hg amend`, which is frustrating and slow.
Reviewed By: chadaustin
Differential Revision: D18005409
fbshipit-source-id: ef93313859919298be78204046eb08bcadc5398e
Summary:
This diff allows passing a watchman version number override
via the environment as well as via the cmake `WATCHMAN_VERSION_OVERRIDE`
option.
To help invalidate the build I've added a new section to the manifest
files that allows listing out additional env vars that the project
hashes should be sensitive to. The effect of this is that we'll
re-run the cmake configure step if the listed env vars are changed.
Reviewed By: Ben0mega
Differential Revision: D17865896
fbshipit-source-id: 8ea5572b0b9b7af95ec5c310e494cb17a139ced4
Summary: testpilot's defaults assume a bigger machine than some of our laptops
Reviewed By: fanzeyi
Differential Revision: D17878120
fbshipit-source-id: e01f1f9c77a4f5f011051c9c642dbe934c66bc0b
Summary: This is the first step towards removing `watchman/thirdparty/tap.{cpp,h}`
Reviewed By: chadaustin
Differential Revision: D17775680
fbshipit-source-id: d6ac32c3b2489e1713fb132b0bb46d848c56811f
Summary: This helps to squash out some flakiness
Reviewed By: pkaush
Differential Revision: D17804696
fbshipit-source-id: decd8e5dd37d802c62cae1168c4f4d72c0fc5c83
Summary: As it turns out, several of the `fizz` dependencies require it to have been built with tests enabled, so it's just easier to build them always, IIRC they only waste 1-2 minutes of time.
Reviewed By: lnicco
Differential Revision: D17837758
fbshipit-source-id: dd0c73b3aaf72831ce702dbcecd4e3ff627a4901
Summary:
Proxygen no longer uses `fbcode_builder` to run its tests, so whatever the purpose of D17158685, these `fbcode_builder` configs no longer affect Proxygen, and can be reverted to their original state.
Since the general design pattern for `fbcode_builder` has been to link everything as `.so`s, let's return to this (which helps fix Bistro's build).
Also, let's not waste time building & linking tests for libraries that are not the library under test. That is:
- Before: The Bistro build also builds tests for wangle, proxygen, etc. This is a result of some accidental changes in D17158685.
- After: We explicitly don't build test for any of the 4 dependencies here. This is OK because each project also has its own `fbcode_builder_config.py`, which **does** build tests.
This latter part should result in a build-time reduction.
Reviewed By: lnicco
Differential Revision: D17819858
fbshipit-source-id: 7cad1bed86b2f0c3934b0fc5d6fb33e6a2ee2695
Summary:
We are seeing random segment fault originating from OpenSSL on macOS when
Mononoke fetching is enabled.
The cause is that on macOS we are actually linking against libcurl shipped with
the system instead of ours. That copy of libcurl is linked with macOS's
libcrypto instead of the one we compiles during Eden's build, and it seems that
version of libcrypto does not provide concurrency safety.
The solution is to build curl on macOS and make sure it is linked to our
OpenSSL that has the concurrency callbacks registered.
Reviewed By: wez
Differential Revision: D17657822
fbshipit-source-id: 85abdf3be10b3903a5efc6b3a91624c7258de790
Summary:
We were troubleshooting an issue with the eden tests on windows
where the boost dlls where not being found during gtest discovery.
When we compute the environment, we were only including INST/bin in the
PATH on windows. On Windows, the dlls are searched for in the PATH, and
since boost installs those into its `lib` dir we were missing those.
This diff causes `lib` dirs to get added to PATH on windows in the same
manner that we would add them to `LD_LIBRARY_PATH` on linux.
Reviewed By: pkaush
Differential Revision: D17694542
fbshipit-source-id: 143a907e6d30d8c12360caa43c8d9c26ff8c88c6