Commit Graph

227 Commits

Author SHA1 Message Date
Mohamed Bassem
4f19f3f2e0 Update the CheckAtomic CMake module to check for __atomic_fetch_add_4 instead of __atomic_is_lock_free
Summary:
When building with clang, the build fails with:

```
-- Looking for __atomic_is_lock_free in atomic
-- Looking for __atomic_is_lock_free in atomic - not found
CMake Error at cmake/CheckAtomic.cmake:90 (message):
  Host compiler appears to require libatomic, but cannot find it.
Call Stack (most recent call first):
  CMakeLists.txt:75 (include)
```

And the error is:

```
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: error: conflicting types for '__atomic_is_lock_free'
  CHECK_FUNCTION_EXISTS(void);
  ^
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
                              ^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: note: '__atomic_is_lock_free' is a builtin with type 'int (unsigned long, const volatile void *)'
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
                              ^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:17:25: error: too few arguments to function call, expected 2, have 0
  CHECK_FUNCTION_EXISTS();
  ~~~~~~~~~~~~~~~~~~~~~ ^
```

LLVM's CheckAtomic (https://fburl.com/bk14shjt) uses `__atomic_fetch_add_4` so I'm modifying the configs to use it as well to check for the existence of the library.

Reviewed By: yfeldblum

Differential Revision: D19497168

fbshipit-source-id: 64f77487efd16dba49055f6c4cb1cdd0fc4ae6da
2020-01-23 02:08:30 -08:00
Lukas Piatkowski
9e71027a0f opensource/fbcode_builder/getdeps.py: support cargo dependencies in cargo builds
Summary:
With this change the getdeps' CargoBuilder will support depencies between cargo builds.

The way how it works is documented in the code and required few assumptions about how a cargo project has to be defined in order to support this.

This change also adds the "mononoke" manifest and few Cargo.toml files to the mononoke project to prove that this new feature works.

Reviewed By: farnz

Differential Revision: D19468912

fbshipit-source-id: f299733cbbc2ec7bca399c898ec8d710334d0fa9
2020-01-20 23:01:17 -08:00
Shrikrishna Khare
f2ac7143d9 fbcode_builder: getdeps: OpenNSA: install more headers
Summary:
FBOSS needs to use header files from more directories in OpenNSA,
so make those available in include directory.

Differential Revision: D19461874

fbshipit-source-id: 1fe3fbbc39477baecf9cd4f7c7a964be40cbbb12
2020-01-18 10:54:09 -08:00
Shrikrishna Khare
1d1286fb8a fbcode_builder: getdeps: OpenNSA update source URL
Summary:
This is an interim solution that unblocks failing FBOSS OSS builds.

Broadcom's OpenNSA is available on github and contains large precompiled
libraries. For large files (> 100Mb), github uses git-lfs.

git clones to OpenNSA started failing yesterday with:

Error downloading object: lib/x86-64/libopennsa.a (7f21d94): Smudge error:
Error downloading lib/x86-64/libopennsa.a
(7f21d941ac32ccae38adb05b3386be739bdccd502a878364d386e7a5cb172f35): batch
response: This repository is over its data quota. Account responsible for LFS
bandwidth should purchase more data packs to restore access.

Github documents (refer:
https://help.github.com/en/github/managing-large-files/about-storage-and-bandwidth-usage)
“When you download a file tracked with Git LFS, the total file size is counted
against the repository owner’s bandwidth limit”. It appears we are hitting that
limit.

Thus, github is not a good choice here. Even if we create a 'github release
tarball', the tarball continues to carry lfs files and thus downloading those
counts towards the quota (which is limited).

Broadcom paid some amount to bump up the quota, but it is matter of time before
we hit it esp given that these builds run on-diff.

A better solution is to download a tarball (that contains real files, not lfs),
and then allow Facebook lfs to cache it.

While github does not support that, it seems bitbucket does.

Thus, we forked OpenNSA from github into a bitbucket repo and this patch
modifies the manifest to download a tarball from bitbucket that will be cached.

Broadcom is working on a cleaner solution but this will keep our FBOSS OSS build
clean in the interim.

Reviewed By: wez

Differential Revision: D19437386

fbshipit-source-id: b1213a186a0af57b9fd71b30e8899e80affd4cc0
2020-01-17 10:31:17 -08:00
Shrikrishna Khare
87c3455819 fbcode_builder: getdeps: fboss remove OpenNSL manifest
Summary:
Broadcom has formally replaced OpenNSL with newer OpenNSA:
https://github.com/Broadcom-Network-Switching-Software/OpenNSA/blob/master/README.md

With D16401189, FBOSS no longer depends on OpenNSL, but uses the new OpenNSA
instead. Remove this manifest.

Reviewed By: wez

Differential Revision: D19413708

fbshipit-source-id: f3662f1101d59a5e0e59cb7238797d652564ff90
2020-01-15 15:52:26 -08:00
Shrikrishna Khare
cbe74a776b OpenNSA: Don't use opennsl, switch to bcm-only API
Summary:
With the release of OpenNSA (https://github.com/Broadcom-Network-Switching-Software/OpenNSA), we can stop using opennsal altogether!

This diff was created using following steps:

cd fboss/agent
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl_spl_//g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl/bcm/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OPENNSL/BCM/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OpenNSL/Bcm/g' {} \; # for symbols like snmpBcmTransmittedPkts2048to4095Octets

Furthermore:

- In all TARGETS under fboss/agent replace wrapped_opennsl_symbols with wrapped_bcm_symbols.
- hg mv agent/facebook/test/OpenNSLInterface.h agent/facebook/test/BcmInterface.h
- BcmEgress.cpp remove operator== overload for opennsl API altogether, and remove operator== BCM API version from inside unnamed namespace, and put it in fboss namespace.
- BcmCinter.h and BcmCinter.cpp now have some duplicate definitions as opennsl_ got renamed to bcm_. Delete one of the definitions "as appropriate".
- Similarly, remove duplicate definitions in FakeSdk.h and FakeSdk.cpp
- Similarly, remove duplicate definitions in SdkTracer.cpp (remove definitions calling CALL_WRAPPERS_RV_NO_CINTER). And lastly, remove CALL_WRAPPERS_RV_NO_CINTER itself.
- arc lint
- Remove OpennslCompatTests.cpp
- Fix ./agent/hw/bcm/tests/facebook/BcmEgressTest.cpp to have #include
  "fboss/agent/facebook/test/MockOpenNSL.h" instead of MockBcm.h

TODO:
- This patch retains the directory structure (facebook/ subdirectory), which will be fixed in subsequent patches.
- Remove opennsl references from TARGETS file, additional cleanup etc.
- Remove all oss directories...

Differential Revision: D16401189

fbshipit-source-id: 607b6c0f97ffcacf3707f6b7a4c9454cc0b24476
2020-01-14 22:11:49 -08:00
Shrikrishna Khare
dbb417440b fbcode_builder: getdeps: fboss: rename OpenBCM to OpenNSA
Summary:
Broadcom provides this library and they decided to rename it from OpenBCM to
OpenNSA. Thus, rename corresponding fbcode_builder code.

Reviewed By: wez

Differential Revision: D19396687

fbshipit-source-id: 8233dbf4de9342b5a0e54ae275d6c73d43abe6d0
2020-01-14 22:11:48 -08:00
Adam Simpkins
db881eb225 getdeps: enable the INSTALL_PYTHON_LIB option for Eden builds on Windows
Summary: Enable `INSTALL_PYTHON_LIB=ON` for internal builds of Eden on Windows.

Reviewed By: pkaush

Differential Revision: D19060630

fbshipit-source-id: 40d4f8784958fbeaad4f0d332c55fa0cb810cb5e
2020-01-13 16:10:58 -08:00
Wez Furlong
a01a9ce0b2 getdeps: dyndeps: gracefully handle empty files
Summary:
Don't error out if we can't read the ELF/MACH-O header; just treat
it is not an object.

Reviewed By: chadaustin, simpkins

Differential Revision: D19253434

fbshipit-source-id: c5ecc7f0bc7a20e2611b7e2ff754355155f095da
2020-01-09 07:30:32 -08:00
Lukas Piatkowski
0e13cf88dc rust-shed: move stats to the shed
Summary: This change also introduces a dummy no-op stats implementation for non-fbcode builds.

Reviewed By: farnz

Differential Revision: D19193954

fbshipit-source-id: f513368a2387987bf129ddf935217c90b7c84f2a
2020-01-02 11:10:59 -08:00
Lukasz Piatkowski
d2520b62e2 Fixes for travic CI build (#3)
Summary:
This should fix the Travis CI builds. It adds rust toolchain support inside docker and sets the required THRIFT env variable.
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/3

Reviewed By: krallin

Differential Revision: D18905608

Pulled By: lukaspiatkowski

fbshipit-source-id: 5db1eff6f215a6617d8acaa0c99a62d45225956b
2019-12-10 04:56:56 -08:00
Adam Simpkins
32e625852c write build output to a log file
Summary:
Update the getdeps builders to save all command output to a log file in the
build directory.  This makes it easier to go back after the fact and look at
the config options selected by the build run.

Reviewed By: wez

Differential Revision: D18777059

fbshipit-source-id: 86d3ffa48ec992fe90c59a3835e18f08310ed88c
2019-12-06 11:59:47 -08:00
Lukas Piatkowski
c2b400ca53 rust toolchain: use RUSTC_BOOTSTRAP when building
Reviewed By: mitrandir77

Differential Revision: D18850917

fbshipit-source-id: f211dfbfadf4b5b0cb372cf8aeb9a232c4b3a739
2019-12-06 06:05:36 -08:00
Lukas Piatkowski
23cec6bfd0 getdeps: add manifest for Rust toolchain
Reviewed By: ikostia

Differential Revision: D18814410

fbshipit-source-id: 889306e51c03a29c2b72e34b7dc60db21f11ff97
2019-12-05 08:47:12 -08:00
Lukas Piatkowski
f90766f95b rust-shed: change the workspace member definition to point to the newly moved project directories
Summary: The projects have been moved to a "shed/" subdirectory, so the root Cargo.toml with workspace has to be adjusted to that move.

Reviewed By: farnz

Differential Revision: D18807189

fbshipit-source-id: 0fd66fa7edd38ab4fdf905872f38fac57ae0230e
2019-12-04 06:44:14 -08:00
Adam Simpkins
e3dff39f45 getdeps: add an install_fb_python_executable() function to the CMake utilities
Summary:
Add a `install_fb_python_executable()` function to `FBPythonBinary.cmake` for
helping to install python executables generated with
`add_fb_python_executable()`.  This primarily helps by automatically looking
up the correct output file to install from the generated targets.

Reviewed By: wez

Differential Revision: D18774539

fbshipit-source-id: 4b397580d72ac448f21d1db6d2cdd653cf3635df
2019-12-03 21:42:27 -08:00
Wez Furlong
829c722974 getdeps: rocksdb: don't enable -Werror
Summary:
It's impossible for a consumer of a released version of
rocksdb to do anything about this except not use it, and this particular
version of rocksdb ships with a number of shadow warnings.

Disable warning to error promotion.

See also: https://twitter.com/pcwalton/status/1201679307552083968

Reviewed By: chadaustin

Differential Revision: D18785637

fbshipit-source-id: 1db2b00b3c397d6c0b8f05b9d1c658877685c961
2019-12-03 15:07:56 -08:00
Shrikrishna Khare
b4394c413d fbcode_builder: getdeps: fboss: OpenBCM depends on git-lfs
Summary:
OpenBCM libraries are stored with git LFS. As a result, fetcher fetches LFS pointers and not the contents. Use git-lfs to pull the real contents before copying to install dir using NoopBuilder.
In future, if more builders require git-lfs, we would consider installing
git-lfs as part of the sandcastle infra as against repeating similar
 logic for each builder that requires git-lfs.

Reviewed By: wez

Differential Revision: D18759806

fbshipit-source-id: f988a0460107bc0685e7aba107daba9ed88f71e7
2019-12-03 11:43:41 -08:00
Shrikrishna Khare
b2f5da202e fbcode_builder: getdeps: fboss: add git-lfs fetcher
Summary:
As titled. This is needed by OpenBCM, and in future, possibly by other
manifests as well.

Reviewed By: wez

Differential Revision: D18759807

fbshipit-source-id: d445dfa382cea4bf96443ab9889926a4abbf0757
2019-12-03 11:43:41 -08:00
Ahmed Soliman
30ee55c5c5 Install wheel and cython on fbcode_builder venv environments
Summary:
This is an update to the fbcode_builder codebase to allow setting up the python virtualenv with python dependencies installed. I've included wheel and cython (with a pinned version to 0.28.6 which is the only version that works with thriftpy3 at the moment, due to https://github.com/cython/cython/issues/2985) as standard packages since these are required by some of our top-level dependencies (folly and thrift)

As far as I know, there are no other projects that use PYTHON_VENV at the moment except LogDevice so the impact should be minimal.

Reviewed By: lucaspmelo

Differential Revision: D18758383

fbshipit-source-id: 264941311c5e3a19dc4ef2bb78c9a1baa34dfd8c
2019-12-02 07:38:34 -08:00
Shrikrishna Khare
c865230548 fbcode_builder: getdeps: fboss: add OpenBCM fetcher
Summary: As titled

Differential Revision: D18726457

fbshipit-source-id: 717a6cda4a78e1b246f20328df1f01b78560cac5
2019-12-01 12:36:50 -08:00
Lukas Piatkowski
ce1dd22f6e rust-shed: unify Cargo.toml definitions
Reviewed By: farnz

Differential Revision: D18726052

fbshipit-source-id: 4f18430342ab6fd4fc82cc7d03e21f3e50e0ce25
2019-11-29 01:59:02 -08:00
Lukas Piatkowski
a8ebf55e26 fbcode_builder: add cargo builder for Rust projects
Summary: The cargo builder will be used to verify if an opensource Rust project passes Cargo build, test and (optionally) documentation build.

Reviewed By: markbt

Differential Revision: D18636934

fbshipit-source-id: e982e6a017eb32913e2994e7457c8add2e9d6b95
2019-11-28 04:59:38 -08:00
Anton Frolov
161179c4c6 Make it possible to compile openr through CMakeLists.txt
Summary:
This diff fixes `CMakeLists.txt` to enable building `openr` tests using CMake:
1. It adds `add_openr_test` CMake function that adds executable target, registers it as test, links it with bunch of libraries like GTest and GMock, etc...
2. There is no `openr/tests/OpenrModuleTestBase.cpp` anywhere in the source tree, so this commit replaces it with `openr/common/Flags.cpp`.

Reviewed By: jstrizich

Differential Revision: D18584028

fbshipit-source-id: 07d854ef98d0d2509889a08ad042a371101a2825
2019-11-27 00:24:06 -08:00
Adam Simpkins
2a1bc3cdad Update the eden manifest to drop internal FB-specific dependencies
Summary:
Remove the Facebook-specific dependencies from the Eden manifest: these
dependencies are now available directly in the Eden repository.

Reviewed By: chadaustin

Differential Revision: D18588009

fbshipit-source-id: 590c74e38e4e05939e6955839af3ebb959f9251b
2019-11-22 13:00:07 -08:00
Adam Simpkins
f71bfaacad getdeps: remove the fb-mercurial-rust target
Summary:
Remove the standalone fb-mercurial-rust target that was an internal-only
dependency for the Eden build.  This build step is now done entirely in the
Eden build.

Reviewed By: fanzeyi

Differential Revision: D18623943

fbshipit-source-id: c62a1155ddd1c0a6b2270c472176ba25194c6145
2019-11-22 13:00:06 -08:00
Koray Polat
ddba97ec96 Add SAI-1.4.0 to opensource build
Summary:
- Added sai to getdeps
- added SAI to FBOSS as a dependency
- Included path of SAI headers to FBOSS's cmake file.

Reviewed By: shri-khare

Differential Revision: D18629127

fbshipit-source-id: 119a6b7f2b64bd84414d9e16a903bc8df48a35e7
2019-11-21 22:52:08 -08:00
Adam Simpkins
7d0226157f getdeps: update eden dependencies
Summary:
Include tools/lfs in the shipit path map for now, since it is needed to
download vendored Rust crates on internal builds.  Also correctly indicate
that the fb-mercurial* dependencies also apply only to internal builds.

Reviewed By: chadaustin

Differential Revision: D18621421

fbshipit-source-id: 74f00bf318ff9aefdca8696a61a8f33587749e66
2019-11-20 16:04:33 -08:00
Adam Simpkins
2eb88ab683 getdeps: automatically detect if a build appears to be Facebook-internal
Summary:
Automatically detect the `--facebook-internal` flag based on the current
repository project name.

Reviewed By: wez

Differential Revision: D18621358

fbshipit-source-id: f2b3018169b151811eec455863a8bfc17667d4d8
2019-11-20 16:04:33 -08:00
Adam Simpkins
d2aeffb44d getdeps: update FBPythonBinary.cmake to generate executable files on Windows
Summary:
On Windows, compile a small C executable to prepend to the zip file, to allow
the resulting executable files to be run directly on Windows, without needing
to explicitly invoke the Python interpreter to run the output file.

Reviewed By: wez

Differential Revision: D17733616

fbshipit-source-id: 989a93851412d0bbe1e7857aa9111db082f67a4c
2019-11-18 18:41:00 -08:00
Victor Zverovich
3285a8f909 Replace Folly Format with fmt in logger to reduce binary size
Summary:
Now that fmt is available in Folly builds (D14813810), use it to reduce binary code size in Folly Logger. This is done by moving most of the formatting logic behind the type-erased `vformat` API. Previously it was instantiated for all combinations of formatting argument types used in calls to `FB_LOGF` and `XLOGF` in a program.

The effect of this change can be illustrated by looking at symbol sizes as given by `nm -S -td` for the following test function:

```
void test_log() {
  FB_LOGF(logger, WARN, "num events: {:06d}, duration: {:6.3f}", 1234, 5.6789);
}
```
compiled in `opt` mode.

`nm` before:

```
0000000004236736 0000000000000231 T test_log()
0000000004236992 0000000000001002 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
```

`nm` after:

```
0000000004237536 0000000000000231 T test_log()
0000000004237792 0000000000000251 W std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > folly::LogStreamProcessor::formatLogString<int, double>(folly::Range<char const*>, int const&, double const&)
0000000004238048 0000000000000740 W folly::LogStreamProcessor::vformatLogString[abi:cxx11](folly::Range<char const*>, fmt::v5::format_args, bool&)
```

Before we had one 1002 byte instantiation of `formatLogString<int, double>`. With this change it was reduced 4x to 251 bytes and non-template function `vformatLogString` was added which is shared among all logging calls. The size of `test_log` remained unchanged. There are even bigger savings from Folly Formatter instantiations which are no longer needed, e.g.

```
0000000004238032 0000000000001363 W _ZNK5folly13BaseFormatterINS_9FormatterILb0EJRKiRKdEEELb0EJS3_S5_EEclIZNKS7_8appendToINSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEEEENSt9enable_ifIXsr12IsSomeStringIT_EE5valueEvE4typeERSH_EUlNS_5RangeIPKcEEE_EEvSK_
```

So in total this change results in ~5x per-call/instantiation binary size. It is possible to reduce binary size even further but it is not done in the current diff to keep it manageable.

In addition to binary size improvements, switching to fmt will potentially

* allow catching errors in format strings at compile time,
* simplify future migration to C++20 [`std::format`](http://eel.is/c++draft/format).

Reviewed By: simpkins

Differential Revision: D15485589

fbshipit-source-id: 06db4436839f11c2c3dbed7b36658e2193343411
2019-11-18 05:53:08 -08:00
Wez Furlong
eee586ea4f getdeps: fix node/yarn detection for watchman builds
Summary:
When we detect that we are building out of the fbsource
monorepo internal to FB, pre-set some environment variables to help
projects detect our internally pinned versions of node and yarn.

Previously, the detection logic was deciding that the `yarn` shell
script was the executable to invoke, but since that is a bash script
it would fail to spawn.

Reviewed By: simpkins

Differential Revision: D18523871

fbshipit-source-id: b932d2b0ccd7b79d6f9cd74d363bc426c288e38f
2019-11-16 16:17:32 -08:00
John Strizich
69842e578c remove libnl depedency from open source build
Summary: We removed this dependency in the code a while ago.

Reviewed By: saifhhasan

Differential Revision: D18366401

fbshipit-source-id: daf1c7865fe0c312694bdf395f9d7ace99f70395
2019-11-06 21:17:08 -08:00
Luca Niccolini
f35bbe3ec5 windows build
Summary: ^

Reviewed By: mjoras

Differential Revision: D18256970

fbshipit-source-id: 4d07b961ab5ab145b620d9e10fa67116236bbf80
2019-10-31 19:03:26 -07:00
Wez Furlong
e84030a082 getdeps: [EASY] pass yarn offline mirror location for fbsource builds
Summary:
Our CI environment cannot directly connect to the internet,
and even if it could, doing so is undesirable to fetch javascript deps.
We maintain an offline mirror of packages used by the build(s) so that
we don't have to go out to the internet.

When running in fbsource, configure the environment to use that offline
mirror.

Reviewed By: chadaustin

Differential Revision: D18061773

fbshipit-source-id: 1a5e112f23c1baaedfb3dff0c4c2a1641f6bb9a1
2019-10-25 20:32:21 -07:00
Wez Furlong
0383f82752 getdeps: [EASY] tell testpilot to use long output mode for failures
Summary: Ask testpilot to include more output in test failures.

Reviewed By: fanzeyi

Differential Revision: D18061772

fbshipit-source-id: 0c14092557c21396c877d3b1776c5707437a117c
2019-10-25 20:32:21 -07:00
Wez Furlong
585d83c817 getdeps: memoize eden prefetched dirs
Summary:
currently, the implementation of `eden prefetch` calls into
a mercurial function that is overly eager in making network connections,
which results in what should be a fast NOP second prefetch call taking
more time than is desirable.

This diff adds a little cache to avoid repeatedly calling prefetch
for the same directory more than once for the life of the getdeps
process.

Given the usage pattern of getdeps it is OK that we don't provide
a way to invalidate this cache.

Reviewed By: fanzeyi

Differential Revision: D18005408

fbshipit-source-id: 0ec3f477da1043a5a715704b512c81fcfaa0acde
2019-10-25 10:20:59 -07:00
Wez Furlong
2471f855c2 getdeps: don't use fbsource hash in the simple shipit fetcher
Summary:
This avoids invalidating the entire build in response
to just running `hg amend`, which is frustrating and slow.

Reviewed By: chadaustin

Differential Revision: D18005409

fbshipit-source-id: ef93313859919298be78204046eb08bcadc5398e
2019-10-21 04:09:13 -07:00
Wez Furlong
27296ed517 getdeps: regard IOError as transient when fetching URLs
Summary:
this should help avoid this particular error:

```
IOError: [Errno socket error] [Errno 11001] getaddrinfo failed
```

Reviewed By: chadaustin

Differential Revision: D17886598

fbshipit-source-id: bd9f5b84ebd7ca5c339be3afec3a975fa907d052
2019-10-17 02:19:41 -07:00
Wez Furlong
39cb261ced watchman: pass version info down to watchman build via environment
Summary:
This diff allows passing a watchman version number override
via the environment as well as via the cmake `WATCHMAN_VERSION_OVERRIDE`
option.

To help invalidate the build I've added a new section to the manifest
files that allows listing out additional env vars that the project
hashes should be sensitive to.  The effect of this is that we'll
re-run the cmake configure step if the listed env vars are changed.

Reviewed By: Ben0mega

Differential Revision: D17865896

fbshipit-source-id: 8ea5572b0b9b7af95ec5c310e494cb17a139ced4
2019-10-17 02:19:41 -07:00
Wez Furlong
38a7106453 getdeps: constrain testpilot to our number of jobs
Summary: testpilot's defaults assume a bigger machine than some of our laptops

Reviewed By: fanzeyi

Differential Revision: D17878120

fbshipit-source-id: e01f1f9c77a4f5f011051c9c642dbe934c66bc0b
2019-10-17 02:19:40 -07:00
Andres Suarez
4f5c1eb215 Relicense getdeps from BSD to MIT
Summary: See https://fb.workplace.com/groups/osssupport/permalink/2846876118694318/

Reviewed By: wez

Differential Revision: D17750243

fbshipit-source-id: 9e149df1f8e09203820f50d0fcac4a5cecf52e33
2019-10-10 13:20:05 -07:00
Wez Furlong
ba2c416069 watchman: migrate wildmatch test to gtest
Summary: This is the first step towards removing `watchman/thirdparty/tap.{cpp,h}`

Reviewed By: chadaustin

Differential Revision: D17775680

fbshipit-source-id: d6ac32c3b2489e1713fb132b0bb46d848c56811f
2019-10-09 21:01:16 -07:00
Wez Furlong
f5b643a4e3 getdeps: ask testpilot to re-run failed tests up to 3 times
Summary: This helps to squash out some flakiness

Reviewed By: pkaush

Differential Revision: D17804696

fbshipit-source-id: decd8e5dd37d802c62cae1168c4f4d72c0fc5c83
2019-10-09 21:01:15 -07:00
Alexey Spiridonov
b93ff4720d Re-enable BUILD_TESTS for fbcode_builder
Summary: As it turns out, several of the `fizz` dependencies require it to have been built with tests enabled, so it's just easier to build them always, IIRC they only waste 1-2 minutes of time.

Reviewed By: lnicco

Differential Revision: D17837758

fbshipit-source-id: dd0c73b3aaf72831ce702dbcecd4e3ff627a4901
2019-10-09 15:20:00 -07:00
Zeyi (Rice) Fan
70b00d43aa eden: make eden build with Rust datapack
Reviewed By: wez

Differential Revision: D17494253

fbshipit-source-id: 45bab057e5cefd8f2808be96ba31814c5196051b
2019-10-09 13:03:07 -07:00
Alexey Spiridonov
3fdcdad190 Revert / fix up D17158685
Summary:
Proxygen no longer uses `fbcode_builder` to run its tests, so whatever the purpose of D17158685, these `fbcode_builder` configs no longer affect Proxygen, and can be reverted to their original state.

Since the general design pattern for `fbcode_builder` has been to link everything as `.so`s, let's return to this (which helps fix Bistro's build).

Also, let's not waste time building & linking tests for libraries that are not the library under test. That is:
 - Before: The Bistro build also builds tests for wangle, proxygen, etc. This is a result of some accidental changes in D17158685.
 - After: We explicitly don't build test for any of the 4 dependencies here. This is OK because each project also has its own `fbcode_builder_config.py`, which **does** build tests.

This latter part should result in a build-time reduction.

Reviewed By: lnicco

Differential Revision: D17819858

fbshipit-source-id: 7cad1bed86b2f0c3934b0fc5d6fb33e6a2ee2695
2019-10-08 21:47:29 -07:00
Zeyi (Rice) Fan
7f8aae31f9 getdeps: add libcurl and make Eden use libcurl instead of system curl
Summary:
We are seeing random segment fault originating from OpenSSL on macOS when
Mononoke fetching is enabled.

The cause is that on macOS we are actually linking against libcurl shipped with
the system instead of ours. That copy of libcurl is linked with macOS's
libcrypto instead of the one we compiles during Eden's build, and it seems that
version of libcrypto does not provide concurrency safety.

The solution is to build curl on macOS and make sure it is linked to our
OpenSSL that has the concurrency callbacks registered.

Reviewed By: wez

Differential Revision: D17657822

fbshipit-source-id: 85abdf3be10b3903a5efc6b3a91624c7258de790
2019-10-04 11:00:45 -07:00
Wez Furlong
0db060013e watchman: getdeps: Capture artifacts from Github Actions CI builds (#750)
Summary:
This diff teaches the generated Github Actions configuration to copy the
executable outputs from the leaf project into an artifacts dir, and then capture
the artifacts in the build.

This page has details on the artifacts mechanism used by Github Actions:
https://github.com/actions/upload-artifact

This page has the "Artifacts" link on the RHS where an example of the artifacts
can be downloaded:
https://github.com/facebook/watchman/pull/750/checks?check_run_id=244088199

Here's the actual download:
https://github.com/facebook/watchman/suites/247296465/artifacts/74686

And here's what the inside of that zip file contains for the watchman build:

```
$ find . -ls
49685444        0 drwxr-xr-x    6 wez              mygroup      192 Oct  1 18:50 .
49685487       16 -rw-r--r--    1 wez              mygroup     6148 Oct  1 18:51 ./.DS_Store
49685446        0 drwxr-xr-x    5 wez              mygroup      160 Oct  1 18:51 ./mac
49685501       16 -rw-r--r--    1 wez              mygroup     6148 Oct  1 18:51 ./mac/.DS_Store
49685453        0 drwxr-xr-x    3 wez              mygroup       96 Oct  1 18:50 ./mac/bin
49685455     5432 -rwxr-xr-x    1 wez              mygroup  2778500 Oct  2 01:02 ./mac/bin/watchman
49685454        0 drwxr-xr-x    7 wez              mygroup      224 Oct  1 18:50 ./mac/lib
49685457      296 -rwxr-xr-x    1 wez              mygroup   149096 Oct  2 01:02 ./mac/lib/libgflags.2.2.dylib
49685460     1072 -rwxr-xr-x    1 wez              mygroup   546348 Oct  2 01:02 ./mac/lib/libssl.1.1.dylib
49685459      272 -rwxr-xr-x    1 wez              mygroup   135524 Oct  2 01:02 ./mac/lib/liblz4.1.dylib
49685458      352 -rwxr-xr-x    1 wez              mygroup   176900 Oct  2 01:02 ./mac/lib/libglog.0.dylib
49685456     4888 -rwxr-xr-x    1 wez              mygroup  2501012 Oct  2 01:02 ./mac/lib/libcrypto.1.1.dylib
49685445        0 drwxr-xr-x    5 wez              mygroup      160 Oct  1 18:50 ./linux
49685486       16 -rw-r--r--    1 wez              mygroup     6148 Oct  1 18:50 ./linux/.DS_Store
49685448        0 drwxr-xr-x    3 wez              mygroup       96 Oct  1 18:50 ./linux/bin
49685450   196264 -rwxr-xr-x    1 wez              mygroup 100486648 Oct  2 01:16 ./linux/bin/watchman
49685449        0 drwxr-xr-x    4 wez              mygroup      128 Oct  1 18:50 ./linux/lib
49685451     3864 -rwxr-xr-x    1 wez              mygroup  1976888 Oct  2 01:16 ./linux/lib/libgflags.so.2.2
49685452     2208 -rwxr-xr-x    1 wez              mygroup  1127336 Oct  2 01:16 ./linux/lib/libglog.so.0
49685447        0 drwxr-xr-x    4 wez              mygroup      128 Oct  1 18:51 ./windows
49685513       16 -rw-r--r--    1 wez              mygroup     6148 Oct  1 18:51 ./windows/.DS_Store
49685461        0 drwxr-xr-x    5 wez              mygroup      160 Oct  2 01:44 ./windows/bin
49685464     7880 -rwxr-xr-x    1 wez              mygroup  4030976 Oct  2 01:44 ./windows/bin/watchman.exe
49685463      488 -rwxr-xr-x    1 wez              mygroup   249856 Oct  2 01:44 ./windows/bin/glog.dll
49685462      544 -rwxr-xr-x    1 wez              mygroup   278016 Oct  2 01:44 ./windows/bin/gflags.dll
```

The `main.yml` file was updated by running:

```
$ opensource/fbcode_builder/getdeps.py generate-github-actions --output-file watchman/.github/workflows/main.yml watchman
```

Pull Request resolved: https://github.com/facebook/watchman/pull/750

Test Plan: See description above

Reviewed By: pkaush

Differential Revision: D17705148

Pulled By: wez

fbshipit-source-id: 4f3c466180f5a1cb9fa31a9b3b3f117876f8b89d
2019-10-04 10:03:38 -07:00
Wez Furlong
bbb6bc98ab getdeps: ensure that INST/lib/*.dll are found in PATH on windows
Summary:
We were troubleshooting an issue with the eden tests on windows
where the boost dlls where not being found during gtest discovery.

When we compute the environment, we were only including INST/bin in the
PATH on windows.  On Windows, the dlls are searched for in the PATH, and
since boost installs those into its `lib` dir we were missing those.

This diff causes `lib` dirs to get added to PATH on windows in the same
manner that we would add them to `LD_LIBRARY_PATH` on linux.

Reviewed By: pkaush

Differential Revision: D17694542

fbshipit-source-id: 143a907e6d30d8c12360caa43c8d9c26ff8c88c6
2019-10-03 12:56:59 -07:00