Summary: Actually use vendored Rust crates when building with CMake. The `--offline` option make sure cargo do not go to the Internet for missing crates.
Reviewed By: simpkins
Differential Revision: D20542672
fbshipit-source-id: ab4af40150c6af8b531a75f503a4fa848b3914da
Summary:
If you have built the `eden/scm` subdirectory with `make local` in your
repository it will have left a bunch of build output artifacts in the source
tree. Update the getdeps shipit path map for eden to explicitly exclude many
of these build artifacts.
In particular:
- Exclude `eden/scm/build/` since this directory can contains a lot of files.
- Exclude all `*.pyc` files since there may be a reasonable number of these.
- Exclude several Cargo-related files since these will cause problems when
trying to build Rust code.
Reviewed By: chadaustin
Differential Revision: D20570720
fbshipit-source-id: a60dec4854ae470fdb58e9651fd8a3b910c76004
Summary:
OpenNSA has prebuilt SDK but kernel modules have to be built from sources. Add
a utility script to build and extend packaging script.
In future, we can consider folding this build into fbcode_builder itself.
Differential Revision: D20549883
fbshipit-source-id: f9475b7e0223e9f357117d7d7d27df8904fa1d73
Summary: Updated fmt version to be on par with buck build. It was causing inconsistencies.
Reviewed By: vitaut
Differential Revision: D20528011
fbshipit-source-id: d9e04ed2c28b839eaeff24120162c4db4732fa55
Summary:
This enables fetching blobs from the LFS server. For now, this is limited to
fetching them, but the protocol specify ways to also upload. That second part
will matter for commit cloud and when pushing code to the server.
One caveat to this code is that the LFS server is not mocked in tests, and thus
requests are done directly to the server. I chose very small blobs to limit the
disruption to the server, by setting a test specific user-agent, we should be
able to monitor traffic due to tests and potentially rate limit it.
Reviewed By: DurhamG
Differential Revision: D20445628
fbshipit-source-id: beb3acb3f69dd27b54f8df7ccb95b04192deca30
Summary:
Next OpenNSA release with increased stack size (needed for FBOSS) is available.
Start linking with it.
Differential Revision: D20062997
fbshipit-source-id: 9938270c322087ac3990861aa1ddd3b9ea1148ac
Summary:
Ideally we'd be using mkscratch, but this still isn't shipped
to our Windows systems.
Pick a path that is more friendly to our corp windows environment by default.
Reviewed By: pkaush
Differential Revision: D20342277
fbshipit-source-id: c85bccee6701adc03b26c92ba217b18bd684257a
Summary:
Replace the methods to get CPU and memory usage statistics:
- For the memory: use `VmRSS` of `/proc/[pid]/status`: http://man7.org/linux/man-pages/man5/proc.5.html
- For the CPU%: calculate the process is occupied how much percentage of the CPU time, use `getrusage()`: http://man7.org/linux/man-pages/man2/getrusage.2.html
- Implemented like the sigar: https://our.intern.facebook.com/intern/diffusion/FBS/browse/master/third-party/sigar/src/sigar.c?commit=4f945812675131ea64cb3d143350b1414f34a351&lines=111-169
- Formula:
- CPU% = `process used time` during the period / `time period` * 100
- `time period` = current query timestamp - last query timestamp
- `process used time` = current `process total time` - last query `process total time`
- `process total time` = CPU time used in user mode + CPU time used in system mode // get from the API `ru_utime` and `ru_stime`
Remove the `fbzmq::ResourceMonitor` and `sigar`:
- Change and rename the UT
- `ResourceMonitorTest.cpp` -> `SystemMetricsTest.cpp`
- `ResourceMonitor` -> `SystemMetricsTest` in `openr/tests/OpenrSystemTest.cpp`
- Remove `ResourceMonitor` code and dependency for `Watchdog` and `ZmqMonitor`
- Remove `sigar` dependency used in building
Reviewed By: saifhhasan
Differential Revision: D20049944
fbshipit-source-id: 00b90c8558dc5f0fb18cc31a09b9666a47b096fe
Summary:
This diff adds fb303 dependency to Open/R. fb303 is FB's open-sourced project
which provides basic infrastructure for thrift service e.g. base functions,
counters supports etc.
Using this in Open/R will allow both internal & external users leverage counters
via thrift APIs
Reviewed By: xiangxu1121
Differential Revision: D20134243
fbshipit-source-id: 23782ec99dc749dd27dc19f2b2b023f1c229608b
Summary:
The SAI diag shell implementation requires Python3.7.6.
Previous diff in the stack added Python3.7.6 manifest.
Make use of it to build diag shell and link with Python lib.
Differential Revision: D20064315
fbshipit-source-id: c855f8a9f554066e8150ce1d867828a84161961b
Summary:
As titled.
FBOSS depends on Python3.7.6, subsequent diff in the stack adds Python3.7.6
depedency for FBOSS.
Differential Revision: D20064316
fbshipit-source-id: 27b328dc25e326f3927ea6cb003fb7bb45732d61
Summary:
We need to add a separate manifest for SAI 1.4 and build that too, but that is
for later diffs, for now, we need 1.5, so change it.
Differential Revision: D20019016
fbshipit-source-id: 8630340ac7c8b7d94e6b768a2efa96266b2cc3e0
Summary:
The Buck build of RocksDB supports LZ4, so if an Eden instance's
object cache ends up compressed with LZ4 compression, the opensource
getdeps build cannot open it. Enable LZ4 in the open source getdeps
build of rocksdb.
Reviewed By: pkaush
Differential Revision: D19961878
fbshipit-source-id: 9a4f37e12bd2b40a0c5dcf2dd1684876886fae26
Summary: There are downstream oss builds showing build failure caused by introduction of metadata. We want to make them proceed to build so we won't get errors in CI.
Reviewed By: iahs
Differential Revision: D19900007
fbshipit-source-id: 4201448f7980b53e407fd2bc1c74ed4ffa8e18c1
Summary:
Take the README.md from
7ead0e29e4/README.md
and apply it on Eden repo.
Re-add the Cargo.toml file that declares Cargo workspace.
Re-add fbcode_builder/getdeps manifest for Mononoke
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/13
Test Plan:
./build/fbcode_builder/getdeps.py build mononoke
./build/fbcode_builder/getdeps.py test mononoke
Reviewed By: ahornby
Differential Revision: D19833059
Pulled By: lukaspiatkowski
fbshipit-source-id: fb37e13306c0b9969a7c4e52b05e1a66a577022f
Summary:
Instead of fetching a tarball for the vendored Rust crates, let's just use
what's present in third-party/rust.
Reviewed By: wez, fanzeyi
Differential Revision: D19770580
fbshipit-source-id: ca6615a5f5c672d9a5527ad47feab6bdd33a39db
Summary: Mononoke is becoming part of Eden repository, remove it's getdeps manifest until we finalize the move.
Reviewed By: krallin
Differential Revision: D19769090
fbshipit-source-id: 9b471686728e9ff28317f9157f5f8a1834c5f5e4
Summary: Remove the shared CheckAtomic cmake module out of the shared dir and back to the projects that need it.
Reviewed By: lukaspiatkowski
Differential Revision: D19553656
fbshipit-source-id: 5e89b5b9448ef6d6c57ef904a652e9f9a1d5dbb3
Summary:
When building with clang, the build fails with:
```
-- Looking for __atomic_is_lock_free in atomic
-- Looking for __atomic_is_lock_free in atomic - not found
CMake Error at cmake/CheckAtomic.cmake:90 (message):
Host compiler appears to require libatomic, but cannot find it.
Call Stack (most recent call first):
CMakeLists.txt:75 (include)
```
And the error is:
```
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: error: conflicting types for '__atomic_is_lock_free'
CHECK_FUNCTION_EXISTS(void);
^
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: note: '__atomic_is_lock_free' is a builtin with type 'int (unsigned long, const volatile void *)'
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:17:25: error: too few arguments to function call, expected 2, have 0
CHECK_FUNCTION_EXISTS();
~~~~~~~~~~~~~~~~~~~~~ ^
```
LLVM's CheckAtomic (https://fburl.com/bk14shjt) uses `__atomic_fetch_add_4` so I'm modifying the configs to use it as well to check for the existence of the library.
Reviewed By: yfeldblum
Differential Revision: D19497168
fbshipit-source-id: 64f77487efd16dba49055f6c4cb1cdd0fc4ae6da
Summary:
With this change the getdeps' CargoBuilder will support depencies between cargo builds.
The way how it works is documented in the code and required few assumptions about how a cargo project has to be defined in order to support this.
This change also adds the "mononoke" manifest and few Cargo.toml files to the mononoke project to prove that this new feature works.
Reviewed By: farnz
Differential Revision: D19468912
fbshipit-source-id: f299733cbbc2ec7bca399c898ec8d710334d0fa9
Summary:
FBOSS needs to use header files from more directories in OpenNSA,
so make those available in include directory.
Differential Revision: D19461874
fbshipit-source-id: 1fe3fbbc39477baecf9cd4f7c7a964be40cbbb12
Summary:
This is an interim solution that unblocks failing FBOSS OSS builds.
Broadcom's OpenNSA is available on github and contains large precompiled
libraries. For large files (> 100Mb), github uses git-lfs.
git clones to OpenNSA started failing yesterday with:
Error downloading object: lib/x86-64/libopennsa.a (7f21d94): Smudge error:
Error downloading lib/x86-64/libopennsa.a
(7f21d941ac32ccae38adb05b3386be739bdccd502a878364d386e7a5cb172f35): batch
response: This repository is over its data quota. Account responsible for LFS
bandwidth should purchase more data packs to restore access.
Github documents (refer:
https://help.github.com/en/github/managing-large-files/about-storage-and-bandwidth-usage)
“When you download a file tracked with Git LFS, the total file size is counted
against the repository owner’s bandwidth limit”. It appears we are hitting that
limit.
Thus, github is not a good choice here. Even if we create a 'github release
tarball', the tarball continues to carry lfs files and thus downloading those
counts towards the quota (which is limited).
Broadcom paid some amount to bump up the quota, but it is matter of time before
we hit it esp given that these builds run on-diff.
A better solution is to download a tarball (that contains real files, not lfs),
and then allow Facebook lfs to cache it.
While github does not support that, it seems bitbucket does.
Thus, we forked OpenNSA from github into a bitbucket repo and this patch
modifies the manifest to download a tarball from bitbucket that will be cached.
Broadcom is working on a cleaner solution but this will keep our FBOSS OSS build
clean in the interim.
Reviewed By: wez
Differential Revision: D19437386
fbshipit-source-id: b1213a186a0af57b9fd71b30e8899e80affd4cc0
Summary:
Broadcom has formally replaced OpenNSL with newer OpenNSA:
https://github.com/Broadcom-Network-Switching-Software/OpenNSA/blob/master/README.md
With D16401189, FBOSS no longer depends on OpenNSL, but uses the new OpenNSA
instead. Remove this manifest.
Reviewed By: wez
Differential Revision: D19413708
fbshipit-source-id: f3662f1101d59a5e0e59cb7238797d652564ff90
Summary:
With the release of OpenNSA (https://github.com/Broadcom-Network-Switching-Software/OpenNSA), we can stop using opennsal altogether!
This diff was created using following steps:
cd fboss/agent
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl_spl_//g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl/bcm/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OPENNSL/BCM/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OpenNSL/Bcm/g' {} \; # for symbols like snmpBcmTransmittedPkts2048to4095Octets
Furthermore:
- In all TARGETS under fboss/agent replace wrapped_opennsl_symbols with wrapped_bcm_symbols.
- hg mv agent/facebook/test/OpenNSLInterface.h agent/facebook/test/BcmInterface.h
- BcmEgress.cpp remove operator== overload for opennsl API altogether, and remove operator== BCM API version from inside unnamed namespace, and put it in fboss namespace.
- BcmCinter.h and BcmCinter.cpp now have some duplicate definitions as opennsl_ got renamed to bcm_. Delete one of the definitions "as appropriate".
- Similarly, remove duplicate definitions in FakeSdk.h and FakeSdk.cpp
- Similarly, remove duplicate definitions in SdkTracer.cpp (remove definitions calling CALL_WRAPPERS_RV_NO_CINTER). And lastly, remove CALL_WRAPPERS_RV_NO_CINTER itself.
- arc lint
- Remove OpennslCompatTests.cpp
- Fix ./agent/hw/bcm/tests/facebook/BcmEgressTest.cpp to have #include
"fboss/agent/facebook/test/MockOpenNSL.h" instead of MockBcm.h
TODO:
- This patch retains the directory structure (facebook/ subdirectory), which will be fixed in subsequent patches.
- Remove opennsl references from TARGETS file, additional cleanup etc.
- Remove all oss directories...
Differential Revision: D16401189
fbshipit-source-id: 607b6c0f97ffcacf3707f6b7a4c9454cc0b24476
Summary:
Broadcom provides this library and they decided to rename it from OpenBCM to
OpenNSA. Thus, rename corresponding fbcode_builder code.
Reviewed By: wez
Differential Revision: D19396687
fbshipit-source-id: 8233dbf4de9342b5a0e54ae275d6c73d43abe6d0
Summary: Enable `INSTALL_PYTHON_LIB=ON` for internal builds of Eden on Windows.
Reviewed By: pkaush
Differential Revision: D19060630
fbshipit-source-id: 40d4f8784958fbeaad4f0d332c55fa0cb810cb5e
Summary:
Don't error out if we can't read the ELF/MACH-O header; just treat
it is not an object.
Reviewed By: chadaustin, simpkins
Differential Revision: D19253434
fbshipit-source-id: c5ecc7f0bc7a20e2611b7e2ff754355155f095da
Summary:
This should fix the Travis CI builds. It adds rust toolchain support inside docker and sets the required THRIFT env variable.
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/3
Reviewed By: krallin
Differential Revision: D18905608
Pulled By: lukaspiatkowski
fbshipit-source-id: 5db1eff6f215a6617d8acaa0c99a62d45225956b
Summary:
Update the getdeps builders to save all command output to a log file in the
build directory. This makes it easier to go back after the fact and look at
the config options selected by the build run.
Reviewed By: wez
Differential Revision: D18777059
fbshipit-source-id: 86d3ffa48ec992fe90c59a3835e18f08310ed88c
Summary: The projects have been moved to a "shed/" subdirectory, so the root Cargo.toml with workspace has to be adjusted to that move.
Reviewed By: farnz
Differential Revision: D18807189
fbshipit-source-id: 0fd66fa7edd38ab4fdf905872f38fac57ae0230e
Summary:
Add a `install_fb_python_executable()` function to `FBPythonBinary.cmake` for
helping to install python executables generated with
`add_fb_python_executable()`. This primarily helps by automatically looking
up the correct output file to install from the generated targets.
Reviewed By: wez
Differential Revision: D18774539
fbshipit-source-id: 4b397580d72ac448f21d1db6d2cdd653cf3635df
Summary:
It's impossible for a consumer of a released version of
rocksdb to do anything about this except not use it, and this particular
version of rocksdb ships with a number of shadow warnings.
Disable warning to error promotion.
See also: https://twitter.com/pcwalton/status/1201679307552083968
Reviewed By: chadaustin
Differential Revision: D18785637
fbshipit-source-id: 1db2b00b3c397d6c0b8f05b9d1c658877685c961
Summary:
OpenBCM libraries are stored with git LFS. As a result, fetcher fetches LFS pointers and not the contents. Use git-lfs to pull the real contents before copying to install dir using NoopBuilder.
In future, if more builders require git-lfs, we would consider installing
git-lfs as part of the sandcastle infra as against repeating similar
logic for each builder that requires git-lfs.
Reviewed By: wez
Differential Revision: D18759806
fbshipit-source-id: f988a0460107bc0685e7aba107daba9ed88f71e7
Summary:
As titled. This is needed by OpenBCM, and in future, possibly by other
manifests as well.
Reviewed By: wez
Differential Revision: D18759807
fbshipit-source-id: d445dfa382cea4bf96443ab9889926a4abbf0757
Summary:
This is an update to the fbcode_builder codebase to allow setting up the python virtualenv with python dependencies installed. I've included wheel and cython (with a pinned version to 0.28.6 which is the only version that works with thriftpy3 at the moment, due to https://github.com/cython/cython/issues/2985) as standard packages since these are required by some of our top-level dependencies (folly and thrift)
As far as I know, there are no other projects that use PYTHON_VENV at the moment except LogDevice so the impact should be minimal.
Reviewed By: lucaspmelo
Differential Revision: D18758383
fbshipit-source-id: 264941311c5e3a19dc4ef2bb78c9a1baa34dfd8c
Summary: The cargo builder will be used to verify if an opensource Rust project passes Cargo build, test and (optionally) documentation build.
Reviewed By: markbt
Differential Revision: D18636934
fbshipit-source-id: e982e6a017eb32913e2994e7457c8add2e9d6b95
Summary:
This diff fixes `CMakeLists.txt` to enable building `openr` tests using CMake:
1. It adds `add_openr_test` CMake function that adds executable target, registers it as test, links it with bunch of libraries like GTest and GMock, etc...
2. There is no `openr/tests/OpenrModuleTestBase.cpp` anywhere in the source tree, so this commit replaces it with `openr/common/Flags.cpp`.
Reviewed By: jstrizich
Differential Revision: D18584028
fbshipit-source-id: 07d854ef98d0d2509889a08ad042a371101a2825
Summary:
Remove the Facebook-specific dependencies from the Eden manifest: these
dependencies are now available directly in the Eden repository.
Reviewed By: chadaustin
Differential Revision: D18588009
fbshipit-source-id: 590c74e38e4e05939e6955839af3ebb959f9251b
Summary:
Remove the standalone fb-mercurial-rust target that was an internal-only
dependency for the Eden build. This build step is now done entirely in the
Eden build.
Reviewed By: fanzeyi
Differential Revision: D18623943
fbshipit-source-id: c62a1155ddd1c0a6b2270c472176ba25194c6145
Summary:
- Added sai to getdeps
- added SAI to FBOSS as a dependency
- Included path of SAI headers to FBOSS's cmake file.
Reviewed By: shri-khare
Differential Revision: D18629127
fbshipit-source-id: 119a6b7f2b64bd84414d9e16a903bc8df48a35e7
Summary:
Include tools/lfs in the shipit path map for now, since it is needed to
download vendored Rust crates on internal builds. Also correctly indicate
that the fb-mercurial* dependencies also apply only to internal builds.
Reviewed By: chadaustin
Differential Revision: D18621421
fbshipit-source-id: 74f00bf318ff9aefdca8696a61a8f33587749e66
