Summary:
- Indicate that the EDEN_VERSION_OVERRIDE environment variable affects the
build.
- Exclude eden/config.py from the shipit path map, as this file is
auto-generated in open source builds.
- Include pexpect as a dependency, as this is needed for some of the
integration tests.
Reviewed By: genevievehelsel
Differential Revision: D21000163
fbshipit-source-id: 8eec378f66487229c995f637c4787eae530c7845
Summary:
This diff extracts the fbsource commit hash and the date of that
commit and maintains that in place of just the commit hash that
we were previously extracting.
This data is exported into the environment that we pass on to
builders so that it is available if they choose to use it.
In a follow on diff I'll use this to default a version number
in the watchman project.
Reviewed By: fanzeyi
Differential Revision: D20949666
fbshipit-source-id: dc12bffe5f0efc4297b15ba0140c4c67a23ab0fd
Summary:
In coverage collection mode a special module loader is prepended to
`sys.meta_path`. In very specific conditions this module loader can end up
returning a loader pointing to a _completely wrong module_. When importing
symbols from the wrong module errors occur.
The conditions to trigger the bug are:
- running in coverage collection mode, enabling the custom loader
- the test binary is a zip (e.g. par_style=fastzip)
- having a module name where the end part matches the name of a builtin Python
module
When these conditions were met, the special loader would return the builtin
Python module instead of the expected module. E.g. when loading a module like
`myapp.somemod.platform` in a zip style binary.
The custom loader first calls `imp.find_module()` to find the module it wants
to return a wrapped loader for. This fails for modules included in the test
binary, because the builtin `imp` module can not load from zips. This was the
trigger leading to the call to the buggy code.
When the initial call to `imp.find_module()` failed, the custom loader would
try a second call, asking the internal loader to this time try any path on
`sys.path`. For most module names this call would also fail, making the custom
loader return `None`, after which Python tries other loaders on `sys.path`.
However, when the final part of the module that was asked to load matches the
name of a Python builtin module, then the second call to the `imp` module would
succeed, returning a loader for the builtin module. E.g. `platform` when asking
for `myapp.somemod.platform`.
This diff fixes the issue by removing the broken second call to the internal
loader. This will never have worked, we just have not triggered or noticed
triggering the wrong loading before.
Differential Revision: D20798119
fbshipit-source-id: dffb54e308106a81af21b63c5ee64c6ca2041920
Summary:
I was testing vs2019 vs vs2017 and realized that
we weren't reconfiguring when the toolchain was changed;
this resolves that.
Reviewed By: genevievehelsel
Differential Revision: D20795118
fbshipit-source-id: db80f090367cacfcc6b53887b77cf949f9cef0e6
Summary:
On Windows the build artifacts cannot be easily run directly from the build
output directory without installing them. The `$PATH` environment variable
needs to be set correctly so that the runtime library dependencies can be
found.
This updates the builder code to emit a `run.ps1` wrapper script in the build
output directory that sets `$PATH` to support running build artifacts directly
from the build directory.
Additionally, this updates the CMake-specific builder to set properly when
running the tests with `ctest`.
Reviewed By: wez
Differential Revision: D20688290
fbshipit-source-id: 5d0f4d685692bca7e37370bd88309cf7634d87f0
Summary:
This still requires support from EdenFS in order to do much
of use, but it takes us a step closer:
* Pull in cpptoml when building with Eden support
* On Windows, when we locate the `.eden` directory, load and parse
the config file in order to determine the socket path
* If the EdenView constructor throws, treat it as a terminal error
so that we don't fallback to the regular filesystem watcher.
This is important because current EdenFS builds don't implement
the journal thrift API endpoint yet.
Reviewed By: pkaush
Differential Revision: D20504752
fbshipit-source-id: 48bbad49f1641698aa7d7b85674e3ddf4d4e617d
Summary:
We have a global `--install-prefix` argument that can be used to set
the prefix for all projects, but that is only suitable if you are running with
sufficient privileges to install each of the deps to that location during the
build. Cmake dependency resolution won't work from the build directory in that
situation; it can only see the final installed location and it will error out
if those files are not present, or link against the currently installed version
instead of the version we just built; not great!
This commit adds a project specific `--project-install-prefix` that can be used
on just the leaf project in a set of deps. That sidesteps the dependency
concern because only the last stage is built in that mode. This option
can technically be applied to an arbitrary set of projects, but in light
of the above, in practice it only makes sense to use it for the final
cmake project. Only the CMakeBuilder respects this option.
In the watchman repo, this commit adjusts the autogen.sh script to allow
specifying the installation prefix; it defaults to `/usr/local` as you
might expect.
refs: https://github.com/facebook/watchman/issues/760
Reviewed By: yfeldblum
Differential Revision: D20674439
fbshipit-source-id: 52799dbd47f3c295e2d6469ee2b74cedeaa20138
Summary:
Not sure if this is a transient issue, but the URL we've been using to obtain boost has started to return 403 errors.
Switch to a sourceforge download link instead.
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: chadaustin
Differential Revision: D20739351
fbshipit-source-id: 47483c675d59201a410c9d2a8f6db0f63ea5da69
Summary:
My recent change to ensure that we were using python3 to launch everything failed on windows: the GH actions environment has `python.exe` in the path and it is python version 3. There is no such thing as `python3` in that environment, although there is a `python2`.
Refs: https://github.com/facebook/watchman/pull/797
Reviewed By: chadaustin
Differential Revision: D20740411
fbshipit-source-id: 0e40590ccedc18e327ebb84901e2509588fdd0ff
Summary: This diff makes it possible to relay on the thrift structures from configerator in OSS.
Reviewed By: ahornby
Differential Revision: D20279457
fbshipit-source-id: 39df1c7a6f78b10a0a5bdeeebe476249ab674cc8
Summary:
This takes some pressure off both cpu and memory
on a laptop.
Reviewed By: pkaush
Differential Revision: D20562474
fbshipit-source-id: a058c71c47f25c3a2b3c1e34a0d0caf83e642021
Summary:
On linux we didn't account for the `--final-install-prefix` argument
which meant that the binaries would always be rewritten to be relative to
the destdir.
This commit fixes that.
Refs: https://github.com/facebook/watchman/issues/760
(this doesn't deal with the compiled in statedir being in the scratch path though)
Reviewed By: simpkins
Differential Revision: D20659749
fbshipit-source-id: 1e8e198a58361882249c33a67f54a7d97b849257
Summary:
The GH actions defaults for git prevent it from being able
to checkout the fbthrift repo due the length of the java related
files in the fbthrift repo.
This commit tells git to use long filenames and allows the checkout
to succeed.
Reviewed By: fanzeyi
Differential Revision: D20659750
fbshipit-source-id: 060b36d312d52a0769cf2f2dd9af60f7446f94a8
Summary:
Ensure that we are referencing python3 in the paths
that we generate for the github actions workflows, and remove
some shebangs that influence how our internal linters process
the python code.
Reviewed By: fanzeyi
Differential Revision: D20659747
fbshipit-source-id: 6f300f8e91edf7701bb27babc7b1418958cf0a10
Summary: Actually use vendored Rust crates when building with CMake. The `--offline` option make sure cargo do not go to the Internet for missing crates.
Reviewed By: simpkins
Differential Revision: D20542672
fbshipit-source-id: ab4af40150c6af8b531a75f503a4fa848b3914da
Summary:
If you have built the `eden/scm` subdirectory with `make local` in your
repository it will have left a bunch of build output artifacts in the source
tree. Update the getdeps shipit path map for eden to explicitly exclude many
of these build artifacts.
In particular:
- Exclude `eden/scm/build/` since this directory can contains a lot of files.
- Exclude all `*.pyc` files since there may be a reasonable number of these.
- Exclude several Cargo-related files since these will cause problems when
trying to build Rust code.
Reviewed By: chadaustin
Differential Revision: D20570720
fbshipit-source-id: a60dec4854ae470fdb58e9651fd8a3b910c76004
Summary:
OpenNSA has prebuilt SDK but kernel modules have to be built from sources. Add
a utility script to build and extend packaging script.
In future, we can consider folding this build into fbcode_builder itself.
Differential Revision: D20549883
fbshipit-source-id: f9475b7e0223e9f357117d7d7d27df8904fa1d73
Summary: Updated fmt version to be on par with buck build. It was causing inconsistencies.
Reviewed By: vitaut
Differential Revision: D20528011
fbshipit-source-id: d9e04ed2c28b839eaeff24120162c4db4732fa55
Summary:
This enables fetching blobs from the LFS server. For now, this is limited to
fetching them, but the protocol specify ways to also upload. That second part
will matter for commit cloud and when pushing code to the server.
One caveat to this code is that the LFS server is not mocked in tests, and thus
requests are done directly to the server. I chose very small blobs to limit the
disruption to the server, by setting a test specific user-agent, we should be
able to monitor traffic due to tests and potentially rate limit it.
Reviewed By: DurhamG
Differential Revision: D20445628
fbshipit-source-id: beb3acb3f69dd27b54f8df7ccb95b04192deca30
Summary:
Next OpenNSA release with increased stack size (needed for FBOSS) is available.
Start linking with it.
Differential Revision: D20062997
fbshipit-source-id: 9938270c322087ac3990861aa1ddd3b9ea1148ac
Summary:
Ideally we'd be using mkscratch, but this still isn't shipped
to our Windows systems.
Pick a path that is more friendly to our corp windows environment by default.
Reviewed By: pkaush
Differential Revision: D20342277
fbshipit-source-id: c85bccee6701adc03b26c92ba217b18bd684257a
Summary:
Replace the methods to get CPU and memory usage statistics:
- For the memory: use `VmRSS` of `/proc/[pid]/status`: http://man7.org/linux/man-pages/man5/proc.5.html
- For the CPU%: calculate the process is occupied how much percentage of the CPU time, use `getrusage()`: http://man7.org/linux/man-pages/man2/getrusage.2.html
- Implemented like the sigar: https://our.intern.facebook.com/intern/diffusion/FBS/browse/master/third-party/sigar/src/sigar.c?commit=4f945812675131ea64cb3d143350b1414f34a351&lines=111-169
- Formula:
- CPU% = `process used time` during the period / `time period` * 100
- `time period` = current query timestamp - last query timestamp
- `process used time` = current `process total time` - last query `process total time`
- `process total time` = CPU time used in user mode + CPU time used in system mode // get from the API `ru_utime` and `ru_stime`
Remove the `fbzmq::ResourceMonitor` and `sigar`:
- Change and rename the UT
- `ResourceMonitorTest.cpp` -> `SystemMetricsTest.cpp`
- `ResourceMonitor` -> `SystemMetricsTest` in `openr/tests/OpenrSystemTest.cpp`
- Remove `ResourceMonitor` code and dependency for `Watchdog` and `ZmqMonitor`
- Remove `sigar` dependency used in building
Reviewed By: saifhhasan
Differential Revision: D20049944
fbshipit-source-id: 00b90c8558dc5f0fb18cc31a09b9666a47b096fe
Summary:
This diff adds fb303 dependency to Open/R. fb303 is FB's open-sourced project
which provides basic infrastructure for thrift service e.g. base functions,
counters supports etc.
Using this in Open/R will allow both internal & external users leverage counters
via thrift APIs
Reviewed By: xiangxu1121
Differential Revision: D20134243
fbshipit-source-id: 23782ec99dc749dd27dc19f2b2b023f1c229608b
Summary:
The SAI diag shell implementation requires Python3.7.6.
Previous diff in the stack added Python3.7.6 manifest.
Make use of it to build diag shell and link with Python lib.
Differential Revision: D20064315
fbshipit-source-id: c855f8a9f554066e8150ce1d867828a84161961b
Summary:
As titled.
FBOSS depends on Python3.7.6, subsequent diff in the stack adds Python3.7.6
depedency for FBOSS.
Differential Revision: D20064316
fbshipit-source-id: 27b328dc25e326f3927ea6cb003fb7bb45732d61
Summary:
We need to add a separate manifest for SAI 1.4 and build that too, but that is
for later diffs, for now, we need 1.5, so change it.
Differential Revision: D20019016
fbshipit-source-id: 8630340ac7c8b7d94e6b768a2efa96266b2cc3e0
Summary:
The Buck build of RocksDB supports LZ4, so if an Eden instance's
object cache ends up compressed with LZ4 compression, the opensource
getdeps build cannot open it. Enable LZ4 in the open source getdeps
build of rocksdb.
Reviewed By: pkaush
Differential Revision: D19961878
fbshipit-source-id: 9a4f37e12bd2b40a0c5dcf2dd1684876886fae26
Summary: There are downstream oss builds showing build failure caused by introduction of metadata. We want to make them proceed to build so we won't get errors in CI.
Reviewed By: iahs
Differential Revision: D19900007
fbshipit-source-id: 4201448f7980b53e407fd2bc1c74ed4ffa8e18c1
Summary:
Take the README.md from
7ead0e29e4/README.md
and apply it on Eden repo.
Re-add the Cargo.toml file that declares Cargo workspace.
Re-add fbcode_builder/getdeps manifest for Mononoke
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/13
Test Plan:
./build/fbcode_builder/getdeps.py build mononoke
./build/fbcode_builder/getdeps.py test mononoke
Reviewed By: ahornby
Differential Revision: D19833059
Pulled By: lukaspiatkowski
fbshipit-source-id: fb37e13306c0b9969a7c4e52b05e1a66a577022f
Summary:
Instead of fetching a tarball for the vendored Rust crates, let's just use
what's present in third-party/rust.
Reviewed By: wez, fanzeyi
Differential Revision: D19770580
fbshipit-source-id: ca6615a5f5c672d9a5527ad47feab6bdd33a39db
Summary: Mononoke is becoming part of Eden repository, remove it's getdeps manifest until we finalize the move.
Reviewed By: krallin
Differential Revision: D19769090
fbshipit-source-id: 9b471686728e9ff28317f9157f5f8a1834c5f5e4
Summary: Remove the shared CheckAtomic cmake module out of the shared dir and back to the projects that need it.
Reviewed By: lukaspiatkowski
Differential Revision: D19553656
fbshipit-source-id: 5e89b5b9448ef6d6c57ef904a652e9f9a1d5dbb3
Summary:
When building with clang, the build fails with:
```
-- Looking for __atomic_is_lock_free in atomic
-- Looking for __atomic_is_lock_free in atomic - not found
CMake Error at cmake/CheckAtomic.cmake:90 (message):
Host compiler appears to require libatomic, but cannot find it.
Call Stack (most recent call first):
CMakeLists.txt:75 (include)
```
And the error is:
```
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: error: conflicting types for '__atomic_is_lock_free'
CHECK_FUNCTION_EXISTS(void);
^
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:7:3: note: '__atomic_is_lock_free' is a builtin with type 'int (unsigned long, const volatile void *)'
<command line>:1:31: note: expanded from here
#define CHECK_FUNCTION_EXISTS __atomic_is_lock_free
^
/usr/share/cmake-3.10/Modules/CheckFunctionExists.c:17:25: error: too few arguments to function call, expected 2, have 0
CHECK_FUNCTION_EXISTS();
~~~~~~~~~~~~~~~~~~~~~ ^
```
LLVM's CheckAtomic (https://fburl.com/bk14shjt) uses `__atomic_fetch_add_4` so I'm modifying the configs to use it as well to check for the existence of the library.
Reviewed By: yfeldblum
Differential Revision: D19497168
fbshipit-source-id: 64f77487efd16dba49055f6c4cb1cdd0fc4ae6da
Summary:
With this change the getdeps' CargoBuilder will support depencies between cargo builds.
The way how it works is documented in the code and required few assumptions about how a cargo project has to be defined in order to support this.
This change also adds the "mononoke" manifest and few Cargo.toml files to the mononoke project to prove that this new feature works.
Reviewed By: farnz
Differential Revision: D19468912
fbshipit-source-id: f299733cbbc2ec7bca399c898ec8d710334d0fa9
Summary:
FBOSS needs to use header files from more directories in OpenNSA,
so make those available in include directory.
Differential Revision: D19461874
fbshipit-source-id: 1fe3fbbc39477baecf9cd4f7c7a964be40cbbb12
Summary:
This is an interim solution that unblocks failing FBOSS OSS builds.
Broadcom's OpenNSA is available on github and contains large precompiled
libraries. For large files (> 100Mb), github uses git-lfs.
git clones to OpenNSA started failing yesterday with:
Error downloading object: lib/x86-64/libopennsa.a (7f21d94): Smudge error:
Error downloading lib/x86-64/libopennsa.a
(7f21d941ac32ccae38adb05b3386be739bdccd502a878364d386e7a5cb172f35): batch
response: This repository is over its data quota. Account responsible for LFS
bandwidth should purchase more data packs to restore access.
Github documents (refer:
https://help.github.com/en/github/managing-large-files/about-storage-and-bandwidth-usage)
“When you download a file tracked with Git LFS, the total file size is counted
against the repository owner’s bandwidth limit”. It appears we are hitting that
limit.
Thus, github is not a good choice here. Even if we create a 'github release
tarball', the tarball continues to carry lfs files and thus downloading those
counts towards the quota (which is limited).
Broadcom paid some amount to bump up the quota, but it is matter of time before
we hit it esp given that these builds run on-diff.
A better solution is to download a tarball (that contains real files, not lfs),
and then allow Facebook lfs to cache it.
While github does not support that, it seems bitbucket does.
Thus, we forked OpenNSA from github into a bitbucket repo and this patch
modifies the manifest to download a tarball from bitbucket that will be cached.
Broadcom is working on a cleaner solution but this will keep our FBOSS OSS build
clean in the interim.
Reviewed By: wez
Differential Revision: D19437386
fbshipit-source-id: b1213a186a0af57b9fd71b30e8899e80affd4cc0
Summary:
Broadcom has formally replaced OpenNSL with newer OpenNSA:
https://github.com/Broadcom-Network-Switching-Software/OpenNSA/blob/master/README.md
With D16401189, FBOSS no longer depends on OpenNSL, but uses the new OpenNSA
instead. Remove this manifest.
Reviewed By: wez
Differential Revision: D19413708
fbshipit-source-id: f3662f1101d59a5e0e59cb7238797d652564ff90
Summary:
With the release of OpenNSA (https://github.com/Broadcom-Network-Switching-Software/OpenNSA), we can stop using opennsal altogether!
This diff was created using following steps:
cd fboss/agent
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl_spl_//g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/opennsl/bcm/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OPENNSL/BCM/g' {} \;
find . ! -path "*\/oss*" -type f \( ! -iname "*TARGETS*" \) -exec sed -i -e 's/OpenNSL/Bcm/g' {} \; # for symbols like snmpBcmTransmittedPkts2048to4095Octets
Furthermore:
- In all TARGETS under fboss/agent replace wrapped_opennsl_symbols with wrapped_bcm_symbols.
- hg mv agent/facebook/test/OpenNSLInterface.h agent/facebook/test/BcmInterface.h
- BcmEgress.cpp remove operator== overload for opennsl API altogether, and remove operator== BCM API version from inside unnamed namespace, and put it in fboss namespace.
- BcmCinter.h and BcmCinter.cpp now have some duplicate definitions as opennsl_ got renamed to bcm_. Delete one of the definitions "as appropriate".
- Similarly, remove duplicate definitions in FakeSdk.h and FakeSdk.cpp
- Similarly, remove duplicate definitions in SdkTracer.cpp (remove definitions calling CALL_WRAPPERS_RV_NO_CINTER). And lastly, remove CALL_WRAPPERS_RV_NO_CINTER itself.
- arc lint
- Remove OpennslCompatTests.cpp
- Fix ./agent/hw/bcm/tests/facebook/BcmEgressTest.cpp to have #include
"fboss/agent/facebook/test/MockOpenNSL.h" instead of MockBcm.h
TODO:
- This patch retains the directory structure (facebook/ subdirectory), which will be fixed in subsequent patches.
- Remove opennsl references from TARGETS file, additional cleanup etc.
- Remove all oss directories...
Differential Revision: D16401189
fbshipit-source-id: 607b6c0f97ffcacf3707f6b7a4c9454cc0b24476
Summary:
Broadcom provides this library and they decided to rename it from OpenBCM to
OpenNSA. Thus, rename corresponding fbcode_builder code.
Reviewed By: wez
Differential Revision: D19396687
fbshipit-source-id: 8233dbf4de9342b5a0e54ae275d6c73d43abe6d0
Summary: Enable `INSTALL_PYTHON_LIB=ON` for internal builds of Eden on Windows.
Reviewed By: pkaush
Differential Revision: D19060630
fbshipit-source-id: 40d4f8784958fbeaad4f0d332c55fa0cb810cb5e
Summary:
Don't error out if we can't read the ELF/MACH-O header; just treat
it is not an object.
Reviewed By: chadaustin, simpkins
Differential Revision: D19253434
fbshipit-source-id: c5ecc7f0bc7a20e2611b7e2ff754355155f095da
Summary:
This should fix the Travis CI builds. It adds rust toolchain support inside docker and sets the required THRIFT env variable.
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/3
Reviewed By: krallin
Differential Revision: D18905608
Pulled By: lukaspiatkowski
fbshipit-source-id: 5db1eff6f215a6617d8acaa0c99a62d45225956b
