Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/67
With this change it will be possible to build dependencies of and run integration tests using getdeps.py.
This is the first goal of Q4 as per https://fb.quip.com/v8YzAYNSYgot: "Get Open Source version of integration tests running on Legocastle".
Before this diff:
The OSS integration tests run now on GitHub by:
- Building some test dependencies with getdeps.py
- Building some test dependencies with homebrew/apt-get
- Running tests via python script
The OSS integration tests were not running on Sandcastle.
After this diff:
The OSS integration tests run on Github by:
- Building and executing tests via getdeps.py (execution of tests happens by getdeps.py calling Make calling python script)
The OSS integration tests run on Sandcastle using the same getdeps.py setup as Github.
Reviewed By: krallin
Differential Revision: D24253268
fbshipit-source-id: cae249b72d076222673b8bbe4ec21866dcdbb253
Summary:
This diff adds all third party dependencies that are required by getdeps to be able to build and runn Mononoke's integration tests.
Also add a stub Makefile with no-op steps that will be filled in next diff.
Reviewed By: ahornby
Differential Revision: D24251894
fbshipit-source-id: 67384ecfd0ced6762dddc3c6e61feb1240b1162d
Summary: It turns out that perf_buffer API is only available in libbpf 0.2.0, which hasn't been released yet. To unblock katran, I grabbed the latest libbpf revision and created a temporary manifest. To avoid relying on this "beta" version libbpf on other projects only katran uses it. I'll make sure to remove it in favor of the official libbpf 0.2.0 once it's released.
Reviewed By: anakryiko
Differential Revision: D24156655
fbshipit-source-id: 32f6e04079a862fbfe96fd5475678cfa4ae1b3db
Summary: Time to update libbpf version (the latest release is now 0.1.1).
Reviewed By: udippant
Differential Revision: D24063680
fbshipit-source-id: 715ac74e9671f0f8ed5b8fe9174fe4070fc0f991
Summary: This doesn't do anything anymore and is going to be removed in D23993306, let's remove it here first.
Reviewed By: yns88
Differential Revision: D23993954
fbshipit-source-id: 4d7dd5f992e34be7a0da16ce7cf59810407649c4
Summary: If retries is 0, then `0 < 0` is false, meaning we will skip the `while` loop completely and just try to read retcode, which was never assigned.
Reviewed By: fanzeyi
Differential Revision: D23999523
fbshipit-source-id: fac4a1104eba3585fb52fc8d83163cb1a87b8fee
Summary: It seems after updating zstd to 1.4.5. `Dllexport` for zstd.dll was not being picked up correctly. Instead of having zstd being a runtime dependency let's try statically link it to avoid the DLL issue.
Reviewed By: vitaut
Differential Revision: D23970349
fbshipit-source-id: 3b14dddb64d410cb9546c416f27d73b7604b21ba
Summary:
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/12
The OpenSSL version on Mac doesn't work well with EdenSCM and Mononoke integration, just use the one from getdeps/brew.
Also remove the now redundant "DEVELOPER_DIR" since the modern XCode version works.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/63
Reviewed By: StanislavGlebik
Differential Revision: D23927022
Pulled By: lukaspiatkowski
fbshipit-source-id: 6b6b3baa33d49b567b9aa6178cbd20b7ae9edc89
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/51
This diff extends capabilities of CargoBuilder in getdeps so that individual manifests can be build even without workspaces. Thanks to that a build for edenapi/tools can be made and its artifacts can be used in mononoke integration tests.
Reviewed By: StanislavGlebik
Differential Revision: D23574887
fbshipit-source-id: 8a974a6b5235d36a44fe082aad55cd380d84dd09
Summary: no need to install when not building tests
Reviewed By: wez
Differential Revision: D23714375
fbshipit-source-id: 6f34ab59ed2155df5646ad279d1347e904f393c4
Summary:
in getdeps we currently don't build and run the tests
There are a few issues:
1. we need to also build tests for fizz, wangle, mvfst since proxygen tests include headers only exported if building tests in dependencies
2. we use `ExternalProject_add` for gtest/gmock. but doesn't seem to be playing nicely with getdeps
Reviewed By: dddmello, mjoras
Differential Revision: D16934955
fbshipit-source-id: fb1c52237f9f0c71da86643409972c94d16e6a71
Summary: properly find the required GMock version (1.8.0) and allow building tests in getdeps
Reviewed By: mjoras
Differential Revision: D16935741
fbshipit-source-id: 46f62511e2feaf553d028e286a862aa5b30393c6
Summary: also always install fizz test headers for mvfst and proxygen tests to consume without needing to build fizz tests
Reviewed By: yfeldblum
Differential Revision: D23676344
fbshipit-source-id: 7ae78c81c2d67bb8da135fcd69d4be119b50a27e
Summary: they were all transitively pulling it from folly
Reviewed By: mjoras
Differential Revision: D23683292
fbshipit-source-id: 2085a580584891b3fd0960c14505c0f675a11bd5
Summary:
Update the `README.md` file in the fbcode_builder subdirectory to document the
current getdeps.py build system rather than the older fbcode_builder scripts
that required each project to keep their own python-based build files in their
project directory.
Reviewed By: chadaustin, wez
Differential Revision: D23591356
fbshipit-source-id: 75a099a10793e68a2b59696682010c4dff47ec69
Summary:
Update FindGflags.cmake to work on recent CentOS and RedHat distributions.
On these distributions the CMake package configuration installed by the
gflags-devel RPM is slightly broken, and sets `gflags_INCLUDE_DIR` to a
directory that does not exist. This happens because `/lib64` is symlinked to
`/usr/lib64`, and CMake ends up searching `/lib64` before `/usr/lib64` by
default when searching for packages. Therefore it finds `gflags-config.cmake`
via the `/lib64` symlink. However, `gflags-config.cmake` computes the include
directory relative to where its config file was found, and this relative path
computation only works when using the actual `/usr/lib64` path where it was
installed. When found via `/lib64` it returns a bogus `//include` path that
does not exist.
This updates `FindGflags.cmake` to verify if the `gflags_INCLUDE_DIR` path
actually exists, and set it to a sane location instead.
This also updates the code to use the `gflags-shared` target that is exported
by the default gflags package configuration if it was only built as a shared
library.
Pull Request resolved: https://github.com/facebook/folly/pull/1409
Reviewed By: yfeldblum
Differential Revision: D23588288
fbshipit-source-id: b68a717953ae0521f568d7bcfd05ca33cd6dc578
Summary:
We use Re2 in D22877942 for parsing multiple path prefix data fetch logging,
this introduces the dependency for eden's opensource builds.
Reviewed By: chadaustin
Differential Revision: D23431175
fbshipit-source-id: 44b399e92cb89ba1403295ecd10bc8f8d769b02c
Summary:
- Added a commandline flag to ignore testpilot even when it's available
- Fixed an oversight that caused getdeps to return 0 even if ctest tests have failed.
Reviewed By: wez
Differential Revision: D23161362
fbshipit-source-id: 1ed97c481315e2b39f1128680386390930194970
Summary:
Now that the subprocess output is no longer piped, we can see a weird situation
where the command being run is displayed after the compilation step:
+ cd /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/build/eden && \
[1/13] rust_job_pool
Finished release [optimized] target(s) in 0.75s
[12/13] Install the project...
-- Install configuration: "RelWithDebInfo"
-- Installing: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfs
-- Set runtime path of "/data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfs" to ""
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfsctl
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/lib/libbackingstore_rs.a
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/lib/libbackingstore.a
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/include/eden/scm/lib/backingstore/c_api/HgNativeBackingStore.h
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/include/eden/scm/lib/backingstore/c_api/RustBackingStore.h
+ /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/cmake-Ncng4tsJb6gdOu40ggy14-YtgNQD43 (4cb1fa6379)k5ev0n-FXq99I/bin/cmake \
+ --build \
+ /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/build/eden \
+ --target \
+ install \
+ --config \
+ Release \
+ -j \
+ 24
This is a bit awkward. Flushing stdout's buffer allows for the ordering to be
correct.
Reviewed By: wez
Differential Revision: D23079405
fbshipit-source-id: e2bf25b098d6ab4a788a5ec07deb635a42cae18c
Summary:
Redirecting stdout means that ninja/cmake won't act as if it's invoked
interactively, ie: it will buffer the output, show every single files being
compiled (instead of just a line or progress), etc. This results in a fairly
janky UX when getdeps is used on the command line. By not redirecting stdout,
we get immediate feedback about the tests being run, and the files being
compiled.
Reviewed By: wez
Differential Revision: D22967815
fbshipit-source-id: 872ddbf421065686c384a3a876d0acb8832baf2e
Summary:
port serdes object type is introduced in 1.5.2
upgrading tp2 sai dependency to 1.5.2
note that for first time, oss build failed, i had to back out removing http_proxy and https_proxy indicated here - D17429928
Differential Revision: D22888202
fbshipit-source-id: c0f90b1caa01603d25b0559188ae961df1dd13d5
Summary:
- I find the ability to make a branch and play with CI on GitHub super handy
- Removing the `- master` limitation gives me this ability
Reviewed By: yi-xian
Differential Revision: D22771835
fbshipit-source-id: 8e8839cb860ab4d1dfa0dda590afaf165127f60d
Summary:
- Replace deprecated method
- Stops spam to my console as I run with show deprecation warnings
Reviewed By: yi-xian
Differential Revision: D22776372
fbshipit-source-id: d29193e4c4c26d7facfabf9038dcb33c1af92434
Summary:
- Make OpenR build all deps from source until we remove fbzmq as a dep
- Allow a project to move it's Open Source CI forward to an alternative version of ubuntu via a new `--ubuntu-version` parameter
Reviewed By: wez
Differential Revision: D22768987
fbshipit-source-id: 07205efbd0c87a702638cf30b84a2850d064fa8e
Summary: `SDKROOT` is a requirement if we manually specify the location of the compiler on macOS. Otherwise it wouldn't be able to find the system libraries headers.
Reviewed By: wez
Differential Revision: D22577887
fbshipit-source-id: 98140e6f9e564d665db085d21023986b240b3732
Summary:
- OpenR building has been broken for quite awhile and it seems this is the main cause
Example CI Failure: https://github.com/facebook/openr/runs/631232675?check_suite_focus=true
Reviewed By: wez
Differential Revision: D22737174
fbshipit-source-id: 31d44c36d495295d30fe3ddc81991dafed100703
Summary:
This diff adds a minimal workflow for running integrations tests for Mononoke. Currently only one test is run and it fails.
This also splits the regular Mononoke CI into separate files for Linux and Mac to match the current style in Eden repo.
There are the "scopeguard::defer" fixes here that somehow escaped the CI tests.
Some tweaks have been made to "integration_runner_real.py" to make it runnable outside FB context.
Lastly the change from using "[[ -v ... ]" to "[[ -n "${...:-}" ]]; in "library.sh" was made because the former is not supported by the default Bash version preinstalled on modern MacOS.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/26
Reviewed By: krallin
Differential Revision: D22541344
Pulled By: lukaspiatkowski
fbshipit-source-id: 5023d147823166a8754be852c29b1e7b0e6d9f5f
Summary:
When we build boost 1.69.0 with newer version of Xcode, it will fail with:
```
clang: error: unknown argument: '-fcoalesce-templates'
```
This commit fixes this build failure by telling boost's build system that we are building with clang on macOS.
Reviewed By: chadaustin
Differential Revision: D22417488
fbshipit-source-id: 0b3d22835abbba6d06812c656acb0311a60d8c67
Summary:
Fixes include:
1. Passing "GETDEPS_BUILD_DIR" and "GETDEPS_INSTALL_DIR" env variable and using them in eden/scm/Makefile rather than assuming the source code is always in the same place regardless getdeps arguments (it isn't).
2. Added "fbthrift-source" and "fb303-source" to avoid unnecessary compilation (at least of fb303) and to put fbthrift and fb303 source code in an easy to locate place inside getdeps' "installed" folder.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/25
Test Plan: sandcastle, check oss-eden_scm-darwin-getdeps
Reviewed By: farnz
Differential Revision: D22431872
Pulled By: lukaspiatkowski
fbshipit-source-id: 8ccbb090713ec085a5dd56df509eb58ab6fb9e34
Summary: This commit adds a flag `--retry` to getdeps and teach it to run retry failed test. This allows us to still pass the tests when there are some flaky tests presents.
Reviewed By: wez
Differential Revision: D22291063
fbshipit-source-id: 572af48a52ceb4a9abbf530cc0154ded0120c0de
Summary:
After some experimenting, it is a little awkward if we want to specify a relative path based on the executable location. We'd need to add a bunch of path calculations to make it right, and I don't think the added complexity is really worth the effort.
As a result, let's just remove the use of relative path, and if we ever want to ship a copy of Python distribution, we can place it under the same directory as the binary.
Reviewed By: chadaustin
Differential Revision: D22394180
fbshipit-source-id: 86d27f6d16a03fe08826b5e5eafcef2a1c77997f
Summary:
In order to do what the title says, this diff does:
1. Add the `eden/oss/.../third-party/rust/.../Cargo.toml` files. As mentioned in the previous diff, those are required by GitHub so that the third party dependencies that are local in fbsource are properly defined with a "git" dependency in order for Cargo to "link" crates properly.
2. Changes to `eden/scm/Makefile` to add build/install commands for getdeps to invoke. Those command knowing that they are called from withing getdeps context they link the dependencies brought by getdeps into their proper places that match their folder layout in fbsource. Those Makefile commands also pass a GETDEPS_BUILD env to the setup.py invocations so that it knows it is being called withing a getdeps build.
3. Changes to `eden/scm/setup.py` that add "thriftasset" that makes use of the getdeps.py provided "thrift" binary to build .py files out of thrift files.
4. Changes to `distutils_rust` to use the vendored crates dir provided by getdeps.
5. Changes to `getdeps/builder.py` and `getdeps/manifest.py` that enable more fine-grained configuratior of how Makefile builds are invoked.
6. Changes to `getdeps/buildopts.py` and `getdeps/manifest.py` to disable overriding PATH and pkgconfig env, so that "eden/scm" builds in getdeps using system libraries rather than getdeps-provided ones (NOTE: I've tried to use getdeps provided libraries, but the trickiest bit was that Rust links with Python, which is currently not providable by getdeps, so if you try to build everything the system provided Python libraries will collide with getdeps provided ones)
7. Added `opensource/fbcode_builder/manifests/eden_scm` for the getdeps build.
Reviewed By: quark-zju
Differential Revision: D22336485
fbshipit-source-id: 244d10c9e06ee83de61e97e62a1f2a2184d2312f
Summary:
In EdenFS's latest Windows package. We are seeing DLL import errors coming from `asyncio` as it requires a system native module `_overlapped.pyd`.
The underlying cause is because when we build EdenFS CLI on Sandcastle, we are linking with Python 3.6.2. The Python36.dll shipped with the EdenFS package is also coming from that version.
However, on Windows laptop. We have Python 3.6.3. Since we are not shipping the Python system libraries with us. It uses the libraries installed in the system, and it attempts to import the `_overlapped.pyd` located at `C:\Pythone36\DLLs\`. This version is compiled against Python 3.6.3, which is incompatible with the Python36.dll we are using.
----
To resolve this, we need either ship an embedded copy of Python along with EdenFS, or teach EdenFS to use the Python distribution installed in the system. This commit tweaks the executable we prepend to the archive created with zipapp to locate `Python3.dll` dynamically. This allows us to use the Python installed in the system so we can avoid the version mismatch issue.
With this setup, we can also be shipping an embedded Python version along with EdenFS, and the Python loader can look for that path. This is demonstrated with the relative DLL loading `..\python`.
In theory, we can have a package structure like this:
```
.
├── python
│ ├── ....
│ └── python3.dll
└── bin
├── ...
├── edenfsctl.exe
└── edenfs.exe
```
Reviewed By: xavierd
Differential Revision: D22325210
fbshipit-source-id: 96a3f9503e7865a5f9d95710ff13f019afcf04f1
