Summary:
Update the `README.md` file in the fbcode_builder subdirectory to document the
current getdeps.py build system rather than the older fbcode_builder scripts
that required each project to keep their own python-based build files in their
project directory.
Reviewed By: chadaustin, wez
Differential Revision: D23591356
fbshipit-source-id: 75a099a10793e68a2b59696682010c4dff47ec69
Summary:
Update FindGflags.cmake to work on recent CentOS and RedHat distributions.
On these distributions the CMake package configuration installed by the
gflags-devel RPM is slightly broken, and sets `gflags_INCLUDE_DIR` to a
directory that does not exist. This happens because `/lib64` is symlinked to
`/usr/lib64`, and CMake ends up searching `/lib64` before `/usr/lib64` by
default when searching for packages. Therefore it finds `gflags-config.cmake`
via the `/lib64` symlink. However, `gflags-config.cmake` computes the include
directory relative to where its config file was found, and this relative path
computation only works when using the actual `/usr/lib64` path where it was
installed. When found via `/lib64` it returns a bogus `//include` path that
does not exist.
This updates `FindGflags.cmake` to verify if the `gflags_INCLUDE_DIR` path
actually exists, and set it to a sane location instead.
This also updates the code to use the `gflags-shared` target that is exported
by the default gflags package configuration if it was only built as a shared
library.
Pull Request resolved: https://github.com/facebook/folly/pull/1409
Reviewed By: yfeldblum
Differential Revision: D23588288
fbshipit-source-id: b68a717953ae0521f568d7bcfd05ca33cd6dc578
Summary:
We use Re2 in D22877942 for parsing multiple path prefix data fetch logging,
this introduces the dependency for eden's opensource builds.
Reviewed By: chadaustin
Differential Revision: D23431175
fbshipit-source-id: 44b399e92cb89ba1403295ecd10bc8f8d769b02c
Summary:
- Added a commandline flag to ignore testpilot even when it's available
- Fixed an oversight that caused getdeps to return 0 even if ctest tests have failed.
Reviewed By: wez
Differential Revision: D23161362
fbshipit-source-id: 1ed97c481315e2b39f1128680386390930194970
Summary:
Now that the subprocess output is no longer piped, we can see a weird situation
where the command being run is displayed after the compilation step:
+ cd /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/build/eden && \
[1/13] rust_job_pool
Finished release [optimized] target(s) in 0.75s
[12/13] Install the project...
-- Install configuration: "RelWithDebInfo"
-- Installing: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfs
-- Set runtime path of "/data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfs" to ""
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/bin/edenfsctl
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/lib/libbackingstore_rs.a
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/lib/libbackingstore.a
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/include/eden/scm/lib/backingstore/c_api/HgNativeBackingStore.h
-- Up-to-date: /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/eden/include/eden/scm/lib/backingstore/c_api/RustBackingStore.h
+ /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/installed/cmake-Ncng4tsJb6gdOu40ggy14-YtgNQD43 (4cb1fa6379)k5ev0n-FXq99I/bin/cmake \
+ --build \
+ /data/users/xavierd/scratch/dataZusersZxavierdZfbsource/fbcode_builder_getdeps/build/eden \
+ --target \
+ install \
+ --config \
+ Release \
+ -j \
+ 24
This is a bit awkward. Flushing stdout's buffer allows for the ordering to be
correct.
Reviewed By: wez
Differential Revision: D23079405
fbshipit-source-id: e2bf25b098d6ab4a788a5ec07deb635a42cae18c
Summary:
Redirecting stdout means that ninja/cmake won't act as if it's invoked
interactively, ie: it will buffer the output, show every single files being
compiled (instead of just a line or progress), etc. This results in a fairly
janky UX when getdeps is used on the command line. By not redirecting stdout,
we get immediate feedback about the tests being run, and the files being
compiled.
Reviewed By: wez
Differential Revision: D22967815
fbshipit-source-id: 872ddbf421065686c384a3a876d0acb8832baf2e
Summary:
port serdes object type is introduced in 1.5.2
upgrading tp2 sai dependency to 1.5.2
note that for first time, oss build failed, i had to back out removing http_proxy and https_proxy indicated here - D17429928
Differential Revision: D22888202
fbshipit-source-id: c0f90b1caa01603d25b0559188ae961df1dd13d5
Summary:
- I find the ability to make a branch and play with CI on GitHub super handy
- Removing the `- master` limitation gives me this ability
Reviewed By: yi-xian
Differential Revision: D22771835
fbshipit-source-id: 8e8839cb860ab4d1dfa0dda590afaf165127f60d
Summary:
- Replace deprecated method
- Stops spam to my console as I run with show deprecation warnings
Reviewed By: yi-xian
Differential Revision: D22776372
fbshipit-source-id: d29193e4c4c26d7facfabf9038dcb33c1af92434
Summary:
- Make OpenR build all deps from source until we remove fbzmq as a dep
- Allow a project to move it's Open Source CI forward to an alternative version of ubuntu via a new `--ubuntu-version` parameter
Reviewed By: wez
Differential Revision: D22768987
fbshipit-source-id: 07205efbd0c87a702638cf30b84a2850d064fa8e
Summary: `SDKROOT` is a requirement if we manually specify the location of the compiler on macOS. Otherwise it wouldn't be able to find the system libraries headers.
Reviewed By: wez
Differential Revision: D22577887
fbshipit-source-id: 98140e6f9e564d665db085d21023986b240b3732
Summary:
- OpenR building has been broken for quite awhile and it seems this is the main cause
Example CI Failure: https://github.com/facebook/openr/runs/631232675?check_suite_focus=true
Reviewed By: wez
Differential Revision: D22737174
fbshipit-source-id: 31d44c36d495295d30fe3ddc81991dafed100703
Summary:
This diff adds a minimal workflow for running integrations tests for Mononoke. Currently only one test is run and it fails.
This also splits the regular Mononoke CI into separate files for Linux and Mac to match the current style in Eden repo.
There are the "scopeguard::defer" fixes here that somehow escaped the CI tests.
Some tweaks have been made to "integration_runner_real.py" to make it runnable outside FB context.
Lastly the change from using "[[ -v ... ]" to "[[ -n "${...:-}" ]]; in "library.sh" was made because the former is not supported by the default Bash version preinstalled on modern MacOS.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/26
Reviewed By: krallin
Differential Revision: D22541344
Pulled By: lukaspiatkowski
fbshipit-source-id: 5023d147823166a8754be852c29b1e7b0e6d9f5f
Summary:
When we build boost 1.69.0 with newer version of Xcode, it will fail with:
```
clang: error: unknown argument: '-fcoalesce-templates'
```
This commit fixes this build failure by telling boost's build system that we are building with clang on macOS.
Reviewed By: chadaustin
Differential Revision: D22417488
fbshipit-source-id: 0b3d22835abbba6d06812c656acb0311a60d8c67
Summary:
Fixes include:
1. Passing "GETDEPS_BUILD_DIR" and "GETDEPS_INSTALL_DIR" env variable and using them in eden/scm/Makefile rather than assuming the source code is always in the same place regardless getdeps arguments (it isn't).
2. Added "fbthrift-source" and "fb303-source" to avoid unnecessary compilation (at least of fb303) and to put fbthrift and fb303 source code in an easy to locate place inside getdeps' "installed" folder.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/25
Test Plan: sandcastle, check oss-eden_scm-darwin-getdeps
Reviewed By: farnz
Differential Revision: D22431872
Pulled By: lukaspiatkowski
fbshipit-source-id: 8ccbb090713ec085a5dd56df509eb58ab6fb9e34
Summary: This commit adds a flag `--retry` to getdeps and teach it to run retry failed test. This allows us to still pass the tests when there are some flaky tests presents.
Reviewed By: wez
Differential Revision: D22291063
fbshipit-source-id: 572af48a52ceb4a9abbf530cc0154ded0120c0de
Summary:
After some experimenting, it is a little awkward if we want to specify a relative path based on the executable location. We'd need to add a bunch of path calculations to make it right, and I don't think the added complexity is really worth the effort.
As a result, let's just remove the use of relative path, and if we ever want to ship a copy of Python distribution, we can place it under the same directory as the binary.
Reviewed By: chadaustin
Differential Revision: D22394180
fbshipit-source-id: 86d27f6d16a03fe08826b5e5eafcef2a1c77997f
Summary:
In order to do what the title says, this diff does:
1. Add the `eden/oss/.../third-party/rust/.../Cargo.toml` files. As mentioned in the previous diff, those are required by GitHub so that the third party dependencies that are local in fbsource are properly defined with a "git" dependency in order for Cargo to "link" crates properly.
2. Changes to `eden/scm/Makefile` to add build/install commands for getdeps to invoke. Those command knowing that they are called from withing getdeps context they link the dependencies brought by getdeps into their proper places that match their folder layout in fbsource. Those Makefile commands also pass a GETDEPS_BUILD env to the setup.py invocations so that it knows it is being called withing a getdeps build.
3. Changes to `eden/scm/setup.py` that add "thriftasset" that makes use of the getdeps.py provided "thrift" binary to build .py files out of thrift files.
4. Changes to `distutils_rust` to use the vendored crates dir provided by getdeps.
5. Changes to `getdeps/builder.py` and `getdeps/manifest.py` that enable more fine-grained configuratior of how Makefile builds are invoked.
6. Changes to `getdeps/buildopts.py` and `getdeps/manifest.py` to disable overriding PATH and pkgconfig env, so that "eden/scm" builds in getdeps using system libraries rather than getdeps-provided ones (NOTE: I've tried to use getdeps provided libraries, but the trickiest bit was that Rust links with Python, which is currently not providable by getdeps, so if you try to build everything the system provided Python libraries will collide with getdeps provided ones)
7. Added `opensource/fbcode_builder/manifests/eden_scm` for the getdeps build.
Reviewed By: quark-zju
Differential Revision: D22336485
fbshipit-source-id: 244d10c9e06ee83de61e97e62a1f2a2184d2312f
Summary:
In EdenFS's latest Windows package. We are seeing DLL import errors coming from `asyncio` as it requires a system native module `_overlapped.pyd`.
The underlying cause is because when we build EdenFS CLI on Sandcastle, we are linking with Python 3.6.2. The Python36.dll shipped with the EdenFS package is also coming from that version.
However, on Windows laptop. We have Python 3.6.3. Since we are not shipping the Python system libraries with us. It uses the libraries installed in the system, and it attempts to import the `_overlapped.pyd` located at `C:\Pythone36\DLLs\`. This version is compiled against Python 3.6.3, which is incompatible with the Python36.dll we are using.
----
To resolve this, we need either ship an embedded copy of Python along with EdenFS, or teach EdenFS to use the Python distribution installed in the system. This commit tweaks the executable we prepend to the archive created with zipapp to locate `Python3.dll` dynamically. This allows us to use the Python installed in the system so we can avoid the version mismatch issue.
With this setup, we can also be shipping an embedded Python version along with EdenFS, and the Python loader can look for that path. This is demonstrated with the relative DLL loading `..\python`.
In theory, we can have a package structure like this:
```
.
├── python
│ ├── ....
│ └── python3.dll
└── bin
├── ...
├── edenfsctl.exe
└── edenfs.exe
```
Reviewed By: xavierd
Differential Revision: D22325210
fbshipit-source-id: 96a3f9503e7865a5f9d95710ff13f019afcf04f1
Summary:
Koray reported issue with OpenNSA while using build VM and observed that we
don't hit the issue on devserver thanks to downloading the lfs cached file.
The root cause:
Broadcom added License notice and uploaded new 6.5.19 with same name.
This changed the hash though - causing fbcode builder to complain.
Fix it by adjusting the hash.
See the first line in the recursive diff of before vs. after: P134467317
(the broken symlinks is an unrelated existing issue, which we have reported to
Broadcom).
Reviewed By: bkoray
Differential Revision: D22280971
fbshipit-source-id: 781079df426f83901509225156cf77a3966d3301
Summary: 6.5.19 is now available, switch OSS to pick that instead of old 6.5.17.
Reviewed By: rsunkad
Differential Revision: D22199286
fbshipit-source-id: 231346df8d2f918d2226cfe17b01bde12c18a5a7
Summary:
Due to Thrift design of "include" statements in fbcode the thrift structures has to be contained in folders that are identical to the folder layout inside fbcode.
This diff changes the folder layout on Cargp.toml files and in fbcode_builder, there will be a next diff that changes this for ShipIt as well.
Reviewed By: ikostia
Differential Revision: D22208707
fbshipit-source-id: 65f9cafed2f0fcc8398a3887dfa622de9e139f68
Summary:
We had to fork OpenNSA and clone from it, see D19437386 for details.
Broadcom has now started hosting OpenNSA as a tarball.
Thus, we no longer need to maintain a fork (yay!)
This patch points opennsa manifest to fetch opennsa from this new location.
Reviewed By: bkoray
Differential Revision: D22175932
fbshipit-source-id: 51cd777ab836e4f191d78fbb2312925e446ca38f
Summary: This bug can be triggered when your computer name contains emoji. getdeps.py will fail to create this file due to Python attempts to write the file as cp1252 (Windows's default encoding)
Reviewed By: wez
Differential Revision: D22171935
fbshipit-source-id: fc3be2d1050c17ddbe05a0fc91d6613865f092ce
Summary:
As per https://github.com/actions/virtual-environments/issues/709 there started to be some issies with Ubuntu envs running out of space. This should fix it.
Also our Cargo builds use a lot of space, changing them to be non-incremental and removing debug symbols keeps the build fast, but greatly reduces the disk space usage leaving us enough space on GitHub Actions virtual machines.
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/23
Reviewed By: farnz
Differential Revision: D22160020
Pulled By: lukaspiatkowski
fbshipit-source-id: c23393e310c15ebf5a18b80f0bb5f1f894d24849
Summary:
Two changes here:
1. The `[patch.crates-io]` section of `third-party/rust/Cargo.toml` is being now copied over to workspaces generated by autocargo for OSS and in the runtime generated Cargo.toml file for cargo-fbcode builds. Without that some projects could be buildable in Buck internally, but not externally on GitHub due to missing patches.
2. If a `[workspace]` Cargo.toml file is being generated and there is already a generated Cargo.toml file in the same directory then instead of overriding that file the `[workspace]` (and `[patch]`) sections are appended to that Cargo.toml file.
Reviewed By: farnz
Differential Revision: D22023144
fbshipit-source-id: dec54491c36c2ee0ab29eefb722b3eceaef6ffe1
Summary:
I think LZ4 was disabled accidentally in our rocksdb getdeps build in
D21319896. Enable it again on macOS and Linux, because otherwise this
breaks people with EdenFS mounts containing LZ4-compressed proxy
hashes.
Reviewed By: xavierd
Differential Revision: D21990356
fbshipit-source-id: b9166c2992ae51f09de3fa9a4f114143aa008f43
Summary:
In 0ae204a978c11ddefafd81bd319a078239a44c1c the 'projects_dir' option
became a required constructor argument since it is called within the
constructor. However, it has not been adjusted in the subclasses that
used to set the option after instantiation. This commit fixes the
'shell_builder' and the 'debian_system_builder'.
Pull Request resolved: https://github.com/facebook/openr/pull/50
Test Plan:
1. Go to build directory: `cd build`
2. Run the `shell_builder` & `debian_system_builder`:
- `python fbcode_builder/shell_builder.py`
- `python debian_system_builder/debian_system_builder.py`
`shell_builder` output before:
```
Traceback (most recent call last):
File "fbcode_builder/shell_builder.py", line 102, in <module>
builder = ShellFBCodeBuilder()
File "/home/butjar/tu/ma/openr/build/fbcode_builder/fbcode_builder.py", line 93, in __init__
self._github_dir = self.option('projects_dir')
File "/home/butjar/tu/ma/openr/build/fbcode_builder/fbcode_builder.py", line 108, in option
raise RuntimeError('Option {0} is required'.format(name))
RuntimeError: Option projects_dir is required
```
`shell_builder` output after:
```
set -exo pipefail
export CCACHE_DIR='/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr/.ccache' CC="ccache ${CC:-gcc}" CXX="ccache ${CXX:-g++}"
### Diagnostics ###
# Builder ShellFBCodeBuilder(google/googletest:cmake_defines={u'BUILD_GTEST': u'ON', u'BUILD_SHARED_LIBS': u'OFF'}, google/googletest:git_hash=u'release-1.8.1', facebook/openr:local_repo_dir='/home/butjar/tu/ma/openr', facebook/zstd:git_hash=ShellQuoted(u'$(git describe --abbrev=0 --tags origin/master)'), openr/build:cmake_defines={u'ADD_ROOT_TESTS': u'OFF'}, thom311/libnl:git_hash=u'libnl3_2_25', projects_dir=u'/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr', fmtlib/fmt:git_hash=u'5.3.0', wangle/wangle/build:cmake_defines={u'BUILD_TESTS': u'OFF'}, prefix=u'/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr/installed', fizz/fizz/build:cmake_defines={u'BUILD_TESTS': u'ON'}, ccache_dir=u'/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr/.ccache', zeromq/libzmq:git_hash=u'v4.2.2', make_parallelism=4, jedisct1/libsodium:git_hash=u'stable')
hostname
cat /etc/issue || echo no /etc/issue
g++ --version || echo g++ not installed
cmake --version || echo cmake not installed
### Check out fmtlib/fmt, workdir build ###
mkdir -p '/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr' && cd '/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr'
git clone https://github.com/'fmtlib/fmt'
mkdir -p '/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr'/'fmt'/'build' && cd '/home/butjar/.fbcode_builder-sZshomesZsbutjarsZstusZsmasZsopenr'/'fmt'/'build'
git checkout '5.3.0'
### Build and install fmtlib/fmt ###
...
```
Reviewed By: steven1327
Differential Revision: D21865881
Pulled By: saifhhasan
fbshipit-source-id: dfd78127d3b2c78721f84a3ecafe0b7198c38f06
Summary: Currently output from build command is decoded with "surrogateescape" error handler, but when writing to log files/stdout we don't specify error handlers to be also "surrogateescape" according to https://docs.python.org/3/library/codecs.html#error-handlers, which could cause exception when there's surrogate characters logged in message.
Reviewed By: yfeldblum
Differential Revision: D21850411
fbshipit-source-id: 21c51d1ab2132171ae29f2d1fbe42655ebee94c5
Summary:
Update fmt version to 6.2.1 for better compatibility with the version used in fbcode. Among other things this fixes fbthrift build failure on Travis:
```
/home/fbthrift/thrift/lib/cpp2/async/RocketClientChannel.cpp:70:67: required from here
/home/install/include/fmt/core.h:492:3: error: static assertion failed: don't know how to format the type, include fmt/ostream.h if it provides an operator<< that should be used
```
which is caused by trying to format an enum class without a formatter - only supported as of 6.0.
Reviewed By: stevegury, avalonalex
Differential Revision: D21860076
fbshipit-source-id: 1857ab65822956b005980b8dfff7a967508f507c
Summary:
Now that we've deployed the new eden build with globfiles
support, we can enable the use of eden prefetch in the getdeps
build when running inside an EdenFS mount on Windows.
Reviewed By: fanzeyi
Differential Revision: D21692689
fbshipit-source-id: b42e778901976cf0385ec31056c227b2049162dc
Summary:
I'm doing this mostly so that we can avoid using git
to fetch the sources.
Reviewed By: chadaustin
Differential Revision: D21508201
fbshipit-source-id: 54d4635d8938659bea962e90bd829d237f1ed221
Summary:
We use drive letter substitutions to workaround Windows
filename length limitations. On my personal Windows system running
python 3.8.2 realpath inside an ssh session manages to resolve to
the full filename which causes the boost build to fail.
Avoid that!
Reviewed By: chadaustin
Differential Revision: D21507851
fbshipit-source-id: 1220c1c85d2124ddc51f42cefff2ce00e10c55c9
Summary:
For large projects, with lots of tests, running all the tests can take a lot
of time, but for quick development iteration, only a subset of the tests may
be needed to run.
On non-Windows platforms, this can be easily achieved by manually executing
the individual tests binaries and use the builtin filtering mechanism of that
test binary to achieve the goal. On Windows, this can quickly become
impossible as DLLs might not be available, and the right PATH would need to
be manually specified by hand to execute the tests binaries[0].
To solve this, let's simply provide a unified way of running specific tests
by passing in a regexp. Both testpilot and CTest do support regex to execute
specific tests. My understanding is that cargo doesn't yet allows regex, but
will in the future.
[0]: And a missing DLLs would produce no output when executed from
PowerShell, which makes this very confusing.
Reviewed By: wez
Differential Revision: D21484774
fbshipit-source-id: ee32e950e25bb2a498a2b364a447955a917b0590
Summary:
Previously, the Windows build was litered with warnings of the form (typo included):
warning C4250: 'C1': inherits 'C2::C2::method' via dominance
note: see declaration of 'C2::method'
Microsoft doesn't offer any recommendation, and the internet suggest that the
right `using` should silence it. That's unfortunately not the case, adding:
using C2::method
In `C1` doesn't do anything, and the compiler still complains :(.
Since the warning appears to be non-actionable, and looks more like a
"notice" than a warning, let's just silence it.
Reviewed By: wez
Differential Revision: D21395095
fbshipit-source-id: ae661b3ed61303e6361b8a15d9e7c6b9627ea8c1
