Summary:
D15588809 introduced manifest and customer builder for iproute2. While the
Iproute2Builder copies include/* from build to install directory, it does not
copy lib/* which contains the libnetlink.a
As a result, the build system looks up libnetlink.a on local machine (which it
may not find).
Fix it by explicitly coping lib/* as well.
Reviewed By: wez
Differential Revision: D15708835
fbshipit-source-id: 3f5906eb48ce982b6e887cca0d3c11314425beab
Summary: This diff allows zmq socket to run on folly::fibers
Reviewed By: saifhhasan
Differential Revision: D15113447
fbshipit-source-id: 8a0e1f91d719a9deda48837d4ff1065408280b72
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
libnl is one of the dependencies for FBOSS. This patch adds a manifest
file to build libnl.
Reviewed By: wez
Differential Revision: D15686853
fbshipit-source-id: f7549df6dc1005630193b024be6e1ea330cc6646
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
libcurl is one of the dependencies for FBOSS. This patch adds a manifest
file to build libcurl.
Reviewed By: wez
Differential Revision: D15683387
fbshipit-source-id: 8df5c413e7dff06d1a19e0ce3b1706bff2f1ab54
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
libusb is one of the dependencies for FBOSS. This patch adds a manifest
file to build libusb. FBOSS does not need libudev, so we avoid unncessary
dependency by passing right argument to ./configure.
Reviewed By: wez
Differential Revision: D15683386
fbshipit-source-id: 1d35caecf4012ddc0df24a7305019d522b4763fa
Summary:
We sometimes see some flakeyness in our internal CI, so we can live
without it there.
Reviewed By: pkaush
Differential Revision: D15695124
fbshipit-source-id: 1d76ae89e245d9c95937e8818826c544c0ae8fc3
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
libmnl is one of the dependencies for FBOSS. This patch adds a manifest file
to build the specific version of libmnl for FBOSS.
Reviewed By: wez
Differential Revision: D15633176
fbshipit-source-id: dd51997ae950f53842711fbf2d044e84cdf437e7
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
OpenNSL is one of the dependencies for FBOSS.
OpenNSL is github repository with headers and prebuilt opaque library. Thus, we
can't/don't need to build it. Thus, OpenNSL manifest uses NopBuilder.
fbcode_builder expects the library to be in installed/lib and headers to be in
installed/include, but OpenNSL lib structure is different. Thus, add explicit
[install.fields] rule for bin/wedge => lib. Once a rule(s) is provided, we must
provide rule(s) for all, so add rule for include => include as well.
Reviewed By: wez
Differential Revision: D15593639
fbshipit-source-id: facb36ff7b9a31f5952ecbc1a9fdb834c0e68d11
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
iproute2 is one of the dependencies for FBOSS. This patch adds a manifest file
to build the specific version of iproute2 needed for FBOSS.
Additionally, the default git clone depth of 100 is insufficient for the
version of iproute2 FBOSS depends on. Thus, this patch extends the git SCHEMA
to add optional argument depth. The default remains 100.
The usual /configure --prefix does not work for iproute2. Thus, we need to add
a custom builder that:
- copies sources to build directory, builds, and
- installs to installed directory using DEST_DIR.
- it must also explicitly copy include from build dir to install dir
Reviewed By: wez
Differential Revision: D15588809
fbshipit-source-id: ac5eab24134e078d88b85b4be433c78b05ef8ce5
Summary: it seems since D15286181, we updated docker os_image to ubuntu18 and gcc7. this diff changes the README.md to reflect the change.
Reviewed By: yfeldblum
Differential Revision: D15513580
fbshipit-source-id: 0a3518c21c912903921946850bd08c948f09c295
Summary:
This should enable test pilot to skip broken/flakey tests.
The `--tag-new-tests` flag is only appropriate for code that has been
landed on master and is used by the FB infra to classify new tests
appropriately.
For continuous builds we use a test collection with different parameters
from the normal developer facing flow so that the infra can re-assess
their status and enable/disable high-signal/noisy tests.
Depends: D15495344
Reviewed By: Ben0mega
Differential Revision: D15500955
fbshipit-source-id: f3b7976cec6a5cf70f5d128b38bde11620b26918
Summary:
getdeps.py's find_existing_win32_subst_for_path function tries to reuse an existing build directory alias. (On Windows, the build directory is aliased to a drive letter to shorten paths.) If this function does not find and existing build directory alias, getdeps.py invalidates many of its caches.
On my Windows machine, find_existing_win32_subst_for_path always fails, so all of my builds are super slow. This happens because find_existing_win32_subst_for_path is given a path with a lower-case drive letter ("c:\users\..."), but the subst command returns paths with an upper-case drive letter ("C:\users\...").
When comparing paths, use ntpath.normpath. This makes the comparison case-insensitive for drive letters. (It also makes the comparison case-insensitive for other path components.)
On Linux and macOS, this diff should not change behavior.
Reviewed By: wez
Differential Revision: D15466096
fbshipit-source-id: 1669aa0a6eaeb6012154f3a9e24eba3f835262c6
Summary:
with the reshuffling around getdeps.py, and changes in folly/thrift/wangle,
our internal package stopped building. This diff brings it back to life. It is
a little bit interwined:
* Introduce a `--facebook-internal` switch to getdeps that causes `fb` in the
build context to be set to `on` rather than `off`. This allows specifying
options in the manifest that are FB specific.
* Introduce cmake options to control the WATCHMAN_STATE_DIR and whether cmake
should attempt to create WATCHMAN_STATE_DIR when it is installed.
* For FB specific builds on macOS, set the state dir to match our existing location
and disable creation of WATCHMAN_STATE_DIR because that will ultimately be
handling during package installation and we don't have privs for that on
the machine generating the package.
* Adjust Facebook CI scripts to use getdeps for building and testing watchman
Reviewed By: simpkins
Differential Revision: D15337970
fbshipit-source-id: 8ab3c1f810decf5f0b16337dc1cbe6dc73d26426
Summary:
The schedule type is used to differentiate between a build run during
code review (and thus runs code that isn't yet in the master branch),
from continuous or other types of run that operate on landed code.
This doesn't change any behavior yet; this diff just adds
the plumbing to pass down an optional arbitrary schedule type string
from the CI system.
In the future, we'll use the schedule type to influence the behavior
of running tests.
Reviewed By: strager
Differential Revision: D15300120
fbshipit-source-id: 3b46afef2ff171b3fa095763dd5006a54ea328b8
Summary:
if we find ccache in the path, then we instruct cmake to use
it as the compiler launcher.
Reviewed By: pkaush
Differential Revision: D15375441
fbshipit-source-id: 602fe54742a5ec07b0533bd2cf63cb905b051e85
Summary:
We were computing `SCRATCH/install` and hashing based on that value,
but the build stuff was later computing `SCRATCH/installed` and passing that
to the builders.
Fixup the mismatch.
Reviewed By: simpkins
Differential Revision: D15337969
fbshipit-source-id: 70288f2d9286aaacf4c1f6e0dac4680a55edac6d
Summary:
This diff updates docker os_image to ubuntu18 and gcc7 in travis settings.
The dependencies that we work with internally (e.g. gcc7, boost and so on) are not present in the ubuntu-16.04. Further, the support for gcc5 is going away. So maintaining this compatibility for these platforms is painful, especially for new projects, such as mvfst.
Reviewed By: simpkins
Differential Revision: D15286181
fbshipit-source-id: eb0da9556cdb17d0aa132ac2aa4c35f9c6eccd97
Summary: We need this for dependencies that require submodules
Reviewed By: saifhhasan, GirasoleY
Differential Revision: D15282792
fbshipit-source-id: b0cc8d645e73668252409934fd6741fb211e30ae
Summary:
On Windows, the writing operation would fail with:
```
Traceback (most recent call last):
File ".\opensource\fbcode_builder\getdeps.py", line 400, in <module>
sys.exit(main())
File ".\opensource\fbcode_builder\getdeps.py", line 393, in main
return args.func(args)
File ".\opensource\fbcode_builder\getdeps.py", line 236, in run
change_status = fetcher.update()
File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 451, in update
return mapping.mirror(self.build_options.fbsource_dir, self.repo_dir)
File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 400, in mirror
f.write(name + "\n")
File "C:\Python36\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 104-105: character maps to <undefined>
```
and this is caused by a file in libgit2: https://github.com/libgit2/libgit2/blob/master/tests/resources/status/%E8%BF%99, which is intended to test handling non-ASCII path.
Python on Windows will write file in cp1252 encoding by default, which does not contain that Chines character. (Caveat: that file on my system doesn't have the correct file name as well, it is being encoded in IBM861 for some reason. However the characters in IBM861 does not exist in CP1252 either)
Reviewed By: wez
Differential Revision: D15281521
fbshipit-source-id: 8a75e32bc1042167c945d67e26b549fda83b6b41
Summary:
The goal is to return an error code > 127 in the case of a
transient, retryable, infrastructure error. This diff generates
those in the case of failure in downloading a URL or from interacting
with LFS.
Reviewed By: strager
Differential Revision: D15266838
fbshipit-source-id: 4f52a791320123968869032c37912dded464a86e
Summary:
the cmake `add_test` and related functions allow specifying
the WORKING_DIRECTORY to use for tests. We weren't respecting this
value, so this diff looks up the WORKING_DIRECTORY from the ctest
json info and adjusts the buck test info json blob that we pass
on the testpilot.
Since that interface only allows passing an argv array, we use
the `cmake -E chdir` command to run a command in a specified
directory in a portable manner.
Reviewed By: strager
Differential Revision: D15274012
fbshipit-source-id: 1f02d461d73178745794703d455494e31c2e09ed
Summary: mstch is no longer required for fbthrift, we can remove it from oss build spec
Reviewed By: vitaut
Differential Revision: D15280267
fbshipit-source-id: 5008e54af9c927f23b0d6acbf0d9beb6e004eee1
Summary: mstch is no longer required for fbthrift, we can remove it from manifests
Reviewed By: vitaut
Differential Revision: D15279652
fbshipit-source-id: 1772de7ab51fbfe048808f66290c4ca79de60608
Summary:
This controls whether tests are built or not.
They are not built by default. When `--enable-tests` is turned on,
tests are enabled for the named project only, not all of the deps.
This results in a faster build, because eg: the number of tests in
folly is very large and consumers of folly don't want to spend so
much time waiting to build tests when really all they want to do
is build their own project.
Reviewed By: strager
Differential Revision: D15246336
fbshipit-source-id: 2e22fd60c983b059f71639602712087f2a188ec6
Summary:
This is needed to correctly pick up changes made to
eg: cmake.defines sections in the manifest for first-party
projects.
Reviewed By: strager
Differential Revision: D15246337
fbshipit-source-id: 35e525e885f87d6136d5ff3b94ebf34516ab947c
Summary: support for fbcode_build and travis build
Reviewed By: siyengar
Differential Revision: D15184766
fbshipit-source-id: 5117164c324d3d3c867592a88670a53b6f862477
Summary:
This is useful especially on Windows where the up-to-date
checks for the dependencies take a long time.
The idea is that you might run this to start:
```
$ getdeps.py build eden
```
and then while in the edit/debug/build iteration cycle:
```
$ getdeps.py build --no-deps eden
```
Reviewed By: pkaush
Differential Revision: D15200352
fbshipit-source-id: 086f2f49db967ef4d1914a69fa80067104d79136
Summary:
Previously, we were using autoconf to build sqlite,
but that isn't available on Windows. Instead, here's a builder
that generates a little cmake configuration for building and
installing sqlite.
Using cmake for this means that we can test the same builder
on all platforms that need to pull in sqlite.
Reviewed By: simpkins
Differential Revision: D15179387
fbshipit-source-id: fccf969182565982bd5be55545a2d2625aa99124
Summary:
When running in FB infra, prefer to download from our local LFS
server rather than going out to the internet.
Fall back to a normal internet download if the LFS get fails for some reason.
Upload to LFS after successfully verifying the hash for the downloaded archive.
Add a subcommand that performs a fetch for all possible platforms so that it
is easier to ensure that the lfs-pointers file is up to date.
Reviewed By: simpkins
Differential Revision: D14978660
fbshipit-source-id: 240fc32fc7003d1e06c88b80d85054dae36e2f31
Summary:
In order to pull in the treemanifest and other libraries
from our mercurial repo, add a manifest file for it, and then
adjust the logic in our cmake module to look for it.
The fb-mercurial manifest just copies the source tree to the
installation dir. In the future, we could teach it to invoke
the build for real.
Reviewed By: simpkins
Differential Revision: D14969806
fbshipit-source-id: cb270c5003a1c134eeea92c7481a84938f1c5957
Summary:
This prints out the installation or source prefix for a given project.
This is useful in eg: packaging scripts to figure out where they can
find the built artifacts.
Reviewed By: simpkins
Differential Revision: D14967378
fbshipit-source-id: 7e1b5de2ca7219af24cfb07b4b42de22aa410469
Summary:
If `testpilot` is available, generate a buck compatible json file describing the available test binaries and feed that to testpilot to have it run the tests.
A later (yet to be written) diff will be able to pass appropriate flags down to testpilot in continuous runs and that will allow testpilot to auto-disable and file tasks for tests in the opensource builds.
Reviewed By: simpkins
Differential Revision: D14766856
fbshipit-source-id: 4e144ff18f6788cf5e830d29788eabd2dbbae46a
Summary:
The loader makes it possible to monkey patch the functions
that are responsible for loading manifests. It is intended to be use
in tests that are run in bucks sandboxed environment and that don't
have direct access to the manifest files on disk.
Reviewed By: simpkins
Differential Revision: D14781326
fbshipit-source-id: 18f69f8ce5768dc605b1a9388a80b7b7b9ffe0f4
Summary:
previously, a relatively lame hash was computed to use
for the build directory based on some hash of the source directory.
That was good enough to get things off the ground, but in the
interest of being able to cache the build outputs and safely
invalidate them we need a slightly more rigorous implementation.
This diff computes a hash based on the manifest contents and
relevant environmental factors.
The hash is used to name the build directory which will ultimately
be cached eg: using the travis/appveyor cache directory configuration
and some other means for the FB internal CI.
The hash needs to be sufficient that we change the hash when
the manifests change. We can tolerate a false positive change
in hash (it just means that a build will take longer), but
cannot tolerate a false negative (which would result in an
incorrect build).
Reviewed By: simpkins
Differential Revision: D14710332
fbshipit-source-id: ebc2e74eafc6f3305d4412a82195bc9fb9dfa615
Summary:
This command schedules a facebook specific sandcastle job for the current
commit in your repo for each of the platforms we have support for in
sandcastle.
You can use `--dry-run` to have it print out the job specs.
To support this, I've moved around some of the support utilities
to make it easier to import them.
Reviewed By: simpkins
Differential Revision: D14710330
fbshipit-source-id: fb1e2a2ce78e52894291159514977da97028b37f
Summary:
Adds a `test` subcommand that runs the tests for project.
We're mostly interested in the 1st party facebook projects for this.
The `sandcastle` flow will run the `test` subcommand just for the "leaf" project--the one named on the command line.
Reviewed By: simpkins
Differential Revision: D14710331
fbshipit-source-id: 7d04a46cfd723894d61018de2f230140b52285ac
Summary:
This fixes a TODO; in our CI environment we want to use the
real shipit, so we'll use this flag to make that happen.
Reviewed By: simpkins
Differential Revision: D14695576
fbshipit-source-id: 64ee72c210e2472d295dcbd39c86549273b68452
Summary:
This makes it possible to disable eden (and thus thrift) support.
I've defaulted this to off so that we don't break the existing
watchman CI.
Reviewed By: simpkins, strager
Differential Revision: D14726767
fbshipit-source-id: 0f4d0597d901a91850f1ba6d71609c059c064c22
Summary:
adds the command that is used to drive a build.
The dependencies of the named project are computed using
the same mechanism behind the `list-deps` subcommand.
If a manifest specifies that it uses the `cmake` builder then
a dependency on `cmake` is synthesized so that we can download
a recent version of cmake ahead of time.
The build command uses the fetcher to update the src and then
executes the builder.
Reviewed By: simpkins
Differential Revision: D14691011
fbshipit-source-id: 31ae59614651ef021a9505e89c13b5717b440071
Summary:
this could do with a better name; the NopBuilder doesn't actually
build anything, but instead copies some files to the destination location.
This is used together with eg: cmake to install pre-built binaries downloaded
from a tarball.
Reviewed By: simpkins
Differential Revision: D14691015
fbshipit-source-id: a938e977aa4ec5a664bdb8085ff708319a204594
Summary:
the boost builder knows how to perform the non-standard
configure and build for boost.
Ideally we'd just build this statically and be happy but there are
some nuances I've observed while building on different platforms:
* One of our projects (thrift or wangle) explicitly requests static
boost linkage for reasons unspecified
* on darwin the install_name is broken when building dynamic libs
For the sake of expediency in getting getdeps up and running, the
solution for the moment is to build static on posix systems and
build both static and shared on windows systems.
Reviewed By: simpkins
Differential Revision: D14691009
fbshipit-source-id: 634770a6f53c3ada42d1877cc6c3dacc6eed7d18
Summary:
the openssl builder knows how to perform the non-standard
configuration and build steps to build openssl.
On Linux systems the manifests for our projects don't mention
openssl, causing them to pick up the system openssl.
On Mac, apple don't ship openssl headers so we need to build our own.
On Windows there is no standard openssl installation so we also need
to build our own.
As a result, this builder only works on windows and mac.
Reviewed By: simpkins
Differential Revision: D14691010
fbshipit-source-id: 9f8979f9eaeb5209c290cf4f43c97c0cb43d13a2
Summary:
this builder is used to bootstrap the ninja build tool.
On Windows and mac the manifest for ninja is set to download a pre-built executable.
While pre-built executables are available for linux they aren't portable enough
for our purposes so we need to be able to build it for ourselves.
Reviewed By: simpkins
Differential Revision: D14690992
fbshipit-source-id: b60fd02ad04f58dc7c2931280341791270609737
Summary:
the cmake builder knows how to use cmake to configure a build
for (preferably) and out-of-src build. The `cmake.defines` section of
the manifest is used to pass `-Dkey=value` options to the cmake configure
command line.
We prefer to use `ninja` to execute the build so that we can use more
cores than 1 on Windows and just for consistency across platforms
with mac and linux.
Reviewed By: simpkins
Differential Revision: D14690998
fbshipit-source-id: 8102e8b4a47da515ca001772788ed0e5f2645ad7
Summary:
the autoconf builder performs an out-of-source build using
the autoconf suite to configure the build rules.
The `autoconf.args` section from the manifest is passed to the `./configure`
command line.
If an `autogen.sh` script is present then it will be used to regenerate
a missing `configure` script, otherwise we'll try `autoreconf`.
Reviewed By: simpkins
Differential Revision: D14691002
fbshipit-source-id: ab8cceafb833dab513d5a50c65f4c895a4f40047
Summary:
the make builder runs `make` in the source directory.
The `make.args` section from the manifest is passed as arguments
to the `make` invocation.
Reviewed By: simpkins
Differential Revision: D14690996
fbshipit-source-id: 180d657ad05f0c0266a8c1d30979d8d1473958c9
