Summary:
This diff allows passing a watchman version number override
via the environment as well as via the cmake `WATCHMAN_VERSION_OVERRIDE`
option.
To help invalidate the build I've added a new section to the manifest
files that allows listing out additional env vars that the project
hashes should be sensitive to. The effect of this is that we'll
re-run the cmake configure step if the listed env vars are changed.
Reviewed By: Ben0mega
Differential Revision: D17865896
fbshipit-source-id: 8ea5572b0b9b7af95ec5c310e494cb17a139ced4
Summary:
Throws an exception when:
* The name specified in the manifest mismatches the filename
* Duplicated manifest -- with the sub-directory support it is now able to have multiple manifest files with the same name
Reviewed By: chadaustin
Differential Revision: D17438460
fbshipit-source-id: ac7ad0b701beb15f0e91bb05cd1ec8fe378ad5b6
Summary:
Add a new builder that can extract Python wheel files, and re-package them
for consumption by our add_fb_python_library() and add_fb_python_executable()
CMake functions. This is useful for dependencies on packages from PyPI.
At the moment this code only handles architecture-independent pure-Python
packages. It shouldn't be too hard to extend this to handle more complex
wheels, but for now I only need to use it for some pure-Python wheels and so I
haven't tested with more complex wheel files.
This also includes two new manifests for python-six and python-toml that take
use this new builder.
Reviewed By: wez
Differential Revision: D17401216
fbshipit-source-id: d6f74565887c3f004e1c06503dc9ec81599dd697
Summary:
This should make it easier to eg: enable fbzmq on macos at a later
time, and also makes things more explicit about what is being built, and
importantly, by listing these in the manifest ensures that we have a
hash change if we change this list; we wouldn't trigger such a change
if the list were encoded solely in builder.py
Reviewed By: chadaustin
Differential Revision: D17133149
fbshipit-source-id: caf0dd45e262188eeefafe0868ef95ad257a4950
Summary:
Fix printing the manifest context in the error message if we cannot find a
project fetcher. Previously the context in the message would be printed as
something like `<getdeps.manifest.ManifestContext object at 0x7fcce987e610>`,
now it shows instead as something like
`{distro=ubuntu, distro_vers=18.04, fb=off, os=linux, test=off}`
Reviewed By: chadaustin
Differential Revision: D17089208
fbshipit-source-id: c16549b61030d813b7b5ff9f65966436dc1e1898
Summary:
Add a ContextGenerator class so that we actually use the correct per-project
context when loading projects and computing dependencies.
Previously commands like `build` and `test` would change the contexts for each
project as they iterated through and performed the build. However, they did
not do this when first loading the projects. This could cause them to use
different context values when loading dependencies than when performing the
build. For instance, this could cause issues if a project depends on
`googletest` only when testing is enabled, as the code previously did not set
the "test" parameter when evaluating dependencies.
Reviewed By: pkaush
Differential Revision: D16477396
fbshipit-source-id: c1e055f07de1cb960861d19594e3bda20a2ccd87
Summary:
Check that all variable names are valid when loading manifest files.
This ensures that getdeps.py will error out if someone makes a typo in a
variable name, rather than treating it as never equal to anything.
Reviewed By: pkaush
Differential Revision: D16477397
fbshipit-source-id: 030e0642ff4a08db8eb74a0a0223e03d53e4880f
Summary:
Some people want to use getdeps with Python 2.7. This looks easy to do, so take a step toward Python 2.7 support by fixing getdeps' tests when run with Python 2.7.
For Python 3, this diff should not change behavior.
This diff should address https://github.com/facebook/bistro/issues/35.
Reviewed By: snarkmaster
Differential Revision: D16435667
fbshipit-source-id: f5c262b12995b609263341c4de26dac7f9b12b70
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
iproute2 is one of the dependencies for FBOSS. This patch adds a manifest file
to build the specific version of iproute2 needed for FBOSS.
Additionally, the default git clone depth of 100 is insufficient for the
version of iproute2 FBOSS depends on. Thus, this patch extends the git SCHEMA
to add optional argument depth. The default remains 100.
The usual /configure --prefix does not work for iproute2. Thus, we need to add
a custom builder that:
- copies sources to build directory, builds, and
- installs to installed directory using DEST_DIR.
- it must also explicitly copy include from build dir to install dir
Reviewed By: wez
Differential Revision: D15588809
fbshipit-source-id: ac5eab24134e078d88b85b4be433c78b05ef8ce5
Summary:
Previously, we were using autoconf to build sqlite,
but that isn't available on Windows. Instead, here's a builder
that generates a little cmake configuration for building and
installing sqlite.
Using cmake for this means that we can test the same builder
on all platforms that need to pull in sqlite.
Reviewed By: simpkins
Differential Revision: D15179387
fbshipit-source-id: fccf969182565982bd5be55545a2d2625aa99124
Summary:
When running in FB infra, prefer to download from our local LFS
server rather than going out to the internet.
Fall back to a normal internet download if the LFS get fails for some reason.
Upload to LFS after successfully verifying the hash for the downloaded archive.
Add a subcommand that performs a fetch for all possible platforms so that it
is easier to ensure that the lfs-pointers file is up to date.
Reviewed By: simpkins
Differential Revision: D14978660
fbshipit-source-id: 240fc32fc7003d1e06c88b80d85054dae36e2f31
Summary:
previously, a relatively lame hash was computed to use
for the build directory based on some hash of the source directory.
That was good enough to get things off the ground, but in the
interest of being able to cache the build outputs and safely
invalidate them we need a slightly more rigorous implementation.
This diff computes a hash based on the manifest contents and
relevant environmental factors.
The hash is used to name the build directory which will ultimately
be cached eg: using the travis/appveyor cache directory configuration
and some other means for the FB internal CI.
The hash needs to be sufficient that we change the hash when
the manifests change. We can tolerate a false positive change
in hash (it just means that a build will take longer), but
cannot tolerate a false negative (which would result in an
incorrect build).
Reviewed By: simpkins
Differential Revision: D14710332
fbshipit-source-id: ebc2e74eafc6f3305d4412a82195bc9fb9dfa615
Summary:
This fixes a TODO; in our CI environment we want to use the
real shipit, so we'll use this flag to make that happen.
Reviewed By: simpkins
Differential Revision: D14695576
fbshipit-source-id: 64ee72c210e2472d295dcbd39c86549273b68452
Summary:
this could do with a better name; the NopBuilder doesn't actually
build anything, but instead copies some files to the destination location.
This is used together with eg: cmake to install pre-built binaries downloaded
from a tarball.
Reviewed By: simpkins
Differential Revision: D14691015
fbshipit-source-id: a938e977aa4ec5a664bdb8085ff708319a204594
Summary:
the boost builder knows how to perform the non-standard
configure and build for boost.
Ideally we'd just build this statically and be happy but there are
some nuances I've observed while building on different platforms:
* One of our projects (thrift or wangle) explicitly requests static
boost linkage for reasons unspecified
* on darwin the install_name is broken when building dynamic libs
For the sake of expediency in getting getdeps up and running, the
solution for the moment is to build static on posix systems and
build both static and shared on windows systems.
Reviewed By: simpkins
Differential Revision: D14691009
fbshipit-source-id: 634770a6f53c3ada42d1877cc6c3dacc6eed7d18
Summary:
the openssl builder knows how to perform the non-standard
configuration and build steps to build openssl.
On Linux systems the manifests for our projects don't mention
openssl, causing them to pick up the system openssl.
On Mac, apple don't ship openssl headers so we need to build our own.
On Windows there is no standard openssl installation so we also need
to build our own.
As a result, this builder only works on windows and mac.
Reviewed By: simpkins
Differential Revision: D14691010
fbshipit-source-id: 9f8979f9eaeb5209c290cf4f43c97c0cb43d13a2
Summary:
this builder is used to bootstrap the ninja build tool.
On Windows and mac the manifest for ninja is set to download a pre-built executable.
While pre-built executables are available for linux they aren't portable enough
for our purposes so we need to be able to build it for ourselves.
Reviewed By: simpkins
Differential Revision: D14690992
fbshipit-source-id: b60fd02ad04f58dc7c2931280341791270609737
Summary:
the cmake builder knows how to use cmake to configure a build
for (preferably) and out-of-src build. The `cmake.defines` section of
the manifest is used to pass `-Dkey=value` options to the cmake configure
command line.
We prefer to use `ninja` to execute the build so that we can use more
cores than 1 on Windows and just for consistency across platforms
with mac and linux.
Reviewed By: simpkins
Differential Revision: D14690998
fbshipit-source-id: 8102e8b4a47da515ca001772788ed0e5f2645ad7
Summary:
the autoconf builder performs an out-of-source build using
the autoconf suite to configure the build rules.
The `autoconf.args` section from the manifest is passed to the `./configure`
command line.
If an `autogen.sh` script is present then it will be used to regenerate
a missing `configure` script, otherwise we'll try `autoreconf`.
Reviewed By: simpkins
Differential Revision: D14691002
fbshipit-source-id: ab8cceafb833dab513d5a50c65f4c895a4f40047
Summary:
the make builder runs `make` in the source directory.
The `make.args` section from the manifest is passed as arguments
to the `make` invocation.
Reviewed By: simpkins
Differential Revision: D14690996
fbshipit-source-id: 180d657ad05f0c0266a8c1d30979d8d1473958c9
Summary:
a builder knows how to build and install a project.
Later diffs add concrete implementations of the BuilderBase
Reviewed By: simpkins
Differential Revision: D14691018
fbshipit-source-id: 89b14614b5160353cd7e59f27037afcdf6229eb7
Summary:
This fetcher knows how to transform a 1st party project
from fbsource into approximately the same shape as ShipIt produces
for the github repo mirrors. It does this by reading shipit
mapping information from the manifest file for the project.
Since this fetcher uses data in the manifest and is implemented
directly in the getdeps codebase, it is suitable for iterating
on the opensource builds directly out of fbsource on both devservers
and laptops inside FB.
Reviewed By: simpkins
Differential Revision: D14691012
fbshipit-source-id: 05f68a7be64a2e465937b24b8825d25d3348ed13
Summary:
This fetcher knows how to take a 1st party FB project
from fbsource and transform it to the same shape as our github
repos using ShipIt. The transformation creates a transformed
mirror of the code in scratch space (rather than mutating fbsource
directly).
This can only be used in environments where shipit is available.
A later diff implements an alternative that works in more environments.
Reviewed By: simpkins
Differential Revision: D14691013
fbshipit-source-id: 539e307755c9fc0a098a235868ab622652061494
Summary:
this fetcher knows how to download a URL that references
an archive and extract the sources from it. Compressed tarballs
and zipfiles are supported.
Reviewed By: simpkins
Differential Revision: D14690994
fbshipit-source-id: ffbbd1d9479b0d5aa3e5cd13fe11a560e9d01c2a
Summary: this fetcher knows how to check out sources via git.
Reviewed By: simpkins
Differential Revision: D14691000
fbshipit-source-id: 60f1ffbfed7b32a019aef6aa70ae0903f2782451
Summary:
Fetchers are used to fetch a source directory from a
source defined in a manifest file.
More details can be found in comments on the various methods.
The Manifest class offers a create_fetcher method for constructing
an appropriate fetcher. This is just a stub in this commit, but
will expand over the course of the next few diffs as concrete
fetcher instances are added.
Reviewed By: simpkins
Differential Revision: D14691001
fbshipit-source-id: 8c9038eacae3345e9403d5d1304bf979a9ee1555
Summary:
Adds a parser for manifest files that describe projects
that may be either 1st party or 3rd party.
A selection of manifest files appears in a later diff in this stack.
This diff provides the raw parser and a couple of low level helpers.
It also defines a mechanism for validating the schema to help catch
structural (rather than semantic) issues with manifest file contents.
Later diffs in the stack add helpers for accessing the data in
a higher level way.
Reviewed By: sinancepel
Differential Revision: D14691003
fbshipit-source-id: 7d2930a3359ede0f6e21fdc45686b083ab7a9ffa