Summary:
In order to pull in the treemanifest and other libraries
from our mercurial repo, add a manifest file for it, and then
adjust the logic in our cmake module to look for it.
The fb-mercurial manifest just copies the source tree to the
installation dir. In the future, we could teach it to invoke
the build for real.
Reviewed By: simpkins
Differential Revision: D14969806
fbshipit-source-id: cb270c5003a1c134eeea92c7481a84938f1c5957
Summary: this cuts over the FB internal CI builds to the updated getdeps.py.
Reviewed By: simpkins
Differential Revision: D14967377
fbshipit-source-id: 183d1d7014c9b587affacf86ff185c3bae703fec
Summary:
This prints out the installation or source prefix for a given project.
This is useful in eg: packaging scripts to figure out where they can
find the built artifacts.
Reviewed By: simpkins
Differential Revision: D14967378
fbshipit-source-id: 7e1b5de2ca7219af24cfb07b4b42de22aa410469
Summary:
If `testpilot` is available, generate a buck compatible json file describing the available test binaries and feed that to testpilot to have it run the tests.
A later (yet to be written) diff will be able to pass appropriate flags down to testpilot in continuous runs and that will allow testpilot to auto-disable and file tasks for tests in the opensource builds.
Reviewed By: simpkins
Differential Revision: D14766856
fbshipit-source-id: 4e144ff18f6788cf5e830d29788eabd2dbbae46a
Summary:
The loader makes it possible to monkey patch the functions
that are responsible for loading manifests. It is intended to be use
in tests that are run in bucks sandboxed environment and that don't
have direct access to the manifest files on disk.
Reviewed By: simpkins
Differential Revision: D14781326
fbshipit-source-id: 18f69f8ce5768dc605b1a9388a80b7b7b9ffe0f4
Summary:
previously, a relatively lame hash was computed to use
for the build directory based on some hash of the source directory.
That was good enough to get things off the ground, but in the
interest of being able to cache the build outputs and safely
invalidate them we need a slightly more rigorous implementation.
This diff computes a hash based on the manifest contents and
relevant environmental factors.
The hash is used to name the build directory which will ultimately
be cached eg: using the travis/appveyor cache directory configuration
and some other means for the FB internal CI.
The hash needs to be sufficient that we change the hash when
the manifests change. We can tolerate a false positive change
in hash (it just means that a build will take longer), but
cannot tolerate a false negative (which would result in an
incorrect build).
Reviewed By: simpkins
Differential Revision: D14710332
fbshipit-source-id: ebc2e74eafc6f3305d4412a82195bc9fb9dfa615
Summary:
This command schedules a facebook specific sandcastle job for the current
commit in your repo for each of the platforms we have support for in
sandcastle.
You can use `--dry-run` to have it print out the job specs.
To support this, I've moved around some of the support utilities
to make it easier to import them.
Reviewed By: simpkins
Differential Revision: D14710330
fbshipit-source-id: fb1e2a2ce78e52894291159514977da97028b37f
Summary:
Adds a `test` subcommand that runs the tests for project.
We're mostly interested in the 1st party facebook projects for this.
The `sandcastle` flow will run the `test` subcommand just for the "leaf" project--the one named on the command line.
Reviewed By: simpkins
Differential Revision: D14710331
fbshipit-source-id: 7d04a46cfd723894d61018de2f230140b52285ac
Summary:
This fixes a TODO; in our CI environment we want to use the
real shipit, so we'll use this flag to make that happen.
Reviewed By: simpkins
Differential Revision: D14695576
fbshipit-source-id: 64ee72c210e2472d295dcbd39c86549273b68452
Summary:
This makes it possible to disable eden (and thus thrift) support.
I've defaulted this to off so that we don't break the existing
watchman CI.
Reviewed By: simpkins, strager
Differential Revision: D14726767
fbshipit-source-id: 0f4d0597d901a91850f1ba6d71609c059c064c22
Summary:
adds the command that is used to drive a build.
The dependencies of the named project are computed using
the same mechanism behind the `list-deps` subcommand.
If a manifest specifies that it uses the `cmake` builder then
a dependency on `cmake` is synthesized so that we can download
a recent version of cmake ahead of time.
The build command uses the fetcher to update the src and then
executes the builder.
Reviewed By: simpkins
Differential Revision: D14691011
fbshipit-source-id: 31ae59614651ef021a9505e89c13b5717b440071
Summary:
this could do with a better name; the NopBuilder doesn't actually
build anything, but instead copies some files to the destination location.
This is used together with eg: cmake to install pre-built binaries downloaded
from a tarball.
Reviewed By: simpkins
Differential Revision: D14691015
fbshipit-source-id: a938e977aa4ec5a664bdb8085ff708319a204594
Summary:
the boost builder knows how to perform the non-standard
configure and build for boost.
Ideally we'd just build this statically and be happy but there are
some nuances I've observed while building on different platforms:
* One of our projects (thrift or wangle) explicitly requests static
boost linkage for reasons unspecified
* on darwin the install_name is broken when building dynamic libs
For the sake of expediency in getting getdeps up and running, the
solution for the moment is to build static on posix systems and
build both static and shared on windows systems.
Reviewed By: simpkins
Differential Revision: D14691009
fbshipit-source-id: 634770a6f53c3ada42d1877cc6c3dacc6eed7d18
Summary:
the openssl builder knows how to perform the non-standard
configuration and build steps to build openssl.
On Linux systems the manifests for our projects don't mention
openssl, causing them to pick up the system openssl.
On Mac, apple don't ship openssl headers so we need to build our own.
On Windows there is no standard openssl installation so we also need
to build our own.
As a result, this builder only works on windows and mac.
Reviewed By: simpkins
Differential Revision: D14691010
fbshipit-source-id: 9f8979f9eaeb5209c290cf4f43c97c0cb43d13a2
Summary:
this builder is used to bootstrap the ninja build tool.
On Windows and mac the manifest for ninja is set to download a pre-built executable.
While pre-built executables are available for linux they aren't portable enough
for our purposes so we need to be able to build it for ourselves.
Reviewed By: simpkins
Differential Revision: D14690992
fbshipit-source-id: b60fd02ad04f58dc7c2931280341791270609737
Summary:
the cmake builder knows how to use cmake to configure a build
for (preferably) and out-of-src build. The `cmake.defines` section of
the manifest is used to pass `-Dkey=value` options to the cmake configure
command line.
We prefer to use `ninja` to execute the build so that we can use more
cores than 1 on Windows and just for consistency across platforms
with mac and linux.
Reviewed By: simpkins
Differential Revision: D14690998
fbshipit-source-id: 8102e8b4a47da515ca001772788ed0e5f2645ad7
Summary:
the autoconf builder performs an out-of-source build using
the autoconf suite to configure the build rules.
The `autoconf.args` section from the manifest is passed to the `./configure`
command line.
If an `autogen.sh` script is present then it will be used to regenerate
a missing `configure` script, otherwise we'll try `autoreconf`.
Reviewed By: simpkins
Differential Revision: D14691002
fbshipit-source-id: ab8cceafb833dab513d5a50c65f4c895a4f40047
Summary:
the make builder runs `make` in the source directory.
The `make.args` section from the manifest is passed as arguments
to the `make` invocation.
Reviewed By: simpkins
Differential Revision: D14690996
fbshipit-source-id: 180d657ad05f0c0266a8c1d30979d8d1473958c9
Summary:
a builder knows how to build and install a project.
Later diffs add concrete implementations of the BuilderBase
Reviewed By: simpkins
Differential Revision: D14691018
fbshipit-source-id: 89b14614b5160353cd7e59f27037afcdf6229eb7
Summary:
While the command isn't necessarily super useful
on its own, it does show that the plumbing for walking the
deps is functioning, and that is important when it comes
to building.
The output lists the projects in the order that they
would be built.
The `fetch` command has been augmented to add a `--recursive`
flag that uses the same mechanism to recursively fetch
the dependencies.
Reviewed By: simpkins
Differential Revision: D14691004
fbshipit-source-id: b00bf6ad4742f8bb0a70698f71a5fe03d6a1f453
Summary:
Adds a command that can be used to trigger a fetch for a project
```
$ ./opensource/fbcode_builder/getdeps.py fetch zstd
Cloning https://github.com/facebook/zstd.git...
---
+ git clone --depth=100 https://github.com/facebook/zstd.git /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git
Cloning into '/data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git'...
remote: Enumerating objects: 3816, done.
remote: Counting objects: 100% (3816/3816), done.
remote: Compressing objects: 100% (1415/1415), done.
remote: Total 3816 (delta 2556), reused 3312 (delta 2288), pack-reused 0
Receiving objects: 100% (3816/3816), 2.93 MiB | 9.59 MiB/s, done.
Resolving deltas: 100% (2556/2556), done.
Updating /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git -> v1.3.8
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git fetch origin v1.3.8
From https://github.com/facebook/zstd
* tag v1.3.8 -> FETCH_HEAD
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git checkout FETCH_HEAD
Note: checking out 'FETCH_HEAD'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 470344d Merge pull request #1479 from facebook/visualTest
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git submodule update --init
```
Reviewed By: simpkins
Differential Revision: D14691008
fbshipit-source-id: 3afa391360518a08ebd6ff97f5b8b4993f10c4e8
Summary:
This fetcher knows how to transform a 1st party project
from fbsource into approximately the same shape as ShipIt produces
for the github repo mirrors. It does this by reading shipit
mapping information from the manifest file for the project.
Since this fetcher uses data in the manifest and is implemented
directly in the getdeps codebase, it is suitable for iterating
on the opensource builds directly out of fbsource on both devservers
and laptops inside FB.
Reviewed By: simpkins
Differential Revision: D14691012
fbshipit-source-id: 05f68a7be64a2e465937b24b8825d25d3348ed13
Summary:
This fetcher knows how to take a 1st party FB project
from fbsource and transform it to the same shape as our github
repos using ShipIt. The transformation creates a transformed
mirror of the code in scratch space (rather than mutating fbsource
directly).
This can only be used in environments where shipit is available.
A later diff implements an alternative that works in more environments.
Reviewed By: simpkins
Differential Revision: D14691013
fbshipit-source-id: 539e307755c9fc0a098a235868ab622652061494
Summary:
this fetcher knows how to download a URL that references
an archive and extract the sources from it. Compressed tarballs
and zipfiles are supported.
Reviewed By: simpkins
Differential Revision: D14690994
fbshipit-source-id: ffbbd1d9479b0d5aa3e5cd13fe11a560e9d01c2a
Summary: this fetcher knows how to check out sources via git.
Reviewed By: simpkins
Differential Revision: D14691000
fbshipit-source-id: 60f1ffbfed7b32a019aef6aa70ae0903f2782451
Summary:
Fetchers are used to fetch a source directory from a
source defined in a manifest file.
More details can be found in comments on the various methods.
The Manifest class offers a create_fetcher method for constructing
an appropriate fetcher. This is just a stub in this commit, but
will expand over the course of the next few diffs as concrete
fetcher instances are added.
Reviewed By: simpkins
Differential Revision: D14691001
fbshipit-source-id: 8c9038eacae3345e9403d5d1304bf979a9ee1555
Summary:
The build options class contains some environmental and
build related information that will be passed down to fetcher
and builder objects that will be introduced in later diffs.
Reviewed By: simpkins
Differential Revision: D14691007
fbshipit-source-id: e8fe7322f590667ac28a5a3925a072056df0b3e3
Summary:
This will feed into the manifest context for system
dependent manifest sections. This is essentially the same
code borrowed from the watchman and eden getdeps.py.
Reviewed By: simpkins
Differential Revision: D14690997
fbshipit-source-id: 3d3ae146237a2cd49609aaa2bd0e785ebe21f02c
Summary:
this module adds some functions that help with copying
directory trees. The copytree function is aware of eden and will
issue a prefetch prior to walking the directory.
Reviewed By: simpkins
Differential Revision: D14691006
fbshipit-source-id: 079bf850756f61aca17978453d07bc73b2f91788
Summary:
This runs a command, raising an exception if it exits with a non-zero error status.
It prints out the arguments in a mostly copy-and-pasteable form, with PATH-like
env vars pretty printed to make it easier to see what is being invoked; here's
an example of how cmake is being invoked later in this stack:
```
---
+ CMAKE_PREFIX_PATH=\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/ninja-5d7ec7:\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/cmake-91dc9a:\
+ PKG_CONFIG_PATH=\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/ninja-5d7ec7/lib/pkgconfig:\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/cmake-91dc9a/lib/pkgconfig:\
+ cd /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/build/zstd-470344 && \
+ cmake configure /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git/build/cmake -DCMAKE_INST
ALL_PREFIX=/data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/zstd-470344 -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=R
elWithDebInfo -G Ninja
```
Reviewed By: simpkins
Differential Revision: D14690999
fbshipit-source-id: cdb0c681c7dfdfdc6e8c96bf4830bfbcf666411b
Summary:
This is the new getdeps entrypoint. It lives in `opensource/fbcode_builder` so that
it is synced into the various opensource projects at FB.
In the incarnation in this diff is has a single subcommand that can be used to validate
a manifest file.
Reviewed By: sinancepel
Differential Revision: D14690995
fbshipit-source-id: 53f8ec9bafb7b1930970079e3ce20f496ac1c188
Summary:
Add some helpers for manipulating environment variables
that will hold either compiler flags or paths.
These are used in later diffs when setting up arguments/environment
for eg: cmake.
Reviewed By: sinancepel, simpkins
Differential Revision: D14690993
fbshipit-source-id: 7f9753cd99d968550fe9e3ba8b7017a44683061e
Summary:
These are ported over from the logic in the watchman and eden getdeps
scripts, with additions to help bootstrap a build environment.
These are sufficient to build watchman with thrift support on windows, mac and
linux, and eden on mac and linux when combined with the getdeps code that
follows in later diffs in this stack.
Reviewed By: simpkins
Differential Revision: D14691005
fbshipit-source-id: 7f8b02fedcdc020e2d0e758c466959d8161d4587
Summary:
Adds a parser for manifest files that describe projects
that may be either 1st party or 3rd party.
A selection of manifest files appears in a later diff in this stack.
This diff provides the raw parser and a couple of low level helpers.
It also defines a mechanism for validating the schema to help catch
structural (rather than semantic) issues with manifest file contents.
Later diffs in the stack add helpers for accessing the data in
a higher level way.
Reviewed By: sinancepel
Differential Revision: D14691003
fbshipit-source-id: 7d2930a3359ede0f6e21fdc45686b083ab7a9ffa
Summary:
As part of folding getdeps into fbcode_builder, this
expression parser is needed to allow constrained and deterministic
conditionals in the manifest file format.
For example, the watchman manifest will use this cargo-inspired syntax
for system dependent sections:
```
[dependencies]
folly
[dependencies.not(os=windows)]
thrift
```
Reviewed By: sinancepel
Differential Revision: D14691014
fbshipit-source-id: 080bcdb20579da40d225799f5f22debe65708b03
Summary:
This argument is unnecessary since D14222323, which should be deployed
everywhere by now. Passing in the extra repo argument forces hg to open the
repository object twice, which is unnecessary.
We already set the current working directory to the repository when invoking
the import helper, so hg will find the repository correctly from that.
Reviewed By: pkaush
Differential Revision: D15179667
fbshipit-source-id: 838cbee91748c41c713731187608a9823b971a53
Summary:
Update HgImporter to use `hg debugedenimporthelper` instead of the separate
`hg_import_helper.py` script by default on all platforms now. The
`debugedenimporthelper` subcommand has existed in our deployed versions of hg
for a while now.
Reviewed By: pkaush
Differential Revision: D15179668
fbshipit-source-id: 2fb8c4c9f92aed54c84899d6643f746baac73327
Summary:
This diff tries to improve the Thrift Mononoke backing store implementation by:
* Re-creating thrift client for every request since thrift has builtin connection pooling
* Using Eden's CPU thread pool for processing data fetched from remote
* Calling thrift methods within correct executor
Reviewed By: wez
Differential Revision: D15170818
fbshipit-source-id: c8be70755a851f63fb62e4d4f36833703283565e
Summary: We intend to break support for flatmanifest hg, so require that treemanifest is available in the build.
Reviewed By: wez
Differential Revision: D15057150
fbshipit-source-id: 449399cfb9d018f3b722598091eead1bd5d7c77d
Summary: Flatmanifest is on its way out. Remove support for falling back to it if a tree import fails.
Reviewed By: pkaush
Differential Revision: D15056459
fbshipit-source-id: a4df820322ee354d77f50a0ec92e9705d0f152ec
Summary: In the past we used different log level to get more logs. Going back to the original log level. In future, we will be able to set it from the cli.
Reviewed By: chadaustin
Differential Revision: D15159110
fbshipit-source-id: 115718ed667d9896bbd60653f586ae5665598df9
Summary:
Interestingly, the request doesn't fail, but does leave
our stack garbage intact. Let's be sure to zero it out to make
it more obvious what is happening.
While we're in here, let's make sure that we get the same
results from both ends of the socket pair.
Reviewed By: simpkins
Differential Revision: D14994593
fbshipit-source-id: 9aec957dfcd80d88c3d8fbce6bf45480502ea812