Summary:
With this change the getdeps' CargoBuilder will support depencies between cargo builds.
The way how it works is documented in the code and required few assumptions about how a cargo project has to be defined in order to support this.
This change also adds the "mononoke" manifest and few Cargo.toml files to the mononoke project to prove that this new feature works.
Reviewed By: farnz
Differential Revision: D19468912
fbshipit-source-id: f299733cbbc2ec7bca399c898ec8d710334d0fa9
Summary:
Automatically detect the `--facebook-internal` flag based on the current
repository project name.
Reviewed By: wez
Differential Revision: D18621358
fbshipit-source-id: f2b3018169b151811eec455863a8bfc17667d4d8
Summary:
This updates fbcode_builder to try and automatically detect the current
repository's project name by looking for a `.projectid` file in the repository
root. If the project name can be detected:
- The current repository will automatically be used as the source directory
for this project (instead of fetching the sources into the scratch
directory).
- If an explicit project name was not specified on the command line, the
current project will be built by default.
This also changes the repository detection logic to use the current working
directory, rather than the directory where the fbcode_builder code lives.
This will allow this logic to work even if we move fbcode_builder into its own
repository in the future, and have projects depend on it using git submodules.
This does mean that callers need to invoke fbcode_builder.py from inside the
repository.
Reviewed By: wez
Differential Revision: D17088938
fbshipit-source-id: f14d28fdcfaa330ff837ea52b8bdd4e358b81c61
Summary:
Never cache first-party projects that use ShipIt. Previously the code checked
the `shipit_fbcode_builder` property, which controlled whether or not
the `fbcode_builder` sources should be included in the project's ShipIt
mapping. This setting is enabled for most but not all projects that use
ShipIt.
This resulted in projects that use ShipIt but that do not include the fbcode
builder sources being incorrectly cached. This caused getdeps.py to not
run the SimpleShipitTransformerFetcher properly when their sources changed.
Reviewed By: wez
Differential Revision: D17493522
fbshipit-source-id: 57be5ac94ae44f56ccb3ce60ba23fac5d68bce0f
Summary:
This is useful when working on changes to some of the builder functions,
to skip ever trying to use cached results.
Reviewed By: chadaustin
Differential Revision: D17401219
fbshipit-source-id: fb7d5ea45618653957b9bd6a62eed91d8334824d
Summary:
This commit adds a getdeps command that is able to generate
a workflow file for the GitHub Actions CI environment.
The workflow file could be expressed more simply using the matrix
syntax and with three steps (checkout, build, test), but I chose to
break out the steps for each of the dependencies because the UX
while waiting on the build is much nicer that way: the steps show
during and live log tailing for the section of the build that is
underway. If they were all lumped into a single build step then
the logs from the boost section of the build dominate and make
the github UI work very hard.
Pull Request resolved: https://github.com/facebook/watchman/pull/743
Test Plan:
https://github.com/facebook/watchman/pull/743 successfully
executes the github actions CI flow.
```
$ opensource/fbcode_builder/getdeps.py generate-github-actions --output-file watchman/.github/workflows/main.yml watchman
```
Reviewed By: simpkins
Differential Revision: D17384915
Pulled By: wez
fbshipit-source-id: 9a9e5a3e806c18f6cc38ba1cb7059740cda01ad4
Summary:
This diff adds a small abstraction that allows for uploading
and downloading from an artifact cache.
We try to download from this cache at build time, but will only
try to populate it for continuous builds--those are built from
code that has been reviewed and landed on master. This restriction
helps to avoid thrashing the cache with works in progress and
results in a slightly more trustworthy state of the cache contents.
In addition to this, we choose only to cache third party projects.
The rationale is that our first party projects move too quickly to
be worth caching, especially since the cache granularity is for
the whole project rather than just changed elements of a given
project.
In a later diff I will introduce some implementations of the
cache class that work with eg: Travis or Circle CI caching.
Reviewed By: simpkins
Differential Revision: D16873307
fbshipit-source-id: 2bfb69e36615791747b499073586562f2ca48be9
Summary:
Feed the first-party oncall data through to the testpilot
invocation. This will set the owner of the test in continuous runs.
The oncall is passed through via the sandcastle module, which means
that you will have to manually pass it through for local testing
(see test plan), but it should automatically get picked up when
sandcastle schedules the `test` step of the job.
Reviewed By: chadaustin
Differential Revision: D17146802
fbshipit-source-id: a4f0e65853a46ed8709594c96db859ede2530b00
Summary:
Add arguments to getdeps.py to allow overriding the source, build, and install
directories a per-project basis. The arguments take the form `[PROJECT:]PATH`
If the `PROJECT` portion is omitted, it defaults to the current project being
built.
In particular this makes it possible to specify `--src-dir .` to tell
getdeps.py to find the project sources from the current directory rather than
downloading them.
Reviewed By: wez
Differential Revision: D16778011
fbshipit-source-id: f33b87213ace04abb66334f588babdf59df91964
Summary:
Most of the getdeps subcommands operate on a single project, and some of the
argument parsing and initial logic to load the project is largely the same.
This consolidates that logic into a base class so that we can share this code
across subcommands, instead of repeating it.
This also unifies the behavior so that by default all commands enable tests on
the specified project, and disable test on dependent projects. Making sure
all commands use the same behavior here is important as whether are not tests
are enabled can affect the project configuration, and therefore affect its
getdeps hash.
Reviewed By: wez
Differential Revision: D16778010
fbshipit-source-id: 044f99ad6cdd4a56f843276cec8ead786249ee7a
Summary:
Move code that computes project hashes to ManifestLoader. ManifestLoader is
the only class that has all of the information necessary to compute the
project hashes correctly. The ManifestLoader object can also cache previously
computed hashes, so that we don't have to keep computing hashes for projects
over and over again. Previously the `BuildOptions.compute_dirs()` function
would end up re-computing hashes for all dependencies each time it was called.
Reviewed By: strager
Differential Revision: D16477401
fbshipit-source-id: ce03642114f91ce4f859f612e6b2e747cf1653be
Summary:
The ManifestLoader contains all of the state needed to create a fetcher
object, so define a helper method on this object to create a fetcher.
Reviewed By: strager
Differential Revision: D16477395
fbshipit-source-id: 6de0942fe6b8de26c18c82bf99343f5467dc006a
Summary:
Add a new ManifestLoader class to handle loading manifests and computing
dependencies.
For now the main thing this class does is maintain the `manifest_by_name`
mapping. In subsequent diffs we should be able to move some additional logic
into this class, which will help clean up the code and eliminate some redudant
work. In particular, we can have this class cache project hashes, which will
avoid re-computing hashes over and over again for the same projects as we do
in many cases today. We should also be able to save and re-use some of the
project dependency ordering computation in some cases as well.
Reviewed By: strager
Differential Revision: D16477400
fbshipit-source-id: f06f62f77d8443fccaa69fe4c1306e39c395b325
Summary:
Update `BuildOpts.compute_dirs()` to use the correct project-specific manifest
context when computing project hashes. Previously it was incorrectly using
the initial project's context when evaluating all dependencies. This would
result in some projects potentially seeing the wrong values for variables that
may change from project to project (like `test`).
Reviewed By: pkaush
Differential Revision: D16477398
fbshipit-source-id: 6c23f5e5e19b2402000a138b3920b79044446041
Summary:
Add a ContextGenerator class so that we actually use the correct per-project
context when loading projects and computing dependencies.
Previously commands like `build` and `test` would change the contexts for each
project as they iterated through and performed the build. However, they did
not do this when first loading the projects. This could cause them to use
different context values when loading dependencies than when performing the
build. For instance, this could cause issues if a project depends on
`googletest` only when testing is enabled, as the code previously did not set
the "test" parameter when evaluating dependencies.
Reviewed By: pkaush
Differential Revision: D16477396
fbshipit-source-id: c1e055f07de1cb960861d19594e3bda20a2ccd87
Summary:
getdeps allowed common arguments to be specified either before or after
the subcommand name. However, the arguments were only actually honored
if they were specified on the command line after the subcommand name.
Specifying any of these arguments before the subcommand would simply be
ignored, and the default value would be used. This happened since the
subcommand argument's default value overwrote the value specified by the
user from the main command parser.
This fixes the behavior by using `argparse.SUPPRESS` so that argparse
won't set the value at all if it was not specified. This allows the
value from the main parser to be used correctly if the argument was not
seen by the subcommand parser.
Reviewed By: chadaustin
Differential Revision: D16435358
fbshipit-source-id: f3893ef7a27fb018247f249f313d6438472e0140
Summary:
This diff adds a `fixup-dyn-deps` subcommand that is intended to
aid in packaging on multiple platforms.
Its purpose is to copy a set of executable object files from the getdeps
installation directories and place them into an installation staging
area that will then be used to create some kind of package (rpm, tarball etc.).
The dynamic dependencies of the executables are determined and also copied
into the destination area, and the important part: the execute is rewritten
such that it will load the deps out of an alternate installation prefix.
The implementation of this command draws on similar scripts in use for
the watchman and eden packaging on windows and macos. This diff adds
linux support using the `patchelf` utility.
Reviewed By: pkaush
Differential Revision: D16101902
fbshipit-source-id: 5885125971947139407841e08c0cf9f35fdf5895
Summary: On Windows "--vcvars-path" can be passed to point to the toolchain we want to use.
Reviewed By: wez
Differential Revision: D15926044
fbshipit-source-id: 2b0cde793f7c7f8473b78afde8794640bae351f3
Summary:
with the reshuffling around getdeps.py, and changes in folly/thrift/wangle,
our internal package stopped building. This diff brings it back to life. It is
a little bit interwined:
* Introduce a `--facebook-internal` switch to getdeps that causes `fb` in the
build context to be set to `on` rather than `off`. This allows specifying
options in the manifest that are FB specific.
* Introduce cmake options to control the WATCHMAN_STATE_DIR and whether cmake
should attempt to create WATCHMAN_STATE_DIR when it is installed.
* For FB specific builds on macOS, set the state dir to match our existing location
and disable creation of WATCHMAN_STATE_DIR because that will ultimately be
handling during package installation and we don't have privs for that on
the machine generating the package.
* Adjust Facebook CI scripts to use getdeps for building and testing watchman
Reviewed By: simpkins
Differential Revision: D15337970
fbshipit-source-id: 8ab3c1f810decf5f0b16337dc1cbe6dc73d26426
Summary:
The schedule type is used to differentiate between a build run during
code review (and thus runs code that isn't yet in the master branch),
from continuous or other types of run that operate on landed code.
This doesn't change any behavior yet; this diff just adds
the plumbing to pass down an optional arbitrary schedule type string
from the CI system.
In the future, we'll use the schedule type to influence the behavior
of running tests.
Reviewed By: strager
Differential Revision: D15300120
fbshipit-source-id: 3b46afef2ff171b3fa095763dd5006a54ea328b8
Summary:
The goal is to return an error code > 127 in the case of a
transient, retryable, infrastructure error. This diff generates
those in the case of failure in downloading a URL or from interacting
with LFS.
Reviewed By: strager
Differential Revision: D15266838
fbshipit-source-id: 4f52a791320123968869032c37912dded464a86e
Summary:
This controls whether tests are built or not.
They are not built by default. When `--enable-tests` is turned on,
tests are enabled for the named project only, not all of the deps.
This results in a faster build, because eg: the number of tests in
folly is very large and consumers of folly don't want to spend so
much time waiting to build tests when really all they want to do
is build their own project.
Reviewed By: strager
Differential Revision: D15246336
fbshipit-source-id: 2e22fd60c983b059f71639602712087f2a188ec6
Summary:
This is needed to correctly pick up changes made to
eg: cmake.defines sections in the manifest for first-party
projects.
Reviewed By: strager
Differential Revision: D15246337
fbshipit-source-id: 35e525e885f87d6136d5ff3b94ebf34516ab947c
Summary:
This is useful especially on Windows where the up-to-date
checks for the dependencies take a long time.
The idea is that you might run this to start:
```
$ getdeps.py build eden
```
and then while in the edit/debug/build iteration cycle:
```
$ getdeps.py build --no-deps eden
```
Reviewed By: pkaush
Differential Revision: D15200352
fbshipit-source-id: 086f2f49db967ef4d1914a69fa80067104d79136
Summary:
This prints out the installation or source prefix for a given project.
This is useful in eg: packaging scripts to figure out where they can
find the built artifacts.
Reviewed By: simpkins
Differential Revision: D14967378
fbshipit-source-id: 7e1b5de2ca7219af24cfb07b4b42de22aa410469
Summary:
previously, a relatively lame hash was computed to use
for the build directory based on some hash of the source directory.
That was good enough to get things off the ground, but in the
interest of being able to cache the build outputs and safely
invalidate them we need a slightly more rigorous implementation.
This diff computes a hash based on the manifest contents and
relevant environmental factors.
The hash is used to name the build directory which will ultimately
be cached eg: using the travis/appveyor cache directory configuration
and some other means for the FB internal CI.
The hash needs to be sufficient that we change the hash when
the manifests change. We can tolerate a false positive change
in hash (it just means that a build will take longer), but
cannot tolerate a false negative (which would result in an
incorrect build).
Reviewed By: simpkins
Differential Revision: D14710332
fbshipit-source-id: ebc2e74eafc6f3305d4412a82195bc9fb9dfa615
Summary:
This command schedules a facebook specific sandcastle job for the current
commit in your repo for each of the platforms we have support for in
sandcastle.
You can use `--dry-run` to have it print out the job specs.
To support this, I've moved around some of the support utilities
to make it easier to import them.
Reviewed By: simpkins
Differential Revision: D14710330
fbshipit-source-id: fb1e2a2ce78e52894291159514977da97028b37f
Summary:
Adds a `test` subcommand that runs the tests for project.
We're mostly interested in the 1st party facebook projects for this.
The `sandcastle` flow will run the `test` subcommand just for the "leaf" project--the one named on the command line.
Reviewed By: simpkins
Differential Revision: D14710331
fbshipit-source-id: 7d04a46cfd723894d61018de2f230140b52285ac
Summary:
This fixes a TODO; in our CI environment we want to use the
real shipit, so we'll use this flag to make that happen.
Reviewed By: simpkins
Differential Revision: D14695576
fbshipit-source-id: 64ee72c210e2472d295dcbd39c86549273b68452
Summary:
adds the command that is used to drive a build.
The dependencies of the named project are computed using
the same mechanism behind the `list-deps` subcommand.
If a manifest specifies that it uses the `cmake` builder then
a dependency on `cmake` is synthesized so that we can download
a recent version of cmake ahead of time.
The build command uses the fetcher to update the src and then
executes the builder.
Reviewed By: simpkins
Differential Revision: D14691011
fbshipit-source-id: 31ae59614651ef021a9505e89c13b5717b440071
Summary:
While the command isn't necessarily super useful
on its own, it does show that the plumbing for walking the
deps is functioning, and that is important when it comes
to building.
The output lists the projects in the order that they
would be built.
The `fetch` command has been augmented to add a `--recursive`
flag that uses the same mechanism to recursively fetch
the dependencies.
Reviewed By: simpkins
Differential Revision: D14691004
fbshipit-source-id: b00bf6ad4742f8bb0a70698f71a5fe03d6a1f453
Summary:
Adds a command that can be used to trigger a fetch for a project
```
$ ./opensource/fbcode_builder/getdeps.py fetch zstd
Cloning https://github.com/facebook/zstd.git...
---
+ git clone --depth=100 https://github.com/facebook/zstd.git /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git
Cloning into '/data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git'...
remote: Enumerating objects: 3816, done.
remote: Counting objects: 100% (3816/3816), done.
remote: Compressing objects: 100% (1415/1415), done.
remote: Total 3816 (delta 2556), reused 3312 (delta 2288), pack-reused 0
Receiving objects: 100% (3816/3816), 2.93 MiB | 9.59 MiB/s, done.
Resolving deltas: 100% (2556/2556), done.
Updating /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git -> v1.3.8
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git fetch origin v1.3.8
From https://github.com/facebook/zstd
* tag v1.3.8 -> FETCH_HEAD
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git checkout FETCH_HEAD
Note: checking out 'FETCH_HEAD'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 470344d Merge pull request #1479 from facebook/visualTest
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git submodule update --init
```
Reviewed By: simpkins
Differential Revision: D14691008
fbshipit-source-id: 3afa391360518a08ebd6ff97f5b8b4993f10c4e8
Summary:
The build options class contains some environmental and
build related information that will be passed down to fetcher
and builder objects that will be introduced in later diffs.
Reviewed By: simpkins
Differential Revision: D14691007
fbshipit-source-id: e8fe7322f590667ac28a5a3925a072056df0b3e3
Summary:
This will feed into the manifest context for system
dependent manifest sections. This is essentially the same
code borrowed from the watchman and eden getdeps.py.
Reviewed By: simpkins
Differential Revision: D14690997
fbshipit-source-id: 3d3ae146237a2cd49609aaa2bd0e785ebe21f02c
Summary:
This is the new getdeps entrypoint. It lives in `opensource/fbcode_builder` so that
it is synced into the various opensource projects at FB.
In the incarnation in this diff is has a single subcommand that can be used to validate
a manifest file.
Reviewed By: sinancepel
Differential Revision: D14690995
fbshipit-source-id: 53f8ec9bafb7b1930970079e3ce20f496ac1c188
