Summary:
This just strips off a trailing '/' from the `INSTALL_DIR` property that we
set on python targets, to avoid having double-slashes in the paths that we use
during installation. This shouldn't really have any material difference other
than cleaning up the paths that get printed during the installation phase.
Reviewed By: chadaustin
Differential Revision: D17089207
fbshipit-source-id: ab36bb76c19fa60fe037f7a5290ccfd6bdbf13b0
Summary:
Update fboss to use `ThriftCppLibrary.cmake` directly from fbcode_builder
rather than maintaining its own copy of this file.
Reviewed By: chadaustin
Differential Revision: D17005421
fbshipit-source-id: a64df426118df6088b47f09410dad7b8b7e79a43
Summary:
Update `add_thrift_cpp2_library()` to pass in the correct `-I` flags when
invoking the thrift compiler so that it can find all of the other thrift files
that this library depends on.
D16062657 was a previous attempt to do this, but suffered from a few problems:
- It required all dependencies to be defined before
`add_thrift_cpp2_library()` was called. This requires users to carefully
order their CMake files and subdirectory include ordering.
- It only handled one level of dependencies, and did not propagate include
paths for deeper dependencies.
- It set the include path for dependencies to the source directory path where
the dependency was originally built, rather than the directory where the
thrift file for that dependency would be installed.
This change does require CMake 3.8+. Previous versions of CMake do not
support using generator expressions to generate multiple arguments for a
custom command.
Reviewed By: strager
Differential Revision: D17005381
fbshipit-source-id: 31190beba94b4d1010445375a5e2791450230f7d
Summary:
All of the requisite headers are now open sourced and eden
has been converted to use them. This removes any logic that references
this directory and kills the now unused cmake file.
Reviewed By: chadaustin
Differential Revision: D16994732
fbshipit-source-id: 573b8b50540ee64590682be2a54aa94659f12368
Summary:
Most stat code in fbcode was moved in to open source under
fbcode/fb303. The only exception that we use internally is the
MonotonicCounter class. This diff moves all of our stats code to use
the open source headers and updates cmake files to pull in the open
source fb303.
Reviewed By: chadaustin, shri-khare
Differential Revision: D16969960
fbshipit-source-id: 12b7abb54c956c242c8e27eb69fd96925c7e61f7
Summary:
Update add_thrift_cpp2_library() to add a PUBLIC_HEADER property to the
generated library target, so that the generated headers can be installed using
an `install()` call. Also add a `HEADER_INSTALL_DIR` property which indicates
the directory they should be installed into. A `INCLUDE_DIR` argument was
added to customize the include installation path, should anyone care to do so.
This also removes code that was previously incorrectly installing the
generated headers into the source tree.
Reviewed By: wez
Differential Revision: D16853404
fbshipit-source-id: 6f640b2bd347c99804e074fca0209dda37c8a9cf
Summary:
The external Travis CI builds for fbzmq are still using the older
fbcode_builder spec (as opposed to the newer getdeps manifest file).
D16577367 moved the CMakeLists.txt file to the top-level directory, but the
fbzmq spec file was still looking for it in the fbzmq subdirectory.
Reviewed By: jstrizich
Differential Revision: D17005361
fbshipit-source-id: 3f7664eadfb60ec7606124a14445b44ae586b8a7
Summary:
Fix the fbzmq pathmap listed in its manifest file to match the path map
actually used by ShipIt. The fact that this was broken was why internal
getdeps builds did not detect that D16577367 broke the build.
Reviewed By: wez, jstrizich
Differential Revision: D17005360
fbshipit-source-id: 046ff58ad70c03b860c3fccebaba975808df244d
Summary:
Update ThriftCppLibrary.cmake to record that the output depends on the thrift
`cpp2` template files. The `THRIFT_TEMPLATE_FILES` variable was defined in
D16062657 but doesn't appear to be used anywhere. This uses it, and also
updates it to only match the C++ template files.
Reviewed By: wez
Differential Revision: D16738442
fbshipit-source-id: aa4097e9ec1d009ef1e3ee2efd03ffa19a82ed12
Summary:
Add explicit double quotes around arguments that are intended to be exactly a
single parameter. Also line wrap to 80 characters.
Reviewed By: wez
Differential Revision: D16738443
fbshipit-source-id: b8c9b6d50da72b44e23eaf5effbc384b6cbbc1c0
Summary:
Change ThriftCppLibrary to use `fb_cmake_parse_args()`
This code doesn't really care much about handling empty arguments, so we could
use the standard `cmake_parse_arguments()` function here instead of
`fb_cmake_parse_args()`. However, it is slightly more convenient that
`fb_cmake_parse_args()` automatically errors out on unhandled arguments, so
that we don't have to check for this ourselves.
Reviewed By: wez
Differential Revision: D16738445
fbshipit-source-id: 67b9761f2ceaf7665bd336290bb44af32fec6f4a
Summary:
The manifest file for iproute2 previously listed the commit to check out as
7ca63aef7d1b0c808da0040c6b366ef7a61f38c1. However, this is not a commit, but
a tag object that refers to commit 92a0236a3cdf3438000834121b7ea8a09f1f52b1.
This caused getdeps to think that iproute2 was always out-of-date, even when
it was already on the correct commit, since it was comparing current commit ID
against the tag ID.
This updates the manifest to refer to the commit ID instead of the tag ID.
Reviewed By: shri-khare
Differential Revision: D17005418
fbshipit-source-id: c7145c73b66ae93a2e9097ddcf8c53237b135d32
Summary:
This simplifies things for the OSS CI situation in
the short term, especially since Eden isn't quite buildable
or usable outside FB at this time.
Reviewed By: chadaustin
Differential Revision: D16953346
fbshipit-source-id: 028150bb3cfc9a8ccbe1197321214b56ad359463
Summary:
does not affect the github build.
this is for fb CI
Reviewed By: dddmello
Differential Revision: D16935982
fbshipit-source-id: 61656fa600f19438b5aa89486486ed4ee103dea4
Summary:
In the Facebook infrastructure the testpilot runner talks
to a service to determine whether a given test should be run locally.
The remote service was returning an error for some tests because they
didn't have enough configuration specified.
This diff associates some configuration with the tests based on the
host type and makes things happier.
Reviewed By: Ben0mega
Differential Revision: D16894682
fbshipit-source-id: b6f59b112991160ef26cc126e98041d8908a6273
Summary:
Add some CMake functions for building standalone executables from Python
source files. This generates executables similar to PEX
(https://github.com/pantsbuild/pex).
In the future this could potentially be leveraged to directly build XAR files
(https://github.com/facebookincubator/xar).
The main advantages of these functions is that they allow easily defining
"libraries" of python files, and their dependencies, which can then be used
and packaged as part of multiple different standalone executables.
Reviewed By: wez
Differential Revision: D16722499
fbshipit-source-id: e1d829b911dc428e5438b5cf9cebf99b3fb6ce24
Summary:
Update the generated `run_cmake.py` script to allow the caller to specify that
they just want to run a build without the install step.
Reviewed By: wez
Differential Revision: D16778007
fbshipit-source-id: 1859aca2b80fa7b099b4790682a6508e0185f2a0
Summary:
This cleans up how the `CMAKE_ENV` and `CMAKE_DEFINE_ARGS` variables are
written in the generated `run_cmake.py` script that we emit for CMake-based
projects.
We now emit each entry in these variables on separate lines, just to improve
readability. (Both of these variables tend to have a number of entries and
are very long if emitted on a single line.)
This also replaces the `-DCMAKE_INSTALL_PREFIX` entry in `CMAKE_DEFINE_ARGS`
to have it correctly honor the `INSTALL_DIR` variable defined in
`run_cmake.py`. This makes `run_cmake.py` still do the right thing if someone
manually edits it to change the `INSTALL_DIR` value.
Reviewed By: wez
Differential Revision: D16778006
fbshipit-source-id: fee5d25748b87b5d9c57ee2edf8de5e586e872ee
Summary:
Add arguments to getdeps.py to allow overriding the source, build, and install
directories a per-project basis. The arguments take the form `[PROJECT:]PATH`
If the `PROJECT` portion is omitted, it defaults to the current project being
built.
In particular this makes it possible to specify `--src-dir .` to tell
getdeps.py to find the project sources from the current directory rather than
downloading them.
Reviewed By: wez
Differential Revision: D16778011
fbshipit-source-id: f33b87213ace04abb66334f588babdf59df91964
Summary:
This makes getdeps.py no longer crash if used in the folly repository.
Folly does not depend on any other Facebook projects, so it does not
include a `build/deps` directory.
Reviewed By: wez
Differential Revision: D16778009
fbshipit-source-id: 41be53d862f41b62154b081eb90ddba8742658fe
Summary:
Most of the getdeps subcommands operate on a single project, and some of the
argument parsing and initial logic to load the project is largely the same.
This consolidates that logic into a base class so that we can share this code
across subcommands, instead of repeating it.
This also unifies the behavior so that by default all commands enable tests on
the specified project, and disable test on dependent projects. Making sure
all commands use the same behavior here is important as whether are not tests
are enabled can affect the project configuration, and therefore affect its
getdeps hash.
Reviewed By: wez
Differential Revision: D16778010
fbshipit-source-id: 044f99ad6cdd4a56f843276cec8ead786249ee7a
Summary:
Fix the BuildOptions class to correctly honor the `host_type` parameter that
it was constructed with when constructing the manifest evaluation context.
I accidentally broke this behavior in D16477396, and incorrectly had this code
path default to using the current host system rather than the value passed in
from the command line.
Reviewed By: wez
Differential Revision: D16779579
fbshipit-source-id: de911daaa643f6303fd35149775ab25d3f64d34f
Summary:
In response to review feedback for D16477400 and D16477401, update
`ManifestLoader.load_all_manifests()` to only update its data for projects
that have not previously been loaded. This helps ensure that code using a
single `ManifestLoader` object cannot have two in-memory `Manifest` objects
for the same project, and that existing data (such as project hashes) can't be
invalidated if a manifest is later loaded from updated on-disk data.
Reviewed By: pkaush
Differential Revision: D16586682
fbshipit-source-id: 50b1979ec55f2ad6901629cd852293a8f6ca903f
Summary:
While developing on a project it is often convenient to be able to invoke its
build manually, rather than always needing to re-run `getdeps.py`. This
updates the CMakeBuilder to also emit a script that can be used to manually
run CMake outside of `getdeps.py`.
The CMakeBuilder is the only builder that this really matters for right now,
as pretty much all of the projects where we do first-party development use
CMake for their build system.
Reviewed By: pkaush
Differential Revision: D16477399
fbshipit-source-id: c8a14af158af7b32d6c799ef685b037e68b748ff
Summary:
Move code that computes project hashes to ManifestLoader. ManifestLoader is
the only class that has all of the information necessary to compute the
project hashes correctly. The ManifestLoader object can also cache previously
computed hashes, so that we don't have to keep computing hashes for projects
over and over again. Previously the `BuildOptions.compute_dirs()` function
would end up re-computing hashes for all dependencies each time it was called.
Reviewed By: strager
Differential Revision: D16477401
fbshipit-source-id: ce03642114f91ce4f859f612e6b2e747cf1653be
Summary:
The ManifestLoader contains all of the state needed to create a fetcher
object, so define a helper method on this object to create a fetcher.
Reviewed By: strager
Differential Revision: D16477395
fbshipit-source-id: 6de0942fe6b8de26c18c82bf99343f5467dc006a
Summary:
Add a new ManifestLoader class to handle loading manifests and computing
dependencies.
For now the main thing this class does is maintain the `manifest_by_name`
mapping. In subsequent diffs we should be able to move some additional logic
into this class, which will help clean up the code and eliminate some redudant
work. In particular, we can have this class cache project hashes, which will
avoid re-computing hashes over and over again for the same projects as we do
in many cases today. We should also be able to save and re-use some of the
project dependency ordering computation in some cases as well.
Reviewed By: strager
Differential Revision: D16477400
fbshipit-source-id: f06f62f77d8443fccaa69fe4c1306e39c395b325
Summary:
Update `BuildOpts.compute_dirs()` to use the correct project-specific manifest
context when computing project hashes. Previously it was incorrectly using
the initial project's context when evaluating all dependencies. This would
result in some projects potentially seeing the wrong values for variables that
may change from project to project (like `test`).
Reviewed By: pkaush
Differential Revision: D16477398
fbshipit-source-id: 6c23f5e5e19b2402000a138b3920b79044446041
Summary:
Add a ContextGenerator class so that we actually use the correct per-project
context when loading projects and computing dependencies.
Previously commands like `build` and `test` would change the contexts for each
project as they iterated through and performed the build. However, they did
not do this when first loading the projects. This could cause them to use
different context values when loading dependencies than when performing the
build. For instance, this could cause issues if a project depends on
`googletest` only when testing is enabled, as the code previously did not set
the "test" parameter when evaluating dependencies.
Reviewed By: pkaush
Differential Revision: D16477396
fbshipit-source-id: c1e055f07de1cb960861d19594e3bda20a2ccd87
Summary:
Check that all variable names are valid when loading manifest files.
This ensures that getdeps.py will error out if someone makes a typo in a
variable name, rather than treating it as never equal to anything.
Reviewed By: pkaush
Differential Revision: D16477397
fbshipit-source-id: 030e0642ff4a08db8eb74a0a0223e03d53e4880f
Summary: bring this in line with other facebook opensource projects
Reviewed By: saifhhasan
Differential Revision: D16577367
fbshipit-source-id: d762658505f824cc180c55ea4485cecf525b8fdc
Summary:
Simply linking `GMOCK_LIBS` into a binary was not telling CMake that the binary depends on `gmock` being built. So, let's add that dependency explicitly.
This wasn't breaking in production because we typically build with `-j 4`, and `gmock` was getting built before the first dependent binary would attempt to link.
Also, since `rsocket` bundles its own `gmock`, it is just a waste of time to compile a system-level gmock. It's not a real dependency.
NB: The change in `fbcode_builder.py` is needed because now that `rsocket` no longer depends on anything on Github, driver programs that were unconditionally setting `projects_dir` started to fail to build `rsocket`.
Reviewed By: simpkins
Differential Revision: D16461572
fbshipit-source-id: 1e95654e96256e7ed37d42e702b5433bf2fe5328
Summary:
flake8 (in Python 2.7 mode) complains that `typing` is mentioned in type annotations but is not defined:
```
fbcode_builder/getdeps/buildopts.py:251:21: F821 undefined name 'typing'
subst_mapping, # type: typing.Mapping[str, str]
^
fbcode_builder/getdeps/buildopts.py:253:5: F821 undefined name 'typing'
# type: (...) -> typing.Optional[str]
^
2 F821 undefined name 'typing'
2
```
Import `typing` explicitly to silence this warning.
Because `typing` may be unavailable, import it conditionally. (Because it's only referenced in comments, failing to import `typing` should have no effect at run time.)
Reviewed By: snarkmaster
Differential Revision: D16435696
fbshipit-source-id: 78a4a7b07acc46aa998f02b54b1a6e52c1daafde
Summary: Add a dependency from the eden open source build to the fb303 open source build and switch EdenServiceHandler to BaseService.
Reviewed By: simpkins
Differential Revision: D15528156
fbshipit-source-id: 2ca5c31dd9fcc9bac43fd399b27f33b6f2c5ebfc
Summary:
Some people want to use getdeps with Python 2.7. This looks easy to do, so take a step toward Python 2.7 support by fixing getdeps' tests when run with Python 2.7.
For Python 3, this diff should not change behavior.
This diff should address https://github.com/facebook/bistro/issues/35.
Reviewed By: snarkmaster
Differential Revision: D16435667
fbshipit-source-id: f5c262b12995b609263341c4de26dac7f9b12b70
Summary:
getdeps allowed common arguments to be specified either before or after
the subcommand name. However, the arguments were only actually honored
if they were specified on the command line after the subcommand name.
Specifying any of these arguments before the subcommand would simply be
ignored, and the default value would be used. This happened since the
subcommand argument's default value overwrote the value specified by the
user from the main command parser.
This fixes the behavior by using `argparse.SUPPRESS` so that argparse
won't set the value at all if it was not specified. This allows the
value from the main parser to be used correctly if the argument was not
seen by the subcommand parser.
Reviewed By: chadaustin
Differential Revision: D16435358
fbshipit-source-id: f3893ef7a27fb018247f249f313d6438472e0140
Summary:
If `path_search()` returns that CMake is not available, raise an Exception and
fail the build. This makes the failure somewhat easier to to identify.
Without this the code would continue and would try to invoke `subprocess` with
a value of `None` in the command argument list. This error is slightly harder
to debug, since it isn't clear which command argument or environment variable
is not a string.
Reviewed By: chadaustin
Differential Revision: D16354623
fbshipit-source-id: be972b02cb13bc70db0f867da70e5bf4c6cec46d
Summary:
The scratch path is used as part of the hash computation for each project. We
need to make sure this path is always normalized to ensure that we compute the
hashes consistently.
Reviewed By: chadaustin
Differential Revision: D16354624
fbshipit-source-id: 39b5362620bdc247cd7e7f1333dac319b354dc6f
Summary:
getdeps currently ends up calling `path_search()` repeatedly searching for
various C/C++ compilers in $PATH. It ends up doing this multiple times for
each dependency as it computes the project hashes. This updates the
`path_search()` function to cache its results so that we don't keep performing
the same searches on the file system over and over again.
Reviewed By: chadaustin
Differential Revision: D16354625
fbshipit-source-id: 116293bd2f636632517d26436b2332e6c10624f1
Summary: eden.thrift includes fb303_core.thrift, so any cmake target that depends on eden.thrift should pull in fb303_core.thrift's include directories and libraries. Implement that machinery in CppThriftLibrary.cmake.
Reviewed By: wez
Differential Revision: D16062657
fbshipit-source-id: d5d962960e767a138a9b634a12aebccf72d6ef43
Summary: Add a manifest and CMakeLists.txt for building fb303 with cmake.
Reviewed By: wez
Differential Revision: D15480895
fbshipit-source-id: d47f6ef9b9383b79b31a6170c7d4c9e8337de4d8
Summary:
Fix an error about source directories being included in the include
path in add_thrift_cpp2_library. I don't know why this was failing for
fb303 and not for other projects, but adding a generation expression
appears to fix the issue.
Reviewed By: strager
Differential Revision: D15725024
fbshipit-source-id: 6c02fed6c6703733cf9e0b130c0f90b70e3ea300
Summary:
Fix dyndeps.py to find dumpbin.exe in more situations.
Previously the code looked for dumpbin.exe under Visual Studio directories
named either `BuildTools` or `Community`. On a system with an MSVC 2017
Professional install it is instead located under a directory named
`Professional`. This updates the glob to allow any directory name here.
Reviewed By: pkaush
Differential Revision: D16207692
fbshipit-source-id: 1a57ec29653a89fd3e751b3e4408a298b4632b11
Summary: is_objfile() is used to find the executable files in the given project. Getdeps will only find and copy the dependencies of the binaries identified by this function. On Windows we will copy only the dependencies for an exe file.
Reviewed By: chadaustin
Differential Revision: D16185962
fbshipit-source-id: f6b5089401b242514a845e3a97b3804051d93c1c
Summary:
Mapping scratch dir <from> -> <to> show up in the output of all the show dir commands on Windows, so removing it.
example: getdeps.py show-inst-dir eden
Reviewed By: wez
Differential Revision: D16092494
fbshipit-source-id: 910288a8d23c1d68c5e70b7b2dbb36ef53a326fc
Summary: This adds a subclass to copy the dynamic dependencies on Windows.
Reviewed By: wez
Differential Revision: D16110433
fbshipit-source-id: 14d876947f3ec504382fef0d459367a7119ff6cb
Summary:
This diff adds a `fixup-dyn-deps` subcommand that is intended to
aid in packaging on multiple platforms.
Its purpose is to copy a set of executable object files from the getdeps
installation directories and place them into an installation staging
area that will then be used to create some kind of package (rpm, tarball etc.).
The dynamic dependencies of the executables are determined and also copied
into the destination area, and the important part: the execute is rewritten
such that it will load the deps out of an alternate installation prefix.
The implementation of this command draws on similar scripts in use for
the watchman and eden packaging on windows and macos. This diff adds
linux support using the `patchelf` utility.
Reviewed By: pkaush
Differential Revision: D16101902
fbshipit-source-id: 5885125971947139407841e08c0cf9f35fdf5895
Summary: I want to use this logic outside of a builder implementation
Reviewed By: pkaush
Differential Revision: D16101914
fbshipit-source-id: db3c9ac6c84a92ab84a18dddb931953b0a51f127
Summary:
The intent is to use this in a future diff to make packaging a first
class concept; it can be used similarly to how we munge dynamic deps on darwin
systems.
Reviewed By: shri-khare
Differential Revision: D16099931
fbshipit-source-id: c66f994b7c07e903fbf2989946b17a0991e12a91
Summary:
We're seeing a failure on macOS in a CI environment but don't have much context:
```
Stderr:
+ GETDEPS=opensource/fbcode_builder/getdeps.py
+ opensource/fbcode_builder/getdeps.py test --facebook-internal watchman
Traceback (most recent call last):
File "opensource/fbcode_builder/getdeps.py", line 436, in <module>
sys.exit(main())
File "opensource/fbcode_builder/getdeps.py", line 422, in main
return args.func(args)
File "opensource/fbcode_builder/getdeps.py", line 342, in run
builder.run_tests(install_dirs, schedule_type=args.schedule_type)
File "/data/sandcastle/boxes/trunk-hg-fbcode-fbsource/fbcode/opensource/fbcode_builder/getdeps/builder.py", line 372, in run_tests
buck_test_info = list_tests()
File "/data/sandcastle/boxes/trunk-hg-fbcode-fbsource/fbcode/opensource/fbcode_builder/getdeps/builder.py", line 352, in list_tests
data = json.loads(output.decode("utf-8"))
File "/opt/homebrew/Cellar/python27/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/__init__.py", line 338, in loads
return _default_decoder.decode(s)
File "/opt/homebrew/Cellar/python27/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 366, in decode
obj, end = self.raw_decode(s, idx=_w(s, 0).end())
File "/opt/homebrew/Cellar/python27/2.7.10_2/Frameworks/Python.framework/Versions/2.7/lib/python2.7/json/decoder.py", line 384, in raw_decode
raise ValueError("No JSON object could be decoded")
ValueError: No JSON object could be decoded
```
Add some more context to that error message so we can learn more.
Reviewed By: pkaush
Differential Revision: D16061528
fbshipit-source-id: 23603a5d18651d20641ef1987b7094e73a9b1dbe
Summary: On Windows "--vcvars-path" can be passed to point to the toolchain we want to use.
Reviewed By: wez
Differential Revision: D15926044
fbshipit-source-id: 2b0cde793f7c7f8473b78afde8794640bae351f3
Summary:
Remove all the autoconf files.
And fully switch to the cmake build
Reviewed By: udippant
Differential Revision: D16066986
fbshipit-source-id: 9e3082dded77ce85449cf5a3a03bed31c16b711f
Summary:
The source we were downloading from (netfilter.org) should work, but
unfortunately, is donw. This is causing getdeps FBOSS builds to fail. For now,
download from a mirror which seems to be up.
If this becomes a frequent issue with this repository/others, we can consider a
different solution.
Also, not sure how lfs pointer edit was missed by previous commit, added here.
Reviewed By: phshaikh
Differential Revision: D16070052
fbshipit-source-id: dd9278254c77e1a5845ca331a9eeb36f2bf03071
Summary:
This is a hack for now to allow fbsource-based watchman getdeps builds to
succeed. In the long run we need to update Watchman's CMakeLists.txt to
depend on a proper EdenFS build, rather than copying Eden's thrift files into
Watchman's source tree.
Reviewed By: wez
Differential Revision: D15753320
fbshipit-source-id: 2dea483c2053c4ea1cf64021c4bbc3239fce645d
Summary:
The previous diffs in the stack add fbcode_builder manifests for FBOSS
dependencies iproute2 and OpenNSL. The other FBOSS dependencies already have
fbcode_builder support. Thus, add FBOSS manifest.
Additionally, this patch also modifies fboss CMakeLists.txt to get FBOSS to
build using fbcode_builder.
Reviewed By: wez
Differential Revision: D15626298
fbshipit-source-id: 31802ef695ae69b08f526bbeb299c59f3b9d44be
Summary:
D15683387 introduced manifest for libcurl and used cmake instead of autoconf
(rationale: having cmake build curl would make it easy as we want curl to build
on windows and macos).
if libcurl finds ldap.h, then it builds with ldap enabled.
On my devserver, ldap library is not installed, and thus libcurl builds without
it, and links with FBOSS binaries fine.
In sandbox, ldap seems to be installed, thus libcurl builds with ldap. However,
linking fails for FBOSS binaries as those don't link against ldap - FBOSS does
not depend on ldap.
Thus, we need a way to configure libcurl with ldap disabled. [cmake.defines]
CURL_DISABLE_LDAP=OFF, does not quite do it. Thus, am going with autoconf, with
args --disable-ldap for now for Linux and cmake for non-Linux environments.
Reviewed By: wez
Differential Revision: D15730947
fbshipit-source-id: 2ce48976ee785401d550720cbdccb22f9d416675
Summary:
D15588809 introduced manifest and customer builder for iproute2. While the
Iproute2Builder copies include/* from build to install directory, it does not
copy lib/* which contains the libnetlink.a
As a result, the build system looks up libnetlink.a on local machine (which it
may not find).
Fix it by explicitly coping lib/* as well.
Reviewed By: wez
Differential Revision: D15708835
fbshipit-source-id: 3f5906eb48ce982b6e887cca0d3c11314425beab
Summary: This diff allows zmq socket to run on folly::fibers
Reviewed By: saifhhasan
Differential Revision: D15113447
fbshipit-source-id: 8a0e1f91d719a9deda48837d4ff1065408280b72
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
libnl is one of the dependencies for FBOSS. This patch adds a manifest
file to build libnl.
Reviewed By: wez
Differential Revision: D15686853
fbshipit-source-id: f7549df6dc1005630193b024be6e1ea330cc6646
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
libcurl is one of the dependencies for FBOSS. This patch adds a manifest
file to build libcurl.
Reviewed By: wez
Differential Revision: D15683387
fbshipit-source-id: 8df5c413e7dff06d1a19e0ce3b1706bff2f1ab54
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
libusb is one of the dependencies for FBOSS. This patch adds a manifest
file to build libusb. FBOSS does not need libudev, so we avoid unncessary
dependency by passing right argument to ./configure.
Reviewed By: wez
Differential Revision: D15683386
fbshipit-source-id: 1d35caecf4012ddc0df24a7305019d522b4763fa
Summary:
We sometimes see some flakeyness in our internal CI, so we can live
without it there.
Reviewed By: pkaush
Differential Revision: D15695124
fbshipit-source-id: 1d76ae89e245d9c95937e8818826c544c0ae8fc3
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
libmnl is one of the dependencies for FBOSS. This patch adds a manifest file
to build the specific version of libmnl for FBOSS.
Reviewed By: wez
Differential Revision: D15633176
fbshipit-source-id: dd51997ae950f53842711fbf2d044e84cdf437e7
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
OpenNSL is one of the dependencies for FBOSS.
OpenNSL is github repository with headers and prebuilt opaque library. Thus, we
can't/don't need to build it. Thus, OpenNSL manifest uses NopBuilder.
fbcode_builder expects the library to be in installed/lib and headers to be in
installed/include, but OpenNSL lib structure is different. Thus, add explicit
[install.fields] rule for bin/wedge => lib. Once a rule(s) is provided, we must
provide rule(s) for all, so add rule for include => include as well.
Reviewed By: wez
Differential Revision: D15593639
fbshipit-source-id: facb36ff7b9a31f5952ecbc1a9fdb834c0e68d11
Summary:
This is towards getting open source FBOSS to build using fbcode_builder.
iproute2 is one of the dependencies for FBOSS. This patch adds a manifest file
to build the specific version of iproute2 needed for FBOSS.
Additionally, the default git clone depth of 100 is insufficient for the
version of iproute2 FBOSS depends on. Thus, this patch extends the git SCHEMA
to add optional argument depth. The default remains 100.
The usual /configure --prefix does not work for iproute2. Thus, we need to add
a custom builder that:
- copies sources to build directory, builds, and
- installs to installed directory using DEST_DIR.
- it must also explicitly copy include from build dir to install dir
Reviewed By: wez
Differential Revision: D15588809
fbshipit-source-id: ac5eab24134e078d88b85b4be433c78b05ef8ce5
Summary: it seems since D15286181, we updated docker os_image to ubuntu18 and gcc7. this diff changes the README.md to reflect the change.
Reviewed By: yfeldblum
Differential Revision: D15513580
fbshipit-source-id: 0a3518c21c912903921946850bd08c948f09c295
Summary:
This should enable test pilot to skip broken/flakey tests.
The `--tag-new-tests` flag is only appropriate for code that has been
landed on master and is used by the FB infra to classify new tests
appropriately.
For continuous builds we use a test collection with different parameters
from the normal developer facing flow so that the infra can re-assess
their status and enable/disable high-signal/noisy tests.
Depends: D15495344
Reviewed By: Ben0mega
Differential Revision: D15500955
fbshipit-source-id: f3b7976cec6a5cf70f5d128b38bde11620b26918
Summary:
getdeps.py's find_existing_win32_subst_for_path function tries to reuse an existing build directory alias. (On Windows, the build directory is aliased to a drive letter to shorten paths.) If this function does not find and existing build directory alias, getdeps.py invalidates many of its caches.
On my Windows machine, find_existing_win32_subst_for_path always fails, so all of my builds are super slow. This happens because find_existing_win32_subst_for_path is given a path with a lower-case drive letter ("c:\users\..."), but the subst command returns paths with an upper-case drive letter ("C:\users\...").
When comparing paths, use ntpath.normpath. This makes the comparison case-insensitive for drive letters. (It also makes the comparison case-insensitive for other path components.)
On Linux and macOS, this diff should not change behavior.
Reviewed By: wez
Differential Revision: D15466096
fbshipit-source-id: 1669aa0a6eaeb6012154f3a9e24eba3f835262c6
Summary:
with the reshuffling around getdeps.py, and changes in folly/thrift/wangle,
our internal package stopped building. This diff brings it back to life. It is
a little bit interwined:
* Introduce a `--facebook-internal` switch to getdeps that causes `fb` in the
build context to be set to `on` rather than `off`. This allows specifying
options in the manifest that are FB specific.
* Introduce cmake options to control the WATCHMAN_STATE_DIR and whether cmake
should attempt to create WATCHMAN_STATE_DIR when it is installed.
* For FB specific builds on macOS, set the state dir to match our existing location
and disable creation of WATCHMAN_STATE_DIR because that will ultimately be
handling during package installation and we don't have privs for that on
the machine generating the package.
* Adjust Facebook CI scripts to use getdeps for building and testing watchman
Reviewed By: simpkins
Differential Revision: D15337970
fbshipit-source-id: 8ab3c1f810decf5f0b16337dc1cbe6dc73d26426
Summary:
The schedule type is used to differentiate between a build run during
code review (and thus runs code that isn't yet in the master branch),
from continuous or other types of run that operate on landed code.
This doesn't change any behavior yet; this diff just adds
the plumbing to pass down an optional arbitrary schedule type string
from the CI system.
In the future, we'll use the schedule type to influence the behavior
of running tests.
Reviewed By: strager
Differential Revision: D15300120
fbshipit-source-id: 3b46afef2ff171b3fa095763dd5006a54ea328b8
Summary:
if we find ccache in the path, then we instruct cmake to use
it as the compiler launcher.
Reviewed By: pkaush
Differential Revision: D15375441
fbshipit-source-id: 602fe54742a5ec07b0533bd2cf63cb905b051e85
Summary:
We were computing `SCRATCH/install` and hashing based on that value,
but the build stuff was later computing `SCRATCH/installed` and passing that
to the builders.
Fixup the mismatch.
Reviewed By: simpkins
Differential Revision: D15337969
fbshipit-source-id: 70288f2d9286aaacf4c1f6e0dac4680a55edac6d
Summary:
This diff updates docker os_image to ubuntu18 and gcc7 in travis settings.
The dependencies that we work with internally (e.g. gcc7, boost and so on) are not present in the ubuntu-16.04. Further, the support for gcc5 is going away. So maintaining this compatibility for these platforms is painful, especially for new projects, such as mvfst.
Reviewed By: simpkins
Differential Revision: D15286181
fbshipit-source-id: eb0da9556cdb17d0aa132ac2aa4c35f9c6eccd97
Summary: We need this for dependencies that require submodules
Reviewed By: saifhhasan, GirasoleY
Differential Revision: D15282792
fbshipit-source-id: b0cc8d645e73668252409934fd6741fb211e30ae
Summary:
On Windows, the writing operation would fail with:
```
Traceback (most recent call last):
File ".\opensource\fbcode_builder\getdeps.py", line 400, in <module>
sys.exit(main())
File ".\opensource\fbcode_builder\getdeps.py", line 393, in main
return args.func(args)
File ".\opensource\fbcode_builder\getdeps.py", line 236, in run
change_status = fetcher.update()
File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 451, in update
return mapping.mirror(self.build_options.fbsource_dir, self.repo_dir)
File "C:\open\fbsource\fbcode\opensource\fbcode_builder\getdeps\fetcher.py", line 400, in mirror
f.write(name + "\n")
File "C:\Python36\lib\encodings\cp1252.py", line 19, in encode
return codecs.charmap_encode(input,self.errors,encoding_table)[0]
UnicodeEncodeError: 'charmap' codec can't encode characters in position 104-105: character maps to <undefined>
```
and this is caused by a file in libgit2: https://github.com/libgit2/libgit2/blob/master/tests/resources/status/%E8%BF%99, which is intended to test handling non-ASCII path.
Python on Windows will write file in cp1252 encoding by default, which does not contain that Chines character. (Caveat: that file on my system doesn't have the correct file name as well, it is being encoded in IBM861 for some reason. However the characters in IBM861 does not exist in CP1252 either)
Reviewed By: wez
Differential Revision: D15281521
fbshipit-source-id: 8a75e32bc1042167c945d67e26b549fda83b6b41
Summary:
The goal is to return an error code > 127 in the case of a
transient, retryable, infrastructure error. This diff generates
those in the case of failure in downloading a URL or from interacting
with LFS.
Reviewed By: strager
Differential Revision: D15266838
fbshipit-source-id: 4f52a791320123968869032c37912dded464a86e
Summary:
the cmake `add_test` and related functions allow specifying
the WORKING_DIRECTORY to use for tests. We weren't respecting this
value, so this diff looks up the WORKING_DIRECTORY from the ctest
json info and adjusts the buck test info json blob that we pass
on the testpilot.
Since that interface only allows passing an argv array, we use
the `cmake -E chdir` command to run a command in a specified
directory in a portable manner.
Reviewed By: strager
Differential Revision: D15274012
fbshipit-source-id: 1f02d461d73178745794703d455494e31c2e09ed
Summary: mstch is no longer required for fbthrift, we can remove it from oss build spec
Reviewed By: vitaut
Differential Revision: D15280267
fbshipit-source-id: 5008e54af9c927f23b0d6acbf0d9beb6e004eee1
Summary: mstch is no longer required for fbthrift, we can remove it from manifests
Reviewed By: vitaut
Differential Revision: D15279652
fbshipit-source-id: 1772de7ab51fbfe048808f66290c4ca79de60608
Summary:
This controls whether tests are built or not.
They are not built by default. When `--enable-tests` is turned on,
tests are enabled for the named project only, not all of the deps.
This results in a faster build, because eg: the number of tests in
folly is very large and consumers of folly don't want to spend so
much time waiting to build tests when really all they want to do
is build their own project.
Reviewed By: strager
Differential Revision: D15246336
fbshipit-source-id: 2e22fd60c983b059f71639602712087f2a188ec6
Summary:
This is needed to correctly pick up changes made to
eg: cmake.defines sections in the manifest for first-party
projects.
Reviewed By: strager
Differential Revision: D15246337
fbshipit-source-id: 35e525e885f87d6136d5ff3b94ebf34516ab947c
Summary: support for fbcode_build and travis build
Reviewed By: siyengar
Differential Revision: D15184766
fbshipit-source-id: 5117164c324d3d3c867592a88670a53b6f862477
Summary:
This is useful especially on Windows where the up-to-date
checks for the dependencies take a long time.
The idea is that you might run this to start:
```
$ getdeps.py build eden
```
and then while in the edit/debug/build iteration cycle:
```
$ getdeps.py build --no-deps eden
```
Reviewed By: pkaush
Differential Revision: D15200352
fbshipit-source-id: 086f2f49db967ef4d1914a69fa80067104d79136
Summary:
Previously, we were using autoconf to build sqlite,
but that isn't available on Windows. Instead, here's a builder
that generates a little cmake configuration for building and
installing sqlite.
Using cmake for this means that we can test the same builder
on all platforms that need to pull in sqlite.
Reviewed By: simpkins
Differential Revision: D15179387
fbshipit-source-id: fccf969182565982bd5be55545a2d2625aa99124
Summary:
When running in FB infra, prefer to download from our local LFS
server rather than going out to the internet.
Fall back to a normal internet download if the LFS get fails for some reason.
Upload to LFS after successfully verifying the hash for the downloaded archive.
Add a subcommand that performs a fetch for all possible platforms so that it
is easier to ensure that the lfs-pointers file is up to date.
Reviewed By: simpkins
Differential Revision: D14978660
fbshipit-source-id: 240fc32fc7003d1e06c88b80d85054dae36e2f31
Summary:
In order to pull in the treemanifest and other libraries
from our mercurial repo, add a manifest file for it, and then
adjust the logic in our cmake module to look for it.
The fb-mercurial manifest just copies the source tree to the
installation dir. In the future, we could teach it to invoke
the build for real.
Reviewed By: simpkins
Differential Revision: D14969806
fbshipit-source-id: cb270c5003a1c134eeea92c7481a84938f1c5957
Summary:
This prints out the installation or source prefix for a given project.
This is useful in eg: packaging scripts to figure out where they can
find the built artifacts.
Reviewed By: simpkins
Differential Revision: D14967378
fbshipit-source-id: 7e1b5de2ca7219af24cfb07b4b42de22aa410469
Summary:
If `testpilot` is available, generate a buck compatible json file describing the available test binaries and feed that to testpilot to have it run the tests.
A later (yet to be written) diff will be able to pass appropriate flags down to testpilot in continuous runs and that will allow testpilot to auto-disable and file tasks for tests in the opensource builds.
Reviewed By: simpkins
Differential Revision: D14766856
fbshipit-source-id: 4e144ff18f6788cf5e830d29788eabd2dbbae46a
Summary:
The loader makes it possible to monkey patch the functions
that are responsible for loading manifests. It is intended to be use
in tests that are run in bucks sandboxed environment and that don't
have direct access to the manifest files on disk.
Reviewed By: simpkins
Differential Revision: D14781326
fbshipit-source-id: 18f69f8ce5768dc605b1a9388a80b7b7b9ffe0f4
Summary:
previously, a relatively lame hash was computed to use
for the build directory based on some hash of the source directory.
That was good enough to get things off the ground, but in the
interest of being able to cache the build outputs and safely
invalidate them we need a slightly more rigorous implementation.
This diff computes a hash based on the manifest contents and
relevant environmental factors.
The hash is used to name the build directory which will ultimately
be cached eg: using the travis/appveyor cache directory configuration
and some other means for the FB internal CI.
The hash needs to be sufficient that we change the hash when
the manifests change. We can tolerate a false positive change
in hash (it just means that a build will take longer), but
cannot tolerate a false negative (which would result in an
incorrect build).
Reviewed By: simpkins
Differential Revision: D14710332
fbshipit-source-id: ebc2e74eafc6f3305d4412a82195bc9fb9dfa615
Summary:
This command schedules a facebook specific sandcastle job for the current
commit in your repo for each of the platforms we have support for in
sandcastle.
You can use `--dry-run` to have it print out the job specs.
To support this, I've moved around some of the support utilities
to make it easier to import them.
Reviewed By: simpkins
Differential Revision: D14710330
fbshipit-source-id: fb1e2a2ce78e52894291159514977da97028b37f
Summary:
Adds a `test` subcommand that runs the tests for project.
We're mostly interested in the 1st party facebook projects for this.
The `sandcastle` flow will run the `test` subcommand just for the "leaf" project--the one named on the command line.
Reviewed By: simpkins
Differential Revision: D14710331
fbshipit-source-id: 7d04a46cfd723894d61018de2f230140b52285ac
Summary:
This fixes a TODO; in our CI environment we want to use the
real shipit, so we'll use this flag to make that happen.
Reviewed By: simpkins
Differential Revision: D14695576
fbshipit-source-id: 64ee72c210e2472d295dcbd39c86549273b68452
Summary:
This makes it possible to disable eden (and thus thrift) support.
I've defaulted this to off so that we don't break the existing
watchman CI.
Reviewed By: simpkins, strager
Differential Revision: D14726767
fbshipit-source-id: 0f4d0597d901a91850f1ba6d71609c059c064c22
Summary:
adds the command that is used to drive a build.
The dependencies of the named project are computed using
the same mechanism behind the `list-deps` subcommand.
If a manifest specifies that it uses the `cmake` builder then
a dependency on `cmake` is synthesized so that we can download
a recent version of cmake ahead of time.
The build command uses the fetcher to update the src and then
executes the builder.
Reviewed By: simpkins
Differential Revision: D14691011
fbshipit-source-id: 31ae59614651ef021a9505e89c13b5717b440071
Summary:
this could do with a better name; the NopBuilder doesn't actually
build anything, but instead copies some files to the destination location.
This is used together with eg: cmake to install pre-built binaries downloaded
from a tarball.
Reviewed By: simpkins
Differential Revision: D14691015
fbshipit-source-id: a938e977aa4ec5a664bdb8085ff708319a204594
Summary:
the boost builder knows how to perform the non-standard
configure and build for boost.
Ideally we'd just build this statically and be happy but there are
some nuances I've observed while building on different platforms:
* One of our projects (thrift or wangle) explicitly requests static
boost linkage for reasons unspecified
* on darwin the install_name is broken when building dynamic libs
For the sake of expediency in getting getdeps up and running, the
solution for the moment is to build static on posix systems and
build both static and shared on windows systems.
Reviewed By: simpkins
Differential Revision: D14691009
fbshipit-source-id: 634770a6f53c3ada42d1877cc6c3dacc6eed7d18
Summary:
the openssl builder knows how to perform the non-standard
configuration and build steps to build openssl.
On Linux systems the manifests for our projects don't mention
openssl, causing them to pick up the system openssl.
On Mac, apple don't ship openssl headers so we need to build our own.
On Windows there is no standard openssl installation so we also need
to build our own.
As a result, this builder only works on windows and mac.
Reviewed By: simpkins
Differential Revision: D14691010
fbshipit-source-id: 9f8979f9eaeb5209c290cf4f43c97c0cb43d13a2
Summary:
this builder is used to bootstrap the ninja build tool.
On Windows and mac the manifest for ninja is set to download a pre-built executable.
While pre-built executables are available for linux they aren't portable enough
for our purposes so we need to be able to build it for ourselves.
Reviewed By: simpkins
Differential Revision: D14690992
fbshipit-source-id: b60fd02ad04f58dc7c2931280341791270609737
Summary:
the cmake builder knows how to use cmake to configure a build
for (preferably) and out-of-src build. The `cmake.defines` section of
the manifest is used to pass `-Dkey=value` options to the cmake configure
command line.
We prefer to use `ninja` to execute the build so that we can use more
cores than 1 on Windows and just for consistency across platforms
with mac and linux.
Reviewed By: simpkins
Differential Revision: D14690998
fbshipit-source-id: 8102e8b4a47da515ca001772788ed0e5f2645ad7
Summary:
the autoconf builder performs an out-of-source build using
the autoconf suite to configure the build rules.
The `autoconf.args` section from the manifest is passed to the `./configure`
command line.
If an `autogen.sh` script is present then it will be used to regenerate
a missing `configure` script, otherwise we'll try `autoreconf`.
Reviewed By: simpkins
Differential Revision: D14691002
fbshipit-source-id: ab8cceafb833dab513d5a50c65f4c895a4f40047
Summary:
the make builder runs `make` in the source directory.
The `make.args` section from the manifest is passed as arguments
to the `make` invocation.
Reviewed By: simpkins
Differential Revision: D14690996
fbshipit-source-id: 180d657ad05f0c0266a8c1d30979d8d1473958c9
Summary:
a builder knows how to build and install a project.
Later diffs add concrete implementations of the BuilderBase
Reviewed By: simpkins
Differential Revision: D14691018
fbshipit-source-id: 89b14614b5160353cd7e59f27037afcdf6229eb7
Summary:
While the command isn't necessarily super useful
on its own, it does show that the plumbing for walking the
deps is functioning, and that is important when it comes
to building.
The output lists the projects in the order that they
would be built.
The `fetch` command has been augmented to add a `--recursive`
flag that uses the same mechanism to recursively fetch
the dependencies.
Reviewed By: simpkins
Differential Revision: D14691004
fbshipit-source-id: b00bf6ad4742f8bb0a70698f71a5fe03d6a1f453
Summary:
Adds a command that can be used to trigger a fetch for a project
```
$ ./opensource/fbcode_builder/getdeps.py fetch zstd
Cloning https://github.com/facebook/zstd.git...
---
+ git clone --depth=100 https://github.com/facebook/zstd.git /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git
Cloning into '/data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git'...
remote: Enumerating objects: 3816, done.
remote: Counting objects: 100% (3816/3816), done.
remote: Compressing objects: 100% (1415/1415), done.
remote: Total 3816 (delta 2556), reused 3312 (delta 2288), pack-reused 0
Receiving objects: 100% (3816/3816), 2.93 MiB | 9.59 MiB/s, done.
Resolving deltas: 100% (2556/2556), done.
Updating /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git -> v1.3.8
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git fetch origin v1.3.8
From https://github.com/facebook/zstd
* tag v1.3.8 -> FETCH_HEAD
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git checkout FETCH_HEAD
Note: checking out 'FETCH_HEAD'.
You are in 'detached HEAD' state. You can look around, make experimental
changes and commit them, and you can discard any commits you make in this
state without impacting any branches by performing another checkout.
If you want to create a new branch to retain commits you create, you may
do so (now or later) by using -b with the checkout command again. Example:
git checkout -b <new-branch-name>
HEAD is now at 470344d Merge pull request #1479 from facebook/visualTest
---
+ git -C /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git submodule update --init
```
Reviewed By: simpkins
Differential Revision: D14691008
fbshipit-source-id: 3afa391360518a08ebd6ff97f5b8b4993f10c4e8
Summary:
This fetcher knows how to transform a 1st party project
from fbsource into approximately the same shape as ShipIt produces
for the github repo mirrors. It does this by reading shipit
mapping information from the manifest file for the project.
Since this fetcher uses data in the manifest and is implemented
directly in the getdeps codebase, it is suitable for iterating
on the opensource builds directly out of fbsource on both devservers
and laptops inside FB.
Reviewed By: simpkins
Differential Revision: D14691012
fbshipit-source-id: 05f68a7be64a2e465937b24b8825d25d3348ed13
Summary:
This fetcher knows how to take a 1st party FB project
from fbsource and transform it to the same shape as our github
repos using ShipIt. The transformation creates a transformed
mirror of the code in scratch space (rather than mutating fbsource
directly).
This can only be used in environments where shipit is available.
A later diff implements an alternative that works in more environments.
Reviewed By: simpkins
Differential Revision: D14691013
fbshipit-source-id: 539e307755c9fc0a098a235868ab622652061494
Summary:
this fetcher knows how to download a URL that references
an archive and extract the sources from it. Compressed tarballs
and zipfiles are supported.
Reviewed By: simpkins
Differential Revision: D14690994
fbshipit-source-id: ffbbd1d9479b0d5aa3e5cd13fe11a560e9d01c2a
Summary: this fetcher knows how to check out sources via git.
Reviewed By: simpkins
Differential Revision: D14691000
fbshipit-source-id: 60f1ffbfed7b32a019aef6aa70ae0903f2782451
Summary:
Fetchers are used to fetch a source directory from a
source defined in a manifest file.
More details can be found in comments on the various methods.
The Manifest class offers a create_fetcher method for constructing
an appropriate fetcher. This is just a stub in this commit, but
will expand over the course of the next few diffs as concrete
fetcher instances are added.
Reviewed By: simpkins
Differential Revision: D14691001
fbshipit-source-id: 8c9038eacae3345e9403d5d1304bf979a9ee1555
Summary:
The build options class contains some environmental and
build related information that will be passed down to fetcher
and builder objects that will be introduced in later diffs.
Reviewed By: simpkins
Differential Revision: D14691007
fbshipit-source-id: e8fe7322f590667ac28a5a3925a072056df0b3e3
Summary:
This will feed into the manifest context for system
dependent manifest sections. This is essentially the same
code borrowed from the watchman and eden getdeps.py.
Reviewed By: simpkins
Differential Revision: D14690997
fbshipit-source-id: 3d3ae146237a2cd49609aaa2bd0e785ebe21f02c
Summary:
this module adds some functions that help with copying
directory trees. The copytree function is aware of eden and will
issue a prefetch prior to walking the directory.
Reviewed By: simpkins
Differential Revision: D14691006
fbshipit-source-id: 079bf850756f61aca17978453d07bc73b2f91788
Summary:
This runs a command, raising an exception if it exits with a non-zero error status.
It prints out the arguments in a mostly copy-and-pasteable form, with PATH-like
env vars pretty printed to make it easier to see what is being invoked; here's
an example of how cmake is being invoked later in this stack:
```
---
+ CMAKE_PREFIX_PATH=\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/ninja-5d7ec7:\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/cmake-91dc9a:\
+ PKG_CONFIG_PATH=\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/ninja-5d7ec7/lib/pkgconfig:\
+ /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/cmake-91dc9a/lib/pkgconfig:\
+ cd /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/build/zstd-470344 && \
+ cmake configure /data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/repos/github.com-facebook-zstd.git/build/cmake -DCMAKE_INST
ALL_PREFIX=/data/users/wez/scratch/dataZusersZwezZfbsource/fbcode_builder_getdeps/installed/zstd-470344 -DBUILD_SHARED_LIBS=OFF -DCMAKE_BUILD_TYPE=R
elWithDebInfo -G Ninja
```
Reviewed By: simpkins
Differential Revision: D14690999
fbshipit-source-id: cdb0c681c7dfdfdc6e8c96bf4830bfbcf666411b
Summary:
This is the new getdeps entrypoint. It lives in `opensource/fbcode_builder` so that
it is synced into the various opensource projects at FB.
In the incarnation in this diff is has a single subcommand that can be used to validate
a manifest file.
Reviewed By: sinancepel
Differential Revision: D14690995
fbshipit-source-id: 53f8ec9bafb7b1930970079e3ce20f496ac1c188
Summary:
Add some helpers for manipulating environment variables
that will hold either compiler flags or paths.
These are used in later diffs when setting up arguments/environment
for eg: cmake.
Reviewed By: sinancepel, simpkins
Differential Revision: D14690993
fbshipit-source-id: 7f9753cd99d968550fe9e3ba8b7017a44683061e
Summary:
These are ported over from the logic in the watchman and eden getdeps
scripts, with additions to help bootstrap a build environment.
These are sufficient to build watchman with thrift support on windows, mac and
linux, and eden on mac and linux when combined with the getdeps code that
follows in later diffs in this stack.
Reviewed By: simpkins
Differential Revision: D14691005
fbshipit-source-id: 7f8b02fedcdc020e2d0e758c466959d8161d4587
Summary:
Adds a parser for manifest files that describe projects
that may be either 1st party or 3rd party.
A selection of manifest files appears in a later diff in this stack.
This diff provides the raw parser and a couple of low level helpers.
It also defines a mechanism for validating the schema to help catch
structural (rather than semantic) issues with manifest file contents.
Later diffs in the stack add helpers for accessing the data in
a higher level way.
Reviewed By: sinancepel
Differential Revision: D14691003
fbshipit-source-id: 7d2930a3359ede0f6e21fdc45686b083ab7a9ffa
Summary:
As part of folding getdeps into fbcode_builder, this
expression parser is needed to allow constrained and deterministic
conditionals in the manifest file format.
For example, the watchman manifest will use this cargo-inspired syntax
for system dependent sections:
```
[dependencies]
folly
[dependencies.not(os=windows)]
thrift
```
Reviewed By: sinancepel
Differential Revision: D14691014
fbshipit-source-id: 080bcdb20579da40d225799f5f22debe65708b03
Summary:
Initial commit to include the fbcode_builder sources in the
eden github repository.
fbshipit-source-id: 49098cecda04a7e9dd9dcc0b569fffc96f0f719b
