Summary:
Neither the default "urlretrieve" nor "pycurl" works fine for all of FBOSS'
dependencies. One or other fails (opennsa, libmnl, exprtk, etc...) with these
two options. So, adding support for "wget".
Tried to use python's wget.download() but this also internally uses
"urlretrieve" and hence it fails for some of the FBOSS' dependencies.
So, using wget command directly instead.
Reviewed By: shri-khare
Differential Revision: D40841867
fbshipit-source-id: 229b3064f3faad4b32cf8a3c43fef53f2bb001fd
Summary:
X-link: https://github.com/facebookincubator/katran/pull/180
The urlretrieve method originally doesn't add this header, and in katran we noticed that the libmnl wasn't fetched anymore. After investigation, the request is blocked with error 403 because this header is missing. Adding it to the request.
Reviewed By: avasylev
Differential Revision: D40386108
fbshipit-source-id: a9b4b937ca12a04bc701d3f945681914151c0110
Summary:
Relative paths can not be calculated between drives, so we should not realpath
here because we don't realpath elsewhere.
Reviewed By: MichaelCuevas
Differential Revision: D38378132
fbshipit-source-id: a20d5c5273a2d09166792fbb9862f67d8040fdd3
Summary:
A request to record_results failed, which caused a folly gcc 8 build
regression to slip in, which broke the Watchman and EdenFS builds.
Instead, turn record_results requests into a logged warning.
Reviewed By: xavierd
Differential Revision: D38170729
fbshipit-source-id: d59ae5fa48b5cbcf649281108d958ff95e319e35
Summary:
Occasionally we see reports [1] that /proc/meminfo is missing or has
unexpected values on Linux. Instead of failing the build, guess a
reasonable value, at the risk that this will unnecessarily limit the
build's concurrency.
[1] https://github.com/facebook/watchman/issues/1040
Reviewed By: genevievehelsel
Differential Revision: D38126612
fbshipit-source-id: 9d9d9f6003703acf6dffcdd5b2022f0f7b3aa691
Summary:
TPX is now available on Windows, the testpilot code is thus no longer
necessary.
Reviewed By: fanzeyi
Differential Revision: D37476157
fbshipit-source-id: 885881de2e06eab4990c6c96dcef8f51614a17f5
Summary:
TPX on Windows is being worked on, but not yet ready for prime time. Let's make
sure that getdeps isn't trying to use TPX on Windows for now.
Reviewed By: splhack
Differential Revision: D37474152
fbshipit-source-id: e636b0b68f6f73472f28295c280803ac44fafebf
Summary:
Fix local test runs. Previously, they were silently passing, because
they skipped tests that were unknown to tpx. Instead, treat local runs
separately from CI diff runs.
Reviewed By: xavierd
Differential Revision: D37287032
fbshipit-source-id: a3a88e117cd0e078e18c0f795bae467768d812a5
Summary:
Before the change:
```
% uname -m -s
Darwin arm64
% ./autogen.sh
...
ld: warning: ignoring file /private/var/.../installed/openssl-OovnrH2WrHG18y7xi0irAegZLOlNsDyeOGb8BBEcoGw/lib/libssl.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
ld: warning: ignoring file /private/var/.../installed/openssl-OovnrH2WrHG18y7xi0irAegZLOlNsDyeOGb8BBEcoGw/lib/libcrypto.dylib, building for macOS-arm64 but attempting to link with file built for macOS-x86_64
...
```
After the change build succeeds.
X-link: https://github.com/facebook/watchman/pull/1032
Reviewed By: fanzeyi
Differential Revision: D37137060
Pulled By: genevievehelsel
fbshipit-source-id: a8381c39821bbd29ddea1d1009b94f5ed6cacb21
Summary: Adding patching support for all platforms using `git apply`
Reviewed By: xavierd
Differential Revision: D36939728
fbshipit-source-id: ccf6e4ccd30257c6928ba804a76882d4d2f71700
Summary:
This function is expected to return a tuple of 3 elements. This doesn't happen when it fails opening the `/etc/os-release` file.
X-link: https://github.com/facebook/watchman/pull/1025
Reviewed By: fanzeyi
Differential Revision: D36736667
Pulled By: chadaustin
fbshipit-source-id: a6839235d6c5ebb50267cd78f1c9590b2fb180d7
Summary:
Without ctest installed, `getdeps.py test $project` would produce a
bad error message about failing to run a `''` command. Instead,
explicitly error early when a command is required.
Reviewed By: genevievehelsel
Differential Revision: D36535929
fbshipit-source-id: d300c6b1b0c124b56dffffae0a71bee9b7f12fe7
Summary:
Watchman's vagrant images rely on mapping a watchman source directory
fetched from fbsource into the VM. Thus, they sometimes look like
fbsource, and that's largely okay. However, the vendored cargo support
breaks. Instead, explicitly skip vendored cargo when there is no
fbsource dir.
Reviewed By: xavierd
Differential Revision: D36386674
fbshipit-source-id: 61f2af19507fecd2342cfc94bbb7120ab91c33b4
Summary:
Applies new import merging and sorting from µsort v1.0.
When merging imports, µsort will make a best-effort to move associated
comments to match merged elements, but there are known limitations due to
the diynamic nature of Python and developer tooling. These changes should
not produce any dangerous runtime changes, but may require touch-ups to
satisfy linters and other tooling.
Note that µsort uses case-insensitive, lexicographical sorting, which
results in a different ordering compared to isort. This provides a more
consistent sorting order, matching the case-insensitive order used when
sorting import statements by module name, and ensures that "frog", "FROG",
and "Frog" always sort next to each other.
For details on µsort's sorting and merging semantics, see the user guide:
https://usort.readthedocs.io/en/stable/guide.html#sorting
Reviewed By: lisroach
Differential Revision: D36402214
fbshipit-source-id: b641bfa9d46242188524d4ae2c44998922a62b4c
Summary:
Applies the black-fbsource codemod with the new build of pyfmt.
paintitblack
Reviewed By: lisroach
Differential Revision: D36324783
fbshipit-source-id: 280c09e88257e5e569ab729691165d8dedd767bc
Summary:
This adds the ability to specify a patch file to be applied on top of any library in getdeps.
For example, zlib doesn't support static linking with CMake, but there's a small patch that can be applied to support static linking. This is the specific use case this diff is intended to support.
Reviewed By: bigfootjon
Differential Revision: D35410512
fbshipit-source-id: d1af0ddf9ec45ef28aa902c06735af86817ac194
Summary: Since Bistro is slated to be deprecated, let us turn off OSS CI for it.
Reviewed By: AnonymousMonkey2021
Differential Revision: D35347845
fbshipit-source-id: be23bcfaa098f77162f27ecfed6469b8db7d3401
Summary:
On GitHub Actions macos-10.15 image, some of the build dependencies
for Watchman reference
/usr/local/opt/openssl@1.1/lib/libcrypto.1.1.dylib and others
reference
/usr/local/Cellar/openssl@1.1/1.1.1m/lib/libcrypto.1.1.dylib.
To avoid overwriting libcrypto in the artifact directory, key
processed_deps on the destination path.
Reviewed By: xavierd
Differential Revision: D34882663
fbshipit-source-id: 60d021bf9956d9b3e34e8945ccec234d31031369
Summary:
If getdeps tests run from fbcode, tpx guesses that the test type is FBCODE rather than FBTEST. This makes UTF ignore the buildsystem=getdeps config, which leads tests to be ignored.
This diff sets the test type explicitly.
More context in [this post](https://fb.workplace.com/groups/tpx.users/permalink/1309289352870499/).
Reviewed By: sandeepkumarpani888
Differential Revision: D34271402
fbshipit-source-id: 2c6c37d706f22d214cd2f7622a7124b4c3f92ba0
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/110
Pull Request resolved: https://github.com/facebookexperimental/rust-shed/pull/27
Make it so that changes to rust-shed or other common rust source are used locally vendored, so they don't need to be pushed to github before they are visible in a build.
There was already some support for cargo vendoring in getdeps, but it was limited to dependencies between manifests built with cargo builder. This wasn't enough to build something like eden (cmake is main entry point, with later calls cargo) or eden_scm (make is main entry point, with later calls to cargo), so this diff adds a cargo prepare step for getdeps other primary build systems.
The cargo vendoring is done by using a cargo config file to point to the source files used by getdeps. It has two modes:
1. per crate, existing mode which is already automatic for cargo to cargo manifest dependencies. To use it for a non cargo build manifest, add crate.pathmap
2. per git url, existing mode which was only use for crates.io third-party crates, now can be enabled by setting cargo.cargo_config_file
Reviewed By: yancouto
Differential Revision: D33895469
fbshipit-source-id: 7b13c0b679532492a336ce217de875c25fe1be90
Summary:
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/106
Pull Request resolved: https://github.com/facebookexperimental/eden/pull/107
Summary
* Add EdenFS builds on external CI now EdenSCM is good
* Mac builds on github actions by using brew for system dependencies
To make this work had to fix some path ordering issues with install directories for Linux and Mac, and generalise the homebrew path fixups we were doing for bison to all the used homebrew packages.
Previously Installed packages were being added after system paths, so our own installed thing might be ignored. On github these meant system python 3.9 was being used for hg tests rather than our specified 3.8 (this showed we have some test fails on python 3.9 with "SystemError: deallocated bytearray object has exported buffers", that are beyond the scope of this diff to fix)
Also needed to include the getdeps generated python into the generated edenscmdeps3.zip archive setup.py produces otherwise EdenFS tests failed to import thrift.Thrift
Eden tests are hanging when run externally about half way through, so disable them on github actions for now as this PR is already fairly large. They work when run locally on an internal devserver, so probably some bit of environment necessary is not defined in the test runner
Reviewed By: chadaustin
Differential Revision: D34116505
fbshipit-source-id: d0d628db5daabc28d0bd8997cd5c1bc885ed1e73
Summary:
Pull Request resolved: https://github.com/facebook/watchman/pull/998
A detailed description of this change is included in the new comment.
Reviewed By: fanzeyi
Differential Revision: D34124030
fbshipit-source-id: f37c458bae135357e16b256e09f0d6eb8f1f996d
Summary:
Watchman advertises support for FreeBSD but it's regressed a couple
ways. Ensure it builds on FreeBSD again with a Vagrantfile.
Reviewed By: ahornby
Differential Revision: D33989289
fbshipit-source-id: ff906bc219974bafbc349740ce6bddfe5bd2908f
Summary: Add some basic support for FreeBSD to getdeps.
Reviewed By: ahornby
Differential Revision: D33989129
fbshipit-source-id: 42ff5f160b7e19c12196bb2e52a726f7815487bd
Summary:
The new boost has an MSVC detection algorithm that 1) I don't
understand and 2) does not even seem deterministic. Since our internal
builds are pinned to vc142, force that in the manifest.
Reviewed By: fanzeyi
Differential Revision: D34085081
fbshipit-source-id: d8ce4d8e28aad14f6fa60f9227dafb546ce321e5
Summary: Make the --facebook-internal flag value available from BuildOptions so can have jobs of both values where required
Reviewed By: xavierd
Differential Revision: D33945588
fbshipit-source-id: 5d0e1b42973bd3e411a4a71075915bdb2755f0eb
Summary:
In order to speed up build times on a mac, add homebrew support to
getdeps.
Homebrew packages can be declared in a manifest using the `homebrew` header.
Futher, ahornby has added manifest entries for homebrew packages which are
included in this diff and also included a change to use the correct version of
openssl. Without this openssl change, homebrew cmake configure finds an old
openssl 1.0.2 install.
This diff provides a 2x speed up for building folly:
Timings for clean getdeps folly build on mid-2018 2.9Ghz i9 6 core intel macbook pro with 32GB RAM:
With new homebrew system deps:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build --allow-system-packages folly
real 17m39.329s
user 76m10.317s
sys 5m50.163s
```
Without:
```
rm -rf /Users/ahornby/.scratch/UsersZahornbyZfbsource/fbcode_builder_getdeps/
time ./opensource/fbcode_builder/getdeps.py build folly
real 32m10.344s
user 105m53.448s
sys 15m57.858s
```
Reviewed By: ahornby
Differential Revision: D33842632
fbshipit-source-id: ac785d4a8dcfa31b77292bddd9e747022ac36e3b
Summary: Add expressions for shipit entries so that we can map different directories if needed
Differential Revision: D33947996
fbshipit-source-id: e765769ae5a6ab90055b63284e538b965d6071b3
Summary: Create a new fbsource property for manifests so we can still use monorepo rust toolchain when running builds with --no-facebook-internal
Differential Revision: D33945559
fbshipit-source-id: 606af62e03d09c3529018af4478bcafecd0cd640
Summary: Boost 1.69 does not build on Fedora 35, so upgrade to Boost 1.78.
Reviewed By: ahornby
Differential Revision: D33855670
fbshipit-source-id: 85469a835a1dab1a7d5222413b1f1349bdcff280
Summary:
If a getdeps project creates a bind mount in a subdirectory, we don't
want the shipit transformer to copy all of the bind mount's contents
to the ship-transformed output.
This manifested when I was using Vagrant inside of Watchman and
`getdeps.py fetch watchman` would copy all of the Vagrant VMs too.
Reviewed By: ahornby
Differential Revision: D33855553
fbshipit-source-id: e2bc50ecfe0d067490c95be05d074d6a844f30d5
Summary:
There's no reason to block a getdeps build on prefetching its sources,
so issue all prefetches in the background.
Reviewed By: genevievehelsel
Differential Revision: D33855396
fbshipit-source-id: 1ba01b0587e9bc0e74e6bba5b8571af76bf2516d
Summary:
A long time ago, getdeps scheduled each build up to the number of hardware threads. For some builds, that was too heavy, so it got throttled back to only ever use half the hardware threads. This left parallelism on the table for CPU-bound projects that don't use much RAM per compilation.
This commit makes better use of the hardware with finer-grained logic that allows each manifest to specify a `job_weight_mib` estimate in MiB, and limit concurrency to `available_ram / job_weight`.
Reviewed By: ahornby
Differential Revision: D33754018
fbshipit-source-id: 785bed6c6cfe3c473244e0806a77cec1fc119e1f
Summary:
Move cargo logic to separate file as builder.py was getting a bit large
It's just a move of source file for CargoBuilder. Diff looks a bit big as I did hg cp to preserve history
Differential Revision: D33888925
fbshipit-source-id: 2d57343535ab087e09876edba4d00a6f0234fcd0
Summary:
Populate the new environment variable if relevant path for it found.
For the OSS getdeps builds of things depending on fb303 this will allow them to be able to be able to find the fb303 thrift files from the declared fb303 dependency, rather than needing duplicates/stubs inside each package.
Also adds python as dependency to fbthrift manifest, found it needed it when testing
Reviewed By: HarveyHunt
Differential Revision: D33771979
fbshipit-source-id: 9635221c0e6a2ac225d7d0babe76f1c919b583de
Summary:
Add support for overriding os, distro and distro version to command line when inspecting system packages so one can requested see ubuntu 18.04 package from other OS. Makes testing easier
Used shlex to shell unquote the value to be tested in the getdeps expression evaluator. getdeps expression parser didn't tolerate 18.04 as . is special char to getdeps expressions, needed to be "18.04"
Reviewed By: quark-zju
Differential Revision: D33741323
fbshipit-source-id: d83397c7fb5180a4d985d0d8ae7b3ff33b72f828
Summary:
Now that we have some type annotations, we might as well support
running the type checker.
Reviewed By: xavierd
Differential Revision: D33715757
fbshipit-source-id: baf693e4b2415e0e1aa50b569b744ca0cfb91337
