Commit Graph

29 Commits

Author SHA1 Message Date
Luke Granger-Brown
d7d4a92695 buildBazelPackage: add "fetchConfigured" for fetching only things required for the build
"bazel fetch" will, by default, fetch everything that _might_ be used,
including things that will later be discarded due to the way the build
is configured.

Concretely, this means that for some builds of Java packages, this will
avoid failures where the builder tries to retrieve the JDK from /usr/share/java
(or equivalent).

This also means that for most packages we can fetch _fewer_ dependencies,
since the standard tree pruning for artifacts to fetch will take effect.

fetchConfigured is disabled by default since it changes the fetch hashes
of tensorflow/tensorflow2 (since it ends up fetching less).
2020-07-17 18:29:58 +02:00
Matthew Bauer
1e19cc37bd build-bazel-package: pass proxy env vars 2020-05-09 13:35:47 -05:00
Matthew Bauer
2a8d05627b build-bazel-package: switch hash mode to “flat”
flat hashes can be substituted through hashed-mirrors, while recursive
hashes can’t. This is especially important for Bazel since the bazel
fetch dependencies can come from multiple different methods (git,
http, ftp, etc.). To do this, we create tar archives from the
output/external directory, which is then extracted to build. All of
the Bazel hashes are all updated.
2020-05-09 13:33:26 -05:00
Matthew Bauer
184cd9f6ff build-bazel-package: fix linkopt flags
These values were incorrect. We need to use NIX_LDFLAGS, not
NIX_LD_FLAGS. Also need to prefix all flags with -Wl, for GCC to
accept it.
2020-04-10 15:41:39 -04:00
Andreas Rammhold
fa6c5c3dc7
buildBazelPackage: prime fixed output derivations
All bazel fixed output derivations should be specific to the bazel
version that was used to generate them. There is not guarantee that the
build will still succeed or reproduces (without the cached fixed output)
if the fetch phase wasn't rerun with a different bazel version.

In the past bazel had been bumped but not all those packages that have
fixed outputs from bazel builds. This lead to compiling and somewhat
working TF versions that couldn't be reproduced without the cached fixed
outputs.
2020-03-03 01:12:26 +01:00
Michael Lingelbach
34296980d1 tensorflow: 1.15.0 -> 1.15.1
* Apply glibc 2.3 patch
* build tensorflow with bazel_1
* Bump openssl version to 1.1
2020-03-02 10:07:37 +01:00
Uri Baghin
10d1a04b02 bazel-deps: remove 2020-02-04 12:35:47 +11:00
Uri Baghin
3dc9c1b885 bazel-remote: 2019-01-12 -> 2020-01-29 2020-02-04 02:26:15 +01:00
Florian Klink
a905fcb40d buildBazelPackage: set $USER environment variable (#74538)
tensorflow assumes $USER to be set to something, otherwise it complains
like this:

```
FATAL: $USER is not set, and unable to look up name of current user: (error: 0): Success
Traceback (most recent call last):
  File "./configure.py", line 1602, in <module>
    main()
  File "./configure.py", line 1399, in main
    _TF_MAX_BAZEL_VERSION)
  File "./configure.py", line 478, in check_bazel_version
    ['bazel', '--batch', '--bazelrc=/dev/null', 'version'])
  File "./configure.py", line 156, in run_shell
    output = subprocess.check_output(cmd)
  File "/nix/store/drr8qcgiccfc5by09r5zc30flgwh1mbx-python3-3.7.5/lib/python3.7/subprocess.py", line 411, in check_output
    **kwargs).stdout
  File "/nix/store/drr8qcgiccfc5by09r5zc30flgwh1mbx-python3-3.7.5/lib/python3.7/subprocess.py", line 512, in run
    output=stdout, stderr=stderr)
```

Spotted while changing the hash of its fixed-output derivation on
purpose.

We could also set this in the tensorflow-specific part, but very likely,
other programs will fail as well.
2019-11-28 16:14:45 -08:00
Andrew Dunham
5e24b9e4ed buildBazelPackage: also set the SSL_CERT_FILE environment variable 2019-11-28 13:39:20 -08:00
Andrew Dunham
3f86f21207 buildBazelPackage: allow specifying whether to remove rules_cc 2019-11-27 10:43:34 +01:00
Nikolay Amiantov
280f17c893 buildBazelPackage: remove rules_cc
It's a new builtin dependency from Bazel 0.29.
2019-10-02 00:46:55 +03:00
Nikolay Amiantov
3b3618ea3c buildBazelPackage: remove top-level symlinks
They always can be regenerated during the actual build, and they are sometimes
random, e.g in Tensorflow;

platforms -> NIX_BUILD_TOP/tmp/install/35282f5123611afa742331368e9ae529/_embedded_binaries/platforms
2019-08-19 11:23:17 +03:00
Timo Kaufmann
e458a34012 buildBazelPackage: add flags for build and fetch
They sometimes take separate flags.
2019-07-31 13:28:44 +03:00
Nikolay Amiantov
a956694aba buildBazelPackage: clear markers
Turns out markers are non-deterministic after all and even our patching still
doesn't solve this problem completely. For example (tensorflow deps, this is a
complete diff so actual dependencies don't differ):

30509c30509
< bc527ff00916b15caee38793bca8f294c748df4a256de55c5199281be0489e73  result/@bazel_skylib.marker
---
> 4e0303e815c78df1e43d4b88dfe65e73046e0c6157fb10aa9a4e8b910113cd9c  result/@bazel_skylib.marker
31045c31045
< fa13d04b2316214c3b4008b52546c2d5b633e006f6f019d597bb3f9745bacf7b  result/@bazel_toolchains.marker
---
> b36174bf5535e5157801b6de30c35ee03a03fe57766306393c3d65dd65cbebf4  result/@bazel_toolchains.marker
31144c31144
< b0ce4a3ac29ac22528336dd3a54b5b7af9ecc43bef2a2630713c1981a5cbbb51  result/@build_bazel_rules_swift.marker
---
> 7492528068ec4f8e7ace2ecf8f933ec4e1b2235bd7426ce6f70177919f1cd05e  result/@build_bazel_rules_swift.marker
36245c36245
< be2993536a8233d63251b664caf35b1e7cd57d194ab2a39a293876c232d6bbd0  result/@io_bazel_rules_closure.marker
---
> b6655cc3f2c78525e5a724d8a4e93b1e7f09f1e09fc817d231109e7f39103e88  result/@io_bazel_rules_closure.marker
36329c36329
< 087bc674c9509dfe157400d111db4a13eeb45fc76aeccd490cee9aad6771ecad  result/@io_bazel_rules_docker.marker
---
> f920ec07315ec71e800b05cd22b2a341c0a80807c6e335ee81739b13c532b422  result/@io_bazel_rules_docker.marker
79544d79543
< 85893a05a817036c61f6cd9f8247757baa1654f473c494ce4fc5253c2bbd2790  result/@platforms.marker

And here's an example of differences:

$ cat result-a/@bazel_skylib.marker
7dc7472d37424ba5ec6a5532765bc911
$MANAGED
cat result-b/@bazel_skylib.marker
a8f3f577798201157128e8e9934c4705
$MANAGED

Instead of trying to patch these markers further we now completely clear them.
Nix hacks for ignoring markers are restored and expanded so that we don't even
attempt to parse the marker.
2019-07-31 11:28:06 +03:00
Wael Nasreddine
a8f174ff7c buildBazelPackage: autodetect nix toolchain instead of Xcode on Darwin (#65308)
* buildBazelPackage: autodetect nix toolchain instead of Xcode on Darwin

* do not export the variables outside of Darwin

* remove unecessary parens

* move comment within the darwin check
2019-07-25 11:43:45 +02:00
Nikolay Amiantov
4dad0ae560 buildBazelPackage: cleanup hacks and enforce them for all builds
Timestamp verification skip is no longer needed (not sure why). Generally we
better off always using the environment hack for all packages because that
ensures all NIX_* flags are correctly applied.

One possible improvement in future is to filter only NIX_* variables to
passthru in Bazel.
2019-07-17 10:40:38 +03:00
Timo Kaufmann
34387bcf5b buildBazelPackage: fix preConfigure without dependencies 2019-06-28 22:44:21 +02:00
Timo Kaufmann
255686e8fc bzildBazelPackage: fix fetch with no markers
Previously the installPhase of the fixed ouput derivation would fail for
a package that has no markers, since `sed` would complain about having
no input files. If we use `find` instead of bash globs, that problem
goes away.
2019-06-28 22:44:20 +02:00
Timo Kaufmann
883725b22f buildBazelPackage: disable multithreaded fetching
To work around https://github.com/bazelbuild/bazel/issues/6502.
2019-06-20 20:02:51 +02:00
Timo Kaufmann
0cfd90a109 buildBazelPackage: fix directory symlink handling
The previous behaviour would work fine as long as `symlink` is a link to
a file. If is a link to a directory though, the new `ln` wouldn't
overwrite it but would create a new link *in that directory* (with the
name of the link source).

Instead, we can precompute the target location, then first remove the
symlink and write the new one in its place.
2019-06-20 18:35:16 +02:00
Uri Baghin
130d987ab1 bazel-remote: init at 2019-01-12 2019-03-11 11:10:59 +01:00
Uri Baghin
718a82b108 buildBazelPackage: autodetect nix toolchain instead of Xcode one on macOS (#56033) 2019-02-19 16:44:05 +00:00
Uri Baghin
2b2854e522 bazel-deps: 2018-11-01 -> 2019-02-01 2019-02-01 17:30:36 +11:00
Wael M. Nasreddine
86a5535b2f
bazel-watcher: init at 0.5.0 2018-09-29 13:33:00 -07:00
Wael M. Nasreddine
90b7b4a509
build-bazel-package: remove any .git, .svn and .hg from external 2018-09-29 13:28:15 -07:00
Wael M. Nasreddine
18aa9b0b65
build-bazel-package: prefix bazel with the USER variable
Bazel computes the default value of output_user_root before parsing the
flag[0]. The computation of the default value involves getting the $USER
from the environment. I don't have that variable when building with
sandbox enabled.

[0]: 9323c57607/src/main/cpp/startup_options.cc (L123-L124)
2018-09-29 13:28:12 -07:00
Uri Baghin
ed98822350 bazel-deps: init at 2018-05-31 (#43018) 2018-07-09 22:38:45 +00:00
Nikolay Amiantov
430e0f4a80 buildBazelPackage: init
A separate function for building Bazel-bazed packages. Internally it splits the
build into two phases, fetching and building.

Users are expected to provide `fetchArgs.sha256` -- checksum of fetched
dependencies. Local dependencies should be removed in `fetchArgs.preInstall`.
Overall `fetchArgs` and `buildArgs` can be used to add specific steps to fetch
and build.
2018-02-20 20:39:49 +03:00