Limit the number of parallel actions spawned by Bazel to the number of
physical CPUs. (Assuming a CPU with hyperthreading enabled)
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Update to Java 11
changelog_begin
changelog_end
* Fix RoundingMode deprecation warnings
* Fix dep-ann warning
* Integer constructor
* JavaX annotation dependency
* javax.xml.bind was removed in Java 11
Using Guava as a replacement, since it is already a project dependency.
* JDK 11 no longer has a separate JRE tree
* Remove unused jdk_nix import
* remove now redundant jdk11_nix
* Java 8 --> 9 increased Instant.now() precision
See https://bugs.openjdk.java.net/browse/JDK-8068730
The precision of `Instant.now()` increased between Java 8 and Java 9.
On Linux and MacOS this doesn't seem to be a problem, as the precision
still seems to be at micro seconds. However, on Windows this now causes
errors of the following form:
```
java.lang.IllegalArgumentException: Conversion of Instant
2021-11-05T13:58:56.726875100Z to microsecond granularity would result
in loss of precision.
```
Suggesting that it now offers sub-microsecond precision.
`TimestampConversion.instantToMicros` had a check to fail if the
conversion lead to a loss of precision. In the specific failing test
case this is not a concern, so this adds a `roundInstantToMicros`
variant that avoids this kind of error.
* TMP round timestamps
* Revert "TMP round timestamps"
This reverts commit af8e261278.
* Skip versions before 1.6.0 in migration tests
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Drop Scala 2.12 support
This only includes the CI/build system infrastructure
changes. Dropping compatibility layers from our code for 2.12 can be
done separately.
This is fine even in the context of backport builds since we already
disable the Scala 2.12 job for those anyway.
fixes#11315
changelog_begin
changelog_end
* Update bazel-java-deps.bzl
Co-authored-by: pbatko-da <pawel.batko@digitalasset.com>
Co-authored-by: pbatko-da <pawel.batko@digitalasset.com>
* Fix rules_scala Windows launcher
* Define nixpkgs_java_configure
To generate a java_runtime based on a Nix provided JDK similar to how it
is done with the Python and CC toolchain.
changelog_begin
changelog_end
* Enable Nix provided Java runtime on Linux and MacOS
* Define dadew_java_configure
To import a JDK and JAVA_HOME from dadew similar to
nixpkgs_java_configure.
* Enable dadew Java runtime on Windows
* Remove now unused java_home_runtime
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
We are seeing lots of OOM issues on Windows after having switched to
the JSON format. They seem to happen after the actual build has
happened while writing out the exec log so trying to revert back to
the binary format seems promising.
changelog_begin
changelog_end
* Bump bazel memory limit to 3gb
We keep getting oom’s on Windows from Bazel itself afaict. I suspect
our build has just gotten too big. Having to increase the memory is a
bit worrying but it also doesn’t seem reasonable to expect the memory
size to stay indepenent of the build size.
changelog_begin
changelog_end
* copypasta
changelog_begin
changelog_end
* Enable --incompatible_remote_results_ignore_disk
> If set to true, --noremote_upload_local_results and --noremote_accept_cached will not apply to the disk cache. If a combined cache is used: --noremote_upload_local_results will cause results to be written to the disk cache, but not uploaded to the remote cache. --noremote_accept_cached will result in Bazel checking for results in the disk cache, but not in the remote cache. See #8216 for details.
changelog_begin
changelog_end
* Fix compatibility workspace
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Bump bazel max jvm memory
I kept the starting memory the same. Happy to bump that as well if you
think it makes sense.
changelog_begin
changelog_end
* Bump xms as well
changelog_begin
changelog_end
* Switch Scala version Bazel config
changelog_begin
changelog_end
This allows to easily test with Scala 2.12.x locally. The `scala_2_13` Bazel
configuration is no longer needed becase now Scala 2.13.x is the default.
* Address https://github.com/digital-asset/daml/pull/9848#discussion_r642381690
Details are in the comments but this seems to workaround the annoying
tls issues we have been dealing with for the past few days.
changelog_begin
changelog_end
CHANGELOG_BEGIN
- Our Linux binaries are now built on Ubuntu 20.04 instead of 16.04. We
do not expect any user-level impact, but please reach out if you
do notice any issue that might be caused by this.
CHANGELOG_END
* Add a Scala 2.13 build pipeline
This adds initial support for multiple Scala versions controlled via
the DAML_SCALA_VERSION env var and a CI job to make sure we don’t
regress. For now we only test //libs-scala/ports/... which seemed like
the easiest starting point I could find. We can incrementally expand
that over time.
changelog_begin
changelog_end
* Document pinning
changelog_begin
changelog_end
* Address review comments
changelog_begin
changelog_end
* Upgrade nixpkgs revision
* Remove unused minio
It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.
* Update Bazel on Windows
changelog_begin
changelog_end
* Fix hlint warnings
The nixpkgs update implied an hlint update which enabled new warnings.
* Fix "Error applying patch"
Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.
* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681
Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.
```
git diff ':(exclude)WORKSPACE'
```
* Update rules_nixpkgs
* nixpkgs location expansion escaping
* Drop --noincompatible_windows_native_test_wrapper
* client_server_test using sh_inline_test
client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.
This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.
* daml_test using sh_inline_test
* daml_doc_test using sh_inline_test
* _daml_validate_test using sh_inline_test
* damlc_compile_test using sh_inline_test
* client_server_test find .exe on Windows
* Bump Windows cache for Bazel update
Remove `clean --expunge` after merge.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
For some reason, platform_suffix doesn’t seem to provide enough
isolation to fix the “undeclared inclusion” errors even though it does
fix the issues for me locally.
This PR tries to address the problem by switching from
`platform_suffix` to modifying the actual URL of the cache.
To avoid leaking stuff from the local cache, I’ve added a clean
--expunge for now. We should be able to remove this once nodes have
been reset tomorrow. It will slow down nodes but that is clearly
better than having everything fail.
changelog_begin
changelog_end
* Bump cache suffix
As discussed, we are going to bump this every time we feel like
resetting the cache might help. This is a temporary measure to get
some metrics on how often things break and if resetting the cache
helps.
changelog_begin
changelog_end
* Update configure-bazel as well
changelog_begin
changelog_end
It seems to use the default JVM options, which is 1/4 of available RAM,
or 8 GB on a machine with 32 GB of memory. This seems quite unnecessary;
it appears to work fine with much less than that.
CHANGELOG_BEGIN
CHANGELOG_END
The docs build is currently not reproducible as it include to-the-minute
time-of-build information. It also includes some Sphinx binary caches
which I suppose will also not be reproducible (though I have not checked
the details there).
This commit attempts to remove all sources of non-reproducibility from
the docs build, though this is hard to test without having a stable,
older release to compare with.
CHANGELOG_BEGIN
CHANGELOG_END
* bazel: 0.28.1 --> 1.1.0
* bazel-watcher sha256
* Fix missing line in patch
* proto_source_root --> strip_import_prefix
See https://github.com/bazelbuild/bazel/issues/7153 for details.
* Update rules_nixpkgs
Required to avoid errors of the form
```
ERROR: An error occurred during the fetch of repository 'node_nix':
parameter 'sep' may not be specified by name, for call to method split(sep, maxsplit = None) of 'string'
```
and
```
ERROR: An error occurred during the fetch of repository 'node_nix':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'nodejs'.
Command: [/Users/runner/.nix-profile/bin/nix-build, /private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/node_nix/nix/bazel.nix, "-A", "nodejs", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
src/main/tools/process-tools.cc:173: "setitimer": Invalid argument
```
* Update rules_scala
* .proto has been removed, use [ProtoInfo] instead
See
https://docs.bazel.build/versions/1.1.0/be/protocol-buffer.html#proto_library
* python3_nix add nix_file attribute
To avoid the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/BUILD:66:1: //:nix_python3_runtime depends on @python3_nix//:bin/python in repository @python3_nix which failed to fetch. no such package '@python3_nix//': Traceback (most recent call last):
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'python3'.
Command: [/home/aj/.nix-profile/bin/nix-build, "-E", "import <nixpkgs> { config = {}; overlays = []; }", "-A", "python3", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
error: anonymous function at /home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs.nix:3:1 called with unexpected argument 'config', at (string):1:1
```
* rules_haskell unnamed string.split(_, maxsplit = _)
The keyword argument may no longer be named.
* string.replace(_, _, maxsplit = _) may not be named
* Move proto sources from deps to data
Fixes
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation
```
* Define sha256 for haskell_ghc__paths
Bazel 1.1.0 fails on missing hashes.
* Disable --incompatible_windows_native_test_wrapper
* //compiler/daml-extension don't modify sources
Modifying sources in-place can cause issues on Windows, where build
actions are not sandboxed and changes on sources can affect other build
steps.
* bazel-genfiles --> bazel-bin
The bazel-genfiles symlink has been removed since Bazel 1.0.
See https://github.com/bazelbuild/bazel/issues/8651
* Mark dev_env_tool repository rule as configure
See
https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule
* Move data deps into data attribute
* Mark dev_env_tool as local = True
* Manually fetch @makensis_dev_env
* Remove git-revision from dependency graph
The navigator binary depended on the current git-revision. The issue is
that any target/test depending on the git-revision will have to be
rebuilt/rerun on every commit.
The navigator package itself was careful to avoid unnecessary
rebuilds/reruns. However, the SDK release tarball depends on the
navigator binary and thereby on the git-revision. The daml-assistant
integration tests, which depend on the SDK release tarball, therefore
had to be rerun on every commit.
This change removes the git-revision from the navigator alltogether. For
issue reporting the SDK version is still available.
* Remove unused git-revision and workspace_status
The `.bazelrc` file specified `--test_env` as a `test` option only.
However, this is a `build` option as well. Only specifying it for `test`
caused Bazel to unnecessarily redo the analysis phase:
```
INFO: Build option --test_env has changed, discarding analysis cache.
```
This makes sure that C dependencies like gRPC or zlib get compiled
with optimizations. I patched rules_haskell to use -O instead of -O2
since the latter slows down compilation while not making things
faster (according to my measurements).
* Bazel: 0.24.0 -> 0.27.0
* Update rules_haskell for Bazel 0.27 compatibility
* Update bazel-deps and bazel-watcher
* Windows escape JVM flags
* load commands at top of .bzl file
Bazel 0.27 no longer allows load commands that are not at the beginning
of the file.
* Update Bazel rules
* subpackage boundary
* native is not defined in BUILD files
* yarn: @bazel/hide-bazel-files
Seems to be required since latest rules_nodejs version. Otherwise, yarn
fails with errors about existing BUILD or BUILD.bazel files.
* grpc-java plugin visibility
* Update fat_cc_library
* Nix Python3 toolchain
* Iteration over depset
* dev_env_package: Create symlinks one level deeper
To prevent symlinking the BUILD file as well. The nested BUILD file
confuses Bazel as of 0.27 and rules_nodejs cannot find the node
executable anymore.
* Update rules_nodejs
* Add managed_directories for node_modules
* hie-bios: Extract bazel-genfiles from bazel info
Bazel 0.27 changed the genfiles location which breaks the hie-core test
on macOS.
* update cc_wrapper to Bazel 0.27
* bazel info -> bazel info bazel-genfiles
* Fix typo in BUILD
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update rules_haskell
- rules_haskell now handles the global package db within Bazel
https://github.com/tweag/rules_haskell/pull/859
- We no longer use the Nix provided c2hs. So, we drop it.
- Rename `ghcWithC2hs` to `ghcStatic` to clarify that that's where the
static linking patches are applied.
- Extend package-db patches to align Nix store paths with the new $out.
This works around a restriction in current rules_haskell, where
the paths in the package config files must have the same prefix as
the path to the package config files themselves.
- Don't exclude haskell libraries from extra-libraries entries.
* Drop redundant unix-compat override
This is a left-over from when the package was patched.
* Windows GHC bindist includes ffi header
* Drop unused language-c Nix override
This allows you to use the disk cache (you cannot have both enabled at
the same time) locally and we disabled fetching from the cache on
Windows (outside of CI) anyway, so passing --remote_http_cache was a noop.
