Commit Graph

91 Commits

Author SHA1 Message Date
Moritz Kiefer
aced78f0b4
Drop dadew uninstall (#12566)
Shouldn’t be necessary anymore now that all nodes have been reset and
it makes things slower and potentially more flaky.

changelog_begin
changelog_end
2022-01-25 12:11:16 +00:00
Moritz Kiefer
a150737d17
Upgrade msys2 (#12545)
* Upgrade msys2

We’ve seen some weird errors with outdated signatures and it seems
plausible that an upgrade may help.

changelog_begin
changelog_end

* .

changelog_begin
changelog_end
2022-01-24 10:58:34 +00:00
Moritz Kiefer
15cb84093b
Switch back to binary execution logs (#11440)
We are seeing lots of OOM issues on Windows after having switched to
the JSON format. They seem to happen after the actual build has
happened while writing out the exec log so trying to revert back to
the binary format seems promising.

changelog_begin
changelog_end
2021-10-28 08:55:35 +00:00
Moritz Kiefer
68d7f83a5f
Switch bazel exec logs to json format (#11430)
* Switch bazel exec logs to json format

changelog_begin
changelog_end

* change file extension

changelog_begin
changelog_end
2021-10-27 13:44:56 +00:00
Andreas Herrmann
7b94b0674e
Map shortened scala test suite names to long names on Windows (#10628)
* Generate short to long name mapping in aspect

Maps shortened test names in da_scala_test_suite on Windows to their
long name on Linux and MacOS.

Names are shortened on Windows to avoid exceeding MAX_PATH.

* Script to generate scala test name mapping

* Generate scala-test-suite-name-map.json on Windows

changelog_begin
changelog_end

* Generate UTF-8 with Unix line endings

Otherwise the file will be formatted using UTF-16 with CRLF line
endings, which confuses `jq` on Linux.

* Apply Scala test name remapping before ES upload

* Pipe bazel output into intermediate file

Bazel writes the output of --experimental_show_artifacts to stderr
instead of stdout. In Powershell this means that these outputs are not
plain strings, but instead error objects. Simply redirecting these to
stdout and piping them into further processing will lead to
indeterministically missing items or indeterministically introduced
additional newlines which may break paths.

To work around this we extract the error message from error objects,
introduce appropriate newlines, and write the output to a temporary file
before further processing.

This solution is taken and adapted from
https://stackoverflow.com/a/48671797/841562

* Add copyright header

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-08-24 17:03:45 +02:00
Andreas Herrmann
fe1b642e02
Don't fail if logs directory exists already (#10423)
changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-07-27 13:43:02 +00:00
Andreas Herrmann
630d021331
Collect build-event-stream (#9831)
* Generate Bazel logs and upload to GCS

changelog_begin
changelog_end

* Move git_*_sha into variables template

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-06-11 12:16:08 +02:00
Moritz Kiefer
0c514b3096
Fix Windows test skipping (#9881)
= doesn’t seem to work. -ceq is case insensitive comparison. I think
-eq would also work here but I see no point in worrying about case.

changelog_begin
changelog_end
2021-06-02 10:18:51 +00:00
Gary Verhaegen
b58d30b9e3
skip Windows tests on release (#9656)
Caching doesn't seem to work very well here. On a release, we build an
old commit, which has already been tested twice (once as a commit on
`main`, once as part of the release PR).

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-12 14:23:05 +02:00
Moritz Kiefer
979e12fa68
Move artifact publishing out of yaml files (#9071)
* Move artifact publishing out of yaml files

The current publishing process pretty much hardcodes the set of
artifacts we publish in the yaml config. This is a problem because we
always release from `main` so the yaml files are always
identical. However, we will add new artifacts over time and this
starts falling apart. This PR changes this such that the process
described in the yaml files is very generic and just uploads and
downloads everything in a directory whereas the details are handled in
bash scripts that will come from the respective release branch and are
therefore version-dependent.

As usual for these type of changes, I don’t have a great way to test
this. I did do some due diligence to test that at least the artifacts
are published correctly and I can download them but I can’t test the
actual publishing.

changelog_begin
changelog_end

* Update ci/copy-unix-release-artifacts.sh

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Update ci/copy-windows-release-artifacts.sh

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Update ci/publish-artifactory.sh

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2021-03-11 11:44:02 +01:00
Gary Verhaegen
403990dfd7
reenable Windows cache (#8329)
See #8326.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-17 15:58:01 +01:00
Moritz Kiefer
b32789025e
Reset Windows cache (#8326)
changelog_begin
changelog_end
2020-12-17 12:24:07 +00:00
Gary Verhaegen
2e25dfdc82
reenable Windows cache (#7426)
All machines have been reset.

CHANGELOG_BEGIN
CHANGELOG_END
2020-09-16 23:54:35 +02:00
Gary Verhaegen
f98b92d7ba
reset Windows cache (#7423)
CHANGELOG_BEGIN
CHANGELOG_END
2020-09-16 22:38:40 +02:00
Gary Verhaegen
5f897e699f
reenable Windows cache (#7216)
All machines have been reset following #7212.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-24 20:02:08 +00:00
Moritz Kiefer
5936644970
Bump windows cache (#7212)
changelog_begin
changelog_end
2020-08-24 18:19:28 +02:00
Gary Verhaegen
8b089f5689
remove clean expunge (#7061)
All nodes have been reset following #7056.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-07 14:11:02 +02:00
Moritz Kiefer
31c2ce0220
Bump Windows cache (#7056)
The "output was not created" errors seem to have become very
frequent. While taking out nodes seems to work as a bandaid, I’d like
to see if resetting the cache buys us a few days of not having to deal
with this. Admittedly, I don’t really have an explanation for why
resetting the cache should help if taking out the machines seems to do
something (suggesting that it hasn’t propagated fully).

changelog_begin
changelog_end
2020-08-07 09:05:32 +00:00
Moritz Kiefer
12c9612566
Remove redundant clean --expunge (#6890)
After the node resets this should hopefully not be necessary
anymore (we still had an issue this morning but I believe all nodes
that hit the issue also got the fix and if not, I’ll schedule a
targetted clean --expunge). I’ve also added node_modules to
.bazelignore to match the other node_modules directories.

changelog_begin
changelog_end
2020-07-28 09:49:36 +02:00
Moritz Kiefer
5668576b78
Upgrade rules-nodejs to the latest release (#6870)
changelog_begin
changelog_end
2020-07-27 16:50:23 +00:00
Gary Verhaegen
deb7f838ea
remove clean --expunge from Windows builds (#6844)
Following #6761, all nodes have been reset, so we can get back to fast,
cached builds.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-24 03:02:49 +02:00
Andreas Herrmann
4b1438276c
Update Bazel 2.1.0 --> 3.3.1 (#6761)
* Upgrade nixpkgs revision

* Remove unused minio

It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.

* Update Bazel on Windows

changelog_begin
changelog_end

* Fix hlint warnings

The nixpkgs update implied an hlint update which enabled new warnings.

* Fix "Error applying patch"

Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.

* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681

Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.

```
git diff ':(exclude)WORKSPACE'
```

* Update rules_nixpkgs

* nixpkgs location expansion escaping

* Drop --noincompatible_windows_native_test_wrapper

* client_server_test using sh_inline_test

client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.

This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.

* daml_test using sh_inline_test

* daml_doc_test using sh_inline_test

* _daml_validate_test using sh_inline_test

* damlc_compile_test using sh_inline_test

* client_server_test find .exe on Windows

* Bump Windows cache for Bazel update

Remove `clean --expunge` after merge.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-07-23 09:46:04 +02:00
Gary Verhaegen
f173ea966a
remove clean --expunge (#6783)
All nodes have been restarted following #6770, let's get our Windows
nodes back to some semblance of performance.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-17 15:00:10 +00:00
Moritz Kiefer
147a2700c0
Bump Windows cache (#6770)
To “fix” the “output was not created” errors.

changelog_begin
changelog_end
2020-07-17 12:41:10 +02:00
Gary Verhaegen
b4915a4bd7
remove clean --expunge from Windows builds (#6605)
Following #6604, I have reset all of our Windows nodes, so hopefully
there is no broken local cache remaining.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-03 19:24:54 +00:00
Moritz Kiefer
ade99dd2c1
Reset windows cache (#6604)
We are seeing caching errors again.

changelog_begin
changelog_end
2020-07-03 16:36:35 +00:00
Gary Verhaegen
952320801f
remove clean --expunge (#6394)
Following #6383, all nodes have been reset.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-17 22:35:09 +02:00
Moritz Kiefer
7e0a684857
Bump Windows cache (#6383)
changelog_begin
changelog_end
2020-06-17 19:33:26 +02:00
Gary Verhaegen
087d03f9f6
remove bazel cleanup (#6331)
See #6321

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-12 20:21:45 +02:00
Andreas Herrmann
d1e422580a
Increment Windows cache URL (#6321)
We've seen a series of failures of the form
```
ERROR: D:/a/1/s/daml-assistant/integration-tests/BUILD.bazel:162:1: output 'daml-assistant/integration-tests/create-daml-app-tests.exe' was not created
ERROR: D:/a/1/s/daml-assistant/integration-tests/BUILD.bazel:162:1: not all outputs were created or valid
```
across multiple machines. We suspect cache poisoning as the cause. This
increments the cache URL to effectively clear the cache.

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-06-12 15:33:38 +02:00
Moritz Kiefer
90246ded32
Remove clean --expunge again (#6275)
We manually reset all Windows nodes so this is no longer necessary.

changelog_begin
changelog_end
2020-06-10 01:23:25 +02:00
Moritz Kiefer
20d26394e1
Modify the cache URL instead of relying on platform_suffix (#6273)
For some reason, platform_suffix doesn’t seem to provide enough
isolation to fix the “undeclared inclusion” errors even though it does
fix the issues for me locally.

This PR tries to address the problem by switching from
`platform_suffix` to modifying the actual URL of the cache.

To avoid leaking stuff from the local cache, I’ve added a clean
--expunge for now. We should be able to remove this once nodes have
been reset tomorrow. It will slow down nodes but that is clearly
better than having everything fail.

changelog_begin
changelog_end
2020-06-09 17:05:19 +02:00
Gary Verhaegen
b9fbba7fc5
shorten Windows CI username (#6190)
Keeping CI working on Windows involves a constant fight against
MAX_PATH, which is a very short 260 characters. As the username appears
in some paths, sometimes multiple times, we can save a few precious
characters by having it shorter.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-06 15:03:15 +02:00
Andreas Herrmann
4c99f67814
Publish Bazel logs (#5821)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-05-04 11:14:40 +02:00
Moritz Kiefer
590833608c
Fix Windows CI (#5648)
The python install step started failing because there is a new pip
release so we get a warning. I tried to just upgrade python but the
new python manifest depends on functions that need an upgrade of scoop
so I upgraded that as well. Not quite sure which of those upgrades
fixes the issue …

changelog_begin
changelog_end
2020-04-21 13:31:57 +00:00
Gary Verhaegen
433f484188
stop producing/publishing the Bazel execution logs (#4527)
They can weigh close to 1GB, and the internal Azure networks are
unreliable enough that this adds up significant amounts of time to our
already slow CI pipeline (usually around a minute, but I've seen at
least one case where it took almost 28 minutes).

Given that we pretty much never look a them, 🔥.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-14 14:34:01 +01:00
Moritz Kiefer
9b529ad8cb
Cleanup typescript hacks for Windows (#4167)
changelog_begin
changelog_end
2020-01-22 17:19:45 +01:00
Moritz Kiefer
970d9891fd
Disable all the TS stuff on Windows (#4163)
* Disable all the TS stuff on Windows

changelog_begin
changelog_end

* disable jest explicitly

* more disabling

* :sadpanda:

* Replace @language_support_ts_deps on Windows

Provides dummy content so that `load` commands are still valid on
Windows without `yarn_install`.

* disable daml-ledger-fetch on windows

* shut up buildifier

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-22 16:07:20 +01:00
Moritz Kiefer
2653c6f1fe
Disable eslint_test on Windows (#4161)
* Disable eslint_test on Windows

This is currently breaking CI completely so for now let’s disable it.

changelog_begin
changelog_end

* clean expunge
2020-01-22 14:43:15 +01:00
Andreas Herrmann
91f5dd573f stack update (#3935)
* Remove manual stack update

* Update rules_haskell

* rules_haskell_worker_dependencies after bazel-haskell-deps

* Update rules_haskell Windows patch

* make cabal haddock optional

* Don't generate Haddocks on stack_snapshot

Fails with ghc-lib and takes more time to build.

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-06 10:24:15 +00:00
Andreas Herrmann
02f806bf22 Manually run stack update (#3590)
* Add stack to dev-env

* Manually run stack update before build

To work around multiple concurrent attempts to take the
hackage-security-lock.
2019-11-22 15:16:48 +00:00
Andreas Herrmann
f4d0eb636e Replace Hazel by stack_snapshot (#2743)
* Unmangled libz.so and libbz2.so

* Use stack_snapshot instead of Hazel

* Remove Hazel

* Define stack_snapshot

* Update rules_haskell

* Document stack_snapshot

* Clean stack's lock file from aborted builds
2019-11-22 14:24:08 +00:00
Moritz Kiefer
0539e3e8d0
Prefetch nodejs_dev_env on Windows (#3447)
Maybe this helps ¯\_(ツ)_/¯
2019-11-13 12:47:07 +01:00
Andreas Herrmann
ed359ef714
Drop Bazel dadew prefetch (#3411)
This should no longer be necessary since `dadew enable` has been dropped
from `dev_env_tool`. `dadew enable` caused a call to `scoop config`
which caused write collisions when executed in parallel.
2019-11-11 13:21:02 +01:00
Moritz Kiefer
e6bc0213ac
Prefetch all dev-env tools (#3409)
* Prefetch all dev-env tools

maybe this helps with some of the CI flakiness that we have seen.

* no enable
2019-11-11 12:33:52 +01:00
Andreas Herrmann
33e47828e3
Bazel 1.1 (#3249)
* bazel: 0.28.1 --> 1.1.0

* bazel-watcher sha256

* Fix missing line in patch

* proto_source_root --> strip_import_prefix

See https://github.com/bazelbuild/bazel/issues/7153 for details.

* Update rules_nixpkgs

Required to avoid errors of the form
```
ERROR: An error occurred during the fetch of repository 'node_nix':
   parameter 'sep' may not be specified by name, for call to method split(sep, maxsplit = None) of 'string'
```

and
```
ERROR: An error occurred during the fetch of repository 'node_nix':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
		_execute_or_fail(repository_ctx, <3 more arguments>)
	File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
		fail(<1 more arguments>)

Cannot build Nix attribute 'nodejs'.
Command: [/Users/runner/.nix-profile/bin/nix-build, /private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/node_nix/nix/bazel.nix, "-A", "nodejs", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
src/main/tools/process-tools.cc:173: "setitimer": Invalid argument
```

* Update rules_scala

* .proto has been removed, use [ProtoInfo] instead

See
https://docs.bazel.build/versions/1.1.0/be/protocol-buffer.html#proto_library

* python3_nix add nix_file attribute

To avoid the following error

```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/BUILD:66:1: //:nix_python3_runtime depends on @python3_nix//:bin/python in repository @python3_nix which failed to fetch. no such package '@python3_nix//': Traceback (most recent call last):
        File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
                _execute_or_fail(repository_ctx, <3 more arguments>)
        File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
                fail(<1 more arguments>)

Cannot build Nix attribute 'python3'.
Command: [/home/aj/.nix-profile/bin/nix-build, "-E", "import <nixpkgs> { config = {}; overlays = []; }", "-A", "python3", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
error: anonymous function at /home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs.nix:3:1 called with unexpected argument 'config', at (string):1:1
```

* rules_haskell unnamed string.split(_, maxsplit = _)

The keyword argument may no longer be named.

* string.replace(_, _, maxsplit = _) may not be named

* Move proto sources from deps to data

Fixes

```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation
```

* Define sha256 for haskell_ghc__paths

Bazel 1.1.0 fails on missing hashes.

* Disable --incompatible_windows_native_test_wrapper

* //compiler/daml-extension don't modify sources

Modifying sources in-place can cause issues on Windows, where build
actions are not sandboxed and changes on sources can affect other build
steps.

* bazel-genfiles --> bazel-bin

The bazel-genfiles symlink has been removed since Bazel 1.0.
See https://github.com/bazelbuild/bazel/issues/8651

* Mark dev_env_tool repository rule as configure

See
https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule

* Move data deps into data attribute

* Mark dev_env_tool as local = True

* Manually fetch @makensis_dev_env
2019-11-11 10:06:03 +01:00
Michał Majcherski
2afefb7883
windows: root build (#1894)
* windows: root build

* windows: fixed haskell bindings tests

* windows: disable client_server_test test

* windows: marking daml_test flaky due to #1907

* windows: removing da-hs-damlc-app run from build.ps1

* windows: disable hie-core alias of currently disabled target
2019-06-28 12:55:31 +02:00
Michał Majcherski
7878120d5b
Fix hanging FFI call to scenario service problem on Windows (#1868)
* windows: hanging GRPC FFI call problem resolution

* windows: fix visual test

* windows: enabled back again some of disabled daml-assitant tests

* marking daml_ghc_integration_test as large
2019-06-26 11:45:52 +02:00
Michał Majcherski
471e022002 windows: CI - more targets (#1720) 2019-06-18 08:20:31 +00:00
Michał Majcherski
285c766e16
windows: ledger and extractor tests; improved msys2 installation (#1643)
* windows: ledger tests

* windows: extractor tests

* windows: combine msys2 manifest files
2019-06-17 09:40:28 +02:00