Commit Graph

193 Commits

Author SHA1 Message Date
Moritz Kiefer
c44f33736e
Simplify speedy benchmark (#10079)
This PR drops two things:

1. The check that the benchmark hasn’t been modified. This hasn’t ever
been useful and it keeps being annoying.
2. It stops the comparison against the old version and instead just
benchmarks the current version. We really only care about the day to
day changes. Comparing against an arbitrary year old version has lost
all meaning at this point.

changelog_begin
changelog_end
2021-06-22 15:11:47 +00:00
Moritz Kiefer
8399039a8a
Remove broken dep from release job (#9963)
Now that we skip the scala_2_12 job this dep results in us never
running the release job which is clearly not intentional.

changelog_begin
changelog_end
2021-06-11 07:53:50 +02:00
Gary Verhaegen
df0f3ff7b0
maybe fix deleted release tags (#9891)
* maybe fix deleted release tags

Hard to know for sure until the next broken release.

CHANGELOG_BEGIN
CHANGELOG_END

* Update azure-pipelines.yml

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-06-02 16:06:56 +00:00
Moritz Kiefer
e033277804
Switch to our own nodes for releases (#9845)
Ubuntu 16.04 nodes are about to go away and there dosen’t seem to be a
compelling reason to use the hosted nodes here anyway.

This is obviously hard/impossible to test without a release but I did
do some basic sanity checks: The release build seems to rely on the
following tools (+ things like rm which are clearly there)
1. git
2. gpg
3. sha256sum
4. gcloud
5. curl

All of those are installed on our own nodes. If this doesn’t work we
could also run (parts of) this in dev-env but for now this should do
the job.

changelog_begin
changelog_end

.

changelog_begin
changelog_end

.

changelog_begin
changelog_end
2021-05-31 09:18:56 +00:00
Gary Verhaegen
646c956457
new windows signing (#9786)
CHANGELOG_BEGIN
CHANGELOG_END
2021-05-25 16:23:17 +02:00
Gary Verhaegen
fb6f72c81d
debug signtool on main (#9780)
I don't know what's going on. I was hoping I could try to debug from
PRs, but that doesn't seem to work.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-24 13:26:41 +00:00
Gary Verhaegen
61293c3ec4
maybe fix azuresigntool invocation (#9779)
🤷

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 17:01:27 +00:00
Gary Verhaegen
45bca6e68b
test_windows_signing: install for u (#9776)
Turns out "`--global`" means "for this user".

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 15:49:31 +02:00
Gary Verhaegen
b8afa02e12
test_windows_signing: add quotes (#9773)
Otherwise Bash interprets the baskslashes and it's not a path anymore.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 11:12:09 +00:00
Gary Verhaegen
5717356a42
fix test_windows_signing job name (#9771)
😢

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 10:33:25 +00:00
Gary Verhaegen
f5c5b634eb
prepare for EV Windows signing (#9758)
Setting up a non-disruptive way to test out EV signing of our Windows
artifacts.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 10:46:45 +02:00
Moritz Kiefer
b1738c7202
Switch Scala default to 2.13 (#9699)
changelog_begin
changelog_end
2021-05-17 15:04:53 +00:00
Gary Verhaegen
17f8bcd37d
fix release job (#9166)
Three issues here:

1. The release job runs on an Azure-hosted agent, so it doesn't have the
   `reset_caches.sh` script (and doesn't need it).
2. The `bash-lib` step should not run if the current job has already
   failed.
3. The `skip-github` jobs should also not run if the job  has failed.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-17 09:46:02 +00:00
Gary Verhaegen
089a11443c
release job: remove extra bash-lib (#9136)
The filesystem is not reset between steps in the same job, so there is
no need to add the `bash-lib` multiple times.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-15 07:53:26 +00:00
Moritz Kiefer
5dddf0aead
Add missing checkout step (#9133)
The revamped release process relies on publish.sh coming from the
release tag. However without the checkout that doesn’t work. This
resulted in us trying to push the EE tarball for a branch that does
not have it yet.

changelog_begin
changelog_end
2021-03-14 13:58:21 +01:00
Moritz Kiefer
707c86aa93
Fix GCS upload on releases (#9111)
* Fix GCS upload on releases

changelog_begin
changelog_end

* Update azure-pipelines.yml

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2021-03-12 11:39:56 +01:00
Moritz Kiefer
d1031ba1e9
Fix typo in azure-pipelines.yml (#9087)
changelog_begin
changelog_end
2021-03-11 13:27:18 +00:00
Moritz Kiefer
979e12fa68
Move artifact publishing out of yaml files (#9071)
* Move artifact publishing out of yaml files

The current publishing process pretty much hardcodes the set of
artifacts we publish in the yaml config. This is a problem because we
always release from `main` so the yaml files are always
identical. However, we will add new artifacts over time and this
starts falling apart. This PR changes this such that the process
described in the yaml files is very generic and just uploads and
downloads everything in a directory whereas the details are handled in
bash scripts that will come from the respective release branch and are
therefore version-dependent.

As usual for these type of changes, I don’t have a great way to test
this. I did do some due diligence to test that at least the artifacts
are published correctly and I can download them but I can’t test the
actual publishing.

changelog_begin
changelog_end

* Update ci/copy-unix-release-artifacts.sh

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Update ci/copy-windows-release-artifacts.sh

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Update ci/publish-artifactory.sh

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2021-03-11 11:44:02 +01:00
Gary Verhaegen
f0ca332983
fix release notification (#9075)
CHANGELOG_BEGIN
CHANGELOG_END
2021-03-10 17:12:14 +00:00
Gary Verhaegen
32e8c66d8d
fix typo in release notification (#9069)
So that on non-automated releases we get nothing instead of an empty
diamond.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-10 11:38:12 +00:00
Gary Verhaegen
41d62ea5c0
send CI failures on separate channel (#9038)
This will send CI failures to `#ci-failures-daml`.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-05 14:10:30 +00:00
Gary Verhaegen
e26d006f6f
slack: reduce @here mentions (#9037)
The people who care about these alerts monitor the channel closely
enough anyway, and having frequent automated @here bells ringing makes
it harder for individuals to highlight important messages.

CHANGELOG_BEGIN
CHANGELOG_END
2021-03-05 13:55:58 +00:00
Moritz Kiefer
ba6ba9019f
Release Scala 2.13 artifacts (#8858)
* Release Scala 2.13 artifacts

changelog_begin
changelog_end

* Dedup default scala version

changelog_begin
changelog_end
2021-02-17 14:32:04 +01:00
Gary Verhaegen
cd33c2015c
ci: use setvar to set variables (#8664)
My goal here is to investigate the new warning Azure has been showing
for the past few days:

> ##[warning]%25 detected in ##vso command. In March 2021, the agent command parser will be updated to unescape this to %. To opt out of this behavior, set a job level variable DECODE_PERCENTS to false. Setting to true will force this behavior immediately. More information can be found at https://github.com/microsoft/azure-pipelines-agent/blob/master/docs/design/percentEncoding.md

As far as I'm aware we are not deliberately passing in any `%25` in any
of our `vso` commands, so I was a bit surprised by this.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-09 11:42:34 +01:00
Gary Verhaegen
2143862a79
ci/prs: add artifactory credentials (#8752)
Changes already made on both Artifactory and Azure; this is mainly to
test.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-09 10:36:15 +00:00
Gary Verhaegen
fef712bf60
Upgrade linux nodes to 20.04 (#8617)
CHANGELOG_BEGIN

- Our Linux binaries are now built on Ubuntu 20.04 instead of 16.04. We
  do not expect any user-level impact, but please reach out if you
  do notice any issue that might be caused by this.

CHANGELOG_END
2021-01-27 17:38:34 +01:00
Andreas Herrmann
5693394650
Publish JARs for trigger service and OAuth 2.0 middleware (#8614)
* include oauth2 logback config in release tarball

overlooked in https://github.com/digital-asset/daml/pull/8611

* Release trigger-service and oauth2-middleware JARs

changelog_begin
changelog_end

* drop from artifacts.yaml

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-01-25 09:14:13 +00:00
Gary Verhaegen
532f996f12
ci: clean-up hard drive more often (#8582)
CHANGELOG_BEGIN
CHANGELOG_END
2021-01-20 17:22:53 +01:00
Gary Verhaegen
e6d1f1399b
fix out-of-rotation release notifs (#8491)
Only ping people on rotation for automated releases.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-13 14:44:23 +01:00
Gary Verhaegen
742feb58be
trigger PRs job on generated PRs (#8489)
CHANGELOG_BEGIN
CHANGELOG_END
2021-01-13 11:30:17 +00:00
Gary Verhaegen
19ed19c3c6
fix Artifactory hashes (#8414)
Apparently it's been silently broken since introduction all the way back
in #7196.

Huge thanks to @SamirTalwar-DA for reporting.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-06 15:36:16 +01:00
Gary Verhaegen
f45a065c54
ci: pin down Ubuntu versions (#8388)
For a couple weeks now there has been a warning on the Azure Pipelines
web UI that says `ubuntu-latest` is in the process of switching from
18.04 to 20.04. I am not aware of any specific issue this would cause
for our particular workflows, but I don't like my dependencies changing
from under me.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-05 10:39:59 +01:00
Gary Verhaegen
7edc57712d
separate CI config for main & prs (#8374)
This is another take on #8276, with the same underlying motivation.
However, this approach is mostly duplication-free, which seems better,
especially given the already-pretty-sorry state of our CI config.

Like #8276, this is done in 2 commits for ease of review. The first
commit is wholly unintresting and just copies `azure-pipelines.yml` to
both `ci/prs.yml` and `ci/build.yml`; the second commit removes from
each part what it shouldn't have. The intention is for `ci/build.yml` to
have all of the common parts.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-04 20:45:30 +01:00
Moritz Kiefer
b324fc4551
Port //daml-lf/(transaction|language) to Scala 2.13 (#8343)
changelog_begin
changelog_end
2021-01-04 10:54:38 +01:00
Gary Verhaegen
a925f0174c
update copyright notices for 2021 (#8257)
* update copyright notices for 2021

To be merged on 2021-01-01.

CHANGELOG_BEGIN
CHANGELOG_END

* patch-bazel-windows & da-ghc-lib
2021-01-01 19:49:51 +01:00
Gary Verhaegen
93f449d245
rename master to main (#8245)
As we strive for more inclusiveness, we are becoming less comfortable
with historically-charged terms being used in our everyday work.

This is targeted for merge on Dec 26, _after_ the necessary
corresponding changes at both the GitHub and Azure Pipelines levels.

CHANGELOG_BEGIN

- DAML Connect development is now conducted from the `main` branch,
  rather than the `master` one. If you had any dependency on the
  digital-asset/daml repository, you will need to update this parameter.

CHANGELOG_END
2020-12-27 14:19:07 +01:00
Moritz Kiefer
cb57ad374c
Port //daml-lf/data to Scala 2.13 (#8321)
* Port //daml-lf/data to Scala 2.13

changelog_begin
changelog_end

* factor common ImmArraySeq code to version-agnostic file

- ImmArraySeq itself is agnostic; the 2.12 and 2.13 versions contain
  implementation mixins/superclasses for parts that must be specific.  The 2.13
  version will collapse into the agnostic version when 2.12 support is no longer
  desired

* factor common InsertOrdMap code to version-agnostic file

- InsertOrdMap itself is agnostic; the 2.12 and 2.13 versions contain
  implementation mixins/superclasses for parts that must be specific.  The 2.13
  version will collapse into the agnostic version when 2.12 support is no longer
  desired

* factor common InsertOrdSet code to version-agnostic file

- InsertOrdSet itself is agnostic; the 2.12 and 2.13 versions contain
  implementation mixins/superclasses for parts that must be specific.  The 2.13
  version will collapse into the agnostic version when 2.12 support is no longer
  desired

* factor Map removal

* Move ImmArraySeq back into ImmArray

changelog_begin
changelog_end

* Type assertion instead of symbol

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-12-18 13:20:15 +01:00
Moritz Kiefer
7d6e8f8d5c
Do not checkout release commit in scala 2.13 (#8291)
* Do not checkout release commit in scala 2.13

At least for now, this fails because older commits don’t have the
necessary infrastructure so far. At some point we probably do want to
run this on release commits but that can wait a bit and this unblocks
release PRs for bugfix releases.

changelog_begin
changelog_end

* Update azure-pipelines.yml

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2020-12-15 10:13:24 +00:00
Moritz Kiefer
1b32044a74
Build //libs-scala/... on 2.13 (#8282)
* Build //libs-scala/... on 2.13

One test is unfortunately disabled at the moment since I utterly
failed to figure out why I get a ClassNotFoundException on 2.13.

changelog_begin
changelog_end

* Copyright headers

changelog_begin
changelog_end

* I can’t bazel today

changelog_begin
changelog_end

* Apply suggestions from code review

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Update libs-scala/resources/src/main/2.13/com/daml/resources/UnitCanBuildFrom.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* No split on view

changelog_begin
changelog_end

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-12-14 17:36:53 +00:00
Moritz Kiefer
d24ef94f36
Add a Scala 2.13 build pipeline (#8271)
* Add a Scala 2.13 build pipeline

This adds initial support for multiple Scala versions controlled via
the DAML_SCALA_VERSION env var and a CI job to make sure we don’t
regress. For now we only test //libs-scala/ports/... which seemed like
the easiest starting point I could find. We can incrementally expand
that over time.

changelog_begin
changelog_end

* Document pinning

changelog_begin
changelog_end

* Address review comments

changelog_begin
changelog_end
2020-12-14 12:42:26 +01:00
Gary Verhaegen
029c655adc
blackduck: open PR on NOTICES file change (#8215)
CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 10:08:28 +01:00
Gary Verhaegen
46b7decbca
notify_user: consider changelog entries (#8166)
The current implementation of the notify_user job sometimes reports
success while the build has actually failed. Azure does not provide a
way to query the overall state of the current build, so a general
solution to this problem does not seem possible (see #6796 for an
example of a failed attempt). However, all reported cases were
specifically about the `check_changelog_entry` job, which we can easily
query for, so this PR does that.

Note: originally pushed without a changelog entry to test new
notification mechanism.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-04 12:16:34 +00:00
Gary Verhaegen
3e4fec8951
release process: tweak automated messages (#8150)
I believe those will be a bit clearer.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-03 13:44:24 +00:00
Gary Verhaegen
b23304c691
add default capability to macos (#5915)
This is the macOS part of #5912, which I have separated because our
macOS nodes have a different deployment process so it seemed easier to
track the deployment of the change separately.

CHANGELOG_BEGIN
CHANGELOG_END
2020-11-25 15:34:33 +01:00
Gary Verhaegen
ffd9da60ee
remove finished job (#7881)
It looks like Azure is correctly reporting the overall job completion
again. See #7790 for details.

CHANGELOG_BEGIN
CHANGELOG_END
2020-11-04 12:38:47 +01:00
fabiotudone-da
0665650c30
Remove integrity check V1 and ledger dumps upload (#7839)
* Remove integrity check V1 and ledger dumps upload

CHANGELOG_BEGIN
CHANGELOG_END

* Revert unrelated changes

* Fix packages

* Formatting

* Simplify kvutils/BUILD.bazel

* Format kvutils/BUILD.bazel
2020-10-29 15:53:48 +01:00
Gary Verhaegen
5f8405331d
work around Azure/GitHub disagreement (#7790)
Azure used to report the status of the entire build to GitHub, which we
use as the "required check" for PRs to be merged. Ir doesn't do that
anymore which means we can't merge anything. It's unclear whether or not
that is a deliberate change.

This attempts to work around that by creating an extra job that depends
on all the other, which GitHub could depend on.

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-23 12:58:08 +02:00
Gary Verhaegen
84bf9c7aea
fix Slack message on compat_pr failure (#7775)
The message is built from the current commit message. Since this checks
out master, the Slack message ends up being a bit confusing.

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-22 13:21:03 +02:00
Gary Verhaegen
5945040810
ci/check_changelog: reduce race condition (#7734)
PR #7732 illustrated a bug in our current changelog check: it only worked
if the PR was up-to-date with `master` at the time of pushing. This was
not the intention. Fortunately, the fix is fairly easy: check out the PR
commit before running the check, rather than run it from the merge
commit (as Azure does by default).

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-19 18:25:59 +02:00
Gary Verhaegen
8c1fbf6225
ci/bash_lib: generalize save_gcp_data (#7599)
This PR extends the existing `save_gcp_data` function to handle any
`gsutil` command. This is done to support existence checking using
`gsutil ls` for private artifacts in release checking (`ci/cron`).

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-08 18:37:14 +02:00