Commit Graph

401 Commits

Author SHA1 Message Date
Gary Verhaegen
115d03fd7b
skip Oracle test on forks (#8930)
Fork builds do not have the required credentials.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-24 12:19:57 +00:00
Moritz Kiefer
8e8284c907
Retry on HttpException in release check (#8909)
ConnectionTimeout gets thrown as HttpException which seems very
sensible to retry.

changelog_begin
changelog_end
2021-02-22 17:37:08 +01:00
Moritz Kiefer
c309ef915c
Stream stdout of verify_signatures (#8911)
There is really no reason to first capture this to a String and then
putStrLn it. That only causes issues since if we crash with a non-zero
exitcode we won’t output anything.

changelog_begin
changelog_end
2021-02-22 16:03:25 +00:00
Moritz Kiefer
b47c1a974c
Stop pretending strings are booleans (#8885)
* Stop pretending strings are booleans

Sorry for all the mess here. I’m not capable of programming in yaml.

It turns out is_release is a string not a boolean and

```
and('false', eq('true', 'true'))
```

is true.

I hate everything about this.

changelog_begin
changelog_end

* Name bash step to reduce confusion

changelog_begin
changelog_end

* fix version in test

changelog_begin
changelog_end

* names can’t have spaces apparently

changelog_begin
changelog_end
2021-02-17 16:13:54 +00:00
Moritz Kiefer
2717e6a164
Fixup condition for running publish_mvn_npm (#8884)
* Fixup condition for running publish_mvn_npm

This needs to run for both linux and linux-scala-2.13

changelog_begin
changelog_end

* Update ci/build-unix.yml

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2021-02-17 15:52:33 +01:00
Moritz Kiefer
5843b193fd
Fixup scala 2.13 check (#8881)
* Fixup scala 2.13 check

Somehow I managed to misread the helpcheck and get confused by my
experiments and thought semver produces an exit code of 1,0,-1 but
actually it writes that to stdout.

changelog_begin
changelog_end

* Update ci/build.yml

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Update ci/build.yml

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2021-02-17 15:23:58 +01:00
Moritz Kiefer
ba6ba9019f
Release Scala 2.13 artifacts (#8858)
* Release Scala 2.13 artifacts

changelog_begin
changelog_end

* Dedup default scala version

changelog_begin
changelog_end
2021-02-17 14:32:04 +01:00
Moritz Kiefer
182eb825c8
Port Ledger API Test Tool to Scala 2.13 (#8875)
* Port Ledger API Test Tool to Scala 2.13

And with that we’re finally at //... building on Scala 2.13.

changelog_begin
changelog_end

* Fix build on 2.12

changelog_begin
changelog_end

* Fix kvutils export on 2.13

changelog_begin
changelog_end
2021-02-17 12:34:25 +01:00
Stephen Compall
b94b5f92f3
experimental Oracle support in json-api (#8596)
* separate OracleQueries from PostgresQueries

- with some changes from 8161e63189 courtesy @cocreature

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* abstract BIGINT

* json, signatories, observers columns

* compatible lastOffset

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* oracle functions for select (single template ID), insert

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* add oracle branch to integration tests

* oracle CLI configuration for json-api

* run integration tests with ojdbc in classpath

* update maven_install for ojdbc

* drop table if exists for Oracle

* make create DDLs and drops more planned out; drop in reverse order for Oracle integrity

* repin maven

* port agreement_text

* port (by removal) array part of ledger offset update

* use CASE instead of JSON map lookup for multiparty offset update

* simplify self types

* fix contract archival

* repin

* remove selectContracts in favor of selectContractsMultiTemplate

* move Oracle test execution to separate build target

* move websocket test to itlib

* make a bad array instance for Oracle

* report actually-available JDBC drivers only

* configure Oracle test from CI

* attempt with platforms and constraints

* a mismash of bazel to get it to conditionally enable oracle testing

* fix dep resolution in Scala 2.13

* make the Oracle test a stub (inits and does empty DB query)

* remove commented unused deps

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* repin

* we never supply a value for the surrogate ID columns

- suggested by @cocreature; thanks

* add not null to json in DB-specific place

- suggested by @cocreature; thanks

* why DBContractKey

- suggested by @cocreature; thanks

* textType isn't finalized

- suggested by @cocreature; thanks

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-02-17 03:50:35 -05:00
Moritz Kiefer
7ae28beff9
More Scala 2.13 cleanup (#8855)
This fixes Scaladoc and our pom file generation.

It also clears up the confusing error around gatling and removes a
redundant dependency on sbt (no idea why we had that in the first
place) both of which resulted in Scala 2.12 dependencies in our 2.13
lockfile which is obviously bad.

With this, we should now be ready to publish Scala 2.13 artifacts once
the ledger API test tool PR lands.

changelog_begin
changelog_end
2021-02-16 09:39:16 +00:00
Moritz Kiefer
f02b62a5e8
Port non-repudiation middleware to Scala 2.13 (#8853)
Tests are still missing and blocked on #8821.

The main change here is the switch from `ArraySeq[Byte]` to
`ArraySeq.ofByte`. `ArraySeq` allows for boxed and unboxed
representaitons. That means that `ArraySeq[Byte]unsafeArray` does not always return
an Array[Byte] (boxed version would be Array[AnyRef]).

Apparently collection-compat has taken the yolo approach and pretends
it can give you an Array[Byte] anyway 🤷 Scala 2.13 on the other
hand, does things properly in this regard which means the code relying
on `unsafeArray` fails to compile.

`ArraySeq.ofByte` is the specialized unboxed version where none of
this is an issue on both 2.13 and 2.12.

changelog_begin
changelog_end
2021-02-16 08:30:34 +01:00
Moritz Kiefer
777749539c
Draw the rest of the Scala 2.13 owl (#8852)
* Draw the rest of the Scala 2.13 owl

Not quite but pretty close and this switches us over from inclusions
to exclusions which makes it much easier to track.

Ledger API test tool should be fixed by #8821. Non-repudiation needs a
tiny bit of work since unwrapArray doesn’t work the same on 2.13 but
shouldn’t be hard to fix.

changelog_begin
changelog_end

* Fix ScriptService tests

Those tests were all dumb. They asserted on a fixed order while the
function to sort the things was broken so we ended up with the random
Map order which is unsurprisingly not the same.

This is easily fixed by fixing the sort function.

There is also a second issue with query not sorting.

changelog_begin
changelog_end

* Turns out if you fix one test the next one breaks

And clearly nobody ever tested this or give this a second thought.

changelog_begin
changelog_end
2021-02-15 19:20:24 +00:00
Moritz Kiefer
4d55b58f19
Port extractor to Scala 2.13 (#8845)
* Port extractor to Scala 2.13

changelog_begin
changelog_end

* Less silly view transforms

changelog_begin
changelog_end
2021-02-15 16:52:06 +01:00
Moritz Kiefer
91b185dcc3
Port Navigator to Scala 2.13 (#8847)
changelog_begin
changelog_end
2021-02-15 14:55:36 +00:00
Moritz Kiefer
9cd3909c31
Port the rest of //language-support/... to Scala 2.13 (#8846)
changelog_begin
changelog_end
2021-02-15 15:08:00 +01:00
Moritz Kiefer
f02e0fe42b
Generate Scala code compatible with Scala 2.13 (#8841)
fixes #8498

This fixes the error in 2.13 wtr to the location change of Predef. It
doesn’t yet address the warning wtr to the import of higherKinds. For
now, our build ignores that warning. Trying to figure out if we can
get away with a breaking change here or if we need to hide that change
behind a flag but either way, no need to block fixing the actual error
on that.

changelog_begin
changelog_end
2021-02-15 09:06:40 +01:00
Gary Verhaegen
df0086d26f
ci/linux: kill machines if they fail to clean up (#8835)
It does not seem like CI machines recover from a failed clean-up. This
is not the most elegant solution possible, but it's a cheap one that
should work.

Not: shutting down the machine in the middle of the build will not
provide an error message to Slack for main branch builds (because the
`tell_slack_failed` step would need to run on the same machine) but will
correctly report failure for PRs (that was the original purpose of the
`collect_build_data` step).

An alternative here would be to give a delay to the shutdown command,
and try to calibrate it so that it's long enough for this job to
correctly report its failure to both Azure and Slack, while making it
short enough that no other job gets assigned to the machine. I'm not
clear enough on how often Azure assigns jobs to try and bet on that.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-12 19:32:14 +01:00
Moritz Kiefer
153a23d315
Port the rest of //ledger-service/... to 2.13 (#8836)
changelog_begin
changelog_end
2021-02-12 17:26:19 +01:00
Moritz Kiefer
2558843bc5
Port //daml-script/... to Scala 2.13 (#8831)
* Port //daml-script/... to Scala 2.13

changelog_begin
changelog_end

* fmt

changelog_begin
changelog_end
2021-02-12 13:01:58 +00:00
Moritz Kiefer
c8b82f8e7a
Port HTTP JSON API binary to 2.13 (#8824)
Tests are still missing. Simply haven’t found the time yet, I’ll
tackle those separately.

changelog_begin
changelog_end
2021-02-12 09:54:47 +01:00
Moritz Kiefer
9dc3d2ace4
Upgrade to Bazel 4.0 (#8798)
* Upgrade to Bazel 4.0

changelog_begin
changelog_end

* Go back to default assignment

changelog_begin
changelog_end
2021-02-11 10:08:47 +01:00
Gary Verhaegen
eb14dcf6e9
bump check_releases timeout (#8812)
It's become quite flaky at just 4h.

Yes, I do plan to actually fix it, but it may take a while.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-10 22:45:25 +00:00
Brian Healey
dd73af06fb
replace deprecated blackduck property (#8811)
CHANGELOG_BEGIN
CHANGELOG_END
2021-02-10 19:54:31 +00:00
Gary Verhaegen
95c2184dcc
bump disk cleanup threashold (#8807)
I've seen [a build] failing with "disk full" after starting with 41GB
free.

[a build]: https://dev.azure.com/digitalasset/daml/_build/results?buildId=69270&view=logs&j=870bb40c-6da0-5bff-67ed-547f10fa97f2&t=deecee86-545a-596e-8b0d-fb7d606fe9f2

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-10 13:39:29 +00:00
Gary Verhaegen
8c45fd6afd
Revert "Disable MacOS CI jobs" (#8806)
This closes INCIDENT-126. Thanks @nycnewman!

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-10 12:59:03 +01:00
Moritz Kiefer
f14a8c14dd
Disable MacOS CI jobs (#8803)
* Disable MacOS CI jobs

5/6 macos nodes are down and we cannot fix it quickly, so to unblock
everyone, let’s disable those jobs for now.

I deliberately did not remove MacOS from releases. Those really should run on MacOS.

changelog_begin
changelog_end

* Undo unnecessary changes

changelog_begin
changelog_end

* Allow skipping macos jobs

changelog_begin
changelog_end
2021-02-10 10:29:21 +01:00
Moritz Kiefer
40e1a39f2c
Port //triggers/... to Scala 2.13 (#8781)
changelog_begin
changelog_end
2021-02-09 13:35:05 +01:00
Gary Verhaegen
cd33c2015c
ci: use setvar to set variables (#8664)
My goal here is to investigate the new warning Azure has been showing
for the past few days:

> ##[warning]%25 detected in ##vso command. In March 2021, the agent command parser will be updated to unescape this to %. To opt out of this behavior, set a job level variable DECODE_PERCENTS to false. Setting to true will force this behavior immediately. More information can be found at https://github.com/microsoft/azure-pipelines-agent/blob/master/docs/design/percentEncoding.md

As far as I'm aware we are not deliberately passing in any `%25` in any
of our `vso` commands, so I was a bit surprised by this.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-09 11:42:34 +01:00
Moritz Kiefer
beb8a2a9ba
Port parts of //ledger-service to Scala 2.13 (#8776)
Unfortunately missing the actual interesting part since porting
`partitionBimap` seems to be rather annoying but this at least gets us
started on the easy parts.

changelog_begin
changelog_end
2021-02-08 11:27:50 +00:00
Moritz Kiefer
773ddada04
Upgrade nixpkgs (#8190)
* Upgrade nixpkgs

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* No ibazel

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* Switch agent pool

changelog_begin
changelog_end

* .

* .

changelog_begin
changelog_end
2021-02-08 11:12:07 +00:00
Moritz Kiefer
6c29f95c61
Add a CI job for testing against Oracle DB (#8763)
* Add a CI job for testing against Oracle DB

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* Cleanup

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* debug

changelog_begin
changelog_end

* no debugging

changelog_begin
changelog_end
2021-02-05 17:36:56 +01:00
Moritz Kiefer
cf3d0876af
Add a prototype for DAML Script dumps (#7934)
* Add a prototype for DAML Script dumps

This is still fairly rough unfortunately but it does at least have
some tests and it doesn’t interact with anything else, so hopefully we
can land this and then parallelize the work from there on.

changelog_begin
changelog_end

* Update daml-script/dump/src/main/scala/com/daml/script/dump/Encode.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* view all the things

changelog_begin
changelog_end

* Update daml-script/dump/src/main/scala/com/daml/script/dump/Dependencies.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Fixup the switch to exists

changelog_begin
changelog_end

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2021-02-05 13:19:20 +01:00
Gary Verhaegen
ab5f62abac
ci/clean-up: s/lsof/pgrep (#8748)
CHANGELOG_BEGIN
CHANGELOG_END
2021-02-04 12:21:55 +01:00
Gary Verhaegen
5734730d50
tweak local cache cleanup (#8738)
For various reasons, my attempts at improving the cache cleanup process
have been delayed. There are, however, two simple, non-controversial
changes I can "backport" without having to wait for consensus on the
whole thing:

1. Increase the threshold. At least for the compat jobs, we have seen
   builds failing after starting with ~32GB free.
2. Kill dangling Bazel processes, which keep some files open and
   sometimes cause the clean-up process to crash.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-03 19:47:52 +01:00
Gary Verhaegen
29c2577fa2
fix perf_speedy report (escaping ") (#8655)
The report currently chokes on quotes in the commit message (see
550aa48fc5). Rather than trying to
correctly excape things in Bash, this PR delegates the quote handling to
jq, because having to deal with Bash embedded in YAML is hard enough.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-03 19:47:27 +01:00
Moritz Kiefer
d7d954335e
Retry asset downloads in check_releases (#8730)
We’ve seen a number of "resource vanished (connection reset by peer)"
errors. Slapping some retries on that should hopefully make the CI job
a bit more robust.

changelog_begin
changelog_end
2021-02-03 12:04:59 +01:00
Moritz Kiefer
c89e00342d
Clean broken entries from the Bazel cache (#8668)
* Clean broken entries from the Bazel cache

This is hopefully a somewhat reasonable workaround for the "output not
created" errors that keep annoying us.

For now, this is just part of the hourly cronjob but we could move it
somewhere else if desired.

changelog_begin
changelog_end

* Fix GCS credentials

changelog_begin
changelog_end
2021-01-28 17:57:09 +00:00
Gary Verhaegen
fef712bf60
Upgrade linux nodes to 20.04 (#8617)
CHANGELOG_BEGIN

- Our Linux binaries are now built on Ubuntu 20.04 instead of 16.04. We
  do not expect any user-level impact, but please reach out if you
  do notice any issue that might be caused by this.

CHANGELOG_END
2021-01-27 17:38:34 +01:00
Gary Verhaegen
fc83f7088d
limit blackduck scans to main (#8647)
CHANGELOG_BEGIN
CHANGELOG_END
2021-01-27 17:29:27 +01:00
Moritz Kiefer
e0c5abd647
Switch to GHC 8.10.3 (#8394)
* Switch to GHC 8.10.3

changelog_begin
changelog_end

* Update bazel-haskell-deps.bzl

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

* Comment on rules_haskell patch

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2021-01-25 11:53:53 +00:00
Andreas Herrmann
5693394650
Publish JARs for trigger service and OAuth 2.0 middleware (#8614)
* include oauth2 logback config in release tarball

overlooked in https://github.com/digital-asset/daml/pull/8611

* Release trigger-service and oauth2-middleware JARs

changelog_begin
changelog_end

* drop from artifacts.yaml

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-01-25 09:14:13 +00:00
Gary Verhaegen
532f996f12
ci: clean-up hard drive more often (#8582)
CHANGELOG_BEGIN
CHANGELOG_END
2021-01-20 17:22:53 +01:00
Gary Verhaegen
01f11109fa
faster disk cleanup check (#8565)
I can't reproduce the `chmod` failure and have no clue how to fix it, so
for now let's just hope it's not too frequent.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-19 19:50:44 +00:00
Gary Verhaegen
3641751571 ci/unix: better cache size reporting (#8556)
Current reports look like:

```
Disk cache small enough:\n20G/home/vsts/.bazel-cache
```

because `echo` does not convert `\n`. An alternative would be to replace
`echo` with `printf`, but I have had enough issues with
subshells-in-strings lately that I prefer just avoiding them when
possible.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-19 18:29:07 +01:00
Gary Verhaegen
8350224aef
ci: fix linux cleanup for spaces (#8531)
I hate spaces.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-15 17:01:44 +00:00
Gary Verhaegen
36175c0a77
ci/cron: remove hidden.json (#8525)
I discovered yesterday that the `snapshots.json` (and actually also the
`versions.json`) file is no longer purely internal to the docs process,
as it was meant to be, but is now depended upon by the assistant. This
means the renaming from `snapshots.json` to `hidden.json` cannot happen,
and we reverted that yesterday in #8513 (& #8514), though that was done
in a bit of a hurry. This PR aims at cleaning up the resulting mess and
achieve a better long-term end state.

I will be manually removing the `hidden.json` file as soon as this is
merged, so that nothing ends up depending on _that_. There is no
occurrence of `hidden.json` outside this docs cron so hopefully this
works out.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-15 16:04:54 +00:00
Gary Verhaegen
cf949b6bae
ci: gc linux cache (#8527)
This is the equivalent of #8515 for Linux. There was some concern that
`bazel` would be upset at having that cache removed, so I spent a fair
amount of time trying to break it (on a Linux VM, as for some reason
`bazel` chooses not to use `~/.cache` on macOS). I could not make
`bazel` unhappy by deleting the whole thing. Deleting random files,
however, did end up producing error messages along the lines of:

```
$ bazel build //...
FATAL: corrupt installation: file '/home/vagrant/.cache/bazel/_bazel_vagrant/install/73d06d52dbf3a8e6ed43f5bf5f115eb0/embedded_tools/src/BUILD' is missing or modified.  Please remove '/home/vagrant/.cache/bazel/_bazel_vagrant/install/73d06d52dbf3a8e6ed43f5bf5f115eb0' and try again.
```

which suggest busting the entire thing as a solution, so I think we're
safe here.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-15 16:29:52 +01:00
Gary Verhaegen
96e9dde360
ci: clean-up disk cache if it gets too big (#8515)
Hopefully this works around our recent CI disk space issues, while 80GB
should be large enough that it only happens once per machine per day, so
perf shouldn't be impacted too much.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-14 19:41:07 +00:00
Gary Verhaegen
7f076bcd56
docs cron: update meta files unconditionally (#8514)
CHANGELOG_BEGIN
CHANGELOG_END
2021-01-14 20:04:48 +01:00
Moritz Kiefer
dfc48a0010
Update both hidden.json and snapshots.json (#8513)
* Update both hidden.json and snapshots.json

The assistant relies on the latter, our docs cronjob on the former. I
have no idea why we have two but keeping them in sync should be fine.

changelog_begin
changelog_end

* maybe I should test if my code compiles before pushing

changelog_begin
changelog_end
2021-01-14 19:07:35 +01:00
Moritz Kiefer
cdb2c2d0ba
Port the rest of //ledger/... to Scala 2.13 (#8497)
* Port the rest of //ledger/... to Scala 2.13

draw the rest of the fcking owl

Omitted for now are the ledger API test tool which has a dependency
only compatible with 2.12 and the generated code of the Scala
codegen (the codegen compiles and runs with 2.13, the generated code
does not).

changelog_begin
changelog_end

* Less symbols

changelog_begin
changelog_end
2021-01-13 22:23:31 +01:00
Moritz Kiefer
84a9488077
Port more of //ledger/... to Scala 2.13 (#8488)
* Port more of //ledger/... to Scala 2.13

changelog_begin
changelog_end

* Remove unusued dependency

changelog_begin
changelog_end

* Rename bf to factory to reflect the fact that it’s now a Factory

changelog_begin
changelog_end

* Use regex match instead of sliding string equalityt

changelog_begin
changelog_end

* regex matches are bad

changelog_begin
changelog_end
2021-01-13 16:30:43 +01:00
Gary Verhaegen
742feb58be
trigger PRs job on generated PRs (#8489)
CHANGELOG_BEGIN
CHANGELOG_END
2021-01-13 11:30:17 +00:00
Moritz Kiefer
290584a378
Port //ledger/ledger-api-client/... to Scala 2.13 (#8478)
* Port //ledger/ledger-api-client/... to Scala 2.13

This pulls in Sandbox next and kvutils as a dependency so those now
build on 2.13 as well.

changelog_begin
changelog_end

* Upgrade scala-colllection-compat

changelog_begin
changelog_end

* Use toVector.sortBy instead of to(LazyList).sortBy

changelog_begin
changelog_end

* Use a view for passing things to varargs

changelog_begin
changelog_end

* avoid symbol literal in CommandClientIT

changelog_begin
changelog_end
2021-01-13 11:35:46 +01:00
Gary Verhaegen
28d9cea05e
remove jfrog-cli (#8477)
As far as I can figure out, this has been unused since #5422.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-12 15:01:01 +00:00
Gary Verhaegen
ce96802a6d
only update NOTICES from main daily compats (#8473)
See #8468 for failure mode of current config.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-12 12:10:27 +00:00
Moritz Kiefer
6b2ac0e0c6
Port parts of //ledger/... to Scala 2.13 (#8470)
* Port parts of //ledger/... to Scala 2.13

Fairly random choice of directories, I just went through them in
alphabetical order. The one thing that I had to disable for now are
the conformance tests since the ledger API test tool has a dependency
not compatible with Scala 2.13.

changelog_begin
changelog_end

* Remove accidentally included //ledger/ledger-api-client/...

doesn’t actually work yet

changelog_begin
changelog_end
2021-01-12 10:55:00 +01:00
Moritz Kiefer
1abde753c9
Build and test //ledger-api/... against Scala 2.13 (#8452)
changelog_begin
changelog_end
2021-01-11 16:53:54 +01:00
Gary Verhaegen
7ca49bacc7
publish docs bundle as pipeline artifact (#8454)
This make the docs bundle available as a download from any build on
Azure. I mostly thought of this as a workaround for @bame-da because of
the Big Sur thing, but I figure that may occasionally useful to other
people too.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-11 13:30:51 +00:00
Moritz Kiefer
b83c4e945d
Bump perf test for scalafmt update (#8444)
changelog_begin
changelog_end
2021-01-09 13:12:54 +01:00
Moritz Kiefer
3a8f5e8aaa
Port //language-support/scala to Scala 2.13 (#8442)
The one thing that is still missing is making the generated Scala code
from the codegen compatible with Scala 2.13 so the examples are
excluded for now.

changelog_begin
changelog_end
2021-01-08 20:28:15 +01:00
Bernhard Elsner
cda93db944
Daml case and logo (#8433)
* Replace many occurrences of DAML with Daml

* Update docs logo

* A few more CLI occurrences

CHANGELOG_BEGIN
- Change DAML capitalization and docs logo
CHANGELOG_END

* Fix some over-eager replacements

* A few mor occurrences in md files

* Address comments in *.proto files

* Change case in comments and strings in .ts files

* Revert changes to frozen proto files

* Also revert LF 1.11

* Update get-daml.sh

* Update windows installer

* Include .py files

* Include comments in .daml files

* More instances in the assistant CLI

* some more help texts
2021-01-08 12:50:15 +00:00
Moritz Kiefer
0af4f522c7
Port the rest //daml-lf/... to Scala 2.13 (#8436)
* Port the rest //daml-lf/... to Scala 2.13

Draw the rest of the owl

changelog_begin
changelog_end

* Update daml-lf/encoder/src/main/scala/com/digitalasset/daml/lf/archive/testing/DamlLfEncoder.scala

Co-authored-by: Remy <remy.haemmerle@daml.com>

Co-authored-by: Remy <remy.haemmerle@daml.com>
2021-01-08 10:38:17 +01:00
Moritz Kiefer
424faa923a
Port damlc dependencies to Scala 2.13 (#8423)
* Port damlc dependencies to Scala 2.13

I got a bit fed up by the fact that going directory by directory
didn’t really work since there are two many interdependencies in
tests (e.g., client tests depend on sandbox, sandbox tests depend on
clients, engine tests depend on DARs which depend on damlc, …).

So before attempting to continue with the per-directory process, this
is a bruteforce approach to break a lot of those cycles by porting all
dependencies of damlc which includes client bindings (for DAML Script)
and Sandbox Classic (also for DAML Script).

If this is too annoying to review let me know and I’ll try to split it
up into a few chunks.

changelog_begin
changelog_end

* Update daml-lf/data/src/main/2.13/com/daml/lf/data/LawlessTraversals.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* fixup lawlesstraversal

changelog_begin
changelog_end

* less iterator more view

changelog_begin
changelog_end

* document safety of unsafeWrapArray

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2021-01-08 07:22:38 +01:00
Moritz Kiefer
0549d07bdc
Port //daml-lf/interpreter to Scala 2.13 (#8421)
* Port //daml-lf/interpreter to Scala 2.13

For now the perf tests are left out since they depend on a DAR built
by damlc which depends on daml script which depends on the world
:exploding-head:

changelog_begin
changelog_end

* Scala 2.13-style to for ImmArray and FrontStack

changelog_begin
changelog_end

* Avoid extra conversion

changelog_begin
changelog_end
2021-01-07 10:13:08 +00:00
Gary Verhaegen
7f7e4f9fdc
fix docs cron (#8412)
Typo introduced in #8384.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-06 13:25:52 +00:00
Moritz Kiefer
02d8e8fe95
Port //daml-lf/(parser|validation) to Scala 2.13 (#8391)
* Port //daml-lf/(parser|validation) to Scala 2.13

changelog_begin
changelog_end

* Rename (Expr|Type)Traversable to (Expr|Type)Iterable

changelog_begin
changelog_end
2021-01-05 13:11:27 +01:00
Gary Verhaegen
f45a065c54
ci: pin down Ubuntu versions (#8388)
For a couple weeks now there has been a warning on the Azure Pipelines
web UI that says `ubuntu-latest` is in the process of switching from
18.04 to 20.04. I am not aware of any specific issue this would cause
for our particular workflows, but I don't like my dependencies changing
from under me.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-05 10:39:59 +01:00
Gary Verhaegen
7edc57712d
separate CI config for main & prs (#8374)
This is another take on #8276, with the same underlying motivation.
However, this approach is mostly duplication-free, which seems better,
especially given the already-pretty-sorry state of our CI config.

Like #8276, this is done in 2 commits for ease of review. The first
commit is wholly unintresting and just copies `azure-pipelines.yml` to
both `ci/prs.yml` and `ci/build.yml`; the second commit removes from
each part what it shouldn't have. The intention is for `ci/build.yml` to
have all of the common parts.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-04 20:45:30 +01:00
Gary Verhaegen
eb9aba680e
ci/cron: remove temp hack (#8385)
CHANGELOG_BEGIN
CHANGELOG_END
2021-01-04 19:06:30 +00:00
Gary Verhaegen
b506393dc8
ci/cron: use proc instead of shell (#8384)
As suggested by @cocreature in #8382.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-04 19:47:13 +01:00
Gary Verhaegen
2c497a3e75
fix typo in docs cron (#8382)
Introduced in #8372.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-04 18:32:57 +01:00
Gary Verhaegen
2083f74cf2
add docs.daml.com/latest (#8372)
This commit changes the docs cron to create a new file
`docs.daml.com/latest`, a simple text file containing the version number
of the latest released version. This is done in response to #8354, to
avoid having to replicate the logic for which version is the latest
across this, the assistant and the get-daml.sh script.

This commit also cleans up a small transitional FIXME in the docs cron
regarding the transition from `snapshots.json` to `hidden.json`.

For the solution to #8354 to be complete, we'll also need to update
get-daml.sh and the assistant to use the new latest file. Note that this
file would only be published on the next stable release, so this commit
also includes a temporary hack to re-generate it (and `versions.json`
and `hidden.json`) unconditionally on every run; this can be removed as
soon as this has run once.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-04 15:14:42 +01:00
Gary Verhaegen
b45e37991f
update test sha for copyright update (#8361)
CHANGELOG_BEGIN
CHANGELOG_END
2021-01-01 22:16:37 +00:00
Gary Verhaegen
a925f0174c
update copyright notices for 2021 (#8257)
* update copyright notices for 2021

To be merged on 2021-01-01.

CHANGELOG_BEGIN
CHANGELOG_END

* patch-bazel-windows & da-ghc-lib
2021-01-01 19:49:51 +01:00
Gary Verhaegen
93f449d245
rename master to main (#8245)
As we strive for more inclusiveness, we are becoming less comfortable
with historically-charged terms being used in our everyday work.

This is targeted for merge on Dec 26, _after_ the necessary
corresponding changes at both the GitHub and Azure Pipelines levels.

CHANGELOG_BEGIN

- DAML Connect development is now conducted from the `main` branch,
  rather than the `master` one. If you had any dependency on the
  digital-asset/daml repository, you will need to update this parameter.

CHANGELOG_END
2020-12-27 14:19:07 +01:00
Moritz Kiefer
f3e481525c
Fix compare.sh again (#8346)
Turns out bash is hard and I’m stupid :sadpanda:

We need to write thoutput to stderr, otherwise this ends up in the
JSON output which obviously is not valid JSON.

changelog_begin
changelog_end
2020-12-18 20:59:38 +00:00
Moritz Kiefer
0391b867ee
Fix compare.sh in perf test (#8344)
We changed the patch to target more than one file which made the
checkout insufficient for restoring the state and then the following
git checkout of current fails with:

```
error: Your local changes to the following files would be overwritten by checkout:
        stack-snapshot.yaml
```

A git reset --hard should make sure everything gets reset.

changelog_begin
changelog_end
2020-12-18 20:59:22 +01:00
nickchapman-da
a7891a32b5
Avoid duplicating template let bindings (#8259)
* update ghc-lib patch hash

* update stack-snapshot.yaml

* update ghc-lib patch again

changelog_begin
changelog_end

* update stack-snapshot.yaml

* bazel run @stackage-unpinned//:pin (on linux)

* update ghc patch -- no singleton tuples!

* update stack-snapshot.yaml

* update stackage_snapshot.json

* expect error just once, not repeated 5 times

* remove 4x: --ghc-option -Werror

* ghc-lib patch

* stackage_snapshot.json

* stackage_snapshot.json (really, last commit was the yaml)

* Revert "remove 4x: --ghc-option -Werror"

This reverts commit 34b65fc7b1.

* ci/da-ghc-lib/compile.yml

* stack-snapshot.yaml

* stackage_snapshot.json

* stack pin on windows
2020-12-18 14:33:28 +00:00
Gary Verhaegen
e51838efff
fail on failed perf check (#8334)
Today the [perf check failed], but we got no notification of it. I'm not
sure what's happening as I can't reproduce any of it locally: not only
does the `bazel run` command work for me (despite the ghc-lib URL
returning a 404 when I try it manually), I also can't reproduce the fact
that Bash, on CI, doesn't seem to fail on either the `bazel run` error
or the fact that on the next line `cat` tries to access a file that
doesn't exist (for which CI does print the error message).

This PR does two things:

- Add an explicit check that _should_ get Bash to actually fail should
  this happen again in the future. It is not a great fix but at least
  we'll know if it happens again (to the best of my knowledge today was
  the first time we hit this).
- Amend the existing patch we apply on the baseline commit to use the
  GCS-hosted ghc-lib packages.

CHANGELOG_BEGIN
CHANGELOG_END

[perf check failed]: https://dev.azure.com/digitalasset/daml/_build/results?buildId=64395&view=results
2020-12-18 13:42:54 +01:00
Gary Verhaegen
604787eb07
fix Slack notifications (#8327)
CHANGELOG_BEGIN
CHANGELOG_END
2020-12-17 14:32:49 +01:00
Moritz Kiefer
b32789025e
Reset Windows cache (#8326)
changelog_begin
changelog_end
2020-12-17 12:24:07 +00:00
Moritz Kiefer
fbf1820f94
Fix slack escaping (#8311)
The previous escaping syntax doesn’t seem to work (after the first
layer you no longer have a variable) and has broken our build
reporting completely.

changelog_begin
changelog_end
2020-12-16 11:18:41 +00:00
Gary Verhaegen
e28699dbb0
fix Slack messages (#8292)
When including commit titles in Slack messages, care has to be taken to
escape what Slack considers [control characters], namely `&`, `>` & `<`.

[control characters]: https://api.slack.com/reference/surfaces/formatting#escaping

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-15 18:11:11 +01:00
Remy
3783a158ff
Bump perf test (#8269)
Needs bumping due to the API changes for GenNode (#8217)

changelog_begin
changelog_end
2020-12-11 17:37:44 +00:00
Brian Healey
6a1e0a633b
Avoid haskell and jvm bazel blackduck scan stomping (#8247)
* Run bazel scan for all but haskell first, then haskell at end so they do not stomp on each other

CHANGELOG_BEGIN
CHANGELOG_END

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* use common prefix between runs so results are aggregated, get branch name from running job rather than assuming master

* DO NOT MERGE: disable all jobs except for blackduck scan

* haskell before full scan

* parens instead of braces

* differentiate haskell and jvm bazel runs with unique code location

* unique code location prefix

* fix syntax

* unique code location to avoid clashing bazel runs

* Use master security-blackduck script helper

* reenable all jobs to make mergeable

* cleanup whitespace

* Use Build.SourceBranchName for branch

* Update ci/cron/daily-compat.yml

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* DO NOT MERGE: skip all jobs but blackduck, skip update notices file step

* reenable all jobs to make mergeable

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2020-12-11 07:56:05 -05:00
Moritz Kiefer
75d28d1242
Bump perf test (#8256)
Needs bumping due to the minor API changes for multi-party submissions

changelog_begin
changelog_end
2020-12-11 12:29:49 +00:00
Gary Verhaegen
917ffe8c8b
daily ci: remove debug output (#8255)
CHANGELOG_BEGIN
CHANGELOG_END
2020-12-11 12:09:29 +00:00
Gary Verhaegen
029c655adc
blackduck: open PR on NOTICES file change (#8215)
CHANGELOG_BEGIN
CHANGELOG_END
2020-12-10 10:08:28 +01:00
Moritz Kiefer
ec0fcb39f1
Fix docs cron again (#8208)
Unfortunately, I missed the fact that we had our own logic for
handling process failures which resulted in uncatchable
exceptions. I’ve changed one place to use the upstream handling and
the other to call `fail` which throws an IOException like I would have
expected.

changelog_begin
changelog_end
2020-12-08 22:06:22 +01:00
Moritz Kiefer
0c7791bc1c
Fix docs cron (#8207)
The change in #8191 to publish daml on sql docs failed because the
versions.json and snapshots.json files don’t exist initially. This PR
fixes that by catching the exception and treating it as an empty file.

changelog_begin
changelog_end
2020-12-08 21:32:33 +01:00
Moritz Kiefer
880d290285
Publish DAML on SQL Docs (#8191)
changelog_begin
changelog_end
2020-12-08 13:49:37 +01:00
Brian Healey
ca294eb14d
add blackduck scan to run on master (#6130) (#8161)
* add blackduck scan to run on master (#6130)

* add blackduck scan

* disable go scanning
exclude entire language-support/ts directory for node scanning
break to multiple lines to make command line params easier to parse

* Increase timeout for blackduck binary scan

* update blackduck scan config

* remove some exclusions, force python3

* exclude GO until path to go executable can be resolved

* added readme explanation of why we want this file

* fail in case of policy violation

* ensure haskell bazel scan completes before running second round scan for bazel jvm and node and other langs

* trigger notices file gen to ensure BOM complete

* remove trailing end of lines

* run with latest detect version and unique code location name changes to wrapper script

* Add blackduck to daily compat job

* DO NOT MERGE: condition false to disable other jobs for testing

* remove parameters not available to cronjob

* Revert changes to regular CI pipeline

CHANGELOG_BEGIN
CHANGELOG_END

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* Do not get branch name from variable

* Upgrade com.fasterxml.jackson.core:jackson-databind to 2.12.0 to address security vulnerability

* Remove disabling of other jobs, set to branch to be used on prod runs

* Apply suggestions from code review

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Address code review comments

* Updated NOTICES file

* Run bazel build, update NOTICES file

* Correct dade-assist

* do not have perms to pipe to dev/null

* Add md file explaining how to update NOTICES file

* Add instructions for running blackduck locally

* Add a link to full security-blackduck readme

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2020-12-07 19:59:39 +00:00
Gary Verhaegen
46b7decbca
notify_user: consider changelog entries (#8166)
The current implementation of the notify_user job sometimes reports
success while the build has actually failed. Azure does not provide a
way to query the overall state of the current build, so a general
solution to this problem does not seem possible (see #6796 for an
example of a failed attempt). However, all reported cases were
specifically about the `check_changelog_entry` job, which we can easily
query for, so this PR does that.

Note: originally pushed without a changelog entry to test new
notification mechanism.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-04 12:16:34 +00:00
nickchapman-da
0867605e1a
Don't loose signatures in template let binding (#8136)
* update ghc-lib patch

* update stack-snapshot.yaml

changelog_begin
changelog_end

* bazel run @stackage-unpinned//:pin (on linux) and update stackage_snapshot.json

* testcase to check type signatures in template-lets are not ignored

* update to ghc-lib commit hash following merge

* update snapshot

* update snapshot after pin on windows

* update stack pin on linux

Co-authored-by: Nick Chapman <nchapman@digitalasset.com>
2020-12-03 15:08:02 +00:00
Gary Verhaegen
485dd1b597
perf_json_http: compress results (#8123)
I wanted to suggest that on the PR but caught it after it was merged. So
I made a note of it, which promptly got lost. As the end of the year
approaches I've started trying to clean up a little.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-01 17:44:43 +01:00
Gary Verhaegen
3cef53135c
ci/cron: do not push artifacts to gcs bucket (#8067)
Having the cron push artifacts to GCP was really only meant to happen
once. I got distracted and worked on other things. This PR closes that
work loop such that the current state and expectations are:

- Every new release pushes to GCP as part of the release process.
- The cron only checks that the GCP backup exists and matches, but does
  not push if it doesn't.

The reason for this is we want the cron job to fail if there are
additional, unexpected files in a release, rather than automatically
commit those files for the long term.

CHANGELOG_BEGIN
CHANGELOG_END
2020-11-25 19:12:03 +01:00
Gary Verhaegen
b23304c691
add default capability to macos (#5915)
This is the macOS part of #5912, which I have separated because our
macOS nodes have a different deployment process so it seemed easier to
track the deployment of the change separately.

CHANGELOG_BEGIN
CHANGELOG_END
2020-11-25 15:34:33 +01:00
nickchapman-da
6a745ed1fa
Support choice observers in 1.dev (#7922)
* Adapt to new desugaring for choice observers.

update hash of ghc patch.

changelog_begin
changelog_end

update ghc patch to final version

update stack-snapshot hashes for ghc-lib(-parser)

update stackage_snapshot.json, following `bazel run @stackage-unpinned//:pin`

expose Optional constructors for desugared code to use

adapt LFConversion to expect a 4-tuple for a desugared choice def/sig

update LFConversion for choice-observers

first example using new choice observer syntax.

fix scala type checker to have correct scoping rules for choice-observers

remove comment from example which says it is broken

improve variable names

improve tests for choice-observer clause

only test choice-observers SINCE 1.dev

add jq queries for choice observeres

make positive statement in jq test which checks choice obserers are present

test behaviour of choice observers

squash me

typo

* test more choice-observer divulgence

* Update documention for choice observers.

changelog_begin
Support choice observers in 1.dev
changelog_end

* fix docs build

* fix daml docs choice-observers example

* address comments: rewording text

* annotate choices observers as early-access in documention

* split out documentation code-snippets which require --target=1.dev

* final tweaks to documentation text
2020-11-18 19:51:15 +00:00
Gary Verhaegen
d4b6b06923
update docker image description (#7915)
Note: this file is meant to represent the content on [Docker Hub], but
syncing is currently a manual process. I will propagate once this is
approved and merged.

[Docker Hub]: https://hub.docker.com/repository/docker/digitalasset/daml-sdk

CHANGELOG_BEGIN
CHANGELOG_END
2020-11-06 16:39:16 +01:00