Commit Graph

489 Commits

Author SHA1 Message Date
Moritz Kiefer
cf445b2e73
Skip platform independence tests on release PRs (#11631)
* Skip platform independence tests on release PRs

changelog_begin
changelog_end

* yaml that shit

changelog_begin
changelog_end
2021-11-10 12:55:33 +00:00
Moisés Ackerman
cfd521ff86
Fix GHC build errors on macOS (#11560)
changelog_begin
changelog_end
2021-11-09 14:42:52 +01:00
Robin Krom
af1bee7b82
precondition lf conversion (#11538)
* interfaces: lf conversion of preconditions

This adds the LF conversion of preconditions from the desugared GHC code
to Daml LF for preconditions. This depends on the corresponding patch of
the GHC fork.

CHANGELOG_BEGIN
CHANGELOG_END

* pin linux stackage snapshot

* added a test

CHANGELOG_BEGIN
CHANGELOG_END

* pinned linux stackage snapshot

CHANGELOG_BEGIN
CHANGELOG_END

* additional InterfacePrecondition test

this checks that that conjunction of all preconditions of intererfaces
and templates is checked.

CHANGELOG_BEGIN
CHANGELOG_END

* pin stackage snapshot

* pin windows stackage snapshot

* update compile.yml (again)

* pin linux stack snapshot

* fix InterfacePrecondition test

* pin windows stackage
2021-11-09 11:43:46 +00:00
Moisés Ackerman
7d68e05f7f
Remove virtual choices (#11482)
* Remove virtual choices

* Remove choices without a body in 'interface' definition
* Remove choices in 'template ... implements' section

part of #11372

changelog_begin
changelog_end

* Remove virtual choices cont.

Switch uses of virtual choices to fixed choice with method implementation

* update snapshot after pin on windows

* Disable failing interface tests with TODO #10810
2021-11-08 17:05:23 +01:00
Hubert Slojewski
9e045c105b
Remove a leftover from check-protobuf-stability.sh (#11513)
CHANGELOG_BEGIN
CHANGELOG_END
2021-11-02 17:11:17 +00:00
Gary Verhaegen
79505b5474
oracle debug (#11386)
Collect Oracle logs for further debugging.

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-01 12:19:01 +01:00
Robert Autenrieth
927b2378c3
Make Oracle tests exclusive (#11408)
* Make Oracle tests exclusive

This attempts to solve connection timeouts we often see in tests.

changelog_begin
changelog_end

* Revert "Make Oracle tests exclusive"

This reverts commit 7804c433b3476542fc156cad9d4da4755740b123.

* Run all Oracle tests sequentially
2021-10-27 16:01:00 +02:00
Robin Krom
12e782b1ea
ifaces: update ghc-parser (#11383)
* ifaces: update ghc-parser

This updates the ghc-parser, including the `ifaceTypeRep` method
generation for interfaces.

CHANGELOG_BEGIN
CHANGELOG_END

* pin stack snapshot on unix

* add tests

* update ghc-lib hash

* pin stackage dependencies on unix

* pinned stackage windows
2021-10-27 11:29:22 +00:00
Sofia Faro
76eb1657d2
Interface fixed choices: ghc parser (#11275)
* Interface fixed choices: ghc parser

(WIP)

changelog_begin
changelog_end

* .

* ..

* ...

* stackage unpin unix

* ...

* fold create/observer/signatory into fake lfconversion primitives

* stackage unpin

* .....

* ......

* ........

* .........

* . . . . .

* unpin stackage on unix

* ...... ......

* ...

* . . . . .

* unpin stackage on unix

* . . . . . . . . .

* unpin stackage on unix

* Loosen restriction on ETo/FromAnyChoice fake primitives

* uncomment exercise in Interface.daml

* remove comments

* !!!

* the final re-pinning, part 1

* the final re-pinning: part 2
2021-10-20 17:45:13 +00:00
Samir Talwar
139b6f3b9e
CI: Set PROJ_DIR inside the bash lib, not outside. (#11236)
* CI: Set `PROJ_DIR` inside the bash lib, not outside.

We can't reference variables set outside.

CHANGELOG_BEGIN
CHANGELOG_END

* CI: Remove an unnecessary `export`.
2021-10-13 14:55:20 +02:00
Robert Autenrieth
66adbc0366
DPP-463 switch conformance tests to append only (#11101)
* Use append-only schema in tests

changelog_begin
changelog_end

* Fix Oracle build

* Update ledger/ledger-on-memory/BUILD.bazel

Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>

* Run more conformance tests on append-only

* Run pruning tests in split-participant mode

* Disable multi-participant test in split participant

* Add missing ContractIdIT test

Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
2021-10-11 23:31:30 +02:00
Robin Krom
3115d104aa
daml script interfaces (#11124)
* interfaces: introduce TemplateOrInterface class in stdlib

The template typeclass is to strong for many applications. The new
constraint `TemplateOrInterface` only contains the methods to convert
contract IDs and choices.

CHANGELOG_BEGIN
CHANGELOG_END

* ghc-lib update, interface script tests

* pinned stackage on unix

* added missing implementation in preprocessor

* added test

* fixing tests

* remove Iface type

* pinned stackage windows

* make sure createAndExercise is not called on interfaces
2021-10-11 18:22:40 +02:00
nicu-da
5abedac9ab
Check the commit message to skip the compatibility check [KVL-1111] (#11162)
* Check the commit message to skip the compatibility check

CHANGELOG_BEGIN

CHANGELOG_END

* Use git trailers to skip the check conditionally
2021-10-08 02:18:36 -07:00
Samir Talwar
d3810af47e
CI: Upgrade Nix. (#11173)
Inspired by seeing `curl` TLS errors in a different repository's CI.

I don't know if this will help, but it's worth a try. Note that the
domain name has changed.

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-08 09:13:50 +00:00
Gary Verhaegen
fcd4549569
test release rotation: factor out selection (#11161)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-07 12:46:55 +00:00
nicu-da
a3b90499fd
Check proto compatiblity with target head for PRs [KVL-1110] (#11123)
* Add the ability to check protobuf compatibility with the branch head alongside the latest release tag.

CHANGELOG_BEGIN
CI - add check to PR for protobuf breaking changes against the target branch head
CHANGELOG_END
2021-10-07 08:55:38 +00:00
Stefano Baghino
265e36255c
Add a notice to the release testing rotation file (#11142)
Reminds people adding new users to the rotation that they should also
be added to DACH-NY/daml-language-ad-hoc to have a box in the cloud
for Windows testing.

Adds the possibility of adding comments at the beginning of the rotation file.

changelog_begin
changelog_end
2021-10-06 21:08:11 +00:00
Remy
9cffa1faf6
LF: check LF transaction protobuf aginst local snapshots (#11064)
instead of last stable branch.

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-04 20:10:41 +02:00
Remy
37d94aaef6
Add unit for daily performance tests (#11107)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-04 13:00:38 +02:00
Robin Krom
9fd6326ae7
interfaces: consuming/non-consuming iface choices (#11009)
* interfaces: consuming/non-consuming iface choices

We add the consumption behaviour to the interface choice definition and
typecheck accordingly.

CHANGELOG_BEGIN
CHANGELOG_END

update to new ghc-lib, conversion implementation

* update ghc-lib

* pinning stackage on unix

* pin stackage on windows
2021-09-30 15:56:28 +00:00
Gary Verhaegen
6ae3afa8fb
fix perf reporting (#11073)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-29 17:39:46 +00:00
Gary Verhaegen
12e0c72d5c
fix blackduck logic (#11049)
We believe the Blakduck logic is currently faulty. We have had a
violation on an NPM dependency, and Blackduck keeps reporting it despite
our having removed the dependency.

We believe that what is happening is that, in the first step of
checking, we udpate the Haskell dependencies, _and then check the
validity of the whole project_, which includes the NPM deps. Because
that fails, we never get to the step where we actually update the NPM
deps, and Blackduck is stuck forever.

The solution is to not fail on violations for the Haskell update steps.
Haskell deps are still checked in the second step, because, again, it is
checking the whole project.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-28 14:28:19 +00:00
Gary Verhaegen
59ad995282
fix buf check (#11014)
`SYSTEM_PULLREQUEST_TARGETBRANCH` is not defined on main builds.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-24 10:23:48 +02:00
fabiotudone-da
f5d2135f97
Check protobuf compatibility of main and PR commits w.r.t. previous stable release [KVL-1109] (#10950)
* Check protobuf compatibility of release commits w.r.t. previous stable release

CHANGELOG_BEGIN
CHANGELOG_END

* Remove blank line

* Don't persist credentials

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* check-protobuf-against-stable.sh: SRC_DIR -> PROJECT ROOT + simplify

* Don't set LATEST_STABLE global in a function

* Simplify by using only the main work tree

* Simplify further as the check will be only run from `main`

* Move the check to `ci/build.yml` so that it is also run on PRs

* Enter the development environment to use tools

* Make variables read-only

* Support release branches and PRs targeting them

* Fix and document the reference tag finding logic

* Fix SYSTEM_PULLREQUEST_TARGETBRANCH and print it

* Don't log the source branch

* Fix comment formatting

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Enable Slack integration

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Don't check if the branch is a release one

...as the check won't be run on release branches.

* Add compatibility_stable_protobuf to collect_build_data

* Do not activate dev-env globally but only in sub-shells

* Add an explanation about why the check is not run on release branch commits

* Simplify further by leveraging `buf`'s ability to compare against branches

* Use `buf`'s `tag` locator instead of `branch`

* Split buf checks by module and remove previous manual check

* Explain how to run locally

* Use more future-proof WIRE_JSON for participant-integration-api

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>

* Use stricter FILE for the ledger gRPC API

* Propose an explanation for WIRE in kvutils

* Fix comment typo

* Re-introduce linting configuration for kvutils

* Simplify explanation for KVUtils' breaking check rule

* Remove extra (C) header from 3rd-party proto

* Don't touch the copyright of google/rpc/status.proto

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
2021-09-23 15:50:33 +00:00
akshayshirahatti-da
721575ea73
[JSON-API] Postgres perf job (#10986)
* Addition of a postgres perf job

CHANGELOG_BEGIN
CHANGELOG_END

* use a single job with strategy matrix

* cleanup test job from main pipeline
2021-09-23 09:16:58 +01:00
Gary Verhaegen
28b8d9a1f7
bump dotnet (#10979)
This bumps dotnet to the version required by the latest azuresigntool,
and pins azuresigntool for the future.

As usual for live CI upgrades, this will be rolled out using the
blue/green approach. I'll keep each deployed commit in this PR.

For future reference, this is PR [#10979].

[#10979]: https://github.com/digital-asset/daml/pull/10979

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-22 16:39:40 +00:00
Moritz Kiefer
4acf34b5fc
Add conversions from/to interfaces in Daml parser (#10954)
* Add conversions from/to interfaces in Daml parser

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end
2021-09-21 18:24:12 +00:00
Moritz Kiefer
cac8391d4c
Bump ghc-lib to include daml interfaces parser (#10747)
changelog_begin
changelog_end
2021-09-16 17:52:16 +00:00
fabiotudone-da
be4e06427f
Ledger API Test Tool: support --additional tests [KVL-1100] (#10829)
* Support adding tests as an hidden option

* Simplify existing suites

CHANGELOG_BEGIN
CHANGELOG_END

* Remove stale conformance suites from build.yml

* `--add` -> ``--additional`

* Re-add `--all-tests` as deprecated CLI option to be tested

* Move sandbox-classic pruning test to wall clock again

* Run KVCommandDeduplicationIT for sandbox append-only

* Tidy-up

* Also add participant pruning test to ledger-on-memory/single-participant

* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only

* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only

* Add KVCommandDeduplicationIT to ledger-on-memory append-only

* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant

* Tidy-up

* Use KVCommandDeduplicationIT for sandbox-on-x too

* Fix merge

Add max dedup duration arg to all the test suites that include command dedup tests

* Make `--include` and `--additional` mutually exclusive

* Uniform formatting of multi-line strings

* Move exclusions after additions as they are applied last

* Re-disable deduplication test on sandbox with static time

* Re-disable deduplication test on sandbox-on-x
2021-09-14 14:00:30 +02:00
akshayshirahatti-da
fb5ab5dca2
setvar doesn't like new lines in assignment, refactor (#10842)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-10 14:04:24 +00:00
akshayshirahatti-da
7824201b8d
Get rid of extra new line and prune precision for all gatling values (#10820)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-09 08:37:50 +00:00
akshayshirahatti-da
35a853fd30
Async logging for http-json-perf tests and publish logs as a pipeline artifact (#10803)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-07 23:42:30 +01:00
Gary Verhaegen
a349217f78
get oracle perf test output (#10800)
We can only reference variables set in a named step.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-07 23:14:44 +02:00
Gary Verhaegen
793253ca87
bump cleanup threshold (#10771)
I've witnessed a build ([link], though that will likely expire soon)
that failed with a "No space left on device" error after skipping the
cleanup step because the machine still had 68GB free.

[link]: https://dev.azure.com/digitalasset/daml/_build/results?buildId=87591&view=logs&j=870bb40c-6da0-5bff-67ed-547f10fa97f2&t=deecee86-545a-596e-8b0d-fb7d606fe9f2

With the machines only having 200GB disk size total, cleaning up at 80
is probably going to start hampering the overall efficiency of the
cache. It may be time to think about increasing the disk size itself (or
finding ways to reduce the size requirements of our builds). Important
note, though: we can't actually increase the macOS disk size very much.

The failure happened on the `compatibility_linux` job.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-03 18:22:18 +02:00
akshayshirahatti-da
f68a12930b
Report some oracle_json_perf numbers on slack (#10754)
* Add avg, stddev, p90, p99, requests_per_second numbers to be reported on slack similar to speedy_perf

changelog_begin
changelog_end

* changes based on code review

* fix failing job due to breaking function export
2021-09-02 17:44:01 +01:00
akshayshirahatti-da
4525b8c265
[JSON-API] vanilla oracle_perf ci job (#10688)
* vanilla job test on main pipeline

changelog_begin
changelog_end

* move job to daily compat tests

* add timeout to dev-env and changes based on code review
2021-09-02 09:31:56 +01:00
Moritz Kiefer
e9c8af5024
Bump ghc-lib to include dropped parsing code for generic templates (#10735)
* Bump ghc-lib to include dropped parsing code for generic templates

changelog_begin
changelog_end

* bump snapshot

changelog_begin
changelog_end

* drop old generics file

changelog_begin
changelog_end

* drop other broken file

changelog_begin
changelog_end

* Bump again

changelog_begin
changelog_end

* bump to merged commit

changelog_begin
changelog_end

* and bump snapshots

changelog_begin
changelog_end
2021-09-01 15:35:40 +02:00
Andreas Herrmann
7b94b0674e
Map shortened scala test suite names to long names on Windows (#10628)
* Generate short to long name mapping in aspect

Maps shortened test names in da_scala_test_suite on Windows to their
long name on Linux and MacOS.

Names are shortened on Windows to avoid exceeding MAX_PATH.

* Script to generate scala test name mapping

* Generate scala-test-suite-name-map.json on Windows

changelog_begin
changelog_end

* Generate UTF-8 with Unix line endings

Otherwise the file will be formatted using UTF-16 with CRLF line
endings, which confuses `jq` on Linux.

* Apply Scala test name remapping before ES upload

* Pipe bazel output into intermediate file

Bazel writes the output of --experimental_show_artifacts to stderr
instead of stdout. In Powershell this means that these outputs are not
plain strings, but instead error objects. Simply redirecting these to
stdout and piping them into further processing will lead to
indeterministically missing items or indeterministically introduced
additional newlines which may break paths.

To work around this we extract the error message from error objects,
introduce appropriate newlines, and write the output to a temporary file
before further processing.

This solution is taken and adapted from
https://stackoverflow.com/a/48671797/841562

* Add copyright header

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-08-24 17:03:45 +02:00
Stefano Baghino
b578b0e7cf
Reminder to put an empty line between subject and body (#10638)
* Reminder to put and empty line between subject and body

changelog_begin
changelog_end

* Update ci/check-changelog.sh

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2021-08-20 11:28:30 +00:00
Robin Krom
0c1878530b
test: test for platform independent dars (#10535)
We add a CI test to check that dars don't depend on the underlying
operating system where the dar is build.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-17 18:59:12 +02:00
Moritz Kiefer
3cc9de564f
Introduce dependency on Oracle image version (#10597)
We want to rerun tests when the image changes which this PR should
hopefully accomplish.

changelog_begin
changelog_end
2021-08-17 13:28:26 +00:00
Moritz Kiefer
fd9d8724fc
Upgrade to Oracle 19.12 (#10589)
* Upgrade to Oracle 19.12

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end

* .

changelog_begin
changelog_end
2021-08-17 13:16:41 +02:00
Moritz Kiefer
147e8f2b96
Skip subject in changelog check (#10584)
* Skip subject in changelog check

This matches what unreleased.sh does.

Ideally we’d probably share the code but this is bash and I do not
like bash so I cannot be bothered to do this right now.

changelog_begin
changelog_end

* better error message

changelog_begin
changelog_end
2021-08-16 12:01:17 +00:00
Samir Talwar
41e60f7302
Upgrade to Scala 2.12.14 and 2.13.6. (#10573)
* Upgrade to Scala 2.12.14.

* Upgrade Scala to 2.13.6.

CHANGELOG_BEGIN
CHANGELOG_END

* compability: Re-pin the Maven dependencies.
2021-08-12 14:25:53 +00:00
Moritz Kiefer
5e7f8e6a49
Handle not quite empty directory digests in cache cleanup (#10536)
Dropping the strict source name patch has resulted in empty
directories which Bazel is happy about but then fail at
runtime. They’re not quite empty, the digest points to 0x0a 0x00
so we match on that.

changelog_begin
changelog_end
2021-08-10 15:06:10 +00:00
Moritz Kiefer
ebab670576
Disable pushing Bazel metrics on PRs from forks (#10520)
We don’t have credentials there so the job just fails.

changelog_begin
changelog_end
2021-08-09 11:54:26 +00:00
Robin Krom
36f3ba8a6a
lsp-types patch for platform independence (#10288)
* fix shake rebuild tests

* patch pretty printing in ghc

* pin linux deps

* pin windows deps

* cosmetics

* address review

* update ghc commit hash

* pin linux stackage deps

* update stack snapshot

* pin windows deps
2021-08-04 21:33:52 +02:00
Moritz Kiefer
b0498ce4b1
Publish test logs on Oracle build failures (#10445)
changelog_begin
changelog_end
2021-07-29 12:58:44 +02:00
Gary Verhaegen
c3a3d60083
don't call Gary, he's on holiday (#10400)
I'm still not sure how or why this happens, but if we can detect it
"early" to fail and try to debug, we can also just try to fix it 🤷

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-24 09:12:42 +02:00
Gary Verhaegen
da9f8e3007
clean-up perf tests (#10355)
We've removed some features, this can be simplified a bit now.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-22 12:07:19 +00:00