Commit Graph

7207 Commits

Author SHA1 Message Date
Moritz Kiefer
1b428be7d2
Add ledger API test tool tests for rollback projections (#9778)
* Add ledger API test tool tests for rollback projections

This adds 3 tests for projections under rollback nodes.

The first one is relatively clear hopefully and tests divulgence.

The other two are a bit more intricate. For both of those we can also
not test too much via the ledger API since we don’t actually get
access to rollback nodes. However, it still seems useful to at least
exercise those code paths and make sure they don’t do anything
horribbly wrong.

The second test tests the normalization rules from
https://github.com/digital-asset/daml/blob/main/docs/source/concepts/ledger-model/ledger-exceptions.rst#privacy

The last one tests a more complex structure with deeply nested
rollback nodes and different informees.

changelog_begin
changelog_end

* Update ledger/test-common/src/main/daml/semantic/Exceptions.daml

Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>

Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
2021-05-25 08:53:43 +02:00
Gary Verhaegen
fb6f72c81d
debug signtool on main (#9780)
I don't know what's going on. I was hoping I could try to debug from
PRs, but that doesn't seem to work.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-24 13:26:41 +00:00
azure-pipelines[bot]
459de1c74d
update NOTICES file (#9781)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-05-22 10:34:05 +02:00
Gary Verhaegen
61293c3ec4
maybe fix azuresigntool invocation (#9779)
🤷

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 17:01:27 +00:00
Robert Autenrieth
66b8f19407
DPP-406 split append only migration (#9777)
* Rename migrations

* Split the append-only migration

changelog_begin
changelog_end
2021-05-21 16:36:19 +02:00
Gary Verhaegen
45bca6e68b
test_windows_signing: install for u (#9776)
Turns out "`--global`" means "for this user".

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 15:49:31 +02:00
Victor Peter Rouven Müller
ed36caa99a
Make use of nameOf for naming telemetry traces in participant-integration-api (#9762)
* Make use of nameOf for naming telemetry traces in participant-integration-api

changelog_begin
changelog_end

* Add copyright header to Telemetry.scala

* Add a simple nameof macro lib which has one function to return the fqn of the current method

* Make use of the new nameof lib within the repo to shorten the code further

* Remove old dependency on nameof macro

* Readd lines which have been accidentally removed
2021-05-21 12:34:55 +00:00
Marton Nagy
6f20d78056
Makes sandbox-classic compression configurable (#9770)
* Adds configuration parameter and wiring
* Only append only schema is supported (for mutating schema it has no effect)

changelog_begin
changelog_end
2021-05-21 12:12:20 +00:00
Gary Verhaegen
b8afa02e12
test_windows_signing: add quotes (#9773)
Otherwise Bash interprets the baskslashes and it's not a path anymore.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 11:12:09 +00:00
Gary Verhaegen
4af6608185
fix signing machine (#9772)
Turns out PowerShell is not Bash. Who knew? 🤷

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 12:36:56 +02:00
Gary Verhaegen
5717356a42
fix test_windows_signing job name (#9771)
😢

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 10:33:25 +00:00
Remy
5855a9941f
LF: Drop CONTRACT_ERROR (#9754)
CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 09:12:29 +00:00
Gary Verhaegen
f5c5b634eb
prepare for EV Windows signing (#9758)
Setting up a non-disruptive way to test out EV signing of our Windows
artifacts.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-21 10:46:45 +02:00
Moritz Kiefer
88c83ec476
Mark //daml-assistant/daml-helper:test-deployment as flaky (#9769)
Fails a bit too often to waste people’s time with restarts.

changelog_begin
changelog_end
2021-05-21 09:57:19 +02:00
Andreas Herrmann
fa0815bc11
Example export (#9756)
* Pretty print daml.yaml in Daml ledger export

changelog_begin
changelog_end

* Use dynamic port in example-export generation

* Add daml.yaml to Daml ledger export golden test

* Sort data-dependencies for reproducibility

* Replace hashes in data-dependencies with a placeholder

To avoid test failure when any of these change.

* normalize data-dependencies path in scala client

So that the bazel build action is more reproducible itself.

* Fix Scala 2.12 build

* Factor out deleteRecursively

changelog_begin
changelog_end

* Generate ledger export in temp-dir

Build actions are not sandboxed in Windows. The ledger export will also
created a directory `deps` to store all the DALFs in. On Windows this
directory may persist across Bazel builds and may cause errors due to
attempts to overwrite an existing directory.

To avoid these issues the ledger export is generated into a temporary
directory and only the wanted files are moved to the expected output
locations in the bazel execroot afterwards.

* Close source

* Use replace instead of replaceFirst

The latter matches regex pattern instead of just substring matching
which fails with Windows paths due to `\` being interpreted as an escape
character.

* Normalize Windows \ to Unix /

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-05-21 09:36:25 +02:00
azure-pipelines[bot]
3b33397bfe
update NOTICES file (#9767)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-05-21 07:13:07 +00:00
Moritz Kiefer
4f97a4da8e
Avoid test log collisions across platforms (#9763)
Currently they overwrite each other which is clearly not helpful.

changelog_begin
changelog_end
2021-05-21 09:11:16 +02:00
Moritz Kiefer
6f631da09d
Use munit for diffs in ledger-api-test-tool (#9765)
As everyone knows that has worked on the ledger API test tool, diffx
makes compilation very very slow (>150s for TransactionServiceIT).

As some people (myself included) know, trying to get diffx to not be
slow is also a giant nightmare (have fun debugging mutually recursive
implicits derived by Scala macros)

So this PR takes a different approach:

Drop diffx completely and replace it by munit. This leads to slightly
different diffs but I think they’re close enough and importantly munit
involves absolutely no macros for those diffs so compilation takes 5s
for TransactionServiceIT.

changelog_begin
changelog_end
2021-05-21 08:40:17 +02:00
Brian Healey
92568eb2fe
Make hikari connection pool timeout configurable (#9750)
* 2 seconds rather than 250ms for default hikari connection pool timeout

CHANGELOG_BEGIN
[jdbc ledger] increase default hikari connection pool timeout to 2s from 250ms
CHANGELOG_END

* extract hikari connection timeout configuration

* default of 250ms

* read hikari connection timeout default from config rather than hardcoding for sandbox next

* remove unused import
2021-05-20 17:51:30 -04:00
Marton Nagy
38cb8b4d3a
Fix flaky CMConcurrentSetConflicting conformance test (#9766)
By relaxing the assertion to two possible GRPC error codes.
Also adds dependency of libs-scala/grpc-utils to test suites.

changelog_begin
changelog_end
2021-05-20 20:57:27 +00:00
Robert Autenrieth
d2bce1628e
DPP-335 Implement data migration for the append-only schema (#9481)
* Implement data migration

changelog_begin
changelog_end

* Rename migration

* Update test

* Move the append-only migrations

* Enable data migration test

* Fixup after script rename

* Check append-only migration hashes

* Add missing comments

* Simplify query

* Add vacuum migration

* Rename unknown workflow id

* Only use head migration script

* Rename history table

* Fix comments

* Align previous schema change after rebase

changelog_begin
changelog_end

Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
2021-05-20 21:33:18 +02:00
Sofia Faro
765d7e3f53
Throw ContractError in template precondition field (#9760)
* Throw ContractError in template precondition field

changelog_begin
changelog_end

* s/pre-condition/precondition/where relevant
2021-05-20 19:10:25 +00:00
Moritz Kiefer
f5a03b3af6
Stop leaking Sdk installations from assistant integration tests (#9751)
Each of these installations is about 1gb so it’s not that unlikely
that this plays a significant factor in our out of disk space errors.

Although I think we cleanup partially so I’m not sure how large the
part of the SDK installation is that gets leftover.

changelog_begin
changelog_end
2021-05-20 20:20:12 +02:00
Remy
a2775605d8
LF: change format of ArithmeticError message (#9761)
in order to follow LF conventional syntax.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-20 20:10:31 +02:00
Moritz Kiefer
90180eb87f
Make sure we don’t start leaking directories in compat tests (#9759)
As evidenced by this change, we don’t actually need the safeguard
here. i think this was just copied over from the main workspace where
we did leak installations. I’d like to know when we start leaking
installations here in the future so this PR removes the catch.

changelog_begin
changelog_end
2021-05-20 13:59:11 +00:00
Victor Peter Rouven Müller
beb2d138f2
Add cli option & system property to enable json only logging for http json service (#9725)
changelog_begin
- [Ledger HTTP Json Service] Logging output can now be in JSON either via providing the cli option `--log-encoder json` or via setting the env var `LOG_FORMAT_JSON=true`
changelog_end
2021-05-20 13:28:34 +00:00
Moritz Kiefer
3827f82a7a
Fix consumedBy in rollback nodes for scenario service (#9746)
changelog_begin
changelog_end
2021-05-20 12:07:13 +00:00
Sofia Faro
2ba5ca7ca2
Make DA.Assert throw AssertionFailed instead of GeneralError (#9747)
* Make DA.Assert throw AssertionFailed instead of GeneralError

changelog_begin

- [Daml Standard Library] `assert`, `(===)`, and other assertion functions (see DA.Assert) now use a new `CanAssert` typeclass constraint instead of `CanAbort`, in preparation for exceptions support.

changelog_end

* Add CanAssert instances for Script and Trigger.

* buildifier-fix

* update script test runner output

* Fix flag and add tests.

* update script test runner again
2021-05-20 12:01:50 +00:00
Moritz Kiefer
b9c36c9a5b
Disable per commit windows compat tests (#9755)
* Disable per commit windows compat tests

Windows version of #9370 to further reduce queues until we either get
more nodes or find another solution. We still test everything in the
daily run.

changelog_begin
changelog_end

* remove success check for skipped windows compat

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2021-05-20 10:37:59 +00:00
Miklos
b1ca310866
ledger: Damlification of Scala files (#9667)
* Damlification of Scala files (primarily comments and strings).

* Corrected a typo.
CHANGELOG_BEGIN
CHANGELOG_END

* Fixed build.

* Fixed test case for acronyms.

* Reformatted.
2021-05-20 10:21:04 +00:00
Marton Nagy
40cf26fb8c
Log improvements for parallel indexer (#9752)
changelog_begin
changelog_end
2021-05-20 11:15:46 +02:00
Miklos
3bf6531e3c
ledger-service, navigator: Damlification of Scala files (#9669)
* Damlification of Scala files (primarily comments and strings).

* Comment tidying.
CHANGELOG_BEGIN
CHANGELOG_END
2021-05-20 09:14:13 +00:00
Marton Nagy
f947c02c74
Extend ConfigManagementServiceIT with tests on generation checking (#9753)
changelog_begin
changelog_end
2021-05-20 09:11:32 +00:00
Miklos
f5c84a2aa4
Daml-LF: Damlification of Scala/Haskell files (#9666)
CHANGELOG_BEGIN
CHANGELOG_END
2021-05-20 09:52:33 +02:00
Remy
0d931b2f4b
Speedy: implement Arithmetic builtin exceptions (#9653)
part of #8020

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-19 21:46:48 +02:00
Moritz Kiefer
5f954da841
Fix flakiness in hot reload tests (#9748)
I started looking into failures in the integration tests and this
seems by far the most prevalent one (maybe even the only one but that
needs more investigation). The issue here is that sometimes waiting
40s isn’t enough. On the other hand, sometimes we wait unnecessarily
long which is also bad.

The fix in this PR is to wait until we see the "reload complete"
message which should be exactly the right amount of time to wait.

changelog_begin
changelog_end
2021-05-19 18:48:00 +00:00
Remy
40b23810d2
Compiler: clean convertion builtin name (#9724)
This follows up #9716

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-19 19:16:29 +02:00
Marton Nagy
7b1a776dd7
Dpp 384 todo cleanup postgres storage backend (#9727)
* Remove TODO regarding isolation level and guarantees at initialisation: that should not be an issue since the indexer is designed to be running in a singleton fashion (at most one at the time)
* Remove TODO regarding configuration conflict checking, with the assumptions:
1) This type of conflict checking for parallel ingestion is not an issue because the committer should take care of it
2) Conlfict checking for sandbox-classic behavior is not changed (logic taken in the same form)
* Remove configuration field from parameters table

changelog_begin
changelog_end
2021-05-19 16:37:33 +02:00
Remy
2e9bb21091
Engine: clean conversion builtin names (#9726)
Rename conversion builtins in the engine to follow
archive conventions.

This follows up #9716

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-19 13:02:16 +00:00
Andreas Herrmann
8efdde6144
FutureResourceOwner --> ResourceOwner.forTry (#9720)
changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-05-19 14:58:51 +02:00
Sofia Faro
1794a6f969
Refactor GenerateStablePackages (#9744)
* Refactor GenerateStablePackages

Summary of changes:

* Split GenerateStablePackages executable into a library (containing the stable package data) and an executable (that writes out the packages)

* Hook up data-dependencies test to the new library so we don't have to manually update the number of stable packages every time we add a stable package or release a new LF version.

changelog_begin
changelog_end

* buildifier-fix
2021-05-19 13:57:09 +01:00
azure-pipelines[bot]
14cfe9916e
update compat versions for 1.14.0-snapshot.20210518.6953.0.a6c7b86a (#9741)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-05-19 14:28:09 +02:00
Gary Verhaegen
83d60f0561
more frequent Windows cache cleanup (#9743)
We'd like this to run more than once an hour, and Azure doesn't support
that as a cron.

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-19 14:24:47 +02:00
Andreas Herrmann
10177d239e
Add golden test for example Daml ledger export (#9732)
* client_server_build - user defined outputs

Let the user define a list of output files on client_server_build.

changelog_begin
changelog_end

* daml-script runner expose main(config)

changelog_begin
changelog_end

* Add an example Daml ledger export to the docs

* Build Daml ledger export

changelog_begin
changelog_end

* sh_inline_test

Support toolchain arguments to sh_inline_test. Usefuly for make variable
expansion, e.g. for the POSIX toolchain.

* Add a golden test for Daml ledger export

* Test args files as well

* Use sed from POSIX toolchain

* Add normalization comment to top-level comment

* Ignore trailing CR on Windows

The JSON formatting of the args file uses Windows line endings on
Windows. Therefore, the args.json output technically differs from the
expected output. We're happy to ignore the difference in CRLF here.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-05-19 12:02:14 +00:00
Moritz Kiefer
bb5dd4cbf3
Fix locking of envScenarioContexts (#9736)
This fixes a race condition in our handling of scenario contexts. See
the comments for details. I verified with a bunch of extra logging
that this is what is actually failing in our tests. I’ll try to
upstream the logging separately since ideally I’d like to have that in CI.

I ran all integration tests with --runs_per_test=20 over night and
with this change I’m no longer able to get it to flake so dropping the
flaky marker.

fixes #6910

changelog_begin
changelog_end
2021-05-19 12:51:30 +02:00
Remy
fcbba1cd95
LF: rename convertion builtins to be more obvious (#9716)
The name of some builtins will be exposed as part of the Exception
message. This PR, try to make conversion builtins more consistent and
more obvious, before we could not rename those.

This is part of #8020

CHANGELOG_BEGIN
CHANGELOG_END
2021-05-19 10:11:26 +00:00
Sofia Faro
b55613be6c
Parallelize the data-dependencies test. (#9733)
* Parallelize the data-dependencies test.

It runs about twice as fast on my laptop (240s -> 120s) when allowed to
run tests in parallel. I'm not sure this will work or make a huge
difference when it comes to timeouts in CI, but it's worth a shot.

changelog_begin
changelog_end

* TASTY_NUM_THREADS := 3

* set bazel tag for test
2021-05-19 10:03:00 +00:00
Sofia Faro
3209188fcc
Add a ledger model page for exceptions. (#9396)
* Add a ledger model page for exceptions.

This page describes the changes to the ledger model as part of the exceptions feature. The changes should be kept separate for now, but once exceptions are stable and out for a while, they should probably be incorporated into the rest of the daml ledger model.

Part of #8020. **This PR is to be merged only when we start releasing exceptions.**

changelog_begin

- [docs] The Daml ledger model has been updated to describe the changes introduced by the new exceptions feature. See  here: https://docs.daml.com/concepts/ledger-model/ledger-exceptions.html

changelog_end

* Apply suggestions from code review

Co-authored-by: Ognjen Maric <ognjen.maric@digitalasset.com>

* Round 2

* Fix and simplify "after" relation!

* Easy fixes

* Diagrams for structure and integrity sections.

* Add missing diagrams :)

* clarify consistency section

* Authorization.

* Editing.

* fix label reference

Co-authored-by: Ognjen Maric <ognjen.maric@digitalasset.com>
2021-05-19 09:43:18 +00:00
Moritz Kiefer
6245e3456c
Drop damlc integration test todo (#9735)
I have no idea what this is supposed to mean. I suspect it’s leftover
from Daml < 1.1 or something like that.

changelog_begin
changelog_end
2021-05-19 11:26:46 +02:00
Kamil Bożek
ae523c7e2f
ledger-api-bench-tool - support for multiple streams [DPP-372] (#9691)
* Added resource management and the LedgerIdentityService

* Added the TransactionService

* Added stream configuration options

* Options for ledger configuration

* Minor improvements

* Refactored packages

* CHANGELOG_BEGIN
- [Integration Kit] - Created the ledger-api-bench-tool prototype for benchmarking ledger transaction streaming capabilities
CHANGELOG_END

* Handler for stream completion

* Transaction stream observer with count metrics

* Generic CountingStreamObserver

* Size metrics

* Rounding results

* Simplified observers hierarchy

* Reporting period program argument

* Metric[T] trait

* Named streams

* Renamed reporting-period parameter to log-interval

* Template filters

* Configurable ledger offset parameter

* Delay metric

* Improved metrics naming

* Consumption speed metric

CHANGELOG_BEGIN
- [Integration Kit] - added metrics to the ledger-api-bench-tool
CHANGELOG_END

* Minor fix

* Fixed stream type parameter name

* Multi-stream support

* Pretty print config

* Fixed rounding of metric values

* Pretty summary

* Minor change

* CLI parameters for concurrency settings

* Minor change

* Actor-based metrics

* Resource management for the typed ActorSystem

* Abstract MetricsManager

* Immutable metrics

* TransactionMetrics

* Cleaner TransactionService

* Factoring out stream observer from the TransactionService

* Fixed transaction trees filters

* CHANGELOG_BEGIN
- [Integration Kit] - support for multiple streams in the ledger-api-bench-tool
CHANGELOG_END

* Removed sleeping in the stream observer

* Improved README.md

* Fixed build errors in Scala 2.13

* Fix counting metric total rate

* Improved concurrency CLI parameters

* Changed the log level of stream completed notification

* Renamed stream observer
2021-05-19 10:07:34 +02:00