Commit Graph

7994 Commits

Author SHA1 Message Date
fabiotudone-da
d4cb1f945e
KV ledgers: disable participant side command de-duplication [KVL-1083] (#11095)
* Extract common code for command dedup conformance tests

CHANGELOG_BEGIN

CHANGELOG_END

* Revert changes for parallel tests

* Fix compile warnings

* Disable participant deduplication for KV

CHANGELOG_BEGIN
CHANGELOG_END

* Always call submitRequestAndAssertDeduplication with parties

* Fix party in CommandDeduplicationBase

* Don't test KV-based command deduplication with static time as it's broken

* Don't disable participant-side command deduplication for sandbox-next on static time

* Revert "Don't test KV-based command deduplication with static time as it's broken"

This reverts commit 3dac41aa

* Add missing dependency

* Don't cross-test this ledger API test tool with existing versions

* Add the current PR to the list of PRs that resulted in exclusions

* Apply @garyverhaegen-da's suggestion for compatibility exclusions

Co-authored-by: Nicu Reut <nicu.reut@digitalasset.com>
2021-10-05 13:31:53 +02:00
Hubert Slojewski
b9a5a83175
Minor: fix trace logging in TimeBasedWriteSetSelector (#11117)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-05 08:30:22 +02:00
Victor Peter Rouven Müller
018e90854c
[JSON-API] Make key_hash idx non unique (#11102)
* Add failing test that covers the bug

* Fix on conflict error for inserts into the contracts table

changelog_begin

- [JSON-API] make key_hash indexes non-unique, this fixes a bug where a duplicate key conflict was raised on the query store when the same contract was being witnessed twice by two separate parties

changelog_end

* move test to parent so as to test oracle query store

* make key_hash indexes non-unique

* use recordFromFields

Co-authored-by: Akshay <akshay.shirahatti@digitalasset.com>
2021-10-05 06:28:04 +01:00
Robert Autenrieth
429f43731a
Remove HA indexer integration test on H2 (#11104)
This test was supposed to always fail, but sometimes it passes

changelog_begin
changelog_end
2021-10-04 22:52:54 +02:00
Remy
9cffa1faf6
LF: check LF transaction protobuf aginst local snapshots (#11064)
instead of last stable branch.

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-04 20:10:41 +02:00
Andreas Herrmann
d7ee278518
Optional table prefix for trigger service (#11047)
* Fix typo postgres --> oracle

* Move tablePrefix into base jdbcConfig

* Add table.prefix in trigger service migrations

* Add tablePrefix to trigger service db table names

changelog_begin
* [Trigger Service] Enable the new `tablePrefix` setting in the `--jdbc`
  flag to add a prefix to all tables used by the trigger service to
  avoid collisions with other components using the same db-schema.
changelog_end

* Add tablePrefix config test for trigger service

* Fix Oracle test

* Allow existing schema in trigger service

CHANGELOG_BEGIN
* [Trigger Service] Enable the new ``--allow-existing-schema`` flag to
  initialize the trigger service on a database with a pre-existing
  schema.
CHANGELOG_END

* Don't ignore CLI flag value

* Update triggers/service/src/main/scala/com/digitalasset/daml/lf/engine/trigger/dao/DbTriggerDao.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Use fragment interpolation

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2021-10-04 17:02:36 +00:00
nicu-da
268b2d3da0
Command deduplication - test case that uses CommandService and CommandSubmissionService [KVL-1106] (#11098)
Add test case for command deduplication which uses both the CommandService and the CommandSubmissionService

CHANGELOG_BEGIN

CHANGELOG_END

* Run mixed CommandService/CommandSubmissionService test for all the possible combinations of calls
2021-10-04 09:42:07 -07:00
nickchapman-da
fac05f657b
bypass value enrichment for the performance test (#11112)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-04 16:22:48 +01:00
Stefano Baghino
4860271a53
Port file was not written to... where? (#11108)
* Port file was not written to... where?

Adds the information on where the port file could not be written.
From the current phrasing it appears the intention was to add the
information but ultimately wasn't.

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/11108#pullrequestreview-770115835
2021-10-04 15:21:56 +00:00
akshayshirahatti-da
a8d703b996
Increase akka server request-timeout for HttpServiceIntegrationTests (#11109)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-04 12:09:27 +01:00
Remy
37d94aaef6
Add unit for daily performance tests (#11107)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-04 13:00:38 +02:00
Robert Autenrieth
eb4ac8a397
Increase timeouts for resetting database (#11103)
... between tests.

changelog_begin
changelog_end
2021-10-04 10:57:43 +02:00
Sofia Faro
db75f7da0f
interface methods: Scala Typechecker (#11097)
* interface methods: Scala Typechecker

Part of #11006.
I tried to match the behavior of the Haskell typechecker.

changelog_begin
changelog_end

* scalafmt

* Fix type
2021-09-30 16:41:49 +00:00
akshayshirahatti-da
d3e6f16c61
[JSON-API] Migrating tests to use sandbox next (#11034)
* Changes to use sandbox next for our integration tests

CHANGELOG_BEGIN
CHANGELOG_END

* remove sandbox classic dependency for HttpServiceTestFixture and perf tests

* rely on sandbox next fixture test class

* add missing dependencies for http-json-oracle

* changes based on code review comments

* Add tag to skip test case for scala_2_12 and also add a jdbc backend for sandbox spun up for perf tests

* Reduce size of contracts for archiving test
2021-09-30 17:29:37 +01:00
Moisés Ackerman
ffc8d6894f
Data dependencies deserialize imports (#11037)
* Extend ModRef type to include ImpSpec

* Extract 'importOriginFromPackageRef' from 'genModule'

* emitModRef with EmptyImpSpec for all items in the metadata value 'imports'

* Generalize simpleImportTest into dataDependenciesTest, for tests with a multiple-module library and app

* Test that orphan instances are propagated in data-dependencies

changelog_begin
changelog_end
2021-09-30 16:27:12 +00:00
azure-pipelines[bot]
4b31bf088f
update compat versions for 1.17.0 (#11075)
* update compat versions for 1.17.0

CHANGELOG_BEGIN
CHANGELOG_END

* add exclusions

* Revert "add exclusions"

This reverts commit 13a5083081.

* Fix command dedup for kv sandbox

changelog_begin
changelog_end

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-09-30 16:14:45 +00:00
Robin Krom
9fd6326ae7
interfaces: consuming/non-consuming iface choices (#11009)
* interfaces: consuming/non-consuming iface choices

We add the consumption behaviour to the interface choice definition and
typecheck accordingly.

CHANGELOG_BEGIN
CHANGELOG_END

update to new ghc-lib, conversion implementation

* update ghc-lib

* pinning stackage on unix

* pin stackage on windows
2021-09-30 15:56:28 +00:00
Moisés Ackerman
229ce4776f
Data dependencies serialize imports (#11038)
* Extract 'convertQualifiedModuleName' from 'convertQualified'

* Generate stub definition with orphan module imports

* Test that generated $$imports value has the right contents

* Update InternedTypes test to expect $$imports-related types

changelog_begin
changelog_end
2021-09-30 15:25:26 +00:00
nicu-da
5d9ec65cc6
Add test case for parallel command deduplication using mixed clients [KVL-1090] (#11093)
* Add test case for parallel command deduplication using mixed submission through the CommandService and the CommandSubmissionService

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-30 12:10:14 +00:00
Samir Talwar
d64d965230
kvutils: Print state updates before and after normalization. (#11096)
On integrity check failure.

This makes it easier to debug failures, as normalization can make the
state update mostly useless.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-30 13:56:30 +02:00
azure-pipelines[bot]
be216aa872
update NOTICES file (#11089)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-09-30 11:44:02 +00:00
nicu-da
517e866df5
Extract common code for command dedup conformance tests [KVL-1090] (#11092)
* Extract common code for command dedup conformance tests

CHANGELOG_BEGIN

CHANGELOG_END
2021-09-30 11:20:46 +00:00
John Sullivan
7a4963b43d
add --max-inbound-message-size flag to daml ledger export (#11087)
Add support for `--max-inbound-message-size` flag to the [Ledger Export](https://docs.daml.com/tools/export/index.html) tool.

CHANGELOG_BEGIN

- [Daml Assistant] Add support for `--max-inbound-message-size` flag to the Ledger Export tool.

CHANGELOG_END
2021-09-30 12:17:53 +02:00
Moritz Kiefer
26d10b267d
Drop NodeId type parameter (#10921)
changelog_begin
changelog_end
2021-09-30 09:13:54 +00:00
Leonid Rozenberg
3cbd9229aa
More NonEmpty functions (#10930)
* Add NonEmpty cons.

* Add NonEmpty find.

* Add NonEmpty delete, deleteBy.

CHANGELOG_BEGIN
Add cons, find, delete and deleteBy functions to NonEmpty module.
CHANGELOG_END

* Simplify type signator of delete result.
2021-09-30 09:13:20 +00:00
Sofia Faro
bcd468625c
interface methods: Speedy (#11076)
* interface methods: Scala AST

Part of #11006
This PR takes care of the Scala AST and decoder.
It leaves the encoder, type checker, and speedy for later.

changelog_begin
changelog_end

* scalafmt

* AstRewriter

* interface methods: Speedy

Part of #11006

changelog_begin
changelog_end

* scalafmt

* Remove InterfaceMethods test

* Update daml-lf/archive/src/main/scala/com/digitalasset/daml/lf/archive/DecodeV1.scala

Co-authored-by: Remy <remy.haemmerle@daml.com>

Co-authored-by: Remy <remy.haemmerle@daml.com>
2021-09-30 08:38:08 +00:00
pbatko-da
85adaab550
[DPP-417][DPP-595] Error codes switching - follow-up (#11074)
Renaming CLI flag --use-error-codes-v2 to --use-self-service-error-codes

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-30 10:30:25 +02:00
Stefano Baghino
d7b280b60c
Remove debug print in daml-lf/interpreter build (#11091)
Seems to be a leftover from #11067

changelog_begin
changelog_end
2021-09-30 07:43:38 +00:00
Remy
1ed6428980
LF: move archive snapshots in a separate directory (#11081)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-30 08:43:55 +02:00
nicu-da
5e424f8d1c
Command dedup conformance suites readme (#11051)
* Add readme which explains the differences between different command deduplication suites

CHANGELOG_BEGIN
CHANGELOG_END

* Update readme
2021-09-29 19:43:17 -07:00
Robert Autenrieth
8290347f74
DPP-496 HA indexer integration test (#11033)
* Add indexer stability test

changelog_begin
changelog_end

* Add test for DebugStorageBackend

* Reduce the log output for IndexerStabilitySpec

* Add Oracle test

* Rename file

* Increase timeout

* Rename DebugStorageBackend

* Address review comments

* Fixup rename

* fixup address review comments

* Refactor IndexerStabilitySpec

* Reset can only be called on an aborted read service

* Non-HA test can fail for multiple reasons

* Fix logback for Oracle tests

Otherwise logback complains about missing classes at startup

* Fix Oracle jdbc url

* Improve assert message

* Improve log output

Otherwise the user has to wait for 10min to get useful log output about the connection issue

* Change default log ids

The previous defaults are not valid for Oracle

* fmt
2021-09-29 22:03:56 +02:00
Marton Nagy
735c3090a3
Switch from InputStream to Byte Array at the binary content JDBC transport (#11072)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-29 21:44:24 +02:00
Gary Verhaegen
6ae3afa8fb
fix perf reporting (#11073)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-29 17:39:46 +00:00
fabiotudone-da
bf801a6e1b
Fix SimpleDeduplicationBasic when participant deduplication is disabled (#11083)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-29 16:58:01 +00:00
pbatko-da
62234dc763
[Short] Remove unused code (#11079)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-29 17:45:53 +02:00
Sofia Faro
c1d1521a14
interface methods: Scala AST (#11070)
* interface methods: Scala AST

Part of #11006
This PR takes care of the Scala AST and decoder.
It leaves the encoder, type checker, and speedy for later.

changelog_begin
changelog_end

* scalafmt

* AstRewriter
2021-09-29 14:11:23 +00:00
nicu-da
7dd9c2d3f0
Remove expectations for internal failures from parallel command dedup tests (#11061)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-29 05:07:01 -07:00
nickchapman-da
51125991e5
Explicit discard in daml-lf/interpreter (#11067)
* Explicit discard to avoid NonUnitStatement.

CHANGELOG_BEGIN
CHANGELOG_END

* define lf_scalacopts_stricter in scala.bzl
2021-09-29 11:24:42 +00:00
azure-pipelines[bot]
554b36c817
rotate release duty after 1.18.0-snapshot.20210928.7948.0.b4d00317 (#11059)
@stefanobaghino-da is taking care of 1.18.0-snapshot.20210928.7948.0.b4d00317 (#11058), so they get pushed back to the end of the line.

Please do not merge this before #11058.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-09-29 13:13:39 +02:00
Moritz Kiefer
d156964f0c
Release SDK 1.17.0 (#11062)
changelog_begin
changelog_end
2021-09-29 10:44:34 +00:00
Moritz Kiefer
6d8cf7089c
Handle interface methods in LF conversion (#11054)
changelog_begin
changelog_end
2021-09-29 12:23:44 +02:00
Robin Krom
87ecf1fe63
daml-ledger: better errors for non-json responses. (#11055)
This fixes #10985.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-29 11:46:54 +02:00
tudor-da
ec2d26f31f
[Divulgence pruning] Fixes divulgence pruning offset update query (#11046)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-29 08:35:23 +00:00
azure-pipelines[bot]
f13c6d61ec
release 1.18.0-snapshot.20210928.7948.0.b4d00317 (#11058)
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.

@stefanobaghino-da is in charge of this release.

Commit log:
```
b4d00317b4 detect unsynchronized contract table and retry (#10617)
3d779cfe12 [Mutable state cache] Fix initialization offset (#11024)
5458aa890c Switch on NonUnitStatements warning in daml-lf/transaction (#11048)
9641fd5f83 auth middleware: no print secret (#11050)
a885f52c4d [DPP-417] [DPP-595] Add error code version switching mechanism (#11035)
12e0c72d5c fix blackduck logic (#11049)
eb87b3439b kvutils: Add the logging context for ledger state operations. (#11030)
b7daa5f7d8 Address remaining dependabot alerts (#11045)
5e43f8c703 es: drop jobs-* indices (#10857)
03203b7360 Define encoding/decoding for module imports (#11036)
57a15972c7 Setting timeoutToleranceMillis to 10 minutes to prevent flakiness (#11043)
df59f3fe8e Fix Navigator dependabot alerts (#11044)
6bf45a344a Upgrade Navigator to Webpack 5 (#11040)
80e217e11f [DPP-622] Add conformance tests that verifies TLSv1.0 and TLSv1 are disabled. (#10983)
626e1fbd7d Small lf value.cids regression fix (found by canton unit tests) (#11032)
a4629a4450 KV: Ignore daml_lf_1.proto when checking for KV protobuf compatibility (#11021)
ee9be65d68 kvutils: Add metadata to `Err` [KVL-1032] (#10992)
0d3ae6e14c interface methods: Haskell Typechecker (#11028)
e36eb46f59 Resolve `set-value` to 4.0.1 and above (#11029)
4075624cfc interface methods: Haskell AST (#11018)
7c1fd50469 Bump grunt-browserify (#11026)
5f3f5824f2 Upgrade webpack-dev-server in Navigator (#11025)
91be1e1659 Drop matchdep dependency from docs build (#11023)
fe9aeffeaf Increase es disk size (#11019)
eac7963b87 LF: Refactor ProtoTest.scala (#11020)
e79a30aa80 For the client binding propagate the full original completion [KVL-1112] (#10879)
59ad995282 fix buf check (#11014)
abc3e6691c Increase the tolarance for handover of control in HaCoordinatorSpec (#10997)
19b2bf477f LF: Cosmetic clean-up in the Speedy Compiler (#11015)
cb0e41f101 LF: Add interface support to the Preprocessor (#11013)
c33297c5fe Remove `transactionNormalization` flag. (#11010)
f5d2135f97 Check protobuf compatibility of `main` and PR commits w.r.t. previous stable release [KVL-1109] (#10950)
bf8b75dab5 interface methods: Add protobuf definitions. (#11005)
88e1430a42 Make LargeTransactionTest use ValueEnricher, so it can work with normalized transactions coming out of the engine. (#11003)
a043926c30 rotate release duty after 1.17.0-snapshot.20210921.7889.0.1b473c2b (#10972)
35666caae8 Add MINIMAL pragma for Additive type class (#11001)
8de162baec [DPP-586] Upgrade to netty 4.1.67.Final and netty-tcnative-boringssl-static 2.0.40.Final (#10956)
871d03ba65 release 1.18.0-snapshot.20210922.7908.0.ced4a272 (#10998)
721575ea73 [JSON-API] Postgres perf job (#10986)
f2d9f07417 Release RC2 for SDK 1.17.0 (#10996)
```
Changelog:
```
- [JSON API] Under rare conditions, a multi-template query backed by database
  could have an ACS portion that doesn't match its transaction stream, if
  updated concurrently.  This conditions is now checked and accounted for.
  See `issue #10617 <https://github.com/digital-asset/daml/pull/10617>`__.
- The OAuth2 Middleware now obfuscates its Client Secret when logging
  its config.
- [Integration Kit] We have added ``loggingContext`` as an implicit
  parameter to more _kvutils_ trait methods. Implementors may need to do
  the same in their trait implementations. This can make it easier to
  log with the appropriate context.
java-client-bindings - the original full completion is included in the `CompletionResponse` when available
Daml on SQL, Integration Kit, Sandbox: Drop support for TLS 1.0 and 1.1 in Ledger API.
```

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-09-29 10:17:47 +02:00
Moritz Kiefer
d1805a3000
More dependabot fun (#11063)
at least it keeps my commit count up I guess

changelog_begin
changelog_end
2021-09-29 10:11:40 +02:00
azure-pipelines[bot]
07b273aab5
update NOTICES file (#11060)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-09-29 06:57:10 +00:00
Stephen Compall
b4d00317b4
detect unsynchronized contract table and retry (#10617)
* enumerating out-of-sync offsets at the DB level

* cleanup in lastOffset

* write the latest-requested-or-read offset when catching up

- Writing only the latest-read, as before, would imply unsynced offsets
  that are actually well-synced.  This puts the DB in a more uniform
  state, i.e. it should actually reflect the single value that the
  fetchAndPersist loop tries to catch everything up to.

* detecting lagging offsets from the unsynced-offsets set

- Treating every unsynced offset as a lag would make us needlessly retry
  perfectly synchronized query results.

* add Foldable1 derived from Foldable for NonEmpty

* nicer version of the unsynced function

* ConnectionIO scalaz monad

* rename Offset.ordering to `Offset ordering` so it can be imported verbatim

* finish aggregating in the lag-detector function, compiles

* port sjd

* XTag, a scalaz 7.3-derived tag to allow stacked tags

* make the complicated aggregation properly testable

* extra semantic corner cases I didn't think of

* tests for laggingOffsets

* a way to rerun queries if the laggingOffsets check reveals inconsistency

* if bookmark is ever different, we always have to rerun anyway

* boolean blindness

* incorporate laggingOffsets into fetchAndPersistBracket

* split fetchAndPersist from getTermination and clean up its arguments

* just compose functors

* add looping to fetchAndPersistBracket

* more mvo tests

* test unsyncedOffsets, too

* Lagginess collector

* supply more likely actual data with mvo tests; don't trust Java equals

* rework minimumViableOffsets to track sync states across template IDs

* extra note

* fix the tests to work against the stricter mvo

* move surrogatesToDomains call

* more tests for lagginess accumulator

* add changelog

CHANGELOG_BEGIN
- [JSON API] Under rare conditions, a multi-template query backed by database
  could have an ACS portion that doesn't match its transaction stream, if
  updated concurrently.  This conditions is now checked and accounted for.
  See `issue #10617 <https://github.com/digital-asset/daml/pull/10617>`__.
CHANGELOG_END

* port toSeq to Scala 2.12

* handle a corner case with offsets being too close to expected values

* didn't need XTag
2021-09-28 20:47:42 +00:00
tudor-da
3d779cfe12
[Mutable state cache] Fix initialization offset (#11024)
* The mutable state cache index is initialized to the current ledger end

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-28 22:32:19 +02:00
nickchapman-da
5458aa890c
Switch on NonUnitStatements warning in daml-lf/transaction (#11048)
* Add NonUnitStatements warning in daml-lf/transaction & fix using discard or similar.

CHANGELOG_BEGIN
CHANGELOG_END

* address comment

* prefer parens to braces for single statement map/foreach bodies
2021-09-28 15:34:31 +00:00
Gary Verhaegen
9641fd5f83
auth middleware: no print secret (#11050)
At the moment, the OAuth2 Middleware prints its config at startup.
Unfortunately that config includes a secret, which presumably we'd
prefer not to show.

CHANGELOG_BEGIN
- The OAuth2 Middleware now obfuscates its Client Secret when logging
  its config.
CHANGELOG_END
2021-09-28 17:00:01 +02:00