Commit Graph

7779 Commits

Author SHA1 Message Date
Gary Verhaegen
8cdeb7539a
make auth middleware aware that actAs => readAs (#10825)
It seems a bit silly that the auth middleware would refuse a token that
has an `actAs` when it needs a `readAs`.

Reference: https://docs.daml.com/app-dev/authorization.html#access-tokens-and-claims

CHANGELOG_BEGIN
- The auth middleware will now consider that actAs credentials are valid
  for readAs requests.
CHANGELOG_END
2021-09-09 11:03:55 +00:00
fabiotudone-da
e263d43aca
Ledger API test tool: test that record time is monotonically increasing [KVL-1053] (#10552)
* Test that record time is monotonically increasing

CHANGELOG_BEGIN
- [Ledger API Specification] Introduced a new conformance test about recort time in completions being monotonically increasing
CHANGELOG_END

* Address early review comments

* Fix unused import

* Simplify main assertion

* Add assertion about expected number of record times

* Print less record times in main assert's error message

* Make wartremover:Option2Iterable happy

* Use a submitting party

* operations -> submissions

* Create contracts rather than allocating parties

* Shorten comment

* Explicitly sort checkpoints by offset

* Shorten short identified

* Don't assume the ledger is at the beginning

* Simplify the test

* Make the new test optional

* Apply review suggestions
2021-09-09 10:26:24 +00:00
nicu-da
e5793a317b
Account for multi participant conformance tests when updating the config [KVL-1092] (#10788)
* Split command deduplication conformance tests into non-kv and kv tests.

Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.

CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END

* Run participant side/committer side deduplication tests based on the ledger implementation

* Try to update the time model on all the participants to account for multi participant setups.

When running with multi participants only one participant can update the configuration. Unfortunately to figure out which one exactly can do so it's not that easy so to make sure the configuration changes for our tests succeed we try to update it on all the participants.

CHANGELOG_BEGIN

CHANGELOG_END

* Fix on memory build file

* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT

* Rollback compat import

* Exclude command dedup from the reference test

* Exclude command deduplication from the reference export

* Format bazel

* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory

* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox

* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export

* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers

* Format bazel file

* Exclude CommandDeduplicationIT from sandbox

* Apply suggestions from code review

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>

* Add comments for command deduplication IT

* Rename to be consistent with scala namings

* Update formatting and comments

* Code review renames

* Add compat import

* Remove unused import

* Use a scala 2.12 compatible approach to receive all the configured participants in the conformance tests

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
2021-09-09 09:54:31 +00:00
Miklos
2e39d7b647
Deprecate Commands.deduplication_time field [KVL-1096] (#10802)
CHANGELOG_BEGIN
- [Ledger API Specification] `Commands.deduplication_time` field has been deprecated, please use `Commands.deduplication_duration` instead.
CHANGELOG_END
2021-09-09 11:30:17 +02:00
akshayshirahatti-da
7824201b8d
Get rid of extra new line and prune precision for all gatling values (#10820)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-09 08:37:50 +00:00
fabiotudone-da
a03f52a15e
Add ledger API tests about command deduplication information in completions [KVL-1057] (#10748)
* Add Ledger API Test Tool tests about command deduplication information present in completions

CHANGELOG_BEGIN
CHANGELOG_END

* Address review comments

* Fix offset reference and move new tests to a separate, optional suite

* Cover rejections as well

* Test both the command and command submission services

* Run new test suite (only) on sandbox-classic append-only and daml-on-sql

* Use the append-only schema with daml-on-sql for CommandDeduplicationInfoIT

* Fix tests except offset, successful completions only.

* Remove completion offset test as it's not supported by most ledgers

* Remove support for multiple submissions as rejection completions are not being tested for the moment

* Consolidate test cases for faster run

* Avoid forbidden characters in short identifiers

* Clarify assert

* Remove wrong test about deduplication time being preserved in completion

* Adhere to the Scala style guide

* Eliminate some code duplication

* fmt

* Make 3e404be compile

* Never assume a specific deduplication period format in completions

* Code cleanup

* Pass party to completionStreamRequest to avoid ILLEGAL_ARG failures

* Enable in append-only mode for all ledgers that support it

* Clarify that the new tests are append-only-only

* Update ledger/ledger-on-sql/BUILD.bazel

Fix suite name

* Update ledger/sandbox-classic/BUILD.bazel

Fix suite name

* fmt
2021-09-09 07:56:14 +00:00
nicu-da
61a07b1986
Add command line option max-deduplication-duration for sandbox and KV [KVL-1098] (#10816)
* Add command line option max-deduplication-duration which allows us to configure the max deduplication duration for sandbox and KV ledgers.

The main usage of this is specifying different max deduplication durations for tests.

CHANGELOG_BEGIN

CHANGELOG_END

* Code review changes
2021-09-08 16:41:56 +00:00
Moritz Kiefer
0800504955
Fix DuplicateSubmissionID test exclusions (#10819)
0db15e6759
renamed those tests so we need to adjust for it.

changelog_begin
changelog_end
2021-09-08 18:28:10 +02:00
Andreas Herrmann
1ff34ffbe8
Remove workaround for akka supervisor issue (#10818)
The workaround for https://github.com/akka/akka/issues/29841 is no
longer required as that issue has been fixed upstream and we have
updated to a version that includes the fix.

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-09-08 17:55:57 +02:00
Moritz Kiefer
6d347d4e88
Add daml interfaces to the Haskell LF AST (#10811)
* Add daml interfaces to the Haskell LF AST

Just copied from Sofia’s PR with no changes and stubbed all usages of
it that aren’t trivial.

changelog_begin
changelog_end
2021-09-08 15:48:02 +00:00
Moritz Kiefer
cecb27fe16
Stop using controller can syntax in skeleton template (#10799)
changelog_begin
changelog_end
2021-09-08 15:28:12 +02:00
Moritz Kiefer
8830eaa2ec
Drop controller can syntax from compiler tests (#10807)
You definitely want to review this with whitespace disabled.

changelog_begin
changelog_end
2021-09-08 15:28:09 +02:00
Miklos
9038a8051b
Update code documentation for participant-state API [KVL-1094] (#10812)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-08 10:37:54 +00:00
nicu-da
60ffb79fb1
Command deduplication - Participant side/committer side command deduplication conformance tests split [KVL-1093] (#10784)
* Split command deduplication conformance tests into non-kv and kv tests.

Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.

CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END

* Run participant side/committer side deduplication tests based on the ledger implementation

* Fix on memory build file

* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT

* Rollback compat import

* Exclude command dedup from the reference test

* Exclude command deduplication from the reference export

* Format bazel

* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory

* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox

* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export

* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers

* Format bazel file

* Exclude CommandDeduplicationIT from sandbox

* Apply suggestions from code review

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>

* Add comments for command deduplication IT

* Rename to be consistent with scala namings

* Update formatting and comments

* Add naming prefix for tests to avoid clashes

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
2021-09-08 10:27:07 +00:00
Moritz Kiefer
1e6540f8f2
Stop using controller _ can syntax in daml-lf tests (#10805)
If we want to kill this, as a first step let’s stop using it in tests.

changelog_begin
changelog_end
2021-09-08 11:49:46 +02:00
fabiotudone-da
0db15e6759
Ledger API test tool: ensure that test identifiers are unique (#10804)
* Ensure that test identifiers are unique

CHANGELOG_BEGIN
CHANGELOG_END

* Update ledger/ledger-api-test-tool/src/test/suite/scala/com/daml/ledger/api/testtool/NamesSpec.scala

Improve test description

* Improve test clarity and efficiency

* Uniform short IDs' naming convention for config and package mgmt. test cases
2021-09-08 08:50:35 +00:00
azure-pipelines[bot]
f1e378a60b
rotate release duty after 1.17.0-snapshot.20210907.7759.0.35a853fd (#10809)
@akrmn is taking care of 1.17.0-snapshot.20210907.7759.0.35a853fd (#10808), so they get pushed back to the end of the line.

Please do not merge this before #10808.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-09-08 10:28:57 +02:00
azure-pipelines[bot]
1464177bf8
release 1.17.0-snapshot.20210907.7759.0.35a853fd (#10808)
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.

@akrmn is in charge of this release.

Commit log:
```
35a853fd30 Async logging for http-json-perf tests and  publish logs as a pipeline artifact (#10803)
a349217f78 get oracle perf test output (#10800)
5f448f2603 Cleanup DataSource initialization (#10794)
0e250d02e3 Fix parallel ingestion initialization (#10797)
b8bd5e6399 interface PoC: protobuf definitions (#10796)
a1da025b78 FreePort draw from outside ephemeral port range (#10774)
945a86c3f2 Fix a race in a test of the RecoveringIndexer (#10792)
4093bbd58c fix macOS Bazel cache (#10795)
ed99fe5aa5 ParticipantPruningIT added events for canton safe-pruning and multi-node synchronization (#10775)
5b28aefbce Deprecate kvutils rejections generated only by participant state v1 [KVL-1090] (#10791)
116d6a5994 DPP-577 Use BIGINT instead of TIMESTAMP (#10761)
cdf4bf1138 Check compatibility of timestamps (#10793)
098a08c58b Add Moisés to release rotation (#10789)
7ae199f027 Follow-up to h2 storage backend user/password in url: Review feedback switch to regex and move to pure function (#10541)
a5999abf8e remove dead code (#10765)
4b7391b0d5 LedgerApiTestTool prints skipped (excluded) tests (#10726)
a0f0fb3684 [DPP-418] Enable outstanding test case for private key decryption (#10778)
d750666f8e Do not drop details when converting between gRPC `Status` classes [KVL-1084] (#10745)
5ec9a234db Fix link to the gRPC health checks in the driver for PostgreSQL docs (#10779)
8538fe97b4 HA protected ledger initialization for append only schema [DPP-551] (#10705)
23b0fe7ac2 Fix exclusions for command service tests (#10777)
c4e0a755d4 LF: drop ad-hoc ImmArray builders (#10763)
303ba90595 participant-state: Re-enable integration test for command deduplication [KVL-1089] (#10751)
793253ca87 bump cleanup threshold (#10771)
4d67684737 Fix flaky DBLock tests (#10770)
751587125f Fix exclusions for command dedup in compat tests (#10767)
50fecfb9ef Wrap missing label names in quotes (#10749)
b1c6e87803 fix claims check in auth middleware (#10768)
5595d55c79 [JSON-API] Use the token from incoming requests to update the package list (#10602)
dbafea0e48 Add unit tests for DBLockStorageBackend [DPP-495] (#10746)
e5a6d70182 Added buffer size metrics for getTransactions/getTransactionTrees (#10744)
f76c868ee4 update NOTICES file (#10762)
bb908d04fb Create a link to party management service in main Ledger API page. (#10138)
e45d852ed4 Fixes participant to do retries on startup as waiting for DB connectivity (#10759)
f68a12930b Report some oracle_json_perf numbers on slack  (#10754)
7270ee3c71 Handle dynamic port in auth middleware client (trigger service) (#10755)
ff1308ee3d [Docs] Add info on logs on Kubernetes & metrics in the ops section (#10525)
d26739087f Update buf image [kvl-1049] (#10752)
d2180cf60b Update exclusion for command deduplication to include full version range (#10750)
41881ba2ba Command dedup: migrate kvutils to use v2 services [KVL-1049] (#10679)
4525b8c265 [JSON-API] vanilla oracle_perf ci job (#10688)
16df8a5e35 [Short] Remove unused code (#10719)
b28afcf7ae [DPP-438] Update docs on metrics that no longer use <party_name> in their name (#10728)
0662025f4c Clarify what the `buf` image is and how it should be used (#10741)
9071a05c76 sandbox-classic: Reintroduce SqlLedger tests for the mutable index. (#10722)
f576cdfd06 fix flaky test in RecoveringIndexer (#9619)
f6a75b42f3 ledger-api-common: Do not mock final classes. (#10733)
e9c8af5024 Bump ghc-lib to include dropped parsing code for generic templates (#10735)
862a2901fb rotate release duty after 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10731)
12cb92b431 update compat versions for 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10737)
73c0de8db4 [DPP-578] Temporarily disable flaky test - Mockito plugin issue (#10734)
963bcb17eb release 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10730)
c11323ddb1 LF: Refactor engine test reinterpret (#10724)
a995fa3cd2 [DPP-574] Update docs about TLS: encrypted private key (#10727)
66970b7226 Not sharing the absolute Deadline object. (#10713)
c6c304b778 Improve script error on invalid script identifier format (#10702)
b748fd6b67 Support TLS in Daml Helper --json-api requests (#10709)
```
Changelog:
```
kvutils - protobuf rejections generated by the participant state v1 API are deprecated (Inconsistent, Disputed, ResourcesExhausted, PartyNotKnownOnLedger)
[Docs] Fixed link to the gRPC health checks in the driver for PostgreSQL docs

participant-state: Emit completions (CommandRejected) for duplicate commands when using pre-execution
- Fix a bug in the auth middleware where insufficient credentials could
  still give access to list of running triggers.
- [JSON API] The cli option `--access-token-file` is now deprecated. It
    is not needed anymore and you can safely remove it. Reason is that
    the operations which prior required a token at startup are now done
    on demand using the token of the incoming request.
- [Docs] Information was added in the `Operating Daml` section on how to aggregate logs on Kubernetes in conjuction with Daml services & what options exists for exporting metrics from daml services (not Kubernetes specific)

participant-state:
- Migrate to use only v2 Read/Write Services. This includes the use of new models for rejections/updates/submission results.
- Calculate deduplicate until for committer side deduplication based on the submitter given deduplication period
   -  if using the deprecated deduplicateUntil then just set the given timestamp
   -  if using duration then calculate the new deduplicateUntil by using the formula (submissionTime + deduplicationDuration + minSkew)
  -   if using offset deduplication then calculate deduplicateUntil by using the formula (submissionTime + maxDeduplicationDuration + minSkew)
  -  if the deduplication period is not set then we don't set deduplicateUntil
- Emit completions with status `ALREADY_EXISTS` for duplicate commands

- [Java Bindings] DamlLedgerClient.Builder allows to set a timeout for command using `withTimeout`.

- [Daml Assistant] The `daml ledger` commands now accepts `--tls` in
  combination with `--json-api` to access a JSON API behind a TLS
  reverse proxy.

```

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-09-08 10:27:20 +02:00
tudor-da
deb8a8b740
[Divulgence pruning] Daml SDK docs update [DPP-536] (#10764)
* [Divulgence pruning] Minor update to participant_pruning_service.proto divulgence docs

CHANGELOG_BEGIN
[Ledger API Specification] Participant pruning of all divulged contracts is fully implemented: Participant operators can choose to prune all immediately and retroactively divulged contracts, by setting the newly-added prune_all_divulged_contracts flag in the ParticipantPruningService/Prune request.
CHANGELOG_END

* Enrich `Daml Participant Pruning` in the `Operating Daml` docs section

* Apply suggestions from code review

Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>

Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>
2021-09-08 07:58:55 +00:00
nickchapman-da
8405ad58fd
Enhance InMemoryLedger to use the ValueEnricher (#10757)
* Enhance InMemoryLedger to use the ValueEnricher.

Allowing SandboxServer to run Engine with the default transactionNormalization=false

CHANGELOG_BEGIN
CHANGELOG_END

* enrich only when responding to verbose API queries
2021-09-08 08:47:47 +01:00
akshayshirahatti-da
35a853fd30
Async logging for http-json-perf tests and publish logs as a pipeline artifact (#10803)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-07 23:42:30 +01:00
Gary Verhaegen
a349217f78
get oracle perf test output (#10800)
We can only reference variables set in a named step.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-07 23:14:44 +02:00
Marton Nagy
5f448f2603
Cleanup DataSource initialization (#10794)
* Introduce VerifiedDataSource to capture async verification and factoring
* Move compatibility check to VerifiedDataSource
* Wire to FlywayMigrations
* Cleanup HikariDataSourceOwner

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-07 16:01:22 +00:00
Marton Nagy
0e250d02e3
Fix parallel ingestion initialization (#10797)
* by completing the Promise on failure track as well

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-07 15:23:58 +00:00
Robin Krom
b8bd5e6399
interface PoC: protobuf definitions (#10796)
* interface PoC: protobuf definitions

The protobuf changes for the Daml interface PoC.

CHANGELOG_BEGIN
CHANGELOG_END

* dummy interface decoder/encoder
2021-09-07 16:58:49 +02:00
Andreas Herrmann
a1da025b78
FreePort draw from outside ephemeral port range (#10774)
* Test case for LockedFreePort not colliding with port 0

changelog_begin
changelog_end

* Discover dynamic port range on Linux

* Random port generator outside ephemeral range

* remove dev comments

* Draw FreePort from outside the ephemeral port range

Note, there is a race condition between the socket being closed and the
lock-file being created in LockedFreePort. This is not a new issue, it
was already present with the previous port 0 based implementation.

LockedFreePort handles this by attempting to find a free port and taking
a file lock multiple times.

But, it could happen that A `find`s port N, and obtains the lock, but
doesn't bind port N again, yet; then B binds port N during `find`; then
A attempts to bind port N before B could release it again and fails
because B still holds it.

* Select dynamic port range based on OS

* Detect dynamic port range on MacOS and Windows

* Import sysctl from Nix on MacOS

changelog_begin
changelog_end

* Windows line separator

* FreePort helpers visibility

* Use more informative exception types

* Use a more light weight unit test

* Add comments

* Fix Windows

* Update libs-scala/ports/src/main/scala/com/digitalasset/ports/FreePort.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update libs-scala/ports/src/main/scala/com/digitalasset/ports/FreePort.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Add a comment to clarify the generated port range

* fmt

* unused import

* Split libs-scala/ports

Splits the FreePort and LockedFreePort components into a separate
library as this is only used for testing purposes.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2021-09-07 15:41:43 +02:00
mziolekda
945a86c3f2
Fix a race in a test of the RecoveringIndexer (#10792)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-07 14:36:56 +02:00
Gary Verhaegen
4093bbd58c
fix macOS Bazel cache (#10795)
macOS filesystems have been case-insensitive by default for years, and
in particular our laptops are, so if we want the cache to work as
expected, CI should be too.

Note: this does not apply to Nix, because the Nix partition is a
case-sensitive image per @nycnewman's script on laptops too.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-07 13:31:57 +02:00
Oliver Seeliger
ed99fe5aa5
ParticipantPruningIT added events for canton safe-pruning and multi-node synchronization (#10775)
* Hardening ParticipantPruningIT divulgence tests to not flake on canton-enterprise

worked out with @tudor-da

CHANGELOG_BEGIN
CHANGELOG_END

* Also feed other participant events

* Switch canton conformance tests to parallel indexer

* More stability fixes
2021-09-07 12:41:43 +02:00
nicu-da
5b28aefbce
Deprecate kvutils rejections generated only by participant state v1 [KVL-1090] (#10791)
* Deprecated protobuf DamlTransactionRejection reasons which are generated by the v1 participant state API only.

CHANGELOG_BEGIN
kvutils - protobuf rejections generated by the participant state v1 API are deprecated (Inconsistent, Disputed, ResourcesExhausted, PartyNotKnownOnLedger)
CHANGELOG_END

* Reorganize proto fields in the order of the  field number
2021-09-07 08:39:00 +00:00
Robert Autenrieth
116d6a5994
DPP-577 Use BIGINT instead of TIMESTAMP (#10761)
* Use bigint instead of timestamp

changelog_begin
changelog_end

* Add tests

* Fix deduplicateUntil

* Fix some places that still use timestamp

* Fix some places that still use timestamp

* DbDto uses only Long instead of Instant values

* Fix some places that still use Timestamp

* Remove unused classes

* Fix some places that still use Timestamp
2021-09-07 10:00:13 +02:00
Robert Autenrieth
cdf4bf1138
Check compatibility of timestamps (#10793)
changelog_begin
changelog_end
2021-09-07 08:58:42 +02:00
Moritz Kiefer
098a08c58b
Add Moisés to release rotation (#10789)
* Add Moisés to release rotation

We do weekly snapshot releases and rotate who does the release and
tests it. This is all described in
https://github.com/digital-asset/daml/blob/main/release/RELEASE.md but
feel free to reach if you need help or anything is unclear.

changelog_begin
changelog_end

* switch gh accounts

changelog_begin
changelog_end
2021-09-06 16:59:41 +00:00
Oliver Seeliger
7ae199f027
Follow-up to h2 storage backend user/password in url: Review feedback switch to regex and move to pure function (#10541)
* Review feedback switch to regex and move to pure function

changelog_begin
changelog_end

* Review feedback

* Attempt at making scala 2.12 test-compile
2021-09-06 17:24:17 +02:00
nickchapman-da
a5999abf8e
remove dead code (#10765)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-06 13:35:37 +00:00
tudor-da
4b7391b0d5
LedgerApiTestTool prints skipped (excluded) tests (#10726)
* Document multi-participant-only pruning test

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-06 13:50:54 +02:00
pbatko-da
a0f0fb3684
[DPP-418] Enable outstanding test case for private key decryption (#10778)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-06 13:25:38 +02:00
Hubert Slojewski
d750666f8e
Do not drop details when converting between gRPC Status classes [KVL-1084] (#10745)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-06 12:22:15 +02:00
Stefano Baghino
5ec9a234db
Fix link to the gRPC health checks in the driver for PostgreSQL docs (#10779)
Fixes #10776

changelog_begin
[Docs] Fixed link to the gRPC health checks in the driver for PostgreSQL docs
changelog_end
2021-09-06 11:36:33 +02:00
Marton Nagy
8538fe97b4
HA protected ledger initialization for append only schema [DPP-551] (#10705)
* HA protected ledger initialization for the appendOnly [DPP-551]

* Preparation - Refactoring ParallelIndexerFactory: pulling initialization, resource factory and subscription apart

CHANGELOG_BEGIN
CHANGELOG_END

* Preparation - Refactoring: untangle JdbcIndexer and FlywayMigrations

CHANGELOG_BEGIN
CHANGELOG_END

* Preparation - Refactoring: simplifying Indexer type

CHANGELOG_BEGIN
CHANGELOG_END

* Preparation - Refactoring: simplifying initialized JdbcIndexer type

CHANGELOG_BEGIN
CHANGELOG_END

* Move initialization code to InitializeParallelIngestion (which does initialization in a HA protected zone)

CHANGELOG_BEGIN
CHANGELOG_END

* Fixes mutable indexer initialization

* Indexer initializes for every RecoveringIndexer restart now
* Also fixes reset: it happend only once, at the beginning

CHANGELOG_BEGIN
CHANGELOG_END

* Minor changes based on review

CHANGELOG_BEGIN
CHANGELOG_END

* Small change in mutable initialization based on review

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-06 09:02:33 +00:00
Moritz Kiefer
23b0fe7ac2
Fix exclusions for command service tests (#10777)
changelog_begin
changelog_end
2021-09-06 10:54:43 +02:00
Remy
c4e0a755d4
LF: drop ad-hoc ImmArray builders (#10763)
Since we switch to scala 2.13, ImmArray companion object extends
`Factory`. Hence:

- the `apply` methods of `ImmArray` override the one from `Factory`

- we can use the notation `.to(ImmArray)` to convert an `Iterable` to
  `ImmArray`

This PR drops those `apply` ImmArray. Conversion from Iterable to
`ImmArray` should use the `.to(ImmArray)`.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-04 16:10:07 +02:00
nicu-da
303ba90595
participant-state: Re-enable integration test for command deduplication [KVL-1089] (#10751)
* Re-write command deduplication integration test for participant state to be valid for the deduplication changes.

The test now accounts for completions being generated for duplicate commands.

CHANGELOG_BEGIN

CHANGELOG_END

* Update check for successfull transaction

* Emit command rejected completions for duplicate commands rejections when using pre-execution

CHANGELOG_BEGIN
participant-state: Emit completions (CommandRejected) for duplicate commands when using pre-execution
CHANGELOG_END

* Update test for emitted duplicate command update

* Added test for new case.

* Revert redundant test case.

This reverts commit 8ef4364f

* Test corner case of dropping not deduplicated transaction rejection.

* Code tidying.

* Reformatted.

Co-authored-by: Miklos Erdelyi <miklos.erdelyi@digitalasset.com>
2021-09-03 17:48:54 +00:00
Gary Verhaegen
793253ca87
bump cleanup threshold (#10771)
I've witnessed a build ([link], though that will likely expire soon)
that failed with a "No space left on device" error after skipping the
cleanup step because the machine still had 68GB free.

[link]: https://dev.azure.com/digitalasset/daml/_build/results?buildId=87591&view=logs&j=870bb40c-6da0-5bff-67ed-547f10fa97f2&t=deecee86-545a-596e-8b0d-fb7d606fe9f2

With the machines only having 200GB disk size total, cleaning up at 80
is probably going to start hampering the overall efficiency of the
cache. It may be time to think about increasing the disk size itself (or
finding ways to reduce the size requirements of our builds). Important
note, though: we can't actually increase the macOS disk size very much.

The failure happened on the `compatibility_linux` job.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-03 18:22:18 +02:00
Marton Nagy
4d67684737
Fix flaky DBLock tests (#10770)
As it turned out normal connection closeing does not imply released resources on the database side

* use eventually for testing connection lost cases
* refactor a case to use proper release instead of connection close
* increases a timeout to lower risk of flakyness

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-03 14:36:40 +00:00
Moritz Kiefer
751587125f
Fix exclusions for command dedup in compat tests (#10767)
We need to differentiate between 3 cases:

1. Ledger API test tool versions that only support exclusion of whole
suites.
2. Ledger API test tool versions where the tests had different names.
3. and the current state

changelog_begin
changelog_end
2021-09-03 14:04:46 +00:00
Moritz Kiefer
50fecfb9ef
Wrap missing label names in quotes (#10749)
* Wrap missing label names in quotes

See
https://discuss.daml.com/t/why-might-i-be-getting-missing-record-label-identity-from-the-ledger-api-when-submitting-a-command/3059/5
this has caused some confusion for users since it’s not obvious that
it is the label name if you have a very generic label name like
`identity`.

I have no strong feelings on whether this should be single or double
quotes so happy to change it.

changelog_begin
changelog_end
2021-09-03 13:52:55 +00:00
Gary Verhaegen
b1c6e87803
fix claims check in auth middleware (#10768)
The claims check in the auth middleware was switched around: in effect,
it checked that we did not receive _more_ than we asked for, rather than
checking we receive _at least_ what we asked for.

Of course this would still not let anyone run any trigger without the
proper access token, but it would let people list running triggers and
request (or stop) trigger executions.

CHANGELOG_BEGIN
- Fix a bug in the auth middleware where insufficient credentials could
  still give access to list of running triggers.
CHANGELOG_END
2021-09-03 13:31:26 +00:00
Victor Peter Rouven Müller
5595d55c79
[JSON-API] Use the token from incoming requests to update the package list (#10602)
* Use the token from incoming requests to update the package list

changelog_begin
changelog_end

* Lazily initialize the ledger client

* Fix ee integration tests

* Fix package reloading behaviour by using a semaphore to check for ongoing updates

* Refactor out the semaphore code into a concurrency utility class

* Use correct locking for the updateTask so every thread always uses an up to date task

* Remove unused imports in utils.Concurrent & remove packages from the tests

* Hide & mark the token file cli option deprecated because we dont need it anymore and only keep it so client deployment code doesn't break

* Fix scala 2.12 build by adding more type annotations

* Update ledger-service/http-json-cli/src/main/scala/com/daml/http/OptionParser.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/PackageService.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Readd pgkManagementClient after it was removed accidentally (but now it's lazy)

* Remove concurrent object & use atomic boolean instead of a mutex because it makes more sense

* Replace semaphore with countdownlatch

* Refactor the caching into a separate class

* Use Instant instead of LocalDateTime

* Remove that ** of bad synchonization and do stupid simple synchronization because it JUST WORKS, besides adapt when we want to reload

* Remove await in tests because it can result in buggy tests

* remove unused code in WebSocketService.scala

* Unhide the access-token-file option as per request of Stefano

* Less implicit jwts per request of Stefano

* Try making some code more readable as by request of Akshay

* Use more shark because it expresses better than flatMaps if I don't need the arg

* Move defs in predicate in WebsocketService.scala around

* Try to minimize diff further in WebsocketService.scala

* Fix build and minimize diff in WebSocketService.scala further

* Minimize diff of function getTransactionSourceForParty in WebSocketService.scala

* Share the ec in WebSocketService.scala to minimize the diff

* Minimize in function predicate in WebSocketService.scala

* Further minimize in function predicate in WebSocketService.scala

* Change some case classes to be normal classes but with apply method

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/PackageService.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Get rid of implicit jwt tokens, the world is already confusing and full of implicits enough

* Improve readability

* Integrate the new LedgerClient which does not depend on a leder id

* Fix tests

* Apply suggestions from code review

thanks to @S11001001

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Apply further review comments

* Remove outcommented code

* Deprecate access token file option in the description too

changelog_begin

- [JSON API] The cli option `--access-token-file` is now deprecated. It
    is not needed anymore and you can safely remove it. Reason is that
    the operations which prior required a token at startup are now done
    on demand using the token of the incoming request.

changelog_end

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2021-09-03 10:12:01 +00:00
Marton Nagy
dbafea0e48
Add unit tests for DBLockStorageBackend [DPP-495] (#10746)
* Add unit tests for DBLockStorageBackend

* Add tests
* Fixes minor bug in Oracle implementation
* Adds inside mixin to StorageBackendPostgresSpec for convenience

CHANGELOG_BEGIN
CHANGELOG_END

* Apply suggestions from code review

Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>

Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>
2021-09-03 09:18:50 +00:00