* interfaces: protobuf encoder haskell side
This is the implementation of the protobuf encoder on the haskell side.
CHANGELOG_BEGIN
CHANGELOG_END
* missing decoder cases for added interface data constructor
* Add conformance test for KV committer usage of max deduplication duration as deduplication duration
CHANGELOG_BEGIN
CHANGELOG_END
* Run AppendOnlyKVCommandDeduplicationIT for KV ledgers that use the append-only schema
This only handles the interface definition, not the implementation in
the template. There are also a few rough edges:
1. It maks all choices as consuming.
2. it ignores locations
But for a poc that doesn’t seem too bad.
The tests don’t do anything super useful since the typechecker falls
over but I checked tha tthe generated LF looks more or less reasonable.
changelog_begin
changelog_end
On Sept 8 our ES cluster became unresponsive. I tried connecting to the
machines.
One machine had an ES Docker container that claimed to have started 7
weeks ago and stopped 5 weeks ago, while the machine's own uptime was 5
weeks. I assume GCP had decided to restart it for some reason. The init
script had failed on missing a TTY, hence the addition of the
`DEBIAN_FRONTEND` env var.
Two machines had a Docker container that had stopped on that day, resp.
6h and 2h before I started investigating. It wasn't immediately clear
what had caused the containers to stop.
On all three of these machines, I was abble to manually restart the
containers and they were abble to reform a cluster, though the state of
the cluster was red (missing shards).
The last two machines simply did not respond to SSH connection attempts.
Assuming it might help, I decided to try to restart the machines. As GCP
does not allow restarting individual machines when they're part of a
managed instance roup, I tried clicking the "rolling restart" button
on the GCP console, which seemed like it would restart the machines. I
carefully selected "restart" (and not "replace"), started the process,
and watched GCP proceed to immediately replace all five machines, losing
all data in the process.
I then started a new cluster and used bigger (and more) machines to
reingest all of the data, and then fell back to the existing
configuration for the "steady" state. I'll try to keep a better eye on
the state of the cluster from now on. In particular, we should not have
a node down for 5 weeks without noticing.
I'll also try to find some time to look into backing up the Kibana
configuration, as that's the one thing we can't just reingest at the
moment.
CHANGELOG_BEGIN
CHANGELOG_END
* Always use max_deduplication_duration as deduplication period for committer side deduplication
Doing this we get the following guarantees:
- Deduplication period is constant for all the submissions therefore this guarantees consistency between "backward-looking" and "forward-looking" deduplication
CHANGELOG_BEGIN
kvutils - committer side deduplication always uses max_deduplication_duration + min_skew as a deduplication period for all the requests.
CHANGELOG_END
* Set max_deduplication_duration to 10 seconds for all the conformance tests
* Update KV command deduplication conformance tests to account for max deduplication duration used as deduplication period for all the requests
* Add max deduplication wait for multi-participant conformance test
* Overwrite the submitter info deduplication period in the transaction committer so that we always set the max deduplication duration
* Update ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/participant/state/kvutils/committer/transaction/TransactionCommitterSpec.scala
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Add separate step for overwriting the deduplication period
* Code review - reformatting and renames
* Add comment for setting the deduplication duration
* [docs] Replace AdoptOpenJDK suggestion by Adoptium
In the installation instructions, suggest Adoptium as JDK source
[AdoptOpenJDK has moved to the Eclipse foundation](
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/)
* Without changelog entry
CHANGELOG_BEGIN
CHANGELOG_END
Manual release process. @stefanobaghino-da is in charge of this release.
Commit log (since 1.17.0-snapshot.20210907.7759.0.35a853fd):
```
976ca400eb Force JSON API to refresh packages for GET requests to /v1/query (#10835)
e42cd3a1c6 LF: add interface to scala AST and decoder (#10830)
07b72d0f3e Update Ledger API's proto documentation [KVL-1094] (#10815)
648acf52f8 Fix typo in app-arch.rst (#10798)
dc9fbb5aa9 Stringify errors in create-daml-app before calling alert (#10831)
90b008ef0d Stop using `controller … can` syntax in ledger tests (#10814)
e490b69471 Clarifications of code ownership in /ledger and /ledger-api (#10742)
8cdeb7539a make auth middleware aware that actAs => readAs (#10825)
e263d43aca Ledger API test tool: test that record time is monotonically increasing [KVL-1053] (#10552)
e5793a317b Account for multi participant conformance tests when updating the config [KVL-1092] (#10788)
2e39d7b647 Deprecate `Commands.deduplication_time` field [KVL-1096] (#10802)
7824201b8d Get rid of extra new line and prune precision for all gatling values (#10820)
a03f52a15e Add ledger API tests about command deduplication information in completions [KVL-1057] (#10748)
61a07b1986 Add command line option max-deduplication-duration for sandbox and KV [KVL-1098] (#10816)
0800504955 Fix DuplicateSubmissionID test exclusions (#10819)
1ff34ffbe8 Remove workaround for akka supervisor issue (#10818)
6d347d4e88 Add daml interfaces to the Haskell LF AST (#10811)
cecb27fe16 Stop using controller can syntax in skeleton template (#10799)
8830eaa2ec Drop controller can syntax from compiler tests (#10807)
9038a8051b Update code documentation for `participant-state` API [KVL-1094] (#10812)
60ffb79fb1 Command deduplication - Participant side/committer side command deduplication conformance tests split [KVL-1093] (#10784)
1e6540f8f2 Stop using `controller _ can` syntax in daml-lf tests (#10805)
0db15e6759 Ledger API test tool: ensure that test identifiers are unique (#10804)
f1e378a60b rotate release duty after 1.17.0-snapshot.20210907.7759.0.35a853fd (#10809)
1464177bf8 release 1.17.0-snapshot.20210907.7759.0.35a853fd (#10808)
deb8a8b740 [Divulgence pruning] Daml SDK docs update [DPP-536] (#10764)
8405ad58fd Enhance InMemoryLedger to use the ValueEnricher (#10757)
```
Changelog (since 1.17.0-snapshot.20210907.7759.0.35a853fd):
```
- [JSON API] Fixed a bug that prevented the JSON API to be aware of
packages uploaded directly via the Ledger API.
- The auth middleware will now consider that actAs credentials are valid
for readAs requests.
- [Ledger API Specification] Introduced a new conformance test about recort time in completions being monotonically increasing
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
- [Ledger API Specification] `Commands.deduplication_time` field has been deprecated, please use `Commands.deduplication_duration` instead.
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
[Ledger API Specification] Participant pruning of all divulged contracts is fully implemented: Participant operators can choose to prune all immediately and retroactively divulged contracts, by setting the newly-added prune_all_divulged_contracts flag in the ParticipantPruningService/Prune request.
```
changelog_begin
changelog_end
* Initial changes to add a surrogate_template_id cache to reduce db queries
CHANGELOG_BEGIN
CHANGELOG_END
* refactoring and addition of tests
* Code review based changes to use Contextual Logger and json-api metrics instance
* make max cache entries/size configurable
* Rename cache max entries default variable
* Add failing test that covers the bug we found in #10823
* Fix /v1/query endpoint bug
changelog_begin
- [JSON API] Fixed a bug that prevented the JSON API to be aware of
packages uploaded directly via the Ledger API.
changelog_end
* Stop using `controller … can` syntax in ledger tests
We’re planning to deprecate that in SDK 2.0 to reduce the confusion
around the implicit observer behavior so in preparation for that, this
PR drops the syntax from all ledger tests.
changelog_begin
changelog_end
* shuffle let above usage
changelog_begin
changelog_end
It seems a bit silly that the auth middleware would refuse a token that
has an `actAs` when it needs a `readAs`.
Reference: https://docs.daml.com/app-dev/authorization.html#access-tokens-and-claims
CHANGELOG_BEGIN
- The auth middleware will now consider that actAs credentials are valid
for readAs requests.
CHANGELOG_END
* Test that record time is monotonically increasing
CHANGELOG_BEGIN
- [Ledger API Specification] Introduced a new conformance test about recort time in completions being monotonically increasing
CHANGELOG_END
* Address early review comments
* Fix unused import
* Simplify main assertion
* Add assertion about expected number of record times
* Print less record times in main assert's error message
* Make wartremover:Option2Iterable happy
* Use a submitting party
* operations -> submissions
* Create contracts rather than allocating parties
* Shorten comment
* Explicitly sort checkpoints by offset
* Shorten short identified
* Don't assume the ledger is at the beginning
* Simplify the test
* Make the new test optional
* Apply review suggestions
* Split command deduplication conformance tests into non-kv and kv tests.
Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.
CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END
* Run participant side/committer side deduplication tests based on the ledger implementation
* Try to update the time model on all the participants to account for multi participant setups.
When running with multi participants only one participant can update the configuration. Unfortunately to figure out which one exactly can do so it's not that easy so to make sure the configuration changes for our tests succeed we try to update it on all the participants.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix on memory build file
* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT
* Rollback compat import
* Exclude command dedup from the reference test
* Exclude command deduplication from the reference export
* Format bazel
* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory
* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox
* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export
* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers
* Format bazel file
* Exclude CommandDeduplicationIT from sandbox
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add comments for command deduplication IT
* Rename to be consistent with scala namings
* Update formatting and comments
* Code review renames
* Add compat import
* Remove unused import
* Use a scala 2.12 compatible approach to receive all the configured participants in the conformance tests
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
CHANGELOG_BEGIN
- [Ledger API Specification] `Commands.deduplication_time` field has been deprecated, please use `Commands.deduplication_duration` instead.
CHANGELOG_END
* Add Ledger API Test Tool tests about command deduplication information present in completions
CHANGELOG_BEGIN
CHANGELOG_END
* Address review comments
* Fix offset reference and move new tests to a separate, optional suite
* Cover rejections as well
* Test both the command and command submission services
* Run new test suite (only) on sandbox-classic append-only and daml-on-sql
* Use the append-only schema with daml-on-sql for CommandDeduplicationInfoIT
* Fix tests except offset, successful completions only.
* Remove completion offset test as it's not supported by most ledgers
* Remove support for multiple submissions as rejection completions are not being tested for the moment
* Consolidate test cases for faster run
* Avoid forbidden characters in short identifiers
* Clarify assert
* Remove wrong test about deduplication time being preserved in completion
* Adhere to the Scala style guide
* Eliminate some code duplication
* fmt
* Make 3e404be compile
* Never assume a specific deduplication period format in completions
* Code cleanup
* Pass party to completionStreamRequest to avoid ILLEGAL_ARG failures
* Enable in append-only mode for all ledgers that support it
* Clarify that the new tests are append-only-only
* Update ledger/ledger-on-sql/BUILD.bazel
Fix suite name
* Update ledger/sandbox-classic/BUILD.bazel
Fix suite name
* fmt
* Add command line option max-deduplication-duration which allows us to configure the max deduplication duration for sandbox and KV ledgers.
The main usage of this is specifying different max deduplication durations for tests.
CHANGELOG_BEGIN
CHANGELOG_END
* Code review changes
The workaround for https://github.com/akka/akka/issues/29841 is no
longer required as that issue has been fixed upstream and we have
updated to a version that includes the fix.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Add daml interfaces to the Haskell LF AST
Just copied from Sofia’s PR with no changes and stubbed all usages of
it that aren’t trivial.
changelog_begin
changelog_end
* Split command deduplication conformance tests into non-kv and kv tests.
Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.
CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END
* Run participant side/committer side deduplication tests based on the ledger implementation
* Fix on memory build file
* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT
* Rollback compat import
* Exclude command dedup from the reference test
* Exclude command deduplication from the reference export
* Format bazel
* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory
* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox
* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export
* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers
* Format bazel file
* Exclude CommandDeduplicationIT from sandbox
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add comments for command deduplication IT
* Rename to be consistent with scala namings
* Update formatting and comments
* Add naming prefix for tests to avoid clashes
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Ensure that test identifiers are unique
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/ledger-api-test-tool/src/test/suite/scala/com/daml/ledger/api/testtool/NamesSpec.scala
Improve test description
* Improve test clarity and efficiency
* Uniform short IDs' naming convention for config and package mgmt. test cases
@akrmn is taking care of 1.17.0-snapshot.20210907.7759.0.35a853fd (#10808), so they get pushed back to the end of the line.
Please do not merge this before #10808.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.
@akrmn is in charge of this release.
Commit log:
```
35a853fd30 Async logging for http-json-perf tests and publish logs as a pipeline artifact (#10803)
a349217f78 get oracle perf test output (#10800)
5f448f2603 Cleanup DataSource initialization (#10794)
0e250d02e3 Fix parallel ingestion initialization (#10797)
b8bd5e6399 interface PoC: protobuf definitions (#10796)
a1da025b78 FreePort draw from outside ephemeral port range (#10774)
945a86c3f2 Fix a race in a test of the RecoveringIndexer (#10792)
4093bbd58c fix macOS Bazel cache (#10795)
ed99fe5aa5 ParticipantPruningIT added events for canton safe-pruning and multi-node synchronization (#10775)
5b28aefbce Deprecate kvutils rejections generated only by participant state v1 [KVL-1090] (#10791)
116d6a5994 DPP-577 Use BIGINT instead of TIMESTAMP (#10761)
cdf4bf1138 Check compatibility of timestamps (#10793)
098a08c58b Add Moisés to release rotation (#10789)
7ae199f027 Follow-up to h2 storage backend user/password in url: Review feedback switch to regex and move to pure function (#10541)
a5999abf8e remove dead code (#10765)
4b7391b0d5 LedgerApiTestTool prints skipped (excluded) tests (#10726)
a0f0fb3684 [DPP-418] Enable outstanding test case for private key decryption (#10778)
d750666f8e Do not drop details when converting between gRPC `Status` classes [KVL-1084] (#10745)
5ec9a234db Fix link to the gRPC health checks in the driver for PostgreSQL docs (#10779)
8538fe97b4 HA protected ledger initialization for append only schema [DPP-551] (#10705)
23b0fe7ac2 Fix exclusions for command service tests (#10777)
c4e0a755d4 LF: drop ad-hoc ImmArray builders (#10763)
303ba90595 participant-state: Re-enable integration test for command deduplication [KVL-1089] (#10751)
793253ca87 bump cleanup threshold (#10771)
4d67684737 Fix flaky DBLock tests (#10770)
751587125f Fix exclusions for command dedup in compat tests (#10767)
50fecfb9ef Wrap missing label names in quotes (#10749)
b1c6e87803 fix claims check in auth middleware (#10768)
5595d55c79 [JSON-API] Use the token from incoming requests to update the package list (#10602)
dbafea0e48 Add unit tests for DBLockStorageBackend [DPP-495] (#10746)
e5a6d70182 Added buffer size metrics for getTransactions/getTransactionTrees (#10744)
f76c868ee4 update NOTICES file (#10762)
bb908d04fb Create a link to party management service in main Ledger API page. (#10138)
e45d852ed4 Fixes participant to do retries on startup as waiting for DB connectivity (#10759)
f68a12930b Report some oracle_json_perf numbers on slack (#10754)
7270ee3c71 Handle dynamic port in auth middleware client (trigger service) (#10755)
ff1308ee3d [Docs] Add info on logs on Kubernetes & metrics in the ops section (#10525)
d26739087f Update buf image [kvl-1049] (#10752)
d2180cf60b Update exclusion for command deduplication to include full version range (#10750)
41881ba2ba Command dedup: migrate kvutils to use v2 services [KVL-1049] (#10679)
4525b8c265 [JSON-API] vanilla oracle_perf ci job (#10688)
16df8a5e35 [Short] Remove unused code (#10719)
b28afcf7ae [DPP-438] Update docs on metrics that no longer use <party_name> in their name (#10728)
0662025f4c Clarify what the `buf` image is and how it should be used (#10741)
9071a05c76 sandbox-classic: Reintroduce SqlLedger tests for the mutable index. (#10722)
f576cdfd06 fix flaky test in RecoveringIndexer (#9619)
f6a75b42f3 ledger-api-common: Do not mock final classes. (#10733)
e9c8af5024 Bump ghc-lib to include dropped parsing code for generic templates (#10735)
862a2901fb rotate release duty after 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10731)
12cb92b431 update compat versions for 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10737)
73c0de8db4 [DPP-578] Temporarily disable flaky test - Mockito plugin issue (#10734)
963bcb17eb release 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10730)
c11323ddb1 LF: Refactor engine test reinterpret (#10724)
a995fa3cd2 [DPP-574] Update docs about TLS: encrypted private key (#10727)
66970b7226 Not sharing the absolute Deadline object. (#10713)
c6c304b778 Improve script error on invalid script identifier format (#10702)
b748fd6b67 Support TLS in Daml Helper --json-api requests (#10709)
```
Changelog:
```
kvutils - protobuf rejections generated by the participant state v1 API are deprecated (Inconsistent, Disputed, ResourcesExhausted, PartyNotKnownOnLedger)
[Docs] Fixed link to the gRPC health checks in the driver for PostgreSQL docs
participant-state: Emit completions (CommandRejected) for duplicate commands when using pre-execution
- Fix a bug in the auth middleware where insufficient credentials could
still give access to list of running triggers.
- [JSON API] The cli option `--access-token-file` is now deprecated. It
is not needed anymore and you can safely remove it. Reason is that
the operations which prior required a token at startup are now done
on demand using the token of the incoming request.
- [Docs] Information was added in the `Operating Daml` section on how to aggregate logs on Kubernetes in conjuction with Daml services & what options exists for exporting metrics from daml services (not Kubernetes specific)
participant-state:
- Migrate to use only v2 Read/Write Services. This includes the use of new models for rejections/updates/submission results.
- Calculate deduplicate until for committer side deduplication based on the submitter given deduplication period
- if using the deprecated deduplicateUntil then just set the given timestamp
- if using duration then calculate the new deduplicateUntil by using the formula (submissionTime + deduplicationDuration + minSkew)
- if using offset deduplication then calculate deduplicateUntil by using the formula (submissionTime + maxDeduplicationDuration + minSkew)
- if the deduplication period is not set then we don't set deduplicateUntil
- Emit completions with status `ALREADY_EXISTS` for duplicate commands
- [Java Bindings] DamlLedgerClient.Builder allows to set a timeout for command using `withTimeout`.
- [Daml Assistant] The `daml ledger` commands now accepts `--tls` in
combination with `--json-api` to access a JSON API behind a TLS
reverse proxy.
```
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* [Divulgence pruning] Minor update to participant_pruning_service.proto divulgence docs
CHANGELOG_BEGIN
[Ledger API Specification] Participant pruning of all divulged contracts is fully implemented: Participant operators can choose to prune all immediately and retroactively divulged contracts, by setting the newly-added prune_all_divulged_contracts flag in the ParticipantPruningService/Prune request.
CHANGELOG_END
* Enrich `Daml Participant Pruning` in the `Operating Daml` docs section
* Apply suggestions from code review
Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>
Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>
* Enhance InMemoryLedger to use the ValueEnricher.
Allowing SandboxServer to run Engine with the default transactionNormalization=false
CHANGELOG_BEGIN
CHANGELOG_END
* enrich only when responding to verbose API queries
