* Support adding tests as an hidden option
* Simplify existing suites
CHANGELOG_BEGIN
CHANGELOG_END
* Remove stale conformance suites from build.yml
* `--add` -> ``--additional`
* Re-add `--all-tests` as deprecated CLI option to be tested
* Move sandbox-classic pruning test to wall clock again
* Run KVCommandDeduplicationIT for sandbox append-only
* Tidy-up
* Also add participant pruning test to ledger-on-memory/single-participant
* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only
* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only
* Add KVCommandDeduplicationIT to ledger-on-memory append-only
* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant
* Tidy-up
* Use KVCommandDeduplicationIT for sandbox-on-x too
* Fix merge
Add max dedup duration arg to all the test suites that include command dedup tests
* Make `--include` and `--additional` mutually exclusive
* Uniform formatting of multi-line strings
* Move exclusions after additions as they are applied last
* Re-disable deduplication test on sandbox with static time
* Re-disable deduplication test on sandbox-on-x
* [Ledger API error codes] ErrorCode interfaces and generator
* Implementation of annotation processor
* Implementation of DocItem generator
* Unit testing of the generator and error code logging
CHANGELOG_BEGIN
CHANGELOG_END
* Addressed review comments
- Add support for specifying either 1.2 or 1.3 as minimum TLS versions for ledger api server.
- Log enabled protocols (~TLS versions) and cipher suites at server and client startup.
- Add integration tests against Sandbox-classic and Sandbox
CHANGELOG_BEGIN
Sandbox: Add CLI flag to select minimum enabled TLS version for ledger API server.
CHANGELOG_END
* Minor tweaks.
* Use deduplication period instead of deduplication time.
* Introduced change ID and consistent use of deduplication duration/period.
* Consistent use of deduplication duration/period.
* Added ALREADY_EXISTS to the described gRPC errors.
* Minor tweak.
CHANGELOG_BEGIN
CHANGELOG_END
* Emphasize that applications should not change the deduplication time upon a resubmission
* Fix definition of change ID and always refer to it
* Try to clarify "Application-specific IDs"
* Make command deduplication explanation unspecific w.r.t. direction
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* Extract deduplication "features" into a configuration to be used around the tests.
Better naming for assertions that support sync and async deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Fix broken test and use consistency for tests
As explained in #10853, we recently lost our ES cluster. While I'm not
planning on trusting Google's "rolling restart" feature ever again, we
can't exclude the possibility of future similar outages (without a
significant investment in the cluster, which I don't think we want to
do).
Losing the cluster is not a huge issue as we can always reingest the
data. Worst case we lose visibility for a few days. At least, as far as
the bazel logs are concerned.
Losing the Kibana data is a lot more annoying, as that is not derived
data and thus cannot be reingested. This PR aims to add a backup
mechanism for our Kibana configuration.
CHANGELOG_BEGIN
CHANGELOG_END
* Update daml documentation to reflect the deprecation of the package/internal access token
CHANGELOG_BEGIN
CHANGELOG_END
* drop the whole internal access token section
Following #10763, we drop the ad-hoc builders for `FrontStack`.
* Building a `Fronstack` from individuals elements should be done with
standard scala buidler.
* Building a `Fronstack` from a `TraversableOne` should be done with
the scala 2.13 `.to(FrontStack)` methd
* Building a `Fronstack` from a `ImmArray` should be done with the
`toImmArray` method.
CHANGELOG_BEGIN
CHANGELOG_END
* interfaces: protobuf encoder haskell side
This is the implementation of the protobuf encoder on the haskell side.
CHANGELOG_BEGIN
CHANGELOG_END
* missing decoder cases for added interface data constructor
* Add conformance test for KV committer usage of max deduplication duration as deduplication duration
CHANGELOG_BEGIN
CHANGELOG_END
* Run AppendOnlyKVCommandDeduplicationIT for KV ledgers that use the append-only schema
This only handles the interface definition, not the implementation in
the template. There are also a few rough edges:
1. It maks all choices as consuming.
2. it ignores locations
But for a poc that doesn’t seem too bad.
The tests don’t do anything super useful since the typechecker falls
over but I checked tha tthe generated LF looks more or less reasonable.
changelog_begin
changelog_end
On Sept 8 our ES cluster became unresponsive. I tried connecting to the
machines.
One machine had an ES Docker container that claimed to have started 7
weeks ago and stopped 5 weeks ago, while the machine's own uptime was 5
weeks. I assume GCP had decided to restart it for some reason. The init
script had failed on missing a TTY, hence the addition of the
`DEBIAN_FRONTEND` env var.
Two machines had a Docker container that had stopped on that day, resp.
6h and 2h before I started investigating. It wasn't immediately clear
what had caused the containers to stop.
On all three of these machines, I was abble to manually restart the
containers and they were abble to reform a cluster, though the state of
the cluster was red (missing shards).
The last two machines simply did not respond to SSH connection attempts.
Assuming it might help, I decided to try to restart the machines. As GCP
does not allow restarting individual machines when they're part of a
managed instance roup, I tried clicking the "rolling restart" button
on the GCP console, which seemed like it would restart the machines. I
carefully selected "restart" (and not "replace"), started the process,
and watched GCP proceed to immediately replace all five machines, losing
all data in the process.
I then started a new cluster and used bigger (and more) machines to
reingest all of the data, and then fell back to the existing
configuration for the "steady" state. I'll try to keep a better eye on
the state of the cluster from now on. In particular, we should not have
a node down for 5 weeks without noticing.
I'll also try to find some time to look into backing up the Kibana
configuration, as that's the one thing we can't just reingest at the
moment.
CHANGELOG_BEGIN
CHANGELOG_END
* Always use max_deduplication_duration as deduplication period for committer side deduplication
Doing this we get the following guarantees:
- Deduplication period is constant for all the submissions therefore this guarantees consistency between "backward-looking" and "forward-looking" deduplication
CHANGELOG_BEGIN
kvutils - committer side deduplication always uses max_deduplication_duration + min_skew as a deduplication period for all the requests.
CHANGELOG_END
* Set max_deduplication_duration to 10 seconds for all the conformance tests
* Update KV command deduplication conformance tests to account for max deduplication duration used as deduplication period for all the requests
* Add max deduplication wait for multi-participant conformance test
* Overwrite the submitter info deduplication period in the transaction committer so that we always set the max deduplication duration
* Update ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/participant/state/kvutils/committer/transaction/TransactionCommitterSpec.scala
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Add separate step for overwriting the deduplication period
* Code review - reformatting and renames
* Add comment for setting the deduplication duration
* [docs] Replace AdoptOpenJDK suggestion by Adoptium
In the installation instructions, suggest Adoptium as JDK source
[AdoptOpenJDK has moved to the Eclipse foundation](
https://blog.adoptopenjdk.net/2021/08/goodbye-adoptopenjdk-hello-adoptium/)
* Without changelog entry
CHANGELOG_BEGIN
CHANGELOG_END
Manual release process. @stefanobaghino-da is in charge of this release.
Commit log (since 1.17.0-snapshot.20210907.7759.0.35a853fd):
```
976ca400eb Force JSON API to refresh packages for GET requests to /v1/query (#10835)
e42cd3a1c6 LF: add interface to scala AST and decoder (#10830)
07b72d0f3e Update Ledger API's proto documentation [KVL-1094] (#10815)
648acf52f8 Fix typo in app-arch.rst (#10798)
dc9fbb5aa9 Stringify errors in create-daml-app before calling alert (#10831)
90b008ef0d Stop using `controller … can` syntax in ledger tests (#10814)
e490b69471 Clarifications of code ownership in /ledger and /ledger-api (#10742)
8cdeb7539a make auth middleware aware that actAs => readAs (#10825)
e263d43aca Ledger API test tool: test that record time is monotonically increasing [KVL-1053] (#10552)
e5793a317b Account for multi participant conformance tests when updating the config [KVL-1092] (#10788)
2e39d7b647 Deprecate `Commands.deduplication_time` field [KVL-1096] (#10802)
7824201b8d Get rid of extra new line and prune precision for all gatling values (#10820)
a03f52a15e Add ledger API tests about command deduplication information in completions [KVL-1057] (#10748)
61a07b1986 Add command line option max-deduplication-duration for sandbox and KV [KVL-1098] (#10816)
0800504955 Fix DuplicateSubmissionID test exclusions (#10819)
1ff34ffbe8 Remove workaround for akka supervisor issue (#10818)
6d347d4e88 Add daml interfaces to the Haskell LF AST (#10811)
cecb27fe16 Stop using controller can syntax in skeleton template (#10799)
8830eaa2ec Drop controller can syntax from compiler tests (#10807)
9038a8051b Update code documentation for `participant-state` API [KVL-1094] (#10812)
60ffb79fb1 Command deduplication - Participant side/committer side command deduplication conformance tests split [KVL-1093] (#10784)
1e6540f8f2 Stop using `controller _ can` syntax in daml-lf tests (#10805)
0db15e6759 Ledger API test tool: ensure that test identifiers are unique (#10804)
f1e378a60b rotate release duty after 1.17.0-snapshot.20210907.7759.0.35a853fd (#10809)
1464177bf8 release 1.17.0-snapshot.20210907.7759.0.35a853fd (#10808)
deb8a8b740 [Divulgence pruning] Daml SDK docs update [DPP-536] (#10764)
8405ad58fd Enhance InMemoryLedger to use the ValueEnricher (#10757)
```
Changelog (since 1.17.0-snapshot.20210907.7759.0.35a853fd):
```
- [JSON API] Fixed a bug that prevented the JSON API to be aware of
packages uploaded directly via the Ledger API.
- The auth middleware will now consider that actAs credentials are valid
for readAs requests.
- [Ledger API Specification] Introduced a new conformance test about recort time in completions being monotonically increasing
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
- [Ledger API Specification] `Commands.deduplication_time` field has been deprecated, please use `Commands.deduplication_duration` instead.
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
[Ledger API Specification] Participant pruning of all divulged contracts is fully implemented: Participant operators can choose to prune all immediately and retroactively divulged contracts, by setting the newly-added prune_all_divulged_contracts flag in the ParticipantPruningService/Prune request.
```
changelog_begin
changelog_end
* Initial changes to add a surrogate_template_id cache to reduce db queries
CHANGELOG_BEGIN
CHANGELOG_END
* refactoring and addition of tests
* Code review based changes to use Contextual Logger and json-api metrics instance
* make max cache entries/size configurable
* Rename cache max entries default variable
* Add failing test that covers the bug we found in #10823
* Fix /v1/query endpoint bug
changelog_begin
- [JSON API] Fixed a bug that prevented the JSON API to be aware of
packages uploaded directly via the Ledger API.
changelog_end
* Stop using `controller … can` syntax in ledger tests
We’re planning to deprecate that in SDK 2.0 to reduce the confusion
around the implicit observer behavior so in preparation for that, this
PR drops the syntax from all ledger tests.
changelog_begin
changelog_end
* shuffle let above usage
changelog_begin
changelog_end
It seems a bit silly that the auth middleware would refuse a token that
has an `actAs` when it needs a `readAs`.
Reference: https://docs.daml.com/app-dev/authorization.html#access-tokens-and-claims
CHANGELOG_BEGIN
- The auth middleware will now consider that actAs credentials are valid
for readAs requests.
CHANGELOG_END
* Test that record time is monotonically increasing
CHANGELOG_BEGIN
- [Ledger API Specification] Introduced a new conformance test about recort time in completions being monotonically increasing
CHANGELOG_END
* Address early review comments
* Fix unused import
* Simplify main assertion
* Add assertion about expected number of record times
* Print less record times in main assert's error message
* Make wartremover:Option2Iterable happy
* Use a submitting party
* operations -> submissions
* Create contracts rather than allocating parties
* Shorten comment
* Explicitly sort checkpoints by offset
* Shorten short identified
* Don't assume the ledger is at the beginning
* Simplify the test
* Make the new test optional
* Apply review suggestions
* Split command deduplication conformance tests into non-kv and kv tests.
Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.
CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END
* Run participant side/committer side deduplication tests based on the ledger implementation
* Try to update the time model on all the participants to account for multi participant setups.
When running with multi participants only one participant can update the configuration. Unfortunately to figure out which one exactly can do so it's not that easy so to make sure the configuration changes for our tests succeed we try to update it on all the participants.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix on memory build file
* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT
* Rollback compat import
* Exclude command dedup from the reference test
* Exclude command deduplication from the reference export
* Format bazel
* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory
* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox
* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export
* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers
* Format bazel file
* Exclude CommandDeduplicationIT from sandbox
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add comments for command deduplication IT
* Rename to be consistent with scala namings
* Update formatting and comments
* Code review renames
* Add compat import
* Remove unused import
* Use a scala 2.12 compatible approach to receive all the configured participants in the conformance tests
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
CHANGELOG_BEGIN
- [Ledger API Specification] `Commands.deduplication_time` field has been deprecated, please use `Commands.deduplication_duration` instead.
CHANGELOG_END
