* Extract common code for command dedup conformance tests
CHANGELOG_BEGIN
CHANGELOG_END
* Revert changes for parallel tests
* Fix compile warnings
* Disable participant deduplication for KV
CHANGELOG_BEGIN
CHANGELOG_END
* Always call submitRequestAndAssertDeduplication with parties
* Fix party in CommandDeduplicationBase
* Don't test KV-based command deduplication with static time as it's broken
* Don't disable participant-side command deduplication for sandbox-next on static time
* Revert "Don't test KV-based command deduplication with static time as it's broken"
This reverts commit 3dac41aa
* Add missing dependency
* Don't cross-test this ledger API test tool with existing versions
* Add the current PR to the list of PRs that resulted in exclusions
* Apply @garyverhaegen-da's suggestion for compatibility exclusions
Co-authored-by: Nicu Reut <nicu.reut@digitalasset.com>
Add test case for command deduplication which uses both the CommandService and the CommandSubmissionService
CHANGELOG_BEGIN
CHANGELOG_END
* Run mixed CommandService/CommandSubmissionService test for all the possible combinations of calls
* Add test case for parallel command deduplication using mixed submission through the CommandService and the CommandSubmissionService
CHANGELOG_BEGIN
CHANGELOG_END
Add CLI flag to select minimum enabled TLS version for participant server
CHANGELOG_BEGIN
Sandbox: Add CLI flag `--min-tls-version` to select minimum enabled TLS version for participant server.
CHANGELOG_END
* Reorder the observers for StreamConsumer to first filter and then take just one element, not the other way around
CHANGELOG_BEGIN
CHANGELOG_END
* Refactor the test to get better stacktraces during failures
* Remove unused import
* Extract deduplication "features" into a configuration to be used around the tests.
Better naming for assertions that support sync and async deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Fix broken test and use consistency for tests
* ledger-api-test-tool - Add conformance test for parallel command deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Add import for 2.12 compat
* Add silencer plugin
* Split parallel command deduplication scenario into it's own test suite
* Add the parallel command deduplication test to append only ledgers
* Run parallel command deduplication tests for append only ledgers
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Code review renames
* Add compat import
* Run the test concurrently
* Support adding tests as an hidden option
* Simplify existing suites
CHANGELOG_BEGIN
CHANGELOG_END
* Remove stale conformance suites from build.yml
* `--add` -> ``--additional`
* Re-add `--all-tests` as deprecated CLI option to be tested
* Move sandbox-classic pruning test to wall clock again
* Run KVCommandDeduplicationIT for sandbox append-only
* Tidy-up
* Also add participant pruning test to ledger-on-memory/single-participant
* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only
* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only
* Add KVCommandDeduplicationIT to ledger-on-memory append-only
* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant
* Tidy-up
* Use KVCommandDeduplicationIT for sandbox-on-x too
* Fix merge
Add max dedup duration arg to all the test suites that include command dedup tests
* Make `--include` and `--additional` mutually exclusive
* Uniform formatting of multi-line strings
* Move exclusions after additions as they are applied last
* Re-disable deduplication test on sandbox with static time
* Re-disable deduplication test on sandbox-on-x
- Add support for specifying either 1.2 or 1.3 as minimum TLS versions for ledger api server.
- Log enabled protocols (~TLS versions) and cipher suites at server and client startup.
- Add integration tests against Sandbox-classic and Sandbox
CHANGELOG_BEGIN
Sandbox: Add CLI flag to select minimum enabled TLS version for ledger API server.
CHANGELOG_END
* Extract deduplication "features" into a configuration to be used around the tests.
Better naming for assertions that support sync and async deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Fix broken test and use consistency for tests
* Add conformance test for KV committer usage of max deduplication duration as deduplication duration
CHANGELOG_BEGIN
CHANGELOG_END
* Run AppendOnlyKVCommandDeduplicationIT for KV ledgers that use the append-only schema
* Always use max_deduplication_duration as deduplication period for committer side deduplication
Doing this we get the following guarantees:
- Deduplication period is constant for all the submissions therefore this guarantees consistency between "backward-looking" and "forward-looking" deduplication
CHANGELOG_BEGIN
kvutils - committer side deduplication always uses max_deduplication_duration + min_skew as a deduplication period for all the requests.
CHANGELOG_END
* Set max_deduplication_duration to 10 seconds for all the conformance tests
* Update KV command deduplication conformance tests to account for max deduplication duration used as deduplication period for all the requests
* Add max deduplication wait for multi-participant conformance test
* Overwrite the submitter info deduplication period in the transaction committer so that we always set the max deduplication duration
* Update ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/participant/state/kvutils/committer/transaction/TransactionCommitterSpec.scala
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Add separate step for overwriting the deduplication period
* Code review - reformatting and renames
* Add comment for setting the deduplication duration
* Test that record time is monotonically increasing
CHANGELOG_BEGIN
- [Ledger API Specification] Introduced a new conformance test about recort time in completions being monotonically increasing
CHANGELOG_END
* Address early review comments
* Fix unused import
* Simplify main assertion
* Add assertion about expected number of record times
* Print less record times in main assert's error message
* Make wartremover:Option2Iterable happy
* Use a submitting party
* operations -> submissions
* Create contracts rather than allocating parties
* Shorten comment
* Explicitly sort checkpoints by offset
* Shorten short identified
* Don't assume the ledger is at the beginning
* Simplify the test
* Make the new test optional
* Apply review suggestions
* Split command deduplication conformance tests into non-kv and kv tests.
Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.
CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END
* Run participant side/committer side deduplication tests based on the ledger implementation
* Try to update the time model on all the participants to account for multi participant setups.
When running with multi participants only one participant can update the configuration. Unfortunately to figure out which one exactly can do so it's not that easy so to make sure the configuration changes for our tests succeed we try to update it on all the participants.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix on memory build file
* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT
* Rollback compat import
* Exclude command dedup from the reference test
* Exclude command deduplication from the reference export
* Format bazel
* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory
* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox
* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export
* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers
* Format bazel file
* Exclude CommandDeduplicationIT from sandbox
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add comments for command deduplication IT
* Rename to be consistent with scala namings
* Update formatting and comments
* Code review renames
* Add compat import
* Remove unused import
* Use a scala 2.12 compatible approach to receive all the configured participants in the conformance tests
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
CHANGELOG_BEGIN
- [Ledger API Specification] `Commands.deduplication_time` field has been deprecated, please use `Commands.deduplication_duration` instead.
CHANGELOG_END
* Add Ledger API Test Tool tests about command deduplication information present in completions
CHANGELOG_BEGIN
CHANGELOG_END
* Address review comments
* Fix offset reference and move new tests to a separate, optional suite
* Cover rejections as well
* Test both the command and command submission services
* Run new test suite (only) on sandbox-classic append-only and daml-on-sql
* Use the append-only schema with daml-on-sql for CommandDeduplicationInfoIT
* Fix tests except offset, successful completions only.
* Remove completion offset test as it's not supported by most ledgers
* Remove support for multiple submissions as rejection completions are not being tested for the moment
* Consolidate test cases for faster run
* Avoid forbidden characters in short identifiers
* Clarify assert
* Remove wrong test about deduplication time being preserved in completion
* Adhere to the Scala style guide
* Eliminate some code duplication
* fmt
* Make 3e404be compile
* Never assume a specific deduplication period format in completions
* Code cleanup
* Pass party to completionStreamRequest to avoid ILLEGAL_ARG failures
* Enable in append-only mode for all ledgers that support it
* Clarify that the new tests are append-only-only
* Update ledger/ledger-on-sql/BUILD.bazel
Fix suite name
* Update ledger/sandbox-classic/BUILD.bazel
Fix suite name
* fmt
* Split command deduplication conformance tests into non-kv and kv tests.
Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.
CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END
* Run participant side/committer side deduplication tests based on the ledger implementation
* Fix on memory build file
* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT
* Rollback compat import
* Exclude command dedup from the reference test
* Exclude command deduplication from the reference export
* Format bazel
* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory
* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox
* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export
* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers
* Format bazel file
* Exclude CommandDeduplicationIT from sandbox
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add comments for command deduplication IT
* Rename to be consistent with scala namings
* Update formatting and comments
* Add naming prefix for tests to avoid clashes
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Ensure that test identifiers are unique
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/ledger-api-test-tool/src/test/suite/scala/com/daml/ledger/api/testtool/NamesSpec.scala
Improve test description
* Improve test clarity and efficiency
* Uniform short IDs' naming convention for config and package mgmt. test cases
* Hardening ParticipantPruningIT divulgence tests to not flake on canton-enterprise
worked out with @tudor-da
CHANGELOG_BEGIN
CHANGELOG_END
* Also feed other participant events
* Switch canton conformance tests to parallel indexer
* More stability fixes
* Add deduplication and submission id to the proto submitter info so we can build the v2 completion info
* Use v2 rejections and store the enriched model in protobuf
* Update v1 SubmissionResult to v2 SubmissionResult
* Update v1 imports to v2 imports
* Enable API deduplication for ledgers that don't have it implemented yet (all kv)
* Update tests for new rejections
* Disable buf check for oneof
* Code review - fix formatting and test assertions
* Update imports for proto changes
* Add tests for rejection conversions
* Use entry id as submission id when submission id is missing
* Calculate deduplication period for rejection based on the set deduplicate until by calculating the duration relative to record time
* Code review consistent naming and comments
* Submission id is optional in the completion info
the submissionId is not set for participant.state.v1 entries, therefore for the transition from v1 to the v2 API we need to account for the missing submission ids by making it optional in the completion info.
* Update messages and grpc status codes for rejections to keep backward compatibility
* Deprecate proto deduplicate_until
* Set deduplication value in the commit context for all the deduplication periods. This change allows us to re-enable the committer deduplication tests
* Set ledger configuration for tests
* Command dedup: In the committer set the deduplicate until context value based on the given deduplication period [KVL-1049] (#10743)
CHANGELOG_BEGIN
participant-state:
- Migrate to use only v2 Read/Write Services. This includes the use of new models for rejections/updates/submission results.
- Calculate deduplicate until for committer side deduplication based on the submitter given deduplication period
- if using the deprecated deduplicateUntil then just set the given timestamp
- if using duration then calculate the new deduplicateUntil by using the formula (submissionTime + deduplicationDuration + minSkew)
- if using offset deduplication then calculate deduplicateUntil by using the formula (submissionTime + maxDeduplicationDuration + minSkew)
- if the deduplication period is not set then we don't set deduplicateUntil
- Emit completions with status `ALREADY_EXISTS` for duplicate commands
CHANGELOG_END
* Enable pruning in the sandbox-classic when the append-only schema is used
CHANGELOG_BEGIN
- [Sandbox] - Participant pruning is enabled in the sandbox-classic when the append-only schema is used
CHANGELOG_END
* Added pruning conformance tests to the build definition of the sandbox-classic
* Changed pruning IT time to the wall-clock time
* Added a comment explaining changes made
* Do not enable pruning when pruneAllDivulgedContracts option is enabled
Adding support for accepting server's private key as an encrypted file (since storing unencrypted private key in a file system might be a risk).
Encrypted private key is assumed to be encrypted using AES or similar algorithm. The details necessary to decrypt it are be obtained from a secrets server over HTTP as JSON document. The URL to secret's server is supplied through the new `--secrets-url` CLI parameter.
One can supply private in either plaintext (old behavior) or ciphertext: if a private key's file ends with .enc suffix it is assumed to be ciphertext. Otherwise it is assumed to be plain text.
CHANGELOG_BEGIN
- [DPP-418] [Participant] Add support for supplying server's private key as an encrypted file and then decrypting it with the help of a secrets server.
CHANGELOG_END
* Add `deduplication_duration` to `deduplication_period` and deprecate `deduplication_time`
CHANGELOG_BEGIN
ledger-api - add `deduplication_duration` as a future replacement for `deduplication_time` in the command proto definition
CHANGELOG_END
* Add tests for deduplication period validation
* All divulgence events pruning
* Add `pruneAllDivulgedContracts` to `IndexParticipantPruningService`
* Add `participant_pruned_all_contracts_up_to_inclusive` in `parameters` table
* `pruneEvents` prunes all divulgence events
* Adapt `ParticipantPruningIT` to check for divulgence with and without `pruna_all_divulged_contracts`
* Create multi-participant pruning test in ledger-on-memory
CHANGELOG_BEGIN
CHANGELOG_END
* CommonStorageBackend
* Enrich pruning descriptions
* Logging for command completions pruning
* Move divulgence pruning back to the beginning of pruning sequence
* Addressed review comments
* PRDivulgenceArchivalPruning adapted for privacy-aware ledgers
* Rebased to main
* Disable ParticipantPruningIT tests targeting append-only schema in mutable schema conformance tests in ledger-on-sql
* Adapted order of SQL migration queries
* Documentation of appendonlydao.JdbcLedgerDao.prune
* Apply suggestions from code review
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
CHANGELOG_BEGIN
ledger-api - Command deduplication period can now be specified by setting `deduplication_offset` instead of `deduplication_time` (only valid for v2 WriteService). This change is backwards compatible.
CHANGELOG_END
* Propagate the enriched deduplicationPeriod instead of deduplication duration
* Update the Haskell bindings for the new deduplication period
* Calculate the deduplicateUntil using the new deduplication period for backward compat
* Use consistent naming for deduplication_period
* Cleanup command timeout extraction from deduplication period
* Add the required deduplication_offset to deduplication instead of deduplication_start
* Update haskell bindings to support deduplication_offset
* Add support for deduplication_offset in the ledger-api
* Remove the timestamp-based deduplication from our models to simplify upgrade for users
* Add optional conformance test for offset based deduplication
* Remove buf rule for FIELD_SAME_ONEOF as our change is backwards compatible
* Disable FIELD_SAME_ONEOF buf check for commands file
* Apply suggestions from code review
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Update comment for deduplication period
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Use `extra` in the port file runner, rather than `temporary`.
* ledger-api-test-tool-on-canton: Use the port check runner.
Much simpler than the port file runner for our purposes.
* Replace `runner` with `runner_with_port_file`.
Rather than expecting a particular set of command-line-arguments, we use
templating.
CHANGELOG_BEGIN
CHANGELOG_END
* Rename the `runner_with_port_check` target to the default.
* ledger-api-test-tool: Make IntelliJ happy with TransactionServiceIT.
* ledger-api-test-tool: Collect TransactionServiceIT tests into groups.
* ledger-api-test-tool: Split TransactionServiceIT into lots of suites.
CHANGELOG_BEGIN
- [Ledger API Test Tool] The ``TransactionServiceIT`` test suite has
been split into many test suites. If you are including or excluding
it, you will need to use the new test suite names, or you can use
"TransactionService" as a prefix for all of them.
If you are including or excluding individual tests, you will need to
update your arguments with the new test suite. You can find the new
test suite by running the test tool with the ``--list-all``
flag and looking for the test's short identifier. The short
identifiers have not changed, with the exception of
``TXNoContractKey``, which has been renamed to ``CKNoContractKey`` and
is now in the ``ContractKeysIT`` test suite.
CHANGELOG_END
