99% of our usecases use Value[ContractId] so this PR just fixes it.
The few other usescases are:
1. Value[Nothing] which we use for keys. This is technically more
precise but we benefit very little from it.
2. Value[String] mostly because in a few places we are lazy.
We don’t have any code which benefits from being polymorphic in the
contract id type.
changelog_begin
changelog_end
* Extract deduplication "features" into a configuration to be used around the tests.
Better naming for assertions that support sync and async deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Fix broken test and use consistency for tests
* ledger-api-test-tool - Add conformance test for parallel command deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Add import for 2.12 compat
* Add silencer plugin
* Split parallel command deduplication scenario into it's own test suite
* Add the parallel command deduplication test to append only ledgers
* Run parallel command deduplication tests for append only ledgers
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Code review renames
* Add compat import
* Run the test concurrently
RCA: if at parallel indexer initialization some error happening, then a promise never completes, which causes an initialization future never complete
Expected: failure should be propagated, and recovering indexer should retry 10 seconds later
Actual: failure not propagated, a zombie future freezes initialization, preventing retries
Impact: this is a corner case - if no problems at indexer initialization, the issues is not surfacing
* Extracts critical logic into helper function initializeHandle
* Adds unit tests for initializeHandle
* Fixes issue by completing the promise in all cases
CHANGELOG_BEGIN
CHANGELOG_END
* Avoid slow-progress timeout StoppedProgressing to avoid flakiness
CHANGELOG_BEGIN
CHANGELOG_END
* Make schema migration retries and backoff configurable
* Review feedback - use RetryStrategy.constant instead
* Remove unused tailrec import
* Simplifications and runF by Marton
* Rename config options from retry to attempt and default to 30 attempts
* Create a new grpc exception for each duplicate result
The metadata in the exception is not thread safe, and when being converted into server headers netty.Utils.convertServerHeaders, it calls discardAll which mutates the metadata. Because this was reused for all duplicate exceptions then we got corrupted metadata.
CHANGELOG_BEGIN
CHANGELOG_END
* Do not call duplicate command exception twice
* Added --database-connection-pool-size parameter to the sandbox CLI
* Improved maximum ledger time lookup query
* Improved active contract lookup query
* Improved max event sequential id query
CHANGELOG_BEGIN
- [Sandbox] - Added a CLI parameter for configuring the number of connections in the database connection pool used for serving ledger API requests
CHANGELOG_END
* [Daml error codes API] Further implementations
* Implements ErrorCode.asGrpcError (and test)
* Error code logging now accepts correlation id and an extra context map
* Full error context is included into enriched logging context
CHANGELOG_BEGIN
CHANGELOG_END
* Fixed Scala 2.12 compilation issues
* fixes#9975 - enable json index for all fields that we run JSON_EXISTS queries against
CHANGELOG_BEGIN
CHANGELOG_END
* fixes#9975 - enable json index for all fields that we run JSON_EXISTS queries against
recompute sha
CHANGELOG_BEGIN
CHANGELOG_END
* Support adding tests as an hidden option
* Simplify existing suites
CHANGELOG_BEGIN
CHANGELOG_END
* Remove stale conformance suites from build.yml
* `--add` -> ``--additional`
* Re-add `--all-tests` as deprecated CLI option to be tested
* Move sandbox-classic pruning test to wall clock again
* Run KVCommandDeduplicationIT for sandbox append-only
* Tidy-up
* Also add participant pruning test to ledger-on-memory/single-participant
* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only
* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only
* Add KVCommandDeduplicationIT to ledger-on-memory append-only
* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant
* Tidy-up
* Use KVCommandDeduplicationIT for sandbox-on-x too
* Fix merge
Add max dedup duration arg to all the test suites that include command dedup tests
* Make `--include` and `--additional` mutually exclusive
* Uniform formatting of multi-line strings
* Move exclusions after additions as they are applied last
* Re-disable deduplication test on sandbox with static time
* Re-disable deduplication test on sandbox-on-x
* [Ledger API error codes] ErrorCode interfaces and generator
* Implementation of annotation processor
* Implementation of DocItem generator
* Unit testing of the generator and error code logging
CHANGELOG_BEGIN
CHANGELOG_END
* Addressed review comments
- Add support for specifying either 1.2 or 1.3 as minimum TLS versions for ledger api server.
- Log enabled protocols (~TLS versions) and cipher suites at server and client startup.
- Add integration tests against Sandbox-classic and Sandbox
CHANGELOG_BEGIN
Sandbox: Add CLI flag to select minimum enabled TLS version for ledger API server.
CHANGELOG_END
* Extract deduplication "features" into a configuration to be used around the tests.
Better naming for assertions that support sync and async deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Fix broken test and use consistency for tests
Following #10763, we drop the ad-hoc builders for `FrontStack`.
* Building a `Fronstack` from individuals elements should be done with
standard scala buidler.
* Building a `Fronstack` from a `TraversableOne` should be done with
the scala 2.13 `.to(FrontStack)` methd
* Building a `Fronstack` from a `ImmArray` should be done with the
`toImmArray` method.
CHANGELOG_BEGIN
CHANGELOG_END
* Add conformance test for KV committer usage of max deduplication duration as deduplication duration
CHANGELOG_BEGIN
CHANGELOG_END
* Run AppendOnlyKVCommandDeduplicationIT for KV ledgers that use the append-only schema
* Always use max_deduplication_duration as deduplication period for committer side deduplication
Doing this we get the following guarantees:
- Deduplication period is constant for all the submissions therefore this guarantees consistency between "backward-looking" and "forward-looking" deduplication
CHANGELOG_BEGIN
kvutils - committer side deduplication always uses max_deduplication_duration + min_skew as a deduplication period for all the requests.
CHANGELOG_END
* Set max_deduplication_duration to 10 seconds for all the conformance tests
* Update KV command deduplication conformance tests to account for max deduplication duration used as deduplication period for all the requests
* Add max deduplication wait for multi-participant conformance test
* Overwrite the submitter info deduplication period in the transaction committer so that we always set the max deduplication duration
* Update ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/participant/state/kvutils/committer/transaction/TransactionCommitterSpec.scala
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Add separate step for overwriting the deduplication period
* Code review - reformatting and renames
* Add comment for setting the deduplication duration
* Initial changes to add a surrogate_template_id cache to reduce db queries
CHANGELOG_BEGIN
CHANGELOG_END
* refactoring and addition of tests
* Code review based changes to use Contextual Logger and json-api metrics instance
* make max cache entries/size configurable
* Rename cache max entries default variable
* Stop using `controller … can` syntax in ledger tests
We’re planning to deprecate that in SDK 2.0 to reduce the confusion
around the implicit observer behavior so in preparation for that, this
PR drops the syntax from all ledger tests.
changelog_begin
changelog_end
* shuffle let above usage
changelog_begin
changelog_end
* Test that record time is monotonically increasing
CHANGELOG_BEGIN
- [Ledger API Specification] Introduced a new conformance test about recort time in completions being monotonically increasing
CHANGELOG_END
* Address early review comments
* Fix unused import
* Simplify main assertion
* Add assertion about expected number of record times
* Print less record times in main assert's error message
* Make wartremover:Option2Iterable happy
* Use a submitting party
* operations -> submissions
* Create contracts rather than allocating parties
* Shorten comment
* Explicitly sort checkpoints by offset
* Shorten short identified
* Don't assume the ledger is at the beginning
* Simplify the test
* Make the new test optional
* Apply review suggestions
* Split command deduplication conformance tests into non-kv and kv tests.
Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.
CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END
* Run participant side/committer side deduplication tests based on the ledger implementation
* Try to update the time model on all the participants to account for multi participant setups.
When running with multi participants only one participant can update the configuration. Unfortunately to figure out which one exactly can do so it's not that easy so to make sure the configuration changes for our tests succeed we try to update it on all the participants.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix on memory build file
* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT
* Rollback compat import
* Exclude command dedup from the reference test
* Exclude command deduplication from the reference export
* Format bazel
* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory
* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox
* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export
* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers
* Format bazel file
* Exclude CommandDeduplicationIT from sandbox
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add comments for command deduplication IT
* Rename to be consistent with scala namings
* Update formatting and comments
* Code review renames
* Add compat import
* Remove unused import
* Use a scala 2.12 compatible approach to receive all the configured participants in the conformance tests
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
CHANGELOG_BEGIN
- [Ledger API Specification] `Commands.deduplication_time` field has been deprecated, please use `Commands.deduplication_duration` instead.
CHANGELOG_END
* Add Ledger API Test Tool tests about command deduplication information present in completions
CHANGELOG_BEGIN
CHANGELOG_END
* Address review comments
* Fix offset reference and move new tests to a separate, optional suite
* Cover rejections as well
* Test both the command and command submission services
* Run new test suite (only) on sandbox-classic append-only and daml-on-sql
* Use the append-only schema with daml-on-sql for CommandDeduplicationInfoIT
* Fix tests except offset, successful completions only.
* Remove completion offset test as it's not supported by most ledgers
* Remove support for multiple submissions as rejection completions are not being tested for the moment
* Consolidate test cases for faster run
* Avoid forbidden characters in short identifiers
* Clarify assert
* Remove wrong test about deduplication time being preserved in completion
* Adhere to the Scala style guide
* Eliminate some code duplication
* fmt
* Make 3e404be compile
* Never assume a specific deduplication period format in completions
* Code cleanup
* Pass party to completionStreamRequest to avoid ILLEGAL_ARG failures
* Enable in append-only mode for all ledgers that support it
* Clarify that the new tests are append-only-only
* Update ledger/ledger-on-sql/BUILD.bazel
Fix suite name
* Update ledger/sandbox-classic/BUILD.bazel
Fix suite name
* fmt
* Add command line option max-deduplication-duration which allows us to configure the max deduplication duration for sandbox and KV ledgers.
The main usage of this is specifying different max deduplication durations for tests.
CHANGELOG_BEGIN
CHANGELOG_END
* Code review changes
* Split command deduplication conformance tests into non-kv and kv tests.
Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.
CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END
* Run participant side/committer side deduplication tests based on the ledger implementation
* Fix on memory build file
* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT
* Rollback compat import
* Exclude command dedup from the reference test
* Exclude command deduplication from the reference export
* Format bazel
* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory
* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox
* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export
* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers
* Format bazel file
* Exclude CommandDeduplicationIT from sandbox
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add comments for command deduplication IT
* Rename to be consistent with scala namings
* Update formatting and comments
* Add naming prefix for tests to avoid clashes
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Ensure that test identifiers are unique
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/ledger-api-test-tool/src/test/suite/scala/com/daml/ledger/api/testtool/NamesSpec.scala
Improve test description
* Improve test clarity and efficiency
* Uniform short IDs' naming convention for config and package mgmt. test cases
* Enhance InMemoryLedger to use the ValueEnricher.
Allowing SandboxServer to run Engine with the default transactionNormalization=false
CHANGELOG_BEGIN
CHANGELOG_END
* enrich only when responding to verbose API queries
* Hardening ParticipantPruningIT divulgence tests to not flake on canton-enterprise
worked out with @tudor-da
CHANGELOG_BEGIN
CHANGELOG_END
* Also feed other participant events
* Switch canton conformance tests to parallel indexer
* More stability fixes
* Deprecated protobuf DamlTransactionRejection reasons which are generated by the v1 participant state API only.
CHANGELOG_BEGIN
kvutils - protobuf rejections generated by the participant state v1 API are deprecated (Inconsistent, Disputed, ResourcesExhausted, PartyNotKnownOnLedger)
CHANGELOG_END
* Reorganize proto fields in the order of the field number
* Use bigint instead of timestamp
changelog_begin
changelog_end
* Add tests
* Fix deduplicateUntil
* Fix some places that still use timestamp
* Fix some places that still use timestamp
* DbDto uses only Long instead of Instant values
* Fix some places that still use Timestamp
* Remove unused classes
* Fix some places that still use Timestamp
* Review feedback switch to regex and move to pure function
changelog_begin
changelog_end
* Review feedback
* Attempt at making scala 2.12 test-compile
* HA protected ledger initialization for the appendOnly [DPP-551]
* Preparation - Refactoring ParallelIndexerFactory: pulling initialization, resource factory and subscription apart
CHANGELOG_BEGIN
CHANGELOG_END
* Preparation - Refactoring: untangle JdbcIndexer and FlywayMigrations
CHANGELOG_BEGIN
CHANGELOG_END
* Preparation - Refactoring: simplifying Indexer type
CHANGELOG_BEGIN
CHANGELOG_END
* Preparation - Refactoring: simplifying initialized JdbcIndexer type
CHANGELOG_BEGIN
CHANGELOG_END
* Move initialization code to InitializeParallelIngestion (which does initialization in a HA protected zone)
CHANGELOG_BEGIN
CHANGELOG_END
* Fixes mutable indexer initialization
* Indexer initializes for every RecoveringIndexer restart now
* Also fixes reset: it happend only once, at the beginning
CHANGELOG_BEGIN
CHANGELOG_END
* Minor changes based on review
CHANGELOG_BEGIN
CHANGELOG_END
* Small change in mutable initialization based on review
CHANGELOG_BEGIN
CHANGELOG_END
Since we switch to scala 2.13, ImmArray companion object extends
`Factory`. Hence:
- the `apply` methods of `ImmArray` override the one from `Factory`
- we can use the notation `.to(ImmArray)` to convert an `Iterable` to
`ImmArray`
This PR drops those `apply` ImmArray. Conversion from Iterable to
`ImmArray` should use the `.to(ImmArray)`.
CHANGELOG_BEGIN
CHANGELOG_END
* Re-write command deduplication integration test for participant state to be valid for the deduplication changes.
The test now accounts for completions being generated for duplicate commands.
CHANGELOG_BEGIN
CHANGELOG_END
* Update check for successfull transaction
* Emit command rejected completions for duplicate commands rejections when using pre-execution
CHANGELOG_BEGIN
participant-state: Emit completions (CommandRejected) for duplicate commands when using pre-execution
CHANGELOG_END
* Update test for emitted duplicate command update
* Added test for new case.
* Revert redundant test case.
This reverts commit 8ef4364f
* Test corner case of dropping not deduplicated transaction rejection.
* Code tidying.
* Reformatted.
Co-authored-by: Miklos Erdelyi <miklos.erdelyi@digitalasset.com>
As it turned out normal connection closeing does not imply released resources on the database side
* use eventually for testing connection lost cases
* refactor a case to use proper release instead of connection close
* increases a timeout to lower risk of flakyness
CHANGELOG_BEGIN
CHANGELOG_END
* Use the token from incoming requests to update the package list
changelog_begin
changelog_end
* Lazily initialize the ledger client
* Fix ee integration tests
* Fix package reloading behaviour by using a semaphore to check for ongoing updates
* Refactor out the semaphore code into a concurrency utility class
* Use correct locking for the updateTask so every thread always uses an up to date task
* Remove unused imports in utils.Concurrent & remove packages from the tests
* Hide & mark the token file cli option deprecated because we dont need it anymore and only keep it so client deployment code doesn't break
* Fix scala 2.12 build by adding more type annotations
* Update ledger-service/http-json-cli/src/main/scala/com/daml/http/OptionParser.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/PackageService.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Readd pgkManagementClient after it was removed accidentally (but now it's lazy)
* Remove concurrent object & use atomic boolean instead of a mutex because it makes more sense
* Replace semaphore with countdownlatch
* Refactor the caching into a separate class
* Use Instant instead of LocalDateTime
* Remove that ** of bad synchonization and do stupid simple synchronization because it JUST WORKS, besides adapt when we want to reload
* Remove await in tests because it can result in buggy tests
* remove unused code in WebSocketService.scala
* Unhide the access-token-file option as per request of Stefano
* Less implicit jwts per request of Stefano
* Try making some code more readable as by request of Akshay
* Use more shark because it expresses better than flatMaps if I don't need the arg
* Move defs in predicate in WebsocketService.scala around
* Try to minimize diff further in WebsocketService.scala
* Fix build and minimize diff in WebSocketService.scala further
* Minimize diff of function getTransactionSourceForParty in WebSocketService.scala
* Share the ec in WebSocketService.scala to minimize the diff
* Minimize in function predicate in WebSocketService.scala
* Further minimize in function predicate in WebSocketService.scala
* Change some case classes to be normal classes but with apply method
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/PackageService.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Get rid of implicit jwt tokens, the world is already confusing and full of implicits enough
* Improve readability
* Integrate the new LedgerClient which does not depend on a leder id
* Fix tests
* Apply suggestions from code review
thanks to @S11001001
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Apply further review comments
* Remove outcommented code
* Deprecate access token file option in the description too
changelog_begin
- [JSON API] The cli option `--access-token-file` is now deprecated. It
is not needed anymore and you can safely remove it. Reason is that
the operations which prior required a token at startup are now done
on demand using the token of the incoming request.
changelog_end
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Add deduplication and submission id to the proto submitter info so we can build the v2 completion info
* Use v2 rejections and store the enriched model in protobuf
* Update v1 SubmissionResult to v2 SubmissionResult
* Update v1 imports to v2 imports
* Enable API deduplication for ledgers that don't have it implemented yet (all kv)
* Update tests for new rejections
* Disable buf check for oneof
* Code review - fix formatting and test assertions
* Update imports for proto changes
* Add tests for rejection conversions
* Use entry id as submission id when submission id is missing
* Calculate deduplication period for rejection based on the set deduplicate until by calculating the duration relative to record time
* Code review consistent naming and comments
* Submission id is optional in the completion info
the submissionId is not set for participant.state.v1 entries, therefore for the transition from v1 to the v2 API we need to account for the missing submission ids by making it optional in the completion info.
* Update messages and grpc status codes for rejections to keep backward compatibility
* Deprecate proto deduplicate_until
* Set deduplication value in the commit context for all the deduplication periods. This change allows us to re-enable the committer deduplication tests
* Set ledger configuration for tests
* Command dedup: In the committer set the deduplicate until context value based on the given deduplication period [KVL-1049] (#10743)
CHANGELOG_BEGIN
participant-state:
- Migrate to use only v2 Read/Write Services. This includes the use of new models for rejections/updates/submission results.
- Calculate deduplicate until for committer side deduplication based on the submitter given deduplication period
- if using the deprecated deduplicateUntil then just set the given timestamp
- if using duration then calculate the new deduplicateUntil by using the formula (submissionTime + deduplicationDuration + minSkew)
- if using offset deduplication then calculate deduplicateUntil by using the formula (submissionTime + maxDeduplicationDuration + minSkew)
- if the deduplication period is not set then we don't set deduplicateUntil
- Emit completions with status `ALREADY_EXISTS` for duplicate commands
CHANGELOG_END
I previously changed `SqlLedgerSpec` to use the append-only index, not
realizing that we have both test classes. This removes the
(now-redundant) `SqlLedgerSpecAppendOnly`, and copies `SqlLedgerSpec` to
`SqlLedgerOnMutableIndexSpec`. The only differences are:
- the mutable index tests check for synchronous commits, and
- the append-only index tests check for rejection reason details
(which are not present in the mutable index schema).
Previously I had added some tests to `SqlLedgerSpec` but not
`SqlLedgerSpecAppendOnly`, so they had diverged a little.
I also think this makes it clearer that where the tests diverge, we want
to keep the append-only tests, not the mutable ones.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-common: Replace `URL` with a trait so we can fake it.
This means we don't have to use Mockito to stub out `URL`, which is
final and therefore requires magic to stub.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-common: Move tests that open streams into SecretsUrlTest.
* ledger-api-common: Make SecretsUrl constructors simple methods.
* ledger-api-common: Reduce concrete implementations of SecretsUrl.
* Augment completion.proto with deduplication-related info
CHANGELOG_BEGIN
CHANGELOG_END
* Explicitly specify fields not yet filled in when building Completion
* Time-based deduplication periods are measured in record time of completions
* Add deduplication_offset as a deduplication_period option
* Don't skip proto field numbers
* CompletionFromTransaction: use default Completion constructor
* submission_rank: reserve proto field for future use
* Add comment about reserved proto field
* Add command deduplication columns to completions in append-only schemas
CHANGELOG_BEGIN
CHANGELOG_END
* Remove duplicated `application_id` field from Postgres and H2 schemas
* Make `application_id` and `submission_id` nullable
* Re-generate migrations' sha256
* Tidy-up
* Add RW logic for new command dedup columns in completions
* Declare `deduplication_start` as a nullable column
* Fix UpdateToDbDtoSpec
* Fix merge from main
* Replace Either3 with an ADT in StorageBackendTestValues
* Test with non-zero nanos
* Fix UpdateToDbDtoSpec
* Also change Schema.scala
* Simplify DbDto.CommandCompletion construction
* Fix merge from `main` (migration SHAs and filename)
* Fix UpdateToDbDto: restore transactionId
* Fix completions' application_id: it must be non-nullable in all schemas, as in PSQL
* Actually use newly read columns
* Add DB round-trip tests for new fields in StorageBackendTestValues
* Format
* StorageBackendTestValues: property type someSubmissionId
* UpdateToDbDto: remove duplicated logic
* Simplify UpdateToDbDto.commandCompletion
* Fix comment in CompletionFromTransaction.toApiDeduplicationPeriod
* Fix CompletionFromTransaction.toApiCompletion
* Fix deduplication_offset in Schema
* Make parameters explicit in CompletionFromTransaction and CompletionStorageBackendTemplate calls
* Shorten error message in CompletionFromTransaction
* Fix StorageBackendTestsCompletions
* Fix CompletionFromTransaction
* Separate command dedup-related test cases in UpdateToDbDtoSpec
* Simplify further UpdateToDbDto.commandCompletion
* Simplify CompletionFromTransaction.toApiCompletion
* Format
* Enable pruning in the sandbox-classic when the append-only schema is used
CHANGELOG_BEGIN
- [Sandbox] - Participant pruning is enabled in the sandbox-classic when the append-only schema is used
CHANGELOG_END
* Added pruning conformance tests to the build definition of the sandbox-classic
* Changed pruning IT time to the wall-clock time
* Added a comment explaining changes made
* Do not enable pruning when pruneAllDivulgedContracts option is enabled
* Enable append-only schema in sandbox
changelog_begin
changelog_end
* Add test
* Unhide the compression flag in sandbox
* Make append-only config non-optional
* Prune divulgence events before migration to append-only
* Prune all divulgence events with NULL offset
* Check pruning request against migration offset
* Added unit test for migration offset validation
CHANGELOG_BEGIN
CHANGELOG_END
* Removed unnecessary discards
Soft references allow the garbage collector to collect the values referenced
by soft references in case of memory pressure. Since these caches are not the
source of truth for the data, it's better to remove values from the cache
instead of having the process die due to OOM (not enough heap or excessive GC).
Fixes DPP-561.
CHANGELOG_BEGIN
CHANGELOG_END
* Add the LedgerClientWithoutLedgerId class
* Minimize diff
changelog_begin
changelog_end
* Minimize diff further
* Add missing license header & reformat
* Update language-support/scala/bindings-akka/src/main/scala/com/digitalasset/ledger/client/binding/LedgerClientBinding.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update ledger/ledger-api-client/src/main/scala/com/digitalasset/ledger/client/LedgerClient.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Add changelog
changelog_begin
- [Ledger Client Scala Bindings] A new variant of the LedgerClient class
was added called `LedgerClientWithoutLedgerId`. This class does not
need a ledger id at initialization. It was added to allow skipping
any checks at initialization for use cases where either the
ledger id is not known at initalization or no valid token can be fed
at initialization for checking the ledger id. Furthermore for each
classes `ActiveContractSetClient`, `CommandClient`, `PackageClient`,
`TransactionClient`, `VersionClient` now exists a variant which
doesn't depend on a ledger id at initialization and instead requires
one for every function as parameter. Moreover the existing classes
are extending these classes with overriding the methods and setting
the default of the parameter with the given ledger id from
initialization. The class `LedgerClientWithoutLedgerId` already
makes usage of these variants e.g. `PackageClientWithoutLedgerId`.
changelog_end
* More changelog
changelog_begin
- [Ledger Client Scala Bindings] The function `transactionSource` of the
class `LedgerClientBinding` now optionally accepts a token which is
passed on to the unterlying call.
changelog_end
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox-classic: Remove default parameters in `SqlLedger.Owner`.
The defaults can be dangerous.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-classic: Revert LedgerResource's index schema to the mutable one.
CHANGELOG_BEGIN
- [Integration Kit] Changes the Offset format to contain a version and therefore reduces the highest index size by one byte
CHANGELOG_END
* participant-integration-api: On submit and wait, capture a deadline.
We need this deadline to make sure we terminate the stream.
* ledger-api-client: Use the specified expiry time for tracking.
Not the command deduplication time.
CHANGELOG_BEGIN
- [Ledger API Server] The command deduplication time is no longer used
for determining the period of time to track the command before giving
up. Instead, the gRPC deadline is used. If no deadline is provided
(or if the deadline exceeds the command tracker retention period), the
tracker retention period is used instead.
CHANGELOG_END
* ledger-api-client: Keep supporting the `deduplication_time` timeout.
* ledger-api-client: Improve comments in CommandTrackerFlow and its tests.
Co-Authored-By: Miklos <57664299+miklos-da@users.noreply.github.com>
* sandbox-classic: Add more tests for `SqlLedger#publishTransaction`.
* sandbox-classic: Simplify the SqlLedger test constructors.
* sandbox-classic: Improve the out of bounds assertions in SqlLedgerSpec.
* sandbox-classic: Add ErrorInfo metadata for rejections.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-classic: Enable the append-only index schema in tests.
* sandbox-classic: Assert rejection details in SqlLedgerSpec.
* sandbox-classic: Simplify `Rejection`.
Inline methods that aren't used externally.
* sandbox-classic: Extract out some duplication in SqlLedgerSpec.
* Unhide the append-only schema flags
changelog_begin
- [Sandbox, participant] Added a flag to enable a new append-only database schema.
This schema was designed to support significantly higher performance.
In a future release, all applications will automatically migrate to the new schema.
changelog_end
* Improve CLI flag descriptions
* ledger-api-client: Rename `maxDeduplicationTime` to `maximumExpiryTime`.
* ledger-api-client: Add tests for command tracker timeouts.
* ledger-api-client: Cap the deduplication period at the expiry time.
* participant-integration-api: Make `trackerRetentionPeriod` a JDuration.
* ledger-api-client: Don't use `JDuration` if we can avoid it.
* participant-integration-api: Add a test for command service timeout.
* Use the tracker retention period as the maximum expiry time.
CHANGELOG_BEGIN
- [Ledger API Server] The command service now uses the tracker retention
period (typically specified with the ``--tracker-retention-period``
command-line argument) as the maximum time to wait for a command to
arrive on the completion stream. After this time, the command will
time out, though it may still complete in the future. Previously, the
deduplication period was used, but it was likely the tracker would be
terminated before that anyway.
The default tracker retention period is 5 minutes, unless otherwise
specified.
CHANGELOG_END
* participant-integration-api: Add a test for TrackerMap parameters.
* participant-integration-api: Use `maximumCommandTimeout`.
Instead of `maximumExpiryTime`.
* ledger-api-client: Shorten timeouts in CommandTrackerFlowTest.
* participant-integration-api: Make `LocalServices` a non-case class.
* participant-integration-api: Split `ApiCommandService.LocalServices`.
The tracker uses the completion functions, and the service uses the
transaction functions, but we don't need them in one place.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Add tests for `ApiCommandService`.
Adding support for accepting server's private key as an encrypted file (since storing unencrypted private key in a file system might be a risk).
Encrypted private key is assumed to be encrypted using AES or similar algorithm. The details necessary to decrypt it are be obtained from a secrets server over HTTP as JSON document. The URL to secret's server is supplied through the new `--secrets-url` CLI parameter.
One can supply private in either plaintext (old behavior) or ciphertext: if a private key's file ends with .enc suffix it is assumed to be ciphertext. Otherwise it is assumed to be plain text.
CHANGELOG_BEGIN
- [DPP-418] [Participant] Add support for supplying server's private key as an encrypted file and then decrypting it with the help of a secrets server.
CHANGELOG_END
* bindings-akka: Parameterize RetryInfo with its value type.
* ledger-api-client: Wrap command submissions in a new class.
This will allow me to pass a deadline alongside the commands in the
future.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-client: Inline `CommandRetryFlow.Value`.
* bindings-akka: Inline `CommandRetryFlow`'s `createRetry`.
It's always the same: `retryInfo.value`.
* Pass divulged contracts in KVUtils
CHANGELOG_BEGIN
[Integration Kit] KV-based ledgers pass contract instances through the Write/ReadService, removing the need
for backfilling divulged contract lookups.
Note: KV Ledgers that have been created before this change will still be relying on backfilling lookups of divulged contracts,
hence pruning of all divulged contracts may result in failing lookups for divulged contracts.
CHANGELOG_END
* Enriched TransactionCommitterSpec with assertions for divulgence and disclosure
* Refactored
* Removed DamlTransactionEntry.divulgedContracts
* Added the contractInstance to DivulgenceEntry
* Refactored
* Removed DamlTransactionEntry.divulgedContracts
* Added the contractInstance to DivulgenceEntry
* Addressed review comments
* Addressed Miklos' review comments
* ledger-on-sql: Ensure all databases start auto-incrementing from 1.
* ledger-on-sql: Do not increment the dispatcher head on start.
I don't know how this ever made it in, because it's clearly wrong.
Unfortunately we don't have tests that restart the participant/ledger
server while keeping the database intact, so this is very difficult to
test at a high level.
CHANGELOG_BEGIN
CHANGELOG_END
* Add `deduplication_duration` to `deduplication_period` and deprecate `deduplication_time`
CHANGELOG_BEGIN
ledger-api - add `deduplication_duration` as a future replacement for `deduplication_time` in the command proto definition
CHANGELOG_END
* Add tests for deduplication period validation
* All divulgence events pruning
* Add `pruneAllDivulgedContracts` to `IndexParticipantPruningService`
* Add `participant_pruned_all_contracts_up_to_inclusive` in `parameters` table
* `pruneEvents` prunes all divulgence events
* Adapt `ParticipantPruningIT` to check for divulgence with and without `pruna_all_divulged_contracts`
* Create multi-participant pruning test in ledger-on-memory
CHANGELOG_BEGIN
CHANGELOG_END
* CommonStorageBackend
* Enrich pruning descriptions
* Logging for command completions pruning
* Move divulgence pruning back to the beginning of pruning sequence
* Addressed review comments
* PRDivulgenceArchivalPruning adapted for privacy-aware ledgers
* Rebased to main
* Disable ParticipantPruningIT tests targeting append-only schema in mutable schema conformance tests in ledger-on-sql
* Adapted order of SQL migration queries
* Documentation of appendonlydao.JdbcLedgerDao.prune
* Apply suggestions from code review
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
CHANGELOG_BEGIN
ledger-api - Command deduplication period can now be specified by setting `deduplication_offset` instead of `deduplication_time` (only valid for v2 WriteService). This change is backwards compatible.
CHANGELOG_END
* Propagate the enriched deduplicationPeriod instead of deduplication duration
* Update the Haskell bindings for the new deduplication period
* Calculate the deduplicateUntil using the new deduplication period for backward compat
* Use consistent naming for deduplication_period
* Cleanup command timeout extraction from deduplication period
* Add the required deduplication_offset to deduplication instead of deduplication_start
* Update haskell bindings to support deduplication_offset
* Add support for deduplication_offset in the ledger-api
* Remove the timestamp-based deduplication from our models to simplify upgrade for users
* Add optional conformance test for offset based deduplication
* Remove buf rule for FIELD_SAME_ONEOF as our change is backwards compatible
* Disable FIELD_SAME_ONEOF buf check for commands file
* Apply suggestions from code review
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Update comment for deduplication period
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* participant-integration-api: Clean up failed trackers.
Otherwise they can hang around forever.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Ensure that all trackers are closed.
* participant-integration-api: Ensure that waiting trackers are closed.
* participant-integration-api: Store the tracker map future in the state.
* participant-integration-api: Fix a race in `TrackerMap`.
If the supplied `Future` to `AsyncResourceState` is too fast, the state
may not be set. We need to initialize the state immediately.
* participant-integration-api: Add more comments to `TrackerMap`.
Create normalized TXs when a partial TX is finalised.
Except in limited cases! (i.e for scenario-runner, sandbox)
CHANGELOG_BEGIN
CHANGELOG_END
normalize values in the engine as they are converted from speedy-values
fix 2.12 build
backout redundant change
ensure byKey field is correctly normalized when constructed by engine
rename flag: valueNormalization -> transactionNormalization
improve comment
delete commented-out code
rename: toValueNorm --> toNormalizedValue
rename: (SValue.) toValue --> toUnNormalizedValue
revert changes to ptx so that the interface to insertCreate() etc is Value-based (not SValue-based)
improve comments
respell: toUnNormalizedValue --> toUnnormalizedValue
fix build
* participant-integration-api: Make `TrackerMap` a `Tracker`.
Mostly by moving parameters around.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Clean up `TrackerMap` a little more.
* participant-integration-api: Make `TrackerMap` generic.
* participant-integration-api: Move `Tracker.WithLastSubmission`.
It's only used by `TrackerMap`, so now it's an internal class there.
* participant-integration-api: Only provide the key to `newTracker`.
* participant-integration-api: Subsume the `TrackerMap` cleanup schedule.
* participant-integration-api: Construct the tracker map outside.
* ledger-api-common: Remove some unnecessary braces.
* participant-integration-api: Prettify `TrackerMap` some more.
* participant-integration-api: Make `TrackerMap.selfCleaning` a class.
* participant-integration-api: Add some tests for TrackerMap.
* participant-integration-api: Convert a method to `Runnable`, 2.12-style.
Apparently underscores aren't good enough.
* ledger-api-client: Delete CompletionSuccess#unapply.
It doesn't work on Scala 2.12.
Preparation, small fixes
* Remove unnecessary TODO
* Fix FieldStrategy.idempotentInsert type parameters
Changing the DB-Schema
This change adapts the DB schema for all supported backends. After this change we only populate the party_entries table, and on the query side we reconstruct the state from this.
* Drop the party table
* Add indexing to party_entries
Adapting StorageBackend: ingestion
Since we only ingest party_entries, the party population needs to be removed.
* Drop the party table in ingestion code
* Fixes test
Adapting StorageBackend: queries
Queries needs to be adapted to construct the state from the read side.
* Rewrite queries.
* Fixes reset implementations.
Adapting JdbcLedgerDao
Since underlying storage changed, JdbcLedgerDao can be simplified: no special treatment needed with duplicate errors, since these errors are impossible to happen.
Removing JdbcLedgerDao tests, and adding a new test, testing the behavior of the new event-source party model. Please note: this database refactoring only applies to the append-only schema, so for the mutating schema the test is disabled.
During implementation a bug surfaced: it was not possible anymore to store the is_local information via JdbcLedgerDao.storePartyEntry. Although this bug is a minor issue, since that method is only used from single participant environment, still a fix was also implemented for this, by passing a magic participantId upon non-local party storage.
* Simplify storePartyEntry.
* Fixes bug introduced by append-only.
* adds/adapts tests
Refactoring: remove not used duplicateKeyError from StorageBackend
Changes to JdbcLedgerDao rendered this duplicateKeyError unused.
* Removes unused duplicateKeyError
Adapting sandbox-classic
In sandbox-classic it is not allowed to have updates for parties. Essentially the updates concerning already existent parties were dropped silently with logging without effect.
Here I started by pinning down this behaviour in the SqlLedgerSpec and SqlLedgerSpecAppendOnly. These tests were implemented with the original code in mind.
Then adapted the SqlLedger method: making sure of uniqueness by first trying to lookup the to-be-persisted party.
* Added tests grabbing a hold on original behavior
* Adapted implementation to ensure same behavior
Switching to correct is_local derivation for party queries as per review
* Adapting implementation: switching to aggregated table and a query on that
* Introducing QueryStrategy.booleanOrAggregationFunction to support Oracle
* Moving party related queries to PartyStorageBackendTemplate
* Fixes JdbcLedgerDaoPartiesSpec tests, and add another test case
Also:
* Align Update interface documentation
* Switching to explicit optionality in party query implementation asa per review
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
CHANGELOG_BEGIN
CHANGELOG_END
This PR makes possible to check for contract IDs suffix during
preprocessing.
This is the first part of the task 3 described in #10504.
CHANGELOG_BEGIN
CHANGELOG_END
* Augment completion.proto with deduplication-related info
CHANGELOG_BEGIN
CHANGELOG_END
* Explicitly specify fields not yet filled in when building Completion
* Time-based deduplication periods are measured in record time of completions
* Add deduplication_offset as a deduplication_period option
* Don't skip proto field numbers
* CompletionFromTransaction: use default Completion constructor
* submission_rank: reserve proto field for future use
* Add comment about reserved proto field
Previously, if the max deduplication time was extended, the participant
_might_ retain the old time for certain submissions.
CHANGELOG_BEGIN
- [Ledger API Server] The API server manages a single command tracker
per (application ID × submitters) pair. This tracker would read the
current ledger configuration's maximum deduplication time on creation,
but never updated it, leading to trackers that might inadvertently
reject a submission when it should have been accepted. The tracker now
reads the latest ledger configuration.
CHANGELOG_END
* Time conversion duration between buffer event and API domain transaction
* Compute partiesTemplates inversion mapping outside events transformation
* Other small optimizations
CHANGELOG_BEGIN
CHANGELOG_END
* Add StorageBackend tests
changelog_begin
changelog_end
* Fix Oracle tests
* Do not use empty byte arrays
* Format
* Fix after rebase
* Substitute type params with type bounded ingest method
* Remove empty line
* Assert on configuration contents
* Fix Oracle build
* Add tests for ingestion initialization
* fmt
* Add test for leftover data after reset
* Add resetAll
* Use resetAll between tests
Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
* participant-integration-api: Add logging to RecoveringIndexerSpec.
It's flaky, and I would like to know exactly what's going on here.
* participant-integration-api: Attempt to fix RecoveringIndexerSpec.
The checks for the health status are racy. I'm hoping increasing the
timeouts will help a little.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Inline timeouts in RecoveringIndexerSpec.
The `LedgerConfigurationSubscriptionFromIndexSpec` was flaky due to
stubs not specifying all behavior for `IndexConfigManagementService`.
This fixes the underlying issue by avoiding stubs in favor of fakes,
which means that _all_ behavior must be modelled.
Note: Martin Fowler has an excellent, terse description of [the various
forms of test doubles][TestDouble].
[TestDouble]: https://www.martinfowler.com/bliki/TestDouble.html
CHANGELOG_BEGIN
CHANGELOG_END
* Upgrade to a newer canton version (post 0.27.0 snapshot version)
with canton-community configuration that supports higher throughput.
changelog_begin
changelog_end
* Disable flaky reject DeeplyNestedValueIT:Reject tests that time out half the time
As stated in #10504 the contract ID freshness check cannot be
implemented correctly in general.
This PR drops the support for this (buggy) check.
This corresponds to the fist task of #10504.
CHANGELOG_BEGIN
CHANGELOG_END
This was limited a while back during the initial development of
_ledger-on-sql_ for reasons I can't remember. Let's stop doing that.
CHANGELOG_BEGIN
CHANGELOG_END
* Refactor ParameterStorageBackend
- a single method for atomic ledger initialization
- a single method to look up the ledger end
changelog_begin
changelog_end
* Add a test
* Fix reading event sequential ids
* Remove debug statements
* Allow ledgerEnd on an empty database
* Initialization is not safe to call concurrently
* Remove leftovers from isolation level change
* Use unit return type
for initialization methods
* Allow getParticipantId on an empty database
* Use exceptions instead of a return type ADT
* Don't use Try for initialization
* Clean up parameters table
* Simplify parameter storage api
* Address review suggestion
* Address review comment
* Address review comment
* Prefer ledger id over participant id mismatch
* Address review comment
* Move type definition
* Remove useleess new keyword
* Renove unused import
* Inline result mapping
* Fix reporting of mismatching participantId
* participant-integration-api: Construct completions in one place.
* sandbox-classic: Inline `CompletionFromTransaction#apply`.
It's only used here; there's no reason to keep it in the
_participant-integration-api_.
* participant-integration-api: Store a status gRPC protobuf.
Instead of storing the status code and message, we store a serialized
`google.rpc.Status` protocol buffers message. This allows us to pass
through any additional information reported by the driver `ReadService`.
The migration is only done for the append-only database, and preserves
old data in the existing columns. New data will only be written to the
new column.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Improve comments in migrations.
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* participant-integration-api: Further improvements to migrations.
* participant-integration-api: Store the rejection status as 3 columns.
Serializing the details but keeping the code and message columns
populated.
* participant-integration-api: Publish the indexer protobuf to Maven.
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* Use `extra` in the port file runner, rather than `temporary`.
* ledger-api-test-tool-on-canton: Use the port check runner.
Much simpler than the port file runner for our purposes.
* Replace `runner` with `runner_with_port_file`.
Rather than expecting a particular set of command-line-arguments, we use
templating.
CHANGELOG_BEGIN
CHANGELOG_END
* Rename the `runner_with_port_check` target to the default.
* Use the port file and dynamic port generation in client/server tests.
This creates a runner named `runner_with_port_file` which knows how to
interpolate two variables, `%PORT_FILE%` and `%PORT%`. This allows us to
use the `port-file` argument to the kvutils runner rather than
hard-coding a port for conformance tests.
For now, we only use this for generating the kvutils reference ledger
export.
CHANGELOG_BEGIN
CHANGELOG_END
* Simplify the runner_with_port_file considerably.
It doesn't need to check if the port is open; we trust that the process
will do it.
This also makes sure the port file will be cleaned up, and reduces the
number of dependencies by making use of more functions in `extra`.
* Simplify port file generation in the new client-server runner.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Simplify the runner_with_port_file further.
This doesn't need to work if the server doesn't take a port file.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Move `DeduplicationPeriod` to ledger-api-domain so that it ca be reused and passed down from the ledger-api-client to into the v2.SubmitterInfo
CHANGELOG_BEGIN
participant-state - move `DeduplicationPeriod` to ledger-api-domain
CHANGELOG_END
* Revert unrelated changes
* ledger-api-test-tool: Make IntelliJ happy with TransactionServiceIT.
* ledger-api-test-tool: Collect TransactionServiceIT tests into groups.
* ledger-api-test-tool: Split TransactionServiceIT into lots of suites.
CHANGELOG_BEGIN
- [Ledger API Test Tool] The ``TransactionServiceIT`` test suite has
been split into many test suites. If you are including or excluding
it, you will need to use the new test suite names, or you can use
"TransactionService" as a prefix for all of them.
If you are including or excluding individual tests, you will need to
update your arguments with the new test suite. You can find the new
test suite by running the test tool with the ``--list-all``
flag and looking for the test's short identifier. The short
identifiers have not changed, with the exception of
``TXNoContractKey``, which has been renamed to ``CKNoContractKey`` and
is now in the ``ContractKeysIT`` test suite.
CHANGELOG_END
* ledger-grpc: Fix the directory paths.
This brings _ledger-grpc_ in line with other projects. Scala main files
should be under "src/main/scala", and test files should be under
"src/test/suite/scala".
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-grpc: Moar imports.
* Normalize transactions & values as a separate pass. Use for simpler defintiion of isReplayedBy.
CHANGELOG_BEGIN
CHANGELOG_END
normalize transaction version
* remove stray import from bad merge which breaks scala 2_12 build
* change isReplayedBy to only norm its RIGHT (replay) argument
* add forgotton normalization for ValueEmum
* switch to use existing value normalization code (remove my newly coded duplicate code)
* normalize submittedTransaction before calling engine.validate
* dont call normalizeTx from Engine.validate
* *do* call normalizeTx from Engine.validate
* Set ErrorInfo metadata flag for definite_answer, which is propagated from the completion status
CHANGELOG_BEGIN
ledger-api-client - Propagate definite_answer as metadata in the GRPC response for submit/submitAndWait
CHANGELOG_END
* Keep alphabetical order for bazel build files
* Add test for inclusion of metadata
* Formatting
* Use explicit types to track failures when submitting a request for execution
To distinguish from execution failures (represented by `CompletionFailure`) which is also exposed as part of the akka-bindings, we introduced `TrackingCompletionFailures` which can also represent the failure to add the request to the execution queue.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix formatting
* Apply suggestions from code review
Co-authored-by: Hubert Slojewski <hubert.slojewski@digitalasset.com>
* Inline handling of errors in the tracker to eliminate the need for the secondary promise and simplify the code
* Update testing for the new error handling
* Remove brackets and make code compatible with 2.12
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Review cleanup and use inside for cleaner tests
Co-authored-by: Hubert Slojewski <hubert.slojewski@digitalasset.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Upgrade Scalatest to v3.2.9.
Because of some coupling we also have to upgrade Scalaz to the latest
v7.2 point release, v7.2.33.
The Scalatest changes are quite involved because the JAR has been broken
up into several smaller JARs. Because Bazel expects us to specify all
dependencies and doesn't allow transitive dependencies to be used
directly, this means that we need to specify the explicit Scalatest
components that we use.
As you can imagine, this results in quite a big set of changes. They
are, however, constrained to dependency management; all the code remains
the same.
CHANGELOG_BEGIN
CHANGELOG_END
* http-json-oracle: Fix a Scalatest dependency.
* ledger-api-client: Fix a Scalatest dependency.
* ledger-api-test-tool: Add some basic unit tests for test names.
* ledger-api-test-tool: Ensure that test names are not prefixes of others.
This makes sure that we can include or exclude any given test, without
affecting others.
* ledger-api-test-tool:Ensure that all tests have different names.
Looks like we had some copy-pasta.
CHANGELOG_BEGIN
CHANGELOG_END
* Track command response using an Either instead of passing the completion with the grpc code.
This makes it clearer as to the result of command tracking. We no longer count on the grpc status to determine if there was an error or not, and instead use types for that.
CHANGELOG_BEGIN
akka-bindings: `LedgerClientBinding.commands` now returns a flow of `Either[CompletionFailure, CompletionSuccess]` instead of `Completion` for clearer error handling. For backwards compatiblity the new return type can be turned back into a `Completion` using `CompletionResponse.toCompletion`
CHANGELOG_END
* Fix formatting
* Code review changes
- remove usages of Symbol in tests
- clean curly braces
* Remove change added from another PR
* Fix import
* Fix import
* Fix retry flow and extract one more match case
* Un-nest matches to a single level for simplicity
* fix typo
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Be consistent in assertions and prefer `inside` for pattern matching
* Inline CompletionResponse to use the full type
* Use simpler matcher
* Formatting
* Add a way to convert back an `Either[CompletionFailure, CompletionSuccess]` to a `Completion` for backwards compatibility. This simplifies update for systems that are tightly coupled to `Completion`
* Add test for converting to/from CompletionResponse
* Remove unnecessary brackets
* Add missing header
* Use checked exceptions to preserve backwards compatiblity
* Fix unapply
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* kvutils: Add property-based tests for the offset builder.
* kvutils: Avoid using `BigInt` in offset parsing.
This also means we can theoretically support negative offsets, even if
in practice it's probably a bad idea.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Further property-based tests for OffsetBuilder.
* kvutils: Improve a comment in OffsetBuilder.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add flag to enable/disable command deduplication
* Remove flags for configuration as it should not be exposed externally
* Move deduplicationEnabled flag to the write service.
The deduplication enabled flag is tightly coupled to the WriteService implementation so the flag has been moved to the WriteService trait so that it's explicitly defined.
CHANGELOG_BEGIN
Command deduplication is enabled/disabled based on the write service implementation.
v1 WriteService enables command deduplication while v2 WriteService disabled it
CHANGELOG_END
* Rename deduplication flag
Set sandbox deduplication to true and proxy write service to the delegage
* H2 Storage backend support for canton jdbc urls with user/password
The H2 database's JdbcDataSource does not like seeing user/password
properties in the jdbc url raising errors upon hikari connection pool
initialization:
`org.h2.jdbc.JdbcSQLNonTransientConnectionException: Duplicate property "USER"`
expecting to be able to set user and password separately from passing on
the jdbc url.
As h2 is not supported in production and to get canton integration tests past this
resorting to parsing the jdbc url looking for user/password properties, removing
them from the url and instead setting them explicitly on the data source object.
changelog_begin
changelog_end
* Review feedback
