To enable direct IndexServer building (and enable two step ApiServer build process)
This will be needed for KV WriteService proxy: a wrapper is being build to serve deduplication time conversion, which needs access to completion service.
changelog_begin
changelog_end
kvutils - The deduplication period is no longer overwritten by the max deduplication duration, therefore there is no need to change the default max deduplication duration
changelog_begin
changelog_end
It never did anything in Sandbox Next. Packages are always loaded
eagerly in KV ledgers.
CHANGELOG_BEGIN
- [Sandbox] The ``--eager-package-loading`` flag has been deprecated. It
hasn't actually done anything for many releases; packages are always
loaded eagerly. This does not affect Daml on SQL, which does support
lazy package loading.
CHANGELOG_END
* Remove the mutating schema
changelog_begin
- [Participant] All participants now use the new append-only schema. Existing databases will
automatically upgrade to the new schema the first time a participant/ledger is started.
changelog_end
* Fix post-commit validation test
* Remove append-only flags from CI
* Don't crash when using deprecated flag
* Increase timeout for reset service tests
* Fix typos in parameter names
* Restore removed test
* Restore removed CLI check
* Improve CLI parameter description
* [Self-service error codes] Implement V2 in Authorizer
CHANGELOG_BEGIN
CHANGELOG_END
* Added unit test for authorize (non-streamed)
* Fix after rebase
* Do not expose the error codes switching mechanism to the Java bindings
* Adjust InternalAuthorizationError to be SystemInternalAssumptionViolated
* Parameter names in test
* Testing AuthorizationInterceptor with regard to returned error codes
* Do not use default error code version switchers at instance creation
* Addressed Pawel's review comments
* Using ErrorFactories for error dispatching
* Pass loggingContext to Authorizer where available
* Generic internal authorization error
* Propagate error version switch to KeyValueConsumption
CHANGELOG_BEGIN
CHANGELOG_END
* Introduce v2 (self-service) KV error codes behind the CLI switch
* Test v2 codes and fix them
* Keep newline separating methods in KeyValueConsumption
* TransactionRejections: don't wrap updates in `Some`
* Factor errorVersionsTable
* Reorder imports
* Split "convert rejection to proto models and back to expected grpc code"
* Remove unneeded Option returned by decodeTransactionRejectionEntry
* Formatting fix
* Fix 7537e93d
* kvutils: Remove `VersionedOffsetBuilder.apply`.
Doesn't do anything.
* ledger-on-memory: Use `VersionedOffsetBuilder`.
* indexer-benchmark: Use `VersionedOffsetBuilder`.
* ledger-on-sql: Use `VersionedOffsetBuilder`.
* kvutils: Use `VersionedOffsetBuilder` in tests.
* kvutils: Create a case class for `VersionedOffsetBuilder#split`.
* kvutils: Delete unused methods from the offset builders.
* kvutils: Use `Bytes#startWith` to check the offset version.
* kvutils: Move offset splitting into `VersionedOffset`.
* kvutils: Extract out versioned offset generators.
* kvutils: Replace `OffsetBuilder` with `VersionedOffsetMutator`.
This takes care of the last usages of `OffsetBuilder`, which were to
modify the lowest component of the offset.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Randomize the offset version in testing.
To make sure we don't use a hard-coded offset version anywhere.
* kvutils: `Random.between` is not available in Scala 2.12.
* kvutils: Move offset mutation methods to `VersionedOffset`.
* kvutils: Move the versioned offset construction into `VersionedOffset`.
The `VersionedOffsetBuilder` is still useful as it remembers the
version, meaning we only need to specify it once.
* ledger-api-test-tool: Add command line argument --static-time which makes the test tool aware that the ledger is running in static time
For command deduplication conformance tests this allows us to forward the time so that the tests can run in static time mode as well
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/Cli.scala
* Do not run conformance test that cannot run in static time
* Remove hidden from static time flag for ledger api test tool
* ledger-api-test-tool: Clarify parts of the deduplication tests.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* Use append-only schema in tests
changelog_begin
changelog_end
* Fix Oracle build
* Update ledger/ledger-on-memory/BUILD.bazel
Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
* Run more conformance tests on append-only
* Run pruning tests in split-participant mode
* Disable multi-participant test in split participant
* Add missing ContractIdIT test
Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
* Extract common code for command dedup conformance tests
CHANGELOG_BEGIN
CHANGELOG_END
* Revert changes for parallel tests
* Fix compile warnings
* Disable participant deduplication for KV
CHANGELOG_BEGIN
CHANGELOG_END
* Always call submitRequestAndAssertDeduplication with parties
* Fix party in CommandDeduplicationBase
* Don't test KV-based command deduplication with static time as it's broken
* Don't disable participant-side command deduplication for sandbox-next on static time
* Revert "Don't test KV-based command deduplication with static time as it's broken"
This reverts commit 3dac41aa
* Add missing dependency
* Don't cross-test this ledger API test tool with existing versions
* Add the current PR to the list of PRs that resulted in exclusions
* Apply @garyverhaegen-da's suggestion for compatibility exclusions
Co-authored-by: Nicu Reut <nicu.reut@digitalasset.com>
Adding:
- a mechanism to switch between error code versions: `ErrorCodesVersionSwitcher`,
- the corresponding CLI flag `--use-error-codes-v2` for sandbox-classic, sandbox and ledger-on-X.
---
As of this PR:
- `ErrorCodesVersionSwitcher` is never used in production code (apart from a no-op usage in `ApiTransactionService` to prove the point),
- `--use-error-codes-v2` flag is set to hidden (until new error have been adopted).
These will be used in the subsequent stories from the same epic: https://digitalasset.atlassian.net/browse/DPP-417
* Extract deduplication "features" into a configuration to be used around the tests.
Better naming for assertions that support sync and async deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Fix broken test and use consistency for tests
* ledger-api-test-tool - Add conformance test for parallel command deduplication
CHANGELOG_BEGIN
CHANGELOG_END
* Add import for 2.12 compat
* Add silencer plugin
* Split parallel command deduplication scenario into it's own test suite
* Add the parallel command deduplication test to append only ledgers
* Run parallel command deduplication tests for append only ledgers
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Code review renames
* Add compat import
* Run the test concurrently
* Added --database-connection-pool-size parameter to the sandbox CLI
* Improved maximum ledger time lookup query
* Improved active contract lookup query
* Improved max event sequential id query
CHANGELOG_BEGIN
- [Sandbox] - Added a CLI parameter for configuring the number of connections in the database connection pool used for serving ledger API requests
CHANGELOG_END
* Support adding tests as an hidden option
* Simplify existing suites
CHANGELOG_BEGIN
CHANGELOG_END
* Remove stale conformance suites from build.yml
* `--add` -> ``--additional`
* Re-add `--all-tests` as deprecated CLI option to be tested
* Move sandbox-classic pruning test to wall clock again
* Run KVCommandDeduplicationIT for sandbox append-only
* Tidy-up
* Also add participant pruning test to ledger-on-memory/single-participant
* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only
* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only
* Add KVCommandDeduplicationIT to ledger-on-memory append-only
* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant
* Tidy-up
* Use KVCommandDeduplicationIT for sandbox-on-x too
* Fix merge
Add max dedup duration arg to all the test suites that include command dedup tests
* Make `--include` and `--additional` mutually exclusive
* Uniform formatting of multi-line strings
* Move exclusions after additions as they are applied last
* Re-disable deduplication test on sandbox with static time
* Re-disable deduplication test on sandbox-on-x
- Add support for specifying either 1.2 or 1.3 as minimum TLS versions for ledger api server.
- Log enabled protocols (~TLS versions) and cipher suites at server and client startup.
- Add integration tests against Sandbox-classic and Sandbox
CHANGELOG_BEGIN
Sandbox: Add CLI flag to select minimum enabled TLS version for ledger API server.
CHANGELOG_END
* Add conformance test for KV committer usage of max deduplication duration as deduplication duration
CHANGELOG_BEGIN
CHANGELOG_END
* Run AppendOnlyKVCommandDeduplicationIT for KV ledgers that use the append-only schema
* Always use max_deduplication_duration as deduplication period for committer side deduplication
Doing this we get the following guarantees:
- Deduplication period is constant for all the submissions therefore this guarantees consistency between "backward-looking" and "forward-looking" deduplication
CHANGELOG_BEGIN
kvutils - committer side deduplication always uses max_deduplication_duration + min_skew as a deduplication period for all the requests.
CHANGELOG_END
* Set max_deduplication_duration to 10 seconds for all the conformance tests
* Update KV command deduplication conformance tests to account for max deduplication duration used as deduplication period for all the requests
* Add max deduplication wait for multi-participant conformance test
* Overwrite the submitter info deduplication period in the transaction committer so that we always set the max deduplication duration
* Update ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/participant/state/kvutils/committer/transaction/TransactionCommitterSpec.scala
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Add separate step for overwriting the deduplication period
* Code review - reformatting and renames
* Add comment for setting the deduplication duration
* Add Ledger API Test Tool tests about command deduplication information present in completions
CHANGELOG_BEGIN
CHANGELOG_END
* Address review comments
* Fix offset reference and move new tests to a separate, optional suite
* Cover rejections as well
* Test both the command and command submission services
* Run new test suite (only) on sandbox-classic append-only and daml-on-sql
* Use the append-only schema with daml-on-sql for CommandDeduplicationInfoIT
* Fix tests except offset, successful completions only.
* Remove completion offset test as it's not supported by most ledgers
* Remove support for multiple submissions as rejection completions are not being tested for the moment
* Consolidate test cases for faster run
* Avoid forbidden characters in short identifiers
* Clarify assert
* Remove wrong test about deduplication time being preserved in completion
* Adhere to the Scala style guide
* Eliminate some code duplication
* fmt
* Make 3e404be compile
* Never assume a specific deduplication period format in completions
* Code cleanup
* Pass party to completionStreamRequest to avoid ILLEGAL_ARG failures
* Enable in append-only mode for all ledgers that support it
* Clarify that the new tests are append-only-only
* Update ledger/ledger-on-sql/BUILD.bazel
Fix suite name
* Update ledger/sandbox-classic/BUILD.bazel
Fix suite name
* fmt
* Split command deduplication conformance tests into non-kv and kv tests.
Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.
CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END
* Run participant side/committer side deduplication tests based on the ledger implementation
* Fix on memory build file
* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT
* Rollback compat import
* Exclude command dedup from the reference test
* Exclude command deduplication from the reference export
* Format bazel
* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory
* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox
* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export
* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers
* Format bazel file
* Exclude CommandDeduplicationIT from sandbox
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add comments for command deduplication IT
* Rename to be consistent with scala namings
* Update formatting and comments
* Add naming prefix for tests to avoid clashes
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add deduplication and submission id to the proto submitter info so we can build the v2 completion info
* Use v2 rejections and store the enriched model in protobuf
* Update v1 SubmissionResult to v2 SubmissionResult
* Update v1 imports to v2 imports
* Enable API deduplication for ledgers that don't have it implemented yet (all kv)
* Update tests for new rejections
* Disable buf check for oneof
* Code review - fix formatting and test assertions
* Update imports for proto changes
* Add tests for rejection conversions
* Use entry id as submission id when submission id is missing
* Calculate deduplication period for rejection based on the set deduplicate until by calculating the duration relative to record time
* Code review consistent naming and comments
* Submission id is optional in the completion info
the submissionId is not set for participant.state.v1 entries, therefore for the transition from v1 to the v2 API we need to account for the missing submission ids by making it optional in the completion info.
* Update messages and grpc status codes for rejections to keep backward compatibility
* Deprecate proto deduplicate_until
* Set deduplication value in the commit context for all the deduplication periods. This change allows us to re-enable the committer deduplication tests
* Set ledger configuration for tests
* Command dedup: In the committer set the deduplicate until context value based on the given deduplication period [KVL-1049] (#10743)
CHANGELOG_BEGIN
participant-state:
- Migrate to use only v2 Read/Write Services. This includes the use of new models for rejections/updates/submission results.
- Calculate deduplicate until for committer side deduplication based on the submitter given deduplication period
- if using the deprecated deduplicateUntil then just set the given timestamp
- if using duration then calculate the new deduplicateUntil by using the formula (submissionTime + deduplicationDuration + minSkew)
- if using offset deduplication then calculate deduplicateUntil by using the formula (submissionTime + maxDeduplicationDuration + minSkew)
- if the deduplication period is not set then we don't set deduplicateUntil
- Emit completions with status `ALREADY_EXISTS` for duplicate commands
CHANGELOG_END
* Enable append-only schema in sandbox
changelog_begin
changelog_end
* Add test
* Unhide the compression flag in sandbox
* Make append-only config non-optional
* Upgrade Scalatest to v3.2.9.
Because of some coupling we also have to upgrade Scalaz to the latest
v7.2 point release, v7.2.33.
The Scalatest changes are quite involved because the JAR has been broken
up into several smaller JARs. Because Bazel expects us to specify all
dependencies and doesn't allow transitive dependencies to be used
directly, this means that we need to specify the explicit Scalatest
components that we use.
As you can imagine, this results in quite a big set of changes. They
are, however, constrained to dependency management; all the code remains
the same.
CHANGELOG_BEGIN
CHANGELOG_END
* http-json-oracle: Fix a Scalatest dependency.
* ledger-api-client: Fix a Scalatest dependency.
* sandbox: Do less when resetting.
We don't need to reconstruct some basic things.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: When testing the reset service, keep trying.
If we scale up the test duration, we also need to scale up the timeouts.
* timer-utils: If `RetryStrategy` eventually fails, explain what happened.
For example, if the attempts limit is exceeded, say so.
* Revert "timer-utils: If `RetryStrategy` eventually fails, explain what happened."
This reverts commit 6647681688.
* participant-integration-api: Make the initial ledger config optional.
CHANGELOG_BEGIN
- [Integration Kit] The initial ledger configuration is now optional;
if it is not specified, the participant will not attempt to submit a
configuration, but instead wait for something else to provision the
ledger with a configuration.
This behavior is intended to be used by ledger drivers that have
alternative means of setting up the initial ledger configuration. If
no configuration is provisioned, the participant server will not be
able to submit commands to the ledger.
CHANGELOG_END
* participant-integration-api: Avoid `Option#zip`.
It returns an `Iterable` in Scala 2.12, which is not helpful.
* participant-integration-api: Encapsulate the initial configuration.
* participant-integration-api: Reduce usage of `LedgerConfiguration`.
* Inline `LedgerConfiguration` wherever it's used.
Most things don't need all its constituent parts; this reduces the
amount of unused properties.
CHANGELOG_BEGIN
- [Integration Kit] The ``LedgerConfiguration`` class has been
removed in favor of ``InitialLedgerConfiguration``. Its usage
has been changed accordingly, with the ``configurationLoadTimeout``
property becoming part of ``ApiServerConfig`` instead.
The default options provided by ``LedgerConfiguration`` have been
removed; you are now encouraged to come up with sensible values for
your own ledger. The ``Configuration.reasonableInitialConfiguration``
value may help.
CHANGELOG_END
* Correct the initial configuration submission delay for KV ledgers.
* kvutils: Mark supertype unused parameters as unused.
* kvutils: Extract out common configuration submission delays.
These values are specific to kvutils; other drivers should come up with
their own.
* configuration: Delete `NoGeneration`, as it's unused.
* participant-integration-api: Move test constants to a companion object.
TelemetrySpecBase doesn't need to expose those through inheritance.
* Avoid mocks in the configuration management service test.
* Add tests for `ConfigManagementService.GetTimeModel`.
* Add tests for `ConfigManagementService.SetTimeModel`.
* Disable configuration management until the ledger has a configuration.
CHANGELOG_BEGIN
- [API Server] The configuration management service previously returned
the participant-specified default configuration if none was found on
the ledger. This can be misleading, and so the ``GetTimeModel``
endpoint now returns a gRPC ``NOT_FOUND`` error if no configuration
has been found yet. Similarly, the ``SetTimeModel`` endpoint returns a
gRPC ``UNAVAILABLE`` error.
This should only happen if the participant gives up waiting for the
ledger to provide a configuration, which is very unlikely in a
production setting. In this case, the ledger will also not respond to
command submissions.
CHANGELOG_END
* participant-integration-api: Extract some test variables for clarity.
In ApiConfigManagementServiceSpec.
* sandbox: Improve some variable and parameter names in tests for clarity.
* Improve the error message in case of a missing ledger configuration.
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* ledger-api-common: Improve an error status constructor name.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* participant-state: Give `ChangeId` its own file.
* participant-state: Introduce `v1.CompletionInfo` for a while.
`CompletionInfo` is identical to `SubmitterInfo`. Its purpose is to make
the migration to v2 easier. It should not make it into the final
version.
* participant-integration-api: Switch to the v2 participant state API.
This means that:
- the API server and indexer expect v2 API traits
- adapters are provided if you need to elevate your v1 API usage
- the indexer internally uses v2 Updates
- rejections are switched over to use the v2 format
- Sandbox Classic uses v2 as the internal representation too (because
it interacts directly with the underlying indexer representation,
and is therefore tightly coupled)
_kvutils_ and other users of the `StandaloneApiServer` and
`StandaloneIndexerServer` use the adapters.
CHANGELOG_BEGIN
- [Integration Kit] The API server and indexer have switched over to v2
of the participant-state API. You can continue to use the v1 API, but
you will need to wrap your ``ReadService`` and ``WriteService``
objects in the ``AdaptedV1ReadService`` and ``AdaptedV1WriteSerivce``
classes.
CHANGELOG_END
* participant-state: Remove v1.CompletionInfo.
It's served its purpose.
* kvutils: Remove an unnecessary line from `Runner`.
* ledger-api-common: Delete a TODO; we'll track it elsewhere.
* participant-integration-api: Use full words in `JdbcLedgerDao`.
Just a little bit of cleanup.
* ledger-api-common: Extract out the random submission ID generator.
And introduce a trait, because, well, this is still the JVM.
The conversion of SValue to Value already ensures the resulting value
has a serializable type. Here we add a check to ensure it does not
overpass the maximum allow nesting.
* participant-state: Remove the aliases to `Ref.LedgerString`.
Use the aliases directly in `Ref` instead.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lf/data: Improve comments on ID aliases.
* daml-lf/data: Fix a typo in a comment.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* participant-state: Remove the `ParticipantId` alias.
This alias adds nothing. By using `Ref.ParticipantId` directly, many
packages can remove their dependency on the _participant-state_ package.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-state: Remove the `PackageId` and `Party` aliases.
They don't add anything. Let's just use `Ref`.
* kvutils: Restore missing compat imports.
* ledger-offset: Move `Offset` to a new package.
CHANGELOG_BEGIN
- [Integration Kit] The ``Offset`` type has been moved to a new Maven
package, ``ledger-offset``, from the ``participant-state`` package.
The Java package has been renamed to ``com.daml.ledger.offset``. If
you are using this type, you will need to update your dependencies and
imports.
CHANGELOG_END
* Avoid rewrapping offsets for no reason.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* participant-integration-api: Sort some imports.
* participant-integration-api: Fix dependencies for the Oracle tests.
I didn't add `ledger-offset`.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* ledger-configuration: Extract configuration from participant-state.
The configuration is often used without the state, and doesn't need to
be versioned in the same way.
CHANGELOG_BEGIN
- [Integration Kit] The ledger configuration classes, ``Configuration``,
``LedgerInitialConditions``, and ``TimeModel``, have been moved from
*participant-state* to a separate package named
*ledger-configuration*, in the Java package
``com.daml.ledger.configuration``. You will need to update your
dependencies and imports.
CHANGELOG_END
* participant-state: Remove the `LedgerId` aliases.
* ledger-configuration: Rename `TimeModel` to `LedgerTimeModel`.
This avoids confusion with the protobuf-generated `TimeModel` classes.
CHANGELOG_BEGIN
- [Integration Kit] ``TimeModel`` has been renamed to
``LedgerTimeModel``. If you are using the ledger configuration classes
directly, you may need to update your code.
CHANGELOG_END
* ledger-configuration: Remove colons in LedgerInitialConditions' docs.
* kvutils: Restore a missing compat import.
* participant-integration-api: Add ledger-configuration to Oracle tests.
* sandbox-common: Fix `--max-ledger-time-skew` docs.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* LF: change type from Try to Either in archive module
This is the first part of restructuring errors in archive module.
This is part of #9974.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* remove type alias
* apply stephen suggestion
* fix after rebase
* fix test
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
I don't see a reason why it's part of the participant state API, and
it definitely doesn't need to change between v1 and v2.
CHANGELOG_BEGIN
- [Integration Kit] The class ``SeedService`` has been moved from the
*participant-state* Maven package to the *participant-integration-api*
Maven package, under the Java package name
``com.daml.platform.apiserver`` to reflect its usage by the API
server, not the participant state API. If you use this class directly,
you will need to change your imports.
CHANGELOG_END
