* Set ErrorInfo metadata flag for definite_answer, which is propagated from the completion status
CHANGELOG_BEGIN
ledger-api-client - Propagate definite_answer as metadata in the GRPC response for submit/submitAndWait
CHANGELOG_END
* Keep alphabetical order for bazel build files
* Add test for inclusion of metadata
* Formatting
* Use explicit types to track failures when submitting a request for execution
To distinguish from execution failures (represented by `CompletionFailure`) which is also exposed as part of the akka-bindings, we introduced `TrackingCompletionFailures` which can also represent the failure to add the request to the execution queue.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix formatting
* Apply suggestions from code review
Co-authored-by: Hubert Slojewski <hubert.slojewski@digitalasset.com>
* Inline handling of errors in the tracker to eliminate the need for the secondary promise and simplify the code
* Update testing for the new error handling
* Remove brackets and make code compatible with 2.12
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Review cleanup and use inside for cleaner tests
Co-authored-by: Hubert Slojewski <hubert.slojewski@digitalasset.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Upgrade Scalatest to v3.2.9.
Because of some coupling we also have to upgrade Scalaz to the latest
v7.2 point release, v7.2.33.
The Scalatest changes are quite involved because the JAR has been broken
up into several smaller JARs. Because Bazel expects us to specify all
dependencies and doesn't allow transitive dependencies to be used
directly, this means that we need to specify the explicit Scalatest
components that we use.
As you can imagine, this results in quite a big set of changes. They
are, however, constrained to dependency management; all the code remains
the same.
CHANGELOG_BEGIN
CHANGELOG_END
* http-json-oracle: Fix a Scalatest dependency.
* ledger-api-client: Fix a Scalatest dependency.
* ledger-api-test-tool: Add some basic unit tests for test names.
* ledger-api-test-tool: Ensure that test names are not prefixes of others.
This makes sure that we can include or exclude any given test, without
affecting others.
* ledger-api-test-tool:Ensure that all tests have different names.
Looks like we had some copy-pasta.
CHANGELOG_BEGIN
CHANGELOG_END
* Track command response using an Either instead of passing the completion with the grpc code.
This makes it clearer as to the result of command tracking. We no longer count on the grpc status to determine if there was an error or not, and instead use types for that.
CHANGELOG_BEGIN
akka-bindings: `LedgerClientBinding.commands` now returns a flow of `Either[CompletionFailure, CompletionSuccess]` instead of `Completion` for clearer error handling. For backwards compatiblity the new return type can be turned back into a `Completion` using `CompletionResponse.toCompletion`
CHANGELOG_END
* Fix formatting
* Code review changes
- remove usages of Symbol in tests
- clean curly braces
* Remove change added from another PR
* Fix import
* Fix import
* Fix retry flow and extract one more match case
* Un-nest matches to a single level for simplicity
* fix typo
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Be consistent in assertions and prefer `inside` for pattern matching
* Inline CompletionResponse to use the full type
* Use simpler matcher
* Formatting
* Add a way to convert back an `Either[CompletionFailure, CompletionSuccess]` to a `Completion` for backwards compatibility. This simplifies update for systems that are tightly coupled to `Completion`
* Add test for converting to/from CompletionResponse
* Remove unnecessary brackets
* Add missing header
* Use checked exceptions to preserve backwards compatiblity
* Fix unapply
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* kvutils: Add property-based tests for the offset builder.
* kvutils: Avoid using `BigInt` in offset parsing.
This also means we can theoretically support negative offsets, even if
in practice it's probably a bad idea.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Further property-based tests for OffsetBuilder.
* kvutils: Improve a comment in OffsetBuilder.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add flag to enable/disable command deduplication
* Remove flags for configuration as it should not be exposed externally
* Move deduplicationEnabled flag to the write service.
The deduplication enabled flag is tightly coupled to the WriteService implementation so the flag has been moved to the WriteService trait so that it's explicitly defined.
CHANGELOG_BEGIN
Command deduplication is enabled/disabled based on the write service implementation.
v1 WriteService enables command deduplication while v2 WriteService disabled it
CHANGELOG_END
* Rename deduplication flag
Set sandbox deduplication to true and proxy write service to the delegage
* H2 Storage backend support for canton jdbc urls with user/password
The H2 database's JdbcDataSource does not like seeing user/password
properties in the jdbc url raising errors upon hikari connection pool
initialization:
`org.h2.jdbc.JdbcSQLNonTransientConnectionException: Duplicate property "USER"`
expecting to be able to set user and password separately from passing on
the jdbc url.
As h2 is not supported in production and to get canton integration tests past this
resorting to parsing the jdbc url looking for user/password properties, removing
them from the url and instead setting them explicitly on the data source object.
changelog_begin
changelog_end
* Review feedback
* Add tests for rejections in LedgerConfigurationSubscriptionFromIndex.
* Create a proper state enum in LedgerConfigurationSubscriptionFromIndex.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Do less when resetting.
We don't need to reconstruct some basic things.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: When testing the reset service, keep trying.
If we scale up the test duration, we also need to scale up the timeouts.
* timer-utils: If `RetryStrategy` eventually fails, explain what happened.
For example, if the attempts limit is exceeded, say so.
* Revert "timer-utils: If `RetryStrategy` eventually fails, explain what happened."
This reverts commit 6647681688.
* participant-integration-api: Always wait for the first config lookup.
This means we don't have to guess how long a `lookupConfiguration()`
round trip will take. Instead we can just submit the initial
configuration if one is not found.
Some drivers may still need to wait a while for other reasons.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Explain `delayBeforeSubmitting`.
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* participant-integration-api: Make the initial ledger config optional.
CHANGELOG_BEGIN
- [Integration Kit] The initial ledger configuration is now optional;
if it is not specified, the participant will not attempt to submit a
configuration, but instead wait for something else to provision the
ledger with a configuration.
This behavior is intended to be used by ledger drivers that have
alternative means of setting up the initial ledger configuration. If
no configuration is provisioned, the participant server will not be
able to submit commands to the ledger.
CHANGELOG_END
* participant-integration-api: Avoid `Option#zip`.
It returns an `Iterable` in Scala 2.12, which is not helpful.
* Add optional submission id to commands.proto
This allows to propagate a submission id. If no id is submitted (the submission id is empty) then we generate a new submission id
CHANGELOG_BEGIN
Add optional submission_id to the commands.proto.
CHANGELOG_END
* Update haskell bindings to include the submission id
* Code review - rename submission id extractor
* Code review - update comment and remove braces from if block
* Fix braces
* participant-integration-api: Encapsulate the initial configuration.
* participant-integration-api: Reduce usage of `LedgerConfiguration`.
* Inline `LedgerConfiguration` wherever it's used.
Most things don't need all its constituent parts; this reduces the
amount of unused properties.
CHANGELOG_BEGIN
- [Integration Kit] The ``LedgerConfiguration`` class has been
removed in favor of ``InitialLedgerConfiguration``. Its usage
has been changed accordingly, with the ``configurationLoadTimeout``
property becoming part of ``ApiServerConfig`` instead.
The default options provided by ``LedgerConfiguration`` have been
removed; you are now encouraged to come up with sensible values for
your own ledger. The ``Configuration.reasonableInitialConfiguration``
value may help.
CHANGELOG_END
* Correct the initial configuration submission delay for KV ledgers.
* kvutils: Mark supertype unused parameters as unused.
* kvutils: Extract out common configuration submission delays.
These values are specific to kvutils; other drivers should come up with
their own.
* configuration: Delete `NoGeneration`, as it's unused.
* participant-integration-api: Move test constants to a companion object.
TelemetrySpecBase doesn't need to expose those through inheritance.
* Avoid mocks in the configuration management service test.
* Add tests for `ConfigManagementService.GetTimeModel`.
* Add tests for `ConfigManagementService.SetTimeModel`.
* Disable configuration management until the ledger has a configuration.
CHANGELOG_BEGIN
- [API Server] The configuration management service previously returned
the participant-specified default configuration if none was found on
the ledger. This can be misleading, and so the ``GetTimeModel``
endpoint now returns a gRPC ``NOT_FOUND`` error if no configuration
has been found yet. Similarly, the ``SetTimeModel`` endpoint returns a
gRPC ``UNAVAILABLE`` error.
This should only happen if the participant gives up waiting for the
ledger to provide a configuration, which is very unlikely in a
production setting. In this case, the ledger will also not respond to
command submissions.
CHANGELOG_END
* participant-integration-api: Extract some test variables for clarity.
In ApiConfigManagementServiceSpec.
* sandbox: Improve some variable and parameter names in tests for clarity.
* Improve the error message in case of a missing ledger configuration.
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* ledger-api-common: Improve an error status constructor name.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Indexer db MigrateOnEmptySchemaAndStart startup mode and migrateOnly hook
changelog_begin
Indexer: The indexer now supports the MigrateOnEmptySchemaAndStart startup mode that only performs migrations
on brand-new databases. In addition "real upgrades" are now supported via a new "StandaloneIndexerServer.migrateOnly"
entry point.
changelog_end
* Adding `additionalMigrationPaths` facility for testing on canton side
part of #9974
- rename `msg` field to `message`
- drop `engine.Error.Validation.Generic`
- rename argument `where` to `location`
- Builtin set the `dependsOnTime` before throwing the callback `SResultNeedTime`
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Use `Scheduler` instead of `Materializer`.
Simpler API, and we can inject a test scheduler to make the tests more
reliable.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Use a test scheduler in tests.
This makes the tests faster and more reliable.
* participant-integration-api: Cancel timeout when the config is found.
* participant-integration-api: Fail properly if config lookup fails.
* participant-integration-api: Handle failures in provisioning config.
* participant-integration-api: Test shutting down the config provisioner.
* participant-integration-api: Use the scheduler cancellations.
More useful than a boolean.
* participant-integration-api: Handle submission ID generation failure.
Unfortunately this is untestable, because the only output is logging,
which we can't test right now.
* resources-akka: Add `ResourceOwner.forCancellable`.
* participant-integration-api: Simplify the config provisioner.
This makes it a set of functions; we don't need a class. There is no
exposed behavior.
* participant-integration-api: Use the services EC to initialize config.
* participant-integration-api: Add comments around `release`.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Revert "participant-integration-api: Handle submission ID generation failure."
This reverts commit 72b13771a7.
* participant-integration-api: Factor out a `LedgerConfigProvider` class.
Again.
TBH, it is cleaner than multiple parameter methods.
Also adds more Scaladoc.
* resources-akka: `ResourceOwner.forCancellable` is now generic.
It returns whatever type the `acquire` function returns, to allow for
subtypes of `Cancellable`.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* participant-integration-api: Split `LedgerConfigProvider` into two.
We now have a read-only `CurrentLedgerConfiguration`, and a write-only
`LedgerConfigProvisioner`.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Improve logging in LedgerConfigProvisioner.
Making use of the logging context.
* participant-integration-api: Extract a CurrentLedgerConfiguration trait.
This highlighted a few issues in the tests where we were unnecessarily
calling methods.
* participant-integration-api: Split LedgerConfigProviderSpec.
We can test the two components individually now.
* participant-integration-api: Add a test for streaming the config.
* participant-integration-api: Add tests for provisioning configuration.
* participant-integration-api: Move configuration code into a new package.
* participant-integration-api: Fake ledger config in tests.
Now that there's an interface, we can use it without worrying about
streaming from the index.
* participant-integration-api: Use `List`, not `Seq`, in test fixtures.
Scala 2.12 thinks `Seq.apply` yields a possibly-mutable sequence, which
is incompatible with `Source.apply`.
* participant-integration-api: Explain how `LedgerConfigProvider` works.
* Stop execution when conflicts are detected and remove any retries.
As the current logic for retries had a flaw where we would continue the flow even if we had conflicts, we're removing the retries and fixing the flaw by dropping the transaction. In the future we might add back the retries if we need them.
CHANGELOG_BEGIN
Fix bug where post-execution conflict detection was ignored in the pre-execution workflow
CHANGELOG_END
* Extract common pre-execution test code
* Add test for pre-execution conflict detection
* Finish submission aggregator only on actual success
* Code review formatting
* Use recover to handle conflicts, simpler and clearer than transform
* Update test to verify submission is not submitted when dropping transaction
* logging-entries: Make `LoggingEntries` a non-case class.
There's no reason for it to need `equals`, etc.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-domain: Convert commands into a logging value.
Instead of having a function, let's use `ToLoggingContext`.
This also adds a couple of missing items, and always logs `workflowId`.
* participant-state: Convert updates into a logging value.
Instead of having a function, let's use `ToLoggingContext`.
This changes some of the logging context structure, but otherwise
everything remains the same.
* Make sure Scaladoc is lined up for modified code.
- Drop the wrapper `Versioned` and use directly the `VersionedValue` proto class. Values
should not be handled alone outside the Engine.
- Add benchmarks to aggregate (de)serialization and (en/de)coding
- This also fixes a recursion limit decoding issue that appears with #10393.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-state: Give `ChangeId` its own file.
* participant-state: Introduce `v1.CompletionInfo` for a while.
`CompletionInfo` is identical to `SubmitterInfo`. Its purpose is to make
the migration to v2 easier. It should not make it into the final
version.
* participant-integration-api: Switch to the v2 participant state API.
This means that:
- the API server and indexer expect v2 API traits
- adapters are provided if you need to elevate your v1 API usage
- the indexer internally uses v2 Updates
- rejections are switched over to use the v2 format
- Sandbox Classic uses v2 as the internal representation too (because
it interacts directly with the underlying indexer representation,
and is therefore tightly coupled)
_kvutils_ and other users of the `StandaloneApiServer` and
`StandaloneIndexerServer` use the adapters.
CHANGELOG_BEGIN
- [Integration Kit] The API server and indexer have switched over to v2
of the participant-state API. You can continue to use the v1 API, but
you will need to wrap your ``ReadService`` and ``WriteService``
objects in the ``AdaptedV1ReadService`` and ``AdaptedV1WriteSerivce``
classes.
CHANGELOG_END
* participant-state: Remove v1.CompletionInfo.
It's served its purpose.
* kvutils: Remove an unnecessary line from `Runner`.
* ledger-api-common: Delete a TODO; we'll track it elsewhere.
* participant-integration-api: Use full words in `JdbcLedgerDao`.
Just a little bit of cleanup.
* ledger-api-common: Extract out the random submission ID generator.
And introduce a trait, because, well, this is still the JVM.
CHANGELOG_BEGIN
[Integration Kit] Moved definitions of `DamlSubmission` and `DamlSubmissionBatch` to a separate proto file under the package `com.daml.ledger.participant.state.kvutils.wire`. In case you are directly referencing these messages you will have to update your imports.
CHANGELOG_END
* Stable packages bypass version restrictions.
Part of #10029
changelog_begin
changelog_end
* Add missing bypass
* scalafmt
* Fix tests
* fix script runner
* Fix scenario service
* Dont make stable packages configurable
* unused import
* revert unnecessary formatting changes
* Added a StableOnly engine mode and a conformance test that uses it
* buildifier-fix
* Inline LanguageVersions.StableOnly
* scalafmt
* Remove EngineMode in favor of representing the version range directly
* Add -unsafe to flag name
* Apply suggestions from code review
* participant-integration-api: Keep the state API prefixed.
So we can easily change the version.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-classic: Keep the state API prefixed.
