CHANGELOG_BEGIN
ledger-api - Command deduplication period can now be specified by setting `deduplication_offset` instead of `deduplication_time` (only valid for v2 WriteService). This change is backwards compatible.
CHANGELOG_END
* Propagate the enriched deduplicationPeriod instead of deduplication duration
* Update the Haskell bindings for the new deduplication period
* Calculate the deduplicateUntil using the new deduplication period for backward compat
* Use consistent naming for deduplication_period
* Cleanup command timeout extraction from deduplication period
* Add the required deduplication_offset to deduplication instead of deduplication_start
* Update haskell bindings to support deduplication_offset
* Add support for deduplication_offset in the ledger-api
* Remove the timestamp-based deduplication from our models to simplify upgrade for users
* Add optional conformance test for offset based deduplication
* Remove buf rule for FIELD_SAME_ONEOF as our change is backwards compatible
* Disable FIELD_SAME_ONEOF buf check for commands file
* Apply suggestions from code review
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Update comment for deduplication period
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* participant-integration-api: Clean up failed trackers.
Otherwise they can hang around forever.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Ensure that all trackers are closed.
* participant-integration-api: Ensure that waiting trackers are closed.
* participant-integration-api: Store the tracker map future in the state.
* participant-integration-api: Fix a race in `TrackerMap`.
If the supplied `Future` to `AsyncResourceState` is too fast, the state
may not be set. We need to initialize the state immediately.
* participant-integration-api: Add more comments to `TrackerMap`.
Create normalized TXs when a partial TX is finalised.
Except in limited cases! (i.e for scenario-runner, sandbox)
CHANGELOG_BEGIN
CHANGELOG_END
normalize values in the engine as they are converted from speedy-values
fix 2.12 build
backout redundant change
ensure byKey field is correctly normalized when constructed by engine
rename flag: valueNormalization -> transactionNormalization
improve comment
delete commented-out code
rename: toValueNorm --> toNormalizedValue
rename: (SValue.) toValue --> toUnNormalizedValue
revert changes to ptx so that the interface to insertCreate() etc is Value-based (not SValue-based)
improve comments
respell: toUnNormalizedValue --> toUnnormalizedValue
fix build
* participant-integration-api: Make `TrackerMap` a `Tracker`.
Mostly by moving parameters around.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Clean up `TrackerMap` a little more.
* participant-integration-api: Make `TrackerMap` generic.
* participant-integration-api: Move `Tracker.WithLastSubmission`.
It's only used by `TrackerMap`, so now it's an internal class there.
* participant-integration-api: Only provide the key to `newTracker`.
* participant-integration-api: Subsume the `TrackerMap` cleanup schedule.
* participant-integration-api: Construct the tracker map outside.
* ledger-api-common: Remove some unnecessary braces.
* participant-integration-api: Prettify `TrackerMap` some more.
* participant-integration-api: Make `TrackerMap.selfCleaning` a class.
* participant-integration-api: Add some tests for TrackerMap.
* participant-integration-api: Convert a method to `Runnable`, 2.12-style.
Apparently underscores aren't good enough.
* ledger-api-client: Delete CompletionSuccess#unapply.
It doesn't work on Scala 2.12.
Preparation, small fixes
* Remove unnecessary TODO
* Fix FieldStrategy.idempotentInsert type parameters
Changing the DB-Schema
This change adapts the DB schema for all supported backends. After this change we only populate the party_entries table, and on the query side we reconstruct the state from this.
* Drop the party table
* Add indexing to party_entries
Adapting StorageBackend: ingestion
Since we only ingest party_entries, the party population needs to be removed.
* Drop the party table in ingestion code
* Fixes test
Adapting StorageBackend: queries
Queries needs to be adapted to construct the state from the read side.
* Rewrite queries.
* Fixes reset implementations.
Adapting JdbcLedgerDao
Since underlying storage changed, JdbcLedgerDao can be simplified: no special treatment needed with duplicate errors, since these errors are impossible to happen.
Removing JdbcLedgerDao tests, and adding a new test, testing the behavior of the new event-source party model. Please note: this database refactoring only applies to the append-only schema, so for the mutating schema the test is disabled.
During implementation a bug surfaced: it was not possible anymore to store the is_local information via JdbcLedgerDao.storePartyEntry. Although this bug is a minor issue, since that method is only used from single participant environment, still a fix was also implemented for this, by passing a magic participantId upon non-local party storage.
* Simplify storePartyEntry.
* Fixes bug introduced by append-only.
* adds/adapts tests
Refactoring: remove not used duplicateKeyError from StorageBackend
Changes to JdbcLedgerDao rendered this duplicateKeyError unused.
* Removes unused duplicateKeyError
Adapting sandbox-classic
In sandbox-classic it is not allowed to have updates for parties. Essentially the updates concerning already existent parties were dropped silently with logging without effect.
Here I started by pinning down this behaviour in the SqlLedgerSpec and SqlLedgerSpecAppendOnly. These tests were implemented with the original code in mind.
Then adapted the SqlLedger method: making sure of uniqueness by first trying to lookup the to-be-persisted party.
* Added tests grabbing a hold on original behavior
* Adapted implementation to ensure same behavior
Switching to correct is_local derivation for party queries as per review
* Adapting implementation: switching to aggregated table and a query on that
* Introducing QueryStrategy.booleanOrAggregationFunction to support Oracle
* Moving party related queries to PartyStorageBackendTemplate
* Fixes JdbcLedgerDaoPartiesSpec tests, and add another test case
Also:
* Align Update interface documentation
* Switching to explicit optionality in party query implementation asa per review
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
CHANGELOG_BEGIN
CHANGELOG_END
This PR makes possible to check for contract IDs suffix during
preprocessing.
This is the first part of the task 3 described in #10504.
CHANGELOG_BEGIN
CHANGELOG_END
* Augment completion.proto with deduplication-related info
CHANGELOG_BEGIN
CHANGELOG_END
* Explicitly specify fields not yet filled in when building Completion
* Time-based deduplication periods are measured in record time of completions
* Add deduplication_offset as a deduplication_period option
* Don't skip proto field numbers
* CompletionFromTransaction: use default Completion constructor
* submission_rank: reserve proto field for future use
* Add comment about reserved proto field
Previously, if the max deduplication time was extended, the participant
_might_ retain the old time for certain submissions.
CHANGELOG_BEGIN
- [Ledger API Server] The API server manages a single command tracker
per (application ID × submitters) pair. This tracker would read the
current ledger configuration's maximum deduplication time on creation,
but never updated it, leading to trackers that might inadvertently
reject a submission when it should have been accepted. The tracker now
reads the latest ledger configuration.
CHANGELOG_END
* Time conversion duration between buffer event and API domain transaction
* Compute partiesTemplates inversion mapping outside events transformation
* Other small optimizations
CHANGELOG_BEGIN
CHANGELOG_END
* Add StorageBackend tests
changelog_begin
changelog_end
* Fix Oracle tests
* Do not use empty byte arrays
* Format
* Fix after rebase
* Substitute type params with type bounded ingest method
* Remove empty line
* Assert on configuration contents
* Fix Oracle build
* Add tests for ingestion initialization
* fmt
* Add test for leftover data after reset
* Add resetAll
* Use resetAll between tests
Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
* participant-integration-api: Add logging to RecoveringIndexerSpec.
It's flaky, and I would like to know exactly what's going on here.
* participant-integration-api: Attempt to fix RecoveringIndexerSpec.
The checks for the health status are racy. I'm hoping increasing the
timeouts will help a little.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Inline timeouts in RecoveringIndexerSpec.
The `LedgerConfigurationSubscriptionFromIndexSpec` was flaky due to
stubs not specifying all behavior for `IndexConfigManagementService`.
This fixes the underlying issue by avoiding stubs in favor of fakes,
which means that _all_ behavior must be modelled.
Note: Martin Fowler has an excellent, terse description of [the various
forms of test doubles][TestDouble].
[TestDouble]: https://www.martinfowler.com/bliki/TestDouble.html
CHANGELOG_BEGIN
CHANGELOG_END
* Upgrade to a newer canton version (post 0.27.0 snapshot version)
with canton-community configuration that supports higher throughput.
changelog_begin
changelog_end
* Disable flaky reject DeeplyNestedValueIT:Reject tests that time out half the time
As stated in #10504 the contract ID freshness check cannot be
implemented correctly in general.
This PR drops the support for this (buggy) check.
This corresponds to the fist task of #10504.
CHANGELOG_BEGIN
CHANGELOG_END
This was limited a while back during the initial development of
_ledger-on-sql_ for reasons I can't remember. Let's stop doing that.
CHANGELOG_BEGIN
CHANGELOG_END
* Refactor ParameterStorageBackend
- a single method for atomic ledger initialization
- a single method to look up the ledger end
changelog_begin
changelog_end
* Add a test
* Fix reading event sequential ids
* Remove debug statements
* Allow ledgerEnd on an empty database
* Initialization is not safe to call concurrently
* Remove leftovers from isolation level change
* Use unit return type
for initialization methods
* Allow getParticipantId on an empty database
* Use exceptions instead of a return type ADT
* Don't use Try for initialization
* Clean up parameters table
* Simplify parameter storage api
* Address review suggestion
* Address review comment
* Address review comment
* Prefer ledger id over participant id mismatch
* Address review comment
* Move type definition
* Remove useleess new keyword
* Renove unused import
* Inline result mapping
* Fix reporting of mismatching participantId
* participant-integration-api: Construct completions in one place.
* sandbox-classic: Inline `CompletionFromTransaction#apply`.
It's only used here; there's no reason to keep it in the
_participant-integration-api_.
* participant-integration-api: Store a status gRPC protobuf.
Instead of storing the status code and message, we store a serialized
`google.rpc.Status` protocol buffers message. This allows us to pass
through any additional information reported by the driver `ReadService`.
The migration is only done for the append-only database, and preserves
old data in the existing columns. New data will only be written to the
new column.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Improve comments in migrations.
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* participant-integration-api: Further improvements to migrations.
* participant-integration-api: Store the rejection status as 3 columns.
Serializing the details but keeping the code and message columns
populated.
* participant-integration-api: Publish the indexer protobuf to Maven.
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* Use `extra` in the port file runner, rather than `temporary`.
* ledger-api-test-tool-on-canton: Use the port check runner.
Much simpler than the port file runner for our purposes.
* Replace `runner` with `runner_with_port_file`.
Rather than expecting a particular set of command-line-arguments, we use
templating.
CHANGELOG_BEGIN
CHANGELOG_END
* Rename the `runner_with_port_check` target to the default.
* Use the port file and dynamic port generation in client/server tests.
This creates a runner named `runner_with_port_file` which knows how to
interpolate two variables, `%PORT_FILE%` and `%PORT%`. This allows us to
use the `port-file` argument to the kvutils runner rather than
hard-coding a port for conformance tests.
For now, we only use this for generating the kvutils reference ledger
export.
CHANGELOG_BEGIN
CHANGELOG_END
* Simplify the runner_with_port_file considerably.
It doesn't need to check if the port is open; we trust that the process
will do it.
This also makes sure the port file will be cleaned up, and reduces the
number of dependencies by making use of more functions in `extra`.
* Simplify port file generation in the new client-server runner.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Simplify the runner_with_port_file further.
This doesn't need to work if the server doesn't take a port file.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Move `DeduplicationPeriod` to ledger-api-domain so that it ca be reused and passed down from the ledger-api-client to into the v2.SubmitterInfo
CHANGELOG_BEGIN
participant-state - move `DeduplicationPeriod` to ledger-api-domain
CHANGELOG_END
* Revert unrelated changes
* ledger-api-test-tool: Make IntelliJ happy with TransactionServiceIT.
* ledger-api-test-tool: Collect TransactionServiceIT tests into groups.
* ledger-api-test-tool: Split TransactionServiceIT into lots of suites.
CHANGELOG_BEGIN
- [Ledger API Test Tool] The ``TransactionServiceIT`` test suite has
been split into many test suites. If you are including or excluding
it, you will need to use the new test suite names, or you can use
"TransactionService" as a prefix for all of them.
If you are including or excluding individual tests, you will need to
update your arguments with the new test suite. You can find the new
test suite by running the test tool with the ``--list-all``
flag and looking for the test's short identifier. The short
identifiers have not changed, with the exception of
``TXNoContractKey``, which has been renamed to ``CKNoContractKey`` and
is now in the ``ContractKeysIT`` test suite.
CHANGELOG_END
* ledger-grpc: Fix the directory paths.
This brings _ledger-grpc_ in line with other projects. Scala main files
should be under "src/main/scala", and test files should be under
"src/test/suite/scala".
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-grpc: Moar imports.
* Normalize transactions & values as a separate pass. Use for simpler defintiion of isReplayedBy.
CHANGELOG_BEGIN
CHANGELOG_END
normalize transaction version
* remove stray import from bad merge which breaks scala 2_12 build
* change isReplayedBy to only norm its RIGHT (replay) argument
* add forgotton normalization for ValueEmum
* switch to use existing value normalization code (remove my newly coded duplicate code)
* normalize submittedTransaction before calling engine.validate
* dont call normalizeTx from Engine.validate
* *do* call normalizeTx from Engine.validate
* Set ErrorInfo metadata flag for definite_answer, which is propagated from the completion status
CHANGELOG_BEGIN
ledger-api-client - Propagate definite_answer as metadata in the GRPC response for submit/submitAndWait
CHANGELOG_END
* Keep alphabetical order for bazel build files
* Add test for inclusion of metadata
* Formatting
* Use explicit types to track failures when submitting a request for execution
To distinguish from execution failures (represented by `CompletionFailure`) which is also exposed as part of the akka-bindings, we introduced `TrackingCompletionFailures` which can also represent the failure to add the request to the execution queue.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix formatting
* Apply suggestions from code review
Co-authored-by: Hubert Slojewski <hubert.slojewski@digitalasset.com>
* Inline handling of errors in the tracker to eliminate the need for the secondary promise and simplify the code
* Update testing for the new error handling
* Remove brackets and make code compatible with 2.12
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Review cleanup and use inside for cleaner tests
Co-authored-by: Hubert Slojewski <hubert.slojewski@digitalasset.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Upgrade Scalatest to v3.2.9.
Because of some coupling we also have to upgrade Scalaz to the latest
v7.2 point release, v7.2.33.
The Scalatest changes are quite involved because the JAR has been broken
up into several smaller JARs. Because Bazel expects us to specify all
dependencies and doesn't allow transitive dependencies to be used
directly, this means that we need to specify the explicit Scalatest
components that we use.
As you can imagine, this results in quite a big set of changes. They
are, however, constrained to dependency management; all the code remains
the same.
CHANGELOG_BEGIN
CHANGELOG_END
* http-json-oracle: Fix a Scalatest dependency.
* ledger-api-client: Fix a Scalatest dependency.
* ledger-api-test-tool: Add some basic unit tests for test names.
* ledger-api-test-tool: Ensure that test names are not prefixes of others.
This makes sure that we can include or exclude any given test, without
affecting others.
* ledger-api-test-tool:Ensure that all tests have different names.
Looks like we had some copy-pasta.
CHANGELOG_BEGIN
CHANGELOG_END
* Track command response using an Either instead of passing the completion with the grpc code.
This makes it clearer as to the result of command tracking. We no longer count on the grpc status to determine if there was an error or not, and instead use types for that.
CHANGELOG_BEGIN
akka-bindings: `LedgerClientBinding.commands` now returns a flow of `Either[CompletionFailure, CompletionSuccess]` instead of `Completion` for clearer error handling. For backwards compatiblity the new return type can be turned back into a `Completion` using `CompletionResponse.toCompletion`
CHANGELOG_END
* Fix formatting
* Code review changes
- remove usages of Symbol in tests
- clean curly braces
* Remove change added from another PR
* Fix import
* Fix import
* Fix retry flow and extract one more match case
* Un-nest matches to a single level for simplicity
* fix typo
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Be consistent in assertions and prefer `inside` for pattern matching
* Inline CompletionResponse to use the full type
* Use simpler matcher
* Formatting
* Add a way to convert back an `Either[CompletionFailure, CompletionSuccess]` to a `Completion` for backwards compatibility. This simplifies update for systems that are tightly coupled to `Completion`
* Add test for converting to/from CompletionResponse
* Remove unnecessary brackets
* Add missing header
* Use checked exceptions to preserve backwards compatiblity
* Fix unapply
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* kvutils: Add property-based tests for the offset builder.
* kvutils: Avoid using `BigInt` in offset parsing.
This also means we can theoretically support negative offsets, even if
in practice it's probably a bad idea.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Further property-based tests for OffsetBuilder.
* kvutils: Improve a comment in OffsetBuilder.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Add flag to enable/disable command deduplication
* Remove flags for configuration as it should not be exposed externally
* Move deduplicationEnabled flag to the write service.
The deduplication enabled flag is tightly coupled to the WriteService implementation so the flag has been moved to the WriteService trait so that it's explicitly defined.
CHANGELOG_BEGIN
Command deduplication is enabled/disabled based on the write service implementation.
v1 WriteService enables command deduplication while v2 WriteService disabled it
CHANGELOG_END
* Rename deduplication flag
Set sandbox deduplication to true and proxy write service to the delegage
* H2 Storage backend support for canton jdbc urls with user/password
The H2 database's JdbcDataSource does not like seeing user/password
properties in the jdbc url raising errors upon hikari connection pool
initialization:
`org.h2.jdbc.JdbcSQLNonTransientConnectionException: Duplicate property "USER"`
expecting to be able to set user and password separately from passing on
the jdbc url.
As h2 is not supported in production and to get canton integration tests past this
resorting to parsing the jdbc url looking for user/password properties, removing
them from the url and instead setting them explicitly on the data source object.
changelog_begin
changelog_end
* Review feedback
* Add tests for rejections in LedgerConfigurationSubscriptionFromIndex.
* Create a proper state enum in LedgerConfigurationSubscriptionFromIndex.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Do less when resetting.
We don't need to reconstruct some basic things.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: When testing the reset service, keep trying.
If we scale up the test duration, we also need to scale up the timeouts.
* timer-utils: If `RetryStrategy` eventually fails, explain what happened.
For example, if the attempts limit is exceeded, say so.
* Revert "timer-utils: If `RetryStrategy` eventually fails, explain what happened."
This reverts commit 6647681688.
* participant-integration-api: Always wait for the first config lookup.
This means we don't have to guess how long a `lookupConfiguration()`
round trip will take. Instead we can just submit the initial
configuration if one is not found.
Some drivers may still need to wait a while for other reasons.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Explain `delayBeforeSubmitting`.
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* participant-integration-api: Make the initial ledger config optional.
CHANGELOG_BEGIN
- [Integration Kit] The initial ledger configuration is now optional;
if it is not specified, the participant will not attempt to submit a
configuration, but instead wait for something else to provision the
ledger with a configuration.
This behavior is intended to be used by ledger drivers that have
alternative means of setting up the initial ledger configuration. If
no configuration is provisioned, the participant server will not be
able to submit commands to the ledger.
CHANGELOG_END
* participant-integration-api: Avoid `Option#zip`.
It returns an `Iterable` in Scala 2.12, which is not helpful.
* Add optional submission id to commands.proto
This allows to propagate a submission id. If no id is submitted (the submission id is empty) then we generate a new submission id
CHANGELOG_BEGIN
Add optional submission_id to the commands.proto.
CHANGELOG_END
* Update haskell bindings to include the submission id
* Code review - rename submission id extractor
* Code review - update comment and remove braces from if block
* Fix braces
* participant-integration-api: Encapsulate the initial configuration.
* participant-integration-api: Reduce usage of `LedgerConfiguration`.
* Inline `LedgerConfiguration` wherever it's used.
Most things don't need all its constituent parts; this reduces the
amount of unused properties.
CHANGELOG_BEGIN
- [Integration Kit] The ``LedgerConfiguration`` class has been
removed in favor of ``InitialLedgerConfiguration``. Its usage
has been changed accordingly, with the ``configurationLoadTimeout``
property becoming part of ``ApiServerConfig`` instead.
The default options provided by ``LedgerConfiguration`` have been
removed; you are now encouraged to come up with sensible values for
your own ledger. The ``Configuration.reasonableInitialConfiguration``
value may help.
CHANGELOG_END
* Correct the initial configuration submission delay for KV ledgers.
* kvutils: Mark supertype unused parameters as unused.
* kvutils: Extract out common configuration submission delays.
These values are specific to kvutils; other drivers should come up with
their own.
* configuration: Delete `NoGeneration`, as it's unused.
* participant-integration-api: Move test constants to a companion object.
TelemetrySpecBase doesn't need to expose those through inheritance.
* Avoid mocks in the configuration management service test.
* Add tests for `ConfigManagementService.GetTimeModel`.
* Add tests for `ConfigManagementService.SetTimeModel`.
* Disable configuration management until the ledger has a configuration.
CHANGELOG_BEGIN
- [API Server] The configuration management service previously returned
the participant-specified default configuration if none was found on
the ledger. This can be misleading, and so the ``GetTimeModel``
endpoint now returns a gRPC ``NOT_FOUND`` error if no configuration
has been found yet. Similarly, the ``SetTimeModel`` endpoint returns a
gRPC ``UNAVAILABLE`` error.
This should only happen if the participant gives up waiting for the
ledger to provide a configuration, which is very unlikely in a
production setting. In this case, the ledger will also not respond to
command submissions.
CHANGELOG_END
* participant-integration-api: Extract some test variables for clarity.
In ApiConfigManagementServiceSpec.
* sandbox: Improve some variable and parameter names in tests for clarity.
* Improve the error message in case of a missing ledger configuration.
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* ledger-api-common: Improve an error status constructor name.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Indexer db MigrateOnEmptySchemaAndStart startup mode and migrateOnly hook
changelog_begin
Indexer: The indexer now supports the MigrateOnEmptySchemaAndStart startup mode that only performs migrations
on brand-new databases. In addition "real upgrades" are now supported via a new "StandaloneIndexerServer.migrateOnly"
entry point.
changelog_end
* Adding `additionalMigrationPaths` facility for testing on canton side
part of #9974
- rename `msg` field to `message`
- drop `engine.Error.Validation.Generic`
- rename argument `where` to `location`
- Builtin set the `dependsOnTime` before throwing the callback `SResultNeedTime`
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Use `Scheduler` instead of `Materializer`.
Simpler API, and we can inject a test scheduler to make the tests more
reliable.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Use a test scheduler in tests.
This makes the tests faster and more reliable.
* participant-integration-api: Cancel timeout when the config is found.
* participant-integration-api: Fail properly if config lookup fails.
* participant-integration-api: Handle failures in provisioning config.
* participant-integration-api: Test shutting down the config provisioner.
* participant-integration-api: Use the scheduler cancellations.
More useful than a boolean.
* participant-integration-api: Handle submission ID generation failure.
Unfortunately this is untestable, because the only output is logging,
which we can't test right now.
* resources-akka: Add `ResourceOwner.forCancellable`.
* participant-integration-api: Simplify the config provisioner.
This makes it a set of functions; we don't need a class. There is no
exposed behavior.
* participant-integration-api: Use the services EC to initialize config.
* participant-integration-api: Add comments around `release`.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* Revert "participant-integration-api: Handle submission ID generation failure."
This reverts commit 72b13771a7.
* participant-integration-api: Factor out a `LedgerConfigProvider` class.
Again.
TBH, it is cleaner than multiple parameter methods.
Also adds more Scaladoc.
* resources-akka: `ResourceOwner.forCancellable` is now generic.
It returns whatever type the `acquire` function returns, to allow for
subtypes of `Cancellable`.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* participant-integration-api: Split `LedgerConfigProvider` into two.
We now have a read-only `CurrentLedgerConfiguration`, and a write-only
`LedgerConfigProvisioner`.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Improve logging in LedgerConfigProvisioner.
Making use of the logging context.
* participant-integration-api: Extract a CurrentLedgerConfiguration trait.
This highlighted a few issues in the tests where we were unnecessarily
calling methods.
* participant-integration-api: Split LedgerConfigProviderSpec.
We can test the two components individually now.
* participant-integration-api: Add a test for streaming the config.
* participant-integration-api: Add tests for provisioning configuration.
* participant-integration-api: Move configuration code into a new package.
* participant-integration-api: Fake ledger config in tests.
Now that there's an interface, we can use it without worrying about
streaming from the index.
* participant-integration-api: Use `List`, not `Seq`, in test fixtures.
Scala 2.12 thinks `Seq.apply` yields a possibly-mutable sequence, which
is incompatible with `Source.apply`.
* participant-integration-api: Explain how `LedgerConfigProvider` works.
* Stop execution when conflicts are detected and remove any retries.
As the current logic for retries had a flaw where we would continue the flow even if we had conflicts, we're removing the retries and fixing the flaw by dropping the transaction. In the future we might add back the retries if we need them.
CHANGELOG_BEGIN
Fix bug where post-execution conflict detection was ignored in the pre-execution workflow
CHANGELOG_END
* Extract common pre-execution test code
* Add test for pre-execution conflict detection
* Finish submission aggregator only on actual success
* Code review formatting
* Use recover to handle conflicts, simpler and clearer than transform
* Update test to verify submission is not submitted when dropping transaction
* logging-entries: Make `LoggingEntries` a non-case class.
There's no reason for it to need `equals`, etc.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-domain: Convert commands into a logging value.
Instead of having a function, let's use `ToLoggingContext`.
This also adds a couple of missing items, and always logs `workflowId`.
* participant-state: Convert updates into a logging value.
Instead of having a function, let's use `ToLoggingContext`.
This changes some of the logging context structure, but otherwise
everything remains the same.
* Make sure Scaladoc is lined up for modified code.
- Drop the wrapper `Versioned` and use directly the `VersionedValue` proto class. Values
should not be handled alone outside the Engine.
- Add benchmarks to aggregate (de)serialization and (en/de)coding
- This also fixes a recursion limit decoding issue that appears with #10393.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-state: Give `ChangeId` its own file.
* participant-state: Introduce `v1.CompletionInfo` for a while.
`CompletionInfo` is identical to `SubmitterInfo`. Its purpose is to make
the migration to v2 easier. It should not make it into the final
version.
* participant-integration-api: Switch to the v2 participant state API.
This means that:
- the API server and indexer expect v2 API traits
- adapters are provided if you need to elevate your v1 API usage
- the indexer internally uses v2 Updates
- rejections are switched over to use the v2 format
- Sandbox Classic uses v2 as the internal representation too (because
it interacts directly with the underlying indexer representation,
and is therefore tightly coupled)
_kvutils_ and other users of the `StandaloneApiServer` and
`StandaloneIndexerServer` use the adapters.
CHANGELOG_BEGIN
- [Integration Kit] The API server and indexer have switched over to v2
of the participant-state API. You can continue to use the v1 API, but
you will need to wrap your ``ReadService`` and ``WriteService``
objects in the ``AdaptedV1ReadService`` and ``AdaptedV1WriteSerivce``
classes.
CHANGELOG_END
* participant-state: Remove v1.CompletionInfo.
It's served its purpose.
* kvutils: Remove an unnecessary line from `Runner`.
* ledger-api-common: Delete a TODO; we'll track it elsewhere.
* participant-integration-api: Use full words in `JdbcLedgerDao`.
Just a little bit of cleanup.
* ledger-api-common: Extract out the random submission ID generator.
And introduce a trait, because, well, this is still the JVM.
CHANGELOG_BEGIN
[Integration Kit] Moved definitions of `DamlSubmission` and `DamlSubmissionBatch` to a separate proto file under the package `com.daml.ledger.participant.state.kvutils.wire`. In case you are directly referencing these messages you will have to update your imports.
CHANGELOG_END
* Stable packages bypass version restrictions.
Part of #10029
changelog_begin
changelog_end
* Add missing bypass
* scalafmt
* Fix tests
* fix script runner
* Fix scenario service
* Dont make stable packages configurable
* unused import
* revert unnecessary formatting changes
* Added a StableOnly engine mode and a conformance test that uses it
* buildifier-fix
* Inline LanguageVersions.StableOnly
* scalafmt
* Remove EngineMode in favor of representing the version range directly
* Add -unsafe to flag name
* Apply suggestions from code review
* participant-integration-api: Keep the state API prefixed.
So we can easily change the version.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-classic: Keep the state API prefixed.
* ledger-api-domain: Store the deduplication duration in `Commands`.
We can compute the max deduplication time from this later.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-domain: Refactor out `Commands#deduplicateUntil`.
* participant-integration-api: Reduce usage of the state RejectionReason.
Use a local `Rejection` enumeration instead.
This also changes an implicit conversion to be a somewhat-explicit
conversion.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-classic: Internalize rejection reasons.
Instead of using RejectionReasonV0, we can declare them locally.
* kvutils: Create internal rejection reasons in the committer.
These can be converted to state rejection reasons later.
By using internal rejection representations, we're not coupled to the
state API.
* sandbox-classic: Remove unused code from `Rejection`.
* participant-integration-api: Push the rejection reason conversion down.
Just a little bit further.
* participant-integration-api: Add tests for converting rejection reasons.
* participant-integration-api: Remove an unused test dependency.
* [Divulgence pruning] Conformance tests implementation
CHANGELOG_BEGIN
[Integration kit] Extended the Ledger API test tool with tests for the pruning of all divulgence events.
CHANGELOG_END
* Addressed review comments
The conversion of SValue to Value already ensures the resulting value
has a serializable type. Here we add a check to ensure it does not
overpass the maximum allow nesting.
* ledger-configuration: More rigorous tests for LedgerTimeModel.
Let's actually check the return values, not just the constructor.
* ledger-configuration: Create a type that represents time out of range.
Only used in testing for now.
* ledger-configuration: Return a structured error from `checkTime`.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-configuration: We like braces.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Removing previous Async commit features
Previous async commit features had
- async commit configured by conifg-param
- special treatments to stil force sync commit for certain threadpools
- special treatment to stil force sync commit on transaction level for certain transactions.
This is a preparation step to clean the path for adding a new approach for async commit treatment:
- only session/connection level async configuration
- no transaction level special treatments
- only enable async commit for specific Connection pools (where it is needed / is safe)
* Add DataSourceStorageBackend
- to spawn DataSources in a controlled fashion
these will be needed in upcoming commits for the HikariCP
- DataSources can have Connection init hooks defined with the help of the InitHookDataSourceProxy (this is needed for HA implementation)
- added DataSourceConfig to capture needed level of fine-tuning for DataSource creation
* Switches to DataSource wrapping in HikariCP instantiation
* Adds DBLockStorageBackend
- this is the abstraction and the implementation of database level locking
- with support for Oracle and Postgres
* Adds HaCoordinator and implementation
* Wiring of HaCoordinator in parallel indexer
* Adds feature flag
changelog_begin
changelog_end
* Re-add `application_id` to DamlCommandDedupKey
CHANGELOG_BEGIN
CHANGELOG_END
* Fix test
* Merge main
* commandDedupKey: reorder calls to proto builder setters according to proto order
* Empty commit to set the changelog
CHANGELOG_BEGIN
[Integration Kit] The command de-duplication key now also includes the daml application ID
CHANGELOG_END
* participant-integration-api: Fill out stubs in ApiSubmissionServiceSpec.
This test fails sometimes and it's hard to figure out why because of the
number of benign stack traces in the logs, caused by missing stubs.
By filling out the stubs so that `LedgerConfigProvider` can use them, we
get much clearer test logs, which should hopefully help us diagnose the
error properly next time the test fails in CI.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Fix flake in ApiSubmissionServiceSpec.
Writing the configuration is non-deterministic and unnecessary.
* participant-integration-api: Use resources in ApiSubmissionServiceSpec.
Not always the best idea to use a resource after it's been released. And
this isn't Rust, so we can't force that at compile time.
CHANGELOG_BEGIN
* [Integration Kit] Made `daml_kvutils.proto`'s location follow its proto package and moved `LedgerExportEntry` into a separate proto file. You may have to update your proto import statements in case you are directly importing proto files from the kvutils library.
CHANGELOG_END
* participant-state: Remove the aliases to `Ref.LedgerString`.
Use the aliases directly in `Ref` instead.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lf/data: Improve comments on ID aliases.
* daml-lf/data: Fix a typo in a comment.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* [Mutable cache] Resolve with full lookup on missed cache thread-throughs
* Do not store negative lookups
* Metrics for counting divulgence and full lookups
* Metrics for counting read-through not found
CHANGELOG_BEGIN
CHANGELOG_END
* Added comment detailing secondary lookups for divulgence
* Apply suggestions from code review
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
* daml-lf/data: Move ID aliases to `Ref` from _ledger-api-common_.
This allows us to remove a lot of dependencies on _ledger-api-common_,
and use these aliases in other places where that module is not used.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Remove an unused import.
* http-json-oracle: Remove `ledger-api-common` as a dependency.
* bindings-rxjava: Remove a now-unused dependency.
* participant-state: Remove the Daml-LF transaction aliases.
They add nothing.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-state: Move the `ContractInst` alias to where it's used.
Either inlining it or extracting out a similar alias elsewhere.
This further removes dependencies on _participant-state_.
* Fix a couple of typos.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* [DOCS] Add documentation for the JSON API metrics
changelog_begin
- [JSON-API] You can now find a section `Metrics` in the http-json api documentation explaining how to enable metrics and which are available
changelog_end
* Fix rst build warnings
* Update docs/source/json-api/metrics.rst
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Adapt metrics doc to state that it IS an exhaustive list and remove wrong copy pasta text & add info about prometheus
* Update the legal values for the metrics reporter cli option
* shorten the description, the change prior was unnecessary ._.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Remove Optional[Location] field from transaction nodes.
xoptLocation
working but very messy!
cleanup PartialTransaction code
pass locationInfo in CompleteTransaction
dont pass locationInfo in SumittedTransaction
pass optLocation to CheckAuthorization
temp reinstate xoptLocation in ExerciseContextInfo
cleanup Conversions.scala
remove Optional[Location] field from transaction nodes.
CHANGELOG_BEGIN
CHANGELOG_END
* fix bad merge
* fix build after merge
* rename
* doc comment, and question
* participant-state: Remove the `ParticipantId` alias.
This alias adds nothing. By using `Ref.ParticipantId` directly, many
packages can remove their dependency on the _participant-state_ package.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-state: Remove the `PackageId` and `Party` aliases.
They don't add anything. Let's just use `Ref`.
* kvutils: Restore missing compat imports.
* ledger-configuration: Correct the protobuf package.
CHANGELOG_BEGIN
- [Integration Kit] The *ledger_configuration.proto* Protobuf definition
has been repackaged under ``com.daml.ledger.configuration``, and the
Java and C# packages have been renamed accordingly. If you are using
this Protobuf definition, you will need to update your imports. The
Maven artifact name has not changed.
CHANGELOG_END
* ledger-configuration: Resolve a naming collision in Scala 2.12.
* Add a `--max-connection-attempts` command line option to the Ledger API Test Tool
BEGIN_CHANGELOG
- [Integration Kit] Add a `--max-connection-attempts` command line option to the Ledger API Test Tool
END_CHANGELOG
* Remove `.unbounded()` from the cmdline option
* fmt
* Restore `.unbounded()` in `[endpoints...]`
BEGIN_CHANGELOG
- [Integration Kit] Add a `--max-connection-attempts` command line option to the Ledger API Test Tool
END_CHANGELOG
* Remove stale comment
CHANGELOG_BEGIN
- [Integration Kit] Add a `--max-connection-attempts` command line option to the Ledger API Test Tool
CHANGELOG_END
* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/infrastructure/LedgerTestCasesRunner.scala
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* fmt
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
CHANGELOG_BEGIN
* [Integration Kit] Removed trace_context field from Ledger API and its bindings as we now have trace context propagation support via gRPC metadata. If you are constructing or consuming Ledger API requests or responses directly, you may need to update your code.
CHANGELOG_END
* ledger-configuration: Add tests for Configuration.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-configuration: Reject a negative maximum deduplication time.
* ledger-configuration: Remove duplication in Configuration.
* ledger-configuration: Make some Configuration methods private.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* ledger-configuration: Remove some extra braces.
* ledger-configuration: Use tables to simplify rejection tests.
* ledger-configuration: Add a test for an unknown version.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* ledger-offset: Move `Offset` to a new package.
CHANGELOG_BEGIN
- [Integration Kit] The ``Offset`` type has been moved to a new Maven
package, ``ledger-offset``, from the ``participant-state`` package.
The Java package has been renamed to ``com.daml.ledger.offset``. If
you are using this type, you will need to update your dependencies and
imports.
CHANGELOG_END
* Avoid rewrapping offsets for no reason.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* participant-integration-api: Sort some imports.
* participant-integration-api: Fix dependencies for the Oracle tests.
I didn't add `ledger-offset`.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* ledger-configuration: Extract configuration from participant-state.
The configuration is often used without the state, and doesn't need to
be versioned in the same way.
CHANGELOG_BEGIN
- [Integration Kit] The ledger configuration classes, ``Configuration``,
``LedgerInitialConditions``, and ``TimeModel``, have been moved from
*participant-state* to a separate package named
*ledger-configuration*, in the Java package
``com.daml.ledger.configuration``. You will need to update your
dependencies and imports.
CHANGELOG_END
* participant-state: Remove the `LedgerId` aliases.
* ledger-configuration: Rename `TimeModel` to `LedgerTimeModel`.
This avoids confusion with the protobuf-generated `TimeModel` classes.
CHANGELOG_BEGIN
- [Integration Kit] ``TimeModel`` has been renamed to
``LedgerTimeModel``. If you are using the ledger configuration classes
directly, you may need to update your code.
CHANGELOG_END
* ledger-configuration: Remove colons in LedgerInitialConditions' docs.
* kvutils: Restore a missing compat import.
* participant-integration-api: Add ledger-configuration to Oracle tests.
* sandbox-common: Fix `--max-ledger-time-skew` docs.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* [In-memory fan-out] Oracle support
CHANGELOG_BEGIN
CHANGELOG_END
* Correct so Oracle backend works with fan out enabled - add implicit conversion and correct view defaulting for BLOB fields
Add missing tests to oracle append only spec as it lagged behind postgres and h2
Regen sql sha for oracle
CHANGELOG_BEGIN
CHANGELOG_END
* avoid global import of implicit that is not needed
* reduce duplication on conformance test targets
* update sha for oracle append only flyway scripts
Co-authored-by: Brian Healey <brian.healey@digitalasset.com>
* LF: change type from Try to Either in archive module
This is the first part of restructuring errors in archive module.
This is part of #9974.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* remove type alias
* apply stephen suggestion
* fix after rebase
* fix test
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* CHANGELOG_BEGIN
Fix crash in ConfigManagement- and PackageManagement- services on duplicate submissionsIds from different participants.
Extend ledger-api-test-tool to cover the duplicate submissionId cases.
CHANGELOG_END
* Addressing review comments and fixing failing tests
* format
* address more review comments
* adopt after rebase to main
I don't see a reason why it's part of the participant state API, and
it definitely doesn't need to change between v1 and v2.
CHANGELOG_BEGIN
- [Integration Kit] The class ``SeedService`` has been moved from the
*participant-state* Maven package to the *participant-integration-api*
Maven package, under the Java package name
``com.daml.platform.apiserver`` to reflect its usage by the API
server, not the participant state API. If you use this class directly,
you will need to change your imports.
CHANGELOG_END
* ledger-on-sql: Abstract out the log entry ID allocator.
This is so we don't need to depend on `SeedService`.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Remove `allocateRandomLogEntryId` from production code.
* kvutils: Remove `SubmissionValidator.create`, as it's only for tests.
And remove default parameters for
`SubmissionValidator.createForTimeMode`.
* ledger-on-sql: Use the random log entry ID allocator.
In Sandbox, we still use the seed service, but this is not necessary
for ledger-on-sql tests.
* kvutils: Make `LogEntryIdAllocator.random` an `object`.
* ledger-on-sql: Move `RandomLogEntryIdAllocator` here.
It's not used anywhere else.
* ledger-on-sql: Add some documentation about the application code.
Just to make it clear it's not used in production.
* Print ledger-api-bench-tool final report when a stream in ended by shutting down API server
CHANGELOG_BEGIN
CHANGELOG_END
* Log ledger-api-bench-tool failure with specific header to make it compatible with the lr-sidecar
* Print final report on graceful shutdown even if streams are not complete
* trigger build
* LF: Simplify archive reader.
- decouple Reader and Decoder
- introduce case class to handle hash, proto payload, and version
CHANGELOG_BEGIN
CHANGELOG_END
* Address Moritz' review
* cosmetic
The debug logs don’t seem all that useful and we don’t have them for
other sources, e.g., TransactionSource. They also come with error logs
for failed sources which again is inconsistent with other sources,
noisy and confusing.
changelog_begin
changelog_end
* [In-memory fan-out] Fix execution contexts in ReadOnlySqlLedgerWithMutableCache
* servicesExecutionContext is used in all non-resource components
* Remove widespread use of resourceContext
CHANGELOG_BEGIN
CHANGELOG_END
* [In-memory fan-out] TransactionLogUpdatesConversions optimization
* Lazily convert to transaction trees in TransactionLogUpdatesConversions.ToTransactionTree
* Addressed review comments
* daml-lf/data: Truncate party names in log output, on request.
The party name can grow quite long, so we offer ledger implementors the
opportunity to truncate it in structured log output.
Unfortunately, because we use Logback through the global
`LoggerFactory`, there is no place to inject logging configuration. This
means we also need to use global, mutable state to configure logging
output. I have added a `LoggingConfiguration` class+object in Daml-LF
Data, which may not be the best place, but I can't think of a better
one right now. I suggest we leave it there until it has reason to grow,
at which point we may want to move it.
CHANGELOG_BEGIN
CHANGELOG_END
* logging-entries: Make `ToLoggingValue` mixin-able.
* participant-integration-api: Truncate parties in filters when logging.
* participant-integration-api: Cast to `Party` for logging.
Invalid input should not break the request at this point. No assertions.
* daml-lf/data: Move `Party to LoggingValue` to a new package.
This avoids the transitive dependency issue most of the time.
* daml-lf-data: Move the `Identifier` logging to another package.
Again, reduces the need for transitively depending on _logging-entries_.
I don't find the output from Bazel very helpful most of the time. It
does, however, produce a lot of noise which I have to filter through
when something else goes wrong.
CHANGELOG_BEGIN
CHANGELOG_END
* logging-entries: Split from contextualized-logging.
This allows us to introduce it to Daml-LF without bringing in the
Logback, Logstash, and gRPC dependencies.
CHANGELOG_BEGIN
CHANGELOG_END
* logging-entries: Fix dependencies for 2.12.
* logging-entries: Missed one more Scala 2.12 dependency.
* release: Publish logging-entries.
* participant-integration-api: Remove the subscription ID.
Pretty sure it's not used except for a single log line, which makes it
useless as a correlation ID.
If we want to correlate logs, let's add a correlation ID.
* participant-integration-api: Move transaction requests to trace logging.
Most of the useful information is already in the logging context. We
don't need to log the data structure too.
CHANGELOG_BEGIN
- [Ledger API Server] The amount of data logged in the API transaction
service has been reduced at INFO level. Please enable TRACE logging to
log the request data structures.
CHANGELOG_END
* participant-integration-api: Reorder methods in ApiTransactionService.
* participant-integration-api: Add the word "request" to some log lines.
* participant-integration-api: Add a logging prefix for string offsets.
* ledger-api-domain: `immutable.Set` -> `Set`.
It's an alias.
* participant-integration-api: Log transaction filters on subscription.
* participant-integration-api: Log transaction filters.
Just the parties isn't enough information.
* participant-integration-api: Log the entire transaction request.
Structured, because otherwise it's hard to throw things away later.
* contextualized-logging: Avoid `View` because it's not in Scala 2.12.
* contextualized-logging: Add tests for booleans.
* contextualized-logging: Avoid methods that accept views.
Scala 2.12 really doesn't like me when I do that.
* participant-integration-api: One more try at building with Scala 2.12.
Currently it is impossible to test new features until they land in the
default LF version. This is clearly not great. This PR releases one
Ledger API test tool per LF version (with the LF version being in the
name) to make it easier for Canton and others to test new features.
Note that at least the way things are setup now we won’t go back in
time. We will only publish stable, preview & dev but not older
versions. If needed, we could expand that in the future.
changelog_begin
changelog_end
* A test case for crashing participant in a multi-node env with divulged contracts
* Fixed test naming
CHANGELOG_BEGIN
CHANGELOG_END
* Additional usability check for the disclosed contract
* A newline
* Verify that fetching an archived contract is not possible
* Second test case
* A `ledger-api-test-tool` test case for divulging a contract with key twice on multiple participants [DPP-433] (#9970)
* A test case for divulging a contract with a key twice on multiple participants
CHANGELOG_BEGIN
CHANGELOG_END
* Use autogenerated archive method
* Trigger build
* Increased timeout for TransactionServiceIT:TXInvisibleTransactionTreeByEventId
* contextualized-logging: Automatically convert logging values to strings.
For now, this has almost the same behavior, but it allows us to
customize the output in the future.
The main change is that the log format has gone from:
context: {a=b, x=1, foo=bar}
to:
context: {a: "b", x: "1", foo: "bar"}
* contextualized-logging: Move `writeTo` inside `LoggingValue`.
* contextualized-logging: Allow for more than just strings.
`null`, numbers, and sequences are now correctly logged.
The log format has gone from:
context: {a: "b", x: "1", foo: "bar", parties: "[alice, bob]"}
to:
context: {a: "b", x: 1, foo: "bar", parties: ["alice", "bob"]}
CHANGELOG_BEGIN
- The log output of Daml components has changed so that the structured
part is closer to JSON. This allows us to distinguish and parse
numbers and lists. If you are parsing this log output, you may need to
change your parser.
The log output has changed from:
.. code-block::
context: {a=b, x=1, foo=bar, parties=[alice, bob]}
to:
.. code-block::
context: {a: "b", x: 1, foo: "bar", parties: ["alice", "bob"]}
CHANGELOG_END
* contextualized-logging: Extract the string serializer.
* Use non-string logging where possible.
* contextualized-logging: Split logging values from serialization.
So that callers don't have to know about Jackson.
* contextualized-logging: `SeqView` is `Iterable`. Don't need both.
* contextualized-logging: Make `ToStringToLoggingValue` a `val`.
Co-Authored-By: Stephen Compall <stephen.compall@daml.com>
* contextualized-logging: Add a transient dependency for 2.12 only.
This required more infrastructure than I thought it would.
* kvutils: Make it explicit that we're logging the hashes of archives.
The implicit was found to be a little confusing.
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Oracle compliant append only schema
CHANGELOG_BEGIN
CHANGELOG_END
WIP : oracle on new appendonly schema
* diff to postgres dump, create consolidated view
* diff to postgres dump to ensure all oracle setup is equiv
* recompute sha for changed oracle flyway scripts
* drop old tables to prevent clash on name of new participant_events view
* recompute sha for flyway script
* prelim oracle StorageBackend
* Adds support for special preparedStatement for oracle
* Add support and wires to setObject by default everywhere
* Add the full OracleField suite with TODOs for convenience
* Wires OracleField suite to OracleFieldStrategy
changelog_begin
changelog_end
* enable debug version of oracle driver
* conversion Instant -> Timestamp for oracle
* WIP: primitive println debugging
* Passing PackagesSpec with appendonlyschema on Oracle
Rename size column to siz to avoid reserved word clash, including migration script for postgres
* include sha for new postgres migration script
* add missing copyright header
* cleanup
* passing party spec for appendonly on oracle
* passing configuration spec for appendonly on oracle
* scalafmt
* bazel buildifier reformat
* use db generic FETCH NEXT n ROW ONLY rather than limit for cross db compat
* siz instead of size for packages table on all dbs and schema
* revert enabling oracle jdbc debug
* Support Array[String] -> String conversion (and vice versa) for JSON array
Remove as aliases for tables as this does not work with oracle
Extract submitters clause for all db types
Use append transaction injector for oracle append only spec
* scalafmt
* correct oracle failing active contract spec tests
* wire in JdbcLedgerDaoCompletionsSpec
* remove semi-colons for ending statements that are problematic for oracle driver
* all tests up to divulgence passing for append only on oracle
* all appendonly tests passing on oracle
* remove ignore on fall back to limit-based query with consistent results
* do not change name of size column in packages table for mutable schema for all DBs
* do not change name of size column in packages table for mutable schema for all DBs
* standalone oracle appendonly schema script
regen shas on flyway scripts
revert some cosmetic refactoring in CommonStorageBackend
* Fixes conversion to parties from Oracle-JSON at flatEventWitnessesColumn
* Switches from composit SQLs to single SQLs at prepared statements to accommodate Oracle limitation
* Fixes arrayIntersectionWhereClause by applying patch from mutable Oracle schema integration
* Fixes queries with empty startExclusive Offsets
* First draw adding Oracle conformance test suites to CI
* wire in the oracle conformance tests for CI
* Use cross-db fetch next $n rows only syntax instead of limit syntax that works only for postgres/h2
* rename siz to package_size
* recompute shas
* scalafmt and include sha check for oracle append only flyway script
* correct missing package_size rename
* remove some todos -- correct corrupted V1__Init.sql
* Update ledger/ledger-on-sql/src/test/lib/scala/com/daml/ledger/on/sql/MainWithEphemeralOracleUser.scala
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* correct version number for postgres rename column scripts
* remove unnecessary migration tables for oracle append only
* review feedback: rename createEventFilter as requested, remove todos
* review feedback: case consistency
* review feedback: update todos with issue markers
* review feedback: cleanup
* review feedback: OracleField and OracleSchema cleanup
* Fixing Table generators to use preparedData for convenience
* Placing TODOs for refactorings later
* Renames initial append-only oracle script, for convenience
* Falls back to original behavior as far prepared statements go at couple of queries
Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* Resolve contract keys conflicts in disclosed contracts (#9948)
* build queries for nullifications of past key
* format code
CHANGELOG_BEGIN
In case a contract key is already present in a past contract in the contract table nullify it. This gets rid of contract keys of previously disclosed contracts. We never discover that disclosed contracts get archived, so we get conflicts on such past keys
CHANGELOG_END
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* correct oracle implemantation (new)
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Add missing indices for index initialization
changelog_begin
changelog_end
* Do not create unnecessary indices
* Create indices in H2
* Fix Oracle migration hash
* Verify all migration hashes
* Add exception tests
changelog_begin
changelog_end
* Add another test
This one tests a transaction that can only be
produced by a privacy-aware ledger.
* Address review comments
* CHANGELOG_BEGIN
Log ledger api validation failures at info level
CHANGELOG_END
* log validation failures in Api*Service family of classes
* address review comments
changelog_begin
- [JSON-API] Timing metrics which measure how long the processing of a command submission request takes on the ledger are now available
changelog_end
changelog_begin
- [JSON-API] The database operations (regardless of in-memory or postgres) contain now metrics for fetching contracts by id or key (seperate metrics foreach)
- [JSON-API] The time for a repsonse payload construction of a request is now also tracked in a metric
changelog_end
* [JSON-API] Add more timing metrics
changelog_begin
- [JSON-API] Timing metrics are now available for party management, package management, command submission and query endpoints.
- [JSON-API] Also added a timing metric for parsing and decoding of incoming json payloads
changelog_end
* Add comments to new metrics
* Split metrics up more & remove obsolete metric
* Split up timers for query endpoints
* Creating OR predicate to reduce length of string literals
CHANGELOG_BEGIN
CHANGELOG_END
* removing unnecessary thrown exception
CHANGELOG_BEGIN
CHANGELOG_END
* switching to camel case for constants
CHANGELOG_BEGIN
CHANGELOG_END
* run format
CHANGELOG_BEGIN
CHANGELOG_END
* remove tolist conversion
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt
Co-authored-by: Brian Healey <brian.healey@digitalasset.com>
* Preparation: pull up initialisation
* Preparation: introduce partyArrayContext for more flexibility
* Preparation: add append-only-from-scratch feature to FlywayMigrations
* Add squashed Flyway migration for append-only H2
* Add support for H2 StorageBackend
* Extend conformance test suites for H2
* Extend JDBCLedgerDao unit test suites for H2
* Extend sandbox-classic unit test suites for H2
* Fixing typo, some renames based on review
changelog_begin
changelog_end
* Do not terminate transactions/transaction trees streams when no end-offset provided in the ledger-api-bench-tool
CHANGELOG_BEGIN
CHANGELOG_END
* Minor change
* [JSON-API] Concurrent query etc. metrics
changelog_begin
- [JSON-API] The metrics which describe the amount of these concurrent events is now available:
- running http request
- command submissions
- package allocations
- party allocations
changelog_end
* Rename running metrics to throughput ones & add comments on the metrics
* Adjust names of other existing metrics too, to have for the json api a more consistent metrics naming
* truncate party name for 64 chars for multi-valued submitters to avoid excessively long metrics name
CHANGELOG_BEGIN
[metrics] Limit size of multi party metrics to ease consumption
CHANGELOG_END
* Use just first party for dynamic metrics when there are multiple parties involved
* Log context of all updates written to the database
CHANGELOG_BEGIN
For every update in the index db log the full context at the INFO level.
CHANGELOG_END
* one more missing log
This is a preparation for H2 integration: reworking/opening up the ingestion DSL
* Pull more execution logic to PGTable, as preparing for batch insert execution
* Pull out Table from PGTable, add support for deletion (preparation for JDBC batch execution)
* Abstract over Schema (preparation for JDBC batch execution)
* Opening up Field, abstract over table creation (preparation for JDBC batch execution)
* Move code to common (preparation for JDBC batch execution)
* Add generic JDBC batch execution
* Add generic JDBC batch deletion and some more generic Field-s
* Pulling out PGSmallintOptional to support the postgres flavor
* Generalizing idempotentInsert in FieldStrategy
* Capture non empty data logic and comment in a util function
* Seal access to Schema internals
changelog_begin
changelog_end
* Drop pkgIds argument from Engine.validatePackages
Doesn’t do anything useful, we just use it to validate that it matches
the keys of pkgs.
changelog_begin
changelog_end
* add test showing broken behavior
* Fix buggy tests
changelog_begin
changelog_end
Co-authored-by: Remy Haemmerle <Remy.Haemmerle@daml.com>
* oracle varray to json
CHANGELOG_BEGIN
CHANGELOG_END
* various cleanup, mostly removing printlns
* remove oracle connection from conversions and dependencies, cleaning up vestigial scala side varray code
* fix lazylist/stream issue by using java stream. remove braces
This PR is part of the effort to improve engine Errors. Here we
classify all the errors the engine can report in 4 categories:
- Errors happening during package loading
- Error happening during command/transaction preprocessing
- Error happening during interpretation
- Error happening during transaction validation
part of #9974
CHANGELOG_BEGIN
CHANGELOG_END
* Adds Indexer state to GRPC health checks [DPP-434]
CHANGELOG_BEGIN
[Integration Kit] The state of the participant indexer can now be checked via the GRPC health endpoint
CHANGELOG_END
* Addressed review comments
This PR adds the 3 additional testcases from #9941
Should have added that right when I touched this and fixed the bug but
better late than never.
changelog_begin
changelog_end
* Allow two different time formats as input for the metrics reporting interval and accordingly revert to using the old test for the CommonCliSpecBase in sandbox-common
changelog_begin
- for applications which support the --metrics-reporter-interval cli option, these now support both the java and scala duration string format (e.g. PT1M30S and 1.5min)
changelog_end
* Replace the thrown RuntimeException in the DurationFormat reader with an IllegalArgumentException
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Try parsing java duration first
* Add test which covers the scala duration format too in the CommonCliSpecBase of sandbox-common
* Add comment about java duration format
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Introduce metrics in the http-json service
changelog_begin
[HTTP/JSON API]
- metrics reporting can now be chosen via the command line option --metrics-reporter (currently hidden), valid options are console, csv://, graphite:// and prometheus://
- metrics reporting interval can also now be chosen via a command line option --metrics-reporting-interval (currently hidden)
changelog_end
* Move MetricsReporter and it's dependencies into //ledger/metrics
* Restore non-ugly formatting for that one section in Endpoints.scala
* Update ledger/sandbox-common/src/test/lib/scala/platform/sandbox/cli/CommonCliSpecBase.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Hide metrics option for http-json
* Propagate context exception for the parseUri function in MetricsReporter.scala
* Move cliHint value above parseUri function to have a better structure (it's used once before it's defined and once after it's defined, which is weird to me)
* Use better value name than optMr & optFd in cliopts Metrics.scala
* Remove import order changes & whitespace changes
* Revert usage of Nanoseconds for conversion from scala duration to java duration to usage of Seconds
* Shorten hideIfRequested function
* Fix another rearranged import
* Fix another whitespace removal
* Readd metrics cli option to sandbox after refactoring
* Add missing return type annotation for invalidRead in MetricsReporter
* Readd newline in https OptionParser.scala
* Remove unecessary import
* Update ledger/sandbox-common/src/main/scala/platform/sandbox/cli/CommonCliBase.scala
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Align setter & config name for metricsReportingInterval setting too in CommonCliBase.scala
* Rename http_json_api in Metrics.scala of metrics project to HttpJsonApi
* Reformat CommonCliBase.scala of sandbox-common project
* Fix CommonCliSpecBase test of sandbox
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Made three tests more flexible in terms of expected error message:
1. CommandServiceIT:CSReturnStackTrace extended parsing to support details= and ;-based format of error properties - found by @daravep
2. CommandServiceIT:CSCreateAndBadExerciseChoice reenable previously expected prior to 7bc925e4d2 as a viable error output
3. CommandSubmissionCompletionIT:CSCRefuseBadChoice - same as 2.
changelog_begin
changelog_end
* [In-memory fan-out] BuffersUpdater implementation
* Instantiated the TransactionsBuffer in ReadOnlySqlLedgerWithMutableCache
* Implementations and tests for the BuffersUpdater which updates:
* * The MutableCacheBackedContractStore
* * TransactionsBuffer
* Introduced `buffered-ledger-api-streams-unsafe` CLI flag for enabling in-memory fan-out flow
* Enabled the `buffered-ledger-api-streams-unsafe` in ledger-on-sql conformance test for append-only schema
CHANGELOG_BEGIN
CHANGELOG_END
* [In-memory fan-out] BuffersUpdater implementation
* Instantiated the TransactionsBuffer in ReadOnlySqlLedgerWithMutableCache
* Implementations and tests for the BuffersUpdater which updates:
* * The MutableCacheBackedContractStore
* * TransactionsBuffer
* Introduced `buffered-ledger-api-streams-unsafe` CLI flag for enabling in-memory fan-out flow
* Enabled the `buffered-ledger-api-streams-unsafe` in ledger-on-sql conformance test for append-only schema
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-test-tool: Improve logging when reporting waiting failed.
We use a stringified version of the participant when reporting on a
failed `waitForParties` call.
* ledger-api-test-tool: Only wait for parties on participants under test.
Most tests only use one participant, so we can speed up waiting
considerably if we don't wait on the other participants in those cases.
CHANGELOG_BEGIN
CHANGELOG_END
* Switch from shortcut to support nullable types to explicit Optional PGField-s
* Add scaladoc to type parameters of PGField
* Typo fix
changelog_begin
changelog_end
TODO Cleanup: postgre-batching
* First draw of PGSchema and PostgresDbBatch
* Add runtime check of input to prevent string-array serialisation issues
* Remove TODO pointing out missing indexes: since those deletes are operating on tables of small size, and since execution of those queries only at initialisation: this performance improvement is not necessary
changelog_begin
changelog_end
* CHANGELOG_BEGIN
Add metrics counting the elements in transaction, transaction-tree, completion and acs streams
CHANGELOG_END
* format scala
* address review comments
* Uniform configuration of the command service across all ledgers
* CHANGELOG_BEGIN
Modify command line options of the ledgers based on the kvutils/app library:
- Remove max-commands-in-flight sub-option from the participant group.
- Add command service configuration options in line with sandbox and daml-on-sql: max-commands-in-flight, max-parallel-submissions, input-buffer-size
CHANGELOG_END
* Test cases added for lookupKeyState and lookupContractState
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/participant-integration-api/src/test/lib/scala/platform/store/dao/JdbcLedgerDaoContractsAppendOnlySpec.scala
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* Move ledger-api-bench-tool classes to individual files
CHANGELOG_BEGIN
CHANGELOG_END
* min-consumption-speed SLO for the ledger-api-bench-tool
CHANGELOG_BEGIN
- [Integration Kit] - ledger-api-bench-tool - new consumption speed service-level objective for the ledger-api-bench-tool, configurable with a CLI parameter --min-consumption-speed
CHANGELOG_END
* Redefined consumption speed metric calculation
* Moved objectives to a separate package
Before this change, any write-service was restricted to the predefined
set of rejection reasons which couldn't capture the complex canton
rejection reasons.
In order to open up the path for a redesign, I've migrated the current
rejection reasons into a V0 version and opened the abstract class for
customized implementations. I've also added a temporary method to tunnel
synchronous rejections through the ledger api server.
CHANGELOG_BEGIN
- Participant-state: Introduced versioned form of rejection reason
CHANGELOG_END
* Upgrade protobuf-java and scalapb.
CHANGELOG_BEGIN
CHANGELOG_END
* Bazel: Generate ScalaPB classes without `unknownFields`.
This is very annoying because it means that extractors need an extra
`_` for no good reason. It also breaks Circe's automatic derivation.
Including unknown fields is the default behavior in Protocol Buffers for
Java now, and we'll probably have to reckon with it at some point, but
let's kick that can down the road a little.
* Upgrade protobuf-java to 3.17.1.
This is identical to 3.17.0, but let's track it anyway.
* Bazel: Move ScalaPB versions into their own file.
They don't need to go into the generated one.
* Fix missing microseconds
* More lenient error string matching
* Fix typo
* Fix ledger time tests
Previously, the computed blinding info
did not contain any divulgence, which lead to
no divulgence events being stored.
* Implement JdbcLedgerDao tests for the append-only schema
changelog_begin
changelog_end
* Apply review comment
* Simplified metrics names
* Metrics returning domain values instead of formatted strings
* Always returning metric Value objects with possibly optional contents
* Use the new metric Value classes
* Remove the old metric reporting mechanism
* CHANGELOG_BEGIN
CHANGELOG_END
* Fixed 2.12 build
* Removed random generators from metric tests
* Fixed 2.12 build
* Improved readability of the size metric calculation
* Minor change
* `ledger-api-bench-tool`: max consumption delay SLO [DPP-400] (#9785)
* ServiceLevelObjective trait
* Added copyright to a new file
* A model for combining metrics and objectives in type families
* Mechanism for returning metric violation information
* Return error code when SLOs violated
* Use type parameter for the result of transaction service methods
* max-delay command line parameter for the max delay SLO
* Logging violated objectives in the final report
* Simplified size rate metric value
* CHANGELOG_BEGIN
- [Integration Kit] - ledger-api-bench-tool - new parameter max-delay for specifying the service-level objective for max delay
CHANGELOG_END
* Fixed tests
* Simplified MetricsManager.Message trait
* Fixed build errors
* Changed objectives violated message
* Removed DelayObjective trait
* Comments improvement
* Ordering for MaxDelay.Value
* Removed redundant method from the ServiceLevelObjective trait
* Add ledger API test tool tests for rollback projections
This adds 3 tests for projections under rollback nodes.
The first one is relatively clear hopefully and tests divulgence.
The other two are a bit more intricate. For both of those we can also
not test too much via the ledger API since we don’t actually get
access to rollback nodes. However, it still seems useful to at least
exercise those code paths and make sure they don’t do anything
horribbly wrong.
The second test tests the normalization rules from
https://github.com/digital-asset/daml/blob/main/docs/source/concepts/ledger-model/ledger-exceptions.rst#privacy
The last one tests a more complex structure with deeply nested
rollback nodes and different informees.
changelog_begin
changelog_end
* Update ledger/test-common/src/main/daml/semantic/Exceptions.daml
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
* Make use of nameOf for naming telemetry traces in participant-integration-api
changelog_begin
changelog_end
* Add copyright header to Telemetry.scala
* Add a simple nameof macro lib which has one function to return the fqn of the current method
* Make use of the new nameof lib within the repo to shorten the code further
* Remove old dependency on nameof macro
* Readd lines which have been accidentally removed
