This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.
This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.
Apologies for the giant PR, I don’t see a way to keep it smaller.
changelog_begin
changelog_end
* resources: Move builders into //ledger/ledger-resources.
Keep the actual constructors in a trait, but instantiate it when working
with ledger code.
This allows us to later introduce an extra "context" type parameter to
ResourceOwner.
* resources-akka: Move the builders in to //ledger/ledger-resources.
* resources: Introduce an abstract `Context` parameter for owners.
This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.
This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.
Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.
CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
`Context`, which is filled in by the corresponding `Context` class in
the _ledger-resources_ dependency. This allows us to pass extra
information through resource acquisition.
CHANGELOG_END
* ledger-resources: Move `ResourceOwner` here from `resources`.
* ledger-resources: Remove dependencies from outside //ledger.
* ledger-resource: Wrap the acquisition execution context in `Context`.
So we can add a logging context to it.
* resources: Pass the Context, not the ExecutionContext, to Resource.
* Avoid importing `HasExecutionContext`.
* ledger-resources: Publish to Maven Central.
* resources: Make the small changes suggested by @stefanobaghino-da.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* ledger-resources: Pull out a trait for test resource contexts.
Saves a few lines of code.
* Restore some imports that were accidentally wildcarded.
* resources: Replace an `implicit def` with a couple of imports.
* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.
Try and use the right execution context where possible.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
This PR creates 3 validation modes:
* `Strict`: Specifies that the committer should validate the package
before committing them to the ledger. When using this mode, the
packages committed to the ledger can be fully trusted and do not
have to be validated when loaded into the engine.
* `Lenient`: Specifies that the committer should perform a fast
validation of the packages before committing them to the ledger.
This mode is useful for ledger integrations that cannot handle
long-running submissions (> 10s). When using this mode, the
packages committed to the ledger cannot be trusted and must be
validated every time they are loaded into the engine.
* `No`: Specifies that the committer should not perform any
validation the packages before committing them to the ledger. This
should be used only by non distributed ledgers, like DAML-on-SQL,
where the validation done in the API server can be trusted.
This PR creates 3 preloading modes:
* `Synchronous` : Specifies that the packages should be preloading
into the engine before committed.
* `Asynchronous`: Specifies that the packages should be preloaded into
the engine asynchronously with the rest of the commit process. This
mode is useful for ledger integrations that cannot handle
long-running submissions (> 10s). Failure of the preloading process
will not affect the commit.
* `No`: Specifies that the packages should not be preloaded into
the engine.
CHANGELOG_BEGIN
- [Integration Kit] In kvutils, add metric
daml.kvutils.committer.package_upload.validate_timer to track
package validation time.
CHANGELOG_END
* Move in-mem writer's `ledgerStateAccess.inTransaction` down to committer
* Move `BatchedSubmissionValidator` and spec into `batch` subpackage
* Add `StateAccessingValidatingCommitter` and inherit it in batching one
* Document `StateAccessingValidatingCommitter`
* Generalize the committer for `InMemoryLedgerReaderWriter`
* `envelope` -> `submissionEnvelope` in validating committers
* Add `PreExecutingValidatingCommitter` and sub-components
* Add retry in case of conflict in `PreExecutingValidatingCommitter`
CHANGELOG_BEGIN
CHANGELOG_END
* Fix compilation error
* Hook pre-execution in `daml-on-memory-kv`
* Add fake time updates provider
* Fix `BatchedValidatingCommitterSpec`
* Don't use batched writer with pre-execution
* Fix conflict detection
* Fix out-of-time-bounds detection
* Prefix/unprefix serialized log entry IDs in pre-execution write sets
* Fix: produce an out-of-bounds rejection log entry in transaction rejected cases too
* Fix `SubmissionResult` return in case of repeated pre-exec conflict
* Fidelity level 1: sequential pre-execution
* Documentation for pre-execution support in DAML-on-Memory KV
* Add ledger-on-memory conformance test with pre-execution enabled
* Revert "Fix: produce an out-of-bounds rejection log entry in transaction rejected cases too"
This reverts commit 4df7e26b
* Fix test
* Improve naming and documentation
* Address review comments
* Fix test
* Fix wrong implementation used for `ParticipantStateIntegrationSpecBase` tests
* Address review comments
* Address review comments
* Address minor review comments
* Move public code into daml-integration-api
CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
Also fixes#5635, removing usage of `ReadService` from the `StandaloneApiServer`.
The configuration stream of the Ledger API LedgerConfigurationService is now properly backed by the configuration entries instead of just serving the initial configuration.
CHANGELOG_BEGIN
[DAML Integration Kit]: ``StandaloneApiServer`` can now be run in a read-only mode.
- The type of the constructor parameter ``writeService`` of ``StandaloneApiServer`` changed to ``Option[WriteService]``. Passing ``None`` will not start any of the admin services, the command service, and the command submission service.
- The constructor parameter ``readService`` of ``StandaloneApiServer`` has been removed.
- A new constructor parameter ``ledgerId`` has been added to ``StandaloneApiServer``. It is used to verify that that ``StandaloneApiServer`` is run against an index storage for the same ledgerId. Initialization is aborted if this is not the case.
[DAML Integration Kit]: The ``LedgerConfigurationService`` now properly streams configuration changes.
CHANGELOG_END
* Extract caching from participant-state as a library
This will be used to keep a cache of values to cut on LF translation cost when serving transactions.
changelog_begin
changelog_end
* Add dependency where missing
- Only upload packages during the initial startup.
- Avoid loading packages during subsequent resets
- Share an engine between Ledger API Server and Committer
* Use a randomized H2 URL to simular in-memory
The reset service test assumes to get a completely new ledger for each
test case. But because we use H2 in-memory with db_close_delay=1 and the
same H2 database name, the second test case gets the remnants of the
first test case.
Since we know that sandbox in-memory uses an H2 in-memory URL, we can
simply use SandboxBackend.H2Database for ResetServiceInMemoryIT.
CHANGELOG_BEGIN
[Sandbox] Drastically lower the time needed to do a reset via the
ResetService.
CHANGELOG_END
* kvutils: Cache state value conversions from bytes.
This seems to have a decent speedup in ledger-on-memory.
CHANGELOG_BEGIN
- [Ledger Integration Kit] Submissions now look up ledger values from a
cache where possible, improving performance when there's contention over
certain resources (e.g. common packages). The cache size currently
defaults to 64 MB.
CHANGELOG_END
* kvutils: Make the SubmissionValidator statue value cache configurable.
* kvutils: Report state value cache metrics.
* kvutils: Add a suffix to a Long literal because WartRemover is unhappy.
Strangely, it doesn't fail on my machine.
* kvutils: Extract caching out into its own file.
* kvutils: Move the `bytesToStateValue` call into `cache.get`.
* kvutils: Move caching to its own package.
* kvutils: Inject the state value cache.
* kvutils: Default to no state value cache.
* kvutils: Accept a state value cache size in megabytes, not bytes.
* kvutils: Move cache building from `Config` to the `caching` package.
* kvutils: Replace Guava's cache with Caffeine.
* kvutils: Simplify caching configuration.
* sandbox: Enable state value caching by default.
CHANGELOG_BEGIN
- [Sandbox] State values deserialization is now cached, with a fixed
cache size of 128MB.
CHANGELOG_END
* Changelog commit.
CHANGELOG_BEGIN
- [Ledger Integration Kit] The state value cache is now opt-in, with a
default of no cache at all.
CHANGELOG_END
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* sandbox: Move the events page size configuration value into config.
* sandbox: Pass `config` directly into JdbcIndexerFactory.
* sandbox: Reorder `eventsPageSize` before `metrics` in parameters.
* sandbox: Move `seeding` into `ApiServerConfig`.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Name all parameters of `JdbcLedgerDao.writeOwner`.
Co-Authored-By: stefano.baghino@digitalasset.com
Contributes to #4231.
Remove checkpoints from Participant Server storage:
- Removed code to store checkpoints in the index database.
- Remove existing checkpoint rows in ledger_entries and participant_command_completions.
- Removed ObservedTimeServiceBackend
This commit modifies the java migration V2_1__Rebuild_Acs. This is safe to do, because:
a) any even semi recent persistent sandbox had already gone through this migration and won't re-run it
b) A new database doesn't even have entries yet to migrate.
CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Removed the ``Hearbeat`` state update.
[Sandbox] Checkpoints are no longer emitted in regular intervals in wall
clock time mode.
CHANGELOG_END
* kvutils: Remove an unnecessary `@SuppressWarnings`.
* kvutils: Reduce the scope of fields and methods in `Committer`.
* kvutils: Inject the metric registry into `KeyValueCommitting`.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Inject the metric registry into the committers.
* kvutils: Inject the metric registry into `ProcessTransactionSubmission`.
* kvutils: Avoid shared metric registries in tests.
* kvutils: Recreate the metrics registry per participant state.
* kvutils: Add trailing commas to parameter lists.
Flagrantly encouraged by @stefanobaghino-da.
* recovering-indexer: Don't re-use the metric registry in tests.
* sandbox: Clean up `MetricsReporting` a little.
Make sure it closes both reporters, and avoid starting things in a
constructor.
* sandbox: Add hidden options for enable metrics reporting.
* sandbox: Add a disambiguating name to the DB connection/thread pools.
CHANGELOG_BEGIN
- [Sandbox] DB connection pool metrics names have changed slightly, from
``daml.index.db.connection`` to ``daml.index.db.connection.sandbox``.
- [Ledger Integration Kit] DB connection pool metrics names have changed
to disambiguate the StandaloneApiServer from the
StandaloneIndexerServer. The former now has a ``.ledger-api-server``
suffix, and the latter now has a ``.indexer`` suffix.
CHANGELOG_END
* sandbox-next: Use the same metrics registry for the API and indexer.
* sandbox: Give a useful error message on an invalid metrics reporter.
And simplify the error messages.
With the arguments `--client-auth=foo --metrics-reporter=foo`, we now
get the output:
```
Error: Option --client-auth failed when given 'foo'. Must be one of
"none", "optional", or "require".
Error: Option --metrics-reporter failed when given 'foo'. Must be one of
"console", or "csv:PATH".
Try --help for more information.
```
* sandbox: Pull out more helpers in `MetricsReporting`.
* sandbox: Rename MetricsReporter classes so they don't clash.
* sandbox: Wrap the `name` parameter in a `ServerName` tagged string.
For safety. Yours, not mine.
* sandbox: Push metrics to Graphite with `--metrics-reporter=graphite`.
* sandbox: Make `MetricsReporter.Graphite` singly-lazy, not doubly-.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: Replace `ServerName` with `ServerRole`.
* sandbox: Fix usage of `ServerRole.Testing` in `LedgerResource`.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* kvutils/app|sandbox: Rename `system` to `actorSystem`.
* sandbox: Pass a materializer in to the StandaloneIndexerServer.
There's no need for it to construct one when the caller always has one
available.
CHANGELOG_BEGIN
CHANGELOG_END
* recovering-indexer: Use `SubmissionId` instead of `LedgerString`.
Where appropriate.
* sandbox: Use the materializer implicitly in JdbcIndexer.
* sandbox: Don't let just anyone construct a DbDispatcher.
It's the job of the JdbcLedgerDao.
* sandbox: Clean up the DbDispatcher a little.
* sandbox: JdbcLedgerDao now creates its own execution context.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Make `defaultNumberOfShortLivedConnections` private.
* kvutils: Remove the unnecessary execution context from the test base.
* kvutils: Remove the unnecessary execution context from the writer.
* ledger-on-sql: Make a proper owner so it has a proper execution context.
This means the parallelization now needs to come from the test, so I've
augmented ParticipantStateIntegrationSpecBase to take a proper execution
context instead of the serial one that ScalaTest provides, with a
default of `ExecutionContext.global`.
* ledger-on-memory: Make a proper owner with a proper execution context.
* kvutils/app: Remove `executionContext` from LedgerFactory.
Shouldn't need it in `ResourceOwner`. I was bad.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-memory: Make ResourceOwners real classes.
* ledger-on-sql: Make the ResourceOwner a real class.
* ledger-on-sql: Cause side effects on resource acquisition.
Not on owner construction.
This renames methods backing the `ListKnownParties` request to
from `parties`, `getParties` or `listParties` to `listKnownParties`.
CHANGELOG_BEGIN
- [Ledger API Server] Renamed two metrics:
``daml.index.parties`` was renamed to ``daml.index.list_known_parties``
``daml.index.db.get_parties`` was renamed to ``daml.index.db.list_known_parties``
CHANGELOG_END
* sandbox: Re-use the root actor system in the StandaloneIndexerServer.
* kvutils/app: Don't use the ActorSystem execution context randomly.
Instead, make `Runner` a proper ResourceOwner, with an `acquire` method.
* sandbox: Re-use the root actor system in the StandaloneApiServer.
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Remove the now-unused `ResourceOwner.sequence` functions.
They weren't well-thought-out anyway; they acquire resources
sequentially, rather than in parallel.
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.
InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.
* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.
* Run all conformance tests in the repository in verbose mode.
This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.
This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:
- ledger-api-test-tool-on-canton
- ledger-on-memory
- ledger-on-sql
- sandbox
Fixes#4225.
CHANGELOG_BEGIN
CHANGELOG_END
This moves IndexerIT into its own package, and swaps the dependency from
reference-v2 to ledger-on-memory.
This test should ideally live in the sandbox code, but because it
depends on ledger-on-memory, it's easier to keep it separate.
Also rewrites a lot of the code because the API is different. The tests
should now be clearer too.
I've also marked the test as flaky, because, well, it is.
CHANGELOG_BEGIN
CHANGELOG_END
