* libs-scala/ports: Wrap socket ports in a type, `Port`.
* sandbox: Use `Port` for the API server port, and propagate.
CHANGELOG_BEGIN
CHANGELOG_END
* extractor: Use `Port` for the server port.
* ports: Make Port a compile-time class only.
* ports: Allow port 0; it can be specified by a user.
* ports: Publish to Maven Central.
* Make kvutils work with the new contract id scheme
CHANGELOG_BEGIN
- [KVUtils] uses random contract id. Contract ids are made of 65 hexa decimal characters.
CHANGELOG_END
Co-authored-by: Jussi Mäki <jussi.maki@digitalasset.com>
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.
InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.
* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.
* Run all conformance tests in the repository in verbose mode.
This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.
This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:
- ledger-api-test-tool-on-canton
- ledger-on-memory
- ledger-on-sql
- sandbox
Fixes#4225.
CHANGELOG_BEGIN
CHANGELOG_END
* Add TTL field to protobuf
* Add command deduplication to index service
* Wire command deduplication to DAO
* Implement in-memory command deduplication
* Remove Deduplicator
* Implement JDBC command deduplication
* Add TTL field to domain commands
* Deduplicate commands in the submission service
CHANGELOG_BEGIN
- [Sandbox] Implement a new command submission deduplication mechanism
based on a time-to-live (TTL) for commands.
See https://github.com/digital-asset/daml/issues/4193
CHANGELOG_END
* Remove unused command service parameter
* fixup protobuf
* Add configuration for TTL
* Fix Haskell bindings
* Rename SQL table
* Add command deduplication test
* Redesign command deduplication queries
* Address review comment
* Address review comment
* Address review comments
* Make command deduplication test optional
* Disable more tests
* Address review comments
* Address review comments
* Refine test
* Address review comments
* scalafmt
* Truncate new table on reset
* Store original command result
* Rename table columns
... to be consistent with other upcoming tables
* Rename migrations to solve conflicts
Fixes#4193.
* Add overridable indexer, api and auth configuration to `LedgerFactory`
CHANGELOG_BEGIN
CHANGELOG_END
* Add overridable indexer and api metrics creation to `LedgerFactory`
CHANGELOG_BEGIN
CHANGELOG_END
* Add overridable api's `TimeServiceBackend` to `LedgerFactory`
* 🎨 Fix formatting
* Port SDK ledgers based on `Runner` (and the sandbox) to `TimeServiceBackend`
* Revert to `TimeProvider` for committer usage and to `None` default for API server.
Also removed now unused `TimeServiceProvider.wallClock()`.
* Move TimeServiceBackend back to the API server.
* 🎨 Remove unneeded argument passed for parameter w/default
* Restore sandbox ledger time support
* Simplify passing a `TimeProvider` to the sandbox ledger
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* sandbox: If the ledger ID isn't provided, use the one in the database.
Previously, we would fail if working against an existing ledger, and not
explicitly providing the ledger ID. This was the case even if the ledger
ID was randomly generated initially.
CHANGELOG_BEGIN
- [Sandbox] If no ledger ID is provided when running against an existing
ledger, use the existing ID. Previously, Sandbox would fail to start.
CHANGELOG_END
* sandbox: The ReadOnlySqlLedger should always receive a ledger ID.
It's read-only; it can't create one.
* sandbox: Stop using `equal` in SqlLedgerSpec.
* sandbox: Test that the ledger ID is as specified in SqlLedgerSpec.
* sandbox: Let the top-level runner handle a ledger ID mismatch.
And clean up the log text.
* sandbox: Initialize the ledger properly when the ID is dynamic.
* sandbox: Use `Vector`, not `List`, for SqlLedger initialization.
Append with Vector, good. List, bad.
* ledger-api-common: Make `LedgerApiMode.Dynamic` an object.
And add Java-style static factory methods.
* kvutils/app | ledger-on-{memory,sql}: Make `ledgerId` optional.
It should be generated or retrieved from the persistence layer by the
ledger itself.
* kvutils: Make the ledger ID optional in the tests.
* ledger-on-sql: Store the ledger ID, and reject conflicting IDs.
* ledger-on-sql: Make more things final.
* ledger-on-sql: Document the `ledger_meta.table_key` column better.
* sandbox: Don't hardcode the number of packages in the test DAR.
It changes.
* ledger-on-sql: Merge the `head` resource owner with the `dispatcher`.
* sandbox: Use backticks to simplify pattern match in ReadOnlySqlLedger.
* ledger-on-sql: Extract methods in `owner`.
* Add more tests to default run of Ledger API Test Tool
Furthermore, drop TimeIT, which is a sandbox-only property (and tested already as part of its integration tests)
CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Ledger API Test Tool default tests modified. Use --list for the updated list of default tests. Time service test dropped from the suite.
CHANGELOG_END
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380635979
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380636989
* Optimize imports
* Only run semantic tests on Canton
* kvutils: Make it easier to import the Key and Value types.
Moved them into a companion object.
* kvutils: Flatten the nested `for` comprehensions in SubmissionValidator.
* kvutils: Test that results succeed before querying state updates.
If they don't, we get idle timeouts and we don't see the error, which is
unhelpful.
* sandbox: Increase log levels for failed submissions.
* kvutils: Avoid converting DamlStateKey -> ByteString -> Array[Byte].
Just convert once.
* ledger-on-sql: Use the `SubmissionValidator` from kvutils.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Remove unnecessary interpolation in ApiSubmissionService.
* resources: Allow for startup exceptions without no stack trace.
* sandbox: Remove dependency duplication in the BUILD file.
* ledger-on-(memory|sql): Extract `Main` into its own file tree.
This is so we can remove the dependency on kvutils/app from the library.
logback.xml goes with it.
* sandbox: Implement "Sandbox Next" on top of ledger-on-sql.
CHANGELOG_BEGIN
- [Sandbox] Preliminary work to rebuild Sandbox on top of the DAML
Ledger Integration Kit. Currently not exposed through the CLI.
CHANGELOG_END
* ledger-on-sql: Publish to Maven Central.
* sandbox: If an unknown database is passed to Sandbox Next, handle it.
* resources: Catch errors in owner construction or acquisition.
Sometimes we're not in a future yet.
* resources: Don't run ProgramResource error handling inside the EC.
Otherwise we end up in deadlock which is only broken by
`awaitTermination` timing out.
* resources: Make it easy to use a resource and then release it.
* ledger-on-sql: Move database transaction methods to `Database`.
* ledger-on-sql: Make sure the ledger picks up where it left off.
This means seeding the dispatcher with the correct head, rather than
starting at 1 every time.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Move `Index` and `StartIndex` to the package object.
They're used everywhere.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Don't allow unnamed in-memory databases.
It seems that changing the connection pool size at runtime causes weird
issues, predominantly that sometimes all connections are dropped and
therefore the in-memory database is lost after migration. Obviously, it
then stops working.
This works around this problem by simply not allowing unnamed in-memory
databases, guaranteeing that we can share the database between the
various connection pools.
This will probably be necessary anyway when attempting to share the
database between the ledger and the index.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Make it clear we're generating a new JDBC URL in tests.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Pull out a superclass for the various integration tests.
* ledger-on-sql: Use Flyway to migrate tables in an idempotent manner.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Make it easy to run it with Bazel for experimentation.
* ledger-on-sql: Test that migrations will never change in the future.
* ledger-on-sql: Add a prefix of "ledger_" to the tables.
This is so we don't accidentally conflict with the index when the
schemas are shared.
I am letting myself modify the migrations because the existing
migrations haven't been merged into `master` yet.
* ledger-on-sql: Explain why we change the pool size after migration.
* ledger-api-test-tool: Tests now specify a timeout scale, not a timeout.
Makes it easier to change the default timeout in one place.
* ledger-api-test-tool: Increase timeouts on slow tests.
These tests produce a lot of volume and can make CI flaky.
* ledger-on-sql: Only tear down PostgreSQL in tests.
For other databases, we just create a new file for each test case.
* ledger-on-sql: Reduce the log output in tests.
* ledger-on-sql: Use a separate connection pool for Flyway when possible.
Apparently `setMaximumPoolSize` doesn't really have the desired effect
after the connection pool has already been used. The new test case will
be flaky if we process more than one commit in parallel.
For SQLite, it seems to be OK.
* ledger-on-sql: Use a separate connection pool for Flyway with SQLite.
Except in memory.
* ledger-on-sql: Use a separate PostgreSQL database for each test.
Because performance, innit. Don't have to tear them down.
* ledger-on-sql: Move the `Queries` values inside the `RDBMS` objects.
* ledger-on-sql: Go into even more detail about pool size hijinks.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Don't bother cleaning up after integration tests.
Turns out Bazel cleans up before each test run, so we only have one
set of outputs at a time. This is far more useful for debugging anyway.
* ledger-on-sql: Pull out a test-lib to simplify the conformance tests.
* ledger-on-sql: Turn `Database` from a sealed trait into a case class.
* ledger-on-sql: Support for PostgreSQL!
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run the conformance tests against PostgreSQL.
* ledger-on-sql: Run the LotsOfParties conformance tests.
* ledger-on-sql: Use PostgreSQL's `RETURNING` keyword to save a query.
* ledger-on-sql: Ensure the reader connection pool is read-only.
* ledger-on-sql: Avoid cloning ByteStrings as byte arrays where possible.
Instead, use `ByteString#newInput()`, which returns an `InputStream`.
This is supported by H2 and PostgreSQL, but unfortunately not SQLite.
* ledger-on-sql: Run integration tests in parallel.
* Update the PostgreSQL and SQLite JDBC drivers.
* kvutils: Use resources in KeyValueParticipantState, not AutoCloseable.
CHANGELOG_BEGIN
- [kvutils] The simplified API now uses ``com.digitalasset.resources``
to manage acquiring and releasing resources instead of ``Closeable``.
CHANGELOG_END
* ledger-on-posix-filesystem: Remove dead code due to a refactoring fail.
* kvutils: Make logback.xml a base file, rather than the only option.
* kvutils/app: Simplify logback.base.xml.
Mostly by getting rid of unnecessary appenders.
* ledger-on-sql: Add trace logging for all database work.
* Upgrade H2; there's a few useful bug fixes.
* kvutils/app: Let the user override the server JDBC URL.
* kvutils/app: Provide a way to specify the ledger ID.
* ledger-on-sql: If there are missing entries in the log, fail on read.
This can happen right now because we insert in parallel. Next step: stop
doing that.
* ledger-on-sql: Stop writing in parallel; it causes race conditions.
This unfortunately means we also stop _reading_ in parallel, which is
less fun.
* Revert "ledger-on-sql: Use a sequential log entry ID."
This reverts commit c58265bf43.
* ledger-on-sql: Create tables with columns that are not nullable.
* ledger-on-sql: The H2 conformance tests now work, just slowly.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run tests against H2 and SQLite on memory and disk.
* ledger-on-sql: Allow H2 to read from the log in parallel with writes.
* ledger-on-sql: Use UUIDs for entry IDs, not 8 random bytes.
* ledger-on-sql: Make sure to log the correlation ID.
* ledger-on-sql: Do less while holding a database connection.
* ledger-on-sql: Log the connection.
* ledger-on-sql: We don't need to tell H2 not to drop the connection.
The connection pool takes care of that.
* ledger-on-sql: Disable H2 conformance tests on CI; they're too slow.
* ledger-on-sql: Rename `loggingContext` to `logCtx`.
* ledger-on-sql: Don't abuse log contexts; put the data in the message.
* ledger-on-sql: Make the connection log line easier to read.
And pull out the logger, even if it's only used once.
* ledger-on-sql: Run the unit tests against (H2, SQLite) * (memory, file).
* ledger-on-sql: Implement a ledger using the simplified kvutils on JDBC.
This tests against H2; PostgreSQL coming soon.
No changelog because we're not shipping this yet.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/app: Extract ledger construction into a trait.
* ledger-on-sql: Run the conformance tests against H2.
* ledger-on-sql: Use H2-specific queries to perform actions atomically.
* ledger-on-sql: Extract queries to `Queries` and `H2Queries`.
* ledger-on-sql: SQLite can't have more than one connection at a time.
* ledger-on-sql: Run conformance tests against SQLite.
* ledger-on-sql: Fix error message if the JDBC URL is missing.
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* kvutils/app: Rename LedgerConstructor to LedgerFactory.
* ledger-on-sql: Don't release connections early.
* ledger-on-sql: Rename `withConnection` so it's more meaningful.
* ledger-on-sql: Use a sequential log entry ID.
* ledger-on-sql: Disable H2 conformance tests.
Until we can figure out why some fail.
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
