* kvutils: Avoid casting `ArgumentCaptor` and friends in tests.
Instead, use generics the way they're intended.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: In KeyValueParticipantStateWriterSpec, drop the Option.
* ledger-api-test-tool: Fix warnings flagged by IntelliJ IDEA.
* ledger-api-test-tool: Open-world mode.
In open-world mode, parties aren't allocated; their names are just
reserved for the test case, so that no other test will accidentally use
the same party name.
This is so we can test ledgers which dynamically allocate parties, such
as Sandbox.
* sandbox: Run conformance tests in "open-world" mode.
This means that the tests don't explicitly allocate parties (except for
a few), instead relying on Sandbox's implicit party allocation feature.
This is not enabled for Sandbox Next yet.
* sandbox-next: Implicit party allocation.
This is added to the command submission service.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-next: Don't implicitly allocate pre-existing parties.
* ledger-api-test-tool: Move pre-allocation into ParticipantTestContext.
* ledger-api-test-tool: We can reserve parties or wait for them. Not both.
Make illegal states unrepresentable as early as possible.
* sandbox: Name ApiSubmissionService's private methods a little better.
* sandbox: Move ApiSubmissionService's conditional logic into methods.
* sandbox: Document why we set `implicitPartyAllocation` to `false`.
* sandbox: Document why `implicitPartyAllocation` is dangerous.
* kvutils: On error opening an envelope, throw the correct message.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: On error when querying state, throw the correct error.
* kvutils|ledger-on-sql: Remove unnecessary curly braces around `throw`.
* sandbox: Re-use the root actor system in the StandaloneIndexerServer.
* kvutils/app: Don't use the ActorSystem execution context randomly.
Instead, make `Runner` a proper ResourceOwner, with an `acquire` method.
* sandbox: Re-use the root actor system in the StandaloneApiServer.
CHANGELOG_BEGIN
CHANGELOG_END
* resources: Remove the now-unused `ResourceOwner.sequence` functions.
They weren't well-thought-out anyway; they acquire resources
sequentially, rather than in parallel.
* Allow `LedgerFactory` to provide a full-blown `ReadWriterService` rather than a `LedgerReaderWriter` (needed by at least vDAML)
CHANGELOG_BEGIN
CHANGELOG_END
* Address review points
* Address Samir's review point in `SqlLedgerFactory`
* Finish addressing Samir's review point in `SqlLedgerFactory`
* Split reader and writer owners in `LedgerFactory` as suggested by Gerolf
* Remove unneeded `val` from `KeyvalueParticipantState[Reader|Writer]` constructor params
* Remove unneeded type parameter from `app.Runner`
* Leave `LedgerFactory` a full ledger builder but split hierarchy upwards and clarify responsibilities
* Rename `SimpleLedgerFactory` to `KeyValueLedgerFactory`
* sandbox-next: Make the Runner a real ResourceOwner.
* sandbox: Don't construct the ResetService twice.
* sandbox: Inline and simplify methods in StandaloneApiServer.
* resources: Define a `ResettableResource`, which can be `reset()`.
`reset()` releases the resource, performs an optional reset operation,
and then re-acquires it, binding it to the same variable.
* resources: Pass the resource value into the reset operation.
* sandbox: Fix warnings in `TestCommands`.
* sandbox-next: Add the ResetService.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Make sure the SandboxResetService resets asynchronously.
It was being too clever and negating its own asynchronous behavior.
* sandbox-next: Forbid no seeding.
This double negative is really hard to phrase well.
* sandbox-next: Implement ResetService for a persistent ledger.
* sandbox: Delete the comment heading StandaloneIndexerServer.
It's no longer meaningful.
* sandbox-next: No need to wrap the SandboxResetService in an owner.
* sandbox-next: Bump the ResetService test timeouts.
It looks like it's definitely slower than on Sandbox Classic™. Gonna
look into this as part of future work.
* Revert to previous asynchronous reset behavior
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* libs-scala/ports: Wrap socket ports in a type, `Port`.
* sandbox: Use `Port` for the API server port, and propagate.
CHANGELOG_BEGIN
CHANGELOG_END
* extractor: Use `Port` for the server port.
* ports: Make Port a compile-time class only.
* ports: Allow port 0; it can be specified by a user.
* ports: Publish to Maven Central.
* kvutils: Make the `KeyValueParticipantStateReader` tests more rigorous.
If the `offset` is specified as `None`, expect it to be `None`, not
just anything.
* kvutils: Simplify `KeyValueParticipantStateReader#stateUpdates`.
Construct the Source with `Source.apply`, not `Source.fromIterator`.
* kvutils: Use multiple entry IDs in `KeyValueParticipantStateReaderSpec`.
* kvutils: Add basic tests to `KeyValueParticipantStateReaderSpec`.
* kvutils: Add heartbeats to `LedgerReader`'s `events` output.
Heartbeats are optional, to be delivered by the ledger if and when it
deems necessary.
* sandbox-next: An observing time service backend using Akka streams.
* sandbox-next: A regular heartbeat based on Akka Streams' `tick`.
* sandbox: Replace `TimeServiceBackend.withObserver` with `.observing`.
More code, but it's more decoupled, so can more easily be sent to the
underlying backend in Sandbox Next.
CHANGELOG_BEGIN
- [Sandbox] Fixed a bug in the command completions stream when running
Sandbox in static time. Previously, upon updating the time, the old
time was emitted on the completions stream. The new time is now
emitted.
CHANGELOG_END
* sandbox: TimeServiceBackend should only emit accepted changes.
* ledger-on-memory: Use `LedgerRecord` directly.
* ledger-on-memory: Stream heartbeats to the log.
* ledger-on-memory: Encapsulate mutations behind locks at all times.
* ledger-on-memory: Differentiate between reading and writing.
* ledger-on-memory: Factor out appending to the log.
* kvutils: Move the heartbeat test into the base from ledger-on-memory.
* kvutils: Log when the submission validation fails unexpectedly.
* ledger-on-sql: Add a script to hash all migrations.
* ledger-on-sql: Publish heartbeats to the log, and stream them out.
* ledger-on-sql: Log if publishing the heartbeat failed.
* ledger-on-sql: Wrap all queries in `Try`.
Just to make sure that we don't throw from a function that returns `Try`
or `Future`.
* ledger-on-sql: Allow `Long` values as the heartbeat timestamp.
`INTEGER` really does mean 32-bit, apparently.
* sandbox-next: Pipe heartbeats to the ledger.
* ledger-on-sql: Make sure we publish the correct head after a heartbeat.
Off-by-one errors are the best errors.
* ledger-on-(memory|sql): Just accept heartbeats, not their owner.
* sandbox: Update CIDs in tests to account for the extra heartbeat.
* ledger-on-memory: Fix a reference to variable in a comment.
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* ledger-on-sql: `flatMap` over `Try` rather than `Future` when possible.
* sandbox: Make sure the heartbeat queues are thread-safe.
* kvutils: Remove `LoggingContext` from the interfaces.
Keep it internally. This means we'll drop any context, but otherwise
things should work as expected.
* sandbox-next: Pull out the heartbeat interval into a constant.
* ledger-on-sql|sandbox: Clarify large levels of nesting.
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Freeze DAML-LF 1.8
Two minor points that I did not mention in the previous PR:
We also include the renaming of structural records to `struct` and the
renaming of `Map` to `TextMap`.
There are some minor changes around the LF encoder tests which need to
be able to emit package metadata properly so I’ve added it to the
parser. Sorry for not splitting that out.
Following the process used for the DAML-LF 1.7 release, this does not
yet include the frozen proto file.
changelog_begin
- [DAML-LF] Release DAML-LF 1.8:
* Rename structural records to ``Struct``. Note that
structural records are not exposed in DAML.
* Rename ``Map`` to ``TextMap``.
* Add type synonyms. Note that type synonyms are not serializable.
* Add package metadata, i.e., package names and versions.
Note that the default output of ``damlc`` is stil DAML-LF 1.7. You
can produce DAML-LF 1.8 by passing ``--target=1.8``.
changelog_end
* Update encoder
* Update java codegen tests
* Update comment in scala codegen
* Handle TSynApp in interface reader
* Bump lf_stable_version to 1.7
* Fix kvutils tests
* Make kvutils work with the new contract id scheme
CHANGELOG_BEGIN
- [KVUtils] uses random contract id. Contract ids are made of 65 hexa decimal characters.
CHANGELOG_END
Co-authored-by: Jussi Mäki <jussi.maki@digitalasset.com>
* sandbox: Name the arguments to `ApiServices.create` for clarity.
* sandbox: Clarify numbers and types in configuration classes.
* sandbox-next: Log the correct port on startup.
* sandbox-next: Connect up the command configuration.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-next: Wire up TLS configuration.
* sandbox-next: Wire up the maximum inbound message size.
* sandbox-next: Set the global log level if specified.
And if it's not specified, default to the level in logback.xml, INFO.
* sandbox-next: Connect up the submission configuration.
* sandbox-next: Log the correct ledger ID.
* sandbox-next: Use `TimeProvider.UTC`.
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.
InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.
* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.
* Run all conformance tests in the repository in verbose mode.
This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.
This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:
- ledger-api-test-tool-on-canton
- ledger-on-memory
- ledger-on-sql
- sandbox
Fixes#4225.
CHANGELOG_BEGIN
CHANGELOG_END
* Add overridable indexer, api and auth configuration to `LedgerFactory`
CHANGELOG_BEGIN
CHANGELOG_END
* Add overridable indexer and api metrics creation to `LedgerFactory`
CHANGELOG_BEGIN
CHANGELOG_END
* Add overridable api's `TimeServiceBackend` to `LedgerFactory`
* 🎨 Fix formatting
* Port SDK ledgers based on `Runner` (and the sandbox) to `TimeServiceBackend`
* Revert to `TimeProvider` for committer usage and to `None` default for API server.
Also removed now unused `TimeServiceProvider.wallClock()`.
* Move TimeServiceBackend back to the API server.
* 🎨 Remove unneeded argument passed for parameter w/default
* Restore sandbox ledger time support
* Simplify passing a `TimeProvider` to the sandbox ledger
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* sandbox: If the ledger ID isn't provided, use the one in the database.
Previously, we would fail if working against an existing ledger, and not
explicitly providing the ledger ID. This was the case even if the ledger
ID was randomly generated initially.
CHANGELOG_BEGIN
- [Sandbox] If no ledger ID is provided when running against an existing
ledger, use the existing ID. Previously, Sandbox would fail to start.
CHANGELOG_END
* sandbox: The ReadOnlySqlLedger should always receive a ledger ID.
It's read-only; it can't create one.
* sandbox: Stop using `equal` in SqlLedgerSpec.
* sandbox: Test that the ledger ID is as specified in SqlLedgerSpec.
* sandbox: Let the top-level runner handle a ledger ID mismatch.
And clean up the log text.
* sandbox: Initialize the ledger properly when the ID is dynamic.
* sandbox: Use `Vector`, not `List`, for SqlLedger initialization.
Append with Vector, good. List, bad.
* ledger-api-common: Make `LedgerApiMode.Dynamic` an object.
And add Java-style static factory methods.
* kvutils/app | ledger-on-{memory,sql}: Make `ledgerId` optional.
It should be generated or retrieved from the persistence layer by the
ledger itself.
* kvutils: Make the ledger ID optional in the tests.
* ledger-on-sql: Store the ledger ID, and reject conflicting IDs.
* ledger-on-sql: Make more things final.
* ledger-on-sql: Document the `ledger_meta.table_key` column better.
* sandbox: Don't hardcode the number of packages in the test DAR.
It changes.
* ledger-on-sql: Merge the `head` resource owner with the `dispatcher`.
* sandbox: Use backticks to simplify pattern match in ReadOnlySqlLedger.
* ledger-on-sql: Extract methods in `owner`.
* Add type-level strings in DAML.
This PR adds a `PromotedText` stable package, with `PromotedText` type, which is used to encode type-level strings from DAML into DAML-LF. The reason for this is to preserve the `HasField` instance argument. This PR adds a test that `HasField` is succesfully reconstructed incontexts, during data-dependencies, which wasn't possible before.
changelog_begin
changelog_end
* adresss comments
* fix overly specific tests
* Use KeyHasher to serialize contract keys in kvutils
- Use Value instead of VersionedValue in GlobalKey as the versioning does not make sense here
and may be misleading as the a value with a different version but same meaning would still
be the same key.
- Relocate the KeyHasher to ledger-api-common so kvutils can use it (otherwise cyclic dependencies)
- Replace storing of the contract key as a VersionedValue with the hash produced by KeyHasher.
This is backwards incompatible. A compatible option would require us to query the key with both
the old way and the new way which is unattenable. We're making a calculated breaking change.
CHANGELOG_BEGIN
- [DAML Ledger Integration Kit] Serialize contract keys using a hash instead of the value in kvutils.
This is a backwards incompatible change to kvutils.
CHANGELOG_END
* Use proper hasher for contract keys and not KeyHasher
- Use Hash.scala, not KeyHasher.scala.
- Add hash to GlobalKey as we want the hash to be computed from the inside.
The use of KeyHasher will be later deprecated and replaced by this.
* Use "sealed abstract case class" trick instead of private ctor
and rebase fix
* Revert change to unsupported value version decode error
* Reformat code
* Add kvutils changelog entry and bump the version
* kvutils: Extract a committer from the uses of `SubmissionValidator`.
This makes the clock injectable too.
* kvutils: Provide logging contexts in the `Runner`.
* sandbox: Remove the `StaticAllowBackwards` time provider type.
It's not used anywhere.
* sandbox: Fix warnings in CliSpec.
* sandbox: Ensure that we cannot specify both static and wall-clock time.
* sandbox-next: Crash if wall clock time is not specified.
* sandbox-next: Document more known issues in the new Sandbox.
* sandbox: Add a Clock (and some tests) to TimeServiceBackend.
* sandbox-next: Support static time.
CHANGELOG_BEGIN
- [Sandbox Next] Re-establish static time mode.
CHANGELOG_END
* ledger-on-(memory|sql): Expect a `() => Instant`, not a `Clock`.
* kvutils/ledger-on-sql: Avoid a race condition in dispatching.
This changes the API of kvutils to allow for passing data out of the
transaction, which makes it much easier to ensure the new head makes it
to the dispatcher. Previously, we would use an `AtomicLong` to
communicate the data, but this was problematic because values could
arrive out in the wrong order. For example:
- log index 5 is committed
- `head` is updated to 6
- the dispatcher is signalled with a head of 6
- log index 6 is committed
- log index 7 is committed
- `head` is updated to 8
- `head` is updated to 7
- the dispatcher is signalled with a head of 7
In this scenario, we would have to wait until a new commit comes in
before the indexer finds out about log index 7.
* kvutils: Just return an `Either`from `SubmissionValidator`.
It was either that or introduce yet another type to split
`SubmissionValidated` into two.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Make ValidationFailed extend NoStackTrace.
* kvutils: Make it easier to import the Key and Value types.
Moved them into a companion object.
* kvutils: Flatten the nested `for` comprehensions in SubmissionValidator.
* kvutils: Test that results succeed before querying state updates.
If they don't, we get idle timeouts and we don't see the error, which is
unhelpful.
* sandbox: Increase log levels for failed submissions.
* kvutils: Avoid converting DamlStateKey -> ByteString -> Array[Byte].
Just convert once.
* ledger-on-sql: Use the `SubmissionValidator` from kvutils.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Remove unnecessary interpolation in ApiSubmissionService.
* Translate unsupported kinds to a special Erased type
This should simplify `data-dependencies` and avoid issues like #4470
since we can match on the type instead of having to guess which types
can and which cannot be translated back to DAML.
changelog_begin
changelog_end
* Prevent building of en/de-coders of Cid, Val, and Nid outside Value/Transaction Coder
* DAML-LF: remove constrains on Cid for LF encoder/decoder
* Address Samir coments
* resources: Allow for startup exceptions without no stack trace.
* sandbox: Remove dependency duplication in the BUILD file.
* ledger-on-(memory|sql): Extract `Main` into its own file tree.
This is so we can remove the dependency on kvutils/app from the library.
logback.xml goes with it.
* sandbox: Implement "Sandbox Next" on top of ledger-on-sql.
CHANGELOG_BEGIN
- [Sandbox] Preliminary work to rebuild Sandbox on top of the DAML
Ledger Integration Kit. Currently not exposed through the CLI.
CHANGELOG_END
* ledger-on-sql: Publish to Maven Central.
* sandbox: If an unknown database is passed to Sandbox Next, handle it.
* resources: Catch errors in owner construction or acquisition.
Sometimes we're not in a future yet.
* resources: Don't run ProgramResource error handling inside the EC.
Otherwise we end up in deadlock which is only broken by
`awaitTermination` timing out.
* kvutils: Add idle timeouts to all participant state tests.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Add nesting to test names in the participant state tests.
And clean up the names.
* kvutils: Use durations in participant state tests.
Makes it way easier to read than `addMicros(…)` calls.
* daml-lf/data: Fix warnings in Time.scala.
* daml-lf/data: Add `Timestamp#add`.
* kvutils: An MRT of (start time + 1 second) is too short for CI.
Increasing the MRT to (now + 10 seconds).
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Add a prefix of "ledger_" to the Flyway migrations table.
And while we're at it, make this a constant in the code and use Flyway
placeholders to substitute it in.
We don't have any users right now, so changing the migrations should be
fine.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run the indexer on PostgreSQL when possible.
Specifically, when running the PostgreSQL conformance tests.
It will also work when running it as a binary, but you need to specify
both the `--server-jdbc-url` and `--jdbc-url` switches. This is to keep
kvutils/app simple.
* ledger-on-sql: Make `ExtraConfig` final.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* resources: Revert `ResourceOwner.failed` to `ResourceOwner[Nothing]`.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* kvutils|ledger: Don't use Scala's global execution context.
I have just discovered that Scala's global execution context uses daemon
threads, which means they don't stick around past the end of the `main`
function. This is bad when application startup fails and we end up with
no other threads; the application terminates before it has a chance to
print the error message, which is very unhelpful.
We resolve this by creating a cached thread pool in the
`ProgramResource` class, specifically used for acquisition and release
of resources.
To do this, we need to pass the execution context through to the
resource constructors, which makes usage a little more verbose, because
now we can't use lambda functions (which can't take implicit arguments
such as the `executionContext`).
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils-app|reference-v2: Sort `Runner` and `ReferenceServer` methods.
clean up String Identifier in daml-lf
- Separate LedgerString from ContractIdString
- Drop TransactionIdString from daml-lf
- Create a new ContractIdString (aka. ContractIdStringV1) for new contractId comparable with relative contractIds
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Use `ResourceOwner#use` to simplify tests.
* kvutils: Merge in `RestartSpec` from ledger-on-sql.
CHANGELOG_BEGIN
- [KVUtils] Added a test case to the participant state tests to ensure
your ledger state is resumable upon restart.
CHANGELOG_END
* ledger-on-posix-filesystem: Extract paths into their own class.
* ledger-on-posix-filesystem: Seed the dispatcher with the correct head.
* kvutils: In participant state tests, be clear about different strings.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
