* Push down completion requests to data access layer
This is largely a refactoring. The externally observable behavior is unchanged, but:
- a sub-dao is created for command completions (with the intent of breaking up the dao completely in future commits)
- the command completions dao can, in theory, directly fetch completions off the index
- in practice this is not implemented here to keep this PR as small as possible
Filtering ledger entries to get completions is moved to a function that is in turn used by:
- the ledger dao
- the in-memory sandbox
The plan for the former is to add a new table where completion-relevant data is stored so that it can be fetched quickly.
The plan for the latter is to get rid of it once DAML-on-SQL ships.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix off-by-one error in the in-memory sandbox
* Add type-level strings in DAML.
This PR adds a `PromotedText` stable package, with `PromotedText` type, which is used to encode type-level strings from DAML into DAML-LF. The reason for this is to preserve the `HasField` instance argument. This PR adds a test that `HasField` is succesfully reconstructed incontexts, during data-dependencies, which wasn't possible before.
changelog_begin
changelog_end
* adresss comments
* fix overly specific tests
* Use KeyHasher to serialize contract keys in kvutils
- Use Value instead of VersionedValue in GlobalKey as the versioning does not make sense here
and may be misleading as the a value with a different version but same meaning would still
be the same key.
- Relocate the KeyHasher to ledger-api-common so kvutils can use it (otherwise cyclic dependencies)
- Replace storing of the contract key as a VersionedValue with the hash produced by KeyHasher.
This is backwards incompatible. A compatible option would require us to query the key with both
the old way and the new way which is unattenable. We're making a calculated breaking change.
CHANGELOG_BEGIN
- [DAML Ledger Integration Kit] Serialize contract keys using a hash instead of the value in kvutils.
This is a backwards incompatible change to kvutils.
CHANGELOG_END
* Use proper hasher for contract keys and not KeyHasher
- Use Hash.scala, not KeyHasher.scala.
- Add hash to GlobalKey as we want the hash to be computed from the inside.
The use of KeyHasher will be later deprecated and replaced by this.
* Use "sealed abstract case class" trick instead of private ctor
and rebase fix
* Revert change to unsupported value version decode error
* Reformat code
* Add kvutils changelog entry and bump the version
CHANGELOG_BEGIN
- [Sandbox] Static time mode is being deprecated in the future. A warning has been added to notify users of this fact.
- [Sandbox] Scenarios are being deprecated in the future, in favor of `DAML Script <https://docs.daml.com/daml-script/>`_. A warning has been added to notify users of this fact.
CHANGELOG_END
* kvutils: Extract a committer from the uses of `SubmissionValidator`.
This makes the clock injectable too.
* kvutils: Provide logging contexts in the `Runner`.
* sandbox: Remove the `StaticAllowBackwards` time provider type.
It's not used anywhere.
* sandbox: Fix warnings in CliSpec.
* sandbox: Ensure that we cannot specify both static and wall-clock time.
* sandbox-next: Crash if wall clock time is not specified.
* sandbox-next: Document more known issues in the new Sandbox.
* sandbox: Add a Clock (and some tests) to TimeServiceBackend.
* sandbox-next: Support static time.
CHANGELOG_BEGIN
- [Sandbox Next] Re-establish static time mode.
CHANGELOG_END
* ledger-on-(memory|sql): Expect a `() => Instant`, not a `Clock`.
* Add more tests to default run of Ledger API Test Tool
Furthermore, drop TimeIT, which is a sandbox-only property (and tested already as part of its integration tests)
CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Ledger API Test Tool default tests modified. Use --list for the updated list of default tests. Time service test dropped from the suite.
CHANGELOG_END
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380635979
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380636989
* Optimize imports
* Only run semantic tests on Canton
* kvutils/ledger-on-sql: Avoid a race condition in dispatching.
This changes the API of kvutils to allow for passing data out of the
transaction, which makes it much easier to ensure the new head makes it
to the dispatcher. Previously, we would use an `AtomicLong` to
communicate the data, but this was problematic because values could
arrive out in the wrong order. For example:
- log index 5 is committed
- `head` is updated to 6
- the dispatcher is signalled with a head of 6
- log index 6 is committed
- log index 7 is committed
- `head` is updated to 8
- `head` is updated to 7
- the dispatcher is signalled with a head of 7
In this scenario, we would have to wait until a new commit comes in
before the indexer finds out about log index 7.
* kvutils: Just return an `Either`from `SubmissionValidator`.
It was either that or introduce yet another type to split
`SubmissionValidated` into two.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Make ValidationFailed extend NoStackTrace.
* Simplify participant index access for contract lookups
CHANGELOG_BEGIN
[Sandbox] Participant index contract lookups simplified, should speed up command interpretation.
CHANGELOG_END
* Deserialize contracts off of the database connection thread pool
* Return proper code for invalid authentication
CHANGELOG_BEGIN
[Sandbox] If authentication is enabled, requests without a valid
authentication are going to be rejected with an ``UNAUTHENTICATED``
return code instead of ``PERMISSION_DENIED``.
CHANGELOG_END
* Reduce logging noise from java-rxbindings tests
* Fix rxjava bindings tests to match new behavior
* Fix extractor tests to match new behavior
* Address https://github.com/digital-asset/daml/pull/4485#discussion_r378507478
* kvutils: Make it easier to import the Key and Value types.
Moved them into a companion object.
* kvutils: Flatten the nested `for` comprehensions in SubmissionValidator.
* kvutils: Test that results succeed before querying state updates.
If they don't, we get idle timeouts and we don't see the error, which is
unhelpful.
* sandbox: Increase log levels for failed submissions.
* kvutils: Avoid converting DamlStateKey -> ByteString -> Array[Byte].
Just convert once.
* ledger-on-sql: Use the `SubmissionValidator` from kvutils.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Remove unnecessary interpolation in ApiSubmissionService.
* sandbox: Move more resource acquisition into the `owner`.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Reimplement SandboxClientResource as a resources.Resource.
* codegen: Use resources in TestUtil.
* sandbox: Manage PostgreSQL in tests with ResourceOwners.
* Translate unsupported kinds to a special Erased type
This should simplify `data-dependencies` and avoid issues like #4470
since we can match on the type instead of having to guess which types
can and which cannot be translated back to DAML.
changelog_begin
changelog_end
* sandbox: Don't hold on to old resources when resetting.
Now there's one hell of a memory leak.
CHANGELOG_BEGIN
- [Sandbox] Fixed a memory leak when using the ResetService; not
everything was cleaned up correctly.
CHANGELOG_END
* sandbox: Split out SandboxClientResource from SandboxServerResource.
Gonna replace SandboxServerResource with a ResourceOwner acquisition.
* sandbox: Don't capture the API server in the SandboxServer resource.
When we reset, this is stored forever, leading to a memory leak.
Tested by rewriting the SandboxServerResource to use
`SandboxServer.owner`.
* sandbox: Revert the test client resource to calling `shutdownNow()`.
* sandbox: Make sure the fixture is recreated properly on each test run.
* sandbox: Make `SandboxState` a non-case class.
The `toString()` was unnecessarily heavy.
* sandbox: Futures, futures everywhere.
Avoid a race condition where the server is stopped before it starts by
storing a `Future[SandboxState]` rather than the `SandboxState` itself.
This doesn't trigger the same memory leak as storing a
`Resource[SandboxState]` because we don't capture the object itself in
the `flatMap` in the same way with `Future`.
* sandbox: Remove an unused parameter left in for debugging.
* sandbox: Replace `@VisibleForTesting` with a comment.
* sandbox: Add more comments to the weird logic in SandboxServer.
* sandbox: Get rid of the `Port` type alias; it was confusing.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Shuffle participants used in tests
* Add no-wait-for-parties flag to the test tool
This flag is needed for privacy-preserving ledgers that do not expose
party allocations to non-submitting participants.
* Add shuffle-participants flag to test-tool
CHANGELOG_BEGIN
CHANGELOG_END
* Address code review. Make --no-wait-for-parties hidden.
In a committed transaction (i.e. a transaction with only absolute
contract ids), local divulgence is a non-concept.
Therefore local divulgences don't need to be treated and all code
related to it can be removed.
This also removes some duplication between InMemoryLedger and SqlLedger.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix indexing of transactions referencing divulged contracts
When a transaction referenced a divulged contract that had no contract
metadata (e.g. it was only known to the participant via divulgence),
the indexer failed with InactiveDependencyError.
This fixes the issue by modifying lookupContractLet query to return
LetUnknown if the contract is known, but has no metadata.
No changelog entry to add as this bug did not affect any users as this
surfaced with privacy extensions. Canton not affected as they always
include divulged contracts, even if they already have been divulged.
CHANGELOG_BEGIN
CHANGELOG_END
* Failing test-case for lookupContractLet
* Address PR review
* Second pass addressing review
* Prevent building of en/de-coders of Cid, Val, and Nid outside Value/Transaction Coder
* DAML-LF: remove constrains on Cid for LF encoder/decoder
* Address Samir coments
* resources: Allow for startup exceptions without no stack trace.
* sandbox: Remove dependency duplication in the BUILD file.
* ledger-on-(memory|sql): Extract `Main` into its own file tree.
This is so we can remove the dependency on kvutils/app from the library.
logback.xml goes with it.
* sandbox: Implement "Sandbox Next" on top of ledger-on-sql.
CHANGELOG_BEGIN
- [Sandbox] Preliminary work to rebuild Sandbox on top of the DAML
Ledger Integration Kit. Currently not exposed through the CLI.
CHANGELOG_END
* ledger-on-sql: Publish to Maven Central.
* sandbox: If an unknown database is passed to Sandbox Next, handle it.
* resources: Catch errors in owner construction or acquisition.
Sometimes we're not in a future yet.
* resources: Don't run ProgramResource error handling inside the EC.
Otherwise we end up in deadlock which is only broken by
`awaitTermination` timing out.
* sandbox: Fix RecoveringIndexer so it doesn't wait too long.
If the indexer restart delay is too short, RecoveringIndexer will still
wait a second. This is not an issue in production, because the delay is
typically 10 seconds, but in tests, where we use 100 milliseconds, it's
enough to make the test flaky.
We need to wait in one-second increments to allow for quick
cancellation; we don't want the user to have to wait 10 seconds if the
process is terminated while restarting.
The fix is to take the minimum of 1 second and the remaining restart
time.
This also adds STDOUT logging back into the tests; it was invaluable
when debugging the issue.
I am not including a changelog entry, because this only affects tests.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Add an assertion in IndexerIT to ensure it stops quickly.
We already test the logs, but it's probably wise to test that the
indexer doesn't wait 10 seconds.
* kvutils: Add idle timeouts to all participant state tests.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Add nesting to test names in the participant state tests.
And clean up the names.
* kvutils: Use durations in participant state tests.
Makes it way easier to read than `addMicros(…)` calls.
* daml-lf/data: Fix warnings in Time.scala.
* daml-lf/data: Add `Timestamp#add`.
* kvutils: An MRT of (start time + 1 second) is too short for CI.
Increasing the MRT to (now + 10 seconds).
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Add a prefix of "ledger_" to the Flyway migrations table.
And while we're at it, make this a constant in the code and use Flyway
placeholders to substitute it in.
We don't have any users right now, so changing the migrations should be
fine.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run the indexer on PostgreSQL when possible.
Specifically, when running the PostgreSQL conformance tests.
It will also work when running it as a binary, but you need to specify
both the `--server-jdbc-url` and `--jdbc-url` switches. This is to keep
kvutils/app simple.
* ledger-on-sql: Make `ExtraConfig` final.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* resources: Revert `ResourceOwner.failed` to `ResourceOwner[Nothing]`.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* kvutils|ledger: Don't use Scala's global execution context.
I have just discovered that Scala's global execution context uses daemon
threads, which means they don't stick around past the end of the `main`
function. This is bad when application startup fails and we end up with
no other threads; the application terminates before it has a chance to
print the error message, which is very unhelpful.
We resolve this by creating a cached thread pool in the
`ProgramResource` class, specifically used for acquisition and release
of resources.
To do this, we need to pass the execution context through to the
resource constructors, which makes usage a little more verbose, because
now we can't use lambda functions (which can't take implicit arguments
such as the `executionContext`).
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils-app|reference-v2: Sort `Runner` and `ReferenceServer` methods.
This has outlived its usefulness as an experiment in how to implement a
kvutils-based leger, and is now just a source of flaky test runs.
🔥
CHANGELOG_BEGIN
CHANGELOG_END
`App` is fairly problematic. It depends on `DelayedInit`, which is
deprecated because it has some surprising behavior.
Best to avoid it for anything more than a simple demo application.
Details: https://github.com/scala/bug/issues/4330
CHANGELOG_BEGIN
CHANGELOG_END
clean up String Identifier in daml-lf
- Separate LedgerString from ContractIdString
- Drop TransactionIdString from daml-lf
- Create a new ContractIdString (aka. ContractIdStringV1) for new contractId comparable with relative contractIds
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Use `ResourceOwner#use` to simplify tests.
* kvutils: Merge in `RestartSpec` from ledger-on-sql.
CHANGELOG_BEGIN
- [KVUtils] Added a test case to the participant state tests to ensure
your ledger state is resumable upon restart.
CHANGELOG_END
* ledger-on-posix-filesystem: Extract paths into their own class.
* ledger-on-posix-filesystem: Seed the dispatcher with the correct head.
* kvutils: In participant state tests, be clear about different strings.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* sandbox: Move common API and indexer deps into their own packages.
Primarily `com.digitalasset.platform.index.store`.
* sandbox: Move API server deps into the `apiserver` and common packages.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Rename `….platform.index.store` to `….platform.store`.
* sandbox: Split `JdbcIndex` & co. out from `apiserver` into `index`.
Moar packages! And this one is nice and clean; there's only a few
dependencies.
* sandbox: Move `InMemoryPackageStore` into its own `packages` package.
`packages` package. What a tongue twister.
* sandbox: Fix warnings in `TransactionConversion`.
* sandbox: Inline `SandboxContractStore` into `LedgerBackedIndexService`.
* sandbox: Fix warnings in `InMemoryLedger` and `SqlLedger`.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Previously we were only cleaning up and exiting if loading the DAML
packages failed. Now we also handle API server errors.
An example of where this might happen is if the ledger ID specified is
different to the existing ledger ID, causing an error.
One day we'll kill the ResetService and all of this will go away. One
day…
CHANGELOG_BEGIN
- [Sandbox] Fix an error that stops the server from exiting cleanly if
API server initialization fails.
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* kvutils: Use matchers in ParticipantStateIntegrationSpecBase.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: In participant state tests, always test record time.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-posix-filesystem: Run `scalafmt`.
* ledger-on-posix-filesystem: Just use file channel locks. Seriously.
* ledger-on-posix-filesystem: Rename `FileSystemLock#run` to `apply`.
* ledger-on-posix-filesystem: Conformance tests now run quickly.
So we can stop constraining the number of concurrent test runs.
* ledger-on-posix-filesystem: Lock the entire ledger with a file lock.
This ensures we can't have two separate ledger servers running against
the same directory.
This is mostly an experiment to better understand how to do this on an
SQL ledger for #4039.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-posix-filesystem: Run `scalafmt`.
* ledger-on-posix-filesystem: Use `Resource#use` in `FileSystemLock`.
* resources: Move `Resource#use` to `ResourceOwner#use`.
Makes more sense for the same object to govern both `acquire()` and
`release()`.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* resources: Make it easy to use a resource and then release it.
* ledger-on-sql: Move database transaction methods to `Database`.
* ledger-on-sql: Make sure the ledger picks up where it left off.
This means seeding the dispatcher with the correct head, rather than
starting at 1 every time.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Move `Index` and `StartIndex` to the package object.
They're used everywhere.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
The `sandbox-perf` build has been failing for a while with the following
errors:
```
INFO: From Generating benchmark code for //ledger/sandbox-perf:sandbox-perf_codegen:
JMH benchmark generation: JMH Benchmark generator failed
JMH benchmark generation: Benchmark classes should not be final.
[com.digitalasset.platform.sandbox.perf.LargeTransactionBench]
JMH benchmark generation: The instantiated @State class cannot be abstract.
[com.digitalasset.platform.sandbox.perf.PerfBenchState]
```
However, these errors are ignored; running the benchmarks runs the
`AcsBench` benchmark and ignores the fact `LargeTransactionBench` and
`SimpleBench` failed to compile.
I've fixed the errors by making sure that classes are not final, and
that `SimpleBench` uses a concrete state class.
This also introduces a cheap patch to the Scala JMH Bazel rules that
makes sure they fail if there are any errors.
I'm not really sure how to patch the Bazel rules properly, but someone
else might have an idea. :-)
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Unfortunately, downgrading `scalafmt` resulted in files that would be
reformatted to invalid code, because lines would be merged but trailing
commas would be left in.
I've manually run it everywhere and fixed these issues so people don't
have to fix them as they go.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Don't allow unnamed in-memory databases.
It seems that changing the connection pool size at runtime causes weird
issues, predominantly that sometimes all connections are dropped and
therefore the in-memory database is lost after migration. Obviously, it
then stops working.
This works around this problem by simply not allowing unnamed in-memory
databases, guaranteeing that we can share the database between the
various connection pools.
This will probably be necessary anyway when attempting to share the
database between the ledger and the index.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Make it clear we're generating a new JDBC URL in tests.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Pull out a superclass for the various integration tests.
* ledger-on-sql: Use Flyway to migrate tables in an idempotent manner.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Make it easy to run it with Bazel for experimentation.
* ledger-on-sql: Test that migrations will never change in the future.
* ledger-on-sql: Add a prefix of "ledger_" to the tables.
This is so we don't accidentally conflict with the index when the
schemas are shared.
I am letting myself modify the migrations because the existing
migrations haven't been merged into `master` yet.
* ledger-on-sql: Explain why we change the pool size after migration.
* ledger-api-test-tool: Tests now specify a timeout scale, not a timeout.
Makes it easier to change the default timeout in one place.
* ledger-api-test-tool: Increase timeouts on slow tests.
These tests produce a lot of volume and can make CI flaky.
* ledger-on-sql: Only tear down PostgreSQL in tests.
For other databases, we just create a new file for each test case.
* ledger-on-sql: Reduce the log output in tests.
* ledger-on-sql: Use a separate connection pool for Flyway when possible.
Apparently `setMaximumPoolSize` doesn't really have the desired effect
after the connection pool has already been used. The new test case will
be flaky if we process more than one commit in parallel.
For SQLite, it seems to be OK.
* ledger-on-sql: Use a separate connection pool for Flyway with SQLite.
Except in memory.
* ledger-on-sql: Use a separate PostgreSQL database for each test.
Because performance, innit. Don't have to tear them down.
* ledger-on-sql: Move the `Queries` values inside the `RDBMS` objects.
* ledger-on-sql: Go into even more detail about pool size hijinks.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
It affects basically every file, so let's do it all at once.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Three problems here:
1. On failure, the Sandbox didn't close down the metrics reporting
thread, meaning the application would hang.
2. If there's a failure:
a. we didn't print it, and
b. we didn't exit with the correct status code.
These are now handled.
CHANGELOG_BEGIN
- [Sandbox] On initialization error, report the error correctly and exit
with a status code of 1. Previously, the program would hang
indefinitely. (This regression was introduced in v0.13.41.)
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* sandbox: Ensure that the Flyway tests actually pick up the SQL files.
* sandbox: Upgrade Flyway from v5 to v6.
CHANGELOG_BEGIN
- [Sandbox] Upgrade the Flyway database migrations library from v5 to v6.
CHANGELOG_END
* Add logging context information to ledger services
Rolls back that AllowedType implicit, which made the logging context
less usable and didn't provide that much of an advantage. Replaced with
helpers where it makes sense that take care of turning the value safely
into a string.
Closes#3699
CHANGELOG_BEGIN
CHANGELOG_END
* Remove unused import
* Address https://github.com/digital-asset/daml/pull/4205#discussion_r370704306
* ledger-on-sql: Don't bother cleaning up after integration tests.
Turns out Bazel cleans up before each test run, so we only have one
set of outputs at a time. This is far more useful for debugging anyway.
* ledger-on-sql: Pull out a test-lib to simplify the conformance tests.
* ledger-on-sql: Turn `Database` from a sealed trait into a case class.
* ledger-on-sql: Support for PostgreSQL!
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run the conformance tests against PostgreSQL.
* ledger-on-sql: Run the LotsOfParties conformance tests.
* ledger-on-sql: Use PostgreSQL's `RETURNING` keyword to save a query.
* ledger-on-sql: Ensure the reader connection pool is read-only.
* ledger-on-sql: Avoid cloning ByteStrings as byte arrays where possible.
Instead, use `ByteString#newInput()`, which returns an `InputStream`.
This is supported by H2 and PostgreSQL, but unfortunately not SQLite.
* ledger-on-sql: Run integration tests in parallel.
* Update the PostgreSQL and SQLite JDBC drivers.
* postgresql-testing: Extract `PostgresAround` from the Sandbox tests.
CHANGELOG_BEGIN
CHANGELOG_END
* postgresql-testing: Split into several files.
* postgresql-testing: Pull out the host and database names into variables.
* postgresql-testing: Don't publish to Maven.
I have no idea why the build wasn't failing before.
* postgresql-testing: Assimilate the `@postgresql_dev_env` dependency.
* kvutils: Use resources in KeyValueParticipantState, not AutoCloseable.
CHANGELOG_BEGIN
- [kvutils] The simplified API now uses ``com.digitalasset.resources``
to manage acquiring and releasing resources instead of ``Closeable``.
CHANGELOG_END
* ledger-on-posix-filesystem: Remove dead code due to a refactoring fail.
* Support DAML-LF type synonyms in scala.
CHANGELOG_BEGIN
CHANGELOG_END
* dont create synonymns in GenerateSimpleDalf
* extend DAML-LF parser to support type synonyms
* test: expand type synonyms correctly
* ledger-api-test-tool-on-canton: Upgrade Canton to v0.7.0.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-test-tool-on-canton: Use the health endpoint instead of `nc`.
Netty prints a warning when kicking it with `nc -z` from Nix's
netcat-gnu. Not other netcats though… don't know why.
* ledger-api-test-tool-on-canton: Let IntelliJ format the shell script.
* ledger-api-test-tool-on-canton: Stop Canton with SIGINT.
For some reason, Canton doesn't print out a load of errors if it's
stopped with SIGINT (Ctrl+C). SIGTERM, on the other hand…
* ledger-api-test-tool-on-canton: Write Canton logs to STDOUT.
Now the logs are no longer noisy, we can just pump them to STDOUT. This
should make debugging any transient issues on CI a lot easier.
* libs-scala: Make sure `RetryStrategy` works if `attempts.isEmpty`.
* ledger-on-posix-filesystem: Only retry on expected lock errors.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-posix-filesystem: Only create the lock attempt directory once.
Soooo much faster.
* ledger-on-posix-filesystem: Don't run the unit tests on Windows.
* kvutils: Make logback.xml a base file, rather than the only option.
* kvutils/app: Simplify logback.base.xml.
Mostly by getting rid of unnecessary appenders.
* ledger-on-sql: Add trace logging for all database work.
* Upgrade H2; there's a few useful bug fixes.
* kvutils/app: Let the user override the server JDBC URL.
* kvutils/app: Provide a way to specify the ledger ID.
* ledger-on-sql: If there are missing entries in the log, fail on read.
This can happen right now because we insert in parallel. Next step: stop
doing that.
* ledger-on-sql: Stop writing in parallel; it causes race conditions.
This unfortunately means we also stop _reading_ in parallel, which is
less fun.
* Revert "ledger-on-sql: Use a sequential log entry ID."
This reverts commit c58265bf43.
* ledger-on-sql: Create tables with columns that are not nullable.
* ledger-on-sql: The H2 conformance tests now work, just slowly.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run tests against H2 and SQLite on memory and disk.
* ledger-on-sql: Allow H2 to read from the log in parallel with writes.
* ledger-on-sql: Use UUIDs for entry IDs, not 8 random bytes.
* ledger-on-sql: Make sure to log the correlation ID.
* ledger-on-sql: Do less while holding a database connection.
* ledger-on-sql: Log the connection.
* ledger-on-sql: We don't need to tell H2 not to drop the connection.
The connection pool takes care of that.
* ledger-on-sql: Disable H2 conformance tests on CI; they're too slow.
* ledger-on-sql: Rename `loggingContext` to `logCtx`.
* ledger-on-sql: Don't abuse log contexts; put the data in the message.
* ledger-on-sql: Make the connection log line easier to read.
And pull out the logger, even if it's only used once.
* ledger-on-sql: Run the unit tests against (H2, SQLite) * (memory, file).
* engine: the natural order in Transaction is pre-order traversal
- Do not use anymore SortedMap for nodes in transaction
- Drop unused transaction traversal
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lg: improve generator of transactions
* Introduce new JWT payload format
... the reader still supports old formats
CHANGELOG_BEGIN
[Sandbox] The sandbox uses a new payload format for authentication tokens (JWTs).
The old format is deprecated, but still works.
[JSON API] The HTTP JSON API now uses the same payload format for authentication tokens as the sandbox.
The old format is deprecated, but still works.
CHANGELOG_END
* Add helper function for getting token party
* Support sandbox tokens in JSON API
* Add warning for deprecated formats
* Update documentation
* Add explicit test for new format
* Update JSON API documentation
* Fix test
CHANGELOG_BEGIN
[Sandbox] Metrics are now namespaced by "daml" and their names have been
standardize to snake_case
CHANGELOG_END
Please note that HikariJDBC records its own metrics but this is outside
of our reach, so connection-pool-related metrics will keep being called
with camelCase names.
Prohibit contract IDs in contract keys and add key maintainers to exercises
CHANGELOG_BEGIN
- [DAML-LF] Prohibit contract IDs in contract keys completely. Previously, creating keys containing absolute (but not relative) contract IDs was allowed, but `lookupByKey` on such a key would crash.
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
Co-authored-by: Stephen Compall <scompall@nocandysw.com>
This was accidentally set to 1.7 before for no reason. Note that this
simply meant that we did not use the stable module when compiling with
--target=1.6 since we filter out newer stable packages.
Nevertheless, I’ve also added a test that verifies that we do not
accidentally introduce dependencies on packages in newer LF versions.
changelog_begin
changelog_end
* sandbox: Tidy up the RecoveringIndexerSpec a little.
* sandbox: Move the JdbcLedgerDao owner constructor to its object.
* sandbox/damlonx: Make sure the RecoveringIndexer terminates.
Previously, if there was a failure and a termination at the same time,
we could recover after terminating, which means we'd end up in a weird
state. This makes sure we stop correctly.
Unfortunately, I couldn't find a good place to put these tests apart
from in reference-v2, as they need to depend on both _kvutils_ and
_sandbox_. Perhaps they belong in their own Bazel package.
This might address #3945; it's hard to say.
CHANGELOG_BEGIN
- [Indexer] Potentially fix a bug when recovering from failure.
CHANGELOG_END
* reference-v2: Address a race condition in the IndexerIT assertions.
* sandbox: Make sure the RecoveringIndexer can stop during a restart.
* reference-v2: Use Mockito to simulate errors in IndexerIT.
* sandbox: When the RecoveringIndexer is stopped, stop quickly.
* reference-v2: Make sure each test has its own metrics registry.
* Replace TestNamedLoggerFactory with in-memory appender
* sandbox: Test RecoveringIndexerSpec with a mock logger.
Using a real logger was causing massive stack traces in all other tests
that didn't have `RecoveringIndexerSpec$Appender` available, and adding
the appender programmatically didn't work for some reason.
The mock gives much nicer error messages on failure, too.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* libs-scala/resources: Move `Resource` and friends to their own package.
CHANGELOG_BEGIN
- [Ledger API Server] Publish the resource management code as a library
under ``com.digitalasset:resources``.
CHANGELOG_END
* libs-scala/resources: Add to the artifacts list.
* libs-scala/resources: Factor out the test helpers into test/lib.
* libs-scala/resources-akka: Make the Akka resources an optional dep.
* EC256 rather than EC512 algo
CHANGELOG_BEGIN
[Ledger API Authorization]Support EC256 algorithm for JWT rather than EC512
CHANGELOG_END
* support both 256 and 512 ECDSA algo
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* scalafmt
* correct comment, avoid unnecessary private def
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* correct BAZEL formatter
* Introduce ContextualizedLogger and LoggingContext
A ContextualizedLogger is a wrapper around an Slf4j logger. It uses
call-by-name parameters to only construct logging lines when necessary.
The underlying context can be used by invoking withoutContext.
More interestingly, every call to one of its logging methods requires
the presence of an implicit LoggingContext.
A LoggingContext is a way of definining a set of pairs that a
ContextualizedLogger can log without the need of it being specified for
every call.
A new context can be created with newLoggingContext and, given an
implicit LoggingContext, pairs can be added or overwritten using
withEnrichedLoggingContext.
Pairs in the context will be appended to each event logged by a
ContextualizedLogger in the following form:
some text logged explicitly (context: {key1=value1, key2=value2)
If the underlying Logger is configured to use the Logstash encoder the
keys and values in the LoggingContext will also be added to the logging
event encoded in JSON format.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestion in https://github.com/digital-asset/daml/pull/4046#pullrequestreview-343224692
Thanks to @SamirTalwar
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
* Address https://github.com/digital-asset/daml/pull/4046#discussion_r366874330
Co-authored-by: Samir Talwar <samir@noodlesandwich.com>
* ledger-on-sql: Implement a ledger using the simplified kvutils on JDBC.
This tests against H2; PostgreSQL coming soon.
No changelog because we're not shipping this yet.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/app: Extract ledger construction into a trait.
* ledger-on-sql: Run the conformance tests against H2.
* ledger-on-sql: Use H2-specific queries to perform actions atomically.
* ledger-on-sql: Extract queries to `Queries` and `H2Queries`.
* ledger-on-sql: SQLite can't have more than one connection at a time.
* ledger-on-sql: Run conformance tests against SQLite.
* ledger-on-sql: Fix error message if the JDBC URL is missing.
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* kvutils/app: Rename LedgerConstructor to LedgerFactory.
* ledger-on-sql: Don't release connections early.
* ledger-on-sql: Rename `withConnection` so it's more meaningful.
* ledger-on-sql: Use a sequential log entry ID.
* ledger-on-sql: Disable H2 conformance tests.
Until we can figure out why some fail.
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
Fixes#3984.
CHANGELOG_BEGIN
[Sandbox] Fix bug that can cause the transaction stream to not terminate.
See `issue #3984 <https://github.com/digital-asset/daml/issues/3984>`__.
CHANGELOG_END
The bug is that the ledger end gets updated without writing any ledger entries,
for example when uploading packages or in case of duplicate commands.
The termination of the stream (i.e. the comparison of offsets) only happened
on actual ledger entries. If there are none, then the stream just waits for the next
entry to be written.
This change moves the stream termination/completion to BEFORE potential entries
are loaded. The stream knows that after a certain offset has been reached, that it can
be terminated/completed.
* kvutils, etc: Remove duplicate copyright notices.
* ledger-on-posix-filesystem: Implement a simplified KV ledger on FS.
CHANGELOG_BEGIN
- A simple KVUtils-based ledger that runs against a POSIX filesystem.
Intended as a reference implementation for testing and documentation.
CHANGELOG_END
* ledger-on-posix-filesystem: Make it runnable.
* ledger-on-posix-filesystem: Chunk long state paths for macOS.
macOS only supports file names of up to 255 characters.
* ledger-on-posix-filesystem: Run conformance tests.
The lock is temporarily implemented as an in-memory mutex.
* ledger-on-posix-filesystem: Move the paths into the class.
* ledger-on-posix-filesystem: Lock using the file system.
We use the movement of a directory as an atomic operation.
* ledger-on-posix-filesystem: Use a sequential log entry ID.
* ledger-on-posix-filesystem: Check exception types using patterns.
* ledger-on-posix-filesystem: Test that state key collisions can't happen.
* ledger-on-posix-filesystem: Document the various directories.
* ledger-on-posix-filesystem: Use a *persistent* sequential log entry ID.
Because this is a persistent store. Duh.
* ledger-on-posix-filesystem: Don't worry about using Futures.
It's a global lock anyway.
* Factor out common values
* Validate LookupByKey nodes
Fixes#3543
CHANGELOG_BEGIN
- [Sandbox] Validate LookupByKey transaction nodes.
See `issue #3543 <https://github.com/digital-asset/daml/issues/3543>`_.
CHANGELOG_END
* Add failing tests
* LookupByKey validation should not respect privacy
* lookupByKey should use stakeholders, not divulgees
Fixes#2311
CHANGELOG_BEGIN
- [Ledger API] lookupByKey now requires the submitter to be a stakeholder on the referenced contract.
See `issue #2311 <https://github.com/digital-asset/daml/issues/2311>`_.
CHANGELOG_END
* Adapt tests to new behavior
Our users seem to stumble over this problem over and over again. Since there's
no technical reason to ban `submitMustFail` during sandbix initialization, we
can also allow for it.
CHANGELOG_BEGIN
- [Sandbox] Allow ``submitMustFail`` in scenarios used for sandbox initialization.
CHANGELOG_END
