* kvutils: Use `ResourceOwner#use` to simplify tests.
* kvutils: Merge in `RestartSpec` from ledger-on-sql.
CHANGELOG_BEGIN
- [KVUtils] Added a test case to the participant state tests to ensure
your ledger state is resumable upon restart.
CHANGELOG_END
* ledger-on-posix-filesystem: Extract paths into their own class.
* ledger-on-posix-filesystem: Seed the dispatcher with the correct head.
* kvutils: In participant state tests, be clear about different strings.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* sandbox: Move common API and indexer deps into their own packages.
Primarily `com.digitalasset.platform.index.store`.
* sandbox: Move API server deps into the `apiserver` and common packages.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Rename `….platform.index.store` to `….platform.store`.
* sandbox: Split `JdbcIndex` & co. out from `apiserver` into `index`.
Moar packages! And this one is nice and clean; there's only a few
dependencies.
* sandbox: Move `InMemoryPackageStore` into its own `packages` package.
`packages` package. What a tongue twister.
* sandbox: Fix warnings in `TransactionConversion`.
* sandbox: Inline `SandboxContractStore` into `LedgerBackedIndexService`.
* sandbox: Fix warnings in `InMemoryLedger` and `SqlLedger`.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Previously we were only cleaning up and exiting if loading the DAML
packages failed. Now we also handle API server errors.
An example of where this might happen is if the ledger ID specified is
different to the existing ledger ID, causing an error.
One day we'll kill the ResetService and all of this will go away. One
day…
CHANGELOG_BEGIN
- [Sandbox] Fix an error that stops the server from exiting cleanly if
API server initialization fails.
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* kvutils: Use matchers in ParticipantStateIntegrationSpecBase.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: In participant state tests, always test record time.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-posix-filesystem: Run `scalafmt`.
* ledger-on-posix-filesystem: Just use file channel locks. Seriously.
* ledger-on-posix-filesystem: Rename `FileSystemLock#run` to `apply`.
* ledger-on-posix-filesystem: Conformance tests now run quickly.
So we can stop constraining the number of concurrent test runs.
* ledger-on-posix-filesystem: Lock the entire ledger with a file lock.
This ensures we can't have two separate ledger servers running against
the same directory.
This is mostly an experiment to better understand how to do this on an
SQL ledger for #4039.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-posix-filesystem: Run `scalafmt`.
* ledger-on-posix-filesystem: Use `Resource#use` in `FileSystemLock`.
* resources: Move `Resource#use` to `ResourceOwner#use`.
Makes more sense for the same object to govern both `acquire()` and
`release()`.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* resources: Make it easy to use a resource and then release it.
* ledger-on-sql: Move database transaction methods to `Database`.
* ledger-on-sql: Make sure the ledger picks up where it left off.
This means seeding the dispatcher with the correct head, rather than
starting at 1 every time.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Move `Index` and `StartIndex` to the package object.
They're used everywhere.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
The `sandbox-perf` build has been failing for a while with the following
errors:
```
INFO: From Generating benchmark code for //ledger/sandbox-perf:sandbox-perf_codegen:
JMH benchmark generation: JMH Benchmark generator failed
JMH benchmark generation: Benchmark classes should not be final.
[com.digitalasset.platform.sandbox.perf.LargeTransactionBench]
JMH benchmark generation: The instantiated @State class cannot be abstract.
[com.digitalasset.platform.sandbox.perf.PerfBenchState]
```
However, these errors are ignored; running the benchmarks runs the
`AcsBench` benchmark and ignores the fact `LargeTransactionBench` and
`SimpleBench` failed to compile.
I've fixed the errors by making sure that classes are not final, and
that `SimpleBench` uses a concrete state class.
This also introduces a cheap patch to the Scala JMH Bazel rules that
makes sure they fail if there are any errors.
I'm not really sure how to patch the Bazel rules properly, but someone
else might have an idea. :-)
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Unfortunately, downgrading `scalafmt` resulted in files that would be
reformatted to invalid code, because lines would be merged but trailing
commas would be left in.
I've manually run it everywhere and fixed these issues so people don't
have to fix them as they go.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Don't allow unnamed in-memory databases.
It seems that changing the connection pool size at runtime causes weird
issues, predominantly that sometimes all connections are dropped and
therefore the in-memory database is lost after migration. Obviously, it
then stops working.
This works around this problem by simply not allowing unnamed in-memory
databases, guaranteeing that we can share the database between the
various connection pools.
This will probably be necessary anyway when attempting to share the
database between the ledger and the index.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Make it clear we're generating a new JDBC URL in tests.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Pull out a superclass for the various integration tests.
* ledger-on-sql: Use Flyway to migrate tables in an idempotent manner.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Make it easy to run it with Bazel for experimentation.
* ledger-on-sql: Test that migrations will never change in the future.
* ledger-on-sql: Add a prefix of "ledger_" to the tables.
This is so we don't accidentally conflict with the index when the
schemas are shared.
I am letting myself modify the migrations because the existing
migrations haven't been merged into `master` yet.
* ledger-on-sql: Explain why we change the pool size after migration.
* ledger-api-test-tool: Tests now specify a timeout scale, not a timeout.
Makes it easier to change the default timeout in one place.
* ledger-api-test-tool: Increase timeouts on slow tests.
These tests produce a lot of volume and can make CI flaky.
* ledger-on-sql: Only tear down PostgreSQL in tests.
For other databases, we just create a new file for each test case.
* ledger-on-sql: Reduce the log output in tests.
* ledger-on-sql: Use a separate connection pool for Flyway when possible.
Apparently `setMaximumPoolSize` doesn't really have the desired effect
after the connection pool has already been used. The new test case will
be flaky if we process more than one commit in parallel.
For SQLite, it seems to be OK.
* ledger-on-sql: Use a separate connection pool for Flyway with SQLite.
Except in memory.
* ledger-on-sql: Use a separate PostgreSQL database for each test.
Because performance, innit. Don't have to tear them down.
* ledger-on-sql: Move the `Queries` values inside the `RDBMS` objects.
* ledger-on-sql: Go into even more detail about pool size hijinks.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
It affects basically every file, so let's do it all at once.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Three problems here:
1. On failure, the Sandbox didn't close down the metrics reporting
thread, meaning the application would hang.
2. If there's a failure:
a. we didn't print it, and
b. we didn't exit with the correct status code.
These are now handled.
CHANGELOG_BEGIN
- [Sandbox] On initialization error, report the error correctly and exit
with a status code of 1. Previously, the program would hang
indefinitely. (This regression was introduced in v0.13.41.)
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* sandbox: Ensure that the Flyway tests actually pick up the SQL files.
* sandbox: Upgrade Flyway from v5 to v6.
CHANGELOG_BEGIN
- [Sandbox] Upgrade the Flyway database migrations library from v5 to v6.
CHANGELOG_END
* Add logging context information to ledger services
Rolls back that AllowedType implicit, which made the logging context
less usable and didn't provide that much of an advantage. Replaced with
helpers where it makes sense that take care of turning the value safely
into a string.
Closes#3699
CHANGELOG_BEGIN
CHANGELOG_END
* Remove unused import
* Address https://github.com/digital-asset/daml/pull/4205#discussion_r370704306
* ledger-on-sql: Don't bother cleaning up after integration tests.
Turns out Bazel cleans up before each test run, so we only have one
set of outputs at a time. This is far more useful for debugging anyway.
* ledger-on-sql: Pull out a test-lib to simplify the conformance tests.
* ledger-on-sql: Turn `Database` from a sealed trait into a case class.
* ledger-on-sql: Support for PostgreSQL!
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run the conformance tests against PostgreSQL.
* ledger-on-sql: Run the LotsOfParties conformance tests.
* ledger-on-sql: Use PostgreSQL's `RETURNING` keyword to save a query.
* ledger-on-sql: Ensure the reader connection pool is read-only.
* ledger-on-sql: Avoid cloning ByteStrings as byte arrays where possible.
Instead, use `ByteString#newInput()`, which returns an `InputStream`.
This is supported by H2 and PostgreSQL, but unfortunately not SQLite.
* ledger-on-sql: Run integration tests in parallel.
* Update the PostgreSQL and SQLite JDBC drivers.
* postgresql-testing: Extract `PostgresAround` from the Sandbox tests.
CHANGELOG_BEGIN
CHANGELOG_END
* postgresql-testing: Split into several files.
* postgresql-testing: Pull out the host and database names into variables.
* postgresql-testing: Don't publish to Maven.
I have no idea why the build wasn't failing before.
* postgresql-testing: Assimilate the `@postgresql_dev_env` dependency.
* kvutils: Use resources in KeyValueParticipantState, not AutoCloseable.
CHANGELOG_BEGIN
- [kvutils] The simplified API now uses ``com.digitalasset.resources``
to manage acquiring and releasing resources instead of ``Closeable``.
CHANGELOG_END
* ledger-on-posix-filesystem: Remove dead code due to a refactoring fail.
* Support DAML-LF type synonyms in scala.
CHANGELOG_BEGIN
CHANGELOG_END
* dont create synonymns in GenerateSimpleDalf
* extend DAML-LF parser to support type synonyms
* test: expand type synonyms correctly
* ledger-api-test-tool-on-canton: Upgrade Canton to v0.7.0.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-test-tool-on-canton: Use the health endpoint instead of `nc`.
Netty prints a warning when kicking it with `nc -z` from Nix's
netcat-gnu. Not other netcats though… don't know why.
* ledger-api-test-tool-on-canton: Let IntelliJ format the shell script.
* ledger-api-test-tool-on-canton: Stop Canton with SIGINT.
For some reason, Canton doesn't print out a load of errors if it's
stopped with SIGINT (Ctrl+C). SIGTERM, on the other hand…
* ledger-api-test-tool-on-canton: Write Canton logs to STDOUT.
Now the logs are no longer noisy, we can just pump them to STDOUT. This
should make debugging any transient issues on CI a lot easier.
* libs-scala: Make sure `RetryStrategy` works if `attempts.isEmpty`.
* ledger-on-posix-filesystem: Only retry on expected lock errors.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-posix-filesystem: Only create the lock attempt directory once.
Soooo much faster.
* ledger-on-posix-filesystem: Don't run the unit tests on Windows.
* kvutils: Make logback.xml a base file, rather than the only option.
* kvutils/app: Simplify logback.base.xml.
Mostly by getting rid of unnecessary appenders.
* ledger-on-sql: Add trace logging for all database work.
* Upgrade H2; there's a few useful bug fixes.
* kvutils/app: Let the user override the server JDBC URL.
* kvutils/app: Provide a way to specify the ledger ID.
* ledger-on-sql: If there are missing entries in the log, fail on read.
This can happen right now because we insert in parallel. Next step: stop
doing that.
* ledger-on-sql: Stop writing in parallel; it causes race conditions.
This unfortunately means we also stop _reading_ in parallel, which is
less fun.
* Revert "ledger-on-sql: Use a sequential log entry ID."
This reverts commit c58265bf43.
* ledger-on-sql: Create tables with columns that are not nullable.
* ledger-on-sql: The H2 conformance tests now work, just slowly.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run tests against H2 and SQLite on memory and disk.
* ledger-on-sql: Allow H2 to read from the log in parallel with writes.
* ledger-on-sql: Use UUIDs for entry IDs, not 8 random bytes.
* ledger-on-sql: Make sure to log the correlation ID.
* ledger-on-sql: Do less while holding a database connection.
* ledger-on-sql: Log the connection.
* ledger-on-sql: We don't need to tell H2 not to drop the connection.
The connection pool takes care of that.
* ledger-on-sql: Disable H2 conformance tests on CI; they're too slow.
* ledger-on-sql: Rename `loggingContext` to `logCtx`.
* ledger-on-sql: Don't abuse log contexts; put the data in the message.
* ledger-on-sql: Make the connection log line easier to read.
And pull out the logger, even if it's only used once.
* ledger-on-sql: Run the unit tests against (H2, SQLite) * (memory, file).
* engine: the natural order in Transaction is pre-order traversal
- Do not use anymore SortedMap for nodes in transaction
- Drop unused transaction traversal
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lg: improve generator of transactions
* Introduce new JWT payload format
... the reader still supports old formats
CHANGELOG_BEGIN
[Sandbox] The sandbox uses a new payload format for authentication tokens (JWTs).
The old format is deprecated, but still works.
[JSON API] The HTTP JSON API now uses the same payload format for authentication tokens as the sandbox.
The old format is deprecated, but still works.
CHANGELOG_END
* Add helper function for getting token party
* Support sandbox tokens in JSON API
* Add warning for deprecated formats
* Update documentation
* Add explicit test for new format
* Update JSON API documentation
* Fix test
CHANGELOG_BEGIN
[Sandbox] Metrics are now namespaced by "daml" and their names have been
standardize to snake_case
CHANGELOG_END
Please note that HikariJDBC records its own metrics but this is outside
of our reach, so connection-pool-related metrics will keep being called
with camelCase names.
Prohibit contract IDs in contract keys and add key maintainers to exercises
CHANGELOG_BEGIN
- [DAML-LF] Prohibit contract IDs in contract keys completely. Previously, creating keys containing absolute (but not relative) contract IDs was allowed, but `lookupByKey` on such a key would crash.
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
Co-authored-by: Stephen Compall <scompall@nocandysw.com>
This was accidentally set to 1.7 before for no reason. Note that this
simply meant that we did not use the stable module when compiling with
--target=1.6 since we filter out newer stable packages.
Nevertheless, I’ve also added a test that verifies that we do not
accidentally introduce dependencies on packages in newer LF versions.
changelog_begin
changelog_end
* sandbox: Tidy up the RecoveringIndexerSpec a little.
* sandbox: Move the JdbcLedgerDao owner constructor to its object.
* sandbox/damlonx: Make sure the RecoveringIndexer terminates.
Previously, if there was a failure and a termination at the same time,
we could recover after terminating, which means we'd end up in a weird
state. This makes sure we stop correctly.
Unfortunately, I couldn't find a good place to put these tests apart
from in reference-v2, as they need to depend on both _kvutils_ and
_sandbox_. Perhaps they belong in their own Bazel package.
This might address #3945; it's hard to say.
CHANGELOG_BEGIN
- [Indexer] Potentially fix a bug when recovering from failure.
CHANGELOG_END
* reference-v2: Address a race condition in the IndexerIT assertions.
* sandbox: Make sure the RecoveringIndexer can stop during a restart.
* reference-v2: Use Mockito to simulate errors in IndexerIT.
* sandbox: When the RecoveringIndexer is stopped, stop quickly.
* reference-v2: Make sure each test has its own metrics registry.
* Replace TestNamedLoggerFactory with in-memory appender
* sandbox: Test RecoveringIndexerSpec with a mock logger.
Using a real logger was causing massive stack traces in all other tests
that didn't have `RecoveringIndexerSpec$Appender` available, and adding
the appender programmatically didn't work for some reason.
The mock gives much nicer error messages on failure, too.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* libs-scala/resources: Move `Resource` and friends to their own package.
CHANGELOG_BEGIN
- [Ledger API Server] Publish the resource management code as a library
under ``com.digitalasset:resources``.
CHANGELOG_END
* libs-scala/resources: Add to the artifacts list.
* libs-scala/resources: Factor out the test helpers into test/lib.
* libs-scala/resources-akka: Make the Akka resources an optional dep.
* EC256 rather than EC512 algo
CHANGELOG_BEGIN
[Ledger API Authorization]Support EC256 algorithm for JWT rather than EC512
CHANGELOG_END
* support both 256 and 512 ECDSA algo
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* scalafmt
* correct comment, avoid unnecessary private def
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* correct BAZEL formatter
* Introduce ContextualizedLogger and LoggingContext
A ContextualizedLogger is a wrapper around an Slf4j logger. It uses
call-by-name parameters to only construct logging lines when necessary.
The underlying context can be used by invoking withoutContext.
More interestingly, every call to one of its logging methods requires
the presence of an implicit LoggingContext.
A LoggingContext is a way of definining a set of pairs that a
ContextualizedLogger can log without the need of it being specified for
every call.
A new context can be created with newLoggingContext and, given an
implicit LoggingContext, pairs can be added or overwritten using
withEnrichedLoggingContext.
Pairs in the context will be appended to each event logged by a
ContextualizedLogger in the following form:
some text logged explicitly (context: {key1=value1, key2=value2)
If the underlying Logger is configured to use the Logstash encoder the
keys and values in the LoggingContext will also be added to the logging
event encoded in JSON format.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestion in https://github.com/digital-asset/daml/pull/4046#pullrequestreview-343224692
Thanks to @SamirTalwar
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
* Address https://github.com/digital-asset/daml/pull/4046#discussion_r366874330
Co-authored-by: Samir Talwar <samir@noodlesandwich.com>
* ledger-on-sql: Implement a ledger using the simplified kvutils on JDBC.
This tests against H2; PostgreSQL coming soon.
No changelog because we're not shipping this yet.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/app: Extract ledger construction into a trait.
* ledger-on-sql: Run the conformance tests against H2.
* ledger-on-sql: Use H2-specific queries to perform actions atomically.
* ledger-on-sql: Extract queries to `Queries` and `H2Queries`.
* ledger-on-sql: SQLite can't have more than one connection at a time.
* ledger-on-sql: Run conformance tests against SQLite.
* ledger-on-sql: Fix error message if the JDBC URL is missing.
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* kvutils/app: Rename LedgerConstructor to LedgerFactory.
* ledger-on-sql: Don't release connections early.
* ledger-on-sql: Rename `withConnection` so it's more meaningful.
* ledger-on-sql: Use a sequential log entry ID.
* ledger-on-sql: Disable H2 conformance tests.
Until we can figure out why some fail.
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
Fixes#3984.
CHANGELOG_BEGIN
[Sandbox] Fix bug that can cause the transaction stream to not terminate.
See `issue #3984 <https://github.com/digital-asset/daml/issues/3984>`__.
CHANGELOG_END
The bug is that the ledger end gets updated without writing any ledger entries,
for example when uploading packages or in case of duplicate commands.
The termination of the stream (i.e. the comparison of offsets) only happened
on actual ledger entries. If there are none, then the stream just waits for the next
entry to be written.
This change moves the stream termination/completion to BEFORE potential entries
are loaded. The stream knows that after a certain offset has been reached, that it can
be terminated/completed.
* kvutils, etc: Remove duplicate copyright notices.
* ledger-on-posix-filesystem: Implement a simplified KV ledger on FS.
CHANGELOG_BEGIN
- A simple KVUtils-based ledger that runs against a POSIX filesystem.
Intended as a reference implementation for testing and documentation.
CHANGELOG_END
* ledger-on-posix-filesystem: Make it runnable.
* ledger-on-posix-filesystem: Chunk long state paths for macOS.
macOS only supports file names of up to 255 characters.
* ledger-on-posix-filesystem: Run conformance tests.
The lock is temporarily implemented as an in-memory mutex.
* ledger-on-posix-filesystem: Move the paths into the class.
* ledger-on-posix-filesystem: Lock using the file system.
We use the movement of a directory as an atomic operation.
* ledger-on-posix-filesystem: Use a sequential log entry ID.
* ledger-on-posix-filesystem: Check exception types using patterns.
* ledger-on-posix-filesystem: Test that state key collisions can't happen.
* ledger-on-posix-filesystem: Document the various directories.
* ledger-on-posix-filesystem: Use a *persistent* sequential log entry ID.
Because this is a persistent store. Duh.
* ledger-on-posix-filesystem: Don't worry about using Futures.
It's a global lock anyway.
* Factor out common values
* Validate LookupByKey nodes
Fixes#3543
CHANGELOG_BEGIN
- [Sandbox] Validate LookupByKey transaction nodes.
See `issue #3543 <https://github.com/digital-asset/daml/issues/3543>`_.
CHANGELOG_END
* Add failing tests
* LookupByKey validation should not respect privacy
* lookupByKey should use stakeholders, not divulgees
Fixes#2311
CHANGELOG_BEGIN
- [Ledger API] lookupByKey now requires the submitter to be a stakeholder on the referenced contract.
See `issue #2311 <https://github.com/digital-asset/daml/issues/2311>`_.
CHANGELOG_END
* Adapt tests to new behavior
Our users seem to stumble over this problem over and over again. Since there's
no technical reason to ban `submitMustFail` during sandbix initialization, we
can also allow for it.
CHANGELOG_BEGIN
- [Sandbox] Allow ``submitMustFail`` in scenarios used for sandbox initialization.
CHANGELOG_END
* Initial version of simplified transaction reader/writer API.
* Code tidying.
* Ran ./fmt.sh
* Added verification of pre-declared vs. actual state updates to example.
* kvutils: Run `scalafmt`.
* kvutils: Move ParticipantStateIntegrationSpecBase to src/test/lib.
* kvutils: Clean up warnings in ParticipantStateIntegrationSpecBase.
* ledger-on-memory: Move InMemoryLedgerReaderWriter to its own package.
* ledger-on-memory: Add a Main class with a simple configuration.
* reference-v2/ledger-on-memory: Wait for DARs to upload.
* ledger-on-memory: Run conformance tests.
* kvutils/app: Extract the in-memory ledger `Main` class into a Runner.
* reference-v2: Move InMemoryKVParticipantState out of kvutils.
It's not used elsewhere.
* kvutils/app: Fix some of the config help text.
* kvutils/app: Don't register the archive files twice.
CHANGELOG_BEGIN
- [kvutils/app] A simple way to run KVUtils-based participants.
- [ledger-in-memory] Split out the simple in-memory ledger, previously
part of kvutils, into its own JAR.
CHANGELOG_END
* Revert "kvutils/app: Don't register the archive files twice."
This reverts commit 2489ae6964.
Turns out we need this for preloading, which is a useful optimization.
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
CHANGELOG_BEGIN
[Sandbox] Fix the Sandbox command line help text to reflect the recent
change that the default address is ``localhost`` (``127.0.0.1`` or
``::1``)
CHANGELOG_END
This is done to restrict the usage of NamedLoggerFactory and most usages of MetricsRegistry
to a single Bazel package, thus allowing a possible replacement to not have to become a
dependency of multiple packages.
CHANGELOG_BEGIN
CHANGELOG_END
* ECDA512 algorithm support
* ECDA512
* happy day test for ECDA512 algorithm
* failure test for wrong key for ECDA512 algorithm
* add ability to use EC cert file
* update docs
* scalafmt
* Correct documentation
CHANGELOG_BEGIN
[Ledger API Authorization] Support elliptic curve algorithm for JWT verification
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* correct docs warning
Currently there is a constraint on the postgres `ledger_entries` table
ensuring that submitter, application_id and command_id are set on
transactions being persisted. These fields are coming from the submitter
info that will be empty when the transaction accepted event is received
on a participant which is not hosting the submitter.
This change updates the check to ensure that all of these fields are
set OR none of them are.
The same change has been made to the H2 migrations with the addition of
correcting the `is not null` check (was previously using `!= null`).
For postgres, as the check is just loosening the existing constraint of all fields
requiring a value I think there is no risk that the new constraint will fail
to apply to existing data. For H2 as the original constraint was
incorrect there is a possibility that existing data may be incorrect -
however as the same component is responsible for writing to H2 and postgres
I will assume the data was correctly persisted regardless of the constraint.
Plus as H2 is typically only used for ephemeral tests, there is
likely no existing data to migrate.
CHANGELOG_BEGIN
- [Sandbox] Loosens database schema to allow persistence of transaction
ledger entries where no submitter info is present (typically when the
submitter is hosted by another participant node).
CHANGELOG_END
* Do not assume submitter info is present in the message
* Make kvutils conversion utilities public
Required for implementing extensions on top of kvutils.
* Make kvutils.Conversions private to com.daml.ledger.participant.state
* Add test case for allowing optional submitter info
* Add changelog markers
CHANGELOG_BEGIN
CHANGELOG_END
* Move most of the remaining serializable types to stable LF packages
The only serializable types left in DAML stdlib after this PR are the
following:
- DA.Upgrade:MetaEquiv
- DA.Random:Minstd
- DA.Next.Set:Set
- DA.Next.Map:Map
- DA.Generics:MetaSel0
- DA.Generics:MetaData0
- DA.Generics:DecidedStrictness
- DA.Generics:SourceStrictness
- DA.Generics:SourceUnpackedness
- DA.Generics:Associativity
- DA.Generics:Infix0
- DA.Generics:Fixity
- DA.Generics:K1
- DA.Generics:Par1
- DA.Generics:U1
- DA.Internal.Prelude:Optional
Ignoring the Generics stuff which isn’t very urgent imho and the
Upgrade stuff which is probably going to change significantly anyway,
this leaves us with the weird Random module, the wrappers around
TextMap which will go away anyway and DA.Internal.Prelude:Optional
which shouldn’t exist in the first place (I’ll address that in a
separate PR).
CHANGELOG_BEGIN
- [DAML Compiler] Move more types from daml-stdlib to standalone LF
packages. The module names for the types have also changed
slightly. This only matters over the Ledger API when you specify the
module name explicitly. In DAML you should continue to use the
existing module names.
- The types from ``DA.Semigroup` are now in a separate package under
``DA.Semigroup.Types``
- The types from ``DA.Monoid` are now in a separate package under
``DA.Monoid.Types``
- The types from ``DA.Time` are now in a separate package under
``DA.Time.Types``
- The types from ``DA.Validation` are now in a separate package
under ``DA.Validation.Types``
- The types from ``DA.Logic` are now in a separate package under
``DA.Logic.Types``
- The types from `DA.Date` are now in a separate package under
`DA.Date.Types`.
- The `Down` type from `DA.Internal.Prelude` is now in a separate
package under `DA.Internal.Down`.
CHANGELOG_end
* Fix serializability of RelTime
* fix daml-docs
* Fix tests
* Expose address config option in StandaloneApiServer
Before this PR, it was not possible to set the ledger-api-server
address and so it defaulted to localhost. Now, there is a configuration
variable address in the ApiServerConfig.
* Bind to the loopback address by default instead of the public interface.
CHANGELOG_BEGIN
- [daml-on-x-server] Made ledger api server to bind to localhost by
default instead to the public interface for security reasons.
CHANGELOG_END
* Start on daml test-scripts
* Run all `Script a` as test cases
* LedgerClient: Expose PackageManagementClient
To enable DAR uploads
* Upload the DAR to the ledger
* Start sandbox if no ledger specified
* Format daml test-script
* Fix deprecation warning on ActorMaterializer
* Add test-case //daml-script/tests:test_daml_script_test_runner
* Add daml test-script command
CHANGELOG_BEGIN
- [DAML Script - Experimental] Allow running DAML scripts as test-cases.
Executing ``daml test-script --dar mydar.dar`` will execute all
definitions matching the type ``Script a`` as test-cases.
See `#3687 <https://github.com/digital-asset/daml/issues/3687>`__.
CHANGELOG_END
* daml-test-script enable logging
* Remove outdated TODO comment
* daml script-test More elaborate test-caseo
Compare to expected output and add failing test-case
* daml test-script Don't abort on test-failure
Before the test runner would abort on the first failed test-case. This
occasionally introduce additional test-failures if the sandbox was
torn down half-way through execution.
* ./fmt.sh
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
* sandbox: Create a monadic `ResourceOwner` to manage resources.
* sandbox: Rewrite `ResourceOwner` to be async.
* sandbox: Make sure failed resources are closed immediately.
* sandbox: Better naming in `Open`.
* sandbox: Rename `Open` to `Resource`, and open/close to acquire/release.
* sandbox: Convert `() => AutoCloseable` into `ResourceOwner`.
* sandbox: Refactor the LedgerApiServer in terms of resources.
* sandbox: Explicitly convert `() => AutoCloseable` to `ResourceOwner`.
Explicit > Implicit, right?
* sandbox: Create helpers for converting things to ResourceOwners.
Because I tried to start using them and there was so much code being
written at once.
* sandbox: Simplify construction of JdbcLedgerDao.
* sandbox: Releasing resources should be idempotent.
In that we should only do it once.
* sandbox: Fix the ResetService by closing the API services _first_.
They need to be shut down before the gRPC server.
* sandbox: Don't try and shut down PostgreSQL twice in tests.
* sandbox: Actually run the assertions in ResourceOwnerSpec.
Facepalm.
* sandbox: Test `Resource.sequence` more rigorously.
* sandbox: Move the helpers around `Resource` into `Resource.apply`.
* sandbox: Convert LedgerApiServer resource owners to classes.
* sandbox: Make `ResourceOwner` a monad too, delegating to `Resource`.
* sandbox: Turn `LedgerApiServer` into a ResourceOwner.
* sandbox: Simplify the public signature of `Resource.apply`.
* sandbox: Use ResourceOwners to simplify DB resource management.
This is one hell of a change. Sorry.
* sandbox: Try not to nest `Await.result` calls.
Causes issues when running in a `DirectExecutionContext`.
* sandbox: Turn index subscriptions into resources.
* sandbox: Fix warnings in RecoveringIndexerSpec.
* sandbox: Always release before recovering the indexer.
* sandbox: Add `flatten` and `transformWith` to `Resource`.
* sandbox: If releasing twice in parallel, the second should wait.
* sandbox: If the indexer recovers, clean up the old subscription.
* sandbox: Convert StandaloneIndexerServer into a resource owner.
* sandbox: Convert StandaloneApiServer into a resource owner.
* reference-v2: Rewrite ReferenceServer in terms of resources.
CHANGELOG_BEGIN
- [Reference v2] On an exception, shut down everything and crash.
Previously, the server would stay in a half-running state.
CHANGELOG_END
* sandbox: Rewrite SandboxServer in terms of resources.
* sandbox: Write the port file in a Future.
* sandbox: JdbcIndexer no longer needs to manage the actorSystem.
* sandbox: Shut down the LedgerApiServer when closing the Sandbox.
* sandbox: Rename `Resource.pure` to `Resource.successful`.
* sandbox: Rename `Resource.sequence_` to `sequenceIgnoringValues`.
* sandbox: Delete `CloseableResource`.
It's only used in once place. Just inline it.
* sandbox: `LedgerDao` no longer needs to be closeable.
* sandbox: Delete implicit materializers where they're not used.
* http-json: Wait for the Sandbox to start in tests.
* sandbox: Convert `scheduleHeartbeats` into a ResourceOwner.
* reference-v2: Explain why we steal ownership of the actor system.
* sandbox: Document why we only release resources once.
* sandbox: Add clues to ResourceOwnerSpec.
* http-json: Fix HttpServiceTestFixture to pass auth service through.
* codegen-sample-app: In ScalaCodeGenIT, wait for the server to start.
* Moves CommandCompletionIT to Ledger API Test Tool
Closes#2841
Also refactors test observers into a single API and unifies the existing implementations of the DirectExecutionContext
* Allow explicit offsets to get checkpoints
* Raise checkpoint test timeout
* Consolidate checkpoint tests in a single test case
* Fix compilation issues
* Exclude command submission/completion tests for rev-v2
* Drop test not relevant for Ledger API implementations
* Fix compilation errors
Closes#2842Closes#2843
The separation between high-level and low-level is not particularly meaningful, as the low-level variant issues commands with a component used by the high-level variant.
The separation aims at performing integration tests at two different levels but this can be addressed by a single suite of integration tests and relying on unit testing for individual components.
The first two tests:
- 'having many transactions all of them has a unique event id' reads transactions from an empty ledger and doesn't really test anything, so this has been dropped because the uniqueness of event identifiers is already more than covered by many tests in the 'TransactionService' suite.
- 'handle bad Decimals correctly' ensures the ledger api rejects unrepresentable decimals and it have been re-implemented as 'Reject unrepresentable numeric values' in the 'CommandService' suite.
The other tests are about command-and-exercise:
- 'process valid commands with workflow ids successfully' -> 'Implement create-and-exercise correctly'
- 'fail for invalid create arguments' -> 'Fail create-and-exercise on bad create arguments'
- 'fail for invalid choice arguments' -> 'Fail create-and-exercise on bad choice arguments'
- 'fail for an invalid choice' -> 'Fail create-and-exercise on invalid choice'
* Upgrade to Akka 2.6.1, akka-http 10.1.11 and Scala 2.12.10
Akka 2.6.1 Upgrade Changes
- Materializer in place of ActorMaterializer
- Source.future instead of Source.fromFuture
- The Scheduler.schedule method has been deprecated in favor of selecting scheduleWithFixedDelay or scheduleAtFixedRate
- onDownstreamFinish(cause: Throwable)
- ActorAttributes.supervisionStrategy(...) in place of ActorMaterializerSettings.withSupervisionStrategy
See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html
* Akka 2.6.1 Upgrade Changes
- onDownstreamFinish(cause: Throwable)
See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html
* code review: remove unnecessary supervision strategy
* sandbox: Fix warnings in RecoveringIndexerSpec.
* sandbox: Test RecoveringIndexer logs, and stop dumping them to STDERR.
Printing logs during test execution is really annoying. This stubs the
logger factory with a TestNamedLoggerFactory which captures them in a
list instead, for testing purposes.
* sandbox: Remove unnecessary comments from {,Simple}NamedLoggerFactory.
* ledger-api-common: Remove unused parts of the NamedLoggerFactory.
Using the default configuration as a base for this test meant binding on 6865 instead of using dynamic port binding, which could cause flaky errors if some other test is occupying the port
* Move Any wrappers and Archive to stable packages
There are no actual API or functionality changes in this PR but the
logic for locating the stable packages has slightly changed since the
Any wrappers package only makes sense for LF 1.7. To address this, we
simply filter out stable packages for newer LF versions since it
doesn’t make sense to depend on those anyway.
CHANGELOG_BEGIN
- [DAML Compiler] Move ``Archive`` type to a separate DALF.
CHANGELOG_END
* More comments
* Fix java codegen tests
* fix more tests
* Force cache reset on Windows
* Revert "Force cache reset on Windows"
This reverts commit 9f2b7d70b2.
This test suite was introduced in #3870 and seems to be extremely
flaky both locally (I get about 16/20 failures) and on CI.
The error is
- should return completions of commands that are submitted after subscription *** FAILED *** (4 seconds, 945 milliseconds)
Set() did not contain all elements of List("5104", "5100", "5105", "5109", "5103", "5106", "5107", "5102", "5108", "5101") (CommandClientIT.scala:330)
After bumping the timeout, I don’t get any failures locally.
Happy to choose another timeout, I have no clue what a sensible value
is here.
* Port FailingCommandsIT to the Ledger API Test Tool
Closes#2850
The FailingCommandsIT suite contains tests for mismatching ledger
identifier (which is already abundantly covered by the Ledger API Test
Tool) and for bad labels in contract creation, both via submit and
submitAndWait.
Some of the tests previously part of the CommandService Ledger API Test
Tool suite have been moved to a new home in CommandSubmissionCompletion
to reflect the fact that those use the submission/completion workflow
instead of leveraging the submit-and-wait alternatives.
There are also suites that for some reason cover subsequent invocations
of the same services but I'm not sure of the meaning of those and I
would drop them, unless we have a good explanation for those.
* Optimize imports
* Move Scala Ledger API client integration tests
Closes#2845
The Scala Ledger API client tests belong to the
//ledger/ledger-api-client package.
* Reduce noise coming from tests
* sandbox: Move SandboxServer's helper classes to the companion object.
And make them `private`, `final`, and non-`case`.
* sandbox: Drop SandboxServer.apply and just call the constructor.
* sandbox: Move the SandboxServer#SandboxState class into the object.
Leave the `resetAndRestartServer` method behind, though.
* Implementation for the configuration management service
- Add configuration generation to the response of SetTimeModel
- Implement the ConfigManagementService
- Implement integration test into test tool
This is still a draft as it has unsolved FIXMEs and
it conflicts with #3744 which should go first.
The main conflict is with changes to PartyAllocationResponse which cannot
no longer reference "domain.PartyDetail" as we would have a cyclic dependency
(participant-state contains Configuration which we point to from domain.ConfigurationEntry).
The still open issues are:
- Revisit PartyDetail
- Naming: LedgerConfigurationService and ConfigManagementService are not talking
about the same configuration and it feels confusing.
- Remove duplication of ConfigurationEntry? Do we need both domain.ConfigurationEntry
and ledger.store.ConfigurationEntry? Only difference is in the types of participantId
and submissionId.
* Address review part 1
* Fix up tests after rebase and address PR review
* Post-merge fixes
* Add missing config MRT checks and fixes to tests
- Check config MRT in InMemoryLedger and SqlLedger
- Use proper source of time in ConfigManagement
- Separate out ConfigManagementServiceIT in sandbox conformance tsets
* Reformat
Closes#2857
- Also deals with the long ignored todo about not using the context
- Attempts at restoring this as a non-flaky test
- Attempts at running the same test across runtime environments
* sandbox: No point creating a class just for the `close` method.
The LedgerApiServer class doesn't actually have much behavior apart from
closing. The `startServer` method is called during construction, which
definitely counts as spooky action at a distance. This change moves that
behavior to `LedgerApiServer.start` (renamed from `LedgerApiServer
.create` to make it more explicit).
There's not much point creating a class for the rest, so all behavior is
now in that `start` method.
* sandbox: Build up a list of "closeables" as the LedgerApiServer starts.
* sandbox: If the LedgerApiServer fails to come up, stop the parts.
* sandbox: Apply @stefanobaghino-da's refactoring suggestions.
* sandbox: Don't log while building the gRPC server; log at the end.
* sandbox: Turns out we should not stop the server when it works.
Facepalm.
* Ledger api test tool party flakiness for multi-participant runs
Caused by synchronize not waiting for parties to fully exist on
all participants.
* Reenable canton test run
* Update canton to 0.5.2
* Revert "Update canton to 0.5.2"
This reverts commit d0e530c652.
* Add package_entries table
* Change PublicPackageUpload event to cover list of packages.
Add PublicPackageUploadRejected.
* Produce new package update events in KeyValueConsumption
* Update signature of uploadPackages
* Cleanup InMemoryKVParticipantState. Add submissionId to uploadPackages.
* Fix up InMemoryKVParticipantStateIT
* Initial ledger dao changes for package entries
Drop the participant_id as we never expect to see
entries of other participants. This should be done
for party_entries as well.
* Drop UploadPackagesResult
* Implement getPackageEntries and refactor callers
* Add maxRecordTime to uploadPackages
* First cut at updating ApiPackageManagementService
* Update tests, wire through the packageEntries
* Don't extend IndexPackagesService in InMemoryPackageStore
It does not implement the full interface and it isn't used
directly as one anyway.
* Drop maximum_record_time from package_management_service
Adding maximum record times touches the whole stack. Leaving
this change to another PR.
* Wire through the removal of maximum_record_time.
And remove dead code from InMemoryKVParticipantState
* Remove checking for duplicate package uploads
This aligns with the behaviour of WriteService.
* Reformat
* Fix PackageManagementService after adding of submission_id to the service
CHANGELOG_BEGIN
- [Sandbox] Restore 0.13.38 logging behaviour.
- [Navigator] Restore 0.13.38 logging behaviour.
- [Extractor] Restore 0.13.38 logging behaviour.
- [Internals] As of 0.13.39, we merged a number of internal JAR files in
the SDK tarball to reduce its size. These jars used to be standalone
JARs you could invoke as e.g. ``java -jar sandbox.jar <args>``. As a
result of merging the jars, they lost their individual ``logback.xml``
configuration file. Although running the jars directly was (and is
still) not supported, note that you can now achieve the same behaviour
with e.g. ``java -Dlogback.configurationFile=sandbox-logback.xml -jar
daml-sdk.jar sandbox <args>``.
CHANGELOG_END
* sandbox/reference-v2: Split `Config` into three.
- Reference v2 CLI configuration
- Standalone API server configuration
- Standalone indexer configuration
* reference-v2: Simplify CLI parsing of TLS config and extra participants.
Turns out we _always_ want to support extra participants. They're
optional, after all.
* Revert "Remove docker bazel target for sandbox (#3802)"
This reverts commit 48a381ca49.
* sandbox: Use the public `openjdk` Docker image rather than GCR's.
Turns out you need to authenticate with Google Container Registry
regardless of whether the image is public or not. We don't want people
to have to bother logging in to GCR just to get Bazel working.
* sandbox: Split the StandaloneApiServer from the StandaloneIndexerServer.
* sandbox: Move StandaloneApiServer's nested classes to its companion.
And make them private.
* reference-v2: Rename `IndexServer` to `ApiServer`.
* sandbox: Move the services into the `apiserver.services` package.
* sandbox: Move SandboxEventIdFormatter's vals to the top.
* sandbox: StandaloneApiServer helper classes don't need to be Products.
* docs: Fix links for LedgerApiServer and Standalone{Api,Indexer}Server.
* sandbox: Delete a misleading comment from `StandaloneApiServer`.
* sandbox: Rename SandboxEventIdFormatter to EventIdFormatter.
* Move all datatypes out of daml-prim
This moves the remaining two modules DA.Types and GHC.Tuple to
separate LF packages with stable identifiers.
The only data types remaining are the ones for typeclasses which will
disappear once we move this to type synonyms.
CHANGELOG_BEGIN
- [DAML Compiler] The modules DA.Types and GHC.Tuple from daml-prim
have been moved to separate packages.
CHANGELOG_END
* Fix codegen tests
* Fix DarReader test
* Fix kvutils tests
* Fix jdbcdao tests
* Fix hs ledger bindings tests
* Add ledger and participant ID to claims
CHANGELOG_BEGIN
- [Ledger] AuthService implementations can now restrict the validity of access tokens to a single ledger or participant.
- [Sandbox] The sandbox JWT authentication now respects the ledgerId and participantId fields of the token payload.
CHANGELOG_END
* Add tests for ledger and participant in claims
* Address review comment
* Address review comment
* Fix tests
* Fix tests
* ledger-api-scala-logging: Fix errors in IntelliJ IDEA.
The Bazel plugin for IntelliJ doesn't seem to be smart enough to be able
to handle a Scala library that is part `src` directory and part
generated code from another Bazel rule. It just ignores the second part.
This means that IntelliJ cannot find the *ServiceLogging classes, as
they're not represented on the Bazel-generated classpath, and so
complains with lots of errors when working on the equivalent Api*Service
files.
To fix this, we can split these in two, compiling the base traits to
`ledger-api-scala-logging-base` and then the generated code separately.
It does result in an extra Bazel dependency for the users of
ledger-api-scala-logging, as Bazel doesn't realise transitive
dependencies for us.
* Release: Add `ledger-api-scala-logging-base` to the Maven list.
* ledger: Document the health checks.
* sandbox: Build a Docker image.
* sandbox: Create a sample Kubernetes YAML file.
* sandbox: Add health probes to the sample Kubernetes configuration file.
Startup and liveness are tested with a simple TCP connection to port
6865. Readiness checks are done with `grpc-health-probe`, which is added
to the Sandbox container image.
* sandbox: Link to kubernetes.yaml in the README and provide a disclaimer.
// changelog removed as it's not actually relevant to users
* sandbox: Don't try and build `sandbox-image-base` on Windows.
* Apply suggestions from code review
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: HikariJdbcConnectionProvider.start(), rather than construction.
It's too easy to construct things; I'd like it to be obvious that we're
starting something (in this case a connection pool).
* sandbox: In SqlLedgerSpec, stop ledgers before shutting down PostgreSQL.
* sandbox: Perform JDBC health checks on a timer.
The problem with hooking into existing requests is that if the Sandbox
is running in a load balancer and reports itself as unhealthy, it might
be taken out of the load balancer. Without any requests going through,
it'll never realize it's healthy again, and so will remain thinking it's
unhealthy forever.
* sandbox: Inline SqlExecutor into DbDispatcher.
There's no clear delineation of responsibilities. I'll try and separate
some of this back out again later.
* sandbox: Make it clear that we're "starting" the DbDispatcher.
* sandbox: Simplify the promise-based behavior in DbDispatcher.
* sandbox: Rename `noOfShortLivedConnections` to `maxConnections`.
The timeout for witnessing the party allocation is now increased
to 30 seconds (from 10 seconds). Additionally the LotsOfParties
conformance test is marked as flaky.
* Sandbox: Remove streaming connections
The separate database connection pool for streaming connections
was only used for the active contracts stream. However, a single
db connection was being occupied until the last active contract was
streamed over the Ledger API to the client. This effectively means
that only ever 2 concurrent active contract streams could exist.
No need to say that this is bad design.
The following changes happened:
- remove the db connection pool for streaming connections
- replace the streaming mechanism for active contracts with
the already existing pagination mechanism in JdbcLedgerDao
- change the pagination mechanism to actually use database level
limit and offset instead of doing the pagination "client side"
- configure the HikariDataSource with the metric registry
CHANGELOG_BEGIN
- [Sandbox] Improve loading of active contracts for the Sandbox SQL backend.
CHANGELOG_END
* Extract PaginatingAsyncStream from JdbcLedgerDao for testing
* Reset metrics registry before each test
* sandbox: If the Flyway migrations fail, crash the process.
Otherwise running it in the background can make it look like
everything's OK.
* reference-v2: If the Flyway migrations fail, crash the process.
Otherwise running it in the background can make it look like
everything's OK.
* sandbox: Remove an errant debugging `println`.
* sandbox: Use `DirectExecutionContext.implicitEC`.
* convert party allocation to the new committer pattern
* scalafmt
* Use map with optional values for kvutils input state
We need to do this to detect when a submission has maliciously
omitted an input key.
This is a first step towards making sure that the package ids for
types defined in daml-prim and daml-stdlib don’t change. This PR
mostly adds all the necessary infrastructure for that and moves
GHC.Types and GHC.Prim to make sure it works.
Until data-dependencies are really solid and we have verified that we
no longer have performance issues with an increasing number of Haskell
packages, we still include the source files in daml-prim and then just
rewrite the references.
We will also need to add tests that these packages really have stable
ids but I’ll leave that for separate PRs since this doesn’t make that
much sense anyway until all of the types have moved to stable
packages.
CHANGELOG_BEGIN
- [DAML Compiler] The modules GHC.Prim and GHC.Types from daml-prim
have been moved to separate packages.
CHANGELOG_END
* Track used packages during whole of engine submit
- Introduce MutableCompiledPackages interface
- Add TrackingCompiledPackages that tracks fetched packages
- Make used packages in transaction optional to distinguish between
missing dependencies and empty set of dependencies.
* Reimplement package dependency tracking
- Compute direct dependencies of a package during decoding
- Compute transitive dependencies of a package when adding a package
to engine.
- Annotate the resulting transaction with package dependencies
in Engine.submit.
* Create Ast.Package with proper direct deps in scenario service
While we don't have use for direct dependencies of a package in
scenario service (only Engine.submit needs it), it's better to be
accurate.
This of course overapproximates the direct dependencies.
* Compile a each new package once in ConcurrentCompiledPackages
* Revert "ledger-api-test-tool: The endpoints are mandatory. (#3611)"
This reverts commit 603ee9367b.
* Restore LedgerTestContext requirement
* Exit early if there are no participants to test
* ledger-api-test-tool: Increase the duration when watching health.
This should hopefully stop CI from flaking out.
* reference-v2/sandbox: Avoid unnecessary companion object constructors.
I like indirection… when it does something.
* ledger: Propagate empty health checks throughout the services.
* reference: Remove duplication from the ReferenceServer object.
* ledger-api-common: Actually query a "reporter" in the health service.
* ledger-api-common: Report health per-component when required.
* ledger-api-health: Use a Map to represent components for health checks.
* sandbox: Fix warnings in SqlLedgerSpec.
* ledger-api-common: Throw GrpcHealthService errors inside the Future.
* ledger: Implement health checks against the PostgreSQL connection.
Without proper testing, because I am not great at this.
* sandbox: Remove duplication and fix warnings in PostgresAround.
* sandbox: Test the SQL Ledger's health reporting on failure.
* sandbox: Don't report as unhealthy until 3 connections fail.
* ledger-api-health: Remove unused parts of the API.
Bit of premature design there.
* sandbox: Rename the "ledger" health check to "write".
* participant-state: Add the ReportsHealth trait to ReadService.
* ledger-api-common: `Future.fromTry(Try(…))` -> `Future(…)`.
* ledger-api-common: Make it clearer that StubReporter closes over health.
* ledger-api-common: Explain the HealthService watch tests with comments.
* sandbox: Clean up SqlLedger a bit.
* sandbox: Don't try and stop PostgreSQL twice in PostgresAround.
* bazel_tools: Windows rlocation lookups need to be with forward slashes.
* release: Fix case of "true".
* ledger-api-common: Make `GrpcHealthService::matchResponse` return a Try.
* ledger-api-common: Make `GrpcHealthServiceSpec` async.
* sandbox: Make a couple of DB classes final.
* sandbox: Avoid importing `X._` in PostgresAround.
* sandbox: Add clues to the SqlLedgerSpec's multiple assertions.
* sandbox: If PostgreSQL doesn't come back up, keep retrying.
* sandbox: Remove duplication in SqlLedgerSpec.
* sandbox: In SqlLedgerSpec, actually wait for the health to change.
* sandbox: In PostgresAround, make stopping PostgreSQL idempotent.
* sandbox: Simplify the SqlLedgerSpec to make it work on CI.
It's worth a shot.
* ledger-api-common: Simplify the GrpcHealthServiceSpec a little.
And add a changelog.
CHANGELOG_BEGIN
- [Ledger API Server] Add a health check endpoint conforming to the
`GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
- [Ledger API Server] Add health checks for index database connectivity.
- [Participant State API] Add a mandatory ``currentHealth()`` method to
``IndexService``, ``ReadService`` and ``WriteService``.
CHANGELOG_END
* sandbox: Improve the Javadoc layout for DbDispatcher.
* sandbox: Capitalize constants in SqlExecutor.
* ledger-api-health: Convert HealthStatus to an abstract class.
* Add participant-state configuration protobuf
- Move the "DamlConfiguration" from kvutils into participant-state/protobuf/ledger_configuration.proto.
- Add version number and spec (ala transaction.proto)
This is a preparation for indexing the configuration and having one canonical serialization for it.
* Initial thoughts on indexing the configuration
* Implement indexing of ledger configuration changes
* Add record time to all Updates. Wire through participant id.
and rename V7__Add_configuration to V8.
* Add ledger_configuration_java_proto to release artifacts
* Fix up release of ledger_configuration_java_proto
* Suggestions from review
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Apply suggestions from code review
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* address rebase issues
* Resolve compilation errors after rebase/merge
* happy formatting for scala and bazel and friends
* Drop "openWorld" setting from configuration
And refactor the tests to allocate parties explicitly.
* Fix up migration and tests
* Drop authorizedParticipantIds from configuration
Implement configuration authorization in kvutils using the previous
the participant id of the previous configuration.
* Post-rebase fixes
* Add missing migrations
* Apply suggestions from code review
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Add missing mavenUpload to ledger-api-server
* Remove stateUpdateRecordTime
* Address code review
- Address PR review
- Merge TimeModelImpl and the traits. Remove TimeModel from ledger-api-common.
- Throw `Err` from KeyValueConsumption on parse errors instead of assert/sys.error
* Reformat
* Add missing protobuf file
* Fix compilation after TimeModel changes. Add version logs to participant-state{,-kvutils}.
* Fix TestUtil.scala build
* Apply suggestions from code review
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Address review
- synchronized access to ledgerConfiguration in InMemoryLedger
- store rejection if configuration generation is wrong
* Update ledger/participant-state/protobuf/ledger_configuration.rst
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Add authentication to Java identity client
Contributes to #3626
No changelog added right now, it will be added with the PR that
effectively closes the ticket.
Moves `LedgerCallCredentials` to a shared package.
Establishes the test framework for other Java clients.
* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731
* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731
* Rebase and fix compilation errors
* Fix type of jar in artifacts.yaml
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Fix more errors emerged after rebase
* Add dependency for DAML assistant ITs
* Add missing dependency
* Fix wrong dependency
* fix release
* Move AuthorizationIT to Sandbox ITs
* Use refactored code in Extractor
* Apply a few improvements, see body
- Run previously ignored tests
- Make token signing private
- Fix stream observer utility comment more accurate
- Add read-only token helper
* Add r/w token split tests
* Fix formatting and imports
* Address https://github.com/digital-asset/daml/pull/3598#discussion_r349769546
* Address https://github.com/digital-asset/daml/pull/3598#discussion_r349769801
* Fix inaccurate test
A test was incorrectly trying to get a token with an invalid signature
* Address reviews
* kvutils: Nest all InMemoryKVParticipantState tests inside the block.
* kvutils: Simplify InMemoryKVParticipantStateIT where possible.
* kvutils: In InMemoryKVParticipantStateIT, move assertions to the end.
It's confusing when the assertions are before the submissions.
* kvutils: Move test helpers into a companion object.
* kvutils: Use BeforeAndAfterEach in testing to remove duplication.
* kuvtils: In testing config updates, always increment the generation.
* kvutils: Assert something in the "initial conditions" integration test.
* ledger-api-test-tool: Use the default Akka execution context.
Akka will use the global fork-join pool by default, which is fine for
our purposes. There's no longer a need to use a specific execution
context, because we no longer use it to limit the work in progress.
That's now done by the parallelism parameter to Akka Streams'
`mapAsyncUnordered`.
* ledger-api-test-tool: Shut down the ActorSystem.
* Improve Navigator output
* Fix Navigator not using the access token
* Add RSA signatures for JWT tokens
* Remove unused method
* Add timeouts for reading JWKS
* Fix test
* Rename method for consistency
* Improve comment
* More renaming for consistency
* CHANGELOG
CHANGELOG_BEGIN
- [Sandbox] Add CLI options to start the sandbox with JWT based authentication with RSA signed tokens.
See `issue #3155 <https://github.com/digital-asset/daml/issues/3155>`__ .
- [Navigator] Fixed a bug where the `--access-token-file` option did not work correctly.
CHANGELOG_END
* Make JwksVerifier limits configurable
* Make SimpleHttpServer private
The shared-secret-based authentication is there exclusively for testing,
this PR makes sure that this is correctly reported through the CLI and
hides this until further notice.
Makes sure that the flag clearly says that this option is not meant for
production use cases.
A warning is already printed when this feature is used, both at startup
and every time this verifier is used.
CHANGELOG_BEGIN
- [Sandbox] The ``--auth-jwt-hs256`` is renamed to
``--auth-jwt-hs256-unsafe``: you are advised to _not_ use this JWT token
signing way in a production environment.
CHANGELOG_END
Closes#3327
CHANGELOG_BEGIN
[Ledger]
* Allow non-alphanumeric characters in ledger api server participant ids (space, colon, hash, slash, dot). Proper fix for change originally attempted in v0.13.36. See issue `issue #3327 <https://github.com/digital-asset/daml/issues/3327>`__.
CHANGELOG_END
Proper fix replacing f39bf7fc5a which does not work
* grpc-utils: Simpler GrpcStatus extractors when you know the code.
And don't care about the description.
* grpc-utils: Simpler GrpcException extractors when you know the code.
And don't care about the description.
* Apply suggestions from code review
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Add `new` to `SpecificGrpc{Exception,Status}` constructor calls.
Also, don't blindly apply GitHub suggestions.
* grpc-definitions: Delete health_service.proto
We can use the version in io.grpc:grpc-services instead.
* ledger: Delete ledger/API.md.
* sandbox: Fix warnings in ApiServices flagged by IntelliJ.
* sandbox: Implement a dummy grpc.health.v1.Health.Check endpoint.
* sandbox: Implement a dummy grpc.health.v1.Health.Watch endpoint.
* sandbox: Drop repeated elements from grpc.health.v1.Health.Watch.
* sandbox: Wrap the HealthService in basic tests.
* sandbox: Stop streaming the server health too.
* ledger-api-test-tool: Health check tests.
* Add a changelog entry for the health check endpoints.
CHANGELOG_BEGIN
- [Ledger API] Add healthcheck endpoints, conforming to the
`GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
It is always ``SERVING`` for now.
- [DAML Ledger Integration Kit] Add conformance test coverage for the
``grpc.health.v1.Health`` service.
CHANGELOG_END
* ledger-api-integration-tests: Increment the number of services.
* Apply suggestions from code review
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: Use `AkkaBeforeAndAfterAll` in the HealthServiceSpec.
In an attempt to get it working on CI.
* sandbox: Change `dropRepeated` to `DropRepeated()`.
Keep it in one file.
* test-common: Use `Delayed.by` in `TimeBoundObserver`.
* test-common: Close the source when `TimeBoundObserver` completes.
* ./fmt.sh
That'll teach me not to `--no-verify` just because it's a merge commit.
* sandbox: Inline `HealthService.suppress`.
At some point it was being used twice.
* sandbox: Increase the timeout for HealthServiceSpec.
* sandbox: Reimplement HealthService using the Scala protobuf types.
* sandbox: Generate an Akka-compatible trait for the health service.
And refactor a lot of test code to make it easy to test.
* ledger-api-common: Move the HealthService here.
* rs-grpc-testing-utils: Publish to Maven.
* rs-grpc-testing-utils: Add Maven coordinates.
CHANGELOG_BEGIN
[DAML Integration Toolkit]
* Add ledger api test tool `--load-scale-factor` option that allows dialing up
or down the workload applied by scale tests (such as the
`TransactionScaleIT` suite). This allows improving the performance of
different ledger over time.
CHANGELOG_END
* Split read and write claims
Fixes#3398
CHANGELOG_BEGIN
- [Sandbox] Added support JWT tokens that only authorize to read data, but not to act on the ledger.
CHANGELOG_END
Instead of passing around MetricsManager just for the timedFuture
method, we pass the underlying MetricRegistry directly and use
the timedFuture function as an actual function (passing in a timer metric).
The Sandbox still starts the JmxReporter, but the integrator of the Indexer
and the Ledger API Server needs to enable the reporting facilities themselves.
CHANGELOG_BEGIN
- [Ledger] Ledger API Server and Indexer now accept an instance of ``MetricRegistry`` as parameters. This gives implementors of ledger integrations the most flexibility to set up metrics reporting that works best for them.
CHANGELOG_END
