* grpc-definitions: Delete health_service.proto
We can use the version in io.grpc:grpc-services instead.
* ledger: Delete ledger/API.md.
* sandbox: Fix warnings in ApiServices flagged by IntelliJ.
* sandbox: Implement a dummy grpc.health.v1.Health.Check endpoint.
* sandbox: Implement a dummy grpc.health.v1.Health.Watch endpoint.
* sandbox: Drop repeated elements from grpc.health.v1.Health.Watch.
* sandbox: Wrap the HealthService in basic tests.
* sandbox: Stop streaming the server health too.
* ledger-api-test-tool: Health check tests.
* Add a changelog entry for the health check endpoints.
CHANGELOG_BEGIN
- [Ledger API] Add healthcheck endpoints, conforming to the
`GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
It is always ``SERVING`` for now.
- [DAML Ledger Integration Kit] Add conformance test coverage for the
``grpc.health.v1.Health`` service.
CHANGELOG_END
* ledger-api-integration-tests: Increment the number of services.
* Apply suggestions from code review
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: Use `AkkaBeforeAndAfterAll` in the HealthServiceSpec.
In an attempt to get it working on CI.
* sandbox: Change `dropRepeated` to `DropRepeated()`.
Keep it in one file.
* test-common: Use `Delayed.by` in `TimeBoundObserver`.
* test-common: Close the source when `TimeBoundObserver` completes.
* ./fmt.sh
That'll teach me not to `--no-verify` just because it's a merge commit.
* sandbox: Inline `HealthService.suppress`.
At some point it was being used twice.
* sandbox: Increase the timeout for HealthServiceSpec.
* sandbox: Reimplement HealthService using the Scala protobuf types.
* sandbox: Generate an Akka-compatible trait for the health service.
And refactor a lot of test code to make it easy to test.
* ledger-api-common: Move the HealthService here.
* rs-grpc-testing-utils: Publish to Maven.
* rs-grpc-testing-utils: Add Maven coordinates.
CHANGELOG_BEGIN
[DAML Integration Toolkit]
* Add ledger api test tool `--load-scale-factor` option that allows dialing up
or down the workload applied by scale tests (such as the
`TransactionScaleIT` suite). This allows improving the performance of
different ledger over time.
CHANGELOG_END
* Split read and write claims
Fixes#3398
CHANGELOG_BEGIN
- [Sandbox] Added support JWT tokens that only authorize to read data, but not to act on the ledger.
CHANGELOG_END
Instead of passing around MetricsManager just for the timedFuture
method, we pass the underlying MetricRegistry directly and use
the timedFuture function as an actual function (passing in a timer metric).
The Sandbox still starts the JmxReporter, but the integrator of the Indexer
and the Ledger API Server needs to enable the reporting facilities themselves.
CHANGELOG_BEGIN
- [Ledger] Ledger API Server and Indexer now accept an instance of ``MetricRegistry`` as parameters. This gives implementors of ledger integrations the most flexibility to set up metrics reporting that works best for them.
CHANGELOG_END
CHANGELOG_BEGIN
- [DAML Ledger Integration Kit] The Ledger API Test Tool no longer shows
individual test duration colored based on how they lasted.
CHANGELOG_END
* Split out ledger api test tool transaction tests related to scale
This enables ledger providers to meet conformance targets first before
tackling the scale / large transactions tests.
* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/tests/TransactionScale.scala
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/tests/TransactionScale.scala
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
Fixes#3513.
* Add metrics to track commands
- Meter: CommandSubmission.failedCommandInterpretations
- Timer: CommandSubmission.submittedTransactions
* Add metrics to JdbcIndexer
- Gauge: JdbcIndexer.lastReceivedRecordTime
- Gauge: JdbcIndexer.lastReceivedOffset
- Gauge: JdbcIndexer.currentRecordTimeLag
- Timer: JdbcIndexer.processedStateUpdates
* Allow setting a custom JMX domain.
We don't want to use com.digitalasset.platform.sandbox
for the ledger-api server and indexer.
* Add interceptor that automatically meters incoming grpc service calls.
CHANGELOG_BEGIN
- [Ledger] Add various metrics to track gRPC requests, command submissions, and state update processing. See `#3513 <https://github.com/digital-asset/daml/issues/3513>`__.
CHANGELOG_END
* add error codes for overloaded and internal error
* change wording in error description strings
* scala formatting
* use string interpolation intead of concatenation
* address issues found in peer review
- moved comparableTransaction[tree]s helpers to companion object
as suggested in https://github.com/digital-asset/daml/pull/3475#pullrequestreview-317474464
- applied helpers to two additional tests
- strip offset in addition to command-ids as offsets drift apart over time
depending on differing heartbeats and different sets of hosted party and thus different
party-relevant events
- modified two tests to be single-node as look-up by transaction/eventid assumes the events
are available locally
* Add InsertOrdSet
A set with deterministic ordering.
* Add party to participant mappings as inputs to kvutils submissions
* Add InsertOrdSetTest, remove fromSet. Sort packages and parties.
* Apply suggestions from code review
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Add AbstractSet and SetLike traits to InsertOrdSet. Remove (Non)EmptyInsertOrdSet.
* Add helper to produce an empty zip files.
This is used to generate empty sources and javadoc jars for
deploy jars later on.
* Create empty auxilliary jars.
da_java_binary:
- empty javadoc jar
- empty sources jar
da_java_proto_library:
- empty javadoc jar
- the sources jar is automatically generated by java_proto_library as a side effect
da_scala_binary:
- empty javadoc jar
- empty sources jar
* Support maven upload for jar-deploy and jar-proto
For jar-deploy targets we don't check for internal dependencies,
because these should already be contained in the (fat-)jar itself.
Additionally, the release program now uploads javadocs and sources
for jar-proto and jar-deploy as well to comply with maven central.
* Upload ledger-api-test-tool and kvutils + dependencies to maven central.
This is the diff running the output of the release without and with these changes.
A few artifacts now also get their javadoc and sources uploaded (mostly to bintray,
but now they are ready for a maven central upload).
ledger-api-test-tool has the scala version removed from the artifact as it is a
deploy jar and nobody should care which specific scala version is used.
Only in release/com/daml/ledger/participant-state-kvutils-java-proto/100.13.35: participant-state-kvutils-java-proto-100.13.35-javadoc.jar
Only in release/com/daml/ledger/participant-state-kvutils-java-proto/100.13.35: participant-state-kvutils-java-proto-100.13.35-sources.jar
Only in release/com/daml/ledger/testtool: ledger-api-test-tool
Only in release-before/com/daml/ledger/testtool: ledger-api-test-tool_2.12
Only in release/com/digitalasset/daml/lf/engine/trigger/runner_2.12/100.13.35: runner_2.12-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml/lf/engine/trigger/runner_2.12/100.13.35: runner_2.12-100.13.35-sources.jar
Only in release/com/digitalasset/daml-lf-blindinginfo-java-proto/100.13.35: daml-lf-blindinginfo-java-proto-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml-lf-blindinginfo-java-proto/100.13.35: daml-lf-blindinginfo-java-proto-100.13.35-sources.jar
Only in release/com/digitalasset/daml-lf-transaction-java-proto/100.13.35: daml-lf-transaction-java-proto-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml-lf-transaction-java-proto/100.13.35: daml-lf-transaction-java-proto-100.13.35-sources.jar
Only in release/com/digitalasset/daml-lf-value-java-proto/100.13.35: daml-lf-value-java-proto-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml-lf-value-java-proto/100.13.35: daml-lf-value-java-proto-100.13.35-sources.jar
Only in release/com/digitalasset/damlc/100.13.35: damlc-100.13.35-javadoc.jar
Only in release/com/digitalasset/damlc/100.13.35: damlc-100.13.35-sources.jar
Only in release/com/digitalasset/extractor/100.13.35: extractor-100.13.35-javadoc.jar
Only in release/com/digitalasset/extractor/100.13.35: extractor-100.13.35-sources.jar
Only in release/com/digitalasset/ledger-service/http-json-deploy/100.13.35: http-json-deploy-100.13.35-javadoc.jar
Only in release/com/digitalasset/ledger-service/http-json-deploy/100.13.35: http-json-deploy-100.13.35-sources.jar
Only in release/com/digitalasset/navigator/100.13.35: navigator-100.13.35-javadoc.jar
Only in release/com/digitalasset/navigator/100.13.35: navigator-100.13.35-sources.jar
* ledger-api-test-tool-on-canton: Upgrade Canton to v0.4.0.
* workspace: Expose a Java library from the Canton external dependency.`
* workspace: Use double quotes for multiline strings too.
* Don't duplicate dependencies on `@postgresql_dev_env` binaries.
If you depend on `@postgresql_dev_env//:all`, you don't need anything
else.
* ledger-api-test-tool: Remove PostgreSQL dep from conformance tests.
It's already a dependency of the server.
* Check internal dependencies of maven artifacts
All artifacts that are uploaded to maven need to also have
their dependencies uploaded to maven.
* inline tuple matching
* remove unnecessary parenthesis
* mark missing artifacts for maven upload
* ledger-api-test-tool-on-canton: Run Canton.
* ledger-api-test-tool-on-canton: Run the SemanticTests against Canton.
This was _so_ much work.
These will probably be flaky, so can't merge them in until we fix the
underlying issues with the tests around multi-participant allocation.
* ledger-api-test-tool: Wait for parties to arrive on all participants.
Thanks to Canton for exacerbating this bug.
Can't turn on all the tests on Canton yet as there are other spurious
failures. Will investigate next.
* ledger-api-test-tool: Move all contract key tests to ContractKeys.
If a ledger doesn't support ContractKeys, they need to be able to turn
these tests off.
* ledger-api-test-tool: If a test name is misspelled, fail immediately.
I keep getting names slightly wrong in the `--exclude` arguments and
wondering why it didn't work.
* client_server: Revert client_server_test.bzl.
Turns out I don't know what I'm doing. ¯\_(ツ)_/¯
* sandbox: Don't call `.toString` on an array. Doesn't do much.
* dev-env: Don't untar netcat automatically.
It needs to be installed with Pacman.
* Use TreeMap for storing transaction nodes
Going forward we require deterministic serialization for transaction nodes.
This PR switches the `Transaction.nodes` to use a TreeMap instead of HashMap,
which gives us deterministic serialization order for the transaction nodes.
An alternative would be to sort the transaction nodes when serializing, but
I feel avoiding HashMap altogether is the safer option as this may bite us
elsewhere as well (e.g. tests, new serialization code, etc.).
* Address review
* Remove unused import
The contract keys accessed in the context of a `fetchByKey` were not
properly checking that the contract was active and visible.
The checks for contract's visibility and activeness are now done from
one place.
* address warning: add type annotations, shortened syntax, optimise imports
* Specify test size to avoid bazel warning
address warning: add type annotation
* make bazel buildifier happier with call order
* no parens for a no side effect pure method
Instead of using the value of the state at the time of subscription,
we need to call getHead explicitly.
Otherwise it could happen that the state gets updated between reading it
and completing the subscription, which means the stream would miss an update
of head.
This also removes the type parameter of SignalDispatcher again, as it is best
that this class doesn't even know the data type involved.
Two tests issued a create and exercise on different participants without
allowing for a delay publishing the contract. Added two missing eventually
statements to two TransactionService tests.
* ledger-api-test-tool: Allocate participants and parties declaratively.
Part of a solution to #3380.
Rather than allocating participants and parties line by line, instruct
the test to construct a number up-front.
This will make it easier to serve multi-participant ledgers, as we can
make sure we wait for all parties to be available on all participants
before starting the test. Once we can do that, tests will hopefully be
much less flaky on ledgers other than the Sandbox.
* ledger-api-test-tool: Rename `provision` to `allocate` and document it.
* ledger-api-test-tool: Make Allocation functions private where possible.
* ledger-api-test-tool: Fix the comment for `LedgerTestContext::allocate`.
It got the signature wrong.
* ledger-api-test-tool: Just use `Duration::toString` for logging.
* ledger-api-test-tool: Fix reporter highlighting.
Previously, the closing parenthesis around the test duration was not
highlighted. Now neither are.
Closes#2844.
It turns out that this test doesn't really exacerbate any backpressure;
it just fires lots of commands. In the interest of not losing this, I've
ported it to a test in TransactionService.
Without the throttling, we _do_ hit resource exhaustion on Sandbox with
the PostgreSQL server backend. Capping the number of commands at 500,
rather than 1000, seems to solve this.
* Implement proper stream pagination
The previous pagination mechanism for streaming ledger entries was implemented
as a recursive method call with manually concatenating akka Sources.
However, this didn't work properly and resulted in all the subsources being
forced immediately, resulting in parallel requests for all pages
(0-100, 100-200, 200-300, ...) instead of the expected mechanism of loading
the first page and on upstream request (i.e. client requests for more data
over grpc) loading the next page of data.
The current mechanism uses Source.unfoldAsync which makes the paging
mechanism work as expected: sequential loading of pages on demand.
* Serialize and deserialize Transactions outside the SQL Executor
This frees up the SQL Executor threads sooner for other work
and the conversion work only happens on a by need basis when
the consumer requests more data.
