* sandbox: Split the StandaloneApiServer from the StandaloneIndexerServer.
* sandbox: Move StandaloneApiServer's nested classes to its companion.
And make them private.
* reference-v2: Rename `IndexServer` to `ApiServer`.
* sandbox: Move the services into the `apiserver.services` package.
* sandbox: Move SandboxEventIdFormatter's vals to the top.
* sandbox: StandaloneApiServer helper classes don't need to be Products.
* docs: Fix links for LedgerApiServer and Standalone{Api,Indexer}Server.
* sandbox: Delete a misleading comment from `StandaloneApiServer`.
* sandbox: Rename SandboxEventIdFormatter to EventIdFormatter.
* Move all datatypes out of daml-prim
This moves the remaining two modules DA.Types and GHC.Tuple to
separate LF packages with stable identifiers.
The only data types remaining are the ones for typeclasses which will
disappear once we move this to type synonyms.
CHANGELOG_BEGIN
- [DAML Compiler] The modules DA.Types and GHC.Tuple from daml-prim
have been moved to separate packages.
CHANGELOG_END
* Fix codegen tests
* Fix DarReader test
* Fix kvutils tests
* Fix jdbcdao tests
* Fix hs ledger bindings tests
* Add ledger and participant ID to claims
CHANGELOG_BEGIN
- [Ledger] AuthService implementations can now restrict the validity of access tokens to a single ledger or participant.
- [Sandbox] The sandbox JWT authentication now respects the ledgerId and participantId fields of the token payload.
CHANGELOG_END
* Add tests for ledger and participant in claims
* Address review comment
* Address review comment
* Fix tests
* Fix tests
* ledger-api-scala-logging: Fix errors in IntelliJ IDEA.
The Bazel plugin for IntelliJ doesn't seem to be smart enough to be able
to handle a Scala library that is part `src` directory and part
generated code from another Bazel rule. It just ignores the second part.
This means that IntelliJ cannot find the *ServiceLogging classes, as
they're not represented on the Bazel-generated classpath, and so
complains with lots of errors when working on the equivalent Api*Service
files.
To fix this, we can split these in two, compiling the base traits to
`ledger-api-scala-logging-base` and then the generated code separately.
It does result in an extra Bazel dependency for the users of
ledger-api-scala-logging, as Bazel doesn't realise transitive
dependencies for us.
* Release: Add `ledger-api-scala-logging-base` to the Maven list.
* ledger: Document the health checks.
* sandbox: Build a Docker image.
* sandbox: Create a sample Kubernetes YAML file.
* sandbox: Add health probes to the sample Kubernetes configuration file.
Startup and liveness are tested with a simple TCP connection to port
6865. Readiness checks are done with `grpc-health-probe`, which is added
to the Sandbox container image.
* sandbox: Link to kubernetes.yaml in the README and provide a disclaimer.
// changelog removed as it's not actually relevant to users
* sandbox: Don't try and build `sandbox-image-base` on Windows.
* Apply suggestions from code review
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: HikariJdbcConnectionProvider.start(), rather than construction.
It's too easy to construct things; I'd like it to be obvious that we're
starting something (in this case a connection pool).
* sandbox: In SqlLedgerSpec, stop ledgers before shutting down PostgreSQL.
* sandbox: Perform JDBC health checks on a timer.
The problem with hooking into existing requests is that if the Sandbox
is running in a load balancer and reports itself as unhealthy, it might
be taken out of the load balancer. Without any requests going through,
it'll never realize it's healthy again, and so will remain thinking it's
unhealthy forever.
* sandbox: Inline SqlExecutor into DbDispatcher.
There's no clear delineation of responsibilities. I'll try and separate
some of this back out again later.
* sandbox: Make it clear that we're "starting" the DbDispatcher.
* sandbox: Simplify the promise-based behavior in DbDispatcher.
* sandbox: Rename `noOfShortLivedConnections` to `maxConnections`.
The timeout for witnessing the party allocation is now increased
to 30 seconds (from 10 seconds). Additionally the LotsOfParties
conformance test is marked as flaky.
* Sandbox: Remove streaming connections
The separate database connection pool for streaming connections
was only used for the active contracts stream. However, a single
db connection was being occupied until the last active contract was
streamed over the Ledger API to the client. This effectively means
that only ever 2 concurrent active contract streams could exist.
No need to say that this is bad design.
The following changes happened:
- remove the db connection pool for streaming connections
- replace the streaming mechanism for active contracts with
the already existing pagination mechanism in JdbcLedgerDao
- change the pagination mechanism to actually use database level
limit and offset instead of doing the pagination "client side"
- configure the HikariDataSource with the metric registry
CHANGELOG_BEGIN
- [Sandbox] Improve loading of active contracts for the Sandbox SQL backend.
CHANGELOG_END
* Extract PaginatingAsyncStream from JdbcLedgerDao for testing
* Reset metrics registry before each test
* sandbox: If the Flyway migrations fail, crash the process.
Otherwise running it in the background can make it look like
everything's OK.
* reference-v2: If the Flyway migrations fail, crash the process.
Otherwise running it in the background can make it look like
everything's OK.
* sandbox: Remove an errant debugging `println`.
* sandbox: Use `DirectExecutionContext.implicitEC`.
* convert party allocation to the new committer pattern
* scalafmt
* Use map with optional values for kvutils input state
We need to do this to detect when a submission has maliciously
omitted an input key.
This is a first step towards making sure that the package ids for
types defined in daml-prim and daml-stdlib don’t change. This PR
mostly adds all the necessary infrastructure for that and moves
GHC.Types and GHC.Prim to make sure it works.
Until data-dependencies are really solid and we have verified that we
no longer have performance issues with an increasing number of Haskell
packages, we still include the source files in daml-prim and then just
rewrite the references.
We will also need to add tests that these packages really have stable
ids but I’ll leave that for separate PRs since this doesn’t make that
much sense anyway until all of the types have moved to stable
packages.
CHANGELOG_BEGIN
- [DAML Compiler] The modules GHC.Prim and GHC.Types from daml-prim
have been moved to separate packages.
CHANGELOG_END
* Track used packages during whole of engine submit
- Introduce MutableCompiledPackages interface
- Add TrackingCompiledPackages that tracks fetched packages
- Make used packages in transaction optional to distinguish between
missing dependencies and empty set of dependencies.
* Reimplement package dependency tracking
- Compute direct dependencies of a package during decoding
- Compute transitive dependencies of a package when adding a package
to engine.
- Annotate the resulting transaction with package dependencies
in Engine.submit.
* Create Ast.Package with proper direct deps in scenario service
While we don't have use for direct dependencies of a package in
scenario service (only Engine.submit needs it), it's better to be
accurate.
This of course overapproximates the direct dependencies.
* Compile a each new package once in ConcurrentCompiledPackages
* Revert "ledger-api-test-tool: The endpoints are mandatory. (#3611)"
This reverts commit 603ee9367b.
* Restore LedgerTestContext requirement
* Exit early if there are no participants to test
* ledger-api-test-tool: Increase the duration when watching health.
This should hopefully stop CI from flaking out.
* reference-v2/sandbox: Avoid unnecessary companion object constructors.
I like indirection… when it does something.
* ledger: Propagate empty health checks throughout the services.
* reference: Remove duplication from the ReferenceServer object.
* ledger-api-common: Actually query a "reporter" in the health service.
* ledger-api-common: Report health per-component when required.
* ledger-api-health: Use a Map to represent components for health checks.
* sandbox: Fix warnings in SqlLedgerSpec.
* ledger-api-common: Throw GrpcHealthService errors inside the Future.
* ledger: Implement health checks against the PostgreSQL connection.
Without proper testing, because I am not great at this.
* sandbox: Remove duplication and fix warnings in PostgresAround.
* sandbox: Test the SQL Ledger's health reporting on failure.
* sandbox: Don't report as unhealthy until 3 connections fail.
* ledger-api-health: Remove unused parts of the API.
Bit of premature design there.
* sandbox: Rename the "ledger" health check to "write".
* participant-state: Add the ReportsHealth trait to ReadService.
* ledger-api-common: `Future.fromTry(Try(…))` -> `Future(…)`.
* ledger-api-common: Make it clearer that StubReporter closes over health.
* ledger-api-common: Explain the HealthService watch tests with comments.
* sandbox: Clean up SqlLedger a bit.
* sandbox: Don't try and stop PostgreSQL twice in PostgresAround.
* bazel_tools: Windows rlocation lookups need to be with forward slashes.
* release: Fix case of "true".
* ledger-api-common: Make `GrpcHealthService::matchResponse` return a Try.
* ledger-api-common: Make `GrpcHealthServiceSpec` async.
* sandbox: Make a couple of DB classes final.
* sandbox: Avoid importing `X._` in PostgresAround.
* sandbox: Add clues to the SqlLedgerSpec's multiple assertions.
* sandbox: If PostgreSQL doesn't come back up, keep retrying.
* sandbox: Remove duplication in SqlLedgerSpec.
* sandbox: In SqlLedgerSpec, actually wait for the health to change.
* sandbox: In PostgresAround, make stopping PostgreSQL idempotent.
* sandbox: Simplify the SqlLedgerSpec to make it work on CI.
It's worth a shot.
* ledger-api-common: Simplify the GrpcHealthServiceSpec a little.
And add a changelog.
CHANGELOG_BEGIN
- [Ledger API Server] Add a health check endpoint conforming to the
`GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
- [Ledger API Server] Add health checks for index database connectivity.
- [Participant State API] Add a mandatory ``currentHealth()`` method to
``IndexService``, ``ReadService`` and ``WriteService``.
CHANGELOG_END
* sandbox: Improve the Javadoc layout for DbDispatcher.
* sandbox: Capitalize constants in SqlExecutor.
* ledger-api-health: Convert HealthStatus to an abstract class.
* Add participant-state configuration protobuf
- Move the "DamlConfiguration" from kvutils into participant-state/protobuf/ledger_configuration.proto.
- Add version number and spec (ala transaction.proto)
This is a preparation for indexing the configuration and having one canonical serialization for it.
* Initial thoughts on indexing the configuration
* Implement indexing of ledger configuration changes
* Add record time to all Updates. Wire through participant id.
and rename V7__Add_configuration to V8.
* Add ledger_configuration_java_proto to release artifacts
* Fix up release of ledger_configuration_java_proto
* Suggestions from review
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Apply suggestions from code review
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* address rebase issues
* Resolve compilation errors after rebase/merge
* happy formatting for scala and bazel and friends
* Drop "openWorld" setting from configuration
And refactor the tests to allocate parties explicitly.
* Fix up migration and tests
* Drop authorizedParticipantIds from configuration
Implement configuration authorization in kvutils using the previous
the participant id of the previous configuration.
* Post-rebase fixes
* Add missing migrations
* Apply suggestions from code review
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Add missing mavenUpload to ledger-api-server
* Remove stateUpdateRecordTime
* Address code review
- Address PR review
- Merge TimeModelImpl and the traits. Remove TimeModel from ledger-api-common.
- Throw `Err` from KeyValueConsumption on parse errors instead of assert/sys.error
* Reformat
* Add missing protobuf file
* Fix compilation after TimeModel changes. Add version logs to participant-state{,-kvutils}.
* Fix TestUtil.scala build
* Apply suggestions from code review
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Address review
- synchronized access to ledgerConfiguration in InMemoryLedger
- store rejection if configuration generation is wrong
* Update ledger/participant-state/protobuf/ledger_configuration.rst
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Add authentication to Java identity client
Contributes to #3626
No changelog added right now, it will be added with the PR that
effectively closes the ticket.
Moves `LedgerCallCredentials` to a shared package.
Establishes the test framework for other Java clients.
* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731
* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731
* Rebase and fix compilation errors
* Fix type of jar in artifacts.yaml
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Fix more errors emerged after rebase
* Add dependency for DAML assistant ITs
* Add missing dependency
* Fix wrong dependency
* fix release
* Move AuthorizationIT to Sandbox ITs
* Use refactored code in Extractor
* Apply a few improvements, see body
- Run previously ignored tests
- Make token signing private
- Fix stream observer utility comment more accurate
- Add read-only token helper
* Add r/w token split tests
* Fix formatting and imports
* Address https://github.com/digital-asset/daml/pull/3598#discussion_r349769546
* Address https://github.com/digital-asset/daml/pull/3598#discussion_r349769801
* Fix inaccurate test
A test was incorrectly trying to get a token with an invalid signature
* Address reviews
* kvutils: Nest all InMemoryKVParticipantState tests inside the block.
* kvutils: Simplify InMemoryKVParticipantStateIT where possible.
* kvutils: In InMemoryKVParticipantStateIT, move assertions to the end.
It's confusing when the assertions are before the submissions.
* kvutils: Move test helpers into a companion object.
* kvutils: Use BeforeAndAfterEach in testing to remove duplication.
* kuvtils: In testing config updates, always increment the generation.
* kvutils: Assert something in the "initial conditions" integration test.
* ledger-api-test-tool: Use the default Akka execution context.
Akka will use the global fork-join pool by default, which is fine for
our purposes. There's no longer a need to use a specific execution
context, because we no longer use it to limit the work in progress.
That's now done by the parallelism parameter to Akka Streams'
`mapAsyncUnordered`.
* ledger-api-test-tool: Shut down the ActorSystem.
* Improve Navigator output
* Fix Navigator not using the access token
* Add RSA signatures for JWT tokens
* Remove unused method
* Add timeouts for reading JWKS
* Fix test
* Rename method for consistency
* Improve comment
* More renaming for consistency
* CHANGELOG
CHANGELOG_BEGIN
- [Sandbox] Add CLI options to start the sandbox with JWT based authentication with RSA signed tokens.
See `issue #3155 <https://github.com/digital-asset/daml/issues/3155>`__ .
- [Navigator] Fixed a bug where the `--access-token-file` option did not work correctly.
CHANGELOG_END
* Make JwksVerifier limits configurable
* Make SimpleHttpServer private
The shared-secret-based authentication is there exclusively for testing,
this PR makes sure that this is correctly reported through the CLI and
hides this until further notice.
Makes sure that the flag clearly says that this option is not meant for
production use cases.
A warning is already printed when this feature is used, both at startup
and every time this verifier is used.
CHANGELOG_BEGIN
- [Sandbox] The ``--auth-jwt-hs256`` is renamed to
``--auth-jwt-hs256-unsafe``: you are advised to _not_ use this JWT token
signing way in a production environment.
CHANGELOG_END
Closes#3327
CHANGELOG_BEGIN
[Ledger]
* Allow non-alphanumeric characters in ledger api server participant ids (space, colon, hash, slash, dot). Proper fix for change originally attempted in v0.13.36. See issue `issue #3327 <https://github.com/digital-asset/daml/issues/3327>`__.
CHANGELOG_END
Proper fix replacing f39bf7fc5a which does not work
* grpc-utils: Simpler GrpcStatus extractors when you know the code.
And don't care about the description.
* grpc-utils: Simpler GrpcException extractors when you know the code.
And don't care about the description.
* Apply suggestions from code review
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Add `new` to `SpecificGrpc{Exception,Status}` constructor calls.
Also, don't blindly apply GitHub suggestions.
* grpc-definitions: Delete health_service.proto
We can use the version in io.grpc:grpc-services instead.
* ledger: Delete ledger/API.md.
* sandbox: Fix warnings in ApiServices flagged by IntelliJ.
* sandbox: Implement a dummy grpc.health.v1.Health.Check endpoint.
* sandbox: Implement a dummy grpc.health.v1.Health.Watch endpoint.
* sandbox: Drop repeated elements from grpc.health.v1.Health.Watch.
* sandbox: Wrap the HealthService in basic tests.
* sandbox: Stop streaming the server health too.
* ledger-api-test-tool: Health check tests.
* Add a changelog entry for the health check endpoints.
CHANGELOG_BEGIN
- [Ledger API] Add healthcheck endpoints, conforming to the
`GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
It is always ``SERVING`` for now.
- [DAML Ledger Integration Kit] Add conformance test coverage for the
``grpc.health.v1.Health`` service.
CHANGELOG_END
* ledger-api-integration-tests: Increment the number of services.
* Apply suggestions from code review
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: Use `AkkaBeforeAndAfterAll` in the HealthServiceSpec.
In an attempt to get it working on CI.
* sandbox: Change `dropRepeated` to `DropRepeated()`.
Keep it in one file.
* test-common: Use `Delayed.by` in `TimeBoundObserver`.
* test-common: Close the source when `TimeBoundObserver` completes.
* ./fmt.sh
That'll teach me not to `--no-verify` just because it's a merge commit.
* sandbox: Inline `HealthService.suppress`.
At some point it was being used twice.
* sandbox: Increase the timeout for HealthServiceSpec.
* sandbox: Reimplement HealthService using the Scala protobuf types.
* sandbox: Generate an Akka-compatible trait for the health service.
And refactor a lot of test code to make it easy to test.
* ledger-api-common: Move the HealthService here.
* rs-grpc-testing-utils: Publish to Maven.
* rs-grpc-testing-utils: Add Maven coordinates.
CHANGELOG_BEGIN
[DAML Integration Toolkit]
* Add ledger api test tool `--load-scale-factor` option that allows dialing up
or down the workload applied by scale tests (such as the
`TransactionScaleIT` suite). This allows improving the performance of
different ledger over time.
CHANGELOG_END
* Split read and write claims
Fixes#3398
CHANGELOG_BEGIN
- [Sandbox] Added support JWT tokens that only authorize to read data, but not to act on the ledger.
CHANGELOG_END
Instead of passing around MetricsManager just for the timedFuture
method, we pass the underlying MetricRegistry directly and use
the timedFuture function as an actual function (passing in a timer metric).
The Sandbox still starts the JmxReporter, but the integrator of the Indexer
and the Ledger API Server needs to enable the reporting facilities themselves.
CHANGELOG_BEGIN
- [Ledger] Ledger API Server and Indexer now accept an instance of ``MetricRegistry`` as parameters. This gives implementors of ledger integrations the most flexibility to set up metrics reporting that works best for them.
CHANGELOG_END
