* Upgrade Scala dependencies for 2.13 compatibility
This upgrades a bunch of Scala libraries to versions that have 2.13
support. There are two libraries that are still missing:
- diffson, this has a new version but with significant breaking
changes and it is only used in Naigator console which I hope to kill
before I have to worry about this.
- ai.x:diff, this is used in the ledger API test tool. The library is
abondened but there are a few alternatives.
changelog_begin
changelog_end
* Fix pureconfig
changelog_begin
changelog_end
* Fix Navigator
changelog_begin
changelog_end
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.
This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.
Apologies for the giant PR, I don’t see a way to keep it smaller.
changelog_begin
changelog_end
* Add multi-party submissions to the ledger API
CHANGELOG_BEGIN
* [Ledger API] Command submission requests now contain new optional
fields used for multi-party submissions. Such submissions currently
return UNIMPLEMENTED errors, they will be enabled in the future.
CHANGELOG_END
* Adapt Haskell bindings
* Handle the new ledger API fields
* Fix SubmitAndWaitDummyCommand
* Remove unused methods
* Redesign multi-party auth tests
* Remove direct access of request party
* Apply review comments
* Improve protobuf comments
* Multi-party tracker map
* Fix validation logic
* Consistent metric naming
* Document offset inclusivity and exclusivity in Protobuf
As brought up in https://discuss.daml.com/t/using-offsets/1731, the
inclusivity and exclusivity of start and end offsets as exposed by
the Ledger API is not well documented.
This commit clearly states which offset is inclusive and which is
exclusive on all relevant endpoints (command completion and
transaction services).
changelog_begin
changelog_end
* Address https://github.com/digital-asset/daml/pull/8145#pullrequestreview-543758867
* Bazel: Reduce the visibility of some Protobuf targets.
* Bazel: Make Protobuf JAR visibility parameterizable.
Default to private, and explicitly make it public where it's needed.
CHANGELOG_BEGIN
CHANGELOG_END
* [DPP-84] Employ parallel unnesting instead of batching on PostgreSQL
changelog_begin
[Integration Kit] When using a PostgreSQL-based index, leveraging native
parallel unnesting allows to more efficiently index new transactions.
changelog_end
* Address review https://github.com/digital-asset/daml/pull/8042#pullrequestreview-541759596
* ledger-api: Use `proto_jars`.
CHANGELOG_BEGIN
- [Ledger API] The Scala JARs containing the gRPC definitions no longer
contain the *.proto files used to generate the ScalaPB-based classes.
CHANGELOG_END
* Create a source JAR for *.proto files in `proto_jars`.
* ledger-api: Publish the protobuf sources as "ledger-api-proto".
CHANGELOG_BEGIN
- [Ledger API] The *.proto files containing the gRPC definitions are now
provided by a new Maven Central artifact, with the group "com.daml"
and the artifact name "ledger-api-proto".
CHANGELOG_END
* release: We don't need the "main-jar" option.
* Bazel: Proto JARs will always have a Maven artifact suffix.
* Bazel: Simplify Protobuf source file TAR and JAR targets.
* Bazel: Extract out Protobuf functions.
* Participant pruning ledger api server support ported from Canton
CHANGELOG_BEGIN
- [Ledger API]: The preview of `ParticipantPruningService` enables ledger participants to prune the "front" of ledger state at the participant including the ledger api server index.
CHANGELOG_END
* Review feedback from Stefano
* Add pruning tests plus missed command completions change
* Review feedback from Robert
* Improved test readability by having populate helper return offsets
* Review feedback
* Ledger api changes to pruning api and disable canton pruning test
- Change return result to PruneResponse
- Change type of PruneRequest.prune_up_to to string
* Review feedback: Use ApiOffsetSConverter for logged offsets
* Include proto files in ScalaPB source jar
* Include proto files in ScalaPB jar
* remove empty line
* use ledger_api_proto_source_root variable
* Fix formatting
CHANGELOG_BEGIN
- Added protobuf source files to ledger-api-scalapb artifact
CHANGELOG_END
This change exposes a new endpoint with information about the ledger API version.
The current approach is to read contents of ledger-api/VERSION and provide it as it is to the user.
CHANGELOG_BEGIN
- ledger API version endpoint
CHANGELOG_END
* resources: Move builders into //ledger/ledger-resources.
Keep the actual constructors in a trait, but instantiate it when working
with ledger code.
This allows us to later introduce an extra "context" type parameter to
ResourceOwner.
* resources-akka: Move the builders in to //ledger/ledger-resources.
* resources: Introduce an abstract `Context` parameter for owners.
This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.
This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.
Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.
CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
`Context`, which is filled in by the corresponding `Context` class in
the _ledger-resources_ dependency. This allows us to pass extra
information through resource acquisition.
CHANGELOG_END
* ledger-resources: Move `ResourceOwner` here from `resources`.
* ledger-resources: Remove dependencies from outside //ledger.
* ledger-resource: Wrap the acquisition execution context in `Context`.
So we can add a logging context to it.
* resources: Pass the Context, not the ExecutionContext, to Resource.
* Avoid importing `HasExecutionContext`.
* ledger-resources: Publish to Maven Central.
* resources: Make the small changes suggested by @stefanobaghino-da.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* ledger-resources: Pull out a trait for test resource contexts.
Saves a few lines of code.
* Restore some imports that were accidentally wildcarded.
* resources: Replace an `implicit def` with a couple of imports.
* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.
Try and use the right execution context where possible.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* concurrent: Tag DirectExecutionContext.
1. Tag `DirectExecutionContext` as `ExecutionContext[Nothing]`, thereby
stating that it works for any tagged `Future`.
2. Move `DirectExecutionContext` to the _libs-scala/concurrent_
library, as it requires it and it's tiny.
CHANGELOG_BEGIN
CHANGELOG_END
* concurrent: Fix the privacy of `DirectExecutionContextInternal`.
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
In order to introduce versioning of the ledger api we
first add a central container of the version for tracking
purposes.
CHANGELOG_BEGIN
CHANGELOG_END
This PR implements a part of the proposal from #7093.
Here packages are validated in the participant node before to be sent to the ledger.
CHANGELOG_BEGIN
- [Ledger-API] participant node validate Dar before uploading to the ledger.
This may increase upload time significantly.
CHANGELOG_END
* remove unused definitions, params, args from ledger API Scala code
CHANGELOG_BEGIN
- [Ledger API] withTimeProvider removed from CommandClient; this method
has done nothing since the new ledger time model was introduced in
1.0.0. See `issue #6985 <https://github.com/digital-asset/daml/pull/6985>`__.
CHANGELOG_END
* percolate withTimeProvider and label removal elsewhere
* Use Bazel builtin pkg_tar rule
* Use @rules_pkg//:pkg.bzl%pkg_tar
The pkg_tar rule builtin to Bazel has been deprecated.
See https://docs.bazel.build/versions/master/be/pkg.html
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* participant-integration-api: `GrpcServerOwner` -> `GrpcServer.Owner`.
Mostly so I can create a test class named `GrpcServerSpec`.
* ports: Move the free port search from postgresql-testing.
* participant-integration-api: Test the basics of GrpcServer.
This uses the HelloService to make sure the server behaves normally.
* ledger-api-client: Extract out channel configuration from LedgerClient.
So we can test it independently of the LedgerClient itself.
* ledger-api-client: Increase the default maximum inbound header size.
Increased from 8 KB to 1 MB.
* participant-integration-api: Reduce the maximum error message size.
Truncate GRPC error descriptions to 256 KB.
* participant-integration-api: Use `Port.Dynamic` instead of `FreePort`.
In tests.
* participant-integration-api: Explicit null checks when they're shorter.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* ledger-api-client: Reduce the max inbound message size back to 8 KB.
And reduce the maximum size of an error description pushed out by the
server accordingly.
CHANGELOG_BEGIN
- [Integration Kit] Truncate GPRC error messages at 4 KB. This ensures
that we won't trigger a protocol error when sending errors to the
client.
CHANGELOG_END
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
This was never intentional, nobody even knew that this was possible
and we have an alternative, documented way of getting this via github
releases.
To avoid introducing this issue again, I’ve removed non-jar artifact
types from the Maven upload script.
fixes#448
changelog_begin
changelog_end
On closing the SingleThreadExecutionSequencerPool, all sequencers in the
pool are set to null (presumably to encourage garbage collection).
However, you can still ask for the next executor, which will then return
`null`, causing all manner of problems down the line.
This change forces the failure early, making it easier to debug, by
checking whether the pool is closed and if so, throwing an
`IllegalStateException`.
We see these null pointer exceptions in the `BotTest` test logs from
time to time.
CHANGELOG_BEGIN
- [RxJava Bindings] We now fail faster, with a more meaningful error,
when RxJava flows continue running after shutting down the client.
CHANGELOG_END
* disable Any wart
* first pass removal of Any suppressions for false positives
* second pass removal of Any suppressions for false positives
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* third pass removal of Any suppressions for false positives
* fourth pass removal of Any suppressions for false positives
* reformat newly single-suppressions into single lines
- suggested by @SamirTalwar-DA; thanks
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
We saw a few issues in CI where `setup` might be called twice, which
resulted in a test fail for no good reason. Rather than crashing, we
can just do nothing the second time.
CHANGELOG_BEGIN
CHANGELOG_END
* Use com.daml as groupId for all artifacts
CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END
* Add 2 additional maven related checks to the release binary
1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId
* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
* Remove old time model from ledger config
CHANGELOG_BEGIN
- [Ledger API] Fields related to the old ledger time
model have been removed from the configuration
management service and the ledger configuration service.
CHANGELOG_END
* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/tests/LedgerConfigurationService.scala
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
Contributes to #4194.
Closes#4231.
Closes#5022.
CHANGELOG_BEGIN
- [Ledger API] The protobuf fields ledger_effective_time and maximum_record_time have been removed from
command submission. These fields were previously deprecated following the introduction
of a new ledger time model. See issue `#4194 <https://github.com/digital-asset/daml/issues/4194>`__.
[Java Bindings] removed the usage of ledgerEffectiveTime and
maximumRecordTime, and instead added minLedgerTimeAbsolute and
minLedgerTimeRelative in CommandSubmissionClient and CommandClient
CHANGELOG_END
* sandbox: Capture timing metrics for API server calls.
`timer` is a superset of `meter`, so this doesn't lose any existing
behavior; just adds new behavior.
CHANGELOG_BEGIN
- [Ledger API Server] Added timing metrics for all GRPC endpoints.
CHANGELOG_END
* sandbox: Rename SandboxClientResource to GrpcClientResource.
* sample-service: Clean up warnings.
* sandbox: Add tests for MetricsInterceptor.
* sandbox: Split the API metrics interceptor from the naming.
* sandbox: Use `MetricRegistry.name` instead of string interpolation.
* rs-grpc-akka: Restrict the test library to the DAML workspace.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Tighten result type
Command execution can't result in a sequencer error
* New helper method for extracting used contracts
* New error clause
* Add a DAO query for the maximum time of contracts
* Implement algorithm for finding ledger time
CHANGELOG_BEGIN
CHANGELOG_END
* fixup ledgerTimeHelper
* Use new ledger time algorithm
* Mark LET/MRT as deprecated
CHANGELOG_BEGIN
- [Ledger API] DAML ledgers have switched to a new ledger time model.
The ledger_effective_time and maximum_record_time fields of command submission are deprecated,
the ledger time of transactions is instead set automatically by the ledger API server.
Ledger time is no longer strictly monotonically increasing, but only follows causal monotonicity:
ledger time of transactions is greater than or equal to the ledger time of any used contract.
See `#4345 <https://github.com/digital-asset/daml/issues/4345>`__.
CHANGELOG_END
* Add ledger time skew check
* Remove command updater
LET/MRT are now deprecated, this class is now useless
* Remove old time model validator
* Switch to new time model check: kvutils
* Switch to new time model check: in-memory ledger
* Switch to new time model check: SqlLedger
* Use initial ledger config
* Ignore user provided LET
* Use TimeProvider in submission services
* Use deduplication_time in daml-script runner
- Also remove unnecessary command completion output of CommandTracker.
- Remove usage of maximum record time in CommandTracker.
* Use arbitrary default value for deduplication time
* Use built-in Instant ordering
* Remove obsolete test
* Remove obsolete test: CommandStaticTimeIT
* Refactor test: TransactionMRTCompliance
* Disable test: CommandTrackerFlow timeout
* thread maxDeduplicationTime through to CommandTracker
* Improve test
* Refactor command client configuration
* Deduplication time should always use UTC
* Add missing method in TimedIndexService after rebase
* Put more details into the deduplication error response.
* Use system time for command dedup submittedAt.
* Use explicit UTC time source in command validator
* Revert CommandTracker[Flow] to previous completion-recovering-behavior
* Adapt scala client command config to new config params
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* rs-grpc-bridge: Separate the test library from tests.
I don't want _logback-test.xml_ shipping with the
TestExecutionSequencerFactory.
* rs-grpc-bridge: Actually run the tests.
How we managed to miss this, I don't know.
CHANGELOG_BEGIN
CHANGELOG_END
A "stable offset" in the context of the Participant Server is the offset
that was provided by the ledger backend (be it kvutils, corda, daml on sql).
The Participant Server does not keep a participant-local offset anymore.
In a single domain/kvutil setup, this makes offsets stable across participants,
since all participants will see the same offset for the same transaction.
The following changes were needed to achieve this:
- The participant server always uses the offset provided by the backend
AS IS (no more +1 magic).
- Offsets provided to the Ledger API in requests must be treated as
startExclusive and endInclusive (previously beginInclusive and
endExclusive).
CHANGELOG_BEGIN
[Ledger API]: Offsets have been redefined. Instead of being represented
by a number or a structured string, an offset is now an opaque string
that can be compared lexicographically.
[DAML Integration Kit]: The bounds for ``Dispatcher`` are now
startExclusive and endInclusive.
CHANGELOG_END
---------
ledger api:
ledger_offset.proto
Changed definition of offsets, since they can now be compared
lexicographically.
---------
participant-state-api:
Offset:
Changed from Array[Long] to ByteString. Ledgers need to make sure that the
offsets produced are strictly monotonically increasing according to
lexicographical order.
---------
akka-streams:
Dispatcher, DispatcherImpl, SubSource:
Changed interval handling to exclusive/inclusive.
---------
ledger-on-memory:
InMemoryLedgerReaderWriter, InMemoryState:
Changed interval handling to exclusive/inclusive.
---------
ledger-on-sql:
CommonQueries, SqlLedgerReaderWriter:
Change interval in query and boundary handling.
---------
kvutils:
KeyValueParticipantStateReader, KVOffset:
Convenience functions for kvutils to add or remove sub-indexes for
offsets.
KV ledger implementations can use KVOffset to construct a structured offset.
---------
Participant Server:
JdbcLedgerDao:
Use Offset instead of Long.
Fetch offsets directly as Offset from the database with proper anorm
integration.
Change interval handling to exclusive/inclusive.
CommandCompletionsReader, CommandCompletionsTable:
Change interval handling to exclusive/inclusive.
BaseLedger:
Use Offset instead of Long.
Change interval handling to exclusive/inclusive.
Conversions:
Anorm integration for using Offset in queries and result parsers.
JdbcIndexer:
Remove references to "extenalLedgerEnd" and participant-local Long
offset (headRef).
---------
sandbox:
In general:
Use the Offset type everywhere instead of Long.
SQL migrations:
Change all offset columns to bytea or BINARY.
LedgerBackedIndexService:
Proper bounds checking has been pushed down to Dispatcher, which
allowed simplifying the acceptedTransactions implementation.
InMemoryLedger, LedgerEntries:
Change interval handling to exclusive/inclusive.
Transaction lookup by ID is now O(n) because transaction IDs are not
necessarily the same as the offset.
SqlLedger:
Remove external offset references.
CHANGELOG_BEGIN
[Documentation] Ledger API documentation clarifies how witness_parties
are determined depending on whether CreatedEvents are served as part of
the flat or full transaction stream.
CHANGELOG_END
* sandbox: Add a database test for storing and retrieving parties.
* sandbox: Add database queries for selecting one or many parties.
* ledger-api-test-tool: Add a test for `ListKnownParties`.
* sandbox: Add an endpoint to retrieve a single party's details.
CHANGELOG_BEGIN
- [Ledger API] Added an endpoint to retrieve a single party's details at
``com.digitalasset.ledger.api.v1.admin.PartyManagementService.GetParty``.
Please consult the ledger API reference documentation for more
information.
CHANGELOG_END
* sandbox: Add an endpoint to retrieve a multiple parties' details.
CHANGELOG_BEGIN
- [Ledger API] Added an endpoint to retrieve multiple parties's details at
``com.digitalasset.ledger.api.v1.admin.PartyManagementService.GetParties``.
Please consult the ledger API reference documentation for more
information.
CHANGELOG_END
* sandbox: Getting a single party is a special case of multiple parties.
So let's use that code path and stop duplicating work.
* sandbox: Remove `GetParty`, as it's subsumed by `GetParties`.
"Subsumed" is a great word.
* Always return error on duplicate submissions
* Remove unnecessary submission information
Now that duplicate submissions always return an error,
we don't need to store the original submission result.
CHANGELOG_BEGIN
CHANGELOG_END
* Rename ttl to deduplicationTime/deduplicateUntil
* Store absolute deduplicateUntil in domain commands
* Fix my own initials
* Remove CommandDeduplicationEntry
Instead, use CommandDeduplicationResult everywhere,
removing the extra layer.
* Add TTL field to protobuf
* Add command deduplication to index service
* Wire command deduplication to DAO
* Implement in-memory command deduplication
* Remove Deduplicator
* Implement JDBC command deduplication
* Add TTL field to domain commands
* Deduplicate commands in the submission service
CHANGELOG_BEGIN
- [Sandbox] Implement a new command submission deduplication mechanism
based on a time-to-live (TTL) for commands.
See https://github.com/digital-asset/daml/issues/4193
CHANGELOG_END
* Remove unused command service parameter
* fixup protobuf
* Add configuration for TTL
* Fix Haskell bindings
* Rename SQL table
* Add command deduplication test
* Redesign command deduplication queries
* Address review comment
* Address review comment
* Address review comments
* Make command deduplication test optional
* Disable more tests
* Address review comments
* Address review comments
* Refine test
* Address review comments
* scalafmt
* Truncate new table on reset
* Store original command result
* Rename table columns
... to be consistent with other upcoming tables
* Rename migrations to solve conflicts
Fixes#4193.
* sandbox: Move more resource acquisition into the `owner`.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Reimplement SandboxClientResource as a resources.Resource.
* codegen: Use resources in TestUtil.
* sandbox: Manage PostgreSQL in tests with ResourceOwners.
* sandbox: Don't hold on to old resources when resetting.
Now there's one hell of a memory leak.
CHANGELOG_BEGIN
- [Sandbox] Fixed a memory leak when using the ResetService; not
everything was cleaned up correctly.
CHANGELOG_END
* sandbox: Split out SandboxClientResource from SandboxServerResource.
Gonna replace SandboxServerResource with a ResourceOwner acquisition.
* sandbox: Don't capture the API server in the SandboxServer resource.
When we reset, this is stored forever, leading to a memory leak.
Tested by rewriting the SandboxServerResource to use
`SandboxServer.owner`.
* sandbox: Revert the test client resource to calling `shutdownNow()`.
* sandbox: Make sure the fixture is recreated properly on each test run.
* sandbox: Make `SandboxState` a non-case class.
The `toString()` was unnecessarily heavy.
* sandbox: Futures, futures everywhere.
Avoid a race condition where the server is stopped before it starts by
storing a `Future[SandboxState]` rather than the `SandboxState` itself.
This doesn't trigger the same memory leak as storing a
`Resource[SandboxState]` because we don't capture the object itself in
the `flatMap` in the same way with `Future`.
* sandbox: Remove an unused parameter left in for debugging.
* sandbox: Replace `@VisibleForTesting` with a comment.
* sandbox: Add more comments to the weird logic in SandboxServer.
* sandbox: Get rid of the `Port` type alias; it was confusing.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
This allows using this from `akka-typed` where you otherwise get an
error that you `cannot create top-level actor from the outside on
ActorSystem with custom user guardian`. See
https://github.com/akka/akka-http/issues/1873#issuecomment-367655223
for the similar issue in `akka-http`. The resolution matches the fix
that `akka-http` https://github.com/akka/akka-http/pull/1878 and
`akka-streams` https://github.com/akka/akka/pull/24460 have used.
I’ve followed the approach taken by `akka-streams` which matches on
the system and only uses a system actor if it gets an
ExtendedActorSystem. For my purposes, it would also be fine to change
the signature to require an `ExtendedActorSystem` but that seems like
a larger change.
changelog_begin
changelog_end
* sandbox: Create a monadic `ResourceOwner` to manage resources.
* sandbox: Rewrite `ResourceOwner` to be async.
* sandbox: Make sure failed resources are closed immediately.
* sandbox: Better naming in `Open`.
* sandbox: Rename `Open` to `Resource`, and open/close to acquire/release.
* sandbox: Convert `() => AutoCloseable` into `ResourceOwner`.
* sandbox: Refactor the LedgerApiServer in terms of resources.
* sandbox: Explicitly convert `() => AutoCloseable` to `ResourceOwner`.
Explicit > Implicit, right?
* sandbox: Create helpers for converting things to ResourceOwners.
Because I tried to start using them and there was so much code being
written at once.
* sandbox: Simplify construction of JdbcLedgerDao.
* sandbox: Releasing resources should be idempotent.
In that we should only do it once.
* sandbox: Fix the ResetService by closing the API services _first_.
They need to be shut down before the gRPC server.
* sandbox: Don't try and shut down PostgreSQL twice in tests.
* sandbox: Actually run the assertions in ResourceOwnerSpec.
Facepalm.
* sandbox: Test `Resource.sequence` more rigorously.
* sandbox: Move the helpers around `Resource` into `Resource.apply`.
* sandbox: Convert LedgerApiServer resource owners to classes.
* sandbox: Make `ResourceOwner` a monad too, delegating to `Resource`.
* sandbox: Turn `LedgerApiServer` into a ResourceOwner.
* sandbox: Simplify the public signature of `Resource.apply`.
* sandbox: Use ResourceOwners to simplify DB resource management.
This is one hell of a change. Sorry.
* sandbox: Try not to nest `Await.result` calls.
Causes issues when running in a `DirectExecutionContext`.
* sandbox: Turn index subscriptions into resources.
* sandbox: Fix warnings in RecoveringIndexerSpec.
* sandbox: Always release before recovering the indexer.
* sandbox: Add `flatten` and `transformWith` to `Resource`.
* sandbox: If releasing twice in parallel, the second should wait.
* sandbox: If the indexer recovers, clean up the old subscription.
* sandbox: Convert StandaloneIndexerServer into a resource owner.
* sandbox: Convert StandaloneApiServer into a resource owner.
* reference-v2: Rewrite ReferenceServer in terms of resources.
CHANGELOG_BEGIN
- [Reference v2] On an exception, shut down everything and crash.
Previously, the server would stay in a half-running state.
CHANGELOG_END
* sandbox: Rewrite SandboxServer in terms of resources.
* sandbox: Write the port file in a Future.
* sandbox: JdbcIndexer no longer needs to manage the actorSystem.
* sandbox: Shut down the LedgerApiServer when closing the Sandbox.
* sandbox: Rename `Resource.pure` to `Resource.successful`.
* sandbox: Rename `Resource.sequence_` to `sequenceIgnoringValues`.
* sandbox: Delete `CloseableResource`.
It's only used in once place. Just inline it.
* sandbox: `LedgerDao` no longer needs to be closeable.
* sandbox: Delete implicit materializers where they're not used.
* http-json: Wait for the Sandbox to start in tests.
* sandbox: Convert `scheduleHeartbeats` into a ResourceOwner.
* reference-v2: Explain why we steal ownership of the actor system.
* sandbox: Document why we only release resources once.
* sandbox: Add clues to ResourceOwnerSpec.
* http-json: Fix HttpServiceTestFixture to pass auth service through.
* codegen-sample-app: In ScalaCodeGenIT, wait for the server to start.
* Moves CommandCompletionIT to Ledger API Test Tool
Closes#2841
Also refactors test observers into a single API and unifies the existing implementations of the DirectExecutionContext
* Allow explicit offsets to get checkpoints
* Raise checkpoint test timeout
* Consolidate checkpoint tests in a single test case
* Fix compilation issues
* Exclude command submission/completion tests for rev-v2
* Drop test not relevant for Ledger API implementations
* Fix compilation errors
* Upgrade to Akka 2.6.1, akka-http 10.1.11 and Scala 2.12.10
Akka 2.6.1 Upgrade Changes
- Materializer in place of ActorMaterializer
- Source.future instead of Source.fromFuture
- The Scheduler.schedule method has been deprecated in favor of selecting scheduleWithFixedDelay or scheduleAtFixedRate
- onDownstreamFinish(cause: Throwable)
- ActorAttributes.supervisionStrategy(...) in place of ActorMaterializerSettings.withSupervisionStrategy
See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html
* Akka 2.6.1 Upgrade Changes
- onDownstreamFinish(cause: Throwable)
See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html
* code review: remove unnecessary supervision strategy
* Implementation for the configuration management service
- Add configuration generation to the response of SetTimeModel
- Implement the ConfigManagementService
- Implement integration test into test tool
This is still a draft as it has unsolved FIXMEs and
it conflicts with #3744 which should go first.
The main conflict is with changes to PartyAllocationResponse which cannot
no longer reference "domain.PartyDetail" as we would have a cyclic dependency
(participant-state contains Configuration which we point to from domain.ConfigurationEntry).
The still open issues are:
- Revisit PartyDetail
- Naming: LedgerConfigurationService and ConfigManagementService are not talking
about the same configuration and it feels confusing.
- Remove duplication of ConfigurationEntry? Do we need both domain.ConfigurationEntry
and ledger.store.ConfigurationEntry? Only difference is in the types of participantId
and submissionId.
* Address review part 1
* Fix up tests after rebase and address PR review
* Post-merge fixes
* Add missing config MRT checks and fixes to tests
- Check config MRT in InMemoryLedger and SqlLedger
- Use proper source of time in ConfigManagement
- Separate out ConfigManagementServiceIT in sandbox conformance tsets
* Reformat
* Add package_entries table
* Change PublicPackageUpload event to cover list of packages.
Add PublicPackageUploadRejected.
* Produce new package update events in KeyValueConsumption
* Update signature of uploadPackages
* Cleanup InMemoryKVParticipantState. Add submissionId to uploadPackages.
* Fix up InMemoryKVParticipantStateIT
* Initial ledger dao changes for package entries
Drop the participant_id as we never expect to see
entries of other participants. This should be done
for party_entries as well.
* Drop UploadPackagesResult
* Implement getPackageEntries and refactor callers
* Add maxRecordTime to uploadPackages
* First cut at updating ApiPackageManagementService
* Update tests, wire through the packageEntries
* Don't extend IndexPackagesService in InMemoryPackageStore
It does not implement the full interface and it isn't used
directly as one anyway.
* Drop maximum_record_time from package_management_service
Adding maximum record times touches the whole stack. Leaving
this change to another PR.
* Wire through the removal of maximum_record_time.
And remove dead code from InMemoryKVParticipantState
* Remove checking for duplicate package uploads
This aligns with the behaviour of WriteService.
* Reformat
* Fix PackageManagementService after adding of submission_id to the service
* grpc-definitions: Delete health_service.proto
We can use the version in io.grpc:grpc-services instead.
* ledger: Delete ledger/API.md.
* sandbox: Fix warnings in ApiServices flagged by IntelliJ.
* sandbox: Implement a dummy grpc.health.v1.Health.Check endpoint.
* sandbox: Implement a dummy grpc.health.v1.Health.Watch endpoint.
* sandbox: Drop repeated elements from grpc.health.v1.Health.Watch.
* sandbox: Wrap the HealthService in basic tests.
* sandbox: Stop streaming the server health too.
* ledger-api-test-tool: Health check tests.
* Add a changelog entry for the health check endpoints.
CHANGELOG_BEGIN
- [Ledger API] Add healthcheck endpoints, conforming to the
`GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
It is always ``SERVING`` for now.
- [DAML Ledger Integration Kit] Add conformance test coverage for the
``grpc.health.v1.Health`` service.
CHANGELOG_END
* ledger-api-integration-tests: Increment the number of services.
* Apply suggestions from code review
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: Use `AkkaBeforeAndAfterAll` in the HealthServiceSpec.
In an attempt to get it working on CI.
* sandbox: Change `dropRepeated` to `DropRepeated()`.
Keep it in one file.
* test-common: Use `Delayed.by` in `TimeBoundObserver`.
* test-common: Close the source when `TimeBoundObserver` completes.
* ./fmt.sh
That'll teach me not to `--no-verify` just because it's a merge commit.
* sandbox: Inline `HealthService.suppress`.
At some point it was being used twice.
* sandbox: Increase the timeout for HealthServiceSpec.
* sandbox: Reimplement HealthService using the Scala protobuf types.
* sandbox: Generate an Akka-compatible trait for the health service.
And refactor a lot of test code to make it easy to test.
* ledger-api-common: Move the HealthService here.
* rs-grpc-testing-utils: Publish to Maven.
* rs-grpc-testing-utils: Add Maven coordinates.
* bazel: 0.28.1 --> 1.1.0
* bazel-watcher sha256
* Fix missing line in patch
* proto_source_root --> strip_import_prefix
See https://github.com/bazelbuild/bazel/issues/7153 for details.
* Update rules_nixpkgs
Required to avoid errors of the form
```
ERROR: An error occurred during the fetch of repository 'node_nix':
parameter 'sep' may not be specified by name, for call to method split(sep, maxsplit = None) of 'string'
```
and
```
ERROR: An error occurred during the fetch of repository 'node_nix':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'nodejs'.
Command: [/Users/runner/.nix-profile/bin/nix-build, /private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/node_nix/nix/bazel.nix, "-A", "nodejs", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
src/main/tools/process-tools.cc:173: "setitimer": Invalid argument
```
* Update rules_scala
* .proto has been removed, use [ProtoInfo] instead
See
https://docs.bazel.build/versions/1.1.0/be/protocol-buffer.html#proto_library
* python3_nix add nix_file attribute
To avoid the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/BUILD:66:1: //:nix_python3_runtime depends on @python3_nix//:bin/python in repository @python3_nix which failed to fetch. no such package '@python3_nix//': Traceback (most recent call last):
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'python3'.
Command: [/home/aj/.nix-profile/bin/nix-build, "-E", "import <nixpkgs> { config = {}; overlays = []; }", "-A", "python3", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
error: anonymous function at /home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs.nix:3:1 called with unexpected argument 'config', at (string):1:1
```
* rules_haskell unnamed string.split(_, maxsplit = _)
The keyword argument may no longer be named.
* string.replace(_, _, maxsplit = _) may not be named
* Move proto sources from deps to data
Fixes
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation
```
* Define sha256 for haskell_ghc__paths
Bazel 1.1.0 fails on missing hashes.
* Disable --incompatible_windows_native_test_wrapper
* //compiler/daml-extension don't modify sources
Modifying sources in-place can cause issues on Windows, where build
actions are not sandboxed and changes on sources can affect other build
steps.
* bazel-genfiles --> bazel-bin
The bazel-genfiles symlink has been removed since Bazel 1.0.
See https://github.com/bazelbuild/bazel/issues/8651
* Mark dev_env_tool repository rule as configure
See
https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule
* Move data deps into data attribute
* Mark dev_env_tool as local = True
* Manually fetch @makensis_dev_env
* Pre-port cleanup of helpers
* Further clean-up
* Remove dead code
* More cleaning and flattening
* Port first tests from CommandClientIT
* More cleanup and more dead code removal
* Port exercise by key test
* Port observer disclosure tests
* Port tests, remove duplicates and dead code
* Fix compilation error and remove more dead code
* Remove unused import
* Spread helpers to single users, remove materializer from helpers
* More test porting and cleanup
* Remove more dead code
* Restore tests deleted by mistake
* Restore final qualifier removed by mistake
* Fix typo
* Optimize imports
* Move timeout from helper to tests
* Remove unused class
* Never gonna give you up
* Address https://github.com/digital-asset/daml/pull/3320#discussion_r342018581
* Address https://github.com/digital-asset/daml/pull/3320#discussion_r342025161
* Remove removed test :)
* Update bazel-common to fix javadoc issues
Specifically, to fix the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel:7:1: in javadoc_library rule //ledger-api/rs-grpc-bridge:rs-grpc-bridge_javadoc:
Traceback (most recent call last):
File "/home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel", line 7
javadoc_library(name = 'rs-grpc-bridge_javadoc')
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/com_github_google_bazel_common/tools/javadoc/javadoc.bzl", line 27, in _javadoc_library
dep.java.transitive_deps
object of type 'JavaSkylarkApiProvider' has no field 'transitive_deps'
```
* Define Maven deps using rules_jvm_external
* Pin artifacts
* Remove bazel-deps generated targets
* Remove bazel-deps
* Switch to rules_jvm_external targets
* update bazel documentation
* pom_file: There are no more bazel-deps targets
* BAZEL-JVM.md `maven_install` typo
This removes the need for clients to handle the special case where the
stream might be empty.
Now the clients can always assume that they receive at least one
response element in the stream.
* stack setup for HLB and Nim
* build_packages.sh and daml-ledger.cabal
* update README
* export-package for daml-ledger
* export-package, run here
* find tmp daml-ledger
* update README for review comments
* ref daml instructions
* further improve the process and documentation in response to comments
* have -with-rtsopts match the bazel build
* fixes for Gary
* add Numeric.java
* ledger-api: rename `decimal` field to `numeric` in value protobuf
* Address Gerolf's comment
* ledger-api: add missing renammings
* ledger-api: relax syntax of numbers that can be sent as numerics
* extractor: fix
* leger-api: change format of number though ledger api
* daml-lf: fix numeric regexp
* ledger: fix tests
* New acceptance test tool prototype
* Fix transaction filter helper
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r308081386
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r308082432
Moreover, creates specific helpers for test DAML code.
* Move helpers from session to context
Remove a circular dependency between session and context, and limits the
usage of implicits.
* Run tests with ledgers out of process
* Prepare for drop in
* Temporary timeout increase
* Move bulk of the logic to the test runner
* Have a map of tests for CLI integration
* Timeout per-test, runner uses context, renaming
* Allow usage of time service, integrate semantic tests
* Integrate CLI SSL configuration
* Integrate failure expectation CLI option
* Integrate with failure output verbosity CLI option
* Integrate with test listing and picking CLI options
* Integrate with timeout scaling CLI option
* Integrate with command TTL CLI option
* Review CODEOWNERS
* Drop previously unimplemented CLI parameters
* Expand test todo list to encompass latest additions
* Deduplicate some testing infrastructure, swap in new test tool
* Remove unused rule
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311045444
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311051974
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311035287
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311055290
* WIP
* Make gRPC thread factory spawn non-deamon threads
* Remove debug prints, complete divulgence test (missing assertion messages)
* Try to detect assertion failure line number
* Add Sandbox binary that starts an ephemeral postgres instance
This allows us to use the client_server_test bazel macro to run against
the sandbox backed by postgres.
Fixes#1543
* Add Sandbox binary that starts an ephemeral postgres instance
This allows us to use the client_server_test bazel macro to run against
the sandbox backed by postgres.
Fixes#1543
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r311400695
* Remove TlsConfiguration duplicate, move original to commons
* Make assertion more readable
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Fix build errors introduced in previous commit
- exclude time service tests from reference-v2
- add copyright header to TlsConfiguration
* Complete divulgence test and address review comments
Restores comments from original divulgence tests and strives to be as
close to them as possible.
- Address https://github.com/digital-asset/daml/pull/2282#discussion_r311429169
- Address https://github.com/digital-asset/daml/pull/2282#discussion_r311431156
- Address https://github.com/digital-asset/daml/pull/2282#discussion_r311431913
* Delete old divulgence tests
* Restore methods to fetch contracts by template identifier
* Run conformance tests on in-memory sandbox
* Run conformance tests on the persistent sandbox
* Remove workflow identifier workaround
* New contract keys test (#2452)
* Set channel type for LedgerSession
* Add ContractKeysIT
* Run tests against reference-server v2
* Removed the old ContractKeysIT. However, ...
... I cannot remove ContractKeysChecks completely, because we have
another test path for DARs created prior to DAML-LF 1.6, which doesn't
have the submitter == maintainer restriction for contract keys.
* Use codegen instead of custom code
* Add missing copyright headers
* Drop sandbox conformance test suite (seems to not work on Windows CI)
* Make time simpler
* Test tool tests to run only semantic tests
* Make CI behave more consistently
* Add missing files to run tests on Postrgres
* Fix flaky divulgence test, introduce readable identifiers
* Increase sandbox conformance tests timeout
* Address https://github.com/digital-asset/daml/pull/2282#discussion_r312363246
* Reintroduce test resource extraction
* Read semantic tests from the classpath rather than Bazel
* Use codegen types for ContractKeys integration test
allocateParty returns Primitive.Party and create and exercise also
accept Primitive.Party. This way we don't have to keep a String and
Primitive.Party representation around.
* Bump timeout for sandbox conformance tests
* Remove dependency on Bazel runfiles
This is a breaking change on the ledger api. So far we could rely on the
assumption that contractId == eventId. This assumption doesn't hold
anymore in a daml-on-x setting, where the eventId is created by the
api server, but the absolute contractId is created by the ledger
implementation.
Instead of going through weird contortions to store the relevant data in
the existing database schema, it is more viable in the long term to
remove that field and instead provide facilities to lookup transactions
by contractId.
Contributes to #2068.
Moves party allocation to its own WriteService trait (similar to package upload).
Changes uploadDar to uploadPackages, so that the write service does not depend on the DAR file format.
Adds or improves many comments.
* store archive size in `DarReader`
* rename `SandboxTemplateStore` to `SandboxPackageStore`
* store package info in `SandboxPackageStore`
* introduce package upload / read to write / index services
not tested yet, just a ton of plumbing
* WIP test the package service
* Fix build errors after rebase
* Move packages service to v2
* Ledger API client uses ledger API types
* Fix ReflectionIT
* Correctly handle uploading invalid dar files
* Fix reading DAR entry file sizes
* Improve package management IT
* Improve handling of duplicate packages
* Fix language-support build
* Use unique party and command names
* Rename lfpackage to language
* Rename SandboxPackageStore to InMemoryPackageStore
* Remove getCurrentTime ledger method
* Improve package management IT
* Move InMemoryActiveContracts and InMemoryPackageStore
* Use case object for UploadDarResult.Ok
* Address review comments
* Update release notes
Fixes#1311
* Use BazelRunfiles in test
* Fixes#1204: Release bindings and codegens to Maven Central.
Upload the Java and Scala Bindings with the respective code
generator binaries to Sonatype Open Source Repository
Host for synchronization with Maven Central.
* bazel build
* Moving it into ledger-api/http-json.
application-api module name does not make sense, API stands for
application programming interface... so application application
programming interface? Nah.
* Allow slashes in ledger strings
Using '/' as the divider is a natural choice for some ledger backend
implementations and in application/workflow etc. identifiers.
* Update value.proto to include slash in LedgerStrings
* Update release notes
Fixes#1312
This PR adds support for party management in the sandbox:
Both the in-memory and the SQL backend track a list of known
parties, and implicitly add any party mentioned in a transaction.
New calls were added to the IndexService and the WriteService
for managing parties. These calls are wired to the above mentioned
persistence backends, and to a new API service.
A corresponding client interface was added to the scala API client.
An integration test was added for checking that a call to allocate a
party succeeds.
An integration test for the sandbox was added for checking that the
sandbox implicitly adds all parties mentioned in a transaction.
Add tTansactionServiceIT into Ledger API Test Tool as an optional test.
Available tests can be listed with the --list option.
Tests can be included with --include, and excluded with --exclude.
Fixes#1372, #1472
