* Use TreeMap for storing transaction nodes
Going forward we require deterministic serialization for transaction nodes.
This PR switches the `Transaction.nodes` to use a TreeMap instead of HashMap,
which gives us deterministic serialization order for the transaction nodes.
An alternative would be to sort the transaction nodes when serializing, but
I feel avoiding HashMap altogether is the safer option as this may bite us
elsewhere as well (e.g. tests, new serialization code, etc.).
* Address review
* Remove unused import
The contract keys accessed in the context of a `fetchByKey` were not
properly checking that the contract was active and visible.
The checks for contract's visibility and activeness are now done from
one place.
* address warning: add type annotations, shortened syntax, optimise imports
* Specify test size to avoid bazel warning
address warning: add type annotation
* make bazel buildifier happier with call order
* no parens for a no side effect pure method
* Implement proper stream pagination
The previous pagination mechanism for streaming ledger entries was implemented
as a recursive method call with manually concatenating akka Sources.
However, this didn't work properly and resulted in all the subsources being
forced immediately, resulting in parallel requests for all pages
(0-100, 100-200, 200-300, ...) instead of the expected mechanism of loading
the first page and on upstream request (i.e. client requests for more data
over grpc) loading the next page of data.
The current mechanism uses Source.unfoldAsync which makes the paging
mechanism work as expected: sequential loading of pages on demand.
* Serialize and deserialize Transactions outside the SQL Executor
This frees up the SQL Executor threads sooner for other work
and the conversion work only happens on a by need basis when
the consumer requests more data.
* Refactor proposal for "KeyValueCommitting"
This only changes the package uploads as an example.
Embrace the imperative style:
* we carry a lot of state through the commit steps anyway and immutable maps are actually
dangerous: we may end up reading from the wrong one etc (has happened).
* performs much better with transactions that have thousands of outputs (the current monadic commit would
have quite a lot of overhead for setting all these).
Drop the "Option[DamlStateValue]". This was only for dedup value and the extra complexity for catching the case
where some inputs weren't fetched isn't worth the complexity (tests will catch this anyway).
Return an Iterable[(DamlStateKey, DamlStateValue)]. This way we're clear about the ordering of the outputs.
* Address review and simplify
* Port the change to use daemon thread for preload executor
So that JVM doesn't get stuck when exiting.
* Fix typo
* Add client_server_build and integrity_test rules
And use them to implement ledger dump of the reference
server and to check it.
* Only build and test ledger dump on Linux. Only run tests relevant to dump.
* Make client_server_build quiet in happy path
* Reformat
* Remove unnecessary runfiles for client_server_build
* Fix the import paths of {value,transaction,blindinginfo}.proto
This cleans the imports in daml_kvutils.proto:
-import "daml-lf/transaction/src/main/protobuf/transaction.proto";
+import "com/digitalasset/daml/lf/transaction.proto";
This shouldn't be a breaking change to the outside as the proto files
themselves are not published, only the java libraries generated from them.
* Update missed deps
* Make Err an abstract class so it is throwable when used from Java
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/Err.scala
bring back sealed
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
kvutils now supports dumping the ledger to a file via environment
variable: "KVUTILS_LEDGER_DUMP=/tmp/ledger.dump".
The integrity checker tool allows re-processing of ledger dumps to
validate that all submissions, state and log entries can be parsed
and processed. This forms the basis of kvutils data continuity test
suite.
Currently the checker strictly enforces that re-processed
outputs match the inputs exactly, which for now guarantees forwards
and backwards compatibility. The checker will be adapted when
forwards-incompatible changes or new kvutils message versions arise.
A select collection of ledger dumps will be maintained in a separate
repository (to be decided) and these dumps are to be validated by
the DAML CI.
* Update bazel-common to fix javadoc issues
Specifically, to fix the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel:7:1: in javadoc_library rule //ledger-api/rs-grpc-bridge:rs-grpc-bridge_javadoc:
Traceback (most recent call last):
File "/home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel", line 7
javadoc_library(name = 'rs-grpc-bridge_javadoc')
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/com_github_google_bazel_common/tools/javadoc/javadoc.bzl", line 27, in _javadoc_library
dep.java.transitive_deps
object of type 'JavaSkylarkApiProvider' has no field 'transitive_deps'
```
* Define Maven deps using rules_jvm_external
* Pin artifacts
* Remove bazel-deps generated targets
* Remove bazel-deps
* Switch to rules_jvm_external targets
* update bazel documentation
* pom_file: There are no more bazel-deps targets
* BAZEL-JVM.md `maven_install` typo
* Add failing test case for transient contract keys
* Add kvutils tests for transient contracts and keys
* Fix handling of transient contracts and keys in kvutils
* Fix contract key uniqueness check in kvutils
Archival of a contract with a key and recreation within the
same transaction is now allowed in kvutils.
* Add assertions to check that new contract has been created with same key
* Preload packages to engine during upload
* Improve logging in KeyValueCommitting and add timing information
* Fix scenario service tests now that logging is done in interpreter
* Rename referencedContracts to divulgedContracts
We do not want to provide all referenced contracts as that would
require the ReadService to be able to read all those contracts, or
we would need to bundle the referenced contracts with the transaction.
The new type should match what was discussed in #2488.
* Add changelog entry to participant.state.v1.Version
* Remove divulgedTo from DivulgedContract
We decided not to prematurely add this feature.
This allows using the dispatcher in situations where the index is common,
but different data streams are materialized from it.
The practical reason for this is to allow using a single dispatcher to read
from both the ledger_entries and configuration_entries tables, which share
the same ledger offset construction.
The time model is being reworked and the current stack is not prepared
to handle this additional check well.
We will reimplement time model related checks once the redesign has been
completed.
The kvutils envelope is meant to be used whenever a kvutils produced
data is stored and transmitted over the network. It adds both versioning
and compression to the original message.
* Initial version for configuration submissions and authorization
* Refactor config submission based on review. Add tests.
* Cleanup test output and comments
* kvutils test-suite
- Add unit tests for kvutils
- Thread the input state into Commit monad
- Fix a bug in handling rejected transactions (from previous refactoring)
* Address PR review
- Add submissionId to configuration changes.
- Revert the dedup of the ParticipantNotAuthorized message. This did not buy much
and we need to duplicate it anyway when modularizing kvutils.
* kvutils refactoring
Move transaction submission processing to its own class and refactor to
stop early on errors.
Throw a known error rather than use sys.error.
* Refactor package and party allocation
* Introduce "Commit" computation abstraction
This also fixes an issue where the command dedup entry was not created
when a later validation failed and a rejection log entry was created.
* Drop input_log_entries. Add contract instance to DamlContractState.
* Add copyright headers
* Add KeyValueCommitting.submissionOutputs
This cleans up implementations that need to know the outputs of a submission
up-front.
* Support for contract keys to kvutils
* Fix handling of CreateEvents with contract keys in semantic tester
* Add ContractKeysIT to tests and fix issues related to fetchByKey
* Address code review
* make every PublicPackageUploaded update contain exactly one package
* implement update subscriptions that start with offsets with the same major number
* address review comments
* connenct LedgerAPI party and package management with KVUtils
* formatting
* address review comments
* add participantId parameter to sandbox and indexer
* annotate TODO's with github issue numbers
* Add external offset column to parameters
We only ever need to use the external offset when restarting the api
server from a persistent data store to resume the ReadService state
update stream from the last processed offset.
In the PostgresIndexer we then need to store the external offset while
adding a new ledger entry.
* Allow disabling the jmx reporter in metricsmanager
* Implement PostgresIndexer component
* Introduce read-only Ledger and Dao
* SandboxEventIdFormatter treats transactionId as opaque string
* Introduce LedgerBackedIndexService
This can be reused both in the Sandbox and the ApiServer
* Implement PostgresIndex component
* Reference Service based on kvutils-v2 and postgres(index|indexer)
* Disable conformance test for ReferenceServer
* provide asynchronous interfaces for party and package management on the participant-state
* provide submission id for correlating req with resp on Write- and ReadService
* add asynchronous party allocation in KVUtils
* start conversion of the package upload and party allocation to synchronous paradigm
* implement synchronous versions of party allocamtion and packacge update
* document recent changes and unify concepts between v1 and v2 versions of
the participant-state interface
* scala formatting
* address review comments
Moves party allocation to its own WriteService trait (similar to package upload).
Changes uploadDar to uploadPackages, so that the write service does not depend on the DAR file format.
Adds or improves many comments.
* Introduce contract key to exercise node
Track the contract key in NodeExercises. This is in
preparation for adding the key into exercise events, and
for support for contract keys in kvutils, where we need
to be able to compute the outputs of the submission, e.g.
the contract key that would be unset by a consuming exercise.
This introduces transaction version 8.
* Move release note on contract keys in exercises to unreleased.rst
* Address Stephen's review
- Drop changelog entry for this change as it is not user facing
- Add info on version 8 to transaction.rst. Fix typo in version 7.
- Reorder VersionTimeline as per review
- Only compare the NodeExercise 'key' if it is set in the original
transaction (in order to compare transactions in a backwards
compatible manner)
* Unbork unreleased.rst
* Add missing domain.RejectionReason cases
* Use java.time.Instant instead of damlf Timestamp
* Make applicationId, submitterParty, commandId optional in LedgerEntry.Transaction
