Fixes#3513.
* Add metrics to track commands
- Meter: CommandSubmission.failedCommandInterpretations
- Timer: CommandSubmission.submittedTransactions
* Add metrics to JdbcIndexer
- Gauge: JdbcIndexer.lastReceivedRecordTime
- Gauge: JdbcIndexer.lastReceivedOffset
- Gauge: JdbcIndexer.currentRecordTimeLag
- Timer: JdbcIndexer.processedStateUpdates
* Allow setting a custom JMX domain.
We don't want to use com.digitalasset.platform.sandbox
for the ledger-api server and indexer.
* Add interceptor that automatically meters incoming grpc service calls.
CHANGELOG_BEGIN
- [Ledger] Add various metrics to track gRPC requests, command submissions, and state update processing. See `#3513 <https://github.com/digital-asset/daml/issues/3513>`__.
CHANGELOG_END
* add error codes for overloaded and internal error
* change wording in error description strings
* scala formatting
* use string interpolation intead of concatenation
* address issues found in peer review
* Check internal dependencies of maven artifacts
All artifacts that are uploaded to maven need to also have
their dependencies uploaded to maven.
* inline tuple matching
* remove unnecessary parenthesis
* mark missing artifacts for maven upload
* ledger-api-test-tool-on-canton: Run Canton.
* ledger-api-test-tool-on-canton: Run the SemanticTests against Canton.
This was _so_ much work.
These will probably be flaky, so can't merge them in until we fix the
underlying issues with the tests around multi-participant allocation.
* ledger-api-test-tool: Wait for parties to arrive on all participants.
Thanks to Canton for exacerbating this bug.
Can't turn on all the tests on Canton yet as there are other spurious
failures. Will investigate next.
* ledger-api-test-tool: Move all contract key tests to ContractKeys.
If a ledger doesn't support ContractKeys, they need to be able to turn
these tests off.
* ledger-api-test-tool: If a test name is misspelled, fail immediately.
I keep getting names slightly wrong in the `--exclude` arguments and
wondering why it didn't work.
* client_server: Revert client_server_test.bzl.
Turns out I don't know what I'm doing. ¯\_(ツ)_/¯
* sandbox: Don't call `.toString` on an array. Doesn't do much.
* dev-env: Don't untar netcat automatically.
It needs to be installed with Pacman.
* Use TreeMap for storing transaction nodes
Going forward we require deterministic serialization for transaction nodes.
This PR switches the `Transaction.nodes` to use a TreeMap instead of HashMap,
which gives us deterministic serialization order for the transaction nodes.
An alternative would be to sort the transaction nodes when serializing, but
I feel avoiding HashMap altogether is the safer option as this may bite us
elsewhere as well (e.g. tests, new serialization code, etc.).
* Address review
* Remove unused import
* address warning: add type annotations, shortened syntax, optimise imports
* Specify test size to avoid bazel warning
address warning: add type annotation
* make bazel buildifier happier with call order
* no parens for a no side effect pure method
* Implement proper stream pagination
The previous pagination mechanism for streaming ledger entries was implemented
as a recursive method call with manually concatenating akka Sources.
However, this didn't work properly and resulted in all the subsources being
forced immediately, resulting in parallel requests for all pages
(0-100, 100-200, 200-300, ...) instead of the expected mechanism of loading
the first page and on upstream request (i.e. client requests for more data
over grpc) loading the next page of data.
The current mechanism uses Source.unfoldAsync which makes the paging
mechanism work as expected: sequential loading of pages on demand.
* Serialize and deserialize Transactions outside the SQL Executor
This frees up the SQL Executor threads sooner for other work
and the conversion work only happens on a by need basis when
the consumer requests more data.
* Divulged contract visibility in multi-participant environments #3351
* Ledger api server time service optional ability for testing #3225
* Allow ledger api server to share DAML-on-X DAML engine #2975
* Allow ledger api server participant ids with LedgerString chars #3327
* Ledger api server includes SQL description in errors #3324
* ledger-api-test-tool: Use Scala's `Duration` for timeouts.
* ledger-api-test-tool: Remove the parentheses from a getter.
* ledger-api-test-tool: Run tests sequentially.
Using Akka Streams to manage the concurrency.
* ledger-api-test-tool: Parallelize across available processors.
* sandbox/reference-v2: The test timeout scale factor is no longer needed.
Now that we don't run all tests at once, the measurements are much more
accurate.
* sandbox/reference-v2: Run LotsOfPartiesIT 11 times instead of 16.
* ledger-api-test-tool: Highlight slow tests in yellow and red.
* ledger-api-test-tool: Fix some warnings in the Reporter.
* ledger-api-test-tool: Sort the tests afterwards for better concurrency.
Turns out that Akka Streams tries to maintain order at the expense of
running more things, which is not ideal. This is a hack to let us use
`.mapAsyncUnordered` and then re-sort into the original order at the
end.
Ideally `.mapAsync` would detect when subsequent futures have finished
and start running more, even if the first hasn't finished yet.
* ledger-api-test-tool: Inline `availableProcessors` into Config.
* new library ledger-service/db-backend
* borrow contracts table schema from extractor
* borrow contract insertion, removing some data to be unused
* match contract schema with insert function
* factor insertContract arguments
* offset table declarations
* CLI argument for query store
* surrogate template IDs
* compute surrogate template IDs on-the-fly
* database init action
* incoherent typeclasses, eh
* newtype SurrogateTpId
* offset fetch/update functions
* bad sql
* bulk insert contracts, function for selecting contracts
* expose contract column name for query's usage
* Initializing DB on startup if configured
* dropping existing tables as part of initialization
* fix some query syntax errors
* createSchema flag
* function for streaming transactions with jwt party selected
* formatting
* usage
* collect acs contracts and the ledger offset at the end
* lastOffset
* fixing merge conflicts, updating the way 3rd party deps are specified
* Moving ContractDao into http-json module
so it can take domain AST as an input
* cleanup
* injecting new dependencies
* split transaction batches into inserts and deletes
* generate sql for deleting contracts
* `fetch_sources = True` for java_deps
* make the delete-constructed fragment more efficient; handle empty list here
* pass logHandler for insertContracts
* ContractDao returns ConnectionIO, it's up to the caller to wrap query into a transaction
* fixing typo
* minor cleanup, moving fromLedgerApi factory function into corresponding companion objects
* don't need it any more
* GetActiveContractsResponse => domain.Contract factory
* make concatFragment private
* add partition graph; move other contract-fetching experiments to ContractsFetch
* experimenting with akka sources
* introducing domain.Offset to work around API's empty/null offset cases
* minor cleanup
* decompose fetchActiveContractsFromOffset
* missed via
* ACS splitting graph
* finish doc for ACS splitting graph
* remove unneeded stages
* WIP
* lazily read a stream of ConnectionIO into a single ConnectionIO
* cancel on IO error
* figuring out how to put all the pieces together
* graph WIP
* Removing workflowId from the JSON API
* simplify acsAndBoundary; describe other flow pieces
* WIP
* use Vector in InsertDeleteStep; add variant for ACS (no deletes)
* `org.wartremover.warts.NonUnitStatements` enforced in `http-json` module
* evaluate InsertDeleteStep to a ConnectionIO
* database variant of LfValueCodec, using numbers for numbers
* convert input to JSON, combine insert plans, connect rest of contractsToOffsetIo
* remove strict contractsToOffset sink
* moving dao methods into an object
* putting pieces together
* contractsFromOffset WIP
* should be it
* cleanup
* cleanup
* contractsIo that takes List[domain.TemplateId.RequiredPkg])
* contractsIo that takes List[domain.TemplateId.RequiredPkg])
* cleanup
* put all pieces together, testing
something does not work yet
* diff is not required to return anything
that is why Sink.lastOption that gives Option[domain.Offset]
* factor out tuple split
* use traverse syntax in contractsIo2
* factor explicit flow steps out of graph DSL; remove aggregate
* locally model the Absolute/Begin distinction for offset bookmarking in DB
* Adding test cases to run HTTP Service with Postgres backend
the same set of test cases, run with and without DB backend
* make better use of domain.Offset in OffsetBoundary
* monomorphize InsertDeleteStep#append
* Disabling a test that fails with DB backend
* add release note
* add release note about workflowId
* a test case that checks the number or stored contracts
* trying to figure out why Postgres test fails on Windows with NPE
* ledger-api-test-tool: Port LotsOfPartiesIT.
* ledger-api-test-tool: Run LotsOfPartiesIT with multiple participants.
Odds of me getting this wrong: 80%. Let's find out!
* ledger-api-test-tool: More underscores, methinks.
* ledger-api-test-tool: Improve the readability of the LotsOfParties test.
Instead of all the functional magic, just do things twice. It's even
shorter in some cases.
* ledger-api-test-tool: Not so many parties. :-(
* sandbox: Extract duplication between conformance test definitions.
* sandbox: Run the LotsOfPartiesIT conformance test separately.
* Revert "ledger-api-test-tool: Not so many parties. :-("
This reverts commit 8039a7cc9b.
* sandbox: Add the copyright header to sandbox_conformance_test.bzl.
* Run `./fmt.sh`.
* ledger-api-test-tool: Generalize `sandbox_conformance_test`.
I want to use it with the DAML-on-X reference v2.
* api-server-damlonx/reference-v2: Use `server_conformance_test`.
* api-server-damlonx/reference-v2: Run LotsOfPartiesIT separately.
* Pre-port cleanup of helpers
* Further clean-up
* Remove dead code
* More cleaning and flattening
* Port first tests from CommandClientIT
* More cleanup and more dead code removal
* Port exercise by key test
* Port observer disclosure tests
* Port tests, remove duplicates and dead code
* Fix compilation error and remove more dead code
* Remove unused import
* Spread helpers to single users, remove materializer from helpers
* More test porting and cleanup
* Remove more dead code
* Restore tests deleted by mistake
* Restore final qualifier removed by mistake
* Fix typo
* Optimize imports
* Move timeout from helper to tests
* Remove unused class
* Never gonna give you up
* Address https://github.com/digital-asset/daml/pull/3320#discussion_r342018581
* Address https://github.com/digital-asset/daml/pull/3320#discussion_r342025161
* Remove removed test :)
* Fix the import paths of {value,transaction,blindinginfo}.proto
This cleans the imports in daml_kvutils.proto:
-import "daml-lf/transaction/src/main/protobuf/transaction.proto";
+import "com/digitalasset/daml/lf/transaction.proto";
This shouldn't be a breaking change to the outside as the proto files
themselves are not published, only the java libraries generated from them.
* Update missed deps
The global implicit disclosure map was missing the `#` prefix for contract ids, which caused subsequent contract id lookups to not find the right contract.
Instead of loading and deserializing the transaction, we store the
stakeholders in tables contract_signatories and contract_observers.
To avoid having to do an extra roundtrip for fetching the stakeholders,
we aggregate the parties at query time into a % separated string (% is
not a valid character for parties), which we then split again in the
application.
Fixes#2330.
* Update bazel-common to fix javadoc issues
Specifically, to fix the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel:7:1: in javadoc_library rule //ledger-api/rs-grpc-bridge:rs-grpc-bridge_javadoc:
Traceback (most recent call last):
File "/home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel", line 7
javadoc_library(name = 'rs-grpc-bridge_javadoc')
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/com_github_google_bazel_common/tools/javadoc/javadoc.bzl", line 27, in _javadoc_library
dep.java.transitive_deps
object of type 'JavaSkylarkApiProvider' has no field 'transitive_deps'
```
* Define Maven deps using rules_jvm_external
* Pin artifacts
* Remove bazel-deps generated targets
* Remove bazel-deps
* Switch to rules_jvm_external targets
* update bazel documentation
* pom_file: There are no more bazel-deps targets
* BAZEL-JVM.md `maven_install` typo
* speedy: limit allocation of trivial SValue constants
* daml-lf: limit allocation of constants for Value, Exp, Type
* weaken numeric equality in ledger service test
* Properly fill eventId for active contracts
This gets rid of the last remaining bit that assumes
contractId==eventId.
Fixes#65.
Contributes to #2068.
* Do not conflate eventId and contractId in the daml-lf interpreter
* Do not treat contractId as eventId in Ledger.scala
* Remember the transaction that divulged a contract.
* In this scope we can treat divulged contracts the same as disclosed ones
* revert a few more syntactical changes to make the overall diff smaller
* retain the same behavior on the scenario service api
* fix unreleased after rebase
Since we still have issues with parallel writes when using H2,
we should properly only use a single DB connection and executor thread
for that.
Because we didn't do that before for Sandbox, tests like PackageManagement
appeared to be flaky due to racy inserts in H2.
