* participant-integration-api: Move test constants to a companion object.
TelemetrySpecBase doesn't need to expose those through inheritance.
* Avoid mocks in the configuration management service test.
* Add tests for `ConfigManagementService.GetTimeModel`.
* Add tests for `ConfigManagementService.SetTimeModel`.
* Disable configuration management until the ledger has a configuration.
CHANGELOG_BEGIN
- [API Server] The configuration management service previously returned
the participant-specified default configuration if none was found on
the ledger. This can be misleading, and so the ``GetTimeModel``
endpoint now returns a gRPC ``NOT_FOUND`` error if no configuration
has been found yet. Similarly, the ``SetTimeModel`` endpoint returns a
gRPC ``UNAVAILABLE`` error.
This should only happen if the participant gives up waiting for the
ledger to provide a configuration, which is very unlikely in a
production setting. In this case, the ledger will also not respond to
command submissions.
CHANGELOG_END
* participant-integration-api: Extract some test variables for clarity.
In ApiConfigManagementServiceSpec.
* sandbox: Improve some variable and parameter names in tests for clarity.
* Improve the error message in case of a missing ledger configuration.
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* ledger-api-common: Improve an error status constructor name.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* [Mutable cache] Resolve with full lookup on missed cache thread-throughs
* Do not store negative lookups
* Metrics for counting divulgence and full lookups
* Metrics for counting read-through not found
CHANGELOG_BEGIN
CHANGELOG_END
* Added comment detailing secondary lookups for divulgence
* Apply suggestions from code review
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
* [DOCS] Add documentation for the JSON API metrics
changelog_begin
- [JSON-API] You can now find a section `Metrics` in the http-json api documentation explaining how to enable metrics and which are available
changelog_end
* Fix rst build warnings
* Update docs/source/json-api/metrics.rst
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Adapt metrics doc to state that it IS an exhaustive list and remove wrong copy pasta text & add info about prometheus
* Update the legal values for the metrics reporter cli option
* shorten the description, the change prior was unnecessary ._.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Resolve contract keys conflicts in disclosed contracts (#9948)
* build queries for nullifications of past key
* format code
CHANGELOG_BEGIN
In case a contract key is already present in a past contract in the contract table nullify it. This gets rid of contract keys of previously disclosed contracts. We never discover that disclosed contracts get archived, so we get conflicts on such past keys
CHANGELOG_END
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* correct oracle implemantation (new)
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
changelog_begin
- [JSON-API] Timing metrics which measure how long the processing of a command submission request takes on the ledger are now available
changelog_end
changelog_begin
- [JSON-API] The database operations (regardless of in-memory or postgres) contain now metrics for fetching contracts by id or key (seperate metrics foreach)
- [JSON-API] The time for a repsonse payload construction of a request is now also tracked in a metric
changelog_end
* [JSON-API] Add more timing metrics
changelog_begin
- [JSON-API] Timing metrics are now available for party management, package management, command submission and query endpoints.
- [JSON-API] Also added a timing metric for parsing and decoding of incoming json payloads
changelog_end
* Add comments to new metrics
* Split metrics up more & remove obsolete metric
* Split up timers for query endpoints
* [JSON-API] Concurrent query etc. metrics
changelog_begin
- [JSON-API] The metrics which describe the amount of these concurrent events is now available:
- running http request
- command submissions
- package allocations
- party allocations
changelog_end
* Rename running metrics to throughput ones & add comments on the metrics
* Adjust names of other existing metrics too, to have for the json api a more consistent metrics naming
* truncate party name for 64 chars for multi-valued submitters to avoid excessively long metrics name
CHANGELOG_BEGIN
[metrics] Limit size of multi party metrics to ease consumption
CHANGELOG_END
* Use just first party for dynamic metrics when there are multiple parties involved
* Introduce metrics in the http-json service
changelog_begin
[HTTP/JSON API]
- metrics reporting can now be chosen via the command line option --metrics-reporter (currently hidden), valid options are console, csv://, graphite:// and prometheus://
- metrics reporting interval can also now be chosen via a command line option --metrics-reporting-interval (currently hidden)
changelog_end
* Move MetricsReporter and it's dependencies into //ledger/metrics
* Restore non-ugly formatting for that one section in Endpoints.scala
* Update ledger/sandbox-common/src/test/lib/scala/platform/sandbox/cli/CommonCliSpecBase.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Hide metrics option for http-json
* Propagate context exception for the parseUri function in MetricsReporter.scala
* Move cliHint value above parseUri function to have a better structure (it's used once before it's defined and once after it's defined, which is weird to me)
* Use better value name than optMr & optFd in cliopts Metrics.scala
* Remove import order changes & whitespace changes
* Revert usage of Nanoseconds for conversion from scala duration to java duration to usage of Seconds
* Shorten hideIfRequested function
* Fix another rearranged import
* Fix another whitespace removal
* Readd metrics cli option to sandbox after refactoring
* Add missing return type annotation for invalidRead in MetricsReporter
* Readd newline in https OptionParser.scala
* Remove unecessary import
* Update ledger/sandbox-common/src/main/scala/platform/sandbox/cli/CommonCliBase.scala
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Align setter & config name for metricsReportingInterval setting too in CommonCliBase.scala
* Rename http_json_api in Metrics.scala of metrics project to HttpJsonApi
* Reformat CommonCliBase.scala of sandbox-common project
* Fix CommonCliSpecBase test of sandbox
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* CHANGELOG_BEGIN
Add metrics counting the elements in transaction, transaction-tree, completion and acs streams
CHANGELOG_END
* format scala
* address review comments
* Removes TODO related to BatchingParallelIngestionPipe type: the result of the ingestion is not needed at the moment
* Removes TODO related to groupedWithin: at this point the introduced latency seems not significant
* Refactor AsyncSupport
* Removing some TODOs related to postponed work-items
* moving mutable initialisation code does not make sense without refactoring/segregating the initialisation part of the LedgerDao. That effort is postponed as per group decision.
* Injecting the dispatcher makes little sense at this point, further features might require it.
* Fixes type derivation in ParallelIndexerFactory
* Moves instrumentedBufferedSource to metrics (+ unit tests)
* Removes TODO related to logging: primarily to pass down the initializing LoggingContext is fine. (further logging discussion is in progress related to ingestion at the time of writing)
* Refactor type cast towards synthetic zero-element generation
* Expose configuration for the instrumented input, lowers default.
* Remove TODO related wiring of indexer HealthChecks: there is already an issue for this: https://github.com/digital-asset/daml/issues/2733
* Remove TODO related to lowering connectionTimeout for Hikari database-connection pool: it is already the minimum, no smaller connection timeout is available for selection.
changelog_begin
changelog_end
Motivation of this PR: with the help of this combined method, wiring of the
append-only schema ingestion becomes possible.
* Add proper implementation for dao
* Adapts sandbox-classic usage
changelog_begin
changelog_end
The intent of this change is to make the first step in this direction
in order to support dpp-336 work of sanbdox-classic integration by
injecting DAO functionality via this interface.
Level of quality is still pre-production, hence TODO comments.
Planned next step: move StorageBackend interface, and implementation
to platform/store, to it's final place.
* Introduce StorageBackend interface
* Decouple event-seq-id assignment logic from storage specific batching
* Pull out of batching step from input mapping, execution too
* Switch to stateless DAO functions
* Switch to DBDispatcher instead of custom JDBC Connection pool
* Introduce/adapt metrics
* Naturally extend configuration
* Move RunningBatch layer to ParallelIndexerFactory
* Remove dead code
changelog_begin
changelog_end
* Add unit tests for Telemetry
CHANGELOG_BEGIN
CHANGELOG_END
* Make Tracers injectable, improve the TelemetryContextSpec, add a metrics-test-lib package
* Change the TelemetryContext to use SpanAttributes instead of raw AttributeKeys
and remove an unused object along with its dependencies
CHANGELOG_BEGIN
CHANGELOG_END
* Change the Tracer name and the instrumentation name from participant to com.daml.telemetry
* Add telemetry classes from the oem integration kit and use it for command submissions
* Change submitTransaction to submitTransactionWithTelemetry and add a deprecation
* Fix tests
* Revert "Change submitTransaction to submitTransactionWithTelemetry and add a deprecation"
CHANGELOG_BEGIN
- [Integration Kit] TelemetryContext has been introduced to the WriteService.submitTransaction method to support distributed tracing
CHANGELOG_END
* Time lag between contract state events and general dispatcher
* Emit current event sequential id from the indexer.
* Time update registering metric
CHANGELOG_BEGIN
CHANGELOG_END
* Integrate mutable state cache into ReadOnlySqlLedger
* Create tiered ReadOnlySqlLedger for accomodating both legacy and new caching stores
* Integrate event stream lifecycle responsibility into MutableCacheBackedContractStore
* Add missing SQL fragment to ContractsReader fetch query
CHANGELOG_BEGIN
CHANGELOG_END
* Addressed review comments
* ContractStateEventsReader draft based on the POC
* Code formatting
* Moved the contract state event reader as a method to the TransactionsReader.
(required to ease parallel stories development)
* Basic unit tests for reading contract state events
* Removed the dependence on not yet present column.
* Workaround for the lack of 'event_sequential_id' column in the 'parameters' table.
The workaround is intended to be replaced with a proper solution when the append-only schema comes in
* Fixed the unit test for contract state events
* Re-enabled all ledger dao suites
* Included 'JdbcLedgerDaoContractEventsStreamSpec' in the H2 db suites
* Intermediary DTO for contract state events
* Added a comment explaining introduction of the RawContractStateEvent
* Simplified contract state event reading query
* Easier transition for the append-only schema
* Minor refactor
* Minor change
* Ingest contract key for consuming exercise nodes
* CHANGELOG_BEGIN
- [Integration Kit] new streaming query for contract state events
- [Integration Kit] indexing contract keys for consuming exercise events
CHANGELOG_END
* Minor change
* Fixed formatting
* Ingesting key values for consuming exercised events - H2 database
* Addressed review comments
* Moving changes from the 'dao' to the 'appendonlydao' package - ContractStateEventsReader
* Moving changes from the 'dao' to the 'appendonlydao' package - indexing create_contract_key for exercise nodes
* Reduced code duplication - minor
* Reverted changes to the JdbcLedgerDaoSuite
Tests for the ContractStateEventsReader will come in the future
* Restored original naming for the indexed create key
* Improved comments + throwing UnsupportedOperationException
* Added ledger_effective_time to the ContractStateEvent.Created
* Update a metric name
Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>
* Simplified GlobalKey imports
* Added a comment on events stream parallelism level
* Inline event_kind types in the SQL query for contract state events
* A constant for the event sequential id edge case
* Removed unused imports
Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>
* LedgerDao and ContractsReader interface updates for mutable state cache implementation
* Factored the LfValueSerialization cache out of the ContractStore
* Implemented state lookup methods (at valid_at)
This PR does not contain tests for the new lookup methods.
The tests have been extracted in a separate branch and will be merged
when the DAO integration testing suite will be adapted
for the append-only schema.
CHANGELOG_BEGIN
CHANGELOG_END
* Addressed review comments
* Introduce parallel indexer
* Adds parallel indexer in PoC quality
* Adds relevant metrics, and wiring to the parallel indexer code
* Minor fixes to tracing in TransactionsReader
changelog_begin
changelog_end
* Tag all todos
... with 'append-only', so that they are easier to find.
* Refactor metrics
* Remove AverageCounter
Co-authored-by: Robert Autenrieth <robert.autenrieth@digitalasset.com>
* [DPP-142] Explicitly deflate/inflate data outside of the index
* [DPP-142] Explicitly deflate/inflate data outside of the index - review fixes - exposing prepare update parallelism as param
changelog_begin
[Integration Kit] Compression and decompression of stored DAML-LF values
is now executed outside of the index database, allowing to make more
efficient use of the participant resources when indexing.
changelog_end
* Pipelined transaction indexing
CHANGELOG_BEGIN
[Integration Kit] The participant indexer (for PostgreSQL)
can now execute DAML transaction insertions in three pipelined stages.
CHANGELOG_END
* Make participant-integration-api test suite `large` for BAZEL
* Fixed constant timeout for MacOS builds
* Moved ledger end guard to TransactionReader
* Removed TransactionServiceResponseValidator
* Removed MetadataUpdate intermediary level from Update
* Added back store_ledger_entry timer
* Updated comment for idempotent insertions.
* Port more of //ledger/... to Scala 2.13
changelog_begin
changelog_end
* Remove unusued dependency
changelog_begin
changelog_end
* Rename bf to factory to reflect the fact that it’s now a Factory
changelog_begin
changelog_end
* Use regex match instead of sliding string equalityt
changelog_begin
changelog_end
* regex matches are bad
changelog_begin
changelog_end
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.
As announced, this will be merged on Saturday to avoid too many conflicts.
changelog_begin
changelog_end
* Port damlc dependencies to Scala 2.13
I got a bit fed up by the fact that going directory by directory
didn’t really work since there are two many interdependencies in
tests (e.g., client tests depend on sandbox, sandbox tests depend on
clients, engine tests depend on DARs which depend on damlc, …).
So before attempting to continue with the per-directory process, this
is a bruteforce approach to break a lot of those cycles by porting all
dependencies of damlc which includes client bindings (for DAML Script)
and Sandbox Classic (also for DAML Script).
If this is too annoying to review let me know and I’ll try to split it
up into a few chunks.
changelog_begin
changelog_end
* Update daml-lf/data/src/main/2.13/com/daml/lf/data/LawlessTraversals.scala
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* fixup lawlesstraversal
changelog_begin
changelog_end
* less iterator more view
changelog_begin
changelog_end
* document safety of unsafeWrapArray
changelog_begin
changelog_end
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
