Commit Graph

2875 Commits

Author SHA1 Message Date
Remy
bef0f15d70
[LF] make qualified choice name a LF concept (#15599) 2022-11-17 22:44:18 +00:00
Remy
3dee676cbb
[LF] making LF 1.15 stable (#15614) 2022-11-17 23:30:11 +01:00
azure-pipelines[bot]
c1ffefaaed
bump canton to 20221116 (#15605)
Co-authored-by: Azure Pipelines Daml Build <support@digitalasset.com>
Co-authored-by: Remy Haemmerle <Remy.Haemmerle@daml.com>
2022-11-17 15:05:11 +00:00
Nicu Reut
525831c4db
Fix flaky gRPC server metrics test (#15612) 2022-11-17 15:12:08 +01:00
Oliver Seeliger
936594f523
Deduplication flake: allow inflight-request error (#15609)
In rare situations a canton participant returns a SUBMISSION_ALREADY_IN_FLIGHT
error instead of DUPLICATE_COMMAND due to a possible race condition between
publishing the TransactionAccepted read service update and updating internal
in-flight transactions. Particularly in Oracle runs this can result in a
subsequence "duplicate" command to still be considered in-flight.

Extended the tests to also accept the "in-flight" error code.

Changelog_begin
Changelog_end
2022-11-17 11:13:51 +01:00
Nicu Reut
253150e6e2
Change OpenTelemetry timer unit to seconds [PLEN-44] (#15585) 2022-11-17 09:52:24 +00:00
Marton Nagy
4f4cd910da
Open up ChoiceCoder for usage in another repo (#15592)
[CHANGELOG_BEGIN]
[CHANGELOG_END]
2022-11-16 13:24:59 +01:00
Luc Bourlier
e24ce7ab4f
Adds labels to the Akka HTTP metrics [PLEN-12] (#15436)
* Adds labels to the Akka HTTP metrics

CHANGELOG_BEGIN
CHANGELOG_END

* code improvements

* A bit of headroom for the latency tests
2022-11-16 11:08:45 +01:00
atriantafyllos-da
5ddd0646c4
Removed http_json_api metrics from ledger metrics [DPP-1307]. (#15552)
changelog_begin
changelog_end
2022-11-15 14:56:41 +01:00
Nicu Reut
b1edee94f8
Add gRPC server interceptor with golden metrics to all gRPC services [PLEN-10] (#15477) 2022-11-15 09:50:21 +01:00
Nicu Reut
5b958ae9b7
Use the testing metrics instead of opentelemetry testing sdk [PLEN-43] (#15555) 2022-11-14 13:15:21 +00:00
Simon Maxen
a10ffa819c
Default memory based rate limiting to off (#15546) 2022-11-14 12:48:33 +00:00
mziolekda
60ab087e71
Fix copy pasta in ErrorCategory (#15543)
* Fix copy pasta in ErrorCategory

* update documentation
2022-11-14 08:39:57 +01:00
tudor-da
bd36919e75
Index and server driver_metadata (#15478)
* Create support in Sandbox-on-x with mock contract driver metadata
* Assert driver metadata in conformance tests

changelog_begin
changelog_end

run-full-compat: true
2022-11-11 09:55:34 +01:00
Nicu Reut
7ce5d05f2a
Add grpc server interceptor with golden metrics [PLEN-9] (#15447) 2022-11-10 17:50:41 +00:00
Moritz Kiefer
c7be81f0c4
Fix error handling in time service (#15521)
updateTime can return an error and the code correctly tried to turn
that into invalidArgument and logged it. However, the Either returned
by that then got completely discarded and the overall request
succeeded which definitely seems like a bug.

I added a test and verified that it fails withotu this change.

The code seems fairly messy. There are a few reasons for that:

1. We use Future[Either[…]] and we treat failures in the Either
   differently from future failures. I'm not quite convinced that
   makes sense but for this PR I didn’t want to introduce unrelated
   changed.
2. Future[Either[…]] without some form of EitherT is inherently messy
   but afaik you don’t use cats here so I kept it like this.
2022-11-10 11:40:25 +00:00
Luc Bourlier
4354c5d06b
Support for golden signals metrics on akka-http APIs [PLEN-12] (#15310)
* Support for golden signals metrics on akka-http

Adds metrics support needed for throughput, error rate, latency,
and data size on both the classic http endpoints and
the websocket endpoints.

Installed the support on the JSON API endpoints.

CHANGELOG_BEGIN

JSON API - additional metrics for endpoints

daml_http_json_api_requests_total
daml_http_json_api_errors_total
daml_http_json_api_requests_duration_seconds
daml_http_json_api_requests_size_bytes
daml_http_json_api_responses_size_bytes

daml_http_json_api_websocket_messages_received_total
daml_http_json_api_websocket_messages_received_size_bytes
daml_http_json_api_websocket_messages_sent_total
daml_http_json_api_websocket_messages_sent_size_bytes

CHANGELOG_END


Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2022-11-09 16:55:12 +01:00
tudor-da
346bb48904
Extract Ledger API error definitions in //ledger/ledger-api-errors [DPP-1269] (#15439)
* Extract Ledger API error definitions in //ledger/error-definitions

changelog_begin
SERVICE_INTERNAL_ERROR error code is created in CommonErrors and replaces the usage of LEDGER_API_INTERNAL_ERROR in ledger-api-client (as a fix).
changelog_end

* Tests in //ledger/error do not depend on //ledger/ledger-api-errors

* Remove unused dependency from //ledger/error

changelog_begin
changelog_end

* Address Martino's comment
2022-11-07 17:24:43 +01:00
pbatko-da
3d83948cdb
[DPP-1144] Do not fail when benchtool-tests package is not found by Benchtool (#15455)
changelog_begin
changelog_end
2022-11-07 10:05:11 +00:00
pbatko-da
ba55d37467
[DPP-1141] Run VACUUM ANALYZE from Benchtool at the end of submission step (PG only) (#15385)
changelog_begin
changelog_end
2022-11-04 16:41:55 +01:00
tudor-da
ce464c2421
Precompute scala pb serialized size at creation [DPP-1301] (#15408)
* Precompute ScalaPB serializedSize for stream messages

changelog_begin
Introduces a gRPC streams optimization targeting complex protobuf payloads.
When enabled, it can allow up to 60-80% throughput increase for GetTransactions/GetTransactionTrees/GetActiveContracts.
The optimization is toggleable by an additional config parameter for the API server: `api-server.optimize-grpc-streams-throughput`
changelog_end

* Address Marton's review comment

* Update ledger/participant-integration-api/src/main/scala/platform/store/ScalaPbStreamingOptimizations.scala

Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>

* Rename to withPrecomputedSerializedSize

Co-authored-by: mziolekda <marcin.ziolek@digitalasset.com>
2022-11-04 10:31:15 +00:00
pbatko-da
553674a850
[DPP-1144] Retrieving a Daml packageId from Ledger API server for Benchtool (#15400)
changelog_begin
changelog_end
2022-11-03 11:46:51 +01:00
atriantafyllos-da
d8f907b928
Added groupable class in GroupTag [DPP-1296]. (#15382)
CHANGELOG_BEGIN
CHANGELOG_END
2022-11-03 10:57:56 +01:00
Nicu Reut
5f192c1839
Code clean-up for OpenTelemetry implementation [PLEN-29] (#15357) 2022-11-01 17:13:07 +00:00
Nicu Reut
7077a4dce1
Add global labels to the OpenTelemetry metrics implementation [PLEN-5] (#15407) 2022-11-01 15:28:53 +00:00
Nicu Reut
c93b9a5c9d
Add the metrics context to the metric creation API [PLEN-5] (#15378) 2022-11-01 11:47:27 +00:00
tudor-da
4730eda113
Stream subscription delay in LedgerApiBenchTool (#15352)
changelog_begin
changelog_end
2022-10-28 10:22:57 +00:00
Nicu Reut
e8c7e29ba2
Add labels through the MetricsContext in the metrics API [PLEN-5] (#15349) 2022-10-27 09:26:03 +02:00
Remy
0bba409d0e
[LF] Make ApiCommand.Exercise work directly with type constructor (#15360)
CHANGELOG_BEGIN
CHANGELOG_END
2022-10-26 18:13:38 +00:00
pbatko-da
6fa4abf91b
[DPP-1231] Add a conformance test where we update a non-local party's annotations #15287
changelog_begin
changelog_end
2022-10-26 10:25:24 +02:00
Nicu Reut
29a2c54682
Add opentelemetry metrics implementation [PLEN-29] (#15346) 2022-10-25 17:00:24 +00:00
Nicu Reut
8da0a84183
Introduce explicit timer API for stopping a timer [PLEN-29] (#15344) 2022-10-25 14:44:32 +00:00
Nicu Reut
f550b69463
Reorganize the metrics package structure [PLEN-29] (#15312) 2022-10-25 10:05:40 +00:00
Nicu Reut
8957a3a426
Add opentelemetry metrics integration on the same prometheus endpoint [PLEN-4] (#15232) 2022-10-24 16:50:27 +00:00
tudor-da
1a4ef5b238
Revert "Upgrade dependencies to their latest stable release (#15216)" (#15309)
This reverts commit 1454497f

changelog_begin
changelog_end
2022-10-24 08:33:37 +02:00
atriantafyllos-da
322de044b6
Dpp 1273 elevate qualification on the important metrics to debug (#15294)
CHANGELOG_BEGIN
CHANGELOG_END
2022-10-23 19:23:40 +00:00
Nicu Reut
5b4c7f7677
Extract metrics API that is decoupled from dropwizard [PLEN-28] (#15231) 2022-10-21 16:30:21 +02:00
Marton Nagy
20a639467b
Implement wiring of additional gRPC services (#15301)
changelog_begin
changelog_end
2022-10-21 15:59:35 +02:00
pbatko-da
fd86551873
[DPP-1278] Shuffle party related domain classes (#15276)
changelog_begin
changelog_end

Status quo:
ledger-api-domain's domain object contained three classes related to parties:
 1. `PartyDetails`, the oldest, which didn't contain the metadata field and was used both in client and indexer code,
 2. `ParticipantPartyDetails`, added recently during the participant-local metadata extensions, similar to the `PartyDetails` above but additionally contains the metadata field,
 3. `PartyRecord` which held participant-local metadata extensions for a party.

Changes:
 1) `PartyDetails` gets renamed to `IndexerPartyDetails` and together with `PartyEntry` gets moved to participant-state-index Bazel package and is removed from client code,
 2) `ParticipantPartyDetails` gets renamed to `PartyDetails` and is now used in the client code instead.
 3) `PartyRecord` gets moved to participant-local-store Bazel package
2022-10-20 09:23:34 +00:00
atriantafyllos-da
65b022c51a
Fixed summary of currentRecordTimeLag [DPP-1226] (#15241)
CHANGELOG_BEGIN
CHANGELOG_END
2022-10-19 14:34:00 +00:00
pbatko-da
bbaaac3058
[DPP-1278] Handle almost all the remaining 'um-for-hub' todos (#15261)
* InMemUserMngtImpl: Introducece custom in-mem user repr to better handle resource versions. Fix not updating isDeactivated property
* Generalize concurrent change control tests for users and party records
* Move backend files such that related localstore files are separated from other files
* Remove unneeded nesting in ParticipantParty object
* Document rights in revoke/grant requests as optional. More conformance tests for users rights

changelog_begin
changelog_end
2022-10-19 10:41:10 +02:00
atriantafyllos-da
e5c418a913
Bisected daml.execution.cache.register_update [DPP-1248]. (#15274)
CHANGELOG_BEGIN
* Removed daml_execution_cache_register_update metric
* Added:
  - daml_execution_cache_contract_state_register_update
  - daml_execution_cache_key_state_register_update
CHANGELOG_END
2022-10-18 17:33:13 +00:00
pbatko-da
ddfa7c1d9d
[DPP-1278] Do not test the exact matching of error messages in ObjectMeta related conformance tests (#15277)
changelog_begin
changelog_end
2022-10-18 17:41:24 +02:00
Gerolf Seitz
1454497f3a
Upgrade dependencies to their latest stable release (#15216)
The outdated dependencies where determined by running
`bazel run @maven//:outdated`.

Not all dependencies are brought up to the latest version available on
Maven Central, for the following reasons:

Nobody dares to touch DB:
  com.oracle.database.jdbc:ojdbc8 [19.14.0.0 -> 21.7.0.0]
  com.oracle.database.jdbc.debug:ojdbc8_g [19.14.0.0 -> 21.7.0.0]
  com.h2database:h2 [2.1.210 -> 2.1.214]
  org.flywaydb:flyway-core [8.4.1 -> 9.4.0]

License issues beyond version 2.6.x:
  com.typesafe.akka:akka-actor_2.13 [2.6.18 -> 2.7.0-M3]
  com.typesafe.akka:akka-actor-testkit-typed_2.13 [2.6.18 -> 2.7.0-M3]
  com.typesafe.akka:akka-actor-typed_2.13 [2.6.18 -> 2.7.0-M3]
  com.typesafe.akka:akka-http_2.13 [10.2.8 -> 10.4.0-M1]
  com.typesafe.akka:akka-http-spray-json_2.13 [10.2.8 -> 10.4.0-M1]
  com.typesafe.akka:akka-http-testkit_2.13 [10.2.8 -> 10.4.0-M1]
  com.typesafe.akka:akka-slf4j_2.13 [2.6.18 -> 2.7.0-M3]
  com.typesafe.akka:akka-stream_2.13 [2.6.18 -> 2.7.0-M3]
  com.typesafe.akka:akka-stream-testkit_2.13 [2.6.18 -> 2.7.0-M3]
  com.typesafe.akka:akka-testkit_2.13 [2.6.18 -> 2.7.0-M3]

Not a stable release:
  com.chuusai:shapeless_2.13 [2.3.3 -> 2.4.0-M1]
  io.circe:circe-core_2.13 [0.14.3 -> 0.15.0-M1]
  io.circe:circe-generic_2.13 [0.14.3 -> 0.15.0-M1]
  io.circe:circe-parser_2.13 [0.14.3 -> 0.15.0-M1]
  org.scalameta:munit_2.13 [0.7.29 -> 1.0.0-M6]
  org.scalactic:scalactic_2.13 [3.2.14 -> 3.3.0-SNAP3]
  org.scalatest:scalatest_2.13 [3.2.14 -> 3.3.0-SNAP3]
  org.scalatestplus:scalacheck-1-15_2.13 [3.2.11.0 -> 3.3.0.0-SNAP3]
  org.scalatestplus:selenium-3-141_2.13 [3.2.10.0 -> 3.3.0.0-SNAP3]
  org.scalatestplus:testng-6-7_2.13 [3.2.10.0 -> 3.3.0.0-SNAP3]
  org.tpolecat:doobie-core_2.13 [0.13.4 -> 1.0.0-RC2]
  org.tpolecat:doobie-hikari_2.13 [0.13.4 -> 1.0.0-RC2]
  org.tpolecat:doobie-postgres_2.13 [0.13.4 -> 1.0.0-RC2]
  com.fasterxml.jackson.core:jackson-core [2.13.4 -> 2.14.0-rc2]
  com.fasterxml.jackson.core:jackson-databind [2.13.4 -> 2.14.0-rc2]

Netty + boringssl + grpc + protobuf need be kept in sync, and the latest
versions also aren't stable releases:
  io.netty:netty-buffer [4.1.79.Final -> 5.0.0.Alpha2]
  io.netty:netty-codec-http2 [4.1.79.Final -> 5.0.0.Alpha2]
  io.netty:netty-handler [4.1.79.Final -> 5.0.0.Alpha2]
  io.netty:netty-handler-proxy [4.1.79.Final -> 5.0.0.Alpha2]
  io.netty:netty-resolver [4.1.79.Final -> 5.0.0.Alpha2]
  com.google.protobuf:protobuf-java [3.21.7 -> 4.0.0-rc-2]

Scalaz changed a few annoying things in 7.3, so let's stick to 7.2:
  org.scalaz:scalaz-core_2.13 [7.2.34 -> 7.4.0-M12]
  org.scalaz:scalaz-scalacheck-binding_2.13 [7.2.34-scalacheck-1.15 -> 7.4.0-M12]
  org.scalacheck:scalacheck_2.13 [1.15.4 -> 1.17.0]

Determined by the rules_scala:
  org.scala-lang:scala-library [2.13.8 -> 2.13.10]

CHANGELOG_BEGIN
Upgraded to the latest stable version of the following JVM dependencies:

ch.qos.logback:logback-classic:1.4.4
ch.qos.logback:logback-core:1.4.4
com.auth0:java-jwt:4.1.0
com.auth0:jwks-rsa:0.21.2
com.fasterxml.jackson.core:jackson-core:2.13.4
com.fasterxml.jackson.core:jackson-databind:2.13.4
com.github.ben-manes.caffeine:caffeine:3.1.1
com.github.pathikrit:better-files_2.13:3.9.1
com.github.pureconfig:pureconfig-core_2.13:0.17.1
com.github.pureconfig:pureconfig-generic-base_2.13:0.17.1"
com.github.pureconfig:pureconfig-generic_2.13:0.17.1
com.github.pureconfig:pureconfig_2.13:0.17.1
com.github.scopt:scopt_2.13:4.1.0
com.google.code.gson:gson:2.9.1
com.lihaoyi:pprint_2._13:0.8.0"
com.lihaoyi:sourcecode_2.13:0.3.0
com.sparkjava:spark-core:2.9.4
com.squareup:javapoet:1.13.0
com.storm-enroute:scalameter-core_2.13:0.21
com.storm-enroute:scalameter_2.13:0.21
com.thesamet.scalapb:scalapb-json4s_2.13:0.12.0
com.typesafe.scala-logging:scala-logging_2.13:3.9.5
com.zaxxer:HikariCP:5.0.1
commons-io:commons-io:2.11.0
eu.rekawek.toxiproxy:toxiproxy-java:2.1.7
io.circe:circe-core_2.13:0.14.3
io.circe:circe-generic-extras_2.13:0.14.2
io.circe:circe-generic_2.13:0.14.3
io.circe:circe-parser_2.13:0.14.3
io.circe:circe-yaml_2.13:0.14.1
io.github.paoloboni:spray-json-derived-codecs_2.13:2.3.9
io.reactivex.rxjava2:rxjava:2.2.21
io.spray:spray-json_2.13:1.3.6
javax.annotation:javax.annotation-api:1.3.2
javax.ws.rs:javax.ws.rs-api:2.1.1
junit:junit-dep:4.11
junit:junit:4.13.2
net.logstash.logback:logstash-logback-encoder:7.2
org.apache.commons:commons-lang3:3.12.0
org.apache.commons:commons-text:1.10.0
org.awaitility:awaitility:4.2.0
org.checkerframework:checker-qual:3.26.0
org.checkerframework:checker:3.26.0
org.codehaus.janino:janino:3.1.8
org.freemarker:freemarker-gae:2.3.31
org.jline:jline-reader:3.21.0
org.jline:jline:3.21.0
org.junit.jupiter:junit-jupiter-api:5.9.1
org.junit.jupiter:junit-jupiter-engine:5.9.1
org.junit.platform:junit-platform-engine:1.9.1
org.junit.platform:junit-platform-runner:1.9.1
org.mockito:mockito-core:4.6.1
org.mockito:mockito-inline:4.6.1
org.mockito:mockito-scala_2.13:1.17.12
org.pcollections:pcollections:3.2.0
org.playframework.anorm:anorm-akka_2.13:2.7.0
org.playframework.anorm:anorm_2.13:2.7.0
org.postgresql:postgresql:42.5.0
org.reactivestreams:reactive-streams-tck:1.0.4
org.reactivestreams:reactive-streams:1.0.4
org.reflections:reflections:0.10.2
org.sangria-graphql:sangria-ast_2.13:3.4.0
org.sangria-graphql:sangria-core_2.13:3.4.0
org.sangria-graphql:sangria-derivation_2.13:3.4.0
org.sangria-graphql:sangria-parser_2.13:3.4.0
org.sangria-graphql:sangria-spray-json_2.13:1.0.3
org.scala-lang.modules:scala-parallel-collections_2.13:1.0.4
org.scalactic:scalactic_2.13:3.2.10
org.scalameta:munit_2.13:0.7.29
org.scalatest:scalatest_2.13:3.2.10
org.scalatestplus:scalacheck-1-15_2.13:3.2.10.0
org.scalatestplus:selenium-3-141_2.13:3.2.10.0
org.scalatestplus:testng-6-7_2.13:3.2.10.0
org.scalaz:scalaz-core_2.13:7.2.34
org.scalaz:scalaz-scalacheck-binding_2.13:7.2.34-scalacheck-1.15
org.seleniumhq.selenium:selenium-java:4.5.0
org.slf4j:slf4j-api:2.0.3
org.slf4j:slf4j-simple:2.0.3
org.testng:testng:6.7
org.typelevel:cats-core_2.13:2.8.0
org.typelevel:cats-kernel_2.13:2.8.0
org.typelevel:cats-laws_2.13:2.8.0
org.typelevel:paiges-core_2.13:0.4.2
org.wartremover:wartremover_2.13.8:3.0.6
org.xerial:sqlite-jdbc:3.39.3.0
CHANGELOG_END
2022-10-18 08:50:23 +02:00
pbatko-da
61e85b19a2
[DPP-1278] Remove custom daml-script UserManagementStore (#15266)
changelog_begin
changelog_end
2022-10-17 20:58:41 +02:00
pbatko-da
04a5d2a12a
[DPP-1278] Address a portion of um-for-hub todos left in the codebase (#15255)
changelog_begin
changelog_end
2022-10-17 17:51:58 +00:00
pbatko-da
0d88435de1
Do not catch fatal errors in error intercpetor and hikari connection provider (#15263)
changelog_begin
changelog_end
2022-10-17 16:39:08 +00:00
pbatko-da
f0c2275726
[DPP-1278] User/party management: do not assert on detailed error messages in conformance tests as they should not be subject to compatibility checking (#15251)
changelog_begin
changelog_end

By doing this change we are loosing some detailed test coverage, but preventing future spurious compatibility tests failures.
2022-10-17 15:29:47 +00:00
pbatko-da
971e4f09a4
[DPP-1278] Treat empty string displayName as lack of displayName on DB result parsing (#15238)
changelog_begin
changelog_end
2022-10-17 14:48:05 +02:00
mziolekda
4f08bedc17
Print all kinds of problems in hikari connection checker (#15248)
CHANGELOG_BEGIN
CHANGELOG_END
2022-10-17 11:13:41 +02:00