Commit Graph

3002 Commits

Author SHA1 Message Date
mziolekda
6501782dea
Reenable IDP canton tests (#16252) 2023-02-07 19:31:37 +01:00
Nicu Reut
20f1227d5c
Remove latest offset gauge [PLEN-103] (#16247) 2023-02-07 10:08:02 +01:00
pbatko-da
391ba32201
[DPP-1395] Adding pruning benchmarking capability to the benchtool + non-transient contracts (#16201)
Changes:
1. Add support for defining pruning benchmarks.
Currently the new top-level `unary` section allow defining only a single `pruning` section.
Declaring both top sections `unary` and `streams` in a single workflow config is unsupported.
```
unary:
  - type: pruning
    name: pruning-or-101
    max_duration_objective: 600s
    prune_all_divulged_contracts: false
```

2. Add support for optionally submitting non-transient contracts.
```
submission:
  ...
  allow_non_transient_contracts: true
```
2023-02-07 08:40:56 +01:00
Sergey Kisel
2713e2b4b4
Identity Provider Config Service conformance tests [DPP-1339] (#16230)
* Identity Provider Config Service conformance tests [DPP-1339]

* formatting

* address code review

run-full-compat:true

* address code review concerns from Pawel

* address code review concerns by Pawel

* remove accidentially added stuff

* fix canton tests

---------

Co-authored-by: Marcin Ziolek <marcin.ziolek@digitalasset.com>
2023-02-06 20:44:44 +00:00
Sergey Kisel
5c89f8be42
Check for party existence in the index party management (#16228) 2023-02-06 13:02:37 +00:00
pbatko-da
804fc47ecb
[DPP-1395] Use one randomness privider in the benchtool (#16235) 2023-02-06 10:36:23 +01:00
Remy
aad8725ea7
[damlc] change default output to 1.15 (#16157) 2023-02-06 08:58:47 +01:00
Sergey Kisel
f6eaf3e2e9
Identity Provider Config Service: Validation rule changes (#16233) 2023-02-03 16:45:55 +00:00
Nicu Reut
a9ff1376f4
Use in-memory metrics for indexer benchmark banner [PLEN-99] (#16218) 2023-02-03 16:16:12 +00:00
azure-pipelines[bot]
2e449b0d91
bump canton to 20230202 (#16229)
* bump canton to 20230202

CHANGELOG_BEGIN
CHANGELOG_END

* DACH-NY/canton#11206 Fix dev option flags for Canton in `ledger-api-test-tool-on-canton`

---------

Co-authored-by: Azure Pipelines Daml Build <support@digitalasset.com>
Co-authored-by: Kirill Zhuravlev <kirill.zhuravlev@digitalasset.com>
2023-02-03 13:48:34 +01:00
pbatko-da
51f302b607
[DPP-1400] Refactor and simplify the pruning backend tests (#16220)
A concern with the existing pruning backend tests is that they are relatively complex particularly because:
- in order to fetch data from the indexdb they use production sql queries which might have builtin knowledge of pruning offsets,
- production sql queries are not designed for ergonomic one-off usage.

This PR introduces a set of queries, one for each table subject to pruning, each returning a record for each row its table. This leads to more a concise and straightforward way assert what data has or hasn't been pruned.
2023-02-03 11:41:51 +00:00
Nicu Reut
234e7a1fad
[metrics] Improve the gauges API to allow closing a gauge [PLEN-107] (#16208) 2023-02-02 11:51:44 +00:00
Sergey Kisel
b51ab760ec
Use allocated party id for the subsequent steps (#16210) 2023-02-01 11:55:08 +00:00
Sergey Kisel
c04019de4e
Restrict the use of non-allocated party in User Mnmgt API for ParticipantAdmin [DPP-1385] (#16111) 2023-01-28 11:49:11 +01:00
pbatko-da
58cb5386ed
[DPP-1390] Introduce separate 'index-service.acs-streams.contract-processing-parallelism' config key (#16182)
Previously the config key `index-service.events-processing-parallelism` was used both when fetching ACS from DB
and for buffered transaction reader.
Now these resposibilities are separated into two new config keys:
1. `index-service.acs-streams.contract-processing-parallelism` (which mirrors the tx streams configs)
2. `index-service.buffered-events-processing-parallelism`

Also moving a larger chunk of code from TransactionReader to ACSReader (which mirrors the tx stream readers)
2023-01-27 18:05:02 +00:00
pbatko-da
73a1997902
[DPP-1363] Use largest smaller power of two for Akka async input buffers for ETQ streams (#16166) 2023-01-27 14:06:55 +00:00
pbatko-da
bbd5966e9f
[DPP-1390] Improve SoX config docs wording for ETQ related config keys (#16169) 2023-01-27 13:34:47 +01:00
Nicu Reut
a87412b68c
Introduce LabeledMetricsFactory trait to mark metrics factories that support labels [PLEN-98] (#16159) 2023-01-26 16:01:37 +00:00
Stephen Compall
81b4e17f00
reduce package service calls to avoid THREADPOOL_OVERLOADED errors (#16116)
* confirm that monadifying the package fetch still suppresses the error

* thread ExecutionContext from request

- makes the error less likely
- but still fairly easy to repro with 3 tabs

* experiment with setting executor

* explain that the cache isn't a cache

* random order, maybe

- #3090 mentions keeping the order as a goal; I don't see why we should,
  though

* random order with groups of 8

* embed the decoding

- this slows down the processing of a group, yielding somewhat less
  granular contention
- and also makes hits cost much less, at the cost of making granular
  contention more expensive

* reduce diff size before resolution

- this won't improve contention, but does nearly eliminate the cost of
  resolution for already-resolved packages, making hits nearly free
  (amortized)

* randomize groups instead

- while groups themselves can overlap with this arrangement, each
  costing ParallelLoadFactor granular contention, on average it seems to
  perform a little better due to groups never overlapping

* refactor StatusEnvelope to utils

* constant 250ms retry

* detect contention earlier and skip decode

* factor traverseFM
2023-01-26 10:18:13 -05:00
Luc Bourlier
e809344c00
Metrics for node status [PLEN-63] (#16052)
* Metrics for node status [PLEN-63]

CHANGELOG_BEGIN

* Node status metric: daml_health_status

CHANGELOG_END
2023-01-26 12:01:14 +01:00
Sergey Kisel
efcc6620ed
Implement IDP ID check transactionally within persistence layer [DPP-1386] (#16134) 2023-01-26 11:21:18 +01:00
pbatko-da
232774a3bf
[DPP-1063] Addressing benchtool flaky tests - second attempt (#16161) 2023-01-26 10:50:13 +01:00
Nicu Reut
fee45895d8
[metrics] Remove the API allowing us to read the state of a counter [PLEN-97] (#16141) 2023-01-26 09:05:56 +01:00
Nicu Reut
1f5a4e1368
Improve threadpool overloaded rejection message [PLEN-96] (#16160) 2023-01-26 09:05:12 +01:00
tudor-da
2291d7c728
Fix ExplicitDisclosureIT:EDDuplicates for Canton (#16158) 2023-01-25 18:42:57 +00:00
pbatko-da
48bd296e80
[DPP-1363] Refactor IdPageSizing (#16138) 2023-01-25 17:12:32 +00:00
Luc Bourlier
dec4738f53
Adds documentation for pruning metrics (#16099)
* Adds documentation for pruning metrics

CHANGELOG_BEGIN

CHANGELOG_END

* Apply suggestions from code review

Co-authored-by: da-katmurp <117752519+da-katmurp@users.noreply.github.com>

Co-authored-by: Luc Bourlier <lb@skyluc.org>
Co-authored-by: da-katmurp <117752519+da-katmurp@users.noreply.github.com>
2023-01-25 18:10:48 +01:00
pbatko-da
652f672cd5
Fix flaky benchtool tests by fixing random seed (#16143) 2023-01-25 15:38:00 +01:00
Sergey Kisel
b42267298f
Reorder auth operations to check default auth service first (#16128) 2023-01-25 15:03:33 +01:00
azure-pipelines[bot]
187a2ea25d
bump canton to 20230124 (#16149)
* bump canton to 20230124
* reactive canton dev test

Co-authored-by: Azure Pipelines Daml Build <support@digitalasset.com>
Co-authored-by: Remy Haemmerle <Remy.Haemmerle@daml.com>
2023-01-25 14:01:42 +00:00
atriantafyllos-da
4d026f02a1
TraceId in LAPI [DPP-1372]. (#15995)
changelog_begin
changelog_end
2023-01-24 23:10:51 +00:00
Nicu Reut
17596abd1b
Expose opentelemetry jvm metrics alongside dropwizard metrics [PLEN-67] (#16133) 2023-01-24 19:58:20 +01:00
Nicu Reut
efcf62ac0a
Simplify and split interface for executor services [PLEN-96] (#16139) 2023-01-24 19:57:58 +01:00
Nicu Reut
09807d33a7
Fix literal string for metrics documentation (#16136) 2023-01-24 16:41:54 +00:00
pbatko-da
b627a803b7
[DPP-1367] Make the ACS pruning tests work for Canton (#16114) 2023-01-24 11:34:04 +00:00
Nicu Reut
20d7e2becd
Remove thread check dependency on metrics [PLEN-96] (#16113)
Co-authored-by: Simon Maxen <simon.maxen@digitalasset.com>
2023-01-24 11:06:06 +00:00
Nicu Reut
85eff2930f
Configure OpenTelemetry using the sdk [PLEN-68] (#16083) 2023-01-24 08:53:25 +01:00
oliverse-da
b3f84bfca5
Flaky test fix of TransactionServiceVisibilityIT (#16125)
* Flaky test fix of TransactionServiceVisibilityIT

when running against OracleDb.

For example we have seen TransactionServiceVisibilityIT:TXTreeBlinding
time out waiting for one of the three participants to write to oracle
db.

changelog_begin
changelog_end

* PR feedback
2023-01-23 18:46:13 +00:00
Sergey Kisel
ef32584b94
Log level change to INFO for IdentityProviderAwareAuthService (#16126) 2023-01-23 16:23:35 +00:00
pbatko-da
e1aea25fdc
[DPP-1363] Handle outstanding ETQ TODOs: Part 2 (#16094)
Changes:
* Move acs config keys into into its own case class + config key renames.
* Apply the global limit of parallel event id queries (shared with tx streams) to acs streams.
* Replace the acs limit of parallel event payload queries with the global limit (shared with tx streams).
* Assert on participant_meta_table in StorageBackendTestsInitializeIngestion test
2023-01-23 16:32:15 +01:00
Remy
18faa81608
[Speedy] slight refactoring of contract key handeling (#16097)
* cleanup PartialTransaction API

* Put SValue + GlobalKey in cached contract key

* slight change of the ContractStateMahcine API

* drop (unsafe) builder for GlobalKey
2023-01-23 14:40:45 +00:00
tudor-da
179b865a20
Move ExplicitDisclosureIT to test tool 1.15 (#16091)
* Move ExplicitDisclosureIT to test tool 1.15
since it's not tied to the LF dev version anymore in the Engine.

* Merge ED and interfaces conformance test targets
2023-01-19 18:14:53 +01:00
azure-pipelines[bot]
ed93f5e5c6
bump canton to 20230118 (#16102)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines Daml Build <support@digitalasset.com>
2023-01-19 16:13:19 +01:00
Sergey Kisel
ee1e26900c
Identity Provider Management: Authorization tests [DPP-1338] (#15994) 2023-01-19 15:23:43 +01:00
pbatko-da
c5df0112fc
Make ActiveAtOffsetAfterLedgerEnd conformance test more robust (#16106) 2023-01-19 15:21:36 +01:00
Sergey Kisel
3d35f9a652
Integrate IdentityProviderAwareAuthService into LedgerApiServer (#16084) 2023-01-19 14:53:59 +01:00
Sergey Kisel
b546f6f8b5
Restrict the use of foreign IDP parties within user management API [DPP-1383] (#16025) 2023-01-19 14:10:30 +01:00
Remy
10fdd77fe8
[ED] replace ImmArray[Byte] by Bytes in EngineEnrichedContractMetadata (#16105) 2023-01-19 11:36:15 +00:00
Nicu Reut
43be887732
Change rate limiter stream check to use gauges [PLEN-95] (#16098) 2023-01-19 11:22:07 +00:00
pbatko-da
0d18d04f61
[DPP-1013] Benchtool: parse more config keys as durations and support more optional config keys (#16090)
Also expand benchtool readme.md
2023-01-18 11:30:29 +00:00