Commit Graph

8075 Commits

Author SHA1 Message Date
akshayshirahatti-da
f68a12930b
Report some oracle_json_perf numbers on slack (#10754)
* Add avg, stddev, p90, p99, requests_per_second numbers to be reported on slack similar to speedy_perf

changelog_begin
changelog_end

* changes based on code review

* fix failing job due to breaking function export
2021-09-02 17:44:01 +01:00
Andreas Herrmann
7270ee3c71
Handle dynamic port in auth middleware client (trigger service) (#10755)
* Do not require redirect URI upfront

changelog_begin
changelog_end

This allows users to configure the redirect URI after the middleware
client binding occurred such that dynamic port selection can be taken
into account.

* Use port 0 in trigger service fixture

changelog_begin
changelog_end

* Update triggers/service/auth/src/test/scala/com/daml/auth/middleware/oauth2/TestMiddleware.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update triggers/service/auth/src/test/scala/com/daml/auth/middleware/oauth2/TestMiddleware.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update triggers/service/auth/src/test/scala/com/daml/auth/middleware/oauth2/TestMiddleware.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update triggers/service/auth/src/test/scala/com/daml/auth/middleware/oauth2/TestMiddleware.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update triggers/service/auth/src/test/scala/com/daml/auth/middleware/oauth2/TestMiddleware.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2021-09-02 18:06:48 +02:00
Victor Peter Rouven Müller
ff1308ee3d
[Docs] Add info on logs on Kubernetes & metrics in the ops section (#10525)
* [Docs] Add info on logs on Kubernetes & metrics in the ops section

changelog_begin

- [Docs] Information was added in the `Operating Daml` section on how to aggregate logs on Kubernetes in conjuction with Daml services & what options exists for exporting metrics from daml services (not Kubernetes specific)

changelog_end

* Move the new docs into a seperate section 'Operating Daml Connect'

* Move section again, now it's a subsection under Operating Daml

* Update docs/source/ops/connect/content.rst

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Apply suggestions from code review

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Rename file

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2021-09-02 14:52:27 +00:00
nicu-da
d26739087f
Update buf image [kvl-1049] (#10752)
* Update buf image with new definitions

CHANGELOG_BEGIN

CHANGELOG_END

* Remove exceptions from buf config as the base image was updated
2021-09-02 06:04:13 -07:00
nicu-da
d2180cf60b
Update exclusion for command deduplication to include full version range (#10750)
CHANGELOG_BEGIN

CHANGELOG_END
2021-09-02 05:56:44 -07:00
nicu-da
41881ba2ba
Command dedup: migrate kvutils to use v2 services [KVL-1049] (#10679)
* Add deduplication and submission id to the proto submitter info so we can build the v2 completion info

* Use v2 rejections and store the enriched model in protobuf

* Update v1 SubmissionResult to v2 SubmissionResult

* Update v1 imports to v2 imports

* Enable API deduplication for ledgers that don't have it implemented yet (all kv)

* Update tests for new rejections

* Disable buf check for oneof

* Code review - fix formatting and test assertions

* Update imports for proto changes

* Add tests for rejection conversions

* Use entry id as submission id when submission id is missing

* Calculate deduplication period for rejection based on the set deduplicate until by calculating the duration relative to record time

* Code review consistent naming and comments
* Submission id is optional in the completion info

the submissionId is not set for participant.state.v1 entries, therefore for the transition from v1 to the v2 API we need to account for the missing submission ids by making it optional in the completion info.

* Update messages and grpc status codes for rejections to keep backward compatibility

* Deprecate proto deduplicate_until
* Set deduplication value in the commit context for all the deduplication periods. This change allows us to re-enable the committer deduplication tests

* Set ledger configuration for tests

* Command dedup: In the committer set the deduplicate until context value based on the given deduplication period [KVL-1049] (#10743)


CHANGELOG_BEGIN
participant-state:  
- Migrate to use only v2 Read/Write Services. This includes the use of new models for rejections/updates/submission results.
- Calculate deduplicate until for committer side deduplication based on the submitter given deduplication period
   -  if using the deprecated deduplicateUntil then just set the given timestamp
   -  if using duration then calculate the new deduplicateUntil by using the formula (submissionTime + deduplicationDuration + minSkew)
  -   if using offset deduplication then calculate deduplicateUntil by using the formula (submissionTime + maxDeduplicationDuration + minSkew)
  -  if the deduplication period is not set then we don't set deduplicateUntil
- Emit completions with status `ALREADY_EXISTS` for duplicate commands

CHANGELOG_END
2021-09-02 04:44:55 -07:00
akshayshirahatti-da
4525b8c265
[JSON-API] vanilla oracle_perf ci job (#10688)
* vanilla job test on main pipeline

changelog_begin
changelog_end

* move job to daily compat tests

* add timeout to dev-env and changes based on code review
2021-09-02 09:31:56 +01:00
pbatko-da
16df8a5e35
[Short] Remove unused code (#10719)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-02 10:08:54 +02:00
pbatko-da
b28afcf7ae
[DPP-438] Update docs on metrics that no longer use <party_name> in their name (#10728)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-02 10:07:36 +02:00
fabiotudone-da
0662025f4c
Clarify what the buf image is and how it should be used (#10741)
* Clarify what is the buf image

CHANGELOG_BEGIN
CHANGELOG_END

* Expand comment on releases

* Improve comment wording
2021-09-02 09:27:07 +02:00
Samir Talwar
9071a05c76
sandbox-classic: Reintroduce SqlLedger tests for the mutable index. (#10722)
I previously changed `SqlLedgerSpec` to use the append-only index, not
realizing that we have both test classes. This removes the
(now-redundant) `SqlLedgerSpecAppendOnly`, and copies `SqlLedgerSpec` to
`SqlLedgerOnMutableIndexSpec`. The only differences are:

- the mutable index tests check for synchronous commits, and
- the append-only index tests check for rejection reason details
  (which are not present in the mutable index schema).

Previously I had added some tests to `SqlLedgerSpec` but not
`SqlLedgerSpecAppendOnly`, so they had diverged a little.

I also think this makes it clearer that where the tests diverge, we want
to keep the append-only tests, not the mutable ones.

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-02 07:35:35 +02:00
mziolekda
f576cdfd06
fix flaky test in RecoveringIndexer (#9619)
* fix flaky test in RecoveringIndexer
changelog_begin
changelog_end

* move the log to the re-subscription callback

* address review comments

CHANGELOG_BEGIN
CHANGELOG_END

* Fixed compilation errors

Co-authored-by: Kamil Bozek <kamil.bozek@digitalasset.com>
2021-09-01 15:30:53 +00:00
Samir Talwar
f6a75b42f3
ledger-api-common: Do not mock final classes. (#10733)
* ledger-api-common: Replace `URL` with a trait so we can fake it.

This means we don't have to use Mockito to stub out `URL`, which is
final and therefore requires magic to stub.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-api-common: Move tests that open streams into SecretsUrlTest.

* ledger-api-common: Make SecretsUrl constructors simple methods.

* ledger-api-common: Reduce concrete implementations of SecretsUrl.
2021-09-01 13:36:30 +00:00
Moritz Kiefer
e9c8af5024
Bump ghc-lib to include dropped parsing code for generic templates (#10735)
* Bump ghc-lib to include dropped parsing code for generic templates

changelog_begin
changelog_end

* bump snapshot

changelog_begin
changelog_end

* drop old generics file

changelog_begin
changelog_end

* drop other broken file

changelog_begin
changelog_end

* Bump again

changelog_begin
changelog_end

* bump to merged commit

changelog_begin
changelog_end

* and bump snapshots

changelog_begin
changelog_end
2021-09-01 15:35:40 +02:00
azure-pipelines[bot]
862a2901fb
rotate release duty after 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10731)
@aherrmann-da is taking care of 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10730), so they get pushed back to the end of the line.

Please do not merge this before #10730.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-09-01 14:03:30 +02:00
azure-pipelines[bot]
12cb92b431
update compat versions for 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10737)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-09-01 10:49:09 +00:00
pbatko-da
73c0de8db4
[DPP-578] Temporarily disable flaky test - Mockito plugin issue (#10734)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-01 11:20:52 +02:00
azure-pipelines[bot]
963bcb17eb
release 1.17.0-snapshot.20210831.7702.0.f058c2f1 (#10730)
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.

@aherrmann-da is in charge of this release.

Commit log:
```
f058c2f16f DPP-368 clean up flags (#10711)
90ad24fd6a  [Divulgence pruning] Prune immediate divulgence [DPP-513] (#10691)
183934b89b Command dedup: add columns to completions in append-only schemas [KVL-1057] (#10652)
27c13334b6 LF: Drop outdated TODOs (#10725)
9be577a7fb Enable pruning in the sandbox-classic when the append-only schema is used [DPP-567] (#10708)
9f072aeba0 Ledger-API Conformance test for Contract ID V0 (#10717)
867547c579 DPP-368 enable append-only flag in sandbox (#10710)
bdc511eaa9 [DPP-438] Change open-ended metric names into static ones (by removing partyName part) (#10706)
0c820063f1 [Divulgence pruning] Prune all divulged contracts only after migration offset [DPP-483] (#10661)
2555dbb30b Use soft references for values in the caches (#10715)
7fd590694e Add LedgerClientWithoutLedgerId next to the LedgerClient (#10681)
856c69c425 participant-integration-api: Increase a test timeout. (#10721)
b86d07d498 remove non functioning oracle json array indices (#10720)
f5e17567f9 sandbox-classic: Remove default parameters in `SqlLedger.Owner`. (#10718)
9ef3377864 LF: Update specification with Contract ID Comparability check (#10703)
e5c4734780 enable JSON search index on Oracle by default (#10539)
1ded42f185 [DPP-418] Protect TLS keys - follow-up cleanup (#10696)
3fcd986f0b Introduce a new `Offset` format [KVL-1063] (#10668)
a5781a6682 update NOTICES file (#10714)
89855054b5 participant-integration-api: Use deadlines, not deduplication times, for expiring trackers. [KVL-1009] (#10704)
65025c26b8 sandbox-classic: Add ErrorInfo metadata for rejections. [KVL-1048] (#10707)
97bda3ca36 LF: V1 Contract ID check in Preprocessor (#10687)
c2f90efbba Add CLI option to force disabling of participant deduplication (#10698)
6016633bb7 Construct ParsedModule directly in Daml Repl (#10701)
bbdf16aacf DPP-368 unhide append-only CLI flags (#10697)
a41b1349df Use the tracker retention period as the maximum expiry time. [KVL-1009] (#10700)
e750ba5c45 Make warning less scary. (#10699)
5f120bdf87 rotate release duty after 1.17.0-snapshot.20210824.7647.0.640fb683 (#10660)
a17253fb63 DPP-535 Verify postgres version (#10577)
301ce53a71 participant-integration-api: Add tests for ApiCommandService. [KVL-1009] (#10689)
bd01a211f4 [DPP-418] Protect Participant TLS keys (#10629)
7ee13243e1 update NOTICES file (#10695)
7c392f32dc update NOTICES file (#10693)
3db654e9dd update NOTICES file (#10690)
eff09c010a ledger-api-client: Wrap command submissions in a new class. [KVL-1009] (#10683)
d54adb2543  Ledger-API: Conformance tests for contract IDs suffixing (#10654)
aa2e869ede [Divulgence pruning] Pass divulged contract arguments through kvutils Write/ReadService [DPP-535] (#10598)
1a7831390e Disable DeeplyNestedValueIT suite against canton in Daml repo (#10686)
b5f9be3fbc participant-integration-api: Standardize tracker retention naming. (#10682)
2aa632e3aa ledger-on-sql: Do not increment the dispatcher head on start. (#10684)
eabb19d7f7 [ledger-api] Add deduplication_duration to deduplication period [kvl-1047] (#10676)
96ad9b5ab8 [Divulgence pruning] All divulgence events pruning [DPP-483] (#10634)
0b7980d6eb Update rules_haskell (#10674)
284edfc4cd Fix FlywayMigrations datasource (#10666)
adbe65fae1 Document ActionFail vs CanAbort (#10657)
52e7a6d928 update compat versions for 1.17.0-snapshot.20210824.7647.0.640fb683 (#10667)
f42e6b6877 Expose pending contracts in triggers (#10672)
7cc698948c Add multiple ways of specifying deduplication [KVL-1047] (#10601)
53be19f86c participant-integration-api: Ensure that all waiting, failed, and closed trackers are cleaned up. (#10662)
b27cde6440 participant-integration-api: Move tracker code around, and tidy up tests. (#10663)
fc9d35935a Drop alternative rules from dlint config (#10646)
5204d3ce7a Include committers in PartialTransaction root context (#10665)
387c68b9eb Normalize transaction values within the engine (#10648)
430c1cccf3 release 1.17.0-snapshot.20210824.7647.0.640fb683 (#10659)
ef239fdd1a participant-integration-api: Move `TrackerMap` code around. [KVL-1009] (#10653)
```
Changelog:
```
- [Sandbox] - Participant pruning is enabled in the sandbox-classic when the append-only schema is used

- [Ledger Client Scala Bindings] A new variant of the LedgerClient class
    was added called `LedgerClientWithoutLedgerId`. This class does not
    need a ledger id at initialization. It was added to allow skipping
    any checks at initialization for use cases where either the
    ledger id is not known at initalization or no valid token can be fed
    at initialization for checking the ledger id. Furthermore for each
    classes `ActiveContractSetClient`, `CommandClient`, `PackageClient`,
    `TransactionClient`, `VersionClient` now exists a variant which
    doesn't depend on a ledger id at initialization and instead requires
    one for every function as parameter. Moreover the existing classes
    are extending these classes with overriding the methods and setting
    the default of the parameter with the given ledger id from
    initialization. The class `LedgerClientWithoutLedgerId` already
    makes usage of these variants e.g. `PackageClientWithoutLedgerId`.

- [Ledger Client Scala Bindings] The function `transactionSource` of the
    class `LedgerClientBinding` now optionally accepts a token which is
    passed on to the unterlying call.

- [JSON API] The Oracle database schema has changed; if using
  ``--query-store-jdbc-config``, you must rebuild the database by adding
  ``,start-mode=create-only``.  See #10539.
- [Trigger Service] ``--help`` no longer advertises unsupported JDBC
  options from JSON API.
- [JSON API] [EE only] By default, on Oracle, sets up a JSON search
  index to speed up the queries endpoints.  However, Oracle versions
  prior to 19.12 have an unrecoverably buggy implementation of this
  index; in addition, the current implementation fails on queries with
  strings >256 bytes, with no way to disable the index for that query.
  Pass the ``disableContractPayloadIndexing=true`` option as part of
  ``--query-store-jdbc-config`` to disable this index when creating the
  schema.
  See `issue #10539 <https://github.com/digital-asset/daml/pull/10539>`__.

- [Integration Kit] Changes the Offset format to contain a version and therefore reduces the highest index size by one byte

- [Ledger API Server] The command deduplication time is no longer used
  for determining the period of time to track the command before giving
  up. Instead, the gRPC deadline is used. If no deadline is provided
  (or if the deadline exceeds the command tracker retention period), the
   tracker retention period is used instead.

- [Daml Repl] Fix a bug where bindings with out of scope types would result in error in following lines.

- [Sandbox, participant] Added a flag to enable a new append-only database schema.
  This schema was designed to support significantly higher performance.
  In a future release, all applications will automatically migrate to the new schema.
- [Ledger API Server] The command service now uses the tracker retention
  period (typically specified with the ``--tracker-retention-period``
  command-line argument) as the maximum time to wait for a command to
  arrive on the completion stream. After this time, the command will
  time out, though it may still complete in the future. Previously, the
  deduplication period was used, but it was likely the tracker would be
  terminated before that anyway.

  The default tracker retention period is 5 minutes, unless otherwise
  specified.
- [DPP-418] [Participant] Add support for supplying server's private key as an encrypted file and then decrypting it with the help of a secrets server.
[Integration Kit] KV-based ledgers pass contract instances through the Write/ReadService, removing the need
for backfilling divulged contract lookups.
Note: KV Ledgers that have been created before this change will still be relying on backfilling lookups of divulged contracts,
hence pruning of all divulged contracts may result in failing lookups for divulged contracts.
ledger-api - add `deduplication_duration` as a future replacement for `deduplication_time` in the command proto definition
ledger-api - Command deduplication period can now be specified by setting `deduplication_offset` instead of `deduplication_time` (only valid for v2 WriteService). This change is backwards compatible.
```

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-09-01 11:01:02 +02:00
Remy
c11323ddb1
LF: Refactor engine test reinterpret (#10724)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-01 09:57:16 +02:00
pbatko-da
a995fa3cd2
[DPP-574] Update docs about TLS: encrypted private key (#10727)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-01 09:52:11 +02:00
Dharmaraj Parmar
66970b7226
Not sharing the absolute Deadline object. (#10713)
Fixes #10383

CHANGELOG_BEGIN
- [Java Bindings] DamlLedgerClient.Builder allows to set a timeout for command using `withTimeout`.
CHANGELOG_END

Co-authored-by: Stefano Baghino <stefano.baghino@digitalasset.com>
2021-09-01 08:54:57 +02:00
Moritz Kiefer
c6c304b778
Improve script error on invalid script identifier format (#10702)
fixes #9271

changelog_begin
changelog_end
2021-09-01 08:45:49 +02:00
Moritz Kiefer
b748fd6b67
Support TLS in Daml Helper --json-api requests (#10709)
This PR accepts the --tls flag but rejects the flags to set certs
because those don’t really make sense here (or are at least not easily
settable).

Unfortunately no great way to test this without a huge amount of test
infrastructure to setup a custom TLS reverse proxy for tests.

If someone has an idea to add a reasonable test, I’m all ears.

changelog_begin

- [Daml Assistant] The `daml ledger` commands now accepts `--tls` in
  combination with `--json-api` to access a JSON API behind a TLS
  reverse proxy.

changelog_end
2021-09-01 08:45:42 +02:00
Robert Autenrieth
f058c2f16f
DPP-368 clean up flags (#10711)
* Make batching parallelism configurable

changelog_begin
changelog_end

* Fail if incompatible cli flags are used
2021-08-31 23:43:32 +00:00
tudor-da
90ad24fd6a
[Divulgence pruning] Prune immediate divulgence [DPP-513] (#10691)
* [Divulgence pruning] Prune disclosed contracts
* Adapt CommonStorageBackend.pruneEvents to prune all immediately divulged events
* Adapt ParticipantPruningIT to assert immediate divulgence pruning
* Adapt ParticipantPruningIT tests for divulgence pruning to assert the ACS before and after pruning

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt conformance tests

* Adapt disclosure query for Oracle

* Addressed review comments

* Rebased
2021-08-31 20:19:08 +00:00
fabiotudone-da
183934b89b
Command dedup: add columns to completions in append-only schemas [KVL-1057] (#10652)
* Augment completion.proto with deduplication-related info

CHANGELOG_BEGIN
CHANGELOG_END

* Explicitly specify fields not yet filled in when building Completion

* Time-based deduplication periods are measured in record time of completions

* Add deduplication_offset as a deduplication_period option

* Don't skip proto field numbers

* CompletionFromTransaction: use default Completion constructor

* submission_rank: reserve proto field for future use

* Add comment about reserved proto field

* Add command deduplication columns to completions in append-only schemas

CHANGELOG_BEGIN
CHANGELOG_END

* Remove duplicated `application_id` field from Postgres and H2 schemas

* Make `application_id` and `submission_id` nullable

* Re-generate migrations' sha256

* Tidy-up

* Add RW logic for new command dedup columns in completions

* Declare `deduplication_start` as a nullable column

* Fix UpdateToDbDtoSpec

* Fix merge from main

* Replace Either3 with an ADT in StorageBackendTestValues

* Test with non-zero nanos

* Fix UpdateToDbDtoSpec

* Also change Schema.scala

* Simplify DbDto.CommandCompletion construction

* Fix merge from `main` (migration SHAs and filename)

* Fix UpdateToDbDto: restore transactionId

* Fix completions' application_id: it must be non-nullable in all schemas, as in PSQL

* Actually use newly read columns

* Add DB round-trip tests for new fields in StorageBackendTestValues

* Format

* StorageBackendTestValues: property type someSubmissionId

* UpdateToDbDto: remove duplicated logic

* Simplify UpdateToDbDto.commandCompletion

* Fix comment in CompletionFromTransaction.toApiDeduplicationPeriod

* Fix CompletionFromTransaction.toApiCompletion

* Fix deduplication_offset in Schema

* Make parameters explicit in CompletionFromTransaction and CompletionStorageBackendTemplate calls

* Shorten error message in CompletionFromTransaction

* Fix StorageBackendTestsCompletions

* Fix CompletionFromTransaction

* Separate command dedup-related test cases in UpdateToDbDtoSpec

* Simplify further UpdateToDbDto.commandCompletion

* Simplify CompletionFromTransaction.toApiCompletion

* Format
2021-08-31 18:22:53 +00:00
Remy
27c13334b6
LF: Drop outdated TODOs (#10725)
* LF: Drop outdated TODOs

CHANGELOG_BEGIN
CHANGELOG_END

* Address Moritz' Review
2021-08-31 18:02:29 +00:00
Kamil Bozek
9be577a7fb
Enable pruning in the sandbox-classic when the append-only schema is used [DPP-567] (#10708)
* Enable pruning in the sandbox-classic when the append-only schema is used

CHANGELOG_BEGIN
- [Sandbox] - Participant pruning is enabled in the sandbox-classic when the append-only schema is used
CHANGELOG_END

* Added pruning conformance tests to the build definition of the sandbox-classic

* Changed pruning IT time to the wall-clock time

* Added a comment explaining changes made

* Do not enable pruning when pruneAllDivulgedContracts option is enabled
2021-08-31 18:41:41 +02:00
Remy
9f072aeba0
Ledger-API Conformance test for Contract ID V0 (#10717)
CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 18:17:29 +02:00
Robert Autenrieth
867547c579
DPP-368 enable append-only flag in sandbox (#10710)
* Enable append-only schema in sandbox

changelog_begin
changelog_end

* Add test

* Unhide the compression flag in sandbox

* Make append-only config non-optional
2021-08-31 16:46:53 +02:00
pbatko-da
bdc511eaa9
[DPP-438] Change open-ended metric names into static ones (by removing partyName part) (#10706)
* [DPP-438] Change open-ended metric names into static ones (by removing partyName part)

CHANGELOG_BEGIN
CHANGELOG_END

* revert changes in Ctx.scala

* 1
2021-08-31 16:42:41 +02:00
tudor-da
0c820063f1
[Divulgence pruning] Prune all divulged contracts only after migration offset [DPP-483] (#10661)
* Prune divulgence events before migration to append-only
 * Prune all divulgence events with NULL offset
 * Check pruning request against migration offset
 * Added unit test for migration offset validation

CHANGELOG_BEGIN
CHANGELOG_END

* Removed unnecessary discards
2021-08-31 14:16:36 +00:00
Gerolf Seitz
2555dbb30b
Use soft references for values in the caches (#10715)
Soft references allow the garbage collector to collect the values referenced
by soft references in case of memory pressure. Since these caches are not the
source of truth for the data, it's better to remove values from the cache
instead of having the process die due to OOM (not enough heap or excessive GC).

Fixes DPP-561.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 13:51:14 +00:00
Victor Peter Rouven Müller
7fd590694e
Add LedgerClientWithoutLedgerId next to the LedgerClient (#10681)
* Add the LedgerClientWithoutLedgerId class

* Minimize diff

changelog_begin
changelog_end

* Minimize diff further

* Add missing license header & reformat

* Update language-support/scala/bindings-akka/src/main/scala/com/digitalasset/ledger/client/binding/LedgerClientBinding.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update ledger/ledger-api-client/src/main/scala/com/digitalasset/ledger/client/LedgerClient.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Add changelog

changelog_begin

- [Ledger Client Scala Bindings] A new variant of the LedgerClient class
    was added called `LedgerClientWithoutLedgerId`. This class does not
    need a ledger id at initialization. It was added to allow skipping
    any checks at initialization for use cases where either the
    ledger id is not known at initalization or no valid token can be fed
    at initialization for checking the ledger id. Furthermore for each
    classes `ActiveContractSetClient`, `CommandClient`, `PackageClient`,
    `TransactionClient`, `VersionClient` now exists a variant which
    doesn't depend on a ledger id at initialization and instead requires
    one for every function as parameter. Moreover the existing classes
    are extending these classes with overriding the methods and setting
    the default of the parameter with the given ledger id from
    initialization. The class `LedgerClientWithoutLedgerId` already
    makes usage of these variants e.g. `PackageClientWithoutLedgerId`.

changelog_end

* More changelog

changelog_begin

- [Ledger Client Scala Bindings] The function `transactionSource` of the
    class `LedgerClientBinding` now optionally accepts a token which is
    passed on to the unterlying call.

changelog_end

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2021-08-31 14:11:52 +01:00
Samir Talwar
856c69c425
participant-integration-api: Increase a test timeout. (#10721)
The default patience timeout of 150ms is often too quick for CI. Let's
increase it to 1s.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 13:06:46 +00:00
Brian Healey
b86d07d498
remove non functioning oracle json array indices (#10720)
CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 12:59:46 +00:00
Samir Talwar
f5e17567f9
sandbox-classic: Remove default parameters in SqlLedger.Owner. (#10718)
* sandbox-classic: Remove default parameters in `SqlLedger.Owner`.

The defaults can be dangerous.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox-classic: Revert LedgerResource's index schema to the mutable one.
2021-08-31 12:37:19 +00:00
Remy
9ef3377864
LF: Update specification with Contract ID Comparability check (#10703)
This is part of #10504.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 13:01:25 +02:00
Stephen Compall
e5c4734780
enable JSON search index on Oracle by default (#10539)
* unconditionally enable JSON search index on Oracle

In '1kb of data' and larger Oracle integration tests:

ORA-29902: error in executing ODCIIndexStart() routine
ORA-20000: Oracle Text error:
DRG-50943: query token too long on line 1 on column 3

From https://docs.oracle.com/en/database/oracle/oracle-database/19/errmg/DRG-10000.html#GUID-46BC3B3F-4DB7-4EB4-85DA-55E9461966CB

    Cause: A query token is longer than 256 bytes

    Action: Rewrite query

* add changelog

CHANGELOG_BEGIN
- [JSON API] The Oracle database schema has changed; if using
  ``--query-store-jdbc-config``, you must rebuild the database by adding
  ``,start-mode=create-only``.  See #10539.
CHANGELOG_END

* test only 1kb

* extra flag in db config string

* let Queries backends configure themselves from maps

* new Queries constructor dataflow to better support config values

* remove fields as we go, isolating backend-specific from -agnostic conf

- we use StateT to avoid the problems that will definitely arise if we
  don't DRY.

* fix up DbConfig including DbStartupMode

* start to uncouple json-api's config from db-utils

* two JdbcConfigs with different purposes/scopes

- also moves db-utils contents to com.daml.dbutils

* adapt trigger service to refactoring

* fix JdbcConfig leftovers

* adapt http-json-cli to new JdbcConfig

* remove extra ConfigCompanion

* explain more about the QueryBackend/Queries distinction

* split SupportedJdbcDriver into two phases with a tparam

* use SupportedJdbcDriver.TC instead of SupportedJdbcDriver as the nullary typeclass

* patch around all the moved objects with imports

* missed import from moving ConnectionPool to dbutils

* use new 2-phase SupportedJdbcDriver for ContractDao setup

* left off part of a comment

* more q.queries imports

* other imports from the dbutils move

* nested JdbcConfig

* configure the driver in each backend-specific test

* very confusing error, but make the imports nicer and it goes away

* nested JdbcConfig in perf

* missing newline

* port contractdao-bench

* test new option parsing all the way through QueryBackend

* disable search index for some tests, enable for others

* add changelog

CHANGELOG_BEGIN
- [Trigger Service] ``--help`` no longer advertises unsupported JDBC
  options from JSON API.

- [JSON API] [EE only] By default, on Oracle, sets up a JSON search
  index to speed up the queries endpoints.  However, Oracle versions
  prior to 19.12 have an unrecoverably buggy implementation of this
  index; in addition, the current implementation fails on queries with
  strings >256 bytes, with no way to disable the index for that query.

  Pass the ``disableContractPayloadIndexing=true`` option as part of
  ``--query-store-jdbc-config`` to disable this index when creating the
  schema.
  See `issue #10539 <https://github.com/digital-asset/daml/pull/10539>`__.
CHANGELOG_END

* port failure tests

* init version table last, drop first

- suggested by @realvictorprm; thanks

* rename split DBConfig.scala

- suggested by @realvictorprm; thanks

* move imports to not be in alphabetical order

- suggested by @realvictorprm; thanks

* remove createSchema

- suggested by @realvictorprm; thanks

* Revert "test only 1kb"

This reverts commit 616e173e63.

* port to scala 2.12

- bug in unused imports

- old name `-` for `removed`
2021-08-31 09:29:36 +00:00
pbatko-da
1ded42f185
[DPP-418] Protect TLS keys - follow-up cleanup (#10696)
CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 10:42:47 +02:00
Hubert Slojewski
3fcd986f0b
Introduce a new Offset format [KVL-1063] (#10668)
CHANGELOG_BEGIN

- [Integration Kit] Changes the Offset format to contain a version and therefore reduces the highest index size by one byte

CHANGELOG_END
2021-08-31 08:29:15 +00:00
azure-pipelines[bot]
a5781a6682
update NOTICES file (#10714)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-08-31 06:58:13 +00:00
Samir Talwar
89855054b5
participant-integration-api: Use deadlines, not deduplication times, for expiring trackers. [KVL-1009] (#10704)
* participant-integration-api: On submit and wait, capture a deadline.

We need this deadline to make sure we terminate the stream.

* ledger-api-client: Use the specified expiry time for tracking.

Not the command deduplication time.

CHANGELOG_BEGIN
- [Ledger API Server] The command deduplication time is no longer used
  for determining the period of time to track the command before giving
  up. Instead, the gRPC deadline is used. If no deadline is provided
  (or if the deadline exceeds the command tracker retention period), the
   tracker retention period is used instead.
CHANGELOG_END

* ledger-api-client: Keep supporting the `deduplication_time` timeout.

* ledger-api-client: Improve comments in CommandTrackerFlow and its tests.

Co-Authored-By: Miklos <57664299+miklos-da@users.noreply.github.com>
2021-08-30 19:22:43 +00:00
Samir Talwar
65025c26b8
sandbox-classic: Add ErrorInfo metadata for rejections. [KVL-1048] (#10707)
* sandbox-classic: Add more tests for `SqlLedger#publishTransaction`.

* sandbox-classic: Simplify the SqlLedger test constructors.

* sandbox-classic: Improve the out of bounds assertions in SqlLedgerSpec.

* sandbox-classic: Add ErrorInfo metadata for rejections.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox-classic: Enable the append-only index schema in tests.

* sandbox-classic: Assert rejection details in SqlLedgerSpec.

* sandbox-classic: Simplify `Rejection`.

Inline methods that aren't used externally.

* sandbox-classic: Extract out some duplication in SqlLedgerSpec.
2021-08-30 18:42:00 +00:00
Remy
97bda3ca36
LF: V1 Contract ID check in Preprocessor (#10687)
This PR makes possible to reject V0 contract IDs during preprocessing.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-30 20:14:55 +02:00
mziolekda
c2f90efbba
Add CLI option to force disabling of participant deduplication (#10698)
* Add CLI option to force disabling of participant deduplication

CHANGELOG_BEGIN
CHANGELOG_END

* address review comments
2021-08-30 17:45:59 +00:00
Moritz Kiefer
6016633bb7
Construct ParsedModule directly in Daml Repl (#10701)
This is a relatively large change unfortunately which unfortunately
requires reimplementing parts of the logic of the typechecker & core
compilation. I don’t think it is too bad but we might want to think
over time if we can factor this better.

This fixes #10073 and fixes #10664 by referencing the exact types
instead of going via the renamer.

There are some minor changes around error messages for "module not
found" errors. This is because these are now caught in the
typechecker instead of in our own code. We could keep the errors but
it requires duplicating even more logic and I don’t really see what it
buys us so I think I prefer the approach here.

changelog_begin

- [Daml Repl] Fix a bug where bindings with out of scope types would result in error in following lines.

changelog_end
2021-08-30 17:28:16 +02:00
Robert Autenrieth
bbdf16aacf
DPP-368 unhide append-only CLI flags (#10697)
* Unhide the append-only schema flags

changelog_begin
- [Sandbox, participant] Added a flag to enable a new append-only database schema.
  This schema was designed to support significantly higher performance.
  In a future release, all applications will automatically migrate to the new schema.
changelog_end

* Improve CLI flag descriptions
2021-08-30 15:23:16 +02:00
Samir Talwar
a41b1349df
Use the tracker retention period as the maximum expiry time. [KVL-1009] (#10700)
* ledger-api-client: Rename `maxDeduplicationTime` to `maximumExpiryTime`.

* ledger-api-client: Add tests for command tracker timeouts.

* ledger-api-client: Cap the deduplication period at the expiry time.

* participant-integration-api: Make `trackerRetentionPeriod` a JDuration.

* ledger-api-client: Don't use `JDuration` if we can avoid it.

* participant-integration-api: Add a test for command service timeout.

* Use the tracker retention period as the maximum expiry time.

CHANGELOG_BEGIN
- [Ledger API Server] The command service now uses the tracker retention
  period (typically specified with the ``--tracker-retention-period``
  command-line argument) as the maximum time to wait for a command to
  arrive on the completion stream. After this time, the command will
  time out, though it may still complete in the future. Previously, the
  deduplication period was used, but it was likely the tracker would be
  terminated before that anyway.

  The default tracker retention period is 5 minutes, unless otherwise
  specified.
CHANGELOG_END

* participant-integration-api: Add a test for TrackerMap parameters.

* participant-integration-api: Use `maximumCommandTimeout`.

Instead of `maximumExpiryTime`.

* ledger-api-client: Shorten timeouts in CommandTrackerFlowTest.
2021-08-30 13:21:14 +00:00
Robert Autenrieth
e750ba5c45
Make warning less scary. (#10699)
The removed line is implicit to our support guarantees.

changelog_begin
changelog_end
2021-08-30 14:59:17 +02:00