Commit Graph

7696 Commits

Author SHA1 Message Date
Robert Autenrieth
867547c579
DPP-368 enable append-only flag in sandbox (#10710)
* Enable append-only schema in sandbox

changelog_begin
changelog_end

* Add test

* Unhide the compression flag in sandbox

* Make append-only config non-optional
2021-08-31 16:46:53 +02:00
pbatko-da
bdc511eaa9
[DPP-438] Change open-ended metric names into static ones (by removing partyName part) (#10706)
* [DPP-438] Change open-ended metric names into static ones (by removing partyName part)

CHANGELOG_BEGIN
CHANGELOG_END

* revert changes in Ctx.scala

* 1
2021-08-31 16:42:41 +02:00
tudor-da
0c820063f1
[Divulgence pruning] Prune all divulged contracts only after migration offset [DPP-483] (#10661)
* Prune divulgence events before migration to append-only
 * Prune all divulgence events with NULL offset
 * Check pruning request against migration offset
 * Added unit test for migration offset validation

CHANGELOG_BEGIN
CHANGELOG_END

* Removed unnecessary discards
2021-08-31 14:16:36 +00:00
Gerolf Seitz
2555dbb30b
Use soft references for values in the caches (#10715)
Soft references allow the garbage collector to collect the values referenced
by soft references in case of memory pressure. Since these caches are not the
source of truth for the data, it's better to remove values from the cache
instead of having the process die due to OOM (not enough heap or excessive GC).

Fixes DPP-561.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 13:51:14 +00:00
Victor Peter Rouven Müller
7fd590694e
Add LedgerClientWithoutLedgerId next to the LedgerClient (#10681)
* Add the LedgerClientWithoutLedgerId class

* Minimize diff

changelog_begin
changelog_end

* Minimize diff further

* Add missing license header & reformat

* Update language-support/scala/bindings-akka/src/main/scala/com/digitalasset/ledger/client/binding/LedgerClientBinding.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update ledger/ledger-api-client/src/main/scala/com/digitalasset/ledger/client/LedgerClient.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Add changelog

changelog_begin

- [Ledger Client Scala Bindings] A new variant of the LedgerClient class
    was added called `LedgerClientWithoutLedgerId`. This class does not
    need a ledger id at initialization. It was added to allow skipping
    any checks at initialization for use cases where either the
    ledger id is not known at initalization or no valid token can be fed
    at initialization for checking the ledger id. Furthermore for each
    classes `ActiveContractSetClient`, `CommandClient`, `PackageClient`,
    `TransactionClient`, `VersionClient` now exists a variant which
    doesn't depend on a ledger id at initialization and instead requires
    one for every function as parameter. Moreover the existing classes
    are extending these classes with overriding the methods and setting
    the default of the parameter with the given ledger id from
    initialization. The class `LedgerClientWithoutLedgerId` already
    makes usage of these variants e.g. `PackageClientWithoutLedgerId`.

changelog_end

* More changelog

changelog_begin

- [Ledger Client Scala Bindings] The function `transactionSource` of the
    class `LedgerClientBinding` now optionally accepts a token which is
    passed on to the unterlying call.

changelog_end

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2021-08-31 14:11:52 +01:00
Samir Talwar
856c69c425
participant-integration-api: Increase a test timeout. (#10721)
The default patience timeout of 150ms is often too quick for CI. Let's
increase it to 1s.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 13:06:46 +00:00
Brian Healey
b86d07d498
remove non functioning oracle json array indices (#10720)
CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 12:59:46 +00:00
Samir Talwar
f5e17567f9
sandbox-classic: Remove default parameters in SqlLedger.Owner. (#10718)
* sandbox-classic: Remove default parameters in `SqlLedger.Owner`.

The defaults can be dangerous.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox-classic: Revert LedgerResource's index schema to the mutable one.
2021-08-31 12:37:19 +00:00
Remy
9ef3377864
LF: Update specification with Contract ID Comparability check (#10703)
This is part of #10504.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 13:01:25 +02:00
Stephen Compall
e5c4734780
enable JSON search index on Oracle by default (#10539)
* unconditionally enable JSON search index on Oracle

In '1kb of data' and larger Oracle integration tests:

ORA-29902: error in executing ODCIIndexStart() routine
ORA-20000: Oracle Text error:
DRG-50943: query token too long on line 1 on column 3

From https://docs.oracle.com/en/database/oracle/oracle-database/19/errmg/DRG-10000.html#GUID-46BC3B3F-4DB7-4EB4-85DA-55E9461966CB

    Cause: A query token is longer than 256 bytes

    Action: Rewrite query

* add changelog

CHANGELOG_BEGIN
- [JSON API] The Oracle database schema has changed; if using
  ``--query-store-jdbc-config``, you must rebuild the database by adding
  ``,start-mode=create-only``.  See #10539.
CHANGELOG_END

* test only 1kb

* extra flag in db config string

* let Queries backends configure themselves from maps

* new Queries constructor dataflow to better support config values

* remove fields as we go, isolating backend-specific from -agnostic conf

- we use StateT to avoid the problems that will definitely arise if we
  don't DRY.

* fix up DbConfig including DbStartupMode

* start to uncouple json-api's config from db-utils

* two JdbcConfigs with different purposes/scopes

- also moves db-utils contents to com.daml.dbutils

* adapt trigger service to refactoring

* fix JdbcConfig leftovers

* adapt http-json-cli to new JdbcConfig

* remove extra ConfigCompanion

* explain more about the QueryBackend/Queries distinction

* split SupportedJdbcDriver into two phases with a tparam

* use SupportedJdbcDriver.TC instead of SupportedJdbcDriver as the nullary typeclass

* patch around all the moved objects with imports

* missed import from moving ConnectionPool to dbutils

* use new 2-phase SupportedJdbcDriver for ContractDao setup

* left off part of a comment

* more q.queries imports

* other imports from the dbutils move

* nested JdbcConfig

* configure the driver in each backend-specific test

* very confusing error, but make the imports nicer and it goes away

* nested JdbcConfig in perf

* missing newline

* port contractdao-bench

* test new option parsing all the way through QueryBackend

* disable search index for some tests, enable for others

* add changelog

CHANGELOG_BEGIN
- [Trigger Service] ``--help`` no longer advertises unsupported JDBC
  options from JSON API.

- [JSON API] [EE only] By default, on Oracle, sets up a JSON search
  index to speed up the queries endpoints.  However, Oracle versions
  prior to 19.12 have an unrecoverably buggy implementation of this
  index; in addition, the current implementation fails on queries with
  strings >256 bytes, with no way to disable the index for that query.

  Pass the ``disableContractPayloadIndexing=true`` option as part of
  ``--query-store-jdbc-config`` to disable this index when creating the
  schema.
  See `issue #10539 <https://github.com/digital-asset/daml/pull/10539>`__.
CHANGELOG_END

* port failure tests

* init version table last, drop first

- suggested by @realvictorprm; thanks

* rename split DBConfig.scala

- suggested by @realvictorprm; thanks

* move imports to not be in alphabetical order

- suggested by @realvictorprm; thanks

* remove createSchema

- suggested by @realvictorprm; thanks

* Revert "test only 1kb"

This reverts commit 616e173e63.

* port to scala 2.12

- bug in unused imports

- old name `-` for `removed`
2021-08-31 09:29:36 +00:00
pbatko-da
1ded42f185
[DPP-418] Protect TLS keys - follow-up cleanup (#10696)
CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 10:42:47 +02:00
Hubert Slojewski
3fcd986f0b
Introduce a new Offset format [KVL-1063] (#10668)
CHANGELOG_BEGIN

- [Integration Kit] Changes the Offset format to contain a version and therefore reduces the highest index size by one byte

CHANGELOG_END
2021-08-31 08:29:15 +00:00
azure-pipelines[bot]
a5781a6682
update NOTICES file (#10714)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-08-31 06:58:13 +00:00
Samir Talwar
89855054b5
participant-integration-api: Use deadlines, not deduplication times, for expiring trackers. [KVL-1009] (#10704)
* participant-integration-api: On submit and wait, capture a deadline.

We need this deadline to make sure we terminate the stream.

* ledger-api-client: Use the specified expiry time for tracking.

Not the command deduplication time.

CHANGELOG_BEGIN
- [Ledger API Server] The command deduplication time is no longer used
  for determining the period of time to track the command before giving
  up. Instead, the gRPC deadline is used. If no deadline is provided
  (or if the deadline exceeds the command tracker retention period), the
   tracker retention period is used instead.
CHANGELOG_END

* ledger-api-client: Keep supporting the `deduplication_time` timeout.

* ledger-api-client: Improve comments in CommandTrackerFlow and its tests.

Co-Authored-By: Miklos <57664299+miklos-da@users.noreply.github.com>
2021-08-30 19:22:43 +00:00
Samir Talwar
65025c26b8
sandbox-classic: Add ErrorInfo metadata for rejections. [KVL-1048] (#10707)
* sandbox-classic: Add more tests for `SqlLedger#publishTransaction`.

* sandbox-classic: Simplify the SqlLedger test constructors.

* sandbox-classic: Improve the out of bounds assertions in SqlLedgerSpec.

* sandbox-classic: Add ErrorInfo metadata for rejections.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox-classic: Enable the append-only index schema in tests.

* sandbox-classic: Assert rejection details in SqlLedgerSpec.

* sandbox-classic: Simplify `Rejection`.

Inline methods that aren't used externally.

* sandbox-classic: Extract out some duplication in SqlLedgerSpec.
2021-08-30 18:42:00 +00:00
Remy
97bda3ca36
LF: V1 Contract ID check in Preprocessor (#10687)
This PR makes possible to reject V0 contract IDs during preprocessing.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-30 20:14:55 +02:00
mziolekda
c2f90efbba
Add CLI option to force disabling of participant deduplication (#10698)
* Add CLI option to force disabling of participant deduplication

CHANGELOG_BEGIN
CHANGELOG_END

* address review comments
2021-08-30 17:45:59 +00:00
Moritz Kiefer
6016633bb7
Construct ParsedModule directly in Daml Repl (#10701)
This is a relatively large change unfortunately which unfortunately
requires reimplementing parts of the logic of the typechecker & core
compilation. I don’t think it is too bad but we might want to think
over time if we can factor this better.

This fixes #10073 and fixes #10664 by referencing the exact types
instead of going via the renamer.

There are some minor changes around error messages for "module not
found" errors. This is because these are now caught in the
typechecker instead of in our own code. We could keep the errors but
it requires duplicating even more logic and I don’t really see what it
buys us so I think I prefer the approach here.

changelog_begin

- [Daml Repl] Fix a bug where bindings with out of scope types would result in error in following lines.

changelog_end
2021-08-30 17:28:16 +02:00
Robert Autenrieth
bbdf16aacf
DPP-368 unhide append-only CLI flags (#10697)
* Unhide the append-only schema flags

changelog_begin
- [Sandbox, participant] Added a flag to enable a new append-only database schema.
  This schema was designed to support significantly higher performance.
  In a future release, all applications will automatically migrate to the new schema.
changelog_end

* Improve CLI flag descriptions
2021-08-30 15:23:16 +02:00
Samir Talwar
a41b1349df
Use the tracker retention period as the maximum expiry time. [KVL-1009] (#10700)
* ledger-api-client: Rename `maxDeduplicationTime` to `maximumExpiryTime`.

* ledger-api-client: Add tests for command tracker timeouts.

* ledger-api-client: Cap the deduplication period at the expiry time.

* participant-integration-api: Make `trackerRetentionPeriod` a JDuration.

* ledger-api-client: Don't use `JDuration` if we can avoid it.

* participant-integration-api: Add a test for command service timeout.

* Use the tracker retention period as the maximum expiry time.

CHANGELOG_BEGIN
- [Ledger API Server] The command service now uses the tracker retention
  period (typically specified with the ``--tracker-retention-period``
  command-line argument) as the maximum time to wait for a command to
  arrive on the completion stream. After this time, the command will
  time out, though it may still complete in the future. Previously, the
  deduplication period was used, but it was likely the tracker would be
  terminated before that anyway.

  The default tracker retention period is 5 minutes, unless otherwise
  specified.
CHANGELOG_END

* participant-integration-api: Add a test for TrackerMap parameters.

* participant-integration-api: Use `maximumCommandTimeout`.

Instead of `maximumExpiryTime`.

* ledger-api-client: Shorten timeouts in CommandTrackerFlowTest.
2021-08-30 13:21:14 +00:00
Robert Autenrieth
e750ba5c45
Make warning less scary. (#10699)
The removed line is implicit to our support guarantees.

changelog_begin
changelog_end
2021-08-30 14:59:17 +02:00
azure-pipelines[bot]
5f120bdf87
rotate release duty after 1.17.0-snapshot.20210824.7647.0.640fb683 (#10660)
@SamirTalwar-DA is taking care of 1.17.0-snapshot.20210824.7647.0.640fb683 (#10659), so they get pushed back to the end of the line.

Please do not merge this before #10659.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-08-30 12:14:38 +02:00
Robert Autenrieth
a17253fb63
DPP-535 Verify postgres version (#10577)
* Check Postgres version at startup

changelog_begin
changelog_end

* Check compatiblity when creating datasource

* Remove unused import

* remove unused import

* Fix version parsing

* Fix compiler error

* Simplify code

* Don't use the patch version

It might be missing or contain non-number characters

* Revert needless change

* Fix regular expression

* Add test
2021-08-30 10:20:34 +02:00
Samir Talwar
301ce53a71
participant-integration-api: Add tests for ApiCommandService. [KVL-1009] (#10689)
* participant-integration-api: Make `LocalServices` a non-case class.

* participant-integration-api: Split `ApiCommandService.LocalServices`.

The tracker uses the completion functions, and the service uses the
transaction functions, but we don't need them in one place.

CHANGELOG_BEGIN
CHANGELOG_END

* participant-integration-api: Add tests for `ApiCommandService`.
2021-08-30 09:54:40 +02:00
pbatko-da
bd01a211f4
[DPP-418] Protect Participant TLS keys (#10629)
Adding support for accepting server's private key as an encrypted file (since storing unencrypted private key in a file system might be a risk).

Encrypted private key is assumed to be encrypted using AES or similar algorithm. The details necessary to decrypt it are be obtained from a secrets server over HTTP as JSON document. The URL to secret's server is supplied through the new `--secrets-url` CLI parameter.

One can supply private in either plaintext (old behavior) or ciphertext: if a private key's file ends with .enc suffix it is assumed to be ciphertext. Otherwise it is assumed to be plain text.

CHANGELOG_BEGIN
- [DPP-418] [Participant] Add support for supplying server's private key as an encrypted file and then decrypting it with the help of a secrets server.
CHANGELOG_END
2021-08-30 09:24:52 +02:00
azure-pipelines[bot]
7ee13243e1
update NOTICES file (#10695)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-08-30 07:08:53 +00:00
azure-pipelines[bot]
7c392f32dc
update NOTICES file (#10693)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-08-28 07:07:20 +00:00
azure-pipelines[bot]
3db654e9dd
update NOTICES file (#10690)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-08-27 08:28:03 +00:00
Samir Talwar
eff09c010a
ledger-api-client: Wrap command submissions in a new class. [KVL-1009] (#10683)
* bindings-akka: Parameterize RetryInfo with its value type.

* ledger-api-client: Wrap command submissions in a new class.

This will allow me to pass a deadline alongside the commands in the
future.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-api-client: Inline `CommandRetryFlow.Value`.

* bindings-akka: Inline `CommandRetryFlow`'s `createRetry`.

It's always the same: `retryInfo.value`.
2021-08-26 17:44:12 +02:00
Remy
d54adb2543
Ledger-API: Conformance tests for contract IDs suffixing (#10654)
This is part of #10504

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-26 16:14:02 +02:00
tudor-da
aa2e869ede
[Divulgence pruning] Pass divulged contract arguments through kvutils Write/ReadService [DPP-535] (#10598)
* Pass divulged contracts in KVUtils

CHANGELOG_BEGIN
[Integration Kit] KV-based ledgers pass contract instances through the Write/ReadService, removing the need
for backfilling divulged contract lookups.
Note: KV Ledgers that have been created before this change will still be relying on backfilling lookups of divulged contracts,
hence pruning of all divulged contracts may result in failing lookups for divulged contracts.
CHANGELOG_END

* Enriched TransactionCommitterSpec with assertions for divulgence and disclosure

* Refactored
* Removed DamlTransactionEntry.divulgedContracts
* Added the contractInstance to DivulgenceEntry

* Refactored
* Removed DamlTransactionEntry.divulgedContracts
* Added the contractInstance to DivulgenceEntry

* Addressed review comments

* Addressed Miklos' review comments
2021-08-26 13:34:26 +00:00
Oliver Seeliger
1a7831390e
Disable DeeplyNestedValueIT suite against canton in Daml repo (#10686)
Due to flakes. The same DeeplyNestedValueIT suite runs apparently
without flakes in the canton repo.

changelog_begin
changelog_end
2021-08-26 14:54:53 +02:00
Samir Talwar
b5f9be3fbc
participant-integration-api: Standardize tracker retention naming. (#10682)
Let's be clear that the "retention period" is for tracking.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-26 13:56:22 +02:00
Samir Talwar
2aa632e3aa
ledger-on-sql: Do not increment the dispatcher head on start. (#10684)
* ledger-on-sql: Ensure all databases start auto-incrementing from 1.

* ledger-on-sql: Do not increment the dispatcher head on start.

I don't know how this ever made it in, because it's clearly wrong.

Unfortunately we don't have tests that restart the participant/ledger
server while keeping the database intact, so this is very difficult to
test at a high level.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-26 13:43:12 +02:00
nicu-da
eabb19d7f7
[ledger-api] Add deduplication_duration to deduplication period [kvl-1047] (#10676)
* Add `deduplication_duration` to `deduplication_period` and deprecate `deduplication_time`

CHANGELOG_BEGIN
ledger-api - add `deduplication_duration` as a future replacement for `deduplication_time` in the command proto definition
CHANGELOG_END

* Add tests for deduplication period validation
2021-08-26 09:33:01 +00:00
tudor-da
96ad9b5ab8
[Divulgence pruning] All divulgence events pruning [DPP-483] (#10634)
* All divulgence events pruning
* Add `pruneAllDivulgedContracts` to `IndexParticipantPruningService`
* Add `participant_pruned_all_contracts_up_to_inclusive` in `parameters` table
* `pruneEvents` prunes all divulgence events
* Adapt `ParticipantPruningIT` to check for divulgence with and without `pruna_all_divulged_contracts`
* Create multi-participant pruning test in ledger-on-memory

CHANGELOG_BEGIN
CHANGELOG_END

* CommonStorageBackend
* Enrich pruning descriptions
* Logging for command completions pruning
* Move divulgence pruning back to the beginning of pruning sequence

* Addressed review comments

* PRDivulgenceArchivalPruning adapted for privacy-aware ledgers

* Rebased to main

* Disable ParticipantPruningIT tests targeting append-only schema in mutable schema conformance tests in ledger-on-sql

* Adapted order of SQL migration queries

* Documentation of appendonlydao.JdbcLedgerDao.prune

* Apply suggestions from code review

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
2021-08-26 08:45:01 +00:00
Andreas Herrmann
0b7980d6eb
Update rules_haskell (#10674)
* update rules_haskell

changelog_begin
changelog_end

* Build proto3-suite lib and exe in same repo

changelog_begin
changelog_end

* drop unneeded stack_snapshot packages

* Use packages.bzl consistently

* Sort stack_snapshot attribute values

* Remove unused constant

* drop unneeded stack_snapshot packages in compat

changelog_begin
changelog_end

* Sort stack_snapshot attribute values in compat

* Remove redundant dicts.add in compat

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-08-26 10:41:58 +02:00
Marton Nagy
284edfc4cd
Fix FlywayMigrations datasource (#10666)
* Use StorageBackend.createDatasource instead
* remove dependencies
* refactor FlywayMigrations interface for simplicity

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-26 02:22:52 +02:00
Leonid Rozenberg
adbe65fae1
Document ActionFail vs CanAbort (#10657)
CHANGELOG_BEGIN
CHANGELOG_END
2021-08-25 20:51:29 +02:00
azure-pipelines[bot]
52e7a6d928
update compat versions for 1.17.0-snapshot.20210824.7647.0.640fb683 (#10667)
CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-08-25 16:29:37 +02:00
Moritz Kiefer
f42e6b6877
Expose pending contracts in triggers (#10672)
fixes #10671

changelog_begin
changelog_end
2021-08-25 13:23:15 +00:00
nicu-da
7cc698948c
Add multiple ways of specifying deduplication [KVL-1047] (#10601)
CHANGELOG_BEGIN
ledger-api - Command deduplication period can now be specified by setting `deduplication_offset` instead of `deduplication_time` (only valid for v2 WriteService). This change is backwards compatible.
CHANGELOG_END

* Propagate the enriched deduplicationPeriod instead of deduplication duration

* Update the Haskell bindings for the new deduplication period

* Calculate the deduplicateUntil using the new deduplication period for backward compat

* Use consistent naming for deduplication_period

* Cleanup command timeout extraction from deduplication period

* Add the required deduplication_offset to deduplication instead of deduplication_start

* Update haskell bindings to support deduplication_offset

* Add support for deduplication_offset in the ledger-api

* Remove the timestamp-based deduplication from our models to simplify upgrade for users

* Add optional conformance test for offset based deduplication

* Remove buf rule for FIELD_SAME_ONEOF as our change is backwards compatible

* Disable FIELD_SAME_ONEOF buf check for commands file

* Apply suggestions from code review

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Update comment for deduplication period

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2021-08-25 05:58:03 -07:00
Samir Talwar
53be19f86c
participant-integration-api: Ensure that all waiting, failed, and closed trackers are cleaned up. (#10662)
* participant-integration-api: Clean up failed trackers.

Otherwise they can hang around forever.

CHANGELOG_BEGIN
CHANGELOG_END

* participant-integration-api: Ensure that all trackers are closed.

* participant-integration-api: Ensure that waiting trackers are closed.

* participant-integration-api: Store the tracker map future in the state.

* participant-integration-api: Fix a race in `TrackerMap`.

If the supplied `Future` to `AsyncResourceState` is too fast, the state
may not be set. We need to initialize the state immediately.

* participant-integration-api: Add more comments to `TrackerMap`.
2021-08-25 12:53:13 +00:00
Samir Talwar
b27cde6440
participant-integration-api: Move tracker code around, and tidy up tests. (#10663)
* participant-integration-api: Delete `SizeCappedMap`. It's unused.

* participant-integration-api: `TrackerImpl` → `QueueBackedTracker`.

CHANGELOG_BEGIN
CHANGELOG_END

* participant-integration-api: Clean up QueueBackedTrackerSpec.
2021-08-25 10:55:26 +00:00
Moritz Kiefer
fc9d35935a
Drop alternative rules from dlint config (#10646)
We don’t have Alternative in daml-stdlib so these don’t make any sense.

changelog_begin
changelog_end
2021-08-25 12:52:51 +02:00
Moritz Kiefer
5204d3ce7a
Include committers in PartialTransaction root context (#10665)
This gets us into a state where PartialTransaction always has the
authorization context which is much more sensible than having the
committers in speedy while the rest is in ptx.

The reason for having the split in the first place was the scenario
service but now that we create a new speedy machine per submission
that’s no longer a valid reason.

changelog_begin
changelog_end
2021-08-25 10:45:45 +00:00
nickchapman-da
387c68b9eb
Normalize transaction values within the engine (#10648)
Create normalized TXs when a partial TX is finalised.

Except in limited cases! (i.e for scenario-runner, sandbox)

CHANGELOG_BEGIN
CHANGELOG_END

normalize values in the engine as they are converted from speedy-values

fix 2.12 build

backout redundant change

ensure byKey field is correctly normalized when constructed by engine

rename flag: valueNormalization -> transactionNormalization

improve comment

delete commented-out code

rename: toValueNorm --> toNormalizedValue

rename: (SValue.) toValue --> toUnNormalizedValue

revert changes to ptx so that the interface to insertCreate() etc is Value-based (not SValue-based)

improve comments

respell: toUnNormalizedValue --> toUnnormalizedValue

fix build
2021-08-25 09:53:26 +01:00
azure-pipelines[bot]
430c1cccf3
release 1.17.0-snapshot.20210824.7647.0.640fb683 (#10659)
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.

@SamirTalwar-DA is in charge of this release.

Commit log:
```
640fb68342 Make Index DB enable multiple party additions [DPP-546] (#10623)
b22de6893b LF: Contract ID suffix check in Preprocessor (#10642)
7b94b0674e Map shortened scala test suite names to long names on Windows (#10628)
6e4a24c28c participant-state: Generate correct gRPC error codes by v2 `WriteService` [KVL-1081] (#10656)
663781a159 Update curl 7.73.0 --> 7.78.0 (#10655)
a471f15aa8 Dpp-558 Fix startexclusive queries on oracle (#10649)
e99254fd3e Augment `completion.proto` with deduplication-related info [KVL-1057] (#10619)
a00608c8b3 participant-integration-api: Accommodate changes to max dedup time. (#10650)
29c546c949 [Divulgence pruning] Added `prune_all_divulged_contracts` to PruneRequest [DPP-534] (#10635)
dea57ca099 In-memory fan-out optimizations (#10558)
77eb366627 [JSON-API] key_hash field to speed up fetchByKey queries (#10631)
5001329c9d LF: Comparisons fail at runtime if comparing local vs global CIDs (#10630)
055be4b3cb Disable deprecation warnings for data-dependencies (#10647)
c1559357f6 clean-up gsg README (#10641)
8501832757 DPP-468 StorageBackend tests (#10529)
4e08b47dd1 update NOTICES file (#10645)
733590d591 ledger-api-health: Use the Scala health status values everywhere. (#10640)
5b837ec2d9 Ledger API: add `buf` checks [KVL-1045] (#10625)
f77cd0a81c participant-integration-api: Attempt to fix RecoveringIndexerSpec. (#10639)
9d7f60f03e participant-integration-api: Fix a flaky test. (#10637)
4a9331ca55 Upgrade Nixpkgs [KVL-1045] (#10624)
01b6e89551 update compat versions for 1.17.0-snapshot.20210817.7604.0.0c187853 (#10610)
b578b0e7cf Reminder to put an empty line between subject and body (#10638)
c0fbad1f6e participant-integration-api: Remove `limitMaxCommandsInFlight`. (#10626)
b4af6d1155 Canton testing: Mark one more DeeplyNestedValueIT test flaky (#10636)
e807f4a918 Upgrade to a newer canton version (post 0.27.0 snapshot version) (#10632)
c4513f2eea Oracle append-only schema: enable contract id index on participant_events_xxxx tables (#10633)
3598e0928c LF: Drop contract ID Freshness check (#10620)
37c999e237 ledger-on-sql: Increase the concurrency for conformance tests. (#10622)
46e8c7d762 DPP-460 Extract constant for event sequential IDs (#10564)
121047e0ea DPP-460 Parameter storage consolidation (#10472)
569612ab73 Drop broken symlink config (#10616)
6d0109fca8 Support $ in daml-lf identifiers in the parser (#10609)
c38703ec84 participant-integration-api: Store a status gRPC protobuf. [KVL-1005] (#10600)
0af5b49484 make FinalReason a case class (#10614)
8dd136fc7d bazel-tools: Replace `runner` with either `runner_with_port_check` or `runner_with_port_file`. (#10615)
19c3d28ebf Remove GenMissingString class because it is not used (#10608)
3227e860e0 Use the port file and dynamic port generation in client/server tests. (#10604)
fb19bcb7f3 fix gsg-trigger template (#10611)
b2077026d6 rotate release duty after 1.17.0-snapshot.20210817.7604.0.0c187853 (#10606)
975a5fb32b Move DeduplicationPeriod to ledger-api-domain [KVL-1047] (#10590)
f8a1820cc8 release 1.17.0-snapshot.20210817.7604.0.0c187853 (#10605)
64abf8aaa9 update NOTICES file (#10607)
```
Changelog:
```
[Integration Kit] Corrected gRPC error codes returned by v2 `WriteService` adaptor.
- [Ledger API Server] The API server manages a single command tracker
  per (application ID × submitters) pair. This tracker would read the
  current ledger configuration's maximum deduplication time on creation,
  but never updated it, leading to trackers that might inadvertently
  reject a submission when it should have been accepted. The tracker now
  reads the latest ledger configuration.
- Update schema version for http-json-api query store with new key_hash field
- Improved performance for fetchByKey query which now uses key_hash field
participant-state - move `DeduplicationPeriod` to ledger-api-domain
dc4629fff9 update NOTICES file (#10580)
8b0a0e7bbc update NOTICES file (#10578)
4b8b67a1b5 Upgrade Scalatest to v3.2.9. (#10576)
41e60f7302 Upgrade to Scala 2.12.14 and 2.13.6. (#10573)
c447898cc9 Fix display of unhandled exceptions in the scenario service (#10572)
4e1a90dffd Enable --incompatible_remote_results_ignore_disk (#10571)
d183ecc6d6 rotate release duty after 1.17.0-snapshot.20210811.7560.0.4f9de4ba (#10556)
76ecb44fbd update compat versions for 1.17.0-snapshot.20210811.7565.0.f1a55aa4 (#10563)
86a03fa7c0 Bump bazel max jvm memory (#10569)
1cc136c033 update NOTICES file (#10565)
c69880ccf1 ledger-api-test-tool: Enforce test naming standards. (#10562)
ee34d0f80c Track command - use types for error handling instead of grpc statuses [KVL-1005] (#10503)
93c25f36b6 Release 1.17 snapshot (#10560)
```
Changelog:
```
- [Ledger API Test Tool] The ``TransactionServiceIT`` test suite has
  been split into many test suites. If you are including or excluding
  it, you will need to use the new test suite names, or you can use
  "TransactionService" as a prefix for all of them.
  If you are including or excluding individual tests, you will need to
  update your arguments with the new test suite. You can find the new
  test suite by running the test tool with the ``--list-all``
  flag and looking for the test's short identifier. The short
  identifiers have not changed, with the exception of
  ``TXNoContractKey``, which has been renamed to ``CKNoContractKey`` and
  is now in the ``ContractKeysIT`` test suite.
* [Daml export] You can now set the ``--all-parties`` option to generate
  a ledger export as seen by all known parties.
ledger-api-client - Propagate definite_answer as metadata in the GRPC response for submit/submitAndWait
[JSON API] Ledger connection errors are now logged at every attempt
akka-bindings: `LedgerClientBinding.commands` now returns a flow of `Either[CompletionFailure, CompletionSuccess]` instead of `Completion` for clearer error handling. For backwards compatiblity the new return type can be turned back into a `Completion` using `CompletionResponse.toCompletion`
```

```

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
2021-08-25 08:43:57 +00:00
Samir Talwar
ef239fdd1a
participant-integration-api: Move TrackerMap code around. [KVL-1009] (#10653)
* participant-integration-api: Make `TrackerMap` a `Tracker`.

Mostly by moving parameters around.

CHANGELOG_BEGIN
CHANGELOG_END

* participant-integration-api: Clean up `TrackerMap` a little more.

* participant-integration-api: Make `TrackerMap` generic.

* participant-integration-api: Move `Tracker.WithLastSubmission`.

It's only used by `TrackerMap`, so now it's an internal class there.

* participant-integration-api: Only provide the key to `newTracker`.

* participant-integration-api: Subsume the `TrackerMap` cleanup schedule.

* participant-integration-api: Construct the tracker map outside.

* ledger-api-common: Remove some unnecessary braces.

* participant-integration-api: Prettify `TrackerMap` some more.

* participant-integration-api: Make `TrackerMap.selfCleaning` a class.

* participant-integration-api: Add some tests for TrackerMap.

* participant-integration-api: Convert a method to `Runnable`, 2.12-style.

Apparently underscores aren't good enough.

* ledger-api-client: Delete CompletionSuccess#unapply.

It doesn't work on Scala 2.12.
2021-08-25 06:30:39 +00:00
Marton Nagy
640fb68342
Make Index DB enable multiple party additions [DPP-546] (#10623)
Preparation, small fixes

* Remove unnecessary TODO
* Fix FieldStrategy.idempotentInsert type parameters

Changing the DB-Schema

This change adapts the DB schema for all supported backends. After this change we only populate the party_entries table, and on the query side we reconstruct the state from this.

* Drop the party table
* Add indexing to party_entries

Adapting StorageBackend: ingestion

Since we only ingest party_entries, the party population needs to be removed.

* Drop the party table in ingestion code
* Fixes test

Adapting StorageBackend: queries

Queries needs to be adapted to construct the state from the read side.

* Rewrite queries.
* Fixes reset implementations.

Adapting JdbcLedgerDao

Since underlying storage changed, JdbcLedgerDao can be simplified: no special treatment needed with duplicate errors, since these errors are impossible to happen.
Removing JdbcLedgerDao tests, and adding a new test, testing the behavior of the new event-source party model. Please note: this database refactoring only applies to the append-only schema, so for the mutating schema the test is disabled.
During implementation a bug surfaced: it was not possible anymore to store the is_local information via JdbcLedgerDao.storePartyEntry. Although this bug is a minor issue, since that method is only used from single participant environment, still a fix was also implemented for this, by passing a magic participantId upon non-local party storage.

* Simplify storePartyEntry.
* Fixes bug introduced by append-only.
* adds/adapts tests

Refactoring: remove not used duplicateKeyError from StorageBackend

Changes to JdbcLedgerDao rendered this duplicateKeyError unused.

* Removes unused duplicateKeyError

Adapting sandbox-classic

In sandbox-classic it is not allowed to have updates for parties. Essentially the updates concerning already existent parties were dropped silently with logging without effect.
Here I started by pinning down this behaviour in the SqlLedgerSpec and SqlLedgerSpecAppendOnly. These tests were implemented with the original code in mind.
Then adapted the SqlLedger method: making sure of uniqueness by first trying to lookup the to-be-persisted party.

* Added tests grabbing a hold on original behavior
* Adapted implementation to ensure same behavior

Switching to correct is_local derivation for party queries as per review

* Adapting implementation: switching to aggregated table and a query on that
* Introducing QueryStrategy.booleanOrAggregationFunction to support Oracle
* Moving party related queries to PartyStorageBackendTemplate
* Fixes JdbcLedgerDaoPartiesSpec tests, and add another test case

Also:

* Align Update interface documentation
* Switching to explicit optionality in party query implementation asa per review

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-24 19:24:27 +02:00