Commit Graph

159 Commits

Author SHA1 Message Date
Samir Talwar
a9f6afbfde
kvutils: Rename VersionedOffset to KVOffset. (#11286)
There is no longer a non-versioned offset, so this is kind of
meaningless.

I also deleted `VersionedOffsetMutator`, which should not have made it
in.

CHANGELOG_BEGIN
CHANGELOG_END
2021-10-19 09:34:19 +00:00
fabiotudone-da
98cf8d86b3
KV: introduce v2 error codes behind a CLI switch [KVL-1140] (#11224)
* Propagate error version switch to KeyValueConsumption

CHANGELOG_BEGIN
CHANGELOG_END

* Introduce v2 (self-service) KV error codes behind the CLI switch

* Test v2 codes and fix them

* Keep newline separating methods in KeyValueConsumption

* TransactionRejections: don't wrap updates in `Some`

* Factor errorVersionsTable

* Reorder imports

* Split "convert rejection to proto models and back to expected grpc code"

* Remove unneeded Option returned by decodeTransactionRejectionEntry

* Formatting fix

* Fix 7537e93d
2021-10-19 08:32:57 +00:00
Samir Talwar
8f94cffdd1
kvutils: Use VersionedOffsetBuilder where possible, and introduce VersionedOffsetMutator. [KVL-1154] (#11277)
* kvutils: Remove `VersionedOffsetBuilder.apply`.

Doesn't do anything.

* ledger-on-memory: Use `VersionedOffsetBuilder`.

* indexer-benchmark: Use `VersionedOffsetBuilder`.

* ledger-on-sql: Use `VersionedOffsetBuilder`.

* kvutils: Use `VersionedOffsetBuilder` in tests.

* kvutils: Create a case class for `VersionedOffsetBuilder#split`.

* kvutils: Delete unused methods from the offset builders.

* kvutils: Use `Bytes#startWith` to check the offset version.

* kvutils: Move offset splitting into `VersionedOffset`.

* kvutils: Extract out versioned offset generators.

* kvutils: Replace `OffsetBuilder` with `VersionedOffsetMutator`.

This takes care of the last usages of `OffsetBuilder`, which were to
modify the lowest component of the offset.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Randomize the offset version in testing.

To make sure we don't use a hard-coded offset version anywhere.

* kvutils: `Random.between` is not available in Scala 2.12.

* kvutils: Move offset mutation methods to `VersionedOffset`.

* kvutils: Move the versioned offset construction into `VersionedOffset`.

The `VersionedOffsetBuilder` is still useful as it remembers the
version, meaning we only need to specify it once.
2021-10-18 16:50:52 +00:00
nicu-da
68a93cf70f
kvutils - Use the ledger configured time for command dedup [KVL-1149] (#11239)
CHANGELOG_BEGIN
kvutils - Command deduplication uses the configured time model (static/wall) and not running always using wall-clock
CHANGELOG_END
2021-10-15 05:56:45 -07:00
nicu-da
b6f3f2087a
Extract the log_entry from daml_kvutils.proto [KVL-1090] (#11193)
CHANGELOG_BEGIN
CHANGELOG_END
2021-10-12 11:23:11 +00:00
Robert Autenrieth
66adbc0366
DPP-463 switch conformance tests to append only (#11101)
* Use append-only schema in tests

changelog_begin
changelog_end

* Fix Oracle build

* Update ledger/ledger-on-memory/BUILD.bazel

Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>

* Run more conformance tests on append-only

* Run pruning tests in split-participant mode

* Disable multi-participant test in split participant

* Add missing ContractIdIT test

Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
2021-10-11 23:31:30 +02:00
nicu-da
7dd9c2d3f0
Remove expectations for internal failures from parallel command dedup tests (#11061)
CHANGELOG_BEGIN
CHANGELOG_END
2021-09-29 05:07:01 -07:00
Samir Talwar
eb87b3439b
kvutils: Add the logging context for ledger state operations. (#11030)
We do some logging in proprietary ledger drivers which would benefit
from correct propagation of the context.

CHANGELOG_BEGIN
- [Integration Kit] We have added ``loggingContext`` as an implicit
  parameter to more _kvutils_ trait methods. Implementors may need to do
  the same in their trait implementations. This can make it easier to
  log with the appropriate context.
CHANGELOG_END
2021-09-28 15:25:34 +02:00
nicu-da
f4adee91ca
Add conformance test for command deduplication using the CommandService [KVL-1099] (#10883)
* Add conformance test for command deduplication using the CommandService

CHANGELOG_BEGIN

CHANGELOG_END
2021-09-16 11:16:39 +00:00
nicu-da
b4328b3dc3
ledger-api-test-tool - Add conformance test for parallel command deduplication using CommandSubmissionService [KVL-1099] (#10869)
* Extract deduplication "features" into a configuration to be used around the tests.
Better naming for assertions that support sync and async deduplication

CHANGELOG_BEGIN

CHANGELOG_END

* Fix broken test and use consistency for tests

* ledger-api-test-tool - Add conformance test for parallel command deduplication

CHANGELOG_BEGIN
CHANGELOG_END

* Add import for 2.12 compat

* Add silencer plugin

* Split parallel command deduplication scenario into it's own test suite

* Add the parallel command deduplication test to append only ledgers

* Run parallel command deduplication tests for append only ledgers

* Apply suggestions from code review

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>

* Code review renames

* Add compat import

* Run the test concurrently
2021-09-15 12:15:13 +00:00
fabiotudone-da
be4e06427f
Ledger API Test Tool: support --additional tests [KVL-1100] (#10829)
* Support adding tests as an hidden option

* Simplify existing suites

CHANGELOG_BEGIN
CHANGELOG_END

* Remove stale conformance suites from build.yml

* `--add` -> ``--additional`

* Re-add `--all-tests` as deprecated CLI option to be tested

* Move sandbox-classic pruning test to wall clock again

* Run KVCommandDeduplicationIT for sandbox append-only

* Tidy-up

* Also add participant pruning test to ledger-on-memory/single-participant

* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only

* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only

* Add KVCommandDeduplicationIT to ledger-on-memory append-only

* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant

* Tidy-up

* Use KVCommandDeduplicationIT for sandbox-on-x too

* Fix merge

Add max dedup duration arg to all the test suites that include command dedup tests

* Make `--include` and `--additional` mutually exclusive

* Uniform formatting of multi-line strings

* Move exclusions after additions as they are applied last

* Re-disable deduplication test on sandbox with static time

* Re-disable deduplication test on sandbox-on-x
2021-09-14 14:00:30 +02:00
nicu-da
8a9d19a99b
Command deduplication - KV conformance test for usage of max deduplication duration [KVL-1098] (#10846)
* Add conformance test for KV committer usage of max deduplication duration as deduplication duration

CHANGELOG_BEGIN

CHANGELOG_END

* Run AppendOnlyKVCommandDeduplicationIT for KV ledgers that use the append-only schema
2021-09-13 06:48:31 -07:00
nicu-da
1c4f173f3a
Command deduplication - kvutils - Always use max deduplication duration as deduplication period [KVL-1098] (#10824)
* Always use max_deduplication_duration as deduplication period for committer side deduplication

Doing this we get the following guarantees:
- Deduplication period is constant for all the submissions therefore this guarantees consistency between "backward-looking" and "forward-looking" deduplication

CHANGELOG_BEGIN
kvutils - committer side deduplication always uses max_deduplication_duration + min_skew as a deduplication period for all the requests.
CHANGELOG_END

* Set max_deduplication_duration to 10 seconds for all the conformance tests

* Update KV command deduplication conformance tests to account for max deduplication duration used as deduplication period for all the requests

* Add max deduplication wait for multi-participant conformance test

* Overwrite the submitter info deduplication period in the transaction committer so that we always set the max deduplication duration

* Update ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/participant/state/kvutils/committer/transaction/TransactionCommitterSpec.scala

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>

* Add separate step for overwriting the deduplication period

* Code review - reformatting and renames

* Add comment for setting the deduplication duration
2021-09-10 09:05:21 -07:00
fabiotudone-da
a03f52a15e
Add ledger API tests about command deduplication information in completions [KVL-1057] (#10748)
* Add Ledger API Test Tool tests about command deduplication information present in completions

CHANGELOG_BEGIN
CHANGELOG_END

* Address review comments

* Fix offset reference and move new tests to a separate, optional suite

* Cover rejections as well

* Test both the command and command submission services

* Run new test suite (only) on sandbox-classic append-only and daml-on-sql

* Use the append-only schema with daml-on-sql for CommandDeduplicationInfoIT

* Fix tests except offset, successful completions only.

* Remove completion offset test as it's not supported by most ledgers

* Remove support for multiple submissions as rejection completions are not being tested for the moment

* Consolidate test cases for faster run

* Avoid forbidden characters in short identifiers

* Clarify assert

* Remove wrong test about deduplication time being preserved in completion

* Adhere to the Scala style guide

* Eliminate some code duplication

* fmt

* Make 3e404be compile

* Never assume a specific deduplication period format in completions

* Code cleanup

* Pass party to completionStreamRequest to avoid ILLEGAL_ARG failures

* Enable in append-only mode for all ledgers that support it

* Clarify that the new tests are append-only-only

* Update ledger/ledger-on-sql/BUILD.bazel

Fix suite name

* Update ledger/sandbox-classic/BUILD.bazel

Fix suite name

* fmt
2021-09-09 07:56:14 +00:00
nicu-da
60ffb79fb1
Command deduplication - Participant side/committer side command deduplication conformance tests split [KVL-1093] (#10784)
* Split command deduplication conformance tests into non-kv and kv tests.

Because kv ledgers have both participant and committer side deduplication, the conformance tests have been duplicated into different suited which can target either both deduplications or just participant side deduplication.
CommandDeduplicationIT is a default suite and should be run for ledgers with participant side only deduplication.
KVCommandDeduplicationIT is optional and should be enabled for any KV ledgers.

CHANGELOG_BEGIN
ledger-api-test-tool - CommandDeduplicationIT tests only participant side deduplication, it should be disabled for KV ledgers. Added KVCommandDeduplicationIT which should be run for KV ledgers
CHANGELOG_END

* Run participant side/committer side deduplication tests based on the ledger implementation

* Fix on memory build file

* Daml-on-sql is not a kv ledger so revert the changes so that it runs the default command deduplication IT

* Rollback compat import

* Exclude command dedup from the reference test

* Exclude command deduplication from the reference export

* Format bazel

* Set the exclude to client args for the ledger-api-test-tool and not for the ledger-on-memory

* Exclude CommandDeduplicationIT from non repudiation conformance tests and include KVCommandDeduplicationIT because it runs using sandbox

* Exclude HealthServiceIT from the reference-ledger-export as it has no impact on the export

* Exclude CommandDeduplicationIT from the ledger-on-sql conformance-test as they are KV ledgers

* Format bazel file

* Exclude CommandDeduplicationIT from sandbox

* Apply suggestions from code review

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>

* Add comments for command deduplication IT

* Rename to be consistent with scala namings

* Update formatting and comments

* Add naming prefix for tests to avoid clashes

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
2021-09-08 10:27:07 +00:00
tudor-da
4b7391b0d5
LedgerApiTestTool prints skipped (excluded) tests (#10726)
* Document multi-participant-only pruning test

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-06 13:50:54 +02:00
nicu-da
41881ba2ba
Command dedup: migrate kvutils to use v2 services [KVL-1049] (#10679)
* Add deduplication and submission id to the proto submitter info so we can build the v2 completion info

* Use v2 rejections and store the enriched model in protobuf

* Update v1 SubmissionResult to v2 SubmissionResult

* Update v1 imports to v2 imports

* Enable API deduplication for ledgers that don't have it implemented yet (all kv)

* Update tests for new rejections

* Disable buf check for oneof

* Code review - fix formatting and test assertions

* Update imports for proto changes

* Add tests for rejection conversions

* Use entry id as submission id when submission id is missing

* Calculate deduplication period for rejection based on the set deduplicate until by calculating the duration relative to record time

* Code review consistent naming and comments
* Submission id is optional in the completion info

the submissionId is not set for participant.state.v1 entries, therefore for the transition from v1 to the v2 API we need to account for the missing submission ids by making it optional in the completion info.

* Update messages and grpc status codes for rejections to keep backward compatibility

* Deprecate proto deduplicate_until
* Set deduplication value in the commit context for all the deduplication periods. This change allows us to re-enable the committer deduplication tests

* Set ledger configuration for tests

* Command dedup: In the committer set the deduplicate until context value based on the given deduplication period [KVL-1049] (#10743)


CHANGELOG_BEGIN
participant-state:  
- Migrate to use only v2 Read/Write Services. This includes the use of new models for rejections/updates/submission results.
- Calculate deduplicate until for committer side deduplication based on the submitter given deduplication period
   -  if using the deprecated deduplicateUntil then just set the given timestamp
   -  if using duration then calculate the new deduplicateUntil by using the formula (submissionTime + deduplicationDuration + minSkew)
  -   if using offset deduplication then calculate deduplicateUntil by using the formula (submissionTime + maxDeduplicationDuration + minSkew)
  -  if the deduplication period is not set then we don't set deduplicateUntil
- Emit completions with status `ALREADY_EXISTS` for duplicate commands

CHANGELOG_END
2021-09-02 04:44:55 -07:00
tudor-da
90ad24fd6a
[Divulgence pruning] Prune immediate divulgence [DPP-513] (#10691)
* [Divulgence pruning] Prune disclosed contracts
* Adapt CommonStorageBackend.pruneEvents to prune all immediately divulged events
* Adapt ParticipantPruningIT to assert immediate divulgence pruning
* Adapt ParticipantPruningIT tests for divulgence pruning to assert the ACS before and after pruning

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt conformance tests

* Adapt disclosure query for Oracle

* Addressed review comments

* Rebased
2021-08-31 20:19:08 +00:00
Remy
9f072aeba0
Ledger-API Conformance test for Contract ID V0 (#10717)
CHANGELOG_BEGIN
CHANGELOG_END
2021-08-31 18:17:29 +02:00
Remy
d54adb2543
Ledger-API: Conformance tests for contract IDs suffixing (#10654)
This is part of #10504

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-26 16:14:02 +02:00
Samir Talwar
2aa632e3aa
ledger-on-sql: Do not increment the dispatcher head on start. (#10684)
* ledger-on-sql: Ensure all databases start auto-incrementing from 1.

* ledger-on-sql: Do not increment the dispatcher head on start.

I don't know how this ever made it in, because it's clearly wrong.

Unfortunately we don't have tests that restart the participant/ledger
server while keeping the database intact, so this is very difficult to
test at a high level.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-26 13:43:12 +02:00
tudor-da
96ad9b5ab8
[Divulgence pruning] All divulgence events pruning [DPP-483] (#10634)
* All divulgence events pruning
* Add `pruneAllDivulgedContracts` to `IndexParticipantPruningService`
* Add `participant_pruned_all_contracts_up_to_inclusive` in `parameters` table
* `pruneEvents` prunes all divulgence events
* Adapt `ParticipantPruningIT` to check for divulgence with and without `pruna_all_divulged_contracts`
* Create multi-participant pruning test in ledger-on-memory

CHANGELOG_BEGIN
CHANGELOG_END

* CommonStorageBackend
* Enrich pruning descriptions
* Logging for command completions pruning
* Move divulgence pruning back to the beginning of pruning sequence

* Addressed review comments

* PRDivulgenceArchivalPruning adapted for privacy-aware ledgers

* Rebased to main

* Disable ParticipantPruningIT tests targeting append-only schema in mutable schema conformance tests in ledger-on-sql

* Adapted order of SQL migration queries

* Documentation of appendonlydao.JdbcLedgerDao.prune

* Apply suggestions from code review

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
2021-08-26 08:45:01 +00:00
Samir Talwar
37c999e237
ledger-on-sql: Increase the concurrency for conformance tests. (#10622)
This was limited a while back during the initial development of
_ledger-on-sql_ for reasons I can't remember. Let's stop doing that.

CHANGELOG_BEGIN
CHANGELOG_END
2021-08-19 14:04:09 +00:00
Robert Autenrieth
121047e0ea
DPP-460 Parameter storage consolidation (#10472)
* Refactor ParameterStorageBackend

- a single method for atomic ledger initialization
- a single method to look up the ledger end

changelog_begin
changelog_end

* Add a test

* Fix reading event sequential ids

* Remove debug statements

* Allow ledgerEnd on an empty database

* Initialization is not safe to call concurrently

* Remove leftovers from isolation level change

* Use unit return type

for initialization methods

* Allow getParticipantId on an empty database

* Use exceptions instead of a return type ADT

* Don't use Try for initialization

* Clean up parameters table

* Simplify parameter storage api

* Address review suggestion

* Address review comment

* Address review comment

* Prefer ledger id over participant id mismatch

* Address review comment

* Move type definition

* Remove useleess new keyword

* Renove unused import

* Inline result mapping

* Fix reporting of mismatching participantId
2021-08-19 10:45:17 +02:00
Samir Talwar
4b8b67a1b5
Upgrade Scalatest to v3.2.9. (#10576)
* Upgrade Scalatest to v3.2.9.

Because of some coupling we also have to upgrade Scalaz to the latest
v7.2 point release, v7.2.33.

The Scalatest changes are quite involved because the JAR has been broken
up into several smaller JARs. Because Bazel expects us to specify all
dependencies and doesn't allow transitive dependencies to be used
directly, this means that we need to specify the explicit Scalatest
components that we use.

As you can imagine, this results in quite a big set of changes. They
are, however, constrained to dependency management; all the code remains
the same.

CHANGELOG_BEGIN
CHANGELOG_END

* http-json-oracle: Fix a Scalatest dependency.

* ledger-api-client: Fix a Scalatest dependency.
2021-08-12 23:19:35 +00:00
Samir Talwar
fe1c678531
sandbox: Attempt to make the reset service tests less flaky. (#10507)
* sandbox: Do less when resetting.

We don't need to reconstruct some basic things.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: When testing the reset service, keep trying.

If we scale up the test duration, we also need to scale up the timeouts.

* timer-utils: If `RetryStrategy` eventually fails, explain what happened.

For example, if the attempts limit is exceeded, say so.

* Revert "timer-utils: If `RetryStrategy` eventually fails, explain what happened."

This reverts commit 6647681688.
2021-08-06 15:26:25 +00:00
Samir Talwar
07da936421
participant-integration-api: Always wait for the first config lookup. (#10500)
* participant-integration-api: Always wait for the first config lookup.

This means we don't have to guess how long a `lookupConfiguration()`
round trip will take. Instead we can just submit the initial
configuration if one is not found.

Some drivers may still need to wait a while for other reasons.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Explain `delayBeforeSubmitting`.

Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>

Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
2021-08-06 12:47:31 +00:00
Samir Talwar
df78f9cb1e
Replace LedgerConfiguration with InitialLedgerConfiguration or the load timeout. [KVL-1058] (#10487)
* participant-integration-api: Encapsulate the initial configuration.

* participant-integration-api: Reduce usage of `LedgerConfiguration`.

* Inline `LedgerConfiguration` wherever it's used.

Most things don't need all its constituent parts; this reduces the
amount of unused properties.

CHANGELOG_BEGIN
- [Integration Kit] The ``LedgerConfiguration`` class has been
  removed in favor of ``InitialLedgerConfiguration``. Its usage
  has been changed accordingly, with the ``configurationLoadTimeout``
  property becoming part of ``ApiServerConfig`` instead.

  The default options provided by ``LedgerConfiguration`` have been
  removed; you are now encouraged to come up with sensible values for
  your own ledger. The ``Configuration.reasonableInitialConfiguration``
  value may help.
CHANGELOG_END

* Correct the initial configuration submission delay for KV ledgers.

* kvutils: Mark supertype unused parameters as unused.

* kvutils: Extract out common configuration submission delays.

These values are specific to kvutils; other drivers should come up with
their own.

* configuration: Delete `NoGeneration`, as it's unused.
2021-08-05 16:31:45 +00:00
mziolekda
f1c7548b7a
remove unstable marker for append-only features (#10465)
CHANGELOG_BEGIN
CHANGELOG_END
2021-08-03 10:09:11 +00:00
Remy
72cf2f36d3
LF: replace bazel keyword stable by default (#10410)
to refer to the compiler default LF output.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-27 09:27:03 +02:00
Samir Talwar
bd12170af2
participant-state: Remove the ParticipantId, PackageId, and Party aliases. [KVL-1002] (#10308)
* participant-state: Remove the `ParticipantId` alias.

This alias adds nothing. By using `Ref.ParticipantId` directly, many
packages can remove their dependency on the _participant-state_ package.

CHANGELOG_BEGIN
CHANGELOG_END

* participant-state: Remove the `PackageId` and `Party` aliases.

They don't add anything. Let's just use `Ref`.

* kvutils: Restore missing compat imports.
2021-07-19 12:31:25 +00:00
Samir Talwar
cb29f34d4b
ledger-offset: Move Offset to a new package. [KVL-1002] (#10296)
* ledger-offset: Move `Offset` to a new package.

CHANGELOG_BEGIN
- [Integration Kit] The ``Offset`` type has been moved to a new Maven
  package, ``ledger-offset``, from the ``participant-state`` package.
  The Java package has been renamed to ``com.daml.ledger.offset``. If
  you are using this type, you will need to update your dependencies and
  imports.
CHANGELOG_END

* Avoid rewrapping offsets for no reason.

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>

* participant-integration-api: Sort some imports.

* participant-integration-api: Fix dependencies for the Oracle tests.

I didn't add `ledger-offset`.

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2021-07-15 17:53:03 +02:00
Samir Talwar
70b12f04d0
ledger-on-sql: Remove conformance tests for H2 and SQLite files. (#10291)
We don't use these in any kind of production setting, and it reduces the
CI overhead a lot.

CHANGELOG_BEGIN
CHANGELOG_END
2021-07-15 13:31:25 +00:00
Samir Talwar
a9a0b70b85
ledger-configuration: Extract the configuration from participant-state. [KVL-1002] (#10279)
* ledger-configuration: Extract configuration from participant-state.

The configuration is often used without the state, and doesn't need to
be versioned in the same way.

CHANGELOG_BEGIN
- [Integration Kit] The ledger configuration classes, ``Configuration``,
  ``LedgerInitialConditions``, and ``TimeModel``, have been moved from
  *participant-state* to a separate package named
  *ledger-configuration*, in the Java package
  ``com.daml.ledger.configuration``. You will need to update your
  dependencies and imports.
CHANGELOG_END

* participant-state: Remove the `LedgerId` aliases.

* ledger-configuration: Rename `TimeModel` to `LedgerTimeModel`.

This avoids confusion with the protobuf-generated `TimeModel` classes.

CHANGELOG_BEGIN
- [Integration Kit] ``TimeModel`` has been renamed to
  ``LedgerTimeModel``. If you are using the ledger configuration classes
  directly, you may need to update your code.
CHANGELOG_END

* ledger-configuration: Remove colons in LedgerInitialConditions' docs.

* kvutils: Restore a missing compat import.

* participant-integration-api: Add ledger-configuration to Oracle tests.

* sandbox-common: Fix `--max-ledger-time-skew` docs.

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2021-07-15 13:03:19 +00:00
tudor-da
ba8e829c8d
[In-memory fan-out] Oracle support (#10263)
* [In-memory fan-out] Oracle support

CHANGELOG_BEGIN
CHANGELOG_END

* Correct so Oracle backend works with fan out enabled - add implicit conversion and correct view defaulting for BLOB fields
Add missing tests to oracle append only spec as it lagged behind postgres and h2
Regen sql sha for oracle

CHANGELOG_BEGIN
CHANGELOG_END

* avoid global import of implicit that is not needed

* reduce duplication on conformance test targets

* update sha for oracle append only flyway scripts

Co-authored-by: Brian Healey <brian.healey@digitalasset.com>
2021-07-14 21:10:12 +00:00
Samir Talwar
1f35db17c7
ledger-on-sql: Use random log entry ID allocation so we don't depend on SeedService. [KVL-1002] (#10255)
* ledger-on-sql: Abstract out the log entry ID allocator.

This is so we don't need to depend on `SeedService`.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Remove `allocateRandomLogEntryId` from production code.

* kvutils: Remove `SubmissionValidator.create`, as it's only for tests.

And remove default parameters for
`SubmissionValidator.createForTimeMode`.

* ledger-on-sql: Use the random log entry ID allocator.

In Sandbox, we still use the seed service, but this is not necessary
for ledger-on-sql tests.

* kvutils: Make `LogEntryIdAllocator.random` an `object`.

* ledger-on-sql: Move `RandomLogEntryIdAllocator` here.

It's not used anywhere else.

* ledger-on-sql: Add some documentation about the application code.

Just to make it clear it's not used in production.
2021-07-13 17:10:30 +00:00
Brian Healey
678cab0327
10050 append only schema on oracle (#10051)
* Oracle compliant append only schema

CHANGELOG_BEGIN
CHANGELOG_END

WIP : oracle on new appendonly schema

* diff to postgres dump, create consolidated view

* diff to postgres dump to ensure all oracle setup is equiv

* recompute sha for changed oracle flyway scripts

* drop old tables to prevent clash on name of new participant_events view

* recompute sha for flyway script

* prelim oracle StorageBackend

* Adds support for special preparedStatement for oracle

* Add support and wires to setObject by default everywhere
* Add the full OracleField suite with TODOs for convenience
* Wires OracleField suite to OracleFieldStrategy

changelog_begin
changelog_end

* enable debug version of oracle driver

* conversion Instant -> Timestamp for oracle

* WIP: primitive println debugging

* Passing PackagesSpec with appendonlyschema on Oracle
Rename size column to siz to avoid reserved word clash, including migration script for postgres

* include sha for new postgres migration script

* add missing copyright header

* cleanup

* passing party spec for appendonly on oracle

* passing configuration spec for appendonly on oracle

* scalafmt

* bazel buildifier reformat

* use db generic FETCH NEXT n ROW ONLY rather than limit for cross db compat

* siz instead of size for packages table on all dbs and schema

* revert enabling oracle jdbc debug

* Support Array[String] -> String conversion (and vice versa) for JSON array
Remove as aliases for tables as this does not work with oracle
Extract submitters clause for all db types
Use append transaction injector for oracle append only spec

* scalafmt

* correct oracle failing active contract spec tests

* wire in JdbcLedgerDaoCompletionsSpec

* remove semi-colons for ending statements that are problematic for oracle driver

* all tests up to divulgence passing for append only on oracle

* all appendonly tests passing on oracle

* remove ignore on fall back to limit-based query with consistent results

* do not change name of size column in packages table for mutable schema for all DBs

* do not change name of size column in packages table for mutable schema for all DBs

* standalone oracle appendonly schema script
regen shas on flyway scripts
revert some cosmetic refactoring in CommonStorageBackend

* Fixes conversion to parties from Oracle-JSON at flatEventWitnessesColumn

* Switches from composit SQLs to single SQLs at prepared statements to accommodate Oracle limitation

* Fixes arrayIntersectionWhereClause by applying patch from mutable Oracle schema integration

* Fixes queries with empty startExclusive Offsets

* First draw adding Oracle conformance test suites to CI

* wire in the oracle conformance tests for CI

* Use cross-db fetch next $n rows only syntax instead of limit syntax that works only for postgres/h2

* rename siz to package_size

* recompute shas

* scalafmt and include sha check for oracle append only flyway script

* correct missing package_size rename

* remove some todos -- correct corrupted V1__Init.sql

* Update ledger/ledger-on-sql/src/test/lib/scala/com/daml/ledger/on/sql/MainWithEphemeralOracleUser.scala

Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>

* correct version number for postgres rename column scripts

* remove unnecessary migration tables for oracle append only

* review feedback: rename createEventFilter as requested, remove todos

* review feedback: case consistency

* review feedback: update todos with issue markers

* review feedback: cleanup

* review feedback: OracleField and OracleSchema cleanup

* Fixing Table generators to use preparedData for convenience

* Placing TODOs for refactorings later

* Renames initial append-only oracle script, for convenience

* Falls back to original behavior as far prepared statements go at couple of queries

Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
2021-06-28 18:51:48 -04:00
Marton Nagy
0fbc1ee67d
H2 support for append only schema (part2) [DPP-394] (#10030)
* Preparation: pull up initialisation
* Preparation: introduce partyArrayContext for more flexibility
* Preparation: add append-only-from-scratch feature to FlywayMigrations
* Add squashed Flyway migration for append-only H2
* Add support for H2 StorageBackend
* Extend conformance test suites for H2
* Extend JDBCLedgerDao unit test suites for H2
* Extend sandbox-classic unit test suites for H2
* Fixing typo, some renames based on review

changelog_begin
changelog_end
2021-06-17 15:34:25 +02:00
tudor-da
e49c7c8db7
[In-memory fan-out] BufferedTransactionsReader implementation [DPP-413] [DPP-414] (#9882)
* [In-memory fan-out] BufferedTransactionsReader implementation
* getTransactions
* getTransactionTrees

CHANGELOG_BEGIN
CHANGELOG_END

* Address review comments
2021-06-15 16:29:53 +02:00
Remy
cbcec32e70
LF: clean up the remaining Exception TODOs (#9913)
CHANGELOG_BEGIN
CHANGELOG_END
2021-06-08 11:49:02 +02:00
tudor-da
b4247c1511
[In-memory fan-out] BuffersUpdater implementation (#9858)
* [In-memory fan-out] BuffersUpdater implementation
* Instantiated the TransactionsBuffer in ReadOnlySqlLedgerWithMutableCache
* Implementations and tests for the BuffersUpdater which updates:
* * The MutableCacheBackedContractStore
* * TransactionsBuffer
* Introduced `buffered-ledger-api-streams-unsafe` CLI flag for enabling in-memory fan-out flow
* Enabled the `buffered-ledger-api-streams-unsafe` in ledger-on-sql conformance test for append-only schema

CHANGELOG_BEGIN
CHANGELOG_END

* [In-memory fan-out] BuffersUpdater implementation
* Instantiated the TransactionsBuffer in ReadOnlySqlLedgerWithMutableCache
* Implementations and tests for the BuffersUpdater which updates:
* * The MutableCacheBackedContractStore
* * TransactionsBuffer
* Introduced `buffered-ledger-api-streams-unsafe` CLI flag for enabling in-memory fan-out flow
* Enabled the `buffered-ledger-api-streams-unsafe` in ledger-on-sql conformance test for append-only schema

CHANGELOG_BEGIN
CHANGELOG_END
2021-06-07 09:21:12 +00:00
tudor-da
a645971e72
Enable mutable-contract-state-cache for ledger-on-SQL append-only tests (#9836)
CHANGELOG_BEGIN
CHANGELOG_END
2021-05-28 15:33:38 +00:00
Victor Peter Rouven Müller
beb2d138f2
Add cli option & system property to enable json only logging for http json service (#9725)
changelog_begin
- [Ledger HTTP Json Service] Logging output can now be in JSON either via providing the cli option `--log-encoder json` or via setting the env var `LOG_FORMAT_JSON=true`
changelog_end
2021-05-20 13:28:34 +00:00
Stephen Compall
ca9e89b3da
check whether collection.compat is unused when compiling for Scala 2.12 (#9604)
* check whether collection.compat is unused when compiling for Scala 2.12

- Instead of always suppressing warnings for collection.compat._,
  we should only do it for Scala 2.13

- We can also reduce boilerplate by automatically adding this
  option when both silencer_plugin and collection-compat are
  present

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused import

* remove another unused import

* remove even more unused imports

* missed compat dependency

* more missed compat dependencies

* missed compat dependency

* use scala_deps in scaladoc_jar

- #8423 inlined the major version expansion, but this seems to
  have been prior to proper support by scaladoc_jar

* restore custom handling of participant-integration-api

- fixing scaladoc_jar isn't worth it for a single case, as with
  deps vs scala_deps
2021-05-11 21:54:14 +00:00
Kamil Bożek
39afcc4857
Fix index size limit issues [DPP-347] (#9477)
* Added a new test suite for testing limit API values - ValueLimitsIT

* Change index on participant_command_completions table
This change fixes issues with commands with large submitters number on sandbox-classic

* Added key_hash column to the ledger *state table
The new column is now the primary key of the table. Its values are hashed values of the 'key' column which allows to mitigate limitations on the index row size

* Backfill key_hash for ledger_state table

* Dynamic state table prefix form the backfill migration

* Removed redundant comments

* Backfill migration for all the db types

* Added missing copyright comment

* Fixed migration order after a rebase

* Added missing checksums for sql migrations

* Temporarily removed copyrights from one of sql migrations

* Removed unnecessary NOT NULL constraint

* Removed submitters from the index participant_command_completion_offset_application_idx in the append-only schema

* Disabled the test for old platforms

* CHANGELOG_BEGIN
- [Integration Kit] - a new test suite ValueLimitsIT for testing edge case values
- [Integration Kit] - modified index on the participant_command_completions table to avoid issuess with a large number of submitters
- [Sandbox] - added the key_hash column to the *state table
CHANGELOG_END

* Disabled concurrent testing for the ValueLimitsIT:VLLargeSubmittersNumberCreateContract test case

* Recomputed migrations checksums for the participant-integration-api

* Increase the sandbox-on-x queue size to 500
The motivation is that running conformance tests failed with RESOURCE_EXHAUSTED error due to 200 limit on the queue.

* Minor improvement

* Inlined key hashing for migrations to avoid external dependencies

* Minor improvement

* Fixed migrations hash
2021-05-04 12:23:37 +02:00
Moritz Kiefer
dce6a16649
Add Ledger API test tool tests for exceptions (#9400)
* Add Ledger API test tool tests for exceptions

changelog_begin
changelog_end

* Update daml-lf/language/daml-lf.bzl

Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>

* Address review comments

changelog_begin
changelog_end

* Shuffle around test

changelog_begin
changelog_end

Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
2021-05-03 18:40:52 +02:00
Samir Talwar
e1af564bcc
Switch from @silent to @nowarn. (#9498)
* Switch from `@silent` to `@nowarn`.

This annotation is native to Scala 2.12.13+ and 2.13.2+. It replaces
most usages of `@silent`.

I had to get creative about a couple of use cases that didn't work.
Specifically:

  1.  Suppressing deprecation warnings works, but Scala 2.12 erroneously
      complains that the `@nowarn` is unnecessary.  I had to suppress
      this warning too with `-Ywarn-unused:-nowarn`.
  2.  I can't seem to suppress the warning, "The outer reference in this
      type test cannot be checked at run time." Instead, I have
      refactored the code to remove the warning.

We still need to use the silencer plugin to suppress some warnings about
unused imports (because of compatibility between Scala 2.12 and 2.13),
but this means we no longer need the library, and therefore it is not a
transitive dependency that downstream consumers need to worry about.

CHANGELOG_BEGIN
CHANGELOG_END

* Add some comments around `@nowarn` support.

* language-support/scala: Fix a warning suppression.

* Revert to the default warnings.

Compatibility was complaining.
2021-04-26 19:46:14 +00:00
Hubert Slojewski
f4316bddf3
KVL-874 Add telemetry classes from the oem integration kit and use it for command submissions (#9436)
* Add telemetry classes from the oem integration kit and use it for command submissions

* Change submitTransaction to submitTransactionWithTelemetry and add a deprecation

* Fix tests

* Revert "Change submitTransaction to submitTransactionWithTelemetry and add a deprecation"

CHANGELOG_BEGIN

- [Integration Kit] TelemetryContext has been introduced to the WriteService.submitTransaction method to support distributed tracing

CHANGELOG_END
2021-04-23 13:00:39 +02:00
Robert Autenrieth
3193027558
DPP-334 Manually partition the event table (#9394)
* Manually partition the event table

changelog_begin
changelog_end

* fix delete from participant_events

* Run conformance test with append only schema

* Adapt indexer to partitioned tables

* Clean up schema

* Fix pruning

* Improve comments

* Store event offset for divulgence events

* Add back create key value for exercises

* Fix migration checksum
2021-04-19 21:12:28 +02:00
Robert Autenrieth
301dcd9e40
DPP-316: Enable the use of the append only index database (#9368)
* Enable append-only schema in index

changelog_begin
changelog_end

* Enable append-only schema in indexer

* Add CLI flags to configure append-only indexer

* Fix CLI flag name

* Remove unused parameter

* Change CLI flag description

... it's independent of Postgres

* Refactor how indexer config is specified in CLI

* Upper case constants
2021-04-12 21:11:03 +00:00