Commit Graph

54 Commits

Author SHA1 Message Date
pbatko-da
fbb3ba55c2
Simplify implementations of BaseError trait [DPP-606] (#13185)
1. Remove `BaseError.Impl`, `LoggingTransactionErrorImpl` and `LoggingPackageServiceError` 
and instead provide more direct `DamlError` and `DamlErrorWithDefiniteAnswer`.


2. Remove custom implementation of `TransactionError.rpcStatus` and instead provide simpler one in `DamlErr.r.rpcStatus` (which works by first calling `code.asGrpcStatus` and then converting the result to `com.google.rpc.status.Status`).

3. Remove `GrpcStatus.toProto` and instead use `DamlError.rpcStatus`.

4. Use `asGrpcStatus` and `asGrpcError`  instead of `asGrpcStatusFromContext` and `asGrpcErrorFromContext` where possible.


changelog_begin
changelog_end
2022-03-10 12:12:14 +01:00
pbatko-da
c7c211e4df
Remove most of ErrorFactories [DPP-606] (#13146)
changelog_begin
changelog_end
2022-03-07 16:15:21 +01:00
tudor-da
ea48173350
Switch daml-lf/engine and leger-api-tests/tool not depend on kvutils (#13107)
changelog_begin
changelog_end
2022-03-01 15:41:12 +01:00
pbatko-da
15b54a870e
[User management] Enable UM for sandbox-on-x and ledger-on-sql conformance tests (#13041)
The goal is to have a conformance test using Oracle and with UM enabled.

changelog_begin
changelog_end
2022-02-23 15:42:48 +01:00
mziolekda
a235a66f38
Remove unsafe from the buffered-ledger-api-streams (#13022)
* Remove unsafe from the buffered-ledger-api-streams

CHANGELOG_BEGIN
CHANGELOG_END

* and a bazel file
2022-02-22 19:02:05 +01:00
Samir Talwar
ac12112dad
ledger-api-tests: Remove the performance-testing functionality. (#12948)
* ledger-api-tests: Remove the performance-testing functionality.

This has been superseded by the Ledger API Bench Tool.

CHANGELOG_BEGIN
- [Ledger API Test Tool] The performance tests have been removed from
  this tool, in favor of the Ledger API Bench Tool, which is much more
  capable and configurable.
CHANGELOG_END

* ledger-on-sql/sandbox-on-x: Stop running performance tests.

We don't check the results anyway.
2022-02-15 17:06:23 +00:00
tudor-da
2a06a0a78c
Conflict checking enabled by default (#12924)
changelog_begin
changelog_end
2022-02-15 15:51:42 +01:00
tudor-da
307931d5b4
Extracted kvutils Config to //leger/ledger-runner-common [DPP-865] (#12889)
* Extracted kvutils Config to //leger/ledger-runner-config

changelog_begin
changelog_end

* Sandbox-on-X logback.xml does not depend on the kvutils/app one

* Remove unused dependencies

* Add copyright header back

* Remove dependency of SoX on kvutils
2022-02-14 13:52:06 +00:00
tudor-da
dd6dab6e6b
[Sandbox-on-X] Address various TODOs (#12823)
* Removed no-action TODOs

changelog_begin
changelog_end

* Assert self-service error codes in SequenceSpec

* Move InternallyInconsistentKeys and InternallyDuplicateKeys to LedgerApiErrors.WriteServiceRejections

* Restructured ledger bridge metrics
* Buffers before each processing stage

* Do not log the failure stacktrace on ledgerId lookup

* Addressed review comments

* Buffer size before prepare submission is 128
2022-02-11 17:03:41 +00:00
Samir Talwar
83ed13d799
Split the ledger-api-test-tool into libraries, under /ledger/ledger-api-tests. [KVL-1076] (#12771)
* Move //ledger/ledger-api-test-tool to //ledger/ledger-api-tests/tool.

And conformance.bzl to //ledger/ledger-api-tests.

* Move test tool infra to //ledger/ledger-api-tests/infrastructure.

* Split the ledger API test tool into smaller libraries.

There is no change to behavior.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-api-tests: Encapsulate performance tests.

* ledger-api-tests: Extract a "runner" library.

* ledger-api-tests: Publish libraries to Maven Central.

* ledger-api-tests: Fix Scaladoc so it compiles correctly.

Unfortunately I had to remove some Scaladoc in ActiveContractsServiceIT
which is incorrectly interpreted as not being attached to anything.
Fortunately, it's not actually that helpful.

* compatibility: Fix paths to the Ledger API Test Tool.

* Fix release types for ledger API test libraries.

* test-common: Publish generated Scala.

These libraries are dependencies of the ledger API tests components.

* ledger-api-tests: Use "com.daml" as the group ID everywhere.

* ledger-api-tests: Check that tests in `--additional` are valid.

* compatibility: Fix a reference to ledger-api-test-tool I missed.
2022-02-10 13:25:23 +00:00
tudor-da
c9dcd1c493
Update kvutils Config with missing flags options from SandboxConfig [DPP-864] (#12629)
* Update kvutils Config and SandboxConfig
* Remove `eager-package-loading` and `scenario` from SandboxConfig
* Add `profile-dir` and `stackTraces` to kvutils Config
* Configurable authService in kvutils

changelog_begin
changelog_end

* Addressed Samir's review comments

* Add back Maximum supported deduplication duration enforcement of 1 hour
2022-02-07 20:13:04 +00:00
tudor-da
d63651be58
[Sandbox-on-X] Duplicate party allocations explicit rejection (#12767)
* Added test case in PartyManagementServiceIT

changelog_begin
changelog_end

run-full-compat: true
2022-02-07 14:14:19 +01:00
Marton Nagy
5a2dffd79e
Add multi-db sandbox-on-x conformance tests [DPP-802] (#12585)
* Enhances SandboxOnXRunner with manipulateConfig
* Adds Postgres and Oracle conformance test runner binaries
* Adapts BUILD.bazel

changelog_begin
changelog_end
2022-02-04 15:28:20 +01:00
pbatko-da
97676bdb2c
Increase time-out in OngoingStreamAuthIT (#12747) 2022-02-04 12:44:40 +01:00
nicu-da
c6ed12dc6c
[ledger-api-test-tool] - Remove static time flag which was replaced by a feature descriptor (#12714) 2022-02-02 09:30:50 -08:00
tudor-da
117c9202cd
Bump sandbox-on-x-it-tests size (#12681)
changelog_begin
changelog_end
2022-01-31 17:12:12 +00:00
tudor-da
bb5722cd1b
Move sandbox-classic test lib and IT tests to Sandbox-on-x (#12641)
* Moved SandboxFixture and IT tests to Sandbox-on-X

changelog_begin
changelog_end

* Addressed review comments
2022-01-31 12:49:46 +01:00
tudor-da
cfa8d30491
[Sandbox-on-X] Ledger-side in-memory command deduplication [DPP-872] (#12596)
* [Sandbox-on-X] Ledger-side in-memory command deduplication

changelog_begin
changelog_end

* Enirched SequenceSpec test

* Participant-side deduplication compatibility tests exclusions

* Fix LedgerConfigurationServiceIT

* Disable command deduplication tests for SoX without conflict checking

* Remove redundant max-dedup-seconds bridge config

* Default max deduplication time 5 minutes

* Deduplication queue length metrics

* Add back removed ledger config test in SequenceSpec
2022-01-28 11:42:25 +00:00
mziolekda
16a13e21b7
Remove redundant conformance tests from ledger-on-sql (#12639)
- Remove duplcate postgres test
- Move oracle tests under the common framework
- Move in-memory-fan-out test to sandbox-on-x

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-27 19:41:05 +00:00
Samir Talwar
ac3ca1d20c
ledger-api-test-tool: Enable command deduplication tests by default [KVL-1261] (#12541)
* ledger-api-test-tool: Enable command deduplication tests by default.

CHANGELOG_BEGIN
- [Ledger API Test Tool] More command deduplication tests have been
  enabled by default. This should not affect testing a fully functioning
  ledger. If any tests fail, they can be temporarily excluded.
CHANGELOG_END
2022-01-25 12:51:45 +01:00
tudor-da
bd0ae0239b
Startup initialization header for runners [DPP-860] (#12525)
* Startup header of kvutils Runner

changelog_begin
changelog_end

* Added initialization header for Sandbox-on-X

* Updated Sandbox-on-X init message

* Updated initialization log for kvutils.Runner
2022-01-21 21:11:22 +00:00
Samir Talwar
4d26f0899e
ledger-api-test-tool: Enable ParticipantPruningIT for everyone. [KVL-1261] (#12529)
* ledger-api-test-tool: Enable `ParticipantPruningIT` for everyone.

CHANGELOG_BEGIN
- [Ledger API Test Tool] The ``ParticipantPruningIT`` test case has
  been enabled by default. You may need to disable specific test cases
  accordingly.
CHANGELOG_END

run-full-compat: true

* Enable more ParticipantPruningIT tests across the board.

* ledger-api-test-tool: Exclude a multi-participant test by default.

The test, `PRImmediateAndRetroactiveDivulgence`, will now only run if
more than one participant is specified.

* ledger-api-test-tool: Use a `for` comprehension for readability.

Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>

Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
2022-01-21 16:16:57 +00:00
Marton Nagy
5ab9eae033
Make mutable-contract-state-cache the default and only option (#12528)
* remove configuration flag
* remove ReadOnlySqlLedgerWithTranslationCache
* adapt condebase

changelog_begin
changelog_end
2022-01-21 16:47:33 +01:00
tudor-da
2267d1a15d
sandbox-classic uses Sandbox-on-X internals [DPP-860] (#12466)
* Sandbox-classic uses Sandbox-on-X internals

changelog_begin
changelog_end

* Removed conformance tests targeting in-memory ledger

* Adapt pruning test exclusions

* Refactoring

* Dedicated execution context for resource management

* Rebased

* Keep Sandbox logging initialization header

* Addressed Samir's review comments

* Addressed Martino's review comments
* Wired more CLI params from SandboxConfig to SoX config
2022-01-21 09:44:26 +00:00
Samir Talwar
e9e1b06da5
ledger-api-test-tool: Enable MultiPartySubmissionIT for everyone. [KVL-1261] (#12500)
* ledger-api-test-tool: Enable `MultiPartySubmissionIT` for everyone.

CHANGELOG_BEGIN
- [Ledger API Test Tool] The ``MultiPartySubmissionIT`` test case has
  been enabled by default. You will most likely not need to change
  anything.
CHANGELOG_END

* ledger-api-test-tool-on-canton: Enable all MultiPartySubmissionIT tests.

* ledger-api-test-tool: Correct MultiPartySubmissionIT for LF v1.8.
2022-01-20 18:12:46 +01:00
Samir Talwar
f03fa6a5f1
ledger-api-test-tool: Govern ContractIdIT test runs through a feature. (#12454)
* ledger-api-test-tool: Rename "ParticipantFeature" to "Features".

* ledger-api-test-tool: Govern ContractIdIT test runs through a feature.

Rather than making it optional, we can expose a feature that instructs
whether we support certain kinds of contract IDs.

This accidentally works for Canton too as it defaults to "false" for
both options, which holds for Canton.

CHANGELOG_BEGIN
- [Ledger API Test Tool] The "ContractIdIT" test is now run by default,
  using ledger feature flags to determine which test cases are run. You
  will need to configure the ``StandaloneApiService`` accordingly.
CHANGELOG_END

* ledger-api: Add some documentation to `ContractIdFeatures`.

* compatibility: Update for ContractIdIT.

run-full-compat: true

* ledger-api: Clarify the contract ID features and remove v0 support.

We also remove the tests for v0 contract IDs from ContractIdIT.

* ledger-api-test-tool: Re-introduce tests for v0 contract IDs.

* ledger-api-test-tool: Case class with named parameters in ContractIdIT.

* ledger-api: All ledgers must support suffixed contract IDs.

* compatibilty: Add a start for ContractIdIT.

We can't exclude a test until it exists.

run-full-compat: true
2022-01-19 22:40:12 +01:00
tudor-da
27b8ca0f17
Publish Sandbox-on-X app lib jar (#12414)
changelog_begin
changelog_end
2022-01-14 09:30:08 +01:00
Moritz Kiefer
b7ded08614
Publish sandbox on x as a standalone JAR (#12386)
Checked with @tudor-da that pkvutils does not rely on this being
available as a thin library.

changelog_begin
changelog_end
2022-01-13 11:47:57 +01:00
Samir Talwar
9c77a3c94b
kvutils: Remove the DAR upload parameters from the runner. (#12366)
* kvutils: Remove the DAR upload parameters from the runner.

No production ledger needs to upload DARs on participant startup; this
feature is primarily for users of Sandbox. The feature never worked in
the case of multiple participants and was only ever used in testing.

This also removes the associated functionality from Sandbox-on-X as they
share a configuration object. Hopefully this won't be an issue.

Some tests were using this feature, so I have updated them to upload the
DARs through the PackageManagementService instead.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Remove the `writePackagesService` factory constructor.

We no longer use it.

* daml-lf/engine: Revert an unnecessary change.

* daml-script/test: Shut down the channel after uploading the DAR.
2022-01-12 12:57:02 +01:00
nicu-da
8bc5804520
Add feature descriptor for the support for max deduplication duration [KVL-1218] (#12355)
Co-authored-by: Hubert Slojewski <hubert.slojewski@digitalasset.com>
2022-01-12 10:00:55 +00:00
tudor-da
8a2449b87f
TagWithLedgerEndSpec (#12346)
changelog_begin
changelog_end
2022-01-11 15:12:52 +00:00
nicu-da
50deee3ef1
[ledger-api-test-tool] - Merge KVCommandDeduplicationIT and CommandDeduplicationIT (#12257)
changelog_begin
[ledger-api-test-tool] - Merge command deduplication ledger API tests (`KVCommandDeduplicationIT`, `CommandDeduplicationIT`) into a single suite(`CommandDeduplicationIT`) which uses feature descriptors to handle different participant behaviors
changelog_end
2022-01-11 01:53:55 -08:00
tudor-da
e86b307fa5
[Sandbox-on-X with conflict checking] Full conformance test suite (#12309)
changelog_begin
changelog_end
2022-01-07 18:50:06 +01:00
tudor-da
8d0bbde7eb
[SoX conflict checking] Configuration upload conflict checking (#12291)
* [SoX conflict checking] Configuration upload conflict checking

CHANGELOG_BEGIN
CHANGELOG_END

* Created static-time SoX conformance test target
2022-01-07 11:03:48 +00:00
tudor-da
d4ebce6f44
[SoX with conflict checking] Implement party allocation validation for transactions conflict checks [DPP-837] (#12283)
* Implement party allocation validation for transactions conflict checks

CHANGELOG_BEGIN
CHANGELOG_END

* Propagate loggingContext to all processing stages
2022-01-06 13:04:15 +01:00
tudor-da
46ffec6a1a
[Sandbox-on-X] Transaction conflict checking for contracts and keys [DPP-825] (#12204)
* [Sandbox-on-X] Implement transaction submission conflict checking
* Implement the conflict checking ledger bridge
* Adds `conformance-test-conflict-checking` using the --enable-conflict-checking flag

CHANGELOG_BEGIN
CHANGELOG_END

* Dedicated threadpool for ledger bridge

* Fixed BridgeWriteService after rebase

* Update copyright headers

* Address Robert's review comments

* Revert constraining the conflict-checking parallelism

* Ledger bridge uses services thread pool

* Enable PARTICIPANT_DEDUPLICATION_SUPPORT feature flag
2022-01-06 09:06:29 +01:00
Gary Verhaegen
d2e2c21684
update copyright headers (#12240)
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.

I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-03 16:36:51 +00:00
Simon Maxen
8f43c034ec
Add transaction statistics to completion info (#12224)
* Add transaction statistics to completion info

changelog_begin
changelog_end

* reformat
2021-12-21 23:27:30 +00:00
tudor-da
ea7d49d4d9
Implement Sandbox-on-X runner [DPP-823] (#12150)
* Implement Sandbox-on-X runner

CHANGELOG_BEGIN
CHANGELOG_END

* Address Marton's comments

* Extracted AkkaSubmissionsBridge

* Addressed second round of review comments
2021-12-17 07:38:16 +00:00
Remy
58e69ade1a
LF: replace "dev" LF version by "1.dev" in bazel files (#11894)
the more consistent, as asked by Moritz in review of #11820

CHANGELOG_BEGIN
CHANGELOG_END
2021-11-26 12:23:17 +00:00
nicu-da
25e5bbb753
kvutils - Removed unused setting from conformance tests [KVL-1174] (#11785)
kvutils - The deduplication period is no longer overwritten by the max deduplication duration, therefore there is no need to change the default max deduplication duration

changelog_begin
changelog_end
2021-11-19 05:41:45 -08:00
Robert Autenrieth
355352f2d1
DPP-650 Remove the mutating schema (#11211)
* Remove the mutating schema

changelog_begin
- [Participant] All participants now use the new append-only schema. Existing databases will
  automatically upgrade to the new schema the first time a participant/ledger is started.
changelog_end

* Fix post-commit validation test

* Remove append-only flags from CI

* Don't crash when using deprecated flag

* Increase timeout for reset service tests

* Fix typos in parameter names

* Restore removed test

* Restore removed CLI check

* Improve CLI parameter description
2021-10-21 14:40:35 +02:00
mziolekda
04d8f753b2
Clean up sandbox-on-x conformance tests. (#10766)
* Clean up sandbox-on-x conformance tests.

CHANGELOG_BEGIN
CHANGELOG_END

* fix a flaky test in config management
2021-09-20 07:52:46 +00:00
fabiotudone-da
be4e06427f
Ledger API Test Tool: support --additional tests [KVL-1100] (#10829)
* Support adding tests as an hidden option

* Simplify existing suites

CHANGELOG_BEGIN
CHANGELOG_END

* Remove stale conformance suites from build.yml

* `--add` -> ``--additional`

* Re-add `--all-tests` as deprecated CLI option to be tested

* Move sandbox-classic pruning test to wall clock again

* Run KVCommandDeduplicationIT for sandbox append-only

* Tidy-up

* Also add participant pruning test to ledger-on-memory/single-participant

* Remove KVCommandDeduplicationIT on ledger-on-memory/append-only

* Run the full suite plus pruning (rather than just pruning) for ledger-on-memory with multiple participants and append-only

* Add KVCommandDeduplicationIT to ledger-on-memory append-only

* Exclude ConfigManagementServiceIT from ledger-on-memory append-only multi-participant

* Tidy-up

* Use KVCommandDeduplicationIT for sandbox-on-x too

* Fix merge

Add max dedup duration arg to all the test suites that include command dedup tests

* Make `--include` and `--additional` mutually exclusive

* Uniform formatting of multi-line strings

* Move exclusions after additions as they are applied last

* Re-disable deduplication test on sandbox with static time

* Re-disable deduplication test on sandbox-on-x
2021-09-14 14:00:30 +02:00
tudor-da
4b7391b0d5
LedgerApiTestTool prints skipped (excluded) tests (#10726)
* Document multi-participant-only pruning test

CHANGELOG_BEGIN
CHANGELOG_END
2021-09-06 13:50:54 +02:00
tudor-da
90ad24fd6a
[Divulgence pruning] Prune immediate divulgence [DPP-513] (#10691)
* [Divulgence pruning] Prune disclosed contracts
* Adapt CommonStorageBackend.pruneEvents to prune all immediately divulged events
* Adapt ParticipantPruningIT to assert immediate divulgence pruning
* Adapt ParticipantPruningIT tests for divulgence pruning to assert the ACS before and after pruning

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt conformance tests

* Adapt disclosure query for Oracle

* Addressed review comments

* Rebased
2021-08-31 20:19:08 +00:00
tudor-da
96ad9b5ab8
[Divulgence pruning] All divulgence events pruning [DPP-483] (#10634)
* All divulgence events pruning
* Add `pruneAllDivulgedContracts` to `IndexParticipantPruningService`
* Add `participant_pruned_all_contracts_up_to_inclusive` in `parameters` table
* `pruneEvents` prunes all divulgence events
* Adapt `ParticipantPruningIT` to check for divulgence with and without `pruna_all_divulged_contracts`
* Create multi-participant pruning test in ledger-on-memory

CHANGELOG_BEGIN
CHANGELOG_END

* CommonStorageBackend
* Enrich pruning descriptions
* Logging for command completions pruning
* Move divulgence pruning back to the beginning of pruning sequence

* Addressed review comments

* PRDivulgenceArchivalPruning adapted for privacy-aware ledgers

* Rebased to main

* Disable ParticipantPruningIT tests targeting append-only schema in mutable schema conformance tests in ledger-on-sql

* Adapted order of SQL migration queries

* Documentation of appendonlydao.JdbcLedgerDao.prune

* Apply suggestions from code review

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
2021-08-26 08:45:01 +00:00
Samir Talwar
4b8b67a1b5
Upgrade Scalatest to v3.2.9. (#10576)
* Upgrade Scalatest to v3.2.9.

Because of some coupling we also have to upgrade Scalaz to the latest
v7.2 point release, v7.2.33.

The Scalatest changes are quite involved because the JAR has been broken
up into several smaller JARs. Because Bazel expects us to specify all
dependencies and doesn't allow transitive dependencies to be used
directly, this means that we need to specify the explicit Scalatest
components that we use.

As you can imagine, this results in quite a big set of changes. They
are, however, constrained to dependency management; all the code remains
the same.

CHANGELOG_BEGIN
CHANGELOG_END

* http-json-oracle: Fix a Scalatest dependency.

* ledger-api-client: Fix a Scalatest dependency.
2021-08-12 23:19:35 +00:00
Remy
41009f79cc
Compiler: Make LF 1.14 the default output (#10394)
CHANGELOG_BEGIN

- [Compiler] Default compiler output LF version is now 1.14

CHANGELOG_END
2021-07-28 12:14:56 +02:00
Samir Talwar
c3bc26fd1c
daml-lf/data: Move ID aliases to Ref from _ledger-api-common_. [KVL-1002] (#10323)
* daml-lf/data: Move ID aliases to `Ref` from _ledger-api-common_.

This allows us to remove a lot of dependencies on _ledger-api-common_,
and use these aliases in other places where that module is not used.

CHANGELOG_BEGIN
CHANGELOG_END

* participant-integration-api: Remove an unused import.

* http-json-oracle: Remove `ledger-api-common` as a dependency.

* bindings-rxjava: Remove a now-unused dependency.
2021-07-20 11:01:19 +00:00