Commit Graph

928 Commits

Author SHA1 Message Date
Stephen Compall
15350a7bc2
disable warts.Any and remove most suppressions (#6132)
* disable Any wart

* first pass removal of Any suppressions for false positives

* second pass removal of Any suppressions for false positives

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* third pass removal of Any suppressions for false positives

* fourth pass removal of Any suppressions for false positives

* reformat newly single-suppressions into single lines

- suggested by @SamirTalwar-DA; thanks
2020-05-28 16:53:41 +00:00
Robert Autenrieth
28251ba296
Don't store archived divulged contracts (#6139)
* Don't store archived divulged contracts

Specifically, don't keep contracts that
have been divulged AND archived
in the same transaction.

Even if the archival of a divulged contract
might not be visible to all locally hosted
parties, the ledger would reject all
commands that try to use archived
contracts anyway.

CHANGELOG_BEGIN
CHANGELOG_END

* Add test

* Fix compile error
2020-05-28 15:10:08 +02:00
Miklos
39daca8b82
Dropped 'Batched' infix from all ledger-on-memory class names. (#6141) 2020-05-28 13:51:46 +02:00
Miklos
6108c922c6
Support parallel validation for ledger-on-memory (#6111) 2020-05-28 11:11:47 +02:00
Remy
3f29ac84f0
DAML-LF: cleanup last refs to Absolute Contract IDs (#6126)
follow up of #5991

CHANGELOG_BEGIN
CHANGELOG_END
2020-05-27 18:05:48 +02:00
Miklos
895a709498
Standardize command-line parameter for max inbound message size (#6093) 2020-05-27 11:55:49 +02:00
Stefano Baghino
acc5a21c59
Fix issues cause by migration V25 and V26 (#6107)
* Fix issues cause by migration V25 and V26

Fixes #6017

Please note that migrating sandbox-classic from version 1.0.0 to 1.1.1 will cause
undefined behavior if the result is used to back a running ledger setup.

No migration should end at 1.1.1 and they should move past that until the version
that applies this fix to make sure the index database is in a consistent state.

Full write-up on the issue here: https://github.com/digital-asset/daml/issues/6017#issuecomment-634040972

changelog_begin
[Sandbox Classic] Fix issue in migrating to version 1.1.1. If you did migrate to
version 1.1.1, do not use the resulting setup but make sure to migrate until this
version, which fixes the issue
See https://github.com/digital-asset/daml/issues/6017
changelog_end

* Add trailing newline

* Fix wrong hash for migration

* Address https://github.com/digital-asset/daml/pull/6107#discussion_r430469918
2020-05-27 08:37:58 +02:00
Leonid Shlyapnikov
937aa88ce7
transaction stream dashboard (#6059)
changelog_begin
changelog_end
2020-05-26 21:07:49 -04:00
Simon Maxen
cfec886e71
Ensure that on first call to ReadService.stateUpdates beginOffset is … (#6103)
* Ensure that on first call to ReadService.stateUpdates beginOffset is None

Fixes #6023

CHANGELOG_BEGIN
CHANGELOG_END

Also restores correct advice in CONTRIBUTING.md

* Update with review comments
2020-05-26 17:36:02 +01:00
Stefano Baghino
9af85e56e9
Introduce DAML-LF value caching for transaction service (#6052)
* Introduce DAML-LF value caching for transaction service

Allows to keep the DAML-LF values in the most recently indexed events in memory,
so that they don't have to be deserialized on the client from their Protobuf encoding.

Closes #6044

CHANGELOG_BEGIN
[Sandbox] The --max-lf-value-translation-cache-entries option allows to set a
number of events for which DAML-LF values are cached. Could reduce latency in
serving transactions for consumers that are reasonably fast.
CHANGELOG_END

* Add missing dependency

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076003

* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/sandboxnext/Runner.scala

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428071324

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076905

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428081294

* Fix fatal warnings

* //ledger/caching has to be used whenever sandbox is run

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-05-26 08:33:53 +00:00
Remy
9e456a1016
DAML-LF: Kill RelativeContractId (#5991)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-25 22:30:45 +02:00
Miklos
9a0cce155e
Introduce batched submission validator (#6004) 2020-05-25 18:25:10 +02:00
Miklos
32625da92f
Pass on command ID as correlation ID for transaction submissions (#6051)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-20 17:51:03 +02:00
Stefano Baghino
6f1e051648
Integrate metrics for cache (#6043)
* Integrate metrics for cache

Centralizes the creation of instrumented caches in a single point and adds
metrics coming from Caffeine into the mix.

changelog_begin
[Ledger API Server] if --max-state-value-cache-size is greater than zero, the
following additional metrics will be recorded under the daml.kvutils.submission.validator.state_value_cache namespace:
hits, misses, load_successes, load_failures, load_total_time, evictions and evicted_weight
changelog_end

* Fix Bazel build file formatting

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427902339

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427904794

* Review DropwizardStatsCounter

- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905074
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905184
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905307
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905387
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905650

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427906243

* Fix implicit numeric widening fatal warning

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427960762

* Fix infinite loop in metrics
2020-05-20 15:51:45 +02:00
Stephen Compall
d7cc5e07c3
fix ACS race condition in sandbox-classic; enable random Websocket testing (#5969)
* Revert "disable test pending tracking down duplicates"

This reverts commit 8b5f9dfa04.

* activeContracts tells caller the ledgerEnd, rather than accepting activeAt argument

* naive port of http-json tests to sandbox-next

* Revert "naive port of http-json tests to sandbox-next"

This reverts commit 91d4590c90.

* lock before grabbing acs and ledgerEnd offset in InMemoryLedger

- as suggested by @gerolf-da; thanks
2020-05-19 13:26:37 -04:00
Stefano Baghino
53a7e28126
Document CommandConfiguration (#6031)
changelog_begin
changelog_end
2020-05-19 15:33:38 +00:00
Samir Talwar
5915dc158d
Create a Docker image for Sandbox Next. (#6019)
This image can be built with:

```
bazel run //ledger/sandbox:sandbox-next-image
```

This will, unfortunately, also run a container, which you will need to
kill with _Ctrl+C_. You can use `bazel build`, but this will only build
the image tarballs, which you need to import into your Docker image
repository yourself.

CHANGELOG_BEGIN
CHANGELOG_END
2020-05-19 13:11:45 +00:00
Samir Talwar
d76c4ec949
ledger-api-client: Make it Closeable. (#6014)
CHANGELOG_BEGIN
- Add `LedgerClient#close`, which will shut down the channel and await
  termination. This is optional; the channel will still be shut down on
  JVM exit if this method is not called.
CHANGELOG_END

This is offered as a compromise for those who would like to ensure
resources are shut down cleanly, while not making the API more
complicated. I originally wanted to make `LedgerClient.fromBuilder`
return a `Resource[LedgerClient]`, but the `Resource` API would lead to
an increased learning curve for users.
2020-05-19 08:09:01 +00:00
Oliver Seeliger
c0af52950c
Fix ledger api test tool multi-node CommandService test flake (#5998)
by adding synchronize call between an alpha-participant-create and the
corresponding beta-participant-exercise.

changelog_begin
changelog_end

We occasionally run into this in Canton
2020-05-15 13:41:57 +02:00
Samir Talwar
8cd3cc414e
ledger/metrics: Display graphs with counts alongside mean timings. (#5994)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-15 08:44:34 +00:00
Stefano Baghino
be914d3b26
Measure translation time for full result from DB (#5983)
Furthermore, uses a work stealing thread pool for work JdbcLedgerDao has
to perform outside of the database thread pool. This solves some
flakiness observed in conformance tests against PostgreSQL-backed
ledgers.

changelog_begin
changelog_end
2020-05-15 09:02:02 +02:00
Remy
3eed4d1a22
Ledger: remove all reference to RelativeContractId (#5985)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-14 19:12:32 +02:00
Gerolf Seitz
a4f75c3cef
Fix quadratic runtime when reading events (#5981)
Fixes #5980.

CHANGELOG_BEGIN
[Sandbox] Improved performance when loading transactions with many
events. See `issue #5980
<https://github.com/digital-asset/daml/issues/5980>`__.
CHANGELOG_END
2020-05-14 13:31:15 +00:00
Samir Talwar
57a8d0b37e
CI: Run PostgreSQL once for all Scala tests. (#5919) 2020-05-14 09:06:34 +02:00
Remy
deedec9b9f
Engine: Remove optionality of contract ID Seeding. (#5966)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-13 20:29:14 +02:00
Remy
9019b3a33c
Ledger: Remove optionality of contract ID seeding. (#5965)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-13 17:39:07 +02:00
Stefano Baghino
b0ad8d7a60
Centralize and time serialization (#5946)
changelog_begin
OVERRIDES CHANGELOG ENTRY FROM 2a8c93a614
[Ledger Integration Kit] Added new metrics for
``daml.index.db.*.translation`` to measure the time spent
translating to and from the serialized DAML-LF values when
fetched from the participant index.
changelog_end
2020-05-13 17:06:46 +02:00
Stefano Baghino
e39adbe69e
Cleanup: move unused translation helpers to migrations (#5963)
* Cleanup: move unused translation helpers to migrations

These helpers are no longer used in the main source tree.

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/5963#discussion_r424415151
2020-05-13 16:35:05 +02:00
Oliver Seeliger
cc880e290b
Port PerformanceEnvelope.TransactionSize perf test from Canton (#5955)
* Port PerformanceEnvelope.TransactionSize perf test from Canton

to LedgerApiTestTool --perf-tests

changelog_begin
[Ledger Api Test Tool] Added the TransactionSize performance benchmark test.
changelog_end

* Review feedback from Samir

* Formatting
2020-05-13 12:03:46 +00:00
Remy
021f4af394
Sandbox classic: Emulate legacy contract ID scheme (#5929)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-13 12:56:41 +02:00
Stefano Baghino
9ff36a13cf
Extract caching from participant-state as a library (#5949)
* Extract caching from participant-state as a library

This will be used to keep a cache of values to cut on LF translation cost when serving transactions.

changelog_begin
changelog_end

* Add dependency where missing
2020-05-12 19:45:55 +02:00
Remy
1cd412fa83
Sandbox: addapt scenario loader to the new contract ID scheme (#5942)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-12 13:45:14 +02:00
Stefano Baghino
7e0377dd76
Offload transaction write preparation from DB thread (#5930)
* Offload transaction write preparation from DB thread

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/5930#discussion_r423049118

* Solve compilation issue

* Move type into companion object

* Address https://github.com/digital-asset/daml/pull/5930#discussion_r423170046
2020-05-11 22:51:32 +00:00
Stefano Baghino
dd6af4e276
Centralize index DB metrics naming (#5927)
* Centralize and review index DB metrics naming

Closes #5922

changelog_begin
changelog_end

* Remove unnecessary factory

* Fix compilation issue
2020-05-11 12:59:02 +00:00
Stefano Baghino
2a8c93a614
Offload value deserialization from DB threadpool and add metrics (#5917)
* Offload value deserialization from DB threadpool and add metrics

Fixes #5909

CHANGELOG_BEGIN
[Ledger Integration Kit] Added new metrics for
``daml.index.db.*.deserialization`` to measure the duration of
the translation of the serialized DAML-LF values when fetched
from the participant index.
CHANGELOG_END

* Address https://github.com/digital-asset/daml/pull/5917#discussion_r422282582

* Address https://github.com/digital-asset/daml/pull/5917#discussion_r422282901

* Address https://github.com/digital-asset/daml/pull/5917#discussion_r422283768

* Address https://github.com/digital-asset/daml/pull/5917#discussion_r422285920

* Address https://github.com/digital-asset/daml/pull/5917#discussion_r422285674
2020-05-11 10:41:49 +02:00
mziolekda
e0ab70df64
Convert transaction committer to applicative style (#5901)
* Convert transaction committer to applicative style

* few renames following code review

* Test that `TransactionCommitter` won't overflow when performing many reads

* Remove useless `TransactionCommitterSpec`

* Apply suggestions from code review

Apply suggestions

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>

* address more review comments

Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2020-05-09 11:03:05 +02:00
fabiotudone-da
5e4164c12d
LedgerFactory.metricsRegistry -> LedgerFactory.metrics: Metrics (#5914)
* `LedgerFactory.metricsRegistry` -> `LedgerFactory.metrics: Metrics`

CHANGELOG_BEGIN
CHANGELOG_END

* Address review comment
2020-05-08 19:00:58 +02:00
Samir Talwar
208d4a50da
postgresql-testing: Use a new, random database name every time. (#5911)
* postgresql-testing: Store the JDBC URL separately.

* postgresql-testing: Expose the username and password.

* postgresql-testing: Get the caller to create the database.

And make sure it's a random one, not "test".

CHANGELOG_BEGIN
CHANGELOG_END

* postgresql-testing: Only store the JDBC URL for tests.

Less mutable state, innit.

* postgresql-testing: Capture the individual JDBC URL parameters.

* Bazel: Fix PostgreSQL binary paths.

* postgresql-testing: Just recreate the database in PostgresAroundEach.

There's no need to restart the process with a different data directory.
2020-05-08 12:36:42 +00:00
Stefano Baghino
846391ac72
Use a table specifically for transaction trees witnesses (#5897)
* Use a table specifically for transaction trees witnesses

This allows to not re-compute those every time using the flat transaction
witnesses and the complement.

Furthermore, witness tables indexing is redefined to a single primary key
covering both event identifier and witness party instead of two
separated indexes. This has proven to be a relevant improvement for
performance.

Fixes #5883

changelog_begin
[Ledger API Server] Significant improvements in serving transaction trees. See #5883.
changelog_end

* Fix migration SHA
2020-05-07 19:19:28 +02:00
Samir Talwar
17f4dc2716
ledger-on-sql: Remove the reference to the removed "common" migrations. (#5888)
This is producing a warning.

CHANGELOG_BEGIN
CHANGELOG_END
2020-05-07 12:56:56 +00:00
Samir Talwar
a0635fa397
ledger-on-sql: Use dedicated database execution contexts. (#5869)
* ledger-on-sql: Use dedicated database execution contexts.

This may or may not stop us from monopolizing API server threads.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-sql: Name arguments to `Database` and `UninitializedDatabase`.
2020-05-07 11:52:24 +00:00
nickchapman-da
53b24793f4
Simplify and clarify the public interface to Speedy. (#5881)
* Simplify and clarify the public interface to Speedy.

- Remove `isFinal`. A client just uses `run()`.
- Remove `toSValue`. The value in available in `SResultFinalValue(v: SValue)`.
- A client never directly access the `.ctrl` (or `.returnValue`) components.
- A client may use `setExpressionToEvaluate(expr)` to evaluate a new expression on an existing machine.

changelog_begin
changelog_end

* remove while loop which executes just once

* avoid unnecessary mutation when running speedy
2020-05-07 08:55:16 +00:00
Robert Autenrieth
9ebfd01f83
Add script to dump metrics summary (#5828)
* Add script to dump metrics summary

CHANGELOG_BEGIN
CHANGELOG_END

* Sort results by default

* Address review comments

* Run prettier

* Refactor code

* Make script executable

* Address review comment

* Always take latest data point
2020-05-06 17:13:14 +02:00
Leonid Shlyapnikov
a79377cb2d
Improve DAML LF Party, PackageName, PackageId fromString error messages (#5855)
* Add description to daml.lf.data.StringModule implementations

Make sure DAML LF Party deserialization picks up the added description.

changelog_begin
changelog_end

* addressing code review comments. @S11001001, thanks.
2020-05-06 10:02:49 -04:00
Robert Autenrieth
7e448d810c
Always use the latest ledger config (#5669)
CHANGELOG_BEGIN
- [Sandbox] The ledger API server will now always use the most recent ledger configuration.
  Until a ledger configuration is read from the ledger, command submissions will fail with the UNAVAILABLE error.
CHANGELOG_END

In kvutils, the first ledger configuration change needs
to have a generation one higher than the one returned
by getLedgerInitialConditions().

Remove initial config writing from sandbox as it's now written by the ledger API server
2020-05-06 12:12:23 +02:00
Stefano Baghino
df12c529c8
Fix broken migration v25 with v29 (#5861)
The v25 migration did not read transactions in order, causing #5659.

This PR re-arranges migrations so that ledger_entries can be re-read to fix the issue before dropping it in v30 (which used to be v29).

Fixes #5659.

changelog_begin
[Sandbox Classic] There is a chance that migrating from Sandbox 0.13.55 to Sandbox Classic 1.0.0 could have
introduced contracts falsely reported as active when in fact they are not. Migrating to Sandbox Classic
1.1.0 will fix the issue. For more details, see #5659.
changelog_end
2020-05-06 10:08:32 +00:00
Gerolf Seitz
4d606689bb
Centralize metric names (#5790)
Centralizing the metrics allows us to have a good overview of the
existing metrics.

CHANGELOG_BEGIN
CHANGELOG_END
2020-05-05 16:55:29 +02:00
Stephen Compall
2b23b41391
JSON decoding support for GenMap (#5223)
* add GenMap to the "all types" test generators

* report bad GenMap format with DeserializationError, not MatchError

* document GenMap JSON

* notes on missing features

* enable -Xsource:2.13 in transaction

* make an Order instance for Value resolvable, but unimplemented

* use the skeleton from SValue ordering to make a Value ordering skeleton

* add Party Order

* add Order instance for SortedLookupList

* add Order for FrontStack, deriving everything

* factor the Order lookup, and tie a knot in the recursive Value instances

* we're going to need this Iterator thing again

* replacing Order#contramap with version that supports equalIsNatural

* use new equalBy, orderBy for FrontStack, SortedLookupList, ImmArray

* _2 comparator, upgrade Name Equal to an Order

* incorporate lookup for enums, variants into Value order; record/struct cases

* Enum/Variant comparison

* looking up the singleton implicitly won't work for non-`object`s, alas

* test Order laws for values of all primitive types

* test Order laws for record and variant types

* test Order laws for enum types

* test that enum strings are not compared

* use checkLaws for Value Equal as well

* test that enums match order to constructor rank

* factor genAddend and genAddendNoListMap

* reintroduce Order for TypedValueGenerators

* more addend order

* record, variant order cases

* record cons order

* deriving Order while decoding from JSON

* make ApiCodecCompressed's Cid codec based on the typeclass

* test how the Value ordering and the underlying projected value orderings line up

- hint: they don't, yet
- this is also a template for how we'll check the fidelity with SValue
  ordering

* test how the Value ordering and SValue ordering line up

- hint: they don't, yet

* typed Arbitrarys have access to Order

* produce proper ValueGenMap

* inj requires Order, sometimes

- we encode this as "all the time" but there is a type-level unification
  approach to remove this requirement in some cases

* make inj a function

* test that order doesn't matter for JSON decoder

* use Utf8 order for TVG text; don't pretend that base equal works

* sort JSON GenMaps, and check for duplicates

* make injarb use IntroCtx

* remove stray import

* Order instances for Bytes, Hash, AbsoluteContractId

* require Order[Cid] to decode JSON to LF values

* clean up map reordering test

* remove unused Instant instance

* fake Order instance no longer needed, valid instance defined

* test parity of global AbsoluteContractId order and SContractId order

* bazel fmt

* test AbsoluteContractId Order lawfulness

* test duplicate key detection

CHANGELOG_BEGIN
- [JSON API] Prepare full support for the planned GenMap primitive type.
  See `issue #5031 <https://github.com/digital-asset/daml/issues/5031>`_.
CHANGELOG_END
2020-05-05 10:27:37 -04:00
Oliver Seeliger
4c0ffdf4df
Upgrade scala compiler silencer to 1.6.0 (#5822)
* Upgrade scala compiler silencer to 1.6.0

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt build bazel file to new targets

* Switch to silencer plugin scala 2.12.11 per Samir's feedback

rather than 2.12.8

* Add missed bazel files

* Review feedback from Leo
2020-05-04 21:30:46 +00:00
Stefano Baghino
89eb6c64d2
Integrate PostCommitValidation with JdbcLedgerDao and SqlLedger (#5781)
* Integrate PostCommitValidation with JdbcLedgerDao and SqlLedger

Closes #5035
Closes #5663

changelog_begin
[Sandbox] Skip unnecessary double post-commit validation inherited by sandbox-classic, expect performance improvement
changelog_end

* Ensure SqlLedger recovers from failures and logs them when publishing a transaction

* Remove unused import

* Remove tests for ledger entries

* Fix completions test to make them compile

* Fix compilation errors in tests, address self-review items, apply necessary fixes

- address https://github.com/digital-asset/daml/pull/5781#pullrequestreview-403293667
- address https://github.com/digital-asset/daml/pull/5781#pullrequestreview-403378192

* Pass TransactionTimeModelComplianceIT

* Minor tweaks to variable naming

* Fix failing tests

* Stop deduplicating commands on failures

* Attempt at making sandbox-classic allocate parties implicitly

* Remove implicit party allocation test (without full server) for SQL backed sandbox-classic

* Removing ImplicitPartyAdditionIT (covered in conformance tests)

* Add migrations

* Fix test for ledger DAO with post-commit validation against PostgreSQL

* Update PostgresIT

* Fix missing/wrong items from previous commits

* Don't perform batch processing of enqueued persistence entries

* Rebase against master
2020-05-04 15:28:18 +00:00