Commit Graph

111 Commits

Author SHA1 Message Date
Stefano Baghino
fc69db02b4
Add a library that defines Resource factory methods for gRPC servers and channels (#8604)
* Add a library that defines Resource factory methods for gRPC servers and channels

This is the first of four PRs in which 6ea70c4b45
has been broken up to facilitate review.

The endgame is to have the non-repudiation prototype merged. These
factory methods have been used to make resource management easier,
specifically as part of testing.

changelog_begin
changelog_end

* Let warning log output in tests

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Address https://github.com/digital-asset/daml/pull/8604#discussion_r562634032

* Address https://github.com/digital-asset/daml/pull/8604#discussion_r562634865

* Address https://github.com/digital-asset/daml/pull/8604#discussion_r562635148

* Address https://github.com/digital-asset/daml/pull/8604#discussion_r562635684

* Add resources-grpc to release/artifacts.yaml

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-01-25 10:32:36 +01:00
Moritz Kiefer
84a9488077
Port more of //ledger/... to Scala 2.13 (#8488)
* Port more of //ledger/... to Scala 2.13

changelog_begin
changelog_end

* Remove unusued dependency

changelog_begin
changelog_end

* Rename bf to factory to reflect the fact that it’s now a Factory

changelog_begin
changelog_end

* Use regex match instead of sliding string equalityt

changelog_begin
changelog_end

* regex matches are bad

changelog_begin
changelog_end
2021-01-13 16:30:43 +01:00
Moritz Kiefer
22ce940954
Upgrade scalafmt and enable trailing commas (#8437)
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.

As announced, this will be merged on Saturday to avoid too many conflicts.

changelog_begin
changelog_end
2021-01-09 11:37:37 +01:00
Samir Talwar
6d2140072f
kvutils: Wrap raw key and value bytestrings in types. (#8408)
* kvutils: Wrap raw key and value bytestrings in types.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Delete `Bytes`.

* kvutils: Inline LedgerStateOperations.{Key, Value}.

* kvutils: Extract out a `Raw` trait.

* ledger-on-memory: Use state reader type aliases.

* ledger-on-sql: Provide implicit conversions for Raw.*.

* kvutils: Merge raw weights.

* kvutils: Move `Raw` into the kvutils package.

* kvutils: Seal `Raw`.

* kvutils: Delete the `SubmissionEnvelope` alias.

* kvutils: Use Raw#size when computing cache weight.

* kvutils: Improve clarity in a couple of places.

* kvutils: Rename the `Raw` trait to `Raw.Bytes`.

* kvutils: Rename `Raw.Pair` to `Raw.KeyValuePair`.
2021-01-07 15:28:14 +00:00
Gary Verhaegen
427179ab27
fix (some) copyright headers (#8422)
This commit fixes a few copyright headers that have been missed in the
automatic update on Jan 1, as well as the generation code in the compat
workspace so it generates the right headers.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-07 11:48:59 +01:00
Gary Verhaegen
a925f0174c
update copyright notices for 2021 (#8257)
* update copyright notices for 2021

To be merged on 2021-01-01.

CHANGELOG_BEGIN
CHANGELOG_END

* patch-bazel-windows & da-ghc-lib
2021-01-01 19:49:51 +01:00
Samir Talwar
4447685755
ledger-on-memory: Don't bother with fingerprints. [KVL-747] (#8340)
* ledger-on-memory: Don't bother with fingerprints.

Instead, just compare values during post-execution.

This has no semantic change, as the "fingerprints" were simply the
serialized bytestrings of the DAML state values. We now just compare the
DAML state values instead.

This change also removes fingerprints entirely from the kvutils code.
Ledgers that use fingerprinting can (and already do) can transform the
state values to include those fingerprints, if necessary.

CHANGELOG_BEGIN
- [Integration Kit] kvutils no longer supports fingerprints directly.
  Instead, if the ledger fingerprints the code, you can use the newer,
  more generic APIs to transform the state values to include those
  fingerprints, if necessary.
CHANGELOG_END

* kvutils: Make `PreExecutingSubmissionValidator` non-final.

It doesn't have a corresponding interface/trait, and it's useful to be
able to mock it.
2020-12-18 17:07:28 +00:00
Samir Talwar
cb0f681932
kvutils: Remove fingerprints from the PreExecutingValidatingCommitter. [KVL-747] (#8337)
* ledger-on-memory: Create type aliases for caches.

* kvutils: Move caching out of the committer and into the constructor.

* kvutils: Move `reader.map` into the conflict detector.

* kvutils: Remove `LogResult` from the `PreExecutingValidatingCommitter`.

By making it covariant, we can just use `Any`, as we don't care about
the actual type.

* kvutils: Pull the state type out of `PreExecutingValidatingCommitter`.

It is now generic over state types.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-18 08:13:14 +00:00
Samir Talwar
72b00a05f2
kvutils: Avoid serializing keys and values except when necessary. [KVL-747] (#8319)
* kvutils: Let users pass an `Iterable`, not just a `Seq`, to StateReader.

* kvutils: Avoid serializing keys and values except when necessary.

This simplifies the state-reading operations considerably.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Remove knowledge on fingerprints from conflict detection.

Generics forever.

* kvutils: Move FingerprintedReadSet.

* kvutils: Remove the `fingerprint` test helper.

* kvutils: Make equality-based post execution even more generic.

* kvutils: Be less variant in the PostExecutionConflictDetector types.
2020-12-18 06:54:40 +00:00
Robert Autenrieth
f3d8f05070
Enable multi-party submissions [KVL-708] (#8266)
* Remove single-party check

CHANGELOG_BEGIN
- [Ledger API] The ledger API now supports multi-party submissions.
  In order to use multi-party submissions, use the new act_as and
  read_as fields in submission requests.
CHANGELOG_END

* Remove usage of temporary SubmitterInfo methods

* Fix validator tests

* Fix auth test

* Add multi-party tests to the ledger API

* Fix compile errors

* Improve tests

* Remove temporary single-party method from SubmitterInfo

* Remove temporary single-party method from SubmitterInfo companion object

* Remove temporary single-party method from TxEntry

* Run multi-party submission ITs for ledger-on-memory and ledger-on-sql

* Minor improvement

Co-authored-by: Kamil Bozek <kamil.bozek@digitalasset.com>
2020-12-17 13:42:39 +01:00
Samir Talwar
021697fff7
kvutils: Remove fingerprints from the pre-executing validator. [KVL-747] (#8296)
* kvutils: Remove fingerprints from the pre-executing validator.

The submission validator doesn't care about fingerprints, it only needs
to know about them in order to discard them from the state value. This
introduces a new typeclass, `HasDamlStateValue`, which can be customized
for various state value types to make submission validation generic.

A little more code for now, but we're setting things up so it can be
deleted later.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Add more comments for `PreExecutingSubmissionValidator`.
2020-12-16 15:15:08 +00:00
Samir Talwar
54687a579c
kvutils: Reduce the surface area of fingerprints in pre-execution. [KVL-747] (#8289)
* kvutils: Move read set generation to the commit strategy.

* kvutils: Make the submission validator abstract over the read set.

* kvutils: Split post-execution conflict detection from finalization.

* kvutils: Ignore the read set in the post-execution finalizer.

* kvutils: Extract a trait for conflict detection.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Extract out a trait for post-execution finalization.

* kvutils: Generalize the write set in PreExecutingValidatingCommitter.

* kvutils: In submission validation, create the input state once.

* kvutils: Generalize PreExecutingCommitStrategy over keys and values.

* kvutils: Extract KeyNotPresentInInputException to the top level.

* kvutils: Inline type aliases where they're only used once or twice.

* kvutils: Improve naming and commentary around post-execution.
2020-12-15 12:52:08 +00:00
Samir Talwar
79a91669b2
kvutils: Use mockito-scala's ArgumentMatchersSugar. (#8218)
CHANGELOG_BEGIN
CHANGELOG_END
2020-12-09 15:34:38 +00:00
Samir Talwar
fdde69a599
kvutils: Throw if CommitContext#read encounters missing input state. [KVL-757] (#8194)
* kvutils: Throw if CommitContext#read encounters missing input state.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Revert CommitContext#get.

* ledger-on-memory: The pre-execution tests are no longer flaky.

* kvutils: Update the Scaladoc for CommitContext to mention exceptions.

* kvutils: Moar CommitContext testing.
2020-12-08 13:25:11 +00:00
Samir Talwar
73514c53ea
kvutils: Fix a flaky batching queue test. (#8199)
This changes the API so that we can wait for the queue to finish before
asserting that it is, indeed, finished.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-08 13:17:26 +00:00
Moritz Kiefer
7e05dc7932
Upgrade rules-scala and scalatest (#8187)
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.

This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.

Apologies for the giant PR, I don’t see a way to keep it smaller.

changelog_begin
changelog_end
2020-12-08 06:59:23 +01:00
Samir Talwar
17c8ce788f
ledger-on-memory: Mark pre-execution conformance tests as flaky. (#8153)
This is a temporary measure while we figure out what's wrong.

CHANGELOG_BEGIN
CHANGELOG_END
2020-12-03 15:47:49 +00:00
Oliver Seeliger
7a2a2700cf
Introduce optional ledger-api-test-tool tests with ParticipantPruningIT (#8041)
CHANGELOG_BEGIN
CHANGELOG_END
2020-11-23 19:09:39 +01:00
Samir Talwar
d3b5f30a70
Delete the Docker image targets. (#7932)
We don't use them, and we definitely don't want to support them.

CHANGELOG_BEGIN
CHANGELOG_END
2020-11-11 07:40:06 +00:00
Samir Talwar
7f679b9896
resources: Customizable contexts. (#7678)
* resources: Move builders into //ledger/ledger-resources.

Keep the actual constructors in a trait, but instantiate it when working
with ledger code.

This allows us to later introduce an extra "context" type parameter to
ResourceOwner.

* resources-akka: Move the builders in to //ledger/ledger-resources.

* resources: Introduce an abstract `Context` parameter for owners.

This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.

This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.

Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.

CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
  `Context`, which is filled in by the corresponding `Context` class in
  the _ledger-resources_ dependency. This allows us to pass extra
  information through resource acquisition.
CHANGELOG_END

* ledger-resources: Move `ResourceOwner` here from `resources`.

* ledger-resources: Remove dependencies from outside //ledger.

* ledger-resource: Wrap the acquisition execution context in `Context`.

So we can add a logging context to it.

* resources: Pass the Context, not the ExecutionContext, to Resource.

* Avoid importing `HasExecutionContext`.

* ledger-resources: Publish to Maven Central.

* resources: Make the small changes suggested by @stefanobaghino-da.

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* ledger-resources: Pull out a trait for test resource contexts.

Saves a few lines of code.

* Restore some imports that were accidentally wildcarded.

* resources: Replace an `implicit def` with a couple of imports.

* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.

Try and use the right execution context where possible.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-10-20 09:26:28 +00:00
Robert Autenrieth
8f57b9bf85
Add ability to start only indexer or only ledger API server [KVL-584] (#7574)
* Add ability to start only indexer or lapi server

CHANGELOG_BEGIN
CHANGELOG_END

* Change command line arguments

* Use Resource.unit

* Fix MultiParticipantFixture

* Add a new conformance test

* Improve retrying connecting to the database

* Improve naming

* Introduce shardName

and use it to create unique metric names

* Fix a merge error

* Remove unused comment

* Fix test

* Run conformance-test-split-participant in batch mode

Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
2020-10-12 09:46:20 +02:00
Samir Talwar
f0c1eb207c
concurrent: Tag DirectExecutionContext. (#7517)
* concurrent: Tag DirectExecutionContext.

1.  Tag `DirectExecutionContext` as `ExecutionContext[Nothing]`, thereby
    stating that it works for any tagged `Future`.
2.  Move `DirectExecutionContext` to the _libs-scala/concurrent_
    library, as it requires it and it's tiny.

CHANGELOG_BEGIN
CHANGELOG_END

* concurrent: Fix the privacy of `DirectExecutionContextInternal`.

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-09-29 17:23:57 +00:00
Samir Talwar
9fa77b61a2
kvutils: Sort the output state before writing. [KVL-558] (#7515)
* kvutils: Always commit serially.

Committing in parallel will cause the set of writes to be emitted in a
random order.

CHANGELOG_BEGIN
- [Integration Kit] In kvutils, the ``BatchedSubmissionValidator`` no
  longer has a parameter for commit parallelism. Commits are now always
  written serially to preserve order.
CHANGELOG_END

* kvutils: Sort the output state before writing.

We previously wrote the output state in a random order, leading to
writes that could not be easily compared for integrity. Sorting them
allows us to validate not just the values, but the order.

This means that the exporter no longer sorts the write set, and the
integrity checker does not sort before checking.

CHANGELOG_BEGIN
- [Integration Kit] In kvutils, state is now sorted before committing.
  This allows us to provide stronger guarantees with regards to the
  serialized write sets.

  If you have implemented your own ``CommitStrategy``, you should also
  ensure the output state is sorted before committing.
CHANGELOG_END

* ledger-on-sql: Implement SQL typeclasses as values, not functions.

Let's pretend performance is important here.

* kvutils: Allow Iterables in batching ledger state operations.

A `Map[Key, Value]` is an `Iterable[(Key, Value)]`, but not a `Seq`.
This allows us to pass a `Map` straight from the committer.

* kvutils: Construct a `SortedMap` in the commit strategy.

This should be more performant than generating and then sorting a
`Vector`.

* kvutils: Sort state updates when committing after pre-execution.
2020-09-29 17:07:11 +00:00
Moritz Kiefer
86f89d5026
Split ledger-on-memory Main into multiple files (#7428)
I’d like to use this in a test fixture so this PR splits up the main
and exposes an API that gives you back the resource without having to
import Main.

changelog_begin
changelog_end
2020-09-17 17:42:27 +02:00
Samir Talwar
fe7d56f099
kvutils/tools: Split the tools by directory. (#7306)
* kvutils/tools: Split the integrity checkers by package.

The two versions of the integrity checker are intertwined when they
really don't rely on each other at all. This splits them into two
separate packages to make the distinction clear.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils/tools: Split the integrity checkers by directory.

This splits the integrity checkers further into separate directories, to
make it clear they do not interact at all.

* kvutils/tools: Rename "integrity-check" to "integrity-check-v1".

* ledger-on-memory: Recommend ledger exports (v3), not ledger dumps (v1).

* kvutils/tools: Split the benchmarks from integrity-check-v2.
2020-09-02 12:45:56 +00:00
Samir Talwar
aaf70c4add
kvutils: Increase the streaming updates timeout in integration tests. (#7278)
* ledger-on-(memory + sql): Include logback as a test dependency.

Otherwise logs don't show up when we get failures, which makes debugging
a lot harder.

* kvutils: On unit test timeout, show a useful error.

* kvutils: Increase the streaming updates timeout in integration tests.

This is timing out every now and again on CI, and I can reproduce it
easily by running the tests in parallel.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-31 14:40:35 +00:00
Samir Talwar
aafb4a27f9
kvutils: Simplify exporting by pushing the mutations outwards. (#7215)
* kvutils: Add `override` annotations for the exporters.

* kvutils: Test the ledger export (v2).

* kvutils: Mark integrity tests as small.

* kvutils: Add assertions to the FileBasedLedgerDataExporter.

Just confirming my understanding of how this works. Verified with the
Ledger API Test Tool and some proprietary code.

* kvutils: Move the definition of `CorrelationId` to the package file.

* kvutils: Pass a submission write set through the batch pipeline.

This allows for less mutability all over the place.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Move export finishing into its own type.

* kvutils: Move some nested types upwards.

* kvutils: Split Deserialization from Serialization.

* kvutils: Extract out common behavior in (De)Serialization.

* kvutils: Don't use a singleton for LedgerDataExporter.

Instead, construct it once.

* kvutils: Make sure we close the export file when we're done.

* kvutils: Simplify `Debug` in the same manner as `LedgerDataExport`.

* kvutils: Fix a test name.

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>

* kvutils: Remove backticks around "export".

* kvutils: Move the test `BatchedSubmissionValidator#apply` into the test.

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
2020-08-25 17:52:59 +00:00
Samir Talwar
5d1ef35b09
kvutils: Move execution contexts to the submission methods. (#7071)
* kvutils: Move execution contexts to the submission methods.

Previously, the execution contexts were provided at the constructor
level, which meant the submission validator and associated components
would use the resource acquisition execution context. This context is
intended purely for resource acquisition; if we need an execution
context for processing data, we have much more control when we
explicitly create one.

Implementors will still need to avoid passing an execution context
implicitly into the constructors of their own components, instead
favoring an explicit one that is to be used during `commit` or reading
events.

CHANGELOG_BEGIN
- [Integration Kit] kvutils now expects execution contexts to be passed
  in to the various ``SubmissionValidator``, ``LedgerStateAccess``, and
  ``LedgerStateOperations`` methods. This is a source-breaking change.

  Instead of providing an execution context implicitly to your ledger
  implementation, you are encouraged to construct the relevant contexts
  for different operations explicitly. Please refer to the open-source
  implementations as a reference.
CHANGELOG_END

* ledger-on-memory: Don't use the acquisition context for commits.

Instead, use the materializer execution context; it may not be the best
choice but at least it's appropriate.

* kvutils: Standardize method parameter formatting.

* kvutils: Provide old state  APIs as `LedgerStateAccess.v1_4`.
2020-08-18 11:50:33 +00:00
curiousleo-da
e1bf421075
Publish ledger-on-memory (#7117)
CHANGELOG_BEGIN

- [Integration Kit] Publish package `com.daml:ledger-on-memory`.

CHANGELOG_END
2020-08-13 14:32:04 +02:00
Samir Talwar
015b3b6534
ledger-on-memory: Split the tests into a test suite. (#7069)
* ledger-on-memory: Split the tests into a test suite.

The ledger-on-memory tests are sporadically failing on CI because they
hit the timeout of 60s, which is the default for a "small" test target.
This is not outside the realm of possibility, as tests can take a while
on CI when lots of other things are running.

To mitigate this, rather than increasing the timeout, I am splitting the
tests up into a suite, which means they'll run as 3 targets, not 1, each
with a 60s timeout. This means they're far more likely to complete in
the allocated time.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-memory: Don't implicitly widen numerics.

Not sure why this breaks on CI and not on my machine, but 🤷🏽‍♂️.
2020-08-10 09:07:03 +00:00
Samir Talwar
2d53b01598
ledger-on-sql: Provide a dedicated committer execution context. (#7063)
* ledger-on-sql: When failing to acquire a connection, specify where.

* kvutils: Pass the execution context through the ValidatingCommitter.

The SubmissionValidator shouldn't be getting it through the constructor,
but this will do for now.

* kvutils: Remove the executionContext from BatchingLedgerStateOperations.

* ledger-on-sql: Inject the committer.

* ledger-on-sql: Construct a single-threaded executor for the committer.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-sql: Trailing commas are the best kind of commas.

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>

Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
2020-08-10 09:03:15 +00:00
Stephen Compall
3d5e96c9bb
remove unused definitions, params, args from sandbox Scala code (#7035)
* remove unused definitions, params, args from sandbox Scala code

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused loggingContext from sandbox

* pass pageSize along in JdbcLedgerDaoTransactionsSpec

- seems to have been the intent of the parameter, and at the moment it
  is semantically identical
2020-08-06 17:12:59 +00:00
fabiotudone-da
84bbca1a6d
Pre-execution support for DAML on memory KV (#6793)
* Move in-mem writer's `ledgerStateAccess.inTransaction` down to committer

* Move `BatchedSubmissionValidator` and spec into `batch` subpackage

* Add `StateAccessingValidatingCommitter` and inherit it in batching one

* Document `StateAccessingValidatingCommitter`

* Generalize the committer for `InMemoryLedgerReaderWriter`

* `envelope` -> `submissionEnvelope` in validating committers

* Add `PreExecutingValidatingCommitter` and sub-components

* Add retry in case of conflict in `PreExecutingValidatingCommitter`

CHANGELOG_BEGIN
CHANGELOG_END

* Fix compilation error

* Hook pre-execution in `daml-on-memory-kv`

* Add fake time updates provider

* Fix `BatchedValidatingCommitterSpec`

* Don't use batched writer with pre-execution

* Fix conflict detection

* Fix out-of-time-bounds detection

* Prefix/unprefix serialized log entry IDs in pre-execution write sets

* Fix: produce an out-of-bounds rejection log entry in transaction rejected cases too

* Fix `SubmissionResult` return in case of repeated pre-exec conflict

* Fidelity level 1: sequential pre-execution

* Documentation for pre-execution support in DAML-on-Memory KV

* Add ledger-on-memory conformance test with pre-execution enabled

* Revert "Fix: produce an out-of-bounds rejection log entry in transaction rejected cases too"

This reverts commit 4df7e26b

* Fix test

* Improve naming and documentation

* Address review comments

* Fix test

* Fix wrong implementation used for `ParticipantStateIntegrationSpecBase` tests

* Address review comments

* Address review comments

* Address minor review comments
2020-08-05 08:00:27 +00:00
Stefano Baghino
e972872128
Properly use LoggingContext in Participant Server (#6924)
* Properly use LoggingContext in Participant Server

Fixes #6837

Spreads usage of LoggingContext and ContextualizedLogger throughout the participant server.

changelog_begin
[Sandbox/Integration Kit] We have enriched the contextual information
exposed by the Ledger API server. You should note richer logging information,
which can be read either via unstructured or structured logging frameworks.
A paragraph on how to configure structured logging has been added to the docs.
For more on the issue, see https://github.com/digital-asset/daml/issues/6837.
changelog_end

* Make everything compile

* Leave only actual commands on trace level on submission

* Shorter log messages on submission

* Don't add values down the call stack

* Document good practices regarding the logging context

* Add context to write service implementations

* Add logging context to tests

* Document logging for DAML on SQL

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463503013

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463508117

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463515665

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463513157

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463516359

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463516895

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463518813

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463520210

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463521501

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463521593

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525453

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525560

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525672

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525742

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463526837

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527054

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527523

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527814

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527958

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527900

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527997

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463528050

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463542877

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543051

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543614

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543692
2020-07-31 12:40:13 +00:00
Stefano Baghino
c16f17a721
Remove mentions of fidelity levels from the Ledger API test tool (#6893)
* Remove mentions of fidelity levels from the Ledger API test tool

Fixes #6868

changelog_begin
[Integration Kit] The performance test names have been changed.
To learn more about the available tests, consult the documentation
for the Ledger API Test Tool and run it with --list.
Docs: https://docs.daml.com/tools/ledger-api-test-tool/index.html
changelog_end

* Update performance envelope tests

* Address https://github.com/digital-asset/daml/pull/6893#discussion_r461655325
2020-07-28 18:24:46 +00:00
Robert Autenrieth
7ce9748066
Split sandbox code into separate packages (#6695)
* Move public code into daml-integration-api

CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
2020-07-17 17:06:06 +02:00
Gerolf Seitz
35b50992f0
Allow running StandaloneApiServer in read-only mode. (#6721)
Also fixes #5635, removing usage of `ReadService` from the `StandaloneApiServer`.

The configuration stream of the Ledger API LedgerConfigurationService is now properly backed by the configuration entries instead of just serving the initial configuration.

CHANGELOG_BEGIN
[DAML Integration Kit]: ``StandaloneApiServer`` can now be run in a read-only mode.
  - The type of the constructor parameter ``writeService`` of ``StandaloneApiServer`` changed to ``Option[WriteService]``. Passing ``None`` will not start any of the admin services, the command service, and the command submission service.
  - The constructor parameter ``readService`` of ``StandaloneApiServer`` has been removed.
  - A new constructor parameter ``ledgerId`` has been added to ``StandaloneApiServer``. It is used to verify that that ``StandaloneApiServer`` is run against an index storage for the same ledgerId. Initialization is aborted if this is not the case.
[DAML Integration Kit]: The ``LedgerConfigurationService`` now properly streams configuration changes.
CHANGELOG_END
2020-07-17 09:57:00 +02:00
Miklos
761787ce4b
[kvutils] Pre-executing submission validator (#6711)
* Removed engine constrcutor parameter.

* Basic abstractions for pre-executing submission validator.

* Fix build.

* Shared byteToLogEntryId with BatchedSubmissionValidator.

* Added some more metrics.

* Removed involved participants from PreExecutionResult.
CHANGELOG_BEGIN
CHANGELOG_END

* Reformatted/added headers.

* Fix build.

* Code tidying.

* Moved PreExecutionOutput into a separate file.

* Moved type definition into companion object.

* Added some tests for PreExecutingSubmissionValidator.

* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/validator/preexecution/LedgerStateReaderWithFingerprints.scala

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>

* Removed left-in code.

* Added some assertions for .zip operations.

* Test result for unexpected message.

* Test setting min/max record time.

* Reuse code for generating a key-value pair for log entries.

* Tests for CachingDamlLedgerStateReaderWithFingerprints.

* Store empty values as well with their fingerprints.

* Moved weight function into companion object.

* Test that the results come back in right order.

* Code tidying.

* Basic test case for LogAppenderPreExecutingCommitStrategy. Added missing headers.

* Code tidying.

* Fix build.

* Fix build.

* Revert "Store empty values as well with their fingerprints."

This reverts commit ca68ea967f.

# Conflicts:
#	ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/validator/caching/CachingDamlLedgerStateReaderWithFingerprintsSpec.scala

* Added some docs.

* Added test case for ensuring that we don't cache None values.

* Fix build.

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
2020-07-15 08:55:19 +00:00
Miklos
f60ec5c2ed
[kvutils] Handle offsets in case of multiple updates from the same log entry (#6677) 2020-07-10 19:11:32 +02:00
Stefano Baghino
31995ee000
Ledger API Test Tool: deprecate unneeded tests and options (#6651)
* Ledger API Test Tool: deprecate unneeded tests and options

Furthermore, now all active tests are now run be default without needing
to specify --all-tests. The scheduler makes sure to run non-isolated
tests last (and sequentially).

Fixes #3747
Fixes #6518

A bit of historical context: tests in the Ledger API Test Tool used to
be the sandbox integration tests. As part of the Ledger API Test Tool
project, those tests have been ported one-to-one, with little review of
those same tests.

Examining the tests that cause #3747, it became evident that those tests
were originally put in place to verify that an isolated sandbox spun up
for a single test could withstand a number of commands slightly lower
than what at the time was the hard-coded back-pressure threshold. Hence,
implementing back-pressure on the Ledger API Test Tool would have
basically negated the usefulness of those tests. Furthermore, those
tests can be easily passed by simply raising the back-pressure
threshold. As such, they do not convey any meaningful information for
the DAML ledger implementor. This means we are retiring those tests. The
tests will be nominally kept in for a deprecation period to not break
existing build scripts, but running them explicitly will simply show
them as skipped. The documentation has been updated with the information
necessary to users to deal with retired tests.

Since all tests are now run by default --all-tests is being deprecated.
--load-scale-factor is also deprecated as it was used only by
TransactionScaleIT.

changelog_begin
[Ledger API Test Tool] The LotsOfPartiesIT and TransactionScaleIT test
suite have been deemed not providing relevant signal to DAML ledger
implementers and have been retired. The tests will be nominally kept in
but will be skipped while they are in a deprecation period. You are
advised to remove explicit references to those tests before they are
fully removed.
[Ledger API Test Tool] All tests are now run by default. The --all-tests
option is now ineffective and deprecated. You are advised to remove its
usages from your build scripts. Non-isolated tests that could affect the
global state of the ledger and interfere with other tests are now
automatically scheduled by the test tool to run sequentially at the end
of the run.
[Ledger API Test Tool] The --load-scale-factor option is now unused and
deprecated. You are advised to remove its usages from your build
scripts.
changelog_end

* Add test for deprecated CLI options

* Address https://github.com/digital-asset/daml/pull/6651#discussion_r452100158

* Address https://github.com/digital-asset/daml/pull/6651#discussion_r452126405

* Address https://github.com/digital-asset/daml/pull/6651#discussion_r452143811

* Address test failures
2020-07-09 14:01:32 +00:00
Miklos
1ffb28cde9
Renamed some default variables (#6627)
* BatchedSubmissionValidatorParameters.default => reasonableDefault

* Config.default() => createDefault()

* ExtraConfig.default => reasonableDefault
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-06 16:52:47 +00:00
Miklos
ccfb0ac94e
Make interpretation cost available for LedgerWriter/WriteService (#6515)
* Measure time of interpretation and store it in CommandExecutionResult.

* Added parameters for passing in interpretation time for WriteService and LedgerWriter.

* Code tidying.

* Added CommitMetadata parameter to LedgerWriter.

* Thread through interpretation time. Aggregate interpretation times for a batch.

* Take max of interpretation times.

* Calculate interpretation time in nanos.

* Moved CommitMetadata into separate file.
CHANGELOG_BEGIN
CHANGELOG_END

* Apply suggestions from code review

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Delegate deprecated methods to method with new signature.

* Code tidying.

* Suppress deprecation warnings.

* Made interpretation cost optional in CommitMetadata.

* Test that we populate interpretation time.

* Code tidying.

* Apply suggestions from code review

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Code tidying.

* Some more tests. Do not return interpretation cost for a batch if it only contains non-transaction submissions.

* Reformatted.

* Reformatted.

* Named arbitrary interpretation cost.

* Reverted changes for BatchingLedgerWriter.

* Always drop commit metadata for batches and don't report interpretation cost.

* More specific expectations.

* Include authorization check and blinding in interpretation time.

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-06-30 16:45:12 +00:00
Samir Talwar
7f7eaa35e7
Use Distroless for the Java Docker base image. (#6537)
* Use Distroless for the Java Docker base image.

We switched away from Distroless because it was causing issues with
`docker pull` when you had Docker configured to use `gcloud` for
authentication, but weren't actually authenticated.

Adding `docker-credential-gcloud` to dev-env should hopefully fix this,
meaning we can switch back to a base image that is better-maintained.

CHANGELOG_BEGIN
CHANGELOG_END

* Bump rules_docker to v0.14.3.

This fixes an issue when running `bazel sync`:

```
ERROR: java.io.IOException: Error downloading [http://central.maven.org/maven2/javax/servlet/javax.servlet-api/3.0.1/javax.servlet-api-3.0.1.jar] to [...]/external/javax_servlet_api/javax.servlet-api-3.0.1.jar: Unknown host: central.maven.org
```
2020-06-30 12:01:43 +00:00
Samir Talwar
8b972fb1d6
sandbox: Inject the dispatcher into the BaseLedger. (#6497)
* sandbox: Create proper `ResourceOwner` implementations.

This allows us to use the resource acquisition execution context, rather
than using DirectExecutionContext.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Simplify the construction of SqlLedger.

* sandbox: Inject the dispatcher into the BaseLedger.

* sandbox: Make sure the SqlLedger objects are closed.

Tiny regression. No one noticed. It's OK.

* sandbox: Simplify ReadOnlySqlLedger.

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* sandbox: Pull out functions to make SqlLedger.Owner easier to read.

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* ledger-api-common: Factor out wrapping Dispatcher in a ResourceOwner.

* sandbox: Move the PersistenceQueue into a ResourceOwner.

* ledger-api-common: Add a comma.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-06-29 13:24:07 +00:00
Remy
46b68bd92a
ledger test tool: refactor conformance test dars (#6506)
* ledger test tool: refactor conformance test dars

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-26 13:39:00 +02:00
Remy
6d27825acb
Engine: introduce config to control value/transaction output version (#6428)
* Engine: introduce configuration to control value/transaction output version

This advances the state of #5164

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-26 13:38:38 +02:00
Miklos
9dd78fc508
Ledger data export for non-determinism checks (#6418)
CHANGELOG_BEGIN
CHANGELOG_END
2020-06-19 17:52:11 +02:00
Samir Talwar
6496ddd903
caching | kvutils | sandbox: Use a size-based cache for LF value translation. (#6432)
* caching: Split caches into new files.

* caching: Rename `Cache.from` to `WeightedCache.from`.

* caching: Move `Configuration` inside `WeightedCache`.

* caching: Add test cases.

* caching: Allow for Caffeine builders to be covariant.

* caching: When instrumenting the Caffeine cache, compose, don't inherit.

* caching: Add a size-based cache.

* caching: Extract out common test cases into base classes.

* caching: Use the size-based cache for LF value translation.

CHANGELOG_BEGIN
CHANGELOG_END

* caching: Simplify the eviction tests.

* caching: Increase the encapsulation in CaffeineCache.

* caching: Commas are important.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-06-19 14:42:40 +00:00
Miklos
b0c1b5a2b3
Improve batched submission validation related docs (#6186) 2020-06-03 15:20:09 +02:00
Leonid Shlyapnikov
12c05c408a
Experiment: replacing row OFFSET with ledger offset (#6083)
* Experiment: replacing row OFFSET with ledger offset in the flat transactions query for one party.

if this improves the perf numbers, the rest of the queries can be updated.

* Flat transaction query optimization:

replacing row offset with ledger offset

* Flat transaction query optimization:

replacing row offset with ledger offset

* transaction tree query optimization

changelog_begin

[Sandbox-next/Postgres]
Flat Transaction Stream, Transaction Tree Stream SQL query optimizations.
Pagination based on Ledger Offset instead of SQL Row Offset.

changelog_end

* Addressing code review comments
2020-05-29 11:49:46 -04:00