Commit Graph

63 Commits

Author SHA1 Message Date
Miklos
b0c1b5a2b3
Improve batched submission validation related docs (#6186) 2020-06-03 15:20:09 +02:00
Leonid Shlyapnikov
12c05c408a
Experiment: replacing row OFFSET with ledger offset (#6083)
* Experiment: replacing row OFFSET with ledger offset in the flat transactions query for one party.

if this improves the perf numbers, the rest of the queries can be updated.

* Flat transaction query optimization:

replacing row offset with ledger offset

* Flat transaction query optimization:

replacing row offset with ledger offset

* transaction tree query optimization

changelog_begin

[Sandbox-next/Postgres]
Flat Transaction Stream, Transaction Tree Stream SQL query optimizations.
Pagination based on Ledger Offset instead of SQL Row Offset.

changelog_end

* Addressing code review comments
2020-05-29 11:49:46 -04:00
Miklos
655c0a5425
Fix flaky tests for ledger-on-memory (#6160) 2020-05-29 15:33:10 +02:00
Stephen Compall
15350a7bc2
disable warts.Any and remove most suppressions (#6132)
* disable Any wart

* first pass removal of Any suppressions for false positives

* second pass removal of Any suppressions for false positives

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* third pass removal of Any suppressions for false positives

* fourth pass removal of Any suppressions for false positives

* reformat newly single-suppressions into single lines

- suggested by @SamirTalwar-DA; thanks
2020-05-28 16:53:41 +00:00
Miklos
39daca8b82
Dropped 'Batched' infix from all ledger-on-memory class names. (#6141) 2020-05-28 13:51:46 +02:00
Miklos
6108c922c6
Support parallel validation for ledger-on-memory (#6111) 2020-05-28 11:11:47 +02:00
Miklos
895a709498
Standardize command-line parameter for max inbound message size (#6093) 2020-05-27 11:55:49 +02:00
Miklos
9a0cce155e
Introduce batched submission validator (#6004) 2020-05-25 18:25:10 +02:00
Stefano Baghino
6f1e051648
Integrate metrics for cache (#6043)
* Integrate metrics for cache

Centralizes the creation of instrumented caches in a single point and adds
metrics coming from Caffeine into the mix.

changelog_begin
[Ledger API Server] if --max-state-value-cache-size is greater than zero, the
following additional metrics will be recorded under the daml.kvutils.submission.validator.state_value_cache namespace:
hits, misses, load_successes, load_failures, load_total_time, evictions and evicted_weight
changelog_end

* Fix Bazel build file formatting

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427902339

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427904794

* Review DropwizardStatsCounter

- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905074
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905184
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905307
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905387
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905650

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427906243

* Fix implicit numeric widening fatal warning

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427960762

* Fix infinite loop in metrics
2020-05-20 15:51:45 +02:00
Oliver Seeliger
cc880e290b
Port PerformanceEnvelope.TransactionSize perf test from Canton (#5955)
* Port PerformanceEnvelope.TransactionSize perf test from Canton

to LedgerApiTestTool --perf-tests

changelog_begin
[Ledger Api Test Tool] Added the TransactionSize performance benchmark test.
changelog_end

* Review feedback from Samir

* Formatting
2020-05-13 12:03:46 +00:00
Stefano Baghino
9ff36a13cf
Extract caching from participant-state as a library (#5949)
* Extract caching from participant-state as a library

This will be used to keep a cache of values to cut on LF translation cost when serving transactions.

changelog_begin
changelog_end

* Add dependency where missing
2020-05-12 19:45:55 +02:00
fabiotudone-da
5e4164c12d
LedgerFactory.metricsRegistry -> LedgerFactory.metrics: Metrics (#5914)
* `LedgerFactory.metricsRegistry` -> `LedgerFactory.metrics: Metrics`

CHANGELOG_BEGIN
CHANGELOG_END

* Address review comment
2020-05-08 19:00:58 +02:00
Robert Autenrieth
7e448d810c
Always use the latest ledger config (#5669)
CHANGELOG_BEGIN
- [Sandbox] The ledger API server will now always use the most recent ledger configuration.
  Until a ledger configuration is read from the ledger, command submissions will fail with the UNAVAILABLE error.
CHANGELOG_END

In kvutils, the first ledger configuration change needs
to have a generation one higher than the one returned
by getLedgerInitialConditions().

Remove initial config writing from sandbox as it's now written by the ledger API server
2020-05-06 12:12:23 +02:00
Gerolf Seitz
4d606689bb
Centralize metric names (#5790)
Centralizing the metrics allows us to have a good overview of the
existing metrics.

CHANGELOG_BEGIN
CHANGELOG_END
2020-05-05 16:55:29 +02:00
Gerolf Seitz
e821168909
Reset service improvements for sadbonx (#5728)
- Only upload packages during the initial startup.
- Avoid loading packages during subsequent resets
- Share an engine between Ledger API Server and Committer

* Use a randomized H2 URL to simular in-memory

The reset service test assumes to get a completely new ledger for each
test case. But because we use H2 in-memory with db_close_delay=1 and the
same H2 database name, the second test case gets the remnants of the
first test case.

Since we know that sandbox in-memory uses an H2 in-memory URL, we can
simply use SandboxBackend.H2Database for ResetServiceInMemoryIT.

CHANGELOG_BEGIN
[Sandbox] Drastically lower the time needed to do a reset via the
ResetService.
CHANGELOG_END
2020-04-30 17:06:21 +02:00
mziolekda
bbd21605f5
add readme that details the usage of integrity check tool (#5685)
* add readme that details the usage of integrity check tool

* Apply suggestions from code review

Addressing review comments

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-04-28 09:18:35 +02:00
Samir Talwar
7d094fac2b
kvutils: Don't report ledger metrics by default. (#5604)
Instead, opt-in explicitly in _ledger-on-memory_ and _ledger-on-sql_.

Wrapping the operations can confuse other users of `LedgerStateAccess`.

CHANGELOG_BEGIN
CHANGELOG_END
2020-04-17 14:16:02 +00:00
Samir Talwar
9186f5ccb6
kvutils | ledger-on-(memory|sql): Read metrics. (#5576)
* ledger-on-memory: Add metrics to the operations.

* ledger-on-memory: Speed up reads by using views.

* kvutils: Time event processing.

* ledger-on-(memory|sql): Make classes private and final where possible.

* kvutils: Factor out a TimedLedgerStateOperations class.

CHANGELOG_BEGIN
- [Ledger Integration Kit] Report timing metrics for ledger state
  operations.
CHANGELOG_END

* ledger-on-sql: Record log read metrics.

* ledger-on-sql: Record database transaction timing metrics.

CHANGELOG_BEGIN
- [Sandbox] Record ledger database timing metrics under "daml.ledger".
CHANGELOG_END

* ledger-on-sql: Time queries.

* metrics: Add graphs for read events and DAML-on-SQL.

* ledger-on-memory: Simplify the tuple swap in reading the log.

Co-Authored-By: Miklos <57664299+miklos-da@users.noreply.github.com>

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2020-04-16 11:11:06 +00:00
fabiotudone-da
1bb1517c08
[TestTool] Provide performance tests for all performance envelopes (#5473)
* Produce performance tests for all envelopes

CHANGELOG_BEGIN
[TestTool] Provide performance tests for all performance envelopes
CHANGELOG_END

* Follow Scala's recommended pattern for enum-like hierarchies
2020-04-08 09:59:29 +02:00
Samir Talwar
6563048d5b
kvutils: Cache state value conversions from bytes. (#5426)
* kvutils: Cache state value conversions from bytes.

This seems to have a decent speedup in ledger-on-memory.

CHANGELOG_BEGIN
- [Ledger Integration Kit] Submissions now look up ledger values from a
  cache where possible, improving performance when there's contention over
  certain resources (e.g. common packages). The cache size currently
  defaults to 64 MB.
CHANGELOG_END

* kvutils: Make the SubmissionValidator statue value cache configurable.

* kvutils: Report state value cache metrics.

* kvutils: Add a suffix to a Long literal because WartRemover is unhappy.

Strangely, it doesn't fail on my machine.

* kvutils: Extract caching out into its own file.

* kvutils: Move the `bytesToStateValue` call into `cache.get`.

* kvutils: Move caching to its own package.

* kvutils: Inject the state value cache.

* kvutils: Default to no state value cache.

* kvutils: Accept a state value cache size in megabytes, not bytes.

* kvutils: Move cache building from `Config` to the `caching` package.

* kvutils: Replace Guava's cache with Caffeine.

* kvutils: Simplify caching configuration.

* sandbox: Enable state value caching by default.

CHANGELOG_BEGIN
- [Sandbox] State values deserialization is now cached, with a fixed
  cache size of 128MB.
CHANGELOG_END

* Changelog commit.

CHANGELOG_BEGIN
- [Ledger Integration Kit] The state value cache is now opt-in, with a
  default of no cache at all.
CHANGELOG_END
2020-04-06 17:30:32 +00:00
fabiotudone-da
2a5a15d625
PingPong throughput and latency benchmarks for on-mem and on-sql (#5377)
* Draft of PingPong throughput and latency benchmarks for on-mem and on-sql ledgers

* Augment `ParticipantTestContext` and remove `LedgerApiServer` hack

* Separate performance tests into distinct category with concurrency 1

* 🎨

* Package performance tests in separate DAR

* Have performance tests excluded by default and run exclusively if passed

* Fix rebase

* Simplify `BenchmarkReporter`

* Make `concurrencyOverride` into an `Option`

* Clarify command line usage, prevent regular and perf. tests together

* Fix preventing regular and perf. tests together

* Split `PingPong`, `PingPongExplode` and `Cycle` benchmarks' model

CHANGELOG_BEGIN
- [TestTool] Add `PingPong` performance envelope test
CHANGELOG_END

* Explicitly name `concurrencyOverride`

* Fix formatting

* Lower bar for CI run of performance envelope tests

* Make benchmark output file configurable

* Improve messages and report config option name

* Use exit status 64 for "bad command line usage" as in BSD
2020-04-06 18:01:10 +02:00
Gerolf Seitz
a2d785e3ee
Use com.daml as root package (#5343)
Packages com.digitalasset.daml and com.daml have been unified under com.daml

Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.

CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
2020-04-05 19:49:57 +02:00
Gerolf Seitz
5ddbd5c511
Implement timed command deduplication in kvutils (#5242)
* Implement timed command deduplication in kvutils

This adds a field deduplication_time to DamlCommandDedupValue for
deduplication timeout checking.

* Bump kvutils version to 4

* Fix CommandTracker pulling commandResultIn multiple times
Now that the timeouts are generated out of band, we have 2
"unsynchronized" places that pull on commandResultIn.
Whenever we pull, we need to check that commandResultIn hasn't been
pulled before.

* Add inStaticTimeMode flag to enable command dedup in sandbox-next with static-time

Fixes #4624.

CHANGELOG_BEGIN
[kvutils] KVUtils now respects the command deduplciation time instead of
deduplicating commands forever.
CHANGELOG_END
2020-04-03 10:44:35 +02:00
Samir Talwar
1d9c7d2574
kvutils: Remove the LedgerEntry trait; it's no longer necessary. (#5383)
* kvutils: Remove the LedgerEntry trait; it's no longer necessary.

This was introduced to allow for heartbeats, which no longer exist.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Make LedgerRecord a case class again.

We used to store the envelope as an array of bytes, which doesn't have a
value-based `equals` method and therefore should not be used in a case
class. We now use a `ByteString`, so this is no longer an issue.
2020-04-03 07:19:45 +00:00
Samir Talwar
a535a54014
kvutils: Do a little less inside the submission transaction. (#5368)
* kvutils: Use `.view` in SubmissionValidator.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Don't compute missing inputs unless we're asked.

* ledger-on-memory: Do less in `InMemoryLedgerStateOperations`.

* ledger-on-memory: Use `RangeSource` instead of `OneAfterAnother`.

Should be faster to just take a slice.

* ledger-on-memory: Don't bother locking when reading.

We're only reading the log, which is append-only; we never mutate
existing data. This means we don't need to lock to read it.

* ledger-on-memory: Make it impossible to construct a state with data.
2020-04-02 14:51:18 +00:00
Gerolf Seitz
effceda612
Remove heartbeats in participant server and kvutils (#5315)
Contributes to #4231.

Remove checkpoints from Participant Server storage:
- Removed code to store checkpoints in the index database.
- Remove existing checkpoint rows in ledger_entries and participant_command_completions.
- Removed ObservedTimeServiceBackend

This commit modifies the java migration V2_1__Rebuild_Acs. This is safe to do, because:
a) any even semi recent persistent sandbox had already gone through this migration and won't re-run it
b) A new database doesn't even have entries yet to migrate.

CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Removed the ``Hearbeat`` state update.
[Sandbox] Checkpoints are no longer emitted in regular intervals in wall
clock time mode.
CHANGELOG_END
2020-04-01 12:34:23 +02:00
Gerolf Seitz
329320bad9
Organize maven coordinates (#5272)
* Use com.daml as groupId for all artifacts

CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END

* Add 2 additional maven related checks to the release binary

1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId

* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
2020-04-01 11:41:18 +02:00
mziolekda
0f3c0d66d8
docker image for ledger-on-memory (#5256)
* docker image for ledger-on-memory

CHANGELOG_BEGIN
CHANGELOG_END

* Apply suggestions from code review

Applying Samir's suggestions to the README file

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

* Update ledger/ledger-on-memory/README.md

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-03-29 10:54:12 +02:00
mziolekda
ce62bfb596
Support for large grpc messages in ledger-on-memory (#5230)
* Support for large grpc messages in ledger-on-memory

* adress review comments

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 14:00:24 +01:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Samir Talwar
1ce67fbd2f
kvutils: Metrics on the write path. (#5212)
* kvutils: Add metrics to SubmissionValidator.

* kvutils: Add metrics to KeyValueSubmission.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-26 15:17:42 +00:00
Samir Talwar
2887f28cc1
kvutils: Inject the metrics registry instead of using SharedMetricRegistries. (#5161)
* kvutils: Remove an unnecessary `@SuppressWarnings`.

* kvutils: Reduce the scope of fields and methods in `Committer`.

* kvutils: Inject the metric registry into `KeyValueCommitting`.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Inject the metric registry into the committers.

* kvutils: Inject the metric registry into `ProcessTransactionSubmission`.

* kvutils: Avoid shared metric registries in tests.

* kvutils: Recreate the metrics registry per participant state.

* kvutils: Add trailing commas to parameter lists.

Flagrantly encouraged by @stefanobaghino-da.

* recovering-indexer: Don't re-use the metric registry in tests.
2020-03-24 19:41:19 +00:00
Remy
d834b9b029
Sandbox: make contract id seeding flag public (#5153)
* Sandbox: Reveal contract id seeding flag

CHANGELOG_BEGIN
- [Sandbox] Add support for random contract identifiers.  See section
  `Contract Identifiers Generation` section in
  docs/source/tools/sandbox.rst
CHANGELOG_END
2020-03-24 15:51:23 +01:00
Gerolf Seitz
cf032a02cf
Introduce Stable Offsets (#4953)
A "stable offset" in the context of the Participant Server is the offset
that was provided by the ledger backend (be it kvutils, corda, daml on sql).

The Participant Server does not keep a participant-local offset anymore.
In a single domain/kvutil setup, this makes offsets stable across participants,
since all participants will see the same offset for the same transaction.

The following changes were needed to achieve this:
- The participant server always uses the offset provided by the backend
  AS IS (no more +1 magic).
- Offsets provided to the Ledger API in requests must be treated as
  startExclusive and endInclusive (previously beginInclusive and
  endExclusive).

CHANGELOG_BEGIN
[Ledger API]: Offsets have been redefined. Instead of being represented
by a number or a structured string, an offset is now an opaque string
that can be compared lexicographically.
[DAML Integration Kit]: The bounds for ``Dispatcher`` are now
startExclusive and endInclusive.
CHANGELOG_END

---------
ledger api:
ledger_offset.proto
  Changed definition of offsets, since they can now be compared
  lexicographically.

---------
participant-state-api:
Offset:
  Changed from Array[Long] to ByteString. Ledgers need to make sure that the
  offsets produced are strictly monotonically increasing according to
  lexicographical order.

---------
akka-streams:
Dispatcher, DispatcherImpl, SubSource:
  Changed interval handling to exclusive/inclusive.

---------
ledger-on-memory:
InMemoryLedgerReaderWriter, InMemoryState:
  Changed interval handling to exclusive/inclusive.

---------
ledger-on-sql:
CommonQueries, SqlLedgerReaderWriter:
  Change interval in query and boundary handling.

---------
kvutils:
KeyValueParticipantStateReader, KVOffset:
  Convenience functions for kvutils to add or remove sub-indexes for
  offsets.
  KV ledger implementations can use KVOffset to construct a structured offset.

---------
Participant Server:
JdbcLedgerDao:
  Use Offset instead of Long.
  Fetch offsets directly as Offset from the database with proper anorm
  integration.
  Change interval handling to exclusive/inclusive.

CommandCompletionsReader, CommandCompletionsTable:
  Change interval handling to exclusive/inclusive.

BaseLedger:
  Use Offset instead of Long.
  Change interval handling to exclusive/inclusive.

Conversions:
  Anorm integration for using Offset in queries and result parsers.

JdbcIndexer:
  Remove references to "extenalLedgerEnd" and participant-local Long
  offset (headRef).

---------
sandbox:
In general:
  Use the Offset type everywhere instead of Long.

SQL migrations:
  Change all offset columns to bytea or BINARY.

LedgerBackedIndexService:
  Proper bounds checking has been pushed down to Dispatcher, which
  allowed simplifying the acceptedTransactions implementation.

InMemoryLedger, LedgerEntries:
  Change interval handling to exclusive/inclusive.
  Transaction lookup by ID is now O(n) because transaction IDs are not
  necessarily the same as the offset.

SqlLedger:
  Remove external offset references.
2020-03-18 08:43:41 +01:00
Miklos
c3068b2955
Allow reusing same ValidatingCommitter for multiple participants (#5041)
* Pass submitting participant ID as parameter to ValidatingCommitter.

* Made reference to processSubmission.
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-17 13:06:39 +00:00
Samir Talwar
b9acad2c6d
kvutils: Don't supply execution contexts to ResourceOwners. (#5010)
* kvutils: Remove the unnecessary execution context from the test base.

* kvutils: Remove the unnecessary execution context from the writer.

* ledger-on-sql: Make a proper owner so it has a proper execution context.

This means the parallelization now needs to come from the test, so I've
augmented ParticipantStateIntegrationSpecBase to take a proper execution
context instead of the serial one that ScalaTest provides, with a
default of `ExecutionContext.global`.

* ledger-on-memory: Make a proper owner with a proper execution context.

* kvutils/app: Remove `executionContext` from LedgerFactory.

Shouldn't need it in `ResourceOwner`. I was bad.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-memory: Make ResourceOwners real classes.

* ledger-on-sql: Make the ResourceOwner a real class.

* ledger-on-sql: Cause side effects on resource acquisition.

Not on owner construction.
2020-03-16 19:05:23 +00:00
Miklos
04a0723c81
Switch from Array[Byte] to ByteString on simplified ledger integration API (#4984)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-16 18:02:50 +01:00
mergify[bot]
95c1e5edaa
Allow LedgerFactory to provide a full-blown ReadWriterService (#4792)
* Allow `LedgerFactory` to provide a full-blown `ReadWriterService` rather than a `LedgerReaderWriter` (needed by at least vDAML)

CHANGELOG_BEGIN
CHANGELOG_END

* Address review points

* Address Samir's review point in `SqlLedgerFactory`

* Finish addressing Samir's review point in `SqlLedgerFactory`

* Split reader and writer owners in `LedgerFactory` as suggested by Gerolf

* Remove unneeded `val` from `KeyvalueParticipantState[Reader|Writer]` constructor params

* Remove unneeded type parameter from `app.Runner`

* Leave `LedgerFactory` a full ledger builder but split hierarchy upwards and clarify responsibilities

* Rename `SimpleLedgerFactory` to `KeyValueLedgerFactory`
2020-03-05 12:02:23 +00:00
Samir Talwar
89c23ef85d
Heartbeats for Sandbox-Next, ledger-on-sql, and ledger-on-memory. (#4755)
* kvutils: Make the `KeyValueParticipantStateReader` tests more rigorous.

If the `offset` is specified as `None`, expect it to be `None`, not
just anything.

* kvutils: Simplify `KeyValueParticipantStateReader#stateUpdates`.

Construct the Source with `Source.apply`, not `Source.fromIterator`.

* kvutils: Use multiple entry IDs in `KeyValueParticipantStateReaderSpec`.

* kvutils: Add basic tests to `KeyValueParticipantStateReaderSpec`.

* kvutils: Add heartbeats to `LedgerReader`'s `events` output.

Heartbeats are optional, to be delivered by the ledger if and when it
deems necessary.

* sandbox-next: An observing time service backend using Akka streams.

* sandbox-next: A regular heartbeat based on Akka Streams' `tick`.

* sandbox: Replace `TimeServiceBackend.withObserver` with `.observing`.

More code, but it's more decoupled, so can more easily be sent to the
underlying backend in Sandbox Next.

CHANGELOG_BEGIN
- [Sandbox] Fixed a bug in the command completions stream when running
  Sandbox in static time. Previously, upon updating the time, the old
  time was emitted on the completions stream. The new time is now
  emitted.
CHANGELOG_END

* sandbox: TimeServiceBackend should only emit accepted changes.

* ledger-on-memory: Use `LedgerRecord` directly.

* ledger-on-memory: Stream heartbeats to the log.

* ledger-on-memory: Encapsulate mutations behind locks at all times.

* ledger-on-memory: Differentiate between reading and writing.

* ledger-on-memory: Factor out appending to the log.

* kvutils: Move the heartbeat test into the base from ledger-on-memory.

* kvutils: Log when the submission validation fails unexpectedly.

* ledger-on-sql: Add a script to hash all migrations.

* ledger-on-sql: Publish heartbeats to the log, and stream them out.

* ledger-on-sql: Log if publishing the heartbeat failed.

* ledger-on-sql: Wrap all queries in `Try`.

Just to make sure that we don't throw from a function that returns `Try`
or `Future`.

* ledger-on-sql: Allow `Long` values as the heartbeat timestamp.

`INTEGER` really does mean 32-bit, apparently.

* sandbox-next: Pipe heartbeats to the ledger.

* ledger-on-sql: Make sure we publish the correct head after a heartbeat.

Off-by-one errors are the best errors.

* ledger-on-(memory|sql): Just accept heartbeats, not their owner.

* sandbox: Update CIDs in tests to account for the extra heartbeat.

* ledger-on-memory: Fix a reference to variable in a comment.

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* ledger-on-sql: `flatMap` over `Try` rather than `Future` when possible.

* sandbox: Make sure the heartbeat queues are thread-safe.

* kvutils: Remove `LoggingContext` from the interfaces.

Keep it internally. This means we'll drop any context, but otherwise
things should work as expected.

* sandbox-next: Pull out the heartbeat interval into a constant.

* ledger-on-sql|sandbox: Clarify large levels of nesting.

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-02 19:44:42 +00:00
Remy
e7ee8143cb
[kvutils] use new contract id scheme (#4741)
* Make kvutils work with the new contract id scheme

CHANGELOG_BEGIN
- [KVUtils] uses random contract id. Contract ids are made of 65 hexa decimal characters.
CHANGELOG_END

Co-authored-by: Jussi Mäki <jussi.maki@digitalasset.com>
2020-03-02 17:50:38 +01:00
Gerolf Seitz
d617922618
Remove InMemoryKVParticipantState (#4674)
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.

InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.

* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.

* Run all conformance tests in the repository in verbose mode.

This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.

This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:

  - ledger-api-test-tool-on-canton
  - ledger-on-memory
  - ledger-on-sql
  - sandbox




Fixes #4225.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-26 15:45:35 +01:00
Robert Autenrieth
79e7ca0627
Implement new command deduplication (#4467)
* Add TTL field to protobuf

* Add command deduplication to index service

* Wire command deduplication to DAO

* Implement in-memory command deduplication

* Remove Deduplicator

* Implement JDBC command deduplication

* Add TTL field to domain commands

* Deduplicate commands in the submission service

CHANGELOG_BEGIN
- [Sandbox] Implement a new command submission deduplication mechanism
  based on a time-to-live (TTL) for commands.
  See https://github.com/digital-asset/daml/issues/4193
CHANGELOG_END

* Remove unused command service parameter

* fixup protobuf

* Add configuration for TTL

* Fix Haskell bindings

* Rename SQL table

* Add command deduplication test

* Redesign command deduplication queries

* Address review comment

* Address review comment

* Address review comments

* Make command deduplication test optional

* Disable more tests

* Address review comments

* Address review comments

* Refine test

* Address review comments

* scalafmt

* Truncate new table on reset

* Store original command result

* Rename table columns

... to be consistent with other upcoming tables

* Rename migrations to solve conflicts

Fixes #4193.
2020-02-26 12:00:02 +01:00
fabiotudone-da
b2a79df0d8
Expand Runner's flexibility as a basis for participant servers (#4669)
* Add overridable indexer, api and auth configuration to `LedgerFactory`

CHANGELOG_BEGIN
CHANGELOG_END

* Add overridable indexer and api metrics creation to `LedgerFactory`

CHANGELOG_BEGIN
CHANGELOG_END

* Add overridable api's `TimeServiceBackend` to `LedgerFactory`

* 🎨 Fix formatting

* Port SDK ledgers based on `Runner` (and the sandbox) to `TimeServiceBackend`

* Revert to `TimeProvider` for committer usage and to `None` default for API server.
Also removed now unused `TimeServiceProvider.wallClock()`.

* Move TimeServiceBackend back to the API server.

* 🎨 Remove unneeded argument passed for parameter w/default

* Restore sandbox ledger time support

* Simplify passing a `TimeProvider` to the sandbox ledger

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-02-26 09:31:26 +01:00
Samir Talwar
2ec710b0e3
Sandbox & kvutils: Use the existing ledger ID if no ledger ID is specified. (#4607)
* sandbox: If the ledger ID isn't provided, use the one in the database.

Previously, we would fail if working against an existing ledger, and not
explicitly providing the ledger ID. This was the case even if the ledger
ID was randomly generated initially.

CHANGELOG_BEGIN
- [Sandbox] If no ledger ID is provided when running against an existing
  ledger, use the existing ID. Previously, Sandbox would fail to start.
CHANGELOG_END

* sandbox: The ReadOnlySqlLedger should always receive a ledger ID.

It's read-only; it can't create one.

* sandbox: Stop using `equal` in SqlLedgerSpec.

* sandbox: Test that the ledger ID is as specified in SqlLedgerSpec.

* sandbox: Let the top-level runner handle a ledger ID mismatch.

And clean up the log text.

* sandbox: Initialize the ledger properly when the ID is dynamic.

* sandbox: Use `Vector`, not `List`, for SqlLedger initialization.

Append with Vector, good. List, bad.

* ledger-api-common: Make `LedgerApiMode.Dynamic` an object.

And add Java-style static factory methods.

* kvutils/app | ledger-on-{memory,sql}: Make `ledgerId` optional.

It should be generated or retrieved from the persistence layer by the
ledger itself.

* kvutils: Make the ledger ID optional in the tests.

* ledger-on-sql: Store the ledger ID, and reject conflicting IDs.

* ledger-on-sql: Make more things final.

* ledger-on-sql: Document the `ledger_meta.table_key` column better.

* sandbox: Don't hardcode the number of packages in the test DAR.

It changes.

* ledger-on-sql: Merge the `head` resource owner with the `dispatcher`.

* sandbox: Use backticks to simplify pattern match in ReadOnlySqlLedger.

* ledger-on-sql: Extract methods in `owner`.
2020-02-20 10:35:16 +00:00
Samir Talwar
f231ebc1b3
Sandbox-Next: Re-establish static time support. (#4581)
* kvutils: Extract a committer from the uses of `SubmissionValidator`.

This makes the clock injectable too.

* kvutils: Provide logging contexts in the `Runner`.

* sandbox: Remove the `StaticAllowBackwards` time provider type.

It's not used anywhere.

* sandbox: Fix warnings in CliSpec.

* sandbox: Ensure that we cannot specify both static and wall-clock time.

* sandbox-next: Crash if wall clock time is not specified.

* sandbox-next: Document more known issues in the new Sandbox.

* sandbox: Add a Clock (and some tests) to TimeServiceBackend.

* sandbox-next: Support static time.

CHANGELOG_BEGIN
- [Sandbox Next] Re-establish static time mode.
CHANGELOG_END

* ledger-on-(memory|sql): Expect a `() => Instant`, not a `Clock`.
2020-02-18 17:03:15 +00:00
Stefano Baghino
8df29ac19e
Add more tests to default run of Ledger API Test Tool (#4561)
* Add more tests to default run of Ledger API Test Tool

Furthermore, drop TimeIT, which is a sandbox-only property (and tested already as part of its integration tests)

CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Ledger API Test Tool default tests modified. Use --list for the updated list of default tests. Time service test dropped from the suite.
CHANGELOG_END

* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380635979

* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380636989

* Optimize imports

* Only run semantic tests on Canton
2020-02-18 13:59:46 +00:00
Samir Talwar
b203ccaade
kvutils & ledger-on-sql: Avoid a race condition in dispatching. (#4546)
* kvutils/ledger-on-sql: Avoid a race condition in dispatching.

This changes the API of kvutils to allow for passing data out of the
transaction, which makes it much easier to ensure the new head makes it
to the dispatcher. Previously, we would use an `AtomicLong` to
communicate the data, but this was problematic because values could
arrive out in the wrong order. For example:

  - log index 5 is committed
  - `head` is updated to 6
  - the dispatcher is signalled with a head of 6
  - log index 6 is committed
  - log index 7 is committed
  - `head` is updated to 8
  - `head` is updated to 7
  - the dispatcher is signalled with a head of 7

In this scenario, we would have to wait until a new commit comes in
before the indexer finds out about log index 7.

* kvutils: Just return an `Either`from `SubmissionValidator`.

It was either that or introduce yet another type to split
`SubmissionValidated` into two.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Make ValidationFailed extend NoStackTrace.
2020-02-18 09:08:16 +00:00
Samir Talwar
6577ae2556
ledger-on-sql: Switch to the extra-simplified KVUtils API. (#4532)
* kvutils: Make it easier to import the Key and Value types.

Moved them into a companion object.

* kvutils: Flatten the nested `for` comprehensions in SubmissionValidator.

* kvutils: Test that results succeed before querying state updates.

If they don't, we get idle timeouts and we don't see the error, which is
unhelpful.

* sandbox: Increase log levels for failed submissions.

* kvutils: Avoid converting DamlStateKey -> ByteString -> Array[Byte].

Just convert once.

* ledger-on-sql: Use the `SubmissionValidator` from kvutils.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Remove unnecessary interpolation in ApiSubmissionService.
2020-02-15 22:00:04 +00:00
Miklos
a440a3fabd
Simplified API fixes (#4505)
Removed LedgerStateAccess.participantId and renamed LedgerReader.retrieveLedgerId => ledgerId.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-13 19:17:07 +01:00
Miklos
c7183979fd
Added submission validator (#4437)
CHANGELOG_BEGIN
CHANGELOG_END
2020-02-10 17:07:16 +01:00