Commit Graph

814 Commits

Author SHA1 Message Date
Remy
50694c7abe
DAML-LF. fix parameter and self binder in choices. (#5444)
* DAML-LF. fix parameter and self binder in choices.

* fix

* DAML-LF: address moritz's comments

* changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-04-06 14:32:30 +00:00
Robert Autenrieth
8397467712
Wait for removing deduplication entries (#5442)
* Wait for removing deduplication entries

This addresses a potential race condition

CHANGELOG_BEGIN
CHANGELOG_END

* Fix broken code
2020-04-06 15:54:20 +02:00
Gerolf Seitz
a2d785e3ee
Use com.daml as root package (#5343)
Packages com.digitalasset.daml and com.daml have been unified under com.daml

Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.

CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
2020-04-05 19:49:57 +02:00
Stefano Baghino
a7474f4c50
Fix test for groupContiguous (#5432)
* Fix test for groupContiguous

Automatic generation of test values was prone to cause flakiness, removed in favor of a simpler test case.

changelog_begin
changelog_end

* Relax order sensitivity

* Update ledger/sandbox/src/test/suite/scala/com/digitalasset/platform/store/dao/events/GroupContiguousSpec.scala

Co-Authored-By: Remy <remy.haemmerle@daml.com>

* Fix compilation issue

Co-authored-by: Remy <remy.haemmerle@daml.com>
2020-04-03 16:58:03 +00:00
Gerolf Seitz
e3f3ebcb9b
Suppress DB related logging on sandbox startup. (#5419)
Fixes #5401

CHANGELOG_BEGIN
CHANGELOG_END
2020-04-03 15:28:47 +02:00
Gerolf Seitz
16240b31cc
Add comments to ApiOffset and SandboxOffset (#5414)
CHANGELOG_BEGIN
CHANGELOG_END
2020-04-03 14:26:38 +02:00
Samir Talwar
4de360a47a
sandbox: Move server parameters into the Config objects. (#5413)
* sandbox: Move the events page size configuration value into config.

* sandbox: Pass `config` directly into JdbcIndexerFactory.

* sandbox: Reorder `eventsPageSize` before `metrics` in parameters.

* sandbox: Move `seeding` into `ApiServerConfig`.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Name all parameters of `JdbcLedgerDao.writeOwner`.

Co-Authored-By: stefano.baghino@digitalasset.com
2020-04-03 11:20:05 +00:00
Gerolf Seitz
5ddbd5c511
Implement timed command deduplication in kvutils (#5242)
* Implement timed command deduplication in kvutils

This adds a field deduplication_time to DamlCommandDedupValue for
deduplication timeout checking.

* Bump kvutils version to 4

* Fix CommandTracker pulling commandResultIn multiple times
Now that the timeouts are generated out of band, we have 2
"unsynchronized" places that pull on commandResultIn.
Whenever we pull, we need to check that commandResultIn hasn't been
pulled before.

* Add inStaticTimeMode flag to enable command dedup in sandbox-next with static-time

Fixes #4624.

CHANGELOG_BEGIN
[kvutils] KVUtils now respects the command deduplciation time instead of
deduplicating commands forever.
CHANGELOG_END
2020-04-03 10:44:35 +02:00
Stefano Baghino
e2cab1c162
Integrate new active contracts reader in Ledger API (#5397)
changelog_begin
changelog_end
2020-04-03 08:17:42 +00:00
Samir Talwar
1d9c7d2574
kvutils: Remove the LedgerEntry trait; it's no longer necessary. (#5383)
* kvutils: Remove the LedgerEntry trait; it's no longer necessary.

This was introduced to allow for heartbeats, which no longer exist.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Make LedgerRecord a case class again.

We used to store the envelope as an array of bytes, which doesn't have a
value-based `equals` method and therefore should not be used in a case
class. We now use a `ByteString`, so this is no longer an issue.
2020-04-03 07:19:45 +00:00
Stefano Baghino
446892a80e
Read active contracts for the Ledger API from new schema (#5378)
changelog_begin
changelog_end
2020-04-02 22:27:15 +00:00
Stefano Baghino
40e41cb927
Migrate create_consumed_at to offset (#5371)
* Migrate create_consumed_at to offset

This is a leftover from the stable offsets migration and causes issues when serving active contracts from the new schema.

changelog_begin
changelog_end

* No decoding necessary for create_consumed_at
2020-04-02 20:24:01 +02:00
Samir Talwar
a535a54014
kvutils: Do a little less inside the submission transaction. (#5368)
* kvutils: Use `.view` in SubmissionValidator.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils: Don't compute missing inputs unless we're asked.

* ledger-on-memory: Do less in `InMemoryLedgerStateOperations`.

* ledger-on-memory: Use `RangeSource` instead of `OneAfterAnother`.

Should be faster to just take a slice.

* ledger-on-memory: Don't bother locking when reading.

We're only reading the log, which is append-only; we never mutate
existing data. This means we don't need to lock to read it.

* ledger-on-memory: Make it impossible to construct a state with data.
2020-04-02 14:51:18 +00:00
Samir Talwar
e3ff700a67
sandbox: Wait longer in tests to start the Sandbox. (#5380)
This can fail in CI, especially when we're running tests in parallel,
and starting PostgreSQL as well.

I've increased the limit from 30 seconds to 1 minute.

This is hopefully not a permanent fix; I'm going to look into doing this
without an `Await.result`.

CHANGELOG_BEGIN
CHANGELOG_END
2020-04-02 14:44:00 +00:00
Stefano Baghino
b838d6f1ca
Refactor flat events range queries (#5365)
* Refactor flat events range queries

By factoring out query routing, this logic can be re-used to serve active contracts.

changelog_begin
changelog_end

* Fix copyright notice header
2020-04-02 10:07:42 +00:00
Robert Autenrieth
3945708192
Don't deduplicate failed submissions (#5333)
* Pass explicit submitter and command Id

* Stop deduplicating commands on submission failure

CHANGELOG_BEGIN
CHANGELOG_END

* Add test
2020-04-01 23:38:23 +00:00
Stefano Baghino
c23564b0cc
Use snake_case for all metrics (#5348)
* Use snake_case for all metrics

This makes sure we're using the snake_case for all metrics. DAML-on-X originally asked us for this.

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/5348#issuecomment-607326336
2020-04-01 18:53:02 +00:00
Stefano Baghino
f85782e1ba
Integrate transaction trees from new schema in Ledger API (#5346)
* Integrate transaction trees from new schema in Ledger API

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/5331#discussion_r401490187
2020-04-01 16:45:40 +00:00
Samir Talwar
fca8895954
kvutils | sandbox: Wrap metric names in a value type. (#5306)
* participant-state-metrics: Wrap metric names in a value type.

For safety, and for simplicity when building upon prefixes.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Use MetricName within MetricsNaming.

* kvutils: Use `MetricName` in `Committer`.

* sandbox | kvutils: Extract common metric prefixes.

* sandbox: Remove a redundant visibility modifier in `MetricsNaming`.

* participant-state-metrics: `MetricName` doesn't need to be a case class.

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-04-01 15:35:28 +00:00
Stefano Baghino
5cfe296253
Read transaction trees from new schema (#5331)
Integration to come in a follow-up PR.

changelog_begin
changelog_end
2020-04-01 13:52:15 +00:00
Robert Autenrieth
9e37ed4d46
Retry if time lookup failed (#5327)
* Retry if time lookup failed

An error while looking up the maximum ledger time for the used contracts
likely means that one of the contracts is already not active anymore,
which can happen under contention.
Retry finding a suitable ledger time in this case instead of crashing.

CHANGELOG_BEGIN
CHANGELOG_END

* Handle archived contracts in JDBC max time lookup

* Don't retry if ledger time lookup fails

The retry might not fix anything.
Instead, log a helpful message.
2020-04-01 11:52:08 +00:00
Gerolf Seitz
effceda612
Remove heartbeats in participant server and kvutils (#5315)
Contributes to #4231.

Remove checkpoints from Participant Server storage:
- Removed code to store checkpoints in the index database.
- Remove existing checkpoint rows in ledger_entries and participant_command_completions.
- Removed ObservedTimeServiceBackend

This commit modifies the java migration V2_1__Rebuild_Acs. This is safe to do, because:
a) any even semi recent persistent sandbox had already gone through this migration and won't re-run it
b) A new database doesn't even have entries yet to migrate.

CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Removed the ``Hearbeat`` state update.
[Sandbox] Checkpoints are no longer emitted in regular intervals in wall
clock time mode.
CHANGELOG_END
2020-04-01 12:34:23 +02:00
Gerolf Seitz
329320bad9
Organize maven coordinates (#5272)
* Use com.daml as groupId for all artifacts

CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END

* Add 2 additional maven related checks to the release binary

1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId

* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
2020-04-01 11:41:18 +02:00
Stefano Baghino
d83d590ba4
Refactor tree events query (#5328)
This will allow sharing parts of the query.

changelog_begin
changelog_end
2020-04-01 09:40:37 +00:00
Gerolf Seitz
9b5c5e23b9
Make wall clock time the default time mode for sandbox (#5301)
CHANGELOG_BEGIN
[Sandbox] Wall Clock Time mode (``--wall-clock-time``) is now the
default.
CHANGELOG_END
2020-04-01 08:53:16 +02:00
Stefano Baghino
f8c9ab6354
Integrate new flat transactions reader with Ledger API (#5305)
* Integrate new flat transactions reader with Ledger API

- fixes outstanding issue with package id ignored when filtering
- adds --events-page-size CLI option to set the fetch size when streaming transactions

changelog_begin
[Sandbox] The --events-page-size option allows to tweak the fetch size when streaming
transactions.
changelog_end

* Address https://github.com/digital-asset/daml/pull/5285#discussion_r400711685

* Fix outstanding compilation errors

* Address https://github.com/digital-asset/daml/pull/5305#discussion_r400966816
2020-03-31 16:09:03 +00:00
Stefano Baghino
aeefaa1d0c
Add queries to get flat transactions straight from the index (#5285)
* Add queries to get flat transactions straight from the index

changelog_begin
changelog_end

* Fix compilation errors

* Remove unused method, loosen transaction check

* Fix grammar in test case description

Thanks @SamirTalwar-DA

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-03-31 14:13:14 +02:00
Samir Talwar
a9ca9d3956
kvutils: Time transaction acquisition and release. (#5292)
* kvutils: Do less in the ledger transaction.

* kvutils: Make `SubmissionValidator#runValidation` tail-recursive.

Because otherwise IntelliJ is very unhappy.

* kvutils: Time transaction acquisition and release.

CHANGELOG_BEGIN
- [Ledger Integration Kit] Metrics for acquiring and releasing
  transactions.
CHANGELOG_END

* kvutils: Include the word "lock" in validator transaction lock metrics.

* kvutils: Record successful + failed transaction acquisitions separately.
2020-03-31 10:21:13 +00:00
Samir Talwar
499bb924cf
kvutils: Prefix metrics with "daml.". (#5289)
Just to get them consistent with the others.

CHANGELOG_BEGIN
- [Ledger Integration Kit] Prefixed all metrics with "daml." for
  consistency.
CHANGELOG_END
2020-03-31 08:18:29 +00:00
Andreas Herrmann
89a9f5c7d2
tarball reproducibility (#5258)
* integration-tests reproducibility

* package-app reproducibility

* Make remaining tar czf reproducible

* package-app

CHANGELOG_BEGIN
CHANGELOG_END

* Reproducibility of remaing tar invocation

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-03-31 10:09:52 +02:00
Samir Talwar
2e9d7adfa9
sandbox|kvutils: Use MetricRegistry.name to create metric names. (#5274)
* kvutils: Use `MetricRegistry.name` to create metric names.

String interpolation is 4TL.

* sandbox: Make the metric names of the services consistent with kvutils.

CHANGELOG_BEGIN
- [Sandbox] Move the service metrics from ``daml.sandbox.indexService``
  and ``daml.sandbox.writeService`` to ``daml.services.index`` and
  ``daml.services.write`` respectively. This brings Sandbox in line with
  the Ledger Integration Kit.
CHANGELOG_END

* sandbox: Use `MetricRegistry.name` to create metric names.

String interpolation is 4TL.
2020-03-30 20:14:06 +00:00
Remy
3fc1175436
Engine: force recent version for serializing transactions and values. (#5232) 2020-03-30 16:29:18 +02:00
Samir Talwar
66a7fc6384
kvutils/app: Publish metrics someplace. (#5267)
* kvutils/app: Publish JVM metrics.

* kvutils/app: Sort the metrics prefixes.

* kvutils/app: Allow for specifying a metrics reporter as a CLI argument.

No changelog as this is hidden.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Remove an unnecessary string interpolation mark.

Co-Authored-By: Miklos <57664299+miklos-da@users.noreply.github.com>

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2020-03-30 13:00:42 +00:00
Stefano Baghino
746b774e82
Add canonical string representation for identifiers (#5270)
* Add canonical string representation for identifiers

An identifier is represented as a string as a package identifier and a qualified name separated by a colon.

changelog_begin
changelog_end

* Simplify explicit type signature for HexString

Thanks @remyhaemmerle-da

Co-Authored-By: Remy <remy.haemmerle@daml.com>

* Fix hashing test expected output

* Fix key hasher test expected output

Co-authored-by: Remy <remy.haemmerle@daml.com>
2020-03-30 12:55:11 +00:00
Samir Talwar
28b21d07c7
ledger-on-sql: Squash the migrations. (#5265)
Before this reaches production-ready status, it's time to squash the
migrations. This will improve startup performance a little.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-30 12:51:33 +00:00
Remy
4cb3ea5c7c
Ledger: clean up contract id seeding (#5260)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-30 10:13:42 +02:00
Robert Autenrieth
f8cd628c2f
Remove old time model from ledger config (#5216)
* Remove old time model from ledger config

CHANGELOG_BEGIN
- [Ledger API] Fields related to the old ledger time
  model have been removed from the configuration
  management service and the ledger configuration service.
CHANGELOG_END

* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/tests/LedgerConfigurationService.scala

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-30 08:46:22 +02:00
mziolekda
0f3c0d66d8
docker image for ledger-on-memory (#5256)
* docker image for ledger-on-memory

CHANGELOG_BEGIN
CHANGELOG_END

* Apply suggestions from code review

Applying Samir's suggestions to the README file

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

* Update ledger/ledger-on-memory/README.md

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-03-29 10:54:12 +02:00
Robert Autenrieth
0a83309f1b
TTL cleanup (#5246)
* Remove documentation for a removed option

CHANGELOG_BEGIN
- [DAML Integration Kit] The CLI option command-submission-ttl-scale-factor was
  removed, as the LET/MRT/TTL fields have recently been removed
  from the command submission service.
CHANGELOG_END

* Minor renaming
2020-03-28 00:13:08 +01:00
Remy
1b37f6c482
DAML-LF: redesign absolute contract ids (#5207)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 19:07:42 +01:00
Samir Talwar
7ef0972856
sandbox: On reset, wait for the API server to start before replacing it. (#5251)
* sandbox: On reset, wait for the API server to start before replacing it.

Hopefully this addresses the flickering behavior we're seeing in CI.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Scale the retries in ResetServiceIT along with everything else.
2020-03-27 16:29:56 +00:00
Stefano Baghino
9522c6e4b2
Allow to have both verbose and succinct row parsers (#5248)
* Allow to have both verbose and succinct row parsers

This will be used when we need to stream transactions and not individual
events.

changelog_begin
changelog_end

* Succinct parser should have verbose = false
2020-03-27 17:00:36 +01:00
mziolekda
ce62bfb596
Support for large grpc messages in ledger-on-memory (#5230)
* Support for large grpc messages in ledger-on-memory

* adress review comments

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 14:00:24 +01:00
Stefano Baghino
4543917f8b
Refactor query preparation for flat events (#5241)
* Refactor query preparation for flat events

This will allow to share code with the stream endpoints.

changelog_begin
changelog_end

* Allow for multiple requesting parties

* Safer approach to build queries

Thanks @SamirTalwar-DA

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

* Fix compilation errors

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-03-27 12:41:08 +00:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Gary Verhaegen
9f3900d870
pin copyright headers (#5227)
When files should be immutable, they should also be protected from the
copyright headers script.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-26 23:30:23 +01:00
Samir Talwar
5c913106fc
sandbox: Report JVM metrics. (#5217)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-26 18:31:59 +00:00
Jussi Mäki
e64777c1a6
Introduce DamlSubmissionBatch and the BatchingLedgerWriter (#4964)
* Introduce DamlSubmissionBatch and the BatchingLedgerWriter

This introduces the DamlSubmissionBatch message to group
submissions into a single message and extends Envelope to
carry the batch. We're using the envelope wrapping for consistency
and compatibility.

We're adding this to kvutils version 3 as it has not yet been released
into the wild and as this is not backwards incompatible change.

Support for batching is implemented with the BatchingLedgerWriter
that wraps a LedgerWriter and groups submissions into a batch based
on size and time duration.

For implementing the validation of a batch we will require some rework
in the SubmissionValidator to be able to produce multiple "LogResult"s,
e.g. commit on the in-memory ledger results in an "Index" which is used
to signal new head to dispatcher. With a batch we'd need to pick max index.

CHANGELOG_BEGIN
CHANGELOG_END

* Add missing copyright header

* Address code reviews

* Post rebase fixes

* Rename BATCH -> SUBMISSION_BATCH

* Address code reviews, add further tests and cleanup.

* Add test for DefaultBatchingQueue.close

* Use generous timeouts

* Renamed BatchMessage => SubmissionBatchMessage. Added default boolean parameter value. Added simple test case.

* Removed unused include.

* Address final code review

Co-authored-by: Miklos Erdelyi <miklos.erdelyi@digitalasset.com>
2020-03-26 16:37:52 +00:00
Samir Talwar
1ce67fbd2f
kvutils: Metrics on the write path. (#5212)
* kvutils: Add metrics to SubmissionValidator.

* kvutils: Add metrics to KeyValueSubmission.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-26 15:17:42 +00:00
Ognjen Maric
f7aa2a3db0
Add keys with maintainers to Fetch nodes (#5186)
Add keys with maintainers to Fetch nodes

The new field is populated by the interpreter whenever the fetched
contract has a key. Used for contract key reinterpretation in Canton.

CHANGELOG_BEGIN

- [DAML-LF] Add keys with maintainers to Fetch nodes in transactions.

CHANGELOG_END
2020-03-26 14:07:17 +01:00