Commit Graph

317 Commits

Author SHA1 Message Date
Robert Autenrieth
2159de08ef
Remove index transformation (#6789)
The index is not part of the public API,
and we want to always enable metrics.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-20 09:20:59 +02:00
Robert Autenrieth
1074736316
Add time model documentation (#6705)
* Update existing docs, removing references to old time model
* Add detailed time model docs
* Rename ledger effective time to ledger time

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-17 17:54:26 +02:00
Robert Autenrieth
7ce9748066
Split sandbox code into separate packages (#6695)
* Move public code into daml-integration-api

CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
2020-07-17 17:06:06 +02:00
fabiotudone-da
ebf312873c
Add CachingDamlLedgerStateReaderWithFingerprints factory method (#6769)
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-17 16:07:54 +02:00
Gerolf Seitz
35b50992f0
Allow running StandaloneApiServer in read-only mode. (#6721)
Also fixes #5635, removing usage of `ReadService` from the `StandaloneApiServer`.

The configuration stream of the Ledger API LedgerConfigurationService is now properly backed by the configuration entries instead of just serving the initial configuration.

CHANGELOG_BEGIN
[DAML Integration Kit]: ``StandaloneApiServer`` can now be run in a read-only mode.
  - The type of the constructor parameter ``writeService`` of ``StandaloneApiServer`` changed to ``Option[WriteService]``. Passing ``None`` will not start any of the admin services, the command service, and the command submission service.
  - The constructor parameter ``readService`` of ``StandaloneApiServer`` has been removed.
  - A new constructor parameter ``ledgerId`` has been added to ``StandaloneApiServer``. It is used to verify that that ``StandaloneApiServer`` is run against an index storage for the same ledgerId. Initialization is aborted if this is not the case.
[DAML Integration Kit]: The ``LedgerConfigurationService`` now properly streams configuration changes.
CHANGELOG_END
2020-07-17 09:57:00 +02:00
Remy
8d74754450
kv tools: fix typos in kvutils tools README (#6758)
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-16 16:59:34 +02:00
Remy
15da222939
DAML-LF: move some type definitions out of Transaction object (#6739)
SubmittedTransaction and CommittedTransaction are moved from com.daml.lf.transaction.Transaction to
com.daml.lf.transaction

This helps intelliJ type inference.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-16 09:13:30 +02:00
Moritz Kiefer
91302d0f4a
Fix merge conflict in KVUtils test adapter (#6748)
NodeId from the Transaction object got deprecated.

changelog_begin
changelog_end
2020-07-15 19:30:01 +00:00
Remy
471863570a
Replay benchmark: adapt dar to ledger export (#6661)
Try to match transactions from the ledger export to a different non original dar.
This is useful to benchmark a different compilation of the same original daml source.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-15 21:10:35 +02:00
Miklos
f7bbfc2a1b
[kvutils] Added initial set of metrics for pre-execution (#6738)
* Added metrics for decoding and total time of pre-execution.
CHANGELOG_BEGIN
CHANGELOG_END

* Code tidying.

* Added timer for tracking time spent with generating write sets.
2020-07-15 14:27:14 +00:00
Miklos
8e04b2add9
[kvutils] Do not expose fingerprints to Committers (#6737) 2020-07-15 14:59:44 +02:00
Miklos
761787ce4b
[kvutils] Pre-executing submission validator (#6711)
* Removed engine constrcutor parameter.

* Basic abstractions for pre-executing submission validator.

* Fix build.

* Shared byteToLogEntryId with BatchedSubmissionValidator.

* Added some more metrics.

* Removed involved participants from PreExecutionResult.
CHANGELOG_BEGIN
CHANGELOG_END

* Reformatted/added headers.

* Fix build.

* Code tidying.

* Moved PreExecutionOutput into a separate file.

* Moved type definition into companion object.

* Added some tests for PreExecutingSubmissionValidator.

* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/validator/preexecution/LedgerStateReaderWithFingerprints.scala

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>

* Removed left-in code.

* Added some assertions for .zip operations.

* Test result for unexpected message.

* Test setting min/max record time.

* Reuse code for generating a key-value pair for log entries.

* Tests for CachingDamlLedgerStateReaderWithFingerprints.

* Store empty values as well with their fingerprints.

* Moved weight function into companion object.

* Test that the results come back in right order.

* Code tidying.

* Basic test case for LogAppenderPreExecutingCommitStrategy. Added missing headers.

* Code tidying.

* Fix build.

* Fix build.

* Revert "Store empty values as well with their fingerprints."

This reverts commit ca68ea967f.

# Conflicts:
#	ledger/participant-state/kvutils/src/test/suite/scala/com/daml/ledger/validator/caching/CachingDamlLedgerStateReaderWithFingerprintsSpec.scala

* Added some docs.

* Added test case for ensuring that we don't cache None values.

* Fix build.

Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
2020-07-15 08:55:19 +00:00
Gary Verhaegen
8176fb0c8d
fix the the typo (#6723)
```
s/the the /the /
```

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-14 16:55:33 +00:00
Remy
121a6a7d77
DAMLLF: clean up Transaction definitions (#6714)
* LF: move NodeId from value to transaction

* LF: remove some `private` keywords in Transaction

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-14 14:42:58 +02:00
fabiotudone-da
3f2938ecf6
Add KVUtils pre-execution round-trip test (#6671)
* Round-trip pre-execution tests for all committers

* Address review comments

* Address review comments
2020-07-14 09:29:07 +00:00
Miklos
1edc02a4f7
Populate deduplicateUntil in DamlOutOfTimeBoundsLogEntry by Committers (#6672)
* Removed unused code.

* Basic interface draft for producing pre-execution results.

* Share code for running all steps for a committer.

* Code tidying.

* Removed type parameter Submission from Committer.

* Simplify creation and using of committers.

* ExecuteSubmission => SubmissionExecutor

* Code tidying.

* Code tidying.
CHANGELOG_BEGIN
CHANGELOG_END

* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/committer/Committer.scala

Co-authored-by: Andreas Lochbihler <36634420+andreaslochbihler-da@users.noreply.github.com>

* Fixed build.

* Basic test cases for dryRun.

* Set to-be-notified participants to be everyone.

* Some tests for runSteps.

* Fixing merge.

* Organized imports.

* Added separate timer for pre-execution.

* Set min/max record time from context.

* preexecution => preExecution

* Code tidying.

* Code tidying.

* Set out-of-time-bounds log entry from context.

* Code tidying.

* Code tidying.

* Keep track of deduplication time window for transactions.

* Set deduplicateUntil in the generated log entry.

* Made min/max record time optional in pre-execution result.

Co-authored-by: Andreas Lochbihler <36634420+andreaslochbihler-da@users.noreply.github.com>
2020-07-10 17:50:02 +00:00
Miklos
f60ec5c2ed
[kvutils] Handle offsets in case of multiple updates from the same log entry (#6677) 2020-07-10 19:11:32 +02:00
Miklos
2e836dc6b3
Enable pre-executing submissions (#6592) 2020-07-10 19:03:20 +02:00
Miklos
4b94c27600
Generate out-of-time-bounds rejection log entries (#6630) 2020-07-09 14:28:44 +02:00
fabiotudone-da
9a135fa6f2
Revert unneeded kvutils format version bump (#6644)
* Revert unneeded kvutils format version bump

CHANGELOG_BEGIN
CHANGELOG_END

* Re-add change comment as part of the current (5) version.
2020-07-08 12:28:25 +02:00
Remy
e09755ca08
Replay transaction (#6464)
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-08 10:21:48 +02:00
fabiotudone-da
474e92c13c
Track min and max record time in CommitContext when pre-executing (#6591) 2020-07-08 09:53:44 +02:00
Stephen Compall
4cd419623a
replace traverseU and sequenceU with traverse and sequence (#6594)
* replace traverseU and sequenceU with traverse and sequence

- with -Ypartial-unification on, the extra Unapply typeclass lookup is
  unnecessary

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* limit imports; we only need *> and void
2020-07-07 15:01:58 +00:00
Miklos
1ffb28cde9
Renamed some default variables (#6627)
* BatchedSubmissionValidatorParameters.default => reasonableDefault

* Config.default() => createDefault()

* ExtraConfig.default => reasonableDefault
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-06 16:52:47 +00:00
Samir Talwar
304f8d4201
kvutils/integrity-check: Clean up code around printing. (#6619)
* kvutils/integrity-check: Always reset the output color.

Contrary to popular belief, terminal colors are hard. Resetting to
"white" isn't good enough; you don't know which color the user's
terminal should be set to. We need to print `AnsiColor.RESET` to ensure
the terminal is properly reset.

This changes the way we print so we _always_ set the color and then
reset it in one operation, making it impossible to accidentally leave
the terminal using a specific color. It also fixes the above bug as a
side effect.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils/integrity-check: Print the total outside `go`.

* kvutils/integrity-check: Push the error outwards.

I'd rather avoid calling `sys.exit` inside a loop if I can.

* kvutils/integrity-check: Implicits for colors.

Before:

```scala
Print.green("Hello!")
```

After:

```scala
println("Hello!".green)
```
2020-07-06 15:23:17 +00:00
Samir Talwar
f1a00f40bf
kvutils/integrity-check: Rewrite without Await. (#6614)
* kvutils/integrity-check: When there's an exception, crash.

The integrity checker would previously hang around for a long time in
the event of an exception. This makes sure all threads are shut down
immediately.

* kvutils/integrity-check: Increase the timeout per submission.

Larger submissions time out on my machine.

CHANGELOG_BEGIN
CHANGELOG_END

* kvutils/integrity-check: Rewrite without `Await`.

* kvutils/integrity-check: Close the file after running.
2020-07-06 10:59:11 +00:00
Remy
dbd0215dd0
DAML-LF: cleanup TransactionBuilder (#6610)
small clean up in the testing library TransactionBuilder, bsaically add a Submitted and a Committed version for empty transaction constant and build method.

plus a bit of cleaning in tests using TransactionBuilder


CHANGELOG_BEGIN
CHANGELOG_END
2020-07-06 11:25:36 +02:00
Miklos
c789b65bd8
Do not expose log entry ID to committers (#6603)
* Removed archived_by_entry from CommitContext and DamlContractState.

* Removed references to getEntryId from test code.

* Do not pass log entry ID to committers.

* Log entry ID is no longer required for computing submission outputs.

* Log entry ID is no longer required for computing submission outputs.

* Code tidying.
CHANGELOG_BEGIN
CHANGELOG_END
2020-07-06 08:30:55 +00:00
Miklos
52fc0635f1
Handle out-of-time-bounds log entry (#6568) 2020-07-03 15:27:27 +02:00
fabiotudone-da
43f99f45fc
Add support for fingerprints in CommitContext (#6567) 2020-07-03 10:57:49 +02:00
Martin Huschenbett
873e0163cd
Drop the global from globalImplicitDisclosure and globalDivulgence (#6588)
Since we have only absolute contract ids, there is no more distinction
between local and global disclosure/divulgence. Let's please remove the
`global` prefix since it causes confusion (at least for me it did).

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-03 08:43:31 +02:00
Stephen Compall
86148c398b
make @silent annotations narrower (#6570)
* check that all @silent annotations are used

CHANGELOG_BEGIN
CHANGELOG_END

* make all @silent annotations very specific about what they're silencing
2020-07-01 20:53:22 +00:00
Miklos
5302c57593
Make record time optional for committers (#6538) 2020-07-01 19:20:07 +02:00
fabiotudone-da
bf36f32fca
CommitContext records accessed input keys (#6561)
* `CommitContext` records accessed input keys

CHANGELOG_BEGIN
CHANGELOG_END

* Simplify state and tests for `accessedInputKeys`.

* Add note about stable VS deterministic iteration order

* Add test about not recording input keys that are not accessed.

* Use regular a regular set for accessed keys as nor iteration stability nor determinism are needed
2020-07-01 16:57:37 +00:00
Miklos
ccfb0ac94e
Make interpretation cost available for LedgerWriter/WriteService (#6515)
* Measure time of interpretation and store it in CommandExecutionResult.

* Added parameters for passing in interpretation time for WriteService and LedgerWriter.

* Code tidying.

* Added CommitMetadata parameter to LedgerWriter.

* Thread through interpretation time. Aggregate interpretation times for a batch.

* Take max of interpretation times.

* Calculate interpretation time in nanos.

* Moved CommitMetadata into separate file.
CHANGELOG_BEGIN
CHANGELOG_END

* Apply suggestions from code review

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Delegate deprecated methods to method with new signature.

* Code tidying.

* Suppress deprecation warnings.

* Made interpretation cost optional in CommitMetadata.

* Test that we populate interpretation time.

* Code tidying.

* Apply suggestions from code review

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Code tidying.

* Some more tests. Do not return interpretation cost for a batch if it only contains non-transaction submissions.

* Reformatted.

* Reformatted.

* Named arbitrary interpretation cost.

* Reverted changes for BatchingLedgerWriter.

* Always drop commit metadata for batches and don't report interpretation cost.

* More specific expectations.

* Include authorization check and blinding in interpretation time.

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-06-30 16:45:12 +00:00
Miklos
f01a8e66dc
Remove record time from DAML state values (#6526) 2020-06-30 17:53:08 +02:00
Samir Talwar
2404d18b5d
kvutils: When creating the reference ledger dump, do it less eagerly. (#6525)
Our build is intermittently failing on CI on the target,
`//ledger/participant-state/kvutils:reference-ledger-dump`.

Creating the reference ledger dump exercises ledger-on-memory with the
Ledger API Test Tool as a side effect. The real goal is to run any kind
of kvutils ledger and generate a load of data. With that in mind, the
idea is to be somewhat stable, not fast.

This decreases the number of concurrent tests running at once to
generate the reference dump from the number of CPUs (presumably 8 on CI,
though I'm not sure) to 4.

It does not change the number of concurrent tests when running
`//ledger/ledger-on-memory/conformance-test`. This will still default to
the number of CPUs.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-29 15:02:08 +00:00
Stefano Baghino
36a9042594
Remove localDisclosures from BlindingInfo entirely (#6509)
* Address https://github.com/digital-asset/daml/pull/6507#discussion_r446113575

* drop blindinginfo.proto

changelog_begin
changelog_end

* drop BlindingCoder

* Remove blindinginfo Protobuf definition JAR

changelog_begin
[DAML-LF] The blindinginfo Protobuf definition JAR, which was previously unused, has been pulled from the artifacts released on Maven
changelog_end

Co-authored-by: Remy Haemmerle <Remy.Haemmerle@daml.com>
2020-06-26 14:50:49 +00:00
Remy
46b68bd92a
ledger test tool: refactor conformance test dars (#6506)
* ledger test tool: refactor conformance test dars

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-26 13:39:00 +02:00
Remy
6d27825acb
Engine: introduce config to control value/transaction output version (#6428)
* Engine: introduce configuration to control value/transaction output version

This advances the state of #5164

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-26 13:38:38 +02:00
Stephen Compall
a51d0db8ff
set scalac -Xsource:2.13 -Ypartial-unification globally (#6469)
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts

* add now-referenced scalaz-core where needed

* work around bad type signatures in scalatest Aggregating, Containing

* unused Any suppression

* work around bad partial-unification wrought by type alias

* remove unused Conversions import

- not required in 4f68cfc480 either, so unsure how it's survived this long

* work around Future.traverse; remove unused show import

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused bounds

* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed

* longer comment on what the options do

- suggested by @stefanobaghino-da; thanks

* forget Future.traverse, just use scalaz, it knows how to do this
2020-06-24 16:51:24 -04:00
Miklos
9dd78fc508
Ledger data export for non-determinism checks (#6418)
CHANGELOG_BEGIN
CHANGELOG_END
2020-06-19 17:52:11 +02:00
Samir Talwar
6496ddd903
caching | kvutils | sandbox: Use a size-based cache for LF value translation. (#6432)
* caching: Split caches into new files.

* caching: Rename `Cache.from` to `WeightedCache.from`.

* caching: Move `Configuration` inside `WeightedCache`.

* caching: Add test cases.

* caching: Allow for Caffeine builders to be covariant.

* caching: When instrumenting the Caffeine cache, compose, don't inherit.

* caching: Add a size-based cache.

* caching: Extract out common test cases into base classes.

* caching: Use the size-based cache for LF value translation.

CHANGELOG_BEGIN
CHANGELOG_END

* caching: Simplify the eviction tests.

* caching: Increase the encapsulation in CaffeineCache.

* caching: Commas are important.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-06-19 14:42:40 +00:00
Remy
86ceeea8d8
LF: shield value versioning in Transaction builder. (#6415)
We want to avoid that code outside LF version is own LF values.

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-18 17:20:47 +02:00
Remy
cbe423fc6c
LF: rename library transaction-scalacheck to transaction-test-lib (#6347)
* LF: rename library transaction-scalacheck to transaction-test-lib

CHANGELOG_BEGIN
CHANGELOG_END

* move files in com/daml

* missing change in release/artifacts.yaml

* remove 'com/dam' from the path
2020-06-17 10:52:40 -04:00
Miklos
50ad201b11
Make conflict explanation deterministic (#6380) 2020-06-17 15:59:32 +02:00
Remy
39ceac7a2c
LF: Change engine output from GenTransaction to VersionedTransaction (#6311)
CHANGELOG_BEGIN
CHANGELOG_END
2020-06-15 17:35:15 +02:00
Remy
45746ac604
Ledger: move TransactionBuilder in LF and use it more broadly (#6322)
CHANGELOG_BEGIN
CHANGELOG_END
2020-06-15 14:13:14 +02:00
Robert Autenrieth
31ccc8b9f5
Add cache for contract arguments during execution (#6289)
* Add cache for contract arguments during execution

Fixes #5754

CHANGELOG_BEGIN
CHANGELOG_END

* Add proper ADT for cache keys

* scalafmt

* Address review comments

* Separate contract and event caches

* Address review comments
2020-06-15 10:46:18 +02:00
Remy
5051383aca
Ledger: remove optionality of submissionSeed in TransactionMeta (#6307)
CHANGELOG_BEGIN
CHANGELOG_END
2020-06-11 17:49:59 +02:00