* kvutils/tools: Split the integrity checkers by package.
The two versions of the integrity checker are intertwined when they
really don't rely on each other at all. This splits them into two
separate packages to make the distinction clear.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/tools: Split the integrity checkers by directory.
This splits the integrity checkers further into separate directories, to
make it clear they do not interact at all.
* kvutils/tools: Rename "integrity-check" to "integrity-check-v1".
* ledger-on-memory: Recommend ledger exports (v3), not ledger dumps (v1).
* kvutils/tools: Split the benchmarks from integrity-check-v2.
* kvutils: Implement a basic protocol buffer-based export.
* kvutils: Add a header to the protobuf-based export.
So if we decide to go for a v4, it's easier to figure this stuff out.
* kvutils: Use our own header format for export.
The idea is to be somewhat forward-compatible, so relying on protobuf
to always be the format of the future seems a little odd. We may switch.
* kvutils: Construct the importer up-front during integrity checks.
* kvutils: Move the various importers and exporters to versioned packages.
* kvutils: Move the export version to the package object.
* kvutils: Create helpers to construct importers and exporters from paths.
* kvutils: Process either v2 or v3 exports with the integrity checker.
* kvutils: Sort the write set during export.
* kvutils: Switch to v3 of the export.
The integrity checker will work with either version.
CHANGELOG_BEGIN
- [Integration Kit] kvutils-based ledgers will now export the ledger
using a new serialization format based on Protocol Buffers, which we
are referring to as "v3". Existing "v2" exports can still be read and
verified using the integrity checker, which can read both old and new
versions.
CHANGELOG_END
* kvutils: Aggregate export data in a sorted map.
We need to sort it anyway; might as well do it on input.
* kvutils: Improve a test string.
* kvutils: Use the v3 importer in the replay test.
* kvutils: Remove v2 of the export format.
We're pretty sure no one is using it.
* ledger-on-(memory + sql): Include logback as a test dependency.
Otherwise logs don't show up when we get failures, which makes debugging
a lot harder.
* kvutils: On unit test timeout, show a useful error.
* kvutils: Increase the streaming updates timeout in integration tests.
This is timing out every now and again on CI, and I can reproduce it
easily by running the tests in parallel.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Factor out a base class for ledger exporter implementations.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Use piped streams in LedgerDataExporterSpecBase.
* kvutils: Rename "FileBased…" to "SerializationBasedLedgerDataExporter".
* kvutils: Wrap Deserialization in a class for polymorphism purposes.
* kvutils: Pull out submission info variables in the ledger export test.
* kvutils: Inline (De)Serialization into the exporter.
* kvutils: Replace usage of Deserialization with LedgerDataImporter.
* kvutils: Don't synchronize on making an InMemorySubmissionAggregator.
* kvutils: Pass a SubmissionInfo to LedgerDataExporter.
* kvutils: Reflow a long string to make it readable.
* kvutils: Factor out ledger dump timers into a class.
* kvutils: Use `Paths` in the integrity checkers.
Strings give me the heebie-jeebies.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Add `override` annotations for the exporters.
* kvutils: Test the ledger export (v2).
* kvutils: Mark integrity tests as small.
* kvutils: Add assertions to the FileBasedLedgerDataExporter.
Just confirming my understanding of how this works. Verified with the
Ledger API Test Tool and some proprietary code.
* kvutils: Move the definition of `CorrelationId` to the package file.
* kvutils: Pass a submission write set through the batch pipeline.
This allows for less mutability all over the place.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Move export finishing into its own type.
* kvutils: Move some nested types upwards.
* kvutils: Split Deserialization from Serialization.
* kvutils: Extract out common behavior in (De)Serialization.
* kvutils: Don't use a singleton for LedgerDataExporter.
Instead, construct it once.
* kvutils: Make sure we close the export file when we're done.
* kvutils: Simplify `Debug` in the same manner as `LedgerDataExport`.
* kvutils: Fix a test name.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* kvutils: Remove backticks around "export".
* kvutils: Move the test `BatchedSubmissionValidator#apply` into the test.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* [KVL-222] Add participant id to index metadata dump
changelog_begin
changelog_end
* Test SqlLedger participant id initialization
* Test JdbcIndexer participant id initialization
* Make RecoveringIndexerSpec final and remove unused trait
* kvutils: Use `eventually` in BatchingQueueSpec to fix a flaky test.
We cannot assume that the queue will shut down faster than we can check
its state.
* kvutils: Increase timeouts in BatchingQueueSpec to fix flakiness.
50 milliseconds is pretty fast when tests are running in parallel.
CHANGELOG_BEGIN
CHANGELOG_END
* Refactor SQLLedger initialization routine
Small refactoring to make initialization a bit more readable. Performed
while moving forward with the addition of the participant identifier to
the parameters table (so a few minor details have leaked into this PR).
changelog_begin
changelog_end
* Fix compilation errors
* Address https://github.com/digital-asset/daml/pull/7200#discussion_r474630880
* Fix test, lower test logging noise
* kvutils: Move execution contexts to the submission methods.
Previously, the execution contexts were provided at the constructor
level, which meant the submission validator and associated components
would use the resource acquisition execution context. This context is
intended purely for resource acquisition; if we need an execution
context for processing data, we have much more control when we
explicitly create one.
Implementors will still need to avoid passing an execution context
implicitly into the constructors of their own components, instead
favoring an explicit one that is to be used during `commit` or reading
events.
CHANGELOG_BEGIN
- [Integration Kit] kvutils now expects execution contexts to be passed
in to the various ``SubmissionValidator``, ``LedgerStateAccess``, and
``LedgerStateOperations`` methods. This is a source-breaking change.
Instead of providing an execution context implicitly to your ledger
implementation, you are encouraged to construct the relevant contexts
for different operations explicitly. Please refer to the open-source
implementations as a reference.
CHANGELOG_END
* ledger-on-memory: Don't use the acquisition context for commits.
Instead, use the materializer execution context; it may not be the best
choice but at least it's appropriate.
* kvutils: Standardize method parameter formatting.
* kvutils: Provide old state APIs as `LedgerStateAccess.v1_4`.
Right now the error is logged, but the program stalls, because the
exception is outside any `Future`, and so the cleanup never happens.
CHANGELOG_BEGIN
CHANGELOG_END
* Use case class for grouping components needed for replay for a given commit strategy.
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/participant-state/kvutils/tools/src/main/scala/com/daml/ledger/participant/state/kvutils/tools/export/CommitStrategySupport.scala
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Reformatted.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: When failing to acquire a connection, specify where.
* kvutils: Pass the execution context through the ValidatingCommitter.
The SubmissionValidator shouldn't be getting it through the constructor,
but this will do for now.
* kvutils: Remove the executionContext from BatchingLedgerStateOperations.
* ledger-on-sql: Inject the committer.
* ledger-on-sql: Construct a single-threaded executor for the committer.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Trailing commas are the best kind of commas.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* add -Ywarn-unused to all scalac options
* remove some unused arguments
* remove some unused definitions
* remove some unused variable names
* suppress some unused variable names
* changeExtension doesn't use baseName
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* work around no plugins in scenario interpreter perf tests
* remove many more unused things
* remove more unused things, restore some used things
* remove more unused things, restore a couple signature mistakes
* missed import
* unused argument
* remove more unused loggingContexts
* some unused code in triggers
* some unused code in sandbox and kvutils
* some unused code in repl-service and daml-script
* some unused code in bindings-rxjava tests
* some unused code in triggers runner
* more comments on silent usages
- suggested by @cocreature; thanks
* fix missing reference in TestCommands
* more unused in triggers
* more unused in sandbox
* more unused in daml-script
* more unused in ledger-client tests
* more unused in triggers
* more unused in kvutils
* more unused in daml-script
* more unused in sandbox
* remove unused in ledger-api-test-tool
* suppress final special case for codegen unused warnings
.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
* one more unused in daml-script
* special scaladoc rules may need silencer, too
* unused in compatibility/sandbox-migration
* more commas, a different way to `find`
- suggested by @remyhaemmerle-da; thanks
* remove unused definitions, params, args from kvutils Scala code
CHANGELOG_BEGIN
CHANGELOG_END
* label desired default for enclose compression argument, should it come into use
- suggested by @fabiotudone-da; thanks
* type-alias a couple of ProcessSubmission's args to label what they are
- suggested by @fabiotudone-da; thanks
* reformat after fixing merge
* more unused in kvutils
* define enclose's "default" compression as a constant
- suggested by @miklos-da; thanks
https://github.com/digital-asset/daml/pull/6992#discussion_r466489923
* Create scala library for integrity checking tools.
CHANGELOG_BEGIN
CHANGELOG_END
* Moved integrity checking drivers into separate package.
* First define the scala library then the rest.
* Added missing header.
* Moved all export related code to under package kvutils.tools.export.
* Added missing header.
* Make all binaries depend on the library and not need sources.
* Move in-mem writer's `ledgerStateAccess.inTransaction` down to committer
* Move `BatchedSubmissionValidator` and spec into `batch` subpackage
* Add `StateAccessingValidatingCommitter` and inherit it in batching one
* Document `StateAccessingValidatingCommitter`
* Generalize the committer for `InMemoryLedgerReaderWriter`
* `envelope` -> `submissionEnvelope` in validating committers
* Add `PreExecutingValidatingCommitter` and sub-components
* Add retry in case of conflict in `PreExecutingValidatingCommitter`
CHANGELOG_BEGIN
CHANGELOG_END
* Fix compilation error
* Hook pre-execution in `daml-on-memory-kv`
* Add fake time updates provider
* Fix `BatchedValidatingCommitterSpec`
* Don't use batched writer with pre-execution
* Fix conflict detection
* Fix out-of-time-bounds detection
* Prefix/unprefix serialized log entry IDs in pre-execution write sets
* Fix: produce an out-of-bounds rejection log entry in transaction rejected cases too
* Fix `SubmissionResult` return in case of repeated pre-exec conflict
* Fidelity level 1: sequential pre-execution
* Documentation for pre-execution support in DAML-on-Memory KV
* Add ledger-on-memory conformance test with pre-execution enabled
* Revert "Fix: produce an out-of-bounds rejection log entry in transaction rejected cases too"
This reverts commit 4df7e26b
* Fix test
* Improve naming and documentation
* Address review comments
* Fix test
* Fix wrong implementation used for `ParticipantStateIntegrationSpecBase` tests
* Address review comments
* Address review comments
* Address minor review comments
* Added ledger writer that chooses between instances based on estimate interpretation cost.
CHANGELOG_BEGIN
CHANGELOG_END
* Code tidying.
* Delegate to pre-executing writer in case thershold is set to 0.
* Added ability to change metrics.
* Added metrics.
* Code tidying.
* Update ledger/participant-state/kvutils/src/main/scala/com/daml/ledger/participant/state/kvutils/api/InterpretationCostBasedLedgerWriterChooser.scala
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* add -Xlint:doc-detached
- reverts 1feae964e3 from #6798
* attach several scaladocs where they'll actually be included
* no changelog
* attach several more scaladocs where they'll actually be included
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* Fix time model error message
CHANGELOG_BEGIN
CHANGELOG_END
* Restore ledger time based error message
* Add test for the error message
* Fix the error message
now for real?
* daml-on-sql: Pull out a new `Main` object that wraps sandbox-classic.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-on-sql: Fail if a JDBC URL is not provided or not for PostgreSQL.
* sandbox-classic: Rename the conformance test H2 database.
* daml-on-sql + sandbox-classic: Report configuration errors cleanly.
This means letting `ProgramResource` catch the errors, log, and exit.
* daml-on-sql: Change the name logged on startup.
* daml-on-sql: Change the default participant ID.
* sandbox-common: Give the ledger name its own tagged string type.
* sandbox-classic: Generate random ledger IDs using the ledger name.
* daml-on-sql: Remove the banner, replacing it with a blank line.
* daml-on-sql: Enable strong seeding by default.
And weak seeding in the conformance tests.
* sandbox-classic: Move the ledger name to a separate parameter.
It's not really configurable.
* sandbox-classic: Move LedgerName from sandbox-common.
* daml-on-sql: Remove "-participant" from the participant ID.
* daml-on-sql: Use `Name` where possible.
* daml-on-sql: Make the ledger ID mandatory.
* Revert "sandbox-classic: Move LedgerName from sandbox-common."
This reverts commit 0dad1584a7.
* daml-on-sql: Print "DAML-on-SQL" in the CLI help, not "Sandbox".
* daml-on-sql + sandbox + sandbox-classic: Split out custom CLI parsing. (#6846)
* participant-state: Simplify naming the seeding modes.
* Populate out-of-time-bounds entry when we set min/max record time.
* Populate out-of-time-bounds entry when we set max record time for ConfigCommitter.
* Code tidying.
* Do not throw in case no min/max record time has been specified and there's no out-of-time-bounds log entry.
CHANGELOG_BEGIN
CHANGELOG_END
* Code tidying.
* Add ledger-on-memory conformance test with pre-execution enabled
* Revert "Add ledger-on-memory conformance test with pre-execution enabled"
This reverts commit d2c4364a
* Apply suggestions from code review
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
CHANGELOG_BEGIN
[Engine] - Change the callback for contract key from `GlobalKey => Option[ContractId]` to `GlobalKeyWithMaintainers => Option[ContractId]`
CHANGELOG_END
* Update existing docs, removing references to old time model
* Add detailed time model docs
* Rename ledger effective time to ledger time
CHANGELOG_BEGIN
CHANGELOG_END
* Move public code into daml-integration-api
CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
Also fixes#5635, removing usage of `ReadService` from the `StandaloneApiServer`.
The configuration stream of the Ledger API LedgerConfigurationService is now properly backed by the configuration entries instead of just serving the initial configuration.
CHANGELOG_BEGIN
[DAML Integration Kit]: ``StandaloneApiServer`` can now be run in a read-only mode.
- The type of the constructor parameter ``writeService`` of ``StandaloneApiServer`` changed to ``Option[WriteService]``. Passing ``None`` will not start any of the admin services, the command service, and the command submission service.
- The constructor parameter ``readService`` of ``StandaloneApiServer`` has been removed.
- A new constructor parameter ``ledgerId`` has been added to ``StandaloneApiServer``. It is used to verify that that ``StandaloneApiServer`` is run against an index storage for the same ledgerId. Initialization is aborted if this is not the case.
[DAML Integration Kit]: The ``LedgerConfigurationService`` now properly streams configuration changes.
CHANGELOG_END
SubmittedTransaction and CommittedTransaction are moved from com.daml.lf.transaction.Transaction to
com.daml.lf.transaction
This helps intelliJ type inference.
CHANGELOG_BEGIN
CHANGELOG_END
Try to match transactions from the ledger export to a different non original dar.
This is useful to benchmark a different compilation of the same original daml source.
CHANGELOG_BEGIN
CHANGELOG_END
* Added metrics for decoding and total time of pre-execution.
CHANGELOG_BEGIN
CHANGELOG_END
* Code tidying.
* Added timer for tracking time spent with generating write sets.
