* add silent_annotations option to da scala bazel functions
* use silent_annotations for several scala targets
* use silencer_plugin instead when the lib isn't used
* use silent_annotations for several more scala targets
* use silencer_lib for strange indirect requirement for running tests
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* silent_annotations support for scaladoc
Currently the list of node that correspond to a "by key" operations
is tracked in the Transaction metadata. this PR move this information
into the nodes themself.
This is a preparatory work to include this information in the serialization
format for transactions
This advances the state of #7622
CHANGELOG_BEGIN
CHANGELOG_END
This PR fixes a bug in the PackageCommitter introduced recently.
Dependencies not decoded are assumed to be already preloaded and
should not bee looked for.
CHANGELOG_BEGIN
CHANGELOG_END
* Add ability to start only indexer or lapi server
CHANGELOG_BEGIN
CHANGELOG_END
* Change command line arguments
* Use Resource.unit
* Fix MultiParticipantFixture
* Add a new conformance test
* Improve retrying connecting to the database
* Improve naming
* Introduce shardName
and use it to create unique metric names
* Fix a merge error
* Remove unused comment
* Fix test
* Run conformance-test-split-participant in batch mode
Co-authored-by: tudor-da <tudor.voicu@digitalasset.com>
This PR creates 3 validation modes:
* `Strict`: Specifies that the committer should validate the package
before committing them to the ledger. When using this mode, the
packages committed to the ledger can be fully trusted and do not
have to be validated when loaded into the engine.
* `Lenient`: Specifies that the committer should perform a fast
validation of the packages before committing them to the ledger.
This mode is useful for ledger integrations that cannot handle
long-running submissions (> 10s). When using this mode, the
packages committed to the ledger cannot be trusted and must be
validated every time they are loaded into the engine.
* `No`: Specifies that the committer should not perform any
validation the packages before committing them to the ledger. This
should be used only by non distributed ledgers, like DAML-on-SQL,
where the validation done in the API server can be trusted.
This PR creates 3 preloading modes:
* `Synchronous` : Specifies that the packages should be preloading
into the engine before committed.
* `Asynchronous`: Specifies that the packages should be preloaded into
the engine asynchronously with the rest of the commit process. This
mode is useful for ledger integrations that cannot handle
long-running submissions (> 10s). Failure of the preloading process
will not affect the commit.
* `No`: Specifies that the packages should not be preloaded into
the engine.
CHANGELOG_BEGIN
- [Integration Kit] In kvutils, add metric
daml.kvutils.committer.package_upload.validate_timer to track
package validation time.
CHANGELOG_END
Make the hardcoded timeout for party allocaction/package upload configurable
This is a short term fix to remediate issues with uploading packages
that take a considerable amount of time to decode and validate and
therefore exhausting the 30 seconds.
Adding a maximum record parameter to the ledger API like we already have
for the config management service is not as straight forward for the
package upload, because one has to account for the time in transit as
well. This topic needs further analysis, but in the meantime making the
timeout configurable and setting the default to 2 minutes should provide
enough headroom to alleviate existing issues with package upload timing.
Contributes to #6880
CHANGELOG_BEGIN
[Integration Kit]: The hardcoded timeout for party
allocation and package uploads in the Ledger API Server can be configured via ParticipantConfig and
the default value is now set to 2 minutes. See
`issue #6880 <https://github.com/digital-asset/daml/issues/6880>`__.
CHANGELOG_END
* kvutils: Document the deterministic nature of commit strategies.
* kvutils: Run the commit strategies in as parallel a fashion as possible.
Trees are faster than lists.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Move `serializeStateKey` into its own class.
* concurrent: Tag DirectExecutionContext.
1. Tag `DirectExecutionContext` as `ExecutionContext[Nothing]`, thereby
stating that it works for any tagged `Future`.
2. Move `DirectExecutionContext` to the _libs-scala/concurrent_
library, as it requires it and it's tiny.
CHANGELOG_BEGIN
CHANGELOG_END
* concurrent: Fix the privacy of `DirectExecutionContextInternal`.
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* kvutils: Always commit serially.
Committing in parallel will cause the set of writes to be emitted in a
random order.
CHANGELOG_BEGIN
- [Integration Kit] In kvutils, the ``BatchedSubmissionValidator`` no
longer has a parameter for commit parallelism. Commits are now always
written serially to preserve order.
CHANGELOG_END
* kvutils: Sort the output state before writing.
We previously wrote the output state in a random order, leading to
writes that could not be easily compared for integrity. Sorting them
allows us to validate not just the values, but the order.
This means that the exporter no longer sorts the write set, and the
integrity checker does not sort before checking.
CHANGELOG_BEGIN
- [Integration Kit] In kvutils, state is now sorted before committing.
This allows us to provide stronger guarantees with regards to the
serialized write sets.
If you have implemented your own ``CommitStrategy``, you should also
ensure the output state is sorted before committing.
CHANGELOG_END
* ledger-on-sql: Implement SQL typeclasses as values, not functions.
Let's pretend performance is important here.
* kvutils: Allow Iterables in batching ledger state operations.
A `Map[Key, Value]` is an `Iterable[(Key, Value)]`, but not a `Seq`.
This allows us to pass a `Map` straight from the committer.
* kvutils: Construct a `SortedMap` in the commit strategy.
This should be more performant than generating and then sorting a
`Vector`.
* kvutils: Sort state updates when committing after pre-execution.
* participant-integration-api: Inject health checks into the API server.
CHANGELOG_BEGIN
- [Integration Kit] The ``StandaloneApiServer`` now takes a
``healthChecks`` parameter, which should identify the health checks to
be exposed over the gRPC Health Checking Protocol. This will
typically look something like::
healthChecks = new HealthChecks("read" -> readService, "write" -> writeService)
Integrators may also wish to expose the health of more components.
All components wishing to report their health must implement the
``ReportsHealth`` trait.
CHANGELOG_END
* sandbox + kvutils: Add the "read" component back to the health checks.
* remove failedTransactions field from ScenarioLedger.RichTransaction
changelog_begin
changelog_end
* remove Blinding.checkAuthorizationAndBlind; fixup callers to use Blinding.blind when blindningInfo is required
* rename/relocate: ScenarioLedger.CommitError.FailedAuthorizations --> SError.DamlEFailedAuthorization
* fix types to demonstate that at most one FailedAuthorization is detected/reported
* address small review comments
This PR uses the new data structure introduced in #7220.
Additionnally this fix `Value Equal instance` which was considering
<a: x, b: y> different from <b:y, a:x>.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/tools: Split the integrity checkers by package.
The two versions of the integrity checker are intertwined when they
really don't rely on each other at all. This splits them into two
separate packages to make the distinction clear.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils/tools: Split the integrity checkers by directory.
This splits the integrity checkers further into separate directories, to
make it clear they do not interact at all.
* kvutils/tools: Rename "integrity-check" to "integrity-check-v1".
* ledger-on-memory: Recommend ledger exports (v3), not ledger dumps (v1).
* kvutils/tools: Split the benchmarks from integrity-check-v2.
* kvutils: Implement a basic protocol buffer-based export.
* kvutils: Add a header to the protobuf-based export.
So if we decide to go for a v4, it's easier to figure this stuff out.
* kvutils: Use our own header format for export.
The idea is to be somewhat forward-compatible, so relying on protobuf
to always be the format of the future seems a little odd. We may switch.
* kvutils: Construct the importer up-front during integrity checks.
* kvutils: Move the various importers and exporters to versioned packages.
* kvutils: Move the export version to the package object.
* kvutils: Create helpers to construct importers and exporters from paths.
* kvutils: Process either v2 or v3 exports with the integrity checker.
* kvutils: Sort the write set during export.
* kvutils: Switch to v3 of the export.
The integrity checker will work with either version.
CHANGELOG_BEGIN
- [Integration Kit] kvutils-based ledgers will now export the ledger
using a new serialization format based on Protocol Buffers, which we
are referring to as "v3". Existing "v2" exports can still be read and
verified using the integrity checker, which can read both old and new
versions.
CHANGELOG_END
* kvutils: Aggregate export data in a sorted map.
We need to sort it anyway; might as well do it on input.
* kvutils: Improve a test string.
* kvutils: Use the v3 importer in the replay test.
* kvutils: Remove v2 of the export format.
We're pretty sure no one is using it.
* ledger-on-(memory + sql): Include logback as a test dependency.
Otherwise logs don't show up when we get failures, which makes debugging
a lot harder.
* kvutils: On unit test timeout, show a useful error.
* kvutils: Increase the streaming updates timeout in integration tests.
This is timing out every now and again on CI, and I can reproduce it
easily by running the tests in parallel.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Factor out a base class for ledger exporter implementations.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Use piped streams in LedgerDataExporterSpecBase.
* kvutils: Rename "FileBased…" to "SerializationBasedLedgerDataExporter".
* kvutils: Wrap Deserialization in a class for polymorphism purposes.
* kvutils: Pull out submission info variables in the ledger export test.
* kvutils: Inline (De)Serialization into the exporter.
* kvutils: Replace usage of Deserialization with LedgerDataImporter.
* kvutils: Don't synchronize on making an InMemorySubmissionAggregator.
* kvutils: Pass a SubmissionInfo to LedgerDataExporter.
* kvutils: Reflow a long string to make it readable.
* kvutils: Factor out ledger dump timers into a class.
* kvutils: Use `Paths` in the integrity checkers.
Strings give me the heebie-jeebies.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Add `override` annotations for the exporters.
* kvutils: Test the ledger export (v2).
* kvutils: Mark integrity tests as small.
* kvutils: Add assertions to the FileBasedLedgerDataExporter.
Just confirming my understanding of how this works. Verified with the
Ledger API Test Tool and some proprietary code.
* kvutils: Move the definition of `CorrelationId` to the package file.
* kvutils: Pass a submission write set through the batch pipeline.
This allows for less mutability all over the place.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Move export finishing into its own type.
* kvutils: Move some nested types upwards.
* kvutils: Split Deserialization from Serialization.
* kvutils: Extract out common behavior in (De)Serialization.
* kvutils: Don't use a singleton for LedgerDataExporter.
Instead, construct it once.
* kvutils: Make sure we close the export file when we're done.
* kvutils: Simplify `Debug` in the same manner as `LedgerDataExport`.
* kvutils: Fix a test name.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* kvutils: Remove backticks around "export".
* kvutils: Move the test `BatchedSubmissionValidator#apply` into the test.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* [KVL-222] Add participant id to index metadata dump
changelog_begin
changelog_end
* Test SqlLedger participant id initialization
* Test JdbcIndexer participant id initialization
* Make RecoveringIndexerSpec final and remove unused trait
* kvutils: Use `eventually` in BatchingQueueSpec to fix a flaky test.
We cannot assume that the queue will shut down faster than we can check
its state.
* kvutils: Increase timeouts in BatchingQueueSpec to fix flakiness.
50 milliseconds is pretty fast when tests are running in parallel.
CHANGELOG_BEGIN
CHANGELOG_END
* Refactor SQLLedger initialization routine
Small refactoring to make initialization a bit more readable. Performed
while moving forward with the addition of the participant identifier to
the parameters table (so a few minor details have leaked into this PR).
changelog_begin
changelog_end
* Fix compilation errors
* Address https://github.com/digital-asset/daml/pull/7200#discussion_r474630880
* Fix test, lower test logging noise
* kvutils: Move execution contexts to the submission methods.
Previously, the execution contexts were provided at the constructor
level, which meant the submission validator and associated components
would use the resource acquisition execution context. This context is
intended purely for resource acquisition; if we need an execution
context for processing data, we have much more control when we
explicitly create one.
Implementors will still need to avoid passing an execution context
implicitly into the constructors of their own components, instead
favoring an explicit one that is to be used during `commit` or reading
events.
CHANGELOG_BEGIN
- [Integration Kit] kvutils now expects execution contexts to be passed
in to the various ``SubmissionValidator``, ``LedgerStateAccess``, and
``LedgerStateOperations`` methods. This is a source-breaking change.
Instead of providing an execution context implicitly to your ledger
implementation, you are encouraged to construct the relevant contexts
for different operations explicitly. Please refer to the open-source
implementations as a reference.
CHANGELOG_END
* ledger-on-memory: Don't use the acquisition context for commits.
Instead, use the materializer execution context; it may not be the best
choice but at least it's appropriate.
* kvutils: Standardize method parameter formatting.
* kvutils: Provide old state APIs as `LedgerStateAccess.v1_4`.
Right now the error is logged, but the program stalls, because the
exception is outside any `Future`, and so the cleanup never happens.
CHANGELOG_BEGIN
CHANGELOG_END
* Use case class for grouping components needed for replay for a given commit strategy.
CHANGELOG_BEGIN
CHANGELOG_END
* Update ledger/participant-state/kvutils/tools/src/main/scala/com/daml/ledger/participant/state/kvutils/tools/export/CommitStrategySupport.scala
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Reformatted.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: When failing to acquire a connection, specify where.
* kvutils: Pass the execution context through the ValidatingCommitter.
The SubmissionValidator shouldn't be getting it through the constructor,
but this will do for now.
* kvutils: Remove the executionContext from BatchingLedgerStateOperations.
* ledger-on-sql: Inject the committer.
* ledger-on-sql: Construct a single-threaded executor for the committer.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Trailing commas are the best kind of commas.
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
Co-authored-by: fabiotudone-da <fabio.tudone@digitalasset.com>
* add -Ywarn-unused to all scalac options
* remove some unused arguments
* remove some unused definitions
* remove some unused variable names
* suppress some unused variable names
* changeExtension doesn't use baseName
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* work around no plugins in scenario interpreter perf tests
* remove many more unused things
* remove more unused things, restore some used things
* remove more unused things, restore a couple signature mistakes
* missed import
* unused argument
* remove more unused loggingContexts
* some unused code in triggers
* some unused code in sandbox and kvutils
* some unused code in repl-service and daml-script
* some unused code in bindings-rxjava tests
* some unused code in triggers runner
* more comments on silent usages
- suggested by @cocreature; thanks
* fix missing reference in TestCommands
* more unused in triggers
* more unused in sandbox
* more unused in daml-script
* more unused in ledger-client tests
* more unused in triggers
* more unused in kvutils
* more unused in daml-script
* more unused in sandbox
* remove unused in ledger-api-test-tool
* suppress final special case for codegen unused warnings
.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
* one more unused in daml-script
* special scaladoc rules may need silencer, too
* unused in compatibility/sandbox-migration
* more commas, a different way to `find`
- suggested by @remyhaemmerle-da; thanks
* remove unused definitions, params, args from kvutils Scala code
CHANGELOG_BEGIN
CHANGELOG_END
* label desired default for enclose compression argument, should it come into use
- suggested by @fabiotudone-da; thanks
* type-alias a couple of ProcessSubmission's args to label what they are
- suggested by @fabiotudone-da; thanks
* reformat after fixing merge
* more unused in kvutils
* define enclose's "default" compression as a constant
- suggested by @miklos-da; thanks
https://github.com/digital-asset/daml/pull/6992#discussion_r466489923
