by adding synchronize call between an alpha-participant-create and the
corresponding beta-participant-exercise.
changelog_begin
changelog_end
We occasionally run into this in Canton
* Port PerformanceEnvelope.TransactionSize perf test from Canton
to LedgerApiTestTool --perf-tests
changelog_begin
[Ledger Api Test Tool] Added the TransactionSize performance benchmark test.
changelog_end
* Review feedback from Samir
* Formatting
* Make the contract keys test fail if the transaction is not rejected by the committer
- First, this removes the assert statement in the DAML model, so that the
transaction validation doesn't fail with a transaction abort.
- Second, the gRPC error message should contain Disputed. `InvalidLookup`
should only be used in Sandbox Classic, but due to code sharing, the
validation that triggers this error is also run for Sadbonx.
* Fix the key lookup in kvutils
The current way of transaction validation and key lookup does not work
in the following scenario:
- the transaction coming from the participant has a negative key lookup
- the contract key state is loaded, which points to a contract that has
been created by another transaction since the command interpretation
on the participant
- the contract state for the contractId inside the contract key state
has not been loaded, therefore the visibility and activeness check in
`ProcessTransactionSubmission.lookupKey` returns `None`, even though
there is in fact a contract with such a key.
To mitigate this issue, the `lookupKey` function must not check
activeness or visibility. If the submitter is not allowed to load the
contract via a key lookup, the authorization check should catch that.
Any other situation that is inconsistent should result in a rejection of
the transaction.
A small side effect of this change is that the `lookupContract` function
might try to load a contract that wasn't loaded from the kv state
earlier. This only happens because we first rebuild the entire
transaction before we compare it to the original transaction.
Previously we threw an exception in this case, but this results in a rather
ugly abort of the reinterpretation of the transaction. Therefore the
`lookupContract` function just returns `None`. Eventually the
transaction will be rejected during validation of the nodes, because
the `LookupByKey` nodes will be different.
This would not happen if we reinterpreted the transaction lazily and
compared the nodes as soon as they are produced with the corresponding
node of the original transaction.
CHANGELOG_BEGIN
[kvutils] Fixed the validation of key lookups.
See issue `#5562 <https://github.com/digital-asset/daml/issues/5562>`__.
[Participant Server, Ledger API Test Tool] Use rejection reasons of the participant-state API instead of sandbox specific ones.
CHANGELOG_END
* Validate causal monotonicity of used contract keys
When using LookupByKey, the return contract isn't necessarily fetched or used
any other way. This means that we need an extra pass over all contract keys to
ensure that causal monotonicity is respected. To be able to do this,
there is now a new field `DamlContractKeyState.active_at`, which is set to the
same value `DamlTransactionEntry.ledger_effective_time`.
An additional check in `ProcessTransactionSubmission` uses this field to verify
causal monotonicity.
* Produce performance tests for all envelopes
CHANGELOG_BEGIN
[TestTool] Provide performance tests for all performance envelopes
CHANGELOG_END
* Follow Scala's recommended pattern for enum-like hierarchies
* Add failing test
The test can produce false negatives,
but locally it fails 10 out of 10 times.
* Stop deduplicating commands after rejections
Fixes#5338.
CHANGELOG_BEGIN
CHANGELOG_END
* Only deduplicate successful transactions
* Use pass instead of pure unit
* Handle exceptions
* Draft of PingPong throughput and latency benchmarks for on-mem and on-sql ledgers
* Augment `ParticipantTestContext` and remove `LedgerApiServer` hack
* Separate performance tests into distinct category with concurrency 1
* 🎨
* Package performance tests in separate DAR
* Have performance tests excluded by default and run exclusively if passed
* Fix rebase
* Simplify `BenchmarkReporter`
* Make `concurrencyOverride` into an `Option`
* Clarify command line usage, prevent regular and perf. tests together
* Fix preventing regular and perf. tests together
* Split `PingPong`, `PingPongExplode` and `Cycle` benchmarks' model
CHANGELOG_BEGIN
- [TestTool] Add `PingPong` performance envelope test
CHANGELOG_END
* Explicitly name `concurrencyOverride`
* Fix formatting
* Lower bar for CI run of performance envelope tests
* Make benchmark output file configurable
* Improve messages and report config option name
* Use exit status 64 for "bad command line usage" as in BSD
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* Implement timed command deduplication in kvutils
This adds a field deduplication_time to DamlCommandDedupValue for
deduplication timeout checking.
* Bump kvutils version to 4
* Fix CommandTracker pulling commandResultIn multiple times
Now that the timeouts are generated out of band, we have 2
"unsynchronized" places that pull on commandResultIn.
Whenever we pull, we need to check that commandResultIn hasn't been
pulled before.
* Add inStaticTimeMode flag to enable command dedup in sandbox-next with static-time
Fixes#4624.
CHANGELOG_BEGIN
[kvutils] KVUtils now respects the command deduplciation time instead of
deduplicating commands forever.
CHANGELOG_END
* Use com.daml as groupId for all artifacts
CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END
* Add 2 additional maven related checks to the release binary
1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId
* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
* Remove old time model from ledger config
CHANGELOG_BEGIN
- [Ledger API] Fields related to the old ledger time
model have been removed from the configuration
management service and the ledger configuration service.
CHANGELOG_END
* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/tests/LedgerConfigurationService.scala
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Remove documentation for a removed option
CHANGELOG_BEGIN
- [DAML Integration Kit] The CLI option command-submission-ttl-scale-factor was
removed, as the LET/MRT/TTL fields have recently been removed
from the command submission service.
CHANGELOG_END
* Minor renaming
Contributes to #4194.
Closes#4231.
Closes#5022.
CHANGELOG_BEGIN
- [Ledger API] The protobuf fields ledger_effective_time and maximum_record_time have been removed from
command submission. These fields were previously deprecated following the introduction
of a new ledger time model. See issue `#4194 <https://github.com/digital-asset/daml/issues/4194>`__.
[Java Bindings] removed the usage of ledgerEffectiveTime and
maximumRecordTime, and instead added minLedgerTimeAbsolute and
minLedgerTimeRelative in CommandSubmissionClient and CommandClient
CHANGELOG_END
* Tighten result type
Command execution can't result in a sequencer error
* New helper method for extracting used contracts
* New error clause
* Add a DAO query for the maximum time of contracts
* Implement algorithm for finding ledger time
CHANGELOG_BEGIN
CHANGELOG_END
* fixup ledgerTimeHelper
* Use new ledger time algorithm
* Mark LET/MRT as deprecated
CHANGELOG_BEGIN
- [Ledger API] DAML ledgers have switched to a new ledger time model.
The ledger_effective_time and maximum_record_time fields of command submission are deprecated,
the ledger time of transactions is instead set automatically by the ledger API server.
Ledger time is no longer strictly monotonically increasing, but only follows causal monotonicity:
ledger time of transactions is greater than or equal to the ledger time of any used contract.
See `#4345 <https://github.com/digital-asset/daml/issues/4345>`__.
CHANGELOG_END
* Add ledger time skew check
* Remove command updater
LET/MRT are now deprecated, this class is now useless
* Remove old time model validator
* Switch to new time model check: kvutils
* Switch to new time model check: in-memory ledger
* Switch to new time model check: SqlLedger
* Use initial ledger config
* Ignore user provided LET
* Use TimeProvider in submission services
* Use deduplication_time in daml-script runner
- Also remove unnecessary command completion output of CommandTracker.
- Remove usage of maximum record time in CommandTracker.
* Use arbitrary default value for deduplication time
* Use built-in Instant ordering
* Remove obsolete test
* Remove obsolete test: CommandStaticTimeIT
* Refactor test: TransactionMRTCompliance
* Disable test: CommandTrackerFlow timeout
* thread maxDeduplicationTime through to CommandTracker
* Improve test
* Refactor command client configuration
* Deduplication time should always use UTC
* Add missing method in TimedIndexService after rebase
* Put more details into the deduplication error response.
* Use system time for command dedup submittedAt.
* Use explicit UTC time source in command validator
* Revert CommandTracker[Flow] to previous completion-recovering-behavior
* Adapt scala client command config to new config params
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Sandbox: Reveal contract id seeding flag
CHANGELOG_BEGIN
- [Sandbox] Add support for random contract identifiers. See section
`Contract Identifiers Generation` section in
docs/source/tools/sandbox.rst
CHANGELOG_END
* Integrate transaction lookup on new schema in Ledger API
Re-wires all transaction lookups to the new schema
CHANGELOG_BEGIN
CHANGELOG_END
* Always return the agreement text
CHANGELOG_BEGIN
[Ledger API Server] The metric 'daml.index.lookup_transaction' has been
replaced by 'daml.index.lookup_flat_transaction_by_id' and
'daml.index.lookup_transaction_tree_by_id', which record the same events
but with more granularity regarding the type of lookup.
CHANGELOG_END
* Ensure agreement text invariant in a single place
* Do not compare the order in which witness parties appear in an event
* Hide command identifier from non-submitters in transaction trees
* Fix time assigned to transaction to be the ledger effective time and not the record time
* Store transactions from initial state into the new schema
Avoiding `damlc compile/package` commands (which we would like to deprecate), and replace with plain `damlc build` together with a post dar->dalf extraction step in the couple of places where we actually want the .dalf for testing.
changelog_begin
changelog_end
This would fail only on PostgreSQL because `IN ()` is invalid. H2 seems
to be fine with it.
CHANGELOG_BEGIN
- [Ledger API Server] Support a call to `GetParties` with an empty list
of parties.
CHANGELOG_END
* Share test certificates
This is primarily an attempt at making sure my contribution stats
remain negative but I think it’s a nice cleanup. The only difference
in the certs used by daml-helper which are now used everywhere is that
they use a different CN for the CA and the server. This is required to
make openssl happy (which is used by the daml-helper).
changelog_begin
changelog_end
* Fix script and trigger tests
Currently sandbox only supports TLS if you also enable client
authentication. There is no reason for why this has to be the case and
for things like DABL we want TLS without client authentication so it’s
useful to be able to test this in sandbox. This PR introduces a
`--client-auth` flag that allows you to configure the behavior. The
default is the current one of requiring client authentication.
This PR does not yet update Java clients, however, the Haskell client
supports this already and is used to test this functionality.
I’ve also added a section in the documentation on TLS (there were no
docs at all so far).
changelog_begin
- [DAML Sandbox] When Sandbox is run with TLS enabled, you can now
configure the requirement for client authentication via
``--client-auth``. See
https://docs.daml.com/tools/sandbox.html#running-with-tls for more information.
changelog_end
* ledger-api-test-tool: Fix warnings flagged by IntelliJ IDEA.
* ledger-api-test-tool: Open-world mode.
In open-world mode, parties aren't allocated; their names are just
reserved for the test case, so that no other test will accidentally use
the same party name.
This is so we can test ledgers which dynamically allocate parties, such
as Sandbox.
* sandbox: Run conformance tests in "open-world" mode.
This means that the tests don't explicitly allocate parties (except for
a few), instead relying on Sandbox's implicit party allocation feature.
This is not enabled for Sandbox Next yet.
* sandbox-next: Implicit party allocation.
This is added to the command submission service.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-next: Don't implicitly allocate pre-existing parties.
* ledger-api-test-tool: Move pre-allocation into ParticipantTestContext.
* ledger-api-test-tool: We can reserve parties or wait for them. Not both.
Make illegal states unrepresentable as early as possible.
* sandbox: Name ApiSubmissionService's private methods a little better.
* sandbox: Move ApiSubmissionService's conditional logic into methods.
* sandbox: Document why we set `implicitPartyAllocation` to `false`.
* sandbox: Document why `implicitPartyAllocation` is dangerous.
* sandbox: Add a database test for storing and retrieving parties.
* sandbox: Add database queries for selecting one or many parties.
* ledger-api-test-tool: Add a test for `ListKnownParties`.
* sandbox: Add an endpoint to retrieve a single party's details.
CHANGELOG_BEGIN
- [Ledger API] Added an endpoint to retrieve a single party's details at
``com.digitalasset.ledger.api.v1.admin.PartyManagementService.GetParty``.
Please consult the ledger API reference documentation for more
information.
CHANGELOG_END
* sandbox: Add an endpoint to retrieve a multiple parties' details.
CHANGELOG_BEGIN
- [Ledger API] Added an endpoint to retrieve multiple parties's details at
``com.digitalasset.ledger.api.v1.admin.PartyManagementService.GetParties``.
Please consult the ledger API reference documentation for more
information.
CHANGELOG_END
* sandbox: Getting a single party is a special case of multiple parties.
So let's use that code path and stop duplicating work.
* sandbox: Remove `GetParty`, as it's subsumed by `GetParties`.
"Subsumed" is a great word.
Events in transaction trees should only reference other events
that:
1) are either create or exercise events
2) the requesting parties are a witness of
This applies to recalculated root nodes as well as
the child event ids referenced in exercise nodes.
CHANGELOG_BEGIN
[Sandbox]: fixed projection of transaction trees.
CHANGELOG_END
* Always return error on duplicate submissions
* Remove unnecessary submission information
Now that duplicate submissions always return an error,
we don't need to store the original submission result.
CHANGELOG_BEGIN
CHANGELOG_END
* Rename ttl to deduplicationTime/deduplicateUntil
* Store absolute deduplicateUntil in domain commands
* Fix my own initials
* Remove CommandDeduplicationEntry
Instead, use CommandDeduplicationResult everywhere,
removing the extra layer.
The change to `EventFilter` and to the query in `JdbcLedgerDao` are
"duplicate work", but we need the change in EventFilter for the
InMemoryLedger, and the change in JdbcLedgerDao so that we avoid
fetching a contract that anyway would be discarded later.
CHANGELOG_BEGIN
[Sandbox]: Witnessed contracts for which a party is not a stakeholder
are no longer returned in the active contract stream.
CHANGELOG_END
Fixes#3254.
* Make kvutils work with the new contract id scheme
CHANGELOG_BEGIN
- [KVUtils] uses random contract id. Contract ids are made of 65 hexa decimal characters.
CHANGELOG_END
Co-authored-by: Jussi Mäki <jussi.maki@digitalasset.com>
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.
InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.
* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.
* Run all conformance tests in the repository in verbose mode.
This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.
This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:
- ledger-api-test-tool-on-canton
- ledger-on-memory
- ledger-on-sql
- sandbox
Fixes#4225.
CHANGELOG_BEGIN
CHANGELOG_END
* Add TTL field to protobuf
* Add command deduplication to index service
* Wire command deduplication to DAO
* Implement in-memory command deduplication
* Remove Deduplicator
* Implement JDBC command deduplication
* Add TTL field to domain commands
* Deduplicate commands in the submission service
CHANGELOG_BEGIN
- [Sandbox] Implement a new command submission deduplication mechanism
based on a time-to-live (TTL) for commands.
See https://github.com/digital-asset/daml/issues/4193
CHANGELOG_END
* Remove unused command service parameter
* fixup protobuf
* Add configuration for TTL
* Fix Haskell bindings
* Rename SQL table
* Add command deduplication test
* Redesign command deduplication queries
* Address review comment
* Address review comment
* Address review comments
* Make command deduplication test optional
* Disable more tests
* Address review comments
* Address review comments
* Refine test
* Address review comments
* scalafmt
* Truncate new table on reset
* Store original command result
* Rename table columns
... to be consistent with other upcoming tables
* Rename migrations to solve conflicts
Fixes#4193.
* Split Ledger API Test Tool output
Makes failure pop up even without text coloring (e.g. on Azure Pipelines)
CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Ledger API Test Tool now prints errors as a separate section
CHANGELOG_END
* Successes on the right, failures on the left :)
* Add missing newline
* Add more tests to default run of Ledger API Test Tool
Furthermore, drop TimeIT, which is a sandbox-only property (and tested already as part of its integration tests)
CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Ledger API Test Tool default tests modified. Use --list for the updated list of default tests. Time service test dropped from the suite.
CHANGELOG_END
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380635979
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380636989
* Optimize imports
* Only run semantic tests on Canton
* Shuffle participants used in tests
* Add no-wait-for-parties flag to the test tool
This flag is needed for privacy-preserving ledgers that do not expose
party allocations to non-submitting participants.
* Add shuffle-participants flag to test-tool
CHANGELOG_BEGIN
CHANGELOG_END
* Address code review. Make --no-wait-for-parties hidden.
Unfortunately, downgrading `scalafmt` resulted in files that would be
reformatted to invalid code, because lines would be merged but trailing
commas would be left in.
I've manually run it everywhere and fixed these issues so people don't
have to fix them as they go.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* ledger-on-sql: Pull out a superclass for the various integration tests.
* ledger-on-sql: Use Flyway to migrate tables in an idempotent manner.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Make it easy to run it with Bazel for experimentation.
* ledger-on-sql: Test that migrations will never change in the future.
* ledger-on-sql: Add a prefix of "ledger_" to the tables.
This is so we don't accidentally conflict with the index when the
schemas are shared.
I am letting myself modify the migrations because the existing
migrations haven't been merged into `master` yet.
* ledger-on-sql: Explain why we change the pool size after migration.
* ledger-api-test-tool: Tests now specify a timeout scale, not a timeout.
Makes it easier to change the default timeout in one place.
* ledger-api-test-tool: Increase timeouts on slow tests.
These tests produce a lot of volume and can make CI flaky.
* ledger-on-sql: Only tear down PostgreSQL in tests.
For other databases, we just create a new file for each test case.
* ledger-on-sql: Reduce the log output in tests.
* ledger-on-sql: Use a separate connection pool for Flyway when possible.
Apparently `setMaximumPoolSize` doesn't really have the desired effect
after the connection pool has already been used. The new test case will
be flaky if we process more than one commit in parallel.
For SQLite, it seems to be OK.
* ledger-on-sql: Use a separate connection pool for Flyway with SQLite.
Except in memory.
* ledger-on-sql: Use a separate PostgreSQL database for each test.
Because performance, innit. Don't have to tear them down.
* ledger-on-sql: Move the `Queries` values inside the `RDBMS` objects.
* ledger-on-sql: Go into even more detail about pool size hijinks.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
It affects basically every file, so let's do it all at once.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Factor out common values
* Validate LookupByKey nodes
Fixes#3543
CHANGELOG_BEGIN
- [Sandbox] Validate LookupByKey transaction nodes.
See `issue #3543 <https://github.com/digital-asset/daml/issues/3543>`_.
CHANGELOG_END
* Add failing tests
* LookupByKey validation should not respect privacy
* lookupByKey should use stakeholders, not divulgees
Fixes#2311
CHANGELOG_BEGIN
- [Ledger API] lookupByKey now requires the submitter to be a stakeholder on the referenced contract.
See `issue #2311 <https://github.com/digital-asset/daml/issues/2311>`_.
CHANGELOG_END
* Adapt tests to new behavior
