Furthermore, uses a work stealing thread pool for work JdbcLedgerDao has
to perform outside of the database thread pool. This solves some
flakiness observed in conformance tests against PostgreSQL-backed
ledgers.
changelog_begin
changelog_end
changelog_begin
OVERRIDES CHANGELOG ENTRY FROM 2a8c93a614
[Ledger Integration Kit] Added new metrics for
``daml.index.db.*.translation`` to measure the time spent
translating to and from the serialized DAML-LF values when
fetched from the participant index.
changelog_end
* Extract caching from participant-state as a library
This will be used to keep a cache of values to cut on LF translation cost when serving transactions.
changelog_begin
changelog_end
* Add dependency where missing
* postgresql-testing: Store the JDBC URL separately.
* postgresql-testing: Expose the username and password.
* postgresql-testing: Get the caller to create the database.
And make sure it's a random one, not "test".
CHANGELOG_BEGIN
CHANGELOG_END
* postgresql-testing: Only store the JDBC URL for tests.
Less mutable state, innit.
* postgresql-testing: Capture the individual JDBC URL parameters.
* Bazel: Fix PostgreSQL binary paths.
* postgresql-testing: Just recreate the database in PostgresAroundEach.
There's no need to restart the process with a different data directory.
* Use a table specifically for transaction trees witnesses
This allows to not re-compute those every time using the flat transaction
witnesses and the complement.
Furthermore, witness tables indexing is redefined to a single primary key
covering both event identifier and witness party instead of two
separated indexes. This has proven to be a relevant improvement for
performance.
Fixes#5883
changelog_begin
[Ledger API Server] Significant improvements in serving transaction trees. See #5883.
changelog_end
* Fix migration SHA
* Simplify and clarify the public interface to Speedy.
- Remove `isFinal`. A client just uses `run()`.
- Remove `toSValue`. The value in available in `SResultFinalValue(v: SValue)`.
- A client never directly access the `.ctrl` (or `.returnValue`) components.
- A client may use `setExpressionToEvaluate(expr)` to evaluate a new expression on an existing machine.
changelog_begin
changelog_end
* remove while loop which executes just once
* avoid unnecessary mutation when running speedy
CHANGELOG_BEGIN
- [Sandbox] The ledger API server will now always use the most recent ledger configuration.
Until a ledger configuration is read from the ledger, command submissions will fail with the UNAVAILABLE error.
CHANGELOG_END
In kvutils, the first ledger configuration change needs
to have a generation one higher than the one returned
by getLedgerInitialConditions().
Remove initial config writing from sandbox as it's now written by the ledger API server
The v25 migration did not read transactions in order, causing #5659.
This PR re-arranges migrations so that ledger_entries can be re-read to fix the issue before dropping it in v30 (which used to be v29).
Fixes#5659.
changelog_begin
[Sandbox Classic] There is a chance that migrating from Sandbox 0.13.55 to Sandbox Classic 1.0.0 could have
introduced contracts falsely reported as active when in fact they are not. Migrating to Sandbox Classic
1.1.0 will fix the issue. For more details, see #5659.
changelog_end
* Integrate PostCommitValidation with JdbcLedgerDao and SqlLedger
Closes#5035Closes#5663
changelog_begin
[Sandbox] Skip unnecessary double post-commit validation inherited by sandbox-classic, expect performance improvement
changelog_end
* Ensure SqlLedger recovers from failures and logs them when publishing a transaction
* Remove unused import
* Remove tests for ledger entries
* Fix completions test to make them compile
* Fix compilation errors in tests, address self-review items, apply necessary fixes
- address https://github.com/digital-asset/daml/pull/5781#pullrequestreview-403293667
- address https://github.com/digital-asset/daml/pull/5781#pullrequestreview-403378192
* Pass TransactionTimeModelComplianceIT
* Minor tweaks to variable naming
* Fix failing tests
* Stop deduplicating commands on failures
* Attempt at making sandbox-classic allocate parties implicitly
* Remove implicit party allocation test (without full server) for SQL backed sandbox-classic
* Removing ImplicitPartyAdditionIT (covered in conformance tests)
* Add migrations
* Fix test for ledger DAO with post-commit validation against PostgreSQL
* Update PostgresIT
* Fix missing/wrong items from previous commits
* Don't perform batch processing of enqueued persistence entries
* Rebase against master
* Sandbox: expose back pressure config in CLI
CHANGELOG_BEGIN
[Sandbox]: Added ``--max-commands-in-flight`` as CLI configs. See ``daml sandbox --help``.
[Sandbox Classic]: Added ``--max-commands-in-flight`` and
``--max-parallel-submissions`` as CLI configs. See ``daml sandbox-classic --help``.
CHANGELOG_END
* Bumping the default maxParallelSubmissions to 512 for sandbox classic
We used to use `maxCommandsInFlight * 2` in SqlServer, but it makes more
sense to use `maxParallelSubmissions` there. Since the lower default value of 128
would result in the conformance tests to fail, I'm bumping it to 512
* Use maxCommandsInFlight to configure the parallel submissions for CommandService
* Add a reason text field to RejectReason.Inconsistent (#5180)
CHANGELOG_BEGIN
- Add a reason text field to RejectReason.Inconsistent.
See `#5810 <https://github.com/digital-asset/daml/issues/5810>`__.
CHANGELOG_END
* Change wording in contributing instructions to reflect best practice (#5820)
* Also make add reason text to other reject reasons that don't have it (#5820)
* Update with review comments (#5820)
* Update with review comments (#5820)
* Update with review comments (#5820)
- Only upload packages during the initial startup.
- Avoid loading packages during subsequent resets
- Share an engine between Ledger API Server and Committer
* Use a randomized H2 URL to simular in-memory
The reset service test assumes to get a completely new ledger for each
test case. But because we use H2 in-memory with db_close_delay=1 and the
same H2 database name, the second test case gets the remnants of the
first test case.
Since we know that sandbox in-memory uses an H2 in-memory URL, we can
simply use SandboxBackend.H2Database for ResetServiceInMemoryIT.
CHANGELOG_BEGIN
[Sandbox] Drastically lower the time needed to do a reset via the
ResetService.
CHANGELOG_END
* Adjust PostCommitValidation
- do not require a party for validation (to validate divulged contracts)
- stop validation after first error and don't accumulate rejection reasons
- address 01da7393b3 (r416719399)
changelog_begin
changelog_end
* Use participant state rejection reasons, drop unused submitter parameter from validation
* Implement replacement for post-commit validation
PostCommitValidation is meant as a replacement for ActiveLedgerState and
ActiveLedgerStateManager for JdbcLedgerDao so that the old table can eventually
be retired and post-commit validation can happen on the new participant schema.
This commit simply introduces the implementation and tests it in isolation.
Integration with the existing system will come in a separate contribution.
changelog_begin
changelog_end
* Address https://github.com/digital-asset/daml/pull/5737#discussion_r415918949
* Address https://github.com/digital-asset/daml/pull/5737#discussion_r415921207
* Add tests for the empty ledger scenario and introduce necessary fixes
* Add more tests for fetch and using some prior history
* Add tests for divulged contracts
* sandbox: Split up LedgerApiServer.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Extract out API service construction from LedgerApiServer.
* sandbox: Pull out a case class for GrpcServerOwner event loop params.
* sandbox: Split ownership of `ApiServices` from the object.
In #5419 we set specific loggers to WARN or ERROR so that we avoid noisy
logging by default. However, if the user specifies the --log-level CLI
param, not only the root logger, but also other loggers set in logback.xml
should log at that level.
CHANGELOG_BEGIN
[Sandbox] Properly respect the --log-level CLI parameter
CHANGELOG_END
The duplicated information was not always in sync.
Fixes#5662
CHANGELOG_BEGIN
- [Sandbox] Fix an issue where the sandbox would sometimes reject transactions
with a "INVALID_ARGUMENT: Disputed" error if either of
``min_ledger_time_rel`` or ``min_ledger_time_abs`` was set in the
submission request.
CHANGELOG_END
* Fix hashing of keys in the new contract store
Use GlobalKey.hash instead of KeyHasher.hashKey
Fixes#5680
changelog_begin
changelog_end
* Fix order of fields in migration
* Fix migration naming
* Rename file according to class
* Fix hash of active contracts, not consumed ones
* Make the contract keys test fail if the transaction is not rejected by the committer
- First, this removes the assert statement in the DAML model, so that the
transaction validation doesn't fail with a transaction abort.
- Second, the gRPC error message should contain Disputed. `InvalidLookup`
should only be used in Sandbox Classic, but due to code sharing, the
validation that triggers this error is also run for Sadbonx.
* Fix the key lookup in kvutils
The current way of transaction validation and key lookup does not work
in the following scenario:
- the transaction coming from the participant has a negative key lookup
- the contract key state is loaded, which points to a contract that has
been created by another transaction since the command interpretation
on the participant
- the contract state for the contractId inside the contract key state
has not been loaded, therefore the visibility and activeness check in
`ProcessTransactionSubmission.lookupKey` returns `None`, even though
there is in fact a contract with such a key.
To mitigate this issue, the `lookupKey` function must not check
activeness or visibility. If the submitter is not allowed to load the
contract via a key lookup, the authorization check should catch that.
Any other situation that is inconsistent should result in a rejection of
the transaction.
A small side effect of this change is that the `lookupContract` function
might try to load a contract that wasn't loaded from the kv state
earlier. This only happens because we first rebuild the entire
transaction before we compare it to the original transaction.
Previously we threw an exception in this case, but this results in a rather
ugly abort of the reinterpretation of the transaction. Therefore the
`lookupContract` function just returns `None`. Eventually the
transaction will be rejected during validation of the nodes, because
the `LookupByKey` nodes will be different.
This would not happen if we reinterpreted the transaction lazily and
compared the nodes as soon as they are produced with the corresponding
node of the original transaction.
CHANGELOG_BEGIN
[kvutils] Fixed the validation of key lookups.
See issue `#5562 <https://github.com/digital-asset/daml/issues/5562>`__.
[Participant Server, Ledger API Test Tool] Use rejection reasons of the participant-state API instead of sandbox specific ones.
CHANGELOG_END
* Validate causal monotonicity of used contract keys
When using LookupByKey, the return contract isn't necessarily fetched or used
any other way. This means that we need an extra pass over all contract keys to
ensure that causal monotonicity is respected. To be able to do this,
there is now a new field `DamlContractKeyState.active_at`, which is set to the
same value `DamlTransactionEntry.ledger_effective_time`.
An additional check in `ProcessTransactionSubmission` uses this field to verify
causal monotonicity.
This could lead to slow rollbacks that could cascade on errors that
pile up and keep precious DB threads busy for longer than needed.
changelog_begin
changelog_end
* Create single-party specialized queries to access events
Closes#5620
changelog_begin
changelog_end
* Ensure ordering of events when performing a lookup
- removes unused `is_root` column, closes#5618
- merges `template_package_id` and `template_name` in `template_id`, closes#5619
changelog_begin
[Ledger API] clean up of the participant index, improved performance when requesting transactions by template
changelog_end
* ledger-api-common: Add a metric for command submission validation.
CHANGELOG_BEGIN
- [Ledger API Server] Add a metric for command validation upon
submission, ``daml.commands.validation``.
CHANGELOG_END
* ledger-api-common: Move the submissions metric from sandbox.
* sandbox: Add a test for MetricsInterceptor for non-streaming endpoints.
* sandbox: Simplify MetricsInterceptor by overriding `ServerCall`.
* sandbox: Time all submissions, not just through the submission service.
* sandbox: Don't bother timing transaction submissions twice.
They're already timed by TimedWriteService.
* sandbox: Move command submission metrics to "daml.commands".
CHANGELOG_BEGIN
- [Ledger API Server] A number of command submission metrics have moved,
and we have added a couple.
- ``daml.commands.submissions`` is a new timer that measures all
submissions.
- ``daml.commands.valid_submissions`` is a new meter that counts
valid (unique, interpretable) submissions.
- ``daml.lapi.command_submission_service.failed_command_interpretations``
has been moved to ``daml.commands.failed_command_interpretations``.
- ``daml.lapi.command_submission_service.deduplicated_commands`` has
been moved to ``daml.commands.deduplicated_commands``.
- ``daml.lapi.command_submission_service.delayed_submissions`` has
been moved to ``daml.commands.delayed_submissions``.
- ``daml.lapi.command_submission_service.submitted_transactions`` has
been moved to ``daml.services.write.submit_transaction``.
- All ``daml.execution.*`` metrics have moved to
``daml.commands.execution``.
CHANGELOG_END
* metrics: Update the dashboard to include the new command metrics.
* Add time provider type to ledger API server
Static time sometimes needs special treatment
* Delay submissions of transactions with future time
Fixes#5480.
CHANGELOG_BEGIN
- [Sandbox] The sandbox now properly delays command submissions using
minLedgerTimeAbs or minLedgerTimeRel. See
`issue #5480 <https://github.com/digital-asset/daml/issues/5480>`_.
CHANGELOG_END
* sandbox: Construct PartyConfiguration later.
* sandbox: Provide an option to disable implicit party allocation.
CHANGELOG_BEGIN
- [Sandbox] You can now disable implicit party allocation by passing the
flag, ``--implicit-party-allocation=false``. This makes it easier to
test as you would against another ledger which does not support this
feature.
CHANGELOG_END
* ledger/metrics: Move metric helpers to their own Bazel package.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Use ledger/metrics.
* metrics: Rename `Metrics` to `Timed` and drop the `timed` prefix.
Importing methods is harder than importing objects.
* metrics: Publish to Maven Central.
