* postgresql-testing: Store the JDBC URL separately.
* postgresql-testing: Expose the username and password.
* postgresql-testing: Get the caller to create the database.
And make sure it's a random one, not "test".
CHANGELOG_BEGIN
CHANGELOG_END
* postgresql-testing: Only store the JDBC URL for tests.
Less mutable state, innit.
* postgresql-testing: Capture the individual JDBC URL parameters.
* Bazel: Fix PostgreSQL binary paths.
* postgresql-testing: Just recreate the database in PostgresAroundEach.
There's no need to restart the process with a different data directory.
* Use a table specifically for transaction trees witnesses
This allows to not re-compute those every time using the flat transaction
witnesses and the complement.
Furthermore, witness tables indexing is redefined to a single primary key
covering both event identifier and witness party instead of two
separated indexes. This has proven to be a relevant improvement for
performance.
Fixes#5883
changelog_begin
[Ledger API Server] Significant improvements in serving transaction trees. See #5883.
changelog_end
* Fix migration SHA
* ledger-on-sql: Use dedicated database execution contexts.
This may or may not stop us from monopolizing API server threads.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Name arguments to `Database` and `UninitializedDatabase`.
* Simplify and clarify the public interface to Speedy.
- Remove `isFinal`. A client just uses `run()`.
- Remove `toSValue`. The value in available in `SResultFinalValue(v: SValue)`.
- A client never directly access the `.ctrl` (or `.returnValue`) components.
- A client may use `setExpressionToEvaluate(expr)` to evaluate a new expression on an existing machine.
changelog_begin
changelog_end
* remove while loop which executes just once
* avoid unnecessary mutation when running speedy
CHANGELOG_BEGIN
- [Sandbox] The ledger API server will now always use the most recent ledger configuration.
Until a ledger configuration is read from the ledger, command submissions will fail with the UNAVAILABLE error.
CHANGELOG_END
In kvutils, the first ledger configuration change needs
to have a generation one higher than the one returned
by getLedgerInitialConditions().
Remove initial config writing from sandbox as it's now written by the ledger API server
The v25 migration did not read transactions in order, causing #5659.
This PR re-arranges migrations so that ledger_entries can be re-read to fix the issue before dropping it in v30 (which used to be v29).
Fixes#5659.
changelog_begin
[Sandbox Classic] There is a chance that migrating from Sandbox 0.13.55 to Sandbox Classic 1.0.0 could have
introduced contracts falsely reported as active when in fact they are not. Migrating to Sandbox Classic
1.1.0 will fix the issue. For more details, see #5659.
changelog_end
* add GenMap to the "all types" test generators
* report bad GenMap format with DeserializationError, not MatchError
* document GenMap JSON
* notes on missing features
* enable -Xsource:2.13 in transaction
* make an Order instance for Value resolvable, but unimplemented
* use the skeleton from SValue ordering to make a Value ordering skeleton
* add Party Order
* add Order instance for SortedLookupList
* add Order for FrontStack, deriving everything
* factor the Order lookup, and tie a knot in the recursive Value instances
* we're going to need this Iterator thing again
* replacing Order#contramap with version that supports equalIsNatural
* use new equalBy, orderBy for FrontStack, SortedLookupList, ImmArray
* _2 comparator, upgrade Name Equal to an Order
* incorporate lookup for enums, variants into Value order; record/struct cases
* Enum/Variant comparison
* looking up the singleton implicitly won't work for non-`object`s, alas
* test Order laws for values of all primitive types
* test Order laws for record and variant types
* test Order laws for enum types
* test that enum strings are not compared
* use checkLaws for Value Equal as well
* test that enums match order to constructor rank
* factor genAddend and genAddendNoListMap
* reintroduce Order for TypedValueGenerators
* more addend order
* record, variant order cases
* record cons order
* deriving Order while decoding from JSON
* make ApiCodecCompressed's Cid codec based on the typeclass
* test how the Value ordering and the underlying projected value orderings line up
- hint: they don't, yet
- this is also a template for how we'll check the fidelity with SValue
ordering
* test how the Value ordering and SValue ordering line up
- hint: they don't, yet
* typed Arbitrarys have access to Order
* produce proper ValueGenMap
* inj requires Order, sometimes
- we encode this as "all the time" but there is a type-level unification
approach to remove this requirement in some cases
* make inj a function
* test that order doesn't matter for JSON decoder
* use Utf8 order for TVG text; don't pretend that base equal works
* sort JSON GenMaps, and check for duplicates
* make injarb use IntroCtx
* remove stray import
* Order instances for Bytes, Hash, AbsoluteContractId
* require Order[Cid] to decode JSON to LF values
* clean up map reordering test
* remove unused Instant instance
* fake Order instance no longer needed, valid instance defined
* test parity of global AbsoluteContractId order and SContractId order
* bazel fmt
* test AbsoluteContractId Order lawfulness
* test duplicate key detection
CHANGELOG_BEGIN
- [JSON API] Prepare full support for the planned GenMap primitive type.
See `issue #5031 <https://github.com/digital-asset/daml/issues/5031>`_.
CHANGELOG_END
* Upgrade scala compiler silencer to 1.6.0
CHANGELOG_BEGIN
CHANGELOG_END
* Adapt build bazel file to new targets
* Switch to silencer plugin scala 2.12.11 per Samir's feedback
rather than 2.12.8
* Add missed bazel files
* Review feedback from Leo
* Integrate PostCommitValidation with JdbcLedgerDao and SqlLedger
Closes#5035Closes#5663
changelog_begin
[Sandbox] Skip unnecessary double post-commit validation inherited by sandbox-classic, expect performance improvement
changelog_end
* Ensure SqlLedger recovers from failures and logs them when publishing a transaction
* Remove unused import
* Remove tests for ledger entries
* Fix completions test to make them compile
* Fix compilation errors in tests, address self-review items, apply necessary fixes
- address https://github.com/digital-asset/daml/pull/5781#pullrequestreview-403293667
- address https://github.com/digital-asset/daml/pull/5781#pullrequestreview-403378192
* Pass TransactionTimeModelComplianceIT
* Minor tweaks to variable naming
* Fix failing tests
* Stop deduplicating commands on failures
* Attempt at making sandbox-classic allocate parties implicitly
* Remove implicit party allocation test (without full server) for SQL backed sandbox-classic
* Removing ImplicitPartyAdditionIT (covered in conformance tests)
* Add migrations
* Fix test for ledger DAO with post-commit validation against PostgreSQL
* Update PostgresIT
* Fix missing/wrong items from previous commits
* Don't perform batch processing of enqueued persistence entries
* Rebase against master
* Sandbox: expose back pressure config in CLI
CHANGELOG_BEGIN
[Sandbox]: Added ``--max-commands-in-flight`` as CLI configs. See ``daml sandbox --help``.
[Sandbox Classic]: Added ``--max-commands-in-flight`` and
``--max-parallel-submissions`` as CLI configs. See ``daml sandbox-classic --help``.
CHANGELOG_END
* Bumping the default maxParallelSubmissions to 512 for sandbox classic
We used to use `maxCommandsInFlight * 2` in SqlServer, but it makes more
sense to use `maxParallelSubmissions` there. Since the lower default value of 128
would result in the conformance tests to fail, I'm bumping it to 512
* Use maxCommandsInFlight to configure the parallel submissions for CommandService
* Add a reason text field to RejectReason.Inconsistent (#5180)
CHANGELOG_BEGIN
- Add a reason text field to RejectReason.Inconsistent.
See `#5810 <https://github.com/digital-asset/daml/issues/5810>`__.
CHANGELOG_END
* Change wording in contributing instructions to reflect best practice (#5820)
* Also make add reason text to other reject reasons that don't have it (#5820)
* Update with review comments (#5820)
* Update with review comments (#5820)
* Update with review comments (#5820)
- Only upload packages during the initial startup.
- Avoid loading packages during subsequent resets
- Share an engine between Ledger API Server and Committer
* Use a randomized H2 URL to simular in-memory
The reset service test assumes to get a completely new ledger for each
test case. But because we use H2 in-memory with db_close_delay=1 and the
same H2 database name, the second test case gets the remnants of the
first test case.
Since we know that sandbox in-memory uses an H2 in-memory URL, we can
simply use SandboxBackend.H2Database for ResetServiceInMemoryIT.
CHANGELOG_BEGIN
[Sandbox] Drastically lower the time needed to do a reset via the
ResetService.
CHANGELOG_END
* Adjust PostCommitValidation
- do not require a party for validation (to validate divulged contracts)
- stop validation after first error and don't accumulate rejection reasons
- address 01da7393b3 (r416719399)
changelog_begin
changelog_end
* Use participant state rejection reasons, drop unused submitter parameter from validation
* Implement replacement for post-commit validation
PostCommitValidation is meant as a replacement for ActiveLedgerState and
ActiveLedgerStateManager for JdbcLedgerDao so that the old table can eventually
be retired and post-commit validation can happen on the new participant schema.
This commit simply introduces the implementation and tests it in isolation.
Integration with the existing system will come in a separate contribution.
changelog_begin
changelog_end
* Address https://github.com/digital-asset/daml/pull/5737#discussion_r415918949
* Address https://github.com/digital-asset/daml/pull/5737#discussion_r415921207
* Add tests for the empty ledger scenario and introduce necessary fixes
* Add more tests for fetch and using some prior history
* Add tests for divulged contracts
* sandbox: Split up LedgerApiServer.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Extract out API service construction from LedgerApiServer.
* sandbox: Pull out a case class for GrpcServerOwner event loop params.
* sandbox: Split ownership of `ApiServices` from the object.
In #5419 we set specific loggers to WARN or ERROR so that we avoid noisy
logging by default. However, if the user specifies the --log-level CLI
param, not only the root logger, but also other loggers set in logback.xml
should log at that level.
CHANGELOG_BEGIN
[Sandbox] Properly respect the --log-level CLI parameter
CHANGELOG_END
The duplicated information was not always in sync.
Fixes#5662
CHANGELOG_BEGIN
- [Sandbox] Fix an issue where the sandbox would sometimes reject transactions
with a "INVALID_ARGUMENT: Disputed" error if either of
``min_ledger_time_rel`` or ``min_ledger_time_abs`` was set in the
submission request.
CHANGELOG_END
* Fix hashing of keys in the new contract store
Use GlobalKey.hash instead of KeyHasher.hashKey
Fixes#5680
changelog_begin
changelog_end
* Fix order of fields in migration
* Fix migration naming
* Rename file according to class
* Fix hash of active contracts, not consumed ones
* Make the contract keys test fail if the transaction is not rejected by the committer
- First, this removes the assert statement in the DAML model, so that the
transaction validation doesn't fail with a transaction abort.
- Second, the gRPC error message should contain Disputed. `InvalidLookup`
should only be used in Sandbox Classic, but due to code sharing, the
validation that triggers this error is also run for Sadbonx.
* Fix the key lookup in kvutils
The current way of transaction validation and key lookup does not work
in the following scenario:
- the transaction coming from the participant has a negative key lookup
- the contract key state is loaded, which points to a contract that has
been created by another transaction since the command interpretation
on the participant
- the contract state for the contractId inside the contract key state
has not been loaded, therefore the visibility and activeness check in
`ProcessTransactionSubmission.lookupKey` returns `None`, even though
there is in fact a contract with such a key.
To mitigate this issue, the `lookupKey` function must not check
activeness or visibility. If the submitter is not allowed to load the
contract via a key lookup, the authorization check should catch that.
Any other situation that is inconsistent should result in a rejection of
the transaction.
A small side effect of this change is that the `lookupContract` function
might try to load a contract that wasn't loaded from the kv state
earlier. This only happens because we first rebuild the entire
transaction before we compare it to the original transaction.
Previously we threw an exception in this case, but this results in a rather
ugly abort of the reinterpretation of the transaction. Therefore the
`lookupContract` function just returns `None`. Eventually the
transaction will be rejected during validation of the nodes, because
the `LookupByKey` nodes will be different.
This would not happen if we reinterpreted the transaction lazily and
compared the nodes as soon as they are produced with the corresponding
node of the original transaction.
CHANGELOG_BEGIN
[kvutils] Fixed the validation of key lookups.
See issue `#5562 <https://github.com/digital-asset/daml/issues/5562>`__.
[Participant Server, Ledger API Test Tool] Use rejection reasons of the participant-state API instead of sandbox specific ones.
CHANGELOG_END
* Validate causal monotonicity of used contract keys
When using LookupByKey, the return contract isn't necessarily fetched or used
any other way. This means that we need an extra pass over all contract keys to
ensure that causal monotonicity is respected. To be able to do this,
there is now a new field `DamlContractKeyState.active_at`, which is set to the
same value `DamlTransactionEntry.ledger_effective_time`.
An additional check in `ProcessTransactionSubmission` uses this field to verify
causal monotonicity.
This could lead to slow rollbacks that could cascade on errors that
pile up and keep precious DB threads busy for longer than needed.
changelog_begin
changelog_end
* Create single-party specialized queries to access events
Closes#5620
changelog_begin
changelog_end
* Ensure ordering of events when performing a lookup
- removes unused `is_root` column, closes#5618
- merges `template_package_id` and `template_name` in `template_id`, closes#5619
changelog_begin
[Ledger API] clean up of the participant index, improved performance when requesting transactions by template
changelog_end
Instead, opt-in explicitly in _ledger-on-memory_ and _ledger-on-sql_.
Wrapping the operations can confuse other users of `LedgerStateAccess`.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-common: Add a metric for command submission validation.
CHANGELOG_BEGIN
- [Ledger API Server] Add a metric for command validation upon
submission, ``daml.commands.validation``.
CHANGELOG_END
* ledger-api-common: Move the submissions metric from sandbox.
* sandbox: Add a test for MetricsInterceptor for non-streaming endpoints.
* sandbox: Simplify MetricsInterceptor by overriding `ServerCall`.
* sandbox: Time all submissions, not just through the submission service.
* sandbox: Don't bother timing transaction submissions twice.
They're already timed by TimedWriteService.
* sandbox: Move command submission metrics to "daml.commands".
CHANGELOG_BEGIN
- [Ledger API Server] A number of command submission metrics have moved,
and we have added a couple.
- ``daml.commands.submissions`` is a new timer that measures all
submissions.
- ``daml.commands.valid_submissions`` is a new meter that counts
valid (unique, interpretable) submissions.
- ``daml.lapi.command_submission_service.failed_command_interpretations``
has been moved to ``daml.commands.failed_command_interpretations``.
- ``daml.lapi.command_submission_service.deduplicated_commands`` has
been moved to ``daml.commands.deduplicated_commands``.
- ``daml.lapi.command_submission_service.delayed_submissions`` has
been moved to ``daml.commands.delayed_submissions``.
- ``daml.lapi.command_submission_service.submitted_transactions`` has
been moved to ``daml.services.write.submit_transaction``.
- All ``daml.execution.*`` metrics have moved to
``daml.commands.execution``.
CHANGELOG_END
* metrics: Update the dashboard to include the new command metrics.
* Add time provider type to ledger API server
Static time sometimes needs special treatment
* Delay submissions of transactions with future time
Fixes#5480.
CHANGELOG_BEGIN
- [Sandbox] The sandbox now properly delays command submissions using
minLedgerTimeAbs or minLedgerTimeRel. See
`issue #5480 <https://github.com/digital-asset/daml/issues/5480>`_.
CHANGELOG_END
