* add GenMap to the "all types" test generators
* report bad GenMap format with DeserializationError, not MatchError
* document GenMap JSON
* notes on missing features
* enable -Xsource:2.13 in transaction
* make an Order instance for Value resolvable, but unimplemented
* use the skeleton from SValue ordering to make a Value ordering skeleton
* add Party Order
* add Order instance for SortedLookupList
* add Order for FrontStack, deriving everything
* factor the Order lookup, and tie a knot in the recursive Value instances
* we're going to need this Iterator thing again
* replacing Order#contramap with version that supports equalIsNatural
* use new equalBy, orderBy for FrontStack, SortedLookupList, ImmArray
* _2 comparator, upgrade Name Equal to an Order
* incorporate lookup for enums, variants into Value order; record/struct cases
* Enum/Variant comparison
* looking up the singleton implicitly won't work for non-`object`s, alas
* test Order laws for values of all primitive types
* test Order laws for record and variant types
* test Order laws for enum types
* test that enum strings are not compared
* use checkLaws for Value Equal as well
* test that enums match order to constructor rank
* factor genAddend and genAddendNoListMap
* reintroduce Order for TypedValueGenerators
* more addend order
* record, variant order cases
* record cons order
* deriving Order while decoding from JSON
* make ApiCodecCompressed's Cid codec based on the typeclass
* test how the Value ordering and the underlying projected value orderings line up
- hint: they don't, yet
- this is also a template for how we'll check the fidelity with SValue
ordering
* test how the Value ordering and SValue ordering line up
- hint: they don't, yet
* typed Arbitrarys have access to Order
* produce proper ValueGenMap
* inj requires Order, sometimes
- we encode this as "all the time" but there is a type-level unification
approach to remove this requirement in some cases
* make inj a function
* test that order doesn't matter for JSON decoder
* use Utf8 order for TVG text; don't pretend that base equal works
* sort JSON GenMaps, and check for duplicates
* make injarb use IntroCtx
* remove stray import
* Order instances for Bytes, Hash, AbsoluteContractId
* require Order[Cid] to decode JSON to LF values
* clean up map reordering test
* remove unused Instant instance
* fake Order instance no longer needed, valid instance defined
* test parity of global AbsoluteContractId order and SContractId order
* bazel fmt
* test AbsoluteContractId Order lawfulness
* test duplicate key detection
CHANGELOG_BEGIN
- [JSON API] Prepare full support for the planned GenMap primitive type.
See `issue #5031 <https://github.com/digital-asset/daml/issues/5031>`_.
CHANGELOG_END
* Upgrade scala compiler silencer to 1.6.0
CHANGELOG_BEGIN
CHANGELOG_END
* Adapt build bazel file to new targets
* Switch to silencer plugin scala 2.12.11 per Samir's feedback
rather than 2.12.8
* Add missed bazel files
* Review feedback from Leo
* Integrate PostCommitValidation with JdbcLedgerDao and SqlLedger
Closes#5035Closes#5663
changelog_begin
[Sandbox] Skip unnecessary double post-commit validation inherited by sandbox-classic, expect performance improvement
changelog_end
* Ensure SqlLedger recovers from failures and logs them when publishing a transaction
* Remove unused import
* Remove tests for ledger entries
* Fix completions test to make them compile
* Fix compilation errors in tests, address self-review items, apply necessary fixes
- address https://github.com/digital-asset/daml/pull/5781#pullrequestreview-403293667
- address https://github.com/digital-asset/daml/pull/5781#pullrequestreview-403378192
* Pass TransactionTimeModelComplianceIT
* Minor tweaks to variable naming
* Fix failing tests
* Stop deduplicating commands on failures
* Attempt at making sandbox-classic allocate parties implicitly
* Remove implicit party allocation test (without full server) for SQL backed sandbox-classic
* Removing ImplicitPartyAdditionIT (covered in conformance tests)
* Add migrations
* Fix test for ledger DAO with post-commit validation against PostgreSQL
* Update PostgresIT
* Fix missing/wrong items from previous commits
* Don't perform batch processing of enqueued persistence entries
* Rebase against master
* Sandbox: expose back pressure config in CLI
CHANGELOG_BEGIN
[Sandbox]: Added ``--max-commands-in-flight`` as CLI configs. See ``daml sandbox --help``.
[Sandbox Classic]: Added ``--max-commands-in-flight`` and
``--max-parallel-submissions`` as CLI configs. See ``daml sandbox-classic --help``.
CHANGELOG_END
* Bumping the default maxParallelSubmissions to 512 for sandbox classic
We used to use `maxCommandsInFlight * 2` in SqlServer, but it makes more
sense to use `maxParallelSubmissions` there. Since the lower default value of 128
would result in the conformance tests to fail, I'm bumping it to 512
* Use maxCommandsInFlight to configure the parallel submissions for CommandService
* Add a reason text field to RejectReason.Inconsistent (#5180)
CHANGELOG_BEGIN
- Add a reason text field to RejectReason.Inconsistent.
See `#5810 <https://github.com/digital-asset/daml/issues/5810>`__.
CHANGELOG_END
* Change wording in contributing instructions to reflect best practice (#5820)
* Also make add reason text to other reject reasons that don't have it (#5820)
* Update with review comments (#5820)
* Update with review comments (#5820)
* Update with review comments (#5820)
- Only upload packages during the initial startup.
- Avoid loading packages during subsequent resets
- Share an engine between Ledger API Server and Committer
* Use a randomized H2 URL to simular in-memory
The reset service test assumes to get a completely new ledger for each
test case. But because we use H2 in-memory with db_close_delay=1 and the
same H2 database name, the second test case gets the remnants of the
first test case.
Since we know that sandbox in-memory uses an H2 in-memory URL, we can
simply use SandboxBackend.H2Database for ResetServiceInMemoryIT.
CHANGELOG_BEGIN
[Sandbox] Drastically lower the time needed to do a reset via the
ResetService.
CHANGELOG_END
* Adjust PostCommitValidation
- do not require a party for validation (to validate divulged contracts)
- stop validation after first error and don't accumulate rejection reasons
- address 01da7393b3 (r416719399)
changelog_begin
changelog_end
* Use participant state rejection reasons, drop unused submitter parameter from validation
* Implement replacement for post-commit validation
PostCommitValidation is meant as a replacement for ActiveLedgerState and
ActiveLedgerStateManager for JdbcLedgerDao so that the old table can eventually
be retired and post-commit validation can happen on the new participant schema.
This commit simply introduces the implementation and tests it in isolation.
Integration with the existing system will come in a separate contribution.
changelog_begin
changelog_end
* Address https://github.com/digital-asset/daml/pull/5737#discussion_r415918949
* Address https://github.com/digital-asset/daml/pull/5737#discussion_r415921207
* Add tests for the empty ledger scenario and introduce necessary fixes
* Add more tests for fetch and using some prior history
* Add tests for divulged contracts
* sandbox: Split up LedgerApiServer.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Extract out API service construction from LedgerApiServer.
* sandbox: Pull out a case class for GrpcServerOwner event loop params.
* sandbox: Split ownership of `ApiServices` from the object.
In #5419 we set specific loggers to WARN or ERROR so that we avoid noisy
logging by default. However, if the user specifies the --log-level CLI
param, not only the root logger, but also other loggers set in logback.xml
should log at that level.
CHANGELOG_BEGIN
[Sandbox] Properly respect the --log-level CLI parameter
CHANGELOG_END
The duplicated information was not always in sync.
Fixes#5662
CHANGELOG_BEGIN
- [Sandbox] Fix an issue where the sandbox would sometimes reject transactions
with a "INVALID_ARGUMENT: Disputed" error if either of
``min_ledger_time_rel`` or ``min_ledger_time_abs`` was set in the
submission request.
CHANGELOG_END
* Fix hashing of keys in the new contract store
Use GlobalKey.hash instead of KeyHasher.hashKey
Fixes#5680
changelog_begin
changelog_end
* Fix order of fields in migration
* Fix migration naming
* Rename file according to class
* Fix hash of active contracts, not consumed ones
* Make the contract keys test fail if the transaction is not rejected by the committer
- First, this removes the assert statement in the DAML model, so that the
transaction validation doesn't fail with a transaction abort.
- Second, the gRPC error message should contain Disputed. `InvalidLookup`
should only be used in Sandbox Classic, but due to code sharing, the
validation that triggers this error is also run for Sadbonx.
* Fix the key lookup in kvutils
The current way of transaction validation and key lookup does not work
in the following scenario:
- the transaction coming from the participant has a negative key lookup
- the contract key state is loaded, which points to a contract that has
been created by another transaction since the command interpretation
on the participant
- the contract state for the contractId inside the contract key state
has not been loaded, therefore the visibility and activeness check in
`ProcessTransactionSubmission.lookupKey` returns `None`, even though
there is in fact a contract with such a key.
To mitigate this issue, the `lookupKey` function must not check
activeness or visibility. If the submitter is not allowed to load the
contract via a key lookup, the authorization check should catch that.
Any other situation that is inconsistent should result in a rejection of
the transaction.
A small side effect of this change is that the `lookupContract` function
might try to load a contract that wasn't loaded from the kv state
earlier. This only happens because we first rebuild the entire
transaction before we compare it to the original transaction.
Previously we threw an exception in this case, but this results in a rather
ugly abort of the reinterpretation of the transaction. Therefore the
`lookupContract` function just returns `None`. Eventually the
transaction will be rejected during validation of the nodes, because
the `LookupByKey` nodes will be different.
This would not happen if we reinterpreted the transaction lazily and
compared the nodes as soon as they are produced with the corresponding
node of the original transaction.
CHANGELOG_BEGIN
[kvutils] Fixed the validation of key lookups.
See issue `#5562 <https://github.com/digital-asset/daml/issues/5562>`__.
[Participant Server, Ledger API Test Tool] Use rejection reasons of the participant-state API instead of sandbox specific ones.
CHANGELOG_END
* Validate causal monotonicity of used contract keys
When using LookupByKey, the return contract isn't necessarily fetched or used
any other way. This means that we need an extra pass over all contract keys to
ensure that causal monotonicity is respected. To be able to do this,
there is now a new field `DamlContractKeyState.active_at`, which is set to the
same value `DamlTransactionEntry.ledger_effective_time`.
An additional check in `ProcessTransactionSubmission` uses this field to verify
causal monotonicity.
This could lead to slow rollbacks that could cascade on errors that
pile up and keep precious DB threads busy for longer than needed.
changelog_begin
changelog_end
* Create single-party specialized queries to access events
Closes#5620
changelog_begin
changelog_end
* Ensure ordering of events when performing a lookup
- removes unused `is_root` column, closes#5618
- merges `template_package_id` and `template_name` in `template_id`, closes#5619
changelog_begin
[Ledger API] clean up of the participant index, improved performance when requesting transactions by template
changelog_end
Instead, opt-in explicitly in _ledger-on-memory_ and _ledger-on-sql_.
Wrapping the operations can confuse other users of `LedgerStateAccess`.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-api-common: Add a metric for command submission validation.
CHANGELOG_BEGIN
- [Ledger API Server] Add a metric for command validation upon
submission, ``daml.commands.validation``.
CHANGELOG_END
* ledger-api-common: Move the submissions metric from sandbox.
* sandbox: Add a test for MetricsInterceptor for non-streaming endpoints.
* sandbox: Simplify MetricsInterceptor by overriding `ServerCall`.
* sandbox: Time all submissions, not just through the submission service.
* sandbox: Don't bother timing transaction submissions twice.
They're already timed by TimedWriteService.
* sandbox: Move command submission metrics to "daml.commands".
CHANGELOG_BEGIN
- [Ledger API Server] A number of command submission metrics have moved,
and we have added a couple.
- ``daml.commands.submissions`` is a new timer that measures all
submissions.
- ``daml.commands.valid_submissions`` is a new meter that counts
valid (unique, interpretable) submissions.
- ``daml.lapi.command_submission_service.failed_command_interpretations``
has been moved to ``daml.commands.failed_command_interpretations``.
- ``daml.lapi.command_submission_service.deduplicated_commands`` has
been moved to ``daml.commands.deduplicated_commands``.
- ``daml.lapi.command_submission_service.delayed_submissions`` has
been moved to ``daml.commands.delayed_submissions``.
- ``daml.lapi.command_submission_service.submitted_transactions`` has
been moved to ``daml.services.write.submit_transaction``.
- All ``daml.execution.*`` metrics have moved to
``daml.commands.execution``.
CHANGELOG_END
* metrics: Update the dashboard to include the new command metrics.
* Add time provider type to ledger API server
Static time sometimes needs special treatment
* Delay submissions of transactions with future time
Fixes#5480.
CHANGELOG_BEGIN
- [Sandbox] The sandbox now properly delays command submissions using
minLedgerTimeAbs or minLedgerTimeRel. See
`issue #5480 <https://github.com/digital-asset/daml/issues/5480>`_.
CHANGELOG_END
* ledger-on-memory: Add metrics to the operations.
* ledger-on-memory: Speed up reads by using views.
* kvutils: Time event processing.
* ledger-on-(memory|sql): Make classes private and final where possible.
* kvutils: Factor out a TimedLedgerStateOperations class.
CHANGELOG_BEGIN
- [Ledger Integration Kit] Report timing metrics for ledger state
operations.
CHANGELOG_END
* ledger-on-sql: Record log read metrics.
* ledger-on-sql: Record database transaction timing metrics.
CHANGELOG_BEGIN
- [Sandbox] Record ledger database timing metrics under "daml.ledger".
CHANGELOG_END
* ledger-on-sql: Time queries.
* metrics: Add graphs for read events and DAML-on-SQL.
* ledger-on-memory: Simplify the tuple swap in reading the log.
Co-Authored-By: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* sandbox: Construct PartyConfiguration later.
* sandbox: Provide an option to disable implicit party allocation.
CHANGELOG_BEGIN
- [Sandbox] You can now disable implicit party allocation by passing the
flag, ``--implicit-party-allocation=false``. This makes it easier to
test as you would against another ledger which does not support this
feature.
CHANGELOG_END
* ledger/metrics: Move metric helpers to their own Bazel package.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Use ledger/metrics.
* metrics: Rename `Metrics` to `Timed` and drop the `timed` prefix.
Importing methods is harder than importing objects.
* metrics: Publish to Maven Central.
We started serving the LedgerAPI from the new schema in #5104, #5305, #5346,
and #5397.
We must therefore also add a migration that moves the data from
ledger_entries to the new tables.
To keep the migration relatively stable with respect to refactoring the
code, I copied a bunch of files to
`db.migration.postgres.v25_backfill_participant_events`.
As a bonus, I added a line to logback.xml so that we get the usual
sandbox output on startup.
CHANGELOG_BEGIN
CHANGELOG_END
* participant_contracts write path
Writes contracts alongside events for interpretation and validation.
changelog_begin
changelog_end
* Fix self-referencing foreign key
* Address https://github.com/digital-asset/daml/pull/5522#discussion_r406289145
* Compute divulgence only if necessary
* Do not attempt at persisting witnesses for transient contracts
* Drop new tables when resetting
* sandbox: Time command execution.
* sandbox: Tiny cleanups in StoreBackedCommandExecutor.
* sandbox: Add metrics for fetching within execution.
CHANGELOG_BEGIN
- [Ledger API Server] Add metrics under "daml.execution" to measure
command execution time, and fetching within command execution.
CHANGELOG_END
* sandbox: Revert the ApiSubmissionService metric prefix.
It can be private again.
* sandbox: Record a mark on a meter on command execution retries.
- removes unnecessary reference to offsets
- pushes event identifier translation out to enable code re-use
- event identifier translation is inline in other iterations, no significant perf degradation
- this will allow to share code between event and contract witness table objects
changelog_begin
changelog_end
* sandbox: Remove `ErrorCause.Sequencer`.
It doesn't seem to be constructed anywhere.
Feel free to close this if it's used outside the repository.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Don't deconstruct when we can check types in a pattern match.
* sandbox: Move command execution into its own package.
* sandbox: Make LedgerTimeHelper into a CommandExecutor implementation.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Rename CommandExecutorImpl to StoreBackedCommandExecutor.
And pass it objects, not functions.
* sandbox: Reorder result cases in StoreBackedCommandExecutor.
* sandbox: Inject the LedgerTimeAwareCommandExecutor.
* sandbox: Pull out the default time provider type into a constant.
* sandbox: Name ResetService tests consistently.
* sandbox: Reset the time service when resetting in Static Time mode.
The test, unfortunately, is ``@Ignore`d due to flakiness in CI, so won't
actually be run. However, I _hope_ we're going to remove that
annotation eventually, and it allowed me to test-drive the fix on my
machine, so it's still helpful.
CHANGELOG_BEGIN
- [Sandbox] Fix a regression in the ResetService which did not reset the
TimeService in static time mode.
CHANGELOG_END
Turns out not every participant can support seeding (yet).
CHANGELOG_BEGIN
- [Ledger API Server] Re-introduce an option to disable seeding. This
does not affect Sandbox.
CHANGELOG_END
* Set the `Bearer ` prefix in bindings.
* Make the `Bearer ` prefix in the authorization header mandatory.
* Bearer prefix can be removed from the token file.
CHANGELOG_BEGIN
[Extractor]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[Navigator]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[DAML Script] The ``Bearer `` prefix can be removed from the token file. It
is added automatically.
[DAML Repl] The ``Bearer `` prefix can be removed from the token file. It is
added automatically.
[Scala Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[Java Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[DAML Integration Kit] ``AuthService`` implementations MUST read the
``Authorization`` header and the value of the header MUST start with
``Bearer ``.
CHANGELOG_END
* Produce performance tests for all envelopes
CHANGELOG_BEGIN
[TestTool] Provide performance tests for all performance envelopes
CHANGELOG_END
* Follow Scala's recommended pattern for enum-like hierarchies
There have been reports of sporadic occurrences of
`java.lang.RuntimeException: Lob not found: 49/-2`
when running the participant server with H2.
It seems like using `setBinaryStream` triggers H2 to go through the BLOB
machinery. Since this issue is not easily reproducible, it seems like an
altogether better solution to switch to using `setBytes`. Offsets aren't
that large anyway, so going directly for byte array should be fine.
Better than a broken query anway.
CHANGELOG_BEGIN
CHANGELOG_END
* Add failing test
The test can produce false negatives,
but locally it fails 10 out of 10 times.
* Stop deduplicating commands after rejections
Fixes#5338.
CHANGELOG_BEGIN
CHANGELOG_END
* Only deduplicate successful transactions
* Use pass instead of pure unit
* Handle exceptions
* kvutils: Simplify calculating the weight of a Caffeine cache.
And remove an errant `println` that slipped through the cracks.
Thank you to @ben-manes for the tip!
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Make classes final and defs into vals.
Co-Authored-By: Stefano Baghino <stefano.baghino@digitalasset.com>
Co-authored-by: Stefano Baghino <stefano.baghino@digitalasset.com>
We do this for Sandbox Next™; we can do it for Sandbox Classic™ too. It
doesn't seem to have much of a noticeable impact on the test run time,
and reduces the number of exclusive tests, which is helpful.
CHANGELOG_BEGIN
CHANGELOG_END
* Add legacy proxy gRPC services.
This exposes the services as com.digitalasset as well, to ensure that
applications built with a previous release of the SDK continue to work
with the Ledger API.
Due to how the gRPC reflection service works, this doesn't expose the
com.digitalasset services on the reflection api, and thus grpcurl won't
work with the old services. These scripts need to be updated to refer to
the com.daml services.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Cache state value conversions from bytes.
This seems to have a decent speedup in ledger-on-memory.
CHANGELOG_BEGIN
- [Ledger Integration Kit] Submissions now look up ledger values from a
cache where possible, improving performance when there's contention over
certain resources (e.g. common packages). The cache size currently
defaults to 64 MB.
CHANGELOG_END
* kvutils: Make the SubmissionValidator statue value cache configurable.
* kvutils: Report state value cache metrics.
* kvutils: Add a suffix to a Long literal because WartRemover is unhappy.
Strangely, it doesn't fail on my machine.
* kvutils: Extract caching out into its own file.
* kvutils: Move the `bytesToStateValue` call into `cache.get`.
* kvutils: Move caching to its own package.
* kvutils: Inject the state value cache.
* kvutils: Default to no state value cache.
* kvutils: Accept a state value cache size in megabytes, not bytes.
* kvutils: Move cache building from `Config` to the `caching` package.
* kvutils: Replace Guava's cache with Caffeine.
* kvutils: Simplify caching configuration.
* sandbox: Enable state value caching by default.
CHANGELOG_BEGIN
- [Sandbox] State values deserialization is now cached, with a fixed
cache size of 128MB.
CHANGELOG_END
* Changelog commit.
CHANGELOG_BEGIN
- [Ledger Integration Kit] The state value cache is now opt-in, with a
default of no cache at all.
CHANGELOG_END
* Draft of PingPong throughput and latency benchmarks for on-mem and on-sql ledgers
* Augment `ParticipantTestContext` and remove `LedgerApiServer` hack
* Separate performance tests into distinct category with concurrency 1
* 🎨
* Package performance tests in separate DAR
* Have performance tests excluded by default and run exclusively if passed
* Fix rebase
* Simplify `BenchmarkReporter`
* Make `concurrencyOverride` into an `Option`
* Clarify command line usage, prevent regular and perf. tests together
* Fix preventing regular and perf. tests together
* Split `PingPong`, `PingPongExplode` and `Cycle` benchmarks' model
CHANGELOG_BEGIN
- [TestTool] Add `PingPong` performance envelope test
CHANGELOG_END
* Explicitly name `concurrencyOverride`
* Fix formatting
* Lower bar for CI run of performance envelope tests
* Make benchmark output file configurable
* Improve messages and report config option name
* Use exit status 64 for "bad command line usage" as in BSD
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* Fix test for groupContiguous
Automatic generation of test values was prone to cause flakiness, removed in favor of a simpler test case.
changelog_begin
changelog_end
* Relax order sensitivity
* Update ledger/sandbox/src/test/suite/scala/com/digitalasset/platform/store/dao/events/GroupContiguousSpec.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Fix compilation issue
Co-authored-by: Remy <remy.haemmerle@daml.com>
* sandbox: Move the events page size configuration value into config.
* sandbox: Pass `config` directly into JdbcIndexerFactory.
* sandbox: Reorder `eventsPageSize` before `metrics` in parameters.
* sandbox: Move `seeding` into `ApiServerConfig`.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Name all parameters of `JdbcLedgerDao.writeOwner`.
Co-Authored-By: stefano.baghino@digitalasset.com
* Implement timed command deduplication in kvutils
This adds a field deduplication_time to DamlCommandDedupValue for
deduplication timeout checking.
* Bump kvutils version to 4
* Fix CommandTracker pulling commandResultIn multiple times
Now that the timeouts are generated out of band, we have 2
"unsynchronized" places that pull on commandResultIn.
Whenever we pull, we need to check that commandResultIn hasn't been
pulled before.
* Add inStaticTimeMode flag to enable command dedup in sandbox-next with static-time
Fixes#4624.
CHANGELOG_BEGIN
[kvutils] KVUtils now respects the command deduplciation time instead of
deduplicating commands forever.
CHANGELOG_END
* kvutils: Remove the LedgerEntry trait; it's no longer necessary.
This was introduced to allow for heartbeats, which no longer exist.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Make LedgerRecord a case class again.
We used to store the envelope as an array of bytes, which doesn't have a
value-based `equals` method and therefore should not be used in a case
class. We now use a `ByteString`, so this is no longer an issue.
* Migrate create_consumed_at to offset
This is a leftover from the stable offsets migration and causes issues when serving active contracts from the new schema.
changelog_begin
changelog_end
* No decoding necessary for create_consumed_at
* kvutils: Use `.view` in SubmissionValidator.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Don't compute missing inputs unless we're asked.
* ledger-on-memory: Do less in `InMemoryLedgerStateOperations`.
* ledger-on-memory: Use `RangeSource` instead of `OneAfterAnother`.
Should be faster to just take a slice.
* ledger-on-memory: Don't bother locking when reading.
We're only reading the log, which is append-only; we never mutate
existing data. This means we don't need to lock to read it.
* ledger-on-memory: Make it impossible to construct a state with data.
This can fail in CI, especially when we're running tests in parallel,
and starting PostgreSQL as well.
I've increased the limit from 30 seconds to 1 minute.
This is hopefully not a permanent fix; I'm going to look into doing this
without an `Await.result`.
CHANGELOG_BEGIN
CHANGELOG_END
* Refactor flat events range queries
By factoring out query routing, this logic can be re-used to serve active contracts.
changelog_begin
changelog_end
* Fix copyright notice header
* participant-state-metrics: Wrap metric names in a value type.
For safety, and for simplicity when building upon prefixes.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Use MetricName within MetricsNaming.
* kvutils: Use `MetricName` in `Committer`.
* sandbox | kvutils: Extract common metric prefixes.
* sandbox: Remove a redundant visibility modifier in `MetricsNaming`.
* participant-state-metrics: `MetricName` doesn't need to be a case class.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Retry if time lookup failed
An error while looking up the maximum ledger time for the used contracts
likely means that one of the contracts is already not active anymore,
which can happen under contention.
Retry finding a suitable ledger time in this case instead of crashing.
CHANGELOG_BEGIN
CHANGELOG_END
* Handle archived contracts in JDBC max time lookup
* Don't retry if ledger time lookup fails
The retry might not fix anything.
Instead, log a helpful message.
Contributes to #4231.
Remove checkpoints from Participant Server storage:
- Removed code to store checkpoints in the index database.
- Remove existing checkpoint rows in ledger_entries and participant_command_completions.
- Removed ObservedTimeServiceBackend
This commit modifies the java migration V2_1__Rebuild_Acs. This is safe to do, because:
a) any even semi recent persistent sandbox had already gone through this migration and won't re-run it
b) A new database doesn't even have entries yet to migrate.
CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Removed the ``Hearbeat`` state update.
[Sandbox] Checkpoints are no longer emitted in regular intervals in wall
clock time mode.
CHANGELOG_END
* Use com.daml as groupId for all artifacts
CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END
* Add 2 additional maven related checks to the release binary
1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId
* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
* kvutils: Do less in the ledger transaction.
* kvutils: Make `SubmissionValidator#runValidation` tail-recursive.
Because otherwise IntelliJ is very unhappy.
* kvutils: Time transaction acquisition and release.
CHANGELOG_BEGIN
- [Ledger Integration Kit] Metrics for acquiring and releasing
transactions.
CHANGELOG_END
* kvutils: Include the word "lock" in validator transaction lock metrics.
* kvutils: Record successful + failed transaction acquisitions separately.
Just to get them consistent with the others.
CHANGELOG_BEGIN
- [Ledger Integration Kit] Prefixed all metrics with "daml." for
consistency.
CHANGELOG_END
* kvutils: Use `MetricRegistry.name` to create metric names.
String interpolation is 4TL.
* sandbox: Make the metric names of the services consistent with kvutils.
CHANGELOG_BEGIN
- [Sandbox] Move the service metrics from ``daml.sandbox.indexService``
and ``daml.sandbox.writeService`` to ``daml.services.index`` and
``daml.services.write`` respectively. This brings Sandbox in line with
the Ledger Integration Kit.
CHANGELOG_END
* sandbox: Use `MetricRegistry.name` to create metric names.
String interpolation is 4TL.
* kvutils/app: Publish JVM metrics.
* kvutils/app: Sort the metrics prefixes.
* kvutils/app: Allow for specifying a metrics reporter as a CLI argument.
No changelog as this is hidden.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Remove an unnecessary string interpolation mark.
Co-Authored-By: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
* Add canonical string representation for identifiers
An identifier is represented as a string as a package identifier and a qualified name separated by a colon.
changelog_begin
changelog_end
* Simplify explicit type signature for HexString
Thanks @remyhaemmerle-da
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Fix hashing test expected output
* Fix key hasher test expected output
Co-authored-by: Remy <remy.haemmerle@daml.com>
Before this reaches production-ready status, it's time to squash the
migrations. This will improve startup performance a little.
CHANGELOG_BEGIN
CHANGELOG_END
* Remove old time model from ledger config
CHANGELOG_BEGIN
- [Ledger API] Fields related to the old ledger time
model have been removed from the configuration
management service and the ledger configuration service.
CHANGELOG_END
* Update ledger/ledger-api-test-tool/src/main/scala/com/daml/ledger/api/testtool/tests/LedgerConfigurationService.scala
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Remove documentation for a removed option
CHANGELOG_BEGIN
- [DAML Integration Kit] The CLI option command-submission-ttl-scale-factor was
removed, as the LET/MRT/TTL fields have recently been removed
from the command submission service.
CHANGELOG_END
* Minor renaming
* sandbox: On reset, wait for the API server to start before replacing it.
Hopefully this addresses the flickering behavior we're seeing in CI.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Scale the retries in ResetServiceIT along with everything else.
* Allow to have both verbose and succinct row parsers
This will be used when we need to stream transactions and not individual
events.
changelog_begin
changelog_end
* Succinct parser should have verbose = false
* Introduce DamlSubmissionBatch and the BatchingLedgerWriter
This introduces the DamlSubmissionBatch message to group
submissions into a single message and extends Envelope to
carry the batch. We're using the envelope wrapping for consistency
and compatibility.
We're adding this to kvutils version 3 as it has not yet been released
into the wild and as this is not backwards incompatible change.
Support for batching is implemented with the BatchingLedgerWriter
that wraps a LedgerWriter and groups submissions into a batch based
on size and time duration.
For implementing the validation of a batch we will require some rework
in the SubmissionValidator to be able to produce multiple "LogResult"s,
e.g. commit on the in-memory ledger results in an "Index" which is used
to signal new head to dispatcher. With a batch we'd need to pick max index.
CHANGELOG_BEGIN
CHANGELOG_END
* Add missing copyright header
* Address code reviews
* Post rebase fixes
* Rename BATCH -> SUBMISSION_BATCH
* Address code reviews, add further tests and cleanup.
* Add test for DefaultBatchingQueue.close
* Use generous timeouts
* Renamed BatchMessage => SubmissionBatchMessage. Added default boolean parameter value. Added simple test case.
* Removed unused include.
* Address final code review
Co-authored-by: Miklos Erdelyi <miklos.erdelyi@digitalasset.com>
Add keys with maintainers to Fetch nodes
The new field is populated by the interpreter whenever the fetched
contract has a key. Used for contract key reinterpretation in Canton.
CHANGELOG_BEGIN
- [DAML-LF] Add keys with maintainers to Fetch nodes in transactions.
CHANGELOG_END
Contributes to #4194.
Closes#4231.
Closes#5022.
CHANGELOG_BEGIN
- [Ledger API] The protobuf fields ledger_effective_time and maximum_record_time have been removed from
command submission. These fields were previously deprecated following the introduction
of a new ledger time model. See issue `#4194 <https://github.com/digital-asset/daml/issues/4194>`__.
[Java Bindings] removed the usage of ledgerEffectiveTime and
maximumRecordTime, and instead added minLedgerTimeAbsolute and
minLedgerTimeRelative in CommandSubmissionClient and CommandClient
CHANGELOG_END
* participant-state{,-index}: Move Timed*Service classes from Sandbox.
CHANGELOG_BEGIN
- [Ledger Integration Kit] Metrics for the various read, write, and index
services.
CHANGELOG_END
* kvutils/app: Add timing metrics for read/write/index services.
* participant-state: Move metrics-related code to another Bazel package.
* participant-state-metrics: Add to artifacts.yml.
* participant-state-metrics: Move TimedIndexService back into Sandbox.
Cuts down on dependencies like nobody's business.
Tests would often display this warning:
io.grpc.netty.NettyChannelBuilder buildTransportFactory
WARNING: Both EventLoopGroup and ChannelType should be provided or neither should be, otherwise client may not start. Not provided values will use Nio (NioSocketChannel, NioEventLoopGroup) for compatibility. This will cause an Exception in the future.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Capture timing metrics for API server calls.
`timer` is a superset of `meter`, so this doesn't lose any existing
behavior; just adds new behavior.
CHANGELOG_BEGIN
- [Ledger API Server] Added timing metrics for all GRPC endpoints.
CHANGELOG_END
* sandbox: Rename SandboxClientResource to GrpcClientResource.
* sample-service: Clean up warnings.
* sandbox: Add tests for MetricsInterceptor.
* sandbox: Split the API metrics interceptor from the naming.
* sandbox: Use `MetricRegistry.name` instead of string interpolation.
* rs-grpc-akka: Restrict the test library to the DAML workspace.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Tighten result type
Command execution can't result in a sequencer error
* New helper method for extracting used contracts
* New error clause
* Add a DAO query for the maximum time of contracts
* Implement algorithm for finding ledger time
CHANGELOG_BEGIN
CHANGELOG_END
* fixup ledgerTimeHelper
* Use new ledger time algorithm
* Mark LET/MRT as deprecated
CHANGELOG_BEGIN
- [Ledger API] DAML ledgers have switched to a new ledger time model.
The ledger_effective_time and maximum_record_time fields of command submission are deprecated,
the ledger time of transactions is instead set automatically by the ledger API server.
Ledger time is no longer strictly monotonically increasing, but only follows causal monotonicity:
ledger time of transactions is greater than or equal to the ledger time of any used contract.
See `#4345 <https://github.com/digital-asset/daml/issues/4345>`__.
CHANGELOG_END
* Add ledger time skew check
* Remove command updater
LET/MRT are now deprecated, this class is now useless
* Remove old time model validator
* Switch to new time model check: kvutils
* Switch to new time model check: in-memory ledger
* Switch to new time model check: SqlLedger
* Use initial ledger config
* Ignore user provided LET
* Use TimeProvider in submission services
* Use deduplication_time in daml-script runner
- Also remove unnecessary command completion output of CommandTracker.
- Remove usage of maximum record time in CommandTracker.
* Use arbitrary default value for deduplication time
* Use built-in Instant ordering
* Remove obsolete test
* Remove obsolete test: CommandStaticTimeIT
* Refactor test: TransactionMRTCompliance
* Disable test: CommandTrackerFlow timeout
* thread maxDeduplicationTime through to CommandTracker
* Improve test
* Refactor command client configuration
* Deduplication time should always use UTC
* Add missing method in TimedIndexService after rebase
* Put more details into the deduplication error response.
* Use system time for command dedup submittedAt.
* Use explicit UTC time source in command validator
* Revert CommandTracker[Flow] to previous completion-recovering-behavior
* Adapt scala client command config to new config params
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* kvutils: Remove an unnecessary `@SuppressWarnings`.
* kvutils: Reduce the scope of fields and methods in `Committer`.
* kvutils: Inject the metric registry into `KeyValueCommitting`.
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Inject the metric registry into the committers.
* kvutils: Inject the metric registry into `ProcessTransactionSubmission`.
* kvutils: Avoid shared metric registries in tests.
* kvutils: Recreate the metrics registry per participant state.
* kvutils: Add trailing commas to parameter lists.
Flagrantly encouraged by @stefanobaghino-da.
* recovering-indexer: Don't re-use the metric registry in tests.
* Sandbox: Reveal contract id seeding flag
CHANGELOG_BEGIN
- [Sandbox] Add support for random contract identifiers. See section
`Contract Identifiers Generation` section in
docs/source/tools/sandbox.rst
CHANGELOG_END
The database-backed reset service can (understandably) go a bit slower than the one backed by the in-memory ledger.
This should help avoiding flaky tests.
CHANGELOG_BEGIN
CHANGELOG_END
Some Option2Iterable ignore annotations are not needed, others were needed for unused methods.
In a few occasions we were ignoring the warning for the very purpose for which is was there,
i.e. avoiding an implicit conversion. I'm all for not verifying this rule if we agree we
don't need it.
For ProcessFailedException it was a bit gratuitous, I changed the way in which the exception
message is built.
CHANGELOG_BEGIN
CHANGELOG_END
* Add test to ensure that the reset truncates all tables
The test can be adjusted over time to accomodate for exceptions (which are already there).
Unfortunataly we have to add a new couple of queries to support both Postgres and H2.
Fixes#5130
CHANGELOG_BEGIN
CHANGELOG_END
* Make loose check on configuration_entries
* sandbox: Clean up `MetricsReporting` a little.
Make sure it closes both reporters, and avoid starting things in a
constructor.
* sandbox: Add hidden options for enable metrics reporting.
* sandbox: Add a disambiguating name to the DB connection/thread pools.
CHANGELOG_BEGIN
- [Sandbox] DB connection pool metrics names have changed slightly, from
``daml.index.db.connection`` to ``daml.index.db.connection.sandbox``.
- [Ledger Integration Kit] DB connection pool metrics names have changed
to disambiguate the StandaloneApiServer from the
StandaloneIndexerServer. The former now has a ``.ledger-api-server``
suffix, and the latter now has a ``.indexer`` suffix.
CHANGELOG_END
* sandbox-next: Use the same metrics registry for the API and indexer.
* sandbox: Give a useful error message on an invalid metrics reporter.
And simplify the error messages.
With the arguments `--client-auth=foo --metrics-reporter=foo`, we now
get the output:
```
Error: Option --client-auth failed when given 'foo'. Must be one of
"none", "optional", or "require".
Error: Option --metrics-reporter failed when given 'foo'. Must be one of
"console", or "csv:PATH".
Try --help for more information.
```
* sandbox: Pull out more helpers in `MetricsReporting`.
* sandbox: Rename MetricsReporter classes so they don't clash.
* sandbox: Wrap the `name` parameter in a `ServerName` tagged string.
For safety. Yours, not mine.
* sandbox: Push metrics to Graphite with `--metrics-reporter=graphite`.
* sandbox: Make `MetricsReporter.Graphite` singly-lazy, not doubly-.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: Replace `ServerName` with `ServerRole`.
* sandbox: Fix usage of `ServerRole.Testing` in `LedgerResource`.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
- New seeding type "Static" that uses a fixed seed for generating new seeds.
This is only used in Sandbox and Sandbox-next.
- Remove the fuzzing for submission time in Engine.scala
- DAML-on-SQL: Create new log entry IDs from a provided SeedService.
This allows for generating deterministic transaction IDs in
Sandbox-Next.
Fixes#5107
CHANGELOG_BEGIN
[Sandbox] Add contract-id-seeding=static to allow for predictable contract IDs. This is useful for documentation,
to be able to refer to a specific contract ID instead of having to write "note down the contract ID you see on the screen. we will use it later."
[DAML-on-SQL] Derive the next log entry ID using the provided SeedService. This allows us to also deterministically create transactionIds in static time mode together with `--contract-id-seeding=static`. This should only be used for demos or documentation.
CHANGELOG_END
* sandbox: Remove a few levels of indentation in `ApiCommandService`.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox: Get rid of the `LowLevelCommandServiceAccess` trait.
It only has one implementation. Just use it.
* sandbox: Remove unused type aliases from ApiCommandService.
* kvutils/app|sandbox: Rename `system` to `actorSystem`.
* sandbox: Pass a materializer in to the StandaloneIndexerServer.
There's no need for it to construct one when the caller always has one
available.
CHANGELOG_BEGIN
CHANGELOG_END
* recovering-indexer: Use `SubmissionId` instead of `LedgerString`.
Where appropriate.
* sandbox: Use the materializer implicitly in JdbcIndexer.
by reading parameters.ledger_end column instead of parameters.external_ledger_end.
CHANGELOG_BEGIN
[Ledger API Server]: Upon restart, ledger api server continues consuming unconsumed events rather than
all events from beginning.
CHANGELOG_END
Closes#5121
