CHANGELOG_BEGIN
ledger-api - Command deduplication period can now be specified by setting `deduplication_offset` instead of `deduplication_time` (only valid for v2 WriteService). This change is backwards compatible.
CHANGELOG_END
* Propagate the enriched deduplicationPeriod instead of deduplication duration
* Update the Haskell bindings for the new deduplication period
* Calculate the deduplicateUntil using the new deduplication period for backward compat
* Use consistent naming for deduplication_period
* Cleanup command timeout extraction from deduplication period
* Add the required deduplication_offset to deduplication instead of deduplication_start
* Update haskell bindings to support deduplication_offset
* Add support for deduplication_offset in the ledger-api
* Remove the timestamp-based deduplication from our models to simplify upgrade for users
* Add optional conformance test for offset based deduplication
* Remove buf rule for FIELD_SAME_ONEOF as our change is backwards compatible
* Disable FIELD_SAME_ONEOF buf check for commands file
* Apply suggestions from code review
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Update comment for deduplication period
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* participant-integration-api: Clean up failed trackers.
Otherwise they can hang around forever.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Ensure that all trackers are closed.
* participant-integration-api: Ensure that waiting trackers are closed.
* participant-integration-api: Store the tracker map future in the state.
* participant-integration-api: Fix a race in `TrackerMap`.
If the supplied `Future` to `AsyncResourceState` is too fast, the state
may not be set. We need to initialize the state immediately.
* participant-integration-api: Add more comments to `TrackerMap`.
This gets us into a state where PartialTransaction always has the
authorization context which is much more sensible than having the
committers in speedy while the rest is in ptx.
The reason for having the split in the first place was the scenario
service but now that we create a new speedy machine per submission
that’s no longer a valid reason.
changelog_begin
changelog_end
Create normalized TXs when a partial TX is finalised.
Except in limited cases! (i.e for scenario-runner, sandbox)
CHANGELOG_BEGIN
CHANGELOG_END
normalize values in the engine as they are converted from speedy-values
fix 2.12 build
backout redundant change
ensure byKey field is correctly normalized when constructed by engine
rename flag: valueNormalization -> transactionNormalization
improve comment
delete commented-out code
rename: toValueNorm --> toNormalizedValue
rename: (SValue.) toValue --> toUnNormalizedValue
revert changes to ptx so that the interface to insertCreate() etc is Value-based (not SValue-based)
improve comments
respell: toUnNormalizedValue --> toUnnormalizedValue
fix build
This PR has been created by a script, which is not very smart
and does not have all the context. Please do double-check that
the version prefix is correct before merging.
@SamirTalwar-DA is in charge of this release.
Commit log:
```
640fb68342 Make Index DB enable multiple party additions [DPP-546] (#10623)
b22de6893b LF: Contract ID suffix check in Preprocessor (#10642)
7b94b0674e Map shortened scala test suite names to long names on Windows (#10628)
6e4a24c28c participant-state: Generate correct gRPC error codes by v2 `WriteService` [KVL-1081] (#10656)
663781a159 Update curl 7.73.0 --> 7.78.0 (#10655)
a471f15aa8 Dpp-558 Fix startexclusive queries on oracle (#10649)
e99254fd3e Augment `completion.proto` with deduplication-related info [KVL-1057] (#10619)
a00608c8b3 participant-integration-api: Accommodate changes to max dedup time. (#10650)
29c546c949 [Divulgence pruning] Added `prune_all_divulged_contracts` to PruneRequest [DPP-534] (#10635)
dea57ca099 In-memory fan-out optimizations (#10558)
77eb366627 [JSON-API] key_hash field to speed up fetchByKey queries (#10631)
5001329c9d LF: Comparisons fail at runtime if comparing local vs global CIDs (#10630)
055be4b3cb Disable deprecation warnings for data-dependencies (#10647)
c1559357f6 clean-up gsg README (#10641)
8501832757 DPP-468 StorageBackend tests (#10529)
4e08b47dd1 update NOTICES file (#10645)
733590d591 ledger-api-health: Use the Scala health status values everywhere. (#10640)
5b837ec2d9 Ledger API: add `buf` checks [KVL-1045] (#10625)
f77cd0a81c participant-integration-api: Attempt to fix RecoveringIndexerSpec. (#10639)
9d7f60f03e participant-integration-api: Fix a flaky test. (#10637)
4a9331ca55 Upgrade Nixpkgs [KVL-1045] (#10624)
01b6e89551 update compat versions for 1.17.0-snapshot.20210817.7604.0.0c187853 (#10610)
b578b0e7cf Reminder to put an empty line between subject and body (#10638)
c0fbad1f6e participant-integration-api: Remove `limitMaxCommandsInFlight`. (#10626)
b4af6d1155 Canton testing: Mark one more DeeplyNestedValueIT test flaky (#10636)
e807f4a918 Upgrade to a newer canton version (post 0.27.0 snapshot version) (#10632)
c4513f2eea Oracle append-only schema: enable contract id index on participant_events_xxxx tables (#10633)
3598e0928c LF: Drop contract ID Freshness check (#10620)
37c999e237 ledger-on-sql: Increase the concurrency for conformance tests. (#10622)
46e8c7d762 DPP-460 Extract constant for event sequential IDs (#10564)
121047e0ea DPP-460 Parameter storage consolidation (#10472)
569612ab73 Drop broken symlink config (#10616)
6d0109fca8 Support $ in daml-lf identifiers in the parser (#10609)
c38703ec84 participant-integration-api: Store a status gRPC protobuf. [KVL-1005] (#10600)
0af5b49484 make FinalReason a case class (#10614)
8dd136fc7d bazel-tools: Replace `runner` with either `runner_with_port_check` or `runner_with_port_file`. (#10615)
19c3d28ebf Remove GenMissingString class because it is not used (#10608)
3227e860e0 Use the port file and dynamic port generation in client/server tests. (#10604)
fb19bcb7f3 fix gsg-trigger template (#10611)
b2077026d6 rotate release duty after 1.17.0-snapshot.20210817.7604.0.0c187853 (#10606)
975a5fb32b Move DeduplicationPeriod to ledger-api-domain [KVL-1047] (#10590)
f8a1820cc8 release 1.17.0-snapshot.20210817.7604.0.0c187853 (#10605)
64abf8aaa9 update NOTICES file (#10607)
```
Changelog:
```
[Integration Kit] Corrected gRPC error codes returned by v2 `WriteService` adaptor.
- [Ledger API Server] The API server manages a single command tracker
per (application ID × submitters) pair. This tracker would read the
current ledger configuration's maximum deduplication time on creation,
but never updated it, leading to trackers that might inadvertently
reject a submission when it should have been accepted. The tracker now
reads the latest ledger configuration.
- Update schema version for http-json-api query store with new key_hash field
- Improved performance for fetchByKey query which now uses key_hash field
participant-state - move `DeduplicationPeriod` to ledger-api-domain
dc4629fff9 update NOTICES file (#10580)
8b0a0e7bbc update NOTICES file (#10578)
4b8b67a1b5 Upgrade Scalatest to v3.2.9. (#10576)
41e60f7302 Upgrade to Scala 2.12.14 and 2.13.6. (#10573)
c447898cc9 Fix display of unhandled exceptions in the scenario service (#10572)
4e1a90dffd Enable --incompatible_remote_results_ignore_disk (#10571)
d183ecc6d6 rotate release duty after 1.17.0-snapshot.20210811.7560.0.4f9de4ba (#10556)
76ecb44fbd update compat versions for 1.17.0-snapshot.20210811.7565.0.f1a55aa4 (#10563)
86a03fa7c0 Bump bazel max jvm memory (#10569)
1cc136c033 update NOTICES file (#10565)
c69880ccf1 ledger-api-test-tool: Enforce test naming standards. (#10562)
ee34d0f80c Track command - use types for error handling instead of grpc statuses [KVL-1005] (#10503)
93c25f36b6 Release 1.17 snapshot (#10560)
```
Changelog:
```
- [Ledger API Test Tool] The ``TransactionServiceIT`` test suite has
been split into many test suites. If you are including or excluding
it, you will need to use the new test suite names, or you can use
"TransactionService" as a prefix for all of them.
If you are including or excluding individual tests, you will need to
update your arguments with the new test suite. You can find the new
test suite by running the test tool with the ``--list-all``
flag and looking for the test's short identifier. The short
identifiers have not changed, with the exception of
``TXNoContractKey``, which has been renamed to ``CKNoContractKey`` and
is now in the ``ContractKeysIT`` test suite.
* [Daml export] You can now set the ``--all-parties`` option to generate
a ledger export as seen by all known parties.
ledger-api-client - Propagate definite_answer as metadata in the GRPC response for submit/submitAndWait
[JSON API] Ledger connection errors are now logged at every attempt
akka-bindings: `LedgerClientBinding.commands` now returns a flow of `Either[CompletionFailure, CompletionSuccess]` instead of `Completion` for clearer error handling. For backwards compatiblity the new return type can be turned back into a `Completion` using `CompletionResponse.toCompletion`
```
```
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Azure Pipelines DAML Build <support@digitalasset.com>
* participant-integration-api: Make `TrackerMap` a `Tracker`.
Mostly by moving parameters around.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Clean up `TrackerMap` a little more.
* participant-integration-api: Make `TrackerMap` generic.
* participant-integration-api: Move `Tracker.WithLastSubmission`.
It's only used by `TrackerMap`, so now it's an internal class there.
* participant-integration-api: Only provide the key to `newTracker`.
* participant-integration-api: Subsume the `TrackerMap` cleanup schedule.
* participant-integration-api: Construct the tracker map outside.
* ledger-api-common: Remove some unnecessary braces.
* participant-integration-api: Prettify `TrackerMap` some more.
* participant-integration-api: Make `TrackerMap.selfCleaning` a class.
* participant-integration-api: Add some tests for TrackerMap.
* participant-integration-api: Convert a method to `Runnable`, 2.12-style.
Apparently underscores aren't good enough.
* ledger-api-client: Delete CompletionSuccess#unapply.
It doesn't work on Scala 2.12.
Preparation, small fixes
* Remove unnecessary TODO
* Fix FieldStrategy.idempotentInsert type parameters
Changing the DB-Schema
This change adapts the DB schema for all supported backends. After this change we only populate the party_entries table, and on the query side we reconstruct the state from this.
* Drop the party table
* Add indexing to party_entries
Adapting StorageBackend: ingestion
Since we only ingest party_entries, the party population needs to be removed.
* Drop the party table in ingestion code
* Fixes test
Adapting StorageBackend: queries
Queries needs to be adapted to construct the state from the read side.
* Rewrite queries.
* Fixes reset implementations.
Adapting JdbcLedgerDao
Since underlying storage changed, JdbcLedgerDao can be simplified: no special treatment needed with duplicate errors, since these errors are impossible to happen.
Removing JdbcLedgerDao tests, and adding a new test, testing the behavior of the new event-source party model. Please note: this database refactoring only applies to the append-only schema, so for the mutating schema the test is disabled.
During implementation a bug surfaced: it was not possible anymore to store the is_local information via JdbcLedgerDao.storePartyEntry. Although this bug is a minor issue, since that method is only used from single participant environment, still a fix was also implemented for this, by passing a magic participantId upon non-local party storage.
* Simplify storePartyEntry.
* Fixes bug introduced by append-only.
* adds/adapts tests
Refactoring: remove not used duplicateKeyError from StorageBackend
Changes to JdbcLedgerDao rendered this duplicateKeyError unused.
* Removes unused duplicateKeyError
Adapting sandbox-classic
In sandbox-classic it is not allowed to have updates for parties. Essentially the updates concerning already existent parties were dropped silently with logging without effect.
Here I started by pinning down this behaviour in the SqlLedgerSpec and SqlLedgerSpecAppendOnly. These tests were implemented with the original code in mind.
Then adapted the SqlLedger method: making sure of uniqueness by first trying to lookup the to-be-persisted party.
* Added tests grabbing a hold on original behavior
* Adapted implementation to ensure same behavior
Switching to correct is_local derivation for party queries as per review
* Adapting implementation: switching to aggregated table and a query on that
* Introducing QueryStrategy.booleanOrAggregationFunction to support Oracle
* Moving party related queries to PartyStorageBackendTemplate
* Fixes JdbcLedgerDaoPartiesSpec tests, and add another test case
Also:
* Align Update interface documentation
* Switching to explicit optionality in party query implementation asa per review
Co-authored-by: Simon Meier <meiersi-da@users.noreply.github.com>
CHANGELOG_BEGIN
CHANGELOG_END
This PR makes possible to check for contract IDs suffix during
preprocessing.
This is the first part of the task 3 described in #10504.
CHANGELOG_BEGIN
CHANGELOG_END
* Generate short to long name mapping in aspect
Maps shortened test names in da_scala_test_suite on Windows to their
long name on Linux and MacOS.
Names are shortened on Windows to avoid exceeding MAX_PATH.
* Script to generate scala test name mapping
* Generate scala-test-suite-name-map.json on Windows
changelog_begin
changelog_end
* Generate UTF-8 with Unix line endings
Otherwise the file will be formatted using UTF-16 with CRLF line
endings, which confuses `jq` on Linux.
* Apply Scala test name remapping before ES upload
* Pipe bazel output into intermediate file
Bazel writes the output of --experimental_show_artifacts to stderr
instead of stdout. In Powershell this means that these outputs are not
plain strings, but instead error objects. Simply redirecting these to
stdout and piping them into further processing will lead to
indeterministically missing items or indeterministically introduced
additional newlines which may break paths.
To work around this we extract the error message from error objects,
introduce appropriate newlines, and write the output to a temporary file
before further processing.
This solution is taken and adapted from
https://stackoverflow.com/a/48671797/841562
* Add copyright header
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Augment completion.proto with deduplication-related info
CHANGELOG_BEGIN
CHANGELOG_END
* Explicitly specify fields not yet filled in when building Completion
* Time-based deduplication periods are measured in record time of completions
* Add deduplication_offset as a deduplication_period option
* Don't skip proto field numbers
* CompletionFromTransaction: use default Completion constructor
* submission_rank: reserve proto field for future use
* Add comment about reserved proto field
Previously, if the max deduplication time was extended, the participant
_might_ retain the old time for certain submissions.
CHANGELOG_BEGIN
- [Ledger API Server] The API server manages a single command tracker
per (application ID × submitters) pair. This tracker would read the
current ledger configuration's maximum deduplication time on creation,
but never updated it, leading to trackers that might inadvertently
reject a submission when it should have been accepted. The tracker now
reads the latest ledger configuration.
CHANGELOG_END
* Time conversion duration between buffer event and API domain transaction
* Compute partiesTemplates inversion mapping outside events transformation
* Other small optimizations
CHANGELOG_BEGIN
CHANGELOG_END
* Addition of a key_hash field to speed up fetchByKey queries
CHANGELOG_BEGIN
CHANGELOG_END
* changes to make key_hash and Optional field
CHANGELOG_BEGIN
- Update schema version for http-json-api query store with new key_hash field
- Improved performance for fetchByKey query which now uses key_hash field
CHANGELOG_END
* remove btree index for postgres and other changes based on code review comments
This PR enforces that LF Comparisons (both Ordering and Equality)
fails at runtime if comparing local vs global CID.
This corresponds to the second task of #10504.
CHANGELOG_BEGIN
CHANGELOG_END
The current README dates back to when `create-daml-app` was its own
repo, and seems mostly written from the perspective of being the default
home page for the project. These days, `create-daml-app` serves two
purposes:
1. As the core starting point for the Getting Started Guide, we can
assume people who are totally new to Daml will be following the
getting started guide. How else would they even know about this
template? In that case, I think it's fair to assume the user will *not*
be looking to the README for guidance. Therefore, repeating the
documentation here seems a bit pointless.
2. As a template to quickstart the creation of a "custom" app that is
not the GSG. For that use-case, the README is the place where people
would look for information, but at that point the information they're
looking for are more along the lines of quick refreshers of what
commands to run than explanations on how to use the app as Alice and
Bob.
CHANGELOG_BEGIN
CHANGELOG_END
* Add StorageBackend tests
changelog_begin
changelog_end
* Fix Oracle tests
* Do not use empty byte arrays
* Format
* Fix after rebase
* Substitute type params with type bounded ingest method
* Remove empty line
* Assert on configuration contents
* Fix Oracle build
* Add tests for ingestion initialization
* fmt
* Add test for leftover data after reset
* Add resetAll
* Use resetAll between tests
Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
* participant-integration-api: Add logging to RecoveringIndexerSpec.
It's flaky, and I would like to know exactly what's going on here.
* participant-integration-api: Attempt to fix RecoveringIndexerSpec.
The checks for the health status are racy. I'm hoping increasing the
timeouts will help a little.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Inline timeouts in RecoveringIndexerSpec.
The `LedgerConfigurationSubscriptionFromIndexSpec` was flaky due to
stubs not specifying all behavior for `IndexConfigManagementService`.
This fixes the underlying issue by avoiding stubs in favor of fakes,
which means that _all_ behavior must be modelled.
Note: Martin Fowler has an excellent, terse description of [the various
forms of test doubles][TestDouble].
[TestDouble]: https://www.martinfowler.com/bliki/TestDouble.html
CHANGELOG_BEGIN
CHANGELOG_END
* Update the channels link in Upgrading.md
* Update the nixpkgs-unstable commit
CHANGELOG_BEGIN
CHANGELOG_END
* Fix new `hlint` warnings.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Reminder to put and empty line between subject and body
changelog_begin
changelog_end
* Update ci/check-changelog.sh
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Upgrade to a newer canton version (post 0.27.0 snapshot version)
with canton-community configuration that supports higher throughput.
changelog_begin
changelog_end
* Disable flaky reject DeeplyNestedValueIT:Reject tests that time out half the time
As stated in #10504 the contract ID freshness check cannot be
implemented correctly in general.
This PR drops the support for this (buggy) check.
This corresponds to the fist task of #10504.
CHANGELOG_BEGIN
CHANGELOG_END
This was limited a while back during the initial development of
_ledger-on-sql_ for reasons I can't remember. Let's stop doing that.
CHANGELOG_BEGIN
CHANGELOG_END
* Refactor ParameterStorageBackend
- a single method for atomic ledger initialization
- a single method to look up the ledger end
changelog_begin
changelog_end
* Add a test
* Fix reading event sequential ids
* Remove debug statements
* Allow ledgerEnd on an empty database
* Initialization is not safe to call concurrently
* Remove leftovers from isolation level change
* Use unit return type
for initialization methods
* Allow getParticipantId on an empty database
* Use exceptions instead of a return type ADT
* Don't use Try for initialization
* Clean up parameters table
* Simplify parameter storage api
* Address review suggestion
* Address review comment
* Address review comment
* Prefer ledger id over participant id mismatch
* Address review comment
* Move type definition
* Remove useleess new keyword
* Renove unused import
* Inline result mapping
* Fix reporting of mismatching participantId
* participant-integration-api: Construct completions in one place.
* sandbox-classic: Inline `CompletionFromTransaction#apply`.
It's only used here; there's no reason to keep it in the
_participant-integration-api_.
* participant-integration-api: Store a status gRPC protobuf.
Instead of storing the status code and message, we store a serialized
`google.rpc.Status` protocol buffers message. This allows us to pass
through any additional information reported by the driver `ReadService`.
The migration is only done for the append-only database, and preserves
old data in the existing columns. New data will only be written to the
new column.
CHANGELOG_BEGIN
CHANGELOG_END
* participant-integration-api: Improve comments in migrations.
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* participant-integration-api: Further improvements to migrations.
* participant-integration-api: Store the rejection status as 3 columns.
Serializing the details but keeping the code and message columns
populated.
* participant-integration-api: Publish the indexer protobuf to Maven.
Co-authored-by: Fabio Tudone <fabio.tudone@digitalasset.com>
* Use `extra` in the port file runner, rather than `temporary`.
* ledger-api-test-tool-on-canton: Use the port check runner.
Much simpler than the port file runner for our purposes.
* Replace `runner` with `runner_with_port_file`.
Rather than expecting a particular set of command-line-arguments, we use
templating.
CHANGELOG_BEGIN
CHANGELOG_END
* Rename the `runner_with_port_check` target to the default.
* Use the port file and dynamic port generation in client/server tests.
This creates a runner named `runner_with_port_file` which knows how to
interpolate two variables, `%PORT_FILE%` and `%PORT%`. This allows us to
use the `port-file` argument to the kvutils runner rather than
hard-coding a port for conformance tests.
For now, we only use this for generating the kvutils reference ledger
export.
CHANGELOG_BEGIN
CHANGELOG_END
* Simplify the runner_with_port_file considerably.
It doesn't need to check if the port is open; we trust that the process
will do it.
This also makes sure the port file will be cleaned up, and reduces the
number of dependencies by making use of more functions in `extra`.
* Simplify port file generation in the new client-server runner.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Simplify the runner_with_port_file further.
This doesn't need to work if the server doesn't take a port file.
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
