* [LLP] Allow Indexer to be back-pressured
* Introduce buffer after Indexer
* Remove Indexer conflation
changelog_begin
changelog_end
* Batch in tailer to predefined limit
* Reformatted and updated tests
* bump canton to 20220803
CHANGELOG_BEGIN
CHANGELOG_END
* adjust bootstrap script to changes in Canton console commands
* use canonical dot syntax
Co-authored-by: Azure Pipelines Daml Build <support@digitalasset.com>
Co-authored-by: Andreas Lochbihler <andreas.lochbihler@digitalasset.com>
* Add DA_Internal_Interface to convertTypeDef exclusion check
* Move desugared types/values for interface views into DA.Internal.Desugar
* Convert viewtype and view method declarations
* Typecheck view types and view declarations
* Add unit viewtypes to InterfaceChoiceCollision tests
* Add unit viewtypes to more compiler/damlc tests
* Modify empty interfaces/implements to have viewtypes/views
* Add unit viewtypes to remaining Interface tests
* Test for error thrown when viewtype not specified
* Fix daml-lf/ interface tests to use new syntax / unit views
* Add placeholder view of type Unit to TestInterfaces
CHANGELOG_BEGIN
CHANGELOG_END
* Fix typescript interface tests with unit viewtype
* Add unit viewtype to Java & Scala codegen
* Add unit viewtype to triggers tests interface
* Add unit viewtypes to ledger/test-common
* Add unit viewtypes to ledger-service/http-json
* Fix some damlc tests
* Use viewtype syntax in InterfaceViewNonSerializable test
* Remove HasInferfaceView instances in convertBind
* Add unit view to QualifiedInterface.daml test
* Generate HasInterfaceView instances in DataDependencies
* Add unit viewtypes and views to compiler/damlc/tests
* Document reconstruction of HasInterfaceView
* Move desugared types/values for interface views into DA.Internal.Desugar
* Add desugarable function view w/ EViewInterface desugaring
* Remove templateName and viewtype from generated EViewInterface
* Update desugared-daml
* Proof of concept: use _view -> EViewInterface prim via HasInterfaceView
* Move view, HasInterfaceView to DA.Internal.Interface
* Update desugared-daml tests
* Stop removing and re-generating HasInterfaceView instances
* Do not generate docs for generated "_view_" methods
* Report errors with interface TypeConNames prettily
* Fix InterfaceViewNotSpecified @ERROR to reflect new error text
* Don't import magic `view` method from Prelude to avoid name clash
* Update shake interface goto definition test locations
* Temp disable view spec tests, will re-implement views in speedy
* Add unit viewtype to Java code snippet docs
* Update Interface syntax docs for views
* Add unit viewtype to Transferrable.daml
* Add back "empty" interface implementation to interfaces.rst
* Re-enable view tests by having _view methods on InterfaceViews.daml file
* Remove deprecated commented-out view desugaring code
* Document all options in the HOCON config bundle [DPP-1121]
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Tudor Voicu <tudor.voicu@digitalasset.com>
Co-authored-by: Marton Nagy <marton.nagy@digitalasset.com>
Co-authored-by: Simon Maxen <56595114+simonmaxen-da@users.noreply.github.com>
* Rename TransactionLogUpdate.Transaction to TransactionLogUpdate.TransactionAccepted
changelog_begin
changelog_end
* Implement command completions serving from in memory fanout
changelog_begin
changelog_end
* Refactor to use only one BufferedReader
* Renaming and docs
* Fix compilation issues
* Address review comments
* Adds retryGuard to retry mechanism
* Only allow retrying in lock-busy condition
[CHANGELOG_BEGIN]
Fixes issue: network problems can prevent non-leader indexer to be elected as leader.
[CHANGELOG_END]
* Adds explicit network timeout to main lock acquisition calls.
* Adds mainLockCheckerJdbcNetworkTimeoutMillis configuration with default
[CHANGELOG_BEGIN]
Fixes issue: network problems can cause indexer to stop working until restarting the application.
[CHANGELOG_END]
Populating IndexDB with a set of parties:
```
submission:
...
observers_party_set:
party_name_prefix: MyParty
count: 99
visibility: 0.35
```
where the parties created will be Party-0, Party-1, .., Party-98
and each will have a 35% chance of being an observer on a Foo1/Foo2/Foo3 contract.
NOTE: This PR is not adding party-set support for divulgee parties (i.e. non-stakeholder informees) since a large number of such parties might cause an exponential explosion in the number of created helper "Divulger" contracts. It's also not clear if support for divulgee party-sets will be needed after all.
Using party set in a stream filter:
```
streams:
- type: transaction-trees
name: foo
filters:
- party: Obs-1
templates: [Foo1, Foo2, Foo3]
filter_by_party_set:
party_name_prefix: MyParty
templates: [ ]
...
```
changelog_begin
changelog_end
* Move disclosed contract logic to speedy
This moves the disclosed contract logic from the engine to speedy.
Fixes#14090.
* back to versioned contract instances
CHANGELOG_BEGIN
CHANGELOG_END
This a preparation before the adding support for huge party filters.
Some of the changes:
1. Scan for existing parties even if when there is not submission step configured so that we can enrich stream configs fully.
NOTE: ConfigEnricher will now never fail on missing party.
2. Extract party selection for contract creation to a separate class.
3. Expose `commandGenerationParallelism` and a couple of randomness providers to allow for reproducible party allotment to contracts - useful in tests.
4. Extract `benchtoolFixture` to reduce boilerplate in IT tests
changelog_begin
changelog_end
Recent changes in //ledger/test-common in Daml modules changed the fully qualified names of templates used by Benchtool, even though they hadn't been changed directly.
This made it impossible to filter by template name in a recently created LR IndexDB snapshot.
To avoid spurious changes in fully qualified template names (from Benchtool perspective) we are now packaging Benchtool related Daml modules separately from other modules
changelog_begin
changelog_end
* bump canton to 20220617
CHANGELOG_BEGIN
CHANGELOG_END
* activate conformance test for regression test for exceptions
Co-authored-by: Azure Pipelines Daml Build <support@digitalasset.com>
Co-authored-by: Remy Haemmerle <Remy.Haemmerle@daml.com>
This is a follow up of #14175 and #14176.
Basically
- we kill the `On` mode which is not used anymore.
- we factorize the visibility check.
- we rename `LocalContractNotVisible` into `ContractNotVisible`
CHANGELOG_BEGIN
CHANGELOG_END
Correct metrics documentation shipped with the sdk
* Remove references to obsolete metrics for participant-side de-duplication
* Correct the names of the connection pool metrics
* Add a mention about internal service metrics
* Correct the name for the string interning metric
* Remove obsolete sandbox server role
CHANGELOG_BEGIN
Correct the name for the string interning metric from daml_index_db_loadStringInterningEntries to daml_index_db_load_string_interning_entries
CHANGELOG_END
* Add missing activeness check when reading from globalKeyInputs
changelog_begin
- Engine. Fix lookup by key. See issue Lookup archived contracts may succeed when combined with rollback exceptions #14107 for more details.
changelog_end
* Allow the possibility to have a shared StringInterningView
changelog_begin
changelog_end
* Solve TODO in RecoveringIndexerIntegrationSpec
* Addressed Marton's review comments
* Added thread-safety mentionsin string interning interfaces
* inherited choices are gone, convert to interface
* use toInterface for conformance tests on interfaces
CHANGELOG_BEGIN
- [Scala codegen] Interface choices can no longer be invoked directly on
template IDs, or via ``createAnd`` or ``key`` directly. Instead, use
``toInterface[Ifn]`` before calling the relevant ``exercise*`` method.
The resulting ledger commands now contain the correct interface ID
rather than template ID for looking up the choice, but see #13993 for
a caveat regarding create-and-exercise and exercise-by-key.
CHANGELOG_END
* Adds exponential ID page retrieval to aid low stream initialization
* Adds Working-Memory concept to calculate parameters to aid guaranteed memory footprint
* Switches to dynamic BatchN batching instead of static grouping to aid stream initialization
changelog_begin
changelog_end
The field globalKeyMapping is recently added to
lf.transaction.Transaction.Metadata, and which is
computed by the Engine.
This field might be needed for processing by WriteService
implementors.
This PR adds this to the WriteService.
changelog_begin
Global contract key mapping is added to WriteService.
changelog_end
Non-stakeholder informees for consuming events are obtained by exercising a consuming choice on Foo1, Foo2 or Foo3 contract as a sub-action of divulgers newly introducedDivulgeConsumingExercise choice.
changelog_begin
changelog_end
* disclosures: support in engine
This adds support for contract disclosures in the engine.
Fixes#13863.
CHANGELOG_BEGIN
CHANGELOG_END
* move VersionedContractInstance to Value
* normalization/typechecking
* Update daml-lf/engine/src/main/scala/com/digitalasset/daml/lf/engine/Error.scala
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Update daml-lf/engine/src/main/scala/com/digitalasset/daml/lf/engine/preprocessing/CommandPreprocessor.scala
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Update daml-lf/engine/src/main/scala/com/digitalasset/daml/lf/engine/preprocessing/CommandPreprocessor.scala
Co-authored-by: Remy <remy.haemmerle@daml.com>
* remove processImmArray
* throw exception in foldl
* check for duplicate contract keys in disclosures
* added issue for removal of empty disclosure assertion
* format
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
Simplifies ID fetching logic
Removes dead code / tests
Cleans up factories (Materializer not needed anymore)
Simplifies ConcurrencyLimiter interface
Adds/cleans up unit tests
CHANGELOG_BEGIN
CHANGELOG_END
* add tests for large maps and lists
* add many iterations on exercise
* format
* clean up
CHANGELOG_BEGIN
CHANGELOG_END
* fix broken build
* Incorporate review comments
Previous approach was generating random bytes
and encoding them into UTF-8 strings.
This turned out to be problematic as not every
random byte sequence is valid in the UTF-8 charset, and
`java.lang.String#String(byte[], java.lang.String)` constructor's
documentation states that in such cases its behavior
is unspecified.
In practice the size of the byte array obtained from decoding the
generated string was almost twice the size of the orginal byte array.
Current approach limits the generated bytes to valid ASCII characters,
each of which takes exactly one byte in UTF-8 encoding.
changelog_begin
changelog_end
We obtain non-empty non-stakeholder informees by performing an immediate divulgence of instances of Foo1, Foo2 or Foo3 templates through instances of a helper Divulger template.
The divulgence is controlled by 1) configuring the number of all divulgees to generated and 2) probabilistically picking a non-empty subset from all divulgees for each contract (similar to how observers are picked for each contract with certain probabilities). If the divulgees for a contract are non-empty a Foo1, Foo2 or Foo3 contract is created via a helper choice on a Divulger contract. Otherwise a standard create Foo1, Foo2 or Foo3 command is issued.
Each Divulger contract can divulge any number of Foo1, Foo2 or Foo3 contracts but it's limited to divulging to a fixed set of divulgees. In other words, there is a separate Divulger contract for each non-empty subset of all divulgees.
We expect the set of all divulgees to be very small in practice similar to how only a small number observers (currently at most 3) is currently configured for existing benchmarks. Thus, the set of all subsets is also small.
To be conservative, we allow the number of divulgees to be at most 5.
changelog_begin
changelog_end
- Wait for confirmation for all submissions when setting time model
- Do not ignore authorization errors, this is done so that we are consistent across tests (exclusions should be used if required)
changelog_begin
changelog_end
Previously a fire and forget release process
was implemented, which in corner cases could
result in exceptions, as API stream was still running after DbDispatcher was released.
This PR changes to waiting for all the streams to finish as releasing Dispatcher (with a timeout of 5 seconds).
CHANGELOG_BEGIN
CHANGELOG_END
Fixes Index DB insertion order for childEventId.
Adds backwards compatibility treatment: recovering childEventId
order from the order of events at API stream rendering.
Extends TransactionServiceVisibilityIT with the ordering criterion.
Fixes compatibility tests with exclusion.
Adds documentation to event.proto.
CHANGELOG_BEGIN
Fixing Ledger API Bug: Exercise nodes in transaction trees
have child_event_ids out of order.
CHANGELOG_END
This PR fixes an issue with the ACS ID page retrieval: if there is one
of the id page DB fetching is slow, all the rest of the ID fetching
lanes getting fetched, due to continuous demand.
We implemented a treatment to cap the increased number of ID pages
accumulated in memory, by failing the stream.
Switching to mapAsync in pullWorkerSource naturally will
stop demand, in case a single query takes too long.
This also removes the need for the queue size limitting treatment.
* Switches to mapAsync in pullWorkerSource
* Fixes unit test
* Removes queue limiting feature (as it is not necessary anymore)
CHANGELOG_BEGIN
CHANGELOG_END
* ifaces:support exercising by required interface
This adds support to exercise an interface choice on a contract ID,
where the interface is required by one of the implemented interfaces of
the contract template.
Fixes#13434.
CHANGELOG_BEGIN
CHANGELOG_END
* some improvements based on review
* added a test plus a bugfix
* Update daml-lf/interpreter/src/main/scala/com/digitalasset/daml/lf/speedy/PhaseOne.scala
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
* 2 more interface tests for the command preprocessor
Co-authored-by: Sofia Faro <sofia.faro@digitalasset.com>
* Use event partitions directly
changelog_begin
changelog_end
* it's more complicated
* fix query
* don't use divulgence to look up events
* fix query
* Remove commented out code
* Fix query
* fmt
* Actually remove the events view
at least on Postgres
* Remove view from Oracle
* Remove view from H2
* Fix hash
* fix boolean constants for oracle
* Add new test
This test currently fails on Oracle
changelog_begin
changelog_end
* Fix deleting partial data
... if the indexer crashes before writing the first ledger end
* Adding section to error codes page and updating descriptions
CHANGELOG_BEGIN
CHANGELOG_END
Added a section in the error codes page and improved description to errors
* Removing the multi-line descriptions
* Changing the added sentence
swapping 'due to security reasons' 'for security reasons'
* Updated formatting
Ran the fmt.sh script to give me the correct formatting.
* Updating the generated error codes
CHANGELOG_BEGIN
CHANGELOG_END
Updating the generated error codes
* Implemented race conditions test for mutable contract state cache upload
changelog_begin
[Ledger API] A race condition bug is fixed in the Ledger API mutable contract state cache.
The bug allowed the cache to become corrupted with stale references when the Ledger API
was subjected to concurrent submissions racing to update/fetch the same contract key.
The stale references were causing the Ledger API to erroneously reject submissions in
certain corner-cases.
changelog_end
* Pull cache indexes to StateCache
* Passes the test
* Fixing StateCacheSpec and MutabelCacheBackedContractStoreSpec
* Added invalidate test to ConcurrentCacheCachingSpecBase
* Address Marton's review comments
* Do not log a warn on negative contract lookups
* Update cache outside the source restart
* Move ExecutionContext to the CommandExecutor implementing classes from method arguments
CHANGELOG_BEGIN
CHANGELOG_END
* Added information about the CPU-intensive computations executed by an ExecutionContext
1. Remove `BaseError.Impl`, `LoggingTransactionErrorImpl` and `LoggingPackageServiceError`
and instead provide more direct `DamlError` and `DamlErrorWithDefiniteAnswer`.
2. Remove custom implementation of `TransactionError.rpcStatus` and instead provide simpler one in `DamlErr.r.rpcStatus` (which works by first calling `code.asGrpcStatus` and then converting the result to `com.google.rpc.status.Status`).
3. Remove `GrpcStatus.toProto` and instead use `DamlError.rpcStatus`.
4. Use `asGrpcStatus` and `asGrpcError` instead of `asGrpcStatusFromContext` and `asGrpcErrorFromContext` where possible.
changelog_begin
changelog_end
And add a check (a unit test) that the generated files are up-to-date with what can be generated.
If the check fails then one will need to run `./docs/scripts/gen-error-docs-src.sh` and commit the changes.
changelog_begin
changelog_end
* Re-enable some sandbox tests on MacOS
changelog_begin
changelog_end
* Remove the use of eventually where it is not applicable
changelog_begin
changelog_end
* formatting
changelog_begin
changelog_end
* Revert "formatting"
This reverts commit d2f4bfce1b.
* Revert "Remove the use of eventually where it is not applicable"
This reverts commit 74f0ec5517.
run-full-compat: true
* evantually removal
run-full-compat: true
* Revert "Re-enable some sandbox tests on MacOS"
This reverts commit 5a1544953b.
* Adding comments
changelog_begin
changelog_end
* Update ledger/ledger-api-tests/infrastructure/src/main/scala/com/daml/ledger/api/testtool/infrastructure/Allocation.scala
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
* remove blank lines
changelog_begin
changelog_end
Co-authored-by: Robert Autenrieth <31539813+rautenrieth-da@users.noreply.github.com>
We split kv replay as follow:
- [LF] we introduce a simple protobuf to store submission entry (which
roughly correspond to TRANSACTION_ENTRY and PACKAGE_UPLOAD_ENTRY KV
submission)
- [KV] we create a tool to extract form a ledger export a sequence of
submission entry
- [LF] we move the replay tool from KV to LF, and base it on
submission entries instead of ledger export
The objectify of this split is double:
- on the one side it will simplify maintenance, as it will make API
between KV and LF more clear (depends only on a protobuf definition)
- on the other side, it will made the use of the tool handy, as it
will make possible to benchmark the engine without direct dependency
between KV and LF.
CHANGELOG_BEGIN
CHANGELOG_END
* Improve MessageDigest and Mac instance creation to solve lock contention problem [DPP-956]
changelog_begin
Scalability bottleneck in regard to hashing has been fixed in multiple places.
changelog_end
* Make no-op metering logging debug
* Address statistic defaulting TODOs
* TODO replaced with ticket
* Switch to using assertLedgerMeteringEnd when ledgering end must be populated
* Note in docs that ledger metering is not affected by participant pruning
* Address review comments leftover from DPP-819
* Switch to using cSQL for options and offsets
changelog_begin
changelog_end
Process:
- `git ls-files -z | xargs -0 -n 100 sed -i --follow-symlinks 's/DAML/Daml/g'`
- `git add -p`
- `git restore -p`
- Check there is no unstaged change left.
To review:
- Check for false positives by carefully reviewing the diff in this PR.
- Check for false negatives with `git grep DAML`.
- Quicker check for fals positives:
```
git grep DAML | grep -v migration | grep -v DAML_
```
Fixes#13190
Note: This is the "second half" of #13191, which failed to cover all the
remaining DAMLs because of:
```
$ git ls-files | grep "'"
compiler/damlc/tests/daml-test-files/MangledScenario'.daml
```
CHANGELOG_BEGIN
CHANGELOG_END
changelog_begin
Ledger API Specification: Streaming endpoints will use error code 'SERVER_IS_SHUTTING_DOWN' when the server
is being shut down, whereas before an exception without an error code was used.
Grpc status code in use remains 'UNAVAILABLE'.
changelog_end
In SandboxFixture when mixing in the auth middleware fixture, set
up the ledger client with a user token instead of a claims token
when not running in the claims-token-specific auth tests.
Fixes#12831.
* note about inClaims
* prevent Product, Serializable warts with AuthServiceJWTPayload
CHANGELOG_BEGIN
CHANGELOG_END
