* Port //daml-lf/interpreter to Scala 2.13
For now the perf tests are left out since they depend on a DAR built
by damlc which depends on daml script which depends on the world
:exploding-head:
changelog_begin
changelog_end
* Scala 2.13-style to for ImmArray and FrontStack
changelog_begin
changelog_end
* Avoid extra conversion
changelog_begin
changelog_end
CHANGELOG_BEGIN
* Release preview of LF 1.11. Preview versions can be changed only to
include bug fixes. Changes of LF 1.11 include:
- add support for generic map;
- add support for choice observers;
- reduce archive size thanks to type interning.
CHANGELOG_END
* Port //daml-lf/(parser|validation) to Scala 2.13
changelog_begin
changelog_end
* Rename (Expr|Type)Traversable to (Expr|Type)Iterable
changelog_begin
changelog_end
* LF: Scala type checker handle exceptions
This includes some fixes of the DAML-LF specifications.
This is part of #8020.
CHANGELOG_BEGIN
CHANGELOG_END
As we strive for more inclusiveness, we are becoming less comfortable
with historically-charged terms being used in our everyday work.
This is targeted for merge on Dec 26, _after_ the necessary
corresponding changes at both the GitHub and Azure Pipelines levels.
CHANGELOG_BEGIN
- DAML Connect development is now conducted from the `main` branch,
rather than the `master` one. If you had any dependency on the
digital-asset/daml repository, you will need to update this parameter.
CHANGELOG_END
* Port //daml-lf/data to Scala 2.13
changelog_begin
changelog_end
* factor common ImmArraySeq code to version-agnostic file
- ImmArraySeq itself is agnostic; the 2.12 and 2.13 versions contain
implementation mixins/superclasses for parts that must be specific. The 2.13
version will collapse into the agnostic version when 2.12 support is no longer
desired
* factor common InsertOrdMap code to version-agnostic file
- InsertOrdMap itself is agnostic; the 2.12 and 2.13 versions contain
implementation mixins/superclasses for parts that must be specific. The 2.13
version will collapse into the agnostic version when 2.12 support is no longer
desired
* factor common InsertOrdSet code to version-agnostic file
- InsertOrdSet itself is agnostic; the 2.12 and 2.13 versions contain
implementation mixins/superclasses for parts that must be specific. The 2.13
version will collapse into the agnostic version when 2.12 support is no longer
desired
* factor Map removal
* Move ImmArraySeq back into ImmArray
changelog_begin
changelog_end
* Type assertion instead of symbol
changelog_begin
changelog_end
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* Update protobuf to throw exceptions directly.
Part of #8020. This PR changes the exception protobuf and AST (Haskell
side) to make exceptions be thrown directly via a primitive expression
(EThrow), instead of wrapping them up via AnyException.
changelog_begin
changelog_end
* Rename MakeAnyException to ToAnyException
* update EToAnyException field names
* Missing stuff
* missing scala case
* Make AnyException unserializable
* reindex protobuf builtins
* meaningless change
* refrobulate
* change pretty
* Make choice observers mandatory when available.
This is an invariant of the DAML-LF proto.
The haskell AST allows missing choice observers for any version, because the AST represents what was written in the syntax. However, if we are targeting a new DAML-LF version (>= `featureChoiceObservers`) then `Nothing` is converted to `Just (ENil TParty)` by the haskell encoder (`EncodeV1.hs`) to satisfy the proto invariant.
On the scala side, the decoder (`DecodeV1.scala`) now insists that choice observers are present when_ the version is new. (>= `LV.Features.choiceObservers`).
changelog_begin
changelog_end
* LF: LF encoder generate always choice observer if lf >= 1.dev (#8318)
This is a follow up of #8316.
This advances the state of #7709
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Upgrade Scala dependencies for 2.13 compatibility
This upgrades a bunch of Scala libraries to versions that have 2.13
support. There are two libraries that are still missing:
- diffson, this has a new version but with significant breaking
changes and it is only used in Naigator console which I hope to kill
before I have to worry about this.
- ai.x:diff, this is used in the ledger API test tool. The library is
abondened but there are a few alternatives.
changelog_begin
changelog_end
* Fix pureconfig
changelog_begin
changelog_end
* Fix Navigator
changelog_begin
changelog_end
We use the constants defined in LanguageVersion/TransactionVersions instead
repeating the somewhat magic constants "6", "7", "8", "10" and "dev" in multiple places?
CHANGELOG_BEGIN
CHANGELOG_END
We drop the distinction (at the type level) of Dev and Stable language
version. The two main reason that motivate this choice:
* we never really used this distinction.
* we want to add the concept of "preview" version (which is neither Dev nor Stable)
CHANGELOG_BEGIN
CHANGELOG_END
Changes:
- replaced submitter: Party with actAs: Set[Party] and readAs: Set[Party] in com.daml.ledger.api.domain.Commands
- adapted contract and key lookups for multi-party submissions
* Changed single Commands to accept multiple submitters
* Removed submitter field from Commands
* Replaced submitters with actAs and readAs in Commands
* Moved actAs and readAs to com.daml.ledger.api.Commands
* Contract lookup tests
* Key lookup tests
CHANGELOG_BEGIN
CHANGELOG_END
* Formatted code changes
* Remove unused method
* Formatting fixes
* Simplified logical condition for active contracts visibility
* Removed unused temporary method in Commands
* Unified contract key lookup query for different type of db types
* Simplified ContractReader class structure
* Improved dao unit tests
* Unit tests for lookups as a divulgee
* Reduced code duplication in unit tests
* Minor improvement
* Formatted code
* Fixed a ContractsReader's query
* Improved unit tests
* Explanation for engine's command authorizers and contract readers
* Minor fixes
* Adapted SpannedIndexService for the multi-party submission
* Improve error messages on use of contract ids in contract keys
We got some feedback that "Unexpected contract id" sounds like an
internal error whereas this is really a user error.
changelog_begin
changelog_end
* Include templateId in error message
changelog_begin
changelog_end
With this PR, the Repl now:
* outputs the transaction version (not sure why it was not
there before)
* outputs the node version, because each node is now versioned
independently (See #7788)
* drops the value version, because it is dictated by the node version
(See #7788)
CHANGELOG_BEGIN
CHANGELOG_END
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.
This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.
Apologies for the giant PR, I don’t see a way to keep it smaller.
changelog_begin
changelog_end
We have a builtin SBConsMany use to concatenate a list with a prefix
of variable size. Its appear that most of the concatenation happens
with only one element. For this reason we add a new builtin to handle
the most efficiently possible the case with 1 element.
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
- [DAML-LF] Some types that were deprecated in DAML 1.4 have been
removed. Please fix any deprecation warnings by using the new types
before upgrading.
CHANGELOG_END
* Bazel: Reduce the visibility of some Protobuf targets.
* Bazel: Make Protobuf JAR visibility parameterizable.
Default to private, and explicitly make it public where it's needed.
CHANGELOG_BEGIN
CHANGELOG_END
* Tail-recursive function definitions execute with bounded environment and continuation stacks.
changelog_begin
changelog_end
* be more precise about the throw exception; use thrownBy
* add test for the KFoldr1Map/Reduce case
* Multiple submitters field for Command
* Multiple submitters field for Engine API
* Tests for multi-party submissions in Engine
* Make use of multi-party Engine capabilities in the engine benchmark
* Enable multi-party submission in Engine's API
CHANGELOG_BEGIN
CHANGELOG_END
* Separate unit tests for multi-party engine
* Unit tests for multi-party submission transaction replay
* ledger-api: Use `proto_jars`.
CHANGELOG_BEGIN
- [Ledger API] The Scala JARs containing the gRPC definitions no longer
contain the *.proto files used to generate the ScalaPB-based classes.
CHANGELOG_END
* Create a source JAR for *.proto files in `proto_jars`.
* ledger-api: Publish the protobuf sources as "ledger-api-proto".
CHANGELOG_BEGIN
- [Ledger API] The *.proto files containing the gRPC definitions are now
provided by a new Maven Central artifact, with the group "com.daml"
and the artifact name "ledger-api-proto".
CHANGELOG_END
* release: We don't need the "main-jar" option.
* Bazel: Proto JARs will always have a Maven artifact suffix.
* Bazel: Simplify Protobuf source file TAR and JAR targets.
* Bazel: Extract out Protobuf functions.
String validators from com.daml.lf.data.IdString are use extensively.
This PR optimize the implementation of some of those.
CHANGELOG_BEGIN
CHANGELOG_END
* WIP Draft of formal exception semantics
CHANGELOG_BEGIN
CHANGELOG_END
* Make throw take an AnyException
* Don't allow templates as exceptions
* Make throw a builtin
* Pass the textual error message to to_any_exception
* Rename builtin exceptions to errors
* Rename to_any_exception in make_any_exception
* Fix typing rules for make/from_any_exception
* Add availability notes
* Fix typo
* simplify interface to construct speedy continuations
move code from caller to definition
* relocate comment to the relocated code it refers to!
changelog_begin
changelog_end
* rename var: m -> machine
* relocate instrumentation print() from where KFinished is thrown, to where it is handled in run()
* revert KFinished back to an object (instead of a class taking machne); and revert back to use initialKontStack()
Turns out the `package_dir` is not what we're looking for. Omitting it
fixes the path.
_Before:_
```
1766 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_8/com/digitalasset/daml_lf_1_8/daml_lf.proto
```
_After:_
```
1766 2010-01-01 00:00 protos-0.0.0/com/digitalasset/daml_lf_1_8/daml_lf.proto
```
CHANGELOG_BEGIN
CHANGELOG_END
* kvutils: Use ScalaPB to generate a Scala JAR for daml_kvutils.proto.
* Bazel: Delete the unused `da_java_binary` rule, and inline `_wrap_rule`.
* Bazel: Factor out Java/Scala protobuf class generation into a helper.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lf/archive: Use `proto_jars`.
* Bazel: Remove the visibility modifier from `proto_jars`.
It's too confusing. Just make everything public.
* daml-lf/archive: Push protobuf source tarballs into `proto_jars`.
* Bazel: Add comments to the various parts of `proto_jars`.
* daml-assistant: Do unpleasant things with `location` in Bazel.
As requested by Martin in #8069, we extract most of the changes from
DecodeV1Spec in a separate PR.
This advances the state of #7155
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
kvutils reports LookupByKey node mismatches during validation as Inconsistent
instead of Disputed if they can be due to contention on the contract key
CHANGELOG_END
* Refactor indexing to minimize transaction traversals
changelog_begin
[Integration Kit] Bugfix: daml.index.db.store_transaction metrics were keeping track of empty insertions, skewing down the numbers
[Integration Kit] Performance: minimizing the number of traversals of a transaction to index it, more efficient indexing
changelog_end
* Add empty stakeholders for non-consuming exercises
* Fix test to check usage of pre-computed blinding info
* Remove unnecessary string interpolators
* Shorten creation of net visibility
* Delete obsolete comment
* Add visibility of divulged contracts
* Address review https://github.com/digital-asset/daml/pull/8019#discussion_r527666572
* Address review https://github.com/digital-asset/daml/pull/8019#pullrequestreview-535399441
* Address review https://github.com/digital-asset/daml/pull/8019#discussion_r527681774
* Fix ordering of batches, use explicitly named types to avoid confusion
* Default to an empty set if a divulged contract is not visible in the current transaction
* Shorten references to store transaction metrics when writing
* Address review https://github.com/digital-asset/daml/pull/8019#discussion_r527745640
* Add divulgences from past transactions
* More fine-grained types and signatures
Add a description of how to enable LaTeX input mode when editing the
DAML-LF spec in VS Code. Also add some additional symbols.
CHANGELOG_BEGIN
CHANGELOG_END
* Adapt to new desugaring for choice observers.
update hash of ghc patch.
changelog_begin
changelog_end
update ghc patch to final version
update stack-snapshot hashes for ghc-lib(-parser)
update stackage_snapshot.json, following `bazel run @stackage-unpinned//:pin`
expose Optional constructors for desugared code to use
adapt LFConversion to expect a 4-tuple for a desugared choice def/sig
update LFConversion for choice-observers
first example using new choice observer syntax.
fix scala type checker to have correct scoping rules for choice-observers
remove comment from example which says it is broken
improve variable names
improve tests for choice-observer clause
only test choice-observers SINCE 1.dev
add jq queries for choice observeres
make positive statement in jq test which checks choice obserers are present
test behaviour of choice observers
squash me
typo
* test more choice-observer divulgence
* Update documention for choice observers.
changelog_begin
Support choice observers in 1.dev
changelog_end
* fix docs build
* fix daml docs choice-observers example
* address comments: rewording text
* annotate choices observers as early-access in documention
* split out documentation code-snippets which require --target=1.dev
* final tweaks to documentation text
* DAML-LF: Add interning for type to DAML-LF 1.dev
We add two new features to DAML-LF 1.dev:
* a per package list (or table) of `Type` messages, and
* a new case in the `Type` message which is an index into this table.
In combination, these two features can be used to allow DAML-LF
encoders to perform hash-consing of `Type` messages. We also change the
Haskell implementation of our DAML-LF encoder to do exactly that when
targetting DAML-LF 1.dev.
Doing this has a few benefits:
1. The DALFs produced by `damlc` get smaller: I've seen a case where
the size dropped from 69MB to 45MB.
2. DAML-LF decoders need to decode less data.
3. Decoded packages use less memory because identical structures are
now shared. This is particularly helpful in situations where we need
to keep the interface (or signature) of a package in memory for a
long time.
This PR mostly takes care of the Haskell implementation. However, we
need to make the Scala implementation of the decoder aware of the new
features as well since we have tests that load DAML-LF 1.dev into the
engine. A decoder and _targeted_ tests on the Scala side will follow
in a separate PR.
CHANGELOG_BEGIN
CHANGELOG_END
* Make jq tests aware of type interning
CHANGELOG_BEGIN
CHANGELOG_END
* Improve jq test
CHANGELOG_BEGIN
CHANGELOG_END
* Apply Remy's suggestions
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Improve the imperative bits
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
We introduce in #7835 VersionedNode which associate to each node a
version. In this PR, we update Transaction validation to take into
account this version when comparing nodes.
This advances the state of #7788
CHANGELOG_BEGIN
CHANGELOG_END
Following what have been done in #7944, we add a check in the Scala
version of the LF type checker that rejects invalid kinds (i.e. kinds
of the form k -> Nat).
This PR does the "Scala side" of #7917.
CHANGELOG_BEGIN
CHANGELOG_END
* Require pattern match exhaustiveness in spec.
Changes the rules for case expressions to require exhaustiveness.
This involves moving out the "well-formed case alternative" rules
to a separate judgement, and introducing a "pattern match
exhaustiveness" judgement.
The symbol ⊲ used for pattern match exhaustiveness comes from
sheaf theory and formal topology, where "A ⊲ B₁, …, Bₙ" means
"A is covered by { B₁, …, Bₙ }", indicating an exhaustive
coverage of A by possibly overlapping conditions B₁, …, Bₙ.
For example, see [this nlab page](https://ncatlab.org/nlab/show/posite).
It seemed appropriate.
changelog_begin
changelog_end
* rst syntax
* fix ExhaustVariant and some weird other syntax
* review comments
Remove file describing how contract key will be implement. The file
is not usefull as contract key have been fully implemented. Futhermore
the file contains plenty of out of date information.
CHANGELOG_BEGIN
CHANGELOG_END
This PR prepares the change of inference algorithm described #7788
where each node is version independently. This PR associates to each
node of `VersionedTransaction` a version. In the current state, all
nodes are associated to the version of the transaction itself. The
inference algorithm (that will make those versions potentially
distinct) will be implemented in an upcoming PR.
CHANGELOG_BEGIN
CHANGELOG_END
This PR fixes two bugs in the type checker:
- Fetch are now rejected if the type constructor does not match a
template
- to_any, from_any, and type_rep are now rejected if the type
parameter contains synonyms.
Additionally, this PR adds the type of the validation error that each
test case should throw in the test that ill formed expression are
properly rejected. In the way we fix some wrong test cases.
CHANGELOG_BEGIN
CHANGELOG_END
- Exercises' results are checked after the children have been checked.
- output the mismatching nodes in case of comparison failure.
CHANGELOG_BEGIN
CHANGELOG_END
* DALF Reader: Refactor decoder environment
Currently, the module to be decoded is part of the decoder environment.
This turned out to be unpleasant during my attempts to implement
interning for types since decoding the type interning table does not
happen in the context of a module.
This PR moves the module out of the decoder environment and passes it
to the decoder function for modules directly. Unfornatunately, we still
need to keep the module name in the environment since that is used as a
default when decoding location information. We solve this problem by
making the module name in the environment optional and always filling
it in when decoding in actual module.
CHANGELOG_BEGIN
CHANGELOG_END
* Adapt tests
CHANGELOG_BEGIN
CHANGELOG_END
* Follow Remy's suggestion
CHANGELOG_BEGIN
CHANGELOG_END
This PR drops support for Transaction version 1 to 9. This is a
breaking change that may affect only ledger created with a pre SDK
1.0.0. No Sandbox-based ledger is affected by this change.
This advances the state of #7788
This breaking change was approved by the architecture committed.
CHANGELOG_BEGIN
CHANGELOG_END
* LF: Rotate type checking of case expressions by 90 degrees
This reproduces changes made to the Haskell type checker (in #7873)
into the Scala type checker.
CHANGELOG_BEGIN
CHANGELOG_END
Add two more tests for the following cases:
1. The Expressions the values should be updated with are non-atomoic.
This tests that our internal ANF transformation doesn't destroy the
omptimization. (In fact, it can't really do that because it is run
after the phase which does the optimization. But better safe than
sorry.)
2. A single record field is updated multiple times. This tests that
the last update wins. Unfortunately, we cannot drop the earlier
updates, or at least computing their values, since these values
might be bottom. (We could add an optimization _after_ we've
transformed into ANF but I don't expect any benefits from this in
practice and would hence not do it.)
CHANGELOG_BEGIN
CHANGELOG_END
* Cache computation of top-level values at definition level
Earlier the computation of a top-level value was only cached at the
use-site in SEVal. This introduces SDefinition which contains the same
mechanism as SEVal to cache the computation.
As expected this does not impact performance much:
before: CollectAuthority.bench //daml-lf/scenario-interpreter/CollectAuthority.dar CollectAuthority:test avgt 40 44.267 ± 0.728 ms/op
after: CollectAuthority.bench //daml-lf/scenario-interpreter/CollectAuthority.dar CollectAuthority:test avgt 40 43.693 ± 0.702 ms/op
What this does have a significant impact is on reducing the number of distinct
SValues for things like type class dictionaries etc, so that now we have one
SValue per dictionary rather than one per SEVal.
CHANGELOG_BEGIN
CHANGELOG_END
* Address code review
* Fix speedy tests
* Fix scenario service vs struct projections bug
In a recent change, we have sped up the projection from structural
records by means of an inference in the DAML-LF type checker. This
change was made under the assumption that users of the DAML interpreter
run through package validation before compiling to the internal Speedy
AST. Unfortunately, that assumption was wrong and not covered by our
existing tests. Manually testing of a new release candidate reminded us
of the fact that the scenario service skips package validation for the
sake of faster response times in the IDE.
This PR drops the assumption that package validation is always run.
Instead, we add the old implementation of struct projection, which works
without DAML-LF type checking, back and use it whenever the AST has not
been annotated with the information inferred by the type checker. We do
the same for struct updates.
We improve test coverage by _additionally_ running the `damlc`
integration tests without package validation for the latest stable
version of DAML-LF and DAML-LF 1.dev. These tests would have caught the
issue we only discovered during manual testing.
CHANGELOG_BEGIN
CHANGELOG_END
* Add some explanations
CHANGELOG_BEGIN
CHANGELOG_END
* choice observers, WIP
changelog_begin
changelog_end
fix generator driven test
Node.isReplayedBy, consider observers
add observers to NodeExercise in transaction.proto
add observers to TemplateChoice in Ast.scala
add observers to LF .proto, and Haskell Ast for TemplateChoice
reinstate trailing // for better format
fix validate tests
fix haskell LF decoder when choice-observers field is missing in .proto
fix build
make choice-observers optional in scala AST
make choice-observers optional in Haskell Ast
address comments from Remy and Martin
more review comments
check TransactionVersions.minChoiceObservers in Transaction encode/decode
featureChoiceObservers, and check in haskell type-checker
improve speedy Compiler for empty choice-observers
extend scala LF decoder for optional choice observers
extend scala parser for choices to allow optional choice-observers clause, and test
rename new field in scala Ast -> "choiceObservers"
var rename
extend TypingSpec tests for choice-observers. also add missing negative test for controllers
switch from keyword "ob" to identifier "observers" in scala parser choice syntax
add TODO for featureChoiceObservers to be part of DAML 1.9 (issue 7139)
* replace "NICK" comment markers with "FIXME #7709" comment markers
* Regularize scala mod-parser syntax for controllers in preparation for adding choice-observers.
From keyword "by" to identifier "controllers" + leading ","
changelog_begin
changelog_end
* fix another use of the old "by" syntax -> ", controllers"
This is pretty much a verbatim copy of
https://github.com/digital-asset/daml/pull/7740, but for updates of
structural records instead of projections.
This has no immediate performance impacts since the DAML compiler does
_currently_ not produce any structural record updates. Having both,
projections and updates, have the same runtime characteristics seems to
be a desirable property nevertheless, simply to avoid nasty surprises
should we ever start to use these updates.
Since the compiler does not produce any structural record updates, I
have no way to benchmark this. However, since the code is changed here
in the same way it was changed in the PR mentioned above, I expect the
same saving, which were roughly 80 ns per operation, where the previous
cost of an operation was _at most_ 210 ns.
CHANGELOG_BEGIN
CHANGELOG_END
* Speed up projections of structural records
Currently, for every projection of a structural record, we first do a
linear search through the field names to find the index of the field
value in the record value. Then we use this index to get the actual
value. However, since we started sorting the fields in alphabetic order
both in types and in values, this lookup will always yield the same
result for a fixed record type and a fixed field name, regardless of the
value of the record.
This PR changes the AST node for structural record projection to
contain the index of the field within the record as well. This
information is not contained in the DAML-LF archive itself and must
hence be filled in by the DAML-LF type checker.
The only context in which we use structural record prejections are
typeclass method invocations, which do a little bit more than just the
projection. Unfortunately, typeclass method invocations are the small
unit of work I can benchmark using the setup I have. In the end, they
are also the feature our users really care about.
My benchmarks show that the time a single typeclass method invocation
takes has dropped from ca. 210ns to ca. 130ns. That's a speedup of
ca. 1.6x.
CHANGELOG_BEGIN
[DAML Interpreter] Typeclass method invocation is now 1.6x faster.
CHANGELOG_END
* Apply Remy's suggestions
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Make it compile again
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
This fixes a bug in the Speedy interpreter. With this change, the
interpreter crashes if it attempts to fetch using a contract key that
has an empty set of maintainers. This propagates to exerciseByKey as
this command is compiled using fetchByKey speedy built-in.
This is a breaking change approved by @bame-da .
CHANGELOG_BEGIN
[LF] (Bug fix) enforce non-empty maintainers in contract key during fetchByKey and exericiseByKey.
CHANGELOG_END
As part of multi-party read/write on command submissions we also need
to be able to support multi-party queries in DAML Script. Since the
queries in DAML Studio go via ParticipantView, this PR extends this as
a prerequisite to making use of this in DAML Script.
This does not yet relax any restrictions on command submissions. Those
still require a single committer.
For consistency with the surrounding code, I went for a simple Set
rather than a non-empty Set.
changelog_begin
changelog_end
* add silent_annotations option to da scala bazel functions
* use silent_annotations for several scala targets
* use silencer_plugin instead when the lib isn't used
* use silent_annotations for several more scala targets
* use silencer_lib for strange indirect requirement for running tests
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* silent_annotations support for scaladoc
Currently the list of node that correspond to a "by key" operations
is tracked in the Transaction metadata. this PR move this information
into the nodes themself.
This is a preparatory work to include this information in the serialization
format for transactions
This advances the state of #7622
CHANGELOG_BEGIN
CHANGELOG_END
* LF: enforce non-empty maintainer in contracts key lookup
This fixes a bug in the Speedy interpreter. With this change, the
interpreter crashes if it attempts to lookup using a contract key that
has an empty set of maintainers.
This is a breaking change approved by Bernhard Elsner.
CHANGELOG_BEGIN
- [LF] (Bug fix) enforce non-empty maintainers in contract key during
lookupByKey.
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Address Moritz's review.
* Address Martin's review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
This PR creates 3 validation modes:
* `Strict`: Specifies that the committer should validate the package
before committing them to the ledger. When using this mode, the
packages committed to the ledger can be fully trusted and do not
have to be validated when loaded into the engine.
* `Lenient`: Specifies that the committer should perform a fast
validation of the packages before committing them to the ledger.
This mode is useful for ledger integrations that cannot handle
long-running submissions (> 10s). When using this mode, the
packages committed to the ledger cannot be trusted and must be
validated every time they are loaded into the engine.
* `No`: Specifies that the committer should not perform any
validation the packages before committing them to the ledger. This
should be used only by non distributed ledgers, like DAML-on-SQL,
where the validation done in the API server can be trusted.
This PR creates 3 preloading modes:
* `Synchronous` : Specifies that the packages should be preloading
into the engine before committed.
* `Asynchronous`: Specifies that the packages should be preloaded into
the engine asynchronously with the rest of the commit process. This
mode is useful for ledger integrations that cannot handle
long-running submissions (> 10s). Failure of the preloading process
will not affect the commit.
* `No`: Specifies that the packages should not be preloaded into
the engine.
CHANGELOG_BEGIN
- [Integration Kit] In kvutils, add metric
daml.kvutils.committer.package_upload.validate_timer to track
package validation time.
CHANGELOG_END
This fixes a bug in the Speedy interpreter. With this change, the
interpreter crashes if it attempts to create a contract containing a
contract key that has an empty set of maintainers.
This is a breaking change approved by Bernhard Elsner.
CHANGELOG_BEGIN
- [LF] (Bug fix) enforce non-empty maintainers in contract key during
contract creation.
CHANGELOG_END
* LF: force `actor` field in Exercise Update to be undefined.
This PR forces the field 'actor' from 'Upate.Exercise' to be undefined
starting from LF 1.5.
This breaking change is approved by Bernhard Elsner and Shaul Kfir.
It will affect only handcrafted LF archives, as no compiler from SDK
1.0.0 or latter made use of this field.
CHANGELOG_BEGIN
CHANGELOG_END
* Revert "Revert "Speedy Compiler: factorize and clean code (#7493)""
This reverts commit d99f35b3cd.
* Speedy: Reintroduce usage of muti-binding let.
This PR fixes recursion issue with `closureConvert` and `freeVars`.
The explosion of muti-binding in single binding makes the function
`closureConvert` crash with stackoverflow in some cases.
We reintroduce the usage of multi-binding let in the first pass of the
compiler. The ANF transformation pass will replace them with nested
single binding let. ANF pass do not have issue with deep expression,
as it handle them with trampoline.
CHANGELOG_BEGIN
CHANGELOG_END
We currently pass the token to all builtins named `SBU*` aka the
update builtins. Since we pass the token immediately on all call sites
of these builtins, there's no point in passing the token at all.
This PR removes the token passing for the `SBU*` builtins.
Unfortunately, this does not improve execution performance. However,
since it simplifies the code, I still consider it worthwhile.
CHANGELOG_BEGIN
CHANGELOG_END
We factorize scala code in the compiler:
Here are some example of changes:
* factorize the common part of `compileChoice` and `compileChoiceByKey`
* factorize the common part of `compileFetch` and `compileFetchByKey`
* application patterns like
```
SEApp(expr1, Array(expr2))
```
are replaced by:
```
app(expr1, expr2)`
```
* let binding patterns like
```
val x1 = expr1
val x1Pos = nextPosition()
SLet(x1, ...)
```
are replaced by pattern
```
let(expr){ x1Pos => ... }
```
* function binding pattern like
```
Abs(1){
val x1Pos = nextPosition()
x1
}
```
are replaced by
```
function{ x1Pos => ... }
```
CHANGELOG_BEGIN
CHANGELOG_END
With this PR, the profiler nicely handles the new SDefinitionRef's added in #7472.
We add labels for:
'create'
'exerciseByKey'
'fetch'
'fetchByKey'
'lookupByKey'
CHANGELOG_BEGIN
CHANGELOG_END
This includes:
* Create
* ExceciseByKey
* Fetch
* FetchByKey
* LookupByKey
This change allows us to reference the code used in a template definition from another package instead of having to inline it at the use-site
CHANGELOG_BEGIN
CHANGELOG_END
* Allow speedy execution without performing authorization checks. This is a temporary measure for canton.
This is controlled by a new Speedy machine member:
/* Controls if authorization checks are performed during evaluation */
val checkAuthorization: Boolean,
The default value for `Machine.checkAuthorization` is `true`, but can be set `false` when execution is called from `Engine.reinterpret`, by passing `checkAuthorization = false` to `reinterpret`.
changelog_begin
changelog_end
* CheckAuthorizationMode
* share duplicate code for computing machine contextActors and auth
The Engine does not verify key duplication when construction a
transaction, this task is currently let to the ledger implementation.
This PR provides an utility function to compute the duplicated keys
in a transaction.
CHANGELOG_BEGIN
CHANGELOG_END
* remove failedTransactions field from ScenarioLedger.RichTransaction
changelog_begin
changelog_end
* remove Blinding.checkAuthorizationAndBlind; fixup callers to use Blinding.blind when blindningInfo is required
* rename/relocate: ScenarioLedger.CommitError.FailedAuthorizations --> SError.DamlEFailedAuthorization
* fix types to demonstate that at most one FailedAuthorization is detected/reported
* address small review comments
* Speedy: group compiler parameters into a Config case class.
Speedy compiler is parametrized by several option flags. Those flags
are passed through different calls (in particular in the instance of
`CompiledPackages`.
This PR group all this configuration parameters into a case class to
pass the option in a cleaner way.
CHANGELOG_BEGIN
CHANGELOG_END
This PR implements a part of the proposal from #7093.
Here packages are validated in the participant node before to be sent to the ledger.
CHANGELOG_BEGIN
- [Ledger-API] participant node validate Dar before uploading to the ledger.
This may increase upload time significantly.
CHANGELOG_END
This is part of #7399.
This enable skipping packages validation in the engine when they are
comming from a trusted source.
CHANGELOG_BEGIN
CHANGELOG_END
changelog_begin
changelog_end
Adapt test for small error message change. An improvement! Previously an internal message was shown. Make test flexible enough to pass for old & new message.
Adapt expected output file. Only change is the contract-ids for active-contracts when the test ends.
fix small test bugs in testcase where authorization is wrong
temp disabled 3 tests; needs investigation; see TODO markers
temp adapt 1 test for change in error message
temp disable 1 test. needs invesigation
undo accidentally commited change to .bazelrc
add copyright header to new file
remove testcase (badActorCheck2) which is no longer expected behaviour, when authorization occurs during execution
address comments: be mre private & other tiny changes
appease scala formatter
improve expected error message in KeyNotVisibleStakeholders testcase
fix authorization issues and re-enable the 3 failing tests in EngineTest which now pass
fix auth issue and re-enable ledger-test-tool test: WronglyTypedContractIdIT
fix compatibility
re-enable test in KVUtilsTransactionSpec.scala
The migration script from #7344 ensures that there are no more values
in Sandbox Classic DB encoded with Value version < 6. Hence, we do
not need anymore a special EngineConfig to maintain backward
compatibility of Sandbox classic DB.
CHANGELOG_BEGIN
CHANGELOG_END
* LF: shortcup failure in SValue ordering.
Before this PR comparison of lists, maps, ans structural record had a
always linear complexity.
CHANGELOG_BEGIN
CHANGELOG_END
This reverts changed made in #3416, where the standard scala was
replace by ad-hoc Speedy equality for SValue. As Speedy equality
become more permisive (e.g. it does not check for type constructor of
records, variants, and enumrations or constructor of variants or
enumerations) this is safer to use standard scala eqality for case
class.
CHANGELOG_BEGIN
CHANGELOG_END
* DAML-LF spec: Fix two minor issues around structs
We make it explicit that all field names in a structural record need to
be distince. We also fix a small typo in the indices for the
`ExpStructCon` typing rule.
CHANGELOG_BEGIN
CHANGELOG_END
* Address comments regarding distinctness
CHANGELOG_BEGIN
CHANGELOG_END
This PR uses the new data structure introduced in #7220.
Additionally this fixes `svalue.Equality` and
`svalue.Ordering` which were considering <a: x, b: y>
different from <b:y, a:x>.
CHANGELOG_BEGIN
CHANGELOG_END
This PR uses the new data structure introduced in #7220.
Additionnally this fix `Value Equal instance` which was considering
<a: x, b: y> different from <b:y, a:x>.
CHANGELOG_BEGIN
CHANGELOG_END
This PR perform the following changes:
- reduce function size by spliting them out,
- rename "translate" function prefix to "compile",
- replace 'private' by 'private[this]',
- add 'inline' annotation to function used only once.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix trace statement handling in DAML Script service
This PR makes sure that we get both trace statements coming from the
ledger client as well as from the ledger server. In the script service
those are two different Speedy machines so we need to interleave
them. This is a bit messy so let me explain the steps (apologies for
not splitting it up but I think it’s easier to understand the
motivation when it is a single PR):
1. Abstract over the tracelog interface. The ringbuffer doesn’t makes
sense for our purposes and is annoying to clear.
2. Pass in the respective fields of `Machine` to `Conversions` instead
of the whole machine. This is both a simpler design (passing around
mutable objects scares me) and it allows us to assemble the fields
from different machines.
3. Initialize the ledger machine with a simple ArrayBuffer tracelog.
4. After `submit` and `submitMustFail` copy the tracelog from the
ledger to the client.
fixes#7280
changelog_begin
changelog_end
* Address review feedback
changelog_begin
changelog_end
* Only include stakeholder contracts in result of query
This fixes a bug in the script service. We need to filter out divulged
contracts since this should behave exactly like the ACS endpoint on
the ledger API.
changelog_begin
changelog_end
* Update daml-lf/interpreter/src/main/scala/com/digitalasset/daml/lf/scenario/ScenarioLedger.scala
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Update daml-script/runner/src/main/scala/com/digitalasset/daml/lf/engine/script/LedgerInteraction.scala
Co-authored-by: Remy <remy.haemmerle@daml.com>
* fmt
changelog_begin
changelog_end
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Refactor EnrichedTransaction
This PR prepares for work to interleave _authorization_ with _execution_, which has been an open issue for a long time: #132.
- Split computation of `BlindingInfo` and `FailedAuthorizations` into separate modules.
- Fixup callers: `Blinding.scala` and `ScenarioLedger.scala`, to do:
```
val failedAuthorizations =
AuthorizingTransaction.checkAuthFailures(authorization, tx)
val blindingInfo =
BlindingTransaction.calculateBlindingInfo(tx)
```
- Simplify code to remove `DontAuthorize` and `Authorization` types, which were only needed to restrict the combined _enrichment_ code from doing the authorization check when `blind`ing only is required. Now `Blinding.blind` calls just `calculateBlindingInfo`.
* address comments from Stefano
A `Struct[+X]` is a list of pair `(FieldName, X)` ordered by first
component.
We use this data-structure to represent TStruct LF type.
It will be used in upcomming PRs to sorted fields in Struct values.
CHANGELOG_BEGIN
CHANGELOG_END
* Normalize struct field order in the LF spec.
This commit changes the LF spec to ensure that struct field order is ignored whenever reasonable.
* Rename "type synonym resolution" to "type normalization", and change the rule for structs to also normalize the order of fields.
* Make the struct type "well-formed" only if the struct fields are sorted by field name. (This matches the way "well-formed" is only used for normalized types. E.g. type synonyms are not considered "well-formed" by the spec. This nomenclature should possibly be fixed in a separate PR.)
* Change the struct con expression type rule accordingly.
* Change the definition of a struct value. A struct value is now a struct expression where all the field expressions are values, and all the field names are sorted.
* Change the evaluation rule for struct expressions to also sort the fields by field name in the process.
As far as I know, this spec matches the behavior of the LF engine. I also tried to make the treatment of struct field order in the types match the treatment of struct field order in expressions (i.e. normalized structs always have sorted fields, but non-normalized structs do not).
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Update daml-lf/spec/daml-lf-1.rst
Co-authored-by: Remy <remy.haemmerle@daml.com>
This is primarily intended to improve error reporting in DAML Script,
in particular, https://github.com/digital-asset/daml/issues/7192 which
is a blocker for replacing scenarios by DAML Script.
However, I think it is also a very reasonable feature in general.
changelog_begin
changelog_end
lookupContract catches the exception which results in us continuing
and eventually running into a null-pointer exception.
This is another reason why we should enable the NonUnitStatements wart
but I’ll leave that for a separate PR (last I tried the scenario
service resulted in a ton of false positives).
fixes#7185
changelog_begin
changelog_end
This reverts commit 87a0ee9383.
CHANGELOG_BEGIN
WARNING remove the changelog entry from 87a0ee93 and starting by :
"[Sandbox] By default Sandbox rejects the development versions of ..."
CHANGELOG_END
This PR replaces a few `for (... to/until ...)` loops with `while`
loops which explicitly maintain their counter. In my benchmarks, this
has lead to a 3% performance improvement. Since the affected functions
are the ones used for applying arguments to functions, similar
improvements should be observable regardless of the specific benchmark.
CHANGELOG_BEGIN
CHANGELOG_END
* Add TO_TEXT_CONTRACT_ID primitive
This is the first part of #7114.
This PR
* Adds the primitive to the protobuf.
* Handles decoding and encoding in Haskell and Scala.
* Handles typechecking in Haskell and Scala.
* Handles speedy compilation and interpretation in Scala.
* Updates the specification.
This PR does not yet change the standard library to make use of this
primitive.
changelog_begin
changelog_end
* Apply suggestions from code review
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Avoid extra allocation
changelog_begin
changelog_end
Co-authored-by: Remy <remy.haemmerle@daml.com>
* script service party management test-case
* script service: allocateParty and listKnownParties
Behave like sandbox on party hint. If a hint is given then allocate
exactly that party name and fail if it already exists. Otherwise
generate a fresh party name. For easier debugging the fresh name is
based on the display name.
changelog_begin
changelog_end
* Track parties that were not allocated explicitly
E.g. parties generated by `partyFromText`.
* Handle and test party already exists error
* fmt
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Originally those two options were set up by calling state mutating
method on a build engine.
In this PR, we move this two options to the recently introduced EngineConfig.
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
* [Sandbox] By default Sandbox rejects the development versions of
DAML-LF and transaction format. One has to explicitly use the
command line option `--dev-mode` to allows those versions.
CHANGELOG_END
The methods moved take a single instance of the `Machine` class as an
argument, hence it makes no sense to have them in the companion object.
CHANGELOG_BEGIN
CHANGELOG_END
This PR simplifies the deprecated `getScenarioLedger` to take the
whole engine instead of `compiledPackages` (from the engine) and the
rest of the `outputTransactionVersions` (form the engine conffig).
This prepares for some changes for #5164, where more options for the
engine config will have to be used by the method. Also we will need to
be sure the `compiledPackages` following the option of the engine
config.
CHANGELOG_BEGIN
CHANGELOG_END
This PR addresses a TODO in the implementations of `foldl` and `foldr`.
The continuations for both primitives have to apply the step function to
the accumulcation and the current list item. So far, we've used the
`SEAppAtomicFun` AST node for this purpose. However, since the step
function and both arguments have already been evaluated to values, we
can also use the `enterApplication` function that has recently been
introduced together with our use of ANF.
Benchmarks: Applying `foldr (+) 0` (or `foldl (+) 0`) to the
pre-computed (and pre-allocated) list `[1..1_000_000]` has taken ca.
117s before this change and now takes only ca. 83ms. This is a speedup
of ca. 1.4x.
CHANGELOG_BEGIN
CHANGELOG_END
* add -Ywarn-unused to all scalac options
* remove some unused arguments
* remove some unused definitions
* remove some unused variable names
* suppress some unused variable names
* changeExtension doesn't use baseName
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* work around no plugins in scenario interpreter perf tests
* remove many more unused things
* remove more unused things, restore some used things
* remove more unused things, restore a couple signature mistakes
* missed import
* unused argument
* remove more unused loggingContexts
* some unused code in triggers
* some unused code in sandbox and kvutils
* some unused code in repl-service and daml-script
* some unused code in bindings-rxjava tests
* some unused code in triggers runner
* more comments on silent usages
- suggested by @cocreature; thanks
* fix missing reference in TestCommands
* more unused in triggers
* more unused in sandbox
* more unused in daml-script
* more unused in ledger-client tests
* more unused in triggers
* more unused in kvutils
* more unused in daml-script
* more unused in sandbox
* remove unused in ledger-api-test-tool
* suppress final special case for codegen unused warnings
.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
^
* one more unused in daml-script
* special scaladoc rules may need silencer, too
* unused in compatibility/sandbox-migration
* more commas, a different way to `find`
- suggested by @remyhaemmerle-da; thanks
This is a pre-step for ANF, which will allow ANF expression forms to call `executeMatchAlts` directly, rather than always from the `KMatch` continuation.
changelog_begin
changelog_end
This is another small PR split out from the ANF work.
Improve explore-dar to allow dar file to be selected on the command line, and so allows the json-parser example to be run like this:
bazel run daml-lf/interpreter/perf:explore-dar -- --base JsonParser pipeline --arg 10
As well as a speed test:
bazel run daml-lf/interpreter/perf:speed-json-parser
This example is a nice quick smoke test to check speedy execution is not broken.
changelog_begin
changelog_end
* DAML Engine: Speed up foldr
This PR replaces Speedy's current implementation of `foldr`, which
basically does nothing more than replace the builtin `foldr` with an
expression for its standard implementation in a functional language,
with a more loop-like implementation that takes advantage of the
structure of Speedy. There's additional complexity for the case when
the step function expects only one more argument before it performs
some computation. The exact issue and the solution are explained in a
comment in the code.
I've benchmarked the application of `foldr (+) 0` to the pre-computed
list `[1..100_000]`. With the old implementation of `foldl` this took
ca. 35ms, with the new implementation ca. 11ms. Further experiments
indicate that out of these times ca. 5ms are spent applying the
arguments to `(+)` and performing the addition. Thus, the time actually
spent in `foldr` amounts to 30ms and 6ms, respectively. This means
the new implementation of `foldr` is ca. 5x faster than the old
implementation when the step function takes at least two arguments.
For the case of a step function which expects only one argument before
performing computation, I've benchmarked the application of
`foldr (\_ -> identity) 0` to the pre-computed list `[1..100_00]`. The
measured times have dropped from ca. 42ms to ca. 19ms. Further
experiments that the time spent in `foldr` itself has dropped from
ca. 32ms to 9ms.
CHANGELOG_BEGIN
- [DAML Engine] The performance of `foldr` has been improved by more
than 4x.
CHANGELOG_END
CHANGELOG_BEGIN
CHANGELOG_END
* Add explanation on how extra coomplexity could have been avoided
CHANGELOG_BEGIN
CHANGELOG_END
* Extend the scenario service with DAML Script support
This adds most of the infrastructure for running DAML Script via the
scenario service which means it runs as part of DAML Studio and `daml
test`. This is hidden behind a feature flag so we can land this and
parallelize the remaining tasks. The main things that are missing are:
1. `createAndExerciseCmd` and `exerciseByKeyCmd`.
2. Party management needs some work and listing parties is
unsupported.
3. Time management
4. Potentially some better error handling (we need to go through
SResult and SError and see what is relevant for us).
Overall, it is already in a very usable state and there is a decent
range of tests.
closes#3688
changelog_begin
changelog_end
* Update compiler/damlc/daml-ide-core/src/Development/IDE/Core/Rules/Daml.hs
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Fix name for actor system and pool
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Stratify Speedy builtins into pure/effectful.
This is a preparatory change for the ANF translation, which can treat pure
builtins more efficiently.
- Parent: `SBuiltin`, (which are effectful).
- Child: `SBuiltinPure` (which are pure).
Effectful builtin functions may raise `SpeedyHungry` exceptions or change
machine state. Pure builtins can be treated specially because their evaluation
is immediate.
The interface to effectful builtins remains:
def execute(args: util.ArrayList[SValue], machine: Machine): Unit
For pure builtins the interface is:
def executePure(args: util.ArrayList[SValue]): SValue
changelog_begin
changelog_end
* insert trailing comma and newline in 17x execute method args-list
* fix spello
This PR replaces Speedy's current implementation of `foldl`, which
basically does nothing more than replace the builtine`foldl` with an
expression for its standard implementation in a functional language,
with a more loop-like implementation that takes advantage of the
structure of Speedy.
I've benchmarked the application of `foldl (+) 0` to the pre-computed
list `[1..100_000]`. With the old implementation of `foldl` this took
ca. 31ms, with the new implementation ca. 11ms. Further experiments
indicate that out of these times ca. 5ms are spent applying the
arguments to `(+)` and performing the addition. Thus, the time actually
spent in `foldl` amounts to 6ms and 26ms, respectively. This means
the new implementation of `foldl` is ca. 4.3x faster than the old
implementation.
CHANGELOG_BEGIN
- [DAML Engine] The performance of `foldl` has been improved by more
than 4x.
CHANGELOG_END
I've already grown very tired of almost always having to touch the
`Classify` class whenever I change anything about Speedy. Let's put an
end to this by using reflection instead of explicitly listing all
different AST nodes and continuation types.
The `PrettyLightweight` class has similar problems that can be fixed
using the same idea.
This change might slightly change this output but since this is an
experimentation tool, I deem this acceptable.
CHANGELOG_BEGIN
CHANGELOG_END
* FrontStack: Ensure that claimed invariant is maintained
The documentation of `FrontStack` claims that the head of the
`FQPrepend` is never empty but there's a way to violate this claimed
invariant.
This PR makes sure the invariant is maintained and then uses the
invariant to remove a redundant `else`-branch.
CHANGELOG_BEGIN
CHANGELOG_END
* Remove initials from NOTE
CHANGELOG_BEGIN
CHANGELOG_END
* DAML Engine: Speed up updates of multiple fields in a record
Currently, updating multiple fields of a single record is not very
efficient. If you update, say, `n` fields, as in
```haskell
r with f_1 = ...; ...; f_n = ...
```
`n-1` intermediate records are created and almost immediately
discarded. Each creation of an intermediate record involves shallowly
cloning an array of size `m`, where `m` is the number of fields of the
record type. This does not only cost computation time but also a lot of
avoidable memory allocations. Overall, the cost of such an update is
`O(m*n)` in terms of time and memory.
This PR changes the DAML-LF interpreter such that multiple updates of
the same record are batched together. This avoids the creating of any
intermediate records but only creates the final record resulting from
the updates. This reduces the time complexity to `O(m+n)` and the
memory complexity to `O(m)`.
The code path for updating a single record field remains unchanged
(for now) and hence won't suffer from the additional complexity that
is required for the batching described above.
I've benchmarked this change by creating a single record with `m`
fields and updating all `m` fields in one single update expression,
for `m = 10` and `m = 20`. In both cases, the benchmarks showed a
speedup of ca. 2.5x.
This is part of #5766.
CHANGELOG_BEGIN
- [DAML Engine] Record update expressions of the form
`R with f_1 = E_1; ...; f_n = E_n` are now run as batch updates
in order to improve runtime performance and avoid unneccesary
memory allocations.
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Remy <remy.haemmerle@daml.com>
Co-authored-by: Remy <remy.haemmerle@daml.com>
This is a spinoff from IDE support for DAML Script #6929. There I need
cannot quite use `run` from ScenarioRunner since I need to handle
results differently but I don’t want to replicate all the visibility
and authorization logic from `lookupContract` and `lookupKey`.
changelog_begin
changelog_end
* Use Bazel builtin pkg_tar rule
* Use @rules_pkg//:pkg.bzl%pkg_tar
The pkg_tar rule builtin to Bazel has been deprecated.
See https://docs.bazel.build/versions/master/be/pkg.html
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* daml-lf/parser: Add the ability to parse location annotations
Currently, the DAML-LF parser we use for testing in Scala land does not
understand location information and has in particular no way to produce
`ELocation` AST nodes. Having these nodes will be important in a test
for a feature I'm currently working on in order to demonstrate that my
Speedy AST transformation can actually look through location
information properly.
This PR adds a rule to the parser to allow for parsing location
information and producing `ELocation` nodes.
CHANGELOG_BEGIN
CHANGELOG_END
* Address Nick's feedback
CHANGELOG_BEGIN
CHANGELOG_END
* Evaluation order for exercise & other updates.
This adds more DAML tests and error cases for, finishing off the update
interpretation for now. I believe this is roughly the limit of what we
can test directly in DAML, and the rest is up to DAML-LF testing.
For example, there's no way to test that the contract inactivity check
in EvUpdExercInactive happens before the evaluation of the controller
expression, since there is no way to generate an 'exercise' with actors
from DAML, as far as I could tell, and 'exercise_without_actor' always
evaluates the controller before passing it off to 'exercise'.
For the same reason, the EvUpdExercBadActor check seems impossible to
trigger from DAML itself. The closest is an authorization test that
happens during submit (which I added a test for, because it is kinda
relevant to evaluation order of scenarios, even though scenario
interpretation isn't really important to the spec).
So at the very least we should add LF tests for those two cases.
changelog_begin
changelog_end
* Apply Moritz\'s suggestions
* Add test with fetch of consumed contract
* add -Xlint:doc-detached
- reverts 1feae964e3 from #6798
* attach several scaladocs where they'll actually be included
* no changelog
* attach several more scaladocs where they'll actually be included
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* Draft of new operational semantics.
* make errors deterministic, and be explicit about them
* Flesh out the op semantics for expressions and add tests.
* Nail down the semantics of create
* Apply suggestions from code review
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Add a reminder that u has to be a value
... in the update interpretation section.
changelog_begin
changelog_end
* add a test to track the ghc behavior on multi-lets
* add test for erasable type abstraction
* Add non-erasable type abstraction value test.
* move the ValExpTyAbsNat to 1.7-only value test
Co-authored-by: Remy <remy.haemmerle@daml.com>
* LF: Simplify inference of output transaction/value version.
As part of #5164, we simplify the way transaction and value versions
are inferred for the engine output.
The versioning of a transaction tr is done as follow:
* Let tvMin be the minimal transaction version allowed by the DAML
engine configuration
* Let tvMax be the maximal transaction version allowed by the DAML
engine configuration
* Let n₁, ..., nₘ be the nodes of tr.
* Let pkgᵢ be the package of the template associated to the node nᵢ
* Let lvᵢ be the maximal language version that pkgᵢ uses (directly or
through its dependencies)
* let tvᵢ be the maximal transaction version supported by all engines
that support both lvᵢ and tvMin
* Let vvᵢ be the maximal value version supported by all engines that
support tvᵢ.
* Let tv be the maximal transaction version between tv₁, ..., tvₘ.
* If tv is not greater than tvMax
+ then
- Version the values of each node nᵢ according vvᵢ
- Version tr according tv
+ fail otherwise
CHANGELOG_BEGIN
CHANGELOG_END
* set many extra scalac -Xlint options for all Scala projects
CHANGELOG_BEGIN
CHANGELOG_END
* move NoCopy to its own file
package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
trait NoCopy {
^
* move more traits, classes, and objects to proper packages
- note that `package` is itself a scoping construct, so if your reason
is the apparent aesthetic of placing a bunch of things in one `package
object`, that is easily remedied by deleting the `object` keyword
* fix some type-parameter-shadow warnings
- I'm generally in favor of sensible name-shadowing, following the
"deliberately hide variables that should not be accessed here" school
of thought. But I think type name shadowing isn't quite as valuable
and more likely to confuse than general variable shadowing, so have
experimentally linted it out.
Example warning:
EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
Offset defined in package v1. You may want to rename your type
parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
^
* fix more package-object-classes warnings
* fix an inaccessible warning
ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
^
* enable -Xlint:infer-any
- continuing the saga of #6116, #6132
* enable -explaintypes for more detailed type errors
* missed header for NoCopy; probably should have left it in the package file
* misspelling in comment
* revert -Xlint:doc-detached
- there are a lot of these fixes, and they are noisy, so shifting to a
separate PR
- thanks to @leo-da for pointing out
According transaction specification the field `value_version` "is
optional; if defined, it must be a version ofthe value
specification, and `template_id` shall be consumed according to
that version. Otherwise, it is assumed to be version 1."
We take advantage that the encoding of identifier in value does not
change since version "1" to unset the field `value_version`, hence
using the default interpretation.
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
[Engine] - Change the callback for contract key from `GlobalKey => Option[ContractId]` to `GlobalKeyWithMaintainers => Option[ContractId]`
CHANGELOG_END
We add a method to CompiledPackages that returns the latest language version that package uses directly or indirectly through is dependencies.
This will be used in the context of #5164
CHANGELOG_BEGIN
CHANGELOG_END
SubmittedTransaction and CommittedTransaction are moved from com.daml.lf.transaction.Transaction to
com.daml.lf.transaction
This helps intelliJ type inference.
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
Default new Engine constructor to Engine.StableConfig so it does not
need to be overridden unless you specifically want to run in
Engine.DevConfig mode
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* include DefTemplate's key types under the Ty type variable
* note that contract key types are included by folding over DefTemplates' Tys
- the topo sort from --root now correctly includes contract keys
* test that contract key dependencies get included in the Scala codegen plan
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* Revert ANF changes and add a testcase for evaluation order
After careful consideration, we decided that the change in evaluation
order that was accidentally introduced by the ANF changes should be
considered a breaking change or arguably even a bug and should not
land in 1.3.0.
Therefore, this PR reverts the following commits:
1. 353d0da6f7
2. a45b51042f
3. 04c7b2af7f
4. a624dd7242
5. b3aab72cee
Other PRs mostly had trivial merge conflicts that I resolved. The two
most interesting ones here are probably
1. https://github.com/digital-asset/daml/pull/6576 which was easy to
resolve and the change to return SEValue instead of SExpr is still
nice and useful even if we do not need the guarantees.
2. it https://github.com/digital-asset/daml/pull/6542 which required
some changes since the constructors changed. If you want to review
those changes in detail (they are pretty straightforward so not too
important), it’s probably easiest to check out this PR and run
```
git diff 2cd2a8f2a8
daml-lf/interpreter/src/main/scala/com/digitalasset/daml/lf/speedy/Compiler.scala
```
to see the diff to the parent commit of the first commit that
introduced ANF.
changelog_begin
changelog_end
* replace traverseU and sequenceU with traverse and sequence
- with -Ypartial-unification on, the extra Unapply typeclass lookup is
unnecessary
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* limit imports; we only need *> and void
So that the resulting value is returned from the `execute` function, instead of being assigned to the `returnValue` machine state.
Rename the parent class from `SBuiltinMaybeHungry` to `SBuiltinEffect`, and have this be the parent of a few more subclasses which cannot meat the _pure_ interface.
Declare method `executeEffect` of the parent class `SBuiltinEffect` which has the same type as the original `execute`.
changelog_begin
changelog_end
Ask @nickchapman-da for details on what this transform does and why it's
better. I have no clue.
What I do understand is that current master has a stack-consuming
behaviour that just so happens to not reach the 1mb default stack size
on the tests we run, and that this presumably better version of a let
transform had been left out from the original ANF PR because it was
pushing the stack consumption over that 1mb limit.
Besides reintroducing the "better" let version, this PR applies an
almost full CPS transform to the set of mutually recursive functions
that grow the stack. Combined with the trampoline, this should mean the
ANF transform itself is done in (almost) constant stack space.
The one exception to that is the `flattenAlts` function, which has been
left as a plain imperative loop. This does mean that we consume stack
linearly with the nesting level of alts, so conceivably this could still
blow the stack. There is no fundamental reason that makes this hard, it
just seemed unnecessary for the stack consumption and potentially
damaging to performance (changing a native Java loop on an array to a
recursive list consumption).
To verify the stack-consuming behaviour on master, one can apply this
patch:
```patch
diff --git a/ledger/sandbox/BUILD.bazel b/ledger/sandbox/BUILD.bazel
index a717e66fb..4c953b3ea 100644
--- a/ledger/sandbox/BUILD.bazel
+++ b/ledger/sandbox/BUILD.bazel
@@ -124,6 +124,7 @@ da_scala_library(
da_scala_binary(
name = "sandbox-binary",
+ jvm_flags = ["-Xss256k"],
main_class = "com.daml.platform.sandbox.SandboxMain",
resources = ["src/main/resources/logback.xml"],
visibility = ["//visibility:public"],
```
then run
```
bazel test -t- //ledger/sandbox:conformance-test-contract-id-seeding-memory
```
The same process can be repeated on this branch to verify that, at
least, this now runs under 256kb of stack for all our tests. I don't
know of a good way to get a stronger guarantee on stack allocations.
While my primary motivation here is correctness (and termination, I
guess), this does seem to yield a modest performance improvement. On a
test machine, I got the following results as confidence intervals:
* `master` (b4915a4bd7): 54.00, 54.56
* this branch: 52.54, 53.11
I've only ran the benchmark once on each and don't have enough
experience with it to know how reliable those numbers are. I'll try to
run some more.
CHANGELOG_BEGIN
CHANGELOG_END
small clean up in the testing library TransactionBuilder, bsaically add a Submitted and a Committed version for empty transaction constant and build method.
plus a bit of cleaning in tests using TransactionBuilder
CHANGELOG_BEGIN
CHANGELOG_END
Since we have only absolute contract ids, there is no more distinction
between local and global disclosure/divulgence. Let's please remove the
`global` prefix since it causes confusion (at least for me it did).
CHANGELOG_BEGIN
CHANGELOG_END
The dead code was hanging on by a slender thread. This change cuts that thread.
For over-apps, we now push a new continuation type KOverApp, instead of KArg.
KOverApp contains args of type SExprAtomic, instead of SExpr, and so its
execute() method may use enterApplication, instead of executeApplication.
This allowing lots of code to be removed:
- the continuation types: KArg, KFun, KBuiltin and KPap.
- defs: evaluateArguments and executeApplication.
changelog_begin
changelog_end
Currently, the scenario service ships the disclosure information of a
transaction twice. One of the two copies is completely unused. This PR
simply deletes it. I hope this reduces confusion in the future.
CHANGELOG_BEGIN
CHANGELOG_END
* Distinguish witnesses and divulgences in scenario table view
Currently, we display a "D" (for "Disclosure/Divulgence") in the table
view of the scenario view when a party can see a contract because they
have learnt about it via disclosure.
This change further refines this visibily indicators for disclosure:
We display a "W" for "Witness" (i.e. the party has witnessed the
creation of the contract) and "D" for "Divulgence".
Help for the reviewer: In the scenario service server, witnessing is
called explicit disclosure whereas divulgence is called implicit
disclosure.
CHANGELOG_BEGIN
* [DAML Studio]
When displaying scenario results in table view in DAML Studio, we now
indicate _why_ a party knows about the existence of a contract:
- `S` means the party is a signatory.
- `O` means the party is an observer.
- `W` means the party has witnessed the creation of the contract.
- `D` means the party has learned about the contract via divulgence.
CHANGELOG_END
* Address @cocreature's feedback
I was trying to track down what looks like a new source of flakiness in
the ANF work (#6440), which manifested in a stack overflow on the
recursion of the `go` function in speedy.Compiler. I spent a bit of time
refactoring this to be functional in order to then trampoline it, only
to realize afterwards that I had worked on `freeVars/go` whereas the one
blowing up the stack was `validate/go`.
Obviously, it did nothing to reduce the rate of flakiness on the stack
overflow issue I was trying to address, as it's a separate function.
So there isn't really a good reason for this change, except I had done
it and I do believe the code looks marginally nicer, so might as well
submit it.
CHANGELOG_BEGIN
CHANGELOG_END
* ANF transformation in Speedy.
The idea behind this PR is to transform speedy expressions into a simpler form where all non-atomic sub-expressions are made explicit by the introduction of let-forms. In particular, for the function-application form. These simpler forms allow the execution engine to take advantage of the atomic assumption, and often removes many additional execution steps. In particular the pushing of continuations to allow execution to continue after a compound expression has been reduced to a value.
changelog_begin
changelog_end
* improve comment
* inline functions relocateA/L
* remove comment about scalafmt
* remove commented out alterative def for transformLet1
* improve code by adding incr methods to DepthA/E
* remove (n == 0) special case in trackBindings
* clarify comment further
* improve validate/go to not consume stack for deeply right-nested let-expressions
* address comments from Remy: be private; use final case case; etc
* rename to unsafeCompilationPipeline
* add back some trailing commas
* remove commented-out debug line
* improve comment
* remove dev/debug code in compilationPipeline
* remove commented out code in SEAppGeneral.execute
* undo unrelated code improvement in SValue.scala
* fix compile. object Anf cannot be private
* Measure time of interpretation and store it in CommandExecutionResult.
* Added parameters for passing in interpretation time for WriteService and LedgerWriter.
* Code tidying.
* Added CommitMetadata parameter to LedgerWriter.
* Thread through interpretation time. Aggregate interpretation times for a batch.
* Take max of interpretation times.
* Calculate interpretation time in nanos.
* Moved CommitMetadata into separate file.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Delegate deprecated methods to method with new signature.
* Code tidying.
* Suppress deprecation warnings.
* Made interpretation cost optional in CommitMetadata.
* Test that we populate interpretation time.
* Code tidying.
* Apply suggestions from code review
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Code tidying.
* Some more tests. Do not return interpretation cost for a batch if it only contains non-transaction submissions.
* Reformatted.
* Reformatted.
* Named arbitrary interpretation cost.
* Reverted changes for BatchingLedgerWriter.
* Always drop commit metadata for batches and don't report interpretation cost.
* More specific expectations.
* Include authorization check and blinding in interpretation time.
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
So far, these two flags only worked in sandbox classic because I wasn't
aware the code path consuming the options is not shared between both
implementation. Now, sandbox next is on par with sandbox classic.
CHANGELOG_BEGIN
CHANGELOG_END
