* Implement a simple profiler for DAML scenarios
The profiler runs a single scenario and records timing information when
each function (and some other closures) are entered and left. The
resulting information can be visualized as a flamegraph using
[speedscope](https://www.speedscope.app/).
The profiler works by instrumenting the CEK machine at the heart of
DAML Engine. Unfortunetaly, this causes a very small overhead on
non-profiling runs too. However, in my benchmarks I could not measure
any significant impact on the overall runtime at all. More precisely,
the overhead is as follows:
Every closure now has an additional field called `label`. In
non-profiling runs this field is always set to `null`. This field needs
to be allocated, copied whenever we copy a closure and scanned during
garbage collection. Additionally, whenever we enter a closure, we check
this field and whenever it is _not_ `null`, i.e. never during
non-profiling runs, we record an "open event" and set up a hook for the
corresponding "close event". Thus, the additional cost during
non-profiling runs are a single pointer comparison and a jump beyond
the "then branch".
Since this is still very much in active development, there are no
documentation, other than an entry in a README, and no tests yet. They
will come before we promote this. However, the UX will look very
different then since we already have plans to significantly change it.
CHANGELOG_BEGIN
CHANGELOG_END
* Run scalafmt
* Make profiling argument to PureCompiledPackges optional
* Fix a bunch of tests
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt is so annoying
CHANGELOG_BEGIN
CHANGELOG_END
* Apply simple suggestions
CHANGELOG_BEGIN
CHANGELOG_END
* Simplify and clarify the public interface to Speedy.
- Remove `isFinal`. A client just uses `run()`.
- Remove `toSValue`. The value in available in `SResultFinalValue(v: SValue)`.
- A client never directly access the `.ctrl` (or `.returnValue`) components.
- A client may use `setExpressionToEvaluate(expr)` to evaluate a new expression on an existing machine.
changelog_begin
changelog_end
* remove while loop which executes just once
* avoid unnecessary mutation when running speedy
* Properly mangle/unmangle names in the scenario service
Previously, we just passed in whatever we had in DAML land which broke
the scenario service as soon as a name needed mangling. This PR fixes
mangling when passing identifiers to the scenario service and
unmangles them again when pretty printing.
fixes#5593
changelog_begin
- [DAML Studio] Fix a bug where scenarios with names that are mangled
in DAML-LF resulted in a crash in the scenario service.
changelog_end
* Mangle module names in context updates
* Fix conversion from contract to node ids in the scenario service
Previously we only included the mapping from the ledger. However, this
does not include any contracts which might have been created in a
partial transaction that failed. This caused a
`NoSuchElementException` when converting the error which is obviously
not what we want.
This PR extends the mapping to include any contracts created in the
current partial transaction.
changelog_begin
- [DAML Studio] Fix a bug where a failed transaction that included
references to a transient contract returned a gRPC error instead of
the expected error message.
changelog_end
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
* Update compiler/scenario-service/server/src/main/scala/com/digitalasset/daml/lf/scenario/Conversions.scala
Co-Authored-By: Remy <remy.haemmerle@daml.com>
Co-authored-by: Remy <remy.haemmerle@daml.com>
This is a spin off from the work for making module prefixing
work. There I wanted to be able to use the `GeneratePackageMap` rule
to get access to the modules in an LF package which requires to be in
an Action rather than in IO.
This resulted in a much larger refactoring than I expected, so here is
a list of changes to ease review:
1. I’ve split off Development/IDE/Core/RuleTypes/Daml.hs into its own
Bazel targets to avoid cycles. This allows us to use a rule without
depending on the implementation of that rule.
2. Development/IDE/Core/IdeState/Daml.hs has moved into its own Bazel
target. There is a lot of stuff that needs `withDamlIdeState` now
since the `ghcide` `initialise` function does not add our Shake rule
for getting the GHC session and making all of that depend on
damlc:lib creates a mess.
3. The actual logic of the initialization has stayed the same but
moved into a rule in Development/IDE/Core/Rules/Daml.hs.
4. As mentioned above, a few things that could previously get away
with only the rules provided by ghcide now need our own rules so
they switched to `withDamlIdeState`. This mainly affects `damlc
docs`.
5. Making daml-docs work with the ofInterestRule is a bit tricky.I did
find the issue and included the fix but also disabled the
ofInterestRule since it really does not make sense to generate Core
in the background.
changelog_begin
changelog_end
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* Limit memory usage of scenario service in tests
Previously the scenario service used over 2GB of memory during
//compiler/damlc/tests:integration-dev for absolutely no reason.
This PR adds an option to pass JVM options when starting the scenario
service and sets them to 200MB for the Shake tests and the damlc
integration tests which seems to be more than sufficient.
These option can also be set in `daml.yaml`. I did not change the
defaults since this is shipped to users where I’d rather have high
memory usage than a crashing scenario service and on certain large
DAML codebases we might actually use a fair bit.
changelog_begin
changelog_end
* Update docs/source/tools/assistant.rst
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Add keys with maintainers to Fetch nodes
The new field is populated by the interpreter whenever the fetched
contract has a key. Used for contract key reinterpretation in Canton.
CHANGELOG_BEGIN
- [DAML-LF] Add keys with maintainers to Fetch nodes in transactions.
CHANGELOG_END
Both were previously binding 0.0.0.0, which is inherently insecure. More
importantly to me, that meant running `bazel test //...` essentially
rendered my computer unusable for however long it took (which is
_long_), as it kept popping up focus-stealing dialogs about whether or
not I wanted to trust "java" to open an incoming network connection.
The ScenarioService does not seem to have an existing setup for CLI args
and my Scala-fu is not good enough to add one, so I just changed the
hard-coded path.
The JSON API already had an option, just with the wrong default. This is
technically a breaking change, but I'm hoping to pass it under the
"experimental" flag we still have on the JSON API.
CHANGELOG_BEGIN
- [JSON API - Experimental] As a security improvement, the JSON API
server will now bind on ``127.0.0.1`` by default. Previous behaviour was
to bind on ``0.0.0.0``; you can get that behaviour back by passing in
the (existing) flag ``--address 0.0.0.0``.
- [DAML SDK] The Scenario Service will now bind on ``127.0.0.1``. Previous
behaviour was to bind on ``0.0.0.0``.
CHANGELOG_END
* Use KeyHasher to serialize contract keys in kvutils
- Use Value instead of VersionedValue in GlobalKey as the versioning does not make sense here
and may be misleading as the a value with a different version but same meaning would still
be the same key.
- Relocate the KeyHasher to ledger-api-common so kvutils can use it (otherwise cyclic dependencies)
- Replace storing of the contract key as a VersionedValue with the hash produced by KeyHasher.
This is backwards incompatible. A compatible option would require us to query the key with both
the old way and the new way which is unattenable. We're making a calculated breaking change.
CHANGELOG_BEGIN
- [DAML Ledger Integration Kit] Serialize contract keys using a hash instead of the value in kvutils.
This is a backwards incompatible change to kvutils.
CHANGELOG_END
* Use proper hasher for contract keys and not KeyHasher
- Use Hash.scala, not KeyHasher.scala.
- Add hash to GlobalKey as we want the hash to be computed from the inside.
The use of KeyHasher will be later deprecated and replaced by this.
* Use "sealed abstract case class" trick instead of private ctor
and rebase fix
* Revert change to unsupported value version decode error
* Reformat code
* Add kvutils changelog entry and bump the version
I don’t have a great test case to demonstrate the problem but
`contexts = contexts + (ctxId -> ctx)` is definitely not an atomic
update which means that inserts can get lost.
changelog_begin
changelog_end
If we get interrupted after updating our local view but before doing
the actual context update on the server side bad things happen. To fix
this, we first clone the context then update the clone which avoids
this and is a bit nicer than throwing an `uninterruptibleMask` around
updating the local view and sending the updateCtx gRPC request.
fixes#4525
changelog_begin
changelog_end
This adds the NodeInfo trait to compute the informees of a node and the
required authorizers. This has been separated into its own trait as this
computation is also required (externally) for serialized transaction nodes
in a context where the full deserialization of the node would be prohibitively
expensive.
While doing this change and cleaning up transaction authorization it was
discovered that top-level fetches (which can only be created from scenarios)
did not include the submitter as a witnessing party.
CHANGELOG_BEGIN
[DAML SDK] Fix computation of witnesses of top-level fetch nodes in scenario results ("known to").
CHANGELOG_END
Unfortunately, downgrading `scalafmt` resulted in files that would be
reformatted to invalid code, because lines would be merged but trailing
commas would be left in.
I've manually run it everywhere and fixed these issues so people don't
have to fix them as they go.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Handle unserializable scenario result types
Previously, the conversions made some attempt at guarding against this
by matching on SPap and PClosure. While it would be possible to extend
this to match on STypeRep, this doesn’t actually fix the issue since
this can be nested, e.g., you can have a record with a field that is
an SPAP.
This PR changes this to simply catch any errors thrown from
`toValue`. While this feels a bit ugly, I think it’s a reasonable fix
for now.
changelog_begin
- [DAML Studio] Scenarios with unserializable result types no longer
crash the scenario service.
changelog_end
* Only run test on DAML-LF >= 1.7
* Track used packages during whole of engine submit
- Introduce MutableCompiledPackages interface
- Add TrackingCompiledPackages that tracks fetched packages
- Make used packages in transaction optional to distinguish between
missing dependencies and empty set of dependencies.
* Reimplement package dependency tracking
- Compute direct dependencies of a package during decoding
- Compute transitive dependencies of a package when adding a package
to engine.
- Annotate the resulting transaction with package dependencies
in Engine.submit.
* Create Ast.Package with proper direct deps in scenario service
While we don't have use for direct dependencies of a package in
scenario service (only Engine.submit needs it), it's better to be
accurate.
This of course overapproximates the direct dependencies.
* Compile a each new package once in ConcurrentCompiledPackages
Until now, we broke long party names in the table view in the scenario at
spaces. This looked quite ugly.
Now, we don't break them at spaces. At the same time we also moved the CSS
into a file rather than having it in the code to make changes like this one
easier in the future.
This fixes#3642.
* Update bazel-common to fix javadoc issues
Specifically, to fix the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel:7:1: in javadoc_library rule //ledger-api/rs-grpc-bridge:rs-grpc-bridge_javadoc:
Traceback (most recent call last):
File "/home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel", line 7
javadoc_library(name = 'rs-grpc-bridge_javadoc')
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/com_github_google_bazel_common/tools/javadoc/javadoc.bzl", line 27, in _javadoc_library
dep.java.transitive_deps
object of type 'JavaSkylarkApiProvider' has no field 'transitive_deps'
```
* Define Maven deps using rules_jvm_external
* Pin artifacts
* Remove bazel-deps generated targets
* Remove bazel-deps
* Switch to rules_jvm_external targets
* update bazel documentation
* pom_file: There are no more bazel-deps targets
* BAZEL-JVM.md `maven_install` typo
* Properly fill eventId for active contracts
This gets rid of the last remaining bit that assumes
contractId==eventId.
Fixes#65.
Contributes to #2068.
* Do not conflate eventId and contractId in the daml-lf interpreter
* Do not treat contractId as eventId in Ledger.scala
* Remember the transaction that divulged a contract.
* In this scope we can treat divulged contracts the same as disclosed ones
* revert a few more syntactical changes to make the overall diff smaller
* retain the same behavior on the scenario service api
* fix unreleased after rebase
* Refactor DAML-LF encoder/decoder in prep for string interning
Refactor the encoder/decoder such that all the functions concerned with
interning live next to each other and code will break once the types of
the protobuf messages change.
This PR is a pure refactoring and does not change any functionality.
* Fix typos
* Preload packages to engine during upload
* Improve logging in KeyValueCommitting and add timing information
* Fix scenario service tests now that logging is done in interpreter
* add Numeric.java
* ledger-api: rename `decimal` field to `numeric` in value protobuf
* Address Gerolf's comment
* ledger-api: add missing renammings
* ledger-api: relax syntax of numbers that can be sent as numerics
* extractor: fix
* leger-api: change format of number though ledger api
* daml-lf: fix numeric regexp
* ledger: fix tests
This fixes all flakiness in `damlc test` that I was able to
reproduce. Previously, I got it to fail in about 10% of the cases
whereas now I have successfully run tests 200 times under load without
issues.
There were two issues at play here:
1. We run scenarios in separate threads to be able to kill the running
Shake session quickly even if a scenario has an infinite loop or
something like that (there is a timeout but it’s quite long). This
could result in one of those left-over threads trying to issue a
request while we are already trying to shut down.
To fix that, we wait for the concurrency semaphore to be empty before
shutting down.
2. Just waiting for scenario executions is not quite sufficient as
`runAction` does not wait for all rules to finish (we could just use
runActionSync in `damlc test` but I’d rather make this work
properly). While we do wait for all scenario executions to finish
there is one gRPC request in offInterest that we do not wait for:
gcCtxs.
To fix this, I’ve now routed all gRPC requests through the semaphore
which means that we will also wait for these requests to finish (or
prevent them from spawning).
This makes more sense anyway as scenario executions are mostly fairly
cheap requests while things like setting up the context are expensive
so we want to limit their concurrency.
We should make the concurrency limit configurable but I’ll leave that
for a separate PR.
SS.scenarioServiceClient does not just read the actual client from
some IORef, it registers the available gRPC methods. Apparently we
never knew about this or at least I didn’t.
By only doing this once, we should speed things up a bit and this
fixes once of the assertion failures that we have been seing on
shutdown (pthread_mutex_lock(&mu->mutex) == 0 in sync_posix.cc) which
was caused by trying to register a method from another thread after
destroying the channel.
* Show function names in stack trace on failing scenario
So far, we've only shown the location of the function but not its name.
Now, we add the name of the function as well.
I assume the plan was to implement stack traces. I intend to do that as well
but the message type does not fit my approach. Thus, let's remove it first.
* Print stack traces in the scenario on failure
Currently, we only print the last source location, which is not
particularly helpful for debugging. Now, we put all source locations we
encounter during execution on the continuation stack and print them when
a scenario fails. This PR does not print the names of entered functions
or choices. We leave this for a future PR.
* Address Moritz' comments
The module was only used to format decimal numbers. The only impact the
formatting could have had was to add/strip trailing zeros from the
presentation of the number. However, manual testing shows this did not
happen. Thus, the whole module is now useless.
While experimenting with the pure Haskell gRPC implementation I
noticed that there is a fair amount of duplication in PrettyScenario
so this PR cleans up some of that.
* check that submitter is in maintainers when looking up keys
Fixes#1866. Note that this limitation applies both for `lookupByKey`
and `fetchByKey` -- anything involving retrieving a key is affected.
* add UNTIL-LF to run tests up to a certain version of DAML-LF
* name targets for DAML tests better
* add notes about DAML-LF changes
* commit Test.daml with DAML-LF 1.5 rather than compiling it on the fly
* add scenario tests for #1866
* add warnings about future key behavior in docs
* use flag rather than version when executing
Given that we already made the max message size configurable it only
seems reasonable to also make the timeout configurable and on very
large projects, we do sometimes hit this.
Previously we sometimes ended up declaring the expected exit of the
scenario service as an unexpected exit. This PR addresses this by
adding a new variable that we set to True before we ask the scenario
service to exit by closing stdin.
* Allow controlling the gRPC message limit via daml.yaml
We have had to raise that in the past since it caused issues on large
projects so it makes sense to make it configurable.
* Update unreleased.rst
Co-Authored-By: Beth Aitman <bethaitman@users.noreply.github.com>
* specify the new fields for interning package IDs
* percolating intern tables
* crawl packages for package IDs as a pre-phase: generic prep
* stub case for interned_id in Scala packageref reader
* HasPackageRefs instances for the rest of the ast
* make intern table and use when encoding PackageRefs in v1
* don't need where
* stub out decode for interned package IDs
* no benefit to using uint32 instead of uint64
* percolate in encode one step
* interned case for decoding PackageRefs
* naming details
* intern table decoder
* finish propagating the intern table in encoder
* encode the package ID table
* document the vital assumption of encodeInternedPackageIds
* propagate the intern table through the LF decoder
- done by stacking ReaderT on top of Decode internally,
as discussed with @hurryabit
* daml-lf-proto requires mtl
* stub out interned case in Scala LF decoder
* stub interface decoder function
* get the interned table to most places in InterfaceReader
* support for interned package IDs in Scala decoder
* use ImmArraySeq instead of Vector for Scala intern decode table
* adding that ghc extension didn't make sense
* implement interned ID decoding for InterfaceReader
* scenario service won't have interned package IDs
* test the interned ID resolution in Scala by examining the proto -> AST in detail
* proper precondition for the dev phase of interned IDs testing
* better error reporting for malformed DALFs in intern test
* just import Data.Int
- suggested by @neil-da; thanks
* pass around the lookup function instead of the vector in decoder
- suggested by @neil-da; thanks
* remove derivations for types deleted in e63b012d2d
* rename VersionAware to EncodeCtx
- suggested by @hurryabit; thanks
* rename MDecode to MonadDecode
- suggested by @hurryabit; thanks
* pass a function through the encoder instead of a set
- based on suggestions by @hurryabit and @neil-da; thanks
* daml-ghc test that interned IDs are generated
- suggested by @hurryabit; thanks
* adapt to 5b480c99ec#1844
* add scenario Module decoding to Decode.OfPackage
* use purely data-driven decoding in scenario service in Scala
- decouples scenario service from LF decoder implementation
* make DecodeV1 companion private
* make extension to LFv2 more obvious
Previously, we always reverted back to the table view when the
scenario results changed. This PR changes this to preserve the
selected view.
I’ve also tested that this does not break backwards compatibility: If
you use the newer extension with an older SDK, you will get the
previous behavior of reverting to the table view and you can still
switch by clicking on the button.
Fixes#1675
This is a temporary fix to unblock users that have run into this
limit. The long-term solution here is to use compression and/or make
the limit configurable.
Ironically `managed` didn't turn out to make our code more manageable
and we ended up mostly using it in very isolated places only to then
immediately convert it back to bracket-style functions using `with`.
This PR also removes the use of `managed` from the GcpLogger which was
the only other place where we are using it and it finally kills the
rather silly logic that starting up the scenario service was tied to
having an event logger.
* Rename Value's ContractId to VContractId
* daml-lf: a bit more about PartyId
* daml-lf: Concatenable MatchingStringModule
* daml-lf make clear type used for Scenarios only
* daml-lf create ContractId, LedgerId, TransactionId
* sandbox-sql conversion util
* LedgerName -> LedgerString
* futher type cleanup in the sandbox
* daml-lf add test for LedgerString
* fixing tests
* a bit more safety in the DB
* Address Stephen's comments
* fix rebase
* More fixes for StringModule
* change length of LedgerString (256 -> 255)
* Replace all occurrences of Tagged in DAML-LF AST with newtypes
This must end as it makes the code bases harder to understand without
adding any benefits.
* Add two more usages of mkVal
* Remove Orphans.Lib_hashable
* Remove tagged from package dependencies
* Derive stock classes as such
The current naming suggests, the protobuf encoding and the CLI use different
textual representations of minor versions, which is not true. They only
differ in their types. We push this type conversion to the call sites
of both functions instead.
Up to now, the engine blindly assumed that contract ids pointed to
contracts of the right type. However, this assumption is faulty:
contract ids coming from the Ledger API cannot be type checked
in command translation since we need access to the contract itself
to do so.
This caused some seriously surprising / broken behavior: one could
send an exercise command with the wrong template id and still go
through, or break internal invariants about the type of choices.
This commit fixes this by checking that the type of the contract
instances we fetch is correct at runtime.
cc @hurryabit @dajmaki @remyhaemmerle-da @S11001001 @meiersi-da
* Use DA.Prelude a lot less
* Make LANGUAGE pragmas have the right case
* Don't reexport DA.Prelude
* Remove more DA.Prelude imports
* Remove the nastiest renaming DA.Prelude
* Final removal of DA.Prelude
* Kill DA.Prelude
* Add missing dependencies
* More missing package imports
* More missing package imports
* Update compiler/daml-lf-ast/src/DA/Daml/LF/Ast/Base.hs
* Update compiler/daml-lf-ast/src/DA/Daml/LF/Ast/Base.hs
* clean up v3 contract key case in TransactionVersions
* clean up duplicate cases in ValueVersions.assignVersion
* make LanguageMinorVersion a proper ADT, aliased LanguageVersion.Minor
* port DecodeV1 to LanguageMinorVersion ADT via implicit conversion
- @remyhaemmerle-da might prefer the explicit application of Stable/Dev
* make releasePrecedes private
* add dev versions to the timeline
* copyright header for LanguageMinorVersion
* build the package database for 1.dev, support 1.dev as --target
* test presence and new-ness of "dev" in timeline
* report dev versions in engine info; note 1.dev target in release notes
* spec and governance rules on 1.dev and minor dev in general
* make the governance rule about dev more direct
* 1.x
* missed plural; thanks @leo-da
* further definition of 1.dev in LFv1 spec; thanks @remyhaemmerle-da
* Move from prettyDiagnostic to prettyDiagnostics
* Remove as much pretty print stuff as we can
* Try moving duplicate named functions with similar semantics and identical types to different names
* Change to returning pretty printed outputs from Diagnostics
* Remove a redundant import
