* Implement a simple profiler for DAML scenarios
The profiler runs a single scenario and records timing information when
each function (and some other closures) are entered and left. The
resulting information can be visualized as a flamegraph using
[speedscope](https://www.speedscope.app/).
The profiler works by instrumenting the CEK machine at the heart of
DAML Engine. Unfortunetaly, this causes a very small overhead on
non-profiling runs too. However, in my benchmarks I could not measure
any significant impact on the overall runtime at all. More precisely,
the overhead is as follows:
Every closure now has an additional field called `label`. In
non-profiling runs this field is always set to `null`. This field needs
to be allocated, copied whenever we copy a closure and scanned during
garbage collection. Additionally, whenever we enter a closure, we check
this field and whenever it is _not_ `null`, i.e. never during
non-profiling runs, we record an "open event" and set up a hook for the
corresponding "close event". Thus, the additional cost during
non-profiling runs are a single pointer comparison and a jump beyond
the "then branch".
Since this is still very much in active development, there are no
documentation, other than an entry in a README, and no tests yet. They
will come before we promote this. However, the UX will look very
different then since we already have plans to significantly change it.
CHANGELOG_BEGIN
CHANGELOG_END
* Run scalafmt
* Make profiling argument to PureCompiledPackges optional
* Fix a bunch of tests
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt is so annoying
CHANGELOG_BEGIN
CHANGELOG_END
* Apply simple suggestions
CHANGELOG_BEGIN
CHANGELOG_END
* Simplify and clarify the public interface to Speedy.
- Remove `isFinal`. A client just uses `run()`.
- Remove `toSValue`. The value in available in `SResultFinalValue(v: SValue)`.
- A client never directly access the `.ctrl` (or `.returnValue`) components.
- A client may use `setExpressionToEvaluate(expr)` to evaluate a new expression on an existing machine.
changelog_begin
changelog_end
* remove while loop which executes just once
* avoid unnecessary mutation when running speedy
Remove the `Ctrl` trait and separate `Machine.ctrl: Ctrl` into `Machine.ctrl: SExpr` and `Machine.returnValue: SValue` instead. This allows for avoiding dynamic dispatch on `ctrl` and instead allows for checking a pointer for `null` to decide if we have an expression that needs further break-down or a return value ready to be passed to the next continuation.
To make this check really only a pointer comparison we also needed to remove the abomination of "fully applied partially applied primitives". In order to achieve this, we check whether a PAP will be fully applied afterward when applying the last argument.
On the `collect-authority` benchmark, this increases throughput by around 13%, on another more computation heave benchmark by about 21%.
`collect-authority` benchmark on `master`:
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
112.361 ±(99.9%) 1.965 ms/op [Average]
(min, avg, max) = (107.047, 112.361, 120.745), stdev = 3.493
CI (99.9%): [110.396, 114.326] (assumes normal distribution)
```
`collect-authority` benchmark on this branch:
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
98.196 ±(99.9%) 1.933 ms/op [Average]
(min, avg, max) = (91.580, 98.196, 105.478), stdev = 3.436
CI (99.9%): [96.263, 100.129] (assumes normal distribution)
```
computation heavy benchmark on master
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
44.030 ±(99.9%) 0.742 ms/op [Average]
(min, avg, max) = (42.124, 44.030, 46.781), stdev = 1.319
CI (99.9%): [43.289, 44.772] (assumes normal distribution)
```
computation heavy benchmark on this branch:
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
36.222 ±(99.9%) 0.580 ms/op [Average]
(min, avg, max) = (34.897, 36.222, 39.787), stdev = 1.031
CI (99.9%): [35.643, 36.802] (assumes normal distribution)
```
changelog_begin
changelog_end
* Use triggerId field in trigger start response
* Use triggerId field for stop trigger result
* Fix indentation and make yields consistent
* Use pair constructor for JsObject instead of Map
* Use triggerIds field in list triggers response
changelog_begin
changelog_end
* Adapt ResponseFormat from JSON API
* Add some type annotations
* Use response format with status and errors/result fields
* Update and refactor tests
changelog_begin
changelog_end
Speedy: run() dont step()
- Running the Speedy machine with `run()` instead of `step()`
- Remove: `SResultContinue`
- Add: `SResultFinalValue(_)`
We change the top level control of Speedy: from machine.step() to machine.run, with the control of stepping while the machine returns SResultContinue moved into speedy itself. (And so SResultContinue is removed in favour of SResultFinalValue.) The main advantage of this approach is that the tight while loop can be moved inside the exception handler, rather than having to wrap the handler every step.
changelog_begin
changelog_end
* Endpoint to list all triggers (not yet by party)
* Clean up test code a little
changelog_begin
changelog_end
* Test for listing running triggers
* Respond with JSON list instead of random text
* List triggers by party
Pass party name in request body.
Store another map of party name to set of trigger ids.
Also store party names in the values of the original trigger id map, so
we can update the party map when stopping a trigger.
* Make DAML Triggers and DAML Script default to wall-clock-time
Now that sandbox defaults to wall-clock-time there is no reason why we
should not default in DAML triggers and DAML Script.
changelog_begin
- [DAML Triggers] ``daml trigger`` now defaults to wall clock time if
neither ``--wall-clock-time`` or ``--static-time`` is passed.
- [DAML Script] ``daml script`` now defaults to wall clock time if
neither ``--wall-clock-time`` or ``--static-time`` is passed.
changelog_end
* Make --static-time and --wall-clock-time exclusive
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* Move more trigger tests to scala tests
This PR moves more tests of triggers over to the Scala test suite, in
particular:
- The existing tests there abstract over the time mode and are
instantiated once for wallclock mode and once for static time mode.
- I’ve added tests for TLS and Auth.
- I’ve removed the TLS and Auth tests outside of scala-test since they
are now redundant.
- I’ve added the time tests to the scala-tests stuff since with the
new ledger time model, that’s necessary to actually trigger a
failure if you get static time vs wallclock time wrong (MRT and LET
no longer exist).
I haven’t yet moved all the func tests over, I’ll do that separately
and then we can kill the old tests completely.
changelog_begin
changelog_end
* Factor out test utils into a library
* Improve handling of exposed-modules with data-dependencies
Previously, we tried to rename all modules of a dependency via
--package. This fails if some of those modules are not exported. This
was trivial to hit as a user since the ``daml-trigger`` library made
use of this.
This PR adds a few things to improve the situation:
1. We only rename modules that are exposed. This fixes the issue if
you don’t actually reference a non-exposed module from your
data-dependency.
2. I’ve removed the exposed-modules from daml-trigger. I don’t think
they are essential here given that the module name has `Internal`
in the name and it’s too easy to have something that actually
references the non-exposed module since the types are reexported.
3. I’ve added documentation that mentions this issue.
4. I’ve added a warning if your exposed-modules are excluding some
modules. Maybe worth turning this into an error in the future.
changelog_begin
changelog_end
* Update compiler/damlc/lib/DA/Cli/Damlc/Packaging.hs
Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>
Co-authored-by: associahedron <231829+associahedron@users.noreply.github.com>
* Use com.daml as groupId for all artifacts
CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END
* Add 2 additional maven related checks to the release binary
1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId
* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
This replaces the rather horrible previous setup of having a custom
test runner that spawns 3 separate JVM processes by a single scalatest
test suite that starts sandbox and the JSON API in process.
changelog_begin
changelog_end
This PR adds a new test suite for DAML triggers based on scala test
rather than the client_server_test macro + a custom main. This seems
much nicer than the client_server_test (we get a lot of useful stuff
from scalatest, e.g., useful output of assertion failures, things
don’t blow up after the first test failure, …).
This PR only ports over a small fraction of the tests to make review
easier. The plan is then to port over everything and kill off the
existing test stuff once everything is ported over.
changelog_begin
changelog_end
* Depend on LF version specific daml-libs
* daml-script.dar build multiple LF versions
CHANGELOG_BEGIN
[DAML Script] The `daml-script` library is now available in multiple LF
versions, namely 1.7, 1.8, and 1.dev.
CHANGELOG_END
* daml-trigger.dar build multiple LF versions
[DAML Triggers] The `daml-trigger` library is now available in multiple
LF versions, namely 1.7, 1.8, and 1.dev.
* Keep daml-script.dar available for tests
* Keep daml-trigger.dar available for tests
* daml-libs LF versions integration test
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Contributes to #4194.
Closes#4231.
Closes#5022.
CHANGELOG_BEGIN
- [Ledger API] The protobuf fields ledger_effective_time and maximum_record_time have been removed from
command submission. These fields were previously deprecated following the introduction
of a new ledger time model. See issue `#4194 <https://github.com/digital-asset/daml/issues/4194>`__.
[Java Bindings] removed the usage of ledgerEffectiveTime and
maximumRecordTime, and instead added minLedgerTimeAbsolute and
minLedgerTimeRelative in CommandSubmissionClient and CommandClient
CHANGELOG_END
* Tighten result type
Command execution can't result in a sequencer error
* New helper method for extracting used contracts
* New error clause
* Add a DAO query for the maximum time of contracts
* Implement algorithm for finding ledger time
CHANGELOG_BEGIN
CHANGELOG_END
* fixup ledgerTimeHelper
* Use new ledger time algorithm
* Mark LET/MRT as deprecated
CHANGELOG_BEGIN
- [Ledger API] DAML ledgers have switched to a new ledger time model.
The ledger_effective_time and maximum_record_time fields of command submission are deprecated,
the ledger time of transactions is instead set automatically by the ledger API server.
Ledger time is no longer strictly monotonically increasing, but only follows causal monotonicity:
ledger time of transactions is greater than or equal to the ledger time of any used contract.
See `#4345 <https://github.com/digital-asset/daml/issues/4345>`__.
CHANGELOG_END
* Add ledger time skew check
* Remove command updater
LET/MRT are now deprecated, this class is now useless
* Remove old time model validator
* Switch to new time model check: kvutils
* Switch to new time model check: in-memory ledger
* Switch to new time model check: SqlLedger
* Use initial ledger config
* Ignore user provided LET
* Use TimeProvider in submission services
* Use deduplication_time in daml-script runner
- Also remove unnecessary command completion output of CommandTracker.
- Remove usage of maximum record time in CommandTracker.
* Use arbitrary default value for deduplication time
* Use built-in Instant ordering
* Remove obsolete test
* Remove obsolete test: CommandStaticTimeIT
* Refactor test: TransactionMRTCompliance
* Disable test: CommandTrackerFlow timeout
* thread maxDeduplicationTime through to CommandTracker
* Improve test
* Refactor command client configuration
* Deduplication time should always use UTC
* Add missing method in TimedIndexService after rebase
* Put more details into the deduplication error response.
* Use system time for command dedup submittedAt.
* Use explicit UTC time source in command validator
* Revert CommandTracker[Flow] to previous completion-recovering-behavior
* Adapt scala client command config to new config params
Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Support uploading DARs to the trigger service
This PR adds a new `upload_dar` endpoint that accepts a DAR as a
multi-part form request and adds it to the list of compiled packages.
I’ve also made the DAR passed in on startup optional now given the new
endpoint.
There is no endpoint for deleting a DAR so far but there is none on
the ledger API either so I think this not particularly urgent.
changelog_begin
changelog_end
* Address review comments
Previously the http endpoint for starting a trigger would always
return immediately. Based on the recent refactorings, we now do the
non-IO trigger initialization synchronously and return a failed http
status code with an error message.
This also refactors the code to only have one (mutable) set of
compiled packages which is a prerequisite for dynamic package uploads.
changelog_begin
changelog_end
* sandbox: Fail to start if a time mode is not explicitly specified.
CHANGELOG_BEGIN
- [Sandbox] Sandbox is switching from Static Time mode to Wall Clock
Time mode as the default. To ensure that our users know about this,
for one version, there will be no default time mode. Instead, users
will have to explicitly select their preferred time mode by means of
the `--static-time` or `--wall-clock-time` switches. In the next
release, Wall Clock Time will become the default, and users who are
happy with the defaults will no longer need to specify the time mode.
CHANGELOG_END
* daml-script|triggers: Specify time mode when testing against Sandbox.
* daml-assistant: Default the Sandbox to wall clock time.
CHANGELOG_BEGIN
- [DAML Assistant] Initializing a new DAML project adds a switch to
``daml.yaml`` to ensure Sandbox can continue to start with ``daml
start``::
sandbox-options:
- --wall-clock-time
CHANGELOG_END
* docs: Update the DAML Script and Triggers docs to use Wall Clock time.
It's now what Sandbox will use by default when using `daml init`.
* docs: Change the Quickstart to run Sandbox in wall clock time.
This explains why the contract IDs may vary.
It also updates the manual release testing script to match.
Previously parts of the initialization, in particular, the code for
finding the filter and the heartbeat were part of the Runner. This led
to an akward API and didn’t really make any sense.
Now all of this code is part of a pure `Trigger.fromIdentifier`
method and the runner only takes care of actually running the
ledger. This could also be useful for the trigger service where we
might want to synchronously call `getIdentifier` so users get some
indication if there request even points to a valid trigger
directly. However, this is not tackled by this PR.
changelog_begin
changelog_end
Previously the runner class was in a weird state where it was specific
to a DAR but not to an individual trigger. This meant that you had to
pass around a fair bit of state which got a bit awkward. This PR
addresses this by making the trigger class specific to the trigger.
It also now accepts `CompiledPackages` instead of a DAR which should
make it easier in the trigger service to support dynamic package
uploads.
changelog_begin
changelog_end
Previously we assumed that the module name was globally unique in the
DAR which is definitely not guaranteed. Now we instead detect the
package id of the trigger library based on the type of the trigger we
are running which doesn’t fall apart if there are multiple versions of
the trigger library.
I’ve also removed the check for the package id of the trigger library
since I’d like the trigger runner to be backwarts compatible from now on (we
didn’t break that in a while).
This is slightly ugly since the Runner class is currently not specific
to a single trigger but only the individual methods are aware of the
specific trigger identifier. I’ll refactor this in a separate PR.
changelog_begin
changelog_end
* Share test certificates
This is primarily an attempt at making sure my contribution stats
remain negative but I think it’s a nice cleanup. The only difference
in the certs used by daml-helper which are now used everywhere is that
they use a different CN for the CA and the server. This is required to
make openssl happy (which is used by the daml-helper).
changelog_begin
changelog_end
* Fix script and trigger tests
This adds CLI parametrs for connecting via TLS following the scheme
used by navigator, extractor and `daml ledger`.
changelog_begin
- [DAML Script] Support TLS. Enable it by passing ``--tls``. You can
set certificates for client authentication via ``--pem`` and
``-crt`` and a custom root CA for validating the server certificate
via ``--cacrt``.
- [DAML Triggers - Experimental] Support TLS. Enable it by passing ``--tls``. You can
set certificates for client authentication via ``--pem`` and
``-crt`` and a custom root CA for validating the server certificate
via ``--cacrt``.
changelog_end
