Commit Graph

135 Commits

Author SHA1 Message Date
Remy
66ae38da6c
LF: control of input value version in the engine (#6828)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-13 21:15:52 +02:00
Moritz Kiefer
00aacf21d7
Support setTime/getTime in DAML Script service (#7120)
fixes #7001

changelog_begin
changelog_end
2020-08-13 12:26:17 +00:00
Andreas Herrmann
f764c2f627
Support submitMustFail in DAML Script Service (#7076)
* submitMustFail test-case

* script compat test for submitMustFail

changelog_begin
changelog_end

* Support submitMustFail in DAML Script Service

Pushes handling of `submitMustFail` into the DAML script interpreter
rather than DAML script itself.

* fmt

* Simplify IdeClient.submitMustFail

* Comment on case we don't expect to hit

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-12 08:54:34 +00:00
Moritz Kiefer
bbc2fe56c4
Error out on SResultScenarioCommit in DAML script service (#7089)
This was leftover from my first attempt at implementing the script
service by piggybagging on top of the scenario primitives. The current
implementation instead relies on the codepaths used on an actual
ledger where we get SResultFinalValue at the end so we can kill all of
this.

changelog_begin
changelog_end
2020-08-11 16:37:03 +00:00
Samir Talwar
27f76c4386
Use Future.unit instead of Future.successful(()). (#7080)
And in one instance, `Resource.unit`.

I just think it's easier to read. Too many parentheses make Samir a dull
boy.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-11 09:18:50 +00:00
Moritz Kiefer
e1236d42ed
Make ApplicationId in DAML Script configurable (#7074)
This is set per participant since it is similar to the token file.

fixes #7029

changelog_begin

- [DAML Script/DAML REPL] You can now configure the application id via
  `--application-id` or the `--participant-config`. This is primarily
  useful if you are working against a ledger with authentication and
  need to match the application id in your token.

changelog_end
2020-08-10 17:24:38 +02:00
Moritz Kiefer
733a43aeea
Move ApplicationId in daml-script to ledger client (#7070)
This makes much more sense since for things like the JSON API and the
script service, we don’t actually have the application id as a
separate parameter.

I’ve also cleaned up all the arbitrary hardcoded application id in
various tests in favor of a DEFAULT_APPLICATION_ID value.

Next step is #7029

changelog_begin
changelog_end
2020-08-10 13:22:51 +02:00
Stephen Compall
96624a7677
use -Ywarn-unused for all Scala code (#6907)
* add -Ywarn-unused to all scalac options

* remove some unused arguments

* remove some unused definitions

* remove some unused variable names

* suppress some unused variable names

* changeExtension doesn't use baseName

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* work around no plugins in scenario interpreter perf tests

* remove many more unused things

* remove more unused things, restore some used things

* remove more unused things, restore a couple signature mistakes

* missed import

* unused argument

* remove more unused loggingContexts

* some unused code in triggers

* some unused code in sandbox and kvutils

* some unused code in repl-service and daml-script

* some unused code in bindings-rxjava tests

* some unused code in triggers runner

* more comments on silent usages

- suggested by @cocreature; thanks

* fix missing reference in TestCommands

* more unused in triggers

* more unused in sandbox

* more unused in daml-script

* more unused in ledger-client tests

* more unused in triggers

* more unused in kvutils

* more unused in daml-script

* more unused in sandbox

* remove unused in ledger-api-test-tool

* suppress final special case for codegen unused warnings

.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
      implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
                                                         ^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
      implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
                                                               ^

* one more unused in daml-script

* special scaladoc rules may need silencer, too

* unused in compatibility/sandbox-migration

* more commas, a different way to `find`

- suggested by @remyhaemmerle-da; thanks
2020-08-07 13:16:09 -04:00
Andreas Herrmann
322c9eb20b
Support createAndExerciseCmd in DAML Script Service (#7022)
* testCreateAndExercise test-case

* CreateAndExerciseCommand in DAML Script service

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-06 13:44:53 +00:00
Andreas Herrmann
4ced59102b
Support exerciseByKeyCmd in DAML Script Service (#7026)
* Test exerciseByKeyCmd in DAML Script service

* ExerciseByKeyCommand in DAML Script service

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-08-05 20:24:22 +00:00
Moritz Kiefer
4065bfdece
Move ChoiceName -> String conversion to the edges in DAML Script (#7012)
Just gives us a bit more typesafety until we really need to drop it
and avoids a bunch of ugly ChoiceName.assertFromString

changelog_begin
changelog_end
2020-08-05 15:43:03 +00:00
Moritz Kiefer
8d5ce1136b
Push value conversion into ScriptLedgerClient implementations (#7010)
For the script service, we don’t need any conversion so storing SValue
is faster (and easier since converting back needs type info). For
other client, calling `toValue` is easy enough.

changelog_begin
changelog_end
2020-08-05 16:44:35 +02:00
Moritz Kiefer
ad9d8c24ed
Extend the scenario service with DAML Script support (#6929)
* Extend the scenario service with DAML Script support

This adds most of the infrastructure for running DAML Script via the
scenario service which means it runs as part of DAML Studio and `daml
test`. This is hidden behind a feature flag so we can land this and
parallelize the remaining tasks. The main things that are missing are:

1. `createAndExerciseCmd` and `exerciseByKeyCmd`.
2. Party management needs some work and listing parties is
unsupported.
3. Time management
4. Potentially some better error handling (we need to go through
   SResult and SError and see what is relevant for us).

Overall, it is already in a very usable state and there is a decent
range of tests.

closes #3688

changelog_begin
changelog_end

* Update compiler/damlc/daml-ide-core/src/Development/IDE/Core/Rules/Daml.hs

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

* Fix name for actor system and pool

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2020-08-05 12:56:09 +02:00
Stephen Compall
a8bc188ce3
remove unused definitions, params, args from daml-script Scala code (#6987)
CHANGELOG_BEGIN
CHANGELOG_END
2020-08-04 18:23:50 +00:00
Moritz Kiefer
5ceeb476c8
Remove duplicated tls config parsing from DAML Script/Triggers (#6947)
This was not only unnecessarily duplicated, it also had a bug where
`--crt` behaved like `--pem` instead of setting the cert chain.

I didn’t add new tests since it seems like the wrong place to test
config parsing of a library. We do have tests for TLS in general for
both DAML Script and DAML Triggers.

changelog_begin
changelog_end
2020-07-31 18:25:18 +02:00
Andreas Herrmann
4b1438276c
Update Bazel 2.1.0 --> 3.3.1 (#6761)
* Upgrade nixpkgs revision

* Remove unused minio

It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.

* Update Bazel on Windows

changelog_begin
changelog_end

* Fix hlint warnings

The nixpkgs update implied an hlint update which enabled new warnings.

* Fix "Error applying patch"

Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.

* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681

Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.

```
git diff ':(exclude)WORKSPACE'
```

* Update rules_nixpkgs

* nixpkgs location expansion escaping

* Drop --noincompatible_windows_native_test_wrapper

* client_server_test using sh_inline_test

client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.

This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.

* daml_test using sh_inline_test

* daml_doc_test using sh_inline_test

* _daml_validate_test using sh_inline_test

* damlc_compile_test using sh_inline_test

* client_server_test find .exe on Windows

* Bump Windows cache for Bazel update

Remove `clean --expunge` after merge.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-07-23 09:46:04 +02:00
Samir Talwar
adcbfb45a1
sandbox-classic: Move the configuration from sandbox-common. (#6810)
* sandbox-common: Move the ledgerIdMode up in the configuration.

* sandbox-classic: Move the configuration from sandbox-common.

CHANGELOG_BEGIN
CHANGELOG_END

* Fix miscellaneous warnings caught by IntelliJ IDEA.
2020-07-21 14:43:53 +00:00
Robert Autenrieth
7ce9748066
Split sandbox code into separate packages (#6695)
* Move public code into daml-integration-api

CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
2020-07-17 17:06:06 +02:00
Remy
c32355b9fd
DAML-LF: compute the newest version of LF used by a package (#6747)
We add a method to CompiledPackages that returns the latest language version that package uses directly or indirectly through is dependencies.

This will be used in the context of #5164

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-17 09:40:21 +02:00
Gary Verhaegen
8176fb0c8d
fix the the typo (#6723)
```
s/the the /the /
```

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-14 16:55:33 +00:00
Moritz Kiefer
02c59d4f2a
Revert ANF changes and add a testcase for evaluation order (#6645)
* Revert ANF changes and add a testcase for evaluation order

After careful consideration, we decided that the change in evaluation
order that was accidentally introduced by the ANF changes should be
considered a breaking change or arguably even a bug and should not
land in 1.3.0.

Therefore, this PR reverts the following commits:

1. 353d0da6f7
2. a45b51042f
3. 04c7b2af7f
4. a624dd7242
5. b3aab72cee

Other PRs mostly had trivial merge conflicts that I resolved. The two
most interesting ones here are probably

1. https://github.com/digital-asset/daml/pull/6576 which was easy to
   resolve and the change to return SEValue instead of SExpr is still
   nice and useful even if we do not need the guarantees.
2. it https://github.com/digital-asset/daml/pull/6542 which required
   some changes since the constructors changed. If you want to review
   those changes in detail (they are pretty straightforward so not too
   important), it’s probably easiest to check out this PR and run
   ```
   git diff 2cd2a8f2a8
   daml-lf/interpreter/src/main/scala/com/digitalasset/daml/lf/speedy/Compiler.scala
   ```
   to see the diff to the parent commit of the first commit that
   introduced ANF.

changelog_begin
changelog_end
2020-07-08 12:40:12 +02:00
Stephen Compall
4cd419623a
replace traverseU and sequenceU with traverse and sequence (#6594)
* replace traverseU and sequenceU with traverse and sequence

- with -Ypartial-unification on, the extra Unapply typeclass lookup is
  unnecessary

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* limit imports; we only need *> and void
2020-07-07 15:01:58 +00:00
Moritz Kiefer
c0a313c6c5
Fix port handling in DAML Script over JSON API (#6629)
If you have something like `http://localhost:8080` the port is handled
correctly. However, if you have `http://localhost/abc:8080` the port
will silently be ignored (`http://localhost:8080/abc` would be
correct). That is clearly wrong so this PR fixes it.

changelog_begin

- [DAML Script] Fix an issue where the `port` was ignored for
  non-empty paths in the url when running DAML Script over the JSON API.

changelog_end
2020-07-06 17:34:56 +00:00
Moritz Kiefer
907e4fe678
Fix project versions of DAML Script and DAML Triggers (#6622)
GHC has weird restriction on version numbers which damlc inserits so
we need to use `ghc_version` instead of `sdk_version`. That only makes
a difference for snapshot versions where the `-snapshot.` part is
replaced by `.`.

changelog_begin
changelog_end
2020-07-06 15:07:39 +00:00
Moritz Kiefer
cf4fcb560b
Fix the package version of daml-trigger and daml-script (#6566)
It makes no sense to keep this at 0.0.1.

changelog_begin

- [DAML Script] The DAML Script library now has the version of the
  corresponding SDK.

- [DAML Trigger] The DAML Trigger library now has the version of the
  corresponding SDK.

changelog_end
2020-07-01 19:06:00 +02:00
nickchapman-da
353d0da6f7
Speedy ANF (#6440)
* ANF transformation in Speedy.

The idea behind this PR is to transform speedy expressions into a simpler form where all non-atomic sub-expressions are made explicit by the introduction of let-forms. In particular, for the function-application form. These simpler forms allow the execution engine to take advantage of the atomic assumption, and often removes many additional execution steps. In particular the pushing of continuations to allow execution to continue after a compound expression has been reduced to a value.

changelog_begin
changelog_end

* improve comment

* inline functions relocateA/L

* remove comment about scalafmt

* remove commented out alterative def for transformLet1

* improve code by adding incr methods to DepthA/E

* remove (n == 0) special case in trackBindings

* clarify comment further

* improve validate/go to not consume stack for deeply right-nested let-expressions

* address comments from Remy: be private; use final case case; etc

* rename to unsafeCompilationPipeline

* add back some trailing commas

* remove commented-out debug line

* improve comment

* remove dev/debug code in compilationPipeline

* remove commented out code in SEAppGeneral.execute

* undo unrelated code improvement in SValue.scala

* fix compile. object Anf cannot be private
2020-06-30 20:52:44 +00:00
Moritz Kiefer
c6ec40f2bf
Fix DAML Script test for listKnownParties (#6534)
Apart from the current test being broken since it tested the
vals.get(0) twice 🤦 we can now also test is_local properly
since the issue mentioned in the comment has been fixed in #6533.

changelog_begin
changelog_end
2020-06-30 10:19:20 +02:00
Moritz Kiefer
b0eebba51b
Improve error reporting when running DAML Script over the JSON API (#6481)
It is pretty easy to hit this, e.g., when your templates haven’t been
uploaded to the ledger. Just printing the response doesn’t actually
include the response body which means that you don’t see the actual
error so it’s pretty useless. This PR changes that by printing the
status code and the response body.

All the rest is just test setup to be able to submit a script with a
template that has not been uploaded to the ledger.

changelog_begin
changelog_end
2020-06-25 14:53:58 +02:00
Stephen Compall
a51d0db8ff
set scalac -Xsource:2.13 -Ypartial-unification globally (#6469)
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts

* add now-referenced scalaz-core where needed

* work around bad type signatures in scalatest Aggregating, Containing

* unused Any suppression

* work around bad partial-unification wrought by type alias

* remove unused Conversions import

- not required in 4f68cfc480 either, so unsure how it's survived this long

* work around Future.traverse; remove unused show import

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused bounds

* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed

* longer comment on what the options do

- suggested by @stefanobaghino-da; thanks

* forget Future.traverse, just use scalaz, it knows how to do this
2020-06-24 16:51:24 -04:00
Moritz Kiefer
97c74d4a29
Support multiple auth tokens in DAML Script (#6473)
* Support multiple auth tokens in DAML Script

This piggy backs on top of the already existing --participant-config
feature. While you can argue that it might be slightly confusing that
you have to specify the same participant twice to specify different
auth tokens, I think this actually makes sense: In an ideal
world (ignoring any performance issues) you have one participant per
party anyway and one connection per participant specified in the
config file still seems like a very reasonable model.

changelog_begin

- [DAML Script] You can now use DAML Script with multiple auth
  tokens. This is particularly useful if you are working with the JSON
  API where you can only have one party per token or with an IAM that
  only provides single-party tokens. The tokens are specified in the
  participant configuration passed via `--participant-config` in a new
  ``access_token`` field. The existing `--acess-token-file` flag is still supported if you want to use the same token for all connections. See
  https://docs.daml.com/daml-script/index.html#running-daml-script-against-authenticated-ledgers
  for more details.

changelog_end

* I will never understand rst

changelog_begin
changelog_end
2020-06-24 14:43:29 +02:00
Remy
1b1b4eab2c
Speedy: clean machine builder name (#6427)
* Address comment martin made in #6368

* changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-19 14:58:13 +02:00
Moritz Kiefer
8730555bc6
Clarify docs around getTime (#6425)
changelog_begin
changelog_end
2020-06-19 14:56:53 +02:00
Sofia Faro
f5613785e0
Constant lifting transformation in the LF simplifier. (#6101)
* First draft of constant lifting

changelog_begin
changelog_end

* refactoring

* doing stuff

* run simplifier on template exprs

* remove merge artifact

* Fix ExerciseWithoutActors

* add comments

* fix trace order test

* prefix generated val names with their provenance

* Verification tool bugfix

During the value collection phase, when encountering a record projection on a (yet) undefined value, stop searching this branch instead of throwing an error.

* Bump pattern-match-perf memory limit with cocreature`s blessing

* bump again

* Filter generated identifiers from daml script test runner

changelog_begin
changelog_end

* Fix party literals

* Remove inlineClosedExpr for now.

* Improve comments

* Reset script test locations

* Unhashmap

* disable daml-lf-verify quickstart tests for now

Co-authored-by: Gert-Jan Bottu <gertjan.bottu@kuleuven.be>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-06-18 17:36:28 +00:00
Moritz Kiefer
0a8c1eba4e
Integrate time service into DAML Script (#6417)
This integrates the time service into DAML script thereby covering the
main piece of scenarios that was missing from DAML script.

This PR does two things (they are very related and doing them together
makes it much easier to test):

1. It “fixes” `getTime` to return the ledger time in static mode by
   querying the ledger time service instead of defaulting to the Unix
   epoch which is pretty useless and I would consider the old behavior
   a bug. We keep the old behavior via the JSON API since there is no
   time service.

2. It adds `setTime` to set the ledger time via the time service. This
   is only supported in static time mode (sadbonx and other ledgers do
   not expose the time service in wallclock mode because changing time
   makes it not wallclock) or via the JSON API (no time service).

fixes #6220

changelog_begin

- [DAML Script] DAML Script’s `getTime` now correctly handles time
  changes in static time mode and returns the current time by querying
  the time service rather than defaulting to the Unix epoch. Note that
  when run via the JSON API, it still returns the Unix epoch.

- [DAML Script] Add `setTime` to DAML Script which sets the ledger
  time via the ledger API time service. Note that this is only
  supported when running over gRPC in static time mode.

changelog_end
2020-06-18 18:25:24 +02:00
Remy
5a9e7ebc7c
Speedy: refactor Machine builders (#6368)
In this PR we cleanup the constructor for the speedy Machine.

* We remove the `case`  keyword since `Machine` is a stateful class,
* We replace the pre-existing builders with
  + one generic builder `Machine.apply`,
  + scenario specific builder,

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-18 15:39:55 +02:00
Moritz Kiefer
53bddb54d7
Add listKnownParties to DAML Script (#6369)
* Add listKnownParties to DAML Script

This is particularly useful if you want easy access to a party that
has already been allocated since partyFromText is bad.

For now this is not supported in the JSON API. It should be possible
to add it but I consider it fairly low priority so omitting for now.

changelog_begin

- [DAML Script] Add ``listKnownParties`` and ``listKnownPartiesOn`` to
  query the corresponding ListKnownParties endpoint in the party
  management service.

changelog_end

* Fix `daml script-test` tests by running them sequentially.

changelog_begin
changelog_end
2020-06-17 07:50:29 +02:00
Simon Maxen
5329a4f97f
Ledger id requirement (#6323)
* Add option based constructor for LedgerIdRequirement

changelog_begin
changelog_end

* Make option based consructor the default, deprecate old constructor

* Update with review comments
2020-06-12 15:00:32 +00:00
Stefano Baghino
9af85e56e9
Introduce DAML-LF value caching for transaction service (#6052)
* Introduce DAML-LF value caching for transaction service

Allows to keep the DAML-LF values in the most recently indexed events in memory,
so that they don't have to be deserialized on the client from their Protobuf encoding.

Closes #6044

CHANGELOG_BEGIN
[Sandbox] The --max-lf-value-translation-cache-entries option allows to set a
number of events for which DAML-LF values are cached. Could reduce latency in
serving transactions for consumers that are reasonably fast.
CHANGELOG_END

* Add missing dependency

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076003

* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/sandboxnext/Runner.scala

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428071324

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076905

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428081294

* Fix fatal warnings

* //ledger/caching has to be used whenever sandbox is run

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-05-26 08:33:53 +00:00
Remy
9e456a1016
DAML-LF: Kill RelativeContractId (#5991)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-25 22:30:45 +02:00
nickchapman-da
44d0128f29
Add control to speedy compiler to choose if stack-trace support occurs at runtime. (#6070)
Choices for `stacktracing` are `NoStackTrace` / `FullStackTrace`.

Adapt code so the selection is made by the original caller:
- `engine`
- `scenario-service`
- `repl-service`
- `daml-script` runner
etc

Currently, all callers pass `FullStackTrace` (the existing behaviour), except for the
exploration dev-code: `daml-lf/interpreter/perf/src/main/scala/com/daml/lf/explore`.

The idea is that once this control is in place, we can discuss if we can change how we
might expose it to the user, and/or perhaps change the default behaviour to have
`stacktracing` disabled.

changelog_begin
changelog_end
2020-05-22 16:36:03 +01:00
nickchapman-da
ee4f89378d
Speedy Tail Call Optimization (#6003)
* Speedy Tail call optimization

The goal of this PR is to achieve Tail call optimization #5767

Tail call optimization means that tail-calls execute without consuming resources. In particular, they must not consume stack space.

Speedy has two stacks: The `env`-stack and the `kontStack`. For an optimized tail call in Speedy, we must not extend either. In Speedy, all function calls are executed via the code `enterFullyAppliedFunction`. The behaviour of this code (prior to this PR) is as follows:

(1) Push the values of all args and free-variables on the env-stack (because that's where the code expects to find them), and
(2) Push a KPop continuation on the kontStack, which will restore the env-stack to its original size before returning to the code which made the function call.

We must stop doing both these things. We achieve this as follows:

(1) We address function args and free-vars via a new machine component: the current `frame`.
(2) We make continuations responsible for restoring their own environment.

As well as achieving proper tail calls, we also gain a performance improvement by (a) removing the many pushes to the env-stack, and (b) never having to push (and then later re-enter) a KPop continuation. The args array and the free-vars array already existed, so there is no additional cost associated with constructing these array. The only extra costs (which are smaller than the gains) are that we must manage the new `frame` component of the machine, and we must record frame/env-size information in continuations so they can restore their environment.

To make use of the frame, we need to identify (at compile time) the run-time location for every variable in a speedy expression. This is done during the `closureConvert` phase. At run-time, an environment is now composed of both the existing env-stack and the frame. The only values which now live on the env-stack are those introduced by let-bindings and pattern-match-destructuring. All other are found in the frame.

Changes to SEExpr:
- Introduce a new expression form `SELoc`, with 3 sub classes: SELocS/SELocA/SELocF to represent the run-time location of a variable.
- SELocS/A/F execute by calling corresponding lookup function in Speedy: getEnv(Stack,Arg,Free).
- SEMakeClo takes a list of SELoc instead of list of int.
- During closure conversion all SEVar are replaced by an SELocS/A/F.
- SEVar are not allowed to exist at run-time (just as SEAbs may not exist).
- We adapt the synthesised code for SEBuiltinRecursiveDefinition: FoldL, FoldR, EqualList

It is worth noting the prior code also had the notion of before/after closureConvert, but SEVar was used for both meanings: Prior to closureConvert it meant the relative-index of the enclosing binder (lambda,let,..). After closureConvert it meant the relative-offset from the top of the env-stack where the value would be found at run-time. These are not quite the same! Now we have different sub-types (SEVar vs SELoc), this change of mode is made more explicit.

Run-time changes:
- Use the existing `KFun` continuation as the new `Frame` component.
- `KFun` allows access to both the args of the current application, and the free-vars of the current closure.
- A variable is looked up by it's run-time location (SELocS/A/F)
- A function application is executed (`enterFullyAppliedFunction`), by setting the machine's `frame` component to the new current `KFun`.
- When a continuation (KArg, KMatch, KPushTo, KCatch) is pushed, we record the current Frame and current stack depth within the continuation, so when it is entered, it can call `restoreEnv` to restore the environment to the state when the continuation was pushed.

Changes to Compiler:

- The required changes are to the `closureConvert` and `validate`.
- `closureConvert` `remaps` is now a `Map` from the `SEVar`s relative-index to `SELoc`
- `validate` now tracks 3-ints (maxS,masA,maxF)

changelog_begin
changelog_end

* changes for Remy

* Changes for Martin

* test designed explicitly to blow if the free variables are captured incorrectly

* address more comments

* improve comment about shift in Compiler
2020-05-20 12:37:52 +01:00
Martin Huschenbett
caedc72551
Implement a simple profiler for DAML scenarios (#5957)
* Implement a simple profiler for DAML scenarios

The profiler runs a single scenario and records timing information when
each function (and some other closures) are entered and left. The
resulting information can be visualized as a flamegraph using
[speedscope](https://www.speedscope.app/).

The profiler works by instrumenting the CEK machine at the heart of
DAML Engine. Unfortunetaly, this causes a very small overhead on
non-profiling runs too. However, in my benchmarks I could not measure
any significant impact on the overall runtime at all. More precisely,
the overhead is as follows:

Every closure now has an additional field called `label`. In
non-profiling runs this field is always set to `null`. This field needs
to be allocated, copied whenever we copy a closure and scanned during
garbage collection. Additionally, whenever we enter a closure, we check
this field and whenever it is _not_ `null`, i.e. never during
non-profiling runs, we record an "open event" and set up a hook for the
corresponding "close event". Thus, the additional cost during
non-profiling runs are a single pointer comparison and a jump beyond
the "then branch".

Since this is still very much in active development, there are no
documentation, other than an entry in a README, and no tests yet. They
will come before we promote this. However, the UX will look very
different then since we already have plans to significantly change it.

CHANGELOG_BEGIN
CHANGELOG_END

* Run scalafmt

* Make profiling argument to PureCompiledPackges optional

* Fix a bunch of tests

CHANGELOG_BEGIN
CHANGELOG_END

* scalafmt is so annoying

CHANGELOG_BEGIN
CHANGELOG_END

* Apply simple suggestions

CHANGELOG_BEGIN
CHANGELOG_END
2020-05-18 17:20:44 +00:00
Moritz Kiefer
770a2763b0
Fix a stackoverflow in the script runner (#5989)
Previously, we used the stack to recurse when filling in the command
results which obviously breaks once you have large multi-command
submissions.

You probably want to view the diff with whitespace disabled.

changelog_begin

- [DAML Script] Fix a bug where large multi-command transactions
  produced a stack overflow.

changelog_end
2020-05-15 16:04:18 +02:00
Moritz Kiefer
2853a20eb9
Make max-inbound-message-size in DAML script/repl configurable (#5996)
Fixes #5592

The CLI syntax and the defaults follow the JSON API here.

changelog_begin

- [DAML Script] The maximum inbound message size can now be configured
using `--max-inbound-message-size``. This matches the flag in the JSON
API.

- [DAML REPL] The maximum inbound message size can now be configured
using `--max-inbound-message-size``. This matches the flag in the JSON API.

changelog_end
2020-05-15 07:22:05 -04:00
Remy
861724e4fd
Engine: cleanup SEVal (#5859)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-07 11:44:19 +02:00
nickchapman-da
53b24793f4
Simplify and clarify the public interface to Speedy. (#5881)
* Simplify and clarify the public interface to Speedy.

- Remove `isFinal`. A client just uses `run()`.
- Remove `toSValue`. The value in available in `SResultFinalValue(v: SValue)`.
- A client never directly access the `.ctrl` (or `.returnValue`) components.
- A client may use `setExpressionToEvaluate(expr)` to evaluate a new expression on an existing machine.

changelog_begin
changelog_end

* remove while loop which executes just once

* avoid unnecessary mutation when running speedy
2020-05-07 08:55:16 +00:00
nickchapman-da
b318cb0299
Optimize Speedy machine.ctrl (#5811)
Remove the `Ctrl` trait and separate `Machine.ctrl: Ctrl` into `Machine.ctrl: SExpr` and `Machine.returnValue: SValue` instead. This allows for avoiding dynamic dispatch on `ctrl` and instead allows for checking a pointer for `null` to decide if we have an expression that needs further break-down or a return value ready to be passed to the next continuation.

To make this check really only a pointer comparison we also needed to remove the abomination of "fully applied partially applied primitives". In order to achieve this, we check whether a PAP will be fully applied afterward when applying the last argument.

On the `collect-authority` benchmark, this increases throughput by around 13%, on another more computation heave benchmark by about 21%.

`collect-authority` benchmark on `master`:
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
  112.361 ±(99.9%) 1.965 ms/op [Average]
  (min, avg, max) = (107.047, 112.361, 120.745), stdev = 3.493
  CI (99.9%): [110.396, 114.326] (assumes normal distribution)
```

`collect-authority` benchmark on this branch:
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
  98.196 ±(99.9%) 1.933 ms/op [Average]
  (min, avg, max) = (91.580, 98.196, 105.478), stdev = 3.436
  CI (99.9%): [96.263, 100.129] (assumes normal distribution)
```

computation heavy benchmark on master
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
  44.030 ±(99.9%) 0.742 ms/op [Average]
  (min, avg, max) = (42.124, 44.030, 46.781), stdev = 1.319
  CI (99.9%): [43.289, 44.772] (assumes normal distribution)
```

computation heavy benchmark on this branch:
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
  36.222 ±(99.9%) 0.580 ms/op [Average]
  (min, avg, max) = (34.897, 36.222, 39.787), stdev = 1.031
  CI (99.9%): [35.643, 36.802] (assumes normal distribution)
```

changelog_begin
changelog_end
2020-05-06 20:44:50 +02:00
Remy
b59f024eec
DAML-Script: cleanup to prepare #5811 (#5871)
* DAML-SCRIPT: cleanup to prepare #5811

* a bit more.

CHANGELOG_BEGIN
CHANGELOG_END

* Address Moritz's review

* Update daml-script/runner/src/main/scala/com/digitalasset/daml/lf/engine/script/Runner.scala

Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>

Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
2020-05-06 16:23:59 +00:00
Stephen Compall
2b23b41391
JSON decoding support for GenMap (#5223)
* add GenMap to the "all types" test generators

* report bad GenMap format with DeserializationError, not MatchError

* document GenMap JSON

* notes on missing features

* enable -Xsource:2.13 in transaction

* make an Order instance for Value resolvable, but unimplemented

* use the skeleton from SValue ordering to make a Value ordering skeleton

* add Party Order

* add Order instance for SortedLookupList

* add Order for FrontStack, deriving everything

* factor the Order lookup, and tie a knot in the recursive Value instances

* we're going to need this Iterator thing again

* replacing Order#contramap with version that supports equalIsNatural

* use new equalBy, orderBy for FrontStack, SortedLookupList, ImmArray

* _2 comparator, upgrade Name Equal to an Order

* incorporate lookup for enums, variants into Value order; record/struct cases

* Enum/Variant comparison

* looking up the singleton implicitly won't work for non-`object`s, alas

* test Order laws for values of all primitive types

* test Order laws for record and variant types

* test Order laws for enum types

* test that enum strings are not compared

* use checkLaws for Value Equal as well

* test that enums match order to constructor rank

* factor genAddend and genAddendNoListMap

* reintroduce Order for TypedValueGenerators

* more addend order

* record, variant order cases

* record cons order

* deriving Order while decoding from JSON

* make ApiCodecCompressed's Cid codec based on the typeclass

* test how the Value ordering and the underlying projected value orderings line up

- hint: they don't, yet
- this is also a template for how we'll check the fidelity with SValue
  ordering

* test how the Value ordering and SValue ordering line up

- hint: they don't, yet

* typed Arbitrarys have access to Order

* produce proper ValueGenMap

* inj requires Order, sometimes

- we encode this as "all the time" but there is a type-level unification
  approach to remove this requirement in some cases

* make inj a function

* test that order doesn't matter for JSON decoder

* use Utf8 order for TVG text; don't pretend that base equal works

* sort JSON GenMaps, and check for duplicates

* make injarb use IntroCtx

* remove stray import

* Order instances for Bytes, Hash, AbsoluteContractId

* require Order[Cid] to decode JSON to LF values

* clean up map reordering test

* remove unused Instant instance

* fake Order instance no longer needed, valid instance defined

* test parity of global AbsoluteContractId order and SContractId order

* bazel fmt

* test AbsoluteContractId Order lawfulness

* test duplicate key detection

CHANGELOG_BEGIN
- [JSON API] Prepare full support for the planned GenMap primitive type.
  See `issue #5031 <https://github.com/digital-asset/daml/issues/5031>`_.
CHANGELOG_END
2020-05-05 10:27:37 -04:00
nickchapman-da
572b21e882
Speedy: run() dont step() (#5814)
Speedy: run() dont step()

- Running the Speedy machine with  `run()` instead of `step()`
- Remove: `SResultContinue`
- Add: `SResultFinalValue(_)`

We change the top level control of Speedy: from machine.step() to machine.run, with the control of stepping while the machine returns SResultContinue moved into speedy itself. (And so SResultContinue is removed in favour of SResultFinalValue.) The main advantage of this approach is that the tight while loop can be moved inside the exception handler, rather than having to wrap the handler every step.

changelog_begin
changelog_end
2020-05-04 16:57:47 +01:00