* Speedy Tail call optimization
The goal of this PR is to achieve Tail call optimization #5767
Tail call optimization means that tail-calls execute without consuming resources. In particular, they must not consume stack space.
Speedy has two stacks: The `env`-stack and the `kontStack`. For an optimized tail call in Speedy, we must not extend either. In Speedy, all function calls are executed via the code `enterFullyAppliedFunction`. The behaviour of this code (prior to this PR) is as follows:
(1) Push the values of all args and free-variables on the env-stack (because that's where the code expects to find them), and
(2) Push a KPop continuation on the kontStack, which will restore the env-stack to its original size before returning to the code which made the function call.
We must stop doing both these things. We achieve this as follows:
(1) We address function args and free-vars via a new machine component: the current `frame`.
(2) We make continuations responsible for restoring their own environment.
As well as achieving proper tail calls, we also gain a performance improvement by (a) removing the many pushes to the env-stack, and (b) never having to push (and then later re-enter) a KPop continuation. The args array and the free-vars array already existed, so there is no additional cost associated with constructing these array. The only extra costs (which are smaller than the gains) are that we must manage the new `frame` component of the machine, and we must record frame/env-size information in continuations so they can restore their environment.
To make use of the frame, we need to identify (at compile time) the run-time location for every variable in a speedy expression. This is done during the `closureConvert` phase. At run-time, an environment is now composed of both the existing env-stack and the frame. The only values which now live on the env-stack are those introduced by let-bindings and pattern-match-destructuring. All other are found in the frame.
Changes to SEExpr:
- Introduce a new expression form `SELoc`, with 3 sub classes: SELocS/SELocA/SELocF to represent the run-time location of a variable.
- SELocS/A/F execute by calling corresponding lookup function in Speedy: getEnv(Stack,Arg,Free).
- SEMakeClo takes a list of SELoc instead of list of int.
- During closure conversion all SEVar are replaced by an SELocS/A/F.
- SEVar are not allowed to exist at run-time (just as SEAbs may not exist).
- We adapt the synthesised code for SEBuiltinRecursiveDefinition: FoldL, FoldR, EqualList
It is worth noting the prior code also had the notion of before/after closureConvert, but SEVar was used for both meanings: Prior to closureConvert it meant the relative-index of the enclosing binder (lambda,let,..). After closureConvert it meant the relative-offset from the top of the env-stack where the value would be found at run-time. These are not quite the same! Now we have different sub-types (SEVar vs SELoc), this change of mode is made more explicit.
Run-time changes:
- Use the existing `KFun` continuation as the new `Frame` component.
- `KFun` allows access to both the args of the current application, and the free-vars of the current closure.
- A variable is looked up by it's run-time location (SELocS/A/F)
- A function application is executed (`enterFullyAppliedFunction`), by setting the machine's `frame` component to the new current `KFun`.
- When a continuation (KArg, KMatch, KPushTo, KCatch) is pushed, we record the current Frame and current stack depth within the continuation, so when it is entered, it can call `restoreEnv` to restore the environment to the state when the continuation was pushed.
Changes to Compiler:
- The required changes are to the `closureConvert` and `validate`.
- `closureConvert` `remaps` is now a `Map` from the `SEVar`s relative-index to `SELoc`
- `validate` now tracks 3-ints (maxS,masA,maxF)
changelog_begin
changelog_end
* changes for Remy
* Changes for Martin
* test designed explicitly to blow if the free variables are captured incorrectly
* address more comments
* improve comment about shift in Compiler
First version of static verification tool.
The current state of the tool:
- Reads DAR files.
- Partially evaluates the code.
- Generates constraints for the field and choice to be verified.
- Passes the constraints to an SMT solver.
- Some basic tests.
* Revert "disable test pending tracking down duplicates"
This reverts commit 8b5f9dfa04.
* activeContracts tells caller the ledgerEnd, rather than accepting activeAt argument
* naive port of http-json tests to sandbox-next
* Revert "naive port of http-json tests to sandbox-next"
This reverts commit 91d4590c90.
* lock before grabbing acs and ledgerEnd offset in InMemoryLedger
- as suggested by @gerolf-da; thanks
trigger all releases from master
The 1.1.0 release went wrong and we had to trash it and release 1.1.1
instead. This is an attempt at identifying and correcting the root
cause behind that incident.
To understand the situation, we need to know how releases worked before
1.0. We had a one-line file called `LATEST` that specifies the git SHA and
version tag for the latest release. A change to that file triggered a
release with the specified release tag, built from the source tree of
the specified commit. The `LATEST` file looked something like:
```
f050da78c9 1.0.0-snapshot.20200411.3905.0.f050da78
```
To mark a release as stable, we would change it to look like this:
```
f050da78c9 1.0.0
```
i.e. simply drop the `-snapshot...` suffix. Even though the commit (and
thus the entire source tree we build from) is the same, we would need to
rebuild almost all of our release artifacts, as they embed the version
tag in various places and ways. That worked well as long as we could
assume we were doing trunk-based development, i.e. all releases would
always come from the same (`master`) branch.
When we released 1.0, and started work on 1.1, we had a few bug reports
for 1.0 that we decided should be resolved in a point release. We
decided that the best way to handle that would be to have a branch
starting on the release commit for 1.0, and then backport patches from
`master` to that branch. We adapted our build process to also watch the
`release/1.0.x` branch and, in particular, trigger a new release build if
the `LATEST` file in that branch changed. That worked well.
The plan going forward was to keep doing regular snapshot releases from
the `master` branch, and create support, point releases ("patch" releases
in semver) from dedicated branches.
On April 30, we made a snapshot release as an RC for 1.1.0, by changing
the `LATEST` file in the `master` branch. That release was built on commit
681c862d. On May 6, we decided to take a new snapshot as the RC for
1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new
latest release.
On May 11, we noticed an issue that broke our builds. Without going into
details, an external artifact we depend on had changed in incompatible
ways. After fixing that on `master`, we reasoned that this would also
break the build of the final 1.1.0 release if we just tried to build
7e448d81 again. But as the target release date was May 13, we did not
want to take a new snapshot after that fix, as that would have included
one more week of work in the release, and given us no time to test it.
So we did what we did for the 1.0 branch, as it had worked well: we
created a branch that forked from `master` at commit 7e448d81 and called
it `release/1.1.x`, then cherry-picked the one fix to our build process to
work around the broken download. When the time came to make the final
1.1.0 build on May 13, we naturally picked the `LATEST` file from the
`release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly,
we did not need to update the target commit to include the "broken
download" fix as, in the meantime, the internet had fixed itself, and we
thus reasoned we should go for the exact code of the RC rather than
include an unnecessary, albeit seemingly harmless, change.
Everything went well with the release process. Tests went well too. Then
we got a report that an application that worked against the latest RC
broke with the final 1.1.0. The issue was that we had built the wrong
commit: by branching off at the point of the _target_ commit for the
latest snapshot, we did not have the change to the `LATEST` file that
designated that commit as the target. So the `LATEST` file in
`release/1.1.x` was still pointing to 681c862d.
I believe the root cause for this issue is the fact that we have
scattered our release process over multiple branches, meaning there is
no linear history of what was released and we are relying on people
being able to mentally manage multiple timelines. Therefore, I propose
to fix our release process so this should not happen again by
linearizing the release process, i.e. getting back to a situation where
all releases are made from a single branch, `master`.
Because we do want to be able to release _for_ multiple release branches
(to provide backports and bugfixes), we still need some way to
accommodate that. Having a single `LATEST` file in the same format as
before would not really work well: keeping track of interleaved release
streams on a single file would not really be easier than keeping track
of multiple branches.
My proposed solution is to instead have a multiline LATEST file, so that
all the release branch "tips" can be observed at the same time, and, as
long as we take care to only advance one release branch at a time, we
can easily keep track of each of them. This is what this PR does.
This required a few changes to our release process. Most notably:
- Obviously, as this is the main point of this PR, the build process has
once again been restricted to only trigger new releases from the
`master` branch.
- As our CI machinery cannot easily be made to produce multiple releases
from a single build, the `check_for_release` step will only recognize
a commit as a release trigger if it changes a single line in the
`LATEST` file. This restriction comes in addition to the existing one
that a release commit is only allowed to change either just the
`LATEST` file or both the `LATEST` and
`docs/source/support/release-notes.rst` files.
- The docs publication process has been changed to update _all_
published versions to display the _latest_ release notes page. This
means that the release notes page will always show you all published
versions, regardless of which version of the documentation you're
looking at. This also means that interleaving release notes correctly on
that page is a manual exercise.
- As per the intention of the new process, the `LATEST` file has been
updated to contained all existing post-1.0 stable releases. It should
also include all existing snapshot releases should we have more than one
at a time (say, should we discover an issue with 1.1.1 that required us
to work on a 1.1.2).
- The `release.sh` script has been dramatically simplified as I felt it
was trying to do too much and porting its existing functionality to a
multi-line `LATEST` file would be too hard.
CHANGELOG_BEGIN
CHANGELOG_END
The workspace for the vendored node wrapper script `@nodejs_linux_amd64`
did previously not record a dependency on the nixpkgs provided node
workspace. This patch enforces that dependency by introducing a dummy
read of the vendored node binary.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Note: this is beta-level software. See documentation for the precise
guarantees this does and does not come with. (Documentation does not
exist at the time of opening this PR, but should exist by the time the
first version of this gets published.)
CHANGELOG_BEGIN
- We now publish Sandbox Next as an **ALPHA** standalone jar.
- We now publish the HTTP JSON API as a standalone jar.
CHANGELOG_END
not sending a live event after reconnecting means that the 'loading'
indicater is never updated to 'false'. we could also not set it to
'true' on the 'close' event, but that can lead to a rather unintuitive
behaviour where data is stale and then suddenly updated without any
indication.
CHANGELOG_BEGIN
CHANGELOG_END
* REPL record dot syntax regression test
* REPL: Apply record preprocessor to expressions
CHANGELOG_BEGIN
- [DAML REPL] Record dot syntax is now handled in expressions entered
into the REPL.
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
For some reason this seems to be failing in particular on Linux but
I’ve now also seen it fail on MacOS. My suspicion is that running it
in parallel, somehow corrupts the cache but I don’t understand the details.
changelog_begin
changelog_end
This image can be built with:
```
bazel run //ledger/sandbox:sandbox-next-image
```
This will, unfortunately, also run a container, which you will need to
kill with _Ctrl+C_. You can use `bazel build`, but this will only build
the image tarballs, which you need to import into your Docker image
repository yourself.
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
- Add `LedgerClient#close`, which will shut down the channel and await
termination. This is optional; the channel will still be shut down on
JVM exit if this method is not called.
CHANGELOG_END
This is offered as a compromise for those who would like to ensure
resources are shut down cleanly, while not making the API more
complicated. I originally wanted to make `LedgerClient.fromBuilder`
return a `Resource[LedgerClient]`, but the `Resource` API would lead to
an increased learning curve for users.
* Implement a simple profiler for DAML scenarios
The profiler runs a single scenario and records timing information when
each function (and some other closures) are entered and left. The
resulting information can be visualized as a flamegraph using
[speedscope](https://www.speedscope.app/).
The profiler works by instrumenting the CEK machine at the heart of
DAML Engine. Unfortunetaly, this causes a very small overhead on
non-profiling runs too. However, in my benchmarks I could not measure
any significant impact on the overall runtime at all. More precisely,
the overhead is as follows:
Every closure now has an additional field called `label`. In
non-profiling runs this field is always set to `null`. This field needs
to be allocated, copied whenever we copy a closure and scanned during
garbage collection. Additionally, whenever we enter a closure, we check
this field and whenever it is _not_ `null`, i.e. never during
non-profiling runs, we record an "open event" and set up a hook for the
corresponding "close event". Thus, the additional cost during
non-profiling runs are a single pointer comparison and a jump beyond
the "then branch".
Since this is still very much in active development, there are no
documentation, other than an entry in a README, and no tests yet. They
will come before we promote this. However, the UX will look very
different then since we already have plans to significantly change it.
CHANGELOG_BEGIN
CHANGELOG_END
* Run scalafmt
* Make profiling argument to PureCompiledPackges optional
* Fix a bunch of tests
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt is so annoying
CHANGELOG_BEGIN
CHANGELOG_END
* Apply simple suggestions
CHANGELOG_BEGIN
CHANGELOG_END
fixes#6005
Take a look at #6005 for details. I verified manually that things
still work as expected. It does require bumping the vscode version to
1.38 which was released almost a year ago so hopefully that should be
safe.
changelog_begin
changelog_end
* update rules_nixpkgs
* Use hermetic nixpkgs cc toolchain
CHANGELOG_BEGIN
CHANGELOG_END
* Work around Bazel's cc toolchain autodetection
* Use --crosstool_top for hermetic cc toolchain
When using --incompatible_enable_cc_toolchain_resolution instead
cc actions still depend on
`external/local_config_cc/builtin_include_directory_paths`
as well as
`external/nixpkgs_cc_toolchain_config/builtin_include_directory_paths`.
* override local_config_cc
* remove unused attribute
* Fix posix toolchain on Windows
* nixpkgs cc toolchain not on Windows
* Fix nixpkgs cc toolchain on MacOS
* nixpkgs cc toolchain uses bin/cc
* Use darwin.binutils on MacOS
* Remove clang(++) and gcc (g++) symlinks
The toolchain only considers `bin/cc` and having the other symlinks
around could lead to confusion
* Use hermetic toolchain in compatibility workspace
* Avoid empty linker flags
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Include Bazel patch to cache exclusive tests on Windows
This includes the patch that we already use on Linux and MacOS to fix
caching of things marked exclusive. I’ve kept in the debugging output
that I added in the last patch for now. While our workaround seems to
be working, I’d like to wait a bit longer in case the issue reappears.
changelog_begin
changelog_end
* Actually bump manifest
changelog_begin
changelog_end
* Fix alunchd killing VMWare process at end of script execution
* Fix alunchd killing VMWare process at end of script execution
CHANGELOG_BEGIN
Fix issue with MacOS Catalina Launchd killing VMWare instance on rebuild (AbandonProcessGrop)
CHANGELOG_END
Bash is not easy to add to dev-env because dev-env depends on Bash. This
has not been an issue so far because Bash behaves in very sensible ways
overall and is mostly backwards compatible.
A recent change to `daml-sdk-head` is, however, using some feature of
Bash 4 that is not evailable in Bash 3. Bash 3 is ancient so in an ideal
world that would not be an issue, but macOS still ships with Bash 3 for
some obscure (licensing) reason.
It turns our that the line
```bash
arr=()
```
in Bash 4 sets the variable `arr` to an empty array, whereas it leaves it
unset in Bash 3. This means that the later use of `arr`, in the case
where no further element has been added to the array, will yield an
error in combination with the `set -u` option we are using.
This PR changes the usage pattern from
```bash
"${arr[@]}"
```
which fails on Bash 3 to
```bash
${arr[@]:-}
```
which works as expected. Note that the quotes have been removed: the
quotes in this case are not useful assuming that the flags themselves
are never multiword. Without the quotes, this evaluates to `""` under
Bash 3 (as the variable is not set), which confuses Bazel because now it
thinks it's asked to build the `""` target, which it has no rule for.
Ignoring the quoting issue, the actual fix is to include `:-` inside the
`{}`, which instructs Bash to replace the variable with what follows the
`-` in case the variable is not set (in this case, an empty string), and
therefore not crash on this specific variable not being set despite the
`set -u` option.
CHANGELOG_BEGIN
CHANGELOG_END
* Include the first feature in compat tests for create-daml-app
This adds the step from the create-daml-app tests that applies the
patch for the messaging feature and tests that things will build. This
is the last step before we can actually run the puppeteer tests which
will turn this into an actual compatibility test.
changelog_begin
changelog_end
* temporarily run all tests
changelog_begin
changelog_end
* Fix exclusions
changelog_begin
changelog_end
* Revert "temporarily run all tests"
This reverts commit 7425dd09cf48f2bfd6064b55d857c76d51afc821.
* Remove accidentally committed messaging.patch
changelog_begin
changelog_end
* Include sources directory in the Bazel cache key
This should hopefully fix the “undeclared inclusion” errors we have
been getting daily on CI
The details are in a comment but the short summary is that
the daily cron job is running in D:\a\1 whereas jobs on the same
machine afterwards run in D:\a\2. Because absolute paths leak in some
places, this fucks things up.
changelog_begin
changelog_end
* Add debugging output to the cache
Previously, we used the stack to recurse when filling in the command
results which obviously breaks once you have large multi-command
submissions.
You probably want to view the diff with whitespace disabled.
changelog_begin
- [DAML Script] Fix a bug where large multi-command transactions
produced a stack overflow.
changelog_end
by adding synchronize call between an alpha-participant-create and the
corresponding beta-participant-exercise.
changelog_begin
changelog_end
We occasionally run into this in Canton
Fixes#5592
The CLI syntax and the defaults follow the JSON API here.
changelog_begin
- [DAML Script] The maximum inbound message size can now be configured
using `--max-inbound-message-size``. This matches the flag in the JSON
API.
- [DAML REPL] The maximum inbound message size can now be configured
using `--max-inbound-message-size``. This matches the flag in the JSON API.
changelog_end
Furthermore, uses a work stealing thread pool for work JdbcLedgerDao has
to perform outside of the database thread pool. This solves some
flakiness observed in conformance tests against PostgreSQL-backed
ledgers.
changelog_begin
changelog_end
* daml-sdk-head: Quote all variables.
And use arrays where necessary.
* daml-sdk-head: Use Bash tests (`[[` and `]]`) rather than `test`.
Most importantly, they support the `&&` and `||` operators, because
they're a shell builtin, not a program.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-sdk-head: Refer to `${BASH_SOURCE[0]}` explicitly.
`BASH_SOURCE` is an array. Expanding an array gets the first element, but this
is unclear. We can clarify by explicitly expanding the first element.
* daml-sdk-head: Use `command -v` instead of `which`, as it's more standard.
* daml-sdk-head: Use semicolons judiciously.
In our first design draft, the maintainers of a contract key were
computed from the template payload. Now, they are computed from the
key itself. The rule `EvUpdCreateWithKeySucceed` still reflect the old
behavior but changing a single index from `t` to `v` fixes the issue.
CHANGELOG_BEGIN
CHANGELOG_END
Currently, a call to `create` for a template with a contract key
evaluates the key even before _executing_ the `create`, which is _way_
too early. See #5967 for examples what this implies.
This change moves the evaluation of the key into the execution of the
`create` and therein _after_ the evaluation of the `ensure` clauses,
which is where it belongs according to the DAML-LF specification.
Unfortunately, the specification does not say anything about the
relative evaluation order of the `agreement`, `signatory`, `observer`
and `key` clauses. Thus, I've deciced to evaluate the key after the
other three, which seems to be its natural place given its optionality.
This fixes#5967.
CHANGELOG_BEGIN
CHANGELOG_END
* Consume language for listing [re]actions in a foldAsync linearly
* eagerly consume as many Futures as possible
* restate one foldAsync test with Consume
* report value that failed to match
* add error locations and new `drain` primitive
* use `drain` to properly handle heartbeats at end of stream
* it doesn't make sense to delay the drain step
* rewrite other websocket test; document syntax
* get rid of StreamState, finally
* split Consume to a separate file
* plan split tests
* functions for consuming SplitSeq
* converting SplitSeq into stream actions
* exercise and check recursively
* add one 100-sized sample to the tests
* don't violate Iou's invariant
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* disable test pending tracking down duplicates
- as discussed with @leo-da
* Add debugging output to inclusion errors
This adds some more debugging output to inclusion errors in Bazel
which should hopefully help us track it down. These are the only call sites
in the Bazel source that can produce them. My suspicion is that it’s
coming from HeaderDiscovery but I’m not entirely sure what is off.
We’ll almost certainly have to add more output once we know which of
those 3 cases we hit but let’s do it step by step.
changelog_begin
changelog_end
* Bump url and hash
changelog_begin
changelog_end
changelog_begin
OVERRIDES CHANGELOG ENTRY FROM 2a8c93a614
[Ledger Integration Kit] Added new metrics for
``daml.index.db.*.translation`` to measure the time spent
translating to and from the serialized DAML-LF values when
fetched from the participant index.
changelog_end
