It should not be an issue, but I know of at least 4 people for whom
having envsubst in dev-env causes their local git to get confused and
spit out many lines of warning on each invocation. This also seems to
make git much slower.
Given that we're only using envsubst in this one place, I think it may
be worth replacing.
CHANGELOG_BEGIN
CHANGELOG_END
The `da` tool has not existed for a while now and it's unlikely any new
user will need to know about the upgrade path. I think starting from
1.2.0 it makes sense not to have that documenation around anymore.
Note that we do have references to the upgrade page in the release
notes, so leaving them as is would break building the documentation. My
preference would have been to turn those references into "external"
links to the relevant version, but unfortunately, these are versions
(`0.12.15` and `0.12.18`) that, for various reasons, we cannot build
anymore and do not have in pre-built form. I have therefore decided to
change the links in the release notes to link to the latest existing
version of that page (which is pretty much what it already did), i.e.
specifically link to `1.1.1`.
CHANGELOG_BEGIN
CHANGELOG_END
* Support version ranges in test tool exclusions
This cleans up the map of exclusions for the ledger-api-test-tool part
of the compat tests. For now, you can only use ranges for the inner
map. In principle we could also add it to the outer map but I’ll leave
that for a separate PR.
I’ve also run the script to update versions so this includes the
latest two snapshots.
fixes#5851
changelog_begin
changelog_end
* Address review comments
changelog_begin
changelog_end
* buildifier
changelog_begin
changelog_end
Previously we ran `daml codegen js` and `daml build` in the tests and
we also ran it twice. While this makes sense for the SDK integration
tests, it doesn’t really make sense for the compatibility tests. These
are SDK-only features so they don’t actually test compatibility. These
steps are super slow and running them in the tests also means that we
run them a quadratic number of times.
This PR moves the `daml build` and the `daml codegen js` step to
separate genrules. In the actual test, the only expensive steps are to
run `yarn install` and the actual jest tests. In principle, we could
probably try to factor out the `yarn install` step as well but I’m a
bit scared of coyping around node_modules so I’ll not attempt to do
that for now.
changelog_begin
changelog_end
Version is taken from the env var (or defaulted to 0.0.0) at build-time.
Since those two packages are not build by default by Bazel, we need to
add the env var to the Bash step where they do get explicitly built.
Fixes#6090.
CHANGELOG_BEGIN
- sandbox and http-api fatjars will now display correct version number.
CHANGELOG_END
* Add JSON output to `damlc inspect-dar`
fixes#3117, fixes#5563
This PR adds a `--json` flag to `damlc inspect-dar` that does pretty
much what you expect. So far it mainly exposes the same information
that you already got from `inspect-dar` but it’s easy to add more as
needed.
I’ve also finally added some documentation for `inspect-dar`.
changelog_begin
- [DAML Compiler] ``damlc inspect-dar`` now has a `--json`` flag to
produce machine-readable output. See
https://docs.daml.com/daml/reference/packages.html#inspecting-dars
for more information.
changelog_end
* Sort packages in human-readable output
changelog_begin
changelog_end
* Update docs/source/daml/reference/packages.rst
Co-authored-by: Rohan Jacob-Rao <rohanjr@gmail.com>
Co-authored-by: Rohan Jacob-Rao <rohanjr@gmail.com>
Choices for `stacktracing` are `NoStackTrace` / `FullStackTrace`.
Adapt code so the selection is made by the original caller:
- `engine`
- `scenario-service`
- `repl-service`
- `daml-script` runner
etc
Currently, all callers pass `FullStackTrace` (the existing behaviour), except for the
exploration dev-code: `daml-lf/interpreter/perf/src/main/scala/com/daml/lf/explore`.
The idea is that once this control is in place, we can discuss if we can change how we
might expose it to the user, and/or perhaps change the default behaviour to have
`stacktracing` disabled.
changelog_begin
changelog_end
Minimal database initialization with schemas for running_triggers and dalfs tables. The user passes in the database URL, username and password in a config string argument (approach and code adapted from the JSON API).
In future the idea is to also create a "service" role with permissions to read and write to the new tables. Then the user can pass in the service role to connect to the database when running the service for real.
* Include puppeteer tests in compat tests
This PR adds the puppeteer based tests to the compatibility
tests. This also means that they are now actually compatibility
tests. Before, we only tested the SDK side.
Apart from process management being a nightmare on Windows as usually,
there are two things that might stick out here:
1. I’ve replaced the `sh_binary` wrapper by a `cc_binary`. There is a
lengthy comment explaining why. I think at the moment, we could
actually get rid of the wraper completely and add JAVA to path in
the tests that need it but at least for now, I’d like to keep it
until we are sure that we don’t need to add more to it (and then
it’s also in the git history if we do need to resurrect it).
2. These tests are duplicated now similar to the `daml ledger *`
tests. The reasoning here is different. They depend on the SDK
tarball either way so performance wise there is no reason to keep
them. However, we reference the other file in the docs which means
we cannot change it freely. What we could do is to make this
sufficiently flexible to handle both the `daml start` case and
separate `daml sandbox`/`daml json-api` processes and then we can
reference it in the docs. There is still added complexity for
Windows but that’s necessary for users as well that want to run
this on Windows so that seems unavoidable. (I should probably also
remove my snarky comments 😇) I’d like to kee it duplicated
for this PR and then we can clean it up afterwards.
changelog_begin
changelog_end
* Bump timeouts
changelog_begin
changelog_end
- reduce verbosity of PAP
- see inside applications & lambdas.
- see rhss/body of lets
- see scrut of case (maybe later we add alts!)
- see the name of the definition ref
- see bools
- see LOC(and-nested-expression) -- interesting where the LOCs are places
- see head of kont stack
changelog_begin
changelog_end
This change extends the bazel rule for `compile_daml` to add a stage which will pretty-print the generated .dar file, suitable for human inspection. The generated file is named with a `.dar.pp` suffix, and will only be generated if explicitly requested as a build target or listed as a dependency.
Make use of the new rule by demanding CollectAuthority.dar.pp when the perf benchmark is built.
changelog_begin
changelog_end
Explore the execution of the speedy machine on small examples taken from a dar file
In particular interested in how the execution proceeds for standard patterns of recursion.
changelog_begin
changelog_end
CI currently errors with:
```
Subprocess:
git checkout efe6545c2c
-- docs/source/support/release-notes.rst
failed with exit code 127; output:
---
---
err:
---
Previous HEAD position was 2af134c... WIP: Draft version constraint
generation (#5472)
HEAD is now at efe6545... 1.2.0-snapshot.20200520.4224.0.2af134ca
(#6040)
/bin/sh: 2: --: not found
---
```
because the line
```
latest_release_notes_sha <- shell "git log -n1 --format=%H HEAD -- LATEST"
```
will assign a string that ends in a newline, and then when we try to
construct the shell command:
```
(shell_ $ "git checkout " <> latest_sha <> " -- docs/source/support/release-notes.rst")
```
we actually get two lines for Bash to execute.
CHANGELOG_BEGIN
CHANGELOG_END
* Some refactoring in Speedy Classify, follow last PR.
Improve matching idiom, and shorten two annoyingly long names.
Use 0 for default value of all Clasification.Counts. Then dont need newEmptyCounts() function
changelog_begin
changelog_end
* final class, not case class, for Remy
Have tests which will fail if we screw up the indexing for any of:
- function application args
- the stack (let bindings)
- free-variables of a closure
In particular if we mix up the direction of out indexing. i.e. 0..n-1 VS n-1..0
Try to make the three tests as selective as possible, by only failing for the one aspect they are designed to test.
changelog_begin
changelog_end
Current version yields:
```
Subprocess:
git log -n1 --format=%H master -- LATEST
failed with exit code 128; output:
---
---
err:
---
fatal: bad revision 'master'
---
```
so apparently we can't trust a CI run on master to have a master branch
defined. `HEAD` should work, though.
CHANGELOG_BEGIN
CHANGELOG_END
* Speedy Tail call optimization
The goal of this PR is to achieve Tail call optimization #5767
Tail call optimization means that tail-calls execute without consuming resources. In particular, they must not consume stack space.
Speedy has two stacks: The `env`-stack and the `kontStack`. For an optimized tail call in Speedy, we must not extend either. In Speedy, all function calls are executed via the code `enterFullyAppliedFunction`. The behaviour of this code (prior to this PR) is as follows:
(1) Push the values of all args and free-variables on the env-stack (because that's where the code expects to find them), and
(2) Push a KPop continuation on the kontStack, which will restore the env-stack to its original size before returning to the code which made the function call.
We must stop doing both these things. We achieve this as follows:
(1) We address function args and free-vars via a new machine component: the current `frame`.
(2) We make continuations responsible for restoring their own environment.
As well as achieving proper tail calls, we also gain a performance improvement by (a) removing the many pushes to the env-stack, and (b) never having to push (and then later re-enter) a KPop continuation. The args array and the free-vars array already existed, so there is no additional cost associated with constructing these array. The only extra costs (which are smaller than the gains) are that we must manage the new `frame` component of the machine, and we must record frame/env-size information in continuations so they can restore their environment.
To make use of the frame, we need to identify (at compile time) the run-time location for every variable in a speedy expression. This is done during the `closureConvert` phase. At run-time, an environment is now composed of both the existing env-stack and the frame. The only values which now live on the env-stack are those introduced by let-bindings and pattern-match-destructuring. All other are found in the frame.
Changes to SEExpr:
- Introduce a new expression form `SELoc`, with 3 sub classes: SELocS/SELocA/SELocF to represent the run-time location of a variable.
- SELocS/A/F execute by calling corresponding lookup function in Speedy: getEnv(Stack,Arg,Free).
- SEMakeClo takes a list of SELoc instead of list of int.
- During closure conversion all SEVar are replaced by an SELocS/A/F.
- SEVar are not allowed to exist at run-time (just as SEAbs may not exist).
- We adapt the synthesised code for SEBuiltinRecursiveDefinition: FoldL, FoldR, EqualList
It is worth noting the prior code also had the notion of before/after closureConvert, but SEVar was used for both meanings: Prior to closureConvert it meant the relative-index of the enclosing binder (lambda,let,..). After closureConvert it meant the relative-offset from the top of the env-stack where the value would be found at run-time. These are not quite the same! Now we have different sub-types (SEVar vs SELoc), this change of mode is made more explicit.
Run-time changes:
- Use the existing `KFun` continuation as the new `Frame` component.
- `KFun` allows access to both the args of the current application, and the free-vars of the current closure.
- A variable is looked up by it's run-time location (SELocS/A/F)
- A function application is executed (`enterFullyAppliedFunction`), by setting the machine's `frame` component to the new current `KFun`.
- When a continuation (KArg, KMatch, KPushTo, KCatch) is pushed, we record the current Frame and current stack depth within the continuation, so when it is entered, it can call `restoreEnv` to restore the environment to the state when the continuation was pushed.
Changes to Compiler:
- The required changes are to the `closureConvert` and `validate`.
- `closureConvert` `remaps` is now a `Map` from the `SEVar`s relative-index to `SELoc`
- `validate` now tracks 3-ints (maxS,masA,maxF)
changelog_begin
changelog_end
* changes for Remy
* Changes for Martin
* test designed explicitly to blow if the free variables are captured incorrectly
* address more comments
* improve comment about shift in Compiler
First version of static verification tool.
The current state of the tool:
- Reads DAR files.
- Partially evaluates the code.
- Generates constraints for the field and choice to be verified.
- Passes the constraints to an SMT solver.
- Some basic tests.
* Revert "disable test pending tracking down duplicates"
This reverts commit 8b5f9dfa04.
* activeContracts tells caller the ledgerEnd, rather than accepting activeAt argument
* naive port of http-json tests to sandbox-next
* Revert "naive port of http-json tests to sandbox-next"
This reverts commit 91d4590c90.
* lock before grabbing acs and ledgerEnd offset in InMemoryLedger
- as suggested by @gerolf-da; thanks
trigger all releases from master
The 1.1.0 release went wrong and we had to trash it and release 1.1.1
instead. This is an attempt at identifying and correcting the root
cause behind that incident.
To understand the situation, we need to know how releases worked before
1.0. We had a one-line file called `LATEST` that specifies the git SHA and
version tag for the latest release. A change to that file triggered a
release with the specified release tag, built from the source tree of
the specified commit. The `LATEST` file looked something like:
```
f050da78c9 1.0.0-snapshot.20200411.3905.0.f050da78
```
To mark a release as stable, we would change it to look like this:
```
f050da78c9 1.0.0
```
i.e. simply drop the `-snapshot...` suffix. Even though the commit (and
thus the entire source tree we build from) is the same, we would need to
rebuild almost all of our release artifacts, as they embed the version
tag in various places and ways. That worked well as long as we could
assume we were doing trunk-based development, i.e. all releases would
always come from the same (`master`) branch.
When we released 1.0, and started work on 1.1, we had a few bug reports
for 1.0 that we decided should be resolved in a point release. We
decided that the best way to handle that would be to have a branch
starting on the release commit for 1.0, and then backport patches from
`master` to that branch. We adapted our build process to also watch the
`release/1.0.x` branch and, in particular, trigger a new release build if
the `LATEST` file in that branch changed. That worked well.
The plan going forward was to keep doing regular snapshot releases from
the `master` branch, and create support, point releases ("patch" releases
in semver) from dedicated branches.
On April 30, we made a snapshot release as an RC for 1.1.0, by changing
the `LATEST` file in the `master` branch. That release was built on commit
681c862d. On May 6, we decided to take a new snapshot as the RC for
1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new
latest release.
On May 11, we noticed an issue that broke our builds. Without going into
details, an external artifact we depend on had changed in incompatible
ways. After fixing that on `master`, we reasoned that this would also
break the build of the final 1.1.0 release if we just tried to build
7e448d81 again. But as the target release date was May 13, we did not
want to take a new snapshot after that fix, as that would have included
one more week of work in the release, and given us no time to test it.
So we did what we did for the 1.0 branch, as it had worked well: we
created a branch that forked from `master` at commit 7e448d81 and called
it `release/1.1.x`, then cherry-picked the one fix to our build process to
work around the broken download. When the time came to make the final
1.1.0 build on May 13, we naturally picked the `LATEST` file from the
`release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly,
we did not need to update the target commit to include the "broken
download" fix as, in the meantime, the internet had fixed itself, and we
thus reasoned we should go for the exact code of the RC rather than
include an unnecessary, albeit seemingly harmless, change.
Everything went well with the release process. Tests went well too. Then
we got a report that an application that worked against the latest RC
broke with the final 1.1.0. The issue was that we had built the wrong
commit: by branching off at the point of the _target_ commit for the
latest snapshot, we did not have the change to the `LATEST` file that
designated that commit as the target. So the `LATEST` file in
`release/1.1.x` was still pointing to 681c862d.
I believe the root cause for this issue is the fact that we have
scattered our release process over multiple branches, meaning there is
no linear history of what was released and we are relying on people
being able to mentally manage multiple timelines. Therefore, I propose
to fix our release process so this should not happen again by
linearizing the release process, i.e. getting back to a situation where
all releases are made from a single branch, `master`.
Because we do want to be able to release _for_ multiple release branches
(to provide backports and bugfixes), we still need some way to
accommodate that. Having a single `LATEST` file in the same format as
before would not really work well: keeping track of interleaved release
streams on a single file would not really be easier than keeping track
of multiple branches.
My proposed solution is to instead have a multiline LATEST file, so that
all the release branch "tips" can be observed at the same time, and, as
long as we take care to only advance one release branch at a time, we
can easily keep track of each of them. This is what this PR does.
This required a few changes to our release process. Most notably:
- Obviously, as this is the main point of this PR, the build process has
once again been restricted to only trigger new releases from the
`master` branch.
- As our CI machinery cannot easily be made to produce multiple releases
from a single build, the `check_for_release` step will only recognize
a commit as a release trigger if it changes a single line in the
`LATEST` file. This restriction comes in addition to the existing one
that a release commit is only allowed to change either just the
`LATEST` file or both the `LATEST` and
`docs/source/support/release-notes.rst` files.
- The docs publication process has been changed to update _all_
published versions to display the _latest_ release notes page. This
means that the release notes page will always show you all published
versions, regardless of which version of the documentation you're
looking at. This also means that interleaving release notes correctly on
that page is a manual exercise.
- As per the intention of the new process, the `LATEST` file has been
updated to contained all existing post-1.0 stable releases. It should
also include all existing snapshot releases should we have more than one
at a time (say, should we discover an issue with 1.1.1 that required us
to work on a 1.1.2).
- The `release.sh` script has been dramatically simplified as I felt it
was trying to do too much and porting its existing functionality to a
multi-line `LATEST` file would be too hard.
CHANGELOG_BEGIN
CHANGELOG_END
The workspace for the vendored node wrapper script `@nodejs_linux_amd64`
did previously not record a dependency on the nixpkgs provided node
workspace. This patch enforces that dependency by introducing a dummy
read of the vendored node binary.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Note: this is beta-level software. See documentation for the precise
guarantees this does and does not come with. (Documentation does not
exist at the time of opening this PR, but should exist by the time the
first version of this gets published.)
CHANGELOG_BEGIN
- We now publish Sandbox Next as an **ALPHA** standalone jar.
- We now publish the HTTP JSON API as a standalone jar.
CHANGELOG_END
not sending a live event after reconnecting means that the 'loading'
indicater is never updated to 'false'. we could also not set it to
'true' on the 'close' event, but that can lead to a rather unintuitive
behaviour where data is stale and then suddenly updated without any
indication.
CHANGELOG_BEGIN
CHANGELOG_END
* REPL record dot syntax regression test
* REPL: Apply record preprocessor to expressions
CHANGELOG_BEGIN
- [DAML REPL] Record dot syntax is now handled in expressions entered
into the REPL.
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
For some reason this seems to be failing in particular on Linux but
I’ve now also seen it fail on MacOS. My suspicion is that running it
in parallel, somehow corrupts the cache but I don’t understand the details.
changelog_begin
changelog_end
This image can be built with:
```
bazel run //ledger/sandbox:sandbox-next-image
```
This will, unfortunately, also run a container, which you will need to
kill with _Ctrl+C_. You can use `bazel build`, but this will only build
the image tarballs, which you need to import into your Docker image
repository yourself.
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
- Add `LedgerClient#close`, which will shut down the channel and await
termination. This is optional; the channel will still be shut down on
JVM exit if this method is not called.
CHANGELOG_END
This is offered as a compromise for those who would like to ensure
resources are shut down cleanly, while not making the API more
complicated. I originally wanted to make `LedgerClient.fromBuilder`
return a `Resource[LedgerClient]`, but the `Resource` API would lead to
an increased learning curve for users.
