* Update rules_haskell
* Pin stack_snapshot repositories
* Document stack_snapshot_json
CHANGELOG_BEGIN
CHANGELOG_END
* Don't pin stack_snapshot on Windows
The lock file is generated on Unix and includes unix specific
dependencies, e.g. `unix`. Most developers don't have easy access to a
Windows machine, so regenerating the lock file for Windows would be
inconvenient.
* upgrade stack 2.1.3 --> 2.3.1 on Windows
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* update versions bbased on git tags
We currently run the update script as part of the release process. This
causes an issue (highlighted by @cocreature in #6560): the update script
currently relies on querying the docs website to get the list of
versions, but the docs website is updated on a cron, which means by the
time we run the update script it is not up-to-date yet.
This PR changes the update script to instead rely on the local git repo
and its list of tags, as that is updated synchronously.
CHANGELOG_BEGIN
CHANGELOG_END
* test
* revert test
The issue is described in a comment (since I want to preserve it) but
here’s the brief summary:
Calling `follow` twice can result in contention if we do not wait in
between. That is shown in the log of the JSON API. For some reason
that ends up propagating to a misleading Target closed error from
puppeteer.
This PR addresses the problem by waiting for the list of followers to
be updated which definitely implies that the ledger has seen the
command.
changelog_begin
changelog_end
* upgrade to wartremover 2.4.9
* simplify wart list and list JavaConversions as disabled
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* delete long-obsolete, contradictory comment
* also upgrade wartremover in compatibility (leaving aside maven_install.json)
* update compatibility maven_install.json to match
We take our own libraries from latest_stable_version which changed but
we did not rerun pinning which meant that this did not get updated.
changelog_begin
changelog_end
* ANF transformation in Speedy.
The idea behind this PR is to transform speedy expressions into a simpler form where all non-atomic sub-expressions are made explicit by the introduction of let-forms. In particular, for the function-application form. These simpler forms allow the execution engine to take advantage of the atomic assumption, and often removes many additional execution steps. In particular the pushing of continuations to allow execution to continue after a compound expression has been reduced to a value.
changelog_begin
changelog_end
* improve comment
* inline functions relocateA/L
* remove comment about scalafmt
* remove commented out alterative def for transformLet1
* improve code by adding incr methods to DepthA/E
* remove (n == 0) special case in trackBindings
* clarify comment further
* improve validate/go to not consume stack for deeply right-nested let-expressions
* address comments from Remy: be private; use final case case; etc
* rename to unsafeCompilationPipeline
* add back some trailing commas
* remove commented-out debug line
* improve comment
* remove dev/debug code in compilationPipeline
* remove commented out code in SEAppGeneral.execute
* undo unrelated code improvement in SValue.scala
* fix compile. object Anf cannot be private
I don’t know if this fixes the “Target closed” issues since I never
managed to reproduce them but I’d rather debug them on the latest
version :)
changelog_begin
changelog_end
I broke the tests for Sandbox < 1.2.0 when I switched to explicit
party allocation yesterday since I didn’t realize this only landed in
SDK 1.2.0 (sorry about that).
changelog_begin
changelog_end
This introduces the same trick for sharing the postgres instance that
we use in the main workspace to the compatibility workspace. The bash
code for that is currently duplicated. Didn’t seem worth trying to
make it sufficiently generic that we can share it (it hasn’t changed
since we added it) but if someone disagrees, I’m happy to reconsider
that.
changelog_begin
changelog_end
* Allocate parties in test-tool compat tests
Compatibility is really something you care about in a production
setup. Implicit party allocation is a development feature and not
available on most ledgers. Therefore, this PR switches the tests over
to allocate parties explicitly. For sandbox-next we also disable the
implicit party allocation feature completely which allows us to
include the ClosedWorldIT test. Sadbonx-classic does not allow
disabling it atm.
In principle, we could run in both configurations but I don’t think
the additional costs (we have hundreds of those tests and this would
double the number) are worth it due to the same reasons that I’ve
given at the beginning for why explicit allocation is more important.
changelog_begin
changelog_end
* Fix postgresql tests
changelog_begin
changelog_end
This PR updates the lower bound of the api test tool required to use the
granular test selection feature to the first snapshot that includes it
(as opposed to the floating 0.0.0).
CHANGELOG_BEGIN
CHANGELOG_END
The //:migration-stable test-cases had issues with test results not
being invalidated after an update of the HEAD SDK. This may be due to
the SDK distribution files being undeclared dependencies of the `daml`
binary. This change adds the SDK distribution files to the `data`
attribute of the generated `daml` `cc_binary` to ensure that changes are
noticed by Bazel and cached test-cases invalidated.
The corresponding runfiles tree includes a symlink per file of the SDK
release. At the time of commit this amounts to ~9MiB of symlinks for one
SDK version (0.0.0 in particular).
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This is the final bit extracted from #6314. This PR adds the capability
to filter individual test cases (for either inclusion or exclusion)
while being completely backwards compatible. It also introduces a new
cli option, `--list-all`, to list all test cases, keeping the behaviour
of the existing `--list` option unchanged, i.e. only print test suites.
Unlike #6314, this is still building upon the idea of defining test
suites as maps, preserving the known-good order of tests and thereby not
introducing flakiness into our builds.
CHANGELOG_BEGIN
- [Ledger API Test Tool] ``--exclude`` and ``--include`` now match the
full test name as a prefix, rather than just suite names. Test name is
built by combining the suite name with a test identifier, so this change
should be fully backwards compatible. Run with ``--list-all`` to list
all tests (as opposed to just the test suites with ``--list``).
CHANGELOG_END
Following the changes to exclusions, this is purely running the
update-versions script so we are now at the point where at least a PR
bot that makes these PRs is feasible.
changelog_begin
changelog_end
After adding ranges for sandbox versions, this PR now also adds ranges
for test tool versions. This is particularly useful since this gets us
to the point where adding a new version usually (unless we’ve actually
broken compatibility more than before) does not require any change in
exclusions.
I’ve also cleaned up exclusions that only affect snapshots. While I’ve
previously argued against that since that seemed more annoying to
maintain, the list is small enough now that I’m no longer worried
about that.
changelog_begin
changelog_end
Given that sadbonx-classic came back from the dead my initial
reasoning that we don’t have to include it since it will be gone soon
anyway is obsolete. Therefore, this PR adds both the in-memory and
postgresql variant to the ledger-api-test-tool tests. For other tests,
I did not yet create variants for sadbonx-classic. Not sure how
important that is, we can always add it later.
changelog_begin
changelog_end
* Document compatibility tests
Since I’ve explained at various points what is being tested and this
is something that is useful to non-devs as well, I’ve written up the
tests we currently have and what exactly is covered.
changelog_begin
changelog_end
* Update compatibility/README.md
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Update compatibility/README.md
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Add more detail to sandbox continuity tests
changelog_begin
changelog_end
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Currently we run all migration tests on a separate postgres
instance. This PR changes this to interleave the steps so that we can
run all tests on a single postgres instance.
We do get slightly worse errors for now but the assertion failures
should be informative enough that I don’t think assigning them to the
test that caused them is an issue (and if it is we can just change the
error message).
changelog_begin
changelog_end
* Factor out generic types from migration tests
This PR factors out `Event` and `Transaction` so we can reuse them for
different tests. To make that work we simply remove the type parameter
from ContractId which didn’t help much anyway and store the undecoded
JSON value in the constructors. The use sites are basically unchanged
thanks to pattern synonyms which match the old constructors.
changelog_begin
changelog_end
* Explain missing package id
changelog_begin
changelog_end
* Validate data-dependencies test DARs
In addition to building the coin upgrade DAR with data-dependencies on
the coin1 and coin2 packages built with an older SDK, this will also
define cases that run `damlc validate-dar` on the compiled coin upgrade
package.
changelog_begin
changelog_end
* fmt
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This PR does two things:
1. It makes use of the new sha256sums file to speed up the script to
update the versions significantly. The code here is a bit hacky but
as mentioned in the comment, this script is only run offline atm so
that seems reasonable.
2. It prunes SDK snapshot versions to those for which there is no
corresponding stable release.
changelog_begin
changelog_end
We've seen a series of failures of the form
```
ERROR: D:/a/1/s/daml-assistant/integration-tests/BUILD.bazel:162:1: output 'daml-assistant/integration-tests/create-daml-app-tests.exe' was not created
ERROR: D:/a/1/s/daml-assistant/integration-tests/BUILD.bazel:162:1: not all outputs were created or valid
```
across multiple machines. We suspect cache poisoning as the cause. This
increments the cache URL to effectively clear the cache.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
For some reason, platform_suffix doesn’t seem to provide enough
isolation to fix the “undeclared inclusion” errors even though it does
fix the issues for me locally.
This PR tries to address the problem by switching from
`platform_suffix` to modifying the actual URL of the cache.
To avoid leaking stuff from the local cache, I’ve added a clean
--expunge for now. We should be able to remove this once nodes have
been reset tomorrow. It will slow down nodes but that is clearly
better than having everything fail.
changelog_begin
changelog_end
This factors out the parts of the migration runner that are specific
to the current test so we can easily add new tests.
Currently the postgres instance and the model DAR are shared. It’s
easy to separate that and have them be per test. I don’t have strong
feelings on what the right approach here.
changelog_begin
changelog_end
`waitForSelector` returns immediately if the selector is already
present (which is documented). This means that the waitForSelector
after the second follow isn’t doing anything since we already waited
for after the first follow. `waitForFunction` seemed like the simplest
solution and doesn’t require patching the HTML which is a bit finnicky
in the compat tests.
changelog_begin
changelog_end
This PR extends the existing sandbox migration tests with a query to
the transaction service before and after running each step and
corresponding validations for that in the migration runner.
changelog_begin
changelog_end
* Upgrade puppeteer
We’ve seen a couple of issues in the compatibility tests of the form
```
Error: Protocol error (Runtime.callFunctionOn): Target closed.
```
Looking at the issue tracker in puppeteer this might be fixed in newer
versions and I don’t see why we should stick to a fairly old version
anyway.
changelog_begin
changelog_end
* Upgrade nodejs
changelog_begin
changelog_end
* temporary add a step to kill node_modules
changelog_begin
changelog_end
* Kill live server and try to fix Windows
changelog_begin
changelog_end
* Undo rm
changelog_begin
changelog_end
This is preparatory work for hitting the transaction service (and
others but that one is the most important) which isn’t possible via
DAML Script. To ease review, this PR only switches from a DAML Script
to a custom Scala binary. It does not yet change what is tested.
changelog_begin
changelog_end
* Build CopyTrigger example in compatibility
* Test DAML trigger with test DAML script
Defines a custom client script that runs the trigger runner in the
background and uses a DAML script in the foreground to test that the
trigger runs and its effects are observable.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Use bazel_tools:versions.bzl%versions.is_at_least
Also align versions.is_at_least|most with skylib
The arguments to the Skylib function are in the opposite order. I've
included a docstring to explain
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
We have seen a couple of timeouts on CI. I haven’t reproduced any
locally and even on CI there haven’t been too many (remember that we
run those tests very often in different configurations) so going for a
fairly conversative bump in timeouts but I’m just guessing here.
changelog_begin
changelog_end
* Support version ranges in test tool exclusions
This cleans up the map of exclusions for the ledger-api-test-tool part
of the compat tests. For now, you can only use ranges for the inner
map. In principle we could also add it to the outer map but I’ll leave
that for a separate PR.
I’ve also run the script to update versions so this includes the
latest two snapshots.
fixes#5851
changelog_begin
changelog_end
* Address review comments
changelog_begin
changelog_end
* buildifier
changelog_begin
changelog_end
Previously we ran `daml codegen js` and `daml build` in the tests and
we also ran it twice. While this makes sense for the SDK integration
tests, it doesn’t really make sense for the compatibility tests. These
are SDK-only features so they don’t actually test compatibility. These
steps are super slow and running them in the tests also means that we
run them a quadratic number of times.
This PR moves the `daml build` and the `daml codegen js` step to
separate genrules. In the actual test, the only expensive steps are to
run `yarn install` and the actual jest tests. In principle, we could
probably try to factor out the `yarn install` step as well but I’m a
bit scared of coyping around node_modules so I’ll not attempt to do
that for now.
changelog_begin
changelog_end
* Include puppeteer tests in compat tests
This PR adds the puppeteer based tests to the compatibility
tests. This also means that they are now actually compatibility
tests. Before, we only tested the SDK side.
Apart from process management being a nightmare on Windows as usually,
there are two things that might stick out here:
1. I’ve replaced the `sh_binary` wrapper by a `cc_binary`. There is a
lengthy comment explaining why. I think at the moment, we could
actually get rid of the wraper completely and add JAVA to path in
the tests that need it but at least for now, I’d like to keep it
until we are sure that we don’t need to add more to it (and then
it’s also in the git history if we do need to resurrect it).
2. These tests are duplicated now similar to the `daml ledger *`
tests. The reasoning here is different. They depend on the SDK
tarball either way so performance wise there is no reason to keep
them. However, we reference the other file in the docs which means
we cannot change it freely. What we could do is to make this
sufficiently flexible to handle both the `daml start` case and
separate `daml sandbox`/`daml json-api` processes and then we can
reference it in the docs. There is still added complexity for
Windows but that’s necessary for users as well that want to run
this on Windows so that seems unavoidable. (I should probably also
remove my snarky comments 😇) I’d like to kee it duplicated
for this PR and then we can clean it up afterwards.
changelog_begin
changelog_end
* Bump timeouts
changelog_begin
changelog_end
The workspace for the vendored node wrapper script `@nodejs_linux_amd64`
did previously not record a dependency on the nixpkgs provided node
workspace. This patch enforces that dependency by introducing a dummy
read of the vendored node binary.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
