* Upgrade nixpkgs revision
* Remove unused minio
It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.
* Update Bazel on Windows
changelog_begin
changelog_end
* Fix hlint warnings
The nixpkgs update implied an hlint update which enabled new warnings.
* Fix "Error applying patch"
Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.
* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681
Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.
```
git diff ':(exclude)WORKSPACE'
```
* Update rules_nixpkgs
* nixpkgs location expansion escaping
* Drop --noincompatible_windows_native_test_wrapper
* client_server_test using sh_inline_test
client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.
This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.
* daml_test using sh_inline_test
* daml_doc_test using sh_inline_test
* _daml_validate_test using sh_inline_test
* damlc_compile_test using sh_inline_test
* client_server_test find .exe on Windows
* Bump Windows cache for Bazel update
Remove `clean --expunge` after merge.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Add a --timeout option to `daml ledger` commands
At least in the compatibility tests, we see timeouts fairly regularly
in the tests for fetch-dar. If we see those timeouts in the tests, I
see no reason why our users should never see that especially on
actually distributed ledgers and hardcoded timeouts are a bad idea
anyway so now they are configurable.
I’ve also changed the timeout for the fetch tests (those are the only
once I’ve seen timeout) to 120s.
changelog_begin
- [DAML Assistant] The `daml ledger` commands now accept a `--timeout`
option which can be used to change the timeout for API requests from
the default of 30s.
changelog_end
* Generate parties as uuids to avoid collisions
changelog_begin
changelog_end
* Match on Deadline Exceeded as well
changelog_begin
changelog_end
* Update compatibility/bazel_tools/test_utils/DA/Test/Util.hs
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Update daml-assistant/daml-helper/src/DA/Daml/Helper/Main.hs
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Add some docs
changelog_begin
changelog_end
* Better version checks
changelog_begin
changelog_end
* fix imports
changelog_begin
changelog_end
* This is not how you exception
changelog_begin
changelog_end
* Update daml-assistant/daml-helper/src/DA/Daml/Helper/Main.hs
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
lodash released a non-broken version now and while dependabot isn’t
smart enough to realize this one has a vulnerability it still seems
like a good idea to bump it.
changelog_begin
changelog_end
* Add `platform-version` field to `daml.yaml`
This PR adds the `platform-version` field to `daml.yaml`. Based on the
approach agreed upon in #6558, the logic for this all sits in
`daml-helper` and there are no changes to the assistant.
The details of how the logic work are in a comment so I’m not going to
repeat them here but the commands that are affected are:
- `daml sandbox`
- `daml sandbox-classic`
- `daml json-api`
- `daml start` (but only for sandbox and the JSON API, not for
Navigator or anything else)
For tests, I’ve added a test to the compat workspace that installs two
SDKs simultaneously and tries out various combinations and verifies
that we get the correct version. Open to other ideas for testing this
but that seemed like the most sensible option that actually tests what
we run.
changelog_begin
- [DAML Assistant] You can now specify the version of Sandbox and the
JSON API independently of your SDK version by setting
``platform-version`` in your ``daml.yaml``. This is useful if you
are deploying to a ledger that is running components from a
different SDK version. See
https://docs.daml.com/tools/assistant.html#project-config-file-daml-yaml
for details.
changelog_end
* Run platform-version tests
changelog_begin
changelog_end
* Fix tag globbing
changelog_begin
changelog_end
* fmt
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* Try to fix env vars
changelog_begin
changelog_end
* Remove hardcoded references to 1.2.0
changelog_begin
changelog_end
* Rephrase doc comment
changelog_begin
changelog_end
* get things to compile
changelog_begin
changelog_end
* maybe fix things for realz
changelog_begin
changelog_end
* Remove debugging output
changelog_begin
changelog_end
* Get angry at windows
changelog_begin
changelog_end
* why is windows
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
The default cache directories in yarn don’t work in the Bazel
sandbox. This results in Yarn creating tons of temp directories for
each test that are never cleared up.
This PR changes this to create actual temp directories that are
properly cleaned up.
changelog_begin
changelog_end
The issue is described in a comment (since I want to preserve it) but
here’s the brief summary:
Calling `follow` twice can result in contention if we do not wait in
between. That is shown in the log of the JSON API. For some reason
that ends up propagating to a misleading Target closed error from
puppeteer.
This PR addresses the problem by waiting for the list of followers to
be updated which definitely implies that the ledger has seen the
command.
changelog_begin
changelog_end
* ANF transformation in Speedy.
The idea behind this PR is to transform speedy expressions into a simpler form where all non-atomic sub-expressions are made explicit by the introduction of let-forms. In particular, for the function-application form. These simpler forms allow the execution engine to take advantage of the atomic assumption, and often removes many additional execution steps. In particular the pushing of continuations to allow execution to continue after a compound expression has been reduced to a value.
changelog_begin
changelog_end
* improve comment
* inline functions relocateA/L
* remove comment about scalafmt
* remove commented out alterative def for transformLet1
* improve code by adding incr methods to DepthA/E
* remove (n == 0) special case in trackBindings
* clarify comment further
* improve validate/go to not consume stack for deeply right-nested let-expressions
* address comments from Remy: be private; use final case case; etc
* rename to unsafeCompilationPipeline
* add back some trailing commas
* remove commented-out debug line
* improve comment
* remove dev/debug code in compilationPipeline
* remove commented out code in SEAppGeneral.execute
* undo unrelated code improvement in SValue.scala
* fix compile. object Anf cannot be private
I don’t know if this fixes the “Target closed” issues since I never
managed to reproduce them but I’d rather debug them on the latest
version :)
changelog_begin
changelog_end
I broke the tests for Sandbox < 1.2.0 when I switched to explicit
party allocation yesterday since I didn’t realize this only landed in
SDK 1.2.0 (sorry about that).
changelog_begin
changelog_end
This introduces the same trick for sharing the postgres instance that
we use in the main workspace to the compatibility workspace. The bash
code for that is currently duplicated. Didn’t seem worth trying to
make it sufficiently generic that we can share it (it hasn’t changed
since we added it) but if someone disagrees, I’m happy to reconsider
that.
changelog_begin
changelog_end
* Allocate parties in test-tool compat tests
Compatibility is really something you care about in a production
setup. Implicit party allocation is a development feature and not
available on most ledgers. Therefore, this PR switches the tests over
to allocate parties explicitly. For sandbox-next we also disable the
implicit party allocation feature completely which allows us to
include the ClosedWorldIT test. Sadbonx-classic does not allow
disabling it atm.
In principle, we could run in both configurations but I don’t think
the additional costs (we have hundreds of those tests and this would
double the number) are worth it due to the same reasons that I’ve
given at the beginning for why explicit allocation is more important.
changelog_begin
changelog_end
* Fix postgresql tests
changelog_begin
changelog_end
This PR updates the lower bound of the api test tool required to use the
granular test selection feature to the first snapshot that includes it
(as opposed to the floating 0.0.0).
CHANGELOG_BEGIN
CHANGELOG_END
The //:migration-stable test-cases had issues with test results not
being invalidated after an update of the HEAD SDK. This may be due to
the SDK distribution files being undeclared dependencies of the `daml`
binary. This change adds the SDK distribution files to the `data`
attribute of the generated `daml` `cc_binary` to ensure that changes are
noticed by Bazel and cached test-cases invalidated.
The corresponding runfiles tree includes a symlink per file of the SDK
release. At the time of commit this amounts to ~9MiB of symlinks for one
SDK version (0.0.0 in particular).
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This is the final bit extracted from #6314. This PR adds the capability
to filter individual test cases (for either inclusion or exclusion)
while being completely backwards compatible. It also introduces a new
cli option, `--list-all`, to list all test cases, keeping the behaviour
of the existing `--list` option unchanged, i.e. only print test suites.
Unlike #6314, this is still building upon the idea of defining test
suites as maps, preserving the known-good order of tests and thereby not
introducing flakiness into our builds.
CHANGELOG_BEGIN
- [Ledger API Test Tool] ``--exclude`` and ``--include`` now match the
full test name as a prefix, rather than just suite names. Test name is
built by combining the suite name with a test identifier, so this change
should be fully backwards compatible. Run with ``--list-all`` to list
all tests (as opposed to just the test suites with ``--list``).
CHANGELOG_END
After adding ranges for sandbox versions, this PR now also adds ranges
for test tool versions. This is particularly useful since this gets us
to the point where adding a new version usually (unless we’ve actually
broken compatibility more than before) does not require any change in
exclusions.
I’ve also cleaned up exclusions that only affect snapshots. While I’ve
previously argued against that since that seemed more annoying to
maintain, the list is small enough now that I’m no longer worried
about that.
changelog_begin
changelog_end
Given that sadbonx-classic came back from the dead my initial
reasoning that we don’t have to include it since it will be gone soon
anyway is obsolete. Therefore, this PR adds both the in-memory and
postgresql variant to the ledger-api-test-tool tests. For other tests,
I did not yet create variants for sadbonx-classic. Not sure how
important that is, we can always add it later.
changelog_begin
changelog_end
* Validate data-dependencies test DARs
In addition to building the coin upgrade DAR with data-dependencies on
the coin1 and coin2 packages built with an older SDK, this will also
define cases that run `damlc validate-dar` on the compiled coin upgrade
package.
changelog_begin
changelog_end
* fmt
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This PR does two things:
1. It makes use of the new sha256sums file to speed up the script to
update the versions significantly. The code here is a bit hacky but
as mentioned in the comment, this script is only run offline atm so
that seems reasonable.
2. It prunes SDK snapshot versions to those for which there is no
corresponding stable release.
changelog_begin
changelog_end
`waitForSelector` returns immediately if the selector is already
present (which is documented). This means that the waitForSelector
after the second follow isn’t doing anything since we already waited
for after the first follow. `waitForFunction` seemed like the simplest
solution and doesn’t require patching the HTML which is a bit finnicky
in the compat tests.
changelog_begin
changelog_end
* Upgrade puppeteer
We’ve seen a couple of issues in the compatibility tests of the form
```
Error: Protocol error (Runtime.callFunctionOn): Target closed.
```
Looking at the issue tracker in puppeteer this might be fixed in newer
versions and I don’t see why we should stick to a fairly old version
anyway.
changelog_begin
changelog_end
* Upgrade nodejs
changelog_begin
changelog_end
* temporary add a step to kill node_modules
changelog_begin
changelog_end
* Kill live server and try to fix Windows
changelog_begin
changelog_end
* Undo rm
changelog_begin
changelog_end
* Build CopyTrigger example in compatibility
* Test DAML trigger with test DAML script
Defines a custom client script that runs the trigger runner in the
background and uses a DAML script in the foreground to test that the
trigger runs and its effects are observable.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Use bazel_tools:versions.bzl%versions.is_at_least
Also align versions.is_at_least|most with skylib
The arguments to the Skylib function are in the opposite order. I've
included a docstring to explain
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
We have seen a couple of timeouts on CI. I haven’t reproduced any
locally and even on CI there haven’t been too many (remember that we
run those tests very often in different configurations) so going for a
fairly conversative bump in timeouts but I’m just guessing here.
changelog_begin
changelog_end
* Support version ranges in test tool exclusions
This cleans up the map of exclusions for the ledger-api-test-tool part
of the compat tests. For now, you can only use ranges for the inner
map. In principle we could also add it to the outer map but I’ll leave
that for a separate PR.
I’ve also run the script to update versions so this includes the
latest two snapshots.
fixes#5851
changelog_begin
changelog_end
* Address review comments
changelog_begin
changelog_end
* buildifier
changelog_begin
changelog_end
Previously we ran `daml codegen js` and `daml build` in the tests and
we also ran it twice. While this makes sense for the SDK integration
tests, it doesn’t really make sense for the compatibility tests. These
are SDK-only features so they don’t actually test compatibility. These
steps are super slow and running them in the tests also means that we
run them a quadratic number of times.
This PR moves the `daml build` and the `daml codegen js` step to
separate genrules. In the actual test, the only expensive steps are to
run `yarn install` and the actual jest tests. In principle, we could
probably try to factor out the `yarn install` step as well but I’m a
bit scared of coyping around node_modules so I’ll not attempt to do
that for now.
changelog_begin
changelog_end
* Include puppeteer tests in compat tests
This PR adds the puppeteer based tests to the compatibility
tests. This also means that they are now actually compatibility
tests. Before, we only tested the SDK side.
Apart from process management being a nightmare on Windows as usually,
there are two things that might stick out here:
1. I’ve replaced the `sh_binary` wrapper by a `cc_binary`. There is a
lengthy comment explaining why. I think at the moment, we could
actually get rid of the wraper completely and add JAVA to path in
the tests that need it but at least for now, I’d like to keep it
until we are sure that we don’t need to add more to it (and then
it’s also in the git history if we do need to resurrect it).
2. These tests are duplicated now similar to the `daml ledger *`
tests. The reasoning here is different. They depend on the SDK
tarball either way so performance wise there is no reason to keep
them. However, we reference the other file in the docs which means
we cannot change it freely. What we could do is to make this
sufficiently flexible to handle both the `daml start` case and
separate `daml sandbox`/`daml json-api` processes and then we can
reference it in the docs. There is still added complexity for
Windows but that’s necessary for users as well that want to run
this on Windows so that seems unavoidable. (I should probably also
remove my snarky comments 😇) I’d like to kee it duplicated
for this PR and then we can clean it up afterwards.
changelog_begin
changelog_end
* Bump timeouts
changelog_begin
changelog_end
For some reason this seems to be failing in particular on Linux but
I’ve now also seen it fail on MacOS. My suspicion is that running it
in parallel, somehow corrupts the cache but I don’t understand the details.
changelog_begin
changelog_end
* Include the first feature in compat tests for create-daml-app
This adds the step from the create-daml-app tests that applies the
patch for the messaging feature and tests that things will build. This
is the last step before we can actually run the puppeteer tests which
will turn this into an actual compatibility test.
changelog_begin
changelog_end
* temporarily run all tests
changelog_begin
changelog_end
* Fix exclusions
changelog_begin
changelog_end
* Revert "temporarily run all tests"
This reverts commit 7425dd09cf48f2bfd6064b55d857c76d51afc821.
* Remove accidentally committed messaging.patch
changelog_begin
changelog_end
This is the first part of #5700
It adds tests that build create-daml-app using `daml build` and then
run the codegen and build the UI. Contrary to our main tests these
also run on Windows. This is actually reasonably simple by first
building the typescript libraries on Linux and then downloading them
on Windows.
There are two parts that are still missing from the tests in the main
workspace:
1. Building the extra feature. This should be fairly easy to add.
2. Running the pupeeter tests. At least MacOS and Linux should be
reasonably easy. I don’t know what horrors Windows will throw at
us. This step is what actually makes this a compatibility
test. Currently it doesn’t actually launch Sandbox and the JSON API.
Since this PR is already pretty large, I’d like to tackle those things
separately.
changelog_begin
changelog_end
While the tests are exclusive and I’ve kept them exclusive for
resource usage, the hardcoded port numbers can still cause issues if
for whatever reason you have a postgres instance running outside of
Bazel. This could happen because you simply have a local postgres
instance for testing other things or, what seems to have happened on
CI, due to a leftover postgres instance from a previous test.
While we should do something about the latter, this at least gets us
into a situation where the node isn’t completely broken at this point.
changelog_begin
changelog_end
* Implement version name shortening for Windows
CHANGELOG_BEGIN
CHANGELOG_END
* Temporarily disable `--quick`.
* Revert "Temporarily disable `--quick`."
This reverts commit 025e6c2ddd.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This changes the code for starting postgres to print the log if
`pg_ctl start` fails.
In addition to that it marks the migration tests exclusive since they
rely on this code which has a hardcoded port number.
I tested this on CI by temporarily disabling --quick.
changelog_begin
changelog_end
* Make Bearer prefix optional in Haskell ledger bindings
This matches the change to the JVM bindings in 1.0.0
changelog_begin
- [DAML Assistant] The ``daml ledger`` commands no longer require the
``Bearer `` prefix in the access token file. This matches the behavior
of DAML Script and other SDK tools.
changelog_end
* Fix tasty crash
changelog_begin
changelog_end
* Update compatibility/bazel_tools/daml_ledger/Main.hs
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
* Update SDK versions in compatibility tests
This adds a Haskell script to generate a versions.bzl file that
contains the list of versions as well as their hashes. This should
make it a bit easier to keep things up2date going forward.
The script is a bit slow since downloading all the SDKs takes quite a
while but for now it should be good enough and is much more pleasant
than having to figure this out manually.
changelog_begin
changelog_end
* Address review comments
changelog_begin
changelog_end
* Fix excluded tests
changelog_begin
changelog_end
* Add basic Sandbox data continuity tests
This adds some basic tests that check that data migrations work
properly. For now, I use DAML Script to create and query contracts at
each step. This isn’t perfect since queries can only use the active
contract service but not things like the transaction stream but it’s
clearly better than nothing.
The runner for executing the tests is a simple Haskell executable. It
didn’t really seem useful to throw tasty at this.
I’ve added two sets of tests, one that runs only through stable
versions and one that includes snapshots since migrating through
snapshots is not necessarily equivalent.
Sadly these tests use sandbox-classic since I discovered while writing
these tests that sandbox-next does not actually support migrating data
between SDK versions.
changelog_begin
changelog_end
* Use the sandbox module instead of a custom withSandbox
changelog_begin
changelog_end
* Update compatibility/sandbox-migration/SandboxMigrationRunner.hs
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* daml new script-example
* Build DAML script DAR
* daml_script_dar macro over sdk_version
* Run an individual daml-script test
* DAML script test matrix
* format
CHANGELOG_BEGIN
CHANGELOG_END
* Use named arguments on daml_script_test
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Set JVM memory limits for sandbox
Using the same settings as defined in `@daml//bazel_tools:scala.bzl`.
* Mark tests as large
Their memory consumption is somewhere around 300MiB which is considered
"large" according to
https://docs.bazel.build/versions/master/be/common-definitions.html?cl=head#common-attributes-tests.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Make compat tests work on windows
This required some changes to the daml_sdk rule since the read-only
installation by the assistant breaks Bazel completely. We could only
apply those changes on Windows but I think I prefer the consistency
across platforms here over trying to stay close to how the SDK is
installed on user machines given that the SDK installation is not
something we’ve had issues with.
I’ve excluded the postgresql tests for now. I don’t expect them to be
particularly hard to fix but I’ve already spent almost 2 days on this
and having some tests run on Windows seems like a clear improvement
over running no tests on Windows :)
changelog_begin
changelog_end
* Remove todo
changelog_begin
changelog_end
* Copy daml ledger test to compatibility
Specifically:
- `daml-assistant/daml-helper/test/DA/Daml/Helper/Test/Deployment.hs`
- `libs-haskell/test-utils/DA/Test/Sandbox.hs`
* Adapt daml ledger test for compatibility
Avoid dependencies on the daml repository and make the SDK and platform
components configurable.
* Add test certificates to compatibility
Taken from `@daml//ledger/test-common/test-certificates`.
* Define daml_ledger_test
CHANGELOG_BEGIN
CHANGELOG_END
* Add note on keeping daml ledger tests in sync
* ./fmt.sh
* Strictly require --sdk-version flag
Tasty doesn't allow options without a default value, so we default to
`error` to enforce a value being set.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This PR extends the existing Linux compatibility tests to run on macOS
too. Fixes#5692.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Fix ./fmt.sh check
We accidentally introduced hlint lints when we added the compatibility
workspace. The hlint code in ./fmt.sh correctly detects that there is
a hint but then exits with 0 since it uses the exit code from `echo`
rather than the one from `hlint`. The info that we printed with
`echo` didn’t actually make any sense so I added a new comment as to
why we run hlint here. This requires some setup changes so that the
compatibility workspace gets the same Haskell flags since otherwise
using the same hlint rules obviously does not make sense.
Fixes#5701
changelog_begin
changelog_end
* Update fmt.sh
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Extend compatibility tests to DAML on SQL
This feels a bit hacky since the runfiles don’t work quite like I
would expect them to but it’s at least not more hacky than what we do
for the head-based tests we currently have.
Progress towards #5695
changelog_begin
changelog_end
* Fix runfiles with more bash
changelog_begin
changelog_end
* remove redundant port options
changelog_begin
changelog_end
* Create fewer sandbox targets
changelog_begin
changelog_end
* Apply suggestions from code review
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Fix runfiles snippet
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Abstract over SDK/platform client_server_test
* Define test-matrix using simple list comprehension
CHANGELOG_BEGIN
CHANGELOG_END
* copy right header
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
