* Participant pruning ledger api server support ported from Canton
CHANGELOG_BEGIN
- [Ledger API]: The preview of `ParticipantPruningService` enables ledger participants to prune the "front" of ledger state at the participant including the ledger api server index.
CHANGELOG_END
* Review feedback from Stefano
* Add pruning tests plus missed command completions change
* Review feedback from Robert
* Improved test readability by having populate helper return offsets
* Review feedback
* Ledger api changes to pruning api and disable canton pruning test
- Change return result to PruneResponse
- Change type of PruneRequest.prune_up_to to string
* Review feedback: Use ApiOffsetSConverter for logged offsets
The trigger runner (delibaretely) only cares about the high-level
API. This means that we can change the high-level API without breaking
backwards compatibility in the runner. However, we do of course have
to change the trigger code written in the high-level API.
This PR addresses this by making the code depend on the
version. Unfortunately we cannot use CPP here (something I don’t say
very often) so this is done in a Bazel function.
The last low-level breaking change was the change to `CommandId`
generation so we go back to that.
changelog_begin
changelog_end
* allow updating user state from high-level trigger rule
* fix tests and examples for new rule signature
* fix doc for new rule signature
* add changelog
CHANGELOG_BEGIN
- [Triggers] ``rule`` can now ``get`` and ``put`` the user-defined state just
like ``updateState`` does. It no longer accepts the state as an argument.
You can port your rule function types by replacing ``s -> TriggerA`` with
``TriggerA s``, removing the last argument from the function, and using
``get`` to retrieve the state at the beginning of the ``rule``'s ``do`` block,
if needed.
See `issue #7674 <https://github.com/digital-asset/daml/pull/7674>`__.
CHANGELOG_END
* test that rule userState changes propagate all the way out
* adapt compatibility filtering to #7681
* add ACS reader to TriggerA and TriggerStateA
* propagate changes in TriggerA, TriggerStateA structure
* allow query to be used in updateState and rule
* remove getTemplates
* remove ACS argument from updateState and rule
* fix type parameter order on query
* use query function in all tests and examples
* replace getContracts with query in documentation
* use wildcards instead of otherwise
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* rename TriggerStateA to TriggerUpdateA
- suggested by @cocreature; thanks
* missed renamings of TriggerStateA to TriggerUpdateA
- suggested by @cocreature; thanks
* make the meaning of the rlift functions clearer
* make initialize a TriggerInitializeA instead of a function; remove getContracts
* update tests and examples for new initialize signature
CHANGELOG_BEGIN
- [Triggers] Trigger ``updateState``, ``rule``, and ``initialize`` functions no
longer accept an ``ACS`` argument; instead, they must use the ``query`` action
to query the ACS, similar to the same function in DAML Script.
See `issue #7632 <https://github.com/digital-asset/daml/pull/7632>`__.
CHANGELOG_END
* disable older compatibility trigger builds
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add ActionState to the standard library
* use 1 ActionState, 1 get, 1 put in low-level trigger library
* introduce TriggerStateA for updateState
* fix tests and examples for new updateState signature
CHANGELOG_BEGIN
- [Triggers] The ``updateState`` function now returns a ``TriggerStateA``. This
is an action like ``TriggerA``, but doesn't permit emitting commands. Instead
of taking the state as an argument and returning a new state, you can
manipulate the state with ``get``, ``put``, and ``modify``. Any existing
``updateState`` can be ported by replacing ``s -> expr`` in the lambda
expression with ``-> modify $ \s ->``, and then made to look nicer from there
as desired.
See `issue #7621 <https://github.com/digital-asset/daml/pull/7621>`__.
CHANGELOG_END
* some DAML docs for updateState and TriggerStateA
* add getCommandsInFlight as a TriggerA action
* immediately update commandsInFlight on emitCommands
- delay until the rule had returned made sense before submitCommands was launched
immediately; this also makes `getCommandsInFlight` more sensible
* remove commands-in-flight argument from high-level Trigger rule
CHANGELOG_BEGIN
- [Triggers] The "commands in flight" or ``Map CommandId [Command]`` argument has been
removed from high-level trigger ``rule`` functions; instead, the current
commands-in-flight can be retrieved with the new ``getCommandsInFlight`` function, which
can be done immediately at the beginning of the rule's ``do`` block to preserve exact
existing trigger behavior.
See `issue #7600 <https://github.com/digital-asset/daml/pull/7600>`__.
CHANGELOG_END
* remove commands-in-flight argument from trigger tests
* update doc examples and copy in compatibility
- compatibility change will most likely entail another flag day in compatibility tests,
replacing last_pre_7456_trigger_version
* update doc text for getCommandsInFlight
* test that getCommandsInFlight gets updated during the rule
* flag day for trigger compatibility tests
sdk-version: 1.7.0-snapshot.20201006.5358.0.0c1cadcf
File: src/CopyTrigger.daml
Hidden: no
Range: 55:11-55:19
Source: typecheck
Severity: DsError
Message:
src/CopyTrigger.daml:55:12: error:
• Couldn't match type ‘TriggerA ()’ with ‘() -> TriggerA ()’
Expected type: Party
-> DA.Next.Map.Map CommandId [Command] -> () -> TriggerA ()
Actual type: Party -> ACS -> () -> TriggerA ()
• In the ‘rule’ field of a record
In the expression:
Trigger
{initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}
In an equation for ‘copyTrigger’:
copyTrigger
= Trigger
{initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}
* match docs on TriggerAState to current usage of these fields
* remove emittedCommands, as commandsInFlight is now kept up-to-date
* zoomed from where?
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* conservatively move daml-script, trigger SValue interpreters to common library
* introduce expect and JavaList pattern for converters
* clean up trigger Converter Command interpretation
* add Church Free monad
* add an action language for trigger updates
* add expectE to remove some of the joins
* convert more of the converters to expect
* tool for unrolling Free/Roll
* split handleStepResult up and clean up its pattern
* handleStepFreeResult to interpret TriggerF
* replace Free Church with Pure/Roll free from Script
* newtype for ActionTrigger
* replace update in low-level Trigger with Free TriggerF
* submit one Commands at a time
* boolean blindness strikes again
* log missed TriggerF steps
* comment actual Submit contents
* match #7501 fromPureSExpr sig change in 00b80b8ea3
* avoid using forwardPort in runTrigger
* push State back into DAML, so it can be excluded from the action list
* push Message back into DAML, unifying the action language for initialState and update
* bringing TriggerF into initial state
* really add TriggerF into initial state, with all ports, tested
* add ActionTrigger class, express initialState in its terms
* add all TriggerF actions to existing TriggerA
* Trigger.rule will no longer have Time argument
* rename getS, setS to get, put, matching C.M.T.State from transformers
* make high-level Rule evaluate to the underlying TriggerF sequence
* Assert's testRule doesn't have a transform yet
* move DamlTuple2 to common converter library
- suggested by @cocreature; thanks
* combine the two Frees, provide from Script
* remove time argument from integration tests
CHANGELOG_BEGIN
- [Triggers] The ``Time`` argument was removed from the trigger rule function; instead, it
can be fetched within the ``TriggerA`` ``do`` block by ``getTime``, as with ``Update``
and ``Scenario``. The ``LowLevel`` trigger interface has been redesigned; such triggers
need to be rewritten or ported to high-level triggers.
See `issue #7456 <https://github.com/digital-asset/daml/pull/7456>`_.
CHANGELOG_END
* add trigger rule simulator to support Assert module
* missed new Free module
- left in script per @cocreature
* remove retract as we ended up using foldFree for that purpose instead
- suggested by @cocreature; thanks
* throw ConverterException instead of RuntimeException
- suggested by @cocreature; thanks
* remove Time argument from coin-upgrade-trigger
* port trigger service tests
* port trigger scenario test
* put TriggerSetup and TriggerRule into LowLevel.Trigger instead of unboxed Free
- suggested by @cocreature; thanks
* remove Time argument from trigger compatibility test
* submit commands as soon as each `emitCommands` is sequenced
- we still collect a list, but only for tracking commandsInFlight
* filter out compatibility tests for triggers before now
* remove commented imports, libraries from new shared converter
* make the TriggerF interpreter tail-recursive
* remove unused compatibility trait
* add back new state logging
* remove refactoring comment
* rewrite some LowLevel initialStates in do
* hide Daml.Script.Free from docs
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* remove forwardPortInitialState
- suggested by @cocreature; thanks
* manually port low-level updates
- suggested by @cocreature; thanks
* remove forwardPort
- suggested by @cocreature; thanks
* fail faster on unrecognized TriggerF
- suggested by @cocreature; thanks
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
changelog_begin
changelog_end
Adapt test for small error message change. An improvement! Previously an internal message was shown. Make test flexible enough to pass for old & new message.
Adapt expected output file. Only change is the contract-ids for active-contracts when the test ends.
fix small test bugs in testcase where authorization is wrong
temp disabled 3 tests; needs investigation; see TODO markers
temp adapt 1 test for change in error message
temp disable 1 test. needs invesigation
undo accidentally commited change to .bazelrc
add copyright header to new file
remove testcase (badActorCheck2) which is no longer expected behaviour, when authorization occurs during execution
address comments: be mre private & other tiny changes
appease scala formatter
improve expected error message in KeyNotVisibleStakeholders testcase
fix authorization issues and re-enable the 3 failing tests in EngineTest which now pass
fix auth issue and re-enable ledger-test-tool test: WronglyTypedContractIdIT
fix compatibility
re-enable test in KVUtilsTransactionSpec.scala
CHANGELOG_BEGIN
CHANGELOG_END
Update exclusions to include new snapshot
changelog_begin
changelog_end
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
The new behavior here was only added in SDK 1.4.0 so the tests
fail (as expected) for older SDK releases. We only test against the
latest stable release on each PR so we didn’t catch this directly on CI.
changelog_begin
changelog_end
* submitMustFail test-case
* script compat test for submitMustFail
changelog_begin
changelog_end
* Support submitMustFail in DAML Script Service
Pushes handling of `submitMustFail` into the DAML script interpreter
rather than DAML script itself.
* fmt
* Simplify IdeClient.submitMustFail
* Comment on case we don't expect to hit
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Not quite sure if this will help but I’ve only seen checksum
mismatches here and not for other Maven artifacts so worth a try at least.
changelog_begin
changelog_end
* Factor out tar/gzip reproducibility flags
* use mktgz in package-app
* Bazel managed tar/gzip
* Remove quiet = True
As stated in the comment this is no longer required with Bazel >= 3.0.
* Build package-app as a sh_binary
This way Bazel will manage the runtime dependencies tar, gzip, mktgz,
and patchelf.
package-app.sh changes directory so it needs to make sure that all paths
are absolute and that the runfiles tree/manifest location is forwarded
to programs called by package-app.sh.
* Avoid file path too long errors
* Fix readlink -f on MacOS
* Document abspath
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Upgrade nixpkgs revision
* Remove unused minio
It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.
* Update Bazel on Windows
changelog_begin
changelog_end
* Fix hlint warnings
The nixpkgs update implied an hlint update which enabled new warnings.
* Fix "Error applying patch"
Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.
* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681
Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.
```
git diff ':(exclude)WORKSPACE'
```
* Update rules_nixpkgs
* nixpkgs location expansion escaping
* Drop --noincompatible_windows_native_test_wrapper
* client_server_test using sh_inline_test
client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.
This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.
* daml_test using sh_inline_test
* daml_doc_test using sh_inline_test
* _daml_validate_test using sh_inline_test
* damlc_compile_test using sh_inline_test
* client_server_test find .exe on Windows
* Bump Windows cache for Bazel update
Remove `clean --expunge` after merge.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Add a --timeout option to `daml ledger` commands
At least in the compatibility tests, we see timeouts fairly regularly
in the tests for fetch-dar. If we see those timeouts in the tests, I
see no reason why our users should never see that especially on
actually distributed ledgers and hardcoded timeouts are a bad idea
anyway so now they are configurable.
I’ve also changed the timeout for the fetch tests (those are the only
once I’ve seen timeout) to 120s.
changelog_begin
- [DAML Assistant] The `daml ledger` commands now accept a `--timeout`
option which can be used to change the timeout for API requests from
the default of 30s.
changelog_end
* Generate parties as uuids to avoid collisions
changelog_begin
changelog_end
* Match on Deadline Exceeded as well
changelog_begin
changelog_end
* Update compatibility/bazel_tools/test_utils/DA/Test/Util.hs
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Update daml-assistant/daml-helper/src/DA/Daml/Helper/Main.hs
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Add some docs
changelog_begin
changelog_end
* Better version checks
changelog_begin
changelog_end
* fix imports
changelog_begin
changelog_end
* This is not how you exception
changelog_begin
changelog_end
* Update daml-assistant/daml-helper/src/DA/Daml/Helper/Main.hs
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
lodash released a non-broken version now and while dependabot isn’t
smart enough to realize this one has a vulnerability it still seems
like a good idea to bump it.
changelog_begin
changelog_end
* Add `platform-version` field to `daml.yaml`
This PR adds the `platform-version` field to `daml.yaml`. Based on the
approach agreed upon in #6558, the logic for this all sits in
`daml-helper` and there are no changes to the assistant.
The details of how the logic work are in a comment so I’m not going to
repeat them here but the commands that are affected are:
- `daml sandbox`
- `daml sandbox-classic`
- `daml json-api`
- `daml start` (but only for sandbox and the JSON API, not for
Navigator or anything else)
For tests, I’ve added a test to the compat workspace that installs two
SDKs simultaneously and tries out various combinations and verifies
that we get the correct version. Open to other ideas for testing this
but that seemed like the most sensible option that actually tests what
we run.
changelog_begin
- [DAML Assistant] You can now specify the version of Sandbox and the
JSON API independently of your SDK version by setting
``platform-version`` in your ``daml.yaml``. This is useful if you
are deploying to a ledger that is running components from a
different SDK version. See
https://docs.daml.com/tools/assistant.html#project-config-file-daml-yaml
for details.
changelog_end
* Run platform-version tests
changelog_begin
changelog_end
* Fix tag globbing
changelog_begin
changelog_end
* fmt
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* Try to fix env vars
changelog_begin
changelog_end
* Remove hardcoded references to 1.2.0
changelog_begin
changelog_end
* Rephrase doc comment
changelog_begin
changelog_end
* get things to compile
changelog_begin
changelog_end
* maybe fix things for realz
changelog_begin
changelog_end
* Remove debugging output
changelog_begin
changelog_end
* Get angry at windows
changelog_begin
changelog_end
* why is windows
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
The default cache directories in yarn don’t work in the Bazel
sandbox. This results in Yarn creating tons of temp directories for
each test that are never cleared up.
This PR changes this to create actual temp directories that are
properly cleaned up.
changelog_begin
changelog_end
The issue is described in a comment (since I want to preserve it) but
here’s the brief summary:
Calling `follow` twice can result in contention if we do not wait in
between. That is shown in the log of the JSON API. For some reason
that ends up propagating to a misleading Target closed error from
puppeteer.
This PR addresses the problem by waiting for the list of followers to
be updated which definitely implies that the ledger has seen the
command.
changelog_begin
changelog_end
* ANF transformation in Speedy.
The idea behind this PR is to transform speedy expressions into a simpler form where all non-atomic sub-expressions are made explicit by the introduction of let-forms. In particular, for the function-application form. These simpler forms allow the execution engine to take advantage of the atomic assumption, and often removes many additional execution steps. In particular the pushing of continuations to allow execution to continue after a compound expression has been reduced to a value.
changelog_begin
changelog_end
* improve comment
* inline functions relocateA/L
* remove comment about scalafmt
* remove commented out alterative def for transformLet1
* improve code by adding incr methods to DepthA/E
* remove (n == 0) special case in trackBindings
* clarify comment further
* improve validate/go to not consume stack for deeply right-nested let-expressions
* address comments from Remy: be private; use final case case; etc
* rename to unsafeCompilationPipeline
* add back some trailing commas
* remove commented-out debug line
* improve comment
* remove dev/debug code in compilationPipeline
* remove commented out code in SEAppGeneral.execute
* undo unrelated code improvement in SValue.scala
* fix compile. object Anf cannot be private
I don’t know if this fixes the “Target closed” issues since I never
managed to reproduce them but I’d rather debug them on the latest
version :)
changelog_begin
changelog_end
I broke the tests for Sandbox < 1.2.0 when I switched to explicit
party allocation yesterday since I didn’t realize this only landed in
SDK 1.2.0 (sorry about that).
changelog_begin
changelog_end
This introduces the same trick for sharing the postgres instance that
we use in the main workspace to the compatibility workspace. The bash
code for that is currently duplicated. Didn’t seem worth trying to
make it sufficiently generic that we can share it (it hasn’t changed
since we added it) but if someone disagrees, I’m happy to reconsider
that.
changelog_begin
changelog_end
* Allocate parties in test-tool compat tests
Compatibility is really something you care about in a production
setup. Implicit party allocation is a development feature and not
available on most ledgers. Therefore, this PR switches the tests over
to allocate parties explicitly. For sandbox-next we also disable the
implicit party allocation feature completely which allows us to
include the ClosedWorldIT test. Sadbonx-classic does not allow
disabling it atm.
In principle, we could run in both configurations but I don’t think
the additional costs (we have hundreds of those tests and this would
double the number) are worth it due to the same reasons that I’ve
given at the beginning for why explicit allocation is more important.
changelog_begin
changelog_end
* Fix postgresql tests
changelog_begin
changelog_end
This PR updates the lower bound of the api test tool required to use the
granular test selection feature to the first snapshot that includes it
(as opposed to the floating 0.0.0).
CHANGELOG_BEGIN
CHANGELOG_END
The //:migration-stable test-cases had issues with test results not
being invalidated after an update of the HEAD SDK. This may be due to
the SDK distribution files being undeclared dependencies of the `daml`
binary. This change adds the SDK distribution files to the `data`
attribute of the generated `daml` `cc_binary` to ensure that changes are
noticed by Bazel and cached test-cases invalidated.
The corresponding runfiles tree includes a symlink per file of the SDK
release. At the time of commit this amounts to ~9MiB of symlinks for one
SDK version (0.0.0 in particular).
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This is the final bit extracted from #6314. This PR adds the capability
to filter individual test cases (for either inclusion or exclusion)
while being completely backwards compatible. It also introduces a new
cli option, `--list-all`, to list all test cases, keeping the behaviour
of the existing `--list` option unchanged, i.e. only print test suites.
Unlike #6314, this is still building upon the idea of defining test
suites as maps, preserving the known-good order of tests and thereby not
introducing flakiness into our builds.
CHANGELOG_BEGIN
- [Ledger API Test Tool] ``--exclude`` and ``--include`` now match the
full test name as a prefix, rather than just suite names. Test name is
built by combining the suite name with a test identifier, so this change
should be fully backwards compatible. Run with ``--list-all`` to list
all tests (as opposed to just the test suites with ``--list``).
CHANGELOG_END
After adding ranges for sandbox versions, this PR now also adds ranges
for test tool versions. This is particularly useful since this gets us
to the point where adding a new version usually (unless we’ve actually
broken compatibility more than before) does not require any change in
exclusions.
I’ve also cleaned up exclusions that only affect snapshots. While I’ve
previously argued against that since that seemed more annoying to
maintain, the list is small enough now that I’m no longer worried
about that.
changelog_begin
changelog_end
Given that sadbonx-classic came back from the dead my initial
reasoning that we don’t have to include it since it will be gone soon
anyway is obsolete. Therefore, this PR adds both the in-memory and
postgresql variant to the ledger-api-test-tool tests. For other tests,
I did not yet create variants for sadbonx-classic. Not sure how
important that is, we can always add it later.
changelog_begin
changelog_end
* Validate data-dependencies test DARs
In addition to building the coin upgrade DAR with data-dependencies on
the coin1 and coin2 packages built with an older SDK, this will also
define cases that run `damlc validate-dar` on the compiled coin upgrade
package.
changelog_begin
changelog_end
* fmt
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This PR does two things:
1. It makes use of the new sha256sums file to speed up the script to
update the versions significantly. The code here is a bit hacky but
as mentioned in the comment, this script is only run offline atm so
that seems reasonable.
2. It prunes SDK snapshot versions to those for which there is no
corresponding stable release.
changelog_begin
changelog_end
`waitForSelector` returns immediately if the selector is already
present (which is documented). This means that the waitForSelector
after the second follow isn’t doing anything since we already waited
for after the first follow. `waitForFunction` seemed like the simplest
solution and doesn’t require patching the HTML which is a bit finnicky
in the compat tests.
changelog_begin
changelog_end
* Upgrade puppeteer
We’ve seen a couple of issues in the compatibility tests of the form
```
Error: Protocol error (Runtime.callFunctionOn): Target closed.
```
Looking at the issue tracker in puppeteer this might be fixed in newer
versions and I don’t see why we should stick to a fairly old version
anyway.
changelog_begin
changelog_end
* Upgrade nodejs
changelog_begin
changelog_end
* temporary add a step to kill node_modules
changelog_begin
changelog_end
* Kill live server and try to fix Windows
changelog_begin
changelog_end
* Undo rm
changelog_begin
changelog_end
* Build CopyTrigger example in compatibility
* Test DAML trigger with test DAML script
Defines a custom client script that runs the trigger runner in the
background and uses a DAML script in the foreground to test that the
trigger runs and its effects are observable.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestions from code review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Use bazel_tools:versions.bzl%versions.is_at_least
Also align versions.is_at_least|most with skylib
The arguments to the Skylib function are in the opposite order. I've
included a docstring to explain
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
We have seen a couple of timeouts on CI. I haven’t reproduced any
locally and even on CI there haven’t been too many (remember that we
run those tests very often in different configurations) so going for a
fairly conversative bump in timeouts but I’m just guessing here.
changelog_begin
changelog_end
* Support version ranges in test tool exclusions
This cleans up the map of exclusions for the ledger-api-test-tool part
of the compat tests. For now, you can only use ranges for the inner
map. In principle we could also add it to the outer map but I’ll leave
that for a separate PR.
I’ve also run the script to update versions so this includes the
latest two snapshots.
fixes#5851
changelog_begin
changelog_end
* Address review comments
changelog_begin
changelog_end
* buildifier
changelog_begin
changelog_end
Previously we ran `daml codegen js` and `daml build` in the tests and
we also ran it twice. While this makes sense for the SDK integration
tests, it doesn’t really make sense for the compatibility tests. These
are SDK-only features so they don’t actually test compatibility. These
steps are super slow and running them in the tests also means that we
run them a quadratic number of times.
This PR moves the `daml build` and the `daml codegen js` step to
separate genrules. In the actual test, the only expensive steps are to
run `yarn install` and the actual jest tests. In principle, we could
probably try to factor out the `yarn install` step as well but I’m a
bit scared of coyping around node_modules so I’ll not attempt to do
that for now.
changelog_begin
changelog_end
* Include puppeteer tests in compat tests
This PR adds the puppeteer based tests to the compatibility
tests. This also means that they are now actually compatibility
tests. Before, we only tested the SDK side.
Apart from process management being a nightmare on Windows as usually,
there are two things that might stick out here:
1. I’ve replaced the `sh_binary` wrapper by a `cc_binary`. There is a
lengthy comment explaining why. I think at the moment, we could
actually get rid of the wraper completely and add JAVA to path in
the tests that need it but at least for now, I’d like to keep it
until we are sure that we don’t need to add more to it (and then
it’s also in the git history if we do need to resurrect it).
2. These tests are duplicated now similar to the `daml ledger *`
tests. The reasoning here is different. They depend on the SDK
tarball either way so performance wise there is no reason to keep
them. However, we reference the other file in the docs which means
we cannot change it freely. What we could do is to make this
sufficiently flexible to handle both the `daml start` case and
separate `daml sandbox`/`daml json-api` processes and then we can
reference it in the docs. There is still added complexity for
Windows but that’s necessary for users as well that want to run
this on Windows so that seems unavoidable. (I should probably also
remove my snarky comments 😇) I’d like to kee it duplicated
for this PR and then we can clean it up afterwards.
changelog_begin
changelog_end
* Bump timeouts
changelog_begin
changelog_end
For some reason this seems to be failing in particular on Linux but
I’ve now also seen it fail on MacOS. My suspicion is that running it
in parallel, somehow corrupts the cache but I don’t understand the details.
changelog_begin
changelog_end
* Include the first feature in compat tests for create-daml-app
This adds the step from the create-daml-app tests that applies the
patch for the messaging feature and tests that things will build. This
is the last step before we can actually run the puppeteer tests which
will turn this into an actual compatibility test.
changelog_begin
changelog_end
* temporarily run all tests
changelog_begin
changelog_end
* Fix exclusions
changelog_begin
changelog_end
* Revert "temporarily run all tests"
This reverts commit 7425dd09cf48f2bfd6064b55d857c76d51afc821.
* Remove accidentally committed messaging.patch
changelog_begin
changelog_end
This is the first part of #5700
It adds tests that build create-daml-app using `daml build` and then
run the codegen and build the UI. Contrary to our main tests these
also run on Windows. This is actually reasonably simple by first
building the typescript libraries on Linux and then downloading them
on Windows.
There are two parts that are still missing from the tests in the main
workspace:
1. Building the extra feature. This should be fairly easy to add.
2. Running the pupeeter tests. At least MacOS and Linux should be
reasonably easy. I don’t know what horrors Windows will throw at
us. This step is what actually makes this a compatibility
test. Currently it doesn’t actually launch Sandbox and the JSON API.
Since this PR is already pretty large, I’d like to tackle those things
separately.
changelog_begin
changelog_end
While the tests are exclusive and I’ve kept them exclusive for
resource usage, the hardcoded port numbers can still cause issues if
for whatever reason you have a postgres instance running outside of
Bazel. This could happen because you simply have a local postgres
instance for testing other things or, what seems to have happened on
CI, due to a leftover postgres instance from a previous test.
While we should do something about the latter, this at least gets us
into a situation where the node isn’t completely broken at this point.
changelog_begin
changelog_end
* Implement version name shortening for Windows
CHANGELOG_BEGIN
CHANGELOG_END
* Temporarily disable `--quick`.
* Revert "Temporarily disable `--quick`."
This reverts commit 025e6c2ddd.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This changes the code for starting postgres to print the log if
`pg_ctl start` fails.
In addition to that it marks the migration tests exclusive since they
rely on this code which has a hardcoded port number.
I tested this on CI by temporarily disabling --quick.
changelog_begin
changelog_end
* Make Bearer prefix optional in Haskell ledger bindings
This matches the change to the JVM bindings in 1.0.0
changelog_begin
- [DAML Assistant] The ``daml ledger`` commands no longer require the
``Bearer `` prefix in the access token file. This matches the behavior
of DAML Script and other SDK tools.
changelog_end
* Fix tasty crash
changelog_begin
changelog_end
* Update compatibility/bazel_tools/daml_ledger/Main.hs
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
* Update SDK versions in compatibility tests
This adds a Haskell script to generate a versions.bzl file that
contains the list of versions as well as their hashes. This should
make it a bit easier to keep things up2date going forward.
The script is a bit slow since downloading all the SDKs takes quite a
while but for now it should be good enough and is much more pleasant
than having to figure this out manually.
changelog_begin
changelog_end
* Address review comments
changelog_begin
changelog_end
* Fix excluded tests
changelog_begin
changelog_end
* Add basic Sandbox data continuity tests
This adds some basic tests that check that data migrations work
properly. For now, I use DAML Script to create and query contracts at
each step. This isn’t perfect since queries can only use the active
contract service but not things like the transaction stream but it’s
clearly better than nothing.
The runner for executing the tests is a simple Haskell executable. It
didn’t really seem useful to throw tasty at this.
I’ve added two sets of tests, one that runs only through stable
versions and one that includes snapshots since migrating through
snapshots is not necessarily equivalent.
Sadly these tests use sandbox-classic since I discovered while writing
these tests that sandbox-next does not actually support migrating data
between SDK versions.
changelog_begin
changelog_end
* Use the sandbox module instead of a custom withSandbox
changelog_begin
changelog_end
* Update compatibility/sandbox-migration/SandboxMigrationRunner.hs
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* daml new script-example
* Build DAML script DAR
* daml_script_dar macro over sdk_version
* Run an individual daml-script test
* DAML script test matrix
* format
CHANGELOG_BEGIN
CHANGELOG_END
* Use named arguments on daml_script_test
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Set JVM memory limits for sandbox
Using the same settings as defined in `@daml//bazel_tools:scala.bzl`.
* Mark tests as large
Their memory consumption is somewhere around 300MiB which is considered
"large" according to
https://docs.bazel.build/versions/master/be/common-definitions.html?cl=head#common-attributes-tests.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Make compat tests work on windows
This required some changes to the daml_sdk rule since the read-only
installation by the assistant breaks Bazel completely. We could only
apply those changes on Windows but I think I prefer the consistency
across platforms here over trying to stay close to how the SDK is
installed on user machines given that the SDK installation is not
something we’ve had issues with.
I’ve excluded the postgresql tests for now. I don’t expect them to be
particularly hard to fix but I’ve already spent almost 2 days on this
and having some tests run on Windows seems like a clear improvement
over running no tests on Windows :)
changelog_begin
changelog_end
* Remove todo
changelog_begin
changelog_end
* Copy daml ledger test to compatibility
Specifically:
- `daml-assistant/daml-helper/test/DA/Daml/Helper/Test/Deployment.hs`
- `libs-haskell/test-utils/DA/Test/Sandbox.hs`
* Adapt daml ledger test for compatibility
Avoid dependencies on the daml repository and make the SDK and platform
components configurable.
* Add test certificates to compatibility
Taken from `@daml//ledger/test-common/test-certificates`.
* Define daml_ledger_test
CHANGELOG_BEGIN
CHANGELOG_END
* Add note on keeping daml ledger tests in sync
* ./fmt.sh
* Strictly require --sdk-version flag
Tasty doesn't allow options without a default value, so we default to
`error` to enforce a value being set.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This PR extends the existing Linux compatibility tests to run on macOS
too. Fixes#5692.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Fix ./fmt.sh check
We accidentally introduced hlint lints when we added the compatibility
workspace. The hlint code in ./fmt.sh correctly detects that there is
a hint but then exits with 0 since it uses the exit code from `echo`
rather than the one from `hlint`. The info that we printed with
`echo` didn’t actually make any sense so I added a new comment as to
why we run hlint here. This requires some setup changes so that the
compatibility workspace gets the same Haskell flags since otherwise
using the same hlint rules obviously does not make sense.
Fixes#5701
changelog_begin
changelog_end
* Update fmt.sh
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
* Extend compatibility tests to DAML on SQL
This feels a bit hacky since the runfiles don’t work quite like I
would expect them to but it’s at least not more hacky than what we do
for the head-based tests we currently have.
Progress towards #5695
changelog_begin
changelog_end
* Fix runfiles with more bash
changelog_begin
changelog_end
* remove redundant port options
changelog_begin
changelog_end
* Create fewer sandbox targets
changelog_begin
changelog_end
* Apply suggestions from code review
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Fix runfiles snippet
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Abstract over SDK/platform client_server_test
* Define test-matrix using simple list comprehension
CHANGELOG_BEGIN
CHANGELOG_END
* copy right header
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This is a first step towards testing cross-version
compatibility. It doesn’t actuall do much yet but hopefully it should
be easier to parallelize once we have the initial boilerplate in place
so ideally I’d like to address most missing things and issues in
separate PRs.
changelog_begin
changelog_end
