* Parallelize daml-assistant integration tests.
This mainly involves avoiding changing the working directory or the environment while inside tests.
AFAIK the daml start tests can't be parallelized without starting up a sandbox instance each time, which seems bad. These tests could be separated from the rest...
The magic number "2" was picked via experimentation:
```
1 threads -> 306 s
2 threads -> 199 s
3 threads -> 198 s
4 threads -> 195 s
```
changelog_begin
changelog_end
* buildifier-fix
Each of these installations is about 1gb so it’s not that unlikely
that this plays a significant factor in our out of disk space errors.
Although I think we cleanup partially so I’m not sure how large the
part of the SDK installation is that gets leftover.
changelog_begin
changelog_end
* daml build: add a --access-token-file for remote dependencies
This adds a `ledger.access-token-file` field in the `daml.yaml` project
file and a `--access-token-file` flag to `daml build` to authorize
querying/fetching of remote dependencies.
CHANGELOG_BEGIN
[daml build] A new flag `--access-token-file` is added for the `daml
build` command. It allows the specify the path to an access token to
authenticate against the ledger API. This is needed if the project
depends on a remote Daml package hosted on such a ledger. Alternatively,
the path to the token can also be specified in the `daml.yaml` project
file under the `ledger.access-token-file` field.
CHANGELOG_END
* Merge Maven uploads for different Scala versions
It turns out Maven will abort an existing staging operation if you
create a new one. This means our jobs race against each other. We
could try to fix that by either sequencing the jobs in a clever
way (annoying and can break things like rerunning if only parts
failed), or by creating more profiles (unclear if you can even have
two profiles for the same group id, even if you do, it’s annoying to
merge).
So in this PR I (grudgingly) merged both uploads into the Haskell
script. This isn’t all bad:
1. It moves some logic from bash embedded in yaml string literals into
Haskell code.
2. It duplicates some versions but it removes duplication in other
places so overall not too much worse.
3. It does however, make things slower. We don’t run this stuff in
parallel. That said, the release step is relatively small (< 5min) and
it only runs on Linux.
We could add CLI arguments to make the Scala versions configurable for
local development. Given that this is blocking releases, I wanted to
get something in that works first and then see what we need in that regard.
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* Port readPortfile changes to compat workspace
Unfortunately we cannot easily share code between the two workspaces
so there is some stuff that is copied over and that we need to keep in
sync. This PR ports the changes from #8669 to the compat workspace.
changelog_begin
changelog_end
* maybe I should test if my code compiles but also I don’t want to
changelog_begin
changelog_end
* .
changelog_begin
changelog_end
* telemetry: use an extra cache directory for telemetry
If the sdk's `.daml` directory is not writable, telemetry fails and
kills the IDE. We add a new assistant environment variable "DAML_CACHE".
If set, this directory is used for telemetry cache data.
Fixes#8396.
CHANGELOG_BEGIN
CHANGELOG_END
* try creating directory
* check for already existing machine id file
* set DAML_CACHE to read-only in assistant integration tests
* DAML-LF: Add interning for type to DAML-LF 1.dev
We add two new features to DAML-LF 1.dev:
* a per package list (or table) of `Type` messages, and
* a new case in the `Type` message which is an index into this table.
In combination, these two features can be used to allow DAML-LF
encoders to perform hash-consing of `Type` messages. We also change the
Haskell implementation of our DAML-LF encoder to do exactly that when
targetting DAML-LF 1.dev.
Doing this has a few benefits:
1. The DALFs produced by `damlc` get smaller: I've seen a case where
the size dropped from 69MB to 45MB.
2. DAML-LF decoders need to decode less data.
3. Decoded packages use less memory because identical structures are
now shared. This is particularly helpful in situations where we need
to keep the interface (or signature) of a package in memory for a
long time.
This PR mostly takes care of the Haskell implementation. However, we
need to make the Scala implementation of the decoder aware of the new
features as well since we have tests that load DAML-LF 1.dev into the
engine. A decoder and _targeted_ tests on the Scala side will follow
in a separate PR.
CHANGELOG_BEGIN
CHANGELOG_END
* Make jq tests aware of type interning
CHANGELOG_BEGIN
CHANGELOG_END
* Improve jq test
CHANGELOG_BEGIN
CHANGELOG_END
* Apply Remy's suggestions
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Improve the imperative bits
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
* damlc: Check DAML-LF pattern matches for exhaustiveness
`damlc` has always produced exhaustive pattern matches in DAML-LF since
GHC adds a default branch with a call to `error` message as soon as a
pattern match is not exhaustive. It was a complete oversight on our
side that we did not enforce this properly in DAML-LF. Since `damlc`
has never produced non-exhaustive pattern matches, enforcing this now
and for all DAML-LF versions is only theoretically a breaking change,
namely if people hand-crafted DAML-LF, but not practically.
This check is as under-tested as the rest of our Haskell implementation
of the DAML-LF type checker. Well, all our other tests implicitly check
that the type checker does not give false errors. However, we have no
tests ensuring that the type checker is not too permissive. Fixing this
situation would require a big time investment since we currently don't
have a simple way to produce DAML-LF without going through GHC, which
will always produe well-typed DAML-LF.
I will update the DAML-LF specification wrt pattern matching
exhaustiveness in a separete PR.
This PR does not have a changelog entry since there's no impact for our
users.
CHANGELOG_BEGIN
CHANGELOG_END
* Make check linear in match size not constructor number
CHANGELOG_BEGIN
CHANGELOG_END
* Move lookupWithIndex in Data.List.Extended
CHANGELOG_BEGIN
CHANGELOG_END
* Implement the suggestions
CHANGELOG_BEGIN
CHANGELOG_END
* Haskell: Add hlint rule to suggest foldl' over foldl
`foldl` is lazy in a way that almost never is what you want since it
can cause space leaks without any benefit. `foldl'` does not have this
problem. See https://www.well-typed.com/blog/2014/04/fixing-foldl/ for
more details.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix all existing occurrences of foldl
CHANGELOG_BEGIN
CHANGELOG_END
* daml ledger: Infrastructure for tests
This adds infrastructure for testing `daml ledger` and moves one test
from the assistant integration tests to the newly created tests.
CHANGELOG_BEGIN
CHANGELOG_END
* Update libs-haskell/test-utils/DA/Test/HttpJson.hs
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
* Update libs-haskell/test-utils/DA/Test/HttpJson.hs
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
* small fixes
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
An extraction script that makes the LF reading libraries (DAML LF archive decoding to an AST) usable in other Haskell projects.
This script once existed but is not available any more, so I have resurrected the script, adapted to current usage, and upgraded to the newest stackage LTS that can support it. These libraries _could_ be versioned manually to released SDKs and published if desirable (without making any guarantees about the stability, therefore prefixing the version with `0.`.
CHANGELOG_BEGIN
Tool to extract Haskell libraries for DAML LF archives for use in other Haskell projects
CHANGELOG_END
This gives us visibility into issues w/ sandbox in the
tls/deployment/ledger tests. This change only affects test
output.
changelog_begin
changelog_end
* Upgrade nixpkgs revision
* Remove unused minio
It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.
* Update Bazel on Windows
changelog_begin
changelog_end
* Fix hlint warnings
The nixpkgs update implied an hlint update which enabled new warnings.
* Fix "Error applying patch"
Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.
* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681
Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.
```
git diff ':(exclude)WORKSPACE'
```
* Update rules_nixpkgs
* nixpkgs location expansion escaping
* Drop --noincompatible_windows_native_test_wrapper
* client_server_test using sh_inline_test
client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.
This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.
* daml_test using sh_inline_test
* daml_doc_test using sh_inline_test
* _daml_validate_test using sh_inline_test
* damlc_compile_test using sh_inline_test
* client_server_test find .exe on Windows
* Bump Windows cache for Bazel update
Remove `clean --expunge` after merge.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This limits the JVM max memory and initial memory in a few tests that
look like they might be using more than they have two and that run for
a long time so there is a high chance they end up running in parallel
with something else.
changelog_begin
changelog_end
* Update rules_haskell hie-bios support
* Decouple Haskell ghcide and DAML ghcide
Creates a separate `stack_snapshot` to pull in `ghcide` for the Haskell
IDE use case independent of the `ghcide` for DAML. This allows to update
these two `ghcide` instances independently. As DAML uses `ghcide` the
library updates can be involved if the API experienced breaking changes.
At the same time we may wish to update `ghcide` for Haskell earlier to
make use of new features and stay compatible with rules_haskell's ghcide
support.
* Fix Haddock warnings reported by ghcide
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Fix leakage of SDK installations
This fixes an issue where the integration tests leaked SDK
installations. Easily verified by `ls /tmp/extra-dir*` before and
after running the tests.
changelog_begin
changelog_end
* I hate windows so much
changelog_begin
changelog_end
* Update rules_haskell
- Fixes the issue where a `sh_test` wrapping a `haskell_binary` couldn't
add runfiles as is the case with other Bazel rules.
CHANGELOG_BEGIN
CHANGELOG_END
* Save the runfiles environment at start-up
To work around the fact that `withProgName` overwrites `argv[0]`.
See https://gitlab.haskell.org/ghc/ghc/-/issues/18418.
* damlc_compile_test include damlc runfiles
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Previously that resulted in us telling sandbox,navigator,… to connect
to port 0 which obviously does not work. I’ve changed the types a bit
to avoid accidentally passing on the cli argument to them.
changelog_begin
changelog_end
* Haskell: Add assertFileExists to DA.Test.Util
This PR adds a helper function `assertFileExists` that captures the
`doesFileExist ... >>= assertBool ...` pattern that is very common in
our Haskell test suites. It also adds the inverse
`assertFileDoesNotExist` function. Both functions are now used where
appropriate.
CHANGELOG_BEGIN
CHANGELOG_END
* Add directory dependency to test-util lib
CHANGELOG_BEGIN
CHANGELOG_END
* Split up repl tests to make them faster
This PR splits up the tests into the tests for TLS and Auth and the
tests for the actual functionality.
The func tests use the repl as a library which allows them to be
significantly faster:
1. We only need to start the service process once.
2. We only need to initialize the package db once.
There is at least one other point that I did not address for now and
that is only loading the packages into the repl service once. While
loading them multiple times is a noop, it still has a performance
implication.
Sadly, this has turned out much more messy than I thought it would be
due to various issues with haskeline/repline/tasty/computers. The
details are in a comment in DA.Test.Repl.FuncTests.
changelog_begin
changelog_end
* Try to nuke cache on window selectively
* Enable cache again
- Move deployment tests (deployTest, fetchTest) out of integration-tests.
- Use DA.Test.Sandbox where appropriate.
- Split out code for useful test patterns: i.e. calling commands quietly, getFreePort.
changelog_begin
changelog_end
* Move 'withEnv', call it from daml2ts tests
changelog_begin
changelog_end
* Fix withEnv call to ensure that TASTY_NUM_THREADs is set
withEnv replaces the whole environment so we need to set everything we
care about.
* withEnv replaces the whole environment so we need to set everything we
care about.
* Apparently applying the same fix has destabilized Windows
* Try even harder to get daml assistant tests passing on Windows again
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
This ensures that -l and -p work properly in the integration tests
since they no longer depend on the order.
There is lots of other crap to cleanup in those tests but I’m trying
to keep it to small steps.
changelog_begin
changelog_end
* Integrate create-daml-app into the assistant
fixes#4868
changelog_begin
- [DAML Assistant] Add a new ``daml create-daml-app`` command for creating a project based on
`create-daml-app <https://github.com/digital-asset/create-daml-app>`_.
changelog_end
* Try random things hoping to fix windows
* Try random things hoping to fix things on macos
We have seen a few test failures where the port file was not written
to in time. Apparently this can takes ages on CI for tests that are
not marked exclusive.
changelog_begin
changelog_end
* Support authentication and TLS in DAML repl
changelog_begin
- [DAML Repl - Experimental] You can now connect to a ledger via TLS
by passing ``--tls`` to ``daml repl``
- [DAML Repl - Experimental] You can now connect to a ledger with
authentication by passing the token via ``--access-token-file`` to
``daml repl``.
changelog_end
* try to fix linking on windows
* windows is weird
* gnah
We previously had 3 slightly different but consistently shitty logic
for handling this in the tests for daml-helper daml repl and the
Haskell ledger bindings. This PR introduces a module that is flexible
enough to capture all their needs and hopefully is somewhat less
shitty.
changelog_begin
changelog_end
As mentioned in the title, this is still very experimental and needs
more work before we want to advertise it. However, the code is in a
somewhat reasonable shape, there are tests and I think even in the
current state it is already useful. Also this PR is already getting
very large so I don’t want to hold off much longer before merging this.
It is included in the SDK but hidden from `damlc --help` and `daml
--help` until the most pressing issues are addressed (primarily around
making sure that it doesn’t just shut down if you have a type error
and better error messages in general).
changelog_begin
changelog_end
This was initially done as an optimization but now that we’ve used
telemetry for a while this hasn’t really been necessary and we’ve
spend way more time on people being confused by the fact that
telemetry messages are only flushed on shutdown.
Furthermore, I’m also slighlty worried that flushing the telemetry
messages on shutdown might go wrong on Windows given that everything
there seems to insist on hard-killing proesses.
changelog_begin
changelog_end
We have seen a few telemetry messages with the SDK version set to
`null` which is obviously not very useful. I suspect this is probably
because some users invoke damlc directly. This change is backwards
compatible since the ToJSON instance does encode `version = Just "x"` as
`"version" = "x"` and version = Nothing as `version = null`.
changelog_begin
changelog_end
* Report assistant commands and errors via a logger.
CHANGELOG_BEGIN
CHANGELOG_END
* Hook up daml-assistant to a GCP logger.
* fix test case
* fix more tests
* Check opted in status in assistant.
* Anonimize args that have unexpected characters.
* More agressive anonimization
* add missing containers dependency
Previously, we did not send any message when users simply clicked away
the telemetry popup. Now we send a special message similar to the
opt-out message which only includes the machine id.
I’ve also changed the opt-out message to include the machine id.
changelog_begin
changelog_end
* Shut down when stdin is closed.
changelog_begin
- [DAML Assistant] The DAML assistant will now shut down long-running
processes like sandbox when stdin is
closed. This is mainly useful on Windows, where process APIs often
kill the process in a way that does not allow it to do any cleanup, in
particular, we cannot stop child processes.
changelog_end
For now, the logic for this is only in daml-helper which is the only
thing starting long-running processes (in particular sandbox). There
is a long inline-comment explaining why this is not on the assistant
itself.
fixes#4168
* Refactor damldocs
* Refactor damldocs.
changelog_begin
changelog_end
* More refactoring
* Update copyright headers.
* Add export list for Templates
* Add export list to .Util
* added a little ^
* Improve bash completions for daml-assistant.
* Install bash completion script automatically
* Better default logic for bash completions
* specifically -> explicitly
* Copyright headers
* Better hook logic and refactoring
* Handle non-standard installations more robustly.
* Handle CI env & add changelog note.
CHANGELOG_BEGIN
- [DAML Assistant] Bash completions for the DAML assistant are now
available via ``daml install``. These will be installed automatically
on Linux and Mac. If you use bash and have bash completions installed,
these bash completions let you use the tab key to autocomplete
many DAML Assistant commands, such as ``daml install`` and
``daml version``.
CHANGELOG_END
* Mention bash completion in assistant docs
* Remove promises
* Move Any wrappers and Archive to stable packages
There are no actual API or functionality changes in this PR but the
logic for locating the stable packages has slightly changed since the
Any wrappers package only makes sense for LF 1.7. To address this, we
simply filter out stable packages for newer LF versions since it
doesn’t make sense to depend on those anyway.
CHANGELOG_BEGIN
- [DAML Compiler] Move ``Archive`` type to a separate DALF.
CHANGELOG_END
* More comments
* Fix java codegen tests
* fix more tests
* Force cache reset on Windows
* Revert "Force cache reset on Windows"
This reverts commit 9f2b7d70b2.
* Extend Priority type with Telemetry value
* Include Telemetry logging level in logger
* Log viewing of scenario results
* Update ghcide with telemetry logging level
