* DAML-LF: Add interning for type to DAML-LF 1.dev
We add two new features to DAML-LF 1.dev:
* a per package list (or table) of `Type` messages, and
* a new case in the `Type` message which is an index into this table.
In combination, these two features can be used to allow DAML-LF
encoders to perform hash-consing of `Type` messages. We also change the
Haskell implementation of our DAML-LF encoder to do exactly that when
targetting DAML-LF 1.dev.
Doing this has a few benefits:
1. The DALFs produced by `damlc` get smaller: I've seen a case where
the size dropped from 69MB to 45MB.
2. DAML-LF decoders need to decode less data.
3. Decoded packages use less memory because identical structures are
now shared. This is particularly helpful in situations where we need
to keep the interface (or signature) of a package in memory for a
long time.
This PR mostly takes care of the Haskell implementation. However, we
need to make the Scala implementation of the decoder aware of the new
features as well since we have tests that load DAML-LF 1.dev into the
engine. A decoder and _targeted_ tests on the Scala side will follow
in a separate PR.
CHANGELOG_BEGIN
CHANGELOG_END
* Make jq tests aware of type interning
CHANGELOG_BEGIN
CHANGELOG_END
* Improve jq test
CHANGELOG_BEGIN
CHANGELOG_END
* Apply Remy's suggestions
Co-authored-by: Remy <remy.haemmerle@daml.com>
* Improve the imperative bits
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Remy <remy.haemmerle@daml.com>
* damlc: Check DAML-LF pattern matches for exhaustiveness
`damlc` has always produced exhaustive pattern matches in DAML-LF since
GHC adds a default branch with a call to `error` message as soon as a
pattern match is not exhaustive. It was a complete oversight on our
side that we did not enforce this properly in DAML-LF. Since `damlc`
has never produced non-exhaustive pattern matches, enforcing this now
and for all DAML-LF versions is only theoretically a breaking change,
namely if people hand-crafted DAML-LF, but not practically.
This check is as under-tested as the rest of our Haskell implementation
of the DAML-LF type checker. Well, all our other tests implicitly check
that the type checker does not give false errors. However, we have no
tests ensuring that the type checker is not too permissive. Fixing this
situation would require a big time investment since we currently don't
have a simple way to produce DAML-LF without going through GHC, which
will always produe well-typed DAML-LF.
I will update the DAML-LF specification wrt pattern matching
exhaustiveness in a separete PR.
This PR does not have a changelog entry since there's no impact for our
users.
CHANGELOG_BEGIN
CHANGELOG_END
* Make check linear in match size not constructor number
CHANGELOG_BEGIN
CHANGELOG_END
* Move lookupWithIndex in Data.List.Extended
CHANGELOG_BEGIN
CHANGELOG_END
* Implement the suggestions
CHANGELOG_BEGIN
CHANGELOG_END
* Haskell: Add hlint rule to suggest foldl' over foldl
`foldl` is lazy in a way that almost never is what you want since it
can cause space leaks without any benefit. `foldl'` does not have this
problem. See https://www.well-typed.com/blog/2014/04/fixing-foldl/ for
more details.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix all existing occurrences of foldl
CHANGELOG_BEGIN
CHANGELOG_END
* daml ledger: Infrastructure for tests
This adds infrastructure for testing `daml ledger` and moves one test
from the assistant integration tests to the newly created tests.
CHANGELOG_BEGIN
CHANGELOG_END
* Update libs-haskell/test-utils/DA/Test/HttpJson.hs
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
* Update libs-haskell/test-utils/DA/Test/HttpJson.hs
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
* small fixes
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
An extraction script that makes the LF reading libraries (DAML LF archive decoding to an AST) usable in other Haskell projects.
This script once existed but is not available any more, so I have resurrected the script, adapted to current usage, and upgraded to the newest stackage LTS that can support it. These libraries _could_ be versioned manually to released SDKs and published if desirable (without making any guarantees about the stability, therefore prefixing the version with `0.`.
CHANGELOG_BEGIN
Tool to extract Haskell libraries for DAML LF archives for use in other Haskell projects
CHANGELOG_END
This gives us visibility into issues w/ sandbox in the
tls/deployment/ledger tests. This change only affects test
output.
changelog_begin
changelog_end
* Upgrade nixpkgs revision
* Remove unused minio
It used to be used as a gateway to push the Nix cache to GCS, but has
since been replaced by nix-store-gcs-proxy.
* Update Bazel on Windows
changelog_begin
changelog_end
* Fix hlint warnings
The nixpkgs update implied an hlint update which enabled new warnings.
* Fix "Error applying patch"
Since Bazel 2.2.0 the order of generating `WORKSPACE` and `BUILD` files
and applying patches has been reversed. The allows users to define
patches to these files that will not be immediately overwritten.
However, it also means that patches on another repository's original
`WORKSPACE` file will likely become invalid.
* a948eb7255
* https://github.com/bazelbuild/bazel/issues/10681
Hint: If you're generating a patch with `git` then you can use the
following command to exclude the `WORKSPACE` file.
```
git diff ':(exclude)WORKSPACE'
```
* Update rules_nixpkgs
* nixpkgs location expansion escaping
* Drop --noincompatible_windows_native_test_wrapper
* client_server_test using sh_inline_test
client_server_test used to produce an executable shell script in form of
a text file output. However, since the removal of
`--noincompatible_windows_native_test_wrapper` this no longer works on
Windows since `.sh` files are not directly executable on Windows.
This change fixes the issue by producing the script file in a dedicated
rule and then wrapping it in a `sh_test` rule which also works on
Windows.
* daml_test using sh_inline_test
* daml_doc_test using sh_inline_test
* _daml_validate_test using sh_inline_test
* damlc_compile_test using sh_inline_test
* client_server_test find .exe on Windows
* Bump Windows cache for Bazel update
Remove `clean --expunge` after merge.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This limits the JVM max memory and initial memory in a few tests that
look like they might be using more than they have two and that run for
a long time so there is a high chance they end up running in parallel
with something else.
changelog_begin
changelog_end
* Update rules_haskell hie-bios support
* Decouple Haskell ghcide and DAML ghcide
Creates a separate `stack_snapshot` to pull in `ghcide` for the Haskell
IDE use case independent of the `ghcide` for DAML. This allows to update
these two `ghcide` instances independently. As DAML uses `ghcide` the
library updates can be involved if the API experienced breaking changes.
At the same time we may wish to update `ghcide` for Haskell earlier to
make use of new features and stay compatible with rules_haskell's ghcide
support.
* Fix Haddock warnings reported by ghcide
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Fix leakage of SDK installations
This fixes an issue where the integration tests leaked SDK
installations. Easily verified by `ls /tmp/extra-dir*` before and
after running the tests.
changelog_begin
changelog_end
* I hate windows so much
changelog_begin
changelog_end
* Update rules_haskell
- Fixes the issue where a `sh_test` wrapping a `haskell_binary` couldn't
add runfiles as is the case with other Bazel rules.
CHANGELOG_BEGIN
CHANGELOG_END
* Save the runfiles environment at start-up
To work around the fact that `withProgName` overwrites `argv[0]`.
See https://gitlab.haskell.org/ghc/ghc/-/issues/18418.
* damlc_compile_test include damlc runfiles
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Previously that resulted in us telling sandbox,navigator,… to connect
to port 0 which obviously does not work. I’ve changed the types a bit
to avoid accidentally passing on the cli argument to them.
changelog_begin
changelog_end
* Haskell: Add assertFileExists to DA.Test.Util
This PR adds a helper function `assertFileExists` that captures the
`doesFileExist ... >>= assertBool ...` pattern that is very common in
our Haskell test suites. It also adds the inverse
`assertFileDoesNotExist` function. Both functions are now used where
appropriate.
CHANGELOG_BEGIN
CHANGELOG_END
* Add directory dependency to test-util lib
CHANGELOG_BEGIN
CHANGELOG_END
* Split up repl tests to make them faster
This PR splits up the tests into the tests for TLS and Auth and the
tests for the actual functionality.
The func tests use the repl as a library which allows them to be
significantly faster:
1. We only need to start the service process once.
2. We only need to initialize the package db once.
There is at least one other point that I did not address for now and
that is only loading the packages into the repl service once. While
loading them multiple times is a noop, it still has a performance
implication.
Sadly, this has turned out much more messy than I thought it would be
due to various issues with haskeline/repline/tasty/computers. The
details are in a comment in DA.Test.Repl.FuncTests.
changelog_begin
changelog_end
* Try to nuke cache on window selectively
* Enable cache again
- Move deployment tests (deployTest, fetchTest) out of integration-tests.
- Use DA.Test.Sandbox where appropriate.
- Split out code for useful test patterns: i.e. calling commands quietly, getFreePort.
changelog_begin
changelog_end
* Move 'withEnv', call it from daml2ts tests
changelog_begin
changelog_end
* Fix withEnv call to ensure that TASTY_NUM_THREADs is set
withEnv replaces the whole environment so we need to set everything we
care about.
* withEnv replaces the whole environment so we need to set everything we
care about.
* Apparently applying the same fix has destabilized Windows
* Try even harder to get daml assistant tests passing on Windows again
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
This ensures that -l and -p work properly in the integration tests
since they no longer depend on the order.
There is lots of other crap to cleanup in those tests but I’m trying
to keep it to small steps.
changelog_begin
changelog_end
* Integrate create-daml-app into the assistant
fixes#4868
changelog_begin
- [DAML Assistant] Add a new ``daml create-daml-app`` command for creating a project based on
`create-daml-app <https://github.com/digital-asset/create-daml-app>`_.
changelog_end
* Try random things hoping to fix windows
* Try random things hoping to fix things on macos
We have seen a few test failures where the port file was not written
to in time. Apparently this can takes ages on CI for tests that are
not marked exclusive.
changelog_begin
changelog_end
* Support authentication and TLS in DAML repl
changelog_begin
- [DAML Repl - Experimental] You can now connect to a ledger via TLS
by passing ``--tls`` to ``daml repl``
- [DAML Repl - Experimental] You can now connect to a ledger with
authentication by passing the token via ``--access-token-file`` to
``daml repl``.
changelog_end
* try to fix linking on windows
* windows is weird
* gnah
We previously had 3 slightly different but consistently shitty logic
for handling this in the tests for daml-helper daml repl and the
Haskell ledger bindings. This PR introduces a module that is flexible
enough to capture all their needs and hopefully is somewhat less
shitty.
changelog_begin
changelog_end
As mentioned in the title, this is still very experimental and needs
more work before we want to advertise it. However, the code is in a
somewhat reasonable shape, there are tests and I think even in the
current state it is already useful. Also this PR is already getting
very large so I don’t want to hold off much longer before merging this.
It is included in the SDK but hidden from `damlc --help` and `daml
--help` until the most pressing issues are addressed (primarily around
making sure that it doesn’t just shut down if you have a type error
and better error messages in general).
changelog_begin
changelog_end
This was initially done as an optimization but now that we’ve used
telemetry for a while this hasn’t really been necessary and we’ve
spend way more time on people being confused by the fact that
telemetry messages are only flushed on shutdown.
Furthermore, I’m also slighlty worried that flushing the telemetry
messages on shutdown might go wrong on Windows given that everything
there seems to insist on hard-killing proesses.
changelog_begin
changelog_end
We have seen a few telemetry messages with the SDK version set to
`null` which is obviously not very useful. I suspect this is probably
because some users invoke damlc directly. This change is backwards
compatible since the ToJSON instance does encode `version = Just "x"` as
`"version" = "x"` and version = Nothing as `version = null`.
changelog_begin
changelog_end
* Report assistant commands and errors via a logger.
CHANGELOG_BEGIN
CHANGELOG_END
* Hook up daml-assistant to a GCP logger.
* fix test case
* fix more tests
* Check opted in status in assistant.
* Anonimize args that have unexpected characters.
* More agressive anonimization
* add missing containers dependency
Previously, we did not send any message when users simply clicked away
the telemetry popup. Now we send a special message similar to the
opt-out message which only includes the machine id.
I’ve also changed the opt-out message to include the machine id.
changelog_begin
changelog_end
* Shut down when stdin is closed.
changelog_begin
- [DAML Assistant] The DAML assistant will now shut down long-running
processes like sandbox when stdin is
closed. This is mainly useful on Windows, where process APIs often
kill the process in a way that does not allow it to do any cleanup, in
particular, we cannot stop child processes.
changelog_end
For now, the logic for this is only in daml-helper which is the only
thing starting long-running processes (in particular sandbox). There
is a long inline-comment explaining why this is not on the assistant
itself.
fixes#4168
* Refactor damldocs
* Refactor damldocs.
changelog_begin
changelog_end
* More refactoring
* Update copyright headers.
* Add export list for Templates
* Add export list to .Util
* added a little ^
* Improve bash completions for daml-assistant.
* Install bash completion script automatically
* Better default logic for bash completions
* specifically -> explicitly
* Copyright headers
* Better hook logic and refactoring
* Handle non-standard installations more robustly.
* Handle CI env & add changelog note.
CHANGELOG_BEGIN
- [DAML Assistant] Bash completions for the DAML assistant are now
available via ``daml install``. These will be installed automatically
on Linux and Mac. If you use bash and have bash completions installed,
these bash completions let you use the tab key to autocomplete
many DAML Assistant commands, such as ``daml install`` and
``daml version``.
CHANGELOG_END
* Mention bash completion in assistant docs
* Remove promises
* Move Any wrappers and Archive to stable packages
There are no actual API or functionality changes in this PR but the
logic for locating the stable packages has slightly changed since the
Any wrappers package only makes sense for LF 1.7. To address this, we
simply filter out stable packages for newer LF versions since it
doesn’t make sense to depend on those anyway.
CHANGELOG_BEGIN
- [DAML Compiler] Move ``Archive`` type to a separate DALF.
CHANGELOG_END
* More comments
* Fix java codegen tests
* fix more tests
* Force cache reset on Windows
* Revert "Force cache reset on Windows"
This reverts commit 9f2b7d70b2.
* Extend Priority type with Telemetry value
* Include Telemetry logging level in logger
* Log viewing of scenario results
* Update ghcide with telemetry logging level
* language: compile everything in the source directory
This removes the need to specify a 'main'. Instead we 'source' in
daml.yaml should point to the source root directory.
* Delete obsolete proto3-suite patch
My patch has been upstreamed so no need to keep it around in our repo.
* Upgrade proto3-wire
* Adapt to changes in proto3-suite
Add a command line option to `damlc inspect` which allows for configuring the
details level of the pretty printed DAML-LF. Right now the only difference is
that levels bigger than 0 print all location information. Level 0, which is
the default, also prints location information top level declarations.
This feature is useful for debugging location information.
For example, we print two spaces if there's no annotation between `def` and
the function. This PR fixes this by using the `empty` document instead of the
document containing the empty string. (Please don't ask me why they are
different.)
* Supporting producing sdist tarballs for the HS ledger bindings
The README.md has an explanation for how you can use this.
This should hopefully allow others to experiment with the bindings.
