* Refactor GenerateStablePackages
Summary of changes:
* Split GenerateStablePackages executable into a library (containing the stable package data) and an executable (that writes out the packages)
* Hook up data-dependencies test to the new library so we don't have to manually update the number of stable packages every time we add a stable package or release a new LF version.
changelog_begin
changelog_end
* buildifier-fix
This fixes a race condition in our handling of scenario contexts. See
the comments for details. I verified with a bunch of extra logging
that this is what is actually failing in our tests. I’ll try to
upstream the logging separately since ideally I’d like to have that in CI.
I ran all integration tests with --runs_per_test=20 over night and
with this change I’m no longer able to get it to flake so dropping the
flaky marker.
fixes#6910
changelog_begin
changelog_end
* Parallelize the data-dependencies test.
It runs about twice as fast on my laptop (240s -> 120s) when allowed to
run tests in parallel. I'm not sure this will work or make a huge
difference when it comes to timeouts in CI, but it's worth a shot.
changelog_begin
changelog_end
* TASTY_NUM_THREADS := 3
* set bazel tag for test
* daml build: add a --access-token-file for remote dependencies
This adds a `ledger.access-token-file` field in the `daml.yaml` project
file and a `--access-token-file` flag to `daml build` to authorize
querying/fetching of remote dependencies.
CHANGELOG_BEGIN
[daml build] A new flag `--access-token-file` is added for the `daml
build` command. It allows the specify the path to an access token to
authenticate against the ledger API. This is needed if the project
depends on a remote Daml package hosted on such a ledger. Alternatively,
the path to the token can also be specified in the `daml.yaml` project
file under the `ledger.access-token-file` field.
CHANGELOG_END
* damlc pkg: use cache for already present packages
This only downloads packages from a remote ledger, if the package is not
already present in the dependency db as a normal dependency.
CHANGELOG_BEGIN
CHANGELOG_END
* damlc: Allow package IDs in data-dependencies.
This is the next step outlined in issue #8976. If package id's are
present in the `data-dependency` section of the daml.yaml file, we try
to fetch them (and their transitive dependencies) from the default
ledger of the project.
CHANGELOG_BEGIN
CHANGELOG_END
We used to do this for some packages but it broke cpp. I don’t
actually know why it doesn’t do that anymore but I’ll gladly accept
that fact and turn it on everywhere.
changelog_begin
changelog_end
* feature: damlc lints all files in project directory
fixes#8887.
This changes the input options of `damlc lint` to take several files. If
no file is given, all `.daml` files contained in the directory are
linted.
CHANGELOG_BEGIN
CHANGELOG_END
* Ledger API test tool: test against legacy and preview version.
+ use dictionary instead of alias to map version keywords to LF version
CHANGELOG_BEGIN
CHANGELOG_END
improve previous generalization from #8695
- use lf version instead keyword (like 'stable', 'latest', 'dev') to
tag actual target. This will allow two keywords to map to the same
versions without doing twice the compilation/test work.
- use alias to map keywords tag target to versioned tag target.
- move package manage dar to test_commong.
CHANGELOG_BEGIN
CHANGELOG_END
* damlc: incremental package db initialization
We keep a hash over all dependencies of a project in the package
database metadata and only recompute the package database if a
dependency changes, i.e. the computed hash changes.
Fixes#4413.
Fixes#8409.
CHANGELOG_BEGIN
CHANGELOG_END
* using Fingerprint
* added tests
* format
* use SdkVersion instead of hardcoded version
* added a reference in tests
* factored out project file template
* Remove redundant list of LF versions
After #8472, I realized that there must be a list used for daml-stdlib
and daml-prim already and it turns out there is. I’ve removed that one
in favor of the one added in #8472 since I like having all in one
place and the one from #8472 is created by filtering an existing list
instead of creating a completely separate list like we do here.
changelog_begin
changelog_end
* Introduce SCRIPT_LF_VERSIONS
changelog_begin
changelog_end
The packaging test suite has become just so big that it regularly
exceeds its 15 min timeout. There are some deeper issues here, for
instance that we set up tons of tiny DAML projects and call
`daml build` on each of them. Each of those builds initializes a
package database, which is one of the things that seem to take
noticable time. Unfortunately, fixing these issues is a bigger effort.
This PR is more of a band aid rather than a proper solution: split the
packaging tests into two parts: the packaging tests and the tests for
data-depenendencies. Effectively, this amounts to giving the old
packaging test suite a 30 min time budges, but at least the two parts
can now be run in parallel.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix scenario service vs struct projections bug
In a recent change, we have sped up the projection from structural
records by means of an inference in the DAML-LF type checker. This
change was made under the assumption that users of the DAML interpreter
run through package validation before compiling to the internal Speedy
AST. Unfortunately, that assumption was wrong and not covered by our
existing tests. Manually testing of a new release candidate reminded us
of the fact that the scenario service skips package validation for the
sake of faster response times in the IDE.
This PR drops the assumption that package validation is always run.
Instead, we add the old implementation of struct projection, which works
without DAML-LF type checking, back and use it whenever the AST has not
been annotated with the information inferred by the type checker. We do
the same for struct updates.
We improve test coverage by _additionally_ running the `damlc`
integration tests without package validation for the latest stable
version of DAML-LF and DAML-LF 1.dev. These tests would have caught the
issue we only discovered during manual testing.
CHANGELOG_BEGIN
CHANGELOG_END
* Add some explanations
CHANGELOG_BEGIN
CHANGELOG_END
* Fix qualified name handling in DAML REPL
This fixes the issue reported in
https://discuss.daml.com/t/how-to-use-qualified-template-names-in-repl-queries/1329/7
We need to print names qualified otherwise, the type signature for
following lines will be ambiguous even if users qualified the name in
their input. Figuring out the right name to use in qualification is
tricky but Luckily GHC provides this already so we just have to make
sure to plug it into the right places.
changelog_begin
- [DAML REPL] Fix a bug where you got an error about a name being
ambiguous even if you used a qualified name.
changelog_end
* Update compiler/damlc/daml-compiler/src/DA/Daml/Compiler/Repl.hs
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Factor out typechecking of imports
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
To make `--import` work we also need to expose the
packages. Otherwise, we end up with a ton of confusing errors as soon
as you try anything. Luckily, we can do this fairly cheaply by using
the fact that GHC can expose both unit ids as well as just package
names. Until I realized that I started working on a different approach
of reading the DALFs first to resolve unversioned packages but that
seemed much more messy.
changelog_begin
changelog_end
We have seen timeouts on port file waits occasionally.
Those timeouts are already very high so reducing load is a more
sensible option than bumping the timeout.
changelog_begin
changelog_end
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Extend the scenario service with DAML Script support
This adds most of the infrastructure for running DAML Script via the
scenario service which means it runs as part of DAML Studio and `daml
test`. This is hidden behind a feature flag so we can land this and
parallelize the remaining tasks. The main things that are missing are:
1. `createAndExerciseCmd` and `exerciseByKeyCmd`.
2. Party management needs some work and listing parties is
unsupported.
3. Time management
4. Potentially some better error handling (we need to go through
SResult and SError and see what is relevant for us).
Overall, it is already in a very usable state and there is a decent
range of tests.
closes#3688
changelog_begin
changelog_end
* Update compiler/damlc/daml-ide-core/src/Development/IDE/Core/Rules/Daml.hs
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Fix name for actor system and pool
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* DAML REPL - Explicit package imports
changelog_begin
- [DAML REPL] The REPL no longer imports all modules from the main DALFs
of all specified DARs automatically at start-up. Instead, the REPL
will only import modules from packages specified on the command-line
using the `--import` flag.
changelog_end
* Accept package-name or package-id
* REPL test case for --import flag
* DAML REPL use `UnitId` for import packages
Addressing review comment
https://github.com/digital-asset/daml/pull/6707#discussion_r453731353
* DAML REPL Parse package-name/id at CLI
* DAML REPL Simplify unversioned pkgs
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* REPL: Factor out Dar loading
* REPL accept multiple DARs
* REPL CLI take multiple packages
CHANGELOG_BEGIN
- [DAML REPL] The REPL now supports loading multiple DARs specified at
the command line. The DARs are loaded as data-dependencies, and the
modules of each DAR's main DALF will be imported into the REPL session
at startup.
CHANGELOG_END
* REPL test loading multiple DARs
* REPL Load packages from current session
* REPL func-test test symbols from two DARs
* Update compiler/damlc/daml-compiler/src/DA/Daml/Compiler/Repl.hs
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Split damlc integration tests per DAML-LF version
Currently, we have one test for all stable DAML-LF versions. This test
takes quite a while to run and times out occasionally. This PR splits
the big test into one test per stable DAML-LF version.
CHANGELOG_BEGIN
CHANGELOG_END
* Address some feedback
* Fix haddock typo
* Daml REPL - use data dependencies
Without this a DAR that has a transitive dependency on another DAR
cannot be loaded into the REPL since the interfaces are missing, with
data-dependencies they are reconstructed.
CHANGELOG_BEGIN
- [DAML REPL] The provided DAR is now loaded as a data-dependencies,
meaning that the REPL supports loading DARs with transitive
dependencies.
CHANGELOG_END
* REPL test - use indirect DAR dependency
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* support external anchors
changelog_begin
- `daml docs` now supports an `--input-anchor` argument specifying the
path to a database of external anchors
changelog_end
* Add daml-base-anchors.json to the damlc-dist target
fixes#3150
This PR introduces a patch to GHC to fix the performance of the
pattern match checker in the presence of multiple packages which
is currently significantly (orders of magnitude) slower than having
everything in a single package. I also added a test case that hits
this. Here’s what you need to hit this issue:
1. A typeclass with a functional dependency. `HasField` is the obvious
candidate for this.
2. A lot of instances of this typeclass in a separate package (this is
the only part where the separate package matters).
3. A reasonably large ADT with a bunch of strict fields.
4. A pattern match in the context of some constraints of the
typeclass. The constraints can be completely unused.
In that case, you will get a significant slowdown in the number of
instances, number of constructors and number of constraints (didn’t
verify if it’s linear but it is significant which is all that
matters).
Here’s why this happens:
1. The pattern match checker checks for strict fields if the type is
inhabited.
2. This calls `pmTopNormaliseType_maybe` to normalize a type (the details don’t
matter) which in turn calls into the typechecker. This function is
called very often (presumably linear in the number of constructors
but didn’t verify.)
3. The typechecker has some logic in `improveFromInstEnv` for
generating additional equations by unifying functional
dependencies `a -> b` with constraints in scope
and thereby deducing information about `b`.
4. In the pattern match checker the list of instances of the home
package is empty since the pattern match checker (apparently)
doesn’t actually care about those extra equations. However, the
list of instances in the EPS is not empty. This is the issue here:
By moving it to an external package we suddenly end up with
thousands of instances that we try to unify with the functional
dependencies every time we normalize which happens very often.
Proposed fix:
The solution is rather simple: Since the pattern match checker
apparently does not care about the instances of the home package, it
almost certainly doesn’t care about instances in general so we just
empty the instances of external packages explicitly.
Is the fix correct?
1. I verified that the GHC test suite passes with this patch which
gives me a reasonable level of confidence.
2. I verified that our own test suite passes.
3. The most dodgy part is actually emptying the instance since the
whole EPS stuff is a mutable mess. What could in theory happen is that
the PM ends up loading an interface file that mutates this
again. However, afaiu it is impossible for the PM to need an
interface that the typechecker didnt already need. I did do a bunch
of debugging and this is exactly what I observed in my experiments.
Alternative ideas and upstreaming:
The other option would be to not try and mess with the EPS but somehow
have a conditional flag somewhere in the typechecker env to disable
this logic in the pattern match checker. However, that sounds
significantly more complex so I don’t think it’s worth the effort.
GHC 8.10 has a new pattern match checker that has different
performance characteristics and seems to do much better here so there
is little reason to try and upstream this. I strongly want to avoid
upgrading DAML to 8.10 at this point (too much risk, let’s wait until
things calm down)
changelog_begin
- [DAML Compiler] Fix an issue where compilation slowed down
significantly when code was split up into several packages. See
https://github.com/digital-asset/daml/issues/3150
changelog_end
* Add JSON output to `damlc inspect-dar`
fixes#3117, fixes#5563
This PR adds a `--json` flag to `damlc inspect-dar` that does pretty
much what you expect. So far it mainly exposes the same information
that you already got from `inspect-dar` but it’s easy to add more as
needed.
I’ve also finally added some documentation for `inspect-dar`.
changelog_begin
- [DAML Compiler] ``damlc inspect-dar`` now has a `--json`` flag to
produce machine-readable output. See
https://docs.daml.com/daml/reference/packages.html#inspecting-dars
for more information.
changelog_end
* Sort packages in human-readable output
changelog_begin
changelog_end
* Update docs/source/daml/reference/packages.rst
Co-authored-by: Rohan Jacob-Rao <rohanjr@gmail.com>
Co-authored-by: Rohan Jacob-Rao <rohanjr@gmail.com>
This moves the code from the Shake testing module to a separate
module. This is required since the Shake test setup is not really
intended to test multiple packages. I’ve added a new test suite that
makes use of this and has an expected failure for the issue with
multiple packages. The issue is not actually fixed in this PR, I just
wanted to keep the refactoring separate since it is quite large and
noisy.
changelog_begin
changelog_end
This is a spin off from the work for making module prefixing
work. There I wanted to be able to use the `GeneratePackageMap` rule
to get access to the modules in an LF package which requires to be in
an Action rather than in IO.
This resulted in a much larger refactoring than I expected, so here is
a list of changes to ease review:
1. I’ve split off Development/IDE/Core/RuleTypes/Daml.hs into its own
Bazel targets to avoid cycles. This allows us to use a rule without
depending on the implementation of that rule.
2. Development/IDE/Core/IdeState/Daml.hs has moved into its own Bazel
target. There is a lot of stuff that needs `withDamlIdeState` now
since the `ghcide` `initialise` function does not add our Shake rule
for getting the GHC session and making all of that depend on
damlc:lib creates a mess.
3. The actual logic of the initialization has stayed the same but
moved into a rule in Development/IDE/Core/Rules/Daml.hs.
4. As mentioned above, a few things that could previously get away
with only the rules provided by ghcide now need our own rules so
they switched to `withDamlIdeState`. This mainly affects `damlc
docs`.
5. Making daml-docs work with the ofInterestRule is a bit tricky.I did
find the issue and included the fix but also disabled the
ofInterestRule since it really does not make sense to generate Core
in the background.
changelog_begin
changelog_end
* Split up repl tests to make them faster
This PR splits up the tests into the tests for TLS and Auth and the
tests for the actual functionality.
The func tests use the repl as a library which allows them to be
significantly faster:
1. We only need to start the service process once.
2. We only need to initialize the package db once.
There is at least one other point that I did not address for now and
that is only loading the packages into the repl service once. While
loading them multiple times is a noop, it still has a performance
implication.
Sadly, this has turned out much more messy than I thought it would be
due to various issues with haskeline/repline/tasty/computers. The
details are in a comment in DA.Test.Repl.FuncTests.
changelog_begin
changelog_end
* Try to nuke cache on window selectively
* Enable cache again
* Replace unstable-types tests by a Haskell test suite
The JQ based test suite is absurdly slow, it takes around 60s to run
on my machine. The Haskell test suite on the other hand takes less
than 2s on my machine so this is a speedup of over 30x and I have to
read less bash.
changelog_begin
changelog_end
* Address review comments
* Turn warnings for module name/record name mismatches into errors
The module name warning existed for ages. We started warning about the
record name mismatch in 0.13.55 so I’d like to turn it into an error
before 1.0
changelog_begin
- [DAML Compiler] File names must now match up with module names. This
already produced a warning in previous releases.
- [DAML Compiler] It is now an error to define a record with a single
constructor where the constructor does not match the type
name. To fix the error you need to change ```data X = Y { … }``` to
```data X = X { … }```. This restriction only applies to
single-constructor records. Variants and enums are not affected.
This already produced a warning in 0.13.55.
changelog_end
* Fix integration tests
* Fix docs
* Fix lsp tests
* Make name collision check more strict
This PR extends the name collision check to catch collisions between
A:B (type B in module A) and module A.B.C. For now this is just a
warning and not an error. Once we turn it into an error, we also need
to add this to the Scala collision checker.
There is a fair bit of plumbing required to make warnings work but on
the plus side we get multiple errors at once now instead of erroring
out on the first one.
changelog_begin
- [DAML Compiler] The name collision check has been extended to also
count the case as a collision where you have a type B in module A and a module
A.B.C (but no module A.B). This is a warning in this SDK release but
will become an error in a future release. The typescript codegen is
not usable on packages that don’t uphold this restriction.
changelog_end
* Address review comments
* Integrate create-daml-app into the assistant
fixes#4868
changelog_begin
- [DAML Assistant] Add a new ``daml create-daml-app`` command for creating a project based on
`create-daml-app <https://github.com/digital-asset/create-daml-app>`_.
changelog_end
* Try random things hoping to fix windows
* Try random things hoping to fix things on macos
1.dev is changing in ways wich break this as predicted by the comment,
so let’s freeze this now that we have a release that supports this.
changelog_begin
changelog_end
* Support authentication and TLS in DAML repl
changelog_begin
- [DAML Repl - Experimental] You can now connect to a ledger via TLS
by passing ``--tls`` to ``daml repl``
- [DAML Repl - Experimental] You can now connect to a ledger with
authentication by passing the token via ``--access-token-file`` to
``daml repl``.
changelog_end
* try to fix linking on windows
* windows is weird
* gnah
