* Remove damlc migrate
``damlc migrate`` hasn’t worked for quite a while and we emitted a
warning for months so given that we don’t have plans to make it work
again in the near future, I think it does more harm than good to keep
it around.
changelog_begin
- [DAML Compiler] After being deprecated for a while the ``damlc
migrate`` command has now been removed. See
https://docs.daml.com/upgrade/ for up to date documentation
on model upgrades.
changelog_end
fixes#3704 (by removing the tests 😇)
* yeah the windows cache is once again broken \o/
* Revert "yeah the windows cache is once again broken \o/"
This reverts commit 38d7877aa4.
* Make `debug` a bit more lazy
Previously `debug x >>= f` would print the debug statement when it got
evaluated rather than when the monadic execution got executed. This is
rather confusing. Now we explicitly make it lazy by wrapping the trace
statement in a lambda passed to >>=.
changelog_begin
- [DAML Standard Library] Fix a bug where ``debug`` printed the trace
statement before the action got executed. Note that this means that
``debug`` now has a slightly more restrictive type. You can use
``trace`` directly if this causes problems.
changelog_end
* Update compiler/damlc/tests/src/DA/Test/ShakeIdeClient.hs
Co-Authored-By: Martin Huschenbett <martin.huschenbett@posteo.me>
Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
Given that these two group of tests run for the same combination of LF
versions, I don’t see any reason why we should keep them
separate. Merging them, speeds up the packaging tests by > 50s on my
local machine.
changelog_begin
changelog_end
Rather than decoding everything twice, we can only rewrite references
which is measurably faster (> 5s out of 60s on the packaging tests).
changelog_begin
changelog_end
* Freeze DAML-LF 1.8
Two minor points that I did not mention in the previous PR:
We also include the renaming of structural records to `struct` and the
renaming of `Map` to `TextMap`.
There are some minor changes around the LF encoder tests which need to
be able to emit package metadata properly so I’ve added it to the
parser. Sorry for not splitting that out.
Following the process used for the DAML-LF 1.7 release, this does not
yet include the frozen proto file.
changelog_begin
- [DAML-LF] Release DAML-LF 1.8:
* Rename structural records to ``Struct``. Note that
structural records are not exposed in DAML.
* Rename ``Map`` to ``TextMap``.
* Add type synonyms. Note that type synonyms are not serializable.
* Add package metadata, i.e., package names and versions.
Note that the default output of ``damlc`` is stil DAML-LF 1.7. You
can produce DAML-LF 1.8 by passing ``--target=1.8``.
changelog_end
* Update encoder
* Update java codegen tests
* Update comment in scala codegen
* Handle TSynApp in interface reader
* Bump lf_stable_version to 1.7
* Fix kvutils tests
I’m not really happy with this “fix” but after having spend way too
much time on this, this was the best I came up with. (The details are
in an inline comment). If anyone has better ideas, I’m all ears.
changelog_begin
changelog_end
Replace `daml-lf-repl validate` in packaging tests with `damlc validate-dar`.
Simplify test setup a little by passing tools (damlc,validate, etc) in a record.
changelog_begin
changelog_end
This reduces the number of GHCs to 2 on Linux (regular and DWARF) and
1 on macOS. Given that each derivation is > 1 GB this should hopefully
help a bit.
changelog_begin
changelog_end
Previously, we mapped `dependencies` under
Pkg_$pkgId.originalmodule name and imported them this way. However, we
did not map `dependencies` the same way. This PR unifies the two and
cleans up the import handling logic a bit.
This also fixes imports if we have two packages with the same name but
a different version since the package name (which is the only thing
usable in package-qualified imports) is not sufficient to
disambiguate. I’ve added a test for this.
changelog_begin
changelog_end
These commands were intended for debugging but neither @associahedron
nor I actually use them since running `daml build` and looking at the
generated files in `.daml` is a much more robust solution.
I’ve also deleted some leftover code from the old-style
data-dependencise where we generated actual template instances (not
just dummy instances). We’ve already deleted everything else around
this, this was just leftover by accident.
The only usage was a testcase which I’ve just switched over to using
`daml build`.
changelog_begin
changelog_end
Context
=======
After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.
This PR is a proposal for a new release process designed under the
following constraints:
- Reuse as much as possible of the existing infrastructure, to minimize
effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
not meant for general public consumption, but can still be used by savvy
users without jumping through too many extra hoops (ideally just
swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
release status, with as few changes as possible in-between, so we can be
confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
releases, to avoid discovering non-transient problems while trying to
promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
keep the same approval process for SOC2 auditability.
The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:
```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```
This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.
Version numbers
===============
Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:
- Semantic versioning (which we chose as the version format for the SDK
version number) allows for "prerelease" version numbers as well as
"metadata"; an example of a complete version string would be
`1.2.3-nightly.201+server12.43`. The "main" part of the version string
always has to have 3 numbers separated by dots; the "prerelease"
(after the `-` but before the `+`) and the "metadata" (after the `+`)
parts are optional and, if present, must consist of one or more segments
separated by dots, where a segment can be either a number or an
alphanumeric string. In terms of ordering, metadata is irrelevant and
any version with a prerelease string is before the corresponding "main"
version string alone. Amongst prereleases, segments are compared in
order with purely numeric ones compared as numbers and mixed ones
compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
`-`, or a transition between a number and a letter. Version strings
are compared element-wise, with numeric segments being compared as
numbers. Alphabetic segments are treated specially if they happen to be
one of a handful of magic words (such as "alpha", "beta" or "snapshot"
for example) which count as "qualifiers"; a version string with a
qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
special ordering amongst qualifiers. Other alphabetic segments are
compared alphabetically and count as being "after" their prefix
(`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
separated by `.`, plus an optional (though deprecated) alphanumeric
"version tag" separated by a `-`. I could not find any official
documentation on ordering for the version tag; numeric segments are
compared as numbers.
- npm uses semantic versioning so that is covered already.
After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:
```
0.13.53-alpha.20200214.3025.ef5d32b7
```
where the components are, respectively:
- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
master.
- `3025`: the number of commits in master up to the release commit
(included). Because we have a linear, append-only master branch, this
uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
not strictly speaking necessary, but makes it a lot more convenient to
identify the commit.
The main downsides of this format are:
1. It is not a valid format for GHC packages. We do not publish GHC
packages from the SDK (so far we have instead opted to release our
Haskell code as separate packages entirely), so this should not be an
issue. However, our SDK version currently leaks to `ghc-pkg` as the
version string for the stdlib (and prim) packages. This PR addresses
that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
would see the above version number as `0.13.53.20200214.3025`, which
should be enough to uniquely identify it. Note that, as far as I could
find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
it makes it more scary. However, I have been told that this may be
long enough to cause issues on Windows by pushing us past the max path
size limitation of that "OS". I suggest we try it and see what
happens.
The upsides are:
- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
they have the same date, allowing us to release same-day patches if
needed. (Note: that is, commits that were made on the same day; the
release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
GHC).
Alternatives I considered:
- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
be accepted by all schemes, but will not sort as expected under semantic
versioning (though Maven will be fine). I have no idea how it will sort
under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
This is not valid semantic versioning and is therefore rejected by
npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
what is generally done in the Java world, but we then lose track of what
version is actually in use and I'm concerned about bug reports. This
would also not let us publish to the main Maven repo (at least not more
than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
version formats. However, it would not clearly indicate to humans that
it is not meant as a stable version, and would sort differently under
semantic versioning (which counts it as a prerelease, i.e. before
`0.13.53`) than under maven (which counts it as a patch, so after
`0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
number of prereleases in-between `0.13.52` and the next. This is
currently the fallback plan if Windows path length causes issues. It
would be less convenient to map releases to commits, but it could still
be done via querying the history of the `LATEST` file.
Release notes
=============
> Note: We have decided not to have release notes for snapshot releases.
Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.
But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.
The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.
> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.
New release process
===================
Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.
Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:
1. Before running the release step, CI will checkout the commit
specified in the LATEST file. This requires separating the release
step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
that gets the version to build from an environment variable, with a
default of `0.0.0` to remain consistent with the current `daml-head`
behaviour.
Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.
The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.
I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.
CHANGELOG_BEGIN
CHANGELOG_END
* Add documentation for DAML repl and advertise it
This PR adds some simple docs for ``daml repl`` and adds it to the
release notes.
changelog_begin
- [DAML Repl - Experimental] A new ``daml repl`` command that allows
you to use the ``DAML Script`` API interactively. Take a look at the
`documentation <https://docs.daml.com/daml-repl/>`_ for more
information.
changelog_end
* Update docs/source/daml-repl/index.rst
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* s/Repl/REPL/
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Use package metadata instead of file names to infer unit ids
This PR adds a function that abstracts over whether we get metadata
from a filename (< 1.dev) or directly from the LF metadata.
There is more work to be done here, in particular, I want to clean up
the hacks around daml-prim/daml-stdlib but I’ll leave that for a
separate PR.
changelog_begin
changelog_end
* Update compiler/daml-lf-ast/src/DA/Daml/LF/Ast/Util.hs
Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>
* Refactor getUnitId
Co-authored-by: associahedron <231829+associahedron@users.noreply.github.com>
* Graceful error handling in `daml repl`
This PR changes `daml repl` to handle errors (parse errors, type
errors, unsupported statement errors, script errors) gracefully
and just emit an error message instead of tearing down the whole
process.
This gets the repl into a state where I think it’s sufficiently
user-friendly to be released (obviously there are tons of potential
improvements). The only thing missing before I’m comfortable
mentioning this in release notes and uninternalizing it are docs.
If you think there is something crucial that needs to be addressed
before, let me know.
changelog_begin
changelog_end
* why is windows
As mentioned in the title, this is still very experimental and needs
more work before we want to advertise it. However, the code is in a
somewhat reasonable shape, there are tests and I think even in the
current state it is already useful. Also this PR is already getting
very large so I don’t want to hold off much longer before merging this.
It is included in the SDK but hidden from `damlc --help` and `daml
--help` until the most pressing issues are addressed (primarily around
making sure that it doesn’t just shut down if you have a type error
and better error messages in general).
changelog_begin
changelog_end
This PR adds the necessary infrastructure to produce package metadata
in `damlc`.
For `damlc build` this works exactly as expected. There are a few edge
cases where we don’t have names and/or versions (namely scenarios,
damlc compile and damlc package). We don’t really care about the
metadata for those anyway, so I’ve just set it some default value.
changelog_begin
changelog_end
* Refactor handling of package names and versions
This is a preparatory refactoring PR in preparation for propagating
package metadata into DAML-LF. There are no actual changes in here.
Primarily the changes consist of 3 things:
1. In options, we split the `optMbPackageName` field which previously
contained the unit id into `optMbPackageName` and
`optMbPackageVersion`.
2. We use newtypes for names and versions and try to keep them pretty
much everywhere (the only place missing is `splitUnitId`, I’ll do that
separately).
3. We use `UnitId` where we want `name-version`.
As was probably to be expected, this surfaced some minor issues. They
are pretty much exclusively in debugging or “internal” commands so
I’ve mostly just added notes/todos.
changelog_begin
changelog_end
* cry about applicativedo
`addDays d (- r)` looks a bit confusing and it’s not obvious that
negative numbers even work.
changelog_begin
- [DAML Standard Library] Add `subtractDays` to the DAML Standard Library.
changelog_end
* Add package metadata to DAML-LF proto and the Haskell AST
This adds package metadata (currently only the package name and
version) to DAML-LF and the corresponding Haskell ASTs. This is useful
for debugging and “codegens” (typescript, damlc dependencies, …)
This PR does not yet add it to the Scala side or change the compiler
to actually produce this metadata.
Part of #4412
changelog_begin
changelog_end
* Address review comments
* Add type-level strings in DAML.
This PR adds a `PromotedText` stable package, with `PromotedText` type, which is used to encode type-level strings from DAML into DAML-LF. The reason for this is to preserve the `HasField` instance argument. This PR adds a test that `HasField` is succesfully reconstructed incontexts, during data-dependencies, which wasn't possible before.
changelog_begin
changelog_end
* adresss comments
* fix overly specific tests
* Make DAML's (>>) operator lazy in its second argument
Currently, when you write
```haskell
do
Left "wanted"
error "unwanted"
```
your computation fails with an exception coming from the `error "unwanted"`
call. However, `do`-notation suggests that you actually never get there but
rather bail out at the `Left "wanted"` line. The cause of this mismatch is
that GHC desugars the code above to
```haskell
Left "wanted" >> error "unwanted"
```
and since DAML is strict, we evaluate `error "unwanted"` _before_ evaluating
the application of `(>>)`.
This PR solves the problem by rewriting all expressions of the form `A >> B`,
and hence those of the form `do A; B` as well, to `A >>= \_ -> B`. This gives
the desired semantics with `(>>)` being lazy in its second argument.
However, these semantics only make sense for `Action`s (aka monads). Thus, we
need to restrict the constraint on `(>>)` from `Applicative` to `Action`.
(This is in line with what Haskell does.)
Moreover, the `Action` instance of the `Validation` applicative would lead
to undesriable behavior when using `do`-notation. Thus, we also drop this
`Action` instance and hence force the usage of the `ApplicativeDo` language
extension when using `do`-notation for `Validation`.
CHANGELOG_BEGIN
- [DAML Stdlib] Restrict the ``(>>)`` operator to instances of ``Action``
and make it lazy in its second argument. This gives expressions of the
form ``do A; B`` the desirable semantics of only running ``B`` when
``A`` is a successful action.
- [DAML Stdlib] Remove the ``Action`` and ``ActionFail`` instances for
``Validation`` in ``DA.Validation``. Please enable the ``ApplicativeDo``
language extension if you want to use ``Validation`` with ``do``-notation
and replace ``fail`` with ``DA.Validation.invalid``.
CHANGELOG_END
* Address @cocreature's remarks
CHANGELOG_BEGIN
CHANGELOG_END
We forgot to import the module that defines the duplicate instance
which causes problems if this instance is required for something else,
e.g. because it is a superclass of another instance.
This fixes the issue surfaced by the removal of the Action instance
for Validation in #4552.
changelog_begin
changelog_end
Shouldn’t really make a difference due to laziness but at least it
makes it explicit if we need to decode the archive to the AST or just
need to get the package id.
changelog_begin
changelog_end
On a simple quickstart project we spend about 0.7s in
createProjectPackageDb (all of it in the `withDamlIdeState`
part). This slows down our packaging tests significantly so for now
I’ve added some shortcircuiting if we don’t have any deps and
data-deps. We might want to find a nicer solution here in the
future but the whole code for setting up the package db is going to
move a around a fair bit so I don’t think it makes sense to do this
right now.
changelog_begin
changelog_end
If we get interrupted after updating our local view but before doing
the actual context update on the server side bad things happen. To fix
this, we first clone the context then update the clone which avoids
this and is a bit nicer than throwing an `uninterruptibleMask` around
updating the local view and sending the updateCtx gRPC request.
fixes#4525
changelog_begin
changelog_end
* Fix cross-sdk typeclass reexports in data-dependencies
Sorry for the somewhat large PR, the intermediate states sadly just
change error messages but don’t allow me to test the things they
enable (at least not without considerable effort).
There are a few things this PR fixes:
1. We did not properly expose `dependencies` while compiling
`data-dependencies`. Now they are mapped under a Pkg_$pkgId prefix.
2. We did not pass on `dependencies` to the LF typechecker and
various other places that tried to look up LF packages.
3. We did not filter out duplicate instances, e.g., `daml-prim`
defines `Eq` instances for `Either`. If we define that twice, we
will get an error.
4. The use of Generic in DA.Upgrade doesn’t work
(https://github.com/digital-asset/daml/issues/4470). For now,
I’ve added a somewhat crude hack to filter out typeclasses of a
given name. We should replace this by checking for `Erased`
but that is not yet used in 0.13.51 which I use in the test.
For the test, I’ve checked in a DAR built on 0.13.51 with DAML-LF
1.dev. Not sure if checking in DARs is a reasonable thing to do in the
long term but for now, this is small enough and let’s us move forward
with this.
changelog_begin
changelog_end
* Document envDepInstances
* Document TUnit match
* Update docs for stub binds
* Cleanup and document class rewrites
* Report assistant commands and errors via a logger.
CHANGELOG_BEGIN
CHANGELOG_END
* Hook up daml-assistant to a GCP logger.
* fix test case
* fix more tests
* Check opted in status in assistant.
* Anonimize args that have unexpected characters.
* More agressive anonimization
* add missing containers dependency
changelog_begin
- [DAML Standard Library] Add a ``subtract`` function which is useful
as a replacement for sections of ``(-)``, e.g., ``subtract 1`` is
equivalent to ``\x -> x - 1``.
changelog_end
This is a spin off from my fixes for making data-deps work with
typeclasses cross-SDK.
We only have the interface files for the main dalf so it doesn’t
really make sense to extract the other dalfs. The current behavior of
extracting all dalfs results in them being picked up by
`GeneratePackageMap` even if GHC doesn’t know about them which causes
issues in other placse.
I’ve adapted the collision check to check for transitive dependencies
when creating the project db.
changelog_begin
changelog_end
The string needs to include the full flag instead of only including
--package. Otherwise you get rather unhelpful error messages.
changelog_begin
changelog_end
* Handle multiple constraints on an instance in data-dependencies
Previously we translated `instance (Foo a, Foo b) => Foo (a, b) where`
into `instance Foo a => Foo b => Foo (a, b)` which is a syntax
error. We already did this correctly for `HasField` so this PR mostly
just shuffles things around to always use the non-HasField specific
parts.
changelog_begin
changelog_end
* Prefix fields of DFunHead wtih dfh
* Translate unsupported kinds to a special Erased type
This should simplify `data-dependencies` and avoid issues like #4470
since we can match on the type instead of having to guess which types
can and which cannot be translated back to DAML.
changelog_begin
changelog_end
* Switch to HashSet/HashMap for NormalizedUri/NormalizedFilePath
This matches the change to ghcide in
https://github.com/digital-asset/ghcide/pull/420. Now that we have
optimized Hashable instances it makes sense to use them as much as
possible.
changelog_begin
changelog_end
* debug windows crap
* Fix a bug in vr scenario notifications
* Revert "debug windows crap"
This reverts commit f58fdb92c1.
* attempt to fix windows
who are we kidding, this is not going to work
