On MacOS using Java 13 (at least that was the only combination where
I’ve managed to reproduce this). Sandbox spits out errors of the
following form in ``daml start``
```
java.net.SocketException: Connection reset
```
This appears to be caused by the TCP requests that we send to detect
whether sandbox has started. Switching to using the ``--port-file``
option makes this issue go away.
fixes#4670
changelog_begin
- [DAML Assistant] In certain configurations ``daml start`` produced
a (harmless) ``Connection reset`` log message. This has now been
fixed.
changelog_end
changelog_begin
- [DAML Assistant] You can now specify options for
Sandbox/Navigator/the HTTP JSON API in ``daml.yaml`` via
``sandbox-options``/``navigator-options``/``json-api-options``. These
options will be picked up by running ``daml start``. This is
particularly useful for specifying ``--wall-clock-time`` and for
specifying a fixed ledger ID during development.
changelog_end
fixes#2993
This introduces a `HasSubmit` typeclass (following the naming scheme
of `HasCreate`, …) and instances for `Scenario` and `Script`. This
avoids the need to hide `submit` in every single DAML script.
changelog_begin
- [DAML Standard Library] ``submit`` and ``submitMustFail`` are now
overloaded so that they can be used in both scenarios and DAML script.
changelog_end
* Remove damlc migrate
``damlc migrate`` hasn’t worked for quite a while and we emitted a
warning for months so given that we don’t have plans to make it work
again in the near future, I think it does more harm than good to keep
it around.
changelog_begin
- [DAML Compiler] After being deprecated for a while the ``damlc
migrate`` command has now been removed. See
https://docs.daml.com/upgrade/ for up to date documentation
on model upgrades.
changelog_end
fixes#3704 (by removing the tests 😇)
* yeah the windows cache is once again broken \o/
* Revert "yeah the windows cache is once again broken \o/"
This reverts commit 38d7877aa4.
* Add init-script daml.yaml field
CHANGELOG_BEGIN
[DAML Script -- Experimental] You can now define an ``init-script`` in
the ``daml.yaml`` file. If present, that DAML script will be executed
to initialize the sandbox on ``daml start``. This can be used instead
of the ``scenario`` field.
CHANGELOG_END
* Add integration test for init-script
* Generate JWT token in tests
Addressing review comment
https://github.com/digital-asset/daml/pull/4685#discussion_r383835050
* Remove unnecessary daml calls
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
Context
=======
After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.
This PR is a proposal for a new release process designed under the
following constraints:
- Reuse as much as possible of the existing infrastructure, to minimize
effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
not meant for general public consumption, but can still be used by savvy
users without jumping through too many extra hoops (ideally just
swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
release status, with as few changes as possible in-between, so we can be
confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
releases, to avoid discovering non-transient problems while trying to
promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
keep the same approval process for SOC2 auditability.
The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:
```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```
This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.
Version numbers
===============
Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:
- Semantic versioning (which we chose as the version format for the SDK
version number) allows for "prerelease" version numbers as well as
"metadata"; an example of a complete version string would be
`1.2.3-nightly.201+server12.43`. The "main" part of the version string
always has to have 3 numbers separated by dots; the "prerelease"
(after the `-` but before the `+`) and the "metadata" (after the `+`)
parts are optional and, if present, must consist of one or more segments
separated by dots, where a segment can be either a number or an
alphanumeric string. In terms of ordering, metadata is irrelevant and
any version with a prerelease string is before the corresponding "main"
version string alone. Amongst prereleases, segments are compared in
order with purely numeric ones compared as numbers and mixed ones
compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
`-`, or a transition between a number and a letter. Version strings
are compared element-wise, with numeric segments being compared as
numbers. Alphabetic segments are treated specially if they happen to be
one of a handful of magic words (such as "alpha", "beta" or "snapshot"
for example) which count as "qualifiers"; a version string with a
qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
special ordering amongst qualifiers. Other alphabetic segments are
compared alphabetically and count as being "after" their prefix
(`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
separated by `.`, plus an optional (though deprecated) alphanumeric
"version tag" separated by a `-`. I could not find any official
documentation on ordering for the version tag; numeric segments are
compared as numbers.
- npm uses semantic versioning so that is covered already.
After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:
```
0.13.53-alpha.20200214.3025.ef5d32b7
```
where the components are, respectively:
- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
master.
- `3025`: the number of commits in master up to the release commit
(included). Because we have a linear, append-only master branch, this
uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
not strictly speaking necessary, but makes it a lot more convenient to
identify the commit.
The main downsides of this format are:
1. It is not a valid format for GHC packages. We do not publish GHC
packages from the SDK (so far we have instead opted to release our
Haskell code as separate packages entirely), so this should not be an
issue. However, our SDK version currently leaks to `ghc-pkg` as the
version string for the stdlib (and prim) packages. This PR addresses
that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
would see the above version number as `0.13.53.20200214.3025`, which
should be enough to uniquely identify it. Note that, as far as I could
find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
it makes it more scary. However, I have been told that this may be
long enough to cause issues on Windows by pushing us past the max path
size limitation of that "OS". I suggest we try it and see what
happens.
The upsides are:
- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
they have the same date, allowing us to release same-day patches if
needed. (Note: that is, commits that were made on the same day; the
release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
GHC).
Alternatives I considered:
- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
be accepted by all schemes, but will not sort as expected under semantic
versioning (though Maven will be fine). I have no idea how it will sort
under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
This is not valid semantic versioning and is therefore rejected by
npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
what is generally done in the Java world, but we then lose track of what
version is actually in use and I'm concerned about bug reports. This
would also not let us publish to the main Maven repo (at least not more
than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
version formats. However, it would not clearly indicate to humans that
it is not meant as a stable version, and would sort differently under
semantic versioning (which counts it as a prerelease, i.e. before
`0.13.53`) than under maven (which counts it as a patch, so after
`0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
number of prereleases in-between `0.13.52` and the next. This is
currently the fallback plan if Windows path length causes issues. It
would be less convenient to map releases to commits, but it could still
be done via querying the history of the `LATEST` file.
Release notes
=============
> Note: We have decided not to have release notes for snapshot releases.
Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.
But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.
The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.
> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.
New release process
===================
Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.
Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:
1. Before running the release step, CI will checkout the commit
specified in the LATEST file. This requires separating the release
step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
that gets the version to build from an environment variable, with a
default of `0.0.0` to remain consistent with the current `daml-head`
behaviour.
Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.
The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.
I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.
CHANGELOG_BEGIN
CHANGELOG_END
* Report assistant commands and errors via a logger.
CHANGELOG_BEGIN
CHANGELOG_END
* Hook up daml-assistant to a GCP logger.
* fix test case
* fix more tests
* Check opted in status in assistant.
* Anonimize args that have unexpected characters.
* More agressive anonimization
* add missing containers dependency
We were still using the old token format without the
https://daml.com/ledger-api key. This resulted in a warning on every
call to `daml start` which looks rather ugly. I’ve changed it to
construct the token using the JWT library instead of hardcoding it
which should make it easier to modify in the future
changelog_begin
changelog_end
* Enhanced completions for the assistant
changelog_begin
- [DAML Assistant] The assistant can now do completions for SDK
commands, e.g., ``daml ledger upl<TAB>`` will complete to ``daml
ledger upload-dar``.
changelog_end
* Apply suggestions from code review
Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>
Co-authored-by: associahedron <231829+associahedron@users.noreply.github.com>
It turns out that various environments (hello docker) start with a
closed stdin. This means that things like `daml sandbox` exit
immediately. Since users that care about the new behavior are likely
to know about it, we only enable it if a specific flag is given.
changelog_begin
- [DAML Assistant] The new behavior introduced in ``0.13.51`` to shut
down when stdin is closed is now disabled unless you explicitly
enable it by passing ``--shutdown-stdin-close``.
changelog_end
* Fall back to regular bash completions for command arguments
This addresses the first part of #4369 by falling back to the bash completions.
I sadly still haven’t figured out how we can stop to completing to a
directory if the command also exists as a directory.
changelog_begin
- [DAML Assistant] Bash and Zsh completions will now fall back to
regular file completions after the command argument.
changelog_end
* Shut up hlint
* Shut down when stdin is closed.
changelog_begin
- [DAML Assistant] The DAML assistant will now shut down long-running
processes like sandbox when stdin is
closed. This is mainly useful on Windows, where process APIs often
kill the process in a way that does not allow it to do any cleanup, in
particular, we cannot stop child processes.
changelog_end
For now, the logic for this is only in daml-helper which is the only
thing starting long-running processes (in particular sandbox). There
is a long inline-comment explaining why this is not on the assistant
itself.
fixes#4168
This test just checks that navigator starts and we get a 200
response. This is obviously not a very comprehensive test but it would
have caught the issues that prevented 0.13.44 an 0.13.45 from getting
published so it seems like a good idea to test this.
changelog_begin
changelog_end
* Add Zsh completions for the assistant
This is slightly more annoying for users since at least on Linux there
doesn’t seem to be a user-writable directory that is in `$fpath` by
default. I think on Zsh we could probably write them to
/usr/local/share/zsh/site-functions but I’d rather avoid platform
specific logic here. I would expect that Zsh users are usually
somewhat comfortable with modifying the config and this also matches
other installation instructions, e.g.,
https://github.com/zsh-users/zsh-completions#manual-installation.
On the plus side, the completions look significantly nicer in Zsh
since they include the description of commands.
CHANGELOG_BEGIN
- [DAML Assistant] Zsh completions for the DAML Assistant are now
installed as part of ``daml install``. To activate them you need to
add ``~/.daml/zsh`` to your ``$fpath``, e.g., by adding
``fpath=(~/.daml/zsh $fpath)`` to the beginning of your ``~/.zshrc``
before you call ``compinit``.
CHANGELOG_END
* Fix tests
* Update daml-assistant/src/DA/Daml/Assistant/Install/Completion.hs
Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>
Co-authored-by: associahedron <231829+associahedron@users.noreply.github.com>
* Improve bash completions for daml-assistant.
* Install bash completion script automatically
* Better default logic for bash completions
* specifically -> explicitly
* Copyright headers
* Better hook logic and refactoring
* Handle non-standard installations more robustly.
* Handle CI env & add changelog note.
CHANGELOG_BEGIN
- [DAML Assistant] Bash completions for the DAML assistant are now
available via ``daml install``. These will be installed automatically
on Linux and Mac. If you use bash and have bash completions installed,
these bash completions let you use the tab key to autocomplete
many DAML Assistant commands, such as ``daml install`` and
``daml version``.
CHANGELOG_END
* Mention bash completion in assistant docs
* Remove promises
* Start on daml test-scripts
* Run all `Script a` as test cases
* LedgerClient: Expose PackageManagementClient
To enable DAR uploads
* Upload the DAR to the ledger
* Start sandbox if no ledger specified
* Format daml test-script
* Fix deprecation warning on ActorMaterializer
* Add test-case //daml-script/tests:test_daml_script_test_runner
* Add daml test-script command
CHANGELOG_BEGIN
- [DAML Script - Experimental] Allow running DAML scripts as test-cases.
Executing ``daml test-script --dar mydar.dar`` will execute all
definitions matching the type ``Script a`` as test-cases.
See `#3687 <https://github.com/digital-asset/daml/issues/3687>`__.
CHANGELOG_END
* daml-test-script enable logging
* Remove outdated TODO comment
* daml script-test More elaborate test-caseo
Compare to expected output and add failing test-case
* daml test-script Don't abort on test-failure
Before the test runner would abort on the first failed test-case. This
occasionally introduce additional test-failures if the sandbox was
torn down half-way through execution.
* ./fmt.sh
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
As mentioned in the inline comment, this matches the behavior on the
Scala side and given that most people configure their editors to
always add trailing newlines, this is quite convenient (especially
given that the error message is horrible if you do include the
newline).
CHANGELOG_BEGIN
- [Sandbox] Restore 0.13.38 logging behaviour.
- [Navigator] Restore 0.13.38 logging behaviour.
- [Extractor] Restore 0.13.38 logging behaviour.
- [Internals] As of 0.13.39, we merged a number of internal JAR files in
the SDK tarball to reduce its size. These jars used to be standalone
JARs you could invoke as e.g. ``java -jar sandbox.jar <args>``. As a
result of merging the jars, they lost their individual ``logback.xml``
configuration file. Although running the jars directly was (and is
still) not supported, note that you can now achieve the same behaviour
with e.g. ``java -Dlogback.configurationFile=sandbox-logback.xml -jar
daml-sdk.jar sandbox <args>``.
CHANGELOG_END
* daml assistant expected auth token in Bearer format
* Daml assistant does no validation of the auth token before passing in on to the ledger.
* clarify code with newtype Token
This is a first step towards making sure that the package ids for
types defined in daml-prim and daml-stdlib don’t change. This PR
mostly adds all the necessary infrastructure for that and moves
GHC.Types and GHC.Prim to make sure it works.
Until data-dependencies are really solid and we have verified that we
no longer have performance issues with an increasing number of Haskell
packages, we still include the source files in daml-prim and then just
rewrite the references.
We will also need to add tests that these packages really have stable
ids but I’ll leave that for separate PRs since this doesn’t make that
much sense anyway until all of the types have moved to stable
packages.
CHANGELOG_BEGIN
- [DAML Compiler] The modules GHC.Prim and GHC.Types from daml-prim
have been moved to separate packages.
CHANGELOG_END
This makes sure that navigator picks up things such as
`frontend-config.js` which is needed for custom tabs.
To get there, this PR does two things:
1. Change navigator such that it prefers a config file specified
explicitly via -c over the SDK config file.
2. Change daml-helper to pass in the config file via -c instead of
changing the directory
CHANGELOG_BEGIN
- [DAML Assistant] ``daml ledger navigator`` now loads
``frontend-config.js`` properly.
- [Navigator] Explicit config files passed via ``-c`` are preferred
over ``daml.yaml``.
CHANGELOG_END
* Error handling improvements,
partitioning the Soruce[Error \/ JsValue], folding all errors and
producing the final status and errors element based on the above fold
* Test case + fixes,
for some reason GeneratorDrivenPropertyChecks does not like AsyncFreeSpec
* changing the order of elements in the response,
would be better if we were checking the JSON objects not JSON strings.
* Start moving all the jars into a single jar.
* Fold navigator into daml-sdk jar
* include sandbox
* Remove unnecessary compileDeps.
CHANGELOG_BEGIN
- [DAML SDK] Reduced the size of the DAML SDK by about
60% uncompressed, 70% compressed, by deduplicating Scala
dependencies.
CHANGELOG_END
* update copyright header
* buildifier fix
This reenables the migration checks. The migration command now only
generates stumbs for conversions instead of using the formerly (generic)
upgrade template.
* Add authentication to Java identity client
Contributes to #3626
No changelog added right now, it will be added with the PR that
effectively closes the ticket.
Moves `LedgerCallCredentials` to a shared package.
Establishes the test framework for other Java clients.
* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731
* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731
* Rebase and fix compilation errors
* Fix type of jar in artifacts.yaml
Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
* Fix more errors emerged after rebase
* Add dependency for DAML assistant ITs
* Add missing dependency
* Fix wrong dependency
* fix release
We don't drop full support from the compiler yet but rather ban their use by
adding a check to the preprocessor. We'll remove the actual support as we go
along with fixing the upgrading story.
CHANGELOG_BEGIN
- [DAML Compiler] Make the experimental feature "generic templates"
unavailable. The current implementation is at odds with other, more important
language features still under development.
CHANGELOG_END
* support --access-token-file in SDK
* rename: HostAndPort -> LedgerArgs; HostAndPortFlags -> LedgerFlags
* integration test for --access-token-file
* doc & more help
* address comments: use FilePath
* adapt to renamed arg: --auth-jwt-hs256-unsafe
* avoid use of single quotes on command line
* Set Sdk-Version in DAR Manifest to compiler SDK version.
CHANGELOG_BEGIN
- [DAML Compiler] Bugfix: The Sdk-Version field in a DAR manifest file
now matches the SDK version of the compiler, not the sdk-version field
from daml.yaml. These are usually the same, but they could be different
if you set the DAML_SDK_VERSION environment variable before running
``daml init`` or ``daml build``.
CHANGELOG_END
* Fallback to daml.yaml if env var not set
* add yaml dependency
* Always require sdk-version, and emit warning on mismatch with env var
* More explicit about where override comes from
* Add packaging regression test
* Move packaging tests to a separate test suite
Originally, we had these as part of the integration test suite since
`bazel run damlc build` couldn’t locate `ghc-pkg` but that has been
fixed for a while.
Moving it to a separate test suite speeds things up as the integration
tests are rerun very often and also makes development much more
convenient since the new test suite supports `-p` properly to filter
to specific tests.
For now, I’ve left the upgrading tests as part of the integration
tests. I expect that we probably want to factor those out to another
test suite as well.
* fix warnings
* language: fix for tuple types in source generation
We correct the source generation from dalfs containing TupleN data
types. A test is added to the integration tests of daml assistant and
the `generate-src` command is improved to make this test possible.
* check that tuple type comes from DA.Types
* check that compilation succeeded
* pattern match golf
* check for tuples between 2 and 20
This change allows to only import `module A` from the instances package
if data dependencies are present instead of `module A` and `module
AInstances`.
* language: introduce data-imports
Right now the user experience for importing dalfs and dars from
different sdks is quiet confusing. This PR tries to solve this. We add
an additional field `data-imports` to daml.yaml. These imports can come
from different SDK's and we will generate interface files containing the
data types and their Template instances.
This also simplifies the migration command, as it now always imports the
respective packages as `data-imports`.
* language: cross sdk dalf/dar imports
The final piece for cross sdk imports. With this PR we can import the
data types of packages and dalfs that were created with different sdks.
This is done by generating interface files from dalfs and an 'instances'
package that contains the template instance definitions of template data
types. The instances itself are defined via the `external` keyword,
which is inlined to proper daml-lf instance definitions given in the
respective dalf package.
We test that cross sdk imports work by importing the `simple-dalf` in
the daml-assistant integation tests and running a scenario.
Currently we hide modules, for which we don't define an alias, when we
specify one alias for *one* module. With this change, all modules
without aliases are still exposed.
* language: refactoring of iface file generation and package db setup.
This is a refactoring of the damlc part that creates the package
database and the code generation for interface files. This is a
preparation for the cross sdk imports.
We also add an internal command to damlc to generate generic instances
code and use simple copying via {..} in the migration command. An
additional test checks that migration via generics still works.
* Update compiler/damlc/daml-compiler/src/DA/Daml/Compiler/Upgrade.hs
Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* clearer description for generate-gen-src
* updated documentation
* correct copy/pasta mistake
* added a comment on different build options in migration command.
* Update compiler/damlc/lib/DA/Cli/Damlc.hs
Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* resolve dalf paths from dar manifest
* added a comment on different headers in upgrade modules.
* removed monoid instance for ExtractedDar
