Commit Graph

83 Commits

Author SHA1 Message Date
Remy
15354c3256
DAML-LF: remove submitter is in maintainer check (#5611)
CHANGELOG_BEGIN
CHANGELOG_END
2020-04-23 16:10:39 +02:00
Remy
7bdde70840
Speedy Compiler: clean up. (#5649)
CHANGELOG_BEGIN
CHANGELOG_END
2020-04-22 16:09:54 +02:00
Moritz Kiefer
d8db88e7e1
Make DAML Triggers and DAML Script default to wall-clock-time (#5632)
* Make DAML Triggers and DAML Script default to wall-clock-time

Now that sandbox defaults to wall-clock-time there is no reason why we
should not default in DAML triggers and DAML Script.

changelog_begin

- [DAML Triggers] ``daml trigger`` now defaults to wall clock time if
  neither ``--wall-clock-time`` or ``--static-time`` is passed.

- [DAML Script] ``daml script`` now defaults to wall clock time if
  neither ``--wall-clock-time`` or ``--static-time`` is passed.

changelog_end

* Make --static-time and --wall-clock-time exclusive
2020-04-20 14:42:19 +00:00
Moritz Kiefer
7067fa432e
Add an --output-file option to DAML Script (#5590)
This PR adds an --output-file option to DAML Script that writes the
result of a DAML Script to a file and complements the --input-file option.

changelog_begin

- [DAML Script] ``daml script`` now has a `--output-file`` option that
  can be used to specify a file the result of the script should be
  written to. Similar to ``--input-file`` the result will be output in
  the DAML-LF JSON encoding.

changelog_end
2020-04-17 09:41:02 +02:00
Stephen Compall
91c43f147c
allow secure connection from JSON API to ledger (#5555)
* factor TlsConfiguration parser from extractor

* move TlsConfigurationParser to new library

* link extractor to ledger-service/cli-opts properly

* use TlsConfigurationCli in http-json, pass SslContext to ledger-client

* test TLS options as used in http-json

- the TLS config code is shared with extractor, where it is more fully
  tested; we just do a sanity check here

* doc TLS options for http-json

CHANGELOG_BEGIN
- [JSON API] New ``--pem``, ``--crt``, ``--cacrt``, and ``--tls`` options
  for securing the connection between JSON API server and ledger.
  See `issue #2540 <https://github.com/digital-asset/daml/issues/2540>`__.
CHANGELOG_END

* TLS off in daml-script JSON API test
2020-04-16 13:12:30 -04:00
Remy
9704a39bae
Engine: refactor PartialTransaction context (#5578)
* Engine: refactor PartialTransaction context

CHANGELOG_BEGIN
CHANGELOG_END
2020-04-16 14:39:12 +02:00
Moritz Kiefer
d83387a64d
Improve error messages in daml repl on calls to error (#5565)
* Improve error messages in daml repl on calls to `error`

There were two issues with calls to `error`:

1. This one is harmless but somewhat annoying: When calling `error` we
   run into the log statement in `stepToValue` which prints out the
   error message in a fairly reasonable form (you can argue whether
   Error: User abort: is a super useful prefix but that’s a relatively
   minor issue). Afterwards we then call `println` on the failed
   future. However, that will just print the type of the exception
   which isn’t something we want to show to users. I’ve just disabled
   the println statement if the exception is `SError`.

2. This one is a bigger issue: `throw x` is not the same as
   `Future.failed(x)`. I only fully realized the difference fairly
   recently. The former fails before it produces a future. So `(throw
   x).onComplete(…)` will never execute the callback. The latter is
   just a failed future. It is rather confusing to have a function
   that returns a future but then throws an exception instead of a
   future and it confuses the grpc library which prints out a horrible
   exception. I’ve changed all calls to `throw` in `runWithClients` to
   instead use `Future.failed` and `flatMap` (in the form of
   for-comprehensions).
   There are still a few calls in `run` left which I’ll leave for a
   separate PR.

I think we need to factor out some helper functions here to make this
a bit more manageable (e.g. for the Converter.toFuture) stuff but I’ll
leave that for a separate PR. You probably want to view this with
whitespace diffs disabled.

changelog_begin

- [DAML Repl] DAML Repl now produces better error messages on calls to
  `error` and `abort`.

changelog_end

* Switch stepToValue to return Either
2020-04-15 16:19:36 +00:00
Remy
d91da3be51
Engine: refactoring preprocessing. (#5434)
* Engine: refactor preprocessing

CHANGELOG_BEGIN
CHANGELOG_END
2020-04-09 20:29:03 +02:00
Leonid Shlyapnikov
29e1931f17
Add --port-file command line option to JSON API (#5454)
* Adding `--port-file` support

* ``--port-file`` support

* Updating docs

changelog_begin

[JSON API] Add support for ``--port-file`` command line option.
``--http-port 0 --port-file ./json-api.port`` will pick up a free port
and write it into ``./json-api.port` file.

changelog_end

* reformatting

* Usage grammar

* use bimap

* Adding `PortFiles` utility for creating and deleting port files on JVM exit

* Adding scaladoc explaining that the port file should be deleted on

JVM termination.

* Updating usage and docs to reflect that the file must be unique and

will be deleted on graceful shutdown

* Relying on `java.nio.file.FileAlreadyExistsException` to determine the

case when failed due to the nonunique file name.

* toString instead of Exception.getMessage

java.nio exception's getMessage can be just a file name, need the class
name to capture the error context.

* updatePortFile -> createPortFile

* write to file instead of write into file
2020-04-08 18:48:11 +00:00
Gerolf Seitz
97433743a1
Set the Bearer prefix in bindings. (#5484)
* Set the `Bearer ` prefix in bindings.
* Make the `Bearer ` prefix in the authorization header mandatory.
* Bearer prefix can be removed from the token file.

CHANGELOG_BEGIN
[Extractor]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[Navigator]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[DAML Script] The ``Bearer `` prefix can be removed from the token file. It
is added automatically.
[DAML Repl] The ``Bearer `` prefix can be removed from the token file. It is
added automatically.
[Scala Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[Java Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[DAML Integration Kit] ``AuthService`` implementations MUST read the
``Authorization`` header and the value of the header MUST start with
``Bearer ``.
CHANGELOG_END
2020-04-08 13:07:28 +02:00
Gerolf Seitz
a2d785e3ee
Use com.daml as root package (#5343)
Packages com.digitalasset.daml and com.daml have been unified under com.daml

Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.

CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
2020-04-05 19:49:57 +02:00
Moritz Kiefer
b48ef6d804
Implement party allocation in DAML script over JSON API (#5311)
Probably not super useful but easy enough to implement.

changelog_begin
changelog_end
2020-03-31 20:45:09 +02:00
Moritz Kiefer
cd7f50d6c0
Support submitMustFail when running DAML script over the JSON API (#5308)
changelog_begin
changelog_end
2020-03-31 18:33:57 +02:00
Moritz Kiefer
3ed21c26c7
Validate token parties on submit and query in DAML script (#5297)
This adds a validation step when running DAML script over the JSON API
to ensure that the party in the token matches the party that is passed
as an argument to `submit/query`.

changelog_begin
changelog_end
2020-03-31 14:16:16 +02:00
Moritz Kiefer
99e56ebe22
Only catch specific grpc errors in DAML script (#5291)
It doesn’t really make sense to catch errors like PERMISSION_DENIED
and it only make the error message more confusing and debugging
harder.

changelog_begin
changelog_end
2020-03-31 11:55:52 +02:00
Moritz Kiefer
da5daf3239
Turn script over JSON API tests into a proper test suite (#5278)
This replaces the rather horrible previous setup of having a custom
test runner that spawns 3 separate JVM processes by a single scalatest
test suite that starts sandbox and the JSON API in process.

changelog_begin
changelog_end
2020-03-31 11:55:29 +02:00
Remy
1b37f6c482
DAML-LF: redesign absolute contract ids (#5207)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 19:07:42 +01:00
Moritz Kiefer
0c5166dd8c
Support running DAML script over the JSON API (#5211)
* Support running DAML script over the JSON API

This is still in a somewhat messy state and some things don’t
work (documented in a comment) so I deliberately didn’t add this to
the documentation. However, there are tests and the PR is already
pretty large so I’d like to move the rest to separate PRs to not turn
this into more of a review nightmare than it already is.

changelog_begin
changelog_end

* Address review comments
2020-03-27 11:04:40 +00:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Andreas Herrmann
9e5dff4109
Ship daml-script and daml-trigger libraries in multiple LF versions (#5192)
* Depend on LF version specific daml-libs

* daml-script.dar build multiple LF versions

CHANGELOG_BEGIN
[DAML Script] The `daml-script` library is now available in multiple LF
  versions, namely 1.7, 1.8, and 1.dev.
CHANGELOG_END

* daml-trigger.dar build multiple LF versions

[DAML Triggers] The `daml-trigger` library is now available in multiple
  LF versions, namely 1.7, 1.8, and 1.dev.

* Keep daml-script.dar available for tests

* Keep daml-trigger.dar available for tests

* daml-libs LF versions integration test

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-03-26 10:17:19 +01:00
Robert Autenrieth
a3ddde3a9d
Remove MRT and LET (#5172)
Contributes to #4194.
Closes #4231.
Closes #5022.
CHANGELOG_BEGIN
- [Ledger API] The protobuf fields ledger_effective_time and maximum_record_time have been removed from
  command submission. These fields were previously deprecated following the introduction
  of a new ledger time model. See issue `#4194 <https://github.com/digital-asset/daml/issues/4194>`__.
[Java Bindings] removed the usage of ledgerEffectiveTime and
maximumRecordTime, and instead added minLedgerTimeAbsolute and
minLedgerTimeRelative in CommandSubmissionClient and CommandClient
CHANGELOG_END
2020-03-25 16:50:27 +01:00
Moritz Kiefer
46be910ba1
Abstract over ledger interaction in DAML script (#5184)
This PR adds as `ScriptLedgerClient` trait (happy to change the name
if anyone has a better proposal) that abstracts over the interaction
with the ledger. This will allow us to plug in a different
implementation for interacting with the JSON API so we can run DAML
scripts against DABL or other environments where gRPC is not a
workable option. Note that this PR does not yet add the implementation
for interacting with the JSON API. I’ll leave that for a separate PR.

changelog_begin
changelog_end
2020-03-25 15:26:18 +01:00
Robert Autenrieth
3f597aae16
New ledger time (#5100)
* Tighten result type

Command execution can't result in a sequencer error

* New helper method for extracting used contracts

* New error clause

* Add a DAO query for the maximum time of contracts

* Implement algorithm for finding ledger time

CHANGELOG_BEGIN
CHANGELOG_END

* fixup ledgerTimeHelper

* Use new ledger time algorithm

* Mark LET/MRT as deprecated

CHANGELOG_BEGIN
- [Ledger API] DAML ledgers have switched to a new ledger time model.
  The ledger_effective_time and maximum_record_time fields of command submission are deprecated,
  the ledger time of transactions is instead set automatically by the ledger API server.
  Ledger time is no longer strictly monotonically increasing, but only follows causal monotonicity:
  ledger time of transactions is greater than or equal to the ledger time of any used contract.
  See `#4345 <https://github.com/digital-asset/daml/issues/4345>`__.
CHANGELOG_END

* Add ledger time skew check

* Remove command updater

LET/MRT are now deprecated, this class is now useless

* Remove old time model validator

* Switch to new time model check: kvutils

* Switch to new time model check: in-memory ledger

* Switch to new time model check: SqlLedger

* Use initial ledger config

* Ignore user provided LET

* Use TimeProvider in submission services

* Use deduplication_time in daml-script runner

- Also remove unnecessary command completion output of CommandTracker.
- Remove usage of maximum record time in CommandTracker.

* Use arbitrary default value for deduplication time

* Use built-in Instant ordering

* Remove obsolete test

* Remove obsolete test: CommandStaticTimeIT

* Refactor test: TransactionMRTCompliance

* Disable test: CommandTrackerFlow timeout

* thread maxDeduplicationTime through to CommandTracker

* Improve test

* Refactor command client configuration

* Deduplication time should always use UTC

* Add missing method in TimedIndexService after rebase

* Put more details into the deduplication error response.

* Use system time for command dedup submittedAt.

* Use explicit UTC time source in command validator

* Revert CommandTracker[Flow] to previous completion-recovering-behavior

* Adapt scala client command config to new config params

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-25 09:28:56 +01:00
Moritz Kiefer
a50b7b7dbd
Hide "Slf4jlogger started" log statement (#5163)
This just looks ugly and doesn’t help users at all.

changelog_begin
changelog_end
2020-03-24 17:06:14 +01:00
Andreas Herrmann
4f7c52c981
DAML script runner takes CompiledPackages (#5110)
* daml-script runner from compiledPackages

* Factor out script parameter conversion

* Use ADT for DAML script (Action & Function)

* run --> runwithClients

* DAMl script Runner `run` convenience function

* REPL use pure script runner constructor

* DAML script runner holds script itself

* scalafmt

* comments

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-03-20 17:52:54 +00:00
Moritz Kiefer
be90649186
Support partial patterns in DAML repl (#5093)
* Support partial patterns in DAML repl

This PR improves the support for partial patterns in DAML repl by
making sure that they fail on the line itself rather than some
subsequent line and avoids the partial pattern match warnings on all
following lines.

changelog_begin
changelog_end

* Fix tests
2020-03-19 17:28:40 +00:00
Andreas Herrmann
8527ec4c4b
Infer daml-script package id from script (#5076)
* Factor out common identifier generation

For `DA.Types`, `DA.Internal.Any`, and `Daml.Script`.

* Factor out Script type for DAML scripts

* Adapt DAML script test runners

* Adapt REPL

CHANGELOG_BEGIN
CHANGELOG_END

* ./fmt.sh

* Avoid `unapply`

addressing
https://github.com/digital-asset/daml/pull/5076#discussion_r394526881

* Pure Script.fromIdentifier

* Pure Script.fromDar

* Simplify test script discovery

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-03-19 11:59:06 +00:00
Remy
c2070170b3
Engine: add rank to Enum and Variant SValue (#5048)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-18 14:37:44 +01:00
Samir Talwar
cbeeb5aafc
sandbox: Fail to start if a time mode is not explicitly specified. (#5033)
* sandbox: Fail to start if a time mode is not explicitly specified.

CHANGELOG_BEGIN
- [Sandbox] Sandbox is switching from Static Time mode to Wall Clock
  Time mode as the default. To ensure that our users know about this,
  for one version, there will be no default time mode. Instead, users
  will have to explicitly select their preferred time mode by means of
  the `--static-time` or `--wall-clock-time` switches. In the next
  release, Wall Clock Time will become the default, and users who are
  happy with the defaults will no longer need to specify the time mode.
CHANGELOG_END

* daml-script|triggers: Specify time mode when testing against Sandbox.

* daml-assistant: Default the Sandbox to wall clock time.

CHANGELOG_BEGIN
- [DAML Assistant] Initializing a new DAML project adds a switch to
  ``daml.yaml`` to ensure Sandbox can continue to start with ``daml
  start``::

      sandbox-options:
        - --wall-clock-time
CHANGELOG_END

* docs: Update the DAML Script and Triggers docs to use Wall Clock time.

It's now what Sandbox will use by default when using `daml init`.

* docs: Change the Quickstart to run Sandbox in wall clock time.

This explains why the contract IDs may vary.

It also updates the manual release testing script to match.
2020-03-18 08:25:03 +00:00
Moritz Kiefer
44d843f9ef
Support authentication and TLS in DAML repl (#4998)
* Support authentication and TLS in DAML repl

changelog_begin

- [DAML Repl - Experimental] You can now connect to a ledger via TLS
  by passing ``--tls`` to ``daml repl``

- [DAML Repl - Experimental] You can now connect to a ledger with
  authentication by passing the token via ``--access-token-file`` to
  ``daml repl``.

changelog_end

* try to fix linking on windows

* windows is weird

* gnah
2020-03-16 10:43:57 +01:00
Moritz Kiefer
b3a5c3b28d
Share test certificates (#4982)
* Share test certificates

This is primarily an attempt at making sure my contribution stats
remain negative but I think it’s a nice cleanup. The only difference
in the certs used by daml-helper which are now used everywhere is that
they use a different CN for the CA and the server. This is required to
make openssl happy (which is used by the daml-helper).

changelog_begin
changelog_end

* Fix script and trigger tests
2020-03-13 12:12:34 +01:00
Moritz Kiefer
96f62418c5
Support TLS in DAML script and DAML triggers (#4971)
This adds CLI parametrs for connecting via TLS following the scheme
used by navigator, extractor and `daml ledger`.

changelog_begin

- [DAML Script] Support TLS. Enable it by passing ``--tls``. You can
  set certificates for client authentication via ``--pem`` and
  ``-crt`` and a custom root CA for validating the server certificate
  via ``--cacrt``.

- [DAML Triggers - Experimental] Support TLS. Enable it by passing ``--tls``. You can
  set certificates for client authentication via ``--pem`` and
  ``-crt`` and a custom root CA for validating the server certificate
  via ``--cacrt``.

changelog_end
2020-03-13 09:54:44 +01:00
Moritz Kiefer
1feffdbdb1
Improve error messages when we cannot find packages in the script runner (#4884)
The logic for detecting these needs to be improved but for now this at
least gives a useful error message instead of some internal stacktrace.

changelog_begin
changelog_end
2020-03-06 18:42:57 +01:00
Moritz Kiefer
c7eb3f6b6c
Wrap Script in StateT to make evaluation order a bit less important (#4869)
* Wrap Script in StateT to make evaluation order a bit less important

This PR wraps the Script newtype in `StateT` which means that
evaluation won’t do much so `debug` behaves a bit more sensibly and
you don’t end up evaluating a script that only consists of `pure` and
`>>=` if you do not execute it.

fixes #4821

changelog_begin

- [DAML Script] Fix an issue where ``debug`` messages were output
  before the script was executed.

changelog_end

* Inline StateT and improve error messages
2020-03-06 13:42:11 +01:00
mergify[bot]
6defabe39e
Document migration from scenarios to DAML script (#4834)
* Document migration from scenarios to DAML script

changelog_begin
changelog_end

* Add a more general section on ledger initialization

* fix tests
2020-03-05 16:09:21 +00:00
Moritz Kiefer
0a6be2b341
Overload submit so that it doesn’t collide with DAML script (#4831)
This introduces a `HasSubmit` typeclass (following the naming scheme
of `HasCreate`, …) and instances for `Scenario` and `Script`. This
avoids the need to hide `submit` in every single DAML script.

changelog_begin

- [DAML Standard Library] ``submit`` and ``submitMustFail`` are now
  overloaded so that they can be used in both scenarios and DAML script.

changelog_end
2020-03-05 15:43:35 +01:00
Samir Talwar
a99156252d
libs-scala/ports: Wrap socket ports in a type, Port. (#4784)
* libs-scala/ports: Wrap socket ports in a type, `Port`.

* sandbox: Use `Port` for the API server port, and propagate.

CHANGELOG_BEGIN
CHANGELOG_END

* extractor: Use `Port` for the server port.

* ports: Make Port a compile-time class only.

* ports: Allow port 0; it can be specified by a user.

* ports: Publish to Maven Central.
2020-03-03 08:59:15 +00:00
Shayne Fletcher
66dd112960
Remove pragma 'daml 1.2' (#4702)
changelog_begin
- The pragma 'daml 1.2' is now optional.
changelog_end
2020-02-26 13:17:45 -05:00
Gerolf Seitz
d617922618
Remove InMemoryKVParticipantState (#4674)
This removes the sample/reference implementation of kvutils
InMemoryKVParticipantState.
This used to be the only implementation of kvutils, but now with the
simplified kvutils api we have ledger-on-memory and ledger-on-sql.

InMemoryKVParticipantState was also used for the ledger dump utility,
which now uses ledger-on-memory.

* Runner now supports a multi participant configuration
This change removes the "extra participants" config and goes for consistent
participant setup with --participant.

* Run all conformance tests in the repository in verbose mode.

This means we'll print stack traces on error, which should make it
easier to figure out what's going on with flaky tests on CI.

This doesn't change the default for other users of the
ledger-api-test-tool; we just add the flag for:

  - ledger-api-test-tool-on-canton
  - ledger-on-memory
  - ledger-on-sql
  - sandbox




Fixes #4225.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-26 15:45:35 +01:00
Gary Verhaegen
5a117dc358
introduce new release process (#4513)
Context
=======

After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.

This PR is a proposal for a new release process designed under the
following constraints:

- Reuse as much as possible of the existing infrastructure, to minimize
  effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
  not meant for general public consumption, but can still be used by savvy
  users without jumping through too many extra hoops (ideally just
  swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
  release status, with as few changes as possible in-between, so we can be
  confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
  releases, to avoid discovering non-transient problems while trying to
  promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
  keep the same approval process for SOC2 auditability.

The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:

```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```

This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.

Version numbers
===============

Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:

- Semantic versioning (which we chose as the version format for the SDK
  version number) allows for "prerelease" version numbers as well as
  "metadata"; an example of a complete version string would be
  `1.2.3-nightly.201+server12.43`. The "main" part of the version string
  always has to have 3 numbers separated by dots; the "prerelease"
  (after the `-` but before the `+`) and the "metadata" (after the `+`)
  parts are optional and, if present, must consist of one or more segments
  separated by dots, where a segment can be either a number or an
  alphanumeric string. In terms of ordering, metadata is irrelevant and
  any version with a prerelease string is before the corresponding "main"
  version string alone. Amongst prereleases, segments are compared in
  order with purely numeric ones compared as numbers and mixed ones
  compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
  which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
  `-`, or a transition between a number and a letter. Version strings
  are compared element-wise, with numeric segments being compared as
  numbers. Alphabetic segments are treated specially if they happen to be
  one of a handful of magic words (such as "alpha", "beta" or "snapshot"
  for example) which count as "qualifiers"; a version string with a
  qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
  which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
  special ordering amongst qualifiers. Other alphabetic segments are
  compared alphabetically and count as being "after" their prefix
  (`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
  separated by `.`, plus an optional (though deprecated) alphanumeric
  "version tag" separated by a `-`. I could not find any official
  documentation on ordering for the version tag; numeric segments are
  compared as numbers.
- npm uses semantic versioning so that is covered already.

After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:

```
0.13.53-alpha.20200214.3025.ef5d32b7
```

where the components are, respectively:

- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
  master.
- `3025`: the number of commits in master up to the release commit
  (included). Because we have a linear, append-only master branch, this
  uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
  not strictly speaking necessary, but makes it a lot more convenient to
  identify the commit.

The main downsides of this format are:

1. It is not a valid format for GHC packages. We do not publish GHC
  packages from the SDK (so far we have instead opted to release our
  Haskell code as separate packages entirely), so this should not be an
  issue. However, our SDK version currently leaks to `ghc-pkg` as the
  version string for the stdlib (and prim) packages. This PR addresses
  that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
  would see the above version number as `0.13.53.20200214.3025`, which
  should be enough to uniquely identify it. Note that, as far as I could
  find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
  it makes it more scary. However, I have been told that this may be
  long enough to cause issues on Windows by pushing us past the max path
  size limitation of that "OS". I suggest we try it and see what
  happens.

The upsides are:

- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
  they have the same date, allowing us to release same-day patches if
  needed. (Note: that is, commits that were made on the same day; the
  release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
  immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
  GHC).

Alternatives I considered:

- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
  be accepted by all schemes, but will not sort as expected under semantic
  versioning (though Maven will be fine). I have no idea how it will sort
  under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
  This is not valid semantic versioning and is therefore rejected by
  npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
  what is generally done in the Java world, but we then lose track of what
  version is actually in use and I'm concerned about bug reports. This
  would also not let us publish to the main Maven repo (at least not more
  than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
  version formats. However, it would not clearly indicate to humans that
  it is not meant as a stable version, and would sort differently under
  semantic versioning (which counts it as a prerelease, i.e. before
  `0.13.53`) than under maven (which counts it as a patch, so after
  `0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
  number of prereleases in-between `0.13.52` and the next. This is
  currently the fallback plan if Windows path length causes issues. It
  would be less convenient to map releases to commits, but it could still
  be done via querying the history of the `LATEST` file.

Release notes
=============

> Note: We have decided not to have release notes for snapshot releases.

Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.

But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.

The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.

> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.

New release process
===================

Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.

Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:

1. Before running the release step, CI will checkout the commit
  specified in the LATEST file. This requires separating the release
  step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
  that gets the version to build from an environment variable, with a
  default of `0.0.0` to remain consistent with the current `daml-head`
  behaviour.

Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.

The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.

I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-25 17:01:23 +01:00
Moritz Kiefer
8d81399c0f
Add an experimental DAML script REPL (#4660)
As mentioned in the title, this is still very experimental and needs
more work before we want to advertise it. However, the code is in a
somewhat reasonable shape, there are tests and I think even in the
current state it is already useful. Also this PR is already getting
very large so I don’t want to hold off much longer before merging this.

It is included in the SDK but hidden from `damlc --help` and `daml
--help` until the most pressing issues are addressed (primarily around
making sure that it doesn’t just shut down if you have a type error
and better error messages in general).

changelog_begin
changelog_end
2020-02-24 11:06:27 +01:00
Moritz Kiefer
2fb53497d4
Support authentication in DAML script (#4637)
changelog_begin

- [DAML Script - Experimental] Support running DAML scripts against an
authenticated ledger. The token is passed via ``--access-token-file``.

changelog_end
2020-02-20 17:52:57 +00:00
Samir Talwar
f231ebc1b3
Sandbox-Next: Re-establish static time support. (#4581)
* kvutils: Extract a committer from the uses of `SubmissionValidator`.

This makes the clock injectable too.

* kvutils: Provide logging contexts in the `Runner`.

* sandbox: Remove the `StaticAllowBackwards` time provider type.

It's not used anywhere.

* sandbox: Fix warnings in CliSpec.

* sandbox: Ensure that we cannot specify both static and wall-clock time.

* sandbox-next: Crash if wall clock time is not specified.

* sandbox-next: Document more known issues in the new Sandbox.

* sandbox: Add a Clock (and some tests) to TimeServiceBackend.

* sandbox-next: Support static time.

CHANGELOG_BEGIN
- [Sandbox Next] Re-establish static time mode.
CHANGELOG_END

* ledger-on-(memory|sql): Expect a `() => Instant`, not a `Clock`.
2020-02-18 17:03:15 +00:00
Moritz Kiefer
c64919cd58
Support --help in daml trigger/daml script/daml script-test (#4534)
fixes #4529
fixes #4530
fixes #4531

changelog_begin
changelog_end
2020-02-17 09:09:01 +01:00
Andreas Herrmann
a44e00937b
Add allocatePartyWithHint(On) to DAML Script (#4489)
* allocatePartyWithHint(On)

CHANGELOG_BEGIN
- [DAML Script - Experimental] The participant argument in ``allocatePartyOn`` is wrapped in ``ParticipantName`` to avoid confusion with the ``displayName`` argument.
- [DAML Script - Experimental] Add ``allocatePartyWithHint`` and ``allocatePartyWithHintOn`` that allow to specify the ``partyIdHint`` to the backing participant. See https://github.com/digital-asset/daml/issues/4472.
CHANGELOG_END

* test-cases for allocatePartyWithHint(On)

* DAML formatting

* Supply "" party id hint instead of None

Addressing review comment
https://github.com/digital-asset/daml/pull/4489#discussion_r378245989

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-02-12 14:17:32 +00:00
Moritz Kiefer
e7b8cdba89
Require explicit time mode for DAML script and DAML trigger (#4484)
This should provide a better migration path for people that still rely
on static time by forcing them to make this explicit. Given that both
DAML script and DAML triggers are still experimental, I’m not marking
this as a breaking change

changelog_begin

- [DAML Script - Experimental] The time mode must now always be
  specified explicitly. Use ``--static-time`` to recover the previous
  default time mode.

- [DAML Triggers - Experimental] The time mode must now always be
  specified explicitly. Use ``--static-time`` to recover the previous
  default time mode.

changelog_end
2020-02-12 11:16:25 +01:00
Gerolf Seitz
85db1033d7
Fix memory leak when using the reset service (#4447)
* sandbox: Don't hold on to old resources when resetting.

Now there's one hell of a memory leak.

CHANGELOG_BEGIN
- [Sandbox] Fixed a memory leak when using the ResetService; not
  everything was cleaned up correctly.
CHANGELOG_END

* sandbox: Split out SandboxClientResource from SandboxServerResource.

Gonna replace SandboxServerResource with a ResourceOwner acquisition.

* sandbox: Don't capture the API server in the SandboxServer resource.

When we reset, this is stored forever, leading to a memory leak.

Tested by rewriting the SandboxServerResource to use
`SandboxServer.owner`.

* sandbox: Revert the test client resource to calling `shutdownNow()`.

* sandbox: Make sure the fixture is recreated properly on each test run.

* sandbox: Make `SandboxState` a non-case class.

The `toString()` was unnecessarily heavy.

* sandbox: Futures, futures everywhere.

Avoid a race condition where the server is stopped before it starts by
storing a `Future[SandboxState]` rather than the `SandboxState` itself.

This doesn't trigger the same memory leak as storing a
`Resource[SandboxState]` because we don't capture the object itself in
the `flatMap` in the same way with `Future`.

* sandbox: Remove an unused parameter left in for debugging.

* sandbox: Replace `@VisibleForTesting` with a comment.

* sandbox: Add more comments to the weird logic in SandboxServer.

* sandbox: Get rid of the `Port` type alias; it was confusing.

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-02-10 16:47:39 +00:00
Remy
5fa173ea19
Engine: make Transaction/Node/VersionnedValue mapContractId LF private (#4364)
CHANGELOG_BEGIN
CHANGELOG_END
2020-02-05 16:25:39 +00:00
Moritz Kiefer
ef285456c9
Add sleep function to DAML script (#4410)
changelog_begin

- [DAML Script - Experimental] Add a sleep funciton that pauses
  the script for the given duration. This is primarily useful in tests
  where you repeatedly call query until a certain state is
  reached.

changelog_end

fixes #4199
2020-02-05 16:21:56 +01:00
Moritz Kiefer
072154cfaf
Expose time in DAML script (#4405)
* Expose time in DAML script

changelog_begin

- [DAML Script] Add a ``HasTime`` instance for ``Script`` which allows
  you to get the current time (UTC in wallclock mode, UNIX epoch otherwise)

changelog_end

* reenable tests

* clarify how time works

* fix tests
2020-02-05 13:47:24 +00:00