Commit Graph

175 Commits

Author SHA1 Message Date
Gary Verhaegen
caf07566d2
add commands to extract relevant changelogs (#4905)
In the current state of the release instructions, the person in charge
of the release has to figure out how to produce the changelog. This PR
adds more specific (and hopefully simpler) instructions for producing
relevant changelogs.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-10 12:45:31 +01:00
Moritz Kiefer
c4fa115b67
Remove experimental note about DAML script from daml --help (#4917)
changelog_begin
changelog_end
2020-03-10 11:26:46 +00:00
Gerolf Seitz
72cafaef70
Update release testing to include running daml script (#4915)
Also the line numbers are updated.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-10 08:49:47 +01:00
Gary Verhaegen
8e217d219e
amend release process notes (#4871)
Based on running through the new process twice now, I wanted to try and
clarify some of the steps.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-09 12:50:18 +01:00
mergify[bot]
cb1395e923
Remove damlc migrate (#4816)
* Remove damlc migrate

``damlc migrate`` hasn’t worked for quite a while and we emitted a
warning for months so given that we don’t have plans to make it work
again in the near future, I think it does more harm than good to keep
it around.

changelog_begin

- [DAML Compiler] After being deprecated for a while the ``damlc
  migrate`` command has now been removed. See
  https://docs.daml.com/upgrade/ for up to date documentation
  on model upgrades.

changelog_end

fixes #3704 (by removing the tests 😇)

* yeah the windows cache is once again broken \o/

* Revert "yeah the windows cache is once again broken \o/"

This reverts commit 38d7877aa4.
2020-03-04 20:36:48 +00:00
Moritz Kiefer
ad0336d3a5
Freeze DAML-LF 1.8 proto (#4788)
* Freeze DAML-LF 1.8 proto

This is a copy of the 1.dev proto with the following changes that
overall remove all references to 1.dev:

* Change the versions and the reference to the spec.
* Remove `dev` from version history at the top.
* Remove `GENMAP` and corresponding primitives.
* Remove general equality `EQUAL` and the comments that the
  type-specific equality primitives are only available in < 1.dev
* Remove experimental text primitives.

changelog_begin
changelog_end

* windows is very bad

* bump windows hashes
2020-03-03 14:21:00 +01:00
Samir Talwar
a99156252d
libs-scala/ports: Wrap socket ports in a type, Port. (#4784)
* libs-scala/ports: Wrap socket ports in a type, `Port`.

* sandbox: Use `Port` for the API server port, and propagate.

CHANGELOG_BEGIN
CHANGELOG_END

* extractor: Use `Port` for the server port.

* ports: Make Port a compile-time class only.

* ports: Allow port 0; it can be specified by a user.

* ports: Publish to Maven Central.
2020-03-03 08:59:15 +00:00
Gary Verhaegen
5a117dc358
introduce new release process (#4513)
Context
=======

After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.

This PR is a proposal for a new release process designed under the
following constraints:

- Reuse as much as possible of the existing infrastructure, to minimize
  effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
  not meant for general public consumption, but can still be used by savvy
  users without jumping through too many extra hoops (ideally just
  swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
  release status, with as few changes as possible in-between, so we can be
  confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
  releases, to avoid discovering non-transient problems while trying to
  promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
  keep the same approval process for SOC2 auditability.

The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:

```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```

This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.

Version numbers
===============

Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:

- Semantic versioning (which we chose as the version format for the SDK
  version number) allows for "prerelease" version numbers as well as
  "metadata"; an example of a complete version string would be
  `1.2.3-nightly.201+server12.43`. The "main" part of the version string
  always has to have 3 numbers separated by dots; the "prerelease"
  (after the `-` but before the `+`) and the "metadata" (after the `+`)
  parts are optional and, if present, must consist of one or more segments
  separated by dots, where a segment can be either a number or an
  alphanumeric string. In terms of ordering, metadata is irrelevant and
  any version with a prerelease string is before the corresponding "main"
  version string alone. Amongst prereleases, segments are compared in
  order with purely numeric ones compared as numbers and mixed ones
  compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
  which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
  `-`, or a transition between a number and a letter. Version strings
  are compared element-wise, with numeric segments being compared as
  numbers. Alphabetic segments are treated specially if they happen to be
  one of a handful of magic words (such as "alpha", "beta" or "snapshot"
  for example) which count as "qualifiers"; a version string with a
  qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
  which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
  special ordering amongst qualifiers. Other alphabetic segments are
  compared alphabetically and count as being "after" their prefix
  (`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
  separated by `.`, plus an optional (though deprecated) alphanumeric
  "version tag" separated by a `-`. I could not find any official
  documentation on ordering for the version tag; numeric segments are
  compared as numbers.
- npm uses semantic versioning so that is covered already.

After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:

```
0.13.53-alpha.20200214.3025.ef5d32b7
```

where the components are, respectively:

- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
  master.
- `3025`: the number of commits in master up to the release commit
  (included). Because we have a linear, append-only master branch, this
  uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
  not strictly speaking necessary, but makes it a lot more convenient to
  identify the commit.

The main downsides of this format are:

1. It is not a valid format for GHC packages. We do not publish GHC
  packages from the SDK (so far we have instead opted to release our
  Haskell code as separate packages entirely), so this should not be an
  issue. However, our SDK version currently leaks to `ghc-pkg` as the
  version string for the stdlib (and prim) packages. This PR addresses
  that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
  would see the above version number as `0.13.53.20200214.3025`, which
  should be enough to uniquely identify it. Note that, as far as I could
  find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
  it makes it more scary. However, I have been told that this may be
  long enough to cause issues on Windows by pushing us past the max path
  size limitation of that "OS". I suggest we try it and see what
  happens.

The upsides are:

- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
  they have the same date, allowing us to release same-day patches if
  needed. (Note: that is, commits that were made on the same day; the
  release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
  immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
  GHC).

Alternatives I considered:

- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
  be accepted by all schemes, but will not sort as expected under semantic
  versioning (though Maven will be fine). I have no idea how it will sort
  under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
  This is not valid semantic versioning and is therefore rejected by
  npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
  what is generally done in the Java world, but we then lose track of what
  version is actually in use and I'm concerned about bug reports. This
  would also not let us publish to the main Maven repo (at least not more
  than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
  version formats. However, it would not clearly indicate to humans that
  it is not meant as a stable version, and would sort differently under
  semantic versioning (which counts it as a prerelease, i.e. before
  `0.13.53`) than under maven (which counts it as a patch, so after
  `0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
  number of prereleases in-between `0.13.52` and the next. This is
  currently the fallback plan if Windows path length causes issues. It
  would be less convenient to map releases to commits, but it could still
  be done via querying the history of the `LATEST` file.

Release notes
=============

> Note: We have decided not to have release notes for snapshot releases.

Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.

But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.

The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.

> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.

New release process
===================

Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.

Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:

1. Before running the release step, CI will checkout the commit
  specified in the LATEST file. This requires separating the release
  step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
  that gets the version to build from an environment variable, with a
  default of `0.0.0` to remain consistent with the current `daml-head`
  behaviour.

Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.

The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.

I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-25 17:01:23 +01:00
Moritz Kiefer
38b7e65197
Add documentation for DAML repl and advertise it (#4678)
* Add documentation for DAML repl and advertise it

This PR adds some simple docs for ``daml repl`` and adds it to the
release notes.

changelog_begin

- [DAML Repl - Experimental] A new ``daml repl`` command that allows
  you to use the ``DAML Script`` API interactively. Take a look at the
  `documentation <https://docs.daml.com/daml-repl/>`_ for more
  information.

changelog_end

* Update docs/source/daml-repl/index.rst

Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

* s/Repl/REPL/

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2020-02-25 11:16:31 +00:00
Moritz Kiefer
8d81399c0f
Add an experimental DAML script REPL (#4660)
As mentioned in the title, this is still very experimental and needs
more work before we want to advertise it. However, the code is in a
somewhat reasonable shape, there are tests and I think even in the
current state it is already useful. Also this PR is already getting
very large so I don’t want to hold off much longer before merging this.

It is included in the SDK but hidden from `damlc --help` and `daml
--help` until the most pressing issues are addressed (primarily around
making sure that it doesn’t just shut down if you have a type error
and better error messages in general).

changelog_begin
changelog_end
2020-02-24 11:06:27 +01:00
Samir Talwar
2c4807f3bf
release: Sort artifacts.yaml. (#4502)
I verified that there are no semantic changes, only cosmetic ones, by
running the following before and after, and checking the diff.

```
< release/artifacts.yaml ruby -rjson -ryaml -e 'puts JSON::generate YAML::load $stdin' | jq '. | sort_by(.target)' > ~/Desktop/artifacts.{old,new}.json
```

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-13 13:52:48 +00:00
fabiotudone-da
6e1fd55e45
Publish com.daml.ledger:participant-state-kvutils-app (#4480)
CHANGELOG_BEGIN
CHANGELOG_END
2020-02-12 09:45:52 +01:00
Samir Talwar
6fd402f634
Sandbox: Preliminary work to rebuild Sandbox on top of kvutils. (#4422)
* resources: Allow for startup exceptions without no stack trace.

* sandbox: Remove dependency duplication in the BUILD file.

* ledger-on-(memory|sql): Extract `Main` into its own file tree.

This is so we can remove the dependency on kvutils/app from the library.

logback.xml goes with it.

* sandbox: Implement "Sandbox Next" on top of ledger-on-sql.

CHANGELOG_BEGIN
- [Sandbox] Preliminary work to rebuild Sandbox on top of the DAML
  Ledger Integration Kit. Currently not exposed through the CLI.
CHANGELOG_END

* ledger-on-sql: Publish to Maven Central.

* sandbox: If an unknown database is passed to Sandbox Next, handle it.

* resources: Catch errors in owner construction or acquisition.

Sometimes we're not in a future yet.

* resources: Don't run ProgramResource error handling inside the EC.

Otherwise we end up in deadlock which is only broken by
`awaitTermination` timing out.
2020-02-06 17:34:17 +00:00
Moritz Kiefer
fc87953ed4
Enhanced completions for the assistant (#4420)
* Enhanced completions for the assistant

changelog_begin

- [DAML Assistant] The assistant can now do completions for SDK
  commands, e.g., ``daml ledger upl<TAB>`` will complete to ``daml
  ledger upload-dar``.

changelog_end

* Apply suggestions from code review

Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>

Co-authored-by: associahedron <231829+associahedron@users.noreply.github.com>
2020-02-06 15:16:12 +01:00
Gary Verhaegen
47bd131f15
add copyright headers to yml files (#4407)
We seem to have forgotten about them in the copyright scripts.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-06 12:54:07 +01:00
Moritz Kiefer
bd127d6c47
Flip the default on mavenUpload in artifacts.yaml (#4396)
changelog_begin
changelog_end
2020-02-05 10:32:02 +01:00
Moritz Kiefer
60e32a98c4
Disable release step on macos (#4330)
Originally we ran the release step on both Linux and MacOS to handle
platform dependent artifacts, in particular, damlc.jar. However, we
don’t have any platform dependent artifacts that are uploaded as part
of the release script anymore and I hope we will never have to add any
in the future.

So this PR, removes the code for handling platform dependent artifacts
in the release step and disables the release step on MacOS (while
still setting the variables like we do on Windows).

Currently the release step still costs us ~2 minutes on MacOS which is
already our slowest platform so hopefully this will speed things up a
bit.

changelog_begin
changelog_end
2020-01-31 17:08:47 +01:00
Robin Krom
eacacbf44e
fix release script (#4275)
* fix release script

only upload from the linux machine and escape arguments to `publish`,
otherwise they get interpreted by bazel.

CHANGELOG_BEGIN
CHANGELOG_END

* Update release/src/Main.hs

Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Update release/src/Main.hs

Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* bracket -> bracket_

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-01-30 10:51:39 +01:00
Robin Krom
ce9cd93519
publish npm packages as public (#4263)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-29 17:26:46 +01:00
Robin Krom
2e9fe6afb6
language: add daml-react package to ts libraries (#4259)
* language: add daml-react package to ts libraries

This adds the library formerly known as `daml-react-hook` into the
monorepo. We renamed it to `@daml/react`.

The tests sadly don't work with bazel right now because the local
imports aren't resolved correctly. Local testing with `yarn run test`
works as usual.

CHANGELOG_BEGIN
CHANGELOG_END

* address moritz comments

* get rid of DAVL mentions

* fix eslint warnings

* Update language-support/ts/daml-react/tsconfig.json

Co-Authored-By: Martin Huschenbett <martin.huschenbett@posteo.me>

Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
2020-01-29 15:37:47 +01:00
Shayne Fletcher
0e53ec0a2b Integrate daml2ts into the assistant (#4252)
changelog_begin
- Generate TypeScript bindings to DAML with `daml codegen ts DAR -o OUTDIR`
changelog_end
2020-01-28 22:03:54 +00:00
Shayne Fletcher
9c0fe75b94
Add daml2ts to the SDK release tarball (#4234)
changelog_begin
- daml2ts now included in the SDK
changelog_end
2020-01-27 13:18:05 -05:00
Moritz Kiefer
1a0fb138fb
Ignore missing maven deps on macos (#4201)
Currently, pretty much all of our builds are bottlenecked on
MacOS (mainly because the builders are slower and have worse
caching). The release step adds > 3min to each build which is a bit
annoying. It turns out that removing the calls to `bazel query` which
are used to check for missing Maven dependencies speeds this up by >
5x. Given that the check is platform independent anyway, we can just
disable it on MacOS.

changelog_begin
changelog_end
2020-01-24 16:17:08 +00:00
Robin Krom
811d740df5
upload typescript packages to npm registry (#4185)
* make packages public

This uploads the typescript npm packages of the language support to the
npm registry in the release process.

CHANGELOG_BEGIN
CHANGELOG_END

* address moritz/gary's review

* generate the .npmrc file

* adding debug output

just in case the upload will fail in the next release.

* reverse package order
2020-01-23 18:31:50 +01:00
Moritz Kiefer
2fb107b505 Remove damlc.jar (#4157)
* Remove damlc.jar

We never advertised or published this externally and our only internal
user has moved off this months ago already.

changelog_begin
changelog_end

* Remove dependency from navigator test lib on damlc jar
2020-01-22 16:44:27 +00:00
Samir Talwar
e2a5b26475
contextualized-logging: Move to its own Bazel package. (#4138)
I want to use this in _ledger/ledger-on-sql_, which doesn't depend
directly on Sandbox.

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-21 14:25:06 +01:00
Stefano Baghino
1ad7d6c24c
Replace NamedLoggerFactory (#4097)
* Replace NamedLoggerFactory

CHANGELOG_BEGIN
CHANGELOG_END

* Recover change lost in rebase

* Address https://github.com/digital-asset/daml/pull/4045#discussion_r366883577

* Address https://github.com/digital-asset/daml/pull/4045#discussion_r366885112

* Address two open review comments

Address https://github.com/digital-asset/daml/pull/4045#discussion_r366885448
Address https://github.com/digital-asset/daml/pull/4045#discussion_r366885981

* Address outstanding compilation errors

* Replace mocking with in-memory log collector

* Address https://github.com/digital-asset/daml/pull/4097#pullrequestreview-344639319

* Address https://github.com/digital-asset/daml/pull/4045#discussion_r366886250

The generation of code to have logging in the services has been replaced
by helpers classes. This will allow to enrich the context received at
construction by the service implementations.

* Use ContextualizedLogger for TrackerMap

* Remove deleted logging packages from artifacts.yaml

* Remove remaining deleted artifact from artifacts.yaml

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368640231

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368648027

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368652178

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368654996

* Annotate type of references to logErrorOnCall

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368869575
2020-01-21 10:57:28 +01:00
Samir Talwar
b96e54f640 libs-scala/resources: Move Resource and friends to their own package. (#4066)
* libs-scala/resources: Move `Resource` and friends to their own package.

CHANGELOG_BEGIN
- [Ledger API Server] Publish the resource management code as a library
  under ``com.digitalasset:resources``.
CHANGELOG_END

* libs-scala/resources: Add to the artifacts list.

* libs-scala/resources: Factor out the test helpers into test/lib.

* libs-scala/resources-akka: Make the Akka resources an optional dep.
2020-01-16 14:13:18 +00:00
Andreas Herrmann
6e25d10fe2
daml test-script (#3918)
* Start on daml test-scripts

* Run all `Script a` as test cases

* LedgerClient: Expose PackageManagementClient

To enable DAR uploads

* Upload the DAR to the ledger

* Start sandbox if no ledger specified

* Format daml test-script

* Fix deprecation warning on ActorMaterializer

* Add test-case //daml-script/tests:test_daml_script_test_runner

* Add daml test-script command

CHANGELOG_BEGIN

- [DAML Script - Experimental] Allow running DAML scripts as test-cases.
  Executing ``daml test-script --dar mydar.dar`` will execute all
  definitions matching the type ``Script a`` as test-cases.
  See `#3687 <https://github.com/digital-asset/daml/issues/3687>`__.

CHANGELOG_END

* daml-test-script enable logging

* Remove outdated TODO comment

* daml script-test More elaborate test-caseo

Compare to expected output and add failing test-case

* daml test-script Don't abort on test-failure

Before the test runner would abort on the first failed test-case. This
occasionally introduce additional test-failures if the sandbox was
torn down half-way through execution.

* ./fmt.sh

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-03 15:01:36 +01:00
Gary Verhaegen
878429e3bf
update copyright notices to 2020 (#3939)
copyright update 2020

* update template
* run script: `dade-copyright-headers update .`
* update script
* manual adjustments
* exclude frozen proto files from further header checks (by adding NO_AUTO_COPYRIGHT files)
2020-01-02 21:21:13 +01:00
Stefano Baghino
7020ed3b15
Moves CommandCompletionIT to Ledger API Test Tool (#3901)
* Moves CommandCompletionIT to Ledger API Test Tool

Closes #2841

Also refactors test observers into a single API and unifies the existing implementations of the DirectExecutionContext

* Allow explicit offsets to get checkpoints

* Raise checkpoint test timeout

* Consolidate checkpoint tests in a single test case

* Fix compilation issues

* Exclude command submission/completion tests for rev-v2

* Drop test not relevant for Ledger API implementations

* Fix compilation errors
2019-12-20 15:55:23 +01:00
Rohan Jacob-Rao
2b4a79c68f Tidy up release instructions (#3911)
* Tidy up first step of release instructions

* Minor adjustments to the rest of the instructions

* Fix

* Tweak
2019-12-20 11:12:12 +01:00
Gary Verhaegen
2b3693bf4d
restore 0.13.38 logging defaults for daml jars (#3819)
CHANGELOG_BEGIN
- [Sandbox] Restore 0.13.38 logging behaviour.
- [Navigator] Restore 0.13.38 logging behaviour.
- [Extractor] Restore 0.13.38 logging behaviour.
- [Internals] As of 0.13.39, we merged a number of internal JAR files in
  the SDK tarball to reduce its size. These jars used to be standalone
  JARs you could invoke as e.g. ``java -jar sandbox.jar <args>``. As a
  result of merging the jars, they lost their individual ``logback.xml``
  configuration file. Although running the jars directly was (and is
  still) not supported, note that you can now achieve the same behaviour
  with e.g. ``java -Dlogback.configurationFile=sandbox-logback.xml -jar
  daml-sdk.jar sandbox <args>``.
CHANGELOG_END
2019-12-12 14:02:23 +01:00
Remy
0107228c49
daml-lf: distribute daml_lf_1.7_archive_java_proto (#3792) 2019-12-10 12:24:55 +01:00
Samir Talwar
639eba2c17 ledger-api-scala-logging: Fix errors in IntelliJ IDEA. (#3785)
* ledger-api-scala-logging: Fix errors in IntelliJ IDEA.

The Bazel plugin for IntelliJ doesn't seem to be smart enough to be able
to handle a Scala library that is part `src` directory and part
generated code from another Bazel rule. It just ignores the second part.
This means that IntelliJ cannot find the *ServiceLogging classes, as
they're not represented on the Bazel-generated classpath, and so
complains with lots of errors when working on the equivalent Api*Service
files.

To fix this, we can split these in two, compiling the base traits to
`ledger-api-scala-logging-base` and then the generated code separately.

It does result in an extra Bazel dependency for the users of
ledger-api-scala-logging, as Bazel doesn't realise transitive
dependencies for us.

* Release: Add `ledger-api-scala-logging-base` to the Maven list.
2019-12-09 16:02:59 +00:00
Moritz Kiefer
ab59376cb3 Fix Sdk-Version field in daml-sdk-head (#3759)
In #3706 we fixed the SDK versions of the daml-trigger an daml-script
libraries and (rightfully) stopped filtering out 0.0.0 from the SDK
version check in `damlc build`. However, this broke daml-sdk-head
since we still distributed the DARs in daml-sdk-head with the
Sdk-Version set to whatever the current release is rather than 0.0.0.
2019-12-06 10:08:37 +00:00
Moritz Kiefer
022c60ccb3 Publish the thin trigger library instead of the fat jar (#3739)
This still contains the main class so you can use it like you would
use the fat jar but publishing fat jars to maven central is apparently
bad practise and some peple have asked for the library.

This includes some slight tweaks to the scala_docs rule to make it
capable of coping with the generated source file and a hack in the
release script to avoid it complaining about the scenario proto
library not being published to Maven even though it is included in the
transitive deps.
2019-12-05 13:04:20 +00:00
Stefano Baghino
5254520b80 Spin off TokenHolder into a new library (#3741)
* Spin off TokenHolder into a new library

Avoids having weird dependencies between different packages, makes TokenHolder available on Maven

* Fix auth-utils path
2019-12-05 12:31:14 +00:00
Moritz Kiefer
08b98602b1
Build SDK head tarball directly (#3723)
Previously, we constructed the tarball used by daml-sdk-head by first
building the regular SDK release tarball only to the extract it, patch
the version file and recompress it. This adds about 30-60s to every
invocation of daml-sdk-head.

This PR changes this by factoring out the logic from building the
release tarball into a macro that we instantiate twice, once with the
proper version file and once with a dummy HEAD version file set to
0.0.0.

This does change the format of the sdk-head-tarball slightly, in
particular the files are now located under sdk-0.0.0 instead of
sdk-head. However, this doesn’t matter anyway afaik and I think the
new format makes more sense anyway considering that the regular
release tarballs have something like sdk-0.13.38.
2019-12-04 13:04:15 +01:00
Samir Talwar
5dd38d54e8 sandbox: PostgreSQL health checks. (#3655)
* ledger-api-test-tool: Increase the duration when watching health.

This should hopefully stop CI from flaking out.

* reference-v2/sandbox: Avoid unnecessary companion object constructors.

I like indirection… when it does something.

* ledger: Propagate empty health checks throughout the services.

* reference: Remove duplication from the ReferenceServer object.

* ledger-api-common: Actually query a "reporter" in the health service.

* ledger-api-common: Report health per-component when required.

* ledger-api-health: Use a Map to represent components for health checks.

* sandbox: Fix warnings in SqlLedgerSpec.

* ledger-api-common: Throw GrpcHealthService errors inside the Future.

* ledger: Implement health checks against the PostgreSQL connection.

Without proper testing, because I am not great at this.

* sandbox: Remove duplication and fix warnings in PostgresAround.

* sandbox: Test the SQL Ledger's health reporting on failure.

* sandbox: Don't report as unhealthy until 3 connections fail.

* ledger-api-health: Remove unused parts of the API.

Bit of premature design there.

* sandbox: Rename the "ledger" health check to "write".

* participant-state: Add the ReportsHealth trait to ReadService.

* ledger-api-common: `Future.fromTry(Try(…))` -> `Future(…)`.

* ledger-api-common: Make it clearer that StubReporter closes over health.

* ledger-api-common: Explain the HealthService watch tests with comments.

* sandbox: Clean up SqlLedger a bit.

* sandbox: Don't try and stop PostgreSQL twice in PostgresAround.

* bazel_tools: Windows rlocation lookups need to be with forward slashes.

* release: Fix case of "true".

* ledger-api-common: Make `GrpcHealthService::matchResponse` return a Try.

* ledger-api-common: Make `GrpcHealthServiceSpec` async.

* sandbox: Make a couple of DB classes final.

* sandbox: Avoid importing `X._` in PostgresAround.

* sandbox: Add clues to the SqlLedgerSpec's multiple assertions.

* sandbox: If PostgreSQL doesn't come back up, keep retrying.

* sandbox: Remove duplication in SqlLedgerSpec.

* sandbox: In SqlLedgerSpec, actually wait for the health to change.

* sandbox: In PostgresAround, make stopping PostgreSQL idempotent.

* sandbox: Simplify the SqlLedgerSpec to make it work on CI.

It's worth a shot.

* ledger-api-common: Simplify the GrpcHealthServiceSpec a little.

And add a changelog.

CHANGELOG_BEGIN

- [Ledger API Server] Add a health check endpoint conforming to the
  `GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
- [Ledger API Server] Add health checks for index database connectivity.
- [Participant State API] Add a mandatory ``currentHealth()`` method to
  ``IndexService``, ``ReadService`` and ``WriteService``.

CHANGELOG_END

* sandbox: Improve the Javadoc layout for DbDispatcher.

* sandbox: Capitalize constants in SqlExecutor.

* ledger-api-health: Convert HealthStatus to an abstract class.
2019-11-29 15:07:43 +00:00
associahedron
2713d6978d
Reduce SDK tarball size by 70% by deduplicating Scala dependencies. (#3678)
* Start moving all the jars into a single jar.

* Fold navigator into daml-sdk jar

* include sandbox

* Remove unnecessary compileDeps.

CHANGELOG_BEGIN

- [DAML SDK] Reduced the size of the DAML SDK by about
60% uncompressed, 70% compressed, by deduplicating Scala
dependencies.

CHANGELOG_END

* update copyright header

* buildifier fix
2019-11-29 13:26:47 +00:00
Oliver Seeliger
f11a3ce9a7
Release triggers to maven central (#3674) 2019-11-29 10:25:02 +01:00
Brian Healey
02186ef068 Ledger configuration indexing changes (rebased to master) (#3553)
* Add participant-state configuration protobuf

- Move the "DamlConfiguration" from kvutils into participant-state/protobuf/ledger_configuration.proto.
- Add version number and spec (ala transaction.proto)

This is a preparation for indexing the configuration and having one canonical serialization for it.

* Initial thoughts on indexing the configuration

* Implement indexing of ledger configuration changes

* Add record time to all Updates. Wire through participant id.

and rename V7__Add_configuration to V8.

* Add ledger_configuration_java_proto to release artifacts

* Fix up release of ledger_configuration_java_proto

* Suggestions from review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* address rebase issues

* Resolve compilation errors after rebase/merge

* happy formatting for scala and bazel and friends

* Drop "openWorld" setting from configuration

And refactor the tests to allocate parties explicitly.

* Fix up migration and tests

* Drop authorizedParticipantIds from configuration

Implement configuration authorization in kvutils using the previous
the participant id of the previous configuration.

* Post-rebase fixes

* Add missing migrations

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Add missing mavenUpload to ledger-api-server

* Remove stateUpdateRecordTime

* Address code review

- Address PR review
- Merge TimeModelImpl and the traits. Remove TimeModel from ledger-api-common.
- Throw `Err` from KeyValueConsumption on parse errors instead of assert/sys.error

* Reformat

* Add missing protobuf file

* Fix compilation after TimeModel changes. Add version logs to participant-state{,-kvutils}.

* Fix TestUtil.scala build

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Address review

- synchronized access to ledgerConfiguration in InMemoryLedger
- store rejection if configuration generation is wrong

* Update ledger/participant-state/protobuf/ledger_configuration.rst

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2019-11-27 17:41:23 +01:00
Stefano Baghino
4d922e763b Add authentication to Java identity client (#3630)
* Add authentication to Java identity client

Contributes to #3626

No changelog added right now, it will be added with the PR that
effectively closes the ticket.

Moves `LedgerCallCredentials` to a shared package.

Establishes the test framework for other Java clients.

* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731

* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731

* Rebase and fix compilation errors

* Fix type of jar in artifacts.yaml

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Fix more errors emerged after rebase

* Add dependency for DAML assistant ITs

* Add missing dependency

* Fix wrong dependency

* fix release
2019-11-26 18:51:09 +00:00
Samir Talwar
0ff716df2a Ledger API: Add healthcheck endpoints. (#3573)
* grpc-definitions: Delete health_service.proto

We can use the version in io.grpc:grpc-services instead.

* ledger: Delete ledger/API.md.

* sandbox: Fix warnings in ApiServices flagged by IntelliJ.

* sandbox: Implement a dummy grpc.health.v1.Health.Check endpoint.

* sandbox: Implement a dummy grpc.health.v1.Health.Watch endpoint.

* sandbox: Drop repeated elements from grpc.health.v1.Health.Watch.

* sandbox: Wrap the HealthService in basic tests.

* sandbox: Stop streaming the server health too.

* ledger-api-test-tool: Health check tests.

* Add a changelog entry for the health check endpoints.

CHANGELOG_BEGIN

- [Ledger API] Add healthcheck endpoints, conforming to the
  `GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
  It is always ``SERVING`` for now.

- [DAML Ledger Integration Kit] Add conformance test coverage for the
  ``grpc.health.v1.Health`` service.

CHANGELOG_END

* ledger-api-integration-tests: Increment the number of services.

* Apply suggestions from code review

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* sandbox: Use `AkkaBeforeAndAfterAll` in the HealthServiceSpec.

In an attempt to get it working on CI.

* sandbox: Change `dropRepeated` to `DropRepeated()`.

Keep it in one file.

* test-common: Use `Delayed.by` in `TimeBoundObserver`.

* test-common: Close the source when `TimeBoundObserver` completes.

* ./fmt.sh

That'll teach me not to `--no-verify` just because it's a merge commit.

* sandbox: Inline `HealthService.suppress`.

At some point it was being used twice.

* sandbox: Increase the timeout for HealthServiceSpec.

* sandbox: Reimplement HealthService using the Scala protobuf types.

* sandbox: Generate an Akka-compatible trait for the health service.

And refactor a lot of test code to make it easy to test.

* ledger-api-common: Move the HealthService here.

* rs-grpc-testing-utils: Publish to Maven.

* rs-grpc-testing-utils: Add Maven coordinates.
2019-11-22 14:02:05 +00:00
Stefano Baghino
84e6064d51 Remove unreleased.rst (#3547)
* Start working on getting rid of unreleased.rst

Document new process in CONTRIBUTING.md,
.github/pull_request_template.md and unreleased.rst (for good measure)

Report the previous changelog additions here so that they're not lost in
the mist of times.

CHANGELOG_BEGIN

- [DAML Stdlib] Added the ``NumericScale`` typeclass, which improves the type inference for Numeric literals, and helps catch the creation of out-of-bound Numerics earlier in the compilation process.

- [DAML Triggers] ``emitCommands`` now accepts an additional argument
  that allows you to mark contracts as pending. Those contracts will
  be automatically filtered from the result of ``getContracts`` until
  we receive the corresponding completion/transaction.

- [Navigator] Fixed a bug where Navigator becomes unresponsive if the ledger does not contain any DAML packages.

- [Ledger-API] Add field ``gen_map`` in Protobuf definition for ledger
  api values. This field is used to support generic maps, an new
  feature currently in development.  See issue
  https://github.com/digital-asset/daml/pull/3356 for more details
  about generic maps.

  The Ledger API will send no messages where this field is set, when
  using a stable version of DAML-LF.  However the addition of this
  field may cause pattern-matching exhaustive warnings in the code of
  ledger API clients. Those warnings can be safely ignored until
  GenMap is made stable in an upcoming version of DAML-LF.

- [JSON API - Experimental] CLI configuration to enable serving static content as part of the JSON API daemon:
  ``--static-content "directory=/full/path,prefix=static"``
  This configuration is NOT recommended for production deployment. See issue #2782.

- [Extractor] The app can now work against a Ledger API server that requires client authentication. See `issue #3157 <https://github.com/digital-asset/daml/issues/3157>`__.
- [DAML Script] This release contains a first version of an experimental DAML script
   feature that provides a scenario-like API that is run against an actual ledger.
- [DAML Compiler] The default DAML-LF version is now 1.7. You can
  still produce DAML-LF 1.6 by passing ``--target=1.6`` to ``daml
  build``.

- [JSON API - Experimental] The database schema has changed; if using
  ``--query-store-jdbc-config``, you must rebuild the database by adding
  ``,createSchema=true``.
  See `issue #3461 <https://github.com/digital-asset/daml/pull/3461>`_.

- [JSON API - Experimental] Terminate process immediately after creating schema. See issue #3386.

- [DAML Stdlib] ``fromAnyChoice`` and ``fromAnyContractKey`` now take
  the template type into account.

CHANGELOG_END

* Document new release process to gather changelog additions

* Change the release script to ignore unreleased.rst

* Remove spurious unreleased.rst lines

* Transition to use tags

* Document new way to get changelog additions with tags

* Update release/RELEASE.md

Co-Authored-By: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Address https://github.com/digital-asset/daml/pull/3547#discussion_r348438786

* Document correction process

* Add copyright header to unreleased.sh

* Update CONTRIBUTING.md

Co-Authored-By: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Modify CONTRIBUTING.md after @garyverhaegen-da's proposal

* Make unreleased.sh run per commit and treat tags as case-insensitive

* Fix documentation for replacements
2019-11-20 15:16:57 +00:00
Moritz Kiefer
7deca903bb Add documentation for DAML script and bundle it in the SDK (#3527) 2019-11-19 17:19:57 +00:00
Stefano Baghino
c51ecd3c58
Extractor with authentication (#3514)
* Make Extractor work against Ledger API servers protected by authentication

* Update changelog

* Address https://github.com/digital-asset/daml/pull/3514#pullrequestreview-318811500

- https://github.com/digital-asset/daml/pull/3514#discussion_r347753205
- https://github.com/digital-asset/daml/pull/3514#discussion_r347753629
- https://github.com/digital-asset/daml/pull/3514#discussion_r347754462

* Address https://github.com/digital-asset/daml/pull/3514#discussion_r347755484

* Address https://github.com/digital-asset/daml/pull/3514#discussion_r347752424

* Address https://github.com/digital-asset/daml/pull/3514#discussion_r347756328

* Fix docs

* Fix artifacts.yaml
2019-11-19 17:12:25 +01:00
Moritz Kiefer
e4632f5fb2 Include "daml trigger" in "daml --help" (#3528) 2019-11-19 16:19:17 +01:00
Oliver Seeliger
f50901347f
Release lf scalacheck test jars to maven central (#3519)
as they are dependencies used downstream
2019-11-19 13:10:28 +01:00