Commit Graph

141 Commits

Author SHA1 Message Date
Stefano Baghino
1f0534ea4f
Spin-off BuildInfo into its own (micro-)library (#5004)
* Spin-off BuildInfo into its own (micro-)library

CHANGELOG_BEGIN
CHANGELOG_END

* Fix dependencies

* Remove unused dependency
2020-03-16 09:04:00 +01:00
Moritz Kiefer
048b3fa679
Fix warnings on mismatch between record type and constructor name (#4919)
changelog_begin
changelog_end
2020-03-10 13:01:16 +01:00
Samir Talwar
a99156252d
libs-scala/ports: Wrap socket ports in a type, Port. (#4784)
* libs-scala/ports: Wrap socket ports in a type, `Port`.

* sandbox: Use `Port` for the API server port, and propagate.

CHANGELOG_BEGIN
CHANGELOG_END

* extractor: Use `Port` for the server port.

* ports: Make Port a compile-time class only.

* ports: Allow port 0; it can be specified by a user.

* ports: Publish to Maven Central.
2020-03-03 08:59:15 +00:00
Moritz Kiefer
d68d3eb74a
Freeze DAML-LF 1.8 (#4770)
* Freeze DAML-LF 1.8

Two minor points that I did not mention in the previous PR:

We also include the renaming of structural records to `struct` and the
renaming of `Map` to `TextMap`.

There are some minor changes around the LF encoder tests which need to
be able to emit package metadata properly so I’ve added it to the
parser. Sorry for not splitting that out.

Following the process used for the DAML-LF 1.7 release, this does not
yet include the frozen proto file.

changelog_begin

- [DAML-LF] Release DAML-LF 1.8:

  * Rename structural records to ``Struct``. Note that
    structural records are not exposed in DAML.
  * Rename ``Map`` to ``TextMap``.
  * Add type synonyms. Note that type synonyms are not serializable.
  * Add package metadata, i.e., package names and versions.

  Note that the default output of ``damlc`` is stil DAML-LF 1.7. You
  can produce DAML-LF 1.8 by passing ``--target=1.8``.

changelog_end

* Update encoder

* Update java codegen tests

* Update comment in scala codegen

* Handle TSynApp in interface reader

* Bump lf_stable_version to 1.7

* Fix kvutils tests
2020-03-02 18:29:26 +01:00
Shayne Fletcher
66dd112960
Remove pragma 'daml 1.2' (#4702)
changelog_begin
- The pragma 'daml 1.2' is now optional.
changelog_end
2020-02-26 13:17:45 -05:00
Gary Verhaegen
5a117dc358
introduce new release process (#4513)
Context
=======

After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.

This PR is a proposal for a new release process designed under the
following constraints:

- Reuse as much as possible of the existing infrastructure, to minimize
  effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
  not meant for general public consumption, but can still be used by savvy
  users without jumping through too many extra hoops (ideally just
  swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
  release status, with as few changes as possible in-between, so we can be
  confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
  releases, to avoid discovering non-transient problems while trying to
  promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
  keep the same approval process for SOC2 auditability.

The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:

```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```

This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.

Version numbers
===============

Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:

- Semantic versioning (which we chose as the version format for the SDK
  version number) allows for "prerelease" version numbers as well as
  "metadata"; an example of a complete version string would be
  `1.2.3-nightly.201+server12.43`. The "main" part of the version string
  always has to have 3 numbers separated by dots; the "prerelease"
  (after the `-` but before the `+`) and the "metadata" (after the `+`)
  parts are optional and, if present, must consist of one or more segments
  separated by dots, where a segment can be either a number or an
  alphanumeric string. In terms of ordering, metadata is irrelevant and
  any version with a prerelease string is before the corresponding "main"
  version string alone. Amongst prereleases, segments are compared in
  order with purely numeric ones compared as numbers and mixed ones
  compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
  which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
  `-`, or a transition between a number and a letter. Version strings
  are compared element-wise, with numeric segments being compared as
  numbers. Alphabetic segments are treated specially if they happen to be
  one of a handful of magic words (such as "alpha", "beta" or "snapshot"
  for example) which count as "qualifiers"; a version string with a
  qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
  which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
  special ordering amongst qualifiers. Other alphabetic segments are
  compared alphabetically and count as being "after" their prefix
  (`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
  separated by `.`, plus an optional (though deprecated) alphanumeric
  "version tag" separated by a `-`. I could not find any official
  documentation on ordering for the version tag; numeric segments are
  compared as numbers.
- npm uses semantic versioning so that is covered already.

After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:

```
0.13.53-alpha.20200214.3025.ef5d32b7
```

where the components are, respectively:

- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
  master.
- `3025`: the number of commits in master up to the release commit
  (included). Because we have a linear, append-only master branch, this
  uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
  not strictly speaking necessary, but makes it a lot more convenient to
  identify the commit.

The main downsides of this format are:

1. It is not a valid format for GHC packages. We do not publish GHC
  packages from the SDK (so far we have instead opted to release our
  Haskell code as separate packages entirely), so this should not be an
  issue. However, our SDK version currently leaks to `ghc-pkg` as the
  version string for the stdlib (and prim) packages. This PR addresses
  that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
  would see the above version number as `0.13.53.20200214.3025`, which
  should be enough to uniquely identify it. Note that, as far as I could
  find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
  it makes it more scary. However, I have been told that this may be
  long enough to cause issues on Windows by pushing us past the max path
  size limitation of that "OS". I suggest we try it and see what
  happens.

The upsides are:

- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
  they have the same date, allowing us to release same-day patches if
  needed. (Note: that is, commits that were made on the same day; the
  release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
  immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
  GHC).

Alternatives I considered:

- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
  be accepted by all schemes, but will not sort as expected under semantic
  versioning (though Maven will be fine). I have no idea how it will sort
  under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
  This is not valid semantic versioning and is therefore rejected by
  npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
  what is generally done in the Java world, but we then lose track of what
  version is actually in use and I'm concerned about bug reports. This
  would also not let us publish to the main Maven repo (at least not more
  than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
  version formats. However, it would not clearly indicate to humans that
  it is not meant as a stable version, and would sort differently under
  semantic versioning (which counts it as a prerelease, i.e. before
  `0.13.53`) than under maven (which counts it as a patch, so after
  `0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
  number of prereleases in-between `0.13.52` and the next. This is
  currently the fallback plan if Windows path length causes issues. It
  would be less convenient to map releases to commits, but it could still
  be done via querying the history of the `LATEST` file.

Release notes
=============

> Note: We have decided not to have release notes for snapshot releases.

Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.

But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.

The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.

> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.

New release process
===================

Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.

Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:

1. Before running the release step, CI will checkout the commit
  specified in the LATEST file. This requires separating the release
  step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
  that gets the version to build from an environment variable, with a
  default of `0.0.0` to remain consistent with the current `daml-head`
  behaviour.

Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.

The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.

I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-25 17:01:23 +01:00
Moritz Kiefer
ff90ec643e
Remove hardcoded number of packages from Java codegen test (#4604)
This breaks everytime we change the number of packages in damlc which
is rather annoying.

changelog_begin
changelog_end
2020-02-19 15:45:35 +01:00
associahedron
5db4ac8b0e
Support type-level strings in DAML. (#4571)
* Add type-level strings in DAML.

This PR adds a `PromotedText` stable package, with `PromotedText` type, which is used to encode type-level strings from DAML into DAML-LF. The reason for this is to preserve the `HasField` instance argument. This PR adds a test that `HasField` is succesfully reconstructed incontexts, during data-dependencies, which wasn't possible before.

changelog_begin
changelog_end

* adresss comments

* fix overly specific tests
2020-02-19 13:10:36 +00:00
Samir Talwar
f231ebc1b3
Sandbox-Next: Re-establish static time support. (#4581)
* kvutils: Extract a committer from the uses of `SubmissionValidator`.

This makes the clock injectable too.

* kvutils: Provide logging contexts in the `Runner`.

* sandbox: Remove the `StaticAllowBackwards` time provider type.

It's not used anywhere.

* sandbox: Fix warnings in CliSpec.

* sandbox: Ensure that we cannot specify both static and wall-clock time.

* sandbox-next: Crash if wall clock time is not specified.

* sandbox-next: Document more known issues in the new Sandbox.

* sandbox: Add a Clock (and some tests) to TimeServiceBackend.

* sandbox-next: Support static time.

CHANGELOG_BEGIN
- [Sandbox Next] Re-establish static time mode.
CHANGELOG_END

* ledger-on-(memory|sql): Expect a `() => Instant`, not a `Clock`.
2020-02-18 17:03:15 +00:00
Stefano Baghino
bd32bd6c8f
Return proper code for invalid authentication (#4485)
* Return proper code for invalid authentication

CHANGELOG_BEGIN
[Sandbox] If authentication is enabled, requests without a valid
authentication are going to be rejected with an ``UNAUTHENTICATED``
return code instead of ``PERMISSION_DENIED``.
CHANGELOG_END

* Reduce logging noise from java-rxbindings tests

* Fix rxjava bindings tests to match new behavior

* Fix extractor tests to match new behavior

* Address https://github.com/digital-asset/daml/pull/4485#discussion_r378507478
2020-02-17 11:09:16 +00:00
Moritz Kiefer
d68d197948
Disable scaladoc on the MacOS CI (#4524)
* Disable scaladoc on the MacOS CI

It is still built by default locally.

fixes #4441

changelog_begin
changelog_end

* Change tag name
2020-02-14 14:08:24 +00:00
Samir Talwar
a589f4af0b
sandbox: Move more resource acquisition into the owner. (#4501)
* sandbox: Move more resource acquisition into the `owner`.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Reimplement SandboxClientResource as a resources.Resource.

* codegen: Use resources in TestUtil.

* sandbox: Manage PostgreSQL in tests with ResourceOwners.
2020-02-14 13:52:45 +00:00
Moritz Kiefer
36e188cac8
Translate unsupported kinds to a special Erased type (#4474)
* Translate unsupported kinds to a special Erased type

This should simplify `data-dependencies` and avoid issues like #4470
since we can match on the type instead of having to guess which types
can and which cannot be translated back to DAML.

changelog_begin
changelog_end
2020-02-12 09:03:19 +01:00
Gerolf Seitz
85db1033d7
Fix memory leak when using the reset service (#4447)
* sandbox: Don't hold on to old resources when resetting.

Now there's one hell of a memory leak.

CHANGELOG_BEGIN
- [Sandbox] Fixed a memory leak when using the ResetService; not
  everything was cleaned up correctly.
CHANGELOG_END

* sandbox: Split out SandboxClientResource from SandboxServerResource.

Gonna replace SandboxServerResource with a ResourceOwner acquisition.

* sandbox: Don't capture the API server in the SandboxServer resource.

When we reset, this is stored forever, leading to a memory leak.

Tested by rewriting the SandboxServerResource to use
`SandboxServer.owner`.

* sandbox: Revert the test client resource to calling `shutdownNow()`.

* sandbox: Make sure the fixture is recreated properly on each test run.

* sandbox: Make `SandboxState` a non-case class.

The `toString()` was unnecessarily heavy.

* sandbox: Futures, futures everywhere.

Avoid a race condition where the server is stopped before it starts by
storing a `Future[SandboxState]` rather than the `SandboxState` itself.

This doesn't trigger the same memory leak as storing a
`Resource[SandboxState]` because we don't capture the object itself in
the `flatMap` in the same way with `Future`.

* sandbox: Remove an unused parameter left in for debugging.

* sandbox: Replace `@VisibleForTesting` with a comment.

* sandbox: Add more comments to the weird logic in SandboxServer.

* sandbox: Get rid of the `Port` type alias; it was confusing.

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-02-10 16:47:39 +00:00
Leonid Shlyapnikov
c8c55c4408
Exercise by key (#4049)
* Implement exercise by key

ExerciseCommand got a new required element: `reference` of polymorphic type.

* Add test case: exercise Archive by contractKey

* Add test case for ExerciseCommand JSON protocol

* flatten contract reference in ExerciseCommand JSON protocol

* formatting

* Update exercise by key

* Update documentation

CHANGELOG_BEGIN

- [JSON API - Experimental] Support Exercise by Key. See #4009.

CHANGELOG_END

* Address code review comments
2020-01-18 09:06:52 -05:00
Samir Talwar
5aa33cf7e9 bindings-rxjava: Attempt to fix the flakiness in BotTest. (#3931)
* bindings-rxjava: Fix warnings and tidy up code in BotTest.

* bindings-rxjava: Complete BotTest.TestFlowable properly.

This avoids a stack trace in the logs.

* bindings-rxjava: Use `should have size` in BotTest.

Better error messages when it fails.

* bindings-rxjava: Use `Eventually` to wait, rather than `Thread.sleep`.

This should hopefully fix the flakiness in this test.
2020-01-13 08:46:31 +00:00
Moritz Kiefer
c5b11aa229 Fix DAML module filename in java codegen tests (#4003)
A mismatch between the module and the filename is already a warning
and will become an error in the future.

changelog_begin
changelog_end
2020-01-09 16:52:11 +00:00
Moritz Kiefer
a8b4a84b08
Move most of the remaining serializable types to stable LF packages (#3964)
* Move most of the remaining serializable types to stable LF packages

The only serializable types left in DAML stdlib after this PR are the
following:

- DA.Upgrade:MetaEquiv
- DA.Random:Minstd
- DA.Next.Set:Set
- DA.Next.Map:Map
- DA.Generics:MetaSel0
- DA.Generics:MetaData0
- DA.Generics:DecidedStrictness
- DA.Generics:SourceStrictness
- DA.Generics:SourceUnpackedness
- DA.Generics:Associativity
- DA.Generics:Infix0
- DA.Generics:Fixity
- DA.Generics:K1
- DA.Generics:Par1
- DA.Generics:U1
- DA.Internal.Prelude:Optional

Ignoring the Generics stuff which isn’t very urgent imho and the
Upgrade stuff which is probably going to change significantly anyway,
this leaves us with the weird Random module, the wrappers around
TextMap which will go away anyway and DA.Internal.Prelude:Optional
which shouldn’t exist in the first place (I’ll address that in a
separate PR).

CHANGELOG_BEGIN

- [DAML Compiler] Move more types from daml-stdlib to standalone LF
packages. The module names for the types have also changed
slightly. This only matters over the Ledger API when you specify the
module name explicitly. In DAML you should continue to use the
existing module names.

  - The types from ``DA.Semigroup` are now in a separate package under
  ``DA.Semigroup.Types``
  - The types from ``DA.Monoid` are now in a separate package under
  ``DA.Monoid.Types``
  - The types from ``DA.Time` are now in a separate package under
  ``DA.Time.Types``
  - The types from ``DA.Validation` are now in a separate package
  under ``DA.Validation.Types``
  - The types from ``DA.Logic` are now in a separate package under
  ``DA.Logic.Types``
  - The types from `DA.Date` are now in a separate package under
  `DA.Date.Types`.
  - The `Down` type from `DA.Internal.Prelude` is now in a separate
  package under `DA.Internal.Down`.

CHANGELOG_end

* Fix serializability of RelTime

* fix daml-docs

* Fix tests
2020-01-07 18:25:23 +01:00
Andreas Herrmann
f33e79c787
Remove unused dependencies to da_scala_library (#3938)
* Inline all scala_library dependencies

* Run //:buildifier-fix

* TMP scala_library_suite --> scala_library

* da_scala_library: Enable unused dependency checker

* scala_library: Enable unused dependency checker

* //daml-lf/data:data

* //daml-lf/engine:engine

* //ledger-api/rs-grpc-akka:rs-grpc-akka

* //ledger/participant-state:participant-state

* //ledger/ledger-api-client:ledger-api-client

* //scala-protoc-plugins/scala-logging:scala-logging-lib

* //ledger/ledger-api-scala-logging:ledger-api-scala-logging

* //ledger/ledger-api-common:ledger-api-common

* //ledger-service/utils:utils

* //ledger-service/jwt:jwt

* //ledger/ledger-api-auth:ledger-api-auth

* //extractor:extractor

* //daml-assistant/scala-daml-project-config:scala-daml-project-config

* //language-support/codegen-common:codegen-common

* //language-support/scala/codegen:codegen

* //language-support/codegen-main:codegen-main-lib

* //ledger-service/db-backend:db-backend

* //ledger-service/http-json:http-json

* //daml-lf/scenario-interpreter:scenario-interpreter

* //ledger/sandbox:sandbox

* //navigator/backend:navigator-library

* //daml-assistant/daml-sdk:sdk-lib

* //daml-lf/data-scalacheck:data-scalacheck

* //daml-script/test:test-lib

* //ledger/ledger-api-common:ledger-api-common-scala-tests-lib

* //ledger/test-common:test-common

* //ledger/sandbox:sandbox-scala-tests-lib

* //extractor:extractor-scala-tests-lib

* //language-support/java/bindings:bindings-java-tests-lib

* //language-support/java/bindings-rxjava:bindings-java-tests-lib

* //language-support/scala/bindings-akka-testing:bindings-akka-testing

* //language-support/scala/codegen-testing:codegen-testing

* //language-support/scala/codegen-sample-app:daml-lf-codegen-sample-app

* //language-support/scala/codegen-sample-app:daml-lf-codegen-sample-app-testing

* //language-support/scala/codegen-testing:codegen-testing-testing

* //ledger-api/sample-service:sample-service

* //ledger-api/rs-grpc-akka:rs-grpc-akka-tests-lib

* //ledger/ledger-api-test-tool:ledger-api-test-tool-lib

* //ledger/ledger-api-test-tool:ledger-api-test-tool-tests

* //ledger/participant-state/kvutils:kvutils

* //ledger/sandbox:ledger-api-server

* //ledger/sandbox-perf:sandbox-perf-lib

* //navigator/backend:navigator-tests-library

* UNDO scala_library_suite --> scala_library

This reverts commit ab3eb1ae23139e2ec431ab4551fbb0371e0354e1.

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-06 18:14:21 +01:00
Gary Verhaegen
878429e3bf
update copyright notices to 2020 (#3939)
copyright update 2020

* update template
* run script: `dade-copyright-headers update .`
* update script
* manual adjustments
* exclude frozen proto files from further header checks (by adding NO_AUTO_COPYRIGHT files)
2020-01-02 21:21:13 +01:00
Andreas Herrmann
9fbb787062 Remove unused dependencies to da_scala_test(_suite) (#3925)
* Remove unused scala.bzl imports

* override_targets org.scalatest.scalatest_2.12

Otherwise, rules_scala implicitly adds a different version to scala_test
than other packages transitively depending on scalatest. This causes
unused dependency checker to raise an error.

* Handle rules_scala scalatest in pom_file.bzl

* Inline all scala_test dependencies

So that `unused_dependency_checker = "error"` can be applied to them.

* Run //:buildifier-fix

* TMP scala_test_suite --> scala_test

* da_scala_test: Enable unused dependency checker

* //navigator/backend:navigator-scala-tests

* //ledger/sandbox:sandbox-scala-tests

* //ledger/participant-state/kvutils:kvutils-tests

* //ledger/participant-state:participant-state-tests

* //ledger/ledger-api-scala-logging:ledger-api-scala-logging-test

* //ledger/ledger-api-common:ledger-api-common-scala-tests

* //ledger/ledger-api-client:ledger-api-client-tests

* //ledger/ledger-api-auth:ledger-api-auth-scala-tests

* //ledger-service/lf-value-json:tests

* //ledger-service/jwt:tests

* //ledger-service/http-json:tests

* //ledger-api/rs-grpc-akka:rs-grpc-akka-tests

* //language-support/scala/codegen-sample-app:tests

* //language-support/scala/codegen-sample-app:ScalaCodeGenIT

* //language-support/scala/codegen:tests

* //language-support/scala/bindings-akka:tests

* //language-support/java/codegen:test

* //language-support/java/codegen:ledger-tests

* //language-support/java/bindings-rxjava:bindings-java-tests

* //language-support/codegen-common:test

* //extractor:extractor-scala-tests

* //daml-lf/scenario-interpreter:scenario-interpreter_tests

* //daml-lf/language:language-test

* //daml-lf/interface:tests

* //daml-lf/engine:tests

* //daml-lf/encoder:tests

* //daml-lf/archive:daml_lf_archive_reader_tests

* //daml-assistant/scala-daml-project-config:scala-daml-project-config-tests

* UNDO scala_test_suite --> scala_test

This reverts commit 13ed47ba725e944533ca1157a070cb8dc30569ac.

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2019-12-30 13:49:59 +00:00
Moritz Kiefer
c48b0ee289 Move Any wrappers and Archive to stable packages (#3890)
* Move Any wrappers and Archive to stable packages

There are no actual API or functionality changes in this PR but the
logic for locating the stable packages has slightly changed since the
Any wrappers package only makes sense for LF 1.7. To address this, we
simply filter out stable packages for newer LF versions since it
doesn’t make sense to depend on those anyway.

CHANGELOG_BEGIN

- [DAML Compiler] Move ``Archive`` type to a separate DALF.

CHANGELOG_END

* More comments

* Fix java codegen tests

* fix more tests

* Force cache reset on Windows

* Revert "Force cache reset on Windows"

This reverts commit 9f2b7d70b2.
2019-12-19 01:26:38 +00:00
Gary Verhaegen
2b3693bf4d
restore 0.13.38 logging defaults for daml jars (#3819)
CHANGELOG_BEGIN
- [Sandbox] Restore 0.13.38 logging behaviour.
- [Navigator] Restore 0.13.38 logging behaviour.
- [Extractor] Restore 0.13.38 logging behaviour.
- [Internals] As of 0.13.39, we merged a number of internal JAR files in
  the SDK tarball to reduce its size. These jars used to be standalone
  JARs you could invoke as e.g. ``java -jar sandbox.jar <args>``. As a
  result of merging the jars, they lost their individual ``logback.xml``
  configuration file. Although running the jars directly was (and is
  still) not supported, note that you can now achieve the same behaviour
  with e.g. ``java -Dlogback.configurationFile=sandbox-logback.xml -jar
  daml-sdk.jar sandbox <args>``.
CHANGELOG_END
2019-12-12 14:02:23 +01:00
Leonid Shlyapnikov
dcc7dc913f
Add choice result to the exercise response (#3757)
* Adding choice result to the exercise response, WIP

* Adding choice result to the exercise response, WIP

* Refactoring towards #3390,

ExerciseCommand argument does not always have to be a Record, changing
typearg: JsObject -> JsValue

* Cleanup

* exercise-with-result endpoint

* todo

* Switching /commands/exercise to use SubmitAndWaitForTransactionTree,

populating archived and created from TransactionTree

* removing debug println

* Fixing tests

* Removing `/command/exercise-with-result` endpoint

this one returns only exercise result, `/command/exercise` now returns
exercise result and events

* Updating docs

* Updating docs

CHANGELOG_BEGIN

- [JSON API - Experimental] Expose exercise result. Changed the output
of the ``/command/exercise``. Note ``exerciseResult`` and ``contracts``
in ``{"status":200,"result":{"exerciseResult": ...,"contracts":[...]}``.
See #3314

CHANGELOG_END
2019-12-10 10:09:24 -05:00
Moritz Kiefer
e769264ddf Move all datatypes out of daml-prim (#3791)
* Move all datatypes out of daml-prim

This moves the remaining two modules DA.Types and GHC.Tuple to
separate LF packages with stable identifiers.

The only data types remaining are the ones for typeclasses which will
disappear once we move this to type synonyms.

CHANGELOG_BEGIN

- [DAML Compiler] The modules DA.Types and GHC.Tuple from daml-prim
have been moved to separate packages.

CHANGELOG_END

* Fix codegen tests

* Fix DarReader test

* Fix kvutils tests

* Fix jdbcdao tests

* Fix hs ledger bindings tests
2019-12-10 09:19:16 +01:00
Stefano Baghino
6e17cdba90
Fix typo in JavaDoc (#3793) 2019-12-09 17:58:33 +01:00
Robert Autenrieth
bf2098f038
Check ledger and participant ID in claims (#3781)
* Add ledger and participant ID to claims

CHANGELOG_BEGIN
- [Ledger] AuthService implementations can now restrict the validity of access tokens to a single ledger or participant.
- [Sandbox] The sandbox JWT authentication now respects the ledgerId and participantId fields of the token payload.

CHANGELOG_END

* Add tests for ledger and participant in claims

* Address review comment

* Address review comment

* Fix tests

* Fix tests
2019-12-09 17:55:17 +01:00
Stefano Baghino
18824016c1 Ensure the access token is initialized when constructing a client (#3788)
* Ensure the access token is initialized when constructing a client

CHANGELOG_BEGIN
- [Java Client] Ensure the access token is initialized when using a
deprecated constructor.
CHANGELOG_END

* Improve phrasing and grammar
2019-12-09 14:14:45 +00:00
Gerolf Seitz
9d926d81b0 RxJava Bindings: Allow bot to run on a scheduler (#3404)
* RxJava Bindings: Allow bot to run on a scheduler

It seems that having many bots results in a some sort of deadlock
or blocking of data flow within the flowable network.

Adding some async boundaries to allow for concurrent processing
seems to help.

Fixes #2356

CHANGELOG_BEGIN
- [RxJava Bindings] Added a method to the ``Bot`` class allowing users to specify a ``Scheduler`` to use for running the bot. See `issue #2356 <https://github.com/digital-asset/daml/issues/2356>`__.
CHANGELOG_END

* Go easy, test!
2019-12-06 16:30:25 +00:00
Gerolf Seitz
658365e5eb Java codegen fixes (#3762)
* Fix compiler warnings in generated enums

* Fix warnings in generated equals method for parameterized types.

* Remove warning in equals for records without fields.

CHANGELOG_BEGIN
- [Java Bindings] Removed warnings in code emitted by the Java Codegen.
CHANGELOG_END

* fix compilation error in tests
2019-12-06 15:10:38 +00:00
Stefano Baghino
da9306414b Add authentication support to DamlLedgerClient (#3743)
Add documentation

CHANGELOG_BEGIN
- [Java Bindings] Added authentication support. See `issue #3626 <https://github.com/digital-asset/daml/issues/3626>`__.
CHANGELOG_END
2019-12-05 10:57:51 +00:00
Stefano Baghino
94aafd8121 Add authentication support to Java transaction client (#3733)
* Add authentication support to Java transaction client

* Address https://github.com/digital-asset/daml/pull/3733#discussion_r354182366
2019-12-05 10:17:30 +00:00
Moritz Kiefer
ba30e86911
Split wired-in modules into separate LF packages (#3696)
This is a first step towards making sure that the package ids for
types defined in daml-prim and daml-stdlib don’t change. This PR
mostly adds all the necessary infrastructure for that and moves
GHC.Types and GHC.Prim to make sure it works.

Until data-dependencies are really solid and we have verified that we
no longer have performance issues with an increasing number of Haskell
packages, we still include the source files in daml-prim and then just
rewrite the references.

We will also need to add tests that these packages really have stable
ids but I’ll leave that for separate PRs since this doesn’t make that
much sense anyway until all of the types have moved to stable
packages.

CHANGELOG_BEGIN

- [DAML Compiler] The modules GHC.Prim and GHC.Types from daml-prim
have been moved to separate packages.

CHANGELOG_END
2019-12-04 16:42:48 +01:00
Stefano Baghino
50d689d011 Optimize imports in //language-support/java/bindings-rxjava (#3728) 2019-12-04 11:08:13 +00:00
Stefano Baghino
ba0c5e8b30
Add support for authentication to Java command submission client (#3716)
* Add support for authentication to Java command submission client

* Fix compilation error
2019-12-04 11:31:50 +01:00
Stefano Baghino
36da75e1d6
Add authentication support to Java command completion client (#3713)
* Add authentication support to Java command completion client

* Address https://github.com/digital-asset/daml/pull/3713#discussion_r353629606
2019-12-04 11:31:21 +01:00
Stefano Baghino
8d5b4af0f2
Add support for authentication to Java time client (#3721) 2019-12-04 11:00:30 +01:00
Stefano Baghino
8eb5956f31
Add support for authentication to Java package client (#3720) 2019-12-04 10:42:48 +01:00
Stefano Baghino
0123c659f2 Add authentication support to CommandClient (#3666) 2019-11-29 07:20:39 +00:00
Stefano Baghino
9e9eb68e38 Make ActiveContractsClient work against auth ledgers (#3653)
* Make ActiveContractsClient work against auth ledgers

* Address review comments

- Address https://github.com/digital-asset/daml/pull/3653#discussion_r351667485
- Address https://github.com/digital-asset/daml/pull/3653#discussion_r351667588
2019-11-28 12:36:50 +00:00
Brian Healey
02186ef068 Ledger configuration indexing changes (rebased to master) (#3553)
* Add participant-state configuration protobuf

- Move the "DamlConfiguration" from kvutils into participant-state/protobuf/ledger_configuration.proto.
- Add version number and spec (ala transaction.proto)

This is a preparation for indexing the configuration and having one canonical serialization for it.

* Initial thoughts on indexing the configuration

* Implement indexing of ledger configuration changes

* Add record time to all Updates. Wire through participant id.

and rename V7__Add_configuration to V8.

* Add ledger_configuration_java_proto to release artifacts

* Fix up release of ledger_configuration_java_proto

* Suggestions from review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* address rebase issues

* Resolve compilation errors after rebase/merge

* happy formatting for scala and bazel and friends

* Drop "openWorld" setting from configuration

And refactor the tests to allocate parties explicitly.

* Fix up migration and tests

* Drop authorizedParticipantIds from configuration

Implement configuration authorization in kvutils using the previous
the participant id of the previous configuration.

* Post-rebase fixes

* Add missing migrations

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Add missing mavenUpload to ledger-api-server

* Remove stateUpdateRecordTime

* Address code review

- Address PR review
- Merge TimeModelImpl and the traits. Remove TimeModel from ledger-api-common.
- Throw `Err` from KeyValueConsumption on parse errors instead of assert/sys.error

* Reformat

* Add missing protobuf file

* Fix compilation after TimeModel changes. Add version logs to participant-state{,-kvutils}.

* Fix TestUtil.scala build

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Address review

- synchronized access to ledgerConfiguration in InMemoryLedger
- store rejection if configuration generation is wrong

* Update ledger/participant-state/protobuf/ledger_configuration.rst

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2019-11-27 17:41:23 +01:00
Remy
5b155409a0 DAML-LF iface reader: rename Map to TextMap (#3610)
CHANGELOG_BEGIN

 [DAM-LF Interface Reader]: **Rename** ``PrimTypeMap`` to ``PrimTypeTextMap`` and ``PrimType.Map`` to ``PrimType.TextMap``

CHANGELOG_END
2019-11-27 14:01:35 +00:00
Stefano Baghino
4d922e763b Add authentication to Java identity client (#3630)
* Add authentication to Java identity client

Contributes to #3626

No changelog added right now, it will be added with the PR that
effectively closes the ticket.

Moves `LedgerCallCredentials` to a shared package.

Establishes the test framework for other Java clients.

* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731

* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731

* Rebase and fix compilation errors

* Fix type of jar in artifacts.yaml

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Fix more errors emerged after rebase

* Add dependency for DAML assistant ITs

* Add missing dependency

* Fix wrong dependency

* fix release
2019-11-26 18:51:09 +00:00
Stefano Baghino
25f65f746b
Move DamlLedgerClient to builder-based constructors (#3620)
With the intent of adding more functionality to the Java bindings, in
particular authentication, this PR deprecates existing constructors in
favor of a more flexible builder based approach.

CHANGELOG_BEGIN

- [Java Bindings] Deprecated existing constructors for
``DamlLedgerClient``, please use the static ``newBuilder`` method to
instantiate a builder and use it to create the client, starting from
either a ``NettyChannelBuilder`` or a plain host/port pair.

CHANGELOG_END
2019-11-25 23:19:14 +01:00
Samir Talwar
1bd2ec5b1a grpc-utils: Simpler extractors. (#3594)
* grpc-utils: Simpler GrpcStatus extractors when you know the code.

And don't care about the description.

* grpc-utils: Simpler GrpcException extractors when you know the code.

And don't care about the description.

* Apply suggestions from code review

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Add `new` to `SpecificGrpc{Exception,Status}` constructor calls.

Also, don't blindly apply GitHub suggestions.
2019-11-22 21:09:17 +00:00
Remy
b506e27bde Java-codegen: add support for Generic Maps. (#3549)
* java-bindings: use LinkedHashMap asunderlying Map for DamlGenMap

* Address Samir's comments

* java-codegen: make DamlList, DamlMap and DamlGenMap immutable

* Address Samir's comments

* Adress Gerolf's comments

* more cleanup

* fix value generator

* Address Gerolf's comments:

CHANGELOG_BEGIN

[Java Bindings]
  * Rename ``DamlMap`` to ``DamlTextMap``
  * ``DamlCollectors`` class provides Collectors to build more easily ``DamlList`` and ``DamlTextMap``
  * Change the recommended method to convert ``DamlValue`` containers from/to Java Bindings containers.
    See `docs/source/app-dev/bindings-java/codegen.rst` for more details the new methodology.

CHANGELOG_END

* fix tests
2019-11-22 08:07:53 +00:00
Stefano Baghino
c51ecd3c58
Extractor with authentication (#3514)
* Make Extractor work against Ledger API servers protected by authentication

* Update changelog

* Address https://github.com/digital-asset/daml/pull/3514#pullrequestreview-318811500

- https://github.com/digital-asset/daml/pull/3514#discussion_r347753205
- https://github.com/digital-asset/daml/pull/3514#discussion_r347753629
- https://github.com/digital-asset/daml/pull/3514#discussion_r347754462

* Address https://github.com/digital-asset/daml/pull/3514#discussion_r347755484

* Address https://github.com/digital-asset/daml/pull/3514#discussion_r347752424

* Address https://github.com/digital-asset/daml/pull/3514#discussion_r347756328

* Fix docs

* Fix artifacts.yaml
2019-11-19 17:12:25 +01:00
Remy
027d63a787 java representation for value GenMap (#3515) 2019-11-19 08:53:15 +00:00
Remy
43eb69cbbd DAML-LF: add GenMap tests for LF encoder (#3491)
* daml-lf: fix decoder

* daml-lf: extend testing dar for encoder

* add EOF to .lf files

* change 1.dev to 1.7 for java codegen encoding test

* formatting

* fixes
2019-11-18 10:07:13 +00:00
Remy
4bd98b6cbc DAML-LF: add GenMap in iface reader (#3487)
* daml-lf: add GenMap in iface reader

* remove useless code

* Update language-support/scala/codegen/src/main/scala/com/digitalasset/codegen/lf/LFUtil.scala

Co-Authored-By: Stephen Compall <stephen.compall@daml.com>

* please restart CI
2019-11-17 08:12:10 +00:00