Commit Graph

164 Commits

Author SHA1 Message Date
fabiotudone-da
6e1fd55e45
Publish com.daml.ledger:participant-state-kvutils-app (#4480)
CHANGELOG_BEGIN
CHANGELOG_END
2020-02-12 09:45:52 +01:00
Samir Talwar
6fd402f634
Sandbox: Preliminary work to rebuild Sandbox on top of kvutils. (#4422)
* resources: Allow for startup exceptions without no stack trace.

* sandbox: Remove dependency duplication in the BUILD file.

* ledger-on-(memory|sql): Extract `Main` into its own file tree.

This is so we can remove the dependency on kvutils/app from the library.

logback.xml goes with it.

* sandbox: Implement "Sandbox Next" on top of ledger-on-sql.

CHANGELOG_BEGIN
- [Sandbox] Preliminary work to rebuild Sandbox on top of the DAML
  Ledger Integration Kit. Currently not exposed through the CLI.
CHANGELOG_END

* ledger-on-sql: Publish to Maven Central.

* sandbox: If an unknown database is passed to Sandbox Next, handle it.

* resources: Catch errors in owner construction or acquisition.

Sometimes we're not in a future yet.

* resources: Don't run ProgramResource error handling inside the EC.

Otherwise we end up in deadlock which is only broken by
`awaitTermination` timing out.
2020-02-06 17:34:17 +00:00
Moritz Kiefer
fc87953ed4
Enhanced completions for the assistant (#4420)
* Enhanced completions for the assistant

changelog_begin

- [DAML Assistant] The assistant can now do completions for SDK
  commands, e.g., ``daml ledger upl<TAB>`` will complete to ``daml
  ledger upload-dar``.

changelog_end

* Apply suggestions from code review

Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>

Co-authored-by: associahedron <231829+associahedron@users.noreply.github.com>
2020-02-06 15:16:12 +01:00
Gary Verhaegen
47bd131f15
add copyright headers to yml files (#4407)
We seem to have forgotten about them in the copyright scripts.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-06 12:54:07 +01:00
Moritz Kiefer
bd127d6c47
Flip the default on mavenUpload in artifacts.yaml (#4396)
changelog_begin
changelog_end
2020-02-05 10:32:02 +01:00
Moritz Kiefer
60e32a98c4
Disable release step on macos (#4330)
Originally we ran the release step on both Linux and MacOS to handle
platform dependent artifacts, in particular, damlc.jar. However, we
don’t have any platform dependent artifacts that are uploaded as part
of the release script anymore and I hope we will never have to add any
in the future.

So this PR, removes the code for handling platform dependent artifacts
in the release step and disables the release step on MacOS (while
still setting the variables like we do on Windows).

Currently the release step still costs us ~2 minutes on MacOS which is
already our slowest platform so hopefully this will speed things up a
bit.

changelog_begin
changelog_end
2020-01-31 17:08:47 +01:00
Robin Krom
eacacbf44e
fix release script (#4275)
* fix release script

only upload from the linux machine and escape arguments to `publish`,
otherwise they get interpreted by bazel.

CHANGELOG_BEGIN
CHANGELOG_END

* Update release/src/Main.hs

Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Update release/src/Main.hs

Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* bracket -> bracket_

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-01-30 10:51:39 +01:00
Robin Krom
ce9cd93519
publish npm packages as public (#4263)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-29 17:26:46 +01:00
Robin Krom
2e9fe6afb6
language: add daml-react package to ts libraries (#4259)
* language: add daml-react package to ts libraries

This adds the library formerly known as `daml-react-hook` into the
monorepo. We renamed it to `@daml/react`.

The tests sadly don't work with bazel right now because the local
imports aren't resolved correctly. Local testing with `yarn run test`
works as usual.

CHANGELOG_BEGIN
CHANGELOG_END

* address moritz comments

* get rid of DAVL mentions

* fix eslint warnings

* Update language-support/ts/daml-react/tsconfig.json

Co-Authored-By: Martin Huschenbett <martin.huschenbett@posteo.me>

Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
2020-01-29 15:37:47 +01:00
Shayne Fletcher
0e53ec0a2b Integrate daml2ts into the assistant (#4252)
changelog_begin
- Generate TypeScript bindings to DAML with `daml codegen ts DAR -o OUTDIR`
changelog_end
2020-01-28 22:03:54 +00:00
Shayne Fletcher
9c0fe75b94
Add daml2ts to the SDK release tarball (#4234)
changelog_begin
- daml2ts now included in the SDK
changelog_end
2020-01-27 13:18:05 -05:00
Moritz Kiefer
1a0fb138fb
Ignore missing maven deps on macos (#4201)
Currently, pretty much all of our builds are bottlenecked on
MacOS (mainly because the builders are slower and have worse
caching). The release step adds > 3min to each build which is a bit
annoying. It turns out that removing the calls to `bazel query` which
are used to check for missing Maven dependencies speeds this up by >
5x. Given that the check is platform independent anyway, we can just
disable it on MacOS.

changelog_begin
changelog_end
2020-01-24 16:17:08 +00:00
Robin Krom
811d740df5
upload typescript packages to npm registry (#4185)
* make packages public

This uploads the typescript npm packages of the language support to the
npm registry in the release process.

CHANGELOG_BEGIN
CHANGELOG_END

* address moritz/gary's review

* generate the .npmrc file

* adding debug output

just in case the upload will fail in the next release.

* reverse package order
2020-01-23 18:31:50 +01:00
Moritz Kiefer
2fb107b505 Remove damlc.jar (#4157)
* Remove damlc.jar

We never advertised or published this externally and our only internal
user has moved off this months ago already.

changelog_begin
changelog_end

* Remove dependency from navigator test lib on damlc jar
2020-01-22 16:44:27 +00:00
Samir Talwar
e2a5b26475
contextualized-logging: Move to its own Bazel package. (#4138)
I want to use this in _ledger/ledger-on-sql_, which doesn't depend
directly on Sandbox.

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-21 14:25:06 +01:00
Stefano Baghino
1ad7d6c24c
Replace NamedLoggerFactory (#4097)
* Replace NamedLoggerFactory

CHANGELOG_BEGIN
CHANGELOG_END

* Recover change lost in rebase

* Address https://github.com/digital-asset/daml/pull/4045#discussion_r366883577

* Address https://github.com/digital-asset/daml/pull/4045#discussion_r366885112

* Address two open review comments

Address https://github.com/digital-asset/daml/pull/4045#discussion_r366885448
Address https://github.com/digital-asset/daml/pull/4045#discussion_r366885981

* Address outstanding compilation errors

* Replace mocking with in-memory log collector

* Address https://github.com/digital-asset/daml/pull/4097#pullrequestreview-344639319

* Address https://github.com/digital-asset/daml/pull/4045#discussion_r366886250

The generation of code to have logging in the services has been replaced
by helpers classes. This will allow to enrich the context received at
construction by the service implementations.

* Use ContextualizedLogger for TrackerMap

* Remove deleted logging packages from artifacts.yaml

* Remove remaining deleted artifact from artifacts.yaml

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368640231

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368648027

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368652178

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368654996

* Annotate type of references to logErrorOnCall

* Address https://github.com/digital-asset/daml/pull/4097#discussion_r368869575
2020-01-21 10:57:28 +01:00
Samir Talwar
b96e54f640 libs-scala/resources: Move Resource and friends to their own package. (#4066)
* libs-scala/resources: Move `Resource` and friends to their own package.

CHANGELOG_BEGIN
- [Ledger API Server] Publish the resource management code as a library
  under ``com.digitalasset:resources``.
CHANGELOG_END

* libs-scala/resources: Add to the artifacts list.

* libs-scala/resources: Factor out the test helpers into test/lib.

* libs-scala/resources-akka: Make the Akka resources an optional dep.
2020-01-16 14:13:18 +00:00
Andreas Herrmann
6e25d10fe2
daml test-script (#3918)
* Start on daml test-scripts

* Run all `Script a` as test cases

* LedgerClient: Expose PackageManagementClient

To enable DAR uploads

* Upload the DAR to the ledger

* Start sandbox if no ledger specified

* Format daml test-script

* Fix deprecation warning on ActorMaterializer

* Add test-case //daml-script/tests:test_daml_script_test_runner

* Add daml test-script command

CHANGELOG_BEGIN

- [DAML Script - Experimental] Allow running DAML scripts as test-cases.
  Executing ``daml test-script --dar mydar.dar`` will execute all
  definitions matching the type ``Script a`` as test-cases.
  See `#3687 <https://github.com/digital-asset/daml/issues/3687>`__.

CHANGELOG_END

* daml-test-script enable logging

* Remove outdated TODO comment

* daml script-test More elaborate test-caseo

Compare to expected output and add failing test-case

* daml test-script Don't abort on test-failure

Before the test runner would abort on the first failed test-case. This
occasionally introduce additional test-failures if the sandbox was
torn down half-way through execution.

* ./fmt.sh

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-03 15:01:36 +01:00
Gary Verhaegen
878429e3bf
update copyright notices to 2020 (#3939)
copyright update 2020

* update template
* run script: `dade-copyright-headers update .`
* update script
* manual adjustments
* exclude frozen proto files from further header checks (by adding NO_AUTO_COPYRIGHT files)
2020-01-02 21:21:13 +01:00
Stefano Baghino
7020ed3b15
Moves CommandCompletionIT to Ledger API Test Tool (#3901)
* Moves CommandCompletionIT to Ledger API Test Tool

Closes #2841

Also refactors test observers into a single API and unifies the existing implementations of the DirectExecutionContext

* Allow explicit offsets to get checkpoints

* Raise checkpoint test timeout

* Consolidate checkpoint tests in a single test case

* Fix compilation issues

* Exclude command submission/completion tests for rev-v2

* Drop test not relevant for Ledger API implementations

* Fix compilation errors
2019-12-20 15:55:23 +01:00
Rohan Jacob-Rao
2b4a79c68f Tidy up release instructions (#3911)
* Tidy up first step of release instructions

* Minor adjustments to the rest of the instructions

* Fix

* Tweak
2019-12-20 11:12:12 +01:00
Gary Verhaegen
2b3693bf4d
restore 0.13.38 logging defaults for daml jars (#3819)
CHANGELOG_BEGIN
- [Sandbox] Restore 0.13.38 logging behaviour.
- [Navigator] Restore 0.13.38 logging behaviour.
- [Extractor] Restore 0.13.38 logging behaviour.
- [Internals] As of 0.13.39, we merged a number of internal JAR files in
  the SDK tarball to reduce its size. These jars used to be standalone
  JARs you could invoke as e.g. ``java -jar sandbox.jar <args>``. As a
  result of merging the jars, they lost their individual ``logback.xml``
  configuration file. Although running the jars directly was (and is
  still) not supported, note that you can now achieve the same behaviour
  with e.g. ``java -Dlogback.configurationFile=sandbox-logback.xml -jar
  daml-sdk.jar sandbox <args>``.
CHANGELOG_END
2019-12-12 14:02:23 +01:00
Remy
0107228c49
daml-lf: distribute daml_lf_1.7_archive_java_proto (#3792) 2019-12-10 12:24:55 +01:00
Samir Talwar
639eba2c17 ledger-api-scala-logging: Fix errors in IntelliJ IDEA. (#3785)
* ledger-api-scala-logging: Fix errors in IntelliJ IDEA.

The Bazel plugin for IntelliJ doesn't seem to be smart enough to be able
to handle a Scala library that is part `src` directory and part
generated code from another Bazel rule. It just ignores the second part.
This means that IntelliJ cannot find the *ServiceLogging classes, as
they're not represented on the Bazel-generated classpath, and so
complains with lots of errors when working on the equivalent Api*Service
files.

To fix this, we can split these in two, compiling the base traits to
`ledger-api-scala-logging-base` and then the generated code separately.

It does result in an extra Bazel dependency for the users of
ledger-api-scala-logging, as Bazel doesn't realise transitive
dependencies for us.

* Release: Add `ledger-api-scala-logging-base` to the Maven list.
2019-12-09 16:02:59 +00:00
Moritz Kiefer
ab59376cb3 Fix Sdk-Version field in daml-sdk-head (#3759)
In #3706 we fixed the SDK versions of the daml-trigger an daml-script
libraries and (rightfully) stopped filtering out 0.0.0 from the SDK
version check in `damlc build`. However, this broke daml-sdk-head
since we still distributed the DARs in daml-sdk-head with the
Sdk-Version set to whatever the current release is rather than 0.0.0.
2019-12-06 10:08:37 +00:00
Moritz Kiefer
022c60ccb3 Publish the thin trigger library instead of the fat jar (#3739)
This still contains the main class so you can use it like you would
use the fat jar but publishing fat jars to maven central is apparently
bad practise and some peple have asked for the library.

This includes some slight tweaks to the scala_docs rule to make it
capable of coping with the generated source file and a hack in the
release script to avoid it complaining about the scenario proto
library not being published to Maven even though it is included in the
transitive deps.
2019-12-05 13:04:20 +00:00
Stefano Baghino
5254520b80 Spin off TokenHolder into a new library (#3741)
* Spin off TokenHolder into a new library

Avoids having weird dependencies between different packages, makes TokenHolder available on Maven

* Fix auth-utils path
2019-12-05 12:31:14 +00:00
Moritz Kiefer
08b98602b1
Build SDK head tarball directly (#3723)
Previously, we constructed the tarball used by daml-sdk-head by first
building the regular SDK release tarball only to the extract it, patch
the version file and recompress it. This adds about 30-60s to every
invocation of daml-sdk-head.

This PR changes this by factoring out the logic from building the
release tarball into a macro that we instantiate twice, once with the
proper version file and once with a dummy HEAD version file set to
0.0.0.

This does change the format of the sdk-head-tarball slightly, in
particular the files are now located under sdk-0.0.0 instead of
sdk-head. However, this doesn’t matter anyway afaik and I think the
new format makes more sense anyway considering that the regular
release tarballs have something like sdk-0.13.38.
2019-12-04 13:04:15 +01:00
Samir Talwar
5dd38d54e8 sandbox: PostgreSQL health checks. (#3655)
* ledger-api-test-tool: Increase the duration when watching health.

This should hopefully stop CI from flaking out.

* reference-v2/sandbox: Avoid unnecessary companion object constructors.

I like indirection… when it does something.

* ledger: Propagate empty health checks throughout the services.

* reference: Remove duplication from the ReferenceServer object.

* ledger-api-common: Actually query a "reporter" in the health service.

* ledger-api-common: Report health per-component when required.

* ledger-api-health: Use a Map to represent components for health checks.

* sandbox: Fix warnings in SqlLedgerSpec.

* ledger-api-common: Throw GrpcHealthService errors inside the Future.

* ledger: Implement health checks against the PostgreSQL connection.

Without proper testing, because I am not great at this.

* sandbox: Remove duplication and fix warnings in PostgresAround.

* sandbox: Test the SQL Ledger's health reporting on failure.

* sandbox: Don't report as unhealthy until 3 connections fail.

* ledger-api-health: Remove unused parts of the API.

Bit of premature design there.

* sandbox: Rename the "ledger" health check to "write".

* participant-state: Add the ReportsHealth trait to ReadService.

* ledger-api-common: `Future.fromTry(Try(…))` -> `Future(…)`.

* ledger-api-common: Make it clearer that StubReporter closes over health.

* ledger-api-common: Explain the HealthService watch tests with comments.

* sandbox: Clean up SqlLedger a bit.

* sandbox: Don't try and stop PostgreSQL twice in PostgresAround.

* bazel_tools: Windows rlocation lookups need to be with forward slashes.

* release: Fix case of "true".

* ledger-api-common: Make `GrpcHealthService::matchResponse` return a Try.

* ledger-api-common: Make `GrpcHealthServiceSpec` async.

* sandbox: Make a couple of DB classes final.

* sandbox: Avoid importing `X._` in PostgresAround.

* sandbox: Add clues to the SqlLedgerSpec's multiple assertions.

* sandbox: If PostgreSQL doesn't come back up, keep retrying.

* sandbox: Remove duplication in SqlLedgerSpec.

* sandbox: In SqlLedgerSpec, actually wait for the health to change.

* sandbox: In PostgresAround, make stopping PostgreSQL idempotent.

* sandbox: Simplify the SqlLedgerSpec to make it work on CI.

It's worth a shot.

* ledger-api-common: Simplify the GrpcHealthServiceSpec a little.

And add a changelog.

CHANGELOG_BEGIN

- [Ledger API Server] Add a health check endpoint conforming to the
  `GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
- [Ledger API Server] Add health checks for index database connectivity.
- [Participant State API] Add a mandatory ``currentHealth()`` method to
  ``IndexService``, ``ReadService`` and ``WriteService``.

CHANGELOG_END

* sandbox: Improve the Javadoc layout for DbDispatcher.

* sandbox: Capitalize constants in SqlExecutor.

* ledger-api-health: Convert HealthStatus to an abstract class.
2019-11-29 15:07:43 +00:00
associahedron
2713d6978d
Reduce SDK tarball size by 70% by deduplicating Scala dependencies. (#3678)
* Start moving all the jars into a single jar.

* Fold navigator into daml-sdk jar

* include sandbox

* Remove unnecessary compileDeps.

CHANGELOG_BEGIN

- [DAML SDK] Reduced the size of the DAML SDK by about
60% uncompressed, 70% compressed, by deduplicating Scala
dependencies.

CHANGELOG_END

* update copyright header

* buildifier fix
2019-11-29 13:26:47 +00:00
Oliver Seeliger
f11a3ce9a7
Release triggers to maven central (#3674) 2019-11-29 10:25:02 +01:00
Brian Healey
02186ef068 Ledger configuration indexing changes (rebased to master) (#3553)
* Add participant-state configuration protobuf

- Move the "DamlConfiguration" from kvutils into participant-state/protobuf/ledger_configuration.proto.
- Add version number and spec (ala transaction.proto)

This is a preparation for indexing the configuration and having one canonical serialization for it.

* Initial thoughts on indexing the configuration

* Implement indexing of ledger configuration changes

* Add record time to all Updates. Wire through participant id.

and rename V7__Add_configuration to V8.

* Add ledger_configuration_java_proto to release artifacts

* Fix up release of ledger_configuration_java_proto

* Suggestions from review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* address rebase issues

* Resolve compilation errors after rebase/merge

* happy formatting for scala and bazel and friends

* Drop "openWorld" setting from configuration

And refactor the tests to allocate parties explicitly.

* Fix up migration and tests

* Drop authorizedParticipantIds from configuration

Implement configuration authorization in kvutils using the previous
the participant id of the previous configuration.

* Post-rebase fixes

* Add missing migrations

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Add missing mavenUpload to ledger-api-server

* Remove stateUpdateRecordTime

* Address code review

- Address PR review
- Merge TimeModelImpl and the traits. Remove TimeModel from ledger-api-common.
- Throw `Err` from KeyValueConsumption on parse errors instead of assert/sys.error

* Reformat

* Add missing protobuf file

* Fix compilation after TimeModel changes. Add version logs to participant-state{,-kvutils}.

* Fix TestUtil.scala build

* Apply suggestions from code review

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Address review

- synchronized access to ledgerConfiguration in InMemoryLedger
- store rejection if configuration generation is wrong

* Update ledger/participant-state/protobuf/ledger_configuration.rst

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2019-11-27 17:41:23 +01:00
Stefano Baghino
4d922e763b Add authentication to Java identity client (#3630)
* Add authentication to Java identity client

Contributes to #3626

No changelog added right now, it will be added with the PR that
effectively closes the ticket.

Moves `LedgerCallCredentials` to a shared package.

Establishes the test framework for other Java clients.

* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731

* Address https://github.com/digital-asset/daml/pull/3630#discussion_r350663731

* Rebase and fix compilation errors

* Fix type of jar in artifacts.yaml

Co-Authored-By: Gerolf Seitz <gerolf.seitz@digitalasset.com>

* Fix more errors emerged after rebase

* Add dependency for DAML assistant ITs

* Add missing dependency

* Fix wrong dependency

* fix release
2019-11-26 18:51:09 +00:00
Samir Talwar
0ff716df2a Ledger API: Add healthcheck endpoints. (#3573)
* grpc-definitions: Delete health_service.proto

We can use the version in io.grpc:grpc-services instead.

* ledger: Delete ledger/API.md.

* sandbox: Fix warnings in ApiServices flagged by IntelliJ.

* sandbox: Implement a dummy grpc.health.v1.Health.Check endpoint.

* sandbox: Implement a dummy grpc.health.v1.Health.Watch endpoint.

* sandbox: Drop repeated elements from grpc.health.v1.Health.Watch.

* sandbox: Wrap the HealthService in basic tests.

* sandbox: Stop streaming the server health too.

* ledger-api-test-tool: Health check tests.

* Add a changelog entry for the health check endpoints.

CHANGELOG_BEGIN

- [Ledger API] Add healthcheck endpoints, conforming to the
  `GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
  It is always ``SERVING`` for now.

- [DAML Ledger Integration Kit] Add conformance test coverage for the
  ``grpc.health.v1.Health`` service.

CHANGELOG_END

* ledger-api-integration-tests: Increment the number of services.

* Apply suggestions from code review

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* sandbox: Use `AkkaBeforeAndAfterAll` in the HealthServiceSpec.

In an attempt to get it working on CI.

* sandbox: Change `dropRepeated` to `DropRepeated()`.

Keep it in one file.

* test-common: Use `Delayed.by` in `TimeBoundObserver`.

* test-common: Close the source when `TimeBoundObserver` completes.

* ./fmt.sh

That'll teach me not to `--no-verify` just because it's a merge commit.

* sandbox: Inline `HealthService.suppress`.

At some point it was being used twice.

* sandbox: Increase the timeout for HealthServiceSpec.

* sandbox: Reimplement HealthService using the Scala protobuf types.

* sandbox: Generate an Akka-compatible trait for the health service.

And refactor a lot of test code to make it easy to test.

* ledger-api-common: Move the HealthService here.

* rs-grpc-testing-utils: Publish to Maven.

* rs-grpc-testing-utils: Add Maven coordinates.
2019-11-22 14:02:05 +00:00
Stefano Baghino
84e6064d51 Remove unreleased.rst (#3547)
* Start working on getting rid of unreleased.rst

Document new process in CONTRIBUTING.md,
.github/pull_request_template.md and unreleased.rst (for good measure)

Report the previous changelog additions here so that they're not lost in
the mist of times.

CHANGELOG_BEGIN

- [DAML Stdlib] Added the ``NumericScale`` typeclass, which improves the type inference for Numeric literals, and helps catch the creation of out-of-bound Numerics earlier in the compilation process.

- [DAML Triggers] ``emitCommands`` now accepts an additional argument
  that allows you to mark contracts as pending. Those contracts will
  be automatically filtered from the result of ``getContracts`` until
  we receive the corresponding completion/transaction.

- [Navigator] Fixed a bug where Navigator becomes unresponsive if the ledger does not contain any DAML packages.

- [Ledger-API] Add field ``gen_map`` in Protobuf definition for ledger
  api values. This field is used to support generic maps, an new
  feature currently in development.  See issue
  https://github.com/digital-asset/daml/pull/3356 for more details
  about generic maps.

  The Ledger API will send no messages where this field is set, when
  using a stable version of DAML-LF.  However the addition of this
  field may cause pattern-matching exhaustive warnings in the code of
  ledger API clients. Those warnings can be safely ignored until
  GenMap is made stable in an upcoming version of DAML-LF.

- [JSON API - Experimental] CLI configuration to enable serving static content as part of the JSON API daemon:
  ``--static-content "directory=/full/path,prefix=static"``
  This configuration is NOT recommended for production deployment. See issue #2782.

- [Extractor] The app can now work against a Ledger API server that requires client authentication. See `issue #3157 <https://github.com/digital-asset/daml/issues/3157>`__.
- [DAML Script] This release contains a first version of an experimental DAML script
   feature that provides a scenario-like API that is run against an actual ledger.
- [DAML Compiler] The default DAML-LF version is now 1.7. You can
  still produce DAML-LF 1.6 by passing ``--target=1.6`` to ``daml
  build``.

- [JSON API - Experimental] The database schema has changed; if using
  ``--query-store-jdbc-config``, you must rebuild the database by adding
  ``,createSchema=true``.
  See `issue #3461 <https://github.com/digital-asset/daml/pull/3461>`_.

- [JSON API - Experimental] Terminate process immediately after creating schema. See issue #3386.

- [DAML Stdlib] ``fromAnyChoice`` and ``fromAnyContractKey`` now take
  the template type into account.

CHANGELOG_END

* Document new release process to gather changelog additions

* Change the release script to ignore unreleased.rst

* Remove spurious unreleased.rst lines

* Transition to use tags

* Document new way to get changelog additions with tags

* Update release/RELEASE.md

Co-Authored-By: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Address https://github.com/digital-asset/daml/pull/3547#discussion_r348438786

* Document correction process

* Add copyright header to unreleased.sh

* Update CONTRIBUTING.md

Co-Authored-By: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Modify CONTRIBUTING.md after @garyverhaegen-da's proposal

* Make unreleased.sh run per commit and treat tags as case-insensitive

* Fix documentation for replacements
2019-11-20 15:16:57 +00:00
Moritz Kiefer
7deca903bb Add documentation for DAML script and bundle it in the SDK (#3527) 2019-11-19 17:19:57 +00:00
Stefano Baghino
c51ecd3c58
Extractor with authentication (#3514)
* Make Extractor work against Ledger API servers protected by authentication

* Update changelog

* Address https://github.com/digital-asset/daml/pull/3514#pullrequestreview-318811500

- https://github.com/digital-asset/daml/pull/3514#discussion_r347753205
- https://github.com/digital-asset/daml/pull/3514#discussion_r347753629
- https://github.com/digital-asset/daml/pull/3514#discussion_r347754462

* Address https://github.com/digital-asset/daml/pull/3514#discussion_r347755484

* Address https://github.com/digital-asset/daml/pull/3514#discussion_r347752424

* Address https://github.com/digital-asset/daml/pull/3514#discussion_r347756328

* Fix docs

* Fix artifacts.yaml
2019-11-19 17:12:25 +01:00
Moritz Kiefer
e4632f5fb2 Include "daml trigger" in "daml --help" (#3528) 2019-11-19 16:19:17 +01:00
Oliver Seeliger
f50901347f
Release lf scalacheck test jars to maven central (#3519)
as they are dependencies used downstream
2019-11-19 13:10:28 +01:00
Gerolf Seitz
5f8bf411bd
Upload proto and deploy jars to maven (#3507)
* Add helper to produce an empty zip files.

This is used to generate empty sources and javadoc jars for
deploy jars later on.

* Create empty auxilliary jars.

da_java_binary:
  - empty javadoc jar
  - empty sources jar

da_java_proto_library:
  - empty javadoc jar
  - the sources jar is automatically generated by java_proto_library as a side effect

da_scala_binary:
  - empty javadoc jar
  - empty sources jar

* Support maven upload for jar-deploy and jar-proto

For jar-deploy targets we don't check for internal dependencies,
because these should already be contained in the (fat-)jar itself.

Additionally, the release program now uploads javadocs and sources
for jar-proto and jar-deploy as well to comply with maven central.

* Upload ledger-api-test-tool and kvutils + dependencies to maven central.

This is the diff running the output of the release without and with these changes.
A few artifacts now also get their javadoc and sources uploaded (mostly to bintray,
but now they are ready for a maven central upload).

ledger-api-test-tool has the scala version removed from the artifact as it is a
deploy jar and nobody should care which specific scala version is used.

Only in release/com/daml/ledger/participant-state-kvutils-java-proto/100.13.35: participant-state-kvutils-java-proto-100.13.35-javadoc.jar
Only in release/com/daml/ledger/participant-state-kvutils-java-proto/100.13.35: participant-state-kvutils-java-proto-100.13.35-sources.jar
Only in release/com/daml/ledger/testtool: ledger-api-test-tool
Only in release-before/com/daml/ledger/testtool: ledger-api-test-tool_2.12
Only in release/com/digitalasset/daml/lf/engine/trigger/runner_2.12/100.13.35: runner_2.12-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml/lf/engine/trigger/runner_2.12/100.13.35: runner_2.12-100.13.35-sources.jar
Only in release/com/digitalasset/daml-lf-blindinginfo-java-proto/100.13.35: daml-lf-blindinginfo-java-proto-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml-lf-blindinginfo-java-proto/100.13.35: daml-lf-blindinginfo-java-proto-100.13.35-sources.jar
Only in release/com/digitalasset/daml-lf-transaction-java-proto/100.13.35: daml-lf-transaction-java-proto-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml-lf-transaction-java-proto/100.13.35: daml-lf-transaction-java-proto-100.13.35-sources.jar
Only in release/com/digitalasset/daml-lf-value-java-proto/100.13.35: daml-lf-value-java-proto-100.13.35-javadoc.jar
Only in release/com/digitalasset/daml-lf-value-java-proto/100.13.35: daml-lf-value-java-proto-100.13.35-sources.jar
Only in release/com/digitalasset/damlc/100.13.35: damlc-100.13.35-javadoc.jar
Only in release/com/digitalasset/damlc/100.13.35: damlc-100.13.35-sources.jar
Only in release/com/digitalasset/extractor/100.13.35: extractor-100.13.35-javadoc.jar
Only in release/com/digitalasset/extractor/100.13.35: extractor-100.13.35-sources.jar
Only in release/com/digitalasset/ledger-service/http-json-deploy/100.13.35: http-json-deploy-100.13.35-javadoc.jar
Only in release/com/digitalasset/ledger-service/http-json-deploy/100.13.35: http-json-deploy-100.13.35-sources.jar
Only in release/com/digitalasset/navigator/100.13.35: navigator-100.13.35-javadoc.jar
Only in release/com/digitalasset/navigator/100.13.35: navigator-100.13.35-sources.jar
2019-11-18 14:40:15 +01:00
Gary Verhaegen
8a024c549f release instructions: add label to PR (#3489) 2019-11-15 15:55:36 +00:00
Gerolf Seitz
172ce90597
Upload codegen fat-jars to maven central (#3476) 2019-11-14 18:40:54 +01:00
Gerolf Seitz
e760d4749f Check internal dependencies of maven artifacts (#3454)
* Check internal dependencies of maven artifacts

All artifacts that are uploaded to maven need to also have
their dependencies uploaded to maven.

* inline tuple matching

* remove unnecessary parenthesis

* mark missing artifacts for maven upload
2019-11-14 14:18:54 +01:00
Andreas Herrmann
33e47828e3
Bazel 1.1 (#3249)
* bazel: 0.28.1 --> 1.1.0

* bazel-watcher sha256

* Fix missing line in patch

* proto_source_root --> strip_import_prefix

See https://github.com/bazelbuild/bazel/issues/7153 for details.

* Update rules_nixpkgs

Required to avoid errors of the form
```
ERROR: An error occurred during the fetch of repository 'node_nix':
   parameter 'sep' may not be specified by name, for call to method split(sep, maxsplit = None) of 'string'
```

and
```
ERROR: An error occurred during the fetch of repository 'node_nix':
   Traceback (most recent call last):
	File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
		_execute_or_fail(repository_ctx, <3 more arguments>)
	File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
		fail(<1 more arguments>)

Cannot build Nix attribute 'nodejs'.
Command: [/Users/runner/.nix-profile/bin/nix-build, /private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/node_nix/nix/bazel.nix, "-A", "nodejs", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
src/main/tools/process-tools.cc:173: "setitimer": Invalid argument
```

* Update rules_scala

* .proto has been removed, use [ProtoInfo] instead

See
https://docs.bazel.build/versions/1.1.0/be/protocol-buffer.html#proto_library

* python3_nix add nix_file attribute

To avoid the following error

```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/BUILD:66:1: //:nix_python3_runtime depends on @python3_nix//:bin/python in repository @python3_nix which failed to fetch. no such package '@python3_nix//': Traceback (most recent call last):
        File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
                _execute_or_fail(repository_ctx, <3 more arguments>)
        File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
                fail(<1 more arguments>)

Cannot build Nix attribute 'python3'.
Command: [/home/aj/.nix-profile/bin/nix-build, "-E", "import <nixpkgs> { config = {}; overlays = []; }", "-A", "python3", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
error: anonymous function at /home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs.nix:3:1 called with unexpected argument 'config', at (string):1:1
```

* rules_haskell unnamed string.split(_, maxsplit = _)

The keyword argument may no longer be named.

* string.replace(_, _, maxsplit = _) may not be named

* Move proto sources from deps to data

Fixes

```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation
```

* Define sha256 for haskell_ghc__paths

Bazel 1.1.0 fails on missing hashes.

* Disable --incompatible_windows_native_test_wrapper

* //compiler/daml-extension don't modify sources

Modifying sources in-place can cause issues on Windows, where build
actions are not sandboxed and changes on sources can affect other build
steps.

* bazel-genfiles --> bazel-bin

The bazel-genfiles symlink has been removed since Bazel 1.0.
See https://github.com/bazelbuild/bazel/issues/8651

* Mark dev_env_tool repository rule as configure

See
https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule

* Move data deps into data attribute

* Mark dev_env_tool as local = True

* Manually fetch @makensis_dev_env
2019-11-11 10:06:03 +01:00
Remy
ce36674ea0 daml-lf: clone dev protobuf for LF 1.7 (#3371) 2019-11-07 12:32:38 +00:00
Stefano Baghino
a2725415eb Add artifacts publishing for //ledger/ledger-api-grpc-utils (#3370)
* Add artifacts publishing for //ledger/ledger-api-grpc-utils

* Address https://github.com/digital-asset/daml/pull/3370#discussion_r343509550
2019-11-07 08:13:18 +00:00
Stephen Compall
d0ec381925 json-api: store contracts similar to Extractor's single-table (#3192)
* new library ledger-service/db-backend

* borrow contracts table schema from extractor

* borrow contract insertion, removing some data to be unused

* match contract schema with insert function

* factor insertContract arguments

* offset table declarations

* CLI argument for query store

* surrogate template IDs

* compute surrogate template IDs on-the-fly

* database init action

* incoherent typeclasses, eh

* newtype SurrogateTpId

* offset fetch/update functions

* bad sql

* bulk insert contracts, function for selecting contracts

* expose contract column name for query's usage

* Initializing DB on startup if configured

* dropping existing tables as part of initialization

* fix some query syntax errors

* createSchema flag

* function for streaming transactions with jwt party selected

* formatting

* usage

* collect acs contracts and the ledger offset at the end

* lastOffset

* fixing merge conflicts, updating the way 3rd party deps are specified

* Moving ContractDao into http-json module

so it can take domain AST as an input

* cleanup

* injecting new dependencies

* split transaction batches into inserts and deletes

* generate sql for deleting contracts

* `fetch_sources = True` for java_deps

* make the delete-constructed fragment more efficient; handle empty list here

* pass logHandler for insertContracts

* ContractDao returns ConnectionIO, it's up to the caller to wrap query into a transaction

* fixing typo

* minor cleanup, moving fromLedgerApi factory function into corresponding companion objects

* don't need it any more

* GetActiveContractsResponse => domain.Contract factory

* make concatFragment private

* add partition graph; move other contract-fetching experiments to ContractsFetch

* experimenting with akka sources

* introducing domain.Offset to work around API's empty/null offset cases

* minor cleanup

* decompose fetchActiveContractsFromOffset

* missed via

* ACS splitting graph

* finish doc for ACS splitting graph

* remove unneeded stages

* WIP

* lazily read a stream of ConnectionIO into a single ConnectionIO

* cancel on IO error

* figuring out how to put all the pieces together

* graph WIP

* Removing workflowId from the JSON API

* simplify acsAndBoundary; describe other flow pieces

* WIP

* use Vector in InsertDeleteStep; add variant for ACS (no deletes)

* `org.wartremover.warts.NonUnitStatements` enforced in `http-json` module

* evaluate InsertDeleteStep to a ConnectionIO

* database variant of LfValueCodec, using numbers for numbers

* convert input to JSON, combine insert plans, connect rest of contractsToOffsetIo

* remove strict contractsToOffset sink

* moving dao methods into an object

* putting pieces together

* contractsFromOffset WIP

* should be it

* cleanup

* cleanup

* contractsIo that takes List[domain.TemplateId.RequiredPkg])

* contractsIo that takes List[domain.TemplateId.RequiredPkg])

* cleanup

* put all pieces together, testing

something does not work yet

* diff is not required to return anything

that is why Sink.lastOption that gives Option[domain.Offset]

* factor out tuple split

* use traverse syntax in contractsIo2

* factor explicit flow steps out of graph DSL; remove aggregate

* locally model the Absolute/Begin distinction for offset bookmarking in DB

* Adding test cases to run HTTP Service with Postgres backend

the same set of test cases, run with and without DB backend

* make better use of domain.Offset in OffsetBoundary

* monomorphize InsertDeleteStep#append

* Disabling a test that fails with DB backend

* add release note

* add release note about workflowId

* a test case that checks the number or stored contracts

* trying to figure out why Postgres test fails on Windows with NPE
2019-11-06 13:53:16 -05:00
Jussi Mäki
0ca8580914 Fix the import paths of {value,transaction,blindinginfo}.proto (#3312)
* Fix the import paths of {value,transaction,blindinginfo}.proto

This cleans the imports in daml_kvutils.proto:

  -import "daml-lf/transaction/src/main/protobuf/transaction.proto";
  +import "com/digitalasset/daml/lf/transaction.proto";

This shouldn't be a breaking change to the outside as the proto files
themselves are not published, only the java libraries generated from them.

* Update missed deps
2019-11-01 14:54:55 +00:00
Robert Autenrieth
e887318f4f Move AuthService (#3272)
* Move AuthService and Claims

* Move interceptor and authorization wrappers

* Add artifact

* Address review comments
2019-10-29 15:46:43 +00:00
Moritz Kiefer
6c36777a8e
Release DAML trigger runner to Bintray (#3243)
This should make it a bit easier to use in a managed environment and
has been requested by some users.
2019-10-21 18:14:46 -04:00