Commit Graph

439 Commits

Author SHA1 Message Date
Stefano Baghino
3fa05804bf
Make non-repudiation integration exclusive to enterprise edition (#9219)
* Make non-repudiation integration exclusive to enterprise edition

Closes #9182

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/9219#discussion_r599578614

* Address https://github.com/digital-asset/daml/pull/9219#pullrequestreview-618605511

* Add aliases for CE test targets, thanks to @S11001001 for the feedback

* Uh, turns out you need a main...

* Address https://github.com/digital-asset/daml/pull/9219#discussion_r599717526

* Add missing dependency for Oracle
2021-03-24 20:25:47 +01:00
Moritz Kiefer
a93f89b6f8
Factor out an Oracle test fixture (#9191)
* Factor out an Oracle test fixture

We need to add Oracle support to other components like the trigger
service as well. At that point, we cannot assume that things don’t
stomp on each other so this PR adds a fixture that generates a random
user (which also comes with its own schema, creating new databases is
a bit different in oracle) which we use testing.

changelog_begin
changelog_end

* Update libs-scala/oracle-testing/src/main/scala/testing/oracle/OracleAround.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Less top-level stuff in traits

changelog_begin
changelog_end

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2021-03-19 16:37:36 +01:00
Hubert Slojewski
51f495e758
KVL-203 Share participant state integration test harness (#9143)
* Expose libraries for integration testing purposes

The motivation of these changes is to eliminate manual work and reduce duplication between the SDK and oem-integration-kit repos by reusing the same test fixture for integration testing participant state implementations. Also, the DARs required for running these tests won't need to be manually updated.

CHANGELOG_BEGIN
CHANGELOG_END

* Fix a concurrency issue in integration tests

* Fix Bazel error

* Fix conflict resolution

* Move inline daml-lf to separate dar files

* Add a comment

* Add a missing artifact

* Extract method

* Remove maven tags

* Add a macro for Scala libraries with dar resources

* Improve the macro

* Add missing artifact

* Simplify the tests

* Format signature

* Fix the maven tag

* Add missing copyright headers

* Format bazel files

* Make //ledger/test-common lf version dependent (to avoid jar hell)

* Move da_scala_dar_resources_library to a separate bzl file

* Add missing artifacts

Co-authored-by: Hubert Slojewski <hubert.slojewski@tesco.com>
2021-03-19 16:29:02 +01:00
Stefano Baghino
f0c8b1240d
Integrate non-repudiation into HTTP JSON API (#9180)
* Integrate non-repudiation into HTTP JSON API

changelog_begin
changelog_end

Closes #9094

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596400828

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596408663

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596402533

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596624708

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596624174

* Solve compilation error in daml-script

* Address https://github.com/digital-asset/daml/pull/9180#issuecomment-801875959

changelog_begin
- [JSON API] The JAR for the HTTP JSON API is no longer published on
Maven, use the fat JAR as indicated in the documentation.
changelog_end
2021-03-18 15:15:52 +00:00
nickchapman-da
9b2158508b
Add new variant to Value.scala for builtin-exceptions. (#9084)
* Add new variant to Value.scala for builtin-exceptions.

final case class ValueBuiltinException[+Cid](tag: String, value: Value[Cid]) extends Value[Cid]

And push through the code consequences.
Most places fixed up.
A couple more things to do in this PR (marked NICK)
A couple of things which can be left for later (marked 8020)

fix build

fix another scala match

changelog_begin
changelog_end

* fix any match

* add marker of code which needs attending to in the PR

* extend ledger-api value.proto & fix LfEngineToApi

* undo/comment-out the change to value.proto

* add tests in HashSpec for BuiltinException

* code but dont yet enable value-gen for builtin exceptions

* address comments which suggest we crash in various places

* support BuiltinException in scenario_service.proto

* one more TODO 8020 tag
2021-03-16 13:15:33 +00:00
Gary Verhaegen
1df2270cc9
typo (#9145)
changelog_begin
changelog_end
2021-03-16 11:30:57 +01:00
Moritz Kiefer
8e25227cbb
Fix heartbeating in the JSON API when starting from offset (#9104)
We filter heartbeat ticks until we get the first step message. This is
when starting from the ACS but it is incorrect when starting from an
existing offset where this results in us not emitting heartbeats until
we get the first step message. This PR fixes this by passing along the
initial offset and adds a test for this.

changelog_begin

- [Json Api] Fix a bug where heartbeating on websocket connections did not start until the first transaction was received when resuming from a previous ledger offset. See https://github.com/digital-asset/daml/issues/9102

changelog_end

fixes #9102
2021-03-11 19:37:57 +01:00
Moritz Kiefer
5197374cab
Build SDK EE tarball (#9049)
* Build SDK EE tarball

This sets up the infrastructure to build an SDK EE tarball and allows
for swapping out all files included in the tarball depending on the
edition. As an example, this includes the JSON API with (partial)
Oracle support in the EE tarball.

This PR does not yet address publishing this artifact to Artifactory.
I’ll tackle that in a separate PR.

changelog_begin
changelog_end

* Build in temp dir because Windows is stupid

changelog_begin
changelog_end

* directories are bad

changelog_begin
changelog_end

* Navigator resources are actually needed

changelog_begin
changelog_end
2021-03-08 17:48:00 +01:00
Kamil Bożek
32d4bf92ec
Do not require a JWT token for Health and Reflection services [DPP-277] (#8969)
* Do not require a JWT token for Health and Reflection services

CHANGELOG_BEGIN
- A JWT token is no longer required to call methods of Health and Reflection services
CHANGELOG_END

* Let service's authorizer decide about rejections

* Updated authorization test

* Added integration test for unsecured authorisation test for the Health service

* Added integration test for unsecured authorisation test for the Server Reflection service

* Updated Claims doc comments

* Minor change

* Reduced code duplication with SecuredServiceCallAuthTests and UnsecuredServiceCallAuthTests

* Added copyrights

* Move response status handling logic to Authorizer
2021-03-03 11:05:35 +00:00
Remy
a9b035e92e
LF: release LF 1.12 (#8856)
CHANGELOG_BEGIN

    - [LF] Release LF 1.12. This version reduce the size of transaction

    - [Compiler]: Change the default LF output from 1.8 to 1.11.

CHANGELOG_END
2021-02-24 21:17:33 +01:00
Moritz Kiefer
abf061f694
Fix flakiness in JSON API integration tests (#8933)
The tests start the stream in parallel to submitting commands. This is
problematic since it means that those commands can either be included
in the ACS block or they can come after the ACS block. This PR polls
for the ACS block upfront which makes sure that the commands come afterwards.

changelog_begin
changelog_end
2021-02-24 14:00:19 +01:00
Stephen Compall
2e671e4e5d
non-empty newtypes (#8516)
* non-empty newtypes

* an operation

* add some map/set operations and make everything compile on 2.12 and 2.13

* +-: and :-+, with compatibility layer; docs

* move to nonempty package; add aliases for cons/snoc; fix SeqOps aliases

* ensure 2.12 aliases are inferrable

* groupBy1 and toList, use to prove uniqueSets's invariants

* prove immutability first

* matching variance in aliases

* prove the return property of uniqueSets, and use the proof

* tests for NonEmpty API

* rename sci alias to imm

* move RefinedOps to more obvious location

* more docs

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused imports

* illustrate the scala.collection.Seq problem

* ideas for extension

* tests for toF

* tests for +-:

* explain difference with OneAnd
2021-02-22 08:54:26 -05:00
Remy
d92f2c7003
Ledger: refactor bazel packaging of tedger test tool (#8894)
improve previous generalization from #8695

- use lf version instead keyword (like 'stable', 'latest', 'dev') to
  tag actual target.  This will allow two keywords to map to the same
  versions without doing twice the compilation/test work.

- use alias to map keywords tag target to versioned tag target.

- move package manage dar to test_commong.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-22 11:41:19 +01:00
Moritz Kiefer
7255baeb0f
Switch to @platforms//:incompatible (#8873)
changelog_begin
changelog_end
2021-02-17 10:15:17 +00:00
Stephen Compall
b94b5f92f3
experimental Oracle support in json-api (#8596)
* separate OracleQueries from PostgresQueries

- with some changes from 8161e63189 courtesy @cocreature

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* abstract BIGINT

* json, signatories, observers columns

* compatible lastOffset

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* oracle functions for select (single template ID), insert

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* add oracle branch to integration tests

* oracle CLI configuration for json-api

* run integration tests with ojdbc in classpath

* update maven_install for ojdbc

* drop table if exists for Oracle

* make create DDLs and drops more planned out; drop in reverse order for Oracle integrity

* repin maven

* port agreement_text

* port (by removal) array part of ledger offset update

* use CASE instead of JSON map lookup for multiparty offset update

* simplify self types

* fix contract archival

* repin

* remove selectContracts in favor of selectContractsMultiTemplate

* move Oracle test execution to separate build target

* move websocket test to itlib

* make a bad array instance for Oracle

* report actually-available JDBC drivers only

* configure Oracle test from CI

* attempt with platforms and constraints

* a mismash of bazel to get it to conditionally enable oracle testing

* fix dep resolution in Scala 2.13

* make the Oracle test a stub (inits and does empty DB query)

* remove commented unused deps

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* repin

* we never supply a value for the surrogate ID columns

- suggested by @cocreature; thanks

* add not null to json in DB-specific place

- suggested by @cocreature; thanks

* why DBContractKey

- suggested by @cocreature; thanks

* textType isn't finalized

- suggested by @cocreature; thanks

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-02-17 03:50:35 -05:00
Moritz Kiefer
7ae28beff9
More Scala 2.13 cleanup (#8855)
This fixes Scaladoc and our pom file generation.

It also clears up the confusing error around gatling and removes a
redundant dependency on sbt (no idea why we had that in the first
place) both of which resulted in Scala 2.12 dependencies in our 2.13
lockfile which is obviously bad.

With this, we should now be ready to publish Scala 2.13 artifacts once
the ledger API test tool PR lands.

changelog_begin
changelog_end
2021-02-16 09:39:16 +00:00
Moritz Kiefer
153a23d315
Port the rest of //ledger-service/... to 2.13 (#8836)
changelog_begin
changelog_end
2021-02-12 17:26:19 +01:00
Moritz Kiefer
c8b82f8e7a
Port HTTP JSON API binary to 2.13 (#8824)
Tests are still missing. Simply haven’t found the time yet, I’ll
tackle those separately.

changelog_begin
changelog_end
2021-02-12 09:54:47 +01:00
Moritz Kiefer
beb8a2a9ba
Port parts of //ledger-service to Scala 2.13 (#8776)
Unfortunately missing the actual interesting part since porting
`partitionBimap` seems to be rather annoying but this at least gets us
started on the easy parts.

changelog_begin
changelog_end
2021-02-08 11:27:50 +00:00
Stefano Baghino
42e071f2c2
Add //libs-scala/doobie-slf4j and replace usages of jdkLogHandler (#8731)
The jdkLogHandler provided by Doobie exists purely as an example and the library
itself does not recommend using it in production.

Note that this slightly changes the runtime behavior, logging successful queries
at debug level rather then info. The message itself is preserved from the original
MIT-licensed example.

This uses Slf4j as most of our components, instead of java.util.logging.

changelog_begin
[HTTP JSON API] The server now logs successful queries at debug level
instead of info
[Trigger Service] The server now logs successful queries at debug level
instead of info
changelog_end
2021-02-03 10:45:05 +00:00
Stephen Compall
e51bcd54d4
lengthen timeout for FlowUtilTest from 150ms to 5s (#8639)
- found by @stefanobaghino-da; thanks

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-27 14:48:14 +00:00
Stephen Compall
00a5fe50ca
soft-code Postgres in json-api (#8517)
* SupportedJdbcDriver box for the required DB-specific implicits and magic values

* replace postgres references with the SupportedJdbcDriver box

* explaining the typeclass

* labels for debugging

* allow external initialization of SupportedJdbcDriver, but not usage

* thread SupportedJdbcDriver everywhere, hang it off of ContractDao

* remove unused dep from integration tests

* split Queries into an agnostic part and a DB-specific part

* document withOptPrefix

* reformat

* SQL syntax more amenable to refactoring

* different sets of DDL for different backends

* make everything use queries passed around everywhere (usually via ContractDao)

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-21 07:22:33 -05:00
Stephen Compall
c0c2a59275
hybrid database-then-gRPC websocket query (#8226)
* group database queries while still providing the matchedQueries part of the event

* SQL query for multiquery websocket request all at once

* fetchAndPersist responds with a single bookmark

* unify connection imports

* convert DB results to domain, reassociate with proper query indices

* reassociate multiple matches with proper query indices

* make overlap more likely in testing overlap

* simpler matchedQueries merging for multi-query case

* integrate DB query with metadata-ful StreamQuery's

* expose daoAndFetch; better insertDeleteStepSource doc

* more efficient query path for contract key streams

* missed LogHandler

* persist resolved template IDs; glue the prefiltered set into the stream

* ticks and phantom removal need the state from the ACS

* compile SQL queries for query language predicates on WS

- wrong matchedQueries order

* harmonize order of ACS-vector and Positives

* misc compilation conversions

* WebsocketServicePostgresInt mixin

* update (C) date

* test websocket queries under postgres

* looking for new way to compile queries with proper matchedQueries offsets

* model that querying without matchedQueries requires only one SQL query

* SQL path for contract key streams

* nondeterminism

* fix 3 fetch tests with SQL syntax

* nondeterminism mk 2

* fix multi-party query tests by dealing with nondeterminism properly

* temp logs to track down the contract duplication

* match new scalafmt from #8437

* remove completed TODOs

* add changelog

CHANGELOG_BEGIN
- [JSON API] If the JDBC query store is enabled, it will be used to optimize
  Websocket queries as well as the previously-supported synchronous queries.
  See `issue #8226 <https://github.com/digital-asset/daml/pull/8226>`__.
CHANGELOG_END

* fix up matchedQueries indices

* complete the fast path for by-id queries

* remove AS c

- suggested by @cocreature; thanks

* remove temporary debugging logs

- suggested by @cocreature; thanks
2021-01-14 10:01:17 -05:00
Stephen Compall
89bc670445
fix race condition in DB update when ACS has later contracts than the ledger-end (#8480)
* fix race condition in DB update when ACS has later contracts than the ledger-end

CHANGELOG_BEGIN
- [JSON API] Under rare conditions, a multi-template query backed by Postgres
  could have mismatched snapshots of the ACS for different templates.  These
  conditions are now checked and accounted for.
  See `issue #8226 <https://github.com/digital-asset/daml/pull/8226#issuecomment-756446537>`__.
CHANGELOG_END

* contractsFromOffsetIo already saves the offset to DB

* notes on why we rerun DB update to fix the race condition

* if the ACS last-offset exceeds the tx stream offset, save it in the DB instead
2021-01-13 11:27:33 -05:00
Moritz Kiefer
aebde84842
Upgrade scopt to 4.0.0 (#8466)
* Upgrade scopt to 4.0.0

Scopt 3.x has some issues with Scala 2.13 because it expects an
immutable Seq on 2.13 meaning you cannot just pass in an Array. Rather
than fixing our callsites to convert to an immutable Seq everywhere,
this PR bumps to Scopt 4.0 which goes back to collection.Seq.

and leaving that aside, I’m a fan of upgrading dependencies anyway :)

changelog_begin
changelog_end

* Use val instead of def

changelog_begin
changelog_end
2021-01-11 21:23:02 +01:00
Moritz Kiefer
22ce940954
Upgrade scalafmt and enable trailing commas (#8437)
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.

As announced, this will be merged on Saturday to avoid too many conflicts.

changelog_begin
changelog_end
2021-01-09 11:37:37 +01:00
Bernhard Elsner
cda93db944
Daml case and logo (#8433)
* Replace many occurrences of DAML with Daml

* Update docs logo

* A few more CLI occurrences

CHANGELOG_BEGIN
- Change DAML capitalization and docs logo
CHANGELOG_END

* Fix some over-eager replacements

* A few mor occurrences in md files

* Address comments in *.proto files

* Change case in comments and strings in .ts files

* Revert changes to frozen proto files

* Also revert LF 1.11

* Update get-daml.sh

* Update windows installer

* Include .py files

* Include comments in .daml files

* More instances in the assistant CLI

* some more help texts
2021-01-08 12:50:15 +00:00
Moritz Kiefer
424faa923a
Port damlc dependencies to Scala 2.13 (#8423)
* Port damlc dependencies to Scala 2.13

I got a bit fed up by the fact that going directory by directory
didn’t really work since there are two many interdependencies in
tests (e.g., client tests depend on sandbox, sandbox tests depend on
clients, engine tests depend on DARs which depend on damlc, …).

So before attempting to continue with the per-directory process, this
is a bruteforce approach to break a lot of those cycles by porting all
dependencies of damlc which includes client bindings (for DAML Script)
and Sandbox Classic (also for DAML Script).

If this is too annoying to review let me know and I’ll try to split it
up into a few chunks.

changelog_begin
changelog_end

* Update daml-lf/data/src/main/2.13/com/daml/lf/data/LawlessTraversals.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* fixup lawlesstraversal

changelog_begin
changelog_end

* less iterator more view

changelog_begin
changelog_end

* document safety of unsafeWrapArray

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2021-01-08 07:22:38 +01:00
Moritz Kiefer
746b5f4774
Remove a few redundant toString calls (#8379)
Spotbugs yelled at me and while they don’t do any harm they might be slightly confusing.

changelog_begin
changelog_end
2021-01-05 08:31:15 +01:00
Moritz Kiefer
0541b2b888
Prefer Integer.valueOf over new Integer (#8381)
The latter is deprecated in newer JDKs.

changelog_begin
changelog_end
2021-01-04 17:37:33 +00:00
Gary Verhaegen
a925f0174c
update copyright notices for 2021 (#8257)
* update copyright notices for 2021

To be merged on 2021-01-01.

CHANGELOG_BEGIN
CHANGELOG_END

* patch-bazel-windows & da-ghc-lib
2021-01-01 19:49:51 +01:00
Moritz Kiefer
494077cb3d
Support multi-party submissions in the JSON API (#8328)
* Support multi-party submissions in the JSON API

changelog_begin

- [JSON API] Add support for multi-party submissions by allowing for
multiple actAs parties in the token and passing on readAs to the
ledger.

changelog_end

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/EndpointsCompanion.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-12-17 21:10:09 +00:00
Moritz Kiefer
76b6fd86fb
Upgrade Scala dependencies for 2.13 compatibility (#8268)
* Upgrade Scala dependencies for 2.13 compatibility

This upgrades a bunch of Scala libraries to versions that have 2.13
support. There are two libraries that are still missing:

- diffson, this has a new version but with significant breaking
  changes and it is only used in Naigator console which I hope to kill
  before I have to worry about this.

- ai.x:diff, this is used in the ledger API test tool. The library is
  abondened but there are a few alternatives.

changelog_begin
changelog_end

* Fix pureconfig

changelog_begin
changelog_end

* Fix Navigator

changelog_begin
changelog_end
2020-12-14 09:59:00 +01:00
Moritz Kiefer
7e05dc7932
Upgrade rules-scala and scalatest (#8187)
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.

This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.

Apologies for the giant PR, I don’t see a way to keep it smaller.

changelog_begin
changelog_end
2020-12-08 06:59:23 +01:00
Stephen Compall
8ec103ea0b
speed up sync fetch by contract ID/key with postgres and faster in-memory filtering (#7993)
* mark some of dbbackend private

* fetchBy* functions for Queries

* shift in-memory filtering into the transaction stream

- removes irrelevant contracts from memory as soon as possible for fetch by
  contract ID and key in-memory

* push the three synchronous search varieties into new signatures

* replace the core findByContract* functions with DB-delegating versions

* remove the GADT equality and most of the explicit traversals

- compiles again, finally

* factoring

* ContractDao wrappers for fetchById and fetchByKey

* DB version of findByContractId

* DB version of findByContractKey

* Search is the split of ContractsService

* fix SQL for keys

* trade the typeclass for a sum type

- sealed instead of final because of the path dependency on ContractsService
  instance

* number conversion is done already in ContractDao

* make fetch-by-key tests depend on proper number conversion for SQL

* add changelog

CHANGELOG_BEGIN
- [JSON API] ``/v1/fetch`` now uses the Postgres database, if configured, to
  look up contracts by ID or key, except when querying a contract by ID without
  its corresponding template ID.  The fallback in-memory version of
  ``/v1/fetch`` is also now significantly more efficient for large datasets,
  though still linear.

  You may optionally re-create JSON API's database to take full advantage.
  See `issue #7993 <https://github.com/digital-asset/daml/pull/7993>`__.
CHANGELOG_END

* use search.search for search

- suggested by @cocreature; thanks

* add an index for contract key lookups

- suggested by @cocreature; thanks
2020-11-30 13:51:29 -05:00
Samir Talwar
9976b4cd50
Bazel: Factor out logic around Protobuf JARs. [KVL-714] (#8084)
* kvutils: Use ScalaPB to generate a Scala JAR for daml_kvutils.proto.

* Bazel: Delete the unused `da_java_binary` rule, and inline `_wrap_rule`.

* Bazel: Factor out Java/Scala protobuf class generation into a helper.

CHANGELOG_BEGIN
CHANGELOG_END

* daml-lf/archive: Use `proto_jars`.

* Bazel: Remove the visibility modifier from `proto_jars`.

It's too confusing. Just make everything public.

* daml-lf/archive: Push protobuf source tarballs into `proto_jars`.

* Bazel: Add comments to the various parts of `proto_jars`.

* daml-assistant: Do unpleasant things with `location` in Bazel.
2020-11-27 08:34:53 +00:00
Moritz Kiefer
2d3820ac14
Upgrade akka-http to 10.2 (#8058)
* Upgrade akka-http to 10.2

Follow up to #8048, I left out this upgrade to reduce noise and since
I wasn’t quite sure how involved it was going to be.

changelog_begin
changelog_end

* Reenable transparent HEAD requests

Apparently no longer on by default but we depend on this in waitForHttpServer

changelog_begin
changelog_end
2020-11-25 13:39:25 +01:00
Moritz Kiefer
d83cbdb475
Upgrade akka and akka-http (#8048)
* Upgrade akka and akka-http

Was chasing an issue somewhere and thought this might affect it in
some way. It didn’t but I might as well turn the upgrade into a PR.

changelog_begin
changelog_end

* Fix trigger service tests

changelog_begin
changelog_end

* Downgrade akka-http again

changelog_begin
changelog_end

* Upgrade akka-http again and fix tests

changelog_begin
changelog_end

* Cleanup trigger service

changelog_begin
changelog_end
2020-11-25 10:13:51 +01:00
Moritz Kiefer
45ba167aac
Dedup parties in JwtPayload (#7973)
Previously we didn’t build up the `OneAnd[Set, Party]` properly and
included the one party in the set as well. This was an issue if you
have the same party multiple times, most likely in readAs and
actAs (but not limited to that). This then lead to SQL queries failing
since we tried to insert twice for a given party. This PR fixes that
by properly deduplicating the parties and adding a test for this.

changelog_begin

- [JSON API] Fix a regression introduced in SDK 1.7.0, where using a
  party multiple times in the same JWT token (e.g., readAs and actAs)
  broke database queries for that party. Note that there is never a
  reason to include a party multiple times since actAs implies readAs.

changelog_end
2020-11-16 19:28:11 +01:00
Moritz Kiefer
bdf1c48947
Make HealthService public (#7887)
* Make HealthService public

DABL patches the rest adapter so making this public helps them plug it
together with other things.

Also removes some garbage debug print which I forgot to remove
🤦

changelog_begin
changelog_end

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/HealthService.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* No logging

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-11-04 19:13:54 +01:00
Moritz Kiefer
a74a1c15f4
Add healthcheck endpoints to JSON API (#7852)
* Add healthcheck endpoints to JSON API

This PR adds /livez and /readyz (following k8s naming scheme) that can
be used as liveness and readyness check. There isn’t much we can do
for liveness apart from showing that we can still respond to http
requests but readyness can be a bit more clever and check the ledger
connection as well as the database connection.

changelog_begin

- [JSON API] Add `/livez` and `/readyz` health check endpoints for
  easier integration with k8s and other schedulers.

changelog_end

* I hate windows

changelog_begin
changelog_end
2020-11-02 18:41:49 +01:00
Moritz Kiefer
37f389640a
Use chunked responses in the JSON API (#7836)
HTTP 1.1 exists since 1999 so there isn’t really a good reason not do
use this. In fact, the docs recommend to use Chunked in favor of
CloseDelimited.

changelog_begin
changelog_end
2020-10-28 18:43:10 +01:00
Andreas Herrmann
b452e07d05
Move common jwt cli flags (#7811)
* move cliopts

changelog_begin
changelog_end

* update artifacts.yaml

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-26 17:53:37 +00:00
Moritz Kiefer
22f14b72c5
Test for database failures in JSON API (#7808)
changelog_begin
changelog_end
2020-10-26 18:18:46 +01:00
Andreas Herrmann
cd56af1cdd
Factor out the JWT verifier CLI options (#7780)
* Factor JWT verifier CLI flags

changelog_begin
changelog_end

* Use cli-opts in auth middleware

* Use cli-opts in sandbox cli

* Mark trigger service test as long

These have become prone to timeout on CI.

Increasing the size (timeout) is a temporary fix. A proper
solution is to a) not start a fresh sandbox per test-case and b)
separate the in-mem/db and no-auth/auth configrations into
separate Bazel test targets.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-26 15:10:40 +01:00
Moritz Kiefer
339704dd14
Tests for timeouts to/from ledger in JSON API failure tests (#7791)
* Tests for timeouts to/from ledger in JSON API failure tests

changelog_begin
changelog_end

* Update ledger-service/http-json-testing/src/main/scala/com/daml/http/HttpServiceTestFixture.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Update ledger-service/http-json-testing/src/main/scala/com/daml/http/HttpServiceTestFixture.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Apply suggestions from code review

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-10-26 09:08:03 +01:00
Moritz Kiefer
19a7f326f1
Include streaming API in JSON API failure tests (#7783)
This PR extends the failure tests with one that establishes a
websocket connection, kills the connection and reenables it at the
last offset.

changelog_begin
changelog_end
2020-10-23 09:12:09 +02:00
Moritz Kiefer
9da86a602e
Relax restrictions on JWTs on write endpoints in JSON API (#7712)
The previous restriction was both too lax and too strict now that we
have multi pary queries:

1. It allowed a party in `readAs` for command submissions which just
   fails on the ledger side. Changing this is technically breaking but
   only if you used a token that would have been rejected as soon as you
   enabled auth so that seems very resonable to break.

2. It didn’t allow extra parties in `readAs`.

This PR switches to requiring exactly one party in `actAs` while
supporting multiple parties in `readAs`.

changelog_begin

- [JSON API] JWTs on command submissions can now contain extra parties
  in the `readAs` field. `actAs` is still limited to a single party.

changelog_end
2020-10-20 19:37:44 +02:00
Moritz Kiefer
ea453c35cf
Add tests for connection failures in the JSON API (#7751)
* Add tests for connection failures in the JSON API

This PR adds some toxiproxy based tests to see how the JSON API reacts
if the connection to the ledger is killed. There are a bunch of
inconsistencies here in the tests some of which we might want to
address and the others we should at least document but I’ll leave that
for future PRs.

changelog_begin
changelog_end

* Import HttpServiceTestFixture instead of prefixing

changelog_begin
changelog_end
2020-10-20 18:59:11 +02:00
Stephen Compall
4ca02e0eb6
upgrade doobie from 0.6.0 to 0.9.2 (#7618)
* set doobie version to 0.9.2 and rerun maven pin

* port extractor and some of JSON API

* repin maven

* use doobie's own builder compatibility where required

* use probably bad derivations to supply Blockers where transactEC was required

- The point of using Blocker instead of ExecutionContext seems to be to
  especially emphasize to API users that it isn't appropriate to use an
  ExecutionContext with ordinary behavior.  That is what we have done, which
  should probably change, but just compiling for now.

* fix fragment inspection test for internal restructuring

- This test depends on implementation details of Doobie, so naturally it must be
  altered when that runs.  Fortunately, it's been made easier by the changes
  in this upgrade.

* allow 256 blockers for navigator transaction blocker, like the global EC

* allow as many blockers as the pool size for trigger service

- The transactor shouldn't share ExecutionContext for transactions with the
  caller, so we set up a new one based on configured pool size.

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-16 09:46:20 -04:00
Moritz Kiefer
40335c5ccd
Support specifying the jdbc URL for the JSON API in an env var (#7686)
This PR shuffles things around a bit to make it easier to test and
adds `--query-store-jdbc-config-env` which specifies the name of an
environment variable containing the jdbc URL. The UX for this is
modeled after #7660.

Added a test for the different formats.

fixes #7667

changelog_begin

- [JSON API] The JDBC url can now also be specified via
`--query-store-jdbc-config-env` which reads it from the given
environment variable.

changelog_end
2020-10-14 20:19:35 +02:00
Stephen Compall
bafde51752
add silent_annotations option to da_scala bazel functions (#7668)
* add silent_annotations option to da scala bazel functions

* use silent_annotations for several scala targets

* use silencer_plugin instead when the lib isn't used

* use silent_annotations for several more scala targets

* use silencer_lib for strange indirect requirement for running tests

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* silent_annotations support for scaladoc
2020-10-13 15:44:16 +00:00
Stephen Compall
1a2afd5266
upgrade to Scala 2.12.12 from 2.12.11 (#7661)
* upgrade bazel settings to scala 2.12.12

* upgrade nix scala tool to scala 2.12.12

* upgrade silencer references to scala 2.12.12

* repin for scala 2.12, silencer, wartremover upgrades

* remove numerous occurrences of unused silencer now spotted

* update Scala version in our bazel notes

CHANGELOG_BEGIN
CHANGELOG_END

* update compatibility maven_install.json to match compatibility WORKSPACE
2020-10-13 08:42:14 -04:00
Moritz Kiefer
aa3e5a7dbe
Support multi-party reads on the JSON API (#7571)
* Support multi-party reads on the JSON API

Given that those aren’t going away and we’re instead doubling down on
this and adding multi-party writes as well, the JSON API needs to
support this. This PR only implements the read side (since the ledgers
do not yet support the write side).

This does not deviate from the approach chosen by the JSON API to
infer the parties from the token, we just don’t error out anymore when
more than one party is passed.

changelog_begin
changelog_end

* Apply suggestions from code review

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Remove dependency on doobie_postgres from db-backend

changelog_begin
changelog_end

* Fix offset update

changelog_begin
changelog_end

* Use nonempty sets for parties

changelog_begin
changelog_end

* Fix updateOffset under concurrent transactions

changelog_begin
changelog_end

* Add tests for multi-party websocket queries and fetches

changelog_begin
changelog_end

* fmt

changelog_begin
changelog_end

* Fix perf tests

changelog_begin
changelog_end

* Cleanup

changelog_begin
changelog_end

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/dbbackend/ContractDao.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Move ParsePayload instances, thanks Stephen!

changelog_begin
changelog_end

* More unsubst

changelog_begin
changelog_end

* Fix off by 1 error

changelog_begin
changelog_end

* Remove redundant type annotation

changelog_begin
changelog_end

* fmt

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-10-08 13:46:15 +02:00
Moritz Kiefer
a34963c16d
Include JSON API logback in scala_binary (#7485)
With the introduction of the standalone JAR, we cannot rely on the
assistant anymore to pass the default logback config. Users can still
override the logback config with `-Dlogback.configurationFile` if they
need something else but this provides a more sensible default logging
config than seeing a ton of debug logs from netty.

changelog_begin
changelog_end
2020-09-25 17:52:50 +02:00
Remy
f5694ee2ea
LF: Kill ValueStruct (#7457)
Struct is not serializable and therefore should not appear as Value.

CHANGELOG_BEGIN
CHANGELOG_END
2020-09-22 13:37:59 +02:00
Leonid Shlyapnikov
0730102b99
JSON API Perf Test Runner Query Store Index Support (#7434)
* Postgresql setup for Gatling perf test

* Add JSON API `query-store-index` support

changelog_begin
changelog_end
2020-09-17 16:39:00 -04:00
Leonid Shlyapnikov
d13e7aa184
JSON API daily perf test cron job (#7406)
* Add http-json-perf daily cron job

changelog_begin
changelog_end

* commenting out other jobs so we can manually test the new one

* commenting out other jobs so we can manually test the new one

* Fix the shell script

* Fixing the gs bucket, `gs://http-json bucket does not exist`

* uncomment the other jobs

* timestamp from git log

* get rid of DAR copying

* comment out the other jobs, so we can test it

* uncomment the other jobs
2020-09-16 13:02:02 -04:00
Leonid Shlyapnikov
fe63816431
Write report summary in text format (#7407)
* write summary.txt

changelog_begin
changelog_end

* make `Response time distribution` brackets configurable
2020-09-16 09:31:54 -04:00
Leonid Shlyapnikov
ae2981eaf2
Updating readme (#7398)
changelog_begin
changelog_end
2020-09-15 08:51:57 -04:00
Leonid Shlyapnikov
4fbde3c672
Open-sourcing Gatling statistics reporter (#7325)
* Open sourcing gatling statistics reporter

Running gatling scenarios with `RunLikeGatling` from libs-scala/gatling-utils

* cleaning up

* Replace "\n" with System.lineSeparator

so the formatting test cases pass on windows

* Testing DurationStatistics Monoid laws

* Renaming RunLikeGatling -> CustomRunner
2020-09-11 13:39:05 +00:00
Remy
78e770bf72
LF: Use the new Struct data structure for ValueStruct (#7241)
This PR uses the new data structure introduced in #7220.
Additionnally this fix `Value Equal instance` which was considering
<a: x, b: y> different from <b:y, a:x>.

CHANGELOG_BEGIN
CHANGELOG_END
2020-09-04 18:32:39 +02:00
Leonid Shlyapnikov
6186c5ccfd
Fixing flaky Template ID resolution test (#7307)
* Adding `uniqueModuleEntity`

and making sure that generated domain Template IDs do not
have module entity duplicates, so resolution should work
with no problem.

changelog_begin
changelog_end

* cleaning up
2020-09-02 12:17:37 -04:00
Leonid Shlyapnikov
ccecc46e70
JSON API Perf Scenario Runner (#7295)
* Run gatling scenario from the perf runner main

reports are disabled for now, getting a class not found
when generating them

changelog_begin
changelog_end
2020-09-02 09:00:43 -04:00
Samir Talwar
15cb93eec3
http-json: Use matchers in WebsocketServiceIntegrationTest. (#7302)
This makes diagnosing failures in CI easier, because "true is not false"
is not a very helpful error message.

CHANGELOG_BEGIN
CHANGELOG_END
2020-09-02 11:52:05 +00:00
Moritz Kiefer
abb979a4db
Fix flakiness in multi-key tests (#7298)
There are two sources of flakiness and I’ve seen both on CI:

1. We can get more than one offset at the beginning if things are too
   slow. This is addressed by just filtering those out.
2. The stream completes as soon as the input is closed. This is
   addressed by keeping the stream open and closing it with `take`.

Point 2 is a problem for all tests, see #7293, but I’ll leave the
other tests for separate PRs (I’ve also never seen them flake).

You can observe the failures locally if you add a `Thread.sleep` between
creating the stream future and sending the commands.

changelog_begin
changelog_end
2020-09-02 12:50:51 +02:00
Samir Talwar
2b3cf1b3c0
ledger-api-client: Rename maxInboundMessageSize to maxInboundMetadataSize. (#7290)
* ledger-api-client: `maxInboundMessageSize` -> `maxInboundMetadataSize`.

CHANGELOG_BEGIN
- [Scala Bindings] Rename a field in the ``LedgerClientConfiguration``
  to ``maxInboundMetadataSize``, to match the builder Netty channel
  builder. It was incorrectly named ``maxInboundMessageSize``, which is
  a different channel property that configures the maximum message size,
  not the header size.
CHANGELOG_END

* ledger-api-client: Introduce a `maxInboundMessageSize` config property.

We use this a lot; easier if it's in the configuration.

CHANGELOG_BEGIN
- [Scala Bindings] Replace the
  ``LedgerClientConfiguration.maxInboundMessageSize`` property with a
  new one that represents the maximum size of the response body.
CHANGELOG_END
2020-09-02 08:41:34 +00:00
Gary Verhaegen
989fdc9bde
fix JSON API multikey stream (#7244)
fix JSON API multikey stream

In the current state, the JSON API only handles multiple keys _from
different templates_. This makes it work for multiple keys from the same
template too.

Extracted from #7066 with the following changes:

- Use of a mutable `HashSet` to test for keys, because perf.
- Addition of a test at the JSON API level.

CHANGELOG_BEGIN

- [JSON API] Fix a bug where streaming multiple keys could silently
  ignore some of the given keys.

CHANGELOG_END

* apply @cocreature's patch

https://gist.github.com/cocreature/d35367153a7331dc15cca4e5ea9098f0

* fix fmt
2020-09-01 18:17:13 +02:00
Leonid Shlyapnikov
20e2922b3e
JSON API perf test main that starts sandbox and json-api services (#7283)
* reintroducing the main

* Introducing `ledger-service/http-json-testing`

* cleaning up

* Starting sandbox and json-api from perf-test main

changelog_begin
changelog_end
2020-08-31 22:55:01 +00:00
Leonid Shlyapnikov
1452076eee
Deprecating noop '--application-id' command-line option in JSON API (#7232)
* Deprecate noop `--application-id`

changelog_begin

[JSON API]
Hiding and deprecating `--application-id` command-line option. JSON API never used it.
It is required to instantiate LedgerClientConfiguration and was not used for any command submission.
JSON API uses Application ID specified in the JWT. See #7162

changelog_end

* removing further usage of noop applicationId

* a bit of explanation what this is for
2020-08-26 20:05:00 +00:00
Gary Verhaegen
07657e01c8
fix docs warning (#7243)
Apparently `[[]]` links don't work to external resources. I wanted to
turn the first referencee into a link too but that contains too many
weird characters.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-26 20:27:33 +02:00
Leonid Shlyapnikov
3ecfcc918c
Adding package-max-inbound-message-size (#7147)
* Adding `package-max-inbound-message-size`

this is to allow separate configuration settings for command submission
and package management ledger clients

* Fixing formatting

* Updating docs

changelog_begin

[JSON API] Adding `--package-max-inbound-message-size` command line option.
Optional max inbound message size in bytes used for uploading and downloading package updates. Defaults to the `max-inbound-message-size` setting.

changelog_end

* Addressing code review comments
2020-08-24 14:55:55 +00:00
Moritz Kiefer
820c9ff844
Remove damlLfInstantiate in favor of TypeCon.instantiate (#7166)
fixes #2506

Judging from the issue this was originally introduced to workaround a
bug. I couldn’t actually track down what that bug was but at this
point they are identical so no point keeping this around.

changelog_begin
changelog_end
2020-08-18 12:15:12 +00:00
Leonid Shlyapnikov
cb73542316
http-json-perf cleanup (#7145)
* Removing unused Main and Config,

cleaning up BUILD.bazel: removing common scala options,

* Removing unused dependencies

* Removing unused dependencies

changelog_begin
changelog_end
2020-08-14 17:02:45 +00:00
Leonid Shlyapnikov
5287e5b946
Do not emit offset ticks preceding the ACS events (#7109)
* Introducing `TickTriggerOrStep` ADT, filtering out `TickTrigger`s preceding the initial ACS retrieval

changelog_begin
[JSON API] Filter out offset ticks preceding the ACS events block. See issue: #6940.
changelog_end

* Cleaning up a bit

* Do not emit offset tick unless we know the real offset

wait for LiveBegin message

* Make WebsocketConfig configurable

* Adding offset tick integration tests

reverting WebsocketService to 05d49b37c3 makes these tests fail

* cleaning up

* Refactoring `emitOffsetTicksAndFilterOutEmptySteps`

keep offset instead of StepAndError with offset
2020-08-13 09:00:28 +02:00
Stephen Compall
1737907415
add --address option to trigger service (#7090)
* factor --address, --http-port, --port-file options from http-json to cli-opts

- enabling reuse in trigger service

* use cli-opts for address and http-port options in Trigger service

* mark ServiceConfig and some defaults private

* use --address option to set up server

* document Setter

* test --address option is parsed

* missing (c) headers

* add changelog

CHANGELOG_BEGIN
- [Trigger Service] Accepts a new ``--address`` option to listen for HTTP connections on
  interfaces other than localhost, such as ``0.0.0.0`` for all addresses.
  See `issue #7090 <https://github.com/digital-asset/daml/pull/7090>`__.
CHANGELOG_END
2020-08-12 13:50:00 -04:00
Samir Talwar
27f76c4386
Use Future.unit instead of Future.successful(()). (#7080)
And in one instance, `Resource.unit`.

I just think it's easier to read. Too many parentheses make Samir a dull
boy.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-11 09:18:50 +00:00
Leonid Shlyapnikov
213832572b
Sync query perf scenario, new ACS every query (#7078)
* AsyncQueryNewAcs scenario

sync query with totally new ACS every time

changelog_begin
changelog_end

* cleanup

* cleanup

* fixing numbers

* fixing numbers

* adding groups to self-document the scenario

* Archive and then Create

* cleanup

* silence archive and create

* with ACS of 5000 it takes too long to run
2020-08-10 22:23:46 +00:00
Leonid Shlyapnikov
ba22f5b9a4
AsyncQueryConstantAcs scenario (#7054)
changelog_begin
changelog_end
2020-08-07 09:33:05 -04:00
Leonid Shlyapnikov
a630e3c944
JSON API SyncQueryVariableAcs perf test scenario (#7028)
* Perf test scenario for query with variable ACS, WIP

* WIP

* change ACS with every query

exercise a choice + create a new contract to keep ACS size the same

* change ACS with every query

running exercise and create in parallel with the query

* exercise Archive instead of Transfer

* Adding copyright header

* Thanks @S11001001
2020-08-05 18:31:18 +00:00
Leonid Shlyapnikov
9acb47d994
JSON API command submission perf test (#6906)
* JSON API Perf test scenarios

changelog_begin
changelog_end

* addressing code review, thanks @S11001001

* addressing code review, thanks @S11001001

* cleanup

* cleanup + instructions

* cleanup + instructions

* Updates

* Updates
2020-08-04 18:51:44 +00:00
Gary Verhaegen
109d184d39
improve error message on failed JSON parsing (#6978)
* improve error message on failed JSON parsing

Fixes #6971.

Interestingly, all the other cases in that block already had useful
feedback, not sure why this ones was missing.

CHANGELOG_BEGIN
CHANGELOG_END

* add tests
2020-08-04 14:56:17 +02:00
Leonid Shlyapnikov
05d49b37c3
Introduce //libs-scala/scala-utils (#6935)
* Moving `Statements.discard` from //ledger-server/http-json into //libs-scala/scala-utils

changelog_begin
changelog_end

* Add new module to the published artifacts

* `com.daml.scalautil` instead of `com.daml.scala.util`

@S11001001: That's because if this is in classpath and you import com.daml._,
you have a different scala in scope than the one you expect.
2020-07-30 13:51:07 -04:00
Samir Talwar
98de16606d
daml-on-sql: An elegant Main class, for a more civilized age. (#6829)
* daml-on-sql: Pull out a new `Main` object that wraps sandbox-classic.

CHANGELOG_BEGIN
CHANGELOG_END

* daml-on-sql: Fail if a JDBC URL is not provided or not for PostgreSQL.

* sandbox-classic: Rename the conformance test H2 database.

* daml-on-sql + sandbox-classic: Report configuration errors cleanly.

This means letting `ProgramResource` catch the errors, log, and exit.

* daml-on-sql: Change the name logged on startup.

* daml-on-sql: Change the default participant ID.

* sandbox-common: Give the ledger name its own tagged string type.

* sandbox-classic: Generate random ledger IDs using the ledger name.

* daml-on-sql: Remove the banner, replacing it with a blank line.

* daml-on-sql: Enable strong seeding by default.

And weak seeding in the conformance tests.

* sandbox-classic: Move the ledger name to a separate parameter.

It's not really configurable.

* sandbox-classic: Move LedgerName from sandbox-common.

* daml-on-sql: Remove "-participant" from the participant ID.

* daml-on-sql: Use `Name` where possible.

* daml-on-sql: Make the ledger ID mandatory.

* Revert "sandbox-classic: Move LedgerName from sandbox-common."

This reverts commit 0dad1584a7.

* daml-on-sql: Print "DAML-on-SQL" in the CLI help, not "Sandbox".

* daml-on-sql + sandbox + sandbox-classic: Split out custom CLI parsing. (#6846)

* participant-state: Simplify naming the seeding modes.
2020-07-24 18:54:19 +00:00
Samir Talwar
adcbfb45a1
sandbox-classic: Move the configuration from sandbox-common. (#6810)
* sandbox-common: Move the ledgerIdMode up in the configuration.

* sandbox-classic: Move the configuration from sandbox-common.

CHANGELOG_BEGIN
CHANGELOG_END

* Fix miscellaneous warnings caught by IntelliJ IDEA.
2020-07-21 14:43:53 +00:00
Stephen Compall
4355406259
add more scalac 2.12 warnings (#6798)
* set many extra scalac -Xlint options for all Scala projects

CHANGELOG_BEGIN
CHANGELOG_END

* move NoCopy to its own file

package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
  trait NoCopy {
        ^

* move more traits, classes, and objects to proper packages

- note that `package` is itself a scoping construct, so if your reason
  is the apparent aesthetic of placing a bunch of things in one `package
  object`, that is easily remedied by deleting the `object` keyword

* fix some type-parameter-shadow warnings

- I'm generally in favor of sensible name-shadowing, following the
  "deliberately hide variables that should not be accessed here" school
  of thought.  But I think type name shadowing isn't quite as valuable
  and more likely to confuse than general variable shadowing, so have
  experimentally linted it out.

  Example warning:

EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
 Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
 Offset defined in package v1. You may want to rename your type
 parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
                                                               ^

* fix more package-object-classes warnings

* fix an inaccessible warning

ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
  def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
      ^

* enable -Xlint:infer-any

- continuing the saga of #6116, #6132

* enable -explaintypes for more detailed type errors

* missed header for NoCopy; probably should have left it in the package file

* misspelling in comment

* revert -Xlint:doc-detached

- there are a lot of these fixes, and they are noisy, so shifting to a
  separate PR
- thanks to @leo-da for pointing out
2020-07-21 08:18:01 -04:00
Robert Autenrieth
7ce9748066
Split sandbox code into separate packages (#6695)
* Move public code into daml-integration-api

CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
2020-07-17 17:06:06 +02:00
Leonid Shlyapnikov
9abfff0388
Fixing typo in the JSON API error response (#6735)
changelog_begin
changelog_end
2020-07-15 09:23:40 -04:00
Gary Verhaegen
8176fb0c8d
fix the the typo (#6723)
```
s/the the /the /
```

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-14 16:55:33 +00:00
Stephen Compall
4cd419623a
replace traverseU and sequenceU with traverse and sequence (#6594)
* replace traverseU and sequenceU with traverse and sequence

- with -Ypartial-unification on, the extra Unapply typeclass lookup is
  unnecessary

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* limit imports; we only need *> and void
2020-07-07 15:01:58 +00:00
Stephen Compall
9fb72202d8
use daml-lf scalacopts for ledger-service (#6569)
* add lf_scalacopts to ledger-service/ projects

* handle unused warnings in http-json main

* handle unused warnings in lf-value-json

* handle unused warnings in http-json tests

* handle unused warnings in jwt

* JwtGenerator.generate doesn't use its argument; remove it

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* format bazel files
2020-07-02 13:58:41 +00:00
Stephen Compall
86148c398b
make @silent annotations narrower (#6570)
* check that all @silent annotations are used

CHANGELOG_BEGIN
CHANGELOG_END

* make all @silent annotations very specific about what they're silencing
2020-07-01 20:53:22 +00:00
Remy
46b68bd92a
ledger test tool: refactor conformance test dars (#6506)
* ledger test tool: refactor conformance test dars

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-26 13:39:00 +02:00
Stephen Compall
a51d0db8ff
set scalac -Xsource:2.13 -Ypartial-unification globally (#6469)
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts

* add now-referenced scalaz-core where needed

* work around bad type signatures in scalatest Aggregating, Containing

* unused Any suppression

* work around bad partial-unification wrought by type alias

* remove unused Conversions import

- not required in 4f68cfc480 either, so unsure how it's survived this long

* work around Future.traverse; remove unused show import

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused bounds

* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed

* longer comment on what the options do

- suggested by @stefanobaghino-da; thanks

* forget Future.traverse, just use scalaz, it knows how to do this
2020-06-24 16:51:24 -04:00
Remy
cbe423fc6c
LF: rename library transaction-scalacheck to transaction-test-lib (#6347)
* LF: rename library transaction-scalacheck to transaction-test-lib

CHANGELOG_BEGIN
CHANGELOG_END

* move files in com/daml

* missing change in release/artifacts.yaml

* remove 'com/dam' from the path
2020-06-17 10:52:40 -04:00
Simon Maxen
5329a4f97f
Ledger id requirement (#6323)
* Add option based constructor for LedgerIdRequirement

changelog_begin
changelog_end

* Make option based consructor the default, deprecate old constructor

* Update with review comments
2020-06-12 15:00:32 +00:00
Stephen Compall
15350a7bc2
disable warts.Any and remove most suppressions (#6132)
* disable Any wart

* first pass removal of Any suppressions for false positives

* second pass removal of Any suppressions for false positives

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* third pass removal of Any suppressions for false positives

* fourth pass removal of Any suppressions for false positives

* reformat newly single-suppressions into single lines

- suggested by @SamirTalwar-DA; thanks
2020-05-28 16:53:41 +00:00
Stephen Compall
265cfe6cdf
split http-json tests into tests and integration-tests targets (#6135)
* move integration-type tests in http-json to separate integration-tests target

* remove unused library deps from split tests

* remove resources from unit tests, set to medium size

* move resources to it subtree

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-05-28 11:46:25 -04:00
Stephen Compall
78205afa0d
use 2.13-style collections library syntax in json-api (#6075)
* register scala-collection-compat with java deps list

* add scala-collection-compat to http-json deps

* remove breakOut throughout http-json, replaced with view/to or iterator/to

* use scala 2.13-style `to` calls in http-json

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* use 2.13-style to in lf-value-json

* some fused size comparisons

* remove low-hanging breakOuts in daml-lf

* regenerate maven_install.json for scala-collection-compat

* regenerate maven_install.json for scala-collection-compat

* regenerate maven_install.json for scala-collection-compat
2020-05-27 22:27:25 +00:00
Stephen Compall
1be6b3640f
equalz Scalatest matcher in new daml-lf/scalatest-tools library (#1200)
* equalz Scalatest matcher in new daml-lf/scalatest-tools library

* equalz typing tests

* a 'should' replacing design

* a 'MatcherFactory1' design

- this fails because the TC parameter should be a type member to avoid
  scala/bug#5075 but it is not

* MatcherFactory1 with chained Lub+Equal typeclass

- requires partial-unification at point of use, which is not great

* LubEqual's extra tparam is probably unneeded

* better LtEqual

* demonstrate that HK LubEqual's resolve with DMT should + MatcherFactory

* remove unneeded 3rd param from LubEqual, again

* update dependency specs and license headers

* allow use with should, shouldNot in some cases, preserving the shouldx/shouldNotx alternatives

* move Equalz to libs-scala/scalatest-utils

* rename bzl targets and place in com.daml.scalatest package

* add scalatest-utils to release

* move *SpecCheckLaws, Unnatural to scalatest-utils

* missed scalacheck dep in scalatest-utils

* downstreams of *SpecCheckLaws now get them from scalatest-utils

* test equal-types case as well

* update LF documentation

CHANGELOG_BEGIN
CHANGELOG_END

* whitespace error
2020-05-27 17:25:43 -04:00
Stefano Baghino
9af85e56e9
Introduce DAML-LF value caching for transaction service (#6052)
* Introduce DAML-LF value caching for transaction service

Allows to keep the DAML-LF values in the most recently indexed events in memory,
so that they don't have to be deserialized on the client from their Protobuf encoding.

Closes #6044

CHANGELOG_BEGIN
[Sandbox] The --max-lf-value-translation-cache-entries option allows to set a
number of events for which DAML-LF values are cached. Could reduce latency in
serving transactions for consumers that are reasonably fast.
CHANGELOG_END

* Add missing dependency

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076003

* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/sandboxnext/Runner.scala

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428071324

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076905

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428081294

* Fix fatal warnings

* //ledger/caching has to be used whenever sandbox is run

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-05-26 08:33:53 +00:00
Remy
9e456a1016
DAML-LF: Kill RelativeContractId (#5991)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-25 22:30:45 +02:00
Stephen Compall
d7cc5e07c3
fix ACS race condition in sandbox-classic; enable random Websocket testing (#5969)
* Revert "disable test pending tracking down duplicates"

This reverts commit 8b5f9dfa04.

* activeContracts tells caller the ledgerEnd, rather than accepting activeAt argument

* naive port of http-json tests to sandbox-next

* Revert "naive port of http-json tests to sandbox-next"

This reverts commit 91d4590c90.

* lock before grabbing acs and ledgerEnd offset in InMemoryLedger

- as suggested by @gerolf-da; thanks
2020-05-19 13:26:37 -04:00
Stephen Compall
af52ef3ce9
random IOU create and split testing for websocket streams (#5879)
* Consume language for listing [re]actions in a foldAsync linearly

* eagerly consume as many Futures as possible

* restate one foldAsync test with Consume

* report value that failed to match

* add error locations and new `drain` primitive

* use `drain` to properly handle heartbeats at end of stream

* it doesn't make sense to delay the drain step

* rewrite other websocket test; document syntax

* get rid of StreamState, finally

* split Consume to a separate file

* plan split tests

* functions for consuming SplitSeq

* converting SplitSeq into stream actions

* exercise and check recursively

* add one 100-sized sample to the tests

* don't violate Iou's invariant

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* disable test pending tracking down duplicates

- as discussed with @leo-da
2020-05-13 14:06:11 -04:00
Samir Talwar
208d4a50da
postgresql-testing: Use a new, random database name every time. (#5911)
* postgresql-testing: Store the JDBC URL separately.

* postgresql-testing: Expose the username and password.

* postgresql-testing: Get the caller to create the database.

And make sure it's a random one, not "test".

CHANGELOG_BEGIN
CHANGELOG_END

* postgresql-testing: Only store the JDBC URL for tests.

Less mutable state, innit.

* postgresql-testing: Capture the individual JDBC URL parameters.

* Bazel: Fix PostgreSQL binary paths.

* postgresql-testing: Just recreate the database in PostgresAroundEach.

There's no need to restart the process with a different data directory.
2020-05-08 12:36:42 +00:00
Leonid Shlyapnikov
a79377cb2d
Improve DAML LF Party, PackageName, PackageId fromString error messages (#5855)
* Add description to daml.lf.data.StringModule implementations

Make sure DAML LF Party deserialization picks up the added description.

changelog_begin
changelog_end

* addressing code review comments. @S11001001, thanks.
2020-05-06 10:02:49 -04:00
Stephen Compall
59d5446438
disable https check for JWT tokens in JSON API, warn instead (#5856)
CHANGELOG_BEGIN
- [JSON API] The check that connections are made through a reverse-proxy
  providing HTTPS, ensuring that JWT tokens don't leak, only logs a warning
  rather than rejecting the request.
  See `issue #5856 <https://github.com/digital-asset/daml/pull/5856>`_.
CHANGELOG_END
2020-05-05 19:25:24 -04:00
Stephen Compall
2b23b41391
JSON decoding support for GenMap (#5223)
* add GenMap to the "all types" test generators

* report bad GenMap format with DeserializationError, not MatchError

* document GenMap JSON

* notes on missing features

* enable -Xsource:2.13 in transaction

* make an Order instance for Value resolvable, but unimplemented

* use the skeleton from SValue ordering to make a Value ordering skeleton

* add Party Order

* add Order instance for SortedLookupList

* add Order for FrontStack, deriving everything

* factor the Order lookup, and tie a knot in the recursive Value instances

* we're going to need this Iterator thing again

* replacing Order#contramap with version that supports equalIsNatural

* use new equalBy, orderBy for FrontStack, SortedLookupList, ImmArray

* _2 comparator, upgrade Name Equal to an Order

* incorporate lookup for enums, variants into Value order; record/struct cases

* Enum/Variant comparison

* looking up the singleton implicitly won't work for non-`object`s, alas

* test Order laws for values of all primitive types

* test Order laws for record and variant types

* test Order laws for enum types

* test that enum strings are not compared

* use checkLaws for Value Equal as well

* test that enums match order to constructor rank

* factor genAddend and genAddendNoListMap

* reintroduce Order for TypedValueGenerators

* more addend order

* record, variant order cases

* record cons order

* deriving Order while decoding from JSON

* make ApiCodecCompressed's Cid codec based on the typeclass

* test how the Value ordering and the underlying projected value orderings line up

- hint: they don't, yet
- this is also a template for how we'll check the fidelity with SValue
  ordering

* test how the Value ordering and SValue ordering line up

- hint: they don't, yet

* typed Arbitrarys have access to Order

* produce proper ValueGenMap

* inj requires Order, sometimes

- we encode this as "all the time" but there is a type-level unification
  approach to remove this requirement in some cases

* make inj a function

* test that order doesn't matter for JSON decoder

* use Utf8 order for TVG text; don't pretend that base equal works

* sort JSON GenMaps, and check for duplicates

* make injarb use IntroCtx

* remove stray import

* Order instances for Bytes, Hash, AbsoluteContractId

* require Order[Cid] to decode JSON to LF values

* clean up map reordering test

* remove unused Instant instance

* fake Order instance no longer needed, valid instance defined

* test parity of global AbsoluteContractId order and SContractId order

* bazel fmt

* test AbsoluteContractId Order lawfulness

* test duplicate key detection

CHANGELOG_BEGIN
- [JSON API] Prepare full support for the planned GenMap primitive type.
  See `issue #5031 <https://github.com/digital-asset/daml/issues/5031>`_.
CHANGELOG_END
2020-05-05 10:27:37 -04:00
Oliver Seeliger
4c0ffdf4df
Upgrade scala compiler silencer to 1.6.0 (#5822)
* Upgrade scala compiler silencer to 1.6.0

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt build bazel file to new targets

* Switch to silencer plugin scala 2.12.11 per Samir's feedback

rather than 2.12.8

* Add missed bazel files

* Review feedback from Leo
2020-05-04 21:30:46 +00:00
Stephen Compall
067f3c987d
require https reverse proxy (#5660)
* new --leak-passwords-firesheep-style option; functions to check forwarded protocol

* enforce https reverse-proxy in all JWT-accepting endpoints

* make HttpService.start take config record

* test that X-Forwarded-Proto or Forwarded is enforced

* use new start signature in daml-script tests

* use insecure http mode for ts codegen tests

* note on regex

* use insecure option in daml assistant integration tests

* log allowNonHttps setting

* add non-https option to more places in daml-assistant tests

* add non-https option to getting started guide

* rename --leak-passwords-firesheep-style to --allow-insecure-tokens

- per suggestion by @garyverhaegen-da, @hurryabit

CHANGELOG_BEGIN
- [JSON API] By default, checks that connections are made through a reverse-proxy
  providing HTTPS, ensuring that JWT tokens don't leak.  To disable this check,
  such as for development, pass ``--allow-insecure-tokens``.
  See `issue #5572 <https://github.com/digital-asset/daml/issues/5572>`_.
CHANGELOG_END

* daml start includes --allow-insecure-tokens by default

- as indicated by @cocreature
2020-04-28 14:33:38 -04:00
Leonid Shlyapnikov
102f6b3888
Use a custom Gen instead of Arbitrary, to generate nonempty Vec[Int], (#5763)
* Use a custom Gen instead of Arbitrary, to generate nonempty Vec[Int],

the test could fail on empty input, it expects at least one input message

changelog_begin
changelog_end

* Address code review comments
2020-04-28 16:25:09 +00:00
Stephen Compall
d4086080ea
proper GenMap Gen with key Value order (#5526)
* add Order instances to various types in LF data, transaction

* define scope-dependent Order for Value; test compatibility with SValue order

* support Order in TypedValueGenerators

* generate properly-ordered ValueGenMaps in TypedValueGenerators

* factor genAddend and genAddendNoListMap

* test Value.orderInstance fidelity with well-typed Orders

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* shrink Maps for better debugging

* line up base List and GenMap orders with svalue

* unhandled cases in custom comparator for ApiValueToLfValueConverter tests

- hazards of not using match2

* adapt to PureCompiledPackages changes
2020-04-24 17:57:12 -04:00
Leonid Shlyapnikov
bab018b44e
Allow tokens with zero parties for pass-through endpoints (#5674)
changelog_begin
[JSON API]
Pass through endpoints do not require party specified in the JWT:
- ``/v1/parties``
- ``/v1/parties/allocate``
- ``/v1/packages``
See #5313.
changelog_end
2020-04-22 12:39:20 -04:00
Leonid Shlyapnikov
6aebbaca65
JSON API endpoints for DAR file upload, DALF download (#5613)
* DAR file upload, DALF download, list all DALF package IDs

Reload packages after dar upload

changelog_begin
[JSON API]
Add package management pass-through endpoints. See #4823.
- GET /v1/packages -- returns all package IDs
- GET /v1/packages/<package ID> -- downloads the DALF package
- POST /v1/packages -- uploads a DAR file to the ledger
changelog_end

* Fixing docs

* minor improvements + cleanup
2020-04-21 19:22:50 -04:00
Stephen Compall
91c43f147c
allow secure connection from JSON API to ledger (#5555)
* factor TlsConfiguration parser from extractor

* move TlsConfigurationParser to new library

* link extractor to ledger-service/cli-opts properly

* use TlsConfigurationCli in http-json, pass SslContext to ledger-client

* test TLS options as used in http-json

- the TLS config code is shared with extractor, where it is more fully
  tested; we just do a sanity check here

* doc TLS options for http-json

CHANGELOG_BEGIN
- [JSON API] New ``--pem``, ``--crt``, ``--cacrt``, and ``--tls`` options
  for securing the connection between JSON API server and ledger.
  See `issue #2540 <https://github.com/digital-asset/daml/issues/2540>`__.
CHANGELOG_END

* TLS off in daml-script JSON API test
2020-04-16 13:12:30 -04:00
Leonid Shlyapnikov
89f5ebcde8
Offset ticking fix (#5532)
* Fixing offset tick

changelog_begin
changelog_end
2020-04-10 15:05:19 -04:00
Leonid Shlyapnikov
29e1931f17
Add --port-file command line option to JSON API (#5454)
* Adding `--port-file` support

* ``--port-file`` support

* Updating docs

changelog_begin

[JSON API] Add support for ``--port-file`` command line option.
``--http-port 0 --port-file ./json-api.port`` will pick up a free port
and write it into ``./json-api.port` file.

changelog_end

* reformatting

* Usage grammar

* use bimap

* Adding `PortFiles` utility for creating and deleting port files on JVM exit

* Adding scaladoc explaining that the port file should be deleted on

JVM termination.

* Updating usage and docs to reflect that the file must be unique and

will be deleted on graceful shutdown

* Relying on `java.nio.file.FileAlreadyExistsException` to determine the

case when failed due to the nonunique file name.

* toString instead of Exception.getMessage

java.nio exception's getMessage can be just a file name, need the class
name to capture the error context.

* updatePortFile -> createPortFile

* write to file instead of write into file
2020-04-08 18:48:11 +00:00
Gerolf Seitz
97433743a1
Set the Bearer prefix in bindings. (#5484)
* Set the `Bearer ` prefix in bindings.
* Make the `Bearer ` prefix in the authorization header mandatory.
* Bearer prefix can be removed from the token file.

CHANGELOG_BEGIN
[Extractor]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[Navigator]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[DAML Script] The ``Bearer `` prefix can be removed from the token file. It
is added automatically.
[DAML Repl] The ``Bearer `` prefix can be removed from the token file. It is
added automatically.
[Scala Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[Java Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[DAML Integration Kit] ``AuthService`` implementations MUST read the
``Authorization`` header and the value of the header MUST start with
``Bearer ``.
CHANGELOG_END
2020-04-08 13:07:28 +02:00
Leonid Shlyapnikov
003065fb70
Empty array of party IDs -- error, nothing found -- OK (#5456)
* Empty array of party IDs -- error, nothing found -- OK, as per @hurryabit

changelog_begin
changelog_end

* fixing test description
2020-04-07 13:57:06 +00:00
Leonid Shlyapnikov
644740f3c3
JSON API: Fixing ledger offset ordering (#5458)
changelog_begin
changelog_end
2020-04-07 09:11:41 +02:00
Gerolf Seitz
a2d785e3ee
Use com.daml as root package (#5343)
Packages com.digitalasset.daml and com.daml have been unified under com.daml

Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.

CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
2020-04-05 19:49:57 +02:00
Leonid Shlyapnikov
f71facf7cf
Parties error handling improvements (#5395)
* Return BadRequest(400) with warnings when all party IDs are unknown

changelog_begin
changelog_end

* cleanup
2020-04-03 09:54:43 -04:00
Leonid Shlyapnikov
62b7ab590a
Sync query error reporting improvements (#5326)
* non empty set of template IDs in domain.GetActiveContractsRequest

* Updating expected error message

changelog_begin
changelog_end

* Adding warnings field to domain.ErrorResponse

* Error messages constants

* using domain.AsyncResponse

* added todo

* Updating examples

changelog_begin
changelog_end

* Updating examples

* Fixing the case when head is part of the tail, as pointed by @S11001001

* Fixing the domain.SynResponse JSON reader,

report deserialization error when both 'errors' and 'result' fields
present, as pointed by @S11001001

* Addressing code review comments
2020-04-02 19:47:07 +00:00
Gerolf Seitz
329320bad9
Organize maven coordinates (#5272)
* Use com.daml as groupId for all artifacts

CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END

* Add 2 additional maven related checks to the release binary

1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId

* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
2020-04-01 11:41:18 +02:00
Leonid Shlyapnikov
2b701ea26c
Removing unused domain.JwtPayload JSON formatters, (#5325)
we have been relying on `ledger.api.auth.AuthServiceJWTCodec` since the
refactoring happened

changelog_begin
changelog_end
2020-03-31 21:52:24 +00:00
Leonid Shlyapnikov
6b1c90267a
Report UnknownTemplateIds warning before sending an error (#5318)
* report UnknownTemplateIds warning before sending an error

if no template IDs resolved, send warning before the error, this is for
consistency with the case when at least one ID resolved (we continue
processing request in this case).

changelog_begin
changelog_end

* adding an error message example

* change title

* clarifying what on-message means.

* formatting

* Addressing code review comments
2020-03-31 14:36:48 -04:00
Samir Talwar
52dbcf5d95
postgresql-testing: "Lock" ports when starting PostgreSQL. (#5310)
* http-json: Ask for a free port by specifying port 0.

This will avoid race conditions.

* bindings-akka-testing: Delete RandomPorts; it's unused.

* ports: Fix the Bazel test glob.

* ports: Move FreePort to postgresql-testing and add a test case.

* postgresql-testing: Make `FreePort.find()` return a `Port`.

* postgresql-testing: Lock free ports until the server starts.

This uses a `FileLock`, which should work well on all our
supported operating systems as long as everyone agrees to use it.

CHANGELOG_BEGIN
CHANGELOG_END

* postgresql-testing: Try to find a free port 10 times, then give up.

* postgresql-testing: Use a shared directory for the port lock.

* postgresql-testing: Try an alternative way of getting `%LOCALAPPDATA%`.
2020-03-31 17:13:02 +00:00
Leonid Shlyapnikov
703b588384
Error and disconnect when multiple requests coming over the same WebSocket connection (#5288)
* minor refactoring in preparation for the next test case

* allow only one input request + test, WIP

* allow only one input request + test, WIP

* Error when multiple requests coming over the same websocket connection

This never worked, we never returned anything when 2nd request received,
now we return an error and drop the connection.

changelog_begin
changelog_end

* update docs

* Fixing copyright header

* changing test description

* trying to clean wsMessageHandler up

* Reverting getTransactionSourceForParty changes
2020-03-30 18:08:54 -04:00
Remy
1b37f6c482
DAML-LF: redesign absolute contract ids (#5207)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 19:07:42 +01:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Stephen Compall
332d35a347
stop filtering "phantom archives" when it won't work; reenable it on request (#5003)
* ContractKeyStreamRequest with different types for whether offset-preceded or not

- should replace EnrichedContractKey as the WS StreamQuery type

* add the ContractKeyStreamRequest layer everywhere

* split StreamQuery parse from the other steps

* make StreamQuery type depend on the input JsValue

* only StreamQueryReader is a typeclass now

* scalafmt

* wrong type arg

* letting the request data and phantom archive removal choice flow

- from work with @leo-da

* finish threading request-derived phantom state to removal flow

- from work with @leo-da

* make it clear that hints are contract IDs in StreamQuery

- from work with @leo-da

* treat StreamQueryReader's type parameter fully as a phantom

- from work with @leo-da

* remove unused type alias

- from work with @leo-da

* test fetch resume, with and without various offsetHints

* document offsetHints

* missing ` in rst

- thanks @hurryabit

* rename offsetHint to contractIdAtOffset

CHANGELOG_BEGIN
- [JSON API - Experimental] New field ``contractIdAtOffset`` for fetch-by-key streams
  to restore proper archive filtering.
  See `issue #4511 <https://github.com/digital-asset/daml/issues/4511>`_.
CHANGELOG_END

* we never unify the two ContractKeyStreamRequest types

* doc update for contractIdAtOffset
2020-03-26 15:31:27 -04:00
Leonid Shlyapnikov
24e414cba0
Remove ledgerEffectiveTime, maximumRecordTime and default TTL config. (#5200)
changelog_begin

[JSON API - Experimental]
Remove ``ledgerEffectiveTime`` and ``maximumRecordTime`` fields from command ``meta``. Remove ``--default-ttl`` command line argument. The ledger time and TTL are automatically computed by the submission service now. See #5090.

changelog_end
2020-03-25 17:06:12 -04:00
Robert Autenrieth
a3ddde3a9d
Remove MRT and LET (#5172)
Contributes to #4194.
Closes #4231.
Closes #5022.
CHANGELOG_BEGIN
- [Ledger API] The protobuf fields ledger_effective_time and maximum_record_time have been removed from
  command submission. These fields were previously deprecated following the introduction
  of a new ledger time model. See issue `#4194 <https://github.com/digital-asset/daml/issues/4194>`__.
[Java Bindings] removed the usage of ledgerEffectiveTime and
maximumRecordTime, and instead added minLedgerTimeAbsolute and
minLedgerTimeRelative in CommandSubmissionClient and CommandClient
CHANGELOG_END
2020-03-25 16:50:27 +01:00
Stephen Compall
8d2c13cf43
generating GenMaps for testing (#5115)
* building a GenMap generator

* move Party Order to accessible place; test whether Name Equal is still in use

* add Order instance for SortedLookupList

* switch to Map for genMap's Inj; more Order instances

* remove all Order from TypedValueGenerators

* Revert "add Order instance for SortedLookupList"

This reverts commit 03a59a8249.

* moving the Equal instance means scenario-interpreter no longer direct-deps scalaz

* add a test using TypedValueGenerators GenMap

* remove Party Order

* refmt bazel

* remove stray import

* followup dep change to moving the Equal[Name] instance

* add changelog

CHANGELOG_BEGIN
CHANGELOG_END

* Name equal instance appears to be no longer used, but keeping anyway
2020-03-25 10:28:06 -04:00
Leonid Shlyapnikov
a6a1e75bc1
Streaming API error handling/reporting improvements (#5141)
* Streaming API error handling

* Stopping the stream after emitting the 1st error

* Cleaning up test cases, stricter type for non empty streaming fetch

* Update docs.

changelog_begin

[JSON API - Experimental] Change Streaming API error format to match synchronous API:
``{"status": <400 | 401 | 404 | 500>, "errors": <JSON array of strings> }``. See #4521

changelog_end

* a few words about warnings in the Streaming API

* a few words about warnings in the Streaming API

* Updating command formatting in the JavaScript example.
2020-03-24 19:32:55 +00:00
Stephen Compall
f2b068f0b5
remove List and TextMap support from the JSON query language (#5099)
* remove List and TextMap support from the JSON query language

CHANGELOG_BEGIN
- [JSON API - Experimental] Removed support for maps and lists from the query language.
  See `issue #4855 <https://github.com/digital-asset/daml/issues/4855>`_.
CHANGELOG_END

* GenMap won't be supported in general, remove FIXME

- thanks @leo-da

* more documentation on unsupported query types

- as suggested by @hurryabit; thanks
2020-03-20 16:22:22 -04:00
Leonid Shlyapnikov
ffabc2b3e5
Bumping up http-json tests size to large, adding some extra logging (#5109)
* Bumping up http-json tests size to large, adding some extra logging

* Removing the logging from prod code, adding some in tests.

changelog_begin
changelog_end
2020-03-20 16:48:11 +00:00
nickchapman-da
d81caa9568
rework daml_compile bazel rule (#5070)
Avoiding `damlc compile/package` commands (which we would like to deprecate), and replace with plain `damlc build` together with a post dar->dalf extraction step in the couple of places where we actually want the .dalf for testing.

changelog_begin
changelog_end
2020-03-18 23:06:55 +00:00
Leonid Shlyapnikov
29c340fed0
2nd part of stabilizing flaky websocket integration test (#5079)
* 2nd part of stabilizing flaky websocket integration test,

handling empty events, which we may receive a bunch in the end of the test
when there is no activity

changelog_begin
changelog_end

* fixing `Offset.semigroup`, thanks @S11001001!
2020-03-18 17:04:01 -04:00
Leonid Shlyapnikov
acb41c5829
Fetch by parties endpoint to use new getParties gRPC call (#5027)
* fetch by parties endpoint to use new getParties gRPC call

changelog_begin
changelog_end

* addressing code review comments, thanks @S11001001 and @SamirTalwar-DA

* Update ledger/ledger-api-client/src/main/scala/com/digitalasset/ledger/client/services/admin/PartyManagementClient.scala

thanks @S11001001

Co-Authored-By: Stephen Compall <stephen.compall@daml.com>

* addressing code review comments, thanks @S11001001 and @SamirTalwar-DA

* addressing code review comments, thanks @S11001001 and @SamirTalwar-DA

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-03-16 16:42:01 -04:00
Leonid Shlyapnikov
5c2ed56c8e
Allocate party JSON API endpoint (#4991)
* Allocate party JSON API endpoint

changelog_begin

[JSON API - Experimental]
Allocate party endpoint added: ``/v1/parties/allocate``. See #4638.

changelog_end

* cleanup

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/LedgerClientJwt.scala

Co-Authored-By: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-03-13 14:06:08 -04:00
Stephen Compall
0cb3541666
optional offset argument to WebSocket streams (#4819)
* capture the pattern of an optional stream input prefix

* allow LiveBegins to be appended

- should have been included with #4593

* StreamQuery will parse JsValues instead of Strings

* shortcut in HttpServiceTestFixture; clean up InsertDeleteStepTest

* compatibly add withOptPrefix stage to WS input

* parsing the offset argument

* add offset parsing to input stream, but drop it on the floor

* another form of the two cases

* percolate the starting offset to the next step

* percolate the starting offset to the next step

* percolate the starting offset to the next step

* use transactionsFollowingBoundary in ContractsService if begin offset specified

* integration test for scan from offset in WS query

* tool for catching the first offset in a stream

* documentation

* add changelog entry

CHANGELOG_BEGIN
- [JSON API - Experimental] WebSocket endpoints now support an optional offset argument.
  See `issue #4509 <https://github.com/digital-asset/daml/issues/4509>`_.
CHANGELOG_END

* move offset argument doc to later

- suggested by @hurryabit; thanks

* remove stray TODO

- pointed out by @leo-da; thanks

* redesign withOptPrefix to make the usage a little easier to follow

- suggested by @leo-da; thanks
2020-03-13 12:52:17 -04:00
Leonid Shlyapnikov
bc7d76c9d7
Mark JSON API tests flaky, failing on Windows consistently (#4978)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-12 21:32:46 +00:00
Leonid Shlyapnikov
f69100a4d4
Fixing blinking tests, multiple heartbeats are expected, change heartbeat format, live marker is gone (#4969)
* Fixing tests, multiple heartbeats are expected, change heartbeat format

changelog_begin
changelog_end

* scalafmt
2020-03-12 12:49:15 -04:00
Leonid Shlyapnikov
19f5b86c79
Websocket stream to emit last seen offset instead of heartbeat message (#4886)
* Capture lastSeenOffset in the @volatile var

CHANGELOG_BEGIN
[JSON API - Experimental] Websocket stream now emits last seen offset instead of the heartbeat message.
``{"heartbeat": "ping"}`` is replaced by ``{"events":[],"offset":"<last seen offset>"}``. See #4510.
CHANGELOG_END

* updating docs

* moving the last seen offset into the stream, WIP

* adding in-stream state

* minor docs

* cleanup the heartbeat logic

* minot cleanup

* Change live and heartbeat msg handling + some debug logging (to be removed)

* fixing ts tests, cleaning up

* Adding todo with the reference to the follow-up ticket

* Adding todo with the reference to the follow-up ticket
2020-03-11 19:53:05 +00:00
Gary Verhaegen
75c7d48d5b
bind localhost for json-api and scenarios (#4576)
Both were previously binding 0.0.0.0, which is inherently insecure. More
importantly to me, that meant running `bazel test //...` essentially
rendered my computer unusable for however long it took (which is
_long_), as it kept popping up focus-stealing dialogs about whether or
not I wanted to trust "java" to open an incoming network connection.

The ScenarioService does not seem to have an existing setup for CLI args
and my Scala-fu is not good enough to add one, so I just changed the
hard-coded path.

The JSON API already had an option, just with the wrong default. This is
technically a breaking change, but I'm hoping to pass it under the
"experimental" flag we still have on the JSON API.

CHANGELOG_BEGIN
- [JSON API - Experimental] As a security improvement, the JSON API
server will now bind on ``127.0.0.1`` by default. Previous behaviour was
to bind on ``0.0.0.0``; you can get that behaviour back by passing in
the (existing) flag ``--address 0.0.0.0``.

- [DAML SDK] The Scenario Service will now bind on ``127.0.0.1``. Previous
behaviour was to bind on ``0.0.0.0``.

CHANGELOG_END
2020-03-05 17:10:49 +01:00
Leonid Shlyapnikov
f9057bea19
handle dual LiveBegin case (#4815)
* make append of two liveness markers proper with respect to offsets

- never happens in a real stream, but satisfying some random tests
- test randomly harder

CHANGELOG_BEGIN
CHANGELOG_END

* move Monoid and Semigroup instances for Steps to main code

- suggested by @leo-da; thanks

* stray empty InsertDeleteStep
2020-03-04 14:59:26 -05:00
mergify[bot]
a40190d695
minor cleanup, using Option.exists instead of cata. (#4812)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-04 18:10:04 +00:00
mergify[bot]
c001b4f22c
test that ContractStreamStep#append preserves data sensibly (#4809)
* test that ContractStreamStep#append preserves data sensibly

CHANGELOG_BEGIN
CHANGELOG_END

* (C) header

* test more general associativity
2020-03-04 17:59:02 +00:00
Leonid Shlyapnikov
e80cda53ab
Fix for unknown template ID received from gRPC streaming API (#4800)
* Filter out contracts received from gRPC stream, if template IDs is unknown (pending package reload).

* minor cleanup

CHANGELOG_BEGIN

[JSON API - Experimental] Fix ``java.util.NoSuchElementException: key not found: TemplateId(xxx,yyy,zzz)`` error in the log for query and fetch endpoints. See #4761

CHANGELOG_END
2020-03-04 08:21:06 +00:00
Leonid Shlyapnikov
a3228bcea8
A few more map query tests (#4798)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-03 18:02:58 +00:00
Samir Talwar
a99156252d
libs-scala/ports: Wrap socket ports in a type, Port. (#4784)
* libs-scala/ports: Wrap socket ports in a type, `Port`.

* sandbox: Use `Port` for the API server port, and propagate.

CHANGELOG_BEGIN
CHANGELOG_END

* extractor: Use `Port` for the server port.

* ports: Make Port a compile-time class only.

* ports: Allow port 0; it can be specified by a user.

* ports: Publish to Maven Central.
2020-03-03 08:59:15 +00:00
Leonid Shlyapnikov
93329ac81d
Add create-and-exercise JSON API endpoint (#4754)
* CreateAndExerciseCommand JSON decoding

* create-and-exercise endpoint WIP

* create-and-exercise endpoint + test, WIP

* create-and-exercise test

* Update docs

CHANGELOG_BEGIN

[JSON API - Experimental]
Add ``/v1/create-and-exercise`` JSON API endpoint. See #4507.

CHANGELOG_END

* Cleaning up domain JSON encoder/decoder, trying to get rid of HasTemplate

* Cleaning up domain JSON encoder/decoder, trying to get rid of HasTemplate

* Addressing code review comments
2020-03-02 20:43:58 +00:00
Leonid Shlyapnikov
d93e4382f6
Extract ErrorOps, use liftErr instead of leftMap (#4730)
* Extract ErrorOps, use liftErr instead of leftMap
JSON error formatting cleanup

CHANGELOG_BEGIN
CHANGELOG_END

* Good we have tests for this stuff

* Apply https://github.com/scala/bug/issues/3664 work-around,

so JsonError can be used instead of JsonError.apply

* error formatting
2020-02-26 20:06:26 +00:00
Shayne Fletcher
66dd112960
Remove pragma 'daml 1.2' (#4702)
changelog_begin
- The pragma 'daml 1.2' is now optional.
changelog_end
2020-02-26 13:17:45 -05:00
Stephen Compall
79c6ee7339
add offset to live events; remove liveness marker (#4593)
* move BeginBookmark to util

* adding offsets to steps

* offsetAfter belongs in Txn, not InsertDeleteStep

* make transaction stream a ContractStreamStep.Txn stream

* add several ContractStreamStep append cases

* rewrite 'render' to emit offset in the right places

* make ContractStreamStep#append total again

* check for offset in a few tests

* revert useless whitespace changes

* missed argument

* simpler mapPreservingIds

* rewrite states for new "live" format

* remove invalidated "events" block structure assertions

* make shutdown in withHttpService deterministic, to try to catch race condition

* exhaustiveness checking somehow disabled; fixed fetch flow and all is well

* documentation and changelog

CHANGELOG_BEGIN
- [JSON API - Experimental] Remove ``{"live": true}`` marker from websocket streams;
  instead, live data is indicated by the presence of an "offset".
  See `issue #4593 <https://github.com/digital-asset/daml/pull/4593>`_.
CHANGELOG_END

* be more specific about what liveness marker may be in docs

* fix daml2ts websocket tests

* mention type rules for all cases in offset documentation
2020-02-26 10:17:21 -08:00
Leonid Shlyapnikov
d58bb4597e
Extend /v1/parties endpoint to support fetch by party IDs (#4680)
* Extend /party endpoint to allow specifying party ids

* Extend /party endpoint to allow specifying party ids

* Update docs

CHANGELOG_BEGIN

[JSON API - Experimental] Fetch Parties by their Identifiers. See #4512
``/v1/parties`` endpoint supports POST method now, which expects
a JSON array of party identifiers as an input.

CHANGELOG_END

* minor update

* minor update

* Use type alias

* Add warnings to the sync response

* test cases

* update docs, add test case for an empty input

* cleanup

* cleanup

* Addressing code review comments
2020-02-26 15:24:01 +00:00
Samir Talwar
9bdfbb0534
Avoid opening a server to the world when finding a free port. (#4683)
* Avoid opening a server to the world when finding a free port.

This is very annoying on macOS because we get a focus-stealing popup for
a split second, asking for permission to allow the server through the
firewall. The popup pretty much always disappears before it can even be
read, when the server is closed.

This is almost certainly not an attack vector, because:
  - we only do this in tests,
  - the server is open for only a few milliseconds,
  - nothing is served,
  - and finding the port is tricky, because it's effectively random.

Nevertheless, it's very annoying.

CHANGELOG_BEGIN
CHANGELOG_END

* Extract a Bazel package for finding free ports.

We seem to do it in 4 different places, which I think is enough to
remove the duplication.
2020-02-25 13:09:28 +00:00
Samir Talwar
f231ebc1b3
Sandbox-Next: Re-establish static time support. (#4581)
* kvutils: Extract a committer from the uses of `SubmissionValidator`.

This makes the clock injectable too.

* kvutils: Provide logging contexts in the `Runner`.

* sandbox: Remove the `StaticAllowBackwards` time provider type.

It's not used anywhere.

* sandbox: Fix warnings in CliSpec.

* sandbox: Ensure that we cannot specify both static and wall-clock time.

* sandbox-next: Crash if wall clock time is not specified.

* sandbox-next: Document more known issues in the new Sandbox.

* sandbox: Add a Clock (and some tests) to TimeServiceBackend.

* sandbox-next: Support static time.

CHANGELOG_BEGIN
- [Sandbox Next] Re-establish static time mode.
CHANGELOG_END

* ledger-on-(memory|sql): Expect a `() => Instant`, not a `Clock`.
2020-02-18 17:03:15 +00:00
Stephen Compall
eee815d3c3
json-api: add live-data boundary marker to WS contract streams (#4465)
* Inserts alias

* ContractStreamStep, extending InsertDeleteStep with a boundary

* partitionBimap, an IDS utility for partitionMap on both inserts and deletes

* make the flow setup of acsFollowingAndBoundary a little clearer

* use partitionBimap to simplify websocket response stream

* make the flow setup of transactionsFollowingBoundary a little clearer

* acsFollowingAndBoundary becoming a ContractStreamStep producer

* insertDeleteStepSource also becomes a ContractStreamStep producer

* porting StepAndErrors to ContractStreamStep

* remove Acs constructor of ContractStreamStep

- it's an interesting idea for future potential features, but not useful
  right now so just a needless complication

* adapt convertFilterContracts to presence of LiveBegin marker

* adapt removePhantomArchives to presence of LiveBegin marker

* test that live marker is emitted in the right place

* document liveness marker

* add changelog

CHANGELOG_BEGIN
- [JSON API - Experimental] Add ``{"live": true}`` to WebSocket streams
  to mark the beginning of "live" data.  See `issue #4461
  <https://github.com/digital-asset/daml/issues/4461>`_.

  This marker is a placeholder feature; `issue #4509 bookmarks in query
  streams <https://github.com/digital-asset/daml/issues/4509>`_ will
  obsolete this marker, after which it will no longer be emitted.  When
  building features on the marker, be aware of this forthcoming
  replacement.
CHANGELOG_END

* be a little more prepared for offsets
2020-02-14 10:41:41 -05:00
Samir Talwar
a589f4af0b
sandbox: Move more resource acquisition into the owner. (#4501)
* sandbox: Move more resource acquisition into the `owner`.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Reimplement SandboxClientResource as a resources.Resource.

* codegen: Use resources in TestUtil.

* sandbox: Manage PostgreSQL in tests with ResourceOwners.
2020-02-14 13:52:45 +00:00
Leonid Shlyapnikov
53b95ecb20
Expose total functions behind the endpoint mappings for DABL to re-use (#4462)
* Extract partial functions for DABL to re-use

CHANGELOG_BEGIN
CHANGELOG_END

* Actually DABL needs total functions,

they will map them to different endpoints
2020-02-10 15:00:03 -05:00
Leonid Shlyapnikov
a194ce2a34
Wrap streaming API events in JSON object (#4453)
* Wrap streaming API events in json object

CHANGELOG_BEGIN

[JSON API - Experimental] wrap Streaming API events in JSON object:
   ``{ "events": [ E1, E2, ... ] }``
   See: 4384

CHANGELOG_END

* cleanup
2020-02-07 17:17:42 -05:00
Leonid Shlyapnikov
98ab189e3e
Encode tuples as records in the JSON API docs, add a new test (#4452)
tuples are formatted as records in the response, so this way input format
matches the output

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-07 13:46:36 -05:00
Stephen Compall
8710673e71
with archived responses from WS streams, include template ID alongside contract ID (#4439)
* move InsertDeleteStep to util

* turn append cid argument into a typeclass

* use Cid for appendForgettingDeletes as well

* add secondary data to the deletes in InsertDeleteStep with new covariant tparam

* make StepAndErrors hold ArchivedContracts so ws archives look like exercise archives

* add changelog

CHANGELOG_BEGIN
- [JSON API - Experimental] The format of ``archived`` responses from WebSocket endpoints
  has changed to include template IDs, similar to exercise responses.
  See `issue #4383 <https://github.com/digital-asset/daml/issues/4383>`_.
CHANGELOG_END
2020-02-07 11:24:15 -05:00
Leonid Shlyapnikov
fcc2545a23
Make WebsocketConfig public, so DABL can use it (#4446)
WebSocketService is already public. This does not mean that the interfaces
are stabilized.

CHANGELOG_BEGIN
CHANGELOG_END
2020-02-07 15:56:21 +00:00
Leonid Shlyapnikov
fb6d8afd1f
Rename JSON API endpoints, add version prefix (#4440)
* Rename JSON API endpoints, add version prefix. Combining two breaking changes.

CHANGELOG_BEGIN

[JSON API - Experimental] Rename JSON API endpoints. See #4289 and #3145
    /command/create => /v1/create
    /command/exercise => /v1/exercise
    /contracts/lookup => /v1/fetch
    /contracts/search => /v1/query
    /contracts/searchForever => /v1/stream/query
    /contracts/lookupForever => /v1/stream/fetch
    /parties => /v1/parties

CHANGELOG_END

* minor update

* fix typo in the logging

* experimental disclosure update

* experimental disclosure update

* experimental disclosure update
2020-02-06 16:30:47 -05:00
Stephen Compall
2a2479a3fd
rename "contracts" to "events" in JSON API exercise response (#4436)
* rename "contracts" to "events" in JSON API exercise response

CHANGELOG_BEGIN
- [JSON API - Experimental] Exercise response field "contracts" renamed to "events".
  See `issue #4385 <https://github.com/digital-asset/daml/issues/4385>`_.
CHANGELOG_END

* more events in doc

- pointed out by @leo-da; thanks
2020-02-06 20:13:46 +00:00
Leonid Shlyapnikov
0583f57de6
Fetch by key stream test and doc (#4393)
* Remove phantom archive when streaming events filtered by contract key

* Add fetch test cases, WIP

* Add fetch test cases, WIP

* Add fetch test cases, WIP

* Add fetch test cases, WIP

* Test case to validate phantom archive filter for stream/fetch

* Use `scan` instead of `statefulMapConcat` to filter out phantom archives

advantage(???) immutable state vs mutable

* make `com.digitalasset.http.WebsocketEndpoints.handleWebsocketRequest` public

so DABL can call it directly, `private[http]` does not work since DABL
is under com.projectdabl, see #4190

* Fixing typos

* Scalafmt

* Scalafmt

* Removing unused function

* Update docs

* Remove unused type alias

* CHANGELOG_BEGIN

[JSON API - Experimental]
- Added streaming version of fetch by key: ``/stream/fetch``. See #4075.

CHANGELOG_END

* Address code review comments

* Add test case with todo to address consistent handling of empty requests

* Fixing merge conflicts

* Addressing code review comments

* Addressing code review comments

Use `partition` instead of consecutive intersect and diff
2020-02-06 18:45:53 +00:00
Leonid Shlyapnikov
c15cf3457e
CHANGELOG_BEGIN (#4434)
[JSON API - Experimental]
Added a command line option to allow overriding default TTL.
``--default-ttl <value>`` Optional Time to Live interval to set if not provided in the command. Examples: 30s, 1min, 1h. Defaults to 30 seconds

CHANGELOG_END
2020-02-06 12:36:36 -05:00
Stephen Compall
b85637115f
accept a list of {templateIds, query} as input to searchForever (#4363)
* add SearchForeverRequest with one-or-many JSON model

* the least structured way to flatten a query union into a single stream

* add somewhere for indices to go in StreamQuery

* unzipMap utility

* doc Positive

* collect indices for SearchForever predicate, and use only one Map

* interface for rendering positions

* finish propagating positives from the predicate to the rendering phase

* add matchedQueries to every `created` in the searchForever results

* test that matchedQueries indices are included in query stream blocks

* document query union

* add changelog

CHANGELOG_BEGIN
- [JSON API - Experimental] ``/contracts/searchForever`` accepts multiple queries,
  and includes with each ``created`` result the ``matchedQueries`` indicating which
  queries matched.
  See `issue #4363 <https://github.com/digital-asset/daml/pull/4363>`_.
CHANGELOG_END

* remove unused unzipMap
2020-02-06 10:23:38 -05:00
Stephen Compall
1c8bb9968a
ensure archive of a key happens to left of create of same key in each WebSocket result block (#4390)
* ensure archive of a key happens to left of create of same key in each WebSocket result block

CHANGELOG_BEGIN
- [JSON API - Experimental] In websocket endpoints, if a 'created' and 'archived' contract
  in the same result array share a contract key, the 'archived' is guaranteed to occur
  earlier in the array than the 'created'.
  See `issue #4354 <https://github.com/digital-asset/daml/issues/4354>`_.
CHANGELOG_END

* by key I mean contract key

- suggested by @hurryabit; thanks
2020-02-04 15:59:13 -05:00
Leonid Shlyapnikov
a754a2aa22
Add fetch by key websocket stream, part 1 of 2 (#4212)
* Add fetch by key stream websocket stream, WIP

* Add fetch by key stream websocket stream, WIP

* new formatting rules

* minor documentation cleanup

* refactoring required for the `lookupForever` stream

* Extracted `StreamQuery` typeclass

* Extracted `StreamQuery` typeclass

* Fix typo

* minor cleanup

* Using polymorphic `A: WebSocketService.StreamQuery` to get `lookupForever` functionality

* Remove unused methods

* Cleanup

* cleanup

* cleanup

* Merge remote-tracking branch 'origin/master' into leo-4075-fetch-by-key-stream

# Conflicts:
#	ledger-service/http-json/src/main/scala/com/digitalasset/http/WebSocketService.scala

* Rename the endpoint so it is compliant with #4289

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-31 21:49:45 +00:00
Stephen Compall
7882080207
warn on unknown template IDs in searchForever (#4312)
* in searchForever, warn on unknown template IDs as long as at least one is known

* remove unused resolveTemplateId

* factor out WS request parts in WS integration test

* factor out IOU create

* test early template ID warning in searchForever stream

* document warnings case for searchForever

CHANGELOG_BEGIN
- [JSON API - Experimental] Precede stream with warnings of unknown template IDs, if any,
  rather than failing outright.
  See `issue #4290 <https://github.com/digital-asset/daml/issues/4290>`_.
CHANGELOG_END
2020-01-30 16:13:40 -05:00
Leonid Shlyapnikov
9f57994f2d
Fix the case when start offset > end offset (#4283)
* Fix the case when start offset > end offset

Actually skip the call completely, all needed data supposed to be
already in the query-store, fetched by a concurrent thread/client.
Moving the check into the `getCreatesAndArchivesSince` call

CHANGELOG_BEGIN
CHANGELOG_END

* Remove flaky, want to see the test failing.

* reformat

* Add AbsoluteOffsetOrdering so we can handle both offset formats

* Code review comments

* generators WIP

* generators WIP

* testing the laws

* remove `implicitly` call
2020-01-30 15:35:59 -05:00
Stephen Compall
ca5c646406
test that step conflation yields consistent results with unconflated streams (#4253)
* test that step conflation yields consistent results with unconflated streams

CHANGELOG_BEGIN
CHANGELOG_END

* check that relevant right deletes are definitely included

* use a smaller set of potential Cids to have better overlap testing
2020-01-30 12:55:40 -05:00
Leonid Shlyapnikov
cb465837ef
Fixing package reload scheduling, thanks @lima-da for providing the fix (#4254)
CHANGELOG_BEGIN

[JSON API - Experimental]
Fixing package reload scheduling. The bug was introduced when upgraded
to the most recent Akka library. Also using `scheduleWithFixedDelay` and
awaiting for the reload future to complete. This is to guarantee that we
don't trigger multiple reloads simultaneously. Package reload is
thread-safe though.

CHANGELOG_END
2020-01-30 10:46:38 -05:00
Andreas Herrmann
87b84772ab
Mark //ledger-service/http-json:tests as flaky (#4282)
Latest failure
https://dev.azure.com/digitalasset/daml/_build/results?buildId=30191&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=f58be928-45a1-58e7-c1ad-04869565b3f4&l=6805

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-30 13:22:32 +00:00
Leonid Shlyapnikov
4be77fd889
Temporary work-around, can't reproduce locally: LedgerBegin > LedgerEnd (#4271)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-29 15:29:17 -05:00
Samir Talwar
4d324f6aaf
Run scalafmt on all files and fix the errors. (#4260)
Unfortunately, downgrading `scalafmt` resulted in files that would be
reformatted to invalid code, because lines would be merged but trailing
commas would be left in.

I've manually run it everywhere and fixed these issues so people don't
have to fix them as they go.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-01-29 16:15:49 +00:00
Stephen Compall
d6a20cba34
Avoid ending at different ledger ends for multi-template-ID queries (#4153)
* make searchInMemory do only one transaction stream

* drop unneeded deletes from memory while doing searchInMemory

* remove unused functions

* rename OffsetBookmark to BeginBookmark

* add AtAbsolute option to GetCreatesAndArchivesSince

* add GetTermination raw command

* propagate GetTermination to ContractsFetch

* remove fetching from searchDbOneTpId_

* fetch multiple contract IDs for DB persistence always to a fixed offset

* bugfix and cleanup; no changelog relevant

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-28 17:50:18 -05:00
Leonid Shlyapnikov
231b9c6394 Change Exercise by Key to send ExerciseByKeyCommand (#4209)
instead of lookup by key + ExerciseCommand with retrieved contract ID.

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-24 17:59:47 +00:00
Samir Talwar
f3f7def5d2
postgresql-testing: Extract PostgresAround from the Sandbox t… (#4197)
* postgresql-testing: Extract `PostgresAround` from the Sandbox tests.

CHANGELOG_BEGIN
CHANGELOG_END

* postgresql-testing: Split into several files.

* postgresql-testing: Pull out the host and database names into variables.

* postgresql-testing: Don't publish to Maven.

I have no idea why the build wasn't failing before.

* postgresql-testing: Assimilate the `@postgresql_dev_env` dependency.
2020-01-24 12:48:36 +01:00
Leonid Shlyapnikov
9ab419f96b CHANGELOG_BEGIN (#4192)
[JSON API - Experimental] Rename ``argument`` field to ``payload`` in the ``command/create`` request. See #4189.

CHANGELOG_END
2020-01-24 09:04:03 +01:00
Leonid Shlyapnikov
78b29852d3
JSON API documentation update (#4173)
Document error handling

Document query store

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-23 14:54:12 -05:00
Robert Autenrieth
3f95c2ec13
Change sandbox JWT format (#4070)
* Introduce new JWT payload format
... the reader still supports old formats

CHANGELOG_BEGIN
[Sandbox] The sandbox uses a new payload format for authentication tokens (JWTs).
          The old format is deprecated, but still works.
[JSON API] The HTTP JSON API now uses the same payload format for authentication tokens as the sandbox.
           The old format is deprecated, but still works.
CHANGELOG_END

* Add helper function for getting token party

* Support sandbox tokens in JSON API

* Add warning for deprecated formats

* Update documentation

* Add explicit test for new format

* Update JSON API documentation

* Fix test
2020-01-22 14:26:57 +01:00
Leonid Shlyapnikov
c8c55c4408
Exercise by key (#4049)
* Implement exercise by key

ExerciseCommand got a new required element: `reference` of polymorphic type.

* Add test case: exercise Archive by contractKey

* Add test case for ExerciseCommand JSON protocol

* flatten contract reference in ExerciseCommand JSON protocol

* formatting

* Update exercise by key

* Update documentation

CHANGELOG_BEGIN

- [JSON API - Experimental] Support Exercise by Key. See #4009.

CHANGELOG_END

* Address code review comments
2020-01-18 09:06:52 -05:00
Stephen Compall
3d183b7f3e
for searchForever, use a similar response format to exercise results (#4078)
* for searchForever, use a similar response format to exercise results

CHANGELOG_BEGIN
- [JSON API - Experimental] Response format in ``searchForever`` changed to be more like ``exercise``.
  See `issue #4072 <https://github.com/digital-asset/daml/issues/4072>`__.
CHANGELOG_END

* typo left from add->created replacement

Co-Authored-By: Martin Huschenbett <martin.huschenbett@posteo.me>

Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
2020-01-17 15:44:25 -05:00
Stephen Compall
0520fdfa84
in query argument, rename %templates to templateIds, and nest query under 'query' field (#4082)
* in query argument, rename %templates to templateIds, and nest query under 'query' field

CHANGELOG_BEGIN
- [JSON API - Experimental] In 'search' endpoint arguments, %templates is now templateIds.
  Additionally, all contract query fields must occur under 'query'.
  See `issue #3450 <https://github.com/digital-asset/daml/issues/3450>`__.
CHANGELOG_END

* fix other old query format usages
2020-01-17 14:34:27 -05:00
Remy
21a17fefaa
daml-lf: new cryptographic hash for values (#3849)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-16 22:11:57 +01:00
Brian Healey
a5490658ac
EC256 rather than EC512 algo (#4019)
* EC256 rather than EC512 algo
CHANGELOG_BEGIN
[Ledger API Authorization]Support EC256 algorithm for JWT rather than EC512
CHANGELOG_END

* support both 256 and 512 ECDSA algo

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* scalafmt

* correct comment, avoid unnecessary private def

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* correct BAZEL formatter
2020-01-16 08:55:55 -05:00
Stephen Compall
3ae0438b73
websocket variant of query endpoint (#3936)
* minor improvements on websocket and add websocketIT

* add it for websocket, and support config args

* add one more test case

* make ws config optional

* avoid fromTryCatchNonFatal when derivative already exists

* spelling and missing type parameter

* use richer Matchers in WebsocketServiceIntegrationTest

* scalafmt

* IDEs may love braces but we don't

* utility for simplifying FanOutShape2s; use in ContractsService

* split matSecondOut into generalization; make compile again

* match matSecondOut utility with standard utility methods

* spelling

* getCreatesAndArchivesSince doesn't need to query the transaction boundary

* boolean newtype utility

* split up transactionMessageHandler into components

* decodeAndParsePayload passes through the Jwt

* clean up config and default WS config

* take multiple template IDs for insertDeleteStepSource

* replace websocket return with {errors, add, remove}, based on acsFollowingAndBoundary

* parse ValuePredicate in websocket

* remove unused lfvToJson

* nominal internal state for emitted WS steps-and-errors

* missing copyright headers

* add filtering to convertFilterContracts

* add step conflation to websocket output

* move conflation to static function

* rename /transactions endpoint to /contracts/searchForever

* empty requests are not allowed; numConns is per-service

* option for GetCreatesAndArchiveSince to not terminate; use in WebsocketService

* start of searchForever documentation

* stub searchForever longer test

* use valueOr

* don't run all other tests again with WebsocketServiceIntegrationTest

* start of websocket delta test

* solve init order problem with AbstractHttpServiceIntegrationTestFuns

- previous order caused test set to be cleared; mutation is intuitive
  for sure!

* full flow test, fails for lack of create/exercise yet

* passing full flow test

* full documentation examples

* rename add/remove to created/archived

* cleaner NewBoolean.Named

* document heartbeats

* document subprotocols for searchForever

* note about the tests mysteriously terminating

* ensure create has happened before attempting query in tests

* reorganize multi-step WS test so its states and assertions are clearer

* filter out heartbeats in raw string tests

* factor out ContractDelta

* make exercisePayload easier to read

* filter out heartbeats in conversation test

* remove type lambda

* accept chunked queries

- clients may not be in control of how query bodies are delivered to the
  server, so we should be agnostic in that respect

* add changelog

CHANGELOG_BEGIN

- [JSON API - Experimental] WebSocket contract search at ``/contracts/searchForever``.
  See `issue #3936 <https://github.com/digital-asset/daml/pull/3936>`_.

CHANGELOG_END

* adapt to #3991 template ID strings

* adapt to #3971 argument -> payload

* fix create command for test (string template ID redux)

* adapt to #4014 ResolveTemplateId change

* update copyright headers

* rebuild WS example output to match latest changes

- thanks @leo-da

* SeqOps is not a safe name

* don't need breakOut anymore

* use util library form of partitionMap

- thanks @leo-da for pointing it out

Co-authored-by: lima-da <54044170+lima-da@users.noreply.github.com>
2020-01-15 18:11:45 -05:00
Leonid Shlyapnikov
3381681874 /contracts/search should return HTTP 200 and report unresolved Template IDs in the warnings element (#4014)
* WIP

* test cases updated

* `PackageService.resolveTemplateId` returns `Option[TemplateId]` now

used to return `Error \/ TemplateId`. There are scenarios when unresolved
TemplateID is a warning and not an error, which was the initial design

* CHANGELOG_BEGIN

[JSON API - Experimental]
``/contracts/search`` endpoint reports unresolved template IDs as warnings, see #3771::

    {
        "warnings": {
            "unknownTemplateIds": ["UnknownModule:UnknownEntity"]
        },
        "result": [{...}, ...],
        "status": 200
    }

CHANGELOG_END

* Addressing codereview comments

* `partitionMap` is now part of `http.util.Collections.SeqOps`
2020-01-14 20:17:04 +00:00
Leonid Shlyapnikov
93216a0d71
Template ID to JsString JSON serialization (#3991)
* test cases: domain.TemplateId JSON serialization to JsString

* JSON protocol updated

* Fixing json-api test cases

* test cases: domain.TemplateId JSON serialization to JsString

* JSON protocol updated

* Fixing json-api test cases

* Adapt daml2ts and support libraries

* Update documentation

CHANGELOG_BEGIN

[JSON API - Experimental]
- Use JSON string to encode template IDs. Use colon (``:``) to separate parts of the ID.
  The request format, with optional package ID:
  - "<module>:<entity>"
  - "<package ID>:<module>:<entity>"
  The response always contains fully qualified template ID in the format:
  - "<package ID>:<module>:<entity>"
  See #3647.

CHANGELOG_END

* Minor documentation formatting changes.

Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
2020-01-09 14:43:41 -05:00
Brian Healey
95fbf5c921
ECDA512 algorithm support (#3953)
* ECDA512 algorithm support

* ECDA512

* happy day test for ECDA512 algorithm

* failure test for wrong key for ECDA512 algorithm

* add ability to use EC cert file

* update docs

* scalafmt

* Correct documentation

CHANGELOG_BEGIN
[Ledger API Authorization] Support elliptic curve algorithm for JWT verification
CHANGELOG_END

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* correct docs warning
2020-01-09 07:25:04 -05:00
Nikolai Cherkezishvili
49b90bfb36
Engine: Improve performance of critical parts (#3947) 2020-01-09 19:18:55 +11:00
Leonid Shlyapnikov
e75b98351c Aligning DB contract table with domain.ActiveContract class (#3989)
* Aligning DB contract table with domain.ActiveContract class

adding key,signatories, observers and agreement_text to DB contract table
removing witnessParties

Reading signatories and observers from contracts table

Updating doc, removing witnessParties

Address code review comments, thanks @S11001001

CHANGELOG_BEGIN

[JSON API - Experimental]
- Align ``contract`` table with ``domain.ActiveContract`` class.
  The database schema has changed, if using ``--query-store-jdbc-config``,
  you must rebuild the database by adding ``,createSchema=true``. See #3754.
- ``witnessParties`` field is removed from all JSON responses.

CHANGELOG_END

* Fix TypeScript domain models

remove witnessParties and workflowId fields. workflowId has be removed
from JSON output a while ago.
2020-01-08 20:39:57 +00:00
Leonid Shlyapnikov
2c154f2299 Rename argument to payload in domain.ActiveContract (#3971)
CHANGELOG_BEGIN

[JSON API - Experimental] Rename ``argument`` in active contract to ``payload``. See #3826.

CHANGELOG_END
2020-01-08 13:59:27 +01:00
Leonid Shlyapnikov
29d5995454 CHANGELOG_BEGIN (#3969)
- [JSON API - Experimental] When contract is not found, ``/contracts/lookup`` endpoint returns
  ``{"status":200,"result":null}`` instead of ``{"status":200,"result":{}}``.
  This is consistent with how DAML-LF JSON Encoding spec treats Optional.

CHANGELOG_END
2020-01-07 18:51:00 +00:00
Andreas Herrmann
f33e79c787
Remove unused dependencies to da_scala_library (#3938)
* Inline all scala_library dependencies

* Run //:buildifier-fix

* TMP scala_library_suite --> scala_library

* da_scala_library: Enable unused dependency checker

* scala_library: Enable unused dependency checker

* //daml-lf/data:data

* //daml-lf/engine:engine

* //ledger-api/rs-grpc-akka:rs-grpc-akka

* //ledger/participant-state:participant-state

* //ledger/ledger-api-client:ledger-api-client

* //scala-protoc-plugins/scala-logging:scala-logging-lib

* //ledger/ledger-api-scala-logging:ledger-api-scala-logging

* //ledger/ledger-api-common:ledger-api-common

* //ledger-service/utils:utils

* //ledger-service/jwt:jwt

* //ledger/ledger-api-auth:ledger-api-auth

* //extractor:extractor

* //daml-assistant/scala-daml-project-config:scala-daml-project-config

* //language-support/codegen-common:codegen-common

* //language-support/scala/codegen:codegen

* //language-support/codegen-main:codegen-main-lib

* //ledger-service/db-backend:db-backend

* //ledger-service/http-json:http-json

* //daml-lf/scenario-interpreter:scenario-interpreter

* //ledger/sandbox:sandbox

* //navigator/backend:navigator-library

* //daml-assistant/daml-sdk:sdk-lib

* //daml-lf/data-scalacheck:data-scalacheck

* //daml-script/test:test-lib

* //ledger/ledger-api-common:ledger-api-common-scala-tests-lib

* //ledger/test-common:test-common

* //ledger/sandbox:sandbox-scala-tests-lib

* //extractor:extractor-scala-tests-lib

* //language-support/java/bindings:bindings-java-tests-lib

* //language-support/java/bindings-rxjava:bindings-java-tests-lib

* //language-support/scala/bindings-akka-testing:bindings-akka-testing

* //language-support/scala/codegen-testing:codegen-testing

* //language-support/scala/codegen-sample-app:daml-lf-codegen-sample-app

* //language-support/scala/codegen-sample-app:daml-lf-codegen-sample-app-testing

* //language-support/scala/codegen-testing:codegen-testing-testing

* //ledger-api/sample-service:sample-service

* //ledger-api/rs-grpc-akka:rs-grpc-akka-tests-lib

* //ledger/ledger-api-test-tool:ledger-api-test-tool-lib

* //ledger/ledger-api-test-tool:ledger-api-test-tool-tests

* //ledger/participant-state/kvutils:kvutils

* //ledger/sandbox:ledger-api-server

* //ledger/sandbox-perf:sandbox-perf-lib

* //navigator/backend:navigator-tests-library

* UNDO scala_library_suite --> scala_library

This reverts commit ab3eb1ae23139e2ec431ab4551fbb0371e0354e1.

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-06 18:14:21 +01:00