Commit Graph

122 Commits

Author SHA1 Message Date
Stephen Compall
2a2479a3fd
rename "contracts" to "events" in JSON API exercise response (#4436)
* rename "contracts" to "events" in JSON API exercise response

CHANGELOG_BEGIN
- [JSON API - Experimental] Exercise response field "contracts" renamed to "events".
  See `issue #4385 <https://github.com/digital-asset/daml/issues/4385>`_.
CHANGELOG_END

* more events in doc

- pointed out by @leo-da; thanks
2020-02-06 20:13:46 +00:00
Leonid Shlyapnikov
0583f57de6
Fetch by key stream test and doc (#4393)
* Remove phantom archive when streaming events filtered by contract key

* Add fetch test cases, WIP

* Add fetch test cases, WIP

* Add fetch test cases, WIP

* Add fetch test cases, WIP

* Test case to validate phantom archive filter for stream/fetch

* Use `scan` instead of `statefulMapConcat` to filter out phantom archives

advantage(???) immutable state vs mutable

* make `com.digitalasset.http.WebsocketEndpoints.handleWebsocketRequest` public

so DABL can call it directly, `private[http]` does not work since DABL
is under com.projectdabl, see #4190

* Fixing typos

* Scalafmt

* Scalafmt

* Removing unused function

* Update docs

* Remove unused type alias

* CHANGELOG_BEGIN

[JSON API - Experimental]
- Added streaming version of fetch by key: ``/stream/fetch``. See #4075.

CHANGELOG_END

* Address code review comments

* Add test case with todo to address consistent handling of empty requests

* Fixing merge conflicts

* Addressing code review comments

* Addressing code review comments

Use `partition` instead of consecutive intersect and diff
2020-02-06 18:45:53 +00:00
Leonid Shlyapnikov
c15cf3457e
CHANGELOG_BEGIN (#4434)
[JSON API - Experimental]
Added a command line option to allow overriding default TTL.
``--default-ttl <value>`` Optional Time to Live interval to set if not provided in the command. Examples: 30s, 1min, 1h. Defaults to 30 seconds

CHANGELOG_END
2020-02-06 12:36:36 -05:00
Stephen Compall
b85637115f
accept a list of {templateIds, query} as input to searchForever (#4363)
* add SearchForeverRequest with one-or-many JSON model

* the least structured way to flatten a query union into a single stream

* add somewhere for indices to go in StreamQuery

* unzipMap utility

* doc Positive

* collect indices for SearchForever predicate, and use only one Map

* interface for rendering positions

* finish propagating positives from the predicate to the rendering phase

* add matchedQueries to every `created` in the searchForever results

* test that matchedQueries indices are included in query stream blocks

* document query union

* add changelog

CHANGELOG_BEGIN
- [JSON API - Experimental] ``/contracts/searchForever`` accepts multiple queries,
  and includes with each ``created`` result the ``matchedQueries`` indicating which
  queries matched.
  See `issue #4363 <https://github.com/digital-asset/daml/pull/4363>`_.
CHANGELOG_END

* remove unused unzipMap
2020-02-06 10:23:38 -05:00
Stephen Compall
1c8bb9968a
ensure archive of a key happens to left of create of same key in each WebSocket result block (#4390)
* ensure archive of a key happens to left of create of same key in each WebSocket result block

CHANGELOG_BEGIN
- [JSON API - Experimental] In websocket endpoints, if a 'created' and 'archived' contract
  in the same result array share a contract key, the 'archived' is guaranteed to occur
  earlier in the array than the 'created'.
  See `issue #4354 <https://github.com/digital-asset/daml/issues/4354>`_.
CHANGELOG_END

* by key I mean contract key

- suggested by @hurryabit; thanks
2020-02-04 15:59:13 -05:00
Leonid Shlyapnikov
a754a2aa22
Add fetch by key websocket stream, part 1 of 2 (#4212)
* Add fetch by key stream websocket stream, WIP

* Add fetch by key stream websocket stream, WIP

* new formatting rules

* minor documentation cleanup

* refactoring required for the `lookupForever` stream

* Extracted `StreamQuery` typeclass

* Extracted `StreamQuery` typeclass

* Fix typo

* minor cleanup

* Using polymorphic `A: WebSocketService.StreamQuery` to get `lookupForever` functionality

* Remove unused methods

* Cleanup

* cleanup

* cleanup

* Merge remote-tracking branch 'origin/master' into leo-4075-fetch-by-key-stream

# Conflicts:
#	ledger-service/http-json/src/main/scala/com/digitalasset/http/WebSocketService.scala

* Rename the endpoint so it is compliant with #4289

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-31 21:49:45 +00:00
Stephen Compall
7882080207
warn on unknown template IDs in searchForever (#4312)
* in searchForever, warn on unknown template IDs as long as at least one is known

* remove unused resolveTemplateId

* factor out WS request parts in WS integration test

* factor out IOU create

* test early template ID warning in searchForever stream

* document warnings case for searchForever

CHANGELOG_BEGIN
- [JSON API - Experimental] Precede stream with warnings of unknown template IDs, if any,
  rather than failing outright.
  See `issue #4290 <https://github.com/digital-asset/daml/issues/4290>`_.
CHANGELOG_END
2020-01-30 16:13:40 -05:00
Leonid Shlyapnikov
9f57994f2d
Fix the case when start offset > end offset (#4283)
* Fix the case when start offset > end offset

Actually skip the call completely, all needed data supposed to be
already in the query-store, fetched by a concurrent thread/client.
Moving the check into the `getCreatesAndArchivesSince` call

CHANGELOG_BEGIN
CHANGELOG_END

* Remove flaky, want to see the test failing.

* reformat

* Add AbsoluteOffsetOrdering so we can handle both offset formats

* Code review comments

* generators WIP

* generators WIP

* testing the laws

* remove `implicitly` call
2020-01-30 15:35:59 -05:00
Stephen Compall
ca5c646406
test that step conflation yields consistent results with unconflated streams (#4253)
* test that step conflation yields consistent results with unconflated streams

CHANGELOG_BEGIN
CHANGELOG_END

* check that relevant right deletes are definitely included

* use a smaller set of potential Cids to have better overlap testing
2020-01-30 12:55:40 -05:00
Leonid Shlyapnikov
cb465837ef
Fixing package reload scheduling, thanks @lima-da for providing the fix (#4254)
CHANGELOG_BEGIN

[JSON API - Experimental]
Fixing package reload scheduling. The bug was introduced when upgraded
to the most recent Akka library. Also using `scheduleWithFixedDelay` and
awaiting for the reload future to complete. This is to guarantee that we
don't trigger multiple reloads simultaneously. Package reload is
thread-safe though.

CHANGELOG_END
2020-01-30 10:46:38 -05:00
Andreas Herrmann
87b84772ab
Mark //ledger-service/http-json:tests as flaky (#4282)
Latest failure
https://dev.azure.com/digitalasset/daml/_build/results?buildId=30191&view=logs&j=0da5d1d9-276d-5173-c4c4-9d4d4ed14fdb&t=f58be928-45a1-58e7-c1ad-04869565b3f4&l=6805

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-30 13:22:32 +00:00
Leonid Shlyapnikov
4be77fd889
Temporary work-around, can't reproduce locally: LedgerBegin > LedgerEnd (#4271)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-29 15:29:17 -05:00
Samir Talwar
4d324f6aaf
Run scalafmt on all files and fix the errors. (#4260)
Unfortunately, downgrading `scalafmt` resulted in files that would be
reformatted to invalid code, because lines would be merged but trailing
commas would be left in.

I've manually run it everywhere and fixed these issues so people don't
have to fix them as they go.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-01-29 16:15:49 +00:00
Stephen Compall
d6a20cba34
Avoid ending at different ledger ends for multi-template-ID queries (#4153)
* make searchInMemory do only one transaction stream

* drop unneeded deletes from memory while doing searchInMemory

* remove unused functions

* rename OffsetBookmark to BeginBookmark

* add AtAbsolute option to GetCreatesAndArchivesSince

* add GetTermination raw command

* propagate GetTermination to ContractsFetch

* remove fetching from searchDbOneTpId_

* fetch multiple contract IDs for DB persistence always to a fixed offset

* bugfix and cleanup; no changelog relevant

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-28 17:50:18 -05:00
Leonid Shlyapnikov
231b9c6394 Change Exercise by Key to send ExerciseByKeyCommand (#4209)
instead of lookup by key + ExerciseCommand with retrieved contract ID.

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-24 17:59:47 +00:00
Samir Talwar
f3f7def5d2
postgresql-testing: Extract PostgresAround from the Sandbox t… (#4197)
* postgresql-testing: Extract `PostgresAround` from the Sandbox tests.

CHANGELOG_BEGIN
CHANGELOG_END

* postgresql-testing: Split into several files.

* postgresql-testing: Pull out the host and database names into variables.

* postgresql-testing: Don't publish to Maven.

I have no idea why the build wasn't failing before.

* postgresql-testing: Assimilate the `@postgresql_dev_env` dependency.
2020-01-24 12:48:36 +01:00
Leonid Shlyapnikov
9ab419f96b CHANGELOG_BEGIN (#4192)
[JSON API - Experimental] Rename ``argument`` field to ``payload`` in the ``command/create`` request. See #4189.

CHANGELOG_END
2020-01-24 09:04:03 +01:00
Leonid Shlyapnikov
78b29852d3
JSON API documentation update (#4173)
Document error handling

Document query store

CHANGELOG_BEGIN
CHANGELOG_END
2020-01-23 14:54:12 -05:00
Robert Autenrieth
3f95c2ec13
Change sandbox JWT format (#4070)
* Introduce new JWT payload format
... the reader still supports old formats

CHANGELOG_BEGIN
[Sandbox] The sandbox uses a new payload format for authentication tokens (JWTs).
          The old format is deprecated, but still works.
[JSON API] The HTTP JSON API now uses the same payload format for authentication tokens as the sandbox.
           The old format is deprecated, but still works.
CHANGELOG_END

* Add helper function for getting token party

* Support sandbox tokens in JSON API

* Add warning for deprecated formats

* Update documentation

* Add explicit test for new format

* Update JSON API documentation

* Fix test
2020-01-22 14:26:57 +01:00
Leonid Shlyapnikov
c8c55c4408
Exercise by key (#4049)
* Implement exercise by key

ExerciseCommand got a new required element: `reference` of polymorphic type.

* Add test case: exercise Archive by contractKey

* Add test case for ExerciseCommand JSON protocol

* flatten contract reference in ExerciseCommand JSON protocol

* formatting

* Update exercise by key

* Update documentation

CHANGELOG_BEGIN

- [JSON API - Experimental] Support Exercise by Key. See #4009.

CHANGELOG_END

* Address code review comments
2020-01-18 09:06:52 -05:00
Stephen Compall
3d183b7f3e
for searchForever, use a similar response format to exercise results (#4078)
* for searchForever, use a similar response format to exercise results

CHANGELOG_BEGIN
- [JSON API - Experimental] Response format in ``searchForever`` changed to be more like ``exercise``.
  See `issue #4072 <https://github.com/digital-asset/daml/issues/4072>`__.
CHANGELOG_END

* typo left from add->created replacement

Co-Authored-By: Martin Huschenbett <martin.huschenbett@posteo.me>

Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
2020-01-17 15:44:25 -05:00
Stephen Compall
0520fdfa84
in query argument, rename %templates to templateIds, and nest query under 'query' field (#4082)
* in query argument, rename %templates to templateIds, and nest query under 'query' field

CHANGELOG_BEGIN
- [JSON API - Experimental] In 'search' endpoint arguments, %templates is now templateIds.
  Additionally, all contract query fields must occur under 'query'.
  See `issue #3450 <https://github.com/digital-asset/daml/issues/3450>`__.
CHANGELOG_END

* fix other old query format usages
2020-01-17 14:34:27 -05:00
Remy
21a17fefaa
daml-lf: new cryptographic hash for values (#3849)
CHANGELOG_BEGIN
CHANGELOG_END
2020-01-16 22:11:57 +01:00
Brian Healey
a5490658ac
EC256 rather than EC512 algo (#4019)
* EC256 rather than EC512 algo
CHANGELOG_BEGIN
[Ledger API Authorization]Support EC256 algorithm for JWT rather than EC512
CHANGELOG_END

* support both 256 and 512 ECDSA algo

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* scalafmt

* correct comment, avoid unnecessary private def

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* correct BAZEL formatter
2020-01-16 08:55:55 -05:00
Stephen Compall
3ae0438b73
websocket variant of query endpoint (#3936)
* minor improvements on websocket and add websocketIT

* add it for websocket, and support config args

* add one more test case

* make ws config optional

* avoid fromTryCatchNonFatal when derivative already exists

* spelling and missing type parameter

* use richer Matchers in WebsocketServiceIntegrationTest

* scalafmt

* IDEs may love braces but we don't

* utility for simplifying FanOutShape2s; use in ContractsService

* split matSecondOut into generalization; make compile again

* match matSecondOut utility with standard utility methods

* spelling

* getCreatesAndArchivesSince doesn't need to query the transaction boundary

* boolean newtype utility

* split up transactionMessageHandler into components

* decodeAndParsePayload passes through the Jwt

* clean up config and default WS config

* take multiple template IDs for insertDeleteStepSource

* replace websocket return with {errors, add, remove}, based on acsFollowingAndBoundary

* parse ValuePredicate in websocket

* remove unused lfvToJson

* nominal internal state for emitted WS steps-and-errors

* missing copyright headers

* add filtering to convertFilterContracts

* add step conflation to websocket output

* move conflation to static function

* rename /transactions endpoint to /contracts/searchForever

* empty requests are not allowed; numConns is per-service

* option for GetCreatesAndArchiveSince to not terminate; use in WebsocketService

* start of searchForever documentation

* stub searchForever longer test

* use valueOr

* don't run all other tests again with WebsocketServiceIntegrationTest

* start of websocket delta test

* solve init order problem with AbstractHttpServiceIntegrationTestFuns

- previous order caused test set to be cleared; mutation is intuitive
  for sure!

* full flow test, fails for lack of create/exercise yet

* passing full flow test

* full documentation examples

* rename add/remove to created/archived

* cleaner NewBoolean.Named

* document heartbeats

* document subprotocols for searchForever

* note about the tests mysteriously terminating

* ensure create has happened before attempting query in tests

* reorganize multi-step WS test so its states and assertions are clearer

* filter out heartbeats in raw string tests

* factor out ContractDelta

* make exercisePayload easier to read

* filter out heartbeats in conversation test

* remove type lambda

* accept chunked queries

- clients may not be in control of how query bodies are delivered to the
  server, so we should be agnostic in that respect

* add changelog

CHANGELOG_BEGIN

- [JSON API - Experimental] WebSocket contract search at ``/contracts/searchForever``.
  See `issue #3936 <https://github.com/digital-asset/daml/pull/3936>`_.

CHANGELOG_END

* adapt to #3991 template ID strings

* adapt to #3971 argument -> payload

* fix create command for test (string template ID redux)

* adapt to #4014 ResolveTemplateId change

* update copyright headers

* rebuild WS example output to match latest changes

- thanks @leo-da

* SeqOps is not a safe name

* don't need breakOut anymore

* use util library form of partitionMap

- thanks @leo-da for pointing it out

Co-authored-by: lima-da <54044170+lima-da@users.noreply.github.com>
2020-01-15 18:11:45 -05:00
Leonid Shlyapnikov
3381681874 /contracts/search should return HTTP 200 and report unresolved Template IDs in the warnings element (#4014)
* WIP

* test cases updated

* `PackageService.resolveTemplateId` returns `Option[TemplateId]` now

used to return `Error \/ TemplateId`. There are scenarios when unresolved
TemplateID is a warning and not an error, which was the initial design

* CHANGELOG_BEGIN

[JSON API - Experimental]
``/contracts/search`` endpoint reports unresolved template IDs as warnings, see #3771::

    {
        "warnings": {
            "unknownTemplateIds": ["UnknownModule:UnknownEntity"]
        },
        "result": [{...}, ...],
        "status": 200
    }

CHANGELOG_END

* Addressing codereview comments

* `partitionMap` is now part of `http.util.Collections.SeqOps`
2020-01-14 20:17:04 +00:00
Leonid Shlyapnikov
93216a0d71
Template ID to JsString JSON serialization (#3991)
* test cases: domain.TemplateId JSON serialization to JsString

* JSON protocol updated

* Fixing json-api test cases

* test cases: domain.TemplateId JSON serialization to JsString

* JSON protocol updated

* Fixing json-api test cases

* Adapt daml2ts and support libraries

* Update documentation

CHANGELOG_BEGIN

[JSON API - Experimental]
- Use JSON string to encode template IDs. Use colon (``:``) to separate parts of the ID.
  The request format, with optional package ID:
  - "<module>:<entity>"
  - "<package ID>:<module>:<entity>"
  The response always contains fully qualified template ID in the format:
  - "<package ID>:<module>:<entity>"
  See #3647.

CHANGELOG_END

* Minor documentation formatting changes.

Co-authored-by: Martin Huschenbett <martin.huschenbett@posteo.me>
2020-01-09 14:43:41 -05:00
Brian Healey
95fbf5c921
ECDA512 algorithm support (#3953)
* ECDA512 algorithm support

* ECDA512

* happy day test for ECDA512 algorithm

* failure test for wrong key for ECDA512 algorithm

* add ability to use EC cert file

* update docs

* scalafmt

* Correct documentation

CHANGELOG_BEGIN
[Ledger API Authorization] Support elliptic curve algorithm for JWT verification
CHANGELOG_END

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* correct docs warning
2020-01-09 07:25:04 -05:00
Nikolai Cherkezishvili
49b90bfb36
Engine: Improve performance of critical parts (#3947) 2020-01-09 19:18:55 +11:00
Leonid Shlyapnikov
e75b98351c Aligning DB contract table with domain.ActiveContract class (#3989)
* Aligning DB contract table with domain.ActiveContract class

adding key,signatories, observers and agreement_text to DB contract table
removing witnessParties

Reading signatories and observers from contracts table

Updating doc, removing witnessParties

Address code review comments, thanks @S11001001

CHANGELOG_BEGIN

[JSON API - Experimental]
- Align ``contract`` table with ``domain.ActiveContract`` class.
  The database schema has changed, if using ``--query-store-jdbc-config``,
  you must rebuild the database by adding ``,createSchema=true``. See #3754.
- ``witnessParties`` field is removed from all JSON responses.

CHANGELOG_END

* Fix TypeScript domain models

remove witnessParties and workflowId fields. workflowId has be removed
from JSON output a while ago.
2020-01-08 20:39:57 +00:00
Leonid Shlyapnikov
2c154f2299 Rename argument to payload in domain.ActiveContract (#3971)
CHANGELOG_BEGIN

[JSON API - Experimental] Rename ``argument`` in active contract to ``payload``. See #3826.

CHANGELOG_END
2020-01-08 13:59:27 +01:00
Leonid Shlyapnikov
29d5995454 CHANGELOG_BEGIN (#3969)
- [JSON API - Experimental] When contract is not found, ``/contracts/lookup`` endpoint returns
  ``{"status":200,"result":null}`` instead of ``{"status":200,"result":{}}``.
  This is consistent with how DAML-LF JSON Encoding spec treats Optional.

CHANGELOG_END
2020-01-07 18:51:00 +00:00
Andreas Herrmann
f33e79c787
Remove unused dependencies to da_scala_library (#3938)
* Inline all scala_library dependencies

* Run //:buildifier-fix

* TMP scala_library_suite --> scala_library

* da_scala_library: Enable unused dependency checker

* scala_library: Enable unused dependency checker

* //daml-lf/data:data

* //daml-lf/engine:engine

* //ledger-api/rs-grpc-akka:rs-grpc-akka

* //ledger/participant-state:participant-state

* //ledger/ledger-api-client:ledger-api-client

* //scala-protoc-plugins/scala-logging:scala-logging-lib

* //ledger/ledger-api-scala-logging:ledger-api-scala-logging

* //ledger/ledger-api-common:ledger-api-common

* //ledger-service/utils:utils

* //ledger-service/jwt:jwt

* //ledger/ledger-api-auth:ledger-api-auth

* //extractor:extractor

* //daml-assistant/scala-daml-project-config:scala-daml-project-config

* //language-support/codegen-common:codegen-common

* //language-support/scala/codegen:codegen

* //language-support/codegen-main:codegen-main-lib

* //ledger-service/db-backend:db-backend

* //ledger-service/http-json:http-json

* //daml-lf/scenario-interpreter:scenario-interpreter

* //ledger/sandbox:sandbox

* //navigator/backend:navigator-library

* //daml-assistant/daml-sdk:sdk-lib

* //daml-lf/data-scalacheck:data-scalacheck

* //daml-script/test:test-lib

* //ledger/ledger-api-common:ledger-api-common-scala-tests-lib

* //ledger/test-common:test-common

* //ledger/sandbox:sandbox-scala-tests-lib

* //extractor:extractor-scala-tests-lib

* //language-support/java/bindings:bindings-java-tests-lib

* //language-support/java/bindings-rxjava:bindings-java-tests-lib

* //language-support/scala/bindings-akka-testing:bindings-akka-testing

* //language-support/scala/codegen-testing:codegen-testing

* //language-support/scala/codegen-sample-app:daml-lf-codegen-sample-app

* //language-support/scala/codegen-sample-app:daml-lf-codegen-sample-app-testing

* //language-support/scala/codegen-testing:codegen-testing-testing

* //ledger-api/sample-service:sample-service

* //ledger-api/rs-grpc-akka:rs-grpc-akka-tests-lib

* //ledger/ledger-api-test-tool:ledger-api-test-tool-lib

* //ledger/ledger-api-test-tool:ledger-api-test-tool-tests

* //ledger/participant-state/kvutils:kvutils

* //ledger/sandbox:ledger-api-server

* //ledger/sandbox-perf:sandbox-perf-lib

* //navigator/backend:navigator-tests-library

* UNDO scala_library_suite --> scala_library

This reverts commit ab3eb1ae23139e2ec431ab4551fbb0371e0354e1.

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-06 18:14:21 +01:00
Andreas Herrmann
43bbfeaee6
Remove unused dependencies to da_scala_binary (#3937)
* Inline all scala_binary dependencies

* Run //:buildifier-fix

* da_scala_binary: Enable unused dependency checker

* //compiler/scenario-service/server:scenario-service-raw

* //language-support/scala/codegen:codegen-main

* //daml-lf/encoder:encoder_binary

* //daml-lf/repl:repl

* //language-support/codegen-main:codegen-main

* //language-support/scala/examples:quickstart-scala-bin

* //ledger-api/rs-grpc-akka:rs-grpc-akka-perf

* //ledger-service/jwt:jwt-bin

* //ledger/api-server-damlonx/reference-v2:reference-v2

* //ledger/api-server-damlonx/reference-v2:ephemeral-postgres-reference-server

* //ledger/ledger-api-auth:ledger-api-auth-bin

* //ledger/ledger-api-test-tool:ledger-api-test-tool

* //ledger/participant-state/kvutils/tools:integrity-check

* //navigator/integration-test:navigatortest-jar

* Run //:buildifier-fix

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-06 13:58:56 +01:00
Gary Verhaegen
878429e3bf
update copyright notices to 2020 (#3939)
copyright update 2020

* update template
* run script: `dade-copyright-headers update .`
* update script
* manual adjustments
* exclude frozen proto files from further header checks (by adding NO_AUTO_COPYRIGHT files)
2020-01-02 21:21:13 +01:00
Andreas Herrmann
9fbb787062 Remove unused dependencies to da_scala_test(_suite) (#3925)
* Remove unused scala.bzl imports

* override_targets org.scalatest.scalatest_2.12

Otherwise, rules_scala implicitly adds a different version to scala_test
than other packages transitively depending on scalatest. This causes
unused dependency checker to raise an error.

* Handle rules_scala scalatest in pom_file.bzl

* Inline all scala_test dependencies

So that `unused_dependency_checker = "error"` can be applied to them.

* Run //:buildifier-fix

* TMP scala_test_suite --> scala_test

* da_scala_test: Enable unused dependency checker

* //navigator/backend:navigator-scala-tests

* //ledger/sandbox:sandbox-scala-tests

* //ledger/participant-state/kvutils:kvutils-tests

* //ledger/participant-state:participant-state-tests

* //ledger/ledger-api-scala-logging:ledger-api-scala-logging-test

* //ledger/ledger-api-common:ledger-api-common-scala-tests

* //ledger/ledger-api-client:ledger-api-client-tests

* //ledger/ledger-api-auth:ledger-api-auth-scala-tests

* //ledger-service/lf-value-json:tests

* //ledger-service/jwt:tests

* //ledger-service/http-json:tests

* //ledger-api/rs-grpc-akka:rs-grpc-akka-tests

* //language-support/scala/codegen-sample-app:tests

* //language-support/scala/codegen-sample-app:ScalaCodeGenIT

* //language-support/scala/codegen:tests

* //language-support/scala/bindings-akka:tests

* //language-support/java/codegen:test

* //language-support/java/codegen:ledger-tests

* //language-support/java/bindings-rxjava:bindings-java-tests

* //language-support/codegen-common:test

* //extractor:extractor-scala-tests

* //daml-lf/scenario-interpreter:scenario-interpreter_tests

* //daml-lf/language:language-test

* //daml-lf/interface:tests

* //daml-lf/engine:tests

* //daml-lf/encoder:tests

* //daml-lf/archive:daml_lf_archive_reader_tests

* //daml-assistant/scala-daml-project-config:scala-daml-project-config-tests

* UNDO scala_test_suite --> scala_test

This reverts commit 13ed47ba725e944533ca1157a070cb8dc30569ac.

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2019-12-30 13:49:59 +00:00
Leonid Shlyapnikov
f22d52a2ff Change variant JSON encoding, so it is easier to pattern match on it in TypeScript (#3882)
* Change variant json encoding,

adding integration test

* Add DamlLfTypeLookup dependencies

* Add MetadataReader

* Add test WIP

* Add serialize test cases

* Add serialize test cases, WIP

* Test for variant encoding decoding

* Solving merge conflicts

* Updating roundtrip test

* Minor cleanup

* Addressing code review comments

Add JsonVariant custom matcher

* Update specification

* Update link

* Add test case, WIP

* Add proper template key resolution

* Got rid of choice record ID resolution, resolving choice type and key type

* Fixing logging

* Add Contract Key decoding tests

* cleanup

* cleanup

* Update JSON variant encoding tests

* Add more contract key JSON decoding tests

* Fix variant JSON encoding

* Change value predicate to support new variant encoding

* Change value predicate to support new variant encoding

* Add lookup by contract key test case

where contract key contains variant and record

Add `requiredResource` to bazel utils

CHANGELOG_BEGIN

- [JSON API - Experimental] Change variant JSON encoding. The new format is ``{ tag: data-constructor, value: argument }``.
  For example, if we have: ``data Foo = Bar Int | Baz``, these are all valid JSON encodings for values of type Foo:
  - ``{"tag": "Bar", "value": 42}``
  - ``{"tag": "Baz", "value": {}}``
  See #3622

- [JSON API - Experimental] Fix ``/contracts/lookup` find by contract key.

- [JSON API - Experimental] Fix ``/command/exercise`` to support any LF type as a choice argument.
  See #3390

CHANGELOG_END

* minor cleanup

* Fix copy/paste

* Renaming

* Got rid of DAML LF identifier resolution

resolving DAML LF Type based on command type

* Address code review comments, thanks @S11001001

* Address code review comments, thanks @S11001001

Do not include any error handling here; this partial function should
only match the successful case, JsonVariant.

* Address code review comments, thanks @S11001001

comment

* Address code review comments, thanks @S11001001

using `JsonVariant` for variant encoding/decoding

* Address code review comments, thanks @S11001001

replace `find` and `map` chain with collectFirst

* Update docs/source/json-api/lf-value-specification.rst

Co-Authored-By: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <scompall@nocandysw.com>
2019-12-24 20:55:44 +00:00
Samir Talwar
1794d8a2fd Sandbox: Manage resources and close them properly on failure. (#3871)
* sandbox: Create a monadic `ResourceOwner` to manage resources.

* sandbox: Rewrite `ResourceOwner` to be async.

* sandbox: Make sure failed resources are closed immediately.

* sandbox: Better naming in `Open`.

* sandbox: Rename `Open` to `Resource`, and open/close to acquire/release.

* sandbox: Convert `() => AutoCloseable` into `ResourceOwner`.

* sandbox: Refactor the LedgerApiServer in terms of resources.

* sandbox: Explicitly convert `() => AutoCloseable` to `ResourceOwner`.

Explicit > Implicit, right?

* sandbox: Create helpers for converting things to ResourceOwners.

Because I tried to start using them and there was so much code being
written at once.

* sandbox: Simplify construction of JdbcLedgerDao.

* sandbox: Releasing resources should be idempotent.

In that we should only do it once.

* sandbox: Fix the ResetService by closing the API services _first_.

They need to be shut down before the gRPC server.

* sandbox: Don't try and shut down PostgreSQL twice in tests.

* sandbox: Actually run the assertions in ResourceOwnerSpec.

Facepalm.

* sandbox: Test `Resource.sequence` more rigorously.

* sandbox: Move the helpers around `Resource` into `Resource.apply`.

* sandbox: Convert LedgerApiServer resource owners to classes.

* sandbox: Make `ResourceOwner` a monad too, delegating to `Resource`.

* sandbox: Turn `LedgerApiServer` into a ResourceOwner.

* sandbox: Simplify the public signature of `Resource.apply`.

* sandbox: Use ResourceOwners to simplify DB resource management.

This is one hell of a change. Sorry.

* sandbox: Try not to nest `Await.result` calls.

Causes issues when running in a `DirectExecutionContext`.

* sandbox: Turn index subscriptions into resources.

* sandbox: Fix warnings in RecoveringIndexerSpec.

* sandbox: Always release before recovering the indexer.

* sandbox: Add `flatten` and `transformWith` to `Resource`.

* sandbox: If releasing twice in parallel, the second should wait.

* sandbox: If the indexer recovers, clean up the old subscription.

* sandbox: Convert StandaloneIndexerServer into a resource owner.

* sandbox: Convert StandaloneApiServer into a resource owner.

* reference-v2: Rewrite ReferenceServer in terms of resources.

CHANGELOG_BEGIN

- [Reference v2] On an exception, shut down everything and crash.
  Previously, the server would stay in a half-running state.

CHANGELOG_END

* sandbox: Rewrite SandboxServer in terms of resources.

* sandbox: Write the port file in a Future.

* sandbox: JdbcIndexer no longer needs to manage the actorSystem.

* sandbox: Shut down the LedgerApiServer when closing the Sandbox.

* sandbox: Rename `Resource.pure` to `Resource.successful`.

* sandbox: Rename `Resource.sequence_` to `sequenceIgnoringValues`.

* sandbox: Delete `CloseableResource`.

It's only used in once place. Just inline it.

* sandbox: `LedgerDao` no longer needs to be closeable.

* sandbox: Delete implicit materializers where they're not used.

* http-json: Wait for the Sandbox to start in tests.

* sandbox: Convert `scheduleHeartbeats` into a ResourceOwner.

* reference-v2: Explain why we steal ownership of the actor system.

* sandbox: Document why we only release resources once.

* sandbox: Add clues to ResourceOwnerSpec.

* http-json: Fix HttpServiceTestFixture to pass auth service through.

* codegen-sample-app: In ScalaCodeGenIT, wait for the server to  start.
2019-12-22 15:05:55 +00:00
Stephen Compall
ac7fa36162
typed testing support for LF Enums (#3913)
* runtime-filtering-only-style enum for ValueAddend

- the alternative is a coproduct of singleton types, which would be more
  like the other ValueAddend combinators, but done differently here as
  an experiment

* test an enum case for lf-value-json
2019-12-20 12:14:17 -05:00
Brian Healey
9f13a2fb10 Upgrade to Akka 2.6.1, akka-http 10.1.11 and Scala 2.12.10 (#3903)
* Upgrade to Akka 2.6.1, akka-http 10.1.11 and Scala 2.12.10

Akka 2.6.1 Upgrade Changes
- Materializer in place of ActorMaterializer
- Source.future instead of Source.fromFuture
- The Scheduler.schedule method has been deprecated in favor of selecting scheduleWithFixedDelay or scheduleAtFixedRate
- onDownstreamFinish(cause: Throwable)
- ActorAttributes.supervisionStrategy(...) in place of ActorMaterializerSettings.withSupervisionStrategy

See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html

* Akka 2.6.1 Upgrade Changes
- onDownstreamFinish(cause: Throwable)

See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html

* code review: remove unnecessary supervision strategy
2019-12-19 18:00:59 +00:00
Stephen Compall
7c1e40f391 friendly and gen testing for LF records, variants (#3838)
* drop Nones from records when JSON-encoding

* elide some needless conversions in ApiCodecCompressed

* record with visible Inj

- seems to be running up against scala/scala#5075

* use new 3rdparty lib reference

* construct record types field-by-field

- works around scala/scala#5075

* switch to record style for records' Inj

- suggested by @leo-da

* include some samples to experiment with record

* record parsing

* RecordVa doesn't benefit from kind-projector

* scalafmt

* make RecordVa details private; only intended API is public

* Arbitrary and Shrink record support

* rename RecordVa#Inj to HRec, to accommodate variants in same cons structure

* use record testing in lf-value-json

* add complex record type

* add complex record value and test it

* more documentation for RecordVa

* Table-ify prior record tests

* start adding variant support

* scalacheck support for variants

* more scalacheck support for variants

* variant examples

* example of using align for records

* test variant roundtrip

* test variant in record

* add sample tests for the JSON format of records and variants

* rename record-specific functions in RecordVa

* rename RecordVa to RecVarSpec
2019-12-19 14:48:27 +00:00
lima-da
a801a55d81
update encoder to be urlEncoder (#3893)
* update encoder to be urlEncoder

* fix decoder as well

* move decoder to constant
2019-12-18 17:26:36 -05:00
Leonid Shlyapnikov
8b9fb5942d
Fixing a rare test failure, (#3868)
* Fixing a rare test failure,

two independent scalacheck Gens generate duplicate TemplateIds. Duplicate
in this context means having the same moduleName and entityName.
```
- TemplateIdMap.unique should not contain dups *** FAILED *** (102 milliseconds)
  TestFailedException was thrown during property evaluation.
    Message: None was not equal to Some(TemplateId(zy,n,s))
    Location: (PackageServiceTest.scala:81)
    Occurred when passed generated values (
      arg0 = List(TemplateId(zy,n,s)),
      arg1 = List(TemplateId(lj,n,s), TemplateId(g,n,s))
    )
```

* Addressing code review comments, thanks @S11001001

* Addressing code review comments, thanks @SamirTalwar

* Minor cleanup
2019-12-17 13:00:37 -05:00
Samir Talwar
71e8c20127 Sandbox: Fix warnings in SandboxServer. (#3853)
* sandbox: Move SandboxServer's helper classes to the companion object.

And make them `private`, `final`, and non-`case`.

* sandbox: Drop SandboxServer.apply and just call the constructor.

* sandbox: Move the SandboxServer#SandboxState class into the object.

Leave the `resetAndRestartServer` method behind, though.
2019-12-14 20:52:03 +00:00
Leonid Shlyapnikov
0c741a64cb Fix contract lookup endpoint (#3809)
* Lookup by Contract ID and Contract Key is WIP

* factor out "contract ID or key" JSON decoding

- adapted from fc132253 (#2695)

* Resolving conflicts

* Resolving conflicts

* Lookup by contract ID works

* Testing new contract created by IOU_Transfer can be looked up

* error if key and contractId specified for lookup at the same time

* Lookup by contract key test

* Lookup docs

* re-format with `./fmt.sh`

* minor cleanup

CHANGELOG_BEGIN

- [JSON API - Experimental] Fix and document ``/contracts/lookup`` endpoint. See #3755.

CHANGELOG_END
2019-12-12 22:16:12 +00:00
lima-da
a56849ce73
Json simple websocket (#3796)
* first checkpoint for adding websocket

* unify shared method across websocketEndpoints and Endpoints using EndpointsCompanion objc

* add config for ws backpressure and logging

* add copy right header

* merge and change TODO comments

* resolve conflicts

* disable ws service for now to save merge conflicts while working on improvements
2019-12-11 17:02:35 -05:00
Leonid Shlyapnikov
0b172da49a Add ExceptionOps to extract exception details safely (#3822)
* Add ExceptionOps to extract exception details safely

`Throwable.getMessage` can return `null` which caused unexpected NPE
in JSON API error handling

* Fix formatting

* Remove show instance

* Addressing code review comments,

thanks Stefano!
2019-12-11 18:38:44 +00:00
Leonid Shlyapnikov
dcc7dc913f
Add choice result to the exercise response (#3757)
* Adding choice result to the exercise response, WIP

* Adding choice result to the exercise response, WIP

* Refactoring towards #3390,

ExerciseCommand argument does not always have to be a Record, changing
typearg: JsObject -> JsValue

* Cleanup

* exercise-with-result endpoint

* todo

* Switching /commands/exercise to use SubmitAndWaitForTransactionTree,

populating archived and created from TransactionTree

* removing debug println

* Fixing tests

* Removing `/command/exercise-with-result` endpoint

this one returns only exercise result, `/command/exercise` now returns
exercise result and events

* Updating docs

* Updating docs

CHANGELOG_BEGIN

- [JSON API - Experimental] Expose exercise result. Changed the output
of the ``/command/exercise``. Note ``exerciseResult`` and ``contracts``
in ``{"status":200,"result":{"exerciseResult": ...,"contracts":[...]}``.
See #3314

CHANGELOG_END
2019-12-10 10:09:24 -05:00
Stephen Compall
1388ebd695 compile ValuePredicates to SQL and run json-api queries in Postgres (#3599)
* toSqlWhereClause WIP

* literal cases for SQL predicate generation

* fill in the remaining Literals' sql-equal data

* add other likely elements of the toSqlWhereClause fold

* SQL ListMatch case

* SQL VariantMatch case

* partial SQL RecordSubset case

* SQL Range case

* comments on JSON encoding for DB

* new relationship for the 3 paths in toSqlWhereClause Rec

- conversion to Fragment is further delayed to allow mixed-mode matching
  of records

* handle new Rec semantics in ListMatch (can't drop @> on the floor anymore)

* compile RecordSubset to SQL

* compile MapMatch to SQL

* note optimization for record & map

* optimize = when @> unavailable for record subsets

* optimize range when degenerately =

* don't discard the @> safety of empty test sets

* unnested SQL optional matching

* converting DB JSON to response body JSON

* compiling nested optionals to SQL

* add missing length check to ListMatch

* remove unneeded parens from generated predicates

* remove 1 = 1 leader from generated predicates

* test the generated SQL from toSqlWhereClause

* searchDb WIP

* searchDb integrated

* coerce party to text in selectContracts; log that query

* fixing query

* fixing scala formatting

* removing unused functions

* removing unused type alias

* checking a that search returned exactly what we searched for,

also checking that all contracts got stored in the DB

* factor commonalities of MapMatch and RecordSubset

* cleanup

* cleanup

* changelog

CHANGELOG_BEGIN

- [JSON API - Experimental] Queries will always run against Postgres if Postgres is
  configured.  See `issue #3388 <https://github.com/digital-asset/daml/issues/3388>`_.

CHANGELOG_END

* fix record =-unsafe detection
2019-12-05 11:15:01 -05:00
Stefano Baghino
5254520b80 Spin off TokenHolder into a new library (#3741)
* Spin off TokenHolder into a new library

Avoids having weird dependencies between different packages, makes TokenHolder available on Maven

* Fix auth-utils path
2019-12-05 12:31:14 +00:00