Commit Graph

498 Commits

Author SHA1 Message Date
Samir Talwar
888f02aa4d
ledger-api-common: There's only one way to order absolute offsets. (#10153)
Let's not mess around with injectable orderings.

This has the pleasant side effect of removing dependencies.

CHANGELOG_BEGIN
CHANGELOG_END
2021-06-30 14:17:59 +00:00
Samir Talwar
c929a14486
logging-entries: Split from contextualized-logging. [KVL-996] (#10149)
* logging-entries: Split from contextualized-logging.

This allows us to introduce it to Daml-LF without bringing in the
Logback, Logstash, and gRPC dependencies.

CHANGELOG_BEGIN
CHANGELOG_END

* logging-entries: Fix dependencies for 2.12.

* logging-entries: Missed one more Scala 2.12 dependency.

* release: Publish logging-entries.
2021-06-30 12:08:33 +00:00
Victor Peter Rouven Müller
7521ec9988
[JSON-API] Include contract id in the logging ctx when logging exercise commands (#10131)
changelog_begin

- [JSON-API] The contract id is now also included in the logging ctx when logging exercise commands

changelog_end
2021-06-28 09:12:40 +00:00
Samir Talwar
af9382ce92
contextualized-logging: Reduce the API surface and avoid name collisions. (#10102)
* contextualized-logging: Remove `Map` from the `LoggingContextOf` API.

* contextualized-logging: Encapsulate entries.

CHANGELOG_BEGIN
CHANGELOG_END

* contextualized-logging: Disambiguate `withEnrichedLoggingContext`.

So forthcoming implicit conversions won't confuse the compiler.

* contextualized-logging: Remove a now-unused compat import.

* contextualized-logging: Avoid overloading `LoggingEntries#apply`.

* contextualized-logging: Remove the silencer plugin.
2021-06-24 16:21:26 +00:00
Stephen Compall
01d6770450
json-api perf tests: combine large-ACS frequencies combinatorially instead of in lockstep (#10101)
In #10016, 1% template ID and 1% party-set membership meant _the same_ 1%,
meaning that an index of both couldn't possibly yield interesting results.  This
changes how LargeAcs builds the large ACS so that it's "1% of 1%", as you'd
expect.

CHANGELOG_BEGIN
CHANGELOG_END
2021-06-24 10:00:02 -04:00
Victor Peter Rouven Müller
1f5aa4491c
[JSON-API] Add metrics for ledger command submission timing (#10076)
changelog_begin

- [JSON-API] Timing metrics which measure how long the processing of a command submission request takes on the ledger are now available

changelog_end
2021-06-22 15:59:23 +00:00
Victor Peter Rouven Müller
0391f3529f
[JSON-API] Add db metrics & response construction metrics (#10068)
changelog_begin

- [JSON-API] The database operations (regardless of in-memory or postgres) contain now metrics for fetching contracts by id or key (seperate metrics foreach)
- [JSON-API] The time for a repsonse payload construction of a request is now also tracked in a metric

changelog_end
2021-06-22 10:34:29 +02:00
Stefano Baghino
b796695768
Add per-query offsets to HTTP-JSON API (#9847)
* Add per-query offsets to HTTP-JSON API

changelog_begin
[HTTP/JSON API] The streaming query endpoint now accepts to override the
offset for each query using the `offset` field. You can read more
about it on the documentation.
changelog_end

* Replace mapValues and filterKeys with something that compiles with both 2.12 and 2.13

* Refactor offset comparison

- Addresses https://github.com/digital-asset/daml/pull/9847/files#r642363229
- Addresses https://github.com/digital-asset/daml/pull/9847/files#r642363848

* Add tests (failing), fix definition of starting offset

* Fix failing test

* More tests

* Add tests for verify older behavior still works

* Fix regression

* Remove debug prints

- Addresses https://github.com/digital-asset/daml/pull/9847#discussion_r642362683
- Addresses https://github.com/digital-asset/daml/pull/9847#discussion_r642362774

* Failed attempt at ACS call integration

* Revert "Failed attempt at ACS call integration"

This reverts commit 879e0cfbdb579362674120f93a9c93388ed81bc1.

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r647019447

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r647019965

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r647020825

* Integrate with ACS calls

* Fix three broken tests

* Fix one regression

* Fix last regression

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651586811

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651668995

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651669134

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651669691

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651670134

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651676902

* Fix Scala 2.12 compilation error

* Fix compilation issues on Scala 2.13 and 2.12

* Remove unnecessary dependency

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651824658

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651831880

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651833646

* Address two comments:

- https://github.com/digital-asset/daml/pull/9847#discussion_r651841615
- https://github.com/digital-asset/daml/pull/9847#discussion_r651842951

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651838831

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651949821

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r651943962

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r652492756

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r652537652

* query_offset -> queryOffset

* Document per-query offsets

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r655117581

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r655119961

* Apply suggestions from code review

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r655233517

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r655266806

* Address https://github.com/digital-asset/daml/pull/9847#discussion_r655265717

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-06-21 16:38:07 +02:00
Victor Peter Rouven Müller
e4585295c6
[JSON-API] Add moar timing metrics (#10045)
* [JSON-API] Add more timing metrics

changelog_begin

- [JSON-API] Timing metrics are now available for party management, package management, command submission and query endpoints.
- [JSON-API] Also added a timing metric for parsing and decoding of incoming json payloads

changelog_end

* Add comments to new metrics

* Split metrics up more & remove obsolete metric

* Split up timers for query endpoints
2021-06-21 14:37:36 +00:00
Gary Verhaegen
e12a449c81
s/DABL/Daml Hub/ (#10062)
It's been long enough.

CHANGELOG_BEGIN
CHANGELOG_END
2021-06-21 16:37:17 +02:00
Stephen Compall
3113702dc0
index json-api signatories and observers on Oracle (#9895)
* nvarchar2 keys are text-incompatible, but varchar2 keys are fine

* commit the ACS update before query

* add changelog

CHANGELOG_BEGIN
- [JSON API] The Oracle database schema has changed; if using
  ``--query-store-jdbc-config``, you must rebuild the database by adding
  ``,createSchema=true``.  See #9895.
CHANGELOG_END

* expand the InitDdl set to include materialized views

* replace search index with a materialized view that expands the stakeholders

* allow materialized views to be created in Oracle testing

* join and query the contract_stakeholders table for party-set membership

- restoring a few elements removed by 3e6661128d (#9484)
2021-06-17 11:52:37 -04:00
Stephen Compall
7dfa36fccd
pass supportedJdbcDriverNames implicitly, and only to what actually needs it (#10036)
This solves two warts in the code:

- the validate/createUnsafe double-parse, because scopt doesn't let you flatMap;
- the non-JdbcConfig sub-configs appeared to need to know the JDBC drivers when
  they really don't, because of a quirk in the inherited implementation

That coherence of scopt.Read instances calls for all its dependencies to be
coherent leads us to treat supportedJdbcDriverNames as a nullary typeclass
instance.  This is a nullary typeclass by the same justification as
SupportedJdbcDriver; see scaladoc there for more.

And we solve the latter problem by...adding a type parameter, how else.

CHANGELOG_BEGIN
CHANGELOG_END
2021-06-17 10:40:49 -04:00
Victor Peter Rouven Müller
1852830833
[JSON-API] Concurrent query etc. metrics (#10031)
* [JSON-API] Concurrent query etc. metrics

changelog_begin

- [JSON-API] The metrics which describe the amount of these concurrent events is now available:
	- running http request
	- command submissions
	- package allocations
	- party allocations

changelog_end

* Rename running metrics to throughput ones & add comments on the metrics

* Adjust names of other existing metrics too, to have for the json api a more consistent metrics naming
2021-06-16 17:08:06 +00:00
Stephen Compall
321c4da2dd
test different template-ID frequencies in the "mega" perf scenario (#10016)
* add a new template for testing template-ID-sensitive query

* add a tpid *and* party-set 1% scenario

CHANGELOG_BEGIN
CHANGELOG_END
2021-06-16 12:19:03 -04:00
Victor Peter Rouven Müller
cfee5a2d8b
[JSON-API] Add information from the jwtpayload to the logging context (#9995)
* Add information from the jwtpayload to the logging context

changelog_begin

- [JSON API] For applicable requests actAs, readAs, applicationId & ledgerId are included in the log context

changelog_end

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Revert changes to make the function generic

* Create JwtPayloadG trait from which both payload variants inherit

* Reduce code duplication in Endpoints.scala

* Apply review suggestion

* Update test name to reflect field name changes

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/Endpoints.scala

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-06-16 07:55:50 +00:00
Stephen Compall
401069ef00
prepare for upgrade to Scalaz 7.3.3 (#9997)
* upgrade scalacheck to 1.14.3

* regenerate maven_install files

* some different names and implicits

* remove some fromTryCatchNonFatal

* more porting

* port fromTryCatchNonFatal to attempt

* factor the assertions in SignatureSpec to avoid \/

* deal with invariant \/

* make partial unification do what we want

* \/, parse*, and toNel

* many uses of the .right method

* a legitimate use of fromTryCatchThrowable

* rebuild maven pins

* further invariant \/

* OneAnd and Nel interface changes

* further Either games

* \/ and reformatting

* \/ in http-json

* \/ in http-json

* deprecations

* more invariance

* cleanup unused

* more invariance; http-json compiles

* final either follies

* small 2.12 extra incompatibility

* rebuild deps

* revisit a couple earlier fixes using nicer expressions I learned later

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* repin 2.12
2021-06-15 14:41:48 -04:00
Victor Peter Rouven Müller
3ad9cd58d9
[JSON-API] Log templateId & choice name (if present) on command submissions in the json API (#10005)
* Log templateId & choice name (if present) on command submissions in the json API

changelog_begin

- [JSON API] The template id & choice name (if present) are now logged on command submissions in the Json API (at trace level)

changelog_end

* Move the template id & the choice into the logging context

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/CommandService.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/CommandService.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Fix compile error due to scala 2.12 collection differences

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2021-06-15 17:41:17 +00:00
Victor Peter Rouven Müller
3583fd1885
[JSON-API] Log source and path of incoming http requests & the response with info level (#10007)
changelog_begin

- [JSON-API] The source and the path for incoming http requests are now logged
- [JSON-API] The http response for a request is now logged

changelog_end
2021-06-15 16:30:05 +02:00
Victor Peter Rouven Müller
3d2d4b9696
Don't duplicate log failed futures in the http json api (commandservice specific ones) (#9990)
* Don't duplicate log failed futures in the http json api (commandservice specific)

changelog_begin

- [JSON API] Errors which arise from the CommandService are not logged twice anymore (thus reducing noise)

changelog_end

* Fix duplicate logging of a failed future differently via adding another error case in CommandService.scala

* Restore old formatting

* Improve comment

* Remove the wrong changes I accidentally made, the functions are quite similar xD

* remove the additional type and just make the id optional & remove unecessary comment
2021-06-15 12:27:54 +02:00
Victor Peter Rouven Müller
c75013622c
Just print out the operator name for the operation logging in CommandService.scala (#9989)
changelog_begin
changelog_end
2021-06-14 15:11:10 +00:00
Victor Peter Rouven Müller
603a4a4b42
Add date to log statements in the HTTP JSON service (#9988)
changelog_begin

- [JSON API] log statements now include the date next to the time

changelog_end
2021-06-14 12:05:43 +00:00
Victor Peter Rouven Müller
de7a462082
Log call to submitAndWaitRequest with the command id provided in the log ctx (#9940)
* Log call to submitAndWaitRequest with the command id provided in the log ctx

changelog_begin

- [HTTP-JSON] Calls which trigger a submitAndWaitRequest are logged with the command id provided in the log ctx

changelog_end

* Require the request id to be also in the log ctx

* Log command kind in submitAndWaitRequest of CommandService.scala

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/CommandService.scala

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-06-11 11:01:05 +00:00
Moritz Kiefer
db792a9774
Proxy ledger API health endpoint in JSON API health endpoint (#9945)
* Proxy ledger API health endpoint in JSON API health endpoint

This is a bit more useful than just querying ledger end.

changelog_begin

- [JSON API] The healthcheck endpoint on the JSON API now proxies the
  health check endpoint on the underlying ledger. Previously it only
  queried for ledger end to determine ledger health.

changelog_end

* naming things is hard

changelog_begin
changelog_end

* Fix Scala 2.12

changelog_begin
changelog_end
2021-06-09 20:32:51 +00:00
Victor Peter Rouven Müller
13767b5c5d
Load the correct logback file for the http json service respecting the deployment situation (#9938)
* Load the correct logback file for the http json service respecting the deployment situation

changelog_begin

[HTTP-JSON]
- fixed that json log output could not be enabled via cli options except via usage of env vars

changelog_end

* Move import statement, remove some braces and reformat

* Move system prop dependend logic to cliopts Logging.scala

* Remove PathKind type in cliopts Logging.scala as it's not necessary anymore
2021-06-09 14:09:56 +00:00
Victor Peter Rouven Müller
e9a2e2e7f2
Allow two different time formats as input for the metrics reporting interval (#9926)
* Allow two different time formats as input for the metrics reporting interval and accordingly revert to using the old test for the CommonCliSpecBase in sandbox-common

changelog_begin

- for applications which support the --metrics-reporter-interval cli option, these now support both the java and scala duration string format (e.g. PT1M30S and 1.5min)

changelog_end

* Replace the thrown RuntimeException in the DurationFormat reader with an IllegalArgumentException

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Try parsing java duration first

* Add test which covers the scala duration format too in the CommonCliSpecBase of sandbox-common

* Add comment about java duration format

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-06-08 17:35:47 +00:00
Victor Peter Rouven Müller
4037b1cd9e
Add metrics to the http json service (#9923)
* Introduce metrics in the http-json service

changelog_begin

[HTTP/JSON API]
- metrics reporting can now be chosen via the command line option --metrics-reporter (currently hidden), valid options are console, csv://, graphite:// and prometheus://
- metrics reporting interval can also now be chosen via a command line option --metrics-reporting-interval (currently hidden)

changelog_end

* Move MetricsReporter and it's dependencies into //ledger/metrics

* Restore non-ugly formatting for that one section in Endpoints.scala

* Update ledger/sandbox-common/src/test/lib/scala/platform/sandbox/cli/CommonCliSpecBase.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Hide metrics option for http-json

* Propagate context exception for the parseUri function in MetricsReporter.scala

* Move cliHint value above parseUri function to have a better structure (it's used once before it's defined and once after it's defined, which is weird to me)

* Use better value name than optMr & optFd in cliopts Metrics.scala

* Remove import order changes & whitespace changes

* Revert usage of Nanoseconds for conversion from scala duration to java duration to usage of Seconds

* Shorten hideIfRequested function

* Fix another rearranged import

* Fix another whitespace removal

* Readd metrics cli option to sandbox after refactoring

* Add missing return type annotation for invalidRead in MetricsReporter

* Readd newline in https OptionParser.scala

* Remove unecessary import

* Update ledger/sandbox-common/src/main/scala/platform/sandbox/cli/CommonCliBase.scala

Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>

* Align setter & config name for metricsReportingInterval setting too in CommonCliBase.scala

* Rename http_json_api in Metrics.scala of metrics project to HttpJsonApi

* Reformat CommonCliBase.scala of sandbox-common project

* Fix CommonCliSpecBase test of sandbox

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
Co-authored-by: Miklos <57664299+miklos-da@users.noreply.github.com>
2021-06-08 15:54:01 +00:00
Moritz Kiefer
081c54b411
Revert "Try to unflake JSON API failure tests (#9855)" (#9887)
This has made things worse not better. Now tests that should be
successful hit the timeout.

Still not sure what is actually going wrong so marking it as flaky
Tracked in #9886

changelog_begin
changelog_end
2021-06-02 18:20:19 +02:00
Moritz Kiefer
7d716e6dba
Try to unflake JSON API failure tests (#9855)
On CI we occasionally see the timeout tests fail with
`UnexpectedConnectionClosureException` instead of the expected
server-side timout error. I haven’t managed to reliably reproduce this
or figure out what is going wrong (the closest seems to be
https://github.com/akka/akka-http/issues/3806 but I struggle to see
how that applies here).

This change seems at least promising and at worst, it just speeds up
the tests (by waiting less) and doesn’t change flakiness which still
seems like an improvement.

changelog_begin
changelog_end
2021-06-01 18:39:37 +02:00
Moritz Kiefer
25b7e544ce
Only resolve template id once (#9837)
While looking at errors produced during command submissions I got
confused by the fact that we resolve the template id in the command
service. Turns out there is no reason for doing that, our types are
just not precise enough. This PR fixes that by making sure that we
differentiate between commands where the template id has been resolved
and those where it hasn’t.

changelog_begin
changelog_end
2021-05-31 08:50:52 +02:00
Stephen Compall
ad529e9a8b
http-json-perf: querying larger ACSes with different data patterns (#9530)
* new perf test module LargeAcs to shift many-contract creation to Daml-side

* using MakeIouRange to make blocks of (more or less constant) Ious

* syntax and bad variable quasiquoting

* ACS data distribution plans

* run with 100k contracts, same template, 1% observer frequency

* use CanAssert instead of CanAbort

* query under proper alternative jwt

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* make sure the database accurately represents the ACS
2021-05-25 08:23:31 -04:00
Victor Peter Rouven Müller
beb2d138f2
Add cli option & system property to enable json only logging for http json service (#9725)
changelog_begin
- [Ledger HTTP Json Service] Logging output can now be in JSON either via providing the cli option `--log-encoder json` or via setting the env var `LOG_FORMAT_JSON=true`
changelog_end
2021-05-20 13:28:34 +00:00
Miklos
3bf6531e3c
ledger-service, navigator: Damlification of Scala files (#9669)
* Damlification of Scala files (primarily comments and strings).

* Comment tidying.
CHANGELOG_BEGIN
CHANGELOG_END
2021-05-20 09:14:13 +00:00
Miklos
f5c84a2aa4
Daml-LF: Damlification of Scala/Haskell files (#9666)
CHANGELOG_BEGIN
CHANGELOG_END
2021-05-20 09:52:33 +02:00
Moritz Kiefer
66d039eb5b
Upgrade gatling to version 3.5.1 (#9714)
Ths is required to get a version of Gatling that supports Scala
2.13 (and only that because they do not cross build).

Unfortunately the upgrade is a bit more annoying than I was hoping for:

Our custom gatling utils rely on parsing the simulation log. This is
an internal file format with zero documentation or stabilityt
guarantees and as expected it has changed in incompatible ways during
the upgrade.

Rather than trying to reverse engineer and adapt to changes everytime
we upgrade, this PR switches us to a slightly more supported codepath
by parsing the `stats.json` and `assertions.json` produced by the
highcharts stuff. Afaict this is also what for example the Jenkins
integration relies on so while it’s not completely public API it seems
like the best option I could find.

There are a few pieces of information we can’t get out of those
files. Specifically:

1. maxUsers: we only ever need one users anyway so not really relevant.
2. start, duration, end: no idea why we would want those. we want per
   request metrics not the total duration.
3. geometric mean: slightly annoying, but avg & stdev should be good
enough™.
4. The scenario name: Not really an issue but if it is, we can
   disambiguate by changing request names.

changelog_begin
changelog_end
2021-05-18 16:54:37 +00:00
Victor Peter Rouven Müller
625592e7b5
Rename CorrelationID to InstanceUUID as it has been used wrongly. (#9672)
This ensures that we follow our internal guidance on logging correctly.

changelog_begin
changelog_end
2021-05-12 16:46:14 +00:00
Stephen Compall
ca9e89b3da
check whether collection.compat is unused when compiling for Scala 2.12 (#9604)
* check whether collection.compat is unused when compiling for Scala 2.12

- Instead of always suppressing warnings for collection.compat._,
  we should only do it for Scala 2.13

- We can also reduce boilerplate by automatically adding this
  option when both silencer_plugin and collection-compat are
  present

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused import

* remove another unused import

* remove even more unused imports

* missed compat dependency

* more missed compat dependencies

* missed compat dependency

* use scala_deps in scaladoc_jar

- #8423 inlined the major version expansion, but this seems to
  have been prior to proper support by scaladoc_jar

* restore custom handling of participant-integration-api

- fixing scaladoc_jar isn't worth it for a single case, as with
  deps vs scala_deps
2021-05-11 21:54:14 +00:00
Victor Peter Rouven Müller
4f0ceff7a2
Fix #7170 (#9618)
CHANGELOG_BEGIN

http-json:
- add contextual id as logging context to distinguish different application runs in logs
- add request id as logging context to distinguish different http requests within an application run
- add for non-static endpoints trace logs which show how long processing it took in ns

CHANGELOG_END
2021-05-11 14:15:17 +00:00
Victor Peter Rouven Müller
112c387e5a
Refactor out setGlobalLogLevel into ContextualLogging (#9592)
CHANGELOG_BEGIN

- [Ledger HTTP Json Service] Logging now also tells service name if log level was changed.

CHANGELOG_END
2021-05-06 14:23:39 +00:00
Victor Müller
26a53d8469
Add logging command line option to ledger http service (#9581)
CHANGELOG_BEGIN

- [Ledger HTTP Json service] Logging can now be configured via the `--log-level` cli argument. Valid values are `error`, `warn`, `info` (default), `debug`, `trace`

CHANGELOG_END

Co-authored-by: victor.mueller@digitalasset.com <mueller.vpr@gmail.com>
2021-05-05 17:36:22 +00:00
Moritz Kiefer
de80a6dc60
Drop ValueBuiltinException (#9576)
As discussed, we don’t want to expose this via serializable values at
least for now (and it’s not exposed on the ledger API anyway) so this
PR drops the type.

changelog_begin
changelog_end
2021-05-05 12:32:58 +02:00
Stefano Baghino
94e9ec2966
Remove a couple of warts (#9557)
changelog_begin
changelog_end
2021-05-03 14:37:36 +00:00
Moritz Kiefer
4aca199bbd
Upgrade to Scala 2.13.5 (#9528)
changelog_begin
changelog_end
2021-04-29 13:05:57 +00:00
Stephen Compall
4e712a01a0
add oracle option for json-api perf runner (#9492)
* add oracle option to http-json-perf-binary-ee

* add oracle path to perf Main's JDBC bracket

* adapt to availableJdbcDriverNames; missing deps

* add changelog

CHANGELOG_BEGIN
- [JSON-API Perf] ``--query-store-index=postgres`` must be passed
  to select PostgreSQL query store performance testing; ``true``
  and ``yes`` are no longer supported.
  See `issue #9492 <https://github.com/digital-asset/daml/pull/9492>`__.
CHANGELOG_END
2021-04-27 08:58:44 -04:00
Samir Talwar
8cd3658b14
Switch to an environment variable for enabling Oracle tests. (#9511)
* participant-integration-api: Build Oracle tests, but don't run them.

CHANGELOG_BEGIN
CHANGELOG_END

* triggers: Switch to an environment variable for enabling Oracle tests.

* http-json: Switch to an environment variable for enabling Oracle tests.

* Disable running Oracle tests by default, not building them.

* triggers/service: Remove unused test dependencies.
2021-04-27 11:50:39 +00:00
Stefano Baghino
3e6661128d
Nest stakeholders in contracts table as JSON arrays (#9484)
* Nest stakeholders in contracts table as JSON arrays

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/9484#discussion_r619468382

* Addresses https://github.com/digital-asset/daml/pull/9484#discussion_r619470603 and https://github.com/digital-asset/daml/pull/9484#discussion_r619472278

* Partly address https://github.com/digital-asset/daml/pull/9484#discussion_r619470992

* Update ledger-service/db-backend/src/main/scala/com/digitalasset/http/dbbackend/Queries.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2021-04-27 12:27:25 +02:00
Samir Talwar
e1af564bcc
Switch from @silent to @nowarn. (#9498)
* Switch from `@silent` to `@nowarn`.

This annotation is native to Scala 2.12.13+ and 2.13.2+. It replaces
most usages of `@silent`.

I had to get creative about a couple of use cases that didn't work.
Specifically:

  1.  Suppressing deprecation warnings works, but Scala 2.12 erroneously
      complains that the `@nowarn` is unnecessary.  I had to suppress
      this warning too with `-Ywarn-unused:-nowarn`.
  2.  I can't seem to suppress the warning, "The outer reference in this
      type test cannot be checked at run time." Instead, I have
      refactored the code to remove the warning.

We still need to use the silencer plugin to suppress some warnings about
unused imports (because of compatibility between Scala 2.12 and 2.13),
but this means we no longer need the library, and therefore it is not a
transitive dependency that downstream consumers need to worry about.

CHANGELOG_BEGIN
CHANGELOG_END

* Add some comments around `@nowarn` support.

* language-support/scala: Fix a warning suppression.

* Revert to the default warnings.

Compatibility was complaining.
2021-04-26 19:46:14 +00:00
Moritz Kiefer
f0f8027a47
Set supported jdbc driver names at compile time (#9489)
* Set supported jdbc driver names at compile time

This is mainly to unblock the work on Oracle support in the Ledger API
but I think it’s a sensible thing in general. For the Ledger API,
moving the dependency to the top-level is apparently rather
tricky. Because the SDK bundles everything into a single megajar,
Sandbox depending on the oracle library does also result in the JSON
API and the trigger service will also have the oracle library in scope
and will support Oracle in CE which they should not.

This PR simply hardcodes the list of supported drivers to address
that. Not pretty but does the job.

changelog_begin
changelog_end

* format

* Address review comments

changelog_begin
changelog_end
2021-04-26 09:19:31 +02:00
lima-da
988f3e509e
add ledger connection retry on json-api start (#9450)
* CHANGELOG_BEGIN
add ledger connection retry on json-api start
CHANGELOG_END

* remove white space

* move default maxAttempts to val
2021-04-21 10:08:47 -04:00
Stefano Baghino
43ffa426f3
Test for duplicate contracts when querying on behalf of multiple parties (#9443)
* Test for duplicate contracts when querying on behalf of multiple parties

Fixes #9388

changelog_begin
changelog_end

* Optimize imports

* Thanks to @S11001001 for answering the comment

* Re-structure the test following @S11001001's input in https://github.com/digital-asset/daml/pull/9443#discussion_r616083932 -- thanks
2021-04-20 09:25:40 +00:00
Stephen Compall
4a08fd4db7
remove Array[String] instances from Oracle json-api driver layer (#9425)
* remove Array[String] instances from Oracle json-api driver layer

CHANGELOG_BEGIN
CHANGELOG_END

* fix Postgres integration test to deal with removed implicit API
2021-04-16 16:51:18 +00:00
Stephen Compall
c367ddabef
deduplicate queried contracts with multiple signatories/observers (#9423)
CHANGELOG_BEGIN
CHANGELOG_END
2021-04-15 17:20:44 +00:00
Stephen Compall
9ac74e697d
Oracle json-api: websockets and testing (#9278)
* comparison queries

* name the contract primary key constraint

* use ignore_row_on_dupkey_index instead of merge

- suggested by @cocreature in #9286 f7b2f14294fa33d6804251ce841529a1e2bd298d; thanks

* retrySqlStates for oracle

* enable all non-websocket tests

* name the template_id primary key constraint

* clean up concatFragment calls

* add Websocket tests for oracle

* move iouCreateCommand to be usable by oracle integration tests

* work around Scala 2.12 NPE in Source

* multiquery support for Oracle

* matchedQueries, therefore query-stream support for Oracle

* enable websocket tests

* test '& bar' and 5kb strings

- 5kb string fails on Oracle with
  ORA-01704: string literal too long

* refine the long data cases; gets too long at 4000 bytes as expected

- however, the predicate fails for unknown reason before then; possibly a missed
  escape character case

* handle long data with a fallback

- now the predicate fails in all cases instead of a SQL error, which is...better

* only interpolate true, false, null into JSON predicate conditions

- the problem was with JSON-formatted data; it must be SQL-typed instead

* adapt equal's large-data solution for comparison as well

- only works for numbers and strings, but that's all we need to compare

* move Implicits to Queries

* remove stray spaces in output

* test Oracle query expressions alongside Postgresql

* test that bools aren't compared like numbers and strings

* test @> conjunctions and special {}-query handling

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* note on PASSING ... AS X

- suggested by @cocreature; thanks

* remove printlns; these functions don't really need scaffolding anymore

- suggested by @stefanobaghino-da; thanks
2021-04-12 17:49:19 +00:00
Stephen Compall
fd63cf0c83
use {tag, value} format for SQLizing advanced JSON queries (#9321)
* literal variant test case for compiled query

* nicer error reporting of positional args in Fragments

* a failing advanced case for variant query

* use proper path

* also check nested number roundtrip

* add changelog

CHANGELOG_BEGIN
- [HTTP JSON API] Range queries within variant data would not return matching
  data when using the PostgreSQL backend with JSON API; this is fixed.
  See `issue #9321 <https://github.com/digital-asset/daml/pull/9321>`__.
CHANGELOG_END
2021-04-06 08:58:37 -04:00
Moritz Kiefer
f7b2f14294
Add Oracle support in the trigger service (#9286)
* Add Oracle support in the trigger service

This PR migrates the ddl & queries and adds tests for this. It does
not yet expose this to users. I’ll handle that in a separate PR.

changelog_begin
changelog_end

* use getOrElse

changelog_begin
changelog_end
2021-03-31 18:39:08 +02:00
Stefano Baghino
a35400f461
Fix copy-pasta in HTTP JSON API EE options (#9296)
changelog_begin
changelog_end
2021-03-31 08:57:51 +02:00
Stephen Compall
480216e50b
Oracle json-api: more queries (#9159)
* support scalaz.Foldable1 in Fragments.in

* incorporating signatories and observers in Oracle contract query

* join syntax; allowed aggregation

* aggregate the signatories and observers independently before join

- prior: ERROR at line 8 (the GROUP BY line):
         ORA-00932: inconsistent datatypes: expected - got CLOB

* make toSqlWhereClause portable, mostly

* name the constraints for debugging

* import cleanup

* skip inserting contract on conflict (for read committed)

* support lookup by contract ID

* remove ::jsonb from fetch-by-key for Oracle

* proper key comparison and retrieval

* on conflict ignore for signatories and observers

* contract ID, party, offset, package ID column types

* template module and entity name types

- nvarchar2 for name type because

        ,template_module_name CLOB NOT NULL
        ,template_entity_name CLOB NOT NULL
        ,UNIQUE (package_id, template_module_name, template_entity_name)
        )
    , Error Msg = ORA-02329: column of datatype LOB cannot be unique or a primary key

* type-aware == and @> output for Oracle

* pick arbitrary maximum module/entity name size

      Cause: Error : 1450, Position : 0, Sql =
      CREATE TABLE
        template_id
        (tpid NUMBER(19,0) GENERATED ALWAYS AS IDENTITY NOT NULL PRIMARY KEY
        ,package_id NVARCHAR2(64) NOT NULL
        ,template_module_name NVARCHAR2(1594) NOT NULL
        ,template_entity_name NVARCHAR2(1594) NOT NULL
        ,UNIQUE (package_id, template_module_name, template_entity_name)
        )
    , Error Msg = ORA-01450: maximum key length (6398) exceeded

* happy path for query-less queries

* done todo

CHANGELOG_BEGIN
CHANGELOG_END

* handle 2.13 deprecation

* factor NVARCHAR2(255)s

- suggested by @cocreature; thanks

* deal with where only a signatory OR observer matches

- suggested by @cocreature; thanks
2021-03-29 15:29:12 -04:00
Gary Verhaegen
cbba75a2c0
oracle: switch to entreprise (#9254)
* oracle: switch to entreprise

CHANGELOG_BEGIN
CHANGELOG_END

* --network host

changelog_begin
changelog_end

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-03-29 16:56:30 +02:00
Hubert Slojewski
cb422e2f67
KVL-810 Report the correct gRPC error code on failures due to concurrent transactions (#9218)
* Fix gRPC status codes for inconsistency rejections and DamlLf errors
Also, add unit tests and exclude failing compatibility and conformance tests

CHANGELOG_BEGIN

- [Integration Kit] Fix gRPC status codes for inconsistency rejections and DamlLf errors (ContractNotFound, ReplayMismatch) by changing them from INVALID_ARGUMENT to ABORTED

CHANGELOG_END
2021-03-26 15:26:33 +01:00
Stefano Baghino
1b7491b8c9
Remove code duplication of driver names in HTTP JSON API (#9239)
changelog_begin
changelog_end

Closes #9229
2021-03-25 10:44:54 +00:00
Stefano Baghino
3fa05804bf
Make non-repudiation integration exclusive to enterprise edition (#9219)
* Make non-repudiation integration exclusive to enterprise edition

Closes #9182

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/9219#discussion_r599578614

* Address https://github.com/digital-asset/daml/pull/9219#pullrequestreview-618605511

* Add aliases for CE test targets, thanks to @S11001001 for the feedback

* Uh, turns out you need a main...

* Address https://github.com/digital-asset/daml/pull/9219#discussion_r599717526

* Add missing dependency for Oracle
2021-03-24 20:25:47 +01:00
Moritz Kiefer
a93f89b6f8
Factor out an Oracle test fixture (#9191)
* Factor out an Oracle test fixture

We need to add Oracle support to other components like the trigger
service as well. At that point, we cannot assume that things don’t
stomp on each other so this PR adds a fixture that generates a random
user (which also comes with its own schema, creating new databases is
a bit different in oracle) which we use testing.

changelog_begin
changelog_end

* Update libs-scala/oracle-testing/src/main/scala/testing/oracle/OracleAround.scala

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* Less top-level stuff in traits

changelog_begin
changelog_end

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2021-03-19 16:37:36 +01:00
Hubert Slojewski
51f495e758
KVL-203 Share participant state integration test harness (#9143)
* Expose libraries for integration testing purposes

The motivation of these changes is to eliminate manual work and reduce duplication between the SDK and oem-integration-kit repos by reusing the same test fixture for integration testing participant state implementations. Also, the DARs required for running these tests won't need to be manually updated.

CHANGELOG_BEGIN
CHANGELOG_END

* Fix a concurrency issue in integration tests

* Fix Bazel error

* Fix conflict resolution

* Move inline daml-lf to separate dar files

* Add a comment

* Add a missing artifact

* Extract method

* Remove maven tags

* Add a macro for Scala libraries with dar resources

* Improve the macro

* Add missing artifact

* Simplify the tests

* Format signature

* Fix the maven tag

* Add missing copyright headers

* Format bazel files

* Make //ledger/test-common lf version dependent (to avoid jar hell)

* Move da_scala_dar_resources_library to a separate bzl file

* Add missing artifacts

Co-authored-by: Hubert Slojewski <hubert.slojewski@tesco.com>
2021-03-19 16:29:02 +01:00
Stefano Baghino
f0c8b1240d
Integrate non-repudiation into HTTP JSON API (#9180)
* Integrate non-repudiation into HTTP JSON API

changelog_begin
changelog_end

Closes #9094

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596400828

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596408663

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596402533

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596624708

* Address https://github.com/digital-asset/daml/pull/9180#discussion_r596624174

* Solve compilation error in daml-script

* Address https://github.com/digital-asset/daml/pull/9180#issuecomment-801875959

changelog_begin
- [JSON API] The JAR for the HTTP JSON API is no longer published on
Maven, use the fat JAR as indicated in the documentation.
changelog_end
2021-03-18 15:15:52 +00:00
nickchapman-da
9b2158508b
Add new variant to Value.scala for builtin-exceptions. (#9084)
* Add new variant to Value.scala for builtin-exceptions.

final case class ValueBuiltinException[+Cid](tag: String, value: Value[Cid]) extends Value[Cid]

And push through the code consequences.
Most places fixed up.
A couple more things to do in this PR (marked NICK)
A couple of things which can be left for later (marked 8020)

fix build

fix another scala match

changelog_begin
changelog_end

* fix any match

* add marker of code which needs attending to in the PR

* extend ledger-api value.proto & fix LfEngineToApi

* undo/comment-out the change to value.proto

* add tests in HashSpec for BuiltinException

* code but dont yet enable value-gen for builtin exceptions

* address comments which suggest we crash in various places

* support BuiltinException in scenario_service.proto

* one more TODO 8020 tag
2021-03-16 13:15:33 +00:00
Gary Verhaegen
1df2270cc9
typo (#9145)
changelog_begin
changelog_end
2021-03-16 11:30:57 +01:00
Moritz Kiefer
8e25227cbb
Fix heartbeating in the JSON API when starting from offset (#9104)
We filter heartbeat ticks until we get the first step message. This is
when starting from the ACS but it is incorrect when starting from an
existing offset where this results in us not emitting heartbeats until
we get the first step message. This PR fixes this by passing along the
initial offset and adds a test for this.

changelog_begin

- [Json Api] Fix a bug where heartbeating on websocket connections did not start until the first transaction was received when resuming from a previous ledger offset. See https://github.com/digital-asset/daml/issues/9102

changelog_end

fixes #9102
2021-03-11 19:37:57 +01:00
Moritz Kiefer
5197374cab
Build SDK EE tarball (#9049)
* Build SDK EE tarball

This sets up the infrastructure to build an SDK EE tarball and allows
for swapping out all files included in the tarball depending on the
edition. As an example, this includes the JSON API with (partial)
Oracle support in the EE tarball.

This PR does not yet address publishing this artifact to Artifactory.
I’ll tackle that in a separate PR.

changelog_begin
changelog_end

* Build in temp dir because Windows is stupid

changelog_begin
changelog_end

* directories are bad

changelog_begin
changelog_end

* Navigator resources are actually needed

changelog_begin
changelog_end
2021-03-08 17:48:00 +01:00
Kamil Bożek
32d4bf92ec
Do not require a JWT token for Health and Reflection services [DPP-277] (#8969)
* Do not require a JWT token for Health and Reflection services

CHANGELOG_BEGIN
- A JWT token is no longer required to call methods of Health and Reflection services
CHANGELOG_END

* Let service's authorizer decide about rejections

* Updated authorization test

* Added integration test for unsecured authorisation test for the Health service

* Added integration test for unsecured authorisation test for the Server Reflection service

* Updated Claims doc comments

* Minor change

* Reduced code duplication with SecuredServiceCallAuthTests and UnsecuredServiceCallAuthTests

* Added copyrights

* Move response status handling logic to Authorizer
2021-03-03 11:05:35 +00:00
Remy
a9b035e92e
LF: release LF 1.12 (#8856)
CHANGELOG_BEGIN

    - [LF] Release LF 1.12. This version reduce the size of transaction

    - [Compiler]: Change the default LF output from 1.8 to 1.11.

CHANGELOG_END
2021-02-24 21:17:33 +01:00
Moritz Kiefer
abf061f694
Fix flakiness in JSON API integration tests (#8933)
The tests start the stream in parallel to submitting commands. This is
problematic since it means that those commands can either be included
in the ACS block or they can come after the ACS block. This PR polls
for the ACS block upfront which makes sure that the commands come afterwards.

changelog_begin
changelog_end
2021-02-24 14:00:19 +01:00
Stephen Compall
2e671e4e5d
non-empty newtypes (#8516)
* non-empty newtypes

* an operation

* add some map/set operations and make everything compile on 2.12 and 2.13

* +-: and :-+, with compatibility layer; docs

* move to nonempty package; add aliases for cons/snoc; fix SeqOps aliases

* ensure 2.12 aliases are inferrable

* groupBy1 and toList, use to prove uniqueSets's invariants

* prove immutability first

* matching variance in aliases

* prove the return property of uniqueSets, and use the proof

* tests for NonEmpty API

* rename sci alias to imm

* move RefinedOps to more obvious location

* more docs

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused imports

* illustrate the scala.collection.Seq problem

* ideas for extension

* tests for toF

* tests for +-:

* explain difference with OneAnd
2021-02-22 08:54:26 -05:00
Remy
d92f2c7003
Ledger: refactor bazel packaging of tedger test tool (#8894)
improve previous generalization from #8695

- use lf version instead keyword (like 'stable', 'latest', 'dev') to
  tag actual target.  This will allow two keywords to map to the same
  versions without doing twice the compilation/test work.

- use alias to map keywords tag target to versioned tag target.

- move package manage dar to test_commong.

CHANGELOG_BEGIN
CHANGELOG_END
2021-02-22 11:41:19 +01:00
Moritz Kiefer
7255baeb0f
Switch to @platforms//:incompatible (#8873)
changelog_begin
changelog_end
2021-02-17 10:15:17 +00:00
Stephen Compall
b94b5f92f3
experimental Oracle support in json-api (#8596)
* separate OracleQueries from PostgresQueries

- with some changes from 8161e63189 courtesy @cocreature

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* abstract BIGINT

* json, signatories, observers columns

* compatible lastOffset

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* oracle functions for select (single template ID), insert

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* add oracle branch to integration tests

* oracle CLI configuration for json-api

* run integration tests with ojdbc in classpath

* update maven_install for ojdbc

* drop table if exists for Oracle

* make create DDLs and drops more planned out; drop in reverse order for Oracle integrity

* repin maven

* port agreement_text

* port (by removal) array part of ledger offset update

* use CASE instead of JSON map lookup for multiparty offset update

* simplify self types

* fix contract archival

* repin

* remove selectContracts in favor of selectContractsMultiTemplate

* move Oracle test execution to separate build target

* move websocket test to itlib

* make a bad array instance for Oracle

* report actually-available JDBC drivers only

* configure Oracle test from CI

* attempt with platforms and constraints

* a mismash of bazel to get it to conditionally enable oracle testing

* fix dep resolution in Scala 2.13

* make the Oracle test a stub (inits and does empty DB query)

* remove commented unused deps

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* repin

* we never supply a value for the surrogate ID columns

- suggested by @cocreature; thanks

* add not null to json in DB-specific place

- suggested by @cocreature; thanks

* why DBContractKey

- suggested by @cocreature; thanks

* textType isn't finalized

- suggested by @cocreature; thanks

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-02-17 03:50:35 -05:00
Moritz Kiefer
7ae28beff9
More Scala 2.13 cleanup (#8855)
This fixes Scaladoc and our pom file generation.

It also clears up the confusing error around gatling and removes a
redundant dependency on sbt (no idea why we had that in the first
place) both of which resulted in Scala 2.12 dependencies in our 2.13
lockfile which is obviously bad.

With this, we should now be ready to publish Scala 2.13 artifacts once
the ledger API test tool PR lands.

changelog_begin
changelog_end
2021-02-16 09:39:16 +00:00
Moritz Kiefer
153a23d315
Port the rest of //ledger-service/... to 2.13 (#8836)
changelog_begin
changelog_end
2021-02-12 17:26:19 +01:00
Moritz Kiefer
c8b82f8e7a
Port HTTP JSON API binary to 2.13 (#8824)
Tests are still missing. Simply haven’t found the time yet, I’ll
tackle those separately.

changelog_begin
changelog_end
2021-02-12 09:54:47 +01:00
Moritz Kiefer
beb8a2a9ba
Port parts of //ledger-service to Scala 2.13 (#8776)
Unfortunately missing the actual interesting part since porting
`partitionBimap` seems to be rather annoying but this at least gets us
started on the easy parts.

changelog_begin
changelog_end
2021-02-08 11:27:50 +00:00
Stefano Baghino
42e071f2c2
Add //libs-scala/doobie-slf4j and replace usages of jdkLogHandler (#8731)
The jdkLogHandler provided by Doobie exists purely as an example and the library
itself does not recommend using it in production.

Note that this slightly changes the runtime behavior, logging successful queries
at debug level rather then info. The message itself is preserved from the original
MIT-licensed example.

This uses Slf4j as most of our components, instead of java.util.logging.

changelog_begin
[HTTP JSON API] The server now logs successful queries at debug level
instead of info
[Trigger Service] The server now logs successful queries at debug level
instead of info
changelog_end
2021-02-03 10:45:05 +00:00
Stephen Compall
e51bcd54d4
lengthen timeout for FlowUtilTest from 150ms to 5s (#8639)
- found by @stefanobaghino-da; thanks

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-27 14:48:14 +00:00
Stephen Compall
00a5fe50ca
soft-code Postgres in json-api (#8517)
* SupportedJdbcDriver box for the required DB-specific implicits and magic values

* replace postgres references with the SupportedJdbcDriver box

* explaining the typeclass

* labels for debugging

* allow external initialization of SupportedJdbcDriver, but not usage

* thread SupportedJdbcDriver everywhere, hang it off of ContractDao

* remove unused dep from integration tests

* split Queries into an agnostic part and a DB-specific part

* document withOptPrefix

* reformat

* SQL syntax more amenable to refactoring

* different sets of DDL for different backends

* make everything use queries passed around everywhere (usually via ContractDao)

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-21 07:22:33 -05:00
Stephen Compall
c0c2a59275
hybrid database-then-gRPC websocket query (#8226)
* group database queries while still providing the matchedQueries part of the event

* SQL query for multiquery websocket request all at once

* fetchAndPersist responds with a single bookmark

* unify connection imports

* convert DB results to domain, reassociate with proper query indices

* reassociate multiple matches with proper query indices

* make overlap more likely in testing overlap

* simpler matchedQueries merging for multi-query case

* integrate DB query with metadata-ful StreamQuery's

* expose daoAndFetch; better insertDeleteStepSource doc

* more efficient query path for contract key streams

* missed LogHandler

* persist resolved template IDs; glue the prefiltered set into the stream

* ticks and phantom removal need the state from the ACS

* compile SQL queries for query language predicates on WS

- wrong matchedQueries order

* harmonize order of ACS-vector and Positives

* misc compilation conversions

* WebsocketServicePostgresInt mixin

* update (C) date

* test websocket queries under postgres

* looking for new way to compile queries with proper matchedQueries offsets

* model that querying without matchedQueries requires only one SQL query

* SQL path for contract key streams

* nondeterminism

* fix 3 fetch tests with SQL syntax

* nondeterminism mk 2

* fix multi-party query tests by dealing with nondeterminism properly

* temp logs to track down the contract duplication

* match new scalafmt from #8437

* remove completed TODOs

* add changelog

CHANGELOG_BEGIN
- [JSON API] If the JDBC query store is enabled, it will be used to optimize
  Websocket queries as well as the previously-supported synchronous queries.
  See `issue #8226 <https://github.com/digital-asset/daml/pull/8226>`__.
CHANGELOG_END

* fix up matchedQueries indices

* complete the fast path for by-id queries

* remove AS c

- suggested by @cocreature; thanks

* remove temporary debugging logs

- suggested by @cocreature; thanks
2021-01-14 10:01:17 -05:00
Stephen Compall
89bc670445
fix race condition in DB update when ACS has later contracts than the ledger-end (#8480)
* fix race condition in DB update when ACS has later contracts than the ledger-end

CHANGELOG_BEGIN
- [JSON API] Under rare conditions, a multi-template query backed by Postgres
  could have mismatched snapshots of the ACS for different templates.  These
  conditions are now checked and accounted for.
  See `issue #8226 <https://github.com/digital-asset/daml/pull/8226#issuecomment-756446537>`__.
CHANGELOG_END

* contractsFromOffsetIo already saves the offset to DB

* notes on why we rerun DB update to fix the race condition

* if the ACS last-offset exceeds the tx stream offset, save it in the DB instead
2021-01-13 11:27:33 -05:00
Moritz Kiefer
aebde84842
Upgrade scopt to 4.0.0 (#8466)
* Upgrade scopt to 4.0.0

Scopt 3.x has some issues with Scala 2.13 because it expects an
immutable Seq on 2.13 meaning you cannot just pass in an Array. Rather
than fixing our callsites to convert to an immutable Seq everywhere,
this PR bumps to Scopt 4.0 which goes back to collection.Seq.

and leaving that aside, I’m a fan of upgrading dependencies anyway :)

changelog_begin
changelog_end

* Use val instead of def

changelog_begin
changelog_end
2021-01-11 21:23:02 +01:00
Moritz Kiefer
22ce940954
Upgrade scalafmt and enable trailing commas (#8437)
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.

As announced, this will be merged on Saturday to avoid too many conflicts.

changelog_begin
changelog_end
2021-01-09 11:37:37 +01:00
Bernhard Elsner
cda93db944
Daml case and logo (#8433)
* Replace many occurrences of DAML with Daml

* Update docs logo

* A few more CLI occurrences

CHANGELOG_BEGIN
- Change DAML capitalization and docs logo
CHANGELOG_END

* Fix some over-eager replacements

* A few mor occurrences in md files

* Address comments in *.proto files

* Change case in comments and strings in .ts files

* Revert changes to frozen proto files

* Also revert LF 1.11

* Update get-daml.sh

* Update windows installer

* Include .py files

* Include comments in .daml files

* More instances in the assistant CLI

* some more help texts
2021-01-08 12:50:15 +00:00
Moritz Kiefer
424faa923a
Port damlc dependencies to Scala 2.13 (#8423)
* Port damlc dependencies to Scala 2.13

I got a bit fed up by the fact that going directory by directory
didn’t really work since there are two many interdependencies in
tests (e.g., client tests depend on sandbox, sandbox tests depend on
clients, engine tests depend on DARs which depend on damlc, …).

So before attempting to continue with the per-directory process, this
is a bruteforce approach to break a lot of those cycles by porting all
dependencies of damlc which includes client bindings (for DAML Script)
and Sandbox Classic (also for DAML Script).

If this is too annoying to review let me know and I’ll try to split it
up into a few chunks.

changelog_begin
changelog_end

* Update daml-lf/data/src/main/2.13/com/daml/lf/data/LawlessTraversals.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* fixup lawlesstraversal

changelog_begin
changelog_end

* less iterator more view

changelog_begin
changelog_end

* document safety of unsafeWrapArray

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2021-01-08 07:22:38 +01:00
Moritz Kiefer
746b5f4774
Remove a few redundant toString calls (#8379)
Spotbugs yelled at me and while they don’t do any harm they might be slightly confusing.

changelog_begin
changelog_end
2021-01-05 08:31:15 +01:00
Moritz Kiefer
0541b2b888
Prefer Integer.valueOf over new Integer (#8381)
The latter is deprecated in newer JDKs.

changelog_begin
changelog_end
2021-01-04 17:37:33 +00:00
Gary Verhaegen
a925f0174c
update copyright notices for 2021 (#8257)
* update copyright notices for 2021

To be merged on 2021-01-01.

CHANGELOG_BEGIN
CHANGELOG_END

* patch-bazel-windows & da-ghc-lib
2021-01-01 19:49:51 +01:00
Moritz Kiefer
494077cb3d
Support multi-party submissions in the JSON API (#8328)
* Support multi-party submissions in the JSON API

changelog_begin

- [JSON API] Add support for multi-party submissions by allowing for
multiple actAs parties in the token and passing on readAs to the
ledger.

changelog_end

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/EndpointsCompanion.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-12-17 21:10:09 +00:00
Moritz Kiefer
76b6fd86fb
Upgrade Scala dependencies for 2.13 compatibility (#8268)
* Upgrade Scala dependencies for 2.13 compatibility

This upgrades a bunch of Scala libraries to versions that have 2.13
support. There are two libraries that are still missing:

- diffson, this has a new version but with significant breaking
  changes and it is only used in Naigator console which I hope to kill
  before I have to worry about this.

- ai.x:diff, this is used in the ledger API test tool. The library is
  abondened but there are a few alternatives.

changelog_begin
changelog_end

* Fix pureconfig

changelog_begin
changelog_end

* Fix Navigator

changelog_begin
changelog_end
2020-12-14 09:59:00 +01:00
Moritz Kiefer
7e05dc7932
Upgrade rules-scala and scalatest (#8187)
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.

This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.

Apologies for the giant PR, I don’t see a way to keep it smaller.

changelog_begin
changelog_end
2020-12-08 06:59:23 +01:00
Stephen Compall
8ec103ea0b
speed up sync fetch by contract ID/key with postgres and faster in-memory filtering (#7993)
* mark some of dbbackend private

* fetchBy* functions for Queries

* shift in-memory filtering into the transaction stream

- removes irrelevant contracts from memory as soon as possible for fetch by
  contract ID and key in-memory

* push the three synchronous search varieties into new signatures

* replace the core findByContract* functions with DB-delegating versions

* remove the GADT equality and most of the explicit traversals

- compiles again, finally

* factoring

* ContractDao wrappers for fetchById and fetchByKey

* DB version of findByContractId

* DB version of findByContractKey

* Search is the split of ContractsService

* fix SQL for keys

* trade the typeclass for a sum type

- sealed instead of final because of the path dependency on ContractsService
  instance

* number conversion is done already in ContractDao

* make fetch-by-key tests depend on proper number conversion for SQL

* add changelog

CHANGELOG_BEGIN
- [JSON API] ``/v1/fetch`` now uses the Postgres database, if configured, to
  look up contracts by ID or key, except when querying a contract by ID without
  its corresponding template ID.  The fallback in-memory version of
  ``/v1/fetch`` is also now significantly more efficient for large datasets,
  though still linear.

  You may optionally re-create JSON API's database to take full advantage.
  See `issue #7993 <https://github.com/digital-asset/daml/pull/7993>`__.
CHANGELOG_END

* use search.search for search

- suggested by @cocreature; thanks

* add an index for contract key lookups

- suggested by @cocreature; thanks
2020-11-30 13:51:29 -05:00
Samir Talwar
9976b4cd50
Bazel: Factor out logic around Protobuf JARs. [KVL-714] (#8084)
* kvutils: Use ScalaPB to generate a Scala JAR for daml_kvutils.proto.

* Bazel: Delete the unused `da_java_binary` rule, and inline `_wrap_rule`.

* Bazel: Factor out Java/Scala protobuf class generation into a helper.

CHANGELOG_BEGIN
CHANGELOG_END

* daml-lf/archive: Use `proto_jars`.

* Bazel: Remove the visibility modifier from `proto_jars`.

It's too confusing. Just make everything public.

* daml-lf/archive: Push protobuf source tarballs into `proto_jars`.

* Bazel: Add comments to the various parts of `proto_jars`.

* daml-assistant: Do unpleasant things with `location` in Bazel.
2020-11-27 08:34:53 +00:00
Moritz Kiefer
2d3820ac14
Upgrade akka-http to 10.2 (#8058)
* Upgrade akka-http to 10.2

Follow up to #8048, I left out this upgrade to reduce noise and since
I wasn’t quite sure how involved it was going to be.

changelog_begin
changelog_end

* Reenable transparent HEAD requests

Apparently no longer on by default but we depend on this in waitForHttpServer

changelog_begin
changelog_end
2020-11-25 13:39:25 +01:00
Moritz Kiefer
d83cbdb475
Upgrade akka and akka-http (#8048)
* Upgrade akka and akka-http

Was chasing an issue somewhere and thought this might affect it in
some way. It didn’t but I might as well turn the upgrade into a PR.

changelog_begin
changelog_end

* Fix trigger service tests

changelog_begin
changelog_end

* Downgrade akka-http again

changelog_begin
changelog_end

* Upgrade akka-http again and fix tests

changelog_begin
changelog_end

* Cleanup trigger service

changelog_begin
changelog_end
2020-11-25 10:13:51 +01:00
Moritz Kiefer
45ba167aac
Dedup parties in JwtPayload (#7973)
Previously we didn’t build up the `OneAnd[Set, Party]` properly and
included the one party in the set as well. This was an issue if you
have the same party multiple times, most likely in readAs and
actAs (but not limited to that). This then lead to SQL queries failing
since we tried to insert twice for a given party. This PR fixes that
by properly deduplicating the parties and adding a test for this.

changelog_begin

- [JSON API] Fix a regression introduced in SDK 1.7.0, where using a
  party multiple times in the same JWT token (e.g., readAs and actAs)
  broke database queries for that party. Note that there is never a
  reason to include a party multiple times since actAs implies readAs.

changelog_end
2020-11-16 19:28:11 +01:00
Moritz Kiefer
bdf1c48947
Make HealthService public (#7887)
* Make HealthService public

DABL patches the rest adapter so making this public helps them plug it
together with other things.

Also removes some garbage debug print which I forgot to remove
🤦

changelog_begin
changelog_end

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/HealthService.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* No logging

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-11-04 19:13:54 +01:00
Moritz Kiefer
a74a1c15f4
Add healthcheck endpoints to JSON API (#7852)
* Add healthcheck endpoints to JSON API

This PR adds /livez and /readyz (following k8s naming scheme) that can
be used as liveness and readyness check. There isn’t much we can do
for liveness apart from showing that we can still respond to http
requests but readyness can be a bit more clever and check the ledger
connection as well as the database connection.

changelog_begin

- [JSON API] Add `/livez` and `/readyz` health check endpoints for
  easier integration with k8s and other schedulers.

changelog_end

* I hate windows

changelog_begin
changelog_end
2020-11-02 18:41:49 +01:00
Moritz Kiefer
37f389640a
Use chunked responses in the JSON API (#7836)
HTTP 1.1 exists since 1999 so there isn’t really a good reason not do
use this. In fact, the docs recommend to use Chunked in favor of
CloseDelimited.

changelog_begin
changelog_end
2020-10-28 18:43:10 +01:00
Andreas Herrmann
b452e07d05
Move common jwt cli flags (#7811)
* move cliopts

changelog_begin
changelog_end

* update artifacts.yaml

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-26 17:53:37 +00:00
Moritz Kiefer
22f14b72c5
Test for database failures in JSON API (#7808)
changelog_begin
changelog_end
2020-10-26 18:18:46 +01:00
Andreas Herrmann
cd56af1cdd
Factor out the JWT verifier CLI options (#7780)
* Factor JWT verifier CLI flags

changelog_begin
changelog_end

* Use cli-opts in auth middleware

* Use cli-opts in sandbox cli

* Mark trigger service test as long

These have become prone to timeout on CI.

Increasing the size (timeout) is a temporary fix. A proper
solution is to a) not start a fresh sandbox per test-case and b)
separate the in-mem/db and no-auth/auth configrations into
separate Bazel test targets.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-26 15:10:40 +01:00
Moritz Kiefer
339704dd14
Tests for timeouts to/from ledger in JSON API failure tests (#7791)
* Tests for timeouts to/from ledger in JSON API failure tests

changelog_begin
changelog_end

* Update ledger-service/http-json-testing/src/main/scala/com/daml/http/HttpServiceTestFixture.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Update ledger-service/http-json-testing/src/main/scala/com/daml/http/HttpServiceTestFixture.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Apply suggestions from code review

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-10-26 09:08:03 +01:00
Moritz Kiefer
19a7f326f1
Include streaming API in JSON API failure tests (#7783)
This PR extends the failure tests with one that establishes a
websocket connection, kills the connection and reenables it at the
last offset.

changelog_begin
changelog_end
2020-10-23 09:12:09 +02:00
Moritz Kiefer
9da86a602e
Relax restrictions on JWTs on write endpoints in JSON API (#7712)
The previous restriction was both too lax and too strict now that we
have multi pary queries:

1. It allowed a party in `readAs` for command submissions which just
   fails on the ledger side. Changing this is technically breaking but
   only if you used a token that would have been rejected as soon as you
   enabled auth so that seems very resonable to break.

2. It didn’t allow extra parties in `readAs`.

This PR switches to requiring exactly one party in `actAs` while
supporting multiple parties in `readAs`.

changelog_begin

- [JSON API] JWTs on command submissions can now contain extra parties
  in the `readAs` field. `actAs` is still limited to a single party.

changelog_end
2020-10-20 19:37:44 +02:00
Moritz Kiefer
ea453c35cf
Add tests for connection failures in the JSON API (#7751)
* Add tests for connection failures in the JSON API

This PR adds some toxiproxy based tests to see how the JSON API reacts
if the connection to the ledger is killed. There are a bunch of
inconsistencies here in the tests some of which we might want to
address and the others we should at least document but I’ll leave that
for future PRs.

changelog_begin
changelog_end

* Import HttpServiceTestFixture instead of prefixing

changelog_begin
changelog_end
2020-10-20 18:59:11 +02:00
Stephen Compall
4ca02e0eb6
upgrade doobie from 0.6.0 to 0.9.2 (#7618)
* set doobie version to 0.9.2 and rerun maven pin

* port extractor and some of JSON API

* repin maven

* use doobie's own builder compatibility where required

* use probably bad derivations to supply Blockers where transactEC was required

- The point of using Blocker instead of ExecutionContext seems to be to
  especially emphasize to API users that it isn't appropriate to use an
  ExecutionContext with ordinary behavior.  That is what we have done, which
  should probably change, but just compiling for now.

* fix fragment inspection test for internal restructuring

- This test depends on implementation details of Doobie, so naturally it must be
  altered when that runs.  Fortunately, it's been made easier by the changes
  in this upgrade.

* allow 256 blockers for navigator transaction blocker, like the global EC

* allow as many blockers as the pool size for trigger service

- The transactor shouldn't share ExecutionContext for transactions with the
  caller, so we set up a new one based on configured pool size.

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-16 09:46:20 -04:00
Moritz Kiefer
40335c5ccd
Support specifying the jdbc URL for the JSON API in an env var (#7686)
This PR shuffles things around a bit to make it easier to test and
adds `--query-store-jdbc-config-env` which specifies the name of an
environment variable containing the jdbc URL. The UX for this is
modeled after #7660.

Added a test for the different formats.

fixes #7667

changelog_begin

- [JSON API] The JDBC url can now also be specified via
`--query-store-jdbc-config-env` which reads it from the given
environment variable.

changelog_end
2020-10-14 20:19:35 +02:00
Stephen Compall
bafde51752
add silent_annotations option to da_scala bazel functions (#7668)
* add silent_annotations option to da scala bazel functions

* use silent_annotations for several scala targets

* use silencer_plugin instead when the lib isn't used

* use silent_annotations for several more scala targets

* use silencer_lib for strange indirect requirement for running tests

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* silent_annotations support for scaladoc
2020-10-13 15:44:16 +00:00
Stephen Compall
1a2afd5266
upgrade to Scala 2.12.12 from 2.12.11 (#7661)
* upgrade bazel settings to scala 2.12.12

* upgrade nix scala tool to scala 2.12.12

* upgrade silencer references to scala 2.12.12

* repin for scala 2.12, silencer, wartremover upgrades

* remove numerous occurrences of unused silencer now spotted

* update Scala version in our bazel notes

CHANGELOG_BEGIN
CHANGELOG_END

* update compatibility maven_install.json to match compatibility WORKSPACE
2020-10-13 08:42:14 -04:00
Moritz Kiefer
aa3e5a7dbe
Support multi-party reads on the JSON API (#7571)
* Support multi-party reads on the JSON API

Given that those aren’t going away and we’re instead doubling down on
this and adding multi-party writes as well, the JSON API needs to
support this. This PR only implements the read side (since the ledgers
do not yet support the write side).

This does not deviate from the approach chosen by the JSON API to
infer the parties from the token, we just don’t error out anymore when
more than one party is passed.

changelog_begin
changelog_end

* Apply suggestions from code review

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Remove dependency on doobie_postgres from db-backend

changelog_begin
changelog_end

* Fix offset update

changelog_begin
changelog_end

* Use nonempty sets for parties

changelog_begin
changelog_end

* Fix updateOffset under concurrent transactions

changelog_begin
changelog_end

* Add tests for multi-party websocket queries and fetches

changelog_begin
changelog_end

* fmt

changelog_begin
changelog_end

* Fix perf tests

changelog_begin
changelog_end

* Cleanup

changelog_begin
changelog_end

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/dbbackend/ContractDao.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* Move ParsePayload instances, thanks Stephen!

changelog_begin
changelog_end

* More unsubst

changelog_begin
changelog_end

* Fix off by 1 error

changelog_begin
changelog_end

* Remove redundant type annotation

changelog_begin
changelog_end

* fmt

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-10-08 13:46:15 +02:00
Moritz Kiefer
a34963c16d
Include JSON API logback in scala_binary (#7485)
With the introduction of the standalone JAR, we cannot rely on the
assistant anymore to pass the default logback config. Users can still
override the logback config with `-Dlogback.configurationFile` if they
need something else but this provides a more sensible default logging
config than seeing a ton of debug logs from netty.

changelog_begin
changelog_end
2020-09-25 17:52:50 +02:00
Remy
f5694ee2ea
LF: Kill ValueStruct (#7457)
Struct is not serializable and therefore should not appear as Value.

CHANGELOG_BEGIN
CHANGELOG_END
2020-09-22 13:37:59 +02:00
Leonid Shlyapnikov
0730102b99
JSON API Perf Test Runner Query Store Index Support (#7434)
* Postgresql setup for Gatling perf test

* Add JSON API `query-store-index` support

changelog_begin
changelog_end
2020-09-17 16:39:00 -04:00
Leonid Shlyapnikov
d13e7aa184
JSON API daily perf test cron job (#7406)
* Add http-json-perf daily cron job

changelog_begin
changelog_end

* commenting out other jobs so we can manually test the new one

* commenting out other jobs so we can manually test the new one

* Fix the shell script

* Fixing the gs bucket, `gs://http-json bucket does not exist`

* uncomment the other jobs

* timestamp from git log

* get rid of DAR copying

* comment out the other jobs, so we can test it

* uncomment the other jobs
2020-09-16 13:02:02 -04:00
Leonid Shlyapnikov
fe63816431
Write report summary in text format (#7407)
* write summary.txt

changelog_begin
changelog_end

* make `Response time distribution` brackets configurable
2020-09-16 09:31:54 -04:00
Leonid Shlyapnikov
ae2981eaf2
Updating readme (#7398)
changelog_begin
changelog_end
2020-09-15 08:51:57 -04:00
Leonid Shlyapnikov
4fbde3c672
Open-sourcing Gatling statistics reporter (#7325)
* Open sourcing gatling statistics reporter

Running gatling scenarios with `RunLikeGatling` from libs-scala/gatling-utils

* cleaning up

* Replace "\n" with System.lineSeparator

so the formatting test cases pass on windows

* Testing DurationStatistics Monoid laws

* Renaming RunLikeGatling -> CustomRunner
2020-09-11 13:39:05 +00:00
Remy
78e770bf72
LF: Use the new Struct data structure for ValueStruct (#7241)
This PR uses the new data structure introduced in #7220.
Additionnally this fix `Value Equal instance` which was considering
<a: x, b: y> different from <b:y, a:x>.

CHANGELOG_BEGIN
CHANGELOG_END
2020-09-04 18:32:39 +02:00
Leonid Shlyapnikov
6186c5ccfd
Fixing flaky Template ID resolution test (#7307)
* Adding `uniqueModuleEntity`

and making sure that generated domain Template IDs do not
have module entity duplicates, so resolution should work
with no problem.

changelog_begin
changelog_end

* cleaning up
2020-09-02 12:17:37 -04:00
Leonid Shlyapnikov
ccecc46e70
JSON API Perf Scenario Runner (#7295)
* Run gatling scenario from the perf runner main

reports are disabled for now, getting a class not found
when generating them

changelog_begin
changelog_end
2020-09-02 09:00:43 -04:00
Samir Talwar
15cb93eec3
http-json: Use matchers in WebsocketServiceIntegrationTest. (#7302)
This makes diagnosing failures in CI easier, because "true is not false"
is not a very helpful error message.

CHANGELOG_BEGIN
CHANGELOG_END
2020-09-02 11:52:05 +00:00
Moritz Kiefer
abb979a4db
Fix flakiness in multi-key tests (#7298)
There are two sources of flakiness and I’ve seen both on CI:

1. We can get more than one offset at the beginning if things are too
   slow. This is addressed by just filtering those out.
2. The stream completes as soon as the input is closed. This is
   addressed by keeping the stream open and closing it with `take`.

Point 2 is a problem for all tests, see #7293, but I’ll leave the
other tests for separate PRs (I’ve also never seen them flake).

You can observe the failures locally if you add a `Thread.sleep` between
creating the stream future and sending the commands.

changelog_begin
changelog_end
2020-09-02 12:50:51 +02:00
Samir Talwar
2b3cf1b3c0
ledger-api-client: Rename maxInboundMessageSize to maxInboundMetadataSize. (#7290)
* ledger-api-client: `maxInboundMessageSize` -> `maxInboundMetadataSize`.

CHANGELOG_BEGIN
- [Scala Bindings] Rename a field in the ``LedgerClientConfiguration``
  to ``maxInboundMetadataSize``, to match the builder Netty channel
  builder. It was incorrectly named ``maxInboundMessageSize``, which is
  a different channel property that configures the maximum message size,
  not the header size.
CHANGELOG_END

* ledger-api-client: Introduce a `maxInboundMessageSize` config property.

We use this a lot; easier if it's in the configuration.

CHANGELOG_BEGIN
- [Scala Bindings] Replace the
  ``LedgerClientConfiguration.maxInboundMessageSize`` property with a
  new one that represents the maximum size of the response body.
CHANGELOG_END
2020-09-02 08:41:34 +00:00
Gary Verhaegen
989fdc9bde
fix JSON API multikey stream (#7244)
fix JSON API multikey stream

In the current state, the JSON API only handles multiple keys _from
different templates_. This makes it work for multiple keys from the same
template too.

Extracted from #7066 with the following changes:

- Use of a mutable `HashSet` to test for keys, because perf.
- Addition of a test at the JSON API level.

CHANGELOG_BEGIN

- [JSON API] Fix a bug where streaming multiple keys could silently
  ignore some of the given keys.

CHANGELOG_END

* apply @cocreature's patch

https://gist.github.com/cocreature/d35367153a7331dc15cca4e5ea9098f0

* fix fmt
2020-09-01 18:17:13 +02:00
Leonid Shlyapnikov
20e2922b3e
JSON API perf test main that starts sandbox and json-api services (#7283)
* reintroducing the main

* Introducing `ledger-service/http-json-testing`

* cleaning up

* Starting sandbox and json-api from perf-test main

changelog_begin
changelog_end
2020-08-31 22:55:01 +00:00
Leonid Shlyapnikov
1452076eee
Deprecating noop '--application-id' command-line option in JSON API (#7232)
* Deprecate noop `--application-id`

changelog_begin

[JSON API]
Hiding and deprecating `--application-id` command-line option. JSON API never used it.
It is required to instantiate LedgerClientConfiguration and was not used for any command submission.
JSON API uses Application ID specified in the JWT. See #7162

changelog_end

* removing further usage of noop applicationId

* a bit of explanation what this is for
2020-08-26 20:05:00 +00:00
Gary Verhaegen
07657e01c8
fix docs warning (#7243)
Apparently `[[]]` links don't work to external resources. I wanted to
turn the first referencee into a link too but that contains too many
weird characters.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-26 20:27:33 +02:00
Leonid Shlyapnikov
3ecfcc918c
Adding package-max-inbound-message-size (#7147)
* Adding `package-max-inbound-message-size`

this is to allow separate configuration settings for command submission
and package management ledger clients

* Fixing formatting

* Updating docs

changelog_begin

[JSON API] Adding `--package-max-inbound-message-size` command line option.
Optional max inbound message size in bytes used for uploading and downloading package updates. Defaults to the `max-inbound-message-size` setting.

changelog_end

* Addressing code review comments
2020-08-24 14:55:55 +00:00
Moritz Kiefer
820c9ff844
Remove damlLfInstantiate in favor of TypeCon.instantiate (#7166)
fixes #2506

Judging from the issue this was originally introduced to workaround a
bug. I couldn’t actually track down what that bug was but at this
point they are identical so no point keeping this around.

changelog_begin
changelog_end
2020-08-18 12:15:12 +00:00
Leonid Shlyapnikov
cb73542316
http-json-perf cleanup (#7145)
* Removing unused Main and Config,

cleaning up BUILD.bazel: removing common scala options,

* Removing unused dependencies

* Removing unused dependencies

changelog_begin
changelog_end
2020-08-14 17:02:45 +00:00
Leonid Shlyapnikov
5287e5b946
Do not emit offset ticks preceding the ACS events (#7109)
* Introducing `TickTriggerOrStep` ADT, filtering out `TickTrigger`s preceding the initial ACS retrieval

changelog_begin
[JSON API] Filter out offset ticks preceding the ACS events block. See issue: #6940.
changelog_end

* Cleaning up a bit

* Do not emit offset tick unless we know the real offset

wait for LiveBegin message

* Make WebsocketConfig configurable

* Adding offset tick integration tests

reverting WebsocketService to 05d49b37c3 makes these tests fail

* cleaning up

* Refactoring `emitOffsetTicksAndFilterOutEmptySteps`

keep offset instead of StepAndError with offset
2020-08-13 09:00:28 +02:00
Stephen Compall
1737907415
add --address option to trigger service (#7090)
* factor --address, --http-port, --port-file options from http-json to cli-opts

- enabling reuse in trigger service

* use cli-opts for address and http-port options in Trigger service

* mark ServiceConfig and some defaults private

* use --address option to set up server

* document Setter

* test --address option is parsed

* missing (c) headers

* add changelog

CHANGELOG_BEGIN
- [Trigger Service] Accepts a new ``--address`` option to listen for HTTP connections on
  interfaces other than localhost, such as ``0.0.0.0`` for all addresses.
  See `issue #7090 <https://github.com/digital-asset/daml/pull/7090>`__.
CHANGELOG_END
2020-08-12 13:50:00 -04:00
Samir Talwar
27f76c4386
Use Future.unit instead of Future.successful(()). (#7080)
And in one instance, `Resource.unit`.

I just think it's easier to read. Too many parentheses make Samir a dull
boy.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-11 09:18:50 +00:00
Leonid Shlyapnikov
213832572b
Sync query perf scenario, new ACS every query (#7078)
* AsyncQueryNewAcs scenario

sync query with totally new ACS every time

changelog_begin
changelog_end

* cleanup

* cleanup

* fixing numbers

* fixing numbers

* adding groups to self-document the scenario

* Archive and then Create

* cleanup

* silence archive and create

* with ACS of 5000 it takes too long to run
2020-08-10 22:23:46 +00:00
Leonid Shlyapnikov
ba22f5b9a4
AsyncQueryConstantAcs scenario (#7054)
changelog_begin
changelog_end
2020-08-07 09:33:05 -04:00
Leonid Shlyapnikov
a630e3c944
JSON API SyncQueryVariableAcs perf test scenario (#7028)
* Perf test scenario for query with variable ACS, WIP

* WIP

* change ACS with every query

exercise a choice + create a new contract to keep ACS size the same

* change ACS with every query

running exercise and create in parallel with the query

* exercise Archive instead of Transfer

* Adding copyright header

* Thanks @S11001001
2020-08-05 18:31:18 +00:00
Leonid Shlyapnikov
9acb47d994
JSON API command submission perf test (#6906)
* JSON API Perf test scenarios

changelog_begin
changelog_end

* addressing code review, thanks @S11001001

* addressing code review, thanks @S11001001

* cleanup

* cleanup + instructions

* cleanup + instructions

* Updates

* Updates
2020-08-04 18:51:44 +00:00
Gary Verhaegen
109d184d39
improve error message on failed JSON parsing (#6978)
* improve error message on failed JSON parsing

Fixes #6971.

Interestingly, all the other cases in that block already had useful
feedback, not sure why this ones was missing.

CHANGELOG_BEGIN
CHANGELOG_END

* add tests
2020-08-04 14:56:17 +02:00
Leonid Shlyapnikov
05d49b37c3
Introduce //libs-scala/scala-utils (#6935)
* Moving `Statements.discard` from //ledger-server/http-json into //libs-scala/scala-utils

changelog_begin
changelog_end

* Add new module to the published artifacts

* `com.daml.scalautil` instead of `com.daml.scala.util`

@S11001001: That's because if this is in classpath and you import com.daml._,
you have a different scala in scope than the one you expect.
2020-07-30 13:51:07 -04:00
Samir Talwar
98de16606d
daml-on-sql: An elegant Main class, for a more civilized age. (#6829)
* daml-on-sql: Pull out a new `Main` object that wraps sandbox-classic.

CHANGELOG_BEGIN
CHANGELOG_END

* daml-on-sql: Fail if a JDBC URL is not provided or not for PostgreSQL.

* sandbox-classic: Rename the conformance test H2 database.

* daml-on-sql + sandbox-classic: Report configuration errors cleanly.

This means letting `ProgramResource` catch the errors, log, and exit.

* daml-on-sql: Change the name logged on startup.

* daml-on-sql: Change the default participant ID.

* sandbox-common: Give the ledger name its own tagged string type.

* sandbox-classic: Generate random ledger IDs using the ledger name.

* daml-on-sql: Remove the banner, replacing it with a blank line.

* daml-on-sql: Enable strong seeding by default.

And weak seeding in the conformance tests.

* sandbox-classic: Move the ledger name to a separate parameter.

It's not really configurable.

* sandbox-classic: Move LedgerName from sandbox-common.

* daml-on-sql: Remove "-participant" from the participant ID.

* daml-on-sql: Use `Name` where possible.

* daml-on-sql: Make the ledger ID mandatory.

* Revert "sandbox-classic: Move LedgerName from sandbox-common."

This reverts commit 0dad1584a7.

* daml-on-sql: Print "DAML-on-SQL" in the CLI help, not "Sandbox".

* daml-on-sql + sandbox + sandbox-classic: Split out custom CLI parsing. (#6846)

* participant-state: Simplify naming the seeding modes.
2020-07-24 18:54:19 +00:00
Samir Talwar
adcbfb45a1
sandbox-classic: Move the configuration from sandbox-common. (#6810)
* sandbox-common: Move the ledgerIdMode up in the configuration.

* sandbox-classic: Move the configuration from sandbox-common.

CHANGELOG_BEGIN
CHANGELOG_END

* Fix miscellaneous warnings caught by IntelliJ IDEA.
2020-07-21 14:43:53 +00:00
Stephen Compall
4355406259
add more scalac 2.12 warnings (#6798)
* set many extra scalac -Xlint options for all Scala projects

CHANGELOG_BEGIN
CHANGELOG_END

* move NoCopy to its own file

package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
  trait NoCopy {
        ^

* move more traits, classes, and objects to proper packages

- note that `package` is itself a scoping construct, so if your reason
  is the apparent aesthetic of placing a bunch of things in one `package
  object`, that is easily remedied by deleting the `object` keyword

* fix some type-parameter-shadow warnings

- I'm generally in favor of sensible name-shadowing, following the
  "deliberately hide variables that should not be accessed here" school
  of thought.  But I think type name shadowing isn't quite as valuable
  and more likely to confuse than general variable shadowing, so have
  experimentally linted it out.

  Example warning:

EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
 Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
 Offset defined in package v1. You may want to rename your type
 parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
                                                               ^

* fix more package-object-classes warnings

* fix an inaccessible warning

ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
  def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
      ^

* enable -Xlint:infer-any

- continuing the saga of #6116, #6132

* enable -explaintypes for more detailed type errors

* missed header for NoCopy; probably should have left it in the package file

* misspelling in comment

* revert -Xlint:doc-detached

- there are a lot of these fixes, and they are noisy, so shifting to a
  separate PR
- thanks to @leo-da for pointing out
2020-07-21 08:18:01 -04:00
Robert Autenrieth
7ce9748066
Split sandbox code into separate packages (#6695)
* Move public code into daml-integration-api

CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
2020-07-17 17:06:06 +02:00
Leonid Shlyapnikov
9abfff0388
Fixing typo in the JSON API error response (#6735)
changelog_begin
changelog_end
2020-07-15 09:23:40 -04:00
Gary Verhaegen
8176fb0c8d
fix the the typo (#6723)
```
s/the the /the /
```

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-14 16:55:33 +00:00
Stephen Compall
4cd419623a
replace traverseU and sequenceU with traverse and sequence (#6594)
* replace traverseU and sequenceU with traverse and sequence

- with -Ypartial-unification on, the extra Unapply typeclass lookup is
  unnecessary

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* limit imports; we only need *> and void
2020-07-07 15:01:58 +00:00
Stephen Compall
9fb72202d8
use daml-lf scalacopts for ledger-service (#6569)
* add lf_scalacopts to ledger-service/ projects

* handle unused warnings in http-json main

* handle unused warnings in lf-value-json

* handle unused warnings in http-json tests

* handle unused warnings in jwt

* JwtGenerator.generate doesn't use its argument; remove it

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* format bazel files
2020-07-02 13:58:41 +00:00
Stephen Compall
86148c398b
make @silent annotations narrower (#6570)
* check that all @silent annotations are used

CHANGELOG_BEGIN
CHANGELOG_END

* make all @silent annotations very specific about what they're silencing
2020-07-01 20:53:22 +00:00
Remy
46b68bd92a
ledger test tool: refactor conformance test dars (#6506)
* ledger test tool: refactor conformance test dars

CHANGELOG_BEGIN
CHANGELOG_END
2020-06-26 13:39:00 +02:00
Stephen Compall
a51d0db8ff
set scalac -Xsource:2.13 -Ypartial-unification globally (#6469)
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts

* add now-referenced scalaz-core where needed

* work around bad type signatures in scalatest Aggregating, Containing

* unused Any suppression

* work around bad partial-unification wrought by type alias

* remove unused Conversions import

- not required in 4f68cfc480 either, so unsure how it's survived this long

* work around Future.traverse; remove unused show import

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused bounds

* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed

* longer comment on what the options do

- suggested by @stefanobaghino-da; thanks

* forget Future.traverse, just use scalaz, it knows how to do this
2020-06-24 16:51:24 -04:00
Remy
cbe423fc6c
LF: rename library transaction-scalacheck to transaction-test-lib (#6347)
* LF: rename library transaction-scalacheck to transaction-test-lib

CHANGELOG_BEGIN
CHANGELOG_END

* move files in com/daml

* missing change in release/artifacts.yaml

* remove 'com/dam' from the path
2020-06-17 10:52:40 -04:00
Simon Maxen
5329a4f97f
Ledger id requirement (#6323)
* Add option based constructor for LedgerIdRequirement

changelog_begin
changelog_end

* Make option based consructor the default, deprecate old constructor

* Update with review comments
2020-06-12 15:00:32 +00:00
Stephen Compall
15350a7bc2
disable warts.Any and remove most suppressions (#6132)
* disable Any wart

* first pass removal of Any suppressions for false positives

* second pass removal of Any suppressions for false positives

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* third pass removal of Any suppressions for false positives

* fourth pass removal of Any suppressions for false positives

* reformat newly single-suppressions into single lines

- suggested by @SamirTalwar-DA; thanks
2020-05-28 16:53:41 +00:00
Stephen Compall
265cfe6cdf
split http-json tests into tests and integration-tests targets (#6135)
* move integration-type tests in http-json to separate integration-tests target

* remove unused library deps from split tests

* remove resources from unit tests, set to medium size

* move resources to it subtree

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-05-28 11:46:25 -04:00
Stephen Compall
78205afa0d
use 2.13-style collections library syntax in json-api (#6075)
* register scala-collection-compat with java deps list

* add scala-collection-compat to http-json deps

* remove breakOut throughout http-json, replaced with view/to or iterator/to

* use scala 2.13-style `to` calls in http-json

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* use 2.13-style to in lf-value-json

* some fused size comparisons

* remove low-hanging breakOuts in daml-lf

* regenerate maven_install.json for scala-collection-compat

* regenerate maven_install.json for scala-collection-compat

* regenerate maven_install.json for scala-collection-compat
2020-05-27 22:27:25 +00:00
Stephen Compall
1be6b3640f
equalz Scalatest matcher in new daml-lf/scalatest-tools library (#1200)
* equalz Scalatest matcher in new daml-lf/scalatest-tools library

* equalz typing tests

* a 'should' replacing design

* a 'MatcherFactory1' design

- this fails because the TC parameter should be a type member to avoid
  scala/bug#5075 but it is not

* MatcherFactory1 with chained Lub+Equal typeclass

- requires partial-unification at point of use, which is not great

* LubEqual's extra tparam is probably unneeded

* better LtEqual

* demonstrate that HK LubEqual's resolve with DMT should + MatcherFactory

* remove unneeded 3rd param from LubEqual, again

* update dependency specs and license headers

* allow use with should, shouldNot in some cases, preserving the shouldx/shouldNotx alternatives

* move Equalz to libs-scala/scalatest-utils

* rename bzl targets and place in com.daml.scalatest package

* add scalatest-utils to release

* move *SpecCheckLaws, Unnatural to scalatest-utils

* missed scalacheck dep in scalatest-utils

* downstreams of *SpecCheckLaws now get them from scalatest-utils

* test equal-types case as well

* update LF documentation

CHANGELOG_BEGIN
CHANGELOG_END

* whitespace error
2020-05-27 17:25:43 -04:00
Stefano Baghino
9af85e56e9
Introduce DAML-LF value caching for transaction service (#6052)
* Introduce DAML-LF value caching for transaction service

Allows to keep the DAML-LF values in the most recently indexed events in memory,
so that they don't have to be deserialized on the client from their Protobuf encoding.

Closes #6044

CHANGELOG_BEGIN
[Sandbox] The --max-lf-value-translation-cache-entries option allows to set a
number of events for which DAML-LF values are cached. Could reduce latency in
serving transactions for consumers that are reasonably fast.
CHANGELOG_END

* Add missing dependency

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076003

* Update ledger/sandbox/src/main/scala/com/digitalasset/platform/sandboxnext/Runner.scala

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428071324

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428076905

* Address https://github.com/digital-asset/daml/pull/6052#discussion_r428081294

* Fix fatal warnings

* //ledger/caching has to be used whenever sandbox is run

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-05-26 08:33:53 +00:00
Remy
9e456a1016
DAML-LF: Kill RelativeContractId (#5991)
CHANGELOG_BEGIN
CHANGELOG_END
2020-05-25 22:30:45 +02:00
Stephen Compall
d7cc5e07c3
fix ACS race condition in sandbox-classic; enable random Websocket testing (#5969)
* Revert "disable test pending tracking down duplicates"

This reverts commit 8b5f9dfa04.

* activeContracts tells caller the ledgerEnd, rather than accepting activeAt argument

* naive port of http-json tests to sandbox-next

* Revert "naive port of http-json tests to sandbox-next"

This reverts commit 91d4590c90.

* lock before grabbing acs and ledgerEnd offset in InMemoryLedger

- as suggested by @gerolf-da; thanks
2020-05-19 13:26:37 -04:00
Stephen Compall
af52ef3ce9
random IOU create and split testing for websocket streams (#5879)
* Consume language for listing [re]actions in a foldAsync linearly

* eagerly consume as many Futures as possible

* restate one foldAsync test with Consume

* report value that failed to match

* add error locations and new `drain` primitive

* use `drain` to properly handle heartbeats at end of stream

* it doesn't make sense to delay the drain step

* rewrite other websocket test; document syntax

* get rid of StreamState, finally

* split Consume to a separate file

* plan split tests

* functions for consuming SplitSeq

* converting SplitSeq into stream actions

* exercise and check recursively

* add one 100-sized sample to the tests

* don't violate Iou's invariant

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* disable test pending tracking down duplicates

- as discussed with @leo-da
2020-05-13 14:06:11 -04:00
Samir Talwar
208d4a50da
postgresql-testing: Use a new, random database name every time. (#5911)
* postgresql-testing: Store the JDBC URL separately.

* postgresql-testing: Expose the username and password.

* postgresql-testing: Get the caller to create the database.

And make sure it's a random one, not "test".

CHANGELOG_BEGIN
CHANGELOG_END

* postgresql-testing: Only store the JDBC URL for tests.

Less mutable state, innit.

* postgresql-testing: Capture the individual JDBC URL parameters.

* Bazel: Fix PostgreSQL binary paths.

* postgresql-testing: Just recreate the database in PostgresAroundEach.

There's no need to restart the process with a different data directory.
2020-05-08 12:36:42 +00:00
Leonid Shlyapnikov
a79377cb2d
Improve DAML LF Party, PackageName, PackageId fromString error messages (#5855)
* Add description to daml.lf.data.StringModule implementations

Make sure DAML LF Party deserialization picks up the added description.

changelog_begin
changelog_end

* addressing code review comments. @S11001001, thanks.
2020-05-06 10:02:49 -04:00
Stephen Compall
59d5446438
disable https check for JWT tokens in JSON API, warn instead (#5856)
CHANGELOG_BEGIN
- [JSON API] The check that connections are made through a reverse-proxy
  providing HTTPS, ensuring that JWT tokens don't leak, only logs a warning
  rather than rejecting the request.
  See `issue #5856 <https://github.com/digital-asset/daml/pull/5856>`_.
CHANGELOG_END
2020-05-05 19:25:24 -04:00
Stephen Compall
2b23b41391
JSON decoding support for GenMap (#5223)
* add GenMap to the "all types" test generators

* report bad GenMap format with DeserializationError, not MatchError

* document GenMap JSON

* notes on missing features

* enable -Xsource:2.13 in transaction

* make an Order instance for Value resolvable, but unimplemented

* use the skeleton from SValue ordering to make a Value ordering skeleton

* add Party Order

* add Order instance for SortedLookupList

* add Order for FrontStack, deriving everything

* factor the Order lookup, and tie a knot in the recursive Value instances

* we're going to need this Iterator thing again

* replacing Order#contramap with version that supports equalIsNatural

* use new equalBy, orderBy for FrontStack, SortedLookupList, ImmArray

* _2 comparator, upgrade Name Equal to an Order

* incorporate lookup for enums, variants into Value order; record/struct cases

* Enum/Variant comparison

* looking up the singleton implicitly won't work for non-`object`s, alas

* test Order laws for values of all primitive types

* test Order laws for record and variant types

* test Order laws for enum types

* test that enum strings are not compared

* use checkLaws for Value Equal as well

* test that enums match order to constructor rank

* factor genAddend and genAddendNoListMap

* reintroduce Order for TypedValueGenerators

* more addend order

* record, variant order cases

* record cons order

* deriving Order while decoding from JSON

* make ApiCodecCompressed's Cid codec based on the typeclass

* test how the Value ordering and the underlying projected value orderings line up

- hint: they don't, yet
- this is also a template for how we'll check the fidelity with SValue
  ordering

* test how the Value ordering and SValue ordering line up

- hint: they don't, yet

* typed Arbitrarys have access to Order

* produce proper ValueGenMap

* inj requires Order, sometimes

- we encode this as "all the time" but there is a type-level unification
  approach to remove this requirement in some cases

* make inj a function

* test that order doesn't matter for JSON decoder

* use Utf8 order for TVG text; don't pretend that base equal works

* sort JSON GenMaps, and check for duplicates

* make injarb use IntroCtx

* remove stray import

* Order instances for Bytes, Hash, AbsoluteContractId

* require Order[Cid] to decode JSON to LF values

* clean up map reordering test

* remove unused Instant instance

* fake Order instance no longer needed, valid instance defined

* test parity of global AbsoluteContractId order and SContractId order

* bazel fmt

* test AbsoluteContractId Order lawfulness

* test duplicate key detection

CHANGELOG_BEGIN
- [JSON API] Prepare full support for the planned GenMap primitive type.
  See `issue #5031 <https://github.com/digital-asset/daml/issues/5031>`_.
CHANGELOG_END
2020-05-05 10:27:37 -04:00
Oliver Seeliger
4c0ffdf4df
Upgrade scala compiler silencer to 1.6.0 (#5822)
* Upgrade scala compiler silencer to 1.6.0

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt build bazel file to new targets

* Switch to silencer plugin scala 2.12.11 per Samir's feedback

rather than 2.12.8

* Add missed bazel files

* Review feedback from Leo
2020-05-04 21:30:46 +00:00
Stephen Compall
067f3c987d
require https reverse proxy (#5660)
* new --leak-passwords-firesheep-style option; functions to check forwarded protocol

* enforce https reverse-proxy in all JWT-accepting endpoints

* make HttpService.start take config record

* test that X-Forwarded-Proto or Forwarded is enforced

* use new start signature in daml-script tests

* use insecure http mode for ts codegen tests

* note on regex

* use insecure option in daml assistant integration tests

* log allowNonHttps setting

* add non-https option to more places in daml-assistant tests

* add non-https option to getting started guide

* rename --leak-passwords-firesheep-style to --allow-insecure-tokens

- per suggestion by @garyverhaegen-da, @hurryabit

CHANGELOG_BEGIN
- [JSON API] By default, checks that connections are made through a reverse-proxy
  providing HTTPS, ensuring that JWT tokens don't leak.  To disable this check,
  such as for development, pass ``--allow-insecure-tokens``.
  See `issue #5572 <https://github.com/digital-asset/daml/issues/5572>`_.
CHANGELOG_END

* daml start includes --allow-insecure-tokens by default

- as indicated by @cocreature
2020-04-28 14:33:38 -04:00
Leonid Shlyapnikov
102f6b3888
Use a custom Gen instead of Arbitrary, to generate nonempty Vec[Int], (#5763)
* Use a custom Gen instead of Arbitrary, to generate nonempty Vec[Int],

the test could fail on empty input, it expects at least one input message

changelog_begin
changelog_end

* Address code review comments
2020-04-28 16:25:09 +00:00
Stephen Compall
d4086080ea
proper GenMap Gen with key Value order (#5526)
* add Order instances to various types in LF data, transaction

* define scope-dependent Order for Value; test compatibility with SValue order

* support Order in TypedValueGenerators

* generate properly-ordered ValueGenMaps in TypedValueGenerators

* factor genAddend and genAddendNoListMap

* test Value.orderInstance fidelity with well-typed Orders

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* shrink Maps for better debugging

* line up base List and GenMap orders with svalue

* unhandled cases in custom comparator for ApiValueToLfValueConverter tests

- hazards of not using match2

* adapt to PureCompiledPackages changes
2020-04-24 17:57:12 -04:00
Leonid Shlyapnikov
bab018b44e
Allow tokens with zero parties for pass-through endpoints (#5674)
changelog_begin
[JSON API]
Pass through endpoints do not require party specified in the JWT:
- ``/v1/parties``
- ``/v1/parties/allocate``
- ``/v1/packages``
See #5313.
changelog_end
2020-04-22 12:39:20 -04:00
Leonid Shlyapnikov
6aebbaca65
JSON API endpoints for DAR file upload, DALF download (#5613)
* DAR file upload, DALF download, list all DALF package IDs

Reload packages after dar upload

changelog_begin
[JSON API]
Add package management pass-through endpoints. See #4823.
- GET /v1/packages -- returns all package IDs
- GET /v1/packages/<package ID> -- downloads the DALF package
- POST /v1/packages -- uploads a DAR file to the ledger
changelog_end

* Fixing docs

* minor improvements + cleanup
2020-04-21 19:22:50 -04:00
Stephen Compall
91c43f147c
allow secure connection from JSON API to ledger (#5555)
* factor TlsConfiguration parser from extractor

* move TlsConfigurationParser to new library

* link extractor to ledger-service/cli-opts properly

* use TlsConfigurationCli in http-json, pass SslContext to ledger-client

* test TLS options as used in http-json

- the TLS config code is shared with extractor, where it is more fully
  tested; we just do a sanity check here

* doc TLS options for http-json

CHANGELOG_BEGIN
- [JSON API] New ``--pem``, ``--crt``, ``--cacrt``, and ``--tls`` options
  for securing the connection between JSON API server and ledger.
  See `issue #2540 <https://github.com/digital-asset/daml/issues/2540>`__.
CHANGELOG_END

* TLS off in daml-script JSON API test
2020-04-16 13:12:30 -04:00
Leonid Shlyapnikov
89f5ebcde8
Offset ticking fix (#5532)
* Fixing offset tick

changelog_begin
changelog_end
2020-04-10 15:05:19 -04:00
Leonid Shlyapnikov
29e1931f17
Add --port-file command line option to JSON API (#5454)
* Adding `--port-file` support

* ``--port-file`` support

* Updating docs

changelog_begin

[JSON API] Add support for ``--port-file`` command line option.
``--http-port 0 --port-file ./json-api.port`` will pick up a free port
and write it into ``./json-api.port` file.

changelog_end

* reformatting

* Usage grammar

* use bimap

* Adding `PortFiles` utility for creating and deleting port files on JVM exit

* Adding scaladoc explaining that the port file should be deleted on

JVM termination.

* Updating usage and docs to reflect that the file must be unique and

will be deleted on graceful shutdown

* Relying on `java.nio.file.FileAlreadyExistsException` to determine the

case when failed due to the nonunique file name.

* toString instead of Exception.getMessage

java.nio exception's getMessage can be just a file name, need the class
name to capture the error context.

* updatePortFile -> createPortFile

* write to file instead of write into file
2020-04-08 18:48:11 +00:00
Gerolf Seitz
97433743a1
Set the Bearer prefix in bindings. (#5484)
* Set the `Bearer ` prefix in bindings.
* Make the `Bearer ` prefix in the authorization header mandatory.
* Bearer prefix can be removed from the token file.

CHANGELOG_BEGIN
[Extractor]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[Navigator]: The ``Bearer `` prefix can be removed from the token file.
It is added automatically.
[DAML Script] The ``Bearer `` prefix can be removed from the token file. It
is added automatically.
[DAML Repl] The ``Bearer `` prefix can be removed from the token file. It is
added automatically.
[Scala Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[Java Bindings] The ``Bearer `` prefix can be removed from the token. It is
added automatically.
[DAML Integration Kit] ``AuthService`` implementations MUST read the
``Authorization`` header and the value of the header MUST start with
``Bearer ``.
CHANGELOG_END
2020-04-08 13:07:28 +02:00
Leonid Shlyapnikov
003065fb70
Empty array of party IDs -- error, nothing found -- OK (#5456)
* Empty array of party IDs -- error, nothing found -- OK, as per @hurryabit

changelog_begin
changelog_end

* fixing test description
2020-04-07 13:57:06 +00:00
Leonid Shlyapnikov
644740f3c3
JSON API: Fixing ledger offset ordering (#5458)
changelog_begin
changelog_end
2020-04-07 09:11:41 +02:00
Gerolf Seitz
a2d785e3ee
Use com.daml as root package (#5343)
Packages com.digitalasset.daml and com.daml have been unified under com.daml

Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.

CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
2020-04-05 19:49:57 +02:00
Leonid Shlyapnikov
f71facf7cf
Parties error handling improvements (#5395)
* Return BadRequest(400) with warnings when all party IDs are unknown

changelog_begin
changelog_end

* cleanup
2020-04-03 09:54:43 -04:00
Leonid Shlyapnikov
62b7ab590a
Sync query error reporting improvements (#5326)
* non empty set of template IDs in domain.GetActiveContractsRequest

* Updating expected error message

changelog_begin
changelog_end

* Adding warnings field to domain.ErrorResponse

* Error messages constants

* using domain.AsyncResponse

* added todo

* Updating examples

changelog_begin
changelog_end

* Updating examples

* Fixing the case when head is part of the tail, as pointed by @S11001001

* Fixing the domain.SynResponse JSON reader,

report deserialization error when both 'errors' and 'result' fields
present, as pointed by @S11001001

* Addressing code review comments
2020-04-02 19:47:07 +00:00
Gerolf Seitz
329320bad9
Organize maven coordinates (#5272)
* Use com.daml as groupId for all artifacts

CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END

* Add 2 additional maven related checks to the release binary

1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId

* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
2020-04-01 11:41:18 +02:00
Leonid Shlyapnikov
2b701ea26c
Removing unused domain.JwtPayload JSON formatters, (#5325)
we have been relying on `ledger.api.auth.AuthServiceJWTCodec` since the
refactoring happened

changelog_begin
changelog_end
2020-03-31 21:52:24 +00:00
Leonid Shlyapnikov
6b1c90267a
Report UnknownTemplateIds warning before sending an error (#5318)
* report UnknownTemplateIds warning before sending an error

if no template IDs resolved, send warning before the error, this is for
consistency with the case when at least one ID resolved (we continue
processing request in this case).

changelog_begin
changelog_end

* adding an error message example

* change title

* clarifying what on-message means.

* formatting

* Addressing code review comments
2020-03-31 14:36:48 -04:00
Samir Talwar
52dbcf5d95
postgresql-testing: "Lock" ports when starting PostgreSQL. (#5310)
* http-json: Ask for a free port by specifying port 0.

This will avoid race conditions.

* bindings-akka-testing: Delete RandomPorts; it's unused.

* ports: Fix the Bazel test glob.

* ports: Move FreePort to postgresql-testing and add a test case.

* postgresql-testing: Make `FreePort.find()` return a `Port`.

* postgresql-testing: Lock free ports until the server starts.

This uses a `FileLock`, which should work well on all our
supported operating systems as long as everyone agrees to use it.

CHANGELOG_BEGIN
CHANGELOG_END

* postgresql-testing: Try to find a free port 10 times, then give up.

* postgresql-testing: Use a shared directory for the port lock.

* postgresql-testing: Try an alternative way of getting `%LOCALAPPDATA%`.
2020-03-31 17:13:02 +00:00
Leonid Shlyapnikov
703b588384
Error and disconnect when multiple requests coming over the same WebSocket connection (#5288)
* minor refactoring in preparation for the next test case

* allow only one input request + test, WIP

* allow only one input request + test, WIP

* Error when multiple requests coming over the same websocket connection

This never worked, we never returned anything when 2nd request received,
now we return an error and drop the connection.

changelog_begin
changelog_end

* update docs

* Fixing copyright header

* changing test description

* trying to clean wsMessageHandler up

* Reverting getTransactionSourceForParty changes
2020-03-30 18:08:54 -04:00
Remy
1b37f6c482
DAML-LF: redesign absolute contract ids (#5207)
CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 19:07:42 +01:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Stephen Compall
332d35a347
stop filtering "phantom archives" when it won't work; reenable it on request (#5003)
* ContractKeyStreamRequest with different types for whether offset-preceded or not

- should replace EnrichedContractKey as the WS StreamQuery type

* add the ContractKeyStreamRequest layer everywhere

* split StreamQuery parse from the other steps

* make StreamQuery type depend on the input JsValue

* only StreamQueryReader is a typeclass now

* scalafmt

* wrong type arg

* letting the request data and phantom archive removal choice flow

- from work with @leo-da

* finish threading request-derived phantom state to removal flow

- from work with @leo-da

* make it clear that hints are contract IDs in StreamQuery

- from work with @leo-da

* treat StreamQueryReader's type parameter fully as a phantom

- from work with @leo-da

* remove unused type alias

- from work with @leo-da

* test fetch resume, with and without various offsetHints

* document offsetHints

* missing ` in rst

- thanks @hurryabit

* rename offsetHint to contractIdAtOffset

CHANGELOG_BEGIN
- [JSON API - Experimental] New field ``contractIdAtOffset`` for fetch-by-key streams
  to restore proper archive filtering.
  See `issue #4511 <https://github.com/digital-asset/daml/issues/4511>`_.
CHANGELOG_END

* we never unify the two ContractKeyStreamRequest types

* doc update for contractIdAtOffset
2020-03-26 15:31:27 -04:00
Leonid Shlyapnikov
24e414cba0
Remove ledgerEffectiveTime, maximumRecordTime and default TTL config. (#5200)
changelog_begin

[JSON API - Experimental]
Remove ``ledgerEffectiveTime`` and ``maximumRecordTime`` fields from command ``meta``. Remove ``--default-ttl`` command line argument. The ledger time and TTL are automatically computed by the submission service now. See #5090.

changelog_end
2020-03-25 17:06:12 -04:00
Robert Autenrieth
a3ddde3a9d
Remove MRT and LET (#5172)
Contributes to #4194.
Closes #4231.
Closes #5022.
CHANGELOG_BEGIN
- [Ledger API] The protobuf fields ledger_effective_time and maximum_record_time have been removed from
  command submission. These fields were previously deprecated following the introduction
  of a new ledger time model. See issue `#4194 <https://github.com/digital-asset/daml/issues/4194>`__.
[Java Bindings] removed the usage of ledgerEffectiveTime and
maximumRecordTime, and instead added minLedgerTimeAbsolute and
minLedgerTimeRelative in CommandSubmissionClient and CommandClient
CHANGELOG_END
2020-03-25 16:50:27 +01:00
Stephen Compall
8d2c13cf43
generating GenMaps for testing (#5115)
* building a GenMap generator

* move Party Order to accessible place; test whether Name Equal is still in use

* add Order instance for SortedLookupList

* switch to Map for genMap's Inj; more Order instances

* remove all Order from TypedValueGenerators

* Revert "add Order instance for SortedLookupList"

This reverts commit 03a59a8249.

* moving the Equal instance means scenario-interpreter no longer direct-deps scalaz

* add a test using TypedValueGenerators GenMap

* remove Party Order

* refmt bazel

* remove stray import

* followup dep change to moving the Equal[Name] instance

* add changelog

CHANGELOG_BEGIN
CHANGELOG_END

* Name equal instance appears to be no longer used, but keeping anyway
2020-03-25 10:28:06 -04:00
Leonid Shlyapnikov
a6a1e75bc1
Streaming API error handling/reporting improvements (#5141)
* Streaming API error handling

* Stopping the stream after emitting the 1st error

* Cleaning up test cases, stricter type for non empty streaming fetch

* Update docs.

changelog_begin

[JSON API - Experimental] Change Streaming API error format to match synchronous API:
``{"status": <400 | 401 | 404 | 500>, "errors": <JSON array of strings> }``. See #4521

changelog_end

* a few words about warnings in the Streaming API

* a few words about warnings in the Streaming API

* Updating command formatting in the JavaScript example.
2020-03-24 19:32:55 +00:00
Stephen Compall
f2b068f0b5
remove List and TextMap support from the JSON query language (#5099)
* remove List and TextMap support from the JSON query language

CHANGELOG_BEGIN
- [JSON API - Experimental] Removed support for maps and lists from the query language.
  See `issue #4855 <https://github.com/digital-asset/daml/issues/4855>`_.
CHANGELOG_END

* GenMap won't be supported in general, remove FIXME

- thanks @leo-da

* more documentation on unsupported query types

- as suggested by @hurryabit; thanks
2020-03-20 16:22:22 -04:00
Leonid Shlyapnikov
ffabc2b3e5
Bumping up http-json tests size to large, adding some extra logging (#5109)
* Bumping up http-json tests size to large, adding some extra logging

* Removing the logging from prod code, adding some in tests.

changelog_begin
changelog_end
2020-03-20 16:48:11 +00:00
nickchapman-da
d81caa9568
rework daml_compile bazel rule (#5070)
Avoiding `damlc compile/package` commands (which we would like to deprecate), and replace with plain `damlc build` together with a post dar->dalf extraction step in the couple of places where we actually want the .dalf for testing.

changelog_begin
changelog_end
2020-03-18 23:06:55 +00:00
Leonid Shlyapnikov
29c340fed0
2nd part of stabilizing flaky websocket integration test (#5079)
* 2nd part of stabilizing flaky websocket integration test,

handling empty events, which we may receive a bunch in the end of the test
when there is no activity

changelog_begin
changelog_end

* fixing `Offset.semigroup`, thanks @S11001001!
2020-03-18 17:04:01 -04:00
Leonid Shlyapnikov
acb41c5829
Fetch by parties endpoint to use new getParties gRPC call (#5027)
* fetch by parties endpoint to use new getParties gRPC call

changelog_begin
changelog_end

* addressing code review comments, thanks @S11001001 and @SamirTalwar-DA

* Update ledger/ledger-api-client/src/main/scala/com/digitalasset/ledger/client/services/admin/PartyManagementClient.scala

thanks @S11001001

Co-Authored-By: Stephen Compall <stephen.compall@daml.com>

* addressing code review comments, thanks @S11001001 and @SamirTalwar-DA

* addressing code review comments, thanks @S11001001 and @SamirTalwar-DA

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-03-16 16:42:01 -04:00
Leonid Shlyapnikov
5c2ed56c8e
Allocate party JSON API endpoint (#4991)
* Allocate party JSON API endpoint

changelog_begin

[JSON API - Experimental]
Allocate party endpoint added: ``/v1/parties/allocate``. See #4638.

changelog_end

* cleanup

* Update ledger-service/http-json/src/main/scala/com/digitalasset/http/LedgerClientJwt.scala

Co-Authored-By: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-03-13 14:06:08 -04:00