Commit Graph

66 Commits

Author SHA1 Message Date
Samir Talwar
6035840d8a
Upgrade Scala 2.12 to v2.12.13. (#9280)
* Upgrade Scala 2.12 to v2.12.13.

This is being pulled in anyway because of Maven/Gradle/etc's fun
"favor the most recent" resolution mechanism. The version of Akka we
depend upon transitively depends on Scala 2.12.13, so any downstream
consumers will see that as the Scala version required.

Bringing the Daml repo in line means no more confusion.

CHANGELOG_BEGIN
- [Scala Bindings] If you are using Daml on Scala 2.12, it now depends
  on Scala v2.12.13 (from v2.12.12).
CHANGELOG_END

* Scala 2.12.13 is the default version in our pinned version of nixpkgs.

* Upgrade Scala 2.13's Wartremover version.

* Rename `scala_version_rule` to `scala_version_configure`.

* Add a test case to ensure the Scala versions are the same everywhere.

* Add tests for the Scala JAR versions in maven_install_*.json

* gatling-utils: Change the sort order of the expected CSV in tests.

I don't know why this changed, but it seems to be stable.

* compatibility: `scala_version` -> `scala_version_configure`.

* Bazel: Disable the Scala version tests on Windows.

* compatibility: Upgrade Wartremover to Scala 2.12.13.
2021-03-30 11:02:18 +00:00
Moritz Kiefer
0f1d2a9c47
Upgrade akka to 2.6.13 (#9026)
Seems to have a few useful bugfixes
https://akka.io/blog/news/2021/02/23/akka-2.6.13-released. Unclear if
we’re affected by those bugs but I’d rather not worry about whether we are.

changelog_begin
changelog_end
2021-03-05 10:31:37 +01:00
Luc Bourlier
09c46cf666
Upgrades opentelemetry to 0.16.0 (#8862)
CHANGELOG_BEGIN

- [Dependencies] Upgrade io.opentelemetry:opentelemetry-api to 0.16.0

- [Dependencies] New dependency: io.opentelemetry:opentelemetry-context:0.16.0

CHANGELOG_END
2021-02-23 18:21:36 +01:00
Kamil Bożek
9b70f4c106
Replace ai.x.diff library with com.softwaremill.diffx [DPP-192] (#8821)
* Replaced diff with diffx

* Explicitly added magnolia and mercator dependencies to fix automatical type class derivation

* CHANGELOG_BEGIN
CHANGELOG_END

* Removed unnecessary Diff type class instance for Seq[T]

* Removed ai.x.diff leftovers

* Added an explanatory comment for magnolia and mercator dependencies

* Formatted changes

* Added optional scaladoc parameter to Bazel's da_scala_library_suite()

* Formatted changes
2021-02-17 10:56:23 +01:00
Stephen Compall
b94b5f92f3
experimental Oracle support in json-api (#8596)
* separate OracleQueries from PostgresQueries

- with some changes from 8161e63189 courtesy @cocreature

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* abstract BIGINT

* json, signatories, observers columns

* compatible lastOffset

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* oracle functions for select (single template ID), insert

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* add oracle branch to integration tests

* oracle CLI configuration for json-api

* run integration tests with ojdbc in classpath

* update maven_install for ojdbc

* drop table if exists for Oracle

* make create DDLs and drops more planned out; drop in reverse order for Oracle integrity

* repin maven

* port agreement_text

* port (by removal) array part of ledger offset update

* use CASE instead of JSON map lookup for multiparty offset update

* simplify self types

* fix contract archival

* repin

* remove selectContracts in favor of selectContractsMultiTemplate

* move Oracle test execution to separate build target

* move websocket test to itlib

* make a bad array instance for Oracle

* report actually-available JDBC drivers only

* configure Oracle test from CI

* attempt with platforms and constraints

* a mismash of bazel to get it to conditionally enable oracle testing

* fix dep resolution in Scala 2.13

* make the Oracle test a stub (inits and does empty DB query)

* remove commented unused deps

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* repin

* we never supply a value for the surrogate ID columns

- suggested by @cocreature; thanks

* add not null to json in DB-specific place

- suggested by @cocreature; thanks

* why DBContractKey

- suggested by @cocreature; thanks

* textType isn't finalized

- suggested by @cocreature; thanks

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-02-17 03:50:35 -05:00
Moritz Kiefer
7ae28beff9
More Scala 2.13 cleanup (#8855)
This fixes Scaladoc and our pom file generation.

It also clears up the confusing error around gatling and removes a
redundant dependency on sbt (no idea why we had that in the first
place) both of which resulted in Scala 2.12 dependencies in our 2.13
lockfile which is obviously bad.

With this, we should now be ready to publish Scala 2.13 artifacts once
the ledger API test tool PR lands.

changelog_begin
changelog_end
2021-02-16 09:39:16 +00:00
Stefano Baghino
491a1b29f4
Non-repudiation: add persistent, SQL-backed signed commands repo (#8723)
* Non-repudiation: add persistent, SQL-backed signed commands repo

Closes https://github.com/digital-asset/daml/issues/8634

Contributes to https://github.com/digital-asset/daml/issues/8658

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/8723#discussion_r569228934

* Fix issues after rebase, address https://github.com/digital-asset/daml/pull/8723#discussion_r569227969

* Fix certificate table and column names, add timestamp to persistent store, address https://github.com/digital-asset/daml/pull/8723#discussion_r569223780

* Fix outstanding compilation errors

* Fix tests after rebase

* Address comments from https://github.com/digital-asset/daml/pull/8723#pullrequestreview-582106893

- Address https://github.com/digital-asset/daml/pull/8723#discussion_r569216711
- Address https://github.com/digital-asset/daml/pull/8723#discussion_r569217401

* Address https://github.com/digital-asset/daml/pull/8723#discussion_r569215383
2021-02-04 14:15:36 +00:00
Moritz Kiefer
f261e0489a
Bump akka to 2.6.12 to fix oom issue (#8698)
This includes a fix for https://github.com/akka/akka/issues/29966
which we hit in some of our tests.

changelog_begin
changelog_end
2021-01-30 07:47:10 +01:00
Moritz Kiefer
6326822907
Upgrade grpc and protobuf libraries (#8558)
* Upgrade grpc and protobuf libraries

changelog_begin

- [Dependencies] Upgrade
  iou.grpc:grpc-(api|core|netty|protobuf|services|stub) to
  1.35.0.

- [Dependencies] Upgrade
  io.netty:netty-(codec-http2|handler|handler-proxy|resolver)
  to 4.1.58.Final.

- [Dependencies] Upgrade
  io.netty:netty-tcnative-boringssl-static to 2.0.36.Final.

- [Dependencies] Upgrade com.google.protobuf:protobuf-java to 3.14.0.

changelog_end

* Factor out netty and grpc version

changelog_begin
changelog_end
2021-01-26 14:17:18 +00:00
Moritz Kiefer
a5bd99a368
Upgrade akka libraries (#8585)
changelog_begin
changelog_end
2021-01-20 19:49:10 +00:00
Moritz Kiefer
290584a378
Port //ledger/ledger-api-client/... to Scala 2.13 (#8478)
* Port //ledger/ledger-api-client/... to Scala 2.13

This pulls in Sandbox next and kvutils as a dependency so those now
build on 2.13 as well.

changelog_begin
changelog_end

* Upgrade scala-colllection-compat

changelog_begin
changelog_end

* Use toVector.sortBy instead of to(LazyList).sortBy

changelog_begin
changelog_end

* Use a view for passing things to varargs

changelog_begin
changelog_end

* avoid symbol literal in CommandClientIT

changelog_begin
changelog_end
2021-01-13 11:35:46 +01:00
Moritz Kiefer
6b2ac0e0c6
Port parts of //ledger/... to Scala 2.13 (#8470)
* Port parts of //ledger/... to Scala 2.13

Fairly random choice of directories, I just went through them in
alphabetical order. The one thing that I had to disable for now are
the conformance tests since the ledger API test tool has a dependency
not compatible with Scala 2.13.

changelog_begin
changelog_end

* Remove accidentally included //ledger/ledger-api-client/...

doesn’t actually work yet

changelog_begin
changelog_end
2021-01-12 10:55:00 +01:00
Moritz Kiefer
aebde84842
Upgrade scopt to 4.0.0 (#8466)
* Upgrade scopt to 4.0.0

Scopt 3.x has some issues with Scala 2.13 because it expects an
immutable Seq on 2.13 meaning you cannot just pass in an Array. Rather
than fixing our callsites to convert to an immutable Seq everywhere,
this PR bumps to Scopt 4.0 which goes back to collection.Seq.

and leaving that aside, I’m a fan of upgrading dependencies anyway :)

changelog_begin
changelog_end

* Use val instead of def

changelog_begin
changelog_end
2021-01-11 21:23:02 +01:00
Andreas Herrmann
7855b023ac
Oauth template (#8453)
* Use ActorSystem and ExecutionContext from RequestContext

* Factor out middleware server class

To avoid passing around config and state manually.

changelog_begin
changelog_end

* Depend on databricks/sjsonnet

changelog_begin
changelog_end

* Generate request params from jsonnet template

changelog_begin
changelog_end

* Split middleware test suite sources

* Add test suite for request templates

* fmt

* TriggerServiceFixture template arguments

* Use null to indicate missing applicationId claim

Addressing
https://github.com/digital-asset/daml/pull/8453/files#r555025173

* Fix invalid path on Windows

* Close request template source

* Avoid repeated re-reading of Jsonnet files

https://github.com/digital-asset/daml/pull/8453/files#r555044262

* Factor out template argument mappings

* factor out template error handling

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2021-01-11 17:27:46 +00:00
Moritz Kiefer
df35aed5bb
Remove Navigator console (#8392)
* Remove Navigator console

This was a labs feature so we can remove it without a deprecation
cycle. It doesn’t have any known users and is almost completely
superseded by DAML REPL.

@bame-da agreed to the removal.

This also has the nice side effect of eliminating 1 of our 2
dependencies that were not Scala 2.13 compatible.

changelog_begin

- [Navigator Console] The labs feature Navigator Console has been
  removed. Users are encouraged to use DAML REPL instead.

changelog_end

* Kill Navigator database docs completely

changelog_begin
changelog_end
2021-01-05 13:20:48 +01:00
Gary Verhaegen
a925f0174c
update copyright notices for 2021 (#8257)
* update copyright notices for 2021

To be merged on 2021-01-01.

CHANGELOG_BEGIN
CHANGELOG_END

* patch-bazel-windows & da-ghc-lib
2021-01-01 19:49:51 +01:00
Moritz Kiefer
cb57ad374c
Port //daml-lf/data to Scala 2.13 (#8321)
* Port //daml-lf/data to Scala 2.13

changelog_begin
changelog_end

* factor common ImmArraySeq code to version-agnostic file

- ImmArraySeq itself is agnostic; the 2.12 and 2.13 versions contain
  implementation mixins/superclasses for parts that must be specific.  The 2.13
  version will collapse into the agnostic version when 2.12 support is no longer
  desired

* factor common InsertOrdMap code to version-agnostic file

- InsertOrdMap itself is agnostic; the 2.12 and 2.13 versions contain
  implementation mixins/superclasses for parts that must be specific.  The 2.13
  version will collapse into the agnostic version when 2.12 support is no longer
  desired

* factor common InsertOrdSet code to version-agnostic file

- InsertOrdSet itself is agnostic; the 2.12 and 2.13 versions contain
  implementation mixins/superclasses for parts that must be specific.  The 2.13
  version will collapse into the agnostic version when 2.12 support is no longer
  desired

* factor Map removal

* Move ImmArraySeq back into ImmArray

changelog_begin
changelog_end

* Type assertion instead of symbol

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-12-18 13:20:15 +01:00
Moritz Kiefer
d24ef94f36
Add a Scala 2.13 build pipeline (#8271)
* Add a Scala 2.13 build pipeline

This adds initial support for multiple Scala versions controlled via
the DAML_SCALA_VERSION env var and a CI job to make sure we don’t
regress. For now we only test //libs-scala/ports/... which seemed like
the easiest starting point I could find. We can incrementally expand
that over time.

changelog_begin
changelog_end

* Document pinning

changelog_begin
changelog_end

* Address review comments

changelog_begin
changelog_end
2020-12-14 12:42:26 +01:00
Moritz Kiefer
76b6fd86fb
Upgrade Scala dependencies for 2.13 compatibility (#8268)
* Upgrade Scala dependencies for 2.13 compatibility

This upgrades a bunch of Scala libraries to versions that have 2.13
support. There are two libraries that are still missing:

- diffson, this has a new version but with significant breaking
  changes and it is only used in Naigator console which I hope to kill
  before I have to worry about this.

- ai.x:diff, this is used in the ledger API test tool. The library is
  abondened but there are a few alternatives.

changelog_begin
changelog_end

* Fix pureconfig

changelog_begin
changelog_end

* Fix Navigator

changelog_begin
changelog_end
2020-12-14 09:59:00 +01:00
curiousleo-da
f4fa88c977
Trace index DB -> Ledger API read path [LPF-223] (#8233)
CHANGELOG_BEGIN
- [Integration Kit] Emit OpenTelemetry events for transactions streamed
  to the client from the Ledger API
CHANGELOG_END
2020-12-10 15:06:17 +01:00
Moritz Kiefer
7e05dc7932
Upgrade rules-scala and scalatest (#8187)
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.

This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.

Apologies for the giant PR, I don’t see a way to keep it smaller.

changelog_begin
changelog_end
2020-12-08 06:59:23 +01:00
Brian Healey
ca294eb14d
add blackduck scan to run on master (#6130) (#8161)
* add blackduck scan to run on master (#6130)

* add blackduck scan

* disable go scanning
exclude entire language-support/ts directory for node scanning
break to multiple lines to make command line params easier to parse

* Increase timeout for blackduck binary scan

* update blackduck scan config

* remove some exclusions, force python3

* exclude GO until path to go executable can be resolved

* added readme explanation of why we want this file

* fail in case of policy violation

* ensure haskell bazel scan completes before running second round scan for bazel jvm and node and other langs

* trigger notices file gen to ensure BOM complete

* remove trailing end of lines

* run with latest detect version and unique code location name changes to wrapper script

* Add blackduck to daily compat job

* DO NOT MERGE: condition false to disable other jobs for testing

* remove parameters not available to cronjob

* Revert changes to regular CI pipeline

CHANGELOG_BEGIN
CHANGELOG_END

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>

* Do not get branch name from variable

* Upgrade com.fasterxml.jackson.core:jackson-databind to 2.12.0 to address security vulnerability

* Remove disabling of other jobs, set to branch to be used on prod runs

* Apply suggestions from code review

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

* Address code review comments

* Updated NOTICES file

* Run bazel build, update NOTICES file

* Correct dade-assist

* do not have perms to pipe to dev/null

* Add md file explaining how to update NOTICES file

* Add instructions for running blackduck locally

* Add a link to full security-blackduck readme

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2020-12-07 19:59:39 +00:00
Stefano Baghino
bc0f6ab35e
[DPP-84] Employ parallel unnesting instead of batching on PostgreSQL (#8042)
* [DPP-84] Employ parallel unnesting instead of batching on PostgreSQL

changelog_begin
[Integration Kit] When using a PostgreSQL-based index, leveraging native
parallel unnesting allows to more efficiently index new transactions.
changelog_end

* Address review https://github.com/digital-asset/daml/pull/8042#pullrequestreview-541759596
2020-12-01 12:06:36 +00:00
Moritz Kiefer
2d3820ac14
Upgrade akka-http to 10.2 (#8058)
* Upgrade akka-http to 10.2

Follow up to #8048, I left out this upgrade to reduce noise and since
I wasn’t quite sure how involved it was going to be.

changelog_begin
changelog_end

* Reenable transparent HEAD requests

Apparently no longer on by default but we depend on this in waitForHttpServer

changelog_begin
changelog_end
2020-11-25 13:39:25 +01:00
Moritz Kiefer
d83cbdb475
Upgrade akka and akka-http (#8048)
* Upgrade akka and akka-http

Was chasing an issue somewhere and thought this might affect it in
some way. It didn’t but I might as well turn the upgrade into a PR.

changelog_begin
changelog_end

* Fix trigger service tests

changelog_begin
changelog_end

* Downgrade akka-http again

changelog_begin
changelog_end

* Upgrade akka-http again and fix tests

changelog_begin
changelog_end

* Cleanup trigger service

changelog_begin
changelog_end
2020-11-25 10:13:51 +01:00
Stephen Compall
4ca02e0eb6
upgrade doobie from 0.6.0 to 0.9.2 (#7618)
* set doobie version to 0.9.2 and rerun maven pin

* port extractor and some of JSON API

* repin maven

* use doobie's own builder compatibility where required

* use probably bad derivations to supply Blockers where transactEC was required

- The point of using Blocker instead of ExecutionContext seems to be to
  especially emphasize to API users that it isn't appropriate to use an
  ExecutionContext with ordinary behavior.  That is what we have done, which
  should probably change, but just compiling for now.

* fix fragment inspection test for internal restructuring

- This test depends on implementation details of Doobie, so naturally it must be
  altered when that runs.  Fortunately, it's been made easier by the changes
  in this upgrade.

* allow 256 blockers for navigator transaction blocker, like the global EC

* allow as many blockers as the pool size for trigger service

- The transactor shouldn't share ExecutionContext for transactions with the
  caller, so we set up a new one based on configured pool size.

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-16 09:46:20 -04:00
Stephen Compall
1a2afd5266
upgrade to Scala 2.12.12 from 2.12.11 (#7661)
* upgrade bazel settings to scala 2.12.12

* upgrade nix scala tool to scala 2.12.12

* upgrade silencer references to scala 2.12.12

* repin for scala 2.12, silencer, wartremover upgrades

* remove numerous occurrences of unused silencer now spotted

* update Scala version in our bazel notes

CHANGELOG_BEGIN
CHANGELOG_END

* update compatibility maven_install.json to match compatibility WORKSPACE
2020-10-13 08:42:14 -04:00
Brian Healey
b2e1dd85cd
Upgrade Jackson to 2.11.2 to address security vulnerabilities (#7375)
CHANGELOG_BEGIN
Upgrade Jackson to 2.11.2 to address security vulnerabilities
CHANGELOG_END

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
2020-09-10 14:32:01 -04:00
Leonid Shlyapnikov
ccecc46e70
JSON API Perf Scenario Runner (#7295)
* Run gatling scenario from the perf runner main

reports are disabled for now, getting a class not found
when generating them

changelog_begin
changelog_end
2020-09-02 09:00:43 -04:00
Samir Talwar
d2561896e1
Upgrade Guava from v24 to v29. (#7261)
This brings `ByteStreams` out of beta.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-28 12:55:04 +00:00
Leonid Shlyapnikov
9acb47d994
JSON API command submission perf test (#6906)
* JSON API Perf test scenarios

changelog_begin
changelog_end

* addressing code review, thanks @S11001001

* addressing code review, thanks @S11001001

* cleanup

* cleanup + instructions

* cleanup + instructions

* Updates

* Updates
2020-08-04 18:51:44 +00:00
Moritz Kiefer
bcb2a8ce65
Mark pureconfig-macros as neverlink (#6952)
pureconfig-macros pulls in the scala-compiler library which in turn
pulls in jansi-native which is somewhat problematic as it’s not
well supported outside of x86. It also just adds roughly 10% unused
junk to the http-json fat JAR. Afaict, pureconfig-macros is only used
at compile time so we can remove it.

changelog_begin
changelog_end
2020-08-04 10:01:58 +02:00
Stefano Baghino
0982a01ce4
🔥 (#6809)
changelog_begin
changelog_end
2020-07-21 18:32:58 +00:00
Stephen Compall
7622862312
upgrade to Wartremover 2.4.9 (#6551)
* upgrade to wartremover 2.4.9

* simplify wart list and list JavaConversions as disabled

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* delete long-obsolete, contradictory comment

* also upgrade wartremover in compatibility (leaving aside maven_install.json)

* update compatibility maven_install.json to match
2020-07-01 15:02:58 +00:00
Stefano Baghino
748d17bbe0
Bump Flyway version to 6.5 (#6520)
* Bump Flyway version to 6.5

Prevents incurring into https://github.com/flyway/flyway/issues/2759 (which was apparently solved in 6.4.0)

changelog_begin
changelog_end

* Comply with changed method signature
2020-06-29 14:07:21 +00:00
Brian Healey
27ecc62b11
Update spark version to update jetty (#6488)
CHANGELOG_BEGIN
Update spark version to update jetty to address security vulnerabilities
CHANGELOG_END

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
2020-06-25 10:51:36 -04:00
Brian Healey
f903ba70a6
Upgrade auth0 jwks-rsa version to 0.11.0 (#6218)
CHANGELOG_BEGIN
Upgrade auth0 jwks-rsa version to 0.11.0
CHANGELOG_END
2020-06-11 13:59:30 -04:00
Stephen Compall
78205afa0d
use 2.13-style collections library syntax in json-api (#6075)
* register scala-collection-compat with java deps list

* add scala-collection-compat to http-json deps

* remove breakOut throughout http-json, replaced with view/to or iterator/to

* use scala 2.13-style `to` calls in http-json

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* use 2.13-style to in lf-value-json

* some fused size comparisons

* remove low-hanging breakOuts in daml-lf

* regenerate maven_install.json for scala-collection-compat

* regenerate maven_install.json for scala-collection-compat

* regenerate maven_install.json for scala-collection-compat
2020-05-27 22:27:25 +00:00
Brian Healey
d162a4a14e
Update Jackson library version to latest published 2.11.0 (#6115)
* Update Jackson library version to latest published to resolve several reported vulnerabilities

CHANGELOG_BEGIN
Upgrade jackson version to 2.11.0 from 2.9.9.3
CHANGELOG_END

* regenerate updated jackson dependencies
2020-05-27 17:22:33 -04:00
Brian Healey
cfab7dab63
Update netty and iogrpc version (#6063)
* io.grpc:grpc-xxxx to 1.29.0 (from 1.22.1)
io.netty:netty-xxxx to .1.50.Final (from 4.1.37.Final)
io.nett.netty-tcp-native-boringssl-static to 2.0.30.Final (from 2.0.25.Final)

To resolve open vulnerabilities with these versions
netty-4.1.37.Final vulnerabilities
 BDSA-2018-4022 (Medium)
 BDSA-2019-2610 (Medium)
 BDSA-2019-3119 (CVE-2019-16869) (Medium)
 BDSA-2020-0130 (Medium)
 BDSA BDSA-2019-4230 (CVE-2019-20445) (Low)
 BDSA BDSA-2019-4231 (CVE-2019-20444) (Low)
 BDSA BDSA-2020-0666 (CVE-2020-11612) (Low)
 BDSA BDSA-2019-2642 (Low)
 BDSA BDSA-2019-2649 (Low)
 BDSA BDSA-2019-2643 (Low)

CHANGELOG_BEGIN
Upgrade io.grpc:grpc-xxxxx and io.netty:netty-xxx version to latest
released to avoid exposure to reported security vulnerabilities in
currently used versions
CHANGELOG_END
–

* Update spray versions to address vulnerabilities
CVE-2018-18853 and CVE-2018-18854

CHANGELOG_BEGIN
Upgrade io.grpc:grpc-xxxxx and io.netty:netty-xxx version to latest
released to avoid exposure to reported security vulnerabilities in
currently used versions
CHANGELOG_END

* do not change io.grpc version since reflection seems to be misbehaving

* Clarify how to bump grpc/netty/protobuf versions

Also "downgrade" netty to 4.1.48, according to
https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty

CHANGELOG_BEGIN
CHANGELOG_END

* Load protobuf deps after haskell deps to avoid loading an older version of rules_cc

* Upgrade protoc and protobuf-java to 3.11.0

* buildifier reformat

* regen unique int after rebase

* remove commented patch

Co-authored-by: Gerolf Seitz <gerolf.seitz@daml.com>
2020-05-27 13:44:34 -04:00
Shayne Fletcher
79563ee74b
ToxiProxy config (#6114)
changelog_begin
changelog_end
2020-05-27 07:08:45 -04:00
Oliver Seeliger
4c0ffdf4df
Upgrade scala compiler silencer to 1.6.0 (#5822)
* Upgrade scala compiler silencer to 1.6.0

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt build bazel file to new targets

* Switch to silencer plugin scala 2.12.11 per Samir's feedback

rather than 2.12.8

* Add missed bazel files

* Review feedback from Leo
2020-05-04 21:30:46 +00:00
Samir Talwar
6563048d5b
kvutils: Cache state value conversions from bytes. (#5426)
* kvutils: Cache state value conversions from bytes.

This seems to have a decent speedup in ledger-on-memory.

CHANGELOG_BEGIN
- [Ledger Integration Kit] Submissions now look up ledger values from a
  cache where possible, improving performance when there's contention over
  certain resources (e.g. common packages). The cache size currently
  defaults to 64 MB.
CHANGELOG_END

* kvutils: Make the SubmissionValidator statue value cache configurable.

* kvutils: Report state value cache metrics.

* kvutils: Add a suffix to a Long literal because WartRemover is unhappy.

Strangely, it doesn't fail on my machine.

* kvutils: Extract caching out into its own file.

* kvutils: Move the `bytesToStateValue` call into `cache.get`.

* kvutils: Move caching to its own package.

* kvutils: Inject the state value cache.

* kvutils: Default to no state value cache.

* kvutils: Accept a state value cache size in megabytes, not bytes.

* kvutils: Move cache building from `Config` to the `caching` package.

* kvutils: Replace Guava's cache with Caffeine.

* kvutils: Simplify caching configuration.

* sandbox: Enable state value caching by default.

CHANGELOG_BEGIN
- [Sandbox] State values deserialization is now cached, with a fixed
  cache size of 128MB.
CHANGELOG_END

* Changelog commit.

CHANGELOG_BEGIN
- [Ledger Integration Kit] The state value cache is now opt-in, with a
  default of no cache at all.
CHANGELOG_END
2020-04-06 17:30:32 +00:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Samir Talwar
3d6ecc13b6
Sandbox: Configurable metrics output. (#5113)
* sandbox: Clean up `MetricsReporting` a little.

Make sure it closes both reporters, and avoid starting things in a
constructor.

* sandbox: Add hidden options for enable metrics reporting.

* sandbox: Add a disambiguating name to the DB connection/thread pools.

CHANGELOG_BEGIN
- [Sandbox] DB connection pool metrics names have changed slightly, from
  ``daml.index.db.connection`` to ``daml.index.db.connection.sandbox``.
- [Ledger Integration Kit] DB connection pool metrics names have changed
  to disambiguate the StandaloneApiServer from the
  StandaloneIndexerServer. The former now has a ``.ledger-api-server``
  suffix, and the latter now has a ``.indexer`` suffix.
CHANGELOG_END

* sandbox-next: Use the same metrics registry for the API and indexer.

* sandbox: Give a useful error message on an invalid metrics reporter.

And simplify the error messages.

With the arguments `--client-auth=foo --metrics-reporter=foo`, we now
get the output:

```
Error: Option --client-auth failed when given 'foo'. Must be one of
  "none", "optional", or "require".
Error: Option --metrics-reporter failed when given 'foo'. Must be one of
  "console", or "csv:PATH".
Try --help for more information.
```

* sandbox: Pull out more helpers in `MetricsReporting`.

* sandbox: Rename MetricsReporter classes so they don't clash.

* sandbox: Wrap the `name` parameter in a `ServerName` tagged string.

For safety. Yours, not mine.

* sandbox: Push metrics to Graphite with `--metrics-reporter=graphite`.

* sandbox: Make `MetricsReporter.Graphite` singly-lazy, not doubly-.

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* sandbox: Replace `ServerName` with `ServerRole`.

* sandbox: Fix usage of `ServerRole.Testing` in `LedgerResource`.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-03-23 10:59:13 +00:00
Moritz Kiefer
6453a73968
Add first prototype of triggers as a service (TaaS) (#4287)
* Add first prototype of triggers as a service (TaaS)

This is an extremely basic version of the trigger as a service thingy.

Right now, it supports spawning triggers and stopping them but nothing
else.

There is a very simple test to check that it’s not completely broken.

changelog_begin
changelog_end

* Apply suggestions from code review

Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

* remove debugging output

* remove leftover import

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2020-01-31 12:53:45 +01:00
Samir Talwar
2737303ac1
sandbox: Upgrade Flyway from v5 to v6. (#4207)
* sandbox: Ensure that the Flyway tests actually pick up the SQL files.

* sandbox: Upgrade Flyway from v5 to v6.

CHANGELOG_BEGIN
- [Sandbox] Upgrade the Flyway database migrations library from v5 to v6.
CHANGELOG_END
2020-01-24 19:36:31 +01:00
Samir Talwar
0be3e72a12
ledger-on-sql: PostgreSQL support. (#4204)
* ledger-on-sql: Don't bother cleaning up after integration tests.

Turns out Bazel cleans up before each test run, so we only have one
set of outputs at a time. This is far more useful for debugging anyway.

* ledger-on-sql: Pull out a test-lib to simplify the conformance tests.

* ledger-on-sql: Turn `Database` from a sealed trait into a case class.

* ledger-on-sql: Support for PostgreSQL!

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-sql: Run the conformance tests against PostgreSQL.

* ledger-on-sql: Run the LotsOfParties conformance tests.

* ledger-on-sql: Use PostgreSQL's `RETURNING` keyword to save a query.

* ledger-on-sql: Ensure the reader connection pool is read-only.

* ledger-on-sql: Avoid cloning ByteStrings as byte arrays where possible.

Instead, use `ByteString#newInput()`, which returns an `InputStream`.
This is supported by H2 and PostgreSQL, but unfortunately not SQLite.

* ledger-on-sql: Run integration tests in parallel.

* Update the PostgreSQL and SQLite JDBC drivers.
2020-01-24 15:33:11 +01:00
Samir Talwar
2e3c6b28bd
ledger-on-sql: Get H2 working in the conformance tests. (#4155)
* kvutils: Make logback.xml a base file, rather than the only option.

* kvutils/app: Simplify logback.base.xml.

Mostly by getting rid of unnecessary appenders.

* ledger-on-sql: Add trace logging for all database work.

* Upgrade H2; there's a few useful bug fixes.

* kvutils/app: Let the user override the server JDBC URL.

* kvutils/app: Provide a way to specify the ledger ID.

* ledger-on-sql: If there are missing entries in the log, fail on read.

This can happen right now because we insert in parallel. Next step: stop
doing that.

* ledger-on-sql: Stop writing in parallel; it causes race conditions.

This unfortunately means we also stop _reading_ in parallel, which is
less fun.

* Revert "ledger-on-sql: Use a sequential log entry ID."

This reverts commit c58265bf43.

* ledger-on-sql: Create tables with columns that are not nullable.

* ledger-on-sql: The H2 conformance tests now work, just slowly.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-sql: Run tests against H2 and SQLite on memory and disk.

* ledger-on-sql: Allow H2 to read from the log in parallel with writes.

* ledger-on-sql: Use UUIDs for entry IDs, not 8 random bytes.

* ledger-on-sql: Make sure to log the correlation ID.

* ledger-on-sql: Do less while holding a database connection.

* ledger-on-sql: Log the connection.

* ledger-on-sql: We don't need to tell H2 not to drop the connection.

The connection pool takes care of that.

* ledger-on-sql: Disable H2 conformance tests on CI; they're too slow.

* ledger-on-sql: Rename `loggingContext` to `logCtx`.

* ledger-on-sql: Don't abuse log contexts; put the data in the message.

* ledger-on-sql: Make the connection log line easier to read.

And pull out the logger, even if it's only used once.

* ledger-on-sql: Run the unit tests against (H2, SQLite) * (memory, file).
2020-01-22 16:54:51 +01:00
Stefano Baghino
5231bab423
Introduce ContextualizedLogger and LoggingContext (#4046)
* Introduce ContextualizedLogger and LoggingContext

A ContextualizedLogger is a wrapper around an Slf4j logger. It uses
call-by-name parameters to only construct logging lines when necessary.
The underlying context can be used by invoking withoutContext.

More interestingly, every call to one of its logging methods requires
the presence of an implicit LoggingContext.

A LoggingContext is a way of definining a set of pairs that a
ContextualizedLogger can log without the need of it being specified for
every call.

A new context can be created with newLoggingContext and, given an
implicit LoggingContext, pairs can be added or overwritten using
withEnrichedLoggingContext.

Pairs in the context will be appended to each event logged by a
ContextualizedLogger in the following form:

some text logged explicitly (context: {key1=value1, key2=value2)

If the underlying Logger is configured to use the Logstash encoder the
keys and values in the LoggingContext will also be added to the logging
event encoded in JSON format.

CHANGELOG_BEGIN
CHANGELOG_END

* Apply suggestion in https://github.com/digital-asset/daml/pull/4046#pullrequestreview-343224692

Thanks to @SamirTalwar

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

* Address https://github.com/digital-asset/daml/pull/4046#discussion_r366874330

Co-authored-by: Samir Talwar <samir@noodlesandwich.com>
2020-01-16 13:05:40 +01:00