Commit Graph

28 Commits

Author SHA1 Message Date
Stephen Compall
7622862312
upgrade to Wartremover 2.4.9 (#6551)
* upgrade to wartremover 2.4.9

* simplify wart list and list JavaConversions as disabled

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* delete long-obsolete, contradictory comment

* also upgrade wartremover in compatibility (leaving aside maven_install.json)

* update compatibility maven_install.json to match
2020-07-01 15:02:58 +00:00
Stefano Baghino
748d17bbe0
Bump Flyway version to 6.5 (#6520)
* Bump Flyway version to 6.5

Prevents incurring into https://github.com/flyway/flyway/issues/2759 (which was apparently solved in 6.4.0)

changelog_begin
changelog_end

* Comply with changed method signature
2020-06-29 14:07:21 +00:00
Brian Healey
27ecc62b11
Update spark version to update jetty (#6488)
CHANGELOG_BEGIN
Update spark version to update jetty to address security vulnerabilities
CHANGELOG_END

Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
2020-06-25 10:51:36 -04:00
Brian Healey
f903ba70a6
Upgrade auth0 jwks-rsa version to 0.11.0 (#6218)
CHANGELOG_BEGIN
Upgrade auth0 jwks-rsa version to 0.11.0
CHANGELOG_END
2020-06-11 13:59:30 -04:00
Stephen Compall
78205afa0d
use 2.13-style collections library syntax in json-api (#6075)
* register scala-collection-compat with java deps list

* add scala-collection-compat to http-json deps

* remove breakOut throughout http-json, replaced with view/to or iterator/to

* use scala 2.13-style `to` calls in http-json

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* use 2.13-style to in lf-value-json

* some fused size comparisons

* remove low-hanging breakOuts in daml-lf

* regenerate maven_install.json for scala-collection-compat

* regenerate maven_install.json for scala-collection-compat

* regenerate maven_install.json for scala-collection-compat
2020-05-27 22:27:25 +00:00
Brian Healey
d162a4a14e
Update Jackson library version to latest published 2.11.0 (#6115)
* Update Jackson library version to latest published to resolve several reported vulnerabilities

CHANGELOG_BEGIN
Upgrade jackson version to 2.11.0 from 2.9.9.3
CHANGELOG_END

* regenerate updated jackson dependencies
2020-05-27 17:22:33 -04:00
Brian Healey
cfab7dab63
Update netty and iogrpc version (#6063)
* io.grpc:grpc-xxxx to 1.29.0 (from 1.22.1)
io.netty:netty-xxxx to .1.50.Final (from 4.1.37.Final)
io.nett.netty-tcp-native-boringssl-static to 2.0.30.Final (from 2.0.25.Final)

To resolve open vulnerabilities with these versions
netty-4.1.37.Final vulnerabilities
 BDSA-2018-4022 (Medium)
 BDSA-2019-2610 (Medium)
 BDSA-2019-3119 (CVE-2019-16869) (Medium)
 BDSA-2020-0130 (Medium)
 BDSA BDSA-2019-4230 (CVE-2019-20445) (Low)
 BDSA BDSA-2019-4231 (CVE-2019-20444) (Low)
 BDSA BDSA-2020-0666 (CVE-2020-11612) (Low)
 BDSA BDSA-2019-2642 (Low)
 BDSA BDSA-2019-2649 (Low)
 BDSA BDSA-2019-2643 (Low)

CHANGELOG_BEGIN
Upgrade io.grpc:grpc-xxxxx and io.netty:netty-xxx version to latest
released to avoid exposure to reported security vulnerabilities in
currently used versions
CHANGELOG_END
–

* Update spray versions to address vulnerabilities
CVE-2018-18853 and CVE-2018-18854

CHANGELOG_BEGIN
Upgrade io.grpc:grpc-xxxxx and io.netty:netty-xxx version to latest
released to avoid exposure to reported security vulnerabilities in
currently used versions
CHANGELOG_END

* do not change io.grpc version since reflection seems to be misbehaving

* Clarify how to bump grpc/netty/protobuf versions

Also "downgrade" netty to 4.1.48, according to
https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty

CHANGELOG_BEGIN
CHANGELOG_END

* Load protobuf deps after haskell deps to avoid loading an older version of rules_cc

* Upgrade protoc and protobuf-java to 3.11.0

* buildifier reformat

* regen unique int after rebase

* remove commented patch

Co-authored-by: Gerolf Seitz <gerolf.seitz@daml.com>
2020-05-27 13:44:34 -04:00
Shayne Fletcher
79563ee74b
ToxiProxy config (#6114)
changelog_begin
changelog_end
2020-05-27 07:08:45 -04:00
Oliver Seeliger
4c0ffdf4df
Upgrade scala compiler silencer to 1.6.0 (#5822)
* Upgrade scala compiler silencer to 1.6.0

CHANGELOG_BEGIN
CHANGELOG_END

* Adapt build bazel file to new targets

* Switch to silencer plugin scala 2.12.11 per Samir's feedback

rather than 2.12.8

* Add missed bazel files

* Review feedback from Leo
2020-05-04 21:30:46 +00:00
Samir Talwar
6563048d5b
kvutils: Cache state value conversions from bytes. (#5426)
* kvutils: Cache state value conversions from bytes.

This seems to have a decent speedup in ledger-on-memory.

CHANGELOG_BEGIN
- [Ledger Integration Kit] Submissions now look up ledger values from a
  cache where possible, improving performance when there's contention over
  certain resources (e.g. common packages). The cache size currently
  defaults to 64 MB.
CHANGELOG_END

* kvutils: Make the SubmissionValidator statue value cache configurable.

* kvutils: Report state value cache metrics.

* kvutils: Add a suffix to a Long literal because WartRemover is unhappy.

Strangely, it doesn't fail on my machine.

* kvutils: Extract caching out into its own file.

* kvutils: Move the `bytesToStateValue` call into `cache.get`.

* kvutils: Move caching to its own package.

* kvutils: Inject the state value cache.

* kvutils: Default to no state value cache.

* kvutils: Accept a state value cache size in megabytes, not bytes.

* kvutils: Move cache building from `Config` to the `caching` package.

* kvutils: Replace Guava's cache with Caffeine.

* kvutils: Simplify caching configuration.

* sandbox: Enable state value caching by default.

CHANGELOG_BEGIN
- [Sandbox] State values deserialization is now cached, with a fixed
  cache size of 128MB.
CHANGELOG_END

* Changelog commit.

CHANGELOG_BEGIN
- [Ledger Integration Kit] The state value cache is now opt-in, with a
  default of no cache at all.
CHANGELOG_END
2020-04-06 17:30:32 +00:00
Samir Talwar
3d6ecc13b6
Sandbox: Configurable metrics output. (#5113)
* sandbox: Clean up `MetricsReporting` a little.

Make sure it closes both reporters, and avoid starting things in a
constructor.

* sandbox: Add hidden options for enable metrics reporting.

* sandbox: Add a disambiguating name to the DB connection/thread pools.

CHANGELOG_BEGIN
- [Sandbox] DB connection pool metrics names have changed slightly, from
  ``daml.index.db.connection`` to ``daml.index.db.connection.sandbox``.
- [Ledger Integration Kit] DB connection pool metrics names have changed
  to disambiguate the StandaloneApiServer from the
  StandaloneIndexerServer. The former now has a ``.ledger-api-server``
  suffix, and the latter now has a ``.indexer`` suffix.
CHANGELOG_END

* sandbox-next: Use the same metrics registry for the API and indexer.

* sandbox: Give a useful error message on an invalid metrics reporter.

And simplify the error messages.

With the arguments `--client-auth=foo --metrics-reporter=foo`, we now
get the output:

```
Error: Option --client-auth failed when given 'foo'. Must be one of
  "none", "optional", or "require".
Error: Option --metrics-reporter failed when given 'foo'. Must be one of
  "console", or "csv:PATH".
Try --help for more information.
```

* sandbox: Pull out more helpers in `MetricsReporting`.

* sandbox: Rename MetricsReporter classes so they don't clash.

* sandbox: Wrap the `name` parameter in a `ServerName` tagged string.

For safety. Yours, not mine.

* sandbox: Push metrics to Graphite with `--metrics-reporter=graphite`.

* sandbox: Make `MetricsReporter.Graphite` singly-lazy, not doubly-.

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* sandbox: Replace `ServerName` with `ServerRole`.

* sandbox: Fix usage of `ServerRole.Testing` in `LedgerResource`.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-03-23 10:59:13 +00:00
Moritz Kiefer
6453a73968
Add first prototype of triggers as a service (TaaS) (#4287)
* Add first prototype of triggers as a service (TaaS)

This is an extremely basic version of the trigger as a service thingy.

Right now, it supports spawning triggers and stopping them but nothing
else.

There is a very simple test to check that it’s not completely broken.

changelog_begin
changelog_end

* Apply suggestions from code review

Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

* remove debugging output

* remove leftover import

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2020-01-31 12:53:45 +01:00
Samir Talwar
2737303ac1
sandbox: Upgrade Flyway from v5 to v6. (#4207)
* sandbox: Ensure that the Flyway tests actually pick up the SQL files.

* sandbox: Upgrade Flyway from v5 to v6.

CHANGELOG_BEGIN
- [Sandbox] Upgrade the Flyway database migrations library from v5 to v6.
CHANGELOG_END
2020-01-24 19:36:31 +01:00
Samir Talwar
0be3e72a12
ledger-on-sql: PostgreSQL support. (#4204)
* ledger-on-sql: Don't bother cleaning up after integration tests.

Turns out Bazel cleans up before each test run, so we only have one
set of outputs at a time. This is far more useful for debugging anyway.

* ledger-on-sql: Pull out a test-lib to simplify the conformance tests.

* ledger-on-sql: Turn `Database` from a sealed trait into a case class.

* ledger-on-sql: Support for PostgreSQL!

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-sql: Run the conformance tests against PostgreSQL.

* ledger-on-sql: Run the LotsOfParties conformance tests.

* ledger-on-sql: Use PostgreSQL's `RETURNING` keyword to save a query.

* ledger-on-sql: Ensure the reader connection pool is read-only.

* ledger-on-sql: Avoid cloning ByteStrings as byte arrays where possible.

Instead, use `ByteString#newInput()`, which returns an `InputStream`.
This is supported by H2 and PostgreSQL, but unfortunately not SQLite.

* ledger-on-sql: Run integration tests in parallel.

* Update the PostgreSQL and SQLite JDBC drivers.
2020-01-24 15:33:11 +01:00
Samir Talwar
2e3c6b28bd
ledger-on-sql: Get H2 working in the conformance tests. (#4155)
* kvutils: Make logback.xml a base file, rather than the only option.

* kvutils/app: Simplify logback.base.xml.

Mostly by getting rid of unnecessary appenders.

* ledger-on-sql: Add trace logging for all database work.

* Upgrade H2; there's a few useful bug fixes.

* kvutils/app: Let the user override the server JDBC URL.

* kvutils/app: Provide a way to specify the ledger ID.

* ledger-on-sql: If there are missing entries in the log, fail on read.

This can happen right now because we insert in parallel. Next step: stop
doing that.

* ledger-on-sql: Stop writing in parallel; it causes race conditions.

This unfortunately means we also stop _reading_ in parallel, which is
less fun.

* Revert "ledger-on-sql: Use a sequential log entry ID."

This reverts commit c58265bf43.

* ledger-on-sql: Create tables with columns that are not nullable.

* ledger-on-sql: The H2 conformance tests now work, just slowly.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-sql: Run tests against H2 and SQLite on memory and disk.

* ledger-on-sql: Allow H2 to read from the log in parallel with writes.

* ledger-on-sql: Use UUIDs for entry IDs, not 8 random bytes.

* ledger-on-sql: Make sure to log the correlation ID.

* ledger-on-sql: Do less while holding a database connection.

* ledger-on-sql: Log the connection.

* ledger-on-sql: We don't need to tell H2 not to drop the connection.

The connection pool takes care of that.

* ledger-on-sql: Disable H2 conformance tests on CI; they're too slow.

* ledger-on-sql: Rename `loggingContext` to `logCtx`.

* ledger-on-sql: Don't abuse log contexts; put the data in the message.

* ledger-on-sql: Make the connection log line easier to read.

And pull out the logger, even if it's only used once.

* ledger-on-sql: Run the unit tests against (H2, SQLite) * (memory, file).
2020-01-22 16:54:51 +01:00
Stefano Baghino
5231bab423
Introduce ContextualizedLogger and LoggingContext (#4046)
* Introduce ContextualizedLogger and LoggingContext

A ContextualizedLogger is a wrapper around an Slf4j logger. It uses
call-by-name parameters to only construct logging lines when necessary.
The underlying context can be used by invoking withoutContext.

More interestingly, every call to one of its logging methods requires
the presence of an implicit LoggingContext.

A LoggingContext is a way of definining a set of pairs that a
ContextualizedLogger can log without the need of it being specified for
every call.

A new context can be created with newLoggingContext and, given an
implicit LoggingContext, pairs can be added or overwritten using
withEnrichedLoggingContext.

Pairs in the context will be appended to each event logged by a
ContextualizedLogger in the following form:

some text logged explicitly (context: {key1=value1, key2=value2)

If the underlying Logger is configured to use the Logstash encoder the
keys and values in the LoggingContext will also be added to the logging
event encoded in JSON format.

CHANGELOG_BEGIN
CHANGELOG_END

* Apply suggestion in https://github.com/digital-asset/daml/pull/4046#pullrequestreview-343224692

Thanks to @SamirTalwar

Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>

* Address https://github.com/digital-asset/daml/pull/4046#discussion_r366874330

Co-authored-by: Samir Talwar <samir@noodlesandwich.com>
2020-01-16 13:05:40 +01:00
Andreas Herrmann
942c0b7c9e Makes scalatest available in IntelliJ (#4032)
The target override for scalatest was pointing to a meta package that
was empty itself and reexported scalactic and scalatest. It seems this
confused the IntelliJ plugin. This instead introduces dedicated
overrides for scalactic and scalatest.

CHANGELOG_BEGIN
CHANGELOG_END

Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
2020-01-13 17:43:52 +00:00
Miklos
1124931f78
Simplified participant-side reader/writer APIs (#3923)
Initial version of simplified participant-side reader/writer APIs.
2020-01-10 12:20:32 +01:00
Brian Healey
cf684d2595
update dropwizrd to version 4.1.2 (#3990)
CHANGELOG_BEGIN
[Ledger API - Metrics] update dropwizard to version 4.1.2
CHANGELOG_END
2020-01-09 07:25:47 -05:00
Brian Healey
9f13a2fb10 Upgrade to Akka 2.6.1, akka-http 10.1.11 and Scala 2.12.10 (#3903)
* Upgrade to Akka 2.6.1, akka-http 10.1.11 and Scala 2.12.10

Akka 2.6.1 Upgrade Changes
- Materializer in place of ActorMaterializer
- Source.future instead of Source.fromFuture
- The Scheduler.schedule method has been deprecated in favor of selecting scheduleWithFixedDelay or scheduleAtFixedRate
- onDownstreamFinish(cause: Throwable)
- ActorAttributes.supervisionStrategy(...) in place of ActorMaterializerSettings.withSupervisionStrategy

See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html

* Akka 2.6.1 Upgrade Changes
- onDownstreamFinish(cause: Throwable)

See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html

* code review: remove unnecessary supervision strategy
2019-12-19 18:00:59 +00:00
Brian Healey
e8c6364d32
Update akka-http version to 10.1.12, akka-stream version to 2.5.26, and corresponding uptick to scala 2.13 (#3899) 2019-12-18 23:14:28 -05:00
Brian Healey
d8089f44b0
Uptick akka-http version to 10.1.10 / uptick scala version to 2.12.9 (#3896) 2019-12-18 16:22:39 -05:00
Brian Healey
23edf52d36
resolve jackson transitive clash (#3775) 2019-12-09 07:47:53 -05:00
Robert Autenrieth
4c3f6014f6
Implement support for RSA-signed JWT tokens (#3526)
* Improve Navigator output

* Fix Navigator not using the access token

* Add RSA signatures for JWT tokens

* Remove unused method

* Add timeouts for reading JWKS

* Fix test

* Rename method for consistency

* Improve comment

* More renaming for consistency

* CHANGELOG

CHANGELOG_BEGIN
- [Sandbox] Add CLI options to start the sandbox with JWT based authentication with RSA signed tokens.
  See `issue #3155 <https://github.com/digital-asset/daml/issues/3155>`__ .
- [Navigator] Fixed a bug where the `--access-token-file` option did not work correctly.
CHANGELOG_END

* Make JwksVerifier limits configurable

* Make SimpleHttpServer private
2019-11-25 16:29:24 +01:00
mziolekda
0a2641ba78
Remove dead dependency on jboss logging library (#3333)
* remove dead dependency om jboss logging library

* remove references to jboss from license file
2019-11-05 11:36:43 +01:00
Jussi Mäki
4e440bd0a6 Add kvutils ledger dumps and integrity checker (#3167)
kvutils now supports dumping the ledger to a file via environment
variable: "KVUTILS_LEDGER_DUMP=/tmp/ledger.dump".

The integrity checker tool allows re-processing of ledger dumps to
validate that all submissions, state and log entries can be parsed
and processed. This forms the basis of kvutils data continuity test
suite.

Currently the checker strictly enforces that re-processed
outputs match the inputs exactly, which for now guarantees forwards
and backwards compatibility. The checker will be adapted when
forwards-incompatible changes or new kvutils message versions arise.

A select collection of ledger dumps will be maintained in a separate
repository (to be decided) and these dumps are to be validated by
the DAML CI.
2019-10-30 17:02:00 +00:00
Andreas Herrmann
03037ca8bd fetch_sources = True (#3279) 2019-10-29 14:43:56 +00:00
Andreas Herrmann
2bd1db490a
Replace bazel-deps by rules_jvm_external (#3253)
* Update bazel-common to fix javadoc issues

Specifically, to fix the following error

```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel:7:1: in javadoc_library rule //ledger-api/rs-grpc-bridge:rs-grpc-bridge_javadoc:
Traceback (most recent call last):
        File "/home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel", line 7
                javadoc_library(name = 'rs-grpc-bridge_javadoc')
        File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/com_github_google_bazel_common/tools/javadoc/javadoc.bzl", line 27, in _javadoc_library
                dep.java.transitive_deps
object of type 'JavaSkylarkApiProvider' has no field 'transitive_deps'
```

* Define Maven deps using rules_jvm_external

* Pin artifacts

* Remove bazel-deps generated targets

* Remove bazel-deps

* Switch to rules_jvm_external targets

* update bazel documentation

* pom_file: There are no more bazel-deps targets

* BAZEL-JVM.md `maven_install` typo
2019-10-28 13:53:14 +01:00