* Add a Scala 2.13 build pipeline
This adds initial support for multiple Scala versions controlled via
the DAML_SCALA_VERSION env var and a CI job to make sure we don’t
regress. For now we only test //libs-scala/ports/... which seemed like
the easiest starting point I could find. We can incrementally expand
that over time.
changelog_begin
changelog_end
* Document pinning
changelog_begin
changelog_end
* Address review comments
changelog_begin
changelog_end
* Upgrade Scala dependencies for 2.13 compatibility
This upgrades a bunch of Scala libraries to versions that have 2.13
support. There are two libraries that are still missing:
- diffson, this has a new version but with significant breaking
changes and it is only used in Naigator console which I hope to kill
before I have to worry about this.
- ai.x:diff, this is used in the ledger API test tool. The library is
abondened but there are a few alternatives.
changelog_begin
changelog_end
* Fix pureconfig
changelog_begin
changelog_end
* Fix Navigator
changelog_begin
changelog_end
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.
This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.
Apologies for the giant PR, I don’t see a way to keep it smaller.
changelog_begin
changelog_end
* add blackduck scan to run on master (#6130)
* add blackduck scan
* disable go scanning
exclude entire language-support/ts directory for node scanning
break to multiple lines to make command line params easier to parse
* Increase timeout for blackduck binary scan
* update blackduck scan config
* remove some exclusions, force python3
* exclude GO until path to go executable can be resolved
* added readme explanation of why we want this file
* fail in case of policy violation
* ensure haskell bazel scan completes before running second round scan for bazel jvm and node and other langs
* trigger notices file gen to ensure BOM complete
* remove trailing end of lines
* run with latest detect version and unique code location name changes to wrapper script
* Add blackduck to daily compat job
* DO NOT MERGE: condition false to disable other jobs for testing
* remove parameters not available to cronjob
* Revert changes to regular CI pipeline
CHANGELOG_BEGIN
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* Do not get branch name from variable
* Upgrade com.fasterxml.jackson.core:jackson-databind to 2.12.0 to address security vulnerability
* Remove disabling of other jobs, set to branch to be used on prod runs
* Apply suggestions from code review
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Address code review comments
* Updated NOTICES file
* Run bazel build, update NOTICES file
* Correct dade-assist
* do not have perms to pipe to dev/null
* Add md file explaining how to update NOTICES file
* Add instructions for running blackduck locally
* Add a link to full security-blackduck readme
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* [DPP-84] Employ parallel unnesting instead of batching on PostgreSQL
changelog_begin
[Integration Kit] When using a PostgreSQL-based index, leveraging native
parallel unnesting allows to more efficiently index new transactions.
changelog_end
* Address review https://github.com/digital-asset/daml/pull/8042#pullrequestreview-541759596
* Upgrade akka-http to 10.2
Follow up to #8048, I left out this upgrade to reduce noise and since
I wasn’t quite sure how involved it was going to be.
changelog_begin
changelog_end
* Reenable transparent HEAD requests
Apparently no longer on by default but we depend on this in waitForHttpServer
changelog_begin
changelog_end
* Upgrade akka and akka-http
Was chasing an issue somewhere and thought this might affect it in
some way. It didn’t but I might as well turn the upgrade into a PR.
changelog_begin
changelog_end
* Fix trigger service tests
changelog_begin
changelog_end
* Downgrade akka-http again
changelog_begin
changelog_end
* Upgrade akka-http again and fix tests
changelog_begin
changelog_end
* Cleanup trigger service
changelog_begin
changelog_end
* set doobie version to 0.9.2 and rerun maven pin
* port extractor and some of JSON API
* repin maven
* use doobie's own builder compatibility where required
* use probably bad derivations to supply Blockers where transactEC was required
- The point of using Blocker instead of ExecutionContext seems to be to
especially emphasize to API users that it isn't appropriate to use an
ExecutionContext with ordinary behavior. That is what we have done, which
should probably change, but just compiling for now.
* fix fragment inspection test for internal restructuring
- This test depends on implementation details of Doobie, so naturally it must be
altered when that runs. Fortunately, it's been made easier by the changes
in this upgrade.
* allow 256 blockers for navigator transaction blocker, like the global EC
* allow as many blockers as the pool size for trigger service
- The transactor shouldn't share ExecutionContext for transactions with the
caller, so we set up a new one based on configured pool size.
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
Upgrade Jackson to 2.11.2 to address security vulnerabilities
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* Run gatling scenario from the perf runner main
reports are disabled for now, getting a class not found
when generating them
changelog_begin
changelog_end
pureconfig-macros pulls in the scala-compiler library which in turn
pulls in jansi-native which is somewhat problematic as it’s not
well supported outside of x86. It also just adds roughly 10% unused
junk to the http-json fat JAR. Afaict, pureconfig-macros is only used
at compile time so we can remove it.
changelog_begin
changelog_end
* upgrade to wartremover 2.4.9
* simplify wart list and list JavaConversions as disabled
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* delete long-obsolete, contradictory comment
* also upgrade wartremover in compatibility (leaving aside maven_install.json)
* update compatibility maven_install.json to match
* Bump Flyway version to 6.5
Prevents incurring into https://github.com/flyway/flyway/issues/2759 (which was apparently solved in 6.4.0)
changelog_begin
changelog_end
* Comply with changed method signature
CHANGELOG_BEGIN
Update spark version to update jetty to address security vulnerabilities
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* register scala-collection-compat with java deps list
* add scala-collection-compat to http-json deps
* remove breakOut throughout http-json, replaced with view/to or iterator/to
* use scala 2.13-style `to` calls in http-json
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* use 2.13-style to in lf-value-json
* some fused size comparisons
* remove low-hanging breakOuts in daml-lf
* regenerate maven_install.json for scala-collection-compat
* regenerate maven_install.json for scala-collection-compat
* regenerate maven_install.json for scala-collection-compat
* Update Jackson library version to latest published to resolve several reported vulnerabilities
CHANGELOG_BEGIN
Upgrade jackson version to 2.11.0 from 2.9.9.3
CHANGELOG_END
* regenerate updated jackson dependencies
* io.grpc:grpc-xxxx to 1.29.0 (from 1.22.1)
io.netty:netty-xxxx to .1.50.Final (from 4.1.37.Final)
io.nett.netty-tcp-native-boringssl-static to 2.0.30.Final (from 2.0.25.Final)
To resolve open vulnerabilities with these versions
netty-4.1.37.Final vulnerabilities
BDSA-2018-4022 (Medium)
BDSA-2019-2610 (Medium)
BDSA-2019-3119 (CVE-2019-16869) (Medium)
BDSA-2020-0130 (Medium)
BDSA BDSA-2019-4230 (CVE-2019-20445) (Low)
BDSA BDSA-2019-4231 (CVE-2019-20444) (Low)
BDSA BDSA-2020-0666 (CVE-2020-11612) (Low)
BDSA BDSA-2019-2642 (Low)
BDSA BDSA-2019-2649 (Low)
BDSA BDSA-2019-2643 (Low)
CHANGELOG_BEGIN
Upgrade io.grpc:grpc-xxxxx and io.netty:netty-xxx version to latest
released to avoid exposure to reported security vulnerabilities in
currently used versions
CHANGELOG_END
–
* Update spray versions to address vulnerabilities
CVE-2018-18853 and CVE-2018-18854
CHANGELOG_BEGIN
Upgrade io.grpc:grpc-xxxxx and io.netty:netty-xxx version to latest
released to avoid exposure to reported security vulnerabilities in
currently used versions
CHANGELOG_END
* do not change io.grpc version since reflection seems to be misbehaving
* Clarify how to bump grpc/netty/protobuf versions
Also "downgrade" netty to 4.1.48, according to
https://github.com/grpc/grpc-java/blob/master/SECURITY.md#netty
CHANGELOG_BEGIN
CHANGELOG_END
* Load protobuf deps after haskell deps to avoid loading an older version of rules_cc
* Upgrade protoc and protobuf-java to 3.11.0
* buildifier reformat
* regen unique int after rebase
* remove commented patch
Co-authored-by: Gerolf Seitz <gerolf.seitz@daml.com>
* Upgrade scala compiler silencer to 1.6.0
CHANGELOG_BEGIN
CHANGELOG_END
* Adapt build bazel file to new targets
* Switch to silencer plugin scala 2.12.11 per Samir's feedback
rather than 2.12.8
* Add missed bazel files
* Review feedback from Leo
* kvutils: Cache state value conversions from bytes.
This seems to have a decent speedup in ledger-on-memory.
CHANGELOG_BEGIN
- [Ledger Integration Kit] Submissions now look up ledger values from a
cache where possible, improving performance when there's contention over
certain resources (e.g. common packages). The cache size currently
defaults to 64 MB.
CHANGELOG_END
* kvutils: Make the SubmissionValidator statue value cache configurable.
* kvutils: Report state value cache metrics.
* kvutils: Add a suffix to a Long literal because WartRemover is unhappy.
Strangely, it doesn't fail on my machine.
* kvutils: Extract caching out into its own file.
* kvutils: Move the `bytesToStateValue` call into `cache.get`.
* kvutils: Move caching to its own package.
* kvutils: Inject the state value cache.
* kvutils: Default to no state value cache.
* kvutils: Accept a state value cache size in megabytes, not bytes.
* kvutils: Move cache building from `Config` to the `caching` package.
* kvutils: Replace Guava's cache with Caffeine.
* kvutils: Simplify caching configuration.
* sandbox: Enable state value caching by default.
CHANGELOG_BEGIN
- [Sandbox] State values deserialization is now cached, with a fixed
cache size of 128MB.
CHANGELOG_END
* Changelog commit.
CHANGELOG_BEGIN
- [Ledger Integration Kit] The state value cache is now opt-in, with a
default of no cache at all.
CHANGELOG_END
* sandbox: Clean up `MetricsReporting` a little.
Make sure it closes both reporters, and avoid starting things in a
constructor.
* sandbox: Add hidden options for enable metrics reporting.
* sandbox: Add a disambiguating name to the DB connection/thread pools.
CHANGELOG_BEGIN
- [Sandbox] DB connection pool metrics names have changed slightly, from
``daml.index.db.connection`` to ``daml.index.db.connection.sandbox``.
- [Ledger Integration Kit] DB connection pool metrics names have changed
to disambiguate the StandaloneApiServer from the
StandaloneIndexerServer. The former now has a ``.ledger-api-server``
suffix, and the latter now has a ``.indexer`` suffix.
CHANGELOG_END
* sandbox-next: Use the same metrics registry for the API and indexer.
* sandbox: Give a useful error message on an invalid metrics reporter.
And simplify the error messages.
With the arguments `--client-auth=foo --metrics-reporter=foo`, we now
get the output:
```
Error: Option --client-auth failed when given 'foo'. Must be one of
"none", "optional", or "require".
Error: Option --metrics-reporter failed when given 'foo'. Must be one of
"console", or "csv:PATH".
Try --help for more information.
```
* sandbox: Pull out more helpers in `MetricsReporting`.
* sandbox: Rename MetricsReporter classes so they don't clash.
* sandbox: Wrap the `name` parameter in a `ServerName` tagged string.
For safety. Yours, not mine.
* sandbox: Push metrics to Graphite with `--metrics-reporter=graphite`.
* sandbox: Make `MetricsReporter.Graphite` singly-lazy, not doubly-.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: Replace `ServerName` with `ServerRole`.
* sandbox: Fix usage of `ServerRole.Testing` in `LedgerResource`.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Add first prototype of triggers as a service (TaaS)
This is an extremely basic version of the trigger as a service thingy.
Right now, it supports spawning triggers and stopping them but nothing
else.
There is a very simple test to check that it’s not completely broken.
changelog_begin
changelog_end
* Apply suggestions from code review
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* remove debugging output
* remove leftover import
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* sandbox: Ensure that the Flyway tests actually pick up the SQL files.
* sandbox: Upgrade Flyway from v5 to v6.
CHANGELOG_BEGIN
- [Sandbox] Upgrade the Flyway database migrations library from v5 to v6.
CHANGELOG_END
* ledger-on-sql: Don't bother cleaning up after integration tests.
Turns out Bazel cleans up before each test run, so we only have one
set of outputs at a time. This is far more useful for debugging anyway.
* ledger-on-sql: Pull out a test-lib to simplify the conformance tests.
* ledger-on-sql: Turn `Database` from a sealed trait into a case class.
* ledger-on-sql: Support for PostgreSQL!
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run the conformance tests against PostgreSQL.
* ledger-on-sql: Run the LotsOfParties conformance tests.
* ledger-on-sql: Use PostgreSQL's `RETURNING` keyword to save a query.
* ledger-on-sql: Ensure the reader connection pool is read-only.
* ledger-on-sql: Avoid cloning ByteStrings as byte arrays where possible.
Instead, use `ByteString#newInput()`, which returns an `InputStream`.
This is supported by H2 and PostgreSQL, but unfortunately not SQLite.
* ledger-on-sql: Run integration tests in parallel.
* Update the PostgreSQL and SQLite JDBC drivers.
* kvutils: Make logback.xml a base file, rather than the only option.
* kvutils/app: Simplify logback.base.xml.
Mostly by getting rid of unnecessary appenders.
* ledger-on-sql: Add trace logging for all database work.
* Upgrade H2; there's a few useful bug fixes.
* kvutils/app: Let the user override the server JDBC URL.
* kvutils/app: Provide a way to specify the ledger ID.
* ledger-on-sql: If there are missing entries in the log, fail on read.
This can happen right now because we insert in parallel. Next step: stop
doing that.
* ledger-on-sql: Stop writing in parallel; it causes race conditions.
This unfortunately means we also stop _reading_ in parallel, which is
less fun.
* Revert "ledger-on-sql: Use a sequential log entry ID."
This reverts commit c58265bf43.
* ledger-on-sql: Create tables with columns that are not nullable.
* ledger-on-sql: The H2 conformance tests now work, just slowly.
CHANGELOG_BEGIN
CHANGELOG_END
* ledger-on-sql: Run tests against H2 and SQLite on memory and disk.
* ledger-on-sql: Allow H2 to read from the log in parallel with writes.
* ledger-on-sql: Use UUIDs for entry IDs, not 8 random bytes.
* ledger-on-sql: Make sure to log the correlation ID.
* ledger-on-sql: Do less while holding a database connection.
* ledger-on-sql: Log the connection.
* ledger-on-sql: We don't need to tell H2 not to drop the connection.
The connection pool takes care of that.
* ledger-on-sql: Disable H2 conformance tests on CI; they're too slow.
* ledger-on-sql: Rename `loggingContext` to `logCtx`.
* ledger-on-sql: Don't abuse log contexts; put the data in the message.
* ledger-on-sql: Make the connection log line easier to read.
And pull out the logger, even if it's only used once.
* ledger-on-sql: Run the unit tests against (H2, SQLite) * (memory, file).
* Introduce ContextualizedLogger and LoggingContext
A ContextualizedLogger is a wrapper around an Slf4j logger. It uses
call-by-name parameters to only construct logging lines when necessary.
The underlying context can be used by invoking withoutContext.
More interestingly, every call to one of its logging methods requires
the presence of an implicit LoggingContext.
A LoggingContext is a way of definining a set of pairs that a
ContextualizedLogger can log without the need of it being specified for
every call.
A new context can be created with newLoggingContext and, given an
implicit LoggingContext, pairs can be added or overwritten using
withEnrichedLoggingContext.
Pairs in the context will be appended to each event logged by a
ContextualizedLogger in the following form:
some text logged explicitly (context: {key1=value1, key2=value2)
If the underlying Logger is configured to use the Logstash encoder the
keys and values in the LoggingContext will also be added to the logging
event encoded in JSON format.
CHANGELOG_BEGIN
CHANGELOG_END
* Apply suggestion in https://github.com/digital-asset/daml/pull/4046#pullrequestreview-343224692
Thanks to @SamirTalwar
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
* Address https://github.com/digital-asset/daml/pull/4046#discussion_r366874330
Co-authored-by: Samir Talwar <samir@noodlesandwich.com>
The target override for scalatest was pointing to a meta package that
was empty itself and reexported scalactic and scalatest. It seems this
confused the IntelliJ plugin. This instead introduces dedicated
overrides for scalactic and scalatest.
CHANGELOG_BEGIN
CHANGELOG_END
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
* Upgrade to Akka 2.6.1, akka-http 10.1.11 and Scala 2.12.10
Akka 2.6.1 Upgrade Changes
- Materializer in place of ActorMaterializer
- Source.future instead of Source.fromFuture
- The Scheduler.schedule method has been deprecated in favor of selecting scheduleWithFixedDelay or scheduleAtFixedRate
- onDownstreamFinish(cause: Throwable)
- ActorAttributes.supervisionStrategy(...) in place of ActorMaterializerSettings.withSupervisionStrategy
See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html
* Akka 2.6.1 Upgrade Changes
- onDownstreamFinish(cause: Throwable)
See https://doc.akka.io/docs/akka/current/project/migration-guide-2.5.x-2.6.x.html
* code review: remove unnecessary supervision strategy
* Improve Navigator output
* Fix Navigator not using the access token
* Add RSA signatures for JWT tokens
* Remove unused method
* Add timeouts for reading JWKS
* Fix test
* Rename method for consistency
* Improve comment
* More renaming for consistency
* CHANGELOG
CHANGELOG_BEGIN
- [Sandbox] Add CLI options to start the sandbox with JWT based authentication with RSA signed tokens.
See `issue #3155 <https://github.com/digital-asset/daml/issues/3155>`__ .
- [Navigator] Fixed a bug where the `--access-token-file` option did not work correctly.
CHANGELOG_END
* Make JwksVerifier limits configurable
* Make SimpleHttpServer private
* grpc-definitions: Delete health_service.proto
We can use the version in io.grpc:grpc-services instead.
* ledger: Delete ledger/API.md.
* sandbox: Fix warnings in ApiServices flagged by IntelliJ.
* sandbox: Implement a dummy grpc.health.v1.Health.Check endpoint.
* sandbox: Implement a dummy grpc.health.v1.Health.Watch endpoint.
* sandbox: Drop repeated elements from grpc.health.v1.Health.Watch.
* sandbox: Wrap the HealthService in basic tests.
* sandbox: Stop streaming the server health too.
* ledger-api-test-tool: Health check tests.
* Add a changelog entry for the health check endpoints.
CHANGELOG_BEGIN
- [Ledger API] Add healthcheck endpoints, conforming to the
`GRPC Health Checking Protocol <https://github.com/grpc/grpc/blob/master/doc/health-checking.md>`_.
It is always ``SERVING`` for now.
- [DAML Ledger Integration Kit] Add conformance test coverage for the
``grpc.health.v1.Health`` service.
CHANGELOG_END
* ledger-api-integration-tests: Increment the number of services.
* Apply suggestions from code review
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* sandbox: Use `AkkaBeforeAndAfterAll` in the HealthServiceSpec.
In an attempt to get it working on CI.
* sandbox: Change `dropRepeated` to `DropRepeated()`.
Keep it in one file.
* test-common: Use `Delayed.by` in `TimeBoundObserver`.
* test-common: Close the source when `TimeBoundObserver` completes.
* ./fmt.sh
That'll teach me not to `--no-verify` just because it's a merge commit.
* sandbox: Inline `HealthService.suppress`.
At some point it was being used twice.
* sandbox: Increase the timeout for HealthServiceSpec.
* sandbox: Reimplement HealthService using the Scala protobuf types.
* sandbox: Generate an Akka-compatible trait for the health service.
And refactor a lot of test code to make it easy to test.
* ledger-api-common: Move the HealthService here.
* rs-grpc-testing-utils: Publish to Maven.
* rs-grpc-testing-utils: Add Maven coordinates.
kvutils now supports dumping the ledger to a file via environment
variable: "KVUTILS_LEDGER_DUMP=/tmp/ledger.dump".
The integrity checker tool allows re-processing of ledger dumps to
validate that all submissions, state and log entries can be parsed
and processed. This forms the basis of kvutils data continuity test
suite.
Currently the checker strictly enforces that re-processed
outputs match the inputs exactly, which for now guarantees forwards
and backwards compatibility. The checker will be adapted when
forwards-incompatible changes or new kvutils message versions arise.
A select collection of ledger dumps will be maintained in a separate
repository (to be decided) and these dumps are to be validated by
the DAML CI.
* Update bazel-common to fix javadoc issues
Specifically, to fix the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel:7:1: in javadoc_library rule //ledger-api/rs-grpc-bridge:rs-grpc-bridge_javadoc:
Traceback (most recent call last):
File "/home/aj/tweag.io/da/da-bazel-1.1/ledger-api/rs-grpc-bridge/BUILD.bazel", line 7
javadoc_library(name = 'rs-grpc-bridge_javadoc')
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/com_github_google_bazel_common/tools/javadoc/javadoc.bzl", line 27, in _javadoc_library
dep.java.transitive_deps
object of type 'JavaSkylarkApiProvider' has no field 'transitive_deps'
```
* Define Maven deps using rules_jvm_external
* Pin artifacts
* Remove bazel-deps generated targets
* Remove bazel-deps
* Switch to rules_jvm_external targets
* update bazel documentation
* pom_file: There are no more bazel-deps targets
* BAZEL-JVM.md `maven_install` typo
