Commit Graph

99 Commits

Author SHA1 Message Date
Stephen Compall
2e671e4e5d
non-empty newtypes (#8516)
* non-empty newtypes

* an operation

* add some map/set operations and make everything compile on 2.12 and 2.13

* +-: and :-+, with compatibility layer; docs

* move to nonempty package; add aliases for cons/snoc; fix SeqOps aliases

* ensure 2.12 aliases are inferrable

* groupBy1 and toList, use to prove uniqueSets's invariants

* prove immutability first

* matching variance in aliases

* prove the return property of uniqueSets, and use the proof

* tests for NonEmpty API

* rename sci alias to imm

* move RefinedOps to more obvious location

* more docs

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused imports

* illustrate the scala.collection.Seq problem

* ideas for extension

* tests for toF

* tests for +-:

* explain difference with OneAnd
2021-02-22 08:54:26 -05:00
Moritz Kiefer
7ae28beff9
More Scala 2.13 cleanup (#8855)
This fixes Scaladoc and our pom file generation.

It also clears up the confusing error around gatling and removes a
redundant dependency on sbt (no idea why we had that in the first
place) both of which resulted in Scala 2.12 dependencies in our 2.13
lockfile which is obviously bad.

With this, we should now be ready to publish Scala 2.13 artifacts once
the ledger API test tool PR lands.

changelog_begin
changelog_end
2021-02-16 09:39:16 +00:00
Kamil Bożek
926fb5997a
Improve logging in ledger API server [DPP-231] (#8676)
* Added logging incoming requests in API services: Submission, ConfigManagement, PackageManagement, ParticipantPruning and PartyManagement

CHANGELOG_BEGIN
- Logging incoming requests in API services
CHANGELOG_END

* Logging transactions and transaction trees streamed by the ApiTransactionService

CHANGELOG_BEGIN
- Logging transactions and transaction trees returned by the ApiTransactionService
CHANGELOG_END

* Logging storing the db in JdbcLedgerDao

* Changed log severity

* Factored out logging util to ContextualizedLogger

* Review improvements

* Removed unused import

* Formatted changes

* Logging completions stream items

* Fixed log message

* Logging complete transactions and transaction trees

* Removed duplicated keys from logging context

* Formatted changes

* Reduced logging for completions and transactions

* Removed redundant log

* Removed update* prefix in Indexer's logging context

* Minor improvement

* Minor improvement
2021-02-15 11:45:49 +01:00
Stefano Baghino
42e071f2c2
Add //libs-scala/doobie-slf4j and replace usages of jdkLogHandler (#8731)
The jdkLogHandler provided by Doobie exists purely as an example and the library
itself does not recommend using it in production.

Note that this slightly changes the runtime behavior, logging successful queries
at debug level rather then info. The message itself is preserved from the original
MIT-licensed example.

This uses Slf4j as most of our components, instead of java.util.logging.

changelog_begin
[HTTP JSON API] The server now logs successful queries at debug level
instead of info
[Trigger Service] The server now logs successful queries at debug level
instead of info
changelog_end
2021-02-03 10:45:05 +00:00
Stefano Baghino
c541df2c38
Restrict extensibility of ServerResourceOwner (#8660)
* Restrict extensibility of ServerResourceOwner

Following https://github.com/digital-asset/daml/pull/8649 this commit
fixes the last open comment from https://github.com/digital-asset/daml/pull/8604
as explained in the comment https://github.com/digital-asset/daml/pull/8604#issuecomment-766620042

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/8660#discussion_r565946621
2021-01-28 10:47:28 +00:00
Samir Talwar
48daf33586
resources: Add Resource#transform, and mirror Resource#transform{,With} in ResourceOwner. (#8649)
* resources: Add `Resource#transform`.

CHANGELOG_BEGIN
CHANGELOG_END

* resources: Fix some Scaladoc warnings.

* resources: Add `transform` and `transformWith` to ResourceOwners.
2021-01-27 17:58:13 +00:00
Samir Talwar
ecab863d68
resources: Sometimes, Future#andThen is simpler. (#8650)
CHANGELOG_BEGIN
CHANGELOG_END
2021-01-27 17:58:06 +00:00
Samir Talwar
694a7213d1
resources: Remove the default factories. (#8645)
* resources: Wrapper constructors around the normal Resource types.

* resources: Factor out `ResourceFactories#apply`.

* resources: Remove the global factories.

They're no longer necessary, and could cause confusion. Users should use
their own factory object, not a generic one.

CHANGELOG_BEGIN
CHANGELOG_END
2021-01-27 15:56:29 +00:00
Samir Talwar
d2e0cb2689
resources: Less context. (#8643)
* resources: Extract out `Resource.nest` into its own class.

* resources: Avoid needing a context for "pure" resources.

CHANGELOG_BEGIN
CHANGELOG_END

* resources: Turns out the higher kinds import fails in Scala 2.13.
2021-01-27 12:50:23 +00:00
Stefano Baghino
ed89e58a76
Enable ledger-api-test-tool to be used as a library (#8607)
* Enable ledger-api-test-tool to be used as a library

This is the third of four PRs in which 6ea70c4b45
has been broken up to facilitate review.

The endgame is to have the non-repudiation prototype merged. The
Ledger API test tool will be used to check that the proxy can be
safely put in front of a conformant ledger implementation without
causing conformance tests to not pass. The ability to use is as
a library facilitates the process, ensuring we don't have to set
up three running processes in Bazel (the participant, the proxy
and the test tool).

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/8607#discussion_r563685495

* Address https://github.com/digital-asset/daml/pull/8607#discussion_r563688955

* Address https://github.com/digital-asset/daml/pull/8607#discussion_r563692051
2021-01-25 14:12:50 +00:00
Stefano Baghino
7de057192c
Enhance grpc-test-utils and grpc-reverse-proxy to use Resources (#8606)
* Enhance grpc-test-utils and grpc-reverse-proxy to use Resources

This is the second of four PRs in which 6ea70c4b45
has been broken up to facilitate review.

The endgame is to have the non-repudiation prototype merged. The
grpc-test-utils and grpc-server-proxy libraries have been
enhanced to use the ResouceOwner/Resource abstraction to handle
the lifecycle of components, making resource management easier
when testing.

changelog_begin
changelog_end

* Solve shadowing issue with more specific imports

* Address https://github.com/digital-asset/daml/pull/8606#discussion_r563628581
2021-01-25 13:27:48 +01:00
Stefano Baghino
fc69db02b4
Add a library that defines Resource factory methods for gRPC servers and channels (#8604)
* Add a library that defines Resource factory methods for gRPC servers and channels

This is the first of four PRs in which 6ea70c4b45
has been broken up to facilitate review.

The endgame is to have the non-repudiation prototype merged. These
factory methods have been used to make resource management easier,
specifically as part of testing.

changelog_begin
changelog_end

* Let warning log output in tests

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Address https://github.com/digital-asset/daml/pull/8604#discussion_r562634032

* Address https://github.com/digital-asset/daml/pull/8604#discussion_r562634865

* Address https://github.com/digital-asset/daml/pull/8604#discussion_r562635148

* Address https://github.com/digital-asset/daml/pull/8604#discussion_r562635684

* Add resources-grpc to release/artifacts.yaml

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2021-01-25 10:32:36 +01:00
Stefano Baghino
9ed787cb3d
Add grpc-reverse-proxy with optional server interceptors (#8481)
* Factor 'withServices' out of ServerReflectionClientSpec

* Add grpc-reverse-proxy with optional server interceptors

changelog_begin
changelog_end

* Switch from Protobuf byte strings to native byte arrays

* Fix year in copyright headers

* Address https://github.com/digital-asset/daml/pull/8481#discussion_r555915935

* Add missing maven_coordinates tag

* Address https://github.com/digital-asset/daml/pull/8481#discussion_r555916684
2021-01-13 15:41:30 +01:00
Moritz Kiefer
22ce940954
Upgrade scalafmt and enable trailing commas (#8437)
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.

As announced, this will be merged on Saturday to avoid too many conflicts.

changelog_begin
changelog_end
2021-01-09 11:37:37 +01:00
Stefano Baghino
c44a62afd0
Add grpc-server-reflection-client (#8438)
* Add grpc-server-reflection-client

Adds a small library designed to return in a simple package (a Future[Set[...]])
a full description of all services available on a gRPC server (if it exposes the
standard reflection endpoint), which is enough to easily generate a full reverse
proxy.

changelog_begin
changelog_end

* Address https://github.com/digital-asset/daml/pull/8438#discussion_r553900804

* Address https://github.com/digital-asset/daml/pull/8438#discussion_r553902902

* Address https://github.com/digital-asset/daml/pull/8438#discussion_r553904143
2021-01-08 16:33:25 +00:00
Gary Verhaegen
a925f0174c
update copyright notices for 2021 (#8257)
* update copyright notices for 2021

To be merged on 2021-01-01.

CHANGELOG_BEGIN
CHANGELOG_END

* patch-bazel-windows & da-ghc-lib
2021-01-01 19:49:51 +01:00
Moritz Kiefer
cb57ad374c
Port //daml-lf/data to Scala 2.13 (#8321)
* Port //daml-lf/data to Scala 2.13

changelog_begin
changelog_end

* factor common ImmArraySeq code to version-agnostic file

- ImmArraySeq itself is agnostic; the 2.12 and 2.13 versions contain
  implementation mixins/superclasses for parts that must be specific.  The 2.13
  version will collapse into the agnostic version when 2.12 support is no longer
  desired

* factor common InsertOrdMap code to version-agnostic file

- InsertOrdMap itself is agnostic; the 2.12 and 2.13 versions contain
  implementation mixins/superclasses for parts that must be specific.  The 2.13
  version will collapse into the agnostic version when 2.12 support is no longer
  desired

* factor common InsertOrdSet code to version-agnostic file

- InsertOrdSet itself is agnostic; the 2.12 and 2.13 versions contain
  implementation mixins/superclasses for parts that must be specific.  The 2.13
  version will collapse into the agnostic version when 2.12 support is no longer
  desired

* factor Map removal

* Move ImmArraySeq back into ImmArray

changelog_begin
changelog_end

* Type assertion instead of symbol

changelog_begin
changelog_end

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-12-18 13:20:15 +01:00
Moritz Kiefer
1b32044a74
Build //libs-scala/... on 2.13 (#8282)
* Build //libs-scala/... on 2.13

One test is unfortunately disabled at the moment since I utterly
failed to figure out why I get a ClassNotFoundException on 2.13.

changelog_begin
changelog_end

* Copyright headers

changelog_begin
changelog_end

* I can’t bazel today

changelog_begin
changelog_end

* Apply suggestions from code review

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

* Update libs-scala/resources/src/main/2.13/com/daml/resources/UnitCanBuildFrom.scala

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

* No split on view

changelog_begin
changelog_end

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-12-14 17:36:53 +00:00
Moritz Kiefer
d24ef94f36
Add a Scala 2.13 build pipeline (#8271)
* Add a Scala 2.13 build pipeline

This adds initial support for multiple Scala versions controlled via
the DAML_SCALA_VERSION env var and a CI job to make sure we don’t
regress. For now we only test //libs-scala/ports/... which seemed like
the easiest starting point I could find. We can incrementally expand
that over time.

changelog_begin
changelog_end

* Document pinning

changelog_begin
changelog_end

* Address review comments

changelog_begin
changelog_end
2020-12-14 12:42:26 +01:00
Moritz Kiefer
76b6fd86fb
Upgrade Scala dependencies for 2.13 compatibility (#8268)
* Upgrade Scala dependencies for 2.13 compatibility

This upgrades a bunch of Scala libraries to versions that have 2.13
support. There are two libraries that are still missing:

- diffson, this has a new version but with significant breaking
  changes and it is only used in Naigator console which I hope to kill
  before I have to worry about this.

- ai.x:diff, this is used in the ledger API test tool. The library is
  abondened but there are a few alternatives.

changelog_begin
changelog_end

* Fix pureconfig

changelog_begin
changelog_end

* Fix Navigator

changelog_begin
changelog_end
2020-12-14 09:59:00 +01:00
Moritz Kiefer
7e05dc7932
Upgrade rules-scala and scalatest (#8187)
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.

This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.

Apologies for the giant PR, I don’t see a way to keep it smaller.

changelog_begin
changelog_end
2020-12-08 06:59:23 +01:00
Samir Talwar
052f69cde9
ledger-api: Use proto_jars, and publish Protobuf sources separately from the Scala classes. [KVL-714] (#8091)
* ledger-api: Use `proto_jars`.

CHANGELOG_BEGIN
- [Ledger API] The Scala JARs containing the gRPC definitions no longer
  contain the *.proto files used to generate the ScalaPB-based classes.
CHANGELOG_END

* Create a source JAR for *.proto files in `proto_jars`.

* ledger-api: Publish the protobuf sources as "ledger-api-proto".

CHANGELOG_BEGIN
- [Ledger API] The *.proto files containing the gRPC definitions are now
  provided by a new Maven Central artifact, with the group "com.daml"
  and the artifact name "ledger-api-proto".
CHANGELOG_END

* release: We don't need the "main-jar" option.

* Bazel: Proto JARs will always have a Maven artifact suffix.

* Bazel: Simplify Protobuf source file TAR and JAR targets.

* Bazel: Extract out Protobuf functions.
2020-11-27 17:14:48 +00:00
Samir Talwar
87eee303e3
resources: Release sequenced resources in parallel. (#7991)
* resources: Release sequenced resources in parallel.

This isn't used much, but has been bothering me for a while. While we
acquire the resources in parallel, we used to release them sequentially.
This reimplements `sequence` so they're released all at once.

CHANGELOG_BEGIN
CHANGELOG_END

* resources: Drop an unnecessary `.map`.

* resources: Fix the Scaladoc for `sequence`.
2020-11-18 09:33:44 +00:00
Andreas Herrmann
90dc3a5669
Implement token refresh in auth middleware (#7981)
* Obtain refresh token from Auth0

Auth0 requires the `offline_access` scope to be set to return a refresh
token.

See https://auth0.com/docs/tokens/refresh-tokens/get-refresh-tokens

Additionally, the `audience` claim needs to be set to obtain a JWT
access token and a refresh token.

See https://auth0.com/docs/tokens/refresh-tokens

changelog_begin
changelog_end

* Implement refresh endpoint on auth middleware

Following the refresh spec [1] and Auth0 documentation [2].

[1]: https://tools.ietf.org/html/rfc6749#section-6
[2]: https://auth0.com/docs/tokens/refresh-tokens/use-refresh-tokens

* Adapt Auth0 example configuration

Ignore any requests outside the ledger-api audience.

Don't throw on missing query fields. Otherwise the unhandled exception
would prevent unrelated requests from succeeding. E.g. token refresh
requests would always fail.

* Forward unauthorized/forbidden response on refresh

* re-use precomputed token payload

* Implement token refresh in auth test server

Reuses the association between authorization code and token payload to
associate refresh tokens and token payload.

Adds an expiry to the generated token to make tokens distinguishable
across refresh.

* obtain refresh token in test client

* Test auth server refresh token

* auth test server clock configurable

The clock used to define token expiry is configurable

* Override default clock in test fixture

* implement an adjustable clock

* Test token refresh with adjustable clock

* Test token expiry on /auth backend

* Test case for auth middleware /refresh endpoint

* handle malformed code/refresh token in auth server

* Forward client errors on middleware refresh

* Test middleware refresh failure

* Clarify meaning of offline accesss

* Remove redundant testing only comment

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-11-17 12:06:42 +00:00
Robin Krom
5bfff4e9ba
sandbox: fail on already existing port-file. (#7929)
Fixes #7806. This aligns the port file behaviour of the sandbox with the
HTTP JSON API.

CHANGELOG_BEGIN
CHANGELOG_END
2020-11-17 11:08:37 +01:00
Stephen Compall
b35c9fcb3d
slow down trigger on RESOURCE_EXHAUSTED failures (#7820)
* restate the submit stage as a Flow and derived Sink

* take submit out of the trigger-to-submit flow

* type for the failures produced directly by command submission

* directly connect the msgSource failure queue to the submitter output

* parens

* slow down submission as we exceed max parallel submissions

* restricting alterF so it will be usable with ConcurrentMap

* disable buffer for the delay

* split out the delay function

* drafting a retry loop

* degenerate test for retry loop, factoring the forAllFuture utility

* map input to retrying properly

* make retrying accessible to tests

* test happy path and fix off-by-one

* further tests for retrying

* reveal that elements can get lost

* more determinism in test

* let failures block further elements from being attempted

- Previously failures would go into a separate queue, where they awaited expiry
  of their delay and further initial upstream elements were given their first
  tries.  However, closing the upstream could mean that queue was dropped, and
  detecting that situation is not trivial.  So, instead, we don't use a separate
  queue.

* plug retrying into the trigger submission flow

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove throttle; pendingCommandIds may leak

* report random parameter on failure

* revert comment about throttling

* explanation for fail in the error queue

- suggested by @cocreature; thanks
2020-11-10 21:36:48 +00:00
Andreas Herrmann
9758b2f85b
Check trigger dao migrations digests (#7908)
* Check the trigger dao migrations digest

Following the example of the corresponding ledger on SQL tests.

The digests had to be updated as both of them had gone out of sync.
The init digest presumably due to the change in #7226 and the one for
adding the access token during review of #7890.

changelog_begin
changelog_end

* define abstract migrations test

* Use abstract migrations test in trigger service tests

* use abstract migrations test in ledger on SQL

* Retain check for number of .sql resources

* Factor out the hash-migrations script

* Consistent shell settings

Addressing review comment

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-11-06 09:20:32 +00:00
Samir Talwar
2801c22cab
timer-utils: Widen the bounds for duration checks. (#7760)
I saw a failure in CI due to timer fluctuation.

> 94 milliseconds was not around 100 milliseconds [95, 1000)

This widens the lower bound from 5ms to 20ms to be safe.

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-21 12:19:27 +02:00
Samir Talwar
7f679b9896
resources: Customizable contexts. (#7678)
* resources: Move builders into //ledger/ledger-resources.

Keep the actual constructors in a trait, but instantiate it when working
with ledger code.

This allows us to later introduce an extra "context" type parameter to
ResourceOwner.

* resources-akka: Move the builders in to //ledger/ledger-resources.

* resources: Introduce an abstract `Context` parameter for owners.

This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.

This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.

Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.

CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
  `Context`, which is filled in by the corresponding `Context` class in
  the _ledger-resources_ dependency. This allows us to pass extra
  information through resource acquisition.
CHANGELOG_END

* ledger-resources: Move `ResourceOwner` here from `resources`.

* ledger-resources: Remove dependencies from outside //ledger.

* ledger-resource: Wrap the acquisition execution context in `Context`.

So we can add a logging context to it.

* resources: Pass the Context, not the ExecutionContext, to Resource.

* Avoid importing `HasExecutionContext`.

* ledger-resources: Publish to Maven Central.

* resources: Make the small changes suggested by @stefanobaghino-da.

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* ledger-resources: Pull out a trait for test resource contexts.

Saves a few lines of code.

* Restore some imports that were accidentally wildcarded.

* resources: Replace an `implicit def` with a couple of imports.

* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.

Try and use the right execution context where possible.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-10-20 09:26:28 +00:00
Stephen Compall
bafde51752
add silent_annotations option to da_scala bazel functions (#7668)
* add silent_annotations option to da scala bazel functions

* use silent_annotations for several scala targets

* use silencer_plugin instead when the lib isn't used

* use silent_annotations for several more scala targets

* use silencer_lib for strange indirect requirement for running tests

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* silent_annotations support for scaladoc
2020-10-13 15:44:16 +00:00
Stephen Compall
1a2afd5266
upgrade to Scala 2.12.12 from 2.12.11 (#7661)
* upgrade bazel settings to scala 2.12.12

* upgrade nix scala tool to scala 2.12.12

* upgrade silencer references to scala 2.12.12

* repin for scala 2.12, silencer, wartremover upgrades

* remove numerous occurrences of unused silencer now spotted

* update Scala version in our bazel notes

CHANGELOG_BEGIN
CHANGELOG_END

* update compatibility maven_install.json to match compatibility WORKSPACE
2020-10-13 08:42:14 -04:00
Samir Talwar
aec25d2a49
ledger-on-sql: Use tagged execution contexts and data sources in Database. (#7525)
* metrics: Support tagged Futures when timing.

* ledger-on-sql: Use tagged execution contexts in `Database`.

We have to deal with multiple execution contexts in `Database`. This
makes it possible to use them implicitly, which is much cleaner.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-sql: Simplify `Database` a little.

* ledger-on-sql: Make the connection pool implicit.

* ledger-on-sql: Move the execution context into the connection pool.

* ledger-on-sql: Make connection pools more implicit.

* ledger-on-sql: Use the `sc` prefix for `scala.concurrent`.

* ledger-on-sql: Remove an unnecessary import.
2020-10-02 15:16:05 +00:00
Samir Talwar
f0c1eb207c
concurrent: Tag DirectExecutionContext. (#7517)
* concurrent: Tag DirectExecutionContext.

1.  Tag `DirectExecutionContext` as `ExecutionContext[Nothing]`, thereby
    stating that it works for any tagged `Future`.
2.  Move `DirectExecutionContext` to the _libs-scala/concurrent_
    library, as it requires it and it's tiny.

CHANGELOG_BEGIN
CHANGELOG_END

* concurrent: Fix the privacy of `DirectExecutionContextInternal`.

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-09-29 17:23:57 +00:00
Stephen Compall
d48e9d251d
ExecutionContext[EC] phantom for control, Future[EC, A] (#7347)
* add phantom-tagged ExecutionContext and Future to scala-utils concurrent package

* many new operations for Futures

* Future, ExecutionContext combinators from porting ledger-on-sql

- picked from 546b84ab9cdf4de2d93ec5682bdee6cfd6b385f8

* move Future, ExecutionContext companions into normal package

* lots of new docs

* many new Future utilities

* working zipWith

* tests for ExecutionContext resolution, showing what will be picked under different scenarios

* even more tests for ExecutionContext resolution

* tests showing some well-typed and ill-typed Future combinator usage

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* missed scalafmt

* one more doc note

* split concurrent package to concurrent library

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-09-17 07:36:50 +00:00
Leonid Shlyapnikov
fe63816431
Write report summary in text format (#7407)
* write summary.txt

changelog_begin
changelog_end

* make `Response time distribution` brackets configurable
2020-09-16 09:31:54 -04:00
Leonid Shlyapnikov
4fbde3c672
Open-sourcing Gatling statistics reporter (#7325)
* Open sourcing gatling statistics reporter

Running gatling scenarios with `RunLikeGatling` from libs-scala/gatling-utils

* cleaning up

* Replace "\n" with System.lineSeparator

so the formatting test cases pass on windows

* Testing DurationStatistics Monoid laws

* Renaming RunLikeGatling -> CustomRunner
2020-09-11 13:39:05 +00:00
Stephen Compall
0b72b6f716
more trigger service logs; tagged LoggingContext (#7205)
* get a LoggingContext into the TriggerRunnerImpl

* make some implicits more implicitly scoped and explicitly ascribed

* make some private/final markings

* most of JsonFormat[Identifier] is in companion

* experimental LoggingContext with phantom type

* ActorContext#log isn't really doing that much

* more details of LoggingContextOf

* make LoggingContextOf compile

* add trigger message logging, yet without context

* fix parent compile errors

* use Config as the phantom for its own logging extensions

* LocalDateTimeFormat cleanup

* switch TriggerRunner to contextual logging

* add trigger definition ID to logs

* log trigger-submitted commands, fix trigger test compile

* log trigger stopping and DAR uploads

* add context to PostStop/PreRestart logs

* add changelog

CHANGELOG_BEGIN
- [Triggers] More detailed logging of trigger actions and trigger service actions.
  See `issue #7205 <https://github.com/digital-asset/daml/pull/7205>`_.
CHANGELOG_END

* missed copyright header

* switch to Unit, scala/bug#9240 fixed
2020-09-03 17:47:09 +00:00
Samir Talwar
c8b10091db
participant-integration-api: Never use a delay of zero. (#7175)
* participant-integration-api: Never use a delay of zero.

If `akka.pattern.after` is passed a delay of zero, it will execute the
body synchronously, potentially leading to a stack overflow error.

This error was observed in tests.

CHANGELOG_BEGIN
CHANGELOG_END

* timer-utils: Add tests for Delayed.Future.

* timer-utils: Add tests for RetryStrategy.

* timer-utils: Remove duplication in RetryStrategy tests.

* timer-utils: Allow for more wiggle room in the RetryStrategy tests.

* timer-utils: Fail after retrying the correct number of times.

* timer-utils: Ensure we don't overflow the stack in RetryStrategy.

* timer-utils: Reject a negative number of retry attempts.
2020-08-19 10:43:31 +00:00
Samir Talwar
27f76c4386
Use Future.unit instead of Future.successful(()). (#7080)
And in one instance, `Resource.unit`.

I just think it's easier to read. Too many parentheses make Samir a dull
boy.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-11 09:18:50 +00:00
Stephen Compall
96624a7677
use -Ywarn-unused for all Scala code (#6907)
* add -Ywarn-unused to all scalac options

* remove some unused arguments

* remove some unused definitions

* remove some unused variable names

* suppress some unused variable names

* changeExtension doesn't use baseName

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* work around no plugins in scenario interpreter perf tests

* remove many more unused things

* remove more unused things, restore some used things

* remove more unused things, restore a couple signature mistakes

* missed import

* unused argument

* remove more unused loggingContexts

* some unused code in triggers

* some unused code in sandbox and kvutils

* some unused code in repl-service and daml-script

* some unused code in bindings-rxjava tests

* some unused code in triggers runner

* more comments on silent usages

- suggested by @cocreature; thanks

* fix missing reference in TestCommands

* more unused in triggers

* more unused in sandbox

* more unused in daml-script

* more unused in ledger-client tests

* more unused in triggers

* more unused in kvutils

* more unused in daml-script

* more unused in sandbox

* remove unused in ledger-api-test-tool

* suppress final special case for codegen unused warnings

.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
      implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
                                                         ^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
      implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
                                                               ^

* one more unused in daml-script

* special scaladoc rules may need silencer, too

* unused in compatibility/sandbox-migration

* more commas, a different way to `find`

- suggested by @remyhaemmerle-da; thanks
2020-08-07 13:16:09 -04:00
Stephen Compall
294068ae1d
remove unused definitions, params, args from libs-scala and test tool Scala code (#7051)
* remove unused definitions, params, args from libs-scala and test tool Scala code

CHANGELOG_BEGIN
CHANGELOG_END

* collapse `Some(_) | None` into _

- suggested by @SamirTalwar-DA; thanks

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-08-06 18:26:22 +00:00
Stephen Compall
3196ef7575
fix flaky 'restart triggers after shutdown' test (#6926)
* reenable 'restart triggers after shutdown'

CHANGELOG_BEGIN
CHANGELOG_END

* wait for everything to shut down before completing a withTriggerService fixture

- similar to a change to HttpServiceFixture.withHttpService in #4593,
  but without the suppression of shutdown errors

* label the WithDb tests

* in CI, test only 'recover packages after shutdown', 50 times

* experiment: Process#destroy appears to be async

* is it in the in-between period?

* partial -> total

* replace some booleans with assertions for better error reporting

* make triggerLog concurrent

* close channel and file in other error cases for port locking

- suggested by @leo-da; thanks

* use port locking instead of port 0 for trigger service fixtures

* destroy one service at a time

* missed continuation in build script

* use assertion language for "restart triggers with update errors"

* Revert "is it in the in-between period?"

This reverts commit 211ebfe9d2.

* use better assertion language for "restart triggers with update errors"

* restore full CI build
2020-08-04 08:15:08 -04:00
Stephen Compall
3a0a3228ec
fix null pointer in PortLock#lock (#6941)
java.lang.NullPointerException:
  at com.daml.ports.PortLock$Locked.unlock(PortLock.scala:55)
  at com.daml.ports.PortLock$.lock(PortLock.scala:41)
  at com.daml.ports.LockedFreePort$.find(LockedFreePort.scala:15)
  at com.daml.lf.engine.trigger.TriggerServiceFixture$.$anonfun$withTriggerService$1(TriggerServiceFixture.scala:65)

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-31 10:59:39 -04:00
Stefano Baghino
e972872128
Properly use LoggingContext in Participant Server (#6924)
* Properly use LoggingContext in Participant Server

Fixes #6837

Spreads usage of LoggingContext and ContextualizedLogger throughout the participant server.

changelog_begin
[Sandbox/Integration Kit] We have enriched the contextual information
exposed by the Ledger API server. You should note richer logging information,
which can be read either via unstructured or structured logging frameworks.
A paragraph on how to configure structured logging has been added to the docs.
For more on the issue, see https://github.com/digital-asset/daml/issues/6837.
changelog_end

* Make everything compile

* Leave only actual commands on trace level on submission

* Shorter log messages on submission

* Don't add values down the call stack

* Document good practices regarding the logging context

* Add context to write service implementations

* Add logging context to tests

* Document logging for DAML on SQL

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463503013

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463508117

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463515665

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463513157

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463516359

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463516895

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463518813

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463520210

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463521501

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463521593

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525453

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525560

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525672

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525742

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463526837

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527054

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527523

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527814

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527958

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527900

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527997

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463528050

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463542877

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543051

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543614

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543692
2020-07-31 12:40:13 +00:00
Leonid Shlyapnikov
05d49b37c3
Introduce //libs-scala/scala-utils (#6935)
* Moving `Statements.discard` from //ledger-server/http-json into //libs-scala/scala-utils

changelog_begin
changelog_end

* Add new module to the published artifacts

* `com.daml.scalautil` instead of `com.daml.scala.util`

@S11001001: That's because if this is in classpath and you import com.daml._,
you have a different scala in scope than the one you expect.
2020-07-30 13:51:07 -04:00
Samir Talwar
99463577af
triggers + ports: Lock free ports before binding. (#6911)
* triggers: Use `FreePort.find()`.

* ports: Move `LockedFreePort` from postgresql-testing for reuse.

* triggers: Use `LockedFreePort` to avoid race conditions.

* ports + triggers: Move common port testing into the ports library.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-29 14:29:55 +00:00
Samir Talwar
d6fc2bbb58
ledger-api-client + participant-integration-api: Increase the default maximum inbound error size, and truncate errors well before that. (#6807)
* participant-integration-api: `GrpcServerOwner` -> `GrpcServer.Owner`.

Mostly so I can create a test class named `GrpcServerSpec`.

* ports: Move the free port search from postgresql-testing.

* participant-integration-api: Test the basics of GrpcServer.

This uses the HelloService to make sure the server behaves normally.

* ledger-api-client: Extract out channel configuration from LedgerClient.

So we can test it independently of the LedgerClient itself.

* ledger-api-client: Increase the default maximum inbound header size.

Increased from 8 KB to 1 MB.

* participant-integration-api: Reduce the maximum error message size.

Truncate GRPC error descriptions to 256 KB.

* participant-integration-api: Use `Port.Dynamic` instead of `FreePort`.

In tests.

* participant-integration-api: Explicit null checks when they're shorter.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* ledger-api-client: Reduce the max inbound message size back to 8 KB.

And reduce the maximum size of an error description pushed out by the
server accordingly.

CHANGELOG_BEGIN
- [Integration Kit] Truncate GPRC error messages at 4 KB. This ensures
  that we won't trigger a protocol error when sending errors to the
  client.
CHANGELOG_END

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-07-21 15:50:33 +00:00
Stephen Compall
a51d0db8ff
set scalac -Xsource:2.13 -Ypartial-unification globally (#6469)
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts

* add now-referenced scalaz-core where needed

* work around bad type signatures in scalatest Aggregating, Containing

* unused Any suppression

* work around bad partial-unification wrought by type alias

* remove unused Conversions import

- not required in 4f68cfc480 either, so unsure how it's survived this long

* work around Future.traverse; remove unused show import

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused bounds

* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed

* longer comment on what the options do

- suggested by @stefanobaghino-da; thanks

* forget Future.traverse, just use scalaz, it knows how to do this
2020-06-24 16:51:24 -04:00
Stephen Compall
15350a7bc2
disable warts.Any and remove most suppressions (#6132)
* disable Any wart

* first pass removal of Any suppressions for false positives

* second pass removal of Any suppressions for false positives

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* third pass removal of Any suppressions for false positives

* fourth pass removal of Any suppressions for false positives

* reformat newly single-suppressions into single lines

- suggested by @SamirTalwar-DA; thanks
2020-05-28 16:53:41 +00:00
Stephen Compall
1be6b3640f
equalz Scalatest matcher in new daml-lf/scalatest-tools library (#1200)
* equalz Scalatest matcher in new daml-lf/scalatest-tools library

* equalz typing tests

* a 'should' replacing design

* a 'MatcherFactory1' design

- this fails because the TC parameter should be a type member to avoid
  scala/bug#5075 but it is not

* MatcherFactory1 with chained Lub+Equal typeclass

- requires partial-unification at point of use, which is not great

* LubEqual's extra tparam is probably unneeded

* better LtEqual

* demonstrate that HK LubEqual's resolve with DMT should + MatcherFactory

* remove unneeded 3rd param from LubEqual, again

* update dependency specs and license headers

* allow use with should, shouldNot in some cases, preserving the shouldx/shouldNotx alternatives

* move Equalz to libs-scala/scalatest-utils

* rename bzl targets and place in com.daml.scalatest package

* add scalatest-utils to release

* move *SpecCheckLaws, Unnatural to scalatest-utils

* missed scalacheck dep in scalatest-utils

* downstreams of *SpecCheckLaws now get them from scalatest-utils

* test equal-types case as well

* update LF documentation

CHANGELOG_BEGIN
CHANGELOG_END

* whitespace error
2020-05-27 17:25:43 -04:00