Commit Graph

79 Commits

Author SHA1 Message Date
Moritz Kiefer
7e05dc7932
Upgrade rules-scala and scalatest (#8187)
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.

This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.

Apologies for the giant PR, I don’t see a way to keep it smaller.

changelog_begin
changelog_end
2020-12-08 06:59:23 +01:00
Samir Talwar
052f69cde9
ledger-api: Use proto_jars, and publish Protobuf sources separately from the Scala classes. [KVL-714] (#8091)
* ledger-api: Use `proto_jars`.

CHANGELOG_BEGIN
- [Ledger API] The Scala JARs containing the gRPC definitions no longer
  contain the *.proto files used to generate the ScalaPB-based classes.
CHANGELOG_END

* Create a source JAR for *.proto files in `proto_jars`.

* ledger-api: Publish the protobuf sources as "ledger-api-proto".

CHANGELOG_BEGIN
- [Ledger API] The *.proto files containing the gRPC definitions are now
  provided by a new Maven Central artifact, with the group "com.daml"
  and the artifact name "ledger-api-proto".
CHANGELOG_END

* release: We don't need the "main-jar" option.

* Bazel: Proto JARs will always have a Maven artifact suffix.

* Bazel: Simplify Protobuf source file TAR and JAR targets.

* Bazel: Extract out Protobuf functions.
2020-11-27 17:14:48 +00:00
Samir Talwar
87eee303e3
resources: Release sequenced resources in parallel. (#7991)
* resources: Release sequenced resources in parallel.

This isn't used much, but has been bothering me for a while. While we
acquire the resources in parallel, we used to release them sequentially.
This reimplements `sequence` so they're released all at once.

CHANGELOG_BEGIN
CHANGELOG_END

* resources: Drop an unnecessary `.map`.

* resources: Fix the Scaladoc for `sequence`.
2020-11-18 09:33:44 +00:00
Andreas Herrmann
90dc3a5669
Implement token refresh in auth middleware (#7981)
* Obtain refresh token from Auth0

Auth0 requires the `offline_access` scope to be set to return a refresh
token.

See https://auth0.com/docs/tokens/refresh-tokens/get-refresh-tokens

Additionally, the `audience` claim needs to be set to obtain a JWT
access token and a refresh token.

See https://auth0.com/docs/tokens/refresh-tokens

changelog_begin
changelog_end

* Implement refresh endpoint on auth middleware

Following the refresh spec [1] and Auth0 documentation [2].

[1]: https://tools.ietf.org/html/rfc6749#section-6
[2]: https://auth0.com/docs/tokens/refresh-tokens/use-refresh-tokens

* Adapt Auth0 example configuration

Ignore any requests outside the ledger-api audience.

Don't throw on missing query fields. Otherwise the unhandled exception
would prevent unrelated requests from succeeding. E.g. token refresh
requests would always fail.

* Forward unauthorized/forbidden response on refresh

* re-use precomputed token payload

* Implement token refresh in auth test server

Reuses the association between authorization code and token payload to
associate refresh tokens and token payload.

Adds an expiry to the generated token to make tokens distinguishable
across refresh.

* obtain refresh token in test client

* Test auth server refresh token

* auth test server clock configurable

The clock used to define token expiry is configurable

* Override default clock in test fixture

* implement an adjustable clock

* Test token refresh with adjustable clock

* Test token expiry on /auth backend

* Test case for auth middleware /refresh endpoint

* handle malformed code/refresh token in auth server

* Forward client errors on middleware refresh

* Test middleware refresh failure

* Clarify meaning of offline accesss

* Remove redundant testing only comment

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-11-17 12:06:42 +00:00
Robin Krom
5bfff4e9ba
sandbox: fail on already existing port-file. (#7929)
Fixes #7806. This aligns the port file behaviour of the sandbox with the
HTTP JSON API.

CHANGELOG_BEGIN
CHANGELOG_END
2020-11-17 11:08:37 +01:00
Stephen Compall
b35c9fcb3d
slow down trigger on RESOURCE_EXHAUSTED failures (#7820)
* restate the submit stage as a Flow and derived Sink

* take submit out of the trigger-to-submit flow

* type for the failures produced directly by command submission

* directly connect the msgSource failure queue to the submitter output

* parens

* slow down submission as we exceed max parallel submissions

* restricting alterF so it will be usable with ConcurrentMap

* disable buffer for the delay

* split out the delay function

* drafting a retry loop

* degenerate test for retry loop, factoring the forAllFuture utility

* map input to retrying properly

* make retrying accessible to tests

* test happy path and fix off-by-one

* further tests for retrying

* reveal that elements can get lost

* more determinism in test

* let failures block further elements from being attempted

- Previously failures would go into a separate queue, where they awaited expiry
  of their delay and further initial upstream elements were given their first
  tries.  However, closing the upstream could mean that queue was dropped, and
  detecting that situation is not trivial.  So, instead, we don't use a separate
  queue.

* plug retrying into the trigger submission flow

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove throttle; pendingCommandIds may leak

* report random parameter on failure

* revert comment about throttling

* explanation for fail in the error queue

- suggested by @cocreature; thanks
2020-11-10 21:36:48 +00:00
Andreas Herrmann
9758b2f85b
Check trigger dao migrations digests (#7908)
* Check the trigger dao migrations digest

Following the example of the corresponding ledger on SQL tests.

The digests had to be updated as both of them had gone out of sync.
The init digest presumably due to the change in #7226 and the one for
adding the access token during review of #7890.

changelog_begin
changelog_end

* define abstract migrations test

* Use abstract migrations test in trigger service tests

* use abstract migrations test in ledger on SQL

* Retain check for number of .sql resources

* Factor out the hash-migrations script

* Consistent shell settings

Addressing review comment

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-11-06 09:20:32 +00:00
Samir Talwar
2801c22cab
timer-utils: Widen the bounds for duration checks. (#7760)
I saw a failure in CI due to timer fluctuation.

> 94 milliseconds was not around 100 milliseconds [95, 1000)

This widens the lower bound from 5ms to 20ms to be safe.

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-21 12:19:27 +02:00
Samir Talwar
7f679b9896
resources: Customizable contexts. (#7678)
* resources: Move builders into //ledger/ledger-resources.

Keep the actual constructors in a trait, but instantiate it when working
with ledger code.

This allows us to later introduce an extra "context" type parameter to
ResourceOwner.

* resources-akka: Move the builders in to //ledger/ledger-resources.

* resources: Introduce an abstract `Context` parameter for owners.

This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.

This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.

Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.

CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
  `Context`, which is filled in by the corresponding `Context` class in
  the _ledger-resources_ dependency. This allows us to pass extra
  information through resource acquisition.
CHANGELOG_END

* ledger-resources: Move `ResourceOwner` here from `resources`.

* ledger-resources: Remove dependencies from outside //ledger.

* ledger-resource: Wrap the acquisition execution context in `Context`.

So we can add a logging context to it.

* resources: Pass the Context, not the ExecutionContext, to Resource.

* Avoid importing `HasExecutionContext`.

* ledger-resources: Publish to Maven Central.

* resources: Make the small changes suggested by @stefanobaghino-da.

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* ledger-resources: Pull out a trait for test resource contexts.

Saves a few lines of code.

* Restore some imports that were accidentally wildcarded.

* resources: Replace an `implicit def` with a couple of imports.

* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.

Try and use the right execution context where possible.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-10-20 09:26:28 +00:00
Stephen Compall
bafde51752
add silent_annotations option to da_scala bazel functions (#7668)
* add silent_annotations option to da scala bazel functions

* use silent_annotations for several scala targets

* use silencer_plugin instead when the lib isn't used

* use silent_annotations for several more scala targets

* use silencer_lib for strange indirect requirement for running tests

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* silent_annotations support for scaladoc
2020-10-13 15:44:16 +00:00
Stephen Compall
1a2afd5266
upgrade to Scala 2.12.12 from 2.12.11 (#7661)
* upgrade bazel settings to scala 2.12.12

* upgrade nix scala tool to scala 2.12.12

* upgrade silencer references to scala 2.12.12

* repin for scala 2.12, silencer, wartremover upgrades

* remove numerous occurrences of unused silencer now spotted

* update Scala version in our bazel notes

CHANGELOG_BEGIN
CHANGELOG_END

* update compatibility maven_install.json to match compatibility WORKSPACE
2020-10-13 08:42:14 -04:00
Samir Talwar
aec25d2a49
ledger-on-sql: Use tagged execution contexts and data sources in Database. (#7525)
* metrics: Support tagged Futures when timing.

* ledger-on-sql: Use tagged execution contexts in `Database`.

We have to deal with multiple execution contexts in `Database`. This
makes it possible to use them implicitly, which is much cleaner.

CHANGELOG_BEGIN
CHANGELOG_END

* ledger-on-sql: Simplify `Database` a little.

* ledger-on-sql: Make the connection pool implicit.

* ledger-on-sql: Move the execution context into the connection pool.

* ledger-on-sql: Make connection pools more implicit.

* ledger-on-sql: Use the `sc` prefix for `scala.concurrent`.

* ledger-on-sql: Remove an unnecessary import.
2020-10-02 15:16:05 +00:00
Samir Talwar
f0c1eb207c
concurrent: Tag DirectExecutionContext. (#7517)
* concurrent: Tag DirectExecutionContext.

1.  Tag `DirectExecutionContext` as `ExecutionContext[Nothing]`, thereby
    stating that it works for any tagged `Future`.
2.  Move `DirectExecutionContext` to the _libs-scala/concurrent_
    library, as it requires it and it's tiny.

CHANGELOG_BEGIN
CHANGELOG_END

* concurrent: Fix the privacy of `DirectExecutionContextInternal`.

Co-authored-by: Stephen Compall <stephen.compall@daml.com>

Co-authored-by: Stephen Compall <stephen.compall@daml.com>
2020-09-29 17:23:57 +00:00
Stephen Compall
d48e9d251d
ExecutionContext[EC] phantom for control, Future[EC, A] (#7347)
* add phantom-tagged ExecutionContext and Future to scala-utils concurrent package

* many new operations for Futures

* Future, ExecutionContext combinators from porting ledger-on-sql

- picked from 546b84ab9cdf4de2d93ec5682bdee6cfd6b385f8

* move Future, ExecutionContext companions into normal package

* lots of new docs

* many new Future utilities

* working zipWith

* tests for ExecutionContext resolution, showing what will be picked under different scenarios

* even more tests for ExecutionContext resolution

* tests showing some well-typed and ill-typed Future combinator usage

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* missed scalafmt

* one more doc note

* split concurrent package to concurrent library

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-09-17 07:36:50 +00:00
Leonid Shlyapnikov
fe63816431
Write report summary in text format (#7407)
* write summary.txt

changelog_begin
changelog_end

* make `Response time distribution` brackets configurable
2020-09-16 09:31:54 -04:00
Leonid Shlyapnikov
4fbde3c672
Open-sourcing Gatling statistics reporter (#7325)
* Open sourcing gatling statistics reporter

Running gatling scenarios with `RunLikeGatling` from libs-scala/gatling-utils

* cleaning up

* Replace "\n" with System.lineSeparator

so the formatting test cases pass on windows

* Testing DurationStatistics Monoid laws

* Renaming RunLikeGatling -> CustomRunner
2020-09-11 13:39:05 +00:00
Stephen Compall
0b72b6f716
more trigger service logs; tagged LoggingContext (#7205)
* get a LoggingContext into the TriggerRunnerImpl

* make some implicits more implicitly scoped and explicitly ascribed

* make some private/final markings

* most of JsonFormat[Identifier] is in companion

* experimental LoggingContext with phantom type

* ActorContext#log isn't really doing that much

* more details of LoggingContextOf

* make LoggingContextOf compile

* add trigger message logging, yet without context

* fix parent compile errors

* use Config as the phantom for its own logging extensions

* LocalDateTimeFormat cleanup

* switch TriggerRunner to contextual logging

* add trigger definition ID to logs

* log trigger-submitted commands, fix trigger test compile

* log trigger stopping and DAR uploads

* add context to PostStop/PreRestart logs

* add changelog

CHANGELOG_BEGIN
- [Triggers] More detailed logging of trigger actions and trigger service actions.
  See `issue #7205 <https://github.com/digital-asset/daml/pull/7205>`_.
CHANGELOG_END

* missed copyright header

* switch to Unit, scala/bug#9240 fixed
2020-09-03 17:47:09 +00:00
Samir Talwar
c8b10091db
participant-integration-api: Never use a delay of zero. (#7175)
* participant-integration-api: Never use a delay of zero.

If `akka.pattern.after` is passed a delay of zero, it will execute the
body synchronously, potentially leading to a stack overflow error.

This error was observed in tests.

CHANGELOG_BEGIN
CHANGELOG_END

* timer-utils: Add tests for Delayed.Future.

* timer-utils: Add tests for RetryStrategy.

* timer-utils: Remove duplication in RetryStrategy tests.

* timer-utils: Allow for more wiggle room in the RetryStrategy tests.

* timer-utils: Fail after retrying the correct number of times.

* timer-utils: Ensure we don't overflow the stack in RetryStrategy.

* timer-utils: Reject a negative number of retry attempts.
2020-08-19 10:43:31 +00:00
Samir Talwar
27f76c4386
Use Future.unit instead of Future.successful(()). (#7080)
And in one instance, `Resource.unit`.

I just think it's easier to read. Too many parentheses make Samir a dull
boy.

CHANGELOG_BEGIN
CHANGELOG_END
2020-08-11 09:18:50 +00:00
Stephen Compall
96624a7677
use -Ywarn-unused for all Scala code (#6907)
* add -Ywarn-unused to all scalac options

* remove some unused arguments

* remove some unused definitions

* remove some unused variable names

* suppress some unused variable names

* changeExtension doesn't use baseName

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* work around no plugins in scenario interpreter perf tests

* remove many more unused things

* remove more unused things, restore some used things

* remove more unused things, restore a couple signature mistakes

* missed import

* unused argument

* remove more unused loggingContexts

* some unused code in triggers

* some unused code in sandbox and kvutils

* some unused code in repl-service and daml-script

* some unused code in bindings-rxjava tests

* some unused code in triggers runner

* more comments on silent usages

- suggested by @cocreature; thanks

* fix missing reference in TestCommands

* more unused in triggers

* more unused in sandbox

* more unused in daml-script

* more unused in ledger-client tests

* more unused in triggers

* more unused in kvutils

* more unused in daml-script

* more unused in sandbox

* remove unused in ledger-api-test-tool

* suppress final special case for codegen unused warnings

.../com/daml/sample/mymain/ContractIdNT.scala:24: warning: parameter value ev 0 in method ContractIdNT Value is never used
      implicit def `ContractIdNT Value`[a_a1dk](implicit `ev 0`: ` lfdomainapi`.Value[a_a1dk]): ` lfdomainapi`.Value[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
                                                         ^
.../com/daml/sample/mymain/ContractIdNT.scala:41: warning: parameter value eva_a1dk in method ContractIdNT LfEncodable is never used
      implicit def `ContractIdNT LfEncodable`[a_a1dk](implicit eva_a1dk: ` lfdomainapi`.encoding.LfEncodable[a_a1dk]): ` lfdomainapi`.encoding.LfEncodable[_root_.com.daml.sample.MyMain.ContractIdNT[a_a1dk]] = {
                                                               ^

* one more unused in daml-script

* special scaladoc rules may need silencer, too

* unused in compatibility/sandbox-migration

* more commas, a different way to `find`

- suggested by @remyhaemmerle-da; thanks
2020-08-07 13:16:09 -04:00
Stephen Compall
294068ae1d
remove unused definitions, params, args from libs-scala and test tool Scala code (#7051)
* remove unused definitions, params, args from libs-scala and test tool Scala code

CHANGELOG_BEGIN
CHANGELOG_END

* collapse `Some(_) | None` into _

- suggested by @SamirTalwar-DA; thanks

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>

Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
2020-08-06 18:26:22 +00:00
Stephen Compall
3196ef7575
fix flaky 'restart triggers after shutdown' test (#6926)
* reenable 'restart triggers after shutdown'

CHANGELOG_BEGIN
CHANGELOG_END

* wait for everything to shut down before completing a withTriggerService fixture

- similar to a change to HttpServiceFixture.withHttpService in #4593,
  but without the suppression of shutdown errors

* label the WithDb tests

* in CI, test only 'recover packages after shutdown', 50 times

* experiment: Process#destroy appears to be async

* is it in the in-between period?

* partial -> total

* replace some booleans with assertions for better error reporting

* make triggerLog concurrent

* close channel and file in other error cases for port locking

- suggested by @leo-da; thanks

* use port locking instead of port 0 for trigger service fixtures

* destroy one service at a time

* missed continuation in build script

* use assertion language for "restart triggers with update errors"

* Revert "is it in the in-between period?"

This reverts commit 211ebfe9d2.

* use better assertion language for "restart triggers with update errors"

* restore full CI build
2020-08-04 08:15:08 -04:00
Stephen Compall
3a0a3228ec
fix null pointer in PortLock#lock (#6941)
java.lang.NullPointerException:
  at com.daml.ports.PortLock$Locked.unlock(PortLock.scala:55)
  at com.daml.ports.PortLock$.lock(PortLock.scala:41)
  at com.daml.ports.LockedFreePort$.find(LockedFreePort.scala:15)
  at com.daml.lf.engine.trigger.TriggerServiceFixture$.$anonfun$withTriggerService$1(TriggerServiceFixture.scala:65)

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-31 10:59:39 -04:00
Stefano Baghino
e972872128
Properly use LoggingContext in Participant Server (#6924)
* Properly use LoggingContext in Participant Server

Fixes #6837

Spreads usage of LoggingContext and ContextualizedLogger throughout the participant server.

changelog_begin
[Sandbox/Integration Kit] We have enriched the contextual information
exposed by the Ledger API server. You should note richer logging information,
which can be read either via unstructured or structured logging frameworks.
A paragraph on how to configure structured logging has been added to the docs.
For more on the issue, see https://github.com/digital-asset/daml/issues/6837.
changelog_end

* Make everything compile

* Leave only actual commands on trace level on submission

* Shorter log messages on submission

* Don't add values down the call stack

* Document good practices regarding the logging context

* Add context to write service implementations

* Add logging context to tests

* Document logging for DAML on SQL

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463503013

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463508117

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463515665

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463513157

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463516359

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463516895

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463518813

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463520210

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463521501

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463521593

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525453

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525560

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525672

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463525742

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463526837

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527054

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527523

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527814

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527958

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527900

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463527997

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463528050

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463542877

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543051

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543614

* Address https://github.com/digital-asset/daml/pull/6924#discussion_r463543692
2020-07-31 12:40:13 +00:00
Leonid Shlyapnikov
05d49b37c3
Introduce //libs-scala/scala-utils (#6935)
* Moving `Statements.discard` from //ledger-server/http-json into //libs-scala/scala-utils

changelog_begin
changelog_end

* Add new module to the published artifacts

* `com.daml.scalautil` instead of `com.daml.scala.util`

@S11001001: That's because if this is in classpath and you import com.daml._,
you have a different scala in scope than the one you expect.
2020-07-30 13:51:07 -04:00
Samir Talwar
99463577af
triggers + ports: Lock free ports before binding. (#6911)
* triggers: Use `FreePort.find()`.

* ports: Move `LockedFreePort` from postgresql-testing for reuse.

* triggers: Use `LockedFreePort` to avoid race conditions.

* ports + triggers: Move common port testing into the ports library.

CHANGELOG_BEGIN
CHANGELOG_END
2020-07-29 14:29:55 +00:00
Samir Talwar
d6fc2bbb58
ledger-api-client + participant-integration-api: Increase the default maximum inbound error size, and truncate errors well before that. (#6807)
* participant-integration-api: `GrpcServerOwner` -> `GrpcServer.Owner`.

Mostly so I can create a test class named `GrpcServerSpec`.

* ports: Move the free port search from postgresql-testing.

* participant-integration-api: Test the basics of GrpcServer.

This uses the HelloService to make sure the server behaves normally.

* ledger-api-client: Extract out channel configuration from LedgerClient.

So we can test it independently of the LedgerClient itself.

* ledger-api-client: Increase the default maximum inbound header size.

Increased from 8 KB to 1 MB.

* participant-integration-api: Reduce the maximum error message size.

Truncate GRPC error descriptions to 256 KB.

* participant-integration-api: Use `Port.Dynamic` instead of `FreePort`.

In tests.

* participant-integration-api: Explicit null checks when they're shorter.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* ledger-api-client: Reduce the max inbound message size back to 8 KB.

And reduce the maximum size of an error description pushed out by the
server accordingly.

CHANGELOG_BEGIN
- [Integration Kit] Truncate GPRC error messages at 4 KB. This ensures
  that we won't trigger a protocol error when sending errors to the
  client.
CHANGELOG_END

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-07-21 15:50:33 +00:00
Stephen Compall
a51d0db8ff
set scalac -Xsource:2.13 -Ypartial-unification globally (#6469)
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts

* add now-referenced scalaz-core where needed

* work around bad type signatures in scalatest Aggregating, Containing

* unused Any suppression

* work around bad partial-unification wrought by type alias

* remove unused Conversions import

- not required in 4f68cfc480 either, so unsure how it's survived this long

* work around Future.traverse; remove unused show import

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove unused bounds

* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed

* longer comment on what the options do

- suggested by @stefanobaghino-da; thanks

* forget Future.traverse, just use scalaz, it knows how to do this
2020-06-24 16:51:24 -04:00
Stephen Compall
15350a7bc2
disable warts.Any and remove most suppressions (#6132)
* disable Any wart

* first pass removal of Any suppressions for false positives

* second pass removal of Any suppressions for false positives

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* third pass removal of Any suppressions for false positives

* fourth pass removal of Any suppressions for false positives

* reformat newly single-suppressions into single lines

- suggested by @SamirTalwar-DA; thanks
2020-05-28 16:53:41 +00:00
Stephen Compall
1be6b3640f
equalz Scalatest matcher in new daml-lf/scalatest-tools library (#1200)
* equalz Scalatest matcher in new daml-lf/scalatest-tools library

* equalz typing tests

* a 'should' replacing design

* a 'MatcherFactory1' design

- this fails because the TC parameter should be a type member to avoid
  scala/bug#5075 but it is not

* MatcherFactory1 with chained Lub+Equal typeclass

- requires partial-unification at point of use, which is not great

* LubEqual's extra tparam is probably unneeded

* better LtEqual

* demonstrate that HK LubEqual's resolve with DMT should + MatcherFactory

* remove unneeded 3rd param from LubEqual, again

* update dependency specs and license headers

* allow use with should, shouldNot in some cases, preserving the shouldx/shouldNotx alternatives

* move Equalz to libs-scala/scalatest-utils

* rename bzl targets and place in com.daml.scalatest package

* add scalatest-utils to release

* move *SpecCheckLaws, Unnatural to scalatest-utils

* missed scalacheck dep in scalatest-utils

* downstreams of *SpecCheckLaws now get them from scalatest-utils

* test equal-types case as well

* update LF documentation

CHANGELOG_BEGIN
CHANGELOG_END

* whitespace error
2020-05-27 17:25:43 -04:00
Stefano Baghino
6f1e051648
Integrate metrics for cache (#6043)
* Integrate metrics for cache

Centralizes the creation of instrumented caches in a single point and adds
metrics coming from Caffeine into the mix.

changelog_begin
[Ledger API Server] if --max-state-value-cache-size is greater than zero, the
following additional metrics will be recorded under the daml.kvutils.submission.validator.state_value_cache namespace:
hits, misses, load_successes, load_failures, load_total_time, evictions and evicted_weight
changelog_end

* Fix Bazel build file formatting

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427902339

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427904794

* Review DropwizardStatsCounter

- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905074
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905184
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905307
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905387
- address https://github.com/digital-asset/daml/pull/6043#discussion_r427905650

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427906243

* Fix implicit numeric widening fatal warning

* Address https://github.com/digital-asset/daml/pull/6043#discussion_r427960762

* Fix infinite loop in metrics
2020-05-20 15:51:45 +02:00
Stefano Baghino
1018e2744b
Add put and getIfPresent methods to Cache (#6007)
* Add put method to Cache

changelog_begin
changelog_end

* Add getIfPresent method
2020-05-18 11:53:08 +00:00
Samir Talwar
57a8d0b37e
CI: Run PostgreSQL once for all Scala tests. (#5919) 2020-05-14 09:06:34 +02:00
Stefano Baghino
9ff36a13cf
Extract caching from participant-state as a library (#5949)
* Extract caching from participant-state as a library

This will be used to keep a cache of values to cut on LF translation cost when serving transactions.

changelog_begin
changelog_end

* Add dependency where missing
2020-05-12 19:45:55 +02:00
Samir Talwar
208d4a50da
postgresql-testing: Use a new, random database name every time. (#5911)
* postgresql-testing: Store the JDBC URL separately.

* postgresql-testing: Expose the username and password.

* postgresql-testing: Get the caller to create the database.

And make sure it's a random one, not "test".

CHANGELOG_BEGIN
CHANGELOG_END

* postgresql-testing: Only store the JDBC URL for tests.

Less mutable state, innit.

* postgresql-testing: Capture the individual JDBC URL parameters.

* Bazel: Fix PostgreSQL binary paths.

* postgresql-testing: Just recreate the database in PostgresAroundEach.

There's no need to restart the process with a different data directory.
2020-05-08 12:36:42 +00:00
Robert Autenrieth
7e448d810c
Always use the latest ledger config (#5669)
CHANGELOG_BEGIN
- [Sandbox] The ledger API server will now always use the most recent ledger configuration.
  Until a ledger configuration is read from the ledger, command submissions will fail with the UNAVAILABLE error.
CHANGELOG_END

In kvutils, the first ledger configuration change needs
to have a generation one higher than the one returned
by getLedgerInitialConditions().

Remove initial config writing from sandbox as it's now written by the ledger API server
2020-05-06 12:12:23 +02:00
Moritz Kiefer
0e5aa4bebf
Remove -o config from PostgresAround (#5707)
We already specify those values in the config file.

changelog_begin
changelog_end
2020-04-23 16:08:37 +00:00
Samir Talwar
16a0902c3a
resources: Don't time out ProgramResource on startup. (#5564)
* resources: Rename `tearDownDuration` to `tearDownTimeout`.

* resources: Don't time out `ProgramResource` on startup.

Let it go for as long as it needs to.

Sometimes Sandbox takes a while to start, especially if you're
preloading large DARs. We should not try and predict how long it will
take.

CHANGELOG_BEGIN
CHANGELOG_END
2020-04-15 13:35:02 +00:00
Leonid Shlyapnikov
b1c87e9f12
Fixing scaladoc (#5506)
changelog_begin
changelog_end
2020-04-09 06:01:18 +00:00
Leonid Shlyapnikov
29e1931f17
Add --port-file command line option to JSON API (#5454)
* Adding `--port-file` support

* ``--port-file`` support

* Updating docs

changelog_begin

[JSON API] Add support for ``--port-file`` command line option.
``--http-port 0 --port-file ./json-api.port`` will pick up a free port
and write it into ``./json-api.port` file.

changelog_end

* reformatting

* Usage grammar

* use bimap

* Adding `PortFiles` utility for creating and deleting port files on JVM exit

* Adding scaladoc explaining that the port file should be deleted on

JVM termination.

* Updating usage and docs to reflect that the file must be unique and

will be deleted on graceful shutdown

* Relying on `java.nio.file.FileAlreadyExistsException` to determine the

case when failed due to the nonunique file name.

* toString instead of Exception.getMessage

java.nio exception's getMessage can be just a file name, need the class
name to capture the error context.

* updatePortFile -> createPortFile

* write to file instead of write into file
2020-04-08 18:48:11 +00:00
Gerolf Seitz
a2d785e3ee
Use com.daml as root package (#5343)
Packages com.digitalasset.daml and com.daml have been unified under com.daml

Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.

CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
2020-04-05 19:49:57 +02:00
Gerolf Seitz
329320bad9
Organize maven coordinates (#5272)
* Use com.daml as groupId for all artifacts

CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END

* Add 2 additional maven related checks to the release binary

1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId

* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
2020-04-01 11:41:18 +02:00
Samir Talwar
52dbcf5d95
postgresql-testing: "Lock" ports when starting PostgreSQL. (#5310)
* http-json: Ask for a free port by specifying port 0.

This will avoid race conditions.

* bindings-akka-testing: Delete RandomPorts; it's unused.

* ports: Fix the Bazel test glob.

* ports: Move FreePort to postgresql-testing and add a test case.

* postgresql-testing: Make `FreePort.find()` return a `Port`.

* postgresql-testing: Lock free ports until the server starts.

This uses a `FileLock`, which should work well on all our
supported operating systems as long as everyone agrees to use it.

CHANGELOG_BEGIN
CHANGELOG_END

* postgresql-testing: Try to find a free port 10 times, then give up.

* postgresql-testing: Use a shared directory for the port lock.

* postgresql-testing: Try an alternative way of getting `%LOCALAPPDATA%`.
2020-03-31 17:13:02 +00:00
Gary Verhaegen
1872c668a5
replace DAML Authors with DA in copyright headers (#5228)
Change requested by Manoj.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-27 01:26:10 +01:00
Samir Talwar
1e13b8679e
postgresql-testing: On failure, logs are no longer an Option. (#5190)
`logs` used to be an `Option[Seq[String]]`. They were changed to
`Seq[String]` with a default of `Seq.empty`, but the code to print them
still assumed an `Option`, which means they were printing as
"ArrayBuffer(line 1, line 2, line 3)" instead of, well, nicely.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-26 08:09:24 +00:00
Stefano Baghino
181df8f3ab
Act on Option2Iterable wart (#5154)
Some Option2Iterable ignore annotations are not needed, others were needed for unused methods.

In a few occasions we were ignoring the warning for the very purpose for which is was there,
i.e. avoiding an implicit conversion. I'm all for not verifying this rule if we agree we
don't need it.

For ProcessFailedException it was a bit gratuitous, I changed the way in which the exception
message is built.

CHANGELOG_BEGIN
CHANGELOG_END
2020-03-24 11:25:05 +00:00
Stefano Baghino
1f0534ea4f
Spin-off BuildInfo into its own (micro-)library (#5004)
* Spin-off BuildInfo into its own (micro-)library

CHANGELOG_BEGIN
CHANGELOG_END

* Fix dependencies

* Remove unused dependency
2020-03-16 09:04:00 +01:00
mergify[bot]
e785bb7402
Sandbox: Re-use actor systems between the StandaloneApiServer and StandaloneIndexerServer. (#4827)
* sandbox: Re-use the root actor system in the StandaloneIndexerServer.

* kvutils/app: Don't use the ActorSystem execution context randomly.

Instead, make `Runner` a proper ResourceOwner, with an `acquire` method.

* sandbox: Re-use the root actor system in the StandaloneApiServer.

CHANGELOG_BEGIN
CHANGELOG_END

* resources: Remove the now-unused `ResourceOwner.sequence` functions.

They weren't well-thought-out anyway; they acquire resources
sequentially, rather than in parallel.
2020-03-05 14:50:18 +00:00
Samir Talwar
85998f8f06
Sandbox-Next: Add the ResetService. (#4802)
* sandbox-next: Make the Runner a real ResourceOwner.

* sandbox: Don't construct the ResetService twice.

* sandbox: Inline and simplify methods in StandaloneApiServer.

* resources: Define a `ResettableResource`, which can be `reset()`.

`reset()` releases the resource, performs an optional reset operation,
and then re-acquires it, binding it to the same variable.

* resources: Pass the resource value into the reset operation.

* sandbox: Fix warnings in `TestCommands`.

* sandbox-next: Add the ResetService.

CHANGELOG_BEGIN
CHANGELOG_END

* sandbox: Make sure the SandboxResetService resets asynchronously.

It was being too clever and negating its own asynchronous behavior.

* sandbox-next: Forbid no seeding.

This double negative is really hard to phrase well.

* sandbox-next: Implement ResetService for a persistent ledger.

* sandbox: Delete the comment heading StandaloneIndexerServer.

It's no longer meaningful.

* sandbox-next: No need to wrap the SandboxResetService in an owner.

* sandbox-next: Bump the ResetService test timeouts.

It looks like it's definitely slower than on Sandbox Classic™. Gonna
look into this as part of future work.

* Revert to previous asynchronous reset behavior

Co-authored-by: Gerolf Seitz <gerolf.seitz@digitalasset.com>
2020-03-04 14:40:35 +00:00
Samir Talwar
a73b7459a7
resources: Resource.unit, as an alias for Resource.successful(()). (#4786)
* resources: `Resource.unit`, as an alias for `Resource.successful(())`.

CHANGELOG_BEGIN
CHANGELOG_END

* resources: Add `ResourceOwner.unit`.
2020-03-03 16:47:54 +00:00