Commit Graph

297 Commits

Author SHA1 Message Date
Andreas Herrmann
8bceeb13de
Handle token expiry in trigger service (#8037)
* Enable adjustable clock in trigger service tests

changelog_begin
changelog_end

* Test user side token expiry

* Test service side token refresh

* Use AccessToken wrapper in TriggerRunnerImpl

* Store refresh token in trigger DB

* add refresh token to trigger runner config

* TriggerTokenExpired message to server

* TriggerTokenRefresh message to server

* refresh trigger token and update db

* Restart trigger with fresh token

* Test second token expiry

* Refresh token on running trigger

changelog_begin
* [Triggers] UNAUTHENTICATED errors will now terminate the trigger.
  These errors are no longer available for handling in the trigger DAML
  code. Instead, they are forwarded to the trigger service for handling,
  e.g. access token refresh.
changelog_end

* todo note

* Move triggerRunnerName and getRunner into object

* Factor out token refresh

* Factor out getActiveContracts

* factor out create command

* Add logging to token refresh

* Handle token expiry in TriggerRunner

TriggerRunnerImpl throws a dedicated exception when it fails on an
expired access token (any unauthenticated error to be precise).
The TriggerRunner supervisor reacts to this child failure by
requesting a token refresh and restart on the trigger server and
stopping itself.
The trigger server requests a new access and refresh token on the auth
middleware and restarts the trigger.

This works around an issue with actor supervisors in akka-actor-typed.
A stop supervisor wrapped within a restart supervisor will not cause a
stop as expected. Instead, the restart supervisor will trigger as well
and restart the actor. The work around uses a custom behavior
interceptor to emulate the appropriate stop supervisors as closely as
possible. We cannot properly emulate ChildFailed signals this way, so
we use dedicated messages intead.

* throw --> Future.failedo

* getOrFail helper

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-12-02 16:17:45 +00:00
Samir Talwar
9976b4cd50
Bazel: Factor out logic around Protobuf JARs. [KVL-714] (#8084)
* kvutils: Use ScalaPB to generate a Scala JAR for daml_kvutils.proto.

* Bazel: Delete the unused `da_java_binary` rule, and inline `_wrap_rule`.

* Bazel: Factor out Java/Scala protobuf class generation into a helper.

CHANGELOG_BEGIN
CHANGELOG_END

* daml-lf/archive: Use `proto_jars`.

* Bazel: Remove the visibility modifier from `proto_jars`.

It's too confusing. Just make everything public.

* daml-lf/archive: Push protobuf source tarballs into `proto_jars`.

* Bazel: Add comments to the various parts of `proto_jars`.

* daml-assistant: Do unpleasant things with `location` in Bazel.
2020-11-27 08:34:53 +00:00
Moritz Kiefer
2d3820ac14
Upgrade akka-http to 10.2 (#8058)
* Upgrade akka-http to 10.2

Follow up to #8048, I left out this upgrade to reduce noise and since
I wasn’t quite sure how involved it was going to be.

changelog_begin
changelog_end

* Reenable transparent HEAD requests

Apparently no longer on by default but we depend on this in waitForHttpServer

changelog_begin
changelog_end
2020-11-25 13:39:25 +01:00
Moritz Kiefer
d83cbdb475
Upgrade akka and akka-http (#8048)
* Upgrade akka and akka-http

Was chasing an issue somewhere and thought this might affect it in
some way. It didn’t but I might as well turn the upgrade into a PR.

changelog_begin
changelog_end

* Fix trigger service tests

changelog_begin
changelog_end

* Downgrade akka-http again

changelog_begin
changelog_end

* Upgrade akka-http again and fix tests

changelog_begin
changelog_end

* Cleanup trigger service

changelog_begin
changelog_end
2020-11-25 10:13:51 +01:00
Moritz Kiefer
9a6c33a7e3
Pass application_id through auth middleware (#8011)
fixes #7978

There is no new test in the trigger service since the existing test
for the custom application id already hits this. The difference is
that now the test authorization server will produce a token with the
application id set to what we request rather than the wildcard token
we used before.

changelog_begin
changelog_end
2020-11-19 12:00:54 +01:00
Moritz Kiefer
21aaf8b173
Use futures for database interaction in trigger service (#7999)
akka-http gets a bit unhappy if you block within requests and we also
use the unsafeToFuture in the JSON API so it seems sensible to do the
same in both.

I’ve moved out the initDb option out of the Server actor both because
it seemed cleaner than calling sys.exit in the actor and because it
was annoying to fit it in.

changelog_begin
changelog_end
2020-11-19 09:13:05 +01:00
Andreas Herrmann
59f40cb54e
Hide docs of Daml.Trigger.Internal (#7992)
changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-11-18 12:26:06 +00:00
Andreas Herrmann
14f71ed20c
Split trigger service tests into a test suite (#7983)
This is to reduce the likelyhood of these tests timint out. Bazel will
generate a dedicated test target per `.scala` file. Meaning the tests
can run in parallel and each test target should have a shorter overall
runtime.

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-11-17 13:14:24 +00:00
Andreas Herrmann
90dc3a5669
Implement token refresh in auth middleware (#7981)
* Obtain refresh token from Auth0

Auth0 requires the `offline_access` scope to be set to return a refresh
token.

See https://auth0.com/docs/tokens/refresh-tokens/get-refresh-tokens

Additionally, the `audience` claim needs to be set to obtain a JWT
access token and a refresh token.

See https://auth0.com/docs/tokens/refresh-tokens

changelog_begin
changelog_end

* Implement refresh endpoint on auth middleware

Following the refresh spec [1] and Auth0 documentation [2].

[1]: https://tools.ietf.org/html/rfc6749#section-6
[2]: https://auth0.com/docs/tokens/refresh-tokens/use-refresh-tokens

* Adapt Auth0 example configuration

Ignore any requests outside the ledger-api audience.

Don't throw on missing query fields. Otherwise the unhandled exception
would prevent unrelated requests from succeeding. E.g. token refresh
requests would always fail.

* Forward unauthorized/forbidden response on refresh

* re-use precomputed token payload

* Implement token refresh in auth test server

Reuses the association between authorization code and token payload to
associate refresh tokens and token payload.

Adds an expiry to the generated token to make tokens distinguishable
across refresh.

* obtain refresh token in test client

* Test auth server refresh token

* auth test server clock configurable

The clock used to define token expiry is configurable

* Override default clock in test fixture

* implement an adjustable clock

* Test token refresh with adjustable clock

* Test token expiry on /auth backend

* Test case for auth middleware /refresh endpoint

* handle malformed code/refresh token in auth server

* Forward client errors on middleware refresh

* Test middleware refresh failure

* Clarify meaning of offline accesss

* Remove redundant testing only comment

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-11-17 12:06:42 +00:00
Moritz Kiefer
10792ed6fd
Make application ID configurable in trigger service (#7974)
* Make application ID configurable in trigger service

fixes #7671

changelog_begin

- [Trigger Service] The application id used by a trigger can now be
  configured by an optional `applicationId` in the start request.

changelog_end

* Update triggers/service/src/main/scala/com/digitalasset/daml/lf/engine/trigger/Server.scala

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2020-11-17 10:25:30 +01:00
Moritz Kiefer
d8bbd2e8bd
Cleanup trigger message types (#7964)
* Cleanup trigger message types

This properly separates the messages accepted by the internal
TriggerRunnerImpl from the ones accepted by TriggerRunner. This also
shows a bug/redundancy where we had a bunch of code in
TriggerRunnerImpl to handle `Stop` but as evidenced by the types now,
we never actually send this message. We send it to TriggerRunner which
then tears down the child with it.

It also shuffles around the server message type to make it clear where
it belongs to.

And of course, I managed once again to include debugging output from a
previous PR so this is now removed as well 🤦

changelog_begin
changelog_end

* Update triggers/service/src/main/scala/com/digitalasset/daml/lf/engine/trigger/TriggerRunnerImpl.scala

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>

Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
2020-11-16 12:27:51 +01:00
Moritz Kiefer
b6e04c4fee
Fix protobuf decoding in trigger service (#7967)
We need to go via the methods in Reader to make sure that we get our
increased protobuf recursion limit. Otherwise, we fail when trying to
read from the database on anything non-trivial. I’ve verified that the
definition I’ve added is sufficient to break the default limit.

changelog_begin

- [Trigger Service] Fix a bug where complex models resulted in a fatal
  error when restoring the state from the database due to an incorrect
  protobuf recursion limit.

changelog_end
2020-11-16 12:16:03 +01:00
Moritz Kiefer
dbf28e0794
Revamp trigger status endpoint (#7961)
* Revamp trigger status endpoint

fixes #7951

The previous endpoint was a memory leak, nothing got persisted across
restarts and it omitted useful information like the metadata of the
trigger. The information is useful for testing, so I abstracted over
it so we can do what we did before in testing.

As for the endpoint, it now queries the actor for its current status
and only returns that and includes the metadata in the response.

As mentioned in #7951, I do think there is value in some kind of
history and potentially something including trace statements but I’d
like to do that properly instead of the hacky thing we have atm.

changelog_begin

- [Trigger Service] The trigger status endpoint /v1/triggers/:id now
  includes metadata about the trigger like the party and the trigger
  id. The logs field has been replaced by a status field.

changelog_end

* Fix ACS query test

changelog_begin
changelog_end
2020-11-13 15:59:34 +01:00
Andreas Herrmann
1964920ae1
Trigger auth failure (#7952)
* Factor out oauth2 test server body into a class

changelog_begin
changelog_end

* move start into class

* mutable authorized parties

* add function sfor party authorization

* manage party authorization in trigger fixture

* enable deleting cookies

* Add access denied test cases

* Track readAs and actAs claims

* Enable ignored auth tests

* fmt

* add method comments

* Update triggers/service/auth/src/main/scala/com/daml/oauth/server/Server.scala

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Update triggers/service/auth/src/main/scala/com/daml/oauth/server/Server.scala

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* Explain party authorization modes

* inline expected status codes

* fix closing brace

* use shouldBe instead of should equal

* Use shouldBe instead of should equal

* Explain revoking access and deleting cookies

* foreach requires a total function

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-11-12 23:09:30 +00:00
Moritz Kiefer
07d5a19d50
Cleanup trigger service endpoints (#7950)
* Cleanup trigger service endpoints

fixes #6333

changelog_begin

- [Trigger Service]

  Endpoints have been rearranged to be more consistent:

  | New endpoint              | Old endpoint       | Functionality                |
  |---------------------------|--------------------|------------------------------|
  | GET `/v1/triggers`        | `/v1/list`         | List triggers                |
  | POST `/v1/triggers`       | `/v1/start`        | Start trigger                |
  | GET `/v1/triggers/:id`    | `/v1/status/:id`   | Trigger status               |
  | DELETE `/v1/triggers/:id` | `/v1/triggers/:id` | Stop/delete trigger          |
  | POST `/v1/packages`       | `/v1/upload_dar`   | Upload DAR                   |
  | GET `/livez`              | `/v1/health`       | liveness check               |

changelog_end

* Fix integration tests

changelog_begin
changelog_end
2020-11-12 11:23:28 +01:00
Stephen Compall
b35c9fcb3d
slow down trigger on RESOURCE_EXHAUSTED failures (#7820)
* restate the submit stage as a Flow and derived Sink

* take submit out of the trigger-to-submit flow

* type for the failures produced directly by command submission

* directly connect the msgSource failure queue to the submitter output

* parens

* slow down submission as we exceed max parallel submissions

* restricting alterF so it will be usable with ConcurrentMap

* disable buffer for the delay

* split out the delay function

* drafting a retry loop

* degenerate test for retry loop, factoring the forAllFuture utility

* map input to retrying properly

* make retrying accessible to tests

* test happy path and fix off-by-one

* further tests for retrying

* reveal that elements can get lost

* more determinism in test

* let failures block further elements from being attempted

- Previously failures would go into a separate queue, where they awaited expiry
  of their delay and further initial upstream elements were given their first
  tries.  However, closing the upstream could mean that queue was dropped, and
  detecting that situation is not trivial.  So, instead, we don't use a separate
  queue.

* plug retrying into the trigger submission flow

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* remove throttle; pendingCommandIds may leak

* report random parameter on failure

* revert comment about throttling

* explanation for fail in the error queue

- suggested by @cocreature; thanks
2020-11-10 21:36:48 +00:00
Andreas Herrmann
70d7c97e7f
Handle authorization failure in the auth middleware and trigger service (#7920)
* oauth error response

* oauth server parties parameter

* implement request denial in oauth test server

* Test oauth test server for access denied

* test unauthorized party on auth middleware

* Handle OAuth2 login flow error response in auth middleware

* Forward login error to auth middleware client

* fmt

* fix server test

* fmt

* Handle login failure in trigger service

* Test unauthorized trigger start

* Cleanup authCallback

* Update authentication specification

- The auth middleware accepts an arbitrary callback URI on /login
- The auth middleware will forward OAuth2 authorization failures to the
  client service.

changelog_begin
changelog_end

* Update triggers/service/authentication.md

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-11-10 09:27:21 +00:00
Andreas Herrmann
9758b2f85b
Check trigger dao migrations digests (#7908)
* Check the trigger dao migrations digest

Following the example of the corresponding ledger on SQL tests.

The digests had to be updated as both of them had gone out of sync.
The init digest presumably due to the change in #7226 and the one for
adding the access token during review of #7890.

changelog_begin
changelog_end

* define abstract migrations test

* Use abstract migrations test in trigger service tests

* use abstract migrations test in ledger on SQL

* Retain check for number of .sql resources

* Factor out the hash-migrations script

* Consistent shell settings

Addressing review comment

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-11-06 09:20:32 +00:00
Andreas Herrmann
c61abf0ab0
newtype wrapper for access token in trigger service (#7906)
changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-11-05 14:07:04 +00:00
Moritz Kiefer
31a3f25800
Add a --port-file option to the trigger service (#7907)
fixes #7097

changelog_begin

- [Triggers] The trigger service now has a `--port-file` option
  matching the corresponding option in the JSON API.

changelog_end
2020-11-05 15:03:34 +01:00
Moritz Kiefer
edc243963c
Expose Event from the high-level trigger API (#7899)
This allows you to write somewhat useful update functions since you
can inspect the transactions. Created and Archived are kept abstract
but we expose fromCreated/fromArchived to interact with them.

fixes #6968

changelog_begin

- [DAML Triggers] The `Daml.Trigger` module now reexports `Event`
  which avoids having to import `Daml.Trigger.LowLevel` for implementing
  a non-trivial `updateState` function.

changelog_end
2020-11-05 15:02:17 +01:00
Moritz Kiefer
2a3e8095b8
Support multiple --dar options in the trigger service (#7903)
Limiting this to a single one makes little sense and while you can
work around it by uploading more packages, that can be annoying during
development.

fixes #6332

changelog_begin

- [Triggers] The trigger service now accepts multiple `--dar`` options.

changelog_end
2020-11-05 14:01:06 +01:00
Andreas Herrmann
4f92ed13ed
Store access token in trigger dao (#7890)
* Test all four trigger service configurations

- non-authenticated with in-memory backend
- non-authenticated with database backend
- authenticated with in-memory backend
- authenticated with database backend

changelog_begin
changelog_end

* add access token to trigger dao

* Simplify readRunningTriggers

* fmt

* Rename V2__Add_access_token.sha25 to V2__Add_access_token.sha256

* Apply suggestions from code review

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-11-05 09:45:40 +00:00
Moritz Kiefer
4fe9196ceb
Improve error on query failures in toACS (#7898)
Getting a pattern match failure here if you accidentally pass in an
archived contract is a bit cruel and now that this is DAML Script
where we can handle the failure, we can do better.

changelog_begin
changelog_end
2020-11-05 10:38:54 +01:00
Andreas Herrmann
dc7ec9ef45
Refactor trigger service test fixture (#7880)
* Separate trigger service test fixture

Define a dedicated fixture per resource instead of generating all
resources in a single fixture function. This allows to control the
lifetime of resources individually.

Reuse the same sandbox and toxiproxy and auth middleware for the test
suite instead of restarting new resources for each test case. This is to
reduce overall test runtime.

changelog_begin
changelog_end

* fix exec context

changelog_begin
changelog_end

* Use owned resources where appropriate

Avoid Await result and use owned resources instead.

* Reduce test timeout

* Be more patient for Windows

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-11-04 15:00:28 +00:00
Moritz Kiefer
05b4f95967
Avoid verbose mode in DAML Script and DAML triggers (#7870)
We only expose the DAML values to users which don’t depend on this, so
we might as well avoid the unnecessary costs associated with verbose
mode especially since those are expected to increase in the
foreseeable future.

changelog_begin
changelog_end
2020-11-03 18:37:13 +01:00
Stephen Compall
6e26d8c408
properly stream commands from initialState trigger actions (#7812)
* graph component for running UnfoldStates

* proper doc for flatMapConcatNode

* nicer internal signatures

* refactor TriggerMsg encoder

* restate trigger sink in the graph DSL

* UnfoldState functions that might be useful

* express trigger sink with graphed initial state

* fix doc comment

* add SourceShape2, a SourceShape with two outlets

* add toSource for UnfoldState

* test for iterator

* do not submit for initialState out-of-band; feed into the graph instead

* factor the forAll Future pattern for testing

* test flatMapConcatNode directly

* add changelog

CHANGELOG_BEGIN
- [Triggers] Each trigger will submit up to 8 commands at a time, rather than
  submitting as fast as possible.  This applies to submissions in progress,
  rather than completed but pending submissions; the latter's limit is still
  subject to the ledger's own limits.
  See `issue #7812 <https://github.com/digital-asset/daml/pull/7812>`__.
CHANGELOG_END

* remove uncons and append, which were not needed

* log initial evaluated state again

* shape port aliases

- suggested by @cocreature; thanks
2020-10-27 17:36:24 +00:00
Andreas Herrmann
b452e07d05
Move common jwt cli flags (#7811)
* move cliopts

changelog_begin
changelog_end

* update artifacts.yaml

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-26 17:53:37 +00:00
Andreas Herrmann
cd56af1cdd
Factor out the JWT verifier CLI options (#7780)
* Factor JWT verifier CLI flags

changelog_begin
changelog_end

* Use cli-opts in auth middleware

* Use cli-opts in sandbox cli

* Mark trigger service test as long

These have become prone to timeout on CI.

Increasing the size (timeout) is a temporary fix. A proper
solution is to a) not start a fresh sandbox per test-case and b)
separate the in-mem/db and no-auth/auth configrations into
separate Bazel test targets.

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-26 15:10:40 +01:00
Stephen Compall
667e29b242
getCommandsInFlight can be used in updateState (#7787)
* make ActionTriggerAny no longer have orphan instances

* add commands-in-flight reader to TriggerUpdateA

* make getCommandsInFlight a method shared by update and rule

* add changelog

CHANGELOG_BEGIN
- [Triggers] ``getCommandsInFlight`` may be used in a high-level trigger's
  ``updateState``, as well as its ``rule``.
  See `issue #7787 <https://github.com/digital-asset/daml/pull/7787>`__.
CHANGELOG_END

* test use of getCommandsInFlight in an updateState

* inexplicable parse error

* Internal no longer exports queryContractId
2020-10-23 10:00:58 -04:00
Stephen Compall
26d49d0a68
trigger request submissions backpressure the transaction stream (#7753)
* let free interpreter suspend on a SubmitRequest

* move UnfoldState to separate file, start a custom flow for flatmapping to it

* proper handlers, getting tangled in the mutable state of it all

* tests and conversion utilities for UnfoldState

* working UnfoldState flow with far fewer tricky details, passes test

* note on laziness

* missed t update

* test empty lists better

* add bifunctor for UnfoldState

* refactor message filtering

* split message parsing from the fold

* add copyright headers to new files

* let the flow from an UnfoldState emit the states

* test flatMapConcatStates's special output semantics

* pass submit's Future back to the holder of the SubmitRequest

* parallel submission stage, here with limit 8

* feed a stream of SubmitRequests to the command submission stage

* don't fail request submit stream stage on submit failure

- There was no way before for submission errors to report themselves to the free
  interpreter, which just saw Unit; now we don't drop the Future on the floor,
  so we properly turn "normal" submission errors into success, and actually
  propagate any "real" errors to the stream instead of logging.

* note that initialState command submissions are not throttled or backpressured

* add the parallel submission size to the failure queue size

- defense against deadlock should we later change to propagate enqueuing
  failures to the stream

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END

* clean up the initial namespace of getTriggerSink
2020-10-23 10:00:38 -04:00
Andreas Herrmann
51fef07984
Verify token in auth middleware (#7773)
* Configure a token verifier on the auth middleware

changelog_begin
changelog_end

* Verify the token in the auth middleware

changelog_begin
changelog_end

* Test token verification in auth middleware

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-22 10:11:21 +02:00
Andreas Herrmann
e907fc368f
Require authorization on list and status endpoints (#7761)
Requires authorization for `readAs:<party>` claims for the list and
status endpoints of the trigger service. In case of list the party is
provided by the request entity. However, in case of status the party is
determined by querying for the running trigger instance in the same way
as for the stop endpoint.

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-21 13:43:38 +00:00
Andreas Herrmann
48527e21a5
Authorize on trigger service stop endpoint (#7750)
changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-20 17:52:48 +02:00
Stephen Compall
1d638c29cb
queryContractId and queryContractKey for high-level triggers (#7726)
* queryContractId and queryContractKey, trivially

* add changelog

CHANGELOG_BEGIN
- [Triggers] Two new functions are available for querying the ACS:
  ``queryContractId``, for looking up a contract by ID, and ``queryContractKey``
  for looking one up by key.
  See `issue #7726 <https://github.com/digital-asset/daml/pull/7726>`__.
CHANGELOG_END

* more efficient, direct queryContractId implementation

* flip getContractById's arguments, avoid 'flip'

* test queryContractKey

* test queryContractId
2020-10-20 09:50:59 -04:00
Samir Talwar
7f679b9896
resources: Customizable contexts. (#7678)
* resources: Move builders into //ledger/ledger-resources.

Keep the actual constructors in a trait, but instantiate it when working
with ledger code.

This allows us to later introduce an extra "context" type parameter to
ResourceOwner.

* resources-akka: Move the builders in to //ledger/ledger-resources.

* resources: Introduce an abstract `Context` parameter for owners.

This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.

This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.

Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.

CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
  `Context`, which is filled in by the corresponding `Context` class in
  the _ledger-resources_ dependency. This allows us to pass extra
  information through resource acquisition.
CHANGELOG_END

* ledger-resources: Move `ResourceOwner` here from `resources`.

* ledger-resources: Remove dependencies from outside //ledger.

* ledger-resource: Wrap the acquisition execution context in `Context`.

So we can add a logging context to it.

* resources: Pass the Context, not the ExecutionContext, to Resource.

* Avoid importing `HasExecutionContext`.

* ledger-resources: Publish to Maven Central.

* resources: Make the small changes suggested by @stefanobaghino-da.

Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>

* ledger-resources: Pull out a trait for test resource contexts.

Saves a few lines of code.

* Restore some imports that were accidentally wildcarded.

* resources: Replace an `implicit def` with a couple of imports.

* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.

Try and use the right execution context where possible.

Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
2020-10-20 09:26:28 +00:00
Andreas Herrmann
60fe244e1b
Use auth middleware in trigger service /v1/start endpoint (#7654)
* Authorize trigger service on middleware

changelog_begin
changelog_end

* Trigger service auth callback handler

* Forward token

* Do not pin the application ID in the access token

The trigger service will assign an individual application ID to each
trigger based on its UUID. Requiring tokens on the granularity of
application IDs would break the idea of storing the token in a cookie to
be able to use it across multiple requests.

changelog_begin
changelog_end

* todo persist trigger token

* Add a state parameter to middleware login

* add documentation comments

* typo

* fmt

* Align Party type between middleware and trigger service

The middleware was using `com.daml.lf.data.Ref.Party` while the trigger
service is using `com.daml.ledger.api.refinements.ApiTypes.Party` which
requires conversions. This aligns the types to avoid such conversions.

* optional application id in oauth2 test server

* align party types

* configure auth middleware in trigger service tests

* handle empty cookie header

* follow redirects in trigger service tests

* keep track of cookies

* keep track of cookies

* Replace any previous Cookie header

Otherwise on old daml-ledger-token cookie might persist and be preferred
over a newly added instance.

* DEBUG

* Configure test ledger client readAs claims

* fmt

* docstrings

* remove debug output

* Avoid endless redirect loops

When the replay still fails to authorize on the middleware then we do
not want to attempt another login flow.

* Store callback routes in authCallbacks

* fmt

* Push AuthTestConfig into test target

https://github.com/digital-asset/daml/pull/7654#discussion_r506510193

* Unbind oauth2 server after middleware

https://github.com/digital-asset/daml/pull/7654/files#r506513251

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-16 15:37:36 +00:00
Stephen Compall
4ca02e0eb6
upgrade doobie from 0.6.0 to 0.9.2 (#7618)
* set doobie version to 0.9.2 and rerun maven pin

* port extractor and some of JSON API

* repin maven

* use doobie's own builder compatibility where required

* use probably bad derivations to supply Blockers where transactEC was required

- The point of using Blocker instead of ExecutionContext seems to be to
  especially emphasize to API users that it isn't appropriate to use an
  ExecutionContext with ordinary behavior.  That is what we have done, which
  should probably change, but just compiling for now.

* fix fragment inspection test for internal restructuring

- This test depends on implementation details of Doobie, so naturally it must be
  altered when that runs.  Fortunately, it's been made easier by the changes
  in this upgrade.

* allow 256 blockers for navigator transaction blocker, like the global EC

* allow as many blockers as the pool size for trigger service

- The transactor shouldn't share ExecutionContext for transactions with the
  caller, so we set up a new one based on configured pool size.

* no changelog

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-16 09:46:20 -04:00
Stephen Compall
6d39ac6984
allow updating user state from high-level trigger rule (#7674)
* allow updating user state from high-level trigger rule

* fix tests and examples for new rule signature

* fix doc for new rule signature

* add changelog

CHANGELOG_BEGIN
- [Triggers] ``rule`` can now ``get`` and ``put`` the user-defined state just
  like ``updateState`` does.  It no longer accepts the state as an argument.
  You can port your rule function types by replacing ``s -> TriggerA`` with
  ``TriggerA s``, removing the last argument from the function, and using
  ``get`` to retrieve the state at the beginning of the ``rule``'s ``do`` block,
  if needed.
  See `issue #7674 <https://github.com/digital-asset/daml/pull/7674>`__.
CHANGELOG_END

* test that rule userState changes propagate all the way out

* adapt compatibility filtering to #7681
2020-10-14 14:51:25 +00:00
Stephen Compall
bda13392fd
replace getContracts with query Action, usable in initialize, updateState, rule (#7632)
* add ACS reader to TriggerA and TriggerStateA

* propagate changes in TriggerA, TriggerStateA structure

* allow query to be used in updateState and rule

* remove getTemplates

* remove ACS argument from updateState and rule

* fix type parameter order on query

* use query function in all tests and examples

* replace getContracts with query in documentation

* use wildcards instead of otherwise

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* rename TriggerStateA to TriggerUpdateA

- suggested by @cocreature; thanks

* missed renamings of TriggerStateA to TriggerUpdateA

- suggested by @cocreature; thanks

* make the meaning of the rlift functions clearer

* make initialize a TriggerInitializeA instead of a function; remove getContracts

* update tests and examples for new initialize signature

CHANGELOG_BEGIN
- [Triggers] Trigger ``updateState``, ``rule``, and ``initialize`` functions no
  longer accept an ``ACS`` argument; instead, they must use the ``query`` action
  to query the ACS, similar to the same function in DAML Script.
  See `issue #7632 <https://github.com/digital-asset/daml/pull/7632>`__.
CHANGELOG_END

* disable older compatibility trigger builds

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-10-13 13:20:43 -04:00
Remy
dc34d4f637
LF: add PackageInterface a lightweigh Package (#7577)
CHANGELOG_BEGIN
CHANGELOG_END
2020-10-13 12:41:22 +02:00
Remy
ea2a637a11
LF: decouple template from record in the Scala Ast. (#7631)
CHANGELOG_BEGIN
CHANGELOG_END
2020-10-12 11:28:30 +02:00
Stephen Compall
664a0c0076
add Action to high-level trigger updateState (#7621)
* add ActionState to the standard library

* use 1 ActionState, 1 get, 1 put in low-level trigger library

* introduce TriggerStateA for updateState

* fix tests and examples for new updateState signature

CHANGELOG_BEGIN
- [Triggers] The ``updateState`` function now returns a ``TriggerStateA``.  This
  is an action like ``TriggerA``, but doesn't permit emitting commands.  Instead
  of taking the state as an argument and returning a new state, you can
  manipulate the state with ``get``, ``put``, and ``modify``.  Any existing
  ``updateState`` can be ported by replacing ``s -> expr`` in the lambda
  expression with ``-> modify $ \s ->``, and then made to look nicer from there
  as desired.
  See `issue #7621 <https://github.com/digital-asset/daml/pull/7621>`__.
CHANGELOG_END

* some DAML docs for updateState and TriggerStateA
2020-10-09 13:56:24 -04:00
Andreas Herrmann
8b9c237031
map ledger-api claims to scopes for auth0 (#7629)
* Define mapping from claims to scope

changelog_begin
changelog_end

* Update auth0 instructions

changelog_begin
changelog_end

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-09 18:43:56 +02:00
Stephen Compall
c296e22a80
remove submitCommands, ActionTrigger from high-level Trigger API (#7619)
- for parity with what was allowed pre-#7456

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-09 07:34:33 +00:00
Stephen Compall
a1d8e8dc33
replace commands-in-flight argument to trigger rule with getCommandsInFlight action (#7600)
* add getCommandsInFlight as a TriggerA action

* immediately update commandsInFlight on emitCommands

- delay until the rule had returned made sense before submitCommands was launched
  immediately; this also makes `getCommandsInFlight` more sensible

* remove commands-in-flight argument from high-level Trigger rule

CHANGELOG_BEGIN
- [Triggers] The "commands in flight" or ``Map CommandId [Command]`` argument has been
  removed from high-level trigger ``rule`` functions; instead, the current
  commands-in-flight can be retrieved with the new ``getCommandsInFlight`` function, which
  can be done immediately at the beginning of the rule's ``do`` block to preserve exact
  existing trigger behavior.
  See `issue #7600 <https://github.com/digital-asset/daml/pull/7600>`__.
CHANGELOG_END

* remove commands-in-flight argument from trigger tests

* update doc examples and copy in compatibility

- compatibility change will most likely entail another flag day in compatibility tests,
  replacing last_pre_7456_trigger_version

* update doc text for getCommandsInFlight

* test that getCommandsInFlight gets updated during the rule

* flag day for trigger compatibility tests

sdk-version: 1.7.0-snapshot.20201006.5358.0.0c1cadcf
File:     src/CopyTrigger.daml
Hidden:   no
Range:    55:11-55:19
Source:   typecheck
Severity: DsError
Message:
  src/CopyTrigger.daml:55:12: error:
  • Couldn't match type ‘TriggerA ()’ with ‘() -> TriggerA ()’
  Expected type: Party
-> DA.Next.Map.Map CommandId [Command] -> () -> TriggerA ()
  Actual type: Party -> ACS -> () -> TriggerA ()
  • In the ‘rule’ field of a record
  In the expression:
  Trigger
  {initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
  rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}
  In an equation for ‘copyTrigger’:
  copyTrigger
  = Trigger
  {initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
  rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}

* match docs on TriggerAState to current usage of these fields

* remove emittedCommands, as commandsInFlight is now kept up-to-date

* zoomed from where?

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-10-08 12:03:56 -04:00
Stephen Compall
dc204a37a7
remove known command ID from trigger state once Transaction and Completion message are seen (#7598)
- no assumption about order

- multiple transactions/completions are treated as one

CHANGELOG_BEGIN
CHANGELOG_END
2020-10-07 18:32:49 +00:00
Stephen Compall
ff706d5762
Trigger monad's Submit emits a command ID instead of accepting one (#7587)
* make TriggerF's Submit emit a command ID instead of accepting one

- LowLevel updated

- interpreter in Scala updated, preserving Message semantics as closely as possible

* remove nextCommandId and all management from high-level Trigger API implementation

* a note about TriggerF and simulateRule

* port submitCommands calls from tests

CHANGELOG_BEGIN
- [Triggers] The CommandIds as accessed from trigger DAML code are now exactly the command
  IDs used in command submission to the ledger; as such, they will vary randomly from run
  to run of the trigger rule.  To enable this, the low-level ``submitCommands`` function
  no longer accepts a command ID, instead returning one; there is no change to the
  corresponding high-level ``emitCommands`` function, so high-level triggers should only
  see improved and easier-to-understand logging.
  See `issue #7587 <https://github.com/digital-asset/daml/pull/7587>`__.
CHANGELOG_END

* port Scala-side assertion of trigger test

* rename commandIdMap to commandIdsUsed

* test that command IDs are like the ones we expect from the runner
2020-10-07 08:34:41 -04:00
Andreas Herrmann
8dee7ac9fc
Make trigger service auth middleware /login endpoint compatible with Auth0 (#7565)
* Make /login endpoint compatible to auth0

- Make the authorization and token endpoints configurable
- Use `application/x-www-form-urlencoded` as specified in
  https://tools.ietf.org/html/rfc6749#section-4.1.3
- Check the status code of the token endpoint response
- Fix the type of the token's `expires_in` field
  (`Int` instead of `String)

changelog_begin
changelog_end

* Add Auth0 testing instructions

* Use native application type on Auth0

* scope to claims mapping todo note

Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
2020-10-05 13:44:46 +00:00
Stephen Compall
0e71a2d3d2
interpret allowed updates and other actions in a free Trigger monad, like Script (#7456)
* conservatively move daml-script, trigger SValue interpreters to common library

* introduce expect and JavaList pattern for converters

* clean up trigger Converter Command interpretation

* add Church Free monad

* add an action language for trigger updates

* add expectE to remove some of the joins

* convert more of the converters to expect

* tool for unrolling Free/Roll

* split handleStepResult up and clean up its pattern

* handleStepFreeResult to interpret TriggerF

* replace Free Church with Pure/Roll free from Script

* newtype for ActionTrigger

* replace update in low-level Trigger with Free TriggerF

* submit one Commands at a time

* boolean blindness strikes again

* log missed TriggerF steps

* comment actual Submit contents

* match #7501 fromPureSExpr sig change in 00b80b8ea3

* avoid using forwardPort in runTrigger

* push State back into DAML, so it can be excluded from the action list

* push Message back into DAML, unifying the action language for initialState and update

* bringing TriggerF into initial state

* really add TriggerF into initial state, with all ports, tested

* add ActionTrigger class, express initialState in its terms

* add all TriggerF actions to existing TriggerA

* Trigger.rule will no longer have Time argument

* rename getS, setS to get, put, matching C.M.T.State from transformers

* make high-level Rule evaluate to the underlying TriggerF sequence

* Assert's testRule doesn't have a transform yet

* move DamlTuple2 to common converter library

- suggested by @cocreature; thanks

* combine the two Frees, provide from Script

* remove time argument from integration tests

CHANGELOG_BEGIN
- [Triggers] The ``Time`` argument was removed from the trigger rule function; instead, it
  can be fetched within the ``TriggerA`` ``do`` block by ``getTime``, as with ``Update``
  and ``Scenario``.  The ``LowLevel`` trigger interface has been redesigned; such triggers
  need to be rewritten or ported to high-level triggers.
  See `issue #7456 <https://github.com/digital-asset/daml/pull/7456>`_.
CHANGELOG_END

* add trigger rule simulator to support Assert module

* missed new Free module

- left in script per @cocreature

* remove retract as we ended up using foldFree for that purpose instead

- suggested by @cocreature; thanks

* throw ConverterException instead of RuntimeException

- suggested by @cocreature; thanks

* remove Time argument from coin-upgrade-trigger

* port trigger service tests

* port trigger scenario test

* put TriggerSetup and TriggerRule into LowLevel.Trigger instead of unboxed Free

- suggested by @cocreature; thanks

* remove Time argument from trigger compatibility test

* submit commands as soon as each `emitCommands` is sequenced

- we still collect a list, but only for tracking commandsInFlight

* filter out compatibility tests for triggers before now

* remove commented imports, libraries from new shared converter

* make the TriggerF interpreter tail-recursive

* remove unused compatibility trait

* add back new state logging

* remove refactoring comment

* rewrite some LowLevel initialStates in do

* hide Daml.Script.Free from docs

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>

* remove forwardPortInitialState

- suggested by @cocreature; thanks

* manually port low-level updates

- suggested by @cocreature; thanks

* remove forwardPort

- suggested by @cocreature; thanks

* fail faster on unrecognized TriggerF

- suggested by @cocreature; thanks

Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
2020-10-02 14:18:13 -04:00