* Use cliopts.Http for OAuth 2.0 middleware
changelog_begin
- [OAuth 2.0 Middleware] You can now configure the address that the
middleware listens to using the ``--address`` flag.
The port that the middleware listens to is now configured using the
``--http-port`` flag, use 0 to dynamically choose a free port.
You can now configure a port file where the chosen port will be
written to using the ``--port-file`` flag.
changelog_end
* Add test-case for OAuth 2.0 middleware port file
* Don't forget to close source
* Fix integration test
* Update triggers/service/auth/src/main/scala/com/daml/auth/middleware/oauth2/Config.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add livez endpoint to auth middleware
* Add OAuth 2.0 middleware to Daml SDK
* unhide trigger service auth flags
changelog_begin
- [Triggers] The trigger service now supports authorization through an
auth middleware. The feature is enabled using the `--auth` and
`--auth-callback` command-line flags. Please refer to the
Authorization chapter of the trigger service documentation for further
instructions.
- [OAuth 2.0 middleware] Daml Connect now includes an implementation of
the auth middleware API that supports OAuth 2.0 Authorization Code
Grant. Please refer to the Auth Middleware and OAuth 2.0 Auth
Middleware chapters of the documentation.
changelog_end
* drop early access flag on triggers
Daml triggers, the trigger service, and the auth middleware are no
longer marked as early access features.
changelog_begin
- [Triggers] Daml Triggers and the Trigger Service are no longer in
early access status.
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Include auth challenge in response body
changelog_begin
changelog_end
Some browsers make it difficult to access the `WWW-Authenticate`
response header from Javascript. For example, Firefox 84.0.2 (64-bit) on
Linux does not expose the `WWW-Authenticate` header in the result of the
`fetch` function, independent of the server's access control headers.
In that case the header is only accessible through the `XMLHttpRequest`
API, which is more cumbersome to use.
This adds the challenge to the response body in JSON format as well to
avoid exposing users to any such browser related issues.
* Make Secure Set-Cookie attribute configurable
changelog_begin
changelog_end
Since Chrome 80 Set-Cookie with the Secure attribute enabled is rejected
for connections that don't use https [1]. This includes localhost.
Firefox, at least as of version 84.0.2, allows such cookies on
localhost.
This adds a command-line flag to the authorization middleware to make
the value of the Secure attribute configurable. This way it can be
disabled for development purposes.
[1]: https://blog.chromium.org/2019/10/developers-get-ready-for-new.html
* Expand description on --cookie-secure flag
* Fix TriggerServiceFixture
* Fix middleware TestFixture
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Don't always redirect to /login automatically
The redirect mode can be configured to never redirect, always redirect,
or redirect based on the request type (redirect for text/html).
In case of no redirect the auth middleware client will reply with 401
Unauthorized with a custom WWW-Authenticate challenge to login on the
auth middleware.
* Make login to redirect configurable on trigger service
By default the trigger service will redirect for HTML requests and not
redirect for JSON requests. The test suite uses automatic redirect as
the OAuth2 test server works without user interaction.
changelog_begin
changelog_end
* Preserve path and query in authMiddlewareUri
This is necessary if the auth middleware lies behind a reverse proxy
with a path prefix or a similar setup.
* Bump default auth middleware login timeout
One minute was to short for a login cycle that requires manual user
input.
* Set token cookie properties
`path = "/"` is required so that the `/login` endpoint can reliably
override the cookie value for other endoints such as `/cb` or
`/v1/triggers`.
* Test redirectToLogin modes
* Redirect on HTML
https://github.com/digital-asset/daml/pull/8532#discussion_r559368335
* Use pass/reject in onRedirectToLogin
8db2bff9af (r559370308)
* default login timeout 5min
https://github.com/digital-asset/daml/pull/8532#discussion_r559535511
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Remove redundant list of LF versions
After #8472, I realized that there must be a list used for daml-stdlib
and daml-prim already and it turns out there is. I’ve removed that one
in favor of the one added in #8472 since I like having all in one
place and the one from #8472 is created by filtering an existing list
instead of creating a completely separate list like we do here.
changelog_begin
changelog_end
* Introduce SCRIPT_LF_VERSIONS
changelog_begin
changelog_end
* Use ActorSystem and ExecutionContext from RequestContext
* Factor out middleware server class
To avoid passing around config and state manually.
changelog_begin
changelog_end
* Depend on databricks/sjsonnet
changelog_begin
changelog_end
* Generate request params from jsonnet template
changelog_begin
changelog_end
* Split middleware test suite sources
* Add test suite for request templates
* fmt
* TriggerServiceFixture template arguments
* Use null to indicate missing applicationId claim
Addressing
https://github.com/digital-asset/daml/pull/8453/files#r555025173
* Fix invalid path on Windows
* Close request template source
* Avoid repeated re-reading of Jsonnet files
https://github.com/digital-asset/daml/pull/8453/files#r555044262
* Factor out template argument mappings
* factor out template error handling
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This PR updates scalafmt and enables trailingCommas =
multiple. Unfortunately, scalafmt broke the version field which means
we cannot fully preserve the rest of the config. I’ve made some
attempts to stay reasonably close to the original config but couldn’t
find an exact equivalent in a lot of cases. I don’t feel strongly
about any of the settings so happy to change them to something else.
As announced, this will be merged on Saturday to avoid too many conflicts.
changelog_begin
changelog_end
* Replace many occurrences of DAML with Daml
* Update docs logo
* A few more CLI occurrences
CHANGELOG_BEGIN
- Change DAML capitalization and docs logo
CHANGELOG_END
* Fix some over-eager replacements
* A few mor occurrences in md files
* Address comments in *.proto files
* Change case in comments and strings in .ts files
* Revert changes to frozen proto files
* Also revert LF 1.11
* Update get-daml.sh
* Update windows installer
* Include .py files
* Include comments in .daml files
* More instances in the assistant CLI
* some more help texts
* Bounded auth middleware client callback store
changelog_begin
changelog_end
* Json format for login response
* Add middleware client binding to test fixture
* Test middleware client callback store size
* Make max auth callbacks and timeout configurable on trigger service
changelog_begin
changelog_end
* Bounded pending login requests at auth middleware
* Make max logins and timeout configurable on middleware
* Test middleware login store size
* fmt
* Fix Windows
failed with `"localhost" != "127.0.0.1"`. Hardcode "localhost" to avoid
platform specific resolution.
* Use FiniteDuration for login timeout
* Respond with 503 on full login request store
Addresses https://github.com/digital-asset/daml/pull/8351#pullrequestreview-560859604
* Add tests for RequestStore
* Lower DefaultMaxLoginRequests
b48050eb91 (r552649275)
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Make middleware callback URI configurable
changelog_begin
changelog_end
* Make trigger service callback URI configurable
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Extract actor system and execution context from request context
* Extract actor system and execution context from request context
* Extract actor system, materializer, and execution context from request context
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Bundle hpp with damlc.
This PR eliminates the red squiggly lines you get when opening up the
standard library in daml studio (e.g. after a Go To Definitien).
(When CPP is used, Go To Definition is still pointing to the wrong place,
because the line numbers are messed up. But it's better than having a
million error messages.)
We could also remove --cpp flag in a separate PR, though that may be
considered a breaking change.
changelog_begin
changelog_end
* cleanup locateRunfiles jank
* Revert "cleanup locateRunfiles jank"
This reverts commit 82552003ae.
This prevents the `dade-copyright-headers` script from changing those
files (and from checking them for copyright headers, unfortunately).
CHANGELOG_BEGIN
CHANGELOG_END
* Factor out authorization middleware client
changelog_begin
changelog_end
* Factor out token refresh
* Factor out auth request
* Factor out auth middleware URIs
* factor out auth directive
* Factor out login directive
* Add login response type to api library
* Allow error handling in login callback
* Factor error handling out of authorize directive
* Move tagged token types into middleware api
* Factor out the auth middleware client
* Dedicated exception types in middleware client
* Handle auth middleware client exception
Restores the behavior of authorize before factoring out the client
* expose middleware URIs
* Use the middleware client in the middleware tests
* Use localhost/CALLBACK in testing
https://github.com/digital-asset/daml/pull/8244#discussion_r540328001
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Test authentication on upload_dar endpoint
changelog_begin
changelog_end
* require authentication on upload_dar endpoint
* push Directive into auth
* Fully upload request before auth redirection
* Make HTTP entity upload parameters configurable
changelog_begin
changelog_end
* Shorten help message
https://github.com/digital-asset/daml/pull/8193#discussion_r538428368
* maxHttpEntityUploadSize as Long
https://github.com/digital-asset/daml/pull/8193#discussion_r538431773
* use DefaultMaxInboundMessageSize for DefaultMaxHttpEntityUploadSize
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Middleware test resources
- Make OAuthServer accessible
- Don't access suiteResource directly in the test-suite
* OAuth2 test server test resources
- Make Server accessible
- Don't access suiteResource directly in the test-suite
* Use a deny list in OAuth2 test server
It is simpler to configure a set of disallowed parties and extend it on
demand than switching between blanket approval and allow list modes.
changelog_begin
changelog_end
* Handle admin claims in test server
Requests with admin claims are granted by default and denied if admin
requests have been disallowed.
* reset admin state in test fixtures
* test admin tokens in OAuth2 test server
* test admin claims in OAuth2 middleware
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
changelog_begin
changelog_end
This is to avoid a race condition where the old trigger runner may not
yet have completed shutdown by the time that the token has been
refreshed and we attempt to start a new trigger runner.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.
This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.
Apologies for the giant PR, I don’t see a way to keep it smaller.
changelog_begin
changelog_end
* Rename and separate auth targets and packages
Split out the API types `Request`/`Response` into their own Bazel
targets, so that the trigger service does not have to depend on the full
middleware and OAuth2 test server targets.
Also rename packages: Generic auth code goes into `com.daml.auth`,
OAuth2 specific code goes into `com.daml.auth.oauth2` or
`com.daml.auth.middleware.oauth2`.
changelog_begin
changelog_end
* Remove empty test-case
This is a left-over from #7226.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Enable adjustable clock in trigger service tests
changelog_begin
changelog_end
* Test user side token expiry
* Test service side token refresh
* Use AccessToken wrapper in TriggerRunnerImpl
* Store refresh token in trigger DB
* add refresh token to trigger runner config
* TriggerTokenExpired message to server
* TriggerTokenRefresh message to server
* refresh trigger token and update db
* Restart trigger with fresh token
* Test second token expiry
* Refresh token on running trigger
changelog_begin
* [Triggers] UNAUTHENTICATED errors will now terminate the trigger.
These errors are no longer available for handling in the trigger DAML
code. Instead, they are forwarded to the trigger service for handling,
e.g. access token refresh.
changelog_end
* todo note
* Move triggerRunnerName and getRunner into object
* Factor out token refresh
* Factor out getActiveContracts
* factor out create command
* Add logging to token refresh
* Handle token expiry in TriggerRunner
TriggerRunnerImpl throws a dedicated exception when it fails on an
expired access token (any unauthenticated error to be precise).
The TriggerRunner supervisor reacts to this child failure by
requesting a token refresh and restart on the trigger server and
stopping itself.
The trigger server requests a new access and refresh token on the auth
middleware and restarts the trigger.
This works around an issue with actor supervisors in akka-actor-typed.
A stop supervisor wrapped within a restart supervisor will not cause a
stop as expected. Instead, the restart supervisor will trigger as well
and restart the actor. The work around uses a custom behavior
interceptor to emulate the appropriate stop supervisors as closely as
possible. We cannot properly emulate ChildFailed signals this way, so
we use dedicated messages intead.
* throw --> Future.failedo
* getOrFail helper
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* kvutils: Use ScalaPB to generate a Scala JAR for daml_kvutils.proto.
* Bazel: Delete the unused `da_java_binary` rule, and inline `_wrap_rule`.
* Bazel: Factor out Java/Scala protobuf class generation into a helper.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lf/archive: Use `proto_jars`.
* Bazel: Remove the visibility modifier from `proto_jars`.
It's too confusing. Just make everything public.
* daml-lf/archive: Push protobuf source tarballs into `proto_jars`.
* Bazel: Add comments to the various parts of `proto_jars`.
* daml-assistant: Do unpleasant things with `location` in Bazel.
* Upgrade akka-http to 10.2
Follow up to #8048, I left out this upgrade to reduce noise and since
I wasn’t quite sure how involved it was going to be.
changelog_begin
changelog_end
* Reenable transparent HEAD requests
Apparently no longer on by default but we depend on this in waitForHttpServer
changelog_begin
changelog_end
* Upgrade akka and akka-http
Was chasing an issue somewhere and thought this might affect it in
some way. It didn’t but I might as well turn the upgrade into a PR.
changelog_begin
changelog_end
* Fix trigger service tests
changelog_begin
changelog_end
* Downgrade akka-http again
changelog_begin
changelog_end
* Upgrade akka-http again and fix tests
changelog_begin
changelog_end
* Cleanup trigger service
changelog_begin
changelog_end
fixes#7978
There is no new test in the trigger service since the existing test
for the custom application id already hits this. The difference is
that now the test authorization server will produce a token with the
application id set to what we request rather than the wildcard token
we used before.
changelog_begin
changelog_end
akka-http gets a bit unhappy if you block within requests and we also
use the unsafeToFuture in the JSON API so it seems sensible to do the
same in both.
I’ve moved out the initDb option out of the Server actor both because
it seemed cleaner than calling sys.exit in the actor and because it
was annoying to fit it in.
changelog_begin
changelog_end
This is to reduce the likelyhood of these tests timint out. Bazel will
generate a dedicated test target per `.scala` file. Meaning the tests
can run in parallel and each test target should have a shorter overall
runtime.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Obtain refresh token from Auth0
Auth0 requires the `offline_access` scope to be set to return a refresh
token.
See https://auth0.com/docs/tokens/refresh-tokens/get-refresh-tokens
Additionally, the `audience` claim needs to be set to obtain a JWT
access token and a refresh token.
See https://auth0.com/docs/tokens/refresh-tokens
changelog_begin
changelog_end
* Implement refresh endpoint on auth middleware
Following the refresh spec [1] and Auth0 documentation [2].
[1]: https://tools.ietf.org/html/rfc6749#section-6
[2]: https://auth0.com/docs/tokens/refresh-tokens/use-refresh-tokens
* Adapt Auth0 example configuration
Ignore any requests outside the ledger-api audience.
Don't throw on missing query fields. Otherwise the unhandled exception
would prevent unrelated requests from succeeding. E.g. token refresh
requests would always fail.
* Forward unauthorized/forbidden response on refresh
* re-use precomputed token payload
* Implement token refresh in auth test server
Reuses the association between authorization code and token payload to
associate refresh tokens and token payload.
Adds an expiry to the generated token to make tokens distinguishable
across refresh.
* obtain refresh token in test client
* Test auth server refresh token
* auth test server clock configurable
The clock used to define token expiry is configurable
* Override default clock in test fixture
* implement an adjustable clock
* Test token refresh with adjustable clock
* Test token expiry on /auth backend
* Test case for auth middleware /refresh endpoint
* handle malformed code/refresh token in auth server
* Forward client errors on middleware refresh
* Test middleware refresh failure
* Clarify meaning of offline accesss
* Remove redundant testing only comment
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Make application ID configurable in trigger service
fixes#7671
changelog_begin
- [Trigger Service] The application id used by a trigger can now be
configured by an optional `applicationId` in the start request.
changelog_end
* Update triggers/service/src/main/scala/com/digitalasset/daml/lf/engine/trigger/Server.scala
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Cleanup trigger message types
This properly separates the messages accepted by the internal
TriggerRunnerImpl from the ones accepted by TriggerRunner. This also
shows a bug/redundancy where we had a bunch of code in
TriggerRunnerImpl to handle `Stop` but as evidenced by the types now,
we never actually send this message. We send it to TriggerRunner which
then tears down the child with it.
It also shuffles around the server message type to make it clear where
it belongs to.
And of course, I managed once again to include debugging output from a
previous PR so this is now removed as well 🤦
changelog_begin
changelog_end
* Update triggers/service/src/main/scala/com/digitalasset/daml/lf/engine/trigger/TriggerRunnerImpl.scala
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
We need to go via the methods in Reader to make sure that we get our
increased protobuf recursion limit. Otherwise, we fail when trying to
read from the database on anything non-trivial. I’ve verified that the
definition I’ve added is sufficient to break the default limit.
changelog_begin
- [Trigger Service] Fix a bug where complex models resulted in a fatal
error when restoring the state from the database due to an incorrect
protobuf recursion limit.
changelog_end
* Revamp trigger status endpoint
fixes#7951
The previous endpoint was a memory leak, nothing got persisted across
restarts and it omitted useful information like the metadata of the
trigger. The information is useful for testing, so I abstracted over
it so we can do what we did before in testing.
As for the endpoint, it now queries the actor for its current status
and only returns that and includes the metadata in the response.
As mentioned in #7951, I do think there is value in some kind of
history and potentially something including trace statements but I’d
like to do that properly instead of the hacky thing we have atm.
changelog_begin
- [Trigger Service] The trigger status endpoint /v1/triggers/:id now
includes metadata about the trigger like the party and the trigger
id. The logs field has been replaced by a status field.
changelog_end
* Fix ACS query test
changelog_begin
changelog_end
* Factor out oauth2 test server body into a class
changelog_begin
changelog_end
* move start into class
* mutable authorized parties
* add function sfor party authorization
* manage party authorization in trigger fixture
* enable deleting cookies
* Add access denied test cases
* Track readAs and actAs claims
* Enable ignored auth tests
* fmt
* add method comments
* Update triggers/service/auth/src/main/scala/com/daml/oauth/server/Server.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update triggers/service/auth/src/main/scala/com/daml/oauth/server/Server.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Explain party authorization modes
* inline expected status codes
* fix closing brace
* use shouldBe instead of should equal
* Use shouldBe instead of should equal
* Explain revoking access and deleting cookies
* foreach requires a total function
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* restate the submit stage as a Flow and derived Sink
* take submit out of the trigger-to-submit flow
* type for the failures produced directly by command submission
* directly connect the msgSource failure queue to the submitter output
* parens
* slow down submission as we exceed max parallel submissions
* restricting alterF so it will be usable with ConcurrentMap
* disable buffer for the delay
* split out the delay function
* drafting a retry loop
* degenerate test for retry loop, factoring the forAllFuture utility
* map input to retrying properly
* make retrying accessible to tests
* test happy path and fix off-by-one
* further tests for retrying
* reveal that elements can get lost
* more determinism in test
* let failures block further elements from being attempted
- Previously failures would go into a separate queue, where they awaited expiry
of their delay and further initial upstream elements were given their first
tries. However, closing the upstream could mean that queue was dropped, and
detecting that situation is not trivial. So, instead, we don't use a separate
queue.
* plug retrying into the trigger submission flow
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* remove throttle; pendingCommandIds may leak
* report random parameter on failure
* revert comment about throttling
* explanation for fail in the error queue
- suggested by @cocreature; thanks
* oauth error response
* oauth server parties parameter
* implement request denial in oauth test server
* Test oauth test server for access denied
* test unauthorized party on auth middleware
* Handle OAuth2 login flow error response in auth middleware
* Forward login error to auth middleware client
* fmt
* fix server test
* fmt
* Handle login failure in trigger service
* Test unauthorized trigger start
* Cleanup authCallback
* Update authentication specification
- The auth middleware accepts an arbitrary callback URI on /login
- The auth middleware will forward OAuth2 authorization failures to the
client service.
changelog_begin
changelog_end
* Update triggers/service/authentication.md
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Check the trigger dao migrations digest
Following the example of the corresponding ledger on SQL tests.
The digests had to be updated as both of them had gone out of sync.
The init digest presumably due to the change in #7226 and the one for
adding the access token during review of #7890.
changelog_begin
changelog_end
* define abstract migrations test
* Use abstract migrations test in trigger service tests
* use abstract migrations test in ledger on SQL
* Retain check for number of .sql resources
* Factor out the hash-migrations script
* Consistent shell settings
Addressing review comment
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
fixes#7097
changelog_begin
- [Triggers] The trigger service now has a `--port-file` option
matching the corresponding option in the JSON API.
changelog_end
This allows you to write somewhat useful update functions since you
can inspect the transactions. Created and Archived are kept abstract
but we expose fromCreated/fromArchived to interact with them.
fixes#6968
changelog_begin
- [DAML Triggers] The `Daml.Trigger` module now reexports `Event`
which avoids having to import `Daml.Trigger.LowLevel` for implementing
a non-trivial `updateState` function.
changelog_end
Limiting this to a single one makes little sense and while you can
work around it by uploading more packages, that can be annoying during
development.
fixes#6332
changelog_begin
- [Triggers] The trigger service now accepts multiple `--dar`` options.
changelog_end
* Test all four trigger service configurations
- non-authenticated with in-memory backend
- non-authenticated with database backend
- authenticated with in-memory backend
- authenticated with database backend
changelog_begin
changelog_end
* add access token to trigger dao
* Simplify readRunningTriggers
* fmt
* Rename V2__Add_access_token.sha25 to V2__Add_access_token.sha256
* Apply suggestions from code review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Getting a pattern match failure here if you accidentally pass in an
archived contract is a bit cruel and now that this is DAML Script
where we can handle the failure, we can do better.
changelog_begin
changelog_end
* Separate trigger service test fixture
Define a dedicated fixture per resource instead of generating all
resources in a single fixture function. This allows to control the
lifetime of resources individually.
Reuse the same sandbox and toxiproxy and auth middleware for the test
suite instead of restarting new resources for each test case. This is to
reduce overall test runtime.
changelog_begin
changelog_end
* fix exec context
changelog_begin
changelog_end
* Use owned resources where appropriate
Avoid Await result and use owned resources instead.
* Reduce test timeout
* Be more patient for Windows
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
We only expose the DAML values to users which don’t depend on this, so
we might as well avoid the unnecessary costs associated with verbose
mode especially since those are expected to increase in the
foreseeable future.
changelog_begin
changelog_end
* graph component for running UnfoldStates
* proper doc for flatMapConcatNode
* nicer internal signatures
* refactor TriggerMsg encoder
* restate trigger sink in the graph DSL
* UnfoldState functions that might be useful
* express trigger sink with graphed initial state
* fix doc comment
* add SourceShape2, a SourceShape with two outlets
* add toSource for UnfoldState
* test for iterator
* do not submit for initialState out-of-band; feed into the graph instead
* factor the forAll Future pattern for testing
* test flatMapConcatNode directly
* add changelog
CHANGELOG_BEGIN
- [Triggers] Each trigger will submit up to 8 commands at a time, rather than
submitting as fast as possible. This applies to submissions in progress,
rather than completed but pending submissions; the latter's limit is still
subject to the ledger's own limits.
See `issue #7812 <https://github.com/digital-asset/daml/pull/7812>`__.
CHANGELOG_END
* remove uncons and append, which were not needed
* log initial evaluated state again
* shape port aliases
- suggested by @cocreature; thanks
* Factor JWT verifier CLI flags
changelog_begin
changelog_end
* Use cli-opts in auth middleware
* Use cli-opts in sandbox cli
* Mark trigger service test as long
These have become prone to timeout on CI.
Increasing the size (timeout) is a temporary fix. A proper
solution is to a) not start a fresh sandbox per test-case and b)
separate the in-mem/db and no-auth/auth configrations into
separate Bazel test targets.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* make ActionTriggerAny no longer have orphan instances
* add commands-in-flight reader to TriggerUpdateA
* make getCommandsInFlight a method shared by update and rule
* add changelog
CHANGELOG_BEGIN
- [Triggers] ``getCommandsInFlight`` may be used in a high-level trigger's
``updateState``, as well as its ``rule``.
See `issue #7787 <https://github.com/digital-asset/daml/pull/7787>`__.
CHANGELOG_END
* test use of getCommandsInFlight in an updateState
* inexplicable parse error
* Internal no longer exports queryContractId
* let free interpreter suspend on a SubmitRequest
* move UnfoldState to separate file, start a custom flow for flatmapping to it
* proper handlers, getting tangled in the mutable state of it all
* tests and conversion utilities for UnfoldState
* working UnfoldState flow with far fewer tricky details, passes test
* note on laziness
* missed t update
* test empty lists better
* add bifunctor for UnfoldState
* refactor message filtering
* split message parsing from the fold
* add copyright headers to new files
* let the flow from an UnfoldState emit the states
* test flatMapConcatStates's special output semantics
* pass submit's Future back to the holder of the SubmitRequest
* parallel submission stage, here with limit 8
* feed a stream of SubmitRequests to the command submission stage
* don't fail request submit stream stage on submit failure
- There was no way before for submission errors to report themselves to the free
interpreter, which just saw Unit; now we don't drop the Future on the floor,
so we properly turn "normal" submission errors into success, and actually
propagate any "real" errors to the stream instead of logging.
* note that initialState command submissions are not throttled or backpressured
* add the parallel submission size to the failure queue size
- defense against deadlock should we later change to propagate enqueuing
failures to the stream
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* clean up the initial namespace of getTriggerSink
* Configure a token verifier on the auth middleware
changelog_begin
changelog_end
* Verify the token in the auth middleware
changelog_begin
changelog_end
* Test token verification in auth middleware
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Requires authorization for `readAs:<party>` claims for the list and
status endpoints of the trigger service. In case of list the party is
provided by the request entity. However, in case of status the party is
determined by querying for the running trigger instance in the same way
as for the stop endpoint.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* queryContractId and queryContractKey, trivially
* add changelog
CHANGELOG_BEGIN
- [Triggers] Two new functions are available for querying the ACS:
``queryContractId``, for looking up a contract by ID, and ``queryContractKey``
for looking one up by key.
See `issue #7726 <https://github.com/digital-asset/daml/pull/7726>`__.
CHANGELOG_END
* more efficient, direct queryContractId implementation
* flip getContractById's arguments, avoid 'flip'
* test queryContractKey
* test queryContractId
* resources: Move builders into //ledger/ledger-resources.
Keep the actual constructors in a trait, but instantiate it when working
with ledger code.
This allows us to later introduce an extra "context" type parameter to
ResourceOwner.
* resources-akka: Move the builders in to //ledger/ledger-resources.
* resources: Introduce an abstract `Context` parameter for owners.
This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.
This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.
Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.
CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
`Context`, which is filled in by the corresponding `Context` class in
the _ledger-resources_ dependency. This allows us to pass extra
information through resource acquisition.
CHANGELOG_END
* ledger-resources: Move `ResourceOwner` here from `resources`.
* ledger-resources: Remove dependencies from outside //ledger.
* ledger-resource: Wrap the acquisition execution context in `Context`.
So we can add a logging context to it.
* resources: Pass the Context, not the ExecutionContext, to Resource.
* Avoid importing `HasExecutionContext`.
* ledger-resources: Publish to Maven Central.
* resources: Make the small changes suggested by @stefanobaghino-da.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* ledger-resources: Pull out a trait for test resource contexts.
Saves a few lines of code.
* Restore some imports that were accidentally wildcarded.
* resources: Replace an `implicit def` with a couple of imports.
* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.
Try and use the right execution context where possible.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Authorize trigger service on middleware
changelog_begin
changelog_end
* Trigger service auth callback handler
* Forward token
* Do not pin the application ID in the access token
The trigger service will assign an individual application ID to each
trigger based on its UUID. Requiring tokens on the granularity of
application IDs would break the idea of storing the token in a cookie to
be able to use it across multiple requests.
changelog_begin
changelog_end
* todo persist trigger token
* Add a state parameter to middleware login
* add documentation comments
* typo
* fmt
* Align Party type between middleware and trigger service
The middleware was using `com.daml.lf.data.Ref.Party` while the trigger
service is using `com.daml.ledger.api.refinements.ApiTypes.Party` which
requires conversions. This aligns the types to avoid such conversions.
* optional application id in oauth2 test server
* align party types
* configure auth middleware in trigger service tests
* handle empty cookie header
* follow redirects in trigger service tests
* keep track of cookies
* keep track of cookies
* Replace any previous Cookie header
Otherwise on old daml-ledger-token cookie might persist and be preferred
over a newly added instance.
* DEBUG
* Configure test ledger client readAs claims
* fmt
* docstrings
* remove debug output
* Avoid endless redirect loops
When the replay still fails to authorize on the middleware then we do
not want to attempt another login flow.
* Store callback routes in authCallbacks
* fmt
* Push AuthTestConfig into test target
https://github.com/digital-asset/daml/pull/7654#discussion_r506510193
* Unbind oauth2 server after middleware
https://github.com/digital-asset/daml/pull/7654/files#r506513251
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* set doobie version to 0.9.2 and rerun maven pin
* port extractor and some of JSON API
* repin maven
* use doobie's own builder compatibility where required
* use probably bad derivations to supply Blockers where transactEC was required
- The point of using Blocker instead of ExecutionContext seems to be to
especially emphasize to API users that it isn't appropriate to use an
ExecutionContext with ordinary behavior. That is what we have done, which
should probably change, but just compiling for now.
* fix fragment inspection test for internal restructuring
- This test depends on implementation details of Doobie, so naturally it must be
altered when that runs. Fortunately, it's been made easier by the changes
in this upgrade.
* allow 256 blockers for navigator transaction blocker, like the global EC
* allow as many blockers as the pool size for trigger service
- The transactor shouldn't share ExecutionContext for transactions with the
caller, so we set up a new one based on configured pool size.
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* allow updating user state from high-level trigger rule
* fix tests and examples for new rule signature
* fix doc for new rule signature
* add changelog
CHANGELOG_BEGIN
- [Triggers] ``rule`` can now ``get`` and ``put`` the user-defined state just
like ``updateState`` does. It no longer accepts the state as an argument.
You can port your rule function types by replacing ``s -> TriggerA`` with
``TriggerA s``, removing the last argument from the function, and using
``get`` to retrieve the state at the beginning of the ``rule``'s ``do`` block,
if needed.
See `issue #7674 <https://github.com/digital-asset/daml/pull/7674>`__.
CHANGELOG_END
* test that rule userState changes propagate all the way out
* adapt compatibility filtering to #7681
* add ACS reader to TriggerA and TriggerStateA
* propagate changes in TriggerA, TriggerStateA structure
* allow query to be used in updateState and rule
* remove getTemplates
* remove ACS argument from updateState and rule
* fix type parameter order on query
* use query function in all tests and examples
* replace getContracts with query in documentation
* use wildcards instead of otherwise
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* rename TriggerStateA to TriggerUpdateA
- suggested by @cocreature; thanks
* missed renamings of TriggerStateA to TriggerUpdateA
- suggested by @cocreature; thanks
* make the meaning of the rlift functions clearer
* make initialize a TriggerInitializeA instead of a function; remove getContracts
* update tests and examples for new initialize signature
CHANGELOG_BEGIN
- [Triggers] Trigger ``updateState``, ``rule``, and ``initialize`` functions no
longer accept an ``ACS`` argument; instead, they must use the ``query`` action
to query the ACS, similar to the same function in DAML Script.
See `issue #7632 <https://github.com/digital-asset/daml/pull/7632>`__.
CHANGELOG_END
* disable older compatibility trigger builds
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add ActionState to the standard library
* use 1 ActionState, 1 get, 1 put in low-level trigger library
* introduce TriggerStateA for updateState
* fix tests and examples for new updateState signature
CHANGELOG_BEGIN
- [Triggers] The ``updateState`` function now returns a ``TriggerStateA``. This
is an action like ``TriggerA``, but doesn't permit emitting commands. Instead
of taking the state as an argument and returning a new state, you can
manipulate the state with ``get``, ``put``, and ``modify``. Any existing
``updateState`` can be ported by replacing ``s -> expr`` in the lambda
expression with ``-> modify $ \s ->``, and then made to look nicer from there
as desired.
See `issue #7621 <https://github.com/digital-asset/daml/pull/7621>`__.
CHANGELOG_END
* some DAML docs for updateState and TriggerStateA
* add getCommandsInFlight as a TriggerA action
* immediately update commandsInFlight on emitCommands
- delay until the rule had returned made sense before submitCommands was launched
immediately; this also makes `getCommandsInFlight` more sensible
* remove commands-in-flight argument from high-level Trigger rule
CHANGELOG_BEGIN
- [Triggers] The "commands in flight" or ``Map CommandId [Command]`` argument has been
removed from high-level trigger ``rule`` functions; instead, the current
commands-in-flight can be retrieved with the new ``getCommandsInFlight`` function, which
can be done immediately at the beginning of the rule's ``do`` block to preserve exact
existing trigger behavior.
See `issue #7600 <https://github.com/digital-asset/daml/pull/7600>`__.
CHANGELOG_END
* remove commands-in-flight argument from trigger tests
* update doc examples and copy in compatibility
- compatibility change will most likely entail another flag day in compatibility tests,
replacing last_pre_7456_trigger_version
* update doc text for getCommandsInFlight
* test that getCommandsInFlight gets updated during the rule
* flag day for trigger compatibility tests
sdk-version: 1.7.0-snapshot.20201006.5358.0.0c1cadcf
File: src/CopyTrigger.daml
Hidden: no
Range: 55:11-55:19
Source: typecheck
Severity: DsError
Message:
src/CopyTrigger.daml:55:12: error:
• Couldn't match type ‘TriggerA ()’ with ‘() -> TriggerA ()’
Expected type: Party
-> DA.Next.Map.Map CommandId [Command] -> () -> TriggerA ()
Actual type: Party -> ACS -> () -> TriggerA ()
• In the ‘rule’ field of a record
In the expression:
Trigger
{initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}
In an equation for ‘copyTrigger’:
copyTrigger
= Trigger
{initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}
* match docs on TriggerAState to current usage of these fields
* remove emittedCommands, as commandsInFlight is now kept up-to-date
* zoomed from where?
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* make TriggerF's Submit emit a command ID instead of accepting one
- LowLevel updated
- interpreter in Scala updated, preserving Message semantics as closely as possible
* remove nextCommandId and all management from high-level Trigger API implementation
* a note about TriggerF and simulateRule
* port submitCommands calls from tests
CHANGELOG_BEGIN
- [Triggers] The CommandIds as accessed from trigger DAML code are now exactly the command
IDs used in command submission to the ledger; as such, they will vary randomly from run
to run of the trigger rule. To enable this, the low-level ``submitCommands`` function
no longer accepts a command ID, instead returning one; there is no change to the
corresponding high-level ``emitCommands`` function, so high-level triggers should only
see improved and easier-to-understand logging.
See `issue #7587 <https://github.com/digital-asset/daml/pull/7587>`__.
CHANGELOG_END
* port Scala-side assertion of trigger test
* rename commandIdMap to commandIdsUsed
* test that command IDs are like the ones we expect from the runner
* Make /login endpoint compatible to auth0
- Make the authorization and token endpoints configurable
- Use `application/x-www-form-urlencoded` as specified in
https://tools.ietf.org/html/rfc6749#section-4.1.3
- Check the status code of the token endpoint response
- Fix the type of the token's `expires_in` field
(`Int` instead of `String)
changelog_begin
changelog_end
* Add Auth0 testing instructions
* Use native application type on Auth0
* scope to claims mapping todo note
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* conservatively move daml-script, trigger SValue interpreters to common library
* introduce expect and JavaList pattern for converters
* clean up trigger Converter Command interpretation
* add Church Free monad
* add an action language for trigger updates
* add expectE to remove some of the joins
* convert more of the converters to expect
* tool for unrolling Free/Roll
* split handleStepResult up and clean up its pattern
* handleStepFreeResult to interpret TriggerF
* replace Free Church with Pure/Roll free from Script
* newtype for ActionTrigger
* replace update in low-level Trigger with Free TriggerF
* submit one Commands at a time
* boolean blindness strikes again
* log missed TriggerF steps
* comment actual Submit contents
* match #7501 fromPureSExpr sig change in 00b80b8ea3
* avoid using forwardPort in runTrigger
* push State back into DAML, so it can be excluded from the action list
* push Message back into DAML, unifying the action language for initialState and update
* bringing TriggerF into initial state
* really add TriggerF into initial state, with all ports, tested
* add ActionTrigger class, express initialState in its terms
* add all TriggerF actions to existing TriggerA
* Trigger.rule will no longer have Time argument
* rename getS, setS to get, put, matching C.M.T.State from transformers
* make high-level Rule evaluate to the underlying TriggerF sequence
* Assert's testRule doesn't have a transform yet
* move DamlTuple2 to common converter library
- suggested by @cocreature; thanks
* combine the two Frees, provide from Script
* remove time argument from integration tests
CHANGELOG_BEGIN
- [Triggers] The ``Time`` argument was removed from the trigger rule function; instead, it
can be fetched within the ``TriggerA`` ``do`` block by ``getTime``, as with ``Update``
and ``Scenario``. The ``LowLevel`` trigger interface has been redesigned; such triggers
need to be rewritten or ported to high-level triggers.
See `issue #7456 <https://github.com/digital-asset/daml/pull/7456>`_.
CHANGELOG_END
* add trigger rule simulator to support Assert module
* missed new Free module
- left in script per @cocreature
* remove retract as we ended up using foldFree for that purpose instead
- suggested by @cocreature; thanks
* throw ConverterException instead of RuntimeException
- suggested by @cocreature; thanks
* remove Time argument from coin-upgrade-trigger
* port trigger service tests
* port trigger scenario test
* put TriggerSetup and TriggerRule into LowLevel.Trigger instead of unboxed Free
- suggested by @cocreature; thanks
* remove Time argument from trigger compatibility test
* submit commands as soon as each `emitCommands` is sequenced
- we still collect a list, but only for tracking commandsInFlight
* filter out compatibility tests for triggers before now
* remove commented imports, libraries from new shared converter
* make the TriggerF interpreter tail-recursive
* remove unused compatibility trait
* add back new state logging
* remove refactoring comment
* rewrite some LowLevel initialStates in do
* hide Daml.Script.Free from docs
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* remove forwardPortInitialState
- suggested by @cocreature; thanks
* manually port low-level updates
- suggested by @cocreature; thanks
* remove forwardPort
- suggested by @cocreature; thanks
* fail faster on unrecognized TriggerF
- suggested by @cocreature; thanks
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add 'alter' function to DA.Map
CHANGELOG_BEGIN
CHANGELOG_END
* swap arguments for alter
- suggested by @cocreature; thanks
* alter moved to stdlib, remove from trigger lib
* use markdown formatting for alter doc
- suggested by @sofiafaro-da; thanks
* fix code block formatting
- suggested by @sofiafaro-da; thanks
* add Ord (ContractId a) instance
* change inner list in activeContracts to map
* adapt Internal module to new map
* support rest of Trigger daml lib
* rename adjust to alter, its Data.Map name
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* distribute type arguments for ACS activeContracts
- suggested by @cocreature; thanks
* swap arguments for alter
- suggested by @cocreature; thanks
* reformat groupActiveContracts signature
- suggested by @cocreature; thanks
* Add design doc for authentication in the trigger service
This is a draft of how I currently imagine authentication in the
trigger service to work. Since the authentication middleware has to be
pluggable in the end anyway we need public documentation in the end
anyway and I find this much easier to manage than a google doc outside
of the repo which just never sees updates.
changelog_begin
changelog_end
* Update triggers/service/authentication.md
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* use GenMaps for trigger ACS
* Next removed for 1.dev
* temp port trigger test code to 1.dev only
* run trigger tests on 1.dev only
* move pending back to TextMap
* include trigger service in the 1.dev test lineup
- it takes >2min, so shouldn't be permanent
* add Ord TypeRep and Ord TemplateTypeRep when possible
* swap names in Internal to reduce the diff
* try to enable cpp for triggers compilation
$ bazel build //triggers/daml:daml-trigger-1.dev
<snip>
File: daml/Daml/Trigger/Internal.daml
Hidden: no
Range: 103:-1-103:-1
Source: CPP
Severity: DsError
Message: 22 in hpp-0.6.1:Hpp.CmdLine
File: daml/Daml/Trigger.daml
Hidden: no
Range: 103:-1-103:-1
Source: CPP
Severity: DsError
Message: 22 in hpp-0.6.1:Hpp.CmdLine
ERROR: Creation of DAR file failed.
<snip>
* remove problematic options for invoking cpp
hpp: Couldn't open input file: -Werror
CallStack (from HasCallStack):
error, called at src/Hpp/CmdLine.hs:103:22 in hpp-0.6.1:Hpp.CmdLine
* enough cpp so default and 1.dev triggers compile
* cpp needed for docs as well
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* return trigger service to testing against sdk default lf version
* run trigger integration test against sdk default and LF 1.dev
* return trigger scenario test to SDK default LF version
* avoid import warnings in trigger lib
* Windows manifests a missing file differently. Hilarious
* Generate hoogle docs for daml script/triggers
This PR switches over the documentation generation for daml script and
daml triggers to the multi-page format we already use for the standard
library and extends it to also generate hoogle documentation.
All 3 hoogle files are combined in a single hoogle_db.tar.gz archive.
Since the location in the multi-page format is different, I’ve added
redirects.
I verified locally, that I can generate the hoogle database and that
the links point to the right places.
changelog_begin
changelog_end
* Fix baseurl for daml-stdlib
changelog_begin
changelog_end
This is clearly a breaking change but triggers are still alpha so we
can get away this (confirmed with Bernhard).
changelog_begin
- [DAML Trigger] Daml.Trigger.Assert now uses DAML Script instead of scenarios.
changelog_end
* get a LoggingContext into the TriggerRunnerImpl
* make some implicits more implicitly scoped and explicitly ascribed
* make some private/final markings
* most of JsonFormat[Identifier] is in companion
* experimental LoggingContext with phantom type
* ActorContext#log isn't really doing that much
* more details of LoggingContextOf
* make LoggingContextOf compile
* add trigger message logging, yet without context
* fix parent compile errors
* use Config as the phantom for its own logging extensions
* LocalDateTimeFormat cleanup
* switch TriggerRunner to contextual logging
* add trigger definition ID to logs
* log trigger-submitted commands, fix trigger test compile
* log trigger stopping and DAR uploads
* add context to PostStop/PreRestart logs
* add changelog
CHANGELOG_BEGIN
- [Triggers] More detailed logging of trigger actions and trigger service actions.
See `issue #7205 <https://github.com/digital-asset/daml/pull/7205>`_.
CHANGELOG_END
* missed copyright header
* switch to Unit, scala/bug#9240 fixed
* ledger-api-client: `maxInboundMessageSize` -> `maxInboundMetadataSize`.
CHANGELOG_BEGIN
- [Scala Bindings] Rename a field in the ``LedgerClientConfiguration``
to ``maxInboundMetadataSize``, to match the builder Netty channel
builder. It was incorrectly named ``maxInboundMessageSize``, which is
a different channel property that configures the maximum message size,
not the header size.
CHANGELOG_END
* ledger-api-client: Introduce a `maxInboundMessageSize` config property.
We use this a lot; easier if it's in the configuration.
CHANGELOG_BEGIN
- [Scala Bindings] Replace the
``LedgerClientConfiguration.maxInboundMessageSize`` property with a
new one that represents the maximum size of the response body.
CHANGELOG_END
* Add a dummy OAuth2 authorization server
This is intended for use in the trigger service integration tests not
as an artifact that we ship to users.
changelog_begin
changelog_end
* Address review feedback
changelog_begin
changelog_end
* Remove authentication service from trigger service
This rips out the integration with ref-ledger-authenticator and
together with that the weird http basic auth which doesn’t actually
work for ref-ledger-authenticator outside of its testing mode and also
includes some very sketchy stuff like storing user
passwords (encrypted but still).
While the exact details of the new approach towards auth are still not
completely clear, it is clear that it is going to be sufficiently
different from what we did before that starting from a clean slate is
easier.
changelog_begin
- [Trigger Service] Parties are now specified in request bodies as
opposed to via HTTP Basic auth. This is done in preparation for
running the trigger service against authenticated ledgers.
changelog_end
* Fix integration tests
changelog_begin
changelog_end
* depend on flyway in trigger service; duplicate FlywayMigrations from sandbox
* bring in postgres/h2 detection, other data-source elements needed; use scalalogging
* use DataSource instead of DriverManager for DbTriggerDao, pass to flyway too
- initialization order isn't quite right yet
* hardcode postgres in triggers flyway path
* found an init order that doesn't fail in tests
* better pools for production, and don't forget to close them
* move most init SQL to first migration file
* make logHandler implicit again
- seems to have been an accidental implication of #6334
* move remaining create to V1__Init.sql
* use Prod poolSize
* prove that PostStop signals are never seen (in tests)
* move all the shutdown to Stop message
* remove dead code in DbFlywayMigrations
* always "initialize" database
CHANGELOG_BEGIN
- [Trigger Service] The database format has changed to allow migrations in
future releases. Databases are always initialized or migrated to the current
version on start, so use of ``--init-db`` is no longer required.
See `issue #7092 <https://github.com/digital-asset/daml/pull/7092>`__.
CHANGELOG_END
* it should always be obvious when there is no escape
* remove logs from experiments
* remove receiveSignal PostStop
* remove unused imports
* ledger-api-client: Add integration tests for the simple stuff.
* sandbox-common: Make `SandboxFixtureWithAuth` a mixin.
This makes it useful with `SandboxNextFixture` as well as
`SandboxFixture`.
Also, add types to non-private fields and methods, and make more fields
protected rather than public.
* ledger-api-client: Add tests to make sure the token is passed through.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-common: Tokens are for auth, not auth.
fixes#7114
This PR changes the Show instance of ContractId and flips the switch
on triggers and DAML Script to run in off-ledger mode.
It also adds a test that for DAML Script we actually get back the
correct contract id.
There is a bit of a design decision here in how we want to print
contract ids, so let me list the options I considered. $cid will stand
for the actual cid and all options are wrapped in markdown inline
code.
1. `"$cid"`. Indistinguishable from string. Suggests that there might
be an IsString instance for ContractId.
2. `<$cid>`. Matches the dummy `<contract-id>` but it’s not a dummy so
I don’t think matching that is benefitial.
3. `$cid`. Easy to spot (contract ids start with # and have no
spaces), clearly not a string but might look slightly weird.
changelog_begin
- [DAML Script/DAML Triggers] When using DAML-LF 1.dev, the `Show` instance of `ContractId` will now display the actual contract id instead of a dummy `<contract-id>` value. Note that this only applies to DAML Script and DAML Triggers not to ledger code.
changelog_end
* factor --address, --http-port, --port-file options from http-json to cli-opts
- enabling reuse in trigger service
* use cli-opts for address and http-port options in Trigger service
* mark ServiceConfig and some defaults private
* use --address option to set up server
* document Setter
* test --address option is parsed
* missing (c) headers
* add changelog
CHANGELOG_BEGIN
- [Trigger Service] Accepts a new ``--address`` option to listen for HTTP connections on
interfaces other than localhost, such as ``0.0.0.0`` for all addresses.
See `issue #7090 <https://github.com/digital-asset/daml/pull/7090>`__.
CHANGELOG_END
* test only 'restart triggers with update errors' in CI, 100 times
CHANGELOG_BEGIN
CHANGELOG_END
* build fewer things and try 1000 runs
* test restart after shutdown, 200 times
* distrust the log
* report which withTriggerService is being used to provide log dividers
* log when TriggerStarting, TriggerStarted received in 'starting' state
- In mainline, we only add logs when we add triggers in the 'running'
stat in Server.apply. Let's see if we aren't sending these messages
to the `Server.start` actor when it is still in the 'starting' loop
* log same thing in starting and running states
* revert build experiment and extra logs
- keep the fixture location log, as it's a very useful divider when
parsing test results
* turn on NonUnitStatements in trigger runner, service main code
* add some friendly discards
* add a not-so-friendly discard
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* one more discard to clarify an Await.ready's type
* explain unbind issue
fixes#7030
This deliberately ignores the trigger service. The main reason for
setting the ApplicationId at the moment is authentication and this is
still very WIP in the trigger service, so I don’t think it makes sense
to add this in some form to the API until that has settled.
changelog_begin
- [DAML Trigger] You can now configure the application id via
`--application-id` in `daml trigger`. This is primarily useful if
you run with authentication.
changelog_end
* remove unused definitions, params, args from ledger API Scala code
CHANGELOG_BEGIN
- [Ledger API] withTimeProvider removed from CommandClient; this method
has done nothing since the new ledger time model was introduced in
1.0.0. See `issue #6985 <https://github.com/digital-asset/daml/pull/6985>`__.
CHANGELOG_END
* percolate withTimeProvider and label removal elsewhere
* reenable 'restart triggers after shutdown'
CHANGELOG_BEGIN
CHANGELOG_END
* wait for everything to shut down before completing a withTriggerService fixture
- similar to a change to HttpServiceFixture.withHttpService in #4593,
but without the suppression of shutdown errors
* label the WithDb tests
* in CI, test only 'recover packages after shutdown', 50 times
* experiment: Process#destroy appears to be async
* is it in the in-between period?
* partial -> total
* replace some booleans with assertions for better error reporting
* make triggerLog concurrent
* close channel and file in other error cases for port locking
- suggested by @leo-da; thanks
* use port locking instead of port 0 for trigger service fixtures
* destroy one service at a time
* missed continuation in build script
* use assertion language for "restart triggers with update errors"
* Revert "is it in the in-between period?"
This reverts commit 211ebfe9d2.
* use better assertion language for "restart triggers with update errors"
* restore full CI build
This was not only unnecessarily duplicated, it also had a bug where
`--crt` behaved like `--pem` instead of setting the cert chain.
I didn’t add new tests since it seems like the wrong place to test
config parsing of a library. We do have tests for TLS in general for
both DAML Script and DAML Triggers.
changelog_begin
changelog_end
This code has a few problems: the auth service test mode is not enabled, the
admin ledger should be separate from main ledger if auth is enabled, and the
retry strategy not executed.
The auth service is not used in these tests yet and we have an example
of how to do it correctly in the auth service fixture (and auth service
client tests). We can try to reuse that fixture instead or copy the
code over when necessary.
changelog_begin
changelog_end
* Trigger service tests: Make sure toxiproxy server is running before connecting client
changelog_begin
changelog_end
* Undo infuriatingly wrong camel case
* triggers: Use `FreePort.find()`.
* ports: Move `LockedFreePort` from postgresql-testing for reuse.
* triggers: Use `LockedFreePort` to avoid race conditions.
* ports + triggers: Move common port testing into the ports library.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-on-sql: Pull out a new `Main` object that wraps sandbox-classic.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-on-sql: Fail if a JDBC URL is not provided or not for PostgreSQL.
* sandbox-classic: Rename the conformance test H2 database.
* daml-on-sql + sandbox-classic: Report configuration errors cleanly.
This means letting `ProgramResource` catch the errors, log, and exit.
* daml-on-sql: Change the name logged on startup.
* daml-on-sql: Change the default participant ID.
* sandbox-common: Give the ledger name its own tagged string type.
* sandbox-classic: Generate random ledger IDs using the ledger name.
* daml-on-sql: Remove the banner, replacing it with a blank line.
* daml-on-sql: Enable strong seeding by default.
And weak seeding in the conformance tests.
* sandbox-classic: Move the ledger name to a separate parameter.
It's not really configurable.
* sandbox-classic: Move LedgerName from sandbox-common.
* daml-on-sql: Remove "-participant" from the participant ID.
* daml-on-sql: Use `Name` where possible.
* daml-on-sql: Make the ledger ID mandatory.
* Revert "sandbox-classic: Move LedgerName from sandbox-common."
This reverts commit 0dad1584a7.
* daml-on-sql: Print "DAML-on-SQL" in the CLI help, not "Sandbox".
* daml-on-sql + sandbox + sandbox-classic: Split out custom CLI parsing. (#6846)
* participant-state: Simplify naming the seeding modes.
* Method for service account request
* Extend domain and factor out sa/secure
* Methods to list service accounts and poll for service account
changelog_begin
changelog_end
* Run auth client tests 20 times
* Revert "Run auth client tests 20 times"
This reverts commit 5264ce6f30.
* Assert creds list for new service account is empty
* Request credential method
* Method to get a new credential id
* Method for fetching credential
* Login method (for ledger access token)
* Make authorize style consistent with rest
* Remove redundant test (subset of second test)
* Run auth client tests 20 times
* Fail future for http error responses and refine retry strategy
* Scalafmt
* Revert "Run auth client tests 20 times"
This reverts commit 2568dc6b47.
* Test getting service account before request
* sandbox-common: Move the ledgerIdMode up in the configuration.
* sandbox-classic: Move the configuration from sandbox-common.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix miscellaneous warnings caught by IntelliJ IDEA.
* set many extra scalac -Xlint options for all Scala projects
CHANGELOG_BEGIN
CHANGELOG_END
* move NoCopy to its own file
package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
trait NoCopy {
^
* move more traits, classes, and objects to proper packages
- note that `package` is itself a scoping construct, so if your reason
is the apparent aesthetic of placing a bunch of things in one `package
object`, that is easily remedied by deleting the `object` keyword
* fix some type-parameter-shadow warnings
- I'm generally in favor of sensible name-shadowing, following the
"deliberately hide variables that should not be accessed here" school
of thought. But I think type name shadowing isn't quite as valuable
and more likely to confuse than general variable shadowing, so have
experimentally linted it out.
Example warning:
EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
Offset defined in package v1. You may want to rename your type
parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
^
* fix more package-object-classes warnings
* fix an inaccessible warning
ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
^
* enable -Xlint:infer-any
- continuing the saga of #6116, #6132
* enable -explaintypes for more detailed type errors
* missed header for NoCopy; probably should have left it in the package file
* misspelling in comment
* revert -Xlint:doc-detached
- there are a lot of these fixes, and they are noisy, so shifting to a
separate PR
- thanks to @leo-da for pointing out
* WIP Call authorize before start trigger
* Intro auth service client
* Auth service client test setup
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt
* Use Spray JSON format and slightly better uri handling
* Get auth client test to work and fiddle with uris
* Remove changes to Server
* Add package for URI
* Format
* Copyrights
* Update sandbox packages
* Use default wallclock time for auth service sandbox
* Move public code into daml-integration-api
CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
* Add details about validity windows
* Reword description of auth service concepts
* Elaborate on auth flow dealing with token/credential expiry
CHANGELOG_BEGIN
CHANGELOG_END
* Revert ANF changes and add a testcase for evaluation order
After careful consideration, we decided that the change in evaluation
order that was accidentally introduced by the ANF changes should be
considered a breaking change or arguably even a bug and should not
land in 1.3.0.
Therefore, this PR reverts the following commits:
1. 353d0da6f7
2. a45b51042f
3. 04c7b2af7f
4. a624dd7242
5. b3aab72cee
Other PRs mostly had trivial merge conflicts that I resolved. The two
most interesting ones here are probably
1. https://github.com/digital-asset/daml/pull/6576 which was easy to
resolve and the change to return SEValue instead of SExpr is still
nice and useful even if we do not need the guarantees.
2. it https://github.com/digital-asset/daml/pull/6542 which required
some changes since the constructors changed. If you want to review
those changes in detail (they are pretty straightforward so not too
important), it’s probably easiest to check out this PR and run
```
git diff 2cd2a8f2a8
daml-lf/interpreter/src/main/scala/com/digitalasset/daml/lf/speedy/Compiler.scala
```
to see the diff to the parent commit of the first commit that
introduced ANF.
changelog_begin
changelog_end
* replace traverseU and sequenceU with traverse and sequence
- with -Ypartial-unification on, the extra Unapply typeclass lookup is
unnecessary
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* limit imports; we only need *> and void
The running trigger store is now written to synchronously by the server, so there's no need for retry logic.
Also assert equality of trigger ids rather than an arbitrary predicate, as it gives better error messages and we don't use other types of predicates for this function.
CHANGELOG_BEGIN
CHANGELOG_END
GHC has weird restriction on version numbers which damlc inserits so
we need to use `ghc_version` instead of `sdk_version`. That only makes
a difference for snapshot versions where the `-snapshot.` part is
replaced by `.`.
changelog_begin
changelog_end
It makes no sense to keep this at 0.0.1.
changelog_begin
- [DAML Script] The DAML Script library now has the version of the
corresponding SDK.
- [DAML Trigger] The DAML Trigger library now has the version of the
corresponding SDK.
changelog_end
* Add optional trigger id to startTrigger
CHANGELOG_BEGIN
CHANGELOG_END
* Read running trigger table in DbTriggerDao
And get rid of traverse type application!
(since we have partial unification on)
* Restart triggers in database
* Test for trigger restart on recovery
* Add order by clause and remove accidental overloading
* Add min/maxRestartInterval to configs and use in trigger restart strategy
* Adapt tests with triggers failing due to lack of network
changelog_begin
changelog_end
* Adapt tests for triggers with errors
* Remove comment about number of restarts
* Use a small initial restart interval for testing
* Remove old restart params
* Move maxInboundMessageSize to LedgerConfig
* Rename TriggerRunnerConfig to TriggerRestartConfig
* ANF transformation in Speedy.
The idea behind this PR is to transform speedy expressions into a simpler form where all non-atomic sub-expressions are made explicit by the introduction of let-forms. In particular, for the function-application form. These simpler forms allow the execution engine to take advantage of the atomic assumption, and often removes many additional execution steps. In particular the pushing of continuations to allow execution to continue after a compound expression has been reduced to a value.
changelog_begin
changelog_end
* improve comment
* inline functions relocateA/L
* remove comment about scalafmt
* remove commented out alterative def for transformLet1
* improve code by adding incr methods to DepthA/E
* remove (n == 0) special case in trackBindings
* clarify comment further
* improve validate/go to not consume stack for deeply right-nested let-expressions
* address comments from Remy: be private; use final case case; etc
* rename to unsafeCompilationPipeline
* add back some trailing commas
* remove commented-out debug line
* improve comment
* remove dev/debug code in compilationPipeline
* remove commented out code in SEAppGeneral.execute
* undo unrelated code improvement in SValue.scala
* fix compile. object Anf cannot be private
* Don't modify running trigger store via the trigger runner actor
changelog_begin
changelog_end
* Remove runner actor from RunningTrigger type
And parent actor from TriggerRunnerImpl
* Add to running trigger store on start request
* Update tests for new trigger store convention
* Fix test with network outage
* Move top level case classes to package object
changelog_begin
changelog_end
* Move Server Message classes to separate file
* Server in Server.scala and ServiceMain in ServiceMain.scala
* Copyright headers
The Server object and especially the apply method is extremely big and
hard to follow. There are a great many functions nested within the apply
method. This PR attempt to organize things, mostly by moving local
functions into methods of the Server class. I think this makes things
easier to follow, and I think it's more conventional.
Note that with this change I added some implicit parameters to the
Server constructor, which I think makes sense for these kinds of values
(contexts, sequencer factory, etc.) I also moved the Message trait to
the top level, but we will probably it its own file, following this PR.
There are a few other shallow tweaks that I made along the way.
Open to feedback on the high level structuring, as I'm not super
familiar with idiomatic design in Scala.
changelog_begin
changelog_end
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts
* add now-referenced scalaz-core where needed
* work around bad type signatures in scalatest Aggregating, Containing
* unused Any suppression
* work around bad partial-unification wrought by type alias
* remove unused Conversions import
- not required in 4f68cfc480 either, so unsure how it's survived this long
* work around Future.traverse; remove unused show import
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* remove unused bounds
* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed
* longer comment on what the options do
- suggested by @stefanobaghino-da; thanks
* forget Future.traverse, just use scalaz, it knows how to do this
* Move database initialization to Server apply method
* Reorder dao/server creation
* Read packages from database on startup
* Test starting a trigger after a shutdown
changelog_begin
changelog_end
This is needed to recover state after the service shuts down or crashes.
We add a method to the RunningTriggerDao to persistPackages. This only
does something in the case of a DbTriggerDao. In any case the Server
keeps a package map in memory as it's required to construct a trigger runner.
Uploads of existing packages is considered harmless.
changelog_begin
changelog_end
Avoid the use of Fragment.const which interprets raw strings as SQL
queries without any checks. Use the `sql` string interpolation which I
found out does the right thing with Strings and other simple types.
CHANGELOG_BEGIN
CHANGELOG_END
Trigger service: Refactor Server.addDar to take encoded dar
This is so we can write the encoded packages to the database if we have
one (without re-encoding them).
changelog_begin
changelog_end
In this PR we cleanup the constructor for the speedy Machine.
* We remove the `case` keyword since `Machine` is a stateful class,
* We replace the pre-existing builders with
+ one generic builder `Machine.apply`,
+ scenario specific builder,
CHANGELOG_BEGIN
CHANGELOG_END
* Trigger service: Refactor testing with and without database
Following the approach taken in the HTTP JSON API.
Define an AbstractTriggerServiceTest class with abstract jdbcConfig
member. This is instantiated with None and Some in the in-memory and
database test classes respectively. Tests for the individual modes can
be added in these concrete classes.
changelog_begin
changelog_end
* Move db init and destroy into beforeEach/afterEach to avoid concurrent tests clashing
Trigger Service: Refactor running trigger access using DAO interface
This introduces a RunningTriggerDao trait and two classes that implement
it, one for in-memory and one for database storage. This moves a lot of
code out of the Server class and into the DAO classes. The class chosen
at runtime depends on whether a JDBC config has been provided.
The database DAO also has methods for initialization and clean up used by
the server main function and the test fixture.
This is a pure refactoring (modulo some minor error message changes) so
functionality should stay the same.
CHANGELOG_BEGIN
CHANGELOG_END
* Add option based constructor for LedgerIdRequirement
changelog_begin
changelog_end
* Make option based consructor the default, deprecate old constructor
* Update with review comments
