* Middleware test resources
- Make OAuthServer accessible
- Don't access suiteResource directly in the test-suite
* OAuth2 test server test resources
- Make Server accessible
- Don't access suiteResource directly in the test-suite
* Use a deny list in OAuth2 test server
It is simpler to configure a set of disallowed parties and extend it on
demand than switching between blanket approval and allow list modes.
changelog_begin
changelog_end
* Handle admin claims in test server
Requests with admin claims are granted by default and denied if admin
requests have been disallowed.
* reset admin state in test fixtures
* test admin tokens in OAuth2 test server
* test admin claims in OAuth2 middleware
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
changelog_begin
changelog_end
This is to avoid a race condition where the old trigger runner may not
yet have completed shutdown by the time that the token has been
refreshed and we attempt to start a new trigger runner.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
This is necessary to at least attempt an upgrade to 2.13 and
generally, I want to keep our rulesets up2date. rules-scala forces the
version of scalatest so we have to bump that at the same time.
This requires changes to basically all Scala test suites since the
import structure has changed and a bunch of things (primarily
scalacheck support) got split out.
Apologies for the giant PR, I don’t see a way to keep it smaller.
changelog_begin
changelog_end
* Rename and separate auth targets and packages
Split out the API types `Request`/`Response` into their own Bazel
targets, so that the trigger service does not have to depend on the full
middleware and OAuth2 test server targets.
Also rename packages: Generic auth code goes into `com.daml.auth`,
OAuth2 specific code goes into `com.daml.auth.oauth2` or
`com.daml.auth.middleware.oauth2`.
changelog_begin
changelog_end
* Remove empty test-case
This is a left-over from #7226.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Enable adjustable clock in trigger service tests
changelog_begin
changelog_end
* Test user side token expiry
* Test service side token refresh
* Use AccessToken wrapper in TriggerRunnerImpl
* Store refresh token in trigger DB
* add refresh token to trigger runner config
* TriggerTokenExpired message to server
* TriggerTokenRefresh message to server
* refresh trigger token and update db
* Restart trigger with fresh token
* Test second token expiry
* Refresh token on running trigger
changelog_begin
* [Triggers] UNAUTHENTICATED errors will now terminate the trigger.
These errors are no longer available for handling in the trigger DAML
code. Instead, they are forwarded to the trigger service for handling,
e.g. access token refresh.
changelog_end
* todo note
* Move triggerRunnerName and getRunner into object
* Factor out token refresh
* Factor out getActiveContracts
* factor out create command
* Add logging to token refresh
* Handle token expiry in TriggerRunner
TriggerRunnerImpl throws a dedicated exception when it fails on an
expired access token (any unauthenticated error to be precise).
The TriggerRunner supervisor reacts to this child failure by
requesting a token refresh and restart on the trigger server and
stopping itself.
The trigger server requests a new access and refresh token on the auth
middleware and restarts the trigger.
This works around an issue with actor supervisors in akka-actor-typed.
A stop supervisor wrapped within a restart supervisor will not cause a
stop as expected. Instead, the restart supervisor will trigger as well
and restart the actor. The work around uses a custom behavior
interceptor to emulate the appropriate stop supervisors as closely as
possible. We cannot properly emulate ChildFailed signals this way, so
we use dedicated messages intead.
* throw --> Future.failedo
* getOrFail helper
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* kvutils: Use ScalaPB to generate a Scala JAR for daml_kvutils.proto.
* Bazel: Delete the unused `da_java_binary` rule, and inline `_wrap_rule`.
* Bazel: Factor out Java/Scala protobuf class generation into a helper.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lf/archive: Use `proto_jars`.
* Bazel: Remove the visibility modifier from `proto_jars`.
It's too confusing. Just make everything public.
* daml-lf/archive: Push protobuf source tarballs into `proto_jars`.
* Bazel: Add comments to the various parts of `proto_jars`.
* daml-assistant: Do unpleasant things with `location` in Bazel.
* Upgrade akka-http to 10.2
Follow up to #8048, I left out this upgrade to reduce noise and since
I wasn’t quite sure how involved it was going to be.
changelog_begin
changelog_end
* Reenable transparent HEAD requests
Apparently no longer on by default but we depend on this in waitForHttpServer
changelog_begin
changelog_end
* Upgrade akka and akka-http
Was chasing an issue somewhere and thought this might affect it in
some way. It didn’t but I might as well turn the upgrade into a PR.
changelog_begin
changelog_end
* Fix trigger service tests
changelog_begin
changelog_end
* Downgrade akka-http again
changelog_begin
changelog_end
* Upgrade akka-http again and fix tests
changelog_begin
changelog_end
* Cleanup trigger service
changelog_begin
changelog_end
fixes#7978
There is no new test in the trigger service since the existing test
for the custom application id already hits this. The difference is
that now the test authorization server will produce a token with the
application id set to what we request rather than the wildcard token
we used before.
changelog_begin
changelog_end
akka-http gets a bit unhappy if you block within requests and we also
use the unsafeToFuture in the JSON API so it seems sensible to do the
same in both.
I’ve moved out the initDb option out of the Server actor both because
it seemed cleaner than calling sys.exit in the actor and because it
was annoying to fit it in.
changelog_begin
changelog_end
This is to reduce the likelyhood of these tests timint out. Bazel will
generate a dedicated test target per `.scala` file. Meaning the tests
can run in parallel and each test target should have a shorter overall
runtime.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Obtain refresh token from Auth0
Auth0 requires the `offline_access` scope to be set to return a refresh
token.
See https://auth0.com/docs/tokens/refresh-tokens/get-refresh-tokens
Additionally, the `audience` claim needs to be set to obtain a JWT
access token and a refresh token.
See https://auth0.com/docs/tokens/refresh-tokens
changelog_begin
changelog_end
* Implement refresh endpoint on auth middleware
Following the refresh spec [1] and Auth0 documentation [2].
[1]: https://tools.ietf.org/html/rfc6749#section-6
[2]: https://auth0.com/docs/tokens/refresh-tokens/use-refresh-tokens
* Adapt Auth0 example configuration
Ignore any requests outside the ledger-api audience.
Don't throw on missing query fields. Otherwise the unhandled exception
would prevent unrelated requests from succeeding. E.g. token refresh
requests would always fail.
* Forward unauthorized/forbidden response on refresh
* re-use precomputed token payload
* Implement token refresh in auth test server
Reuses the association between authorization code and token payload to
associate refresh tokens and token payload.
Adds an expiry to the generated token to make tokens distinguishable
across refresh.
* obtain refresh token in test client
* Test auth server refresh token
* auth test server clock configurable
The clock used to define token expiry is configurable
* Override default clock in test fixture
* implement an adjustable clock
* Test token refresh with adjustable clock
* Test token expiry on /auth backend
* Test case for auth middleware /refresh endpoint
* handle malformed code/refresh token in auth server
* Forward client errors on middleware refresh
* Test middleware refresh failure
* Clarify meaning of offline accesss
* Remove redundant testing only comment
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* Make application ID configurable in trigger service
fixes#7671
changelog_begin
- [Trigger Service] The application id used by a trigger can now be
configured by an optional `applicationId` in the start request.
changelog_end
* Update triggers/service/src/main/scala/com/digitalasset/daml/lf/engine/trigger/Server.scala
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Cleanup trigger message types
This properly separates the messages accepted by the internal
TriggerRunnerImpl from the ones accepted by TriggerRunner. This also
shows a bug/redundancy where we had a bunch of code in
TriggerRunnerImpl to handle `Stop` but as evidenced by the types now,
we never actually send this message. We send it to TriggerRunner which
then tears down the child with it.
It also shuffles around the server message type to make it clear where
it belongs to.
And of course, I managed once again to include debugging output from a
previous PR so this is now removed as well 🤦
changelog_begin
changelog_end
* Update triggers/service/src/main/scala/com/digitalasset/daml/lf/engine/trigger/TriggerRunnerImpl.scala
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
We need to go via the methods in Reader to make sure that we get our
increased protobuf recursion limit. Otherwise, we fail when trying to
read from the database on anything non-trivial. I’ve verified that the
definition I’ve added is sufficient to break the default limit.
changelog_begin
- [Trigger Service] Fix a bug where complex models resulted in a fatal
error when restoring the state from the database due to an incorrect
protobuf recursion limit.
changelog_end
* Revamp trigger status endpoint
fixes#7951
The previous endpoint was a memory leak, nothing got persisted across
restarts and it omitted useful information like the metadata of the
trigger. The information is useful for testing, so I abstracted over
it so we can do what we did before in testing.
As for the endpoint, it now queries the actor for its current status
and only returns that and includes the metadata in the response.
As mentioned in #7951, I do think there is value in some kind of
history and potentially something including trace statements but I’d
like to do that properly instead of the hacky thing we have atm.
changelog_begin
- [Trigger Service] The trigger status endpoint /v1/triggers/:id now
includes metadata about the trigger like the party and the trigger
id. The logs field has been replaced by a status field.
changelog_end
* Fix ACS query test
changelog_begin
changelog_end
* Factor out oauth2 test server body into a class
changelog_begin
changelog_end
* move start into class
* mutable authorized parties
* add function sfor party authorization
* manage party authorization in trigger fixture
* enable deleting cookies
* Add access denied test cases
* Track readAs and actAs claims
* Enable ignored auth tests
* fmt
* add method comments
* Update triggers/service/auth/src/main/scala/com/daml/oauth/server/Server.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update triggers/service/auth/src/main/scala/com/daml/oauth/server/Server.scala
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Explain party authorization modes
* inline expected status codes
* fix closing brace
* use shouldBe instead of should equal
* Use shouldBe instead of should equal
* Explain revoking access and deleting cookies
* foreach requires a total function
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* restate the submit stage as a Flow and derived Sink
* take submit out of the trigger-to-submit flow
* type for the failures produced directly by command submission
* directly connect the msgSource failure queue to the submitter output
* parens
* slow down submission as we exceed max parallel submissions
* restricting alterF so it will be usable with ConcurrentMap
* disable buffer for the delay
* split out the delay function
* drafting a retry loop
* degenerate test for retry loop, factoring the forAllFuture utility
* map input to retrying properly
* make retrying accessible to tests
* test happy path and fix off-by-one
* further tests for retrying
* reveal that elements can get lost
* more determinism in test
* let failures block further elements from being attempted
- Previously failures would go into a separate queue, where they awaited expiry
of their delay and further initial upstream elements were given their first
tries. However, closing the upstream could mean that queue was dropped, and
detecting that situation is not trivial. So, instead, we don't use a separate
queue.
* plug retrying into the trigger submission flow
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* remove throttle; pendingCommandIds may leak
* report random parameter on failure
* revert comment about throttling
* explanation for fail in the error queue
- suggested by @cocreature; thanks
* oauth error response
* oauth server parties parameter
* implement request denial in oauth test server
* Test oauth test server for access denied
* test unauthorized party on auth middleware
* Handle OAuth2 login flow error response in auth middleware
* Forward login error to auth middleware client
* fmt
* fix server test
* fmt
* Handle login failure in trigger service
* Test unauthorized trigger start
* Cleanup authCallback
* Update authentication specification
- The auth middleware accepts an arbitrary callback URI on /login
- The auth middleware will forward OAuth2 authorization failures to the
client service.
changelog_begin
changelog_end
* Update triggers/service/authentication.md
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Check the trigger dao migrations digest
Following the example of the corresponding ledger on SQL tests.
The digests had to be updated as both of them had gone out of sync.
The init digest presumably due to the change in #7226 and the one for
adding the access token during review of #7890.
changelog_begin
changelog_end
* define abstract migrations test
* Use abstract migrations test in trigger service tests
* use abstract migrations test in ledger on SQL
* Retain check for number of .sql resources
* Factor out the hash-migrations script
* Consistent shell settings
Addressing review comment
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
fixes#7097
changelog_begin
- [Triggers] The trigger service now has a `--port-file` option
matching the corresponding option in the JSON API.
changelog_end
This allows you to write somewhat useful update functions since you
can inspect the transactions. Created and Archived are kept abstract
but we expose fromCreated/fromArchived to interact with them.
fixes#6968
changelog_begin
- [DAML Triggers] The `Daml.Trigger` module now reexports `Event`
which avoids having to import `Daml.Trigger.LowLevel` for implementing
a non-trivial `updateState` function.
changelog_end
Limiting this to a single one makes little sense and while you can
work around it by uploading more packages, that can be annoying during
development.
fixes#6332
changelog_begin
- [Triggers] The trigger service now accepts multiple `--dar`` options.
changelog_end
* Test all four trigger service configurations
- non-authenticated with in-memory backend
- non-authenticated with database backend
- authenticated with in-memory backend
- authenticated with database backend
changelog_begin
changelog_end
* add access token to trigger dao
* Simplify readRunningTriggers
* fmt
* Rename V2__Add_access_token.sha25 to V2__Add_access_token.sha256
* Apply suggestions from code review
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Getting a pattern match failure here if you accidentally pass in an
archived contract is a bit cruel and now that this is DAML Script
where we can handle the failure, we can do better.
changelog_begin
changelog_end
* Separate trigger service test fixture
Define a dedicated fixture per resource instead of generating all
resources in a single fixture function. This allows to control the
lifetime of resources individually.
Reuse the same sandbox and toxiproxy and auth middleware for the test
suite instead of restarting new resources for each test case. This is to
reduce overall test runtime.
changelog_begin
changelog_end
* fix exec context
changelog_begin
changelog_end
* Use owned resources where appropriate
Avoid Await result and use owned resources instead.
* Reduce test timeout
* Be more patient for Windows
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
We only expose the DAML values to users which don’t depend on this, so
we might as well avoid the unnecessary costs associated with verbose
mode especially since those are expected to increase in the
foreseeable future.
changelog_begin
changelog_end
* graph component for running UnfoldStates
* proper doc for flatMapConcatNode
* nicer internal signatures
* refactor TriggerMsg encoder
* restate trigger sink in the graph DSL
* UnfoldState functions that might be useful
* express trigger sink with graphed initial state
* fix doc comment
* add SourceShape2, a SourceShape with two outlets
* add toSource for UnfoldState
* test for iterator
* do not submit for initialState out-of-band; feed into the graph instead
* factor the forAll Future pattern for testing
* test flatMapConcatNode directly
* add changelog
CHANGELOG_BEGIN
- [Triggers] Each trigger will submit up to 8 commands at a time, rather than
submitting as fast as possible. This applies to submissions in progress,
rather than completed but pending submissions; the latter's limit is still
subject to the ledger's own limits.
See `issue #7812 <https://github.com/digital-asset/daml/pull/7812>`__.
CHANGELOG_END
* remove uncons and append, which were not needed
* log initial evaluated state again
* shape port aliases
- suggested by @cocreature; thanks
* Factor JWT verifier CLI flags
changelog_begin
changelog_end
* Use cli-opts in auth middleware
* Use cli-opts in sandbox cli
* Mark trigger service test as long
These have become prone to timeout on CI.
Increasing the size (timeout) is a temporary fix. A proper
solution is to a) not start a fresh sandbox per test-case and b)
separate the in-mem/db and no-auth/auth configrations into
separate Bazel test targets.
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* make ActionTriggerAny no longer have orphan instances
* add commands-in-flight reader to TriggerUpdateA
* make getCommandsInFlight a method shared by update and rule
* add changelog
CHANGELOG_BEGIN
- [Triggers] ``getCommandsInFlight`` may be used in a high-level trigger's
``updateState``, as well as its ``rule``.
See `issue #7787 <https://github.com/digital-asset/daml/pull/7787>`__.
CHANGELOG_END
* test use of getCommandsInFlight in an updateState
* inexplicable parse error
* Internal no longer exports queryContractId
* let free interpreter suspend on a SubmitRequest
* move UnfoldState to separate file, start a custom flow for flatmapping to it
* proper handlers, getting tangled in the mutable state of it all
* tests and conversion utilities for UnfoldState
* working UnfoldState flow with far fewer tricky details, passes test
* note on laziness
* missed t update
* test empty lists better
* add bifunctor for UnfoldState
* refactor message filtering
* split message parsing from the fold
* add copyright headers to new files
* let the flow from an UnfoldState emit the states
* test flatMapConcatStates's special output semantics
* pass submit's Future back to the holder of the SubmitRequest
* parallel submission stage, here with limit 8
* feed a stream of SubmitRequests to the command submission stage
* don't fail request submit stream stage on submit failure
- There was no way before for submission errors to report themselves to the free
interpreter, which just saw Unit; now we don't drop the Future on the floor,
so we properly turn "normal" submission errors into success, and actually
propagate any "real" errors to the stream instead of logging.
* note that initialState command submissions are not throttled or backpressured
* add the parallel submission size to the failure queue size
- defense against deadlock should we later change to propagate enqueuing
failures to the stream
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* clean up the initial namespace of getTriggerSink
* Configure a token verifier on the auth middleware
changelog_begin
changelog_end
* Verify the token in the auth middleware
changelog_begin
changelog_end
* Test token verification in auth middleware
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Requires authorization for `readAs:<party>` claims for the list and
status endpoints of the trigger service. In case of list the party is
provided by the request entity. However, in case of status the party is
determined by querying for the running trigger instance in the same way
as for the stop endpoint.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* queryContractId and queryContractKey, trivially
* add changelog
CHANGELOG_BEGIN
- [Triggers] Two new functions are available for querying the ACS:
``queryContractId``, for looking up a contract by ID, and ``queryContractKey``
for looking one up by key.
See `issue #7726 <https://github.com/digital-asset/daml/pull/7726>`__.
CHANGELOG_END
* more efficient, direct queryContractId implementation
* flip getContractById's arguments, avoid 'flip'
* test queryContractKey
* test queryContractId
* resources: Move builders into //ledger/ledger-resources.
Keep the actual constructors in a trait, but instantiate it when working
with ledger code.
This allows us to later introduce an extra "context" type parameter to
ResourceOwner.
* resources-akka: Move the builders in to //ledger/ledger-resources.
* resources: Introduce an abstract `Context` parameter for owners.
This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.
This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.
Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.
CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
`Context`, which is filled in by the corresponding `Context` class in
the _ledger-resources_ dependency. This allows us to pass extra
information through resource acquisition.
CHANGELOG_END
* ledger-resources: Move `ResourceOwner` here from `resources`.
* ledger-resources: Remove dependencies from outside //ledger.
* ledger-resource: Wrap the acquisition execution context in `Context`.
So we can add a logging context to it.
* resources: Pass the Context, not the ExecutionContext, to Resource.
* Avoid importing `HasExecutionContext`.
* ledger-resources: Publish to Maven Central.
* resources: Make the small changes suggested by @stefanobaghino-da.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* ledger-resources: Pull out a trait for test resource contexts.
Saves a few lines of code.
* Restore some imports that were accidentally wildcarded.
* resources: Replace an `implicit def` with a couple of imports.
* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.
Try and use the right execution context where possible.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Authorize trigger service on middleware
changelog_begin
changelog_end
* Trigger service auth callback handler
* Forward token
* Do not pin the application ID in the access token
The trigger service will assign an individual application ID to each
trigger based on its UUID. Requiring tokens on the granularity of
application IDs would break the idea of storing the token in a cookie to
be able to use it across multiple requests.
changelog_begin
changelog_end
* todo persist trigger token
* Add a state parameter to middleware login
* add documentation comments
* typo
* fmt
* Align Party type between middleware and trigger service
The middleware was using `com.daml.lf.data.Ref.Party` while the trigger
service is using `com.daml.ledger.api.refinements.ApiTypes.Party` which
requires conversions. This aligns the types to avoid such conversions.
* optional application id in oauth2 test server
* align party types
* configure auth middleware in trigger service tests
* handle empty cookie header
* follow redirects in trigger service tests
* keep track of cookies
* keep track of cookies
* Replace any previous Cookie header
Otherwise on old daml-ledger-token cookie might persist and be preferred
over a newly added instance.
* DEBUG
* Configure test ledger client readAs claims
* fmt
* docstrings
* remove debug output
* Avoid endless redirect loops
When the replay still fails to authorize on the middleware then we do
not want to attempt another login flow.
* Store callback routes in authCallbacks
* fmt
* Push AuthTestConfig into test target
https://github.com/digital-asset/daml/pull/7654#discussion_r506510193
* Unbind oauth2 server after middleware
https://github.com/digital-asset/daml/pull/7654/files#r506513251
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* set doobie version to 0.9.2 and rerun maven pin
* port extractor and some of JSON API
* repin maven
* use doobie's own builder compatibility where required
* use probably bad derivations to supply Blockers where transactEC was required
- The point of using Blocker instead of ExecutionContext seems to be to
especially emphasize to API users that it isn't appropriate to use an
ExecutionContext with ordinary behavior. That is what we have done, which
should probably change, but just compiling for now.
* fix fragment inspection test for internal restructuring
- This test depends on implementation details of Doobie, so naturally it must be
altered when that runs. Fortunately, it's been made easier by the changes
in this upgrade.
* allow 256 blockers for navigator transaction blocker, like the global EC
* allow as many blockers as the pool size for trigger service
- The transactor shouldn't share ExecutionContext for transactions with the
caller, so we set up a new one based on configured pool size.
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* allow updating user state from high-level trigger rule
* fix tests and examples for new rule signature
* fix doc for new rule signature
* add changelog
CHANGELOG_BEGIN
- [Triggers] ``rule`` can now ``get`` and ``put`` the user-defined state just
like ``updateState`` does. It no longer accepts the state as an argument.
You can port your rule function types by replacing ``s -> TriggerA`` with
``TriggerA s``, removing the last argument from the function, and using
``get`` to retrieve the state at the beginning of the ``rule``'s ``do`` block,
if needed.
See `issue #7674 <https://github.com/digital-asset/daml/pull/7674>`__.
CHANGELOG_END
* test that rule userState changes propagate all the way out
* adapt compatibility filtering to #7681
* add ACS reader to TriggerA and TriggerStateA
* propagate changes in TriggerA, TriggerStateA structure
* allow query to be used in updateState and rule
* remove getTemplates
* remove ACS argument from updateState and rule
* fix type parameter order on query
* use query function in all tests and examples
* replace getContracts with query in documentation
* use wildcards instead of otherwise
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* rename TriggerStateA to TriggerUpdateA
- suggested by @cocreature; thanks
* missed renamings of TriggerStateA to TriggerUpdateA
- suggested by @cocreature; thanks
* make the meaning of the rlift functions clearer
* make initialize a TriggerInitializeA instead of a function; remove getContracts
* update tests and examples for new initialize signature
CHANGELOG_BEGIN
- [Triggers] Trigger ``updateState``, ``rule``, and ``initialize`` functions no
longer accept an ``ACS`` argument; instead, they must use the ``query`` action
to query the ACS, similar to the same function in DAML Script.
See `issue #7632 <https://github.com/digital-asset/daml/pull/7632>`__.
CHANGELOG_END
* disable older compatibility trigger builds
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add ActionState to the standard library
* use 1 ActionState, 1 get, 1 put in low-level trigger library
* introduce TriggerStateA for updateState
* fix tests and examples for new updateState signature
CHANGELOG_BEGIN
- [Triggers] The ``updateState`` function now returns a ``TriggerStateA``. This
is an action like ``TriggerA``, but doesn't permit emitting commands. Instead
of taking the state as an argument and returning a new state, you can
manipulate the state with ``get``, ``put``, and ``modify``. Any existing
``updateState`` can be ported by replacing ``s -> expr`` in the lambda
expression with ``-> modify $ \s ->``, and then made to look nicer from there
as desired.
See `issue #7621 <https://github.com/digital-asset/daml/pull/7621>`__.
CHANGELOG_END
* some DAML docs for updateState and TriggerStateA
* add getCommandsInFlight as a TriggerA action
* immediately update commandsInFlight on emitCommands
- delay until the rule had returned made sense before submitCommands was launched
immediately; this also makes `getCommandsInFlight` more sensible
* remove commands-in-flight argument from high-level Trigger rule
CHANGELOG_BEGIN
- [Triggers] The "commands in flight" or ``Map CommandId [Command]`` argument has been
removed from high-level trigger ``rule`` functions; instead, the current
commands-in-flight can be retrieved with the new ``getCommandsInFlight`` function, which
can be done immediately at the beginning of the rule's ``do`` block to preserve exact
existing trigger behavior.
See `issue #7600 <https://github.com/digital-asset/daml/pull/7600>`__.
CHANGELOG_END
* remove commands-in-flight argument from trigger tests
* update doc examples and copy in compatibility
- compatibility change will most likely entail another flag day in compatibility tests,
replacing last_pre_7456_trigger_version
* update doc text for getCommandsInFlight
* test that getCommandsInFlight gets updated during the rule
* flag day for trigger compatibility tests
sdk-version: 1.7.0-snapshot.20201006.5358.0.0c1cadcf
File: src/CopyTrigger.daml
Hidden: no
Range: 55:11-55:19
Source: typecheck
Severity: DsError
Message:
src/CopyTrigger.daml:55:12: error:
• Couldn't match type ‘TriggerA ()’ with ‘() -> TriggerA ()’
Expected type: Party
-> DA.Next.Map.Map CommandId [Command] -> () -> TriggerA ()
Actual type: Party -> ACS -> () -> TriggerA ()
• In the ‘rule’ field of a record
In the expression:
Trigger
{initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}
In an equation for ‘copyTrigger’:
copyTrigger
= Trigger
{initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}
* match docs on TriggerAState to current usage of these fields
* remove emittedCommands, as commandsInFlight is now kept up-to-date
* zoomed from where?
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* make TriggerF's Submit emit a command ID instead of accepting one
- LowLevel updated
- interpreter in Scala updated, preserving Message semantics as closely as possible
* remove nextCommandId and all management from high-level Trigger API implementation
* a note about TriggerF and simulateRule
* port submitCommands calls from tests
CHANGELOG_BEGIN
- [Triggers] The CommandIds as accessed from trigger DAML code are now exactly the command
IDs used in command submission to the ledger; as such, they will vary randomly from run
to run of the trigger rule. To enable this, the low-level ``submitCommands`` function
no longer accepts a command ID, instead returning one; there is no change to the
corresponding high-level ``emitCommands`` function, so high-level triggers should only
see improved and easier-to-understand logging.
See `issue #7587 <https://github.com/digital-asset/daml/pull/7587>`__.
CHANGELOG_END
* port Scala-side assertion of trigger test
* rename commandIdMap to commandIdsUsed
* test that command IDs are like the ones we expect from the runner
* Make /login endpoint compatible to auth0
- Make the authorization and token endpoints configurable
- Use `application/x-www-form-urlencoded` as specified in
https://tools.ietf.org/html/rfc6749#section-4.1.3
- Check the status code of the token endpoint response
- Fix the type of the token's `expires_in` field
(`Int` instead of `String)
changelog_begin
changelog_end
* Add Auth0 testing instructions
* Use native application type on Auth0
* scope to claims mapping todo note
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* conservatively move daml-script, trigger SValue interpreters to common library
* introduce expect and JavaList pattern for converters
* clean up trigger Converter Command interpretation
* add Church Free monad
* add an action language for trigger updates
* add expectE to remove some of the joins
* convert more of the converters to expect
* tool for unrolling Free/Roll
* split handleStepResult up and clean up its pattern
* handleStepFreeResult to interpret TriggerF
* replace Free Church with Pure/Roll free from Script
* newtype for ActionTrigger
* replace update in low-level Trigger with Free TriggerF
* submit one Commands at a time
* boolean blindness strikes again
* log missed TriggerF steps
* comment actual Submit contents
* match #7501 fromPureSExpr sig change in 00b80b8ea3
* avoid using forwardPort in runTrigger
* push State back into DAML, so it can be excluded from the action list
* push Message back into DAML, unifying the action language for initialState and update
* bringing TriggerF into initial state
* really add TriggerF into initial state, with all ports, tested
* add ActionTrigger class, express initialState in its terms
* add all TriggerF actions to existing TriggerA
* Trigger.rule will no longer have Time argument
* rename getS, setS to get, put, matching C.M.T.State from transformers
* make high-level Rule evaluate to the underlying TriggerF sequence
* Assert's testRule doesn't have a transform yet
* move DamlTuple2 to common converter library
- suggested by @cocreature; thanks
* combine the two Frees, provide from Script
* remove time argument from integration tests
CHANGELOG_BEGIN
- [Triggers] The ``Time`` argument was removed from the trigger rule function; instead, it
can be fetched within the ``TriggerA`` ``do`` block by ``getTime``, as with ``Update``
and ``Scenario``. The ``LowLevel`` trigger interface has been redesigned; such triggers
need to be rewritten or ported to high-level triggers.
See `issue #7456 <https://github.com/digital-asset/daml/pull/7456>`_.
CHANGELOG_END
* add trigger rule simulator to support Assert module
* missed new Free module
- left in script per @cocreature
* remove retract as we ended up using foldFree for that purpose instead
- suggested by @cocreature; thanks
* throw ConverterException instead of RuntimeException
- suggested by @cocreature; thanks
* remove Time argument from coin-upgrade-trigger
* port trigger service tests
* port trigger scenario test
* put TriggerSetup and TriggerRule into LowLevel.Trigger instead of unboxed Free
- suggested by @cocreature; thanks
* remove Time argument from trigger compatibility test
* submit commands as soon as each `emitCommands` is sequenced
- we still collect a list, but only for tracking commandsInFlight
* filter out compatibility tests for triggers before now
* remove commented imports, libraries from new shared converter
* make the TriggerF interpreter tail-recursive
* remove unused compatibility trait
* add back new state logging
* remove refactoring comment
* rewrite some LowLevel initialStates in do
* hide Daml.Script.Free from docs
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* remove forwardPortInitialState
- suggested by @cocreature; thanks
* manually port low-level updates
- suggested by @cocreature; thanks
* remove forwardPort
- suggested by @cocreature; thanks
* fail faster on unrecognized TriggerF
- suggested by @cocreature; thanks
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add 'alter' function to DA.Map
CHANGELOG_BEGIN
CHANGELOG_END
* swap arguments for alter
- suggested by @cocreature; thanks
* alter moved to stdlib, remove from trigger lib
* use markdown formatting for alter doc
- suggested by @sofiafaro-da; thanks
* fix code block formatting
- suggested by @sofiafaro-da; thanks
* add Ord (ContractId a) instance
* change inner list in activeContracts to map
* adapt Internal module to new map
* support rest of Trigger daml lib
* rename adjust to alter, its Data.Map name
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* distribute type arguments for ACS activeContracts
- suggested by @cocreature; thanks
* swap arguments for alter
- suggested by @cocreature; thanks
* reformat groupActiveContracts signature
- suggested by @cocreature; thanks
* Add design doc for authentication in the trigger service
This is a draft of how I currently imagine authentication in the
trigger service to work. Since the authentication middleware has to be
pluggable in the end anyway we need public documentation in the end
anyway and I find this much easier to manage than a google doc outside
of the repo which just never sees updates.
changelog_begin
changelog_end
* Update triggers/service/authentication.md
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
Co-authored-by: Stephen Compall <stephen.compall@daml.com>
* use GenMaps for trigger ACS
* Next removed for 1.dev
* temp port trigger test code to 1.dev only
* run trigger tests on 1.dev only
* move pending back to TextMap
* include trigger service in the 1.dev test lineup
- it takes >2min, so shouldn't be permanent
* add Ord TypeRep and Ord TemplateTypeRep when possible
* swap names in Internal to reduce the diff
* try to enable cpp for triggers compilation
$ bazel build //triggers/daml:daml-trigger-1.dev
<snip>
File: daml/Daml/Trigger/Internal.daml
Hidden: no
Range: 103:-1-103:-1
Source: CPP
Severity: DsError
Message: 22 in hpp-0.6.1:Hpp.CmdLine
File: daml/Daml/Trigger.daml
Hidden: no
Range: 103:-1-103:-1
Source: CPP
Severity: DsError
Message: 22 in hpp-0.6.1:Hpp.CmdLine
ERROR: Creation of DAR file failed.
<snip>
* remove problematic options for invoking cpp
hpp: Couldn't open input file: -Werror
CallStack (from HasCallStack):
error, called at src/Hpp/CmdLine.hs:103:22 in hpp-0.6.1:Hpp.CmdLine
* enough cpp so default and 1.dev triggers compile
* cpp needed for docs as well
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* return trigger service to testing against sdk default lf version
* run trigger integration test against sdk default and LF 1.dev
* return trigger scenario test to SDK default LF version
* avoid import warnings in trigger lib
* Windows manifests a missing file differently. Hilarious
* Generate hoogle docs for daml script/triggers
This PR switches over the documentation generation for daml script and
daml triggers to the multi-page format we already use for the standard
library and extends it to also generate hoogle documentation.
All 3 hoogle files are combined in a single hoogle_db.tar.gz archive.
Since the location in the multi-page format is different, I’ve added
redirects.
I verified locally, that I can generate the hoogle database and that
the links point to the right places.
changelog_begin
changelog_end
* Fix baseurl for daml-stdlib
changelog_begin
changelog_end
This is clearly a breaking change but triggers are still alpha so we
can get away this (confirmed with Bernhard).
changelog_begin
- [DAML Trigger] Daml.Trigger.Assert now uses DAML Script instead of scenarios.
changelog_end
* get a LoggingContext into the TriggerRunnerImpl
* make some implicits more implicitly scoped and explicitly ascribed
* make some private/final markings
* most of JsonFormat[Identifier] is in companion
* experimental LoggingContext with phantom type
* ActorContext#log isn't really doing that much
* more details of LoggingContextOf
* make LoggingContextOf compile
* add trigger message logging, yet without context
* fix parent compile errors
* use Config as the phantom for its own logging extensions
* LocalDateTimeFormat cleanup
* switch TriggerRunner to contextual logging
* add trigger definition ID to logs
* log trigger-submitted commands, fix trigger test compile
* log trigger stopping and DAR uploads
* add context to PostStop/PreRestart logs
* add changelog
CHANGELOG_BEGIN
- [Triggers] More detailed logging of trigger actions and trigger service actions.
See `issue #7205 <https://github.com/digital-asset/daml/pull/7205>`_.
CHANGELOG_END
* missed copyright header
* switch to Unit, scala/bug#9240 fixed
* ledger-api-client: `maxInboundMessageSize` -> `maxInboundMetadataSize`.
CHANGELOG_BEGIN
- [Scala Bindings] Rename a field in the ``LedgerClientConfiguration``
to ``maxInboundMetadataSize``, to match the builder Netty channel
builder. It was incorrectly named ``maxInboundMessageSize``, which is
a different channel property that configures the maximum message size,
not the header size.
CHANGELOG_END
* ledger-api-client: Introduce a `maxInboundMessageSize` config property.
We use this a lot; easier if it's in the configuration.
CHANGELOG_BEGIN
- [Scala Bindings] Replace the
``LedgerClientConfiguration.maxInboundMessageSize`` property with a
new one that represents the maximum size of the response body.
CHANGELOG_END
* Add a dummy OAuth2 authorization server
This is intended for use in the trigger service integration tests not
as an artifact that we ship to users.
changelog_begin
changelog_end
* Address review feedback
changelog_begin
changelog_end
* Remove authentication service from trigger service
This rips out the integration with ref-ledger-authenticator and
together with that the weird http basic auth which doesn’t actually
work for ref-ledger-authenticator outside of its testing mode and also
includes some very sketchy stuff like storing user
passwords (encrypted but still).
While the exact details of the new approach towards auth are still not
completely clear, it is clear that it is going to be sufficiently
different from what we did before that starting from a clean slate is
easier.
changelog_begin
- [Trigger Service] Parties are now specified in request bodies as
opposed to via HTTP Basic auth. This is done in preparation for
running the trigger service against authenticated ledgers.
changelog_end
* Fix integration tests
changelog_begin
changelog_end
* depend on flyway in trigger service; duplicate FlywayMigrations from sandbox
* bring in postgres/h2 detection, other data-source elements needed; use scalalogging
* use DataSource instead of DriverManager for DbTriggerDao, pass to flyway too
- initialization order isn't quite right yet
* hardcode postgres in triggers flyway path
* found an init order that doesn't fail in tests
* better pools for production, and don't forget to close them
* move most init SQL to first migration file
* make logHandler implicit again
- seems to have been an accidental implication of #6334
* move remaining create to V1__Init.sql
* use Prod poolSize
* prove that PostStop signals are never seen (in tests)
* move all the shutdown to Stop message
* remove dead code in DbFlywayMigrations
* always "initialize" database
CHANGELOG_BEGIN
- [Trigger Service] The database format has changed to allow migrations in
future releases. Databases are always initialized or migrated to the current
version on start, so use of ``--init-db`` is no longer required.
See `issue #7092 <https://github.com/digital-asset/daml/pull/7092>`__.
CHANGELOG_END
* it should always be obvious when there is no escape
* remove logs from experiments
* remove receiveSignal PostStop
* remove unused imports
* ledger-api-client: Add integration tests for the simple stuff.
* sandbox-common: Make `SandboxFixtureWithAuth` a mixin.
This makes it useful with `SandboxNextFixture` as well as
`SandboxFixture`.
Also, add types to non-private fields and methods, and make more fields
protected rather than public.
* ledger-api-client: Add tests to make sure the token is passed through.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-common: Tokens are for auth, not auth.
fixes#7114
This PR changes the Show instance of ContractId and flips the switch
on triggers and DAML Script to run in off-ledger mode.
It also adds a test that for DAML Script we actually get back the
correct contract id.
There is a bit of a design decision here in how we want to print
contract ids, so let me list the options I considered. $cid will stand
for the actual cid and all options are wrapped in markdown inline
code.
1. `"$cid"`. Indistinguishable from string. Suggests that there might
be an IsString instance for ContractId.
2. `<$cid>`. Matches the dummy `<contract-id>` but it’s not a dummy so
I don’t think matching that is benefitial.
3. `$cid`. Easy to spot (contract ids start with # and have no
spaces), clearly not a string but might look slightly weird.
changelog_begin
- [DAML Script/DAML Triggers] When using DAML-LF 1.dev, the `Show` instance of `ContractId` will now display the actual contract id instead of a dummy `<contract-id>` value. Note that this only applies to DAML Script and DAML Triggers not to ledger code.
changelog_end
* factor --address, --http-port, --port-file options from http-json to cli-opts
- enabling reuse in trigger service
* use cli-opts for address and http-port options in Trigger service
* mark ServiceConfig and some defaults private
* use --address option to set up server
* document Setter
* test --address option is parsed
* missing (c) headers
* add changelog
CHANGELOG_BEGIN
- [Trigger Service] Accepts a new ``--address`` option to listen for HTTP connections on
interfaces other than localhost, such as ``0.0.0.0`` for all addresses.
See `issue #7090 <https://github.com/digital-asset/daml/pull/7090>`__.
CHANGELOG_END
* test only 'restart triggers with update errors' in CI, 100 times
CHANGELOG_BEGIN
CHANGELOG_END
* build fewer things and try 1000 runs
* test restart after shutdown, 200 times
* distrust the log
* report which withTriggerService is being used to provide log dividers
* log when TriggerStarting, TriggerStarted received in 'starting' state
- In mainline, we only add logs when we add triggers in the 'running'
stat in Server.apply. Let's see if we aren't sending these messages
to the `Server.start` actor when it is still in the 'starting' loop
* log same thing in starting and running states
* revert build experiment and extra logs
- keep the fixture location log, as it's a very useful divider when
parsing test results
* turn on NonUnitStatements in trigger runner, service main code
* add some friendly discards
* add a not-so-friendly discard
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* one more discard to clarify an Await.ready's type
* explain unbind issue
fixes#7030
This deliberately ignores the trigger service. The main reason for
setting the ApplicationId at the moment is authentication and this is
still very WIP in the trigger service, so I don’t think it makes sense
to add this in some form to the API until that has settled.
changelog_begin
- [DAML Trigger] You can now configure the application id via
`--application-id` in `daml trigger`. This is primarily useful if
you run with authentication.
changelog_end
* remove unused definitions, params, args from ledger API Scala code
CHANGELOG_BEGIN
- [Ledger API] withTimeProvider removed from CommandClient; this method
has done nothing since the new ledger time model was introduced in
1.0.0. See `issue #6985 <https://github.com/digital-asset/daml/pull/6985>`__.
CHANGELOG_END
* percolate withTimeProvider and label removal elsewhere
* reenable 'restart triggers after shutdown'
CHANGELOG_BEGIN
CHANGELOG_END
* wait for everything to shut down before completing a withTriggerService fixture
- similar to a change to HttpServiceFixture.withHttpService in #4593,
but without the suppression of shutdown errors
* label the WithDb tests
* in CI, test only 'recover packages after shutdown', 50 times
* experiment: Process#destroy appears to be async
* is it in the in-between period?
* partial -> total
* replace some booleans with assertions for better error reporting
* make triggerLog concurrent
* close channel and file in other error cases for port locking
- suggested by @leo-da; thanks
* use port locking instead of port 0 for trigger service fixtures
* destroy one service at a time
* missed continuation in build script
* use assertion language for "restart triggers with update errors"
* Revert "is it in the in-between period?"
This reverts commit 211ebfe9d2.
* use better assertion language for "restart triggers with update errors"
* restore full CI build
This was not only unnecessarily duplicated, it also had a bug where
`--crt` behaved like `--pem` instead of setting the cert chain.
I didn’t add new tests since it seems like the wrong place to test
config parsing of a library. We do have tests for TLS in general for
both DAML Script and DAML Triggers.
changelog_begin
changelog_end
This code has a few problems: the auth service test mode is not enabled, the
admin ledger should be separate from main ledger if auth is enabled, and the
retry strategy not executed.
The auth service is not used in these tests yet and we have an example
of how to do it correctly in the auth service fixture (and auth service
client tests). We can try to reuse that fixture instead or copy the
code over when necessary.
changelog_begin
changelog_end
* Trigger service tests: Make sure toxiproxy server is running before connecting client
changelog_begin
changelog_end
* Undo infuriatingly wrong camel case
* triggers: Use `FreePort.find()`.
* ports: Move `LockedFreePort` from postgresql-testing for reuse.
* triggers: Use `LockedFreePort` to avoid race conditions.
* ports + triggers: Move common port testing into the ports library.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-on-sql: Pull out a new `Main` object that wraps sandbox-classic.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-on-sql: Fail if a JDBC URL is not provided or not for PostgreSQL.
* sandbox-classic: Rename the conformance test H2 database.
* daml-on-sql + sandbox-classic: Report configuration errors cleanly.
This means letting `ProgramResource` catch the errors, log, and exit.
* daml-on-sql: Change the name logged on startup.
* daml-on-sql: Change the default participant ID.
* sandbox-common: Give the ledger name its own tagged string type.
* sandbox-classic: Generate random ledger IDs using the ledger name.
* daml-on-sql: Remove the banner, replacing it with a blank line.
* daml-on-sql: Enable strong seeding by default.
And weak seeding in the conformance tests.
* sandbox-classic: Move the ledger name to a separate parameter.
It's not really configurable.
* sandbox-classic: Move LedgerName from sandbox-common.
* daml-on-sql: Remove "-participant" from the participant ID.
* daml-on-sql: Use `Name` where possible.
* daml-on-sql: Make the ledger ID mandatory.
* Revert "sandbox-classic: Move LedgerName from sandbox-common."
This reverts commit 0dad1584a7.
* daml-on-sql: Print "DAML-on-SQL" in the CLI help, not "Sandbox".
* daml-on-sql + sandbox + sandbox-classic: Split out custom CLI parsing. (#6846)
* participant-state: Simplify naming the seeding modes.
* Method for service account request
* Extend domain and factor out sa/secure
* Methods to list service accounts and poll for service account
changelog_begin
changelog_end
* Run auth client tests 20 times
* Revert "Run auth client tests 20 times"
This reverts commit 5264ce6f30.
* Assert creds list for new service account is empty
* Request credential method
* Method to get a new credential id
* Method for fetching credential
* Login method (for ledger access token)
* Make authorize style consistent with rest
* Remove redundant test (subset of second test)
* Run auth client tests 20 times
* Fail future for http error responses and refine retry strategy
* Scalafmt
* Revert "Run auth client tests 20 times"
This reverts commit 2568dc6b47.
* Test getting service account before request
* sandbox-common: Move the ledgerIdMode up in the configuration.
* sandbox-classic: Move the configuration from sandbox-common.
CHANGELOG_BEGIN
CHANGELOG_END
* Fix miscellaneous warnings caught by IntelliJ IDEA.
* set many extra scalac -Xlint options for all Scala projects
CHANGELOG_BEGIN
CHANGELOG_END
* move NoCopy to its own file
package.scala:18: warning: it is not recommended to define classes/objects inside of package objects.
If possible, define trait NoCopy in package data instead.
trait NoCopy {
^
* move more traits, classes, and objects to proper packages
- note that `package` is itself a scoping construct, so if your reason
is the apparent aesthetic of placing a bunch of things in one `package
object`, that is easily remedied by deleting the `object` keyword
* fix some type-parameter-shadow warnings
- I'm generally in favor of sensible name-shadowing, following the
"deliberately hide variables that should not be accessed here" school
of thought. But I think type name shadowing isn't quite as valuable
and more likely to confuse than general variable shadowing, so have
experimentally linted it out.
Example warning:
EventsTableFlatEventsRangeQueries.scala:11: warning: type parameter
Offset defined in trait EventsTableFlatEventsRangeQueries shadows class
Offset defined in package v1. You may want to rename your type
parameter, or possibly remove it.
private[events] sealed trait EventsTableFlatEventsRangeQueries[Offset] {
^
* fix more package-object-classes warnings
* fix an inaccessible warning
ContractsService.scala:197: warning: method searchDb in class ContractsService references private class ContractsFetch.
Classes which cannot access ContractsFetch may be unable to override searchDb.
def searchDb(dao: dbbackend.ContractDao, fetch: ContractsFetch)(
^
* enable -Xlint:infer-any
- continuing the saga of #6116, #6132
* enable -explaintypes for more detailed type errors
* missed header for NoCopy; probably should have left it in the package file
* misspelling in comment
* revert -Xlint:doc-detached
- there are a lot of these fixes, and they are noisy, so shifting to a
separate PR
- thanks to @leo-da for pointing out
* WIP Call authorize before start trigger
* Intro auth service client
* Auth service client test setup
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt
* Use Spray JSON format and slightly better uri handling
* Get auth client test to work and fiddle with uris
* Remove changes to Server
* Add package for URI
* Format
* Copyrights
* Update sandbox packages
* Use default wallclock time for auth service sandbox
* Move public code into daml-integration-api
CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
* Add details about validity windows
* Reword description of auth service concepts
* Elaborate on auth flow dealing with token/credential expiry
CHANGELOG_BEGIN
CHANGELOG_END
* Revert ANF changes and add a testcase for evaluation order
After careful consideration, we decided that the change in evaluation
order that was accidentally introduced by the ANF changes should be
considered a breaking change or arguably even a bug and should not
land in 1.3.0.
Therefore, this PR reverts the following commits:
1. 353d0da6f7
2. a45b51042f
3. 04c7b2af7f
4. a624dd7242
5. b3aab72cee
Other PRs mostly had trivial merge conflicts that I resolved. The two
most interesting ones here are probably
1. https://github.com/digital-asset/daml/pull/6576 which was easy to
resolve and the change to return SEValue instead of SExpr is still
nice and useful even if we do not need the guarantees.
2. it https://github.com/digital-asset/daml/pull/6542 which required
some changes since the constructors changed. If you want to review
those changes in detail (they are pretty straightforward so not too
important), it’s probably easiest to check out this PR and run
```
git diff 2cd2a8f2a8
daml-lf/interpreter/src/main/scala/com/digitalasset/daml/lf/speedy/Compiler.scala
```
to see the diff to the parent commit of the first commit that
introduced ANF.
changelog_begin
changelog_end
* replace traverseU and sequenceU with traverse and sequence
- with -Ypartial-unification on, the extra Unapply typeclass lookup is
unnecessary
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* limit imports; we only need *> and void
The running trigger store is now written to synchronously by the server, so there's no need for retry logic.
Also assert equality of trigger ids rather than an arbitrary predicate, as it gives better error messages and we don't use other types of predicates for this function.
CHANGELOG_BEGIN
CHANGELOG_END
GHC has weird restriction on version numbers which damlc inserits so
we need to use `ghc_version` instead of `sdk_version`. That only makes
a difference for snapshot versions where the `-snapshot.` part is
replaced by `.`.
changelog_begin
changelog_end
It makes no sense to keep this at 0.0.1.
changelog_begin
- [DAML Script] The DAML Script library now has the version of the
corresponding SDK.
- [DAML Trigger] The DAML Trigger library now has the version of the
corresponding SDK.
changelog_end
* Add optional trigger id to startTrigger
CHANGELOG_BEGIN
CHANGELOG_END
* Read running trigger table in DbTriggerDao
And get rid of traverse type application!
(since we have partial unification on)
* Restart triggers in database
* Test for trigger restart on recovery
* Add order by clause and remove accidental overloading
* Add min/maxRestartInterval to configs and use in trigger restart strategy
* Adapt tests with triggers failing due to lack of network
changelog_begin
changelog_end
* Adapt tests for triggers with errors
* Remove comment about number of restarts
* Use a small initial restart interval for testing
* Remove old restart params
* Move maxInboundMessageSize to LedgerConfig
* Rename TriggerRunnerConfig to TriggerRestartConfig
* ANF transformation in Speedy.
The idea behind this PR is to transform speedy expressions into a simpler form where all non-atomic sub-expressions are made explicit by the introduction of let-forms. In particular, for the function-application form. These simpler forms allow the execution engine to take advantage of the atomic assumption, and often removes many additional execution steps. In particular the pushing of continuations to allow execution to continue after a compound expression has been reduced to a value.
changelog_begin
changelog_end
* improve comment
* inline functions relocateA/L
* remove comment about scalafmt
* remove commented out alterative def for transformLet1
* improve code by adding incr methods to DepthA/E
* remove (n == 0) special case in trackBindings
* clarify comment further
* improve validate/go to not consume stack for deeply right-nested let-expressions
* address comments from Remy: be private; use final case case; etc
* rename to unsafeCompilationPipeline
* add back some trailing commas
* remove commented-out debug line
* improve comment
* remove dev/debug code in compilationPipeline
* remove commented out code in SEAppGeneral.execute
* undo unrelated code improvement in SValue.scala
* fix compile. object Anf cannot be private
* Don't modify running trigger store via the trigger runner actor
changelog_begin
changelog_end
* Remove runner actor from RunningTrigger type
And parent actor from TriggerRunnerImpl
* Add to running trigger store on start request
* Update tests for new trigger store convention
* Fix test with network outage
* Move top level case classes to package object
changelog_begin
changelog_end
* Move Server Message classes to separate file
* Server in Server.scala and ServiceMain in ServiceMain.scala
* Copyright headers
The Server object and especially the apply method is extremely big and
hard to follow. There are a great many functions nested within the apply
method. This PR attempt to organize things, mostly by moving local
functions into methods of the Server class. I think this makes things
easier to follow, and I think it's more conventional.
Note that with this change I added some implicit parameters to the
Server constructor, which I think makes sense for these kinds of values
(contexts, sequencer factory, etc.) I also moved the Message trait to
the top level, but we will probably it its own file, following this PR.
There are a few other shallow tweaks that I made along the way.
Open to feedback on the high level structuring, as I'm not super
familiar with idiomatic design in Scala.
changelog_begin
changelog_end
* add -Xsource:2.13, -Ypartial-unification to common_scalacopts
* add now-referenced scalaz-core where needed
* work around bad type signatures in scalatest Aggregating, Containing
* unused Any suppression
* work around bad partial-unification wrought by type alias
* remove unused Conversions import
- not required in 4f68cfc480 either, so unsure how it's survived this long
* work around Future.traverse; remove unused show import
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* remove unused bounds
* remove -Ypartial-unification and -Xsource:2.13 where they were explicitly passed
* longer comment on what the options do
- suggested by @stefanobaghino-da; thanks
* forget Future.traverse, just use scalaz, it knows how to do this
* Move database initialization to Server apply method
* Reorder dao/server creation
* Read packages from database on startup
* Test starting a trigger after a shutdown
changelog_begin
changelog_end
This is needed to recover state after the service shuts down or crashes.
We add a method to the RunningTriggerDao to persistPackages. This only
does something in the case of a DbTriggerDao. In any case the Server
keeps a package map in memory as it's required to construct a trigger runner.
Uploads of existing packages is considered harmless.
changelog_begin
changelog_end
Avoid the use of Fragment.const which interprets raw strings as SQL
queries without any checks. Use the `sql` string interpolation which I
found out does the right thing with Strings and other simple types.
CHANGELOG_BEGIN
CHANGELOG_END
Trigger service: Refactor Server.addDar to take encoded dar
This is so we can write the encoded packages to the database if we have
one (without re-encoding them).
changelog_begin
changelog_end
In this PR we cleanup the constructor for the speedy Machine.
* We remove the `case` keyword since `Machine` is a stateful class,
* We replace the pre-existing builders with
+ one generic builder `Machine.apply`,
+ scenario specific builder,
CHANGELOG_BEGIN
CHANGELOG_END
* Trigger service: Refactor testing with and without database
Following the approach taken in the HTTP JSON API.
Define an AbstractTriggerServiceTest class with abstract jdbcConfig
member. This is instantiated with None and Some in the in-memory and
database test classes respectively. Tests for the individual modes can
be added in these concrete classes.
changelog_begin
changelog_end
* Move db init and destroy into beforeEach/afterEach to avoid concurrent tests clashing
Trigger Service: Refactor running trigger access using DAO interface
This introduces a RunningTriggerDao trait and two classes that implement
it, one for in-memory and one for database storage. This moves a lot of
code out of the Server class and into the DAO classes. The class chosen
at runtime depends on whether a JDBC config has been provided.
The database DAO also has methods for initialization and clean up used by
the server main function and the test fixture.
This is a pure refactoring (modulo some minor error message changes) so
functionality should stay the same.
CHANGELOG_BEGIN
CHANGELOG_END
* Add option based constructor for LedgerIdRequirement
changelog_begin
changelog_end
* Make option based consructor the default, deprecate old constructor
* Update with review comments
Includes a fair amount of refactoring to smooth the logic with/without a database. All tests now run with and without a database.
CHANGELOG_BEGIN
CHANGELOG_END
* Touch up logTriggerStatus
* Touch up removeRunningTrigger
* Touch up addRunningTrigger
* Remove IntelliJ (scalastyle I think) warnings about public members without type annotations
CHANGELOG_BEGIN
CHANGELOG_END
* Redundant brackets
* Stop running trigger under DB write failure
changelog_begin
changelog_end
* Another fix relating to initialization retries
changelog_begin
changelog_end
* Handle the error case directly
This adds a function withTriggerServiceAndDb which runs a test twice, once with and once without a database, and succeeds if both succeed. This will be useful for reusing test logic with both backends and making sure behaviour is consistent. I have used this function where possible, but it won't work for everything until stop is implemented on the DB side.
At the moment this new function squashes two tests into one making it hard to tell whether it failed with or without the database. In a future PR I will investigate using an abstract class to run the tests separately (hopefully with altered descriptions).
This feature required a few changes in the process, mainly:
- Use PostgresAroundAll to connect/disconnect to the database before and after all tests run
- Add a destroy method to the TriggerDao to reset the database between tests
- Use the TriggerDao in the withTriggerService functions to initialize / clean up the database at the start / end of each test
- Sort trigger instances from list using Scala's sort, not relying on Postgres' ordering of UUIDs. This also means we need to use UUIDs for trigger instances in the tests and sort nonempty vectors in expected results.
* Insert running trigger to DB when using one
If the DB write fails, the server sends itself a
TriggerInitializationFailure message so that the corresponding trigger
runner is stopped and the table is in sync with the actors.
We still need to retry writes here.
Includes basic test that runs the server with a JDBC config set and adds
a trigger, expecting a new entry to be added to the DB. However does not
check the running trigger table which we can do once reads are
implemented.
changelog_begin
changelog_end
* Await on future in test
* Update to new assertTriggerIds
* Apply scalafmt suggestions
* Create index on party token
* Read db in list command
* Update comment in test script
* Remove outdated comment
* Fix strings in insert and select
* Clean up test
* Add a second trigger in the db test
* Fix comment in test script
* Comment db tables
* Order trigger instances in list command
* Comment about TriggerDao execution context
* Store trigger history
changelog_begin
changelog_end
* Harvest trigger histories
changelog_begin
changelog_end
* Switch to Vector over List (and other bits and bobs)
* Use a better verb for updating trigger status method
* Add a comment
* Fix mangled comments
* Pass JdbcConfig object to TriggerDao apply
* No need to return TriggerDao from init db
* Refactor introducing RunningTrigger type
* Rename triggerId -> triggerInstance and triggerOrigId -> triggerName
Note this also changes the start request parameter name to triggerName.
However I have not yet renamed triggerId in the response messages. We
should probably make it triggerInstance there too but in a later PR.
changelog_begin
changelog_end
* disable Any wart
* first pass removal of Any suppressions for false positives
* second pass removal of Any suppressions for false positives
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* third pass removal of Any suppressions for false positives
* fourth pass removal of Any suppressions for false positives
* reformat newly single-suppressions into single lines
- suggested by @SamirTalwar-DA; thanks
* Don't update running triggers until we know the trigger is running
changelog_begin
changelog_end
* Don't update running triggers until we know the trigger is running
Minimal database initialization with schemas for running_triggers and dalfs tables. The user passes in the database URL, username and password in a config string argument (approach and code adapted from the JSON API).
In future the idea is to also create a "service" role with permissions to read and write to the new tables. Then the user can pass in the service role to connect to the database when running the service for real.
* Implement a simple profiler for DAML scenarios
The profiler runs a single scenario and records timing information when
each function (and some other closures) are entered and left. The
resulting information can be visualized as a flamegraph using
[speedscope](https://www.speedscope.app/).
The profiler works by instrumenting the CEK machine at the heart of
DAML Engine. Unfortunetaly, this causes a very small overhead on
non-profiling runs too. However, in my benchmarks I could not measure
any significant impact on the overall runtime at all. More precisely,
the overhead is as follows:
Every closure now has an additional field called `label`. In
non-profiling runs this field is always set to `null`. This field needs
to be allocated, copied whenever we copy a closure and scanned during
garbage collection. Additionally, whenever we enter a closure, we check
this field and whenever it is _not_ `null`, i.e. never during
non-profiling runs, we record an "open event" and set up a hook for the
corresponding "close event". Thus, the additional cost during
non-profiling runs are a single pointer comparison and a jump beyond
the "then branch".
Since this is still very much in active development, there are no
documentation, other than an entry in a README, and no tests yet. They
will come before we promote this. However, the UX will look very
different then since we already have plans to significantly change it.
CHANGELOG_BEGIN
CHANGELOG_END
* Run scalafmt
* Make profiling argument to PureCompiledPackges optional
* Fix a bunch of tests
CHANGELOG_BEGIN
CHANGELOG_END
* scalafmt is so annoying
CHANGELOG_BEGIN
CHANGELOG_END
* Apply simple suggestions
CHANGELOG_BEGIN
CHANGELOG_END
* Simplify and clarify the public interface to Speedy.
- Remove `isFinal`. A client just uses `run()`.
- Remove `toSValue`. The value in available in `SResultFinalValue(v: SValue)`.
- A client never directly access the `.ctrl` (or `.returnValue`) components.
- A client may use `setExpressionToEvaluate(expr)` to evaluate a new expression on an existing machine.
changelog_begin
changelog_end
* remove while loop which executes just once
* avoid unnecessary mutation when running speedy
Remove the `Ctrl` trait and separate `Machine.ctrl: Ctrl` into `Machine.ctrl: SExpr` and `Machine.returnValue: SValue` instead. This allows for avoiding dynamic dispatch on `ctrl` and instead allows for checking a pointer for `null` to decide if we have an expression that needs further break-down or a return value ready to be passed to the next continuation.
To make this check really only a pointer comparison we also needed to remove the abomination of "fully applied partially applied primitives". In order to achieve this, we check whether a PAP will be fully applied afterward when applying the last argument.
On the `collect-authority` benchmark, this increases throughput by around 13%, on another more computation heave benchmark by about 21%.
`collect-authority` benchmark on `master`:
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
112.361 ±(99.9%) 1.965 ms/op [Average]
(min, avg, max) = (107.047, 112.361, 120.745), stdev = 3.493
CI (99.9%): [110.396, 114.326] (assumes normal distribution)
```
`collect-authority` benchmark on this branch:
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
98.196 ±(99.9%) 1.933 ms/op [Average]
(min, avg, max) = (91.580, 98.196, 105.478), stdev = 3.436
CI (99.9%): [96.263, 100.129] (assumes normal distribution)
```
computation heavy benchmark on master
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
44.030 ±(99.9%) 0.742 ms/op [Average]
(min, avg, max) = (42.124, 44.030, 46.781), stdev = 1.319
CI (99.9%): [43.289, 44.772] (assumes normal distribution)
```
computation heavy benchmark on this branch:
```
Result "com.daml.lf.speedy.perf.CollectAuthority.bench":
36.222 ±(99.9%) 0.580 ms/op [Average]
(min, avg, max) = (34.897, 36.222, 39.787), stdev = 1.031
CI (99.9%): [35.643, 36.802] (assumes normal distribution)
```
changelog_begin
changelog_end
* Use triggerId field in trigger start response
* Use triggerId field for stop trigger result
* Fix indentation and make yields consistent
* Use pair constructor for JsObject instead of Map
* Use triggerIds field in list triggers response
changelog_begin
changelog_end
* Adapt ResponseFormat from JSON API
* Add some type annotations
* Use response format with status and errors/result fields
* Update and refactor tests
changelog_begin
changelog_end
Speedy: run() dont step()
- Running the Speedy machine with `run()` instead of `step()`
- Remove: `SResultContinue`
- Add: `SResultFinalValue(_)`
We change the top level control of Speedy: from machine.step() to machine.run, with the control of stepping while the machine returns SResultContinue moved into speedy itself. (And so SResultContinue is removed in favour of SResultFinalValue.) The main advantage of this approach is that the tight while loop can be moved inside the exception handler, rather than having to wrap the handler every step.
changelog_begin
changelog_end
* Endpoint to list all triggers (not yet by party)
* Clean up test code a little
changelog_begin
changelog_end
* Test for listing running triggers
* Respond with JSON list instead of random text
* List triggers by party
Pass party name in request body.
Store another map of party name to set of trigger ids.
Also store party names in the values of the original trigger id map, so
we can update the party map when stopping a trigger.
* Make DAML Triggers and DAML Script default to wall-clock-time
Now that sandbox defaults to wall-clock-time there is no reason why we
should not default in DAML triggers and DAML Script.
changelog_begin
- [DAML Triggers] ``daml trigger`` now defaults to wall clock time if
neither ``--wall-clock-time`` or ``--static-time`` is passed.
- [DAML Script] ``daml script`` now defaults to wall clock time if
neither ``--wall-clock-time`` or ``--static-time`` is passed.
changelog_end
* Make --static-time and --wall-clock-time exclusive
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* Move more trigger tests to scala tests
This PR moves more tests of triggers over to the Scala test suite, in
particular:
- The existing tests there abstract over the time mode and are
instantiated once for wallclock mode and once for static time mode.
- I’ve added tests for TLS and Auth.
- I’ve removed the TLS and Auth tests outside of scala-test since they
are now redundant.
- I’ve added the time tests to the scala-tests stuff since with the
new ledger time model, that’s necessary to actually trigger a
failure if you get static time vs wallclock time wrong (MRT and LET
no longer exist).
I haven’t yet moved all the func tests over, I’ll do that separately
and then we can kill the old tests completely.
changelog_begin
changelog_end
* Factor out test utils into a library
* Improve handling of exposed-modules with data-dependencies
Previously, we tried to rename all modules of a dependency via
--package. This fails if some of those modules are not exported. This
was trivial to hit as a user since the ``daml-trigger`` library made
use of this.
This PR adds a few things to improve the situation:
1. We only rename modules that are exposed. This fixes the issue if
you don’t actually reference a non-exposed module from your
data-dependency.
2. I’ve removed the exposed-modules from daml-trigger. I don’t think
they are essential here given that the module name has `Internal`
in the name and it’s too easy to have something that actually
references the non-exposed module since the types are reexported.
3. I’ve added documentation that mentions this issue.
4. I’ve added a warning if your exposed-modules are excluding some
modules. Maybe worth turning this into an error in the future.
changelog_begin
changelog_end
* Update compiler/damlc/lib/DA/Cli/Damlc/Packaging.hs
Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>
Co-authored-by: associahedron <231829+associahedron@users.noreply.github.com>
* Use com.daml as groupId for all artifacts
CHANGELOG_BEGIN
[SDK] Changed the groupId for Maven artifacts to ``com.daml``.
CHANGELOG_END
* Add 2 additional maven related checks to the release binary
1. Check that all maven upload artifacts use com.daml as the groupId
2. Check that all maven upload artifacts have a unique artifactId
* Address @cocreature's comments in https://github.com/digital-asset/daml/pull/5272#pullrequestreview-385026181
This replaces the rather horrible previous setup of having a custom
test runner that spawns 3 separate JVM processes by a single scalatest
test suite that starts sandbox and the JSON API in process.
changelog_begin
changelog_end
This PR adds a new test suite for DAML triggers based on scala test
rather than the client_server_test macro + a custom main. This seems
much nicer than the client_server_test (we get a lot of useful stuff
from scalatest, e.g., useful output of assertion failures, things
don’t blow up after the first test failure, …).
This PR only ports over a small fraction of the tests to make review
easier. The plan is then to port over everything and kill off the
existing test stuff once everything is ported over.
changelog_begin
changelog_end
