* kvutils: Use ScalaPB to generate a Scala JAR for daml_kvutils.proto.
* Bazel: Delete the unused `da_java_binary` rule, and inline `_wrap_rule`.
* Bazel: Factor out Java/Scala protobuf class generation into a helper.
CHANGELOG_BEGIN
CHANGELOG_END
* daml-lf/archive: Use `proto_jars`.
* Bazel: Remove the visibility modifier from `proto_jars`.
It's too confusing. Just make everything public.
* daml-lf/archive: Push protobuf source tarballs into `proto_jars`.
* Bazel: Add comments to the various parts of `proto_jars`.
* daml-assistant: Do unpleasant things with `location` in Bazel.
* restate the submit stage as a Flow and derived Sink
* take submit out of the trigger-to-submit flow
* type for the failures produced directly by command submission
* directly connect the msgSource failure queue to the submitter output
* parens
* slow down submission as we exceed max parallel submissions
* restricting alterF so it will be usable with ConcurrentMap
* disable buffer for the delay
* split out the delay function
* drafting a retry loop
* degenerate test for retry loop, factoring the forAllFuture utility
* map input to retrying properly
* make retrying accessible to tests
* test happy path and fix off-by-one
* further tests for retrying
* reveal that elements can get lost
* more determinism in test
* let failures block further elements from being attempted
- Previously failures would go into a separate queue, where they awaited expiry
of their delay and further initial upstream elements were given their first
tries. However, closing the upstream could mean that queue was dropped, and
detecting that situation is not trivial. So, instead, we don't use a separate
queue.
* plug retrying into the trigger submission flow
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* remove throttle; pendingCommandIds may leak
* report random parameter on failure
* revert comment about throttling
* explanation for fail in the error queue
- suggested by @cocreature; thanks
* graph component for running UnfoldStates
* proper doc for flatMapConcatNode
* nicer internal signatures
* refactor TriggerMsg encoder
* restate trigger sink in the graph DSL
* UnfoldState functions that might be useful
* express trigger sink with graphed initial state
* fix doc comment
* add SourceShape2, a SourceShape with two outlets
* add toSource for UnfoldState
* test for iterator
* do not submit for initialState out-of-band; feed into the graph instead
* factor the forAll Future pattern for testing
* test flatMapConcatNode directly
* add changelog
CHANGELOG_BEGIN
- [Triggers] Each trigger will submit up to 8 commands at a time, rather than
submitting as fast as possible. This applies to submissions in progress,
rather than completed but pending submissions; the latter's limit is still
subject to the ledger's own limits.
See `issue #7812 <https://github.com/digital-asset/daml/pull/7812>`__.
CHANGELOG_END
* remove uncons and append, which were not needed
* log initial evaluated state again
* shape port aliases
- suggested by @cocreature; thanks
* make ActionTriggerAny no longer have orphan instances
* add commands-in-flight reader to TriggerUpdateA
* make getCommandsInFlight a method shared by update and rule
* add changelog
CHANGELOG_BEGIN
- [Triggers] ``getCommandsInFlight`` may be used in a high-level trigger's
``updateState``, as well as its ``rule``.
See `issue #7787 <https://github.com/digital-asset/daml/pull/7787>`__.
CHANGELOG_END
* test use of getCommandsInFlight in an updateState
* inexplicable parse error
* Internal no longer exports queryContractId
* let free interpreter suspend on a SubmitRequest
* move UnfoldState to separate file, start a custom flow for flatmapping to it
* proper handlers, getting tangled in the mutable state of it all
* tests and conversion utilities for UnfoldState
* working UnfoldState flow with far fewer tricky details, passes test
* note on laziness
* missed t update
* test empty lists better
* add bifunctor for UnfoldState
* refactor message filtering
* split message parsing from the fold
* add copyright headers to new files
* let the flow from an UnfoldState emit the states
* test flatMapConcatStates's special output semantics
* pass submit's Future back to the holder of the SubmitRequest
* parallel submission stage, here with limit 8
* feed a stream of SubmitRequests to the command submission stage
* don't fail request submit stream stage on submit failure
- There was no way before for submission errors to report themselves to the free
interpreter, which just saw Unit; now we don't drop the Future on the floor,
so we properly turn "normal" submission errors into success, and actually
propagate any "real" errors to the stream instead of logging.
* note that initialState command submissions are not throttled or backpressured
* add the parallel submission size to the failure queue size
- defense against deadlock should we later change to propagate enqueuing
failures to the stream
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* clean up the initial namespace of getTriggerSink
* queryContractId and queryContractKey, trivially
* add changelog
CHANGELOG_BEGIN
- [Triggers] Two new functions are available for querying the ACS:
``queryContractId``, for looking up a contract by ID, and ``queryContractKey``
for looking one up by key.
See `issue #7726 <https://github.com/digital-asset/daml/pull/7726>`__.
CHANGELOG_END
* more efficient, direct queryContractId implementation
* flip getContractById's arguments, avoid 'flip'
* test queryContractKey
* test queryContractId
* resources: Move builders into //ledger/ledger-resources.
Keep the actual constructors in a trait, but instantiate it when working
with ledger code.
This allows us to later introduce an extra "context" type parameter to
ResourceOwner.
* resources-akka: Move the builders in to //ledger/ledger-resources.
* resources: Introduce an abstract `Context` parameter for owners.
This replaces the concrete `ExecutionContext`. While it _can_ be an
execution context, it really doesn't matter as long as we can get at one
somehow.
This is being introduced so we can wrap the context in a container,
either for type tagging or to include extra information.
Because our current context _is_ `ExecutionContext`, and an implicit is
provided to extract it, we can end up with two ways to get the same
value. We use shadowing to prevent this. This problem should go away in
the near future when a new context type is added.
CHANGELOG_BEGIN
- [Integration Kit] The `ResourceOwner` type is now parameterized by a
`Context`, which is filled in by the corresponding `Context` class in
the _ledger-resources_ dependency. This allows us to pass extra
information through resource acquisition.
CHANGELOG_END
* ledger-resources: Move `ResourceOwner` here from `resources`.
* ledger-resources: Remove dependencies from outside //ledger.
* ledger-resource: Wrap the acquisition execution context in `Context`.
So we can add a logging context to it.
* resources: Pass the Context, not the ExecutionContext, to Resource.
* Avoid importing `HasExecutionContext`.
* ledger-resources: Publish to Maven Central.
* resources: Make the small changes suggested by @stefanobaghino-da.
Co-Authored-By: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* ledger-resources: Pull out a trait for test resource contexts.
Saves a few lines of code.
* Restore some imports that were accidentally wildcarded.
* resources: Replace an `implicit def` with a couple of imports.
* participant-integration-api: Simplify the JdbcLedgerDaoBackend tests.
Try and use the right execution context where possible.
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* allow updating user state from high-level trigger rule
* fix tests and examples for new rule signature
* fix doc for new rule signature
* add changelog
CHANGELOG_BEGIN
- [Triggers] ``rule`` can now ``get`` and ``put`` the user-defined state just
like ``updateState`` does. It no longer accepts the state as an argument.
You can port your rule function types by replacing ``s -> TriggerA`` with
``TriggerA s``, removing the last argument from the function, and using
``get`` to retrieve the state at the beginning of the ``rule``'s ``do`` block,
if needed.
See `issue #7674 <https://github.com/digital-asset/daml/pull/7674>`__.
CHANGELOG_END
* test that rule userState changes propagate all the way out
* adapt compatibility filtering to #7681
* add ACS reader to TriggerA and TriggerStateA
* propagate changes in TriggerA, TriggerStateA structure
* allow query to be used in updateState and rule
* remove getTemplates
* remove ACS argument from updateState and rule
* fix type parameter order on query
* use query function in all tests and examples
* replace getContracts with query in documentation
* use wildcards instead of otherwise
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* rename TriggerStateA to TriggerUpdateA
- suggested by @cocreature; thanks
* missed renamings of TriggerStateA to TriggerUpdateA
- suggested by @cocreature; thanks
* make the meaning of the rlift functions clearer
* make initialize a TriggerInitializeA instead of a function; remove getContracts
* update tests and examples for new initialize signature
CHANGELOG_BEGIN
- [Triggers] Trigger ``updateState``, ``rule``, and ``initialize`` functions no
longer accept an ``ACS`` argument; instead, they must use the ``query`` action
to query the ACS, similar to the same function in DAML Script.
See `issue #7632 <https://github.com/digital-asset/daml/pull/7632>`__.
CHANGELOG_END
* disable older compatibility trigger builds
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* add ActionState to the standard library
* use 1 ActionState, 1 get, 1 put in low-level trigger library
* introduce TriggerStateA for updateState
* fix tests and examples for new updateState signature
CHANGELOG_BEGIN
- [Triggers] The ``updateState`` function now returns a ``TriggerStateA``. This
is an action like ``TriggerA``, but doesn't permit emitting commands. Instead
of taking the state as an argument and returning a new state, you can
manipulate the state with ``get``, ``put``, and ``modify``. Any existing
``updateState`` can be ported by replacing ``s -> expr`` in the lambda
expression with ``-> modify $ \s ->``, and then made to look nicer from there
as desired.
See `issue #7621 <https://github.com/digital-asset/daml/pull/7621>`__.
CHANGELOG_END
* some DAML docs for updateState and TriggerStateA
* add getCommandsInFlight as a TriggerA action
* immediately update commandsInFlight on emitCommands
- delay until the rule had returned made sense before submitCommands was launched
immediately; this also makes `getCommandsInFlight` more sensible
* remove commands-in-flight argument from high-level Trigger rule
CHANGELOG_BEGIN
- [Triggers] The "commands in flight" or ``Map CommandId [Command]`` argument has been
removed from high-level trigger ``rule`` functions; instead, the current
commands-in-flight can be retrieved with the new ``getCommandsInFlight`` function, which
can be done immediately at the beginning of the rule's ``do`` block to preserve exact
existing trigger behavior.
See `issue #7600 <https://github.com/digital-asset/daml/pull/7600>`__.
CHANGELOG_END
* remove commands-in-flight argument from trigger tests
* update doc examples and copy in compatibility
- compatibility change will most likely entail another flag day in compatibility tests,
replacing last_pre_7456_trigger_version
* update doc text for getCommandsInFlight
* test that getCommandsInFlight gets updated during the rule
* flag day for trigger compatibility tests
sdk-version: 1.7.0-snapshot.20201006.5358.0.0c1cadcf
File: src/CopyTrigger.daml
Hidden: no
Range: 55:11-55:19
Source: typecheck
Severity: DsError
Message:
src/CopyTrigger.daml:55:12: error:
• Couldn't match type ‘TriggerA ()’ with ‘() -> TriggerA ()’
Expected type: Party
-> DA.Next.Map.Map CommandId [Command] -> () -> TriggerA ()
Actual type: Party -> ACS -> () -> TriggerA ()
• In the ‘rule’ field of a record
In the expression:
Trigger
{initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}
In an equation for ‘copyTrigger’:
copyTrigger
= Trigger
{initialize = \ _acs -> (), updateState = \ _acs _message () -> (),
rule = copyRule, registeredTemplates = AllInDar, heartbeat = None}
* match docs on TriggerAState to current usage of these fields
* remove emittedCommands, as commandsInFlight is now kept up-to-date
* zoomed from where?
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* make TriggerF's Submit emit a command ID instead of accepting one
- LowLevel updated
- interpreter in Scala updated, preserving Message semantics as closely as possible
* remove nextCommandId and all management from high-level Trigger API implementation
* a note about TriggerF and simulateRule
* port submitCommands calls from tests
CHANGELOG_BEGIN
- [Triggers] The CommandIds as accessed from trigger DAML code are now exactly the command
IDs used in command submission to the ledger; as such, they will vary randomly from run
to run of the trigger rule. To enable this, the low-level ``submitCommands`` function
no longer accepts a command ID, instead returning one; there is no change to the
corresponding high-level ``emitCommands`` function, so high-level triggers should only
see improved and easier-to-understand logging.
See `issue #7587 <https://github.com/digital-asset/daml/pull/7587>`__.
CHANGELOG_END
* port Scala-side assertion of trigger test
* rename commandIdMap to commandIdsUsed
* test that command IDs are like the ones we expect from the runner
* conservatively move daml-script, trigger SValue interpreters to common library
* introduce expect and JavaList pattern for converters
* clean up trigger Converter Command interpretation
* add Church Free monad
* add an action language for trigger updates
* add expectE to remove some of the joins
* convert more of the converters to expect
* tool for unrolling Free/Roll
* split handleStepResult up and clean up its pattern
* handleStepFreeResult to interpret TriggerF
* replace Free Church with Pure/Roll free from Script
* newtype for ActionTrigger
* replace update in low-level Trigger with Free TriggerF
* submit one Commands at a time
* boolean blindness strikes again
* log missed TriggerF steps
* comment actual Submit contents
* match #7501 fromPureSExpr sig change in 00b80b8ea3
* avoid using forwardPort in runTrigger
* push State back into DAML, so it can be excluded from the action list
* push Message back into DAML, unifying the action language for initialState and update
* bringing TriggerF into initial state
* really add TriggerF into initial state, with all ports, tested
* add ActionTrigger class, express initialState in its terms
* add all TriggerF actions to existing TriggerA
* Trigger.rule will no longer have Time argument
* rename getS, setS to get, put, matching C.M.T.State from transformers
* make high-level Rule evaluate to the underlying TriggerF sequence
* Assert's testRule doesn't have a transform yet
* move DamlTuple2 to common converter library
- suggested by @cocreature; thanks
* combine the two Frees, provide from Script
* remove time argument from integration tests
CHANGELOG_BEGIN
- [Triggers] The ``Time`` argument was removed from the trigger rule function; instead, it
can be fetched within the ``TriggerA`` ``do`` block by ``getTime``, as with ``Update``
and ``Scenario``. The ``LowLevel`` trigger interface has been redesigned; such triggers
need to be rewritten or ported to high-level triggers.
See `issue #7456 <https://github.com/digital-asset/daml/pull/7456>`_.
CHANGELOG_END
* add trigger rule simulator to support Assert module
* missed new Free module
- left in script per @cocreature
* remove retract as we ended up using foldFree for that purpose instead
- suggested by @cocreature; thanks
* throw ConverterException instead of RuntimeException
- suggested by @cocreature; thanks
* remove Time argument from coin-upgrade-trigger
* port trigger service tests
* port trigger scenario test
* put TriggerSetup and TriggerRule into LowLevel.Trigger instead of unboxed Free
- suggested by @cocreature; thanks
* remove Time argument from trigger compatibility test
* submit commands as soon as each `emitCommands` is sequenced
- we still collect a list, but only for tracking commandsInFlight
* filter out compatibility tests for triggers before now
* remove commented imports, libraries from new shared converter
* make the TriggerF interpreter tail-recursive
* remove unused compatibility trait
* add back new state logging
* remove refactoring comment
* rewrite some LowLevel initialStates in do
* hide Daml.Script.Free from docs
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* remove forwardPortInitialState
- suggested by @cocreature; thanks
* manually port low-level updates
- suggested by @cocreature; thanks
* remove forwardPort
- suggested by @cocreature; thanks
* fail faster on unrecognized TriggerF
- suggested by @cocreature; thanks
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* use GenMaps for trigger ACS
* Next removed for 1.dev
* temp port trigger test code to 1.dev only
* run trigger tests on 1.dev only
* move pending back to TextMap
* include trigger service in the 1.dev test lineup
- it takes >2min, so shouldn't be permanent
* add Ord TypeRep and Ord TemplateTypeRep when possible
* swap names in Internal to reduce the diff
* try to enable cpp for triggers compilation
$ bazel build //triggers/daml:daml-trigger-1.dev
<snip>
File: daml/Daml/Trigger/Internal.daml
Hidden: no
Range: 103:-1-103:-1
Source: CPP
Severity: DsError
Message: 22 in hpp-0.6.1:Hpp.CmdLine
File: daml/Daml/Trigger.daml
Hidden: no
Range: 103:-1-103:-1
Source: CPP
Severity: DsError
Message: 22 in hpp-0.6.1:Hpp.CmdLine
ERROR: Creation of DAR file failed.
<snip>
* remove problematic options for invoking cpp
hpp: Couldn't open input file: -Werror
CallStack (from HasCallStack):
error, called at src/Hpp/CmdLine.hs:103:22 in hpp-0.6.1:Hpp.CmdLine
* enough cpp so default and 1.dev triggers compile
* cpp needed for docs as well
* no changelog
CHANGELOG_BEGIN
CHANGELOG_END
* return trigger service to testing against sdk default lf version
* run trigger integration test against sdk default and LF 1.dev
* return trigger scenario test to SDK default LF version
* avoid import warnings in trigger lib
* Windows manifests a missing file differently. Hilarious
This is clearly a breaking change but triggers are still alpha so we
can get away this (confirmed with Bernhard).
changelog_begin
- [DAML Trigger] Daml.Trigger.Assert now uses DAML Script instead of scenarios.
changelog_end
* get a LoggingContext into the TriggerRunnerImpl
* make some implicits more implicitly scoped and explicitly ascribed
* make some private/final markings
* most of JsonFormat[Identifier] is in companion
* experimental LoggingContext with phantom type
* ActorContext#log isn't really doing that much
* more details of LoggingContextOf
* make LoggingContextOf compile
* add trigger message logging, yet without context
* fix parent compile errors
* use Config as the phantom for its own logging extensions
* LocalDateTimeFormat cleanup
* switch TriggerRunner to contextual logging
* add trigger definition ID to logs
* log trigger-submitted commands, fix trigger test compile
* log trigger stopping and DAR uploads
* add context to PostStop/PreRestart logs
* add changelog
CHANGELOG_BEGIN
- [Triggers] More detailed logging of trigger actions and trigger service actions.
See `issue #7205 <https://github.com/digital-asset/daml/pull/7205>`_.
CHANGELOG_END
* missed copyright header
* switch to Unit, scala/bug#9240 fixed
* ledger-api-client: `maxInboundMessageSize` -> `maxInboundMetadataSize`.
CHANGELOG_BEGIN
- [Scala Bindings] Rename a field in the ``LedgerClientConfiguration``
to ``maxInboundMetadataSize``, to match the builder Netty channel
builder. It was incorrectly named ``maxInboundMessageSize``, which is
a different channel property that configures the maximum message size,
not the header size.
CHANGELOG_END
* ledger-api-client: Introduce a `maxInboundMessageSize` config property.
We use this a lot; easier if it's in the configuration.
CHANGELOG_BEGIN
- [Scala Bindings] Replace the
``LedgerClientConfiguration.maxInboundMessageSize`` property with a
new one that represents the maximum size of the response body.
CHANGELOG_END
* ledger-api-client: Add integration tests for the simple stuff.
* sandbox-common: Make `SandboxFixtureWithAuth` a mixin.
This makes it useful with `SandboxNextFixture` as well as
`SandboxFixture`.
Also, add types to non-private fields and methods, and make more fields
protected rather than public.
* ledger-api-client: Add tests to make sure the token is passed through.
CHANGELOG_BEGIN
CHANGELOG_END
* sandbox-common: Tokens are for auth, not auth.
fixes#7030
This deliberately ignores the trigger service. The main reason for
setting the ApplicationId at the moment is authentication and this is
still very WIP in the trigger service, so I don’t think it makes sense
to add this in some form to the API until that has settled.
changelog_begin
- [DAML Trigger] You can now configure the application id via
`--application-id` in `daml trigger`. This is primarily useful if
you run with authentication.
changelog_end
* Move public code into daml-integration-api
CHANGELOG_BEGIN
[DAML Integration Kit]: Removed sandbox specific code from the API intended to be used by ledger integrations. Use the maven coordinates ``com.daml:participant-integration-api:VERSION`` instead of ``com.daml:ledger-api-server`` or ``com.daml:sandbox``.
CHANGELOG_END
* Add option based constructor for LedgerIdRequirement
changelog_begin
changelog_end
* Make option based consructor the default, deprecate old constructor
* Update with review comments
Packages com.digitalasset.daml and com.daml have been unified under com.daml
Ledger API and DAML-LF DEV protos have also been moved from `com/digitalasset`
to `com/daml` on the file system.
Protos for already released DAML LF versions (1.6, 1.7, 1.8) stay in the
package `com.digitalasset`.
CHANGELOG_BEGIN
[SDK] All Java and Scala packages starting with
``com.digitalasset.daml`` and ``com.digitalasset`` are now consolidated
under ``com.daml``. Simply changing imports should be enough to
migrate your code.
CHANGELOG_END
* Move more trigger tests to scala tests
This PR moves more tests of triggers over to the Scala test suite, in
particular:
- The existing tests there abstract over the time mode and are
instantiated once for wallclock mode and once for static time mode.
- I’ve added tests for TLS and Auth.
- I’ve removed the TLS and Auth tests outside of scala-test since they
are now redundant.
- I’ve added the time tests to the scala-tests stuff since with the
new ledger time model, that’s necessary to actually trigger a
failure if you get static time vs wallclock time wrong (MRT and LET
no longer exist).
I haven’t yet moved all the func tests over, I’ll do that separately
and then we can kill the old tests completely.
changelog_begin
changelog_end
* Factor out test utils into a library
This replaces the rather horrible previous setup of having a custom
test runner that spawns 3 separate JVM processes by a single scalatest
test suite that starts sandbox and the JSON API in process.
changelog_begin
changelog_end
This PR adds a new test suite for DAML triggers based on scala test
rather than the client_server_test macro + a custom main. This seems
much nicer than the client_server_test (we get a lot of useful stuff
from scalatest, e.g., useful output of assertion failures, things
don’t blow up after the first test failure, …).
This PR only ports over a small fraction of the tests to make review
easier. The plan is then to port over everything and kill off the
existing test stuff once everything is ported over.
changelog_begin
changelog_end
* Depend on LF version specific daml-libs
* daml-script.dar build multiple LF versions
CHANGELOG_BEGIN
[DAML Script] The `daml-script` library is now available in multiple LF
versions, namely 1.7, 1.8, and 1.dev.
CHANGELOG_END
* daml-trigger.dar build multiple LF versions
[DAML Triggers] The `daml-trigger` library is now available in multiple
LF versions, namely 1.7, 1.8, and 1.dev.
* Keep daml-script.dar available for tests
* Keep daml-trigger.dar available for tests
* daml-libs LF versions integration test
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
Contributes to #4194.
Closes#4231.
Closes#5022.
CHANGELOG_BEGIN
- [Ledger API] The protobuf fields ledger_effective_time and maximum_record_time have been removed from
command submission. These fields were previously deprecated following the introduction
of a new ledger time model. See issue `#4194 <https://github.com/digital-asset/daml/issues/4194>`__.
[Java Bindings] removed the usage of ledgerEffectiveTime and
maximumRecordTime, and instead added minLedgerTimeAbsolute and
minLedgerTimeRelative in CommandSubmissionClient and CommandClient
CHANGELOG_END
* sandbox: Fail to start if a time mode is not explicitly specified.
CHANGELOG_BEGIN
- [Sandbox] Sandbox is switching from Static Time mode to Wall Clock
Time mode as the default. To ensure that our users know about this,
for one version, there will be no default time mode. Instead, users
will have to explicitly select their preferred time mode by means of
the `--static-time` or `--wall-clock-time` switches. In the next
release, Wall Clock Time will become the default, and users who are
happy with the defaults will no longer need to specify the time mode.
CHANGELOG_END
* daml-script|triggers: Specify time mode when testing against Sandbox.
* daml-assistant: Default the Sandbox to wall clock time.
CHANGELOG_BEGIN
- [DAML Assistant] Initializing a new DAML project adds a switch to
``daml.yaml`` to ensure Sandbox can continue to start with ``daml
start``::
sandbox-options:
- --wall-clock-time
CHANGELOG_END
* docs: Update the DAML Script and Triggers docs to use Wall Clock time.
It's now what Sandbox will use by default when using `daml init`.
* docs: Change the Quickstart to run Sandbox in wall clock time.
This explains why the contract IDs may vary.
It also updates the manual release testing script to match.
Previously parts of the initialization, in particular, the code for
finding the filter and the heartbeat were part of the Runner. This led
to an akward API and didn’t really make any sense.
Now all of this code is part of a pure `Trigger.fromIdentifier`
method and the runner only takes care of actually running the
ledger. This could also be useful for the trigger service where we
might want to synchronously call `getIdentifier` so users get some
indication if there request even points to a valid trigger
directly. However, this is not tackled by this PR.
changelog_begin
changelog_end
Previously the runner class was in a weird state where it was specific
to a DAR but not to an individual trigger. This meant that you had to
pass around a fair bit of state which got a bit awkward. This PR
addresses this by making the trigger class specific to the trigger.
It also now accepts `CompiledPackages` instead of a DAR which should
make it easier in the trigger service to support dynamic package
uploads.
changelog_begin
changelog_end
Previously we assumed that the module name was globally unique in the
DAR which is definitely not guaranteed. Now we instead detect the
package id of the trigger library based on the type of the trigger we
are running which doesn’t fall apart if there are multiple versions of
the trigger library.
I’ve also removed the check for the package id of the trigger library
since I’d like the trigger runner to be backwarts compatible from now on (we
didn’t break that in a while).
This is slightly ugly since the Runner class is currently not specific
to a single trigger but only the individual methods are aware of the
specific trigger identifier. I’ll refactor this in a separate PR.
changelog_begin
changelog_end
* Share test certificates
This is primarily an attempt at making sure my contribution stats
remain negative but I think it’s a nice cleanup. The only difference
in the certs used by daml-helper which are now used everywhere is that
they use a different CN for the CA and the server. This is required to
make openssl happy (which is used by the daml-helper).
changelog_begin
changelog_end
* Fix script and trigger tests
This adds CLI parametrs for connecting via TLS following the scheme
used by navigator, extractor and `daml ledger`.
changelog_begin
- [DAML Script] Support TLS. Enable it by passing ``--tls``. You can
set certificates for client authentication via ``--pem`` and
``-crt`` and a custom root CA for validating the server certificate
via ``--cacrt``.
- [DAML Triggers - Experimental] Support TLS. Enable it by passing ``--tls``. You can
set certificates for client authentication via ``--pem`` and
``-crt`` and a custom root CA for validating the server certificate
via ``--cacrt``.
changelog_end
Context
=======
After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.
This PR is a proposal for a new release process designed under the
following constraints:
- Reuse as much as possible of the existing infrastructure, to minimize
effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
not meant for general public consumption, but can still be used by savvy
users without jumping through too many extra hoops (ideally just
swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
release status, with as few changes as possible in-between, so we can be
confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
releases, to avoid discovering non-transient problems while trying to
promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
keep the same approval process for SOC2 auditability.
The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:
```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```
This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.
Version numbers
===============
Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:
- Semantic versioning (which we chose as the version format for the SDK
version number) allows for "prerelease" version numbers as well as
"metadata"; an example of a complete version string would be
`1.2.3-nightly.201+server12.43`. The "main" part of the version string
always has to have 3 numbers separated by dots; the "prerelease"
(after the `-` but before the `+`) and the "metadata" (after the `+`)
parts are optional and, if present, must consist of one or more segments
separated by dots, where a segment can be either a number or an
alphanumeric string. In terms of ordering, metadata is irrelevant and
any version with a prerelease string is before the corresponding "main"
version string alone. Amongst prereleases, segments are compared in
order with purely numeric ones compared as numbers and mixed ones
compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
`-`, or a transition between a number and a letter. Version strings
are compared element-wise, with numeric segments being compared as
numbers. Alphabetic segments are treated specially if they happen to be
one of a handful of magic words (such as "alpha", "beta" or "snapshot"
for example) which count as "qualifiers"; a version string with a
qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
special ordering amongst qualifiers. Other alphabetic segments are
compared alphabetically and count as being "after" their prefix
(`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
separated by `.`, plus an optional (though deprecated) alphanumeric
"version tag" separated by a `-`. I could not find any official
documentation on ordering for the version tag; numeric segments are
compared as numbers.
- npm uses semantic versioning so that is covered already.
After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:
```
0.13.53-alpha.20200214.3025.ef5d32b7
```
where the components are, respectively:
- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
master.
- `3025`: the number of commits in master up to the release commit
(included). Because we have a linear, append-only master branch, this
uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
not strictly speaking necessary, but makes it a lot more convenient to
identify the commit.
The main downsides of this format are:
1. It is not a valid format for GHC packages. We do not publish GHC
packages from the SDK (so far we have instead opted to release our
Haskell code as separate packages entirely), so this should not be an
issue. However, our SDK version currently leaks to `ghc-pkg` as the
version string for the stdlib (and prim) packages. This PR addresses
that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
would see the above version number as `0.13.53.20200214.3025`, which
should be enough to uniquely identify it. Note that, as far as I could
find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
it makes it more scary. However, I have been told that this may be
long enough to cause issues on Windows by pushing us past the max path
size limitation of that "OS". I suggest we try it and see what
happens.
The upsides are:
- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
they have the same date, allowing us to release same-day patches if
needed. (Note: that is, commits that were made on the same day; the
release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
GHC).
Alternatives I considered:
- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
be accepted by all schemes, but will not sort as expected under semantic
versioning (though Maven will be fine). I have no idea how it will sort
under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
This is not valid semantic versioning and is therefore rejected by
npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
what is generally done in the Java world, but we then lose track of what
version is actually in use and I'm concerned about bug reports. This
would also not let us publish to the main Maven repo (at least not more
than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
version formats. However, it would not clearly indicate to humans that
it is not meant as a stable version, and would sort differently under
semantic versioning (which counts it as a prerelease, i.e. before
`0.13.53`) than under maven (which counts it as a patch, so after
`0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
number of prereleases in-between `0.13.52` and the next. This is
currently the fallback plan if Windows path length causes issues. It
would be less convenient to map releases to commits, but it could still
be done via querying the history of the `LATEST` file.
Release notes
=============
> Note: We have decided not to have release notes for snapshot releases.
Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.
But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.
The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.
> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.
New release process
===================
Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.
Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:
1. Before running the release step, CI will checkout the commit
specified in the LATEST file. This requires separating the release
step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
that gets the version to build from an environment variable, with a
default of `0.0.0` to remain consistent with the current `daml-head`
behaviour.
Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.
The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.
I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.
CHANGELOG_BEGIN
CHANGELOG_END
* Add first prototype of triggers as a service (TaaS)
This is an extremely basic version of the trigger as a service thingy.
Right now, it supports spawning triggers and stopping them but nothing
else.
There is a very simple test to check that it’s not completely broken.
changelog_begin
changelog_end
* Apply suggestions from code review
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* remove debugging output
* remove leftover import
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Daml.Trigger.Assert for trigger testing API
Requires extracting part of Daml.Trigger into Daml.Trigger.Internal to
get access to internal data constructors and functionality
CHANGELOG_BEGIN
- [DAML Trigger - Experimental] Trigger testing functionality is now
available in the module Daml.Trigger.Assert.
CHANGELOG_END
* Set exposed-modules to hide Daml.Trigger.Internal
* API docs for Daml.Trigger.Assert
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
* Bazel test for trigger scenario
* daml-triggers: Allow testing trigger rules in scenarios
CHANGELOG_BEGIN
- [DAML Triggers - Experimental] DAML triggers can now be tested in
scenarios. Specifically, a trigger's ``rule`` can be executed in a
scenario and assertions performed on the emitted commands.
CHANGELOG_END
* Allow assertions on create commands
CHANGELOG_BEGIN
* [DAML stdlib] Add `CanAbort` instance for `Either Text`.
CHANGELOG_END
* Add convenience to construct ACS for testRule
* Add assertions for exercise and exerciseByKey
* fix assert message
* Test assertExercise(ByKey)Cmd
* unpackCommands --> flattenCommands
* Add API documentation
* Document that command ids start from "0"
* generalise command assertions to CanAbortm
* export ACSBuilder type
* Better haddocs for trigger command assertions
* explicit let
* ./fmt.sh
* Fix runfiles on Windows
* Add reference to Bazel issue
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
* Implement heartbeat messages in trigger runner.
* Add heartbeat to Daml.Trigger
CHANGELOG_BEGIN
- [DAML Triggers - Experimental] DAML triggers can now configure a heartbeat message to be sent at regular time interval.
CHANGELOG_END
* Add DAML trigger heartbeat test-case
* ./fmts.h
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
These have failed quite a few times on Windows and occasionally also
on MacOS.
This test, first fixes a small issue where the tests were actually
using the times from completions instead of only the timings from
creations. (that technically shouldn’t be an issue but it’s at least
confusing since the error claims to test creations).
In addition to that, this PR changes the condition to allow for the
times to be equal since especially on Windows we don’t seem to have a
very high resolution and the tests are remarkably quick so sometimes
the times can be identical.
I’ve slightly rephrased the condition since I got confused by the fact
that we test for the negated condition.
changelog_begin
changelog_end
