Add documentation for upgrading using DAML script and triggers (#5465)

* Add documentation for upgrading using DAML script and triggers

This PR adds documentation for how you can automate the upgrade
process from the Coin example using DAML Script and DAML Triggers.

Having written this, I think we might want to add some integration
tests for this but it’s rather complex to setup since it ties together
so many pieces so I’d like to leave it out for now. We do at least
test that the code compiles and we all knows that code that compiles
never has bugs.

changelog_begin
changelog_end

* Apply suggestions from code review

Co-Authored-By: associahedron <231829+associahedron@users.noreply.github.com>

Co-authored-by: associahedron <231829+associahedron@users.noreply.github.com>

.gitignore

@@ -91,3 +91,4 @@ compiler/damlc/output
 language-support/hs/bindings/gen
 language-support/hs/bindings/*.cabal
 language-support/hs/bindings/examples/nim/*.cabal
+.daml/

docs/BUILD.bazel

@@ -651,6 +651,28 @@ daml_build_test(
     project_dir = "source/upgrade/example/coin-upgrade",
 )
 
+daml_build_test(
+    name = "daml-upgrade-example-upgrade-script",
+    dar_dict = {
+        ":daml-upgrade-example-v1": "path/to/coin-1.0.0.dar",
+        ":daml-upgrade-example-v2": "path/to/coin-2.0.0.dar",
+        ":daml-upgrade-example-upgrade": "path/to/coin-upgrade-1.0.0.dar",
+        "//daml-script/daml:daml-script.dar": "path/to/daml-script.dar",
+    },
+    project_dir = "source/upgrade/example/coin-initiate-upgrade",
+)
+
+daml_build_test(
+    name = "daml-upgrade-example-upgrade-trigger",
+    dar_dict = {
+        ":daml-upgrade-example-v1": "path/to/coin-1.0.0.dar",
+        ":daml-upgrade-example-v2": "path/to/coin-2.0.0.dar",
+        ":daml-upgrade-example-upgrade": "path/to/coin-upgrade-1.0.0.dar",
+        "//triggers/daml:daml-trigger.dar": "path/to/daml-trigger.dar",
+    },
+    project_dir = "source/upgrade/example/coin-upgrade-trigger",
+)
+
 filegroup(
     name = "daml-intro-1",
     srcs = glob(

docs/source/upgrade/automation.rst

@@ -0,0 +1,167 @@
+.. Copyright (c) 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
+.. SPDX-License-Identifier: Apache-2.0
+
+.. _upgrade-automation:
+
+Automating the Upgrade Process
+##############################
+
+In this section, we are going to automate the upgrade of our coin
+process using :doc:`DAML Script</daml-script/index>` and :doc:`DAML
+Triggers </triggers/index>`. Note that automation for upgrades is
+specific to an individual application, just like the upgrade models.
+Nevertheless, we have found that the pattern shown here
+occurs frequently.
+
+Structuring the Upgrade
+=======================
+
+There are three kinds of actions performed during the upgrade:
+
+#. Alice creates ``UpgradeCoinProposal`` contracts. We assume here,
+   that Alice wants to upgrade all ``Coin`` contracts she has
+   issued. Since the ``UpgradeCoinProposal`` proposal is specific to
+   each owner, Alice has to create one ``UpgradeCoinProposal`` per
+   owner. There can be potentially many owners but this step only has
+   to be performed once assuming Alice will not issue more ``Coin``
+   contracts after this point.
+#. Bob and other owners accept the ``UpgradeCoinProposal``. To keep
+   this example simple, we assume that there are only coins issued by
+   Alice. Therefore, each owner has to accept at most one proposal.
+#. As owners accept upgrade proposals, Alice has to upgrade each
+   coin. This means that she has to execute the upgrade choice once
+   for each coin. Owners will not all accept the upgrade at the same
+   time and some might never accept it. Therefore, this should be a
+   long-running process that upgrades all coins of a given owner as
+   soon as they accept the upgrade.
+
+Given those constraints, we are going to use the following tools for
+the upgrade:
+
+#. A DAML script that will be executed once by Alice and creates an
+   ``UpgradeCoinProposal`` contract for each owner.
+#. Navigator to accept the ``UpgradeCoinProposal`` as Bob. While we
+   could also use a DAML script to accept the proposal, this step will
+   often be exposed as part of a web UI so doing it interactively in
+   Navigator resembles that workflow more closely.
+#. A long-running DAML trigger that upgrades all ``Coin`` contracts
+   for which there is a corresponding ``UpgradeCoinAgreement``.
+
+Implementation of the DAML Script
+=================================
+
+In our DAML Script, we are first going to query the ACS (Active Contract Set) to find all
+``Coin`` contracts issued by us. Next, we are going to extract the
+owner of each of those contracts and remove any duplicates coming from
+multiple coins issued to the same owner. Finally, we iterate over the
+owners and create an ``UpgradeCoinAgreement`` contract for each owner.
+
+.. literalinclude:: example/coin-initiate-upgrade/daml/InitiateUpgrade.daml
+   :language: daml
+   :start-after: -- INITIATE_UPGRADE_BEGIN
+   :end-before: -- INITIATE_UPGRADE_END
+
+Implementation of the DAML Trigger
+==================================
+
+Our trigger does not need any custom user state and no heartbeat so
+the only interesting field in its definition is the rule.
+
+.. literalinclude:: example/coin-upgrade-trigger/daml/UpgradeTrigger.daml
+   :language: daml
+   :start-after: -- TRIGGER_BOILERPLATE_BEGIN
+   :end-before: -- TRIGGER_BOILERPLATE_END
+
+In our rule, we first filter out all agreements and coins issued by
+us. Next, we iterate over all agreements. For each agreement we filter
+the coins by the owner of the agreement and finally upgrade the coin
+by exercising the ``Upgrade`` choice. We mark the coin as pending
+which temporarily removes it from the ACS and therefore stops the
+trigger from trying to upgrade the same coin multiple times if the
+rule is triggered in quick succession.
+
+.. literalinclude:: example/coin-upgrade-trigger/daml/UpgradeTrigger.daml
+   :language: daml
+   :start-after: -- TRIGGER_RULE_BEGIN
+   :end-before: -- TRIGGER_RULE_END
+
+The trigger is a long-running process and the rule will be executed
+whenever the state of the ledger changes. So whenever an owner accepts
+an upgrade proposal, the trigger will run the rule and upgrade all
+coins of that owner.
+
+Deploying and Executing the Upgrade
+===================================
+
+Now that we defined our DAML script and our trigger, it is time to use
+them! If you still have Sandbox running from the previous section,
+stop it to clear out all data before continuing.
+
+First, we start sandbox passing in the ``coin-upgrade`` DAR. Since a
+DAR includes all transitive dependencies, this includes ``coin-1.0.0``
+and ``coin-2.0.0``.
+
+.. code-block:: none
+
+   $ cd example/coin-upgrade
+   $ daml sandbox .daml/dist/coin-upgrade-1.0.0.dar
+
+To simplify the setup here, we use a DAML script to create 3 parties
+Alice, Bob and Charlie and two ``Coin`` contracts issues by Alice, one
+owned by Bob and one owned by Charlie.
+
+.. literalinclude:: example/coin-initiate-upgrade/daml/InitiateUpgrade.daml
+   :language: daml
+   :start-after: -- SETUP_SCRIPT_BEGIN
+   :end-before: -- SETUP_SCRIPT_END
+
+Run the script as follows:
+
+.. code-block:: none
+
+   $ cd example/coin-initiate-upgrade
+   $ daml build
+   $ daml script --dar=.daml/dist/coin-initiate-upgrade-1.0.0.dar --script-name=InitiateUpgrade:setup --ledger-host=localhost --ledger-port=6865 --wall-clock-time
+
+If you now start Navigator from the ``coin-initiate-upgrade``
+directory and log in as Alice, you can see the two ``Coin`` contracts.
+
+Next, we run the trigger for Alice. The trigger will keep running throughout the
+rest of this example.
+
+.. code-block:: none
+
+   $ cd example/coin-upgrade-trigger
+   $ daml build
+   $ daml trigger --dar=.daml/dist/coin-upgrade-trigger-1.0.0.dar --trigger-name=UpgradeTrigger:upgradeTrigger --ledger-host=localhost --ledger-port=6865 --ledger-party=Alice --wall-clock-time
+
+With the trigger running, we can now run the script to create the
+``UpgradeCoinProposal`` contracts (we could also have done that before
+starting the trigger). The script takes an argument of type
+``Party``. We can pass this in via the ``--input-file`` argument which
+we will point to a file ``party.json`` containing ``"Alice"``. This allows us to
+change the party without having to change the code of the script.
+
+.. code-block:: none
+
+   $ cd example/coin-initiate-upgrade
+   $ daml build
+   $ daml script --dar=.daml/dist/coin-initiate-upgrade-1.0.0.dar --script-name=InitiateUpgrade:initiateUpgrade --ledger-host=localhost --ledger-port=6865 --wall-clock-time --input-file=party.json
+
+At this point, our trigger is running and the ``UpgradeCoinProposal``
+contracts for Bob and Charlie have been created. What is left to do is
+to accept the proposals. Our trigger will then automatically pick them
+up and upgrade the ``Coin`` contracts.
+
+First, start Navigator and log in as Bob. Click on the
+``UpgradeCoinProposal`` and accept it. If you now go back to the
+contracts tab, you can see that the ``Coin`` contract has been
+archived and instead there is a new ``CoinWithAmount`` upgrade. Our
+trigger has successfully upgraded the ``Coin``!
+
+Next, log in as Charlie and accept the ``UpgradeCoinProposal``. Just
+like for Bob, you can see that the ``Coin`` contract has been archived
+and instead there is a new ``CoinWithAmount`` contract.
+
+Since we upgraded all ``Coin`` contracts issued by Alice, we can now stop the
+trigger and declare the update successful.

docs/source/upgrade/example/coin-1.0.0/daml.yaml

@@ -11,4 +11,3 @@ dependencies:
 # END
 parties: [Alice, Bob]
 source: .
-build-options: [--target=1.8]

docs/source/upgrade/example/coin-2.0.0/daml.yaml

@@ -11,4 +11,3 @@ dependencies:
 # END
 parties: [Alice, Bob]
 source: .
-build-options: [--target=1.8]

docs/source/upgrade/example/coin-initiate-upgrade/daml.yaml

@@ -0,0 +1,18 @@
+# Copyright (c) 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+sdk-version: 0.0.0
+# BEGIN
+name: coin-initiate-upgrade
+version: 1.0.0
+dependencies:
+  - daml-prim
+  - daml-stdlib
+  - path/to/daml-script.dar
+data-dependencies:
+  - path/to/coin-upgrade-1.0.0.dar
+  - path/to/coin-1.0.0.dar
+  - path/to/coin-2.0.0.dar
+# END
+parties: [Alice, Bob, Charlie]
+source: .

docs/source/upgrade/example/coin-initiate-upgrade/daml/InitiateUpgrade.daml

@@ -0,0 +1,35 @@
+-- Copyright (c) 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
+-- SPDX-License-Identifier: Apache-2.0
+
+module InitiateUpgrade where
+
+import DA.Foldable
+import DA.List
+import Daml.Script
+
+import CoinV1
+import UpgradeFromCoinV1
+
+-- INITIATE_UPGRADE_BEGIN
+initiateUpgrade : Party -> Script ()
+initiateUpgrade issuer = do
+  coins <- query @Coin issuer
+  let myCoins = filter (\(_cid, c) -> c.issuer == issuer) coins
+  let owners = dedup $ map (\(_cid, c) -> c.owner) myCoins
+  forA_ owners $ \owner -> do
+    debug ("Creating upgrade proposal for: " <> show owner)
+    submit issuer $ createCmd (UpgradeCoinProposal issuer owner)
+-- INITIATE_UPGRADE_END
+
+-- SETUP_SCRIPT_BEGIN
+setup : Script ()
+setup = do
+  alice <- allocatePartyWithHint "Alice" (PartyIdHint "Alice")
+  bob <- allocatePartyWithHint "Bob" (PartyIdHint "Bob")
+  charlie <- allocatePartyWithHint "Charlie" (PartyIdHint "Charlie")
+  bobProposal <- submit alice $ createCmd (CoinProposal alice bob)
+  submit bob $ exerciseCmd bobProposal CoinProposal_Accept
+  charlieProposal <- submit alice $ createCmd (CoinProposal alice charlie)
+  submit charlie $ exerciseCmd charlieProposal CoinProposal_Accept
+  pure ()
+-- SETUP_SCRIPT_END

docs/source/upgrade/example/coin-initiate-upgrade/party.json

@@ -0,0 +1 @@
+"Alice"

docs/source/upgrade/example/coin-upgrade-trigger/daml.yaml

@@ -0,0 +1,18 @@
+# Copyright (c) 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
+# SPDX-License-Identifier: Apache-2.0
+
+sdk-version: 0.0.0
+# BEGIN
+name: coin-upgrade-trigger
+version: 1.0.0
+dependencies:
+  - daml-prim
+  - daml-stdlib
+  - path/to/daml-trigger.dar
+data-dependencies:
+  - path/to/coin-upgrade-1.0.0.dar
+  - path/to/coin-1.0.0.dar
+  - path/to/coin-2.0.0.dar
+# END
+parties: [Alice, Bob]
+source: .

docs/source/upgrade/example/coin-upgrade-trigger/daml/UpgradeTrigger.daml

@@ -0,0 +1,63 @@
+-- Copyright (c) 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
+-- SPDX-License-Identifier: Apache-2.0
+
+module UpgradeTrigger where
+
+import DA.Assert
+import DA.Foldable
+import qualified DA.Next.Map as Map
+import DA.Next.Map (Map)
+import Daml.Trigger
+import Daml.Trigger.Assert
+
+import CoinV1
+import UpgradeFromCoinV1
+
+-- TRIGGER_BOILERPLATE_BEGIN
+upgradeTrigger : Trigger ()
+upgradeTrigger = Trigger with
+  initialize = \_acs -> ()
+  updateState = \_acs _msg () -> ()
+  registeredTemplates = AllInDar
+  heartbeat = None
+  rule = triggerRule
+-- TRIGGER_BOILERPLATE_END
+
+-- TRIGGER_RULE_BEGIN
+triggerRule : Party -> ACS -> Time -> Map CommandId [Command] -> () -> TriggerA ()
+triggerRule issuer acs _ _ _ = do
+  let agreements =
+        filter (\(_cid, agreement) -> agreement.issuer == issuer) $
+        getContracts @UpgradeCoinAgreement acs
+  let allCoins =
+        filter (\(_cid, coin) -> coin.issuer == issuer) $
+        getContracts @Coin acs
+  forA_ agreements $ \(agreementCid, agreement) -> do
+    let coinsForOwner = filter (\(_cid, coin) -> coin.owner == agreement.owner) allCoins
+    forA_ coinsForOwner $ \(coinCid, _) ->
+      emitCommands
+        [exerciseCmd agreementCid (Upgrade coinCid)]
+        [toAnyContractId coinCid]
+-- TRIGGER_RULE_END
+
+-- TODO (MK) The Bazel rule atm doesn’t run this scenario, we should fix that.
+test = scenario do
+  alice <- getParty "Alice"
+  bob <- getParty "Bob"
+
+  coinProposal <- submit alice $ create (CoinProposal alice bob)
+  coin <- submit bob $ exercise coinProposal CoinProposal_Accept
+
+  upgradeProposal <- submit alice $ create (UpgradeCoinProposal alice bob)
+  upgradeAgreement <- submit bob $ exercise upgradeProposal Accept
+
+  let acs = toACS coin <> toACS upgradeAgreement
+
+  commands <- testRule upgradeTrigger alice acs Map.empty ()
+  let flatCommands = flattenCommands commands
+  assertExerciseCmd flatCommands $ \(cid, choiceArg) -> do
+    cid === upgradeAgreement
+    choiceArg === Upgrade coin
+  -- TODO (MK) It would be nice to test for the absence of certain commands as well
+  -- or ideally just assert that the list of emitted commands matches an expected
+  -- list of commands.

docs/source/upgrade/example/coin-upgrade/daml.yaml

@@ -14,4 +14,3 @@ data-dependencies:
 # END
 parties: [Alice, Bob]
 source: .
-build-options: [--target=1.8]

docs/source/upgrade/example/coin-upgrade/daml/UpgradeFromCoinV1.daml

@@ -15,6 +15,8 @@ template UpgradeCoinProposal
   where
     signatory issuer
     observer owner
+    key (issuer, owner) : (Party, Party)
+    maintainer key._1
     choice Accept : ContractId UpgradeCoinAgreement
       controller owner
       do create UpgradeCoinAgreement with ..
@@ -27,6 +29,8 @@ template UpgradeCoinAgreement
     owner : Party
   where
     signatory issuer, owner
+    key (issuer, owner) : (Party, Party)
+    maintainer key._1
     nonconsuming choice Upgrade : ContractId CoinWithAmount
       with
         coinId : ContractId Coin

docs/source/upgrade/index.rst

@@ -6,6 +6,11 @@
 Upgrading and extending DAML applications
 #########################################
 
+.. toctree::
+   :hidden:
+
+   automation
+
 **Note:** Cross-SDK upgrades require DAML-LF 1.8 or newer.
 This is the default starting from SDK 1.0. For older releases add
 ``build-options: ["--target=1.8"]`` to your ``daml.yaml`` to select
@@ -225,8 +230,17 @@ Finally, we point a browser to http://localhost:4000 and can effect the coin upg
 
 #. Login as Alice
     #. Select Templates tab.
-    #. Create an `UpgradeCoinProposal` with Alice as issuer and Bob as owner.
+    #. Create an ``UpgradeCoinProposal`` with Alice as issuer and Bob as owner.
 #. Login as Bob
-    #. Exercise the `Accept` choice of the upgrade proposal, creating an `UpgradeCoinAgreement`.
+    #. Exercise the ``Accept`` choice of the upgrade proposal, creating an ``UpgradeCoinAgreement``.
 #. Login again as Alice
-    #. Use the `UpgradeCoinAgreement` repeatedly to upgrade any coin for which Alice is issuer and Bob is owner.
+    #. Use the ``UpgradeCoinAgreement`` repeatedly to upgrade any coin for which Alice is issuer and Bob is owner.
+
+Further Steps
+=============
+
+For the upgrade of our coin model above, we performed all steps
+manually via Navigator. However, if Alice had issued millions of
+coins, performing all upgrading steps manually becomes infeasible.  It
+thus becomes necessary to automate these steps. We will go through a
+potential implementation of an automated upgrade in the :ref:`next section <upgrade-automation>`.

rules_daml/daml.bzl

@@ -128,6 +128,7 @@ _daml_build = rule(
         ),
         "dar_dict": attr.label_keyed_string_dict(
             mandatory = True,
+            allow_files = True,
             doc = "Other DAML projects referenced by this DAML project.",
         ),
         "dar": attr.output(