daml/daml-lf/governance.rst

.. Copyright (c) 2019 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
.. SPDX-License-Identifier: Apache-2.0

DAML-LF Governance process
==========================

These are the principles and rules by which changes to three data
formats are governed by the DAML Runtime team:

* values,
* transactions, and
* language.

Overview
--------

* *DAML-LF serializable values* or (DAML-LF values for brevity in the
  rest of this document) is the universe of DAML-LF values that can be
  stored on the ledger. Currently nominal records, nominal variants,
  lists, and primitive types.

* *DAML-LF transactions* are the data structure that defines an update
  to a ledger. Currently comprised of create, exercise, and fetch
  nodes.

* The *DAML-LF language* is the typed core calculus that DAML gets
  compiled to, and that the ledger runs.

* DAML-LF values and transactions are evolved without ever breaking
  backwards compatibility. They are evolved separatedly, with
  transactions building on top of values.

* Specification documents are provided for DAML-LF values, DAML-LF
  transactions, and for the DAML-LF language.  

* Every merged change to DAML-LF values, transactions, or language
  *must* come with changes to the appropriate specification
  document. Moreover, a one-line summary of each version must be
  present in the ``.proto`` files for DAML-LF values, transactions, and
  language.

* Every change to DAML-LF values or transactions *must* be checked for
  governance rules conformance and approved by Stephen Compall, or by
  Francesco Mazzoli if Stephen is on vacation.  Every change to the
  DAML-LF language *must* be checked for governance rules conformance
  and approved by Francesco Mazzoli, or by Rémy Haemmerle if Francesco
  is on vacation.

  These checks are for governance and versioning only; the substance of
  DAML-LF language changes should be made in consultation with the DAML
  Language team, in particular Martin Huschenbett.  Proposed changes can
  be filed as tickets under Milestone "DAML-LF Spec & Validation",
  labels "component/daml-lf" and "discussion".

* Every ledger implementation must declare what DAML-LF value,
  transaction, and language versions it supports. This information will
  be most likely tied to what DAML-LF engine the ledger is running, but
  additional constraints external to the DAML-LF engine can be put in
  place, if necessary. Note that a running ledger can never drop support
  for value, transaction, or language versions, since it must be able to
  replay every transaction ever run on the ledger. However the ledger is
  free to refuse to accept new code running with older DAML-LF versions
  while still maintaining support for existing code internally.

* Client libraries and other tooling mostly care about DAML-LF values
  and DAML-LF serializable types. Deciding how these components handle
  versions is not part of the DAML-LF governance progress. For
  example, a certain version of the DAML-LF Scala codegen might
  support a specified set of DAML-LF versions -- much like ledger
  implementations.

Values
------

Each DAML-LF value and DAML-LF transaction comes with a version number,
indicating how the respective specification should be consulted for
rules regarding that value or transaction.

Every change to the ``.proto`` specification must yield a specification
capable of consuming all values produced by any prior version of the
specification. New fields can be added, the human-readable names of
existing fields can be changed, and the semantics of fields can be
changed according to the version number included with the enclosing
value or transaction.  Moreover, messages produced via the new proto may
be consumed via any prior proto, but *only* to the extent of completing
without error and providing the version number.  The remaining contents
of the message are neither required to make sense, nor preserve the
semantics of older versions.

This "old data can be consumed" form is the only form of compatibility
supported.  That is because there is no way for a consumer to anticipate
whether new, unreadable data may be essential to consuming a given
value.  Consumers can detect gracefully if they get a value or a
transaction that they do not support, and *must* reject it rather than
"best-effort" consuming it in such a case, because any field might be
ignored when within values/transactions of later versions.

These versioning rules are described in more detail, with examples, in
the value specification.

For DAML-LF values we have a single document for all the versions, with
annotations on version-specific features (in the form of "since version
XYZ").

Note how the versioning is not per-node, but as a whole. The version is
decided as the DAML-LF engine generates a value -- at contract creation
time and contract exercise time (for choice arguments). Client libraries
are free to present all values according to latest DAML-LF value they
understand.

Permanent backward compatibility
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

*This section is non-normative.*

Unlike the language, values and transactions have only one major version
each; in terms of how language is versioned, they only have minor
versions.  Their treatment is different for a few reasons:

- We expect DAML-LF values to be often stored in a long-term fashion -
  for example a ledger needs to do so.

- It should be possible to revalidate old transactions at any time;
  requiring users to maintain an archive of every DAML-LF engine version
  they ever used in order to be able to revalidate would be extremely
  onerous, and ledger operators should be able to upgrade the engine
  freely.

- While we could have major revisions and have each ledger
  deployment to have a blessed set of major revisions, like we plan do
  for code, we expect the impact of such a decision to be much greater,
  since there is essentially only one consumer of DAML-LF code (the
  server), while DAML-LF values will be stored all over the place, and
  the disruption of breaking changes would be too burdensome.

- It is *much* easier to preserve compatibility of simple first-order
  data, compared to a full-blown language.

Transactions
------------

Transactions are specified and versioned according to the same rules as
those for values, so everything above applies equally to transactions as
it does for values.  The only difference is that, as a separate
specification, transactions have their own versions, independent of
value versions.

Transactions incorporate values; however, this does not mean that the
transaction's version depends on any of those values' versions.  Every
occurrence of a value in a transaction is accompanied by a version of
the value specification under which that value should be considered.

As with values, transactions are versioned as a whole, with the
DAML-LF engine picking a version when it creates a new transaction.

Language
--------

The DAML-LF *language* is versioned using a major and minor component.
Increasing the major component allows us to drop features or change
semantics of existing features, and implies creating an entirely new
``.proto`` message for the new major version. Changes to the minor
component cannot break backwards compatibility, and operate on an
existing major version and its ``.proto`` message in a backwards
compatible way; "compatibility" here means exactly what it does for
values and transactions, i.e. "old data can be read"; older serialized
``.proto`` messages from the same major version are still readable and
their semantics are unchanged.

Note that some major or minor version bumps might not change the
``.proto`` at all, but just add / change semantics in the
specification. For example, the string format for decimal numbers in
DAML-LF values might be made more lenient, resulting in a minor version
bump with no ``.proto`` schema changes (but probably a comment).

Also note that the DAML-LF versioning is independent from the DAML
surface language versioning.

We have one specification document per major version, with each document
noting the differences between minor versions within that major version.

"dev" version
~~~~~~~~~~~~~

Every DAML-LF major version includes a minor version, the *dev* version,
which we use as staging area for the next stable minor version of that
major version.

The "dev" DAML-LF major version can be changed freely without
compatibility considerations to prior "dev" versions.  Since "dev" is
always considered to be newer than every stable minor version of that
major version, it must be backward-compatible with all such stable
versions.

All newly-implemented minor version features or changes must be staged
in the *dev* version for later release as a stable group of
backward-compatible features.

The DAML-LF dev version is enabled in the sandbox and ledger server, but
will never be emitted by damlc unless explicitly requested via
``--target 1.dev`` or similar.

Working with LF data
--------------------

The DAML Runtime team provides libraries to read DAML-LF values, DAML-LF
transactions, and DAML-LF packages in a version-aware manner, to aid the
implementation of readers and writers.

With "version-aware" we mean that the libraries encoding and decoding
data structures are aware of what versions they support, and will fail
gracefully if they encounter unknown versions.  Note that the value and
transaction specifications *require* that consumers are version-aware in
this way.  Because this only becomes more complex as the specifications
evolve, we strongly recommend that JVM-hosted applications use our
libraries for encoding and decoding.

The ``daml-lf/transaction`` library for DAML-LF values and transactions
includes:

.. todo include this at some point? - Specification

- ``.proto`` definitions for both, in two different files, with the
  transaction one referencing the value one.
- Enforcement of the specifications' rules, not just the proto structure
- Data structures to work with values and transactions from at least
  Scala and Java, and functions to decode / encode between the
  ``.proto`` and the data structures.

The ``daml-lf/lfpackage`` library for the DAML-LF language includes:

.. todo include this at some point? - Specifications, one per major revision

- Data structures to work with DAML-LF packages. Currently, this is an
  AST that supports a set of DAML-LF versions, spanning across multiple
  major revisions, either by having a larger type or by converting some
  features into other features (e.g. if we add dependent functions the
  library would convert non-dependent functions and foralls into
  dependent functions).

``lfpackage`` is built upon the ``daml-lf/archive`` library, which
includes:

- The language ``.proto`` files, with a top-level sum type to identify
  the major revision.

The ``daml-lf/validation`` library includes:

- Enforcement of the language specifications' rules, not just the proto
  structure, against ``lfpackage``'s AST
- Checking properties of the synthesis of the different specifications,
  e.g. that a particular value conforms to a type

Engine
------

The DAML Runtime team provides the DAML-LF engine. The DAML-LF engine
will advertise what versions of the DAML-LF values and language it
accepts (in commands and packages respectively). Note that the DAML-LF
engine also produces versioned DAML-LF values and transactions. We do
not currently specify the algorithm that decides what version the
DAML-LF engine will use when producing values and transactions.

We do however guarantee that, for the engine:

- Let *er* = DAML-LF engine release,

- let *lv* = DAML-LF language version,

- let *ts* = contract template signature,

- let *vt* = value or transaction;

- we use the deterministic algorithm *vtv* to decide the DAML-LF value /
  transaction version, ``er × lv × ts × vt -> version``;

- If we fix *lv*, *ts*, and *vt*, *vtv* is monotonically decreasing. In
  other words, if the current release of the DAML-LF engine produces
  values of version ``X`` for values regarding a certain contract
  template, future releases will produce DAML-LF values of some version
  ``Y`` such that ``Y ≤ X``.

This allows deployments to use newer DAML-LF language versions while
still working with older clients.

This set of rules applies only to the DAML engine; producers of values
and transactions are not generally required to adhere to these rules,
and may simply use the latest versions of the specifications they
support.

In any case, once a versioned DAML-LF value / transaction is produced by
the DAML-LF engine, the committer must store it together with its
version, which is fixed forever. Note that a ledger implementation is
free to store transactions in some new better format, as long as the
original versioned transaction is stored too so that the whole ledger
can be verified at will.

More on version selection
~~~~~~~~~~~~~~~~~~~~~~~~~

*This section is non-normative.*

We are leaving what version to pick for values and transactions up to
the DAML-LF engine, and not specifying how the DAML-LF engine picks
these versions. This is because we have a few options that we do not
want to commit to right now. Specifically:

- Pin the DAML-LF value version to the DAML-LF language version.  Or in
  other words have a map from DAML-LF language version to what DAML-LF
  value and transaction version to generate. This is problematic because
  it prevents us from generating values compatible with older clients
  with newer versions of the DAML-LF language. For example, say we have
  DAML-LF language 1.0 pinned to DAML-LF value 1. Then we upgrade
  DAML-LF to version 1.1, which adds support for a new numeric type,
  which is available in DAML-LF value 2. With this pinning scheme, *all*
  contract templates defined in DAML-LF 1.1 would use DAML-LF values 2,
  even if they do not use the new numeric type in question. And thus old
  clients will be unable to read all values generated from DAML-LF 1.1,
  even if many contract templates in DAML-LF 1.1 won't make use of the
  new numeric type at all.

- Have the DAML-LF engine to generate values and transactions of the
  lowest version possible. This is the best and "smartest" solution,
  given that values and transactions are forever backwards compatible.
  However, we aren't fully cognizant of what else this might imply.

- Have the users of DAML specify what DAML-LF value version to use for
  each template. This is probably the most "principled" solution, in the
  sense that it's the easiest to support while providing predictable
  results. However it burdens the users with something they are not
  burdened with today, and embeds a highly esoteric choice into the
  writing of each surface language template.

So in the meantime we make the weaker promise in the previous section,
which we might make stronger in the future.