Context
=======
After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.
This PR is a proposal for a new release process designed under the
following constraints:
- Reuse as much as possible of the existing infrastructure, to minimize
effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
not meant for general public consumption, but can still be used by savvy
users without jumping through too many extra hoops (ideally just
swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
release status, with as few changes as possible in-between, so we can be
confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
releases, to avoid discovering non-transient problems while trying to
promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
keep the same approval process for SOC2 auditability.
The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:
```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```
This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.
Version numbers
===============
Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:
- Semantic versioning (which we chose as the version format for the SDK
version number) allows for "prerelease" version numbers as well as
"metadata"; an example of a complete version string would be
`1.2.3-nightly.201+server12.43`. The "main" part of the version string
always has to have 3 numbers separated by dots; the "prerelease"
(after the `-` but before the `+`) and the "metadata" (after the `+`)
parts are optional and, if present, must consist of one or more segments
separated by dots, where a segment can be either a number or an
alphanumeric string. In terms of ordering, metadata is irrelevant and
any version with a prerelease string is before the corresponding "main"
version string alone. Amongst prereleases, segments are compared in
order with purely numeric ones compared as numbers and mixed ones
compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
`-`, or a transition between a number and a letter. Version strings
are compared element-wise, with numeric segments being compared as
numbers. Alphabetic segments are treated specially if they happen to be
one of a handful of magic words (such as "alpha", "beta" or "snapshot"
for example) which count as "qualifiers"; a version string with a
qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
special ordering amongst qualifiers. Other alphabetic segments are
compared alphabetically and count as being "after" their prefix
(`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
separated by `.`, plus an optional (though deprecated) alphanumeric
"version tag" separated by a `-`. I could not find any official
documentation on ordering for the version tag; numeric segments are
compared as numbers.
- npm uses semantic versioning so that is covered already.
After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:
```
0.13.53-alpha.20200214.3025.ef5d32b7
```
where the components are, respectively:
- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
master.
- `3025`: the number of commits in master up to the release commit
(included). Because we have a linear, append-only master branch, this
uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
not strictly speaking necessary, but makes it a lot more convenient to
identify the commit.
The main downsides of this format are:
1. It is not a valid format for GHC packages. We do not publish GHC
packages from the SDK (so far we have instead opted to release our
Haskell code as separate packages entirely), so this should not be an
issue. However, our SDK version currently leaks to `ghc-pkg` as the
version string for the stdlib (and prim) packages. This PR addresses
that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
would see the above version number as `0.13.53.20200214.3025`, which
should be enough to uniquely identify it. Note that, as far as I could
find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
it makes it more scary. However, I have been told that this may be
long enough to cause issues on Windows by pushing us past the max path
size limitation of that "OS". I suggest we try it and see what
happens.
The upsides are:
- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
they have the same date, allowing us to release same-day patches if
needed. (Note: that is, commits that were made on the same day; the
release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
GHC).
Alternatives I considered:
- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
be accepted by all schemes, but will not sort as expected under semantic
versioning (though Maven will be fine). I have no idea how it will sort
under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
This is not valid semantic versioning and is therefore rejected by
npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
what is generally done in the Java world, but we then lose track of what
version is actually in use and I'm concerned about bug reports. This
would also not let us publish to the main Maven repo (at least not more
than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
version formats. However, it would not clearly indicate to humans that
it is not meant as a stable version, and would sort differently under
semantic versioning (which counts it as a prerelease, i.e. before
`0.13.53`) than under maven (which counts it as a patch, so after
`0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
number of prereleases in-between `0.13.52` and the next. This is
currently the fallback plan if Windows path length causes issues. It
would be less convenient to map releases to commits, but it could still
be done via querying the history of the `LATEST` file.
Release notes
=============
> Note: We have decided not to have release notes for snapshot releases.
Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.
But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.
The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.
> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.
New release process
===================
Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.
Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:
1. Before running the release step, CI will checkout the commit
specified in the LATEST file. This requires separating the release
step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
that gets the version to build from an environment variable, with a
default of `0.0.0` to remain consistent with the current `daml-head`
behaviour.
Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.
The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.
I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.
CHANGELOG_BEGIN
CHANGELOG_END
* GSG: Explain Sandbox and JSON API better and improve wording in intro
Also adapt to lack of sign up.
changelog_begin
changelog_end
* Address comments
* Add documentation for DAML repl and advertise it
This PR adds some simple docs for ``daml repl`` and adds it to the
release notes.
changelog_begin
- [DAML Repl - Experimental] A new ``daml repl`` command that allows
you to use the ``DAML Script`` API interactively. Take a look at the
`documentation <https://docs.daml.com/daml-repl/>`_ for more
information.
changelog_end
* Update docs/source/daml-repl/index.rst
Co-Authored-By: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* s/Repl/REPL/
Co-authored-by: Andreas Herrmann <42969706+aherrmann-da@users.noreply.github.com>
* Rework description of using the app with recent changes
changelog_begin
changelog_end
* Formatting
Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Wording
Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Address more comments
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Explain UI better in app arch section
Still rough but better than the void of information there currently.
changelog_begin
changelog_end
* Finish sentence
* Reword introduction and explanation of codegen
CHANGELOG_BEGIN
CHANGELOG_END
* Say DAR instead of archive
Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Remove 'i'll explain later'
* Include links to sections of this guide
* More links
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Update MainView in App Arch section
* Update Feed to MessageList
* Update MessageEdit and MainView components and description in First Feature section
CHANGELOG_BEGIN
CHANGELOG_END
* Underline length
Co-Authored-By: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* Add more tests to default run of Ledger API Test Tool
Furthermore, drop TimeIT, which is a sandbox-only property (and tested already as part of its integration tests)
CHANGELOG_BEGIN
[DAML Ledger Integration Kit] Ledger API Test Tool default tests modified. Use --list for the updated list of default tests. Time service test dropped from the suite.
CHANGELOG_END
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380635979
* Address https://github.com/digital-asset/daml/pull/4561#discussion_r380636989
* Optimize imports
* Only run semantic tests on Canton
* Fix UI intro para
* Tweak UI explanation
* Address some feedback about the intro text
* More changes to wording about DAML
* Small wording fixes to feature section
changelog_begin
changelog_end
* Start drafting new quickstart guide
* Show how to run the app
* Very rough instructions for playing with the app
* Explain core of User template
* Start explaining UI code
* Example of daml react hook (useQuery for allUsers)
* Talk about daml2ts and start describing the new feature
* Start talking about DAML for posts feature
* Add ui file referenced in text
* Start describing changes to UI for Post feature
* Rework feature section wording as Messaging instead of Posts
* Write about additions to MainController
* Talk about new components before the view and controller (bottom up style)
* Describe additions to MainView (may change if we inline MainView into MainController)
* Adapt to create-daml-app removing MainController
* Fix code snippet and try to highlight tsx but fail
* Split guide into sections and rename some sections
* Improve start of app arch section
* Minor edits to code snippets and wording
* Update setup instructions with codegen step
* Update UI components in 'before' code
* Move and update section explaining TS codegen
* Copy in new DAML files
* Update UI code for messaging feature and some of the explanatory text
* Start reworking DAML feature section
* Redo DAML feature section
* Edit initial section
* Edit intro para of arch section
* Edit start of DAML explanation
* Edit template explanation
* Minor edit to UI explanation
* Improve wording of DAML explanation
* Rework sig/obs explanation
* Update User.daml file from create-daml-app and label AddFriend choice
* Explain AddFriend choice better
* Move new GSG to experimental features section
* Undo accidental change to vscode settings
changelog_begin
changelog_end
* Copyright headers
* Revert unwanted change
* Remove typescript highlighting which doesn't work
* Tweak explanation of code generation
* Remove driven
If you click on a section title, we hijack the link to animate
it. However, we did not update the location hash. This makes it quite
annoying to copy links to a specific section.
This PR sets the jquery animate callback to update the hash as
well. I’ve also included a small fix for the link on the section title
which previously produced an error in the JS console since "#" is not
a valid jquery selector.
changelog_begin
changelog_end
* Inserts alias
* ContractStreamStep, extending InsertDeleteStep with a boundary
* partitionBimap, an IDS utility for partitionMap on both inserts and deletes
* make the flow setup of acsFollowingAndBoundary a little clearer
* use partitionBimap to simplify websocket response stream
* make the flow setup of transactionsFollowingBoundary a little clearer
* acsFollowingAndBoundary becoming a ContractStreamStep producer
* insertDeleteStepSource also becomes a ContractStreamStep producer
* porting StepAndErrors to ContractStreamStep
* remove Acs constructor of ContractStreamStep
- it's an interesting idea for future potential features, but not useful
right now so just a needless complication
* adapt convertFilterContracts to presence of LiveBegin marker
* adapt removePhantomArchives to presence of LiveBegin marker
* test that live marker is emitted in the right place
* document liveness marker
* add changelog
CHANGELOG_BEGIN
- [JSON API - Experimental] Add ``{"live": true}`` to WebSocket streams
to mark the beginning of "live" data. See `issue #4461
<https://github.com/digital-asset/daml/issues/4461>`_.
This marker is a placeholder feature; `issue #4509 bookmarks in query
streams <https://github.com/digital-asset/daml/issues/4509>`_ will
obsolete this marker, after which it will no longer be emitted. When
building features on the marker, be aware of this forthcoming
replacement.
CHANGELOG_END
* be a little more prepared for offsets
This should provide a better migration path for people that still rely
on static time by forcing them to make this explicit. Given that both
DAML script and DAML triggers are still experimental, I’m not marking
this as a breaking change
changelog_begin
- [DAML Script - Experimental] The time mode must now always be
specified explicitly. Use ``--static-time`` to recover the previous
default time mode.
- [DAML Triggers - Experimental] The time mode must now always be
specified explicitly. Use ``--static-time`` to recover the previous
default time mode.
changelog_end
* expand documentation on WebSockets subprotocols to a more-visible section
CHANGELOG_BEGIN
CHANGELOG_END
* combine reference in websocket section to auth section
* resection
* subprotocols sometimes called protocols
- suggested by @hurryabit; thanks
We've created a cool video that happens to have the wrong link in it
(missin l in the extension), and changing videos is more expensive than
changing text files.
Also, the current URL is a tad long, so I've added a shorter one.
Note: as this is part of the docs bundle (actually generating an HTML
redirect page), this won't be live until next version is published.
CHANGELOG_BEGIN
CHANGELOG_END
* Support multiple-packages in `damlc ide`
changelog_begin
- [DAML Studio] You can now open DAML Studio in the root of a
multi-package project instead of opening it separately for each
package. Take a look at the documentation for details on how to set
this up.
changelog_end
There are a few caveats here:
1. You need a ``daml.yaml`` in the root of your project directory. I
think this is somewhat sensible but we should add a warning to VSCode
if you open it in a directory that does not have a ``daml.yaml`` (in a
separate PR).
2. Changes are not picked up accross dependencies. This is a larger
undertaking and given the current setup simply impossible (we don’t
know that the source files of one package belong to the DAR referenced
in the ``dependencies`` field of the other package. We can make this a
bit better by at least detecting that the ``.dar`` has changed but
let’s do that separately.
3. Since ``daml init`` runs once on startup, it will run in the root
directory instead of initializing the package db of the individual
packages. This is fixable but will conflict with #4391 so let’s
address this separately.
I’ve added docs to the daml studio section that explain the caveats.
* Use the proper sdk version in lsp-tests
* move InsertDeleteStep to util
* turn append cid argument into a typeclass
* use Cid for appendForgettingDeletes as well
* add secondary data to the deletes in InsertDeleteStep with new covariant tparam
* make StepAndErrors hold ArchivedContracts so ws archives look like exercise archives
* add changelog
CHANGELOG_BEGIN
- [JSON API - Experimental] The format of ``archived`` responses from WebSocket endpoints
has changed to include template IDs, similar to exercise responses.
See `issue #4383 <https://github.com/digital-asset/daml/issues/4383>`_.
CHANGELOG_END
* rename "contracts" to "events" in JSON API exercise response
CHANGELOG_BEGIN
- [JSON API - Experimental] Exercise response field "contracts" renamed to "events".
See `issue #4385 <https://github.com/digital-asset/daml/issues/4385>`_.
CHANGELOG_END
* more events in doc
- pointed out by @leo-da; thanks
* Remove phantom archive when streaming events filtered by contract key
* Add fetch test cases, WIP
* Add fetch test cases, WIP
* Add fetch test cases, WIP
* Add fetch test cases, WIP
* Test case to validate phantom archive filter for stream/fetch
* Use `scan` instead of `statefulMapConcat` to filter out phantom archives
advantage(???) immutable state vs mutable
* make `com.digitalasset.http.WebsocketEndpoints.handleWebsocketRequest` public
so DABL can call it directly, `private[http]` does not work since DABL
is under com.projectdabl, see #4190
* Fixing typos
* Scalafmt
* Scalafmt
* Removing unused function
* Update docs
* Remove unused type alias
* CHANGELOG_BEGIN
[JSON API - Experimental]
- Added streaming version of fetch by key: ``/stream/fetch``. See #4075.
CHANGELOG_END
* Address code review comments
* Add test case with todo to address consistent handling of empty requests
* Fixing merge conflicts
* Addressing code review comments
* Addressing code review comments
Use `partition` instead of consecutive intersect and diff
[JSON API - Experimental]
Added a command line option to allow overriding default TTL.
``--default-ttl <value>`` Optional Time to Live interval to set if not provided in the command. Examples: 30s, 1min, 1h. Defaults to 30 seconds
CHANGELOG_END
* add SearchForeverRequest with one-or-many JSON model
* the least structured way to flatten a query union into a single stream
* add somewhere for indices to go in StreamQuery
* unzipMap utility
* doc Positive
* collect indices for SearchForever predicate, and use only one Map
* interface for rendering positions
* finish propagating positives from the predicate to the rendering phase
* add matchedQueries to every `created` in the searchForever results
* test that matchedQueries indices are included in query stream blocks
* document query union
* add changelog
CHANGELOG_BEGIN
- [JSON API - Experimental] ``/contracts/searchForever`` accepts multiple queries,
and includes with each ``created`` result the ``matchedQueries`` indicating which
queries matched.
See `issue #4363 <https://github.com/digital-asset/daml/pull/4363>`_.
CHANGELOG_END
* remove unused unzipMap
* ensure archive of a key happens to left of create of same key in each WebSocket result block
CHANGELOG_BEGIN
- [JSON API - Experimental] In websocket endpoints, if a 'created' and 'archived' contract
in the same result array share a contract key, the 'archived' is guaranteed to occur
earlier in the array than the 'created'.
See `issue #4354 <https://github.com/digital-asset/daml/issues/4354>`_.
CHANGELOG_END
* by key I mean contract key
- suggested by @hurryabit; thanks
In the integrity concepts, a party is defined as `C`, but referenced
later as `A`. As the rest of the docs use `A` for the party and `C` for
the contract, this seems to be a mistake. This corrects that
inconsistency.
This commit also clarifies a sentence and fixes a typo.
CHANGELOG_BEGIN
CHANGELOG_END
* Updated roadmap for Jan. 2020
* Removed Canton update
* Wording fixes
CHANGELOG_BEGIN
Removed previous roadmap (September 2019)
Added new one for January 2020
CHANGELOG_END
* Release 0.13.50
changelog_begin
changelog_end
* force clean build on Windows
apparently our cache is borked again …
* undo windows changes
Co-authored-by: Moritz Kiefer <moritz.kiefer@purelyfunctional.org>
* in searchForever, warn on unknown template IDs as long as at least one is known
* remove unused resolveTemplateId
* factor out WS request parts in WS integration test
* factor out IOU create
* test early template ID warning in searchForever stream
* document warnings case for searchForever
CHANGELOG_BEGIN
- [JSON API - Experimental] Precede stream with warnings of unknown template IDs, if any,
rather than failing outright.
See `issue #4290 <https://github.com/digital-asset/daml/issues/4290>`_.
CHANGELOG_END
Before, we didn't mention in the docs that sum-types need to derive from
(Eq,Show) if they are to be used as arguments of templates or choices.
Now we do.
CHANGELOG_BEGIN
CHANGELOG_END
