daml/release/RELEASE.md

339 lines
15 KiB
Markdown
Raw Normal View History

2019-04-04 12:55:22 +03:00
# Making a Release
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
First, you need to decide whether you are making a technical snapshot
("prerelease" at the github level, hereafter "snapshot") or an officially
supported release (hereafter "stable release"). For the latter, there are
extra steps marked as **[STABLE]** in the following instructions. You have to
skip those if you are making a snapshot release, that is, one intended mostly
for internal use and early testing, but which makes no promises about future
compatibility.
In either case, before going through the following list, you need to know which
commit you want to create the release from, `$SHA`. For a stable release, it is
highly recommended that this be the same commit as the latest existing
snapshot, so we "bless" an existing, tested version of the SDK rather than try
our luck with a random new one. For a snapshot, this should generally be the
latest commit on master.
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
1. **[STABLE]** Coordinate with the product and marketing teams to define
release highlights, tweets, blog posts, as well as timeline for publishing
the release. Define a version number, `$VERSION`. The following command may
be useful as a starting point; it will list all changes between the previous
stable release and the latest snapshot release:
```
./release.sh changes stable latest
```
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
1. Pull the latest master branch of the `daml` repository and create a new,
clean branch off it.
- For a snapshot, run `./release.sh snapshot HEAD`.
- If applicable (e.g. latest release was a stable one), edit `LATEST` to
update the release "version". For example, change:
```
6ea118d6142d2a937286b0a7bf9846dbcdb1751b 0.13.56-snapshot.20200318.3529.0.6ea118d6
```
to:
```
6ea118d6142d2a937286b0a7bf9846dbcdb1751b 0.13.57-snapshot.20200318.3529.0.6ea118d6
```
- For a stable release, run `echo "$SHA $VERSION" > LATEST`.
- Ideally, for a stable release, the resulting change is only to cut off
the prerelease part of the version number (the `-snapshot...`).
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
1. **[STABLE]** In `docs/source/support/release-notes.rst`, add a new header
and label for the new version. (See previous releases as examples.)
Remove unreleased.rst (#3547) * Start working on getting rid of unreleased.rst Document new process in CONTRIBUTING.md, .github/pull_request_template.md and unreleased.rst (for good measure) Report the previous changelog additions here so that they're not lost in the mist of times. CHANGELOG_BEGIN - [DAML Stdlib] Added the ``NumericScale`` typeclass, which improves the type inference for Numeric literals, and helps catch the creation of out-of-bound Numerics earlier in the compilation process. - [DAML Triggers] ``emitCommands`` now accepts an additional argument that allows you to mark contracts as pending. Those contracts will be automatically filtered from the result of ``getContracts`` until we receive the corresponding completion/transaction. - [Navigator] Fixed a bug where Navigator becomes unresponsive if the ledger does not contain any DAML packages. - [Ledger-API] Add field ``gen_map`` in Protobuf definition for ledger api values. This field is used to support generic maps, an new feature currently in development. See issue https://github.com/digital-asset/daml/pull/3356 for more details about generic maps. The Ledger API will send no messages where this field is set, when using a stable version of DAML-LF. However the addition of this field may cause pattern-matching exhaustive warnings in the code of ledger API clients. Those warnings can be safely ignored until GenMap is made stable in an upcoming version of DAML-LF. - [JSON API - Experimental] CLI configuration to enable serving static content as part of the JSON API daemon: ``--static-content "directory=/full/path,prefix=static"`` This configuration is NOT recommended for production deployment. See issue #2782. - [Extractor] The app can now work against a Ledger API server that requires client authentication. See `issue #3157 <https://github.com/digital-asset/daml/issues/3157>`__. - [DAML Script] This release contains a first version of an experimental DAML script feature that provides a scenario-like API that is run against an actual ledger. - [DAML Compiler] The default DAML-LF version is now 1.7. You can still produce DAML-LF 1.6 by passing ``--target=1.6`` to ``daml build``. - [JSON API - Experimental] The database schema has changed; if using ``--query-store-jdbc-config``, you must rebuild the database by adding ``,createSchema=true``. See `issue #3461 <https://github.com/digital-asset/daml/pull/3461>`_. - [JSON API - Experimental] Terminate process immediately after creating schema. See issue #3386. - [DAML Stdlib] ``fromAnyChoice`` and ``fromAnyContractKey`` now take the template type into account. CHANGELOG_END * Document new release process to gather changelog additions * Change the release script to ignore unreleased.rst * Remove spurious unreleased.rst lines * Transition to use tags * Document new way to get changelog additions with tags * Update release/RELEASE.md Co-Authored-By: Gary Verhaegen <gary.verhaegen@digitalasset.com> * Address https://github.com/digital-asset/daml/pull/3547#discussion_r348438786 * Document correction process * Add copyright header to unreleased.sh * Update CONTRIBUTING.md Co-Authored-By: Gary Verhaegen <gary.verhaegen@digitalasset.com> * Modify CONTRIBUTING.md after @garyverhaegen-da's proposal * Make unreleased.sh run per commit and treat tags as case-insensitive * Fix documentation for replacements
2019-11-20 18:16:57 +03:00
Once we are ready to make a release stable, preliminary release
notes will already have been published to the blog, e.g., the
preliminary release notes for 1.0 were at
https://blog.daml.com/release-notes/1.0.
These release notes now have to be converted to Rst so we can
include them in the documentation. You can do that manually or you
can use `pandoc` to create a first version and then fine tune
that. For the latter, download everything inside `<div
class="section post-body">` from the web page displaying the
release notes and save it as `release_notes.html`. Then you can run
`pandoc release_notes.html -o release_notes.rst`. Now copy those
release notes under the header you created above in
`docs/source/support/release-notes.rst` and start editing
them. Here are a couple of things that you should pay attention to:
1. Try to match the formatting of previous release notes.
1. Make sure that links from the release notes to our documentation
point to the documentation for the version you are about to
release. In particular, this means that in Rst terminology all
of these are external links.
1. Pandoc does not seem to preserve markup of inline code blocks so
you will have to manually add wrap them in double backslashes.
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
1. Once this is done, create a GitHub pull request (PR) with the above changes
to the `LATEST` and (for a stable release) `release-notes.rst` files.
It is important that your PR changes no other file.
1. Get a review and approval on your PR and then merge it into master.
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
**[STABLE]** For a stable release, the approval **MUST** be from the team
lead of the Language, Runtime or Product team.
1. Once the CI checks have passed for the corresponding master build, the release
should be available on Maven Central and GitHub, and have a Git tag.
The release should be visible on GitHub with _prerelease_ status, meaning it's
not yet ready for users to consume. The release notes should not be defined yet
and will be adjusted later on. Maven central has a delay of around 20 minutes
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
until the new version is visible. (**Note:** The 20-minute delay is for
artifacts to be available through e.g. `mvn build`; the delay for artifacts to
show up in web searches on the Maven Central website is up to two hours. Do not
worry if the artifacts do not show on the website yet.)
1. Tests for the getting started guide
You will have to run through those steps on both Linux/MacOS and
on Windows. See the platform specific-instructions below for more
details on the order of steps you need to follow.
1. For these steps you will need the documentation for the
release you are about to make. Documentation is published at
every hour so if you wait for a bit you can go to
https://docs.daml.com/$VERSION/getting-started/index.html.
Otherwise, check out the commit that you are referencing in the `LATEST` file
and build documentation locally via `./docs/scripts/preview.sh`.
1. Create a new project using `daml new create-daml-app create-daml-app`
and switch to the project directory using `cd create-daml-app`.
1. Build the project using `daml build`.
1. Run the JavaScript codegen using `daml codegen js .daml/dist/create-daml-app-0.1.0.dar -o daml.js`.
1. Install yarn dependencies using `cd ui && yarn install`.
1. Run `daml start` from the project root directory.
1. In a separate terminal run `yarn start` from the `ui` directory.
1. Open two browser windows (you want to see them simultaneously ideally) at `localhost:3000`.
1. Log in as `Alice` in the first window, log in as `Bob` in the second window.
1. Make `Alice` follow `Bob`. Verify that `Bob` appears in the
list of users `Alice` is following. Verify in the other
browser window that `Alice` shows up in `Bob`s network.
1. Make `Bob` follow `Alice`. Verify that `Alice` appears in
the list of users `Bob` is following. Verify in the other
browser window that `Bob` shows up in `Alice`s network.
1. Kill the `daml start` process and the `yarn start` process.
1. Open the your first feature section of the GSG, e.g., from
https://docs.daml.com/$VERSION/getting-started/first-feature.html
if you did not build docs locally.
1. Run `daml studio --replace=always` from the project root
directory and open `User.daml`.
1. Copy the `Message` template from the documentation to the end of `User.daml`.
1. Copy the `SendMessage` choice from the documentation to the
`User` template below the `Follow` choice.
1. Close VSCode.
1. Run `daml build && daml codegen js .daml/dist/create-daml-app-0.1.0.dar -o daml.js`.
1. From the `ui` directory run `yarn install --force --frozen-lockfile`.
1. Run `code .` from the project root directory (the extension is
already installed, no need to use `daml studio`).
1. Create `MessageList.tsx`, `MessageEdit.tsx` and modify
`MainView.tsx` as described in the documentation.
1. Verify that you do not see errors in the typescript code in VSCode.
1. Close VSCode.
1. Run `daml start` from the project root directory.
1. In a separate terminal, run `yarn start` from the `ui` directory.
1. As before, open two browser windows at `localhost:3000` and log
in as `Alice` and `Bob`.
1. Make `Alice` follow `Bob`.
1. From `Bob`, select `Alice` in the `Select a follower` drop down,
insert `hi alice` in the message field and click on `Send`.
1. Verify that `Alice` has received the message in the other window.
1. Make `Bob` follow `Alice`.
1. From `Alice`, select `Bob` in the drop down insert `hi bob` in
the message field and click on `Send`.
1. Verify that `Bob` has received the message in the other window.
1. Run through the following test plan on one of Linux or macOS:
1. Install the new SDK using:
```
curl -sSL https://get.daml.com/ | sh -s $(cat LATEST | gawk '{print $2}')
```
Note: this assumes you have the up-to-date `LATEST` file, either because
you just checked out master or because you're still on the release PR
commit.
1. Run `daml version --assistant=yes` and verify that the new version is
selected as the assistant version and the default version for new projects.
1. Tests for `quickstart-java`
While this is no longer the default getting started guide we still test it
for now since it covers things not covered by the new GSG
(Navigator, scenarios, Maven artifacts, …)
1. Create a new project with `daml new quickstart quickstart-java`
and switch to it using `cd quickstart`.
1. Verify the new version is specified in `daml.yaml` as the `sdk-version`.
1. Run `daml start`. Your browser should be opened automatically at
`http://localhost:7500`. Login as `Alice` and verify that there is
1 contract and 3 templates. Close the tab and kill `daml start` using
`Ctrl-C`.
1. Run `daml build`.
1. In 3 separate terminals (since each command blocks), run:
1. `daml sandbox --wall-clock-time --port 6865 .daml/dist/quickstart-0.0.1.dar`
1. `daml script --dar .daml/dist/quickstart-0.0.1.dar --script-name Setup:initialize --ledger-host localhost --ledger-port 6865 --wall-clock-time && daml navigator server localhost 6865 --port 7500`
1. `daml codegen java && mvn compile exec:java@run-quickstart`
> Note: It takes some time for our artifacts to be available on Maven
> Central. If you try running the last command before the artifacts are
> available, you will get a "not found" error. Trying to build again _in
> the next 24 hours_ will result in:
>
> ```
> Failure to find ... was cached in the local repository, resolution will not be reattempted until the update interval of digitalasset-releases has elapsed or updates are forced
> ```
>
> This is Maven telling you it has locally cached that "not found" result
> and will consider it valid for 24h. To bypass that and force Maven to
> try the network call again, add a `-U` option, as in
> `mvn compile exec:java@run-quickstart -U`. Note that this is required to
> bypass your local cache of the failure; it will not be required for a
> user trying to run the quickstart after the artifacts have been
> published.
1. Point your browser to `http://localhost:7500`, login as `Alice` and verify
that there is 1 contract, 3 templates and 1 owned IOU.
1. Check that `curl http://localhost:8080/iou` returns:
```
{"0":{"issuer":"EUR_Bank","owner":"Alice","currency":"EUR","amount":100.0000000000,"observers":[]}}
```
1. Kill all processes.
1. Run `daml studio --replace=always`. This should open VSCode and trigger
the DAML extension that's bundled with the new SDK version. (The new
VSCode extension will not be in the marketplace at this point.)
1. Open `daml/Main.daml`.
1. Click on `Scenario results` above `setup` and wait for the scenario
results to appear.
1. Add `+` at the end of line 12, after `"Alice"` and confirm you get an
error in line 13.
1. Add `1` after the `+` and confirm you get an error in line 12.
1. Delete the `+1` and the `e` in `Alice` and verify that the scenario
results are updated to the misspelled name.
1. Right click on `eurBank` in line 18 and verify that "Go to Definition"
takes you to the definition in line 15.
1. Close all files.
1. Run through the tests for the getting started guide described above.
1. On your PR, add the comment:
> Manual tests passed on [Linux/macOS].
specifying which platform you tested on.
1. Run through the following test plan on Windows. This is slightly shortened to
make testing faster and since most issues are not platform specific.
1. Close any running SDK instance in PowerShell (Navigator or Sandbox).
1. Download and run the Windows installer (the `.exe` file) from
`https://github.com/digital-asset/daml/releases`.
1. If asked if you want to remove an existing installation, click `Yes`.
1. Open a new PowerShell.
1. Run `daml new quickstart` to create a new project and switch to it using
`cd quickstart`.
1. Run `daml start`.
1. Open your browser at `http://localhost:7500`, verify that you can login as
Alice and there is one template and one contract.
1. Kill `daml start` with `Ctrl-C`.
1. Run `daml studio --replace=always` and open `daml/Main.daml`. Verify that
the scenario result appears within 30 seconds.
1. Add `+` at the end of line 26 after `"Alice"` and verify that you get an
error.
1. Run through the tests for the getting started guide described above.
1. On your PR, add the comment:
> Manual tests passed on Windows.
1. If the release is bad, delete the release from [the releases
page](https://github.com/digital-asset/daml/releases). Mention why it is bad
as a comment on your PR, and **stop the process here**.
1. Add the label `Standard-Change` to your PR.
1. Go to [the releases page](https://github.com/digital-asset/daml/releases)
and edit the release to look better. For both types of release, the release
title should be the version number (i.e. same as the git tag). For a
snapshot, the message should be set to
> This is a snapshot release. Use at your own risk.
For a stable release, the message should contain the team-lead-approved
release notes, and the "prerelease" checkbox should be unticked.
1. Announce the release on the relevant internal Slack channels (#product-daml,
\#team-daml). Add release notes in a thread under your announcement. For a
stable release, these are the notes decided with the product team; for a
snapshot release, include both the changes in this release (i.e. since the
last snapshot) and the complete list of changes since the last stable
release. Use the raw output of `unreleased.sh`.
You can produce the changes since the previous (snapshot or stable) release
by running:
```
./release.sh changes previous latest
```
and the changes between the latest stable and the previous release with:
```
./release.sh changes stable previous
```
1. **[STABLE]** Coordinate with product (& marketing) for the relevant public
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
announcements (public Slack, Twitter, etc.).
1. **[STABLE]** Documentation is published automatically once the release is
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
public on Github, however it runs on an hourly job and takes about 20
minutes to complete, so it could take up to an hour and a half depending on
when the prerelease tag was removed.
Thanks for making a release!