daml/release.sh

163 lines
5.0 KiB
Bash
Raw Normal View History

introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
#!/usr/bin/env bash
# Copyright (c) 2020 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
# SPDX-License-Identifier: Apache-2.0
set -euo pipefail
trigger all releases from master (#6016) trigger all releases from master The 1.1.0 release went wrong and we had to trash it and release 1.1.1 instead. This is an attempt at identifying and correcting the root cause behind that incident. To understand the situation, we need to know how releases worked before 1.0. We had a one-line file called `LATEST` that specifies the git SHA and version tag for the latest release. A change to that file triggered a release with the specified release tag, built from the source tree of the specified commit. The `LATEST` file looked something like: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0-snapshot.20200411.3905.0.f050da78 ``` To mark a release as stable, we would change it to look like this: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0 ``` i.e. simply drop the `-snapshot...` suffix. Even though the commit (and thus the entire source tree we build from) is the same, we would need to rebuild almost all of our release artifacts, as they embed the version tag in various places and ways. That worked well as long as we could assume we were doing trunk-based development, i.e. all releases would always come from the same (`master`) branch. When we released 1.0, and started work on 1.1, we had a few bug reports for 1.0 that we decided should be resolved in a point release. We decided that the best way to handle that would be to have a branch starting on the release commit for 1.0, and then backport patches from `master` to that branch. We adapted our build process to also watch the `release/1.0.x` branch and, in particular, trigger a new release build if the `LATEST` file in that branch changed. That worked well. The plan going forward was to keep doing regular snapshot releases from the `master` branch, and create support, point releases ("patch" releases in semver) from dedicated branches. On April 30, we made a snapshot release as an RC for 1.1.0, by changing the `LATEST` file in the `master` branch. That release was built on commit 681c862d. On May 6, we decided to take a new snapshot as the RC for 1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new latest release. On May 11, we noticed an issue that broke our builds. Without going into details, an external artifact we depend on had changed in incompatible ways. After fixing that on `master`, we reasoned that this would also break the build of the final 1.1.0 release if we just tried to build 7e448d81 again. But as the target release date was May 13, we did not want to take a new snapshot after that fix, as that would have included one more week of work in the release, and given us no time to test it. So we did what we did for the 1.0 branch, as it had worked well: we created a branch that forked from `master` at commit 7e448d81 and called it `release/1.1.x`, then cherry-picked the one fix to our build process to work around the broken download. When the time came to make the final 1.1.0 build on May 13, we naturally picked the `LATEST` file from the `release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly, we did not need to update the target commit to include the "broken download" fix as, in the meantime, the internet had fixed itself, and we thus reasoned we should go for the exact code of the RC rather than include an unnecessary, albeit seemingly harmless, change. Everything went well with the release process. Tests went well too. Then we got a report that an application that worked against the latest RC broke with the final 1.1.0. The issue was that we had built the wrong commit: by branching off at the point of the _target_ commit for the latest snapshot, we did not have the change to the `LATEST` file that designated that commit as the target. So the `LATEST` file in `release/1.1.x` was still pointing to 681c862d. I believe the root cause for this issue is the fact that we have scattered our release process over multiple branches, meaning there is no linear history of what was released and we are relying on people being able to mentally manage multiple timelines. Therefore, I propose to fix our release process so this should not happen again by linearizing the release process, i.e. getting back to a situation where all releases are made from a single branch, `master`. Because we do want to be able to release _for_ multiple release branches (to provide backports and bugfixes), we still need some way to accommodate that. Having a single `LATEST` file in the same format as before would not really work well: keeping track of interleaved release streams on a single file would not really be easier than keeping track of multiple branches. My proposed solution is to instead have a multiline LATEST file, so that all the release branch "tips" can be observed at the same time, and, as long as we take care to only advance one release branch at a time, we can easily keep track of each of them. This is what this PR does. This required a few changes to our release process. Most notably: - Obviously, as this is the main point of this PR, the build process has once again been restricted to only trigger new releases from the `master` branch. - As our CI machinery cannot easily be made to produce multiple releases from a single build, the `check_for_release` step will only recognize a commit as a release trigger if it changes a single line in the `LATEST` file. This restriction comes in addition to the existing one that a release commit is only allowed to change either just the `LATEST` file or both the `LATEST` and `docs/source/support/release-notes.rst` files. - The docs publication process has been changed to update _all_ published versions to display the _latest_ release notes page. This means that the release notes page will always show you all published versions, regardless of which version of the documentation you're looking at. This also means that interleaving release notes correctly on that page is a manual exercise. - As per the intention of the new process, the `LATEST` file has been updated to contained all existing post-1.0 stable releases. It should also include all existing snapshot releases should we have more than one at a time (say, should we discover an issue with 1.1.1 that required us to work on a 1.1.2). - The `release.sh` script has been dramatically simplified as I felt it was trying to do too much and porting its existing functionality to a multi-line `LATEST` file would be too hard. CHANGELOG_BEGIN CHANGELOG_END
2020-05-19 20:18:10 +03:00
DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
uhoh() {
echo "
It looks like this script failed to complete. Please check the status
of the LATEST file and consider running this script again."
}
trap uhoh EXIT
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
STABLE_REGEX="\d+\.\d+\.\d+"
VERSION_REGEX="^${STABLE_REGEX}(-snapshot\.\d{8}\.\d+(\.\d+)?\.[0-9a-f]{8})?$"
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
trigger all releases from master (#6016) trigger all releases from master The 1.1.0 release went wrong and we had to trash it and release 1.1.1 instead. This is an attempt at identifying and correcting the root cause behind that incident. To understand the situation, we need to know how releases worked before 1.0. We had a one-line file called `LATEST` that specifies the git SHA and version tag for the latest release. A change to that file triggered a release with the specified release tag, built from the source tree of the specified commit. The `LATEST` file looked something like: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0-snapshot.20200411.3905.0.f050da78 ``` To mark a release as stable, we would change it to look like this: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0 ``` i.e. simply drop the `-snapshot...` suffix. Even though the commit (and thus the entire source tree we build from) is the same, we would need to rebuild almost all of our release artifacts, as they embed the version tag in various places and ways. That worked well as long as we could assume we were doing trunk-based development, i.e. all releases would always come from the same (`master`) branch. When we released 1.0, and started work on 1.1, we had a few bug reports for 1.0 that we decided should be resolved in a point release. We decided that the best way to handle that would be to have a branch starting on the release commit for 1.0, and then backport patches from `master` to that branch. We adapted our build process to also watch the `release/1.0.x` branch and, in particular, trigger a new release build if the `LATEST` file in that branch changed. That worked well. The plan going forward was to keep doing regular snapshot releases from the `master` branch, and create support, point releases ("patch" releases in semver) from dedicated branches. On April 30, we made a snapshot release as an RC for 1.1.0, by changing the `LATEST` file in the `master` branch. That release was built on commit 681c862d. On May 6, we decided to take a new snapshot as the RC for 1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new latest release. On May 11, we noticed an issue that broke our builds. Without going into details, an external artifact we depend on had changed in incompatible ways. After fixing that on `master`, we reasoned that this would also break the build of the final 1.1.0 release if we just tried to build 7e448d81 again. But as the target release date was May 13, we did not want to take a new snapshot after that fix, as that would have included one more week of work in the release, and given us no time to test it. So we did what we did for the 1.0 branch, as it had worked well: we created a branch that forked from `master` at commit 7e448d81 and called it `release/1.1.x`, then cherry-picked the one fix to our build process to work around the broken download. When the time came to make the final 1.1.0 build on May 13, we naturally picked the `LATEST` file from the `release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly, we did not need to update the target commit to include the "broken download" fix as, in the meantime, the internet had fixed itself, and we thus reasoned we should go for the exact code of the RC rather than include an unnecessary, albeit seemingly harmless, change. Everything went well with the release process. Tests went well too. Then we got a report that an application that worked against the latest RC broke with the final 1.1.0. The issue was that we had built the wrong commit: by branching off at the point of the _target_ commit for the latest snapshot, we did not have the change to the `LATEST` file that designated that commit as the target. So the `LATEST` file in `release/1.1.x` was still pointing to 681c862d. I believe the root cause for this issue is the fact that we have scattered our release process over multiple branches, meaning there is no linear history of what was released and we are relying on people being able to mentally manage multiple timelines. Therefore, I propose to fix our release process so this should not happen again by linearizing the release process, i.e. getting back to a situation where all releases are made from a single branch, `master`. Because we do want to be able to release _for_ multiple release branches (to provide backports and bugfixes), we still need some way to accommodate that. Having a single `LATEST` file in the same format as before would not really work well: keeping track of interleaved release streams on a single file would not really be easier than keeping track of multiple branches. My proposed solution is to instead have a multiline LATEST file, so that all the release branch "tips" can be observed at the same time, and, as long as we take care to only advance one release branch at a time, we can easily keep track of each of them. This is what this PR does. This required a few changes to our release process. Most notably: - Obviously, as this is the main point of this PR, the build process has once again been restricted to only trigger new releases from the `master` branch. - As our CI machinery cannot easily be made to produce multiple releases from a single build, the `check_for_release` step will only recognize a commit as a release trigger if it changes a single line in the `LATEST` file. This restriction comes in addition to the existing one that a release commit is only allowed to change either just the `LATEST` file or both the `LATEST` and `docs/source/support/release-notes.rst` files. - The docs publication process has been changed to update _all_ published versions to display the _latest_ release notes page. This means that the release notes page will always show you all published versions, regardless of which version of the documentation you're looking at. This also means that interleaving release notes correctly on that page is a manual exercise. - As per the intention of the new process, the `LATEST` file has been updated to contained all existing post-1.0 stable releases. It should also include all existing snapshot releases should we have more than one at a time (say, should we discover an issue with 1.1.1 that required us to work on a 1.1.2). - The `release.sh` script has been dramatically simplified as I felt it was trying to do too much and porting its existing functionality to a multi-line `LATEST` file would be too hard. CHANGELOG_BEGIN CHANGELOG_END
2020-05-19 20:18:10 +03:00
function file_ends_with_newline() {
[[ $(tail -c1 "$1" | wc -l) -gt 0 ]]
}
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
check() {
local sha ver ver_sha
trigger all releases from master (#6016) trigger all releases from master The 1.1.0 release went wrong and we had to trash it and release 1.1.1 instead. This is an attempt at identifying and correcting the root cause behind that incident. To understand the situation, we need to know how releases worked before 1.0. We had a one-line file called `LATEST` that specifies the git SHA and version tag for the latest release. A change to that file triggered a release with the specified release tag, built from the source tree of the specified commit. The `LATEST` file looked something like: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0-snapshot.20200411.3905.0.f050da78 ``` To mark a release as stable, we would change it to look like this: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0 ``` i.e. simply drop the `-snapshot...` suffix. Even though the commit (and thus the entire source tree we build from) is the same, we would need to rebuild almost all of our release artifacts, as they embed the version tag in various places and ways. That worked well as long as we could assume we were doing trunk-based development, i.e. all releases would always come from the same (`master`) branch. When we released 1.0, and started work on 1.1, we had a few bug reports for 1.0 that we decided should be resolved in a point release. We decided that the best way to handle that would be to have a branch starting on the release commit for 1.0, and then backport patches from `master` to that branch. We adapted our build process to also watch the `release/1.0.x` branch and, in particular, trigger a new release build if the `LATEST` file in that branch changed. That worked well. The plan going forward was to keep doing regular snapshot releases from the `master` branch, and create support, point releases ("patch" releases in semver) from dedicated branches. On April 30, we made a snapshot release as an RC for 1.1.0, by changing the `LATEST` file in the `master` branch. That release was built on commit 681c862d. On May 6, we decided to take a new snapshot as the RC for 1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new latest release. On May 11, we noticed an issue that broke our builds. Without going into details, an external artifact we depend on had changed in incompatible ways. After fixing that on `master`, we reasoned that this would also break the build of the final 1.1.0 release if we just tried to build 7e448d81 again. But as the target release date was May 13, we did not want to take a new snapshot after that fix, as that would have included one more week of work in the release, and given us no time to test it. So we did what we did for the 1.0 branch, as it had worked well: we created a branch that forked from `master` at commit 7e448d81 and called it `release/1.1.x`, then cherry-picked the one fix to our build process to work around the broken download. When the time came to make the final 1.1.0 build on May 13, we naturally picked the `LATEST` file from the `release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly, we did not need to update the target commit to include the "broken download" fix as, in the meantime, the internet had fixed itself, and we thus reasoned we should go for the exact code of the RC rather than include an unnecessary, albeit seemingly harmless, change. Everything went well with the release process. Tests went well too. Then we got a report that an application that worked against the latest RC broke with the final 1.1.0. The issue was that we had built the wrong commit: by branching off at the point of the _target_ commit for the latest snapshot, we did not have the change to the `LATEST` file that designated that commit as the target. So the `LATEST` file in `release/1.1.x` was still pointing to 681c862d. I believe the root cause for this issue is the fact that we have scattered our release process over multiple branches, meaning there is no linear history of what was released and we are relying on people being able to mentally manage multiple timelines. Therefore, I propose to fix our release process so this should not happen again by linearizing the release process, i.e. getting back to a situation where all releases are made from a single branch, `master`. Because we do want to be able to release _for_ multiple release branches (to provide backports and bugfixes), we still need some way to accommodate that. Having a single `LATEST` file in the same format as before would not really work well: keeping track of interleaved release streams on a single file would not really be easier than keeping track of multiple branches. My proposed solution is to instead have a multiline LATEST file, so that all the release branch "tips" can be observed at the same time, and, as long as we take care to only advance one release branch at a time, we can easily keep track of each of them. This is what this PR does. This required a few changes to our release process. Most notably: - Obviously, as this is the main point of this PR, the build process has once again been restricted to only trigger new releases from the `master` branch. - As our CI machinery cannot easily be made to produce multiple releases from a single build, the `check_for_release` step will only recognize a commit as a release trigger if it changes a single line in the `LATEST` file. This restriction comes in addition to the existing one that a release commit is only allowed to change either just the `LATEST` file or both the `LATEST` and `docs/source/support/release-notes.rst` files. - The docs publication process has been changed to update _all_ published versions to display the _latest_ release notes page. This means that the release notes page will always show you all published versions, regardless of which version of the documentation you're looking at. This also means that interleaving release notes correctly on that page is a manual exercise. - As per the intention of the new process, the `LATEST` file has been updated to contained all existing post-1.0 stable releases. It should also include all existing snapshot releases should we have more than one at a time (say, should we discover an issue with 1.1.1 that required us to work on a 1.1.2). - The `release.sh` script has been dramatically simplified as I felt it was trying to do too much and porting its existing functionality to a multi-line `LATEST` file would be too hard. CHANGELOG_BEGIN CHANGELOG_END
2020-05-19 20:18:10 +03:00
if ! file_ends_with_newline LATEST; then
echo "LATEST file does not end with newline. Please correct."
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
exit 1
fi
trigger all releases from master (#6016) trigger all releases from master The 1.1.0 release went wrong and we had to trash it and release 1.1.1 instead. This is an attempt at identifying and correcting the root cause behind that incident. To understand the situation, we need to know how releases worked before 1.0. We had a one-line file called `LATEST` that specifies the git SHA and version tag for the latest release. A change to that file triggered a release with the specified release tag, built from the source tree of the specified commit. The `LATEST` file looked something like: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0-snapshot.20200411.3905.0.f050da78 ``` To mark a release as stable, we would change it to look like this: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0 ``` i.e. simply drop the `-snapshot...` suffix. Even though the commit (and thus the entire source tree we build from) is the same, we would need to rebuild almost all of our release artifacts, as they embed the version tag in various places and ways. That worked well as long as we could assume we were doing trunk-based development, i.e. all releases would always come from the same (`master`) branch. When we released 1.0, and started work on 1.1, we had a few bug reports for 1.0 that we decided should be resolved in a point release. We decided that the best way to handle that would be to have a branch starting on the release commit for 1.0, and then backport patches from `master` to that branch. We adapted our build process to also watch the `release/1.0.x` branch and, in particular, trigger a new release build if the `LATEST` file in that branch changed. That worked well. The plan going forward was to keep doing regular snapshot releases from the `master` branch, and create support, point releases ("patch" releases in semver) from dedicated branches. On April 30, we made a snapshot release as an RC for 1.1.0, by changing the `LATEST` file in the `master` branch. That release was built on commit 681c862d. On May 6, we decided to take a new snapshot as the RC for 1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new latest release. On May 11, we noticed an issue that broke our builds. Without going into details, an external artifact we depend on had changed in incompatible ways. After fixing that on `master`, we reasoned that this would also break the build of the final 1.1.0 release if we just tried to build 7e448d81 again. But as the target release date was May 13, we did not want to take a new snapshot after that fix, as that would have included one more week of work in the release, and given us no time to test it. So we did what we did for the 1.0 branch, as it had worked well: we created a branch that forked from `master` at commit 7e448d81 and called it `release/1.1.x`, then cherry-picked the one fix to our build process to work around the broken download. When the time came to make the final 1.1.0 build on May 13, we naturally picked the `LATEST` file from the `release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly, we did not need to update the target commit to include the "broken download" fix as, in the meantime, the internet had fixed itself, and we thus reasoned we should go for the exact code of the RC rather than include an unnecessary, albeit seemingly harmless, change. Everything went well with the release process. Tests went well too. Then we got a report that an application that worked against the latest RC broke with the final 1.1.0. The issue was that we had built the wrong commit: by branching off at the point of the _target_ commit for the latest snapshot, we did not have the change to the `LATEST` file that designated that commit as the target. So the `LATEST` file in `release/1.1.x` was still pointing to 681c862d. I believe the root cause for this issue is the fact that we have scattered our release process over multiple branches, meaning there is no linear history of what was released and we are relying on people being able to mentally manage multiple timelines. Therefore, I propose to fix our release process so this should not happen again by linearizing the release process, i.e. getting back to a situation where all releases are made from a single branch, `master`. Because we do want to be able to release _for_ multiple release branches (to provide backports and bugfixes), we still need some way to accommodate that. Having a single `LATEST` file in the same format as before would not really work well: keeping track of interleaved release streams on a single file would not really be easier than keeping track of multiple branches. My proposed solution is to instead have a multiline LATEST file, so that all the release branch "tips" can be observed at the same time, and, as long as we take care to only advance one release branch at a time, we can easily keep track of each of them. This is what this PR does. This required a few changes to our release process. Most notably: - Obviously, as this is the main point of this PR, the build process has once again been restricted to only trigger new releases from the `master` branch. - As our CI machinery cannot easily be made to produce multiple releases from a single build, the `check_for_release` step will only recognize a commit as a release trigger if it changes a single line in the `LATEST` file. This restriction comes in addition to the existing one that a release commit is only allowed to change either just the `LATEST` file or both the `LATEST` and `docs/source/support/release-notes.rst` files. - The docs publication process has been changed to update _all_ published versions to display the _latest_ release notes page. This means that the release notes page will always show you all published versions, regardless of which version of the documentation you're looking at. This also means that interleaving release notes correctly on that page is a manual exercise. - As per the intention of the new process, the `LATEST` file has been updated to contained all existing post-1.0 stable releases. It should also include all existing snapshot releases should we have more than one at a time (say, should we discover an issue with 1.1.1 that required us to work on a 1.1.2). - The `release.sh` script has been dramatically simplified as I felt it was trying to do too much and porting its existing functionality to a multi-line `LATEST` file would be too hard. CHANGELOG_BEGIN CHANGELOG_END
2020-05-19 20:18:10 +03:00
while read line; do
sha=$(echo "$line" | gawk '{print $1}')
ver=$(echo "$line" | gawk '{print $2}')
if ! echo "$ver" | grep -q -P $VERSION_REGEX; then
echo "Invalid version number in LATEST file, needs manual correction."
echo "Offending version: '$ver'."
trigger all releases from master (#6016) trigger all releases from master The 1.1.0 release went wrong and we had to trash it and release 1.1.1 instead. This is an attempt at identifying and correcting the root cause behind that incident. To understand the situation, we need to know how releases worked before 1.0. We had a one-line file called `LATEST` that specifies the git SHA and version tag for the latest release. A change to that file triggered a release with the specified release tag, built from the source tree of the specified commit. The `LATEST` file looked something like: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0-snapshot.20200411.3905.0.f050da78 ``` To mark a release as stable, we would change it to look like this: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0 ``` i.e. simply drop the `-snapshot...` suffix. Even though the commit (and thus the entire source tree we build from) is the same, we would need to rebuild almost all of our release artifacts, as they embed the version tag in various places and ways. That worked well as long as we could assume we were doing trunk-based development, i.e. all releases would always come from the same (`master`) branch. When we released 1.0, and started work on 1.1, we had a few bug reports for 1.0 that we decided should be resolved in a point release. We decided that the best way to handle that would be to have a branch starting on the release commit for 1.0, and then backport patches from `master` to that branch. We adapted our build process to also watch the `release/1.0.x` branch and, in particular, trigger a new release build if the `LATEST` file in that branch changed. That worked well. The plan going forward was to keep doing regular snapshot releases from the `master` branch, and create support, point releases ("patch" releases in semver) from dedicated branches. On April 30, we made a snapshot release as an RC for 1.1.0, by changing the `LATEST` file in the `master` branch. That release was built on commit 681c862d. On May 6, we decided to take a new snapshot as the RC for 1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new latest release. On May 11, we noticed an issue that broke our builds. Without going into details, an external artifact we depend on had changed in incompatible ways. After fixing that on `master`, we reasoned that this would also break the build of the final 1.1.0 release if we just tried to build 7e448d81 again. But as the target release date was May 13, we did not want to take a new snapshot after that fix, as that would have included one more week of work in the release, and given us no time to test it. So we did what we did for the 1.0 branch, as it had worked well: we created a branch that forked from `master` at commit 7e448d81 and called it `release/1.1.x`, then cherry-picked the one fix to our build process to work around the broken download. When the time came to make the final 1.1.0 build on May 13, we naturally picked the `LATEST` file from the `release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly, we did not need to update the target commit to include the "broken download" fix as, in the meantime, the internet had fixed itself, and we thus reasoned we should go for the exact code of the RC rather than include an unnecessary, albeit seemingly harmless, change. Everything went well with the release process. Tests went well too. Then we got a report that an application that worked against the latest RC broke with the final 1.1.0. The issue was that we had built the wrong commit: by branching off at the point of the _target_ commit for the latest snapshot, we did not have the change to the `LATEST` file that designated that commit as the target. So the `LATEST` file in `release/1.1.x` was still pointing to 681c862d. I believe the root cause for this issue is the fact that we have scattered our release process over multiple branches, meaning there is no linear history of what was released and we are relying on people being able to mentally manage multiple timelines. Therefore, I propose to fix our release process so this should not happen again by linearizing the release process, i.e. getting back to a situation where all releases are made from a single branch, `master`. Because we do want to be able to release _for_ multiple release branches (to provide backports and bugfixes), we still need some way to accommodate that. Having a single `LATEST` file in the same format as before would not really work well: keeping track of interleaved release streams on a single file would not really be easier than keeping track of multiple branches. My proposed solution is to instead have a multiline LATEST file, so that all the release branch "tips" can be observed at the same time, and, as long as we take care to only advance one release branch at a time, we can easily keep track of each of them. This is what this PR does. This required a few changes to our release process. Most notably: - Obviously, as this is the main point of this PR, the build process has once again been restricted to only trigger new releases from the `master` branch. - As our CI machinery cannot easily be made to produce multiple releases from a single build, the `check_for_release` step will only recognize a commit as a release trigger if it changes a single line in the `LATEST` file. This restriction comes in addition to the existing one that a release commit is only allowed to change either just the `LATEST` file or both the `LATEST` and `docs/source/support/release-notes.rst` files. - The docs publication process has been changed to update _all_ published versions to display the _latest_ release notes page. This means that the release notes page will always show you all published versions, regardless of which version of the documentation you're looking at. This also means that interleaving release notes correctly on that page is a manual exercise. - As per the intention of the new process, the `LATEST` file has been updated to contained all existing post-1.0 stable releases. It should also include all existing snapshot releases should we have more than one at a time (say, should we discover an issue with 1.1.1 that required us to work on a 1.1.2). - The `release.sh` script has been dramatically simplified as I felt it was trying to do too much and porting its existing functionality to a multi-line `LATEST` file would be too hard. CHANGELOG_BEGIN CHANGELOG_END
2020-05-19 20:18:10 +03:00
exit 1
fi
if ! is_stable $ver; then
ver_sha=$(echo $ver | sed 's/.*\.//')
if ! [ "${sha:0:8}" = "$ver_sha" ]; then
echo "$ver does not match $sha, please correct. ($ver_sha != ${sha:0:8})"
trigger all releases from master (#6016) trigger all releases from master The 1.1.0 release went wrong and we had to trash it and release 1.1.1 instead. This is an attempt at identifying and correcting the root cause behind that incident. To understand the situation, we need to know how releases worked before 1.0. We had a one-line file called `LATEST` that specifies the git SHA and version tag for the latest release. A change to that file triggered a release with the specified release tag, built from the source tree of the specified commit. The `LATEST` file looked something like: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0-snapshot.20200411.3905.0.f050da78 ``` To mark a release as stable, we would change it to look like this: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0 ``` i.e. simply drop the `-snapshot...` suffix. Even though the commit (and thus the entire source tree we build from) is the same, we would need to rebuild almost all of our release artifacts, as they embed the version tag in various places and ways. That worked well as long as we could assume we were doing trunk-based development, i.e. all releases would always come from the same (`master`) branch. When we released 1.0, and started work on 1.1, we had a few bug reports for 1.0 that we decided should be resolved in a point release. We decided that the best way to handle that would be to have a branch starting on the release commit for 1.0, and then backport patches from `master` to that branch. We adapted our build process to also watch the `release/1.0.x` branch and, in particular, trigger a new release build if the `LATEST` file in that branch changed. That worked well. The plan going forward was to keep doing regular snapshot releases from the `master` branch, and create support, point releases ("patch" releases in semver) from dedicated branches. On April 30, we made a snapshot release as an RC for 1.1.0, by changing the `LATEST` file in the `master` branch. That release was built on commit 681c862d. On May 6, we decided to take a new snapshot as the RC for 1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new latest release. On May 11, we noticed an issue that broke our builds. Without going into details, an external artifact we depend on had changed in incompatible ways. After fixing that on `master`, we reasoned that this would also break the build of the final 1.1.0 release if we just tried to build 7e448d81 again. But as the target release date was May 13, we did not want to take a new snapshot after that fix, as that would have included one more week of work in the release, and given us no time to test it. So we did what we did for the 1.0 branch, as it had worked well: we created a branch that forked from `master` at commit 7e448d81 and called it `release/1.1.x`, then cherry-picked the one fix to our build process to work around the broken download. When the time came to make the final 1.1.0 build on May 13, we naturally picked the `LATEST` file from the `release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly, we did not need to update the target commit to include the "broken download" fix as, in the meantime, the internet had fixed itself, and we thus reasoned we should go for the exact code of the RC rather than include an unnecessary, albeit seemingly harmless, change. Everything went well with the release process. Tests went well too. Then we got a report that an application that worked against the latest RC broke with the final 1.1.0. The issue was that we had built the wrong commit: by branching off at the point of the _target_ commit for the latest snapshot, we did not have the change to the `LATEST` file that designated that commit as the target. So the `LATEST` file in `release/1.1.x` was still pointing to 681c862d. I believe the root cause for this issue is the fact that we have scattered our release process over multiple branches, meaning there is no linear history of what was released and we are relying on people being able to mentally manage multiple timelines. Therefore, I propose to fix our release process so this should not happen again by linearizing the release process, i.e. getting back to a situation where all releases are made from a single branch, `master`. Because we do want to be able to release _for_ multiple release branches (to provide backports and bugfixes), we still need some way to accommodate that. Having a single `LATEST` file in the same format as before would not really work well: keeping track of interleaved release streams on a single file would not really be easier than keeping track of multiple branches. My proposed solution is to instead have a multiline LATEST file, so that all the release branch "tips" can be observed at the same time, and, as long as we take care to only advance one release branch at a time, we can easily keep track of each of them. This is what this PR does. This required a few changes to our release process. Most notably: - Obviously, as this is the main point of this PR, the build process has once again been restricted to only trigger new releases from the `master` branch. - As our CI machinery cannot easily be made to produce multiple releases from a single build, the `check_for_release` step will only recognize a commit as a release trigger if it changes a single line in the `LATEST` file. This restriction comes in addition to the existing one that a release commit is only allowed to change either just the `LATEST` file or both the `LATEST` and `docs/source/support/release-notes.rst` files. - The docs publication process has been changed to update _all_ published versions to display the _latest_ release notes page. This means that the release notes page will always show you all published versions, regardless of which version of the documentation you're looking at. This also means that interleaving release notes correctly on that page is a manual exercise. - As per the intention of the new process, the `LATEST` file has been updated to contained all existing post-1.0 stable releases. It should also include all existing snapshot releases should we have more than one at a time (say, should we discover an issue with 1.1.1 that required us to work on a 1.1.2). - The `release.sh` script has been dramatically simplified as I felt it was trying to do too much and porting its existing functionality to a multi-line `LATEST` file would be too hard. CHANGELOG_BEGIN CHANGELOG_END
2020-05-19 20:18:10 +03:00
exit 1
fi
fi
done < LATEST
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
}
is_stable() {
local version="$1"
echo "$version" | grep -q -P "^${STABLE_REGEX}$"
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
}
make_snapshot() {
local sha prefix commit_date number_of_commits commit_sha_8
sha=$1
prefix=$2
commit_date=$(git log -n1 --format=%cd --date=format:%Y%m%d $sha)
number_of_commits=$(git rev-list --count $sha)
commit_sha_8=$(git log -n1 --format=%h --abbrev=8 $sha)
echo "$1 $2-snapshot.$commit_date.$number_of_commits.0.$commit_sha_8"
}
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
display_help() {
cat <<EOF
This script is meant to help with managing releases. Usage:
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
$0 snapshot SHA PREFIX
Prints the snapshot line for commit SHA as a release candidate for
version PREFIX. For example:
trigger all releases from master (#6016) trigger all releases from master The 1.1.0 release went wrong and we had to trash it and release 1.1.1 instead. This is an attempt at identifying and correcting the root cause behind that incident. To understand the situation, we need to know how releases worked before 1.0. We had a one-line file called `LATEST` that specifies the git SHA and version tag for the latest release. A change to that file triggered a release with the specified release tag, built from the source tree of the specified commit. The `LATEST` file looked something like: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0-snapshot.20200411.3905.0.f050da78 ``` To mark a release as stable, we would change it to look like this: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0 ``` i.e. simply drop the `-snapshot...` suffix. Even though the commit (and thus the entire source tree we build from) is the same, we would need to rebuild almost all of our release artifacts, as they embed the version tag in various places and ways. That worked well as long as we could assume we were doing trunk-based development, i.e. all releases would always come from the same (`master`) branch. When we released 1.0, and started work on 1.1, we had a few bug reports for 1.0 that we decided should be resolved in a point release. We decided that the best way to handle that would be to have a branch starting on the release commit for 1.0, and then backport patches from `master` to that branch. We adapted our build process to also watch the `release/1.0.x` branch and, in particular, trigger a new release build if the `LATEST` file in that branch changed. That worked well. The plan going forward was to keep doing regular snapshot releases from the `master` branch, and create support, point releases ("patch" releases in semver) from dedicated branches. On April 30, we made a snapshot release as an RC for 1.1.0, by changing the `LATEST` file in the `master` branch. That release was built on commit 681c862d. On May 6, we decided to take a new snapshot as the RC for 1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new latest release. On May 11, we noticed an issue that broke our builds. Without going into details, an external artifact we depend on had changed in incompatible ways. After fixing that on `master`, we reasoned that this would also break the build of the final 1.1.0 release if we just tried to build 7e448d81 again. But as the target release date was May 13, we did not want to take a new snapshot after that fix, as that would have included one more week of work in the release, and given us no time to test it. So we did what we did for the 1.0 branch, as it had worked well: we created a branch that forked from `master` at commit 7e448d81 and called it `release/1.1.x`, then cherry-picked the one fix to our build process to work around the broken download. When the time came to make the final 1.1.0 build on May 13, we naturally picked the `LATEST` file from the `release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly, we did not need to update the target commit to include the "broken download" fix as, in the meantime, the internet had fixed itself, and we thus reasoned we should go for the exact code of the RC rather than include an unnecessary, albeit seemingly harmless, change. Everything went well with the release process. Tests went well too. Then we got a report that an application that worked against the latest RC broke with the final 1.1.0. The issue was that we had built the wrong commit: by branching off at the point of the _target_ commit for the latest snapshot, we did not have the change to the `LATEST` file that designated that commit as the target. So the `LATEST` file in `release/1.1.x` was still pointing to 681c862d. I believe the root cause for this issue is the fact that we have scattered our release process over multiple branches, meaning there is no linear history of what was released and we are relying on people being able to mentally manage multiple timelines. Therefore, I propose to fix our release process so this should not happen again by linearizing the release process, i.e. getting back to a situation where all releases are made from a single branch, `master`. Because we do want to be able to release _for_ multiple release branches (to provide backports and bugfixes), we still need some way to accommodate that. Having a single `LATEST` file in the same format as before would not really work well: keeping track of interleaved release streams on a single file would not really be easier than keeping track of multiple branches. My proposed solution is to instead have a multiline LATEST file, so that all the release branch "tips" can be observed at the same time, and, as long as we take care to only advance one release branch at a time, we can easily keep track of each of them. This is what this PR does. This required a few changes to our release process. Most notably: - Obviously, as this is the main point of this PR, the build process has once again been restricted to only trigger new releases from the `master` branch. - As our CI machinery cannot easily be made to produce multiple releases from a single build, the `check_for_release` step will only recognize a commit as a release trigger if it changes a single line in the `LATEST` file. This restriction comes in addition to the existing one that a release commit is only allowed to change either just the `LATEST` file or both the `LATEST` and `docs/source/support/release-notes.rst` files. - The docs publication process has been changed to update _all_ published versions to display the _latest_ release notes page. This means that the release notes page will always show you all published versions, regardless of which version of the documentation you're looking at. This also means that interleaving release notes correctly on that page is a manual exercise. - As per the intention of the new process, the `LATEST` file has been updated to contained all existing post-1.0 stable releases. It should also include all existing snapshot releases should we have more than one at a time (say, should we discover an issue with 1.1.1 that required us to work on a 1.1.2). - The `release.sh` script has been dramatically simplified as I felt it was trying to do too much and porting its existing functionality to a multi-line `LATEST` file would be too hard. CHANGELOG_BEGIN CHANGELOG_END
2020-05-19 20:18:10 +03:00
$ $0 snapshot cc880e2 0.1.2
cc880e290b2311d0bf05d58c7d75c50784c0131c 0.1.2-snapshot.20200513.4174.0.cc880e29
trigger all releases from master (#6016) trigger all releases from master The 1.1.0 release went wrong and we had to trash it and release 1.1.1 instead. This is an attempt at identifying and correcting the root cause behind that incident. To understand the situation, we need to know how releases worked before 1.0. We had a one-line file called `LATEST` that specifies the git SHA and version tag for the latest release. A change to that file triggered a release with the specified release tag, built from the source tree of the specified commit. The `LATEST` file looked something like: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0-snapshot.20200411.3905.0.f050da78 ``` To mark a release as stable, we would change it to look like this: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0 ``` i.e. simply drop the `-snapshot...` suffix. Even though the commit (and thus the entire source tree we build from) is the same, we would need to rebuild almost all of our release artifacts, as they embed the version tag in various places and ways. That worked well as long as we could assume we were doing trunk-based development, i.e. all releases would always come from the same (`master`) branch. When we released 1.0, and started work on 1.1, we had a few bug reports for 1.0 that we decided should be resolved in a point release. We decided that the best way to handle that would be to have a branch starting on the release commit for 1.0, and then backport patches from `master` to that branch. We adapted our build process to also watch the `release/1.0.x` branch and, in particular, trigger a new release build if the `LATEST` file in that branch changed. That worked well. The plan going forward was to keep doing regular snapshot releases from the `master` branch, and create support, point releases ("patch" releases in semver) from dedicated branches. On April 30, we made a snapshot release as an RC for 1.1.0, by changing the `LATEST` file in the `master` branch. That release was built on commit 681c862d. On May 6, we decided to take a new snapshot as the RC for 1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new latest release. On May 11, we noticed an issue that broke our builds. Without going into details, an external artifact we depend on had changed in incompatible ways. After fixing that on `master`, we reasoned that this would also break the build of the final 1.1.0 release if we just tried to build 7e448d81 again. But as the target release date was May 13, we did not want to take a new snapshot after that fix, as that would have included one more week of work in the release, and given us no time to test it. So we did what we did for the 1.0 branch, as it had worked well: we created a branch that forked from `master` at commit 7e448d81 and called it `release/1.1.x`, then cherry-picked the one fix to our build process to work around the broken download. When the time came to make the final 1.1.0 build on May 13, we naturally picked the `LATEST` file from the `release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly, we did not need to update the target commit to include the "broken download" fix as, in the meantime, the internet had fixed itself, and we thus reasoned we should go for the exact code of the RC rather than include an unnecessary, albeit seemingly harmless, change. Everything went well with the release process. Tests went well too. Then we got a report that an application that worked against the latest RC broke with the final 1.1.0. The issue was that we had built the wrong commit: by branching off at the point of the _target_ commit for the latest snapshot, we did not have the change to the `LATEST` file that designated that commit as the target. So the `LATEST` file in `release/1.1.x` was still pointing to 681c862d. I believe the root cause for this issue is the fact that we have scattered our release process over multiple branches, meaning there is no linear history of what was released and we are relying on people being able to mentally manage multiple timelines. Therefore, I propose to fix our release process so this should not happen again by linearizing the release process, i.e. getting back to a situation where all releases are made from a single branch, `master`. Because we do want to be able to release _for_ multiple release branches (to provide backports and bugfixes), we still need some way to accommodate that. Having a single `LATEST` file in the same format as before would not really work well: keeping track of interleaved release streams on a single file would not really be easier than keeping track of multiple branches. My proposed solution is to instead have a multiline LATEST file, so that all the release branch "tips" can be observed at the same time, and, as long as we take care to only advance one release branch at a time, we can easily keep track of each of them. This is what this PR does. This required a few changes to our release process. Most notably: - Obviously, as this is the main point of this PR, the build process has once again been restricted to only trigger new releases from the `master` branch. - As our CI machinery cannot easily be made to produce multiple releases from a single build, the `check_for_release` step will only recognize a commit as a release trigger if it changes a single line in the `LATEST` file. This restriction comes in addition to the existing one that a release commit is only allowed to change either just the `LATEST` file or both the `LATEST` and `docs/source/support/release-notes.rst` files. - The docs publication process has been changed to update _all_ published versions to display the _latest_ release notes page. This means that the release notes page will always show you all published versions, regardless of which version of the documentation you're looking at. This also means that interleaving release notes correctly on that page is a manual exercise. - As per the intention of the new process, the `LATEST` file has been updated to contained all existing post-1.0 stable releases. It should also include all existing snapshot releases should we have more than one at a time (say, should we discover an issue with 1.1.1 that required us to work on a 1.1.2). - The `release.sh` script has been dramatically simplified as I felt it was trying to do too much and porting its existing functionality to a multi-line `LATEST` file would be too hard. CHANGELOG_BEGIN CHANGELOG_END
2020-05-19 20:18:10 +03:00
Any non-ambiguous git commit reference can be given as SHA.
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
$0 new snapshot
Updates LATEST to add current commit as a new snapshot. Figures out
prefix version by keeping the same if the first line in LATEST is a
snapshot, and incrementing minor if the first line is stable.
$0 check
trigger all releases from master (#6016) trigger all releases from master The 1.1.0 release went wrong and we had to trash it and release 1.1.1 instead. This is an attempt at identifying and correcting the root cause behind that incident. To understand the situation, we need to know how releases worked before 1.0. We had a one-line file called `LATEST` that specifies the git SHA and version tag for the latest release. A change to that file triggered a release with the specified release tag, built from the source tree of the specified commit. The `LATEST` file looked something like: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0-snapshot.20200411.3905.0.f050da78 ``` To mark a release as stable, we would change it to look like this: ``` f050da78c9c8727b889bdac286156f19e2f938c4 1.0.0 ``` i.e. simply drop the `-snapshot...` suffix. Even though the commit (and thus the entire source tree we build from) is the same, we would need to rebuild almost all of our release artifacts, as they embed the version tag in various places and ways. That worked well as long as we could assume we were doing trunk-based development, i.e. all releases would always come from the same (`master`) branch. When we released 1.0, and started work on 1.1, we had a few bug reports for 1.0 that we decided should be resolved in a point release. We decided that the best way to handle that would be to have a branch starting on the release commit for 1.0, and then backport patches from `master` to that branch. We adapted our build process to also watch the `release/1.0.x` branch and, in particular, trigger a new release build if the `LATEST` file in that branch changed. That worked well. The plan going forward was to keep doing regular snapshot releases from the `master` branch, and create support, point releases ("patch" releases in semver) from dedicated branches. On April 30, we made a snapshot release as an RC for 1.1.0, by changing the `LATEST` file in the `master` branch. That release was built on commit 681c862d. On May 6, we decided to take a new snapshot as the RC for 1.1.0; we changed `LATEST` in `master` to designate 7e448d81 as the new latest release. On May 11, we noticed an issue that broke our builds. Without going into details, an external artifact we depend on had changed in incompatible ways. After fixing that on `master`, we reasoned that this would also break the build of the final 1.1.0 release if we just tried to build 7e448d81 again. But as the target release date was May 13, we did not want to take a new snapshot after that fix, as that would have included one more week of work in the release, and given us no time to test it. So we did what we did for the 1.0 branch, as it had worked well: we created a branch that forked from `master` at commit 7e448d81 and called it `release/1.1.x`, then cherry-picked the one fix to our build process to work around the broken download. When the time came to make the final 1.1.0 build on May 13, we naturally picked the `LATEST` file from the `release/1.1.x` branch and dropped the `-snapshot...` suffix. Importantly, we did not need to update the target commit to include the "broken download" fix as, in the meantime, the internet had fixed itself, and we thus reasoned we should go for the exact code of the RC rather than include an unnecessary, albeit seemingly harmless, change. Everything went well with the release process. Tests went well too. Then we got a report that an application that worked against the latest RC broke with the final 1.1.0. The issue was that we had built the wrong commit: by branching off at the point of the _target_ commit for the latest snapshot, we did not have the change to the `LATEST` file that designated that commit as the target. So the `LATEST` file in `release/1.1.x` was still pointing to 681c862d. I believe the root cause for this issue is the fact that we have scattered our release process over multiple branches, meaning there is no linear history of what was released and we are relying on people being able to mentally manage multiple timelines. Therefore, I propose to fix our release process so this should not happen again by linearizing the release process, i.e. getting back to a situation where all releases are made from a single branch, `master`. Because we do want to be able to release _for_ multiple release branches (to provide backports and bugfixes), we still need some way to accommodate that. Having a single `LATEST` file in the same format as before would not really work well: keeping track of interleaved release streams on a single file would not really be easier than keeping track of multiple branches. My proposed solution is to instead have a multiline LATEST file, so that all the release branch "tips" can be observed at the same time, and, as long as we take care to only advance one release branch at a time, we can easily keep track of each of them. This is what this PR does. This required a few changes to our release process. Most notably: - Obviously, as this is the main point of this PR, the build process has once again been restricted to only trigger new releases from the `master` branch. - As our CI machinery cannot easily be made to produce multiple releases from a single build, the `check_for_release` step will only recognize a commit as a release trigger if it changes a single line in the `LATEST` file. This restriction comes in addition to the existing one that a release commit is only allowed to change either just the `LATEST` file or both the `LATEST` and `docs/source/support/release-notes.rst` files. - The docs publication process has been changed to update _all_ published versions to display the _latest_ release notes page. This means that the release notes page will always show you all published versions, regardless of which version of the documentation you're looking at. This also means that interleaving release notes correctly on that page is a manual exercise. - As per the intention of the new process, the `LATEST` file has been updated to contained all existing post-1.0 stable releases. It should also include all existing snapshot releases should we have more than one at a time (say, should we discover an issue with 1.1.1 that required us to work on a 1.1.2). - The `release.sh` script has been dramatically simplified as I felt it was trying to do too much and porting its existing functionality to a multi-line `LATEST` file would be too hard. CHANGELOG_BEGIN CHANGELOG_END
2020-05-19 20:18:10 +03:00
Checks that each line of the LATEST file is well-formed.
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
Any other invocation will display this help message.
EOF
}
if [ -z "${1+x}" ]; then
display_help
exit 1
fi
commit_belongs_to_release_branch() {
git branch --all --format='%(refname:short)' --contains="$1" \
| grep -q -E '^origin/(master$|release/)'
}
new_snapshot () {
local sha latest latest_prefix prefix new_line tmp
sha=$(git rev-parse HEAD)
if ! commit_belongs_to_release_branch $sha; then
echo "WARNING: Commit does not belong to a release branch."
fi
latest=$(head -1 LATEST | awk '{print $2}')
latest_prefix=$(echo $latest | grep -o -P "$STABLE_REGEX" | head -1)
if is_stable $latest; then
# As part of our normal processes, this case should never happen.
# Versions in the LATEST file are supposed to be ordered (highest
# version at the top), and we're supposed to start creating 1.5
# snapshots before we have fully settled on the 1.4 stable. So the
# normal case for the top two lines of the LATEST file is either two
# snapshots or one snapshot then one stable.
#
# Still, if it does encounter that case, the only sensible thing the
# script can do is bump the version. (Well, or exit 1, I guess.)
prefix=$(echo $latest_prefix | jq -Rr '. | split(".") | [.[0], (.[1] | tonumber + 1 | tostring), "0"] | join(".")')
else
prefix=$latest_prefix
fi
new_line=$(make_snapshot $sha $prefix)
tmp=$(mktemp)
cp LATEST $tmp
if is_stable $latest; then
# This case should not happen (see above), but if it does, we need to
# add the new snapshot while keeping the existing stable.
cat <(echo $new_line) $tmp > LATEST
else
# This is the only case consistent with all of our other processes
# working as expected: top of LATEST is a snapshot and we replace it
# with the new one.
cat <(echo $new_line) <(tail -n +2 $tmp) > LATEST
fi
}
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
case $1 in
snapshot)
if [ -n "${2+x}" ] && [ -n "${3+x}" ]; then
if ! commit_belongs_to_release_branch $2; then
echo "WARNING: Commit does not belong to a release branch."
fi
make_snapshot $(git rev-parse $2) $3
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
else
display_help
fi
;;
new)
if [ $# -eq 2 ] && [ "$2" == 'snapshot' ]; then
new_snapshot
else
display_help
fi
;;
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
check)
check
;;
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
*)
display_help
;;
introduce new release process (#4513) Context ======= After multiple discussions about our current release schedule and process, we've come to the conclusion that we need to be able to make a distinction between technical snapshots and marketing releases. In other words, we need to be able to create a bundle for early adopters to test without making it an officially-supported version, and without necessarily implying everyone should go through the trouble of upgrading. The underlying goal is to have less frequent but more stable "official" releases. This PR is a proposal for a new release process designed under the following constraints: - Reuse as much as possible of the existing infrastructure, to minimize effort but also chances of disruptions. - Have the ability to create "snapshot"/"nightly"/... releases that are not meant for general public consumption, but can still be used by savvy users without jumping through too many extra hoops (ideally just swapping in a slightly-weirder version string). - Have the ability to promote an existing snapshot release to "official" release status, with as few changes as possible in-between, so we can be confident that the official release is what we tested as a prerelease. - Have as much of the release pipeline shared between the two types of releases, to avoid discovering non-transient problems while trying to promote a snapshot to an official release. - Triggerring a release should still be done through a PR, so we can keep the same approval process for SOC2 auditability. The gist of this proposal is to replace the current `VERSION` file with a `LATEST` file, which would have the following format: ``` ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7 ``` This file would be maintained with a script to reduce manual labor in producing the version string. Other than that, the process will be largely the same, with releases triggered by changes to this `LATEST` and the release notes files. Version numbers =============== Because one of the goals is to reduce the velocity of our published version numbers, we need a different version scheme for our snapshot releases. Fortunately, most version schemes have some support for that; unfortunately, the SDK sits at the intersection of three different version schemes that have made incompatible choices. Without going into too much detail: - Semantic versioning (which we chose as the version format for the SDK version number) allows for "prerelease" version numbers as well as "metadata"; an example of a complete version string would be `1.2.3-nightly.201+server12.43`. The "main" part of the version string always has to have 3 numbers separated by dots; the "prerelease" (after the `-` but before the `+`) and the "metadata" (after the `+`) parts are optional and, if present, must consist of one or more segments separated by dots, where a segment can be either a number or an alphanumeric string. In terms of ordering, metadata is irrelevant and any version with a prerelease string is before the corresponding "main" version string alone. Amongst prereleases, segments are compared in order with purely numeric ones compared as numbers and mixed ones compared lexicographically. So 1.2.3 is more recent than 1.2.3-1, which is itself less recent than 1.2.3-2. - Maven version strings are any number of segments separated by a `.`, a `-`, or a transition between a number and a letter. Version strings are compared element-wise, with numeric segments being compared as numbers. Alphabetic segments are treated specially if they happen to be one of a handful of magic words (such as "alpha", "beta" or "snapshot" for example) which count as "qualifiers"; a version string with a qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`, which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a special ordering amongst qualifiers. Other alphabetic segments are compared alphabetically and count as being "after" their prefix (`1.2.3-really-final-this-time` counts as being released after `1.2.3`). - GHC package numbers are comprised of any number of numeric segments separated by `.`, plus an optional (though deprecated) alphanumeric "version tag" separated by a `-`. I could not find any official documentation on ordering for the version tag; numeric segments are compared as numbers. - npm uses semantic versioning so that is covered already. After much more investigation than I'd care to admit, I have come up with the following compromise as the least-bad solution. First, obviously, the version string for stable/marketing versions is going to be "standard" semver, i.e. major.minor.patch, all numbers, which works, and sorts as expected, for all three schemes. For snapshot releases, we shall use the following (semver) format: ``` 0.13.53-alpha.20200214.3025.ef5d32b7 ``` where the components are, respectively: - `0.13.53`: the expected version string of the next "stable" release. - `alpha`: a marker that hopefully scares people enough. - `20200214`: the date of the release commit, which _MUST_ be on master. - `3025`: the number of commits in master up to the release commit (included). Because we have a linear, append-only master branch, this uniquely identifies the commit. - `ef5d32b7ù : the first 8 characters of the release commit sha. This is not strictly speaking necessary, but makes it a lot more convenient to identify the commit. The main downsides of this format are: 1. It is not a valid format for GHC packages. We do not publish GHC packages from the SDK (so far we have instead opted to release our Haskell code as separate packages entirely), so this should not be an issue. However, our SDK version currently leaks to `ghc-pkg` as the version string for the stdlib (and prim) packages. This PR addresses that by tweaking the compiler to remove the offending bits, so `ghc-pkg` would see the above version number as `0.13.53.20200214.3025`, which should be enough to uniquely identify it. Note that, as far as I could find out, this number would never be exposed to users. 2. It is rather long, which I think is good from a human perspective as it makes it more scary. However, I have been told that this may be long enough to cause issues on Windows by pushing us past the max path size limitation of that "OS". I suggest we try it and see what happens. The upsides are: - It clearly indicates it is an unstable release (`alpha`). - It clearly indicates how old it is, by including the date. - To humans, it is immediately obvious which version is "later" even if they have the same date, allowing us to release same-day patches if needed. (Note: that is, commits that were made on the same day; the release date itself is irrelevant here.) - It contains the git sha so the commit built for that release is immediately obvious. - It sorts correctly under all schemes (modulo the modification for GHC). Alternatives I considered: - Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would be accepted by all schemes, but will not sort as expected under semantic versioning (though Maven will be fine). I have no idea how it will sort under GHC. - Not having any non-numeric component, e.g. `0.13.53.20200214.3025`. This is not valid semantic versioning and is therefore rejected by npm. - Not having detailed info: just go with `0.13.53-snapshot`. This is what is generally done in the Java world, but we then lose track of what version is actually in use and I'm concerned about bug reports. This would also not let us publish to the main Maven repo (at least not more than once), as artifacts there are supposed to be immutable. - No having a qualifier: `0.13.53-3025` would be acceptable to all three version formats. However, it would not clearly indicate to humans that it is not meant as a stable version, and would sort differently under semantic versioning (which counts it as a prerelease, i.e. before `0.13.53`) than under maven (which counts it as a patch, so after `0.13.53`). - Just counting releases: `0.13.53-alpha.1`, where we just count the number of prereleases in-between `0.13.52` and the next. This is currently the fallback plan if Windows path length causes issues. It would be less convenient to map releases to commits, but it could still be done via querying the history of the `LATEST` file. Release notes ============= > Note: We have decided not to have release notes for snapshot releases. Release notes are a bit tricky. Because we want the ability to make snapshot releases, then later on promote them to stable releases, it follows that we want to build commits from the past. However, if we decide post-hoc that a commit is actually a good candidate for a release, there is no way that commit can have the appropriate release notes: it cannot know what version number it's getting, and, moreover, we now track changes in commit messages. And I do not think anyone wants to go back to the release notes file being a merge bottleneck. But release notes need to be published to the releases blog upon releasing a stable version, and the docs website needs to be updated and include them. The only sensible solution here is to pick up the release notes as of the commit that triggers the release. As the docs cron runs asynchronously, this means walking down the git history to find the relevant commit. > Note: We could probably do away with the asynchronicity at this point. > It was originally included to cover for the possibility of a release > failing. If we are releasing commits from the past after they have been > tested, this should not be an issue anymore. If the docs generation were > part of the synchronous release step, it would have direct access to the > correct release notes without having to walk down the git history. > > However, I think it is more prudent to keep this change as a future step, > after we're confident the new release scheme does indeed produce much more > reliable "stable" releases. New release process =================== Just like releases are currently controlled mostly by detecting changes to the `VERSION` file, the new process will be controlled by detecting changes to the `LATEST` file. The format of that file will include both the version string and the corresponding SHA. Upon detecting a change to the `LATEST` file, CI will run the entire release process, just like it does now with the VERSION file. The main differences are: 1. Before running the release step, CI will checkout the commit specified in the LATEST file. This requires separating the release step from the build step, which in my opinion is cleaner anyway. 2. The `//:VERSION` Bazel target is replaced by a repository rule that gets the version to build from an environment variable, with a default of `0.0.0` to remain consistent with the current `daml-head` behaviour. Some of the manual steps will need to be skipped for a snapshot release. See amended `release/RELEASE.md` in this commit for details. The main caveat of this approach is that the official release will be a different binary from the corresponding snapshot. It will have been built from the same source, but with a different version string. This is somewhat mitigated by Bazel caching, meaning any build step that does not depend on the version string should use the cache and produce identical results. I do not think this can be avoided when our artifact includes its own version number. I must note, though, that while going through the changes required after removing the `VERSION` file, I have been quite surprised at the sheer number of things that actually depend on the SDK version number. I believe we should look into reducing that over time. CHANGELOG_BEGIN CHANGELOG_END
2020-02-25 19:01:23 +03:00
esac
trap - EXIT