The formatting check used to be git-repo-dependent (see #4985), which is
preventing our candidate 0.13.55 from building.
This PR introduces a temporary hack to disable format checking on
release PRs & commits. It should be reverted once 0.13.55 is released.
CHANGELOG_BEGIN
CHANGELOG_END
This PR changes the documentation release process to publish the
documentation for releases tagged "prerelease" on GitHub, while
discarding them when deciding on the latest version (the one that shows
on `/` on the docs site) and omitting them from the `versions.json` file
(meaning they do not appear on the dropdown).
This PR also makes a bit of cleanup/bug fixing:
- The change in `nix` toolset name (#4724) needs to be protected by a
version check, as we checkout older versions of the repo during docs
build.
- The data types BlogSubmit and BlogId seem to have survived the "dead
code detection" in #4956.
- The documentation build step had not been updated to pass down the
correct version string (#4513).
CHANGELOG_BEGIN
CHANGELOG_END
There have been a few GitHub glitches last week that resulted in a few
commits on master not being associated with a PR (though they really
were created from merging a PR, and the correct PR number is in their
title).
This makes the report script crash on not finding the PR, so this PR
fixes that. And a comment.
CHANGELOG_BEGIN
CHANGELOG_END
In the current setup, the report generation starts at 4. However, the
daily killing of all machines also starts at 4, giving the report little
chance of ever finishing.
CHANGELOG_BEGIN
CHANGELOG_END
During the latest attempt at making a snapshot release (#4749),
everything seemingly went well except for the step of signing the
Windows installer.
Based on a very obscure erorr message (Access Denied) and a bit of
Google search, my current hypothesis is that the signing fails because
the artifact produced by Bazel is read-only. This was not an issue in
the previous setup because we were getting the installer from an Azure
internal download between different jobs. We are now getting the binary
directly from Bazel.
This also adds a `set -e` to the relevant Bash snippet, because ideally
they should always have one.
CHANGELOG_BEGIN
CHANGELOG_END
Somehow, in the current setup, the publish steps do not get executed on
master. This is what Azure reports:
```
Evaluating: and(succeeded(), eq('$(is_release)', 'true'),
eq(variables['Build.SourceBranchName'], 'master'), eq('linux', 'linux'))
Expanded: and(True, eq('$(is_release)', 'true'),
eq(variables['Build.SourceBranchName'], 'master'), eq('linux', 'linux'))
Result: False
```
So it looks like, in the condition, `${{parameters.is_release}}`
evaluates to the literal string `$(is_release)`. If we look at the point
of invocation of the ~function~ template, we can see:
```
- template: ci/build-unix.yml
parameters:
release_tag: $(release_tag)
name: 'linux'
is_release: $(is_release)
```
so it does not seem completely crazy. However, according to the
documentation, we should expect that to be replaced by the value of the
corresponding variable, as per:
```
variables:
release_sha: $[ dependencies.check_for_release.outputs['out.release_sha'] ]
release_tag: $[ coalesce(dependencies.check_for_release.outputs['out.release_tag'], '0.0.0') ]
trigger_sha: $[ dependencies.check_for_release.outputs['out.trigger_sha'] ]
is_release: $[ dependencies.check_for_release.outputs['out.is_release'] ]
```
What's interesting here is that, within `build-unix.yml`, we are also
using `release_tag` in the exact same way:
```
- bash: ./build.sh "_$(uname)"
displayName: 'Build'
env:
DAML_SDK_RELEASE_VERSION: ${{parameters.release_tag}}
```
and this time output from the build seems to show the value being
correctly substituted:
```
damlc - Compiler and IDE backend for the Digital Asset Modelling
Language
SDK Version: 0.13.55-snapshot.20200226.3266.d58bb459
Usage: <interactive> COMMAND
Invoke the DAML compiler. Use -h for help.
```
My current guess is that the (undocumented, as far as I can tell)
evaluation order is as follows:
1. In the template, syntactically replace all the parameters.
2. In the job definition, replace the call to the template with the code
of the template. So it is as if we had written the template directly in
the `azure-pipelines.yml` file, with `$(release_tag)` and
`$(is_release)`.
3. Run the build. When we reach the time to run this specific job,
we can evaluate the expressions for the variables and replace them in
the rest of the job.
So what is going wrong? I believe the issue is with the quotes,
preventing the substitution of `is_release`. They came directly from the
[documented
syntax](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/conditions?view=azure-devops&tabs=yaml#use-a-template-parameter-as-part-of-a-condition),
but if the above evaluation order is correct, they should not be there.
There are actually two things going wrong here. The first one is that
the syntax `$()` is used to substitute a value in what Azure considers a
string. This is the case for `env` keys. However, the `condition` key
is not a string, it is an Azure "expression". Expressions have their own
evaluation rules and syntax, and in particular, `$()` is not a
substitution rule there, so when it sees `$()` in a string in an
expression (due to the quoptes), it leaves it alone.
Removing the quotes does not directly help, though, as we then end with
```
condition: eq($(is_release), 'true')
```
and `$()` is not valid syntax in an expression. The way to use variables
in an expression is `variables.name` (or `variables["name"]`, because
why have only one?).
So that means we have to pass variables to the template in different
ways depending on how they will be used. So much fun.
CHANGELOG_BEGIN
CHANGELOG_END
The existing approach is a historical accident. The reason for the
additional tarball/install jobs was that, in my original attempt, the
build steps would still build the current commit, as opposed to the
target commit.
This is not such an issue on Linux, but setting up the build environment
on Windows and macOS _again_ for no good reason is a pure waste of time
(and effort in getting it right). Now that the build steps build the
target commit (with the env var set), we can go back to the way things
were previously: just take the build products directly from the build
step.
CHANGELOG_BEGIN
CHANGELOG_END
This disables the PDF docs builds on MacOS on CI (they are still built
locally by default) and removes them from the Nix closure by
introducing a separate ci-cached attribute that filters out texlive.
Since we built `nix-build nix -A tools -A cached` on CI, I’ve also
removed all the Tex stuff from tools which only means that it ends up
in PATH which nobody seems to care about.
changelog_begin
changelog_end
Context
=======
After multiple discussions about our current release schedule and
process, we've come to the conclusion that we need to be able to make a
distinction between technical snapshots and marketing releases. In other
words, we need to be able to create a bundle for early adopters to test
without making it an officially-supported version, and without
necessarily implying everyone should go through the trouble of
upgrading. The underlying goal is to have less frequent but more stable
"official" releases.
This PR is a proposal for a new release process designed under the
following constraints:
- Reuse as much as possible of the existing infrastructure, to minimize
effort but also chances of disruptions.
- Have the ability to create "snapshot"/"nightly"/... releases that are
not meant for general public consumption, but can still be used by savvy
users without jumping through too many extra hoops (ideally just
swapping in a slightly-weirder version string).
- Have the ability to promote an existing snapshot release to "official"
release status, with as few changes as possible in-between, so we can be
confident that the official release is what we tested as a prerelease.
- Have as much of the release pipeline shared between the two types of
releases, to avoid discovering non-transient problems while trying to
promote a snapshot to an official release.
- Triggerring a release should still be done through a PR, so we can
keep the same approval process for SOC2 auditability.
The gist of this proposal is to replace the current `VERSION` file with
a `LATEST` file, which would have the following format:
```
ef5d32b7438e481de0235c5538aedab419682388 0.13.53-alpha.20200214.3025.ef5d32b7
```
This file would be maintained with a script to reduce manual labor in
producing the version string. Other than that, the process will be
largely the same, with releases triggered by changes to this `LATEST`
and the release notes files.
Version numbers
===============
Because one of the goals is to reduce the velocity of our published
version numbers, we need a different version scheme for our snapshot
releases. Fortunately, most version schemes have some support for that;
unfortunately, the SDK sits at the intersection of three different
version schemes that have made incompatible choices. Without going into
too much detail:
- Semantic versioning (which we chose as the version format for the SDK
version number) allows for "prerelease" version numbers as well as
"metadata"; an example of a complete version string would be
`1.2.3-nightly.201+server12.43`. The "main" part of the version string
always has to have 3 numbers separated by dots; the "prerelease"
(after the `-` but before the `+`) and the "metadata" (after the `+`)
parts are optional and, if present, must consist of one or more segments
separated by dots, where a segment can be either a number or an
alphanumeric string. In terms of ordering, metadata is irrelevant and
any version with a prerelease string is before the corresponding "main"
version string alone. Amongst prereleases, segments are compared in
order with purely numeric ones compared as numbers and mixed ones
compared lexicographically. So 1.2.3 is more recent than 1.2.3-1,
which is itself less recent than 1.2.3-2.
- Maven version strings are any number of segments separated by a `.`, a
`-`, or a transition between a number and a letter. Version strings
are compared element-wise, with numeric segments being compared as
numbers. Alphabetic segments are treated specially if they happen to be
one of a handful of magic words (such as "alpha", "beta" or "snapshot"
for example) which count as "qualifiers"; a version string with a
qualifier is "before" its prefix (`1.2.3` is before `1.2.3-alpha.3`,
which is the same as `1.2.3-alpha3` or `1.2.3-alpha-3`), and there is a
special ordering amongst qualifiers. Other alphabetic segments are
compared alphabetically and count as being "after" their prefix
(`1.2.3-really-final-this-time` counts as being released after `1.2.3`).
- GHC package numbers are comprised of any number of numeric segments
separated by `.`, plus an optional (though deprecated) alphanumeric
"version tag" separated by a `-`. I could not find any official
documentation on ordering for the version tag; numeric segments are
compared as numbers.
- npm uses semantic versioning so that is covered already.
After much more investigation than I'd care to admit, I have come up
with the following compromise as the least-bad solution. First,
obviously, the version string for stable/marketing versions is going to
be "standard" semver, i.e. major.minor.patch, all numbers, which works,
and sorts as expected, for all three schemes. For snapshot releases, we
shall use the following (semver) format:
```
0.13.53-alpha.20200214.3025.ef5d32b7
```
where the components are, respectively:
- `0.13.53`: the expected version string of the next "stable" release.
- `alpha`: a marker that hopefully scares people enough.
- `20200214`: the date of the release commit, which _MUST_ be on
master.
- `3025`: the number of commits in master up to the release commit
(included). Because we have a linear, append-only master branch, this
uniquely identifies the commit.
- `ef5d32b7ù : the first 8 characters of the release commit sha. This is
not strictly speaking necessary, but makes it a lot more convenient to
identify the commit.
The main downsides of this format are:
1. It is not a valid format for GHC packages. We do not publish GHC
packages from the SDK (so far we have instead opted to release our
Haskell code as separate packages entirely), so this should not be an
issue. However, our SDK version currently leaks to `ghc-pkg` as the
version string for the stdlib (and prim) packages. This PR addresses
that by tweaking the compiler to remove the offending bits, so `ghc-pkg`
would see the above version number as `0.13.53.20200214.3025`, which
should be enough to uniquely identify it. Note that, as far as I could
find out, this number would never be exposed to users.
2. It is rather long, which I think is good from a human perspective as
it makes it more scary. However, I have been told that this may be
long enough to cause issues on Windows by pushing us past the max path
size limitation of that "OS". I suggest we try it and see what
happens.
The upsides are:
- It clearly indicates it is an unstable release (`alpha`).
- It clearly indicates how old it is, by including the date.
- To humans, it is immediately obvious which version is "later" even if
they have the same date, allowing us to release same-day patches if
needed. (Note: that is, commits that were made on the same day; the
release date itself is irrelevant here.)
- It contains the git sha so the commit built for that release is
immediately obvious.
- It sorts correctly under all schemes (modulo the modification for
GHC).
Alternatives I considered:
- Pander to GHC: 0.13.53-alpha-20200214-3025-ef5d32b7. This format would
be accepted by all schemes, but will not sort as expected under semantic
versioning (though Maven will be fine). I have no idea how it will sort
under GHC.
- Not having any non-numeric component, e.g. `0.13.53.20200214.3025`.
This is not valid semantic versioning and is therefore rejected by
npm.
- Not having detailed info: just go with `0.13.53-snapshot`. This is
what is generally done in the Java world, but we then lose track of what
version is actually in use and I'm concerned about bug reports. This
would also not let us publish to the main Maven repo (at least not more
than once), as artifacts there are supposed to be immutable.
- No having a qualifier: `0.13.53-3025` would be acceptable to all three
version formats. However, it would not clearly indicate to humans that
it is not meant as a stable version, and would sort differently under
semantic versioning (which counts it as a prerelease, i.e. before
`0.13.53`) than under maven (which counts it as a patch, so after
`0.13.53`).
- Just counting releases: `0.13.53-alpha.1`, where we just count the
number of prereleases in-between `0.13.52` and the next. This is
currently the fallback plan if Windows path length causes issues. It
would be less convenient to map releases to commits, but it could still
be done via querying the history of the `LATEST` file.
Release notes
=============
> Note: We have decided not to have release notes for snapshot releases.
Release notes are a bit tricky. Because we want the ability to make
snapshot releases, then later on promote them to stable releases, it
follows that we want to build commits from the past. However, if we
decide post-hoc that a commit is actually a good candidate for a
release, there is no way that commit can have the appropriate release
notes: it cannot know what version number it's getting, and, moreover,
we now track changes in commit messages. And I do not think anyone wants
to go back to the release notes file being a merge bottleneck.
But release notes need to be published to the releases blog upon
releasing a stable version, and the docs website needs to be updated and
include them.
The only sensible solution here is to pick up the release notes as of
the commit that triggers the release. As the docs cron runs
asynchronously, this means walking down the git history to find the
relevant commit.
> Note: We could probably do away with the asynchronicity at this point.
> It was originally included to cover for the possibility of a release
> failing. If we are releasing commits from the past after they have been
> tested, this should not be an issue anymore. If the docs generation were
> part of the synchronous release step, it would have direct access to the
> correct release notes without having to walk down the git history.
>
> However, I think it is more prudent to keep this change as a future step,
> after we're confident the new release scheme does indeed produce much more
> reliable "stable" releases.
New release process
===================
Just like releases are currently controlled mostly by detecting
changes to the `VERSION` file, the new process will be controlled by
detecting changes to the `LATEST` file. The format of that file will
include both the version string and the corresponding SHA.
Upon detecting a change to the `LATEST` file, CI will run the entire
release process, just like it does now with the VERSION file. The main
differences are:
1. Before running the release step, CI will checkout the commit
specified in the LATEST file. This requires separating the release
step from the build step, which in my opinion is cleaner anyway.
2. The `//:VERSION` Bazel target is replaced by a repository rule
that gets the version to build from an environment variable, with a
default of `0.0.0` to remain consistent with the current `daml-head`
behaviour.
Some of the manual steps will need to be skipped for a snapshot release.
See amended `release/RELEASE.md` in this commit for details.
The main caveat of this approach is that the official release will be a
different binary from the corresponding snapshot. It will have been
built from the same source, but with a different version string. This is
somewhat mitigated by Bazel caching, meaning any build step that does
not depend on the version string should use the cache and produce
identical results. I do not think this can be avoided when our artifact
includes its own version number.
I must note, though, that while going through the changes required after
removing the `VERSION` file, I have been quite surprised at the sheer number of
things that actually depend on the SDK version number. I believe we should
look into reducing that over time.
CHANGELOG_BEGIN
CHANGELOG_END
They can weigh close to 1GB, and the internal Azure networks are
unreliable enough that this adds up significant amounts of time to our
already slow CI pipeline (usually around a minute, but I've seen at
least one case where it took almost 28 minutes).
Given that we pretty much never look a them, 🔥.
CHANGELOG_BEGIN
CHANGELOG_END
* Update CI nix version
For `--option http2 false` to take effect requires Nix 2.3.2.
CHANGELOG_BEGIN
CHANGELOG_END
* Set option `http2 = false` dev-env nix config
This is less likely to overlook an instance than manually adding
`--option http2 false` to each Nix invocation.
Setting `--option htt2p false` also had no effect on the multi-user Nix
installation on the Linux CI machines due to
```
WARNING: option '--disk_cache' was expanded to from both option '--config linux' (source /nix/store/2xnfb2l39d2b4nxw5vwmqz5hjwhw0caw-daml-bazelrc) and option '--config linux' (source /nix/store/2xnfb2l39d2b4nxw5vwmqz5hjwhw0caw-daml-bazelrc)
```
Co-authored-by: Andreas Herrmann <andreash87@gmx.ch>
As part of our SOC2/ISO certification, we need to be able to evidence a
list of "Standard Changes" for the DAML SDK project. This commit adds a
script that extracts, for a given month, the list of Standard Changes
that happened along with relevant information (author, reviewer, etc.).
It also adds a definition for a monthly cron job on Azure to run the
script and send the result to Slack, @-mentioning Martin.
CHANGELOG_BEGIN
CHANGELOG_END
One of the outputs of our brainstorming about how to make CI better was
that it is annoying to have to "babysit" pull requests. This PR attempts
to introduce a notification mechanism by which Azure will notify people
on Slack when a build finishes, so they know they need to go and rerun
or merge the corresponding PR.
This commit also changes the existing $Slack.URL variable to
$Slack.team-daml, to make more explicit where the Slack message is being
sent to (Slack works with one token per destination channel). Both
$Slack.URL and $Slack.team-daml are currently defined as the same token
in Azure.
CHANGELOG_BEGIN
CHANGELOG_END
Originally we ran the release step on both Linux and MacOS to handle
platform dependent artifacts, in particular, damlc.jar. However, we
don’t have any platform dependent artifacts that are uploaded as part
of the release script anymore and I hope we will never have to add any
in the future.
So this PR, removes the code for handling platform dependent artifacts
in the release step and disables the release step on MacOS (while
still setting the variables like we do on Windows).
Currently the release step still costs us ~2 minutes on MacOS which is
already our slowest platform so hopefully this will speed things up a
bit.
changelog_begin
changelog_end
Currently, pretty much all of our builds are bottlenecked on
MacOS (mainly because the builders are slower and have worse
caching). The release step adds > 3min to each build which is a bit
annoying. It turns out that removing the calls to `bazel query` which
are used to check for missing Maven dependencies speeds this up by >
5x. Given that the check is platform independent anyway, we can just
disable it on MacOS.
changelog_begin
changelog_end
* make packages public
This uploads the typescript npm packages of the language support to the
npm registry in the release process.
CHANGELOG_BEGIN
CHANGELOG_END
* address moritz/gary's review
* generate the .npmrc file
* adding debug output
just in case the upload will fail in the next release.
* reverse package order
This patch changes the call to the GitHub API that translates the
release notes from markdown to HTML to use gfm instead of plain
markdown. gfm is a superset of markdown that adds the following:
- GitHub usernames (`@`-mentions) are turned into links to the user's
profile page.
- Issues and PR numbers (`#1234`) are turned into links to the
corresponding issue or PR.
- Existing git shas are turned into links to the corresponding commit.
An example of this feature missing is the release notes for
[v0.13.42](https://blog.daml.com/release-notes/0.13.42-1), where
intended links such as
> - Rename argument in active contract to payload. See #3826.
are not rendered.
CHANGELOG_BEGIN
CHANGELOG_END
We have recently added the option for this script to not build some
versions (because they are too old and external dependencies have
changed from under them). We also have changed the GitHub call to get
all the history of releases.
This PR changes the logic to generate the `versions.json` file so that
it only contains versions that we have either built or copied over.
Consequently, it also changes the logic to decide whether this job
should run to depend only on the latest version, rather than the whole
list of versions.
CHANGELOG_BEGIN
CHANGELOG_END
The GitHub API is paginated (30 items by default). This creates two
problems:
1. At the moment, older versions silently drop from the docs website,
without us having made any explicit decision about it.
2. When we prepare a new version, it gets created as a pre-release
version. Our script filters that out, but that happens on our end so
we end up with 29 published versions and the list is different form the
existing one. If the prerelease then gets dropped, the oldest version
comes back.
It is possible that we will sometime decide we do not want to keep old
documentation around forever, but that should be an explicit decision.
This patch changes the logic to fetch the list of versions from GitHub
so that we always get all the published versions (barring race
conditions inherent to that kind of paginated API).
CHANGELOG_BEGIN
CHANGELOG_END
The docs build is currently not reproducible as it include to-the-minute
time-of-build information. It also includes some Sphinx binary caches
which I suppose will also not be reproducible (though I have not checked
the details there).
This commit attempts to remove all sources of non-reproducibility from
the docs build, though this is hard to test without having a stable,
older release to compare with.
CHANGELOG_BEGIN
CHANGELOG_END
The latest changes to the docs cron have introduced a bug whereby the
"latest" version is determined including prereleases.
CHANGELOG_BEGIN
CHANGELOG_END
This commit makes two conceptually independent changes:
1. It adds a checksum file to each version folder. This allows the
script to detect when a version has not been correctly uploaded.
2. It changes the script to first download all the docs website, and
then reuse existing version folders where appropriate (i.e. when their
folder matches its checksum).
The hope is that this will reduce the time it takes to deploy a new
version, as only the current version should be rebuilt (in addition to
previous, failed versions).
The first time this cron runs (upon next release as per the current
setup), however, it will still rebuild all existing versions as they do
not currently have a checksum.
CHANGELOG_BEGIN
CHANGELOG_END
This commit aims at mitigating two issues we have noticed with the
0.13.41 release:
1. The initial cron run for that release got interrupted at the 50
minutes mark, which happened to be right in the middle of the s3 upload.
This means it had already changed the versions.json file, but had not
finished updating the actual html files. Right now, the docs.daml.com
website shows version 0.13.41 in the drop-down, but actually displays
the content for 0.13.40. Additionally, trying to explicitly visit the
website for 0.13.41 (https://docs.daml.com/0.13.41) yields a 404. Note
that this also means the cron job did not reach the "tell HubSpot"
point, so 0.13.41 did not get announced.
2. As the script also did not reach the "clear cache" step, subsequent
runs have been rebuilding the documentation for no reason as the
sequence of steps was: check versions.json through HTTP, get cached one,
see it's not up-to-date, build docs, check versions.json through s3 API,
bypassing the cache, see it's up-to-date, stop.
To address those issues, this PR changes the cron to:
1. Increase the timeout to 2h instead of 50 minutes.
2. Always check the versions.json file through s3, rather than go
through the HTTP cache first.
These are not complete solutions but I'm not sure how to do better given
that s3 does not have atomic operations.
* re-add cleanup for /tmp to remove 700ish mb of unneeded temp files made by the sdk installer
* Set WORKDIR to daml user home dir so that sdk tools can create files
* add daml sdk config defaults for auto-install and update-check sdk install RUN command
* add --no-cache to apk add to reduce size a little
* add line return to end of daml-config.yaml
Currently if the docs script fails, the Slack message we get mentions the commit title of the docs version that failed to build, which is not super useful. This ensures we get back to the current commit regardless of what happens with the Haskell script.
* bazel: 0.28.1 --> 1.1.0
* bazel-watcher sha256
* Fix missing line in patch
* proto_source_root --> strip_import_prefix
See https://github.com/bazelbuild/bazel/issues/7153 for details.
* Update rules_nixpkgs
Required to avoid errors of the form
```
ERROR: An error occurred during the fetch of repository 'node_nix':
parameter 'sep' may not be specified by name, for call to method split(sep, maxsplit = None) of 'string'
```
and
```
ERROR: An error occurred during the fetch of repository 'node_nix':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'nodejs'.
Command: [/Users/runner/.nix-profile/bin/nix-build, /private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/node_nix/nix/bazel.nix, "-A", "nodejs", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
src/main/tools/process-tools.cc:173: "setitimer": Invalid argument
```
* Update rules_scala
* .proto has been removed, use [ProtoInfo] instead
See
https://docs.bazel.build/versions/1.1.0/be/protocol-buffer.html#proto_library
* python3_nix add nix_file attribute
To avoid the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/BUILD:66:1: //:nix_python3_runtime depends on @python3_nix//:bin/python in repository @python3_nix which failed to fetch. no such package '@python3_nix//': Traceback (most recent call last):
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'python3'.
Command: [/home/aj/.nix-profile/bin/nix-build, "-E", "import <nixpkgs> { config = {}; overlays = []; }", "-A", "python3", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
error: anonymous function at /home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs.nix:3:1 called with unexpected argument 'config', at (string):1:1
```
* rules_haskell unnamed string.split(_, maxsplit = _)
The keyword argument may no longer be named.
* string.replace(_, _, maxsplit = _) may not be named
* Move proto sources from deps to data
Fixes
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation
```
* Define sha256 for haskell_ghc__paths
Bazel 1.1.0 fails on missing hashes.
* Disable --incompatible_windows_native_test_wrapper
* //compiler/daml-extension don't modify sources
Modifying sources in-place can cause issues on Windows, where build
actions are not sandboxed and changes on sources can affect other build
steps.
* bazel-genfiles --> bazel-bin
The bazel-genfiles symlink has been removed since Bazel 1.0.
See https://github.com/bazelbuild/bazel/issues/8651
* Mark dev_env_tool repository rule as configure
See
https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule
* Move data deps into data attribute
* Mark dev_env_tool as local = True
* Manually fetch @makensis_dev_env
* Shrink the docker image for the SDK by 57%
Wiping out the `/tmp` dir after installing the SDK does wonders.
@associahedron I wonder if we should do this in the assistant?
* Update release notes
Previously, we were installing the SDK as root which is probably not a
good idea. This PR adds a new `daml` user and fixes PATH (`$HOME` and
`~` both don’t work in this context).
* Fixes#1725: Correct Maven credential variables in CI release script.
Update documentation that refered explicitly to the old version, to
use refer to new version.
* Fixes#1204: Release bindings and codegens to Maven Central.
Upload the Java and Scala Bindings with the respective code
generator binaries to Sonatype Open Source Repository
Host for synchronization with Maven Central.
* webide: build webide image when sdk releases
* add scripts which check the latest version of sdk. If webide docker
image version does not exist or is older than the sdk version, it will
kick off a build of the webide docker image
* add job to azure cron
* webide: minor response to review
* windows: fixed daml-lf tests for Windows by using Bazel's rlocation
* more consistent logging on CI; publishing Windows test logs on failure
* windows: fix daml-lf engine tests
* windows: add diff tool to msys
This is a first step towards improving our docs release process. The
goal here is to get rid of the manual "publish docs" step. This is done
as a periodic check because we only want to run this for "published"
releases, i.e. the ones that are not marked as prerelease. Because the
act of publishing a release is a manual step that Azure cannot trigger
on, we instead opt for a periodic check.
Not included in this piece of work:
- Any change to the docs themselves; the goal here is to automate the
current process as a first step. Future plans for the docs themselves
include adding links to older versions of the docs.
- A better way to detect docs are already up-to-date, and abort if so.
- Including older versions of the docs.
- Switching the DNS record from the current AWS S3 bucket to this new
GCS bucket. That will be a manual step once we're happy with how the
new bucket works.
This reverts commit 3d8acde916.
For some reason that commit seems to have resulted in a lot of
"unexpected end of file" errors during cache downloads. I do not know
what is going on here or how to fix it so let’s revert it for now.
* release: make 'ci/release.sh' runnable for dry runs.
release-dry-run.sh is outdated and duplicates logic from ci/release.sh, so it
got deleted.
* ledger-api-test-tool: release the tool together with the SDK components.
* ledger-api-test-tool: update docs to reflect distribution mechanism.
* ledger-api-test-tool: further docs refinements.
* Add Ledger API Test Tool mention into release notes.
The ci/release.sh fails if the BUILD_SOURCEBRANCHNAME environment
variable is not set. Although this variable is normally set by the
CI system, it is sometimes useful to run the script manually and
simply adding an 'invalid' default to the check of the env variable
means that the script still works if the variable is unbound.
The newer version seems to segfault on MacOS quite often so let’s
downgrade for now. We should also try to see if we can find a
reasonable way of reproducing this and report it upstream.
As multiple platforms will create different annotated tags (because an
annotated tag includes a tag time), they will conflict on trying to
push. Therefore, we go for a lightweight tag for now, as those are
simple pointers to a commit and git will recognize that they are the
same and there is no conflict.
This rewrites the release script to be a lot simpler and significantly
faster:
- The artifacts are now declared in a separate yaml file which should
make it easier for people to modify and doesn’t clutter the actual
code.
- There is only a constant number of calls to Bazel which speeds up
the script quite a bit.
I verified that the release artifacts are the same that we got
before and I traced the calls to the jfrog binary in a fake release
and ignoring order they are identical.
This adds `ci/azure-cleanup`, containing a script that talks to azure pipelines, removing agents older than 25 hours in a specific pool.
Machines are meant to be killed after 24 hours anyway, make sure they're properly unregistered from Azure Pipelines, too.
By doing this, we don't need to unregister nodes manually on shutdown.
Idea is to execute this every time a new agent is provisioned, it has cloned the repo. We intend to clone the repo and pre-warm the caches there anyhow.
WIP until the repo fetching and cache pre-warming is present, too.
cc @zimbatm
### Pull Request Checklist
- [x] Read and understand the [contribution guidelines](https://github.com/digital-asset/daml/blob/master/CONTRIBUTING.md)
- [x] Include appropriate tests
- [x] Set a descriptive title and thorough description
- [x] Add a reference to the [issue this PR will solve](https://github.com/digital-asset/daml/issues), if appropriate
- [x] Add a line to the [release notes](https://github.com/digital-asset/daml/blob/master/docs/source/support/release-notes.rst), if appropriate
NOTE: CI is not automatically run on non-members pull-requests for security
reasons. The reviewer will have to comment with `/AzurePipelines run` to
trigger the build.
Azure Pipelines has direct integration with GitHub, so we're just using
that. Releases on GitHub have to target a tag, so we also need to push
the tag as an intermediate step; we also need to include the platform
name in the artifact to avoid overwriting from different builds.
The two "GitHub release" steps depend on two Azure variables that are
not defined in the pipeline script. This may look like it should not
work, but in fact it does, because these variables are set by the
release script.
In Azure Pipelines, any build step can set variables for the next build
steps by outputting specially-formatted text to stdout. This text will
not appear in the build output displayed by Azure Pipelines, e.g.:
```
echo '##vso[task.setvariable variable=sauce]tomatoes'
```
would define the Azure variable `sauce` to have the string `tomatoes` as
its value for the next build steps.
See [0] for details.
[0]: https://docs.microsoft.com/en-us/azure/devops/pipelines/process/variables?view=azure-devops&tabs=yaml%2Cbatch#set-in-script
* nix: add the more providers to terraform
* docs: make tarballs more reproducible
* ci: use the linux-pool pool
* ci: tweak the nix installation
handle the case where the user is root and on ubuntu
* infra: terraform fmt
* infra: add Azure Pipeline agents
* ci: only enable linux-pool for internal PRs
Without this PR, this variable ends up being set to the string
"variables['System.PullRequest.IsFork']" which meant that we never
uploaded to the Bazel cache.
