The docs build is currently not reproducible as it include to-the-minute
time-of-build information. It also includes some Sphinx binary caches
which I suppose will also not be reproducible (though I have not checked
the details there).
This commit attempts to remove all sources of non-reproducibility from
the docs build, though this is hard to test without having a stable,
older release to compare with.
CHANGELOG_BEGIN
CHANGELOG_END
The latest changes to the docs cron have introduced a bug whereby the
"latest" version is determined including prereleases.
CHANGELOG_BEGIN
CHANGELOG_END
This commit makes two conceptually independent changes:
1. It adds a checksum file to each version folder. This allows the
script to detect when a version has not been correctly uploaded.
2. It changes the script to first download all the docs website, and
then reuse existing version folders where appropriate (i.e. when their
folder matches its checksum).
The hope is that this will reduce the time it takes to deploy a new
version, as only the current version should be rebuilt (in addition to
previous, failed versions).
The first time this cron runs (upon next release as per the current
setup), however, it will still rebuild all existing versions as they do
not currently have a checksum.
CHANGELOG_BEGIN
CHANGELOG_END
This commit aims at mitigating two issues we have noticed with the
0.13.41 release:
1. The initial cron run for that release got interrupted at the 50
minutes mark, which happened to be right in the middle of the s3 upload.
This means it had already changed the versions.json file, but had not
finished updating the actual html files. Right now, the docs.daml.com
website shows version 0.13.41 in the drop-down, but actually displays
the content for 0.13.40. Additionally, trying to explicitly visit the
website for 0.13.41 (https://docs.daml.com/0.13.41) yields a 404. Note
that this also means the cron job did not reach the "tell HubSpot"
point, so 0.13.41 did not get announced.
2. As the script also did not reach the "clear cache" step, subsequent
runs have been rebuilding the documentation for no reason as the
sequence of steps was: check versions.json through HTTP, get cached one,
see it's not up-to-date, build docs, check versions.json through s3 API,
bypassing the cache, see it's up-to-date, stop.
To address those issues, this PR changes the cron to:
1. Increase the timeout to 2h instead of 50 minutes.
2. Always check the versions.json file through s3, rather than go
through the HTTP cache first.
These are not complete solutions but I'm not sure how to do better given
that s3 does not have atomic operations.
* re-add cleanup for /tmp to remove 700ish mb of unneeded temp files made by the sdk installer
* Set WORKDIR to daml user home dir so that sdk tools can create files
* add daml sdk config defaults for auto-install and update-check sdk install RUN command
* add --no-cache to apk add to reduce size a little
* add line return to end of daml-config.yaml
Currently if the docs script fails, the Slack message we get mentions the commit title of the docs version that failed to build, which is not super useful. This ensures we get back to the current commit regardless of what happens with the Haskell script.
* bazel: 0.28.1 --> 1.1.0
* bazel-watcher sha256
* Fix missing line in patch
* proto_source_root --> strip_import_prefix
See https://github.com/bazelbuild/bazel/issues/7153 for details.
* Update rules_nixpkgs
Required to avoid errors of the form
```
ERROR: An error occurred during the fetch of repository 'node_nix':
parameter 'sep' may not be specified by name, for call to method split(sep, maxsplit = None) of 'string'
```
and
```
ERROR: An error occurred during the fetch of repository 'node_nix':
Traceback (most recent call last):
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'nodejs'.
Command: [/Users/runner/.nix-profile/bin/nix-build, /private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/node_nix/nix/bazel.nix, "-A", "nodejs", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/private/var/tmp/_bazel_runner/17d2b3954f1c6dcf5414d5453467df9a/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
src/main/tools/process-tools.cc:173: "setitimer": Invalid argument
```
* Update rules_scala
* .proto has been removed, use [ProtoInfo] instead
See
https://docs.bazel.build/versions/1.1.0/be/protocol-buffer.html#proto_library
* python3_nix add nix_file attribute
To avoid the following error
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/BUILD:66:1: //:nix_python3_runtime depends on @python3_nix//:bin/python in repository @python3_nix which failed to fetch. no such package '@python3_nix//': Traceback (most recent call last):
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 149
_execute_or_fail(repository_ctx, <3 more arguments>)
File "/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/io_tweag_rules_nixpkgs/nixpkgs/nixpkgs.bzl", line 318, in _execute_or_fail
fail(<1 more arguments>)
Cannot build Nix attribute 'python3'.
Command: [/home/aj/.nix-profile/bin/nix-build, "-E", "import <nixpkgs> { config = {}; overlays = []; }", "-A", "python3", "--out-link", "bazel-support/nix-out-link", "-I", "nixpkgs=/home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs"]
Return code: 1
Error output:
error: anonymous function at /home/aj/.cache/bazel/_bazel_aj/5f825ad28f8e070f999ba37395e46ee5/external/nixpkgs/nixpkgs.nix:3:1 called with unexpected argument 'config', at (string):1:1
```
* rules_haskell unnamed string.split(_, maxsplit = _)
The keyword argument may no longer be named.
* string.replace(_, _, maxsplit = _) may not be named
* Move proto sources from deps to data
Fixes
```
ERROR: /home/aj/tweag.io/da/da-bazel-1.1/daml-lf/archive/BUILD.bazel:150:1: in deps attribute of scala_test rule //daml-lf/archive:daml_lf_archive_reader_tests_test_suite_src_test_scala_com_digitalasset_daml_lf_archive_DecodeV1Spec.scala: '//daml-lf/archive:daml_lf_1.6_archive_proto_srcs' does not have mandatory providers: 'JavaInfo'. Since this rule was created by the macro 'da_scala_test_suite', the error might have been caused by the macro implementation
```
* Define sha256 for haskell_ghc__paths
Bazel 1.1.0 fails on missing hashes.
* Disable --incompatible_windows_native_test_wrapper
* //compiler/daml-extension don't modify sources
Modifying sources in-place can cause issues on Windows, where build
actions are not sandboxed and changes on sources can affect other build
steps.
* bazel-genfiles --> bazel-bin
The bazel-genfiles symlink has been removed since Bazel 1.0.
See https://github.com/bazelbuild/bazel/issues/8651
* Mark dev_env_tool repository rule as configure
See
https://docs.bazel.build/versions/1.1.0/skylark/lib/globals.html#repository_rule
* Move data deps into data attribute
* Mark dev_env_tool as local = True
* Manually fetch @makensis_dev_env
* Shrink the docker image for the SDK by 57%
Wiping out the `/tmp` dir after installing the SDK does wonders.
@associahedron I wonder if we should do this in the assistant?
* Update release notes
Previously, we were installing the SDK as root which is probably not a
good idea. This PR adds a new `daml` user and fixes PATH (`$HOME` and
`~` both don’t work in this context).
* Fixes#1725: Correct Maven credential variables in CI release script.
Update documentation that refered explicitly to the old version, to
use refer to new version.
* Fixes#1204: Release bindings and codegens to Maven Central.
Upload the Java and Scala Bindings with the respective code
generator binaries to Sonatype Open Source Repository
Host for synchronization with Maven Central.
* webide: build webide image when sdk releases
* add scripts which check the latest version of sdk. If webide docker
image version does not exist or is older than the sdk version, it will
kick off a build of the webide docker image
* add job to azure cron
* webide: minor response to review
* windows: fixed daml-lf tests for Windows by using Bazel's rlocation
* more consistent logging on CI; publishing Windows test logs on failure
* windows: fix daml-lf engine tests
* windows: add diff tool to msys
This is a first step towards improving our docs release process. The
goal here is to get rid of the manual "publish docs" step. This is done
as a periodic check because we only want to run this for "published"
releases, i.e. the ones that are not marked as prerelease. Because the
act of publishing a release is a manual step that Azure cannot trigger
on, we instead opt for a periodic check.
Not included in this piece of work:
- Any change to the docs themselves; the goal here is to automate the
current process as a first step. Future plans for the docs themselves
include adding links to older versions of the docs.
- A better way to detect docs are already up-to-date, and abort if so.
- Including older versions of the docs.
- Switching the DNS record from the current AWS S3 bucket to this new
GCS bucket. That will be a manual step once we're happy with how the
new bucket works.
This reverts commit 3d8acde916.
For some reason that commit seems to have resulted in a lot of
"unexpected end of file" errors during cache downloads. I do not know
what is going on here or how to fix it so let’s revert it for now.
* release: make 'ci/release.sh' runnable for dry runs.
release-dry-run.sh is outdated and duplicates logic from ci/release.sh, so it
got deleted.
* ledger-api-test-tool: release the tool together with the SDK components.
* ledger-api-test-tool: update docs to reflect distribution mechanism.
* ledger-api-test-tool: further docs refinements.
* Add Ledger API Test Tool mention into release notes.
The ci/release.sh fails if the BUILD_SOURCEBRANCHNAME environment
variable is not set. Although this variable is normally set by the
CI system, it is sometimes useful to run the script manually and
simply adding an 'invalid' default to the check of the env variable
means that the script still works if the variable is unbound.
