* build canton 3.x in build.yml
* Apply suggestions from code review
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Apply suggestions from code review
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* apply suggestions
---------
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* introduce a canton2 directory
* rename to canton-3x and only keep the bazel rules
* add canton-3x to bazelignore
* fix visibility rules
* add a build-canton-3x script
* add build-canton-3x.sh temporarily to prs.yml to test it out
* try sending a message to stack when failing
* backport 243125adee
* restore prs.yml and add the job to azure-cron.yml
* restore tell-slack-failed condition
* address comments
* display the revision at which canton was cloned, to ease debugging
Turns out some people depend on it. I still think they shouldn't, and we
should work with them to help them move away, but short-term the right
thing to do is to not block their upgrade.
So that the Docker image file depends on when the image was created
rather than the branch. I think that makes more sense overall, but we
need this to include the OpenSSL bump into 2.3.1.
CHANGELOG_BEGIN
CHANGELOG_END
CHANGELOG_BEGIN
- The base image used by the `digitalasset/daml-sdk` Docker image has
been bumped from Ubuntu Focal to Ubuntu Kinetic. Please remember that
this image is meant for demonstration purposes only and is not supported
for use in any kind of production setup.
CHANGELOG_END
Same goal as #14055, but this should fix this week's image (we build
from the release commit, so #14055 only fixes future images), as well as
allow us to rebuild past images should we ever need to.
CHANGELOG_BEGIN
CHANGELOG_END
This upgrades vsce to the latest version and also markdown-it which is
unfortunately still pinned to an older version with a vulnerability.
There are some minor changes required to our invocations of vsce. I
tried to test this locally up to the point where it fails due to me
not having a token. We’ll only fully see it working after the next
stable release unfortunately.
changelog_begin
changelog_end
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
Currently CI is failing with a 401. Confusingly, it doesn’t fail
during this step but fails during the upload below later because
$MARKET ends up pointing to garbage. This does not solve the failure
but it produces a more sensible error.
changelog_begin
changelog_end
* Clean broken entries from the Bazel cache
This is hopefully a somewhat reasonable workaround for the "output not
created" errors that keep annoying us.
For now, this is just part of the hourly cronjob but we could move it
somewhere else if desired.
changelog_begin
changelog_end
* Fix GCS credentials
changelog_begin
changelog_end
CHANGELOG_BEGIN
- Our Linux binaries are now built on Ubuntu 20.04 instead of 16.04. We
do not expect any user-level impact, but please reach out if you
do notice any issue that might be caused by this.
CHANGELOG_END
As we strive for more inclusiveness, we are becoming less comfortable
with historically-charged terms being used in our everyday work.
This is targeted for merge on Dec 26, _after_ the necessary
corresponding changes at both the GitHub and Azure Pipelines levels.
CHANGELOG_BEGIN
- DAML Connect development is now conducted from the `main` branch,
rather than the `master` one. If you had any dependency on the
digital-asset/daml repository, you will need to update this parameter.
CHANGELOG_END
Nobody should be using it anyway, but at least now they can trust it's
from us. If they bother to check.
CHANGELOG_BEGIN
- The SDK Docker image is now signed. Reminder: this is a dev-only
image, with absolutely no support for any kind of production use-case.
To verify the signature, use the `docker trust inspect` command.
You can also set the `DOCKER_CONTENT_TRUST` environment variable to 1
to instruct Docker commands to only pull and run signed images. Keep
in mind, however, that this only checks that there is a signature, not
that the signer is who you expect it to be. For optimal security, you
should manually check the signature once with `docker trust inspect
--pretty` and then pin the image hash rather than relying on tags.
The expected output of the `docker sign inspect` command should
mention a signer named `automation` with a public key ID matching
533a6e09faa512f974f217668580da1ceb6aa5b00aad34ea1240afc7d249703f
(note that the `--pretty` output only shows the first 12 chars) and a
repository key matching
f5dc2aee6aed2d05d7eda75db7aa2b3fac7fc67afbb880d03535d5a5295a0d3b.
CHANGELOG_END
This is a preparatory step for moving at least some of the logic of
checking signatures to this script. The reasoning for putting signatures
in the same script basically boils down to "it already has GitHub
pagination".
I also removed the `run.sh` wrapper because it did not add anything
anymore. It used to be useful, but across various changes it's sort of
lost its purpose.
CHANGELOG_BEGIN
CHANGELOG_END
add default machine capability
We semi-regularly need to do work that has the potential to disrupt a
machine's local cache, rendering it broken for other streams of work.
This can include upgrading nix, upgrading Bazel, debugging caching
issues, or anything related to Windows.
Right now we do not have any good solution for these situations. We can
either not do those streams of work, or we can proceed with them and
just accept that all other builds may get affected depending on which
machine they get assigned to. Debugging broken nodes is particularly
tricky as we do not have any way to force a build to run on a given
node.
This PR aims at providing a better alternative by (ab)using an Azure
Pipelines feature called
[capabilities](https://docs.microsoft.com/en-us/azure/devops/pipelines/agents/agents?view=azure-devops&tabs=browser#capabilities).
The idea behind capabilities is that you assign a set of tags to a
machine, and then a job can express its
[demands](https://docs.microsoft.com/en-us/azure/devops/pipelines/process/demands?view=azure-devops&tabs=yaml),
i.e. specify a set of tags machines need to have in order to run it.
Support for this is fairly badly documented. We can gather from the
documentation that a job can specify two things about a capability
(through its `demands`): that a given tag exists, and that a given tag
has an exact specified value. In particular, a job cannot specify that a
capability should _not_ be present, meaning we cannot rely on, say,
adding a "broken" tag to broken machines.
Documentation on how to set capabilities for an agent is basically
nonexistent, but [looking at the
code](https://github.com/microsoft/azure-pipelines-agent/blob/master/src/Microsoft.VisualStudio.Services.Agent/Capabilities/UserCapabilitiesProvider.cs)
indicates that they can be set by using a simple `key=value`-formatted
text file, provided we can find the right place to put this file.
This PR adds this file to our Linux, macOS and Windows node init scripts
to define an `assignment` capability and adds a demand for a `default`
value on each job. From then on, when we hit a case where we want a PR
to run on a specific node, and to prevent other PRs from running on that
node, we can manually override the capability from the Azure UI and
update the demand in the relevant YAML file in the PR.
CHANGELOG_BEGIN
CHANGELOG_END
With the current setup, we always push whatever version GitHub considers
as the latest, which is defined by date. This means that at the moment a
patch release could overwrite a less recent but higher-version release,
essentially downgrading the SDK to a previous, presumably less good user
experience.
This patches the upload process to choose the highest-numbered release
instead of the most recent one by date.
CHANGELOG_BEGIN
CHANGELOG_END
* Publish docker images for snapshot releases as well
We got a couple of requests for this and given the current release
process this seems very reasonable. Snapshot releases are clearly
marked as such so I think there this is fine.
We could think about differentiating between a stable release version
marked as a prerelease and not publish the docker image for that but
given our current release process where the stable version is based on
an already existing snapshot and therefore testing is very unlikely to
fail when we mark it as stable, that doesn’t seem worth the complexity
here.
changelog_begin
changelog_end
* Update azure-cron.yml
Co-Authored-By: Samir Talwar <samir.talwar@digitalasset.com>
Co-authored-by: Samir Talwar <samir.talwar@digitalasset.com>
One of the outputs of our brainstorming about how to make CI better was
that it is annoying to have to "babysit" pull requests. This PR attempts
to introduce a notification mechanism by which Azure will notify people
on Slack when a build finishes, so they know they need to go and rerun
or merge the corresponding PR.
This commit also changes the existing $Slack.URL variable to
$Slack.team-daml, to make more explicit where the Slack message is being
sent to (Slack works with one token per destination channel). Both
$Slack.URL and $Slack.team-daml are currently defined as the same token
in Azure.
CHANGELOG_BEGIN
CHANGELOG_END
Currently, when cron jobs fail, this is completely silent. After #4178,
in which we realized the VSCode extension had not been updated for about
a month, I believe it may be a good idea to get error messages sent to
Slack when a cron job fails.
We'll need to monitor for verbosity, but hopefully they will work most
of the time.
CHANGELOG_BEGIN
CHANGELOG_END
This has changed in 0a26591849 which
broke the build. We only build the latest release so changing this
should be safe and not break older releases.
changelog_begin
changelog_end
changelog_begin
- [DAML SDK] Docker images for this release and releases in the future
are built using the Dockerfile of the corresponding git tag and are
therefore stable. Previously, they were updated whenever the
Dockerfile changed.
changelog_end
This commit aims at mitigating two issues we have noticed with the
0.13.41 release:
1. The initial cron run for that release got interrupted at the 50
minutes mark, which happened to be right in the middle of the s3 upload.
This means it had already changed the versions.json file, but had not
finished updating the actual html files. Right now, the docs.daml.com
website shows version 0.13.41 in the drop-down, but actually displays
the content for 0.13.40. Additionally, trying to explicitly visit the
website for 0.13.41 (https://docs.daml.com/0.13.41) yields a 404. Note
that this also means the cron job did not reach the "tell HubSpot"
point, so 0.13.41 did not get announced.
2. As the script also did not reach the "clear cache" step, subsequent
runs have been rebuilding the documentation for no reason as the
sequence of steps was: check versions.json through HTTP, get cached one,
see it's not up-to-date, build docs, check versions.json through s3 API,
bypassing the cache, see it's up-to-date, stop.
To address those issues, this PR changes the cron to:
1. Increase the timeout to 2h instead of 50 minutes.
2. Always check the versions.json file through s3, rather than go
through the HTTP cache first.
These are not complete solutions but I'm not sure how to do better given
that s3 does not have atomic operations.
