This commit changes the docs cron to create a new file
`docs.daml.com/latest`, a simple text file containing the version number
of the latest released version. This is done in response to #8354, to
avoid having to replicate the logic for which version is the latest
across this, the assistant and the get-daml.sh script.
This commit also cleans up a small transitional FIXME in the docs cron
regarding the transition from `snapshots.json` to `hidden.json`.
For the solution to #8354 to be complete, we'll also need to update
get-daml.sh and the assistant to use the new latest file. Note that this
file would only be published on the next stable release, so this commit
also includes a temporary hack to re-generate it (and `versions.json`
and `hidden.json`) unconditionally on every run; this can be removed as
soon as this has run once.
CHANGELOG_BEGIN
CHANGELOG_END
As we strive for more inclusiveness, we are becoming less comfortable
with historically-charged terms being used in our everyday work.
This is targeted for merge on Dec 26, _after_ the necessary
corresponding changes at both the GitHub and Azure Pipelines levels.
CHANGELOG_BEGIN
- DAML Connect development is now conducted from the `main` branch,
rather than the `master` one. If you had any dependency on the
digital-asset/daml repository, you will need to update this parameter.
CHANGELOG_END
Turns out bash is hard and I’m stupid :sadpanda:
We need to write thoutput to stderr, otherwise this ends up in the
JSON output which obviously is not valid JSON.
changelog_begin
changelog_end
We changed the patch to target more than one file which made the
checkout insufficient for restoring the state and then the following
git checkout of current fails with:
```
error: Your local changes to the following files would be overwritten by checkout:
stack-snapshot.yaml
```
A git reset --hard should make sure everything gets reset.
changelog_begin
changelog_end
Today the [perf check failed], but we got no notification of it. I'm not
sure what's happening as I can't reproduce any of it locally: not only
does the `bazel run` command work for me (despite the ghc-lib URL
returning a 404 when I try it manually), I also can't reproduce the fact
that Bash, on CI, doesn't seem to fail on either the `bazel run` error
or the fact that on the next line `cat` tries to access a file that
doesn't exist (for which CI does print the error message).
This PR does two things:
- Add an explicit check that _should_ get Bash to actually fail should
this happen again in the future. It is not a great fix but at least
we'll know if it happens again (to the best of my knowledge today was
the first time we hit this).
- Amend the existing patch we apply on the baseline commit to use the
GCS-hosted ghc-lib packages.
CHANGELOG_BEGIN
CHANGELOG_END
[perf check failed]: https://dev.azure.com/digitalasset/daml/_build/results?buildId=64395&view=results
* Run bazel scan for all but haskell first, then haskell at end so they do not stomp on each other
CHANGELOG_BEGIN
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* use common prefix between runs so results are aggregated, get branch name from running job rather than assuming master
* DO NOT MERGE: disable all jobs except for blackduck scan
* haskell before full scan
* parens instead of braces
* differentiate haskell and jvm bazel runs with unique code location
* unique code location prefix
* fix syntax
* unique code location to avoid clashing bazel runs
* Use master security-blackduck script helper
* reenable all jobs to make mergeable
* cleanup whitespace
* Use Build.SourceBranchName for branch
* Update ci/cron/daily-compat.yml
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* DO NOT MERGE: skip all jobs but blackduck, skip update notices file step
* reenable all jobs to make mergeable
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Unfortunately, I missed the fact that we had our own logic for
handling process failures which resulted in uncatchable
exceptions. I’ve changed one place to use the upstream handling and
the other to call `fail` which throws an IOException like I would have
expected.
changelog_begin
changelog_end
The change in #8191 to publish daml on sql docs failed because the
versions.json and snapshots.json files don’t exist initially. This PR
fixes that by catching the exception and treating it as an empty file.
changelog_begin
changelog_end
* add blackduck scan to run on master (#6130)
* add blackduck scan
* disable go scanning
exclude entire language-support/ts directory for node scanning
break to multiple lines to make command line params easier to parse
* Increase timeout for blackduck binary scan
* update blackduck scan config
* remove some exclusions, force python3
* exclude GO until path to go executable can be resolved
* added readme explanation of why we want this file
* fail in case of policy violation
* ensure haskell bazel scan completes before running second round scan for bazel jvm and node and other langs
* trigger notices file gen to ensure BOM complete
* remove trailing end of lines
* run with latest detect version and unique code location name changes to wrapper script
* Add blackduck to daily compat job
* DO NOT MERGE: condition false to disable other jobs for testing
* remove parameters not available to cronjob
* Revert changes to regular CI pipeline
CHANGELOG_BEGIN
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* Do not get branch name from variable
* Upgrade com.fasterxml.jackson.core:jackson-databind to 2.12.0 to address security vulnerability
* Remove disabling of other jobs, set to branch to be used on prod runs
* Apply suggestions from code review
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Address code review comments
* Updated NOTICES file
* Run bazel build, update NOTICES file
* Correct dade-assist
* do not have perms to pipe to dev/null
* Add md file explaining how to update NOTICES file
* Add instructions for running blackduck locally
* Add a link to full security-blackduck readme
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
I wanted to suggest that on the PR but caught it after it was merged. So
I made a note of it, which promptly got lost. As the end of the year
approaches I've started trying to clean up a little.
CHANGELOG_BEGIN
CHANGELOG_END
Having the cron push artifacts to GCP was really only meant to happen
once. I got distracted and worked on other things. This PR closes that
work loop such that the current state and expectations are:
- Every new release pushes to GCP as part of the release process.
- The cron only checks that the GCP backup exists and matches, but does
not push if it doesn't.
The reason for this is we want the cron job to fail if there are
additional, unexpected files in a release, rather than automatically
commit those files for the long term.
CHANGELOG_BEGIN
CHANGELOG_END
This is the macOS part of #5912, which I have separated because our
macOS nodes have a different deployment process so it seemed easier to
track the deployment of the change separately.
CHANGELOG_BEGIN
CHANGELOG_END
GitHub requests that when we use the API without a token, which is what
we do here, we use a user-agent header that allows them to contact us in
case there is an issue they need to discuss. This PR updates that
address, and cleans-up a bit of duplication around it.
CHANGELOG_BEGIN
CHANGELOG_END
This PR allows the script to run without GCP credentials. It will
obviously then skip the bits that require GCP credentials, but that
still leaves it with plenty of things to do.
Because checking all releases can still be quite long (around an hour on
CI, and my personal connection is a bit slower), this also introduces a
new parameter that restricts the number of releases to test.
CHANGELOG_BEGIN
CHANGELOG_END
Currently the return code of the function is the return code of the
`eval "$restore_trap"` line, whereas semantically we want the return
code of the `gsutil` call. This is not an issue in most cases as the
`set -e` should kick in, but if the function appears as the condition in
an `if` statement the `-e` flag is suspended.
The main use-case right now is that the daily license check is _not_
uploading artifacts.
CHANGELOG_BEGIN
CHANGELOG_END
Change the path used to push to the backup gcs bucket to match what is
put by the release script. This needs to get merged before we run the
next daily.
CHANGELOG_BEGIN
CHANGELOG_END
If we don't already have a copy of an artifact in our "disaster
recovery" storage box, put one.
Note: as implemented, this upload mechanism happens only if the release
was successfully verified signature-wise, so this should not result in
us saving broken artifacts. Also, CI does not have deletion or overwrite
access to this bucket, so overall this should be pretty safe.
CHANGELOG_BEGIN
CHANGELOG_END
This PR extends the existing `save_gcp_data` function to handle any
`gsutil` command. This is done to support existence checking using
`gsutil ls` for private artifacts in release checking (`ci/cron`).
CHANGELOG_BEGIN
CHANGELOG_END
At the moment, because the signature check appears in a `if` statement,
failed signatures do not actually fail the script and would thus still
result in "success" messages to Slack.
CHANGELOG_BEGIN
CHANGELOG_END
Yes, this is how I write Haskell. I'm told it's an improvement over
Bash.
Jokes aside, plan is to chip away at the Bash script, starting with
replacing the outermost loop with a proper "get _all_ releases" call
from Haskell, but I like keeping things working in small steps, and even
long-term I have no desire to reimplement the gpg signature checking
code in Haskell.
I have tested that things still work (on my machine); the only
difference is that we now only get the full output all at once at the
end, rather than one signature at a time. I don't think anyone is
looking at the output in real-time, so this should not be a huge issue.
CHANGELOG_BEGIN
CHANGELOG_END
This is a preparatory step for moving at least some of the logic of
checking signatures to this script. The reasoning for putting signatures
in the same script basically boils down to "it already has GitHub
pagination".
I also removed the `run.sh` wrapper because it did not add anything
anymore. It used to be useful, but across various changes it's sort of
lost its purpose.
CHANGELOG_BEGIN
CHANGELOG_END
This is a first, very incomplete step in the spirit of small,
incremental PRs. Known missing features:
- Should check all versions, not just the 30 most recent ones.
- Should also download from GCP backup and compare.
- Should alert on Slack if anything is unexpected.
- Should handle versions prior to us starting to sign (and do what?).
- Should also check artifacts in Artifactory, not just GitHub Releases.
- Optionally should save to GCP if we don't have a backup already.
So at the moment it's just downloading the artifacts for the 30 most
recent releases and printing a message stating whether we have a
signature and whether it's valid.
CHANGELOG_BEGIN
CHANGELOG_END
* Remove reference to release-notes.rst
https://github.com/digital-asset/daml/pull/7458 shuffled this
around. While we could update it, it doesn’t really make any sense. We
post our release notes to the blog now and not in the docs so this
whole checkout procedure is redundant. This is also true if we wanted
to make a bugfix release for a release < 1.5 where this file still
existed. The trigger_sha is always on master (following our current
release process) so the file would still not exist.
I did also remove it from the docs cronjob. We never reupload old docs
so this doesn’t make a difference.
changelog_begin
changelog_end
* Stupid whitespace change because windows is pissing me off
changelog_begin
changelog_end
* Add http-json-perf daily cron job
changelog_begin
changelog_end
* commenting out other jobs so we can manually test the new one
* commenting out other jobs so we can manually test the new one
* Fix the shell script
* Fixing the gs bucket, `gs://http-json bucket does not exist`
* uncomment the other jobs
* timestamp from git log
* get rid of DAR copying
* comment out the other jobs, so we can test it
* uncomment the other jobs
It's been pointed out to me that some people actually plan their work,
and for them, it would be useful to know at least a day in advance that
they will be doing a release. This PR attempts to accommodate that.
Note: this will run as a new, separate cron. I'll do the Azure setup
after this gets approved.
CHANGELOG_BEGIN
CHANGELOG_END
On the last release, the job succeeded despite no being able to create
the compat PR. This fixes:
- The curl call to actually return non-0 on non-2xx HTTP response.
- The way in which we encode the credentials.
This also attempts to create a Bash library, hopefully this time in a
way that doesn't get destroyed by our release process. IIUC pipeline
instructions (YAML files) are all parsed and read before any execution,
so by embedding the Bash library in a template we should get the correct
version (i.e. the one that is running the pipeline) even when checking
out other commits.
CHANGELOG_BEGIN
CHANGELOG_END
