* Run bazel scan for all but haskell first, then haskell at end so they do not stomp on each other
CHANGELOG_BEGIN
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* use common prefix between runs so results are aggregated, get branch name from running job rather than assuming master
* DO NOT MERGE: disable all jobs except for blackduck scan
* haskell before full scan
* parens instead of braces
* differentiate haskell and jvm bazel runs with unique code location
* unique code location prefix
* fix syntax
* unique code location to avoid clashing bazel runs
* Use master security-blackduck script helper
* reenable all jobs to make mergeable
* cleanup whitespace
* Use Build.SourceBranchName for branch
* Update ci/cron/daily-compat.yml
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* DO NOT MERGE: skip all jobs but blackduck, skip update notices file step
* reenable all jobs to make mergeable
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
Unfortunately, I missed the fact that we had our own logic for
handling process failures which resulted in uncatchable
exceptions. I’ve changed one place to use the upstream handling and
the other to call `fail` which throws an IOException like I would have
expected.
changelog_begin
changelog_end
The change in #8191 to publish daml on sql docs failed because the
versions.json and snapshots.json files don’t exist initially. This PR
fixes that by catching the exception and treating it as an empty file.
changelog_begin
changelog_end
* add blackduck scan to run on master (#6130)
* add blackduck scan
* disable go scanning
exclude entire language-support/ts directory for node scanning
break to multiple lines to make command line params easier to parse
* Increase timeout for blackduck binary scan
* update blackduck scan config
* remove some exclusions, force python3
* exclude GO until path to go executable can be resolved
* added readme explanation of why we want this file
* fail in case of policy violation
* ensure haskell bazel scan completes before running second round scan for bazel jvm and node and other langs
* trigger notices file gen to ensure BOM complete
* remove trailing end of lines
* run with latest detect version and unique code location name changes to wrapper script
* Add blackduck to daily compat job
* DO NOT MERGE: condition false to disable other jobs for testing
* remove parameters not available to cronjob
* Revert changes to regular CI pipeline
CHANGELOG_BEGIN
CHANGELOG_END
Signed-off-by: Brian Healey <brian.healey@digitalasset.com>
* Do not get branch name from variable
* Upgrade com.fasterxml.jackson.core:jackson-databind to 2.12.0 to address security vulnerability
* Remove disabling of other jobs, set to branch to be used on prod runs
* Apply suggestions from code review
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
* Address code review comments
* Updated NOTICES file
* Run bazel build, update NOTICES file
* Correct dade-assist
* do not have perms to pipe to dev/null
* Add md file explaining how to update NOTICES file
* Add instructions for running blackduck locally
* Add a link to full security-blackduck readme
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
The current implementation of the notify_user job sometimes reports
success while the build has actually failed. Azure does not provide a
way to query the overall state of the current build, so a general
solution to this problem does not seem possible (see #6796 for an
example of a failed attempt). However, all reported cases were
specifically about the `check_changelog_entry` job, which we can easily
query for, so this PR does that.
Note: originally pushed without a changelog entry to test new
notification mechanism.
CHANGELOG_BEGIN
CHANGELOG_END
* update ghc-lib patch
* update stack-snapshot.yaml
changelog_begin
changelog_end
* bazel run @stackage-unpinned//:pin (on linux) and update stackage_snapshot.json
* testcase to check type signatures in template-lets are not ignored
* update to ghc-lib commit hash following merge
* update snapshot
* update snapshot after pin on windows
* update stack pin on linux
Co-authored-by: Nick Chapman <nchapman@digitalasset.com>
I wanted to suggest that on the PR but caught it after it was merged. So
I made a note of it, which promptly got lost. As the end of the year
approaches I've started trying to clean up a little.
CHANGELOG_BEGIN
CHANGELOG_END
Having the cron push artifacts to GCP was really only meant to happen
once. I got distracted and worked on other things. This PR closes that
work loop such that the current state and expectations are:
- Every new release pushes to GCP as part of the release process.
- The cron only checks that the GCP backup exists and matches, but does
not push if it doesn't.
The reason for this is we want the cron job to fail if there are
additional, unexpected files in a release, rather than automatically
commit those files for the long term.
CHANGELOG_BEGIN
CHANGELOG_END
This is the macOS part of #5912, which I have separated because our
macOS nodes have a different deployment process so it seemed easier to
track the deployment of the change separately.
CHANGELOG_BEGIN
CHANGELOG_END
* Adapt to new desugaring for choice observers.
update hash of ghc patch.
changelog_begin
changelog_end
update ghc patch to final version
update stack-snapshot hashes for ghc-lib(-parser)
update stackage_snapshot.json, following `bazel run @stackage-unpinned//:pin`
expose Optional constructors for desugared code to use
adapt LFConversion to expect a 4-tuple for a desugared choice def/sig
update LFConversion for choice-observers
first example using new choice observer syntax.
fix scala type checker to have correct scoping rules for choice-observers
remove comment from example which says it is broken
improve variable names
improve tests for choice-observer clause
only test choice-observers SINCE 1.dev
add jq queries for choice observeres
make positive statement in jq test which checks choice obserers are present
test behaviour of choice observers
squash me
typo
* test more choice-observer divulgence
* Update documention for choice observers.
changelog_begin
Support choice observers in 1.dev
changelog_end
* fix docs build
* fix daml docs choice-observers example
* address comments: rewording text
* annotate choices observers as early-access in documention
* split out documentation code-snippets which require --target=1.dev
* final tweaks to documentation text
Note: this file is meant to represent the content on [Docker Hub], but
syncing is currently a manual process. I will propagate once this is
approved and merged.
[Docker Hub]: https://hub.docker.com/repository/docker/digitalasset/daml-sdk
CHANGELOG_BEGIN
CHANGELOG_END
GitHub requests that when we use the API without a token, which is what
we do here, we use a user-agent header that allows them to contact us in
case there is an issue they need to discuss. This PR updates that
address, and cleans-up a bit of duplication around it.
CHANGELOG_BEGIN
CHANGELOG_END
This PR allows the script to run without GCP credentials. It will
obviously then skip the bits that require GCP credentials, but that
still leaves it with plenty of things to do.
Because checking all releases can still be quite long (around an hour on
CI, and my personal connection is a bit slower), this also introduces a
new parameter that restricts the number of releases to test.
CHANGELOG_BEGIN
CHANGELOG_END
Currently the return code of the function is the return code of the
`eval "$restore_trap"` line, whereas semantically we want the return
code of the `gsutil` call. This is not an issue in most cases as the
`set -e` should kick in, but if the function appears as the condition in
an `if` statement the `-e` flag is suspended.
The main use-case right now is that the daily license check is _not_
uploading artifacts.
CHANGELOG_BEGIN
CHANGELOG_END
Change the path used to push to the backup gcs bucket to match what is
put by the release script. This needs to get merged before we run the
next daily.
CHANGELOG_BEGIN
CHANGELOG_END
If we don't already have a copy of an artifact in our "disaster
recovery" storage box, put one.
Note: as implemented, this upload mechanism happens only if the release
was successfully verified signature-wise, so this should not result in
us saving broken artifacts. Also, CI does not have deletion or overwrite
access to this bucket, so overall this should be pretty safe.
CHANGELOG_BEGIN
CHANGELOG_END
This PR extends the existing `save_gcp_data` function to handle any
`gsutil` command. This is done to support existence checking using
`gsutil ls` for private artifacts in release checking (`ci/cron`).
CHANGELOG_BEGIN
CHANGELOG_END
At the moment, because the signature check appears in a `if` statement,
failed signatures do not actually fail the script and would thus still
result in "success" messages to Slack.
CHANGELOG_BEGIN
CHANGELOG_END
Yes, this is how I write Haskell. I'm told it's an improvement over
Bash.
Jokes aside, plan is to chip away at the Bash script, starting with
replacing the outermost loop with a proper "get _all_ releases" call
from Haskell, but I like keeping things working in small steps, and even
long-term I have no desire to reimplement the gpg signature checking
code in Haskell.
I have tested that things still work (on my machine); the only
difference is that we now only get the full output all at once at the
end, rather than one signature at a time. I don't think anyone is
looking at the output in real-time, so this should not be a huge issue.
CHANGELOG_BEGIN
CHANGELOG_END
This is a preparatory step for moving at least some of the logic of
checking signatures to this script. The reasoning for putting signatures
in the same script basically boils down to "it already has GitHub
pagination".
I also removed the `run.sh` wrapper because it did not add anything
anymore. It used to be useful, but across various changes it's sort of
lost its purpose.
CHANGELOG_BEGIN
CHANGELOG_END
This is a first, very incomplete step in the spirit of small,
incremental PRs. Known missing features:
- Should check all versions, not just the 30 most recent ones.
- Should also download from GCP backup and compare.
- Should alert on Slack if anything is unexpected.
- Should handle versions prior to us starting to sign (and do what?).
- Should also check artifacts in Artifactory, not just GitHub Releases.
- Optionally should save to GCP if we don't have a backup already.
So at the moment it's just downloading the artifacts for the 30 most
recent releases and printing a message stating whether we have a
signature and whether it's valid.
CHANGELOG_BEGIN
CHANGELOG_END
This does not get used very often so it is likely nobody will remember
how it works when we do use it. It's And due to the ordering Azure makes
of jobs in its UI, it's very easy to miss that there is a final,
Linux-based step and the values are actually printed there.
So this adds a little note to remind us of that.
Note that as this changes the `ci/patch_bazel_windows` folder, this will
also generate a new Bazel, so this PR will also update the Scoop
reference.
CHANGELOG_BEGIN
CHANGELOG_END
* Remove reference to release-notes.rst
https://github.com/digital-asset/daml/pull/7458 shuffled this
around. While we could update it, it doesn’t really make any sense. We
post our release notes to the blog now and not in the docs so this
whole checkout procedure is redundant. This is also true if we wanted
to make a bugfix release for a release < 1.5 where this file still
existed. The trigger_sha is always on master (following our current
release process) so the file would still not exist.
I did also remove it from the docs cronjob. We never reupload old docs
so this doesn’t make a difference.
changelog_begin
changelog_end
* Stupid whitespace change because windows is pissing me off
changelog_begin
changelog_end
At the moment, with the `curl` call comfortably nested inside an `if`
statement, `curl` failures are interpreted as simply going down the
`else` branch of the conditional. Hoisting it up to a separate variable
declaration makes the script fail on the curl call itself.
CHANGELOG_BEGIN
CHANGELOG_END
* Add http-json-perf daily cron job
changelog_begin
changelog_end
* commenting out other jobs so we can manually test the new one
* commenting out other jobs so we can manually test the new one
* Fix the shell script
* Fixing the gs bucket, `gs://http-json bucket does not exist`
* uncomment the other jobs
* timestamp from git log
* get rid of DAR copying
* comment out the other jobs, so we can test it
* uncomment the other jobs
In #7386 I inadvertently removed the default value for the argument
`check-changelog.sh` takes. This did not break CI because CI always
passes the argument explicitly, but it did break local workflows for
some developers.
Apologies.
CHANGELOG_BEGIN
CHANGELOG_END
