* fix blackduck
BlackDuck is currently failing to run because it creates multiple report
files and then expects a single one. This is due to #16552 changing only
one of the output file names.
This PR resolves this by:
1. Changing the name everywhere, so all runs aggregate their results on
the same file.
2. Changing the PR step code to expect the given file name, rather than
try and glob it.
My understanding of the BlackDuck setup is very fuzzy so please
double-check carefully.
See #16600 for an example of what the update would be from this PR.
* Update daily-compat.yml
main instead of local-main-maven
---------
Co-authored-by: Brian Healey <brian.healey@digitalasset.com>
From output of currently failed jobs:
```
The key 'detect.npm.dependency.types.excluded' in property source
'commandLineArgs' contained a value that could not be reasonably
converted to the properties type. The exception was: Unable to parse raw
value 'NON_PRODUCTION' and coerce it into type 'NoneEnum or
NpmDependencyType'. Value was must be one of NONE,DEV,PEER
```
It looks like I was a bit overeager on #14995.
CHANGELOG_BEGIN
CHANGELOG_END
- The `check_releases` job seems to take between 4.5h and `TIMEOUT`
these days, with no clear cause for the large span in running time.
Hopefully bumping the limit from 4.5 to 8 hours will give us some
breathing room (this is not by any means a long-term solution, I'm well
aware of that).
- I couldn't find anybody who seemed to care about running the
create-daml-app compatibility tests on macOS, and they're super flaky
(typically requiring 5 reruns to succeed).
CHANGELOG_BEGIN
CHANGELOG_END
run-full-compat: true
After discussion with the team, we've reached the conclusion that the
JSON API is not changing enough for these performance tests to bring
much value: we're not in a situation where we think there's a high risk
of performance regressions, nor trying to focus on performance
improvements.
Note that this is only removing the tests from the daily run; the team
does see value in keeping the code for these tests around in case any
performance-focused work does arise in the future.
(This PR also disables running the `check_releases` job outside of main
branch runs, because there's really never been a good reason to.)
Fixes#14608.
CHANGELOG_BEGIN
CHANGELOG_END
run-full-compat: true
Since yesterday, this test is failing with an OOM error. I hoped I'd
have time to investigate today, but since I didn't and I'm off next week
disabling the test for now seems like the best option to let the overall
daily tests succeed.
The last known success was on 0bf28176a7,
and the first failure was on 444fff1fcf.
We don't test every commit, so any commit in-between could be the cause
of the issue.
Note that the failure does not seem to be flaky, but further testing may
be warranted.
CHANGELOG_BEGIN
CHANGELOG_END
There were two issues:
1. The `Directory.listFilesRecursive` function returns paths that
include its first argument, so there was never any overlap between
the sets.
2. That full path doesn't exist on AWS, and was given to AWS because
`s3Path opts f` returns just `f` if it starts with `/`.
This means that, on the one hand, the script tried to delete
_everything_ from the top-level, which is a bit bad, but, on the other
hand, the script did not actually manage to delete anything at all,
which is basically the status quo.
Note that deleting everything isn't _that_ bad as we reupload right
after and we have a cache in front on the site, so this would only
impact users if they were the first to try and view a given page in that
small time window between deletion and reupload.
This PR should fix both issues by removing the version-specific prefix.
CHANGELOG_BEGIN
CHANGELOG_END
There's a small, subtle bug that made its way into #14287: apparently
`/dev/stdout` is not defined on CI.
On the bright side, we don't actually need the `out` parameter, so
removing it seems like the easiest fix here.
CHANGELOG_BEGIN
CHANGELOG_END
* Merge ci/cron/wednesday.yml open_pr function into ci/bash-lib.yml
* remove unused function reset from ci/cron/wednesday.yml
* Request review from current release person for rotation PR
changelog_begin
changelog_end
The check_releases job has been a major player in the flakiness of the
daily test lately, simply by _timing out_ despite its 6h limit.
There are smarter, more "permanent" fixes we could implement here, but
as a quick stopgap measure I wanted to try out how much faster we would
go if we didn't need to reestablish a GCloud identity for each file.
CHANGELOG_BEGIN
CHANGELOG_END
run-full-compat: true
After yet another report of a ghost file, I stared at the code even
harder than before and finally figured out what was going wrong (or at
least one way it was going wrong): `listDirectory` is not recursive, so
the current code would only delete top-level files.
Hopefully this fixes that. I'll try to do some manual cleanup too.
CHANGELOG_BEGIN
CHANGELOG_END
With the new 2.0.0 release process, we're not releasing from the main CI
build anymore, so the compat update PRs are not getting created.
This restores that by making those PR creations part of the daily CI
build.
CHANGELOG_BEGIN
CHANGELOG_END
- Add notifications so I don't miss te updates.
- Add explicit `fetch` so we can detect if the same branch already
exists.
- Run from `main` so we geet a fresh view in case of reruns.
CHANGELOG_BEGIN
CHANGELOG_END
After 2.0, the docs need to be built alongside Canton files, and that is
taken care of by the [docs.daml.com] repo.
[docs.daml.com]: https://github.com/digital-asset/docs.daml.com
CHANGELOG_BEGIN
CHANGELOG_END
Process:
- `git ls-files | xargs sed -i 's/DAML/Daml/g'`
- `git add -p`
To review:
- Check for false positives by carefully reviewing the diff in this PR.
- Check for false negatives with `git grep DAML`.
CHANGELOG_BEGIN
- Removed all traces of DAML. This may affect error messages and debug
outputs.
CHANGELOG_END
* Don't error when user tokens are provided for the perf runner
changelog_begin
changelog_end
* Better token parsing error handling & extract user id to allocate the user if so
* make the code prettier
* Fix & simplify token parsing
* fix formatting of bazel file
* Update ledger-service/http-json-perf/src/main/scala/com/daml/http/perf/Main.scala
Co-authored-by: Raymond Roestenburg <98821776+ray-roestenburg-da@users.noreply.github.com>
* correctly handle the response of createUser & don't throw if no LedgerId was found in the token
* Wrap exceptions in Futures
* Refactor Main.scala of the perf runner completely & remove dependency on user provided JWT's
* Minimize diff
* simplify code further
* Update ledger-service/http-json-perf/src/main/scala/com/daml/http/perf/Main.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update ledger-service/http-json-perf/src/main/scala/com/daml/http/perf/Main.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Update ledger-service/http-json-perf/src/main/scala/com/daml/http/perf/Main.scala
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
* Fix build
* Don't pass a jwt to the perf runner anymore everywhere it was used & fix ledger id to be right
* Minimize diff
Co-authored-by: Raymond Roestenburg <98821776+ray-roestenburg-da@users.noreply.github.com>
Co-authored-by: Stefano Baghino <43749967+stefanobaghino-da@users.noreply.github.com>
This should hopefully get CI working again.
There are two changes in here:
1. We can no longer change our patched Bazel. I didn’t switch away
from the current patched version for now (we upload it to gcp bucket
so it still works fine even if we cannot build it) but if we upgrade,
we need to go to an unpatched version for now.
2. We need to get `az` from dev-env. I tested the self service compat
job stuff and it works fine with this but there is a chance other
parts don’t.
changelog_begin
changelog_end
Most of this is just documentation. The actual idea here is fairly
simple:
- The assembly repo pushes to S3 on each release (not yet implemented).
- The cronjob skips releases that are already there.
- Updating the top-level just piggybacks on what we already have.
changelog_begin
changelog_end
This follows the approach we used for the BazelCache stuff and splits
the 3 different operations in 3 separate modules + one Github utility
module to make it a bit easier to follow.
This is pure reshuffling, no functional change.
changelog_begin
changelog_end
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
* revert to use synopsys detect 7.1.0 run-full-compat: true
CHANGELOG_BEGIN
CHANGELOG_END
* run on branch run full compat
run-full-compat: true
* run on branch run full compat
run-full-compat: true
* revert changes to run branch
Reminds people adding new users to the rotation that they should also
be added to DACH-NY/daml-language-ad-hoc to have a box in the cloud
for Windows testing.
Adds the possibility of adding comments at the beginning of the rotation file.
changelog_begin
changelog_end
We believe the Blakduck logic is currently faulty. We have had a
violation on an NPM dependency, and Blackduck keeps reporting it despite
our having removed the dependency.
We believe that what is happening is that, in the first step of
checking, we udpate the Haskell dependencies, _and then check the
validity of the whole project_, which includes the NPM deps. Because
that fails, we never get to the step where we actually update the NPM
deps, and Blackduck is stuck forever.
The solution is to not fail on violations for the Haskell update steps.
Haskell deps are still checked in the second step, because, again, it is
checking the whole project.
CHANGELOG_BEGIN
CHANGELOG_END
