We believe the Blakduck logic is currently faulty. We have had a
violation on an NPM dependency, and Blackduck keeps reporting it despite
our having removed the dependency.
We believe that what is happening is that, in the first step of
checking, we udpate the Haskell dependencies, _and then check the
validity of the whole project_, which includes the NPM deps. Because
that fails, we never get to the step where we actually update the NPM
deps, and Blackduck is stuck forever.
The solution is to not fail on violations for the Haskell update steps.
Haskell deps are still checked in the second step, because, again, it is
checking the whole project.
CHANGELOG_BEGIN
CHANGELOG_END
* Add avg, stddev, p90, p99, requests_per_second numbers to be reported on slack similar to speedy_perf
changelog_begin
changelog_end
* changes based on code review
* fix failing job due to breaking function export
* vanilla job test on main pipeline
changelog_begin
changelog_end
* move job to daily compat tests
* add timeout to dev-env and changes based on code review
Dropping the strict source name patch has resulted in empty
directories which Bazel is happy about but then fail at
runtime. They’re not quite empty, the digest points to 0x0a 0x00
so we match on that.
changelog_begin
changelog_end
This PR drops two things:
1. The check that the benchmark hasn’t been modified. This hasn’t ever
been useful and it keeps being annoying.
2. It stops the comparison against the old version and instead just
benchmarks the current version. We really only care about the day to
day changes. Comparing against an arbitrary year old version has lost
all meaning at this point.
changelog_begin
changelog_end
The patch that it downloads via Nix was taken from a GH PR instead of a
commit such that the hash is not fully stable. This adds a patch to
download the relevant patch directly from a GitHub commit.
changelog_begin
changelog_end
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
* ci/cron/check: remove dade-assist calls
We can only run this in a context where the dev-env is already set up
anyway, as that's how we get Bazel to build the script in the first
place.
CHANGELOG_BEGIN
CHANGELOG_END
* remove skip_java logic
Two quick improvements I made while waiting on #9039:
- Avoid loading Java. When looking at the logs flow by this seemed to be
taking a huge amount of time.
- Isolate the gcloud config files, which allows for running gcloud
downloads in parallel.
Together these reduce the `check_releases` runtime from about 5 hours to
about 2. There's much more (and smarter) work needed on this, but this
was really easy to do.
CHANGELOG_BEGIN
CHANGELOG_END
* Conduitify download in release file
Apologies, my ocd kicked in when I saw this in another PR.
changelog_begin
changelog_end
* fixup deps
changelog_begin
changelog_end
* i should stop working
changelog_begin
changelog_end
I think the retry is clobbering the files. Here is my theory:
- The HTTP request is lazy, i.e. it starts producing a byte stream
before it has finished downloading.
- The connection somehow crashes in the middle of that lazy handling,
possibly because the Haskell code blocks for too long on something
else and GCP thus closes the connection. (If this is true, making sure
we download the entire thing before we start writing may make the
download more reliable.) This explains why we get a "resource vanished"
and not a plain 404 to start with.
- The retry policy doesn't know anything about HTTP requests; it just
sees an IO action throwing an exception and restarts the whole thing.
- Because the IO action opens the file in Append mode, we thus end up
with a file that is too big and has its "starting bytes" multiple
times. That obviously fails to sign-check.
If this is what happens then the retry does not help at all, which does
seem to be what we've been observing (though I haven't tracked the exact
error rate too closely). The fix would likely be as simple as changing
`IO.AppendMode to IO.WriteMode (which truncates, per [documentation]).
[documentation]: https://hackage.haskell.org/package/base-4.14.1.0/docs/System-IO.html
CHANGELOG_BEGIN
CHANGELOG_END
There is really no reason to first capture this to a String and then
putStrLn it. That only causes issues since if we crash with a non-zero
exitcode we won’t output anything.
changelog_begin
changelog_end
The report currently chokes on quotes in the commit message (see
550aa48fc5). Rather than trying to
correctly excape things in Bash, this PR delegates the quote handling to
jq, because having to deal with Bash embedded in YAML is hard enough.
CHANGELOG_BEGIN
CHANGELOG_END
We’ve seen a number of "resource vanished (connection reset by peer)"
errors. Slapping some retries on that should hopefully make the CI job
a bit more robust.
changelog_begin
changelog_end
* Clean broken entries from the Bazel cache
This is hopefully a somewhat reasonable workaround for the "output not
created" errors that keep annoying us.
For now, this is just part of the hourly cronjob but we could move it
somewhere else if desired.
changelog_begin
changelog_end
* Fix GCS credentials
changelog_begin
changelog_end
CHANGELOG_BEGIN
- Our Linux binaries are now built on Ubuntu 20.04 instead of 16.04. We
do not expect any user-level impact, but please reach out if you
do notice any issue that might be caused by this.
CHANGELOG_END
I discovered yesterday that the `snapshots.json` (and actually also the
`versions.json`) file is no longer purely internal to the docs process,
as it was meant to be, but is now depended upon by the assistant. This
means the renaming from `snapshots.json` to `hidden.json` cannot happen,
and we reverted that yesterday in #8513 (& #8514), though that was done
in a bit of a hurry. This PR aims at cleaning up the resulting mess and
achieve a better long-term end state.
I will be manually removing the `hidden.json` file as soon as this is
merged, so that nothing ends up depending on _that_. There is no
occurrence of `hidden.json` outside this docs cron so hopefully this
works out.
CHANGELOG_BEGIN
CHANGELOG_END
* Update both hidden.json and snapshots.json
The assistant relies on the latter, our docs cronjob on the former. I
have no idea why we have two but keeping them in sync should be fine.
changelog_begin
changelog_end
* maybe I should test if my code compiles before pushing
changelog_begin
changelog_end
For a couple weeks now there has been a warning on the Azure Pipelines
web UI that says `ubuntu-latest` is in the process of switching from
18.04 to 20.04. I am not aware of any specific issue this would cause
for our particular workflows, but I don't like my dependencies changing
from under me.
CHANGELOG_BEGIN
CHANGELOG_END
