When no service account is explicitly selected, GCP provides a default
one, which happens to have way more access rights than we're comfortable
with. I'm not quite sure how the total lack of a service account slipped
through here, but I've noticed today so I'm changing it.
CHANGELOG_BEGIN
CHANGELOG_END
As the title suggests. We already disable all communication between CI
nodes through network rules, but we currently get a lot of noise from
GCP logging violations to those rules from Windows trying to feel its
way out for file share buddies.
CHANGELOG_BEGIN
CHANGELOG_END
As usualy, this branch will contain intermediate commits that may serve
as an audit log of sorts.
Our Terraform configuration has been slightly broken by two recent
changes:
- The nixpkgs upgrade in #12280 means a new version of our GCP plugin
for Terraform, which as a breaking change added a required argument to
`google_project_iam_member`. The new version also results in a number
of smaller changes in the way Terraform handles default arguments, which
doesn't result in any changes to our configuration files or to the
behaviour of our deployed infrastructure, but does require re-syncing
the Terraform state (by running `terraform apply`, which would
essentially be a no-op if it were not for the next bullet point).
- The nix configuration changes in #12265 have changed the Linux CI
nodes configuration but have not been deployed yet.
This PR is an audit log of the steps taken to rectfy those and bring us
back to a state where our deployed configuration and our recorded
Terraform state both agree with our current `main` branch tip.
CHANGELOG_BEGIN
CHANGELOG_END
* devenv: Factor out a function to get the Nix version.
* devenv: On newer versions of Nix, use `NIX_USER_CONF_FILES`.
This stacks, rather than overwrites.
* devenv: Append Nix caches, instead of overwriting them.
The "extra-" prefix tells Nix to append.
We also switch to non-deprecated configuration keys.
CHANGELOG_BEGIN
CHANGELOG_END
* devenv: Just require Nix v2.4 or newer.
* devenv: `NIX_USER_CONF_FILES` may not be set already.
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
The cluster shut down today. I'm still not sure why this happens, but
cycling the entire cluster seems to solve it so here goes.
CHANGELOG_BEGIN
CHANGELOG_END
As part of the 2.4 release, the Nix installer has been changed to take
care of the volume setup (which we don't want it to do here). Because
that requires root access, they've decided to make multi-user install
the default, and to disable single-user install.
We could do an in-depth review of the difference and adapt our setup to
use a multi-user setup (we do use the multi-user setup on Linux, so
there's precedent), but as an immediate fix, we can keep our single-user
setup _and_ get the latest Nix by using the 2.3.16 installer and then
upgrading from within Nix itself. This _should_ keep working at least
for a while, as Linux still defaults to single-user.
CHANGELOG_BEGIN
CHANGELOG_END
Hoogle has been down for at least 24h accoridng to user reports. What
seems to be happening is that our nixpkgs pinning is not taking effect,
and the nixpkgs version of Hoogle already includes the patch we are
trying to add. This confuses nix, which fails, and thus the boot
sequence is broken.
I've applied the minimal possible patch here (i.e. enforce the pin),
which gets things running again. I've already deployed this change.
We may want to look at bumping the nixpkgs snapshot.
CHANGELOG_BEGIN
CHANGELOG_END
This fixes an issue where we try to upload a single JSON blob that is
bigger than the limit of 500MB. We could also raise the limit, I guess,
but that seems more error-prone.
CHANGELOG_BEGIN
CHANGELOG_END
If a job fails at build time, there are not test logs to process.
Currently this means the ingestion process is going to be stuck in an
endless loop of retrying that job and failing on the missing file.
This change should let us process jobs with no test files.
CHANGELOG_BEGIN
CHANGELOG_END
We are currently ingesting Bazel events in two forms:
In the `events-*` indices, each Bazel event is recorded as a separate ES
object, with the corresponding job name as a field that can serve to
aggregate all of the events for a given job.
In the `jobs-*` indices, each job is ingested as a single (composite) ES
object, with the individual events as elements in a list-type field.
When I set up the cluster, I wasn't sure which one would be more useful,
so I included both. We now have a bit more usage experience and it turns
out the `events-*` form is the only one we use, so I think we should
stop ingesting evrything twice and from now on create only the
`events-*` ones.
CHANGELOG_BEGIN
CHANGELOG_END
Disks are currently at 75% fullness, so seems like a good idea to bump a
bit. #10857 should reduce our needs, too, so this should last us a
while.
CHANGELOG_BEGIN
CHANGELOG_END
This bumps dotnet to the version required by the latest azuresigntool,
and pins azuresigntool for the future.
As usual for live CI upgrades, this will be rolled out using the
blue/green approach. I'll keep each deployed commit in this PR.
For future reference, this is PR [#10979].
[#10979]: https://github.com/digital-asset/daml/pull/10979
CHANGELOG_BEGIN
CHANGELOG_END
As explained in #10853, we recently lost our ES cluster. While I'm not
planning on trusting Google's "rolling restart" feature ever again, we
can't exclude the possibility of future similar outages (without a
significant investment in the cluster, which I don't think we want to
do).
Losing the cluster is not a huge issue as we can always reingest the
data. Worst case we lose visibility for a few days. At least, as far as
the bazel logs are concerned.
Losing the Kibana data is a lot more annoying, as that is not derived
data and thus cannot be reingested. This PR aims to add a backup
mechanism for our Kibana configuration.
CHANGELOG_BEGIN
CHANGELOG_END
On Sept 8 our ES cluster became unresponsive. I tried connecting to the
machines.
One machine had an ES Docker container that claimed to have started 7
weeks ago and stopped 5 weeks ago, while the machine's own uptime was 5
weeks. I assume GCP had decided to restart it for some reason. The init
script had failed on missing a TTY, hence the addition of the
`DEBIAN_FRONTEND` env var.
Two machines had a Docker container that had stopped on that day, resp.
6h and 2h before I started investigating. It wasn't immediately clear
what had caused the containers to stop.
On all three of these machines, I was abble to manually restart the
containers and they were abble to reform a cluster, though the state of
the cluster was red (missing shards).
The last two machines simply did not respond to SSH connection attempts.
Assuming it might help, I decided to try to restart the machines. As GCP
does not allow restarting individual machines when they're part of a
managed instance roup, I tried clicking the "rolling restart" button
on the GCP console, which seemed like it would restart the machines. I
carefully selected "restart" (and not "replace"), started the process,
and watched GCP proceed to immediately replace all five machines, losing
all data in the process.
I then started a new cluster and used bigger (and more) machines to
reingest all of the data, and then fell back to the existing
configuration for the "steady" state. I'll try to keep a better eye on
the state of the cluster from now on. In particular, we should not have
a node down for 5 weeks without noticing.
I'll also try to find some time to look into backing up the Kibana
configuration, as that's the one thing we can't just reingest at the
moment.
CHANGELOG_BEGIN
CHANGELOG_END
macOS filesystems have been case-insensitive by default for years, and
in particular our laptops are, so if we want the cache to work as
expected, CI should be too.
Note: this does not apply to Nix, because the Nix partition is a
case-sensitive image per @nycnewman's script on laptops too.
CHANGELOG_BEGIN
CHANGELOG_END
* Generate short to long name mapping in aspect
Maps shortened test names in da_scala_test_suite on Windows to their
long name on Linux and MacOS.
Names are shortened on Windows to avoid exceeding MAX_PATH.
* Script to generate scala test name mapping
* Generate scala-test-suite-name-map.json on Windows
changelog_begin
changelog_end
* Generate UTF-8 with Unix line endings
Otherwise the file will be formatted using UTF-16 with CRLF line
endings, which confuses `jq` on Linux.
* Apply Scala test name remapping before ES upload
* Pipe bazel output into intermediate file
Bazel writes the output of --experimental_show_artifacts to stderr
instead of stdout. In Powershell this means that these outputs are not
plain strings, but instead error objects. Simply redirecting these to
stdout and piping them into further processing will lead to
indeterministically missing items or indeterministically introduced
additional newlines which may break paths.
To work around this we extract the error message from error objects,
introduce appropriate newlines, and write the output to a temporary file
before further processing.
This solution is taken and adapted from
https://stackoverflow.com/a/48671797/841562
* Add copyright header
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
ES died (again) over the weekend, so I had to manually connect to each
node in order to restore it, and thus made another migration. This time
I opted to make a change, though. Lack of memory is a bit of a weak
hypothesis for the observed behaviour, but it's the only one I have at
the moment and, given how reliably ES has been crashing so far, it's
fairly easy to test.
CHANGELOG_BEGIN
CHANGELOG_END
The cluster died yesterday. As part of recovery, I connected to the
machines and made manual changes. To ensure that we get back to a known,
documented setup I then proceeded to do a full blue -> green migration,
having not tainted any of the green machines with manual interventions.
CHANGELOG_BEGIN
CHANGELOG_END
A few small tweaks, but the most important change is giving up on
preemptible instances (for now at least), because I saw GCP kill 8 out
of the 10 nodes at exactly the same time and I can't really expect the
cluster to sruvive that.
CHANGELOG_BEGIN
CHANGELOG_END
Two changes at the Terraform level, both with no impact on the actual
GCP state:
- There is no reason to make this value a `variable`: variables in
Terraforma are meant to be supplied at the CLI. `local` is the right
abstraction here (i.e. set in the file directly).
- Using an unordered `for_each` set rather than a list so we don't have
positional identity, meaning when adding someone at the top we don't
need to destroy and recreate everyone else.
CHANGELOG_BEGIN
CHANGELOG_END
This PR contains many small changes:
- A small refactoring whereby the "es-init" machine is now
(syntactically) integrated with the two instance groups, to cut down a
bit on repetition.
- The feeder machine is now preemptible, because I've seen it recover
enough times that I'm confident this will not cause any issue.
- Indices are now sharded.
- Return values from ES are filtered, cutting down a bit on network
usage and memory requirements to produce the responses.
- Bulk uploads for a single job are now done in parallel. This results
in about a 2x speedup for ingestion.
- crontab was changed to very minute instead of every 5 minutes.
CHANGELOG_BEGIN
CHANGELOG_END
We currently have about 1% (28 out of 2756) of our build logs that have
invalid JSON files. They are all about a `-profile` file being
incomplete, and since those files represent a single JSON object we
can't do smarter things like filtering invalid individual lines.
I haven't looked deeply into _why_ we create invalid files, but this
should let our ingestion process make some progress in the meantime.
CHANGELOG_BEGIN
CHANGELOG_END
* ci/windows: disable spool
We're not expecting to print anything, and @RPS5' security newsletter
says this is a vector of attack.
CHANGELOG_BEGIN
CHANGELOG_END
* increase no-spool to 6
* Windows name truncation causing collisions
* update main group
* remove temp group
This PR adds a machine that will, every 5 minutes, look at the GCS
bucket that stores Bazel metrics and push whatever it finds to
ElasticSearch.
A huge part of this commit is based on @aherrmann-da's work. You can
assume that all the good bits are his.
CHANGELOG_BEGIN
CHANGELOG_END
This PR adds a Kibana instance to each ES node, and duplicates the load
balancer mechanism to expose both raw ES and Kibana.
CHANGELOG_BEGIN
CHANGELOG_END
This PR adds a basic ES cluster to our infrastructure, completely open
and unprotected but only accessible through VPN.
And, as of yet, through its IP address. I'm not sure whether it's worth
adding a DNS for it.
CHANGELOG_BEGIN
CHANGELOG_END
This morning we started with very restricted CI pools (2/6 for Windows
and 7/20 for Linux), apparently because the region we run in (us-east1)
has three zones, two of them were unable to allocate new nodes, and the
default policy is to distribute nodes evenly between zones.
I've manually changed the distribution policy. Unfortunately this option
is not yet available in our version of the GCP Terraform plugin.
CHANGELOG_BEGIN
CHANGELOG_END
We've recently seen a few cases where the macOS nodes ended up not
having the cache partition mounted. So far this has only happened on
semi-broken nodes (guest VM still up and running but host unable to
connect to it), so I haven't been able to actually poke at a broken
machine, but I believe this should allow a machine in such a state to
recover.
While we haven't observed a similar issue on Linux nodes (as far as I'm
aware), I have made similar changes there to keep both scripts in sync.
CHANGELOG_BEGIN
CHANGELOG_END
This includes https://github.com/ndmitchell/hoogle/pull/367.
As usual, I unfortunately cannot test this myself so please review
carefully.
Note that this will slightly increase compile times since we will now
build hoogle. However, we still only build hoogle rather than
everything which takes less than 2min on my very weak personal
laptop. We could integrate this with our nix cache but for now, I’m
not that worried about this.
changelog_begin
changelog_end
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
After #9362, hoogle stopped returning any result. It was up and running,
but with an empty database.
Problem was two-fold:
1. In the switch to `nix`, I lost track of the fact that we were
previously doing all the work in `~/hoogle` rather than `~/`, which
broke the periodic refresh.
2. The initial setup has been broken for months; machines have been
initializing with an empty DB and only getting data on the first
refresh since #7370.
CHANGELOG_BEGIN
CHANGELOG_END
This is a demonstration, commit-by-commit, of how to use the blue/green
deployment for Hoogle servers.
The actual change (using nix) is stolen from #9352.
CHANGELOG_BEGIN
CHANGELOG_END
