When we set this up years ago (#1566), it was a way to get a more recent
Docker version than was then available through the default Ubuntu 16.04
apt repository. Nowadays, this actually makes us lag behind, to the
point where the 2.3.1 image isn't building.
CHANGELOG_BEGIN
CHANGELOG_END
Running the `docker` command on today's Ubuntu images crashes the
kernel. (Which is super reassuring from a security pov.)
CHANGELOG_BEGIN
CHANGELOG_END
Process:
- `git ls-files -z | xargs -0 -n 100 sed -i --follow-symlinks 's/DAML/Daml/g'`
- `git add -p`
- `git restore -p`
- Check there is no unstaged change left.
To review:
- Check for false positives by carefully reviewing the diff in this PR.
- Check for false negatives with `git grep DAML`.
- Quicker check for fals positives:
```
git grep DAML | grep -v migration | grep -v DAML_
```
Fixes#13190
Note: This is the "second half" of #13191, which failed to cover all the
remaining DAMLs because of:
```
$ git ls-files | grep "'"
compiler/damlc/tests/daml-test-files/MangledScenario'.daml
```
CHANGELOG_BEGIN
CHANGELOG_END
The cluster shuts down about once every two weeks and takes a couple
hous to get back up. It's been off for a few days right now and as far
as I'm aware nobody noticed.
My personal assessment is that this is costing us more in maintenance
(not to mention running) costs than what we're getting out of it.
CHANGELOG_BEGIN
CHANGELOG_END
However that happened, we were stuck with Nix 2.3.15 (or 2.3.16 in some
cases) on our macOS nodes. This PR is a minor edition to the Nix
initialization commands to switch from 2.4 to "latest", but I wil lalso
use it to record the changes I just did manually to the cluster.
The cluster is currently composed of two parts:
- 7 machines running Catalina (10.15.7).
- 1 machine running Monterey (12.2).
Unfortunately they use different setup. The Catalina ones are described
by the state of the repo (in theory, though keeping them in sync is
manual); in order to update those, I have:
1. Taken one node off the CI pool (`builder1epjj7`).
2. On that node, run the following commands:
```
cd ~/daml/infra/macos/3-running-box
vagrant destroy -f
rm ~/images/*
vagrant box remove macinbox
vagrant box remove azure-ci-node
rm -r ~/.vagrant.d/boxes/macinbox-06032020.tar.gz
softwareupdate -d --fetch-full-installer --full-installer-version 10.15.7
cd ~/daml/infra/macos/1-create-box
sudo macinbox --box-format vmware_desktop --disk 250 --memory 32768 --cpu 10 --user-script user-script.sh
cd ../2-common-box
vagrant up
vagrant package --output ~/images/initialized-$(date +%Y%m%d).box
vagrant destroy -f
cd
./run-agent.sh
```
This leaves us with that node running an updated box. The new box is
in `~/images/initialized-$(date)`.
3. Send that file to all the other nodes with `scp`.
4. Reboot all the nodes (after deactivating & waiting for jobs to
finish).
For the Monterey node, images (steps 1 and 2 in this repo) are currently
created by @nycnewman on another machine I don't have access to, so I
took a slightly different approach: I took the existing image, started
it from the `3-running-box` folder as usual, manually updated Nix there,
then repackaged that.
CHANGELOG_BEGIN
CHANGELOG_END
Goals:
- Reflect manual changes from #12996 in Terraform.
- Reflect manual changes from #12997 in Terraform.
- Update plugins to wirk with #12926.
- Keep running services working through the changes.
Details in commits.
CHANGELOG_BEGIN
CHANGELOG_END
Since #12645, we added a new pipeline, so we need to add a corresponding
entry.
As for #12645, the content of the files and the directory structure is
taken directly from a live CI node, as printed by the (now-named)
`workdirs` step.
CHANGELOG_BEGIN
CHANGELOG_END
We've been using an old version of Terraform for a long time now. The
main blocker used to be that there was no post-0.12 version of `secret`,
but that has now been resolved: there's a new fork, with new maintainers
(blessed by the original one and accepted by the Terraform registry)
[here].
I'll be upgrading one version at a time as 0.x versions are considered
major (and thus potentially breaking).
[here]: https://github.com/numtide/terraform-provider-secret
See https://github.com/digital-asset/daml/pull/12670 for details.
CHANGELOG_BEGIN
CHANGELOG_END
The Bazel cache on Windows includes absolute paths. The normal process
for Azure is to dynamically allocate new top-level folders for each new
bbuild that runs on a given machine. The result of that is that we get
about a one in three chance to get caching for any single Windows build
(it's actually not _quite_ that because we don't run different builds an
equal number of times).
This PR is an attempt at pinning the folder to job mapping by mucking
around in [Azure internals], which may or may not have bad consequences
down the line, assuming it works at all.
[Azure internals]: https://github.com/microsoft/azure-pipelines-agent/blob/master/docs/jobdirectories.md
CHANGELOG_BEGIN
CHANGELOG_END
Our Windows CI nodes seem completely overwhelmed today, with typical
wait times above half an hour before jobs even start. This isn't fun, so
I'd like to double our capacity for a few hours.
CHANGELOG_BEGIN
CHANGELOG_END
When no service account is explicitly selected, GCP provides a default
one, which happens to have way more access rights than we're comfortable
with. I'm not quite sure how the total lack of a service account slipped
through here, but I've noticed today so I'm changing it.
CHANGELOG_BEGIN
CHANGELOG_END
As the title suggests. We already disable all communication between CI
nodes through network rules, but we currently get a lot of noise from
GCP logging violations to those rules from Windows trying to feel its
way out for file share buddies.
CHANGELOG_BEGIN
CHANGELOG_END
As usualy, this branch will contain intermediate commits that may serve
as an audit log of sorts.
Our Terraform configuration has been slightly broken by two recent
changes:
- The nixpkgs upgrade in #12280 means a new version of our GCP plugin
for Terraform, which as a breaking change added a required argument to
`google_project_iam_member`. The new version also results in a number
of smaller changes in the way Terraform handles default arguments, which
doesn't result in any changes to our configuration files or to the
behaviour of our deployed infrastructure, but does require re-syncing
the Terraform state (by running `terraform apply`, which would
essentially be a no-op if it were not for the next bullet point).
- The nix configuration changes in #12265 have changed the Linux CI
nodes configuration but have not been deployed yet.
This PR is an audit log of the steps taken to rectfy those and bring us
back to a state where our deployed configuration and our recorded
Terraform state both agree with our current `main` branch tip.
CHANGELOG_BEGIN
CHANGELOG_END
* devenv: Factor out a function to get the Nix version.
* devenv: On newer versions of Nix, use `NIX_USER_CONF_FILES`.
This stacks, rather than overwrites.
* devenv: Append Nix caches, instead of overwriting them.
The "extra-" prefix tells Nix to append.
We also switch to non-deprecated configuration keys.
CHANGELOG_BEGIN
CHANGELOG_END
* devenv: Just require Nix v2.4 or newer.
* devenv: `NIX_USER_CONF_FILES` may not be set already.
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
The cluster shut down today. I'm still not sure why this happens, but
cycling the entire cluster seems to solve it so here goes.
CHANGELOG_BEGIN
CHANGELOG_END
As part of the 2.4 release, the Nix installer has been changed to take
care of the volume setup (which we don't want it to do here). Because
that requires root access, they've decided to make multi-user install
the default, and to disable single-user install.
We could do an in-depth review of the difference and adapt our setup to
use a multi-user setup (we do use the multi-user setup on Linux, so
there's precedent), but as an immediate fix, we can keep our single-user
setup _and_ get the latest Nix by using the 2.3.16 installer and then
upgrading from within Nix itself. This _should_ keep working at least
for a while, as Linux still defaults to single-user.
CHANGELOG_BEGIN
CHANGELOG_END
Hoogle has been down for at least 24h accoridng to user reports. What
seems to be happening is that our nixpkgs pinning is not taking effect,
and the nixpkgs version of Hoogle already includes the patch we are
trying to add. This confuses nix, which fails, and thus the boot
sequence is broken.
I've applied the minimal possible patch here (i.e. enforce the pin),
which gets things running again. I've already deployed this change.
We may want to look at bumping the nixpkgs snapshot.
CHANGELOG_BEGIN
CHANGELOG_END
This fixes an issue where we try to upload a single JSON blob that is
bigger than the limit of 500MB. We could also raise the limit, I guess,
but that seems more error-prone.
CHANGELOG_BEGIN
CHANGELOG_END
If a job fails at build time, there are not test logs to process.
Currently this means the ingestion process is going to be stuck in an
endless loop of retrying that job and failing on the missing file.
This change should let us process jobs with no test files.
CHANGELOG_BEGIN
CHANGELOG_END
We are currently ingesting Bazel events in two forms:
In the `events-*` indices, each Bazel event is recorded as a separate ES
object, with the corresponding job name as a field that can serve to
aggregate all of the events for a given job.
In the `jobs-*` indices, each job is ingested as a single (composite) ES
object, with the individual events as elements in a list-type field.
When I set up the cluster, I wasn't sure which one would be more useful,
so I included both. We now have a bit more usage experience and it turns
out the `events-*` form is the only one we use, so I think we should
stop ingesting evrything twice and from now on create only the
`events-*` ones.
CHANGELOG_BEGIN
CHANGELOG_END
Disks are currently at 75% fullness, so seems like a good idea to bump a
bit. #10857 should reduce our needs, too, so this should last us a
while.
CHANGELOG_BEGIN
CHANGELOG_END
This bumps dotnet to the version required by the latest azuresigntool,
and pins azuresigntool for the future.
As usual for live CI upgrades, this will be rolled out using the
blue/green approach. I'll keep each deployed commit in this PR.
For future reference, this is PR [#10979].
[#10979]: https://github.com/digital-asset/daml/pull/10979
CHANGELOG_BEGIN
CHANGELOG_END
As explained in #10853, we recently lost our ES cluster. While I'm not
planning on trusting Google's "rolling restart" feature ever again, we
can't exclude the possibility of future similar outages (without a
significant investment in the cluster, which I don't think we want to
do).
Losing the cluster is not a huge issue as we can always reingest the
data. Worst case we lose visibility for a few days. At least, as far as
the bazel logs are concerned.
Losing the Kibana data is a lot more annoying, as that is not derived
data and thus cannot be reingested. This PR aims to add a backup
mechanism for our Kibana configuration.
CHANGELOG_BEGIN
CHANGELOG_END
On Sept 8 our ES cluster became unresponsive. I tried connecting to the
machines.
One machine had an ES Docker container that claimed to have started 7
weeks ago and stopped 5 weeks ago, while the machine's own uptime was 5
weeks. I assume GCP had decided to restart it for some reason. The init
script had failed on missing a TTY, hence the addition of the
`DEBIAN_FRONTEND` env var.
Two machines had a Docker container that had stopped on that day, resp.
6h and 2h before I started investigating. It wasn't immediately clear
what had caused the containers to stop.
On all three of these machines, I was abble to manually restart the
containers and they were abble to reform a cluster, though the state of
the cluster was red (missing shards).
The last two machines simply did not respond to SSH connection attempts.
Assuming it might help, I decided to try to restart the machines. As GCP
does not allow restarting individual machines when they're part of a
managed instance roup, I tried clicking the "rolling restart" button
on the GCP console, which seemed like it would restart the machines. I
carefully selected "restart" (and not "replace"), started the process,
and watched GCP proceed to immediately replace all five machines, losing
all data in the process.
I then started a new cluster and used bigger (and more) machines to
reingest all of the data, and then fell back to the existing
configuration for the "steady" state. I'll try to keep a better eye on
the state of the cluster from now on. In particular, we should not have
a node down for 5 weeks without noticing.
I'll also try to find some time to look into backing up the Kibana
configuration, as that's the one thing we can't just reingest at the
moment.
CHANGELOG_BEGIN
CHANGELOG_END
macOS filesystems have been case-insensitive by default for years, and
in particular our laptops are, so if we want the cache to work as
expected, CI should be too.
Note: this does not apply to Nix, because the Nix partition is a
case-sensitive image per @nycnewman's script on laptops too.
CHANGELOG_BEGIN
CHANGELOG_END
* Generate short to long name mapping in aspect
Maps shortened test names in da_scala_test_suite on Windows to their
long name on Linux and MacOS.
Names are shortened on Windows to avoid exceeding MAX_PATH.
* Script to generate scala test name mapping
* Generate scala-test-suite-name-map.json on Windows
changelog_begin
changelog_end
* Generate UTF-8 with Unix line endings
Otherwise the file will be formatted using UTF-16 with CRLF line
endings, which confuses `jq` on Linux.
* Apply Scala test name remapping before ES upload
* Pipe bazel output into intermediate file
Bazel writes the output of --experimental_show_artifacts to stderr
instead of stdout. In Powershell this means that these outputs are not
plain strings, but instead error objects. Simply redirecting these to
stdout and piping them into further processing will lead to
indeterministically missing items or indeterministically introduced
additional newlines which may break paths.
To work around this we extract the error message from error objects,
introduce appropriate newlines, and write the output to a temporary file
before further processing.
This solution is taken and adapted from
https://stackoverflow.com/a/48671797/841562
* Add copyright header
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
ES died (again) over the weekend, so I had to manually connect to each
node in order to restore it, and thus made another migration. This time
I opted to make a change, though. Lack of memory is a bit of a weak
hypothesis for the observed behaviour, but it's the only one I have at
the moment and, given how reliably ES has been crashing so far, it's
fairly easy to test.
CHANGELOG_BEGIN
CHANGELOG_END
The cluster died yesterday. As part of recovery, I connected to the
machines and made manual changes. To ensure that we get back to a known,
documented setup I then proceeded to do a full blue -> green migration,
having not tainted any of the green machines with manual interventions.
CHANGELOG_BEGIN
CHANGELOG_END
A few small tweaks, but the most important change is giving up on
preemptible instances (for now at least), because I saw GCP kill 8 out
of the 10 nodes at exactly the same time and I can't really expect the
cluster to sruvive that.
CHANGELOG_BEGIN
CHANGELOG_END
