- Change machine types to be closer to GCP profiles.
- Set partition sizes correctly on Windows (bigger disks mean little if
partitions don't match).
- Enable full caching on disk access as the only downside seems to be
"risk of data loss if the machine shuts down unexpectedly", and we
never want a machine to come back if it shut down.
The daily restart is now working, I think we can switch over. I'm
keeping the GCP configuration around for the time being in case we need
to roll back for some reason; if everything goes smoothly I'll remove
all of it in a month or so.
I've seen a few "disk full" errors recently on CI, and they don't quite
make sense because we do have this "reset cache" step that looks at
available disk space and cleans up if needed.
So I dug a little bit more and found this discrepancy: the real hard
drives are 400g, the virtual disks are 200g. So what may happen is we
still have plenty of space on the real drives (well, they're virtual,
but bear with me), while the virtual ones are full. And since the
clean-up step only looks at the free space on the real drives, it goes
ahead without cleaning up and then when we try to download stuff there's
no free space on the virtual drives.
This didn't use to be a problem because the size of the virtual drives
mapped to the size of the real ones. This PR aims at restoring that
equivalence.
When we set this up years ago (#1566), it was a way to get a more recent
Docker version than was then available through the default Ubuntu 16.04
apt repository. Nowadays, this actually makes us lag behind, to the
point where the 2.3.1 image isn't building.
CHANGELOG_BEGIN
CHANGELOG_END
Running the `docker` command on today's Ubuntu images crashes the
kernel. (Which is super reassuring from a security pov.)
CHANGELOG_BEGIN
CHANGELOG_END
Process:
- `git ls-files -z | xargs -0 -n 100 sed -i --follow-symlinks 's/DAML/Daml/g'`
- `git add -p`
- `git restore -p`
- Check there is no unstaged change left.
To review:
- Check for false positives by carefully reviewing the diff in this PR.
- Check for false negatives with `git grep DAML`.
- Quicker check for fals positives:
```
git grep DAML | grep -v migration | grep -v DAML_
```
Fixes#13190
Note: This is the "second half" of #13191, which failed to cover all the
remaining DAMLs because of:
```
$ git ls-files | grep "'"
compiler/damlc/tests/daml-test-files/MangledScenario'.daml
```
CHANGELOG_BEGIN
CHANGELOG_END
The cluster shuts down about once every two weeks and takes a couple
hous to get back up. It's been off for a few days right now and as far
as I'm aware nobody noticed.
My personal assessment is that this is costing us more in maintenance
(not to mention running) costs than what we're getting out of it.
CHANGELOG_BEGIN
CHANGELOG_END
However that happened, we were stuck with Nix 2.3.15 (or 2.3.16 in some
cases) on our macOS nodes. This PR is a minor edition to the Nix
initialization commands to switch from 2.4 to "latest", but I wil lalso
use it to record the changes I just did manually to the cluster.
The cluster is currently composed of two parts:
- 7 machines running Catalina (10.15.7).
- 1 machine running Monterey (12.2).
Unfortunately they use different setup. The Catalina ones are described
by the state of the repo (in theory, though keeping them in sync is
manual); in order to update those, I have:
1. Taken one node off the CI pool (`builder1epjj7`).
2. On that node, run the following commands:
```
cd ~/daml/infra/macos/3-running-box
vagrant destroy -f
rm ~/images/*
vagrant box remove macinbox
vagrant box remove azure-ci-node
rm -r ~/.vagrant.d/boxes/macinbox-06032020.tar.gz
softwareupdate -d --fetch-full-installer --full-installer-version 10.15.7
cd ~/daml/infra/macos/1-create-box
sudo macinbox --box-format vmware_desktop --disk 250 --memory 32768 --cpu 10 --user-script user-script.sh
cd ../2-common-box
vagrant up
vagrant package --output ~/images/initialized-$(date +%Y%m%d).box
vagrant destroy -f
cd
./run-agent.sh
```
This leaves us with that node running an updated box. The new box is
in `~/images/initialized-$(date)`.
3. Send that file to all the other nodes with `scp`.
4. Reboot all the nodes (after deactivating & waiting for jobs to
finish).
For the Monterey node, images (steps 1 and 2 in this repo) are currently
created by @nycnewman on another machine I don't have access to, so I
took a slightly different approach: I took the existing image, started
it from the `3-running-box` folder as usual, manually updated Nix there,
then repackaged that.
CHANGELOG_BEGIN
CHANGELOG_END
Goals:
- Reflect manual changes from #12996 in Terraform.
- Reflect manual changes from #12997 in Terraform.
- Update plugins to wirk with #12926.
- Keep running services working through the changes.
Details in commits.
CHANGELOG_BEGIN
CHANGELOG_END
