Commit Graph

192 Commits

Author SHA1 Message Date
Gary Verhaegen
e657624aa7
change Azure subscription (#16741) 2023-04-21 18:16:43 +02:00
Gary Verhaegen
b149ffa8b8
Windows clean up (#16723)
* shut down GCP windows nodes
* shut down periodic-killer
* cycle Windows nodes in daily-reset
2023-04-20 17:06:26 +02:00
Gary Verhaegen
99821e0b66
infra: add a Windows node on Azure (#16705) 2023-04-20 08:32:25 -04:00
Gary Verhaegen
c67ffe403e
remove ubuntu nodes on reset (#16703) 2023-04-19 11:10:00 +00:00
Gary Verhaegen
40f6cb26dc
infra: move Ubuntu to Azure (#16690)
The daily restart is now working, I think we can switch over. I'm
keeping the GCP configuration around for the time being in case we need
to roll back for some reason; if everything goes smoothly I'll remove
all of it in a month or so.
2023-04-13 09:21:38 -04:00
Gary Verhaegen
23ee54ab7b
infra: fix inadvertent shadowing in reset-azure (#16678) 2023-04-11 17:28:23 +02:00
Gary Verhaegen
36a5d2632a
infra: add daily-reset for azure nodes (#16676) 2023-04-10 23:52:35 +02:00
Gary Verhaegen
b54bdcf00d
fix small oversight in azure config (#16629) 2023-03-30 13:39:47 +02:00
Gary Verhaegen
7d69a5975c
ci: set up Ubuntu nodes on Azure (#16610) 2023-03-29 15:21:14 +02:00
Gary Verhaegen
d817383c87
infra: larger virtual disks (#16498)
I've seen a few "disk full" errors recently on CI, and they don't quite
make sense because we do have this "reset cache" step that looks at
available disk space and cleans up if needed.

So I dug a little bit more and found this discrepancy: the real hard
drives are 400g, the virtual disks are 200g. So what may happen is we
still have plenty of space on the real drives (well, they're virtual,
but bear with me), while the virtual ones are full. And since the
clean-up step only looks at the free space on the real drives, it goes
ahead without cleaning up and then when we try to download stuff there's
no free space on the virtual drives.

This didn't use to be a problem because the size of the virtual drives
mapped to the size of the real ones. This PR aims at restoring that
equivalence.
2023-03-09 17:50:55 +00:00
Gary Verhaegen
518248425a
infra: document token rotation (#16461) 2023-03-06 13:50:55 +00:00
Gary Verhaegen
9b50c9f10c
[infra] revert logging change (#16434)
[infra] revert logging change

Reverting #16393 now that Google has fixed things on their side.
2023-03-01 23:01:33 +00:00
Gary Verhaegen
9432f13823
ci: fix Linux nodes (#16393) 2023-02-27 09:34:14 -05:00
Gary Verhaegen
7fb303be3b
update CI node reset permissions (#16216) 2023-02-02 09:06:39 +01:00
Gary Verhaegen
7e59c7e74c
revert #16041 (#16059)
Signatures have been updated, we don't need delete permissions anymore.
2023-01-13 15:40:48 +01:00
Gary Verhaegen
5339642602
infra: add delete permission to assembly account (#16041) 2023-01-11 19:42:15 +00:00
Gary Verhaegen
151e12b81a
bump copyright (#16002)
This is the result of:

- Updating `./COPY` to say `2023`.
- Running `./dev-env/bin/dade-copyright-headers update .`
2023-01-04 18:21:15 +01:00
Gary Verhaegen
65018bcd28
remove Victor's access to data bucket (#15629) 2022-11-24 14:47:37 +01:00
Gary Verhaegen
382b091f77
ci: document node handling a bit more (#15339)
CHANGELOG_BEGIN
CHANGELOG_END
2022-10-26 11:03:27 +02:00
Edward Newman
8e44f3b8bf
M1 build setup using Packer & Tart (#14635)
* M1 build setup using Packer

* Add change log

CHANGELOG_BEGIN
CHANGELOG_END

* Update infra/macos/m1-build/init-2.sh

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>

Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
2022-08-17 07:57:22 -04:00
Gary Verhaegen
8ca7af3030
onboarding: add Chun Lok to release rotation (#14593)
And to our infrastructure's notion of "the ledger clients team"
(obviously abbreviated to "appr").

CHANGELOG_BEGIN
CHANGELOG_END
2022-08-03 12:05:28 +02:00
Gary Verhaegen
bf64e705e9
ci: unpin Docker (#14464)
When we set this up years ago (#1566), it was a way to get a more recent
Docker version than was then available through the default Ubuntu 16.04
apt repository. Nowadays, this actually makes us lag behind, to the
point where the 2.3.1 image isn't building.

CHANGELOG_BEGIN
CHANGELOG_END
2022-07-18 17:52:28 +00:00
Gary Verhaegen
9aec01853b
ci: unpin Ubuntu image (#14143)
Cleaning up after #14126

CHANGELOG_BEGIN
CHANGELOG_END
2022-06-22 13:56:05 +02:00
Gary Verhaegen
feb53f96c1
infra: tighten TLS security (#14239)
This tightens our TLS configuration a bit, mostly by dropping support
for SSL3, TLS1.0 and TLS1.1 on https://hoogle.daml.com,
https://bazel-cache.da-ext.net, https://nix-cache.da-ext.net and the
daml-binaries front (which I don't think we still use).

CHANGELOG_BEGIN
CHANGELOG_END
2022-06-21 14:37:24 +00:00
Gary Verhaegen
d81b4a7071
ci: pin Linux image to yesterday's because today's is broken (#14126)
Running the `docker` command on today's Ubuntu images crashes the
kernel. (Which is super reassuring from a security pov.)

CHANGELOG_BEGIN
CHANGELOG_END
2022-06-08 14:08:23 +00:00
Gary Verhaegen
eefe285f67
remove Stewart from release rotation (#14018)
CHANGELOG_BEGIN
CHANGELOG_END
2022-05-30 17:09:40 +00:00
Gary Verhaegen
dfa648f585
hunt down DAML better (#13195)
Process:

- `git ls-files -z | xargs -0 -n 100 sed -i --follow-symlinks 's/DAML/Daml/g'`
- `git add -p`
- `git restore -p`
- Check there is no unstaged change left.

To review:

- Check for false positives by carefully reviewing the diff in this PR.
- Check for false negatives with `git grep DAML`.
- Quicker check for fals positives:

```
git grep DAML | grep -v migration | grep -v DAML_
```

Fixes #13190

Note: This is the "second half" of #13191, which failed to cover all the
remaining DAMLs because of:

```
$ git ls-files | grep "'"
compiler/damlc/tests/daml-test-files/MangledScenario'.daml
```

CHANGELOG_BEGIN
CHANGELOG_END
2022-03-08 17:04:58 +01:00
Gary Verhaegen
53557dd7de
shut down ElasticSearch (#13151)
The cluster shuts down about once every two weeks and takes a couple
hous to get back up. It's been off for a few days right now and as far
as I'm aware nobody noticed.

My personal assessment is that this is costing us more in maintenance
(not to mention running) costs than what we're getting out of it.

CHANGELOG_BEGIN
CHANGELOG_END
2022-03-04 17:14:15 +01:00
Gary Verhaegen
091a5ac752
appr: add Stewart (#13116)
CHANGELOG_BEGIN
CHANGELOG_END
2022-03-01 23:11:54 +00:00
Gary Verhaegen
fe9d44ffe7
ci: bump Nix on macOS nodes (#13061)
However that happened, we were stuck with Nix 2.3.15 (or 2.3.16 in some
cases) on our macOS nodes. This PR is a minor edition to the Nix
initialization commands to switch from 2.4 to "latest", but I wil lalso
use it to record the changes I just did manually to the cluster.

The cluster is currently composed of two parts:
- 7 machines running Catalina (10.15.7).
- 1 machine running Monterey (12.2).

Unfortunately they use different setup. The Catalina ones are described
by the state of the repo (in theory, though keeping them in sync is
manual); in order to update those, I have:

1. Taken one node off the CI pool (`builder1epjj7`).
2. On that node, run the following commands:
   ```
   cd ~/daml/infra/macos/3-running-box
   vagrant destroy -f
   rm ~/images/*
   vagrant box remove macinbox
   vagrant box remove azure-ci-node
   rm -r ~/.vagrant.d/boxes/macinbox-06032020.tar.gz
   softwareupdate -d --fetch-full-installer --full-installer-version 10.15.7
   cd ~/daml/infra/macos/1-create-box
   sudo macinbox --box-format vmware_desktop --disk 250 --memory 32768 --cpu 10 --user-script user-script.sh
   cd ../2-common-box
   vagrant up
   vagrant package --output ~/images/initialized-$(date +%Y%m%d).box
   vagrant destroy -f
   cd
   ./run-agent.sh
   ```
   This leaves us with that node running an updated box. The new box is
   in `~/images/initialized-$(date)`.
3. Send that file to all the other nodes with `scp`.
4. Reboot all the nodes (after deactivating & waiting for jobs to
   finish).

For the Monterey node, images (steps 1 and 2 in this repo) are currently
created by @nycnewman on another machine I don't have access to, so I
took a slightly different approach: I took the existing image, started
it from the `3-running-box` folder as usual, manually updated Nix there,
then repackaged that.

CHANGELOG_BEGIN
CHANGELOG_END
2022-02-24 01:04:28 +00:00
Gary Verhaegen
583cad5fd6
Fix tf (#13028)
Goals:

- Reflect manual changes from #12996 in Terraform.
- Reflect manual changes from #12997 in Terraform.
- Update plugins to wirk with #12926.
- Keep running services working through the changes.

Details in commits.

CHANGELOG_BEGIN
CHANGELOG_END
2022-02-22 18:33:21 +00:00
Gary Verhaegen
c04fa81d6a
ci: bump Windows workdirs (#12918)
Since #12645, we added a new pipeline, so we need to add a corresponding
entry.

As for #12645, the content of the files and the directory structure is
taken directly from a live CI node, as printed by the (now-named)
`workdirs` step.

CHANGELOG_BEGIN
CHANGELOG_END
2022-02-14 18:49:32 +00:00
Gary Verhaegen
449a68cb33
Fix es (#12845)
A node seemed to have died so I connected to investigate and you know
the rest of this story.

CHANGELOG_BEGIN
CHANGELOG_END
2022-02-09 19:33:25 +00:00
Gary Verhaegen
f08dfa3264
Bump terraform (#12670)
We've been using an old version of Terraform for a long time now. The
main blocker used to be that there was no post-0.12 version of `secret`,
but that has now been resolved: there's a new fork, with new maintainers
(blessed by the original one and accepted by the Terraform registry)
[here].

I'll be upgrading one version at a time as 0.x versions are considered
major (and thus potentially breaking).

[here]: https://github.com/numtide/terraform-provider-secret

See https://github.com/digital-asset/daml/pull/12670 for details.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-31 15:46:59 +01:00
Gary Verhaegen
1fa7f61bb0
ci: pin workdirs on Windows (#12645)
The Bazel cache on Windows includes absolute paths. The normal process
for Azure is to dynamically allocate new top-level folders for each new
bbuild that runs on a given machine. The result of that is that we get
about a one in three chance to get caching for any single Windows build
(it's actually not _quite_ that because we don't run different builds an
equal number of times).

This PR is an attempt at pinning the folder to job mapping by mucking
around in [Azure internals], which may or may not have bad consequences
down the line, assuming it works at all.

[Azure internals]: https://github.com/microsoft/azure-pipelines-agent/blob/master/docs/jobdirectories.md

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-31 10:20:12 +00:00
Gary Verhaegen
b1a917596c
ci: reduce Windows capacity (#12607)
Reverting #12599.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-26 18:17:56 +00:00
Gary Verhaegen
01219d6cdc
ci: temporarily increase Windows capacity (#12599)
Our Windows CI nodes seem completely overwhelmed today, with typical
wait times above half an hour before jobs even start. This isn't fun, so
I'd like to double our capacity for a few hours.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-26 15:03:28 +01:00
Gary Verhaegen
170d839ed0
Fix es (#12554)
It's down again. I wish I knew why it does that.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-24 18:33:59 +01:00
Gary Verhaegen
de2a8c0c04
ci: use service account for Windows nodes (#12489)
When no service account is explicitly selected, GCP provides a default
one, which happens to have way more access rights than we're comfortable
with. I'm not quite sure how the total lack of a service account slipped
through here, but I've noticed today so I'm changing it.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-19 19:58:17 +00:00
Gary Verhaegen
5716d99cd2
Disable printer sharing (#12408)
As the title suggests. We already disable all communication between CI
nodes through network rules, but we currently get a lot of noise from
GCP logging violations to those rules from Windows trying to feel its
way out for file share buddies.

CHANGELOG_BEGIN
CHANGELOG_END

As usualy, this branch will contain intermediate commits that may serve
as an audit log of sorts.
2022-01-13 20:55:18 +00:00
Gary Verhaegen
e34ac20d23
offboarding Akshay (#12396)
😿

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-13 14:40:44 +01:00
Gary Verhaegen
6aa9409e6e
split-releases: gcs accounts for assembly & canton (#12373)
CHANGELOG_BEGIN
CHANGELOG_END
2022-01-13 14:19:08 +01:00
Gary Verhaegen
9f5a2f9778
Fix terraform (#12333)
Our Terraform configuration has been slightly broken by two recent
changes:

- The nixpkgs upgrade in #12280 means a new version of our GCP plugin
  for Terraform, which as a breaking change added a required argument to
  `google_project_iam_member`. The new version also results in a number
  of smaller changes in the way Terraform handles default arguments, which
  doesn't result in any changes to our configuration files or to the
  behaviour of our deployed infrastructure, but does require re-syncing
  the Terraform state (by running `terraform apply`, which would
  essentially be a no-op if it were not for the next bullet point).
- The nix configuration changes in #12265 have changed the Linux CI
  nodes configuration but have not been deployed yet.

This PR is an audit log of the steps taken to rectfy those and bring us
back to a state where our deployed configuration and our recorded
Terraform state both agree with our current `main` branch tip.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-10 21:56:47 +00:00
Gary Verhaegen
648021a2e7
Fix es cluster (#12262)
audit log of actions taken to fix cluster post Winter break

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-05 16:29:48 +00:00
Samir Talwar
854b66ee2f
devenv: Use NIX_USER_CONF_FILES to set caches. (#12265)
* devenv: Factor out a function to get the Nix version.

* devenv: On newer versions of Nix, use `NIX_USER_CONF_FILES`.

This stacks, rather than overwrites.

* devenv: Append Nix caches, instead of overwriting them.

The "extra-" prefix tells Nix to append.

We also switch to non-deprecated configuration keys.

CHANGELOG_BEGIN
CHANGELOG_END

* devenv: Just require Nix v2.4 or newer.

* devenv: `NIX_USER_CONF_FILES` may not be set already.
2022-01-05 13:24:52 +01:00
Gary Verhaegen
93e616475e
Update ci nodes for copyright update (#12255)
Audit log for CI rotation.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-04 15:24:31 +00:00
Gary Verhaegen
d2e2c21684
update copyright headers (#12240)
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.

I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-03 16:36:51 +00:00
Gary Verhaegen
0e30d468f9
expand CI cluster back (#12239)
To be done / merged on Jan 3.

CHANGELOG_BEGIN
CHANGELOG_END
2022-01-03 14:50:08 +00:00
Gary Verhaegen
a51f75d193
give a break to CI (#12238)
I can't think of a good reason to keep 30+ machines running over the
Winter break. I'll bump this back up on Jan 3.

CHANGELOG_BEGIN
CHANGELOG_END
2021-12-26 10:53:08 +01:00
Gary Verhaegen
c5708c96c5
es: mitigate log4j vuln (#12118)
CHANGELOG_BEGIN
CHANGELOG_END
2021-12-10 22:46:02 +00:00