Goals:
- Reflect manual changes from #12996 in Terraform.
- Reflect manual changes from #12997 in Terraform.
- Update plugins to wirk with #12926.
- Keep running services working through the changes.
Details in commits.
CHANGELOG_BEGIN
CHANGELOG_END
We've been using an old version of Terraform for a long time now. The
main blocker used to be that there was no post-0.12 version of `secret`,
but that has now been resolved: there's a new fork, with new maintainers
(blessed by the original one and accepted by the Terraform registry)
[here].
I'll be upgrading one version at a time as 0.x versions are considered
major (and thus potentially breaking).
[here]: https://github.com/numtide/terraform-provider-secret
See https://github.com/digital-asset/daml/pull/12670 for details.
CHANGELOG_BEGIN
CHANGELOG_END
New year, new copyright, new expected unknown issues with various files
that won't be covered by the script and/or will be but shouldn't change.
I'll do the details on Jan 1, but would appreciate this being
preapproved so I can actually get it merged by then.
CHANGELOG_BEGIN
CHANGELOG_END
This PR contains many small changes:
- A small refactoring whereby the "es-init" machine is now
(syntactically) integrated with the two instance groups, to cut down a
bit on repetition.
- The feeder machine is now preemptible, because I've seen it recover
enough times that I'm confident this will not cause any issue.
- Indices are now sharded.
- Return values from ES are filtered, cutting down a bit on network
usage and memory requirements to produce the responses.
- Bulk uploads for a single job are now done in parallel. This results
in about a 2x speedup for ingestion.
- crontab was changed to very minute instead of every 5 minutes.
CHANGELOG_BEGIN
CHANGELOG_END
This PR adds a machine that will, every 5 minutes, look at the GCS
bucket that stores Bazel metrics and push whatever it finds to
ElasticSearch.
A huge part of this commit is based on @aherrmann-da's work. You can
assume that all the good bits are his.
CHANGELOG_BEGIN
CHANGELOG_END
* fixup terraform config
Two changes have happened recently that have invalidated the current
Terraform files:
1. The Terraform version has gone through a major, incompatible upgrade
(#8190); the required updates for this are reflected in the first
commit of this PR.
2. The certificate used to serve [Hoogle](https://hoogle.daml.com) was
about to expire, so Edward created a new one and updated the config
directly. The second commit in this PR updates the Terraform config
to match that new, already-in-prod setting.
Note: This PR applies cleanly, as there are no resulting changes in
Terraform's perception of the target state from 1, and the change from 2
has already been applied through other channels.
CHANGELOG_BEGIN
CHANGELOG_END
* update hoogle cert
CHANGELOG_BEGIN
- Our Linux binaries are now built on Ubuntu 20.04 instead of 16.04. We
do not expect any user-level impact, but please reach out if you
do notice any issue that might be caused by this.
CHANGELOG_END
Our old wildcard certificate has expired. @nycnewman has already updated
our configuration to use new ones; this is just updating the tf files to
match.
CHANGELOG_BEGIN
CHANGELOG_END
It looks like #6761 broke our Terraform setup by upgrading the nixpkgs
snapshot. That this has not been caught earlier is, I suppose, a
testament to how stable our infrastructure has become nowadays.
This is the same issue we had with the Google providers in #6402, i.e.
we are trying to pin the provider versions both at the nix level and at
the terraform level, with no way to force them to stay in sync.
I don't have a good proposal for such a way, and it seems rare and
innocuous enough to not warrant the investment to fix this at a more
fundamental level.
CHANGELOG_BEGIN
CHANGELOG_END
It looks like some nix update has broken our current Terraform setup.
The Google provider plugin has changed its reported version to 0.0.0;
poking at my local nix store seems to indicate we actually get 3.15, but
🤷.
This PR also reverts the infra part of #6400 so we get back to master ==
reality.
CHANGELOG_BEGIN
CHANGELOG_END
We're currently depending on a floating "latest", which is often a bad
idea. Today my machine decided to upgrade the google plugin,w hich is no
specifying some new fields for the GCS objects, and therefore `terraform
plan` doe snot look clean anymore, even though there has been no change
to the terraform files (nor to the infrastructure).
This PR aims to make our Terraform setup more reproducible by pinning
Terraform plugin versions. It's also a way to track the application of
the "new" Terraform setup, as it is technically a standard change
(though hopefully a very safe one).
CHANGELOG_BEGIN
CHANGELOG_END
* nix: add the more providers to terraform
* docs: make tarballs more reproducible
* ci: use the linux-pool pool
* ci: tweak the nix installation
handle the case where the user is root and on ubuntu
* infra: terraform fmt
* infra: add Azure Pipeline agents
* ci: only enable linux-pool for internal PRs
