We are currently ingesting Bazel events in two forms:
In the `events-*` indices, each Bazel event is recorded as a separate ES
object, with the corresponding job name as a field that can serve to
aggregate all of the events for a given job.
In the `jobs-*` indices, each job is ingested as a single (composite) ES
object, with the individual events as elements in a list-type field.
When I set up the cluster, I wasn't sure which one would be more useful,
so I included both. We now have a bit more usage experience and it turns
out the `events-*` form is the only one we use, so I think we should
stop ingesting evrything twice and from now on create only the
`events-*` ones.
CHANGELOG_BEGIN
CHANGELOG_END
Disks are currently at 75% fullness, so seems like a good idea to bump a
bit. #10857 should reduce our needs, too, so this should last us a
while.
CHANGELOG_BEGIN
CHANGELOG_END
This bumps dotnet to the version required by the latest azuresigntool,
and pins azuresigntool for the future.
As usual for live CI upgrades, this will be rolled out using the
blue/green approach. I'll keep each deployed commit in this PR.
For future reference, this is PR [#10979].
[#10979]: https://github.com/digital-asset/daml/pull/10979
CHANGELOG_BEGIN
CHANGELOG_END
As explained in #10853, we recently lost our ES cluster. While I'm not
planning on trusting Google's "rolling restart" feature ever again, we
can't exclude the possibility of future similar outages (without a
significant investment in the cluster, which I don't think we want to
do).
Losing the cluster is not a huge issue as we can always reingest the
data. Worst case we lose visibility for a few days. At least, as far as
the bazel logs are concerned.
Losing the Kibana data is a lot more annoying, as that is not derived
data and thus cannot be reingested. This PR aims to add a backup
mechanism for our Kibana configuration.
CHANGELOG_BEGIN
CHANGELOG_END
On Sept 8 our ES cluster became unresponsive. I tried connecting to the
machines.
One machine had an ES Docker container that claimed to have started 7
weeks ago and stopped 5 weeks ago, while the machine's own uptime was 5
weeks. I assume GCP had decided to restart it for some reason. The init
script had failed on missing a TTY, hence the addition of the
`DEBIAN_FRONTEND` env var.
Two machines had a Docker container that had stopped on that day, resp.
6h and 2h before I started investigating. It wasn't immediately clear
what had caused the containers to stop.
On all three of these machines, I was abble to manually restart the
containers and they were abble to reform a cluster, though the state of
the cluster was red (missing shards).
The last two machines simply did not respond to SSH connection attempts.
Assuming it might help, I decided to try to restart the machines. As GCP
does not allow restarting individual machines when they're part of a
managed instance roup, I tried clicking the "rolling restart" button
on the GCP console, which seemed like it would restart the machines. I
carefully selected "restart" (and not "replace"), started the process,
and watched GCP proceed to immediately replace all five machines, losing
all data in the process.
I then started a new cluster and used bigger (and more) machines to
reingest all of the data, and then fell back to the existing
configuration for the "steady" state. I'll try to keep a better eye on
the state of the cluster from now on. In particular, we should not have
a node down for 5 weeks without noticing.
I'll also try to find some time to look into backing up the Kibana
configuration, as that's the one thing we can't just reingest at the
moment.
CHANGELOG_BEGIN
CHANGELOG_END
macOS filesystems have been case-insensitive by default for years, and
in particular our laptops are, so if we want the cache to work as
expected, CI should be too.
Note: this does not apply to Nix, because the Nix partition is a
case-sensitive image per @nycnewman's script on laptops too.
CHANGELOG_BEGIN
CHANGELOG_END
* Generate short to long name mapping in aspect
Maps shortened test names in da_scala_test_suite on Windows to their
long name on Linux and MacOS.
Names are shortened on Windows to avoid exceeding MAX_PATH.
* Script to generate scala test name mapping
* Generate scala-test-suite-name-map.json on Windows
changelog_begin
changelog_end
* Generate UTF-8 with Unix line endings
Otherwise the file will be formatted using UTF-16 with CRLF line
endings, which confuses `jq` on Linux.
* Apply Scala test name remapping before ES upload
* Pipe bazel output into intermediate file
Bazel writes the output of --experimental_show_artifacts to stderr
instead of stdout. In Powershell this means that these outputs are not
plain strings, but instead error objects. Simply redirecting these to
stdout and piping them into further processing will lead to
indeterministically missing items or indeterministically introduced
additional newlines which may break paths.
To work around this we extract the error message from error objects,
introduce appropriate newlines, and write the output to a temporary file
before further processing.
This solution is taken and adapted from
https://stackoverflow.com/a/48671797/841562
* Add copyright header
Co-authored-by: Andreas Herrmann <andreas.herrmann@tweag.io>
ES died (again) over the weekend, so I had to manually connect to each
node in order to restore it, and thus made another migration. This time
I opted to make a change, though. Lack of memory is a bit of a weak
hypothesis for the observed behaviour, but it's the only one I have at
the moment and, given how reliably ES has been crashing so far, it's
fairly easy to test.
CHANGELOG_BEGIN
CHANGELOG_END
The cluster died yesterday. As part of recovery, I connected to the
machines and made manual changes. To ensure that we get back to a known,
documented setup I then proceeded to do a full blue -> green migration,
having not tainted any of the green machines with manual interventions.
CHANGELOG_BEGIN
CHANGELOG_END
A few small tweaks, but the most important change is giving up on
preemptible instances (for now at least), because I saw GCP kill 8 out
of the 10 nodes at exactly the same time and I can't really expect the
cluster to sruvive that.
CHANGELOG_BEGIN
CHANGELOG_END
Two changes at the Terraform level, both with no impact on the actual
GCP state:
- There is no reason to make this value a `variable`: variables in
Terraforma are meant to be supplied at the CLI. `local` is the right
abstraction here (i.e. set in the file directly).
- Using an unordered `for_each` set rather than a list so we don't have
positional identity, meaning when adding someone at the top we don't
need to destroy and recreate everyone else.
CHANGELOG_BEGIN
CHANGELOG_END
This PR contains many small changes:
- A small refactoring whereby the "es-init" machine is now
(syntactically) integrated with the two instance groups, to cut down a
bit on repetition.
- The feeder machine is now preemptible, because I've seen it recover
enough times that I'm confident this will not cause any issue.
- Indices are now sharded.
- Return values from ES are filtered, cutting down a bit on network
usage and memory requirements to produce the responses.
- Bulk uploads for a single job are now done in parallel. This results
in about a 2x speedup for ingestion.
- crontab was changed to very minute instead of every 5 minutes.
CHANGELOG_BEGIN
CHANGELOG_END
We currently have about 1% (28 out of 2756) of our build logs that have
invalid JSON files. They are all about a `-profile` file being
incomplete, and since those files represent a single JSON object we
can't do smarter things like filtering invalid individual lines.
I haven't looked deeply into _why_ we create invalid files, but this
should let our ingestion process make some progress in the meantime.
CHANGELOG_BEGIN
CHANGELOG_END
* ci/windows: disable spool
We're not expecting to print anything, and @RPS5' security newsletter
says this is a vector of attack.
CHANGELOG_BEGIN
CHANGELOG_END
* increase no-spool to 6
* Windows name truncation causing collisions
* update main group
* remove temp group
This PR adds a machine that will, every 5 minutes, look at the GCS
bucket that stores Bazel metrics and push whatever it finds to
ElasticSearch.
A huge part of this commit is based on @aherrmann-da's work. You can
assume that all the good bits are his.
CHANGELOG_BEGIN
CHANGELOG_END
This PR adds a Kibana instance to each ES node, and duplicates the load
balancer mechanism to expose both raw ES and Kibana.
CHANGELOG_BEGIN
CHANGELOG_END
This PR adds a basic ES cluster to our infrastructure, completely open
and unprotected but only accessible through VPN.
And, as of yet, through its IP address. I'm not sure whether it's worth
adding a DNS for it.
CHANGELOG_BEGIN
CHANGELOG_END
This morning we started with very restricted CI pools (2/6 for Windows
and 7/20 for Linux), apparently because the region we run in (us-east1)
has three zones, two of them were unable to allocate new nodes, and the
default policy is to distribute nodes evenly between zones.
I've manually changed the distribution policy. Unfortunately this option
is not yet available in our version of the GCP Terraform plugin.
CHANGELOG_BEGIN
CHANGELOG_END
We've recently seen a few cases where the macOS nodes ended up not
having the cache partition mounted. So far this has only happened on
semi-broken nodes (guest VM still up and running but host unable to
connect to it), so I haven't been able to actually poke at a broken
machine, but I believe this should allow a machine in such a state to
recover.
While we haven't observed a similar issue on Linux nodes (as far as I'm
aware), I have made similar changes there to keep both scripts in sync.
CHANGELOG_BEGIN
CHANGELOG_END
This includes https://github.com/ndmitchell/hoogle/pull/367.
As usual, I unfortunately cannot test this myself so please review
carefully.
Note that this will slightly increase compile times since we will now
build hoogle. However, we still only build hoogle rather than
everything which takes less than 2min on my very weak personal
laptop. We could integrate this with our nix cache but for now, I’m
not that worried about this.
changelog_begin
changelog_end
Co-authored-by: Gary Verhaegen <gary.verhaegen@digitalasset.com>
After #9362, hoogle stopped returning any result. It was up and running,
but with an empty database.
Problem was two-fold:
1. In the switch to `nix`, I lost track of the fact that we were
previously doing all the work in `~/hoogle` rather than `~/`, which
broke the periodic refresh.
2. The initial setup has been broken for months; machines have been
initializing with an empty DB and only getting data on the first
refresh since #7370.
CHANGELOG_BEGIN
CHANGELOG_END
This is a demonstration, commit-by-commit, of how to use the blue/green
deployment for Hoogle servers.
The actual change (using nix) is stolen from #9352.
CHANGELOG_BEGIN
CHANGELOG_END
This is adapting the same approach as #9137 to the macOS machines. The
setup is very similar, except macOS apparently doesn't require any kind
of `sudo` access in the process.
The main reason for the change here is that while `~/.bazel-cache` is
reasonably fast to clean, cleaning just that has finally caught up to us
with a recent cleanup step that proudly claimed:
```
before: 638Mi free
after: 1.2Gi free
```
So we do need to start cleaning the other one after all.
CHANGELOG_BEGIN
CHANGELOG_END
This is a continuation of #8595 and #8599. I somehow had missed that
`/etc/fstab` can be used to tell `mount` to let users mount some
filesystems with preset options.
This is using the full history of `mount` hardening so should be safe
enough. The option `user` in `/etc/fstab` automatically disables any kind
of `setuid` feature on the mounted filesystem, which is the main attack
vector I know of.
This works flawlessly on my local VM, so hopefully this time's the
charm. (It also happens to be my third PR specifically targeted on this
issue, so, who knows, it may even work.)
CHANGELOG_BEGIN
CHANGELOG_END
* fixup terraform config
Two changes have happened recently that have invalidated the current
Terraform files:
1. The Terraform version has gone through a major, incompatible upgrade
(#8190); the required updates for this are reflected in the first
commit of this PR.
2. The certificate used to serve [Hoogle](https://hoogle.daml.com) was
about to expire, so Edward created a new one and updated the config
directly. The second commit in this PR updates the Terraform config
to match that new, already-in-prod setting.
Note: This PR applies cleanly, as there are no resulting changes in
Terraform's perception of the target state from 1, and the change from 2
has already been applied through other channels.
CHANGELOG_BEGIN
CHANGELOG_END
* update hoogle cert
CHANGELOG_BEGIN
- Our Linux binaries are now built on Ubuntu 20.04 instead of 16.04. We
do not expect any user-level impact, but please reach out if you
do notice any issue that might be caused by this.
CHANGELOG_END
* Replace many occurrences of DAML with Daml
* Update docs logo
* A few more CLI occurrences
CHANGELOG_BEGIN
- Change DAML capitalization and docs logo
CHANGELOG_END
* Fix some over-eager replacements
* A few mor occurrences in md files
* Address comments in *.proto files
* Change case in comments and strings in .ts files
* Revert changes to frozen proto files
* Also revert LF 1.11
* Update get-daml.sh
* Update windows installer
* Include .py files
* Include comments in .daml files
* More instances in the assistant CLI
* some more help texts
Our CI nodes install nix in multi-user mode. This means that changing
cache information is only available to certain trusted users for
security reasons. The CI user is not part of those so the cache info
from dev-env/etc/nix.conf is silently ignored.
We could consider not running in multi-user mode although from a
security pov this seems like a pretty sensible decision and our
signing keys change very rarely so for now, I would keep it.
changelog_begin
changelog_end
As we strive for more inclusiveness, we are becoming less comfortable
with historically-charged terms being used in our everyday work.
This is targeted for merge on Dec 26, _after_ the necessary
corresponding changes at both the GitHub and Azure Pipelines levels.
CHANGELOG_BEGIN
- DAML Connect development is now conducted from the `main` branch,
rather than the `master` one. If you had any dependency on the
digital-asset/daml repository, you will need to update this parameter.
CHANGELOG_END
This is far from perfect but removes the blatantly wrong sections of the
README.
Note: as a README change, this is not really a standard change, but
because the README is under the infra folder, this PR does need the tag
to pass CI.
CHANGELOG_BEGIN
CHANGELOG_END
incident-118: fruitless investigation; revert
This first commit just duplicates the existing configuration. Further
commits will make actual changes so they can be tracked by looking at
individual commits (rather than try to think up the diff by looking at
entirely new files).
CHANGELOG_BEGIN
CHANGELOG_END
This is the macOS part of #5912, which I have separated because our
macOS nodes have a different deployment process so it seemed easier to
track the deployment of the change separately.
CHANGELOG_BEGIN
CHANGELOG_END
