mirror of
https://github.com/digital-asset/daml.git
synced 2024-09-20 01:07:18 +03:00
df0086d26f
It does not seem like CI machines recover from a failed clean-up. This is not the most elegant solution possible, but it's a cheap one that should work. Not: shutting down the machine in the middle of the build will not provide an error message to Slack for main branch builds (because the `tell_slack_failed` step would need to run on the same machine) but will correctly report failure for PRs (that was the original purpose of the `collect_build_data` step). An alternative here would be to give a delay to the shutdown command, and try to calibrate it so that it's long enough for this job to correctly report its failure to both Azure and Slack, while making it short enough that no other job gets assigned to the machine. I'm not clear enough on how often Azure assigns jobs to try and bet on that. CHANGELOG_BEGIN CHANGELOG_END
35 lines
1.2 KiB
YAML
35 lines
1.2 KiB
YAML
# Copyright (c) 2021 Digital Asset (Switzerland) GmbH and/or its affiliates. All rights reserved.
|
|
# SPDX-License-Identifier: Apache-2.0
|
|
|
|
steps:
|
|
- bash: |
|
|
set -euo pipefail
|
|
# Location of the disk cache for CI servers set in their init files:
|
|
# infra/macos/2-common-box/init.sh:echo "build:darwin --disk_cache=~/.bazel-cache" > ~/.bazelrc
|
|
# infra/vsts_agent_linux_startup.sh:echo "build:linux --disk_cache=~/.bazel-cache" > ~/.bazelrc
|
|
|
|
# Linux machines don't seem to recover when this script fails, and they get
|
|
# renewed by the instance_group
|
|
if [ "$(uname -s)" == "Linux" ]; then
|
|
trap "shutdown -h now" EXIT
|
|
fi
|
|
|
|
if [ $(df -m . | sed 1d | awk '{print $4}') -lt 50000 ]; then
|
|
echo "Disk full, cleaning up..."
|
|
disk_cache="$HOME/.bazel-cache"
|
|
rm -rf "$disk_cache"
|
|
echo "removed '$disk_cache'"
|
|
local_cache="$HOME/.cache/bazel"
|
|
if [ -d "$local_cache" ]; then
|
|
for pid in $(pgrep -a -f bazel | awk '{print $1}'); do
|
|
kill -s KILL $pid
|
|
done
|
|
chmod -R +w "$local_cache"
|
|
rm -rf "$local_cache"
|
|
echo "removed '$local_cache'"
|
|
fi
|
|
fi
|
|
df -h .
|
|
trap - EXIT
|
|
displayName: clean-up disk cache
|