enso/.github/workflows/scala-new.yml

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

594 lines
26 KiB
YAML
Raw Normal View History

2023-01-27 03:09:09 +03:00
# This file is auto-generated. Do not edit it manually!
# Edit the enso_build::ci_gen module instead and run `cargo run --package enso-build-ci-gen`.
2022-07-01 04:58:14 +03:00
name: Engine CI
on:
push:
2022-07-01 04:58:14 +03:00
branches:
- develop
pull_request: {}
workflow_dispatch:
inputs:
clean_build_required:
description: Clean before and after the run.
required: false
type: boolean
default: false
jobs:
enso-build-ci-gen-job-cancel-workflow-linux-x86_64:
2022-07-01 04:58:14 +03:00
name: Cancel Previous Runs
if: github.ref != 'refs/heads/develop'
runs-on:
- ubuntu-latest
steps:
- name: Cancel Previous Runs
uses: styfle/cancel-workflow-action@0.12.1
with:
access_token: ${{ github.token }}
2023-07-26 21:13:34 +03:00
permissions:
actions: write
enso-build-ci-gen-job-ci-check-backend-linux-x86_64:
name: Engine (linux, x86_64)
runs-on:
- self-hosted
2022-07-01 04:58:14 +03:00
- Linux
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
2023-10-17 01:59:52 +03:00
uses: s-weigand/setup-conda@v1.2.1
2022-07-01 04:58:14 +03:00
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
2022-07-01 04:58:14 +03:00
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
2022-07-01 04:58:14 +03:00
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
2022-07-01 04:58:14 +03:00
- name: Checking out the repository
2023-10-17 01:59:52 +03:00
uses: actions/checkout@v4
2022-07-01 04:58:14 +03:00
with:
clean: false
submodules: recursive
2022-07-01 04:58:14 +03:00
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
2022-07-01 04:58:14 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: ./run backend ci-check
2022-07-01 04:58:14 +03:00
env:
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
enso-build-ci-gen-job-ci-check-backend-macos-x86_64:
name: Engine (macos, x86_64)
runs-on:
- macos-latest
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
uses: s-weigand/setup-conda@v1.2.1
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
- name: Checking out the repository
uses: actions/checkout@v4
with:
clean: false
submodules: recursive
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: ./run backend ci-check
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
enso-build-ci-gen-job-ci-check-backend-windows-x86_64:
name: Engine (windows, x86_64)
runs-on:
- self-hosted
- Windows
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
uses: s-weigand/setup-conda@v1.2.1
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
- name: Checking out the repository
uses: actions/checkout@v4
with:
clean: false
submodules: recursive
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: ./run backend ci-check
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
enso-build-ci-gen-job-scala-tests-linux-x86_64:
name: Scala Tests (linux, x86_64)
runs-on:
- self-hosted
- Linux
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
uses: s-weigand/setup-conda@v1.2.1
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
- name: Checking out the repository
uses: actions/checkout@v4
with:
clean: false
submodules: recursive
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: ./run backend test scala
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: success() || failure()
name: Engine Test Reporter
2022-07-01 04:58:14 +03:00
uses: dorny/test-reporter@v1
with:
max-annotations: 50
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
name: Engine Tests Report (linux, x86_64)
path: ${{ env.ENSO_TEST_JUNIT_DIR }}/*.xml
path-replace-backslashes: true
reporter: java-junit
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
permissions:
checks: write
enso-build-ci-gen-job-scala-tests-macos-x86_64:
name: Scala Tests (macos, x86_64)
runs-on:
- macos-latest
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
uses: s-weigand/setup-conda@v1.2.1
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
- name: Checking out the repository
uses: actions/checkout@v4
with:
clean: false
submodules: recursive
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: ./run backend test scala
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: success() || failure()
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
name: Engine Test Reporter
uses: dorny/test-reporter@v1
with:
max-annotations: 50
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
name: Engine Tests Report (macos, x86_64)
path: ${{ env.ENSO_TEST_JUNIT_DIR }}/*.xml
path-replace-backslashes: true
2022-07-01 04:58:14 +03:00
reporter: java-junit
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2023-07-26 21:13:34 +03:00
permissions:
checks: write
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
enso-build-ci-gen-job-scala-tests-windows-x86_64:
name: Scala Tests (windows, x86_64)
2022-07-01 04:58:14 +03:00
runs-on:
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
- self-hosted
- Windows
2022-07-01 04:58:14 +03:00
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
2023-10-17 01:59:52 +03:00
uses: s-weigand/setup-conda@v1.2.1
2022-07-01 04:58:14 +03:00
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
2022-07-01 04:58:14 +03:00
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
2022-07-01 04:58:14 +03:00
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
2022-07-01 04:58:14 +03:00
- name: Checking out the repository
2023-10-17 01:59:52 +03:00
uses: actions/checkout@v4
with:
clean: false
submodules: recursive
2022-07-01 04:58:14 +03:00
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
2022-07-01 04:58:14 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
- run: ./run backend test scala
2022-07-01 04:58:14 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: success() || failure()
name: Engine Test Reporter
2022-07-01 04:58:14 +03:00
uses: dorny/test-reporter@v1
with:
max-annotations: 50
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
name: Engine Tests Report (windows, x86_64)
path: ${{ env.ENSO_TEST_JUNIT_DIR }}/*.xml
path-replace-backslashes: true
reporter: java-junit
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
permissions:
checks: write
enso-build-ci-gen-job-standard-library-tests-linux-x86_64:
name: Standard Library Tests (linux, x86_64)
runs-on:
- self-hosted
- Linux
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
uses: s-weigand/setup-conda@v1.2.1
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
- name: Checking out the repository
uses: actions/checkout@v4
with:
clean: false
submodules: recursive
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: ./run backend test standard-library
env:
AWS_ACCESS_KEY_ID: ${{ secrets.ENSO_LIB_S3_AWS_ACCESS_KEY_ID }}
AWS_REGION: ${{ secrets.ENSO_LIB_S3_AWS_REGION }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.ENSO_LIB_S3_AWS_SECRET_ACCESS_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: success() || failure()
name: Standard Library Test Reporter
uses: dorny/test-reporter@v1
with:
max-annotations: 50
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
name: Standard Library Tests Report (linux, x86_64)
path: ${{ env.ENSO_TEST_JUNIT_DIR }}/*/*.xml
path-replace-backslashes: true
2022-07-01 04:58:14 +03:00
reporter: java-junit
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2023-07-26 21:13:34 +03:00
permissions:
checks: write
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
enso-build-ci-gen-job-standard-library-tests-macos-x86_64:
name: Standard Library Tests (macos, x86_64)
2022-07-01 04:58:14 +03:00
runs-on:
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
- macos-latest
2022-07-01 04:58:14 +03:00
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
2023-10-17 01:59:52 +03:00
uses: s-weigand/setup-conda@v1.2.1
2022-07-01 04:58:14 +03:00
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
2022-07-01 04:58:14 +03:00
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
2022-07-01 04:58:14 +03:00
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
2022-07-01 04:58:14 +03:00
- name: Checking out the repository
2023-10-17 01:59:52 +03:00
uses: actions/checkout@v4
2022-07-01 04:58:14 +03:00
with:
clean: false
submodules: recursive
2022-07-01 04:58:14 +03:00
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
2022-07-01 04:58:14 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
- run: ./run backend test standard-library
2022-07-01 04:58:14 +03:00
env:
AWS_ACCESS_KEY_ID: ${{ secrets.ENSO_LIB_S3_AWS_ACCESS_KEY_ID }}
AWS_REGION: ${{ secrets.ENSO_LIB_S3_AWS_REGION }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.ENSO_LIB_S3_AWS_SECRET_ACCESS_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: success() || failure()
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
name: Standard Library Test Reporter
2022-07-01 04:58:14 +03:00
uses: dorny/test-reporter@v1
with:
max-annotations: 50
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
name: Standard Library Tests Report (macos, x86_64)
path: ${{ env.ENSO_TEST_JUNIT_DIR }}/*/*.xml
path-replace-backslashes: true
reporter: java-junit
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
permissions:
checks: write
enso-build-ci-gen-job-standard-library-tests-windows-x86_64:
name: Standard Library Tests (windows, x86_64)
runs-on:
- self-hosted
- Windows
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
uses: s-weigand/setup-conda@v1.2.1
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
- name: Checking out the repository
uses: actions/checkout@v4
with:
clean: false
submodules: recursive
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: ./run backend test standard-library
env:
AWS_ACCESS_KEY_ID: ${{ secrets.ENSO_LIB_S3_AWS_ACCESS_KEY_ID }}
AWS_REGION: ${{ secrets.ENSO_LIB_S3_AWS_REGION }}
AWS_SECRET_ACCESS_KEY: ${{ secrets.ENSO_LIB_S3_AWS_SECRET_ACCESS_KEY }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: success() || failure()
name: Standard Library Test Reporter
uses: dorny/test-reporter@v1
with:
max-annotations: 50
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
name: Standard Library Tests Report (windows, x86_64)
path: ${{ env.ENSO_TEST_JUNIT_DIR }}/*/*.xml
path-replace-backslashes: true
2022-07-01 04:58:14 +03:00
reporter: java-junit
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2023-07-26 21:13:34 +03:00
permissions:
checks: write
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
enso-build-ci-gen-job-verify-license-packages-linux-x86_64:
name: Verify License Packages (linux, x86_64)
runs-on:
- self-hosted
- Linux
steps:
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Setup conda (GH runners only)
uses: s-weigand/setup-conda@v1.2.1
with:
update-conda: false
conda-channels: anaconda, conda-forge
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
name: Installing wasm-pack
uses: jetli/wasm-pack-action@v0.4.0
with:
version: v0.10.2
- name: Expose Artifact API and context information.
uses: actions/github-script@v7
with:
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
- name: Checking out the repository
uses: actions/checkout@v4
with:
clean: false
submodules: recursive
- name: Build Script Setup
run: ./run --help || (git clean -ffdx && ./run --help)
[CI] Engine CI Rework, Part 1 (#9295) I have created PR with the first set of changes for the Engine CI. The changes are small and effectively consist of: 1. Spltting the `verifyLicensePackages`. It is now run only on Linux. There are hardly any time benefits, as the actual job cost is dominated by the overhead of spinning a new job — but it is not expensive in the big picture. 2. Splitting the Scala Tests into separate job. This is probably the biggest "atomic" piece of work we have. 3. Splitting the Standard Library Tests into a separate job. The time is nicely split across the jobs now. The last run has: * 27 min for Scala tests; * 25 min for Standard Library tests; * 24 min for the "rest": the old job containing everything that has not been split. While total CPU time has increased (as jobs are not effectively reusing the same build context), the wall time has decreased significantly. Previously we had ~1 hour of wall time for the old monolithic job, so we are getting more than 2x speedup. The now-slowest Scala tests job is currently comparable with the native Rust tests (and they should improve when the old gui is gone) — which are the slowest job across all CI checks. The PR is pretty minimal. Several future improvements can be made: * Reorganizing and splitting other "heavy" jobs, like the native image generation. * Reusing the built Engine distribution. However, this is probably a lower priority than I initially thought. * Building package takes several minutes, so duplicating this job is not that expensive. * The package is OS-specific. * Scala tests don't really benefit from it, they'd need way more compilation artifacts. It'd make sense to reuse the distribution if we, for example, decided to split more jobs that actually benefit from it, like Standard Library tests. * Reusing the Rust build script binary. * As our self-hosted runners reuse environment, we effectively get this for free. Especially when Rust part of codebase is less frequently changed. * This is however significant cost for the GitHub-hosted runners, affecting our macOS runners. Reusing the binary does not save wall time for jobs that are run in parallel (as we have enough runners), but if we introduce job dependencies that'd force sequential execution of jobs on macOS, this would be a significant need.
2024-03-06 21:56:13 +03:00
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: "(contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean before
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- run: ./run backend sbt '--' verifyLicensePackages
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
- if: failure() && runner.os == 'Windows'
name: List files if failed (Windows)
run: Get-ChildItem -Force -Recurse
- if: failure() && runner.os != 'Windows'
name: List files if failed (non-Windows)
run: ls -lAR
- if: "(always()) && (contains(github.event.pull_request.labels.*.name, 'CI: Clean build required') || inputs.clean_build_required)"
name: Clean after
run: ./run git-clean
env:
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
2022-07-01 04:58:14 +03:00
env:
ENSO_BUILD_SKIP_VERSION_CHECK: "true"