2021-10-30 18:06:21 +03:00
|
|
|
name: GUI CI
|
2022-08-28 15:44:10 +03:00
|
|
|
on:
|
2021-10-30 18:06:21 +03:00
|
|
|
push:
|
|
|
|
branches:
|
|
|
|
- develop
|
|
|
|
pull_request: {}
|
|
|
|
workflow_dispatch: {}
|
|
|
|
jobs:
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-build-backend-linux:
|
|
|
|
name: Build Backend (linux)
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
|
|
|
- self-hosted
|
|
|
|
- Linux
|
|
|
|
- engine
|
2021-10-30 18:06:21 +03:00
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
2021-10-30 18:06:21 +03:00
|
|
|
with:
|
2022-05-23 05:16:04 +03:00
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2021-10-30 18:06:21 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run backend get
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-build-backend-macos:
|
|
|
|
name: Build Backend (macos)
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
2022-05-25 04:19:45 +03:00
|
|
|
- macos-latest
|
2021-10-30 18:06:21 +03:00
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2021-10-30 18:06:21 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run backend get
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-build-backend-windows:
|
|
|
|
name: Build Backend (windows)
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
|
|
|
- self-hosted
|
2022-05-25 04:19:45 +03:00
|
|
|
- Windows
|
2022-05-23 05:16:04 +03:00
|
|
|
- engine
|
2021-10-30 18:06:21 +03:00
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2021-10-30 18:06:21 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run backend get
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-build-wasm-linux:
|
2022-05-23 05:16:04 +03:00
|
|
|
name: Build GUI (WASM) (linux)
|
|
|
|
runs-on:
|
|
|
|
- self-hosted
|
|
|
|
- Linux
|
|
|
|
- engine
|
2021-10-30 18:06:21 +03:00
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2021-10-30 18:06:21 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run --upload-artifacts ${{ runner.os == 'Linux' }} wasm build
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-build-wasm-macos:
|
2022-05-25 04:19:45 +03:00
|
|
|
name: Build GUI (WASM) (macos)
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
2022-05-25 04:19:45 +03:00
|
|
|
- macos-latest
|
2021-10-30 18:06:21 +03:00
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2021-10-30 18:06:21 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run --upload-artifacts ${{ runner.os == 'Linux' }} wasm build
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-build-wasm-windows:
|
2022-05-25 04:19:45 +03:00
|
|
|
name: Build GUI (WASM) (windows)
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
|
|
|
- self-hosted
|
2022-05-25 04:19:45 +03:00
|
|
|
- Windows
|
2022-05-23 05:16:04 +03:00
|
|
|
- engine
|
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2022-05-23 05:16:04 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run --upload-artifacts ${{ runner.os == 'Linux' }} wasm build
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-cancel-workflow-linux:
|
2022-05-25 04:19:45 +03:00
|
|
|
name: Cancel Previous Runs
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
2022-05-25 04:19:45 +03:00
|
|
|
- X64
|
|
|
|
steps:
|
|
|
|
- name: Cancel Previous Runs
|
|
|
|
uses: styfle/cancel-workflow-action@0.9.1
|
|
|
|
with:
|
2022-08-28 15:44:10 +03:00
|
|
|
access_token: ${{ github.token }}
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-lint-linux:
|
2022-05-25 04:19:45 +03:00
|
|
|
name: Lint (linux)
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
|
|
|
- self-hosted
|
2022-05-25 04:19:45 +03:00
|
|
|
- Linux
|
2022-05-23 05:16:04 +03:00
|
|
|
- engine
|
2021-10-30 18:06:21 +03:00
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2022-05-23 05:16:04 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run lint
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-native-test-linux:
|
2022-05-25 04:19:45 +03:00
|
|
|
name: Native GUI tests (linux)
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
2022-05-25 04:19:45 +03:00
|
|
|
- self-hosted
|
|
|
|
- Linux
|
|
|
|
- engine
|
2022-05-23 05:16:04 +03:00
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2021-10-30 18:06:21 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run wasm test --no-wasm
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-package-ide-linux:
|
2022-05-25 04:19:45 +03:00
|
|
|
name: Package IDE (linux)
|
|
|
|
needs:
|
2022-07-01 04:58:14 +03:00
|
|
|
- enso-build-cli-ci-gen-job-build-backend-linux
|
|
|
|
- enso-build-cli-ci-gen-job-build-wasm-linux
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
|
|
|
- self-hosted
|
2022-05-25 04:19:45 +03:00
|
|
|
- Linux
|
2022-05-23 05:16:04 +03:00
|
|
|
- engine
|
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2022-05-23 05:16:04 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run ide build --wasm-source current-ci-run --backend-source current-ci-run
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-package-ide-macos:
|
2022-05-23 05:16:04 +03:00
|
|
|
name: Package IDE (macos)
|
2021-10-30 18:06:21 +03:00
|
|
|
needs:
|
2022-07-01 04:58:14 +03:00
|
|
|
- enso-build-cli-ci-gen-job-build-backend-macos
|
2022-07-05 23:03:29 +03:00
|
|
|
- enso-build-cli-ci-gen-job-build-wasm-linux
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
|
|
|
- macos-latest
|
2021-10-30 18:06:21 +03:00
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2021-10-30 18:06:21 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run ide build --wasm-source current-ci-run --backend-source current-ci-run
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-09-19 22:02:18 +03:00
|
|
|
APPLEID: ${{ secrets.APPLE_NOTARIZATION_USERNAME }}
|
|
|
|
APPLEIDPASS: ${{ secrets.APPLE_NOTARIZATION_PASSWORD }}
|
|
|
|
CSC_IDENTITY_AUTO_DISCOVERY: "true"
|
|
|
|
CSC_KEY_PASSWORD: ${{ secrets.APPLE_CODE_SIGNING_CERT_PASSWORD }}
|
|
|
|
CSC_LINK: ${{ secrets.APPLE_CODE_SIGNING_CERT }}
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-package-ide-windows:
|
2022-05-25 04:19:45 +03:00
|
|
|
name: Package IDE (windows)
|
|
|
|
needs:
|
2022-07-01 04:58:14 +03:00
|
|
|
- enso-build-cli-ci-gen-job-build-backend-windows
|
2022-07-05 23:03:29 +03:00
|
|
|
- enso-build-cli-ci-gen-job-build-wasm-linux
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
|
|
|
- self-hosted
|
2022-05-25 04:19:45 +03:00
|
|
|
- Windows
|
2022-05-23 05:16:04 +03:00
|
|
|
- engine
|
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2022-05-23 05:16:04 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run ide build --wasm-source current-ci-run --backend-source current-ci-run
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
|
|
|
WIN_CSC_KEY_PASSWORD: ${{ secrets.MICROSOFT_CODE_SIGNING_CERT_PASSWORD }}
|
|
|
|
WIN_CSC_LINK: ${{ secrets.MICROSOFT_CODE_SIGNING_CERT }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-07-01 04:58:14 +03:00
|
|
|
enso-build-cli-ci-gen-job-wasm-test-linux:
|
2022-05-25 04:19:45 +03:00
|
|
|
name: WASM GUI tests (linux)
|
2022-05-23 05:16:04 +03:00
|
|
|
runs-on:
|
|
|
|
- self-hosted
|
|
|
|
- Linux
|
|
|
|
- engine
|
|
|
|
steps:
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Setup conda (GH runners only)
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: s-weigand/setup-conda@v1.0.5
|
|
|
|
with:
|
|
|
|
update-conda: false
|
2022-08-28 15:44:10 +03:00
|
|
|
conda-channels: anaconda, conda-forge
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: startsWith(runner.name, 'GitHub Actions') || startsWith(runner.name, 'Hosted Agent')
|
|
|
|
name: Installing wasm-pack
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: jetli/wasm-pack-action@v0.3.0
|
|
|
|
with:
|
2022-06-01 14:44:40 +03:00
|
|
|
version: v0.10.2
|
2022-10-11 00:38:48 +03:00
|
|
|
- name: Expose Artifact API and context information.
|
2022-05-23 05:16:04 +03:00
|
|
|
uses: actions/github-script@v6
|
|
|
|
with:
|
2022-10-11 00:38:48 +03:00
|
|
|
script: "\n core.exportVariable(\"ACTIONS_RUNTIME_TOKEN\", process.env[\"ACTIONS_RUNTIME_TOKEN\"])\n core.exportVariable(\"ACTIONS_RUNTIME_URL\", process.env[\"ACTIONS_RUNTIME_URL\"])\n core.exportVariable(\"GITHUB_RETENTION_DAYS\", process.env[\"GITHUB_RETENTION_DAYS\"])\n console.log(context)\n "
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os == 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: '"c:\Program Files\Git\bin\bash.exe" -c "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"'
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: cmd
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: runner.os != 'Windows'
|
|
|
|
name: Workaround for https://github.com/actions/checkout/issues/590 (non-Windows)
|
2022-10-11 00:38:48 +03:00
|
|
|
run: "git checkout -f $(git -c user.name=x -c user.email=x@x commit-tree $(git hash-object -t tree /dev/null) < /dev/null) || :"
|
2022-07-27 05:37:47 +03:00
|
|
|
shell: bash
|
2022-05-23 05:16:04 +03:00
|
|
|
- name: Checking out the repository
|
2022-07-01 04:58:14 +03:00
|
|
|
uses: actions/checkout@v2
|
2022-05-23 05:16:04 +03:00
|
|
|
with:
|
2022-08-25 01:46:36 +03:00
|
|
|
clean: false
|
2022-09-02 23:02:44 +03:00
|
|
|
submodules: recursive
|
2022-07-01 04:58:14 +03:00
|
|
|
- name: Build Script Setup
|
2022-08-28 15:44:10 +03:00
|
|
|
run: ./run --help
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: "contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean before
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
2022-08-28 15:44:10 +03:00
|
|
|
- run: ./run wasm test --no-native
|
2022-06-01 14:44:40 +03:00
|
|
|
env:
|
2022-08-28 15:44:10 +03:00
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
|
|
|
- if: failure() && runner.os == 'Windows'
|
|
|
|
name: List files if failed (Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: Get-ChildItem -Force -Recurse
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: failure() && runner.os != 'Windows'
|
|
|
|
name: List files if failed (non-Windows)
|
2022-08-10 04:41:44 +03:00
|
|
|
run: ls -lAR
|
2022-11-09 02:15:26 +03:00
|
|
|
- if: "always() && always() && contains(github.event.pull_request.labels.*.name, 'CI: Clean build required')"
|
|
|
|
name: Clean after
|
2022-10-11 00:38:48 +03:00
|
|
|
run: ./run git-clean
|
|
|
|
env:
|
|
|
|
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
|
2022-11-09 02:15:26 +03:00
|
|
|
timeout-minutes: 360
|
Better `release` build time; new maximum-performance `production` profile. (#3498)
### Pull Request Description
Using the new tooling (#3491), I investigated the **performance / compile-time tradeoff** of different codegen options for release mode builds. By scripting the testing procedure, I was able to explore many possible combinations of options, which is important because their interactions (on both application performance and build time) are complex. I found **two candidate profiles** that offer specific advantages over the current `release` settings (`baseline`):
- `thin16`: Supports incremental compiles in 1/3 the time of `baseline` in common cases. Application runs about 2% slower than `baseline`.
- `fat1-O4`: Application performs 13% better than `baseline`. Compile time is almost 3x `baseline`, and non-incremental.
(See key in first chart for the settings defining these profiles.)
We can build faster or run faster, though not in the same build. Because the effect sizes are large enough to be impactful to developer and user experience, respectively, I think we should consider having it both ways. We could **split the `release` profile** into two profiles to serve different purposes:
- `release`: A profile that supports fast developer iteration, while offering realistic performance.
- `production`: A maximally-optimized profile, for nightly builds and actual releases.
Since `wasm-pack` doesn't currently support custom profiles (rustwasm/wasm-pack#1111), we can't use a Cargo profile for `production`; however, we can implement our own profile by overriding rustc flags.
### Performance details
![perf](https://user-images.githubusercontent.com/1047859/170788530-ab6d7910-5253-4a2b-b432-8bfa0b4735ba.png)
As you can see, `thin16` is slightly slower than `baseline`; `fat1-O4` is dramatically faster.
<details>
<summary>Methodology (click to show)</summary>
I developed a procedure for benchmarking "whole application" performance, using the new "open project" workflow (which opens the IDE and loads a complex project), and some statistical analysis to account for variance. To gather this data:
Build the application with profiling:
`./run.sh ide build --profiling-level=debug`
Run the `open_project` workflow repeatedly:
`for i in $(seq 0 9); do dist/ide/linux-unpacked/enso --entry-point profile --workflow open_project --save-profile open_project_thin16_${i}.json; done`
For each profile recorded, take the new `total_self_time` output of the `intervals` tool; gather into CSV:
`echo $(for i in $(seq 0 9); do target/rust/debug/intervals < open_project_thin16_${i}.json | tail -n1 | awk '{print $2}'; do`
(Note that the output of intervals should not be considered stable; this command may need modification in the future. Eventually it would be nice to support formatted outputs...)
The data is ready to graph. I used the `boxplot` method of the [seaborn](https://seaborn.pydata.org/index.html) package, in order to show the distribution of data.
</details>
#### Build times
![thin16](https://user-images.githubusercontent.com/1047859/170788539-1578e41b-bc30-4f30-9b71-0b0181322fa5.png)
In the case of changing a file in `enso-prelude`, with the current `baseline` settings rebuilding takes over 3 minutes. With the `thin16` settings, the same rebuild completes in 40 seconds.
(To gather this data on different hardware or in the future, just run the new `bench-build.sh` script for each case to be measured.)
2022-06-11 01:09:54 +03:00
|
|
|
env:
|
|
|
|
ENSO_BUILD_SKIP_VERSION_CHECK: "true"
|