1
1
mirror of https://github.com/anoma/juvix.git synced 2024-12-16 10:56:14 +03:00
juvix/.github/workflows/ci.yml

312 lines
8.4 KiB
YAML
Raw Normal View History

2023-04-21 20:05:24 +03:00
name: Juvix Compiler CI
"on":
workflow_dispatch:
2023-01-19 17:42:56 +03:00
inputs:
ref:
description: the repository ref to build
required: true
default: main
2022-08-04 20:12:49 +03:00
push:
branches:
- main
pull_request:
branches:
- main
types:
- opened
- reopened
- synchronize
- ready_for_review
Add `juvix format` command (#1886) This PR adds `juvix format` that can be used to format either a single Juvix file or all files in a Juvix project. ## Usage ``` $ juvix format --help Usage: juvix format JUVIX_FILE_OR_PROJECT [--check] [--in-place] Format a Juvix file or Juvix project When the command is run with an unformatted file it prints the reformatted source to standard output. When the command is run with a project directory it prints a list of unformatted files in the project. Available options: JUVIX_FILE_OR_PROJECT Path to a .juvix file or to a directory containing a Juvix project. --check Do not print reformatted sources or unformatted file paths to standard output. --in-place Do not print reformatted sources to standard output. Overwrite the target's contents with the formatted version if the formatted version differs from the original content. -h,--help Show this help text ``` ## Location of main implementation The implementation is split into two components: * The src API: `format` and `formatProject` https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/src/Juvix/Formatter.hs * The CLI interface: https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/app/Commands/Format.hs ## in-place uses polysemy Resource effect The `--in-place` option makes a backup of the target file and restores it if there's an error during processing to avoid data loss. The implementation of this uses the polysemy [Resource effect](https://hackage.haskell.org/package/polysemy-1.9.0.0/docs/Polysemy-Resource.html). The recommended way to interpret the resource effect is to use `resourceToIOFinal` which makes it necessary to change the effects interpretation in main to use `Final IO`: https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/app/Main.hs#L15 ## Format input is `FilePath` The format options uses `FilePath` instead of `AppFile f` for the input file/directory used by other commands. This is because we cannot determine if the input string is a file or directory in the CLI parser (we require IO). I discussed some ideas with @janmasrovira on how to improve this in a way that would also solve other issues with CLI input file/parsing but I want to defer this to a separate PR as this one is already quite large. One consequence of Format using `FilePath` as the input option is that the code that changes the working directory to the root of the project containing the CLI input file is changed to work with `FilePath`: https://github.com/anoma/juvix/blob/f715ef6a531f63c40ac3f629dd9cfea7e867507a/app/TopCommand/Options.hs#L33 ## New dependencies This PR adds new dependencies on `temporary` and `polysemy-zoo`. `temporary` is used for `emptySystemTempFile` in the implementation of the TempFile interpreter for IO: https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/src/Juvix/Data/Effect/Files/IO.hs#L49 `polysemy-zoo` is used for the `Fresh` effect and `absorbMonadThrow` in the implementation of the pure TempFile interpreter: https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/src/Juvix/Data/Effect/Files/Pure.hs#L91 NB: The pure TempFile interpreter is not used, but it seemed a good idea to include it while it's fresh in my mind. * Closes https://github.com/anoma/juvix/issues/1777 --------- Co-authored-by: Jonathan Cubides <jonathan.cubides@uib.no>
2023-03-29 16:51:04 +03:00
concurrency:
group: "${{ github.workflow }}-${{ github.head_ref || github.run_id }}"
cancel-in-progress: true
Add `juvix format` command (#1886) This PR adds `juvix format` that can be used to format either a single Juvix file or all files in a Juvix project. ## Usage ``` $ juvix format --help Usage: juvix format JUVIX_FILE_OR_PROJECT [--check] [--in-place] Format a Juvix file or Juvix project When the command is run with an unformatted file it prints the reformatted source to standard output. When the command is run with a project directory it prints a list of unformatted files in the project. Available options: JUVIX_FILE_OR_PROJECT Path to a .juvix file or to a directory containing a Juvix project. --check Do not print reformatted sources or unformatted file paths to standard output. --in-place Do not print reformatted sources to standard output. Overwrite the target's contents with the formatted version if the formatted version differs from the original content. -h,--help Show this help text ``` ## Location of main implementation The implementation is split into two components: * The src API: `format` and `formatProject` https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/src/Juvix/Formatter.hs * The CLI interface: https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/app/Commands/Format.hs ## in-place uses polysemy Resource effect The `--in-place` option makes a backup of the target file and restores it if there's an error during processing to avoid data loss. The implementation of this uses the polysemy [Resource effect](https://hackage.haskell.org/package/polysemy-1.9.0.0/docs/Polysemy-Resource.html). The recommended way to interpret the resource effect is to use `resourceToIOFinal` which makes it necessary to change the effects interpretation in main to use `Final IO`: https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/app/Main.hs#L15 ## Format input is `FilePath` The format options uses `FilePath` instead of `AppFile f` for the input file/directory used by other commands. This is because we cannot determine if the input string is a file or directory in the CLI parser (we require IO). I discussed some ideas with @janmasrovira on how to improve this in a way that would also solve other issues with CLI input file/parsing but I want to defer this to a separate PR as this one is already quite large. One consequence of Format using `FilePath` as the input option is that the code that changes the working directory to the root of the project containing the CLI input file is changed to work with `FilePath`: https://github.com/anoma/juvix/blob/f715ef6a531f63c40ac3f629dd9cfea7e867507a/app/TopCommand/Options.hs#L33 ## New dependencies This PR adds new dependencies on `temporary` and `polysemy-zoo`. `temporary` is used for `emptySystemTempFile` in the implementation of the TempFile interpreter for IO: https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/src/Juvix/Data/Effect/Files/IO.hs#L49 `polysemy-zoo` is used for the `Fresh` effect and `absorbMonadThrow` in the implementation of the pure TempFile interpreter: https://github.com/anoma/juvix/blob/73952ba15c90f33919c75426cf62ae7e83fd3225/src/Juvix/Data/Effect/Files/Pure.hs#L91 NB: The pure TempFile interpreter is not used, but it seemed a good idea to include it while it's fresh in my mind. * Closes https://github.com/anoma/juvix/issues/1777 --------- Co-authored-by: Jonathan Cubides <jonathan.cubides@uib.no>
2023-03-29 16:51:04 +03:00
env:
SKIP: ormolu,format-juvix-files,typecheck-juvix-examples
VAMPIRREPO: anoma/vamp-ir
VAMPIRVERSION: v0.1.3
jobs:
pre-commit:
2022-04-05 20:57:21 +03:00
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: actions/setup-python@v4
2022-04-05 20:57:21 +03:00
with:
python-version: "3.11"
- uses: pre-commit/action@v3.0.0
ormolu:
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v3
- uses: mrkkrp/ormolu-action@v11
with:
version: 0.5.3.0
extra-args: >-
--ghc-opt -XDerivingStrategies --ghc-opt -XImportQualifiedPost
--ghc-opt -XMultiParamTypeClasses --ghc-opt -XStandaloneDeriving
--ghc-opt -XTemplateHaskell --ghc-opt -XUnicodeSyntax
build-and-test-linux:
runs-on: ubuntu-22.04
steps:
- name: Checkout our repository
uses: actions/checkout@v3
with:
path: main
submodules: true
2022-12-08 19:51:07 +03:00
- name: Cache LLVM and Clang
id: cache-llvm
uses: actions/cache@v3
with:
path: |
C:/Program Files/LLVM
./llvm
key: "${{ runner.os }}-llvm-13"
2022-12-08 19:51:07 +03:00
- name: Install LLVM and Clang
uses: KyleMayes/install-llvm-action@v1
with:
version: "13.0"
cached: "${{ steps.cache-llvm.outputs.cache-hit }}"
2022-12-08 19:51:07 +03:00
- name: Download and extract wasi-sysroot
run: >
curl
https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-15/wasi-sysroot-15.0.tar.gz
-OL
2022-12-08 19:51:07 +03:00
tar xfv wasi-sysroot-15.0.tar.gz
- name: Set WASI_SYSROOT_PATH
run: |
echo "WASI_SYSROOT_PATH=$GITHUB_WORKSPACE/wasi-sysroot" >> $GITHUB_ENV
2022-12-08 19:51:07 +03:00
- name: Add ~/.local/bin to PATH
run: |
echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Install the latest Wasmer version
uses: jaxxstorm/action-install-gh-release@v1.10.0
with:
repo: wasmerio/wasmer
binaries-location: bin
chmod: 0755
- name: Install libs
run: sudo apt install -y libncurses5
- name: Install VampIR for testing
uses: jaxxstorm/action-install-gh-release@v1.10.0
with:
repo: ${{ env.VAMPIRREPO }}
platform: linux
tag: ${{ env.VAMPIRVERSION }}
chmod: 0755
rename-to: vamp-ir
2022-12-08 19:51:07 +03:00
- name: Test VampIR
shell: bash
run: |
vamp-ir --version
2022-12-08 19:51:07 +03:00
- name: Make runtime
2022-12-08 19:51:07 +03:00
run: |
cd main
make runtime
- name: Stack setup
id: stack
uses: freckle/stack-action@v4
with:
working-directory: main
test: false
2022-12-08 19:51:07 +03:00
- name: Install and test Juvix
2022-04-05 20:57:21 +03:00
id: test
if: ${{ success() }}
run: |
cd main
make install
make test
2022-12-08 19:51:07 +03:00
- name: Typecheck and format Juvix examples
if: ${{ success() }}
shell: bash
run: |
cd main
make check-format-juvix-files && make typecheck-juvix-examples
- name: Install Smoke for testing
uses: jaxxstorm/action-install-gh-release@v1.10.0
with:
repo: jonaprieto/smoke
platform: linux
tag: v2.3.2
chmod: 0755
rename-to: smoke
extension-matching: disable
cache: enable
External package dependencies (#2272) This PR adds external git dependency support to the Juvix package format. ## New dependency Git item You can now add a `git` block to the dependencies list: ```yaml name: HelloWorld main: HelloWorld.juvix dependencies: - .juvix-build/stdlib - git: url: https://my.git.repo name: myGitRepo ref: main version: 0.1.0 ``` Git block required fields: * `url`: The URL of the git repository * `ref`: The git reference that should be checked out * `name`: The name for the dependency. This is used to name the directory of the clone, it is required. Perhaps we could come up with a way to automatically name the clone directory. Current ideas are to somehow encode the URL / ref combination or use a UUID. However there's some value in having the clone directory named in a friendly way. NB: * The values of the `name` fields must be unique among the git blocks in the dependencies list. ## Behaviour When dependencies for a package are registered, at the beginning of the compiler pipeline, all remote dependencies are processed: 1. If it doesn't already exist, the remote dependency is cloned to `.juvix-build/deps/$name` 2. `git fetch` is run in the clone 3. `git checkout` at the specified `ref` is run in the clone The clone is then processed by the PathResolver in the same way as path dependencies. NB: * Remote dependencies of transitive dependencies are also processed. * The `git fetch` step is required for the case where the remote is updated. In this case we want the user to be able to update the `ref` field. ## Errors 1. Missing fields in the Git dependency block are YAML parse errors 2. Duplicate `name` values in the dependencies list is an error thrown when the package file is processed 3. The `ref` doesn't exist in the clone or the clone directory is otherwise corrupt. An error with a suggestion to `juvix clean` is given. The package file path is used as the location in the error message. 4. Other `git` command errors (command not found, etc.), a more verbose error is given with the arguments that were passed to the git command. ## Future work 1. Add an offline mode 2. Add a lock file mechanism that resolves branch/tag git refs to commit hashes * closes https://github.com/anoma/juvix/issues/2083 --------- Co-authored-by: Jan Mas Rovira <janmasrovira@gmail.com>
2023-09-01 14:37:06 +03:00
# Smoke tests make git commits
- name: Setup git
shell: bash
run: |
git config --global user.email "tara-juvix@heliax.dev"
git config --global user.name "Tara"
git config --global init.defaultBranch main
- name: Smoke testing
id: smoke-linux
if: ${{ success() }}
run: |
cd main
make smoke-only
build-and-test-macos:
runs-on: macos-12
steps:
- name: Checkout our repository
uses: actions/checkout@v3
with:
path: main
submodules: true
- name: Install Sed
run: |
brew install gnu-sed
echo "$(brew --prefix)/opt/gnu-sed/libexec/gnubin" >> $GITHUB_PATH
- name: Test Sed
run: |
sed --version
- name: Download and extract wasi-sysroot
run: >
curl
https://github.com/WebAssembly/wasi-sdk/releases/download/wasi-sdk-15/wasi-sysroot-15.0.tar.gz
-OL
tar xfv wasi-sysroot-15.0.tar.gz
- name: Set WASI_SYSROOT_PATH
run: |
echo "WASI_SYSROOT_PATH=$GITHUB_WORKSPACE/wasi-sysroot" >> $GITHUB_ENV
- name: Install the latest Wasmer version
uses: jaxxstorm/action-install-gh-release@v1.10.0
with:
repo: wasmerio/wasmer
binaries-location: bin
chmod: 0755
Build and cache smoke binary keyed using icu4c version (#2221) This PR replaces fetching a precompiled binary of smoke with a build/cache for macOS smoke tests on CI. smoke dynamically links to icu4c, so a cached binary of smoke will break when brew bumps the icu4c version. In this PR we use the icu4c version in the cache key of the smoke build to avoid this issue. NB: The smoke build cannot be done as a separate job because the smoke binary must be built using exactly the same version of the macos-12 runner image as the smoke testing step to make sure that the icu4c versions match. Motivation for doing this is this failure: https://github.com/anoma/juvix/actions/runs/5325094406/jobs/9645334642 which uses this release of the runner image https://github.com/actions/runner-images/releases/tag/macOS-12%2F20230618.1 which contains the updated brew version of icu4c. 20230618.1 is currently a prerelease, but will start to run on more jobs shortly. ``` 2023-06-20T17:10:13.2222310Z Copied executables to /Users/runner/.local/bin: 2023-06-20T17:10:13.2223440Z - juvix 2023-06-20T17:10:13.5312790Z dyld[90256]: Library not loaded: '/usr/local/opt/icu4c/lib/libicuuc.72.dylib' 2023-06-20T17:10:13.5331930Z Referenced from: '/Users/runner/hostedtoolcache/jonaprieto/smoke/latest/darwin-x64/smoke' 2023-06-20T17:10:13.5333610Z Reason: tried: '/usr/local/opt/icu4c/lib/libicuuc.72.dylib' (no such file), '/usr/local/lib/libicuuc.72.dylib' (no such file), '/usr/lib/libicuuc.72.dylib' (no such file), '/usr/local/Cellar/icu4c/73.2/lib/libicuuc.72.dylib' (no such file), '/usr/local/lib/libicuuc.72.dylib' (no such file), '/usr/lib/libicuuc.72.dylib' (no such file) 2023-06-20T17:10:13.5334690Z make[1]: *** [smoke-only] Abort trap: 6 2023-06-20T17:10:13.5335310Z make: *** [smoke] Error 2 2023-06-20T17:10:13.5363170Z ##[error]Process completed with exit code 2. ```
2023-06-22 11:10:31 +03:00
# smoke dynamically links to icu4c, so a cached binary of smoke will break
# when brew bumps the icu4c version. In the following steps we use the
# icu4c version in the cache key of the smoke build to avoid this issue.
#
# NB: The smoke build cannot be done as a separate job because the smoke
# binary must be built using exactly the same version of the macos-12
# runner image as the smoke testing step to make sure that the icu4c
# versions match.
- name: Checkout smoke repo
uses: actions/checkout@v3
with:
repository: jonaprieto/smoke
ref: regex-icu
path: smoke-repo
- name: Install ICU4C
run: |
brew install icu4c
brew link icu4c --force
- name: Get ICU4C version
id: icuversion
run: |
ICUVERSION=$(echo -n $(brew list --versions icu4c | head -n 1 | sed -E 's/ /-/g'))
echo "version=$ICUVERSION" >> $GITHUB_OUTPUT
- name: Build smoke
env:
LDFLAGS: -L/usr/local/opt/icu4c/lib
CPPFLAGS: -I/usr/local/opt/icu4c/include
PKG_CONFIG_PATH: /usr/local/opt/icu4c/lib/pkgconfig
uses: freckle/stack-action@v4
with:
test: false
stack-arguments: --copy-bins
working-directory: smoke-repo
cache-prefix: ${{ steps.icuversion.outputs.version }}
pedantic: false
- name: Set homebrew LLVM CC and LIBTOOL vars (macOS)
run: |
echo "CC=$(brew --prefix llvm@15)/bin/clang" >> $GITHUB_ENV
echo "LIBTOOL=$(brew --prefix llvm@15)/bin/llvm-ar" >> $GITHUB_ENV
- name: Make runtime
run: |
cd main
make CC=$CC LIBTOOL=$LIBTOOL runtime
- name: Stack setup
id: stack
uses: freckle/stack-action@v4
with:
working-directory: main
test: false
- name: Add homebrew clang to the PATH (macOS)
run: |
echo "$(brew --prefix llvm@15)/bin" >> $GITHUB_PATH
- name: Install VampIR for testing
uses: jaxxstorm/action-install-gh-release@v1.10.0
with:
repo: ${{ env.VAMPIRREPO }}
platform: darwin
tag: ${{ env.VAMPIRVERSION }}
chmod: 0755
rename-to: vamp-ir
- name: Test VampIR
shell: bash
run: |
vamp-ir --version
- name: Install and test Juvix
if: ${{ success() }}
run: |
cd main
make CC=$CC LIBTOOL=$LIBTOOL install
make CC=$CC LIBTOOL=$LIBTOOL test
- name: Add ~/.local/bin to PATH
run: |
echo "$HOME/.local/bin" >> $GITHUB_PATH
- name: Typecheck and format Juvix examples
if: ${{ success() }}
shell: bash
run: |
cd main
make check-format-juvix-files && make typecheck-juvix-examples
External package dependencies (#2272) This PR adds external git dependency support to the Juvix package format. ## New dependency Git item You can now add a `git` block to the dependencies list: ```yaml name: HelloWorld main: HelloWorld.juvix dependencies: - .juvix-build/stdlib - git: url: https://my.git.repo name: myGitRepo ref: main version: 0.1.0 ``` Git block required fields: * `url`: The URL of the git repository * `ref`: The git reference that should be checked out * `name`: The name for the dependency. This is used to name the directory of the clone, it is required. Perhaps we could come up with a way to automatically name the clone directory. Current ideas are to somehow encode the URL / ref combination or use a UUID. However there's some value in having the clone directory named in a friendly way. NB: * The values of the `name` fields must be unique among the git blocks in the dependencies list. ## Behaviour When dependencies for a package are registered, at the beginning of the compiler pipeline, all remote dependencies are processed: 1. If it doesn't already exist, the remote dependency is cloned to `.juvix-build/deps/$name` 2. `git fetch` is run in the clone 3. `git checkout` at the specified `ref` is run in the clone The clone is then processed by the PathResolver in the same way as path dependencies. NB: * Remote dependencies of transitive dependencies are also processed. * The `git fetch` step is required for the case where the remote is updated. In this case we want the user to be able to update the `ref` field. ## Errors 1. Missing fields in the Git dependency block are YAML parse errors 2. Duplicate `name` values in the dependencies list is an error thrown when the package file is processed 3. The `ref` doesn't exist in the clone or the clone directory is otherwise corrupt. An error with a suggestion to `juvix clean` is given. The package file path is used as the location in the error message. 4. Other `git` command errors (command not found, etc.), a more verbose error is given with the arguments that were passed to the git command. ## Future work 1. Add an offline mode 2. Add a lock file mechanism that resolves branch/tag git refs to commit hashes * closes https://github.com/anoma/juvix/issues/2083 --------- Co-authored-by: Jan Mas Rovira <janmasrovira@gmail.com>
2023-09-01 14:37:06 +03:00
# Smoke tests make git commits
- name: Setup git
shell: bash
run: |
git config --global user.email "tara-juvix@heliax.dev"
git config --global user.name "Tara"
git config --global init.defaultBranch main
- name: Smoke testing (macOS)
id: smoke-macos
if: ${{ success() }}
run: |
cd main
make CC=$CC LIBTOOL=$LIBTOOL smoke