This PR modifies the CI manifest to cache the `cairo-vm-cli` executable
so we don't have to rebuild it on each run.
The ref of the cairo-vm repo is pinned to
`42e04161de82d7e5381258def4b65087c8944660` which is currently the HEAD
ref of the repository. This can be changed in the CI manifest by editing
the `CAIRO_VM_VERSION` variable. Doing so will trigger a rebuild of
`cairo-vm-cli`.
* Closes#2563
Checklist
------------
- [x] Serialization of the Haskell CASM representation to the JSON
format accepted by the Cairo VM.
- [x] Add the `cairo` target to the `compile` commands.
- [x] Output via the Cairo `output` builtin.
- [x] Relativize jumps. Cairo VM doesn't actually support absolute
jumps.
- [x] Test the translation from CASM to Cairo by running the output in
the Cairo VM
- [x] Add Cairo VM to the CI
The alpine abstract packages clang/llvm now meet minimum version
requirements in the Docker container we're using for the linux static
build.
The llvm package must now be installed separately to get the `llvm-ar`
tool.
This PR updates the stackage LTS resolver to `nightly-2024-02-06` which
uses GHC 9.8.1
## Upgrade notes
You will need to update your HLS to
[2.6.0.0](https://github.com/haskell/haskell-language-server/releases/tag/2.6.0.0),
this release contains support for GHC 9.8.1
## Fixes
### `haskeline` / `repline`
We have removed the custom haskeline / repline forks used in the build.
This is because we had trouble overriding haskeline as it is bundled
with GHC and the stackage resolver uses this bundled version. We were
using a custom fork of haskeline to implement
[mapInputT_](15c0685c91/app/Commands/Repl.hs (L409))
in the Juvix REPL, required to implement error handling. This requires
private API from the Haskeline library.
Instead of using a custom fork we use TemplateHaskell to obtain access
to the private API we need. See
[DarkArts.hs](15c0685c91/src/Juvix/Prelude/DarkArts.hs)
and
[HaskelineJB.hs](15c0685c91/app/HaskelineJH.hs).
To obtain access to the private API, we adapted a method from [a Tweag
blogpost](https://www.tweag.io/blog/2021-01-07-haskell-dark-arts-part-i/)
and [repo](https://github.com/tweag/th-jailbreak) - updating it for GHC
9.8.1.
### `aeson-better-errors`
The `aeson-better-errors` library has not been updated to work with
`mtl-2.3.0` so it cannot work with the new stackage resolver. We are
using a [fork](https://github.com/Vekhir/aeson-better-errors.git) which
has been updated.
We should consider replacing this library in future, see
https://github.com/anoma/juvix/issues/2621
### `path`
The `path` library now includes API `splitDrive` and `dropDrive` so we
can remove our versions of those functions from the prelude.
### `with-utf8`
We no longer need to depend on `with-utf8`. We were using this package
for UTF-8 versions of `readFile` and `writeFile` APIs. These APIs are
now available in the `text` package.
### Compiler warnings
GHC 9.8.1 introduces several new compiler warnings.
* We have suppressed `missing-role-annotations` and
`missing-poly-kind-signatures`
* We added our own versions of `head` and `tail` to work around the new
`partial-tx` warning introduced for those functions in `Data.List`.
* We fixed up the code to avoid the
[term-variable-capture](https://ghc.gitlab.haskell.org/ghc/doc/users_guide/using-warnings.html#ghc-flag--Wterm-variable-capture)
warning.
Adds annotations to cells to indicate that it is a call to the stdlib
and might be evaluated faster in the Haskell evaluator.
The syntax for stdlib calls is as follows:
```
[stdlib@add args@<args-term> <left-term> <right-term>]
```
where `add` is the name of the function being called, `<args-term>` is a
nockma term that points to the position of the arguments, and
`<left-term>` and `<right-term>` are the actual components of the cell.
Adds a new version of the lock file that stores the hash (sha256 digest)
of the package file (Package.juvix, juvix.yaml) it was generated from as
a field:
```
# This file was autogenerated by Juvix version 0.5.4.
# Do not edit this file manually.
version: 2
checksum: d05940a4d3dc0e15451d02e1294819c875ba486ee54e26865ba8d190ac7c27c3
dependencies:
- git:
name: stdlib
ref: f68b0614ad695eaa13ead42f3466e0a78219f826
url: https://github.com/anoma/juvix-stdlib.git
dependencies: []
```
The lock file is regenerated when the hash of the package file doesn't
match the value of the `checksum` field, i.e when the user updates the
package file.
Existing lock files are automatically migrated to version 2.
* Closes https://github.com/anoma/juvix/issues/2464
We use the options in `package.yaml`.
```
- -fhide-source-paths
- -fwrite-ide-info -hiedir=.hie
```
If a previously available .hie directory is missing then GHC will
rebuild the whole project with reason: `[HIE file is missing]`. So we
need to cache it to take advantage of incremental builds.
This PR adds external git dependency support to the Juvix package
format.
## New dependency Git item
You can now add a `git` block to the dependencies list:
```yaml
name: HelloWorld
main: HelloWorld.juvix
dependencies:
- .juvix-build/stdlib
- git:
url: https://my.git.repo
name: myGitRepo
ref: main
version: 0.1.0
```
Git block required fields:
* `url`: The URL of the git repository
* `ref`: The git reference that should be checked out
* `name`: The name for the dependency. This is used to name the
directory of the clone, it is required. Perhaps we could come up with a
way to automatically name the clone directory. Current ideas are to
somehow encode the URL / ref combination or use a UUID. However there's
some value in having the clone directory named in a friendly way.
NB:
* The values of the `name` fields must be unique among the git blocks in
the dependencies list.
## Behaviour
When dependencies for a package are registered, at the beginning of the
compiler pipeline, all remote dependencies are processed:
1. If it doesn't already exist, the remote dependency is cloned to
`.juvix-build/deps/$name`
2. `git fetch` is run in the clone
3. `git checkout` at the specified `ref` is run in the clone
The clone is then processed by the PathResolver in the same way as path
dependencies.
NB:
* Remote dependencies of transitive dependencies are also processed.
* The `git fetch` step is required for the case where the remote is
updated. In this case we want the user to be able to update the `ref`
field.
## Errors
1. Missing fields in the Git dependency block are YAML parse errors
2. Duplicate `name` values in the dependencies list is an error thrown
when the package file is processed
3. The `ref` doesn't exist in the clone or the clone directory is
otherwise corrupt. An error with a suggestion to `juvix clean` is given.
The package file path is used as the location in the error message.
4. Other `git` command errors (command not found, etc.), a more verbose
error is given with the arguments that were passed to the git command.
## Future work
1. Add an offline mode
2. Add a lock file mechanism that resolves branch/tag git refs to commit
hashes
* closes https://github.com/anoma/juvix/issues/2083
---------
Co-authored-by: Jan Mas Rovira <janmasrovira@gmail.com>
Stack LTS 21.6 uses GHC 9.4.5, binaries for HLS are available via ghcup.
Changes required:
1. Fix warnings about type level `:` and `[]` used without backticks.
2. Fix warnings about deprecation of builtin `~` - replaced with `import
Data.Type.Equality ( type (~) )` in the Prelude
3. SemVer is no longer a monoid
4. `path-io` now contains the `AnyPath` instances we were defining
(thanks to Jan) so they can be removed.
5. Added `aeson-better-errors-0.9.1.1` as an extra-dep. The reason it is
not part of the resolver is only because it has a strict bound on base
which is not compatible with ghc 9.4.5. To work around this I've set:
```
allow-newer: true
allow-newer-deps:
- aeson-better-errors
```
which relaxed the upper constraint bounds for `aeson-better-errors`
only. When the base constraints have been updated we can remove this
workaround.
6. Use stack2cabal to generate the cabal.project file and to freeze
dependency versions.
https://www.stackage.org/lts-21.6/cabal.config now contains the
constraint `haskeline installed`, which means that the version of
haskeline that is globally installed with GHC 9.4.5 will be used, see:
* https://github.com/commercialhaskell/stackage/issues/7002
GHC 9.4.5 comes with haskeline 0.8.2 preinstalled but our configuration
contains the source-repository-package for haskeline 0.8.2.1 (required
because we're using a fork) so if you try to run` cabal build` you get a
conflict.
Constraints from cabal imports cannot yet be overridden so it's not
possible to get rid of this conflict using the import method. So we need
to use stack2cabal with an explicit freeze file instead.
7. Remove `runTempFilePure` as this is unused and depends on
`Polysemy.Fresh` in `polysemy-zoo` which is not available in the
resolver. It turns out that it's not possible to use the `Fresh` effect
in a pure context anyway, so it was not possible to use
`runTempFilePure` for its original purpose.
8. We now use https://github.com/benz0li/ghc-musl as the base container
for static linux builds, this means we don't need to maintain our own
Docker container for this purpose.
9. The PR for the nightly builds is ready
https://github.com/anoma/juvix-nightly-builds/pull/2, it should be
merged as soon as this PR is merged.
Thanks to @benz0li for maintaining https://github.com/benz0li/ghc-musl
and (along with @TravisCardwell) for help with building the static
binary.
* Closes https://github.com/anoma/juvix/issues/2166
This PR replaces fetching a precompiled binary of smoke with a
build/cache for macOS smoke tests on CI.
smoke dynamically links to icu4c, so a cached binary of smoke will break
when brew bumps the icu4c version. In this PR we use the icu4c version
in the cache key of the smoke build to avoid this issue.
NB: The smoke build cannot be done as a separate job because the smoke
binary must be built using exactly the same version of the macos-12
runner image as the smoke testing step to make sure that the icu4c
versions match.
Motivation for doing this is this failure:
https://github.com/anoma/juvix/actions/runs/5325094406/jobs/9645334642
which uses this release of the runner image
https://github.com/actions/runner-images/releases/tag/macOS-12%2F20230618.1
which contains the updated brew version of icu4c.
20230618.1 is currently a prerelease, but will start to run on more jobs
shortly.
```
2023-06-20T17:10:13.2222310Z Copied executables to /Users/runner/.local/bin:
2023-06-20T17:10:13.2223440Z - juvix
2023-06-20T17:10:13.5312790Z dyld[90256]: Library not loaded: '/usr/local/opt/icu4c/lib/libicuuc.72.dylib'
2023-06-20T17:10:13.5331930Z Referenced from: '/Users/runner/hostedtoolcache/jonaprieto/smoke/latest/darwin-x64/smoke'
2023-06-20T17:10:13.5333610Z Reason: tried: '/usr/local/opt/icu4c/lib/libicuuc.72.dylib' (no such file), '/usr/local/lib/libicuuc.72.dylib' (no such file), '/usr/lib/libicuuc.72.dylib' (no such file), '/usr/local/Cellar/icu4c/73.2/lib/libicuuc.72.dylib' (no such file), '/usr/local/lib/libicuuc.72.dylib' (no such file), '/usr/lib/libicuuc.72.dylib' (no such file)
2023-06-20T17:10:13.5334690Z make[1]: *** [smoke-only] Abort trap: 6
2023-06-20T17:10:13.5335310Z make: *** [smoke] Error 2
2023-06-20T17:10:13.5363170Z ##[error]Process completed with exit code 2.
```
This PR installs the `gnu-sed` for the macOS build.
It can be called with the name `sed`, no `gsed`.
NB: The environmental variables set in one step are only reflected in
the subsequent steps.
This PR:
- Makes `vamp-ir` available in the CI (pre-release 0.1.2)
- [Use a setup-wasmer action to install
`wasmer`](https://github.com/marketplace/actions/setup-wasmer)
- Fixes cache option value for `jaxxstorm/action-install-gh-release`'s
usages
Adds support for:
- #2103
Related:
- https://github.com/anoma/vamp-ir/issues/90
---------
Co-authored-by: Jan Mas Rovira <janmasrovira@gmail.com>
This PR resolves a few bugs in the Makefile targets for formatting and
type checking Juvix files, which were preventing the capture of type
checking errors for our examples and bad formatting for all the Juvix
files in the repository. With this PR, our code should now be clean, and
we can expect every file to be properly formatted and type checked.
Changes made:
- [x] Updated `make format-juvix-files`
- [x] Updated `make check-format-juvix-files`
- [x] Formatted all Juvix files
- [x] Comment a fragment in `examples/milestone/Bank/Bank.juvix`
In the future, we will drastically simplify the Makefile once we improve
the `format` and the `type check` command for example posted here:
- #2066
- #2087
Related:
- #2063
- #2040 (due to some typechecking errors we're not capturing before)
- #2105
- https://github.com/anoma/juvix/issues/2059
Currently we typecheck and check formatting of juvix examples. However
the make target redundantly traverses the juvix files within the
.juvix-build directory contained in each project (which has been created
during previous tests).
This commit cleans the .juvix-build directories before performing each
of these checks.
This PR upgrades our Haskell configurations to compile with version
9.2.7. The checklist below can serve as a guide for similar future
updates:
- [x] Update Stack resolver in `stack.yaml`
- [x] Modify `tested-with` section in `package.yaml`
- [x] Build and push the new compiler docker image, see instructions
here
[docker/README.md](https://github.com/anoma/juvix/blob/main/docker/README.md):
`ghcr.io/paulcadman/ghc-alpine:9.2.7 container`.
- [x] Update Linux Github Action workflow in
`.github/workflows/linux-static-binary.yaml` and adjust
`docker/Dockerfile-ghc-alpine-9.2.7`
- [x] Revise GHC/Stack/Cabal versions in `.devcontainer/Dockerfile`
- [x] Refresh Cabal configuration in `cabal-project`
This pr fixes a bug where the repl would crash if it had the implicit
stdlib dependency and the .juvix-build/stdlib directory did not yet
exist. This bug was not exposed in the smoke tests because the
.juvix-build was never cleared.
---------
Co-authored-by: Paul Cadman <git@paulcadman.dev>
This PR fixes the broken links that refer to Juvix Examples in the
documentation. Ideally, this wouldn't occur since we utilize a link
checker, but this tool only works well for relative links, which was not
the case for the links of the examples. Additionally, I slightly
modified the CI workflow by generating the HTML for the examples first,
followed by the entire book.
- Closes #2001
- Check the output using this: https://jonaprieto.github.io/juvix/
There's an error with this step on linux only, but we cannot see it
because of the `-s` flag on the Makefile call.
This commit removes the `-s` flag so we can diagnose the problem, but
also temporarily ignores the error to avoid blocking other PRs.
NB: This step passes on macOS.
This PR adds `juvix format` that can be used to format either a single
Juvix file or all files in a Juvix project.
## Usage
```
$ juvix format --help
Usage: juvix format JUVIX_FILE_OR_PROJECT [--check] [--in-place]
Format a Juvix file or Juvix project
When the command is run with an unformatted file it prints the reformatted source to standard output.
When the command is run with a project directory it prints a list of unformatted files in the project.
Available options:
JUVIX_FILE_OR_PROJECT Path to a .juvix file or to a directory containing a
Juvix project.
--check Do not print reformatted sources or unformatted file
paths to standard output.
--in-place Do not print reformatted sources to standard output.
Overwrite the target's contents with the formatted
version if the formatted version differs from the
original content.
-h,--help Show this help text
```
## Location of main implementation
The implementation is split into two components:
* The src API: `format` and `formatProject`
73952ba15c/src/Juvix/Formatter.hs
* The CLI interface:
73952ba15c/app/Commands/Format.hs
## in-place uses polysemy Resource effect
The `--in-place` option makes a backup of the target file and restores
it if there's an error during processing to avoid data loss. The
implementation of this uses the polysemy [Resource
effect](https://hackage.haskell.org/package/polysemy-1.9.0.0/docs/Polysemy-Resource.html).
The recommended way to interpret the resource effect is to use
`resourceToIOFinal` which makes it necessary to change the effects
interpretation in main to use `Final IO`:
73952ba15c/app/Main.hs (L15)
## Format input is `FilePath`
The format options uses `FilePath` instead of `AppFile f` for the input
file/directory used by other commands. This is because we cannot
determine if the input string is a file or directory in the CLI parser
(we require IO). I discussed some ideas with @janmasrovira on how to
improve this in a way that would also solve other issues with CLI input
file/parsing but I want to defer this to a separate PR as this one is
already quite large.
One consequence of Format using `FilePath` as the input option is that
the code that changes the working directory to the root of the project
containing the CLI input file is changed to work with `FilePath`:
f715ef6a53/app/TopCommand/Options.hs (L33)
## New dependencies
This PR adds new dependencies on `temporary` and `polysemy-zoo`.
`temporary` is used for `emptySystemTempFile` in the implementation of
the TempFile interpreter for IO:
73952ba15c/src/Juvix/Data/Effect/Files/IO.hs (L49)
`polysemy-zoo` is used for the `Fresh` effect and `absorbMonadThrow` in
the implementation of the pure TempFile interpreter:
73952ba15c/src/Juvix/Data/Effect/Files/Pure.hs (L91)
NB: The pure TempFile interpreter is not used, but it seemed a good idea
to include it while it's fresh in my mind.
* Closes https://github.com/anoma/juvix/issues/1777
---------
Co-authored-by: Jonathan Cubides <jonathan.cubides@uib.no>
This PR changes two things:
- Deploy the documentation website using the (official) deploy action:
- https://github.com/actions/deploy-pages
- Check the documentation generation for all the PRs.
- PR on top of:
- #1905
- #1908
After merging this PR, we could delete the github-pages branch.
This pull request introduces a new workflow that automatically deletes
caches created for closed pull requests. This is necessary because these
caches can quickly accumulate and take up valuable server space, which
is especially important now that we have many pull requests being
created regularly.
This PR goes after:
- #1797
Included in this PR:
- Add Clang formatting (v.15) as part of the pre-commits.
- Add new pre-commits:
- forbid binary files
- check toml files for mdbook
- detect-private-key
- format-juvix-examples: check all the Juvix programs in the `examples`
folder type-check for local runs.
- Remove .pre-commit-hooks and add ormolu for local pre-commit runs
instead.
In this PR, I have updated the README file to reflect the new goals of
the project and highlight related products to Juvix. The ORG files have
been replaced with Markdown for better readability and maintainability.
Additionally, I have added a couple of files to fine-tune the mdbook
settings. These changes, I believe, will make it easier for users to
understand and contribute to the project.🤞
- Closes#1878
- New pre-commit hook to format md, yaml, js, CSS files.
To check the website generation, I have deployed the result here:
Work in progress.
- https://jonaprieto.github.io/juvix
- https://github.com/jonaprieto/juvix
---------
Co-authored-by: Paul Cadman <pcadman@gmail.com>
Co-authored-by: Christopher Goes <cwgoes@pluranimity.org>
Co-authored-by: Jan Mas Rovira <janmasrovira@gmail.com>
* Fixes https://github.com/anoma/juvix/issues/1811
This PR updates:
* The CI workflows to use GHC 9.2.6
* The stack resolver to LTS-20.12
* The cabal.project to point to stackage LTS-20.12
* The linux static build CI to use alpine GHC 9.2.6
NB: You may need to install GHC 9.2.6 and run `cabal update` before
trying the build with `cabal`.
This PR addresses a caching issue in our CI by streamlining each
operating system's build and test processes, reducing CI time. 🤞 Also,
our caching strategy has been updated with the new restore/save actions.
For example, we aim to cache the .stack folder, and if the stack build
is successful, the .stack-build. The building documentation job
continues depending on the Linux build. Upon merging this PR, we get
back to the point where the CI maintain a cache for each OS to be shared
among all PRs, significantly reducing CI testing time. The expected
scenario is as follows. The CI can take, on average, 35' in Linux to
build and test everything. Using caching, that time is reduced to less
than 10'. macOS is a different story. It can easily take one hour, and
even more, the first time to build and test the project. After that, it
might take an average of 20'.
- Caching strategies
[descriptions](https://github.com/actions/cache/blob/main/caching-strategies.md#saving-cache-even-if-the-build-fails)
- Closes#1776
The current Github action responsible for installing Wasmer fails from
time to time, and it also is outdated, not following the new NodeJS 16
requirement by Github.
We could use https://github.com/jaxxstorm/action-install-gh-release
instead, but unfortunately, it does not have the proper support to
expose the binaries contained in an inner folder, as in the case with
the Wasmer release. In the meantime, let's use my
[fork](https://github.com/jonaprieto/action-install-gh-release) while I
open PR to the main repository.
This PR adds some maintenance at different levels to the CI config, the
Make file, and formatting.
- Most of the actions used by the CI related to haskell, ormolu, hlint
and pre-commit have been updated because Github requires NodeJS 16. This
change removes all the old warnings related to nodeJs.
In the case of ormolu, the new version makes us format some files that
were not formatted before, similarly with hlint.
- The CI has been updated to use the latest version of the Smoke testing
framework, which introduced installation of the dependencies for Linux
(libicu66) and macOS (icu4c) in the CI. In the case of macOS, the CI
uses a binary for smoke. For Linux, we use stack to build smoke from the
source. The source here is in a fork of [the official Smoke
repo](https://github.com/SamirTalwar/smoke). Such includes some
features/changes that are not yet in the official repo.
- The Makefile runs the ormolu and hlint targets using as a path for the
binaries the environment variables ORMOLU and HLINT. Thus, export those
variables in your environment before running `make check,` `make format`
or `make hlint`. Otherwise, the Makefile will use the binaries provided
by `stack`.
Co-authored-by: Paul Cadman <git@paulcadman.dev>
We need to use the homebrew Clang/LLVM installation to build the Juvix
runtime because macOS ships with a version of Clang/LLVM that does not
support the wasi target.
The GitHub action runner agent has the homebrew packaged Clang/LLVM
installed but it is not on the shell PATH.
We were using `brew --prefix llvm@14` to point to the homebrew
Clang/LLVM installation. However this breaks when the runner image is
updated to a new version of llvm.
So we switch to using `brew --prefix llvm` which will resolve to the
latest (and in this case only) version of homebrew Clang/LLVM. This will
be stable when Clang/LLVM is updated to a new version.
This PR adds smoke tests using [Smoke
tool](https://github.com/SamirTalwar/smoke) for all the shell tests we
have. One reason for adopting Smoke instead of the previous tool,
`shelltestrunner`, is that tests are declared cleanly and simply using
Smoke Yaml syntax compared to shelltestrunner's syntax.
To add a new smoke test, create a file with the suffix ".smoke.yaml" in
the `tests/smoke` folder. In such a folder, you can also find examples
of how to test the CLI.
* Pin mdbook to 0.4.22
mdbook homebrew version (used by the github action to install mdbook)
was bumped to 0.4.23
eb69fcddba
which seems to be incompatible with Ubuntu 20.04 glibc
* Temporarily enable docs build on PRs
* Revert "Temporarily enable docs build on PRs"
This reverts commit e2cf41016c.
