Adds annotations to cells to indicate that it is a call to the stdlib
and might be evaluated faster in the Haskell evaluator.
The syntax for stdlib calls is as follows:
```
[stdlib@add args@<args-term> <left-term> <right-term>]
```
where `add` is the name of the function being called, `<args-term>` is a
nockma term that points to the position of the arguments, and
`<left-term>` and `<right-term>` are the actual components of the cell.
This PR is a snapshot of the current work on the JuvixAsm -> Nockma
translation. The compilation of Juvix programs to Nockma now works so we
decided to raise this PR now to avoid it getting too large.
## Juvix -> Nockma compilation
You can compile a frontend Juvix file to Nockma as follows:
example.juvix
```
module example;
import Stdlib.Prelude open;
fib : Nat → Nat → Nat → Nat
| zero x1 _ := x1
| (suc n) x1 x2 := fib n x2 (x1 + x2);
fibonacci (n : Nat) : Nat := fib n 0 1;
sumList (xs : List Nat) : Nat :=
for (acc := 0) (x in xs)
acc + x;
main : Nat := fibonacci 9 + sumList [1; 2; 3; 4];
```
```
$ juvix compile -t nockma example.juvix
```
This will generate a file `example.nockma` which can be run using the
nockma evaluator:
```
$ juvix dev nockma eval example.nockma
```
Alternatively you can compile JuvixAsm to Nockma:
```
$ juvix dev asm compile -t nockma example.jva
```
## Tests
We compile an evaluate the JuvixAsm tests in
cb3659e08e/test/Nockma/Compile/Asm/Positive.hs
We currently skip some because either:
1. They are too slow to run in the current evaluator (due to arithmetic
operations using the unjetted nock code from the anoma nock stdlib).
2. They trace data types like lists and booleans which are represented
differently by the asm interpreter and the nock interpreter
3. They operate on raw negative numbers, nock only supports raw natural
numbers
## Next steps
On top of this PR we will work on improving the evaluator so that we can
enable the slow compilation tests.
---------
Co-authored-by: Paul Cadman <git@paulcadman.dev>
Co-authored-by: Lukasz Czajka <lukasz@heliax.dev>
* Closes#2561
* Defines an extended subset of Cairo Assembly, following Section 5 of
[1].
* Adds the commands `juvix dev casm read file.casm` and `juvix dev casm
run file.casm` to print and run `*.casm` files.
* The tests cover CASM semantics. Some are "manual translations" of
corresponding JuvixAsm tests according to the JuvixAsm -> CASM
compilation concept.
* Closes#2392
Changes checklist
-----------------
* [X] Abstract out data types for stored module representation
(`ModuleInfo` in `Juvix.Compiler.Store.Language`)
* [X] Adapt the parser to operate per-module
* [X] Adapt the scoper to operate per-module
* [X] Adapt the arity checker to operate per-module
* [X] Adapt the type checker to operate per-module
* [x] Adapt Core transformations to operate per-module
* [X] Adapt the pipeline functions in `Juvix.Compiler.Pipeline`
* [X] Add `Juvix.Compiler.Pipeline.Driver` which drives the per-module
compilation process
* [x] Implement module saving / loading in `Pipeline.Driver`
* [x] Detect cyclic module dependencies in `Pipeline.Driver`
* [x] Cache visited modules in memory in `Pipeline.Driver` to avoid
excessive disk operations and repeated hash re-computations
* [x] Recompile a module if one of its dependencies needs recompilation
and contains functions that are always inlined.
* [x] Fix identifier dependencies for mutual block creation in
`Internal.fromConcrete`
- Fixed by making textually later definitions depend on earlier ones.
- Now instances are used for resolution only after the textual point of
their definition.
- Similarly, type synonyms will be unfolded only after the textual point
of their definition.
* [x] Fix CLI
* [x] Fix REPL
* [x] Fix highlighting
* [x] Fix HTML generation
* [x] Adapt test suite
This patch dramatically increases the efficiency of `juvix dev root`,
which was unnecessarily parsing all dependencies included in the
`Package.juvix` file. Other commands that do not require the `Package`
will also be faster.
It also refactors some functions so that the `TaggedLock` effect is run
globally.
I've added `singletons-base` as a dependency so we can have `++` on the
type level. We've tried to define a type family ourselves but inference
was not working properly.
This PR updates the juvix-stdlib submodule ref to the current
juvix-stdlib/main.
NB: The Markdown test is not stable after changes to the stdlib - the
ids (deriving from internal name ids?) will change and so the expected
file must be regenerated.
Adds a new version of the lock file that stores the hash (sha256 digest)
of the package file (Package.juvix, juvix.yaml) it was generated from as
a field:
```
# This file was autogenerated by Juvix version 0.5.4.
# Do not edit this file manually.
version: 2
checksum: d05940a4d3dc0e15451d02e1294819c875ba486ee54e26865ba8d190ac7c27c3
dependencies:
- git:
name: stdlib
ref: f68b0614ad695eaa13ead42f3466e0a78219f826
url: https://github.com/anoma/juvix-stdlib.git
dependencies: []
```
The lock file is regenerated when the hash of the package file doesn't
match the value of the `checksum` field, i.e when the user updates the
package file.
Existing lock files are automatically migrated to version 2.
* Closes https://github.com/anoma/juvix/issues/2464
## Overview
This PR makes the compiler pipeline thread-safe so that the test suite
can be run in parallel.
This is achieved by:
* Removing use of `{get, set, with}CurrentDir` functions.
* Adding locking around shared file resources like the the
global-project and internal build directory.
NB: **Locking is disabled for the main compiler target**, as it is
single threaded they are not required.
## Run test suite in parallel
To run the test suite in parallel you must add `--ta '+RTS -N -RTS'` to
your stack test arguments. For example:
```
stack test --fast --ta '+RTS -N -RTS'
```
The `-N` instructs the Haskell runtime to choose the number of threads
to use based on how many processors there are on your machine. You can
use `-Nn` to see the number of threads to `n`.
These flags are already [set in the
Makefile](e6dca22cfd/Makefile (L26))
when you or CI uses `stack test`.
## Locking
The Haskell package
[filelock](https://hackage.haskell.org/package/filelock) is used for
locking. File locks are used instead of MVars because Juvix code does
not control when new threads are created, they are created by the test
suite. This means that MVars created by Juvix code will have no effect,
because they are created independently on each test-suite thread.
Additionally the resources we're locking live on the filesystem and so
can be conveniently tagged by path.
### FileLock
The filelock library is wrapped in a FileLock effect:
e6dca22cfd/src/Juvix/Data/Effect/FileLock/Base.hs (L6-L8)
There is an [IO
interpreter](e6dca22cfd/src/Juvix/Data/Effect/FileLock/IO.hs (L8))
that uses filelock and an [no-op
interpreter](e6dca22cfd/src/Juvix/Data/Effect/FileLock/Permissive.hs (L7))
that just runs actions unconditionally.
### TaggedLock
To make the file locks simpler to use a TaggedLock effect is introduced:
e6dca22cfd/src/Juvix/Data/Effect/TaggedLock/Base.hs (L5-L11)
And convenience function:
e6dca22cfd/src/Juvix/Data/Effect/TaggedLock.hs (L28)
This allows an action to be locked, tagged by a directory that may or
may not exist. For example in the following code, an action is performed
on a directory `root` that may delete the directory before repopulating
the files. So the lockfile cannot be stored in the `root` itself.
e6dca22cfd/src/Juvix/Extra/Files.hs (L55-L60)
## Pipeline
As noted above, we only use locking in the test suite. The main app
target pipeline is single threaded and so locking is unnecessary. So the
interpretation of locks is parameterised so that locking can be disabled
e6dca22cfd/src/Juvix/Compiler/Pipeline/Run.hs (L64)
This PR adds an initial support for Literate Juvix Markdown files, files
with the extension `.juvix.md`.
Here is a small example of such a file: `Test.juvix.md`.
<pre>
# This is a heading
Lorem ...
```juvix
module Test;
type A := a;
fun : A -> A
| _ := a;
```
Other text
</pre>
This initial support enables users to execute common commands such as
typechecking, compilation, and HTML generation. Additionally, a new
command called `markdown` has been introduced. This command replaces
code blocks marked with the juvix attribute with their respective HTML
output, much like the output we obtain when running `juvix html`. In
this version, comments are ignored in the output, including judoc
blocks.
- We intend to use this new feature in combination with this Python
plugin (https://github.com/anoma/juvix-mkdocs) to enhance our
documentation site.
https://github.com/anoma/juvix/assets/1428088/a0c17f36-3d76-42cc-a571-91f885866874
## Future work
Open as issues once this PR is merged, we can work on the following:
- Support imports of Juvix Markdown modules (update the path resolver to
support imports of Literate Markdown files)
- Support (Judoc) comments in md Juvix blocks
- Support Markdown in Judoc blocks
- Update Text editor support, vscode extension and emacs mode (the
highlighting info is a few characters off in the current state)
- Closes#1839
- Closes#1719
The problem with readFile and writeFile from text
[Data.Text.IO](https://hackage.haskell.org/package/text-2.0.2/docs/Data-Text-IO.html)
is that they use the system locale to determine the text encoding
format.
Our assumption is that all Juvix source files are UTF-8 encoded.
I cannot reproduce the issue with using the old APIs on my machine, it
can be reproduced on Arch linux. I'm not sure how to write a specific
test for this.
* Closes https://github.com/anoma/juvix/issues/2472
The special PathResolver puts files from the global package stdlib and
files from the global package description files in scope of the
$root/Package.juvix module.
Currently this means that PackageDescription module is in scope for the
module so that the user can write:
```
module Package;
import Stdlib.Prelude open;
import PackageDescription open;
package : Package :=
mkPackageDefault
(name := "foo")
{ version := mkVersion 0 1 0
; dependencies :=
[ github "anoma" "juvix-stdlib" "adf58a7180b361a022fb53c22ad9e5274ebf6f66"
; github "anoma" "juvix-containers" "v0.7.1"]};
```
This PR adds lock file support to the compiler pipeline. The lock file
is generated whenever a compiler pipeline command (`juvix {compile,
typecheck, repl}`) is run.
The lock file contains all the information necessary to reproduce the
whole dependency source tree. In particular for git dependencies,
branch/tag references are resolved to git hash references.
## Lock file format
The lock file is a YAML `juvix.lock.yaml` file written by the compiler
alongside the package's `juvix.yaml` file.
```
LOCKFILE_SPEC: { dependencies: { DEPENDENCY_SPEC, dependencies: LOCKFILE_SPEC }
DEPENDENCY_SPEC: PATH_SPEC | GIT_SPEC
PATH_SPEC: { path: String }
GIT_SPEC: { git: {url: String, ref: String, name: String } }
```
## Example
Consider a project containing the following `juvix.yaml`:
```yaml
dependencies:
- .juvix-build/stdlib/
- git:
url: https://github.com/anoma/juvix-containers
ref: v0.7.1
name: containers
name: example
version: 1.0.0
```
After running `juvix compile` the following lockfile `juvix.lock.yaml`
is generated.
```yaml
# This file was autogenerated by Juvix version 0.5.1.
# Do not edit this file manually.
dependencies:
- path: .juvix-build/stdlib/
dependencies: []
- git:
name: containers
ref: 3debbc7f5776924eb9652731b3c1982a2ee0ff24
url: https://github.com/anoma/juvix-containers
dependencies:
- git:
name: stdlib
ref: 4facf14d9b2d06b81ce1be1882aa9050f768cb45
url: https://github.com/anoma/juvix-stdlib
dependencies: []
- git:
name: test
ref: a7ac74cac0db92e0b5e349f279d797c3788cdfdd
url: https://github.com/anoma/juvix-test
dependencies:
- git:
name: stdlib
ref: 4facf14d9b2d06b81ce1be1882aa9050f768cb45
url: https://github.com/anoma/juvix-stdlib
dependencies: []
```
For subsequent runs of the juvix compile pipeline, the lock file
dependency information is used.
## Behaviour when package file and lock file are out of sync
If a dependency is specified in `juvix.yaml` that is not present in the
lock file, an error is raised.
Continuing the example above, say we add an additional dependency:
```
dependencies:
- .juvix-build/stdlib/
- git:
url: https://github.com/anoma/juvix-containers
ref: v0.7.1
name: containers
- git:
url: https://github.com/anoma/juvix-test
ref: v0.6.1
name: test
name: example
version: 1.0.0
```
`juvix compile` will throw an error:
```
/Users/paul/tmp/lockfile/dep/juvix.yaml:1:1: error:
The dependency test is declared in the package's juvix.yaml but is not declared in the lockfile: /Users/paul/tmp/lockfile/dep/juvix.lock.json
Try removing /Users/paul/tmp/lockfile/dep/juvix.lock.yaml and then run Juvix again.
```
Closes:
* https://github.com/anoma/juvix/issues/2334
This PR adds external git dependency support to the Juvix package
format.
## New dependency Git item
You can now add a `git` block to the dependencies list:
```yaml
name: HelloWorld
main: HelloWorld.juvix
dependencies:
- .juvix-build/stdlib
- git:
url: https://my.git.repo
name: myGitRepo
ref: main
version: 0.1.0
```
Git block required fields:
* `url`: The URL of the git repository
* `ref`: The git reference that should be checked out
* `name`: The name for the dependency. This is used to name the
directory of the clone, it is required. Perhaps we could come up with a
way to automatically name the clone directory. Current ideas are to
somehow encode the URL / ref combination or use a UUID. However there's
some value in having the clone directory named in a friendly way.
NB:
* The values of the `name` fields must be unique among the git blocks in
the dependencies list.
## Behaviour
When dependencies for a package are registered, at the beginning of the
compiler pipeline, all remote dependencies are processed:
1. If it doesn't already exist, the remote dependency is cloned to
`.juvix-build/deps/$name`
2. `git fetch` is run in the clone
3. `git checkout` at the specified `ref` is run in the clone
The clone is then processed by the PathResolver in the same way as path
dependencies.
NB:
* Remote dependencies of transitive dependencies are also processed.
* The `git fetch` step is required for the case where the remote is
updated. In this case we want the user to be able to update the `ref`
field.
## Errors
1. Missing fields in the Git dependency block are YAML parse errors
2. Duplicate `name` values in the dependencies list is an error thrown
when the package file is processed
3. The `ref` doesn't exist in the clone or the clone directory is
otherwise corrupt. An error with a suggestion to `juvix clean` is given.
The package file path is used as the location in the error message.
4. Other `git` command errors (command not found, etc.), a more verbose
error is given with the arguments that were passed to the git command.
## Future work
1. Add an offline mode
2. Add a lock file mechanism that resolves branch/tag git refs to commit
hashes
* closes https://github.com/anoma/juvix/issues/2083
---------
Co-authored-by: Jan Mas Rovira <janmasrovira@gmail.com>
Stack LTS 21.6 uses GHC 9.4.5, binaries for HLS are available via ghcup.
Changes required:
1. Fix warnings about type level `:` and `[]` used without backticks.
2. Fix warnings about deprecation of builtin `~` - replaced with `import
Data.Type.Equality ( type (~) )` in the Prelude
3. SemVer is no longer a monoid
4. `path-io` now contains the `AnyPath` instances we were defining
(thanks to Jan) so they can be removed.
5. Added `aeson-better-errors-0.9.1.1` as an extra-dep. The reason it is
not part of the resolver is only because it has a strict bound on base
which is not compatible with ghc 9.4.5. To work around this I've set:
```
allow-newer: true
allow-newer-deps:
- aeson-better-errors
```
which relaxed the upper constraint bounds for `aeson-better-errors`
only. When the base constraints have been updated we can remove this
workaround.
6. Use stack2cabal to generate the cabal.project file and to freeze
dependency versions.
https://www.stackage.org/lts-21.6/cabal.config now contains the
constraint `haskeline installed`, which means that the version of
haskeline that is globally installed with GHC 9.4.5 will be used, see:
* https://github.com/commercialhaskell/stackage/issues/7002
GHC 9.4.5 comes with haskeline 0.8.2 preinstalled but our configuration
contains the source-repository-package for haskeline 0.8.2.1 (required
because we're using a fork) so if you try to run` cabal build` you get a
conflict.
Constraints from cabal imports cannot yet be overridden so it's not
possible to get rid of this conflict using the import method. So we need
to use stack2cabal with an explicit freeze file instead.
7. Remove `runTempFilePure` as this is unused and depends on
`Polysemy.Fresh` in `polysemy-zoo` which is not available in the
resolver. It turns out that it's not possible to use the `Fresh` effect
in a pure context anyway, so it was not possible to use
`runTempFilePure` for its original purpose.
8. We now use https://github.com/benz0li/ghc-musl as the base container
for static linux builds, this means we don't need to maintain our own
Docker container for this purpose.
9. The PR for the nightly builds is ready
https://github.com/anoma/juvix-nightly-builds/pull/2, it should be
merged as soon as this PR is merged.
Thanks to @benz0li for maintaining https://github.com/benz0li/ghc-musl
and (along with @TravisCardwell) for help with building the static
binary.
* Closes https://github.com/anoma/juvix/issues/2166
This PR prepares the 0.4.1 release.
* Bump version in package.yaml
* Update version smoke test
* Updates CHANGELOG
NB: The links in the changelog will not work until we create the release
tag.
This PR upgrades our Haskell configurations to compile with version
9.2.7. The checklist below can serve as a guide for similar future
updates:
- [x] Update Stack resolver in `stack.yaml`
- [x] Modify `tested-with` section in `package.yaml`
- [x] Build and push the new compiler docker image, see instructions
here
[docker/README.md](https://github.com/anoma/juvix/blob/main/docker/README.md):
`ghcr.io/paulcadman/ghc-alpine:9.2.7 container`.
- [x] Update Linux Github Action workflow in
`.github/workflows/linux-static-binary.yaml` and adjust
`docker/Dockerfile-ghc-alpine-9.2.7`
- [x] Revise GHC/Stack/Cabal versions in `.devcontainer/Dockerfile`
- [x] Refresh Cabal configuration in `cabal-project`
This PR adds `juvix format` that can be used to format either a single
Juvix file or all files in a Juvix project.
## Usage
```
$ juvix format --help
Usage: juvix format JUVIX_FILE_OR_PROJECT [--check] [--in-place]
Format a Juvix file or Juvix project
When the command is run with an unformatted file it prints the reformatted source to standard output.
When the command is run with a project directory it prints a list of unformatted files in the project.
Available options:
JUVIX_FILE_OR_PROJECT Path to a .juvix file or to a directory containing a
Juvix project.
--check Do not print reformatted sources or unformatted file
paths to standard output.
--in-place Do not print reformatted sources to standard output.
Overwrite the target's contents with the formatted
version if the formatted version differs from the
original content.
-h,--help Show this help text
```
## Location of main implementation
The implementation is split into two components:
* The src API: `format` and `formatProject`
73952ba15c/src/Juvix/Formatter.hs
* The CLI interface:
73952ba15c/app/Commands/Format.hs
## in-place uses polysemy Resource effect
The `--in-place` option makes a backup of the target file and restores
it if there's an error during processing to avoid data loss. The
implementation of this uses the polysemy [Resource
effect](https://hackage.haskell.org/package/polysemy-1.9.0.0/docs/Polysemy-Resource.html).
The recommended way to interpret the resource effect is to use
`resourceToIOFinal` which makes it necessary to change the effects
interpretation in main to use `Final IO`:
73952ba15c/app/Main.hs (L15)
## Format input is `FilePath`
The format options uses `FilePath` instead of `AppFile f` for the input
file/directory used by other commands. This is because we cannot
determine if the input string is a file or directory in the CLI parser
(we require IO). I discussed some ideas with @janmasrovira on how to
improve this in a way that would also solve other issues with CLI input
file/parsing but I want to defer this to a separate PR as this one is
already quite large.
One consequence of Format using `FilePath` as the input option is that
the code that changes the working directory to the root of the project
containing the CLI input file is changed to work with `FilePath`:
f715ef6a53/app/TopCommand/Options.hs (L33)
## New dependencies
This PR adds new dependencies on `temporary` and `polysemy-zoo`.
`temporary` is used for `emptySystemTempFile` in the implementation of
the TempFile interpreter for IO:
73952ba15c/src/Juvix/Data/Effect/Files/IO.hs (L49)
`polysemy-zoo` is used for the `Fresh` effect and `absorbMonadThrow` in
the implementation of the pure TempFile interpreter:
73952ba15c/src/Juvix/Data/Effect/Files/Pure.hs (L91)
NB: The pure TempFile interpreter is not used, but it seemed a good idea
to include it while it's fresh in my mind.
* Closes https://github.com/anoma/juvix/issues/1777
---------
Co-authored-by: Jonathan Cubides <jonathan.cubides@uib.no>
In this PR, I have updated the README file to reflect the new goals of
the project and highlight related products to Juvix. The ORG files have
been replaced with Markdown for better readability and maintainability.
Additionally, I have added a couple of files to fine-tune the mdbook
settings. These changes, I believe, will make it easier for users to
understand and contribute to the project.🤞
- Closes#1878
- New pre-commit hook to format md, yaml, js, CSS files.
To check the website generation, I have deployed the result here:
Work in progress.
- https://jonaprieto.github.io/juvix
- https://github.com/jonaprieto/juvix
---------
Co-authored-by: Paul Cadman <pcadman@gmail.com>
Co-authored-by: Christopher Goes <cwgoes@pluranimity.org>
Co-authored-by: Jan Mas Rovira <janmasrovira@gmail.com>
* Bump version in package.yaml
* Update version smoke test
* Updates installing doc (though link will not work until we've produced
a linux binary after release)
* Updates changelog.org
* Depends on PR #1832
* Closes#1799
* Removes Backend.C.Translation.FromInternal
* Removes `foreign` and `compile` blocks
* Removes unused test files
* Removes the old C runtime
* Removes other dead code
* Depends on PR #1824
* Closes#1556
* Closes#1825
* Closes#1843
* Closes#1729
* Closes#1596
* Closes#1343
* Closes#1382
* Closes#1867
* Closes#1876
* Changes the `juvix compile` command to use the new pipeline.
* Removes the `juvix dev minic` command and the `BackendC` tests.
* Adds the `juvix eval` command.
* Fixes bugs in the Nat-to-integer conversion.
* Fixes bugs in the Internal-to-Core and Core-to-Core.Stripped
translations.
* Fixes bugs in the RemoveTypeArgs transformation.
* Fixes bugs in lambda-lifting (incorrect de Bruijn indices in the types
of added binders).
* Fixes several other bugs in the compilation pipeline.
* Adds a separate EtaExpandApps transformation to avoid quadratic
runtime in the Internal-to-Core translation due to repeated calls to
etaExpandApps.
* Changes Internal-to-Core to avoid generating matches on values which
don't have an inductive type.
---------
Co-authored-by: Paul Cadman <git@paulcadman.dev>
Co-authored-by: janmasrovira <janmasrovira@gmail.com>
This PR redefines the `html` command unifying our previous subcommands
for the HTML backend. You should use the command in the following way to
obtain the same results as before:
- `juvix html src.juvix` -> `juvix html src.juvix --only-source`
- `juvix dev doc src.juvix` -> `juvix html src.juvix`
- Other fixes here include the flag `--non-recursive`, which replaces
the previous behavior in that we now generate all the HTML recursively
by default.
- The flag `--no-print-metadata` is now called `--no-footer`
- Also, another change introduced by this PR is asset handling; for
example, with our canonical Juvix program,
the new output is organized as follows.
```
juvix html HelloWorld.juvix --only-source && tree html/
Copying assets files to test/html/assets
Writing HelloWorld.html
html/
├── assets
│ ├── css
│ │ ├── linuwial.css
│ │ ├── source-ayu-light.css
│ │ └── source-nord.css
│ ├── images
│ │ ├── tara-magicien.png
│ │ ├── tara-seating.svg
│ │ ├── tara-smiling.png
│ │ ├── tara-smiling.svg
│ │ ├── tara-teaching.png
│ │ └── tara-teaching.svg
│ └── js
│ ├── highlight.js
│ └── tex-chtml.js
└── HelloWorld.html
├── Stdlib.Data.Bool.html
├── Stdlib.Data.List.html
├── Stdlib.Data.Maybe.html
├── Stdlib.Data.Nat.html
├── Stdlib.Data.Ord.html
├── Stdlib.Data.Product.html
├── Stdlib.Data.String.html
├── Stdlib.Function.html
├── Stdlib.Prelude.html
└── Stdlib.System.IO.html
```
In addition, for the vscode-plugin, this PR adds two flags,
`--prefix-assets` and `--prefix-url`, for which one provides input to
help vscode find resource locations and Juvix files.
PS. Make sure to run `make clean` the first time you run `make install`
for the first time.
* Add files to embed in the exe to extra-source-files
Cabal requires the files to be specified here because it uses an isolated
build work directory.
* Add cabal configuration files
* Document how to use `cabal` with the project
