Builtin information needs to be propagated from stored modules to REPL
artifacts to avoid "The builtin _ has not been defined" errors.
This PR adds a test suite for the REPL in the Haskell test code. This
means some of the slow smoke tests can be moved to fast haskell unit
tests. In future we should refactor the REPL code by putting in the main
src target and unit testing more features (e.g :doc, :def).
* Closes https://github.com/anoma/juvix/issues/2638
* Closes https://github.com/anoma/juvix/issues/2664
As well as this fix we rename lens scopedIdenName to scopedIdenSrcName.
`scopedIdenSrcName` refers to the name of an identifier from the source
code. The name `scopedIdenName` is confusing because users of
`ScopedIden` may think that this lens refers to the only name associated
with `ScopedIden`, they may want `scopedIdenNameFinal` instead.
See #2670 for an example which triggers the bug.
The nockma case compilation did not correctly compile case expressions
for standard (i.e not list or tuple) constructors.
Existing compilation tests (e.g Tree, Lambda Calculus) did not fail due
to the relevant `fromJust` never being evaluated due to lazy evaluation.
The tests now write out the resulting nockma file to force full
evaluation.
* Closes https://github.com/anoma/juvix/issues/2670
---------
Co-authored-by: Paul Cadman <git@paulcadman.dev>
This PR adds a `anoma` target to the `juvix compile`. This target
compiles a Juvix `main` function to a Nockma/Anoma "function". Unlike
the native, wasm, and nockma targets the main function may have any type
signature.
## Anoma calling convention
[Anoma calls
functions](6a4e15fe9c/lib/anoma/resource.ex (L122))
by evaluating the formula `[call L replace [RL args] @ S]` against a
subject equal to the function. Here `args` is a Nockma term that
evaluates to a tuple of arguments that should be passed to the function.
The anoma target compiles the `main` function to Nockma in the same way
as the nockma target. The main function is then
[wrapped](9a658465ae/src/Juvix/Compiler/Nockma/Translation/FromTree.hs (L627))
to make it compatible with the Anoma calling convention.
## Testing
The anoma calling convention is [unit
tested](9a658465ae/test/Nockma/Eval/Positive.hs (L117))
and [smoke
tested](9a658465ae/tests/smoke/Commands/compile.smoke.yaml (L159)).
This PR also adds versions of the end-to-end compilation tests. Most
tests are included, tests for builtin IO operations and string builtins
are omitted. Other tests that use Strings have been adapted to use other
types that are compatible with this backend.
## Nockma REPL
To facilitate testing the Nockma REPL can now load a nockma file as an
initial subject.
---------
Co-authored-by: Lukasz Czajka <lukasz@heliax.dev>
* Closes#2576
* Adds a JuvixReg transformation `InitBranchVars` which inserts
assignments to initialize variables assigned in other branches. Assumes
the input is in SSA form (which is preserved).
* Adds tests for the `InitBranchVars` transformation.
* Depends on #2647
* Closes#2560
* Adds a transformation of JuvixReg into SSA form.
* Adds an "output variable" field to branching instructions (`Case`,
`Branch`) which indicates the output variable to which the result is
assigned in both branches. The output variable corresponds to top of
stack in JuvixAsm after executing the branches. In the SSA
transformation, differently renamed output variables are unified by
inserting assignment instructions at the end of branches.
* Adds tests for the SSA transformation.
* Depends on #2641.
Now the prelude exports this function:
```
readFile :: (MonadIO m) => Path Abs File -> m Text
readFile = liftIO . Utf8.readFile . toFilePath
```
It is more convenient to use because it uses typed `Path` and works in
any `MonadIO`.
* Implements JuvixReg recursors, which will allow to implement JuvixReg
transformations more succinctly and effectively.
* Adds a transformation framework to JuvixReg.
* Adds identity transformation tests.
* Depends on #2635
# Overview
This pr implements a simple benchmark suite to compare the efficiency of
[`effectful-core`](https://hackage.haskell.org/package/effectful-core)
and [`polysemy`](https://hackage.haskell.org/package/polysemy).
I've implemented the suite with the help of
[`tasty-bench`](https://hackage.haskell.org/package/tasty-bench). It is
a simple benchmarking library that has minimal dependencies and it can
be run with a default main using the same cli options as our
[`tasty`](https://hackage.haskell.org/package/tasty) test suite.
# How to run
```
stack run juvixbench
```
If you only want to run a particular benchmark:
```
stack run juvixbench -- -p "/Output/"
```
# Results
The results show that `effectful` is the clear winner, in some cases it
is extremely close to the raw version.
## State
This benchmark adds the first 2 ^ 22 first naturals:
```
countRaw :: Natural -> Natural
countRaw = go 0
where
go :: Natural -> Natural -> Natural
go acc = \case
0 -> acc
m -> go (acc + m) (pred m)
```
Results:
```
State
Eff State (Static): OK
25.2 ms ± 2.4 ms
Sem State: OK
2.526 s ± 5.1 ms
Raw State: OK
22.3 ms ± 1.5 ms
```
## Output
This benchmark collects the first 2 ^ 21 naturals in a list and adds
them.
```
countdownRaw :: Natural -> Natural
countdownRaw = sum' . reverse . go []
where
go :: [Natural] -> Natural -> [Natural]
go acc = \case
0 -> acc
m -> go (m : acc) (pred m)
```
Results:
```
Eff Output (Dynamic): OK
693 ms ± 61 ms
Eff Accum (Static): OK
553 ms ± 36 ms
Sem Output: OK
2.606 s ± 91 ms
Raw Output: OK
604 ms ± 26 ms
```
## Reader (First Order)
Repeats a constant in a list and adds it. The effects based version ask
the constant value in each iteration.
```
countRaw :: Natural -> Natural
countRaw = sum' . go []
where
go :: [Natural] -> Natural -> [Natural]
go acc = \case
0 -> acc
m -> go (c : acc) (pred m)
```
Results:
```
Reader (First order)
Eff Reader (Static): OK
103 ms ± 6.9 ms
Sem Reader: OK
328 ms ± 31 ms
Raw Reader: OK
106 ms ± 1.9 ms
```
## Reader (Higher Order)
Adds the first 2 ^ 21 naturals. The effects based version use `local`
(from the `Reader`) effect to pass down the argument that counts the
iterations.
```
countRaw :: Natural -> Natural
countRaw = sum' . go []
where
go :: [Natural] -> Natural -> [Natural]
go acc = \case
0 -> acc
m -> go (m : acc) (pred m)
```
Results:
```
Reader (Higher order)
Eff Reader (Static): OK
720 ms ± 56 ms
Sem Reader: OK
2.094 s ± 182 ms
Raw Reader: OK
154 ms ± 2.2 ms
```
## Embed IO
Opens a temporary file and appends a character to it a number of times.
```
countRaw :: Natural -> IO ()
countRaw n =
withSystemTempFile "tmp" $ \_ h -> go h n
where
go :: Handle -> Natural -> IO ()
go h = \case
0 -> return ()
a -> hPutChar h c >> go h (pred a)
```
Results:
```
Embed IO
Raw IO: OK
464 ms ± 12 ms
Eff RIO: OK
487 ms ± 3.5 ms
Sem Embed IO: OK
582 ms ± 33 ms
```
- ⚠️ Depends on #2644
The `effectful` library does not support the `Embed` effect out of the
box. However, it offers `IOE`, which is equivalent to `Embed IO` from
polysemy. In preparation to a possible migration to `effectful`, this pr
hides the general `Embed` effect from the prelude and it exports a
specialized `EmbedIO` in its place.
Since we upgraded to ghc-9.8.1 and dropped the dependency on
[with-utf8](https://hackage.haskell.org/package/with-utf8), the function
`writeFileEnsureLn` no longer has the constraint `MonadMask m`, so the
`embed @IO` has become redundant
The alpine abstract packages clang/llvm now meet minimum version
requirements in the Docker container we're using for the linux static
build.
The llvm package must now be installed separately to get the `llvm-ar`
tool.
This pr addresses a number of problems.
1. It fixes a bug where paths were annotated as operations rather than
paths in the parser.
2. It fixes a bug that happened when unfolding cells in the pretty
printer in order to minimize delimiters. It caused the stdlibcall hints
to be ignored for the unfolded arguments.
3. In order to properly test this, we can't ignore the hints for the Eq
instance, so I've changed that.
4. I've introduced the class NockmaEq for nockma semantic equality. This
is used in the evaluator as well as in the semantic tests.
5. I've added a bigger test. I found these bugs while working with this
file.
* Closes#2578
* Implements JuvixReg parser and pretty printer.
* Adds the `juvix dev reg read file.jvr` command.
* Adds the `reg` target to the `compile` commands.
* Adds tests for the JuvixReg parser.
This allows us to do `juvix dev tree compile --target nockma test.jvt`.
The pipeline was already implemented, this pr only adds the
`TargetNockma` to the list of supported backends so that the cli
recognizes it.
This PR introduces the `pow2` function from the Anoma/Nock stdlib,
replacing the 'Builtin function' version we wrote in the Tree->Nockma
translation. This will be more efficient because it will be jetted in
Anoma.
## New Anoma/nockma stdlib locations
The Nock stdlib now has multiple layers. We now use a Nock term to
to fetch each stdlib symbol's code because this can be obtained from
Urbit's dojo,
for example:
> => anoma !=(add)
[9 20 0 15]
where `anoma` is the symbol where the anoma stdlib is loaded.
The stdlib is the Nock code associated with
6a4e15fe9c/hoon/anoma.hoon
## Tests
This commit also adds unit tests for the stdlib / appendRights
functions. The tests use the evaluator with 'interceptStdlibCalls' both
enabled and disabled.
This pr implements two additional versions of the Juvix Tree evaluator.
Now we have
1. The raw implementation that does not use effects. It throws Haskell
exceptions for errors. It uses `unsafePerformIO` for traces. It relies
on bang patterns to force strictness and guarantee the expected order of
execution (for traces). Avoiding effects allows for improved execution
efficiency.
2. [`polysemy`](https://hackage.haskell.org/package/polysemy-1.9.1.3)
based implementation.
3.
[`effectful-core`](https://hackage.haskell.org/package/effectful-core)
based implementation.
One can specify which evaluator to use thus:
```
juvix dev tree eval --evaluator XXX test.jvt
```
where XXX is one of `raw`, `polysemy`, `effectful`.
# Preliminary benchmarks
More thorough benchmarks should be run, but here are some preliminary
results:
## Test032 (Fibonacci 20)
I've adapted test032 so that it main is a single call to fibonacci of
20.
Command:
```
hyperfine --warmup 2 --runs 10 'juvix dev tree eval test032.jvt --evaluator polysemy' 'juvix dev tree eval test032.jvt --evaluator raw' 'juvix dev tree eval test032.jvt --evaluator e
ffectful'
```
Output:
```
Benchmark 1: juvix dev tree eval test032.jvt --evaluator polysemy
Time (mean ± σ): 2.133 s ± 0.040 s [User: 2.113 s, System: 0.016 s]
Range (min … max): 2.088 s … 2.227 s 10 runs
Benchmark 2: juvix dev tree eval test032.jvt --evaluator raw
Time (mean ± σ): 308.7 ms ± 13.8 ms [User: 293.6 ms, System: 14.1 ms]
Range (min … max): 286.5 ms … 330.1 ms 10 runs
Benchmark 3: juvix dev tree eval test032.jvt --evaluator effectful
Time (mean ± σ): 366.0 ms ± 2.8 ms [User: 345.4 ms, System: 19.4 ms]
Range (min … max): 362.5 ms … 372.6 ms 10 runs
Summary
juvix dev tree eval test032.jvt --evaluator raw ran
1.19 ± 0.05 times faster than juvix dev tree eval test032.jvt --evaluator effectful
6.91 ± 0.34 times faster than juvix dev tree eval test032.jvt --evaluator polysemy
```
## Test034 (Higher-order function composition)
A modified version of test034 where main defined as `call[exp](3, 12)`
Command:
```
hyperfine --warmup 2 --runs 10 'juvix dev tree eval test034.jvt --evaluator polysemy' 'juvix dev tree eval test034.jvt --evaluator raw' 'juvix dev tree eval test034.jvt --evaluator effectful'
```
Output:
```
Benchmark 1: juvix dev tree eval test034.jvt --evaluator polysemy
Time (mean ± σ): 7.025 s ± 0.184 s [User: 6.518 s, System: 0.469 s]
Range (min … max): 6.866 s … 7.327 s 10 runs
Benchmark 2: juvix dev tree eval test034.jvt --evaluator raw
Time (mean ± σ): 835.6 ms ± 7.4 ms [User: 757.2 ms, System: 75.9 ms]
Range (min … max): 824.7 ms … 847.4 ms 10 runs
Benchmark 3: juvix dev tree eval test034.jvt --evaluator effectful
Time (mean ± σ): 1.578 s ± 0.010 s [User: 1.427 s, System: 0.143 s]
Range (min … max): 1.563 s … 1.595 s 10 runs
Summary
juvix dev tree eval test034.jvt --evaluator raw ran
1.89 ± 0.02 times faster than juvix dev tree eval test034.jvt --evaluator effectful
8.41 ± 0.23 times faster than juvix dev tree eval test034.jvt --evaluator polysemy
```
## Test036 (Streams without memoization)
A modified version of test036 where main defined as `call[nth](700,
call[primes]())`
Command:
```
hyperfine --warmup 2 --runs 5 'juvix dev tree eval test036.jvt --evaluator polysemy' 'juvix dev tree eval test036.jvt --evaluator raw' 'juvix dev tree eval test036.jvt --evaluator effectful'
```
Output:
```
Benchmark 1: juvix dev tree eval test036.jvt --evaluator polysemy
Time (mean ± σ): 1.993 s ± 0.026 s [User: 1.946 s, System: 0.043 s]
Range (min … max): 1.969 s … 2.023 s 5 runs
Benchmark 2: juvix dev tree eval test036.jvt --evaluator raw
Time (mean ± σ): 137.5 ms ± 7.1 ms [User: 127.5 ms, System: 8.9 ms]
Range (min … max): 132.8 ms … 149.8 ms 5 runs
Benchmark 3: juvix dev tree eval test036.jvt --evaluator effectful
Time (mean ± σ): 329.0 ms ± 7.3 ms [User: 289.3 ms, System: 37.4 ms]
Range (min … max): 319.9 ms … 336.0 ms 5 runs
Summary
juvix dev tree eval test036.jvt --evaluator raw ran
2.39 ± 0.13 times faster than juvix dev tree eval test036.jvt --evaluator effectful
14.50 ± 0.77 times faster than juvix dev tree eval test036.jvt --evaluator polysemy
```
This PR updates the stackage LTS resolver to `nightly-2024-02-06` which
uses GHC 9.8.1
## Upgrade notes
You will need to update your HLS to
[2.6.0.0](https://github.com/haskell/haskell-language-server/releases/tag/2.6.0.0),
this release contains support for GHC 9.8.1
## Fixes
### `haskeline` / `repline`
We have removed the custom haskeline / repline forks used in the build.
This is because we had trouble overriding haskeline as it is bundled
with GHC and the stackage resolver uses this bundled version. We were
using a custom fork of haskeline to implement
[mapInputT_](15c0685c91/app/Commands/Repl.hs (L409))
in the Juvix REPL, required to implement error handling. This requires
private API from the Haskeline library.
Instead of using a custom fork we use TemplateHaskell to obtain access
to the private API we need. See
[DarkArts.hs](15c0685c91/src/Juvix/Prelude/DarkArts.hs)
and
[HaskelineJB.hs](15c0685c91/app/HaskelineJH.hs).
To obtain access to the private API, we adapted a method from [a Tweag
blogpost](https://www.tweag.io/blog/2021-01-07-haskell-dark-arts-part-i/)
and [repo](https://github.com/tweag/th-jailbreak) - updating it for GHC
9.8.1.
### `aeson-better-errors`
The `aeson-better-errors` library has not been updated to work with
`mtl-2.3.0` so it cannot work with the new stackage resolver. We are
using a [fork](https://github.com/Vekhir/aeson-better-errors.git) which
has been updated.
We should consider replacing this library in future, see
https://github.com/anoma/juvix/issues/2621
### `path`
The `path` library now includes API `splitDrive` and `dropDrive` so we
can remove our versions of those functions from the prelude.
### `with-utf8`
We no longer need to depend on `with-utf8`. We were using this package
for UTF-8 versions of `readFile` and `writeFile` APIs. These APIs are
now available in the `text` package.
### Compiler warnings
GHC 9.8.1 introduces several new compiler warnings.
* We have suppressed `missing-role-annotations` and
`missing-poly-kind-signatures`
* We added our own versions of `head` and `tail` to work around the new
`partial-tx` warning introduced for those functions in `Data.List`.
* We fixed up the code to avoid the
[term-variable-capture](https://ghc.gitlab.haskell.org/ghc/doc/users_guide/using-warnings.html#ghc-flag--Wterm-variable-capture)
warning.
This PR replaces the JuvixAsm -> Nockma translation with a JuvixTree ->
Nockma translation.
We can now enable some of the JuvixTree tests that did not work with
JuvixAsm->Nockma because they were too slow.
## Notes
We have changed [test031: temp stack with
branching](22ee87f0e7/tests/Tree/positive/test031.jvt)
to avoid using negative numbers (because negative integers are not
supported in Nockma).
Three tree tests trace/output lists. Lists are serialised differently by
the asm and nockma pretty printers so they cannot share a single test
output file. We have created separate nockma output files for these
tests (see eg.
[test028.nockma.out](22ee87f0e7/tests/Tree/positive/out/test028.nockma.out)).
* Closes https://github.com/anoma/juvix/issues/2606
---------
Co-authored-by: Paul Cadman <git@paulcadman.dev>
Co-authored-by: Lukasz Czajka <lukasz@heliax.dev>
Quoting the [release notes of gnu
`grep-3.8`](https://lists.gnu.org/archive/html/info-gnu/2022-09/msg00001.html)
> The egrep and fgrep commands, which have been deprecated since
release 2.5.3 (2007), now warn that they are obsolescent and should
be replaced by grep -E and grep -F.
In order to fix the warning, I've replaced all instances of `egrep` with
`grep -E`
This PR adds a https://github.com/casey/just
```
$ just
Available recipes:
build *opts # Build the project. `build runtime` builds only the runtime.
clean *opts # Clean the project. `clean runtime` cleans only the runtime. `clean juvix-build` cleans only juvix-build dirs.
format *opts # Formats all Haskell files in the project. `format changed` formats only changed files. `format FILES` formats individual files.
install # Install juvix
test *filter # Run the tests in the project. Use the filter arg to set a Tasty pattern.
```
---------
Co-authored-by: Jan Mas Rovira <janmasrovira@gmail.com>
* Adds negative tests for the JuvixTree evaluator
* Depends on #2600
* Depends on #2599
* Depends on #2598
* Depends on #2597
* Depends on #2596
* Depends on #2595
* Depends on #2594
* Depends on #2590
* Adds the `--transforms`, `--eval` and `--no-print` options to the
`juvix dev tree read` command.
* Depends on #2598
* Depends on #2597
* Depends on #2596
* Depends on #2595
* Depends on #2594
* Depends on #2590
This PR fixes an issue with importing `syntax alias` and using them from
another module:
Say you have a module defining a syntax alias:
```
module SyntaxAlias;
import Stdlib.Prelude open;
syntax alias MyNat := Nat;
```
and another which imports / uses it:
```
module test073;
import SyntaxAlias open;
main : MyNat := 11;
```
The compiler crashed with the following error:
```
^?!): empty Fold
CallStack (from HasCallStack):
error, called at src/Lens/Micro.hs:711:28 in microlens-0.4.13.1-ARwI8t2x86cAxRs56XPcG1:Lens.Micro
^?!, called at src/Juvix/Compiler/Concrete/Translation/FromParsed/Analysis/Scoping.hs:565:29 in juvix-0.5.5-G7jC6MbkbsJGkMT9u4BkYQ:Juvix.Compiler.Concrete.Translation.FromParsed.Analysis.Scoping
```
We fix this by adding the aliases to the store's [scoper info
table](6649209d26/src/Juvix/Compiler/Store/Scoped/Data/InfoTable.hs (L14))
and then use it to initialise the scoper state when scoping a module.
Now we guarantee that whenever we write a file there is a newline
character at the end, which is a [Unix
convention](https://pubs.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap03.html#tag_03_205).
The juvix prelude now exports `writeFileEnsureLn` and it no longer
exports `writeFile`. If at some point we need the behaviour of
`writeFile` I'd suggest that we export it renamed as `writeFileVerbatim`
or something similar.
* Adds smoke tests for `juvix dev tree` CLI commands.
* Fixes a bug with printing JuvixAsm.
* Depends on #2597
* Depends on #2596
* Depends on #2595
* Depends on #2594
* Depends on #2590
* Adds the `FilterUnreachable` transformation in JuvixTree.
* Depends on #2596
* Depends on #2595
* Depends on #2594
* Depends on #2590
* Depends on #2589
* Depends on #2587
* Moves the "apply" transformation from JuivxAsm to JuvixTree. This
transformation removes the `CallClosures` nodes.
* Makes Nockma compilation tests use JuvixTree instead of JuvixAsm
files.
* Depends on #2594
* Depends on #2590
* Depends on #2589
* Depends on #2587
* Generalizes JuvixCore map and fold recursors to work also for
JuvixTree.
* Adds a transformation framework to JuvixTree.
* Adds identity trasformation tests for JuvixTree.
* Depends on #2590
* Depends on #2589
* Depends on #2587
