## Overview
This PR makes the compiler pipeline thread-safe so that the test suite
can be run in parallel.
This is achieved by:
* Removing use of `{get, set, with}CurrentDir` functions.
* Adding locking around shared file resources like the the
global-project and internal build directory.
NB: **Locking is disabled for the main compiler target**, as it is
single threaded they are not required.
## Run test suite in parallel
To run the test suite in parallel you must add `--ta '+RTS -N -RTS'` to
your stack test arguments. For example:
```
stack test --fast --ta '+RTS -N -RTS'
```
The `-N` instructs the Haskell runtime to choose the number of threads
to use based on how many processors there are on your machine. You can
use `-Nn` to see the number of threads to `n`.
These flags are already [set in the
Makefile](e6dca22cfd/Makefile (L26))
when you or CI uses `stack test`.
## Locking
The Haskell package
[filelock](https://hackage.haskell.org/package/filelock) is used for
locking. File locks are used instead of MVars because Juvix code does
not control when new threads are created, they are created by the test
suite. This means that MVars created by Juvix code will have no effect,
because they are created independently on each test-suite thread.
Additionally the resources we're locking live on the filesystem and so
can be conveniently tagged by path.
### FileLock
The filelock library is wrapped in a FileLock effect:
e6dca22cfd/src/Juvix/Data/Effect/FileLock/Base.hs (L6-L8)
There is an [IO
interpreter](e6dca22cfd/src/Juvix/Data/Effect/FileLock/IO.hs (L8))
that uses filelock and an [no-op
interpreter](e6dca22cfd/src/Juvix/Data/Effect/FileLock/Permissive.hs (L7))
that just runs actions unconditionally.
### TaggedLock
To make the file locks simpler to use a TaggedLock effect is introduced:
e6dca22cfd/src/Juvix/Data/Effect/TaggedLock/Base.hs (L5-L11)
And convenience function:
e6dca22cfd/src/Juvix/Data/Effect/TaggedLock.hs (L28)
This allows an action to be locked, tagged by a directory that may or
may not exist. For example in the following code, an action is performed
on a directory `root` that may delete the directory before repopulating
the files. So the lockfile cannot be stored in the `root` itself.
e6dca22cfd/src/Juvix/Extra/Files.hs (L55-L60)
## Pipeline
As noted above, we only use locking in the test suite. The main app
target pipeline is single threaded and so locking is unnecessary. So the
interpretation of locks is parameterised so that locking can be disabled
e6dca22cfd/src/Juvix/Compiler/Pipeline/Run.hs (L64)
- Depends on #2481
This pr allows inductive type parameters to be any type. Until now, they
had to be exactly `Type`. This allows us to define more general traits
such as the `Monad` and `Functor`, as shown in the new test.
This is only supported under the temporary `--new-typechecker` flag.
Pending work:
Update the positivity checker if necessary (@jonaprieto).
Update the necessary compilation steps in Core (@lukaszcz).
Add compilation tests.
This PR adds the `PackageDescription.Basic` module, available to
Package.juvix files.
```
module Package;
import PackageDescription.Basic open;
package : Package := basicPackage;
```
The `PackageDescription.Basic` module provides a Package type that is
translated to a Juvix Package with all default arguments. It is not
possible to customize a basic package.
A basic package does not depend on the standard library, so loads much
more quickly.
Additionally this PR:
* Adds `juvix init --basic/-b` option to generate a basic Package.juvix.
* Migrates Package.juvix files that only use default arguments, or only
customise the name field, to basic Package files.
* Closes https://github.com/anoma/juvix/issues/2508
- Closes#2362
This pr implements a new typechecking algorithm. This algorithm can be
activated using the global flag `--new-typechecker`. This flag will only
take effect on the compilation pipeline but not the repl.
The main difference between the new and old algorithm is that the new
one inserts holes during typechecking. Thus, it does not require the
arity checker pass.
The new algorithm does not yet implement default arguments. The plan is
to make the change in the following steps:
1. Merge this pr.
2. Merge #2506.
3. Implement default arguments for the new algorithm.
4. Remove the arity checker and the old algorithm.
---------
Co-authored-by: Łukasz Czajka <62751+lukaszcz@users.noreply.github.com>
This PR adds an initial support for Literate Juvix Markdown files, files
with the extension `.juvix.md`.
Here is a small example of such a file: `Test.juvix.md`.
<pre>
# This is a heading
Lorem ...
```juvix
module Test;
type A := a;
fun : A -> A
| _ := a;
```
Other text
</pre>
This initial support enables users to execute common commands such as
typechecking, compilation, and HTML generation. Additionally, a new
command called `markdown` has been introduced. This command replaces
code blocks marked with the juvix attribute with their respective HTML
output, much like the output we obtain when running `juvix html`. In
this version, comments are ignored in the output, including judoc
blocks.
- We intend to use this new feature in combination with this Python
plugin (https://github.com/anoma/juvix-mkdocs) to enhance our
documentation site.
https://github.com/anoma/juvix/assets/1428088/a0c17f36-3d76-42cc-a571-91f885866874
## Future work
Open as issues once this PR is merged, we can work on the following:
- Support imports of Juvix Markdown modules (update the path resolver to
support imports of Literate Markdown files)
- Support (Judoc) comments in md Juvix blocks
- Support Markdown in Judoc blocks
- Update Text editor support, vscode extension and emacs mode (the
highlighting info is a few characters off in the current state)
- Closes#1839
- Closes#1719
We can run `juvix format` on a whole juvix packages instead of
individual files.
This reduces the total time for `make check-format-juvix-files` from 10m
to 3m on my machine.
This PR:
* Modifies entry point `_entryPointBuildDir` to use the `BuildDir` type
instead of `SomeBase Dir`. This allows delayed resolution of the default
build directory which was useful for the Package -> Concrete translation
point below.
* Modifies `juvix dev root` to render the current package as a
Package.juvix file.
* Modifies the Package -> Concrete translation to recognise default
arguments. So, for example, an empty `juvix.yaml` file will be
translated into the following (instead of the `name`, `version`, and
`dependencies` arguments being populated).
module Package;
import Stdlib.Prelude open;
import PackageDescription.V1 open;
package : Package := defaultPackage;
* Adds a temporary command (removed when juvix.yaml support is removed)
`juvix dev migrate-juvix-yaml` that translates `juvix.yaml` into an
equivalent `Package.juvix` in the current project.
* Adds a temporary script `migrate-juvix-yaml.sh` (removed when
juvix.yaml support is removed) which can be run in the project to
translate all Juvix projects in the repository.
* Actually translate all of the `juvix.yaml` files to `Package.juvix`
using the script.
* Part of https://github.com/anoma/juvix/issues/2487
* Closes#2365
* Implements the syntax `f@{x1 := def1; ...; xn := defn}` and `f@?{x1 :=
def1; ..; xn := defn}`. Each definition inside the `@{..}` is an
ordinary function definition. The `@?` version allows partial
application (not all explicit named arguments need to be provided). This
subsumes the old record creation syntax.
The file walker `RecurseFilter` indictates if the the walker recurses
into
a directory which contains a Juvix project.
Previously the file walker would detect juvix.yaml. It now detects both
juvix.yaml and Package.juvix.
* Part of https://github.com/anoma/juvix/issues/2487
When moving to Package.juvix, the package configuration file cannot be
empty. So it's convenient to have a quick way to create a Package.juvix
file (previously you could run `touch juvix.yaml`.
This PR adds the `-n / --non-interactive` option to `juvix init`. This
will create a default `Package.juvix`, using the name of the current
directory to populate the package name.
Similarly for the interactive version of juvix init, if the name of the
current directory is not a valid Juvix module then a fallback name is
used instead.
For example:
```
$ mkdir /tmp/new-package
$ cd /tmp/new-package
$ juvix init -n
$ cat Package.juvix
module Package;
import PackageDescription.V1 open;
package : Package :=
defaultPackage
{name := "new-package";
version := mkVersion 0 0 0;
dependencies := [defaultStdlib]};
```
* Part of https://github.com/anoma/juvix/issues/2487
## Global package configuration
This PR updates the global-project to use `Package.juvix` instead of
`juvix.yaml`.
The global package gets the following `Package.juvix`:
```
module Package;
import PackageDescription.V1 open;
package : Package :=
defaultPackage
{name := "global-juvix-package";
version := mkVersion 0 0 0;
dependencies := [defaultStdlib]};
```
## juvix clean --global
This PR also adds an option `-g/--global` to `juvix clean` that removes
the `$XDG_CONFIG_HOME/juvix/VERSION` directory.
## Testing notes
If you've already run Juvix 0.5.3 you'll need to run `juvix clean -g`
before you'll use the new global project. This will not be an issue for
users of Juvix 0.5.4 (as this version of Juvix will never generate
juvix.yaml in the global project).
Part of:
* https://github.com/anoma/juvix/issues/2487
Paths are used as keys in the PathResolver ResolverCache. These paths
must be normalised because a path may have a different representation
when it is used to set an entry in the cache and when it is used to
lookup from the cache.
Path normalisation is also used in the REPL before a directory is
compared with the standard library path.
* Closes https://github.com/anoma/juvix/issues/2497
`juvix init` now generates a `Package.juvix` file in the current
directory instead of a `juvix.yaml` file. It uses the prompts from the
user to fill in the name and version options.
### Validity check
After the file is generated, the Juvix project in the current directory
is loaded to check that the generated file is valid.
### Version support
Each version of the PackageDescription module must have a corresponding
[PackageDescriptionType](c39c27000c/src/Juvix/Compiler/Pipeline/Package/Loader/Versions.hs (L12))
which defines:
* The path relative to `include/package` that the file exists at (e.g
`PackageDescription/V1.juvix`)
* The name of the Package type in the file (e.g `Package`)
* A function that translates a Package type into a Concrete function
definition that represents the code that is generated in the
`Package.juvix`
* A function that returns a Bool that determines if the generated
function definition depends on the standard library. (i.e that the
standard library needs to be imported by Package.juvix).
The last point is necessary because in the V1 spec, the `SemVer` type
uses `Maybe` to encode the release and meta components of the of the
version:
```
package : Package :=
defaultPackage
{name := "dd";
version := mkVersion 1 2 3 {just "prerel"} {just "meta"}};
```
So if the user specifies a release or meta component in their version -
then the standard library needs to be imported to provide the `just`
constructor. If the user only specifies major, minor and patch
components then the standard library does not need to be imported.
* Closes https://github.com/anoma/juvix/issues/2485
* Closes#2453
* Closes#2432
* Any nonnegative literal `n` is replaced with `fromNat {_} {{_}} n`
where `fromNat` is the builtin conversion function defined in the
`Natural` trait in `Stdlib.Trait.Natural`.
* Any negative literal `-n` is replaced with `fromInt {_} {{_}} -n`
where `fromInt` is the builtin conversion function defined in the
`Integral` trait in `Stdlib.Trait.Integral`.
* Before resolving instance holes, it is checked whether the type holes
introduced for `fromNat` and `fromInt` have been inferred. If not, an
attempt is made to unify them with `Nat` or `Int`. This allows to
type-check e.g. `1 == 1` (there is no hint in the context as to what the
type of `1` should be, so it is decided to be `Nat` after inferring the
hole fails).
We use the options in `package.yaml`.
```
- -fhide-source-paths
- -fwrite-ide-info -hiedir=.hie
```
If a previously available .hie directory is missing then GHC will
rebuild the whole project with reason: `[HIE file is missing]`. So we
need to cache it to take advantage of incremental builds.
In this PR, we ran the Juvix formatter so that we can now freely run
`make format`, `make check`, or `make pre-commit` without any unexpected
file changes.
This goes after:
- https://github.com/anoma/juvix/pull/2486
This PR adds a version number to the module name of the
`PackageDescription` module. This allows us to change the Package file
format in a backwards compatible way in the future.
Now the simplest Package.juvix file looks like:
```
module Package;
import PackageDescription.V1 open;
package : Package := defaultPackage;
```
## Adding new versions
The idea is that new versions of the PackageDescription format will be
added as files of the form:
include/package/PackageDescription/Vx.juvix
The correspondence between a version of the PackageDescription file and
the package type name is recorded in
[`packageDescriptionTypes`](9ba869d836/src/Juvix/Compiler/Pipeline/Package/Loader.hs (L15)).
The `package` identifier type must come from **one** of the versions
of the PackageDescription module.
* Closes https://github.com/anoma/juvix/issues/2452
### Example of nested list formatting using the new method:
```
l : List (List Int) :=
[ [1; 2; 3]
; longLongLongListArg
; [ longLongLongListArg
; longLongLongListArg
; longLongLongListArg
; longLongLongListArg
]
; longLongLongListArg
];
```
* Closes https://github.com/anoma/juvix/issues/2466
Depends on:
* ~~https://github.com/anoma/juvix/pull/2459~~
* https://github.com/anoma/juvix/pull/2462
This PR is part of a series implementing:
* https://github.com/anoma/juvix/issues/2336
This PR adds the package file loading function, including a file
evaluation effect. It integrates this with the existing `readPackage`
function and adds tests / smoke tests.
## Package.juvix format
Instead of `juvix.yaml` (which is still supported currently) users can
now place a `Package.juvix` file in the root of their project. The
simplest `Package.juvix` file you can write is:
```
module Package;
import PackageDescription open;
package : Package := defaultPackage;
```
The
[PackageDescription](35b2f618f0/include/package/PackageDescription.juvix)
module defines the `Package` type. Users can use "go-to definition" in
their IDE from the Package file to see the documentation and
definitions.
Users may also import `Stdlib.Prelude` in their Package file. This is
loaded from the global project. No other module imports are supported.
Notes:
* If a directory contains both `Package.juvix` and `juvix.yaml` then
`Package.juvix` is used in preference.
## Default stdlib dependency
The `Dependency` type has a constructor called `defaultStdlib`. This
means that any project can use the compiler builtin standard library
dependency. With `juvix.yaml` this dependency is only available when the
`dependencies` field is unspecified.
```
module Package;
import PackageDescription open;
package : Package := defaultPackage { dependencies := [defaultStdlib] };
```
## Validation
As well as the standard type checking validation that the Juvix compiler
provides additional validation is made on the file.
* The Package module must contain the identifier `package` and it must
have type `Package` that's obtained from the global `PackageDescription`
module.
* Every dependency specified in the Package.juvix must be unique.
* Closes https://github.com/anoma/juvix/issues/2336
## Examples
### Package with name and version
```
module Package;
import PackageDescription open;
package : Package :=
defaultPackage {name := "a-package";
version := mkVersion 0 1 0};
```
### Package with GitHub dependency
```
module Package;
import PackageDescription open;
package : Package :=
defaultPackage {name := "a-package";
version := mkVersion 0 1 0;
dependencies := [defaultStdlib;
github (org := "anoma";
repo := "juvix-containers";
ref := "v0.7.1")]};
```
## Package with main and buildDir fields
```
module Package;
import Stdlib.Prelude open;
import PackageDescription open;
package : Package :=
defaultPackage {name := "a-package";
version := mkVersion 0 1 0;
dependencies := [defaultStdlib;
github (org := "anoma";
repo := "juvix-containers";
ref := "v0.7.1")];
buildDir := just "/tmp/build";
main := just "HelloWorld.juvix"
};
```
Previously the doctor help links would point to the Juvix documentation
dev URLs.
Now the doctor help links point to the version of the documentation
corresponding to the compiler version.
```
$ juvix doctor
> Checking for clang...
> Checking clang version...
> Checking for wasm-ld...
> Checking that clang supports wasm32...
> Checking that clang supports wasm32-wasi...
> Checking that WASI_SYSROOT_PATH is set...
! Environment variable WASI_SYSROOT_PATH is missing
! https://docs.juvix.org/0.5.2/reference/tooling/doctor/#environment-variable-wasi_sysroot_path-is-not-set
> Checking for wasmer...
> Checking latest Juvix release on Github...
```
Spotted by @agureev
If you have `juvix.yaml` file in a parent of the juvix source directory
in your filesystem then some Compilation tests will fail. This is
because the tests assume that the project root will be in the test
directory.
This PR just adds `juvix.yaml` files to those test directories to avoid
this issue.
Spotted by @agureev
The problem with readFile and writeFile from text
[Data.Text.IO](https://hackage.haskell.org/package/text-2.0.2/docs/Data-Text-IO.html)
is that they use the system locale to determine the text encoding
format.
Our assumption is that all Juvix source files are UTF-8 encoded.
I cannot reproduce the issue with using the old APIs on my machine, it
can be reproduced on Arch linux. I'm not sure how to write a specific
test for this.
* Closes https://github.com/anoma/juvix/issues/2472
This PR introduces FileExt type, and consequently, one can generalise
methods and matches based on the file extension; for example,
`parseInputJuvixAsmFile` is now an app. `parseInputFile FileExtJuvixAsm`
This PR adds `--prefix-id`, `--no-path`, and `only-code` flags to the
HTML backend to manipulate the hyperlinks on the resulting HTML output
and the output itself by only keeping the content of the body in the
Html.
As a usage case, we can support `juvix-standalone` blocks, as
demonstrated in
- https://github.com/anoma/juvix-docs/pull/80
Simplifies arithmetic expressions in the Core optimization phase,
changing e.g. `(x - 1) + 1` to `x`. Such expressions appear as a result
of compiling pattern matching on natural numbers.
The special PathResolver puts files from the global package stdlib and
files from the global package description files in scope of the
$root/Package.juvix module.
Currently this means that PackageDescription module is in scope for the
module so that the user can write:
```
module Package;
import Stdlib.Prelude open;
import PackageDescription open;
package : Package :=
mkPackageDefault
(name := "foo")
{ version := mkVersion 0 1 0
; dependencies :=
[ github "anoma" "juvix-stdlib" "adf58a7180b361a022fb53c22ad9e5274ebf6f66"
; github "anoma" "juvix-containers" "v0.7.1"]};
```
* Closes#2154
* Evaluates closed applications with value arguments when the result
type is zero-order. For example, `3 + 4` is evaluated to `7`, and `id 3`
is evaluated to `3`, but `id id` is not evaluated because the target
type is not zero-order (it's a function type).
* Closes#2426
A coercion from trait `T` to `T'` can be declared with the syntax
```
coercion instance
coeName {A} {{T A}} : T' A := ...
```
Coercions can be seen as instances with special resolution rules.
Coercion resolution rules
-------------------------
* If a non-coercion instance can be applied in a single instance
resolution step, no coercions are considered. No ambiguity results if
there exists some coercion which could be applied, but a non-coercion
instance exists - the non-coercion instances have priority.
* If no non-coercion instance can be applied in a single resolution
step, all minimal coercion paths which lead to an applicable
non-coercion instance are considered. If there is more than one,
ambiguity is reported.
Examples
----------
The following type-checks because:
1. There is no non-coercion instance found for `U String`.
2. There are two minimal coercion paths `U` <- `U1` and `U` <- `U2`, but
only one of them (`U` <- `U2`) ends in an applicable non-coercion
instance (`instU2` for `U2 String`).
```
trait
type U A := mkU {pp : A -> A};
trait
type U1 A := mkU1 {pp : A -> A};
trait
type U2 A := mkU2 {pp : A -> A};
coercion instance
fromU1toU {A} {{U1 A}} : U A :=
mkU@{
pp := U1.pp
};
coercion instance
fromU2toU {A} {{U2 A}} : U A :=
mkU@{
pp := U2.pp
};
instance
instU2 : U2 String := mkU2 id;
main : IO := printStringLn (U.pp "X")
```
The following results in an ambiguity error because:
1. There is no non-coercion instance found for `T Unit`.
2. There are two minimal coercion paths `T` <- `T1` and `T` <- `T2`,
both of which end in applicable non-coercion instances.
```
trait
type T A := mkT { pp : A → A };
trait
type T1 A := mkT1 { pp : A → A };
trait
type T2 A := mkT2 { pp : A → A };
instance
unitT1 : T1 Unit := mkT1 (pp := λ{_ := unit});
instance
unitT2 : T2 Unit := mkT2 (pp := λ{_ := unit});
coercion instance
fromT1toT {A} {{T1 A}} : T A := mkT@{
pp := T1.pp
};
coercion instance
fromT2toT {A} {{T2 A}} : T A := mkT@{
pp := T2.pp
};
main : Unit := T.pp unit;
```
The following type-checks, because there exists a non-coercion instance
for `T2 String`, so the coercion `fromT1toT2` is ignored during instance
resolution.
```
trait
type T1 A := mkT1 {pp : A -> A};
trait
type T2 A := mkT2 {pp : A -> A};
instance
instT1 {A} : T1 A :=
mkT1@{
pp := id
};
coercion instance
fromT1toT2 {A} {{M : T1 A}} : T2 A :=
mkT2@{
pp := T1.pp {{M}}
};
instance
instT2 : T2 String :=
mkT2@{
pp (s : String) : String := s ++str "!"
};
main : String := T2.pp "a";
```
* Introduces the `inline: case` pragma which causes an identifier to be
inlined if it is matched on. This is necessary to support GEB without
compromising optimization for other targets.
* Adapts to the new commits in
https://github.com/anoma/juvix-stdlib/pull/86
