Update various flake inputs and the flake.lock file. Adapt the flake.nix
file, as well as the Rust source code, to accomodate the latest changes
(new clippy warnings, etc.).
Topiary is getting hard to use from the flake, because there are two
conflicting versions: the one that is pulled from Nix to be used in the
CI (checking that files are properly formatted), and the one built into
Nickel via cargo. Both must agree (or at least there might be a
difference in formatting between the two if they aren't the same
version). Since the addition of dynamic loading of grammars, latest
Topiary has become harder to build from Nix.
To avoid all those pitfalls, this commit gets rid of the Topiary as a
flake input, and use `nickel format` instead, ensuring that the
formatting is consistent. As a consequence, Topiary isn't included in
the development shell anymore, but it's arguably not an issue: it was
included before `nickel format`, as we needed a third party formatter,
but now one can just build Nickel locally with their preferred method
and use `nickel format`.
* Doctests
* Add a custom_transform method
* Make the vm private again
* Update doc
* Add some snapshot tests
* Split out closurized and non-closurized variants of eval_record_spine
* Fix doc link
* Add some docs, but not sure where they should go
* Make report_as_str honor the color setting
* Separate the stdout and stderr outputs more intentionally
* Commit the other snapshots
* Explicit imports
* explicit import: more changes.
* Rename typ to format
* Bring back the fallback
* Implement pretty-printer part
* Fix compilation of NLP
* Document imports
* Apply suggestions from code review
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
* minor fix
* explicit imports: add tests
* explicit imports: support importing the same file with different formats
* fmt
* Reinforce explicit imports test against some whitespace changes.
---------
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
* Add attrs to let patterns
* Implement let-rec patterns
* Fix markdown
* More markdown
* Put a test in the right place
* Revert the comment change
* Review comments
* Add let blocks to the AST
* Add parsing and tests
* Fix build
* Missing insta
* Fix it again
* Add a custom parse error
* Try out smallvec
* Support multiple patterns in the ast
* Support constructing let blocks with patterns
* Fix warning
* Review comments
* WIP
* WIP
* Fix typechecking
* Add lsp tests
* Add an error for duplicate bindings in a let block
* Update the manual
* Delete the old pattern file.
* Move test to integration
* Initial draft without Constant and TailVar
* Restrict visibility of RemoveErrorRow to super
* Create trait Subsume for UnifType and UnifRecordRows
* Fix problem about {a: Type}<: {a:Type;b:Type} that was not accepted
* Copy comments (general comment on subsumption and closing a record when there is a record/dictionary subsumption)
* Fix problem (it was error reporting...)
* Rename trait
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
* Correct spelling in comments
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
* Rename trait impl
* Rename trait
* Update comments
* Remove subsumption function from typecheck/mod.rs
* Add test
* Add documentation
* Rename trait (for clippy)
* Add tests
* Modify snapshot lsp
* Update comments core/src/typecheck/subtyping.rs
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
* Update doc/manual/typing.md
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
* Modify comment
* Modify comment
* Revert lsp snapshot
* Cosmetic improvements
This commit performs minor cosmetic improvements in the code handling
subtyping, mostly around comments and function interfaces.
* Update snapshot test for LSP
The support of subtyping for record types makes some types to be
instantiated earlier than before. Given the way terms are visited, when
writing something like `let f = std.array.map in ..`, `std.array.map`
used to be inferred to be of a polymorphic type `forall a b. ...` and
then only was instantiated (because record access is inferred, while the
bound expression is checked). With the new implementation, it is
instantiated as part of the subsumption rule, meaning that it'll appear
differently on the LSP.
All in all, both version of the data shown by the LSP (before this
change and after) are meaningful, in some sense. The polymorphic type is
still shown when it's part of the metadata anyway, in addition to the
inferred monomorphic type. This also doesn't change which expressions
are accepted or not. It's more of an artifact of when we visit terms,
before or after instantiation and how those visits are intertwined with
`check` and `infer`. It's not easy to revert to the previous state of
affairs, and it's in fact not necessarily a good thing either: in the
example above, the LSP would also show a polymorphic type for `f` - the
one of `map` - while in fact `f` has a monomorphic type, so you couldn't
use it in different contexts (say on an `Array Number` and later on a
`Array String`). The current version is showing the real monomorphic
nature of `f`.
---------
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@tweag.io>
* Fix benchmarks on Windows
* Fix tests for Windows
+ Improve failure messages
* Add CI job for Windows
* Disable unused warning for ParseFormatError
* Add `pprof` feature to utils
* Pre-generate contracts
After the introduction of a proper node for types in the AST, we
switched to a lazy way of generating contract: we keep the type as it
is, and convert it to a contract only once it's actually applied. While
simpler, this approach has the drawback of potentially wasting
computations by running the contract generation code many times for the
same contract, as some metrics showed (up to 1.6 thousand times on large
codebases).
This commit add a `contract` field to `Term::Type`, and statically
generates - in the parser - the contract corresponding to a type. This
is what we used to do before the introduction of a type node. Doing so,
we only generate at most one contract per user annotation, regardless of
how many times the contract is applied at runtime.
* Pre-compile match in enum contract
This commit saves some re-compilation of the match expression generated
by contracts converted from enum types by generating the compiled
version directly instead of a match.
* Fix unused import warnings
* More aggressive deduplication on hover.
Deduplicate not just the type annotations, but also the contract
annotations. Also, try to report Dyn less often.
* Add hover metadata for the field of a record.
* Review comments
* Add a comment
* Make the LSP options configurable
Take into account the configuration sent by the client through
[`InitializeParams.initializationOptions`](https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#initializeParams).
This is currently used to set the limits for the background evaluation,
although it also creates the infrastructure for setting more things.
* Fix the doc comments for the config
* Remove a useless explicit Default instance
Replace it by the derived one (thanks clippy)
* Comment fixes
Co-Authored-By: Yann Hamdaoui <yann.hamdaoui@tweag.io>
* Simplify the nls config definition
`#[serde(default)]` applied to a whole struct is enough for it to do the
right thing with the `Default` trait. Makes the module significantly
nicer to read.
---------
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@tweag.io>
When the evaluator in the LSP times out or overflows its stack, the
faulty file is blacklisted so that it doesn't get evaluated any more.
This blacklist used to be permanent, which is generally not desirable
since it makes the LSP unusable.
Change that to only blacklist the file for a fixed delay currently
(hardcoded at 30s).
This commits use a feature of the toml crate allowing to embed span
information into the deserialized data (imported from Nickel). This
makes it possible to give nice error messages and point to within a TOML
file when the application of a Nickel contract to an existing TOML
configuration fails, or in case of unmergeable values.
Doing so requires a bit of boilerplate and brings an additional crate in
(although small). It can also impact deserialization performance,
because we need to first deserialize in an intermediate TOML-like
datastructure. For those reasons, spanned deserialization has been gated
by the `spanned-deser` feature, which is enabled by default, but can be
disabled if this isn't useful.
The background evaluator needs to track dependencies separately, because
it doesn't share file ids with the main cache. This updates the
background dependencies when the foreground ones get updated.
* Add a test
* Invalidate cached terms.
In nls, if `main.ncl` imports `dep.ncl` and `dep.ncl` changes, we need
to invalidate some cached data about `main.ncl` and re-check it.
Previously we were simply resetting `main.ncl` to the "parsed" state in
the cache, but this isn't sufficient because our cached term for
`main.ncl` might depend on `dep.ncl`. Specifically, import resolution of
`main.ncl` transforms the term in a way that depends on whether
`dep.ncl` parsed.
This commit does the most inefficient (but easiest to get correct)
thing: throwing out all the parsed data and re-parsing from scratch.
This can be optimized, but it probably isn't too much slower than the
status quo, because we were re-typechecking from scratch anyway.
* Re-do background evaluation for invalidated files.
When `main.ncl` imports `dep.ncl` and `dep.ncl` changes, we invalidate
`main.ncl` and re-do some checks. This commit adds background
invalidation to the list of checks that we re-do.
* Move the recursive invalidation into Cache
* Reinstate/reword comment
Primitive operators haven't got much love, as opposed to user-facing
interfaces like the stdlib, and they have grown organically since the
beginning of the very first Nickel prototype. As a result, the naming is
inconsistent, with several generations, both in the surface syntax of
Nickel and internally in the Rust codebase.
This PR makes a cleaning pass on primitive operators, by:
1. Use full worlds in the style of the current stdlib: `str` ->
`string`, `num` -> `number`, etc.
2. Introduce pseudo-namespaces: instead of `str_foo` and `record_bar`,
we use `/` as a separator for categories. The previous examples
become `string/foo` and `record/bar`. We don't use `.`, to make it
clear that it's not a normal record access, but just a nice way to
dinstinguish the different categories
3. Align old operators on the current naming in the standard library.
Now that Topiary is published and crates.io (and tree-sitter-nickel as
well), we move away from git revision and use proper public versions for
those. Doing so, we can also crap the special casing that was done
during release to disable those dependencies and we'll now be able to
release on crates.io a version with formatting enabled.
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@tweag.io>
Add `--version` support for the nls binary, using the same scheme as for
the main `nickel` binary.
Doing so, we discovered some underlying feature unification issues that
weren't visible before - but were already there -, namely that the
little stunt we pull off for generating versions in different
environment (in the git repo, building for crates.io and the nixified
version) requires the `string` feature of clap. It just happened that it
was enabled previously for `nickel-lang-cli` by chance by feature
unification (through `comrak`), but in fact building `nickel-lang-cli`
without the default features was failing.
This commit fixes the compilation issue by adding the missing `string`
feature to the `clap` dependency for both the cli and the lsp.
`nickel-lang-lsp` also depends on `nickel-lang-core` without the default
features now (as most of them are useless for the LSP), and we fix the
another compilation error of `nickel-lang-cli` without default features
by making `nickel-lang-core` always export `eval_record_spine`, which
would only be included when the `doc` feature was enabled before, but is
actually used for other purposes now (namely the CLI customize mode).
* Tell lsp about variable bindings in match statements
* Also handle unapplied matches
* insta review
* Add a test getting match branch completions from type inference
This commit introduces or-patterns, which allows to express alternatives
within patterns, including in a deep subpattern.
The compilation of or-patterns is rather simple: we simply try each
alternative until one matches, and use the corresponding bindings.
Typechecking of or-patterns can be done following the same process as
for typechecking a whole match expression (which is also a disjunction
of patterns), although the treatment of bound variables is a bit
different.
Most of the complexity of this commit comes from the fact that we don't
want to make `or` a reserved language keyword, which would break
backward compatibility. This is possible, because `or` in pattern can't
be confused with an identifier, but it requires some tweaking to make
our LALR(1) parser accept this.
This commit adds a new calss of patterns: array patterns. This is a
natural extension, both syntactically and semantically, of existing data
structure patterns in Nickel (in particular of records). Similarly,
arrays pattern can also capture the rest of the pattern (the tail of the
array that hasn't been matched yet) and bind it to a variable.
* Implement pattern guards
This commit implement pattern guards, which are side conditions that can
be added to the branch of a match expression, following the pattern, to
further constrain the matching. This condition is introduced by the
(already existing) `if` keyword. For example:
`match { {list} if list != [] => body }` will match a record with a
unique field `list` that isn't empty.
The compilation is rather straightforward, as a pattern is already
compiled to a tree of if-then-else while also building the bindings
introduced by pattern variables. After all the conditions coming from
the pattern have been tested, we just additionally check for the guard
(injecting the bindings in the condition since the guard can - and most
often does - use variables bound by the pattern).
* Update core/src/term/mod.rs
Co-authored-by: jneem <joeneeman@gmail.com>
* Exclude guarded patterns from tag-only optimization
---------
Co-authored-by: jneem <joeneeman@gmail.com>
* Uniformize destruct and pattern matching logic
When patterns were first introduced, only let-destructuring existed, and
match expressions weren't a thing. Rather than to manually generate code
that would check that a pattern does match the destructured value, we
piggy-backed on contracts instead, which was easier and provided better
error messages. Up to now, this was still how destructuring worked: a
contract is elaborated from the pattern and applied to the matched
value, and should report any error if the value doesn't match. Then,
destructuring is desugared to a series of let-bindings that assume that
the destucture value has the right shape. The generated code is thus
quite simple, as it doesn't have any error handling to do.
When pattern matching landed, we kept the old destructuring
infrastructure, because it was there and worked well. However, there is
duplication: the compilation of match expressions does a work quite
similar to destructuring.
What's more, we're at risk of having a diverging semantics. Some
questions aren't trivial to answer for destructuring, in particular with
respect to lazyness: should `let _ = x in null` force `x`? What about
`let 'Foo _ = x in null`, or `let 'Foo 5 = x in null`?
After some discussion, we decided to give destructuring a simple
semantics in term of pattern matching: `let <pat> = <matched> in <body>`
should be strictly equivalent to `<matched> |> match { <pat> => <body>
}`. Indeed, `match` has a clear semantics around forcing values in lazy
languages. This also agrees with intuition, or at least gives a
reasonable semantics for most cases.
This commit implements this change in practice, by getting rid of the
ad-hoc desugaring of destructuring and rewriting let-destructuring to
actual pattern matching instead.
* Update core/src/term/pattern/compile.rs
Co-authored-by: jneem <joeneeman@gmail.com>
---------
Co-authored-by: jneem <joeneeman@gmail.com>
* Add wildcard patterns
This commit adds a special pattern that doesn't bind anything and match
any value, the wildcard pattern `_`. This pattern doesn't force the
value being matched on.
* Update core/src/typecheck/pattern.rs
Co-authored-by: jneem <joeneeman@gmail.com>
* Fix mishandling of wildcard in enum pattern match compilation
---------
Co-authored-by: jneem <joeneeman@gmail.com>
Most of the time, hovering over a symbol in untyped code would show
`Dyn` even though the symbol has a type annotation - in particular for
records accessed via a non-trivial path, e.g. `foo.bar.baz`. This is an
issue because in particular, we don't get to see the right types for
stdlib symbols when hovering over them.
This commit fixes the issue by taking the type annotation instead of the
type provided by the typechecker when the latter is `Dyn`.
When a function is annotated with a function contract, use this
additional type information to enable completion in the LSP (and also
use the domain as the type of the function's argument in the typing
environment, even if we are in walk mode). Contract data are easy to
fetch during typechecking when available and use them improve the
developer experience by bringing in more static information.
* Run a separate process for each background evaluation job
* Send over only the dependencies, not everything
* Remove ipc-channel
* Kill the child process if it times out
* Add some comments
* [release.sh] update to 1.5.0
* Fix LSP not compiling when format feat is disabled
* Add 1.5 release notes
* Don't mention current release in the manual intro
We used to mention the current release version in the manual intro, but
this is tedious to keep in sync, and not very valuable - this
information is really easy to get by many other mean. The original
intent was to insist that Nickel was past 1.0, and thus was considered
relatively stable and usable in production. The text has been reworded
to say that instead.
* Initial version of background eval
Doesn't support termination, which is probably not good enough
* Kill misbehaving workers
* Clean up the diagnostics
* Improve diagnostics and factor out World from Server
* Fix mistaken ranges for cross-file diagnostics
* Stray dbg
* Fix panic
* Fix rebase issues
* Remove some spurious diagnostics
* Fix link
* Workspace dependencies
* Code review comments
* Add a test
* Issue multiple diagnostics, and ignore partial configurations
* Move eval_permissive to VirtualMachine
* Fix semantics of default values in patterns
The previous semantics of default value in patterns (set with the `?`
annotation) were kinda accidental: at the time they were introduced, it
was much easier to implement default values by the `default` merge
priority. However, it's not aligned with intuition when using default
value in match statement or function argument patterns: the goal here is
mostly to say, if the field isn't defined, I want to use fallback value.
The previous implementation, for example, could fail with merge conflict
or happily merge two default values. In fact, we want to give the
priority to any value that exists in the matched value, whatever is its
actual merge priority.
This commit implements the desired semantics: pattern default values are
only set if the field isn't there, or is there but has no definition. In
that case, we rely on merging to avoid erasing any metadata present
previously (for example, if we set a default value in `{foo | String}`,
we don't want to drop the `String` contract, so we can't use
std.record.update blindly).
* Add From<_> instances to ease Field creation
* Extend match expressions to full patterns
This commit brings together many previous chunks of work to finally make
match expressions accept generic patterns, and not just enum tags.
Pattern compilation was already implemented. Besides changing the
parsing rule for "match" and change the implementation to compile the
pattern and applies the result to the argument, the bulk of the work has
been to correctly typecheck arbitrary structural patterns.
Refer to the comments for the "Match" case in the typechecker for more
details about the algorithm.
* Fix failing test, reword code documentation
* Fix undue unbound identifier when typechecking match
* Add tests for full pattern matching
* Fix null error in pattern compilation
The compilation of record pattern was missing a null check which would
trigger a dynamic type error when some matches aren't exhaustive. This
commit fixes the issue by adding the missing null check.
* Fix clippy warnings
* Limited rewording of some code comments
* Post-rebase fixup of snapshot tests
* Apply suggestions from code review
Co-authored-by: Viktor Kleen <viktor.kleen@tweag.io>
---------
Co-authored-by: Viktor Kleen <viktor.kleen@tweag.io>
* Support enum tags in patterns
Until now, only enum variants were supported within patterns (applied
enums). This commit adds support for bare enum tags as well.
* Improve enum variants handling in stdlib
Make `typeof` returns `'Enum` for enum variants as well, and update
`std.contract.Equal` to properly handle enum tags and enum variants.
* Add tests for enum tag patterns
* Update manual's sample to make tests pass
* Add unwrap_enum_tag primop
Add a primitive operation to extract the argument from an enum variant,
which will prove useful to enum matching and destructuring.
* Add get_tag and is_variant enum primops
* ADTs destructuring
This commit adds support for enum variant patterns in destructuring
forms (let bindings and function declarations). This paves the way for
matching on ADTs once we allow patterns not only appear in a
destructuring binding, but also in a match expression.
* Make get_tag work on simple enum tags as well
* Fix pretty printing of enum_unwrap_variant
* Additional tests for ADTs destructuring
* Fix unification order
A test revelead that the typechecking errors were off sometimes for
destructuring lets: if the destructured value didn't correspond to the
pattern, the type elaborated from the pattern appeared as the inferred
type of the expression, and the inferred type of the expression as the
expected type (i.e. the two were reversed).
This causes incorrect error messages like "extra row" instead of
"missing row". This commits fix the issue by unifying the type deduced
from the pattern and the type inferred from the bound expression in the
right order (unification is symmetric, as far as typechecking is
concerned, but for error reporting we do distinguish between expected
and inferred type).
* Fix type of is_variant to not break parametricity
* Remove useless TODO: tests will catch this
* Format stdlib internals
* Refactoring of pattern destructuring
This commit is a preliminary work for the upcoming ADTs (and more
generally the introduction of pattern matching, instead of just having
destructuring).
The refactoring aims at installing a consistent naming, simplify the
representation of patterns and their associated method, and pave the way
for other patterns that just records. Indeed, the previous
implementation often made the implicit assumptions that patterns were
only record patterns.
* Avoid illegal state for record patterns
Record pattern was previously using a tuple `(open: bool, rest:
Option<LocIdent>)` to represent the presence of either no tail, an
non-capturing tail `..`, or a capturing tail `..rest`. This
representation allows the illegal state `(false, Some("x"))`: indeed, if
the tail is capturing, the record contract is necessarily open.
This commit flattens this representation to a single enum that correctly
represents those three different cases, instead of the 4 allowed by the
previous representation.
* Move the alias field to Pattern
The AST of patterns had a special node for an aliased pattern, which was
a variant containing the alias and a potentital nested pattern. However,
this doesn't model correctly patterns: usually, it doesn't make sense to
stack aliases (and the parser won't accept it), but the previous
representation accepted ASTs for things like `x @ y @ z @ <pat>`, which
incurs additional burden to handle, although it can actually never
happen.
Additionally, the alias of the top pattern was duplicated as an optional
field in the `LetPattern` and `FunPattern` nodes of the `Term` AST.
This commit makes things simpler by storing `alias` as an optional field
directly in the `Pattern` struct, which makes it accessible without
having to pattern match on an enum variant, and forbids nested aliases.
Doing so, we remove the duplication from the `LetPattern` and
`FunPattern`, which now only takes a pattern instead of an optional
identifier and a pattern, leading to code simplification.
* Restore old behavior in typed patterns
The refactoring of patterns has introduced a slightly different
algorithm for typechecking patterns, which isn't entirely
backward-compatible, although it's more consistent. We'll probably rule
out (i.e. depreacte) the offending special cases, but until then, this
commit restores the previous behavior, which fixes a previously failing
test.
* Various renamings in destructuring (now pattern)
This commit only applies pure renaming of several symbols of the
destructuring module for improved clarity. The whole module is also
moved to `term::pattern`, as patterns are just syntactic component of
the term AST.
* Apply suggestions from code review
Co-authored-by: Viktor Kleen <viktor.kleen@tweag.io>
* Post-rebase fixup
---------
Co-authored-by: Viktor Kleen <viktor.kleen@tweag.io>
