Update the release script by adding some missing `git add`, which were
lost when switching to using Topiary from crates.io.
Doing so, we also remove the `--dry-run` step before publishing: the
usual workflow is to perform the dry run test, and if it succeeds,
immediately do the actual publication. This isn't very useful: if the
dry run fails, we could have run the actual publication as well, which
would have fail in the same way, so this makes no difference. However,
if the dry run succeeds, we recompile each package one more time from
scratch, performing the exact same work for publication, which is
wasteful. At this point of the release script, we clearly intend to
release and we should just use `cargo publish` normally.
* Add from/to_tag_and_arg
This commit adds two functions to convert enums to and back from
records, as a tag and an optional argument. Such functions are useful to
handle enums in a general, dynamic way, while pattern matching requires
to know in advance the possible tags.
Additionally, we also implement a `map` function, which can be derived
from the conversions above.
* Update core/stdlib/std.ncl
Co-authored-by: jneem <joeneeman@gmail.com>
* Update core/stdlib/std.ncl
Co-authored-by: jneem <joeneeman@gmail.com>
* Update core/stdlib/std.ncl
Co-authored-by: jneem <joeneeman@gmail.com>
---------
Co-authored-by: jneem <joeneeman@gmail.com>
The background evaluator needs to track dependencies separately, because
it doesn't share file ids with the main cache. This updates the
background dependencies when the foreground ones get updated.
* Add a test
* Invalidate cached terms.
In nls, if `main.ncl` imports `dep.ncl` and `dep.ncl` changes, we need
to invalidate some cached data about `main.ncl` and re-check it.
Previously we were simply resetting `main.ncl` to the "parsed" state in
the cache, but this isn't sufficient because our cached term for
`main.ncl` might depend on `dep.ncl`. Specifically, import resolution of
`main.ncl` transforms the term in a way that depends on whether
`dep.ncl` parsed.
This commit does the most inefficient (but easiest to get correct)
thing: throwing out all the parsed data and re-parsing from scratch.
This can be optimized, but it probably isn't too much slower than the
status quo, because we were re-typechecking from scratch anyway.
* Re-do background evaluation for invalidated files.
When `main.ncl` imports `dep.ncl` and `dep.ncl` changes, we invalidate
`main.ncl` and re-do some checks. This commit adds background
invalidation to the list of checks that we re-do.
* Move the recursive invalidation into Cache
* Reinstate/reword comment
Perhaps surprisingly, `std.record.get_or "a" true {a}` would error out
instead of returning `true`. This commit fixes it by only fetching the
value from the record if the field is present AND it is defined.
Primitive operators haven't got much love, as opposed to user-facing
interfaces like the stdlib, and they have grown organically since the
beginning of the very first Nickel prototype. As a result, the naming is
inconsistent, with several generations, both in the surface syntax of
Nickel and internally in the Rust codebase.
This PR makes a cleaning pass on primitive operators, by:
1. Use full worlds in the style of the current stdlib: `str` ->
`string`, `num` -> `number`, etc.
2. Introduce pseudo-namespaces: instead of `str_foo` and `record_bar`,
we use `/` as a separator for categories. The previous examples
become `string/foo` and `record/bar`. We don't use `.`, to make it
clear that it's not a normal record access, but just a nice way to
dinstinguish the different categories
3. Align old operators on the current naming in the standard library.
This commit update the manual to take into account the latest extensions
of pattern matching, namely wildcard patterns, constant patterns, array
patterns, pattern guards and or-patterns.
Doing so, we also update the examples (in the manual and in the
`examples` directory) to use pattern matching whenever it looks more
idiomatic and make the code more readable.
Co-authored-by: jneem <joeneeman@gmail.com>
Now that Topiary is published and crates.io (and tree-sitter-nickel as
well), we move away from git revision and use proper public versions for
those. Doing so, we can also crap the special casing that was done
during release to disable those dependencies and we'll now be able to
release on crates.io a version with formatting enabled.
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@tweag.io>
Add `--version` support for the nls binary, using the same scheme as for
the main `nickel` binary.
Doing so, we discovered some underlying feature unification issues that
weren't visible before - but were already there -, namely that the
little stunt we pull off for generating versions in different
environment (in the git repo, building for crates.io and the nixified
version) requires the `string` feature of clap. It just happened that it
was enabled previously for `nickel-lang-cli` by chance by feature
unification (through `comrak`), but in fact building `nickel-lang-cli`
without the default features was failing.
This commit fixes the compilation issue by adding the missing `string`
feature to the `clap` dependency for both the cli and the lsp.
`nickel-lang-lsp` also depends on `nickel-lang-core` without the default
features now (as most of them are useless for the LSP), and we fix the
another compilation error of `nickel-lang-cli` without default features
by making `nickel-lang-core` always export `eval_record_spine`, which
would only be included when the `doc` feature was enabled before, but is
actually used for other purposes now (namely the CLI customize mode).
* Add stdlib function array.at_or
This is to return value at given position of array or default in case
provided position is out of bound.
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
* Update manual snippets and snapshot tests
---------
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
* Tell lsp about variable bindings in match statements
* Also handle unapplied matches
* insta review
* Add a test getting match branch completions from type inference
This commit introduces or-patterns, which allows to express alternatives
within patterns, including in a deep subpattern.
The compilation of or-patterns is rather simple: we simply try each
alternative until one matches, and use the corresponding bindings.
Typechecking of or-patterns can be done following the same process as
for typechecking a whole match expression (which is also a disjunction
of patterns), although the treatment of bound variables is a bit
different.
Most of the complexity of this commit comes from the fact that we don't
want to make `or` a reserved language keyword, which would break
backward compatibility. This is possible, because `or` in pattern can't
be confused with an identifier, but it requires some tweaking to make
our LALR(1) parser accept this.
Move the Assert and check function from the local test library to the
stdlib, under the name of `std.test.Assert` and `std.test.assert_all`.
Co-authored-by: Yann Hamdaoui <yann.hamdaoui@gmail.com>
This commit adds a new calss of patterns: array patterns. This is a
natural extension, both syntactically and semantically, of existing data
structure patterns in Nickel (in particular of records). Similarly,
arrays pattern can also capture the rest of the pattern (the tail of the
array that hasn't been matched yet) and bind it to a variable.
* Implement pattern guards
This commit implement pattern guards, which are side conditions that can
be added to the branch of a match expression, following the pattern, to
further constrain the matching. This condition is introduced by the
(already existing) `if` keyword. For example:
`match { {list} if list != [] => body }` will match a record with a
unique field `list` that isn't empty.
The compilation is rather straightforward, as a pattern is already
compiled to a tree of if-then-else while also building the bindings
introduced by pattern variables. After all the conditions coming from
the pattern have been tested, we just additionally check for the guard
(injecting the bindings in the condition since the guard can - and most
often does - use variables bound by the pattern).
* Update core/src/term/mod.rs
Co-authored-by: jneem <joeneeman@gmail.com>
* Exclude guarded patterns from tag-only optimization
---------
Co-authored-by: jneem <joeneeman@gmail.com>
* Uniformize destruct and pattern matching logic
When patterns were first introduced, only let-destructuring existed, and
match expressions weren't a thing. Rather than to manually generate code
that would check that a pattern does match the destructured value, we
piggy-backed on contracts instead, which was easier and provided better
error messages. Up to now, this was still how destructuring worked: a
contract is elaborated from the pattern and applied to the matched
value, and should report any error if the value doesn't match. Then,
destructuring is desugared to a series of let-bindings that assume that
the destucture value has the right shape. The generated code is thus
quite simple, as it doesn't have any error handling to do.
When pattern matching landed, we kept the old destructuring
infrastructure, because it was there and worked well. However, there is
duplication: the compilation of match expressions does a work quite
similar to destructuring.
What's more, we're at risk of having a diverging semantics. Some
questions aren't trivial to answer for destructuring, in particular with
respect to lazyness: should `let _ = x in null` force `x`? What about
`let 'Foo _ = x in null`, or `let 'Foo 5 = x in null`?
After some discussion, we decided to give destructuring a simple
semantics in term of pattern matching: `let <pat> = <matched> in <body>`
should be strictly equivalent to `<matched> |> match { <pat> => <body>
}`. Indeed, `match` has a clear semantics around forcing values in lazy
languages. This also agrees with intuition, or at least gives a
reasonable semantics for most cases.
This commit implements this change in practice, by getting rid of the
ad-hoc desugaring of destructuring and rewriting let-destructuring to
actual pattern matching instead.
* Update core/src/term/pattern/compile.rs
Co-authored-by: jneem <joeneeman@gmail.com>
---------
Co-authored-by: jneem <joeneeman@gmail.com>
* Add new_from_sources function to Program
This is similar to the new_from_path and new_from_paths function. A
new_from_source function already existed, but a new_from_sources was
still missing.
* Abstract the common parts of the Program::new_from functions
* Explain the `File` type parameter to `Input`
* Add wildcard patterns
This commit adds a special pattern that doesn't bind anything and match
any value, the wildcard pattern `_`. This pattern doesn't force the
value being matched on.
* Update core/src/typecheck/pattern.rs
Co-authored-by: jneem <joeneeman@gmail.com>
* Fix mishandling of wildcard in enum pattern match compilation
---------
Co-authored-by: jneem <joeneeman@gmail.com>
When failing to serialize a value, the error sometimes doesn't point to
a usable location (that is, it points to the definition site of some
function somewhere, but it's impossible to know where this function was
actually used, which is usually where the issue is).
This commit enriches the error message of non-serializable errors with a
path, à la JSON pointer (such that it also integrates array indices),
whenever possible. This should make debugging non serializable error
much nicer.
For some reason, the version of Topiary and of the Nickel tree sitter
grammar were lagging behind, before the support for pattern matching was
added, which means current master wasn't able to format a file with
pattersn in it - maybe it was a rollback because of the Topiary install
issue (https://github.com/tweag/topiary/issues/690).
The former issue has been fixed, so this commit update to latest Topiary
in order to get formatting of patterns.
Most of the time, hovering over a symbol in untyped code would show
`Dyn` even though the symbol has a type annotation - in particular for
records accessed via a non-trivial path, e.g. `foo.bar.baz`. This is an
issue because in particular, we don't get to see the right types for
stdlib symbols when hovering over them.
This commit fixes the issue by taking the type annotation instead of the
type provided by the typechecker when the latter is `Dyn`.
When a function is annotated with a function contract, use this
additional type information to enable completion in the LSP (and also
use the domain as the type of the function's argument in the typing
environment, even if we are in walk mode). Contract data are easy to
fetch during typechecking when available and use them improve the
developer experience by bringing in more static information.
Identifiers were required to start with at most one optional underscore,
followed by an alphabetic character (to avoid identifiers like _-'), and
then by any identifier character. This means that identifiers like
`__foo` were invalid.
There is no good reason for that, and is mostly an oversight when
writing the initial regular expression. This commit extends the lexer to
allow zero or more heading underscores before the first alphabetic
character.
A recent commit fixed the `nickel doc` markdown output having spurious
backslashes introduced. However, it also re-introduced a previous issue
- which was why the problematic hard line breaks were introduced in the
first place - which is that the list of types and contracts of a field
and the custom documentation might be squashed together because the
renderer wouldn't properly insert a newline after the generated list.
After some investigation, it seems to be a bug in the comrak renderer.
However, this bug doesn't show when the markdown is parsed from an
actual source (instead of being programmatically generated). It turns
out comrak correctly inserting this required newline relies on the
unspoken assumption that every markdown list item content is wrapped as
a paragraph. This commit thus fixes the issue by inserting the missing
paragraph wrapper.
The code generating markdown documentation from a Nickel configuration
used to unconditionally insert hard line breaks between a field's
signature and the rest of its documentation. However, one can't insert a
hard line break immediately followed by a blocks in commonmark (or put
differently: it's not representable as a source), and those line breaks
were just rendered as spurious backslashes at best, or sometimes
interpreted differently (see #1706 for more details).
A bit of experimentation showed that what comes after this line break is
always a block in the sense of commonmark (even when it's simple
documentation text, it's wrapped as a Markdown paragraph). Thus this
commit just removes the line break altogether, which is not useful and
never legal anyway.
* Add fields_all primop (doesn't ignore empty opts)
* Add empty-opt aware record op variants to stdlib
The design guideline around empty optional fields has been to make most
normal record operation ignore them, to avoid issues like trying to list
the field of a record and access them just to fail on an empty optional
introduced by a contract, which we shouldn't care about.
However, in some cases (typically operating on record contracts), users
might want to actually take empty optional fields into account. Most
primops were already existing in two variants internally (ignore empty
opts and consider all fields). This commit just makes them available as
clearly documented stdlib functions.
* Add `has_field_all`, change `_all` -> `_with_opts`
The previous commit introducing variants of record operators that don't
ignore empty optional fields missed `has_field`, whose variant is added
by this commit.
As decided by the team in the weekly meeting, the `_all` suffix
introduced in the previous commit has been changed to `_with_opts`,
which is a bit longer but less confusing and more explicit.
Using a polymorphic type annotation on a record field, as in
`{id : forall a. a -> a = std.function.id}` would unexpectedly file with
a failure to unify `forall a. a -> a` with `a -> a`, which indicates
that the type annotation wasn't properly instantiated before being used
to check the value of the field.
It turns out the typechecking of the field per se is fine, but the issue
comes from the recursive environment: in a recursive record, we need to
build an environment with types for all record fields _before_ we can
start to actually typecheck each field. When the field isn't annotated,
we use a fresh unification variable. When typechecking the corresponding
field, we unify what's in the type environment (the unification
variable, but which might have been unified with another type at this
point) with the type inferred for the field.
However, when the field has a type annotation, we use this annotation
for the recursive environment instead (which is important to keep
polymorphic types...polymorphic). In this case, we would unify a
polymorphic type with the inferred type, which is the place that was
missing the instantiation.
One possibility is to just add the missing instantiation, but in fact
this unification is useless when there is a type annotation: to avoid
this useless work, we rather remember which fields are annotated, and we
entirely skip the unification for them.
* Run a separate process for each background evaluation job
* Send over only the dependencies, not everything
* Remove ipc-channel
* Kill the child process if it times out
* Add some comments
