Fix failure to parse a syntax case involving unexpected usage of annotated expressions (fixes#11691).
The root cause of the bug was usage of the `matches!` macro in a situation where a `match` statement should have been used to enforce exhaustiveness.
- I have corrected the bug, and reviewed all usages of the `matches!` macro. I didn't find any other *incorrect* usages of `matches!`, but I found and replaced some potentially *fragile* uses. I also simplified some unnecessary uses of `matches!`, replacing them with expressions that are more-obviously correct.
- Stricter parsing: It is now a syntax error for an inline annotation to occur in the RHS of an application (e.g. `fn @Tail_Call recur`)--the precedence in this case may be surprising, so parentheses should be used around the annotated expression.
`parse_all_enso_files.sh`: This PR does not affect any AST in the .enso corpus.
Adjust operator parsing to allow chained conversions, like `3.14 : Integer : Text`.
Change the precedence and associativity of the `:` operator, when used as a binary operator in an expression:
- It is now **left-associative**
- It now has **lower** precedence than `->` (previously they were equal)
# Important Notes
One previously-reasonable syntax has **changed interpretation**: `x->x:Type` is no longer a valid way to write a casting function, and would likely result in a type error. There was 1 instance of this syntax in our .enso sources.
Move documentation into documentable types (implements #11302).
# Important Notes
GUI:
- Distinguish expression and statement
- `Ast.Ast` is still present, as the base class for AST objects. Most references to `Ast.Ast` are now references to `Ast.Expression`. Operations on blocks use `Ast.Statement`.
- `Ast.parse` has been replaced with: `Ast.parseExpression`, `Ast.parseStatement`, and `Ast.parseBlock`
- `syncToCode` is internally context-aware; it parses the provided code appropriately depending on whether its AST is an expression, a statement, or the top level of a module.
- Remove `wrappingExpression` / `innerExpression` APIs: Wrapper types have been eliminated; modifier lines are now fields inside parent types.
- Simplify AST printing:
- Fully implemented autospacing in `concreteChildren` implementations; the type returned by `concreteChildren` now ensures that spacing has been fully resolved.
- Eliminate `printBlock` / `printDocs`: `concreteChildren` is now aware of indentation context, and responsible for indentation of its child lines.
- The `Pattern` type is now parameterized to identify the AST type it constructs. The `Pattern.parseExpression` function helps create a `Pattern<Expression>`.
- Refactor `performCollape` for testability.
- e2e tests: Improve table viz test: It still doesn't pass on my Mac, but these changes are necessary if not sufficient.
Compiler (TreeToIr):
- An expression in statement context is now found in an `ExpressionStatement` wrapper.
- Documentation for a `Function` is now found inside the function node.
- Deduplicate some polyglot-function logic.
Move annotations into fields of Function and ConstructorDefinition.
# Important Notes
New syntax: Constructor argument-definition lines
- Each argument in a type-constructor definition may be specified on its own (indented) line.
Relaxed syntax: Unparenthesized arguments to annotations
- A generic annotation now uses the rest of the line as its argument expression; the expression no longer needs to be parenthesized.
Eliminate `private` modifier-node: `private` is a field in supporting types, or a single-token node in the case of `private` declarations.
# Important Notes
- Rust parser tests: Switch to a builder-style API for defining expected `Function` ASTs to allow further changes to `Function` fields without rewriting all the tests again.
- TreeToIr: Fix discarded module-level `diag`; add a test that covers module diagnostics.
- Syntax: Disallow `private` methods in function blocks. (Previously this was enforced in the compiler.)
Align `Assignment`/`Function` distinction in AST with compiler's implemented semantics:
- The ambiguous case `funcOrVar = expression` is now parsed as a `Function` when in a `Type` definition or in the top level of a module. I.e. it is an `Assignment` in contexts where the RHS is evaluated immediately when the binding is evaluated, and a `Function` in contexts where the RHS is evaluated each time the bound name is evaluated.
- `Assignment` statements now may only occur in function bodies.
Correcting this distinction lays the groundwork for #11302.
Other changes:
- Fixed incorrect source code locations for negative literals and negated expressions.
# Important Notes
New APIs:
- The parser now exposes a `parse_block` entry point, which allows parsing input lines as if in the body of a function. The previous entry point has been renamed to `parse_module`.
The fonts were added (by mistake) anyway, but they are rarely changed, and the scripts add unnecessary complexity.
This also fixes some post- #11287 problems.
Stateless (static) parser interface. Buffer-reuse optimization is now hidden within `Parser` implementation. Fixes#11121 and prevents similar bugs.
# Important Notes
- Also simplify `EnsoParser` API, exposing only a higher-level interface.
* More info when critical failure occurs
Log problematic module to help with debugging critical failure.
* One more exception
* s/System.err/Logger.error/
* maybe append slf4j deps
* fix what looks like a long standing typo
`GeneratedFormatTests.java` not `GeneratedFormatTests..java`
* one more typo
* Fix directory where to look for classpath
`./run java-gen test --skip-version-check` now works. At least locally.
* Local is fine, CI is not. More temporary debugging...
* Ensure project's managedClasspath is exported
Running java tests requires us knowing all additional dependencies as
they have to be added to the classpath manually. That can only be
ensured by invoking the right sbt target.
* Move sbt call after graalvm setup
* removing CI debugging
* Apply suggestions from code review
Co-authored-by: Kaz Wesley <kaz@lambdaverse.org>
---------
Co-authored-by: Kaz Wesley <kaz@lambdaverse.org>
- Fix debug logging for #11088 case--attempt to create an exception that is its own cause fails.
- In case the parser is used after closing, throw an `IllegalStateException` instead of UB. (This case is not known to occur and doesn't seem to be behind the #11121, but we should handle it more safely if it does.)
Removed all `#![feature]` flags, except for `#![feature(test)]`. Once parser benchmarks are ported to something that is compatible with stable rust, we will be able to switch to it.
# Important Notes
The command to run the gui dev environment has been changed. Invoking the old command will print a message about that.
From now on, use `pnpm dev:gui2` in repository root.
Implement full `ArgumentDefinition` syntax for new-lambda arguments, e.g `\a=1 (b:Integer = 23)-> a + b`; add backend support for new lambdas.
Emit an error when any syntactic operator is used outside of its associated syntax (fixes#10473).
Phase out complex arguments for old-lambdas: It is now a syntax error to specify default arguments for an old-lambda. This capability had no usage in real code; affected tests have been updated to test new lambdas. For now, old lambdas can continue to be used with simple arguments; if default arguments are desired, a new-style lambda can be used.
Translate syntax warnings and attach to IR when translating operator applications.
We should ensure that all Trees are checked for warnings and every warning is attached to some IR. That would require a bit of refactoring: In TreeToIr, we could define helpers wrapping every IR constructor and accepting a `Tree` parameter. The `Tree` could be used to populate the `IdentifiedLocation` when constructing the IR type, and then to attach all warnings after constructing the IR object.
# Important Notes
- Update JNI dependency.
- Introduces a `cargo bench` runner for parser.
In a sequence of value-level operators, whitespace does not affect relative precedence. Functional operators still follow the space-precedence rules.
The "functional" operators are: `>> << |> |>> <| <<| : .`, application, and any operator containing `<-` or `->`. All other operators are considered value-level operators.
Asymmetric whitespace can still be used to form *operator sections* of value-level operators, e.g. `+2 * 3` is still equivalent to `x -> (x+2) * 3`.
Precedence of application is unchanged, so `f x+y` is still equivalent to `f (x + y)` and `f x+y * z` is still equivalent to `(f (x + y)) * z`.
Any attempt to use spacing to override value-level operator precedence will be caught by the new enso linter. Mixed spacing (for clarity) in value-operator expressions is allowed, as long as it is consistent with the precedences of the operators.
Closes#10366.
# Important Notes
Precedence warnings:
- The parser emits a warning if the whitespace in an expression is inconsistent with its effective precedence.
- A new enso linter can be run with `./run libraries lint`. It parses all `.enso` files in `distribution/lib` and `test`, and reports any errors or warnings. It can also be run on individual files: `cargo run --release --bin check_syntax -- file1 file2...` (the result may be easier to read than the `./run` output).
- The linter is also run as part of `./run lint`, so it is checked in CI.
Additional language change:
- The exponentiation operator (`^`) now has higher precedence than the multiplication class (`*`, `/`, `%`). This change did not affect any current enso files.
Library changes:
- The libraries have been updated. The new warnings were used to identify all affected code; the changes themselves have not been programmatically verified (in many cases their equivalence relies on the commutativity of string concatenation).
Single-phase whitespace-aware precedence resolution.
#### Performance
Since this is a major refactor of the core of the parser, I benchmarked it; it's about 3% faster.
# Important Notes
- Move operator-identifier recognition to lexer.
- Move compound-token assembly out of precedence resolver
Addresses one of two concerns of #5298 - adds support for `--jvm` argument to allow us to switch from _native image_ built Enso binary (as developed by #10126) to regular JVM based Enso execution. This change _doesn't affect production builds_. The _native executable_ continues to be only built by `engine-runner/buildNativeImage` which is tested on CI, but not in the production jobs.
Outline view and completions for Enso code in VSCode.
# Important Notes
This PR provides the necessary infrastructure for building VSCode extension that includes `enso_parser` library compiled for all supported platforms.
VSCode extension can now use libraries from `sbt` that are `publishM2`-ready. To make that possible a documentation must have been provided and fixed for those modules - hence so many changes in `.scala` classes.
<img width="862" alt="image" src="https://github.com/enso-org/enso/assets/26887752/7374bf41-bdc6-4322-b562-85a2e761de2a">
Last, but not least. The outline view and completions display something.
Add support for private methods. Most of the changes are in parser and compiler. The runtime checking of private functions was already present since #9692
# Important Notes
- Only top-level methods can be declared `private`.
- private method cannot be called from different project
- private method cannot be accessed from polyglot code (private method does not exist for polyglot code)
* Hotfix for finding parser library
Since ydoc is now started by language server, parser is initialized
differently and attempts to find `libenso_parser.so` in
`component/runner` rather than `component` directory.
* Add fallbacks
* fix native image build
This PR introduces a new installer and uninstaller for the Windows platform.
Both are written in Rust and compiled to a single executable. The executable has no dependencies (other than what is included in the Windows), links the C++ runtime statically if needed.
The change is motivated by numerous issues with with the `electron-builder`-generated installers. The new installer should behave better, not have issues with long paths and unblock the `electron-builder` upgrade (which will significantly simplify the workflow definitions).
To build an installer, one needs to provide the unpacked application (generated by `electron-builder`) and the `electron-builder` configuration (with a few minor extensions). Code signing is also supported.
part of #7954
# Important Notes
The workflow is:
- `$ npm install` -- just in case
- `$ npm --workspace=enso-gui2 run build-ydoc-server-polyglot` -- build the `ydocServer.js` bundle
- `$ sbt ydoc-server/assembly` -- build the ydoc server jar
- `env POLYGLOT_YDOC_SERVER=true npm --workspace=enso-gui2 run dev` -- run the dev server with the polyglot ydoc server. Providing `POLYGLOT_YDOC_SERVER_DEBUG=true` env variable enables the chrome debugger
Closes#8836.
Atom constructors can be declared as private (project-private). project-private constructors can be called only from the same project. See the encapsulation.md docs for more info.
---------
Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
Co-authored-by: Kaz Wesley <kaz@lambdaverse.org>
* Text literals: Accept unpaired-surrogate escape codes.
Unpaired surrogates are not allowed by Unicode, but they occur in practice
because many systems accept them; for example, they may be present in filenames
on Windows (which are otherwise constrained to UTF-16).
Programs written in Enso should be able to work with them, if only because they
represent edge cases that should be tested when converting encodings and at
other system boundaries.
- Generalize the representation of interpreted-text-escapes in the lexer, so
that we are not tied to the strict Unicode of Rust's `str`.
- Move some doc-comment code from the parser to test utilities.
- Simplify token serialization.
* Reduce parser dependencies
- `enso-parser-syntax-tree-visitor` is now only used when building tests and debug tools.
- Remove `enso-logging` crate and its macros.
- The main bin for `enso-parser` has been moved to a `check_syntax` tool in `enso-parser-debug`.
Removes a bulk of rust crates that we no longer need, but that added significant install, build and testing time to the Rust parser.
Most significantly, removed `enso-web` and `enso-shapely`, and got rid of many no longer necessary `#![feature]`s. Moved two still used proc-macros from shapely to prelude. The last remaining usage of `web-sys` is within the logger (`console.log`), but we may actually want to keep that one.
This PR updates the Rust toolchain to recent nightly.
Most of the changes are related to fixing newly added warnings and adjusting the feature flags. Also the formatter changed its behavior slightly, causing some whitespace changes.
Other points:
* Changed debug level of the `buildscript` profile to `lint-tables-only` — this should improve the build times and space usage somewhat.
* Moved lint configuration to the worksppace `Cargo.toml` definition. Adjusted the formatter appropriately.
* Removed auto-generated IntelliJ run configurations, as they are not useful anymore.
* Added a few trivial stdlib nightly functions that were removed to our codebase.
* Bumped many dependencies but still not all:
* `clap` bump encountered https://github.com/clap-rs/clap/issues/5407 — for now the warnings were silenced by the lint config.
* `octocrab` — our forked diverged to far with the original, needs more refactoring.
* `derivative` — is unmaintained and has no updated version, despite introducing warnings in the generated code. There is no direct replacement.
This PR removes enso-pack (ensogl-pack) crate.
It still keeps the `enso-runner` JS package, as it is used for CLI argument parser and logger. The runner should be probably refactored (and possible removed altogether).
# Important Notes
I've temporarily extracted the `enso-runner` to `lib/js` directory, as I wanted to avoid keeping pure JS library under `lib/rust`. Attempts at integrating this with `app/ide-desktop` and family caused too much trouble for this PR. The expectation is that the package will be removed or moved elsewhere soon anyway.
Removed `enso-types` crate which had only one reference in unused part of the code. Removed some unused dependencies from `Cargo.toml` files.
# Important Notes
CI has a similar hiccup as before. Please disregard this for now in the review.
Fixes#9331
Fixed issues with wrong initial size and missing edit caret in text widgets.
<img width="311" alt="image" src="https://github.com/enso-org/enso/assets/919491/44f257cc-18a1-4a9f-9ae0-c1dd9b86674e">
# Important Notes
Automated tests for font loading/initial size will follow shortly. The text caret is not really testable, since it is a hosted object visual issue.
Removes the old GUI1 code base and reduces the Rust code footprint by removing unused code.
# Important Notes
Updates build scripts and reformats part of the codebase with the autoformatter.
- Improved performance by batching simulatenous node edits, including metadata updates when dragging many selected nodes together.
- Updated Vue to new version, allowing us to use `defineModel`.
- Fixed#9161
- Unified all handling of auto-blur by making `useAutoBlur` cheap to register - all logic goes through a single window event handler.
- Combined all `ResizeObserver`s into one.
- Fixed the behaviour of repeated toast messages. Now only the latest compilation status is visible at any given time, and the errors disappear once compilation passes.
- Actually fixed broken interaction of node and visualization widths. There no longer is a style feedback loop and the visible node backdrop width no longer jumps or randomly fails to update.
* Test describing the current behavior of chained if then else application
* Chained block should behave just like Group around if_then_else
* Finishing line on BlockStart fixes if_then_else_chained_block
* Only finish the line when there was not start of a macro segment
* Fix tests
* Refine else-body with macro patterns.
* Update test syntax to maintain original semantics
* Few additional tests
---------
Co-authored-by: Kaz Wesley <kaz@lambdaverse.org>
Random.Seed doesn't work in the GUI and the TEXT_ONLY tag doesn't do anything so it was incorrectly showing up in the component browser.
This MR makes Random.Seed private to hide it from the GUI and completely removes the TEXT_ONLY tag which is unused and unimplemented.
Implements #6166.
# Important Notes
- More consistent handling of `default` arguments. `default` is a valid identifier, and only has special meaning when it isn't bound in scope. Since distinguishing the builtin `default` from an identifier called `default` cannot be done until alias analysis has been performed, `default` is now represented in the AST as a regular identifier.
- `TreeToIr`: Remove `insideTypeAscription`. It was only used for bug-for-bug compatibility with the old parser during the transition.
Add `line:column` information to source code references produced by the parser. This information will be used by GUI2 as part of the solution to #8134.
# Important Notes
- `parse_all_enso_files.sh` has been used to ensure this doesn't affect tree structures.
- `parse_all_enso_files.sh` now checks emitted locations for consistency, and has been used to verify that all line:col references match the values found by an independent scan of the source up to the given UTF8 position.
- Validate spans during existing lexer and parser unit tests, and in `enso_parser_debug`.
- Fix lost span info causing failures of updated tests.
# Important Notes
- [x] Output of `parse_all_enso_files.sh` is unchanged since before #7881 (modulo libs changes since then).
- When the parser encounters an input with the first line indented, it now creates a sub-block for lines at than indent level, and emits a syntax error (every indented block must have a parent).
- When the parser encounters a number with a base but no digits (e.g. `0x`), it now emits a `Number` with `None` in the digits field rather than a 0-length digits token.
