Motivation:
```
Caused by: java.lang.AssertionError: No polyglot context is entered. A language or context reference must not be used if there is no polyglot context entered.
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotFastThreadLocals.assertValidGet(PolyglotFastThreadLocals.java:481)
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotFastThreadLocals$ContextReferenceImpl.get(PolyglotFastThreadLocals.java:513)
at org.enso.runtime/org.enso.interpreter.runtime.EnsoContext.get(EnsoContext.java:246)
at org.enso.runtime/org.enso.interpreter.runtime.error.PanicException.computeMessage(PanicException.java:90)
```
Looks like `Throwable` thrown by the execution may be missing a message,
which later crashes when we want to send it over the wire.
Added some guards to prevent NPE.
* Add failing tests
* ExtensiomMethod resolution in the current module might be an error in GlobalNames
* Add another successful test
* Fix compilation error test
* Fix compilation of RedShift_Spec - `setup` is provided in compile scope
I am trying to compile Enso with GraalVM 24.2.0 snapshot to verify that https://github.com/oracle/graal/pull/8266 changes would work for Enso. Among the problems I am facing is a dependency on `org.graalvm.collections` because of using `Pair`. I don't think we need such dependency. Let's avoid it by using our own `record` with two components.
Fixes#11022 and in general fixes#5787 by avoiding calls to Scala's `assert` function and using regular one written in Java that uses `assert` keyword and is thus sensitive to `-ea` command line option of the JVM. Use `assertInJvm` in newly written Scala code to get typical JVM behavior on `assert`.
close#10970
Changelog:
- remove: FileManager actors pool
Language Server executes IO on a separate thread pool for blocking operations and does not block the actor
Loading IR caches and IdMap for a module don't play very well. Rather
than complicating logic and introducing yet another re-compilation
request let's not even attempt to deserialize a module if IdMap is
present. The cost of compiling a single module (right now containing the
project itself) is negligble, compared to a possibility of bugs that
we've encountered so far.
* Rename packages in logging-utils-akka
* Migrate buildInfo to Java
* Rename packages in logging
* Rename package in scala-yaml
* No usage of CompilerDirectives inside pkg
* log errors of initialization of directory watcher
* HashCodeNode does not use com.google.common.base.Objects
* Rename rest of the packages
* fmt
* Fix dependencies on version-output
* Add necessary dependencies to testkit
* Rename instruments in runtime-fat-jar module-info
* Fix compilation errors because of BuildVersion
* Fix logger renames
* Use java.util.List directly
* Fixes after merge
* Improve error message in NativeLauncherSpec
* Fix logger renames
* Fix json version formatting
* Revert "No usage of CompilerDirectives inside pkg"
This reverts commit cc7e078416.
* fmt
* Reduce akka logs in PM and LS
After #10905 we suddenly got a lot of (useless) logs from akka in
project-manager and language-server. After some investigation it turned
out that akka-typed had a special configuration to enable logging in
akka using slf4j.
As we don't need the whole library for that I only stole the
configuration.
* nit
* licensing
* missing license files
* Re-enable IR caching in Language Server
When an IR loaded from cache is being run through the same passes,
"interesting" errors may happen. We must ensure that IR is not run
through phases that have already done their transformations.
With this change, I'm no longer seeing failures after project startup.
* resetScope is also obsolete
* address PR review
* Akka-related startup improvements
Realized that Akka Typed and Spray were not used **at all** yet there
were in the list of dependencies.
Additionally, the former would also show up in the list of Akka's
library-extensions, initialized on startup.
* Hardcode list of library extensions
Hardcoding a list of library extensions is not recommended but I don't
see any other way of disabling expensive Serialization extension, which
we don't seem to use anyway.
* fmt
* licensing update
Scheduling of jobs is asynchronous and nothing prevents it from scheduling execution before compilation leading to stale nodes in GUI. The scenario is easily reproducible by adding a `Thread.sleep` before `EnsureCompiledJob` requests compilation locks.
This change ensures that execution is only scheduled after compilation is finished, when an edit request is being processed.
# Important Notes
This fix was prompted by me getting random stale nodes after edits:
(it would never recover unless another edit was made)
When Ydoc requests to open file and encounters a timeout, it will
attempt to retry the request. At least that's the behaviour since
#10787.
Unfortunately, the write lock on the file might be already held and
identified by the client id and actor reference. The latter is always
determined by the request and therefore spurious. The same client should
always be given the same capability, even on retries.
* Reduce amount of IR copying
This PR ensures that we only perform IR copying when actually needed,
i.e. when one of the fields has changed.
Most of the changes should not be controversial except for equality
change in MetadataStorage. Without it, an empty pass metadata would be
treated as equal, therefore preventing duplication from happening.
* fmt
* missed a few spots
* Optimizations to copying
Avoid expensive computations for static methods.
* Avoid unnecessary unapply calls
Fixes#9749 by:
- [x] Adding `fn` option to `enso-debug-server` instrument - eb3b76e
- [x] Print warnings (if any) to stderr - 4fda04b
- [x] Improving output of `:list` to print out warnings - dbe3c45480
- [x] Print errors to stderr - 1312546
- [x] Exiting on `DataflowError` - 2cc7ef5 and e6fbf73
- [x] Using all of that inside of `runner/*Main` - 7df58ef
The core of the change is in instrumentation that wraps the `main` method and at its end checks for _warnings or errors_ among local variables. When an error is found, it wraps the original return value of `main` with a proxy that delegates to the original value, but also pretends to be _exit exception_ with exit code 173. That one is detected in `Main` launcher to exit the process with exit code 173.
# Important Notes
As a side-effect of this change, one can request an invocation of REPL at the end of any method just by providing a property to the VM:
```bash
$ enso --vm.D=polyglot.enso-debug-server.method-break-point=err_test.main --run err_test.enso --repl
```
stops at the end of `main` method of `err_test.enso` file.
* Add stub of FramePointerAnalysis pass
* Add stub of FramePointerAnalysisTest
* Add more framepointer tests
* FramePointer has constructor for better debugging
* Update tests - FramePointer metadata is also in Info.Def
* FramePointerAnalysis traverses the whole IR.
* Fix test - check for metadata
* Reorder methods in FramePointerAnalysis
* Fix argument processing
* Improve some toString methods for debugging
* Fix parent levels in tests
* FramePointerAnalysis does not copy IRs
* FramePointerAnalysis metadata is Persistable
* FramePointerAnalysis attaches metadata to atom constructors
* Synthetic self argument is handled specially
* IRDumperPass is run as the last compiler pass
This ensure that one can see all the metadata on the IR.
* IRDumper can dump alias analysis metadata
* FramePointerMeta has meaningful toString
* FramePointerAnalysis process case expressions
* Fix javadoc links
* FramePointer is attached to GenericAnnotation expression
* IRDumper handles case type patterns
* FramePointer is attached to argument default value expressions
* Add test for default argument value metadata
* Attach frame pointer to default value expressions even if they have no alias graph
* FramePointerAnalysis can run inline on expression
* Fixes after rebase
* Fix annotation processing
* Add tests for no frame pointers on global symbol usages
* Do not use AliasAnalysis in IrToTruffle.
Also refactor NameResolutionAlgorithm to not use AliasMetadata
* Remove unused methods from LocalScope
* Remove unused import
* fmt
* Fix scala.MatchError for case patterns
* Do not use absolute code locations in the test.
Test on Windows fails because of different length of new lines.
- Enables the `..` autoscoping style for creating Atoms in expressions.
- Add type checking to methods in columns.
- Auto wrap returns from method in expressions into a column as needed.
- Remove `Time_Period.Day` to remove confusion..
In order for widgets not to invalidate expression's results and trigger computations, we now cache self argument to which visualizations should be attached to.
It should help with #10730 but there is still a bug in GUI.
- Closes#10716 by un-marking the `Snowflake_Connection` as `private`.
- Does the same to `SQLServer_Connection`.
- Makes sure that `Snowflake_Connection` has correct visualization and enables links for it.
- Adds a fallback in `to_default_visualization_data` and in Table's `prepare_visualization` so that if `to_js_object` of a given type is malformed, we still get some kind of visualization + debug info about the underlying error.
Flatten hierarchy of alias analysis metadata (`or.genso.compiler.pass.alias.Info`) so that it is easily accessible from Java and rename it.
# Important Notes
- Rename `org.enso.compiler.alias.Graph.Occurence` to `org.enso.compiler.alias.graph.GraphOccurence` and flatten the hierarchy.
- Rename `org.enso.compiler.pass.alias.Info` to `org.enso.compiler.pass.alias.AliasMetadata` and flatten the hierarchy.
Majority of warnings handling is now done via newly introduced nodes. Moreover, the underlying representation of warnings storage in `WithWarnings` was changed from `Warning[]` to `EnsoHashMap`.
# Important Notes
- Remove `ArrayRope`.
Implement full `ArgumentDefinition` syntax for new-lambda arguments, e.g `\a=1 (b:Integer = 23)-> a + b`; add backend support for new lambdas.
Emit an error when any syntactic operator is used outside of its associated syntax (fixes#10473).
Phase out complex arguments for old-lambdas: It is now a syntax error to specify default arguments for an old-lambda. This capability had no usage in real code; affected tests have been updated to test new lambdas. For now, old lambdas can continue to be used with simple arguments; if default arguments are desired, a new-style lambda can be used.
Working on compiler IR is a daunting task. I have therefore added a new system property `enso.compiler.dumpIr` that will help with that. It dumps the encountered IRs to `ir-dumps` directory in the [GraphViz](www.graphviz.org) format. More info in updated docs.
Note that all the functionality to dump IRs to `dot` files was already implemented. This PR just adds the command line option and updates docs.
# Important Notes
- `--dump-graphs` cmd line option is removed as per [Jaroslav's request](https://github.com/enso-org/enso/pull/10740#pullrequestreview-2216676140).
- To dump graphs, use `-Dgraal.Dump=Truffle:2` system property passed via `JAVA_OPTS` env var.
If you run `env JAVA_OPTS='-Denso.compiler.dumpIr=true' enso --run tmp.enso` where `tmp.enso` is, e.g.:
```
from Standard.Base import all
main = 42
```
You will then have something like:
```
$ ls ir-dumps
Standard.Base.Data.Filter_Condition.dot Standard.Base.Data.Time.dot Standard.Base.System.Advanced.dot Standard.Base.Warning.dot
Standard.Base.Data.Locale.dot Standard.Base.Enso_Cloud.Enso_File.dot Standard.Base.System.File.Advanced.dot tmp.dot
Standard.Base.Data.Numeric.dot Standard.Base.Errors.dot Standard.Base.System.File.dot
Standard.Base.Data.Numeric.Internal.dot Standard.Base.Network.HTTP.Internal.dot Standard.Base.System.File.Generic.dot
Standard.Base.Data.Text.Regex.Internal.dot Standard.Base.Runtime.dot Standard.Base.System.Internal.dot
```
You can then visualize any of these with `dot -Tsvg -O ir-dumps/tmp.dot`.
An example how that could look like is
- Close#10622
- Changes `project-manager` and `ensoup` launcher to run the engine/language-server with working directory set to the directory containing currently running project.
- If the working directory is _not_ "the directory containing currently running project", a warning is written to logs. This can happen if the raw `/bin/enso` engine runner is used in a different directory.
- In the Cloud, the `File.new` interprets relative paths as cloud paths relative to the Cloud directory containing the current project. Absolute paths are unaffected.
There is no need to generate unused variables warnings or other linting for IDE and repl users. By default linting is enabled during compilation and for those use-cases it is now disabled via runtime options.
Closes#9883
Plain `Storage failure [AccessDenied].` was rather uninformative when it comes to debugging the underlying problem.
Added more detailed error messages and runners' failures should now sometimes print a detailed message.
References #10662.
Translate syntax warnings and attach to IR when translating operator applications.
We should ensure that all Trees are checked for warnings and every warning is attached to some IR. That would require a bit of refactoring: In TreeToIr, we could define helpers wrapping every IR constructor and accepting a `Tree` parameter. The `Tree` could be used to populate the `IdentifiedLocation` when constructing the IR type, and then to attach all warnings after constructing the IR object.
# Important Notes
- Update JNI dependency.
- Introduces a `cargo bench` runner for parser.
In a sequence of value-level operators, whitespace does not affect relative precedence. Functional operators still follow the space-precedence rules.
The "functional" operators are: `>> << |> |>> <| <<| : .`, application, and any operator containing `<-` or `->`. All other operators are considered value-level operators.
Asymmetric whitespace can still be used to form *operator sections* of value-level operators, e.g. `+2 * 3` is still equivalent to `x -> (x+2) * 3`.
Precedence of application is unchanged, so `f x+y` is still equivalent to `f (x + y)` and `f x+y * z` is still equivalent to `(f (x + y)) * z`.
Any attempt to use spacing to override value-level operator precedence will be caught by the new enso linter. Mixed spacing (for clarity) in value-operator expressions is allowed, as long as it is consistent with the precedences of the operators.
Closes#10366.
# Important Notes
Precedence warnings:
- The parser emits a warning if the whitespace in an expression is inconsistent with its effective precedence.
- A new enso linter can be run with `./run libraries lint`. It parses all `.enso` files in `distribution/lib` and `test`, and reports any errors or warnings. It can also be run on individual files: `cargo run --release --bin check_syntax -- file1 file2...` (the result may be easier to read than the `./run` output).
- The linter is also run as part of `./run lint`, so it is checked in CI.
Additional language change:
- The exponentiation operator (`^`) now has higher precedence than the multiplication class (`*`, `/`, `%`). This change did not affect any current enso files.
Library changes:
- The libraries have been updated. The new warnings were used to identify all affected code; the changes themselves have not been programmatically verified (in many cases their equivalence relies on the commutativity of string concatenation).
Graal Ydoc implementation is currently not being used locally or in the cloud and giving an impression of a slower startup.
Plus it appears that there some issues in the local connection as well.
To limit the impact of it now, let's make it controllable by the same env variable as GUI is.
* initProtocol endpoint returns success when already initialized
* Fix test
* Docs and return error when clientId differs
* fmt
* fix: session management test
* misc: json connection controller
---------
Co-authored-by: Dmitry Bushev <bushevdv@gmail.com>
Issue with Vector nothing to do with this.
The `System.exit 42` component is treated the same way as any other Panic error - it does not interfere with other component evaluation:
After removing the `System.exit 42` component, the workflow works as expected. I have also tried opening the project with the component and then removing it.
Enables `engine.TruffleCompilation` in `std-benchmarks`, collects the logs and dumps compilation into to `System.err` when a benchmark is influenced by dynamic compilation.
Single-phase whitespace-aware precedence resolution.
#### Performance
Since this is a major refactor of the core of the parser, I benchmarked it; it's about 3% faster.
# Important Notes
- Move operator-identifier recognition to lexer.
- Move compound-token assembly out of precedence resolver
- This PR only re-arranges code, splitting the **huge** `processModule` function into a few smaller ones.
- I decided to do it, because when I was working with `processModule` on #9812 I was constantly getting lost in this huge method (this **one** method had 570 lines!) - there is too much happening at once there. Now it's been split into smaller methods, each dealing with one thing.
Reducing the number of log events that we spam regular users. Not the last PR in that area, but already a progress.
Also replaced `fileWalk` with a stream approach hoping that maybe it is a bit more stable on Windows.
* Fix context lock removal
Removal of context lock assumed that one still holds a lock on it. This
is no longer the case when using a `withContextLock` block that
correctly manages the resource. This change fixes the
`IllegalMonitorStateException`.
Closes#10354.
* address review
* Add missing file
* Hiding ContextLock internals from other than ReentrantLocking classes
---------
Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
* Mitiigate LS DDOS scenario on startup
For a medium-size project with a lot of visualizations, Language Server
will be flooded with visualization requests on startup. Due to an
existing limit for the job execution engine, some of those requests
might have been silently dropped.
This change lifts that limit until a better fix can be invented.
Additionally, a slow startup might lead to a timeout when serving open
file request. This change adds some retries as a fallback
mechanism/progress monitoring.
* add runtime-fat-jar to a list of aggregates
Fix#10503 by creating a benchmark and then speeding it up by making sure usage of `InteropLibrary` reminds in partially evaluated code and isn't hidden behind `@TruffleBoundary`.
Addresses one of two concerns of #5298 - adds support for `--jvm` argument to allow us to switch from _native image_ built Enso binary (as developed by #10126) to regular JVM based Enso execution. This change _doesn't affect production builds_. The _native executable_ continues to be only built by `engine-runner/buildNativeImage` which is tested on CI, but not in the production jobs.
- Remove publishing the constructors.
- Fix any missed use in libs.
- Alter tests to generally use auto-scoped calls.
- `on_incomparable` to `on_problems`.
Building Engine, Context, ApplicationConfig and Ydoc was a adding a rather large delay during the initial startup step as all of those were blocking operations.
Moving all of those to the resource initialization step hopes to amortize some of that cost since it can be done in parallel. Had to add a `ComponentSupervisor` (open for a different name suggestion) to ensure that such delayed components are properly closed on shutdown.
# Important Notes
Adding Ydoc has added a visible delay during startup. I'm hoping that we can amortize some of that with this PR:
Now:
* Eliminating circe-yaml
This change adds our very-own YAML parser on top of SnakeYAML. Compared
to Circe parser on top of SnakeYAML. The advantage? In some not-so-distant
future we might actually get rid of circe and the related performance
issues.
The logic is similar to what circe does i.e. analyzing SnakeYAML to
build our own structure.
This change is not complete, as there are still some tests failing, but
most common Configs are already parseable.
We _could_ auto-generate some of the code but still some of the logic
would have to be tweaked by hand; the current logic has a number of
special cases, as I found out the hard way.
* wip: more tests passing
* Fix remaining tests in ConfigSpec
* Fixing YAML decoder for editions
Dropping circe as a decoder for Editions revealed some problems. Turns
out the current implementation had even more special cases to deal with.
* nit
* Allow for empty exports
* Mostly complete encodin part
Replaced almost all `toYAML` locations with SnakeYAML equivalent.
The encoding has to use Java collections for which there exists a
built-in support. If we were to use Scala collections we would have to
deal with tagging, at the very least.
* Remove the last remaining Circe's YAML parser
* Bug fix + further loop optimization
* removal of some dependencies
* Remove circe-yaml
Added a custom SnakeYAML Node updater to mimick the JSON -> YAML -> JSON
conversion needed for updating fields. The algorithm recursively follows
the key-path and inserts the desired Node. This is not a performance
oriented code on purpose.
* Fix compilation issues
`circe-core` was marked as `provided` but no one eventually included it
in the final jar, hence `NoClassFoundException`.
* fix licensing
* Removing obsolete circe definitions
* fmt
* nits
* s/SnakeYamlDecoder/YamlDecoder
* fmt
* Partial revert, PM needs JSON decoders/encoders
* style
* incremental compilation gone wrong
- Rename `Location.Start` to `Location.Left`.
- Rename `Location.End` to `Location.Right`.
- Use auto-scoping for `Location`.
- Tune widgets for `Text.trim`.
- Correct signature of `Text.split`.
- Adjist `generateLocallyUniqueIdent` to not fail on bad signature.
- Part of #9486
- Fixing our tests to not rely on deterministic ordering of created Tables in Database backends
- Before, SQLite and Postgres used to mostly return rows in the order they were inserted in, but Snowflake does not.
- Fixing various parts of Snowflake dialect.
Ultimately, we want to forbid the `from ... export all` syntax. This PR starts by providing a way to explicitly export extension and conversion methods by name.
Stdlib code will be modified in upcoming PR.
# Important Notes
A single name can refer to multiple extension or conversion methods. Exports are not qualified. For example,
```
type My_Type
type Other_Type
My_Type.ext_method x = x
Other_Type.ext_method x = x
```
```
from project.Mod export ext_method
```
will export both `My_Type.ext_method` and `Other_Type.ext_method`.
