Commit Graph

1171 Commits

Author SHA1 Message Date
Hubert Plociniczak
41fe87f2ff
Test benchmark theory (#8668) 2024-01-04 16:28:56 +00:00
Pavel Marek
428e83de36
Remove org.bouncycastle dependency (#8664)
Remove `org.bouncycastle` dependency from `org.enso.runtime`.
2024-01-04 17:16:41 +01:00
Jaroslav Tulach
4222ddccb5
Canonicalize file names before comparing them (#8670) 2024-01-04 15:21:13 +01:00
Hubert Plociniczak
20531d51df
Initialize builtin methods lazily in non-AOT mode (#8654)
Registration of builtin methods is done upfront, similarly to AOT mode, but the initialization is delayed until the first use.
One still has to read the full list of builtin methods and register them in the builtins scope or we would be getting the `NoSuchMethod` errors otherwise. But the most expensive operation, initialization of classes and getting a method via a reflection, is all done on the first usage.

The result is an improved startup and negligible performance impact in regular usage.

Closes #8423.

# Important Notes
Visible speedup.
1. (old) Program returning plain "Hello World" (no stdlib loading)
![Screenshot from 2024-01-02 17-34-35](https://github.com/enso-org/enso/assets/292128/b8348cbb-baf6-4292-8c0e-c8a3492c6583)
2. (new) Program returning plain "Hello World" (no stdlib loading)
![Screenshot from 2024-01-02 17-34-54](https://github.com/enso-org/enso/assets/292128/63dd6c4f-49e2-4a5e-ab95-2460c155b91c)
About 10% improvement for the overall execution.

3. (old) Program printing "Hello World" (some stdlib loading)
![Screenshot from 2024-01-02 17-40-21](https://github.com/enso-org/enso/assets/292128/cf3f464c-8f00-4adf-9499-a6ac86430da0)
4. (new) Program printing "Hello World" (some stdlib loading)
![Screenshot from 2024-01-02 17-40-41](https://github.com/enso-org/enso/assets/292128/3ef99893-ccaf-4ea7-90ba-aa090f904a50)
About 2% improvement for the overall execution.

Minor price to pay for lazy initialization:
![Screenshot from 2024-01-02 17-56-12](https://github.com/enso-org/enso/assets/292128/faa682d8-5370-43d2-9fc9-3454af5749a4)
2024-01-04 08:55:21 +00:00
Jaroslav Tulach
5e3480b1e8
Apply SectionsToBinOp pass to arguments of Section (#8655)
Fixes #8436 by making sure that `SectionsToBinOp` pass is applied also to arguments of `Section`.
2024-01-03 14:37:11 +00:00
Dmitry Bushev
cfab344fbe
Unable to access constructor value in interactive mode (#8626)
close #7184

The constructor value was not accessible because during the re-compilation a new instance of the type was registered in runtime. Then during the execution, an old cached instance of the type was used in method resolution.

Changelog:
- update: the registration of types in runtime
- update: invalidate cached nodes that became a resolution error after applying the edit
2024-01-03 13:23:42 +00:00
Pavel Marek
819741bb4a
Increase warmup for EqualsBenchmarks. (#8636)
After #8620, there is a noticeable slowdown in `EqualsBenchmarks.equalsTrees` as suggested in https://github.com/enso-org/enso/pull/8620#issuecomment-1870776609. After some digging, I realized that the number of warmup iterations is most probably insufficient. Let's increase the warmup for this benchmark and see if we can get its score down again.
2023-12-29 08:47:36 +00:00
Jaroslav Tulach
27431c126c
Make IR caches smaller by storing Graph as a reference (#8635) 2023-12-29 06:15:48 +01:00
Pavel Marek
48a3c14ee5
Format Java sources in benchmark configuration (#8638)
I noticed that sources in `runtime/bench` are not formatted at all. Turns out that the `JavaFormatterPlugin` does not override `javafmt` task for the `Benchmark` configuration. After some failed attempts, I have just redefined the `Benchmark/javafmt` task in the `runtime` project. After all, the `runtime` project is almost the only project where we have any Java benchmarks.

# Important Notes
`javafmtAll` now also formats sources in `runtime/bench/src/java`.
2023-12-28 17:06:06 +00:00
Dmitry Bushev
2d628263ff
Update cache invalidation commands (#8634) 2023-12-26 12:39:17 +00:00
Jaroslav Tulach
07d58f2c02
DataflowError.withoutTrace shall not store a trace (#8608) 2023-12-24 11:07:32 +01:00
Pavel Marek
74436830ce
Fix build of engine benchmarks (#8620)
After #8467, Engine benchmarks are broken, they cannot compile - https://github.com/enso-org/enso/actions/runs/7268987483/job/19805862815#logs

This PR fixes the benchmark build

# Important Notes
Apart from fixing the build of `engine/bench`:
- Don't assemble any fat jars in `runtime/bench`.
- Use our `TestLogProvider` in the benches instead of NOOP provider.
- So that we can at least see warnings and errors in benchmarks.
2023-12-22 11:40:32 +00:00
Pavel Marek
edc98593b9
All unit tests use the same testing logging provider (#8593)
Make sure that the correct test logging provider is loaded in `project-manager/Test`, so that only WARN and ERROR log messages are displayed. Also, make sure that the test log provider parses the correct configuration file - Rename all the `application.conf` files in the test resources to `application-test.conf`.

The problem was introduced in #8467
2023-12-21 13:45:33 +00:00
Radosław Waśko
dfdb547616
Better context info in Type_Error raised from return type checks (#8566)
- Followup to #8502 that adds better error messages
2023-12-20 18:22:47 +00:00
Pavel Marek
4cb2439890
Submodules can be private (#8581) 2023-12-19 19:13:44 +01:00
Jaroslav Tulach
7daad75dd9
Avoid WithWarnings without any warnings (#8583) 2023-12-19 17:03:16 +01:00
Radosław Waśko
f0c2a5fa7f
Opt-in return type checks (#8502)
- Closes #8240
2023-12-19 15:32:30 +00:00
Hubert Plociniczak
1ad11104d3
Stop parsing METADATA section for IdMap (#8574)
We infer old IDs from module's IRs location, instead. This workarounds ill-constructed IdMaps at the start of the project.

If IDE or backend ends up with invalid ID map, I suppose application of edits might still fail.

# Important Notes
Closes #8500 by not doing the parsing at all. Since I couldn't figure why we had invalid metadata in the first place, this might still bite us at some point.
2023-12-19 12:03:33 +00:00
Pavel Marek
21d164ec3e
Run unit tests with truffle-compiler (#8467) 2023-12-18 18:22:16 +01:00
Dmitry Bushev
b2a1cd55d2
Syntax error results in silent error (#8560)
close #7555

Compiler passes after `GenerateMethodBodies` expect the method body to be a function.

After fixing the pass, the compilation returns a proper compiler error:
```
built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Table/0.0.0-dev/src/Data/Column.enso:869:22: error: Methods must have only one definition of the `this` argument, and it must be the first.
869 |     round self round self (decimal_places:Integer = 0) (use_bankers:Boolean = False) = Value_Type.expect_numeric self <|
|                      ^~~~
Aborting due to 1 errors and 0 warnings.
```
2023-12-18 13:12:31 +00:00
Pavel Marek
c1098865f2
Update java formatter sbt plugin (#8543)
Add a local clone of javaFormatter plugin. The upstream is not maintained anymore. And we need to update it to use the newest Google java formatter because the old one, that we use, cannot format sources with Java 8+ syntax.

# Important Notes
Update to Google java formatter 1.18.1 - https://github.com/google/google-java-format/releases/tag/v1.18.1
2023-12-15 14:45:23 +00:00
Dmitry Bushev
56cc9561b1
Fix serialization of alias analysis graph (#8550)
close #8431

Fixes the scenario:
- user sends `executionContext/executeExpression`
- program execution is scheduled
- during the compilation the already compiled `IR` is loaded from the cache (reading invalid alias analysis graph)
- during the codegen the local scope with that aliasing graph is propagated to the runtime
- `EvalNode` compiles the expression to execute with the local scope containing an invalid aliasing graph
- compilation fails in the `AliasAnalysis` pass because of the clashing IDs in the graph
2023-12-15 12:54:52 +00:00
Pavel Marek
4b65e44ef3
EpbLanguage re-uses other TruffleContext support to run tests with assertions enabled (#7882) 2023-12-15 13:31:32 +01:00
Radosław Waśko
95f11abe2c
Try reporting diagnostics in non-strict mode (#8272)
- Makes sure that the compiler will print the diagnostics even if non-strict mode is used
- To prevent printing unexpected stuff to stdout in interactive mode, the diagnostics are printed as 'warning' level log messages in that mode. In strict mode, the messages are printed like before, without changes.
2023-12-14 15:41:35 +00:00
GregoryTravis
1c815a3d45
Better Error Trapping in map (#8307)
* tests

* wip

* wip

* additional warnings

* wip

* wip

* cleanup

* nested wrapping

* multiple nestings

* wraps_error uses looks_for, test for should_fail_with

* wip

* stack trace line fix

* use catch_primitive internally

* fix warning mapping, dtf spec

* just one wrapper checker, vector spec

* missing ctor, back to non-primitive catch

* back to c_p

* put old map back

* wip

* unnest tests

* Array.map on_problems

* wip

* Revert "wip"

This reverts commit c30d171457.

* better test names

* warning logging

* wip

* wip

* move logic into ALH

* doc

* constant

* My_Error.Error

* nested

* doc

* map_primtiive in warning mapper

* composition

* ref spec

* Remove warnings prior to matching on the value

If an expression has warnings and is matched we:
1) extract the warnings
2) execute the branch of a pattern that matches the value
3) attach extracted warnings to the result

This caused warnings to reappear when doing the custom warnings
manipulation.
This is also consistent with how `CaseNode`'s `doWarning` specialization
is defined.

* fix 1

* do not auto unwrap in test error checkers

* nested error matcher

* in problems too

* dtf

* v

* statistics

* wip

* Table_Spec, map_with_index_primitive

* Column_Operations_Spec

* disable warning wrapping and Report_Warning

* unimpl test

* Warnings_Spec

* DCS

* ACG JP

* zip_primitive

* join_helpers

* Lookup_Helpers

* Table

* Data_Formatter

* Value_Type_Helpers

* revert check types changes

* table_helpers

* table tests

* remove st

* do not remove warnings from value

* vec docs, tests for zip, mwi, flat_map

* docs, fixes

* remove nested_error_matcher

* cleanup

* benchmark

* one error

* alter

* add bench to main

* review

* review

* review

* tail call

* changelog

* tail call was not a tail call

* ws

* bad import

* Added missing import

* Update distribution/lib/Standard/Base/0.0.0-dev/src/Data/Array.enso

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>

* review, ref example

* lazy benchmark data

* extra paren

* check outside of catch

* review

* vector too

* actually lazy

* disambiguate Map_Error

* finish rename

* move to extensions

* combine Additional_Warnings error

* rename to map_no_wrap

* do not catch and rethrow

* review

* wip

* remove _primitives entirely

* remove unused should_fail_with function options

* remove expected_warning as function in Problems

---------

Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
2023-12-13 09:38:09 -05:00
Pavel Marek
1bde4c5699
Add org.jline as module to component directory (#8496)
Add `jline` module to the distribution so that our REPL is usable again.

# Important Notes
- No more: "WARNING: Unable to create a system terminal, creating a dumb terminal " warning when starting REPL
- Arrow keys works as expected in REPL
- Back search (the default shortcut `CTRL + R`) works as expected.
2023-12-13 13:47:50 +00:00
Kaz Wesley
ce6c770fd7
Parse inline function signatures (#8470)
Implements #6166.

# Important Notes
- More consistent handling of `default` arguments. `default` is a valid identifier, and only has special meaning when it isn't bound in scope. Since distinguishing the builtin `default` from an identifier called `default` cannot be done until alias analysis has been performed, `default` is now represented in the AST as a regular identifier.
- `TreeToIr`: Remove `insideTypeAscription`. It was only used for bug-for-bug compatibility with the old parser during the transition.
2023-12-12 14:48:44 +00:00
Jaroslav Tulach
80f94a21e1
Compare long and double and BigInteger properly (#8510) 2023-12-12 06:41:43 +01:00
Jaroslav Tulach
e4b2b56a40
Turning Sieve benchmarks into Enso benchmarks (#8475) 2023-12-08 10:27:52 +01:00
Hubert Plociniczak
a14ebd6259
Increase timeout for running launcher command (#8486)
We've been experiencing consistently failures on MacOS due to timeouts.
Doubling the timeout, hoping this will be sufficient to eliminate such
false failures. Will seek alternative solutions if that does not rememdy
the problem on CI.
2023-12-07 16:34:55 +01:00
Hubert Plociniczak
021ff2b40e
Mark some tests as flaky on Windows (#8446) 2023-12-06 11:17:34 +00:00
Hubert Plociniczak
a11bddcb74
Inline execution should support FQNs (#8454)
* Test illustrating problems with FQNs

Inline execution fails with `Compile error: The name `Standard` could
not be found.`.

* Ensure InlineContext carries Package Repos info

Previously, there was no requirement that inline execution should allow
for FQNs. This meant that the omission of Package Repository info went
unnoticed.

In order to be able to refer to `Standard.Visualization.Preprocessor` it
has to be exported as well.
2023-12-06 10:03:06 +01:00
Jaroslav Tulach
7f0cb88fa1
Consistent simple and qualified type name (#8448)
Fixes #8255 by unifying `get_qualified_type_name` and `get_simple_type_name` implementations.
2023-12-06 04:30:24 +00:00
Dmitry Bushev
98d2221b13
Detach oneshot visualization when the evaluation failed (#8463)
close #8429

Changelog:
- fix: detach oneshot visualization in case when the evaluation was failed
2023-12-05 16:15:12 +00:00
Pavel Marek
a67297aebf
Add graalpy packages to the component directory (#8351)
Adds these JAR modules to the `component` directory inside Engine distribution:
- `graal-language-23.1.0`
- `org.bouncycastle.*` - these need to be added for graalpy language

# Important Notes
- Remove `org.bouncycastle.*` packages from `runtime.jar` fat jar.
- Make sure that the `./run` script preinstalls GraalPy standalone distribution before starting engine tests
- Note that using `python -m venv` is only possible from standalone distribution, we cannot distribute `graalpython-launcher`.
- Make sure that installation of `numpy` and its polyglot execution example works.
- Convert `Text` to `TruffleString` before passing to GraalPy - 8ee9a2816f
2023-12-04 11:50:59 +00:00
Hubert Plociniczak
5b38aff294
Part II of making sure that tests clean up after each run (#8432)
* Ensure runtime tests cleanup after run

Context is closed, maps/lists are cleaned up.

* DRY
2023-12-04 10:10:01 +01:00
Jaroslav Tulach
2f676963da
Properly report errors on duplicated constructor names (#8438) 2023-12-01 17:30:34 +01:00
Jaroslav Tulach
b1be8c0faa
Multi line chained operator syntax (#8415) 2023-12-01 11:48:37 +01:00
Jaroslav Tulach
65daaf6f0c
Replacing usage of Source by (Location => String) (#8437) 2023-12-01 08:41:24 +01:00
Jaroslav Tulach
81f06456bf
400x faster with linear hashing of the hash map entries (#8425)
Fixes #5233 by removing `EconomicMap` & co. and using plain old good _linear hashing_. Fixes #8090 by introducing `StorageEntry.removed()` rather than copying the builder on each removal.
2023-12-01 06:43:13 +00:00
Hubert Plociniczak
b6bdf901a9
Cleanup after each test run (#8418)
Reducing leaks when running our test suite.

Potentially fixes #8408.

# Important Notes
Managed to keep thread count and memory in between runs relatively stable.
Initially:
![Screenshot from 2023-11-29 11-06-04](https://github.com/enso-org/enso/assets/292128/af437d8a-9111-4bd6-9033-a59030c7ebed)
Now:
![Screenshot from 2023-11-29 15-57-34](https://github.com/enso-org/enso/assets/292128/3c1f8aef-fe7e-4f5b-a236-12c86ea8b906)

The screenshot illustrates for `RuntimeVisualizationsTest` only. Will need to be applied in other places as well.

Applying the same style to `runtime-with-instruments`.
Before:
![Screenshot from 2023-11-29 16-55-11](https://github.com/enso-org/enso/assets/292128/385e11be-0265-431d-b0d7-c5096df11c50)
After:
![Screenshot from 2023-11-29 16-50-07](https://github.com/enso-org/enso/assets/292128/5e8c28ea-e921-484a-a82c-9f2d3e827e8b)
2023-11-29 19:02:59 +00:00
Jaroslav Tulach
f1cd51ab6c
Convert constants to Enso values (#8416) 2023-11-29 14:57:45 +01:00
Jaroslav Tulach
c889c8e83f
More robust work with caches (#8393) 2023-11-28 09:03:15 +01:00
Hubert Plociniczak
0402e8bafb
Ensure compilation is run with a compilation lock (#8395)
Evaluating visualization expression may trigger a full compilation. A change in #7042 went a bit too far and led to a situation when there could be compilations running at the same time leading to a rather obscure `RedefinedMethodException` when the compilation on one thread already finished. This will make the logic correct again at the price of potentially slowing the processing of visualization.

Closes #8296.

# Important Notes
Should make visualizations a bit more stable as well.
2023-11-27 14:01:23 +00:00
Jaroslav Tulach
7a9a5ba1ff
Rewriting MetadataStorage into Java (#8366) 2023-11-27 13:55:40 +01:00
Jaroslav Tulach
893965ed5c
3% speedup with LazyMap and MetadataStorage (#8359) 2023-11-27 10:28:12 +01:00
Hubert Plociniczak
36996c8938
Don't NPE on op ._ when translating tree to IR (#8381)
Encountered a random NPE when playing with bookclubs. Test case demonstrating the problem is attached.

Threw in a bunch of minor tweaks to logs to make life of the person debugging code more pleasant.
2023-11-23 19:56:56 +00:00
Dmitry Bushev
f1825f3f32
Gather diagnostics from annotations (#8380)
close #8289

Changelog
- feat: `GatherDiagnostics` compiler pass works with annotations
2023-11-23 17:18:39 +00:00
Dmitry Bushev
4b3ba78b52
Add shortcuts to start and stop the backend profiling (#8358)
close #8329

Changelog:
- add: `cmd`+`shift`+`,` and `cmd`+`shift`+`.` shortcuts to start and stop the backend profiling. Profiling data is stored on disk.
2023-11-23 15:31:17 +00:00
Pavel Marek
a9099ddce5
Fix Java log initialization in std-table (#8364) 2023-11-23 14:10:22 +01:00
Hubert Plociniczak
af5be19be8
Minor tweaks post jdk21 upgrade (#8373) 2023-11-23 11:58:49 +00:00
Pavel Marek
268e595ec1
Add Chrome devtools and DAP tools for debugging (#8344)
Adds chrome-inspector tool and Debug Adapter protocol tool for debugging Enso.

# Important Notes
The chrome devtools seems to be broken (tracked in https://github.com/oracle/graal/issues/7636). Even `graalpy --inspect ...` fails (Chrome devtools freezes after connecting to the server). This PR adds Debug adapter protocol tool for debugging as a workaround for chrome inspector. There is docs in [dap.md](https://github.com/enso-org/enso/pull/8344/files#diff-421574b50574cfe546e86d4b3d32d79b8b2087f2fe204f68e5cf2693af43bbe1)
2023-11-22 17:18:41 +00:00
Jaroslav Tulach
1138dfe147
Specify expression to get more advanced results on_return callback (#8331) 2023-11-20 18:47:11 +01:00
Dmitry Bushev
53d1f727da
Always create events log when profiling (#8337)
Changelog:
- update: always create an event log next to the profiling file when the engine is started with the `--profiling-path` flag
- remove: `--profiling-events-log-path` flag
2023-11-20 16:01:25 +00:00
Jaroslav Tulach
705d6f192c
Syntax error on unexpected unary minus (#8333) 2023-11-20 16:14:19 +01:00
Dmitry Bushev
b224f95639
On-deman backend heap and thread dump (#8320)
close #8249

Changelog:
- add: `profiling/snapshot` request that takes a heap dump of the language server and puts it in the `ENSO_DATA_DIRECTORY/profiling` direcotry
2023-11-20 11:41:01 +00:00
Hubert Plociniczak
b8ebed69c3
Report failures when upserting visualizations (#8306)
Attaching or modifying a visualizations returns early on, to avoid a situation when a background job is stalled (by other jobs) and eventually the request timeouts.

This has an unfortunate consequence that any error reported in the `UpsertVisualizationJob` cannot be reported as a directly reply to a request because the sender has already been removed from the list.

Added more logs to discover why we get errors in the first place.

Modified the API a bit so that we carry `VisualizationContext` instead of three parameters all over the place.

Bonus:
Modified `JsonRpcServerTestKit` to implicitly require a position so that we get better error reporting on failures.
2023-11-20 11:38:51 +00:00
Jaroslav Tulach
ba19813511
Speeding up "hello world" example by 16% 2023-11-19 16:38:31 +01:00
Pavel Marek
5a7ad6bfe4
Upgrade enso to GraalVM for jdk 21 (#7991)
Upgrade to GraalVM JDK 21.
```
> java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing)
```

With SDKMan, download with `sdk install java 21-graalce`.

# Important Notes
- After this PR, one can theoretically run enso with any JRE with version at least 21.
- Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution.
- `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded.
- sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368
- Which means that we can have `module-info.java` files only for Java-only projects.
- Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar.
- `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent.
- Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory.
- The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module.
- The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725)
- [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 18:02:36 +00:00
Dmitry Bushev
a286ab7daa
Backend self sampling (#8309)
close #8248

Changelog:
- add: `profiling/start` request starts the sampler and starts collecting runtime events to the log file
- add: `profiling/stop` request stop the sampler and write the profiling data to the `$ENSO_DATA_DIR/profiling` directory
- refactor: rewrite the profiling logic into Java
2023-11-17 15:04:10 +00:00
Hubert Plociniczak
348f5170ab
Avoid NPE during instrumentation (#8317)
Fixes a random crash (*) during instrumentation. Notice how `onTailCallReturn` calls `onReturnValue` with `null` frame.

Bonus: noticed that for some reason we weren't getting logs for `ExecutionService`. This turned out to be the problem with the logger name which by default was `[enso]` not
`[enso.org.enso.interpreter.service.ExecutionService`] and there is some logic there that normalizes the name and assumed a dot after `enso`. This change fixes the logic.

(*)
```
[enso.org.enso.interpreter.service.ExecutionService] Execution of function main failed (Cannot invoke "com.oracle.truffle.api.frame.VirtualFrame.materialize()" because "frame" is null).
java.lang.NullPointerException: Cannot invoke "com.oracle.truffle.api.frame.VirtualFrame.materialize()" because "frame" is null
at org.enso.interpreter.instrument.IdExecutionInstrument$IdEventNodeFactory$IdExecutionEventNode.onReturnValue(IdExecutionInstrument.java:246)
at org.enso.interpreter.instrument.IdExecutionInstrument$IdEventNodeFactory$IdExecutionEventNode.onTailCallReturn(IdExecutionInstrument.java:274)
at org.enso.interpreter.instrument.IdExecutionInstrument$IdEventNodeFactory$IdExecutionEventNode.onReturnExceptional(IdExecutionInstrument.java:258)
at org.graalvm.truffle/com.oracle.truffle.api.instrumentation.ProbeNode$EventProviderChainNode.innerOnReturnExceptional(ProbeNode.java:1395)
at org.graalvm.truffle/com.oracle.truffle.api.instrumentation.ProbeNode$EventChainNode.onReturnExceptional(ProbeNode.java:1031)
at org.graalvm.truffle/com.oracle.truffle.api.instrumentation.ProbeNode.onReturnExceptionalOrUnwind(ProbeNode.java:296)
at org.enso.interpreter.node.ExpressionNodeWrapper.executeGeneric(ExpressionNodeWrapper.java:119)
at org.enso.interpreter.node.ClosureRootNode.execute(ClosureRootNode.java:85)
at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.executeRootNode(OptimizedCallTarget.java:718)
at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.profiledPERoot(OptimizedCallTarget.java:641)
at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callBoundary(OptimizedCallTarget.java:574)
at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.doInvoke(OptimizedCallTarget.java:558)
at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedCallTarget.callDirect(OptimizedCallTarget.java:504)
at jdk.internal.vm.compiler/org.graalvm.compiler.truffle.runtime.OptimizedDirectCallNode.call(OptimizedDirectCallNode.java:69)
at org.enso.interpreter.node.callable.thunk.ThunkExecutorNode.doCached(ThunkExecutorNode.java:69)
at org.enso.interpreter.node.callable.thunk.ThunkExecutorNodeGen.executeAndSpecialize(ThunkExecutorNodeGen.java:207)
at org.enso.interpreter.node.callable.thunk.ThunkExecutorNodeGen.executeThunk(ThunkExecutorNodeGen.java:167)
...
```

# Important Notes
Fixes regressions introduced in #8148 and #8162
2023-11-17 14:38:27 +00:00
Hubert Plociniczak
c649ed87af
Let logging to file respect log level (#8264)
This change fixes a regression introduced in #7918, which prevented the execution from setting the right log level either via env var or parameter.

Now passing either of the options returns logs of the expected level in the log file:
- `ENSO_LOG_TO_FILE_LOG_LEVEL = trace`
- ... `-vv` ...

Fixes #8274
2023-11-14 11:59:27 +00:00
Dmitry Bushev
565a858c5f
Inline Execution (#8148)
close #8132

Update the `executionContext/executeExpression` request to execute expressions in the local scope.
2023-11-13 16:05:05 +00:00
Jaroslav Tulach
94aa58e2a5
Allow IRPass to only access CompilerContext (#8281) 2023-11-13 09:58:30 +01:00
Radosław Waśko
f47ef0c111
Fix rendering of redefined Conversion error (#8245)
- Fixes #7853
2023-11-11 16:27:52 +00:00
Hubert Plociniczak
51abb3e1b0
Apply custom log levels to Truffle logger (#8162)
Previously custom log levels applied only to non-Truffle loggers. To allow it, filtering has to be applied appropriately at two places - first at Java's Handler and then essentially re-confirmed at SLF4J's logger to which the former forwards to.
Filters compose in an `AND` condition, therefore default log level check had to be merged into our custom filters.

`TruffleLogger` has a builtin functionality to perform the filtering when context is configured appropriately. This should be much more efficient than adding a `Filter` to the JUL Handler explicitly.

# Important Notes
```
JAVA_OPTS="-org.enso.compiler.SerializationManager.Logger.level=debug" ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso  --run
```
will now assign a custom log level to `SerializationManager` Logger.
2023-11-10 16:34:04 +00:00
Dmitry Bushev
18c7135769
Fix runtime dataflow error tests (#8273)
@radeusgd pointed out that tests are checking that the engine does not send updates when the dataflow error changes. In the end, it turned out that those tests were not checking what they said, and the engine sent the proper updates.
2023-11-10 14:41:32 +00:00
Dmitry Bushev
210c1907a8
Fix with profiling option (#8266)
close #8156

Followup to #8254. For some reason, I did not commit this change to the original PR.
2023-11-09 09:26:00 +00:00
Dmitry Bushev
5b4716e65a
Enable profiling for all commands of runner executable (#8254)
close #8156
2023-11-08 19:30:28 +00:00
Hubert Plociniczak
f21e09bb65
More logs to debug failed edits (#8251) 2023-11-08 15:06:17 +00:00
Hubert Plociniczak
1388fe1cf9
More fine grained initialization for resources (#8242)
A long running initialization of the component blocks the execution significantly. Removed the `BlockingInitialization` and replaced it with a more fine grained locking.

# Important Notes
Added a simple workaround for potential slow initialization of backend - more retries. We should have a better UX in that case anyway, but due to absence of work on that in old GUI, this will have to do.
The main should be problem should be addressed already by other backend changes. Changes in `app` should only be treated as _just in case something bad happens_.
2023-11-08 08:57:31 +00:00
Hubert Plociniczak
21dc90a0ed
OpenFileCmd sends a reply when finished (#8225)
* OpenFileCmd sends a reply when finished

Lack of reply and therefore a non-determinism on when OpenFile handler
can finish, led to some sporadic instability. Once caches format got
changed, things were taking such a long time that I wasn't able to start
even a basic project (requests would start to timeout).
The change also removes PushContextCmd from synchronous cmds (as
introduced in #798 to remove initialization problems in tests as well as
in real scenarions); the change did the job but was also a bit
controversial.
The change can also help with randomly failing applies (#8174) as IDE kept
closing and opening the project that might have exploited the
race-condition.

* Adapt tests

* Make tests more resilient to out of order messages

* Drop retries that lead to confusing errors

* less random failures

* s/OpenFileNotification/OpenFileRequest
2023-11-07 23:59:42 +01:00
Pavel Marek
88848522cd
Add disable-private-check option (#8202) 2023-11-02 14:25:57 +01:00
Dmitry Bushev
c6cae8cd11
Unlock resources initialization asynchronously (#8206)
Debugging the issue reported by @PabloBuchu when the language server initialization hangs in the cloud. I'm still not sure what is happening in the cloud because I was not able to reproduce it when trying to connect two clients simultaneously.

Another potential source of the issue may be the Scala Future -> Java CompletableFuture conversion, but I didn't find anything suspicious there.
2023-11-02 11:51:37 +00:00
Hubert Plociniczak
2db4f4c5d9
Upgrade directory-watcher library (#8201)
The change upgrades `directory-watcher` library, hoping that it will fix the problem reported in #7695 (there has been a number of bug fixes in MacOS listener since then).

Once upgraded, tests in `WatcherAdapterSpec` because the logic that attempted to ensure the proper initialization order in the test using semaphore was wrong. Now starting the watcher using `watchAsync` which only returns the future when the watcher successfully registers for paths. Ideally authors of the library would make the registration bit public
(3218d68a84/core/src/main/java/io/methvin/watcher/DirectoryWatcher.java (L229C7-L229C20)) but it is the best we can do so far.

Had to adapt to the new API in PathWatcher as well, ensuring the right order of initialization.

Should fix #7695.
2023-11-02 11:24:26 +00:00
Jaroslav Tulach
3fd2249864
Introducing engine/runtime-compiler project (#8197) 2023-11-01 12:42:34 +01:00
GregoryTravis
1480f50207
Overhaul the random number and item generation code (#8127)
Rewrite most of Random.enso.
2023-10-31 15:25:37 +00:00
Jaroslav Tulach
f2cfd7f86c
Allow reassigning of BindingsMap (#8190)
Fixes #8186 by turning `IllegalStateException` into log message. Re-assigning of `BindingsMap` can happen in the IDE where evaluation of modules is repeated again and again. In addition to that avoid dropping errors in compiler without them being noticed.
2023-10-31 10:03:37 +00:00
Hubert Plociniczak
c1c4c8aa18
Use TruffleLogger bound to the engine to prevent illegal usage (#8169)
Using a `TruffleLogger` in `SerializationManager` that is bound to the engine rather than the context prevents reaching an illegal state when using thread pools.

Also cleaned up some tests for consistency.

To verify the fix
```diff
--- a/engine/runtime/src/main/scala/org/enso/compiler/SerializationManager.scala
+++ b/engine/runtime/src/main/scala/org/enso/compiler/SerializationManager.scala
@@ -31,7 +31,7 @@ final class SerializationManager(compiler: Compiler) {
import SerializationManager._

/** The debug logging level. */
-  private val debugLogLevel = Level.FINE
+  private val debugLogLevel = Level.INFO
```
and run
`sbt:enso> runtime/test`

Closes #8147.
2023-10-31 08:53:05 +00:00
Jaroslav Tulach
3d23c6a8d0
Removal of useless ApplicationSaturation phase (#8181) 2023-10-31 06:20:29 +01:00
Jaroslav Tulach
646b47b246
Caches belong to runtime, not IR compiler (#8178) 2023-10-30 16:53:44 +01:00
Jaroslav Tulach
a862ea7948
Eliminate references to Truffle nodes & co. in the compiler (#8172) 2023-10-30 10:57:21 +01:00
Jaroslav Tulach
da21e51bae
Resolve imports and exports via cached BindingsMap (#8160) 2023-10-28 07:43:04 +02:00
Pavel Marek
b084e097c9
Add special handling for Dataflow_Error passed to Runtime.assert (#8168)
Add special handling for `Dataflow_Error` passed as action to `Runtime.assert`. It caused an infinite recursion.
2023-10-27 17:22:15 +00:00
Hubert Plociniczak
13969abc92
Eliminate potential race-condition in PathWatcher (#8154)
The change eliminates a race-condition that can appear between the `PathWatcher` deregistering the last client (and shutting down) and `ReceivesTreeUpdatesHandler` receiving the termination message of that event. In between there could come a message towards the mentioned `PathWatcher` resulting in a timeout.

The scenario has become rather common in CI tests resulting in spurious failures. The problem could also be simulated by adding artificial `Thread.sleep` between sending the reply in `PathWatcher` and stopping the actor.

Fixes #8151.

# Important Notes
Example failure https://github.com/enso-org/enso/actions/runs/6642894609/job/18048711561?pr=8145
To simulate the scenario
```diff
diff --git a/engine/language-server/src/main/scala/org/enso/languageserver/filemanager/PathWatcher.scala b/engine/language-server/src/main/scala/org/enso/languageserver/filemanager/PathWatcher.scala
index 713cfe1182..88afac95cb 100644
--- a/engine/language-server/src/main/scala/org/enso/languageserver/filemanager/PathWatcher.scala
+++ b/engine/language-server/src/main/scala/org/enso/languageserver/filemanager/PathWatcher.scala
@@ -110,6 +110,7 @@ final class PathWatcher(

case UnwatchPath(client) =>
sender() ! CapabilityReleased
+      Thread.sleep(2000)
unregisterClient(root, base, clients - client)
```
2023-10-27 10:17:25 +00:00
Hubert Plociniczak
10f35390a1
Translate IR to Java (#8145)
Towards reduced reliance on Scala semantics.
Translated IR.scala to IR.java and extracted implicits that now need to be imported explicitly.

# Important Notes
1:1 translation. For now `@Identifier` and `@ExternalID` represent the old type aliases but are not verified at compile time.
This is because in a mixed Scala/Java world this seems impossible to employ such frameworks as Checker.
2023-10-26 07:31:08 +00:00
Jaroslav Tulach
5e468f08ad
Introducing CompilerContext.Module (#8144) 2023-10-25 17:11:47 +02:00
Hubert Plociniczak
61a0c8ce3f
Error on invalid capability acquire/release request (#8133)
`capability/acquire` with an invalid method `executionContext/canModify` would previously timeout because the capability router simply didn't support that kind of capability request.
Now rather than timing out, indicating a failure on LS, we report an error.

Closes #8038.
2023-10-24 13:01:10 +00:00
Jaroslav Tulach
e283c78977
Properly convert defaulted arguments before on_call back (#8143) 2023-10-24 13:39:54 +02:00
Jaroslav Tulach
08c9ecb540
Simplify code to run in the browser (#8092)
In order to execute `runtime-parser` in browser, we need to slightly simplify dependencies of our code.

# Important Notes
The actual PR explaining why these changes are desirable is #8094
2023-10-23 14:16:15 +00:00
Radosław Waśko
0c278391fe
Test and improve handling of Date_Time with_timezone=False in Postgres (#8114)
- Fixes #8049
- Adds tests for handling of Date_Time upload/download in Postgres.
- Adds tests for edge cases of handling of Decimal and Binary types in Postgres.
2023-10-21 21:35:13 +00:00
Dmitry Bushev
b1df8b1889
Initialize suggestions database only once (#8116)
close #8033

Changelog:
- update: run language server initialization once
- fix: issues with async `getSuggestionDatabase` message handling in new IDE
- update: implement unique background jobs
- refactor: initialization logic to Java
- refactor: `UniqueJob` to a marker interface
2023-10-21 20:32:13 +00:00
Hubert Plociniczak
cd94b388d1
Report human-readable names for shadowed parameters (#8115)
Improved warning reporting by not reporting IR in user-directed warning
messages. Made sure that fresh names retain some knowledge of the name
which they were created from.

Closes #7963.
2023-10-20 09:01:52 +00:00
Hubert Plociniczak
aa88dfeb2f
More debug info for failed edits (#8107)
Unmasked position info for the failed edits so that it is possible to understand what edits are attempted at all. So far unable to locate the problem based solely on the logs and the stacktrace. Added a test confirming that making edits at the end of the file continues to be fine (that's where the problem usually manifests itself).

# Important Notes
References #7978
2023-10-19 13:46:42 +00:00
Kaz Wesley
2edd2bd7ff
Ensure all spans have document offsets (#8039)
- Validate spans during existing lexer and parser unit tests, and in `enso_parser_debug`.
- Fix lost span info causing failures of updated tests.

# Important Notes
- [x] Output of `parse_all_enso_files.sh` is unchanged since before #7881 (modulo libs changes since then).
- When the parser encounters an input with the first line indented, it now creates a sub-block for lines at than indent level, and emits a syntax error (every indented block must have a parent).
- When the parser encounters a number with a base but no digits (e.g. `0x`), it now emits a `Number` with `None` in the digits field rather than a 0-length digits token.
2023-10-19 12:36:42 +00:00
Hubert Plociniczak
ab2da9e436
Ambiguous imports warnings/errors are serializable (#8093)
`AmbgiuousImport` error and `DuplicatedImport` warning must not have `Source` as one of its fields or compiler will crash during a serialization attempt.
Changed the implementation to provide it as a parameter to the `message` method instead.

Fixes #8089
2023-10-18 21:51:58 +00:00
Hubert Plociniczak
0b58a361ed
Eliminate VCS TimeoutExceptions on startup (#8080)
Having a modest-size files in a project would lead to a timeout when the project was first initialized. This became apparent when testing delivered `.enso-project` files with some data files. After some digging there was a bug in JGit
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=494323) which meant that adding such files was really slow. The implemented fix is not on by default but even with `--renormalization` turned off I did not see improvement.
In the end it didn't make sense to add `data` directory to our version control, or any other files than those in `src` or some meta files in `.enso`. Not including such files eliminates first-use initialization problems.

# Important Notes
To test, pick an existing Enso project with some data files in it (> 100MB) and remove `.enso/.vcs` directory. Previously it would timeout on first try (and work in successive runs). Now it works even on the first try.

The crash:
```
[org.enso.languageserver.requesthandler.vcs.InitVcsHandler] Initialize project request [Number(2)] for [f9a7cd0d-529c-4e1d-a4fa-9dfe2ed79008] failed with: null.
java.util.concurrent.TimeoutException: null
at org.enso.languageserver.effect.ZioExec$.<clinit>(Exec.scala:134)
at org.enso.languageserver.effect.ZioExec.$anonfun$exec$3(Exec.scala:60)
at org.enso.languageserver.effect.ZioExec.$anonfun$exec$3$adapted(Exec.scala:60)
at zio.ZIO.$anonfun$foldCause$4(ZIO.scala:683)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:904)
at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
```
2023-10-18 09:34:08 +00:00
Jaroslav Tulach
a85a57681d
Avoid null values in constant node (#8061)
Prevents [this NullPointerException](https://github.com/enso-org/enso/pull/8044#issuecomment-1763736570) by making sure `ConstantNode` rejects `null` during construction.

# Important Notes
```ruby
from Standard.Base import all
polyglot java import java.util.Random

main =
operator1 = Random.new
```
2023-10-16 15:50:41 +00:00
Hubert Plociniczak
352ad06d2f
Reduce extra output in compilation and tests (#7809)
* Reduce extra output in compilation and tests

I couldn't stand the amount of extra output that we got when compiling
a clean project and when executing regular tests. We should strive to
keep output clean and not print anything additional to stdout/stderr.

* Getting rid of explicit setup by service loading

In order for SL4J to use service loading correctly had to upgrade to
latest slf4j. Unfortunately `TestLogProvider` which essentially
delegates to `logback` provider will lead to spurious ambiguous warnings
on multiple providers. In order to dictate which one to use and
therefore eliminate the warnings we can use the `slf4j.provider` env
var, which is only available in slf4j 2.x.

Now, there is no need to explicitly call `LoggerSetup.get().setup()` as
that is being called during service setup.

* legal review

* linter

* Ensure ConsoleHandler uses the default level

ConsoleHandler's constructor uses `Level.INFO` which is unnecessary for
tests.

* report warnings
2023-10-16 10:57:44 +02:00
Dmitry Bushev
58eeba6a00
Suppress futile visualization warnings (#8056)
close #7819

Warnings
```
Execution of visualization [...] on value [...] of [Panic] failed.
```
are produced when you are typing an incomplete expression in the component browser and can be misleading.
2023-10-16 08:23:43 +00:00