Commit Graph

1062 Commits

Author SHA1 Message Date
Jaroslav Tulach
2aeef4d844
Automatically apply from conversion when runtime argument check fails (#7009) 2023-07-11 07:23:25 +01:00
Dmitry Bushev
4864e5b51d
Add fileModifiedOnDisk notification (#7239)
part of #7178

Changelog:
- add: `text/fileModifiedOnDisk` notification
- update: during the auto-save, check if the file is modified on disk and send the notification. I.e. auto-save does not overwrite the file if it was changed on disk (but the save command does)
- update: IDE handles the file-modified-on-disk notification and reloads the module from disk

# Important Notes
Currently, the auto-save (and the check that the file is modified on disk) is triggered only after the file was edited. The proper check (using the file-watcher service) will be added in the next PR

https://github.com/enso-org/enso/assets/357683/ff91f3e6-2f7a-4c01-a745-98cb140e1964
2023-07-10 16:50:14 +00:00
Hubert Plociniczak
9dcf48a3e0
Cache dataflow errors (#7193)
The logic in instrumentation specifically prohibited caching of panics (added in https://github.com/enso-org/enso/pull/1611).
Except it seems that the restriction was too aggressive because dataflow errors could be cached.
This indirectly fixes the problem with dataflow error in the context of a (temporary) change in execution context, which should keep around the old value.

Closes #7140.

After the fix:
[fix_7140.webm](https://github.com/enso-org/enso/assets/292128/b8d522db-985a-4e57-bbf1-a075d14d39a2)
2023-07-09 14:45:22 +00:00
Hubert Plociniczak
dd06c59380
Initialize Language Server's JSON RPC protocol asynchronously (#7232)
As discovered in #7224, Json RPC protocol was added to the asynchronous resource initialization stage, as part of #6306, but was not in fact initialized at that point.
Instead it was initialized when the server was started to be able to serve correctly the initialization messages. A classic Catch-22. It was really hard to discover this just by looking at the code, but the profiling clearly showed where the time was spent.

This change splits Language Server's protocol into two:
- the first one accepts `heartbeat/init` and `session/initProtocolConnection`
- the second one enriches it with the full set of supported messages

This shifts the initialization from blocking for 0.5 sec to only ~30ms, and performing the second stage asynchronously.

Closes #7224.

# Important Notes
Before the change (blocking server startup):
![Screenshot from 2023-07-05 18-53-24](https://github.com/enso-org/enso/assets/292128/bcfa9043-d00a-4b36-a44c-782a388a16b9)
![Screenshot from 2023-07-05 18-53-10](https://github.com/enso-org/enso/assets/292128/54927787-4c95-46db-bd68-f3a3b82367d5)

After the change (1st stage):
![Screenshot from 2023-07-06 14-02-34](https://github.com/enso-org/enso/assets/292128/d7a7bc34-39dc-46f1-9e64-6d350697c30b)

After the change (2nd, asynchronous initialization, stage):
![Screenshot from 2023-07-06 14-21-17](https://github.com/enso-org/enso/assets/292128/def8c0a1-f211-4fc0-9df0-7c1634312166)
2023-07-07 11:49:23 +00:00
Hubert Plociniczak
a140a04689
A dataflow error with warnings reports the former (#7205)
A dataflow error resulting from calling a value with warnings now
reports only the error rather than the warning.

Closes #7141.
2023-07-05 14:18:36 +02:00
James Dunkerley
4fbe7e3830
Remove Array.new and Array.copy and move Vector functions to builtins. (#7147)
- Removed Array methods: `new`, `copy` and `new_[1234]`.
- New builtins for `Vector.insert`, `Vector.remove` and `Vector.flatten`.
- Replaced `Vector_Builder` use of `Array.copy` to a `Vector.Builder` approach.
2023-07-03 12:41:41 +00:00
Hubert Plociniczak
3c93c25a5a
Reload project's config on rename (#7179)
Package's config information, once loaded, never changed. While there is typically no need for it, this was problematic when the config became out-of-sync with the filesystem, like in the case of project rename action.
In rename, the config's properties would be updated in the FS, but that would never be reflected in module's package. Therefore further compilations would continue to ask for the old namespace.

Most of the changes are cosmetic (s/`.config`/`.getConfig()`) except for the new `reloadConfig` method on `Package` that is being called in `RenameProjectCmd` handler.

Closes #7062.

# Important Notes
The reported `ExecutionFailed` error should have been mostly fixed already via #7143. This change makes sure that all the related warnings are gone as well and the compiler uses the updated namespace.
2023-07-03 09:36:19 +00:00
Dmitry Bushev
66894bd79e
Send info about function values (#7168)
close #6957

Extend `ExpressionUpdate` message and send a function schema if the returned value is a function.
2023-06-30 13:31:48 +00:00
Pavel Marek
ebee8700ce
HashMapNode supports atoms with custom comparators (#7165)
Add proper handling for atoms with custom comparators into the hashing machinery.
2023-06-30 08:57:36 +00:00
Hubert Plociniczak
bf5ddf3de0
Send executionComplete response only on success (#7143)
`executionFailed` instead is sent when an evaulation finishes with a a critical failure or a non-critical error.
The PR tries to miniminally modify the change in the messages exchange so as to avoid a major redesign at this point.

Closes #7002.

# Important Notes
Unblocks IDE which will need to modify to this new setup.
2023-06-29 07:35:28 +00:00
Jaroslav Tulach
91f661fd8b
Documenting how to debug project-manager project (#7153)
```
sbt:project-manager> withDebug run --debugger
```
2023-06-29 05:55:14 +00:00
Pavel Marek
60ef7e0ba3
Referencing self from static method raises compiler error (#7131)
Add a new `NoStaticInSelf` compiler pass that ensures that `self` keyword is used only from instance methods.
2023-06-28 19:04:32 +02:00
James Dunkerley
56688ec1e7
Minor fixes. (#7122)
Mostly stuff to tidy up the static methods in the CB.

- Remove default pattern from `parse_to_table` (caused IDE to freeze).
- Rename any `_` arguments to what they are.
- Merge `Date.now` into `Date.today`
- Merge the Interval constructors into a single constructor.
- Hide various methods.
2023-06-27 18:18:15 +00:00
Jaroslav Tulach
477dd82670
Drop-down widgets for extension functions via UnresolvedSymbol (#7115)
Fixes #6955 by:
- using `visualisationModule` to specify the module where the visualization is to be used
- referring to method in `Meta.get_annotation` with `.method_name` - e.g. unresolved symbol notation
- evaluating arguments to `Meta.get_annotation` in the context of the user module (which can access the extension functions)
2023-06-27 15:19:42 +00:00
Dmitry Bushev
22259e696d
Add method call info for infix operators (#7090)
close #6374

In order to provide the method pointer information, the `IrToTruffle` pass sets the module name and the type name of the builtin node.
2023-06-27 13:11:12 +00:00
Hubert Plociniczak
4e5cb9cca6
Consistent naming of visualization (#7124) 2023-06-26 17:04:35 +00:00
Hubert Plociniczak
ae4666c4d3
Improve TCO in the presence of warnings (#7116)
Partially revert https://github.com/enso-org/enso/pull/6849, which introduced a regression in TCO in the presence of warnings. Rather than modifying the tail call status, `TailCallException` now propagates the extracted warnings and appends them to the final result.

Closes #7093

# Important Notes
Compared to the previous attempt we don't pay the penalty of adding the warnings or even checking for them because it is being dealt in a separate specialization.
2023-06-26 12:38:36 +00:00
Jaroslav Tulach
c4f19e7d66
Vector.map & State (#7078) 2023-06-26 13:13:41 +02:00
Jaroslav Tulach
6abcbcb00c
Moving construction of IR into runtime parser project (#7096) 2023-06-24 06:34:21 +02:00
Hubert Plociniczak
6c777834e4
Don't instrument function bodies (#7097)
Function bodies cannot be instrumented even if the function is right
inside a binding. Consider a scenario when a function is assigned to a
variable and then applied to a `map` method of a really large vector.
The instrumentation will render execution extremely slow.

Alternatively we would still support instrumenting function bodies in
this limited case but take into account the number of times function is
actually called.
2023-06-23 19:16:25 +02:00
Pavel Marek
31aad1d9e6
Always call instance methods on Any (#7033)
Previously, static method calls on `Any` have not worked as expected. For example, `Any.to_text` returned Function instead of Text. That is because the function resolution for `Any.to_text` finds `Any.type.to_text` method on eigentype which expects two `self` arguments, but only one argument is provided.

Note that `Boolean.to_text` worked previously, and returned "Boolean" as expected. This is because the method resolution finds `Any.to_text` method that takes just one `self` argument.

This PR solves this issue by introducing special handling for static method dispatch on `Any`. Simply put, an additional `self` argument is prepended to the argument list.

# Important Notes
A new child node is introduced to `InvokeMethodNode`. This child node is a copy of the current `invokeFunctionNode` with one more `CallArgumentInfo` in its schema.
2023-06-23 14:46:42 +00:00
Pavel Marek
bbd9f437c5
Empty input in REPL does not throw NPE (#7111)
Fix for an annoying NullPointerException thrown when an empty line is fed to REPL, i.e., when one just presses Enter. Now, you can press Enter, creating as many blank prompts, as you want:
![image](https://github.com/enso-org/enso/assets/14013887/8f38e210-da5c-46a0-b5f1-1a5da9e62584)
2023-06-23 12:05:44 +00:00
Dmitry Bushev
3937d21001
Do not invalidate caches when building engine distribution (#7106)
There is an issue that after a clean build, the compiler is unable to resolve some types, i.e
```py
$ ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run ~/enso/projects/Unnamed/

/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:24:11: error: The name `Connection_Details` could not be found.
24 | connect : Connection_Details -> Connection_Options -> Connection ! SQL_Error
|           ^~~~~~~~~~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:24:33: error: The name `Connection_Options` could not be found.
24 | connect : Connection_Details -> Connection_Options -> Connection ! SQL_Error
|                                 ^~~~~~~~~~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:24:55: error: The name `Connection` could not be found.
24 | connect : Connection_Details -> Connection_Options -> Connection ! SQL_Error
|                                                       ^~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:24:68: error: The name `SQL_Error` could not be found.
24 | connect : Connection_Details -> Connection_Options -> Connection ! SQL_Error
|                                                                    ^~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:25:25: error: The name `Connection_Options` could not be found.
25 | connect details options=Connection_Options.Value =
|                         ^~~~~~~~~~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:29:29: error: The name `Widget` could not be found.
29 | connection_details_widget : Widget
|                             ^~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:31:28: error: The name `Vector` could not be found.
31 |     default_constructors = Vector.from_polyglot_array <|
|                            ^~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:32:63: error: The name `False` could not be found.
32 |         DatabaseConnectionDetailsSPI.get_default_constructors False
|                                                               ^~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:36:9: error: The name `Option` could not be found.
36 |         Option name code
|         ^~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:37:5: error: The name `Single_Choice` could not be found.
37 |     Single_Choice display=Display.Always values=choices
|     ^~~~~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:37:27: error: The name `Display` could not be found.
37 |     Single_Choice display=Display.Always values=choices
|                           ^~~~~~~
Aborting due to 11 errors and 0 warnings.
Execution finished with an error: Compilation aborted due to errors.
```

The compiler can't resolve those symbols because the `IR` cache for `Standard.Database.Connection.Database` is missing. What happens during the `buildEngineDistribution` is:

- Compiler processes libraries one by one
- Compiler processes (and compiles and generates caches) `Standard.Database` library
- Compiler starts processing `Standard.Visualization` library
- During the compilation of `Standard.Database.Connection.Database` module, it sees that the module was loaded from the cache. But some of the required dependencies from `Standard.Table` library were not loaded from the cache. But at this point, the `Standard.Table` library has not been processed yet and the caches for it don't exist. The compiler decides that the `Database` file was changed, and the cache is invalid and should be cleaned.
2023-06-22 14:04:40 +00:00
James Dunkerley
90df5adfb0
Fix defect where constructor fields not show for Value constructors. (#7080)
The `IdExecutionService` now uses `getName` not `getDisplayName` for atomConstructor.
`getDisplayName` adjusts the name if the name is `Value` or `Error`, which meant no parameters in IDE.

Before:
![image](https://github.com/enso-org/enso/assets/4699705/ddc564fe-3213-46ec-9684-ab718f8bb0bb)

Now:
![image](https://github.com/enso-org/enso/assets/4699705/59d3ece4-1148-44bb-8a13-7bc3db8dfca2)
2023-06-20 12:47:16 +00:00
James Dunkerley
1859ccbab5
Improving widgets and other minor tweaks. (#7052)
- Removed `module` argument from `enso_project` (new `Project_Description.new` API).
- Removed the custom option from date and time parse/format dropdowns.
- The `format` dropdown uses the value to create the dropdown. (Screenshot below)
- Removed `StorageType` coalescing rules and replaced them with simpler logic in `ObjectStorage`.
- Update signature for `add_row_number` and add aliases.
2023-06-19 19:03:36 +00:00
Dmitry Bushev
8b65509415
Do not add self arguments of static methods to suggestions database (#7074)
The `self` argument of static methods should not be visible to the user.

# Important Notes
![2023-06-19-190618_1243x459_scrot](https://github.com/enso-org/enso/assets/357683/f9b32911-bf8a-4cbc-ab7a-61beaae3c98b)
2023-06-19 17:48:48 +00:00
Jaroslav Tulach
5f572d0680
Avoid dependency on BindingsMap from IR (#7055) 2023-06-19 10:28:31 +02:00
Jaroslav Tulach
7d0461cb6e
Benchmarking Enso version of the Sieve of Eratosthenes algorithm (#7019) 2023-06-19 09:57:42 +02:00
Marcin Kostrzewa
90e413c7db
AI Completions (#5910) 2023-06-18 23:54:06 +02:00
Jaroslav Tulach
6c2ecc615e
Never return null from InteropLibrary calls (#7044)
Preventing `NullPointerException` from debugger and GraalVM Insight.
2023-06-17 04:30:03 +00:00
Hubert Plociniczak
e529f7bb3b
Improvements that significantly reduce the chances of request timeouts (#7042)
Request Timeouts started plaguing IDE due to numerous `executionContext/***Visualization` requests. While caused by a bug they revealed a bigger problem in the Language Server when serving large amounts of requests:
1) Long and short lived jobs are fighting for various locks. Lock contention leads to some jobs waiting for a longer than desired leading to unexpected request timeouts. Increasing timeout value is just delaying the problem.
2) Requests coming from IDE are served almost instantly and handled by various commands. Commands can issue further jobs that serve request. We apparently have and always had a single-thread thread pool for serving such jobs, leading to immediate thread starvation.

Both reasons increase the chances of Request Timeouts when dealing with a large number of requests. For 2) I noticed that while we used to set the `enso-runtime-server.jobParallelism` option descriptor key to some machine-dependent value (most likely > 1), the value set would **only** be available for instrumentation. `JobExecutionEngine` where it is actually used would always get the default, i.e. a single-threaded ThreadPool. This means that this option descriptor was simply misused since its introduction. Moved that option to runtime options so that it can be set and retrieved during normal operation.

Adding parallelism intensified problem 1), because now we could execute multiple jobs and they would compete for resources. It also revealed a scenario for a yet another deadlock scenario, due to invalid order of lock acquisition. See `ExecuteJob` vs `UpsertVisualisationJob` order for details.

Still, a number of requests would continue to randomly timeout due to lock contention. It became apparent that
`Attach/Modify/Detach-VisualisationCmd` should not wait until a triggered `UpsertVisualisationJob` sends a response to the client; long and short lived jobs will always compete for resources and we cannot guarantee that they will not timeout that way. That is why the response is sent immediately from the command handler and not from the job executed after it.

This brings another problematic scenario:
1. `AttachVisualisationCmd` is executed, response sent to the client, `UpsertVisualisationJob` scheduled.
2. In the meantime `ModifyVisualisationCmd` comes and fails; command cannot find the visualization that will only be added by `UpsertVisualisationJob`, which might have not yet been scheduled to run.

Remedied that by checking visualisation-related jobs that are still in progress. It also allowed for cancelling jobs which results wouldn't be used anyway (`ModifyVisualisationCmd` sends its own `UpsertVisualisationJob`). This is not a theoretical scenario, it happened frequently on IDE startup.

This change does not fully solve the rather problematic setup of numerous locks, which are requested by short and long lived jobs. A better design should still be investigated. But it significantly reduces the chances of Request Timeouts which IDE had to deal with.

With this change I haven't been able to experience Request Timeouts for relatively modest projects anymore.

I added the possibility of logging wait times for locks to better investigate further problems.

Closes #7005
2023-06-16 17:57:16 +00:00
James Dunkerley
760fb71798
First part of AWS S3 API, various small fixes. (#6973)
- Add type detection for `Mixed` columns when calling column functions.
- Excel uses column name for missing headers.
- Add aliases for parse functions on text.
- Adjust `Date`, `Time_Of_Day` and `Date_Time` parse functions to not take `Nothing` anymore and provide dropdowns.
- Removed built-in parses.
- All support Locale.
- Add support for missing day or year for parsing a Date.
- All will trim values automatically.
- Added ability to list AWS profiles.
- Added ability to list S3 buckets.
- Workaround for Table.aggregate so default item added works.
2023-06-15 16:20:13 +00:00
Dmitry Bushev
48f0c6f5e8
Scala 2.13.11 and libraries update (#7010)
Update Scala and libraries.
2023-06-14 13:15:57 +00:00
Pavel Marek
67821bf8df
Add compiler pass that discovers ambiguous imports (#6868)
Add a new compiler pass that analyses duplicated and ambiguous symbols from imports
2023-06-14 12:18:57 +02:00
Dmitry Bushev
6249c79ffd
Update sbt-java-formatter plugin (#7011)
Update java formatter plugin. The new version can remove unused imports.
2023-06-12 14:18:48 +00:00
Dmitry Bushev
6de2700209
Do not return getters from global completion response (#7007)
close #6936

Changelog:
- add: new suggestion type Getter that is not exposed to the api
- update: do not return suggestion of type getter when doing a global search (without specifying self types)

Private suggestions and modules mentioned in the issue will be filtered out after we finish the work on the new (refined) exports algorithm.

# Important Notes
![2023-06-09-205327_1088x612_scrot](https://github.com/enso-org/enso/assets/357683/c6b16894-ada0-4ea9-abe8-5efc41949787)
2023-06-12 12:28:35 +00:00
Dmitry Bushev
b2c1f97437
Test case with a function read from a variable (#6986)
Test checking that if the function is read from variable, it does not have the method pointer associated with it.
2023-06-09 21:53:02 +00:00
Hubert Plociniczak
1ce3505e5b
Fix potential NoSuchElementException (#6948)
As demonstrated in https://github.com/enso-org/enso/actions/runs/5175688022/jobs/9323585204?pr=6940

```
org.enso.interpreter.test.instrument.RuntimeAsyncCommandsTest *** ABORTED ***
org.graalvm.polyglot.PolyglotException: java.util.NoSuchElementException
at java.base/java.util.WeakHashMap$HashIterator.nextEntry(WeakHashMap.java:811)
at java.base/java.util.WeakHashMap$EntryIterator.next(WeakHashMap.java:848)
at java.base/java.util.WeakHashMap$EntryIterator.next(WeakHashMap.java:846)
at org.enso.interpreter.runtime.ThreadExecutors.shutdown(ThreadExecutors.java:46)
at org.enso.interpreter.runtime.EnsoContext.shutdown(EnsoContext.java:198)
at org.enso.interpreter.EnsoLanguage.finalizeContext(EnsoLanguage.java:179)
at org.enso.interpreter.EnsoLanguage.finalizeContext(EnsoLanguage.java:65)
at org.graalvm.truffle/com.oracle.truffle.api.LanguageAccessor$LanguageImpl.finalizeContext(LanguageAccessor.java:326)
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotLanguageContext.finalizeContext(PolyglotLanguageContext.java:404)
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextImpl.finalizeContext(PolyglotContextImpl.java:2925)
```
2023-06-09 16:28:03 +00:00
Hubert Plociniczak
5c3d6a7d6f
Eliminate race condition for edit/close/open cmds (#6998)
There was an inherent race condition between edit, close & open commands which could not be prevented solely using locks. `EditFileCmd` triggered `EnsureCompiledJob` which was applying edits collected over time. At the same `CloseFileCmd` and `OpenFileCmd` were executed asynchronously and required locks on compilation unit  and file lock.

Additionally, open file was resetting the module's runtime source irrespective of any edits that could already have been applied with the asynchronous execution in `EnsureCompiledJob`. This was visible especially during early manipulation of the project when open/close was performed due to a bug in IDE (#6843).

Now commands can be run either synchronously or asynchronously. Only that way can we ensure that `close` & `open` commands finish by the time any editions are being applied to module's sources.

Closes #6841.

# Important Notes
In the given video, `"foo"` would be greyed out because it would never be part of the module's (runtime) sources. Therefore no IR would be generated for it or instrumentation, meaning it would be present in `expressionUpdates` information necessary for IDE.

[Kazam_screencast_00014.webm](https://github.com/enso-org/enso/assets/292128/226a17b8-729a-415a-803f-003a9695b2f1)
2023-06-09 12:12:46 +00:00
Pavel Marek
372bc8f0d5
Improve compiler's diagnostic messages (#6931)
Improve and colorize compiler's messages. Heavily inspired by `gcc`.
2023-06-08 17:43:18 +02:00
Jaroslav Tulach
9f39fb30b7
Make sure Enso FrameDescriptors have non-null default value (#6977) 2023-06-08 16:55:40 +02:00
Jaroslav Tulach
e6652b8c9f
Separating instrument related files into runtime-instrument-common project (#6992) 2023-06-08 16:51:50 +02:00
Jaroslav Tulach
3731c9c798
Dates are dates and zones are zones (#6934) 2023-06-08 09:47:44 +02:00
Dmitry Bushev
0a4df4faf3
Skip internal field getters when building suggestions (#6968)
close #6900

This is a follow-up to the discussion of the imports/exports meeting.

Right now we have no control over the visibility of atom constructor arguments. One way to hide them is a convention of filtering getters by an `internal` prefix or suffix.
2023-06-07 12:32:58 +00:00
Radosław Waśko
1931e9e51f
Workaround for to_date_time type errors (#6964)
Related to #6912

It essentially solves it by removing any builtins that would take an EnsoDate/EnsoTimeOfDay/EnsoTimeZone and replacing them with Java utils that do the same operation.

This is not a proper solution - the builtin conversion is still invalid for the date/time types - but at this moment we may just no longer use the invalid conversion so it is much less of an issue. We still need to be aware of this if we want to introduce builtins taking date/time in the future.
2023-06-06 20:28:11 +00:00
Hubert Plociniczak
27feaf6bc5
Eliminate deadlock scenario during file edit/open/close requests (#6920)
At the beginning of the execution `EnsureCompiledJob` acquired write compilation lock. When compiling individual modules it would then
- acquire file lock
- acquire read compilation lock

The second one was spurious since it already kept the write lock. This sequence meant however that `CloseFileCmd` or `OpenFileCmd` can lead to  a deadlock when requests come in close succession. This is because commands:
- acquire file lock
- acquire read compilation lock

So `EnsureCompiledJob` might have the (write) compilation lock but the commands could have file lock. And the second required lock for either the job or the command could never be acquired.

Flipping the order did the trick.

Partially solves #6841.

# Important Notes
For some reason we don't get updates for the newly added node, as illustrated in the screenshot, but that could be related to the close/open action. Will need to dig more.

![Screenshot from 2023-06-01 16-45-17](https://github.com/enso-org/enso/assets/292128/900aa9b3-b2b2-4e4d-93c8-267f92b79352)
2023-06-06 08:05:24 +00:00
Hubert Plociniczak
e9a92a1fb5
Throw panic on "no currying for conversions" (#6940)
Previously, a `RuntimeException` would be thrown when an attempt would be made to curry a conversion function. That is problematic for IDE where `executionFailed` means we can't enter functions due to lack of method pointers info.

Closes #6897.

![Screenshot from 2023-06-02 20-31-03](https://github.com/enso-org/enso/assets/292128/a6c77544-2c47-425c-8ce0-982d837dda5b)

# Important Notes
A more generic solution that allows to recover from execution failures will need a follow up.
2023-06-05 13:16:12 +00:00
Dmitry Bushev
4e9f02258e
Suggestion should contain a list of annotations (#6924)
close #6847

Add annotation names to suggestions.
2023-06-02 07:26:01 +00:00
Jaroslav Tulach
d511602787
Don't swallow exceptions in RuntimeServerTest (#6914) 2023-06-01 14:26:36 +02:00
Dmitry Bushev
ed3f9b306e
Consistent self types (#6867)
close #6800

Update the `executionContext/expressionUpdates` notification and send the list of not applied arguments in addition to the method pointer.

# Important Notes
IDE is updated to support the new API.
2023-05-31 15:47:48 +00:00
Jaroslav Tulach
bc6b9bcf54
Search for IR.Function deeper in the IR tree to find instance methods (#6902) 2023-05-31 10:51:55 +02:00
Pavel Marek
e5c21713e7
Import polyglot java inner classes (#6818)
Adds the ability to import nested inner classes in polyglot java imports.
2023-05-31 09:38:59 +02:00
Jaroslav Tulach
c6cb937c0a
Runtime check of ascribed types (#6790) 2023-05-30 11:07:26 +02:00
Hubert Plociniczak
6eb4737330
Append warnings extracted before tail call execution (#6849)
Throwing `TailCallException` meant that exceptions that were extracted from the expression before the call was made could not be appended. This change catches the `TailCallException`, adds warnings to it and propagates it further, thus ensuring that we don't loose the information.

Closes #6765.

# Important Notes
Removed workarounds introduced in stdlib.
2023-05-29 12:44:15 +00:00
Jaroslav Tulach
e7ee2caa2e
Infrastructure for testing inter project imports and exports (#6840) 2023-05-26 17:33:54 +02:00
Jaroslav Tulach
b33ae479dd
Various test improvements to increase coverage and speed things up (#6820) 2023-05-25 10:18:46 +02:00
Jaroslav Tulach
792cbc448c
Clearly select single specialization with enum dispatch pattern (#6819) 2023-05-24 20:42:24 +02:00
Pavel Marek
fe0a06dcb5
Import/export syntax error have more specific messages (#6808) 2023-05-23 21:10:18 +02:00
Jaroslav Tulach
a49743f00c
Missing conversion of hash key in EqualsNode (#6803) 2023-05-23 10:10:22 +02:00
Dmitry Bushev
7471e38ff6
feat: set constructor args tag values (#6801)
related #6611

Changelog:
- update: set tag values for arguments with ascribed types

# Important Notes
![2023-05-23-002112_697x311_scrot](https://github.com/enso-org/enso/assets/357683/10d1c03b-45b7-4133-b4aa-80d1f1ea8e84)
2023-05-23 08:07:40 +00:00
Pavel Marek
2b1e5cd017
Add a compiler pass to analyze non-existing imported symbols (#6726)
Add diagnosis for unresolved symbols in `from ... import sym1, sym2, ...` statements.

- Adds a new compiler pass, `ImportSymbolAnalysis`, that checks these statements and iterates through the symbols and checks if all the symbols can be resolved.
  - Works with `BindingsMap` metadata.
- Add `ImportExportTest` that creates various modules with various imports/exports and checks their generated `BindingMap`.

---------

Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
2023-05-22 10:41:15 +02:00
GregoryTravis
4f71673718
Decimal/Integer .round and .int #6654 (#6743) 2023-05-19 19:23:17 +00:00
Dmitry Bushev
9ec7415ded
Set suggestion reexports when serializing the library (#6778)
close #6613

Changelog
- feat: during the library serialization, build the exports map and set the reexport field of the suggestion

# Important Notes
IDE does not create additional imports for re-exported symbols.

![2023-05-18-192739_2019x828_scrot](https://github.com/enso-org/enso/assets/357683/5ef20dfe-d6a5-4935-a759-4af10b0817a5)
2023-05-19 17:35:27 +00:00
Jaroslav Tulach
995b3c9f42
Using WarningsLibrary to query for warnings (#6751)
Fixes #6616 by using `WarningsLibrary`.
2023-05-19 13:39:14 +00:00
Radosław Waśko
447786a304
Implement cast for Table and Column (#6711)
Closes #6112
2023-05-19 10:00:20 +00:00
Hubert Plociniczak
a32a2eafba
Only send suggestions updates when type changes (#6755)
The change adds an additional field to `ExpressionUpdates` messages sent by `ProgramExecutionSupport` to indicate if the type of value (or its method pointer) has changed and therefore would potentially require a suggestions' update.

Prior to #3729 that check was done during the instrumentation. However we still want to continue to support "pending expression" functionality therefore `SuggestionsHandler` will use the additional information to filter only the required expression updates.

Most of the changes are related to adapting our tests to the new field.

Closes #6706.

# Important Notes
The associated project now loads and navigates smoothly.
Also attaching a screenshot from the project that illustrates that pending functionality continues to work:
[Kazam_screencast_00006.webm](https://github.com/enso-org/enso/assets/292128/35918841-f84f-4e1c-b1b0-40e45d97e111)
2023-05-18 16:44:38 +00:00
Jaroslav Tulach
8c370008bc
Meta.meta Integer . methods (#6740) 2023-05-18 11:40:44 +02:00
Dmitry Bushev
b7d51ed5c6
Create unique atom getter suggestions (#6694)
related #6611

Prevent creating atom getters with the same name.
2023-05-15 14:03:18 +00:00
Jaroslav Tulach
41bb52c901
Verify ascribed types of parameters really exist (#6584)
Verify ascribed types `(a : Xyz)` are checked for existence.
2023-05-14 07:23:18 +00:00
Dmitry Bushev
706791779b
SuggestionBuilder needs to send ascribedType of constructor parameters (#6655)
close #6611

Changelog:
- update: run compiler passes on the `ascribedType` field of the constructor arguments
- update: suggestion builder uses the type information attached to `ascribedType`
- feat: resolve qualified names in type signatures
2023-05-13 18:33:03 +00:00
Radosław Waśko
f5071a17fd
Improving widgets for take/drop (#6641)
Related to #6410

# Important Notes
- Updated some `Meta` methods (needed for error handling):
- `Meta.Type` now has `name` and `qualified_name`.
- `Meta.Constructor` has `declaring_type` allowing to get the type that this constructor is associated with.
2023-05-12 15:33:15 +00:00
Jaroslav Tulach
1302b693d8
All Vector operations shall be applicable on java.util.ArrayList (#6642)
Fixes #6609 by
- e380e647af - running whole `Vector_Spec` on `java.util.ArrayList`
- 9b1229fe20 - introducing a node to handle interop values

# Important Notes
Contains additional DSL processor fix:
- 415623dcb9 - to not crash the compiler, but to properly report compiler error
2023-05-11 15:29:47 +00:00
Dmitry Bushev
3dd05c2359
Automatic type based dropdown does not include singleton in a union type (#6629)
close #6532

Set tag values to display `Auto` type in dropdowns correctly.

# Important Notes
![2023-05-10-141458_969x566_scrot](https://github.com/enso-org/enso/assets/357683/9a048b3c-d192-4382-bf76-9cbe6c9556d1)
![2023-05-10-141513_991x232_scrot](https://github.com/enso-org/enso/assets/357683/c50e1e73-23a6-4b32-90bf-f849a127c85d)
2023-05-10 16:55:16 +00:00
Jaroslav Tulach
4e7191af46
Make Meta.get_annotation work for constructor (#6633)
Fixes #6612 by searching for the atom constructor on the `target` itself, not its type.
2023-05-10 15:42:28 +00:00
Hubert Plociniczak
cf3624c463
Limit the number of reported warnings (#6577)
Artifically limiting the number of reported warnings to 100. Also added benchmarks with random Ints to investigate perf issues when dealing with warnings (future task).
Ideally we would have a custom set-like collection that allows us internally to specify a maximal number of elements. But `EnsoHashMap` (and potentially `EnsoSet`) are still WIP when it comes to being PE-friendly.

The change also allows for checking if the limit for the number of reported warnings has been reached. It will visualize by adding an additional "Warnings limit reached." to the visualization.

The limit is configurable via `--warnings-limit` parameter to `run`.

Closes #6283.
2023-05-10 11:48:31 +00:00
GregoryTravis
4ba8409def
Add format to the in-memory Column (#6538)
Add format to the in-memory Column

# Important Notes
Also updates .format in date types.
Some rearrangement of date formatting builtins / Java libraries.
2023-05-09 08:47:40 +00:00
Pavel Marek
608c5d8945
Benchmark Engine job runs only engine, not Enso benchmarks (#6534)
Engine Benchmark job runs only engine benchmarks, not Enso benchmarks.

Enso benchmarks do not report their output anywhere, and take more than 5 hours to run nowadays.
We might define a new job in the future and probably rename it to "Library benchmarks".
But that is the responsibility of the lib team.
2023-05-05 21:24:19 +02:00
Jaroslav Tulach
b5578ec2c9
Identify SyntaxError exception and avoid printing a stack trace (#6574) 2023-05-05 15:30:06 +02:00
Jaroslav Tulach
a842130ff3
Let ChangesetBuilder.invalidated search even container elements (#6548) 2023-05-05 11:46:00 +02:00
Hubert Plociniczak
6527312486
Cache result of slow function resolution on Any that is present on a hot path (#6536)
Otherwise things can go horribly slow.
Closes #6523. Follow up on https://github.com/enso-org/enso/pull/6441.
2023-05-04 12:54:28 +00:00
Hubert Plociniczak
4097800f7a
Infer correct synthetic name for nested modules (#6525)
Deeply nested modules (of depth at least 3) would have the incorrect name inferred for synthetic modules. This also became apparent in a much bigger rewrite.

Instead of having module `A.B.C`, as described in the test, it would infer the name to be `B.A.C`, which would break when trying to reference symbols from `C`.

# Important Notes
No ticket reference, as this was something that @radeusgd discovered while working on another component.
The PR includes a minimal example that would previously fail to compile.
2023-05-04 08:05:28 +00:00
Hubert Plociniczak
b93edb050b
Force pending saves if client closes abruptly (#6514)
Force pending saves when JsonSession is terminated for the client.

Potentially closes #6395.
2023-05-03 23:30:20 +00:00
Dmitry Bushev
38212786a3
Fix application config (#6513)
Related #6482
2023-05-02 17:59:44 +00:00
Pavel Marek
3a42d0ce76
Reimplement enso_project as a proper builtin (#6352)
Remove the magical code generation of `enso_project` method from codegen phase and reimplement it as a proper builtin method.

The old behavior of `enso_project` was special, and violated the language semantics (regarding the `self` argument):
- It was implicitly declared in every module, so it could be called without a self argument.
- It can be called with explicit module as self argument, e.g. `Base.enso_project`, or `Visualizations.enso_project`.

Let's avoid implicit methods on modules and let's be explicit. Let's reimplement the `enso_project` as a builtin method. To comply with the language semantics, we will have to change the signature a bit:
- `enso_project` is a static method in the `Standard.Base.Meta.Enso_Project` module.
- It takes an optional `project` argument (instead of taking it as an explicit self argument).

Having the `enso_project` defined as a (shadowed) builtin method, we will automatically have suggestions created for it.

# Important Notes
- Truffle nodes are no longer generated in codegen phase for the `enso_project` method. It is a standard builtin now.
- The minimal import to use `enso_project` is now `from Standard.Base.Meta.Enso_Project import enso_project`.
- Tested implicitly by `org.enso.compiler.ExecCompilerTest#testInvalidEnsoProjectRef`.
2023-05-02 16:40:58 +00:00
Dmitry Bushev
84e59f1403
DataflowAnalysis preserves dependencies order (#6493)
close #6324

Changelog
- feat: DataflowAnalysis compiler pass preserves the order of dependencies. This way when attaching the visualization to the sub-expression, the engine can find the first cached parent node, and properly invalidate it.
- update: runtime visualization test is updated to reproduce the issue

# Important Notes
The dropdown for the column `"LOCATION"` is available right after the Restaurants project startup.

![2023-05-01-171700_1386x975_scrot](https://user-images.githubusercontent.com/357683/235466166-9d25cfa5-0e39-49a3-9c41-93cda59edb81.png)
2023-05-02 14:22:06 +00:00
Hubert Plociniczak
ce4ecc6706
Limit Dead Letter logging (#6482)
Dead Letter logging is occasionally flooding our logs which is confusing to users reporting bugs. Left the possibility of a single report so that we know that something is happening.
2023-05-02 14:00:50 +00:00
Jaroslav Tulach
c626543d02
More reliable shutdown of the EnsoContext to save resources (#6468)
There is 572 threads during `RuntimeServerTest` execution create. They stay around because the context isn't closed after each test finishes. Adding `afterEach` cleanup sequence to `runtime-with-instruments` tests. Improving robustness of overall `EnsoContext` [shutdown](https://github.com/enso-org/enso/pull/6468#discussion_r1180348303).
2023-05-02 11:55:02 +00:00
Hubert Plociniczak
7e33300013
Make execution mode live default for CLI (#6496)
As requested in the design doc and in the ticket.

Closes #6495.
2023-05-02 09:58:41 +00:00
James Dunkerley
6b0c682b08
Add Execution Context control to Text.write (#6459)
- Adjusted `Context.is_enabled` to support default argument (moved built in so can have defaults).
- Made `environment` case-insensitive.
- Bug fix for play button.
- Short hand to execute within an enabled context.
- Forbid file writing if the Output context is disabled with a `Forbidden_Operation` error.
- Add temporary file support via `File.create_temporary_file` which is deleted on exit of JVM.
- Execution Context first pass in `Text.write`.
- Added dry run warning.
- Writes to a temporary file if disabled.
- Created a `DryRunFileManager` which will create and manage the temporary files.
- Added `format` dropdown to `File.read` and `Data.read`.
- Renamed `JSON_File` to `JSON_Format` to be consistent.

(still to unit test).
2023-04-29 08:39:18 +00:00
Jaroslav Tulach
efe904cd9f
Introducing @BuiltinMethod.needsFrame and InlineableNode (#6442)
Fixes #6416 by introducing `InlineableNode`. It runs fast even on GraalVM CE, fixes ([forever broken](https://github.com/enso-org/enso/pull/6442#discussion_r1178782635)) `Debug.eval` with `<|` and [removes discouraged subclassing](https://github.com/enso-org/enso/pull/6442#discussion_r1178778968) of `DirectCallNode`. Introduces `@BuiltinMethod.needsFrame` - something that was requested by #6293. Just in this PR the attribute is optional - its implicit value continues to be derived from `VirtualFrame` presence/absence in the builtin method argument list. A lot of methods had to be modified to pass the `VirtualFrame` parameter along to propagate it where needed.
2023-04-28 15:32:13 +00:00
Hubert Plociniczak
fbfdba2b4a
Don't attempt to resolve hostname for git init (#6463)
During initialization JGit may attempt to resolve hostname. On some systems this can take more than desired triggering timeouts. This change does two things:
- sets the default committer for changes, lack of which probably triggered the check
- sets the default hostname to `localhost` (we don't care), in case something else in JGit still wants to resolve hostname

Closes #6447.

# Important Notes
I wasn't able to reproduce this so relying on @mwu-tow since apparently he can repro it reliably.
2023-04-28 14:29:30 +00:00
Hubert Plociniczak
ae3f9025e3
Invoke instance methods for Any overrides (#6441)
This change modifies method dispatch for methods that override Any's definitions. When an overrided method is invoked statically we call Any's method to stay consistent.
This change primarily addresses the plethora of problems related to `to_text` invocations. It does not attempt to completely modify method dispatch logic.

Closes #6300.
2023-04-28 07:18:37 +00:00
Hubert Plociniczak
c6790f1e9c
Report only unique warnings (#6372)
This change makes sure that reported warnings are unique, based on the value of internal clock tick and ignoring differences in reassignments.

Before:
![Screenshot from 2023-04-20 15-42-55](https://user-images.githubusercontent.com/292128/233415710-925c1045-37c7-49f5-9bc3-bfbfd30270a3.png)
After:
![Screenshot from 2023-04-20 15-27-27](https://user-images.githubusercontent.com/292128/233415807-8cb67bc2-ac37-4db7-924e-ae7619074b5b.png)

On the positive side, no further changes, like in LS, have to be done.


Closes #6257.
2023-04-28 07:16:00 +00:00
Hubert Plociniczak
376415ab17
Mark some tests as (unconditonally) flaky (#6453)
Too many spurious failures on CI now. Will need to be tackled at some
point.
2023-04-27 17:09:46 +02:00
Dmitry Bushev
3401097266
Switch execution environment before the complete notification (#6449)
Fixes the runtime test

```
- should recompute expressions changing an execution environment *** FAILED ***
"[live]" did not equal "[design]" (RuntimeServerTest.scala:2721)
Analysis:
"[live]" -> "[design]"
```
2023-04-27 11:21:07 +00:00
James Dunkerley
0e51131809
Table Visualization and display text changes. (#6382)
- Missing tests from number parsing.
- Fix type signature on some warning methods.
- Fix warnings on `Standard.Database.Data.Table.parse_values`.
- Added test for `Nothing` and empty string on `use_first_row_as_names`.
- New API for `Number.format` taking a simple format string and `Locale`.
- Add ellipsis to truncated `Text.to_display_text`.
- Adjusted built-in `to_display_text` for numbers to not include type (but also to display BigInteger as value).
- Remove `Noise.Generator` interface type.
- Json: Added `to_display_text` to `JS_Object`.
- Time: Added `to_display_text` for `Date`, `Time_Of_Day`, `Date_Time`, `Duration` and `Period`.
- Text: Added `to_display_text` to `Locale`, `Case_Sensitivity`, `Encoding`, `Text_Sub_Range`, `Span`, `Utf_16_Span`.
- System: Added `to_display_text` to `File`, `File_Permissions`, `Process_Result` and `Exit_Code`.
- Network: Added `to_display_text` to `URI`, `HTTP_Status_Code` and `Header`.
- Added `to_display_text` to `Maybe`, `Regression`, `Pair`, `Range`, `Filter_Condition`.
- Added support for `to_js_object` and `to_display_text` to `Random_Number_Generator`.
- Verified all error types have `to_display_text`.
- Removed `BigInt`, `Date`, `Date_Time` and `Time_Of_Day` JS based rendering as using `to_display_text` now.
- Added support for rendering nested structures in the table viz.
2023-04-26 18:15:48 +00:00
Dmitry Bushev
0e5ee35aba
Skip redundant compilations (#6436)
related #6323

The engine can skip compilation when applying changes that do not require execution. It is more efficient to process the changes in a batch, than triggering compilations every time such edit is received.
2023-04-26 17:53:01 +00:00
Jaroslav Tulach
63de18e367
Turn null into UnexpectedExpression when Union type is incomplete (#6415)
Test and fix for #6401.
2023-04-25 14:49:26 +00:00
Radosław Waśko
b8f075a178
Ensure that IO.println does not fail if to_text returned a non-Text value. (#6223) 2023-04-25 10:55:39 +00:00
Jaroslav Tulach
8e08a3b634
Improve inlining of <| on (GraalVM EE) (#6384) 2023-04-25 12:02:51 +02:00
Jaroslav Tulach
e47eb49ea8
Removing need for asynchronous thread to execute ResourceManager finalizers (#6335)
While Enso runs single-threaded, its `ResourceManager` required additional asynchronous thread to execute its _"finalizers"_. What has been necessary back then is no longer needed since _GraalVM 21.1_. GraalVM now provides support for submitting `ThreadLocalAction` that gets then picked and executed via `TruffleSafepoint` locations. This PR uses such mechanism to _"inject"_ finalizer execution into already running Enso evaluation thread.

Requiring more than one thread has complicated Enso's co-existence with other Truffle language. For example Graal.js is strictly singlethreaded and used to refuse (simple) co-existence with Enso. By allowing Enso to perform all its actions in a single thread, the synergy with Graal.js becomes better.
2023-04-20 13:33:45 +02:00
Pavel Marek
4076a64f33
NaN can be used as a key in Map (#6301)
`Number.nan` can be used as a key in `Map`. This PR basically implements the support for [JavaScript's Same Value Zero Equality](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Equality_comparisons_and_sameness#same-value-zero_equality) so that `Number.nan` can be used as a key in `Map`.

# Important Notes
- For NaN, it holds that `Meta.is_same_object Number.nan Number.nan`, and `Number.nan != Number.nan` - inspired by JS spec.
- `Meta.is_same_object x y` implies `Any.== x y`, except for `Number.nan`.
2023-04-20 09:25:36 +00:00
Hubert Plociniczak
6d3151f32d
UnresolvedSymbol is now accepted by Vector.sort (#6334)
`Vector.sort` does some custom method dispatch logic which always expected a function as `by` and `on` arguments. At the same time, `UnresolvedSymbol` is treated like a (to be resolved) `Function` and under normal circumstances there would be no difference between `_.foo` and `.foo` provided as arguments.

Rather than adding an additional phase that does some form of eta-expansion, to accomodate for this custom dispatch, this change only fixes the problem locally. We accept `Function` and `UnresolvedSymbol` and perform the resolution on the fly. Ideally, we would have a specialization on the latter but again, it would be dependent on the contents of the `Vector` so unclear if that is better.

Closes #6276,

# Important Notes
There was a suggestion to somehow modify our codegen to accomodate for this scenario but I went against it. In fact a lot of name literals have `isMethod` flag and that information is used in the passes but it should not control how (late) codegen is done. If we were to make this more generic, I would suggest maybe to add separate eta-expansion pass. But it could affect other things and could be potentially a significant change with limited potential initially, so potential future work item.
2023-04-20 07:58:58 +00:00
Dmitry Bushev
8c9c2d79cd
Keep scheduled execution job on text edit (#6354)
close #6346

Changelog:
- fix: keep scheduled execution job in the queue when applying a text edit
2023-04-19 14:18:36 +00:00
Hubert Plociniczak
f288198b4b
Change order of arguments when controlling context (#6357)
https://github.com/orgs/enso-org/discussions/6344 requested to change the order of arguments when controlling context permissions.

# Important Notes
The change brings it closer to the design doc but IMHO also a bit cumbersome to use (see changed tests) - applications involving default arguments don't play well when the last argument is not the default 🤷 .`
2023-04-19 14:00:27 +00:00
Hubert Plociniczak
0088672f9b
Simplify test for 6248 (#6343)
As per suggestion https://github.com/enso-org/enso/pull/6275#discussion_r1167388449
2023-04-19 11:43:52 +02:00
Jaroslav Tulach
f52ef08be5
Avoid double WithWarnings wrapping (#6348) 2023-04-19 10:43:29 +02:00
Dmitry Bushev
7733b9bdac
Async Language Server resources initialization (#6336)
close #6306

Changelog:
- add: `AsyncResourceInitialization` component
- update: initialize language server resources asynchronously

# Important Notes
Speeds up `session/InitProtocolConnection` request by ~300ms (~10%) 3051ms (before) vs 2740ms (after) on my machine.
2023-04-18 15:55:44 +00:00
Jaroslav Tulach
e5a96b9782
Avoid NullPointerException in --dumpGraphs & test & docs (#6304) 2023-04-18 06:58:06 +02:00
Hubert Plociniczak
f720bd2516
Type of UnresolvedSymbol is Function (#6284)
`Meta.is_a` and `Meta.type_of` now recognize UnresolvedSymbol.
Closes #6277.
2023-04-17 13:44:15 +00:00
Jaroslav Tulach
413661b366
Splitting Gigantic EqualsNode into few Smaller Ones (#6280)
Splitting gigantic `EqualsNode` into few smaller ones
2023-04-17 11:29:54 +02:00
Pavel Marek
b42e910280
sort handles incomparable values (#5998)
* Update type ascriptions in some operators in Any

* Add @GenerateUncached to AnyToTextNode.

Will be used in another node with @GenerateUncached.

* Add tests for "sort handles incomparable types"

* Vector.sort handles incomparable types

* Implement sort handling for different comparators

* Comparison operators in Any do not throw Type_Error

* Fix some issues in Ordering_Spec

* Remove the remaining comparison operator overrides for numbers.

* Consolidate all sorting functionality into a single builtin node.

* Fix warnings attachment in sort

* PrimitiveValuesComparator handles other types than primitives

* Fix byFunc calling

* on function can be called from the builtin

* Fix build of native image

* Update changelog

* Add VectorSortTest

* Builtin method should not throw DataflowError.

If yes, the message is discarded (a bug?)

* TypeOfNode may not return only Type

* UnresolvedSymbol is not supported as `on` argument to Vector.sort_builtin

* Fix docs

* Fix bigint spec in LessThanNode

* Small fixes

* Small fixes

* Nothings and Nans are sorted at the end of default comparator group.

But not at the whole end of the resulting vector.

* Fix checking of `by` parameter - now accepts functions with default arguments.

* Fix changelog formatting

* Fix imports in DebuggingEnsoTest

* Remove Array.sort_builtin

* Add comparison operators to micro-distribution

* Remove Array.sort_builtin

* Replace Incomparable_Values by Type_Error in some tests

* Add on_incomparable argument to Vector.sort_builtin

* Fix after merge - Array.sort delegates to Vector.sort

* Add more tests for problem_behavior on Vector.sort

* SortVectorNode throws only Incomparable_Values.

* Delete Collections helper class

* Add test for expected failure for custom incomparable values

* Cosmetics.

* Fix test expecting different comparators warning

* isNothing is checked via interop

* Remove TruffleLogger from SortVectorNode

* Small review refactorings

* Revert "Remove the remaining comparison operator overrides for numbers."

This reverts commit 0df66b1080.

* Improve bench_download.py tool's `--compare` functionality.

- Output table is sorted by benchmark labels.
- Do not fail when there are different benchmark labels in both runs.

* Wrap potential interop values with `HostValueToEnsoNode`

* Use alter function in Vector_Spec

* Update docs

* Invalid comparison throws Incomparable_Values rather than Type_Error

* Number comparison builtin methods return Nothing in case of incomparables
2023-04-16 16:40:12 +02:00
Hubert Plociniczak
e163353d16
Deprecate Error's to_display_text builtin methods (#6275)
The primary motivation for this change was
https://github.com/enso-org/enso/issues/6248, which requested the possibility of defining `to_display_text` methods of common errors via regular method definitions. Until now one could only define them via builtins.

To be able to support that, polyglot invocation had to report `to_display_text` in the list of (invokable) members, which it didn't. Until now, it only considered fields of constructors and builtin methods. That is now fixed as indicated by the change in `Atom`.

Closes #6248.

# Important Notes
Once most of builtins have been translated to regular Enso code, it became apparent how the usage of `.` at the end of the message is not consistent and inflexible. The pure message should never follow with a dot or it makes it impossible to pretty print consistently for the purpose of error reporting. Otherwise we regularly end up with errors ending with `..` or worse. So I went medieval on the reasons for failures and removed all the dots.
The overall result is mostly the same except now we are much more consistent.

Finally, there was a bit of a good reason for using builtins as it simplified our testing.
Take for example `No_Such_Method.Error`. If we do not import `Errors.Common` module we only rely on builtin error types. The type obviously has the constructor but it **does not have** `to_display_text` in scope; the latter is no longer a builtin method but a regular method. This is not really a problem for users who will always import stdlib but our tests often don't. Hence the number of changes and sometimes lack of human-readable errors there.
2023-04-14 19:14:49 +00:00
Jaroslav Tulach
a74933d10f
Speed cascade of if statements up (#6255)
Fixes #5709. We have a test and a generic fix that improves inlining of every builtin. Everything seems to be faster.
2023-04-14 13:27:23 +00:00
Dmitry Bushev
9ebda56385
Language Server preserves the suggestion updates order (#6268)
Related #6138

Fixes an issue when the `SuggestionsHandler` can send suggestion updates without preserving the order in which they were received.
2023-04-14 12:39:02 +00:00
James Dunkerley
4e92d76665
Add Warning methods onto Any to make working with them easier. (#6176)
Adds `has_warnings`, `remove_warnings` and `throw_on_warning` extension methods.
2023-04-12 15:08:01 +00:00
Hubert Plociniczak
c8ea06c4a6
Provide Forbidden_Operation.to_display_text (#6246)
Added a missing `to_display_text` method which, by convention, is defined via a builtin method.

Also re-enabled and fixed pending tests.

Closes #6227.
2023-04-12 07:53:21 +00:00
Dmitry Bushev
b97fc39214
Remove SQL versions repo (#6242)
close #6232

Changelog:
- remove: `SqlVersionsRepo`
- update: `SuggestionsDatabaseModuleUpdateNotification` message removing the version
- update: cleanup versions repo usages in the language server
2023-04-11 19:22:30 +00:00
Jaroslav Tulach
5b3cf6f503
Consolidate Vector and Array methods (#6218)
Fixes #5011 by making sure the same methods that are on `Vector` are also available on `Array`.
2023-04-11 16:20:11 +00:00
Dmitry Bushev
6663125410
Language Server support for execution environments (#6217)
close #6133

Changelog:
- add: `executionContext/setExecutionEnvironment` request
- update: `executionContext/recompute` request to have optional `executionEnvironment` parameter.
2023-04-11 15:12:27 +00:00
Hubert Plociniczak
3cce3b3caf
Replace IOContexts with execution env and contexts (#6171)
As per design, IOContexts controlled via type signatures are going away. They are replaced by explicit `Context.if_enabled` runtime checks that will be added to particular method implementations.

`production`/`development` `IOPermissions` are replaced with `live` and `design` execution enviornment. Currently, the `live` env has a hardcoded list of allowed contexts i.e. `Input` and `Output`.

# Important Notes
As per design PR-55. Closes #6129. Closes #6131.
2023-04-06 15:47:40 +00:00
Dmitry Bushev
df4491de50
Batch insert suggestions (#6189)
close #6080

Changelog
- add: implement `SuggestionsRepo.insertAll` as a batch SQL insert
- update: `search/getSuggestionsDatabase` returns empty suggestions. Currently, the method is only used at startup and returns the empty response anyway because the libs are not loaded at that point.
- update: serialize only global (defined in the module scope) suggestions during the distribution building. There's no sense in storing the local library suggestions.
- update: sqlite dependency
- remove: unused methods from `SuggestionsRepo`
- remove: Arguments table

# Important Notes
Speeds up libraries loading by ~1 second.

![2023-04-03-173423_2086x324_scrot](https://user-images.githubusercontent.com/357683/229597470-19dcc010-2a34-43e1-87be-60af99afd275.png)
![2023-04-03-173514_2083x321_scrot](https://user-images.githubusercontent.com/357683/229597476-bf5b3c33-6321-4ac9-a0ca-2fb57d257857.png)
2023-04-06 07:47:23 +00:00
Jaroslav Tulach
741b394b0d
Suspended atom fields are evaluated only once (#6151)
Implements #6134.

# Important Notes
One can define lazy atom fields as:
```haskell
type Lazy
Value ~x ~y
```
the evaluation of the `x` and `y` fields is then delayed until they are needed. The evaluation happens once. Then the computed value is kept in the atom for further use.
2023-04-05 23:46:08 +00:00
Jaroslav Tulach
4805193428
Text.to_display_text is (shortened) identity (#6174)
Fixes #5971.
2023-04-05 19:53:07 +00:00
Jaroslav Tulach
519df66d39
Always try to resolve conversion for Any type (#6184)
Fixes #5898 by removing `Catch.panic` and speeding the `sieve.enso` benchmark from 1058 ms to 514 ms. Should there be no dedicated conversion, let's use one defined on `Any` type - e.g. defining a conversion `from(Any)` makes such a conversion is always available.
2023-04-04 09:24:35 +00:00
Jaroslav Tulach
310a2d8ae7
Delay creation of project root EnsoFile until it is really needed (#6149)
Delay creation of `EnsoFile` until it is needed.

# Important Notes
By putting breakpoint into `Atom` constructor I realized few `EnsoProjectNode` instances may be created when parsing the project. It makes no sense to also create `EnsoFile` for them - until it is needed.
2023-04-03 14:03:25 +00:00
Jaroslav Tulach
0d7682b6ec
Adjusting tests to new parser (#6143)
Modification to various tests disabled when #5917 was integrated to pass with new parser. Fixes #5894.

# Important Notes
Some tests can be fixed just by changes on the `IR` side. Some (especially error simulating ones) would benefit from changes in the `Tree` structure or at least @kazcw evaluation.
2023-04-02 21:24:50 +00:00
Radosław Waśko
6ddcb553e5
Date/time support for Postgres. Year/month/day operations on Columns. (#6153)
Closes #6115
2023-03-31 18:37:04 +00:00
Dmitry Bushev
2338e5d8e6
When renaming the project clean old modules instead of updating (#6148)
close #6139
close #6137

When the project is renamed, the engine cleans up affected modules and initiates modules re-indexing to fill the suggestions database with new records. This way it reduces the amount of information stored in the suggestions database and helps implement #6080 optimization.

Changelog:
- remove: rename features from the suggestions database
- update: rename command to initiate modules cleanup and project re-execution
- fix: #6137
2023-03-31 09:40:21 +00:00
Hubert Plociniczak
90612cfa35
Report errors when generating indexes (#6123)
`--compile` command would run the compilation pipeline but silently omit any encountered errors, thus skipping the serialization. This maybe was a good idea in the past but it was problematic now that we generate indexes on build time.
This resulted in rather obscure errors (#6092) for modules that were missing their caches.

The change should significantly improve developers' experience when working on stdlib.

# Important Notes
Making compilation more resilient to sudden cache misses is a separate item to be worked on.
2023-03-30 09:59:49 +00:00
Jaroslav Tulach
8c5cd6e35e
Always print stacktrace to provide some guidance of what is going on (#6128)
Avoid displaying just `Execution finished with an error: java.lang.NullPointerException` without any additional detail.

# Important Notes
When there is an execution error outside of Enso code (identified by the fact that all stack frames belong to `java` language), let's print the whole stack rather than printing nothing. To simulate one can:
```diff
diff --git engine/runtime/src/main/scala/org/enso/compiler/Compiler.scala engine/runtime/src/main/scala/org/enso/compiler/Compiler.scala
index ff0636bc66..42e5eae32e 100644
--- engine/runtime/src/main/scala/org/enso/compiler/Compiler.scala
+++ engine/runtime/src/main/scala/org/enso/compiler/Compiler.scala
@@ -453,7 +453,8 @@ class Compiler(
pool
)
} else {
-          CompletableFuture.completedFuture(ensureParsedAndAnalyzed(module))
+          // CompletableFuture.completedFuture(ensureParsedAndAnalyzed(module))
+          CompletableFuture.completedFuture(())
}
}
```
2023-03-30 08:05:28 +00:00
Jaroslav Tulach
d1c52fef0b
Turn missing body in a binding into a syntax error (#6107)
Test and fix for #5903.
2023-03-28 16:42:18 +00:00
Hubert Plociniczak
b3db805f92
More elaborate error message for invalid text edit (#6108)
It is sometimes impossible to figure out the real reason for invalid text edit request. Added a bit of context to failures to narrow down the cause of the failure.

# Important Notes
Should help with diagnosing issues like #6099.
2023-03-28 14:21:50 +00:00
Jaroslav Tulach
46bd6f6247
Serde testing infrastructure for the Enso compiler (#6062)
The primary delivery of this PR is a design of `SerdeCompilerTest` - a testing suite that allows us to write sample projects, parse them with and without caches and verify they still produce the same `IR`. This is a similar idea to #3723 which compared the old and new parser `IR`s.

With infrastructure like this we can start addressing #5567 without any (significant) fear of breaking something essential.
2023-03-28 08:46:07 +00:00
Dmitry Bushev
aa97787603
Update ZIO library (#6072)
close #6069

Changelog:
- update: ZIO major version bump
2023-03-28 07:58:59 +00:00
Pavel Marek
9bec3a4e71
Treat Boolean.False as false primitive (#6090)
Treat `Boolean.False` and `Boolean.True` as the corresponding primitives. Now, `Boolean.False == False` returns true.

# Important Notes
`False` and `True` constructs, that are converted to `ConstructorNode` during Truffle codegen, are handled specially in `ConstructorNode`. The easiest fix was to implement a similar special handling in `QualifiedAccessorNode`, although not the cleanest one.

A better solution would be to provide transformation of `Boolean.True` IR to a true literal in `ApplicationSaturation` compiler pass. But `ApplicationSaturation` pass does not handle `True`. Moreover, for our case, it is unnecessarily complicated.
2023-03-28 03:41:49 +00:00
Hubert Plociniczak
76409b285d
Ensure new and wrapper nodes inherit UUID (#6067)
Instrumentation of calls involving warning values never really worked because:
1) newly created nodes didn't set the UUID of their children
2) the instrumentable wrappers always had an empty (i.e. null) UUID and
they never referred `get`/`setId` calls to their delegates

On the surface, everything worked fine. Except when one actually relied on the instrumentation of values with warnings for proper setup. Then no instrumentation (replacement of nodes) was performed due to empty UUID (as required by `hasTag` of `FunctionCallInstrumentationNode`).

Closes #6045. Discovered in #5893.
2023-03-27 17:49:20 +00:00
James Dunkerley
bf2545fa04
Use new common parse method throwing less exceptions. (#6075)
Avoiding exceptions by not using parseBest.

Time now in CLI is 1.15s for 500k rows vs 1.65s in GUI.

CLI:
![image](https://user-images.githubusercontent.com/4699705/227711266-bc005b0d-5011-450f-964b-65dd2e437c2e.png)

GUI:
![image](https://user-images.githubusercontent.com/4699705/227711259-f7ddda29-86c7-4eef-a002-4bf0bda6063f.png)

Added it as a function in the shared library so used by both engine and polyglot.
2023-03-27 11:02:10 +00:00
Dmitry Bushev
4c62dc9061
Search suggestions by static attribute (#6036)
close #5874

Changelog:
- add: `isStatic` parameter to `search/completion` request to search by the `static` suggestion attribute
- update: search non-static suggestions when opening component browser

# Important Notes
Component browser doesn't show `Table.new` and `Table.from_rows` suggestions when a `Table` node is selected.

![2023-03-21-151117_1301x877_scrot](https://user-images.githubusercontent.com/357683/226874291-1ff99994-1bb6-41df-96b4-dc5c5178ba41.png)
2023-03-23 15:02:25 +00:00
James Dunkerley
dd009fd1af
Tidy up the public module level statics (#6032)
Tidies up a lot of PUBLIC module statics - marked some as PRIVATE, made some methods of types.
2023-03-22 18:02:37 +00:00
James Dunkerley
546cb0c4ab
Deal with warnings attached to value when making widgets. (#5994)
- Fixes InvokeCallableNode to support warnings.
- Strips warnings from annotations in `get_widget_json`.
- Remove `get_full_annotations_json`.
- Fix warnings on Dialect.
2023-03-22 10:50:19 +00:00
Hubert Plociniczak
8c6fd60aaf
Detect conflicts between exported types and FQNs (#5986)
Exporting types named the same as the module where they are defined in `Main` modules of library components may lead to accidental name conflicts. This became apparent when trying to access `Problem_Behavior` module via a fully qualified name and the compiler rejected it. This is due to the fact that `Main` module exported `Error` type defined in `Standard.Base.Error` module, thus making it impossible to access any other submodules of `Standard.Base.Error` via a fully qualified name.

This change adds a warning to FullyQualifiedNames pass that detects any such future problems.
While only `Error` module was affected, it was widely used in the stdlib, hence the number of changes.

Closes #5902.

# Important Notes
I left out the potential conflict in micro-distribution, thus ensuring we actually detect and report the warning.
2023-03-21 21:09:41 +00:00
Dmitry Bushev
e7ced7905a
feat: early init zio (#6029) 2023-03-21 17:34:02 +01:00
Dmitry Bushev
8125d89676
Delay initialization of JSON-RPC protocol (#5997)
close #5881

Changelog
- add: ProtocolFactory object that initializes and returns the protocol object
- update: Add protocol initialization to the initialization component
2023-03-20 11:04:16 +00:00
Dmitry Bushev
f3c5643f33
Delay initialization of ZIO executors (#5990)
close #5882

Changelog:
- update: move ZIO initialization logic to the initialization component
2023-03-17 23:37:47 +00:00
Dmitry Bushev
b2aa58bf20
Delay writing IR caches (#5957)
close #5911

In interactive mode, perform writing IR caches in the background jobs queue. Background jobs execution is delayed until the first execution is complete.
2023-03-17 22:47:15 +00:00
Jaroslav Tulach
b46be10f63
Introducing Meta.Type (#5956)
Fixing #5768 and #5765 and co. Introducing `Meta.Type` and giving it the desired methods.

# Important Notes
`Type` is no longer a `Meta.Atom`, but it has a dedicated `Meta.Type` representation.
2023-03-17 20:08:18 +00:00
Kaz Wesley
75fda33837
Fix precedence of -1.x (#5830)
Fixes #5826.

# Important Notes
- Change frontend representation of negation.
- Fix a precedence issue: The `.` operators in -1.x and -1.2 must have different precedences.
- Remove a no-longer-needed special case from backend translation.
- Add tests for this case after all translations.
2023-03-17 18:53:34 +00:00
Jaroslav Tulach
c696bf1d87
Don't persist UUIDs in IR caches (#5918)
Give Cache subclasses a chance to control the output stream format. Use that functionality to avoid persisting `UUID` into standard library `.ir` caches. Gets the number of caches down to 42MB from 48MB.

# Important Notes
I believe UUIDs are not really useful for standard libraries and can be omitted. Am I right?
2023-03-17 16:53:53 +00:00
Dmitry Bushev
265fb5d505
Delay initialization of SQL database (#5970)
close #5875

Move database initialization logic to the initialization component.
2023-03-17 13:38:11 +00:00
Dmitry Bushev
69d8864a1f
Fix qualified names in tag values (#5978)
close #5901

Changelog:
- fix: qualified names in tag values

# Important Notes
![2023-03-16-190913_892x345_scrot](https://user-images.githubusercontent.com/357683/225684530-0d60a7a0-8b95-40a9-9ff6-2c112d966991.png)
2023-03-16 17:36:15 +00:00
Hubert Plociniczak
a6487e5105
Don't log installed engines and runtimes in prod (#5900)
The `logAvailableComponentsForDebugging` will check and install all necessary components of GraalVM for every mentioned version. While not harmful, it adds up to startup time.
Additionally added an option in language server startup to skip installation of GraalVM components. The latter is already performed by project-manager when opening the project and it is unnecessary to do it twice. Due to LS' architecture this configuration has to be passed around via multiple configs.

Finally, skipped the attempt to install Python component on Windows - this is not supported by GraalVM atm.

Closes #5749.

# Important Notes
The impact of this problem could be really felt the more versions of Enso and GraalVM one had since it would go through all of them.
2023-03-16 10:36:55 +00:00
Jaroslav Tulach
e666d797c5
Dedicated Module.containsUUID method with a cache (#5945)
Fixes #5781 by caching all UUIDs in each module in a `Map`.
2023-03-15 17:14:59 +00:00
Kaz Wesley
e171fba301
New documentation parser (#5917)
Implement new Enso documentation parser; remove old Scala Enso parser.

Performance: Total time parsing documentation is now ~2ms.

# Important Notes
- Doc parsing is now done only in the frontend.
- Some engine tests had never been switched to the new parser. We should investigate tests that don't pass after the switch: #5894.
- The option to run the old searcher has been removed, as it is obsolete and was already broken before this (see #5909).
- Some interfaces used only by the old searcher have been removed.
2023-03-15 15:43:51 +00:00
Hubert Plociniczak
c21bd3b185
Generate import/export bindings in local cache (#5908)
When generating import/export bindings in local cache, they are included in the distribution.
Additionally, removed the hardcoded value for suggestions cache. Now one can generate them for local as well as for global cache, based on the presence or lack of `--no-global-cache` parameter.

Closes #5890.
2023-03-15 12:40:30 +00:00
Dmitry Bushev
5c6a5dcf60
Do not cancel detach visualization jobs (#5913)
close #5889

Changelog:
- update: make `DetachVisualizationJob` a unique job to make sure that they are not canceled during the re-compilation after `text/applyEdit` command.
2023-03-14 12:31:32 +00:00
Dmitry Bushev
0d82a2af89
Delay background jobs execution (#5904)
close #5892

Changelog:
add: feature to delay background jobs execution
add: start background jobs when program finishes
add: start background jobs on `search/completion` request
2023-03-14 11:26:11 +00:00
Jaroslav Tulach
888307f03f
Meta.get_constructor_fields works for Type (#5869)
Fixes #5805 by returning `[]` as list of fields of `Type`.

# Important Notes
`Type` is recognized as `Meta.is_atom` since #3671. However `Type` isn't an `Atom` internally. We have to provide special handling for it where needed.
2023-03-13 09:58:11 +00:00
Radosław Waśko
263c3ad651
Add a common-polyglot-core-utils project (#5855)
Adds a common project that allows sharing code between the `runtime` and `std-bits`.

Due to classpath separation and the way it is compiled, the classes will be duplicated - we will have one copy for the `runtime` classpath and another copy as a small JAR for `Standard.Base` library.

This is still much better than having the code duplicated - now at least we have a single source of truth for the shared implementations.

Due to the copying we should not expand this project too much, but I encourage to put here any methods that would otherwise require us to copy the code itself.

This may be a good place to put parts of the hashing logic to then allow sharing the logic between the `runtime` and the `MultiValueKey` in the `Table` library (cc: @Akirathan).
2023-03-11 09:27:26 +00:00
Dmitry Bushev
725841467f
Log exceptions with stack traces (#5871)
close #5070

Changelog:
- Include the original exception to log expressions
- Enable logging of Akka Actors' lifecycle events on debug logging level
- Decrease the severity of interruption log messages because interruptions are part of the workflow. The computation can be interrupted at any time, and still be recomputed after. Warnings are just misleading in this case.
2023-03-11 08:23:31 +00:00
Pavel Marek
5f7a4a5a39
Merge ordered and unordered comparators (#5845)
Merge _ordered_ and _unordered_ comparators into a single one.

# Important Notes
Comparator is now required to have only `compare` method:
```
type Comparator
comapre : T -> T -> (Ordering|Nothing)
hash : T -> Integer
```
2023-03-11 05:43:22 +00:00
Jaroslav Tulach
632a303089
Use SimpleFormatter.formatMessage to replace {0} with actual parameters (#5870)
Fixes #5801 to properly format Truffle log records before sending them for further processing.
2023-03-11 00:15:58 +00:00
James Dunkerley
7887fb8d40
Regex.Compile using Truffle Regex, update find, match and match_all (#5785) 2023-03-10 21:49:50 +00:00
Radosław Waśko
91ef8acf35
Review generated Column names (#5850)
Closes #5583 and closes #5157
2023-03-10 19:07:58 +00:00
Jaroslav Tulach
8bbdd1af5b
Meta.is_a consistent with case-type-of check (#5853)
Removing special handling of `AtomConstructor` in `Meta.is_a` check.

# Important Notes
A lot of tests are about to fail. Many of them indirectly call `Meta.is_a` with a constructor rather than type.
2023-03-10 07:41:04 +00:00
Hubert Plociniczak
6769ab0ee7
Downgrade hashing to SHA-1 and other optimizations (#5791)
This change downgrades hashing algorithm used in caching IR and library bindings to SHA-1. It is sufficient and significantly faster for the purpose of simple checksum we use it for.

Additionally, don't calculate the digest for serialized bytes - if we get the expected object type then we are confident about the integrity.

Don't initialize Jackson's ObjectMapper for every metadata serialization/de-serialization. Initialization is very costly.

Avoid unnecessary conversions between Scala and Java. Those back-and-forth `asScala` and `asJava` are pretty expensive.

Finally fix an SBT warning when generating library cache.

Closes https://github.com/enso-org/enso/issues/5763

# Important Notes
The change cuts roughly 0.8-1s from the overall startup.
This change will certainly lead to invalidation of existing caches. It is advised to simply start with a clean slate.
2023-03-09 07:36:59 +00:00
Dmitry Bushev
9397a6ec2f
Pre compute suggestion db during build time (#5698)
Close #5068

Cache suggestions during the `buildEngineDistribution` command, and read them from the disk when the library is loaded. Initial graph coloring takes ~20 seconds vs ~25 seconds on the develop branch.

[peek-develop-branch.webm](https://user-images.githubusercontent.com/357683/223504462-e7d48262-4f5e-4724-b2b0-2cb97fc05140.webm)
[peek-suggestions-branch.webm](https://user-images.githubusercontent.com/357683/223504464-0fe86c04-8c4b-443c-ba96-6c5e2fb1e396.webm)
2023-03-08 12:37:48 +00:00
James Dunkerley
299bfd6b7d
Fixes from the Demo on 2nd March (#5823)
- Fix issue with Geo Map viz.
- Handle invalid format strings better in `Data_Formatter`.
- New constants for the ISO format strings (and a special ENSO_ZONED_DATE_TIME)
- Consistent Date Time format for parsing in all places.
- Avoid throwing exception in datetime parsing.
- Support for milliseconds (well nanoseconds) in Date_Time and Time_Of_Day.
- `Column.map` stays within Enso.
- Allow `Aggregate_Column.Group_By` in `cross_tab` group_by parameter.
2023-03-07 20:58:00 +00:00
Jaroslav Tulach
e3f0888c5f
Coerce values obtained from polyglot calls (#5832)
Coerce values obtained from polyglot calls to fix #5177.

# Important Notes
Adds `IntHolder` class into the `test/Tests` project to simulate access to a class with integer field.
2023-03-07 18:21:30 +00:00
Pavel Marek
b6e2319fcc
Comparators support partial ordering (#5778) 2023-03-07 04:16:38 +00:00
Radosław Waśko
da760aa27d
Review Text/Table.write problem behavior (#5816)
Closes #5114

Added tests for various problems scenarios when writing files.

And ensured that those tests are passing by fixing a few edge cases.
2023-03-07 02:25:13 +00:00
Jaroslav Tulach
161ea45019
Report AbstractTruffleException as a Panic (#5794)
Report `AbstractTruffleException` as a `Panic`. Fixes #5260.
2023-03-06 19:05:25 +00:00
Radosław Waśko
2d29456ed1
Review File/Data read and read_text warnings (#5799)
Closes #5113

Fixes a bug where read-only files would be overwritten if File.write was used in backup mode, and added tests to avoid such regression. To implement it, introduced a `is_writable` property on `File`.
2023-03-06 03:43:38 +00:00
Jaroslav Tulach
f64edd806d
Switch to Truffle PE mode before calling into Enso (#5783)
Enter partial evaluation mode via `CallTarget.call` before invoking `InteropLibrary`. Fixes #5782.
2023-03-03 14:39:37 +00:00
Hubert Plociniczak
941512e0ba
Optimize import/export resolution (#5700)
This change adds serialization and deserialization of library bindings.
In order to be functional, one needs to first generate IR and
serialize bindings using `--compiled <path-to-library>` command. The bindings
will be stored under the library with `.bindings` suffix.
Bindings are being generated during `buildEngineDistribution` task, thus not
requiring any extra steps.

When resolving import/exports the compiler will first try to load
module's bindings from cache. If successful, it will not schedule its
imports/exports for immediate compilation, as we always did, but use the
bindings info to infer the dependent modules.

The current change does not make any optimizations when it comes to
compiling the modules, yet. It only delays the actual
compilation/loading IR from cache so that it can be done in bulk.
Further optimizations will come from this opportunity such as parallel
loading of caches or lazily inferring only the necessary modules.

Part of https://github.com/enso-org/enso/issues/5568 work.
2023-03-01 08:53:29 +00:00
Jaroslav Tulach
a150abbea8
Convert dangling comma into unexpected expression (#5776)
Test and fix for #5691.
2023-02-28 07:44:06 +00:00
Jaroslav Tulach
0536eeb0bb
Offer no suggestion for single constructor (#5747)
Resolving #5055 - avoid putting single constructor into suggestion database.

# Important Notes
Another way to fix #5055 is to keep the single constructor information in the suggestion database and let the IDE filter that out.
2023-02-28 05:58:36 +00:00
James Dunkerley
ba3d45e782
Adjust generateCheckedArgumentRead to return correct expected type (#5769)
Correctly get the expected type and return an IllegalArgument if no Enso type.

![image](https://user-images.githubusercontent.com/4699705/221263862-bbc122ca-b11f-49f3-b7cf-6294ea811a22.png)
2023-02-27 17:23:45 +00:00
James Dunkerley
9ee21e8812
Stability improvements in visualizations. (#5760)
- Handle `WithWarnings` in `IndirectInvokeCallableNode`.
- Handle no RootNode in `ErrorResolver`.
- Allow table vizualisation to cope if no `data` passed.
- Add `Warning.has_warnings` to check if warnings present.
- Adjust `set_value` for `JS_Object` so creates a new object each time.
2023-02-24 21:04:50 +00:00
Jaroslav Tulach
78aab133c7
Avoid nulls in Array.items (#5736)
Put `Nothing` into an empty array rather than `null`. When running with `assert` on (unit tests), check the content of the array and `AssertError` quickly when a `null` is found.
2023-02-22 17:53:36 +00:00
Pavel Marek
58c7ca5401
Performance improvements for Comparators (#5687)
Critical performance improvements after #4067

# Important Notes
- Replace if-then-else expressions in `Any.==` with case expressions.
- Fix caching in `EqualsNode`.
- This includes fixing specializations, along with fallback guard.
2023-02-21 00:56:11 +00:00
Jaroslav Tulach
97bcd87dc6
File.parent can return Nothing (#5699)
`File.parent` can be `Nothing`
2023-02-20 19:13:56 +00:00
Jaroslav Tulach
172f72941b
Two findExceptionMessage methods to extract exception messages consistently (#5684)
Creating two `findExceptionMessage` methods in `HostEnsoUtils` and in `VisualizationResult`. Why two? Because one of them is using `org.graalvm.polyglot` SDK as it runs in _"normal Java"_ mode. The other one is using Truffle API as it is running inside of partially evaluated instrument.

There is a `FindExceptionMessageTest` to guarantee consistency between the two methods. It simulates some exceptions in Enso code and checks that both methods extract the same _"message"_ from the exception. The tests verifies hosted and well as Enso exceptions - however testing other polyglot languages is only possible in other modules - as such I created `PolyglotFindExceptionMessageTest` - but that one doesn't have access to Truffle API - e.g. it doesn't really check the consistency - just that a reasonable message is extracted from a JavaScript exception.

# Important Notes
This is not full fix of #5260 - something needs to be done on the IDE side, as the IDE seems to ignore the delivered JSON message - even if it contains properly extracted exception message.
2023-02-20 11:27:16 +00:00
Jaroslav Tulach
cb3bcbb495
Schedule initialization of JS context in a separate thread (#5680)
Implements the #5643 idea. As soon as `MainModule` creates `Context` for GraalVM execution, it schedules a background task to initialize JavaScript. The initialization finishes sooner than Enso compiler is ready to work, saving time when it is actually needed.

# Important Notes
Only modifies boot sequence of `MainModule` (used in the IDE) and `VerifyJavaScriptIsAvailableTest` (to verify the _"context passing logic"_ works OK between threads). Regular CLI execution remains unchanged for now assuming batch execution may not need JavaScript in all the cases and if it does the initialization speed isn't that critical.
2023-02-17 15:54:36 +00:00
Radosław Waśko
4dcf802831
Ensure that warnings are preserved on Nothing values passing back to Enso through polyglot boundary (#5677)
Fixes #5672

# Important Notes
- Added a subproject `enso-test-java-helpers` which allows the in-Enso tests to add Java helpers for testing.
2023-02-17 13:38:26 +00:00
Pavel Marek
3e8467c204
Update GraalVM to 22.3.1 (#5602)
Updates the engine to GraalVM 22.3.1 version, which contains fixes for:
- Chrome inspector issues - 38eb3b5932

# Important Notes
- Update to GraalVM 22.3.1
- Remove host object wrapping workaround
2023-02-14 15:51:17 +00:00
GregoryTravis
99b9728995
Improve Text.toDisplayString (#5045) (#5598) 2023-02-13 18:38:15 +00:00
Dmitry Bushev
7abc281bd8
Set edition when starting language server (#4117)
When the `project.yaml` specifies an unknown edition, the project is started with the fallback edition (current engine version). It works this way:

- IDE sends project/open request to project manager (specifying only which project to open)
- project manager resolves the project edition (based on `project.yaml`) or selects the fallback if the requested edition cannot be loaded
- project manager starts the language server specifying the edition with the `--server-edition` command line parameter
- language server specifies the correct edition when creates the runtime context
2023-02-11 00:32:01 +00:00
Kaz Wesley
d1af25793a
Port graph editor to new AST (#4113)
Use the Rust parser rather than the Scala parser to parse Enso code in the IDE.

Implements:
- https://www.pivotaltracker.com/story/show/182975925
- https://www.pivotaltracker.com/story/show/182988419
- https://www.pivotaltracker.com/story/show/182970096
- https://www.pivotaltracker.com/story/show/182973659
- https://www.pivotaltracker.com/story/show/182974161
- https://www.pivotaltracker.com/story/show/182974205

There is additional functionality needed before the transition is fully-completed, however I think it's time for this to see review and testing, so I've opened separate issues. In rough order of urgency (these issues are also linked from the corresponding disabled tests):
- #5573
- #5571
- #5572
- #5574

# Important Notes
The implementation is based partly on translation, and partly on new analysis. Method- and operator-related shapes are translated to the old `Ast` variants, so that all the analysis applied to them doesn't need to be ported at this time. Everything else (mostly "macros" in the old AST) is implemented with new analysis.
2023-02-10 18:05:40 +00:00
Hubert Plociniczak
b56d6d74b9
Fix serialization of FQNs (#5600)
Serialization of FQNs' metadata was broken - we attempted to serialiaze concrete modules and that's against the design.

Added a test illustrating the problem which would previously fail during serialization.

Closes #5037

# Important Notes
It is still hard to discover problems like this because SerializationManager creates system threads; when the exception occurs, it is typically not the real cause. Creating regular threads via `createThread` seems to be problematic for `native-image`. There is a simple workaround for the former but will leave it out for another PR to simplify the review.
2023-02-10 12:28:34 +00:00
Pavel Marek
1f8511dab2
Add Comparator conversion for all types (#4067)
Add `Comparator` type class emulation for all types. Migrate all the types in stdlib to this new `Comparator` API. The main documentation is in `Ordering.enso`.

Fixes these pivotals:
- https://www.pivotaltracker.com/story/show/183945328
- https://www.pivotaltracker.com/story/show/183958734
- https://www.pivotaltracker.com/story/show/184380208

# Important Notes
- The new Comparator API forces users to specify both `equals` and `hash` methods on their custom comparators.
- All the `compare_to` overrides were replaced by definition of a custom _ordered_ comparator.
- All the call sites of `x.compare_to y` method were replaced with `Ordering.compare x y`.
- `Ordering.compare` is essentially a shortcut for `Comparable.from x . compare x y`.
- The default comparator for `Any` is `Default_Unordered_Comparator`, which just forwards to the builtin `EqualsNode` and `HashCodeNode` nodes.
- For `x`, one can get its hash with `Comparable.from x . hash x`.
- This makes `hash` as _hidden_ as possible. There are no other public methods to get a hash code of an object.
- Comparing `x` and `y` can be done either by `Ordering.compare x y` or `Comparable.from x . compare x y` instead of `x.compare_to y`.
2023-02-10 09:22:11 +00:00
Dmitry Bushev
b58a5f458b
Send method pointers of constructors (#5593)
Closes #5043

Expression updates of constructors contain method pointers.
2023-02-09 11:49:39 +00:00
Hubert Plociniczak
53b3de1cb1
Recover from serialization failures (#5591)
An exception encountered during serialization prevents engine from continuing because it enters an infinite loop(!).

# Important Notes
The aim of this PR is to make it possible for engine to recover from the serialization failures. Any failure would mean that we enter an infinite loop in deserialization which is in turn waiting for the serialization to finish (which will never happen).
In this particular case FQNs are [referencing concrete modules](https://github.com/enso-org/enso/issues/5037). A separate PR will address that.
2023-02-09 08:22:28 +00:00
Hubert Plociniczak
472580df4d
Report type of expressions returning polyglot values (#4111)
Expressions returning polyglot values were not reporting the type of the result because we have to do additional magic that infers the correct Enso type. Since this is exactly what `TypeOfNode` does, I re-used the logic.

Straightforward solution failed in tests because of assertions:
```
[enso] WARNING: Execution of function main failed (Invalid library usage. Cached library must be adopted by a RootNode before it is executed.).
java.lang.AssertionError: Invalid library usage. Cached library must be adopted by a RootNode before it is executed.
```

That is why this PR replaces `ExecutionEventListener` with `ExecutionEventNodeFactory`.

# Important Notes
Usage of `TypeOfNode` for programs that **do not** import stdlib means that we report types that do not involve stdlib e.g.
`Standard.Builtins.Main.Integer` instead of `Standard.Base.Data.Numbers.Integer`. While surprising, this is correct and I would say desirable. While reviewing the code, notice the difference in expectations in our runtime tests.
2023-02-09 01:06:27 +00:00
James Dunkerley
1c821e22cf
Some fixed form the Anagrams experiment. (#5592)
- Fixes the display of Date, Time_Of_Day and Date_Time so doesn't wrap.
- Adjust serialization of large integer values for JS and display within table.
- Workaround for issue with using `.lines` in the Table (new bug filed).
- Disabled warning on no specified `separator` on `Concatenate`.

Does not include fix for aggregation on integer values outside of `long` range.
2023-02-08 22:17:00 +00:00
Hubert Plociniczak
4405be8a74
Reduce JGit logs verbosity in dev mode (#4106) 2023-02-08 14:03:16 +00:00
Dmitry Bushev
53d5487f47
Lookup method pointers in IDE (#5578)
Closes #5036

Move the logic that looks up method pointers from the language server to IDE. This way we can keep the suggestion updates and expression updates async, otherwise it will hurt the initial startup time of LS.
Fixes the issue when some expression updates does not contain the method pointer.
2023-02-08 00:57:32 +00:00
Hubert Plociniczak
cc7d024495
Stabilize flaky test that re-uses test setup (#5577)
Follow up on
16ba57d465 which marked the tests as flaky.

This test goes back to the original implementation where JGit did all its book-keeping synchronously. When we switched to asynchronous operation, cleaning up the test directory would sometimes not succeed because there would be a race-condition with the ongoing work of JGit.

# Important Notes
In the current setup tagging a test as flaky is only valid for Windows. The other platforms ignore the flag. So we were still getting random failures.
2023-02-07 21:17:17 +00:00
James Dunkerley
c39410b4d1
Adjusting annotations to use Widget types properly. (#5564)
Closes #5038

- Use the proper widget structure.
- Provide new method `get_widget_json` with whole structure, but keep `get_full_annotations_json` in old form.
- Start to get to some reusable functions.
- Added widget to JS_Object field selections.
2023-02-07 07:33:51 +00:00
Jaroslav Tulach
04415a2b5e
Write the log in XML format suitable for VisualVM 'UI Actions' (#4110)
Start `project-manager` with following options to provide first 20s of the startup sequence:
```
$ project-manager --profiling-events-log-path=start.log --profiling-path=start.npss --profiling-time=20
```
once the `start.log` and `start.npss` files are generated (next to each other), open them in GraalVM's VisualVM:
```
$ graalvm/bin/jvisualvm --openfile start.npss
```
analyze.
2023-02-05 06:36:16 +00:00