Commit Graph

1238 Commits

Author SHA1 Message Date
Jaroslav Tulach
2330fdb8af
Provide Persistance for Persistance.Reference (#9326) 2024-03-08 18:23:17 +01:00
Hubert Plociniczak
f80dd9f593
Demonstrate usage of foreign arrow function (#9150)
Including arrow language in the distribution by default. Added a basic example for creating an Arrow array.
Making sure that memory layout agrees with Arrow specification (padding, continuous allocation of memory chunks).
Related to #9118.

This should unblock work on allowing serialization/deserialization to/from Parquet but I'd like to delay it to a follow up ticket as it is going to be a significant amount of specialized work.
2024-03-08 15:20:55 +00:00
Dmitry Bushev
ba1c1d9a94
Use to_display_text to preview warnings (#9325)
close #9200

Changelog:
- feat: use `.to_display_text` method to preview warnings

# Important Notes
![enso-warning-preview](https://github.com/enso-org/enso/assets/357683/ef825cbb-8c33-4d21-998f-80b85070fade)
2024-03-08 12:32:15 +00:00
Jaroslav Tulach
7e498a049d
Trading shouldNotReachHere() for an abstract method (#9312)
Avoiding `shouldNotReachHere()` in favor of less fatal exceptions/constructs.
2024-03-07 13:02:24 +00:00
Dmitry Bushev
acf124e089
Visualization tests maintenance (#9294)
Adjusted a couple of flaky tests reported by @mwu-tow
2024-03-06 15:48:56 +00:00
Jaroslav Tulach
6acec1b30b
Use .. to identify autoscoped constructors (#9285) 2024-03-06 10:28:15 +01:00
Pavel Marek
f02213ae2b
Enable splitting for EqualsSimpleNode (#9268)
Fixes the regression introduced by #9070 in `org.enso.benchmarks.generated.Collections.list_meta_fold` benchmark.

# Important Notes
As can be seen on the graph in IGV:
![image](https://github.com/enso-org/enso/assets/14013887/31b6ceca-4909-4a8f-987f-b456b3fb0a1b)
For some reason, `EqualsSimpleNode` is POLYMORPHIC. That seems to be the most visible performance problem.

First, I tried to introduce `ConditionProfile` with:
```diff
diff --git a/engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin/meta/EqualsNode.java b/engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin/meta/EqualsNode.java
index b368fb7fe..57274b37e 100644
--- a/engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin/meta/EqualsNode.java
+++ b/engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin/meta/EqualsNode.java
@@ -9,6 +9,7 @@ import com.oracle.truffle.api.dsl.Specialization;
import com.oracle.truffle.api.frame.VirtualFrame;
import com.oracle.truffle.api.interop.ArityException;
import com.oracle.truffle.api.nodes.Node;
+import com.oracle.truffle.api.profiles.ConditionProfile;
import org.enso.interpreter.dsl.AcceptsError;
import org.enso.interpreter.dsl.BuiltinMethod;
import org.enso.interpreter.node.EnsoRootNode;
@@ -46,6 +47,7 @@ public final class EqualsNode extends Node {
@Child private EqualsSimpleNode node;
@Child private TypeOfNode types;
@Child private WithConversionNode convert;
+  private final ConditionProfile equalsProfile = ConditionProfile.create();

private static final EqualsNode UNCACHED =
new EqualsNode(EqualsSimpleNodeGen.getUncached(), TypeOfNode.getUncached(), true);
@@ -85,7 +87,7 @@ public final class EqualsNode extends Node {
public boolean execute(
VirtualFrame frame, @AcceptsError Object self, @AcceptsError Object other) {
var areEqual = node.execute(frame, self, other);
-    if (!areEqual) {
+    if (!equalsProfile.profile(areEqual)) {
var selfType = types.execute(self);
var otherType = types.execute(other);
if (selfType != otherType) {

```
But that did not resolve the issue.

My second attempt was to enable splitting for `EqualsSimpleNode` with `@com.oracle.truffle.api.dsl.ReportPolymorphism` annotation, which seems to resolve the issue. The benchmark is back to its original score, and `EqualsSimpleNode` is no longer POLYMORPHIC.
2024-03-05 11:00:16 +00:00
Dmitry Bushev
02bd863c29
Add ascription type information to suggestions database (#9267)
close #9258

Changelog:
- fix: add information about the type descriptions to the suggestion database
2024-03-05 09:02:41 +00:00
Jaroslav Tulach
5676618bad
Autoscoped constructors (#9190)
Fixes #8645 by recognizing `~` prefix to constructor names.
2024-03-04 11:41:02 +00:00
Dmitry Bushev
fad1b45bcf
Fix attaching generic annotations (#9234)
Fixes the issue with attaching generic annotations in complex types.

Annotations in the type body could be lost during the compilation if its constructor was defined at the end of the type definition.
2024-03-01 18:01:50 +00:00
Pavel Marek
96bfcb3e63
Add compiler benchmarks (#9158)
Add compiler benchmarks to `engine/runtime-benchmarks`. All the benchmarks generate source code on the fly into `engine/runtime-benchmarks/target/bench-data` directory. Random data generators are set with the same seed. For the convenience of reviewers, I am attaching the benchmark sources in [bench-data.zip](https://github.com/enso-org/enso/files/14423372/bench-data.zip).

I have created benchmarks that measure the performance of a whole module compilation, and benchmarks that measure the performance of inline compilation. They directly call `run` and `runInline` methods on `org.enso.compiler.Compiler`.

# Important Notes
- The results will be available in https://enso-org.github.io/engine-benchmark-results/engine-benchs.html in a few days after the merge of this PR.
- The benchmark parameters are tweaked so that an average iteration takes less than 400 ms and more than 30 ms.
- Ensured that the benchmarks measure performance of the compiler, for example:
![image](https://github.com/enso-org/enso/assets/14013887/c870f4ad-1418-4812-85f2-ca9664711163)
2024-03-01 13:37:18 +00:00
Radosław Waśko
386132cdab
Fix matching JS strings (#9203)
- Fixes #9202
2024-02-29 15:07:29 +00:00
Dmitry Bushev
6aa213757b
Unexpected warning when running a standalone script (#9114)
close #9109

Fixes the issue when the runner displays unexpected log messages
```
> .\built-distribution\enso-engine-0.0.0-dev-windows-amd64\enso-0.0.0-dev\bin\enso --run .\test.enso
[WARN] [2024-02-20T12:04:21+01:00] [enso.org.enso.interpreter.runtime.SerializationPool] Serialization of module `test` failed: Unable to write cache data for test.`
42
```
2024-02-23 18:19:06 +00:00
Dmitry Bushev
7ff90aa3f7
Synchronize suggestions loading after the reconnect 2 (#9142)
related #8689, #9072

Fixes a race between the language server SQL updating logic and the engine `DeserializeLibrarySuggestionsJob`s when the library suggestions may start loading before the database is properly cleaned up after the reconnect.

# Important Notes
As a side effect, arguments are showing slightly (~1 second) faster due to the lower contention between the engine jobs.

#### Before
https://github.com/enso-org/enso/assets/357683/cbda2da4-9080-4b9b-b836-81e54694d468

#### After
https://github.com/enso-org/enso/assets/357683/bf442284-47be-456d-b1dd-2413b6ad8244
2024-02-23 11:18:01 +00:00
Pavel Marek
21e1284086
Enso tests can be run with filter from cmdline (#9065)
Simplify the `Test.Suite.run_with_filter` to accept a single filter parameter that searches for all the groups and specs that matches that filter. This filter can be a simple text provided from the command line.

# Important Notes
- Pending groups are now printed at the end of the run
- `Test.Suite.run_with_filter` is simplified to accept a single filter parameter that is either `Text` or `Nothing`. See the docs.
- Passing a filter from the command line is therefore straightforward, it is treated as a regex.
- For convenience, I have left all the `main` methods in all the test sources. I have just refactored them to accept the `filter` argument from the command line.
- For example, to run only a single spec from `Vector_Spec.enso`, invoke `enso --run test/Base_Tests/src/Data/Vector_Spec.enso "should allow vector creation with a programmatic constructor"`
- **Majority of the PR is a regex replace** of `^main =` for `main filter=Nothing =` and of `suite.run_with_filter` for `suite.run_with_filter filter`.
- **Fixed some internal engine bugs:**
- `AtomWithHole` allows to specify only one hole - https://github.com/enso-org/enso/pull/9065/files#diff-0f7bb7e85cf86a965de133aa7e6b5958ceb889bd1921c01e00d3a9ceb19626ef
- NaN keys in hash maps are handled in polyglot maps as well - c5257f6c2b78f893214ff67300893b593ea05e21..db4b3c0e9828ee79208d52e02586b24bb845b0d6
2024-02-22 12:31:44 +00:00
Hubert Plociniczak
0133e9ef4f
Drop inefficient SemVer implementation (#9089)
`Bump` library uses parser combinators behind the scenes which are known to be good at expressing grammars but are not performance-oriented.

This change ditches the dependency in favour of an existing Java implementation. `jsemver` implements the full specification, which is probably an overkill in our case, but proved to be an almost drop-in replacement for the previous library.

Closes #8692

# Important Notes
Peformance improvements:
- roughly 50ms compared to the previous approach (from 80ms to 20-40ms)

I don't see any time spent in the new implementation during startup so it could be potentially aggressively inlined.
Further more, we could use a facade and offer our own strip down version of semver.
2024-02-22 09:59:09 +00:00
Jaroslav Tulach
462c96805b
Avoid excessive logging during development (#9101) 2024-02-20 13:49:54 +01:00
Pavel Marek
96082c3bae
Remove akka from runtime (#8953)
There are two projects transitively required by `runtime`, that have akka dependencies:
- `downloader`
- `connected-lock-manager`

This PR replaces the `akka-http` dependency in `downloader` by HttpClient from JDK, and splits `connected-lock-manager` into two projects such that there are no akka classes in `runtime.jar`.

# Important Notes
- Simplify the `downloader` project - remove akka.
- Add HTTP tests to the `downloader` project that uses our `http-test-helper` that is normally used for stdlib tests.
- It required few tweaks so that we can embed that server in a unit test.
- Split `connected-lock-manager` project into two projects - remove akka from `runtime`.
- **Native image build fixes and quality of life improvements:**
- Output of `native-image` is captured 743e167aa4
- The output will no longer be intertwined with the output from other commands on the CI.
- Arguments to the `native-image` are passed via an argument file, not via command line - ba0a69de6e
- This resolves an issue on Windows with "Command line too long", for example in https://github.com/enso-org/enso/actions/runs/7934447148/job/21665456738?pr=8953#step:8:2269
2024-02-19 16:39:05 +00:00
Jaroslav Tulach
a664dd9d56
Equality with conversions (#9070) 2024-02-19 17:18:56 +01:00
Radosław Waśko
cf71a05c3b
Refactor common context handling pattern (#9087) 2024-02-19 11:59:42 +00:00
Dmitry Bushev
a25d716932
Keep single execute expression job in the queue (#9077)
close #8965

Changelog:
- update: keep a single `ExecuteExpressionJob` in the queue
- update: make `ExecuteExpressionCommand` synchronous to preserve the order of commands
- refactor: separate data structures for `Visualization` and `OneshotExpression` to simplify the logic
2024-02-19 09:18:04 +00:00
Hubert Plociniczak
fe0f9046db
Introduce hash seed to invaldiate caches (#9082) 2024-02-16 23:43:30 +00:00
Hubert Plociniczak
d29c2cd66a
Serialize UUID for non-library modules (#9057)
Missing ID's in IR meant that instrumentation wouldn't be applied for loaded modules. This is the reason why after a restart engine wouldn't send **any** expression updates.

Closes #8689.

# Important Notes
After the change
[Kazam_screencast_00038.webm](https://github.com/enso-org/enso/assets/292128/4249287b-6c41-4c9d-b138-e7af59512566)

The video somehow doesn't show that all nodes are loaded after the restart, but once I moved the screen they are there. This appears to be a bug in the recording somehow.
2024-02-15 16:50:27 +00:00
Hubert Plociniczak
5c624d21d7
Revert "Synchronize suggestions loading after the reconnect (#9043)" (#9072) 2024-02-15 13:39:56 +00:00
Dmitry Bushev
702e6c1bb8
Set default timeouts in incremental updates test (#9053)
Fix random failures of `IncrementalUpdatesTest` on CI https://github.com/enso-org/enso/actions/runs/7885050954/job/21515333651#step:8:5274
2024-02-14 17:07:44 +00:00
Radosław Waśko
d45f0fe4df
Check type of self in static dispatch (#8867)
- Fixes #8805
- also btw. fixes #8706
2024-02-14 15:50:13 +00:00
Dmitry Bushev
6efa26b48e
Add support for positionalArgumentsExpressions of text visualizations (#9052)
close #9028

Changelog:
- add: support `positionalArgumentsExpressions` parameter for visualization expressions represented as strings
- add: language server API tests
- add: runtime visualization tests

# Important Notes
https://github.com/enso-org/enso/assets/357683/4e6f207b-ed64-41b9-b94b-96fb9db63ce4
2024-02-13 23:08:30 +00:00
Dmitry Bushev
9c982e07b0
Synchronize suggestions loading after the reconnect (#9043)
related #8689

Fixes a race between the language server SQL updating logic and the engine `DeserializeLibrarySuggestionsJob`s when the library suggestions may start loading before the database is properly cleaned up after the reconnect.
2024-02-13 17:52:15 +00:00
GregoryTravis
3bdd1a0dce
Re-fetch Warnings in Warning.getWarnings() to correctly wrap them (#9011) 2024-02-13 15:12:32 +00:00
Jaroslav Tulach
04161b33e4
Don't print stacktraces of IOException (#9024)
Fixes #8896 by logging `IOException` only with `WARNING` and not `SEVERE`. As such the stacktrace of the exception isn't included in the console and failures to store cache are reported as simple messages, not exceptions with stack trace.
2024-02-13 14:01:42 +00:00
Pavel Marek
5919eda753
Fix incremental compilation of runtime/test (#8975) 2024-02-13 10:05:31 +01:00
Jaroslav Tulach
6ff36e570f
Propagate location to resolution errors (#9025) 2024-02-12 10:25:38 +01:00
Hubert Plociniczak
b00dc9e9c0
Minor cleanups to docker image (#8925)
* Minor cleanups to docker image

* format

* nit
2024-02-11 21:52:47 +01:00
AdRiley
1dc8c1cb24
Merge empty_error (#9000)
We don't need 3 copies of empty_error. This PR makes them one.
2024-02-09 14:18:14 +00:00
Dmitry Bushev
880da9fb56
Cleanup execution contexts when the client disconnects (#9004)
related #8689

Clean up the client's execution contexts when it disconnects from the language server. Dangling execution contexts may slow down the execution when the user reconnects to the language server.
2024-02-09 14:16:34 +00:00
Ikko Eltociear Ashimine
66dfefe810
Fix typo in ContextEventsListener.scala (#8958) 2024-02-09 14:01:02 +00:00
Dmitry Bushev
200a494242
Escape default text arguments in suggestions database (#9010)
Changelog:
- fix: escape default text arguments in suggestions database
2024-02-09 12:57:25 +00:00
Jaroslav Tulach
9a91b7bcc6
Store whole IR.Module in .bindings cache (#8924) 2024-02-09 04:51:45 +01:00
Dmitry Bushev
04e53cc4de
Fix NPE when recomputing the context (#8999)
related #8689

Fixes the NPE during the serialization of update messages.

```
java.lang.NullPointerException: Cannot invoke "java.util.UUID.toString()" because "a" is null
```
2024-02-08 15:40:25 +00:00
Pavel Marek
83fffd9c05
Refactor stdlib tests to the builder API (#8968)
Follow-up of #8890

Refactor the rest of the tests to the builder API (`Test_New`):
- `Image_Tests`
- `Geo_Tests`
- `Google_Api_Test`
- `Examples_Test`
- `AWS_Tests`
- `Meta_Test_Suite_Tests`
- `Visualization_Tests`

# Important Notes
- Unrelated: Fix NPE in `File.new "/" . name`
2024-02-07 13:22:17 +00:00
Hubert Plociniczak
b8612344ca
Disable flaky test (#8967) 2024-02-05 13:41:43 +00:00
Hubert Plociniczak
34e1bacdf1
Further optimizations to test resources (#8936)
TestRuntime should be deprecated as it creates a number of threads and doesn't allow to easily modify ZIO's runtime.
But the biggest drop stems from fixing leaking `FileSystemService` that weren't being closed for every `TextOperationsTest` test.
The change is a follow up on #8892 but this time focused on ZIO usage.

Hopefully fixes #8806 for good.

# Important Notes
Running `language-server/test`.
Before:
![Screenshot from 2024-02-02 09-48-32](https://github.com/enso-org/enso/assets/292128/fb414c74-7d7a-4e7b-8b0c-d25dc3721bbf)

After:
![Screenshot from 2024-02-02 09-46-02](https://github.com/enso-org/enso/assets/292128/db9429df-d861-4f48-818f-888d5bbbb089)
2024-02-02 22:49:48 +00:00
Pavel Marek
4ea2ad5281
Fix Truffle invariants for Nothing with a warning (#8764) 2024-02-02 15:57:29 +00:00
Pavel Marek
a70cbacecf
Remove fansi dependency from runtime-compiler (#8847)
Moves `fansi` dependency from `runtime-compiler` into `runtime`.

# Important Notes
I have not refactored [DiagnosticFormatter.scala](https://github.com/enso-org/enso/pull/8847/files#diff-8e73cf562742d6b0510acfe30af940fb9252e32be27a023f9705908a464e08ed) into Java just yet - I don't know what should be the replacement for now. I have just moved that source from `runtime-compiler` to `runtime`.
2024-02-02 11:45:19 +00:00
Hubert Plociniczak
68ce4ae1b7
Reduced resources for various Akka dispatchers (#8892)
The defaults picked up by Akka tend to make us of all resources which is unnecessary and overwhelming for tests.

Improves #8806, potentially.

Before
![Screenshot from 2024-01-28 22-34-42](https://github.com/enso-org/enso/assets/292128/f80eb66a-2f37-44d5-bcdb-f00a78fe72fd)
After
![Screenshot from 2024-01-31 00-12-10](https://github.com/enso-org/enso/assets/292128/c5223912-5f6e-413c-a0a4-050afa3ed463)

when running the problematic `LibrariesTest`.

Full `language-server` test suite.
Before
![Screenshot from 2024-01-31 00-20-50](https://github.com/enso-org/enso/assets/292128/f1c94a66-6905-4f57-8a7d-7df049714353)
After
![Screenshot from 2024-01-31 00-18-40](https://github.com/enso-org/enso/assets/292128/3a11125e-d593-43df-8d35-1a8915812b2b)

# Important Notes
Note that Executors assigned to Zio and initializers should also be improved. Unfortunately due to various blocking threadpools  it is easy to get timeouts when running the whole suite.
2024-01-31 10:25:17 +00:00
Hubert Plociniczak
081c8c889c
Don't cancel pending visualization's upserts (#8853)
Uniqueness check of `UpsertVisualizationJob` only involved expressionId. Apparently now GUI sends mutliple visualizations for the same expressions and expects all of them to exist. Since previously we would cancel duplicate jobs, this was problematic.

This change makes sure that uniqueness also takes into account visualization id. Fixed a few logs that were not passing arguments properly.

Closes #8801

# Important Notes
I have not noticed any more problems with loading visualizations so the issue appears to be resolved with this change.
Added a unit test case that would previously fail due to cancellation of a job that upserts visualization.
2024-01-30 00:13:43 +00:00
Jaroslav Tulach
084245868d
Canonicalize the temporary directory files (#8879)
Due to some Mac OS X symlinks trickery we need to canonicalize files created by `@Rule TemporaryFolder` to compare them for equality.
2024-01-29 09:06:43 +00:00
Jaroslav Tulach
74b808b00d
Argument block application has no place in type body (#8877)
Fixes #8832 by reporting a `SyntaxError` in `TreeToIr` conversion.
2024-01-28 20:29:14 +00:00
Jaroslav Tulach
9a37357247
Binary operator resolution based on that value (#8779) 2024-01-27 08:38:47 +01:00
Hubert Plociniczak
24007f1db4
Add persistance to Operator.Binary (#8826) 2024-01-23 10:43:26 +01:00
Hubert Plociniczak
dfe867a9cd
Edits are processed in the order of submission (#8787)
This is a quick fix to a long standing problem of
`org.enso.interpreter.service.error.FailedToApplyEditsException` which would prevent backend from processing any more changes, rendering GUI (and backend) virtually useless.
Edits are submitted for (background) processing in the order they are handled. However the order of execution of such tasks is not guaranteed. Most of the time edits are processed in the same order as their requests but when they don't, files get quickly out of sync.

Related to #8770.

# Important Notes
I'm not a fan of this change because it essentially blocks all open/file requests until all edits are processed and we already have logic to deal with that appropriately. Moreover those tasks can and should be processed independently. Since we already had the single thread executor present to ensure correct synchronization of open/file/push commands, we are simply adding edit commands to the list.

Ideally we want to have a specialized executor that executes tasks within the same group sequentially but groups of tasks can be executed in parallel, thus ensuring sufficient throughput. The latter will take much longer and will require significant rewrite of the command execution.

Added tests that would previously fail due to non-deterministic execution.
2024-01-22 23:05:41 +00:00
Radosław Waśko
368e4867b4
Allow secrets in AWS_Credential (#8774)
- Closes #8722
2024-01-19 19:00:56 +00:00
Jaroslav Tulach
6578f35e22
Binary operator must have two arguments (#8789) 2024-01-18 11:12:13 +01:00
Jaroslav Tulach
a764618bd9
Derive --in-project from --run source location (#8775) 2024-01-17 17:19:42 +01:00
Jaroslav Tulach
09f484f00d
Make suspended atom fields work for boxed atoms (#8712)
Fixes #8710 by making sure suspended atom fields support works also for "normal" `Atom` instances without any special `Layout`. Moves all _atom related_ classes into single package and hides as much of classes as possible by making them _package private_.
2024-01-16 20:20:38 +00:00
Dmitry Bushev
51540e2eb2
Add attributes to filesystem events (#8767)
close #8200

Changelog:
- add: `attributes` field to the file event notification
2024-01-16 13:48:53 +00:00
GregoryTravis
f2cb1f097e
Support on_problems=Problem_Behavior.Report_Warning and Map_Error wrapping in Vector.map (#8595)
Implements `Warnings.get_all wrap_errors=True` which wraps warnings attached to values inside vectors with `Map_Error`, which includes the position of the value within the vector. See [the documentation](https://github.com/enso-org/enso/blob/develop/docs/semantics/wrapped-errors.md) for more details.

`get_all wrap_errors=True` does not change the warnings that are attached to values -- it wraps them before returning them to the caller, but does not change the original warnings attached to the values.

Wrapped warnings only appear attached to the vector itself. The values inside the vector do not have their warnings wrapped.

Warning propagation is not changed at all; `Warnings.get_all` (with default `wrap_errors=False`) behaves as before. `get_all wrap_errors=True` is meant to be used primarily by the IDE, although it can be used anywhere this wrapping is desired.
2024-01-16 09:36:22 +00:00
Hubert Plociniczak
5b91f16498
Generate UUIDs on demand (#8728)
Trying to avoid expensive `UUID.randomUUID()` unless we reallly need it.

Closes #8716.

# Important Notes
Some improvement:
![Screenshot from 2024-01-11 15-16-10](https://github.com/enso-org/enso/assets/292128/d8800490-6676-4b71-b178-7ce2e79942e5)

FWIW Total Time for a Hello World example

Before
![Screenshot from 2024-01-12 17-45-56](https://github.com/enso-org/enso/assets/292128/c0bfe7c5-c0a5-4375-8dd9-afb0714ae6c4)

After
![Screenshot from 2024-01-12 17-46-13](https://github.com/enso-org/enso/assets/292128/ea76c413-018f-4b67-9777-85378eb38210)

Memory usage

Before
![Screenshot from 2024-01-12 17-54-54](https://github.com/enso-org/enso/assets/292128/280b1eff-e019-4241-a2a1-07445949d285)

After
![Screenshot from 2024-01-12 17-54-36](https://github.com/enso-org/enso/assets/292128/b6524c8b-2a38-4e51-85eb-63142420f2ff)
2024-01-12 21:50:12 +00:00
Dmitry Bushev
972b359789
Return function schema for atom constructors (#8743)
close #8663

Changelog:
- update: use `MethodRootNode` for the atom constructor function to preserve the call info in runtime
- fix: return function schema for atom constructors
2024-01-12 19:27:51 +00:00
Hubert Plociniczak
3c29a58829
Arrow language (#8512)
Initial implementation of the Arrow language. Closes #7755.
Currently supported logical types are
- Date (days and milliseconds)
- Int (8, 16, 32, 64)

One can currently
- allocate a new fixed-length, nullable Arrow vector - `new[<name-of-the-type>]`
- cast an already existing fixed-length Arrow vector from a memory address - `cast[<name-of-the-type>]`

Closes #7755.
2024-01-12 18:19:36 +00:00
Pavel Marek
6ae35abc46
Fix Runtime.assert (#8742) 2024-01-12 18:47:40 +01:00
Jaroslav Tulach
0e6952710a
Executing (parts of) Truffle TCK with Enso values (#8685) 2024-01-12 07:21:16 +01:00
Hubert Plociniczak
a94cad6bfa
Optimize AliasAnalysis hotspot (#8701)
`scopeFor` is a real hotspot that hasn't been particularly optimized. By using iterator and a simple variable instead of a list and checking its length I was able to get 10% speedup.
Note that it seems tempting to throw the exception within the loop but that seems to create a less optimized code.

# Important Notes
I'm consistently getting ~10% speedup on a hello world example (with standard libraries).

For example I'm no longer seeing 20ms spent in `scopeFor` and most of them are below 10ms:
![Screenshot from 2024-01-08 12-15-30](https://github.com/enso-org/enso/assets/292128/033ece07-eb15-45b5-971f-417bf9f17ef7)

Before
![Screenshot from 2024-01-08 12-17-32](https://github.com/enso-org/enso/assets/292128/3848d7c2-0fe8-4951-b222-c8f40d2daf01)

After
![Screenshot from 2024-01-08 12-17-09](https://github.com/enso-org/enso/assets/292128/524496f5-3cf0-47f4-8b05-edd330080b14)
2024-01-11 15:49:39 +00:00
Hubert Plociniczak
31a0dcef65
Suppress logs for test cases unless a failure is reported (#8694)
The change adds a convenient trait `ReportLogsOnFailure` that, when merged with the test class, will keep logs in memory and only delegate to the underlying appender on failure. For now we only support forwarding to the console which is sufficient.
A corresponding entry in `application-test.conf` has to point to the new `memory` appender. The additional complexity in the implementation ensures that if someone forgets to mixin `ReportLogsOnFailure` logs appear as before i.e. they respect the log level.

As a bonus fixed arguments passed to ScalaTest in build.sbt so that we are now, again, showing timings of individual tests.

Closes #8603.

# Important Notes
Before:
```
[info] VcsManagerTest:
[info] Initializing project
[ERROR] [2024-01-04 17:27:03,366] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must create a repository (3 seconds, 538 milliseconds)
[info] - must fail to create a repository for an already existing project (141 milliseconds)
[info] Save project
[ERROR] [2024-01-04 17:27:08,346] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must create a commit with a timestamp (198 milliseconds)
[ERROR] [2024-01-04 17:27:08,570] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must create a commit with a name (148 milliseconds)
[ERROR] [2024-01-04 17:27:08,741] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must force all pending saves (149 milliseconds)
[info] Status project
[ERROR] [2024-01-04 17:27:08,910] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must report changed files since last commit (148 milliseconds)
[info] Restore project
[ERROR] [2024-01-04 17:27:09,076] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must reset to the last state with committed changes (236 milliseconds)
[ERROR] [2024-01-04 17:27:09,328] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must reset to a named save (pending)
[ERROR] [2024-01-04 17:27:09,520] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must reset to a named save and notify about removed files *** FAILED *** (185 milliseconds)
[info]   Right({
[info]     "jsonrpc" : "2.0",
[info]     "method" : "file/event",
[info]     "params" : {
[info]       "path" : {
[info]         "rootId" : "cd84a4a3-fa50-4ead-8d80-04f6d0d124a3",
[info]         "segments" : [
[info]           "src",
[info]           "Bar.enso"
[info]         ]
[info]       },
[info]       "kind" : "Removed"
[info]     }
[info]   }) did not equal Right({
[info]     "jsonrpc" : "1.0",
[info]     "method" : "file/event",
[info]     "params" : {
[info]       "path" : {
[info]         "rootId" : "cd84a4a3-fa50-4ead-8d80-04f6d0d124a3",
[info]         "segments" : [
[info]           "src",
[info]           "Bar.enso"
[info]         ]
[info]       },
[info]       "kind" : "Removed"
[info]     }
[info]   }) (VcsManagerTest.scala:1343)
[info]   Analysis:
[info]   Right(value: Json$JObject(value: object[jsonrpc -> "2.0",method -> "file/event",params -> {
[info]   "path" : {
[info]     "rootId" : "cd84a4a3-fa50-4ead-8d80-04f6d0d124a3",
[info]     "segments" : [
[info]       "src",
[info]       "Bar.enso"
[info]     ]
[info]   },
[info]   "kind" : "Removed"
[info] }] -> object[jsonrpc -> "1.0",method -> "file/event",params -> {
[info]   "path" : {
[info]     "rootId" : "cd84a4a3-fa50-4ead-8d80-04f6d0d124a3",
[info]     "segments" : [
[info]       "src",
[info]       "Bar.enso"
[info]     ]
[info]   },
[info]   "kind" : "Removed"
[info] }]))
[ERROR] [2024-01-04 17:27:09,734] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] List project saves
[info] - must return all explicit commits (146 milliseconds)
[info] Run completed in 9 seconds, 270 milliseconds.
[info] Total number of tests run: 9
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 8, failed 1, canceled 0, ignored 0, pending 1
[info] *** 1 TEST FAILED ***
```

After:
```
[info] VcsManagerTest:
[info] Initializing project
[info] - must create a repository (3 seconds, 554 milliseconds)
[info] - must fail to create a repository for an already existing project (164 milliseconds)
[info] Save project
[info] - must create a commit with a timestamp (212 milliseconds)
[info] - must create a commit with a name (142 milliseconds)
[info] - must force all pending saves (185 milliseconds)
[info] Status project
[info] - must report changed files since last commit (142 milliseconds)
[info] Restore project
[info] - must reset to the last state with committed changes (202 milliseconds)
[info] - must reset to a named save (pending)
[ERROR] [2024-01-04 17:24:55,738] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/8456553964637757156].
[info] - must reset to a named save and notify about removed files *** FAILED *** (186 milliseconds)
[info]   Right({
[info]     "jsonrpc" : "2.0",
[info]     "method" : "file/event",
[info]     "params" : {
[info]       "path" : {
[info]         "rootId" : "965ed5c8-1760-4284-91f2-1376406fde0d",
[info]         "segments" : [
[info]           "src",
[info]           "Bar.enso"
[info]         ]
[info]       },
[info]       "kind" : "Removed"
[info]     }
[info]   }) did not equal Right({
[info]     "jsonrpc" : "1.0",
[info]     "method" : "file/event",
[info]     "params" : {
[info]       "path" : {
[info]         "rootId" : "965ed5c8-1760-4284-91f2-1376406fde0d",
[info]         "segments" : [
[info]           "src",
[info]           "Bar.enso"
[info]         ]
[info]       },
[info]       "kind" : "Removed"
[info]     }
[info]   }) (VcsManagerTest.scala:1343)
[info]   Analysis:
[info]   Right(value: Json$JObject(value: object[jsonrpc -> "2.0",method -> "file/event",params -> {
[info]   "path" : {
[info]     "rootId" : "965ed5c8-1760-4284-91f2-1376406fde0d",
[info]     "segments" : [
[info]       "src",
[info]       "Bar.enso"
[info]     ]
[info]   },
[info]   "kind" : "Removed"
[info] }] -> object[jsonrpc -> "1.0",method -> "file/event",params -> {
[info]   "path" : {
[info]     "rootId" : "965ed5c8-1760-4284-91f2-1376406fde0d",
[info]     "segments" : [
[info]       "src",
[info]       "Bar.enso"
[info]     ]
[info]   },
[info]   "kind" : "Removed"
[info] }]))
[info] List project saves
[info] - must return all explicit commits (131 milliseconds)
[info] Run completed in 9 seconds, 400 milliseconds.
[info] Total number of tests run: 9
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 8, failed 1, canceled 0, ignored 0, pending 1
[info] *** 1 TEST FAILED ***
```
2024-01-09 09:59:10 +00:00
Jaroslav Tulach
d86c6c472c
Replace Jackson library by simple data I/O streams (#8693) 2024-01-08 13:17:37 +01:00
Jaroslav Tulach
542357addc
Instructions to build Enso with Espresso for GraalVM for JDK21 (#8641) 2024-01-05 10:18:39 +01:00
Jaroslav Tulach
8396bfa165
Avoid using cached bindings when the module isn't loaded from cache (#8669) 2024-01-05 08:03:34 +01:00
Hubert Plociniczak
41fe87f2ff
Test benchmark theory (#8668) 2024-01-04 16:28:56 +00:00
Pavel Marek
428e83de36
Remove org.bouncycastle dependency (#8664)
Remove `org.bouncycastle` dependency from `org.enso.runtime`.
2024-01-04 17:16:41 +01:00
Jaroslav Tulach
4222ddccb5
Canonicalize file names before comparing them (#8670) 2024-01-04 15:21:13 +01:00
Hubert Plociniczak
20531d51df
Initialize builtin methods lazily in non-AOT mode (#8654)
Registration of builtin methods is done upfront, similarly to AOT mode, but the initialization is delayed until the first use.
One still has to read the full list of builtin methods and register them in the builtins scope or we would be getting the `NoSuchMethod` errors otherwise. But the most expensive operation, initialization of classes and getting a method via a reflection, is all done on the first usage.

The result is an improved startup and negligible performance impact in regular usage.

Closes #8423.

# Important Notes
Visible speedup.
1. (old) Program returning plain "Hello World" (no stdlib loading)
![Screenshot from 2024-01-02 17-34-35](https://github.com/enso-org/enso/assets/292128/b8348cbb-baf6-4292-8c0e-c8a3492c6583)
2. (new) Program returning plain "Hello World" (no stdlib loading)
![Screenshot from 2024-01-02 17-34-54](https://github.com/enso-org/enso/assets/292128/63dd6c4f-49e2-4a5e-ab95-2460c155b91c)
About 10% improvement for the overall execution.

3. (old) Program printing "Hello World" (some stdlib loading)
![Screenshot from 2024-01-02 17-40-21](https://github.com/enso-org/enso/assets/292128/cf3f464c-8f00-4adf-9499-a6ac86430da0)
4. (new) Program printing "Hello World" (some stdlib loading)
![Screenshot from 2024-01-02 17-40-41](https://github.com/enso-org/enso/assets/292128/3ef99893-ccaf-4ea7-90ba-aa090f904a50)
About 2% improvement for the overall execution.

Minor price to pay for lazy initialization:
![Screenshot from 2024-01-02 17-56-12](https://github.com/enso-org/enso/assets/292128/faa682d8-5370-43d2-9fc9-3454af5749a4)
2024-01-04 08:55:21 +00:00
Jaroslav Tulach
5e3480b1e8
Apply SectionsToBinOp pass to arguments of Section (#8655)
Fixes #8436 by making sure that `SectionsToBinOp` pass is applied also to arguments of `Section`.
2024-01-03 14:37:11 +00:00
Dmitry Bushev
cfab344fbe
Unable to access constructor value in interactive mode (#8626)
close #7184

The constructor value was not accessible because during the re-compilation a new instance of the type was registered in runtime. Then during the execution, an old cached instance of the type was used in method resolution.

Changelog:
- update: the registration of types in runtime
- update: invalidate cached nodes that became a resolution error after applying the edit
2024-01-03 13:23:42 +00:00
Pavel Marek
819741bb4a
Increase warmup for EqualsBenchmarks. (#8636)
After #8620, there is a noticeable slowdown in `EqualsBenchmarks.equalsTrees` as suggested in https://github.com/enso-org/enso/pull/8620#issuecomment-1870776609. After some digging, I realized that the number of warmup iterations is most probably insufficient. Let's increase the warmup for this benchmark and see if we can get its score down again.
2023-12-29 08:47:36 +00:00
Jaroslav Tulach
27431c126c
Make IR caches smaller by storing Graph as a reference (#8635) 2023-12-29 06:15:48 +01:00
Pavel Marek
48a3c14ee5
Format Java sources in benchmark configuration (#8638)
I noticed that sources in `runtime/bench` are not formatted at all. Turns out that the `JavaFormatterPlugin` does not override `javafmt` task for the `Benchmark` configuration. After some failed attempts, I have just redefined the `Benchmark/javafmt` task in the `runtime` project. After all, the `runtime` project is almost the only project where we have any Java benchmarks.

# Important Notes
`javafmtAll` now also formats sources in `runtime/bench/src/java`.
2023-12-28 17:06:06 +00:00
Dmitry Bushev
2d628263ff
Update cache invalidation commands (#8634) 2023-12-26 12:39:17 +00:00
Jaroslav Tulach
07d58f2c02
DataflowError.withoutTrace shall not store a trace (#8608) 2023-12-24 11:07:32 +01:00
Pavel Marek
74436830ce
Fix build of engine benchmarks (#8620)
After #8467, Engine benchmarks are broken, they cannot compile - https://github.com/enso-org/enso/actions/runs/7268987483/job/19805862815#logs

This PR fixes the benchmark build

# Important Notes
Apart from fixing the build of `engine/bench`:
- Don't assemble any fat jars in `runtime/bench`.
- Use our `TestLogProvider` in the benches instead of NOOP provider.
- So that we can at least see warnings and errors in benchmarks.
2023-12-22 11:40:32 +00:00
Pavel Marek
edc98593b9
All unit tests use the same testing logging provider (#8593)
Make sure that the correct test logging provider is loaded in `project-manager/Test`, so that only WARN and ERROR log messages are displayed. Also, make sure that the test log provider parses the correct configuration file - Rename all the `application.conf` files in the test resources to `application-test.conf`.

The problem was introduced in #8467
2023-12-21 13:45:33 +00:00
Radosław Waśko
dfdb547616
Better context info in Type_Error raised from return type checks (#8566)
- Followup to #8502 that adds better error messages
2023-12-20 18:22:47 +00:00
Pavel Marek
4cb2439890
Submodules can be private (#8581) 2023-12-19 19:13:44 +01:00
Jaroslav Tulach
7daad75dd9
Avoid WithWarnings without any warnings (#8583) 2023-12-19 17:03:16 +01:00
Radosław Waśko
f0c2a5fa7f
Opt-in return type checks (#8502)
- Closes #8240
2023-12-19 15:32:30 +00:00
Hubert Plociniczak
1ad11104d3
Stop parsing METADATA section for IdMap (#8574)
We infer old IDs from module's IRs location, instead. This workarounds ill-constructed IdMaps at the start of the project.

If IDE or backend ends up with invalid ID map, I suppose application of edits might still fail.

# Important Notes
Closes #8500 by not doing the parsing at all. Since I couldn't figure why we had invalid metadata in the first place, this might still bite us at some point.
2023-12-19 12:03:33 +00:00
Pavel Marek
21d164ec3e
Run unit tests with truffle-compiler (#8467) 2023-12-18 18:22:16 +01:00
Dmitry Bushev
b2a1cd55d2
Syntax error results in silent error (#8560)
close #7555

Compiler passes after `GenerateMethodBodies` expect the method body to be a function.

After fixing the pass, the compilation returns a proper compiler error:
```
built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Table/0.0.0-dev/src/Data/Column.enso:869:22: error: Methods must have only one definition of the `this` argument, and it must be the first.
869 |     round self round self (decimal_places:Integer = 0) (use_bankers:Boolean = False) = Value_Type.expect_numeric self <|
|                      ^~~~
Aborting due to 1 errors and 0 warnings.
```
2023-12-18 13:12:31 +00:00
Pavel Marek
c1098865f2
Update java formatter sbt plugin (#8543)
Add a local clone of javaFormatter plugin. The upstream is not maintained anymore. And we need to update it to use the newest Google java formatter because the old one, that we use, cannot format sources with Java 8+ syntax.

# Important Notes
Update to Google java formatter 1.18.1 - https://github.com/google/google-java-format/releases/tag/v1.18.1
2023-12-15 14:45:23 +00:00
Dmitry Bushev
56cc9561b1
Fix serialization of alias analysis graph (#8550)
close #8431

Fixes the scenario:
- user sends `executionContext/executeExpression`
- program execution is scheduled
- during the compilation the already compiled `IR` is loaded from the cache (reading invalid alias analysis graph)
- during the codegen the local scope with that aliasing graph is propagated to the runtime
- `EvalNode` compiles the expression to execute with the local scope containing an invalid aliasing graph
- compilation fails in the `AliasAnalysis` pass because of the clashing IDs in the graph
2023-12-15 12:54:52 +00:00
Pavel Marek
4b65e44ef3
EpbLanguage re-uses other TruffleContext support to run tests with assertions enabled (#7882) 2023-12-15 13:31:32 +01:00
Radosław Waśko
95f11abe2c
Try reporting diagnostics in non-strict mode (#8272)
- Makes sure that the compiler will print the diagnostics even if non-strict mode is used
- To prevent printing unexpected stuff to stdout in interactive mode, the diagnostics are printed as 'warning' level log messages in that mode. In strict mode, the messages are printed like before, without changes.
2023-12-14 15:41:35 +00:00
GregoryTravis
1c815a3d45
Better Error Trapping in map (#8307)
* tests

* wip

* wip

* additional warnings

* wip

* wip

* cleanup

* nested wrapping

* multiple nestings

* wraps_error uses looks_for, test for should_fail_with

* wip

* stack trace line fix

* use catch_primitive internally

* fix warning mapping, dtf spec

* just one wrapper checker, vector spec

* missing ctor, back to non-primitive catch

* back to c_p

* put old map back

* wip

* unnest tests

* Array.map on_problems

* wip

* Revert "wip"

This reverts commit c30d171457.

* better test names

* warning logging

* wip

* wip

* move logic into ALH

* doc

* constant

* My_Error.Error

* nested

* doc

* map_primtiive in warning mapper

* composition

* ref spec

* Remove warnings prior to matching on the value

If an expression has warnings and is matched we:
1) extract the warnings
2) execute the branch of a pattern that matches the value
3) attach extracted warnings to the result

This caused warnings to reappear when doing the custom warnings
manipulation.
This is also consistent with how `CaseNode`'s `doWarning` specialization
is defined.

* fix 1

* do not auto unwrap in test error checkers

* nested error matcher

* in problems too

* dtf

* v

* statistics

* wip

* Table_Spec, map_with_index_primitive

* Column_Operations_Spec

* disable warning wrapping and Report_Warning

* unimpl test

* Warnings_Spec

* DCS

* ACG JP

* zip_primitive

* join_helpers

* Lookup_Helpers

* Table

* Data_Formatter

* Value_Type_Helpers

* revert check types changes

* table_helpers

* table tests

* remove st

* do not remove warnings from value

* vec docs, tests for zip, mwi, flat_map

* docs, fixes

* remove nested_error_matcher

* cleanup

* benchmark

* one error

* alter

* add bench to main

* review

* review

* review

* tail call

* changelog

* tail call was not a tail call

* ws

* bad import

* Added missing import

* Update distribution/lib/Standard/Base/0.0.0-dev/src/Data/Array.enso

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>

* review, ref example

* lazy benchmark data

* extra paren

* check outside of catch

* review

* vector too

* actually lazy

* disambiguate Map_Error

* finish rename

* move to extensions

* combine Additional_Warnings error

* rename to map_no_wrap

* do not catch and rethrow

* review

* wip

* remove _primitives entirely

* remove unused should_fail_with function options

* remove expected_warning as function in Problems

---------

Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
2023-12-13 09:38:09 -05:00
Pavel Marek
1bde4c5699
Add org.jline as module to component directory (#8496)
Add `jline` module to the distribution so that our REPL is usable again.

# Important Notes
- No more: "WARNING: Unable to create a system terminal, creating a dumb terminal " warning when starting REPL
- Arrow keys works as expected in REPL
- Back search (the default shortcut `CTRL + R`) works as expected.
2023-12-13 13:47:50 +00:00
Kaz Wesley
ce6c770fd7
Parse inline function signatures (#8470)
Implements #6166.

# Important Notes
- More consistent handling of `default` arguments. `default` is a valid identifier, and only has special meaning when it isn't bound in scope. Since distinguishing the builtin `default` from an identifier called `default` cannot be done until alias analysis has been performed, `default` is now represented in the AST as a regular identifier.
- `TreeToIr`: Remove `insideTypeAscription`. It was only used for bug-for-bug compatibility with the old parser during the transition.
2023-12-12 14:48:44 +00:00
Jaroslav Tulach
80f94a21e1
Compare long and double and BigInteger properly (#8510) 2023-12-12 06:41:43 +01:00
Jaroslav Tulach
e4b2b56a40
Turning Sieve benchmarks into Enso benchmarks (#8475) 2023-12-08 10:27:52 +01:00
Hubert Plociniczak
a14ebd6259
Increase timeout for running launcher command (#8486)
We've been experiencing consistently failures on MacOS due to timeouts.
Doubling the timeout, hoping this will be sufficient to eliminate such
false failures. Will seek alternative solutions if that does not rememdy
the problem on CI.
2023-12-07 16:34:55 +01:00
Hubert Plociniczak
021ff2b40e
Mark some tests as flaky on Windows (#8446) 2023-12-06 11:17:34 +00:00
Hubert Plociniczak
a11bddcb74
Inline execution should support FQNs (#8454)
* Test illustrating problems with FQNs

Inline execution fails with `Compile error: The name `Standard` could
not be found.`.

* Ensure InlineContext carries Package Repos info

Previously, there was no requirement that inline execution should allow
for FQNs. This meant that the omission of Package Repository info went
unnoticed.

In order to be able to refer to `Standard.Visualization.Preprocessor` it
has to be exported as well.
2023-12-06 10:03:06 +01:00
Jaroslav Tulach
7f0cb88fa1
Consistent simple and qualified type name (#8448)
Fixes #8255 by unifying `get_qualified_type_name` and `get_simple_type_name` implementations.
2023-12-06 04:30:24 +00:00