Commit Graph

508 Commits

Author SHA1 Message Date
Jaroslav Tulach
6a6d7dbff3
Simplify onboarding instructions for engine & project-manager developers (#7181)
The current instructions to _build, use and debug_ `project-manager` and its engine/ls process are complicated and require a lot of symlinks to properly point to each other. This pull requests simplifies all of that by introduction of `ENSO_ENGINE_PATH` and `ENSO_JVM_PATH` environment variables. Then it hides all the complexity behind a simple _sbt command_: `runProjectManagerDistribution --debug`.

# Important Notes
I decided to tackle this problem as I have three repositories with different branches of Enso and switching between them requires me to mangle the symlinks. I hope I will not need to do that anymore with the introduction of the `runProjectManagerDistribution` command.
2023-07-17 17:38:54 +00:00
Pavel Marek
7264d81f2a
Builtin methods can support array-like arguments (#7235)
This PR modifies the builtin method processor such that it forbids arrays of non-primitive and non-guest objects in builtin methods. And provides a proper implementation for the builtin methods in `EnsoFile`.

- Remove last `to_array` calls from `File.enso`
2023-07-17 09:17:39 +02:00
Dmitry Bushev
f691713077
Add endpoint for downloading a project (#7291)
close #7194

Changelog:

- add: `/projects/{project_id}/enso_project` HTTP endpoint returning an `.enso-project` archive structure
- update: archive enso project to a `.enso-project` `.tar.gz` archive
- update: make project `path` a required field
2023-07-14 13:18:08 +00:00
Radosław Waśko
78545b4402
Add safepoints to standard libraries Java polyglot helpers (#7183)
Closes #7129
2023-07-05 14:12:13 +00:00
Jaroslav Tulach
91f661fd8b
Documenting how to debug project-manager project (#7153)
```
sbt:project-manager> withDebug run --debugger
```
2023-06-29 05:55:14 +00:00
Radosław Waśko
2bac9cc844
Execution Context integration for Database write operations (#7072)
Closes #6887
2023-06-27 15:51:21 +00:00
Jaroslav Tulach
6abcbcb00c
Moving construction of IR into runtime parser project (#7096) 2023-06-24 06:34:21 +02:00
Jaroslav Tulach
5f572d0680
Avoid dependency on BindingsMap from IR (#7055) 2023-06-19 10:28:31 +02:00
James Dunkerley
760fb71798
First part of AWS S3 API, various small fixes. (#6973)
- Add type detection for `Mixed` columns when calling column functions.
- Excel uses column name for missing headers.
- Add aliases for parse functions on text.
- Adjust `Date`, `Time_Of_Day` and `Date_Time` parse functions to not take `Nothing` anymore and provide dropdowns.
- Removed built-in parses.
- All support Locale.
- Add support for missing day or year for parsing a Date.
- All will trim values automatically.
- Added ability to list AWS profiles.
- Added ability to list S3 buckets.
- Workaround for Table.aggregate so default item added works.
2023-06-15 16:20:13 +00:00
Dmitry Bushev
35ef358121
Update sqlite jdbc dependency (#7031)
# Important Notes
Update `sqlite-jdbc`
2023-06-15 11:00:18 +00:00
Dmitry Bushev
48f0c6f5e8
Scala 2.13.11 and libraries update (#7010)
Update Scala and libraries.
2023-06-14 13:15:57 +00:00
Pavel Marek
372bc8f0d5
Improve compiler's diagnostic messages (#6931)
Improve and colorize compiler's messages. Heavily inspired by `gcc`.
2023-06-08 17:43:18 +02:00
Jaroslav Tulach
e6652b8c9f
Separating instrument related files into runtime-instrument-common project (#6992) 2023-06-08 16:51:50 +02:00
Jaroslav Tulach
65958cba92
IGV Enso support recognizes lib/scala/pkg as project (#6904)
Addition to #4098 - make sure IGV Enso support can recognize `lib/scala/pkg` as a project.
2023-05-31 15:14:57 +00:00
Pavel Marek
bd70ed6830
Update GraalVM to 22.3.1 JDK17 (#6750)
Upgrade GraalVM to 22.3.2 based on JDK17.

---------

Co-authored-by: somebody1234 <ehern.lee@gmail.com>
2023-05-24 10:39:08 +02:00
Radosław Waśko
41a8257e8d
Separating Redshift connector from Database library into a new AWS library (#6550)
Related to #5777
2023-05-04 17:36:51 +00:00
Pavel Marek
3a42d0ce76
Reimplement enso_project as a proper builtin (#6352)
Remove the magical code generation of `enso_project` method from codegen phase and reimplement it as a proper builtin method.

The old behavior of `enso_project` was special, and violated the language semantics (regarding the `self` argument):
- It was implicitly declared in every module, so it could be called without a self argument.
- It can be called with explicit module as self argument, e.g. `Base.enso_project`, or `Visualizations.enso_project`.

Let's avoid implicit methods on modules and let's be explicit. Let's reimplement the `enso_project` as a builtin method. To comply with the language semantics, we will have to change the signature a bit:
- `enso_project` is a static method in the `Standard.Base.Meta.Enso_Project` module.
- It takes an optional `project` argument (instead of taking it as an explicit self argument).

Having the `enso_project` defined as a (shadowed) builtin method, we will automatically have suggestions created for it.

# Important Notes
- Truffle nodes are no longer generated in codegen phase for the `enso_project` method. It is a standard builtin now.
- The minimal import to use `enso_project` is now `from Standard.Base.Meta.Enso_Project import enso_project`.
- Tested implicitly by `org.enso.compiler.ExecCompilerTest#testInvalidEnsoProjectRef`.
2023-05-02 16:40:58 +00:00
Jaroslav Tulach
efe904cd9f
Introducing @BuiltinMethod.needsFrame and InlineableNode (#6442)
Fixes #6416 by introducing `InlineableNode`. It runs fast even on GraalVM CE, fixes ([forever broken](https://github.com/enso-org/enso/pull/6442#discussion_r1178782635)) `Debug.eval` with `<|` and [removes discouraged subclassing](https://github.com/enso-org/enso/pull/6442#discussion_r1178778968) of `DirectCallNode`. Introduces `@BuiltinMethod.needsFrame` - something that was requested by #6293. Just in this PR the attribute is optional - its implicit value continues to be derived from `VirtualFrame` presence/absence in the builtin method argument list. A lot of methods had to be modified to pass the `VirtualFrame` parameter along to propagate it where needed.
2023-04-28 15:32:13 +00:00
Radosław Waśko
462016a428
Create database table from memory (#6429) 2023-04-27 20:06:17 +00:00
Jaroslav Tulach
e5a96b9782
Avoid NullPointerException in --dumpGraphs & test & docs (#6304) 2023-04-18 06:58:06 +02:00
Dmitry Bushev
df4491de50
Batch insert suggestions (#6189)
close #6080

Changelog
- add: implement `SuggestionsRepo.insertAll` as a batch SQL insert
- update: `search/getSuggestionsDatabase` returns empty suggestions. Currently, the method is only used at startup and returns the empty response anyway because the libs are not loaded at that point.
- update: serialize only global (defined in the module scope) suggestions during the distribution building. There's no sense in storing the local library suggestions.
- update: sqlite dependency
- remove: unused methods from `SuggestionsRepo`
- remove: Arguments table

# Important Notes
Speeds up libraries loading by ~1 second.

![2023-04-03-173423_2086x324_scrot](https://user-images.githubusercontent.com/357683/229597470-19dcc010-2a34-43e1-87be-60af99afd275.png)
![2023-04-03-173514_2083x321_scrot](https://user-images.githubusercontent.com/357683/229597476-bf5b3c33-6321-4ac9-a0ca-2fb57d257857.png)
2023-04-06 07:47:23 +00:00
Radosław Waśko
6f86115498
Proper implementation of Value Types in Table (#6073)
This is the first part of the #5158 umbrella task. It closes #5158, follow-up tasks are listed as a comment in the issue.

- Updates all prototype methods dealing with `Value_Type` with a proper implementation.
- Adds a more precise mapping from in-memory storage to `Value_Type`.
- Adds a dialect-dependent mapping between `SQL_Type` and `Value_Type`.
- Removes obsolete methods and constants on `SQL_Type` that were not portable.
- Ensures that in the Database backend, operation results are computed based on what the Database is meaning to return (by asking the Database about expected types of each operation).
- But also ensures that the result types are sane.
- While SQLite does not officially support a BOOLEAN affinity, we add a set of type overrides to our operations to ensure that Boolean operations will return Boolean values and will not be changed to integers as SQLite would suggest.
- Some methods in SQLite fallback to a NUMERIC affinity unnecessarily, so stuff like `max(text, text)` will keep the `text` type instead of falling back to numeric as SQLite would suggest.
- Adds ability to use custom fetch / builder logic for various types, so that we can support vendor specific types (for example, Postgres dates).

# Important Notes
- There are some TODOs left in the code. I'm still aligning follow-up tasks - once done I will try to add references to relevant tasks in them.
2023-03-31 16:16:18 +00:00
Hubert Plociniczak
90612cfa35
Report errors when generating indexes (#6123)
`--compile` command would run the compilation pipeline but silently omit any encountered errors, thus skipping the serialization. This maybe was a good idea in the past but it was problematic now that we generate indexes on build time.
This resulted in rather obscure errors (#6092) for modules that were missing their caches.

The change should significantly improve developers' experience when working on stdlib.

# Important Notes
Making compilation more resilient to sudden cache misses is a separate item to be worked on.
2023-03-30 09:59:49 +00:00
Hubert Plociniczak
49e2c5ac1b
Generate index when invoking buildStdLib tasks (#6000)
The change adds support for generating suggestions and bindings when using the convenient task for building individual stdlib components. By default commands do not generate index since it adds build time. But `buildStdLibAllWithIndex` will.

Closes #5999.
2023-03-28 22:28:38 +00:00
Dmitry Bushev
aa97787603
Update ZIO library (#6072)
close #6069

Changelog:
- update: ZIO major version bump
2023-03-28 07:58:59 +00:00
Kaz Wesley
e171fba301
New documentation parser (#5917)
Implement new Enso documentation parser; remove old Scala Enso parser.

Performance: Total time parsing documentation is now ~2ms.

# Important Notes
- Doc parsing is now done only in the frontend.
- Some engine tests had never been switched to the new parser. We should investigate tests that don't pass after the switch: #5894.
- The option to run the old searcher has been removed, as it is obsolete and was already broken before this (see #5909).
- Some interfaces used only by the old searcher have been removed.
2023-03-15 15:43:51 +00:00
Radosław Waśko
263c3ad651
Add a common-polyglot-core-utils project (#5855)
Adds a common project that allows sharing code between the `runtime` and `std-bits`.

Due to classpath separation and the way it is compiled, the classes will be duplicated - we will have one copy for the `runtime` classpath and another copy as a small JAR for `Standard.Base` library.

This is still much better than having the code duplicated - now at least we have a single source of truth for the shared implementations.

Due to the copying we should not expand this project too much, but I encourage to put here any methods that would otherwise require us to copy the code itself.

This may be a good place to put parts of the hashing logic to then allow sharing the logic between the `runtime` and the `MultiValueKey` in the `Table` library (cc: @Akirathan).
2023-03-11 09:27:26 +00:00
Jaroslav Tulach
632a303089
Use SimpleFormatter.formatMessage to replace {0} with actual parameters (#5870)
Fixes #5801 to properly format Truffle log records before sending them for further processing.
2023-03-11 00:15:58 +00:00
Dmitry Bushev
9397a6ec2f
Pre compute suggestion db during build time (#5698)
Close #5068

Cache suggestions during the `buildEngineDistribution` command, and read them from the disk when the library is loaded. Initial graph coloring takes ~20 seconds vs ~25 seconds on the develop branch.

[peek-develop-branch.webm](https://user-images.githubusercontent.com/357683/223504462-e7d48262-4f5e-4724-b2b0-2cb97fc05140.webm)
[peek-suggestions-branch.webm](https://user-images.githubusercontent.com/357683/223504464-0fe86c04-8c4b-443c-ba96-6c5e2fb1e396.webm)
2023-03-08 12:37:48 +00:00
Radosław Waśko
da760aa27d
Review Text/Table.write problem behavior (#5816)
Closes #5114

Added tests for various problems scenarios when writing files.

And ensured that those tests are passing by fixing a few edge cases.
2023-03-07 02:25:13 +00:00
Hubert Plociniczak
941512e0ba
Optimize import/export resolution (#5700)
This change adds serialization and deserialization of library bindings.
In order to be functional, one needs to first generate IR and
serialize bindings using `--compiled <path-to-library>` command. The bindings
will be stored under the library with `.bindings` suffix.
Bindings are being generated during `buildEngineDistribution` task, thus not
requiring any extra steps.

When resolving import/exports the compiler will first try to load
module's bindings from cache. If successful, it will not schedule its
imports/exports for immediate compilation, as we always did, but use the
bindings info to infer the dependent modules.

The current change does not make any optimizations when it comes to
compiling the modules, yet. It only delays the actual
compilation/loading IR from cache so that it can be done in bulk.
Further optimizations will come from this opportunity such as parallel
loading of caches or lazily inferring only the necessary modules.

Part of https://github.com/enso-org/enso/issues/5568 work.
2023-03-01 08:53:29 +00:00
Jaroslav Tulach
172f72941b
Two findExceptionMessage methods to extract exception messages consistently (#5684)
Creating two `findExceptionMessage` methods in `HostEnsoUtils` and in `VisualizationResult`. Why two? Because one of them is using `org.graalvm.polyglot` SDK as it runs in _"normal Java"_ mode. The other one is using Truffle API as it is running inside of partially evaluated instrument.

There is a `FindExceptionMessageTest` to guarantee consistency between the two methods. It simulates some exceptions in Enso code and checks that both methods extract the same _"message"_ from the exception. The tests verifies hosted and well as Enso exceptions - however testing other polyglot languages is only possible in other modules - as such I created `PolyglotFindExceptionMessageTest` - but that one doesn't have access to Truffle API - e.g. it doesn't really check the consistency - just that a reasonable message is extracted from a JavaScript exception.

# Important Notes
This is not full fix of #5260 - something needs to be done on the IDE side, as the IDE seems to ignore the delivered JSON message - even if it contains properly extracted exception message.
2023-02-20 11:27:16 +00:00
Radosław Waśko
4dcf802831
Ensure that warnings are preserved on Nothing values passing back to Enso through polyglot boundary (#5677)
Fixes #5672

# Important Notes
- Added a subproject `enso-test-java-helpers` which allows the in-Enso tests to add Java helpers for testing.
2023-02-17 13:38:26 +00:00
Jaroslav Tulach
f53696eda4
sbt runEngineDistribution ...args... to build and execute the engine (#5609)
Automating the assembly of the engine and its execution into a single task. If you are modifying standard libraries, engine sources or Enso tests, you can launch `sbt` and then just:
```
sbt:enso> runEngineDistribution --run test/Tests/src/Data/Maybe_Spec.enso
[info] Engine package created at built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev
[info] Executing built-distribution/enso-engine-...-dev/bin/enso --run test/Tests/src/Data/Maybe_Spec.enso
Maybe:  [5/5, 30ms]
- should have a None variant [14ms]
- should have a Some variant [5ms]
- should provide the `maybe` function [4ms]
- should provide `is_some` [2ms]
- should provide `is_none` [3ms]
5 tests succeeded.
0 tests failed.unEngineDistribution 4s
0 tests skipped.
```
the [runEngineDistribution](3a581f29ee/docs/CONTRIBUTING.md (running-enso)) `sbt` input task makes sure all your sources are properly compiled and only then executes your enso source. Everything ready at a single press of Enter.

# Important Notes
To debug in chrome dev tools, just add `--inspect`:
```
sbt:enso> runEngineDistribution --inspect --run test/Tests/src/Data/Maybe_Spec.enso
E.g. in Chrome open: devtools://devtools/bundled/js_app.html?ws=127.0.0.1:9229/7JsgjXlntK8
```
everything gets build and one can just attach the Enso debugger.
2023-02-16 21:37:12 +00:00
James Dunkerley
1bc27501e6
Remove Column type from Aggregate_Column, simplify Column_Selector, some new File_Formats (#5646)
- Updated `Widget.Vector_Editor` ready for use by IDE team.
- Added `get` to `Row` to make API more aligned.
- Added `first_column`, `second_column` and `last_column` to `Table` APIs.
- Adjusted `Column_Selector` and associated methods to have simpler API.
- Removed `Column` from `Aggregate_Column` constructors.
- Added new `Excel_Workbook` type and added to `Excel_Section`.
- Added new `SQLiteFormatSPI` and `SQLite_Format`.
- Added new `IamgeFormatSPI` and `Image_Format`.
2023-02-16 15:15:49 +00:00
Pavel Marek
3e8467c204
Update GraalVM to 22.3.1 (#5602)
Updates the engine to GraalVM 22.3.1 version, which contains fixes for:
- Chrome inspector issues - 38eb3b5932

# Important Notes
- Update to GraalVM 22.3.1
- Remove host object wrapping workaround
2023-02-14 15:51:17 +00:00
Radosław Waśko
a2a1aefc14
Check cargo return value to ensure issues are reported back to SBT (#5588)
Before, any failures of the Rust-side of the parser build would be swallowed by sbt, for example if I add gibberish to the Rust code I will get:
<img width="474" alt="image" src="https://user-images.githubusercontent.com/1436948/217374050-fd9ddaca-136c-459e-932e-c4b9e630d610.png">

This is problematic, because when users are compiling in SBT they may get confusing errors about Java files not being found whereas the true cause is hard to track down because it is somewhere deep in the logs. We've run into this silent failure when setting up SBT builds together with @GregoryTravis today.

I suggest to change it so that once cargo fails, the build is failed with a helpful message - this way it will be easier to track down the issues.

With these changes we get:
<img width="802" alt="image" src="https://user-images.githubusercontent.com/1436948/217374531-707ae348-4c55-4d62-9a86-93850ad8086b.png">
2023-02-08 10:46:40 +00:00
Jaroslav Tulach
f54464881f
enso4igv can open engine/language-server & co. projects (#4098)
In order to investigate `engine/language-server` project, I need to be able to open its sources in IGV and NetBeans.

# Important Notes
By adding same Java source (this time `package-info.java`) and compiling with our Frgaal compiler the necessary `.enso-sources*` files are generated for `engine/language-server` and then the `enso4igv` plugin can open them and properly understand their compile settings.

![Logical View of language-server project](https://user-images.githubusercontent.com/26887752/215472696-ec9801f3-4692-4bdb-be92-c4d2ab552e60.png)

In addition to that this PR enhances the _"logical view"_ presentation of the project by including all source roots found under `src/*/*`.
2023-01-31 08:40:04 +00:00
Jaroslav Tulach
ca2f108513
Enable asserts in the tests (#4074)
Enso unit tests were running without `-ea` check enabled and as such various invariant checks in Truffle code were not executed. Let's turn the `-ea` flag on and fix all the code misbehaves.
2023-01-26 21:41:35 +00:00
Dmitry Bushev
bf9508603f
Add Meta.get_annotation (#4049)
- add: `GeneralAnnotation` IR node for `@name expression` annotations
- update: compilation pipeline to process the annotation expressions
- update: rewrite `OverloadsResolution` compiler pass so that it keeps the order of module definitions
- add: `Meta.get_annotation` builtin function that returns the result of annotation expression
- misc: improvements (private methods, lazy arguments, build.sbt cleanup)
2023-01-24 21:28:33 +03:00
Marcin Kostrzewa
242bd52942
Unboxed atoms (#3862)
Introduces unboxed (and arity-specialized) storage schemes for Atoms. It results in improvements both in memory consumption and runtime.
Memory wise: instead of using an array, we now use object fields. We also enable unboxing. This cuts a good few pointers in an unboxed object. E.g. a quadruple of integers is now 64 bytes (4x8 bytes for long fields + 16 bytes for layout and constructor pointers + 16 bytes for a class header). It used to be 168 bytes  (4x24 bytes for boxed Longs + 16 bytes for array header + 32 bytes for array contents +  8 bytes for constructor ptr  + 16 bytes for class header), so we're saving 104 bytes a piece. In the least impressive scenarios (all-boxed fields) we're saving 8 bytes per object (saving 16 bytes for array header, using 8 bytes for the new layout field). In the most-benchmarked case (list of longs), we save 32 bytes per cons-cell.
Time wise:
All list-summing benchmarks observe a ~2x speedup. List generation benchmarks get ~25x speedups, probably both due to less GC activity and better allocation characteristics (only allocating one object per Cons, rather than Cons + Object[] for fields). The "map-reverse" family gets a neat 10x speedup (part of the work is reading, which is 2x faster, the other is allocating, which is now 25x faster, we end up with 10x when combined).
2023-01-24 13:03:06 +00:00
Jaroslav Tulach
917176873d
visualizationResultToBytes converts anything that looks like a string (#4045)
Use `InteropLibrary.isString` and `asString` to convert any string value to `byte[]`

# Important Notes
Also contains a support for `Metadata.assertInCode` to help locating the right place in the code snippets.
2023-01-13 12:30:27 +00:00
Dmitry Bushev
9df6448d85
Add Polyglot Support in Runtime Tests (#4016)
`runtime-with-instruments` project sets `-Dgraalvm.locatorDisabled=true` that disables the discovery of available polyglot languages (installed with `gu`). On the other hand, enabling locator makes polyglot languages available, but also makes the program classes and the test classes loaded with different classloaders. This way we're unable to use `EnsoContext` in tests to observe internal context state (there is an exception when you try to cast to `EnsoContext`).

The solution is to move tests with enabled polyglot support, but disabled `EnsoContext` introspection to a separate project.
2023-01-03 14:36:26 +00:00
Jaroslav Tulach
308f4edc21
Tweaks to make benchmarking easier (#3961)
Enabling Frgaal for compilation of benchmarks. Using text blocks for Enso code. Making `--showCompilations` work on GraalVM 22.3 again.
2022-12-08 20:30:19 +00:00
Jaroslav Tulach
030dbe4973
Disabling musl as it isn't capable to load dynamic library (#3917)
Disabling `musl` as it isn't capable to load dynamic library.

# Important Notes
With this change it is possible to:
```
$ sbt  bootstrap
$ sbt  engine-runner/buildNativeImage
$ ./runner --run ./engine/runner/src/test/resources/Factorial.enso 3
6
$ ./runner --run ./engine/runner/src/test/resources/Factorial.enso 4
24
$ ./runner --run ./engine/runner/src/test/resources/Factorial.enso 100
93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000
```

Is it OK, @radeusgd  to disable `musl`? If not, we would have to find a way to link the parser in statically, not dynamically.
2022-12-01 06:43:13 +00:00
Jaroslav Tulach
402ebb2f8e
Upgrade to GraalVM 22.3.0 (#3663)
Upgrading to GraalVM 22.3.0.

# Important Notes
- Removed all deprecated `FrameSlot`, and replaced them with frame indexes - integers.
- Add more information to `AliasAnalysis` so that it also gathers these indexes.
- Add quick build mode option to `native-image` as default for non-release builds
- `graaljs` and `native-image` should now be downloaded via `gu` automatically, as dependencies.
- Remove `engine-runner-native` project - native image is now build straight from `engine-runner`.
- We used to have `engine-runner-native` without `sqldf` in classpath as a workaround for an internal native image bug.
- Fixed chrome inspector integration, such that it shows values of local variables both for current stack frame and caller stack frames.
- There are still many issues with the debugging in general, for example, when there is a polyglot value among local variables, a `NullPointerException` is thrown and no values are displayed.
- Removed some deprecated `native-image` options
- Remove some deprecated Truffle API method calls.
2022-11-23 14:30:48 +00:00
Hubert Plociniczak
deb670785c
Allow multiple exports of the same module (#3897)
Previously, when exporting the same module multiple times only the first statement would count and the rest would be discarded by the compiler.

This change allows for multiple exports of the same module e.g.,
```
export project.F1
from project.F1 export foo
```
Multiple exports may however lead to conflicts when combined with hiding names. Added logic in `ImportResolver` to detect such scenarios.

This fixes https://www.pivotaltracker.com/n/projects/2539304/stories/183092447

# Important Notes
Added a bunch of scenarios to simulate pos and neg results.
2022-11-23 11:40:59 +00:00
Hubert Plociniczak
2a38edecd4
Simple HTTPBin to replace Go equivalent (#3887)
1-to-1 translation of the HTTPBin expected by our testsuite using Java's HttpServer.
Can be started from SBT via
```
sbt:enso> simple-httpbin/run <hostname> <port>
```

# Important Notes
@mwu-tow this will mean we can ditch Go dependency completely and replace it with the above call.
2022-11-18 11:27:27 +00:00
Hubert Plociniczak
85d4337f26
Project save backed by git (#3851)
This change adds support for Version Controlled projects in language server.
Version Control supports operations:
- `init` - initialize VCS for a project
- `save` - commit all changes to the project in VCS
- `restore` - ability to restore project to some past `save`
- `status` - show the status of the project from VCS' perspective
- `list` - show a list of requested saves

# Important Notes
Behind the scenes, Enso's VCS uses git (or rather [jGit](https://www.eclipse.org/jgit/)) but nothing stops us from using a different implementation as long as it conforms to the establish API.
2022-11-14 17:32:39 +00:00
Marcin Kostrzewa
23e04f905f
Another attempt at M1 compilation (#3859) 2022-11-09 15:26:25 +00:00
James Dunkerley
45276b243d
Expanding Derived Columns and Expression Syntax (#3782)
- Added expression ANTLR4 grammar and sbt based build.
- Added expression support to `set` and `filter` on the Database and InMemory `Table`.
- Added expression support to `aggregate` on the Database and InMemory `Table`.
- Removed old aggregate functions (`sum`, `max`, `min` and `mean`) from `Column` types.
- Adjusted database `Column` `+` operator to do concatenation (`||`) when text types.
- Added power operator `^` to both `Column` types.
- Adjust `iif` to allow for columns to be passed for `when_true` and `when_false` parameters.
- Added `is_present` to database `Column` type.
- Added `coalesce`, `min` and `max` functions to both `Column` types performing row based operation.
- Added support for `Date`, `Time_Of_Day` and `Date_Time` constants in database.
- Added `read` method to InMemory `Column` returning `self` (or a slice).

# Important Notes
- Moved approximate type computation to `SQL_Type`.
- Fixed issue in `LongNumericOp` where it was always casting to a double.
- Removed `head` from InMemory Table (still has `first` method).
2022-11-08 15:57:59 +00:00
Jaroslav Tulach
85f71cbfc8
Including enso_parser library in the engine distribution (#3842)
Make sure `libenso_parser.so`, `.dll` or `.dylib` are packaged and included when `sbt buildEngineDistribution`.

# Important Notes
There was [a discussion](https://discord.com/channels/401396655599124480/1036562819644141598) about proper location of the library. It was concluded that _"there's no functional difference between a dylib and a jar."_ and as such the library is placed in `component` folder.

Currently the old parser is still used for parsing. This PR just integrates the build system changes and makes us ready for smooth flipping of the parser in the future as part of #3611.
2022-11-02 17:13:53 +00:00
Radosław Waśko
bc09c7b4c2
Remove obsolete Rust parser experiment, superseded by the much more mature new Rust parser integration (#3815)
We've had an old attempt at integrating a Rust parser with our Scala/Java projects. It seems to have been abandoned and is not used anywhere - it is also superseded by the new integration of the Rust parser. I think it was used as an experiment to see how to approach such an integration.

Since it is not used anymore - it make sense to remove it, because it only adds some (slight, but non-zero) maintenance effort. We can always bring it back from git history if necessary.
2022-10-24 14:56:07 +00:00
James Dunkerley
f0f6deef2a
Load the File_Format types via a ServiceLoader (#3813)
Moves the File.read method into the `File` type.
Uses the ServiceLoader to find all types for the File_Format.
2022-10-24 09:55:18 +00:00
Radosław Waśko
82de8f88bd
Add support for Is_In and Not_In to Filter_Condition (#3790)
Implements https://www.pivotaltracker.com/story/show/183389945
2022-10-15 11:29:59 +00:00
Hubert Plociniczak
811d82c787
Ensure all new parser files are checked for changes (#3800) 2022-10-15 07:06:20 +00:00
Pavel Marek
e9260227c4
Duration type is a builtin type (#3759)
- Reimplement the `Duration` type to a built-in type.
- `Duration` is an interop type.
- Allow Enso method dispatch on `Duration` interop coming from different languages.

# Important Notes
- The older `Duration` type should now be split into new `Duration` builtin type and a `Period` type.
- This PR does not implement `Period` type, so all the `Period`-related functionality is currently not working, e.g., `Date - Period`.
- This PR removes `Integer.milliseconds`, `Integer.seconds`, ..., `Integer.years` extension methods.
2022-10-14 18:08:08 +00:00
Radosław Waśko
592a8516a8
Add Is_Empty, Not_Empty, Like and Not_Like to Filter_Condition (#3775)
Implements https://www.pivotaltracker.com/story/show/183389890
2022-10-10 23:11:04 +00:00
Dmitry Bushev
f168cd0758
Enable Junit reports for Scalatest (#3756)
Changelog
- enable graphical test results for Scalatest tests like we have for [stdlib tests](https://github.com/enso-org/enso/actions/runs/3180677675/jobs/5184855484)

# Important Notes
Sometimes the report is created on a different workflow, like [here](https://github.com/enso-org/enso/actions/runs/3196147595/jobs/5217691808). Turned out it is a [known](https://github.com/dorny/test-reporter/issues/67) issue in GitHub Actions https://github.com/orgs/community/discussions/24616
2022-10-07 02:36:07 +00:00
Hubert Plociniczak
93b5f4116b
Always attempt to build rust parser libs in CI (#3769)
This workarounds issues with cached artifacts.
It was easier to use @radeusgd's suggestion than try to figure out what kind of magic CI does.

Fixes https://www.pivotaltracker.com/story/show/183453152
2022-10-06 00:49:41 +00:00
Hubert Plociniczak
096fcfee82
Generate native image for engine-runner (#3638)
This PR adds a possibility to generate native-image for engine-runner.
Note that due to on-demand loading of stdlib, programs that make use of it are currently not yet supported
(that will be resolved at a later point).
The purpose of this PR is only to make sure that we can generate a bare minimum runner because due to lack TruffleBoundaries or misconfiguration in reflection config, this can get broken very easily.
To generate a native image simply execute:
```
sbt> engine-runner-native/buildNativeImage
... (wait a few minutes)
```
The executable is called `runner` and can be tested via a simple test that is in the resources. To illustrate the benefits
see the timings difference between the non-native and native one:
```
>time built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --no-ir-caches --in-project test/Tests/ --run engine/runner-native/src/test/resources/Factorial.enso 6
720

real	0m4.503s
user	0m9.248s
sys	0m1.494s
> time ./runner --run engine/runner-native/src/test/resources/Factorial.enso 6
720

real	0m0.176s
user	0m0.042s
sys	0m0.038s
```

# Important Notes
Notice that due to a [bug in GraalVM](https://github.com/oracle/graal/issues/4200), which is already fixed in 22.x, and us still being on 21.x for the time being, I had to add a workaround to our sbt build to build a different fat jar for native image. To workaround it I had to exclude sqlite jar. Hence native image task is on `engine-runner-native` and not on `engine-runner`.

Will need to add the above command to CI.
2022-09-22 14:45:10 +00:00
Jaroslav Tulach
9134f9b2d7
EnsoCompilerTest to verify compatibility of parsers (#3723)
Adding new _compatibility test_ `EnsoCompilerTest` to verify the new Rust based parser can produce the same `IR` as the original `AST` based one. The simplest way to execute the test from an empty repository is:
```bash
enso$ sbt bootstrap
enso$ sbt "testOnly *EnsoCompilerTest"
```

There are [GitHub Actions run](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266212#step:9:5187) on Linux as well as [run on Windows](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266370#step:9:5254) that show `EnsoCompilerTest` is being executed by the CI (good, as that means `.so` was properly built and linked to the JVM running the test). The [linux](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266212#step:9:5187) as well as [windows](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266370#step:9:5254) runs also demonstrate that failures in the `EnsoCompilerTest` suite fail the CI.

# Important Notes
Right now [there are five test failures](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266212#step:9:5187) - waiting for @kazcw to make sure `codeRepr()` doesn't contain spaces. However, as this PR is more about the infrastructure, I am disabling the currently failing tests in [031169b](031169bd05)
2022-09-20 15:50:27 +00:00
Radosław Waśko
b304402d8e
Add Period Start and End functions to Date and DateTime (#3695)
Implements https://www.pivotaltracker.com/story/show/183081152
2022-09-13 09:51:08 +00:00
Radosław Waśko
551100af3b
Add Table.distinct function to In-Memory table (#3684)
Implements https://www.pivotaltracker.com/story/show/182307143

# Important Notes
- Modified standard library Java helpers dependencies so that `std-table` module depends on `std-base`, as a provided dependency. This is allowed, because `std-table` is used by the `Standard.Table` Enso module which depends on `Standard.Base` which ensures that the `std-base` is loaded onto the classpath, thus whenever `std-table` is loaded by `Standard.Table`, so is `std-base`. Thus we can rely on classes from `std-base` and its dependencies being _provided_ on the classpath. Thanks to that we can use utilities like `Text_Utils` also in `std-table`, avoiding code duplication. Additional advantage of that is that we don't need to specify ICU4J as a separate dependency for `std-table`, since it is 'taken' from `std-base` already - so we avoid including it in our build packages twice.
2022-09-07 12:28:41 +00:00
Hubert Plociniczak
8575b76b0a
Support pattern matching on constants (#3641)
This change adds support for matching on constants by:
1) extending parser to allow literals in patterns
2) generate branch node for literals

Related to https://www.pivotaltracker.com/story/show/182743559
2022-08-12 13:18:58 +00:00
Jaroslav Tulach
a7bc3c6c89
Verify benchmarks compile and execute in the gate (#3640)
Execution of `sbt runtime/bench` doesn't seem to be part of the gate. As such it can happen a change into the Enso language syntax, standard libraries, runtime & co. can break the benchmarks suite without being noticed. Integrating such PR causes unnecessary disruptions to others using the benchmarks.

Let's make sure verification of the benchmarks (e.g. that they compile and can execute without error) is part of the CI.

# Important Notes
Currently the gate shall fail. The fix is being prepared in parallel PR - #3639. When the two PRs are combined, the gate shall succeed again.
2022-08-11 07:21:44 +00:00
Dmitry Bushev
5e114acbb5
Update Scala to 2.13.8 (#3631)
Update Scala compiler and libraries.
2022-08-08 19:32:55 +00:00
Jaroslav Tulach
030e46bfb4
Switching to Java 19 Frgaal compiler (#3594)
Updating to Frgaal 19.0.0-RC1 to give us access to latest Java features including pattern matching on `record` classes. Builds upon #3421.
2022-07-19 09:28:31 +00:00
Hubert Plociniczak
df0ddf7c0b
Print final test report at the end (#3579)
The option asks to print a final test report for each projects at the
end `sbt> run`.
That way, when running the task in aggregate mode, we have a summary at
the end, rather than somewhere in the large output of the individual
subproject.
2022-07-12 12:58:41 +00:00
James Dunkerley
2527a7bdb2
Update SQLite, PostgreSQL and Redshift drivers (#3571)
Updated the SQLite, PostgreSQL and Redshift drivers.

# Important Notes
Updated the API for Redshift and proved able to connect without the ini file workaround.
2022-07-11 18:39:16 +00:00
Radosław Waśko
7c94fa6a77
Custom Encoding support when writing Delimited files (#3564)
Implements https://www.pivotaltracker.com/story/show/182545847
2022-07-07 00:20:00 +00:00
Michał Wawrzyniec Urbańczyk
43a893cae6
Bump the build script (#3535) 2022-07-01 03:58:14 +02:00
Jaroslav Tulach
8f6367966b
Single (+patchable) LiteralNode (#3536)
This PR merges existing variants of `LiteralNode` (`Integer`, `BigInteger`, `Decimal`, `Text`) into a single `LiteralNode`. It adds `PatchableLiteralNode` variant (with non `final` `value` field) and uses `Node.replace` to modify the AST to be patchable. With such change one can remove the `UnwindHelper` workaround as `IdExecutionInstrument` now sees _patched_ return values without any tricks.
2022-06-21 09:42:23 +00:00
Jaroslav Tulach
c72a6582bc
Avoid whole source reparsing when the IDE performs a simple edit (#3508)
Co-authored-by: Dmitry Bushev <bushevdv@gmail.com>
Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
2022-06-16 16:02:57 +02:00
Hubert Plociniczak
b850844834
Fix runtime-with-instruments dependency on runtime (#3529)
This change makes sure that Runtime configuration of `runtime` is listed
as a dependency of `runtime-with-instruments`.
That way `buildEngineDistribution` which indirectly depends on
`runtime-with-instruments`/assembly triggers compilation for std-bits,
if necessary.

# Important Notes
Minor adjustments for a problem introduced in https://github.com/enso-org/enso/pull/3509
2022-06-15 08:31:19 +00:00
Jaroslav Tulach
dc30e44b60
Register instruments/language in their own compilation units to fix the sbt build issues (#3509)
New plan to [fix the `sbt` build](https://www.pivotaltracker.com/n/projects/2539304/stories/182209126) and its annoying:
```
log.error(
"Truffle Instrumentation is not up to date, " +
"which will lead to runtime errors\n" +
"Fixes have been applied to ensure consistent Instrumentation state, " +
"but compilation has to be triggered again.\n" +
"Please re-run the previous command.\n" +
"(If this for some reason fails, " +
s"please do a clean build of the $projectName project)"
)
```
When it is hard to fix `sbt` incremental compilation, let's restructure our project sources so that each `@TruffleInstrument` and `@TruffleLanguage` registration is in individual compilation unit. Each such unit is either going to be compiled or not going to be compiled as a batch - that will eliminate the `sbt` incremental compilation issues without addressing them in `sbt` itself.

fa2cf6a33ec4a5b2e3370e1b22c2b5f712286a75 is the first step - it introduces `IdExecutionService` and moves all the `IdExecutionInstrument` API up to that interface. The rest of the `runtime` project then depends only on `IdExecutionService`. Such refactoring allows us to move the `IdExecutionInstrument` out of `runtime` project into independent compilation unit.
2022-06-13 14:09:08 +00:00
Hubert Plociniczak
fd46e84e8d
Towards a full-blown builtins DSL (part 3) (#3471)
Auto-generate all builtin methods for builtin `File` type from method signatures.
Similarly, for `ManagedResource` and `Warning`.
Additionally, support for specializations for overloaded and non-overloaded methods is added.
Coverage can be tracked by the number of hard-coded builtin classes that are now deleted.

## Important notes

Notice how `type File` now lacks `prim_file` field and we were able to get rid off all of those
propagating method calls without writing a single builtin node class.
Similarly `ManagedResource` and `Warning` are now builtins and `Prim_Warnings` stub is now gone.
2022-06-13 11:48:34 +00:00
Hubert Plociniczak
b1db359f19
Minor compilation improvements (#3512)
Drop `Core` implementation (replacement for IR) as it (sadly) looks increasingly
unlikely this effort will be continued. Also, it heavily relies
on implicits which increases some compilation time (~1sec from `clean`)

Related to https://www.pivotaltracker.com/story/show/182359029
2022-06-07 14:48:50 +00:00
Hubert Plociniczak
31e3f39c55
Suppress pointless warnings coming from SBT (#3499)
This change introduces a custom LogManager for console that allows for
excluding certain log messages. The primarily reason for introducing
such LogManager/Appender is to stop issuing hundreds of pointless
warnings coming from the analyzing compiler (wrapper around javac) for
classes that are being generated by annotation processors.

The output looks like this:
```
[info] Cannot install GraalVM MBean due to Failed to load org.graalvm.nativebridge.jni.JNIExceptionWrapperEntryPoints
[info] compiling 129 Scala sources and 395 Java sources to /home/hubert/work/repos/enso/enso/engine/runtime/target/scala-2.13/classes ...
[warn] Unexpected javac output: warning: File for type 'org.enso.interpreter.runtime.type.ConstantsGen' created in the last round will not be subject to annotation processing.
[warn] 1 warning.
[info] [Use -Dgraal.LogFile=<path> to redirect Graal log output to a file.]
[info] Cannot install GraalVM MBean due to Failed to load org.graalvm.nativebridge.jni.JNIExceptionWrapperEntryPoints
[info] foojavac Filer
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.number.decimal.CeilMethodGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.resource.TakeNodeGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.error.ThrowErrorMethodGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.number.smallInteger.MultiplyMethodGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.warning.GetWarningsNodeGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.number.smallInteger.BitAndMethodGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.error.ErrorToTextNodeGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.warning.GetValueMethodGen
[warn] Could not determine source for class org.enso.interpreter.runtime.callable.atom.AtomGen$MethodDispatchLibraryExports$Cached
....
```

The output now has over 500 of those and there will be more. Much more
(generated by our and Truffle processors).
There is no way to tell SBT that those are OK. One could potentially
think of splitting compilation into 3 stages (Java processors, Java and
Scala) but that will already complicate the non-trivial build definition
and we may still end up with the initial problem.
This is a fix to make it possible to get reasonable feedback from
compilation without scrolling mutliple screens *every single time*.

Also fixed a spurious warning in javac processor complaining about
creating files in the last round.

Related to https://www.pivotaltracker.com/story/show/182138198
2022-06-01 13:50:46 +00:00
Hubert Plociniczak
33a06c9ddf
Restrict annotation processors in interpreter-dsl (#3497)
`interpreter-dsl` should only attempt to run explicitly specified
processors. That way, even if the generated
`META-INF/services/javax.annotation.processing.Processor` is present,
it does not attempt to apply those processors on itself.

This change makes errors related to
```
[warn] Unexpected javac output: error: Bad service configuration file, or
exception thrown while constructing Processor object:
javax.annotation.processing.Processor: Provider org.enso.interpreter.dsl....
```
a thing of the past. This was supper annoying when switching branches and
required to either clean the project or remove the file by hand.

Related to https://www.pivotaltracker.com/story/show/182297597
2022-05-30 19:30:37 +00:00
Hubert Plociniczak
4918ccb5a3
Make sure formatting is applied to std-bits projects (#3477)
@radeusgd discovered that no formatting was being applied to std-bits projects.
This was caused by the fact that `enso` project didn't aggregate them. Compilation and
packaging still worked because one relied on the output of some tasks but
```
sbt> javafmtAll
```
didn't apply it to `std-bits`.

# Important Notes
Apart from `build.sbt` no manual changes were made.
2022-05-25 09:26:50 +00:00
Dmitry Bushev
f9d2964e83
Update profiling CLI arguments (#3461) 2022-05-24 16:01:26 +03:00
Hubert Plociniczak
688df9825c
Part 2 of system for builtin objects (#3454)
This is the 2nd part of DSL improvements that allow us to generate a lot of
builtins-related boilerplate code.
- [x] generate multiple method nodes for methods/constructors with varargs
- [x] expanded processing to allow for @Builtin to be added to classes and
and generate @BuiltinType classes
- [x] generate code that wraps exceptions to panic via `wrapException`
annotation element (see @Builtin.WrapException`

Also rewrote @Builtin annotations to be more structured and introduced some nesting, such as
@Builtin.Method or @Builtin.WrapException.

This is part of incremental work and a follow up on https://github.com/enso-org/enso/pull/3444.

# Important Notes
Notice the number of boilerplate classes removed to see the impact.
For now only applied to `Array` but should be applicable to other types.
2022-05-19 10:43:47 +00:00
Radosław Waśko
a32644dd85
Make it possible to run Enso with assertions enabled (#3450)
Before, when running Enso with `-ea`, some assertions were broken and the interpreter would not start.

This PR fixes two very minor bugs that were the cause of this - now we can successfully run Enso with `-ea`, to test that any assertions in Truffle or in our own libraries are indeed satisfied.

Additionally, this PR adds a setting to SBT that ensures that IntelliJ uses the right language level (Java 17) for our projects.
2022-05-13 15:38:52 +00:00
Hubert Plociniczak
a2dae60aa9
Generate BuiltinMethods from simple method and constructor signatures (#3444)
A low-hanging fruit where we can automate the generation of many
@BuiltinMethod nodes simply from the runtime's methods signatures.
This change introduces another annotation, @Builtin, to distinguish from
@BuiltinType and @BuiltinMethod processing. @Builtin processing will
always be the first stage of processing and its output will be fed to
the latter.

Note that the return type of Array.length() is changed from `int` to
`long` because we probably don't want to add a ton of specializations
for the former (see comparator nodes for details) and it is fine to cast
it in a small number of places.

Progress is visible in the number of deleted hardcoded classes.

This is an incremental step towards #181499077.

# Important Notes
This process does not attempt to cover all cases. Not yet, at least.
We only handle simple methods and constructors (see removed `Array` boilerplate methods).
2022-05-12 08:42:00 +00:00
Hubert Plociniczak
d8a2832624
Integrate Frgaal in Java libs (#3433)
This PR allows for easier integration of Frgaal in different subprojects.
Related to https://www.pivotaltracker.com/story/show/182093808
2022-05-11 11:21:01 +00:00
Hubert Plociniczak
72c4006cb1
Additional sbt commands to build stdlib packages (#3431)
Auxiliary sbt commands for building individual
stdlib packages.
The commands check if the engine distribution was built at least once,
and only copy the necessary package files if necessary.
So far added:
- `buildStdLibBase`
- `buildStdLibDatabase`
- `buildStdLibTable`
- `buildStdLibImage`
- `buildStdLibGoogle_Api`

Related to [#182014385](https://www.pivotaltracker.com/story/show/182014385)
2022-05-11 10:12:18 +00:00
Jaroslav Tulach
ab1ca54acd
Profile the language server (#3389)
In order to analyse why the `runner.jar` is slow to start, let's _"self sample"_ it using the [sampler library](https://bits.netbeans.org/dev/javadoc/org-netbeans-modules-sampler/org/netbeans/modules/sampler/Sampler.html). As soon as the `Main.main` is launched, the sampling starts and once the server is up, it writes its data into `/tmp/language-server.npss`.

Open the `/tmp/language-server.npss` with [VisualVM](https://visualvm.github.io) - you should have one copy in your
GraalVM `bin/jvisualvm` directory and there has to be a GraalVM to run Enso.

#### Changelog

- add: the `MethodsSampler` that gathers information in `.npss` format
- add: `--profiling` flag that enables the sampler
- add: language server processes the updates in batches
2022-05-10 12:44:05 +00:00
Jaroslav Tulach
21c46901b7
Debug Enso language in ChromeDev tools with --inspect option (#3432)
Finally this pull request proposes `--inspect` option to allow [debugging of `.enso`](e948f2535f/docs/debugger/README.md) in Chrome Developer Tools:

```bash
enso$ ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --inspect --run ./test/Tests/src/Data/Numbers_Spec.enso
Debugger listening on ws://127.0.0.1:9229/Wugyrg9Nm4OUL9YhzdcElmLft71ayZW3LMUPCdPyNAY
For help, see: https://www.graalvm.org/tools/chrome-debugger
E.g. in Chrome open: devtools://devtools/bundled/js_app.html?ws=127.0.0.1:9229/Wugyrg9Nm4OUL9YhzdcElmLft71ayZW3LMUPCdPyNAY
```
copy the printed URL into chrome browser and you should see:

![obrazek](https://user-images.githubusercontent.com/26887752/167235327-8ad15fb2-96d4-4a0c-9e31-ed67ab46578b.png)

One can also debug the `.enso` files in NetBeans or [VS Code with Apache Language Server extension](https://cwiki.apache.org/confluence/display/NETBEANS/Apache+NetBeans+Extension+for+Visual+Studio+Code) just pass in special JVM arguments:
```bash
enso$ JAVA_OPTS=-agentlib:jdwp=transport=dt_socket,server=y,address=8000 ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run ./test/Tests/src/Data/Numbers_Spec.enso
Listening for transport dt_socket at address: 8000
```
and then _Debug/Attach Debugger_. Once connected choose the _Toggle Pause in GraalVM Script_ button in the toolbar (the "G" button):

![obrazek](https://user-images.githubusercontent.com/26887752/167235598-98266c7e-beb5-406b-adc6-8167b3d1b453.png)

and your execution shall stop on the next `.enso` line of code. This mode allows to debug both - the Enso code as well as Java code.

Originally started as an attempt to write test in Java:

* test written in Java
* support for JUnit in `build.sbt`
* compile Java with `-g` - so it can be debugged
* Implementation of `StatementNode` - only gets created when `materialize` request gets to `BlockNode`
2022-05-10 08:55:08 +00:00
Hubert Plociniczak
4bbabc00be
Move Builtin Types and Methods to stdlib (#3363)
This PR replaces hard-coded `@Builtin_Method` and `@Builtin_Type` nodes in Builtins with an automated solution
that a) collects metadata from such annotations b) generates `BuiltinTypes` c) registers builtin methods with corresponding
constructors.
The main differences are:
1) The owner of the builtin method does not necessarily have to be a builtin type
2) You can now mix regular methods and builtin ones in stdlib 
3) No need to keep track of builtin methods and types in various places and register them by hand (a source of many typos or omissions as it found during the process of this PR)

Related to #181497846
Benchmarks also execute within the margin of error.

### Important Notes

The PR got a bit large over time as I was moving various builtin types and finding various corner cases.
Most of the changes however are rather simple c&p from Builtins.enso to the corresponding stdlib module.
Here is the list of the most crucial updates:
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/builtin/Builtins.java` - the core of the changes. We no longer register individual builtin constructors and their methods by hand. Instead, the information about those is read from 2 metadata files generated by annotation processors. When the builtin method is encountered in stdlib, we do not ignore the method. Instead we lookup it up in the list of registered functions (see `getBuiltinFunction` and `IrToTruffle`)
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/callable/atom/AtomConstructor.java` has now information whether it corresponds to the builtin type or not.
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/RuntimeStubsGenerator.scala` - when runtime stubs generator encounters a builtin type, based on the @Builtin_Type annotation, it looks up an existing constructor for it and registers it in the provided scope, rather than creating a new one. The scope of the constructor is also changed to the one coming from stdlib, while ensuring that synthetic methods (for fields) also get assigned correctly
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/IrToTruffle.scala` - when a builtin method is encountered in stdlib we don't generate a new function node for it, instead we look it up in the list of registered builtin methods. Note that Integer and Number present a bit of a challenge because they list a whole bunch of methods that don't have a corresponding method (instead delegating to small/big integer implementations).
During the translation new atom constructors get initialized but we don't want to do it for builtins which have gone through the process earlier, hence the exception
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor not only  generates the actual code fpr nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization 
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor no longer generates only (root) nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/TypeProcessor.java` - Similar to MethodProcessor but handles @Builtin_Type annotations. It doesn't, **yet**, generate any builtin objects.  It also collects the names, as present in stdlib, if any, so that we can generate the names automatically (see generated `types/ConstantsGen.java`)
- `engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin` - various classes annotated with @BuiltinType to ensure that the atom constructor is always properly registered for the builitn. Note that in order to support types fields in those, annotation takes optional `params` parameter (comma separated). 
- `engine/runtime/src/bench/scala/org/enso/interpreter/bench/fixtures/semantic/AtomFixtures.scala` - drop manual creation of test list which seemed to be a relict of the old design
2022-05-05 20:18:06 +02:00
Hubert Plociniczak
79c82da21c
Frgaal integration in sbt (#3421)
* Initial integration with Frgaal in sbt

Half-working since it chokes on generated classes from annotation
processor.

* Replace AutoService with ServiceProvider

For reasons unknown AutoService would fail to initialize and fail to
generate required builtin method classes.
Hidden error message is not particularly revealing on the reason for
that:
```
[error] error: Bad service configuration file, or exception thrown while constructing Processor object: javax.annotation.processing.Processor: Provider com.google.auto.service.processor.AutoServiceProcessor could not be instantiated
```

The sample records is only to demonstrate that we can now use newer Java
features.

* Cleanup + fix benchmark compilation

Bench requires jmh classes which are not available because we obviously
had to limit `java.base` modules to get Frgaal to work nicely.
For now, we default to good ol' javac for Benchmarks.
Limiting Frgaal to runtime for now, if it plays nicely, we can expand it
to other projects.

* Update CHANGELOG

* Remove dummy record class

* Update licenses

* New line

* PR review

* Update legal review

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
2022-05-04 21:18:40 +02:00
James Dunkerley
a4dbc9a37b
Moving Aggregation to Java (#3364) 2022-04-04 09:12:48 +00:00
Dmitry Bushev
23e5216922
Fix docparser (#3370)
Changelog:
- fix the docparser case when there is an unclosed formatted section before a list
- add the test checking the doc generation on the stdlib
2022-03-31 04:44:34 +00:00
Dmitry Bushev
a34c2bcfe6
Implement Documentation Sections API (#3360) 2022-03-25 10:49:37 +03:00
Dmitry Bushev
9d402bd599
Split documentation comment into sections (#3347) 2022-03-21 10:14:25 +03:00
Dmitry Bushev
792c7e3538
Fix dev version check (#3265) 2022-02-15 16:34:33 +01:00
Edward Kmett
0c25ee736c
Upgrade Truffle and Graal to Version 21.3.0 (#3258) 2022-02-11 19:05:13 +03:00
Michał Wawrzyniec Urbańczyk
4baad5f146
Nightly proccess preparations: Setting Enso version through the environment (#3241)
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
Co-authored-by: Radosław Waśko <wasko.radek@gmail.com>
2022-02-07 15:14:32 +01:00
Radosław Waśko
4fc97d5563
Fix a few typos in code (#3223)
* Fix a typo in build.sbt
* Change incorrect Panic.catch reference to Panic.recover
2022-01-13 16:17:19 +01:00
Dmitry Bushev
ca0a6f0bf6
Update Scala to 2.13.7 (#3214) 2021-12-31 17:50:32 +03:00
Michał Wawrzyniec Urbańczyk
66c256a1f7
Self-hosted Engine CI for Windows and Linux (#3182) 2021-12-27 17:56:35 +01:00
Mateusz Czapliński
895291876c
Fix problems related to error visualisation evaluation (#3193)
- Add parser & handler in IDE for `executionContext/visualisationEvaluationFailed` message from Engine (fixes a developer console error "Failed to decode a notification: unknown variant `executionContext/visualisationEvaluationFailed`"). The contents of the error message will now be properly deserialized and printed to Dev Console with appropriate details.
- Fix a bug in an Enso code snippet used internally by the IDE for error visualizations preprocessing. The snippet was using not currently supported double-quote escaping in double-quote delimited strings. This lack of processing is actually a bug in the Engine, and it was reported to the Engine team, but changing the strings to single-quoted makes the snippet also more readable, so it sounds like a win anyway.
    - A test is also added to the Engine CI, verifying that the snippet compiles & works correctly, to protect against similar regressions in the future.

Related: #2815
2021-12-21 10:00:57 +01:00
Adam Obuchowicz
c4d22102cf
Switch to 2021 edition (#3173) 2021-12-01 16:06:57 +01:00
Radosław Waśko
46c31bb9a5
Preinstalling With Dependencies (#1981) 2021-11-23 11:51:17 +03:00
Adam Obuchowicz
942464cbaf
Refactoring: merge utils into prelude; merge workspaces. (#3151) 2021-11-10 14:36:08 +01:00
Dmitry Bushev
8dfbe93df8
Bump the snapshot version 2021-10-01 15:43:02 +03:00
Dmitry Bushev
ae54ae0566
Prepare for the 0.2.31 release (#2004) 2021-10-01 15:39:39 +03:00
Ara Adkins
8d676259b9
Bump the snapshot version 2021-09-23 14:17:45 +01:00
Ara Adkins
1a2c3ddf20
Prepare for the 0.2.30 release (#1999) 2021-09-23 14:16:33 +01:00
Ara Adkins
1cd2706ba8
Load IR Caches from Disk (#1996) 2021-09-18 13:48:13 +01:00
Ara Adkins
2d8dd2876b
Bump the snapshot version 2021-09-16 09:37:15 +01:00
Ara Adkins
c22c41b8df
Prepare for the 0.2.29 release (#1995) 2021-09-16 09:36:08 +01:00
Marcin Kostrzewa
a81257b402
Google Spreadsheet Reading (#1976) 2021-09-03 21:41:12 +02:00
Ara Adkins
c9e6fa2065
Bump the snapshot version 2021-09-02 15:02:33 +01:00
Ara Adkins
3b84547264
Prepare for the 0.2.28 release (#1988) 2021-09-02 15:01:18 +01:00
Marcin Kostrzewa
b73e5e84b3
Redshift Connector (#1985) 2021-09-02 11:28:49 +02:00
Radosław Waśko
826e28a149
Implement library/preinstall (Without Dependencies Yet) (#1972) 2021-08-27 14:01:13 +02:00
Ara Adkins
649fe33ccf
Add support for IR cache locations (#1973) 2021-08-26 11:52:35 +01:00
Ara Adkins
1d16698527
Bump the snapshot version 2021-08-23 13:04:01 +01:00
Ara Adkins
f068579248
Prepare for the 0.2.27 release (#1970) 2021-08-23 12:59:58 +01:00
Ara Adkins
f53744ff53
Bump the snapshot version 2021-08-19 14:44:56 +01:00
Ara Adkins
5820547c81
Prepare for the 0.2.26 release (#1965) 2021-08-19 14:43:26 +01:00
Ara Adkins
2fcc0d7ea5
Bump the snapshot version 2021-08-18 09:57:02 +01:00
Ara Adkins
0cff78b481
Prepare for the 0.2.25 release (#1959) 2021-08-18 09:54:52 +01:00
Radosław Waśko
be6e60509a
Tool To Bump Stdlib Version (#1947) 2021-08-13 18:14:20 +02:00
Ara Adkins
a9c8aa45d4
Bump the snapshot version 2021-08-13 12:04:40 +01:00
Ara Adkins
92cd9fc6c6
Prepare for the 0.2.24 release (#1950) 2021-08-13 12:03:39 +01:00
Radosław Waśko
fac0405fd0
Updating Editions (#1944) 2021-08-12 16:55:23 +02:00
Radosław Waśko
31167fd6ae
Revert "Revert "Library Publishing MVP (#1898)"" (#1933) 2021-08-09 16:00:04 +02:00
Ara Adkins
3b7d6e117c
Bump the snapshot version 2021-08-09 11:48:22 +01:00
Ara Adkins
3fb8e43e98
Prepare for the 0.2.23 release (#1934) 2021-08-09 11:47:14 +01:00
Ara Adkins
a235fb800f
Bump the snapshot version 2021-08-06 14:29:47 +01:00
Ara Adkins
37a8fab046
Prepare for the 0.2.22 release (#1931) 2021-08-06 14:29:16 +01:00
Ara Adkins
3a1fa1f973
Bump the snapshot version 2021-08-05 14:37:17 +01:00
Ara Adkins
c86c09aeff
Prepare for the 0.2.21 release (#1927) 2021-08-05 14:36:37 +01:00
Ara Adkins
2553d9738c
Bump the snapshot version 2021-08-05 09:54:27 +01:00
Ara Adkins
1f0a50066f
Prepare for the 0.2.20 release (#1924) 2021-08-05 09:53:58 +01:00
Ara Adkins
bbffd30c63
Bump the snapshot version 2021-08-03 13:56:59 +01:00
Ara Adkins
cd40671e7a
Prepare for the 0.2.19 release (#1920) 2021-08-03 13:56:30 +01:00
Ara Adkins
ce74f0cf2e
Bump the snapshot version 2021-08-02 17:07:48 +01:00
Ara Adkins
39f794778f
Prepare for the 0.2.18 release (#1918) 2021-08-02 17:07:18 +01:00
Ara Adkins
3302cc2338
Bump the snapshot version 2021-07-28 10:37:25 +01:00
Ara Adkins
d5d61af9a1
Prepare for the 0.2.17 release (#1905) 2021-07-28 10:36:24 +01:00
Ara Adkins
a7478bc573
Revert "Library Publishing MVP (#1898)"
This reverts commit 1bd4e5824e.
2021-07-26 10:32:19 +01:00
Radosław Waśko
1bd4e5824e
Library Publishing MVP (#1898) 2021-07-23 18:16:34 +02:00
Ara Adkins
0af9d4ff1e
Bump the snapshot version 2021-07-23 10:10:08 +01:00
Ara Adkins
2cb6ac5776
Prepare for the 0.2.16 release (#1897) 2021-07-23 10:09:38 +01:00
Radosław Waśko
f3bc88b91e
Library Downloads MVP (#1885) 2021-07-22 08:24:06 +02:00
Dmitry Bushev
980ba8cb65
Add Reexport Field to Suggestions (#1793)
Add the reexport field to suggestions
2021-07-20 19:10:53 +03:00
Marcin Kostrzewa
f55d66cb2c
XLS(X) Reading (#1879) 2021-07-20 13:32:19 +02:00
Radosław Waśko
c0eae9db05
Bump the snapshot version 2021-07-19 19:29:39 +02:00
Radosław Waśko
1aa5cd0831
Prepare for the 0.2.15 release (#1883) 2021-07-19 19:21:23 +02:00
Radosław Waśko
86fcd86055
New Language Server API Implementations / Mocks (#1875) 2021-07-17 16:49:51 +02:00
Ara Adkins
66ba1a598c
Bump the snapshot version 2021-07-15 15:36:13 +01:00
Ara Adkins
721a9ab899
Prepare for the 0.2.14 release (#1874) 2021-07-15 15:35:34 +01:00
Radosław Waśko
2eadf9f19f
Bump Minimal PM/Launcher Versions for New Engine (#1854) 2021-07-13 13:08:00 +02:00
Radosław Waśko
042a0f81f1
Fix Nightly Workflows and Update Release Policy (#1845) 2021-07-10 01:10:31 +02:00
Ara Adkins
8a79e1e033
Bump the snapshot version 2021-07-09 09:18:59 +01:00
Ara Adkins
82623b07a7
Prepare for the 0.2.13 release (#1844) 2021-07-09 09:18:30 +01:00
Radosław Waśko
b58e895b18
Editions Integration (#1832) 2021-07-08 15:38:20 +02:00
Maciej Mikołajek
520cd70d55
Documentation files on disk Part 1. (#1744) 2021-07-07 15:00:33 +02:00
Radosław Waśko
e58b5eb81d
Dynamically Loading Libraries (#1826) 2021-07-06 00:27:14 +02:00
Marcin Kostrzewa
334a022ffd
Import syntax including namespace (#1806) 2021-06-24 12:42:24 +02:00
Ara Adkins
bf97a89ae0
Bump the snapshot version 2021-06-24 08:36:16 +01:00
Ara Adkins
3bcaad42ac
Prepare for the 0.2.11 release (#1818) 2021-06-24 08:35:43 +01:00
Maciej Mikołajek
4173dd810f
Documentation in the Suggestions Database (#1791) 2021-06-22 14:44:48 +02:00
Radosław Waśko
1d124d7770
Libraries Repository (#1804) 2021-06-22 13:35:15 +02:00
Radosław Waśko
241a1e7d74
Define Editions (#1797) 2021-06-18 16:39:45 +02:00
Ara Adkins
a981e72fdf
Provide API access to the project name (#1759) 2021-05-27 16:13:52 +01:00
Ara Adkins
1bdf87ce76
Update the memory limit for Native Image (#1746) 2021-05-20 14:03:14 +01:00
Dmitry Bushev
6c084fc8ae
Add support for detailed logger configuration (#1745) 2021-05-19 09:02:32 +01:00
Dmitry Bushev
c202c92b68
Log akka messages (#1719) 2021-05-18 09:23:13 +01:00
Maciej Mikołajek
f74d386acb
Rewrite Docs Generator from Python to Scala (#1729) 2021-05-17 18:44:21 +02:00
Ara Adkins
48bcebc723
Update to GraalVM 21.1.0 (#1738) 2021-05-14 13:08:39 +01:00
Dmitry Bushev
1b6388702f
Implement masking of sensitive data in logs (#1732) 2021-05-12 16:31:53 +01:00
Ara Adkins
5e00a78d63
Bump the snapshot version 2021-04-28 15:45:58 +01:00
Ara Adkins
b66542cf86
Prepare for the 0.2.11 release (#1708) 2021-04-28 15:45:39 +01:00
Radosław Waśko
fa4775fe58
Remove Unused JAR Launcher Prefix to Fix Issue with Signing (#1672)
* Remove prefix

* Update docs
2021-04-14 13:24:50 +02:00
Ara Adkins
6d3ebedf27
Bump the snapshot version 2021-04-07 17:13:21 +01:00
Ara Adkins
079da0575c
Prepare for the 0.2.10 release (#1656) 2021-04-07 17:12:50 +01:00
Dmitry Bushev
65e9cca5a4
Add Python and R to the GraalVM Bundle (#1644)
CI release pipeline is updated to install
python and R GraalVM components
2021-04-07 17:19:23 +03:00
Ara Adkins
a877e394e7
Bump the snapshot version 2021-03-26 12:28:22 +00:00
Ara Adkins
0212d2184a
Prepare for the 0.2.9 release (#1619) 2021-03-26 12:27:57 +00:00
Dmitry Bushev
534ed305fc
Image Processing Library Prototype (#1450)
Add the Standard.Image library.
2021-03-23 13:16:43 +03:00
Ara Adkins
efe0e859e7
Bump the snapshot version 2021-03-19 10:55:31 +00:00
Ara Adkins
687af4d1c6
Prepare for the 0.2.8 release (#1596) 2021-03-19 10:55:01 +00:00
Radosław Waśko
21f667323e
PostgreSQL Support in Database Library (#1565)
Co-authored-by: Marcin Kostrzewa <marckostrzewa@gmail.com>
2021-03-16 17:53:04 +01:00
Ara Adkins
db03389c20
Bump the snapshot version 2021-03-16 13:14:06 +00:00
Ara Adkins
a2beded54f
Prepare for the 0.2.7 release (#1578) 2021-03-16 13:13:39 +00:00
Ara Adkins
96697ddc97
Fix a crash due to shadowed project names (#1571) 2021-03-16 12:45:19 +00:00
Radosław Waśko
5f8af886e5
Connection and Materialization in the Database Library (#1546) 2021-03-09 19:52:42 +01:00
Ara Adkins
de5cf689c3
Bump the snapshot version 2021-03-02 09:25:06 +00:00
Ara Adkins
d0bd58b584
Prepare for the 0.2.6 release (#1533) 2021-03-02 09:23:32 +00:00
Ara Adkins
2b4f0ef2ad
Bump scalac to 2.13.5 (#1531) 2021-03-01 16:35:57 +00:00
Ara Adkins
03349c1e6c
Bump the snapshot version 2021-02-26 10:43:18 +00:00
Ara Adkins
b79c35205f
Prepare for the 0.2.5 release (#1524) 2021-02-26 10:42:26 +00:00
Radosław Waśko
6544c2478d
Implement the first part of the database library (#1475) 2021-02-25 13:48:18 +00:00
Ara Adkins
25c5519b45
Bump the snapshot version 2021-02-23 09:05:01 +00:00
Ara Adkins
c667805b34
Prepare for the 0.2.4 release (#1499) 2021-02-23 09:04:01 +00:00
Ara Adkins
8ec2a92b77
Upgrade GraalVM to 21.0 (#1487) 2021-02-22 13:32:55 +00:00