Commit Graph

192 Commits

Author SHA1 Message Date
Hubert Plociniczak
49e78adb88
Build distribution for amd64 and aarch64 MacOS (#8407)
* Build distribution for amd64 and aarch64 MacOS

Possible after the GraalVM upgrade.

* Another attempt at building on MacOS M1

* One less hardcoded architecture

* Eliminate one more hardcoded architecture

* add more debug info

* nit
2023-12-05 11:24:02 +01:00
Pavel Marek
a67297aebf
Add graalpy packages to the component directory (#8351)
Adds these JAR modules to the `component` directory inside Engine distribution:
- `graal-language-23.1.0`
- `org.bouncycastle.*` - these need to be added for graalpy language

# Important Notes
- Remove `org.bouncycastle.*` packages from `runtime.jar` fat jar.
- Make sure that the `./run` script preinstalls GraalPy standalone distribution before starting engine tests
- Note that using `python -m venv` is only possible from standalone distribution, we cannot distribute `graalpython-launcher`.
- Make sure that installation of `numpy` and its polyglot execution example works.
- Convert `Text` to `TruffleString` before passing to GraalPy - 8ee9a2816f
2023-12-04 11:50:59 +00:00
Pavel Marek
268e595ec1
Add Chrome devtools and DAP tools for debugging (#8344)
Adds chrome-inspector tool and Debug Adapter protocol tool for debugging Enso.

# Important Notes
The chrome devtools seems to be broken (tracked in https://github.com/oracle/graal/issues/7636). Even `graalpy --inspect ...` fails (Chrome devtools freezes after connecting to the server). This PR adds Debug adapter protocol tool for debugging as a workaround for chrome inspector. There is docs in [dap.md](https://github.com/enso-org/enso/pull/8344/files#diff-421574b50574cfe546e86d4b3d32d79b8b2087f2fe204f68e5cf2693af43bbe1)
2023-11-22 17:18:41 +00:00
James Dunkerley
ecaca12df1
Integrating Enso Cloud with the libraries (part 1...) (#8006)
- Add a `File_For_Read` type. Used for `File_Format` to read files.
- Added `Enso_User` representing the current user in `Enso_Cloud`.
- *Will be later able to list known users.*
- Added `Enso_Secret` representing a value defined in `Enso_Cloud`.
- Value not used within Enso only accessed within polyglot Java.
- Integrated into `Username_And_Password` and can be used within JDBC connections.
- Integrated into HTTP Headers so a secret can be used as a value.
- New `URI_With_Query` with the same API as `URI`. Supporting secrets in the value.
- *Will be integrated with AWS credentials.*
- Added `Enso_File` representing a file or a folder in the cloud.
- Support the same API as `File` (like the `S3_File`).
- *Will support `enso://` URI style access.*
2023-11-20 23:21:14 +00:00
Pavel Marek
5a7ad6bfe4
Upgrade enso to GraalVM for jdk 21 (#7991)
Upgrade to GraalVM JDK 21.
```
> java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing)
```

With SDKMan, download with `sdk install java 21-graalce`.

# Important Notes
- After this PR, one can theoretically run enso with any JRE with version at least 21.
- Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution.
- `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded.
- sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368
- Which means that we can have `module-info.java` files only for Java-only projects.
- Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar.
- `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent.
- Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory.
- The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module.
- The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725)
- [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 18:02:36 +00:00
Jaroslav Tulach
f86e1b3aa5
Using compiler that supports JDK 21 features (#8286) 2023-11-15 17:19:47 +01:00
Radosław Waśko
1114a9bcff
Fix incremental compilation of SPI in Java helper libraries (#8129)
- Fixes the issue that sometimes occurred on CI where old `services` configuration was not cleaned and SPI definitions were leaking between PRs, causing random failures:
```
 should allow selecting table rows based on a boolean column
An unexpected panic was thrown: java.util.ServiceConfigurationError: org.enso.base.file_format.FileFormatSPI: Provider org.enso.database.EnsoConnectionSPI not found
```
- The issue is fixed by detecting unknown SPI classes before the build, and if such classes are detected, cleaning the config and forcing a rebuild of the given library to ensure consistency of the service config.
2023-10-23 09:14:35 +00:00
Dmitry Bushev
1a7e83b80c
Fix warnings in SBT build (#8026)
Fixes warnings on SBT startup

```
$ sbt
[info] welcome to sbt 1.9.0 (GraalVM Community Java 17.0.7)
...
[warn] 5 feature warnings; re-run with -feature for details
[warn] one warning found
[info] loading settings for project enso from build.sbt ...
```
2023-10-12 08:19:45 +00:00
Dmitry Bushev
f348083dfb
Fix bundled GraalVM path (#7948)
close #7871
close #7698

Changelog:
- fix: the `run` script logic to place the GraalVM runtime in the expected directory when building the bundle
- fix: the `makeBundles` SBT logic to place the GraalVM runtime in the expected directory
2023-10-06 18:49:57 +00:00
Hubert Plociniczak
30100605da
Native image should not be tied a particular hardware (#7956)
Added `-march=compatibility` argument to native image builds to ensure that we don't generate binaries that cannot be run on a target machine.
2023-10-03 20:34:00 +02:00
Jaroslav Tulach
15b1989418
Adjusting to GraalVM for JDK21+ (#7855)
Changes required to build on GraalVM for JDK21+ while keeping the support for building on GraalVM for JDK17.
2023-09-22 11:38:53 +00:00
Jaroslav Tulach
ad34a701e4
Upgrading to Frgaal compiler 20.0.1 (#7860) 2023-09-22 09:58:19 +02:00
IsaacTell
12dc39591d
Make reading the license files case insensitive. (#7725)
Remove license files that were duplicated but cased differently.

Co-authored-by: Isaac Tell <isaacftell@gmail.comm>
2023-09-20 23:17:06 +02:00
Jaroslav Tulach
6cbd111bad
Optional Espresso support with ENSO_JAVA=espresso env variable 2023-09-19 15:10:12 +02:00
Jaroslav Tulach
30a62b97bb
Runtime checking of ascribed expression types (#7796) 2023-09-14 14:09:41 +02:00
Hubert Plociniczak
ceb6084be9
Fix sbt task that creates GraalVM package (#7721)
While looking into #7698 discovered that the `buildGraalDistribution`
task would fail to use the old (pre-23.0) GraalVM naming of artifacts.
This has changed since #7176.

The fix does not attempt to fix a problem of packaged GraalVM name in
`runtime`. That, as the ticket mentiones it, is the role of rust build
script that creates the correct bundle.
2023-09-06 13:58:10 +02:00
Hubert Plociniczak
8a60bc6dcd
Replace a custom logger with off the shelf implementation (#7559)
This change replaces Enso's custom logger with an existing, mostly off the shelf logging implementation. The change attempts to provide a 1:1 replacement for the existing solution while requiring only a minimal logic for the initialization.

Loggers are configured completely via `logging-server` section in `application.conf` HOCON file, all initial logback configuration has been removed. This opens up a lot of interesting opportunities because we can benefit from all the well maintained slf4j implementations without being to them in terms of functionality.

Most important differences have been outlined in `docs/infrastructure/logging.md`.

# Important Notes
Addresses:
- #7253
- #6739
2023-09-04 09:40:16 +00:00
Pavel Marek
8e49255d92
Invoke all Enso benchmarks via JMH (#7101)
# Important Notes
#### The Plot

- there used to be two kinds of benchmarks: in Java and in Enso
- those in Java got quite a good treatment
- there even are results updated daily: https://enso-org.github.io/engine-benchmark-results/
- the benchmarks written in Enso used to be 2nd class citizen

#### The Revelation
This PR has the potential to fix it all!
- It designs new [Bench API](88fd6fb988) ready for non-batch execution
- It allows for _single benchmark in a dedicated JVM_ execution
- It provides a simple way to wrap such an Enso benchmark as a Java benchmark
- thus the results of Enso and Java benchmarks are [now unified](https://github.com/enso-org/enso/pull/7101#discussion_r1257504440)

Long live _single benchmarking infrastructure for Java and Enso_!
2023-08-07 12:39:01 +00:00
Michał Wawrzyniec Urbańczyk
04ed4c1868
CI fixes (#7375)
This PR:
* restores the previous naming scheme for 64-bit backend artifacts,
* adds uploading of the Test Results reports.
2023-07-24 15:53:03 +02:00
Pavel Marek
efb39ad91c
Fix GraalVM distribution download on MacOS (#7364)
Follow-up of recent GraalVM update #7176 that fixes downloading of GraalVM for Mac - instead of "darwin", the releases are now named "macos"

# Important Notes
Also re-enables the JDK/GraalVM version check as onLoad hook to the `sbt` process. We used to have that check a long time ago. Provides errors like this one if the `sbt` is run with a different JVM version:
```
[error] GraalVM version mismatch - you are running Oracle GraalVM 20.0.1+9.1 but GraalVM 17.0.7 is expected.
[error] GraalVM version check failed.
```
2023-07-23 09:21:21 +00:00
Pavel Marek
cab6968340
Update GraalVM to 17.0.7 (23.0.0 JDK17) (#7176)
- Previous GraalVM update: https://github.com/enso-org/enso/pull/6750

Removed warnings:
- Remove deprecated `ConditionProfile.createCountingProfile()`.
- Add `@Shared` to some `@Cached` parameters (Truffle now emits warnings about potential `@Share` usage).
- Specialization method names should not start with execute
- Add limit attribute to some specialization methods
- Add `@NeverDefault` for some cached initializer expressions
- Add `@Idempotent` or `@NonIdempotent` where appropriate

BigInteger and potential Node inlining are tracked in follow-up issues.

# Important Notes
For `SDKMan` users:
```
sdk install java 17.0.7-graalce
sdk use java 17.0.7-graalce
```

For other users - download link can be found at https://github.com/graalvm/graalvm-ce-builds/releases/tag/jdk-17.0.7

Release notes: https://www.graalvm.org/release-notes/JDK_17/

R component was dropped from the release 23.0.0, only `python` is available to install via `gu install python`.
2023-07-20 15:11:30 +00:00
Jaroslav Tulach
6a6d7dbff3
Simplify onboarding instructions for engine & project-manager developers (#7181)
The current instructions to _build, use and debug_ `project-manager` and its engine/ls process are complicated and require a lot of symlinks to properly point to each other. This pull requests simplifies all of that by introduction of `ENSO_ENGINE_PATH` and `ENSO_JVM_PATH` environment variables. Then it hides all the complexity behind a simple _sbt command_: `runProjectManagerDistribution --debug`.

# Important Notes
I decided to tackle this problem as I have three repositories with different branches of Enso and switching between them requires me to mangle the symlinks. I hope I will not need to do that anymore with the introduction of the `runProjectManagerDistribution` command.
2023-07-17 17:38:54 +00:00
Dmitry Bushev
48f0c6f5e8
Scala 2.13.11 and libraries update (#7010)
Update Scala and libraries.
2023-06-14 13:15:57 +00:00
Dmitry Bushev
6249c79ffd
Update sbt-java-formatter plugin (#7011)
Update java formatter plugin. The new version can remove unused imports.
2023-06-12 14:18:48 +00:00
Pavel Marek
e5c21713e7
Import polyglot java inner classes (#6818)
Adds the ability to import nested inner classes in polyglot java imports.
2023-05-31 09:38:59 +02:00
Jaroslav Tulach
2bc7d346a8
sbt runEngineDistribution --debug to ease debuggging (#6745) 2023-05-18 15:00:52 +02:00
Radosław Waśko
41a8257e8d
Separating Redshift connector from Database library into a new AWS library (#6550)
Related to #5777
2023-05-04 17:36:51 +00:00
Dmitry Bushev
fe0c8fd9a3
Increase the stack size (#6355)
close #6319
2023-04-19 19:09:05 +00:00
Radosław Waśko
6f86115498
Proper implementation of Value Types in Table (#6073)
This is the first part of the #5158 umbrella task. It closes #5158, follow-up tasks are listed as a comment in the issue.

- Updates all prototype methods dealing with `Value_Type` with a proper implementation.
- Adds a more precise mapping from in-memory storage to `Value_Type`.
- Adds a dialect-dependent mapping between `SQL_Type` and `Value_Type`.
- Removes obsolete methods and constants on `SQL_Type` that were not portable.
- Ensures that in the Database backend, operation results are computed based on what the Database is meaning to return (by asking the Database about expected types of each operation).
- But also ensures that the result types are sane.
- While SQLite does not officially support a BOOLEAN affinity, we add a set of type overrides to our operations to ensure that Boolean operations will return Boolean values and will not be changed to integers as SQLite would suggest.
- Some methods in SQLite fallback to a NUMERIC affinity unnecessarily, so stuff like `max(text, text)` will keep the `text` type instead of falling back to numeric as SQLite would suggest.
- Adds ability to use custom fetch / builder logic for various types, so that we can support vendor specific types (for example, Postgres dates).

# Important Notes
- There are some TODOs left in the code. I'm still aligning follow-up tasks - once done I will try to add references to relevant tasks in them.
2023-03-31 16:16:18 +00:00
Hubert Plociniczak
90612cfa35
Report errors when generating indexes (#6123)
`--compile` command would run the compilation pipeline but silently omit any encountered errors, thus skipping the serialization. This maybe was a good idea in the past but it was problematic now that we generate indexes on build time.
This resulted in rather obscure errors (#6092) for modules that were missing their caches.

The change should significantly improve developers' experience when working on stdlib.

# Important Notes
Making compilation more resilient to sudden cache misses is a separate item to be worked on.
2023-03-30 09:59:49 +00:00
Hubert Plociniczak
49e2c5ac1b
Generate index when invoking buildStdLib tasks (#6000)
The change adds support for generating suggestions and bindings when using the convenient task for building individual stdlib components. By default commands do not generate index since it adds build time. But `buildStdLibAllWithIndex` will.

Closes #5999.
2023-03-28 22:28:38 +00:00
Dmitry Bushev
aa97787603
Update ZIO library (#6072)
close #6069

Changelog:
- update: ZIO major version bump
2023-03-28 07:58:59 +00:00
Pavel Marek
9bec3a4e71
Treat Boolean.False as false primitive (#6090)
Treat `Boolean.False` and `Boolean.True` as the corresponding primitives. Now, `Boolean.False == False` returns true.

# Important Notes
`False` and `True` constructs, that are converted to `ConstructorNode` during Truffle codegen, are handled specially in `ConstructorNode`. The easiest fix was to implement a similar special handling in `QualifiedAccessorNode`, although not the cleanest one.

A better solution would be to provide transformation of `Boolean.True` IR to a true literal in `ApplicationSaturation` compiler pass. But `ApplicationSaturation` pass does not handle `True`. Moreover, for our case, it is unnecessarily complicated.
2023-03-28 03:41:49 +00:00
Kaz Wesley
e171fba301
New documentation parser (#5917)
Implement new Enso documentation parser; remove old Scala Enso parser.

Performance: Total time parsing documentation is now ~2ms.

# Important Notes
- Doc parsing is now done only in the frontend.
- Some engine tests had never been switched to the new parser. We should investigate tests that don't pass after the switch: #5894.
- The option to run the old searcher has been removed, as it is obsolete and was already broken before this (see #5909).
- Some interfaces used only by the old searcher have been removed.
2023-03-15 15:43:51 +00:00
Hubert Plociniczak
c21bd3b185
Generate import/export bindings in local cache (#5908)
When generating import/export bindings in local cache, they are included in the distribution.
Additionally, removed the hardcoded value for suggestions cache. Now one can generate them for local as well as for global cache, based on the presence or lack of `--no-global-cache` parameter.

Closes #5890.
2023-03-15 12:40:30 +00:00
Radosław Waśko
263c3ad651
Add a common-polyglot-core-utils project (#5855)
Adds a common project that allows sharing code between the `runtime` and `std-bits`.

Due to classpath separation and the way it is compiled, the classes will be duplicated - we will have one copy for the `runtime` classpath and another copy as a small JAR for `Standard.Base` library.

This is still much better than having the code duplicated - now at least we have a single source of truth for the shared implementations.

Due to the copying we should not expand this project too much, but I encourage to put here any methods that would otherwise require us to copy the code itself.

This may be a good place to put parts of the hashing logic to then allow sharing the logic between the `runtime` and the `MultiValueKey` in the `Table` library (cc: @Akirathan).
2023-03-11 09:27:26 +00:00
Jaroslav Tulach
632a303089
Use SimpleFormatter.formatMessage to replace {0} with actual parameters (#5870)
Fixes #5801 to properly format Truffle log records before sending them for further processing.
2023-03-11 00:15:58 +00:00
Hubert Plociniczak
6769ab0ee7
Downgrade hashing to SHA-1 and other optimizations (#5791)
This change downgrades hashing algorithm used in caching IR and library bindings to SHA-1. It is sufficient and significantly faster for the purpose of simple checksum we use it for.

Additionally, don't calculate the digest for serialized bytes - if we get the expected object type then we are confident about the integrity.

Don't initialize Jackson's ObjectMapper for every metadata serialization/de-serialization. Initialization is very costly.

Avoid unnecessary conversions between Scala and Java. Those back-and-forth `asScala` and `asJava` are pretty expensive.

Finally fix an SBT warning when generating library cache.

Closes https://github.com/enso-org/enso/issues/5763

# Important Notes
The change cuts roughly 0.8-1s from the overall startup.
This change will certainly lead to invalidation of existing caches. It is advised to simply start with a clean slate.
2023-03-09 07:36:59 +00:00
Dmitry Bushev
9397a6ec2f
Pre compute suggestion db during build time (#5698)
Close #5068

Cache suggestions during the `buildEngineDistribution` command, and read them from the disk when the library is loaded. Initial graph coloring takes ~20 seconds vs ~25 seconds on the develop branch.

[peek-develop-branch.webm](https://user-images.githubusercontent.com/357683/223504462-e7d48262-4f5e-4724-b2b0-2cb97fc05140.webm)
[peek-suggestions-branch.webm](https://user-images.githubusercontent.com/357683/223504464-0fe86c04-8c4b-443c-ba96-6c5e2fb1e396.webm)
2023-03-08 12:37:48 +00:00
Hubert Plociniczak
941512e0ba
Optimize import/export resolution (#5700)
This change adds serialization and deserialization of library bindings.
In order to be functional, one needs to first generate IR and
serialize bindings using `--compiled <path-to-library>` command. The bindings
will be stored under the library with `.bindings` suffix.
Bindings are being generated during `buildEngineDistribution` task, thus not
requiring any extra steps.

When resolving import/exports the compiler will first try to load
module's bindings from cache. If successful, it will not schedule its
imports/exports for immediate compilation, as we always did, but use the
bindings info to infer the dependent modules.

The current change does not make any optimizations when it comes to
compiling the modules, yet. It only delays the actual
compilation/loading IR from cache so that it can be done in bulk.
Further optimizations will come from this opportunity such as parallel
loading of caches or lazily inferring only the necessary modules.

Part of https://github.com/enso-org/enso/issues/5568 work.
2023-03-01 08:53:29 +00:00
Jaroslav Tulach
172f72941b
Two findExceptionMessage methods to extract exception messages consistently (#5684)
Creating two `findExceptionMessage` methods in `HostEnsoUtils` and in `VisualizationResult`. Why two? Because one of them is using `org.graalvm.polyglot` SDK as it runs in _"normal Java"_ mode. The other one is using Truffle API as it is running inside of partially evaluated instrument.

There is a `FindExceptionMessageTest` to guarantee consistency between the two methods. It simulates some exceptions in Enso code and checks that both methods extract the same _"message"_ from the exception. The tests verifies hosted and well as Enso exceptions - however testing other polyglot languages is only possible in other modules - as such I created `PolyglotFindExceptionMessageTest` - but that one doesn't have access to Truffle API - e.g. it doesn't really check the consistency - just that a reasonable message is extracted from a JavaScript exception.

# Important Notes
This is not full fix of #5260 - something needs to be done on the IDE side, as the IDE seems to ignore the delivered JSON message - even if it contains properly extracted exception message.
2023-02-20 11:27:16 +00:00
Jaroslav Tulach
f53696eda4
sbt runEngineDistribution ...args... to build and execute the engine (#5609)
Automating the assembly of the engine and its execution into a single task. If you are modifying standard libraries, engine sources or Enso tests, you can launch `sbt` and then just:
```
sbt:enso> runEngineDistribution --run test/Tests/src/Data/Maybe_Spec.enso
[info] Engine package created at built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev
[info] Executing built-distribution/enso-engine-...-dev/bin/enso --run test/Tests/src/Data/Maybe_Spec.enso
Maybe:  [5/5, 30ms]
- should have a None variant [14ms]
- should have a Some variant [5ms]
- should provide the `maybe` function [4ms]
- should provide `is_some` [2ms]
- should provide `is_none` [3ms]
5 tests succeeded.
0 tests failed.unEngineDistribution 4s
0 tests skipped.
```
the [runEngineDistribution](3a581f29ee/docs/CONTRIBUTING.md (running-enso)) `sbt` input task makes sure all your sources are properly compiled and only then executes your enso source. Everything ready at a single press of Enter.

# Important Notes
To debug in chrome dev tools, just add `--inspect`:
```
sbt:enso> runEngineDistribution --inspect --run test/Tests/src/Data/Maybe_Spec.enso
E.g. in Chrome open: devtools://devtools/bundled/js_app.html?ws=127.0.0.1:9229/7JsgjXlntK8
```
everything gets build and one can just attach the Enso debugger.
2023-02-16 21:37:12 +00:00
Radosław Waśko
a2a1aefc14
Check cargo return value to ensure issues are reported back to SBT (#5588)
Before, any failures of the Rust-side of the parser build would be swallowed by sbt, for example if I add gibberish to the Rust code I will get:
<img width="474" alt="image" src="https://user-images.githubusercontent.com/1436948/217374050-fd9ddaca-136c-459e-932e-c4b9e630d610.png">

This is problematic, because when users are compiling in SBT they may get confusing errors about Java files not being found whereas the true cause is hard to track down because it is somewhere deep in the logs. We've run into this silent failure when setting up SBT builds together with @GregoryTravis today.

I suggest to change it so that once cargo fails, the build is failed with a helpful message - this way it will be easier to track down the issues.

With these changes we get:
<img width="802" alt="image" src="https://user-images.githubusercontent.com/1436948/217374531-707ae348-4c55-4d62-9a86-93850ad8086b.png">
2023-02-08 10:46:40 +00:00
Jaroslav Tulach
41b2aac39f
Removing Unsafe.set_atom_field (#4023)
Introducing `Meta.atom_with_hole` to create an `Atom` _with a hole_ that is then _safely_ filled in later.
2023-01-09 13:39:14 +00:00
Jaroslav Tulach
0041b649eb
IGV can jump to JMH sources & more (#4008)
Improvements to behavior and visual appearance of IGV Enso integration.
2022-12-30 05:30:32 +00:00
Jaroslav Tulach
308f4edc21
Tweaks to make benchmarking easier (#3961)
Enabling Frgaal for compilation of benchmarks. Using text blocks for Enso code. Making `--showCompilations` work on GraalVM 22.3 again.
2022-12-08 20:30:19 +00:00
Jaroslav Tulach
402ebb2f8e
Upgrade to GraalVM 22.3.0 (#3663)
Upgrading to GraalVM 22.3.0.

# Important Notes
- Removed all deprecated `FrameSlot`, and replaced them with frame indexes - integers.
- Add more information to `AliasAnalysis` so that it also gathers these indexes.
- Add quick build mode option to `native-image` as default for non-release builds
- `graaljs` and `native-image` should now be downloaded via `gu` automatically, as dependencies.
- Remove `engine-runner-native` project - native image is now build straight from `engine-runner`.
- We used to have `engine-runner-native` without `sqldf` in classpath as a workaround for an internal native image bug.
- Fixed chrome inspector integration, such that it shows values of local variables both for current stack frame and caller stack frames.
- There are still many issues with the debugging in general, for example, when there is a polyglot value among local variables, a `NullPointerException` is thrown and no values are displayed.
- Removed some deprecated `native-image` options
- Remove some deprecated Truffle API method calls.
2022-11-23 14:30:48 +00:00
Jaroslav Tulach
9cfd6aae26
Proper classpath of engine sources in Enso4Igv plugin (#3810)
This PR modifies `sbt` to record options sent to [frgaal](http://frgaal.org) compiler. Then it reads these options (especially exact classpath used during compilation) from IGV or NetBeans.

# Important Notes
![Open project in IGV](https://user-images.githubusercontent.com/26887752/201684275-b3ee7a37-7b55-4290-b426-75df0280ba32.png)
2022-11-15 07:05:53 +00:00
Marcin Kostrzewa
23e04f905f
Another attempt at M1 compilation (#3859) 2022-11-09 15:26:25 +00:00
James Dunkerley
45276b243d
Expanding Derived Columns and Expression Syntax (#3782)
- Added expression ANTLR4 grammar and sbt based build.
- Added expression support to `set` and `filter` on the Database and InMemory `Table`.
- Added expression support to `aggregate` on the Database and InMemory `Table`.
- Removed old aggregate functions (`sum`, `max`, `min` and `mean`) from `Column` types.
- Adjusted database `Column` `+` operator to do concatenation (`||`) when text types.
- Added power operator `^` to both `Column` types.
- Adjust `iif` to allow for columns to be passed for `when_true` and `when_false` parameters.
- Added `is_present` to database `Column` type.
- Added `coalesce`, `min` and `max` functions to both `Column` types performing row based operation.
- Added support for `Date`, `Time_Of_Day` and `Date_Time` constants in database.
- Added `read` method to InMemory `Column` returning `self` (or a slice).

# Important Notes
- Moved approximate type computation to `SQL_Type`.
- Fixed issue in `LongNumericOp` where it was always casting to a double.
- Removed `head` from InMemory Table (still has `first` method).
2022-11-08 15:57:59 +00:00
Jaroslav Tulach
db13084828
Changing the enso_parser library location to match Intel Mac (#3850) 2022-11-03 12:50:28 +01:00
Jaroslav Tulach
85f71cbfc8
Including enso_parser library in the engine distribution (#3842)
Make sure `libenso_parser.so`, `.dll` or `.dylib` are packaged and included when `sbt buildEngineDistribution`.

# Important Notes
There was [a discussion](https://discord.com/channels/401396655599124480/1036562819644141598) about proper location of the library. It was concluded that _"there's no functional difference between a dylib and a jar."_ and as such the library is placed in `component` folder.

Currently the old parser is still used for parsing. This PR just integrates the build system changes and makes us ready for smooth flipping of the parser in the future as part of #3611.
2022-11-02 17:13:53 +00:00
Radosław Waśko
bc09c7b4c2
Remove obsolete Rust parser experiment, superseded by the much more mature new Rust parser integration (#3815)
We've had an old attempt at integrating a Rust parser with our Scala/Java projects. It seems to have been abandoned and is not used anywhere - it is also superseded by the new integration of the Rust parser. I think it was used as an experiment to see how to approach such an integration.

Since it is not used anymore - it make sense to remove it, because it only adds some (slight, but non-zero) maintenance effort. We can always bring it back from git history if necessary.
2022-10-24 14:56:07 +00:00
Pavel Marek
e9260227c4
Duration type is a builtin type (#3759)
- Reimplement the `Duration` type to a built-in type.
- `Duration` is an interop type.
- Allow Enso method dispatch on `Duration` interop coming from different languages.

# Important Notes
- The older `Duration` type should now be split into new `Duration` builtin type and a `Period` type.
- This PR does not implement `Period` type, so all the `Period`-related functionality is currently not working, e.g., `Date - Period`.
- This PR removes `Integer.milliseconds`, `Integer.seconds`, ..., `Integer.years` extension methods.
2022-10-14 18:08:08 +00:00
Radosław Waśko
592a8516a8
Add Is_Empty, Not_Empty, Like and Not_Like to Filter_Condition (#3775)
Implements https://www.pivotaltracker.com/story/show/183389890
2022-10-10 23:11:04 +00:00
Kaz Wesley
44a031f9f0
Parser: Full constructor syntax for type definitions; Field syntax; Complex operator sections; Template functions; Text improvements; Operator methods; eliminate Unsupported; better ArgumentDefinitions (#3716)
I believe all parse failures remaining after these changes are because the new parser is intentionally stricter about some things. I'll be reviewing those failures and opening a bug to change the library/tests code.

Implements:
- https://www.pivotaltracker.com/story/show/182941610: full type def syntax
- https://www.pivotaltracker.com/story/show/182497490: field syntax
- https://www.pivotaltracker.com/story/show/182497395: complex operator sections
- https://www.pivotaltracker.com/story/show/182497236: template functions
- `codeRepr` without leading whitespace
- text literals: interpret escape sequences in lexer
- the multiline text-literal left-trim algorithm
- type operator-methods
- the `<=` operator is no longer treated as a modifier
- https://www.pivotaltracker.com/story/show/183315038: eliminate Unsupported
- use ArgumentDefinition for type constructor arguments
- more detailed ArgumentDefinition type
2022-10-05 04:45:31 +00:00
Hubert Plociniczak
096fcfee82
Generate native image for engine-runner (#3638)
This PR adds a possibility to generate native-image for engine-runner.
Note that due to on-demand loading of stdlib, programs that make use of it are currently not yet supported
(that will be resolved at a later point).
The purpose of this PR is only to make sure that we can generate a bare minimum runner because due to lack TruffleBoundaries or misconfiguration in reflection config, this can get broken very easily.
To generate a native image simply execute:
```
sbt> engine-runner-native/buildNativeImage
... (wait a few minutes)
```
The executable is called `runner` and can be tested via a simple test that is in the resources. To illustrate the benefits
see the timings difference between the non-native and native one:
```
>time built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --no-ir-caches --in-project test/Tests/ --run engine/runner-native/src/test/resources/Factorial.enso 6
720

real	0m4.503s
user	0m9.248s
sys	0m1.494s
> time ./runner --run engine/runner-native/src/test/resources/Factorial.enso 6
720

real	0m0.176s
user	0m0.042s
sys	0m0.038s
```

# Important Notes
Notice that due to a [bug in GraalVM](https://github.com/oracle/graal/issues/4200), which is already fixed in 22.x, and us still being on 21.x for the time being, I had to add a workaround to our sbt build to build a different fat jar for native image. To workaround it I had to exclude sqlite jar. Hence native image task is on `engine-runner-native` and not on `engine-runner`.

Will need to add the above command to CI.
2022-09-22 14:45:10 +00:00
Jaroslav Tulach
9134f9b2d7
EnsoCompilerTest to verify compatibility of parsers (#3723)
Adding new _compatibility test_ `EnsoCompilerTest` to verify the new Rust based parser can produce the same `IR` as the original `AST` based one. The simplest way to execute the test from an empty repository is:
```bash
enso$ sbt bootstrap
enso$ sbt "testOnly *EnsoCompilerTest"
```

There are [GitHub Actions run](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266212#step:9:5187) on Linux as well as [run on Windows](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266370#step:9:5254) that show `EnsoCompilerTest` is being executed by the CI (good, as that means `.so` was properly built and linked to the JVM running the test). The [linux](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266212#step:9:5187) as well as [windows](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266370#step:9:5254) runs also demonstrate that failures in the `EnsoCompilerTest` suite fail the CI.

# Important Notes
Right now [there are five test failures](https://github.com/enso-org/enso/actions/runs/3087664644/jobs/4993266212#step:9:5187) - waiting for @kazcw to make sure `codeRepr()` doesn't contain spaces. However, as this PR is more about the infrastructure, I am disabling the currently failing tests in [031169b](031169bd05)
2022-09-20 15:50:27 +00:00
Dmitry Bushev
5e114acbb5
Update Scala to 2.13.8 (#3631)
Update Scala compiler and libraries.
2022-08-08 19:32:55 +00:00
Jaroslav Tulach
030e46bfb4
Switching to Java 19 Frgaal compiler (#3594)
Updating to Frgaal 19.0.0-RC1 to give us access to latest Java features including pattern matching on `record` classes. Builds upon #3421.
2022-07-19 09:28:31 +00:00
Jaroslav Tulach
dc30e44b60
Register instruments/language in their own compilation units to fix the sbt build issues (#3509)
New plan to [fix the `sbt` build](https://www.pivotaltracker.com/n/projects/2539304/stories/182209126) and its annoying:
```
log.error(
"Truffle Instrumentation is not up to date, " +
"which will lead to runtime errors\n" +
"Fixes have been applied to ensure consistent Instrumentation state, " +
"but compilation has to be triggered again.\n" +
"Please re-run the previous command.\n" +
"(If this for some reason fails, " +
s"please do a clean build of the $projectName project)"
)
```
When it is hard to fix `sbt` incremental compilation, let's restructure our project sources so that each `@TruffleInstrument` and `@TruffleLanguage` registration is in individual compilation unit. Each such unit is either going to be compiled or not going to be compiled as a batch - that will eliminate the `sbt` incremental compilation issues without addressing them in `sbt` itself.

fa2cf6a33ec4a5b2e3370e1b22c2b5f712286a75 is the first step - it introduces `IdExecutionService` and moves all the `IdExecutionInstrument` API up to that interface. The rest of the `runtime` project then depends only on `IdExecutionService`. Such refactoring allows us to move the `IdExecutionInstrument` out of `runtime` project into independent compilation unit.
2022-06-13 14:09:08 +00:00
Hubert Plociniczak
31e3f39c55
Suppress pointless warnings coming from SBT (#3499)
This change introduces a custom LogManager for console that allows for
excluding certain log messages. The primarily reason for introducing
such LogManager/Appender is to stop issuing hundreds of pointless
warnings coming from the analyzing compiler (wrapper around javac) for
classes that are being generated by annotation processors.

The output looks like this:
```
[info] Cannot install GraalVM MBean due to Failed to load org.graalvm.nativebridge.jni.JNIExceptionWrapperEntryPoints
[info] compiling 129 Scala sources and 395 Java sources to /home/hubert/work/repos/enso/enso/engine/runtime/target/scala-2.13/classes ...
[warn] Unexpected javac output: warning: File for type 'org.enso.interpreter.runtime.type.ConstantsGen' created in the last round will not be subject to annotation processing.
[warn] 1 warning.
[info] [Use -Dgraal.LogFile=<path> to redirect Graal log output to a file.]
[info] Cannot install GraalVM MBean due to Failed to load org.graalvm.nativebridge.jni.JNIExceptionWrapperEntryPoints
[info] foojavac Filer
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.number.decimal.CeilMethodGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.resource.TakeNodeGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.error.ThrowErrorMethodGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.number.smallInteger.MultiplyMethodGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.warning.GetWarningsNodeGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.number.smallInteger.BitAndMethodGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.error.ErrorToTextNodeGen
[warn] Could not determine source for class org.enso.interpreter.node.expression.builtin.warning.GetValueMethodGen
[warn] Could not determine source for class org.enso.interpreter.runtime.callable.atom.AtomGen$MethodDispatchLibraryExports$Cached
....
```

The output now has over 500 of those and there will be more. Much more
(generated by our and Truffle processors).
There is no way to tell SBT that those are OK. One could potentially
think of splitting compilation into 3 stages (Java processors, Java and
Scala) but that will already complicate the non-trivial build definition
and we may still end up with the initial problem.
This is a fix to make it possible to get reasonable feedback from
compilation without scrolling mutliple screens *every single time*.

Also fixed a spurious warning in javac processor complaining about
creating files in the last round.

Related to https://www.pivotaltracker.com/story/show/182138198
2022-06-01 13:50:46 +00:00
Michał Wawrzyniec Urbańczyk
14a01c4635
New IDE build script (#3466) 2022-05-23 04:16:04 +02:00
Hubert Plociniczak
26ef0f6b00
Include only runtime dependencies in std-bits (#3445)
`provided` classifier is completely omitted in the update report so we
cannot filter on that during `.select`.
Instead, we only consider runtime configuration which lists all the
necessary dependencies needed to run things.

This is a follow up on #182093808. With this change frgaal compiler is no
longer packaged.
2022-05-12 10:44:46 +00:00
Hubert Plociniczak
d8a2832624
Integrate Frgaal in Java libs (#3433)
This PR allows for easier integration of Frgaal in different subprojects.
Related to https://www.pivotaltracker.com/story/show/182093808
2022-05-11 11:21:01 +00:00
Hubert Plociniczak
72c4006cb1
Additional sbt commands to build stdlib packages (#3431)
Auxiliary sbt commands for building individual
stdlib packages.
The commands check if the engine distribution was built at least once,
and only copy the necessary package files if necessary.
So far added:
- `buildStdLibBase`
- `buildStdLibDatabase`
- `buildStdLibTable`
- `buildStdLibImage`
- `buildStdLibGoogle_Api`

Related to [#182014385](https://www.pivotaltracker.com/story/show/182014385)
2022-05-11 10:12:18 +00:00
Hubert Plociniczak
79c82da21c
Frgaal integration in sbt (#3421)
* Initial integration with Frgaal in sbt

Half-working since it chokes on generated classes from annotation
processor.

* Replace AutoService with ServiceProvider

For reasons unknown AutoService would fail to initialize and fail to
generate required builtin method classes.
Hidden error message is not particularly revealing on the reason for
that:
```
[error] error: Bad service configuration file, or exception thrown while constructing Processor object: javax.annotation.processing.Processor: Provider com.google.auto.service.processor.AutoServiceProcessor could not be instantiated
```

The sample records is only to demonstrate that we can now use newer Java
features.

* Cleanup + fix benchmark compilation

Bench requires jmh classes which are not available because we obviously
had to limit `java.base` modules to get Frgaal to work nicely.
For now, we default to good ol' javac for Benchmarks.
Limiting Frgaal to runtime for now, if it plays nicely, we can expand it
to other projects.

* Update CHANGELOG

* Remove dummy record class

* Update licenses

* New line

* PR review

* Update legal review

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
2022-05-04 21:18:40 +02:00
Radosław Waśko
14257d07aa
Data analysts should be able to use Text.split, Text.lines and Text.words to break up strings (#3415)
Implements https://www.pivotaltracker.com/story/show/181266184

### Important Notes

Changed example image download to only proceed if the file did not exist before - thus cutting on the build time (the build used to download it _every_ time - which completely failed the build if network is down). A redownload can be forced by performing a fresh repository checkout.
2022-04-26 17:22:53 +02:00
Hubert Plociniczak
059bb8c7e9
Apply automatic formatting prior to turning on checks (#3405)
Result of automatic formatting with `scalafmtAll` and `javafmtAll`.
Prerequisite for https://github.com/enso-org/enso/pull/3394

### Important Notes

This touches a lot of files and might conflict with existing PRs that are in progress. If that's the case, just run
`scalafmtAll` and `javafmtAll` after merge and everything should be in order since formatters should be deterministic.
2022-04-19 12:34:34 +02:00
Dmitry Bushev
792c7e3538
Fix dev version check (#3265) 2022-02-15 16:34:33 +01:00
Edward Kmett
0c25ee736c
Upgrade Truffle and Graal to Version 21.3.0 (#3258) 2022-02-11 19:05:13 +03:00
Michał Wawrzyniec Urbańczyk
4baad5f146
Nightly proccess preparations: Setting Enso version through the environment (#3241)
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
Co-authored-by: Radosław Waśko <wasko.radek@gmail.com>
2022-02-07 15:14:32 +01:00
Dmitry Bushev
ca0a6f0bf6
Update Scala to 2.13.7 (#3214) 2021-12-31 17:50:32 +03:00
Michał Wawrzyniec Urbańczyk
66c256a1f7
Self-hosted Engine CI for Windows and Linux (#3182) 2021-12-27 17:56:35 +01:00
Radosław Waśko
46c31bb9a5
Preinstalling With Dependencies (#1981) 2021-11-23 11:51:17 +03:00
Radosław Waśko
d1882580fb
Simple Fix for Native Image Rebuild (#1989) 2021-09-08 22:59:04 +02:00
Radosław Waśko
0a60e5180a
Update Simple Library Server (#1952) 2021-08-18 10:01:28 +02:00
Radosław Waśko
be6e60509a
Tool To Bump Stdlib Version (#1947) 2021-08-13 18:14:20 +02:00
Radosław Waśko
fac0405fd0
Updating Editions (#1944) 2021-08-12 16:55:23 +02:00
Radosław Waśko
5e86343a9c
Make Legal Notices Order Deterministic (#1891) 2021-07-22 13:00:23 +02:00
Marcin Kostrzewa
f55d66cb2c
XLS(X) Reading (#1879) 2021-07-20 13:32:19 +02:00
Radosław Waśko
2eadf9f19f
Bump Minimal PM/Launcher Versions for New Engine (#1854) 2021-07-13 13:08:00 +02:00
Radosław Waśko
b58e895b18
Editions Integration (#1832) 2021-07-08 15:38:20 +02:00
Marcin Kostrzewa
334a022ffd
Import syntax including namespace (#1806) 2021-06-24 12:42:24 +02:00
Radosław Waśko
db537bf0a8
Add some tweaks to the native builds (#1779) 2021-06-15 10:13:22 +01:00
Ara Adkins
1bdf87ce76
Update the memory limit for Native Image (#1746) 2021-05-20 14:03:14 +01:00
Ara Adkins
48bcebc723
Update to GraalVM 21.1.0 (#1738) 2021-05-14 13:08:39 +01:00
Ara Adkins
9c9f3b2728
Enhance the Standard.Image examples (#1725) 2021-05-05 09:53:36 +01:00
Dmitry Bushev
65e9cca5a4
Add Python and R to the GraalVM Bundle (#1644)
CI release pipeline is updated to install
python and R GraalVM components
2021-04-07 17:19:23 +03:00
Radosław Waśko
d48008be51
Fix symlinks and increase compression level (#1631)
Copying used to follow symlinks, instead I just extract the archive
directly to the destination.
2021-03-31 15:33:02 +02:00
Ara Adkins
8290dedcda
Bump sbt to fix an issue with the scala REPL (#1562) 2021-03-10 12:00:35 +00:00
Ara Adkins
2b4f0ef2ad
Bump scalac to 2.13.5 (#1531) 2021-03-01 16:35:57 +00:00
Radosław Waśko
6544c2478d
Implement the first part of the database library (#1475) 2021-02-25 13:48:18 +00:00
Ara Adkins
8ec2a92b77
Upgrade GraalVM to 21.0 (#1487) 2021-02-22 13:32:55 +00:00
Marcin Kostrzewa
c4a0772190
Polyglot JS Definitions (#1451) 2021-02-08 18:14:54 +01:00
Radosław Waśko
10bccf6b56
Better component handling in the project manager (#1399) 2021-01-15 15:26:51 +00:00
Radosław Waśko
d30a80eedc
Fix the project manager reflection configuration (#1394) 2021-01-14 10:46:01 +00:00
Radosław Waśko
25561e4c8c
Project Manager Native Image (#1318) 2020-12-09 14:58:11 +01:00
Radosław Waśko
9e1b49d245
Forking Language Server in the Project Manager (#1305) 2020-12-02 16:56:47 +01:00
Radosław Waśko
35efd8ea55
Separate ComponentManager from Launcher (#1248) 2020-10-30 12:31:31 +01:00