Commit Graph

470 Commits

Author SHA1 Message Date
Jaroslav Tulach
fe2cf49568
Run whole test/Base_Tests in native image runner (#10296) 2024-06-21 06:03:53 +02:00
Hubert Plociniczak
114b3a5c5e
Cleaning up YAML parsing in preparation for circe-yaml removal (#10309)
The current implementation contains logic that should enable us to make some backward compatibility config changes.
At the same time, the logic is tightly integrated with circe's JSON library, which we want to eventually to get rid off.
Rather than trying to keep it somehow around and maintain via some hacks this PR proposes to ditch that logic completely as we currently have no use-case for such scenarios.

As a result, classes modelling YAML configs now don't have the extra fields and there is 1:1 correspondence.

Performance has also improved although that wasn't the main objective, yet. Follow up PR will attempt to replace `circe-yaml` with `snakeyaml` directly.

In preparation for #9113. Note that the dependency upgrade is necessary because it brings latest available `snakeyaml` (as part of `circe-yaml`).
2024-06-20 13:07:54 +00:00
Jaroslav Tulach
dee9e079d4
Enso language support with parser in VSCode, IGV, etc. (#7054)
Outline view and completions for Enso code in VSCode.

# Important Notes
This PR provides the necessary infrastructure for building VSCode extension that includes `enso_parser` library compiled for all supported platforms.

VSCode extension can now use libraries from `sbt` that are `publishM2`-ready. To make that possible a documentation must have been provided and fixed for those modules - hence so many changes in `.scala` classes.

<img width="862" alt="image" src="https://github.com/enso-org/enso/assets/26887752/7374bf41-bdc6-4322-b562-85a2e761de2a">

Last, but not least. The outline view and completions display something.
2024-06-14 14:01:37 +00:00
Hubert Plociniczak
d6d370925a
Removing dependencies to speed up startup (#10249)
Reducing the number of dependencies. Explicit `cats` are almost gone (present in `cli`). `enumeration` is completely gone.  `cats` is also still included implicitly via `io.circe` but that's a different kind of beast.
Also, really removed `jackson` from dependencies by fixing the dependency on `http-test-helper`.

# Important Notes
In a number of places importing all cats implicits could be simply replaced with a single or two method calls. Not to mention that this will reduce compilation times due to reduced implicit search space.

One example of how the changes affect performance (not only startup):

Before:
![Screenshot from 2024-06-11 12-05-24](https://github.com/enso-org/enso/assets/292128/a1a772a9-635d-4a16-a543-e2fd2124a22c)
Now:
![Screenshot from 2024-06-11 14-27-47](https://github.com/enso-org/enso/assets/292128/b17c7fcc-9a6d-48b9-8200-60708354ee03)
(frequently executed)

![Screenshot from 2024-06-12 12-46-34](https://github.com/enso-org/enso/assets/292128/31bc4dfd-4edc-45c9-9c5d-13e3472089b9)
Also appears to be gone.

This PR is by no means finished. The purge will continue in follow up PRs.
2024-06-12 18:15:36 +00:00
Hubert Plociniczak
4da5e611c9
Replace Jackson serde (#10035)
JSON serialization setup between Language Server and Runtime is a major contributor to startup time. This PR experiments with an alternative implementation that remedies the problem.
The new serializer uses [jsoniter-scala](https://github.com/plokhotnyuk/jsoniter-scala) which by some accounts claims to be really fast. In our case, more importantly, we pay negligible cost of startup setup compared to Jackson which was horribly slow.

# Important Notes
Before:
![Screenshot from 2024-06-06 15-35-18](https://github.com/enso-org/enso/assets/292128/56103b82-777e-459f-966e-abdef25c2430)
After:
![Screenshot from 2024-06-06 15-35-02](https://github.com/enso-org/enso/assets/292128/00a36647-dfae-4dc8-a1b7-bf69069ef109)

Yes. About 0.8sec.
2024-06-11 15:03:12 +00:00
Jaroslav Tulach
4c84b57c49
Publish Persistance API+Javadoc to local m2 repository (#10222) 2024-06-10 14:00:51 +02:00
Radosław Waśko
99a1d052a0
Fixes to Persistance (#10101)
- Follow-up to #9361
- Enables assertions and fixes `count` check
- Tests and fixes null references
- Tests and fixes serializing a deserialized structure - by saving the id of the `Persistance` corresponding to the entry
- After the change to how we determine which `Persistance` instance to use, the most specific one is now used (based on the saved id). This has an unfortunate consequence that `Seq` which is most of the time represented by a subtype of `List`, is now using `PersistScalaList` which is not lazy.
- To alleviate that, we no longer use `Seq` to store some field lazily and instead use a dedicated type for that purpose: `InlineReference`.
2024-06-10 11:49:12 +00:00
Pavel Marek
2ea2a57651
ydoc-server is a separate module (#10156)
- Remove remnants of deprecated Scala parser
- The following projects are now JPMS modules provided on system module-path (in components directory):
  - `ydoc-server`
  - `profiling-utils`
  - `syntax-rust-definition`
- The contents of the aforementioned modules are excluded from both `runner.jar` and `runtime.jar` fat jars.
- Suggestions are serialized and deserialized with our Persistance framework, rather than via the default Java OutputObjectWriter.
2024-06-07 12:56:42 +02:00
Dmitry Bushev
106007cb89
Implement the ydoc js bundle test (#10160)
close #9929

Changelog:
- update: customize Ydoc main hostname and port with environment variables
- add: Ydoc initialization test
2024-06-04 17:45:26 +00:00
Pavel Marek
270f708e4f
Implement private methods (#10060)
Add support for private methods. Most of the changes are in parser and compiler. The runtime checking of private functions was already present since #9692

# Important Notes
- Only top-level methods can be declared `private`.
- private method cannot be called from different project
- private method cannot be accessed from polyglot code (private method does not exist for polyglot code)
2024-05-31 08:00:20 +00:00
Pavel Marek
9027596b1b
Fix build of stdlib benchmarks after ydoc PR (#10140)
A quickfix for a broken build of `std-benchmarks`. A recent error is, e.g., in https://github.com/enso-org/enso/actions/runs/9294745663/job/25580483426#step:7:1343. io.helidon modules are missed on the module-path when invoking std lib benchmarks. This PR adds them via dependency on `componentModulesPath`.
2024-05-30 16:33:48 +00:00
Pavel Marek
4f49a7fad6
Remove spurious warnings from zinc compiler (#10133)
* Ensure all relevant sbt projects have annotationProcSetting

* Remove unused `syntax-rust-definition/javaModuleName` key.

* Add annotationProcSetting to more projects
2024-05-30 18:24:04 +02:00
Pavel Marek
56b289ae79
Move common testing functionality to a separate project (#10112)
Introduce a new `test-utils` project, and moves the `TestBase` there. Moreover, `TestBase` is renamed to `TestUtils` and is no longer an abstract class.

# Important Notes
`test-utils` project does not depend on junit, so it can be used, for example, by any benchmarks as well.
2024-05-29 11:50:03 +00:00
Dmitry Bushev
858e646328
Start Ydoc with the language server (#9862)
- related #7954

Changelog:
- update: Ydoc starts with the language server on the `localhost:1234` by default. The hostname and ports can be configured by setting environment variables `LANGUAGE_SERVER_YDOC_HOSTNAME` and `LANGUAGE_SERVER_YDOC_PORT`
- update: by default `npm dev run` uses the node Ydoc server. You can control it with `POLYGLOT_YDOC_SERVER` env variable. For example,
```
env POLYGLOT_YDOC_SERVER='true' npm --workspace=enso-gui2 run dev
```
To connect to the Ydoc server running on the 1234 port (the one started with the language server)
⠀
```
env POLYGLOT_YDOC_SERVER='ws://127.0.0.1:1235' npm --workspace=enso-gui2 run dev
```
To connect to the provided URL. Can be useful for debugging when you start a separate Ydoc process.
- update: run `npm install` before the engine build. It is required to create the Ydoc JS bundle.
2024-05-28 13:51:42 +00:00
Dmitry Bushev
2c060a2a92
Show JUnit info on test completion (#10097)
Show JUnit test execution info:
```
sbt:enso> ydoc-server/test
[info] Test run org.enso.ydoc.polyfill.web.EventEmitterTest finished: 0 failed, 0 ignored, 2 total, 1.017
[info] Test run org.enso.ydoc.polyfill.web.CryptoTest finished: 0 failed, 0 ignored, 3 total, 0.074s
[info] Test run org.enso.ydoc.polyfill.web.EventTargetTest finished: 0 failed, 0 ignored, 2 total, 0.038s
[info] Test run org.enso.ydoc.polyfill.web.AbortControllerTest finished: 0 failed, 0 ignored, 2 total, 0.041s
[info] Test run org.enso.ydoc.polyfill.ParserPolyfillTest finished: 0 failed, 0 ignored, 3 total, 0.042s
```
2024-05-27 14:40:45 +00:00
Jaroslav Tulach
16c1b74218
Enso Library Feature to execute (a bit of) Base_Tests (#9997) 2024-05-23 08:20:19 +02:00
Jaroslav Tulach
fe28c23658
Table of Persistance.Reference at the end of the stream (#9972)
Fixes #9361 by delaying storing of `Persistance.Reference` instances and creating their table at the end of the stream.
2024-05-18 17:33:04 +00:00
Jaroslav Tulach
5c06535c46
Importing (some of) Standard.Base works from NI runner (#9866) 2024-05-17 14:42:35 +02:00
Hubert Plociniczak
a5ca9f9ac2
Reduce ops when setting exisitng execution env (#9967)
Setting execution environment to the existing one should have no effect.
Should (positively) affect startup in #9789.

# Important Notes
Cancelling jobs and triggering a fresh execute job is expensive and unnecessary, especially on startup, when the result should be the same as before.
2024-05-16 20:59:13 +00:00
Radosław Waśko
1d61c08dff
Followup improvements to the license review tool (#9895)
- Closes #9122
- Promised follow-up of #9782
2024-05-11 07:51:11 +00:00
Jaroslav Tulach
a69d89b274
Reduce set of JDK modules required by Enso (#9868) 2024-05-07 14:13:21 +02:00
Hubert Plociniczak
930f3c593e
Minor follow up to #9528 (#9841)
While playing with the implementation addressed some PR comments of mine and applied DRY.
2024-05-07 08:12:43 +00:00
Pavel Marek
c36cd87b99
Run all benchmarks with -Xss16M (#9839) 2024-05-02 16:18:41 +02:00
Dmitry Bushev
5995a00958
Run ydoc-server with GraalVM (#9528)
part of #7954

# Important Notes
The workflow is:
- `$ npm install` -- just in case
- `$ npm --workspace=enso-gui2 run build-ydoc-server-polyglot` -- build the `ydocServer.js` bundle
- `$ sbt ydoc-server/assembly` -- build the ydoc server jar
- `env POLYGLOT_YDOC_SERVER=true npm --workspace=enso-gui2 run dev` -- run the dev server with the polyglot ydoc server. Providing `POLYGLOT_YDOC_SERVER_DEBUG=true` env variable enables the chrome debugger
2024-05-02 06:28:57 +00:00
Hubert Plociniczak
38df86e33d
Compile parser using musl (#9806)
To workaround problems with `glibc` we need to build parser in the backend with C stdlib statically linked using musl. Uses the arguments presented in #9521.

# Important Notes
No problems in tests so far.
2024-04-29 13:22:41 +00:00
Hubert Plociniczak
25224a8acf
Compile-time shapeless in runtime-compiler (#9805)
Marking the `shapeless` dependency as compile-time only.

# Important Notes
If we want to ditch the dependency completely then, if we want to keep the check in place, we would have to replace with its runtime equivalent:
```
--- a/engine/runtime-compiler/src/main/scala/org/enso/compiler/pass/IRPass.scala
+++ b/engine/runtime-compiler/src/main/scala/org/enso/compiler/pass/IRPass.scala
@@ -148,6 +148,9 @@ object IRPass {
def unsafeAs[T <: Metadata: ClassTag]: T = {
+      if (implicitly[ClassTag[T]].runtimeClass == Metadata.getClass) {
+        throw new InternalError("Type argument must be specified")
+      }
```
Not sure if we want to do it still after this change.

Note that `project-manager` uses that and some other definitions as well.
2024-04-29 09:50:05 +00:00
Jaroslav Tulach
c5bf2384e4
Removing Truffle API dependency from runtime-compiler project (#9785)
Fixes #8888 by removing dependency on Truffle API and `polyglot-api` from `runtime-compiler` project.
2024-04-26 13:14:22 +00:00
Radosław Waśko
df4a7efcdd
Add AWS SSO JARs to the Standard.AWS library (#9782)
As reported by our users, when using the AWS SSO, our code was failing with:
```
Execution finished with an error: To use Sso related properties in the 'xyz' profile, the 'sso' service module must be on the class path.
```

This PR adds the missing JARs to fix that.

Additionally it improves the license review tool UX a bit (parts of #9122):
- sorting the report by amount of problems, so that dependencies with unresolved problems appear at the top,
- semi-automatic helper button to rename package configurations after a version bump,
- button to remove stale entries from config (files or copyrights that disappeared after update),
- button to add custom copyright notice text straight from the report UI,
- button to set a file as the license for the project (creating the `custom-license` file automatically)
- ability to filter processed projects - e.g. `openLegalReviewReport AWS` will only run on the AWS subproject - saving time processing unchanged dependencies,
- updated the license search heuristic, fixing a problem with duplicates:
- if we had dependencies `netty-http` and `netty-http2`, because of a prefix-check logic, the notices for `netty-http` would also appear again for `netty-http2`, which is not valid. I have improved the heuristic to avoid these false positives and removed them from the current report.
- WIP: button to mark a license type as reviewed (not finished in this PR).
2024-04-25 18:44:51 +00:00
Jaroslav Tulach
931baa4276
Towards simplifying runtime-compiler dependencies (#8894) 2024-04-25 10:03:42 +02:00
Hubert Plociniczak
58009b7c04
In-memory suggestions (#9751)
This change replaces an sqllite-backed suggestions' repo with a simple, in-memory, one.
As `completion` functionality has been implemented completely in GUI, there is no need to support it in backend, which simplifies a lot of functionality.

Closes #9650 and #9471.

# Important Notes
Loading suggestions and sending them to GUI on startup is almost instantaneous. Previously it would take ~10s just for `Standard.Base`.
2024-04-22 11:02:17 +00:00
Pavel Marek
0d915d0344
Update to GraalVM 24.0.0 (#9647)
Update the GraalVM-related Maven packages from **23.1.2** to **24.0.0**.

# Important Notes
- Reverted workarounds mentioned in https://github.com/enso-org/enso/issues/9477

The language home search functionality in Truffle changed. It used to find the home for Enso from the class loader of `EnsoLanguage`:
https://github.com/oracle/graal/blob/graal-23.1.2/truffle/src/com.oracle.truffle.polyglot/src/com/oracle/truffle/polyglot/LanguageCache.java#L362

In 24.0.0, the language home is no longer found via class loader, but via system properties:
https://github.com/oracle/graal/blob/graal-24.0.0/truffle/src/com.oracle.truffle.polyglot/src/com/oracle/truffle/polyglot/LanguageCache.java#L353

I provided a "fix" for this by an explicit specification of the `org.graalvm.language.enso.home` system property in our launchers.
2024-04-12 17:01:49 +00:00
Michał Wawrzyniec Urbańczyk
cdf031f61f
[CI] Provide latest FlatBuffers to the Engine build. (#9654)
This PR bumps the FlatBuffers version used by the backend to `24.3.25` (the latest version as of now).

Since the newer FlatBuffers releases come with prebuilt binaries for all platforms we target, we can simplify the build process by simply downloading the required `flatc` binary from the official FlatBuffers GitHub release page. This allows us to remove the dependency on `conda`, which was the only reliable way to get the outdated `flatc`.

The `conda` setup has been removed from the CI steps and the relevant code has been removed from the build script.

The FlatBuffers version is no longer hard-coded in the Rust build script, it is inferred from the `build.sbt` definition (similar to GraalVM).

# Important Notes
This does not affect the GUI binary protocol implementation.

While I initially wanted to update it, it turned out farly non-trivial.

As there are multiple issues with the generated TS code, it was significantly refactored by hand and it is impossible to automatically update it. Work to address this problem is left as [a future task](https://github.com/enso-org/enso/issues/9658).

As the Flatbuffers binary protocol is guaranteed to be compatible between versions (unlike the generated sources), there should be no adverse effects from bumping `flatc` only on the backend side.
2024-04-12 10:10:44 +00:00
Hubert Plociniczak
5d7cbccb77
Potentially fix further std-benchmarks issues (#9541) 2024-03-27 09:31:15 +00:00
Hubert Plociniczak
d488841722
Make sure std-benchmarks has all dependencies (#9504) 2024-03-21 10:57:40 +00:00
James Dunkerley
2f0d99a1cb
Snowflake Connectivity (#9435)
* Initial connection to Snowflake via an account, username and password.

* Fix databases and schemas in Snowflake.
Add warehouses.

* Add warehouse.
Update schema dropdowns.

* Add ability to set warehouse and pass at connect.

* Fix for NPE in license review

* scalafmt

* Separate Snowflake from Database.

* Scala fmt.

* Legal Review

* Avoid using ARROW for snowflake.

* Tidy up Entity_Naming_Properties.

* Fix for separating Entity_Namimg_Properties.

* Allow some tweaking of Postgres dialect to allow snowflake to use as well.

* Working on reading Date, Time and Date Times.

* Changelog.

* Java format.

* Make Snowflake Time and TimeStamp stuff work.
Move some responsibilities to Type_Mapping.

* Make Snowflake Time and TimeStamp stuff work.
Move some responsibilities to Type_Mapping.

* fix

* Update distribution/lib/Standard/Database/0.0.0-dev/src/Connection/Connection.enso

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>

* PR comments.

* Last refactor for PR.

* Fix.

---------

Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
2024-03-20 10:06:12 +00:00
Pavel Marek
c4029eed4f
dry-run benchmarks exits when some benchmark fails (#9397)
If some benchmark fails in dry-run (compileOnly) mode, the whole process exits with non-zero return code. Also fixes failing engine compiler benchmarks.

# Important Notes
Manually added failure:
```diff
diff --git a/engine/runtime-benchmarks/src/main/java/org/enso/interpreter/bench/benchmarks/semantic/ArrayProxyBenchmarks.java b/engine/runtime-benchmarks/src/main/java/org/enso/interpreter/bench/benchmarks/semantic/ArrayProxyBenchmarks.java
index c8d86cecc..f9f4d7cbc 100644
--- a/engine/runtime-benchmarks/src/main/java/org/enso/interpreter/bench/benchmarks/semantic/ArrayProxyBenchmarks.java
+++ b/engine/runtime-benchmarks/src/main/java/org/enso/interpreter/bench/benchmarks/semantic/ArrayProxyBenchmarks.java
@@ -95,7 +95,8 @@ public class ArrayProxyBenchmarks {

@Benchmark
public void sumOverComputingProxy(Blackhole matter) {
-    performBenchmark(matter);
+    //performBenchmark(matter);
+    throw new AssertionError("My error");
}

@Benchmark
```
Run with `sbt "-Dbench.compileOnly=true runtime-benchmarks/benchOnly org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy"` fails with:
```
[info] Running benchmarks [org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy] in compileOnly mode
[info] # JMH version: 1.36
[info] # VM version: JDK 21.0.2, Java HotSpot(TM) 64-Bit Server VM, 21.0.2+13-LTS-jvmci-23.1-b30
[info] # VM invoker: /home/pavel/.sdkman/candidates/java/21.0.2-graal/bin/java
[info] # VM options: -XX:ThreadPriorityPolicy=1 -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCIProduct -XX:-UnlockExperimentalVMOptions -Dslf4j.provider=org.slf4j.nop.NOPServiceProvider -Dbench.compileOnly=true --module-path=/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/nativeimage/23.1.2/nativeimage-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/word/23.1.2/word-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/jniutils/23.1.2/jniutils-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/collections/23.1.2/collections-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/polyglot/polyglot/23.1.2/polyglot-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-api/23.1.2/truffle-api-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-runtime/23.1.2/truffle-runtime-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-compiler/23.1.2/truffle-compiler-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/js/js-language/23.1.2/js-language-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/regex/regex/23.1.2/regex-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/shadowed/icu4j/23.1.2/icu4j-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/python/python-language/23.1.2/python-language-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/python/python-resources/23.1.2/python-resources-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/bouncycastle/bcutil-jdk18on/1.76/bcutil-jdk18on-1.76.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/bouncycastle/bcpkix-jdk18on/1.76/bcpkix-jdk18on-1.76.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk18on/1.76/bcprov-jdk18on-1.76.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/llvm/llvm-api/23.1.2/llvm-api-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-nfi/23.1.2/truffle-nfi-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-nfi-libffi/23.1.2/truffle-nfi-libffi-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/tools/profiler-tool/23.1.2/profiler-tool-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/shadowed/json/23.1.2/json-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/tukaani/xz/1.9/xz-1.9.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/slf4j/slf4j-api/2.0.9/slf4j-api-2.0.9.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/slf4j/slf4j-nop/2.0.9/slf4j-nop-2.0.9.jar:/home/pavel/dev/enso/runtime.jar --add-modules=org.enso.runtime --add-exports=org.slf4j.nop/org.slf4j.nop=org.slf4j
[info] # Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
[info] # Warmup: <none>
[info] # Measurement: 1 iterations, 1 s each
[info] # Timeout: 10 min per iteration
[info] # Threads: 1 thread, will synchronize iterations
[info] # Benchmark mode: Average time, time/op
[info] # Benchmark: org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy
[info] # Run progress: 0.00% complete, ETA 00:00:01
[info] # Fork: N/A, test runs in the host VM
[info] # *** WARNING: Non-forked runs may silently omit JVM options, mess up profilers, disable compiler hints, etc. ***
[info] # *** WARNING: Use non-forked runs only for debugging purposes, not for actual performance runs. ***
[error] SLF4J: Attempting to load provider "org.slf4j.nop.NOPServiceProvider" specified via "slf4j.provider" system property
[info] Iteration   1: <failure>
[info] java.lang.AssertionError: My error
[info] 	at org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy(ArrayProxyBenchmarks.java:99)
[info] 	at org.enso.interpreter.bench.benchmarks.semantic.jmh_generated.ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.sumOverComputingProxy_avgt_jmhStub(ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.java:232)
[info] 	at org.enso.interpreter.bench.benchmarks.semantic.jmh_generated.ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.sumOverComputingProxy_AverageTime(ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.java:173)
[info] 	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[info] 	at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[info] 	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:475)
[info] 	at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:458)
[info] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[info] 	at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[info] 	at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[info] 	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[info] 	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] Benchmark run failed: Benchmark caught the exception
[info] 	at java.base/java.lang.Thread.run(Thread.java:1583)
[error] org.openjdk.jmh.runner.RunnerException: Benchmark caught the exception
[error] 	at org.openjdk.jmh.runner.Runner.runBenchmarks(Runner.java:575)
[error] 	at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:310)
[error] 	at org.openjdk.jmh.runner.Runner.run(Runner.java:209)
[error] 	at org.enso.interpreter.bench.BenchmarksRunner.runCompileOnly(BenchmarksRunner.java:93)
[error] 	at org.enso.interpreter.bench.BenchmarksRunner.run(BenchmarksRunner.java:36)
[error] 	at org.enso.interpreter.bench.benchmarks.RuntimeBenchmarksRunner.main(RuntimeBenchmarksRunner.java:8)
[error] Caused by: org.openjdk.jmh.runner.BenchmarkException: Benchmark error during the run
[error] 	at org.openjdk.jmh.runner.BenchmarkHandler.runIteration(BenchmarkHandler.java:424)
[error] 	at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:281)
[error] 	at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:233)
[error] 	at org.openjdk.jmh.runner.BaseRunner.doSingle(BaseRunner.java:138)
[error] 	at org.openjdk.jmh.runner.BaseRunner.runBenchmarksEmbedded(BaseRunner.java:110)
[error] 	at org.openjdk.jmh.runner.Runner.runBenchmarks(Runner.java:555)
[error] 	... 5 more
[error] 	Suppressed: java.lang.AssertionError: My error
[error] 		at org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy(ArrayProxyBenchmarks.java:99)
[error] 		at org.enso.interpreter.bench.benchmarks.semantic.jmh_generated.ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.sumOverComputingProxy_avgt_jmhStub(ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.java:232)
[error] 		at org.enso.interpreter.bench.benchmarks.semantic.jmh_generated.ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.sumOverComputingProxy_AverageTime(ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.java:173)
[error] 		at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[error] 		at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[error] 		at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:475)
[error] 		at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:458)
[error] 		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error] 		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] 		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] 		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] 		at java.base/java.lang.Thread.run(Thread.java:1583)
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 5 s, completed Mar 13, 2024, 12:49:59 PM
```
2024-03-14 15:21:38 +00:00
Dmitry Bushev
2322b40a22
Cleanup runtime project settings (#9398)
Remove redundant compiler settings from the runtime project.
2024-03-13 14:31:52 +00:00
Pavel Marek
6e498e12cf
sbt graalVMVersionCheck does not fail during reload (#9396)
Reload of sbt used to fail on "Not a valid command: graalVMVersionCheck" This PR fixes this issue.

# Important Notes
Checked:
- Manually running `reload` inside sbt shell
- Temporarily setting a key in the current sbt shell with `set javaOptions += "foo"` and then `reload`.

Invoking sbt with incompatible java:
```
[info] welcome to sbt 1.9.7 (GraalVM Community Java 21)
[info] loading settings for project enso-build from plugins.sbt ...
[info] loading project definition from /home/pavel/dev/enso/project
[info] loading settings for project enso from build.sbt ...
[info] resolving key references (66170 settings) ...
[info] set current project to enso (in build file:/home/pavel/dev/enso/)
[error] Running on GraalVM version 21. Expected GraalVM version 21.0.2.
[warn] Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? (default: r)
```
2024-03-13 12:16:13 +00:00
Pavel Marek
af73768d14
Declare HTTPDownloaderTest as flaky (#9339)
`HTTPDownloaderTest` failed recently transiently. Let's declare it as flaky.
2024-03-12 09:53:55 +00:00
Hubert Plociniczak
f80dd9f593
Demonstrate usage of foreign arrow function (#9150)
Including arrow language in the distribution by default. Added a basic example for creating an Arrow array.
Making sure that memory layout agrees with Arrow specification (padding, continuous allocation of memory chunks).
Related to #9118.

This should unblock work on allowing serialization/deserialization to/from Parquet but I'd like to delay it to a follow up ticket as it is going to be a significant amount of specialized work.
2024-03-08 15:20:55 +00:00
Pavel Marek
96bfcb3e63
Add compiler benchmarks (#9158)
Add compiler benchmarks to `engine/runtime-benchmarks`. All the benchmarks generate source code on the fly into `engine/runtime-benchmarks/target/bench-data` directory. Random data generators are set with the same seed. For the convenience of reviewers, I am attaching the benchmark sources in [bench-data.zip](https://github.com/enso-org/enso/files/14423372/bench-data.zip).

I have created benchmarks that measure the performance of a whole module compilation, and benchmarks that measure the performance of inline compilation. They directly call `run` and `runInline` methods on `org.enso.compiler.Compiler`.

# Important Notes
- The results will be available in https://enso-org.github.io/engine-benchmark-results/engine-benchs.html in a few days after the merge of this PR.
- The benchmark parameters are tweaked so that an average iteration takes less than 400 ms and more than 30 ms.
- Ensured that the benchmarks measure performance of the compiler, for example:
![image](https://github.com/enso-org/enso/assets/14013887/c870f4ad-1418-4812-85f2-ca9664711163)
2024-03-01 13:37:18 +00:00
Hubert Plociniczak
0133e9ef4f
Drop inefficient SemVer implementation (#9089)
`Bump` library uses parser combinators behind the scenes which are known to be good at expressing grammars but are not performance-oriented.

This change ditches the dependency in favour of an existing Java implementation. `jsemver` implements the full specification, which is probably an overkill in our case, but proved to be an almost drop-in replacement for the previous library.

Closes #8692

# Important Notes
Peformance improvements:
- roughly 50ms compared to the previous approach (from 80ms to 20-40ms)

I don't see any time spent in the new implementation during startup so it could be potentially aggressively inlined.
Further more, we could use a facade and offer our own strip down version of semver.
2024-02-22 09:59:09 +00:00
Pavel Marek
9daca288f4
Java and Graal versions are checked before the build starts (#9106)
Add checks of Java and GraalVM versions before the `sbt` project is fully loaded. This ensures that all the devs have exactly the version specified in our `build.sbt`.

# Important Notes
Trying to start `sbt` with a different java versions now results in:

```
$ java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing)
$ sbt
[info] welcome to sbt 1.9.7 (GraalVM Community Java 21)
[info] loading settings for project enso-build from plugins.sbt ...
[info] loading project definition from /home/pavel/dev/enso/project
[info] loading settings for project enso from build.sbt ...
[info] resolving key references (65272 settings) ...
[info] set current project to enso (in build file:/home/pavel/dev/enso/)
[error] Running on GraalVM version 21. Expected GraalVM version 21.0.2.
[error] Total time: 0 s, completed Feb 20, 2024, 1:06:18 PM
[warn] Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? (default: r)
```

```
$ java -version
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment JBR-17.0.10+1-1087.17-jcef (build 17.0.10+1-b1087.17)
OpenJDK 64-Bit Server VM JBR-17.0.10+1-1087.17-jcef (build 17.0.10+1-b1087.17, mixed mode)
$ sbt
[info] welcome to sbt 1.9.7 (JetBrains s.r.o. Java 17.0.10)
[info] loading settings for project enso-build from plugins.sbt ...
[info] loading project definition from /home/pavel/dev/enso/project
[info] compiling 44 Scala sources to /home/pavel/dev/enso/project/target/scala-2.12/sbt-1.0/classes ...
[info] loading settings for project enso from build.sbt ...
[info] resolving key references (65272 settings) ...
[info] set current project to enso (in build file:/home/pavel/dev/enso/)
[warn] Running on non-GraalVM JVM (The actual java.vendor is JetBrains s.r.o.). Expected GraalVM Community java.vendor.
[error] Running on Java version 17. Expected Java version 21.
[error] Total time: 0 s, completed Feb 20, 2024, 1:07:40 PM
[warn] Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? (default: r)
```
2024-02-21 10:33:32 +00:00
Pavel Marek
96082c3bae
Remove akka from runtime (#8953)
There are two projects transitively required by `runtime`, that have akka dependencies:
- `downloader`
- `connected-lock-manager`

This PR replaces the `akka-http` dependency in `downloader` by HttpClient from JDK, and splits `connected-lock-manager` into two projects such that there are no akka classes in `runtime.jar`.

# Important Notes
- Simplify the `downloader` project - remove akka.
- Add HTTP tests to the `downloader` project that uses our `http-test-helper` that is normally used for stdlib tests.
- It required few tweaks so that we can embed that server in a unit test.
- Split `connected-lock-manager` project into two projects - remove akka from `runtime`.
- **Native image build fixes and quality of life improvements:**
- Output of `native-image` is captured 743e167aa4
- The output will no longer be intertwined with the output from other commands on the CI.
- Arguments to the `native-image` are passed via an argument file, not via command line - ba0a69de6e
- This resolves an issue on Windows with "Command line too long", for example in https://github.com/enso-org/enso/actions/runs/7934447148/job/21665456738?pr=8953#step:8:2269
2024-02-19 16:39:05 +00:00
Jaroslav Tulach
f1d4e546d5
Upgrading to GraalVM 21.0.2 (#8883)
Upgrades to latest GraalVM 21.0.2
2024-02-19 12:08:59 +00:00
Cassandra-Clark
88c1cc14f7
8830 add google analytics authentication and reporting to enso (#8907)
Updates Google_Api version for authentication and adds Google Analytics reporting api and run_google_report method.

This is an initial method for proof of concept, with further design changes to follow.

# Important Notes
Updates google-api-client to v 2.2.0 from 1.35.2
Adds google-analytics-data v 0.44.0
2024-02-13 16:23:48 +00:00
Pavel Marek
5919eda753
Fix incremental compilation of runtime/test (#8975) 2024-02-13 10:05:31 +01:00
Jaroslav Tulach
427f5b0410
Keep only module-info.class in the root of runtime-fat-jar (#8961) 2024-02-06 06:23:55 +01:00
Jaroslav Tulach
2028d6b2c1
Better runEngineDistribution --run test/Base_Tests "filter" (#8940) 2024-02-04 11:39:36 +01:00
Pavel Marek
a70cbacecf
Remove fansi dependency from runtime-compiler (#8847)
Moves `fansi` dependency from `runtime-compiler` into `runtime`.

# Important Notes
I have not refactored [DiagnosticFormatter.scala](https://github.com/enso-org/enso/pull/8847/files#diff-8e73cf562742d6b0510acfe30af940fb9252e32be27a023f9705908a464e08ed) into Java just yet - I don't know what should be the replacement for now. I have just moved that source from `runtime-compiler` to `runtime`.
2024-02-02 11:45:19 +00:00