part of #7954
# Important Notes
The workflow is:
- `$ npm install` -- just in case
- `$ npm --workspace=enso-gui2 run build-ydoc-server-polyglot` -- build the `ydocServer.js` bundle
- `$ sbt ydoc-server/assembly` -- build the ydoc server jar
- `env POLYGLOT_YDOC_SERVER=true npm --workspace=enso-gui2 run dev` -- run the dev server with the polyglot ydoc server. Providing `POLYGLOT_YDOC_SERVER_DEBUG=true` env variable enables the chrome debugger
To workaround problems with `glibc` we need to build parser in the backend with C stdlib statically linked using musl. Uses the arguments presented in #9521.
# Important Notes
No problems in tests so far.
Marking the `shapeless` dependency as compile-time only.
# Important Notes
If we want to ditch the dependency completely then, if we want to keep the check in place, we would have to replace with its runtime equivalent:
```
--- a/engine/runtime-compiler/src/main/scala/org/enso/compiler/pass/IRPass.scala
+++ b/engine/runtime-compiler/src/main/scala/org/enso/compiler/pass/IRPass.scala
@@ -148,6 +148,9 @@ object IRPass {
def unsafeAs[T <: Metadata: ClassTag]: T = {
+ if (implicitly[ClassTag[T]].runtimeClass == Metadata.getClass) {
+ throw new InternalError("Type argument must be specified")
+ }
```
Not sure if we want to do it still after this change.
Note that `project-manager` uses that and some other definitions as well.
As reported by our users, when using the AWS SSO, our code was failing with:
```
Execution finished with an error: To use Sso related properties in the 'xyz' profile, the 'sso' service module must be on the class path.
```
This PR adds the missing JARs to fix that.
Additionally it improves the license review tool UX a bit (parts of #9122):
- sorting the report by amount of problems, so that dependencies with unresolved problems appear at the top,
- semi-automatic helper button to rename package configurations after a version bump,
- button to remove stale entries from config (files or copyrights that disappeared after update),
- button to add custom copyright notice text straight from the report UI,
- button to set a file as the license for the project (creating the `custom-license` file automatically)
- ability to filter processed projects - e.g. `openLegalReviewReport AWS` will only run on the AWS subproject - saving time processing unchanged dependencies,
- updated the license search heuristic, fixing a problem with duplicates:
- if we had dependencies `netty-http` and `netty-http2`, because of a prefix-check logic, the notices for `netty-http` would also appear again for `netty-http2`, which is not valid. I have improved the heuristic to avoid these false positives and removed them from the current report.
- WIP: button to mark a license type as reviewed (not finished in this PR).
This change replaces an sqllite-backed suggestions' repo with a simple, in-memory, one.
As `completion` functionality has been implemented completely in GUI, there is no need to support it in backend, which simplifies a lot of functionality.
Closes#9650 and #9471.
# Important Notes
Loading suggestions and sending them to GUI on startup is almost instantaneous. Previously it would take ~10s just for `Standard.Base`.
This PR bumps the FlatBuffers version used by the backend to `24.3.25` (the latest version as of now).
Since the newer FlatBuffers releases come with prebuilt binaries for all platforms we target, we can simplify the build process by simply downloading the required `flatc` binary from the official FlatBuffers GitHub release page. This allows us to remove the dependency on `conda`, which was the only reliable way to get the outdated `flatc`.
The `conda` setup has been removed from the CI steps and the relevant code has been removed from the build script.
The FlatBuffers version is no longer hard-coded in the Rust build script, it is inferred from the `build.sbt` definition (similar to GraalVM).
# Important Notes
This does not affect the GUI binary protocol implementation.
While I initially wanted to update it, it turned out farly non-trivial.
As there are multiple issues with the generated TS code, it was significantly refactored by hand and it is impossible to automatically update it. Work to address this problem is left as [a future task](https://github.com/enso-org/enso/issues/9658).
As the Flatbuffers binary protocol is guaranteed to be compatible between versions (unlike the generated sources), there should be no adverse effects from bumping `flatc` only on the backend side.
* Initial connection to Snowflake via an account, username and password.
* Fix databases and schemas in Snowflake.
Add warehouses.
* Add warehouse.
Update schema dropdowns.
* Add ability to set warehouse and pass at connect.
* Fix for NPE in license review
* scalafmt
* Separate Snowflake from Database.
* Scala fmt.
* Legal Review
* Avoid using ARROW for snowflake.
* Tidy up Entity_Naming_Properties.
* Fix for separating Entity_Namimg_Properties.
* Allow some tweaking of Postgres dialect to allow snowflake to use as well.
* Working on reading Date, Time and Date Times.
* Changelog.
* Java format.
* Make Snowflake Time and TimeStamp stuff work.
Move some responsibilities to Type_Mapping.
* Make Snowflake Time and TimeStamp stuff work.
Move some responsibilities to Type_Mapping.
* fix
* Update distribution/lib/Standard/Database/0.0.0-dev/src/Connection/Connection.enso
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
* PR comments.
* Last refactor for PR.
* Fix.
---------
Co-authored-by: Radosław Waśko <radoslaw.wasko@enso.org>
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
If some benchmark fails in dry-run (compileOnly) mode, the whole process exits with non-zero return code. Also fixes failing engine compiler benchmarks.
# Important Notes
Manually added failure:
```diff
diff --git a/engine/runtime-benchmarks/src/main/java/org/enso/interpreter/bench/benchmarks/semantic/ArrayProxyBenchmarks.java b/engine/runtime-benchmarks/src/main/java/org/enso/interpreter/bench/benchmarks/semantic/ArrayProxyBenchmarks.java
index c8d86cecc..f9f4d7cbc 100644
--- a/engine/runtime-benchmarks/src/main/java/org/enso/interpreter/bench/benchmarks/semantic/ArrayProxyBenchmarks.java
+++ b/engine/runtime-benchmarks/src/main/java/org/enso/interpreter/bench/benchmarks/semantic/ArrayProxyBenchmarks.java
@@ -95,7 +95,8 @@ public class ArrayProxyBenchmarks {
@Benchmark
public void sumOverComputingProxy(Blackhole matter) {
- performBenchmark(matter);
+ //performBenchmark(matter);
+ throw new AssertionError("My error");
}
@Benchmark
```
Run with `sbt "-Dbench.compileOnly=true runtime-benchmarks/benchOnly org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy"` fails with:
```
[info] Running benchmarks [org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy] in compileOnly mode
[info] # JMH version: 1.36
[info] # VM version: JDK 21.0.2, Java HotSpot(TM) 64-Bit Server VM, 21.0.2+13-LTS-jvmci-23.1-b30
[info] # VM invoker: /home/pavel/.sdkman/candidates/java/21.0.2-graal/bin/java
[info] # VM options: -XX:ThreadPriorityPolicy=1 -XX:+UnlockExperimentalVMOptions -XX:+EnableJVMCIProduct -XX:-UnlockExperimentalVMOptions -Dslf4j.provider=org.slf4j.nop.NOPServiceProvider -Dbench.compileOnly=true --module-path=/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/nativeimage/23.1.2/nativeimage-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/word/23.1.2/word-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/jniutils/23.1.2/jniutils-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/collections/23.1.2/collections-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/polyglot/polyglot/23.1.2/polyglot-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-api/23.1.2/truffle-api-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-runtime/23.1.2/truffle-runtime-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-compiler/23.1.2/truffle-compiler-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/js/js-language/23.1.2/js-language-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/regex/regex/23.1.2/regex-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/shadowed/icu4j/23.1.2/icu4j-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/python/python-language/23.1.2/python-language-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/python/python-resources/23.1.2/python-resources-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/bouncycastle/bcutil-jdk18on/1.76/bcutil-jdk18on-1.76.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/bouncycastle/bcpkix-jdk18on/1.76/bcpkix-jdk18on-1.76.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk18on/1.76/bcprov-jdk18on-1.76.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/llvm/llvm-api/23.1.2/llvm-api-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-nfi/23.1.2/truffle-nfi-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-nfi-libffi/23.1.2/truffle-nfi-libffi-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/tools/profiler-tool/23.1.2/profiler-tool-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/shadowed/json/23.1.2/json-23.1.2.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/tukaani/xz/1.9/xz-1.9.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/slf4j/slf4j-api/2.0.9/slf4j-api-2.0.9.jar:/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/slf4j/slf4j-nop/2.0.9/slf4j-nop-2.0.9.jar:/home/pavel/dev/enso/runtime.jar --add-modules=org.enso.runtime --add-exports=org.slf4j.nop/org.slf4j.nop=org.slf4j
[info] # Blackhole mode: compiler (auto-detected, use -Djmh.blackhole.autoDetect=false to disable)
[info] # Warmup: <none>
[info] # Measurement: 1 iterations, 1 s each
[info] # Timeout: 10 min per iteration
[info] # Threads: 1 thread, will synchronize iterations
[info] # Benchmark mode: Average time, time/op
[info] # Benchmark: org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy
[info] # Run progress: 0.00% complete, ETA 00:00:01
[info] # Fork: N/A, test runs in the host VM
[info] # *** WARNING: Non-forked runs may silently omit JVM options, mess up profilers, disable compiler hints, etc. ***
[info] # *** WARNING: Use non-forked runs only for debugging purposes, not for actual performance runs. ***
[error] SLF4J: Attempting to load provider "org.slf4j.nop.NOPServiceProvider" specified via "slf4j.provider" system property
[info] Iteration 1: <failure>
[info] java.lang.AssertionError: My error
[info] at org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy(ArrayProxyBenchmarks.java:99)
[info] at org.enso.interpreter.bench.benchmarks.semantic.jmh_generated.ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.sumOverComputingProxy_avgt_jmhStub(ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.java:232)
[info] at org.enso.interpreter.bench.benchmarks.semantic.jmh_generated.ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.sumOverComputingProxy_AverageTime(ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.java:173)
[info] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[info] at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[info] at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:475)
[info] at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:458)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[info] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[info] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[info] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] Benchmark run failed: Benchmark caught the exception
[info] at java.base/java.lang.Thread.run(Thread.java:1583)
[error] org.openjdk.jmh.runner.RunnerException: Benchmark caught the exception
[error] at org.openjdk.jmh.runner.Runner.runBenchmarks(Runner.java:575)
[error] at org.openjdk.jmh.runner.Runner.internalRun(Runner.java:310)
[error] at org.openjdk.jmh.runner.Runner.run(Runner.java:209)
[error] at org.enso.interpreter.bench.BenchmarksRunner.runCompileOnly(BenchmarksRunner.java:93)
[error] at org.enso.interpreter.bench.BenchmarksRunner.run(BenchmarksRunner.java:36)
[error] at org.enso.interpreter.bench.benchmarks.RuntimeBenchmarksRunner.main(RuntimeBenchmarksRunner.java:8)
[error] Caused by: org.openjdk.jmh.runner.BenchmarkException: Benchmark error during the run
[error] at org.openjdk.jmh.runner.BenchmarkHandler.runIteration(BenchmarkHandler.java:424)
[error] at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:281)
[error] at org.openjdk.jmh.runner.BaseRunner.runBenchmark(BaseRunner.java:233)
[error] at org.openjdk.jmh.runner.BaseRunner.doSingle(BaseRunner.java:138)
[error] at org.openjdk.jmh.runner.BaseRunner.runBenchmarksEmbedded(BaseRunner.java:110)
[error] at org.openjdk.jmh.runner.Runner.runBenchmarks(Runner.java:555)
[error] ... 5 more
[error] Suppressed: java.lang.AssertionError: My error
[error] at org.enso.interpreter.bench.benchmarks.semantic.ArrayProxyBenchmarks.sumOverComputingProxy(ArrayProxyBenchmarks.java:99)
[error] at org.enso.interpreter.bench.benchmarks.semantic.jmh_generated.ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.sumOverComputingProxy_avgt_jmhStub(ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.java:232)
[error] at org.enso.interpreter.bench.benchmarks.semantic.jmh_generated.ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.sumOverComputingProxy_AverageTime(ArrayProxyBenchmarks_sumOverComputingProxy_jmhTest.java:173)
[error] at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:103)
[error] at java.base/java.lang.reflect.Method.invoke(Method.java:580)
[error] at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:475)
[error] at org.openjdk.jmh.runner.BenchmarkHandler$BenchmarkTask.call(BenchmarkHandler.java:458)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:572)
[error] at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:317)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144)
[error] at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642)
[error] at java.base/java.lang.Thread.run(Thread.java:1583)
[error] Nonzero exit code returned from runner: 1
[error] (Compile / run) Nonzero exit code returned from runner: 1
[error] Total time: 5 s, completed Mar 13, 2024, 12:49:59 PM
```
Reload of sbt used to fail on "Not a valid command: graalVMVersionCheck" This PR fixes this issue.
# Important Notes
Checked:
- Manually running `reload` inside sbt shell
- Temporarily setting a key in the current sbt shell with `set javaOptions += "foo"` and then `reload`.
Invoking sbt with incompatible java:
```
[info] welcome to sbt 1.9.7 (GraalVM Community Java 21)
[info] loading settings for project enso-build from plugins.sbt ...
[info] loading project definition from /home/pavel/dev/enso/project
[info] loading settings for project enso from build.sbt ...
[info] resolving key references (66170 settings) ...
[info] set current project to enso (in build file:/home/pavel/dev/enso/)
[error] Running on GraalVM version 21. Expected GraalVM version 21.0.2.
[warn] Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? (default: r)
```
Including arrow language in the distribution by default. Added a basic example for creating an Arrow array.
Making sure that memory layout agrees with Arrow specification (padding, continuous allocation of memory chunks).
Related to #9118.
This should unblock work on allowing serialization/deserialization to/from Parquet but I'd like to delay it to a follow up ticket as it is going to be a significant amount of specialized work.
Add compiler benchmarks to `engine/runtime-benchmarks`. All the benchmarks generate source code on the fly into `engine/runtime-benchmarks/target/bench-data` directory. Random data generators are set with the same seed. For the convenience of reviewers, I am attaching the benchmark sources in [bench-data.zip](https://github.com/enso-org/enso/files/14423372/bench-data.zip).
I have created benchmarks that measure the performance of a whole module compilation, and benchmarks that measure the performance of inline compilation. They directly call `run` and `runInline` methods on `org.enso.compiler.Compiler`.
# Important Notes
- The results will be available in https://enso-org.github.io/engine-benchmark-results/engine-benchs.html in a few days after the merge of this PR.
- The benchmark parameters are tweaked so that an average iteration takes less than 400 ms and more than 30 ms.
- Ensured that the benchmarks measure performance of the compiler, for example:
`Bump` library uses parser combinators behind the scenes which are known to be good at expressing grammars but are not performance-oriented.
This change ditches the dependency in favour of an existing Java implementation. `jsemver` implements the full specification, which is probably an overkill in our case, but proved to be an almost drop-in replacement for the previous library.
Closes#8692
# Important Notes
Peformance improvements:
- roughly 50ms compared to the previous approach (from 80ms to 20-40ms)
I don't see any time spent in the new implementation during startup so it could be potentially aggressively inlined.
Further more, we could use a facade and offer our own strip down version of semver.
Add checks of Java and GraalVM versions before the `sbt` project is fully loaded. This ensures that all the devs have exactly the version specified in our `build.sbt`.
# Important Notes
Trying to start `sbt` with a different java versions now results in:
```
$ java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing)
$ sbt
[info] welcome to sbt 1.9.7 (GraalVM Community Java 21)
[info] loading settings for project enso-build from plugins.sbt ...
[info] loading project definition from /home/pavel/dev/enso/project
[info] loading settings for project enso from build.sbt ...
[info] resolving key references (65272 settings) ...
[info] set current project to enso (in build file:/home/pavel/dev/enso/)
[error] Running on GraalVM version 21. Expected GraalVM version 21.0.2.
[error] Total time: 0 s, completed Feb 20, 2024, 1:06:18 PM
[warn] Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? (default: r)
```
```
$ java -version
openjdk version "17.0.10" 2024-01-16
OpenJDK Runtime Environment JBR-17.0.10+1-1087.17-jcef (build 17.0.10+1-b1087.17)
OpenJDK 64-Bit Server VM JBR-17.0.10+1-1087.17-jcef (build 17.0.10+1-b1087.17, mixed mode)
$ sbt
[info] welcome to sbt 1.9.7 (JetBrains s.r.o. Java 17.0.10)
[info] loading settings for project enso-build from plugins.sbt ...
[info] loading project definition from /home/pavel/dev/enso/project
[info] compiling 44 Scala sources to /home/pavel/dev/enso/project/target/scala-2.12/sbt-1.0/classes ...
[info] loading settings for project enso from build.sbt ...
[info] resolving key references (65272 settings) ...
[info] set current project to enso (in build file:/home/pavel/dev/enso/)
[warn] Running on non-GraalVM JVM (The actual java.vendor is JetBrains s.r.o.). Expected GraalVM Community java.vendor.
[error] Running on Java version 17. Expected Java version 21.
[error] Total time: 0 s, completed Feb 20, 2024, 1:07:40 PM
[warn] Project loading failed: (r)etry, (q)uit, (l)ast, or (i)gnore? (default: r)
```
There are two projects transitively required by `runtime`, that have akka dependencies:
- `downloader`
- `connected-lock-manager`
This PR replaces the `akka-http` dependency in `downloader` by HttpClient from JDK, and splits `connected-lock-manager` into two projects such that there are no akka classes in `runtime.jar`.
# Important Notes
- Simplify the `downloader` project - remove akka.
- Add HTTP tests to the `downloader` project that uses our `http-test-helper` that is normally used for stdlib tests.
- It required few tweaks so that we can embed that server in a unit test.
- Split `connected-lock-manager` project into two projects - remove akka from `runtime`.
- **Native image build fixes and quality of life improvements:**
- Output of `native-image` is captured 743e167aa4
- The output will no longer be intertwined with the output from other commands on the CI.
- Arguments to the `native-image` are passed via an argument file, not via command line - ba0a69de6e
- This resolves an issue on Windows with "Command line too long", for example in https://github.com/enso-org/enso/actions/runs/7934447148/job/21665456738?pr=8953#step:8:2269
Updates Google_Api version for authentication and adds Google Analytics reporting api and run_google_report method.
This is an initial method for proof of concept, with further design changes to follow.
# Important Notes
Updates google-api-client to v 2.2.0 from 1.35.2
Adds google-analytics-data v 0.44.0
- ✅Linting fixes and groups.
- ✅Add `File.from that:Text` and use `File` conversions instead of taking both `File` and `Text` and calling `File.new`.
- ✅Align Unix Epoc with the UTC timezone and add converting from long value to `Date_Time` using it.
- ❌Add simple first logging API allowing writing to log messages from Enso.
- ✅Fix minor style issue where a test type had a empty constructor.
- ❌Added a `long` based array builder.
- Added `File_By_Line` to read a file line by line.
- Added "fast" JSON parser based off Jackson.
- ✅Altered range `to_vector` to be a proxy Vector.
- ✅Added `at` and `get` to `Database.Column`.
- ✅Added `get` to `Table.Column`.
- ✅Added ability to expand `Vector`, `Array` `Range`, `Date_Range` to columns.
- ✅Altered so `expand_to_column` default column name will be the same as the input column (i.e. no `Value` suffix).
- ✅Added ability to expand `Map`, `JS_Object` and `Jackson_Object` to rows with two columns coming out (and extra key column).
- ✅ Fixed bug where couldn't use integer index to expand to rows.
Uniqueness check of `UpsertVisualizationJob` only involved expressionId. Apparently now GUI sends mutliple visualizations for the same expressions and expects all of them to exist. Since previously we would cancel duplicate jobs, this was problematic.
This change makes sure that uniqueness also takes into account visualization id. Fixed a few logs that were not passing arguments properly.
Closes#8801
# Important Notes
I have not noticed any more problems with loading visualizations so the issue appears to be resolved with this change.
Added a unit test case that would previously fail due to cancellation of a job that upserts visualization.
This is a quick fix to a long standing problem of
`org.enso.interpreter.service.error.FailedToApplyEditsException` which would prevent backend from processing any more changes, rendering GUI (and backend) virtually useless.
Edits are submitted for (background) processing in the order they are handled. However the order of execution of such tasks is not guaranteed. Most of the time edits are processed in the same order as their requests but when they don't, files get quickly out of sync.
Related to #8770.
# Important Notes
I'm not a fan of this change because it essentially blocks all open/file requests until all edits are processed and we already have logic to deal with that appropriately. Moreover those tasks can and should be processed independently. Since we already had the single thread executor present to ensure correct synchronization of open/file/push commands, we are simply adding edit commands to the list.
Ideally we want to have a specialized executor that executes tasks within the same group sequentially but groups of tasks can be executed in parallel, thus ensuring sufficient throughput. The latter will take much longer and will require significant rewrite of the command execution.
Added tests that would previously fail due to non-deterministic execution.
Since #8685, there is the following error (warning) message from JPMS plugin when building `runtime-test-instruments`:
```
[error] Returned (10): Vector(/home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/nativeimage/23.1.0/nativeimage-23.1.0.jar, /home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/word/23.1.0/word-23.1.0.jar, /home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/jniutils/23.1.0/jniutils-23.1.0.jar, /home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/collections/23.1.0/collections-23.1.0.jar, /home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/polyglot/polyglot/23.1.0/polyglot-23.1.0.jar, /home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-api/23.1.0/truffle-api-23.1.0.jar, /home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-runtime/23.1.0/truffle-runtime-23.1.0.jar, /home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/truffle/truffle-compiler/23.1.0/truffle-compiler-23.1.0.jar, /home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/graalvm/sdk/polyglot-tck/23.1.0/polyglot-tck-23.1.0.jar, /home/pavel/.cache/coursier/v1/https/repo1.maven.org/maven2/org/netbeans/api/org-openide-util-lookup/RELEASE180/org-openide-util-lookup-RELEASE180.jar)
[error] Expected: (13): List(org.graalvm.sdk:nativeimage:23.1.0, org.graalvm.sdk:word:23.1.0, org.graalvm.sdk:jniutils:23.1.0, org.graalvm.sdk:collections:23.1.0, org.graalvm.polyglot:polyglot:23.1.0, org.graalvm.truffle:truffle-api:23.1.0, org.graalvm.truffle:truffle-runtime:23.1.0, org.graalvm.truffle:truffle-compiler:23.1.0, org.graalvm.sdk:polyglot-tck:23.1.0, org.graalvm.truffle:truffle-tck:23.1.0, org.graalvm.truffle:truffle-tck-common:23.1.0, org.graalvm.truffle:truffle-tck-tests:23.1.0, org.netbeans.api:org-openide-util-lookup:RELEASE180)
```
This PR removes this error message by providing appropriate `libraryDependencies` to `runtime-test-instruments`.
Initial implementation of the Arrow language. Closes#7755.
Currently supported logical types are
- Date (days and milliseconds)
- Int (8, 16, 32, 64)
One can currently
- allocate a new fixed-length, nullable Arrow vector - `new[<name-of-the-type>]`
- cast an already existing fixed-length Arrow vector from a memory address - `cast[<name-of-the-type>]`
Closes#7755.
The change adds a convenient trait `ReportLogsOnFailure` that, when merged with the test class, will keep logs in memory and only delegate to the underlying appender on failure. For now we only support forwarding to the console which is sufficient.
A corresponding entry in `application-test.conf` has to point to the new `memory` appender. The additional complexity in the implementation ensures that if someone forgets to mixin `ReportLogsOnFailure` logs appear as before i.e. they respect the log level.
As a bonus fixed arguments passed to ScalaTest in build.sbt so that we are now, again, showing timings of individual tests.
Closes#8603.
# Important Notes
Before:
```
[info] VcsManagerTest:
[info] Initializing project
[ERROR] [2024-01-04 17:27:03,366] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must create a repository (3 seconds, 538 milliseconds)
[info] - must fail to create a repository for an already existing project (141 milliseconds)
[info] Save project
[ERROR] [2024-01-04 17:27:08,346] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must create a commit with a timestamp (198 milliseconds)
[ERROR] [2024-01-04 17:27:08,570] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must create a commit with a name (148 milliseconds)
[ERROR] [2024-01-04 17:27:08,741] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must force all pending saves (149 milliseconds)
[info] Status project
[ERROR] [2024-01-04 17:27:08,910] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must report changed files since last commit (148 milliseconds)
[info] Restore project
[ERROR] [2024-01-04 17:27:09,076] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must reset to the last state with committed changes (236 milliseconds)
[ERROR] [2024-01-04 17:27:09,328] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must reset to a named save (pending)
[ERROR] [2024-01-04 17:27:09,520] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] - must reset to a named save and notify about removed files *** FAILED *** (185 milliseconds)
[info] Right({
[info] "jsonrpc" : "2.0",
[info] "method" : "file/event",
[info] "params" : {
[info] "path" : {
[info] "rootId" : "cd84a4a3-fa50-4ead-8d80-04f6d0d124a3",
[info] "segments" : [
[info] "src",
[info] "Bar.enso"
[info] ]
[info] },
[info] "kind" : "Removed"
[info] }
[info] }) did not equal Right({
[info] "jsonrpc" : "1.0",
[info] "method" : "file/event",
[info] "params" : {
[info] "path" : {
[info] "rootId" : "cd84a4a3-fa50-4ead-8d80-04f6d0d124a3",
[info] "segments" : [
[info] "src",
[info] "Bar.enso"
[info] ]
[info] },
[info] "kind" : "Removed"
[info] }
[info] }) (VcsManagerTest.scala:1343)
[info] Analysis:
[info] Right(value: Json$JObject(value: object[jsonrpc -> "2.0",method -> "file/event",params -> {
[info] "path" : {
[info] "rootId" : "cd84a4a3-fa50-4ead-8d80-04f6d0d124a3",
[info] "segments" : [
[info] "src",
[info] "Bar.enso"
[info] ]
[info] },
[info] "kind" : "Removed"
[info] }] -> object[jsonrpc -> "1.0",method -> "file/event",params -> {
[info] "path" : {
[info] "rootId" : "cd84a4a3-fa50-4ead-8d80-04f6d0d124a3",
[info] "segments" : [
[info] "src",
[info] "Bar.enso"
[info] ]
[info] },
[info] "kind" : "Removed"
[info] }]))
[ERROR] [2024-01-04 17:27:09,734] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/3607843843826594318].
[info] List project saves
[info] - must return all explicit commits (146 milliseconds)
[info] Run completed in 9 seconds, 270 milliseconds.
[info] Total number of tests run: 9
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 8, failed 1, canceled 0, ignored 0, pending 1
[info] *** 1 TEST FAILED ***
```
After:
```
[info] VcsManagerTest:
[info] Initializing project
[info] - must create a repository (3 seconds, 554 milliseconds)
[info] - must fail to create a repository for an already existing project (164 milliseconds)
[info] Save project
[info] - must create a commit with a timestamp (212 milliseconds)
[info] - must create a commit with a name (142 milliseconds)
[info] - must force all pending saves (185 milliseconds)
[info] Status project
[info] - must report changed files since last commit (142 milliseconds)
[info] Restore project
[info] - must reset to the last state with committed changes (202 milliseconds)
[info] - must reset to a named save (pending)
[ERROR] [2024-01-04 17:24:55,738] [org.enso.languageserver.search.SuggestionsHandler] Cannot read the package definition from [/tmp/8456553964637757156].
[info] - must reset to a named save and notify about removed files *** FAILED *** (186 milliseconds)
[info] Right({
[info] "jsonrpc" : "2.0",
[info] "method" : "file/event",
[info] "params" : {
[info] "path" : {
[info] "rootId" : "965ed5c8-1760-4284-91f2-1376406fde0d",
[info] "segments" : [
[info] "src",
[info] "Bar.enso"
[info] ]
[info] },
[info] "kind" : "Removed"
[info] }
[info] }) did not equal Right({
[info] "jsonrpc" : "1.0",
[info] "method" : "file/event",
[info] "params" : {
[info] "path" : {
[info] "rootId" : "965ed5c8-1760-4284-91f2-1376406fde0d",
[info] "segments" : [
[info] "src",
[info] "Bar.enso"
[info] ]
[info] },
[info] "kind" : "Removed"
[info] }
[info] }) (VcsManagerTest.scala:1343)
[info] Analysis:
[info] Right(value: Json$JObject(value: object[jsonrpc -> "2.0",method -> "file/event",params -> {
[info] "path" : {
[info] "rootId" : "965ed5c8-1760-4284-91f2-1376406fde0d",
[info] "segments" : [
[info] "src",
[info] "Bar.enso"
[info] ]
[info] },
[info] "kind" : "Removed"
[info] }] -> object[jsonrpc -> "1.0",method -> "file/event",params -> {
[info] "path" : {
[info] "rootId" : "965ed5c8-1760-4284-91f2-1376406fde0d",
[info] "segments" : [
[info] "src",
[info] "Bar.enso"
[info] ]
[info] },
[info] "kind" : "Removed"
[info] }]))
[info] List project saves
[info] - must return all explicit commits (131 milliseconds)
[info] Run completed in 9 seconds, 400 milliseconds.
[info] Total number of tests run: 9
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 8, failed 1, canceled 0, ignored 0, pending 1
[info] *** 1 TEST FAILED ***
```
I noticed that sources in `runtime/bench` are not formatted at all. Turns out that the `JavaFormatterPlugin` does not override `javafmt` task for the `Benchmark` configuration. After some failed attempts, I have just redefined the `Benchmark/javafmt` task in the `runtime` project. After all, the `runtime` project is almost the only project where we have any Java benchmarks.
# Important Notes
`javafmtAll` now also formats sources in `runtime/bench/src/java`.
After #8467, Engine benchmarks are broken, they cannot compile - https://github.com/enso-org/enso/actions/runs/7268987483/job/19805862815#logs
This PR fixes the benchmark build
# Important Notes
Apart from fixing the build of `engine/bench`:
- Don't assemble any fat jars in `runtime/bench`.
- Use our `TestLogProvider` in the benches instead of NOOP provider.
- So that we can at least see warnings and errors in benchmarks.
Make sure that the correct test logging provider is loaded in `project-manager/Test`, so that only WARN and ERROR log messages are displayed. Also, make sure that the test log provider parses the correct configuration file - Rename all the `application.conf` files in the test resources to `application-test.conf`.
The problem was introduced in #8467
- After [suggestion](https://github.com/enso-org/enso/pull/8497#discussion_r1429543815) from @JaroslavTulach I have tried reimplementing the URL encoding using just `URLEncode` builtin util. I will see if this does not complicate other followup improvements, but most likely all should work so we should be able to get rid of the unnecessary bloat.
- Closes#8354
- Extends `simple-httpbin` with a simple mock of the Cloud API (currently it checks the token and serves the `/users` endpoint).
- Renames `simple-httpbin` to `http-test-helper`.
