Adds these JAR modules to the `component` directory inside Engine distribution:
- `graal-language-23.1.0`
- `org.bouncycastle.*` - these need to be added for graalpy language
# Important Notes
- Remove `org.bouncycastle.*` packages from `runtime.jar` fat jar.
- Make sure that the `./run` script preinstalls GraalPy standalone distribution before starting engine tests
- Note that using `python -m venv` is only possible from standalone distribution, we cannot distribute `graalpython-launcher`.
- Make sure that installation of `numpy` and its polyglot execution example works.
- Convert `Text` to `TruffleString` before passing to GraalPy - 8ee9a2816f
Upgrade to GraalVM JDK 21.
```
> java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing)
```
With SDKMan, download with `sdk install java 21-graalce`.
# Important Notes
- After this PR, one can theoretically run enso with any JRE with version at least 21.
- Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution.
- `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded.
- sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368
- Which means that we can have `module-info.java` files only for Java-only projects.
- Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar.
- `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent.
- Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory.
- The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module.
- The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725)
- [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
The change upgrades `directory-watcher` library, hoping that it will fix the problem reported in #7695 (there has been a number of bug fixes in MacOS listener since then).
Once upgraded, tests in `WatcherAdapterSpec` because the logic that attempted to ensure the proper initialization order in the test using semaphore was wrong. Now starting the watcher using `watchAsync` which only returns the future when the watcher successfully registers for paths. Ideally authors of the library would make the registration bit public
(3218d68a84/core/src/main/java/io/methvin/watcher/DirectoryWatcher.java (L229C7-L229C20)) but it is the best we can do so far.
Had to adapt to the new API in PathWatcher as well, ensuring the right order of initialization.
Should fix#7695.
Having a modest-size files in a project would lead to a timeout when the project was first initialized. This became apparent when testing delivered `.enso-project` files with some data files. After some digging there was a bug in JGit
(https://bugs.eclipse.org/bugs/show_bug.cgi?id=494323) which meant that adding such files was really slow. The implemented fix is not on by default but even with `--renormalization` turned off I did not see improvement.
In the end it didn't make sense to add `data` directory to our version control, or any other files than those in `src` or some meta files in `.enso`. Not including such files eliminates first-use initialization problems.
# Important Notes
To test, pick an existing Enso project with some data files in it (> 100MB) and remove `.enso/.vcs` directory. Previously it would timeout on first try (and work in successive runs). Now it works even on the first try.
The crash:
```
[org.enso.languageserver.requesthandler.vcs.InitVcsHandler] Initialize project request [Number(2)] for [f9a7cd0d-529c-4e1d-a4fa-9dfe2ed79008] failed with: null.
java.util.concurrent.TimeoutException: null
at org.enso.languageserver.effect.ZioExec$.<clinit>(Exec.scala:134)
at org.enso.languageserver.effect.ZioExec.$anonfun$exec$3(Exec.scala:60)
at org.enso.languageserver.effect.ZioExec.$anonfun$exec$3$adapted(Exec.scala:60)
at zio.ZIO.$anonfun$foldCause$4(ZIO.scala:683)
at zio.internal.FiberRuntime.runLoop(FiberRuntime.scala:904)
at zio.internal.FiberRuntime.evaluateEffect(FiberRuntime.scala:381)
at zio.internal.FiberRuntime.evaluateMessageWhileSuspended(FiberRuntime.scala:504)
at zio.internal.FiberRuntime.drainQueueOnCurrentThread(FiberRuntime.scala:220)
at zio.internal.FiberRuntime.run(FiberRuntime.scala:139)
at akka.dispatch.TaskInvocation.run(AbstractDispatcher.scala:49)
at java.base/java.util.concurrent.ForkJoinTask$RunnableExecuteAction.exec(ForkJoinTask.java:1395)
at java.base/java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:373)
at java.base/java.util.concurrent.ForkJoinPool$WorkQueue.topLevelExec(ForkJoinPool.java:1182)
```
* Reduce extra output in compilation and tests
I couldn't stand the amount of extra output that we got when compiling
a clean project and when executing regular tests. We should strive to
keep output clean and not print anything additional to stdout/stderr.
* Getting rid of explicit setup by service loading
In order for SL4J to use service loading correctly had to upgrade to
latest slf4j. Unfortunately `TestLogProvider` which essentially
delegates to `logback` provider will lead to spurious ambiguous warnings
on multiple providers. In order to dictate which one to use and
therefore eliminate the warnings we can use the `slf4j.provider` env
var, which is only available in slf4j 2.x.
Now, there is no need to explicitly call `LoggerSetup.get().setup()` as
that is being called during service setup.
* legal review
* linter
* Ensure ConsoleHandler uses the default level
ConsoleHandler's constructor uses `Level.INFO` which is unnecessary for
tests.
* report warnings
* Enable log-to-file configuration
PR #7825 enabled parallel logging to a file with a much more
fine-grained log level by default.
However, logging at `TRACE` level on Windows appears to be still
problematic.
This PR reduced the default log level to file from `DEBUG` to `TRACE`
and allows to control it via an environment variable if one wishes to
change the verbosity without making code changes.
* PR comments
* Always log verbose to a file
The change adds an option by default to always log to a file with
verbose log level.
The implementation is a bit tricky because in the most common use-case
we have to always log in verbose mode to a socket and only later apply
the desired log levels. Previously socket appender would respect the
desired log level already before forwarding the log.
If by default we log to a file, verbose mode is simply ignored and does
not override user settings.
To test run `project-manager` with `ENSO_LOGSERVER_APPENDER=console` env
variable. That will output to the console with the default `INFO` level
and `TRACE` log level for the file.
* add docs
* changelog
* Address some PR requests
1. Log INFO level to CONSOLE by default
2. Change runner's default log level from ERROR to WARN
Took a while to figure out why the correct log level wasn't being passed
to the language server, therefore ignoring the (desired) verbose logs
from the log file.
* linter
* 3rd party uses log4j for logging
Getting rid of the warning by adding a log4j over slf4j bridge:
```
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
```
* legal review update
* Make sure tests use test resources
Having `application.conf` in `src/main/resources` and `test/resources`
does not guarantee that in Tests we will pick up the latter. Instead, by
default it seems to do some kind of merge of different configurations,
which is far from desired.
* Ensure native launcher test log to console only
Logging to console and (temporary) files is problematic for Windows.
The CI also revealed a problem with the native configuration because it
was not possible to modify the launcher via env variables as everything
was initialized during build time.
* Adapt to method changes
* Potentially deal with Windows failures
# Important Notes
- Binary LS endpoint is not yet handled.
- The parsing of provided source is not entirely correct, as each line (including imports) is treated as node. The usage of actual enso AST for nodes is not yet implemented.
- Modifications to the graph state are not yet synchronized back to the language server.
This change replaces Enso's custom logger with an existing, mostly off the shelf logging implementation. The change attempts to provide a 1:1 replacement for the existing solution while requiring only a minimal logic for the initialization.
Loggers are configured completely via `logging-server` section in `application.conf` HOCON file, all initial logback configuration has been removed. This opens up a lot of interesting opportunities because we can benefit from all the well maintained slf4j implementations without being to them in terms of functionality.
Most important differences have been outlined in `docs/infrastructure/logging.md`.
# Important Notes
Addresses:
- #7253
- #6739
New rules to recognize `"""` and `'''` at the end of line and color everything that's inside nested block of text as string. It properly _stops the string literal when nested block ends_.
# Important Notes
The [text mate grammar rules](https://macromates.com/manual/en/language_grammars) properly _end_ the text block when the block ends. For example in:
```ruby
main =
x = """
a text
y
```
`a text` is properly recognized as string.
The added benchmark is a basis for a performance investigation.
We compare the performance of the same operation run in Java vs Enso to see what is the overhead and try to get the Enso operations closer to the pure-Java performance.
- Previous GraalVM update: https://github.com/enso-org/enso/pull/6750
Removed warnings:
- Remove deprecated `ConditionProfile.createCountingProfile()`.
- Add `@Shared` to some `@Cached` parameters (Truffle now emits warnings about potential `@Share` usage).
- Specialization method names should not start with execute
- Add limit attribute to some specialization methods
- Add `@NeverDefault` for some cached initializer expressions
- Add `@Idempotent` or `@NonIdempotent` where appropriate
BigInteger and potential Node inlining are tracked in follow-up issues.
# Important Notes
For `SDKMan` users:
```
sdk install java 17.0.7-graalce
sdk use java 17.0.7-graalce
```
For other users - download link can be found at https://github.com/graalvm/graalvm-ce-builds/releases/tag/jdk-17.0.7
Release notes: https://www.graalvm.org/release-notes/JDK_17/
R component was dropped from the release 23.0.0, only `python` is available to install via `gu install python`.
Engine benchmark downloader tool [bench_download.py](https://github.com/enso-org/enso/blob/develop/tools/performance/engine-benchmarks/bench_download.py) can now plot multiple branches in the same charts. The tooltips on the charts, displayed when you hover over some data point, are now broken for some branches. I have no idea why, it might be a technical limitation of the Google charts library. Nevertheless, I have also extended the *selection info* section, that is displayed under every chart, where one can see all the important information, once you click on some data point.
Options added:
- `--branches` specifies list of branches for which all the benchmark data points will be in the plots. The default is `develop` only.
- `--labels` that can limit the number of generated charts.
This PR also **deprecates** the `--compare` option. There is no reason to keep that option around since we can now plot all the branches in the same charts.
An example for plotting benchmarks for PR #7009 with
```
python bench_download.py -v --since 2023-07-01 --until 2023-07-11 --branches develop wip/jtulach/ArgumentConversion
```
is:
# Important Notes
- Deprecate `--compare` option
- Add `--labels` option
- Add `--branches` option
- Add type detection for `Mixed` columns when calling column functions.
- Excel uses column name for missing headers.
- Add aliases for parse functions on text.
- Adjust `Date`, `Time_Of_Day` and `Date_Time` parse functions to not take `Nothing` anymore and provide dropdowns.
- Removed built-in parses.
- All support Locale.
- Add support for missing day or year for parsing a Date.
- All will trim values automatically.
- Added ability to list AWS profiles.
- Added ability to list S3 buckets.
- Workaround for Table.aggregate so default item added works.
Add diagnosis for unresolved symbols in `from ... import sym1, sym2, ...` statements.
- Adds a new compiler pass, `ImportSymbolAnalysis`, that checks these statements and iterates through the symbols and checks if all the symbols can be resolved.
- Works with `BindingsMap` metadata.
- Add `ImportExportTest` that creates various modules with various imports/exports and checks their generated `BindingMap`.
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
`Number.nan` can be used as a key in `Map`. This PR basically implements the support for [JavaScript's Same Value Zero Equality](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Equality_comparisons_and_sameness#same-value-zero_equality) so that `Number.nan` can be used as a key in `Map`.
# Important Notes
- For NaN, it holds that `Meta.is_same_object Number.nan Number.nan`, and `Number.nan != Number.nan` - inspired by JS spec.
- `Meta.is_same_object x y` implies `Any.== x y`, except for `Number.nan`.
* Update type ascriptions in some operators in Any
* Add @GenerateUncached to AnyToTextNode.
Will be used in another node with @GenerateUncached.
* Add tests for "sort handles incomparable types"
* Vector.sort handles incomparable types
* Implement sort handling for different comparators
* Comparison operators in Any do not throw Type_Error
* Fix some issues in Ordering_Spec
* Remove the remaining comparison operator overrides for numbers.
* Consolidate all sorting functionality into a single builtin node.
* Fix warnings attachment in sort
* PrimitiveValuesComparator handles other types than primitives
* Fix byFunc calling
* on function can be called from the builtin
* Fix build of native image
* Update changelog
* Add VectorSortTest
* Builtin method should not throw DataflowError.
If yes, the message is discarded (a bug?)
* TypeOfNode may not return only Type
* UnresolvedSymbol is not supported as `on` argument to Vector.sort_builtin
* Fix docs
* Fix bigint spec in LessThanNode
* Small fixes
* Small fixes
* Nothings and Nans are sorted at the end of default comparator group.
But not at the whole end of the resulting vector.
* Fix checking of `by` parameter - now accepts functions with default arguments.
* Fix changelog formatting
* Fix imports in DebuggingEnsoTest
* Remove Array.sort_builtin
* Add comparison operators to micro-distribution
* Remove Array.sort_builtin
* Replace Incomparable_Values by Type_Error in some tests
* Add on_incomparable argument to Vector.sort_builtin
* Fix after merge - Array.sort delegates to Vector.sort
* Add more tests for problem_behavior on Vector.sort
* SortVectorNode throws only Incomparable_Values.
* Delete Collections helper class
* Add test for expected failure for custom incomparable values
* Cosmetics.
* Fix test expecting different comparators warning
* isNothing is checked via interop
* Remove TruffleLogger from SortVectorNode
* Small review refactorings
* Revert "Remove the remaining comparison operator overrides for numbers."
This reverts commit 0df66b1080.
* Improve bench_download.py tool's `--compare` functionality.
- Output table is sorted by benchmark labels.
- Do not fail when there are different benchmark labels in both runs.
* Wrap potential interop values with `HostValueToEnsoNode`
* Use alter function in Vector_Spec
* Update docs
* Invalid comparison throws Incomparable_Values rather than Type_Error
* Number comparison builtin methods return Nothing in case of incomparables
close#6080
Changelog
- add: implement `SuggestionsRepo.insertAll` as a batch SQL insert
- update: `search/getSuggestionsDatabase` returns empty suggestions. Currently, the method is only used at startup and returns the empty response anyway because the libs are not loaded at that point.
- update: serialize only global (defined in the module scope) suggestions during the distribution building. There's no sense in storing the local library suggestions.
- update: sqlite dependency
- remove: unused methods from `SuggestionsRepo`
- remove: Arguments table
# Important Notes
Speeds up libraries loading by ~1 second.
Add `--compare <bench-run-id-1> <bench-run-id-2>` option to `bench_download.py` script, that prints the difference of benchmark scores for two benchmark runs from GH as a table.
# Important Notes
I tried to add an option to compare two or more branches and visualize them in the graphs, but gave that up after some struggling.
Implement new Enso documentation parser; remove old Scala Enso parser.
Performance: Total time parsing documentation is now ~2ms.
# Important Notes
- Doc parsing is now done only in the frontend.
- Some engine tests had never been switched to the new parser. We should investigate tests that don't pass after the switch: #5894.
- The option to run the old searcher has been removed, as it is obsolete and was already broken before this (see #5909).
- Some interfaces used only by the old searcher have been removed.
Fixing node.js dependencies as:
```bash
enso/tools/enso4igv$ mvn clean install -Pvsix
```
was broken. Making sure the VSIX build is part of the _actions workflow_.
# Important Notes
To reproduce/verify remove `enso/tools/enso4igv/node_modules` first and try to build.
Automating the assembly of the engine and its execution into a single task. If you are modifying standard libraries, engine sources or Enso tests, you can launch `sbt` and then just:
```
sbt:enso> runEngineDistribution --run test/Tests/src/Data/Maybe_Spec.enso
[info] Engine package created at built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev
[info] Executing built-distribution/enso-engine-...-dev/bin/enso --run test/Tests/src/Data/Maybe_Spec.enso
Maybe: [5/5, 30ms]
- should have a None variant [14ms]
- should have a Some variant [5ms]
- should provide the `maybe` function [4ms]
- should provide `is_some` [2ms]
- should provide `is_none` [3ms]
5 tests succeeded.
0 tests failed.unEngineDistribution 4s
0 tests skipped.
```
the [runEngineDistribution](3a581f29ee/docs/CONTRIBUTING.md (running-enso)) `sbt` input task makes sure all your sources are properly compiled and only then executes your enso source. Everything ready at a single press of Enter.
# Important Notes
To debug in chrome dev tools, just add `--inspect`:
```
sbt:enso> runEngineDistribution --inspect --run test/Tests/src/Data/Maybe_Spec.enso
E.g. in Chrome open: devtools://devtools/bundled/js_app.html?ws=127.0.0.1:9229/7JsgjXlntK8
```
everything gets build and one can just attach the Enso debugger.
