Encountered a random NPE when playing with bookclubs. Test case demonstrating the problem is attached.
Threw in a bunch of minor tweaks to logs to make life of the person debugging code more pleasant.
Upgrade to GraalVM JDK 21.
```
> java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing)
```
With SDKMan, download with `sdk install java 21-graalce`.
# Important Notes
- After this PR, one can theoretically run enso with any JRE with version at least 21.
- Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution.
- `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded.
- sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368
- Which means that we can have `module-info.java` files only for Java-only projects.
- Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar.
- `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent.
- Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory.
- The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module.
- The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725)
- [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
@radeusgd pointed out that tests are checking that the engine does not send updates when the dataflow error changes. In the end, it turned out that those tests were not checking what they said, and the engine sent the proper updates.
* OpenFileCmd sends a reply when finished
Lack of reply and therefore a non-determinism on when OpenFile handler
can finish, led to some sporadic instability. Once caches format got
changed, things were taking such a long time that I wasn't able to start
even a basic project (requests would start to timeout).
The change also removes PushContextCmd from synchronous cmds (as
introduced in #798 to remove initialization problems in tests as well as
in real scenarions); the change did the job but was also a bit
controversial.
The change can also help with randomly failing applies (#8174) as IDE kept
closing and opening the project that might have exploited the
race-condition.
* Adapt tests
* Make tests more resilient to out of order messages
* Drop retries that lead to confusing errors
* less random failures
* s/OpenFileNotification/OpenFileRequest
Fixes#8186 by turning `IllegalStateException` into log message. Re-assigning of `BindingsMap` can happen in the IDE where evaluation of modules is repeated again and again. In addition to that avoid dropping errors in compiler without them being noticed.
Using a `TruffleLogger` in `SerializationManager` that is bound to the engine rather than the context prevents reaching an illegal state when using thread pools.
Also cleaned up some tests for consistency.
To verify the fix
```diff
--- a/engine/runtime/src/main/scala/org/enso/compiler/SerializationManager.scala
+++ b/engine/runtime/src/main/scala/org/enso/compiler/SerializationManager.scala
@@ -31,7 +31,7 @@ final class SerializationManager(compiler: Compiler) {
import SerializationManager._
/** The debug logging level. */
- private val debugLogLevel = Level.FINE
+ private val debugLogLevel = Level.INFO
```
and run
`sbt:enso> runtime/test`
Closes#8147.
Towards reduced reliance on Scala semantics.
Translated IR.scala to IR.java and extracted implicits that now need to be imported explicitly.
# Important Notes
1:1 translation. For now `@Identifier` and `@ExternalID` represent the old type aliases but are not verified at compile time.
This is because in a mixed Scala/Java world this seems impossible to employ such frameworks as Checker.
- Fixes#8049
- Adds tests for handling of Date_Time upload/download in Postgres.
- Adds tests for edge cases of handling of Decimal and Binary types in Postgres.
close#8033
Changelog:
- update: run language server initialization once
- fix: issues with async `getSuggestionDatabase` message handling in new IDE
- update: implement unique background jobs
- refactor: initialization logic to Java
- refactor: `UniqueJob` to a marker interface
Improved warning reporting by not reporting IR in user-directed warning
messages. Made sure that fresh names retain some knowledge of the name
which they were created from.
Closes#7963.
Unmasked position info for the failed edits so that it is possible to understand what edits are attempted at all. So far unable to locate the problem based solely on the logs and the stacktrace. Added a test confirming that making edits at the end of the file continues to be fine (that's where the problem usually manifests itself).
# Important Notes
References #7978
- Validate spans during existing lexer and parser unit tests, and in `enso_parser_debug`.
- Fix lost span info causing failures of updated tests.
# Important Notes
- [x] Output of `parse_all_enso_files.sh` is unchanged since before #7881 (modulo libs changes since then).
- When the parser encounters an input with the first line indented, it now creates a sub-block for lines at than indent level, and emits a syntax error (every indented block must have a parent).
- When the parser encounters a number with a base but no digits (e.g. `0x`), it now emits a `Number` with `None` in the digits field rather than a 0-length digits token.
`AmbgiuousImport` error and `DuplicatedImport` warning must not have `Source` as one of its fields or compiler will crash during a serialization attempt.
Changed the implementation to provide it as a parameter to the `message` method instead.
Fixes#8089
Prevents [this NullPointerException](https://github.com/enso-org/enso/pull/8044#issuecomment-1763736570) by making sure `ConstantNode` rejects `null` during construction.
# Important Notes
```ruby
from Standard.Base import all
polyglot java import java.util.Random
main =
operator1 = Random.new
```
* Reduce extra output in compilation and tests
I couldn't stand the amount of extra output that we got when compiling
a clean project and when executing regular tests. We should strive to
keep output clean and not print anything additional to stdout/stderr.
* Getting rid of explicit setup by service loading
In order for SL4J to use service loading correctly had to upgrade to
latest slf4j. Unfortunately `TestLogProvider` which essentially
delegates to `logback` provider will lead to spurious ambiguous warnings
on multiple providers. In order to dictate which one to use and
therefore eliminate the warnings we can use the `slf4j.provider` env
var, which is only available in slf4j 2.x.
Now, there is no need to explicitly call `LoggerSetup.get().setup()` as
that is being called during service setup.
* legal review
* linter
* Ensure ConsoleHandler uses the default level
ConsoleHandler's constructor uses `Level.INFO` which is unnecessary for
tests.
* report warnings
It looks like visualization commands had required context lock unnecessairly. Context manager methods are synchronized and therefore should not need the lock before submitting a correspodning background job.
Additionally, the presence of a context lock leads a deadlock:
1. Consider a long running execution of a program that does not finish within the 5 seconds
2. In the meantime there comes either an `AttachVisualization` or `DetachVisualization` request from the client
The latter will get stuck and eventually timeout out because it cannot acquire the lock withing the required time limits. It is still possible that a single long-running `ExecuteJob` might block other ones (including visualization ones) but that's a separate work.
Fixes some issues present in #7941.
# Important Notes
We need to still investigate `ExecuteJob`s need for context lock which might delay the execution of other background jobs that require it as well (mostly concerned about visualization).
* Q: Is it normal for `--inspect` mode to print two debug urls?
* A: No, it should print just one.
* Q: Putting there a Java breakpoint to find out why it the chromeinspector gets initialized twice might reveal the culprit.
* A: The additional listener is happening [here](https://github.com/enso-org/enso/blob/develop/engine/runner/src/main/scala/org/enso/runner/ContextFactory.scala#L117).
# Important Notes
There is no easy check for a language being present without creating an `Engine`. It was thought creating an `Engine` is cheap operation, but it seems to have some downsides. Let's use `ENSO_JAVA` environment variable to decide whether _experimental Espresso_ support shall be enabled.
Adds the ability to declare a module as *private*. Modifies the parser to add the `private` keyword as a reserved keyword. All the checks for private modules are implemented as an independent *Compiler pass*. No checks are done at runtime.
# Important Notes
- Introduces new keyword - `private` - a reserved keyword.
- Modules that have `private` keyword as the first statement are declared as *private* (Project private)
- Public module cannot have private submodules and vice versa.
- This would require runtime access checks
- See #7088 for the specification.
* Always log verbose to a file
The change adds an option by default to always log to a file with
verbose log level.
The implementation is a bit tricky because in the most common use-case
we have to always log in verbose mode to a socket and only later apply
the desired log levels. Previously socket appender would respect the
desired log level already before forwarding the log.
If by default we log to a file, verbose mode is simply ignored and does
not override user settings.
To test run `project-manager` with `ENSO_LOGSERVER_APPENDER=console` env
variable. That will output to the console with the default `INFO` level
and `TRACE` log level for the file.
* add docs
* changelog
* Address some PR requests
1. Log INFO level to CONSOLE by default
2. Change runner's default log level from ERROR to WARN
Took a while to figure out why the correct log level wasn't being passed
to the language server, therefore ignoring the (desired) verbose logs
from the log file.
* linter
* 3rd party uses log4j for logging
Getting rid of the warning by adding a log4j over slf4j bridge:
```
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
```
* legal review update
* Make sure tests use test resources
Having `application.conf` in `src/main/resources` and `test/resources`
does not guarantee that in Tests we will pick up the latter. Instead, by
default it seems to do some kind of merge of different configurations,
which is far from desired.
* Ensure native launcher test log to console only
Logging to console and (temporary) files is problematic for Windows.
The CI also revealed a problem with the native configuration because it
was not possible to modify the launcher via env variables as everything
was initialized during build time.
* Adapt to method changes
* Potentially deal with Windows failures
- Closes#7461 by introducing a `Date_Time_Formatter` type and making parsing date time formats more robust and safer.
- The default ('simple') set of patterns is slightly simplified and made case insensitive (except for `M/m` and `H/h`) to avoid the `YYYY` vs `yyyy` issues and make it less error prone.
- The `YYYY` now has the same meaning as `yyyy` in simple mode. The old meaning (week-based year) is moved to a _separate mode_, triggered by `Date_Time_Formatter.from_iso_week_date_pattern`.
- Full Java syntax, as well as custom-built Java `DateTimeFormatter` can also be used by `Date_Time_Formatter.from_java`.
- Text-based constants (e.g. `ISO_ZONED_DATE_TIME`) have now become methods on `Date_Time_Formatter`, e.g. `Date_Time_Formatter.iso_zoned_date_time`).
close#7320
Changelog:
- update: enable conversion suggestions
- fix: conversion suggestion building
- fix: conversion suggestion types
- fix: conversion JSON-RPC representation
# Important Notes
For example, the [`Day_Of_Week_From`](5150c14afd/distribution/lib/Standard/Base/0.0.0-dev/src/Data/Time/Day_Of_Week_From.enso) conversion is sent as
```json
{
"type":"Add",
"id":32,
"suggestion":{
"type":"method",
"module":"Standard.Base.Data.Time.Day_Of_Week_From",
"name":"from",
"arguments":[
{
"name":"that",
"reprType":"Standard.Base.Data.Numbers.Integer",
"isSuspended":false,
"hasDefault":false,
"defaultValue":null,
"tagValues":null
},
{
"name":"first_day",
"reprType":"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week",
"isSuspended":false,
"hasDefault":true,
"defaultValue":"Day_Of_Week.Sunday",
"tagValues":[
"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week.Sunday",
"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week.Monday",
"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week.Tuesday",
"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week.Wednesday",
"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week.Thursday",
"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week.Friday",
"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week.Saturday"
]
},
{
"name":"start_at_zero",
"reprType":"Standard.Base.Data.Boolean.Boolean",
"isSuspended":false,
"hasDefault":true,
"defaultValue":"False",
"tagValues":[
"Standard.Base.Data.Boolean.Boolean.True",
"Standard.Base.Data.Boolean.Boolean.False"
]
}
],
"selfType":"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week",
"returnType":"Standard.Base.Data.Time.Day_Of_Week.Day_Of_Week",
"isStatic":false,
"documentation":" Convert from an integer to a Day_Of_Week\n\nArguments:\n- `that`: The first day of the week.\n- `first_day`: The first day of the week.\n- `start_at_zero`: If True, first day of the week is 0 otherwise is 1.",
"annotations":[
]
}
}
```
- Added a `FileSystemSPI` allowing protocol resolution to a target type.
- Separated `Input_Stream` and `Output_Stream` from `File` to allow use in other spaces.
- `File_Format` types `read_web` changed to be `read_stream` working with `InputStream`.
- Added directory listing to `Auto_Detect` allowing for `Data.read` to list a folder.
- Adjusted HTTP to return an `InputStream` not a `byte[]`:
- `Response_Body` adjusted to wrap an `InputStream`.
- Added ability to materialize to either and in-memory vector (<4KB) or a temporary file.
- `Data.fetch` will materialize if not a recognized mime-type.
- Added `HTTP_Error` to handle IO exceptions from the stream.
- `Excel_Format` now supports mime-type and reading a stream.
- `Excel_Workbook` can now get a `Excel_Section` using `read_section`.
- Added S3 APIs:
- `parse_uri`: splits an S3 URI into bucket and key.
- `list_objects`: list the items in a S3 bucket with specified prefix.
- `read_bucket`: list prefixes and keys with a delimiter in a S3 bucket with specified prefix.
- `head`: either head_bucket (tests existance) or head_object API (reads object meta data).
- `get_object`: gets an object from S3 returning as a `Response_Body`.
- Added `S3_File` type acting like a `File`:
- No support for writing in this PR.
- **ToDo:** recursive listing, glob filtering, exists, size.
- Fixed a few invalid type signature line.
- Moved `create` methods for `Postgres_Connection` and `SQLite_Connection` into type instead of module.
- Renamed `Column_Fetcher.Builder` to `Column_Fetcher_Builder`.
- Fixed bug with `select_into` in Dry Run mode creating permanent tables.
**ToDo:** Unit tests.
Closes#7677 by eliminating the _stackoverflow execption_. In general it seems _too adventurous_ to walk members of random foreign objects. There can be anything including cycles. Rather than trying to be too smart in these cases, let's just rely on `InteropLibrary.isIdentical` message.
# Important Notes
Calling `sort` on the `numpy` array no longer yields an error, but the array isn't sorted - that needs a fix on the Python side: https://github.com/oracle/graalpython/issues/354 - once it is in, the elements will be treated as numbers and the sorting happens automatically (without any changes in Enso code).
close#7520
Changelog:
- update: SectionsToBinOp compiler pass produces function application for left sections
- refactor: simplify the registration of builtin methods
This PR addresses two problems mentioned in #7766:
1. A random integer overflow, likely caused by a bug in Rust parser
2. A concurrent access to a methods' map
Re 1: Unable to reproduce but it doesn't mean it won't happen again. Added a try/catch to get in the logs source code that caused it **and** not crash hard when it occurs.
Re 2: Changing methods map from `HashMap` to `ConcurrentHashMap`. Due to a poor design we leaked the underlying structure in a number of places, unnecessairly. `ConcurrentHashMap` does not accept `null` keys therefore due to leaking implementation had to ensure that `methods` of `ModuleScope` never escapes as-is.
Both workarounds should ensure that we don't crash hard when they appear.
Closes#7766
- Closes#7238
- Aligns `update_database_table` to a more consistent and clearer API - `update_rows`.
- Adds a `truncate_table` helper function, to pair up with `drop_table`. Both are `PRIVATE` for now.
- Adds tests for NULLs in keys in `update_rows` and `delete_rows`.
- The behaviour is sometimes unexpected, so instead these fail with `Null_Values_In_Key_Columns`.
- Adds a workaround for https://github.com/oracle/graal/issues/7359
- Adds a workaround for a related bug where a stack frame has no name (its `rootNode.getName() == null`).
- I could not track down this bug to provide a neat repro.
This change replaces Enso's custom logger with an existing, mostly off the shelf logging implementation. The change attempts to provide a 1:1 replacement for the existing solution while requiring only a minimal logic for the initialization.
Loggers are configured completely via `logging-server` section in `application.conf` HOCON file, all initial logback configuration has been removed. This opens up a lot of interesting opportunities because we can benefit from all the well maintained slf4j implementations without being to them in terms of functionality.
Most important differences have been outlined in `docs/infrastructure/logging.md`.
# Important Notes
Addresses:
- #7253
- #6739
- Closes#7633
- Moves `Round_Spec.enso` from published `Standard.Test` into our `test/Tests` project; the `Table_Tests` that depend on it, simply `import enso_dev.Tests`.
- Changes the layout of the local libraries directory:
- It used to be `root/<namespace>/<name>`.
- Now it is `root/<dir>` - the namespace and name are now read from `package.yaml` instead.
- Adds the parent directory of the current project to the default `ENSO_LIBRARY_PATH`.
- It is treated as a secondary path, so the default `ENSO_HOME/lib` still takes precedence.
- This allows projects to reference and load 'sibling' projects easily - the only requirement is for the project to enable `prefer-local-libraries: true` or add the other local project to its edition. The edition resolution logic is **not changed**.
Refactoring deeply nested IR classes to shallow nesting.
Fixes#7017
# Important Notes
It's a big PR but fairly consistent. Split multiple hierarchy levels into subpackages.
close#7604
After moving the rename action to the dashboard, IDE is unaware of the new project name. PR implements a new `refactoring/projectRenamed` notification that is sent from the server to clients and informs them about the changed project name.
# Important Notes
https://github.com/enso-org/enso/assets/357683/7c62726d-217e-4e69-8e48-568e0b7b8c34
While investigating behavior of
```
sbt:std-benchmarks> withDebug benchOnly --dumpGraphs -- Vector_Operations.Max_Stat
```
in IGV I realized there is a deep chain of nodes when reading an element of `Vector` related to work with warnings. There is an invocation of `WarningsLibrary` on `this` - that's probably unnecessary as we know how it is going to resolve. This PR skips such one level of indirection by directly delegating to `this.storage`.
However I haven't seen any effect of this change on peak performance. The library overhead seems to disappear. Anyway I wanted to bring this finding to your attention and perform independent measurement on our CI server.
Fixes#7213 by reacting to new `isBigInteger` and `asBigInteger` messages. Adjusts tests to require `BigInteger` when appropriate. `EnsoBigInteger` no longer extends `Number`. Enso tests to come.
- Closes#5951
- Ensures any SQL warnings reported by the database through the JDBC driver are processed and forwarded to the user.
- These warnings show issues like the implicit name truncation that this PR is also solving. It's good to make sure they are visible as they can help avoid and understand unexpected problems. They should not show up in most standard workflows.
- Adds simple history to our REPL.
Fixes#7279 by detecting missing `getClose()` and yielding an `UnclosedTextLiteral`.
# Important Notes
Special care must be taken for _text blocks_. They have `null` `getClose()`.
- Previous GraalVM update: https://github.com/enso-org/enso/pull/6750
Removed warnings:
- Remove deprecated `ConditionProfile.createCountingProfile()`.
- Add `@Shared` to some `@Cached` parameters (Truffle now emits warnings about potential `@Share` usage).
- Specialization method names should not start with execute
- Add limit attribute to some specialization methods
- Add `@NeverDefault` for some cached initializer expressions
- Add `@Idempotent` or `@NonIdempotent` where appropriate
BigInteger and potential Node inlining are tracked in follow-up issues.
# Important Notes
For `SDKMan` users:
```
sdk install java 17.0.7-graalce
sdk use java 17.0.7-graalce
```
For other users - download link can be found at https://github.com/graalvm/graalvm-ce-builds/releases/tag/jdk-17.0.7
Release notes: https://www.graalvm.org/release-notes/JDK_17/
R component was dropped from the release 23.0.0, only `python` is available to install via `gu install python`.
This PR does three related things:
- Fails more gracefully when a non-string is passed to compile_regex
- Don't pass a non-string to compile_regex
- Allow a Regex param to parse_to_table
This PR modifies the builtin method processor such that it forbids arrays of non-primitive and non-guest objects in builtin methods. And provides a proper implementation for the builtin methods in `EnsoFile`.
- Remove last `to_array` calls from `File.enso`
Less dependencies on `EnsoContext` - `Module` shall exist without it. `Module` is a result of a `Compiler` and shall be created before its execution - e.g. requiring `EnsoContext` (with all its runtime information) is a bit too _demanding_.
# Important Notes
The only reason why `Module` wanted `EnsoContext` was to create its (associated) `Type`. `Type`'s constructor needed a parent type and the code was asking for `Any` from the context. That's unnecessary at creation time - we can just use some constant (like `null`) and turning it into `Any` during execution. Benchmarks show that there is no slowdown doing so.
On a quest to avoid dependencies on `EnsoContext` from `Compiler`. Step one. The ultimate goal is to move `Compiler` and all its `IRPasses` into a dedicated `runtime/compiler` module that could be used from #7054.
- Removed Array methods: `new`, `copy` and `new_[1234]`.
- New builtins for `Vector.insert`, `Vector.remove` and `Vector.flatten`.
- Replaced `Vector_Builder` use of `Array.copy` to a `Vector.Builder` approach.
Package's config information, once loaded, never changed. While there is typically no need for it, this was problematic when the config became out-of-sync with the filesystem, like in the case of project rename action.
In rename, the config's properties would be updated in the FS, but that would never be reflected in module's package. Therefore further compilations would continue to ask for the old namespace.
Most of the changes are cosmetic (s/`.config`/`.getConfig()`) except for the new `reloadConfig` method on `Package` that is being called in `RenameProjectCmd` handler.
Closes#7062.
# Important Notes
The reported `ExecutionFailed` error should have been mostly fixed already via #7143. This change makes sure that all the related warnings are gone as well and the compiler uses the updated namespace.
Mostly stuff to tidy up the static methods in the CB.
- Remove default pattern from `parse_to_table` (caused IDE to freeze).
- Rename any `_` arguments to what they are.
- Merge `Date.now` into `Date.today`
- Merge the Interval constructors into a single constructor.
- Hide various methods.
Fixes#6955 by:
- using `visualisationModule` to specify the module where the visualization is to be used
- referring to method in `Meta.get_annotation` with `.method_name` - e.g. unresolved symbol notation
- evaluating arguments to `Meta.get_annotation` in the context of the user module (which can access the extension functions)
Partially revert https://github.com/enso-org/enso/pull/6849, which introduced a regression in TCO in the presence of warnings. Rather than modifying the tail call status, `TailCallException` now propagates the extracted warnings and appends them to the final result.
Closes#7093
# Important Notes
Compared to the previous attempt we don't pay the penalty of adding the warnings or even checking for them because it is being dealt in a separate specialization.
Function bodies cannot be instrumented even if the function is right
inside a binding. Consider a scenario when a function is assigned to a
variable and then applied to a `map` method of a really large vector.
The instrumentation will render execution extremely slow.
Alternatively we would still support instrumenting function bodies in
this limited case but take into account the number of times function is
actually called.
Previously, static method calls on `Any` have not worked as expected. For example, `Any.to_text` returned Function instead of Text. That is because the function resolution for `Any.to_text` finds `Any.type.to_text` method on eigentype which expects two `self` arguments, but only one argument is provided.
Note that `Boolean.to_text` worked previously, and returned "Boolean" as expected. This is because the method resolution finds `Any.to_text` method that takes just one `self` argument.
This PR solves this issue by introducing special handling for static method dispatch on `Any`. Simply put, an additional `self` argument is prepended to the argument list.
# Important Notes
A new child node is introduced to `InvokeMethodNode`. This child node is a copy of the current `invokeFunctionNode` with one more `CallArgumentInfo` in its schema.
There is an issue that after a clean build, the compiler is unable to resolve some types, i.e
```py
$ ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run ~/enso/projects/Unnamed/
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:24:11: error: The name `Connection_Details` could not be found.
24 | connect : Connection_Details -> Connection_Options -> Connection ! SQL_Error
| ^~~~~~~~~~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:24:33: error: The name `Connection_Options` could not be found.
24 | connect : Connection_Details -> Connection_Options -> Connection ! SQL_Error
| ^~~~~~~~~~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:24:55: error: The name `Connection` could not be found.
24 | connect : Connection_Details -> Connection_Options -> Connection ! SQL_Error
| ^~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:24:68: error: The name `SQL_Error` could not be found.
24 | connect : Connection_Details -> Connection_Options -> Connection ! SQL_Error
| ^~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:25:25: error: The name `Connection_Options` could not be found.
25 | connect details options=Connection_Options.Value =
| ^~~~~~~~~~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:29:29: error: The name `Widget` could not be found.
29 | connection_details_widget : Widget
| ^~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:31:28: error: The name `Vector` could not be found.
31 | default_constructors = Vector.from_polyglot_array <|
| ^~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:32:63: error: The name `False` could not be found.
32 | DatabaseConnectionDetailsSPI.get_default_constructors False
| ^~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:36:9: error: The name `Option` could not be found.
36 | Option name code
| ^~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:37:5: error: The name `Single_Choice` could not be found.
37 | Single_Choice display=Display.Always values=choices
| ^~~~~~~~~~~~~
/home/dbushev/projects/luna/enso/built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/lib/Standard/Database/0.0.0-dev/src/Connection/Database.enso:37:27: error: The name `Display` could not be found.
37 | Single_Choice display=Display.Always values=choices
| ^~~~~~~
Aborting due to 11 errors and 0 warnings.
Execution finished with an error: Compilation aborted due to errors.
```
The compiler can't resolve those symbols because the `IR` cache for `Standard.Database.Connection.Database` is missing. What happens during the `buildEngineDistribution` is:
- Compiler processes libraries one by one
- Compiler processes (and compiles and generates caches) `Standard.Database` library
- Compiler starts processing `Standard.Visualization` library
- During the compilation of `Standard.Database.Connection.Database` module, it sees that the module was loaded from the cache. But some of the required dependencies from `Standard.Table` library were not loaded from the cache. But at this point, the `Standard.Table` library has not been processed yet and the caches for it don't exist. The compiler decides that the `Database` file was changed, and the cache is invalid and should be cleaned.
- Removed `module` argument from `enso_project` (new `Project_Description.new` API).
- Removed the custom option from date and time parse/format dropdowns.
- The `format` dropdown uses the value to create the dropdown. (Screenshot below)
- Removed `StorageType` coalescing rules and replaced them with simpler logic in `ObjectStorage`.
- Update signature for `add_row_number` and add aliases.
Request Timeouts started plaguing IDE due to numerous `executionContext/***Visualization` requests. While caused by a bug they revealed a bigger problem in the Language Server when serving large amounts of requests:
1) Long and short lived jobs are fighting for various locks. Lock contention leads to some jobs waiting for a longer than desired leading to unexpected request timeouts. Increasing timeout value is just delaying the problem.
2) Requests coming from IDE are served almost instantly and handled by various commands. Commands can issue further jobs that serve request. We apparently have and always had a single-thread thread pool for serving such jobs, leading to immediate thread starvation.
Both reasons increase the chances of Request Timeouts when dealing with a large number of requests. For 2) I noticed that while we used to set the `enso-runtime-server.jobParallelism` option descriptor key to some machine-dependent value (most likely > 1), the value set would **only** be available for instrumentation. `JobExecutionEngine` where it is actually used would always get the default, i.e. a single-threaded ThreadPool. This means that this option descriptor was simply misused since its introduction. Moved that option to runtime options so that it can be set and retrieved during normal operation.
Adding parallelism intensified problem 1), because now we could execute multiple jobs and they would compete for resources. It also revealed a scenario for a yet another deadlock scenario, due to invalid order of lock acquisition. See `ExecuteJob` vs `UpsertVisualisationJob` order for details.
Still, a number of requests would continue to randomly timeout due to lock contention. It became apparent that
`Attach/Modify/Detach-VisualisationCmd` should not wait until a triggered `UpsertVisualisationJob` sends a response to the client; long and short lived jobs will always compete for resources and we cannot guarantee that they will not timeout that way. That is why the response is sent immediately from the command handler and not from the job executed after it.
This brings another problematic scenario:
1. `AttachVisualisationCmd` is executed, response sent to the client, `UpsertVisualisationJob` scheduled.
2. In the meantime `ModifyVisualisationCmd` comes and fails; command cannot find the visualization that will only be added by `UpsertVisualisationJob`, which might have not yet been scheduled to run.
Remedied that by checking visualisation-related jobs that are still in progress. It also allowed for cancelling jobs which results wouldn't be used anyway (`ModifyVisualisationCmd` sends its own `UpsertVisualisationJob`). This is not a theoretical scenario, it happened frequently on IDE startup.
This change does not fully solve the rather problematic setup of numerous locks, which are requested by short and long lived jobs. A better design should still be investigated. But it significantly reduces the chances of Request Timeouts which IDE had to deal with.
With this change I haven't been able to experience Request Timeouts for relatively modest projects anymore.
I added the possibility of logging wait times for locks to better investigate further problems.
Closes#7005
- Add type detection for `Mixed` columns when calling column functions.
- Excel uses column name for missing headers.
- Add aliases for parse functions on text.
- Adjust `Date`, `Time_Of_Day` and `Date_Time` parse functions to not take `Nothing` anymore and provide dropdowns.
- Removed built-in parses.
- All support Locale.
- Add support for missing day or year for parsing a Date.
- All will trim values automatically.
- Added ability to list AWS profiles.
- Added ability to list S3 buckets.
- Workaround for Table.aggregate so default item added works.
close#6936
Changelog:
- add: new suggestion type Getter that is not exposed to the api
- update: do not return suggestion of type getter when doing a global search (without specifying self types)
Private suggestions and modules mentioned in the issue will be filtered out after we finish the work on the new (refined) exports algorithm.
# Important Notes
As demonstrated in https://github.com/enso-org/enso/actions/runs/5175688022/jobs/9323585204?pr=6940
```
org.enso.interpreter.test.instrument.RuntimeAsyncCommandsTest *** ABORTED ***
org.graalvm.polyglot.PolyglotException: java.util.NoSuchElementException
at java.base/java.util.WeakHashMap$HashIterator.nextEntry(WeakHashMap.java:811)
at java.base/java.util.WeakHashMap$EntryIterator.next(WeakHashMap.java:848)
at java.base/java.util.WeakHashMap$EntryIterator.next(WeakHashMap.java:846)
at org.enso.interpreter.runtime.ThreadExecutors.shutdown(ThreadExecutors.java:46)
at org.enso.interpreter.runtime.EnsoContext.shutdown(EnsoContext.java:198)
at org.enso.interpreter.EnsoLanguage.finalizeContext(EnsoLanguage.java:179)
at org.enso.interpreter.EnsoLanguage.finalizeContext(EnsoLanguage.java:65)
at org.graalvm.truffle/com.oracle.truffle.api.LanguageAccessor$LanguageImpl.finalizeContext(LanguageAccessor.java:326)
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotLanguageContext.finalizeContext(PolyglotLanguageContext.java:404)
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotContextImpl.finalizeContext(PolyglotContextImpl.java:2925)
```
close#6900
This is a follow-up to the discussion of the imports/exports meeting.
Right now we have no control over the visibility of atom constructor arguments. One way to hide them is a convention of filtering getters by an `internal` prefix or suffix.
Related to #6912
It essentially solves it by removing any builtins that would take an EnsoDate/EnsoTimeOfDay/EnsoTimeZone and replacing them with Java utils that do the same operation.
This is not a proper solution - the builtin conversion is still invalid for the date/time types - but at this moment we may just no longer use the invalid conversion so it is much less of an issue. We still need to be aware of this if we want to introduce builtins taking date/time in the future.
At the beginning of the execution `EnsureCompiledJob` acquired write compilation lock. When compiling individual modules it would then
- acquire file lock
- acquire read compilation lock
The second one was spurious since it already kept the write lock. This sequence meant however that `CloseFileCmd` or `OpenFileCmd` can lead to a deadlock when requests come in close succession. This is because commands:
- acquire file lock
- acquire read compilation lock
So `EnsureCompiledJob` might have the (write) compilation lock but the commands could have file lock. And the second required lock for either the job or the command could never be acquired.
Flipping the order did the trick.
Partially solves #6841.
# Important Notes
For some reason we don't get updates for the newly added node, as illustrated in the screenshot, but that could be related to the close/open action. Will need to dig more.
Previously, a `RuntimeException` would be thrown when an attempt would be made to curry a conversion function. That is problematic for IDE where `executionFailed` means we can't enter functions due to lack of method pointers info.
Closes#6897.
# Important Notes
A more generic solution that allows to recover from execution failures will need a follow up.
close#6800
Update the `executionContext/expressionUpdates` notification and send the list of not applied arguments in addition to the method pointer.
# Important Notes
IDE is updated to support the new API.
Throwing `TailCallException` meant that exceptions that were extracted from the expression before the call was made could not be appended. This change catches the `TailCallException`, adds warnings to it and propagates it further, thus ensuring that we don't loose the information.
Closes#6765.
# Important Notes
Removed workarounds introduced in stdlib.
Add diagnosis for unresolved symbols in `from ... import sym1, sym2, ...` statements.
- Adds a new compiler pass, `ImportSymbolAnalysis`, that checks these statements and iterates through the symbols and checks if all the symbols can be resolved.
- Works with `BindingsMap` metadata.
- Add `ImportExportTest` that creates various modules with various imports/exports and checks their generated `BindingMap`.
---------
Co-authored-by: mergify[bot] <37929162+mergify[bot]@users.noreply.github.com>
Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org>
The change adds an additional field to `ExpressionUpdates` messages sent by `ProgramExecutionSupport` to indicate if the type of value (or its method pointer) has changed and therefore would potentially require a suggestions' update.
Prior to #3729 that check was done during the instrumentation. However we still want to continue to support "pending expression" functionality therefore `SuggestionsHandler` will use the additional information to filter only the required expression updates.
Most of the changes are related to adapting our tests to the new field.
Closes#6706.
# Important Notes
The associated project now loads and navigates smoothly.
Also attaching a screenshot from the project that illustrates that pending functionality continues to work:
[Kazam_screencast_00006.webm](https://github.com/enso-org/enso/assets/292128/35918841-f84f-4e1c-b1b0-40e45d97e111)
close#6611
Changelog:
- update: run compiler passes on the `ascribedType` field of the constructor arguments
- update: suggestion builder uses the type information attached to `ascribedType`
- feat: resolve qualified names in type signatures
Related to #6410
# Important Notes
- Updated some `Meta` methods (needed for error handling):
- `Meta.Type` now has `name` and `qualified_name`.
- `Meta.Constructor` has `declaring_type` allowing to get the type that this constructor is associated with.
Fixes#6609 by
- e380e647af - running whole `Vector_Spec` on `java.util.ArrayList`
- 9b1229fe20 - introducing a node to handle interop values
# Important Notes
Contains additional DSL processor fix:
- 415623dcb9 - to not crash the compiler, but to properly report compiler error
Artifically limiting the number of reported warnings to 100. Also added benchmarks with random Ints to investigate perf issues when dealing with warnings (future task).
Ideally we would have a custom set-like collection that allows us internally to specify a maximal number of elements. But `EnsoHashMap` (and potentially `EnsoSet`) are still WIP when it comes to being PE-friendly.
The change also allows for checking if the limit for the number of reported warnings has been reached. It will visualize by adding an additional "Warnings limit reached." to the visualization.
The limit is configurable via `--warnings-limit` parameter to `run`.
Closes#6283.
Add format to the in-memory Column
# Important Notes
Also updates .format in date types.
Some rearrangement of date formatting builtins / Java libraries.
Engine Benchmark job runs only engine benchmarks, not Enso benchmarks.
Enso benchmarks do not report their output anywhere, and take more than 5 hours to run nowadays.
We might define a new job in the future and probably rename it to "Library benchmarks".
But that is the responsibility of the lib team.
Deeply nested modules (of depth at least 3) would have the incorrect name inferred for synthetic modules. This also became apparent in a much bigger rewrite.
Instead of having module `A.B.C`, as described in the test, it would infer the name to be `B.A.C`, which would break when trying to reference symbols from `C`.
# Important Notes
No ticket reference, as this was something that @radeusgd discovered while working on another component.
The PR includes a minimal example that would previously fail to compile.
Remove the magical code generation of `enso_project` method from codegen phase and reimplement it as a proper builtin method.
The old behavior of `enso_project` was special, and violated the language semantics (regarding the `self` argument):
- It was implicitly declared in every module, so it could be called without a self argument.
- It can be called with explicit module as self argument, e.g. `Base.enso_project`, or `Visualizations.enso_project`.
Let's avoid implicit methods on modules and let's be explicit. Let's reimplement the `enso_project` as a builtin method. To comply with the language semantics, we will have to change the signature a bit:
- `enso_project` is a static method in the `Standard.Base.Meta.Enso_Project` module.
- It takes an optional `project` argument (instead of taking it as an explicit self argument).
Having the `enso_project` defined as a (shadowed) builtin method, we will automatically have suggestions created for it.
# Important Notes
- Truffle nodes are no longer generated in codegen phase for the `enso_project` method. It is a standard builtin now.
- The minimal import to use `enso_project` is now `from Standard.Base.Meta.Enso_Project import enso_project`.
- Tested implicitly by `org.enso.compiler.ExecCompilerTest#testInvalidEnsoProjectRef`.
close#6324
Changelog
- feat: DataflowAnalysis compiler pass preserves the order of dependencies. This way when attaching the visualization to the sub-expression, the engine can find the first cached parent node, and properly invalidate it.
- update: runtime visualization test is updated to reproduce the issue
# Important Notes
The dropdown for the column `"LOCATION"` is available right after the Restaurants project startup.
There is 572 threads during `RuntimeServerTest` execution create. They stay around because the context isn't closed after each test finishes. Adding `afterEach` cleanup sequence to `runtime-with-instruments` tests. Improving robustness of overall `EnsoContext` [shutdown](https://github.com/enso-org/enso/pull/6468#discussion_r1180348303).
- Adjusted `Context.is_enabled` to support default argument (moved built in so can have defaults).
- Made `environment` case-insensitive.
- Bug fix for play button.
- Short hand to execute within an enabled context.
- Forbid file writing if the Output context is disabled with a `Forbidden_Operation` error.
- Add temporary file support via `File.create_temporary_file` which is deleted on exit of JVM.
- Execution Context first pass in `Text.write`.
- Added dry run warning.
- Writes to a temporary file if disabled.
- Created a `DryRunFileManager` which will create and manage the temporary files.
- Added `format` dropdown to `File.read` and `Data.read`.
- Renamed `JSON_File` to `JSON_Format` to be consistent.
(still to unit test).
Fixes#6416 by introducing `InlineableNode`. It runs fast even on GraalVM CE, fixes ([forever broken](https://github.com/enso-org/enso/pull/6442#discussion_r1178782635)) `Debug.eval` with `<|` and [removes discouraged subclassing](https://github.com/enso-org/enso/pull/6442#discussion_r1178778968) of `DirectCallNode`. Introduces `@BuiltinMethod.needsFrame` - something that was requested by #6293. Just in this PR the attribute is optional - its implicit value continues to be derived from `VirtualFrame` presence/absence in the builtin method argument list. A lot of methods had to be modified to pass the `VirtualFrame` parameter along to propagate it where needed.
This change modifies method dispatch for methods that override Any's definitions. When an overrided method is invoked statically we call Any's method to stay consistent.
This change primarily addresses the plethora of problems related to `to_text` invocations. It does not attempt to completely modify method dispatch logic.
Closes#6300.
Fixes the runtime test
```
- should recompute expressions changing an execution environment *** FAILED ***
"[live]" did not equal "[design]" (RuntimeServerTest.scala:2721)
Analysis:
"[live]" -> "[design]"
```
- Missing tests from number parsing.
- Fix type signature on some warning methods.
- Fix warnings on `Standard.Database.Data.Table.parse_values`.
- Added test for `Nothing` and empty string on `use_first_row_as_names`.
- New API for `Number.format` taking a simple format string and `Locale`.
- Add ellipsis to truncated `Text.to_display_text`.
- Adjusted built-in `to_display_text` for numbers to not include type (but also to display BigInteger as value).
- Remove `Noise.Generator` interface type.
- Json: Added `to_display_text` to `JS_Object`.
- Time: Added `to_display_text` for `Date`, `Time_Of_Day`, `Date_Time`, `Duration` and `Period`.
- Text: Added `to_display_text` to `Locale`, `Case_Sensitivity`, `Encoding`, `Text_Sub_Range`, `Span`, `Utf_16_Span`.
- System: Added `to_display_text` to `File`, `File_Permissions`, `Process_Result` and `Exit_Code`.
- Network: Added `to_display_text` to `URI`, `HTTP_Status_Code` and `Header`.
- Added `to_display_text` to `Maybe`, `Regression`, `Pair`, `Range`, `Filter_Condition`.
- Added support for `to_js_object` and `to_display_text` to `Random_Number_Generator`.
- Verified all error types have `to_display_text`.
- Removed `BigInt`, `Date`, `Date_Time` and `Time_Of_Day` JS based rendering as using `to_display_text` now.
- Added support for rendering nested structures in the table viz.
related #6323
The engine can skip compilation when applying changes that do not require execution. It is more efficient to process the changes in a batch, than triggering compilations every time such edit is received.
While Enso runs single-threaded, its `ResourceManager` required additional asynchronous thread to execute its _"finalizers"_. What has been necessary back then is no longer needed since _GraalVM 21.1_. GraalVM now provides support for submitting `ThreadLocalAction` that gets then picked and executed via `TruffleSafepoint` locations. This PR uses such mechanism to _"inject"_ finalizer execution into already running Enso evaluation thread.
Requiring more than one thread has complicated Enso's co-existence with other Truffle language. For example Graal.js is strictly singlethreaded and used to refuse (simple) co-existence with Enso. By allowing Enso to perform all its actions in a single thread, the synergy with Graal.js becomes better.
`Number.nan` can be used as a key in `Map`. This PR basically implements the support for [JavaScript's Same Value Zero Equality](https://developer.mozilla.org/en-US/docs/Web/JavaScript/Equality_comparisons_and_sameness#same-value-zero_equality) so that `Number.nan` can be used as a key in `Map`.
# Important Notes
- For NaN, it holds that `Meta.is_same_object Number.nan Number.nan`, and `Number.nan != Number.nan` - inspired by JS spec.
- `Meta.is_same_object x y` implies `Any.== x y`, except for `Number.nan`.
`Vector.sort` does some custom method dispatch logic which always expected a function as `by` and `on` arguments. At the same time, `UnresolvedSymbol` is treated like a (to be resolved) `Function` and under normal circumstances there would be no difference between `_.foo` and `.foo` provided as arguments.
Rather than adding an additional phase that does some form of eta-expansion, to accomodate for this custom dispatch, this change only fixes the problem locally. We accept `Function` and `UnresolvedSymbol` and perform the resolution on the fly. Ideally, we would have a specialization on the latter but again, it would be dependent on the contents of the `Vector` so unclear if that is better.
Closes#6276,
# Important Notes
There was a suggestion to somehow modify our codegen to accomodate for this scenario but I went against it. In fact a lot of name literals have `isMethod` flag and that information is used in the passes but it should not control how (late) codegen is done. If we were to make this more generic, I would suggest maybe to add separate eta-expansion pass. But it could affect other things and could be potentially a significant change with limited potential initially, so potential future work item.
https://github.com/orgs/enso-org/discussions/6344 requested to change the order of arguments when controlling context permissions.
# Important Notes
The change brings it closer to the design doc but IMHO also a bit cumbersome to use (see changed tests) - applications involving default arguments don't play well when the last argument is not the default 🤷 .`
* Update type ascriptions in some operators in Any
* Add @GenerateUncached to AnyToTextNode.
Will be used in another node with @GenerateUncached.
* Add tests for "sort handles incomparable types"
* Vector.sort handles incomparable types
* Implement sort handling for different comparators
* Comparison operators in Any do not throw Type_Error
* Fix some issues in Ordering_Spec
* Remove the remaining comparison operator overrides for numbers.
* Consolidate all sorting functionality into a single builtin node.
* Fix warnings attachment in sort
* PrimitiveValuesComparator handles other types than primitives
* Fix byFunc calling
* on function can be called from the builtin
* Fix build of native image
* Update changelog
* Add VectorSortTest
* Builtin method should not throw DataflowError.
If yes, the message is discarded (a bug?)
* TypeOfNode may not return only Type
* UnresolvedSymbol is not supported as `on` argument to Vector.sort_builtin
* Fix docs
* Fix bigint spec in LessThanNode
* Small fixes
* Small fixes
* Nothings and Nans are sorted at the end of default comparator group.
But not at the whole end of the resulting vector.
* Fix checking of `by` parameter - now accepts functions with default arguments.
* Fix changelog formatting
* Fix imports in DebuggingEnsoTest
* Remove Array.sort_builtin
* Add comparison operators to micro-distribution
* Remove Array.sort_builtin
* Replace Incomparable_Values by Type_Error in some tests
* Add on_incomparable argument to Vector.sort_builtin
* Fix after merge - Array.sort delegates to Vector.sort
* Add more tests for problem_behavior on Vector.sort
* SortVectorNode throws only Incomparable_Values.
* Delete Collections helper class
* Add test for expected failure for custom incomparable values
* Cosmetics.
* Fix test expecting different comparators warning
* isNothing is checked via interop
* Remove TruffleLogger from SortVectorNode
* Small review refactorings
* Revert "Remove the remaining comparison operator overrides for numbers."
This reverts commit 0df66b1080.
* Improve bench_download.py tool's `--compare` functionality.
- Output table is sorted by benchmark labels.
- Do not fail when there are different benchmark labels in both runs.
* Wrap potential interop values with `HostValueToEnsoNode`
* Use alter function in Vector_Spec
* Update docs
* Invalid comparison throws Incomparable_Values rather than Type_Error
* Number comparison builtin methods return Nothing in case of incomparables
The primary motivation for this change was
https://github.com/enso-org/enso/issues/6248, which requested the possibility of defining `to_display_text` methods of common errors via regular method definitions. Until now one could only define them via builtins.
To be able to support that, polyglot invocation had to report `to_display_text` in the list of (invokable) members, which it didn't. Until now, it only considered fields of constructors and builtin methods. That is now fixed as indicated by the change in `Atom`.
Closes#6248.
# Important Notes
Once most of builtins have been translated to regular Enso code, it became apparent how the usage of `.` at the end of the message is not consistent and inflexible. The pure message should never follow with a dot or it makes it impossible to pretty print consistently for the purpose of error reporting. Otherwise we regularly end up with errors ending with `..` or worse. So I went medieval on the reasons for failures and removed all the dots.
The overall result is mostly the same except now we are much more consistent.
Finally, there was a bit of a good reason for using builtins as it simplified our testing.
Take for example `No_Such_Method.Error`. If we do not import `Errors.Common` module we only rely on builtin error types. The type obviously has the constructor but it **does not have** `to_display_text` in scope; the latter is no longer a builtin method but a regular method. This is not really a problem for users who will always import stdlib but our tests often don't. Hence the number of changes and sometimes lack of human-readable errors there.
close#6232
Changelog:
- remove: `SqlVersionsRepo`
- update: `SuggestionsDatabaseModuleUpdateNotification` message removing the version
- update: cleanup versions repo usages in the language server
As per design, IOContexts controlled via type signatures are going away. They are replaced by explicit `Context.if_enabled` runtime checks that will be added to particular method implementations.
`production`/`development` `IOPermissions` are replaced with `live` and `design` execution enviornment. Currently, the `live` env has a hardcoded list of allowed contexts i.e. `Input` and `Output`.
# Important Notes
As per design PR-55. Closes#6129. Closes#6131.
close#6080
Changelog
- add: implement `SuggestionsRepo.insertAll` as a batch SQL insert
- update: `search/getSuggestionsDatabase` returns empty suggestions. Currently, the method is only used at startup and returns the empty response anyway because the libs are not loaded at that point.
- update: serialize only global (defined in the module scope) suggestions during the distribution building. There's no sense in storing the local library suggestions.
- update: sqlite dependency
- remove: unused methods from `SuggestionsRepo`
- remove: Arguments table
# Important Notes
Speeds up libraries loading by ~1 second.
Implements #6134.
# Important Notes
One can define lazy atom fields as:
```haskell
type Lazy
Value ~x ~y
```
the evaluation of the `x` and `y` fields is then delayed until they are needed. The evaluation happens once. Then the computed value is kept in the atom for further use.
Fixes#5898 by removing `Catch.panic` and speeding the `sieve.enso` benchmark from 1058 ms to 514 ms. Should there be no dedicated conversion, let's use one defined on `Any` type - e.g. defining a conversion `from(Any)` makes such a conversion is always available.
Delay creation of `EnsoFile` until it is needed.
# Important Notes
By putting breakpoint into `Atom` constructor I realized few `EnsoProjectNode` instances may be created when parsing the project. It makes no sense to also create `EnsoFile` for them - until it is needed.
Modification to various tests disabled when #5917 was integrated to pass with new parser. Fixes#5894.
# Important Notes
Some tests can be fixed just by changes on the `IR` side. Some (especially error simulating ones) would benefit from changes in the `Tree` structure or at least @kazcw evaluation.
close#6139close#6137
When the project is renamed, the engine cleans up affected modules and initiates modules re-indexing to fill the suggestions database with new records. This way it reduces the amount of information stored in the suggestions database and helps implement #6080 optimization.
Changelog:
- remove: rename features from the suggestions database
- update: rename command to initiate modules cleanup and project re-execution
- fix: #6137
`--compile` command would run the compilation pipeline but silently omit any encountered errors, thus skipping the serialization. This maybe was a good idea in the past but it was problematic now that we generate indexes on build time.
This resulted in rather obscure errors (#6092) for modules that were missing their caches.
The change should significantly improve developers' experience when working on stdlib.
# Important Notes
Making compilation more resilient to sudden cache misses is a separate item to be worked on.
The primary delivery of this PR is a design of `SerdeCompilerTest` - a testing suite that allows us to write sample projects, parse them with and without caches and verify they still produce the same `IR`. This is a similar idea to #3723 which compared the old and new parser `IR`s.
With infrastructure like this we can start addressing #5567 without any (significant) fear of breaking something essential.
Treat `Boolean.False` and `Boolean.True` as the corresponding primitives. Now, `Boolean.False == False` returns true.
# Important Notes
`False` and `True` constructs, that are converted to `ConstructorNode` during Truffle codegen, are handled specially in `ConstructorNode`. The easiest fix was to implement a similar special handling in `QualifiedAccessorNode`, although not the cleanest one.
A better solution would be to provide transformation of `Boolean.True` IR to a true literal in `ApplicationSaturation` compiler pass. But `ApplicationSaturation` pass does not handle `True`. Moreover, for our case, it is unnecessarily complicated.
Instrumentation of calls involving warning values never really worked because:
1) newly created nodes didn't set the UUID of their children
2) the instrumentable wrappers always had an empty (i.e. null) UUID and
they never referred `get`/`setId` calls to their delegates
On the surface, everything worked fine. Except when one actually relied on the instrumentation of values with warnings for proper setup. Then no instrumentation (replacement of nodes) was performed due to empty UUID (as required by `hasTag` of `FunctionCallInstrumentationNode`).
Closes#6045. Discovered in #5893.
- Fixes InvokeCallableNode to support warnings.
- Strips warnings from annotations in `get_widget_json`.
- Remove `get_full_annotations_json`.
- Fix warnings on Dialect.
Exporting types named the same as the module where they are defined in `Main` modules of library components may lead to accidental name conflicts. This became apparent when trying to access `Problem_Behavior` module via a fully qualified name and the compiler rejected it. This is due to the fact that `Main` module exported `Error` type defined in `Standard.Base.Error` module, thus making it impossible to access any other submodules of `Standard.Base.Error` via a fully qualified name.
This change adds a warning to FullyQualifiedNames pass that detects any such future problems.
While only `Error` module was affected, it was widely used in the stdlib, hence the number of changes.
Closes#5902.
# Important Notes
I left out the potential conflict in micro-distribution, thus ensuring we actually detect and report the warning.
close#5911
In interactive mode, perform writing IR caches in the background jobs queue. Background jobs execution is delayed until the first execution is complete.
Fixing #5768 and #5765 and co. Introducing `Meta.Type` and giving it the desired methods.
# Important Notes
`Type` is no longer a `Meta.Atom`, but it has a dedicated `Meta.Type` representation.
Fixes#5826.
# Important Notes
- Change frontend representation of negation.
- Fix a precedence issue: The `.` operators in -1.x and -1.2 must have different precedences.
- Remove a no-longer-needed special case from backend translation.
- Add tests for this case after all translations.
Give Cache subclasses a chance to control the output stream format. Use that functionality to avoid persisting `UUID` into standard library `.ir` caches. Gets the number of caches down to 42MB from 48MB.
# Important Notes
I believe UUIDs are not really useful for standard libraries and can be omitted. Am I right?
Implement new Enso documentation parser; remove old Scala Enso parser.
Performance: Total time parsing documentation is now ~2ms.
# Important Notes
- Doc parsing is now done only in the frontend.
- Some engine tests had never been switched to the new parser. We should investigate tests that don't pass after the switch: #5894.
- The option to run the old searcher has been removed, as it is obsolete and was already broken before this (see #5909).
- Some interfaces used only by the old searcher have been removed.
When generating import/export bindings in local cache, they are included in the distribution.
Additionally, removed the hardcoded value for suggestions cache. Now one can generate them for local as well as for global cache, based on the presence or lack of `--no-global-cache` parameter.
Closes#5890.
close#5889
Changelog:
- update: make `DetachVisualizationJob` a unique job to make sure that they are not canceled during the re-compilation after `text/applyEdit` command.
close#5892
Changelog:
add: feature to delay background jobs execution
add: start background jobs when program finishes
add: start background jobs on `search/completion` request
Fixes#5805 by returning `[]` as list of fields of `Type`.
# Important Notes
`Type` is recognized as `Meta.is_atom` since #3671. However `Type` isn't an `Atom` internally. We have to provide special handling for it where needed.
Adds a common project that allows sharing code between the `runtime` and `std-bits`.
Due to classpath separation and the way it is compiled, the classes will be duplicated - we will have one copy for the `runtime` classpath and another copy as a small JAR for `Standard.Base` library.
This is still much better than having the code duplicated - now at least we have a single source of truth for the shared implementations.
Due to the copying we should not expand this project too much, but I encourage to put here any methods that would otherwise require us to copy the code itself.
This may be a good place to put parts of the hashing logic to then allow sharing the logic between the `runtime` and the `MultiValueKey` in the `Table` library (cc: @Akirathan).
close#5070
Changelog:
- Include the original exception to log expressions
- Enable logging of Akka Actors' lifecycle events on debug logging level
- Decrease the severity of interruption log messages because interruptions are part of the workflow. The computation can be interrupted at any time, and still be recomputed after. Warnings are just misleading in this case.
Merge _ordered_ and _unordered_ comparators into a single one.
# Important Notes
Comparator is now required to have only `compare` method:
```
type Comparator
comapre : T -> T -> (Ordering|Nothing)
hash : T -> Integer
```
