Unsetting literal sources during the `text/fileClose` operation would also unlink sources from the actual file. That would be never set back during the reopening of the file, resulting in an NPE during application of changes at a later point.
# Important Notes
Revealed during https://github.com/enso-org/enso/pull/4050 which was the first(?) to do close/open of the file.
The exception showing up in GUI and preventing further actions is now gone.
The new `TreeToIr` conversion class inherited usage of `UnhandledEntity` to signal parser errors from the previous `AstToIr` Scala based convertor. Over the time we were improving error recovery - however only _step by step_. This pull request exterminates _all the panics_ for once and forever!
# Important Notes
Unlike Scala, Java has concept of [checked exceptions](http://wiki.apidesign.org/wiki/Checked_exception) - an exception that has to be caught. This PR introduces new checked `SyntaxException` and throws it instead of unchecked `UnhandledEntity`. Because the exception is checked, each method either needs to declare it in its signature or catch it and handle it. No exception can escape or disappear. The main conversion method `TreeToIr.translateModule` doesn't propagate the `SyntaxException`. That provably demonstrates - _all panic states_ are handled and reported as _syntax errors_ in the `IR`.
Component Groups may add additional modules to compilation. This change ensures that whatever modules result from import/export resolution, they are not duplicate. Not only could that lead to unnecessary compilation but also to multiple reports of the same error which would be confusing to users.
This fixes https://www.pivotaltracker.com/story/show/184189980
# Important Notes
No easy way to write unit tests for this, so skipping that on purpose for now.
Use `InteropLibrary.isString` and `asString` to convert any string value to `byte[]`
# Important Notes
Also contains a support for `Metadata.assertInCode` to help locating the right place in the code snippets.
**Vector**
- Adjusted `Vector.sort` to be `Vector.sort order on by`.
- Adjusted other sort to use `order` for direction argument.
- Added `insert`, `remove`, `index_of` and `last_index_of` to `Vector`.
- Added `start` and `if_missing` arguments to `find` on `Vector`, and adjusted default is `Not_Found` error.
- Added type checking to `+` on `Vector`.
- Altered `first`, `second` and `last` to error with `Index_Out_Of_Bounds` on `Vector`.
- Removed `sum`, `exists`, `head`, `init`, `tail`, `rest`, `append`, `prepend` from `Vector`.
**Pair**
- Added `last`, `any`, `all`, `contains`, `find`, `index_of`, `last_index_of`, `reverse`, `each`, `fold` and `reduce` to `Pair`.
- Added `get` to `Pair`.
**Range**
- Added `first`, `second`, `index_of`, `last_index_of`, `reverse` and `reduce` to `Range`.
- Added `at` and `get` to `Range`.
- Added `start` and `if_missing` arguments to `find` on `Range`.
- Simplified `last` and `length` of `Range`.
- Removed `exists` from `Range`.
**List**
- Added `second`, `find`, `index_of`, `last_index_of`, `reverse` and `reduce` to `Range`.
- Added `at` and `get` to `List`.
- Removed `exists` from `List`.
- Made `all` short-circuit if any fail on `List`.
- Altered `is_empty` to not compute the length of `List`.
- Altered `first`, `tail`, `head`, `init` and `last` to error with `Index_Out_Of_Bounds` on `List`.
**Others**
- Added `first`, `second`, `last`, `get` to `Text`.
- Added wrapper methods to the Random_Number_Generator so you can get random values more easily.
- Adjusted `Aggregate_Column` to operate on the first column by default.
- Added `contains_key` to `Map`.
- Added ALIAS to `row_count` and `order_by`.
Potential workaround to line endings problems after `vcs/restore` operation is executed.
# Important Notes
Not really able to reproduce the problem myself so this PR has a lot of _leap of faith_ in it.
Don't propagate errors from `toDisplayString` - construct an error message with `Atom.toString`.
# Important Notes
> currently a failure in to_text is swallowed by `toString` and we cannot detect that something went wrong during the serialization
Not sure how satisfying the solution is, but the error swallowing happens in Truffle and there is little to do with it. We can just catch the error ourselves and produce some meaningful string.
`Text.trim` `what` argument offered `Text` in `tagValues` - that's wrong. Using the `Text` type isn't allowed value for `Text`
# Important Notes
I had to update three other tests to match the new behavior.
Most of the problems with accessing `ArrayOverBuffer` have been resolved by using `CoerceArrayNode` (https://github.com/enso-org/enso/pull/3817). In `Array.sort` we still however specialized on Array which wasn't compatible with `ArrayOverBuffer`. Similarly sorting JS or Python arrays wouldn't work.
Added a specialization to `Array.sort` to deal with that case. A generic specialization (with `hasArrayElements`) not only handles `ArrayOverBuffer` but also polyglot arrays coming from JS or Python. We could have an additional specialization for `ArrayOverBuffer` only (removed in the last commit) that returns `ArrayOverBuffer` rather than `Array` although that adds additional complexity which so far is unnecessary.
Also fixed an example in `Array.enso` by providing a default argument.
Compiler performed name resolution of literals in type signatures but would silently fail to report any problems.
This meant that wrong names or forgotten imports would sneak in to stdlib.
This change introduces 2 main changes:
1) failed name resolutions are appended in `TypeNames` pass
2) `GatherDiagnostics` pass also collects and reports failures from type
signatures IR
Updated stdlib so that it passes given the correct gatekeepers in place.
VCS restore operation was correctly restoring the state of projects to the requested commit. Unfortunately, after the operation file system was becoming out-of-sync with language server's buffers (and IDE's content versions).
A few important changes are introduced here that complicate the interaction between components:
1) `vcs restore` returns an actual diff between the current state and the
requested commit
2) the response is forwarded to buffer registry first rather than to the client
3) the diff is used to identify appropriate collaborative editors and
notify them about the need to reload buffers from file system
4) all clients of affected open buffers are notified of the change via
`text/didChange` notification. If a file was removed and there were open buffers for it, clients will be notified via `file/event` and editor will be stopped
5) only then the client is notified about a successful restore operation
This PR addresses one of the two problems reported in https://www.pivotaltracker.com/story/show/184097084.
# Important Notes
We need to make sure that IDE correctly responds to `text/didChange` notifications.
Implements https://www.pivotaltracker.com/story/show/184032869
# Important Notes
- Currently we get failures in Full joins on Postgres which show a more serious problem - amending equality to ensure that `[NULL = NULL] == True` breaks hash/merge based indexing - so such joins will be extremely inefficient. All our joins currently rely on this notion of equality which will mean all of our DB joins will be extremely inefficient.
- We need to find a solution that will support nulls and still work OK with indices (but after exploring a few approaches: `COALESCE(a = b, a IS NULL AND b is NULL)`, `a IS NOT DISTINCT FROM b`, `(a = b) OR (a IS NULL AND b is NULL)`; all of which did not work (they all result in `ERROR: FULL JOIN is only supported with merge-joinable or hash-joinable join conditions`) I'm less certain that it is possible. Alternatively, we may need to change the NULL semantics to align it with SQL - this seems like likely the simpler solution, allowing us to generate simple, reliable SQL - the NULL=NULL solution will be cornering us into nasty workarounds very dependent on the particular backend.
`runtime-with-instruments` project sets `-Dgraalvm.locatorDisabled=true` that disables the discovery of available polyglot languages (installed with `gu`). On the other hand, enabling locator makes polyglot languages available, but also makes the program classes and the test classes loaded with different classloaders. This way we're unable to use `EnsoContext` in tests to observe internal context state (there is an exception when you try to cast to `EnsoContext`).
The solution is to move tests with enabled polyglot support, but disabled `EnsoContext` introspection to a separate project.
PR adds a flag to `Text` implementation tracking whether it is in a FCD normal form. Then this information can be used in the `Normalizer.compare` method.
| Benchmark name | Old (ms) | With flag (ms)
| --- | --- | ---
| Unicode very short | 40.29 | 40.04
| Unicode medium | 9.07 | 1.99
| Unicode big - random | 115.39 | 0.35
| Unicode big - early difference | 107.02 | 0.54
| Unicode big - late difference | 749.81 | 94.73
| ASCII very short | 28.13 | 31.13
| ASCII medium | 4.58 | 2.26
| ASCII big - random | 42.68 | 0.26
| ASCII big - early difference | 30.91 | 0.32
| ASCII big - late difference | 66.29 | 42.72
Full benchmark output.
[bench_old.txt](https://github.com/enso-org/enso/files/10325202/bench_old.txt)
[bench_new.txt](https://github.com/enso-org/enso/files/10325201/bench_new.txt)
`NestedPatternMatch` pass desugared complex patterns in a very inefficient way resulting in an exponential generation of the number of `case` IR (and Truffle) nodes. Every failed nested pattern would copy all the remaining patterns of the original case expression, in a desugared form. While the execution itself of such deeply nested `case` expression might not have many problems, the time spent in compilation phases certainly was a blocker.
This change desugars deeply nested into individual cases with a fallthrough logic. However the fallthrough logic is implemented directly in Truffle nodes, rather than via IR. That way we can generate much simpler IR for nested patterns.
Consider a simple case of
```
case x of
Cons (Cons a b) Nil -> a + b
Cons a Nil -> a
_ -> 0
```
Before the change, the compiler would generate rather large IR even for those two patterns:
```
case x of
Cons w y -> case w of
Cons a b -> case y of
Nil -> a + b
_ -> case x of
Cons a z -> case z of
Nil -> a
_ -> case x of
_ -> 0
_ -> 0
_ -> case x of
Cons a z -> case z of
Nil -> a
_ -> case x of
_ -> 0
_ -> 0
Cons a z -> case z of
Nil -> a
_ -> case x of
_ -> 0
_ -> 0
```
Now we generate simple patterns with fallthrough semantics and no catch-all branches:
```
case x of
Cons w y -> case w of
Cons a b -> case y of ## fallthrough on failed match ##
Nil -> a + b ## fallthrough on failed match ##
Cons a z -> case z of
Nil -> a ## fallthrough on failed match ##
_ -> 0
```
# Important Notes
If you wonder how much does it improve, then @radeusgd's example in https://www.pivotaltracker.com/story/show/183971366/comments/234688327 used to take at least 8 minutes to compile and run.
Now it takes 5 seconds from cold start.
Also, the example in the benchmark includes compilation time on purpose (that was the main culprit of the slowdown).
For the old implementation I had to kill it after 15 minutes as it still wouldn't finish a single compilation.
Now it runs 2 seconds or less.
Bonus points: This PR will also fix problem reported in https://www.pivotaltracker.com/story/show/184071954 (duplicate errors for nested patterns)
Nested type declaration shall yield a syntax error
# Important Notes
Now the Radek's sample:
```
type Foo
type Bar
main = 42
```
yields
```bash
$ enso --run test.enso
In module test:
Compiler encountered errors:
test.enso[2:9-2:16]: Unexpected declaration in the body of a type.
Aborting due to 1 errors and 0 warnings.
Execution finished with an error: Compilation aborted due to errors.
```
`Any.==` is a builtin method. The semantics is the same as it used to be, except that we no longer assume `x == y` iff `Meta.is_same_object x y`, which used to be the case and caused failures in table tests.
# Important Notes
Measurements from `EqualsBenchmarks` shows that the performance of `Any.==` for recursive atoms increased by roughly 20%, and the performance for primitive types stays roughly the same.
First part of fixing `Text.to_text`.
- add: `pretty` method for pretty printing.
- update: make `Text.to_text` conversion identity for Text
In the next iterations `to_text` will be gradually replaced with `to Text` conversion once the related issues with conversions are fixed.
Implements `getMetaObject` and related messages from Truffle interop for Enso values and types. Turns `Meta.is_a` into builtin and re-uses the same functionality.
# Important Notes
Adds `ValueGenerator` testing infrastructure to provide unified access to special Enso values and builtin types that can be reused by other tests, not just `MetaIsATest` and `MetaObjectTest`.
Use JavaScript to parse and serialise to JSON. Parses to native Enso object.
- `.to_json` now returns a `Text` of the JSON.
- Json methods now `parse`, `stringify` and `from_pairs`.
- New `JSON_Object` representing a JavaScript Object.
- `.to_js_object` allows for types to custom serialize. Returning a `JS_Object`.
- Default JSON format for Atom now has a `type` and `constructor` property (or method to call for as needed to deserialise).
- Removed `.into` support for now.
- Added JSON File Format and SPI to allow `Data.read` to work.
- Added `Data.fetch` API for easy Web download.
- Default visualization for JS Object trunctes, and made Vector default truncate children too.
Fixes defect where types with no constructor crashed on `to_json` (e.g. `Matching_Mode.Last.to_json`.
Adjusted default visualisation for Vector, so it doesn't serialise an array of arrays forever.
Likewise, JS_Object default visualisation is truncated to a small subset.
New convention:
- `.get` returns `Nothing` if a key or index is not present. Takes an `other` argument allowing control of default.
- `.at` error if key or index is not present.
- `Nothing` gains a `get` method allowing for easy propagation.
This removes the special handling of polyglot exceptions and allows matching on Java exceptions in the same way as for any other types.
`Polyglot_Error`, `Panic.catch_java` and `Panic.catch_primitive` are gone
The change mostly deals with the backslash of removing `Polyglot_Error` and two `Panic` methods.
`Panic.catch` was implemented as a builtin instead of delegating to `Panic.catch_primitive` builtin that is now gone.
This fixes https://www.pivotaltracker.com/story/show/182844611
Benchmark to compare _curried and lambda_ based function invocations and a fix to make _curried_ invocation (at least) as fast as the _lambda_ one. Allows us to use _curried invocations_ in standard library again without loosing any speed.
# Important Notes
Execute as:
```
sbt:runtime> benchOnly CurriedFunctionBenchmarks
```
Prior to subsequent bugfixes in this PR the benchmark results were:
- `averageCurried` runs in 0.290 ms
- `averageLambda` runs in 0.122 ms
e.g. _curried invocations_ is more than twice slow. That confirms our findings from the `Array_Proxy` vector benchmarks. The problem is that _function object is not compilation final_. After fixing it we have following results:
- `averageCurried` runs in 0.102 ms
- `averageLambda` runs in 0.111 ms
e.g. both operations are of similar complexity.
- Implemented https://www.pivotaltracker.com/story/show/183913276
- Refactored MultiValueIndex and MultiValueKeys to be more type-safe and more direct about using ordered or unordered maps.
- Added performance tests ensuring we use an efficient algorithm for the joins (the tests will fail for a full O(N*M) scan).
- Removed some duplicate code in the Table library.
- Added optional coloring of test results in terminal to make failures easier to spot.
- Aligned `compare_to` so returns `Type_Error` if `that` is wrong type for `Text`, `Ordering` and `Duration`.
- Add `empty_object`, `empty_array`. `get_or_else`, `at`, `field_names` and `length` to `Json`.
- Fix `Json` serialisation of NaN and Infinity (to "null").
- Added `length`, `at` and `to_vector` to Pair (allowing it to be treated as a Vector).
- Added `running_fold` to the `Vector` and `Range`.
- Added `first` and `last` to the `Vector.Builder`.
- Allow `order_by` to take a single `Sort_Column` or have a mix of `Text` and `Sort_Column.Name` in a `Vector`.
- Allow `select_columns_helper` to take a `Text` value. Allows for a single field in group_by in cross_tab.
- Added `Patch` and `Custom` to HTTP_Method.
- Added running `Statistic` calculation and moved more of the logic from Java to Enso. Performance seems similar to pure Java version now.
Fighting with _too many messages being delivered_ I wrote a test that dumps information about `AvoidIdInstrumentationTag` - every node that has `AvoidIdInstrumentationTag` is excluded from the instrumentation. However, when I look at the output for
```
from Standard.Base import all
import Standard.Visualization
run n = 0.up_to n . map i-> 1.noise * i
```
I see that `1.noise` didn't have the tag. Now there is [AvoidIdInstrumentationTagTest.java](https://github.com/enso-org/enso/pull/3973/files#diff-32cd9240bda2bfe0e5904695ced008daba86fefb3d137ac401997f4265fa50eb) which can be used to collect all programs where _too many messages is being delivered_. Just add a program, identify _isLambda_ and verify all nodes are properly tagged.
```
from Standard.Base import all
main =
value = 1
case value of
1 -> IO.println "one"
2 -> IO.println "two"
-1 -> IO.println "minus one"
_ -> IO.println "other"
```
# Important Notes
Had to write new `ExecCompilerTest` as comparing the AST with old compiler produced miserable results - the old AST wasn't ready for negative constants in `case of` at all.
Apparently
```
git --git-dir .enso/.vcs log refs/heads/master
fatal: ambiguous argument 'refs/heads/master': unknown revision or path not in the working tree.
```
but
```
git --git-dir .enso/.vcs log HEAD
fatal: ambiguous argument 'refs/heads/master': unknown revision or path not in the working tree.
```
works just fine on Windows.
Added some safeguards to avoid propagating weird errors because of retrieving element from an
empty Option.
The presence of `.git` in Enso's data directory (`.enso`), which should be under VCS as well, turned out to be really problematic. Git (and JGit) treat `.enso` as a submodule which prevents as from manual inspection of the repo. Additionally, every commit changed the submodule version, which in turn was reporting invalid/changed status always.
This change renames `.enso/.git` to `.enso/.vcs` which eliminates a lot of problems. Additionally, rather than doing `git add .` we selectively add files to the index, thus preventing most of the issues reported in the ticket.
This change also means that JGit's status needs to filter `.enso/.vcs` as those will be marked as untracked always.
# Important Notes
To manually test the status of the project's repo do
```
git --git-dir .enso/.vcs status
```
Using lambda instead of higher order function.
# Important Notes
Running:
```
sbt:runtime> benchOnly VectorBenchmarks.averageOverArrayProxy
```
speeds up from `0.038 ms/op` to `0.016 ms/op` on my computer. Which seems good enough.
Fixes bug in visualization of host polyglot values - `ExpressionNode` is only wrapped once Chrome inspector instrument is attached to the context. With this fix, when chromeinspector is attached (`enso --run --inspect ...`), all the host values are reinterpreted as text - the assumption is invalidated. But when running as language server, nothing is wrapped.
Allow arbitrary expression evaluation in the chromeinspector console. Moreover, allow modifications of any variable in any stack frame.
# Important Notes
- Implement inline parsing in `EnsoLanguage.parse(InlineParsingRequest)`.
- Debugging experience is affected by this [bug in Truffle](https://github.com/oracle/graal/issues/5513), which causes NPEs when a host object gets into chromeinspector. I tried to implement a workaround, but it does not work all the time. Nevertheless, it should not matter that much - if there is a NPE in the debugger, you can just ignore it, as it should be concealed in the debugger and should not be propagted outside. See comments in the `docs/debugger`.
* Sequence literal (Vector) should preserve warnings
When Vector was created via a sequence literal, we simply dropped any
associated any warnings associated with it.
This change propagates Warnings during the creation of the Vector.
Ideally, it would be sufficient to propagate warnings from the
individual elements to the underlying storage but doesn't go well with
`Vector.fromArray`.
* update changelog
* Array-like structures preserver warnings
Added a WarningsLibrary that exposes `hasWarnings` and `getWarnings`
messages. That way we can have a single storage that defines how to
extract warnings from an Array and the others just delegate to it.
This simplifies logic added to sequence literals to handle warnings.
* Ensure polyglot method calls are warning-free
Since warnings are no longer automatically extracted from Array-like
structures, we delay the operation until an actual polyglot method call
is performed.
Discovered a bug in `Warning.detach_selected_warnings` which was missing
any usage or tests.
* nits
* Support multi-dimensional Vectors with warnings
* Propagate warnings from case branches
* nit
* Propagate all vector warnings when reading element
Previously, accessing an element of an Array-like structure would only
return warnings of that element or of the structure itself.
Now, accessing an element also returns warnings from all its elements as
well.
- Moved `to_default_visualization_data` to `Standard.Visualization`.
- Remove the use of `is_a` in favour of case statements.
- Stop exporting Standard.Base.Error.Common.
- Separate errors to own files.
- Change constructors to be called `Error`.
- Rename `Caught_Panic.Caught_Panic_Data` -> `Caught_Panic.Panic`.
- Rename `Project_Description.Project_Description_Data` ->`Project_Description.Value`
- Rename `Regex_Matcher.Regex_Matcher_Data` -> `Regex_Matcher.Value` (can't come up with anything better!).
- Rename `Range.Value` -> `Range.Between`.
- Rename `Interval.Value` -> `Interval.Between`.
- Rename `Column.Column_Data` -> `Column.Value`.
- Rename `Table.Table_Data` -> `Table.Value`.
- Align all the Error types in Table.
- Removed GEO Json bits from Table.
- `Json.to_table` doesn't have the GEO bits anymore.
- Added `Json.geo_json_to_table` to add the functions back in.
# Important Notes
No more exports from anywhere but Main!
No more `_Data` constructors!
- Moved `Any`, `Error` and `Panic` to `Standard.Base`.
- Separated `Json` and `Range` extensions into own modules.
- Tidied `Case`, `Case_Sensitivity`, `Encoding`, `Matching`, `Regex_Matcher`, `Span`, `Text_Matcher`, `Text_Ordering` and `Text_Sub_Range` in `Standard.Base.Data.Text`.
- Tidied `Standard.Base.Data.Text.Extensions` and stopped it re-exporting anything.
- Tidied `Regex_Mode`. Renamed `Option` to `Regex_Option` and added type to export.
- Tidied up `Regex` space.
- Tidied up `Meta` space.
- Remove `Matching` from export.
- Moved `Standard.Base.Data.Boolean` to `Standard.Base.Boolean`.
# Important Notes
- Moved `to_json` and `to_default_visualization_data` from base types to extension methods.
Converting `Integer.parse` into a builtin and making sure it can parse big values like `100!`. Adding `locale` parameter to `Locale.parse` and making sure it parses `32,5` as `32.5` double in Czech locale.
# Important Notes
Note that one cannot
```
import Standard.Table as Table_Module
```
because of the 2-component name restriction that gets desugared to `Standard.Table.Main` and we have to write
```
import Standard.Table.Main as Table_Module
```
in a few places. Once we move `Json.to_table` extension this can be improved.
Disabling `musl` as it isn't capable to load dynamic library.
# Important Notes
With this change it is possible to:
```
$ sbt bootstrap
$ sbt engine-runner/buildNativeImage
$ ./runner --run ./engine/runner/src/test/resources/Factorial.enso 3
6
$ ./runner --run ./engine/runner/src/test/resources/Factorial.enso 4
24
$ ./runner --run ./engine/runner/src/test/resources/Factorial.enso 100
93326215443944152681699238856266700490715968264381621468592963895217599993229915608941463976156518286253697920827223758251185210916864000000000000000000000000
```
Is it OK, @radeusgd to disable `musl`? If not, we would have to find a way to link the parser in statically, not dynamically.
It appears that when were doing
`from XYZ import all`
the module `XYZ` was also being taken into account during name resolution.
This was unfortunate and became problematic when one had a type with the same name defined in it.
During pattern matching one could not simply do
```
from XYZ import all
...
case ... of
_ : XYZ -> ...
```
since the compiler would complain that we try to pattern match on a type but give it a module.
The module is now excluded from the name resolution, when importing everything from the module.
It appears that this "feature" was used in a number of our tests, so they had to be adapted.
This fixes task 4 in https://www.pivotaltracker.com/story/show/183833055
Rather than hard-coding `.git` in the root of the project, VCS should save data into Enso's data directory (i.e. `.enso`).
This change reshuffles initialization and configuration to store Git VCS metadata by default at `.enso/.git`.
This is follow up to https://github.com/enso-org/enso/pull/3851
# Important Notes
Apparently a custom Git directory in JGit means that it always creates a `.git` **file** with `gitdir` pointing to the custom location.
This is not necessary in our case since all our commands provide that explicitly.
That is why `init` operation removes `.git` file, which may seem a bit counter-intuitive.
By default all polyglot symbols that have been imported were always exported. This means that importing a module that had some polyglot imports brought them into the scope automatically. This didn't follow our desired semantics.
Fixes task 3 in https://www.pivotaltracker.com/story/show/183833055.
- Export all for `Problem_Behavior` (allowing for Report_Warning, Report_Error and Ignore to be trivially used).
- Renamed `Range.Range_Data` to `Range.Value` moved to using `up_to` wherever possible.
- Reviewed `Function`, `IO`, `Polyglot`, `Random`, `Runtime`, `System`.
- `File` now published as type. Some static methods moved to `Data` others into type. Removed `read_bytes` static.
- New `Data` module for reading input data in one place (e.g. `Data.read_file`) will add `Data.connect` later.
- Added `Random` module to the exports.
- Move static methods into `Warning` type and exporting the type not the module.
# Important Notes
- Sorted a few imports into order (ordering by direct import in project, then by from import in project then polyglot and finally self imports).
Upgrading to GraalVM 22.3.0.
# Important Notes
- Removed all deprecated `FrameSlot`, and replaced them with frame indexes - integers.
- Add more information to `AliasAnalysis` so that it also gathers these indexes.
- Add quick build mode option to `native-image` as default for non-release builds
- `graaljs` and `native-image` should now be downloaded via `gu` automatically, as dependencies.
- Remove `engine-runner-native` project - native image is now build straight from `engine-runner`.
- We used to have `engine-runner-native` without `sqldf` in classpath as a workaround for an internal native image bug.
- Fixed chrome inspector integration, such that it shows values of local variables both for current stack frame and caller stack frames.
- There are still many issues with the debugging in general, for example, when there is a polyglot value among local variables, a `NullPointerException` is thrown and no values are displayed.
- Removed some deprecated `native-image` options
- Remove some deprecated Truffle API method calls.
Previously, when exporting the same module multiple times only the first statement would count and the rest would be discarded by the compiler.
This change allows for multiple exports of the same module e.g.,
```
export project.F1
from project.F1 export foo
```
Multiple exports may however lead to conflicts when combined with hiding names. Added logic in `ImportResolver` to detect such scenarios.
This fixes https://www.pivotaltracker.com/n/projects/2539304/stories/183092447
# Important Notes
Added a bunch of scenarios to simulate pos and neg results.
RuntimeStdlibTest now checks that names in type signatures are qualified (aka poor man's typechecker)
Will be merged after the stdlib tidying by James is complete.
Fixes `NullPointerException` in `TreeToIr` by detecting the case and raising `UnexpectedExpression` syntax error.
# Important Notes
This PR prevents the `NullPointerException`. It doesn't change the meaning of `x.` (followed by space) `length`. It just detects when `Tree.OprApp.getRhs() == null` and treats that as an error.
`LocalDate.now()` returns time in system time zone, and git uses UTC. Sometimes it can cause test to fail
```
INFO ide_ci::program::command: sbtℹ️ [info] - must create a commit with a timestamp *** FAILED ***
INFO ide_ci::program::command: sbtℹ️ [info] "2022-11-18T23:05:01.768499800Z" did not start with substring "2022-11-19" (VcsManagerTest.scala:201)
```
- Moved static methods into `Locale` type. Publishing type not module.
- Stop publishing `Nil` and `Cons` from `List`.
- Tidied up `Json` and merged static in to type. Sorted out various type signatures which used a `Constructor`. Now exporting type and extensions.
- Tidied up `Noise` and merge `Generator` into file. Export type not module.
- Moved static method of `Map` into type. Publishing type not module.
# Important Notes
- Move `Text.compare_to` into `Text`.
- Move `Text.to_json` into `Json`.
* Tidy Bound and Interval.
* Fix Interval tests.
* Fix Interval tests.
* Restructure Index_Sub_Range to new Type/Statics.
* Adjust for Vector exported as a type and static methods on it.
* Tidy Maybe.
* Fix issue with Line_Ending_Style.
* Revert Filter_Condition change.
Fix benchmark test issue.
Tidy imports on Index_Sub_Range.
* Revert Filter_Condition change.
Fix benchmark test issue.
Tidy imports on Index_Sub_Range.
* Can't export constructors unless exported from type in module.
* Fix failing tests.
Ignore gitconfig located in the user's home directory when creating Git instance.
For example, I setup git to automatically sign commits. And all git related tests are failing because of that:
```
[info] - should return last X commits *** FAILED ***
[info] org.eclipse.jgit.api.errors.ServiceUnavailableException: Signing service is not available
[info] at org.eclipse.jgit.api.CommitCommand.sign(CommitCommand.java:328)
[info] at org.eclipse.jgit.api.CommitCommand.call(CommitCommand.java:283)
[info] at org.enso.languageserver.vcsmanager.GitSpec$InitialRepoSetup.setup(GitSpec.scala:360)
[info] at org.enso.languageserver.vcsmanager.GitSpec$InitialRepoSetup.setup$(GitSpec.scala:348)
[info] at org.enso.languageserver.vcsmanager.GitSpec$$anon$16.setup(GitSpec.scala:275)
[info] at org.enso.languageserver.vcsmanager.GitSpec$InitialRepoSetup.$init$(GitSpec.scala:346)
[info] at org.enso.languageserver.vcsmanager.GitSpec$$anon$16.<init>(GitSpec.scala:275)
[info] at org.enso.languageserver.vcsmanager.GitSpec.$anonfun$new$35(GitSpec.scala:275)
[info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info] ...
```
- Allow `Map` to store a `Nothing` key (fixes `Vector.distinct` with a `Nothing`).
- Add `column_names` method to `Table` as a shorthand.
- Return data flow error when comparing with Nothing (not a Panic or a Polyglot exception).
- Allow milli and micro second for DateTime and Time Of Day
# Important Notes
- Added a load of tests for the various comparison operators to Numbers_Spec.
It appears that we were always adding builtin methods to the scope of the module and the builtin type that shared the same name.
This resulted in some methods being accidentally available even though they shouldn't.
This change treats differently builtins of types and modules and introduces auto-registration feature for builtins.
By default all builtin methods are registered with a type, unless explicitly defined in the annotation property.
Builtin methods that are auto-registered do not have to be explicitly defined and are registered with the underlying type.
Registration correctly infers the right type, depending whether we deal with static or instance methods.
Builtin methods that are not auto-registered have to be explicitly defined **always**. Modules' builtin methods are the prime example.
# Important Notes
Builtins now carry information whether they are static or not (inferred from the lack of `self` parameter).
They also carry a `autoRegister` property to determine if a builtin method should be automatically registered with the type.
Two fixes in `EnsoCompilerTest` to align new parser with the previous behavior.
# Important Notes
Removes two `@Ignore` annotations. One test was already working. Another one is fixed by Kaz's idea to ignore _"empty block"_ application.
Libraries: Revert changes that were necessitated by a new rule we have decided not to introduce.
Parser:
- Support mixed constructors/bindings in types.
- Disallow zero-length hex sequences in character escapes: `\x`, `\u`, `\u{}`, `\U`, `\U{}` are no longer legal synonyms for `\0` (matches old parser behavior).
This change adds support for Version Controlled projects in language server.
Version Control supports operations:
- `init` - initialize VCS for a project
- `save` - commit all changes to the project in VCS
- `restore` - ability to restore project to some past `save`
- `status` - show the status of the project from VCS' perspective
- `list` - show a list of requested saves
# Important Notes
Behind the scenes, Enso's VCS uses git (or rather [jGit](https://www.eclipse.org/jgit/)) but nothing stops us from using a different implementation as long as it conforms to the establish API.
Computing length of a text takes time. Let's cache it after first computation.
# Important Notes
Wrote `StringBenchmarks` that sums lengths of (the same) `Text` present many time in a `Vector`. Initially it took `383.673 ms` per operation. Then it took `0.031 ms/op`. Looks like the `length` calls are returning instantly as they get cached.
This PR mimics test cases from #3860 and makes sure `IR.Syntax.Error` is constructed at appropriate places rather than just yielding an `UnhandledEntity` exception.
# Important Notes
Merge before #3611 to minimize disruption when changing the parser.
Another set of improvements extracted from #3611. This time it includes a fix to the Rust part of the parser.
# Important Notes
After digging into metadata parsing I realized the positions used to query the BTree data structure are wrong. This PR tries to address that by re-arranging the order of serialized fields and passing `startCode` and `endCode` locations in.
Originally I though I need changes on the Rust side to support `in` operator. Turned out I can do that just with changes on the Java side.
Qualified names in imports were missing UUIDs. Fixed now.
To minimize differences between #3611 and `develop` branch I propose to make following code changes that seem to work fine with the old as well as new parser. In addition to that there are new tests comparing the two parsers.
# Important Notes
Old parser is still used everywhere except `EnsoCompilerTest` where the test compares `IR`s constructed by both parsers.
Another set of fixes extracted from #3611. `CodeLocationsTest` test has been made more flexible to allow _"off by one"_ differences in the offset locations between the old and new parser.
Fix bugs in `TreeToIr` (rewrite) and parser. Implement more undocumented features in parser. Emulate some old parser bugs and quirks for compatibility.
Changes in libs:
- Fix some bugs.
- Clean up some odd syntaxes that the old parser translates idiosyncratically.
- Constructors are now required to precede methods.
# Important Notes
Out of 221 files:
- 215 match the old parser
- 6 contain complex types the old parser is known not to handle correctly
So, compared to the old parser, the new parser parses 103% of files correctly.
This PR adds `Period` type, which is a date-only complement to `Duration` builtin type.
# Important Notes
- `Period` replaces `Date_Period`, and `Time_Period`.
- Added shorthand constructors for `Duration` and `Period`. For example: `Period.days 10` instead of `Period.new days=10`.
- `Period` can be compared to other `Period` in some cases, other cases throw an error.
Define start of Enso epoch as 15th of October 1582 - start of the Gregorian calendar.
# Important Notes
- Some (Gregorian) calendar related functionalities within `Date` and `Date_Time` now produces a warning if the receiving Date/Date_Time is before the epoch start, e.g., `week_of_year`, `is_leap_year`, etc.
1. Changes how we do monadic state – rather than a haskelly solution, we now have an implicit env with mutable data inside. It's better for the JVM. It also opens the possibility to have state ratained on exceptions (previously not possible) – both can now be implemented.
2. Introduces permission check system for IO actions.
Moved loading of Builtin Types and Methods to a static initializer. That way the information is available at Native Image build time and one does not have to update a corresponding entry in `reflect-config` which would be a real pain when we start using it in anger.
Fixes https://www.pivotaltracker.com/story/show/183374932
Most of the time, rather than defining the type of the parameter of the builtin, we want to accept every Array-like object i.e. Vector, Array, polyglot Array etc.
Rather than writing all possible combinations, and likely causing bugs on the way anyway as we already saw, one should use `CoerceArrayNode` to convert to Java's `Object[]`.
Added various test cases to illustrate the problem.
Another part of #3611 with few more `TreeToIr` improvements.
# Important Notes
Unofficial `LoadParser.sh` check from #3611 of all library files now reports just 54 failures out of 222 files - e.g. 75% success rate.
A new test to verify that some `Standard.Base` library files can be parsed by the new parser is added. Use it as:
```bash
$ sbt bootstrap
$ sbt buildEngineDistribution
$ sbt "runtime/testOnly *ParseStdLibTest"
```
By having this test instantly available we can block integration of PRs that would decrease the number of parseable files or introduce new files that aren't paseable.
# Important Notes
The test contains a _black list_ of classes that are currently known to fail. Over the time the _black list_ shall get smaller and smaller.
The main culprit of a Vector slowdown (when compared to Array) was the normalization of the index when accessing the elements. Turns out that the Graal was very persistent on **not** inlining that particular fragment and that was degrading the results in benchmarks.
Being unable to force it to do it (looks like a combination of thunk execution and another layer of indirection) we resorted to just moving the normalization to the builtin method. That makes Array and Vector perform roughly the same.
Moved all handling of invalid index into the builtin as well, simplifying the Enso implementation. This also meant that `Vector.unsafe_at` is now obsolete.
Additionally, added support for negative indices in Array, to behave in the same way as for Vector.
# Important Notes
Note that this workaround only addresses this particular perf issue. I'm pretty sure we will have more of such scenarios.
Before the change `averageOverVector` benchmark averaged around `0.033 ms/op` now it does consistently `0.016 ms/op`, similarly to `averageOverArray`.
Another part of #3611 ready for integration into `develop` branch.
# Important Notes
Test `org.enso.compiler.EnsoCompilerTest.testTestGroup` is ignored as it has problems with source offsets - identifiers don't have the appropriate names due to `Tree.codeRepr()` being _off_.
Improve `Unsupported_Argument_Types` error so that it includes the message from the original exception. `arguments` field is retained, but not included in `to_display_text` method.
- Special precedence rules for case-of so that `:` operator works without parens or nospace-grouping.
- Support an old-lambda syntax: `x->x-> x`. According to the usual rules, the first nospace group would be parsed as an operator section. The expression now parses as a lambda that contains a lambda.
- Match old parser treatment of # in doc comments.
- Tweak precedence so (a : B = c) works.
- Documented constructors.
Trying to invoke a foreign method with non-installed language (either not enabled in the Truffle `Context`, or not installed in the GraalVM distribution) results in `Polyglot_Error`, rather than crashing the entire engine.
- Reimplement the `Duration` type to a built-in type.
- `Duration` is an interop type.
- Allow Enso method dispatch on `Duration` interop coming from different languages.
# Important Notes
- The older `Duration` type should now be split into new `Duration` builtin type and a `Period` type.
- This PR does not implement `Period` type, so all the `Period`-related functionality is currently not working, e.g., `Date - Period`.
- This PR removes `Integer.milliseconds`, `Integer.seconds`, ..., `Integer.years` extension methods.
When trying to resolve an invalid method of a polyglot array we were reaching a state where no specialization applied.
Turns out we can now simplify the logic of inferring polyglot call type for arrays and avoid the crash.
- Implement macro-contexts-lite (`from` is now only a keyword at the beginning of a line)
- Support special nospace-group handling for old lambdas (so expressions like this work: `x-> y-> x + y`)
- Fix a text-escape incompatibility
# Important Notes
- There is now an `OperatorFunction`, which is like a `Function` but has an operator for a name, and likewise an `OperatorTypeSignature`.
Can't instantiate the associated type:
```
Exception in thread "main" java.lang.UnsupportedOperationException: Unsupported operation Value.newInstance(Object...) for 'Internal_Repl_Module___'(language: Java, type: com.oracle.truffle.polyglot.PolyglotMap). You can ensure that the operation is supported using Value.canInstantiate().
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotEngineException.unsupported(PolyglotEngineException.java:137)
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotValueDispatch.unsupported(PolyglotValueDispatch.java:1257)
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotValueDispatch.newInstanceUnsupported(PolyglotValueDispatch.java:613)
at org.graalvm.truffle/com.oracle.truffle.polyglot.PolyglotValueDispatch$InteropValue$NewInstanceNode.doCached(PolyglotValueDispatch.java:4382)
...
```
# Important Notes
I thought we had a CI check that would prevent us from introducing such regressions?
EDIT: we don't
`.find(...)` wasn't equivalent to `.map(...).exists(identity)` in this case because of side-effects related to diagnostics reporting. That is why we were reporting errors only for the first erroneous module.
/cc @jdunkerley
c&p failure, so it was generating a very small Vector. Surprisingly, it indicates similar problems with `@ExplodeLoop` in `CatchTypeBranchNode`.
Kudos to @JaroslavTulach for reporting it.
Missing `@TruffleBoundary` annotations and entries in the configs were preventing us from generating native image. Again.
# Important Notes
@mwu-tow This check is easy to miss so it would be good to have it in CI.
Changelog
- fix reporting of runtime type for values annotated with warning
- fix visualizations of values annotated with warnings
- fix `Runtime.get_stack_trace` failure in interactive mode
This change brings by-type pattern matching to Enso.
One can pattern match on Enso types as well as on polyglot types.
For example,
```
case x of
_ : Integer -> ...
_ : Text -> ...
_ -> ...
```
as well as Java's types
```
case y of
_ : ArrayList -> ...
_ : List -> ...
_ : AbstractList -> ...
_ -> ..
```
It is no longer possible to match a value with a corresponding type constructor.
For example
```
case Date.now of
Date -> ...
```
will no longer match and one should match on the type (`_ : Date`) instead.
```
case Date of
Date -> ...
```
is fine though, as requested in the ticket.
The change required further changes to `type_of` logic which wasn't dealing well with polyglot values.
Implements https://www.pivotaltracker.com/story/show/183188846
# Important Notes
~I discovered late in the game that nested patterns involving type patterns, such as `Const (f : Foo) tail -> ...` are not possible due to the old parser logic.
I would prefer to add it in a separate PR because this one is already getting quite large.~ This is now supported!
PR adds special kind of jobs for visualizations. It should
- prevent cancelling visualization jobs when the program is re-executed (resulting in the fact that visualization is not showing up)
- omit unnecessary executions of visualization jobs (the case when the user goes through menu items in the component browser)
I skipped the tests because testing of the last scenario involves indeterminism. I.e. when preparing the test, we can't control which of submitted visualization jobs will be actually executed, and which will be cancelled.
When nodes get invalidated in the cache, they have to be recomputed. Let the IDE know which of the nodes are pending by sending `Api.ExpressionUpdate.Payload.Pending` message.
# Important Notes
This PR introduces new `Api.ExpressionUpdate.Payload.Pending` message. This message is delivered before re-computation of nodes. Later `Api.ExpressionUpdate.Payload.Value` or other is sent to notify the IDE that a value for given node is available.
Trivial implementation of of the `Api.ExpressionUpdate.Payload.Pending` message in the IDE is provided by this PR to (improperly) visualize pending node status - further improvements needed in follow up PRs.
Each builtin `makeFunction` method has to be currently registered for reflection. Adding registration of `SliceVectorMethodGen` to make following example work ag
```bash
enso$ sbt engine-runner-native/buildNativeImage
enso$ ./runner --run engine/runner-native/src/test/resources/Factorial.enso 6
```
# Important Notes
Regression caused by #3724 - Once the above code is executed in the CI, we'll discover such breakages before integration.
Makes statics static. A type and its instances have different methods defined on them, as it should be. Constructors are now scoped in types, and can be imported/exported.
# Important Notes
The method of fixing stdlib chosen here is to just not. All the conses are exported to make all old code work. All such instances are marked with `TODO Dubious constructor export` so that it can be found and fixed.
This change implements a simple `type_of` method that returns a type of a given value, including for polyglot objects.
The change also allows for pattern matching on various time-related instances. It is a nice-to-have on its own, but it was primarily needed here to write some tests. For equality checks on types we currently can't use `==` due to a known _feature_ which essentially does wrong dispatching. This will be improved in the upcoming statics PR so we agreed that there is no point in duplicating that work and we can replace it later.
Also, note that this PR changes `Meta.is_same_object`. Comparing types revealed that it was wrong when comparing polyglot wrappers over the same value.
Use an `ArraySlice` to slice `Vector`.
Avoids memory copying for the slice function.
# Important Notes
| Test | Ref | New |
| --- | --- | --- |
| New Vector | 71.9 | 71.0 |
| Append Single | 26.0 | 27.7 |
| Append Large | 15.1 | 14.9 |
| Sum | 156.4 | 165.8 |
| Drop First 20 and Sum | 171.2 | 165.3 |
| Drop Last 20 and Sum | 170.7 | 163.0 |
| Filter | 76.9 | 76.9 |
| Filter With Index | 166.3 | 168.3 |
| Partition | 278.5 | 273.8 |
| Partition With Index | 392.0 | 393.7 |
| Each | 101.9 | 102.7 |
- Note: the performance of New and Append has got slower from previous tests.
This PR adds a possibility to generate native-image for engine-runner.
Note that due to on-demand loading of stdlib, programs that make use of it are currently not yet supported
(that will be resolved at a later point).
The purpose of this PR is only to make sure that we can generate a bare minimum runner because due to lack TruffleBoundaries or misconfiguration in reflection config, this can get broken very easily.
To generate a native image simply execute:
```
sbt> engine-runner-native/buildNativeImage
... (wait a few minutes)
```
The executable is called `runner` and can be tested via a simple test that is in the resources. To illustrate the benefits
see the timings difference between the non-native and native one:
```
>time built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --no-ir-caches --in-project test/Tests/ --run engine/runner-native/src/test/resources/Factorial.enso 6
720
real 0m4.503s
user 0m9.248s
sys 0m1.494s
> time ./runner --run engine/runner-native/src/test/resources/Factorial.enso 6
720
real 0m0.176s
user 0m0.042s
sys 0m0.038s
```
# Important Notes
Notice that due to a [bug in GraalVM](https://github.com/oracle/graal/issues/4200), which is already fixed in 22.x, and us still being on 21.x for the time being, I had to add a workaround to our sbt build to build a different fat jar for native image. To workaround it I had to exclude sqlite jar. Hence native image task is on `engine-runner-native` and not on `engine-runner`.
Will need to add the above command to CI.
Turns that if you import a two-part import we had special code that would a) add Main submodule b) add an explicit rename.
b) is problematic because sometimes we only want to import specific names.
E.g.,
```
from Bar.Foo import Bar, Baz
```
would be translated to
```
from Bar.Foo.Main as Foo import Bar, Baz
```
and it should only be translated to
```
from Bar.Foo.Main import Bar, Baz
```
This change detects this scenario and does not add renames in that case.
Fixes [183276486](https://www.pivotaltracker.com/story/show/183276486).
IR cache never really took into account a situation when a binding from the imported module has changed. In other words, it would continue to happily use the serialized metadata without noticing that it changed.
This change forces cache invalidation when any of the imported modules was invalidated (or rather not loaded from cache).
# Important Notes
Added simple test infrastructure that simulates file modifications that would trigger the initial cache invalidation.
If they succeed, cache invalidation is propagated thus causing an error.
`Vector` type is now a builtin type. This requires a bunch of additional builtin methods for its creation:
- Use `Vector.from_array` to convert any array-like structure into a `Vector` [by copy](f628b28f5f)
- Use (already existing) `Vector.from_polyglot_array` to convert any array-like structure into a `Vector` **without** copying
- Use (already existing) `Vector.fill 1 item` to create a singleton `Vector`
Additional, for pattern matching purposes, we had to implement a `VectorBranchNode`. Use following to match on `x` being an instance of `Vector` type:
```
import Standard.Base.Data.Vector
size = case x of
Vector.Vector -> x.length
_ -> 0
```
Finally, `VectorLiterals` pass that transforms `[1,2,3]` to (roughly)
```
a1 = 1
a2 = 2
a3 = 3
Vector (Array (a1,a2, a3))
```
had to be modified to generate
```
a1 = 1
a2 = 2
a3 = 3
Vector.from_array (Array (a1, a2, a3))
```
instead to accomodate to the API changes. As of 025acaa676 all the known CI checks passes. Let's start the review.
# Important Notes
Matching in `case` statement is currently done via `Vector_Data`. Use:
```
case x of
Vector.Vector_Data -> True
```
until a better alternative is found.
The goal of this request is to simplify hello world and other trivial Enso programs. No need to learn any standard library functions, enough to write:
```
main = "Hello World!"
```
and the result is going to be printed:
```bash
enso$ ./built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --run hello.enso
'Hello World!'
```
the result is only printed, if it is not `Nothing`. E.g. if the last statement is `IO.print ...` (which returns `Nothing`), then no value is printed at the end by the launcher.
Many engine PRs modify builtins or other engine internals and then they are subject to [incremental CI runtime errors](https://www.pivotaltracker.com/n/projects/2539304/stories/182868680) as outdated `IR` caches from global space at `~/.local/share/enso/cache/ir/Standard/Builtins/0.0.0-dev/0.0.0-dev/Standard/Builtins` are read in.
This PR provides solution for that problem by explicitly defining `IR.Module` `serialVersionUID`. By changing the `serialVersionUID` one prevents previously saved `IR` caches to be loaded into the running process. Change the `serialVersionUID` whenever you see errors caused by reading outdated `IR` caches in the CI.
# Important Notes
Whenever one needs to avoid loading previous `IR` caches, go to `case class IR.Module` and change the `@SerialVersionUID(3692L)` to **number of your pull request**.
Found a bug when accessing keys via `get(constructor)`. Providing a test and a fix.
# Important Notes
Marcin, is it correct that the whole set of members of `End` is: `[head, tail, Int, is_empty, IntList]`? What does `Int` and `IntList` do there? Shall test test check for their presence? **Answer**: rename `Int` and `IntList` to lowercase and yes, then the members shall be there. Done in [ca9f42a](ca9f42a2b8).
