Implementation of https://www.pivotaltracker.com/story/show/184012743https://user-images.githubusercontent.com/919491/214082311-cf49e43c-1d1f-4654-903c-a4224cd954d8.mp4
This is also a step towards more general widget support. The widget metadata is queried using `Meta.get_annotation` method through a dedicated visualization. For now only `Single_Choice` case is handled, and always all suggestions are is returned.
# Important Notes
There are limitations as to which node segments receive a widget. Only chain method calls are supported now (`thing.method` syntax), and only outside of lambda scope. Widgets in lambdas will require support for visualisations of lambda subexpressions, which is currently missing in the engine. The IDE technically tries to place the widgets there, but the data never arrives. It should work once the engine support is added.
This PR includes a mock for `Meta.get_annotation` call that only supports `Table.at` method. Real implementation is a separate task that is already in progress.
JGit ops generally run fast (as in a few milliseconds) except for the first commit. The initialization + first commit was taking at least 3.5 seconds constistenly, but only in the first test case. Now, this led to frequent timeouts down the chain when the request was expected to finish fast.
The bookkeeping involved some timestamping and other expensive calls in order to calculate clock drift. The default appears to be to run it in a blocking mode, hence adding at least 3 seconds to the first command call.
Setting the job to run in the background makes the cost of the repo initialization acceptable (~300 milliseconds on a cold JVM). The other commands are unaffected and take < 10 milliseconds.
# Important Notes
Added a test to ensure that we don't introduce the regression. Marked it as potentially flaky because it uses timestamps and it is therefore prone to random system hiccups.
There is no need for JGit to try to save file attributes to its config in `<home>/jgit/config`. This change ensures that JGit operates on a stub file, without polluting users' configs, similarly to user's git config.
Additionally, noticed that all FS operations during `init` are rather slow and saw at least on one occassion when the handler timed out because of that (https://www.pivotaltracker.com/story/show/184359934). This change provides an init-specific timeout for `vcs/init`.
# Important Notes
This change should reduce chances of failures during VCS initialization.
When a large long would be passed to a host call expecting a double, it would crash with a
```
Cannot convert '<some long>'(language: Java, type: java.lang.Long) to Java type 'double': Invalid or lossy primitive coercion
```
That is unlikely to be expected by users. It also came up in the Statistics examples during Sum. One could workaround it by forcing the conversion manually with `.to_decimal` but it is not a permanent solution.
Instead this change adds a custom type mapping from Long to Double that will do it behind the scenes with no user interaction. The mapping kicks in only for really large longs.
# Important Notes
Note that the _safe_ range is hardcoded in Truffle and it is not accessible in enso packages. Therefore a simple c&p for that max safe long value was necessary.
In order to investigate `engine/language-server` project, I need to be able to open its sources in IGV and NetBeans.
# Important Notes
By adding same Java source (this time `package-info.java`) and compiling with our Frgaal compiler the necessary `.enso-sources*` files are generated for `engine/language-server` and then the `enso4igv` plugin can open them and properly understand their compile settings.
In addition to that this PR enhances the _"logical view"_ presentation of the project by including all source roots found under `src/*/*`.
In the event when the action of the underlying actor crashes really bad, the result of the future will be `Failure`. All such results will be then wrapped in `Status.Failure` via `pipeTo` (unlike `Success` which just forwards the response).
In some cases we don't handle `Status.Failure` messages, meaning we are rather left in the dark to the reason of the failure with a non-informative message `Received unknown message: class akka.actor.Status$Failure`.
I'm keeping this change small on purpose to keep this change self-contained. I think there are more cases but it needs careful investigation into how messages are being sent.
# Important Notes
This PR does not attempt to fix the underlying problem of the ticket, yet. We should however have a better overview where/why things go wrong.
LS needs to notify runtime that modules' sources need to be reloaded from FS, once its own buffer has been reloaded as well.
# Important Notes
Discovered during integration of https://github.com/enso-org/enso/pull/4050.
The test illustrates the problem if we don't reload module's sources - the sources essentially become stale even though they have changed.
Unsetting literal sources during the `text/fileClose` operation would also unlink sources from the actual file. That would be never set back during the reopening of the file, resulting in an NPE during application of changes at a later point.
# Important Notes
Revealed during https://github.com/enso-org/enso/pull/4050 which was the first(?) to do close/open of the file.
The exception showing up in GUI and preventing further actions is now gone.
Potential workaround to line endings problems after `vcs/restore` operation is executed.
# Important Notes
Not really able to reproduce the problem myself so this PR has a lot of _leap of faith_ in it.
VCS restore operation was correctly restoring the state of projects to the requested commit. Unfortunately, after the operation file system was becoming out-of-sync with language server's buffers (and IDE's content versions).
A few important changes are introduced here that complicate the interaction between components:
1) `vcs restore` returns an actual diff between the current state and the
requested commit
2) the response is forwarded to buffer registry first rather than to the client
3) the diff is used to identify appropriate collaborative editors and
notify them about the need to reload buffers from file system
4) all clients of affected open buffers are notified of the change via
`text/didChange` notification. If a file was removed and there were open buffers for it, clients will be notified via `file/event` and editor will be stopped
5) only then the client is notified about a successful restore operation
This PR addresses one of the two problems reported in https://www.pivotaltracker.com/story/show/184097084.
# Important Notes
We need to make sure that IDE correctly responds to `text/didChange` notifications.
Implements `getMetaObject` and related messages from Truffle interop for Enso values and types. Turns `Meta.is_a` into builtin and re-uses the same functionality.
# Important Notes
Adds `ValueGenerator` testing infrastructure to provide unified access to special Enso values and builtin types that can be reused by other tests, not just `MetaIsATest` and `MetaObjectTest`.
Apparently
```
git --git-dir .enso/.vcs log refs/heads/master
fatal: ambiguous argument 'refs/heads/master': unknown revision or path not in the working tree.
```
but
```
git --git-dir .enso/.vcs log HEAD
fatal: ambiguous argument 'refs/heads/master': unknown revision or path not in the working tree.
```
works just fine on Windows.
Added some safeguards to avoid propagating weird errors because of retrieving element from an
empty Option.
The presence of `.git` in Enso's data directory (`.enso`), which should be under VCS as well, turned out to be really problematic. Git (and JGit) treat `.enso` as a submodule which prevents as from manual inspection of the repo. Additionally, every commit changed the submodule version, which in turn was reporting invalid/changed status always.
This change renames `.enso/.git` to `.enso/.vcs` which eliminates a lot of problems. Additionally, rather than doing `git add .` we selectively add files to the index, thus preventing most of the issues reported in the ticket.
This change also means that JGit's status needs to filter `.enso/.vcs` as those will be marked as untracked always.
# Important Notes
To manually test the status of the project's repo do
```
git --git-dir .enso/.vcs status
```
- Moved `Any`, `Error` and `Panic` to `Standard.Base`.
- Separated `Json` and `Range` extensions into own modules.
- Tidied `Case`, `Case_Sensitivity`, `Encoding`, `Matching`, `Regex_Matcher`, `Span`, `Text_Matcher`, `Text_Ordering` and `Text_Sub_Range` in `Standard.Base.Data.Text`.
- Tidied `Standard.Base.Data.Text.Extensions` and stopped it re-exporting anything.
- Tidied `Regex_Mode`. Renamed `Option` to `Regex_Option` and added type to export.
- Tidied up `Regex` space.
- Tidied up `Meta` space.
- Remove `Matching` from export.
- Moved `Standard.Base.Data.Boolean` to `Standard.Base.Boolean`.
# Important Notes
- Moved `to_json` and `to_default_visualization_data` from base types to extension methods.
Rather than hard-coding `.git` in the root of the project, VCS should save data into Enso's data directory (i.e. `.enso`).
This change reshuffles initialization and configuration to store Git VCS metadata by default at `.enso/.git`.
This is follow up to https://github.com/enso-org/enso/pull/3851
# Important Notes
Apparently a custom Git directory in JGit means that it always creates a `.git` **file** with `gitdir` pointing to the custom location.
This is not necessary in our case since all our commands provide that explicitly.
That is why `init` operation removes `.git` file, which may seem a bit counter-intuitive.
Upgrading to GraalVM 22.3.0.
# Important Notes
- Removed all deprecated `FrameSlot`, and replaced them with frame indexes - integers.
- Add more information to `AliasAnalysis` so that it also gathers these indexes.
- Add quick build mode option to `native-image` as default for non-release builds
- `graaljs` and `native-image` should now be downloaded via `gu` automatically, as dependencies.
- Remove `engine-runner-native` project - native image is now build straight from `engine-runner`.
- We used to have `engine-runner-native` without `sqldf` in classpath as a workaround for an internal native image bug.
- Fixed chrome inspector integration, such that it shows values of local variables both for current stack frame and caller stack frames.
- There are still many issues with the debugging in general, for example, when there is a polyglot value among local variables, a `NullPointerException` is thrown and no values are displayed.
- Removed some deprecated `native-image` options
- Remove some deprecated Truffle API method calls.
`LocalDate.now()` returns time in system time zone, and git uses UTC. Sometimes it can cause test to fail
```
INFO ide_ci::program::command: sbtℹ️ [info] - must create a commit with a timestamp *** FAILED ***
INFO ide_ci::program::command: sbtℹ️ [info] "2022-11-18T23:05:01.768499800Z" did not start with substring "2022-11-19" (VcsManagerTest.scala:201)
```
Ignore gitconfig located in the user's home directory when creating Git instance.
For example, I setup git to automatically sign commits. And all git related tests are failing because of that:
```
[info] - should return last X commits *** FAILED ***
[info] org.eclipse.jgit.api.errors.ServiceUnavailableException: Signing service is not available
[info] at org.eclipse.jgit.api.CommitCommand.sign(CommitCommand.java:328)
[info] at org.eclipse.jgit.api.CommitCommand.call(CommitCommand.java:283)
[info] at org.enso.languageserver.vcsmanager.GitSpec$InitialRepoSetup.setup(GitSpec.scala:360)
[info] at org.enso.languageserver.vcsmanager.GitSpec$InitialRepoSetup.setup$(GitSpec.scala:348)
[info] at org.enso.languageserver.vcsmanager.GitSpec$$anon$16.setup(GitSpec.scala:275)
[info] at org.enso.languageserver.vcsmanager.GitSpec$InitialRepoSetup.$init$(GitSpec.scala:346)
[info] at org.enso.languageserver.vcsmanager.GitSpec$$anon$16.<init>(GitSpec.scala:275)
[info] at org.enso.languageserver.vcsmanager.GitSpec.$anonfun$new$35(GitSpec.scala:275)
[info] at org.scalatest.OutcomeOf.outcomeOf(OutcomeOf.scala:85)
[info] at org.scalatest.OutcomeOf.outcomeOf$(OutcomeOf.scala:83)
[info] ...
```
This change adds support for Version Controlled projects in language server.
Version Control supports operations:
- `init` - initialize VCS for a project
- `save` - commit all changes to the project in VCS
- `restore` - ability to restore project to some past `save`
- `status` - show the status of the project from VCS' perspective
- `list` - show a list of requested saves
# Important Notes
Behind the scenes, Enso's VCS uses git (or rather [jGit](https://www.eclipse.org/jgit/)) but nothing stops us from using a different implementation as long as it conforms to the establish API.
When nodes get invalidated in the cache, they have to be recomputed. Let the IDE know which of the nodes are pending by sending `Api.ExpressionUpdate.Payload.Pending` message.
# Important Notes
This PR introduces new `Api.ExpressionUpdate.Payload.Pending` message. This message is delivered before re-computation of nodes. Later `Api.ExpressionUpdate.Payload.Value` or other is sent to notify the IDE that a value for given node is available.
Trivial implementation of of the `Api.ExpressionUpdate.Payload.Pending` message in the IDE is provided by this PR to (improperly) visualize pending node status - further improvements needed in follow up PRs.
This change adds Autosave action for open buffers. The action is scheduled
after every edit request and is cancelled by every explicit save file request, if
necessary. Successful autosave also notifies any active clients of the buffer.
Related to https://www.pivotaltracker.com/story/show/182721656
# Important Notes
WIP
PR allows to attach metod pointers as a visualization expressions. This way it allows to attach a runtime instrument that enables caching of intermediate expressions.
# Important Notes
ℹ️ API is backward compatible.
To attach the visualization with caching support, the same `executionContext/attachVisualisation` method is used, but `VisualisationConfig` message should contain the message pointer.
While `VisualisationConfiguration` message has changed, the language server accepts both new and old formats to keep visualisations working in IDE.
#### Old format
```json
{
"executionContextId": "UUID",
"visualisationModule": "local.Unnamed.Main",
"expression": "x -> x.to_text"
}
```
#### New format
```json
{
"executionContextId": "UUID",
"expression": {
"module": "local.Unnamed.Main",
"definedOnType": "local.Unnamed.Main",
"name": "encode"
}
}
```
In order to analyse why the `runner.jar` is slow to start, let's _"self sample"_ it using the [sampler library](https://bits.netbeans.org/dev/javadoc/org-netbeans-modules-sampler/org/netbeans/modules/sampler/Sampler.html). As soon as the `Main.main` is launched, the sampling starts and once the server is up, it writes its data into `/tmp/language-server.npss`.
Open the `/tmp/language-server.npss` with [VisualVM](https://visualvm.github.io) - you should have one copy in your
GraalVM `bin/jvisualvm` directory and there has to be a GraalVM to run Enso.
#### Changelog
- add: the `MethodsSampler` that gathers information in `.npss` format
- add: `--profiling` flag that enables the sampler
- add: language server processes the updates in batches
This PR replaces hard-coded `@Builtin_Method` and `@Builtin_Type` nodes in Builtins with an automated solution
that a) collects metadata from such annotations b) generates `BuiltinTypes` c) registers builtin methods with corresponding
constructors.
The main differences are:
1) The owner of the builtin method does not necessarily have to be a builtin type
2) You can now mix regular methods and builtin ones in stdlib
3) No need to keep track of builtin methods and types in various places and register them by hand (a source of many typos or omissions as it found during the process of this PR)
Related to #181497846
Benchmarks also execute within the margin of error.
### Important Notes
The PR got a bit large over time as I was moving various builtin types and finding various corner cases.
Most of the changes however are rather simple c&p from Builtins.enso to the corresponding stdlib module.
Here is the list of the most crucial updates:
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/builtin/Builtins.java` - the core of the changes. We no longer register individual builtin constructors and their methods by hand. Instead, the information about those is read from 2 metadata files generated by annotation processors. When the builtin method is encountered in stdlib, we do not ignore the method. Instead we lookup it up in the list of registered functions (see `getBuiltinFunction` and `IrToTruffle`)
- `engine/runtime/src/main/java/org/enso/interpreter/runtime/callable/atom/AtomConstructor.java` has now information whether it corresponds to the builtin type or not.
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/RuntimeStubsGenerator.scala` - when runtime stubs generator encounters a builtin type, based on the @Builtin_Type annotation, it looks up an existing constructor for it and registers it in the provided scope, rather than creating a new one. The scope of the constructor is also changed to the one coming from stdlib, while ensuring that synthetic methods (for fields) also get assigned correctly
- `engine/runtime/src/main/scala/org/enso/compiler/codegen/IrToTruffle.scala` - when a builtin method is encountered in stdlib we don't generate a new function node for it, instead we look it up in the list of registered builtin methods. Note that Integer and Number present a bit of a challenge because they list a whole bunch of methods that don't have a corresponding method (instead delegating to small/big integer implementations).
During the translation new atom constructors get initialized but we don't want to do it for builtins which have gone through the process earlier, hence the exception
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor not only generates the actual code fpr nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/MethodProcessor.java` - @Builtin_Method processor no longer generates only (root) nodes but also collects and writes the info about them (name, class, params) to a metadata file that is read during builtins initialization
- `lib/scala/interpreter-dsl/src/main/java/org/enso/interpreter/dsl/TypeProcessor.java` - Similar to MethodProcessor but handles @Builtin_Type annotations. It doesn't, **yet**, generate any builtin objects. It also collects the names, as present in stdlib, if any, so that we can generate the names automatically (see generated `types/ConstantsGen.java`)
- `engine/runtime/src/main/java/org/enso/interpreter/node/expression/builtin` - various classes annotated with @BuiltinType to ensure that the atom constructor is always properly registered for the builitn. Note that in order to support types fields in those, annotation takes optional `params` parameter (comma separated).
- `engine/runtime/src/bench/scala/org/enso/interpreter/bench/fixtures/semantic/AtomFixtures.scala` - drop manual creation of test list which seemed to be a relict of the old design
A draft of simple changes to the compiler to expose sum type information. Doesn't break the stdlib & at the same time allows for dropdowns. This is still broken, for example it doesn't handle exporting/importing types, only ones defined in the same module as the signature. Still, seems like a step in the right direction – please provide feedback.
# Important Notes
I've decided to make the variant info part of the type, not the argument – it is a property of the type logically.
Also, I've pushed it as far as I'm comfortable – i.e. to the `SuggestionHandler` – I have no idea if this is enough to show in IDE? cc @4e6
Changelog:
- fix: `search/completion` request with the position parameter.
- fix: `refactoring/renameProject` request. Previously it did not take into account the library namespace (e.g. `local.`)
Result of automatic formatting with `scalafmtAll` and `javafmtAll`.
Prerequisite for https://github.com/enso-org/enso/pull/3394
### Important Notes
This touches a lot of files and might conflict with existing PRs that are in progress. If that's the case, just run
`scalafmtAll` and `javafmtAll` after merge and everything should be in order since formatters should be deterministic.
Changelog:
- add: component groups to package descriptions
- add: `executionContext/getComponentGroups` method that returns component groups of libraries that are currently loaded
- doc: cleanup unimplemented undo/redo commands
- refactor: internal component groups datatype
PR adds a monitor that handles messages between the language server and the runtime and dumps them as a CSV file `/tmp/enso-api-events-*********.csv`
```
UTC timestamp,Direction,Request Id,Message class
```
# Important Notes
⚠️ Monitor is enabled when the log level is set to trace. You should pass `-vv` (very verbose) option to the backend when starting IDE
```
enso -- -vv
```
This changes intends to cleanup some directories that are being left
behind after running `sbt test`:
- a random `foobar` directory was being created in the `engine` project
directory
- every run of a test suite would add more temporary directories in `/tmp`
The change does not make use of `deleteOnExit` which can pretty
unreliable. Instead we recursively delete files in directories and
directories to make sure nothing is left behind.
