* Enable log-to-file configuration
PR #7825 enabled parallel logging to a file with a much more
fine-grained log level by default.
However, logging at `TRACE` level on Windows appears to be still
problematic.
This PR reduced the default log level to file from `DEBUG` to `TRACE`
and allows to control it via an environment variable if one wishes to
change the verbosity without making code changes.
* PR comments
close#7750close#7834
Changelog:
- update: project manager uses the packaged language server to open projects
- fix: remove stack traces from connection errors on initial ping handler request (when the language server is booting)
- update: add engine and edition versions to the `initProtocolConnection` response for easier debug
- update: do not resolve project ensoVersion in the `project/list` to eliminate unnecessary network calls
* Always log verbose to a file
The change adds an option by default to always log to a file with
verbose log level.
The implementation is a bit tricky because in the most common use-case
we have to always log in verbose mode to a socket and only later apply
the desired log levels. Previously socket appender would respect the
desired log level already before forwarding the log.
If by default we log to a file, verbose mode is simply ignored and does
not override user settings.
To test run `project-manager` with `ENSO_LOGSERVER_APPENDER=console` env
variable. That will output to the console with the default `INFO` level
and `TRACE` log level for the file.
* add docs
* changelog
* Address some PR requests
1. Log INFO level to CONSOLE by default
2. Change runner's default log level from ERROR to WARN
Took a while to figure out why the correct log level wasn't being passed
to the language server, therefore ignoring the (desired) verbose logs
from the log file.
* linter
* 3rd party uses log4j for logging
Getting rid of the warning by adding a log4j over slf4j bridge:
```
ERROR StatusLogger Log4j2 could not find a logging implementation. Please add log4j-core to the classpath. Using SimpleLogger to log to the console...
```
* legal review update
* Make sure tests use test resources
Having `application.conf` in `src/main/resources` and `test/resources`
does not guarantee that in Tests we will pick up the latter. Instead, by
default it seems to do some kind of merge of different configurations,
which is far from desired.
* Ensure native launcher test log to console only
Logging to console and (temporary) files is problematic for Windows.
The CI also revealed a problem with the native configuration because it
was not possible to modify the launcher via env variables as everything
was initialized during build time.
* Adapt to method changes
* Potentially deal with Windows failures
* Improve shutdown logic of language server
This PR addresses problems mentioned in #7470 and #7729:
- shutting a language server explicitly will not lead to a soft shutdown
- `project/status` endpoint returns the state of the language server
`LanguageServerController` now also signed up for `ClientConnect`
messages. For it to be unambiguous, we need to carry around the port
number of the language server as a way of identifying the right one.
One can now use `project/status` to additionally determine the state of
the language server.
Also relies on a proper fix for #7765.
* changelog
* PR comments
This change replaces Enso's custom logger with an existing, mostly off the shelf logging implementation. The change attempts to provide a 1:1 replacement for the existing solution while requiring only a minimal logic for the initialization.
Loggers are configured completely via `logging-server` section in `application.conf` HOCON file, all initial logback configuration has been removed. This opens up a lot of interesting opportunities because we can benefit from all the well maintained slf4j implementations without being to them in terms of functionality.
Most important differences have been outlined in `docs/infrastructure/logging.md`.
# Important Notes
Addresses:
- #7253
- #6739
When building the dev IDE version with `./run ide build`, it creates the engine distribution with version `2023.2.1-dev`. This causes the version check to fail preventing opening the projects.
- Closes#6730
- Changes config to allow environment variables to override server host and port
- Adds port scanning to Electron app to ensure the PM is started at a free port
# Important Notes
- `SERVER_PORT=abcd enso.AppImage` does NOT work. It would not be difficult to implement, but it probably needs discussion on how exactly it should be implemented - for example, `SERVER_PORT` is quite a generic name, should the Electron app pass though something like `ENSO_PM_SERVER_PORT` to the PM as `SERVER_PORT` instead?
⚠️ Port scanning is *only* implemented in the JS frontend. It is not implemented:
- In Scala, because the JS/Rust code calling it needs to know the port as well. There shouldn't be any problems with adding port scanning though, if that's desired
- In Rust, because I'm not sure parsing the host and port from a string is a good idea.
- (This also applies to JS, but it *must* work in JS, and port scanning is already a dependency there so it's quite a bit easier)
- QA *will* need a new PM (`sbt buildProjectManagerDistribution` or `./run backend sbt` -> `buildProjectManagerDistribution`), and the path must be supplied as: `-engine.project-manager-path=path/to/new/pm/here`
Looks like that
```
[info] [2023-08-23T13:12:58.119Z] [org.enso.projectmanager.boot.ProjectManager$] Starting Enso Project Manager
Version: 0.0.0-dev
Built with: scala-2.13.11 for GraalVM 17.0.7
Built from: wip/db/fix-dev-version-check* @ 52bc6b8fcf
Built on: Linux (amd64)
```
close#7604
After moving the rename action to the dashboard, IDE is unaware of the new project name. PR implements a new `refactoring/projectRenamed` notification that is sent from the server to clients and informs them about the changed project name.
# Important Notes
https://github.com/enso-org/enso/assets/357683/7c62726d-217e-4e69-8e48-568e0b7b8c34
close#7345#7254 introduced a delayed shutdown timeout. The `LanguageServerGateway` timeout should include the delayed shutdown time to prevent false timeouts.
- Previous GraalVM update: https://github.com/enso-org/enso/pull/6750
Removed warnings:
- Remove deprecated `ConditionProfile.createCountingProfile()`.
- Add `@Shared` to some `@Cached` parameters (Truffle now emits warnings about potential `@Share` usage).
- Specialization method names should not start with execute
- Add limit attribute to some specialization methods
- Add `@NeverDefault` for some cached initializer expressions
- Add `@Idempotent` or `@NonIdempotent` where appropriate
BigInteger and potential Node inlining are tracked in follow-up issues.
# Important Notes
For `SDKMan` users:
```
sdk install java 17.0.7-graalce
sdk use java 17.0.7-graalce
```
For other users - download link can be found at https://github.com/graalvm/graalvm-ce-builds/releases/tag/jdk-17.0.7
Release notes: https://www.graalvm.org/release-notes/JDK_17/
R component was dropped from the release 23.0.0, only `python` is available to install via `gu install python`.
* Delay LS shutdown when last client disconnects
Rather than closing Language Server immediately, we delay the shutdown
until some timeout hits. This gives a chance for new clients to connect
without paying the price of the initialization again.
More importantly, during hibernation/restart, the connection between
client (IDE) and LS is severed so it appears as if client disconnect. In
fact a few moments later IDE would attempt to re-establish the
connection on the same port. Without this change, LS shutsdown and
further attempts to connect on that particular port will fail.
There are still problems on the IDE-side after waking up from
hibernation but it is not related to Language Server.
* Introduce a separate timeout for delayed shutdown
Can't/shouldn't use the same timeout value as for shutdown timeout for
delaying shutdowns initiated by lack of clients.
* Add test demonstrating the new functionality
The current instructions to _build, use and debug_ `project-manager` and its engine/ls process are complicated and require a lot of symlinks to properly point to each other. This pull requests simplifies all of that by introduction of `ENSO_ENGINE_PATH` and `ENSO_JVM_PATH` environment variables. Then it hides all the complexity behind a simple _sbt command_: `runProjectManagerDistribution --debug`.
# Important Notes
I decided to tackle this problem as I have three repositories with different branches of Enso and switching between them requires me to mangle the symlinks. I hope I will not need to do that anymore with the introduction of the `runProjectManagerDistribution` command.
close#7194
Changelog:
- add: `/projects/{project_id}/enso_project` HTTP endpoint returning an `.enso-project` archive structure
- update: archive enso project to a `.enso-project` `.tar.gz` archive
- update: make project `path` a required field
As discovered in #7224, Json RPC protocol was added to the asynchronous resource initialization stage, as part of #6306, but was not in fact initialized at that point.
Instead it was initialized when the server was started to be able to serve correctly the initialization messages. A classic Catch-22. It was really hard to discover this just by looking at the code, but the profiling clearly showed where the time was spent.
This change splits Language Server's protocol into two:
- the first one accepts `heartbeat/init` and `session/initProtocolConnection`
- the second one enriches it with the full set of supported messages
This shifts the initialization from blocking for 0.5 sec to only ~30ms, and performing the second stage asynchronously.
Closes#7224.
# Important Notes
Before the change (blocking server startup):
After the change (1st stage):
After the change (2nd, asynchronous initialization, stage):
It is relatively easy to reach timeouts on weak systems for startup
operations on project manager. Once a timeout is reached, startup will
not proceed any further.
This PR is a bit of an experiment. It adds adds timeout retries to give a bit of a leeway to
under-powered machines and to log some progress on the way, so that we
know that certain actions are still in progress.
Package's config information, once loaded, never changed. While there is typically no need for it, this was problematic when the config became out-of-sync with the filesystem, like in the case of project rename action.
In rename, the config's properties would be updated in the FS, but that would never be reflected in module's package. Therefore further compilations would continue to ask for the old namespace.
Most of the changes are cosmetic (s/`.config`/`.getConfig()`) except for the new `reloadConfig` method on `Package` that is being called in `RenameProjectCmd` handler.
Closes#7062.
# Important Notes
The reported `ExecutionFailed` error should have been mostly fixed already via #7143. This change makes sure that all the related warnings are gone as well and the compiler uses the updated namespace.
Fixes#6787
# Important Notes
I can't get Project Manager compilation to work locally so I guess I'll be relying on CI to verify that it's working correctly?
Of course, QA should be able to catch any problems too - the websocket API hasn't been changed so it should work out of the box with the current dashboard.
This change fixes the rather elusive bug where shutdown hooks could not be fired when shutdown was taking too long and termination was forced.
Under the circumstances described in detail in ticket #6515 there was a small chance that we could have a shutdown race condition. Essentially the messages received when client was disconnected and language server forced the termination could lead to language server not sending the public `ProjectClosed` message which triggers shutdown hook. Now we always do.
Also made sure that multiple `ProjectClosed` messages don't lead to firing multiple shutdown hooks, which was another possibility.
No tests as one would have to be able to introduce different delays in various message handlers to simulate the problem.
Having ability to do such chaos testing would be nice but it is beyond the scope of this ticket.
I was able to reproduce the problem 100% with my specially crafted setup so I'm fairly confident about the change.
Closes#6515.
close#6611
Changelog:
- update: run compiler passes on the `ascribedType` field of the constructor arguments
- update: suggestion builder uses the type information attached to `ascribedType`
- feat: resolve qualified names in type signatures
Dead Letter logging is occasionally flooding our logs which is confusing to users reporting bugs. Left the possibility of a single report so that we know that something is happening.
close#6254
Changelog:
- fix: race when the actor system may be stopped before the shutdown hooks are executed
- fix: project management spec
- fix: recover from `readLine` failure during the shutdown
close#5881
Changelog
- add: ProtocolFactory object that initializes and returns the protocol object
- update: Add protocol initialization to the initialization component
The `logAvailableComponentsForDebugging` will check and install all necessary components of GraalVM for every mentioned version. While not harmful, it adds up to startup time.
Additionally added an option in language server startup to skip installation of GraalVM components. The latter is already performed by project-manager when opening the project and it is unnecessary to do it twice. Due to LS' architecture this configuration has to be passed around via multiple configs.
Finally, skipped the attempt to install Python component on Windows - this is not supported by GraalVM atm.
Closes#5749.
# Important Notes
The impact of this problem could be really felt the more versions of Enso and GraalVM one had since it would go through all of them.
close#5070
Changelog:
- Include the original exception to log expressions
- Enable logging of Akka Actors' lifecycle events on debug logging level
- Decrease the severity of interruption log messages because interruptions are part of the workflow. The computation can be interrupted at any time, and still be recomputed after. Warnings are just misleading in this case.
In order to investigate `engine/language-server` project, I need to be able to open its sources in IGV and NetBeans.
# Important Notes
By adding same Java source (this time `package-info.java`) and compiling with our Frgaal compiler the necessary `.enso-sources*` files are generated for `engine/language-server` and then the `enso4igv` plugin can open them and properly understand their compile settings.
In addition to that this PR enhances the _"logical view"_ presentation of the project by including all source roots found under `src/*/*`.
In order to analyse why the `runner.jar` is slow to start, let's _"self sample"_ it using the [sampler library](https://bits.netbeans.org/dev/javadoc/org-netbeans-modules-sampler/org/netbeans/modules/sampler/Sampler.html). As soon as the `Main.main` is launched, the sampling starts and once the server is up, it writes its data into `/tmp/language-server.npss`.
Open the `/tmp/language-server.npss` with [VisualVM](https://visualvm.github.io) - you should have one copy in your
GraalVM `bin/jvisualvm` directory and there has to be a GraalVM to run Enso.
#### Changelog
- add: the `MethodsSampler` that gathers information in `.npss` format
- add: `--profiling` flag that enables the sampler
- add: language server processes the updates in batches
Result of automatic formatting with `scalafmtAll` and `javafmtAll`.
Prerequisite for https://github.com/enso-org/enso/pull/3394
### Important Notes
This touches a lot of files and might conflict with existing PRs that are in progress. If that's the case, just run
`scalafmtAll` and `javafmtAll` after merge and everything should be in order since formatters should be deterministic.
