mirror of
https://github.com/enso-org/enso.git
synced 2024-11-24 00:27:16 +03:00
e529f7bb3b
Request Timeouts started plaguing IDE due to numerous `executionContext/***Visualization` requests. While caused by a bug they revealed a bigger problem in the Language Server when serving large amounts of requests: 1) Long and short lived jobs are fighting for various locks. Lock contention leads to some jobs waiting for a longer than desired leading to unexpected request timeouts. Increasing timeout value is just delaying the problem. 2) Requests coming from IDE are served almost instantly and handled by various commands. Commands can issue further jobs that serve request. We apparently have and always had a single-thread thread pool for serving such jobs, leading to immediate thread starvation. Both reasons increase the chances of Request Timeouts when dealing with a large number of requests. For 2) I noticed that while we used to set the `enso-runtime-server.jobParallelism` option descriptor key to some machine-dependent value (most likely > 1), the value set would **only** be available for instrumentation. `JobExecutionEngine` where it is actually used would always get the default, i.e. a single-threaded ThreadPool. This means that this option descriptor was simply misused since its introduction. Moved that option to runtime options so that it can be set and retrieved during normal operation. Adding parallelism intensified problem 1), because now we could execute multiple jobs and they would compete for resources. It also revealed a scenario for a yet another deadlock scenario, due to invalid order of lock acquisition. See `ExecuteJob` vs `UpsertVisualisationJob` order for details. Still, a number of requests would continue to randomly timeout due to lock contention. It became apparent that `Attach/Modify/Detach-VisualisationCmd` should not wait until a triggered `UpsertVisualisationJob` sends a response to the client; long and short lived jobs will always compete for resources and we cannot guarantee that they will not timeout that way. That is why the response is sent immediately from the command handler and not from the job executed after it. This brings another problematic scenario: 1. `AttachVisualisationCmd` is executed, response sent to the client, `UpsertVisualisationJob` scheduled. 2. In the meantime `ModifyVisualisationCmd` comes and fails; command cannot find the visualization that will only be added by `UpsertVisualisationJob`, which might have not yet been scheduled to run. Remedied that by checking visualisation-related jobs that are still in progress. It also allowed for cancelling jobs which results wouldn't be used anyway (`ModifyVisualisationCmd` sends its own `UpsertVisualisationJob`). This is not a theoretical scenario, it happened frequently on IDE startup. This change does not fully solve the rather problematic setup of numerous locks, which are requested by short and long lived jobs. A better design should still be investigated. But it significantly reduces the chances of Request Timeouts which IDE had to deal with. With this change I haven't been able to experience Request Timeouts for relatively modest projects anymore. I added the possibility of logging wait times for locks to better investigate further problems. Closes #7005 |
||
|---|---|---|
| .. | ||
| interpreter-dsl-test/src/test/java/org/enso/interpreter/dsl/test | ||
| language-server | ||
| launcher/src | ||
| polyglot-api/src | ||
| runner | ||
| runtime | ||
| runtime-instrument-common/src | ||
| runtime-instrument-id-execution/src/main/java/org/enso/interpreter/instrument | ||
| runtime-instrument-repl-debugger/src/main/java/org/enso/interpreter/instrument | ||
| runtime-instrument-runtime-server/src/main/java/org/enso/interpreter/instrument | ||
| runtime-language-epb/src/main/java/org/enso/interpreter/epb | ||
| runtime-with-instruments/src/test | ||
| runtime-with-polyglot/src/test | ||
| README.md | ||
The Enso Engine
The Enso engine is the codebase responsible for compiling and executing Enso code, as well as providing language server functionality to users of the language. It is subdivided into two major components:
- Language Server: The Enso language service.
- Polyglot API: The truffle-boundary safe API for communication between the language server and the runtime.
- Runner: The command-line interface for Enso.
- Runtime: The compiler and interpreter for Enso.