enso/docs/distribution/launcher.md

layout	title	category	tags	order
developer-doc	Enso Launcher	distribution	distribution launcher	4

Enso Launcher

The launcher is used to run Enso commands (like the REPL, language server etc.) and seamlessly manage Enso versions. This document describes it's features. Its command-line interface is described in the CLI document.

• Launcher Distribution
  • Using Multiple Launcher Versions Side-By-Side
  • Detecting Portable Distribution
• Launcher Build
  • Portability
• Project Management
  • Creating a Project
  • Per-Project Enso Version
  • Project Configuration
• Enso and Graal Version Management
  • GraalVM Override
  • Downloading Enso Releases
  • Downloading GraalVM Releases
• Running Enso Components
  • Running Plugins
• Global User Configuration
• Updating the Launcher
  • Minimal Required Launcher Version
  • Step-by-Step Upgrade
  • Downloading Launcher Releases

Launcher Distribution

The launcher is distributed as a native binary for each platform (Windows, Linux, macOS). It is distributed in a ZIP archive as described in Enso Distribution Layout in two flavors - as packages containing just the launcher binary that can then download and install desired versions of the engine and as bundles that already contain the latest version of Enso engine and Graal runtime corresponding to it.

Using Multiple Launcher Versions Side-By-Side

Multiple portable distributions of the launcher can be used side-by-side. To use multiple installed distributions, some tricks are necessary - before launching a different version, the environment variables ENSO_DATA_DIRECTORY, ENSO_CONFIG_DIRECTORY and ENSO_BIN_DIRECTORY have to be set to directories corresponding to that version.

Detecting Portable Distribution

As described in Enso Distribution Layout, the launcher can either be run in a portable distribution or installed locally. The launcher must detect if its run as the portable or installed distribution. When run, the launcher checks if it is placed in a directory called bin and checks the parent directory for a file called .enso.portable. If such file is found, the launcher runs in portable mode. Otherwise, it runs in installed mode.

Launcher Build

The launcher is built using GraalVM Native Image which compiles the JVM code into a native binary ahead of time, resulting in a small and fast launching executable.

Portability

On Linux, it is possible to statically link all libraries required by the Native Image, thus ensuring portability between Linux distributions.

On Windows and macOS, it is not possible to statically link against system libraries, but this should not hinder portability as the system libraries are generally compatible between distribution versions on these platforms. Non-system dependencies are included in the binary on these platforms as well.

Project Management

The launcher provides basic project management utilities for the command-line user.

Creating a Project

It allows to create an empty project in a specified directory with the basic configuration based on user's config.

Per-Project Enso Version

Project configuration must specify the exact Enso version that should be used inside that project. The launcher automatically detects if it is in a project (by traversing the directory structure). The current project can also be specified by the --path parameter. All components launched inside a project use the version specified in the project configuration, or outside a project, the default version.

The actionables for this section are:

• Decide how to support inexact project bounds (like >=3.1, <4 - when should the launcher check for new versions) and resolvers.
• Decide how to support nightly builds.

Enso and Graal Version Management

The launcher automatically manages required Enso versions. When running inside a project, if the version specified in project configuration is not installed, it is installed automatically.

If an Enso version is no longer needed, it can be removed with the uninstall command.

Moreover, GraalVM distributions tied to the installed Enso versions are managed automatically by the launcher. When a new Enso version is installed, it also ensures that the correct GraalVM version is installed and that it is used for launching this version of Enso. When a managed GraalVM distribution is no longer used by any installed version of Enso, it is automatically removed.

GraalVM Override

While the launcher manages its own installation of GraalVM to ensure that the right JVM version is used to launch each version of Enso, the user can override this mechanism to use the installed system JVM instead. This is an advanced feature and should rarely be used.

Downloading Enso Releases

The releases are discovered and downloaded using the GitHub API. As described in the Release Policy, each release contains a manifest file that is downloaded first. It specifies if this Enso version can be used with the current launcher or an upgrade is needed, as described in Minimal Required Launcher Version. If the version is correct, the binary file containing the Enso components distribution is downloaded. The manifest also specifies which GraalVM version should be used with this version of Enso. If that version of GraalVM is not present on the system it is also downloaded and installed.

Releases marked as broken are ignored by the launcher unless it is specified by an exact version match. In that case it is downloaded, but a warning is printed.

Downloading GraalVM Releases

GraalVM is downloaded from its GitHub releases page using GitHub API, similarly as Enso releases.

Running Enso Components

The primary purpose of the launcher is running various Enso components, namely the REPL, running a project, Enso scripts or the language server.

The launcher automatically infers which Enso version to used, based on the parameters and configuration:

• When running a project or the language server, the version specified in project configuration is used.
• When running the REPL, if the current directory is inside a project, the project version is used, otherwise the default version from the global configuration is used. The current path is the working directory unless overridden with the --path parameter.
• When running an Enso script, if that script is located inside a project, the project version is used, if it is outside a project, the default version is used.

Additional arguments passed to the launcher are forwarded to the launched component. Moreover, options for the JVM that is used to run the components can also be provided, as described in JVM Options.

Running Plugins

If the launcher gets an unknown command foo, it tries to run enso-foo and pass all the arguments that follow. If enso-foo is not found, it fails as normal. This can be used to implement plugins that are launched through the universal launcher. For example, the Enso IDE can provide an enso-ide executable, allowing users to launch the IDE by typing enso ide.

For a plugin to be recognized by the launcher, it needs to support a --synopsis option - running enso-foo --synopsis should print a short description and return with exit code 0, for the plugin to be considered supported. That description will be included in the command listing printed by enso help.

Testing plugins

When testing the launcher, we want to test plugin discovery. To do so, we override the PATH of the tested launcher to a directory containing prepared plugins. On Windows, the environment variables are usually treated as case-insensitive but not all the time. When launching a process with an added environment variable called PATH, that process actually has two variables in its environment - the original Path and the overriden PATH. This can be seen when querying System.getenv() - the returned map contains both Path and PATH with their respective values. However, System.getenv("PATH") uses some platform specific logic, and even if both variables are present in the environment, it actually returns the value corresponding to Path. This is likely the expected behaviour on Windows. So to successfully override the system path on Windows, we need to override Path, not PATH like on Unix-based systems.

Global User Configuration

The launcher allows to edit global user configuration, saved in the config directory inside the Enso distribution structure.

This configuration specifies the default Enso version used outside of projects and used for creating new projects. It also specifies default project metadata used when creating a project with the new command.

Updating the Launcher

Besides managing Enso versions, the launcher has the ability to also update itself. By default it is updated to the latest version available, but it also allows downgrades by specifying a version explicitly.

Minimal Required Launcher Version

Each version of Enso can specify the minimum version of launcher required to run it. This version is specified in a manifest file that should be included as an artifact for every Enso release.

Moreover, if a given project uses some new build features, it may require a newer version of the launcher. Thus, project configuration can also specify a minimal required launcher version.

If the launcher detects that the installed version is older than one of the two criteria above, it offers to automatically upgrade to the latest version and re-run the current command.

Step-by-Step Upgrade

It is possible that in the future, new launcher versions will require some additional logic when upgrading that has not currently been considered. To maintain future-compatibility, each launcher version can define in its manifest a minimum launcher version that can be used to upgrade to it. Any new upgrade logic can then be introduced gradually (by first releasing a new version which knows this new logic but does not require it and later releasing another version that can require this new logic). In that case, updates are performed step-by-step - first this new version that does not require new logic is downloaded and it is used to upgrade to the new version which requires the new logic. If necessary, such upgrade steps can be chained.

The step-by-step upgrade logic is implemented by checking the minimum-version-for-upgrade property in the new launcher's manifest. If the current version is greater or equal to that version, the upgrade proceeds normally. Otherwise, the launcher tries to upgrade to this minimum version (or the closest to it newer non-broken version), recursively - i.e. if this upgrade also cannot be performed directly, it is also performed step-by-step in the same way.

Testing Step-by-Step Upgrade

To test the multi-step upgrade we need multiple launcher executables that report different versions. As building the launcher takes a substantial amount of time and it reports by default the version from build information, we created a simple wrapper in Rust which runs the original launcher executable with additional internal options that tell it to override its version. These options are only available in a development build. Similarly, internal options are used to override the default GitHub repository to a local filesystem based repository for launcher releases, as we want to avoid any network connectivity in tests.

Downloading Launcher Releases

The launcher is released alongside Enso, so each new release of Enso also contains native artifacts for the launcher for each platform. They are downloaded in the same way as Enso distribution. The launcher does not have to be updated as often as Enso itself - it only has to be updated when a project or a new Enso version requires a more recent launcher or the user explicitly wants to.

Fallback Method

To ensure that the launcher can be safely updated even if the distribution scheme changes, there should be support for a fallback upgrade scheme that is used if the default upgrade process fails.

This fallback scheme is only intended for situations where the current default scheme is broken indefinitely. So for simplicity, it does not allow to choose an arbitrary version but only to upgrade to the latest version of the launcher. In the very rare case in which the user wants to downgrade after the default distribution scheme has changed, they have to first upgrade to the latest version of the launcher which will use a new distribution scheme. Then, on that latest version, it may be possible to downgrade back to an old version (which is distributed on the new distribution scheme).

Thus, when migrating to a new distribution scheme, old versions should also be preserved, but the fallback upgrade scheme does not have to keep track of all the versions, but only the latest one.

It can be implemented by uploading the most recent artifacts to some fixed domain, like launcherupgrade.release.enso.org.