Simplify onboarding instructions for engine & project-manager developers (#7181)

The current instructions to _build, use and debug_ `project-manager` and its engine/ls process are complicated and require a lot of symlinks to properly point to each other. This pull requests simplifies all of that by introduction of `ENSO_ENGINE_PATH` and `ENSO_JVM_PATH` environment variables. Then it hides all the complexity behind a simple _sbt command_: `runProjectManagerDistribution --debug`.

# Important Notes
I decided to tackle this problem as I have three repositories with different branches of Enso and switching between them requires me to mangle the symlinks. I hope I will not need to do that anymore with the introduction of the `runProjectManagerDistribution` command.

build.sbt

@@ -2223,7 +2223,7 @@ buildEngineDistributionNoIndex := {
 }
 
 lazy val runEngineDistribution =
-  inputKey[Unit]("Run the engine distribution with arguments")
+  inputKey[Unit]("Run or --debug the engine distribution with arguments")
 runEngineDistribution := {
   buildEngineDistribution.value
   val args: Seq[String] = spaceDelimited("<arg>").parsed
@@ -2234,6 +2234,22 @@ runEngineDistribution := {
   )
 }
 
+lazy val runProjectManagerDistribution =
+  inputKey[Unit](
+    "Run or --debug the project manager distribution with arguments"
+  )
+runProjectManagerDistribution := {
+  buildEngineDistribution.value
+  buildProjectManagerDistribution.value
+  val args: Seq[String] = spaceDelimited("<arg>").parsed
+  DistributionPackage.runProjectManagerPackage(
+    engineDistributionRoot.value,
+    projectManagerDistributionRoot.value,
+    args,
+    streams.value.log
+  )
+}
+
 val allStdBitsSuffix = List("All", "AllWithIndex")
 val stdBitsProjects =
   List(

docs/CONTRIBUTING.md

@@ -669,125 +669,82 @@ Hello, World!
 You can start [IDE](https://github.com/enso-org/enso/tree/develop/gui) with a
 development version of the language server. IDE executable has
 `--external-backend` flag that switches off the bundled backend. That requires
-you to run the project manager process yourself. You can either get a project
+you to run the project manager process yourself. Running development version of
-manager from one of the latest releases on
+the IDE is also possible via the `./run` script in the root of the repository:
-[GitHub](https://github.com/enso-org/enso/releases), or build one using SBT
-`buildProjectManagerDistribution` command.
-
-Running development version of the IDE is possible via the `./run` script in the
-root of the repository:
 
 ```bash
-$ ./run ide start --wasm-profile dev --external-backend
+enso$ ./run gui watch --skip-wasm-opt
 ```
 
-##### Bash
+To build the `project-manager` one needs to launch `sbt` - one way to do it is
+to execute `./run backend sbt`. When in the _sbt prompt_ one can request
+compilation of the `project-manager`:
 
 ```bash
-sbt buildProjectManagerDistribution
+sbt:enso> buildProjectManagerDistribution
-```
-
-##### PowerShell
-
-```powershell
-sbt.bat buildProjectManagerDistribution
 ```
 
 When the command is completed, a development version of the project manager will
 have appeared in the `built-distribution` directory.
 
-The IDE will connect to the running project manager to look up the project and
+Project manager is there to wait for the IDE to connect to it and then launch
-start the language server. The required version of the language server is
+the engine with its embedded language server. To build the engine issue
-specified in the `edition` field of the `package.yaml` project description. Enso
+following command in the _sbt prompt_:
-projects are located in the `~/enso` directory on Unix and `%userprofile%\enso`
-on Windows systems by default.
 
 ```bash
-cat ~/enso/projects/Unnamed/package.yaml
+sbt:enso> buildEngineDistribution
 ```
 
-```yaml
+Once all the components are assembled, it is time to execute them in
-name: Unnamed
+orchestration. One can pass following environment variables to
-namespace: local
+`project-manager`:
-version: 0.0.1
-license: ""
-authors: []
-maintainers: []
-edition: "2021.20-SNAPSHOT"
-prefer-local-libraries: true
-```
 
-We need to set `edition` to a value that will represent the development version.
+- `ENSO_JVM_OPTS` to for example turn
-It should be different from any Enso versions that have already been released.
+  [debugging of the Engine runtime](debugger/README.md) on
-In this case, we chose the `2021.20-SNAPSHOT` (the current development edition).
+- `ENSO_JVM_PATH` to force a fixed GraalVM to execute the engine/language server
-The project manager will look for the appropriate subdirectory in the _engines_
+  process on
-directory of the distribution folder. Distribution paths are printed when you
+- `ENSO_ENGINE_PATH` the path to engine/language server as created by
-run project manager with `-v` verbose logging.
+  `buildEngineDistribution`, usually
+  `<repository-root>/built-distribution/enso-engine-0.0.0-dev-<os>-<arch>/enso-0.0.0-dev/`
 
-Btw. you can specify `ENSO_JVM_OPTS` to turn
+One doesn't need to deal with these options directly, there is an _sbt command_
-[debugging of the Engine runtime](debugger/README.md) on:
+to orchestrate them all:
 
 ```bash
-$ export ENSO_JVM_OPTS=-agentlib:jdwp=transport=dt_socket,address=5005
+sbt:enso> runProjectManagerDistribution
-$ ./built-distribution/enso-project-manager-0.0.0-dev-linux-amd64/enso/bin/project-manager --no-log-masking -v
-[info] [2021-06-16T11:49:33.639Z] [org.enso.projectmanager.boot.ProjectManager$] Starting Project Manager...
-[debug] [2021-06-16T11:49:33.639Z] [org.enso.runtimeversionmanager.distribution.DistributionManager] Detected paths: DistributionPaths(
-  dataRoot = /home/dbv/.local/share/enso,
-  runtimes = /home/dbv/.local/share/enso/runtime,
-  engines  = /home/dbv/.local/share/enso/dist,
-  bundle   = None,
-  config   = /home/dbv/.config/enso,
-  locks    = /run/user/1000/enso/lock,
-  tmp      = /home/dbv/.local/share/enso/tmp
-)
 ```
 
-On Linux it looks for the `~/.local/share/enso/dist/0.2.32-SNAPSHOT/` directory.
+The above command invokes `buildProjectManagerDistribution`,
+`buildEngineDistribution` and then defines `ENSO_ENGINE_PATH` to connect them
+together and also specifies the `ENSO_JVM_PATH` to the JVM `sbt` process runs
+on.
 
-We can build an engine distribution using the `buildEngineDistribution` command
+There also is a simple way to [debug](debugger/README.md). When adding `--debug`
-in SBT.
+option to the _sbt command_:
-
-##### Bash
 
 ```bash
-sbt buildEngineDistribution
+sbt:enso> runProjectManagerDistribution --debug
 ```
 
-##### PowerShell
+the system also sets
-
+`ENSO_JVM_OPTS=-agentlib:jdwp=transport=dt_socket,address=5005`. Just
-```powershell
+[configure your Java IDE](debugger/README.md) to listen on port 5005 before
-sbt.bat buildEngineDistribution
+invoking the command and you'll be able to debug the engine launched by the
-```
+project manager.
-
-And copy the result to the `0.2.32-SNAPSHOT` engines directory of the
-distribution folder.
-
-##### Bash
-
-```bash
-cp -r built-distribution/enso-engine-0.2.32-SNAPSHOT-linux-amd64/enso-0.2.32-SNAPSHOT ~/.local/share/enso/dist/0.2.32-SNAPSHOT
-```
-
-##### PowerShell
-
-```powershell
-cp -r built-distribution/enso-engine-0.2.32-SNAPSHOT-linux-amd64/enso-0.2.32-SNAPSHOT ~/.local/share/enso/dist/0.2.32-SNAPSHOT
-```
-
-Now, when the project manager is running and the engines directory contains the
-required engine version, you can start IDE with the `--no-backend` flag. It will
-pick up the development version of the language server we just prepared.
 
 To summarize, these are the steps required to run IDE with the development
-version of the language server.
+version of the language server:
 
-1. Run the project manager process.
+```bash
-2. Copy or symlink the development version of the engine created with SBT's
+enso$ ./run gui watch --skip-wasm-opt
-   `buildEnginedistribution` command to the engines directory of the Enso
+```
-   distribution folder.
+
-3. Set the `edition` field of the `package.yaml` project definition to the
+together with that also (after launching `./run backend sbt`) following _sbt
-   version that you created in the previous step.
+command_:
-4. Run the IDE with `--no-backend` flag.
+
+```bash
+sbt:enso> runProjectManagerDistribution
+```
 
 #### Language Server Mode
 

engine/launcher/src/main/scala/org/enso/launcher/distribution/DefaultManagers.scala

@@ -49,6 +49,7 @@ object DefaultManagers {
     alwaysInstallMissing: Boolean
   ): RuntimeVersionManager =
     new RuntimeVersionManager(
+      LauncherEnvironment,
       new CLIRuntimeVersionManagementUserInterface(
         globalCLIOptions,
         alwaysInstallMissing

lib/scala/project-manager/src/main/scala/org/enso/projectmanager/versionmanagement/DefaultDistributionConfiguration.scala

@@ -67,6 +67,7 @@ object DefaultDistributionConfiguration
     userInterface: RuntimeVersionManagementUserInterface
   ): RuntimeVersionManager =
     new RuntimeVersionManager(
+      environment               = this.environment,
       userInterface             = userInterface,
       distributionManager       = distributionManager,
       temporaryDirectoryManager = temporaryDirectoryManager,

lib/scala/project-manager/src/test/scala/org/enso/projectmanager/TestDistributionConfiguration.scala

@@ -76,6 +76,7 @@ class TestDistributionConfiguration(
   override def makeRuntimeVersionManager(
     userInterface: RuntimeVersionManagementUserInterface
   ): RuntimeVersionManager = new RuntimeVersionManager(
+    environment               = environment,
     userInterface             = userInterface,
     distributionManager       = distributionManager,
     temporaryDirectoryManager = temporaryDirectoryManager,

lib/scala/runtime-version-manager-test/src/main/scala/org/enso/runtimeversionmanager/test/RuntimeVersionManagerTest.scala

@@ -58,6 +58,7 @@ class RuntimeVersionManagerTest
     val componentConfig = new GraalVMComponentConfiguration
 
     val runtimeVersionManager = new RuntimeVersionManager(
+      env,
       userInterface,
       distributionManager,
       temporaryDirectoryManager,

lib/scala/runtime-version-manager-test/src/test/scala/org/enso/distribution/locking/ConcurrencyTest.scala

@@ -144,6 +144,7 @@ class ConcurrencyTest
       TemporaryDirectoryManager(distributionManager, resourceManager)
     val componentConfig = new GraalVMComponentConfiguration
     val componentsManager = new RuntimeVersionManager(
+      env,
       TestRuntimeVersionManagementUserInterface.default,
       distributionManager,
       temporaryDirectoryManager,

lib/scala/runtime-version-manager/src/main/scala/org/enso/runtimeversionmanager/components/RuntimeVersionManager.scala

@@ -6,6 +6,7 @@ import nl.gn0s1s.bump.SemVer
 import org.enso.cli.OS
 import org.enso.distribution.{
   DistributionManager,
+  Environment,
   FileSystem,
   TemporaryDirectoryManager
 }
@@ -38,6 +39,7 @@ import scala.util.{Failure, Success, Try, Using}
   * @param componentUpdaterFactory the runtime component updater factory
   */
 class RuntimeVersionManager(
+  environment: Environment,
   userInterface: RuntimeVersionManagementUserInterface,
   distributionManager: DistributionManager,
   temporaryDirectoryManager: TemporaryDirectoryManager,
@@ -63,10 +65,16 @@ class RuntimeVersionManager(
     * Returns None if that version is not installed.
     */
   def findGraalRuntime(version: GraalVMVersion): Option[GraalRuntime] = {
-    val name = graalRuntimeNameForVersion(version)
+    val explicitPathOpt = this.environment.getEnvPath("ENSO_JVM_PATH")
-    val graalRuntimeOpt =
+    val graalRuntimeOpt = explicitPathOpt
-      firstExisting(distributionManager.paths.runtimeSearchPaths.map(_ / name))
+      .map(path => {
-        .map { path =>
+        val runtime = GraalRuntime(version, path)
+        runtime.ensureValid()
+        runtime
+      })
+      .orElse {
+        val pathOpt = findGraalRuntimeOnSearchPath(version)
+        pathOpt.map { path =>
           // TODO [RW] for now an exception is thrown if the installation is
           //  corrupted, in #1052 offer to repair the broken installation
           loadGraalRuntime(path).recoverWith { case e: Exception =>
@@ -82,6 +90,7 @@ class RuntimeVersionManager(
             )
           }.get
         }
+      }
     graalRuntimeOpt match {
       case Some(graalRuntime) =>
         logger.info("Found GraalVM runtime [{}].", graalRuntime)
@@ -91,6 +100,13 @@ class RuntimeVersionManager(
     graalRuntimeOpt
   }
 
+  private def findGraalRuntimeOnSearchPath(
+    version: GraalVMVersion
+  ): Option[Path] = {
+    val name = graalRuntimeNameForVersion(version)
+    firstExisting(distributionManager.paths.runtimeSearchPaths.map(_ / name))
+  }
+
   /** Executes the provided action with a requested engine version.
     *
     * The engine is locked with a shared lock, so it is guaranteed that it will
@@ -206,8 +222,20 @@ class RuntimeVersionManager(
     */
   private def getEngine(version: SemVer): Try[Engine] = {
     val name = engineNameForVersion(version)
-    firstExisting(distributionManager.paths.engineSearchPaths.map(_ / name))
+    this.environment
-      .map(loadEngine)
+      .getEnvPath("ENSO_ENGINE_PATH")
+      .map { p =>
+        logger.info("Using explicit ENSO_ENGINE_PATH: " + p)
+        val manifest = loadAndCheckEngineManifest(p)
+        val engine   = Engine(version, p, manifest.get)
+        Success(engine)
+      }
+      .orElse {
+        val f = firstExisting(
+          distributionManager.paths.engineSearchPaths.map(_ / name)
+        )
+        f.map(loadEngine)
+      }
       .getOrElse {
         Failure(ComponentMissingError(s"Engine $version is not installed."))
       }

lib/scala/runtime-version-manager/src/main/scala/org/enso/runtimeversionmanager/runner/Runner.scala

@@ -130,6 +130,7 @@ class Runner(
       )
     }
 
+  final private val JVM_PATH_ENV_VAR    = "ENSO_JVM_PATH"
   final private val JVM_OPTIONS_ENV_VAR = "ENSO_JVM_OPTS"
 
   /** Runs an action giving it a command that can be used to launch the
@@ -186,10 +187,19 @@ class Runner(
 
       val distributionSettings =
         distributionManager.getEnvironmentToInheritSettings
+
+      val javaHome: Option[String] = environment
+        .getEnvPath(JVM_PATH_ENV_VAR)
+        .map { p =>
+          Logger[Runner].info(
+            "Using explicit " + JVM_PATH_ENV_VAR + " JVM: " + p
+          )
+          p.toString()
+        }
+        .orElse(javaCommand.javaHomeOverride)
+
       val extraEnvironmentOverrides =
-        javaCommand.javaHomeOverride
+        javaHome.map("JAVA_HOME" -> _).toSeq ++ distributionSettings.toSeq
-          .map("JAVA_HOME" -> _)
-          .toSeq ++ distributionSettings.toSeq
 
       action(Command(command, extraEnvironmentOverrides))
     }

project/DistributionPackage.scala

@@ -278,6 +278,36 @@ object DistributionPackage {
     exitCode == 0
   }
 
+  def runProjectManagerPackage(
+    engineRoot: File,
+    distributionRoot: File,
+    args: Seq[String],
+    log: Logger
+  ): Boolean = {
+    import scala.collection.JavaConverters._
+
+    val enso = distributionRoot / "bin" / "project-manager"
+    log.info(s"Executing $enso ${args.mkString(" ")}")
+    val pb  = new java.lang.ProcessBuilder()
+    val all = new java.util.ArrayList[String]()
+    all.add(enso.getAbsolutePath())
+    all.addAll(args.asJava)
+    pb.command(all)
+    pb.environment().put("ENSO_ENGINE_PATH", engineRoot.toString())
+    pb.environment().put("ENSO_JVM_PATH", System.getProperty("java.home"))
+    if (args.contains("--debug")) {
+      all.remove("--debug")
+      pb.environment().put("ENSO_JVM_OPTS", WithDebugCommand.DEBUG_OPTION)
+    }
+    pb.inheritIO()
+    val p        = pb.start()
+    val exitCode = p.waitFor()
+    if (exitCode != 0) {
+      log.warn(enso + " finished with exit code " + exitCode)
+    }
+    exitCode == 0
+  }
+
   def fixLibraryManifest(
     packageRoot: File,
     targetVersion: String,