enso/project/NativeImage.scala

Ignoring revisions in .git-blame-ignore-revs. Click here to bypass and see the normal blame view.

491 lines
18 KiB
Scala
Raw Normal View History

2020-07-10 13:57:42 +03:00
import java.io.File
import java.nio.file.Path
2020-07-10 13:57:42 +03:00
import sbt._
2020-07-10 13:57:42 +03:00
import sbt.Keys._
import sbt.internal.util.ManagedLogger
2020-12-09 16:58:11 +03:00
import sbtassembly.AssemblyKeys.assembly
import sbtassembly.AssemblyPlugin.autoImport.assemblyOutputPath
2020-07-10 13:57:42 +03:00
import scala.sys.process._
import java.nio.file.Paths
2020-07-10 13:57:42 +03:00
object NativeImage {
2020-12-09 16:58:11 +03:00
/** Specifies whether the build executable should include debug symbols.
* Should be set to false for production builds. May work only on Linux.
*/
private val includeDebugInfo: Boolean = false
lazy val smallJdk = taskKey[Option[File]]("Location of a minimal JDK")
lazy val additionalCp =
taskKey[Seq[String]](
"Additional class-path entries to be added to the native image"
)
/** List of classes that should be initialized at build time by the native image.
* Note that we strive to initialize as much classes during the native image build
* time as possible, as this reduces the time needed to start the native image.
* One wildcard could theoretically be used instead of the list, but to make things
* more explicit, we use the list.
*/
private val defaultBuildTimeInitClasses = Seq(
"org",
"org.enso",
"scala",
"java",
"sun",
"cats",
"io",
"shapeless",
"com",
"izumi",
"zio",
"enumeratum",
"akka",
"nl",
"ch.qos.logback"
)
val NATIVE_IMAGE_ARG_FILE = "native-image-args.txt"
/** Creates a task that builds a native image for the current project.
2020-12-09 16:58:11 +03:00
*
* This task must be setup in such a way that the assembly JAR is built
* before it starts, as it uses this JAR for the build. Usually this can be
* done by appending `.dependsOn(LocalProject("project-name") / assembly)`.
*
* Additional Native Image configuration can be set for each project by
* editing configuration files in subdirectories of `META-INF/native-image`
* of its resources directory. More information can be found at
* [[https://github.com/oracle/graal/blob/master/substratevm/BuildConfiguration.md]].
*
* @param artifactName name of the artifact to create
* @param staticOnLinux specifies whether to link statically (applies only
* on Linux)
2020-12-09 16:58:11 +03:00
* @param additionalOptions additional options for the Native Image build
* tool
* @param buildMemoryLimitMegabytes a memory limit for the build tool, in
2021-05-14 15:08:39 +03:00
* megabytes; it is good to set this limit to
2020-12-09 16:58:11 +03:00
* make GC more aggressive thus allowing it to
* build successfully even with limited memory
* @param runtimeThreadStackMegabytes the runtime thread stack size; the
* minimum for ZIO to work is higher than the
* default value on some systems
2020-12-09 16:58:11 +03:00
* @param initializeAtRuntime a list of classes that should be initialized at
* run time - useful to set exceptions if build
* time initialization is set to default
* @param initializeAtBuildtime a list of classes that should be initialized at
* build time.
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
* @param includeRuntime Whether `org.enso.runtime` should is included. If yes, then
* it will be passed as a module to the native-image along with other
* Graal and Truffle related modules.
* @param verbose whether to print verbose output from the native image.
*/
def buildNativeImage(
name: String,
staticOnLinux: Boolean,
targetDir: File = null,
additionalOptions: Seq[String] = Seq.empty,
buildMemoryLimitMegabytes: Option[Int] = Some(15608),
runtimeThreadStackMegabytes: Option[Int] = Some(2),
initializeAtRuntime: Seq[String] = Seq.empty,
initializeAtBuildtime: Seq[String] = defaultBuildTimeInitClasses,
mainClass: Option[String] = None,
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
verbose: Boolean = false,
includeRuntime: Boolean = true
2020-12-09 16:58:11 +03:00
): Def.Initialize[Task[Unit]] = Def
.task {
val log = state.value.log
val targetLoc = artifactFile(targetDir, name, false)
def nativeImagePath(prefix: Path)(path: Path): Path = {
val base = path.resolve(prefix)
2020-12-09 16:58:11 +03:00
if (Platform.isWindows)
base.resolve("native-image.cmd")
else base.resolve("native-image")
}
val (javaHome: Path, nativeImagePathResolver) =
smallJdk.value
.map(f =>
(f.toPath(), nativeImagePath(Path.of("lib", "svm", "bin")) _)
)
.filter { case (p, resolver) => resolver(p).toFile.exists() }
.getOrElse(
(
Paths.get(System.getProperty("java.home")),
nativeImagePath(Path.of("bin")) _
)
)
log.info("Native image JAVA_HOME: " + javaHome)
val subProjectRoot = baseDirectory.value
2020-12-09 16:58:11 +03:00
val pathToJAR =
(assembly / assemblyOutputPath).value.toPath.toAbsolutePath.normalize
if (!nativeImagePathResolver(javaHome).toFile.exists()) {
2020-12-09 16:58:11 +03:00
log.error(
"Unexpected: Native Image component not found in the JVM distribution: " + javaHome
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
)
log.error("Is this a GraalVM distribution?")
log.error(
"On older distributions, you can install Native Image with `gu install native-image`."
2020-12-09 16:58:11 +03:00
)
throw new RuntimeException(
"Native Image build failed, " +
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
"because native-image binary was not found."
2020-12-09 16:58:11 +03:00
)
}
if (additionalOptions.contains("--language:java")) {
log.warn(
s"Building ${targetLoc} image with experimental Espresso support!"
)
}
2020-12-09 16:58:11 +03:00
val debugParameters =
if (includeDebugInfo) Seq("-H:GenerateDebugInfo=1") else Seq()
val (staticParameters, pathExts) =
if (staticOnLinux && Platform.isLinux) {
// Note [Static Build On Linux]
val buildCache =
subProjectRoot / "build-cache"
val path = ensureMuslIsInstalled(buildCache, log)
(Seq("--static", "--libc=musl"), Seq(path.toString))
} else (Seq(), Seq())
val configLocation =
subProjectRoot / "native-image-config"
val configs =
if (configLocation.exists()) {
val path = configLocation.toPath.toAbsolutePath
log.debug(s"Picking up Native Image configuration from `$path`.")
Seq(s"-H:ConfigurationFileDirectories=$path")
} else {
log.debug(
"No Native Image configuration found, proceeding without it."
2020-07-10 13:57:42 +03:00
)
2020-12-09 16:58:11 +03:00
Seq()
2020-07-10 13:57:42 +03:00
}
val quickBuildOption =
if (BuildInfo.isReleaseMode) Seq() else Seq("-Ob")
val buildMemoryLimitOptions =
buildMemoryLimitMegabytes.map(megs => s"-J-Xmx${megs}M").toSeq
val runtimeMemoryOptions =
runtimeThreadStackMegabytes.map(megs => s"-R:StackSize=${megs}M").toSeq
2020-12-09 16:58:11 +03:00
val initializeAtBuildtimeOptions =
if (initializeAtBuildtime.isEmpty) Seq()
else {
val classes = initializeAtBuildtime.mkString(",")
Seq(s"--initialize-at-build-time=$classes")
}
2020-12-09 16:58:11 +03:00
val initializeAtRuntimeOptions =
if (initializeAtRuntime.isEmpty) Seq()
else {
val classes = initializeAtRuntime.mkString(",")
Seq(s"--initialize-at-run-time=$classes")
2020-07-10 13:57:42 +03:00
}
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
val runtimeCp = (LocalProject("runtime") / Runtime / fullClasspath).value
val runnerCp =
(LocalProject("engine-runner") / Runtime / fullClasspath).value
val ourCp = (Runtime / fullClasspath).value
val cpToSearch = (ourCp ++ runtimeCp ++ runnerCp).distinct
val componentModules: Seq[String] = JPMSUtils
.filterModulesFromClasspath(
cpToSearch,
JPMSUtils.componentModules,
log,
engine-runner and language-server are separate JPMS modules (#10823) * Use moduleDependencies instead of modulePath * Fix compilation of editions * Fix compilation of distribution-manager * polyglot-api needs to explicitly compile module-info * Fix compilationOrder in library-manager and edition-updater * engine-runner-common is module * JPMSPlugin provides default implementation of compileModuleInfo * Remove unused setting key from JPMSUtils.compileModuleInfo * JPMSPlugin has internalModuleDependencies and exportedModule tasks. * Use BuildVersion instead of buildInfo * Manual compilation of module-info.java is reported as warning * Define org.enso.scalalibs.wrapper meta project. * Fix module check in JPMSPlugin. This is a fix for projects that declare `Compile /exportJars := true` * version-output is a module * ydoc-server uses internalModuleDependencies * persistance is module * engine-common uses internalModuleDependencies * polyglot-api does not override compileModuleInfo task * runtime-parser uses internalModuleDependencies * edition-updater is module * Update moduleDependencies for distribution-manager * editions is module * Fix some dependencies of modules * scala-yaml is a module * Add scala-compiler to scala-libs-wrapper * cli depends on scala-library module * Add dependencies for distribution-manager module * Add some scala-library dependencies in some modules * engine-runner uses internalModuleDependencies * Fix module dependencies of library-manager * Rename org.enso.scalalibs.wrapper to org.enso.scala.wrapper * Add jsoniter-scala-macros to org.enso.scala.wrapper fat module * Fix dependencies of some projects * polyglot-api does not depend on truffle-api * Fix dependencies of some projects * runtime does not use com.google.common * runtime is a module * text-buffer is a module * refactoring-utils is a module * runtime-compiler is a module * runtime-instrument-common is a module * connected-lock-manager is a module * JPMSUtils reports project name in some error messages * Modularize some instruments * module-info compilation is cached * runtime-instrument-runtime-server is module * runtime-language-epb is module * Remove runtime-fat-jar * engine-runner is not a fat jar * JPMSPlugin defines exportedModuleBin task * Redefine componentModulesPaths task * interpreter-dsl is module * Redefine componentModulesPaths task * fmt sbt * scala-libs-wrapper is a modular fat jar * Add some module deps to org.enso.runtime * engine-runner is not a fat jar * Rename package in logging-config * Rename package in logging-service * Rename package in logging-service-logback * Fix dependencies of exportedModuleBin task * Mixed projects have own compileJava task this task does not compile only module-info.java but all the java sources. So that we can see errors more easily. When only module-info.java is compiled, the only errors that we can see are that we did not include some modules on module-path. * Fix definition of exportedModule task. * Remove usages of non-existing buildInfo and replace it with BuildVersion * Fix some dependencies of org.enso.runtime module * module-info compilation is handled directly by FrgaalCompiler * module-info compilation is forced for projects that has only Scala sources with single module-info.java * Fix compilation of org.enso.runtime * manual module-info compilation is not a warning * Rename packages in logging-utils-akka * Create org.enso.language.server.deps.wrapper module * language-server is module * Creat akka-wrapper modular fat jar * fmt * Define common settings for modularFatJarWrapper * Fix compilation of json-rpc-server * Use akka and zio wrappers * language-server depends on org.eclipse.jgit * Fix some dependencies - update library manifests works now! * update library manifests invokes runner directly * buildEngineDistribution does not copy runner.jar * Remove EngineRunnerBootLoader * Fix compilation of std libs * --patch-module and --add-exports are also passed to javac * Rename package in runtime-integration-tests. The package name org.enso.compiler clashes with the package from the module * Remove usage of buildInfo * FrgaalJavaCompiler can deal with non-existing module-info.java when shouldCompileModuleInfo is true. It just generates a warning in such case as it suggests that there is something wrong with the project configuration. * Revert AliasAnalysisTest.scala * Fix dependencies and java cmdline options for runtime-integration-tests * Rename test package * runtime-integration-test depends on logging-service-logback/Test/compile * Rename package in logging-service-logback/Test * Fix FrgaalJavaCompiler creation for projects * Sanitize Test/javaOptions arguments * Sanitize Test/javaOptions arguments * All the JPMSPlugin settings are scoped * Remove unused sbt tasks * modularFatJarWrapperSettings do not override javacOptions * Resolve issue "Cannot find TestLoggerProvider" in runtime-integration-tests * org.enso.runtime module is open * Test that test classes are unconditionally opened and exported * polyglot-api-macros is a module * JPMSPlugin handles --add-opens cmdline option * RuntimeServerTest ensures instruments are initialized * Add some exports to org.enso.runtime.compiler * Add instruments on module-path to runtime-integration-tests * Replace TestLogProviderOnClassPath with TestLogProviderOnModulePath * Replace buildInfo with BuildVersion * Add jpms-wrapper-scalatest * ReportLogsOnFailure is in non-modular testkit project * Add necessary dependencies to testkit project * Revert "Add jpms-wrapper-scalatest" This reverts commit 732b3427a2c4a7051c831f3439b4fc04ba126ae1. * modularize filewatcher and wrap its dependencies * Initial fix for language-server/test * frgaal compiler setting are scoped for Compile and Test * Rename package in language-server/test * Exclude com.sun.jna from wrapper jars * Rename package in library-manager-test * testkit is an automatic module * process-utils is module * akka-wrapper contains akka-http * Some fixes for library-manager-test * Fix dependencies for akka-wrapper * scala-libs-wrapper exports shapeless * lang server deps wrapper exports pureconfig * json-rpc-server requires org.slf4j * Add some dependencies * lang server deps wrapper exports pureconfig.generic * language server test requires bouncycastle provider * language server depends on cli * directory-watcher wrapper requires org.slf4j * WatcherAdapter logs unsuccessful initialization errors * Fix error reporting in WatcherAdapter * Fix rest of the language-server tests * language-server-deps-wrapper depends on scala-libs-wrapper * Fix rest of the language-server tests * Missing module-info.class in an internal project is a warning, not an error * Rename jpms-methvin-directory-watcher-wrapper to a simpler name * compileOrder has to be specified before libraryDependencies * exclude module-info.java from polyglot-api-macros * Remove temporary logging in customFrgaalCompilerSettings * Fix compilation of logging-service-logback * Fix compilation of runtime-benchmarks * Fix runtime-benchmarks/run * HostClassLoader delegates to org.graalvm.polyglot class loader if org.enso.runtime is not on boot layer * org.enso.runtime.lnaguage.epb module must be opened to allow it to be used by annnotation processor * fmt * Fix afetr merge * Add module deps after merge * Print stack trace of the uncaught exception from the annotation processor * Remove akka-actor-typed from akka-wrapper * runtime-instrument-common depends on slf4j * Fix module-path for runtime-instrument-repl-debugger * runtime-benchmarks depends on runtime-language-arrow * --module-path is passed directly to frgaal * Fix some module-related cmd line options for std-benchmarks * Revert "--module-path is passed directly to frgaal" This reverts commit da63f66a0ed2377f78d90d13bdd955f903f0ff19. * Avoid closing of System.err when closing Context * Avoid processing altogether when requested annotations are empty * Pass shouldNotLimitModules opt to frgaal * Pass module-path and add-modules options with -J prefix to frgaal * BenchProcessor annotation processor creates its own truffle module layer * bench-processor and benchmarks-common are modules * fmt * Fix after mege * Enable JMH annotation processor * Fix compileOrder in some projects * Insert TruffleBoundary to QualifiedName. This is a revert * Fix building of engine-runner native image * Add more deps to the native image * Force module-info compilation in instruments. This fixes some weird sbt bug * Don't run engine-runner/assembly from Rust build script * Update docs of JPMSPlugin * fmt * runtime-benchmarks depends on benchmarks-common module * Fix benchmark report writing * std-benchmarks annot processing does not take settings from runtime-benchmarks * Suppress interpreter only warning in annotation processor * Runtime version manager does not expect runtime.jar fat jar * fmt * Fix module entry point * Move some polyglot tests to runtime-integration-tests. Also make their output silent * pkg has no dependency on org.graalvm.truffle * Fix compiler dependencies test * Rename all runtime.jar in fake releases * Add language-server with dependencies to component dir * No module-info.class in target dir is warning not error * language-server does not depend on netbeans lookup uitl * Declare LanguageServerApi service provider in module-info * connected-lock-manager-server is JPMS module * task-progress-notifications is module * Add fansi-wrapper module * Fix compilation of connected-lock-manager-server * Define correct Test/internalModuleDependencies for project-manager * fmt * Fix LauncherRunnerSpec - no runtime.jar * Add fansi-wrapper to runtime-integration-tests and runtime-benchmarks * Fix engine-runner native image build * Use newer JNA version - fixes running of hyperd * DRY * scala-compiler DRY * fmt * More build.sbt refactoring * Include runtime-instrument-id-execution in engine-runner native image * TruffleBoundary for QualifiedName.toString * Finding a needle in a haystack :facepalm: * More scala-library DRY * more mixed-java/scala goodies * Fix compilation of syntax-rust-definition * Test that engine-runner does not depend on language-server * Append rather than assign `moduleDependencies` `++=` is less error prone than `:=`. Also discovered some unnecessary dependencies. * Replace : with File.pathSeparator * [WIP] Make logging in ProjectService more verbose * language-server/test didn't start because of missing lookup and fansi modules * Formatting * org.enso.cli.task.notifications needs Akka and Circe to link * project-manager/test depends on buildEngineDistribution * [WIP] Even more verbose logging for creating projects * [WIP] Even more verbose logging for creating projects * Revert "[WIP] Even more verbose logging for creating projects" This reverts commit a7067c8472c34b6934a0ea6f4d8075628eaf14e6. * Revert "[WIP] Even more verbose logging for creating projects" This reverts commit fc6f53d4f1dd4a86a7b01e9c4f86b35a824bf88b. * Revert "[WIP] Make logging in ProjectService more verbose" This reverts commit 427428e1420984c5faee0ce0981a3b405964df47. * All the project with JPMSPlugin has stripped artifact names * Revert all placeholder fake release components to runtime.jar without version * Eliminate a cross version hack We shouldn't be specifying Scala dependencies with a Scala cross version in the suffix. * Address SBT lint warnings * Revert "Eliminate a cross version hack" This reverts commit 8861dab288cbb0ff7f2f456c8725d75d4442aa9c. * logging-service-logback is mixedJavaScalaProject * fmt * Stripped artifact name contains classifier. This fixes tests as those were named like `artifact-tests.jar`. * Don't use LocalProject unless really needed * Add more logging when BenchProcessor fails * logging-service-logback is not mixed project * Work with java.io.File.getPath to avoid mixing slash and backslash on Windows * Reapply "Eliminate a cross version hack" This reverts commit edaa436ee89670f4065c13dda7593e19fb207b10. * Pass scalaBinaryVersion correctly * Remove scala-compiler from the distribution * Fix IllegalAccessErrors from serde * typos * License review * fmt * Move testLogProviderOnModulePath to TestJPMSConfiguration * logging-service-logback is not a mixed project --------- Co-authored-by: Jaroslav Tulach <jaroslav.tulach@enso.org> Co-authored-by: Hubert Plociniczak <hubert.plociniczak@gmail.com>
2024-09-25 22:33:13 +03:00
projName = (moduleName.value),
scalaBinaryVersion.value,
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
shouldContainAll = true
)
.map(_.data.getAbsolutePath)
val auxCp = additionalCp.value
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
val fullCp =
if (includeRuntime) {
componentModules ++ auxCp
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
} else {
ourCp.map(_.data.getAbsolutePath) ++ auxCp
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
}
val cpStr = fullCp.mkString(File.pathSeparator)
log.debug("Class-path: " + cpStr)
val verboseOpt = if (verbose) Seq("--verbose") else Seq()
var args: Seq[String] =
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
Seq("-cp", cpStr) ++
quickBuildOption ++
2020-12-09 16:58:11 +03:00
debugParameters ++ staticParameters ++ configs ++
Seq("--no-fallback", "--no-server") ++
Seq("-march=compatibility") ++
initializeAtBuildtimeOptions ++
initializeAtRuntimeOptions ++
buildMemoryLimitOptions ++
runtimeMemoryOptions ++
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
additionalOptions ++
Seq("-o", targetLoc.toString())
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
args = mainClass match {
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
case Some(main) =>
args ++
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
Seq(main)
case None =>
args ++
Upgrade enso to GraalVM for jdk 21 (#7991) Upgrade to GraalVM JDK 21. ``` > java -version openjdk version "21" 2023-09-19 OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15) OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing) ``` With SDKMan, download with `sdk install java 21-graalce`. # Important Notes - After this PR, one can theoretically run enso with any JRE with version at least 21. - Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution. - `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded. - sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368 - Which means that we can have `module-info.java` files only for Java-only projects. - Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar. - `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent. - Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory. - The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module. - The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725) - [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 21:02:36 +03:00
Seq("-jar", pathToJAR.toString)
Generate native image for engine-runner (#3638) This PR adds a possibility to generate native-image for engine-runner. Note that due to on-demand loading of stdlib, programs that make use of it are currently not yet supported (that will be resolved at a later point). The purpose of this PR is only to make sure that we can generate a bare minimum runner because due to lack TruffleBoundaries or misconfiguration in reflection config, this can get broken very easily. To generate a native image simply execute: ``` sbt> engine-runner-native/buildNativeImage ... (wait a few minutes) ``` The executable is called `runner` and can be tested via a simple test that is in the resources. To illustrate the benefits see the timings difference between the non-native and native one: ``` >time built-distribution/enso-engine-0.0.0-dev-linux-amd64/enso-0.0.0-dev/bin/enso --no-ir-caches --in-project test/Tests/ --run engine/runner-native/src/test/resources/Factorial.enso 6 720 real 0m4.503s user 0m9.248s sys 0m1.494s > time ./runner --run engine/runner-native/src/test/resources/Factorial.enso 6 720 real 0m0.176s user 0m0.042s sys 0m0.038s ``` # Important Notes Notice that due to a [bug in GraalVM](https://github.com/oracle/graal/issues/4200), which is already fixed in 22.x, and us still being on 21.x for the time being, I had to add a workaround to our sbt build to build a different fat jar for native image. To workaround it I had to exclude sqlite jar. Hence native image task is on `engine-runner-native` and not on `engine-runner`. Will need to add the above command to CI.
2022-09-22 17:45:10 +03:00
}
2020-12-09 16:58:11 +03:00
val targetDirValue = (Compile / target).value
val argFile = targetDirValue.toPath.resolve(NATIVE_IMAGE_ARG_FILE)
IO.writeLines(argFile.toFile, args, append = false)
2020-12-09 16:58:11 +03:00
val pathParts = pathExts ++ Option(System.getenv("PATH")).toSeq
val newPath = pathParts.mkString(File.pathSeparator)
val cmd =
Seq(nativeImagePathResolver(javaHome).toString) ++
verboseOpt ++
Seq("@" + argFile.toAbsolutePath.toString)
2020-12-09 16:58:11 +03:00
log.debug(s"""PATH="$newPath" ${cmd.mkString(" ")}""")
val process =
Process(cmd, None, "PATH" -> newPath)
// All the output from native-image is redirected into a StringBuilder, and printed
// at the end of the build. This mitigates the problem when there are multiple sbt
// commands running in parallel and the output is intertwined.
val sb = new StringBuilder
val processLogger = ProcessLogger(str => {
log.info(str)
sb.append(str + System.lineSeparator())
})
log.info(
s"Started building $targetLoc native image. The output is captured."
)
val retCode = process.!(processLogger)
val targetFile = artifactFile(targetDir, name, true)
if (retCode != 0 || !targetFile.exists()) {
log.error(s"Native Image build of $targetFile failed, with output: ")
println(sb.toString())
2020-12-09 16:58:11 +03:00
throw new RuntimeException("Native Image build failed")
2020-07-10 13:57:42 +03:00
}
log.info(s"$targetLoc native image build successful.")
2020-12-09 16:58:11 +03:00
}
.dependsOn(Compile / compile)
/** Creates a task which watches for changes of any compiled files or
* dependencies and triggers a rebuild if and only if there are any changes.
*
* @param actualBuild reference to the task doing the actual Native Image
* build, usually one returned by [[buildNativeImage]]
* @param artifactName name of the artifact that is expected to be created
* by the native image build
* @return
*/
def incrementalNativeImageBuild(
actualBuild: TaskKey[Unit],
name: String,
targetDir: File = null
): Def.Initialize[Task[Unit]] =
Def.taskDyn {
def rebuild(reason: String) = {
streams.value.log.info(
s"$reason, forcing a rebuild."
)
val artifact = artifactFile(targetDir, name)
if (artifact.exists()) {
artifact.delete()
}
Def.task {
actualBuild.value
}
}
val classpath = (Compile / fullClasspath).value
val filesSet = classpath.flatMap(f => f.data.allPaths.get()).toSet
val store =
streams.value.cacheStoreFactory.make("incremental_native_image")
Tracked.diffInputs(store, FileInfo.hash)(filesSet) {
sourcesDiff: ChangeReport[File] =>
2020-12-09 16:58:11 +03:00
if (sourcesDiff.modified.nonEmpty)
rebuild("Native Image is not up to date")
else if (!artifactFile(targetDir, name).exists())
rebuild("Native Image does not exist")
else
Def.task {
2020-12-09 16:58:11 +03:00
streams.value.log.info(
s"No source changes, $artifactName Native Image is up to date."
)
}
}
}
/** [[File]] representing the artifact called `name` built with the Native
2020-09-09 16:37:26 +03:00
* Image.
*/
def artifactFile(
targetDir: File,
name: String,
withExtension: Boolean = false
): File = {
val artifactName =
if (withExtension && Platform.isWindows) name + ".exe"
else name
if (targetDir == null) {
new File(artifactName).getAbsoluteFile()
} else {
new File(targetDir, artifactName)
}
}
private val muslBundleUrl =
2020-09-09 16:37:26 +03:00
"https://github.com/gradinac/musl-bundle-example/releases/download/" +
"v1.0/musl.tar.gz"
/** Ensures that the `musl` bundle is installed.
*
* Checks for existence of its directory and if it does not exist, downloads
2020-08-28 14:03:09 +03:00
* and extracts the bundle. After extracting it does the required
* initialization (renaming paths to be absolute and creating a shell script
* called `x86_64-linux-musl-gcc`).
2020-08-28 14:03:09 +03:00
*
* `musl` is needed for static builds on Linux.
*
* @param buildCache build-cache directory for the current project
* @param log a logger instance
2020-08-28 14:03:09 +03:00
* @return path to the `musl` bundle binary directory which should be added
* to PATH of the launched native-image
*/
private def ensureMuslIsInstalled(
buildCache: File,
log: ManagedLogger
): Path = {
val muslRoot = buildCache / "musl-1.2.0"
val bundleLocation = muslRoot / "bundle"
2020-08-28 14:03:09 +03:00
val binaryLocation = bundleLocation / "bin"
val gccLocation = binaryLocation / "x86_64-linux-musl-gcc"
2020-08-28 14:03:09 +03:00
def isMuslInstalled =
gccLocation.exists() && gccLocation.isOwnerExecutable
if (!isMuslInstalled) {
log.info(
"`musl` is required for a static build, but it is not installed for " +
"this subproject."
)
try {
log.info("A `musl` bundle will be downloaded.")
buildCache.mkdirs()
val bundle = buildCache / "musl-bundle.tar.gz"
val downloadExitCode = (url(muslBundleUrl) #> bundle).!
if (downloadExitCode != 0) {
log.error("Cannot download `musl` bundle.")
throw new RuntimeException(s"Cannot download `$muslBundleUrl`.")
}
muslRoot.mkdirs()
val tarExitCode = Seq(
"tar",
"xf",
bundle.toPath.toAbsolutePath.toString,
"-C",
muslRoot.toPath.toAbsolutePath.toString
).!
if (tarExitCode != 0) {
log.error(
"An error occurred when extracting the `musl` library bundle."
)
throw new RuntimeException(s"Cannot extract $bundle.")
}
2020-08-28 14:03:09 +03:00
replacePathsInSpecs(
bundleLocation / "lib" / "musl-gcc.specs",
bundleLocation
)
createGCCWrapper(bundleLocation)
log.info("Installed `musl`.")
} catch {
case e: Exception =>
throw new RuntimeException(
"`musl` installation failed. Cannot proceed with a static " +
"Native Image build.",
e
)
}
}
2020-08-28 14:03:09 +03:00
binaryLocation.toPath.toAbsolutePath.normalize
}
/** Replaces paths in `musl-gcc.specs` with absolute paths to the bundle.
2020-08-28 14:03:09 +03:00
*
* The paths in `musl-gcc.specs` start with `/build/bundle` which is not a
* valid path by default. Instead, these prefixes are replaced with an
* absolute path to the bundle.
*
* @param specs reference to `musl-gcc.specs` file
* @param bundleLocation location of the bundle root
*/
private def replacePathsInSpecs(specs: File, bundleLocation: File): Unit = {
val content = IO.read(specs)
val bundlePath = bundleLocation.toPath.toAbsolutePath.normalize.toString
val replaced = content.replace("/build/bundle", bundlePath)
IO.write(specs, replaced)
}
/** Creates a simple shell script called `musl-gcc` which calls the original
2020-08-28 14:03:09 +03:00
* `gcc` and ensures the bundle's configuration (`musl-gcc.specs`) is loaded.
*/
private def createGCCWrapper(bundleLocation: File): Unit = {
val bundlePath = bundleLocation.toPath.toAbsolutePath.normalize.toString
val content =
s"""#!/bin/sh
|exec "$${REALGCC:-gcc}" "$$@" -specs "$bundlePath/lib/musl-gcc.specs"
|""".stripMargin
val wrapper = bundleLocation / "bin" / "x86_64-linux-musl-gcc"
2020-08-28 14:03:09 +03:00
IO.write(wrapper, content)
wrapper.setExecutable(true)
}
2020-07-10 13:57:42 +03:00
}
/* Note [Static Build On Linux]
* ~~~~~~~~~~~~~~~~~~~~~~~~~~~~
* The default `glibc` contains a bug that would cause crashes when downloading
* files form the internet, which is a crucial Launcher functionality. Instead,
* `musl` is suggested by Graal as an alternative libc. The sbt task
* automatically downloads a bundle containing all requirements for a static
* build with `musl`.
*
2020-08-28 14:03:09 +03:00
* The `musl` bundle that we use is not guaranteed to be maintained, so if in
* the future a new version of `musl` comes out and we need to upgrade, we may
* need to create our own infrastructure (a repository with CI jobs for creating
* such bundles). It is especially important to note that the libstdc++ that is
* included in this bundle should also be built using `musl` as otherwise linker
* errors may arise.
*
* Currently, to use `musl`, the `--libc=musl` option has to be added to the
* build and `x86_64-linux-musl-gcc` must be available in the system PATH for the
2020-08-28 14:03:09 +03:00
* native-image. In the future it is possible that a different option will be
* used or that the bundle will not be required anymore if it became
* prepackaged. This task may thus need an update when moving to a newer version
* of Graal.
*
* Currently to make the bundle work correctly with GraalVM 20.2, a shell script
* called `x86_64-linux-musl-gcc` which loads the bundle's configuration is created by the
2020-08-28 14:03:09 +03:00
* task and the paths starting with `/build/bundle` in `musl-gcc.specs` are
* replaced with absolute paths to the bundle location.
*/