enso/docs/infrastructure/sbt.md
Pavel Marek 5a7ad6bfe4
Upgrade enso to GraalVM for jdk 21 (#7991)
Upgrade to GraalVM JDK 21.
```
> java -version
openjdk version "21" 2023-09-19
OpenJDK Runtime Environment GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15)
OpenJDK 64-Bit Server VM GraalVM CE 21+35.1 (build 21+35-jvmci-23.1-b15, mixed mode, sharing)
```

With SDKMan, download with `sdk install java 21-graalce`.

# Important Notes
- After this PR, one can theoretically run enso with any JRE with version at least 21.
- Removed `sbt bootstrap` hack and all the other build time related hacks related to the handling of GraalVM distribution.
- `project-manager` remains backward compatible - it can open older engines with runtimes. New engines now do no longer require a separate runtime to be downloaded.
- sbt does not support compilation of `module-info.java` files in mixed projects - https://github.com/sbt/sbt/issues/3368
- Which means that we can have `module-info.java` files only for Java-only projects.
- Anyway, we need just a single `module-info.class` in the resulting `runtime.jar` fat jar.
- `runtime.jar` is assembled in `runtime-with-instruments` with a custom merge strategy (`sbt-assembly` plugin). Caching is disabled for custom merge strategies, which means that re-assembly of `runtime.jar` will be more frequent.
- Engine distribution contains multiple JAR archives (modules) in `component` directory, along with `runner/runner.jar` that is hidden inside a nested directory.
- The new entry point to the engine runner is [EngineRunnerBootLoader](https://github.com/enso-org/enso/pull/7991/files#diff-9ab172d0566c18456472aeb95c4345f47e2db3965e77e29c11694d3a9333a2aa) that contains a custom ClassLoader - to make sure that everything that does not have to be loaded from a module is loaded from `runner.jar`, which is not a module.
- The new command line for launching the engine runner is in [distribution/bin/enso](https://github.com/enso-org/enso/pull/7991/files#diff-0b66983403b2c329febc7381cd23d45871d4d555ce98dd040d4d1e879c8f3725)
- [Newest version of Frgaal](https://repo1.maven.org/maven2/org/frgaal/compiler/20.0.1/) (20.0.1) does not recognize `--source 21` option, only `--source 20`.
2023-11-17 18:02:36 +00:00

141 lines
5.7 KiB
Markdown

---
layout: developer-doc
title: Build Tools
category: infrastructure
tags: [infrastructure, build]
order: 1
---
# Build Tools
The project is built using the Scala Build Tool which manages dependencies
between the projects as well as external dependencies and allows for incremental
compilation. The build configuration is defined in
[`build.sbt`](../../build.sbt).
<!-- MarkdownTOC levels="2,3" autolink="true" -->
- [Incremental Compilation](#incremental-compilation)
- [Bootstrapping](#bootstrapping)
- [Compile Hooks](#compile-hooks)
- [Helper Tasks](#helper-tasks)
- [Graal and Flatc Version Check](#graal-and-flatc-version-check)
- [Benchmarks](#benchmarks)
- [Build Information](#build-information)
- [Instruments Generation](#instruments-generation)
- [Flatbuffers Generation](#flatbuffers-generation)
- [Ensuring JARs Were Loaded](#ensuring-jars-were-loaded)
- [Debugging Command](#debugging-command)
- [Recompile Parser](#recompile-parser)
- [Native Image](#native-image)
<!-- /MarkdownTOC -->
## Incremental Compilation
To help wit build times, we do not want to rebuild the whole project with every
change, but to only recompile the files that have been affected by the change.
This is handled by sbt which under the hood uses
[zinc](https://github.com/sbt/zinc) (the incremental compiler for Scala). zinc
analyses the compiled files and detects dependencies between them to determine
which files have to be recompiled when something has been changed.
## Compile Hooks
There are some invariants that are specific to our project, so they are not
tracked by sbt, but we want to ensure that they hold to avoid cryptic errors at
compilation or runtime.
To check some state before compilation, we add our tasks as dependencies of
`Compile / compile / compileInputs` by adding the following to the settings of a
particular project.
```
Compile / compile / compileInputs := (Compile / compile / compileInputs)
.dependsOn(preCompileHookTask)
.value
```
Tasks that should be run before compilation, should be attached to the
`compileInputs` task. That is because the actual compilation process is ran in
the task `compileIncremental`. `Compile / compile` depends on
`compileIncremental` but if we add our dependency to `Compile / compile`, it is
considered as independent with `compileIncremental`, so sbt may schedule it to
run in parallel with the actual compilation process. To guarantee that our
pre-flight checks complete _before_ the actual compilation, we add them as a
dependency of `compileInputs` which runs _strictly before_ actual compilation.
To check some invariants _after_ compilation, we can replace the original
`Compile / compile` task with a custom one which does its post-compile checks
and returns the result of `(Compile / compile).value`.
## Helper Tasks
There are additional tasks defined in the [`project`](../../project) directory.
They are used by [`build.sbt`](../../build.sbt) to provide some additional
functionality.
### Graal and Flatc Version Check
[`EnvironmentCheck`](../../project/EnvironmentCheck.scala) defines a helper
function that can be attached to the default `Global / onLoad` state transition
to run a version check when loading the sbt project. This helper function
compares the version of JVM running sbt with GraalVM version defined in
[`build.sbt`](../../build.sbt) and the version of `flatc` installed in the
system with the Flatbuffers library version defined in
[`build.sbt`](../../build.sbt). If the versions do not match it reports an error
telling the user to change to the correct version.
### Benchmarks
[`BenchTasks`](../../project/BenchTasks.scala) defines configuration keys for
benchmarking.
### Build Information
[`BenchTasks`](../../project/BuildInfo.scala) records version information
including what git commit has been used for compiling the project. This
information is used by `enso --version`.
### Instruments Generation
Truffle annotation processor generates a file that registers instruments
provided by the runtime. Unfortunately, with incremental compilation, only the
changed instruments are recompiled and the annotation processor does not detect
this, so un-changed instruments get overwritten.
In the past we had a pre-compile task (see
[FixInstrumentsGeneration](https://github.com/enso-org/enso/blob/8ec2a92b770dea35e47fa9287dbdd1363aabc3c0/project/FixInstrumentsGeneration.scala))
that detected changes to instruments and if only one of them was to be
recompiled, it forced recompilation of all of them, to ensure consistency. This
workaround helped to avoid later runtime issues but sometimes triggered a
cascade of recompilations, which weren't clear to the end user. Instead, to
avoid overwriting entries in META-INF files, individual services were moved to
separate subprojects and during assembly of uber jar we concatenate meta files
with the same service name.
### Flatbuffers Generation
[`GenerateFlatbuffers`](../../project/GenerateFlatbuffers.scala) defines the
task that runs the Flatbuffer compiler `flatc` whenever the flatbuffer
definitions have been changed. It also makes sure that `flatc` is available on
PATH and that its version matches the version of the library. It reports any
errors.
### Debugging Command
[`WithDebugCommand`](../../project/WithDebugCommand.scala) defines a command
that allows to run a task with additional JVM-level flags.
### Recompile Parser
[`RecompileParser`](../../project/RecompileParser.scala) defines a task that can
be attached to the `compile` task in configurations of the `syntax` project.
This task ensures that the `syntax` project is recompiled whenever
`syntax-definition` changes.
## Native Image
[`NativeImage`](../../project/NativeImage.scala) task is described at
[Native Image](native-image.md).