enso/docs/infrastructure/sbt.md

layout	title	category	tags	order
developer-doc	Build Tools	infrastructure	infrastructure build	1

Build Tools

The project is built using the Scala Build Tool which manages dependencies between the projects as well as external dependencies and allows for incremental compilation. The build configuration is defined in build.sbt.

• Incremental Compilation
• Bootstrapping
• Compile Hooks
• Helper Tasks
  • Graal and Flatc Version Check
  • Benchmarks
  • Build Information
  • Instruments Generation
  • Flatbuffers Generation
  • Ensuring JARs Were Loaded
  • Debugging Command
  • Recompile Parser
• Native Image

Incremental Compilation

To help wit build times, we do not want to rebuild the whole project with every change, but to only recompile the files that have been affected by the change. This is handled by sbt which under the hood uses zinc (the incremental compiler for Scala). zinc analyses the compiled files and detects dependencies between them to determine which files have to be recompiled when something has been changed.

Bootstrapping

As described in Java 11 Migration to successfully compile the runtime project, the JVM running sbt must use the overridden JAR for Truffle API. This JAR has to be present during startup of the JVM, but it has to be downloaded from the Maven repository.

To fix this chicken-and-egg problem, we have a special bootstrap task, that has to be ran when setting-up the project (and after a version change of Graal). It makes sure the JAR is downloaded and copied to our directory and terminates the sbt process to ensure that the user restarts it. Without the full restart, the JAR would not be seen by the JVM. So when setting up the project or after changing the version of Graal, before launching the sbt shell, you should first run sbt bootstrap, to make sure the environment is properly prepared.

The logic for copying the JAR is implemented in the bootstrapJARs task in CopyTruffleJAR.

Compile Hooks

There are some invariants that are specific to our project, so they are not tracked by sbt, but we want to ensure that they hold to avoid cryptic errors at compilation or runtime.

To check some state before compilation, we add our tasks as dependencies of Compile / compile / compileInputs by adding the following to the settings of a particular project.

Compile / compile / compileInputs := (Compile / compile / compileInputs)
        .dependsOn(preCompileHookTask)
        .value

Tasks that should be run before compilation, should be attached to the compileInputs task. That is because the actual compilation process is ran in the task compileIncremental. Compile / compile depends on compileIncremental but if we add our dependency to Compile / compile, it is considered as independent with compileIncremental, so sbt may schedule it to run in parallel with the actual compilation process. To guarantee that our pre-flight checks complete before the actual compilation, we add them as a dependency of compileInputs which runs strictly before actual compilation.

To check some invariants after compilation, we can replace the original Compile / compile task with a custom one which does its post-compile checks and returns the result of (Compile / compile).value. An example of such a 'patched' compile task is implemented in FixInstrumentsGeneration.

Helper Tasks

There are additional tasks defined in the project directory. They are used by build.sbt to provide some additional functionality.

Graal and Flatc Version Check

EnvironmentCheck defines a helper function that can be attached to the default Global / onLoad state transition to run a version check when loading the sbt project. This helper function compares the version of JVM running sbt with GraalVM version defined in build.sbt and the version of flatc installed in the system with the Flatbuffers library version defined in build.sbt. If the versions do not match it reports an error telling the user to change to the correct version.

Benchmarks

BenchTasks defines configuration keys for benchmarking.

Build Information

BenchTasks records version information including what git commit has been used for compiling the project. This information is used by enso --version.

Instruments Generation

Truffle annotation processor generates a file that registers instruments provided by the runtime. Unfortunately, with incremental compilation, only the changed instruments are recompiled and the annotation processor does not detect this, so un-changed instruments get un-registered.

To fix this, the pre-compile task defined in FixInstrumentsGeneration detects changes to instruments and if only one of them should be recompiled, it forces recompilation of all of them, to ensure consistency.

For unclear reasons, if this task is attached to Compile / compile / compileInputs, while it runs strictly before compilation, the deleted class files are not always all recompiled. So instead, it is attached directly to Compile / compile. This technically could allow for a data race between this task and the actual compilation that happens in compileIncremental, but in practice it seems to be a stable solution.

Sometimes it is unable to detect the need for recompilation before it takes place. To help that, there is another task that replaces the default compile task, which executes the default compilation task and after it, verifies the consistency of instruments files. As it cannot restart compilation, to preserve consistency it ensures the instruments will be recompiled the next time and stops the current compilation task, asking the user to restart it.

Flatbuffers Generation

GenerateFlatbuffers defines the task that runs the Flatbuffer compiler flatc whenever the flatbuffer definitions have been changed. It also makes sure that flatc is available on PATH and that its version matches the version of the library. It reports any errors.

Ensuring JARs Were Loaded

As described in Bootstrapping, to successfully compile the runtime subproject, the JVM running sbt must load some JARs at startup. The user should run sbt bootstrap to ensure that.

If the compilation proceeds without the bootstrapped JARs it may lead to inconsistent state with some dependencies being undetected and weird errors. To avoid such situations, CopyTruffleJAR defines is a pre-compile task that is executed before compiling the runtime subproject which makes sure that the Truffle JAR is up-to-date. Even if the developer forgets about bootstrap, this pre-compile task will update the Truffle JARs. However, as the JARs are loaded at startup, the JVM has to be restarted. To force the user to restart it, the current sbt process is terminated.

This pre-compile task runs before compiling runtime sources, but some dependencies may have started compiling in the meantime (as sbt schedules independent tasks in parallel). The fact that these dependencies are up-to-date may not be registered by sbt due to the abrupt termination. This does not affect consistency of the build state though - as a result these dependencies may be recompiled again when the compilation is restarted. This is acceptable as this restart is just an additional check to ensure correctness and improve user experience. In normal operation, the restart should never be triggered, as the user should remember to run bootstrap when necessary.

Debugging Command

WithDebugCommand defines a command that allows to run a task with additional JVM-level flags.

Recompile Parser

RecompileParser defines a task that can be attached to the compile task in configurations of the syntax project. This task ensures that the syntax project is recompiled whenever syntax-definition changes.

Native Image

NativeImage task is described at Native Image.