Benchmarks

In this document, we describe the benchmark types used for the runtime - Engine micro benchmarks in the section Engine JMH microbenchmarks and standard library benchmarks in the section Standard library benchmarks, and how and where are the results stored and visualized in the section Visualization.

To track the performance of the engine, we use JMH. There are two types of benchmarks:

micro benchmarks located directly in the runtime-benchmarks SBT project. These benchmarks are written in Java, and are used to measure the performance of specific parts of the engine.
standard library benchmarks located in the test/Benchmarks Enso project. These benchmarks are entirely written in Enso, along with the harness code.

Engine JMH microbenchmarks

These benchmarks are written in Java and are used to measure the performance of specific parts of the engine. The sources are located in the runtime-benchmarks SBT project, under engine/runtime-benchmarks directory.

Running the benchmarks

To run the benchmarks, use bench or benchOnly command in the runtime-benchmarks project - bench runs all the benchmarks and benchOnly runs only one benchmark specified with the fully qualified name. The aforementioned commands are mere shortcuts to the standard JMH launcher. To get the full power of the JMH launcher, invoke simply run with cmdline options passed to the launcher. For the full options summary, see the JMH source code, or invoke run -h.

You can change the parameters to the benchmarks either by modifying the annotations directly in the source code, or by passing the parameters to the JMH runner. For example, to run the benchmarks with 3 warmup iterations and 2 measurement iterations, use:

sbt:runtime-benchmarks> run -w 3 -i 2 <bench-name>

Debugging the benchmarks

Currently, the best way to debug the benchmark is to set the @Fork annotation to 0, and to run withDebug command like this:

withDebug --debugger benchOnly -- <fully qualified benchmark name>

Standard library benchmarks

Unlike the Engine micro benchmarks, these benchmarks are written entirely in Enso and located in the test/Benchmarks Enso project. There are two ways to run these benchmarks:

Running standalone
Running via JMH launcher

Note that to avoid inflating the run-time of the std-lib benchmarks on the CI, some extra benchmarks (which are not measuring important functionality, but may serve as a baseline when trying to understand performance of similar scenarios) are disabled by default. To enable them, set the ENSO_ENABLE_EXTRA_BENCHMARKS environment variable before running any benchmarks.

Running standalone

There is a universal launcher that enlists and executes all available benchmarks in test/Benchmarks project. Run it with

enso$ runEngineDistribution --run test/Benchmarks

command. The launcher accepts additional filter argument which allows one to select a benchmark of one's choice by checking for substrings in group or benchmark name. For example:

enso$ runEngineDistribution --run test/Benchmarks New_Vector

runs all the benchmarks that have New_Vector in their name.

The harness within the project is not meant for any sophisticated benchmarking, but rather for quick local evaluation. See the Bench.measure method documentation for more details. For more sophisticated approach, run the benchmarks via the JMH launcher.

Running via JMH launcher

The JMH launcher is located in std-bits/benchmarks directory, as std-benchmarks SBT project. It is a single Java class with a main method that just delegates to the standard JMH launcher, therefore, supports all the command line options as the standard launcher. For the full options summary, either see the JMH source code, or run the launcher with -h option.

The std-benchmarks SBT project supports bench and benchOnly commands, that work the same as in the runtime-benchmarks project, with the exception that the benchmark name does not have to be specified as a fully qualified name, but as a regular expression. To access the full flexibility of the JMH launcher, run it via Bench/run - for example, to see the help message: Bench/run -h. For example, you can run all the benchmarks that have "New_Vector" in their name with just 3 seconds for warmup iterations and 2 measurement iterations with Bench/run -w 3 -i 2 New_Vector.

Whenever you add or delete any benchmarks from test/Benchmarks project, the generated JMH sources need to be recompiled with Bench/clean; Bench/compile. You do not need to recompile the std-benchmarks project if you only modify the benchmark sources.

Visualization

The benchmarks are invoked as a daily GitHub Action, that can be invoked manually on a specific branch as well. The results are kept in the artifacts produced from the actions. In tools/performance/engine-benchmarks directory, there is a simple Python script for collecting and processing the results. See the README in that directory for more information about how to run that script. This script is invoked regularly on a private machine and the results are published in https://enso-org.github.io/engine-benchmark-results/.

6.0 KiB Raw Blame History