enso/test/Exploratory_Benchmarks/README.md

# Exploring Table operation performance

These benchmarks are used to compare various approaches to computing operations
on Table columns, to find out what best practices should we use for these and
find venues for optimization of the language and Table implementation.

These benchmarks are not meant to be used for tracking performance of the
current implementation itself. That is supposed to be done by another project -
`Table_Benchmarks`.

## Structure

Currently, the benchmarks are split into a few files, each exploring some
separate topic, like mapping a single column, combining two columns with some
operation, or computing an aggregate operation over a column. In each file,
there may be a few Enso types, each representing a separate benchmark. Usually,
we have two benchmarks for each operation type - one dealing with a primitive
value type like integers (`long` in the Java side) and another dealing with a
reference type like `String` or `Date`. We expect the performance
characteristics between these may differ, e.g. because Java allows to use `long`
without boxing, so we compare them separately.

Each Enso type for a given benchmark contains multiple methods which represent
various 'approaches' to computing the same operation.

Each benchmark run has a name that consists of the type it defines it, a dot and
the method representing the particular approach, e.g.
`Boxed_Map_Test.enso_map_as_vector`.

## Running

The runner is very simple. If any options are to be customized, the Enso file
itself needs to be modified. One can run the whole project to run all the
benchmarks, or run only a specific file.

## Analysis

The output of the benchmarks should be saved to a file. Then that file can be
loaded using the Enso workflow in `tools/performance/benchmark-analysis`.

The workflow is tuned to analysing these comparative benchmarks.

At the top, one can select which file is to be analyzed. Below there is a
dropdown allowing to select one particular benchmark (represented by the type,
e.g. `Boxed_Map_Test`). With that selected, one can display a scatter plot
visualization comparing various approaches of that one given benchmark. On the
plot we can see runtimes of subsequent iterations. Later, we drop the first 40
iterations (the number can easily be customized in the workflow) to ensure
sufficient warm-up for each benchmark. Then a table is displayed computing the
average runtime of each approach and how they compare relative to each other - a
dropdown allows to select one benchmark that will be used as a reference point
(100%) for the average runtime comparison.