56635c9a88
The added benchmark is a basis for a performance investigation. We compare the performance of the same operation run in Java vs Enso to see what is the overhead and try to get the Enso operations closer to the pure-Java performance.
2.5 KiB
Exploring Table operation performance
These benchmarks are used to compare various approaches to computing operations on Table columns, to find out what best practices should we use for these and find venues for optimization of the language and Table implementation.
These benchmarks are not meant to be used for tracking performance of the
current implementation itself. That is supposed to be done by another project -
Table_Benchmarks.
Structure
Currently, the benchmarks are split into a few files, each exploring some
separate topic, like mapping a single column, combining two columns with some
operation, or computing an aggregate operation over a column. In each file,
there may be a few Enso types, each representing a separate benchmark. Usually,
we have two benchmarks for each operation type - one dealing with a primitive
value type like integers (long in the Java side) and another dealing with a
reference type like String or Date. We expect the performance
characteristics between these may differ, e.g. because Java allows to use long
without boxing, so we compare them separately.
Each Enso type for a given benchmark contains multiple methods which represent various 'approaches' to computing the same operation.
Each benchmark run has a name that consists of the type it defines it, a dot and
the method representing the particular approach, e.g.
Boxed_Map_Test.enso_map_as_vector.
Running
The runner is very simple. If any options are to be customized, the Enso file itself needs to be modified. One can run the whole project to run all the benchmarks, or run only a specific file.
Analysis
The output of the benchmarks should be saved to a file. Then that file can be
loaded using the Enso workflow in tools/performance/benchmark-analysis.
The workflow is tuned to analysing these comparative benchmarks.
At the top, one can select which file is to be analyzed. Below there is a
dropdown allowing to select one particular benchmark (represented by the type,
e.g. Boxed_Map_Test). With that selected, one can display a scatter plot
visualization comparing various approaches of that one given benchmark. On the
plot we can see runtimes of subsequent iterations. Later, we drop the first 40
iterations (the number can easily be customized in the workflow) to ensure
sufficient warm-up for each benchmark. Then a table is displayed computing the
average runtime of each approach and how they compare relative to each other - a
dropdown allows to select one benchmark that will be used as a reference point
(100%) for the average runtime comparison.