c1259cb4d2
After a discussion, I was really curious that our panics are supposed to be almost free - and while trusting that statement, it was really hard to believe - so I wanted to see for myself - knowing that an experiment is the most robust source of this kind of information - testing that in practice. So I wrote a benchmark comparing various ways of reporting errors, also testing them both at 'shallow' and 'deep' stack traces (adding 200 additional frames) - to see how stack depth affects them, if at all. The panics are indeed blazing fast! Kudos to the engine team. However, it seems that our dataflow errors are relatively slow (and we tend to use them _more_ than panics and want to be using them more and more). This uncovers a possible optimization opportunity. Can we make them as fast as panics?? Analysis of the benchmark results in comment below. |
||
|---|---|---|
| .. | ||
| polyglot-sources/exploratory-benchmark-java-helpers/src/main/java/org/enso/exploratory_benchmark_helpers | ||
| src | ||
| package.yaml | ||
| README.md | ||
Exploring Table operation performance
These benchmarks are used to compare various approaches to computing operations on Table columns, to find out what best practices should we use for these and find venues for optimization of the language and Table implementation.
These benchmarks are not meant to be used for tracking performance of the
current implementation itself. That is supposed to be done by another project -
Table_Benchmarks.
Structure
Currently, the benchmarks are split into a few files, each exploring some
separate topic, like mapping a single column, combining two columns with some
operation, or computing an aggregate operation over a column. In each file,
there may be a few Enso types, each representing a separate benchmark. Usually,
we have two benchmarks for each operation type - one dealing with a primitive
value type like integers (long in the Java side) and another dealing with a
reference type like String or Date. We expect the performance
characteristics between these may differ, e.g. because Java allows to use long
without boxing, so we compare them separately.
Each Enso type for a given benchmark contains multiple methods which represent various 'approaches' to computing the same operation.
Each benchmark run has a name that consists of the type it defines it, a dot and
the method representing the particular approach, e.g.
Boxed_Map_Test.enso_map_as_vector.
Running
The runner is very simple. If any options are to be customized, the Enso file itself needs to be modified. One can run the whole project to run all the benchmarks, or run only a specific file.
Analysis
The output of the benchmarks should be saved to a file. Then that file can be
loaded using the Enso workflow in tools/performance/benchmark-analysis.
The workflow is tuned to analysing these comparative benchmarks.
At the top, one can select which file is to be analyzed. Below there is a
dropdown allowing to select one particular benchmark (represented by the type,
e.g. Boxed_Map_Test). With that selected, one can display a scatter plot
visualization comparing various approaches of that one given benchmark. On the
plot we can see runtimes of subsequent iterations. Later, we drop the first 40
iterations (the number can easily be customized in the workflow) to ensure
sufficient warm-up for each benchmark. Then a table is displayed computing the
average runtime of each approach and how they compare relative to each other - a
dropdown allows to select one benchmark that will be used as a reference point
(100%) for the average runtime comparison.