git.iko.soy/streamly
on 3feeec2 help sign in
streamly/benchmark/README.md
254 lines 6.2 kB history raw
1# Benchmarking
3## Benchmark Dirvers
5Two benchmark drivers are supported:
7* `tasty-bench` (default)
8* `gauge` (enabled by `--use-gauge` build flag)
10## Building a single benchmark suite
12```
13$ cabal build streamly-benchmarks:Prelude.Serial
14```
16or:
18```
19$ cabal build --enable-benchmarks bench:Prelude.Serial
20```
22## Building all benchmarks suites
24```
25$ cabal build --enable-benchmarks streamly-benchmarks
26```
28or:
30```
31$ cabal build --enable-benchmarks all
32```
34Disable optimization, quick build:
36```
37$ cabal build --flag "-opt" ...
38```
40## Build and run single benchmarks:
42The benchmark executables are `tasty-bench` executables unless you have
43passed `--use-gauge` cabal flag when building in which case it is a
44`gauge` executable.
46For quick results you may have to use a large value for `--stdev` or use
47`bench.sh --quick` as described in the following sections.
49```
50$ cabal run bench:Prelude.Serial -- --stdev 100000
51```
53`cabal bench` can be used but you cannot pass arguments (like --stdev):
55```
56$ cabal bench Prelude.Serial
57```
59## Build and run all benchmarks
61Don't try this, it will take too long, use the `bench.sh` method instead.
63```
64$ cabal bench all
65```
67or:
69```
70$ cd benchmark; cabal bench
71```
73or this, note this command does not work from "benchmark" dir:
75```
76$ cabal bench streamly-benchmarks
77```
79## Building and Running Benchmarks with bench.sh
81`<streamly repo>/bin/bench.sh` script is the top level driver for
82running benchmarks. It runs the requested benchmarks and then creates a
83report from the results using the `bench-show` package. Try `bench.sh
84--help` for available options to run it.
86## bench.sh: Quick start
88Run the default benchmark suites:
90```
91$ bench.sh --quick
92```
94You can remove the `--quick` option to run benchmark with lower speed but
95better accuracy, or use `--slow` to even further lower the speed and increase
96the accuracy a bit.
98Show available benchmark suites:
100```
101$ bench.sh --benchmarks help
102```
104Run all benchmark suites in the `serial_grp` group:
106```
107$ bench.sh --benchmarks serial_grp
108```
110Run `Prelude.Serial` and `Data.Parser` benchmark suites:
112```
113$ bench.sh --benchmarks "Prelude.Serial Data.Parser"
114```
116Run all O(1) space complexity benchmarks in `Prelude.Serial` suite:
118```
119$ bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space
120```
122Anything after a `--` is passed to the benchmark executable,
123it basically selects all benchmarks starting with
124`Prelude.Serial/o-1-space` prefix.
126Run a specific benchmark in `Prelude.Serial` suite:
128```
129$ bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space/generation/unfoldr
130```
132Run a benchmark directly instead of running it through `bench.sh`:
134```
135$ cabal run bench:Prelude.Serial -- Prelude.Serial/o-1-space/generation/unfoldr
136```
138The options after `--` are the benchmark executable options.
140## Comparing results of arbitrary runs
142Note: use `--quick` and benchmark selection if you do not intend to wait for a
143while.
145To compare two sets of results, first run the benchmarks at the baseline
146commit:
148```
149$ bench.sh
150```
152And then run with the `--append` option at the commit that you want to compare
153with the baseline. It will show the comparison with the baseline:
155```
156$ bench.sh --append
157```
159Append just adds the next set of results in the same results file. You can keep
160appending more results and all of them will be compared with the baseline.
162## Comparing results of two commits
164Note: use `--quick` and benchmark selection if you do not intend to wait for a
165while.
167You can use `--compare` to compare the previous commit with the head commit:
169```
170$ bench.sh --compare
171```
173To compare the head commit with some other base commit:
175```
176$ bench.sh --compare --base d918833
177```
179To compare two arbitrary commits:
181```
182$ bench.sh --compare --base d918833 --candidate 38aa5f2
183```
185Note that the above may not always work because the script and the benchmarks
186themselves might have changed across the commits. The `--append` method is more
187reliable to compare.
189## Comparing benchmark suites
191First see the available benchmark suites:
193```
194$ bench.sh --benchmarks help
195```
197You will see some benchmark suites end with `_cmp`, these are comparison
198groups. If you run a comparison group benchmark, comparison of all the
199benchmark suites in that group will be shown in the end. For example to compare
200all array benchmark suites:
202```
203$ bench.sh --benchmarks array_cmp
204```
206## Reporting without measuring
208You can use the `--no-measure` option to report the already measured results in
209the benchmarks results file. A results file may collect an arbitrary number of
210results by running with `--append` multiple times. Each benchmark has its own
211results file, for example the `Prelude.Serial` benchmark has the results file at
212`charts/Prelude.Serial/results.csv`.
214You can also manually edit the file to remove a set of results if you like or
215to append results from previously saved results or from some other results
216file. After editing you can run `bench.sh` with the `--no-measure` option to
217see the reports corresponding to the results.
219## Additional benchmark configuration
221### Stream size
223You can specify the stream size (default is 100000) to be used for
224benchmarking:
226```
227$ cabal run bench:Prelude.Serial -- --stream-size 1000000
228```
230### Unicode input
232In the `FileSystem.Handle` benchmark you can specify the input file as an
233environment variable:
235```
236$ export Benchmark_FileSystem_Handle_InputFile=./gutenberg-500.txt
237$ cabal run FileSystem.Handle -- FileSystem.Handle/o-1-space/reduce/read/S.splitOnSeq
238```
240The automatic tests do not test unicode input, this option is useful to specify
241a unicode text file manually.
243## Benchmarking notes
245We run each benchmark in an isolated process to minimize interference
246of benchmarks and to be able to control the RTS memory restrictions per
247benchmark.
249### Gotchas
251Gauge forces a GC before and after the measurement. However, we have observed
252that sometimes the GC stats may not be accurate when the number of iterations
253in the measurement is small (e.g. 1 iteration). In such cases usually the
254number of GCs and GC times would also be 0.