git.iko.soy/streamly
on 62514f2 help sign in
streamly/benchmark/README.md
243 lines 6.0 kB history raw
1## Building a single benchmark suite
3```
4$ cabal build streamly-benchmarks:Prelude.Serial
5```
7or:
9```
10$ cabal build --enable-benchmarks bench:Prelude.Serial
11```
13## Building all benchmarks suites
15```
16$ cabal build --enable-benchmarks streamly-benchmarks
17```
19or:
21```
22$ cabal build --enable-benchmarks all
23```
25Disable optimization, quick build:
27```
28$ cabal build --flag "-opt" ...
29```
31## Build and run single benchmarks:
33For quick results you may have to use `--quick` and possibly some other
34`gauge` options, or better use `bench.sh --quick` as described in the
35following sections.
37```
38$ cabal run bench:Prelude.Serial -- --quick
39```
41`cabal bench` can be used but you cannot pass arguments (like --quick):
43```
44$ cabal bench Prelude.Serial
45```
47## Build and run all benchmarks
49Don't try this, it will take too long, use the `bench.sh` method instead.
51```
52$ cabal bench all
53```
55or:
57```
58$ cd benchmark; cabal bench
59```
61or this, note this command does not work from "benchmark" dir:
63```
64$ cabal bench streamly-benchmarks
65```
67## Building and Running Benchmarks with bench.sh
69`<streamly repo>/bin/bench.sh` script is the top level driver for
70running benchmarks. It runs the requested benchmarks and then creates a
71report from the results using the `bench-show` package. Try `bench.sh
72--help` for available options to run it.
74## bench.sh: Quick start
76Run the default benchmark suites:
78```
79$ bench.sh --quick
80```
82You can remove the `--quick` option to run benchmark with lower speed but
83better accuracy, or use `--slow` to even further lower the speed and increase
84the accuracy a bit.
86Show available benchmark suites:
88```
89$ bench.sh --benchmarks help
90```
92Run all benchmark suites in the `serial_grp` group:
94```
95$ bench.sh --benchmarks serial_grp
96```
98Run `Prelude.Serial` and `Data.Parser` benchmark suites:
100```
101$ bench.sh --benchmarks "Prelude.Serial Data.Parser"
102```
104Run all O(1) space complexity benchmarks in `Prelude.Serial` suite:
106```
107$ bench.sh --benchmarks Prelude.Serial -- Prelude.Serial/o-1-space
108```
110Anything after a `--` is passed to the benchmark executable,
111it basically selects all benchmarks starting with
112`Prelude.Serial/o-1-space` prefix.
114Run a specific benchmark in `Prelude.Serial` suite:
116```
117$ bench.sh --benchmarks Prelude.Serial -- Prelude.Serial/o-1-space/generation/unfoldr
118```
120Run a benchmark directly instead of running it through `bench.sh`:
122```
123$ cabal run bench:Prelude.Serial -- --quick Prelude.Serial/o-1-space/generation/unfoldr
124```
126The options after `--` are the benchmark executable options.
128## Comparing results of arbitrary runs
130Note: use `--quick` and benchmark selection if you do not intend to wait for a
131while.
133To compare two sets of results, first run the benchmarks at the baseline
134commit:
136```
137$ bench.sh
138```
140And then run with the `--append` option at the commit that you want to compare
141with the baseline. It will show the comparison with the baseline:
143```
144$ bench.sh --append
145```
147Append just adds the next set of results in the same results file. You can keep
148appending more results and all of them will be compared with the baseline.
150## Comparing results of two commits
152Note: use `--quick` and benchmark selection if you do not intend to wait for a
153while.
155You can use `--compare` to compare the previous commit with the head commit:
157```
158$ bench.sh --compare
159```
161To compare the head commit with some other base commit:
163```
164$ bench.sh --compare --base d918833
165```
167To compare two arbitrary commits:
169```
170$ bench.sh --compare --base d918833 --candidate 38aa5f2
171```
173Note that the above may not always work because the script and the benchmarks
174themselves might have changed across the commits. The `--append` method is more
175reliable to compare.
177## Comparing benchmark suites
179First see the available benchmark suites:
181```
182$ bench.sh --benchmarks help
183```
185You will see some benchmark suites end with `_cmp`, these are comparison
186groups. If you run a comparison group benchmark, comparison of all the
187benchmark suites in that group will be shown in the end. For example to compare
188all array benchmark suites:
190```
191$ bench.sh --benchmarks array_cmp
192```
194## Reporting without measuring
196You can use the `--no-measure` option to report the already measured results in
197the benchmarks results file. A results file may collect an arbitrary number of
198results by running with `--append` multiple times. Each benchmark has its own
199results file, for example the `Prelude.Serial` benchmark has the results file at
200`charts/Prelude.Serial/results.csv`.
202You can also manually edit the file to remove a set of results if you like or
203to append results from previously saved results or from some other results
204file. After editing you can run `bench.sh` with the `--no-measure` option to
205see the reports corresponding to the results.
207## Additional benchmark configuration
209### Stream size
211You can specify the stream size (default is 100000) to be used for
212benchmarking:
214```
215$ cabal run bench:Prelude.Serial -- --quick --stream-size 1000000
216```
218### Unicode input
220In the `FileSystem.Handle` benchmark you can specify the input file as an
221environment variable:
223```
224$ export Benchmark_FileSystem_Handle_InputFile=./gutenberg-500.txt
225$ cabal run FileSystem.Handle -- --quick FileSystem.Handle/o-1-space/reduce/read/S.splitOnSeq
226```
228The automatic tests do not test unicode input, this option is useful to specify
229a unicode text file manually.
231## Benchmarking notes
233We use gauge instead of criterion for benchmarking. We have fixed
234several issues in gauge inherited from criterion. We have added several
235features as well e.g. rusage stats, running benchmarks in an isolated
236process etc which are crucial to our benchmarking analysis process.
238### Gotchas
240Gauge forces a GC before and after the measurement. However, we have observed
241that sometimes the GC stats may not be accurate when the number of iterations
242in the measurement is small (e.g. 1 iteration). In such cases usually the
243number of GCs and GC times would also be 0.