git.iko.soy/streamly
on ec19422 help sign in
streamly/benchmark/README.md
229 lines 5.4 kB history raw
1## Building a single benchmark suite
3```
4$ cabal build streamly-benchmarks:Prelude.Serial
5```
7or:
9```
10$ cabal build --enable-benchmarks bench:Prelude.Serial
11```
13## Building all benchmarks suites
15```
16$ cabal build --enable-benchmarks streamly-benchmarks
17```
19or:
21```
22$ cabal build --enable-benchmarks all
23```
25Disable optimization, quick build:
27```
28$ cabal build --flag "-opt" ...
29```
31## Build and run single benchmarks:
33For quick results you may have to use `--quick` and possibly some other
34`gauge` options, or better use `bench.sh --quick` as described in the
35following sections.
37```
38$ cabal run bench:Prelude.Serial -- --quick
39```
41`cabal bench` can be used but you cannot pass arguments (like --quick):
43```
44$ cabal bench Prelude.Serial
45```
47## Build and run all benchmarks
49Don't try this, it will take too long, use the `bench.sh` method instead.
51```
52$ cabal bench all
53```
55or:
57```
58$ cd benchmark; cabal bench
59```
61or this, note this command does not work from "benchmark" dir:
63```
64$ cabal bench streamly-benchmarks
65```
67## Building and Running Benchmarks with bench.sh
69`<streamly repo>/bin/bench.sh` script is the top level driver for
70running benchmarks. It runs the requested benchmarks and then creates a
71report from the results using the `bench-show` package. Try `bench.sh
72--help` for available options to run it.
74## bench.sh: Quick start
76Run the default benchmark suites:
78```
79$ bench.sh --quick
80```
82You can remove the `--quick` option to run benchmark with lower speed but
83better accuracy, or use `--slow` to even further lower the speed and increase
84the accuracy a bit.
86Show available benchmark suites:
88```
89$ bench.sh --benchmarks help
90```
92Run all benchmark suites in the `serial_grp` group:
94```
95$ bench.sh --benchmarks serial_grp
96```
98Run `Prelude.Serial` and `Data.Parser` benchmark suites:
100```
101$ bench.sh --benchmarks "Prelude.Serial Data.Parser"
102```
104Run all O(1) space complexity benchmarks in `Prelude.Serial` suite:
106```
107$ bench.sh --benchmarks Prelude.Serial -- Prelude.Serial/o-1-space
108```
110Anything after a `--` is passed to the benchmark executable,
111it basically selects all benchmarks starting with
112`Prelude.Serial/o-1-space` prefix.
114Run a specific benchmark in `Prelude.Serial` suite:
116```
117$ bench.sh --benchmarks Prelude.Serial -- Prelude.Serial/o-1-space/generation/unfoldr
118```
120Run a benchmark directly instead of running it through `bench.sh`:
122```
123$ cabal run bench:Prelude.Serial -- --quick Prelude.Serial/o-1-space/generation/unfoldr
124```
126The options after `--` are the benchmark executable options.
128## Comparing results of arbitrary runs
130Note: use `--quick` and benchmark selection if you do not intend to wait for a
131while.
133To compare two sets of results, first run the benchmarks at the baseline
134commit:
136```
137$ bench.sh
138```
140And then run with the `--append` option at the commit that you want to compare
141with the baseline. It will show the comparison with the baseline:
143```
144$ bench.sh --append
145```
147Append just adds the next set of results in the same results file. You can keep
148appending more results and all of them will be compared with the baseline.
150## Comparing results of two commits
152Note: use `--quick` and benchmark selection if you do not intend to wait for a
153while.
155You can use `--compare` to compare the previous commit with the head commit:
157```
158$ bench.sh --compare
159```
161To compare the head commit with some other base commit:
163```
164$ bench.sh --compare --base d918833
165```
167To compare two arbitrary commits:
169```
170$ bench.sh --compare --base d918833 --candidate 38aa5f2
171```
173Note that the above may not always work because the script and the benchmarks
174themselves might have changed across the commits. The `--append` method is more
175reliable to compare.
177## Comparing benchmark suites
179First see the available benchmark suites:
181```
182$ bench.sh --benchmarks help
183```
185You will see some benchmark suites end with `_cmp`, these are comparison
186groups. If you run a comparison group benchmark, comparison of all the
187benchmark suites in that group will be shown in the end. For example to compare
188all array benchmark suites:
190```
191$ bench.sh --benchmarks array_cmp
192```
194## Reporting without measuring
196You can use the `--no-measure` option to report the already measured results in
197the benchmarks results file. A results file may collect an arbitrary number of
198results by running with `--append` multiple times. Each benchmark has its own
199results file, for example the `Prelude.Serial` benchmark has the results file at
200`charts/Prelude.Serial/results.csv`.
202You can also manually edit the file to remove a set of results if you like or
203to append results from previously saved results or from some other results
204file. After editing you can run `bench.sh` with the `--no-measure` option to
205see the reports corresponding to the results.
207## Additional benchmark configuration
209### Stream size
211You can specify the stream size (default is 100000) to be used for
212benchmarking:
214```
215$ cabal run bench:Prelude.Serial -- --quick --stream-size 1000000
216```
218### Unicode input
220In the `FileSystem.Handle` benchmark you can specify the input file as an
221environment variable:
223```
224$ export Benchmark_FileSystem_Handle_InputFile=./gutenberg-500.txt
225$ cabal run FileSystem.Handle -- --quick FileSystem.Handle/o-1-space/reduce/read/S.splitOnSeq
226```
228The automatic tests do not test unicode input, this option is useful to specify
229a unicode text file manually.