@@ -1,204 +1,86 @@
11# Benchmarking
22
3−## Benchmark Dirvers
3+## Benchmark Drivers
44
55Two benchmark drivers are supported:
66
77* `tasty-bench` (default)
88* `gauge` (enabled by `--use-gauge` build flag)
99
10−## Building a single benchmark suite
11−
12−```
13−$ cabal build streamly-benchmarks:Prelude.Serial
14−```
15−
16−or:
17−
18−```
19−$ cabal build --enable-benchmarks bench:Prelude.Serial
20−```
21−
22−## Building all benchmarks suites
23−
24−```
25−$ cabal build --enable-benchmarks streamly-benchmarks
26−```
27−
28−or:
29−
30−```
31−$ cabal build --enable-benchmarks all
32−```
33−
34−Disable optimization, quick build:
35−
36−```
37−$ cabal build --flag "-opt" ...
38−```
39−
40−## Build and run single benchmarks:
10+## Build and run benchmarks directly
4111
4212The benchmark executables are `tasty-bench` executables unless you have
4313passed `--use-gauge` cabal flag when building in which case it is a
4414`gauge` executable.
4515
46−For quick results you may have to use a large value for `--stdev` or use
47−`bench.sh --quick` as described in the following sections.
48−
4916```
50−$ cabal run bench:Prelude.Serial -- --stdev 100000
51−```
52−
53−`cabal bench` can be used but you cannot pass arguments (like --stdev):
54−
55−```
56−$ cabal bench Prelude.Serial
57−```
58−
59−## Build and run all benchmarks
17+$ cabal run bench:Prelude.Serial # run selected
18+$ cabal run bench:Prelude.Serial -- --help # help on arguments
19+$ cabal run bench:Prelude.Serial -- --stdev 100000 # specify arguments
20+$ cabal run bench:Prelude.Serial --flag fusion-plugin # with fusion-plugin
6021
61−Don't try this, it will take too long, use the `bench.sh` method instead.
22+$ cabal build bench:Prelude.Serial # build selected
23+$ cabal build --enable-benchmarks streamly-benchmarks # build all
24+$ cabal build --enable-benchmarks all # build all, alternate method
6225
63−```
64−$ cabal bench all
65−```
66−
67−or:
68−
69−```
70−$ cd benchmark; cabal bench
71−```
72−
73−or this, note this command does not work from "benchmark" dir:
74−
75−```
76−$ cabal bench streamly-benchmarks
26+$ cabal build --flag "-opt" ... # disable optimization, faster build
7727```
7828
7929## Building and Running Benchmarks with bench.sh
8030
8131`<streamly repo>/bin/bench.sh` script is the top level driver for
8232running benchmarks. It runs the requested benchmarks and then creates a
83−report from the results using the `bench-show` package. Try `bench.sh
84−--help` for available options to run it.
85−
86−## bench.sh: Quick start
87−
88−IMPORTANT NOTE: If you are using nix then you can use `--use-nix`
89−flag so that the `bench-report` executable is built using nix. That
90−can save a lot of time compiling it. However, once it is built it
91−will be cached in the `bin` directory of the repo and used from there
92−every time. You can also build it manually from the cabal file in
33+report from the results using the `bench-show` package.
34+
35+IMPORTANT NOTE: The first time you run this script it may take a long
36+time because it has to build the `bench-report` executable which has a
37+lot of dependencies. If you are using nix then use `--use-nix` flag
38+for the first time so that the `bench-report` executable is built using
39+nix. That can save a lot of time compiling it. However, once it is built
40+it will be cached in the `bin` directory of the repo and used from
41+there every time. You can also build it manually from the cabal file in
9342`benchmark/bench-report` and install it in the `bin` directory.
9443
95−Run the default benchmark suites:
96−
97−```
98−$ bench.sh --quick
99−```
100−
101−You can remove the `--quick` option to run benchmark with lower speed but
102−better accuracy, or use `--slow` to even further lower the speed and increase
103−the accuracy a bit.
104−
105−Show available benchmark suites:
106−
107−```
108−$ bench.sh --benchmarks help
109−```
110−
111−Run all benchmark suites in the `serial_grp` group:
112−
113−```
114−$ bench.sh --benchmarks serial_grp
115−```
116−
117−Run `Prelude.Serial` and `Data.Parser` benchmark suites:
118−
119−```
120−$ bench.sh --benchmarks "Prelude.Serial Data.Parser"
121−```
122−
123−Run all O(1) space complexity benchmarks in `Prelude.Serial` suite:
124−
125−```
126−$ bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space
127−```
128−
129−Anything after a `--` is passed to the benchmark executable,
130−it basically selects all benchmarks starting with
131−`Prelude.Serial/o-1-space` prefix.
132−
133−Run a specific benchmark in `Prelude.Serial` suite:
134−
135−```
136−$ bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
137−```
138−
139−Run a benchmark directly instead of running it through `bench.sh`:
140−
141−```
142−$ cabal run bench:Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
143−```
144−
145−The options after `--` are the benchmark executable options.
146−
147−## Comparing results of arbitrary runs
148−
149−Note: use `--quick` and benchmark selection if you do not intend to wait for a
150−while.
44+## bench.sh: Quick start
15145
152−To compare two sets of results, first run the benchmarks at the baseline
153−commit:
46+Useful commands:
15447
15548```
156−$ bench.sh
157−```
49+$ bin/bench.sh --help
50+$ bin/bench.sh --quick # run all the benchmark suites
51+$ bin/bench.sh --benchmarks help # Show available benchmark suites
52+$ bin/bench.sh --benchmarks serial_grp # Run all serial benchmark suites
53+$ bin/bench.sh --benchmarks "Prelude.Serial Data.Parser" # run selected suites
54+$ bin/bench.sh --no-measure # don't run benchmarks just show previous results
15855
159−And then run with the `--append` option at the commit that you want to compare
160−with the baseline. It will show the comparison with the baseline:
56+# Run all O(1) space complexity benchmarks in `Prelude.Serial` suite
57+$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space
16158
59+# Run a specific benchmark in `Prelude.Serial` suite
60+$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
16261```
163−$ bench.sh --append
164−```
165−
166−Append just adds the next set of results in the same results file. You can keep
167−appending more results and all of them will be compared with the baseline.
168−
169−## Comparing results of two commits
170−
171−Note: use `--quick` and benchmark selection if you do not intend to wait for a
172−while.
173−
174−You can use `--compare` to compare the previous commit with the head commit:
17562
176−```
177−$ bench.sh --compare
178−```
63+Note: `bench.sh` enables fusion-plugin by default.
17964
180−To compare the head commit with some other base commit:
65+## Comparing results with baseline
18166
18267```
183−$ bench.sh --compare --base d918833
184−```
68+# Checkout baseline commit
69+$ bin/bench.sh --quick
18570
186−To compare two arbitrary commits:
71+# Checkout commit with new changes
72+$ bin/bench.sh --quick --append
18773
74+# To add another result to comparisons just repeat the above command on
75+# desired commit
18876```
189−$ bench.sh --compare --base d918833 --candidate 38aa5f2
190−```
191−
192−Note that the above may not always work because the script and the benchmarks
193−themselves might have changed across the commits. The `--append` method is more
194−reliable to compare.
19577
19678## Comparing benchmark suites
19779
19880First see the available benchmark suites:
19981
20082```
201−$ bench.sh --benchmarks help
83+$ bin/bench.sh --benchmarks help
20284```
20385
20486You will see some benchmark suites end with `_cmp`, these are comparison
@@ -207,7 +89,7 @@ benchmark suites in that group will be shown in the end. For example to compare
20789all array benchmark suites:
20890
20991```
210−$ bench.sh --benchmarks array_cmp
92+$ bin/bench.sh --benchmarks array_cmp
21193```
21294
21395## Reporting without measuring
@@ -234,7 +116,7 @@ benchmarking:
234116$ cabal run bench:Prelude.Serial -- --stream-size 1000000
235117```
236118
237−### Unicode input
119+### External input file
238120
239121In the `FileSystem.Handle` benchmark you can specify the input file as an
240122environment variable: