git.iko.soy/streamly
on master help sign in
Simplify/compact some sections
f309c32Harendra Kumar
benchmark/README.md+42−160
compare with master
unifiedsplit
@@ -1,204 +1,86 @@
1# Benchmarking
1# Benchmarking
2
2
3−## Benchmark Dirvers
3+## Benchmark Drivers
4
4
5Two benchmark drivers are supported:
5Two benchmark drivers are supported:
6
6
7* `tasty-bench` (default)
7* `tasty-bench` (default)
8* `gauge` (enabled by `--use-gauge` build flag)
8* `gauge` (enabled by `--use-gauge` build flag)
9
9
10−## Building a single benchmark suite
11−
12−```
13−$ cabal build streamly-benchmarks:Prelude.Serial
14−```
15−
16−or:
17−
18−```
19−$ cabal build --enable-benchmarks bench:Prelude.Serial
20−```
21−
22−## Building all benchmarks suites
23−
24−```
25−$ cabal build --enable-benchmarks streamly-benchmarks
26−```
27−
28−or:
29−
30−```
31−$ cabal build --enable-benchmarks all
32−```
33−
34−Disable optimization, quick build:
35−
36−```
37−$ cabal build --flag "-opt" ...
38−```
39−
40−## Build and run single benchmarks:
10+## Build and run benchmarks directly
41
11
42The benchmark executables are `tasty-bench` executables unless you have
12The benchmark executables are `tasty-bench` executables unless you have
43passed `--use-gauge` cabal flag when building in which case it is a
13passed `--use-gauge` cabal flag when building in which case it is a
44`gauge` executable.
14`gauge` executable.
45
15
46−For quick results you may have to use a large value for `--stdev` or use
47−`bench.sh --quick` as described in the following sections.
48−
49```
16```
50−$ cabal run bench:Prelude.Serial -- --stdev 100000
17+$ cabal run bench:Prelude.Serial # run selected
51−```
18+$ cabal run bench:Prelude.Serial -- --help # help on arguments
52−
19+$ cabal run bench:Prelude.Serial -- --stdev 100000 # specify arguments
53−`cabal bench` can be used but you cannot pass arguments (like --stdev):
54−
55−```
56−$ cabal bench Prelude.Serial
20+$ cabal run bench:Prelude.Serial --flag fusion-plugin # with fusion-plugin
57−```
58−
59−## Build and run all benchmarks
60
21
61−Don't try this, it will take too long, use the `bench.sh` method instead.
22+$ cabal build bench:Prelude.Serial # build selected
23+$ cabal build --enable-benchmarks streamly-benchmarks # build all
24+$ cabal build --enable-benchmarks all # build all, alternate method
62
25
63−```
26+$ cabal build --flag "-opt" ... # disable optimization, faster build
64−$ cabal bench all
65−```
66−
67−or:
68−
69−```
70−$ cd benchmark; cabal bench
71−```
72−
73−or this, note this command does not work from "benchmark" dir:
74−
75−```
76−$ cabal bench streamly-benchmarks
77```
27```
78
28
79## Building and Running Benchmarks with bench.sh
29## Building and Running Benchmarks with bench.sh
80
30
81`<streamly repo>/bin/bench.sh` script is the top level driver for
31`<streamly repo>/bin/bench.sh` script is the top level driver for
82running benchmarks. It runs the requested benchmarks and then creates a
32running benchmarks. It runs the requested benchmarks and then creates a
83−report from the results using the `bench-show` package. Try `bench.sh
33+report from the results using the `bench-show` package.
84−--help` for available options to run it.
34+
85−
35+IMPORTANT NOTE: The first time you run this script it may take a long
86−## bench.sh: Quick start
36+time because it has to build the `bench-report` executable which has a
87−
88−IMPORTANT NOTE: If you are using nix then you can use `--use-nix`
37+lot of dependencies. If you are using nix then use `--use-nix` flag
89−flag so that the `bench-report` executable is built using nix. That
38+for the first time so that the `bench-report` executable is built using
90−can save a lot of time compiling it. However, once it is built it
39+nix. That can save a lot of time compiling it. However, once it is built
91−will be cached in the `bin` directory of the repo and used from there
40+it will be cached in the `bin` directory of the repo and used from
92−every time. You can also build it manually from the cabal file in
41+there every time. You can also build it manually from the cabal file in
93`benchmark/bench-report` and install it in the `bin` directory.
42`benchmark/bench-report` and install it in the `bin` directory.
94
43
95−Run the default benchmark suites:
96−
97−```
98−$ bench.sh --quick
44+## bench.sh: Quick start
99−```
100−
101−You can remove the `--quick` option to run benchmark with lower speed but
102−better accuracy, or use `--slow` to even further lower the speed and increase
103−the accuracy a bit.
104−
105−Show available benchmark suites:
106−
107−```
108−$ bench.sh --benchmarks help
109−```
110−
111−Run all benchmark suites in the `serial_grp` group:
112−
113−```
114−$ bench.sh --benchmarks serial_grp
115−```
116−
117−Run `Prelude.Serial` and `Data.Parser` benchmark suites:
118−
119−```
120−$ bench.sh --benchmarks "Prelude.Serial Data.Parser"
121−```
122−
123−Run all O(1) space complexity benchmarks in `Prelude.Serial` suite:
124−
125−```
126−$ bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space
127−```
128−
129−Anything after a `--` is passed to the benchmark executable,
130−it basically selects all benchmarks starting with
131−`Prelude.Serial/o-1-space` prefix.
132−
133−Run a specific benchmark in `Prelude.Serial` suite:
134−
135−```
136−$ bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
137−```
138−
139−Run a benchmark directly instead of running it through `bench.sh`:
140−
141−```
142−$ cabal run bench:Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
143−```
144−
145−The options after `--` are the benchmark executable options.
146−
147−## Comparing results of arbitrary runs
148−
149−Note: use `--quick` and benchmark selection if you do not intend to wait for a
150−while.
151
45
152−To compare two sets of results, first run the benchmarks at the baseline
46+Useful commands:
153−commit:
154
47
155```
48```
156−$ bench.sh
49+$ bin/bench.sh --help
157−```
50+$ bin/bench.sh --quick # run all the benchmark suites
51+$ bin/bench.sh --benchmarks help # Show available benchmark suites
52+$ bin/bench.sh --benchmarks serial_grp # Run all serial benchmark suites
53+$ bin/bench.sh --benchmarks "Prelude.Serial Data.Parser" # run selected suites
54+$ bin/bench.sh --no-measure # don't run benchmarks just show previous results
158
55
159−And then run with the `--append` option at the commit that you want to compare
56+# Run all O(1) space complexity benchmarks in `Prelude.Serial` suite
160−with the baseline. It will show the comparison with the baseline:
57+$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space
161
58
59+# Run a specific benchmark in `Prelude.Serial` suite
60+$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
162```
61```
163−$ bench.sh --append
164−```
165−
166−Append just adds the next set of results in the same results file. You can keep
167−appending more results and all of them will be compared with the baseline.
168−
169−## Comparing results of two commits
170−
171−Note: use `--quick` and benchmark selection if you do not intend to wait for a
172−while.
173−
174−You can use `--compare` to compare the previous commit with the head commit:
175
62
176−```
63+Note: `bench.sh` enables fusion-plugin by default.
177−$ bench.sh --compare
178−```
179
64
180−To compare the head commit with some other base commit:
65+## Comparing results with baseline
181
66
182```
67```
183−$ bench.sh --compare --base d918833
68+# Checkout baseline commit
184−```
69+$ bin/bench.sh --quick
185
70
186−To compare two arbitrary commits:
71+# Checkout commit with new changes
72+$ bin/bench.sh --quick --append
187
73
74+# To add another result to comparisons just repeat the above command on
75+# desired commit
188```
76```
189−$ bench.sh --compare --base d918833 --candidate 38aa5f2
190−```
191−
192−Note that the above may not always work because the script and the benchmarks
193−themselves might have changed across the commits. The `--append` method is more
194−reliable to compare.
195
77
196## Comparing benchmark suites
78## Comparing benchmark suites
197
79
198First see the available benchmark suites:
80First see the available benchmark suites:
199
81
200```
82```
201−$ bench.sh --benchmarks help
83+$ bin/bench.sh --benchmarks help
202```
84```
203
85
204You will see some benchmark suites end with `_cmp`, these are comparison
86You will see some benchmark suites end with `_cmp`, these are comparison
@@ -207,7 +89,7 @@ benchmark suites in that group will be shown in the end. For example to compare
207all array benchmark suites:
89all array benchmark suites:
208
90
209```
91```
210−$ bench.sh --benchmarks array_cmp
92+$ bin/bench.sh --benchmarks array_cmp
211```
93```
212
94
213## Reporting without measuring
95## Reporting without measuring
@@ -234,7 +116,7 @@ benchmarking:
234$ cabal run bench:Prelude.Serial -- --stream-size 1000000
116$ cabal run bench:Prelude.Serial -- --stream-size 1000000
235```
117```
236
118
237−### Unicode input
119+### External input file
238
120
239In the `FileSystem.Handle` benchmark you can specify the input file as an
121In the `FileSystem.Handle` benchmark you can specify the input file as an
240environment variable:
122environment variable: