@@ -1,18 +1,7 @@
11# Benchmarking
22
3−## Benchmark Drivers
4−
5−Two benchmark drivers are supported:
6−
7−* `tasty-bench` (default)
8−* `gauge` (enabled by `--use-gauge` build flag)
9−
103## Build and run benchmarks directly
114
12−The benchmark executables are `tasty-bench` executables unless you have
13−passed `--use-gauge` cabal flag when building in which case it is a
14−`gauge` executable.
15−
165```
176$ cabal run bench:Prelude.Serial # run selected
187$ cabal run bench:Prelude.Serial -- --help # help on arguments
@@ -26,53 +15,68 @@ $ cabal build --enable-benchmarks all # build all, alternate method
2615$ cabal build --flag "-opt" ... # disable optimization, faster build
2716```
2817
29−## Building and Running Benchmarks with bench.sh
18+## Building and Running Benchmarks with bench-runner
3019
31−`<streamly repo>/bin/bench.sh` script is the top level driver for
20+The executable `bench-runner` is the top level driver for
3221running benchmarks. It runs the requested benchmarks and then creates a
3322report from the results using the `bench-show` package.
3423
35−IMPORTANT NOTE: The first time you run this script it may take a long
24+IMPORTANT NOTE: The first time you run this executable it may take a long
3625time because it has to build the `bench-report` executable which has a
37−lot of dependencies. If you are using nix then use `--use-nix` flag
38−for the first time so that the `bench-report` executable is built using
39−nix. That can save a lot of time compiling it. However, once it is built
40−it will be cached in the `bin` directory of the repo and used from
41−there every time. You can also build it manually from the cabal file in
42−`benchmark/bench-report` and install it in the `bin` directory.
26+lot of dependencies.
27+
28+If you are using nix then use `benchmark/bench-runner/default.nix`. Build it
29+manually and install it in the root of the repository like so,
30+```
31+$ cd benchmark/bench-runner
32+$ nix-shell --run 'cabal install bench-runner --installdir=../../ --overwrite-policy=always'
33+$ cd ../../
34+```
35+
36+You can also use `cabal.project.report` to directly build an run the `bench-runner` like so,
37+```
38+$ cabal run bench-runner --project-file=cabal.project.report -- <bench-runner-args>
39+```
4340
4441## bench.sh: Quick start
4542
43+Assuming `bench-runner` is the executable. You can replace `bench-runner` with
44+`cabal run bench-runner --project-file=cabal.project.report --`
45+
4646Useful commands:
4747
4848```
49−$ bin/bench.sh --help
50−$ bin/bench.sh --quick # run all the benchmark suites
51−$ bin/bench.sh --benchmarks help # Show available benchmark suites
52−$ bin/bench.sh --benchmarks serial_grp # Run all serial benchmark suites
53−$ bin/bench.sh --benchmarks "Prelude.Serial Data.Parser" # run selected suites
54−$ bin/bench.sh --no-measure # don't run benchmarks just show previous results
49+$ ./bench-runner --help
50+$ ./bench-runner --quick # run all the benchmark suites
51+$ ./bench-runner --targets help # Show available benchmark suites
52+$ ./bench-runner --targets serial_grp # Run all serial benchmark suites
53+$ ./bench-runner --targets "Prelude.Serial Data.Parser" # run selected suites
54+$ ./bench-runner --no-measure # don't run benchmarks just show previous results
5555
5656# Run all O(1) space complexity benchmarks in `Prelude.Serial` suite
57−$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space
57+$ ./bench-runner --targets Prelude.Serial --prefix Prelude.Serial/o-1-space
5858
5959# Run a specific benchmark in `Prelude.Serial` suite
60−$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
60+$ ./bench-runner --targets Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
6161```
6262
63−Note: `bench.sh` enables fusion-plugin by default.
63+Note: `bench-runner` enables fusion-plugin by default.
6464
6565## Comparing results with baseline
6666
6767```
6868# Checkout baseline commit
69−$ bin/bench.sh --quick
69+$ ./bench-runner --quick
7070
7171# Checkout commit with new changes
72−$ bin/bench.sh --quick --append
72+$ ./bench-runner --quick --append
7373
7474# To add another result to comparisons just repeat the above command on
7575# desired commit
76+
77+# To display the current results without running the benchmarks.
78+# See "Reporting without measuring" for more info.
79+$ ./bench-runner --no-measure
7680```
7781
7882## Comparing benchmark suites
@@ -80,7 +84,7 @@ $ bin/bench.sh --quick --append
8084First see the available benchmark suites:
8185
8286```
83−$ bin/bench.sh --benchmarks help
87+$ ./bench-runner --targets help
8488```
8589
8690You will see some benchmark suites end with `_cmp`, these are comparison
@@ -89,7 +93,7 @@ benchmark suites in that group will be shown in the end. For example to compare
8993all array benchmark suites:
9094
9195```
92−$ bin/bench.sh --benchmarks array_cmp
96+$ ./bench-runner --targets array_cmp
9397```
9498
9599## Reporting without measuring
@@ -102,7 +106,7 @@ results file, for example the `Prelude.Serial` benchmark has the results file at
102106
103107You can also manually edit the file to remove a set of results if you like or
104108to append results from previously saved results or from some other results
105−file. After editing you can run `bench.sh` with the `--no-measure` option to
109+file. After editing you can run `bench-runner` with the `--no-measure` option to
106110see the reports corresponding to the results.
107111
108112## Additional benchmark configuration
@@ -134,7 +138,7 @@ a unicode text file manually.
134138To run the unicode benchmarks on valid utf8 input, you can do the following,
135139
136140```
137−$ Benchmark_FileSystem_Handle_InputFile=<valid-unicode-filepath> bin/bench.sh --benchmarks Unicode.Stream --cabal-build-options "-f include-strict-utf8"
141+$ Benchmark_FileSystem_Handle_InputFile=<valid-unicode-filepath> ./bench-runner --benchmarks Unicode.Stream --cabal-build-options "-f include-strict-utf8"
138142```
139143
140144## Benchmarking notes
@@ -145,7 +149,8 @@ benchmark.
145149
146150### Gotchas
147151
148−Gauge forces a GC before and after the measurement. However, we have observed
152+The benchmark driver forces a GC before and after the measurement.
153+However, we have observed
149154that sometimes the GC stats may not be accurate when the number of iterations
150155in the measurement is small (e.g. 1 iteration). In such cases usually the
151156number of GCs and GC times would also be 0.
@@ -168,10 +173,10 @@ freeze file so that the versions of all libraries are pinned.
168173
169174There are two ways to find problematic code:
170175
171−1. Run performance benchmarks using `bench.sh`, select the benchmarks
176+1. Run performance benchmarks using `bench-runner`, select the benchmarks
172177 that are taking more than expected time.
1731782. When making a new change, compare with the baseline and select benchmarks
174− with the most regression reported by `bench.sh`.
179+ with the most regression reported by `bench-runner`.
175180
176181Number of allocations are the most stable measure that do not vary from
177182run to run. `cpuTime` and `bytesCopied` may vary. When comparing two