git.iko.soy/streamly
on ffca3ac help sign in
streamly/docs/Developer/Benchmarks.md
326 lines 11 kB history raw
1# Benchmarking
3## Build and run benchmarks directly
5```
6$ cabal run bench:Prelude.Serial # run selected
7$ cabal run bench:Prelude.Serial -- --help # help on arguments
8$ cabal run bench:Prelude.Serial -- --stdev 100000 # specify arguments
9$ cabal run bench:Prelude.Serial --flag fusion-plugin # with fusion-plugin
11$ cabal build bench:Prelude.Serial # build selected
12$ cabal build --enable-benchmarks streamly-benchmarks # build all
13$ cabal build --enable-benchmarks all # build all, alternate method
15$ cabal build --flag "-opt" ... # disable optimization, faster build
16```
18## Building and Running Benchmarks with bench-runner
20The executable `bench-runner` is the top level driver for
21running benchmarks. It runs the requested benchmarks and then creates a
22report from the results using the `bench-show` package.
24IMPORTANT NOTE: The first time you run this executable it may take a long
25time because it has to build the `bench-report` executable which has a
26lot of dependencies.
28If you are using nix then use `benchmark/bench-runner/default.nix`. Build it
29manually and install it in the root of the repository like so,
30```
31$ cd benchmark/bench-runner
32$ nix-shell --run 'cabal install bench-runner --installdir=../../ --overwrite-policy=always'
33$ cd ../../
34```
36You can also use `cabal.project.report` to directly build an run the `bench-runner` like so,
37```
38$ cabal run bench-runner --project-file=cabal.project.report -- <bench-runner-args>
39```
41## bench.sh: Quick start
43Assuming `bench-runner` is the executable. You can replace `bench-runner` with
44`cabal run bench-runner --project-file=cabal.project.report --`
46Useful commands:
48```
49$ ./bench-runner --help
50$ ./bench-runner --quick # run all the benchmark suites
51$ ./bench-runner --targets help # Show available benchmark suites
52$ ./bench-runner --targets serial_grp # Run all serial benchmark suites
53$ ./bench-runner --targets "Prelude.Serial Data.Parser" # run selected suites
54$ ./bench-runner --no-measure # don't run benchmarks just show previous results
56# Run all O(1) space complexity benchmarks in `Prelude.Serial` suite
57$ ./bench-runner --targets Prelude.Serial --prefix Prelude.Serial/o-1-space
59# Run a specific benchmark in `Prelude.Serial` suite
60$ ./bench-runner --targets Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
61```
63Note: `bench-runner` enables fusion-plugin by default.
65## Comparing results with baseline
67```
68# Checkout baseline commit
69$ ./bench-runner --quick
71# Checkout commit with new changes
72$ ./bench-runner --quick --append
74# To add another result to comparisons just repeat the above command on
75# desired commit
77# To display the current results without running the benchmarks.
78# See "Reporting without measuring" for more info.
79$ ./bench-runner --no-measure
80```
82## Comparing benchmark suites
84First see the available benchmark suites:
86```
87$ ./bench-runner --targets help
88```
90You will see some benchmark suites end with `_cmp`, these are comparison
91groups. If you run a comparison group benchmark, comparison of all the
92benchmark suites in that group will be shown in the end. For example to compare
93all array benchmark suites:
95```
96$ ./bench-runner --targets array_cmp
97```
99## Reporting without measuring
101You can use the `--no-measure` option to report the already measured results in
102the benchmarks results file. A results file may collect an arbitrary number of
103results by running with `--append` multiple times. Each benchmark has its own
104results file, for example the `Prelude.Serial` benchmark has the results file at
105`charts/Prelude.Serial/results.csv`.
107You can also manually edit the file to remove a set of results if you like or
108to append results from previously saved results or from some other results
109file. After editing you can run `bench-runner` with the `--no-measure` option to
110see the reports corresponding to the results.
112## Additional benchmark configuration
114### Stream size
116You can specify the stream size (default is 100000) to be used for
117benchmarking:
119```
120$ cabal run bench:Prelude.Serial -- --stream-size 1000000
121```
123### External input file
125In the `FileSystem.Handle` benchmark you can specify the input file as an
126environment variable:
128```
129$ export Benchmark_FileSystem_Handle_InputFile=./gutenberg-500.txt
130$ cabal run FileSystem.Handle -- FileSystem.Handle/o-1-space/reduce/read/S.splitOnSeq
131```
133The automatic tests do not test unicode input, this option is useful to specify
134a unicode text file manually.
136## Running benchmarks on valid UTF8 input
138To run the unicode benchmarks on valid utf8 input, you can do the following,
140```
141$ Benchmark_FileSystem_Handle_InputFile=<valid-unicode-filepath> ./bench-runner --benchmarks Unicode.Stream --cabal-build-options "-f include-strict-utf8"
142```
144## Benchmarking notes
146We run each benchmark in an isolated process to minimize interference
147of benchmarks and to be able to control the RTS memory restrictions per
148benchmark.
150### Gotchas
152The benchmark driver forces a GC before and after the measurement.
153However, we have observed
154that sometimes the GC stats may not be accurate when the number of iterations
155in the measurement is small (e.g. 1 iteration). In such cases usually the
156number of GCs and GC times would also be 0.
158## Diagnosing Performance Issues
160### Reproducible comparison
162When comparing different compilers we need to make sure that we are
163using exactly the same versions of the libraries for apples to apples
164comparison. We have seen cases where a change in the "random" library
165caused allocations regressions in the new version of compiler because of
166the way in which the benchmark code was generated due to the change.
168When it is required to reproduce benchmark results precisely across
169different systems, it is recommended that you create and use a cabal
170freeze file so that the versions of all libraries are pinned.
172### Identifying issues
174There are two ways to find problematic code:
1761. Run performance benchmarks using `bench-runner`, select the benchmarks
177 that are taking more than expected time.
1782. When making a new change, compare with the baseline and select benchmarks
179 with the most regression reported by `bench-runner`.
181Number of allocations are the most stable measure that do not vary from
182run to run. `cpuTime` and `bytesCopied` may vary. When comparing two
183runs for regression the first thing to look at is the difference in
184allocations. Also note that allocations may vary from run to run for
185concurrent benchmarks.
187The next thing to look at is cpuTime. Please note that cpuTime may
188fluctuate quite a bit, you may want to run the relevant benchmarks
189without the --quick mode for confirming and make sure no other load is
190running on the system when measuring.
192Usually the increase is cpuTime is proportional to the increase in
193allocations but sometimes it may increase independently because more cpu
194instructions are being executed. TBD - we should count the instructions
195instead.
197### Inspection Testing
199Before you proceed make sure have to run the benchmarks with
200`inspection` flag on. It may catch any obvious issues or regressions.
202```
203$ cabal build --flag inspection --flag fusion-plugin --enable-benchmarks streamly-benchmarks
204```
206### Compiling with diagnostics
2081. Comment out all other benchmarks in the given benchmark suite, and
209 keep only the one you are examining.
2102. Edit the file and add the following line on top:
212```
213{-# OPTIONS_GHC
214 -ddump-simpl
215 -ddump-to-file
216 -dsuppress-all
217 -Wmissed-specialisations
218 -Wall-missed-specialisations
219 -fplugin-opt=Fusion.Plugin:verbose=2
220 -fplugin-opt=Fusion.Plugin:dump-core
221#-}
222```
2233. Build the benchmark suite with fusion-plugin enabled:
225```
226$ cabal build bench:Prelude.Serial --flag fusion-plugin
227```
229See the `.dump-simpl` file in the cabal build directory. You can find it
230like this:
232```
233$ find dist-newstyle/ -name "*.dump-simpl"
234```
236Make sure you are looking into the right build dir (`--build-dir` may change
237`dist-newstyle` to something else), and check in the appropriate GHC
238version dir.
240### Compiling standalone example
242Sometimes you may want to create a separate program from the benchmark code
243removing the benchmarking harness to simplify and isolate the code for better
244reasoning and simpler core.
246Add the following GHC options at the top of your file, say, example.hs:
248```
249{-# OPTIONS_GHC
250 -ddump-simpl
251 -ddump-to-file
252 -dsuppress-all
253 -Wmissed-specialisations
254 -Wall-missed-specialisations
255 -fplugin Fusion.Plugin
256 -fplugin-opt=Fusion.Plugin:verbose=2
257 -fplugin-opt=Fusion.Plugin:dump-core
258#-}
259```
261Do not include the optimization options in OPTIONS_GHC pragma, instead, specify
262them on the command line. This is to avoid optimization failing if you import
263another module which is not compiled with the same optimization options.
265```
266$ cabal build # build and write ghc environment file
267$ ghc -O2 -fspec-constr-recursive=16 -fmax-worker-args=16 example.hs
268```
270To pinpoint where the optimization is going wrong you can examine the
271plugin generated core files for each optimization pass. The files are
272numbered for each optimization pass. You can compare successive files
273using side-by-side diff and see what the compiler is doing between each
274pass.
276### Diagnosing the Problem
278#### Specialization Issues
280Look for missed specialization messages. When you are comparing against a
281baseline, check if something that was specialized before is no longer
282specialized.
284In the core you have to look for type class dictionaries e.g.
286```
287exc_r6DD = \ @s_a6ai -> try $fMonadCatchIO $fExceptionSomeException
288```
290Search for `$f` in the core.
292#### Fusion Issues
294Look for unfused function warnings emitted by fusion-plugin. You may
295want to take a look at the unfused constructors or functions that
296fusion-plugin is warning about. Beware that:
298* fusion-plugin emits warnings for unfused stuff in intermediate functions as
299 well, those should be ignored.
300* the constructors may remain genuinely unfused unless the loop is
301 closed. So you should look at the warnings in the file where the loop is
302 closed and everything is supposed to be fused.
304Also, look at the core for unfused constructors. At times you may need
305to look for the boxed primitive type constructors e.g. W8# or I#, these
306may not be eliminated, usually, due to strictness issues.
308Often it is useful to diff and compare the core without the problem and
309the core with the problem especially in cases when the problem is due to
310GHC version changes, or smaller changes in the code.
312Note, some operations are inherently fusion breaking, those cannot fuse,
313they are usually annotated so in their documentation.
315### Resolving the problem
317Review the problematic code, see [the optimization
318guide](/docs/Developer/optimization-guidelines.md) for common problems and how to
319solve those. If no obvious issues are found on review, then generate and
320examine the core.
322You may want to add the `Fuse` annotation on some of those constructors
323to make the code fuse. Please note that unnecessary `Fuse` annotations
324may cause unnecessary inlining. Also, make sure that the constructor you
325are adding fuse annotation is not shared by any other code where you may
326not want inlining/fusion.