git.iko.soy/streamly
on 326bcda help sign in
streamly/docs/Developer/Benchmarks.md
321 lines 11 kB history raw
1# Benchmarking
3## Benchmark Drivers
5Two benchmark drivers are supported:
7* `tasty-bench` (default)
8* `gauge` (enabled by `--use-gauge` build flag)
10## Build and run benchmarks directly
12The benchmark executables are `tasty-bench` executables unless you have
13passed `--use-gauge` cabal flag when building in which case it is a
14`gauge` executable.
16```
17$ cabal run bench:Prelude.Serial # run selected
18$ cabal run bench:Prelude.Serial -- --help # help on arguments
19$ cabal run bench:Prelude.Serial -- --stdev 100000 # specify arguments
20$ cabal run bench:Prelude.Serial --flag fusion-plugin # with fusion-plugin
22$ cabal build bench:Prelude.Serial # build selected
23$ cabal build --enable-benchmarks streamly-benchmarks # build all
24$ cabal build --enable-benchmarks all # build all, alternate method
26$ cabal build --flag "-opt" ... # disable optimization, faster build
27```
29## Building and Running Benchmarks with bench.sh
31`<streamly repo>/bin/bench.sh` script is the top level driver for
32running benchmarks. It runs the requested benchmarks and then creates a
33report from the results using the `bench-show` package.
35IMPORTANT NOTE: The first time you run this script it may take a long
36time because it has to build the `bench-report` executable which has a
37lot of dependencies. If you are using nix then use `--use-nix` flag
38for the first time so that the `bench-report` executable is built using
39nix. That can save a lot of time compiling it. However, once it is built
40it will be cached in the `bin` directory of the repo and used from
41there every time. You can also build it manually from the cabal file in
42`benchmark/bench-report` and install it in the `bin` directory.
44## bench.sh: Quick start
46Useful commands:
48```
49$ bin/bench.sh --help
50$ bin/bench.sh --quick # run all the benchmark suites
51$ bin/bench.sh --benchmarks help # Show available benchmark suites
52$ bin/bench.sh --benchmarks serial_grp # Run all serial benchmark suites
53$ bin/bench.sh --benchmarks "Prelude.Serial Data.Parser" # run selected suites
54$ bin/bench.sh --no-measure # don't run benchmarks just show previous results
56# Run all O(1) space complexity benchmarks in `Prelude.Serial` suite
57$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space
59# Run a specific benchmark in `Prelude.Serial` suite
60$ bin/bench.sh --benchmarks Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
61```
63Note: `bench.sh` enables fusion-plugin by default.
65## Comparing results with baseline
67```
68# Checkout baseline commit
69$ bin/bench.sh --quick
71# Checkout commit with new changes
72$ bin/bench.sh --quick --append
74# To add another result to comparisons just repeat the above command on
75# desired commit
76```
78## Comparing benchmark suites
80First see the available benchmark suites:
82```
83$ bin/bench.sh --benchmarks help
84```
86You will see some benchmark suites end with `_cmp`, these are comparison
87groups. If you run a comparison group benchmark, comparison of all the
88benchmark suites in that group will be shown in the end. For example to compare
89all array benchmark suites:
91```
92$ bin/bench.sh --benchmarks array_cmp
93```
95## Reporting without measuring
97You can use the `--no-measure` option to report the already measured results in
98the benchmarks results file. A results file may collect an arbitrary number of
99results by running with `--append` multiple times. Each benchmark has its own
100results file, for example the `Prelude.Serial` benchmark has the results file at
101`charts/Prelude.Serial/results.csv`.
103You can also manually edit the file to remove a set of results if you like or
104to append results from previously saved results or from some other results
105file. After editing you can run `bench.sh` with the `--no-measure` option to
106see the reports corresponding to the results.
108## Additional benchmark configuration
110### Stream size
112You can specify the stream size (default is 100000) to be used for
113benchmarking:
115```
116$ cabal run bench:Prelude.Serial -- --stream-size 1000000
117```
119### External input file
121In the `FileSystem.Handle` benchmark you can specify the input file as an
122environment variable:
124```
125$ export Benchmark_FileSystem_Handle_InputFile=./gutenberg-500.txt
126$ cabal run FileSystem.Handle -- FileSystem.Handle/o-1-space/reduce/read/S.splitOnSeq
127```
129The automatic tests do not test unicode input, this option is useful to specify
130a unicode text file manually.
132## Running benchmarks on valid UTF8 input
134To run the unicode benchmarks on valid utf8 input, you can do the following,
136```
137$ Benchmark_FileSystem_Handle_InputFile=<valid-unicode-filepath> bin/bench.sh --benchmarks Unicode.Stream --cabal-build-options "-f include-strict-utf8"
138```
140## Benchmarking notes
142We run each benchmark in an isolated process to minimize interference
143of benchmarks and to be able to control the RTS memory restrictions per
144benchmark.
146### Gotchas
148Gauge forces a GC before and after the measurement. However, we have observed
149that sometimes the GC stats may not be accurate when the number of iterations
150in the measurement is small (e.g. 1 iteration). In such cases usually the
151number of GCs and GC times would also be 0.
153## Diagnosing Performance Issues
155### Reproducible comparison
157When comparing different compilers we need to make sure that we are
158using exactly the same versions of the libraries for apples to apples
159comparison. We have seen cases where a change in the "random" library
160caused allocations regressions in the new version of compiler because of
161the way in which the benchmark code was generated due to the change.
163When it is required to reproduce benchmark results precisely across
164different systems, it is recommended that you create and use a cabal
165freeze file so that the versions of all libraries are pinned.
167### Identifying issues
169There are two ways to find problematic code:
1711. Run performance benchmarks using `bench.sh`, select the benchmarks
172 that are taking more than expected time.
1732. When making a new change, compare with the baseline and select benchmarks
174 with the most regression reported by `bench.sh`.
176Number of allocations are the most stable measure that do not vary from
177run to run. `cpuTime` and `bytesCopied` may vary. When comparing two
178runs for regression the first thing to look at is the difference in
179allocations. Also note that allocations may vary from run to run for
180concurrent benchmarks.
182The next thing to look at is cpuTime. Please note that cpuTime may
183fluctuate quite a bit, you may want to run the relevant benchmarks
184without the --quick mode for confirming and make sure no other load is
185running on the system when measuring.
187Usually the increase is cpuTime is proportional to the increase in
188allocations but sometimes it may increase independently because more cpu
189instructions are being executed. TBD - we should count the instructions
190instead.
192### Inspection Testing
194Before you proceed make sure have to run the benchmarks with
195`inspection` flag on. It may catch any obvious issues or regressions.
197```
198$ cabal build --flag inspection --flag fusion-plugin --enable-benchmarks streamly-benchmarks
199```
201### Compiling with diagnostics
2031. Comment out all other benchmarks in the given benchmark suite, and
204 keep only the one you are examining.
2052. Edit the file and add the following line on top:
207```
208{-# OPTIONS_GHC
209 -ddump-simpl
210 -ddump-to-file
211 -dsuppress-all
212 -Wmissed-specialisations
213 -Wall-missed-specialisations
214 -fplugin-opt=Fusion.Plugin:verbose=2
215 -fplugin-opt=Fusion.Plugin:dump-core
216#-}
217```
2183. Build the benchmark suite with fusion-plugin enabled:
220```
221$ cabal build bench:Prelude.Serial --flag fusion-plugin
222```
224See the `.dump-simpl` file in the cabal build directory. You can find it
225like this:
227```
228$ find dist-newstyle/ -name "*.dump-simpl"
229```
231Make sure you are looking into the right build dir (`--build-dir` may change
232`dist-newstyle` to something else), and check in the appropriate GHC
233version dir.
235### Compiling standalone example
237Sometimes you may want to create a separate program from the benchmark code
238removing the benchmarking harness to simplify and isolate the code for better
239reasoning and simpler core.
241Add the following GHC options at the top of your file, say, example.hs:
243```
244{-# OPTIONS_GHC
245 -ddump-simpl
246 -ddump-to-file
247 -dsuppress-all
248 -Wmissed-specialisations
249 -Wall-missed-specialisations
250 -fplugin Fusion.Plugin
251 -fplugin-opt=Fusion.Plugin:verbose=2
252 -fplugin-opt=Fusion.Plugin:dump-core
253#-}
254```
256Do not include the optimization options in OPTIONS_GHC pragma, instead, specify
257them on the command line. This is to avoid optimization failing if you import
258another module which is not compiled with the same optimization options.
260```
261$ cabal build # build and write ghc environment file
262$ ghc -O2 -fspec-constr-recursive=16 -fmax-worker-args=16 example.hs
263```
265To pinpoint where the optimization is going wrong you can examine the
266plugin generated core files for each optimization pass. The files are
267numbered for each optimization pass. You can compare successive files
268using side-by-side diff and see what the compiler is doing between each
269pass.
271### Diagnosing the Problem
273#### Specialization Issues
275Look for missed specialization messages. When you are comparing against a
276baseline, check if something that was specialized before is no longer
277specialized.
279In the core you have to look for type class dictionaries e.g.
281```
282exc_r6DD = \ @s_a6ai -> try $fMonadCatchIO $fExceptionSomeException
283```
285Search for `$f` in the core.
287#### Fusion Issues
289Look for unfused function warnings emitted by fusion-plugin. You may
290want to take a look at the unfused constructors or functions that
291fusion-plugin is warning about. Beware that:
293* fusion-plugin emits warnings for unfused stuff in intermediate functions as
294 well, those should be ignored.
295* the constructors may remain genuinely unfused unless the loop is
296 closed. So you should look at the warnings in the file where the loop is
297 closed and everything is supposed to be fused.
299Also, look at the core for unfused constructors. At times you may need
300to look for the boxed primitive type constructors e.g. W8# or I#, these
301may not be eliminated, usually, due to strictness issues.
303Often it is useful to diff and compare the core without the problem and
304the core with the problem especially in cases when the problem is due to
305GHC version changes, or smaller changes in the code.
307Note, some operations are inherently fusion breaking, those cannot fuse,
308they are usually annotated so in their documentation.
310### Resolving the problem
312Review the problematic code, see [the optimization
313guide](/docs/Developer/optimization-guidelines.md) for common problems and how to
314solve those. If no obvious issues are found on review, then generate and
315examine the core.
317You may want to add the `Fuse` annotation on some of those constructors
318to make the code fuse. Please note that unnecessary `Fuse` annotations
319may cause unnecessary inlining. Also, make sure that the constructor you
320are adding fuse annotation is not shared by any other code where you may
321not want inlining/fusion.