git.iko.soy/streamly
on 63aae12 help sign in
streamly/docs/Developer/Benchmarks.md
342 lines 12 kB history raw
1# Benchmarking
3## Build and run benchmarks directly
5```
6$ cabal run bench:Prelude.Serial # run selected
7$ cabal run bench:Prelude.Serial -- --help # help on arguments
8$ cabal run bench:Prelude.Serial -- --stdev 100000 # specify arguments
9$ cabal run bench:Prelude.Serial --flag fusion-plugin # with fusion-plugin
11$ cabal build bench:Prelude.Serial # build selected
12$ cabal build --enable-benchmarks streamly-benchmarks # build all
13$ cabal build --enable-benchmarks all # build all, alternate method
15$ cabal build --flag "-opt" ... # disable optimization, faster build
16```
18## Building and Running Benchmarks with bench-runner
20The executable `bench-runner` is the top level driver for
21running benchmarks. It runs the requested benchmarks and then creates a
22report from the results using the `bench-show` package.
24IMPORTANT NOTE: The first time you run this executable it may take a long
25time because it has to build the `bench-report` executable which has a
26lot of dependencies.
28You can install it once in the root of the repository and use it multiple times.
30You can use `cabal.project.report` to install bench-runner like so:
31```
32$ cabal install bench-runner --project-file=cabal.project.report --installdir=./ --overwrite-policy=always
33$ ./bench-runner <bench-runner-args>
34```
36If you're using nix, you can install bench-runner like so:
37```
38$ cd benchmark/bench-runner
39$ nix-shell --run 'cabal install bench-runner --installdir=../../ --overwrite-policy=always'
40$ cd ../../
41$ ./bench-runner <bench-runner-args>
42```
44You can run the `bench-runner` without installing, like so:
45```
46$ cabal run bench-runner --project-file=cabal.project.report -- <bench-runner-args>
47```
49## bench-runner: Quick start
51Assuming `bench-runner` is the executable. You can replace `./bench-runner` with
52`cabal run bench-runner --project-file=cabal.project.report --`
54Note, you need to pass two mandatory arguments to bench-runner,
55`package-version` and `package-name`. For the streamly-benchmarks package, pass
56these as below:
58```
59bin/bench-runner --package-version 0.0.0 --package-name streamly-benchmarks
60```
62Useful commands:
64```
65$ ./bench-runner --help
66$ ./bench-runner --quick # run all the benchmark suites
67$ ./bench-runner --targets help # Show available benchmark suites
68$ ./bench-runner --targets serial_grp # Run all serial benchmark suites
69$ ./bench-runner --targets "Prelude.Serial Data.Parser" # run selected suites
70$ ./bench-runner --no-measure # don't run benchmarks just show previous results
72# Run all O(1) space complexity benchmarks in `Prelude.Serial` suite
73$ ./bench-runner --targets Prelude.Serial --prefix Prelude.Serial/o-1-space
75# Run a specific benchmark in `Prelude.Serial` suite
76$ ./bench-runner --targets Prelude.Serial --prefix Prelude.Serial/o-1-space.generation.unfoldr
77```
79Note: `bench-runner` enables fusion-plugin by default.
81## Comparing results with baseline
83```
84# Checkout baseline commit
85$ ./bench-runner --quick
87# Checkout commit with new changes
88$ ./bench-runner --quick --append
90# To add another result to comparisons just repeat the above command on
91# desired commit
93# To display the current results without running the benchmarks.
94# See "Reporting without measuring" for more info.
95$ ./bench-runner --no-measure
96```
98## Comparing benchmark suites
100First see the available benchmark suites:
102```
103$ ./bench-runner --targets help
104```
106You will see some benchmark suites end with `_cmp`, these are comparison
107groups. If you run a comparison group benchmark, comparison of all the
108benchmark suites in that group will be shown in the end. For example to compare
109all array benchmark suites:
111```
112$ ./bench-runner --targets array_cmp
113```
115## Reporting without measuring
117You can use the `--no-measure` option to report the already measured results in
118the benchmarks results file. A results file may collect an arbitrary number of
119results by running with `--append` multiple times. Each benchmark has its own
120results file, for example the `Prelude.Serial` benchmark has the results file at
121`charts/Prelude.Serial/results.csv`.
123You can also manually edit the file to remove a set of results if you like or
124to append results from previously saved results or from some other results
125file. After editing you can run `bench-runner` with the `--no-measure` option to
126see the reports corresponding to the results.
128## Additional benchmark configuration
130### Stream size
132You can specify the stream size (default is 100000) to be used for
133benchmarking:
135```
136$ cabal run bench:Prelude.Serial -- --stream-size 1000000
137```
139### External input file
141In the `FileSystem.Handle` benchmark you can specify the input file as an
142environment variable:
144```
145$ export Benchmark_FileSystem_Handle_InputFile=./gutenberg-500.txt
146$ cabal run FileSystem.Handle -- FileSystem.Handle/o-1-space/reduce/read/S.splitOnSeq
147```
149The automatic tests do not test unicode input, this option is useful to specify
150a unicode text file manually.
152## Running benchmarks on valid UTF8 input
154To run the unicode benchmarks on valid utf8 input, you can do the following,
156```
157$ Benchmark_FileSystem_Handle_InputFile=<valid-unicode-filepath> ./bench-runner --benchmarks Unicode.Stream --cabal-build-options "-f include-strict-utf8"
158```
160## Benchmarking notes
162We run each benchmark in an isolated process to minimize interference
163of benchmarks and to be able to control the RTS memory restrictions per
164benchmark.
166### Gotchas
168The benchmark driver forces a GC before and after the measurement.
169However, we have observed
170that sometimes the GC stats may not be accurate when the number of iterations
171in the measurement is small (e.g. 1 iteration). In such cases usually the
172number of GCs and GC times would also be 0.
174## Diagnosing Performance Issues
176### Reproducible comparison
178When comparing different compilers we need to make sure that we are
179using exactly the same versions of the libraries for apples to apples
180comparison. We have seen cases where a change in the "random" library
181caused allocations regressions in the new version of compiler because of
182the way in which the benchmark code was generated due to the change.
184When it is required to reproduce benchmark results precisely across
185different systems, it is recommended that you create and use a cabal
186freeze file so that the versions of all libraries are pinned.
188### Identifying issues
190There are two ways to find problematic code:
1921. Run performance benchmarks using `bench-runner`, select the benchmarks
193 that are taking more than expected time.
1942. When making a new change, compare with the baseline and select benchmarks
195 with the most regression reported by `bench-runner`.
197Number of allocations are the most stable measure that do not vary from
198run to run. `cpuTime` and `bytesCopied` may vary. When comparing two
199runs for regression the first thing to look at is the difference in
200allocations. Also note that allocations may vary from run to run for
201concurrent benchmarks.
203The next thing to look at is cpuTime. Please note that cpuTime may
204fluctuate quite a bit, you may want to run the relevant benchmarks
205without the --quick mode for confirming and make sure no other load is
206running on the system when measuring.
208Usually the increase is cpuTime is proportional to the increase in
209allocations but sometimes it may increase independently because more cpu
210instructions are being executed. TBD - we should count the instructions
211instead.
213### Inspection Testing
215Before you proceed make sure have to run the benchmarks with
216`inspection` flag on. It may catch any obvious issues or regressions.
218```
219$ cabal build --flag inspection --flag fusion-plugin --enable-benchmarks streamly-benchmarks
220```
222### Compiling with diagnostics
2241. Comment out all other benchmarks in the given benchmark suite, and
225 keep only the one you are examining.
2262. Edit the file and add the following line on top:
228```
229{-# OPTIONS_GHC
230 -ddump-simpl
231 -ddump-to-file
232 -dsuppress-all
233 -Wmissed-specialisations
234 -Wall-missed-specialisations
235 -fplugin-opt=Fusion.Plugin:verbose=2
236 -fplugin-opt=Fusion.Plugin:dump-core
237#-}
238```
2393. Build the benchmark suite with fusion-plugin enabled:
241```
242$ cabal build bench:Prelude.Serial --flag fusion-plugin
243```
245See the `.dump-simpl` file in the cabal build directory. You can find it
246like this:
248```
249$ find dist-newstyle/ -name "*.dump-simpl"
250```
252Make sure you are looking into the right build dir (`--build-dir` may change
253`dist-newstyle` to something else), and check in the appropriate GHC
254version dir.
256### Compiling standalone example
258Sometimes you may want to create a separate program from the benchmark code
259removing the benchmarking harness to simplify and isolate the code for better
260reasoning and simpler core.
262Add the following GHC options at the top of your file, say, example.hs:
264```
265{-# OPTIONS_GHC
266 -ddump-simpl
267 -ddump-to-file
268 -dsuppress-all
269 -Wmissed-specialisations
270 -Wall-missed-specialisations
271 -fplugin Fusion.Plugin
272 -fplugin-opt=Fusion.Plugin:verbose=2
273 -fplugin-opt=Fusion.Plugin:dump-core
274#-}
275```
277Do not include the optimization options in OPTIONS_GHC pragma, instead, specify
278them on the command line. This is to avoid optimization failing if you import
279another module which is not compiled with the same optimization options.
281```
282$ cabal build # build and write ghc environment file
283$ ghc -O2 -fspec-constr-recursive=16 -fmax-worker-args=16 example.hs
284```
286To pinpoint where the optimization is going wrong you can examine the
287plugin generated core files for each optimization pass. The files are
288numbered for each optimization pass. You can compare successive files
289using side-by-side diff and see what the compiler is doing between each
290pass.
292### Diagnosing the Problem
294#### Specialization Issues
296Look for missed specialization messages. When you are comparing against a
297baseline, check if something that was specialized before is no longer
298specialized.
300In the core you have to look for type class dictionaries e.g.
302```
303exc_r6DD = \ @s_a6ai -> try $fMonadCatchIO $fExceptionSomeException
304```
306Search for `$f` in the core.
308#### Fusion Issues
310Look for unfused function warnings emitted by fusion-plugin. You may
311want to take a look at the unfused constructors or functions that
312fusion-plugin is warning about. Beware that:
314* fusion-plugin emits warnings for unfused stuff in intermediate functions as
315 well, those should be ignored.
316* the constructors may remain genuinely unfused unless the loop is
317 closed. So you should look at the warnings in the file where the loop is
318 closed and everything is supposed to be fused.
320Also, look at the core for unfused constructors. At times you may need
321to look for the boxed primitive type constructors e.g. W8# or I#, these
322may not be eliminated, usually, due to strictness issues.
324Often it is useful to diff and compare the core without the problem and
325the core with the problem especially in cases when the problem is due to
326GHC version changes, or smaller changes in the code.
328Note, some operations are inherently fusion breaking, those cannot fuse,
329they are usually annotated so in their documentation.
331### Resolving the problem
333Review the problematic code, see [the optimization
334guide](/docs/Developer/optimization-guidelines.md) for common problems and how to
335solve those. If no obvious issues are found on review, then generate and
336examine the core.
338You may want to add the `Fuse` annotation on some of those constructors
339to make the code fuse. Please note that unnecessary `Fuse` annotations
340may cause unnecessary inlining. Also, make sure that the constructor you
341are adding fuse annotation is not shared by any other code where you may
342not want inlining/fusion.