@@ -141,3 +141,173 @@ Gauge forces a GC before and after the measurement. However, we have observed
141that sometimes the GC stats may not be accurate when the number of iterations
141that sometimes the GC stats may not be accurate when the number of iterations
142in the measurement is small (e.g. 1 iteration). In such cases usually the
142in the measurement is small (e.g. 1 iteration). In such cases usually the
143number of GCs and GC times would also be 0.
143number of GCs and GC times would also be 0.
145+## Diagnosing Performance Issues
147+### Reproducible comparison
149+When comparing different compilers we need to make sure that we are
150+using exactly the same versions of the libraries for apples to apples
151+comparison. We have seen cases where a change in the "random" library
152+caused allocations regressions in the new version of compiler because of
153+the way in which the benchmark code was generated due to the change.
155+When it is required to reproduce benchmark results precisely across
156+different systems, it is recommended that you create and use a cabal
157+freeze file so that the versions of all libraries are pinned.
159+### Identifying issues
161+There are two ways to find problematic code:
163+1. Run performance benchmarks using `bench.sh`, select the benchmarks
164+ that are taking more than expected time.
165+2. When making a new change, compare with the baseline and select benchmarks
166+ with the most regression reported by `bench.sh`.
168+Number of allocations are the most stable measure that do not vary from
169+run to run. `cpuTime` and `bytesCopied` may vary. When comparing two
170+runs for regression the first thing to look at is the difference in
171+allocations. Also note that allocations may vary from run to run for
172+concurrent benchmarks.
174+The next thing to look at is cpuTime. Please note that cpuTime may
175+fluctuate quite a bit, you may want to run the relevant benchmarks
176+without the --quick mode for confirming and make sure no other load is
177+running on the system when measuring.
179+Usually the increase is cpuTime is proportional to the increase in
180+allocations but sometimes it may increase independently because more cpu
181+instructions are being executed. TBD - we should count the instructions
184+### Inspection Testing
186+Before you proceed make sure have to run the benchmarks with
187+`inspection` flag on. It may catch any obvious issues or regressions.
190+$ cabal build --flag inspection --flag fusion-plugin --enable-benchmarks streamly-benchmarks
193+### Compiling with diagnostics
195+1. Comment out all other benchmarks in the given benchmark suite, and
196+ keep only the one you are examining.
197+2. Edit the file and add the following line on top:
204+ -Wmissed-specialisations
205+ -Wall-missed-specialisations
206+ -fplugin-opt=Fusion.Plugin:verbose=2
207+ -fplugin-opt=Fusion.Plugin:dump-core
210+3. Build the benchmark suite with fusion-plugin enabled:
213+$ cabal build bench:Prelude.Serial --flag fusion-plugin
216+See the `.dump-simpl` file in the cabal build directory. You can find it
220+$ find dist-newstyle/ -name "*.dump-simpl"
223+Make sure you are looking into the right build dir (`--build-dir` may change
224+`dist-newstyle` to something else), and check in the appropriate GHC
227+### Compiling standalone example
229+Sometimes you may want to create a separate program from the benchmark code
230+removing the benchmarking harness to simplify and isolate the code for better
231+reasoning and simpler core.
233+Add the following GHC options at the top of your file, say, example.hs:
240+ -Wmissed-specialisations
241+ -Wall-missed-specialisations
242+ -fplugin Fusion.Plugin
243+ -fplugin-opt=Fusion.Plugin:verbose=2
244+ -fplugin-opt=Fusion.Plugin:dump-core
248+Do not include the optimization options in OPTIONS_GHC pragma, instead, specify
249+them on the command line. This is to avoid optimization failing if you import
250+another module which is not compiled with the same optimization options.
253+$ cabal build # build and write ghc environment file
254+$ ghc -O2 -fspec-constr-recursive=16 -fmax-worker-args=16 example.hs
257+To pinpoint where the optimization is going wrong you can examine the
258+plugin generated core files for each optimization pass. The files are
259+numbered for each optimization pass. You can compare successive files
260+using side-by-side diff and see what the compiler is doing between each
263+### Diagnosing the Problem
265+#### Specialization Issues
267+Look for missed specialization messages. When you are comparing against a
268+baseline, check if something that was specialized before is no longer
271+In the core you have to look for type class dictionaries e.g.
274+exc_r6DD = \ @s_a6ai -> try $fMonadCatchIO $fExceptionSomeException
277+Search for `$f` in the core.
281+Look for unfused function warnings emitted by fusion-plugin. You may
282+want to take a look at the unfused constructors or functions that
283+fusion-plugin is warning about. Beware that:
285+* fusion-plugin emits warnings for unfused stuff in intermediate functions as
286+ well, those should be ignored.
287+* the constructors may remain genuinely unfused unless the loop is
288+ closed. So you should look at the warnings in the file where the loop is
289+ closed and everything is supposed to be fused.
291+Also, look at the core for unfused constructors. At times you may need
292+to look for the boxed primitive type constructors e.g. W8# or I#, these
293+may not be eliminated, usually, due to strictness issues.
295+Often it is useful to diff and compare the core without the problem and
296+the core with the problem especially in cases when the problem is due to
297+GHC version changes, or smaller changes in the code.
299+Note, some operations are inherently fusion breaking, those cannot fuse,
300+they are usually annotated so in their documentation.
302+### Resolving the problem
304+Review the problematic code, see [the optimization
305+guide](dev/optimization-guidelines.md) for common problems and how to
306+solve those. If no obvious issues are found on review, then generate and
309+You may want to add the `Fuse` annotation on some of those constructors
310+to make the code fuse. Please note that unnecessary `Fuse` annotations
311+may cause unnecessary inlining. Also, make sure that the constructor you
312+are adding fuse annotation is not shared by any other code where you may
313+not want inlining/fusion.