@@ -141,3 +141,173 @@ Gauge forces a GC before and after the measurement. However, we have observed
141141that sometimes the GC stats may not be accurate when the number of iterations
142142in the measurement is small (e.g. 1 iteration). In such cases usually the
143143number of GCs and GC times would also be 0.
144+
145+## Diagnosing Performance Issues
146+
147+### Reproducible comparison
148+
149+When comparing different compilers we need to make sure that we are
150+using exactly the same versions of the libraries for apples to apples
151+comparison. We have seen cases where a change in the "random" library
152+caused allocations regressions in the new version of compiler because of
153+the way in which the benchmark code was generated due to the change.
154+
155+When it is required to reproduce benchmark results precisely across
156+different systems, it is recommended that you create and use a cabal
157+freeze file so that the versions of all libraries are pinned.
158+
159+### Identifying issues
160+
161+There are two ways to find problematic code:
162+
163+1. Run performance benchmarks using `bench.sh`, select the benchmarks
164+ that are taking more than expected time.
165+2. When making a new change, compare with the baseline and select benchmarks
166+ with the most regression reported by `bench.sh`.
167+
168+Number of allocations are the most stable measure that do not vary from
169+run to run. `cpuTime` and `bytesCopied` may vary. When comparing two
170+runs for regression the first thing to look at is the difference in
171+allocations. Also note that allocations may vary from run to run for
172+concurrent benchmarks.
173+
174+The next thing to look at is cpuTime. Please note that cpuTime may
175+fluctuate quite a bit, you may want to run the relevant benchmarks
176+without the --quick mode for confirming and make sure no other load is
177+running on the system when measuring.
178+
179+Usually the increase is cpuTime is proportional to the increase in
180+allocations but sometimes it may increase independently because more cpu
181+instructions are being executed. TBD - we should count the instructions
182+instead.
183+
184+### Inspection Testing
185+
186+Before you proceed make sure have to run the benchmarks with
187+`inspection` flag on. It may catch any obvious issues or regressions.
188+
189+```
190+$ cabal build --flag inspection --flag fusion-plugin --enable-benchmarks streamly-benchmarks
191+```
192+
193+### Compiling with diagnostics
194+
195+1. Comment out all other benchmarks in the given benchmark suite, and
196+ keep only the one you are examining.
197+2. Edit the file and add the following line on top:
198+
199+```
200+{-# OPTIONS_GHC
201+ -ddump-simpl
202+ -ddump-to-file
203+ -dsuppress-all
204+ -Wmissed-specialisations
205+ -Wall-missed-specialisations
206+ -fplugin-opt=Fusion.Plugin:verbose=2
207+ -fplugin-opt=Fusion.Plugin:dump-core
208+#-}
209+```
210+3. Build the benchmark suite with fusion-plugin enabled:
211+
212+```
213+$ cabal build bench:Prelude.Serial --flag fusion-plugin
214+```
215+
216+See the `.dump-simpl` file in the cabal build directory. You can find it
217+like this:
218+
219+```
220+$ find dist-newstyle/ -name "*.dump-simpl"
221+```
222+
223+Make sure you are looking into the right build dir (`--build-dir` may change
224+`dist-newstyle` to something else), and check in the appropriate GHC
225+version dir.
226+
227+### Compiling standalone example
228+
229+Sometimes you may want to create a separate program from the benchmark code
230+removing the benchmarking harness to simplify and isolate the code for better
231+reasoning and simpler core.
232+
233+Add the following GHC options at the top of your file, say, example.hs:
234+
235+```
236+{-# OPTIONS_GHC
237+ -ddump-simpl
238+ -ddump-to-file
239+ -dsuppress-all
240+ -Wmissed-specialisations
241+ -Wall-missed-specialisations
242+ -fplugin Fusion.Plugin
243+ -fplugin-opt=Fusion.Plugin:verbose=2
244+ -fplugin-opt=Fusion.Plugin:dump-core
245+#-}
246+```
247+
248+Do not include the optimization options in OPTIONS_GHC pragma, instead, specify
249+them on the command line. This is to avoid optimization failing if you import
250+another module which is not compiled with the same optimization options.
251+
252+```
253+$ cabal build # build and write ghc environment file
254+$ ghc -O2 -fspec-constr-recursive=16 -fmax-worker-args=16 example.hs
255+```
256+
257+To pinpoint where the optimization is going wrong you can examine the
258+plugin generated core files for each optimization pass. The files are
259+numbered for each optimization pass. You can compare successive files
260+using side-by-side diff and see what the compiler is doing between each
261+pass.
262+
263+### Diagnosing the Problem
264+
265+#### Specialization Issues
266+
267+Look for missed specialization messages. When you are comparing against a
268+baseline, check if something that was specialized before is no longer
269+specialized.
270+
271+In the core you have to look for type class dictionaries e.g.
272+
273+```
274+exc_r6DD = \ @s_a6ai -> try $fMonadCatchIO $fExceptionSomeException
275+```
276+
277+Search for `$f` in the core.
278+
279+#### Fusion Issues
280+
281+Look for unfused function warnings emitted by fusion-plugin. You may
282+want to take a look at the unfused constructors or functions that
283+fusion-plugin is warning about. Beware that:
284+
285+* fusion-plugin emits warnings for unfused stuff in intermediate functions as
286+ well, those should be ignored.
287+* the constructors may remain genuinely unfused unless the loop is
288+ closed. So you should look at the warnings in the file where the loop is
289+ closed and everything is supposed to be fused.
290+
291+Also, look at the core for unfused constructors. At times you may need
292+to look for the boxed primitive type constructors e.g. W8# or I#, these
293+may not be eliminated, usually, due to strictness issues.
294+
295+Often it is useful to diff and compare the core without the problem and
296+the core with the problem especially in cases when the problem is due to
297+GHC version changes, or smaller changes in the code.
298+
299+Note, some operations are inherently fusion breaking, those cannot fuse,
300+they are usually annotated so in their documentation.
301+
302+### Resolving the problem
303+
304+Review the problematic code, see [the optimization
305+guide](dev/optimization-guidelines.md) for common problems and how to
306+solve those. If no obvious issues are found on review, then generate and
307+examine the core.
308+
309+You may want to add the `Fuse` annotation on some of those constructors
310+to make the code fuse. Please note that unnecessary `Fuse` annotations
311+may cause unnecessary inlining. Also, make sure that the constructor you
312+are adding fuse annotation is not shared by any other code where you may
313+not want inlining/fusion.