@@ -5,17 +5,76 @@ stream generation or an unfold operation followed by stream transformation
55functions (e.g. map) and then a stream elimination operation or a fold
66operation.
77
8−The default or most common stream representation used in streamly is `StreamD`
9−which is a direct style stream representation. All the direct style
10−operations in a loop "fuse" together to form a tight machine loop eliminating
11−any intermediate constructors, therefore, reducing allocations and cpu cost.
12−This elimination of intermediate constructors in a loop is known as stream
13−fusion.
8+The default or most common stream representation used in streamly is
9+`StreamD` which is a direct style (compare with CPS representation
10+`StreamK`) stream representation. All the direct style operations in
11+a loop "fuse" together to form a tight machine loop eliminating any
12+intermediate constructors, therefore, reducing allocations and cpu cost.
13+This elimination of intermediate constructors in a stream loop is known
14+as stream fusion.
1415
15−## General
16+## Writing high performance code using streamly
1617
17−To enable stream fusion in direct stream all the operations that are part of a
18−loop must be inlined:
18+This section outlines guidelines for writing code using the available
19+combinators. The guidelines for writing new combinators are provided in
20+the following section.
21+
22+Note that there is no absolute benchmark for performance, most of
23+the time even without following the guidelines you may get excellent
24+performance for the task at hand. Two important points to keep in mind
25+before you optimize:
26+
27+* See if you actually need more performance
28+* See if the code you are optimizing is in fast path
29+
30+### Stream Fusion
31+
32+When your performance requirements are stringent you may want to care
33+about not breaking stream fusion unnecessarily. Certain operations do
34+not fuse and act as a barrier to stream fusion. If these operations are
35+part of a loop, the loop won't fuse completely. In some cases these may
36+be necessary but in others it may be possible to replace these with
37+better fusing operations. These operations include:
38+
39+* Avoid unnecessary use of stream append operations in code that should
40+ fuse. Append operations in general force CPS style breaking stream
41+ fusion. Some direct style append operations can fuse but they won't
42+ scale for more than a few appends.
43+* Avoid unnecessary use of the stream monad if fusion is important. You
44+ may use functor or applicative though.
45+* Concurrent stream operations cannot fuse
46+* Operations involving exception primitives like catch/throw/mask
47+ on elements of the stream cannot fuse.
48+* Stream loops involving FFI calls on the elements of the stream
49+
50+The haddock documentation includes a note when an operation cannot fuse.
51+
52+### Unfolds
53+
54+* Use unfolds especially when higher order operations are involved. For
55+ example, `concatUnfold` can fuse completely whereas `concatMap` would
56+ not fuse.
57+* Use `outerProduct` in `Unfold` module instead of using the monad instance of
58+ streams to fuse nested loops where performance matters. See the unfold
59+ benchmarks for an example. `outerProduct` can give you C like nested loop
60+ performance.
61+
62+### Inlining
63+
64+To make sure fusion can occur INLINE any operations that are part of a
65+stream loop but factored out as separate functions.
66+
67+### Strictness
68+
69+Keep the fold and scan accumulators strict. You could also consider
70+using mutable state in a fold accumulator if the state is large. A
71+large immutable structure as an accumulator may cause optimization
72+issues. See the `WordCount` example for an example of this case.
73+
74+## Writing streamly combinators
75+
76+To enable stream fusion in a direct style stream all the operations that
77+are part of a loop must be inlined:
1978
2079* Use an explicit INLINE pragma on any combinator that could be part of a loop
2180 to ensure they will be inlined.
@@ -24,7 +83,7 @@ loop must be inlined:
2483 is consuming. Use appropriate inline phases to ensure proper inline ordering
2584 when required.
2685
27−## Streams and Unfolds
86+### Streams and Unfolds
2887
2988Stream and Unfold State. Direct style stream and unfold combinators use a
3089state data type to represent the internal state of the stream/unfold generator.
@@ -32,21 +91,52 @@ state data type to represent the internal state of the stream/unfold generator.
3291* In some cases we may have to add a `FUSE` annotation on the state type to
3392 ensure that all the internal join points created by GHC are inlined with the
3493 help of the `fusion-plugin`.
35−* In rare cases we may need to use a strictness annotation on the state.
36− `parseMany` is one such example.
94+* In rare cases we may need to use a strictness annotation on the state
95+ to allow fusion. `parseMany` and `splitOnSeq` are such examples.
3796
3897Step function of a stream or unfold:
3998
40−* Never make the step function recursive. Always use the `Skip` constructor to
41− remove recursion. Recursive step function can introduce optimization barriers
42− that are harder to remove by the GHC simplifier.
43−* Try to make sure that there is a single yield point in the whole state
44− machine. A single yield point is usually desirable, however not always
45− necessary.
99+* Do not introduce unnecessary states. If there is only one entry point of a
100+ state then perhaps you want to collapse the state into the state from which
101+ it is called. More states means more jumps and may affect code locality, and
102+ efficiency of low level code (e.g. register allocation) because of
103+ independent placement of the code blocks in different states.
104+* Keep minimum possible data in state. More variables in state may lead
105+ to poor performance because the state may not fit into registers and the
106+ spill may cause allocations on each iteration of the loop. Mutable state
107+ may help in such cases.
46108* The step function must be annotated such that it gets inlined after the main
47109 combinator (`INLINE_LATE`).
48110
49−## Fold and Parser drivers
111+Multiple yield points or single?:
112+
113+* A single yield point is usually desirable, however, not always necessary.
114+ In some cases multiple yield points may in fact be needed for fusion,
115+ see `splitOnSeq` for an example. Or maybe its fusing because of a
116+ direct yield instead of going through an indirect common yielding
117+ state.
118+
119+Recursion in step function:
120+
121+* In general, we avoid making the step function recursive. Use the
122+ `Skip` constructor to remove recursion. Recursive step function can
123+ introduce optimization barriers that are harder to remove by the GHC
124+ simplifier.
125+
126+ However, in some cases it may be better to have a local recursive
127+ loop. A recursive loop can help us avoid threading around some large
128+ state values every time. Values that do not change across a loop can
129+ be factored in the scope outside the loop (static argument transform),
130+ this way we can create a local loop which is more efficient than
131+ threading around the state in a larger loop. See the `splitOnSeq`
132+ combinator as an example where we use a local recursive loop, it fuses
133+ and is significantly efficient compared to using `Skip`.
134+
135+ In a local recursive loop use SPEC and annotate even the rest of the
136+ loop arguments as strict where needed. We have observed that when the
137+ arguments were not strict the loop does not fuse (splitOnSeq).
138+
139+### Fold and Parser drivers
50140
51141Recursive loop closing operations:
52142
@@ -55,7 +145,7 @@ Recursive loop closing operations:
55145 "Spec Constructor" optimization removes boxed arguments and reduces
56146 allocations.
57147
58−## Fold and Parser combinators
148+### Fold and Parser combinators
59149
60150State of a Fold or Parser:
61151
@@ -70,16 +160,16 @@ The step function of a Fold or Parser:
70160
71161* Never make the fold step recursive, recursion creates an optimization barrier
72162 for the GHC simplifier. Use `Partial` or `Continue` constructors to avoid
73− recursion.
163+ recursion.
74164
75165* Sometimes you may need an explicit INLINE on the step function.
76166
77−## NOINLINE, isolating the closed loop
167+### NOINLINE, isolating the closed loop
78168
79169We want the loop iterations to be optimized and the loop stages to be fused to
80170generate a tighter loop. However, it is not necessarily optimal to inline the
81171whole loop itself into a parent function. For example, consider the following
82−function:
172+function in the `FileSystem.Handle` benchmarks:
83173
84174```
85175{-# NOINLINE readWriteAfter_Stream #-}
@@ -91,13 +181,20 @@ readWriteAfter_Stream inh devNull =
91181
92182If this is inlined into a parent benchmark group list, this leads to
93183many times performance degradation. That's because inlining the loop
94−into a bigger structure may interfere with the optimization of the loop
95−itself and it may not fuse. Whereas it is desirable to INLINE all
184+into a bigger structure interferes with the optimization of the loop
185+body itself and it may not fuse. Whereas it is desirable to INLINE all
96186the stages of a loop, it is often not useful to inline the whole loop
97187itself, in fact we may have to occasionally use a `NOINLINE` so that the
98188compiler does not inline it.
99189
100−## StreamK operations
190+### NoSpecConstr
191+
192+It is not always useful to specialize function calls for all constructors, some
193+constructors may just add to code bloat or add overhead of passing unboxed
194+values in a loop. In such cases the `NoSpecConstr` annotation can be useful.
195+See the `parselMx'` function in `StreamD` module.
196+
197+### StreamK operations
101198
102199StreamK uses foldr/build fusion to a very limited degree. StreamK is not the
103200primary representation in streamly but is used for several operations that
@@ -105,12 +202,21 @@ cannot scale in StreamD representation. StreamK is relatively immune to compiler
105202optimizations. In some cases you may need an INLINE pragma to improve the
106203performance.
107204
108−# GHC Optimizations
205+### How to debug non-fusing code?
206+
207+Strip down the code to a minimal version until it starts fusing and then
208+start building it up from there adding more things incrementally. At
209+each stage keep checking if the code is fusing. At some point it
210+won't fuse. See what we added that made the code not fuse. Go through
211+the guidelines above to check if we did something that that is not
212+recommended. Or raise an issue for GHC to be fixed if possible.
213+
214+# GHC Optimizations In Streamly
109215
110216There are three important levels of optimizations used in streamly:
111217
112−* CPS style stream representation (StreamK) to direct style (StreamD) and
113− vice-versa using rewrite rules
218+* CPS style stream representation (StreamK) to direct style (StreamD)
219+ conversion and vice-versa using rewrite rules
114220* Stream fusion using case-of-case optimization in StreamD
115221* foldr/build fusion using rewrite rules in StreamK
116222