@@ -13,57 +13,88 @@
13-- A CPS style stream using a constructor based representation instead of a
13-- A CPS style stream using a constructor based representation instead of a
14-- function based representation.
14-- function based representation.
16+-- Streamly internally uses two fundamental stream representations, (1) streams
17+-- with an open or arbitrary control flow (we call it StreamK), (2) streams
18+-- with a structured or closed loop control flow (we call it StreamD). The
19+-- higher level stream types can use any of these representations under the
20+-- hood and can interconvert between the two.
16-- StreamD is a non-recursive data type in which the state of the stream and
24-- StreamD is a non-recursive data type in which the state of the stream and
17-- the step function are separate. When the step function is called, a stream
25-- the step function are separate. When the step function is called, a stream
18-- element and the new stream state is yielded. The generated element and the
26-- element and the new stream state is yielded. The generated element and the
19-- state are passed to the next consumer in the loop. The state is threaded
27-- state are passed to the next consumer in the loop. The state is threaded
20-- around in the loop until control returns back to the original step function
28-- around in the loop until control returns back to the original step function
21−-- to run the next step. This creates a closed loop representation with state
29+-- to run the next step. This creates a structured closed loop representation
22−-- of each step being hidden/abstracted or existential within that step. This
30+-- (like "for" loops in C) with state of each step being hidden/abstracted or
31+-- existential within that step. This creates a loop representation identical
23−-- creates a loop reprentation identical to the for an while loop constructs in
32+-- to the "for" or "while" loop constructs in imperative languages, the states
24−-- imperative languages, the states of the steps combined together constitutes
25−-- the state of the loop iteration.
33+-- of the steps combined together constitute the state of the loop iteration.
27−-- Because of a closed representation consing too many elements in this type of
35+-- Internally most combinators use a closed loop representation because it
28−-- stream does not scale. Each cons creates a layer that needs to return the
36+-- provides very high efficiency due to stream fusion. The performance of this
37+-- representation is competitive to the C language implementations.
39+-- Pros and Cons of StreamD:
29−-- control back to the caller. However, this representation can be optimized
41+-- 1) stream-fusion: This representation can be optimized very efficiently by
30−-- very efficiently by the compiler because the state is explicitly separated
42+-- the compiler because the state is explicitly separated from step functions,
31−-- from the functions, represented using pure data constructors and visible to
43+-- represented using pure data constructors and visible to the compiler, the
32−-- the compiler, the stream steps can be fused using case-of-case
44+-- stream steps can be fused using case-of-case transformations and the state
33−-- transformations and the state can be specialized using spec-constructor
45+-- can be specialized using spec-constructor optimization, yielding a C like
34−-- optimization, yielding a C like tight loop/state machine with no
46+-- tight loop/state machine with no constructors, the state is used unboxed and
35−-- constructors and therefore no unnecessary allocation. Because of the closed
47+-- therefore no unnecessary allocation.
36−-- nature of the stream, exception handling is disciplined in this model
48+--
49+-- 2) Because of a closed representation consing too many elements in this type
50+-- of stream does not scale, it will have quadratic performance slowdown. Each
51+-- cons creates a layer that needs to return the control back to the caller.
52+-- Another implementation of cons is possible but that will have to box/unbox
53+-- the state and will not fuse. So effectively cons breaks fusion.
55+-- 3) unconsing an item from the stream breaks fusion, we have to "pause" the
56+-- loop, rebox and save the state.
58+-- 3) Exception handling is easy to implement in this model because control
37−-- because control flow is structured in the loop and cannot be arbitrary.
59+-- flow is structured in the loop and cannot be arbitrary. Therefore,
38−-- Also, because of the closed nature of the loop, it is not possible to uncons
60+-- implementing "bracket" is natural.
39−-- an item from the stream.
61+--
62+-- 4) Round-robin scheduling for co-operative multitasking is easy to implement.
64+-- 5) It fuses well with the direct style Fold implementation.
41−-- This type is amenable to stream fusion.
66+-- StreamK/StreamDK:
43-- StreamDK i.e. the stream defined in this module, like StreamK, is a
68-- StreamDK i.e. the stream defined in this module, like StreamK, is a
44−-- recursive data type which has no explicit state, each step yields an element
69+-- recursive data type which has no explicit state defined using constructors,
45−-- and a computation representing the rest of the stream. Stream state is part
70+-- each step yields an element and a computation representing the rest of the
46−-- of the function representing the rest of the stream. This creates an open
71+-- stream. Stream state is part of the function representing the rest of the
47−-- computation representation, or essentially a continuation passing style
72+-- stream. This creates an open computation representation, or essentially a
48−-- computation. After the stream step is executed, the caller is free to
73+-- continuation passing style computation. After the stream step is executed,
49−-- consume the produced element and then send the control wherever it wants,
74+-- the caller is free to consume the produced element and then send the control
50−-- there is no restriction on the control to return back somewhere, the control
75+-- wherever it wants, there is no restriction on the control to return back
51−-- is free to go anywhere. The caller may decide not to consume the rest of the
76+-- somewhere, the control is free to go anywhere. The caller may decide not to
52−-- stream.
77+-- consume the rest of the stream. This representation is more like a "goto"
53−--
78+-- based implementation in imperative languages.
54−-- This model allows generating arbitrarily large streams lazily using the cons
79+--
55−-- operation. Addition of more elements does not add any cost. Similarly uncons
80+-- Pros and Cons of StreamK:
82+-- 1) The way StreamD can be optimized using stream-fusion, this type can be
83+-- optimized using foldr/build fusion. However, foldr/build has not yet been
84+-- fully implemented for StreamK/StreamDK.
86+-- 2) Using cons is natural in this representation, unlike in StreamD it does
87+-- not have a quadratic slowdown. Currently, we in fact wrap StreamD in StreamK
88+-- to support a better cons operation.
56−-- operation is possible in this stream representation. The control flow can be
90+-- 3) Similarly, uncons is natural in this representation.
57−-- manipulated arbitrarily, there is no structure dictated by the stream. This
91+--
58−-- allows interoperation/interconversion with other monads, and other stream
92+-- 4) Exception handling is not easy to implement because of the "goto" nature
59−-- types. Scalable cons operation and the possibility of uncons operation makes
93+-- of CPS.
60−-- this representation very similar to standard lists. Also, this type
94+--
61−-- facilitates a scalable monadic composition.
95+-- 5) Composable folds are not implemented/proven, however, intuition says that
62−--
96+-- a push style CPS representation should be able to be used along with StreamK
63−-- This model has a cost because the open representation requires a constructor
97+-- to efficiently implement composable folds.
64−-- which may not always be eliminated.
66−-- This type is amenable to foldr/build fusion.
68module Streamly.Streams.StreamDK.Type
99module Streamly.Streams.StreamDK.Type
69 ( Step(..)
100 ( Step(..)