@@ -90,7 +90,7 @@ data ChildEvent a =
9090------------------------------------------------------------------------------
9191
9292-- | Conjunction is used for monadic/product style composition. Disjunction is
93−-- used for fold/sum style composition. We need to distiguish the two types of
93+-- used for fold/sum style composition. We need to distinguish the two types of
9494-- SVars so that the scheduling of the two is independent.
9595data SVarTag = Conjunction | Disjunction deriving Eq
9696
@@ -141,13 +141,65 @@ data SVar m a =
141141------------------------------------------------------------------------------
142142
143143-- TBD use a functor instead of the bare type a?
144−-- XXX remove the Maybe, use "empty" as the base case
145144
146−-- | Represents a monadic stream of values of type 'a' constructed using
147−-- actions in monad 'm'. Streams can be composed sequentially or in parallel;
148−-- in product style compositions (monadic bind multiplies streams in a ListT
149−-- fashion) or in sum style compositions like 'Semigroup', 'Monoid',
150−-- 'Alternative' or variants of these.
145+-- | The type 'Stream m a' represents a monadic stream of values of type 'a'
146+-- constructed using actions in monad 'm'. It uses a stop continuation and a
147+-- yield continuation. You can consider it a rough equivalent of direct style
148+-- type:
149+--
150+-- data Stream m a = Stop | Yield a (Maybe (Stream m a))
151+--
152+-- Our goal is to be able to represent finite as well infinite streams and
153+-- being able to compose a large number of small streams efficiently. In
154+-- addition we want to compose streams in parallel, to facilitate that we
155+-- maintain a local state in an SVar that is shared across and is used for
156+-- synchronization of the streams being composed.
157+--
158+-- Using this type, there are two ways to indicate the end of a stream, one is
159+-- by calling the stop continuation and the other one is by yielding the last
160+-- value along with 'Nothing' as the rest of the stream.
161+--
162+-- Why do we have this redundancy? Why can't we use (a -> Stream m a -> m r) as
163+-- the type of the yield continuation and always use the stop continuation to
164+-- indicate the end of the stream? The reason is that when we compose a large
165+-- number of short or singleton streams then using the stop continuation
166+-- becomes expensive, just to know that there is no next element we have to
167+-- call the continuation, introducing an indirection, it seems when using CPS
168+-- GHC is not able to optimize this out as efficiently as it can be in direct
169+-- style because of the function call involved. In direct style it will just be
170+-- a constructor check and a memory access instead of a function call. So we
171+-- could use:
172+--
173+-- data Stream m a = Stop | Yield a (Stream m a)
174+--
175+-- In CPS style, when we use the 'Maybe' argument of yield to indicate the end
176+-- then just like direct style we can figure out that there is no next element
177+-- without a function call.
178+--
179+-- Then why not get rid of the stop continuation and use only yield to indicate
180+-- the end of stream? The answer is, in that case to indicate the end of the
181+-- stream we would have to yield at least one element so there is no way to
182+-- represent an empty stream.
183+--
184+-- Whenever we make a singleton stream or in general when we build a stream
185+-- strictly i.e. when we know all the elements of the stream in advance we can
186+-- use the last yield to indicate th end of the stream, because we know in
187+-- advance at the time of the last yield that the stream is ending. We build
188+-- singleton streams in the implementation of 'pure' for Applicative and Monad,
189+-- and in 'lift' for MonadTrans, in these places we use yield with 'Nothing' to
190+-- indicate the end of the stream. Note that, the only advantage of Maybe is
191+-- when we have to build a large number of singleton or short streams. For
192+-- larger streams anyway the overhead of a separate stop continuation is not
193+-- significant. This could be significant when we breakdown a large stream into
194+-- its elements, process them in some way and then recompose it from the
195+-- pieces. Zipping streams is one such example. Zipping with streamly is the
196+-- fastest among all streaming libraries.
197+--
198+-- However in a lazy computation we cannot know in advance that the stream is
199+-- ending therefore we cannot use 'Maybe', we use the stop continuation in that
200+-- case. For example when building a stream from a lazy container using a right
201+-- fold.
202+--
151203newtype Stream m a =
152204 Stream {
153205 runStream :: forall r.
@@ -168,6 +220,15 @@ scons a r = Stream $ \_ _ yld -> yld a r
168220snil :: Stream m a
169221snil = Stream $ \_ stp _ -> stp
170222
223+------------------------------------------------------------------------------
224+-- Composing streams
225+------------------------------------------------------------------------------
226+
227+-- Streams can be composed sequentially or in parallel; in product style
228+-- compositions (monadic bind multiplies streams in a ListT fashion) or in sum
229+-- style compositions like 'Semigroup', 'Monoid', 'Alternative' or variants of
230+-- these.
231+
171232------------------------------------------------------------------------------
172233-- Semigroup
173234------------------------------------------------------------------------------