@@ -90,7 +90,7 @@ data ChildEvent a =
90------------------------------------------------------------------------------
90------------------------------------------------------------------------------
92-- | Conjunction is used for monadic/product style composition. Disjunction is
92-- | Conjunction is used for monadic/product style composition. Disjunction is
93−-- used for fold/sum style composition. We need to distiguish the two types of
93+-- used for fold/sum style composition. We need to distinguish the two types of
94-- SVars so that the scheduling of the two is independent.
94-- SVars so that the scheduling of the two is independent.
95data SVarTag = Conjunction | Disjunction deriving Eq
95data SVarTag = Conjunction | Disjunction deriving Eq
@@ -141,13 +141,65 @@ data SVar m a =
141------------------------------------------------------------------------------
141------------------------------------------------------------------------------
143-- TBD use a functor instead of the bare type a?
143-- TBD use a functor instead of the bare type a?
144−-- XXX remove the Maybe, use "empty" as the base case
146−-- | Represents a monadic stream of values of type 'a' constructed using
145+-- | The type 'Stream m a' represents a monadic stream of values of type 'a'
147−-- actions in monad 'm'. Streams can be composed sequentially or in parallel;
146+-- constructed using actions in monad 'm'. It uses a stop continuation and a
148−-- in product style compositions (monadic bind multiplies streams in a ListT
147+-- yield continuation. You can consider it a rough equivalent of direct style
149−-- fashion) or in sum style compositions like 'Semigroup', 'Monoid',
148+-- type:
150−-- 'Alternative' or variants of these.
149+--
150+-- data Stream m a = Stop | Yield a (Maybe (Stream m a))
152+-- Our goal is to be able to represent finite as well infinite streams and
153+-- being able to compose a large number of small streams efficiently. In
154+-- addition we want to compose streams in parallel, to facilitate that we
155+-- maintain a local state in an SVar that is shared across and is used for
156+-- synchronization of the streams being composed.
158+-- Using this type, there are two ways to indicate the end of a stream, one is
159+-- by calling the stop continuation and the other one is by yielding the last
160+-- value along with 'Nothing' as the rest of the stream.
162+-- Why do we have this redundancy? Why can't we use (a -> Stream m a -> m r) as
163+-- the type of the yield continuation and always use the stop continuation to
164+-- indicate the end of the stream? The reason is that when we compose a large
165+-- number of short or singleton streams then using the stop continuation
166+-- becomes expensive, just to know that there is no next element we have to
167+-- call the continuation, introducing an indirection, it seems when using CPS
168+-- GHC is not able to optimize this out as efficiently as it can be in direct
169+-- style because of the function call involved. In direct style it will just be
170+-- a constructor check and a memory access instead of a function call. So we
173+-- data Stream m a = Stop | Yield a (Stream m a)
175+-- In CPS style, when we use the 'Maybe' argument of yield to indicate the end
176+-- then just like direct style we can figure out that there is no next element
177+-- without a function call.
179+-- Then why not get rid of the stop continuation and use only yield to indicate
180+-- the end of stream? The answer is, in that case to indicate the end of the
181+-- stream we would have to yield at least one element so there is no way to
182+-- represent an empty stream.
184+-- Whenever we make a singleton stream or in general when we build a stream
185+-- strictly i.e. when we know all the elements of the stream in advance we can
186+-- use the last yield to indicate th end of the stream, because we know in
187+-- advance at the time of the last yield that the stream is ending. We build
188+-- singleton streams in the implementation of 'pure' for Applicative and Monad,
189+-- and in 'lift' for MonadTrans, in these places we use yield with 'Nothing' to
190+-- indicate the end of the stream. Note that, the only advantage of Maybe is
191+-- when we have to build a large number of singleton or short streams. For
192+-- larger streams anyway the overhead of a separate stop continuation is not
193+-- significant. This could be significant when we breakdown a large stream into
194+-- its elements, process them in some way and then recompose it from the
195+-- pieces. Zipping streams is one such example. Zipping with streamly is the
196+-- fastest among all streaming libraries.
198+-- However in a lazy computation we cannot know in advance that the stream is
199+-- ending therefore we cannot use 'Maybe', we use the stop continuation in that
200+-- case. For example when building a stream from a lazy container using a right
151newtype Stream m a =
203newtype Stream m a =
153 runStream :: forall r.
205 runStream :: forall r.
@@ -168,6 +220,15 @@ scons a r = Stream $ \_ _ yld -> yld a r
168snil :: Stream m a
220snil :: Stream m a
169snil = Stream $ \_ stp _ -> stp
221snil = Stream $ \_ stp _ -> stp
223+------------------------------------------------------------------------------
225+------------------------------------------------------------------------------
227+-- Streams can be composed sequentially or in parallel; in product style
228+-- compositions (monadic bind multiplies streams in a ListT fashion) or in sum
229+-- style compositions like 'Semigroup', 'Monoid', 'Alternative' or variants of
171------------------------------------------------------------------------------
232------------------------------------------------------------------------------
172-- Semigroup
233-- Semigroup
173------------------------------------------------------------------------------
234------------------------------------------------------------------------------