@@ -8,15 +8,20 @@
8-- Stability : released
8-- Stability : released
9-- Portability : GHC
9-- Portability : GHC
11−-- Fast, composable stream producers with ability to terminate, supporting
11+-- Streams represented as state machines, that fuse together when composed
12−-- stream fusion.
12+-- statically, eliminating function calls or intermediate constructor
13+-- allocations - generating tight, efficient loops. Suitable for high
14+-- performance looping operations.
16+-- If you need to call these operations recursively in a loop (i.e. composed
17+-- dynamically) then it is recommended to use the continuation passing style
18+-- (CPS) stream operations from the "Streamly.Data.StreamK" module. 'Stream'
19+-- and 'StreamK' types are interconvertible. See more details in the
20+-- documentation below regarding 'Stream' vs 'StreamK'.
14-- Please refer to "Streamly.Internal.Data.Stream" for more functions that have
22-- Please refer to "Streamly.Internal.Data.Stream" for more functions that have
15-- not yet been released.
23-- not yet been released.
17−-- For continuation passing style (CPS) stream type, please refer to
18−-- the "Streamly.Data.StreamK" module.
20-- Checkout the <https://github.com/composewell/streamly-examples>
25-- Checkout the <https://github.com/composewell/streamly-examples>
21-- repository for many more real world examples of stream programming.
26-- repository for many more real world examples of stream programming.
@@ -49,11 +54,12 @@ module Streamly.Data.Stream
49 -- >>> fromFoldableM = Stream.sequence . fromFoldable
54 -- >>> fromFoldableM = Stream.sequence . fromFoldable
51 -- ** Primitives
56 -- ** Primitives
52− -- | A fused 'Stream' is never constructed using 'cons', we always convert
57+ -- | A fused 'Stream' is never constructed using these primitives, they are
53− -- other type of containers like list into streams, or generate it using
58+ -- typically generated by converting containers like list into streams, or
54− -- custom functions provided in this module. The 'cons' primitive has a
59+ -- generated using custom functions provided in this module. The 'cons'
55− -- rare use in fusing a small number of elements. On the other hand, it is
60+ -- primitive in this module has a rare use in fusing a small number of
61+ -- elements. On the other hand, it is common to construct 'StreamK' stream
56− -- common to construct 'StreamK' stream using the 'StreamK.cons' primitive.
62+ -- using the StreamK.'StreamK.cons' primitive.
@@ -358,32 +364,31 @@ module Streamly.Data.Stream
358 -- , elemIndices
364 -- , elemIndices
360 -- * Combining Two Streams
366 -- * Combining Two Streams
367+ -- | Note that these operations are suitable for statically fusing a few
368+ -- streams, they have a quadratic O(n^2) time complexity wrt to the number
369+ -- of streams. If you want to compose many streams dynamically using binary
370+ -- combining operations see the corresponding operations in
371+ -- "Streamly.Data.StreamK".
373+ -- When fusing more than two streams it is more efficient if the binary
374+ -- operations are composed as a balanced tree rather than a right
375+ -- associative or left associative one e.g.:
377+ -- >>> s1 = Stream.fromList [1,2] `Stream.append` Stream.fromList [3,4]
378+ -- >>> s2 = Stream.fromList [4,5] `Stream.append` Stream.fromList [6,7]
379+ -- >>> s = s1 `Stream.append` s2
361 -- ** Appending
381 -- ** Appending
364 -- ** Interleaving
384 -- ** Interleaving
365− -- | When interleaving more than two streams you may want to interleave
366− -- them pairwise creating a balanced binary merge tree.
367 , interleave
385 , interleave
369 -- ** Merging
387 -- ** Merging
370− -- | When merging more than two streams you may want to merging them
371− -- pairwise creating a balanced binary merge tree.
373− -- Merging of @n@ streams can be performed by combining the streams pair
374− -- wise using 'mergeMapWith' to give O(n * log n) time complexity. If used
375− -- with 'concatMapWith' it will have O(n^2) performance.
377 , mergeBy
388 , mergeBy
378 , mergeByM
389 , mergeByM
380 -- ** Zipping
391 -- ** Zipping
381− -- | When zipping more than two streams you may want to zip them
382− -- pairwise creating a balanced binary tree.
384− -- Zipping of @n@ streams can be performed by combining the streams pair
385− -- wise using 'mergeMapWith' with O(n * log n) time complexity. If used
386− -- with 'concatMapWith' it will have O(n^2) performance.
387 , zipWith
392 , zipWith
388 , zipWithM
393 , zipWithM
389 -- , ZipStream (..)
394 -- , ZipStream (..)
@@ -573,11 +578,11 @@ import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
574-- == Stream Fusion
579-- == Stream Fusion
576−-- The fused 'Stream' type employs stream fusion for C-like performance when
581+-- The fused 'Stream' type in this module employs stream fusion for C-like
577−-- looping over data. It represents a stream source or transformation by
582+-- performance when looping over data. It represents the stream as a state
578−-- defining a state machine with explicit state, and a step function working on
583+-- machine using an explicit state, and a step function working on the state. A
579−-- the state. A typical stream operation consumes elements from the previous
584+-- typical stream operation consumes elements from the previous state machine
580−-- state machine in the pipeline, transforms them and yields new values for the
585+-- in a stream pipeline, transforms the elements and yields new values for the
581-- next stage to consume. The stream operations are modular and represent a
586-- next stage to consume. The stream operations are modular and represent a
582-- single task, they have no knowledge of previous or next operation on the
587-- single task, they have no knowledge of previous or next operation on the
583-- elements.
588-- elements.
@@ -586,7 +591,7 @@ import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
586-- transformation operations and a stream consumer. All these operations taken
591-- transformation operations and a stream consumer. All these operations taken
587-- together form a closed loop processing the stream elements. Elements are
592-- together form a closed loop processing the stream elements. Elements are
588-- transferred between stages using a boxed data constructor. However, all the
593-- transferred between stages using a boxed data constructor. However, all the
589−-- stages of the pipeline are fused together by GHC, eliminating the
594+-- stages of the pipeline are fused together by GHC, eliminating the boxing of
590-- intermediate constructors, and thus forming a tight C like loop without any
595-- intermediate constructors, and thus forming a tight C like loop without any
591-- boxed data being used in the loop.
596-- boxed data being used in the loop.
@@ -601,43 +606,45 @@ import qualified Streamly.Internal.Data.Stream.Chunked as ArrayStream
602-- == Stream vs StreamK
607-- == Stream vs StreamK
604−-- The fused stream model avoids constructor allocations or function call
609+-- The fused stream model avoids constructor allocations and function call
605−-- overheads. However, the stream is represented as a state machine and to
610+-- overheads. However, the stream is represented as a state machine, and to
606−-- generate elements it has to navigate the decision tree of the state machine.
611+-- generate stream elements it has to navigate the decision tree of the state
607−-- Moreover, the state machine is cranked for each element in the stream. This
612+-- machine. Moreover, the state machine is cranked for each element in the
608−-- performs extremely well when the number of states are limited. The state
613+-- stream. This performs extremely well when the number of states are limited.
609−-- machine starts getting expensive as the number of states increase. For
614+-- The state machine starts getting expensive as the number of states increase.
610−-- example, generating a million element stream from a list requires a single
615+-- For example, generating a stream from a list requires a single state and is
611−-- state and is very efficient. However, using fused 'cons' to generate a
616+-- very efficient, even if it has millions of elements. However, using 'cons'
612−-- million element stream would be a disaster.
617+-- to construct a million element stream would be a disaster.
614-- A typical worst case scenario for fused stream model is a large number of
619-- A typical worst case scenario for fused stream model is a large number of
615-- `cons` or `append` operations. A few static `cons` or `append` operations
620-- `cons` or `append` operations. A few static `cons` or `append` operations
616−-- are very fast and much faster than a CPS style stream. However, if we
621+-- are very fast and much faster than a CPS style stream because CPS involves a
622+-- function call for each element whereas fused stream involves a few
623+-- conditional branches in the state machine. However, constructing a large
617−-- construct a large stream using `cons` it introduces as many states in the
624+-- stream using `cons` introduces as many states in the state machine as the
618−-- state machine as the number of elements. If we compose the `cons` as a
625+-- number of elements. If we compose `cons` as a balanced binary tree it will
619−-- binary tree it will take @n * log n@ time to navigate the tree, and @n * n@
626+-- take @n * log n@ time to navigate the tree, and @n * n@ if it is a right
620−-- if it is a right associative composition.
627+-- associative composition.
622−-- For quadratic cases of fused stream, after a certain threshold the CPS
623−-- stream would perform much better and exhibit linear performance behavior.
624-- Operations like 'cons' or 'append'; are typically recursively called to
629-- Operations like 'cons' or 'append'; are typically recursively called to
625-- construct a lazy infinite stream. For such use cases the CPS style 'StreamK'
630-- construct a lazy infinite stream. For such use cases the CPS style 'StreamK'
626−-- type is provided. CPS streams do not have a state machine that needs to be
631+-- should be used. CPS streams do not have a state machine that needs to be
627-- cranked for each element, past state has no effect on the future element
632-- cranked for each element, past state has no effect on the future element
628−-- processing. However, it incurs a function call overhead for each operation
633+-- processing. However, CPS incurs a function call overhead for each element
629−-- for each element, which could be very large overhead compared to fused state
634+-- processed, the overhead could be large compared to a fused state machine
630−-- machines even if it has many states and cranks it for each element. But in
635+-- even if it has many states. However, because of its linear performance
631−-- some cases scales tip in favor of the CPS stream. In those cases even though
636+-- characterstics, after a certain threshold of stream compositions the CPS
632−-- CPS has a large constant overhead, it has a linear performance rather than
637+-- stream would perform much better than the quadratic fused stream operations.
635−-- As a general guideline, if you have to use 'cons' or 'append' or operations
639+-- As a general guideline, you need to use 'StreamK' when you have to use
636−-- of similar nature, at a large scale, then 'StreamK' should be used. When you
640+-- 'cons', 'append' or other operations having quadratic complexity at a large
637−-- need to compose the stream dynamically or recursively, then 'StreamK' should
641+-- scale. Typically, in such cases you need to compose the stream recursively,
642+-- by calling an operation in a loop. The decision to compose the stream is
643+-- taken at run time rather than statically at compile time.
638−-- be used. Typically you would use a dynamically generated 'StreamK' with
645+-- Typically you would compose a 'StreamK' of chunks of data so that the
639−-- chunks of data which can then be processed by statically fused stream
646+-- StreamK overhead is not high, and then process the chunks using 'Stream' by
640−-- pipeline operations.
647+-- using statically fused stream pipeline operations on the chunks.
642-- 'Stream' and 'StreamK' types can be interconverted. See
649-- 'Stream' and 'StreamK' types can be interconverted. See
643-- "Streamly.Data.StreamK" module for conversion operations.
650-- "Streamly.Data.StreamK" module for conversion operations.