@@ -10,12 +10,123 @@
1010-- Maintainer : streamly@composewell.com
1111-- Stability : experimental
1212-- Portability : GHC
13+--
14+-- = Stream Consumers
15+--
16+-- We can classify stream consumers in the following categories in order of
17+-- increasing complexity and power:
18+--
19+-- == Accumulators
20+--
21+-- These are the simplest folds that never fail and never terminate, they
22+-- accumulate the input values forever and always remain @partial@ and
23+-- @complete@ at the same time. It means that we can keep adding more input to
24+-- them or at any time retrieve a consistent result. A
25+-- 'Streamly.Internal.Data.Fold.sum' operation is an example of an accumulator.
26+--
27+-- We can distribute an input stream to two or more accumulators using a @tee@
28+-- style composition. Accumulators cannot be applied on a stream one after the
29+-- other, which we call a @split@ style composition, as the first one itself
30+-- will never terminate, therefore, the next one will never get to run.
31+--
32+-- == Splitters
33+--
34+-- Splitters are accumulators that can terminate. When applied on a stream
35+-- splitters consume part of the stream, thereby, splitting it. Splitters can
36+-- be used in a @split@ style composition where one splitter can be applied
37+-- after the other on an input stream. We can apply a splitter repeatedly on an
38+-- input stream splitting and consuming it in fragments. Splitters never fail,
39+-- therefore, they do not need backtracking, but they can lookahead and return
40+-- unconsumed input. The 'Streamly.Internal.Data.Parser.take' operation is an
41+-- example of a splitter. It terminates after consuming @n@ items. Coupled with
42+-- an accumulator it can be used to split the stream into chunks of fixed size.
43+--
44+-- Consider the example of @takeWhile@ operation, it needs to inspect an
45+-- element for termination decision. However, it does not consume the element
46+-- on which it terminates. To implement @takeWhile@ a splitter will have to
47+-- implement a way to return unconsumed input to the driver.
48+--
49+-- == Parsers
50+--
51+-- Parsers are splitters that can fail and backtrack. Parsers can be composed
52+-- using an @alternative@ style composition where they can backtrack and apply
53+-- another parser if one parser fails. 'Streamly.Internal.Data.Parser.satisfy'
54+-- is a simple example of a parser, it would succeed if the condition is
55+-- satisfied and it would fail otherwise, on failure an alternative parser can
56+-- be used on the same input.
57+--
58+-- = Types for Stream Consumers
59+--
60+-- We use the 'Fold' type to implement the Accumulator and Splitter
61+-- functionality. Parsers are represented by the
62+-- 'Streamly.Internal.Data.Parser.Parser' type. This is a sweet spot to
63+-- balance ease of use, type safety and performance. Using separate
64+-- Accumulator and Splitter types would encode more information in types but it
65+-- would make ease of use, implementation, maintenance effort worse. Combining
66+-- Accumulator, Splitter and Parser into a single
67+-- 'Streamly.Internal.Data.Parser.Parser' type would make ease of use even
68+-- better but type safety and performance worse.
69+--
70+-- One of the design requirements that we have placed for better ease of use
71+-- and code reuse is that 'Streamly.Internal.Data.Parser.Parser' type should be
72+-- a strict superset of the 'Fold' type i.e. it can do everything that a 'Fold'
73+-- can do and more. Therefore, folds can be easily upgraded to parsers and we
74+-- can use parser combinators on folds as well when needed.
75+--
76+-- = Fold Design
77+--
78+-- A fold is represented by a collection of "initial", "step" and "extract"
79+-- functions. The "initial" action generates the initial state of the fold. The
80+-- state is internal to the fold and maintains the accumulated output. The
81+-- "step" function is invoked using the current state and the next input value
82+-- and results in a @Yield@ or @Stop@. A @Yield@ returns the next intermediate
83+-- state of the fold, a @Stop@ indicates that the fold has terminated and
84+-- returns the final value of the accumulator.
85+--
86+-- Every @Yield@ indicates that a new accumulated output is available. The
87+-- accumulated output can be extracted from the state at any point using
88+-- "extract". "extract" can never fail. A fold returns a valid output even
89+-- without any input i.e. even if you call "extract" on "initial" state it
90+-- provides an output. This is not true for parsers.
91+--
92+-- In general, "extract" is used in two cases:
93+--
94+-- * When the fold is used as a scan @extract@ is called on the intermediate
95+-- state every time it is yielded by the fold, the resulting value is yielded
96+-- as a stream.
97+-- * When the fold is used as a regular fold, @extract@ is called once when
98+-- we are done feeding input to the fold.
99+--
100+-- = Alternate Designs
101+--
102+-- An alternate and simpler design would be to return the intermediate output
103+-- via @Yield@ along with the state, instead of using "extract" on the yielded
104+-- state and remove the extract function altogether.
105+--
106+-- This may even facilitate more efficient implementation. Extract from the
107+-- intermediate state after each yield may be more costly compared to the fold
108+-- step itself yielding the output. The fold may have more efficient ways to
109+-- retrieve the output rather than stuffing it in the state and using extract
110+-- on the state.
111+--
112+-- However, removing extract altogether may lead to less optimal code in some
113+-- cases because the driver of the fold needs to thread around the intermediate
114+-- output to return it if the stream stops before the fold could @Stop@. When
115+-- using this approach, the @splitParse (FL.take filesize)@ benchmark shows a
116+-- 2x worse performance even after ensuring everything fuses. So we keep the
117+-- "extract" approach to ensure better perf in all cases.
118+--
119+-- But we could still yield both state and the output in @Yield@, the output
120+-- can be used for the scan use case, instead of using extract. Extract would
121+-- then be used only for the case when the stream stops before the fold
122+-- completes.
13123
14124module Streamly.Internal.Data.Fold.Types
15125 ( Fold (..)
16126 , Fold2 (..)
17127 , simplify
18128 , toListRevF -- experimental
129+ -- $toListRevF
19130 , lmap
20131 , lmapM
21132 , lfilter
@@ -52,13 +163,13 @@ import Streamly.Internal.Data.SVar (MonadAsync)
52163-- Monadic left folds
53164------------------------------------------------------------------------------
54165
55−-- | Represents a left fold over an input stream of values of type @a@ to a
56−-- single value of type @b@ in 'Monad' @m@.
166+-- | Represents a left fold over an input stream consisting of values of type
167+-- @a@ to a single value of type @b@ in 'Monad' @m@.
57168--
58169-- The fold uses an intermediate state @s@ as accumulator. The @step@ function
59−-- updates the state and returns the new updated state. When the fold is done
170+-- updates the state and returns the new state. When the fold is done
60171-- the final result of the fold is extracted from the intermediate state
61−-- representation using the @extract@ function.
172+-- using the @extract@ function.
62173--
63174-- @since 0.7.0
64175
@@ -66,7 +177,7 @@ data Fold m a b =
66177 -- | @Fold @ @ step @ @ initial @ @ extract@
67178 forall s. Fold (s -> a -> m s) (m s) (s -> m b)
68179
69−-- Experimental type to provide a side input to the fold for generating the
180+-- | Experimental type to provide a side input to the fold for generating the
70181-- initial state. For example, if we have to fold chunks of a stream and write
71182-- each chunk to a different file, then we can generate the file name using a
72183-- monadic action. This is a generalized version of 'Fold'.
@@ -208,9 +319,10 @@ instance (Monad m, Floating b) => Floating (Fold m a b) where
208319-- Internal APIs
209320------------------------------------------------------------------------------
210321
211−-- This is more efficient than 'toList'. toList is exactly the same as
212−-- reversing the list after toListRev.
213−--
322+-- $toListRevF
323+-- This is more efficient than 'Streamly.Internal.Data.Fold.toList'. toList is
324+-- exactly the same as reversing the list after 'toListRevF'.
325+
214326-- | Buffers the input stream to a list in the reverse order of the input.
215327--
216328-- /Warning!/ working on large lists accumulated as buffers in memory could be
@@ -283,7 +395,7 @@ lcatMaybes = lfilter isJust . lmap fromJust
283395
284396-- XXX These should become terminating folds.
285397--
286−-- | Take first 'n' elements from the stream and discard the rest.
398+-- | Take first @n@ elements from the stream and discard the rest.
287399--
288400-- @since 0.7.0
289401{-# INLINABLE ltake #-}