@@ -15,53 +15,101 @@
1515-- == Accumulators
1616--
1717-- These are the simplest folds that never fail and never terminate, they
18−-- accumulate the input values forever and always remain @partial@ and
19−-- @complete@ at the same time. It means that we can keep adding more input to
20−-- them or at any time retrieve a consistent result. A
18+-- accumulate the input values forever and can always accept new inputs (never
19+-- terminate) and always have a valid result value. A
2120-- 'Streamly.Internal.Data.Fold.sum' operation is an example of an accumulator.
21+-- Traditional Haskell left folds like 'foldl' are accumulators.
2222--
2323-- We can distribute an input stream to two or more accumulators using a @tee@
2424-- style composition. Accumulators cannot be applied on a stream one after the
25−-- other, which we call a @split@ style composition, as the first one itself
26−-- will never terminate, therefore, the next one will never get to run.
27−--
28−-- == Splitters
29−--
30−-- Splitters are accumulators that can terminate. When applied on a stream
31−-- splitters consume part of the stream, thereby, splitting it. Splitters can
32−-- be used in a @split@ style composition where one splitter can be applied
33−-- after the other on an input stream. We can apply a splitter repeatedly on an
34−-- input stream splitting and consuming it in fragments. Splitters never fail,
35−-- therefore, they do not need backtracking, but they can lookahead and return
36−-- unconsumed input. The 'Streamly.Internal.Data.Parser.take' operation is an
37−-- example of a splitter. It terminates after consuming @n@ items. Coupled with
38−-- an accumulator it can be used to split the stream into chunks of fixed size.
39−--
40−-- Consider the example of @takeWhile@ operation, it needs to inspect an
41−-- element for termination decision. However, it does not consume the element
42−-- on which it terminates. To implement @takeWhile@ a splitter will have to
43−-- implement a way to return unconsumed input to the driver.
25+-- other, which we call a @serial@ append style composition of folds. This is
26+-- because accumulators never terminate, since the first accumulator in a
27+-- series will never terminate, the next one will never get to run.
28+--
29+-- == Terminating Folds
30+--
31+-- Terminating folds are accumulators that can terminate. Once a fold
32+-- terminates it no longer accepts any more inputs. Terminating folds can be
33+-- used in a @serial@ append style composition where one fold can be applied
34+-- after the other on an input stream. We can apply a terminating fold
35+-- repeatedly on an input stream, splitting the stream and consuming it in
36+-- fragments. Terminating folds never fail, therefore, they do not need
37+-- backtracking.
38+--
39+-- The 'Streamly.Internal.Data.Fold.take' operation is an example of a
40+-- terminating fold It terminates after consuming @n@ items. Coupled with an
41+-- accumulator (e.g. sum) it can be used to split and process the stream into
42+-- chunks of fixed size.
43+--
44+-- == Terminating Folds with Leftovers
45+--
46+-- The next upgrade after terminating folds is terminating folds with leftover
47+-- inputs. Consider the example of @takeWhile@ operation, it needs to inspect
48+-- an element for termination decision. However, it does not consume the
49+-- element on which it terminates. To implement @takeWhile@ a terminating fold
50+-- will have to implement a way to return unconsumed input to the fold driver.
51+--
52+-- Single element leftover case is the most common and its easy to implement it
53+-- in terminating folds using a @Done1@ constructor in the 'Step' type which
54+-- indicates that the last element was not consumed by the fold. The following
55+-- additional operations can be implemented as terminating folds if we do that.
56+--
57+-- @
58+-- takeWhile
59+-- groupBy
60+-- wordBy
61+-- @
62+--
63+-- However, it creates several complications. The 'many' combinator requires
64+-- a @Partial1@ ('Partial' with leftover) to handle a @Done1@ from the top
65+-- level fold, for efficient implementation. If the collecting fold in "many"
66+-- returns a @Partial1@ or @Done1@ then what to do with all the elements that
67+-- have been consumed?
68+--
69+-- Similarly, in distribute, if one fold consumes a value and others say its a
70+-- leftover then what do we do? Folds like "many" require the leftover to be
71+-- fed to it again. So in a distribute operation those folds which gave a
72+-- leftover will have to be fed the leftover while the folds that consumed will
73+-- have to be fed the next input. This is very complicated to implement. We
74+-- have the same issue in backtracking parsers being used in a distribute
75+-- operation.
76+--
77+-- To avoid these issues we want to enforce by typing that the collecting folds
78+-- can never return a leftover. So we need a fold type without @Done1@ or
79+-- @Partial1@. This leads us to design folds to never return a leftover and the
80+-- use cases of single leftover are transferred to parsers where we have
81+-- general backtracking mechanism and single leftover is just a special case of
82+-- backtracking.
83+--
84+-- This means: takeWhile, groupBy, wordBy would be implemented as parsers.
85+-- "take 0" can implemented as a fold if we make initial return @Step@ type.
86+-- "takeByTime" can be implemented without @Done1@.
4487--
4588-- == Parsers
4689--
47−-- Parsers are splitters that can fail and backtrack. Parsers can be composed
48−-- using an @alternative@ style composition where they can backtrack and apply
49−-- another parser if one parser fails. 'Streamly.Internal.Data.Parser.satisfy'
50−-- is a simple example of a parser, it would succeed if the condition is
51−-- satisfied and it would fail otherwise, on failure an alternative parser can
52−-- be used on the same input.
90+-- The next upgrade after terminating folds with a leftover are parsers.
91+-- Parsers are terminating folds that can fail and backtrack. Parsers can be
92+-- composed using an @alternative@ style composition where they can backtrack
93+-- and apply another parser if one parser fails.
94+-- 'Streamly.Internal.Data.Parser.satisfy' is a simple example of a parser, it
95+-- would succeed if the condition is satisfied and it would fail otherwise, on
96+-- failure an alternative parser can be used on the same input.
5397--
5498-- = Types for Stream Consumers
5599--
56−-- We use the 'Fold' type to implement the Accumulator and Splitter
57−-- functionality. Parsers are represented by the
58−-- 'Streamly.Internal.Data.Parser.Parser' type. This is a sweet spot to
59−-- balance ease of use, type safety and performance. Using separate
60−-- Accumulator and Splitter types would encode more information in types but it
61−-- would make ease of use, implementation, maintenance effort worse. Combining
62−-- Accumulator, Splitter and Parser into a single
63−-- 'Streamly.Internal.Data.Parser.Parser' type would make ease of use even
64−-- better but type safety and performance worse.
100+-- In streamly, there is no separate type for accumulators. Terminating folds
101+-- are a superset of accumulators and to avoid too many types we represent both
102+-- using the same type, 'Fold'.
103+--
104+-- We do not club the leftovers functionality with terminating folds because of
105+-- the reasons explained earlier. Instead combinators that require leftovers
106+-- are implemented as the 'Streamly.Internal.Data.Parser.Parser' type. This is
107+-- a sweet spot to balance ease of use, type safety and performance. Using
108+-- separate Accumulator and terminating fold types would encode more
109+-- information in types but it would make ease of use, implementation,
110+-- maintenance effort worse. Combining Accumulator, terminating folds and
111+-- Parser into a single 'Streamly.Internal.Data.Parser.Parser' type would make
112+-- ease of use even better but type safety and performance worse.
65113--
66114-- One of the design requirements that we have placed for better ease of use
67115-- and code reuse is that 'Streamly.Internal.Data.Parser.Parser' type should be