@@ -15,53 +15,101 @@
15-- == Accumulators
15-- == Accumulators
17-- These are the simplest folds that never fail and never terminate, they
17-- These are the simplest folds that never fail and never terminate, they
18−-- accumulate the input values forever and always remain @partial@ and
18+-- accumulate the input values forever and can always accept new inputs (never
19−-- @complete@ at the same time. It means that we can keep adding more input to
19+-- terminate) and always have a valid result value. A
20−-- them or at any time retrieve a consistent result. A
21-- 'Streamly.Internal.Data.Fold.sum' operation is an example of an accumulator.
20-- 'Streamly.Internal.Data.Fold.sum' operation is an example of an accumulator.
21+-- Traditional Haskell left folds like 'foldl' are accumulators.
23-- We can distribute an input stream to two or more accumulators using a @tee@
23-- We can distribute an input stream to two or more accumulators using a @tee@
24-- style composition. Accumulators cannot be applied on a stream one after the
24-- style composition. Accumulators cannot be applied on a stream one after the
25−-- other, which we call a @split@ style composition, as the first one itself
25+-- other, which we call a @serial@ append style composition of folds. This is
26+-- because accumulators never terminate, since the first accumulator in a
26−-- will never terminate, therefore, the next one will never get to run.
27+-- series will never terminate, the next one will never get to run.
28−-- == Splitters
29+-- == Terminating Folds
30−-- Splitters are accumulators that can terminate. When applied on a stream
31+-- Terminating folds are accumulators that can terminate. Once a fold
31−-- splitters consume part of the stream, thereby, splitting it. Splitters can
32+-- terminates it no longer accepts any more inputs. Terminating folds can be
32−-- be used in a @split@ style composition where one splitter can be applied
33+-- used in a @serial@ append style composition where one fold can be applied
33−-- after the other on an input stream. We can apply a splitter repeatedly on an
34+-- after the other on an input stream. We can apply a terminating fold
34−-- input stream splitting and consuming it in fragments. Splitters never fail,
35+-- repeatedly on an input stream, splitting the stream and consuming it in
35−-- therefore, they do not need backtracking, but they can lookahead and return
36+-- fragments. Terminating folds never fail, therefore, they do not need
36−-- unconsumed input. The 'Streamly.Internal.Data.Parser.take' operation is an
39+-- The 'Streamly.Internal.Data.Fold.take' operation is an example of a
37−-- example of a splitter. It terminates after consuming @n@ items. Coupled with
40+-- terminating fold It terminates after consuming @n@ items. Coupled with an
38−-- an accumulator it can be used to split the stream into chunks of fixed size.
41+-- accumulator (e.g. sum) it can be used to split and process the stream into
42+-- chunks of fixed size.
44+-- == Terminating Folds with Leftovers
46+-- The next upgrade after terminating folds is terminating folds with leftover
40−-- Consider the example of @takeWhile@ operation, it needs to inspect an
47+-- inputs. Consider the example of @takeWhile@ operation, it needs to inspect
41−-- element for termination decision. However, it does not consume the element
48+-- an element for termination decision. However, it does not consume the
42−-- on which it terminates. To implement @takeWhile@ a splitter will have to
49+-- element on which it terminates. To implement @takeWhile@ a terminating fold
43−-- implement a way to return unconsumed input to the driver.
50+-- will have to implement a way to return unconsumed input to the fold driver.
52+-- Single element leftover case is the most common and its easy to implement it
53+-- in terminating folds using a @Done1@ constructor in the 'Step' type which
54+-- indicates that the last element was not consumed by the fold. The following
55+-- additional operations can be implemented as terminating folds if we do that.
63+-- However, it creates several complications. The 'many' combinator requires
64+-- a @Partial1@ ('Partial' with leftover) to handle a @Done1@ from the top
65+-- level fold, for efficient implementation. If the collecting fold in "many"
66+-- returns a @Partial1@ or @Done1@ then what to do with all the elements that
67+-- have been consumed?
69+-- Similarly, in distribute, if one fold consumes a value and others say its a
70+-- leftover then what do we do? Folds like "many" require the leftover to be
71+-- fed to it again. So in a distribute operation those folds which gave a
72+-- leftover will have to be fed the leftover while the folds that consumed will
73+-- have to be fed the next input. This is very complicated to implement. We
74+-- have the same issue in backtracking parsers being used in a distribute
77+-- To avoid these issues we want to enforce by typing that the collecting folds
78+-- can never return a leftover. So we need a fold type without @Done1@ or
79+-- @Partial1@. This leads us to design folds to never return a leftover and the
80+-- use cases of single leftover are transferred to parsers where we have
81+-- general backtracking mechanism and single leftover is just a special case of
84+-- This means: takeWhile, groupBy, wordBy would be implemented as parsers.
85+-- "take 0" can implemented as a fold if we make initial return @Step@ type.
86+-- "takeByTime" can be implemented without @Done1@.
45-- == Parsers
88-- == Parsers
90+-- The next upgrade after terminating folds with a leftover are parsers.
47−-- Parsers are splitters that can fail and backtrack. Parsers can be composed
91+-- Parsers are terminating folds that can fail and backtrack. Parsers can be
48−-- using an @alternative@ style composition where they can backtrack and apply
92+-- composed using an @alternative@ style composition where they can backtrack
49−-- another parser if one parser fails. 'Streamly.Internal.Data.Parser.satisfy'
93+-- and apply another parser if one parser fails.
50−-- is a simple example of a parser, it would succeed if the condition is
94+-- 'Streamly.Internal.Data.Parser.satisfy' is a simple example of a parser, it
51−-- satisfied and it would fail otherwise, on failure an alternative parser can
95+-- would succeed if the condition is satisfied and it would fail otherwise, on
52−-- be used on the same input.
96+-- failure an alternative parser can be used on the same input.
54-- = Types for Stream Consumers
98-- = Types for Stream Consumers
56−-- We use the 'Fold' type to implement the Accumulator and Splitter
100+-- In streamly, there is no separate type for accumulators. Terminating folds
57−-- functionality. Parsers are represented by the
101+-- are a superset of accumulators and to avoid too many types we represent both
102+-- using the same type, 'Fold'.
104+-- We do not club the leftovers functionality with terminating folds because of
105+-- the reasons explained earlier. Instead combinators that require leftovers
58−-- 'Streamly.Internal.Data.Parser.Parser' type. This is a sweet spot to
106+-- are implemented as the 'Streamly.Internal.Data.Parser.Parser' type. This is
59−-- balance ease of use, type safety and performance. Using separate
107+-- a sweet spot to balance ease of use, type safety and performance. Using
60−-- Accumulator and Splitter types would encode more information in types but it
108+-- separate Accumulator and terminating fold types would encode more
61−-- would make ease of use, implementation, maintenance effort worse. Combining
109+-- information in types but it would make ease of use, implementation,
62−-- Accumulator, Splitter and Parser into a single
110+-- maintenance effort worse. Combining Accumulator, terminating folds and
63−-- 'Streamly.Internal.Data.Parser.Parser' type would make ease of use even
111+-- Parser into a single 'Streamly.Internal.Data.Parser.Parser' type would make
64−-- better but type safety and performance worse.
112+-- ease of use even better but type safety and performance worse.
66-- One of the design requirements that we have placed for better ease of use
114-- One of the design requirements that we have placed for better ease of use
67-- and code reuse is that 'Streamly.Internal.Data.Parser.Parser' type should be
115-- and code reuse is that 'Streamly.Internal.Data.Parser.Parser' type should be