@@ -12,45 +12,115 @@
1212-- We can classify stream consumers in the following categories in order of
1313-- increasing complexity and power:
1414--
15−-- == Accumulators
15+-- == Folds that never terminate
1616--
17−-- These are the simplest folds that never fail and never terminate, they
18−-- accumulate the input values forever and can always accept new inputs (never
19−-- terminate) and always have a valid result value. A
20−-- 'Streamly.Internal.Data.Fold.sum' operation is an example of an accumulator.
17+-- An Accumulator is the simplest type of fold, it never fails and never
18+-- terminates. It can always accept more inputs (never terminates) and the
19+-- accumulator is always valid. For example 'Streamly.Internal.Data.Fold.sum'.
2120-- Traditional Haskell left folds like 'foldl' are accumulators.
2221--
23−-- We can distribute an input stream to two or more accumulators using a @tee@
24−-- style composition. Accumulators cannot be applied on a stream one after the
25−-- other, which we call a @serial@ append style composition of folds. This is
26−-- because accumulators never terminate, since the first accumulator in a
27−-- series will never terminate, the next one will never get to run.
22+-- Accumulators can be composed in parallel where we distribute the input
23+-- stream to all accumulators. Since accumulators never terminate they cannot
24+-- be appended.
2825--
29−-- == Terminating Folds
26+-- An accumulator can be represented as:
3027--
31−-- Terminating folds are accumulators that can terminate. Once a fold
32−-- terminates it no longer accepts any more inputs. Terminating folds can be
33−-- used in a @serial@ append style composition where one fold can be applied
34−-- after the other on an input stream. We can apply a terminating fold
35−-- repeatedly on an input stream, splitting the stream and consuming it in
36−-- fragments. Terminating folds never fail, therefore, they do not need
37−-- backtracking.
28+-- @
29+-- data Fold0 m a b =
30+-- forall s. Fold0
31+-- (s -> a -> m s) -- step
32+-- (m s) -- initial
33+-- (s -> m b) -- extract
34+-- @
35+--
36+-- This is just a traditional left fold, compare with @foldl@. The driver of
37+-- the fold would call @initial@ at the beginning and then keep accumulating
38+-- inputs into its result using @step@ and finally extract the result using
39+-- @extract@.
40+--
41+-- == Folds that terminate after one or more input
42+--
43+-- Terminating folds are accumulators that can terminate, like accumulators
44+-- they do not fail. Once a fold terminates it no longer accepts any more
45+-- inputs. Terminating folds can be appended, the next fold can be
46+-- applied after the first one terminates. Because they cannot fail, they do
47+-- not need backtracking.
3848--
3949-- The 'Streamly.Internal.Data.Fold.take' operation is an example of a
40−-- terminating fold It terminates after consuming @n@ items. Coupled with an
41−-- accumulator (e.g. sum) it can be used to split and process the stream into
42−-- chunks of fixed size.
50+-- terminating fold. It terminates after consuming @n@ items. Coupled with an
51+-- accumulator (e.g. sum) it can be used to process the stream into chunks of
52+-- fixed size.
53+--
54+-- A terminating fold can be represented as:
55+--
56+-- @
57+-- data Step s b
58+-- = Partial !s -- the fold can accept more input
59+-- | Done !b -- the fold is done
60+--
61+-- data Fold1 m a b =
62+-- forall s. Fold1
63+-- (s -> a -> m (Step s b)) -- step
64+-- (m s) -- initial
65+-- (s -> m b) -- extract
66+-- @
4367--
44−-- == Terminating Folds with Leftovers
68+-- The fold driver stops driving the fold as soon as the fold returns a @Done@.
69+-- @extract@ is required only if the fold has not stopped yet and the input
70+-- ends. @extract@ can never be called if the fold is @Done@.
4571--
46−-- The next upgrade after terminating folds is terminating folds with leftover
47−-- inputs. Consider the example of @takeWhile@ operation, it needs to inspect
48−-- an element for termination decision. However, it does not consume the
49−-- element on which it terminates. To implement @takeWhile@ a terminating fold
50−-- will have to implement a way to return unconsumed input to the fold driver.
72+-- Notice that the @initial@ of `Fold1` type does not return a "Step" type,
73+-- therefore, it cannot say "Done" in initial. It always has to consume at
74+-- least one element before it can say "Done" for termination, via the @step@
75+-- function.
5176--
52−-- Single element leftover case is the most common and its easy to implement it
53−-- in terminating folds using a @Done1@ constructor in the 'Step' type which
77+-- == Folds that terminate after 0 or more input
78+--
79+-- The `Fold1` type makes combinators like @take 0@ impossible to implement
80+-- because they need to terminate even before they can consume any elements at
81+-- all. Implementing this requires the @initial@ function to be able to return
82+-- @Done@.
83+--
84+-- @
85+-- data Fold m a b =
86+-- forall s. Fold
87+-- (s -> a -> m (Step s b)) -- step
88+-- (m (Step s b)) -- initial
89+-- (s -> m b) -- extract
90+-- @
91+--
92+-- This is also required if we want to compose terminating folds using an
93+-- Applicative or Monadic composition. @pure@ needs to yield an output without
94+-- having to consume an input.
95+--
96+-- @initial@ now has the ability to terminate the fold without consuming any
97+-- input based on the state of the monad.
98+--
99+-- In some cases it does not make sense to use a fold that does not consume any
100+-- items at all, and it may even lead to an infinite loop. It might make sense
101+-- to use a `Fold1` type for such cases because it guarantees to consume at
102+-- least one input, therefore, guarantees progress. For example, in
103+-- classifySessionsBy or any other splitting operations it may not make sense
104+-- to pass a fold that never consumes an input. However, we do not have a
105+-- separate Fold1 type for the sake of simplicity of types/API.
106+--
107+-- Adding this capability adds a certain amount of complexity in the
108+-- implementation of fold combinators. @initial@ has to always handle two cases
109+-- now. We could potentially not implement this in folds to keep fold
110+-- implementation simpler, and these use cases can be transferred to the parser
111+-- type. However, it would be a bit inconvenient to not have a `take` operation
112+-- or to not be able to use `take 0` if we have it. Also, applicative and
113+-- monadic composition of folds would not be possible.
114+--
115+-- == Terminating Folds with backtracking
116+--
117+-- Consider the example of @takeWhile@ operation, it needs to inspect an
118+-- element for termination decision. However, it does not consume the element
119+-- on which it terminates. To implement @takeWhile@ a terminating fold will
120+-- have to implement a way to return the unconsumed input to the fold driver.
121+--
122+-- Single element leftover case is quite common and its easy to implement it in
123+-- terminating folds by adding a @Done1@ constructor in the 'Step' type which
54124-- indicates that the last element was not consumed by the fold. The following
55125-- additional operations can be implemented as terminating folds if we do that.
56126--
@@ -60,30 +130,74 @@
60130-- wordBy
61131-- @
62132--
63−-- However, it creates several complications. The 'many' combinator requires
64−-- a @Partial1@ ('Partial' with leftover) to handle a @Done1@ from the top
65−-- level fold, for efficient implementation. If the collecting fold in "many"
66−-- returns a @Partial1@ or @Done1@ then what to do with all the elements that
67−-- have been consumed?
68−--
69−-- Similarly, in distribute, if one fold consumes a value and others say its a
70−-- leftover then what do we do? Folds like "many" require the leftover to be
71−-- fed to it again. So in a distribute operation those folds which gave a
72−-- leftover will have to be fed the leftover while the folds that consumed will
73−-- have to be fed the next input. This is very complicated to implement. We
74−-- have the same issue in backtracking parsers being used in a distribute
75−-- operation.
76−--
77−-- To avoid these issues we want to enforce by typing that the collecting folds
78−-- can never return a leftover. So we need a fold type without @Done1@ or
79−-- @Partial1@. This leads us to design folds to never return a leftover and the
80−-- use cases of single leftover are transferred to parsers where we have
81−-- general backtracking mechanism and single leftover is just a special case of
133+-- However, it creates several complications.
134+--
135+-- === Nested backtracking
136+--
137+-- Nesting of backtracking folds increases the amount of backtracking required
138+-- exponentially.
139+--
140+-- For example, the combinator @many inner outer@ applies the outer fold on the
141+-- input stream and applies the inner fold on the results of the outer fold.
142+--
143+-- many :: Monad m => Fold m b c -> Fold m a b -> Fold m a c
144+--
145+-- If the inner fold itself returns a @Done1@ then we need to backtrack all
146+-- the elements that have been consumed by the outer fold to generate that
147+-- value. We need backtracking of more than one element.
148+--
149+-- Arbitrary backtracking requires arbitrary buffering. However, we do not want
150+-- to buffer unconditionally, only if the buffer is needed. One way to do this
151+-- is to use a "Continue" constructor like parsers. When we have nested folds,
152+-- the top level fold always returns a "Continue" to the driver until an output
153+-- is generated by it, this means the top level driver keeps buffering until an
154+-- output is generated via Partial or Done. Intermediate level "Continue" keep
155+-- propagating up to the top level.
156+--
157+-- === Parallel backtracking
158+--
159+-- In compositions like Alternative and Distributive we may have several
160+-- branches. Each branch can backtrack independently. We need to keep the input
161+-- as long as any of the branches need it. We can use a single copy of the
162+-- buffer and maintain it based on all the branches, or we can make each branch
163+-- have its own buffer. The latter approach may be simpler to implement.
164+-- Whenever we branch we can introduce an independent buffer for backtracking.
165+-- Or we can use a newtype that allows branched composition to handle
82166-- backtracking.
83167--
168+-- === Implementation Approach
169+--
170+-- To avoid these issues we can enforce, by using types, that the collecting
171+-- folds can never return a leftover. This leads us to define a type that can
172+-- never return a leftover. The use cases of single leftover can be transferred
173+-- to parsers where we have general backtracking mechanism and single leftover
174+-- is just a special case of backtracking.
175+--
84176-- This means: takeWhile, groupBy, wordBy would be implemented as parsers.
85−-- "take 0" can implemented as a fold if we make initial return @Step@ type.
86−-- "takeInterval" can be implemented without @Done1@.
177+--
178+-- A proposed design is to use the same Step type with Error in Folds as well
179+-- as Parsers. Folds won't use the Error constructor and even if they use, it
180+-- will be equivalent to just throwing an error. They won't have an
181+-- alternative.
182+--
183+-- Because of the complexity of implementing a distributive composition in
184+-- presence of backtracking we could possibly have a type without backtracking
185+-- but with the "Continue" constructor, and use either the Parser type or
186+-- another type for backtracking.
187+--
188+-- == Folds with an additional input
189+--
190+-- The `Fold` type does not allow a dynamic input to be used to generate the
191+-- initial value of the fold accumulator. We can extend the type further to
192+-- allow that:
193+--
194+-- @
195+-- data Refold m i a b =
196+-- forall s. Refold
197+-- (s -> a -> m (Step s b)) -- step
198+-- (i -> m (Step s b)) -- initial
199+-- (s -> m b) -- extract
200+-- @
87201--
88202-- == Parsers
89203--
@@ -95,11 +209,33 @@
95209-- would succeed if the condition is satisfied and it would fail otherwise, on
96210-- failure an alternative parser can be used on the same input.
97211--
212+-- We add @Error@ and @Continue@ to the @Step@ type of fold. @Continue@ is to
213+-- skip producing an output or to backtrack. We also add the ability to
214+-- backtrack in @Partial@ and @Done@.:
215+--
216+-- Also @extract@ now needs to be able to express an error. We could have it
217+-- return the @Step@ type as well but that makes the implementation more
218+-- complicated.
219+--
220+-- @
221+-- data Step s b =
222+-- Partial Int s -- partial result and how much to backtrack
223+-- | Done Int b -- final result and how much to backtrack
224+-- | Continue Int s -- no result and how much to backtrack
225+-- | Error String -- error
226+--
227+-- data Parser m a b =
228+-- forall s. Fold
229+-- (s -> a -> m (Step s b)) -- step
230+-- (m (Step s b)) -- initial
231+-- (s -> m (Either String b)) -- extract
232+-- @
233+--
98234-- = Types for Stream Consumers
99235--
100−-- In streamly, there is no separate type for accumulators. Terminating folds
101−-- are a superset of accumulators and to avoid too many types we represent both
102−-- using the same type, 'Fold'.
236+-- We do not have a separate type for accumulators. Terminating folds are a
237+-- superset of accumulators and to avoid too many types we represent both using
238+-- the same type, 'Fold'.
103239--
104240-- We do not club the leftovers functionality with terminating folds because of
105241-- the reasons explained earlier. Instead combinators that require leftovers
@@ -155,9 +291,9 @@
155291--
156292-- However, removing extract altogether may lead to less optimal code in some
157293-- cases because the driver of the fold needs to thread around the intermediate
158−-- output to return it if the stream stops before the fold could @Done@. When
159−-- using this approach, the @parseMany (FL.take filesize)@ benchmark shows a
160−-- 2x worse performance even after ensuring everything fuses. So we keep the
294+-- output to return it if the stream stops before the fold could return @Done@.
295+-- When using this approach, the @parseMany (FL.take filesize)@ benchmark shows
296+-- a 2x worse performance even after ensuring everything fuses. So we keep the
161297-- "extract" approach to ensure better perf in all cases.
162298--
163299-- But we could still yield both state and the output in @Partial@, the output
@@ -165,14 +301,6 @@
165301-- then be used only for the case when the stream stops before the fold
166302-- completes.
167303--
168−-- = Accumulators and Terminating Folds
169−--
170−-- Folds in this module can be classified in two categories viz. accumulators
171−-- and terminating folds. Accumulators do not have a terminating condition,
172−-- they run forever and consume the entire stream, for example the 'length'
173−-- fold. Terminating folds have a terminating condition and can terminate
174−-- without consuming the entire stream, for example, the 'head' fold.
175−--
176304-- = Monoids
177305--
178306-- Monoids allow generalized, modular folding. The accumulators in this module