@@ -12,45 +12,115 @@
12-- We can classify stream consumers in the following categories in order of
12-- We can classify stream consumers in the following categories in order of
13-- increasing complexity and power:
13-- increasing complexity and power:
15−-- == Accumulators
15+-- == Folds that never terminate
17−-- These are the simplest folds that never fail and never terminate, they
17+-- An Accumulator is the simplest type of fold, it never fails and never
18−-- accumulate the input values forever and can always accept new inputs (never
18+-- terminates. It can always accept more inputs (never terminates) and the
19−-- terminate) and always have a valid result value. A
20−-- 'Streamly.Internal.Data.Fold.sum' operation is an example of an accumulator.
19+-- accumulator is always valid. For example 'Streamly.Internal.Data.Fold.sum'.
21-- Traditional Haskell left folds like 'foldl' are accumulators.
20-- Traditional Haskell left folds like 'foldl' are accumulators.
23−-- We can distribute an input stream to two or more accumulators using a @tee@
22+-- Accumulators can be composed in parallel where we distribute the input
24−-- style composition. Accumulators cannot be applied on a stream one after the
23+-- stream to all accumulators. Since accumulators never terminate they cannot
25−-- other, which we call a @serial@ append style composition of folds. This is
24+-- be appended.
26−-- because accumulators never terminate, since the first accumulator in a
27−-- series will never terminate, the next one will never get to run.
29−-- == Terminating Folds
26+-- An accumulator can be represented as:
31+-- (s -> a -> m s) -- step
33+-- (s -> m b) -- extract
36+-- This is just a traditional left fold, compare with @foldl@. The driver of
37+-- the fold would call @initial@ at the beginning and then keep accumulating
38+-- inputs into its result using @step@ and finally extract the result using
41+-- == Folds that terminate after one or more input
31−-- Terminating folds are accumulators that can terminate. Once a fold
43+-- Terminating folds are accumulators that can terminate, like accumulators
32−-- terminates it no longer accepts any more inputs. Terminating folds can be
44+-- they do not fail. Once a fold terminates it no longer accepts any more
33−-- used in a @serial@ append style composition where one fold can be applied
45+-- inputs. Terminating folds can be appended, the next fold can be
34−-- after the other on an input stream. We can apply a terminating fold
46+-- applied after the first one terminates. Because they cannot fail, they do
35−-- repeatedly on an input stream, splitting the stream and consuming it in
36−-- fragments. Terminating folds never fail, therefore, they do not need
37−-- backtracking.
47+-- not need backtracking.
39-- The 'Streamly.Internal.Data.Fold.take' operation is an example of a
49-- The 'Streamly.Internal.Data.Fold.take' operation is an example of a
40−-- terminating fold It terminates after consuming @n@ items. Coupled with an
50+-- terminating fold. It terminates after consuming @n@ items. Coupled with an
41−-- accumulator (e.g. sum) it can be used to split and process the stream into
51+-- accumulator (e.g. sum) it can be used to process the stream into chunks of
42−-- chunks of fixed size.
52+-- fixed size.
54+-- A terminating fold can be represented as:
58+-- = Partial !s -- the fold can accept more input
59+-- | Done !b -- the fold is done
63+-- (s -> a -> m (Step s b)) -- step
65+-- (s -> m b) -- extract
44−-- == Terminating Folds with Leftovers
68+-- The fold driver stops driving the fold as soon as the fold returns a @Done@.
69+-- @extract@ is required only if the fold has not stopped yet and the input
70+-- ends. @extract@ can never be called if the fold is @Done@.
46−-- The next upgrade after terminating folds is terminating folds with leftover
72+-- Notice that the @initial@ of `Fold1` type does not return a "Step" type,
47−-- inputs. Consider the example of @takeWhile@ operation, it needs to inspect
73+-- therefore, it cannot say "Done" in initial. It always has to consume at
48−-- an element for termination decision. However, it does not consume the
74+-- least one element before it can say "Done" for termination, via the @step@
49−-- element on which it terminates. To implement @takeWhile@ a terminating fold
75+-- function.
50−-- will have to implement a way to return unconsumed input to the fold driver.
77+-- == Folds that terminate after 0 or more input
79+-- The `Fold1` type makes combinators like @take 0@ impossible to implement
80+-- because they need to terminate even before they can consume any elements at
81+-- all. Implementing this requires the @initial@ function to be able to return
87+-- (s -> a -> m (Step s b)) -- step
88+-- (m (Step s b)) -- initial
89+-- (s -> m b) -- extract
92+-- This is also required if we want to compose terminating folds using an
93+-- Applicative or Monadic composition. @pure@ needs to yield an output without
94+-- having to consume an input.
96+-- @initial@ now has the ability to terminate the fold without consuming any
97+-- input based on the state of the monad.
99+-- In some cases it does not make sense to use a fold that does not consume any
100+-- items at all, and it may even lead to an infinite loop. It might make sense
101+-- to use a `Fold1` type for such cases because it guarantees to consume at
102+-- least one input, therefore, guarantees progress. For example, in
103+-- classifySessionsBy or any other splitting operations it may not make sense
104+-- to pass a fold that never consumes an input. However, we do not have a
105+-- separate Fold1 type for the sake of simplicity of types/API.
107+-- Adding this capability adds a certain amount of complexity in the
108+-- implementation of fold combinators. @initial@ has to always handle two cases
109+-- now. We could potentially not implement this in folds to keep fold
110+-- implementation simpler, and these use cases can be transferred to the parser
111+-- type. However, it would be a bit inconvenient to not have a `take` operation
112+-- or to not be able to use `take 0` if we have it. Also, applicative and
113+-- monadic composition of folds would not be possible.
115+-- == Terminating Folds with backtracking
117+-- Consider the example of @takeWhile@ operation, it needs to inspect an
118+-- element for termination decision. However, it does not consume the element
119+-- on which it terminates. To implement @takeWhile@ a terminating fold will
120+-- have to implement a way to return the unconsumed input to the fold driver.
52−-- Single element leftover case is the most common and its easy to implement it
122+-- Single element leftover case is quite common and its easy to implement it in
53−-- in terminating folds using a @Done1@ constructor in the 'Step' type which
123+-- terminating folds by adding a @Done1@ constructor in the 'Step' type which
54-- indicates that the last element was not consumed by the fold. The following
124-- indicates that the last element was not consumed by the fold. The following
55-- additional operations can be implemented as terminating folds if we do that.
125-- additional operations can be implemented as terminating folds if we do that.
@@ -60,30 +130,74 @@
63−-- However, it creates several complications. The 'many' combinator requires
133+-- However, it creates several complications.
64−-- a @Partial1@ ('Partial' with leftover) to handle a @Done1@ from the top
134+--
65−-- level fold, for efficient implementation. If the collecting fold in "many"
135+-- === Nested backtracking
66−-- returns a @Partial1@ or @Done1@ then what to do with all the elements that
136+--
137+-- Nesting of backtracking folds increases the amount of backtracking required
140+-- For example, the combinator @many inner outer@ applies the outer fold on the
141+-- input stream and applies the inner fold on the results of the outer fold.
143+-- many :: Monad m => Fold m b c -> Fold m a b -> Fold m a c
145+-- If the inner fold itself returns a @Done1@ then we need to backtrack all
67−-- have been consumed?
146+-- the elements that have been consumed by the outer fold to generate that
68−--
147+-- value. We need backtracking of more than one element.
69−-- Similarly, in distribute, if one fold consumes a value and others say its a
148+--
70−-- leftover then what do we do? Folds like "many" require the leftover to be
149+-- Arbitrary backtracking requires arbitrary buffering. However, we do not want
71−-- fed to it again. So in a distribute operation those folds which gave a
150+-- to buffer unconditionally, only if the buffer is needed. One way to do this
72−-- leftover will have to be fed the leftover while the folds that consumed will
151+-- is to use a "Continue" constructor like parsers. When we have nested folds,
73−-- have to be fed the next input. This is very complicated to implement. We
152+-- the top level fold always returns a "Continue" to the driver until an output
74−-- have the same issue in backtracking parsers being used in a distribute
153+-- is generated by it, this means the top level driver keeps buffering until an
75−-- operation.
154+-- output is generated via Partial or Done. Intermediate level "Continue" keep
76−--
155+-- propagating up to the top level.
77−-- To avoid these issues we want to enforce by typing that the collecting folds
156+--
78−-- can never return a leftover. So we need a fold type without @Done1@ or
157+-- === Parallel backtracking
79−-- @Partial1@. This leads us to design folds to never return a leftover and the
158+--
80−-- use cases of single leftover are transferred to parsers where we have
159+-- In compositions like Alternative and Distributive we may have several
81−-- general backtracking mechanism and single leftover is just a special case of
160+-- branches. Each branch can backtrack independently. We need to keep the input
161+-- as long as any of the branches need it. We can use a single copy of the
162+-- buffer and maintain it based on all the branches, or we can make each branch
163+-- have its own buffer. The latter approach may be simpler to implement.
164+-- Whenever we branch we can introduce an independent buffer for backtracking.
165+-- Or we can use a newtype that allows branched composition to handle
82-- backtracking.
166-- backtracking.
168+-- === Implementation Approach
170+-- To avoid these issues we can enforce, by using types, that the collecting
171+-- folds can never return a leftover. This leads us to define a type that can
172+-- never return a leftover. The use cases of single leftover can be transferred
173+-- to parsers where we have general backtracking mechanism and single leftover
174+-- is just a special case of backtracking.
84-- This means: takeWhile, groupBy, wordBy would be implemented as parsers.
176-- This means: takeWhile, groupBy, wordBy would be implemented as parsers.
85−-- "take 0" can implemented as a fold if we make initial return @Step@ type.
177+--
86−-- "takeInterval" can be implemented without @Done1@.
178+-- A proposed design is to use the same Step type with Error in Folds as well
179+-- as Parsers. Folds won't use the Error constructor and even if they use, it
180+-- will be equivalent to just throwing an error. They won't have an
183+-- Because of the complexity of implementing a distributive composition in
184+-- presence of backtracking we could possibly have a type without backtracking
185+-- but with the "Continue" constructor, and use either the Parser type or
186+-- another type for backtracking.
188+-- == Folds with an additional input
190+-- The `Fold` type does not allow a dynamic input to be used to generate the
191+-- initial value of the fold accumulator. We can extend the type further to
195+-- data Refold m i a b =
197+-- (s -> a -> m (Step s b)) -- step
198+-- (i -> m (Step s b)) -- initial
199+-- (s -> m b) -- extract
88-- == Parsers
202-- == Parsers
@@ -95,11 +209,33 @@
95-- would succeed if the condition is satisfied and it would fail otherwise, on
209-- would succeed if the condition is satisfied and it would fail otherwise, on
96-- failure an alternative parser can be used on the same input.
210-- failure an alternative parser can be used on the same input.
212+-- We add @Error@ and @Continue@ to the @Step@ type of fold. @Continue@ is to
213+-- skip producing an output or to backtrack. We also add the ability to
214+-- backtrack in @Partial@ and @Done@.:
216+-- Also @extract@ now needs to be able to express an error. We could have it
217+-- return the @Step@ type as well but that makes the implementation more
222+-- Partial Int s -- partial result and how much to backtrack
223+-- | Done Int b -- final result and how much to backtrack
224+-- | Continue Int s -- no result and how much to backtrack
225+-- | Error String -- error
227+-- data Parser m a b =
229+-- (s -> a -> m (Step s b)) -- step
230+-- (m (Step s b)) -- initial
231+-- (s -> m (Either String b)) -- extract
98-- = Types for Stream Consumers
234-- = Types for Stream Consumers
100−-- In streamly, there is no separate type for accumulators. Terminating folds
236+-- We do not have a separate type for accumulators. Terminating folds are a
101−-- are a superset of accumulators and to avoid too many types we represent both
237+-- superset of accumulators and to avoid too many types we represent both using
102−-- using the same type, 'Fold'.
238+-- the same type, 'Fold'.
104-- We do not club the leftovers functionality with terminating folds because of
240-- We do not club the leftovers functionality with terminating folds because of
105-- the reasons explained earlier. Instead combinators that require leftovers
241-- the reasons explained earlier. Instead combinators that require leftovers
@@ -155,9 +291,9 @@
156-- However, removing extract altogether may lead to less optimal code in some
292-- However, removing extract altogether may lead to less optimal code in some
157-- cases because the driver of the fold needs to thread around the intermediate
293-- cases because the driver of the fold needs to thread around the intermediate
158−-- output to return it if the stream stops before the fold could @Done@. When
294+-- output to return it if the stream stops before the fold could return @Done@.
159−-- using this approach, the @parseMany (FL.take filesize)@ benchmark shows a
295+-- When using this approach, the @parseMany (FL.take filesize)@ benchmark shows
160−-- 2x worse performance even after ensuring everything fuses. So we keep the
296+-- a 2x worse performance even after ensuring everything fuses. So we keep the
161-- "extract" approach to ensure better perf in all cases.
297-- "extract" approach to ensure better perf in all cases.
163-- But we could still yield both state and the output in @Partial@, the output
299-- But we could still yield both state and the output in @Partial@, the output
@@ -165,14 +301,6 @@
165-- then be used only for the case when the stream stops before the fold
301-- then be used only for the case when the stream stops before the fold
166-- completes.
302-- completes.
168−-- = Accumulators and Terminating Folds
170−-- Folds in this module can be classified in two categories viz. accumulators
171−-- and terminating folds. Accumulators do not have a terminating condition,
172−-- they run forever and consume the entire stream, for example the 'length'
173−-- fold. Terminating folds have a terminating condition and can terminate
174−-- without consuming the entire stream, for example, the 'head' fold.
176-- = Monoids
304-- = Monoids
178-- Monoids allow generalized, modular folding. The accumulators in this module
306-- Monoids allow generalized, modular folding. The accumulators in this module