@@ -40,11 +40,11 @@
4040-- 'Step' functor provides the following commands to the fold driver
4141-- corresponding to the use cases outlined in the previous para:
4242--
43−-- 1. 'Skip': buffer the current input and go back to a previous
43+-- 1. 'Continue': buffer the current input and go back to a previous
4444-- position in the stream
45−-- 2. 'Yield': buffer the current input and go back to a previous
45+-- 2. 'Partial': buffer the current input and go back to a previous
4646-- position in the stream, drop the buffer before that position.
47−-- 3. 'Stop': parser succeeded, returns how much input was leftover
47+-- 3. 'Done': parser succeeded, returns how much input was leftover
4848-- 4. 'Error': indicates that the parser has failed without a result
4949--
5050-- = How a Parser Works?
@@ -55,19 +55,19 @@
5555--
5656-- The parser may build a new output value from multiple input items. When it
5757-- consumes an input item but needs more input to build a complete output item
58−-- it uses @Skip 0 s@, yielding the intermediate state @s@ and asking the
58+-- it uses @Continue 0 s@, yielding the intermediate state @s@ and asking the
5959-- driver to provide more input. When the parser determines that a new output
60−-- value is complete it can use a @Stop n b@ to terminate the parser with @n@
60+-- value is complete it can use a @Done n b@ to terminate the parser with @n@
6161-- items of input unused and the final value of the accumulator returned as
6262-- @b@. If at any time the parser determines that the parse has failed it can
6363-- return @Error err@.
6464--
65−-- A parser building a collection of values (e.g. a list) can use the @Yield@
65+-- A parser building a collection of values (e.g. a list) can use the @Partial@
6666-- constructor whenever a new item in the output collection is generated. If a
6767-- parser building a collection of values has yielded at least one value then
6868-- it considered successful and cannot fail after that. In the current
6969-- implementation, this is not automatically enforced, there is a rule that the
70−-- parser MUST use only @Stop@ for termination after the first @Yield@, it
70+-- parser MUST use only @Done@ for termination after the first @Partial@, it
7171-- cannot use @Error@. It may be possible to change the implementation so that
7272-- this rule is not required, but there may be some performance cost to it.
7373--
@@ -82,7 +82,7 @@
8282-- = Error Handling
8383--
8484-- When a parser's @step@ function is invoked it may terminate by either a
85−-- 'Stop' or an 'Error' return value. In an 'Alternative' composition an error
85+-- 'Done' or an 'Error' return value. In an 'Alternative' composition an error
8686-- return can make the composed parser backtrack and try another parser.
8787--
8888-- If the stream stops before a parser could terminate then we use the
@@ -145,58 +145,70 @@ import Streamly.Internal.Data.Strict (Tuple3'(..))
145145
146146-- | The return type of a 'Parser' step.
147147--
148−-- A parser is driven by a parse driver one step at a time, at any time the
149−-- driver may @extract@ the result of the parser. The parser may ask the driver
150−-- to backtrack at any point, therefore, the driver holds the input up to a
151−-- point of no return in a backtracking buffer. The buffer grows or shrinks
152−-- based on the return values of the parser step execution.
153−--
154−-- When a parser step is executed it generates a new intermediate state of the
155−-- parse result along with a command to the driver. The command tells the
156−-- driver whether to keep the input stream for a potential backtracking later
157−-- on or drop it, and how much to keep. The constructors of 'Step' represent
158−-- the commands to the driver.
148+-- The parse operation feeds the input stream to the parser one element at a
149+-- time, representing a parse 'Step'. The parser may or may not consume the
150+-- item and returns a result. If the result is 'Partial' we can either extract
151+-- the result or feed more input to the parser. If the result is 'Continue', we
152+-- must feed more input in order to get a result. If the parser returns 'Done'
153+-- then the parser can no longer take any more input.
154+--
155+-- If the result is 'Continue', the parse operation retains the input in a
156+-- backtracking buffer, in case the parser may ask to backtrack in future.
157+-- Whenever a 'Partial n' result is returned we first backtrack by @n@ elements
158+-- in the input and then release any remaining backtracking buffer. Similarly,
159+-- 'Continue n' backtracks to @n@ elements before the current position and
160+-- starts feeding the input from that point for future invocations of the
161+-- parser.
162+--
163+-- If parser is not yet done, we can use the @extract@ operation on the @state@
164+-- of the parser to extract a result. If the parser has not yet yielded a
165+-- result, the operation fails with a 'ParseError' exception. If the parser
166+-- yielded a 'Partial' result in the past the last partial result is returned.
167+-- Therefore, if a parser yields a partial result once it cannot fail later on.
168+--
169+-- The parser can never backtrack beyond the position where the last partial
170+-- result left it at. The parser must ensure that the backtrack position is
171+-- always after that.
159172--
160173-- /Internal/
161174--
162175{-# ANN type Step Fuse #-}
163176data Step s b =
164− Yield Int s
165− -- ^ Go back by @n@ items in the input buffer and drop the rest.
166− -- @Yield n state@ indicates that the parser has yielded a new result
167− -- which is a point of no return. The result can be extracted using
168− -- @extract@. The driver moves back the current position of the cursor by
169− -- @n@ elements and drops any buffer before that. The rule is that if a
170− -- parser has yielded at least once it cannot return a failure result.
171−
172− -- only rewind
173− | Skip Int s
174− -- ^ Go back by @n@ items in the input buffer.
175− -- @Skip n state@ indicates that the parser has consumed the current input
176− -- but no new result has been generated. A new @state@ is generated.
177− -- However, if we use @extract@ on @state@ it will generate a result from
178− -- the previous @Yield@. When @n@ is non-zero it is a backward offset from
179− -- the current position in the stream from which the driver will replay the
180− -- input to the parser. The offset cannot be beyond the latest point of no
181− -- return created by @Yield@.
182−
183− | Stop Int b
184− -- ^ Trim the backtracking buffer keeping only most recent @n@ items.
185− -- @Stop n result@ asks the driver to stop driving the parser because it
186− -- has reached a fixed point and further input will not change the result.
187− -- @n@ is the count of unused input elements which includes the element on
188− -- which 'Stop' occurred.
177+ Partial Int s
178+ -- ^ Partial result with an optional backtrack.
179+ --
180+ -- @Partial count state@ means a partial result is available which
181+ -- can be extracted successfully, @state@ is the opaque state of the
182+ -- parser to be supplied to the next invocation of the step operation.
183+ -- The current input position is reset to @count@ elements back and any
184+ -- input before that is dropped from the backtrack buffer.
185+
186+ | Continue Int s
187+ -- ^ Need more input with an optional backtrack.
188+ --
189+ -- @Continue count state@ means the parser has consumed the current input
190+ -- but no new result is generated, @state@ is the next state of the parser.
191+ -- The current input is retained in the backtrack buffer and the input
192+ -- position is reset to @count@ elements back.
193+
194+ | Done Int b
195+ -- ^ Done with leftover input count and result.
196+ --
197+ -- @Done count result@ means the parser has finished, it will accept no
198+ -- more input, last @count@ elements from the input are unused and the
199+ -- result of the parser is in @result@.
189200
190201 | Error String
191− -- ^ Go back to the start of the backtracking buffer.
192− -- An error makes the parser backtrack to the last checkpoint and try
193− -- another alternative.
202+ -- ^ Parser failed without generating any output.
203+ --
204+ -- The parsing operation may backtrack to the beginning and try another
205+ -- alternative.
194206
195207instance Functor (Step s) where
196208 {-# INLINE fmap #-}
197− fmap _ (Yield n s) = Yield n s
198− fmap _ (Skip n s) = Skip n s
199− fmap f (Stop n b) = Stop n (f b)
209+ fmap _ (Partial n s) = Partial n s
210+ fmap _ (Continue n s) = Continue n s
211+ fmap f (Done n b) = Done n (f b)
200212 fmap _ (Error err) = Error err
201213
202214-- | A parser is a fold that can fail and is represented as @Parser step
@@ -243,7 +255,7 @@ instance Functor m => Functor (Parser m a) where
243255--
244256{-# INLINE_NORMAL yield #-}
245257yield :: Monad m => b -> Parser m a b
246−yield b = Parser (\_ _ -> pure $ Stop 1 b) -- step
258+yield b = Parser (\_ _ -> pure $ Done 1 b) -- step
247259 (pure ()) -- initial
248260 (\_ -> pure b) -- extract
249261
@@ -253,7 +265,7 @@ yield b = Parser (\_ _ -> pure $ Stop 1 b) -- step
253265--
254266{-# INLINE yieldM #-}
255267yieldM :: Monad m => m b -> Parser m a b
256−yieldM b = Parser (\_ _ -> Stop 1 <$> b) -- step
268+yieldM b = Parser (\_ _ -> Done 1 <$> b) -- step
257269 (pure ()) -- initial
258270 (\_ -> b) -- extract
259271
@@ -287,7 +299,7 @@ splitWith func (Parser stepL initialL extractL)
287299 initial = SeqParseL <$> initialL
288300
289301 -- Note: For the composed parse to terminate, the left parser has to be
290− -- a terminating parser returning a Stop at some point.
302+ -- a terminating parser returning a Done at some point.
291303 step (SeqParseL st) a = do
292304 r <- stepL st a
293305 case r of
@@ -295,17 +307,17 @@ splitWith func (Parser stepL initialL extractL)
295307 -- e.g. in ((,) <$> p1 <*> p2) <|> p3, if p2 fails we have to
296308 -- backtrack and start running p3. So we need to keep the input
297309 -- buffered until we know that the applicative cannot fail.
298− Yield n s -> return $ Skip n (SeqParseL s)
299− Skip n s -> return $ Skip n (SeqParseL s)
300− Stop n b -> Skip n <$> (SeqParseR (func b) <$> initialR)
310+ Partial n s -> return $ Continue n (SeqParseL s)
311+ Continue n s -> return $ Continue n (SeqParseL s)
312+ Done n b -> Continue n <$> (SeqParseR (func b) <$> initialR)
301313 Error err -> return $ Error err
302314
303315 step (SeqParseR f st) a = do
304316 r <- stepR st a
305317 return $ case r of
306− Yield n s -> Yield n (SeqParseR f s)
307− Skip n s -> Skip n (SeqParseR f s)
308− Stop n b -> Stop n (f b)
318+ Partial n s -> Partial n (SeqParseR f s)
319+ Continue n s -> Continue n (SeqParseR f s)
320+ Done n b -> Done n (f b)
309321 Error err -> Error err
310322
311323 extract (SeqParseR f sR) = fmap f (extractR sR)
@@ -333,22 +345,22 @@ split_ (Parser stepL initialL extractL) (Parser stepR initialR extractR) =
333345 initial = SeqAL <$> initialL
334346
335347 -- Note: For the composed parse to terminate, the left parser has to be
336− -- a terminating parser returning a Stop at some point.
348+ -- a terminating parser returning a Done at some point.
337349 step (SeqAL st) a = do
338350 (\r i -> case r of
339351 -- Note: this leads to buffering even if we are not in an
340352 -- Alternative composition.
341− Yield n s -> Skip n (SeqAL s)
342− Skip n s -> Skip n (SeqAL s)
343− Stop n _ -> Skip n (SeqAR i)
353+ Partial n s -> Continue n (SeqAL s)
354+ Continue n s -> Continue n (SeqAL s)
355+ Done n _ -> Continue n (SeqAR i)
344356 -- XXX should we sequence initialR monadically?
345357 Error err -> Error err) <$> stepL st a <*> initialR
346358
347359 step (SeqAR st) a = do
348360 (\r -> case r of
349− Yield n s -> Yield n (SeqAR s)
350− Skip n s -> Skip n (SeqAR s)
351− Stop n b -> Stop n b
361+ Partial n s -> Partial n (SeqAR s)
362+ Continue n s -> Continue n (SeqAR s)
363+ Done n b -> Done n b
352364 Error err -> Error err) <$> stepR st a
353365
354366 extract (SeqAR sR) = extractR sR
@@ -400,21 +412,21 @@ alt (Parser stepL initialL extractL) (Parser stepR initialR extractR) =
400412 step (AltParseL cnt st) a = do
401413 r <- stepL st a
402414 case r of
403− Yield n s -> return $ Yield n (AltParseL 0 s)
404− Skip n s -> do
415+ Partial n s -> return $ Partial n (AltParseL 0 s)
416+ Continue n s -> do
405417 assert (cnt + 1 - n >= 0) (return ())
406− return $ Skip n (AltParseL (cnt + 1 - n) s)
407− Stop n b -> return $ Stop n b
418+ return $ Continue n (AltParseL (cnt + 1 - n) s)
419+ Done n b -> return $ Done n b
408420 Error _ -> do
409421 rR <- initialR
410− return $ Skip (cnt + 1) (AltParseR rR)
422+ return $ Continue (cnt + 1) (AltParseR rR)
411423
412424 step (AltParseR st) a = do
413425 r <- stepR st a
414426 return $ case r of
415− Yield n s -> Yield n (AltParseR s)
416− Skip n s -> Skip n (AltParseR s)
417− Stop n b -> Stop n b
427+ Partial n s -> Partial n (AltParseR s)
428+ Continue n s -> Continue n (AltParseR s)
429+ Done n b -> Done n b
418430 Error err -> Error err
419431
420432 extract (AltParseR sR) = extractR sR
@@ -441,19 +453,19 @@ splitMany (Fold fstep finitial fextract) (Parser step1 initial1 extract1) =
441453 r <- step1 st a
442454 let cnt1 = cnt + 1
443455 case r of
444− Yield n s -> do
456+ Partial n s -> do
445457 assert (cnt1 - n >= 0) (return ())
446− return $ Skip n (Tuple3' s (cnt1 - n) fs)
447− Skip n s -> do
458+ return $ Continue n (Tuple3' s (cnt1 - n) fs)
459+ Continue n s -> do
448460 assert (cnt1 - n >= 0) (return ())
449− return $ Skip n (Tuple3' s (cnt1 - n) fs)
450− Stop n b -> do
461+ return $ Continue n (Tuple3' s (cnt1 - n) fs)
462+ Done n b -> do
451463 s <- initial1
452464 fs1 <- fstep fs b
453− return $ Yield n (Tuple3' s 0 fs1)
465+ return $ Partial n (Tuple3' s 0 fs1)
454466 Error _ -> do
455467 xs <- fextract fs
456− return $ Stop cnt1 xs
468+ return $ Done cnt1 xs
457469
458470 -- XXX The "try" may impact performance if this parser is used as a scan
459471 extract (Tuple3' s _ fs) = do
@@ -482,28 +494,28 @@ splitSome (Fold fstep finitial fextract) (Parser step1 initial1 extract1) =
482494 step (Tuple3' st _ (Left fs)) a = do
483495 r <- step1 st a
484496 case r of
485− Yield n s -> return $ Skip n (Tuple3' s undefined (Left fs))
486− Skip n s -> return $ Skip n (Tuple3' s undefined (Left fs))
487− Stop n b -> do
497+ Partial n s -> return $ Continue n (Tuple3' s undefined (Left fs))
498+ Continue n s -> return $ Continue n (Tuple3' s undefined (Left fs))
499+ Done n b -> do
488500 s <- initial1
489501 fs1 <- fstep fs b
490− return $ Yield n (Tuple3' s 0 (Right fs1))
502+ return $ Partial n (Tuple3' s 0 (Right fs1))
491503 Error err -> return $ Error err
492504 step (Tuple3' st cnt (Right fs)) a = do
493505 r <- step1 st a
494506 let cnt1 = cnt + 1
495507 case r of
496− Yield n s -> do
508+ Partial n s -> do
497509 assert (cnt1 - n >= 0) (return ())
498− return $ Yield n (Tuple3' s (cnt1 - n) (Right fs))
499− Skip n s -> do
510+ return $ Partial n (Tuple3' s (cnt1 - n) (Right fs))
511+ Continue n s -> do
500512 assert (cnt1 - n >= 0) (return ())
501− return $ Skip n (Tuple3' s (cnt1 - n) (Right fs))
502− Stop n b -> do
513+ return $ Continue n (Tuple3' s (cnt1 - n) (Right fs))
514+ Done n b -> do
503515 s <- initial1
504516 fs1 <- fstep fs b
505− return $ Yield n (Tuple3' s 0 (Right fs1))
506− Error _ -> Stop cnt1 <$> fextract fs
517+ return $ Partial n (Tuple3' s 0 (Right fs1))
518+ Error _ -> Done cnt1 <$> fextract fs
507519
508520 -- XXX The "try" may impact performance if this parser is used as a scan
509521 extract (Tuple3' s _ (Left fs)) = extract1 s >>= fstep fs >>= fextract
@@ -581,20 +593,20 @@ concatMap func (Parser stepL initialL extractL) = Parser step initial extract
581593 step (ConcatParseL st) a = do
582594 r <- stepL st a
583595 return $ case r of
584− Yield n s -> Skip n (ConcatParseL s)
585− Skip n s -> Skip n (ConcatParseL s)
586− Stop n b -> Skip n (ConcatParseR (func b))
596+ Partial n s -> Continue n (ConcatParseL s)
597+ Continue n s -> Continue n (ConcatParseL s)
598+ Done n b -> Continue n (ConcatParseR (func b))
587599 Error err -> Error err
588600
589601 step (ConcatParseR (Parser stepR initialR extractR)) a = do
590602 st <- initialR
591603 r <- stepR st a
592604 return $ case r of
593− Yield n s ->
594− Yield n (ConcatParseR (Parser stepR (return s) extractR))
595− Skip n s ->
596− Skip n (ConcatParseR (Parser stepR (return s) extractR))
597− Stop n b -> Stop n b
605+ Partial n s ->
606+ Partial n (ConcatParseR (Parser stepR (return s) extractR))
607+ Continue n s ->
608+ Continue n (ConcatParseR (Parser stepR (return s) extractR))
609+ Done n b -> Done n b
598610 Error err -> Error err
599611
600612 extract (ConcatParseR (Parser _ initialR extractR)) =