LaTeX reader: improve withRaw so it can handle cases where...
the token string is modified by a parser (e.g. accent when
it only takes part of a Word token).
Closes #5686. Still not ideal, because we get the whole
`\t0BAR` and not just `\t0` as a raw latex inline command.
But I'm willing to let this be an edge case, since you
can easily work around this by inserting a space, braces,
or raw attribute. The important thing is that we no longer
drop the rest of the document after a raw latex inline
command that gobbles only part of a Word token!
commit79a3449eeb15d2cd3127a1a5910b3f4b37fd2122
authorJohn MacFarlane <jgm@berkeley.edu>
dateWed 14 Aug 2019, 14:28 (7 years ago)
parente85178c
src/Text/Pandoc/Readers/LaTeX/Parsing.hs+3−2
@@ -677,6 +677,7 @@ withRaw :: PandocMonad m => LP m a -> LP m (a, [Tok])
677677withRaw parser = do
678678 inp <- getInput
679679 result <- parser
680− nxt <- option (Tok (initialPos "source") Word "") (lookAhead anyTok)
681− let raw = takeWhile (/= nxt) inp
680+ nxtpos <- option Nothing ((\(Tok pos' _ _) -> Just pos') <$> lookAhead anyTok)
681+ let raw = takeWhile (\(Tok pos _ _) -> maybe True
682+ (\p -> sourceName p /= sourceName pos || pos < p) nxtpos) inp
682683 return (result, raw)
test/command/5686.mdnew file+9
@@ -0,0 +1,9 @@
1+```
2+% pandoc -t native
3+FOO\t0BAR
4+
5+This part does not make it to the html output.
6+^D
7+[Para [Str "FOO",RawInline (Format "tex") "\\t0BAR"]
8+,Para [Str "This",Space,Str "part",Space,Str "does",Space,Str "not",Space,Str "make",Space,Str "it",Space,Str "to",Space,Str "the",Space,Str "html",Space,Str "output."]]
9+```