git.iko.soy/typeable/pandoc
on master help sign in
Merge pull request #1726 from AlexanderS/twiki-parser

TWiki Reader: add new new twiki reader

commit1d268876f8e8f7c1298e3b87cf59f5fd1b0793b8 authorJohn MacFarlane <jgm@berkeley.edu> dateFri 31 Oct 2014, 12:02 (12 years ago) parentdeaefb12cf09aa
7 files changed+941−10
compare with master
unifiedsplit
README+10−8 pandoc.cabal+4−2 src/Text/Pandoc.hs+3 src/Text/Pandoc/Readers/TWiki.hs+526 tests/Tests/Old.hs+3 tests/twiki-reader.native+174 tests/twiki-reader.twiki+221
README+10−8
@@ -18,10 +18,11 @@ Org-mode], [DocBook], [txt2tags], [EPUB] and [Word docx]; and it can write plain
1818[markdown], [reStructuredText], [XHTML], [HTML 5], [LaTeX] (including
1919[beamer] slide shows), [ConTeXt], [RTF], [OPML], [DocBook],
2020[OpenDocument], [ODT], [Word docx], [GNU Texinfo], [MediaWiki markup],
21−[DokuWiki markup], [Haddock markup], [EPUB] (v2 or v3), [FictionBook2],
22−[Textile], [groff man] pages, [Emacs Org-Mode], [AsciiDoc], [InDesign ICML],
23−and [Slidy], [Slideous], [DZSlides], [reveal.js] or [S5] HTML slide shows.
24−It can also produce [PDF] output on systems where LaTeX is installed.
21+[DokuWiki markup], [TWiki markup], [Haddock markup], [EPUB] (v2 or v3),
22+[FictionBook2], [Textile], [groff man] pages, [Emacs Org-Mode], [AsciiDoc],
23+[InDesign ICML], and [Slidy], [Slideous], [DZSlides], [reveal.js] or
24+[S5] HTML slide shows. It can also produce [PDF] output on systems where
25+LaTeX is installed.
2526
2627Pandoc's enhanced version of markdown includes syntax for footnotes,
2728tables, flexible ordered lists, definition lists, fenced code blocks,
@@ -161,8 +162,8 @@ General options
161162 `textile` (Textile), `rst` (reStructuredText), `html` (HTML),
162163 `docbook` (DocBook), `t2t` (txt2tags), `docx` (docx), `epub` (EPUB),
163164 `opml` (OPML), `org` (Emacs Org-mode), `mediawiki` (MediaWiki markup),
164− `haddock` (Haddock markup), or `latex` (LaTeX). If `+lhs` is appended
165− to `markdown`, `rst`,
165+ `twiki` (TWiki markpu), `haddock` (Haddock markup), or `latex` (LaTeX).
166+ If `+lhs` is appended to `markdown`, `rst`,
166167 `latex`, or `html`, the input will be treated as literate Haskell
167168 source: see [Literate Haskell support](#literate-haskell-support),
168169 below. Markdown syntax extensions can be individually enabled or
@@ -253,8 +254,8 @@ Reader options
253254 to curly quotes, `---` to em-dashes, `--` to en-dashes, and
254255 `...` to ellipses. Nonbreaking spaces are inserted after certain
255256 abbreviations, such as "Mr." (Note: This option is significant only when
256− the input format is `markdown`, `markdown_strict`, or `textile`. It
257− is selected automatically when the input format is `textile` or the
257+ the input format is `markdown`, `markdown_strict`, `textile` or `twiki`.
258+ It is selected automatically when the input format is `textile` or the
258259 output format is `latex` or `context`, unless `--no-tex-ligatures`
259260 is used.)
260261
@@ -3182,6 +3183,7 @@ Rosenthal.
31823183[Textile]: http://redcloth.org/textile
31833184[MediaWiki markup]: http://www.mediawiki.org/wiki/Help:Formatting
31843185[DokuWiki markup]: https://www.dokuwiki.org/dokuwiki
3186+[TWiki markup]: http://twiki.org/cgi-bin/view/TWiki/TextFormattingRules
31853187[Haddock markup]: http://www.haskell.org/haddock/doc/html/ch03s08.html
31863188[groff man]: http://developer.apple.com/DOCUMENTATION/Darwin/Reference/ManPages/man7/groff_man.7.html
31873189[Haskell]: http://www.haskell.org/
pandoc.cabal+4−2
@@ -20,8 +20,8 @@ Description: Pandoc is a Haskell library for converting from one markup
2020 markup, OPML, Emacs Org-Mode, txt2tags and Textile, and it can write
2121 markdown, reStructuredText, HTML, LaTeX, ConTeXt, Docbook,
2222 OPML, OpenDocument, ODT, Word docx, RTF, MediaWiki, DokuWiki,
23− Textile, groff man pages, plain text, Emacs Org-Mode, AsciiDoc,
24− Haddock markup, EPUB (v2 and v3), FictionBook2,
23+ TWiki, Textile, groff man pages, plain text, Emacs Org-Mode,
24+ AsciiDoc, Haddock markup, EPUB (v2 and v3), FictionBook2,
2525 InDesign ICML, and several kinds of HTML/javascript
2626 slide shows (S5, Slidy, Slideous, DZSlides, reveal.js).
2727 .
@@ -182,6 +182,7 @@ Extra-Source-Files:
182182 tests/epub/*.epub
183183 tests/epub/*.native
184184 tests/txt2tags.t2t
185+ tests/twiki-reader.twiki
185186
186187Source-repository head
187188 type: git
@@ -289,6 +290,7 @@ Library
289290 Text.Pandoc.Readers.Textile,
290291 Text.Pandoc.Readers.Native,
291292 Text.Pandoc.Readers.Haddock,
293+ Text.Pandoc.Readers.TWiki,
292294 Text.Pandoc.Readers.Docx,
293295 Text.Pandoc.Readers.EPUB,
294296 Text.Pandoc.Writers.Native,
src/Text/Pandoc.hs+3
@@ -77,6 +77,7 @@ module Text.Pandoc
7777 , readHaddock
7878 , readNative
7979 , readJSON
80+ , readTWiki
8081 , readTxt2Tags
8182 , readTxt2TagsNoMacros
8283 , readEPUB
@@ -133,6 +134,7 @@ import Text.Pandoc.Readers.HTML
133134import Text.Pandoc.Readers.Textile
134135import Text.Pandoc.Readers.Native
135136import Text.Pandoc.Readers.Haddock
137+import Text.Pandoc.Readers.TWiki
136138import Text.Pandoc.Readers.Docx
137139import Text.Pandoc.Readers.Txt2Tags
138140import Text.Pandoc.Readers.EPUB
@@ -233,6 +235,7 @@ readers = [ ("native" , StringReader $ \_ s -> return $ readNative s)
233235 ,("html" , mkStringReader readHtml)
234236 ,("latex" , mkStringReader readLaTeX)
235237 ,("haddock" , mkStringReader readHaddock)
238+ ,("twiki" , mkStringReader readTWiki)
236239 ,("docx" , mkBSReader readDocx)
237240 ,("t2t" , mkStringReader readTxt2TagsNoMacros)
238241 ,("epub" , mkBSReader readEPUB)
src/Text/Pandoc/Readers/TWiki.hsnew file+526
@@ -0,0 +1,526 @@
1+{-# LANGUAGE RelaxedPolyRec, FlexibleInstances, TypeSynonymInstances #-}
2+-- RelaxedPolyRec needed for inlinesBetween on GHC < 7
3+{-
4+ Copyright (C) 2014 Alexander Sulfrian <alexander.sulfrian@fu-berlin.de>
5+
6+This program is free software; you can redistribute it and/or modify
7+it under the terms of the GNU General Public License as published by
8+the Free Software Foundation; either version 2 of the License, or
9+(at your option) any later version.
10+
11+This program is distributed in the hope that it will be useful,
12+but WITHOUT ANY WARRANTY; without even the implied warranty of
13+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the
14+GNU General Public License for more details.
15+
16+You should have received a copy of the GNU General Public License
17+along with this program; if not, write to the Free Software
18+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
19+-}
20+
21+{- |
22+ Module : Text.Pandoc.Readers.TWiki
23+ Copyright : Copyright (C) 2014 Alexander Sulfrian
24+ License : GNU GPL, version 2 or above
25+
26+ Maintainer : Alexander Sulfrian <alexander.sulfrian@fu-berlin.de>
27+ Stability : alpha
28+ Portability : portable
29+
30+Conversion of twiki text to 'Pandoc' document.
31+-}
32+module Text.Pandoc.Readers.TWiki ( readTWiki
33+ , readTWikiWithWarnings
34+ ) where
35+
36+import Text.Pandoc.Definition
37+import qualified Text.Pandoc.Builder as B
38+import Text.Pandoc.Options
39+import Text.Pandoc.Parsing hiding (enclosed, macro, nested)
40+import Text.Pandoc.Readers.HTML (htmlTag, isCommentTag)
41+import Data.Monoid (Monoid, mconcat, mempty)
42+import Control.Applicative ((<$>), (<*), (*>), (<$))
43+import Control.Monad
44+import Text.Printf (printf)
45+import Debug.Trace (trace)
46+import Text.Pandoc.XML (fromEntities)
47+import Data.Maybe (fromMaybe)
48+import Text.HTML.TagSoup
49+import Data.Char (isAlphaNum)
50+import qualified Data.Foldable as F
51+
52+-- | Read twiki from an input string and return a Pandoc document.
53+readTWiki :: ReaderOptions -- ^ Reader options
54+ -> String -- ^ String to parse (assuming @'\n'@ line endings)
55+ -> Pandoc
56+readTWiki opts s =
57+ (readWith parseTWiki) def{ stateOptions = opts } (s ++ "\n\n")
58+
59+readTWikiWithWarnings :: ReaderOptions -- ^ Reader options
60+ -> String -- ^ String to parse (assuming @'\n'@ line endings)
61+ -> (Pandoc, [String])
62+readTWikiWithWarnings opts s =
63+ (readWith parseTWikiWithWarnings) def{ stateOptions = opts } (s ++ "\n\n")
64+ where parseTWikiWithWarnings = do
65+ doc <- parseTWiki
66+ warnings <- stateWarnings <$> getState
67+ return (doc, warnings)
68+
69+type TWParser = Parser [Char] ParserState
70+
71+--
72+-- utility functions
73+--
74+
75+tryMsg :: String -> TWParser a -> TWParser a
76+tryMsg msg p = try p <?> msg
77+
78+skip :: TWParser a -> TWParser ()
79+skip parser = parser >> return ()
80+
81+nested :: TWParser a -> TWParser a
82+nested p = do
83+ nestlevel <- stateMaxNestingLevel <$> getState
84+ guard $ nestlevel > 0
85+ updateState $ \st -> st{ stateMaxNestingLevel = stateMaxNestingLevel st - 1 }
86+ res <- p
87+ updateState $ \st -> st{ stateMaxNestingLevel = nestlevel }
88+ return res
89+
90+htmlElement :: String -> TWParser (Attr, String)
91+htmlElement tag = tryMsg tag $ do
92+ (TagOpen _ attr, _) <- htmlTag (~== TagOpen tag [])
93+ content <- manyTill anyChar (endtag <|> endofinput)
94+ return (htmlAttrToPandoc attr, trim content)
95+ where
96+ endtag = skip $ htmlTag (~== TagClose tag)
97+ endofinput = lookAhead $ try $ skipMany blankline >> skipSpaces >> eof
98+ trim = dropWhile (=='\n') . reverse . dropWhile (=='\n') . reverse
99+
100+htmlAttrToPandoc :: [Attribute String] -> Attr
101+htmlAttrToPandoc attrs = (ident, classes, keyvals)
102+ where
103+ ident = fromMaybe "" $ lookup "id" attrs
104+ classes = maybe [] words $ lookup "class" attrs
105+ keyvals = [(k,v) | (k,v) <- attrs, k /= "id" && k /= "class"]
106+
107+parseHtmlContentWithAttrs :: String -> TWParser a -> TWParser (Attr, [a])
108+parseHtmlContentWithAttrs tag parser = do
109+ (attr, content) <- htmlElement tag
110+ parsedContent <- try $ parseContent content
111+ return (attr, parsedContent)
112+ where
113+ parseContent = parseFromString $ nested $ manyTill parser endOfContent
114+ endOfContent = try $ skipMany blankline >> skipSpaces >> eof
115+
116+parseHtmlContent :: String -> TWParser a -> TWParser [a]
117+parseHtmlContent tag p = parseHtmlContentWithAttrs tag p >>= return . snd
118+
119+--
120+-- main parser
121+--
122+
123+parseTWiki :: TWParser Pandoc
124+parseTWiki = do
125+ bs <- mconcat <$> many block
126+ spaces
127+ eof
128+ return $ B.doc bs
129+
130+
131+--
132+-- block parsers
133+--
134+
135+block :: TWParser B.Blocks
136+block = do
137+ tr <- getOption readerTrace
138+ pos <- getPosition
139+ res <- mempty <$ skipMany1 blankline
140+ <|> blockElements
141+ <|> para
142+ skipMany blankline
143+ when tr $
144+ trace (printf "line %d: %s" (sourceLine pos)
145+ (take 60 $ show $ B.toList res)) (return ())
146+ return res
147+
148+blockElements :: TWParser B.Blocks
149+blockElements = choice [ separator
150+ , header
151+ , verbatim
152+ , literal
153+ , list ""
154+ , table
155+ , blockQuote
156+ , noautolink
157+ ]
158+
159+separator :: TWParser B.Blocks
160+separator = tryMsg "separator" $ string "---" >> newline >> return B.horizontalRule
161+
162+header :: TWParser B.Blocks
163+header = tryMsg "header" $ do
164+ string "---"
165+ level <- many1 (char '+') >>= return . length
166+ guard $ level <= 6
167+ classes <- option [] $ string "!!" >> return ["unnumbered"]
168+ skipSpaces
169+ content <- B.trimInlines . mconcat <$> manyTill inline newline
170+ attr <- registerHeader ("", classes, []) content
171+ return $ B.headerWith attr level $ content
172+
173+verbatim :: TWParser B.Blocks
174+verbatim = (htmlElement "verbatim" <|> htmlElement "pre")
175+ >>= return . (uncurry B.codeBlockWith)
176+
177+literal :: TWParser B.Blocks
178+literal = htmlElement "literal" >>= return . rawBlock
179+ where
180+ format (_, _, kvs) = fromMaybe "html" $ lookup "format" kvs
181+ rawBlock (attrs, content) = B.rawBlock (format attrs) content
182+
183+list :: String -> TWParser B.Blocks
184+list prefix = choice [ bulletList prefix
185+ , orderedList prefix
186+ , definitionList prefix]
187+
188+definitionList :: String -> TWParser B.Blocks
189+definitionList prefix = tryMsg "definitionList" $ do
190+ indent <- lookAhead $ string prefix *> (many1 $ string " ") <* string "$ "
191+ elements <- many $ parseDefinitionListItem (prefix ++ concat indent)
192+ return $ B.definitionList elements
193+ where
194+ parseDefinitionListItem :: String -> TWParser (B.Inlines, [B.Blocks])
195+ parseDefinitionListItem indent = do
196+ string (indent ++ "$ ") >> skipSpaces
197+ term <- many1Till inline $ string ": "
198+ line <- listItemLine indent $ string "$ "
199+ return $ (mconcat term, [line])
200+
201+bulletList :: String -> TWParser B.Blocks
202+bulletList prefix = tryMsg "bulletList" $
203+ parseList prefix (char '*') (char ' ')
204+
205+orderedList :: String -> TWParser B.Blocks
206+orderedList prefix = tryMsg "orderedList" $
207+ parseList prefix (oneOf "1iIaA") (string ". ")
208+
209+parseList :: Show a => String -> TWParser Char -> TWParser a -> TWParser B.Blocks
210+parseList prefix marker delim = do
211+ (indent, style) <- lookAhead $ string prefix *> listStyle <* delim
212+ blocks <- many $ parseListItem (prefix ++ indent) (char style <* delim)
213+ return $ case style of
214+ '1' -> B.orderedListWith (1, DefaultStyle, DefaultDelim) blocks
215+ 'i' -> B.orderedListWith (1, LowerRoman, DefaultDelim) blocks
216+ 'I' -> B.orderedListWith (1, UpperRoman, DefaultDelim) blocks
217+ 'a' -> B.orderedListWith (1, LowerAlpha, DefaultDelim) blocks
218+ 'A' -> B.orderedListWith (1, UpperAlpha, DefaultDelim) blocks
219+ _ -> B.bulletList blocks
220+ where
221+ listStyle = do
222+ indent <- many1 $ string " "
223+ style <- marker
224+ return (concat indent, style)
225+
226+parseListItem :: Show a => String -> TWParser a -> TWParser B.Blocks
227+parseListItem prefix marker = string prefix >> marker >> listItemLine prefix marker
228+
229+listItemLine :: Show a => String -> TWParser a -> TWParser B.Blocks
230+listItemLine prefix marker = lineContent >>= parseContent >>= return . mconcat
231+ where
232+ lineContent = do
233+ content <- anyLine
234+ continuation <- optionMaybe listContinuation
235+ return $ filterSpaces content ++ "\n" ++ (maybe "" (" " ++) continuation)
236+ filterSpaces = reverse . dropWhile (== ' ') . reverse
237+ listContinuation = notFollowedBy (string prefix >> marker) >>
238+ string " " >> lineContent
239+ parseContent = parseFromString $ many1 $ nestedList <|> parseInline
240+ parseInline = many1Till inline (lastNewline <|> newlineBeforeNestedList) >>=
241+ return . B.plain . mconcat
242+ nestedList = list prefix
243+ lastNewline = try $ char '\n' <* eof
244+ newlineBeforeNestedList = try $ char '\n' <* lookAhead nestedList
245+
246+table :: TWParser B.Blocks
247+table = try $ do
248+ tableHead <- optionMaybe $ many1Till tableParseHeader newline >>= return . unzip
249+ rows <- many1 tableParseRow
250+ return $ buildTable mempty rows $ fromMaybe (align rows, columns rows) tableHead
251+ where
252+ buildTable caption rows (aligns, heads)
253+ = B.table caption aligns heads rows
254+ align rows = replicate (columCount rows) (AlignDefault, 0)
255+ columns rows = replicate (columCount rows) mempty
256+ columCount rows = length $ head rows
257+
258+tableParseHeader :: TWParser ((Alignment, Double), B.Blocks)
259+tableParseHeader = try $ do
260+ char '|'
261+ leftSpaces <- many spaceChar >>= return . length
262+ char '*'
263+ content <- tableColumnContent (char '*' >> skipSpaces >> char '|')
264+ char '*'
265+ rightSpaces <- many spaceChar >>= return . length
266+ optional tableEndOfRow
267+ return (tableAlign leftSpaces rightSpaces, content)
268+ where
269+ tableAlign left right
270+ | left >= 2 && left == right = (AlignCenter, 0)
271+ | left > right = (AlignRight, 0)
272+ | otherwise = (AlignLeft, 0)
273+
274+tableParseRow :: TWParser [B.Blocks]
275+tableParseRow = many1Till tableParseColumn newline
276+
277+tableParseColumn :: TWParser B.Blocks
278+tableParseColumn = char '|' *> skipSpaces *>
279+ tableColumnContent (skipSpaces >> char '|')
280+ <* skipSpaces <* optional tableEndOfRow
281+
282+tableEndOfRow :: TWParser Char
283+tableEndOfRow = lookAhead (try $ char '|' >> char '\n') >> char '|'
284+
285+tableColumnContent :: Show a => TWParser a -> TWParser B.Blocks
286+tableColumnContent end = manyTill content (lookAhead $ try end) >>= return . B.plain . mconcat
287+ where
288+ content = continuation <|> inline
289+ continuation = try $ char '\\' >> newline >> return mempty
290+
291+blockQuote :: TWParser B.Blocks
292+blockQuote = parseHtmlContent "blockquote" block >>= return . B.blockQuote . mconcat
293+
294+noautolink :: TWParser B.Blocks
295+noautolink = do
296+ (_, content) <- htmlElement "noautolink"
297+ st <- getState
298+ setState $ st{ stateAllowLinks = False }
299+ blocks <- try $ parseContent content
300+ setState $ st{ stateAllowLinks = True }
301+ return $ mconcat blocks
302+ where
303+ parseContent = parseFromString $ many $ block
304+
305+para :: TWParser B.Blocks
306+para = many1Till inline endOfParaElement >>= return . result . mconcat
307+ where
308+ endOfParaElement = lookAhead $ endOfInput <|> endOfPara <|> newBlockElement
309+ endOfInput = try $ skipMany blankline >> skipSpaces >> eof
310+ endOfPara = try $ blankline >> skipMany1 blankline
311+ newBlockElement = try $ blankline >> skip blockElements
312+ result content = if F.all (==Space) content
313+ then mempty
314+ else B.para $ B.trimInlines content
315+
316+
317+--
318+-- inline parsers
319+--
320+
321+inline :: TWParser B.Inlines
322+inline = choice [ whitespace
323+ , br
324+ , macro
325+ , strong
326+ , strongHtml
327+ , strongAndEmph
328+ , emph
329+ , emphHtml
330+ , boldCode
331+ , smart
332+ , link
333+ , htmlComment
334+ , code
335+ , codeHtml
336+ , nop
337+ , autoLink
338+ , str
339+ , symbol
340+ ] <?> "inline"
341+
342+whitespace :: TWParser B.Inlines
343+whitespace = (lb <|> regsp) >>= return
344+ where lb = try $ skipMany spaceChar >> linebreak >> return B.space
345+ regsp = try $ skipMany1 spaceChar >> return B.space
346+
347+br :: TWParser B.Inlines
348+br = try $ string "%BR%" >> return B.linebreak
349+
350+linebreak :: TWParser B.Inlines
351+linebreak = newline >> notFollowedBy newline >> (lastNewline <|> innerNewline)
352+ where lastNewline = eof >> return mempty
353+ innerNewline = return B.space
354+
355+between :: (Show b, Monoid c) => TWParser a -> TWParser b -> (TWParser b -> TWParser c) -> TWParser c
356+between start end p =
357+ mconcat <$> try (start >> notFollowedBy whitespace >> many1Till (p end) end)
358+
359+enclosed :: (Show a, Monoid b) => TWParser a -> (TWParser a -> TWParser b) -> TWParser b
360+enclosed sep p = between sep (try $ sep <* endMarker) p
361+ where
362+ endMarker = lookAhead $ skip endSpace <|> skip (oneOf ".,!?:)|") <|> eof
363+ endSpace = (spaceChar <|> newline) >> return B.space
364+
365+macro :: TWParser B.Inlines
366+macro = macroWithParameters <|> withoutParameters
367+ where
368+ withoutParameters = enclosed (char '%') (\_ -> macroName) >>= return . emptySpan
369+ emptySpan name = buildSpan name [] mempty
370+
371+macroWithParameters :: TWParser B.Inlines
372+macroWithParameters = try $ do
373+ char '%'
374+ name <- macroName
375+ (content, kvs) <- attributes
376+ char '%'
377+ return $ buildSpan name kvs $ B.str content
378+
379+buildSpan :: String -> [(String, String)] -> B.Inlines -> B.Inlines
380+buildSpan className kvs = B.spanWith attrs
381+ where
382+ attrs = ("", ["twiki-macro", className] ++ additionalClasses, kvsWithoutClasses)
383+ additionalClasses = maybe [] words $ lookup "class" kvs
384+ kvsWithoutClasses = [(k,v) | (k,v) <- kvs, k /= "class"]
385+
386+macroName :: TWParser String
387+macroName = do
388+ first <- letter
389+ rest <- many $ alphaNum <|> char '_'
390+ return (first:rest)
391+
392+attributes :: TWParser (String, [(String, String)])
393+attributes = char '{' *> spnl *> many (attribute <* spnl) <* char '}' >>=
394+ return . foldr (either mkContent mkKvs) ([], [])
395+ where
396+ spnl = skipMany (spaceChar <|> newline)
397+ mkContent c ([], kvs) = (c, kvs)
398+ mkContent c (rest, kvs) = (c ++ " " ++ rest, kvs)
399+ mkKvs kv (cont, rest) = (cont, (kv : rest))
400+
401+attribute :: TWParser (Either String (String, String))
402+attribute = withKey <|> withoutKey
403+ where
404+ withKey = try $ do
405+ key <- macroName
406+ char '='
407+ parseValue False >>= return . (curry Right key)
408+ withoutKey = try $ parseValue True >>= return . Left
409+ parseValue allowSpaces = (withQuotes <|> withoutQuotes allowSpaces) >>= return . fromEntities
410+ withQuotes = between (char '"') (char '"') (\_ -> count 1 $ noneOf ['"'])
411+ withoutQuotes allowSpaces
412+ | allowSpaces == True = many1 $ noneOf "}"
413+ | otherwise = many1 $ noneOf " }"
414+
415+nestedInlines :: Show a => TWParser a -> TWParser B.Inlines
416+nestedInlines end = innerSpace <|> nestedInline
417+ where
418+ innerSpace = try $ whitespace <* (notFollowedBy end)
419+ nestedInline = notFollowedBy whitespace >> nested inline
420+
421+strong :: TWParser B.Inlines
422+strong = try $ enclosed (char '*') nestedInlines >>= return . B.strong
423+
424+strongHtml :: TWParser B.Inlines
425+strongHtml = (parseHtmlContent "strong" inline <|> parseHtmlContent "b" inline)
426+ >>= return . B.strong . mconcat
427+
428+strongAndEmph :: TWParser B.Inlines
429+strongAndEmph = try $ enclosed (string "__") nestedInlines >>= return . B.emph . B.strong
430+
431+emph :: TWParser B.Inlines
432+emph = try $ enclosed (char '_') nestedInlines >>= return . B.emph
433+
434+emphHtml :: TWParser B.Inlines
435+emphHtml = (parseHtmlContent "em" inline <|> parseHtmlContent "i" inline)
436+ >>= return . B.emph . mconcat
437+
438+nestedString :: Show a => TWParser a -> TWParser String
439+nestedString end = innerSpace <|> (count 1 nonspaceChar)
440+ where
441+ innerSpace = try $ many1 spaceChar <* notFollowedBy end
442+
443+boldCode :: TWParser B.Inlines
444+boldCode = try $ enclosed (string "==") nestedString >>= return . B.strong . B.code . fromEntities
445+
446+htmlComment :: TWParser B.Inlines
447+htmlComment = htmlTag isCommentTag >> return mempty
448+
449+code :: TWParser B.Inlines
450+code = try $ enclosed (char '=') nestedString >>= return . B.code . fromEntities
451+
452+codeHtml :: TWParser B.Inlines
453+codeHtml = do
454+ (attrs, content) <- parseHtmlContentWithAttrs "code" anyChar
455+ return $ B.codeWith attrs $ fromEntities content
456+
457+autoLink :: TWParser B.Inlines
458+autoLink = try $ do
459+ state <- getState
460+ guard $ stateAllowLinks state
461+ (text, url) <- parseLink
462+ guard $ checkLink (head $ reverse url)
463+ return $ makeLink (text, url)
464+ where
465+ parseLink = notFollowedBy nop >> (uri <|> emailAddress)
466+ makeLink (text, url) = B.link url "" $ B.str text
467+ checkLink c
468+ | c == '/' = True
469+ | otherwise = isAlphaNum c
470+
471+str :: TWParser B.Inlines
472+str = (many1 alphaNum <|> count 1 characterReference) >>= return . B.str
473+
474+nop :: TWParser B.Inlines
475+nop = try $ (skip exclamation <|> skip nopTag) >> followContent
476+ where
477+ exclamation = char '!'
478+ nopTag = stringAnyCase "<nop>"
479+ followContent = many1 nonspaceChar >>= return . B.str . fromEntities
480+
481+symbol :: TWParser B.Inlines
482+symbol = count 1 nonspaceChar >>= return . B.str
483+
484+smart :: TWParser B.Inlines
485+smart = do
486+ getOption readerSmart >>= guard
487+ doubleQuoted <|> singleQuoted <|>
488+ choice [ apostrophe
489+ , dash
490+ , ellipses
491+ ]
492+
493+singleQuoted :: TWParser B.Inlines
494+singleQuoted = try $ do
495+ singleQuoteStart
496+ withQuoteContext InSingleQuote $
497+ many1Till inline singleQuoteEnd >>=
498+ (return . B.singleQuoted . B.trimInlines . mconcat)
499+
500+doubleQuoted :: TWParser B.Inlines
501+doubleQuoted = try $ do
502+ doubleQuoteStart
503+ contents <- mconcat <$> many (try $ notFollowedBy doubleQuoteEnd >> inline)
504+ (withQuoteContext InDoubleQuote $ doubleQuoteEnd >>
505+ return (B.doubleQuoted $ B.trimInlines contents))
506+ <|> (return $ (B.str "\8220") B.<> contents)
507+
508+link :: TWParser B.Inlines
509+link = try $ do
510+ st <- getState
511+ guard $ stateAllowLinks st
512+ setState $ st{ stateAllowLinks = False }
513+ (url, title, content) <- linkText
514+ setState $ st{ stateAllowLinks = True }
515+ return $ B.link url title content
516+
517+linkText :: TWParser (String, String, B.Inlines)
518+linkText = do
519+ string "[["
520+ url <- many1Till anyChar (char ']')
521+ content <- option [B.str url] linkContent
522+ char ']'
523+ return (url, "", mconcat content)
524+ where
525+ linkContent = (char '[') >> many1Till anyChar (char ']') >>= parseLinkContent
526+ parseLinkContent = parseFromString $ many1 inline
tests/Tests/Old.hs+3
@@ -153,6 +153,9 @@ tests = [ testGroup "markdown"
153153 , test "formatting" ["-r", "epub", "-w", "native"]
154154 "epub/formatting.epub" "epub/formatting.native"
155155 ]
156+ , testGroup "twiki"
157+ [ test "reader" ["-r", "twiki", "-w", "native", "-s"]
158+ "twiki-reader.twiki" "twiki-reader.native" ]
156159 , testGroup "other writers" $ map (\f -> testGroup f $ writerTests f)
157160 [ "opendocument" , "context" , "texinfo", "icml"
158161 , "man" , "plain" , "rtf", "org", "asciidoc"
tests/twiki-reader.nativenew file+174
@@ -0,0 +1,174 @@
1+Pandoc (Meta {unMeta = fromList []})
2+[Header 1 ("header",[],[]) [Str "header"]
3+,Header 2 ("header-level-two",[],[]) [Str "header",Space,Str "level",Space,Str "two"]
4+,Header 3 ("header-level-3",[],[]) [Str "header",Space,Str "level",Space,Str "3"]
5+,Header 4 ("header-level-four",[],[]) [Str "header",Space,Emph [Str "level"],Space,Str "four"]
6+,Header 5 ("header-level-5",[],[]) [Str "header",Space,Str "level",Space,Str "5"]
7+,Header 6 ("header-level-6",[],[]) [Str "header",Space,Str "level",Space,Str "6"]
8+,Para [Str "---+++++++",Space,Str "not",Space,Str "a",Space,Str "header"]
9+,Para [Str "--++",Space,Str "not",Space,Str "a",Space,Str "header"]
10+,Header 1 ("emph-and-strong",[],[]) [Str "emph",Space,Str "and",Space,Str "strong"]
11+,Para [Emph [Str "emph"],Space,Strong [Str "strong"]]
12+,Para [Emph [Strong [Str "strong",Space,Str "and",Space,Str "emph"]]]
13+,Para [Strong [Emph [Str "emph",Space,Str "inside"],Space,Str "strong"]]
14+,Para [Strong [Str "strong",Space,Str "with",Space,Emph [Str "emph"]]]
15+,Para [Emph [Strong [Str "strong",Space,Str "inside"],Space,Str "emph"]]
16+,Header 1 ("horizontal-rule",[],[]) [Str "horizontal",Space,Str "rule"]
17+,Para [Str "top"]
18+,HorizontalRule
19+,Para [Str "bottom"]
20+,HorizontalRule
21+,Header 1 ("nop",[],[]) [Str "nop"]
22+,Para [Str "_not",Space,Str "emph_"]
23+,Header 1 ("entities",[],[]) [Str "entities"]
24+,Para [Str "hi",Space,Str "&",Space,Str "low"]
25+,Para [Str "hi",Space,Str "&",Space,Str "low"]
26+,Para [Str "G\246del"]
27+,Para [Str "\777\2730"]
28+,Header 1 ("comments",[],[]) [Str "comments"]
29+,Para [Str "inline",Space,Str "comment"]
30+,Para [Str "between",Space,Str "blocks"]
31+,Header 1 ("linebreaks",[],[]) [Str "linebreaks"]
32+,Para [Str "hi",LineBreak,Str "there"]
33+,Para [Str "hi",LineBreak,Space,Str "there"]
34+,Header 1 ("inline-code",[],[]) [Str "inline",Space,Str "code"]
35+,Para [Code ("",[],[]) "*\8594*",Space,Code ("",[],[]) "typed",Space,Code ("",["haskell"],[]) ">>="]
36+,Header 1 ("code-blocks",[],[]) [Str "code",Space,Str "blocks"]
37+,CodeBlock ("",[],[]) "case xs of\n (_:_) -> reverse xs\n [] -> ['*']"
38+,CodeBlock ("",["haskell"],[]) "case xs of\n (_:_) -> reverse xs\n [] -> ['*']"
39+,Header 1 ("block-quotes",[],[]) [Str "block",Space,Str "quotes"]
40+,Para [Str "Regular",Space,Str "paragraph"]
41+,BlockQuote
42+ [Para [Str "This",Space,Str "is",Space,Str "a",Space,Str "block",Space,Str "quote."]
43+ ,Para [Str "With",Space,Str "two",Space,Str "paragraphs."]]
44+,Para [Str "Nother",Space,Str "paragraph."]
45+,Header 1 ("external-links",[],[]) [Str "external",Space,Str "links"]
46+,Para [Link [Emph [Str "Google"],Space,Str "search",Space,Str "engine"] ("http://google.com","")]
47+,Para [Link [Str "http://johnmacfarlane.net/pandoc/"] ("http://johnmacfarlane.net/pandoc/","")]
48+,Para [Link [Str "http://google.com"] ("http://google.com",""),Space,Link [Str "http://yahoo.com"] ("http://yahoo.com","")]
49+,Para [Link [Str "email",Space,Str "me"] ("mailto:info@example.org","")]
50+,Para [Str "http://google.com"]
51+,Para [Str "http://google.com"]
52+,Para [Str "http://google.com"]
53+,Para [Str "info@example.org"]
54+,Para [Str "info@example.org"]
55+,Para [Str "info@example.org"]
56+,Header 1 ("lists",[],[]) [Str "lists"]
57+,BulletList
58+ [[Plain [Str "Start",Space,Str "each",Space,Str "line"]]
59+ ,[Plain [Str "with",Space,Str "an",Space,Str "asterisk",Space,Str "(*)."]
60+ ,BulletList
61+ [[Plain [Str "More",Space,Str "asterisks",Space,Str "gives",Space,Str "deeper"]
62+ ,BulletList
63+ [[Plain [Str "and",Space,Str "deeper",Space,Str "levels."]]]]]]
64+ ,[Plain [Str "Line",Space,Str "breaks",LineBreak,Str "don't",Space,Str "break",Space,Str "levels."]]
65+ ,[Plain [Str "Continuations",Space,Str "are",Space,Str "also",Space,Str "possible"]
66+ ,BulletList
67+ [[Plain [Str "and",Space,Str "do",Space,Str "not",Space,Str "break",Space,Str "the",Space,Str "list",Space,Str "flow"]]]]
68+ ,[Plain [Str "Level",Space,Str "one"]]]
69+,Para [Str "Any",Space,Str "other",Space,Str "start",Space,Str "ends",Space,Str "the",Space,Str "list."]
70+,OrderedList (1,DefaultStyle,DefaultDelim)
71+ [[Plain [Str "Start",Space,Str "each",Space,Str "line"]]
72+ ,[Plain [Str "with",Space,Str "a",Space,Str "number",Space,Str "(1.)."]
73+ ,OrderedList (1,DefaultStyle,DefaultDelim)
74+ [[Plain [Str "More",Space,Str "number",Space,Str "signs",Space,Str "gives",Space,Str "deeper"]
75+ ,OrderedList (1,DefaultStyle,DefaultDelim)
76+ [[Plain [Str "and",Space,Str "deeper"]]
77+ ,[Plain [Str "levels."]]]]]]
78+ ,[Plain [Str "Line",Space,Str "breaks",LineBreak,Str "don't",Space,Str "break",Space,Str "levels."]]
79+ ,[Plain [Str "Blank",Space,Str "lines"]]]
80+,OrderedList (1,DefaultStyle,DefaultDelim)
81+ [[Plain [Str "end",Space,Str "the",Space,Str "list",Space,Str "and",Space,Str "start",Space,Str "another."]]]
82+,Para [Str "Any",Space,Str "other",Space,Str "start",Space,Str "also",Space,Str "ends",Space,Str "the",Space,Str "list."]
83+,DefinitionList
84+ [([Str "item",Space,Str "1"],
85+ [[Plain [Str "definition",Space,Str "1"]]])
86+ ,([Str "item",Space,Str "2"],
87+ [[Plain [Str "definition",Space,Str "2-1",Space,Str "definition",Space,Str "2-2"]]])
88+ ,([Str "item",Space,Emph [Str "3"]],
89+ [[Plain [Str "definition",Space,Emph [Str "3"]]]])]
90+,OrderedList (1,DefaultStyle,DefaultDelim)
91+ [[Plain [Str "one"]]
92+ ,[Plain [Str "two"]
93+ ,BulletList
94+ [[Plain [Str "two",Space,Str "point",Space,Str "one"]]
95+ ,[Plain [Str "two",Space,Str "point",Space,Str "two"]]]]
96+ ,[Plain [Str "three"]
97+ ,DefinitionList
98+ [([Str "three",Space,Str "item",Space,Str "one"],
99+ [[Plain [Str "three",Space,Str "def",Space,Str "one"]]])]]
100+ ,[Plain [Str "four"]
101+ ,DefinitionList
102+ [([Str "four",Space,Str "def",Space,Str "one"],
103+ [[Plain [Str "this",Space,Str "is",Space,Str "a",Space,Str "continuation"]]])]]
104+ ,[Plain [Str "five"]
105+ ,OrderedList (1,DefaultStyle,DefaultDelim)
106+ [[Plain [Str "five",Space,Str "sub",Space,Str "1"]
107+ ,OrderedList (1,DefaultStyle,DefaultDelim)
108+ [[Plain [Str "five",Space,Str "sub",Space,Str "1",Space,Str "sub",Space,Str "1"]]]]
109+ ,[Plain [Str "five",Space,Str "sub",Space,Str "2"]]]]]
110+,OrderedList (1,DefaultStyle,DefaultDelim)
111+ [[Plain [Str "other"]
112+ ,OrderedList (1,UpperRoman,DefaultDelim)
113+ [[Plain [Str "list"]]
114+ ,[Plain [Str "styles"]]]]
115+ ,[Plain [Str "are"]
116+ ,OrderedList (1,LowerRoman,DefaultDelim)
117+ [[Plain [Str "also"]]
118+ ,[Plain [Str "possible"]]]]
119+ ,[Plain [Str "all"]
120+ ,OrderedList (1,LowerAlpha,DefaultDelim)
121+ [[Plain [Str "the"]]
122+ ,[Plain [Str "different"]]
123+ ,[Plain [Str "styles"]]]]
124+ ,[Plain [Str "are"]
125+ ,OrderedList (1,UpperAlpha,DefaultDelim)
126+ [[Plain [Str "implemented"]]
127+ ,[Plain [Str "and"]]
128+ ,[Plain [Str "supported"]]]]]
129+,Header 1 ("tables",[],[]) [Str "tables"]
130+,Table [] [AlignDefault,AlignDefault] [0.0,0.0]
131+ [[]
132+ ,[]]
133+ [[[Plain [Str "Orange"]]
134+ ,[Plain [Str "Apple"]]]
135+ ,[[Plain [Str "Bread"]]
136+ ,[Plain [Str "Pie"]]]
137+ ,[[Plain [Str "Butter"]]
138+ ,[Plain [Str "Ice",Space,Str "cream"]]]]
139+,Table [] [AlignLeft,AlignLeft] [0.0,0.0]
140+ [[Plain [Str "Orange"]]
141+ ,[Plain [Str "Apple"]]]
142+ [[[Plain [Str "Bread"]]
143+ ,[Plain [Str "Pie"]]]
144+ ,[[Plain [Strong [Str "Butter"]]]
145+ ,[Plain [Str "Ice",Space,Str "cream"]]]]
146+,Table [] [AlignLeft,AlignLeft] [0.0,0.0]
147+ [[Plain [Str "Orange"]]
148+ ,[Plain [Str "Apple"]]]
149+ [[[Plain [Str "Bread",LineBreak,LineBreak,Str "and",Space,Str "cheese"]]
150+ ,[Plain [Str "Pie",LineBreak,LineBreak,Strong [Str "apple"],Space,Str "and",Space,Emph [Str "carrot"]]]]]
151+,Table [] [AlignDefault,AlignDefault,AlignDefault] [0.0,0.0,0.0]
152+ [[]
153+ ,[]
154+ ,[]]
155+ [[[Plain [Str "Orange"]]
156+ ,[Plain [Str "Apple"]]
157+ ,[Plain [Str "more"]]]
158+ ,[[Plain [Str "Bread"]]
159+ ,[Plain [Str "Pie"]]
160+ ,[Plain [Str "more"]]]
161+ ,[[Plain [Str "Butter"]]
162+ ,[Plain [Str "Ice",Space,Str "cream"]]
163+ ,[Plain [Str "and",Space,Str "more"]]]]
164+,Header 1 ("macros",[],[]) [Str "macros"]
165+,Para [Span ("",["twiki-macro","TEST"],[]) []]
166+,Para [Span ("",["twiki-macro","TEST"],[]) [Str ""]]
167+,Para [Span ("",["twiki-macro","TEST"],[]) [Str "content with spaces"]]
168+,Para [Span ("",["twiki-macro","TEST"],[]) [Str "content with spaces"]]
169+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test")]) [Str "content with spaces"]]
170+,Para [Span ("",["twiki-macro","TEST"],[]) [Str "content with spaces ARG1=test"]]
171+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test")]) [Str "content with spaces"]]
172+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test"),("ARG2","test2")]) [Str ""]]
173+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test"),("ARG2","test2")]) [Str ""]]
174+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test"),("ARG2","test2")]) [Str "multiline\ndoes also work"]]]
tests/twiki-reader.twikinew file+221
@@ -0,0 +1,221 @@
1+---+ header
2+
3+---++ header level two
4+
5+---+++ header level 3
6+
7+---++++ header _level_ four
8+
9+---+++++ header level 5
10+
11+---++++++ header level 6
12+
13+---+++++++ not a header
14+
15+ --++ not a header
16+
17+---+ emph and strong
18+
19+_emph_ *strong*
20+
21+__strong and emph__
22+
23+*<i>emph inside</i> strong*
24+
25+*strong with <i>emph</i>*
26+
27+_<b>strong inside</b> emph_
28+
29+---+ horizontal rule
30+
31+top
32+---
33+bottom
34+
35+---
36+
37+---+ nop
38+
39+<nop>_not emph_
40+
41+---+ entities
42+
43+hi & low
44+
45+hi &amp; low
46+
47+G&ouml;del
48+
49+&#777;&#xAAA;
50+
51+---+ comments
52+
53+inline <!-- secret --> comment
54+
55+<!-- secret -->
56+
57+between blocks
58+
59+ <!-- secret -->
60+
61+---+ linebreaks
62+
63+hi%BR%there
64+
65+hi%BR%
66+there
67+
68+---+ inline code
69+
70+<code>*→*</code> =typed= <code class="haskell">>>=</code>
71+
72+---+ code blocks
73+
74+<verbatim>
75+case xs of
76+ (_:_) -> reverse xs
77+ [] -> ['*']
78+</verbatim>
79+
80+<verbatim class="haskell">
81+case xs of
82+ (_:_) -> reverse xs
83+ [] -> ['*']
84+</verbatim>
85+
86+---+ block quotes
87+
88+Regular paragraph
89+<blockquote>
90+This is a block quote.
91+
92+With two paragraphs.
93+</blockquote>
94+Nother paragraph.
95+
96+---+ external links
97+
98+[[http://google.com][<i>Google</i> search engine]]
99+
100+http://johnmacfarlane.net/pandoc/
101+
102+[[http://google.com]] [[http://yahoo.com]]
103+
104+[[mailto:info@example.org][email me]]
105+
106+!http://google.com
107+
108+<nop>http://google.com
109+
110+<noautolink>
111+http://google.com
112+</noautolink>
113+
114+!info@example.org
115+
116+<nop>info@example.org
117+
118+<noautolink>
119+info@example.org
120+</noautolink>
121+
122+---+ lists
123+
124+ * Start each line
125+ * with an asterisk (*).
126+ * More asterisks gives deeper
127+ * and deeper levels.
128+ * Line breaks%BR%don't break levels.
129+ * Continuations
130+ are also possible
131+ * and do not break the list flow
132+ * Level one
133+Any other start ends the list.
134+
135+ 1. Start each line
136+ 1. with a number (1.).
137+ 1. More number signs gives deeper
138+ 1. and deeper
139+ 1. levels.
140+ 1. Line breaks%BR%don't break levels.
141+ 1. Blank lines
142+
143+ 1. end the list and start another.
144+Any other start also
145+ends the list.
146+
147+ $ item 1: definition 1
148+ $ item 2: definition 2-1
149+ definition 2-2
150+ $ item _3_: definition _3_
151+
152+ 1. one
153+ 1. two
154+ * two point one
155+ * two point two
156+ 1. three
157+ $ three item one: three def one
158+ 1. four
159+ $ four def one: this
160+ is a continuation
161+ 1. five
162+ 1. five sub 1
163+ 1. five sub 1 sub 1
164+ 1. five sub 2
165+
166+ 1. other
167+ I. list
168+ I. styles
169+ 1. are
170+ i. also
171+ i. possible
172+ 1. all
173+ a. the
174+ a. different
175+ a. styles
176+ 1. are
177+ A. implemented
178+ A. and
179+ A. supported
180+
181+---+ tables
182+
183+|Orange|Apple|
184+|Bread|Pie|
185+|Butter|Ice cream|
186+
187+|*Orange*|*Apple*|
188+|Bread|Pie|
189+|*Butter*|Ice cream|
190+
191+|*Orange*|*Apple*|
192+|Bread%BR%%BR%and cheese|Pie%BR%%BR%*apple* and <i>carrot</i>|
193+
194+| Orange | Apple | more |
195+| Bread | Pie | more |
196+| Butter | Ice cream | and more |
197+
198+---+ macros
199+
200+%TEST%
201+
202+%TEST{}%
203+
204+%TEST{content with spaces}%
205+
206+%TEST{"content with spaces"}%
207+
208+%TEST{"content with spaces" ARG1="test"}%
209+
210+%TEST{content with spaces ARG1=test}%
211+
212+%TEST{ARG1=test content with spaces}%
213+
214+%TEST{ARG1=test ARG2=test2}%
215+
216+%TEST{ARG1="test" ARG2="test2"}%
217+
218+%TEST{ARG1="test"
219+ARG2="test2"
220+multiline
221+does also work}%