1d26887 · typeable/pandoc

git.iko.soy/typeable/pandoc

on master help sign in

Merge pull request #1726 from AlexanderS/twiki-parser

1d26887John MacFarlane12 years ago

README+10−8

compare with master

unifiedsplit

@@ -18,10 +18,11 @@ Org-mode], [DocBook], [txt2tags], [EPUB] and [Word docx]; and it can write plain

18[markdown], [reStructuredText], [XHTML], [HTML 5], [LaTeX] (including

19[beamer] slide shows), [ConTeXt], [RTF], [OPML], [DocBook],

20[OpenDocument], [ODT], [Word docx], [GNU Texinfo], [MediaWiki markup],

21−[DokuWiki markup], [Haddock markup], [EPUB] (v2 or v3), [FictionBook2],

21+[DokuWiki markup], [TWiki markup], [Haddock markup], [EPUB] (v2 or v3),

22−[Textile], [groff man] pages, [Emacs Org-Mode], [AsciiDoc], [InDesign ICML],

22+[FictionBook2], [Textile], [groff man] pages, [Emacs Org-Mode], [AsciiDoc],

23−and [Slidy], [Slideous], [DZSlides], [reveal.js] or [S5] HTML slide shows.

23+[InDesign ICML], and [Slidy], [Slideous], [DZSlides], [reveal.js] or

24−It can also produce [PDF] output on systems where LaTeX is installed.

24+[S5] HTML slide shows. It can also produce [PDF] output on systems where

25+LaTeX is installed.

26Pandoc's enhanced version of markdown includes syntax for footnotes,

27Pandoc's enhanced version of markdown includes syntax for footnotes,

27tables, flexible ordered lists, definition lists, fenced code blocks,

28tables, flexible ordered lists, definition lists, fenced code blocks,

@@ -161,8 +162,8 @@ General options

161 `textile` (Textile), `rst` (reStructuredText), `html` (HTML),

162 `textile` (Textile), `rst` (reStructuredText), `html` (HTML),

162 `docbook` (DocBook), `t2t` (txt2tags), `docx` (docx), `epub` (EPUB),

163 `docbook` (DocBook), `t2t` (txt2tags), `docx` (docx), `epub` (EPUB),

163 `opml` (OPML), `org` (Emacs Org-mode), `mediawiki` (MediaWiki markup),

164 `opml` (OPML), `org` (Emacs Org-mode), `mediawiki` (MediaWiki markup),

164− `haddock` (Haddock markup), or `latex` (LaTeX). If `+lhs` is appended

165+ `twiki` (TWiki markpu), `haddock` (Haddock markup), or `latex` (LaTeX).

165− to `markdown`, `rst`,

166+ If `+lhs` is appended to `markdown`, `rst`,

166 `latex`, or `html`, the input will be treated as literate Haskell

167 `latex`, or `html`, the input will be treated as literate Haskell

167 source: see [Literate Haskell support](#literate-haskell-support),

168 source: see [Literate Haskell support](#literate-haskell-support),

168 below. Markdown syntax extensions can be individually enabled or

169 below. Markdown syntax extensions can be individually enabled or

@@ -253,8 +254,8 @@ Reader options

253 to curly quotes, `---` to em-dashes, `--` to en-dashes, and

254 to curly quotes, `---` to em-dashes, `--` to en-dashes, and

254 `...` to ellipses. Nonbreaking spaces are inserted after certain

255 `...` to ellipses. Nonbreaking spaces are inserted after certain

255 abbreviations, such as "Mr." (Note: This option is significant only when

256 abbreviations, such as "Mr." (Note: This option is significant only when

256− the input format is `markdown`, `markdown_strict`, or `textile`. It

257+ the input format is `markdown`, `markdown_strict`, `textile` or `twiki`.

257− is selected automatically when the input format is `textile` or the

258+ It is selected automatically when the input format is `textile` or the

258 output format is `latex` or `context`, unless `--no-tex-ligatures`

259 output format is `latex` or `context`, unless `--no-tex-ligatures`

259 is used.)

260 is used.)

260

261

@@ -3182,6 +3183,7 @@ Rosenthal.

3182[Textile]: http://redcloth.org/textile

3183[Textile]: http://redcloth.org/textile

3183[MediaWiki markup]: http://www.mediawiki.org/wiki/Help:Formatting

3184[MediaWiki markup]: http://www.mediawiki.org/wiki/Help:Formatting

3184[DokuWiki markup]: https://www.dokuwiki.org/dokuwiki

3185[DokuWiki markup]: https://www.dokuwiki.org/dokuwiki

3186+[TWiki markup]: http://twiki.org/cgi-bin/view/TWiki/TextFormattingRules

3185[Haddock markup]: http://www.haskell.org/haddock/doc/html/ch03s08.html

3187[Haddock markup]: http://www.haskell.org/haddock/doc/html/ch03s08.html

3186[groff man]: http://developer.apple.com/DOCUMENTATION/Darwin/Reference/ManPages/man7/groff_man.7.html

3188[groff man]: http://developer.apple.com/DOCUMENTATION/Darwin/Reference/ManPages/man7/groff_man.7.html

3187[Haskell]: http://www.haskell.org/

3189[Haskell]: http://www.haskell.org/

pandoc.cabal+4−2

@@ -20,8 +20,8 @@ Description: Pandoc is a Haskell library for converting from one markup

20 markup, OPML, Emacs Org-Mode, txt2tags and Textile, and it can write

21 markdown, reStructuredText, HTML, LaTeX, ConTeXt, Docbook,

22 OPML, OpenDocument, ODT, Word docx, RTF, MediaWiki, DokuWiki,

23− Textile, groff man pages, plain text, Emacs Org-Mode, AsciiDoc,

23+ TWiki, Textile, groff man pages, plain text, Emacs Org-Mode,

24− Haddock markup, EPUB (v2 and v3), FictionBook2,

24+ AsciiDoc, Haddock markup, EPUB (v2 and v3), FictionBook2,

25 InDesign ICML, and several kinds of HTML/javascript

26 slide shows (S5, Slidy, Slideous, DZSlides, reveal.js).

27 .

@@ -182,6 +182,7 @@ Extra-Source-Files:

182 tests/epub/*.epub

183 tests/epub/*.native

184 tests/txt2tags.t2t

185+ tests/twiki-reader.twiki

185

186

186Source-repository head

187Source-repository head

187 type: git

188 type: git

@@ -289,6 +290,7 @@ Library

289 Text.Pandoc.Readers.Textile,

290 Text.Pandoc.Readers.Textile,

290 Text.Pandoc.Readers.Native,

291 Text.Pandoc.Readers.Native,

291 Text.Pandoc.Readers.Haddock,

292 Text.Pandoc.Readers.Haddock,

293+ Text.Pandoc.Readers.TWiki,

292 Text.Pandoc.Readers.Docx,

294 Text.Pandoc.Readers.Docx,

293 Text.Pandoc.Readers.EPUB,

295 Text.Pandoc.Readers.EPUB,

294 Text.Pandoc.Writers.Native,

296 Text.Pandoc.Writers.Native,

src/Text/Pandoc.hs+3

@@ -77,6 +77,7 @@ module Text.Pandoc

77 , readHaddock

78 , readNative

79 , readJSON

80+ , readTWiki

80 , readTxt2Tags

81 , readTxt2Tags

81 , readTxt2TagsNoMacros

82 , readTxt2TagsNoMacros

82 , readEPUB

83 , readEPUB

@@ -133,6 +134,7 @@ import Text.Pandoc.Readers.HTML

133import Text.Pandoc.Readers.Textile

134import Text.Pandoc.Readers.Textile

134import Text.Pandoc.Readers.Native

135import Text.Pandoc.Readers.Native

135import Text.Pandoc.Readers.Haddock

136import Text.Pandoc.Readers.Haddock

137+import Text.Pandoc.Readers.TWiki

136import Text.Pandoc.Readers.Docx

138import Text.Pandoc.Readers.Docx

137import Text.Pandoc.Readers.Txt2Tags

139import Text.Pandoc.Readers.Txt2Tags

138import Text.Pandoc.Readers.EPUB

140import Text.Pandoc.Readers.EPUB

@@ -233,6 +235,7 @@ readers = [ ("native" , StringReader $ \_ s -> return $ readNative s)

233 ,("html" , mkStringReader readHtml)

235 ,("html" , mkStringReader readHtml)

234 ,("latex" , mkStringReader readLaTeX)

236 ,("latex" , mkStringReader readLaTeX)

235 ,("haddock" , mkStringReader readHaddock)

237 ,("haddock" , mkStringReader readHaddock)

238+ ,("twiki" , mkStringReader readTWiki)

236 ,("docx" , mkBSReader readDocx)

239 ,("docx" , mkBSReader readDocx)

237 ,("t2t" , mkStringReader readTxt2TagsNoMacros)

240 ,("t2t" , mkStringReader readTxt2TagsNoMacros)

238 ,("epub" , mkBSReader readEPUB)

241 ,("epub" , mkBSReader readEPUB)

src/Text/Pandoc/Readers/TWiki.hsnew file+526

@@ -0,0 +1,526 @@

1+{-# LANGUAGE RelaxedPolyRec, FlexibleInstances, TypeSynonymInstances #-}

2+-- RelaxedPolyRec needed for inlinesBetween on GHC < 7

3+{-

6+This program is free software; you can redistribute it and/or modify

7+it under the terms of the GNU General Public License as published by

8+the Free Software Foundation; either version 2 of the License, or

9+(at your option) any later version.

10+

11+This program is distributed in the hope that it will be useful,

12+but WITHOUT ANY WARRANTY; without even the implied warranty of

13+MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the

14+GNU General Public License for more details.

15+

16+You should have received a copy of the GNU General Public License

17+along with this program; if not, write to the Free Software

18+Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA

19+-}

20+

21+{- |

22+ Module : Text.Pandoc.Readers.TWiki

24+ License : GNU GPL, version 2 or above

25+

26+ Maintainer : Alexander Sulfrian <alexander.sulfrian@fu-berlin.de>

27+ Stability : alpha

28+ Portability : portable

29+

30+Conversion of twiki text to 'Pandoc' document.

31+-}

32+module Text.Pandoc.Readers.TWiki ( readTWiki

33+ , readTWikiWithWarnings

34+ ) where

35+

36+import Text.Pandoc.Definition

37+import qualified Text.Pandoc.Builder as B

38+import Text.Pandoc.Options

39+import Text.Pandoc.Parsing hiding (enclosed, macro, nested)

40+import Text.Pandoc.Readers.HTML (htmlTag, isCommentTag)

41+import Data.Monoid (Monoid, mconcat, mempty)

42+import Control.Applicative ((<$>), (<*), (*>), (<$))

43+import Control.Monad

44+import Text.Printf (printf)

45+import Debug.Trace (trace)

46+import Text.Pandoc.XML (fromEntities)

47+import Data.Maybe (fromMaybe)

48+import Text.HTML.TagSoup

49+import Data.Char (isAlphaNum)

50+import qualified Data.Foldable as F

51+

52+-- | Read twiki from an input string and return a Pandoc document.

53+readTWiki :: ReaderOptions -- ^ Reader options

54+ -> String -- ^ String to parse (assuming @'\n'@ line endings)

55+ -> Pandoc

56+readTWiki opts s =

57+ (readWith parseTWiki) def{ stateOptions = opts } (s ++ "\n\n")

58+

59+readTWikiWithWarnings :: ReaderOptions -- ^ Reader options

60+ -> String -- ^ String to parse (assuming @'\n'@ line endings)

61+ -> (Pandoc, [String])

62+readTWikiWithWarnings opts s =

63+ (readWith parseTWikiWithWarnings) def{ stateOptions = opts } (s ++ "\n\n")

64+ where parseTWikiWithWarnings = do

65+ doc <- parseTWiki

66+ warnings <- stateWarnings <$> getState

67+ return (doc, warnings)

68+

69+type TWParser = Parser [Char] ParserState

70+

71+--

72+-- utility functions

73+--

74+

75+tryMsg :: String -> TWParser a -> TWParser a

76+tryMsg msg p = try p <?> msg

77+

78+skip :: TWParser a -> TWParser ()

79+skip parser = parser >> return ()

80+

81+nested :: TWParser a -> TWParser a

82+nested p = do

83+ nestlevel <- stateMaxNestingLevel <$> getState

84+ guard $ nestlevel > 0

85+ updateState $ \st -> st{ stateMaxNestingLevel = stateMaxNestingLevel st - 1 }

86+ res <- p

87+ updateState $ \st -> st{ stateMaxNestingLevel = nestlevel }

88+ return res

89+

90+htmlElement :: String -> TWParser (Attr, String)

91+htmlElement tag = tryMsg tag $ do

92+ (TagOpen _ attr, _) <- htmlTag (~== TagOpen tag [])

93+ content <- manyTill anyChar (endtag <|> endofinput)

94+ return (htmlAttrToPandoc attr, trim content)

95+ where

96+ endtag = skip $ htmlTag (~== TagClose tag)

97+ endofinput = lookAhead $ try $ skipMany blankline >> skipSpaces >> eof

98+ trim = dropWhile (=='\n') . reverse . dropWhile (=='\n') . reverse

99+

100+htmlAttrToPandoc :: [Attribute String] -> Attr

101+htmlAttrToPandoc attrs = (ident, classes, keyvals)

102+ where

103+ ident = fromMaybe "" $ lookup "id" attrs

104+ classes = maybe [] words $ lookup "class" attrs

105+ keyvals = [(k,v) | (k,v) <- attrs, k /= "id" && k /= "class"]

106+

107+parseHtmlContentWithAttrs :: String -> TWParser a -> TWParser (Attr, [a])

108+parseHtmlContentWithAttrs tag parser = do

109+ (attr, content) <- htmlElement tag

110+ parsedContent <- try $ parseContent content

111+ return (attr, parsedContent)

112+ where

113+ parseContent = parseFromString $ nested $ manyTill parser endOfContent

114+ endOfContent = try $ skipMany blankline >> skipSpaces >> eof

115+

116+parseHtmlContent :: String -> TWParser a -> TWParser [a]

117+parseHtmlContent tag p = parseHtmlContentWithAttrs tag p >>= return . snd

118+

119+--

120+-- main parser

121+--

122+

123+parseTWiki :: TWParser Pandoc

124+parseTWiki = do

125+ bs <- mconcat <$> many block

126+ spaces

127+ eof

128+ return $ B.doc bs

129+

130+

131+--

132+-- block parsers

133+--

134+

135+block :: TWParser B.Blocks

136+block = do

137+ tr <- getOption readerTrace

138+ pos <- getPosition

139+ res <- mempty <$ skipMany1 blankline

140+ <|> blockElements

141+ <|> para

142+ skipMany blankline

143+ when tr $

144+ trace (printf "line %d: %s" (sourceLine pos)

145+ (take 60 $ show $ B.toList res)) (return ())

146+ return res

147+

148+blockElements :: TWParser B.Blocks

149+blockElements = choice [ separator

150+ , header

151+ , verbatim

152+ , literal

153+ , list ""

154+ , table

155+ , blockQuote

156+ , noautolink

157+ ]

158+

159+separator :: TWParser B.Blocks

160+separator = tryMsg "separator" $ string "---" >> newline >> return B.horizontalRule

161+

162+header :: TWParser B.Blocks

163+header = tryMsg "header" $ do

164+ string "---"

165+ level <- many1 (char '+') >>= return . length

166+ guard $ level <= 6

167+ classes <- option [] $ string "!!" >> return ["unnumbered"]

168+ skipSpaces

169+ content <- B.trimInlines . mconcat <$> manyTill inline newline

170+ attr <- registerHeader ("", classes, []) content

171+ return $ B.headerWith attr level $ content

172+

173+verbatim :: TWParser B.Blocks

174+verbatim = (htmlElement "verbatim" <|> htmlElement "pre")

175+ >>= return . (uncurry B.codeBlockWith)

176+

177+literal :: TWParser B.Blocks

178+literal = htmlElement "literal" >>= return . rawBlock

179+ where

180+ format (_, _, kvs) = fromMaybe "html" $ lookup "format" kvs

181+ rawBlock (attrs, content) = B.rawBlock (format attrs) content

182+

183+list :: String -> TWParser B.Blocks

184+list prefix = choice [ bulletList prefix

185+ , orderedList prefix

186+ , definitionList prefix]

187+

188+definitionList :: String -> TWParser B.Blocks

189+definitionList prefix = tryMsg "definitionList" $ do

190+ indent <- lookAhead $ string prefix *> (many1 $ string " ") <* string "$ "

191+ elements <- many $ parseDefinitionListItem (prefix ++ concat indent)

192+ return $ B.definitionList elements

193+ where

194+ parseDefinitionListItem :: String -> TWParser (B.Inlines, [B.Blocks])

195+ parseDefinitionListItem indent = do

196+ string (indent ++ "$ ") >> skipSpaces

197+ term <- many1Till inline $ string ": "

198+ line <- listItemLine indent $ string "$ "

199+ return $ (mconcat term, [line])

200+

201+bulletList :: String -> TWParser B.Blocks

202+bulletList prefix = tryMsg "bulletList" $

203+ parseList prefix (char '*') (char ' ')

204+

205+orderedList :: String -> TWParser B.Blocks

206+orderedList prefix = tryMsg "orderedList" $

207+ parseList prefix (oneOf "1iIaA") (string ". ")

208+

209+parseList :: Show a => String -> TWParser Char -> TWParser a -> TWParser B.Blocks

210+parseList prefix marker delim = do

211+ (indent, style) <- lookAhead $ string prefix *> listStyle <* delim

212+ blocks <- many $ parseListItem (prefix ++ indent) (char style <* delim)

213+ return $ case style of

214+ '1' -> B.orderedListWith (1, DefaultStyle, DefaultDelim) blocks

215+ 'i' -> B.orderedListWith (1, LowerRoman, DefaultDelim) blocks

216+ 'I' -> B.orderedListWith (1, UpperRoman, DefaultDelim) blocks

217+ 'a' -> B.orderedListWith (1, LowerAlpha, DefaultDelim) blocks

218+ 'A' -> B.orderedListWith (1, UpperAlpha, DefaultDelim) blocks

219+ _ -> B.bulletList blocks

220+ where

221+ listStyle = do

222+ indent <- many1 $ string " "

223+ style <- marker

224+ return (concat indent, style)

225+

226+parseListItem :: Show a => String -> TWParser a -> TWParser B.Blocks

227+parseListItem prefix marker = string prefix >> marker >> listItemLine prefix marker

228+

229+listItemLine :: Show a => String -> TWParser a -> TWParser B.Blocks

230+listItemLine prefix marker = lineContent >>= parseContent >>= return . mconcat

231+ where

232+ lineContent = do

233+ content <- anyLine

234+ continuation <- optionMaybe listContinuation

235+ return $ filterSpaces content ++ "\n" ++ (maybe "" (" " ++) continuation)

236+ filterSpaces = reverse . dropWhile (== ' ') . reverse

237+ listContinuation = notFollowedBy (string prefix >> marker) >>

238+ string " " >> lineContent

239+ parseContent = parseFromString $ many1 $ nestedList <|> parseInline

240+ parseInline = many1Till inline (lastNewline <|> newlineBeforeNestedList) >>=

241+ return . B.plain . mconcat

242+ nestedList = list prefix

243+ lastNewline = try $ char '\n' <* eof

244+ newlineBeforeNestedList = try $ char '\n' <* lookAhead nestedList

245+

246+table :: TWParser B.Blocks

247+table = try $ do

248+ tableHead <- optionMaybe $ many1Till tableParseHeader newline >>= return . unzip

249+ rows <- many1 tableParseRow

250+ return $ buildTable mempty rows $ fromMaybe (align rows, columns rows) tableHead

251+ where

252+ buildTable caption rows (aligns, heads)

253+ = B.table caption aligns heads rows

254+ align rows = replicate (columCount rows) (AlignDefault, 0)

255+ columns rows = replicate (columCount rows) mempty

256+ columCount rows = length $ head rows

257+

258+tableParseHeader :: TWParser ((Alignment, Double), B.Blocks)

259+tableParseHeader = try $ do

260+ char '|'

261+ leftSpaces <- many spaceChar >>= return . length

262+ char '*'

263+ content <- tableColumnContent (char '*' >> skipSpaces >> char '|')

264+ char '*'

265+ rightSpaces <- many spaceChar >>= return . length

266+ optional tableEndOfRow

267+ return (tableAlign leftSpaces rightSpaces, content)

268+ where

269+ tableAlign left right

270+ | left >= 2 && left == right = (AlignCenter, 0)

271+ | left > right = (AlignRight, 0)

272+ | otherwise = (AlignLeft, 0)

273+

274+tableParseRow :: TWParser [B.Blocks]

275+tableParseRow = many1Till tableParseColumn newline

276+

277+tableParseColumn :: TWParser B.Blocks

278+tableParseColumn = char '|' *> skipSpaces *>

279+ tableColumnContent (skipSpaces >> char '|')

280+ <* skipSpaces <* optional tableEndOfRow

281+

282+tableEndOfRow :: TWParser Char

283+tableEndOfRow = lookAhead (try $ char '|' >> char '\n') >> char '|'

284+

285+tableColumnContent :: Show a => TWParser a -> TWParser B.Blocks

286+tableColumnContent end = manyTill content (lookAhead $ try end) >>= return . B.plain . mconcat

287+ where

288+ content = continuation <|> inline

289+ continuation = try $ char '\\' >> newline >> return mempty

290+

291+blockQuote :: TWParser B.Blocks

292+blockQuote = parseHtmlContent "blockquote" block >>= return . B.blockQuote . mconcat

293+

294+noautolink :: TWParser B.Blocks

295+noautolink = do

296+ (_, content) <- htmlElement "noautolink"

297+ st <- getState

298+ setState $ st{ stateAllowLinks = False }

299+ blocks <- try $ parseContent content

300+ setState $ st{ stateAllowLinks = True }

301+ return $ mconcat blocks

302+ where

303+ parseContent = parseFromString $ many $ block

304+

305+para :: TWParser B.Blocks

306+para = many1Till inline endOfParaElement >>= return . result . mconcat

307+ where

308+ endOfParaElement = lookAhead $ endOfInput <|> endOfPara <|> newBlockElement

309+ endOfInput = try $ skipMany blankline >> skipSpaces >> eof

310+ endOfPara = try $ blankline >> skipMany1 blankline

311+ newBlockElement = try $ blankline >> skip blockElements

312+ result content = if F.all (==Space) content

313+ then mempty

314+ else B.para $ B.trimInlines content

315+

316+

317+--

318+-- inline parsers

319+--

320+

321+inline :: TWParser B.Inlines

322+inline = choice [ whitespace

323+ , br

324+ , macro

325+ , strong

326+ , strongHtml

327+ , strongAndEmph

328+ , emph

329+ , emphHtml

330+ , boldCode

331+ , smart

332+ , link

333+ , htmlComment

334+ , code

335+ , codeHtml

336+ , nop

337+ , autoLink

338+ , str

339+ , symbol

340+ ] <?> "inline"

341+

342+whitespace :: TWParser B.Inlines

343+whitespace = (lb <|> regsp) >>= return

344+ where lb = try $ skipMany spaceChar >> linebreak >> return B.space

345+ regsp = try $ skipMany1 spaceChar >> return B.space

346+

347+br :: TWParser B.Inlines

348+br = try $ string "%BR%" >> return B.linebreak

349+

350+linebreak :: TWParser B.Inlines

351+linebreak = newline >> notFollowedBy newline >> (lastNewline <|> innerNewline)

352+ where lastNewline = eof >> return mempty

353+ innerNewline = return B.space

354+

355+between :: (Show b, Monoid c) => TWParser a -> TWParser b -> (TWParser b -> TWParser c) -> TWParser c

356+between start end p =

357+ mconcat <$> try (start >> notFollowedBy whitespace >> many1Till (p end) end)

358+

359+enclosed :: (Show a, Monoid b) => TWParser a -> (TWParser a -> TWParser b) -> TWParser b

360+enclosed sep p = between sep (try $ sep <* endMarker) p

361+ where

362+ endMarker = lookAhead $ skip endSpace <|> skip (oneOf ".,!?:)|") <|> eof

363+ endSpace = (spaceChar <|> newline) >> return B.space

364+

365+macro :: TWParser B.Inlines

366+macro = macroWithParameters <|> withoutParameters

367+ where

368+ withoutParameters = enclosed (char '%') (\_ -> macroName) >>= return . emptySpan

369+ emptySpan name = buildSpan name [] mempty

370+

371+macroWithParameters :: TWParser B.Inlines

372+macroWithParameters = try $ do

373+ char '%'

374+ name <- macroName

375+ (content, kvs) <- attributes

376+ char '%'

377+ return $ buildSpan name kvs $ B.str content

378+

379+buildSpan :: String -> [(String, String)] -> B.Inlines -> B.Inlines

380+buildSpan className kvs = B.spanWith attrs

381+ where

382+ attrs = ("", ["twiki-macro", className] ++ additionalClasses, kvsWithoutClasses)

383+ additionalClasses = maybe [] words $ lookup "class" kvs

384+ kvsWithoutClasses = [(k,v) | (k,v) <- kvs, k /= "class"]

385+

386+macroName :: TWParser String

387+macroName = do

388+ first <- letter

389+ rest <- many $ alphaNum <|> char '_'

390+ return (first:rest)

391+

392+attributes :: TWParser (String, [(String, String)])

393+attributes = char '{' *> spnl *> many (attribute <* spnl) <* char '}' >>=

394+ return . foldr (either mkContent mkKvs) ([], [])

395+ where

396+ spnl = skipMany (spaceChar <|> newline)

397+ mkContent c ([], kvs) = (c, kvs)

398+ mkContent c (rest, kvs) = (c ++ " " ++ rest, kvs)

399+ mkKvs kv (cont, rest) = (cont, (kv : rest))

400+

401+attribute :: TWParser (Either String (String, String))

402+attribute = withKey <|> withoutKey

403+ where

404+ withKey = try $ do

405+ key <- macroName

406+ char '='

407+ parseValue False >>= return . (curry Right key)

408+ withoutKey = try $ parseValue True >>= return . Left

409+ parseValue allowSpaces = (withQuotes <|> withoutQuotes allowSpaces) >>= return . fromEntities

410+ withQuotes = between (char '"') (char '"') (\_ -> count 1 $ noneOf ['"'])

411+ withoutQuotes allowSpaces

412+ | allowSpaces == True = many1 $ noneOf "}"

413+ | otherwise = many1 $ noneOf " }"

414+

415+nestedInlines :: Show a => TWParser a -> TWParser B.Inlines

416+nestedInlines end = innerSpace <|> nestedInline

417+ where

418+ innerSpace = try $ whitespace <* (notFollowedBy end)

419+ nestedInline = notFollowedBy whitespace >> nested inline

420+

421+strong :: TWParser B.Inlines

422+strong = try $ enclosed (char '*') nestedInlines >>= return . B.strong

423+

424+strongHtml :: TWParser B.Inlines

425+strongHtml = (parseHtmlContent "strong" inline <|> parseHtmlContent "b" inline)

426+ >>= return . B.strong . mconcat

427+

428+strongAndEmph :: TWParser B.Inlines

429+strongAndEmph = try $ enclosed (string "__") nestedInlines >>= return . B.emph . B.strong

430+

431+emph :: TWParser B.Inlines

432+emph = try $ enclosed (char '_') nestedInlines >>= return . B.emph

433+

434+emphHtml :: TWParser B.Inlines

435+emphHtml = (parseHtmlContent "em" inline <|> parseHtmlContent "i" inline)

436+ >>= return . B.emph . mconcat

437+

438+nestedString :: Show a => TWParser a -> TWParser String

439+nestedString end = innerSpace <|> (count 1 nonspaceChar)

440+ where

441+ innerSpace = try $ many1 spaceChar <* notFollowedBy end

442+

443+boldCode :: TWParser B.Inlines

444+boldCode = try $ enclosed (string "==") nestedString >>= return . B.strong . B.code . fromEntities

445+

446+htmlComment :: TWParser B.Inlines

447+htmlComment = htmlTag isCommentTag >> return mempty

448+

449+code :: TWParser B.Inlines

450+code = try $ enclosed (char '=') nestedString >>= return . B.code . fromEntities

451+

452+codeHtml :: TWParser B.Inlines

453+codeHtml = do

454+ (attrs, content) <- parseHtmlContentWithAttrs "code" anyChar

455+ return $ B.codeWith attrs $ fromEntities content

456+

457+autoLink :: TWParser B.Inlines

458+autoLink = try $ do

459+ state <- getState

460+ guard $ stateAllowLinks state

461+ (text, url) <- parseLink

462+ guard $ checkLink (head $ reverse url)

463+ return $ makeLink (text, url)

464+ where

465+ parseLink = notFollowedBy nop >> (uri <|> emailAddress)

466+ makeLink (text, url) = B.link url "" $ B.str text

467+ checkLink c

468+ | c == '/' = True

469+ | otherwise = isAlphaNum c

470+

471+str :: TWParser B.Inlines

472+str = (many1 alphaNum <|> count 1 characterReference) >>= return . B.str

473+

474+nop :: TWParser B.Inlines

475+nop = try $ (skip exclamation <|> skip nopTag) >> followContent

476+ where

477+ exclamation = char '!'

478+ nopTag = stringAnyCase "<nop>"

479+ followContent = many1 nonspaceChar >>= return . B.str . fromEntities

480+

481+symbol :: TWParser B.Inlines

482+symbol = count 1 nonspaceChar >>= return . B.str

483+

484+smart :: TWParser B.Inlines

485+smart = do

486+ getOption readerSmart >>= guard

487+ doubleQuoted <|> singleQuoted <|>

488+ choice [ apostrophe

489+ , dash

490+ , ellipses

491+ ]

492+

493+singleQuoted :: TWParser B.Inlines

494+singleQuoted = try $ do

495+ singleQuoteStart

496+ withQuoteContext InSingleQuote $

497+ many1Till inline singleQuoteEnd >>=

498+ (return . B.singleQuoted . B.trimInlines . mconcat)

499+

500+doubleQuoted :: TWParser B.Inlines

501+doubleQuoted = try $ do

502+ doubleQuoteStart

503+ contents <- mconcat <$> many (try $ notFollowedBy doubleQuoteEnd >> inline)

504+ (withQuoteContext InDoubleQuote $ doubleQuoteEnd >>

505+ return (B.doubleQuoted $ B.trimInlines contents))

506+ <|> (return $ (B.str "\8220") B.<> contents)

507+

508+link :: TWParser B.Inlines

509+link = try $ do

510+ st <- getState

511+ guard $ stateAllowLinks st

512+ setState $ st{ stateAllowLinks = False }

513+ (url, title, content) <- linkText

514+ setState $ st{ stateAllowLinks = True }

515+ return $ B.link url title content

516+

517+linkText :: TWParser (String, String, B.Inlines)

518+linkText = do

519+ string "[["

520+ url <- many1Till anyChar (char ']')

521+ content <- option [B.str url] linkContent

522+ char ']'

523+ return (url, "", mconcat content)

524+ where

525+ linkContent = (char '[') >> many1Till anyChar (char ']') >>= parseLinkContent

526+ parseLinkContent = parseFromString $ many1 inline

tests/Tests/Old.hs+3

@@ -153,6 +153,9 @@ tests = [ testGroup "markdown"

153 , test "formatting" ["-r", "epub", "-w", "native"]

154 "epub/formatting.epub" "epub/formatting.native"

155 ]

156+ , testGroup "twiki"

157+ [ test "reader" ["-r", "twiki", "-w", "native", "-s"]

158+ "twiki-reader.twiki" "twiki-reader.native" ]

156 , testGroup "other writers" $ map (\f -> testGroup f $ writerTests f)

159 , testGroup "other writers" $ map (\f -> testGroup f $ writerTests f)

157 [ "opendocument" , "context" , "texinfo", "icml"

160 [ "opendocument" , "context" , "texinfo", "icml"

158 , "man" , "plain" , "rtf", "org", "asciidoc"

161 , "man" , "plain" , "rtf", "org", "asciidoc"

tests/twiki-reader.nativenew file+174

@@ -0,0 +1,174 @@

1+Pandoc (Meta {unMeta = fromList []})

2+[Header 1 ("header",[],[]) [Str "header"]

3+,Header 2 ("header-level-two",[],[]) [Str "header",Space,Str "level",Space,Str "two"]

4+,Header 3 ("header-level-3",[],[]) [Str "header",Space,Str "level",Space,Str "3"]

5+,Header 4 ("header-level-four",[],[]) [Str "header",Space,Emph [Str "level"],Space,Str "four"]

6+,Header 5 ("header-level-5",[],[]) [Str "header",Space,Str "level",Space,Str "5"]

7+,Header 6 ("header-level-6",[],[]) [Str "header",Space,Str "level",Space,Str "6"]

8+,Para [Str "---+++++++",Space,Str "not",Space,Str "a",Space,Str "header"]

9+,Para [Str "--++",Space,Str "not",Space,Str "a",Space,Str "header"]

10+,Header 1 ("emph-and-strong",[],[]) [Str "emph",Space,Str "and",Space,Str "strong"]

11+,Para [Emph [Str "emph"],Space,Strong [Str "strong"]]

12+,Para [Emph [Strong [Str "strong",Space,Str "and",Space,Str "emph"]]]

13+,Para [Strong [Emph [Str "emph",Space,Str "inside"],Space,Str "strong"]]

14+,Para [Strong [Str "strong",Space,Str "with",Space,Emph [Str "emph"]]]

15+,Para [Emph [Strong [Str "strong",Space,Str "inside"],Space,Str "emph"]]

16+,Header 1 ("horizontal-rule",[],[]) [Str "horizontal",Space,Str "rule"]

17+,Para [Str "top"]

18+,HorizontalRule

19+,Para [Str "bottom"]

20+,HorizontalRule

21+,Header 1 ("nop",[],[]) [Str "nop"]

22+,Para [Str "_not",Space,Str "emph_"]

23+,Header 1 ("entities",[],[]) [Str "entities"]

24+,Para [Str "hi",Space,Str "&",Space,Str "low"]

25+,Para [Str "hi",Space,Str "&",Space,Str "low"]

26+,Para [Str "G\246del"]

27+,Para [Str "\777\2730"]

28+,Header 1 ("comments",[],[]) [Str "comments"]

29+,Para [Str "inline",Space,Str "comment"]

30+,Para [Str "between",Space,Str "blocks"]

31+,Header 1 ("linebreaks",[],[]) [Str "linebreaks"]

32+,Para [Str "hi",LineBreak,Str "there"]

33+,Para [Str "hi",LineBreak,Space,Str "there"]

34+,Header 1 ("inline-code",[],[]) [Str "inline",Space,Str "code"]

35+,Para [Code ("",[],[]) "*\8594*",Space,Code ("",[],[]) "typed",Space,Code ("",["haskell"],[]) ">>="]

36+,Header 1 ("code-blocks",[],[]) [Str "code",Space,Str "blocks"]

37+,CodeBlock ("",[],[]) "case xs of\n (_:_) -> reverse xs\n [] -> ['*']"

38+,CodeBlock ("",["haskell"],[]) "case xs of\n (_:_) -> reverse xs\n [] -> ['*']"

39+,Header 1 ("block-quotes",[],[]) [Str "block",Space,Str "quotes"]

40+,Para [Str "Regular",Space,Str "paragraph"]

41+,BlockQuote

42+ [Para [Str "This",Space,Str "is",Space,Str "a",Space,Str "block",Space,Str "quote."]

43+ ,Para [Str "With",Space,Str "two",Space,Str "paragraphs."]]

44+,Para [Str "Nother",Space,Str "paragraph."]

45+,Header 1 ("external-links",[],[]) [Str "external",Space,Str "links"]

46+,Para [Link [Emph [Str "Google"],Space,Str "search",Space,Str "engine"] ("http://google.com","")]

47+,Para [Link [Str "http://johnmacfarlane.net/pandoc/"] ("http://johnmacfarlane.net/pandoc/","")]

48+,Para [Link [Str "http://google.com"] ("http://google.com",""),Space,Link [Str "http://yahoo.com"] ("http://yahoo.com","")]

49+,Para [Link [Str "email",Space,Str "me"] ("mailto:info@example.org","")]

50+,Para [Str "http://google.com"]

51+,Para [Str "http://google.com"]

52+,Para [Str "http://google.com"]

53+,Para [Str "info@example.org"]

54+,Para [Str "info@example.org"]

55+,Para [Str "info@example.org"]

56+,Header 1 ("lists",[],[]) [Str "lists"]

57+,BulletList

58+ [[Plain [Str "Start",Space,Str "each",Space,Str "line"]]

59+ ,[Plain [Str "with",Space,Str "an",Space,Str "asterisk",Space,Str "(*)."]

60+ ,BulletList

61+ [[Plain [Str "More",Space,Str "asterisks",Space,Str "gives",Space,Str "deeper"]

62+ ,BulletList

63+ [[Plain [Str "and",Space,Str "deeper",Space,Str "levels."]]]]]]

64+ ,[Plain [Str "Line",Space,Str "breaks",LineBreak,Str "don't",Space,Str "break",Space,Str "levels."]]

65+ ,[Plain [Str "Continuations",Space,Str "are",Space,Str "also",Space,Str "possible"]

66+ ,BulletList

67+ [[Plain [Str "and",Space,Str "do",Space,Str "not",Space,Str "break",Space,Str "the",Space,Str "list",Space,Str "flow"]]]]

68+ ,[Plain [Str "Level",Space,Str "one"]]]

69+,Para [Str "Any",Space,Str "other",Space,Str "start",Space,Str "ends",Space,Str "the",Space,Str "list."]

70+,OrderedList (1,DefaultStyle,DefaultDelim)

71+ [[Plain [Str "Start",Space,Str "each",Space,Str "line"]]

72+ ,[Plain [Str "with",Space,Str "a",Space,Str "number",Space,Str "(1.)."]

73+ ,OrderedList (1,DefaultStyle,DefaultDelim)

74+ [[Plain [Str "More",Space,Str "number",Space,Str "signs",Space,Str "gives",Space,Str "deeper"]

75+ ,OrderedList (1,DefaultStyle,DefaultDelim)

76+ [[Plain [Str "and",Space,Str "deeper"]]

77+ ,[Plain [Str "levels."]]]]]]

78+ ,[Plain [Str "Line",Space,Str "breaks",LineBreak,Str "don't",Space,Str "break",Space,Str "levels."]]

79+ ,[Plain [Str "Blank",Space,Str "lines"]]]

80+,OrderedList (1,DefaultStyle,DefaultDelim)

81+ [[Plain [Str "end",Space,Str "the",Space,Str "list",Space,Str "and",Space,Str "start",Space,Str "another."]]]

82+,Para [Str "Any",Space,Str "other",Space,Str "start",Space,Str "also",Space,Str "ends",Space,Str "the",Space,Str "list."]

83+,DefinitionList

84+ [([Str "item",Space,Str "1"],

85+ [[Plain [Str "definition",Space,Str "1"]]])

86+ ,([Str "item",Space,Str "2"],

87+ [[Plain [Str "definition",Space,Str "2-1",Space,Str "definition",Space,Str "2-2"]]])

88+ ,([Str "item",Space,Emph [Str "3"]],

89+ [[Plain [Str "definition",Space,Emph [Str "3"]]]])]

90+,OrderedList (1,DefaultStyle,DefaultDelim)

91+ [[Plain [Str "one"]]

92+ ,[Plain [Str "two"]

93+ ,BulletList

94+ [[Plain [Str "two",Space,Str "point",Space,Str "one"]]

95+ ,[Plain [Str "two",Space,Str "point",Space,Str "two"]]]]

96+ ,[Plain [Str "three"]

97+ ,DefinitionList

98+ [([Str "three",Space,Str "item",Space,Str "one"],

99+ [[Plain [Str "three",Space,Str "def",Space,Str "one"]]])]]

100+ ,[Plain [Str "four"]

101+ ,DefinitionList

102+ [([Str "four",Space,Str "def",Space,Str "one"],

103+ [[Plain [Str "this",Space,Str "is",Space,Str "a",Space,Str "continuation"]]])]]

104+ ,[Plain [Str "five"]

105+ ,OrderedList (1,DefaultStyle,DefaultDelim)

106+ [[Plain [Str "five",Space,Str "sub",Space,Str "1"]

107+ ,OrderedList (1,DefaultStyle,DefaultDelim)

108+ [[Plain [Str "five",Space,Str "sub",Space,Str "1",Space,Str "sub",Space,Str "1"]]]]

109+ ,[Plain [Str "five",Space,Str "sub",Space,Str "2"]]]]]

110+,OrderedList (1,DefaultStyle,DefaultDelim)

111+ [[Plain [Str "other"]

112+ ,OrderedList (1,UpperRoman,DefaultDelim)

113+ [[Plain [Str "list"]]

114+ ,[Plain [Str "styles"]]]]

115+ ,[Plain [Str "are"]

116+ ,OrderedList (1,LowerRoman,DefaultDelim)

117+ [[Plain [Str "also"]]

118+ ,[Plain [Str "possible"]]]]

119+ ,[Plain [Str "all"]

120+ ,OrderedList (1,LowerAlpha,DefaultDelim)

121+ [[Plain [Str "the"]]

122+ ,[Plain [Str "different"]]

123+ ,[Plain [Str "styles"]]]]

124+ ,[Plain [Str "are"]

125+ ,OrderedList (1,UpperAlpha,DefaultDelim)

126+ [[Plain [Str "implemented"]]

127+ ,[Plain [Str "and"]]

128+ ,[Plain [Str "supported"]]]]]

129+,Header 1 ("tables",[],[]) [Str "tables"]

130+,Table [] [AlignDefault,AlignDefault] [0.0,0.0]

131+ [[]

132+ ,[]]

133+ [[[Plain [Str "Orange"]]

134+ ,[Plain [Str "Apple"]]]

135+ ,[[Plain [Str "Bread"]]

136+ ,[Plain [Str "Pie"]]]

137+ ,[[Plain [Str "Butter"]]

138+ ,[Plain [Str "Ice",Space,Str "cream"]]]]

139+,Table [] [AlignLeft,AlignLeft] [0.0,0.0]

140+ [[Plain [Str "Orange"]]

141+ ,[Plain [Str "Apple"]]]

142+ [[[Plain [Str "Bread"]]

143+ ,[Plain [Str "Pie"]]]

144+ ,[[Plain [Strong [Str "Butter"]]]

145+ ,[Plain [Str "Ice",Space,Str "cream"]]]]

146+,Table [] [AlignLeft,AlignLeft] [0.0,0.0]

147+ [[Plain [Str "Orange"]]

148+ ,[Plain [Str "Apple"]]]

149+ [[[Plain [Str "Bread",LineBreak,LineBreak,Str "and",Space,Str "cheese"]]

150+ ,[Plain [Str "Pie",LineBreak,LineBreak,Strong [Str "apple"],Space,Str "and",Space,Emph [Str "carrot"]]]]]

151+,Table [] [AlignDefault,AlignDefault,AlignDefault] [0.0,0.0,0.0]

152+ [[]

153+ ,[]

154+ ,[]]

155+ [[[Plain [Str "Orange"]]

156+ ,[Plain [Str "Apple"]]

157+ ,[Plain [Str "more"]]]

158+ ,[[Plain [Str "Bread"]]

159+ ,[Plain [Str "Pie"]]

160+ ,[Plain [Str "more"]]]

161+ ,[[Plain [Str "Butter"]]

162+ ,[Plain [Str "Ice",Space,Str "cream"]]

163+ ,[Plain [Str "and",Space,Str "more"]]]]

164+,Header 1 ("macros",[],[]) [Str "macros"]

165+,Para [Span ("",["twiki-macro","TEST"],[]) []]

166+,Para [Span ("",["twiki-macro","TEST"],[]) [Str ""]]

167+,Para [Span ("",["twiki-macro","TEST"],[]) [Str "content with spaces"]]

168+,Para [Span ("",["twiki-macro","TEST"],[]) [Str "content with spaces"]]

169+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test")]) [Str "content with spaces"]]

170+,Para [Span ("",["twiki-macro","TEST"],[]) [Str "content with spaces ARG1=test"]]

171+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test")]) [Str "content with spaces"]]

172+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test"),("ARG2","test2")]) [Str ""]]

173+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test"),("ARG2","test2")]) [Str ""]]

174+,Para [Span ("",["twiki-macro","TEST"],[("ARG1","test"),("ARG2","test2")]) [Str "multiline\ndoes also work"]]]

tests/twiki-reader.twikinew file+221

@@ -0,0 +1,221 @@

1+---+ header

3+---++ header level two

5+---+++ header level 3

7+---++++ header _level_ four

9+---+++++ header level 5

10+

11+---++++++ header level 6

12+

13+---+++++++ not a header

14+

15+ --++ not a header

16+

17+---+ emph and strong

18+

19+_emph_ *strong*

20+

21+__strong and emph__

22+

23+*emph inside strong*

24+

25+*strong with emph*

26+

27+_strong inside emph_

28+

29+---+ horizontal rule

30+

31+top

32+---

33+bottom

34+

35+---

36+

37+---+ nop

38+

39+<nop>_not emph_

40+

41+---+ entities

42+

43+hi & low

44+

45+hi & low

46+

47+Gödel

48+

49+̉પ

50+

51+---+ comments

52+

53+inline  comment

54+

55+

56+

57+between blocks

58+

59+

60+

61+---+ linebreaks

62+

63+hi%BR%there

64+

65+hi%BR%

66+there

67+

68+---+ inline code

69+

70+<code>*→*</code> =typed= <code class="haskell">>>=</code>

71+

72+---+ code blocks

73+

74+<verbatim>

75+case xs of

76+ (_:_) -> reverse xs

77+ [] -> ['*']

78+</verbatim>

79+

80+<verbatim class="haskell">

81+case xs of

82+ (_:_) -> reverse xs

83+ [] -> ['*']

84+</verbatim>

85+

86+---+ block quotes

87+

88+Regular paragraph

89+<blockquote>

90+This is a block quote.

91+

92+With two paragraphs.

93+</blockquote>

94+Nother paragraph.

95+

96+---+ external links

97+

98+[[http://google.com][Google search engine]]

99+

100+http://johnmacfarlane.net/pandoc/

101+

102+[[http://google.com]] [[http://yahoo.com]]

103+

104+[[mailto:info@example.org][email me]]

105+

106+!http://google.com

107+

108+<nop>http://google.com

109+

110+<noautolink>

111+http://google.com

112+</noautolink>

113+

114+!info@example.org

115+

116+<nop>info@example.org

117+

118+<noautolink>

119+info@example.org

120+</noautolink>

121+

122+---+ lists

123+

124+ * Start each line

125+ * with an asterisk (*).

126+ * More asterisks gives deeper

127+ * and deeper levels.

128+ * Line breaks%BR%don't break levels.

129+ * Continuations

130+ are also possible

131+ * and do not break the list flow

132+ * Level one

133+Any other start ends the list.

134+

135+ 1. Start each line

136+ 1. with a number (1.).

137+ 1. More number signs gives deeper

138+ 1. and deeper

139+ 1. levels.

140+ 1. Line breaks%BR%don't break levels.

141+ 1. Blank lines

142+

143+ 1. end the list and start another.

144+Any other start also

145+ends the list.

146+

147+ $ item 1: definition 1

148+ $ item 2: definition 2-1

149+ definition 2-2

150+ $ item _3_: definition _3_

151+

152+ 1. one

153+ 1. two

154+ * two point one

155+ * two point two

156+ 1. three

157+ $ three item one: three def one

158+ 1. four

159+ $ four def one: this

160+ is a continuation

161+ 1. five

162+ 1. five sub 1

163+ 1. five sub 1 sub 1

164+ 1. five sub 2

165+

166+ 1. other

167+ I. list

168+ I. styles

169+ 1. are

170+ i. also

171+ i. possible

172+ 1. all

173+ a. the

174+ a. different

175+ a. styles

176+ 1. are

177+ A. implemented

178+ A. and

179+ A. supported

180+

181+---+ tables

182+

183+|Orange|Apple|

184+|Bread|Pie|

185+|Butter|Ice cream|

186+

187+|*Orange*|*Apple*|

188+|Bread|Pie|

189+|*Butter*|Ice cream|

190+

191+|*Orange*|*Apple*|

192+|Bread%BR%%BR%and cheese|Pie%BR%%BR%*apple* and carrot|

193+

194+| Orange | Apple | more |

195+| Bread | Pie | more |

196+| Butter | Ice cream | and more |

197+

198+---+ macros

199+

200+%TEST%

201+

202+%TEST{}%

203+

204+%TEST{content with spaces}%

205+

206+%TEST{"content with spaces"}%

207+

208+%TEST{"content with spaces" ARG1="test"}%

209+

210+%TEST{content with spaces ARG1=test}%

211+

212+%TEST{ARG1=test content with spaces}%

213+

214+%TEST{ARG1=test ARG2=test2}%

215+

216+%TEST{ARG1="test" ARG2="test2"}%

217+

218+%TEST{ARG1="test"

219+ARG2="test2"

220+multiline

221+does also work}%