git.iko.soy/typeable/pandoc
on master help sign in
Creole reader (#4002)

* Basic skeleton for creole reader. No real functionality besides preliminary bold and italics yet. * Creole: add support for bold/italic with implicit end at paragraph end. * Creole: add support for headings. * Creole: add support for tilde escaped chars. * Basic skeleton for creole reader. No real functionality besides preliminary bold and italics yet. * Creole: add support for bold/italic with implicit end at paragraph end. * Creole: add support for headings. * Creole: add support for tilde escaped chars. * Add a test suite for the creole parser So far this covers only things the parser already supports. * Added simple parsing of flat unordered lists. * Added tests for unordered lists in creole. * First, wrong(!) implementation of sublists. Fails test, as sublists should not be embedded in a list item! * Implementation of unordered sublists. * Added support for ordered lists to creole reader. * Added utility function to append parsers to Creole reader. * Creole reader: Fixed list item end detection in sub lists. * Tests for creole reader: added more tests for lists. Covering ordered and unordered tests, even mixed. Tests for formatting in list items still missing... * Added "nowiki" blocks. One exception rule is missing... * Creole reader: nowiki: implemented exception for curly brackets. * Creole reader: added inline nowiki. * Creole reader: added horizontalRule. * Creole reader: added auto linking of URIs. * Creole reader: detect horizontalRule as para end. Used the opportunity for a little refactoring. * Creole reader: added forced line breaks. Including test. * Creole reader: implement wiki links. * Creole reader: added image support. * Creole reader: support images as links. * Creole reader: implemented placeholder -- by simply dropping them. * Creole reader: added tests for links. After observing a regression, it was really time... ;-) * Creole reader: fixed links with names. * Creole reader: allow space after first of enclosing tags. Space after the start of formatting tags are allowed with creole, e.g. "there is // italic text // in here" is legal. This problem was discovered using the creole1.0test.txt document from http://www.wikicreole.org/wiki/Creole1.0TestCases See l.57: # // italic item 3 // * Creole reader: fixed links without names. * Creole reader: Tests, sorted into groups. * Creole reader: implemented tables. * Removed redundant import. * Creole reader: add correct escaping of links. * Creole reader: allow handling of e.g. links in parenthesis and quotes. * Creole reader: Modified disclaimer as most of the code is actually by me. * Creole reader: Tests: added escaped links. * Creole reader: preserve leading and trailing space in bold/italic. * Creole reader: detect tables without a leading blank line. * Creole Reader: added official creole1.0test.txt as "old" test. The base document was downloaded from http://www.wikicreole.org/wiki/Creole1.0TestCases. The Wiki, and therefore the test document is Copyright (C) by the contributors. Some rights reserved, license CC BY-SA. http://creativecommons.org/licenses/by-sa/1.0/

commite8be4b0b6d47f74a74a59d9cc49b68a40de314c9 authorSascha Wilde <wilde@sha-bang.de> dateSun 29 Oct 2017, 18:28 (9 years ago) parentd06bbee
3 files changed+236
compare with master
unifiedsplit
test/Tests/Old.hs+4 test/creole-reader.native+95 test/creole-reader.txt+137
test/Tests/Old.hs+4
@@ -156,6 +156,10 @@ tests = [ testGroup "markdown"
156156 , testGroup "ms"
157157 [ testGroup "writer" $ writerTests "ms"
158158 ]
159+ , testGroup "creole"
160+ [ test "reader" ["-r", "creole", "-w", "native", "-s"]
161+ "creole-reader.txt" "creole-reader.native"
162+ ]
159163 ]
160164
161165-- makes sure file is fully closed after reading
test/creole-reader.nativenew file+95
@@ -0,0 +1,95 @@
1+Pandoc (Meta {unMeta = fromList []})
2+[Header 1 ("",[],[]) [Str "Top-level heading (1)"]
3+,Header 2 ("",[],[]) [Str "This a test for creole 0.1 (2)"]
4+,Header 3 ("",[],[]) [Str "This is a Subheading (3)"]
5+,Header 4 ("",[],[]) [Str "Subsub (4)"]
6+,Header 5 ("",[],[]) [Str "Subsubsub (5)"]
7+,Para [Str "The",Space,Str "ending",Space,Str "equal",Space,Str "signs",Space,Str "should",Space,Str "not",Space,Str "be",Space,Str "displayed:"]
8+,Header 1 ("",[],[]) [Str "Top-level heading (1)"]
9+,Header 2 ("",[],[]) [Str "This a test for creole 0.1 (2)"]
10+,Header 3 ("",[],[]) [Str "This is a Subheading (3)"]
11+,Header 4 ("",[],[]) [Str "Subsub (4)"]
12+,Header 5 ("",[],[]) [Str "Subsubsub (5)"]
13+,Para [Str "You",Space,Str "can",Space,Str "make",Space,Str "things",Space,Strong [Str "bold"],Space,Str "or",Space,Emph [Str "italic"],Space,Str "or",Space,Strong [Emph [Str "both"]],Space,Str "or",Space,Emph [Strong [Str "both"]],Str "."]
14+,Para [Str "Character",Space,Str "formatting",Space,Str "extends",Space,Str "across",Space,Str "line",Space,Str "breaks:",Space,Strong [Str "bold,",Space,Str "this",Space,Str "is",Space,Str "still",Space,Str "bold.",Space,Str "This",Space,Str "line",Space,Str "deliberately",Space,Str "does",Space,Str "not",Space,Str "end",Space,Str "in",Space,Str "star-star."]]
15+,Para [Str "Not",Space,Str "bold.",Space,Str "Character",Space,Str "formatting",Space,Str "does",Space,Str "not",Space,Str "cross",Space,Str "paragraph",Space,Str "boundaries."]
16+,Para [Str "You",Space,Str "can",Space,Str "use",Space,Link ("",[],[]) [Str "internal links"] ("internal links",""),Space,Str "or",Space,Link ("",[],[]) [Str "external links"] ("http://www.wikicreole.org",""),Str ",",Space,Str "give",Space,Str "the",Space,Str "link",Space,Str "a",Space,Link ("",[],[]) [Str "different"] ("internal links",""),Space,Str "name."]
17+,Para [Str "Here's",Space,Str "another",Space,Str "sentence:",Space,Str "This",Space,Str "wisdom",Space,Str "is",Space,Str "taken",Space,Str "from",Space,Link ("",[],[]) [Str "Ward Cunningham's"] ("Ward Cunningham's",""),Space,Link ("",[],[]) [Str "Presentation at the Wikisym 06"] ("http://www.c2.com/doc/wikisym/WikiSym2006.pdf",""),Str "."]
18+,Para [Str "Here's",Space,Str "a",Space,Str "external",Space,Str "link",Space,Str "without",Space,Str "a",Space,Str "description:",Space,Link ("",[],[]) [Str "http://www.wikicreole.org"] ("http://www.wikicreole.org","")]
19+,Para [Str "Be",Space,Str "careful",Space,Str "that",Space,Str "italic",Space,Str "links",Space,Str "are",Space,Str "rendered",Space,Str "properly:",Space,Emph [Link ("",[],[]) [Str "My Book Title"] ("http://my.book.example/","")]]
20+,Para [Str "Free",Space,Str "links",Space,Str "without",Space,Str "braces",Space,Str "should",Space,Str "be",Space,Str "rendered",Space,Str "as",Space,Str "well,",Space,Str "like",Space,Link ("",[],[]) [Str "http://www.wikicreole.org/"] ("http://www.wikicreole.org/",""),Space,Str "and",Space,Link ("",[],[]) [Str "http://www.wikicreole.org/users/~example"] ("http://www.wikicreole.org/users/~example",""),Str "."]
21+,Para [Str "Creole1.0",Space,Str "specifies",Space,Str "that",Space,Link ("",[],[]) [Str "http://bar"] ("http://bar",""),Space,Str "and",Space,Link ("",[],[]) [Str "ftp://bar"] ("ftp://bar",""),Space,Str "should",Space,Str "not",Space,Str "render",Space,Str "italic,",Space,Str "something",Space,Str "like",Space,Str "foo:",Emph [Str "bar",Space,Str "should",Space,Str "render",Space,Str "as",Space,Str "italic."]]
22+,Para [Str "You",Space,Str "can",Space,Str "use",Space,Str "this",Space,Str "to",Space,Str "draw",Space,Str "a",Space,Str "line",Space,Str "to",Space,Str "separate",Space,Str "the",Space,Str "page:"]
23+,HorizontalRule
24+,Para [Str "You",Space,Str "can",Space,Str "use",Space,Str "lists,",Space,Str "start",Space,Str "it",Space,Str "at",Space,Str "the",Space,Str "first",Space,Str "column",Space,Str "for",Space,Str "now,",Space,Str "please..."]
25+,Para [Str "unnumbered",Space,Str "lists",Space,Str "are",Space,Str "like"]
26+,BulletList
27+ [[Plain [Str "item",Space,Str "a"]]
28+ ,[Plain [Str "item",Space,Str "b"]]
29+ ,[Plain [Strong [Str "bold",Space,Str "item",Space,Str "c"]]]]
30+,Para [Str "blank",Space,Str "space",Space,Str "is",Space,Str "also",Space,Str "permitted",Space,Str "before",Space,Str "lists",Space,Str "like:"]
31+,BulletList
32+ [[Plain [Str "item",Space,Str "a"]]
33+ ,[Plain [Str "item",Space,Str "b"]]
34+ ,[Plain [Str "item",Space,Str "c"]
35+ ,BulletList
36+ [[Plain [Str "item",Space,Str "c.a"]]]]]
37+,Para [Str "or",Space,Str "you",Space,Str "can",Space,Str "number",Space,Str "them"]
38+,OrderedList (1,DefaultStyle,DefaultDelim)
39+ [[Plain [Link ("",[],[]) [Str "item 1"] ("item 1","")]]
40+ ,[Plain [Str "item",Space,Str "2"]]
41+ ,[Plain [Emph [Space,Str "italic",Space,Str "item",Space,Str "3",Space]]
42+ ,OrderedList (1,DefaultStyle,DefaultDelim)
43+ [[Plain [Str "item",Space,Str "3.1"]]
44+ ,[Plain [Str "item",Space,Str "3.2"]]]]]
45+,Para [Str "up",Space,Str "to",Space,Str "five",Space,Str "levels"]
46+,BulletList
47+ [[Plain [Str "1"]
48+ ,BulletList
49+ [[Plain [Str "2"]
50+ ,BulletList
51+ [[Plain [Str "3"]
52+ ,BulletList
53+ [[Plain [Str "4"]
54+ ,BulletList
55+ [[Plain [Str "5"]]]]]]]]]]]
56+,BulletList
57+ [[Plain [Str "You",Space,Str "can",Space,Str "have",Space,Str "multiline",Space,Str "list",Space,Str "items"]]
58+ ,[Plain [Str "this",Space,Str "is",Space,Str "a",Space,Str "second",Space,Str "multiline",Space,Str "list",Space,Str "item"]]]
59+,Para [Str "You",Space,Str "can",Space,Str "use",Space,Str "nowiki",Space,Str "syntax",Space,Str "if",Space,Str "you",Space,Str "would",Space,Str "like",Space,Str "do",Space,Str "stuff",Space,Str "like",Space,Str "this:"]
60+,CodeBlock ("",[],[]) "Guitar Chord C:\n\n||---|---|---|\n||-0-|---|---|\n||---|---|---|\n||---|-0-|---|\n||---|---|-0-|\n||---|---|---|"
61+,Para [Str "You",Space,Str "can",Space,Str "also",Space,Str "use",Space,Str "it",Space,Str "inline",Space,Str "nowiki",Space,Code ("",[],[]) " in a sentence ",Space,Str "like",Space,Str "this."]
62+,Header 1 ("",[],[]) [Str "Escapes"]
63+,Para [Str "Normal",Space,Str "Link:",Space,Link ("",[],[]) [Str "http://wikicreole.org/"] ("http://wikicreole.org/",""),Space,Str "-",Space,Str "now",Space,Str "same",Space,Str "link,",Space,Str "but",Space,Str "escaped:",Space,Str "http://wikicreole.org/"]
64+,Para [Str "Normal",Space,Str "asterisks:",Space,Str "**not",Space,Str "bold**"]
65+,Para [Str "a",Space,Str "tilde",Space,Str "alone:",Space,Str "~"]
66+,Para [Str "a",Space,Str "tilde",Space,Str "escapes",Space,Str "itself:",Space,Str "~xxx"]
67+,Header 3 ("",[],[]) [Str "Creole 0.2"]
68+,Para [Str "This",Space,Str "should",Space,Str "be",Space,Str "a",Space,Str "flower",Space,Str "with",Space,Str "the",Space,Str "ALT",Space,Str "text",Space,Str "\"this",Space,Str "is",Space,Str "a",Space,Str "flower\"",Space,Str "if",Space,Str "your",Space,Str "wiki",Space,Str "supports",Space,Str "ALT",Space,Str "text",Space,Str "on",Space,Str "images:"]
69+,Para [Image ("",[],[]) [Str "here is a red flower"] ("Red-Flower.jpg","")]
70+,Header 3 ("",[],[]) [Str "Creole 0.4"]
71+,Para [Str "Tables",Space,Str "are",Space,Str "done",Space,Str "like",Space,Str "this:"]
72+,Table [] [AlignDefault,AlignDefault] [0.0,0.0]
73+ [[Plain [Str "header",Space,Str "col1"]]
74+ ,[Plain [Str "header",Space,Str "col2"]]]
75+ [[[Plain [Str "col1"]]
76+ ,[Plain [Str "col2"]]]
77+ ,[[Plain [Str "you"]]
78+ ,[Plain [Str "can"]]]
79+ ,[[Plain [Str "also"]]
80+ ,[Plain [Str "align",LineBreak,Str "it."]]]]
81+,Para [Str "You",Space,Str "can",Space,Str "format",Space,Str "an",Space,Str "address",Space,Str "by",Space,Str "simply",Space,Str "forcing",Space,Str "linebreaks:"]
82+,Para [Str "My",Space,Str "contact",Space,Str "dates:",LineBreak,Str "Pone:",Space,Str "xyz",LineBreak,Str "Fax:",Space,Str "+45",LineBreak,Str "Mobile:",Space,Str "abc"]
83+,Header 3 ("",[],[]) [Str "Creole 0.5"]
84+,Table [] [AlignDefault,AlignDefault] [0.0,0.0]
85+ [[Plain [Str "Header",Space,Str "title"]]
86+ ,[Plain [Str "Another",Space,Str "header",Space,Str "title"]]]
87+ [[[Plain [Code ("",[],[]) " //not italic text// "]]
88+ ,[Plain [Code ("",[],[]) " **not bold text** "]]]
89+ ,[[Plain [Emph [Str "italic",Space,Str "text"]]]
90+ ,[Plain [Strong [Space,Str "bold",Space,Str "text",Space]]]]]
91+,Header 3 ("",[],[]) [Str "Creole 1.0"]
92+,Para [Str "If",Space,Str "interwiki",Space,Str "links",Space,Str "are",Space,Str "setup",Space,Str "in",Space,Str "your",Space,Str "wiki,",Space,Str "this",Space,Str "links",Space,Str "to",Space,Str "the",Space,Str "WikiCreole",Space,Str "page",Space,Str "about",Space,Str "Creole",Space,Str "1.0",Space,Str "test",Space,Str "cases:",Space,Link ("",[],[]) [Str "WikiCreole:Creole1.0TestCases"] ("WikiCreole:Creole1.0TestCases",""),Str "."]
93+,HorizontalRule
94+,Para [Str "The",Space,Str "above",Space,Str "test",Space,Str "document",Space,Str "was",Space,Str "found",Space,Str "on",Space,Link ("",[],[]) [Str "http://www.wikicreole.org/wiki/Creole1.0TestCases"] ("http://www.wikicreole.org/wiki/Creole1.0TestCases",""),Space,Str "and",Space,Str "downloaded",Space,Str "from",Space,Link ("",[],[]) [Str "http://www.wikicreole.org/attach/Creole1.0TestCases/creole1.0test.txt"] ("http://www.wikicreole.org/attach/Creole1.0TestCases/creole1.0test.txt",""),Str "."]
95+,Para [Str "The",Space,Str "Creole",Space,Str "Wiki",Space,Str "is",Space,Str "licensed:",Space,Str "Copyright",Space,Str "(C)",Space,Str "by",Space,Str "the",Space,Str "contributors.",Space,Str "Some",Space,Str "rights",Space,Str "reserved,",Space,Str "license",Space,Link ("",[],[]) [Str "https://creativecommons.org/licenses/by-sa/1.0/"] ("BY-SA",""),Str "."]]
test/creole-reader.txtnew file+137
@@ -0,0 +1,137 @@
1+= Top-level heading (1)
2+== This a test for creole 0.1 (2)
3+=== This is a Subheading (3)
4+==== Subsub (4)
5+===== Subsubsub (5)
6+
7+The ending equal signs should not be displayed:
8+
9+= Top-level heading (1) =
10+== This a test for creole 0.1 (2) ==
11+=== This is a Subheading (3) ===
12+==== Subsub (4) ====
13+===== Subsubsub (5) =====
14+
15+
16+You can make things **bold** or //italic// or **//both//** or //**both**//.
17+
18+Character formatting extends across line breaks: **bold,
19+this is still bold. This line deliberately does not end in star-star.
20+
21+Not bold. Character formatting does not cross paragraph boundaries.
22+
23+You can use [[internal links]] or [[http://www.wikicreole.org|external links]],
24+give the link a [[internal links|different]] name.
25+
26+Here's another sentence: This wisdom is taken from [[Ward Cunningham's]]
27+[[http://www.c2.com/doc/wikisym/WikiSym2006.pdf|Presentation at the Wikisym 06]].
28+
29+Here's a external link without a description: [[http://www.wikicreole.org]]
30+
31+Be careful that italic links are rendered properly: //[[http://my.book.example/|My Book Title]]//
32+
33+Free links without braces should be rendered as well, like http://www.wikicreole.org/ and http://www.wikicreole.org/users/~example.
34+
35+Creole1.0 specifies that http://bar and ftp://bar should not render italic,
36+something like foo://bar should render as italic.
37+
38+You can use this to draw a line to separate the page:
39+----
40+
41+You can use lists, start it at the first column for now, please...
42+
43+unnumbered lists are like
44+* item a
45+* item b
46+* **bold item c**
47+
48+blank space is also permitted before lists like:
49+ * item a
50+ * item b
51+* item c
52+ ** item c.a
53+
54+or you can number them
55+# [[item 1]]
56+# item 2
57+# // italic item 3 //
58+ ## item 3.1
59+ ## item 3.2
60+
61+up to five levels
62+* 1
63+** 2
64+*** 3
65+**** 4
66+***** 5
67+
68+* You can have
69+multiline list items
70+* this is a second multiline
71+list item
72+
73+You can use nowiki syntax if you would like do stuff like this:
74+
75+{{{
76+Guitar Chord C:
77+
78+||---|---|---|
79+||-0-|---|---|
80+||---|---|---|
81+||---|-0-|---|
82+||---|---|-0-|
83+||---|---|---|
84+}}}
85+
86+You can also use it inline nowiki {{{ in a sentence }}} like this.
87+
88+= Escapes =
89+Normal Link: http://wikicreole.org/ - now same link, but escaped: ~http://wikicreole.org/
90+
91+Normal asterisks: ~**not bold~**
92+
93+a tilde alone: ~
94+
95+a tilde escapes itself: ~~xxx
96+
97+=== Creole 0.2 ===
98+
99+This should be a flower with the ALT text "this is a flower" if your wiki supports ALT text on images:
100+
101+{{Red-Flower.jpg|here is a red flower}}
102+
103+=== Creole 0.4 ===
104+
105+Tables are done like this:
106+
107+|=header col1|=header col2|
108+|col1|col2|
109+|you |can |
110+|also |align\\ it. |
111+
112+You can format an address by simply forcing linebreaks:
113+
114+My contact dates:\\
115+Pone: xyz\\
116+Fax: +45\\
117+Mobile: abc
118+
119+=== Creole 0.5 ===
120+
121+|= Header title |= Another header title |
122+| {{{ //not italic text// }}} | {{{ **not bold text** }}} |
123+| //italic text// | ** bold text ** |
124+
125+=== Creole 1.0 ===
126+
127+If interwiki links are setup in your wiki, this links to the WikiCreole page about Creole 1.0 test cases: [[WikiCreole:Creole1.0TestCases]].
128+
129+ ----
130+
131+The above test document was found on
132+http://www.wikicreole.org/wiki/Creole1.0TestCases and downloaded from
133+http://www.wikicreole.org/attach/Creole1.0TestCases/creole1.0test.txt.
134+
135+The Creole Wiki is licensed: Copyright (C) by the contributors. Some
136+rights reserved, license
137+[[BY-SA|https://creativecommons.org/licenses/by-sa/1.0/]].