@@ -16,19 +16,52 @@ The pandoc AST format is defined in the module
1616](https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Definition.html).
1717
1818A "filter" is a program that modifies the AST, between the
19−reader and the writer:
19+reader and the writer.
2020
2121 INPUT --reader--> AST --filter--> AST --writer--> OUTPUT
2222
23−Filters are "pipes" that read from standard input and write to
24−standard output. They consume and produce a JSON representation
25−of the pandoc AST. (In recent versions, this representation
26−includes a `pandoc-api-version` field which refers to a
27−version of `pandoc-types`.) Filters may be written in any programming
28−language. To use a filter, you need only specify it on the
29−command line using `--filter`, e.g.
23+Pandoc supports two kinds of filters:
3024
31− pandoc -s input.txt --filter pandoc-citeproc -o output.htl
25+- **Lua filters** use the Lua language to
26+ define transformations on the pandoc AST. They are
27+ described in a [separate document](lua-filters.html).
28+
29+- **JSON filters**, described here, are pipes that read from
30+ standard input and write to standard output, consuming and
31+ producing a JSON representation of the pandoc AST:
32+
33+ source format
34+ ↓
35+ (pandoc)
36+ ↓
37+ JSON-formatted AST
38+ ↓
39+ (JSON filter)
40+ ↓
41+ JSON-formatted AST
42+ ↓
43+ (pandoc)
44+ ↓
45+ target format
46+
47+Lua filters have a couple of advantages. They use a Lua
48+interpreter that is embedded in pandoc, so you don't need
49+to have any external software installed. And they are
50+usually faster than JSON filters. But if you wish to
51+write your filter in a language other than Lua, you may
52+prefer to use a JSON filter. JSON filters may be written
53+in any programming language.
54+
55+You can use a JSON filter directly in a pipeline:
56+
57+ pandoc -s input.txt -t json | \
58+ pandoc-citeproc | \
59+ pandoc -s -f json -o output.html
60+
61+But it is more convenient to use the `--filter` option,
62+which handles the plumbing automatically:
63+
64+ pandoc -s input.txt --filter pandoc-citeproc -o output.html
3265
3366For a gentle introduction into writing your own filters,
3467continue this guide. There’s also a [list of third party filters
@@ -37,7 +70,7 @@ on the wiki](https://github.com/jgm/pandoc/wiki/Pandoc-Filters).
3770
3871# A simple example
3972
40−Suppose you wanted to replace all level 2+ headers in a markdown
73+Suppose you wanted to replace all level 2+ headings in a markdown
4174document with regular paragraphs, with text in italics. How would you go
4275about doing this?
4376
@@ -47,10 +80,10 @@ like this:
4780 perl -pe 's/^##+ (.*)$/\*\1\*/' source.txt
4881
4982This should work most of the time. But don't forget
50−that ATX style headers can end with a sequence of `#`s
51−that is not part of the header text:
83+that ATX style headings can end with a sequence of `#`s
84+that is not part of the heading text:
5285
53− ## My header ##
86+ ## My heading ##
5487
5588And what if your document contains a line starting with `##` in an HTML
5689comment or delimited code block?
@@ -60,14 +93,14 @@ comment or delimited code block?
6093 -->
6194
6295 ~~~~
63− ### A third level header in standard markdown
96+ ### A third level heading in standard markdown
6497 ~~~~
6598
66−We don't want to touch *these* lines. Moreover, what about setext
67−style second-level headers?
99+We don't want to touch *these* lines. Moreover, what about Setext
100+style second-level heading?
68101
69− A header
70− --------
102+ A heading
103+ ---------
71104
72105We need to handle those too. Finally, can we be sure that adding
73106asterisks to each side of our string will put it in italics?
@@ -76,25 +109,23 @@ end up with bold text, which is not what we want. And what if it contains
76109a regular unescaped asterisk?
77110
78111How would you modify your regular expression to handle these cases? It
79−would be hairy, to say the least. What we need is a real parser.
112+would be hairy, to say the least.
80113
81−Well, pandoc has a real markdown parser, the library function
82−`readMarkdown`. This transforms markdown text to an abstract syntax tree
83−(AST) that represents the document structure. Why not manipulate the
84−AST directly in a short Haskell script, then convert the result back to
85−markdown using `writeMarkdown`?
114+A better approach is to let pandoc handle the parsing, and
115+then modify the AST before the document is written. For this,
116+we can use a filter.
86117
87−First, let's see what this AST looks like. We can use pandoc's `native`
88−output format:
118+To see what sort of AST is produced when pandoc parses our text,
119+we can use pandoc's `native` output format:
89120
90121~~~~
91122% cat test.txt
92−## my header
123+## my heading
93124
94125text with *italics*
95126% pandoc -s -t native test.txt
96127Pandoc (Meta {unMeta = fromList []})
97−[Header 3 ("my-header",[],[]) [Str "My",Space,Str "header"]
128+[Header 2 ("my-heading",[],[]) [Str "My",Space,Str "heading"]
98129, Para [Str "text",Space,Str "with",Space,Emph [Str "italics"]] ]
99130~~~~
100131
@@ -106,136 +137,46 @@ the pandoc AST, see the [haddock documentation for `Text.Pandoc.Definition`].
106137
107138[haddock documentation for `Text.Pandoc.Definition`]: https://hackage.haskell.org/package/pandoc-types
108139
109−Here's a short Haskell script that reads markdown, changes level
110−2+ headers to regular paragraphs, and writes the result as markdown.
111−If you save it as `behead.hs`, you can run it using `runhaskell behead.hs`.
112−It will act like a unix pipe, reading from `stdin` and writing to `stdout`.
113−Or, if you want, you can compile it, using `ghc --make behead`, then run
114−the resulting executable `behead`.
115−
116−~~~~ {.haskell}
117−-- behead.hs
118−import Text.Pandoc
119−import Text.Pandoc.Walk (walk)
120−
121−behead :: Block -> Block
122−behead (Header n _ xs) | n >= 2 = Para [Emph xs]
123−behead x = x
124−
125−readDoc :: String -> Pandoc
126−readDoc s = readMarkdown def s
127−-- or, for pandoc 1.14 and greater, use:
128−-- readDoc s = case readMarkdown def s of
129−-- Right doc -> doc
130−-- Left err -> error (show err)
131−
132−writeDoc :: Pandoc -> String
133−writeDoc doc = writeMarkdown def doc
134−
135−main :: IO ()
136−main = interact (writeDoc . walk behead . readDoc)
137−~~~~
138−
139−The magic here is the `walk` function, which converts
140−our `behead` function (a function from `Block` to `Block`) to
141−a transformation on whole `Pandoc` documents.
142−(See the [haddock documentation for `Text.Pandoc.Walk`].)
143−
144−[haddock documentation for `Text.Pandoc.Walk`]: https://hackage.haskell.org/package/pandoc-types
145−
146−# Queries: listing URLs
147−
148−We can use this same technique to do much more complex transformations
149−and queries. Here's how we could extract all the URLs linked to in
150−a markdown document (again, not an easy task with regular expressions):
151−
152−~~~~ {.haskell}
153−-- extracturls.hs
154−import Text.Pandoc
155−
156−extractURL :: Inline -> [String]
157−extractURL (Link _ _ (u,_)) = [u]
158−extractURL (Image _ _ (u,_)) = [u]
159−extractURL _ = []
160−
161−extractURLs :: Pandoc -> [String]
162−extractURLs = query extractURL
163−
164−readDoc :: String -> Pandoc
165−readDoc = readMarkdown def
166−-- or, for pandoc 1.14, use:
167−-- readDoc s = case readMarkdown def s of
168−-- Right doc -> doc
169−-- Left err -> error (show err)
170−
171−main :: IO ()
172−main = interact (unlines . extractURLs . readDoc)
173−~~~~
174−
175−`query` is the query counterpart of `walk`: it lifts
176−a function that operates on `Inline` elements to one that operates
177−on the whole `Pandoc` AST. The results returned by applying
178−`extractURL` to each `Inline` element are concatenated in the
179−result.
180−
181−# JSON filters
182−
183−`behead.hs` is a very special-purpose program. It reads a
184−specific input format (markdown) and writes a specific output format
185−(HTML), with a specific set of options (here, the defaults).
186−But the basic operation it performs is one that would be useful
187−in many document transformations. It would be nice to isolate the
188−part of the program that transforms the pandoc AST, leaving the rest
189−to pandoc itself. What we want is a *filter* that *just* operates
190−on the AST---or rather, on a JSON representation of the AST that
191−pandoc can produce and consume:
192−
193− source format
194− ↓
195− (pandoc)
196− ↓
197− JSON-formatted AST
198− ↓
199− (filter)
200− ↓
201− JSON-formatted AST
202− ↓
203− (pandoc)
204− ↓
205− target format
206−
207−The module `Text.Pandoc.JSON` (from `pandoc-types`) contains a
208−function `toJSONFilter` that makes it easy to write such
209−filters. Here is a filter version of `behead.hs`:
140+We can use Haskell to create a JSON filter that transforms this
141+AST, replacing each `Header` block with level >= 2 with a `Para`
142+with its contents wrapped inside an `Emph` inline:
210143
211144~~~~ {.haskell}
212145#!/usr/bin/env runhaskell
213−-- behead2.hs
146+-- behead.hs
214147import Text.Pandoc.JSON
215148
216149main :: IO ()
217150main = toJSONFilter behead
218− where behead (Header n _ xs) | n >= 2 = Para [Emph xs]
219− behead x = x
151+
152+behead :: Block -> Block
153+behead (Header n _ xs) | n >= 2 = Para [Emph xs]
154+behead x = x
220155~~~~
221156
222−It can be used this way:
157+The `toJSONFilter` function does two things. First, it lifts
158+the `behead` function (which maps `Block -> Block`) onto a
159+transformation of the entire `Pandoc` AST, walking the AST
160+and transforming each block. Second, it wraps this `Pandoc ->
161+Pandoc` transformation with the necessary JSON serialization
162+and deserialization, producing an executable that consumes
163+JSON from stdin and produces JSON to stdout.
223164
224− pandoc -f SOURCEFORMAT -t json | runhaskell behead2.hs | \
225− pandoc -f json -t TARGETFORMAT
165+To use the filter, make it executable:
226166
227−But it is easier to use the `--filter` option with pandoc:
167+ chmod +x behead.hs
228168
229− pandoc -f SOURCEFORMAT -t TARGETFORMAT --filter ./behead2.hs
169+and then
230170
231−Note that this approach requires that `behead2.hs` be executable,
232−so we must
171+ pandoc -f SOURCEFORMAT -t TARGETFORMAT --filter ./behead.hs
233172
234− chmod +x behead2.hs
173+(It is also necessary that `pandoc-types` be installed in the
174+local package repository: `cabal install pandoc-types` should
175+ensure this.)
235176
236177Alternatively, we could compile the filter:
237178
238− ghc --make behead2.hs
179+ ghc --make behead.hs
239180 pandoc -f SOURCEFORMAT -t TARGETFORMAT --filter ./behead
240181
241182Note that if the filter is placed in the system PATH, then the initial
@@ -243,6 +184,7 @@ Note that if the filter is placed in the system PATH, then the initial
243184multiple instances of `--filter`: the filters will be applied in
244185sequence.
245186
187+
246188# LaTeX for WordPress
247189
248190Another easy example. WordPress blogs require a special format for
@@ -283,7 +225,7 @@ Here's our "beheading" filter in python:
283225#!/usr/bin/env python
284226
285227"""
286−Pandoc filter to convert all level 2+ headers to paragraphs with
228+Pandoc filter to convert all level 2+ headings to paragraphs with
287229emphasized text.
288230"""
289231