@@ -16,19 +16,52 @@ The pandoc AST format is defined in the module
16](https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Definition.html).
16](https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Definition.html).
18A "filter" is a program that modifies the AST, between the
18A "filter" is a program that modifies the AST, between the
19−reader and the writer:
19+reader and the writer.
21 INPUT --reader--> AST --filter--> AST --writer--> OUTPUT
21 INPUT --reader--> AST --filter--> AST --writer--> OUTPUT
23−Filters are "pipes" that read from standard input and write to
23+Pandoc supports two kinds of filters:
24−standard output. They consume and produce a JSON representation
25−of the pandoc AST. (In recent versions, this representation
26−includes a `pandoc-api-version` field which refers to a
27−version of `pandoc-types`.) Filters may be written in any programming
28−language. To use a filter, you need only specify it on the
29−command line using `--filter`, e.g.
25+- **Lua filters** use the Lua language to
26+ define transformations on the pandoc AST. They are
27+ described in a [separate document](lua-filters.html).
29+- **JSON filters**, described here, are pipes that read from
30+ standard input and write to standard output, consuming and
31+ producing a JSON representation of the pandoc AST:
47+Lua filters have a couple of advantages. They use a Lua
48+interpreter that is embedded in pandoc, so you don't need
49+to have any external software installed. And they are
50+usually faster than JSON filters. But if you wish to
51+write your filter in a language other than Lua, you may
52+prefer to use a JSON filter. JSON filters may be written
53+in any programming language.
55+You can use a JSON filter directly in a pipeline:
31− pandoc -s input.txt --filter pandoc-citeproc -o output.htl
57+ pandoc -s input.txt -t json | \
59+ pandoc -s -f json -o output.html
61+But it is more convenient to use the `--filter` option,
62+which handles the plumbing automatically:
64+ pandoc -s input.txt --filter pandoc-citeproc -o output.html
33For a gentle introduction into writing your own filters,
66For a gentle introduction into writing your own filters,
34continue this guide. There’s also a [list of third party filters
67continue this guide. There’s also a [list of third party filters
@@ -37,7 +70,7 @@ on the wiki](https://github.com/jgm/pandoc/wiki/Pandoc-Filters).
38# A simple example
71# A simple example
40−Suppose you wanted to replace all level 2+ headers in a markdown
73+Suppose you wanted to replace all level 2+ headings in a markdown
41document with regular paragraphs, with text in italics. How would you go
74document with regular paragraphs, with text in italics. How would you go
42about doing this?
75about doing this?
@@ -47,10 +80,10 @@ like this:
47 perl -pe 's/^##+ (.*)$/\*\1\*/' source.txt
80 perl -pe 's/^##+ (.*)$/\*\1\*/' source.txt
49This should work most of the time. But don't forget
82This should work most of the time. But don't forget
50−that ATX style headers can end with a sequence of `#`s
83+that ATX style headings can end with a sequence of `#`s
51−that is not part of the header text:
84+that is not part of the heading text:
53− ## My header ##
86+ ## My heading ##
55And what if your document contains a line starting with `##` in an HTML
88And what if your document contains a line starting with `##` in an HTML
56comment or delimited code block?
89comment or delimited code block?
@@ -60,14 +93,14 @@ comment or delimited code block?
63− ### A third level header in standard markdown
96+ ### A third level heading in standard markdown
66−We don't want to touch *these* lines. Moreover, what about setext
99+We don't want to touch *these* lines. Moreover, what about Setext
67−style second-level headers?
100+style second-level heading?
69− A header
102+ A heading
70− --------
103+ ---------
72We need to handle those too. Finally, can we be sure that adding
105We need to handle those too. Finally, can we be sure that adding
73asterisks to each side of our string will put it in italics?
106asterisks to each side of our string will put it in italics?
@@ -76,25 +109,23 @@ end up with bold text, which is not what we want. And what if it contains
76a regular unescaped asterisk?
109a regular unescaped asterisk?
78How would you modify your regular expression to handle these cases? It
111How would you modify your regular expression to handle these cases? It
79−would be hairy, to say the least. What we need is a real parser.
112+would be hairy, to say the least.
81−Well, pandoc has a real markdown parser, the library function
114+A better approach is to let pandoc handle the parsing, and
82−`readMarkdown`. This transforms markdown text to an abstract syntax tree
115+then modify the AST before the document is written. For this,
83−(AST) that represents the document structure. Why not manipulate the
116+we can use a filter.
84−AST directly in a short Haskell script, then convert the result back to
85−markdown using `writeMarkdown`?
87−First, let's see what this AST looks like. We can use pandoc's `native`
118+To see what sort of AST is produced when pandoc parses our text,
88−output format:
119+we can use pandoc's `native` output format:
91% cat test.txt
122% cat test.txt
92−## my header
123+## my heading
94text with *italics*
125text with *italics*
95% pandoc -s -t native test.txt
126% pandoc -s -t native test.txt
96Pandoc (Meta {unMeta = fromList []})
127Pandoc (Meta {unMeta = fromList []})
97−[Header 3 ("my-header",[],[]) [Str "My",Space,Str "header"]
128+[Header 2 ("my-heading",[],[]) [Str "My",Space,Str "heading"]
98, Para [Str "text",Space,Str "with",Space,Emph [Str "italics"]] ]
129, Para [Str "text",Space,Str "with",Space,Emph [Str "italics"]] ]
@@ -106,136 +137,46 @@ the pandoc AST, see the [haddock documentation for `Text.Pandoc.Definition`].
107[haddock documentation for `Text.Pandoc.Definition`]: https://hackage.haskell.org/package/pandoc-types
138[haddock documentation for `Text.Pandoc.Definition`]: https://hackage.haskell.org/package/pandoc-types
109−Here's a short Haskell script that reads markdown, changes level
140+We can use Haskell to create a JSON filter that transforms this
110−2+ headers to regular paragraphs, and writes the result as markdown.
141+AST, replacing each `Header` block with level >= 2 with a `Para`
111−If you save it as `behead.hs`, you can run it using `runhaskell behead.hs`.
142+with its contents wrapped inside an `Emph` inline:
112−It will act like a unix pipe, reading from `stdin` and writing to `stdout`.
113−Or, if you want, you can compile it, using `ghc --make behead`, then run
114−the resulting executable `behead`.
119−import Text.Pandoc.Walk (walk)
121−behead :: Block -> Block
122−behead (Header n _ xs) | n >= 2 = Para [Emph xs]
125−readDoc :: String -> Pandoc
126−readDoc s = readMarkdown def s
127−-- or, for pandoc 1.14 and greater, use:
128−-- readDoc s = case readMarkdown def s of
130−-- Left err -> error (show err)
132−writeDoc :: Pandoc -> String
133−writeDoc doc = writeMarkdown def doc
136−main = interact (writeDoc . walk behead . readDoc)
139−The magic here is the `walk` function, which converts
140−our `behead` function (a function from `Block` to `Block`) to
141−a transformation on whole `Pandoc` documents.
142−(See the [haddock documentation for `Text.Pandoc.Walk`].)
144−[haddock documentation for `Text.Pandoc.Walk`]: https://hackage.haskell.org/package/pandoc-types
146−# Queries: listing URLs
148−We can use this same technique to do much more complex transformations
149−and queries. Here's how we could extract all the URLs linked to in
150−a markdown document (again, not an easy task with regular expressions):
156−extractURL :: Inline -> [String]
157−extractURL (Link _ _ (u,_)) = [u]
158−extractURL (Image _ _ (u,_)) = [u]
161−extractURLs :: Pandoc -> [String]
162−extractURLs = query extractURL
164−readDoc :: String -> Pandoc
165−readDoc = readMarkdown def
166−-- or, for pandoc 1.14, use:
167−-- readDoc s = case readMarkdown def s of
169−-- Left err -> error (show err)
172−main = interact (unlines . extractURLs . readDoc)
175−`query` is the query counterpart of `walk`: it lifts
176−a function that operates on `Inline` elements to one that operates
177−on the whole `Pandoc` AST. The results returned by applying
178−`extractURL` to each `Inline` element are concatenated in the
183−`behead.hs` is a very special-purpose program. It reads a
184−specific input format (markdown) and writes a specific output format
185−(HTML), with a specific set of options (here, the defaults).
186−But the basic operation it performs is one that would be useful
187−in many document transformations. It would be nice to isolate the
188−part of the program that transforms the pandoc AST, leaving the rest
189−to pandoc itself. What we want is a *filter* that *just* operates
190−on the AST---or rather, on a JSON representation of the AST that
191−pandoc can produce and consume:
207−The module `Text.Pandoc.JSON` (from `pandoc-types`) contains a
208−function `toJSONFilter` that makes it easy to write such
209−filters. Here is a filter version of `behead.hs`:
211~~~~ {.haskell}
144~~~~ {.haskell}
212#!/usr/bin/env runhaskell
145#!/usr/bin/env runhaskell
213−-- behead2.hs
146+-- behead.hs
214import Text.Pandoc.JSON
147import Text.Pandoc.JSON
216main :: IO ()
149main :: IO ()
217main = toJSONFilter behead
150main = toJSONFilter behead
152+behead :: Block -> Block
218− where behead (Header n _ xs) | n >= 2 = Para [Emph xs]
153+behead (Header n _ xs) | n >= 2 = Para [Emph xs]
219− behead x = x
154+behead x = x
222−It can be used this way:
157+The `toJSONFilter` function does two things. First, it lifts
158+the `behead` function (which maps `Block -> Block`) onto a
159+transformation of the entire `Pandoc` AST, walking the AST
160+and transforming each block. Second, it wraps this `Pandoc ->
161+Pandoc` transformation with the necessary JSON serialization
162+and deserialization, producing an executable that consumes
163+JSON from stdin and produces JSON to stdout.
224− pandoc -f SOURCEFORMAT -t json | runhaskell behead2.hs | \
165+To use the filter, make it executable:
225− pandoc -f json -t TARGETFORMAT
227−But it is easier to use the `--filter` option with pandoc:
167+ chmod +x behead.hs
229− pandoc -f SOURCEFORMAT -t TARGETFORMAT --filter ./behead2.hs
169+and then
231−Note that this approach requires that `behead2.hs` be executable,
171+ pandoc -f SOURCEFORMAT -t TARGETFORMAT --filter ./behead.hs
234− chmod +x behead2.hs
173+(It is also necessary that `pandoc-types` be installed in the
174+local package repository: `cabal install pandoc-types` should
236Alternatively, we could compile the filter:
177Alternatively, we could compile the filter:
238− ghc --make behead2.hs
179+ ghc --make behead.hs
239 pandoc -f SOURCEFORMAT -t TARGETFORMAT --filter ./behead
180 pandoc -f SOURCEFORMAT -t TARGETFORMAT --filter ./behead
241Note that if the filter is placed in the system PATH, then the initial
182Note that if the filter is placed in the system PATH, then the initial
@@ -243,6 +184,7 @@ Note that if the filter is placed in the system PATH, then the initial
243multiple instances of `--filter`: the filters will be applied in
184multiple instances of `--filter`: the filters will be applied in
246# LaTeX for WordPress
188# LaTeX for WordPress
248Another easy example. WordPress blogs require a special format for
190Another easy example. WordPress blogs require a special format for
@@ -283,7 +225,7 @@ Here's our "beheading" filter in python:
283#!/usr/bin/env python
225#!/usr/bin/env python
286−Pandoc filter to convert all level 2+ headers to paragraphs with
228+Pandoc filter to convert all level 2+ headings to paragraphs with
287emphasized text.
229emphasized text.