@@ -0,0 +1,469 @@
6+Pandoc provides an interface for users to write programs (known
7+as filters) which act on pandoc’s AST.
9+Pandoc consists of a set of readers and writers. When converting
10+a document from one format to another, text is parsed by a
11+reader into pandoc’s intermediate representation of the
12+document---an "abstract syntax tree" or AST---which is then
13+converted by the writer into the target format.
14+The pandoc AST format is defined in the module
15+`Text.Pandoc.Definition` in
16+[pandoc-types](https://hackage.haskell.org/package/pandoc-types).
18+A "filter" is a program that modifies the AST, between the
19+reader and the writer:
21+ INPUT --reader--> AST --filter--> AST --writer--> OUTPUT
23+Filters are "pipes" that read from standard input and write to
24+standard output. They consume and produce a JSON representation
25+of the pandoc AST. (In recent versions, this representation
26+includes a `pandoc-api-version` field which refers to a
27+version of `pandoc-types`.) Filters may be written in any programming
28+language. To use a filter, you need only specify it on the
29+command line using `--filter`, e.g.
31+ pandoc -s input.txt --filter pandoc-citeproc -o output.htl
33+For a gentle introduction into writing your own filters,
34+continue this guide. There’s also a [list of third party filters
35+on the wiki](https://github.com/jgm/pandoc/wiki/Pandoc-Filters).
40+Suppose you wanted to replace all level 2+ headers in a markdown
41+document with regular paragraphs, with text in italics. How would you go
44+A first thought would be to use regular expressions. Something
47+ perl -pe 's/^##+ (.*)$/\*\1\*/' source.txt
49+This should work most of the time. But don't forget
50+that ATX style headers can end with a sequence of `#`s
51+that is not part of the header text:
55+And what if your document contains a line starting with `##` in an HTML
56+comment or delimited code block?
59+ ## This is just a comment
63+ ### A third level header in standard markdown
66+We don't want to touch *these* lines. Moreover, what about setext
67+style second-level headers?
72+We need to handle those too. Finally, can we be sure that adding
73+asterisks to each side of our string will put it in italics?
74+What if the string already contains asterisks around it? Then we'll
75+end up with bold text, which is not what we want. And what if it contains
76+a regular unescaped asterisk?
78+How would you modify your regular expression to handle these cases? It
79+would be hairy, to say the least. What we need is a real parser.
81+Well, pandoc has a real markdown parser, the library function
82+`readMarkdown`. This transforms markdown text to an abstract syntax tree
83+(AST) that represents the document structure. Why not manipulate the
84+AST directly in a short Haskell script, then convert the result back to
85+markdown using `writeMarkdown`?
87+First, let's see what this AST looks like. We can use pandoc's `native`
95+% pandoc -s -t native test.txt
96+Pandoc (Meta {unMeta = fromList []})
97+[Header 3 ("my-header",[],[]) [Str "My",Space,Str "header"]
98+, Para [Str "text",Space,Str "with",Space,Emph [Str "italics"]] ]
101+A `Pandoc` document consists of a `Meta` block (containing
102+metadata like title, authors, and date) and a list of `Block`
103+ elements. In this case, we have two `Block`s, a `Header` and a `Para`.
104+Each has as its content a list of `Inline` elements. For more details on
105+the pandoc AST, see the [haddock documentation for `Text.Pandoc.Definition`].
107+[haddock documentation for `Text.Pandoc.Definition`]: http://hackage.haskell.org/package/pandoc-types
109+Here's a short Haskell script that reads markdown, changes level
110+2+ headers to regular paragraphs, and writes the result as markdown.
111+If you save it as `behead.hs`, you can run it using `runhaskell behead.hs`.
112+It will act like a unix pipe, reading from `stdin` and writing to `stdout`.
113+Or, if you want, you can compile it, using `ghc --make behead`, then run
114+the resulting executable `behead`.
119+import Text.Pandoc.Walk (walk)
121+behead :: Block -> Block
122+behead (Header n _ xs) | n >= 2 = Para [Emph xs]
125+readDoc :: String -> Pandoc
126+readDoc s = readMarkdown def s
127+-- or, for pandoc 1.14 and greater, use:
128+-- readDoc s = case readMarkdown def s of
130+-- Left err -> error (show err)
132+writeDoc :: Pandoc -> String
133+writeDoc doc = writeMarkdown def doc
136+main = interact (writeDoc . walk behead . readDoc)
139+The magic here is the `walk` function, which converts
140+our `behead` function (a function from `Block` to `Block`) to
141+a transformation on whole `Pandoc` documents.
142+(See the [haddock documentation for `Text.Pandoc.Walk`].)
144+[haddock documentation for `Text.Pandoc.Walk`]: http://hackage.haskell.org/package/pandoc-types
146+# Queries: listing URLs
148+We can use this same technique to do much more complex transformations
149+and queries. Here's how we could extract all the URLs linked to in
150+a markdown document (again, not an easy task with regular expressions):
156+extractURL :: Inline -> [String]
157+extractURL (Link _ _ (u,_)) = [u]
158+extractURL (Image _ _ (u,_)) = [u]
161+extractURLs :: Pandoc -> [String]
162+extractURLs = query extractURL
164+readDoc :: String -> Pandoc
165+readDoc = readMarkdown def
166+-- or, for pandoc 1.14, use:
167+-- readDoc s = case readMarkdown def s of
169+-- Left err -> error (show err)
172+main = interact (unlines . extractURLs . readDoc)
175+`query` is the query counterpart of `walk`: it lifts
176+a function that operates on `Inline` elements to one that operates
177+on the whole `Pandoc` AST. The results returned by applying
178+`extractURL` to each `Inline` element are concatenated in the
183+`behead.hs` is a very special-purpose program. It reads a
184+specific input format (markdown) and writes a specific output format
185+(HTML), with a specific set of options (here, the defaults).
186+But the basic operation it performs is one that would be useful
187+in many document transformations. It would be nice to isolate the
188+part of the program that transforms the pandoc AST, leaving the rest
189+to pandoc itself. What we want is a *filter* that *just* operates
190+on the AST---or rather, on a JSON representation of the AST that
191+pandoc can produce and consume:
207+The module `Text.Pandoc.JSON` contains a function `toJSONFilter`
208+that makes it easy to write such filters. Here is a filter
209+version of `behead.hs`:
212+#!/usr/bin/env runhaskell
214+import Text.Pandoc.JSON
217+main = toJSONFilter behead
218+ where behead (Header n _ xs) | n >= 2 = Para [Emph xs]
222+It can be used this way:
224+ pandoc -f SOURCEFORMAT -t json | runhaskell behead2.hs | \
225+ pandoc -f json -t TARGETFORMAT
227+But it is easier to use the `--filter` option with pandoc:
229+ pandoc -f SOURCEFORMAT -t TARGETFORMAT --filter ./behead2.hs
231+Note that this approach requires that `behead2.hs` be executable,
236+Alternatively, we could compile the filter:
238+ ghc --make behead2.hs
239+ pandoc -f SOURCEFORMAT -t TARGETFORMAT --filter ./behead
241+Note that if the filter is placed in the system PATH, then the initial
242+`./` is not needed. Note also that the command line can include
243+multiple instances of `--filter`: the filters will be applied in
246+# LaTeX for WordPress
248+Another easy example. WordPress blogs require a special format for
249+LaTeX math. Instead of `$e=mc^2$`, you need: `$LaTeX e=mc^2$`.
250+How can we convert a markdown document accordingly?
252+Again, it's difficult to do the job reliably with regexes.
253+A `$` might be a regular currency indicator, or it might occur in
254+a comment or code block or inline code span. We just want to find
255+the `$`s that begin LaTeX math. If only we had a parser...
257+We do. Pandoc already extracts LaTeX math, so:
260+#!/usr/bin/env runhaskell
262+import Text.Pandoc.JSON
264+main = toJSONFilter wordpressify
265+ where wordpressify (Math x y) = Math x ("LaTeX " ++ y)
269+Mission accomplished. (I've omitted type signatures here,
270+just to show it can be done.)
273+# But I don't want to learn Haskell!
275+While it's easiest to write pandoc filters in Haskell, it is fairly
276+easy to write them in python using the `pandocfilters` package.
277+The package is in PyPI and can be installed using `pip install
278+pandocfilters` or `easy_install pandocfilters`.
280+Here's our "beheading" filter in python:
283+#!/usr/bin/env python
286+Pandoc filter to convert all level 2+ headers to paragraphs with
290+from pandocfilters import toJSONFilter, Emph, Para
292+def behead(key, value, format, meta):
293+ if key == 'Header' and value[0] >= 2:
294+ return Para([Emph(value[2])])
296+if __name__ == "__main__":
297+ toJSONFilter(behead)
300+`toJSONFilter(behead)` walks the AST and applies the `behead` action
301+to each element. If `behead` returns nothing, the node is unchanged;
302+if it returns an object, the node is replaced; if it returns a list,
303+the new list is spliced in.
305+Note that, although these parameters are not used in this example,
306+`format` provides access to the target format, and `meta` provides access to
307+the document's metadata.
309+There are many examples of python filters in [the pandocfilters
310+repository](http://github.com/jgm/pandocfilters).
312+For a more Pythonic alternative to pandocfilters, see
313+the [panflute](http://scorreia.com/software/panflute/) library.
314+Don't like Python? There are also ports of pandocfilters in
315+[PHP](https://github.com/vinai/pandocfilters-php),
316+[perl](https://metacpan.org/pod/Pandoc::Filter), and
317+[javascript/node.js](https://github.com/mvhenderson/pandoc-filter-node).]
319+Starting with pandoc 2.0, pandoc includes built-in support for
320+writing filters in lua. The lua interpreter is built in to
321+pandoc, so a lua filter does not require any additional software
322+to run. See the [documentation on lua
323+filters](lua-filters.html).
327+So none of our transforms have involved IO. How about a script that
328+reads a markdown document, finds all the inline code blocks with
329+attribute `include`, and replaces their contents with the contents of
333+#!/usr/bin/env runhaskell
335+import Text.Pandoc.JSON
337+doInclude :: Block -> IO Block
338+doInclude cb@(CodeBlock (id, classes, namevals) contents) =
339+ case lookup "include" namevals of
340+ Just f -> return . (CodeBlock (id, classes, namevals)) =<< readFile f
341+ Nothing -> return cb
342+doInclude x = return x
345+main = toJSONFilter doInclude
348+Try this on the following:
350+ Here's the pandoc README:
352+ ~~~~ {include="README"}
353+ this will be replaced by contents of README
358+What if we want to remove every link from a document, retaining
362+#!/usr/bin/env runhaskell
364+import Text.Pandoc.JSON
366+main = toJSONFilter delink
368+delink :: Inline -> [Inline]
369+delink (Link _ txt _) = txt
373+Note that `delink` can't be a function of type `Inline -> Inline`,
374+because the thing we want to replace the link with is not a single
375+`Inline` element, but a list of them. So we make `delink` a function
376+from an `Inline` element to a list of `Inline` elements.
377+`toJSONFilter` can still lift this function to a transformation of type
380+# A filter for ruby text
382+Finally, here's a nice real-world example, developed on the
383+[pandoc-discuss](http://groups.google.com/group/pandoc-discuss/browse_thread/thread/7baea325565878c8) list. Qubyte wrote:
385+> I'm interested in using pandoc to turn my markdown notes on Japanese
386+> into nicely set HTML and (Xe)LaTeX. With HTML5, ruby (typically used to
387+> phonetically read chinese characters by placing text above or to the
388+> side) is standard, and support from browsers is emerging (Webkit based
389+> browsers appear to fully support it). For those browsers that don't
390+> support it yet (notably Firefox) the feature falls back in a nice way
391+> by placing the phonetic reading inside brackets to the side of each
392+> Chinese character, which is suitable for other output formats too. As
393+> for (Xe)LaTeX, ruby is not an issue.
395+> At the moment, I use inline HTML to achieve the result when the
396+> conversion is to HTML, but it's ugly and uses a lot of keystrokes, for
400+> <ruby>ご<rt></rt>飯<rp>（</rp><rt>はん</rt><rp>）</rp></ruby>
403+> sets ご飯 "gohan" with "han" spelt phonetically above the second
404+> character, or to the right of it in brackets if the browser does not
405+> support ruby. I'd like to have something more like
409+> or any keystroke saving convention would be welcome.
411+We came up with the following script, which uses the convention that a
412+markdown link with a URL beginning with a hyphen is interpreted as ruby:
418+import Text.Pandoc.JSON
419+import System.Environment (getArgs)
421+handleRuby :: Maybe Format -> Inline -> Inline
422+handleRuby (Just format) (Link _ [Str ruby] ('-':kanji,_))
423+ | format == Format "html" = RawInline format
424+ $ "<ruby>" ++ kanji ++ "<rp>(</rp><rt>" ++ ruby ++ "</rt><rp>)</rp></ruby>"
425+ | format == Format "latex" = RawInline format
426+ $ "\\ruby{" ++ kanji ++ "}{" ++ ruby ++ "}"
427+ | otherwise = Str ruby
431+main = toJSONFilter handleRuby
434+Note that, when a script is called using `--filter`, pandoc passes
435+it the target format as the first argument. When a function's
436+first argument is of type `Maybe Format`, `toJSONFilter` will
437+automatically assign it `Just` the target format or `Nothing`.
439+We compile our script:
441+ ghc --make handleRuby
445+ % pandoc -F ./handleRuby -t html
448+ <p><ruby>飯<rp>(</rp><rt>はん</rt><rp>)</rp></ruby></p>
449+ % pandoc -F ./handleRuby -t latex
456+1. Put all the regular text in a markdown document in ALL CAPS
457+ (without touching text in URLs or link titles).
459+2. Remove all horizontal rules from a document.
461+3. Renumber all enumerated lists with roman numerals.
463+4. Replace each delimited code block with class `dot` with an
464+ image generated by running `dot -Tpng` (from graphviz) on the
465+ contents of the code block.
467+5. Find all code blocks with class `python` and run them
468+ using the python interpreter, printing the results to the console.