@@ -1,86 +1,156 @@
1−Lua Filters
2−===========
1+% Pandoc Lua Filters
2+% Albert Krewinkel, John MacFarlane
3+% August 21, 2017
4+
5+# Introduction
6+
7+Pandoc has long supported filters, which allow the pandoc
8+abstract syntax tree (AST) to be manipulated between the parsing
9+and the writing phase. Traditional pandoc filters accept a JSON
10+representation of the pandoc AST and produce an altered JSON
11+representation of the AST. They may be written in any
12+programming language, and invoked from pandoc using the
13+`--filter` option.
14+
15+Although traditional filters are very flexible, they have a
16+couple of disadvantages. First, there is some overhead in
17+writing JSON to stdout and reading it from stdin (twice,
18+once on each side of the filter). Second, whether a filter
19+will work will depend on details of the user's environment.
20+A filter may require an interpreter for a certain programming
21+language to be available, as well as a library for manipulating
22+the pandoc AST in JSON form. One cannot simply provide a filter
23+that can be used by anyone who has a certain version of the
24+pandoc executable.
25+
26+Starting with pandoc 2.0, we have made it possible to write
27+filters in lua without any external dependencies at all.
28+A lua interpreter and a lua library for creating pandoc filters
29+is built into the pandoc executable. Pandoc data types
30+are marshalled to lua directly, avoiding the overhead of writing
31+JSON to stdout and reading it from stdin.
32+
33+Here is an example of a lua filter that converts strong emphasis
34+to small caps:
335
4−Pandoc expects lua files to return a list of filters. The filters in that list
5−are called sequentially, each on the result of the previous filter. If there is
6−no value returned by the filter script, then pandoc will try to generate a
7−filter by collecting all top-level functions whose names correspond to those of
8−pandoc elements (e.g., `Str`, `Para`, `Meta`, or `Pandoc`).
36+``` lua
37+return {
38+ {
39+ Strong = function (elem)
40+ return pandoc.SmallCaps(elem.c)
41+ end,
42+ }
43+}
44+```
945
10−Filters are expected to be put into separate files and are passed via the
11−`--lua-filter` command-line argument. E.g., if a filter is defined in a file
12−`current-date.lua`, then it would be applied like this:
46+or equivalently,
1347
14− pandoc --lua-filter=current-date.lua -f markdown MANUAL.txt
48+``` lua
49+function Strong(elem)
50+ return pandoc.SmallCaps(elem.c)
51+end
52+```
1553
16−The `--lua-filter` can be supplied multiple times, causing the filters to be
17−applied sequentially in the order they were given. If other, non-Lua filters are
18−given as well (via `--filter`), then those are executed *after* all Lua filters
19−have been applied.
54+This says: walk the AST, and when you find a Strong element,
55+replace it with a SmallCaps element with the same content.
2056
21−Lua Filter Structure
22−--------------------
57+To run it, save it in a file, say `smallcaps.lua`, and invoke
58+pandoc with `--lua-filter=smallcaps.lua`.
2359
24−Lua filters are tables with element names as keys and values consisting
25−of functions acting on those elements.
60+Here's a quick performance comparison, using a version of the
61+pandoc manual, MANUAL.txt, and versions of the same filter
62+written in compiled Haskell (`smallcaps`) and interpreted Python
63+(`smallcaps.py`):
2664
27−Filter Application
28−------------------
65+| Command | Time |
66+|--------------------------------------------------|------:|
67+| `pandoc MANUAL.txt` | 1.01s |
68+| `pandoc MANUAL.txt --filter ./smallcaps` | 1.36s |
69+| `pandoc MANUAL.txt --filter ./smallcaps.py` | 1.40s |
70+| `pandoc MANUAL.txt --lua-filter ./smallcaps.lua` | 1.03s |
2971
30−For each filter, the document is traversed and each element subjected to
31−the filter. Elements for which the filter contains an entry (i.e. a
32−function of the same name) are passed to lua element filtering function.
33−In other words, filter entries will be called for each corresponding
34−element in the document, getting the respective element as input.
72+As you can see, the lua filter avoids the substantial overhead
73+associated with marshalling to and from JSON over a pipe.
3574
36−The element function's output must be an element of the same type as the
37−input. This means a filter function acting on an inline element must
38−return an inline, and a block element must remain a block element after
39−filter application. Pandoc will throw an error if this condition is
40−violated.
75+# Lua filter structure
76+
77+Lua filters are tables with element names as keys and values
78+consisting of functions acting on those elements.
79+
80+Filters are expected to be put into separate files and are
81+passed via the `--lua-filter` command-line argument. For
82+example, if a filter is defined in a file `current-date.lua`,
83+then it would be applied like this:
84+
85+ pandoc --lua-filter=current-date.lua -f markdown MANUAL.txt
86+
87+The `--lua-filter` can be supplied multiple times, causing the
88+filters to be applied sequentially in the order they were given.
89+If other, non-Lua filters are given as well (via `--filter`),
90+then those are executed *after* all Lua filters have been
91+applied.
92+
93+Pandoc expects each lua file to return a list of filters. The
94+filters in that list are called sequentially, each on the result
95+of the previous filter. If there is no value returned by the
96+filter script, then pandoc will try to generate a single filter
97+by collecting all top-level functions whose names correspond to
98+those of pandoc elements (e.g., `Str`, `Para`, `Meta`, or
99+`Pandoc`). (That is why the two examples above are equivalent.)
100+
101+For each filter, the document is traversed and each element
102+subjected to the filter. Elements for which the filter contains
103+an entry (i.e. a function of the same name) are passed to lua
104+element filtering function. In other words, filter entries will
105+be called for each corresponding element in the document,
106+getting the respective element as input.
107+
108+The element function's output must be an element of the same
109+type as the input. This means a filter function acting on an
110+inline element must return an inline, and a block element must
111+remain a block element after filter application. Pandoc will
112+throw an error if this condition is violated.
41113
42114Elements without matching functions are left untouched.
43115
44116See [module documentation](pandoc-module.html) for a list of pandoc
45117elements.
46118
119+# Pandoc Module
47120
48−Pandoc Module
49−=============
121+The `pandoc` lua module is loaded into the filter's lua
122+environment and provides a set of functions and constants to
123+make creation and manipulation of elements easier. The global
124+variable `pandoc` is bound to the module and should generally
125+not be overwritten for this reason.
50126
51−The `pandoc` lua module is loaded into the filter's lua environment and
52−provides a set of functions and constants to make creation and
53−manipulation of elements easier. The global variable `pandoc` is bound
54−to the module and should generally not be overwritten for this reason.
127+Two major functionalities are provided by the module: element
128+creator functions and access to some of pandoc's main
129+functionalities.
55130
56−Two major functionalities are provided by the module: element creator
57−functions and access to some of pandoc's main functionalities.
131+## Element creation
58132
59−Element creation
60−----------------
133+Element creator functions like `Str`, `Para`, and `Pandoc` are
134+designed to allow easy creation of new elements that are simple
135+to use and can be read back from the lua environment.
136+Internally, pandoc uses these functions to create the lua
137+objects which are passed to element filter functions. This means
138+that elements created via this module will behave exactly as
139+those elements accessible through the filter function parameter.
61140
62−Element creator functions like `Str`, `Para`, and `Pandoc` are designed to
63−allow easy creation of new elements that are simple to use and can be
64−read back from the lua environment. Internally, pandoc uses these
65−functions to create the lua objects which are passed to element filter
66−functions. This means that elements created via this module will behave
67−exactly as those elements accessible through the filter function parameter.
141+## Exposed pandoc functionality
68142
69−Exposed pandoc functionality
70−----------------------------
143+Some filters will require access to certain functions provided
144+by pandoc. This is currently limited to the `read` function
145+which allows to parse strings into pandoc documents from within
146+the lua filter.
71147
72−Some filters will require access to certain functions provided by
73−pandoc. This is currently limited to the `read` function which allows to
74−parse strings into pandoc documents from within the lua filter.
148+# Examples
75149
150+## Macro substitution.
76151
77−Examples
78−--------
79−
80−### Macro substitution.
81−
82−The following filter converts strings containing `{{helloworld}}` with
83−emphasized text.
152+The following filter converts the string `{{helloworld}}` into
153+emphasized text "Hello, World".
84154
85155``` lua
86156return {
@@ -96,9 +166,11 @@ return {
96166}
97167```
98168
99−### Default metadata file
169+## Default metadata file
100170
101−Using the metadata from an external file as default values.
171+This filter causes metadata defined in an external file
172+(`metadata-file.yaml`) to be used as default values in
173+a document's metadata:
102174
103175``` lua
104176-- read metadata file into string
@@ -122,7 +194,10 @@ return {
122194 }
123195```
124196
125−### Setting the date in the metadata
197+## Setting the date in the metadata
198+
199+This filter sets the date in the document's metadata to the
200+current date:
126201
127202```lua
128203function Meta(m)
@@ -131,7 +206,7 @@ function Meta(m)
131206end
132207```
133208
134−### Extracting information about links
209+## Extracting information about links
135210
136211This filter prints a table of all the URLs linked to
137212in the document, together with the number of links to
@@ -168,11 +243,15 @@ function Doc (blocks, meta)
168243end
169244```
170245
171−### Replacing placeholders with their metadata value
246+## Replacing placeholders with their metadata value
247+
248+Lua filter functions are run in the order
249+
250+> *Inlines → Blocks → Meta → Pandoc*.
172251
173−Lua filter functions are run in the order *Inlines → Blocks → Meta → Pandoc*.
174−Passing information from a higher level (e.g., metadata) to a lower level (e.g.,
175−inlines) is still possible by using two filters living in the same file:
252+Passing information from a higher level (e.g., metadata) to a
253+lower level (e.g., inlines) is still possible by using two
254+filters living in the same file:
176255
177256``` lua
178257local vars = {}
@@ -200,8 +279,8 @@ If the contents of file `occupations.md` is
200279
201280``` markdown
202281---
203−name: John MacFarlane
204−occupation: Professor of Philosophy
282+name: Samuel Q. Smith
283+occupation: Professor of Phrenology
205284---
206285
207286Name
@@ -218,10 +297,10 @@ then running `pandoc --lua-filter=meta-vars.lua occupations.md` will output:
218297``` html
219298<dl>
220299<dt>Name</dt>
221−<dd><p><span>John MacFarlane</span></p>
300+<dd><p><span>Samuel Q. Smith</span></p>
222301</dd>
223302<dt>Occupation</dt>
224−<dd><p><span>Professor of Philosophy</span></p>
303+<dd><p><span>Professor of Phrenology</span></p>
225304</dd>
226305</dl>
227306```