git.iko.soy/typeable/pandoc
on 2.12 help sign in
typeable/pandoc/doc/using-the-pandoc-api.md
463 lines 17 kB history raw
1% Using the pandoc API
2% John MacFarlane
4Pandoc can be used as a Haskell library, to write your own
5conversion tools or power a web application. This document
6offers an introduction to using the pandoc API.
8Detailed API documentation at the level of individual functions
9and types is available at
10<https://hackage.haskell.org/package/pandoc>.
12# Pandoc's architecture
14Pandoc is structured as a set of *readers*, which translate
15various input formats into an abstract syntax tree (the
16Pandoc AST) representing a structured document, and a set of
17*writers*, which render this AST into various input formats.
18Pictorially:
20```
21[input format] ==reader==> [Pandoc AST] ==writer==> [output format]
22```
24This architecture allows pandoc to perform $M \times N$
25conversions with $M$ readers and $N$ writers.
27The Pandoc AST is defined in the
28[pandoc-types](https://hackage.haskell.org/package/pandoc-types)
29package. You should start by looking at the Haddock
30documentation for [Text.Pandoc.Definition]. As you'll see, a
31`Pandoc` is composed of some metadata and a list of `Block`s.
32There are various kinds of `Block`, including `Para`
33(paragraph), `Header` (section heading), and `BlockQuote`. Some
34of the `Block`s (like `BlockQuote`) contain lists of `Block`s,
35while others (like `Para`) contain lists of `Inline`s, and still
36others (like `CodeBlock`) contain plain text or nothing.
37`Inline`s are the basic elements of paragraphs. The distinction
38between `Block` and `Inline` in the type system makes it
39impossible to represent, for example, a link (`Inline`) whose
40link text is a block quote (`Block`). This expressive
41limitation is mostly a help rather than a hindrance, since many
42of the formats pandoc supports have similar limitations.
44The best way to explore the pandoc AST is to use `pandoc -t
45native`, which will display the AST corresponding to some
46Markdown input:
48```
49% echo -e "1. *foo*\n2. bar" | pandoc -t native
50[OrderedList (1,Decimal,Period)
51 [[Plain [Emph [Str "foo"]]]
52 ,[Plain [Str "bar"]]]]
53```
55# A simple example
57Here is a simple example of the use of a pandoc reader and
58writer to perform a conversion:
60```haskell
61import Text.Pandoc
62import qualified Data.Text as T
63import qualified Data.Text.IO as TIO
65main :: IO ()
66main = do
67 result <- runIO $ do
68 doc <- readMarkdown def (T.pack "[testing](url)")
69 writeRST def doc
70 rst <- handleError result
71 TIO.putStrLn rst
72```
74Some notes:
761. The first part constructs a conversion pipeline: the input
77 string is passed to `readMarkdown`, and the resulting Pandoc
78 AST (`doc`) is then rendered by `writeRST`. The conversion
79 pipeline is "run" by `runIO`---more on that below.
812. `result` has the type `Either PandocError Text`. We could
82 pattern-match on this manually, but it's simpler in this
83 context to use the `handleError` function from
84 Text.Pandoc.Error. This exits with an appropriate error
85 code and message if the value is a `Left`, and returns the
86 `Text` if the value is a `Right`.
88# The PandocMonad class
90Let's look at the types of `readMarkdown` and `writeRST`:
92```haskell
93readMarkdown :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
94writeRST :: PandocMonad m => WriterOptions -> Pandoc -> m Text
95```
97The `PandocMonad m =>` part is a typeclass constraint.
98It says that `readMarkdown` and `writeRST` define computations
99that can be used in any instance of the `PandocMonad`
100type class. `PandocMonad` is defined in the module
101[Text.Pandoc.Class].
103Two instances of `PandocMonad` are provided: `PandocIO` and
104`PandocPure`. The difference is that computations run in
105`PandocIO` are allowed to do IO (for example, read a file),
106while computations in `PandocPure` are free of any side effects.
107`PandocPure` is useful for sandboxed environments, when you want
108to prevent users from doing anything malicious. To run the
109conversion in `PandocIO`, use `runIO` (as above). To run it in
110`PandocPure`, use `runPure`.
112As you can see from the Haddocks, [Text.Pandoc.Class]
113exports many auxiliary functions that can be used in any
114instance of `PandocMonad`. For example:
116```haskell
117-- | Get the verbosity level.
118getVerbosity :: PandocMonad m => m Verbosity
120-- | Set the verbosity level.
121setVerbosity :: PandocMonad m => Verbosity -> m ()
123-- Get the accumulated log messages (in temporal order).
124getLog :: PandocMonad m => m [LogMessage]
125getLog = reverse <$> getsCommonState stLog
127-- | Log a message using 'logOutput'. Note that 'logOutput' is
128-- called only if the verbosity level exceeds the level of the
129-- message, but the message is added to the list of log messages
130-- that will be retrieved by 'getLog' regardless of its verbosity level.
131report :: PandocMonad m => LogMessage -> m ()
133-- | Fetch an image or other item from the local filesystem or the net.
134-- Returns raw content and maybe mime type.
135fetchItem :: PandocMonad m
136 => String
137 -> m (B.ByteString, Maybe MimeType)
139-- Set the resource path searched by 'fetchItem'.
140setResourcePath :: PandocMonad m => [FilePath] -> m ()
141```
143If we wanted more verbose informational messages
144during the conversion we defined in the previous
145section, we could do this:
147```haskell
148 result <- runIO $ do
149 setVerbosity INFO
150 doc <- readMarkdown def (T.pack "[testing](url)")
151 writeRST def doc
152```
154Note that `PandocIO` is an instance of `MonadIO`, so you can
155use `liftIO` to perform arbitrary IO operations inside a pandoc
156conversion chain.
158# Options
160The first argument of each reader or writer is for
161options controlling the behavior of the reader or writer:
162`ReaderOptions` for readers and `WriterOptions`
163for writers. These are defined in [Text.Pandoc.Options]. It is
164a good idea to study these options to see what can be adjusted.
166`def` (from Data.Default) denotes a default value for
167each kind of option. (You can also use `defaultWriterOptions`
168and `defaultReaderOptions`.) Generally you'll want to use
169the defaults and modify them only when needed, for example:
171```haskell
172 writeRST def{ writerReferenceLinks = True }
173```
175Some particularly important options to know about:
1771. `writerTemplate`: By default, this is `Nothing`, which
178 means that a document fragment will be produced. If you
179 want a full document, you need to specify `Just template`,
180 where `template` is a `Template Text` from
181 [Text.Pandoc.Templates] containing the template's
182 contents (not the path).
1842. `readerExtensions` and `writerExtensions`: These specify
185 the extensions to be used in parsing and rendering.
186 Extensions are defined in [Text.Pandoc.Extensions].
188# Builder
190Sometimes it's useful to construct a Pandoc document
191programmatically. To make this easier we provide the
192module [Text.Pandoc.Builder] `pandoc-types`.
194Because concatenating lists is slow, we use special
195types `Inlines` and `Blocks` that wrap a `Sequence` of
196`Inline` and `Block` elements. These are instances
197of the Monoid typeclass and can easily be concatenated:
199```haskell
200import Text.Pandoc.Builder
202mydoc :: Pandoc
203mydoc = doc $ header 1 (text "Hello!")
204 <> para (emph (text "hello world") <> text ".")
206main :: IO ()
207main = print mydoc
208```
210If you use the `OverloadedStrings` pragma, you can
211simplify this further:
213```haskell
214mydoc = doc $ header 1 "Hello!"
215 <> para (emph "hello world" <> ".")
216```
218Here's a more realistic example. Suppose your boss says: write
219me a letter in Word listing all the filling stations in Chicago
220that take the Voyager card. You find some JSON data in this
221format (`fuel.json`):
223```json
224[ {
225 "state" : "IL",
226 "city" : "Chicago",
227 "fuel_type_code" : "CNG",
228 "zip" : "60607",
229 "station_name" : "Clean Energy - Yellow Cab",
230 "cards_accepted" : "A D M V Voyager Wright_Exp CleanEnergy",
231 "street_address" : "540 W Grenshaw"
232}, ...
233```
235And then use aeson and pandoc to parse the JSON and create
236the Word document:
238```haskell
239{-# LANGUAGE OverloadedStrings #-}
240import Text.Pandoc.Builder
241import Text.Pandoc
242import Data.Monoid ((<>), mempty, mconcat)
243import Data.Aeson
244import Control.Applicative
245import Control.Monad (mzero)
246import qualified Data.ByteString.Lazy as BL
247import qualified Data.Text as T
248import Data.List (intersperse)
250data Station = Station{
251 address :: String
252 , name :: String
253 , cardsAccepted :: [String]
254 } deriving Show
256instance FromJSON Station where
257 parseJSON (Object v) = Station <$>
258 v .: "street_address" <*>
259 v .: "station_name" <*>
260 (words <$> (v .:? "cards_accepted" .!= ""))
261 parseJSON _ = mzero
263createLetter :: [Station] -> Pandoc
264createLetter stations = doc $
265 para "Dear Boss:" <>
266 para "Here are the CNG stations that accept Voyager cards:" <>
267 simpleTable [plain "Station", plain "Address", plain "Cards accepted"]
268 (map stationToRow stations) <>
269 para "Your loyal servant," <>
270 plain (image "JohnHancock.png" "" mempty)
271 where
272 stationToRow station =
273 [ plain (text $ name station)
274 , plain (text $ address station)
275 , plain (mconcat $ intersperse linebreak
276 $ map text $ cardsAccepted station)
277 ]
279main :: IO ()
280main = do
281 json <- BL.readFile "fuel.json"
282 let letter = case decode json of
283 Just stations -> createLetter [s | s <- stations,
284 "Voyager" `elem` cardsAccepted s]
285 Nothing -> error "Could not decode JSON"
286 docx <- runIO (writeDocx def letter) >>= handleError
287 BL.writeFile "letter.docx" docx
288 putStrLn "Created letter.docx"
289```
291Voila! You've written the letter without using Word and
292without looking at the data.
294# Data files
296Pandoc has a number of data files, which can be found in the
297`data/` subdirectory of the repository. These are installed
298with pandoc (or, if pandoc was compiled with the
299`embed_data_files` flag, they are embedded in the binary).
300You can retrieve data files using `readDataFile` from
301Text.Pandoc.Class. `readDataFile` will first look for the
302file in the "user data directory" (`setUserDataDir`,
303`getUserDataDir`), and if it is not found there, it will
304return the default installed with the system.
305To force the use of the default, `setUserDataDir Nothing`.
307# Templates
309Pandoc has its own template system, described in the User's
310Guide. To retrieve the default template for a system,
311use `getDefaultTemplate` from [Text.Pandoc.Templates].
312Note that this looks first in the
313`templates` subdirectory of the user data directory, allowing
314users to override the system defaults. If you want to disable
315this behavior, use `setUserDataDir Nothing`.
317To render a template, use `renderTemplate'`, which takes two
318arguments, a template (String) and a context (any instance
319of ToJSON). If you want to create a context from the metadata
320part of a Pandoc document, use `metaToJSON'` from
321[Text.Pandoc.Writers.Shared]. If you also want to incorporate
322values from variables, use `metaToJSON` instead, and make sure
323`writerVariables` is set in `WriterOptions`.
326# Handling errors and warnings
328`runIO` and `runPure` return an `Either PandocError a`. All errors
329raised in running a `PandocMonad` computation will be trapped
330and returned as a `Left` value, so they can be handled by
331the calling program. To see the constructors for `PandocError`,
332see the documentation for [Text.Pandoc.Error].
334To raise a `PandocError` from inside a `PandocMonad` computation,
335use `throwError`.
337In addition to errors, which stop execution of the conversion
338pipeline, one can generate informational messages.
339Use `report` from [Text.Pandoc.Class] to issue a `LogMessage`.
340For a list of constructors for `LogMessage`, see
341[Text.Pandoc.Logging]. Note that each type of log message
342is associated with a verbosity level. The verbosity level
343(`setVerbosity`/`getVerbosity`) determines whether the report
344will be printed to stderr (when running in `PandocIO`), but
345regardless of verbosity level, all reported messages are stored
346internally and may be retrieved using `getLog`.
348# Walking the AST
350It is often useful to walk the Pandoc AST either to extract
351information (e.g., what are all the URLs linked to in this
352document?, do all the code samples compile?) or to transform a
353document (e.g., increase the level of every section header,
354remove emphasis, or replace specially marked code blocks with
355images). To make this easier and more efficient, `pandoc-types`
356includes a module [Text.Pandoc.Walk].
358Here's the essential documentation:
360```haskell
361class Walkable a b where
362 -- | @walk f x@ walks the structure @x@ (bottom up) and replaces every
363 -- occurrence of an @a@ with the result of applying @f@ to it.
364 walk :: (a -> a) -> b -> b
365 walk f = runIdentity . walkM (return . f)
366 -- | A monadic version of 'walk'.
367 walkM :: (Monad m, Functor m) => (a -> m a) -> b -> m b
368 -- | @query f x@ walks the structure @x@ (bottom up) and applies @f@
369 -- to every @a@, appending the results.
370 query :: Monoid c => (a -> c) -> b -> c
371```
373`Walkable` instances are defined for most combinations of
374Pandoc types. For example, the `Walkable Inline Block`
375instance allows you to take a function `Inline -> Inline`
376and apply it over every inline in a `Block`. And
377`Walkable [Inline] Pandoc` allows you to take a function
378`[Inline] -> [Inline]` and apply it over every maximal
379list of `Inline`s in a `Pandoc`.
381Here's a simple example of a function that promotes
382the levels of headers:
384```haskell
385promoteHeaderLevels :: Pandoc -> Pandoc
386promoteHeaderLevels = walk promote
387 where promote :: Block -> Block
388 promote (Header lev attr ils) = Header (lev + 1) attr ils
389 promote x = x
390```
392`walkM` is a monadic version of `walk`; it can be used, for
393example, when you need your transformations to perform IO
394operations, use PandocMonad operations, or update internal
395state. Here's an example using the State monad to add unique
396identifiers to each code block:
398```haskell
399addCodeIdentifiers :: Pandoc -> Pandoc
400addCodeIdentifiers doc = evalState (walkM addCodeId doc) 1
401 where addCodeId :: Block -> State Int Block
402 addCodeId (CodeBlock (_,classes,kvs) code) = do
403 curId <- get
404 put (curId + 1)
405 return $ CodeBlock (show curId,classes,kvs) code
406 addCodeId x = return x
407```
409`query` is used to collect information from the AST.
410Its argument is a query function that produces a result
411in some monoidal type (e.g. a list). The results are
412concatenated together. Here's an example that returns a
413list of the URLs linked to in a document:
415```haskell
416listURLs :: Pandoc -> [String]
417listURLs = query urls
418 where urls (Link _ _ (src, _)) = [src]
419 urls _ = []
420```
422# Creating a front-end
424All of the functionality of the command-line program `pandoc`
425has been abstracted out in `convertWithOpts` in
426the module [Text.Pandoc.App]. Creating a GUI front-end for
427pandoc is thus just a matter of populating the `Opts`
428structure and calling this function.
430# Notes on using pandoc in web applications
4321. Pandoc's parsers can exhibit pathological behavior on some
433 inputs. So it is always a good idea to wrap uses of pandoc
434 in a timeout function (e.g. `System.Timeout.timeout` from `base`)
435 to prevent DoS attacks.
4372. If pandoc generates HTML from untrusted user input, it is
438 always a good idea to filter the generated HTML through
439 a sanitizer (such as `xss-sanitize`) to avoid security
440 problems.
4423. Using `runPure` rather than `runIO` will ensure that
443 pandoc's functions perform no IO operations (e.g. writing
444 files). If some resources need to be made available, a
445 "fake environment" is provided inside the state available
446 to `runPure` (see `PureState` and its associated functions
447 in [Text.Pandoc.Class]). It is also possible to write
448 a custom instance of `PandocMonad` that, for example,
449 makes wiki resources available as files in the fake environment,
450 while isolating pandoc from the rest of the system.
453[Text.Pandoc.Definition]: https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Definition.html
454[Text.Pandoc.Walk]: https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Walk.html
455[Text.Pandoc.Class]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Class.html
456[Text.Pandoc.Options]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Options.html
457[Text.Pandoc.Extensions]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Extensions.html
458[Text.Pandoc.Builder]: https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Builder.html
459[Text.Pandoc.Templates]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Templates.html
460[Text.Pandoc.Logging]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Logging.html
461[Text.Pandoc.App]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-App.html
462[Text.Pandoc.Error]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Error.html
463[Text.Pandoc.Writers.Shared]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Writers-Shared.html