git.iko.soy/typeable/pandoc
on master help sign in
typeable/pandoc/doc/using-the-pandoc-api.md
476 lines 17 kB history raw
1% Using the pandoc API
2% John MacFarlane
4Pandoc can be used as a Haskell library, to write your own
5conversion tools or power a web application. This document
6offers an introduction to using the pandoc API.
8Detailed API documentation at the level of individual functions
9and types is available at
10<https://hackage.haskell.org/package/pandoc>.
12# Pandoc's architecture
14Pandoc is structured as a set of *readers*, which translate
15various input formats into an abstract syntax tree (the
16Pandoc AST) representing a structured document, and a set of
17*writers*, which render this AST into various output formats.
18Pictorially:
20```
21[input format] ==reader==> [Pandoc AST] ==writer==> [output format]
22```
24This architecture allows pandoc to perform $M \times N$
25conversions with $M$ readers and $N$ writers.
27The Pandoc AST is defined in the
28[pandoc-types](https://hackage.haskell.org/package/pandoc-types)
29package. You should start by looking at the Haddock
30documentation for [Text.Pandoc.Definition]. As you'll see, a
31`Pandoc` is composed of some metadata and a list of `Block`s.
32There are various kinds of `Block`, including `Para`
33(paragraph), `Header` (section heading), and `BlockQuote`. Some
34of the `Block`s (like `BlockQuote`) contain lists of `Block`s,
35while others (like `Para`) contain lists of `Inline`s, and still
36others (like `CodeBlock`) contain plain text or nothing.
37`Inline`s are the basic elements of paragraphs. The distinction
38between `Block` and `Inline` in the type system makes it
39impossible to represent, for example, a link (`Inline`) whose
40link text is a block quote (`Block`). This expressive
41limitation is mostly a help rather than a hindrance, since many
42of the formats pandoc supports have similar limitations.
44The best way to explore the pandoc AST is to use `pandoc -t
45native`, which will display the AST corresponding to some
46Markdown input:
48```
49% echo -e "1. *foo*\n2. bar" | pandoc -t native
50[OrderedList (1,Decimal,Period)
51 [[Plain [Emph [Str "foo"]]]
52 ,[Plain [Str "bar"]]]]
53```
55# A simple example
57Here is a simple example of the use of a pandoc reader and
58writer to perform a conversion:
60```haskell
61import Text.Pandoc
62import qualified Data.Text as T
63import qualified Data.Text.IO as TIO
65main :: IO ()
66main = do
67 result <- runIO $ do
68 doc <- readMarkdown def (T.pack "[testing](url)")
69 writeRST def doc
70 rst <- handleError result
71 TIO.putStrLn rst
72```
74Some notes:
761. The first part constructs a conversion pipeline: the input
77 string is passed to `readMarkdown`, and the resulting Pandoc
78 AST (`doc`) is then rendered by `writeRST`. The conversion
79 pipeline is "run" by `runIO`---more on that below.
812. `result` has the type `Either PandocError Text`. We could
82 pattern-match on this manually, but it's simpler in this
83 context to use the `handleError` function from
84 Text.Pandoc.Error. This exits with an appropriate error
85 code and message if the value is a `Left`, and returns the
86 `Text` if the value is a `Right`.
88# The PandocMonad class
90Let's look at the types of `readMarkdown` and `writeRST`:
92```haskell
93readMarkdown :: (PandocMonad m, ToSources a)
94 => ReaderOptions
95 -> a
96 -> m Pandoc
97writeRST :: PandocMonad m
98 => WriterOptions
99 -> Pandoc
100 -> m Text
101```
103The `PandocMonad m =>` part is a typeclass constraint.
104It says that `readMarkdown` and `writeRST` define computations
105that can be used in any instance of the `PandocMonad`
106type class. `PandocMonad` is defined in the module
107[Text.Pandoc.Class].
109Two instances of `PandocMonad` are provided: `PandocIO` and
110`PandocPure`. The difference is that computations run in
111`PandocIO` are allowed to do IO (for example, read a file),
112while computations in `PandocPure` are free of any side effects.
113`PandocPure` is useful for sandboxed environments, when you want
114to prevent users from doing anything malicious. To run the
115conversion in `PandocIO`, use `runIO` (as above). To run it in
116`PandocPure`, use `runPure`.
118As you can see from the Haddocks, [Text.Pandoc.Class]
119exports many auxiliary functions that can be used in any
120instance of `PandocMonad`. For example:
122```haskell
123-- | Get the verbosity level.
124getVerbosity :: PandocMonad m => m Verbosity
126-- | Set the verbosity level.
127setVerbosity :: PandocMonad m => Verbosity -> m ()
129-- Get the accumulated log messages (in temporal order).
130getLog :: PandocMonad m => m [LogMessage]
131getLog = reverse <$> getsCommonState stLog
133-- | Log a message using 'logOutput'. Note that 'logOutput' is
134-- called only if the verbosity level exceeds the level of the
135-- message, but the message is added to the list of log messages
136-- that will be retrieved by 'getLog' regardless of its verbosity level.
137report :: PandocMonad m => LogMessage -> m ()
139-- | Fetch an image or other item from the local filesystem or the net.
140-- Returns raw content and maybe mime type.
141fetchItem :: PandocMonad m
142 => Text
143 -> m (B.ByteString, Maybe MimeType)
145-- Set the resource path searched by 'fetchItem'.
146setResourcePath :: PandocMonad m => [FilePath] -> m ()
147```
149If we wanted more verbose informational messages
150during the conversion we defined in the previous
151section, we could do this:
153```haskell
154 result <- runIO $ do
155 setVerbosity INFO
156 doc <- readMarkdown def (T.pack "[testing](url)")
157 writeRST def doc
158```
160Note that `PandocIO` is an instance of `MonadIO`, so you can
161use `liftIO` to perform arbitrary IO operations inside a pandoc
162conversion chain.
164`readMarkdown` is polymorphic in its second argument, which
165can be any type that is an instance of the `ToSources`
166typeclass. You can use `Text`, as in the example above.
167But you can also use `[(FilePath, Text)]`, if the input comes
168from multiple files and you want to track source positions
169accurately.
171# Options
173The first argument of each reader or writer is for
174options controlling the behavior of the reader or writer:
175`ReaderOptions` for readers and `WriterOptions`
176for writers. These are defined in [Text.Pandoc.Options]. It is
177a good idea to study these options to see what can be adjusted.
179`def` (from Data.Default) denotes a default value for
180each kind of option. (You can also use `defaultWriterOptions`
181and `defaultReaderOptions`.) Generally you'll want to use
182the defaults and modify them only when needed, for example:
184```haskell
185 writeRST def{ writerReferenceLinks = True }
186```
188Some particularly important options to know about:
1901. `writerTemplate`: By default, this is `Nothing`, which
191 means that a document fragment will be produced. If you
192 want a full document, you need to specify `Just template`,
193 where `template` is a `Template Text` from
194 [Text.Pandoc.Templates] containing the template's
195 contents (not the path).
1972. `readerExtensions` and `writerExtensions`: These specify
198 the extensions to be used in parsing and rendering.
199 Extensions are defined in [Text.Pandoc.Extensions].
201# Builder
203Sometimes it's useful to construct a Pandoc document
204programmatically. To make this easier we provide the
205module [Text.Pandoc.Builder] `pandoc-types`.
207Because concatenating lists is slow, we use special
208types `Inlines` and `Blocks` that wrap a `Sequence` of
209`Inline` and `Block` elements. These are instances
210of the Monoid typeclass and can easily be concatenated:
212```haskell
213import Text.Pandoc.Builder
215mydoc :: Pandoc
216mydoc = doc $ header 1 (text (T.pack "Hello!"))
217 <> para (emph (text (T.pack "hello world")) <> text (T.pack "."))
219main :: IO ()
220main = print mydoc
221```
223If you use the `OverloadedStrings` pragma, you can
224simplify this further:
226```haskell
227mydoc = doc $ header 1 "Hello!"
228 <> para (emph "hello world" <> ".")
229```
231Here's a more realistic example. Suppose your boss says: write
232me a letter in Word listing all the filling stations in Chicago
233that take the Voyager card. You find some JSON data in this
234format (`fuel.json`):
236```json
237[ {
238 "state" : "IL",
239 "city" : "Chicago",
240 "fuel_type_code" : "CNG",
241 "zip" : "60607",
242 "station_name" : "Clean Energy - Yellow Cab",
243 "cards_accepted" : "A D M V Voyager Wright_Exp CleanEnergy",
244 "street_address" : "540 W Grenshaw"
245}, ...
246```
248And then use aeson and pandoc to parse the JSON and create
249the Word document:
251```haskell
252{-# LANGUAGE OverloadedStrings #-}
253import Text.Pandoc.Builder
254import Text.Pandoc
255import Data.Monoid ((<>), mempty, mconcat)
256import Data.Aeson
257import Control.Applicative
258import Control.Monad (mzero)
259import qualified Data.ByteString.Lazy as BL
260import qualified Data.Text as T
261import Data.List (intersperse)
263data Station = Station{
264 address :: T.Text
265 , name :: T.Text
266 , cardsAccepted :: [T.Text]
267 } deriving Show
269instance FromJSON Station where
270 parseJSON (Object v) = Station <$>
271 v .: "street_address" <*>
272 v .: "station_name" <*>
273 (T.words <$> (v .:? "cards_accepted" .!= ""))
274 parseJSON _ = mzero
276createLetter :: [Station] -> Pandoc
277createLetter stations = doc $
278 para "Dear Boss:" <>
279 para "Here are the CNG stations that accept Voyager cards:" <>
280 simpleTable [plain "Station", plain "Address", plain "Cards accepted"]
281 (map stationToRow stations) <>
282 para "Your loyal servant," <>
283 plain (image "JohnHancock.png" "" mempty)
284 where
285 stationToRow station =
286 [ plain (text $ name station)
287 , plain (text $ address station)
288 , plain (mconcat $ intersperse linebreak
289 $ map text $ cardsAccepted station)
290 ]
292main :: IO ()
293main = do
294 json <- BL.readFile "fuel.json"
295 let letter = case decode json of
296 Just stations -> createLetter [s | s <- stations,
297 "Voyager" `elem` cardsAccepted s]
298 Nothing -> error "Could not decode JSON"
299 docx <- runIO (writeDocx def letter) >>= handleError
300 BL.writeFile "letter.docx" docx
301 putStrLn "Created letter.docx"
302```
304Voila! You've written the letter without using Word and
305without looking at the data.
307# Data files
309Pandoc has a number of data files, which can be found in the
310`data/` subdirectory of the repository. These are installed
311with pandoc (or, if pandoc was compiled with the
312`embed_data_files` flag, they are embedded in the binary).
313You can retrieve data files using `readDataFile` from
314Text.Pandoc.Class. `readDataFile` will first look for the
315file in the "user data directory" (`setUserDataDir`,
316`getUserDataDir`), and if it is not found there, it will
317return the default installed with the system.
318To force the use of the default, `setUserDataDir Nothing`.
320# Templates
322Pandoc has its own template system, described in the User's
323Guide. To retrieve the default template for a system,
324use `getDefaultTemplate` from [Text.Pandoc.Templates].
325Note that this looks first in the
326`templates` subdirectory of the user data directory, allowing
327users to override the system defaults. If you want to disable
328this behavior, use `setUserDataDir Nothing`.
330To render a template, use `renderTemplate'`, which takes two
331arguments, a template (Text) and a context (any instance
332of ToJSON). If you want to create a context from the metadata
333part of a Pandoc document, use `metaToJSON'` from
334[Text.Pandoc.Writers.Shared]. If you also want to incorporate
335values from variables, use `metaToJSON` instead, and make sure
336`writerVariables` is set in `WriterOptions`.
339# Handling errors and warnings
341`runIO` and `runPure` return an `Either PandocError a`. All errors
342raised in running a `PandocMonad` computation will be trapped
343and returned as a `Left` value, so they can be handled by
344the calling program. To see the constructors for `PandocError`,
345see the documentation for [Text.Pandoc.Error].
347To raise a `PandocError` from inside a `PandocMonad` computation,
348use `throwError`.
350In addition to errors, which stop execution of the conversion
351pipeline, one can generate informational messages.
352Use `report` from [Text.Pandoc.Class] to issue a `LogMessage`.
353For a list of constructors for `LogMessage`, see
354[Text.Pandoc.Logging]. Note that each type of log message
355is associated with a verbosity level. The verbosity level
356(`setVerbosity`/`getVerbosity`) determines whether the report
357will be printed to stderr (when running in `PandocIO`), but
358regardless of verbosity level, all reported messages are stored
359internally and may be retrieved using `getLog`.
361# Walking the AST
363It is often useful to walk the Pandoc AST either to extract
364information (e.g., what are all the URLs linked to in this
365document?, do all the code samples compile?) or to transform a
366document (e.g., increase the level of every section header,
367remove emphasis, or replace specially marked code blocks with
368images). To make this easier and more efficient, `pandoc-types`
369includes a module [Text.Pandoc.Walk].
371Here's the essential documentation:
373```haskell
374class Walkable a b where
375 -- | @walk f x@ walks the structure @x@ (bottom up) and replaces every
376 -- occurrence of an @a@ with the result of applying @f@ to it.
377 walk :: (a -> a) -> b -> b
378 walk f = runIdentity . walkM (return . f)
379 -- | A monadic version of 'walk'.
380 walkM :: (Monad m, Functor m) => (a -> m a) -> b -> m b
381 -- | @query f x@ walks the structure @x@ (bottom up) and applies @f@
382 -- to every @a@, appending the results.
383 query :: Monoid c => (a -> c) -> b -> c
384```
386`Walkable` instances are defined for most combinations of
387Pandoc types. For example, the `Walkable Inline Block`
388instance allows you to take a function `Inline -> Inline`
389and apply it over every inline in a `Block`. And
390`Walkable [Inline] Pandoc` allows you to take a function
391`[Inline] -> [Inline]` and apply it over every maximal
392list of `Inline`s in a `Pandoc`.
394Here's a simple example of a function that promotes
395the levels of headers:
397```haskell
398promoteHeaderLevels :: Pandoc -> Pandoc
399promoteHeaderLevels = walk promote
400 where promote :: Block -> Block
401 promote (Header lev attr ils) = Header (lev + 1) attr ils
402 promote x = x
403```
405`walkM` is a monadic version of `walk`; it can be used, for
406example, when you need your transformations to perform IO
407operations, use PandocMonad operations, or update internal
408state. Here's an example using the State monad to add unique
409identifiers to each code block:
411```haskell
412addCodeIdentifiers :: Pandoc -> Pandoc
413addCodeIdentifiers doc = evalState (walkM addCodeId doc) 1
414 where addCodeId :: Block -> State Int Block
415 addCodeId (CodeBlock (_,classes,kvs) code) = do
416 curId <- get
417 put (curId + 1)
418 return $ CodeBlock (show curId,classes,kvs) code
419 addCodeId x = return x
420```
422`query` is used to collect information from the AST.
423Its argument is a query function that produces a result
424in some monoidal type (e.g. a list). The results are
425concatenated together. Here's an example that returns a
426list of the URLs linked to in a document:
428```haskell
429listURLs :: Pandoc -> [Text]
430listURLs = query urls
431 where urls (Link _ _ (src, _)) = [src]
432 urls _ = []
433```
435# Creating a front-end
437All of the functionality of the command-line program `pandoc`
438has been abstracted out in `convertWithOpts` in
439the module [Text.Pandoc.App]. Creating a GUI front-end for
440pandoc is thus just a matter of populating the `Opts`
441structure and calling this function.
443# Notes on using pandoc in web applications
4451. Pandoc's parsers can exhibit pathological behavior on some
446 inputs. So it is always a good idea to wrap uses of pandoc
447 in a timeout function (e.g. `System.Timeout.timeout` from `base`)
448 to prevent DoS attacks.
4502. If pandoc generates HTML from untrusted user input, it is
451 always a good idea to filter the generated HTML through
452 a sanitizer (such as `xss-sanitize`) to avoid security
453 problems.
4553. Using `runPure` rather than `runIO` will ensure that
456 pandoc's functions perform no IO operations (e.g. writing
457 files). If some resources need to be made available, a
458 "fake environment" is provided inside the state available
459 to `runPure` (see `PureState` and its associated functions
460 in [Text.Pandoc.Class]). It is also possible to write
461 a custom instance of `PandocMonad` that, for example,
462 makes wiki resources available as files in the fake environment,
463 while isolating pandoc from the rest of the system.
466[Text.Pandoc.Definition]: https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Definition.html
467[Text.Pandoc.Walk]: https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Walk.html
468[Text.Pandoc.Class]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Class.html
469[Text.Pandoc.Options]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Options.html
470[Text.Pandoc.Extensions]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Extensions.html
471[Text.Pandoc.Builder]: https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Builder.html
472[Text.Pandoc.Templates]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Templates.html
473[Text.Pandoc.Logging]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Logging.html
474[Text.Pandoc.App]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-App.html
475[Text.Pandoc.Error]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Error.html
476[Text.Pandoc.Writers.Shared]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Writers-Shared.html