git.iko.soy/typeable/pandoc
on issue5354 help sign in
typeable/pandoc/doc/using-the-pandoc-api.md
462 lines 17 kB history raw
1% Using the pandoc API
2% John MacFarlane
4Pandoc can be used as a Haskell library, to write your own
5conversion tools or power a web application. This document
6offers an introduction to using the pandoc API.
8Detailed API documentation at the level of individual functions
9and types is available at
10<https://hackage.haskell.org/package/pandoc>.
12# Pandoc's architecture
14Pandoc is structured as a set of *readers*, which translate
15various input formats into an abstract syntax tree (the
16Pandoc AST) representing a structured document, and a set of
17*writers*, which render this AST into various input formats.
18Pictorially:
20```
21[input format] ==reader==> [Pandoc AST] ==writer==> [output format]
22```
24This architecture allows pandoc to perform $M \times N$
25conversions with $M$ readers and $N$ writers.
27The Pandoc AST is defined in the
28[pandoc-types](https://hackage.haskell.org/package/pandoc-types)
29package. You should start by looking at the Haddock
30documentation for [Text.Pandoc.Definition]. As you'll see, a
31`Pandoc` is composed of some metadata and a list of `Block`s.
32There are various kinds of `Block`, including `Para`
33(paragraph), `Header` (section heading), and `BlockQuote`. Some
34of the `Block`s (like `BlockQuote`) contain lists of `Block`s,
35while others (like `Para`) contain lists of `Inline`s, and still
36others (like `CodeBlock`) contain plain text or nothing.
37`Inline`s are the basic elements of paragraphs. The distinction
38between `Block` and `Inline` in the type system makes it
39impossible to represent, for example, a link (`Inline`) whose
40link text is a block quote (`Block`). This expressive
41limitation is mostly a help rather than a hindrance, since many
42of the formats pandoc supports have similar limitations.
44The best way to explore the pandoc AST is to use `pandoc -t
45native`, which will display the AST corresponding to some
46Markdown input:
48```
49% echo -e "1. *foo*\n2. bar" | pandoc -t native
50[OrderedList (1,Decimal,Period)
51 [[Plain [Emph [Str "foo"]]]
52 ,[Plain [Str "bar"]]]]
53```
55# A simple example
57Here is a simple example of the use of a pandoc reader and
58writer to perform a conversion:
60```haskell
61import Text.Pandoc
62import qualified Data.Text as T
63import qualified Data.Text.IO as TIO
65main :: IO ()
66main = do
67 result <- runIO $ do
68 doc <- readMarkdown def (T.pack "[testing](url)")
69 writeRST def doc
70 rst <- handleError result
71 TIO.putStrLn rst
72```
74Some notes:
761. The first part constructs a conversion pipeline: the input
77 string is passed to `readMarkdown`, and the resulting Pandoc
78 AST (`doc`) is then rendered by `writeRST`. The conversion
79 pipeline is "run" by `runIO`---more on that below.
812. `result` has the type `Either PandocError Text`. We could
82 pattern-match on this manually, but it's simpler in this
83 context to use the `handleError` function from
84 Text.Pandoc.Error. This exits with an appropriate error
85 code and message if the value is a `Left`, and returns the
86 `Text` if the value is a `Right`.
88# The PandocMonad class
90Let's look at the types of `readMarkdown` and `writeRST`:
92```haskell
93readMarkdown :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
94writeRST :: PandocMonad m => WriterOptions -> Pandoc -> m Text
95```
97The `PandocMonad m =>` part is a typeclass constraint.
98It says that `readMarkdown` and `writeRST` define computations
99that can be used in any instance of the `PandocMonad`
100type class. `PandocMonad` is defined in the module
101[Text.Pandoc.Class].
103Two instances of `PandocMonad` are provided: `PandocIO` and
104`PandocPure`. The difference is that computations run in
105`PandocIO` are allowed to do IO (for example, read a file),
106while computations in `PandocPure` are free of any side effects.
107`PandocPure` is useful for sandboxed environments, when you want
108to prevent users from doing anything malicious. To run the
109conversion in `PandocIO`, use `runIO` (as above). To run it in
110`PandocPure`, use `runPure`.
112As you can see from the Haddocks, [Text.Pandoc.Class]
113exports many auxiliary functions that can be used in any
114instance of `PandocMonad`. For example:
116```haskell
117-- | Get the verbosity level.
118getVerbosity :: PandocMonad m => m Verbosity
120-- | Set the verbosity level.
121setVerbosity :: PandocMonad m => Verbosity -> m ()
123-- Get the accomulated log messages (in temporal order).
124getLog :: PandocMonad m => m [LogMessage]
125getLog = reverse <$> getsCommonState stLog
127-- | Log a message using 'logOutput'. Note that 'logOutput' is
128-- called only if the verbosity level exceeds the level of the
129-- message, but the message is added to the list of log messages
130-- that will be retrieved by 'getLog' regardless of its verbosity level.
131report :: PandocMonad m => LogMessage -> m ()
133-- | Fetch an image or other item from the local filesystem or the net.
134-- Returns raw content and maybe mime type.
135fetchItem :: PandocMonad m
136 => String
137 -> m (B.ByteString, Maybe MimeType)
139-- Set the resource path searched by 'fetchItem'.
140setResourcePath :: PandocMonad m => [FilePath] -> m ()
141```
143If we wanted more verbose informational messages
144during the conversion we defined in the previous
145section, we could do this:
147```haskell
148 result <- runIO $ do
149 setVerbosity INFO
150 doc <- readMarkdown def (T.pack "[testing](url)")
151 writeRST def doc
152```
154Note that `PandocIO` is an instance of `MonadIO`, so you can
155use `liftIO` to perform arbitrary IO operations inside a pandoc
156conversion chain.
158# Options
160The first argument of each reader or writer is for
161options controlling the behavior of the reader or writer:
162`ReaderOptions` for readers and `WriterOptions`
163for writers. These are defined in [Text.Pandoc.Options]. It is
164a good idea to study these options to see what can be adjusted.
166`def` (from Data.Default) denotes a default value for
167each kind of option. (You can also use `defaultWriterOptions`
168and `defaultReaderOptions`.) Generally you'll want to use
169the defaults and modify them only when needed, for example:
171```haskell
172 writeRST def{ writerReferenceLinks = True }
173```
175Some particularly important options to know about:
1771. `writerTemplate`: By default, this is `Nothing`, which
178 means that a document fragment will be produced. If you
179 want a full document, you need to specify `Just template`,
180 where `template` is a String containing the template's
181 contents (not the path).
1832. `readerExtensions` and `writerExtensions`: These specify
184 the extensions to be used in parsing and rendering.
185 Extensions are defined in [Text.Pandoc.Extensions].
187# Builder
189Sometimes it's useful to construct a Pandoc document
190programmatically. To make this easier we provide the
191module [Text.Pandoc.Builder] `pandoc-types`.
193Because concatenating lists is slow, we use special
194types `Inlines` and `Blocks` that wrap a `Sequence` of
195`Inline` and `Block` elements. These are instances
196of the Monoid typeclass and can easily be concatenated:
198```haskell
199import Text.Pandoc.Builder
201mydoc :: Pandoc
202mydoc = doc $ header 1 (text "Hello!")
203 <> para (emph (text "hello world") <> text ".")
205main :: IO ()
206main = print mydoc
207```
209If you use the `OverloadedStrings` pragma, you can
210simplify this further:
212```haskell
213mydoc = doc $ header 1 "Hello!"
214 <> para (emph "hello world" <> ".")
215```
217Here's a more realistic example. Suppose your boss says: write
218me a letter in Word listing all the filling stations in Chicago
219that take the Voyager card. You find some JSON data in this
220format (`fuel.json`):
222```json
223[ {
224 "state" : "IL",
225 "city" : "Chicago",
226 "fuel_type_code" : "CNG",
227 "zip" : "60607",
228 "station_name" : "Clean Energy - Yellow Cab",
229 "cards_accepted" : "A D M V Voyager Wright_Exp CleanEnergy",
230 "street_address" : "540 W Grenshaw"
231}, ...
232```
234And then use aeson and pandoc to parse the JSON and create
235the Word document:
237```haskell
238{-# LANGUAGE OverloadedStrings #-}
239import Text.Pandoc.Builder
240import Text.Pandoc
241import Data.Monoid ((<>), mempty, mconcat)
242import Data.Aeson
243import Control.Applicative
244import Control.Monad (mzero)
245import qualified Data.ByteString.Lazy as BL
246import qualified Data.Text as T
247import Data.List (intersperse)
249data Station = Station{
250 address :: String
251 , name :: String
252 , cardsAccepted :: [String]
253 } deriving Show
255instance FromJSON Station where
256 parseJSON (Object v) = Station <$>
257 v .: "street_address" <*>
258 v .: "station_name" <*>
259 (words <$> (v .:? "cards_accepted" .!= ""))
260 parseJSON _ = mzero
262createLetter :: [Station] -> Pandoc
263createLetter stations = doc $
264 para "Dear Boss:" <>
265 para "Here are the CNG stations that accept Voyager cards:" <>
266 simpleTable [plain "Station", plain "Address", plain "Cards accepted"]
267 (map stationToRow stations) <>
268 para "Your loyal servant," <>
269 plain (image "JohnHancock.png" "" mempty)
270 where
271 stationToRow station =
272 [ plain (text $ name station)
273 , plain (text $ address station)
274 , plain (mconcat $ intersperse linebreak
275 $ map text $ cardsAccepted station)
276 ]
278main :: IO ()
279main = do
280 json <- BL.readFile "fuel.json"
281 let letter = case decode json of
282 Just stations -> createLetter [s | s <- stations,
283 "Voyager" `elem` cardsAccepted s]
284 Nothing -> error "Could not decode JSON"
285 docx <- runIO (writeDocx def letter) >>= handleError
286 BL.writeFile "letter.docx" docx
287 putStrLn "Created letter.docx"
288```
290Voila! You've written the letter without using Word and
291without looking at the data.
293# Data files
295Pandoc has a number of data files, which can be found in the
296`data/` subdirectory of the repository. These are installed
297with pandoc (or, if pandoc was compiled with the
298`embed_data_files` flag, they are embedded in the binary).
299You can retrieve data files using `readDataFile` from
300Text.Pandoc.Class. `readDataFile` will first look for the
301file in the "user data directory" (`setUserDataDir`,
302`getUserDataDir`), and if it is not found there, it will
303return the default installed with the system.
304To force the use of the default, `setUserDataDir Nothing`.
306# Templates
308Pandoc has its own template system, described in the User's
309Guide. To retrieve the default template for a system,
310use `getDefaultTemplate` from [Text.Pandoc.Templates].
311Note that this looks first in the
312`templates` subdirectory of the user data directory, allowing
313users to override the system defaults. If you want to disable
314this behavior, use `setUserDataDir Nothing`.
316To render a template, use `renderTemplate'`, which takes two
317arguments, a template (String) and a context (any instance
318of ToJSON). If you want to create a context from the metadata
319part of a Pandoc document, use `metaToJSON'` from
320[Text.Pandoc.Writers.Shared]. If you also want to incorporate
321values from variables, use `metaToJSON` instead, and make sure
322`writerVariables` is set in `WriterOptions`.
325# Handling errors and warnings
327`runIO` and `runPure` return an `Either PandocError a`. All errors
328raised in running a `PandocMonad` computation will be trapped
329and returned as a `Left` value, so they can be handled by
330the calling program. To see the constructors for `PandocError`,
331see the documentation for [Text.Pandoc.Error].
333To raise a `PandocError` from inside a `PandocMonad` computation,
334use `throwError`.
336In addition to errors, which stop execution of the conversion
337pipeline, one can generate informational messages.
338Use `report` from [Text.Pandoc.Class] to issue a `LogMessage`.
339For a list of cosntructors for `LogMessage`, see
340[Text.Pandoc.Logging]. Note that each type of log message
341is associated with a verbosity level. The verbosity level
342(`setVerbosity`/`getVerbosity`) determines whether the report
343will be printed to stderr (when running in `PandocIO`), but
344regardless of verbosity level, all reported messages are stored
345internally and may be retrieved using `getLog`.
347# Walking the AST
349It is often useful to walk the Pandoc AST either to extract
350information (e.g., what are all the URLs linked to in this
351document?, do all the code samples compile?) or to transform a
352document (e.g., increase the level of every section header,
353remove emphasis, or replace specially marked code blocks with
354images). To make this easier and more efficient, `pandoc-types`
355includes a module [Text.Pandoc.Walk].
357Here's the essential documentation:
359```haskell
360class Walkable a b where
361 -- | @walk f x@ walks the structure @x@ (bottom up) and replaces every
362 -- occurrence of an @a@ with the result of applying @f@ to it.
363 walk :: (a -> a) -> b -> b
364 walk f = runIdentity . walkM (return . f)
365 -- | A monadic version of 'walk'.
366 walkM :: (Monad m, Functor m) => (a -> m a) -> b -> m b
367 -- | @query f x@ walks the structure @x@ (bottom up) and applies @f@
368 -- to every @a@, appending the results.
369 query :: Monoid c => (a -> c) -> b -> c
370```
372`Walkable` instances are defined for most combinations of
373Pandoc types. For example, the `Walkable Inline Block`
374instance allows you to take a function `Inline -> Inline`
375and apply it over every inline in a `Block`. And
376`Walkable [Inline] Pandoc` allows you to take a function
377`[Inline] -> [Inline]` and apply it over every maximal
378list of `Inline`s in a `Pandoc`.
380Here's a simple example of a function that promotes
381the levels of headers:
383```haskell
384promoteHeaderLevels :: Pandoc -> Pandoc
385promoteHeaderLevels = walk promote
386 where promote :: Block -> Block
387 promote (Header lev attr ils) = Header (lev + 1) attr ils
388 promote x = x
389```
391`walkM` is a monadic version of `walk`; it can be used, for
392example, when you need your transformations to perform IO
393operations, use PandocMonad operations, or update internal
394state. Here's an example using the State monad to add unique
395identifiers to each code block:
397```haskell
398addCodeIdentifiers :: Pandoc -> Pandoc
399addCodeIdentifiers doc = evalState (walkM addCodeId doc) 1
400 where addCodeId :: Block -> State Int Block
401 addCodeId (CodeBlock (_,classes,kvs) code) = do
402 curId <- get
403 put (curId + 1)
404 return $ CodeBlock (show curId,classes,kvs) code
405 addCodeId x = return x
406```
408`query` is used to collect information from the AST.
409Its argument is a query function that produces a result
410in some monoidal type (e.g. a list). The results are
411concatenated together. Here's an example that returns a
412list of the URLs linked to in a document:
414```haskell
415listURLs :: Pandoc -> [String]
416listURLs = query urls
417 where urls (Link _ _ (src, _)) = [src]
418 urls _ = []
419```
421# Creating a front-end
423All of the functionality of the command-line program `pandoc`
424has been abstracted out in `convertWithOpts` in
425the module [Text.Pandoc.App]. Creating a GUI front-end for
426pandoc is thus just a matter of populating the `Opts`
427structure and calling this function.
429# Notes on using pandoc in web applications
4311. Pandoc's parsers can exhibit pathological behavior on some
432 inputs. So it is always a good idea to wrap uses of pandoc
433 in a timeout function (e.g. `System.Timeout.timeout` from `base`)
434 to prevent DOS attacks.
4362. If pandoc generates HTML from untrusted user input, it is
437 always a good idea to filter the generated HTML through
438 a sanitizer (such as `xss-sanitize`) to avoid security
439 problems.
4413. Using `runPure` rather than `runIO` will ensure that
442 pandoc's functions perform no IO operations (e.g. writing
443 files). If some resources need to be made available, a
444 "fake environment" is provided inside the state available
445 to `runPure` (see `PureState` and its associated functions
446 in [Text.Pandoc.Class]). It is also possible to write
447 a custom instance of `PandocMonad` that, for example,
448 makes wiki resources available as files in the fake environment,
449 while isolating pandoc from the rest of the system.
452[Text.Pandoc.Definition]: https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Definition.html
453[Text.Pandoc.Walk]: https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Walk.html
454[Text.Pandoc.Class]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Class.html
455[Text.Pandoc.Options]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Options.html
456[Text.Pandoc.Extensions]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Extensions.html
457[Text.Pandoc.Builder]: https://hackage.haskell.org/package/pandoc-types/docs/Text-Pandoc-Builder.html
458[Text.Pandoc.Templates]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Templates.html
459[Text.Pandoc.Logging]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Logging.html
460[Text.Pandoc.App]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-App.html
461[Text.Pandoc.Error]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Error.html
462[Text.Pandoc.Writers.Shared]: https://hackage.haskell.org/package/pandoc/docs/Text-Pandoc-Writers.Shared.html