672a4bd · typeable/pandoc

git.iko.soy/typeable/pandoc

Lua filters: allow filtering of element lists (#6040)

Lists of Inline and Block elements can now be filtered via `Inlines` and `Blocks` functions, respectively. This is helpful if a filter conversion depends on the order of elements rather than a single element. For example, the following filter can be used to remove all spaces before a citation: function isSpaceBeforeCite (spc, cite) return spc and spc.t == 'Space' and cite and cite.t == 'Cite' end function Inlines (inlines) for i = #inlines-1,1,-1 do if isSpaceBeforeCite(inlines[i], inlines[i+1]) then inlines:remove(i) end end return inlines end Closes: #6038

commit672a4bdd1d4a587feaa38613fce64335adaad76d authorAlbert Krewinkel <albert@zeitkraut.de> dateWed 15 Jan 2020, 23:26 (7 years ago) parent400b29d

9 files changed+244−33

compare with master

unifiedsplit

doc/lua-filters.md+71−1 pandoc.cabal+2−2 src/Text/Pandoc/Lua/Filter.hs+68−29 src/Text/Pandoc/Lua/Init.hs+1 src/Text/Pandoc/Lua/Marshaling/List.hs+45 test/Tests/Lua.hs+24−1 test/lua/blocks-filter.lua+8 test/lua/inlines-filter.lua+19 test/lua/meta.lua+6

doc/lua-filters.md+71−1

@@ -136,13 +136,52 @@ Elements without matching functions are left untouched.

136136See [module documentation](#module-pandoc) for a list of pandoc

137137elements.

138138

139−## Execution order

139+## Filters on element sequences

140+

141+For some filtering tasks, the it is necessary to know the order

142+in which elements occur in the document. It is not enough then to

143+inspect a single element at a time.

144+

145+There are two special function names, which can be used to define

146+filters on lists of blocks or lists of inlines.

147+

148+[`Inlines (inlines)`]{#inlines-filter}

149+: If present in a filter, this function will be called on all

150+ lists of inline elements, like the content of a [Para]

151+ (paragraph) block, or the description of an [Image]. The

152+ `inlines` argument passed to the function will be a [List] of

153+ [Inlines] for each call.

154+

155+[`Blocks (blocks)`]{#blocks-filter}

156+: If present in a filter, this function will be called on all

157+ lists of block elements, like the content of a [MetaBlocks]

158+ meta element block, on each item of a list, and the main

159+ content of the [Pandoc] document. The `blocks` argument

160+ passed to the function will be a [List] of [Inlines] for each

161+ call.

162+

163+These filter functions are special in that the result must either

164+be nil, in which case the list is left unchanged, or must be a

165+list of the correct type, i.e., the same type as the input

166+argument. Single elements are **not** allowed as return values,

167+as a single element in this context usually hints at a bug.

168+

169+See ["Remove spaces before normal citations"][Inlines filter

170+example] for an example.

171+

172+This functionality has been added in pandoc 2.9.2.

173+

174+[Inlines filter example]: #remove-spaces-before-citations

175+

176+## Execution Order

140177

141178Element filter functions within a filter set are called in a

142179fixed order, skipping any which are not present:

143180

144181 1. functions for [*Inline* elements](#type-inline),

182+ 2. the [`Inlines`](#inlines-filter) filter function,

145183 2. functions for [*Block* elements](#type-block) ,

184+ 2. the [`Blocks`](#inlines-filter) filter function,

146185 3. the [`Meta`](#type-meta) filter function, and last

147186 4. the [`Pandoc`](#type-pandoc) filter function.

148187

@@ -368,6 +407,34 @@ function Doc (blocks, meta)

368407end

369408```

370409

410+## Remove spaces before citations

411+

412+This filter removes all spaces preceding an "author-in-text"

413+citation. In Markdown, author-in-text citations (e.g.,

414+`@citekey`), must be preceded by a space. If these spaces are

415+undesired, they must be removed with a filter.

416+

417+``` lua

418+local function is_space_before_author_in_text(spc, cite)

419+ return spc and spc.t == 'Space'

420+ and cite and cite.t == 'Cite'

421+ -- there must be only a single citation, and it must have

422+ -- mode 'AuthorInText'

423+ and #cite.citations == 1

424+ and cite.citations[1].mode == 'AuthorInText'

425+end

426+

427+function Inlines (inlines)

428+ -- Go from end to start to avoid problems with shifting indices.

429+ for i = #inlines-1, 1, -1 do

430+ if is_space_before_author_in_text(inlines[i], inlines[i+1]) then

431+ inlines:remove(i)

432+ end

433+ end

434+ return inlines

435+end

436+```

437+

371438## Replacing placeholders with their metadata value

372439

373440Lua filter functions are run in the order

@@ -1650,15 +1717,18 @@ Usage:

16501717[Citation]: #type-citation

16511718[Citations]: #type-citation

16521719[CommonState]: #type-commonstate

1720+[Image]: #type-image

16531721[Inline]: #type-inline

16541722[Inlines]: #type-inline

16551723[List]: #type-list

16561724[ListAttributes]: #type-listattributes

16571725[Meta]: #type-meta

1726+[MetaBlocks]: #type-metablocks

16581727[MetaValue]: #type-metavalue

16591728[MetaValues]: #type-metavalue

16601729[LogMessage]: #type-logmessage

16611730[Pandoc]: #type-pandoc

1731+[Para]: #type-para

16621732[Version]: #type-version

16631733[`pandoc.utils.equals`]: #pandoc.utils.equals

16641734

pandoc.cabal+2−2

@@ -605,9 +605,10 @@ library

605605 Text.Pandoc.Lua.Marshaling.AST,

606606 Text.Pandoc.Lua.Marshaling.AnyValue,

607607 Text.Pandoc.Lua.Marshaling.CommonState,

608+ Text.Pandoc.Lua.Marshaling.Context,

609+ Text.Pandoc.Lua.Marshaling.List,

608610 Text.Pandoc.Lua.Marshaling.MediaBag,

609611 Text.Pandoc.Lua.Marshaling.ReaderOptions,

610− Text.Pandoc.Lua.Marshaling.Context,

611612 Text.Pandoc.Lua.Marshaling.Version,

612613 Text.Pandoc.Lua.Module.MediaBag,

613614 Text.Pandoc.Lua.Module.Pandoc,

@@ -838,4 +839,3 @@ benchmark benchmark-pandoc

838839 -Widentities

839840 -Werror=missing-home-modules

840841 -fhide-source-paths

841−

src/Text/Pandoc/Lua/Filter.hs+68−29

@@ -2,8 +2,8 @@

22{-# LANGUAGE NoImplicitPrelude #-}

33{- |

44Module : Text.Pandoc.Lua.Filter

77License : GNU GPL, version 2 or above

88Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>

99Stability : alpha

@@ -13,25 +13,23 @@ Types and functions for running Lua filters.

1313module Text.Pandoc.Lua.Filter ( LuaFilterFunction

1414 , LuaFilter

1515 , runFilterFile

16− , tryFilter

17− , runFilterFunction

18− , walkMWithLuaFilter

1916 , walkInlines

2017 , walkBlocks

21− , blockElementNames

22− , inlineElementNames

2318 , module Text.Pandoc.Lua.Walk

2419 ) where

2520import Prelude

21+import Control.Applicative ((<|>))

2622import Control.Monad (mplus, (>=>))

2723import Control.Monad.Catch (finally)

2824import Data.Data (Data, DataType, dataTypeConstrs, dataTypeName, dataTypeOf,

2925 showConstr, toConstr, tyconUQname)

3026import Data.Foldable (foldrM)

3127import Data.Map (Map)

28+import Data.Maybe (fromMaybe)

3229import Foreign.Lua (Lua, Peekable, Pushable)

3330import Text.Pandoc.Definition

3431import Text.Pandoc.Lua.Marshaling ()

32+import Text.Pandoc.Lua.Marshaling.List (List (..))

3533import Text.Pandoc.Lua.Walk (SingletonsList (..))

3634import Text.Pandoc.Walk (Walkable (walkM))

3735

@@ -67,7 +65,9 @@ newtype LuaFilter = LuaFilter (Map String LuaFilterFunction)

6765

6866instance Peekable LuaFilter where

6967 peek idx = do

70− let constrs = metaFilterName

68+ let constrs = listOfInlinesFilterName

69+ : listOfBlocksFilterName

70+ : metaFilterName

7171 : pandocFilterNames

7272 ++ blockElementNames

7373 ++ inlineElementNames

@@ -109,22 +109,34 @@ elementOrList x = do

109109 Right res -> [res] <$ Lua.pop 1

110110 Left _ -> Lua.peekList topOfStack `finally` Lua.pop 1

111111

112−-- | Try running a filter for the given element

113−tryFilter :: (Data a, Peekable a, Pushable a)

114− => LuaFilter -> a -> Lua [a]

115−tryFilter (LuaFilter fnMap) x =

116− let filterFnName = showConstr (toConstr x)

117− catchAllName = tyconUQname $ dataTypeName (dataTypeOf x)

118− in

119− case Map.lookup filterFnName fnMap `mplus` Map.lookup catchAllName fnMap of

120− Just fn -> runFilterFunction fn x *> elementOrList x

121− Nothing -> return [x]

122−

123−-- | Apply filter on a sequence of AST elements.

112+-- | Pop and return a value from the stack; if the value at the top of

113+-- the stack is @nil@, return the fallback element.

114+popOption :: Peekable a => a -> Lua a

115+popOption fallback = fromMaybe fallback . Lua.fromOptional <$> Lua.popValue

116+

117+-- | Apply filter on a sequence of AST elements. Both lists and single

118+-- value are accepted as filter function return values.

124119runOnSequence :: (Data a, Peekable a, Pushable a)

125120 => LuaFilter -> SingletonsList a -> Lua (SingletonsList a)

126−runOnSequence lf (SingletonsList xs) =

127− SingletonsList <$> mconcatMapM (tryFilter lf) xs

121+runOnSequence (LuaFilter fnMap) (SingletonsList xs) =

122+ SingletonsList <$> mconcatMapM tryFilter xs

123+ where

124+ tryFilter :: (Data a, Peekable a, Pushable a) => a -> Lua [a]

125+ tryFilter x =

126+ let filterFnName = showConstr (toConstr x)

127+ catchAllName = tyconUQname $ dataTypeName (dataTypeOf x)

128+ in case Map.lookup filterFnName fnMap <|> Map.lookup catchAllName fnMap of

129+ Just fn -> runFilterFunction fn x *> elementOrList x

130+ Nothing -> return [x]

131+

132+-- | Try filtering the given value without type error corrections on

133+-- the return value.

134+runOnValue :: (Data a, Peekable a, Pushable a)

135+ => String -> LuaFilter -> a -> Lua a

136+runOnValue filterFnName (LuaFilter fnMap) x =

137+ case Map.lookup filterFnName fnMap of

138+ Just fn -> runFilterFunction fn x *> popOption x

139+ Nothing -> return x

128140

129141-- | Push a value to the stack via a lua filter function. The filter function is

130142-- called with given element as argument and is expected to return an element.

@@ -138,7 +150,12 @@ runFilterFunction lf x = do

138150

139151walkMWithLuaFilter :: LuaFilter -> Pandoc -> Lua Pandoc

140152walkMWithLuaFilter f =

141− walkInlines f >=> walkBlocks f >=> walkMeta f >=> walkPandoc f

153+ walkInlines f

154+ >=> walkInlineLists f

155+ >=> walkBlocks f

156+ >=> walkBlockLists f

157+ >=> walkMeta f

158+ >=> walkPandoc f

142159

143160mconcatMapM :: (Monad m) => (a -> m [a]) -> [a] -> m [a]

144161mconcatMapM f = fmap mconcat . mapM f

@@ -146,6 +163,9 @@ mconcatMapM f = fmap mconcat . mapM f

146163hasOneOf :: LuaFilter -> [String] -> Bool

147164hasOneOf (LuaFilter fnMap) = any (\k -> Map.member k fnMap)

148165

166+contains :: LuaFilter -> String -> Bool

167+contains (LuaFilter fnMap) = (`Map.member` fnMap)

168+

149169walkInlines :: Walkable (SingletonsList Inline) a => LuaFilter -> a -> Lua a

150170walkInlines lf =

151171 let f :: SingletonsList Inline -> Lua (SingletonsList Inline)

@@ -154,6 +174,14 @@ walkInlines lf =

154174 then walkM f

155175 else return

156176

177+walkInlineLists :: Walkable (List Inline) a => LuaFilter -> a -> Lua a

178+walkInlineLists lf =

179+ let f :: List Inline -> Lua (List Inline)

180+ f = runOnValue listOfInlinesFilterName lf

181+ in if lf `contains` listOfInlinesFilterName

182+ then walkM f

183+ else return

184+

157185walkBlocks :: Walkable (SingletonsList Block) a => LuaFilter -> a -> Lua a

158186walkBlocks lf =

159187 let f :: SingletonsList Block -> Lua (SingletonsList Block)

@@ -162,13 +190,18 @@ walkBlocks lf =

162190 then walkM f

163191 else return

164192

193+walkBlockLists :: Walkable (List Block) a => LuaFilter -> a -> Lua a

194+walkBlockLists lf =

195+ let f :: List Block -> Lua (List Block)

196+ f = runOnValue listOfBlocksFilterName lf

197+ in if lf `contains` listOfBlocksFilterName

198+ then walkM f

199+ else return

200+

165201walkMeta :: LuaFilter -> Pandoc -> Lua Pandoc

166−walkMeta (LuaFilter fnMap) =

167− case Map.lookup "Meta" fnMap of

168− Just fn -> walkM (\(Pandoc meta blocks) -> do

169− meta' <- runFilterFunction fn meta *> singleElement meta

170− return $ Pandoc meta' blocks)

171− Nothing -> return

202+walkMeta lf (Pandoc m bs) = do

203+ m' <- runOnValue "Meta" lf m

204+ return $ Pandoc m' bs

172205

173206walkPandoc :: LuaFilter -> Pandoc -> Lua Pandoc

174207walkPandoc (LuaFilter fnMap) =

@@ -185,6 +218,12 @@ inlineElementNames = "Inline" : constructorsFor (dataTypeOf (Str mempty))

185218blockElementNames :: [String]

186219blockElementNames = "Block" : constructorsFor (dataTypeOf (Para []))

187220

221+listOfInlinesFilterName :: String

222+listOfInlinesFilterName = "Inlines"

223+

224+listOfBlocksFilterName :: String

225+listOfBlocksFilterName = "Blocks"

226+

188227metaFilterName :: String

189228metaFilterName = "Meta"

190229

src/Text/Pandoc/Lua/Init.hs+1

@@ -119,6 +119,7 @@ putConstructorsInRegistry = do

119119 constrsToReg $ Pandoc.Citation mempty mempty mempty Pandoc.AuthorInText 0 0

120120 putInReg "Attr" -- used for Attr type alias

121121 putInReg "ListAttributes" -- used for ListAttributes type alias

122+ putInReg "List" -- pandoc.List

122123 where

123124 constrsToReg :: Data a => a -> Lua ()

124125 constrsToReg = mapM_ (putInReg . showConstr) . dataTypeConstrs . dataTypeOf

src/Text/Pandoc/Lua/Marshaling/List.hsnew file+45

@@ -0,0 +1,45 @@

1+{-# LANGUAGE FlexibleInstances #-}

2+{-# LANGUAGE DeriveDataTypeable #-}

3+{-# LANGUAGE MultiParamTypeClasses #-}

4+{-# LANGUAGE NoImplicitPrelude #-}

5+{-# LANGUAGE UndecidableInstances #-}

6+{- |

7+Module : Text.Pandoc.Lua.Marshaling.List

10+License : GNU GPL, version 2 or above

11+Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>

12+Stability : alpha

13+

14+Marshaling/unmarshaling instances for @pandoc.List@s.

15+-}

16+module Text.Pandoc.Lua.Marshaling.List

17+ ( List (..)

18+ ) where

19+

20+import Prelude

21+import Data.Data (Data)

22+import Foreign.Lua (Peekable, Pushable)

23+import Text.Pandoc.Walk (Walkable (..))

24+import Text.Pandoc.Lua.Util (defineHowTo, pushViaConstructor)

25+

26+import qualified Foreign.Lua as Lua

27+

28+-- | List wrapper which is marshalled as @pandoc.List@.

29+newtype List a = List { fromList :: [a] }

30+ deriving (Data, Eq, Show)

31+

32+instance Pushable a => Pushable (List a) where

33+ push (List xs) =

34+ pushViaConstructor "List" xs

35+

36+instance Peekable a => Peekable (List a) where

37+ peek idx = defineHowTo "get List" $ do

38+ xs <- Lua.peek idx

39+ return $ List xs

40+

41+-- List is just a wrapper, so we can reuse the walk instance for

42+-- unwrapped Hasekll lists.

43+instance Walkable [a] b => Walkable (List a) b where

44+ walkM f = walkM (fmap fromList . f . List)

45+ query f = query (f . List)

test/Tests/Lua.hs+24−1

@@ -23,7 +23,8 @@ import Text.Pandoc.Arbitrary ()

2323import Text.Pandoc.Builder (bulletList, definitionList, displayMath, divWith,

2424 doc, doubleQuoted, emph, header, lineBlock,

2525 linebreak, math, orderedList, para, plain, rawBlock,

26− singleQuoted, space, str, strong)

26+ singleQuoted, space, str, strong,

27+ HasMeta (setMeta))

2728import Text.Pandoc.Class (runIOorExplode, setUserDataDir)

2829import Text.Pandoc.Definition (Block (BlockQuote, Div, Para), Inline (Emph, Str),

2930 Attr, Meta, Pandoc, pandocTypesVersion)

@@ -129,6 +130,28 @@ tests = map (localOption (QuickCheckTests 20))

129130 (doc $ divWith ("", [], kv_before) (para "nil"))

130131 (doc $ divWith ("", [], kv_after) (para "nil"))

131132

133+ , testCase "Filter list of inlines" $

134+ assertFilterConversion "List of inlines"

135+ "inlines-filter.lua"

136+ (doc $ para ("Hello," <> linebreak <> "World! Wassup?"))

137+ (doc $ para "Hello, World! Wassup?")

138+

139+ , testCase "Filter list of blocks" $

140+ assertFilterConversion "List of blocks"

141+ "blocks-filter.lua"

142+ (doc $ para "one." <> para "two." <> para "three.")

143+ (doc $ plain "3")

144+

145+ , testCase "Filter Meta" $

146+ let setMetaBefore = setMeta "old" ("old" :: T.Text)

147+ . setMeta "bool" False

148+ setMetaAfter = setMeta "new" ("new" :: T.Text)

149+ . setMeta "bool" True

150+ in assertFilterConversion "Meta filtering"

151+ "meta.lua"

152+ (setMetaBefore . doc $ mempty)

153+ (setMetaAfter . doc $ mempty)

154+

132155 , testCase "Script filename is set" $

133156 assertFilterConversion "unexpected script name"

134157 "script-name.lua"

test/lua/blocks-filter.luanew file+8

@@ -0,0 +1,8 @@

1+function Blocks (blks)

2+ -- verify that this looks like a `pandoc.List`

3+ if not blks.find or not blks.map or not blks.filter then

4+ error("table doesn't seem to be an instance of pandoc.List")

5+ end

6+ -- return plain block containing the number of elements in the list

7+ return {pandoc.Plain {pandoc.Str(tostring(#blks))}}

8+end

test/lua/inlines-filter.luanew file+19

@@ -0,0 +1,19 @@

1+function isWorldAfterSpace (fst, snd)

2+ return fst and fst.t == 'LineBreak'

3+ and snd and snd.t == 'Str' and snd.text == 'World!'

4+end

6+function Inlines (inlns)

7+ -- verify that this looks like a `pandoc.List`

8+ if not inlns.find or not inlns.map or not inlns.filter then

9+ error("table doesn't seem to be an instance of pandoc.List")

10+ end

11+

12+ -- Remove spaces before string "World"

13+ for i = #inlns-1,1,-1 do

14+ if isWorldAfterSpace(inlns[i], inlns[i+1]) then

15+ inlns[i] = pandoc.Space()

16+ end

17+ end

18+ return inlns

19+end

test/lua/meta.luanew file+6

@@ -0,0 +1,6 @@

1+function Meta (meta)

2+ meta.old = nil

3+ meta.new = "new"

4+ meta.bool = (meta.bool == false)

5+ return meta

6+end