6f8b967 · typeable/pandoc

git.iko.soy/typeable/pandoc

Removed readerSmart and the --smart option; added Ext_smart extension.

Now you will need to do -f markdown+smart instead of -f markdown --smart This change opens the way for writers, in addition to readers, to be sensitive to +smart, but this change hasn't yet been made. API change. Command-line option change. Updated manual.

commit6f8b967d98ea4270aa2492688fbcdfe8bad150b2 authorJohn MacFarlane <jgm@berkeley.edu> dateSat 14 Jan 2017, 18:27 (10 years ago) parent3876b91

15 files changed+55−69

compare with master

unifiedsplit

MANUAL.txt+18−27 pandoc.hs+10−19 src/Text/Pandoc.hs+8−1 src/Text/Pandoc/Extensions.hs+1 src/Text/Pandoc/Options.hs−2 src/Text/Pandoc/Parsing.hs+1−5 src/Text/Pandoc/Readers/CommonMark.hs+1−1 src/Text/Pandoc/Readers/LaTeX.hs+2−2 src/Text/Pandoc/Readers/Markdown.hs+3−3 src/Text/Pandoc/Readers/Org/Inlines.hs+1−1 src/Text/Pandoc/Readers/RST.hs+1−1 src/Text/Pandoc/Readers/TWiki.hs+1−1 tests/Tests/Old.hs+4−4 tests/Tests/Readers/Markdown.hs+2−1 tests/Tests/Readers/Org.hs+2−1

MANUAL.txt+18−27

@@ -394,15 +394,6 @@ Reader options

394394 HTML codes and LaTeX environments. (The LaTeX reader does pass through

395395 untranslatable LaTeX *commands*, even if `-R` is not specified.)

396396

397−`-S`, `--smart`

398−

399−: Produce typographically correct output, converting straight quotes

400− to curly quotes, `---` to em-dashes, `--` to en-dashes, and

401− `...` to ellipses. Nonbreaking spaces are inserted after certain

402− abbreviations, such as "Mr." (Note: This option is selected automatically

403− when the output format is `latex` or `context`, unless `--no-tex-ligatures`

404− is used. It has no effect for `latex` input.)

405−

406397`--old-dashes`

407398

408399: Selects the pandoc <= 1.8.2.1 behavior for parsing smart dashes: `-` before

@@ -721,12 +712,12 @@ Options affecting specific writers

721712 than parsing ligatures for quotation marks and dashes. In

722713 writing LaTeX or ConTeXt, print unicode quotation mark and

723714 dash characters literally, rather than converting them to

724− the standard ASCII TeX ligatures. Note: normally `--smart`

725− is selected automatically for LaTeX and ConTeXt output, but

726− it must be specified explicitly if `--no-tex-ligatures` is

715+ the standard ASCII TeX ligatures. Note: normally the `smart`

716+ extension is selected automatically for LaTeX and ConTeXt output,

717+ but it must be specified explicitly if `--no-tex-ligatures` is

727718 selected. If you use literal curly quotes, dashes, and

728719 ellipses in your source, then you may want to use

729− `--no-tex-ligatures` without `--smart`.

720+ `--no-tex-ligatures` without `+smart`.

730721

731722`--listings`

732723

@@ -2639,20 +2630,6 @@ two trailing spaces on a line.

26392630

26402631Backslash escapes do not work in verbatim contexts.

26412632

2642−Smart punctuation

2643−-----------------

2644−

2645−#### Extension ####

2646−

2647−If the `--smart` option is specified, pandoc will produce typographically

2648−correct output, converting straight quotes to curly quotes, `---` to

2649−em-dashes, `--` to en-dashes, and `...` to ellipses. Nonbreaking spaces

2650−are inserted after certain abbreviations, such as "Mr."

2651−

2652−Note: if your LaTeX template or any included header file call for the

2653−[`csquotes`] package, pandoc will detect this automatically and use

2654−`\enquote{...}` for quoted text.

2655−

26562633Inline formatting

26572634-----------------

26582635

@@ -3410,6 +3387,20 @@ in pandoc, but may be enabled by adding `+EXTENSION` to the format

34103387name, where `EXTENSION` is the name of the extension. Thus, for

34113388example, `markdown+hard_line_breaks` is Markdown with hard line breaks.

34123389

3390+#### Extension: `smart` ####

3391+

3392+Produce typographically correct output, converting straight

3393+quotes to curly quotes, `---` to em-dashes, `--` to en-dashes,

3394+and `...` to ellipses. Nonbreaking spaces are inserted after

3395+certain abbreviations, such as "Mr." (Note: This option is

3396+selected automatically when the output format is `latex` or

3397+`context`, unless `--no-tex-ligatures` is used. It has no

3398+effect for `latex` input.)

3399+

3400+Note: if your LaTeX template or any included header file call

3401+for the [`csquotes`] package, pandoc will detect this

3402+automatically and use `\enquote{...}` for quoted text.

3403+

34133404#### Extension: `angle_brackets_escapable` ####

34143405

34153406Allow `<` and `>` to be backslash-escaped, as they can be in

pandoc.hs+10−19

@@ -112,7 +112,6 @@ convertWithOpts opts args = do

112112 , optSectionDivs = sectionDivs

113113 , optIncremental = incremental

114114 , optSelfContained = selfContained

115− , optSmart = smart

116115 , optOldDashes = oldDashes

117116 , optHtml5 = html5

118117 , optHtmlQTags = htmlQTags

@@ -212,10 +211,6 @@ convertWithOpts opts args = do

212211 let conTeXtOutput = format == "context"

213212 let html5Output = format == "html5"

214213

215− let laTeXInput = "latex" `isPrefixOf` readerName' ||

216− "beamer" `isPrefixOf` readerName'

217−

218−

219214 -- disabling the custom writer for now

220215 writer <- if ".lua" `isSuffixOf` format

221216 -- note: use non-lowercased version writerName

@@ -295,11 +290,15 @@ convertWithOpts opts args = do

295290 uriFragment = "" }

296291 _ -> Nothing

297292

298− let readerOpts = def{ readerSmart = if laTeXInput

299− then texLigatures

300− else smart || (texLigatures &&

301− (laTeXOutput || conTeXtOutput))

302− , readerStandalone = standalone'

293+ {- TODO - smart is now an extension, but we should prob make

294+ - texligatures one too...

295+ let smartExt = if laTeXInput

296+ then texLigatures

297+ else smart || (texLigatures &&

298+ (laTeXOutput || conTeXtOutput))

299+ -}

300+

301+ let readerOpts = def{ readerStandalone = standalone'

303302 , readerParseRaw = parseRaw

304303 , readerColumns = columns

305304 , readerTabStop = tabStop

@@ -547,7 +546,6 @@ data Opt = Opt

547546 , optSectionDivs :: Bool -- ^ Put sections in div tags in HTML

548547 , optIncremental :: Bool -- ^ Use incremental lists in Slidy/Slideous/S5

549548 , optSelfContained :: Bool -- ^ Make HTML accessible offline

550− , optSmart :: Bool -- ^ Use smart typography

551549 , optOldDashes :: Bool -- ^ Parse dashes like pandoc <=1.8.2.1

552550 , optHtml5 :: Bool -- ^ Produce HTML5 in HTML

553551 , optHtmlQTags :: Bool -- ^ Use <q> tags in HTML

@@ -613,7 +611,6 @@ defaultOpts = Opt

613611 , optSectionDivs = False

614612 , optIncremental = False

615613 , optSelfContained = False

616− , optSmart = False

617614 , optOldDashes = False

618615 , optHtml5 = False

619616 , optHtmlQTags = False

@@ -692,15 +689,9 @@ options =

692689 (\opt -> return opt { optParseRaw = True }))

693690 "" -- "Parse untranslatable HTML codes and LaTeX environments as raw"

694691

695− , Option "S" ["smart"]

696− (NoArg

697− (\opt -> return opt { optSmart = True }))

698− "" -- "Use smart quotes, dashes, and ellipses"

699−

700692 , Option "" ["old-dashes"]

701693 (NoArg

702− (\opt -> return opt { optSmart = True

703− , optOldDashes = True }))

694+ (\opt -> return opt { optOldDashes = True }))

704695 "" -- "Use smart quotes, dashes, and ellipses"

705696

706697 , Option "" ["base-header-level"]

src/Text/Pandoc.hs+8−1

@@ -336,7 +336,8 @@ getDefaultExtensions "markdown_github" = githubMarkdownExtensions

336336getDefaultExtensions "markdown" = pandocExtensions

337337getDefaultExtensions "plain" = plainExtensions

338338getDefaultExtensions "org" = extensionsFromList

339− [Ext_citations, Ext_auto_identifiers]

339+ [Ext_citations,

340+ Ext_auto_identifiers]

340341getDefaultExtensions "textile" = extensionsFromList

341342 [Ext_auto_identifiers]

342343getDefaultExtensions "html" = extensionsFromList

@@ -349,6 +350,12 @@ getDefaultExtensions "epub" = extensionsFromList

349350 Ext_native_divs,

350351 Ext_native_spans,

351352 Ext_epub_html_exts]

353+getDefaultExtensions "latex" = extensionsFromList

354+ [Ext_smart,

355+ Ext_auto_identifiers]

356+getDefaultExtensions "context" = extensionsFromList

357+ [Ext_smart,

358+ Ext_auto_identifiers]

352359getDefaultExtensions _ = extensionsFromList

353360 [Ext_auto_identifiers]

354361

src/Text/Pandoc/Extensions.hs+1

@@ -130,6 +130,7 @@ data Extension =

130130 | Ext_line_blocks -- ^ RST style line blocks

131131 | Ext_epub_html_exts -- ^ Recognise the EPUB extended version of HTML

132132 | Ext_shortcut_reference_links -- ^ Shortcut reference links

133+ | Ext_smart -- ^ "Smart" quotes, apostrophes, ellipses, dashes

133134 deriving (Show, Read, Enum, Eq, Ord, Bounded, Data, Typeable, Generic)

134135

135136pandocExtensions :: Extensions

src/Text/Pandoc/Options.hs−2

@@ -54,7 +54,6 @@ import GHC.Generics (Generic)

5454

5555data ReaderOptions = ReaderOptions{

5656 readerExtensions :: Extensions -- ^ Syntax extensions

57− , readerSmart :: Bool -- ^ Smart punctuation

5857 , readerStandalone :: Bool -- ^ Standalone document with header

5958 , readerParseRaw :: Bool -- ^ Parse raw HTML, LaTeX

6059 , readerColumns :: Int -- ^ Number of columns in terminal

@@ -74,7 +73,6 @@ data ReaderOptions = ReaderOptions{

7473instance Default ReaderOptions

7574 where def = ReaderOptions{

7675 readerExtensions = pandocExtensions

77− , readerSmart = False

7876 , readerStandalone = False

7977 , readerParseRaw = False

8078 , readerColumns = 80

src/Text/Pandoc/Parsing.hs+1−5

@@ -1105,15 +1105,11 @@ registerHeader (ident,classes,kvs) header' = do

11051105 updateState $ updateHeaderMap $ insert' header' ident

11061106 return (ident,classes,kvs)

11071107

1108−-- | Fail unless we're in "smart typography" mode.

1109−failUnlessSmart :: (Stream s m a, HasReaderOptions st) => ParserT s st m ()

1110−failUnlessSmart = getOption readerSmart >>= guard

1111−

11121108smartPunctuation :: (HasReaderOptions st, HasLastStrPosition st, HasQuoteContext st m, Stream s m Char)

11131109 => ParserT s st m Inlines

11141110 -> ParserT s st m Inlines

11151111smartPunctuation inlineParser = do

1116− failUnlessSmart

1112+ guardEnabled Ext_smart

11171113 choice [ quoted inlineParser, apostrophe, dash, ellipses ]

11181114

11191115apostrophe :: Stream s m Char => ParserT s st m Inlines

src/Text/Pandoc/Readers/CommonMark.hs+1−1

@@ -43,7 +43,7 @@ import Text.Pandoc.Class (PandocMonad)

4343readCommonMark :: PandocMonad m => ReaderOptions -> String -> m Pandoc

4444readCommonMark opts s = return $

4545 nodeToPandoc $ commonmarkToNode opts' $ pack s

46− where opts' = if readerSmart opts

46+ where opts' = if extensionEnabled Ext_smart (readerExtensions opts)

4747 then [optNormalize, optSmart]

4848 else [optNormalize]

4949

src/Text/Pandoc/Readers/LaTeX.hs+2−2

@@ -187,7 +187,7 @@ mathChars =

187187quoted' :: PandocMonad m => (Inlines -> Inlines) -> LP m String -> LP m () -> LP m Inlines

188188quoted' f starter ender = do

189189 startchs <- starter

190− smart <- getOption readerSmart

190+ smart <- extensionEnabled Ext_smart <$> getOption readerExtensions

191191 if smart

192192 then do

193193 ils <- many (notFollowedBy ender >> inline)

@@ -209,7 +209,7 @@ doubleQuote = do

209209

210210singleQuote :: PandocMonad m => LP m Inlines

211211singleQuote = do

212− smart <- getOption readerSmart

212+ smart <- extensionEnabled Ext_smart <$> getOption readerExtensions

213213 if smart

214214 then quoted' singleQuoted (string "`") (try $ char '\'' >> notFollowedBy letter)

215215 <|> quoted' singleQuoted (string "‘") (try $ char '’' >> notFollowedBy letter)

src/Text/Pandoc/Readers/Markdown.hs+3−3

@@ -1590,7 +1590,7 @@ code = try $ do

15901590math :: PandocMonad m => MarkdownParser m (F Inlines)

15911591math = (return . B.displayMath <$> (mathDisplay >>= applyMacros'))

15921592 <|> (return . B.math <$> (mathInline >>= applyMacros')) <+?>

1593− ((getOption readerSmart >>= guard) *> (return <$> apostrophe)

1593+ (guardEnabled Ext_smart *> (return <$> apostrophe)

15941594 <* notFollowedBy (space <|> satisfy isPunctuation))

15951595

15961596-- Parses material enclosed in *s, **s, _s, or __s.

@@ -1697,7 +1697,7 @@ str = do

16971697 result <- many1 alphaNum

16981698 updateLastStrPos

16991699 let spacesToNbr = map (\c -> if c == ' ' then '\160' else c)

1700− isSmart <- getOption readerSmart

1700+ isSmart <- extensionEnabled Ext_smart <$> getOption readerExtensions

17011701 if isSmart

17021702 then case likelyAbbrev result of

17031703 [] -> return $ return $ B.str result

@@ -2104,7 +2104,7 @@ citation = try $ do

21042104

21052105smart :: PandocMonad m => MarkdownParser m (F Inlines)

21062106smart = do

2107− getOption readerSmart >>= guard

2107+ guardEnabled Ext_smart

21082108 doubleQuoted <|> singleQuoted <|>

21092109 choice (map (return <$>) [apostrophe, dash, ellipses])

21102110

src/Text/Pandoc/Readers/Org/Inlines.hs+1−1

@@ -841,7 +841,7 @@ exportSnippet = try $ do

841841

842842smart :: PandocMonad m => OrgParser m (F Inlines)

843843smart = do

844− getOption readerSmart >>= guard

844+ guardEnabled Ext_smart

845845 doubleQuoted <|> singleQuoted <|>

846846 choice (map (return <$>) [orgApostrophe, orgDash, orgEllipses])

847847 where

src/Text/Pandoc/Readers/RST.hs+1−1

@@ -1309,7 +1309,7 @@ note = try $ do

13091309

13101310smart :: PandocMonad m => RSTParser m Inlines

13111311smart = do

1312− getOption readerSmart >>= guard

1312+ guardEnabled Ext_smart

13131313 doubleQuoted <|> singleQuoted <|>

13141314 choice [apostrophe, dash, ellipses]

13151315

src/Text/Pandoc/Readers/TWiki.hs+1−1

@@ -476,7 +476,7 @@ symbol = count 1 nonspaceChar >>= return . B.str

476476

477477smart :: TWParser B.Inlines

478478smart = do

479− getOption readerSmart >>= guard

479+ guardEnabled Ext_smart

480480 doubleQuoted <|> singleQuoted <|>

481481 choice [ apostrophe

482482 , dash

tests/Tests/Old.hs+4−4

@@ -47,13 +47,13 @@ tests = [ testGroup "markdown"

4747 [ testGroup "writer"

4848 $ writerTests "markdown" ++ lhsWriterTests "markdown"

4949 , testGroup "reader"

50− [ test "basic" ["-r", "markdown", "-w", "native", "-s", "-S"]

50+ [ test "basic" ["-r", "markdown+smart", "-w", "native", "-s"]

5151 "testsuite.txt" "testsuite.native"

5252 , test "tables" ["-r", "markdown", "-w", "native", "--columns=80"]

5353 "tables.txt" "tables.native"

5454 , test "pipe tables" ["-r", "markdown", "-w", "native", "--columns=80"]

5555 "pipe-tables.txt" "pipe-tables.native"

56− , test "more" ["-r", "markdown", "-w", "native", "-s", "-S"]

56+ , test "more" ["-r", "markdown+smart", "-w", "native", "-s"]

5757 "markdown-reader-more.txt" "markdown-reader-more.native"

5858 , lhsReaderTest "markdown+lhs"

5959 ]

@@ -65,8 +65,8 @@ tests = [ testGroup "markdown"

6565 , testGroup "rst"

6666 [ testGroup "writer" (writerTests "rst" ++ lhsWriterTests "rst")

6767 , testGroup "reader"

68− [ test "basic" ["-r", "rst", "-w", "native",

69− "-s", "-S", "--columns=80"] "rst-reader.rst" "rst-reader.native"

68+ [ test "basic" ["-r", "rst+smart", "-w", "native",

69+ "-s", "--columns=80"] "rst-reader.rst" "rst-reader.native"

7070 , test "tables" ["-r", "rst", "-w", "native", "--columns=80"]

7171 "tables.rst" "tables-rstsubset.native"

7272 , lhsReaderTest "rst+lhs"

tests/Tests/Readers/Markdown.hs+2−1

@@ -12,7 +12,8 @@ markdown :: String -> Pandoc

1212markdown = purely $ readMarkdown def

1313

1414markdownSmart :: String -> Pandoc

15−markdownSmart = purely $ readMarkdown def { readerSmart = True }

15+markdownSmart = purely $ readMarkdown def { readerExtensions =

16+ enableExtension Ext_smart $ readerExtensions def }

1617

1718markdownCDL :: String -> Pandoc

1819markdownCDL = purely $ readMarkdown def { readerExtensions = enableExtension

tests/Tests/Readers/Org.hs+2−1

@@ -12,7 +12,8 @@ org :: String -> Pandoc

1212org = purely $ readOrg def

1313

1414orgSmart :: String -> Pandoc

15−orgSmart = purely $ readOrg def { readerSmart = True }

15+orgSmart = purely $ readOrg def { readerExtensions =

16+ enableExtension Ext_smart $ readerExtensions def }

1617

1718infix 4 =:

1819(=:) :: ToString c