git.iko.soy/typeable/pandoc
on master help sign in
swap
b8e0b46634 ahead, 0 behind
566 files changed+17615−11011
since splitsnapshots
unifiedsplit
.circleci/config.yml+2−2 .editorconfig+8 .github/ISSUE_TEMPLATE−11 .github/ISSUE_TEMPLATE/bug_report.md+25 .github/ISSUE_TEMPLATE/config.yml+9 .github/ISSUE_TEMPLATE/feature_request.md+21 .github/workflows/ci.yml+92−8 .github/workflows/nightly.yml+9−2 .github/workflows/release-candidate.yml−1 .gitignore+7 .hlint.yaml+9−5 AUTHORS.md+23 CONTRIBUTING.md+27−19 COPYING.md+1−1 COPYRIGHT+17−26 INSTALL.md+18−15 MANUAL.txt+319−100 Makefile+32−11 README.md+9−7 README.template+6−6 RELEASE-CHECKLIST+1 SECURITY.md+12 app/pandoc.hs+1−3 benchmark/benchmark-pandoc.hs+65−84 benchmark/weigh-pandoc.hs−50 cabal.project+9−4 changelog.md+1437−2 data/default.csl+12−2 data/docx/word/settings.xml+6−16 data/docx/word/styles.xml+15 data/jats.csl−203 data/odt/styles.xml+1−1 data/templates/article.jats_publishing+6−4 data/templates/default.biblatex+10 data/templates/default.bibtex+10 data/templates/default.context+7−5 data/templates/default.dzslides+1−1 data/templates/default.jats_articleauthoring+3−1 data/templates/default.latex+65−40 data/templates/default.revealjs+79−125 data/templates/default.s5+3 data/templates/default.slideous+7 data/templates/default.slidy+5 data/templates/styles.html+2−1 default.nix+13 doc/epub.md+9−2 doc/faqs.md+128 doc/filters.md+2 doc/jats.md+31−7 doc/libraries.md+63 doc/lua-filters.md+192−6 doc/org.md+16−2 doc/using-the-pandoc-api.md+25−12 linux/make_artifacts.sh+19−7 man/pandoc.1+358−89 man/pandoc.1.after+1−1 pandoc.cabal+110−161 release.nix+4 shell.nix+106 src/Text/Pandoc.hs+1−1 src/Text/Pandoc/App.hs+66−43 src/Text/Pandoc/App/CommandLineOptions.hs+64−51 src/Text/Pandoc/App/FormatHeuristics.hs+19−8 src/Text/Pandoc/App/Opt.hs+261−51 src/Text/Pandoc/App/OutputSettings.hs+3−3 src/Text/Pandoc/Asciify.hs+14−391 src/Text/Pandoc/BCP47.hs−112 src/Text/Pandoc/CSS.hs+1−1 src/Text/Pandoc/CSV.hs+2−3 src/Text/Pandoc/Citeproc.hs+185−123 src/Text/Pandoc/Citeproc/BibTeX.hs+297−56 src/Text/Pandoc/Citeproc/Data.hs+7−6 src/Text/Pandoc/Citeproc/Locator.hs+20−4 src/Text/Pandoc/Citeproc/MetaValue.hs+5−3 src/Text/Pandoc/Class/CommonState.hs+1−1 src/Text/Pandoc/Class/IO.hs+24−25 src/Text/Pandoc/Class/PandocMonad.hs+43−36 src/Text/Pandoc/Data.hs+1−1 src/Text/Pandoc/Error.hs+102−49 src/Text/Pandoc/Extensions.hs+26−4 src/Text/Pandoc/Filter.hs+1−1 src/Text/Pandoc/Filter/JSON.hs+1−1 src/Text/Pandoc/Filter/Lua.hs+1−1 src/Text/Pandoc/Filter/Path.hs+1−1 src/Text/Pandoc/Highlighting.hs+6−6 src/Text/Pandoc/Image.hs+1−1 src/Text/Pandoc/ImageSize.hs+17−7 src/Text/Pandoc/Logging.hs+33−10 src/Text/Pandoc/Lua.hs+1−1 src/Text/Pandoc/Lua/ErrorConversion.hs+1−1 src/Text/Pandoc/Lua/Filter.hs+6−3 src/Text/Pandoc/Lua/Global.hs+1−1 src/Text/Pandoc/Lua/Init.hs+15−5 src/Text/Pandoc/Lua/Marshaling.hs+2−2 src/Text/Pandoc/Lua/Marshaling/AST.hs+79−81 src/Text/Pandoc/Lua/Marshaling/AnyValue.hs+1−1 src/Text/Pandoc/Lua/Marshaling/CommonState.hs+2−2 src/Text/Pandoc/Lua/Marshaling/Context.hs+2−2 src/Text/Pandoc/Lua/Marshaling/List.hs+2−2 src/Text/Pandoc/Lua/Marshaling/MediaBag.hs+2−2 src/Text/Pandoc/Lua/Marshaling/PandocError.hs+1−1 src/Text/Pandoc/Lua/Marshaling/ReaderOptions.hs+2−2 src/Text/Pandoc/Lua/Marshaling/SimpleTable.hs+1−1 src/Text/Pandoc/Lua/Marshaling/Version.hs+1−1 src/Text/Pandoc/Lua/Module/MediaBag.hs+18−17 src/Text/Pandoc/Lua/Module/Pandoc.hs+29−23 src/Text/Pandoc/Lua/Module/System.hs+1−1 src/Text/Pandoc/Lua/Module/Types.hs+1−1 src/Text/Pandoc/Lua/Module/Utils.hs+2−2 src/Text/Pandoc/Lua/Packages.hs+10−27 src/Text/Pandoc/Lua/PandocLua.hs+20−13 src/Text/Pandoc/Lua/Util.hs+2−2 src/Text/Pandoc/Lua/Walk.hs+2−2 src/Text/Pandoc/MIME.hs+26−3 src/Text/Pandoc/MediaBag.hs+46−15 src/Text/Pandoc/Options.hs+18−18 src/Text/Pandoc/PDF.hs+34−16 src/Text/Pandoc/Parsing.hs+373−255 src/Text/Pandoc/Process.hs+1−1 src/Text/Pandoc/Readers.hs+56−49 src/Text/Pandoc/Readers/BibTeX.hs+15−9 src/Text/Pandoc/Readers/CSV.hs+8−8 src/Text/Pandoc/Readers/CommonMark.hs+47−5 src/Text/Pandoc/Readers/Creole.hs+5−6 src/Text/Pandoc/Readers/CslJson.hs+6−5 src/Text/Pandoc/Readers/DocBook.hs+95−82 src/Text/Pandoc/Readers/Docx.hs+59−42 src/Text/Pandoc/Readers/Docx/Combine.hs+6−6 src/Text/Pandoc/Readers/Docx/Parse.hs+214−120 src/Text/Pandoc/Readers/Docx/Parse/Styles.hs+32−25 src/Text/Pandoc/Readers/Docx/Util.hs+28−26 src/Text/Pandoc/Readers/DokuWiki.hs+9−8 src/Text/Pandoc/Readers/EPUB.hs+44−36 src/Text/Pandoc/Readers/FB2.hs+57−51 src/Text/Pandoc/Readers/HTML.hs+200−137 src/Text/Pandoc/Readers/HTML/Parsing.hs+1−1 src/Text/Pandoc/Readers/HTML/Table.hs+37−17 src/Text/Pandoc/Readers/HTML/TagCategories.hs+1−1 src/Text/Pandoc/Readers/HTML/Types.hs+1−1 src/Text/Pandoc/Readers/Haddock.hs+11−7 src/Text/Pandoc/Readers/Ipynb.hs+6−4 src/Text/Pandoc/Readers/JATS.hs+37−51 src/Text/Pandoc/Readers/Jira.hs+14−10 src/Text/Pandoc/Readers/LaTeX.hs+252−1415 src/Text/Pandoc/Readers/LaTeX/Citation.hs+210 src/Text/Pandoc/Readers/LaTeX/Inline.hs+397 src/Text/Pandoc/Readers/LaTeX/Lang.hs+200−121 src/Text/Pandoc/Readers/LaTeX/Macro.hs+184 src/Text/Pandoc/Readers/LaTeX/Math.hs+221 src/Text/Pandoc/Readers/LaTeX/Parsing.hs+182−54 src/Text/Pandoc/Readers/LaTeX/SIunitx.hs+152−71 src/Text/Pandoc/Readers/LaTeX/Table.hs+378 src/Text/Pandoc/Readers/LaTeX/Types.hs+15−2 src/Text/Pandoc/Readers/Man.hs+18−11 src/Text/Pandoc/Readers/Markdown.hs+287−239 src/Text/Pandoc/Readers/MediaWiki.hs+21−14 src/Text/Pandoc/Readers/Metadata.hs+45−18 src/Text/Pandoc/Readers/Muse.hs+10−9 src/Text/Pandoc/Readers/Native.hs+8−6 src/Text/Pandoc/Readers/OPML.hs+22−36 src/Text/Pandoc/Readers/Odt.hs+28−25 src/Text/Pandoc/Readers/Odt/Arrows/State.hs+2−2 src/Text/Pandoc/Readers/Odt/ContentReader.hs+19−13 src/Text/Pandoc/Readers/Odt/Generic/Namespaces.hs+2−1 src/Text/Pandoc/Readers/Odt/Generic/Utils.hs+9−24 src/Text/Pandoc/Readers/Odt/Generic/XMLConverter.hs+13−14 src/Text/Pandoc/Readers/Odt/Namespaces.hs+6−5 src/Text/Pandoc/Readers/Odt/StyleReader.hs+21−17 src/Text/Pandoc/Readers/Org.hs+5−8 src/Text/Pandoc/Readers/Org/BlockStarts.hs+1−1 src/Text/Pandoc/Readers/Org/Blocks.hs+59−21 src/Text/Pandoc/Readers/Org/DocumentTree.hs+1−1 src/Text/Pandoc/Readers/Org/ExportSettings.hs+1−1 src/Text/Pandoc/Readers/Org/Inlines.hs+9−8 src/Text/Pandoc/Readers/Org/Meta.hs+11−5 src/Text/Pandoc/Readers/Org/ParserState.hs+1−1 src/Text/Pandoc/Readers/Org/Parsing.hs+6−6 src/Text/Pandoc/Readers/Org/Shared.hs+1−1 src/Text/Pandoc/Readers/RST.hs+85−116 src/Text/Pandoc/Readers/Roff.hs+16−18 src/Text/Pandoc/Readers/TWiki.hs+7−27 src/Text/Pandoc/Readers/Textile.hs+96−90 src/Text/Pandoc/Readers/TikiWiki.hs+6−6 src/Text/Pandoc/Readers/Txt2Tags.hs+48−22 src/Text/Pandoc/Readers/Vimwiki.hs+12−7 src/Text/Pandoc/RoffChar.hs+1−1 src/Text/Pandoc/SelfContained.hs+5−8 src/Text/Pandoc/Shared.hs+52−101 src/Text/Pandoc/Slides.hs+1−1 src/Text/Pandoc/Sources.hs+195 src/Text/Pandoc/Templates.hs+5−1 src/Text/Pandoc/Translations.hs+1−1 src/Text/Pandoc/UTF8.hs+21−24 src/Text/Pandoc/UUID.hs+1−1 src/Text/Pandoc/Writers.hs+9−4 src/Text/Pandoc/Writers/AsciiDoc.hs+11−5 src/Text/Pandoc/Writers/BibTeX.hs+61 src/Text/Pandoc/Writers/CommonMark.hs+1−1 src/Text/Pandoc/Writers/ConTeXt.hs+45−39 src/Text/Pandoc/Writers/CslJson.hs+10−9 src/Text/Pandoc/Writers/Custom.hs+1−1 src/Text/Pandoc/Writers/Docbook.hs+51−8 src/Text/Pandoc/Writers/Docx.hs+177−357 src/Text/Pandoc/Writers/Docx/StyleMap.hs+1−1 src/Text/Pandoc/Writers/Docx/Table.hs+227 src/Text/Pandoc/Writers/Docx/Types.hs+185 src/Text/Pandoc/Writers/DokuWiki.hs+4−2 src/Text/Pandoc/Writers/EPUB.hs+261−211 src/Text/Pandoc/Writers/FB2.hs+61−77 src/Text/Pandoc/Writers/GridTable.hs+157 src/Text/Pandoc/Writers/HTML.hs+115−38 src/Text/Pandoc/Writers/Haddock.hs+13−2 src/Text/Pandoc/Writers/Ipynb.hs+1−1 src/Text/Pandoc/Writers/JATS.hs+99−56 src/Text/Pandoc/Writers/JATS/References.hs+164 src/Text/Pandoc/Writers/JATS/Table.hs+26−6 src/Text/Pandoc/Writers/JATS/Types.hs+15−4 src/Text/Pandoc/Writers/Jira.hs+42−17 src/Text/Pandoc/Writers/LaTeX.hs+54−566 src/Text/Pandoc/Writers/LaTeX/Caption.hs+1−2 src/Text/Pandoc/Writers/LaTeX/Citation.hs+181 src/Text/Pandoc/Writers/LaTeX/Lang.hs+192 src/Text/Pandoc/Writers/LaTeX/Notes.hs+1−1 src/Text/Pandoc/Writers/LaTeX/Table.hs+23−10 src/Text/Pandoc/Writers/LaTeX/Types.hs+2−1 src/Text/Pandoc/Writers/LaTeX/Util.hs+275 src/Text/Pandoc/Writers/Man.hs+3−3 src/Text/Pandoc/Writers/Markdown.hs+83−662 src/Text/Pandoc/Writers/Markdown/Inline.hs+616 src/Text/Pandoc/Writers/Markdown/Types.hs+81 src/Text/Pandoc/Writers/MediaWiki.hs+1−1 src/Text/Pandoc/Writers/Ms.hs+28−10 src/Text/Pandoc/Writers/Muse.hs+6−3 src/Text/Pandoc/Writers/Native.hs+1−1 src/Text/Pandoc/Writers/ODT.hs+28−22 src/Text/Pandoc/Writers/OOXML.hs+22−22 src/Text/Pandoc/Writers/OPML.hs+1−1 src/Text/Pandoc/Writers/OpenDocument.hs+50−54 src/Text/Pandoc/Writers/Org.hs+39−19 src/Text/Pandoc/Writers/Powerpoint.hs+1−1 src/Text/Pandoc/Writers/Powerpoint/Output.hs+148−129 src/Text/Pandoc/Writers/Powerpoint/Presentation.hs+2−3 src/Text/Pandoc/Writers/RST.hs+17−13 src/Text/Pandoc/Writers/RTF.hs+17−11 src/Text/Pandoc/Writers/Roff.hs+1−1 src/Text/Pandoc/Writers/Shared.hs+19−4 src/Text/Pandoc/Writers/TEI.hs+5−4 src/Text/Pandoc/Writers/Texinfo.hs+23−16 src/Text/Pandoc/Writers/Textile.hs+1−1 src/Text/Pandoc/Writers/ZimWiki.hs+5−3 src/Text/Pandoc/XML.hs+42−20 src/Text/Pandoc/XML/Light.hs+89 src/Text/Pandoc/XML/Light/Output.hs+234 src/Text/Pandoc/XML/Light/Proc.hs+139 src/Text/Pandoc/XML/Light/Types.hs+193 stack.yaml+11−18 test/Tests/Command.hs+34−39 test/Tests/Helpers.hs+25−36 test/Tests/Lua.hs+1−3 test/Tests/Lua/Module.hs+3−1 test/Tests/Old.hs+12−26 test/Tests/Readers/Creole.hs+1−3 test/Tests/Readers/Docx.hs+25−7 test/Tests/Readers/DokuWiki.hs−2 test/Tests/Readers/EPUB.hs+2−4 test/Tests/Readers/FB2.hs−2 test/Tests/Readers/HTML.hs+21−8 test/Tests/Readers/JATS.hs+1−2 test/Tests/Readers/Jira.hs+22−2 test/Tests/Readers/LaTeX.hs+2−21 test/Tests/Readers/Man.hs+1−2 test/Tests/Readers/Markdown.hs+4−6 test/Tests/Readers/Muse.hs−2 test/Tests/Readers/Odt.hs+2−3 test/Tests/Readers/Org.hs+1−1 test/Tests/Readers/Org/Block.hs+1−3 test/Tests/Readers/Org/Block/CodeBlock.hs+4−6 test/Tests/Readers/Org/Block/Figure.hs+1−3 test/Tests/Readers/Org/Block/Header.hs+1−3 test/Tests/Readers/Org/Block/List.hs+14−3 test/Tests/Readers/Org/Block/Table.hs+1−3 test/Tests/Readers/Org/Directive.hs+1−3 test/Tests/Readers/Org/Inline.hs+3−5 test/Tests/Readers/Org/Inline/Citation.hs+41−3 test/Tests/Readers/Org/Inline/Note.hs+1−3 test/Tests/Readers/Org/Inline/Smart.hs+1−3 test/Tests/Readers/Org/Meta.hs+11−3 test/Tests/Readers/Org/Shared.hs+1−3 test/Tests/Readers/RST.hs+1−3 test/Tests/Readers/Txt2Tags.hs+1−3 test/Tests/Shared.hs+1−3 test/Tests/Writers/AnnotatedTable.hs−1 test/Tests/Writers/AsciiDoc.hs−2 test/Tests/Writers/ConTeXt.hs+122−115 test/Tests/Writers/Docbook.hs+43−8 test/Tests/Writers/Docx.hs+5−2 test/Tests/Writers/FB2.hs−2 test/Tests/Writers/HTML.hs+57−54 test/Tests/Writers/JATS.hs+135−93 test/Tests/Writers/Jira.hs+60−1 test/Tests/Writers/LaTeX.hs−2 test/Tests/Writers/Markdown.hs−2 test/Tests/Writers/Ms.hs−2 test/Tests/Writers/Muse.hs−1 test/Tests/Writers/Native.hs−2 test/Tests/Writers/OOXML.hs+5−4 test/Tests/Writers/Org.hs+46−13 test/Tests/Writers/Plain.hs−2 test/Tests/Writers/Powerpoint.hs−2 test/Tests/Writers/RST.hs−2 test/Tests/Writers/TEI.hs−2 test/command/1279.md+1−1 test/command/2103.md+8 test/command/2378.md+3−3 test/command/2874.md+2−2 test/command/3516.md+1−5 test/command/3752.md+60 test/command/3792.md+1−1 test/command/4063.md+29 test/command/4164.md+1−1 test/command/4442.md+1−1 test/command/4465.md+15 test/command/5304.md+1−1 test/command/5321.md+2−2 test/command/5367.md+7−3 test/command/5519.md+1−1 test/command/5686.md+1−1 test/command/5708.md+1−5 test/command/6026.md+19 test/command/6541.md+27 test/command/6620.md+3 test/command/6658.md+75 test/command/6709.md+11 test/command/6890.md+2−2 test/command/6925.md+2−2 test/command/6958.md+10 test/command/6993.md+21 test/command/7003.md+37 test/command/7006.md+20 test/command/7009.md+8 test/command/7016.md+48 test/command/7041.md+23 test/command/7042.md+146 test/command/7058.md+6 test/command/7064.md+32 test/command/7067.md+90 test/command/7080.md+8 test/command/7092.md+8 test/command/7099.md+18 test/command/7112.md+15 test/command/7129.md+27 test/command/7132.md+10 test/command/7134.md+16 test/command/7145.md+12 test/command/7155.md+15 test/command/7173.md+137 test/command/7208.md+6 test/command/7214.md+28 test/command/7216.md+19 test/command/7266.md+10 test/command/7272.md+27 test/command/7278.md+38 test/command/7282.md+7 test/command/7288.md+40 test/command/7299.md+23 test/command/7321.md+24 test/command/7323.md+29 test/command/7324.md+25 test/command/7339.md+11 test/command/7340.md+6 test/command/7376.md+16 test/command/7394.md+85 test/command/7397.md+14 test/command/7400.md+9 test/command/7416.md+19 test/command/averroes.bib+95 test/command/biblatex-baez-article.md+1−1 test/command/biblatex-baez-online.md+1−1 test/command/biblatex-basic.md+5−3 test/command/biblatex-cotton.md+3−3 test/command/biblatex-examples.bib−1674 test/command/biblatex-inproceedings.md+1−1 test/command/biblatex-itzhaki.md+1−1 test/command/biblatex-jaffe.md+2−2 test/command/biblatex-moraux.md+1−1 test/command/biblatex-murray.md+3−2 test/command/biblatex-nussbaum.md−63 test/command/biblatex-quotes.md+1−1 test/command/biblatex-sigfridsson.md+1−1 test/command/biblatex-spiegelberg.md+2−2 test/command/biblatex-test-case-conversion.md+2−2 test/command/biblatex-textnormal.md+1−1 test/command/biblatex-wassenberg.md+1−1 test/command/biblatex-wilde.md+1−1 test/command/chap1/spider.png test/command/chap1/text.md+17 test/command/chap2/spider.png test/command/chap2/text.md+5 test/command/defaults-inheritance-1.md+6 test/command/defaults-inheritance-2.md+5 test/command/defaults-inheritance-3.md+6 test/command/defaults3.yaml+4 test/command/defaults4.yaml+3 test/command/defaults5.yaml+2 test/command/defaults6.yaml+2 test/command/defaults7.yaml+2 test/command/defaults8.yaml+2 test/command/defaults9.yaml+1 test/command/duplicate_attributes.md+7 test/command/mmd-metadata.md+20 test/command/newif.md+55 test/command/pandoc-citeproc-119.md+1−1 test/command/pandoc-citeproc-320a.md+3−3 test/command/pandoc-citeproc-322.md+1−1 test/command/pandoc-citeproc-356.md+2−7 test/command/pandoc-citeproc-65.md+1−1 test/command/pandoc-citeproc-70.md+2−2 test/command/pandoc-citeproc-chicago-author-date.md+7−7 test/command/pandoc-citeproc-chicago-fullnote-bibliography.md+15−18 test/command/svg.md+15−17 test/command/toc.md+2−2 test/command/unicode-collation.md+130 test/docbook-reader.docbook+9−2 test/docx/0_level_headers.native+2−2 test/docx/golden/block_quotes.docx test/docx/golden/codeblock.docx test/docx/golden/comments.docx test/docx/golden/custom_style_no_reference.docx test/docx/golden/custom_style_preserve.docx test/docx/golden/custom_style_reference.docx test/docx/golden/definition_list.docx test/docx/golden/document-properties-short-desc.docx test/docx/golden/document-properties.docx test/docx/golden/headers.docx test/docx/golden/image.docx test/docx/golden/inline_code.docx test/docx/golden/inline_formatting.docx test/docx/golden/inline_images.docx test/docx/golden/link_in_notes.docx test/docx/golden/links.docx test/docx/golden/lists.docx test/docx/golden/lists_continuing.docx test/docx/golden/lists_multiple_initial.docx test/docx/golden/lists_restarting.docx test/docx/golden/nested_anchors_in_header.docx test/docx/golden/notes.docx test/docx/golden/raw-blocks.docx test/docx/golden/raw-bookmarks.docx test/docx/golden/table_one_row.docx test/docx/golden/table_with_list_cell.docx test/docx/golden/tables-default-widths.docx test/docx/golden/tables.docx test/docx/golden/track_changes_deletion.docx test/docx/golden/track_changes_insertion.docx test/docx/golden/track_changes_move.docx test/docx/golden/track_changes_scrubbed_metadata.docx test/docx/golden/unicode.docx test/docx/golden/verbatim_subsuper.docx test/docx/image_vml_as_object.docx test/docx/image_vml_as_object.native+2 test/docx/sdt_elements.native+9−10 test/docx/table_captions_no_field.docx test/docx/table_captions_no_field.native+34 test/docx/table_captions_with_field.docx test/docx/table_captions_with_field.native+54 test/docx/table_header_rowspan.docx test/docx/table_header_rowspan.native+189 test/docx/table_one_header_row.docx test/docx/table_one_header_row.native+18 test/docx/table_one_row.docx test/docx/table_one_row.native+4−4 test/docx/table_variable_width.native+12−17 test/docx/table_with_list_cell.native+3−3 test/docx/tables-default-widths.native+92 test/docx/tables.native+9−9 test/epub/features.native+2−2 test/epub/wasteland.epub test/fb2/images.fb2+1−1 test/fb2/test.jpg test/html-reader.native+2−1 test/jats-reader.native+1−1 test/jats-reader.xml+1 test/jira-reader.native+1−1 test/lhs-test.html+2−1 test/lhs-test.html+lhs+2−1 test/lhs-test.latex+3−3 test/lhs-test.latex+lhs+3−3 test/lua/module/pandoc-path.lua+44 test/markdown-reader-more.native+5−33 test/odt/native/tab.native+1 test/odt/odt/tab.odt test/pipe-tables.native+2−10 test/pptx/code-custom.pptx test/pptx/code-custom_templated.pptx test/pptx/code.pptx test/pptx/code_templated.pptx test/pptx/document-properties-short-desc.pptx test/pptx/document-properties-short-desc_templated.pptx test/pptx/document-properties.pptx test/pptx/document-properties_templated.pptx test/pptx/endnotes.pptx test/pptx/endnotes_templated.pptx test/pptx/endnotes_toc.pptx test/pptx/endnotes_toc_templated.pptx test/pptx/images.pptx test/pptx/images_templated.pptx test/pptx/inline_formatting.pptx test/pptx/inline_formatting_templated.pptx test/pptx/lists.pptx test/pptx/lists_templated.pptx test/pptx/raw_ooxml.pptx test/pptx/raw_ooxml_templated.pptx test/pptx/reference_depth.pptx test/pptx/remove_empty_slides.pptx test/pptx/remove_empty_slides_templated.pptx test/pptx/slide_breaks.pptx test/pptx/slide_breaks_slide_level_1.pptx test/pptx/slide_breaks_slide_level_1_templated.pptx test/pptx/slide_breaks_templated.pptx test/pptx/slide_breaks_toc.pptx test/pptx/slide_breaks_toc_templated.pptx test/pptx/speaker_notes.pptx test/pptx/speaker_notes_after_metadata.pptx test/pptx/speaker_notes_after_metadata_templated.pptx test/pptx/speaker_notes_afterheader.pptx test/pptx/speaker_notes_afterheader_templated.pptx test/pptx/speaker_notes_afterseps.pptx test/pptx/speaker_notes_afterseps_templated.pptx test/pptx/speaker_notes_templated.pptx test/pptx/start_numbering_at.pptx test/pptx/start_numbering_at_templated.pptx test/pptx/tables.pptx test/pptx/tables_templated.pptx test/pptx/two_column.pptx test/pptx/two_column_templated.pptx test/rst-reader.native+3−21 test/tables-rstsubset.native+1−9 test/tables.latex+50−35 test/tables.ms+44 test/tables/nordics.latex+28−12 test/tables/planets.latex+15−21 test/tables/students.latex+19−11 test/test-pandoc.hs+18−8 test/txt2tags.native+1−1 test/txt2tags.t2t+1−1 test/writer.context+42−40 test/writer.html4+2−1 test/writer.html5+2−1 test/writer.jats_archiving+10−8 test/writer.jats_articleauthoring+1−1 test/writer.jats_publishing+10−8 test/writer.jira+23−35 test/writer.latex+3−3 test/writer.ms+1−1 test/writer.opml+1−1 test/writer.org+7−7 test/writer.zimwiki+3−3 test/writers-lang-and-dir.context+4−2 test/writers-lang-and-dir.latex+6−6 tools/build-arm.sh+110 tools/changelog-helper.sh+11−4 tools/parseTimings.pl+33 trypandoc/index.html+1−1 trypandoc/trypandoc.hs+3−8 windows/Makefile+1−1 windows/pandoc.wxs+2−2
.circleci/config.yml+2−2
@@ -2,7 +2,7 @@ version: "2.1"
22jobs:
33 build:
44 macos:
5− xcode: 9.4.1
5+ xcode: 11.1.0
66 environment:
77 HOMEBREW_NO_AUTO_UPDATE: 1
88 steps:
@@ -14,7 +14,7 @@ jobs:
1414 - stack-dependencies-{{ checksum "stack.yaml" }}
1515 - run:
1616 name: Install stack
17− command: stack --version || brew update && brew install haskell-stack
17+ command: stack --version || (curl -sSL https://get.haskellstack.org/ | sh -s - -f)
1818 - run:
1919 name: Resolve/Update Dependencies
2020 command: stack --no-terminal setup
.editorconfig+8
@@ -8,6 +8,14 @@ indent_size = 2
88insert_final_newline = true
99trim_trailing_whitespace = true
1010
11+[*.{zip,docx}]
12+charset =
13+end_of_line =
14+indent_style =
15+indent_size =
16+insert_final_newline =
17+trim_trailing_whitespace =
18+
1119[*.{markdown,md}]
1220trim_trailing_whitespace = false
1321
.github/ISSUE_TEMPLATEdeleted−11
@@ -1,11 +0,0 @@
1−Asking a question?
2−Please use the pandoc-discuss mailing list instead:
3−https://groups.google.com/forum/#!forum/pandoc-discuss
4−This tracker is for bug reports and enhancement requests.
5−
6−Reporting a bug?
7−Please include your pandoc version and an example that allows us
8−to reproduce the problem (complete input, exact command line
9−used, expected and actual output). The smaller the example, the
10−better.
11−
.github/ISSUE_TEMPLATE/bug_report.mdnew file+25
@@ -0,0 +1,25 @@
1+---
2+name: Bug report
3+about: Create a report to help us improve
4+title: ''
5+labels: 'bug'
6+assignees: ''
7+
8+---
9+
10+<!--
11+Thank you for reporting an issue! Before you continue, please make sure that you have
12+
13+- searched the issue tracker for similar issues (including closed issues): https://github.com/jgm/pandoc/issues
14+- searched the pandoc-discuss mailing list for solutions: https://groups.google.com/forum/#!forum/pandoc-discuss
15+- reproduced your issue on the latest release of pandoc, or on https://pandoc.org/try
16+
17+Note that this bug tracker is for reporting bugs, not asking questions. For questions, use the pandoc-discuss mailing list.
18+-->
19+
20+**Explain the problem.**
21+Include the exact command line you used and all inputs necessary to reproduce the issue. Please create as minimal an example as possible, to help the maintainers isolate the problem. Explain the output you received and how it differs from what you expected.
22+
23+**Pandoc version?**
24+What version of pandoc are you using, on what OS?
25+
.github/ISSUE_TEMPLATE/config.ymlnew file+9
@@ -0,0 +1,9 @@
1+blank_issues_enabled: false
2+contact_links:
3+ - name: Ask a question
4+ url: https://groups.google.com/forum/#!forum/pandoc-discuss
5+ about: "Discussion forum for pandoc (may be used as an email list)"
6+ - name: Contributing Guidelines
7+ url: https://pandoc.org/CONTRIBUTING.html
8+ about: "Guidelines for submitting bug reports and code improvements"
9+
.github/ISSUE_TEMPLATE/feature_request.mdnew file+21
@@ -0,0 +1,21 @@
1+---
2+name: Feature request
3+about: Suggest an idea for this project
4+title: ''
5+labels: 'enhancement'
6+assignees: ''
7+
8+---
9+
10+<!--
11+Thank you for suggesting a feature! Before you continue, please make sure that you have
12+
13+- read the contributing guidelines: https://pandoc.org/CONTRIBUTING.html
14+- searched the issue tracker for similar issues (including closed issues): https://github.com/jgm/pandoc/issues
15+- searched the pandoc-discuss mailing list for relevant discussions: https://groups.google.com/forum/#!forum/pandoc-discuss
16+-->
17+
18+**Describe your proposed improvement and the problem it solves.**
19+
20+**Describe alternatives you've considered.**
21+
.github/workflows/ci.yml+92−8
@@ -45,23 +45,50 @@ jobs:
4545 versions:
4646 - ghc: '8.0.2'
4747 cabal: '2.4'
48+ prefix: 'v2-'
49+ cabalopts: ''
50+ testopts: ''
4851 - ghc: '8.2.2'
4952 cabal: '2.4'
53+ prefix: 'v2-'
54+ cabalopts: ''
55+ testopts: ''
5056 - ghc: '8.4.4'
5157 cabal: '2.4'
58+ prefix: 'v2-'
59+ cabalopts: '-f-embed_data_files'
60+ testopts: ''
5261 - ghc: '8.6.5'
5362 cabal: '3.2'
63+ prefix: ''
64+ cabalopts: ''
65+ testopts: '--test-option=--hide-successes'
5466 - ghc: '8.8.4'
5567 cabal: '3.2'
56− - ghc: '8.10.2'
68+ prefix: ''
69+ cabalopts: ''
70+ testopts: '--test-option=--hide-successes'
71+ - ghc: '8.10.4'
5772 cabal: '3.2'
73+ prefix: ''
74+ cabalopts: '-ftrypandoc'
75+ testopts: '--test-option=--hide-successes'
76+ - ghc: '9.0.1'
77+ cabal: '3.4'
78+ prefix: ''
79+ cabalopts: ''
80+ testopts: '--test-option=--hide-successes'
5881 steps:
5982 - uses: actions/checkout@v2
6083
84+ # needed by memory
85+ - name: Install numa
86+ run: sudo apt-get install libnuma-dev
87+
6188 # need to install older cabal/ghc versions from ppa repository
6289
6390 - name: Install recent cabal/ghc
64− uses: actions/setup-haskell@v1.1.3
91+ uses: haskell/actions/setup@v1
6592 with:
6693 ghc-version: ${{ matrix.versions.ghc }}
6794 cabal-version: ${{ matrix.versions.cabal }}
@@ -88,14 +115,15 @@ jobs:
88115
89116 - name: Install dependencies
90117 run: |
91− cabal v2-update
92− cabal v2-build --dependencies-only --enable-tests --disable-optimization
118+ cabal ${{ matrix.versions.prefix }}update
119+ cabal ${{ matrix.versions.prefix }}build ${{ matrix.versions.cabalopts }} --dependencies-only --enable-tests --disable-optimization
120+
93121 - name: Build and test
94122 run: |
95− cabal v2-build --enable-tests --disable-optimization 2>&1 | tee build.log
123+ cabal ${{ matrix.versions.prefix }}build ${{ matrix.versions.cabalopts }} --enable-tests --disable-optimization 2>&1 | tee build.log
96124 # fail if warnings in local build
97125 ! grep -q ": *[Ww]arning:" build.log || exit 1
98− cabal v2-test --disable-optimization
126+ cabal ${{ matrix.versions.prefix }}test ${{ matrix.versions.cabalopts }} --disable-optimization ${{ matrix.versions.testopts }}
99127
100128 windows:
101129
@@ -105,7 +133,7 @@ jobs:
105133 - uses: actions/checkout@v2
106134
107135 # We should have ghc 8.10, cabal, stack pre-installed
108− # - uses: actions/setup-haskell@v1.1.3
136+ # - uses: haskell/actions/setup@v1
109137 # with:
110138 # enable-stack: true
111139 # stack-version: 'latest'
@@ -154,7 +182,7 @@ jobs:
154182 - uses: actions/checkout@v2
155183
156184 - name: Install recent cabal/ghc
157− uses: actions/setup-haskell@v1.1.3
185+ uses: haskell/actions/setup@v1
158186 with:
159187 ghc-version: ${{ matrix.versions.ghc }}
160188 cabal-version: ${{ matrix.versions.cabal }}
@@ -189,3 +217,59 @@ jobs:
189217 # fail if warnings in local build
190218 ! grep -q ": *[Ww]arning:" build.log || exit 1
191219 cabal v2-test --disable-optimization
220+
221+ benchmark:
222+
223+ runs-on: ubuntu-18.04
224+ strategy:
225+ fail-fast: true
226+ matrix:
227+ versions:
228+ - ghc: '8.8.4'
229+ cabal: '3.2'
230+ - ghc: '8.10.2'
231+ cabal: '3.2'
232+ steps:
233+ - uses: actions/checkout@v2
234+
235+ # need to install older cabal/ghc versions from ppa repository
236+
237+ - name: Install recent cabal/ghc
238+ uses: haskell/actions/setup@v1
239+ with:
240+ ghc-version: ${{ matrix.versions.ghc }}
241+ cabal-version: ${{ matrix.versions.cabal }}
242+
243+ # declare/restore cached things
244+ # caching doesn't work for scheduled runs yet
245+ # https://github.com/actions/cache/issues/63
246+
247+ - name: Cache cabal global package db
248+ id: cabal-global
249+ uses: actions/cache@v2
250+ with:
251+ path: |
252+ ~/.cabal
253+ key: benchmark-${{ runner.os }}-${{ matrix.versions.ghc }}-${{ matrix.versions.cabal }}-cabal-global-${{ hashFiles('cabal.project') }}
254+
255+ - name: Cache cabal work
256+ id: cabal-local
257+ uses: actions/cache@v2
258+ with:
259+ path: |
260+ dist-newstyle
261+ key: benchmark-${{ runner.os }}-${{ matrix.versions.ghc }}-${{ matrix.versions.cabal }}-cabal-local
262+
263+ - name: Install dependencies
264+ run: |
265+ v2=$([ "${{ matrix.versions.cabal }}" = "2.2" ] && printf 'new' || printf 'v2')
266+ cabal $v2-update
267+ cabal $v2-build --dependencies-only --enable-optimization=1 --enable-benchmarks --disable-tests
268+
269+ - name: Build and test
270+ run: |
271+ v2=$([ "${{ matrix.versions.cabal }}" = "2.2" ] && printf 'new' || printf 'v2')
272+ cabal $v2-build --enable-optimization=1 --enable-benchmarks --disable-tests 2>&1 | tee build.log
273+ # fail if warnings in local build
274+ ! grep -q ": *[Ww]arning:" build.log || exit 1
275+ cabal $v2-bench --enable-optimization=1 --benchmark-options='--timeout=6 +RTS -T -RTS'
.github/workflows/nightly.yml+9−2
@@ -10,15 +10,22 @@ jobs:
1010 runs-on: ubuntu-18.04
1111 steps:
1212 - uses: actions/checkout@v2
13+
14+ - name: Install recent cabal/ghc
15+ uses: haskell/actions/setup@v1
16+ with:
17+ ghc-version: '8.10.3'
18+ cabal-version: '3.2'
19+
1320 - name: Install dependencies
1421 run: |
1522 ghc --version
1623 cabal --version
1724 cabal v2-update
18− cabal v2-build --dependencies-only all
25+ cabal v2-build --dependencies-only
1926 - name: Build
2027 run: |
21− cabal v2-install all
28+ cabal v2-install
2229 strip $HOME/.cabal/bin/pandoc
2330 - name: Install artifact
2431 run: |
.github/workflows/release-candidate.yml−1
@@ -32,7 +32,6 @@ jobs:
3232 - osarch: "windows-x86_64"
3333 arch: "x64"
3434 stack_opts: "-j1"
35−
3635 steps:
3736 - uses: actions/checkout@v1
3837 - name: Install dependencies
.gitignore+7
@@ -1,4 +1,6 @@
11*~
2+.DS_Store
3+*.orig
24deb/.vagrant
35dist/*
46MANUAL.*
@@ -14,6 +16,10 @@ man/man?/*.html
1416*.o
1517*.hi
1618*.pyc
19+.hie/
20+.policeman-evidence/
21+.projectile/
22+.vscode/
1723/COPYING.rtf
1824/COPYRIGHT.txt
1925/cabal-dev/
@@ -26,3 +32,4 @@ data/reference.odt
2632/stack.yaml.lock
2733cabal.project.local
2834/dist-newstyle/
35+pandoc-2*/
.hlint.yaml+9−5
@@ -9,11 +9,13 @@
99# Ignore some builtin hints
1010#
1111- ignore: {name: "Avoid lambda"}
12+- ignore: {name: "Use bimap"}
1213- ignore: {name: "Eta reduce"}
1314- ignore: {name: "Evaluate"}
1415- ignore: {name: "Reduce duplication"} # TODO: could be more fine-grained
1516- ignore: {name: "Use &&&"}
1617- ignore: {name: "Use String"}
18+- ignore: {name: "Use camelCase"}
1719- ignore: {name: "Use fmap"} # specific for GHC 7.8 compat
1820- ignore: {name: "Use isDigit"}
1921
@@ -38,13 +40,10 @@
3840 name: "Redundant return"
3941 within: Text.Pandoc.Citeproc.BibTeX
4042
41−# TODO: check
43+# With recent hlint this rule has given false positives on TH
4244- ignore:
4345 name: "Redundant bracket"
44− within:
45− - Text.Pandoc.Citeproc
46− - Text.Pandoc.Citeproc.BibTeX
47− - Text.Pandoc.Citeproc.MetaValue
46+
4847
4948- ignore:
5049 name: "Use <$>"
@@ -60,6 +59,7 @@
6059 - Text.Pandoc.Citeproc
6160 - Text.Pandoc.Extensions
6261 - Text.Pandoc.Lua.Marshaling.Version
62+ - Text.Pandoc.Lua.Module.Pandoc
6363 - Text.Pandoc.Lua.Module.Utils
6464 - Text.Pandoc.Readers.Odt.ContentReader
6565 - Text.Pandoc.Readers.Odt.Namespaces
@@ -104,5 +104,9 @@
104104 - Text.Pandoc.Readers.Markdown
105105 - Text.Pandoc.Readers.RST
106106
107+# fromRight is only in base >= 4.10
108+- ignore:
109+ name: "Use fromRight"
110+
107111# Define some custom infix operators
108112# - fixity: infixr 3 ~^#^~
AUTHORS.md+23
@@ -20,6 +20,7 @@
2020- Andreas Lööw
2121- Andrew Dunning
2222- Andy Morris
23+- Anti-Distinctlyminty
2324- Antoine Latter
2425- Antonio Terceiro
2526- Arata Mizuki
@@ -35,6 +36,7 @@
3536- Benoit Schweblin
3637- Bjorn Buckwalter
3738- Blake Eryx
39+- Bodigrim
3840- Bradley Kuhn
3941- Brent Yorgey
4042- Brian Leung
@@ -45,6 +47,7 @@
4547- Carlos Sosa
4648- Cécile Chemin
4749- Cédric Couralet
50+- Charanjit Singh
4851- Chris Black
4952- Christian Conkle
5053- Christian Despres
@@ -62,19 +65,23 @@
6265- Daniele D'Orazio
6366- David A Roberts
6467- David Lazar
68+- David Martschenko
6569- David Röthlisberger
6670- Denis Laxalde
6771- Denis Maier
6872- Derek Chen-Becker
6973- Diego Balseiro
74+- Dimitri Sabadie
7075- Dmitry Pogodin
7176- Dmitry Volodin
7277- Douglas Calvert
7378- Eigil Rischel
7479- Emanuel Evans
7580- Emerson Harkin
81+- Emily Bourke
7682- Emily Eisenberg
7783- Eric Kow
84+- Erik Rask
7885- Eric Seidel
7986- Eric Schrijver
8087- Ethan Riley
@@ -84,6 +91,7 @@
8491- Florian Beeres
8592- Florian Eitel
8693- Florian Klink
94+- Florian Kohrt
8795- Francesco Occhipinti
8896- François Gannaz
8997- Frederik Elwert
@@ -128,6 +136,7 @@
128136- Jens Getreu
129137- Jens Petersen
130138- Jeroen de Haas
139+- Jerry Sky
131140- Jesse Rosenthal
132141- Joe Hermaszewski
133142- Joe Hillenbrand
@@ -143,6 +152,7 @@
143152- Josef Svenningsson
144153- Joseph C. Sible
145154- Julien Cretel
155+- Julien Dutant
146156- Juliette Fourcot
147157- Juliusz Gonera
148158- Justin Bogner
@@ -160,6 +170,8 @@
160170- Leif Metcalf
161171- Leonard Rosenthol
162172- Lila
173+- Loïc Grobol
174+- Lorenzo
163175- Lucas Escot
164176- Luke Plant
165177- Marc Schreiber
@@ -193,10 +205,12 @@
193205- Nathan Gass
194206- Neil Mayhew
195207- Nick Bart
208+- Nick Berendsen
196209- Nick Fleisher
197210- Nicolas Kaiser
198211- Nikolay Yakimov
199212- Nils Carlson
213+- Nixon Enraght-Moony
200214- Nokome Bentley
201215- Ole Martin Ruud
202216- Oliver Matthews
@@ -221,6 +235,7 @@
221235- Raymond Ehlers
222236- Recai Oktaş
223237- Roland Hieber
238+- Roman Beránek
224239- RyanGlScott
225240- Salim B
226241- Sascha Wilde
@@ -238,6 +253,7 @@
238253- Stefan Dresselhaus
239254- Sumit Sahrawat
240255- Tarik Graba
256+- Tatiana Porras
241257- Thenaesh Elango
242258- Thomas Hodgson
243259- Thomas Weißschuh
@@ -271,6 +287,7 @@
271287- a-vrma
272288- andrebauer
273289- argent0
290+- badumont
274291- blmage
275292- bucklereed
276293- bumper314
@@ -286,8 +303,11 @@
286303- kaizshang91
287304- lux-lth
288305- lwolfsonkin
306+- mbrackeantidot
289307- nkalvi
290308- niszet
309+- nuew
310+- obcat
291311- oltolm
292312- oquechy
293313- quasicomputational
@@ -299,9 +319,12 @@
299319- shreevatsa.public
300320- takahashim
301321- taotieren
322+- tecosaur
302323- TEC
303324- tgkokk
304325- the-solipsist
326+- timo-a
305327- thsutton
306328- vijayphoenix
307329- wiefling
330+- wuffi
CONTRIBUTING.md+27−19
@@ -244,9 +244,16 @@ Or with stack:
244244
245245 stack test --test-arguments='-p markdown'
246246
247−It is often helpful to add `-j4` (run tests in parallel)
248−and `--hide-successes` (don't clutter output with successes)
249−to the test arguments as well.
247+It is often helpful to add `-j4` (run tests in parallel) and
248+`--hide-successes` (don't clutter output with successes) to the test
249+arguments as well. Collecting all options in a `cabal.project.local`
250+file in the project's root directory can help to keep `cabal`
251+commands short. E.g.:
252+
253+ flags: +embed_data_files
254+ tests: True
255+ test-show-details: direct
256+ test-options: -j4 --hide-successes
250257
251258If you add a new feature to pandoc, please add tests as well, following
252259the pattern of the existing tests. The test suite code is in
@@ -291,6 +298,9 @@ before creating a commit:
291298 git diff --diff-filter=MA --cached --name-only | grep '\.hs$' | \
292299 xargs hlint --hint .hlint.yaml
293300
301+(If you are using GNU `xargs`, add the `-r` option immediately
302+after `xargs`.)
303+
294304Saving this to `.git/hooks/pre-commit`, and making the script
295305executable, will prevent accidental introduction of potentially
296306problematic code.
@@ -308,9 +318,6 @@ With stack:
308318
309319 stack bench
310320
311−You can also build `weigh-pandoc` (`stack build pandoc:weigh-pandoc`)
312−to get some statistics on memory usage. (Eventually this should
313−be incorporated into the benchmark suite.)
314321
315322Using the REPL
316323--------------
@@ -369,7 +376,7 @@ The code
369376--------
370377
371378Pandoc has a publicly accessible git repository on
372−GitHub: <http://github.com/jgm/pandoc>. To get a local copy of the source:
379+GitHub: <https://github.com/jgm/pandoc>. To get a local copy of the source:
373380
374381 git clone https://github.com/jgm/pandoc.git
375382
@@ -379,7 +386,7 @@ the pandoc library is in `src/`, the source for the tests is in
379386
380387The modules `Text.Pandoc.Definition`, `Text.Pandoc.Builder`, and
381388`Text.Pandoc.Generic` are in a separate library `pandoc-types`. The code can
382−be found in <http://github.com/jgm/pandoc-types>.
389+be found in <https://github.com/jgm/pandoc-types>.
383390
384391To build pandoc, you will need a working installation of the
385392[Haskell platform].
@@ -399,11 +406,11 @@ The library is structured as follows:
399406 tree automatically.
400407 - `Text.Pandoc.Readers.*` are the readers, and `Text.Pandoc.Writers.*`
401408 are the writers.
402− - `Text.Pandoc.Biblio` is a utility module for formatting citations
403− using citeproc-hs.
409+ - `Text.Pandoc.Citeproc.*` contain the code for citation handling,
410+ including an interface to the [citeproc] library.
404411 - `Text.Pandoc.Data` is used to embed data files when the `embed_data_files`
405− cabal flag is used. It is generated from `src/Text/Pandoc/Data.hsb` using
406− the preprocessor [hsb2hs].
412+ cabal flag is used.
413+ - `Text.Pandoc.Emoji` is a thin wrapper around [emojis].
407414 - `Text.Pandoc.Highlighting` contains the interface to the
408415 skylighting library, which is used for code syntax highlighting.
409416 - `Text.Pandoc.ImageSize` is a utility module containing functions for
@@ -414,7 +421,6 @@ The library is structured as follows:
414421 - `Text.Pandoc.PDF` contains functions for producing a PDF from a
415422 LaTeX source.
416423 - `Text.Pandoc.Parsing` contains parsing functions used in multiple readers.
417− - `Text.Pandoc.Pretty` is a pretty-printing library specialized to
418424 the needs of pandoc.
419425 - `Text.Pandoc.SelfContained` contains functions for making an HTML
420426 file "self-contained," by importing remotely linked images, CSS,
@@ -442,14 +448,16 @@ you may want to consider submitting a pull request to the
442448[closed issues]: https://github.com/jgm/pandoc/issues?q=is%3Aissue+is%3Aclosed
443449[latest released version]: https://github.com/jgm/pandoc/releases/latest
444450[Nightly builds]: https://github.com/jgm/pandoc/actions?query=workflow%3ANightly
445−[pandoc-discuss]: http://groups.google.com/group/pandoc-discuss
451+[pandoc-discuss]: https://groups.google.com/group/pandoc-discuss
446452[issue tracker]: https://github.com/jgm/pandoc/issues
447−[User's Guide]: http://pandoc.org/MANUAL.html
448−[FAQs]: http://pandoc.org/faqs.html
449−[EditorConfig]: http://editorconfig.org/
450−[Haskell platform]: http://www.haskell.org/platform/
453+[User's Guide]: https://pandoc.org/MANUAL.html
454+[FAQs]: https://pandoc.org/faqs.html
455+[EditorConfig]: https://editorconfig.org/
456+[Haskell platform]: https://www.haskell.org/platform/
451457[hlint]: https://hackage.haskell.org/package/hlint
452−[hsb2hs]: http://hackage.haskell.org/package/hsb2hs
458+[citeproc]: https://hackage.haskell.org/package/citeproc
459+[emojis]: https://hackage.haskell.org/package/emojis
460+[hsb2hs]: https://hackage.haskell.org/package/hsb2hs
453461[pre-commit hook]: https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks
454462[GitHub labels]: https://github.com/jgm/pandoc/labels
455463[good first issue]:https://github.com/jgm/pandoc/labels/good%20first%20issue
COPYING.md+1−1
@@ -357,5 +357,5 @@ into proprietary programs. If your program is a subroutine library,
357357you may consider it more useful to permit linking proprietary
358358applications with the library. If this is what you want to do, use the
359359[GNU Lesser General Public
360−License](http://www.gnu.org/licenses/lgpl.html) instead of this
360+License](https://www.gnu.org/licenses/lgpl.html) instead of this
361361License.
COPYRIGHT+17−26
@@ -1,5 +1,5 @@
11Pandoc
2−Copyright (C) 2006-2020 John MacFarlane <jgm at berkeley dot edu>
2+Copyright (C) 2006-2021 John MacFarlane <jgm at berkeley dot edu>
33
44With the exceptions noted below, this code is released under the [GPL],
55version 2 or later:
@@ -22,11 +22,11 @@ The GNU General Public License is available in the file COPYING.md in
2222the source distribution. On Debian systems, the complete text of the
2323GPL can be found in `/usr/share/common-licenses/GPL`.
2424
25−[GPL]: http://www.gnu.org/copyleft/gpl.html
25+[GPL]: https://www.gnu.org/copyleft/gpl.html
2626
2727Pandoc's complete source code is available from the [Pandoc home page].
2828
29−[Pandoc home page]: http://pandoc.org
29+[Pandoc home page]: https://pandoc.org
3030
3131Pandoc includes some code with different copyrights, or subject to different
3232licenses. The copyright and license statements for these sources are included
@@ -37,7 +37,7 @@ The modules in the `pandoc-types` repository (Text.Pandoc.Definition,
3737Text.Pandoc.Builder, Text.Pandoc.Generics, Text.Pandoc.JSON,
3838Text.Pandoc.Walk) are licensed under the BSD 3-clause license:
3939
40−Copyright (c) 2006-2020, John MacFarlane
40+Copyright (c) 2006-2021, John MacFarlane
4141
4242All rights reserved.
4343
@@ -73,7 +73,7 @@ Pandoc's templates (in `data/templates`) are dual-licensed as either
7373GPL (v2 or higher, same as pandoc) or (at your option) the BSD
74743-clause license.
7575
76−Copyright (c) 2014--2020, John MacFarlane
76+Copyright (c) 2014--2021, John MacFarlane
7777
7878----------------------------------------------------------------------
7979src/Text/Pandoc/Writers/Muse.hs
@@ -83,19 +83,19 @@ Released under the GNU General Public License version 2 or later.
8383
8484----------------------------------------------------------------------
8585src/Text/Pandoc/Writers/Texinfo.hs
86−Copyright (C) 2008-2020 John MacFarlane and Peter Wang
86+Copyright (C) 2008-2021 John MacFarlane and Peter Wang
8787
8888Released under the GNU General Public License version 2 or later.
8989
9090----------------------------------------------------------------------
9191src/Text/Pandoc/Writers/OpenDocument.hs
92−Copyright (C) 2008-2020 Andrea Rossato and John MacFarlane
92+Copyright (C) 2008-2021 Andrea Rossato and John MacFarlane
9393
9494Released under the GNU General Public License version 2 or later.
9595
9696----------------------------------------------------------------------
9797src/Text/Pandoc/Writers/Org.hs
98−Copyright (C) 2010-2020 Puneeth Chaganti, John MacFarlane, and
98+Copyright (C) 2010-2021 Puneeth Chaganti, John MacFarlane, and
9999 Albert Krewinkel
100100
101101Released under the GNU General Public License version 2 or later.
@@ -115,7 +115,7 @@ Released under the GNU General Public License version 2 or later.
115115
116116----------------------------------------------------------------------
117117src/Text/Pandoc/Readers/Textile.hs
118−Copyright (C) 2010-2020 Paul Rivier and John MacFarlane
118+Copyright (C) 2010-2021 Paul Rivier and John MacFarlane
119119
120120Released under the GNU General Public License version 2 or later.
121121
@@ -133,7 +133,7 @@ Released under the GNU General Public License version 2 or later.
133133
134134----------------------------------------------------------------------
135135src/Text/Pandoc/Readers/EPUB.hs
136−Copyright (C) 2014-2020 Matthew Pickering and John MacFarlane
136+Copyright (C) 2014-2021 Matthew Pickering and John MacFarlane
137137
138138Released under the GNU General Public License version 2 or later.
139139
@@ -141,7 +141,7 @@ Released under the GNU General Public License version 2 or later.
141141src/Text/Pandoc/Readers/Org.hs
142142src/Text/Pandoc/Readers/Org/*
143143test/Tests/Readers/Org/*
144−Copyright (C) 2014-2020 Albert Krewinkel
144+Copyright (C) 2014-2021 Albert Krewinkel
145145
146146Released under the GNU General Public License version 2 or later.
147147
@@ -149,7 +149,7 @@ Released under the GNU General Public License version 2 or later.
149149src/Text/Pandoc/Lua.hs
150150src/Text/Pandoc/Lua/*
151151test/lua/*
152−Copyright (C) 2017--2020 Albert Krewinkel and John MacFarlane
152+Copyright (C) 2017--2021 Albert Krewinkel and John MacFarlane
153153
154154Released under the GNU General Public License version 2 or later.
155155
@@ -157,7 +157,7 @@ Released under the GNU General Public License version 2 or later.
157157src/Text/Pandoc/Readers/Jira.hs
158158src/Text/Pandoc/Writers/Jira.hs
159159test/Tests/Readers/Jira.hs
160−Copyright (C) 2019--2020 Albert Krewinkel
160+Copyright (C) 2019--2021 Albert Krewinkel
161161
162162Released under the GNU General Public License version 2 or later.
163163
@@ -169,21 +169,21 @@ Released under the GNU General Public License version 2 or later.
169169
170170----------------------------------------------------------------------
171171data/pandoc.lua
172−Copyright (C) 2017-2020 Albert Krewinkel
172+Copyright (C) 2017-2021 Albert Krewinkel
173173
174174Released under the GNU General Public License version 2 or later.
175175
176176----------------------------------------------------------------------
177177The dzslides template contains JavaScript and CSS from Paul Rouget's
178178dzslides template.
179−http://github.com/paulrouget/dzslides
179+https://github.com/paulrouget/dzslides
180180
181181Released under the Do What the Fuck You Want To Public License.
182182
183183------------------------------------------------------------------------
184−Pandoc embeds a lua interpreter (via hslua).
184+Pandoc embeds a Lua interpreter (via hslua).
185185
186−Copyright © 1994–2019 Lua.org, PUC-Rio.
186+Copyright © 1994–2020 Lua.org, PUC-Rio.
187187
188188Permission is hereby granted, free of charge, to any person obtaining
189189a copy of this software and associated documentation files (the
@@ -203,12 +203,3 @@ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
203203LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
204204OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
205205WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
206−
207−------------------------------------------------------------------------
208−The template pandoc.jats is Copyright 2013--2015 Martin Fenner,
209−released under GPL version 2 or later.
210−
211−The file data/jats.csl is derived from a csl file by Martin Fenner,
212−revised by Martin Paul Eve and then John MacFarlane.
213−"This work is licensed under a Creative Commons Attribution-ShareAlike 3.0
214−License. Originally by Martin Fenner."
INSTALL.md+18−15
@@ -34,6 +34,10 @@ without SVG support), [Python] (to use Pandoc filters), and
3434
3535 choco install rsvg-convert python miktex
3636
37+Using multiple installation methods can result in two separate
38+installations of pandoc; it is recommended to properly uninstall
39+pandoc before switching to an alternative installation method.
40+
3741By default, Pandoc creates PDFs using LaTeX.
3842We recommend installing it via [MiKTeX].
3943
@@ -45,7 +49,7 @@ by downloading [this script][uninstaller]
4549and running it with `perl uninstall-pandoc.pl`.
4650
4751Alternatively, you can install pandoc using
48−[Homebrew](http://brew.sh):
52+[Homebrew](https://brew.sh):
4953
5054 brew install pandoc
5155
@@ -116,7 +120,7 @@ package repositories. For example, on Debian/Ubuntu,
116120you can install it with `apt-get install haskell-platform`.
117121
118122For PDF output, you'll need LaTeX. We recommend installing
119−[TeX Live](http://www.tug.org/texlive/) via your package
123+[TeX Live](https://www.tug.org/texlive/) via your package
120124manager. (On Debian/Ubuntu, `apt-get install texlive`.)
121125
122126## Chrome OS
@@ -238,7 +242,7 @@ The easiest way to build pandoc from source is to use [stack][stack]:
238242
239243 pandoc --help
240244
241− [Not sure where `$CABALDIR` is?](http://www.haskell.org/haskellwiki/Cabal-Install#The_cabal-install_configuration_file)
245+ [Not sure where `$CABALDIR` is?](https://wiki.haskell.org/Cabal-Install#The_cabal-install_configuration_file)
242246
2432475. By default `pandoc` uses the "i;unicode-casemap" method
244248 to sort bibliography entries (RFC 5051). If you would like to
@@ -383,30 +387,29 @@ To run just the markdown benchmarks:
383387
384388
385389[Arch]: https://www.archlinux.org/packages/community/x86_64/pandoc/
386−[Cabal User's Guide]: http://www.haskell.org/cabal/release/latest/doc/users-guide/builders.html#setup-configure-paths
390+[Cabal User's Guide]: https://cabal.readthedocs.io/
387391[Debian]: https://packages.debian.org/pandoc
388392[Fedora]: https://apps.fedoraproject.org/packages/pandoc
389−[FreeBSD ports]: http://www.freshports.org/textproc/hs-pandoc/
390−[GHC]: http://www.haskell.org/ghc/
391−[GPL]: http://www.gnu.org/copyleft/gpl.html
392−[Haskell platform]: http://hackage.haskell.org/platform/
393−[MacPorts]: http://trac.macports.org/browser/trunk/dports/textproc/pandoc/Portfile
393+[FreeBSD ports]: https://www.freshports.org/textproc/hs-pandoc/
394+[GHC]: https://www.haskell.org/ghc/
395+[Haskell platform]: https://hackage.haskell.org/platform/
396+[MacPorts]: https://trac.macports.org/browser/trunk/dports/textproc/pandoc/Portfile
394397[MacTeX]: https://tug.org/mactex/
395−[BasicTeX]: http://www.tug.org/mactex/morepackages.html
398+[BasicTeX]: https://www.tug.org/mactex/morepackages.html
396399[LaTeX]: https://www.latex-project.org
397−[MiKTeX]: http://miktex.org/
400+[MiKTeX]: https://miktex.org/
398401[librsvg]: https://wiki.gnome.org/Projects/LibRsvg
399402[Python]: https://www.python.org
400−[NetBSD]: http://pkgsrc.se/wip/pandoc
403+[NetBSD]: https://pkgsrc.se/wip/pandoc
401404[NixOS]: https://nixos.org/nixos/packages.html
402405[Slackware]: https://www.slackbuilds.org/result/?search=pandoc&sv=
403406[Ubuntu]: https://packages.ubuntu.com/pandoc
404407[download page]: https://github.com/jgm/pandoc/releases/latest
405−[gentoo]: http://packages.gentoo.org/package/app-text/pandoc
408+[gentoo]: https://packages.gentoo.org/package/app-text/pandoc
406409[haskell repository]: https://wiki.archlinux.org/index.php/Haskell_Package_Guidelines#.5Bhaskell.5D
407410[openSUSE]: https://software.opensuse.org/package/pandoc
408−[source tarball]: http://hackage.haskell.org/package/pandoc
411+[source tarball]: https://hackage.haskell.org/package/pandoc
409412[stack]: https://docs.haskellstack.org/en/stable/install_and_upgrade.html
410−[cabal-install]: http://hackage.haskell.org/package/cabal-install
413+[cabal-install]: https://hackage.haskell.org/package/cabal-install
411414[Void]: https://voidlinux.org/
412415[uninstaller]: https://raw.githubusercontent.com/jgm/pandoc/master/macos/uninstall-pandoc.pl
MANUAL.txt+319−100
@@ -1,7 +1,7 @@
11---
22title: Pandoc User's Guide
33author: John MacFarlane
4−date: December 17, 2020
4+date: June 20, 2021
55---
66
77# Synopsis
@@ -136,7 +136,7 @@ to output the generated LaTeX. You can then test it with `pdflatex test.tex`.
136136When using LaTeX, the following packages need to be available
137137(they are included with all recent versions of [TeX Live]):
138138[`amsfonts`], [`amsmath`], [`lm`], [`unicode-math`],
139−[`ifxetex`], [`ifluatex`], [`listings`] (if the
139+[`iftex`], [`listings`] (if the
140140`--listings` option is used), [`fancyvrb`], [`longtable`],
141141[`booktabs`], [`graphicx`] (if the document
142142contains images), [`hyperref`], [`xcolor`],
@@ -180,8 +180,7 @@ footnotes in tables).
180180[`graphicx`]: https://ctan.org/pkg/graphicx
181181[`grffile`]: https://ctan.org/pkg/grffile
182182[`hyperref`]: https://ctan.org/pkg/hyperref
183−[`ifluatex`]: https://ctan.org/pkg/ifluatex
184−[`ifxetex`]: https://ctan.org/pkg/ifxetex
183+[`iftex`]: https://ctan.org/pkg/iftex
185184[`listings`]: https://ctan.org/pkg/listings
186185[`lm`]: https://ctan.org/pkg/lm
187186[`longtable`]: https://ctan.org/pkg/longtable
@@ -281,6 +280,8 @@ header when requesting a document from a URL:
281280 ::: {#output-formats}
282281 - `asciidoc` ([AsciiDoc]) or `asciidoctor` ([AsciiDoctor])
283282 - `beamer` ([LaTeX beamer][`beamer`] slide show)
283+ - `bibtex` ([BibTeX] bibliography)
284+ - `biblatex` ([BibLaTeX] bibliography)
284285 - `commonmark` ([CommonMark] Markdown)
285286 - `commonmark_x` ([CommonMark] Markdown with extensions)
286287 - `context` ([ConTeXt])
@@ -360,15 +361,15 @@ header when requesting a document from a URL:
360361 will be used. On \*nix and macOS systems this will be the `pandoc`
361362 subdirectory of the XDG data directory (by default,
362363 `$HOME/.local/share`, overridable by setting the `XDG_DATA_HOME`
363− environment variable). If that directory does not exist,
364− `$HOME/.pandoc` will be used (for backwards compatibility).
365− In Windows the default user data directory is
364+ environment variable). If that directory does not exist and
365+ `$HOME/.pandoc` exists, it will be used (for backwards compatibility).
366+ On Windows the default user data directory is
366367 `C:\Users\USERNAME\AppData\Roaming\pandoc`.
367368 You can find the default user data directory on your system by
368369 looking at the output of `pandoc --version`.
369− A `reference.odt`, `reference.docx`, `epub.css`, `templates`,
370− `slidy`, `slideous`, or `s5` directory
371− placed in this directory will override pandoc's normal defaults.
370+ Data files placed in this directory (for example, `reference.odt`,
371+ `reference.docx`, `epub.css`, `templates`) will override
372+ pandoc's normal defaults.
372373
373374`-d` *FILE*, `--defaults=`*FILE*
374375
@@ -485,7 +486,7 @@ header when requesting a document from a URL:
485486[Emacs Org mode]: https://orgmode.org
486487[AsciiDoc]: https://www.methods.co.nz/asciidoc/
487488[AsciiDoctor]: https://asciidoctor.org/
488−[DZSlides]: http://paulrouget.com/dzslides/
489+[DZSlides]: https://paulrouget.com/dzslides/
489490[Word docx]: https://en.wikipedia.org/wiki/Office_Open_XML
490491[PDF]: https://www.adobe.com/pdf/
491492[reveal.js]: https://revealjs.com/
@@ -588,8 +589,8 @@ header when requesting a document from a URL:
588589
589590 3. `$PATH` (executable only)
590591
591− Filters and Lua-filters are applied in the order specified
592− on the command line.
592+ Filters, Lua-filters, and citeproc processing are applied in
593+ the order specified on the command line.
593594
594595`-L` *SCRIPT*, `--lua-filter=`*SCRIPT*
595596
@@ -603,26 +604,17 @@ header when requesting a document from a URL:
603604 The `pandoc` Lua module provides helper functions for element
604605 creation. It is always loaded into the script's Lua environment.
605606
606− The following is an example Lua script for macro-expansion:
607−
608− function expand_hello_world(inline)
609− if inline.c == '{{helloworld}}' then
610− return pandoc.Emph{ pandoc.Str "Hello, World" }
611− else
612− return inline
613− end
614− end
615−
616− return {{Str = expand_hello_world}}
607+ See the [Lua filters documentation] for further details.
617608
618609 In order of preference, pandoc will look for Lua filters in
619610
620− 1. a specified full or relative path (executable or
621− non-executable)
611+ 1. a specified full or relative path
622612
623− 2. `$DATADIR/filters` (executable or non-executable)
624− where `$DATADIR` is the user data directory (see
625− `--data-dir`, above).
613+ 2. `$DATADIR/filters` where `$DATADIR` is the user data
614+ directory (see `--data-dir`, above).
615+
616+ Filters, Lua filters, and citeproc processing are applied in
617+ the order specified on the command line.
626618
627619`-M` *KEY*[`=`*VAL*], `--metadata=`*KEY*[`:`*VAL*]
628620
@@ -683,12 +675,12 @@ header when requesting a document from a URL:
683675: Extract images and other media contained in or linked from
684676 the source document to the path *DIR*, creating it if
685677 necessary, and adjust the images references in the document
686− so they point to the extracted files. If the source format is
687− a binary container (docx, epub, or odt), the media is
688− extracted from the container and the original
689− filenames are used. Otherwise the media is read from the
690− file system or downloaded, and new filenames are constructed
691− based on SHA1 hashes of the contents.
678+ so they point to the extracted files. Media are downloaded,
679+ read from the file system, or extracted from a binary
680+ container (e.g. docx), as needed. The original file paths
681+ are used if they are relative paths not containing `..`.
682+ Otherwise filenames are constructed from the SHA1 hash of
683+ the contents.
692684
693685`--abbreviations=`*FILE*
694686
@@ -699,14 +691,15 @@ header when requesting a document from a URL:
699691 system default, use
700692 `pandoc --print-default-data-file=abbreviations`. The only
701693 use pandoc makes of this list is in the Markdown reader.
702− Strings ending in a period that are found in this list will
703− be followed by a nonbreaking space, so that the period will
704− not produce sentence-ending space in formats like LaTeX.
694+ Strings found in this list will be followed by a nonbreaking
695+ space, and the period will not produce sentence-ending space
696+ in formats like LaTeX. The strings may not contain spaces.
705697
706698[`pandocfilters`]: https://github.com/jgm/pandocfilters
707699[PHP]: https://github.com/vinai/pandocfilters-php
708700[perl]: https://metacpan.org/pod/Pandoc::Filter
709701[JavaScript/node.js]: https://github.com/mvhenderson/pandoc-filter-node
702+[Lua filters documentation]: https://pandoc.org/lua-filters.html
710703
711704## General writer options {.options}
712705
@@ -898,6 +891,11 @@ header when requesting a document from a URL:
898891 will not be searched. For example:
899892 `--resource-path=.:test` will search the working directory
900893 and the `test` subdirectory, in that order.
894+ This option can be used repeatedly. Search path components
895+ that come later on the command line will be searched before
896+ those that come earlier, so
897+ `--resource-path foo:bar --resource-path baz:bim` is
898+ equivalent to `--resource-path baz:bim:foo:bar`.
901899
902900`--request-header=`*NAME*`:`*VAL*
903901
@@ -1417,13 +1415,12 @@ results only for basic math, usually you will want to use
14171415`--gladtex`
14181416
14191417: Enclose TeX math in `<eq>` tags in HTML output. The resulting HTML
1420− can then be processed by [GladTeX] to produce images of the typeset
1421− formulas and an HTML file with links to these images.
1422− So, the procedure is:
1418+ can then be processed by [GladTeX] to produce SVG images of the typeset
1419+ formulas and an HTML file with these images embedded.
14231420
14241421 pandoc -s --gladtex input.md -o myfile.htex
1425− gladtex -d myfile-images myfile.htex
1426− # produces myfile.html and images in myfile-images
1422+ gladtex -d image_dir myfile.htex
1423+ # produces myfile.html and images in image_dir
14271424
14281425[MathML]: https://www.w3.org/Math/
14291426[MathJax]: https://www.mathjax.org
@@ -1471,6 +1468,7 @@ Nonzero exit codes have the following meanings:
14711468 24 PandocCiteprocError
14721469 31 PandocEpubSubdirectoryError
14731470 43 PandocPDFError
1471+ 44 PandocXMLError
14741472 47 PandocPDFProgramNotFoundError
14751473 61 PandocHttpError
14761474 62 PandocShouldNeverHappenError
@@ -1483,6 +1481,7 @@ Nonzero exit codes have the following meanings:
14831481 91 PandocMacroLoop
14841482 92 PandocUTF8DecodingError
14851483 93 PandocIpynbDecodingError
1484+ 94 PandocUnsupportedCharsetError
14861485 97 PandocCouldNotFindDataFileError
14871486 99 PandocResourceNotFound
14881487----- ------------------------------------
@@ -1507,6 +1506,16 @@ input-files:
15071506- content.md
15081507# or you may use input-file: with a single value
15091508
1509+# Include options from the specified defaults files.
1510+# The files will be searched for first in the working directory
1511+# and then in the defaults subdirectory of the user data directory.
1512+# The files are included in the same order in which they appear in
1513+# the list. Options specified in this defaults file always have
1514+# priority over the included ones.
1515+defaults:
1516+- defsA
1517+- defsB
1518+
15101519template: letter
15111520standalone: true
15121521self-contained: false
@@ -1564,8 +1573,11 @@ data-dir:
15641573verbosity: INFO
15651574log-file: log.json
15661575
1567−# citeproc, natbib, or biblatex
1576+# citeproc, natbib, or biblatex. This only affects LaTeX
1577+# output. If you want to use citeproc to format citations,
1578+# you should also set 'citeproc: true' (see above).
15681579cite-method: citeproc
1580+
15691581# part, chapter, section, or default:
15701582top-level-division: chapter
15711583abbreviations:
@@ -1664,6 +1676,34 @@ one line:
16641676verbosity: INFO
16651677```
16661678
1679+In fields that expect a file path (or list of file paths), the
1680+following syntax may be used to interpolate environment variables:
1681+
1682+``` yaml
1683+csl: ${HOME}/mycsldir/special.csl
1684+```
1685+
1686+`${USERDATA}` may also be used; this will always resolve to the
1687+user data directory that is current when the defaults file is
1688+parsed, regardless of the setting of the environment
1689+variable `USERDATA`.
1690+
1691+`${.}` will resolve to the directory containing the default
1692+file itself. This allows you to refer to resources contained
1693+in that directory:
1694+
1695+``` yaml
1696+epub-cover-image: ${.}/cover.jpg
1697+epub-metadata: ${.}/meta.xml
1698+resource-path:
1699+- . # the working directory from which pandoc is run
1700+- ${.}/images # the images subdirectory of the directory
1701+ # containing this defaults file
1702+```
1703+
1704+This environment variable interpolation syntax *only* works in
1705+fields that expect file paths.
1706+
16671707Default files can be placed in the `defaults` subdirectory of
16681708the user data directory and used from any directory. For
16691709example, one could create a file specifying defaults for writing
@@ -1891,7 +1931,8 @@ ${ styles.html() }
18911931```
18921932
18931933(If a partial is not found in the directory of the
1894−template, it will also be sought in the `templates`
1934+template and the template path is given as a relative
1935+path, it will also be sought in the `templates`
18951936subdirectory of the user data directory.)
18961937
18971938Partials may optionally be applied to variables using
@@ -2273,8 +2314,8 @@ math equations flush left using [YAML metadata](#layout) or with
22732314
22742315These affect HTML output when [producing slide shows with pandoc].
22752316
2276−All [reveal.js configuration options] are available as variables.
2277−To turn off boolean flags that default to true in reveal.js, use `0`.
2317+`institute`
2318+: author affiliations: can be a list when there are multiple authors
22782319
22792320`revealjs-url`
22802321: base URL for reveal.js documents (defaults to
@@ -2294,6 +2335,9 @@ To turn off boolean flags that default to true in reveal.js, use `0`.
22942335: additional attributes for the title slide of reveal.js slide shows.
22952336 See [background in reveal.js and beamer] for an example.
22962337
2338+All [reveal.js configuration options] are available as variables.
2339+To turn off boolean flags that default to true in reveal.js, use `0`.
2340+
22972341[reveal.js configuration options]: https://revealjs.com/config/
22982342
22992343### Variables for Beamer slides
@@ -3091,6 +3135,13 @@ output format.
30913135Some aspects of [Pandoc's Markdown citation syntax](#citations)
30923136are also accepted in `org` input.
30933137
3138+#### Extension: `element_citations` ####
3139+
3140+In the `jats` output formats, this causes reference items to
3141+be replaced with `<element-citation>` elements. These
3142+elements are not influenced by CSL styles, but all information
3143+on the item is included in tags.
3144+
30943145#### Extension: `ntb` ####
30953146
30963147In the `context` output format this enables the use of [Natural Tables
@@ -3466,6 +3517,9 @@ line must begin with a space.
34663517 | 200 Main St.
34673518 | Berkeley, CA 94718
34683519
3520+Inline formatting (such as emphasis) is allowed in the content,
3521+but not block-level formatting (such as block quotes or lists).
3522+
34693523This syntax is borrowed from [reStructuredText].
34703524
34713525## Lists
@@ -3700,7 +3754,7 @@ definition:
37003754Note that space between items in a definition list is required.
37013755(A variant that loosens this requirement, but disallows "lazy"
37023756hard wrapping, can be activated with `compact_definition_lists`: see
3703−[Non-pandoc extensions], below.)
3757+[Non-default extensions], below.)
37043758
37053759[^3]: I have been influenced by the suggestions of [David
37063760 Wheeler](https://justatheory.com/2009/02/modest-markdown-proposal/).
@@ -4125,10 +4179,12 @@ option is used. All of the metadata will appear in a single block
41254179at the beginning of the document.
41264180
41274181Note that [YAML] escaping rules must be followed. Thus, for example,
4128−if a title contains a colon, it must be quoted. The pipe character
4129−(`|`) can be used to begin an indented block that will be interpreted
4130−literally, without need for escaping. This form is necessary
4131−when the field contains blank lines or block-level formatting:
4182+if a title contains a colon, it must be quoted, and if it contains a
4183+backslash escape, then it must be ensured that it is not treated as a
4184+[YAML escape sequence]. The pipe character (`|`) can be used to begin
4185+an indented block that will be interpreted literally, without need for
4186+escaping. This form is necessary when the field contains blank lines
4187+or block-level formatting:
41324188
41334189 ---
41344190 title: 'This is the title: it contains a colon'
@@ -4142,6 +4198,12 @@ when the field contains blank lines or block-level formatting:
41424198 It consists of two paragraphs.
41434199 ...
41444200
4201+The literal block after the `|` must be indented relative to the
4202+line containing the `|`. If it is not, the YAML will be invalid
4203+and pandoc will not interpret it as metadata. For an overview
4204+of the complex rules governing YAML, see the [Wikipedia entry on
4205+YAML syntax].
4206+
41454207Template variables will be set automatically from the metadata. Thus, for
41464208example, in writing HTML, the variable `abstract` will be set to the HTML
41474209equivalent of the Markdown in the `abstract` field:
@@ -4188,6 +4250,24 @@ will be interpreted as markdown. For example:
41884250 \renewcommand{\section}[1]{\clearpage\oldsection{#1}}
41894251 ```
41904252
4253+Note: the `yaml_metadata_block` extension works with
4254+`commonmark` as well as `markdown` (and it is enabled by default
4255+in `gfm` and `commonmark_x`). However, in these formats the
4256+following restrictions apply:
4257+
4258+- The YAML metadata block must occur at the beginning of the
4259+ document (and there can be only one). If multiple files are
4260+ given as arguments to pandoc, only the first can be a YAML
4261+ metadata block.
4262+
4263+- The leaf nodes of the YAML structure are parsed in isolation from
4264+ each other and from the rest of the document. So, for
4265+ example, you can't use a reference link in these contexts
4266+ if the link definition is somewhere else in the document.
4267+
4268+[YAML escape sequence]: https://yaml.org/spec/1.2/spec.html#id2776092
4269+[Wikipedia entry on YAML syntax]: https://en.m.wikipedia.org/wiki/YAML#Syntax
4270+
41914271## Backslash escapes
41924272
41934273#### Extension: `all_symbols_escapable` ####
@@ -4345,7 +4425,8 @@ you need to enclose text in literal `$` characters, backslash-escape
43454425them and they won't be treated as math delimiters.
43464426
43474427For display math, use `$$` delimiters. (In this case, the delimiters
4348−may be separated from the formula by whitespace.)
4428+may be separated from the formula by whitespace. However, there can be
4429+no blank lines betwen the opening and closing `$$` delimiters.)
43494430
43504431TeX math will be printed in all output formats. How it is rendered
43514432depends on the output format:
@@ -4879,37 +4960,68 @@ Inline and regular footnotes may be mixed freely.
48794960
48804961#### Extension: `citations` ####
48814962
4882−Markdown citations go inside square brackets and are separated
4883−by semicolons. Each citation must have a key, composed of '@' +
4884−the citation identifier from the database, and may optionally
4885−have a prefix, a locator, and a suffix. The citation key must
4886−begin with a letter, digit, or `_`, and may contain
4887−alphanumerics, `_`, and internal punctuation characters
4888−(`:.#$%&-+?<>~/`). Here are some examples:
4889−
4890− Blah blah [see @doe99, pp. 33-35; also @smith04, chap. 1].
4891−
4892− Blah blah [@doe99, pp. 33-35, 38-39 and *passim*].
4893−
4894− Blah blah [@smith04; @doe99].
4895−
4896−`pandoc` detects locator terms in the [CSL locale files].
4897−Either abbreviated or unabbreviated forms are accepted. In the `en-US`
4898−locale, locator terms can be written in either singular or plural forms,
4899−as `book`, `bk.`/`bks.`; `chapter`, `chap.`/`chaps.`; `column`,
4900−`col.`/`cols.`; `figure`, `fig.`/`figs.`; `folio`, `fol.`/`fols.`;
4901−`number`, `no.`/`nos.`; `line`, `l.`/`ll.`; `note`, `n.`/`nn.`; `opus`,
4902−`op.`/`opp.`; `page`, `p.`/`pp.`; `paragraph`, `para.`/`paras.`; `part`,
4903−`pt.`/`pts.`; `section`, `sec.`/`secs.`; `sub verbo`, `s.v.`/`s.vv.`;
4904−`verse`, `v.`/`vv.`; `volume`, `vol.`/`vols.`; `¶`/`¶¶`; `§`/`§§`. If no
4963+To cite a bibliographic item with an identifier foo, use the
4964+syntax `@foo`. Normal citations should be included in square
4965+brackets, with semicolons separating distinct items:
4966+
4967+ Blah blah [@doe99; @smith2000; @smith2004].
4968+
4969+How this is rendered depends on the citation style. In an
4970+author-date style, it might render as
4971+
4972+ Blah blah (Doe 1999, Smith 2000, 2004).
4973+
4974+In a footnote style, it might render as
4975+
4976+ Blah blah.[^1]
4977+
4978+ [^1]: John Doe, "Frogs," *Journal of Amphibians* 44 (1999);
4979+ Susan Smith, "Flies," *Journal of Insects* (2000);
4980+ Susan Smith, "Bees," *Journal of Insects* (2004).
4981+
4982+See the [CSL user documentation] for more information about CSL
4983+styles and how they affect rendering.
4984+
4985+Unless a citation key start with a letter, digit, or `_`,
4986+and contains only alphanumerics and internal punctuation
4987+characters (`:.#$%&-+?<>~/`), it must be surrounded
4988+by curly braces, which are not considered part of the key.
4989+In `@Foo_bar.baz.`, the key is `Foo_bar.baz`. The final
4990+period is not *internal* punctuation, so it is not included in
4991+the key. In `@{Foo_bar.baz.}`, the key is `Foo_bar.baz.`, including
4992+the final period. The curly braces are recommended if you use URLs as
4993+keys: `[@{https://example.com/bib?name=foobar&date=2000}, p. 33]`.
4994+
4995+Citation items may optionally include a prefix, a locator, and
4996+a suffix. In
4997+
4998+ Blah blah [see @doe99, pp. 33-35 and *passim*; @smith04, chap. 1].
4999+
5000+The first item (`doe99`) has prefix `see `, locator `pp. 33-35`,
5001+and suffix `and *passim*`. The second item (`smith04`) has
5002+locator `chap. 1` and no prefix or suffix.
5003+
5004+Pandoc uses some heuristics to separate the locator from the
5005+rest of the subject. It is sensitive to the locator terms
5006+defined in the [CSL locale files]. Either abbreviated or
5007+unabbreviated forms are accepted. In the `en-US` locale, locator
5008+terms can be written in either singular or plural forms, as
5009+`book`, `bk.`/`bks.`; `chapter`, `chap.`/`chaps.`; `column`,
5010+`col.`/`cols.`; `figure`, `fig.`/`figs.`; `folio`,
5011+`fol.`/`fols.`; `number`, `no.`/`nos.`; `line`, `l.`/`ll.`;
5012+`note`, `n.`/`nn.`; `opus`, `op.`/`opp.`; `page`, `p.`/`pp.`;
5013+`paragraph`, `para.`/`paras.`; `part`, `pt.`/`pts.`; `section`,
5014+`sec.`/`secs.`; `sub verbo`, `s.v.`/`s.vv.`; `verse`,
5015+`v.`/`vv.`; `volume`, `vol.`/`vols.`; `¶`/`¶¶`; `§`/`§§`. If no
49055016locator term is used, "page" is assumed.
49065017
4907−`pandoc` will use heuristics to distinguish the locator
4908−from the suffix. In complex cases, the locator can be enclosed
4909−in curly braces:
5018+In complex cases, you can force something to be treated as
5019+a locator by enclosing it in curly braces or prevent parsing
5020+the suffix as locator by prepending curly braces:
49105021
49115022 [@smith{ii, A, D-Z}, with a suffix]
49125023 [@smith, {pp. iv, vi-xi, (xv)-(xvii)} with suffix here]
5024+ [@smith{}, 99 years later]
49135025
49145026A minus sign (`-`) before the `@` will suppress mention of
49155027the author in the citation. This can be useful when the
@@ -4917,13 +5029,26 @@ author is already mentioned in the text:
49175029
49185030 Smith says blah [-@smith04].
49195031
4920−You can also write an in-text citation, as follows:
5032+You can also write an author-in-text citation, by omitting the
5033+square brackets:
49215034
49225035 @smith04 says blah.
49235036
49245037 @smith04 [p. 33] says blah.
49255038
5039+This will cause the author's name to be rendered, followed by
5040+the bibliographical details. Use this form when you want to
5041+make the citation the subject of a sentence.
5042+
5043+When you are using a note style, it is usually better to let
5044+citeproc create the footnotes from citations rather than writing
5045+an explicit note. If you do write an explicit note that
5046+contains a citation, note that normal citations will be put in
5047+parentheses, while author-in-text citations will not. For
5048+this reason, it is sometimes preferable to use the
5049+author-in-text style inside notes when using a note style.
49265050
5051+[CSL user documentation]: https://citationstyles.org/authors/
49275052[CSL]: https://docs.citationstyles.org/en/stable/specification.html
49285053[CSL markup specs]: https://docs.citationstyles.org/en/1.0/release-notes.html#rich-text-markup-within-fields
49295054[Chicago Manual of Style]: https://chicagomanualofstyle.org
@@ -4932,13 +5057,44 @@ You can also write an in-text citation, as follows:
49325057[finding and editing styles]: https://citationstyles.org/authors/
49335058[CSL locale files]: https://github.com/citation-style-language/locales
49345059
4935−## Non-pandoc extensions
5060+## Non-default extensions
49365061
49375062The following Markdown syntax extensions are not enabled by default
49385063in pandoc, but may be enabled by adding `+EXTENSION` to the format
49395064name, where `EXTENSION` is the name of the extension. Thus, for
49405065example, `markdown+hard_line_breaks` is Markdown with hard line breaks.
49415066
5067+#### Extension: `rebase_relative_paths` ####
5068+
5069+Rewrite relative paths for Markdown links and images, depending
5070+on the path of the file containing the link or image link. For
5071+each link or image, pandoc will compute the directory of the
5072+containing file, relative to the working directory, and prepend
5073+the resulting path to the link or image path.
5074+
5075+The use of this extension is best understood by example.
5076+Suppose you have a a subdirectory for each chapter of a book,
5077+`chap1`, `chap2`, `chap3`. Each contains a file `text.md` and a
5078+number of images used in the chapter. You would like to have
5079+`![image](spider.jpg)` in `chap1/text.md` refer to
5080+`chap1/spider.jpg` and `![image](spider.jpg)` in `chap2/text.md`
5081+refer to `chap2/spider.jpg`. To do this, use
5082+
5083+ pandoc chap*/*.md -f markdown+rebase_relative_paths
5084+
5085+Without this extension, you would have to use
5086+`![image](chap1/spider.jpg)` in `chap1/text.md` and
5087+`![image](chap2/spider.jpg)` in `chap2/text.md`. Links with
5088+relative paths will be rewritten in the same way as images.
5089+
5090+Absolute paths and URLs are not changed. Neither are empty
5091+paths or paths consisting entirely of a fragment, e.g., `#foo`.
5092+
5093+Note that relative paths in reference links and images will
5094+be rewritten relative to the file containing the link
5095+reference definition, not the file containing the reference link
5096+or image itself, if these differ.
5097+
49425098#### Extension: `attributes` ####
49435099
49445100Allows attributes to be attached to any inline or block-level
@@ -5256,11 +5412,11 @@ section from a BibTeX, BibLaTeX, or CSL JSON bibliography:
52565412 pandoc chem.bib -s -f biblatex -t markdown
52575413 pandoc chem.json -s -f csljson -t markdown
52585414
5259−`pandoc` can also be used to produce CSL JSON bibliography
5260−from BibTeX, BibLaTeX, or markdown YAML:
5415+Indeed, `pandoc` can convert between any of these
5416+citation formats:
52615417
52625418 pandoc chem.bib -s -f biblatex -t csljson
5263− pandoc chem.yaml -s -f markdown -t csljson
5419+ pandoc chem.yaml -s -f markdown -t biblatex
52645420
52655421Running pandoc on a bibliography file with the `--citeproc`
52665422option will create a formatted bibliography in the format
@@ -5359,6 +5515,34 @@ example:
53595515 }
53605516 }
53615517
5518+
5519+## Citations in note styles
5520+
5521+Pandoc's citation processing is designed to allow you to
5522+move between author-date, numerical, and note styles without
5523+modifying the markdown source. When you're using a note
5524+style, avoid inserting footnotes manually. Instead, insert
5525+citations just as you would in an author-date style---for
5526+example,
5527+
5528+ Blah blah [@foo, p. 33].
5529+
5530+The footnote will be created automatically. Pandoc will take
5531+care of removing the space and moving the note before or
5532+after the period, depending on the setting of
5533+`notes-after-punctuation`, as described below in [Other relevant
5534+metadata fields].
5535+
5536+In some cases you may need to put a citation inside a regular
5537+footnote. Normal citations in footnotes (such as `[@foo, p.
5538+33]`) will be rendered in parentheses. In-text citations (such
5539+as `@foo [p. 33]`) will be rendered without parentheses. (A
5540+comma will be added if appropriate.) Thus:
5541+
5542+ [^1]: Some studies [@foo; @bar, p. 33] show that
5543+ frubulicious zoosnaps are quantical. For a survey
5544+ of the literature, see @baz [chap. 1].
5545+
53625546## Raw content in a style
53635547
53645548To include raw content in a prefix, suffix, delimiter, or term,
@@ -5444,10 +5628,23 @@ A few other metadata fields affect bibliography formatting:
54445628
54455629`lang`
54465630: The `lang` field will affect how the style is localized,
5447− for example in the translation of labels and the use
5448− of quotation marks. (For backwards compatibility,
5449− `locale` may be used instead of `lang`, but this use
5450− is deprecated.)
5631+ for example in the translation of labels, the use
5632+ of quotation marks, and the way items are sorted.
5633+ (For backwards compatibility, `locale` may be used instead
5634+ of `lang`, but this use is deprecated.)
5635+
5636+ A BCP 47 language tag is expected: for example, `en`,
5637+ `de`, `en-US`, `fr-CA`, `ug-Cyrl`. The unicode extension
5638+ syntax (after `-u-`) may be used to specify options for
5639+ collation (sorting) more precisely. Here are some examples:
5640+
5641+ - `zh-u-co-pinyin` -- Chinese with the Pinyin collation.
5642+ - `es-u-co-trad` -- Spanish with the traditional collation
5643+ (with `Ch` sorting after `C`).
5644+ - `fr-u-kb` -- French with "backwards" accent sorting
5645+ (with `coté` sorting after `côte`).
5646+ - `en-US-u-kf-upper` -- English with uppercase letters sorting
5647+ before lower (default is lower before upper).
54515648
54525649`notes-after-punctuation`
54535650: If true (the default), pandoc will put footnote citations
@@ -5459,12 +5656,6 @@ A few other metadata fields affect bibliography formatting:
54595656 punctuation.
54605657
54615658
5462−
5463−
5464−
5465−
5466−
5467−
54685659# Slide shows
54695660
54705661You can use pandoc to produce an HTML + JavaScript slide presentation
@@ -5557,7 +5748,11 @@ rules:
55575748 * A heading at the slide level always starts a new slide.
55585749
55595750 * Headings *below* the slide level in the hierarchy create
5560− headings *within* a slide.
5751+ headings *within* a slide. (In beamer, a "block" will be
5752+ created. If the heading has the class `example`, an
5753+ `exampleblock` environment will be used; if it has the class
5754+ `alert`, an `alertblock` will be used; otherwise a regular
5755+ `block` will be used.)
55615756
55625757 * Headings *above* the slide level in the hierarchy create
55635758 "title slides," which just contain the section title
@@ -5890,6 +6085,15 @@ The following fields are recognized:
58906085
58916086`rights`
58926087 ~ A string value.
6088+
6089+`belongs-to-collection`
6090+ ~ A string value. identifies the name of a collection to which
6091+ the EPUB Publication belongs.
6092+
6093+`group-position`
6094+ ~ The `group-position` field indicates the numeric position in which
6095+ the EPUB Publication belongs relative to other works belonging to
6096+ the same `belongs-to-collection` field.
58936097
58946098`cover-image`
58956099 ~ A string value (path to cover image).
@@ -6126,7 +6330,7 @@ Then edit `my.theme` and use it like this:
61266330If you are not satisfied with the built-in highlighting, or you
61276331want highlight a language that isn't supported, you can use the
61286332`--syntax-definition` option to load a [KDE-style XML syntax definition
6129−file](https://docs.kde.org/stable5/en/applications/katepart/highlight.html).
6333+file](https://docs.kde.org/stable5/en/kate/katepart/highlight.html).
61306334Before writing your own, have a look at KDE's [repository of syntax
61316335definitions](https://github.com/KDE/syntax-highlighting/tree/master/data/syntax).
61326336
@@ -6257,6 +6461,21 @@ subdirectory of your user data directory (see [Templates]).
62576461
62586462[Lua]: https://www.lua.org
62596463
6464+# Reproducible builds
6465+
6466+Some of the document formats pandoc targets (such as EPUB,
6467+docx, and ODT) include build timestamps in the generated document.
6468+That means that the files generated on successive builds will
6469+differ, even if the source does not. To avoid this, set the
6470+`SOURCE_DATE_EPOCH` environment variable, and the timestamp will
6471+be taken from it instead of the current time.
6472+`SOURCE_DATE_EPOCH` should contain an integer unix timestamp
6473+(specifying the number of second since midnight UTC January 1, 1970).
6474+
6475+Some document formats also include a unique identifier. For
6476+EPUB, this can be set explicitly by setting the `identifier`
6477+metadata field (see [EPUB Metadata], above).
6478+
62606479# A note on security
62616480
62626481If you use pandoc to convert user-contributed content in a web
@@ -6285,13 +6504,13 @@ application, here are some things to keep in mind:
628565044. The HTML generated by pandoc is not guaranteed to be safe.
62866505 If `raw_html` is enabled for the Markdown input, users can
62876506 inject arbitrary HTML. Even if `raw_html` is disabled,
6288− users can include dangerous content in attributes for
6289− headings, spans, and code blocks. To be safe, you should
6290− run all the generated HTML through an HTML sanitizer.
6507+ users can include dangerous content in URLs and attributes.
6508+ To be safe, you should run all the generated HTML through
6509+ an HTML sanitizer.
62916510
62926511# Authors
62936512
6294−Copyright 2006--2020 John MacFarlane (jgm@berkeley.edu). Released
6513+Copyright 2006--2021 John MacFarlane (jgm@berkeley.edu). Released
62956514under the [GPL], version 2 or greater. This software carries no
62966515warranty of any kind. (See COPYRIGHT for full copyright and
62976516warranty notices.) For a full list of contributors, see the file
Makefile+32−11
@@ -2,17 +2,31 @@ version?=$(shell grep '^[Vv]ersion:' pandoc.cabal | awk '{print $$2;}')
22pandoc=$(shell find dist -name pandoc -type f -exec ls -t {} \; | head -1)
33SOURCEFILES?=$(shell git ls-tree -r master --name-only | grep "\.hs$$")
44BRANCH?=master
5−RESOLVER?=lts-13
6−GHCOPTS=-fdiagnostics-color=always
5+ARCH=$(shell uname -m)
6+DOCKERIMAGE=registry.gitlab.b-data.ch/ghc/ghc4pandoc:8.10.4
7+COMMIT=$(shell git rev-parse --short HEAD)
8+TIMESTAMP=$(shell date "+%Y%m%d_%H%M")
9+LATESTBENCH=$(word 1,$(shell ls -t bench_*.csv 2>/dev/null))
10+BASELINE?=$(LATESTBENCH)
11+ifeq ($(BASELINE),)
12+BASELINECMD=
13+else
14+BASELINECMD=--baseline $(BASELINE)
15+endif
16+GHCOPTS=-fdiagnostics-color=always -j4 +RTS -A8m -RTS
717WEBSITE=../../web/pandoc.org
818REVISION?=1
19+# For gauge:
20+# BENCHARGS?=--small --ci=0.90 --match=pattern $(PATTERN)
21+# For tasty-bench:
22+BENCHARGS?=--csv bench_$(TIMESTAMP).csv $(BASELINECMD) --timeout=6 +RTS -T -RTS $(if $(PATTERN),--pattern "$(PATTERN)",)
923
1024quick:
11− stack install --ghc-options='$(GHCOPTS)' --install-ghc --flag 'pandoc:embed_data_files' --fast --test --ghc-options='-j +RTS -A64m -RTS' --test-arguments='-j4 --hide-successes $(TESTARGS)'
25+ stack install --ghc-options='$(GHCOPTS)' --install-ghc --flag 'pandoc:embed_data_files' --fast --test --ghc-options='$(GHCOPTS)' --test-arguments='-j4 --hide-successes $(TESTARGS)'
1226
1327quick-cabal:
1428 cabal new-configure . --ghc-options '$(GHCOPTS)' --disable-optimization --enable-tests
15− cabal new-build . --disable-optimization
29+ cabal new-build -j4 . --disable-optimization
1630 cabal new-run test-pandoc --disable-optimization -- --hide-successes $(TESTARGS)
1731
1832full-cabal:
@@ -21,7 +35,7 @@ full-cabal:
2135 cabal new-run test-pandoc --disable-optimization -- --hide-successes $(TESTARGS)
2236
2337full:
24− stack install --flag 'pandoc:embed_data_files' --flag 'pandoc:trypandoc' --bench --no-run-benchmarks --test --test-arguments='-j4 --hide-successes' --ghc-options '-Wall -Werror -fno-warn-unused-do-bind -O0 -j4 $(GHCOPTS)'
38+ stack install --flag 'pandoc:embed_data_files' --flag 'pandoc:trypandoc' --bench --no-run-benchmarks --test --test-arguments='-j4 --hide-successes' --ghc-options '-Wall -Werror -fno-warn-unused-do-bind -O0 $(GHCOPTS)'
2539
2640ghci:
2741 stack ghci --flag 'pandoc:embed_data_files'
@@ -37,11 +51,14 @@ test:
3751ghcid:
3852 ghcid -c "stack repl --flag 'pandoc:embed_data_files'"
3953
40−bench:
41− stack bench --benchmark-arguments='$(BENCHARGS)' --ghc-options '$(GHCOPTS)'
54+ghcid-test:
55+ ghcid -c "stack repl --ghc-options=-XNoImplicitPrelude --flag 'pandoc:embed_data_files' --ghci-options=-fobject-code pandoc:lib pandoc:test-pandoc"
4256
43−weigh:
44− stack build --ghc-options '$(GHCOPTS)' pandoc:weigh-pandoc && stack exec weigh-pandoc
57+bench:
58+ stack bench \
59+ --ghc-options '$(GHCOPTS)' \
60+ --benchmark-arguments='$(BENCHARGS)' 2>&1 | \
61+ tee "bench_latest.txt"
4562
4663reformat:
4764 for f in $(SOURCEFILES); do echo $$f; stylish-haskell -i $$f ; done
@@ -73,10 +90,14 @@ checkdocs:
7390debpkg: man/pandoc.1
7491 docker run -v `pwd`:/mnt \
7592 -v `pwd`/linux/artifacts:/artifacts \
93+ --user $(id -u):$(id -g) \
7694 -e REVISION=$(REVISION) \
7795 -w /mnt \
78− utdemir/ghc-musl:v12-libgmp-ghc8101 bash \
79− /mnt/linux/make_artifacts.sh
96+ --memory=0 \
97+ --rm \
98+ $(DOCKERIMAGE) \
99+ bash \
100+ /mnt/linux/make_artifacts.sh 2>&1 > docker.log
80101
81102man/pandoc.1: MANUAL.txt man/pandoc.1.before man/pandoc.1.after
82103 pandoc $< -f markdown -t man -s \
README.md+9−7
@@ -8,10 +8,10 @@ pandoc --lua-filter tools/update-readme.lua README.template -o README.md
88[![github
99release](https://img.shields.io/github/release/jgm/pandoc.svg?label=current+release)](https://github.com/jgm/pandoc/releases)
1010[![hackage
11−release](https://img.shields.io/hackage/v/pandoc.svg?label=hackage)](http://hackage.haskell.org/package/pandoc)
12−[![homebrew](https://img.shields.io/homebrew/v/pandoc.svg)](http://brewformulas.org/Pandoc)
11+release](https://img.shields.io/hackage/v/pandoc.svg?label=hackage)](https://hackage.haskell.org/package/pandoc)
12+[![homebrew](https://img.shields.io/homebrew/v/pandoc.svg)](https://formulae.brew.sh/formula/pandoc)
1313[![stackage LTS
14−package](http://stackage.org/package/pandoc/badge/lts)](http://stackage.org/lts/package/pandoc)
14+package](https://stackage.org/package/pandoc/badge/lts)](https://www.stackage.org/lts/package/pandoc-types)
1515[![CI
1616tests](https://github.com/jgm/pandoc/workflows/CI%20tests/badge.svg)](https://github.com/jgm/pandoc/actions)
1717[![license](https://img.shields.io/badge/license-GPLv2+-lightgray.svg)](https://www.gnu.org/licenses/gpl.html)
@@ -20,7 +20,7 @@ groups](https://img.shields.io/badge/pandoc-discuss-red.svg?style=social)](https
2020
2121## The universal markup converter
2222
23−Pandoc is a [Haskell](http://haskell.org) library for converting from
23+Pandoc is a [Haskell](https://haskell.org) library for converting from
2424one markup format to another, and a command-line tool that uses this
2525library. It can convert *from*
2626
@@ -95,6 +95,8 @@ It can convert *to*
9595- `asciidoc` ([AsciiDoc](https://www.methods.co.nz/asciidoc/)) or
9696 `asciidoctor` ([AsciiDoctor](https://asciidoctor.org/))
9797- `beamer` ([LaTeX beamer](https://ctan.org/pkg/beamer) slide show)
98+- `bibtex` ([BibTeX](https://ctan.org/pkg/bibtex) bibliography)
99+- `biblatex` ([BibLaTeX](https://ctan.org/pkg/biblatex) bibliography)
98100- `commonmark` ([CommonMark](https://commonmark.org) Markdown)
99101- `commonmark_x` ([CommonMark](https://commonmark.org) Markdown with
100102 extensions)
@@ -173,7 +175,7 @@ It can convert *to*
173175 and JavaScript slide show)
174176- `slidy` ([Slidy](https://www.w3.org/Talks/Tools/Slidy2/) HTML and
175177 JavaScript slide show)
176−- `dzslides` ([DZSlides](http://paulrouget.com/dzslides/) HTML5 +
178+- `dzslides` ([DZSlides](https://paulrouget.com/dzslides/) HTML5 +
177179 JavaScript slide show),
178180- `revealjs` ([reveal.js](https://revealjs.com/) HTML5 + JavaScript
179181 slide show)
@@ -236,7 +238,7 @@ opening a new issue.
236238
237239## License
238240
239−© 2006-2020 John MacFarlane (jgm@berkeley.edu). Released under the
240−[GPL](http://www.gnu.org/copyleft/gpl.html "GNU General Public License"),
241+© 2006-2021 John MacFarlane (jgm@berkeley.edu). Released under the
242+[GPL](https://www.gnu.org/licenses/old-licenses/gpl-2.0.html "GNU General Public License"),
241243version 2 or greater. This software carries no warranty of any kind.
242244(See COPYRIGHT for full copyright and warranty notices.)
README.template+6−6
@@ -7,9 +7,9 @@ Pandoc
77======
88
99[![github release](https://img.shields.io/github/release/jgm/pandoc.svg?label=current+release)](https://github.com/jgm/pandoc/releases)
10−[![hackage release](https://img.shields.io/hackage/v/pandoc.svg?label=hackage)](http://hackage.haskell.org/package/pandoc)
11−[![homebrew](https://img.shields.io/homebrew/v/pandoc.svg)](http://brewformulas.org/Pandoc)
12−[![stackage LTS package](http://stackage.org/package/pandoc/badge/lts)](http://stackage.org/lts/package/pandoc)
10+[![hackage release](https://img.shields.io/hackage/v/pandoc.svg?label=hackage)](https://hackage.haskell.org/package/pandoc)
11+[![homebrew](https://img.shields.io/homebrew/v/pandoc.svg)](https://formulae.brew.sh/formula/pandoc)
12+[![stackage LTS package](https://stackage.org/package/pandoc/badge/lts)](https://www.stackage.org/lts/package/pandoc-types)
1313[![CI tests](https://github.com/jgm/pandoc/workflows/CI%20tests/badge.svg)](https://github.com/jgm/pandoc/actions)
1414[![license](https://img.shields.io/badge/license-GPLv2+-lightgray.svg)](https://www.gnu.org/licenses/gpl.html)
1515[![pandoc-discuss on google groups](https://img.shields.io/badge/pandoc-discuss-red.svg?style=social)](https://groups.google.com/forum/#!forum/pandoc-discuss)
@@ -81,9 +81,9 @@ new issue.
8181License
8282-------
8383
84−© 2006-2020 John MacFarlane (jgm@berkeley.edu). Released under the
84+© 2006-2021 John MacFarlane (jgm@berkeley.edu). Released under the
8585[GPL], version 2 or greater. This software carries no warranty of
8686any kind. (See COPYRIGHT for full copyright and warranty notices.)
8787
88−[GPL]: http://www.gnu.org/copyleft/gpl.html "GNU General Public License"
89−[Haskell]: http://haskell.org
88+[GPL]: https://www.gnu.org/licenses/old-licenses/gpl-2.0.html "GNU General Public License"
89+[Haskell]: https://haskell.org
RELEASE-CHECKLIST+1
@@ -4,6 +4,7 @@
44[ ] Finalize changelog
55[ ] push release candidate branch rc/VERSION and (if it builds
66 successfully) download artifacts
7+[ ] run tools/build-arm.sh to create and download arm64 linux package
78[ ] download macos artifacts from CircleCI
89[ ] Use 'make' in macos and windows artifacts to sign code
910[ ] make update-website
SECURITY.mdnew file+12
@@ -0,0 +1,12 @@
1+# Security Policy
2+
3+## Supported Versions
4+
5+Only the most recent version of pandoc is supported with security updates.
6+
7+## Reporting a Vulnerability
8+
9+To report a vulnerability, email the maintainer, jgm@berkeley.edu.
10+But first please read the section of the manual entitled
11+[A note on security](https://pandoc.org/MANUAL.html#a-note-on-security),
12+which describes some security guarantees pandoc does NOT make.
app/pandoc.hs+1−3
@@ -1,7 +1,6 @@
1−{-# LANGUAGE NoImplicitPrelude #-}
21{- |
32 Module : Main
4− Copyright : Copyright (C) 2006-2020 John MacFarlane
3+ Copyright : Copyright (C) 2006-2021 John MacFarlane
54 License : GNU GPL, version 2 or above
65
76 Maintainer : John MacFarlane <jgm@berkeley@edu>
@@ -12,7 +11,6 @@ Parses command-line options and calls the appropriate readers and
1211writers.
1312-}
1413module Main where
15−import Prelude
1614import qualified Control.Exception as E
1715import Text.Pandoc.App (convertWithOpts, defaultOpts, options, parseOptions)
1816import Text.Pandoc.Error (handleError)
benchmark/benchmark-pandoc.hs+65−84
@@ -1,7 +1,6 @@
1−{-# LANGUAGE NoImplicitPrelude #-}
21{-# LANGUAGE OverloadedStrings #-}
32{-
4−Copyright (C) 2012-2019 John MacFarlane <jgm@berkeley.edu>
3+Copyright (C) 2012-2021 John MacFarlane <jgm@berkeley.edu>
54
65This program is free software; you can redistribute it and/or modify
76it under the terms of the GNU General Public License as published by
@@ -17,106 +16,88 @@ You should have received a copy of the GNU General Public License
1716along with this program; if not, write to the Free Software
1817Foundation, Inc., 59 Temple Place, Suite 330, Boston, MA 02111-1307 USA
1918-}
20−import Prelude
2119import Text.Pandoc
2220import Text.Pandoc.MIME
23−import Control.Monad.Except (throwError, liftIO)
21+import Control.DeepSeq (force)
22+import Control.Monad.Except (throwError)
2423import qualified Text.Pandoc.UTF8 as UTF8
2524import qualified Data.ByteString as B
2625import qualified Data.Text as T
27−import Criterion.Main
28−import Criterion.Types (Config(..))
29−import Data.List (intersect)
30−import Data.Maybe (mapMaybe, catMaybes)
31−import System.Environment (getArgs)
26+import Test.Tasty.Bench
27+-- import Gauge
3228import qualified Data.ByteString.Lazy as BL
33−
34−data Input = InputText {unInputText :: T.Text}
35− | InputBS {unInputBS :: BL.ByteString}
29+import Data.Maybe (mapMaybe)
30+import Data.List (sortOn)
3631
3732readerBench :: Pandoc
3833 -> T.Text
39− -> IO (Maybe Benchmark)
40−readerBench doc name = do
41− let (rdr, rexts) = either (error . show) id . runPure $ getReader name
42− res <- runIO $ do
43− (wtr, wexts) <- getWriter name
44− case (rdr, wtr) of
45− (TextReader r, TextWriter w) -> do
46− setResourcePath ["./test"]
47− inp <- w def{ writerWrapText = WrapAuto
48− , writerExtensions = wexts } doc
49− return (r def{ readerExtensions = rexts } . unInputText, InputText inp)
50− (ByteStringReader r, ByteStringWriter w) -> do
51− setResourcePath ["./test"]
52− tmpl <- Just <$> compileDefaultTemplate name
53− inp <- w def{ writerWrapText = WrapAuto
54− , writerExtensions = wexts
55− , writerTemplate = tmpl } doc
56− liftIO $ BL.writeFile "/tmp/test.odt" inp
57− return (r def{ readerExtensions = rexts } . unInputBS, InputBS inp)
58− _ -> throwError $ PandocSomeError $ "text/bytestring format mismatch: "
59− <> name
60− return $ case res of
61− Right (readerFun, inp) ->
62− Just $ bench (T.unpack $ name <> " reader")
63− $ nf (\i -> either (error . show) id $ runPure (readerFun i))
64− inp
65− Left _ -> Nothing
34+ -> Maybe Benchmark
35+readerBench _ name
36+ | name `elem` ["bibtex", "biblatex", "csljson"] = Nothing
37+readerBench doc name = either (const Nothing) Just $
38+ runPure $ do
39+ (rdr, rexts) <- getReader name
40+ (wtr, wexts) <- getWriter name
41+ case (rdr, wtr) of
42+ (TextReader r, TextWriter w) -> do
43+ inp <- w def{ writerWrapText = WrapAuto
44+ , writerExtensions = wexts } doc
45+ return $ bench (T.unpack name) $
46+ nf (either (error . show) id . runPure . r def) inp
47+ (ByteStringReader r, ByteStringWriter w) -> do
48+ tmpl <- Just <$> compileDefaultTemplate name
49+ inp <- w def{ writerWrapText = WrapAuto
50+ , writerExtensions = wexts
51+ , writerTemplate = tmpl } doc
52+ return $ bench (T.unpack name) $
53+ nf (either (error . show) id .
54+ runPure . r def{readerExtensions = rexts}) inp
55+ _ -> throwError $ PandocSomeError $ "text/bytestring format mismatch: "
56+ <> name
6657
6758getImages :: IO [(FilePath, MimeType, BL.ByteString)]
6859getImages = do
69− ll <- BL.readFile "test/lalune.jpg"
70− mv <- BL.readFile "test/movie.jpg"
71− return [("lalune.jpg", "image/jpg", ll)
72− ,("movie.jpg", "image/jpg", mv)]
60+ ll <- B.readFile "test/lalune.jpg"
61+ mv <- B.readFile "test/movie.jpg"
62+ return [("lalune.jpg", "image/jpg", BL.fromStrict ll)
63+ ,("movie.jpg", "image/jpg", BL.fromStrict mv)]
7364
74−writerBench :: Pandoc
65+writerBench :: [(FilePath, MimeType, BL.ByteString)]
66+ -> Pandoc
7567 -> T.Text
7668 -> Maybe Benchmark
77−writerBench doc name =
78− case res of
79− Right writerFun ->
80− Just $ env getImages $ \imgs ->
81− bench (T.unpack $ name <> " writer")
69+writerBench _ _ name
70+ | name `elem` ["bibtex", "biblatex", "csljson"] = Nothing
71+writerBench imgs doc name = either (const Nothing) Just $
72+ runPure $ do
73+ (wtr, wexts) <- getWriter name
74+ case wtr of
75+ TextWriter writerFun ->
76+ return $ bench (T.unpack name)
77+ $ nf (\d -> either (error . show) id $
78+ runPure $ do
79+ mapM_ (\(fp,mt,bs) -> insertMedia fp (Just mt) bs) imgs
80+ writerFun def{ writerExtensions = wexts} d)
81+ doc
82+ ByteStringWriter writerFun ->
83+ return $ bench (T.unpack name)
8284 $ nf (\d -> either (error . show) id $
83− runPure (do mapM_
84− (\(fp, mt, bs) ->
85− insertMedia fp (Just mt) bs)
86− imgs
87− writerFun d)) doc
88− Left _ -> Nothing
89− where res = runPure $ do
90− (wtr, wexts) <- getWriter name
91− case wtr of
92− TextWriter w ->
93− return $ w def{ writerExtensions = wexts }
94− _ -> throwError $ PandocSomeError
95− $ "could not get text writer for " <> name
85+ runPure $ do
86+ mapM_ (\(fp,mt,bs) -> insertMedia fp (Just mt) bs) imgs
87+ writerFun def{ writerExtensions = wexts} d)
88+ doc
9689
9790main :: IO ()
9891main = do
99− args <- filter (\x -> T.take 1 x /= "-") . fmap T.pack <$> getArgs
100− print args
101− let matchReader (n, _) =
102− null args || ("reader" `elem` args && n `elem` args)
103− matchWriter (n, TextWriter _) =
104− null args || ("writer" `elem` args && n `elem` args)
105− matchWriter _ = False
106− allWriters = map fst (writers :: [(T.Text, Writer PandocPure)])
107− matchedReaders = map fst (filter matchReader readers
108− :: [(T.Text, Reader PandocPure)])
109− matchedWriters = map fst (filter matchWriter writers
110− :: [(T.Text, Writer PandocPure)])
11192 inp <- UTF8.toText <$> B.readFile "test/testsuite.txt"
11293 let opts = def
113− let doc = either (error . show) id $ runPure $ readMarkdown opts inp
114− readerBs <- fmap catMaybes
115− $ mapM (readerBench doc)
116− $ filter (/="haddock")
117− (matchedReaders `intersect` allWriters)
118− -- we need the corresponding writer to generate
119− -- input for the reader
120− let writerBs = mapMaybe (writerBench doc) matchedWriters
121− defaultMainWith defaultConfig{ timeLimit = 6.0 }
122− (writerBs ++ readerBs)
94+ let doc = either (error . show) force $ runPure $ readMarkdown opts inp
95+ defaultMain
96+ [ env getImages $ \imgs ->
97+ bgroup "writers" $ mapMaybe (writerBench imgs doc . fst)
98+ (sortOn fst
99+ writers :: [(T.Text, Writer PandocPure)])
100+ , bgroup "readers" $ mapMaybe (readerBench doc . fst)
101+ (sortOn fst
102+ readers :: [(T.Text, Reader PandocPure)])
103+ ]
benchmark/weigh-pandoc.hsdeleted−50
@@ -1,50 +0,0 @@
1−{-# LANGUAGE NoImplicitPrelude #-}
2−{-# LANGUAGE OverloadedStrings #-}
3−{- |
4− Module : Main
5− Copyright : © 2016-2019 John MacFarlane <jgm@berkeley.edu>
6− License : GNU GPL, version 2 or above
7−
8− Maintainer : John MacFarlane <jgm@berkeley.edu>
9− Stability : alpha
10− Portability : portable
11−
12−Benchmarks to determine resource use of readers and writers.
13−-}
14−import Prelude
15−import Weigh
16−import Text.Pandoc
17−import Data.Text (Text, unpack)
18−
19−main :: IO ()
20−main = do
21− doc <- read <$> readFile "test/testsuite.native"
22− mainWith $ do
23− func "Pandoc document" id doc
24− mapM_
25− (\(n,r) -> weighReader doc n (either (error . show) id . runPure . r def{readerExtensions = pandocExtensions}))
26− [("markdown", readMarkdown)
27− ,("html", readHtml)
28− ,("docbook", readDocBook)
29− ,("latex", readLaTeX)
30− ,("commonmark", readCommonMark)
31− ]
32− mapM_
33− (\(n,w) -> weighWriter doc n (either (error . show) id . runPure . w def))
34− [("markdown", writeMarkdown)
35− ,("html", writeHtml5String)
36− ,("docbook", writeDocbook5)
37− ,("latex", writeLaTeX)
38− ,("commonmark", writeCommonMark)
39− ]
40−
41−weighWriter :: Pandoc -> String -> (Pandoc -> Text) -> Weigh ()
42−weighWriter doc name writer = func (name ++ " writer") writer doc
43−
44−weighReader :: Pandoc -> Text -> (Text -> Pandoc) -> Weigh ()
45−weighReader doc name reader =
46− case lookup name writers of
47− Just (TextWriter writer) ->
48− let inp = either (error . show) id $ runPure $ writer def{ writerWrapText = WrapAuto} doc
49− in func (unpack $ name <> " reader") reader inp
50− _ -> return () -- no writer for reader
cabal.project+9−4
@@ -1,8 +1,13 @@
11packages: pandoc.cabal
2−
32tests: True
3+flags: +embed_data_files
44
5−package pandoc
6− flags: +embed_data_files -trypandoc
7− ghc-options: -j +RTS -A64m -RTS
5+source-repository-package
6+ type: git
7+ location: https://github.com/jgm/doctemplates
8+ tag: 428c26d5303cf7a2b1051fe1ffd9aafe9ba71c81
89
10+source-repository-package
11+ type: git
12+ location: https://github.com/jgm/citeproc
13+ tag: 4ce0501cd6f9c86eee087afcf878c256e49b9615
changelog.md+1437−2
@@ -1,5 +1,1437 @@
11# Revision history for pandoc
22
3+## pandoc 2.14.0.3 (2021-06-22)
4+
5+ * Text.Pandoc.MediaBag `insertMediaBag`: ensure we get a sane mediaPath
6+ for URLs (#7391). In earlier 2.14.x versions, we'd get
7+ incorrect paths for resources downloaded from URLs when the
8+ media are extracted (including in PDF production).
9+ * Text.Pandoc.Parsing: improve `emailAddress` (#7398).
10+ Previously the parser would accept characters in domains
11+ that are illegal in domains, and this sometimes caused it
12+ to gobble bits of the following text.
13+ * txt2tags reader: modify the email address parser so
14+ it still includes form parameters, even after the change to
15+ `emailAddress` in Text.Pandoc.Parsing.
16+ * Text.Pandoc.Readers.Metadata: Fix regression with comment-only YAML
17+ metadata blocks (#7400).
18+ * reveal.js writer and template: better handling of options. Previously
19+ it was impossible to specify false values for options that default to
20+ true (e.g. `center`); setting the option to false just caused the portion
21+ of the template setting the option to be omitted. Now we prepopulate
22+ all the variables with their default values, including them all
23+ unconditionally and allowing them to be overridden.
24+ * Markdown writer: Fix regression in code blocks with attributes (#7397).
25+ Code blocks with a single class but nonempty attributes
26+ were having attributes drop as a result of #7242.
27+ * LaTeX writer:
28+ + Add strut at end of minipage if it contains line breaks.
29+ Without them, the last line is not as tall as it should be in
30+ some cases.
31+ + Always use a minipage for cells with line breaks, when
32+ width information is available (#7393). Otherwise the way we treat them
33+ can lead to content that overflows a cell.
34+ + Use `\strut` instead of `~` before `\\` in empty line.
35+ * Use lts-18.0 stack resolver.
36+ * Require skylighting 0.10.5.2 (adding support for Swift).
37+ * Require commonmark 0.2.1.
38+ * Rephrase section on unsafe HTML in manual.
39+ * Create SECURITY.md
40+
41+## pandoc 2.14.0.2 (2021-06-13)
42+
43+ * Fix MediaBag regressions (#7345). iIn the 2.14 release `--extract-media`
44+ stopped working as before; there could be mismatches between the
45+ paths in the rendered document and the extracted media.
46+ This patch makes several changes that restore the earlier behavior
47+ (while keeping the same API). The `mediaPath` in 2.14 was always
48+ constructed from the SHA1 hash of the media contents. Now, we
49+ preserve the original path unless it's an absolute path or contains
50+ `..` segments (in that case we use a path based on the SHA1 hash of
51+ the contents).
52+
53+ In Text.Pandoc.MediaBag, `mediaDirectory` and `mediaItems` now use the
54+ `mediaPath`, rather than the mediabag key, for the first component of the
55+ tuple. This makes more sense, I think, and fits with the documentation of
56+ these functions; eventually, though, we should rework the API so that
57+ `mediaItems` returns both the keys and the MediaItems.
58+
59+ In Text.Pandoc.Class.IO, rewriting of source paths in `extractMedia` has
60+ been fixed.
61+
62+ In Text.Pandoc.Class.PandocMonad, `fillMediaBag` has been modified so that
63+ it doesn't modify image paths (that was part of the problem in #7345).
64+
65+ We now do path normalization (e.g. `\` separators on Windows) in
66+ writing the media.
67+
68+ * Text.Pandoc.PDF:
69+
70+ + Text.Pandoc.PDF: Fix regression in 2.14 for generation of PDFs with
71+ SVGs (#7344).
72+ + Only print relevant part of environment on `--verbose`. Since
73+ `--verbose` output might be put in an issue, we want to avoid
74+ spilling out secrets in environment variables.
75+
76+ * Markdown reader: fix pipe table regression in 2.11.4 (#7343).
77+ Previously pipe tables with empty headers (that is, a header
78+ line with all empty cells) would be rendered as headerless
79+ tables. This broke in 2.11.4. The fix here is to produce an
80+ AST with an empty table head when a pipe table has all empty
81+ header cells.
82+
83+ * LaTeX reader: don't allow optional `*` on symbol control sequences
84+ (#7340). Generally we allow optional starred variants of LaTeX commands
85+ (since many allow them, and if we don't accept these explicitly,
86+ ignoring the star usually gives acceptable results). But we
87+ don't want to do this for `\(*\)` and similar cases.
88+
89+ * Docx reader: handle absolute URIs in Relationship Target (#7374).
90+
91+ * Docx writer: fix handling of empty table headers (Albert Krewinkel,
92+ #7369). A table header which does not contain any cells is now treated as
93+ an empty header.
94+
95+ * LaTeX writer: Fix regression in table header position (#7347).
96+ In recent versions the table headers were no longer bottom-aligned
97+ (if more than one line). This patch fixes that by using minipages
98+ for table headers in non-simple tables.
99+
100+ * CommonMark writer:
101+
102+ + Do not use simple class for fenced-divs (Jan Tojnar, amends #7242.)
103+ + Do not throw away attributes when `Ext_attributes` is enabled.
104+ `Ext_attributes` covers at least the following:
105+ `Ext_fenced_code_attributes`, `Ext_header_attributes`,
106+ `Ext_inline_code_attributes`, `Ext_link_attributes`.
107+
108+ * Markdown writer:
109+
110+ + Allow `pipe_tables` to be disabled for commonmark formats
111+ (`commonmark_x`, `gfm`) (#7375).
112+ + Re-use functions from Text.Pandoc.Markdown.Inline (Jan Tojnar).
113+
114+ * DocBook writer: Remove non-existent admonitions (Jan Tojnar).
115+ `attention`, `error` and `hint` are reStructuredText specific.
116+
117+ * HTML writer: Don't omit width attribute on div (#7342).
118+
119+ * Text.Pandoc.MIME, `extensionFromMimeType`: add a few special cases.
120+ When we do a reverse lookup in the MIME table, we just get the
121+ last match, so when the same mime type is associated with several
122+ different extensions, we sometimes got weird results, e.g. `.vs`
123+ for `text/plain`. These special cases help us get the most standard
124+ extensions for mime types like `text/plain`.
125+
126+ * Lua utils: fix handling of table headers in `from_simple_table` (Albert
127+ Krewinkel, #7369). Passing an empty list of header cells now results
128+ in an empty table header.
129+
130+ * Text.Pandoc.Citeproc:
131+
132+ + Avoid duplicate classes and attributes on references div.
133+ + Fix regression in citeproc processing (#7376). If inline
134+ references are used (in the metadata `references` field), we
135+ should still only include in the bibliography items that are
136+ actually cited (unless `nocite` is used).
137+
138+ * Require citeproc 0.4.0.1. This fixes a bug which led to doubled
139+ "et al." in some (rare) circumstances.
140+
141+ * MANUAL.txt:
142+
143+ + Mention GladTeX for EPUB export (Sebastian Humenda).
144+ This updates the manual and the web site about the GladTeX usage.
145+ + More details and a useful link for YAML syntax.
146+
147+ * CONTRIBUTING.md: update modules overview (Albert Krewinkel).
148+
149+ * using-the-pandoc-api.md: switch from String to Text (Albert Krewinkel).
150+
151+
152+## pandoc 2.14.0.1 (2021-06-01)
153+
154+ * Commonmark reader: Fix regression in 2.14 with YAML metdata block parsing,
155+ which could cause the document body to be omitted after metadata (#7339).
156+
157+ * HTML reader: fix column width regression in 2.14 (#7334).
158+ Column widths specified with a style attribute were off by a factor of 100.
159+
160+ * Markdown reader: in `rebasePaths`, check for both Windows and Posix
161+ absolute paths. Previously Windows pandoc was treating
162+ `/foo/bar.jpg` as non-absolute.
163+
164+ * Text.Pandoc.Logging: In rendering `LoadedResource`, use relative paths.
165+
166+ * Docx writer: fix regression on captions (#7328). The "Table Caption"
167+ style was no longer getting applied. (It was overwritten by "Compact.")
168+
169+ * Use commonmark-extensions 0.2.1.2
170+
171+## pandoc 2.14 (2021-05-28)
172+
173+ * Change reader types, allowing better tracking of source positions
174+ [API change]. Previously, when multiple file arguments were provided,
175+ pandoc simply concatenated them and passed the contents to the readers,
176+ which took a Text argument. As a result, the readers had no way of knowing
177+ which file was the source of any particular bit of text. This meant that
178+ we couldn't report accurate source positions on errors or include accurate
179+ source positions as attributes in the AST. More seriously, it meant that
180+ we couldn't resolve resource paths relative to the files containing them
181+ (see e.g. #5501, #6632, #6384, #3752).
182+
183+ * Add `rebase_relative_paths` extension (#3752). When enabled, this
184+ extension rewrites relative image and link paths by prepending
185+ the (relative) directory of the containing file. This
186+ behavior is useful when your input sources are split
187+ into multiple files, across several directories, with files
188+ referring to images stored in the same directory. The
189+ extension can be enabled for all markdown and commonmark-based formats.
190+
191+ * Add Text.Pandoc.Sources (exported module), with a `Sources` type and a
192+ `ToSources` class. A `Sources` wraps a list of `(SourcePos, Text)` pairs
193+ [API change]. A parsec `Stream` instance is provided for `Sources`. The
194+ module also exports versions of parsec's `satisfy` and other Char parsers
195+ that track source positions accurately from a `Sources` stream (or any
196+ instance of the new `UpdateSourcePos` class).
197+
198+ * Text.Pandoc.Parsing
199+
200+ + Export the modified Char parsers defined in Text.Pandoc.Sources
201+ instead of the ones parsec provides. Modified parsers to use a
202+ `Sources` as stream [API change].
203+ + Improve include file functions [API change]. Remove old
204+ `insertIncludedFileF`. Give `insertIncludedFile` a more general type,
205+ allowing it to be used where `insertIncludedFileF` was.
206+ + Add parameter to the `citeKey` parser from
207+ Text.Pandoc.Parsing, which controls whether the `@{..}`
208+ syntax is allowed [API change].
209+
210+ * Text.Pandoc.Error: Modified the constructor `PandocParsecError` to take a
211+ `Sources` rather than a `Text` as first argument, so parse error locations
212+ can be accurately reported.
213+
214+ * Fix source position reporting for YAML bibliographies (#7273).
215+
216+ * Issue error message when reader or writer format is malformed
217+ (#7231). Previously we exited with an error status but (due to a bug)
218+ no message.
219+
220+ * Smarter smart quotes (#7216, #2103). Treat a leading `"` with no
221+ closing `"` as a left curly quote. This supports the practice, in
222+ fiction, of continuing paragraphs quoting the same speaker without an
223+ end quote. It also helps with quotes that break over lines in line blocks.
224+
225+ * Markdown reader:
226+
227+ + Use MetaInlines not MetaBlocks for multimarkdown metadata fields.
228+ This gives better results in converting to e.g. pandoc markdown.
229+ + Implement curly-brace syntax for Markdown citation keys (#6026).
230+ The change provides a way to use citation keys that contain
231+ special characters not usable with the standard citation key syntax.
232+ Example: `@{foo_bar{x}'}` for the key `foo_bar{x}`. It also allows
233+ separating citation keys from immediately following text, e.g. `@{foo}A`.
234+
235+ * RST reader:
236+
237+ + Seek include files in the directory of the file
238+ containing the include directive, as RST requires (#6632).
239+ + Use `insertIncludedFile` from Text.Pandoc.Parsing
240+ instead of reproducing much of its code.
241+
242+ * Org reader: Resolve org includes relative to the directory containing the
243+ file containing the INCLUDE directive (#5501).
244+
245+ * ODT reader: Treat tabs as spaces (#7185, niszet).
246+
247+ * Docx reader:
248+
249+ + Add handling of vml image objects (#7257, mbrackeantidot).
250+ + Support new table features (Emily Bourke, #6316): column
251+ spans, row spans, multiple header rows, table description
252+ (parsed as a simple caption), captions, column widths.
253+
254+ * LaTeX reader:
255+
256+ + Improved siunitx support (#6658, #6620).
257+ + Better support for `\xspace` (#7299).
258+ + Improve parsing of `\def` macros. We previously set "verbatim mode"
259+ even for parsing the initial `\def`; this caused problems
260+ for `\def` nested inside another `\def`.
261+ + Implement `\newif`.
262+
263+ * ConTeXt writer: improve ordered lists (#5016, Denis Maier).
264+ Change ordered list from itemize to enumerate. Add new
265+ itemgroup for ordered lists. Remove manual insertion of
266+ width attributes. Use tabular figures in ordered list
267+ enumerators.
268+
269+ * HTML reader:
270+
271+ + Don't fail on unmatched closing "script" tag (Albert Krenkel, #7282).
272+ + Keep h1 tags as normal headers (#2293, Albert Krewinkel).
273+ The tags `<title>` and `<h1 class="title">` often contain the same
274+ information, so the latter was dropped from the document. However, as
275+ this can lead to loss of information, the heading is now always
276+ retained. Use `--shift-heading-level-by=-1` to turn the `<h1>`
277+ into the document title, or a filter to restore the previous behavior.
278+ + Handle relative lengths (e.g. `2*`) in HTML column widths (#4063).
279+ See <https://www.w3.org/TR/html4/types.html#h-6.6>.
280+
281+ * DocBook/JATS readers:
282+
283+ + Fix mathml regression caused by the switch in XML libraries (#7173).
284+ + Fix "phrase" in DocBook: take classes from "role" not "class" (#7195).
285+
286+ * DocBook reader: ensure that first and last names are separated (#6541).
287+
288+ * Jira reader (Albert Krewinkel, #7218):
289+
290+ + Support "smart" links: `[alias|https://example.com|smart-card]` syntax.
291+ + Allow spaces and most unicode characters in attachment links.
292+ + No longer require a newline character after `{noformat}`.
293+ + Only allow URI path segment characters in bare links.
294+ + The `file:` schema is no longer allowed in bare links; these
295+ rarely make sense.
296+
297+ * Plain writer: handle superscript unicode minus (#7276).
298+
299+ * LaTeX writer:
300+
301+ + Better handling of line breaks in simple tables (#7272).
302+ Now we also handle the case where they're embedded in other elements,
303+ e.g. spans.
304+ + For beamer output, support `exampleblock` and `alertblock` (#7278).
305+ A block will be rendered as an `exampleblock` if the heading
306+ has class `example` and an `alertblock` if it has class `alert`.
307+ + Separate successive quote chars with thin space (#6958,
308+ Albert Krewinkel). Successive quote characters are separated with
309+ a thin space to improve readability and to prevent unwanted ligatures.
310+ Detection of these quotes sometimes had failed if the second quote
311+ was nested in a span element.
312+ + Separate successive quote chars with thin space (#6958, Albert
313+ Krewinkel).
314+
315+ * EPUB Writer: Fix belongs-to-collection XML id choice (#7267, nuew).
316+ The epub writer previously used the same XML id for both the book
317+ identifier and the epub collection. This causes an error on epubcheck.
318+
319+ * BibTeX/BibLaTeX writer: Handle `annote` field (#7266).
320+
321+ * ZimWiki writer: allow links and emphasis in headers (#6605,
322+ Albert Krewinkel).
323+
324+ * ConTeXt writer:
325+
326+ + Support blank lines in line blocks (#6564, Albert Krewinkel,
327+ thanks to @denismaier).
328+ + Use span identifiers as reference anchors (#7246, Albert Krewinkel).
329+
330+ * HTML writer:
331+
332+ + Keep attributes from code nested below `pre` tag (#7221,
333+ Albert Krewinkel). If a code block is defined with `<pre><code
334+ class="language-x">…</code></pre>`, where the `<pre>` element has no
335+ attributes, then the attributes from the `<code>` element are used
336+ instead. Any leading `language-` prefix is dropped in the code's
337+ `class` attribute are dropped to improve syntax highlighting.
338+ + Ensure headings only have valid attribs in HTML4 (#5944, Albert
339+ Krewinkel).
340+ + Parse `<header>` as a Div (Albert Krewinkel).
341+
342+ * Org writer:
343+
344+ + Inline latex envs need newlines (#7252, tecosaur).
345+ As specified in https://orgmode.org/manual/LaTeX-fragments.html, an
346+ inline \begin{}...\end{} LaTeX block must start on a new line.
347+ + Use LaTeX style maths deliminators (#7196, tecosaur).
348+
349+ * JATS writer (Albert Krewinkel):
350+
351+ + Use either styled-content or named-content for spans (#7211).
352+ If the element has a content-type attribute, or at least one class,
353+ then that value is used as `content-type` and the span is put inside
354+ a `<named-content>` element. Otherwise a `<styled-content>` element
355+ is used instead.
356+ + Reduce unnecessary use of `<p>` elements for wrapping (#7227).
357+ The `<p>` element is used for wrapping in cases were the contents
358+ would otherwise not be allowed in a certain context. Unnecessary
359+ wrapping is avoided, especially around quotes (`<disp-quote>` elements).
360+ + Convert spans to `<named-content>` elements (#7211). Spans with
361+ attributes are converted to `<named-content>` elements instead of
362+ being wrapped with `<milestone-start/>` and `<milestone-end>`
363+ elements. Milestone elements are not allowed in documents using the
364+ articleauthoring tag set, so this change ensures the creation of valid
365+ documents.
366+ + Add footnote number as label in backmatter (#7210). Footnotes in the
367+ backmatter are given the footnote's number as a label. The
368+ articleauthoring output is unaffected from this change, as footnotes
369+ are placed inline there.
370+ + Escape disallows chars in identifiers. XML identifiers must start
371+ with an underscore or letter, and can contain only a limited set
372+ of punctuation characters. Any IDs not adhering to these rules are
373+ rewritten by writing the offending characters as `Uxxxx`,
374+ where `xxxx` is the character's hex code.
375+
376+ * Jira writer: use `{color}` when span has a color attribute
377+ (Albert Krewinkel, tarleb/jira-wiki-markup#10).
378+
379+ * Docx writer:
380+
381+ + Autoset table width if no column has an explicit width (Albert
382+ Krewinkel).
383+ + Extract Table handling into separate module (Albert Krewinkel).
384+ + Support colspans and rowspans in tables (Albert Krewinkel, #6315).
385+ + Support multirow table headers (Albert Krewinkel).
386+ + Improve integration of settings from reference.docx (#1209).
387+ This change allows users to create a reference.docx that
388+ sets `w:proofState` for spelling or grammar to `dirty`,
389+ so that spell/grammar checking will be triggered on the
390+ generated docx.
391+ + Copy over more settings from reference.docx (#7240). From settings.xml
392+ in the reference-doc, we now include: `zoom`, `embedSystemFonts`,
393+ `doNotTrackMoves`, `defaultTabStop`, `drawingGridHorizontalSpacing`,
394+ `drawingGridVerticalSpacing`, `displayHorizontalDrawingGridEvery`,
395+ `displayVerticalDrawingGridEvery`, `characterSpacingControl`,
396+ `savePreviewPicture`, `mathPr`, `themeFontLang`, `decimalSymbol`,
397+ `listSeparator`, `autoHyphenation`, `compat`.
398+ + Set zoom to 100% by default in settings.xml.
399+ + Align math options more with current Word defaults (e.g. Cambria Math
400+ font).
401+ + Remove `rsid`s from default settings.xml. Word will add these
402+ when revisions are made.
403+
404+ * Ms writer: Handle tables with multiple paragraphs (#7288).
405+ Previously they overflowed the table cell width. We now set line lengths
406+ per-cell and restore them after the table has been written.
407+
408+ * Markdown writer:
409+
410+ + Use cleaner braceless syntax for code blocks with a
411+ single class (#7242, Jan Tojnar).
412+ + Add quotes properly in markdown YAML metadata fields (#7245).
413+ This fixes a bug, which caused the writer to look at the *last*
414+ rather than the *first* character in determining whether quotes
415+ were needed. So we got spurious quotes in some cases and
416+ didn't get necessary quotes in others.
417+ + Use `@{..}` syntax for citations when needed.
418+ + Use fewer unneeded escapes for `#` (see #6259).
419+ + Improve escaping of `@`. We need to escape literal `@` before
420+ `{` because of the new citation syntax.
421+
422+ * Commonmark writer: Use backslash escapes for `<` and `|`...
423+ instead of entities (#7208).
424+
425+ * Powerpoint writer: allow `monofont` to be specified in metadata
426+ (#7187).
427+
428+ * LaTeX template:
429+
430+ + Use non-starred names for xcolor color names (#6109).
431+ This should make svgnames and x11names work properly.
432+ + Fix bad vertical spacing after bibliography (#7234, badumont).
433+ + List of figures before list of tables (#7235, Julien Dutant).
434+ + Move CSL macro definitions before header-includes so they can be
435+ overridden (#7286).
436+ + Improve treatment of CSL `entry-spacing` (#7296).
437+ Previously with the default template settings (`indent` variable
438+ not set), we would get interparagraph spaces separating bib
439+ entries even with `entry-spacing="0"`. On the other hand,
440+ setting `entry-spacing="2"` gave ridiculously large spacing.
441+ This change makes the spacing caused by `entry-spacing` a multiple
442+ of `\parskip` by default, which gives aesthetically reasonable
443+ output. Those who want a larger or smaller unit (e.g. because
444+ they use `indent` which sets `\parskip` to 0) may
445+ `\setlength{\cslentryspacingunit}{10pt}` in header-includes
446+ to override the defaults.
447+ + Move title, author, date up to top of preamble (#7295).
448+ This allows header-includes to use them, and puts them
449+ in a position where you can see them immediately.
450+ + Define commands for zero width non-joiner character
451+ (#6639, Albert Krewinkel). The zero-width non-joiner character
452+ is used to avoid ligatures (e.g. in German).
453+
454+ * ConTeXt template:
455+
456+ + Define `enumerate` itemgroup (#5016, Denis Maier).
457+ + List of figures before list of tables (#7235, Julien Dutant).
458+
459+ * reveal.js template:
460+
461+ + Support `toc-title` (#7171, Florian Kohrt).
462+ + Use `hash: true` by default rather than `history: true` (#6968).
463+
464+ * HTML-based slide shows: add support for `institute` (#7289, Thomas
465+ Hodgson).
466+
467+ * Text.Pandoc.Extensions: Add constructor `Ext_rebase_relative_paths` to
468+ `Extensions` [API change].
469+
470+ * Text.Pandoc.XML.Light: add Eq, Ord instances for Content,
471+ Element, Attr, CDataKind [API change].
472+
473+ * Text.Pandoc.MediaBag:
474+
475+ + Change type to use a `Text` key instead of `[FilePath]`.
476+ We normalize the path and use `/` separators for consistency.
477+ + Export `MediaItem` type [API change].
478+ + Change `MediaBag` type to a map from Text to MediaItem [API change].
479+ + `lookupMedia` now returns a `MediaItem` [API change].
480+ + Change `insertMedia` so it sets the `mediaPath` to a filename based on
481+ the SHA1 hash of the contents. This will be used when contents
482+ are extracted.
483+
484+ * Text.Pandoc.Class.PandocMonad:
485+
486+ + Remove `fetchMediaResource` [API change]. Use `fetchItem` to get
487+ resources in `fillMediaBag`.
488+ + Add informational message in `downloadOrRead` indicating what path
489+ local resources have been loaded from.
490+
491+ * Text.Pandoc.Logging:
492+
493+ + Remove single quotes around paths in messages.
494+ + Add LoadedResource constructor to LogMessage [API change].
495+ This is for INFO-level messages telling where image data has been
496+ loaded from. (This can vary because of the resource path.)
497+
498+ * Text.Pandoc.Asciify: simplify code and export `toAsciiText` [API change].
499+ Instead of encoding a giant (and incomplete) map, we now
500+ just use unicode-transforms to normalize the text to
501+ a canonical decomposition, and manipulate the result.
502+
503+ * App: allow tabs expansion even if file-scope is used (Albert Krewinkel,
504+ #6709). Tabs in plain-text inputs are now handled correctly, even if
505+ the `--file-scope` flag is used.
506+
507+ * Add new internal module Text.Pandoc.Writers.GridTable (Albert Krewinkel).
508+
509+ * Text.Pandoc.Highlighting: Change type of `languagesByExtension`, adding
510+ a parameter for a `SyntaxMap` [API change] (Jan Tojnar, #7241).
511+ Languages defined using `--syntax-definition` were not recognized by
512+ `languagesByExtension`. This patch corrects that, allowing the writers
513+ to see all custom definitions. The LaTeX writer still uses the default
514+ syntax map, but that's okay in that context, since
515+ `--syntax-definition` won't create new listings styles.
516+
517+ * Text.Pandoc.Citeproc:
518+
519+ + Ensure that CSL-related attributes are passed on to a Div with id
520+ 'refs'. Otherwise things like `entry-spacing` won't work when
521+ such Divs are used.
522+ + Use metadata's `lang` for the lang parameter of citeproc, overriding
523+ `localeLanguage`.
524+ + Recognize locators spelled with a capital letter (#7323).
525+ + Add a comma and a space in front of the suffix if it doesn't start
526+ with space or punctuation (#7324).
527+ + Don't detect math elements as locators (#7321).
528+
529+ * Remove Text.Pandoc.BCP47 module [API change]. Use types and functions
530+ from UnicodeCollation.Lang instead. This is a richer implementation
531+ of BCP 47.
532+
533+ * Text.Pandoc.Shared:
534+
535+ + Fix regression in grid tables for wide characters (#7214).
536+ In the translation from String to Text, a char-width-sensitive
537+ `splitAt'` was dropped. This commit reinstates it and uses it to make
538+ `splitTextByInstances` char-width sensitive.
539+ + Add `getLang` (formerly in the now-removed BCP47) [API change].
540+
541+ * Text.Pandoc.SelfContained: use `application/octet-stream`
542+ for unknown mime types instead of halting with an error (#7202).
543+
544+ * Lua filters: respect Inlines/Blocks filter functions in `pandoc.walk_*`
545+ (Albert Krewinkel).
546+
547+ * Add text as build-depend for trypandoc (#7193, Roman Beránek).
548+
549+ * Bump upper-bounds for network-uri, time, attoparsec.
550+
551+ * Use citeproc 0.4.
552+
553+ * Use texmath 0.12.3.
554+
555+ * Use jira-wiki-markup 1.3.5 (Albert Krewinkel).
556+
557+ * Require latest skylighting (fixes a bug in XML syntax highlighting).
558+
559+ * Use latest xml-conduit.
560+
561+ * Use latest commonmark, commonmark-extensions, commonmark-pandoc.
562+
563+ * Use haddock-library-1.10.0 (Albert Krewinkel).
564+
565+ * Allow compilation with base 4.15 (Albert Krewinkel).
566+
567+ * MANUAL:
568+
569+ + Add information about `lang` and bibliography sorting.
570+ + Add info about YAML escape sequences, link to spec (#7152,
571+ Albert Krewinkel).
572+ + Note that `institute` variable works for HTML-based slides.
573+ + Update documentation on citation syntax.
574+ + Add citation example for locators and suffixes (Tristan Stenner)
575+
576+ * Updated and fixed typos in documentation (Charanjit Singh,
577+ Anti-Distinctlyminty, Tatiana Porras, obcat).
578+
579+ * Add instructions for installing pandoc-types before compiling filter.
580+
581+ * INSTALL: add note that parallel installations should be avoided
582+ (#6865).
583+
584+ * Remove `biblatex-nussbaum.md` test. It is basically the same
585+ as `biblaetx-quotes.md`.
586+
587+ * Command tests: fail if a file contains no tests---and fix a
588+ test that failed in that way!
589+
590+ * Use smaller images in tests, reducing the size of the source tarball by 8 MB.
591+
592+
593+## pandoc 2.13 (2021-03-21)
594+
595+ * Support `yaml_metadata_block` extension for `commonmark`, `gfm` (#6537).
596+ This supported is a bit more limited than with pandoc's
597+ `markdown`. The YAML block must be the first thing in the input,
598+ and the leaf notes are parsed in isolation from the rest of
599+ the document. So, for example, you can't use reference
600+ links if the references are defined later in the document.
601+
602+ * Fix fallback to default partials when custom templates are used.
603+ If the directory containing a template does not contain the partial,
604+ it should be sought in the default templates, but this was not
605+ working properly (#7164).
606+
607+ * Handle `nocite` better with `--biblatex` and `--natbib` (#4585).
608+ Previously the nocite metadata field was ignored with these formats.
609+ Now it populates a `nocite-ids` template variable and causes a
610+ `\nocite` command to be issued.
611+
612+ * Text.Pandoc.Citeproc: apply `fixLinks` correctly (#7130). This is code
613+ that incorporates a prefix like `https://doi.org/` into a following link
614+ when appropriate.
615+
616+ * Text.Pandoc.Shared:
617+
618+ + Remove `backslashEscapes`, `escapeStringUsing` [API change]. Replace
619+ these inefficient association list lookups with more efficient escaping
620+ functions in the writers that used them (for a 10-25% performance boost
621+ in org, haddock, rtf, texinfo writers).
622+ + Remove `ToString`, `ToText` typeclasses [API change]. These were needed
623+ for the transition from String to Text, but they are no longer used and
624+ may clash with other things.
625+ + Simplify `compactDL`.
626+
627+ * Text.Pandoc.Parsing:
628+
629+ + Change type of `readWithM` so that it is no longer polymorphic
630+ [API change]. The `ToText` class has been removed, and now that we've
631+ completed the transition to Text we no longer need this to operate
632+ on Strings.
633+ + Remove `F` type synonym [API change]. Muse and Org were defining their
634+ own `F` anyway.
635+
636+ * Text.Pandoc.Readers.Metadata:
637+
638+ + Export `yamlMetaBlock` [API change].
639+ + Make `yamlBsToMeta`, `yamlBsToRefs` polymorphic on the parser state
640+ [API change].
641+
642+ * Markdown reader: Fix regression with `tex_math_backslash` (#7155).
643+
644+ * MediaWiki reader: Allow block-level content in notes (ref) (#7145).
645+
646+ * Jira reader (Albert Krewinkel):
647+
648+ + Fixed parsing of autolinks (i.e., of bare URLs in the text).
649+ Previously an autolink would take up the rest of a line, as spaces
650+ were allowed characters in these items.
651+ + Emoji character sequences no longer cause parsing failures. This was
652+ due to missing backtracking when emoji parsing fails.
653+ + Mark divs created from panels with class "panel".
654+
655+ * RST reader: fix logic for ending comments (#7134). Previously comments
656+ sometimes got extended too far.
657+
658+ * DocBook writer: include Header attributes as XML attributes on
659+ section (Erik Rask). Attributes with key names that are not allowed
660+ as XML attributes are dropped, as are attributes with invalid values
661+ and `xml:id` (DocBook 5) and `id` (DocBook 4).
662+
663+ * Docx writer:
664+
665+ + Make `nsid` in `abstractNum` deterministic. Previously we assigned
666+ a random number, but we don't need random values, so now we just
667+ assign a value based on the list marker.
668+ + Use integral values for `w:tblW` (#7141).
669+
670+ * Jira writer (Albert Krewinkel):
671+
672+ + Block quotes are only rendered as `bq.` if they do not contain a
673+ linebreak.
674+ + Jira writer: improve div/panel handling. Include div attributes in
675+ panels, always render divs with class `panel` as panels, and
676+ avoid nesting of panels.
677+
678+ * HTML writer: Add warnings on duplicate attribute values.
679+ This prevents emitting invalid HTML. Ultimately it would be good to
680+ prevent this in the types themselves, but this is better for now.
681+
682+ * Org writer: Prevent unintended creation of ordered list items (#7132,
683+ Albert Krewinkel). Adjust line wrapping if default wrapping would cause
684+ a line to be read as an ordered list item.
685+
686+ * JATS templates: support 'equal-contrib' attrib for authors (Albert
687+ Krewinkel). Authors who contributed equally to a paper may be marked
688+ with `equal-contrib`.
689+
690+ * reveal.js template: replace JS comment with HTML (#7154, Florian Kohrt).
691+
692+ * Text.Pandoc.Logging: Add `DuplicateAttribute` constructor to `LogMessage`.
693+ [API change]
694+
695+ * Use `-j4` for linux release build. This speeds up the build dramatically
696+ on arm.
697+
698+ * cabal.project: remove ghcoptions. Move flags to top level, so they can
699+ be set differently on the command line.
700+
701+ * Require latest texmath, skylighting, citeproc, jira-wiki-markup.
702+ (The latest skylighting fixes a bad bug with Haskell syntax highlighting.)
703+ Narrow version bounds for texmath, skylighting, and citeproc, since
704+ the test output depend on them.
705+
706+ * Use doclayout 0.3.0.2. This significantly reduces the time and memory
707+ needed to compile pandoc.
708+
709+ * Use `foldl'` instead of `foldl` everywhere.
710+
711+ * Update bounds for random (#7156, Alexey Kuleshevich).
712+
713+ * Remove uses of some partial functions.
714+
715+ * Don't bake in a larger stack size for the executable.
716+
717+ * Test improvements:
718+
719+ + Use `getExecutablePath` from base, avoiding the dependency on
720+ `executable-path`.
721+ + Factor out `setupEnvironment` in Helpers, to avoid code duplication.
722+ + Fix finding of data files by setting teh `pandoc_datadir` environment
723+ variable when we shell out to pandoc. This avoids the need to use
724+ `--data-dir` for the tests, which caused problems finding `pandoc.lua`
725+ when compiling without the `embed_data_files` flag (#7163).
726+
727+ * Benchmark improvements:
728+
729+ + Build `+RTS -A8m -RTS` into default ghc-options for benchmark.
730+ This is necessary to get accurate benchmark results; otherwise we
731+ are largely measuring garbage collecting, some not related to the
732+ current benchmark.
733+ + Allow specifying BASELINE file in 'make bench' for comparison
734+ (otherwise the latest benchmark is chosen by default).
735+ + Force `readFile` in benchmarks early (Bodigrim).
736+
737+ * CONTRIBUTING: suggest using a `cabal.project.local` file (#7153,
738+ Albert Krewinkel).
739+
740+ * Add ghcid-test to Makefile. This loads the test suite in ghcid.
741+
742+
743+## pandoc 2.12 (2021-03-08)
744+
745+ * `--resource-path` now accumulates if specified multiple
746+ times (#6152). Resource paths specified later on the command line are
747+ prepended to those specified earlier. Thus,
748+ `--resource-path foo --resource-path bar:baz` is equivalent to
749+ `--resource-path bar:bas:foo`. (The previous behavior was
750+ for the last `--resource-path` to replace all the rest.)
751+ `resource-path` in defaults files behaves the same way: it
752+ will be prepended to the resource path set by earlier
753+ command line options or defaults files. This change
754+ facilitates the use of multiple defaults files: each can
755+ specify a directory containing resources it refers to
756+ without clobbering the resource paths set by the others.
757+
758+ * Allow defaults files to refer to the home directory, the
759+ user data directory, and the directory containing the defaults file
760+ itself (#5871, #5982, #5977). In fields that expect file paths
761+ (and only in these fields),
762+
763+ + `${VARIABLE}` will expand to the value of the environment variable
764+ `VARIABLE` (and in particular `${HOME}` will expand to the path
765+ of the home directory). A warning will be raised for undefined
766+ variables.
767+ + `${USERDATA}` will expand to the path of the user data
768+ directory in force when the defaults file is being processed.
769+ + `${.}` will expand to the directory containing the defaults file.
770+ (This allows default files to be placed in a directory containing
771+ resources they make use of.)
772+
773+ * When downloading content from URL arguments, be sensitive to
774+ the character encoding (#5600). We can properly handle UTF-8 and latin1
775+ (ISO-8859-1); for others we raise an error. Fall back to latin1 if
776+ no charset is given in the mime type and UTF-8 decoding fails.
777+
778+ * Allow abbreviations that don't end in a period to be
779+ specified using `--abbreviations` (#7124).
780+
781+ * Add new unexported module Text.Pandoc.XML.Light, as well
782+ as Text.Pandoc.XML.Light.Types, Text.Pantoc.XML.Light.Proc,
783+ Text.Pandoc.XML.Light.Output. (Closes #6001, #6565, #7091).
784+
785+ This module exports definitions of `Element` and `Content`
786+ that are isomorphic to xml-light's, but with Text
787+ instead of String. This allows us to keep most of the code in existing
788+ readers that use xml-light, but avoid lots of unnecessary allocation.
789+
790+ We also add versions of the functions from xml-light's
791+ Text.XML.Light.Output and Text.XML.Light.Proc that operate on our
792+ modified XML types, and functions that convert xml-light types to our
793+ types (since some of our dependencies, like texmath, use xml-light).
794+
795+ We export functions that use xml-conduit's parser to produce an
796+ `Element` or `[Content]`. This allows existing pandoc code to use
797+ a better parser without much modification.
798+
799+ The new parser is used in all places where xml-light's parser was
800+ previously used. Benchmarks show a significant performance improvement
801+ in parsing XML-based formats (with docbook, opml, jats, and docx
802+ almost twice as fast, odt and fb2 more than twice as fast).
803+
804+ In addition, the new parser gives us better error reporting than
805+ xml-light. We report XML errors, when possible, using the new
806+ `PandocXMLError` constructor in `PandocError`.
807+
808+ These changes revealed the need for some changes in the tests. The
809+ docbook-reader.docbook test lacked definitions for the entities it used;
810+ these have been added. And the docx golden tests have been updated,
811+ because the new parser does not preserve the order of attributes.
812+
813+ * DocBook reader:
814+
815+ + Avoid expensive tree normalization step, as it is not necessary
816+ with the new XML parser.
817+ + Support `informalfigure` (#7079) (Nils Carlson).
818+
819+ * Docx reader:
820+
821+ + Use Map instead of list for Namespaces. This gives a speedup of
822+ about 5-10%. With this and the XML parsing changes, the docx reader
823+ is now about twice as fast as in the previous release.
824+
825+ * HTML reader:
826+
827+ + Small performance tweaks.
828+ + Also, remove exported class `NamedTag(..)` [API change]. This was just
829+ intended to smooth over the transition from String to Text and is no
830+ longer needed.
831+ + As a result, the functions `isInlineTag` and `isBlockTag`
832+ are no longer polymorphic; they apply to a `Tag Text` [API change].
833+ + Do a lookahead to find the right parser to use. This takes
834+ benchmarks from 34ms to 23ms, with less allocation.
835+ + Fix bad handling of empty `src` attribute in `iframe` (#7099).
836+ If `src` is empty, we simply skip the `iframe`.
837+ If `src` is invalid or cannot be fetched, we issue a warning
838+ nd skip instead of failing with an error.
839+
840+ * JATS reader:
841+
842+ + Avoid tree normalization, which is no longer necessary given the
843+ new XML parser.
844+
845+ * LaTeX reader:
846+
847+ + Don't export `tokenize`, `untokenize` [API change]. These are internal
848+ implementation details, which were only exported for testing.
849+ They don't belong in the public API.
850+ + Improved efficiency of the parser. With these changes the reader
851+ is almost twice as fast as in the last release in our benchmarks.
852+ + Code cleanup, removing some unnecessary things.
853+ + Rewrite `withRaw` so it doesn't rely on fragile assumptions
854+ about token positions (which break when macros are expanded)
855+ (#7092). This requires the addition of `sEnableWithRaw` and
856+ `sRawTokens` in `LaTeXState`, and a new combinator `disablingWithRaw`
857+ to disable collecting of raw tokens in certain contexts.
858+ Add `parseFromToks` to Text.Pandoc.Readers.LaTeX.Parsing.
859+ Fix parsing of single character tokens so it doesn't mess
860+ up the new raw token collecting. These changes slightly increase
861+ allocations and have a small performance impact.
862+ + Handle some bibtex/biblatex-specific commands that used to be
863+ dealt with in pandoc-citeproc (#7049).
864+ + Optimize `satisfyTok`, avoiding unnecessary macro expansion steps.
865+ Benchmarks after this change show 2/3 of the run time and 2/3 of the
866+ allocation of the Feb. 10 benchmarks.
867+ + Removed `sExpanded` in state. This isn't actually needed and checking
868+ it doesn't change anything.
869+ + Improve `braced'`. Remove the parameter, have it parse the
870+ opening brace, and make it more efficient.
871+ + Factor out pieces of the LaTeX reader to make the module smaller.
872+ This reduces memory demands when compiling. Created
873+ Text.Pandoc.Readers.{LaTeX,Math,Citation,Table,Macro,Inline}.
874+ Changed Text.Pandoc.Readers.LaTeX.SIunitx to export a command map
875+ instead of individual commands.
876+ + Handle table cells containing `&` in `\verb` (#7129).
877+
878+ * Make Text.Pandoc.Readers.LaTeX.Types an unexported module [API change].
879+
880+ * Markdown reader:
881+
882+ + Improved handling of mmd link attributes in references (#7080).
883+ Previously they only worked for links that had titles.
884+ + Improved efficiency of the parser (benchmarks show a 15% speedup).
885+
886+ * OPML reader:
887+
888+ + Avoid tree normalization, which is no longer necessary with the
889+ new XML parser.
890+
891+ * ODT reader:
892+
893+ + Finer-grained errors on parse failure (#7091).
894+ + Give more information if the zip container can't be unpacked.
895+
896+ * Org reader:
897+
898+ + Support `task_lists` extension (Albert Krewinkel, #6336).
899+ + Fix bug in org-ref citation parsing (Albert Krewinkel, #7101).
900+ The org-ref syntax allows to list multiple citations separated by
901+ comma. Previously commas were accepted as part of the citation id,
902+ so all citation lists were parsed as one single citation.
903+
904+ * RST reader:
905+
906+ + Use `getTimestamp` instead of `getCurrentTime` to fetch timestamp.
907+ Setting `SOURCE_DATE_EPOCH` will allow reproducible builds.
908+ + RST reader: fix handling of header in CSV tables (#7064).
909+ The interpretation of this line is not affected by the delim option.
910+
911+ * Jira reader:
912+
913+ + Modified the Doc parser to skip leading blank lines. This fixes
914+ parsing of documents which start with multiple blank lines (Albert
915+ Krewinkel, #7095).
916+ + Prevent URLs within link aliases to be treated as autolinks (Albert
917+ Krewinkel, #6944).
918+
919+ * Text.Pandoc.Shared
920+
921+ + Remove formerly exported functions that are no longer used in the
922+ code base: `splitByIndices`, `splitStringByIndicies`, `substitute`,
923+ and `underlineSpan` (which had been deprecated in April 2020)
924+ [API change].
925+ + Export `handleTaskListItem` (Albert Krewinkel) [API change].
926+ + Change `defaultUserDataDirs` to `defaultUserDataDir` [API
927+ change]. We determine what is the default user data directory
928+ by seeing whether the XDG directory and/or legacy
929+ directory exist.
930+
931+ * BibTeX writer:
932+
933+ + BibTeX writer: use doclayout and doctemplate. This change allows
934+ bibtex/biblatex output to wrap as other formats do,
935+ depending on the settings of `--wrap` and `--columns` (#7068).
936+
937+ * CSL JSON writer:
938+
939+ + Output `[]` if no references in input, instead of raising a
940+ PandocAppError as before.
941+
942+ * Docx writer:
943+
944+ + Use `getTimestamp` instead of `getCurrentTime` for timestamp.
945+ Setting `SOURCE_DATE_EPOCH` will allow reproducible builds.
946+
947+ * EPUB writer:
948+
949+ + Use `getTimestamp` instead of `getCurrentTime` for timestamp.
950+ Setting `SOURCE_DATE_EPOCH` will allow reproducible builds (#7093).
951+ This does not suffice to fully enable reproducible in EPUB, since
952+ a unique id is still being generated for each build.
953+ + Support `belongs-to-collection` metadata (#7063) (Nick Berendsen).
954+
955+ * JATS writer:
956+
957+ + Escape special chars in reference elements (Albert Krewinkel).
958+ Prevents the generation of invalid markup if a citation element
959+ contains an ampersand or another character with a special meaning
960+ in XML.
961+
962+ * Jira writer:
963+
964+ + Use Span identifiers as anchors (Albert Krewinkel).
965+ + Use `{noformat}` instead of `{code}` for unknown languages (Albert
966+ Krewinkel). Code blocks which are not marked as a language supported
967+ by Jira are rendered as preformatted text via `{noformat}` blocks.
968+
969+ * LaTeX writer:
970+
971+ + Adjust hypertargets to beginnings of paragraphs (#7078).
972+ Use `\vadjust pre` so that the hypertarget takes you to the beginning
973+ of the paragraph rather than one line down.
974+ This makes a particular difference for links to citations using
975+ `--citeproc` and `link-citations: true`.
976+ + Change BCP47 lang tag from `jp` to `ja` (Mauro Bieg, #7047).
977+ + Use function instead of map for accent lookup (should be
978+ more efficient).
979+ + Split the module to make it easier to compile on low-memory
980+ systems: added Text.Pandoc.Writers.LaTeX.{Util,Citation,Lang}.
981+
982+ * Markdown writer:
983+
984+ + Handle math right before digit. We insert an HTML comment to
985+ avoid a `$` right before a digit, which pandoc will not recognize
986+ as a math delimiter.
987+ + Split the module to make it easier to compile on low-memory
988+ systems: added Text.Pandoc.Writers.Markdown.{Types,Inline}.
989+
990+ * ODT writer:
991+
992+ + Use `getTimestamp` instead of `getCurrentTime` for timestamp.
993+ Setting `SOURCE_DATE_EPOCH` will allow reproducible builds.
994+ + Update default ODT style (Lorenzo). Previously, the "First paragraph"
995+ style inherited from "Standard" but not from "Text body." Now
996+ it is adjusted to inherit from "Text body", to avoid some ugly
997+ spacing issues. It may be necessary to update a custom `reference.odt`
998+ in light of this change.
999+
1000+ * Org writer:
1001+
1002+ + Support `task_lists` extension (Albert Krewinkel, #6336).
1003+
1004+ * Pptx writer:
1005+
1006+ + Use `getTimestamp` instead of `getCurrentTime` for timestamp.
1007+ Setting `SOURCE_DATE_EPOCH` will allow reproducible builds.
1008+
1009+ * JATS templates: tag `author.name` as `string-name` (Albert Krewinkel).
1010+ The partitioning the components of a name into surname, given names,
1011+ etc. is not always possible or not available. Using `author.name`
1012+ allows to give the full name as a fallback to be used when
1013+ `author.surname` is not available.
1014+
1015+ * Add default templates for bibtex and biblatex, so that
1016+ the variables `header-include`, `include-before`, `include-after`
1017+ (or alternatively the command line options
1018+ `--include-in-header`, `--include-before-body`, `--include-after-body`)
1019+ may be used.
1020+
1021+ * LaTeX template:
1022+
1023+ + Update to iftex package (#7073) (Andrew Dunning)
1024+ + Wrap url colours in braces (#7121) (Loïc Grobol).
1025+
1026+ * revealjs template: Add 'center' option for vertical slide centering.
1027+ (maurerle, #7104).
1028+
1029+ * Text.Pandoc.XML: Improve efficiency of `fromEntities`.
1030+
1031+ * Text.Pandoc.MIME
1032+
1033+ + Add exported function `getCharset` [API change].
1034+
1035+ * Text.Pandoc.UTF8: change IO functions to return Text, not String
1036+ [API change]. This affects `readFile`, `getContents`, `writeFileWith`,
1037+ `writeFile`, `putStrWith`, `putStr`, `putStrLnWith`, `putStrLn`.
1038+ `hPutStrWith`, `hPutStr`, `hPutStrLnWith`, `hPutStrLn`, `hGetContents`.
1039+ This avoids the need to uselessly create a linked list of characters
1040+ when emiting output.
1041+
1042+ * Text.Pandoc.App
1043+
1044+ + Add `parseOptionsFromArgs` [API change, new exported function].
1045+ + Add fields for CSL options to `Opt` [API change]:
1046+ `optCSL`, `optbibliography`, `optCitationAbbreviations`.
1047+
1048+ * Text.Pandoc.Citeproc.BibTeX
1049+
1050+ + `Text.Pandoc.Citeproc.writeBibTeXString` now returns
1051+ `Doc Text` instead of `Text` (#7068).
1052+ + Correctly handle `pages` (= `page` in CSL) (#7067).
1053+ + Correctly handle BibLaTeX `langid` (= `language` in CSL, #7067).
1054+ + In BibTeX output, protect foreign titles since there's no language
1055+ field (#7067).
1056+ + Clean up BibTeX parsing (#7049). Previously there was a messy code
1057+ path that gave strange results in some cases, not passing through raw
1058+ tex but trying to extract a string content. This was an artefact of
1059+ trying to handle some special bibtex-specific commands in the BibTeX
1060+ reader. Now we just handle these in the LaTeX reader and simplify
1061+ parsing in the BibTeX reader. This does mean that more raw tex will
1062+ be passed through (and currently this is not sensitive to the
1063+ `raw_tex` extension; this should be fixed).
1064+
1065+ * Text.Pandoc.Citeproc.MetaValue
1066+
1067+ + Correctly parse "raw" date value in markdown references metadata.
1068+ (See jgm/citeproc#53.)
1069+
1070+ * Text.Pandoc.Citeproc
1071+
1072+ + Use https URLs for links (Salim B, #7122).
1073+
1074+ * Text.Pandoc.Class
1075+
1076+ + Add `getTimestamp` [API change]. This attempts to read the
1077+ `SOURCE_DATE_EPOCH` environment variable and parse a UTC time
1078+ from it (treating it as a unix date stamp, see
1079+ https://reproducible-builds.org/specs/source-date-epoch/). If the
1080+ variable is not set or can't be parsed as a unix date stamp, then the
1081+ function returns the current date.
1082+
1083+ * Text.Pandoc.Error
1084+
1085+ + Add `PandocUnsupportedCharsetError` constructor for
1086+ `PandocError` [API change].
1087+ + Export `renderError` [API change].
1088+ + Refactor `handleError` to use `renderError`. This allows us render
1089+ error messages without exiting.
1090+
1091+ * Text.Pandoc.Extensions
1092+
1093+ + `Ext_task_lists` is now supported by org (and turned
1094+ on by default) (Albert Krewinkel, #6336).
1095+ + Remove `Ext_fenced_code_attributes` from allowed commonmark attributes
1096+ (#7097). This attribute was listed as allowed, but it didn't actually
1097+ do anything. Use `attributes` for code attributes and more.
1098+
1099+ * Lua subsystem:
1100+
1101+ + Always load built-in Lua scripts from default data-dir (Albert
1102+ Krewinkel). The Lua modules `pandoc` and `pandoc.List` are now always
1103+ loaded from the system's default data directory. Loading from a
1104+ different directory by overriding the default path, e.g. via
1105+ `--data-dir`, is no longer supported to avoid unexpected behavior
1106+ and to address security concerns.
1107+ + Add module "pandoc.path" (Albert Krewinkel, #6001, #6565).
1108+ The module allows to work with file paths in a convenient and
1109+ platform-independent manner.
1110+ + Use strict evaluation when retrieving AST value from the stack
1111+ (Albert Krewinkel, #6674).
1112+
1113+ * Text.Pandoc.PDF
1114+
1115+ + Disable `smart` extension when building PDF via LaTeX.
1116+ This is to prevent accidental creation of ligatures like
1117+ `` ?` `` and `` !` `` (especially in languages with quotations like
1118+ German), and similar ligature issues. (See jgm/citeproc#54.)
1119+
1120+ * Text.Pandoc.CSV:
1121+
1122+ + Fix parsing of unquoted values (#7112). Previously we didn't allow
1123+ unescaped quotes in unquoted values, but they are allowed
1124+ in CSV.
1125+
1126+ * Test suite:
1127+
1128+ + Use a more robust method for testing the executable. Many
1129+ of our tests require running the pandoc executable. This
1130+ is problematic for a few different reasons. First,
1131+ cabal-install will sometimes run the test suite after
1132+ building the library but before building the executable,
1133+ which means the executable isn't in place for the tests.
1134+ One can work around that by first building, then building
1135+ and running the tests, but that's fragile. Second, we
1136+ have to find the executable. So far, we've done that using
1137+ a function `findPandoc` that attempts to locate it
1138+ relative to the test executable (which can be located
1139+ using findExecutablePath). But the logic here is delicate
1140+ and work with every combination of options. To solve both
1141+ problems, we add an `--emulate` option to the
1142+ `test-pandoc` executable. When `--emulate` occurs as the
1143+ first argument passed to `test-pandoc`, the program simply
1144+ emulates the regular pandoc executable, using the rest of
1145+ the arguments (after `--emulate`). Thus, `test-pandoc
1146+ --emulate -f markdown -t latex` is just like `pandoc -f
1147+ markdown -t latex`. Since all the work is done by library
1148+ functions, implementing this emulation just takes a couple
1149+ lines of code and should be entirely reliable. With this
1150+ change, we can test the pandoc executable by running the
1151+ test program itself (locatable using `findExecutablePath`)
1152+ with the `--emulate` option. This removes the need for the
1153+ fragile `findPandoc` step, and it means we can run our
1154+ integration tests even when we're just building the
1155+ library, not the executable. [Note: part of this change
1156+ involved simplifying some complex handling to set
1157+ environment variables for dynamic library paths. I have
1158+ tested a build with `--enable-dynamic-executable`, and it
1159+ works, but further testing may be needed.]
1160+ + Print accurate location if a test fails (Albert
1161+ Krewinkel). Ensures that tasty-hunit reports the location
1162+ of the failing test instead of the location of the helper
1163+ `test` function.
1164+
1165+ * Documentation: Update URLs and use `https` where possible (#7122,
1166+ Salim B).
1167+
1168+ * Add `doc/libraries.md`, a description of libraries that support pandoc.
1169+
1170+ * MANUAL.txt
1171+
1172+ + MANUAL: block-level formatting is not allowed in line blocks (#7107).
1173+ + Clarify `tex_math_dollars` extension. Note that no blank lines
1174+ are allowed between the delimiters in display math.
1175+ + Add MANUAL section on reproducible builds.
1176+ + Document no template fallback for absolute path (#7077, Nixon
1177+ Enraght-Moony.)
1178+ + Improve docs for cite-method.
1179+ + Update README and man page.
1180+
1181+ * Makefile: in `make bench`, create CSV files for comparison and compare
1182+ against previous benchmark run. Add timestamp to CSV filenames.
1183+
1184+ * cabal.project: don't explicitly set -trypandoc.
1185+ If we do, this can't be overridden on the cabal command line.
1186+
1187+ * doc/lua-filters.md: improve documentation for
1188+ `pandoc.mediabag.insert`, `pandoc.mediabag.fetch`,
1189+ `directory`, `normalize` (Albert Krewinkel).
1190+
1191+ * Allow base64-bytestring-1.2.* (Dmitrii Kovanikov)
1192+
1193+ * Require jira-wiki-markup 1.3.3 (Albert Krewinkel)
1194+
1195+ * Require citeproc 0.3.0.8, which correctly titlecases when titles
1196+ contain non-ASCII characters.
1197+
1198+ * Use skylighting 0.10.4. This version of skylighting uses xml-conduit
1199+ rather than hxt. This speeds up parsing of XML syntax definitions
1200+ fourfold, and removes four packages from pandoc's dependency graph:
1201+ hxt-charproperties, hxt-unicode, hxt-regex-xmlschema, hxt.
1202+
1203+ * Add script `tools/parseTimings.pl` to help pin down which
1204+ modules take the most time and memory to compile.
1205+
1206+ * Avoid unnecessary use of NoImplicitPrelude pragma (#7089) (Albert
1207+ Krewinkel)
1208+
1209+ * Benchmarks
1210+
1211+ + Use the lighter-weight tasty-bench instead of criterion.
1212+ + Run writer benchmarks for binary formats too.
1213+ + Alphabetize benchmarks.
1214+ + Don't run benchmarks for bibliography formats
1215+ (yet; we need a special input for them).
1216+ + Show allocation data
1217+ + Clean up benchmark code.
1218+ + Allow specifying patterns using `-p blah'.
1219+
1220+ * trypandoc: add 2 second timeout.
1221+
1222+ * Use `-split-sections` in creating linux release binary.
1223+ This reduces executable size significantly (by about 30%).
1224+
1225+ * Remove `weigh-pandoc`. It's not really useful any more, now that our
1226+ regular benchmarks include data on allocation.
1227+
1228+ * Improve linux package build process and add script to
1229+ automate building an arm64 binary package.
1230+
1231+
1232+## pandoc 2.11.4 (2021-01-22)
1233+
1234+ * Add `biblatex`, `bibtex` as output formats (closes #7040).
1235+
1236+ * Recognize more extensions as markdown by default (#7034):
1237+ `mkdn`, `mkd`, `mdwn`, `mdown`, `Rmd`.
1238+
1239+ * Implement defaults file inheritance (#6924, David Martschenko).
1240+ Allow defaults files to inherit options from other defaults files by
1241+ specifying them with the following syntax:
1242+ `defaults: [list of defaults files or single defaults file]`.
1243+
1244+ * Fix infinite HTTP requests when writing epubs from URL source (#7013).
1245+ Due to a bug in code added to avoid overwriting the cover image
1246+ if it had the form `fileX.YYY`, pandoc made an endless sequence
1247+ of HTTP requests when writing epub with input from a URL.
1248+
1249+ * Org reader:
1250+
1251+ + Allow multiple pipe chars in todo sequences (Albert Krewinkel, #7014).
1252+ Additional pipe chars, used to separate "action" state from "no further
1253+ action" states, are ignored. E.g., for the following sequence, both
1254+ `DONE` and `FINISHED` are states with no further action required:
1255+ `#+TODO: UNFINISHED | DONE | FINISHED`.
1256+ + Restructure output of captioned code blocks (Albert Krewinkel, #6977).
1257+ The Div wrapper of code blocks with captions now has the class
1258+ "captioned-content". The caption itself is added as a Plain block
1259+ inside a Div of class "caption". This makes it easier to write filters
1260+ which match on captioned code blocks. Existing filters will need to be
1261+ updated.
1262+ + Mark verbatim code with class `verbatim` (Dimitri Sabadie, #6998).
1263+
1264+ * LaTeX reader:
1265+
1266+ + Handle `filecontents` environment (#7003).
1267+ + Put contents of unknown environments in a Div when `raw_tex` is not
1268+ enabled (#6997). (When `raw_tex` is enabled, the whole environment is
1269+ parsed as a raw block.) The class name is the name of the environment.
1270+ Previously, we just included the contents without the surrounding Div,
1271+ but having a record of the environment's boundaries and name can be
1272+ useful.
1273+
1274+ * Mediawiki reader:
1275+
1276+ + Allow space around storng/emph delimiters (#6993).
1277+
1278+ * New module Text.Pandoc.Writers.BibTeX, exporting
1279+ writeBibTeX and writeBibLaTeX. [API change]
1280+
1281+ * LaTeX writer:
1282+
1283+ + Revert table line height increase in 2.11.3 (#6996).
1284+ In 2.11.3 we started adding `\addlinespace`, which produced less dense
1285+ tables. This wasn't an intentional change; I misunderstood a comment in
1286+ the discussion leading up to the change. This commit restores the earlier
1287+ default table appearance. Note that if you want a less dense table, you
1288+ can use something like `\def\arraystretch{1.5}` in your header.
1289+
1290+ * EPUB writer:
1291+
1292+ + Adjust internal links to identifiers defined in raw HTML sections
1293+ after splitting into chapters (#7000).
1294+ + Recognize `Format "html4"`, `Format "html5"` as raw HTML.
1295+ + Adjust internal links to images, links, and tables after splitting into
1296+ chapters. Previously we only did this for Div and Span and Header
1297+ elements (see #7000).
1298+
1299+ * Ms writer:
1300+
1301+ + Don't justify text inside table cells.
1302+
1303+ * JATS writer:
1304+
1305+ + Use `<element-citation>` if `element_citations`
1306+ extension is enabled (Albert Krewinkel).
1307+ + Fix citations (Albert Krewinkel, #7018). By default
1308+ we use formatted citations.
1309+ + Ensure that `<disp-quote>` is always wrapped in `<p>` (#7041).
1310+
1311+ * Markdown writer:
1312+
1313+ + Cleaned up raw formats. We now react appropriately
1314+ to `gfm`, `commonmark`, and `commonmark_x` as raw formats.
1315+
1316+ * RST writer:
1317+
1318+ + Fix bug with dropped content from inside spans with a class in
1319+ some cases (#7039).
1320+
1321+ * Docx writer:
1322+
1323+ + Handle table header using styles (#7008). Instead of hard-coding
1324+ the border and header cell vertical alignment, we now let this
1325+ be determined by the Table style, making use of Word's
1326+ "conditional formatting" for the table's first row. For
1327+ headerless tables, we use the tblLook element to tell Word
1328+ not to apply conditional first-row formatting.
1329+
1330+ * Commonmark writer:
1331+
1332+ + Implement start number on ordered lists (#7009). Previously they always
1333+ started at 1, but according to the spec the start number is respected.
1334+
1335+ * HTML writer:
1336+
1337+ + Fix implicit_figure at end of footnotes (#7006).
1338+
1339+ * ConTeXt template: Remove `\setupthinrules` from default template.
1340+ The width parameter this used is not actually supported,
1341+ and the command didn't do anything.
1342+
1343+ * Text.Pandoc.Extensions:
1344+
1345+ + Add `Ext_element_citations` constructor (Albert Krewinkel).
1346+
1347+ * Text.Pandoc.Citeproc.BibTeX: New unexported function
1348+ `writeBibtexString`.
1349+
1350+ * Text.Pandoc.Citeproc:
1351+
1352+ + Use finer grained imports (Albert Krewinkel).
1353+ + Factor out and export `getStyle` [API change].
1354+ + Export `getReferences` [API change, #7106].
1355+ + Factor out getLang.
1356+
1357+ * Text.Pandoc.Parsing: modify `gridTableWith'` for headerless tables.
1358+ If the table lacks a header, the header row should be an empty
1359+ list. Previously we got a list of empty cells, which caused
1360+ an empty header to be emitted instead of no header. In LaTeX/PDF
1361+ output that meant we got a double top line with space between.
1362+
1363+ * ImageSize: use `viewBox` for SVG if no length, width attributes (#7045).
1364+ This change allows pandoc to extract size information from more SVGs.
1365+
1366+ * Add simple default.nix.
1367+
1368+ * Use commonmark 0.1.1.3.
1369+
1370+ * Use citeproc 0.3.0.5.
1371+
1372+ * Update default CSL to use latest chicago-author-date.csl.
1373+
1374+ * CONTRIBUTING.md: add note on GNU xargs.
1375+
1376+ * MANUAL.txt:
1377+
1378+ + Update description of `-L`/`--lua-filter`.
1379+ + Document use of citations in note styles (#6828).
1380+
1381+## pandoc 2.11.3.2 (2020-12-29)
1382+
1383+ * HTML reader: use renderTags' from Text.Pandoc.Shared (Albert Krewinkel).
1384+ A side effect of this change is that empty `<col>` elements are written
1385+ as self-closing tags in raw HTML blocks.
1386+
1387+ * Asciidoc writer: Add support for writing nested tables (#6972, timo-a).
1388+ Asciidoc supports one level of nesting. If deeper tables are to be
1389+ written, they are omitted and a warning is issued.
1390+
1391+ * Docx writer: fix nested tables with captions (#6983).
1392+ Previously we got unreadable content, because docx seems
1393+ to want a `<w:p>` element (even an empty one) at the end of
1394+ every table cell.
1395+
1396+ * Powerpoint writer: allow arbitrary OOXML in raw inline elements
1397+ (Albert Krewinkel). The raw text is now included verbatim in the
1398+ output. Previously is was parsed into XML elements, which prevented
1399+ the inclusion of partial XML snippets.
1400+
1401+ * LaTeX writer: support colspans and rowspans in tables (#6950,
1402+ Albert Krewinkel). Note that the multirow package is needed for
1403+ rowspans. It is included in the latex template under a variable,
1404+ so that it won't be used unless needed for a table.
1405+
1406+ * HTML writer: don't include p tags in CSL bibliography entries
1407+ (#6966). Fixes a regression in 2.11.3.
1408+
1409+ * Add `meta-description` variable to HTML templates (#6982). This
1410+ is populated by the writer by stringifying the `description`
1411+ field of metadata (Jerry Sky). The `description` meta tag will
1412+ make the generated HTML documents more complete and SEO-friendly.
1413+
1414+ * Citeproc: fix handling of empty URL variables (`DOI`, etc.).
1415+ The `linkifyVariables` function was changing these to links
1416+ which then got treated as non-empty by citeproc, leading
1417+ to wrong results (e.g. ignoring nonempty URL when empty DOI is present).
1418+ See jgm/citeproc#41.
1419+
1420+ * Use citeproc 0.3.0.3. Fixes an issue in author-only citations when
1421+ both an author and translator are present, and an issue with
1422+ citation group delimiters.
1423+
1424+ * Require texmath 0.12.1. This improves siunitx support in math,
1425+ fixes bugs with `\*mod` family operators and arrays, and avoids
1426+ italicizing symbols and operator names in docx output.
1427+
1428+ * Ensure that the perl interpreter used for filters with `.pl`
1429+ extension (wuffi).
1430+
1431+ * MANUAL: note that textarea content is never parsed as Markdown
1432+ (Albert Krewinkel).
1433+
1434+
31435## pandoc 2.11.3.1 (2020-12-18)
41436
51437 * Added some missing files to extra-source-files and data
@@ -388,8 +1820,11 @@
3881820 translated to `C:\/foo/bar`, which caused problems.
3891821 With this fix, the backslashes are removed.
3901822
391− * Text.Pandoc.Logging: Add constructor `ATXHeadingInLHS` constructor
392− to `LogMessage` [API change].
1823+ * Text.Pandoc.Logging:
1824+
1825+ + Add constructor `ATXHeadingInLHS` to `LogMessage` [API change].
1826+ + Add constructor `EnvironmentVariableUndefined` to
1827+ `LogMessage` [API change].
3931828
3941829 * Fix error that is given when people specify `doc` output (#6834,
3951830 gison93).
data/default.csl+12−2
@@ -27,6 +27,9 @@
2727 <email>matthew.g.roth@yale.edu</email>
2828 <uri> https://orcid.org/0000-0001-7902-6331</uri>
2929 </contributor>
30+ <contributor>
31+ <name>Brenton M. Wiernik</name>
32+ </contributor>
3033 <category citation-format="author-date"/>
3134 <category field="generic-base"/>
3235 <summary>The author-date variant of the Chicago style</summary>
@@ -509,8 +512,15 @@
509512 </choose>
510513 </macro>
511514 <macro name="event">
512− <group>
513− <text term="presented at" suffix=" "/>
515+ <group delimiter=" ">
516+ <choose>
517+ <if variable="genre">
518+ <text term="presented at"/>
519+ </if>
520+ <else>
521+ <text term="presented at" text-case="capitalize-first"/>
522+ </else>
523+ </choose>
514524 <text variable="event"/>
515525 </group>
516526 </macro>
data/docx/word/settings.xml+6−16
@@ -1,6 +1,6 @@
11<?xml version="1.0" encoding="UTF-8"?>
22<w:settings xmlns:o="urn:schemas-microsoft-com:office:office" xmlns:r="http://schemas.openxmlformats.org/officeDocument/2006/relationships" xmlns:m="http://schemas.openxmlformats.org/officeDocument/2006/math" xmlns:v="urn:schemas-microsoft-com:vml" xmlns:w10="urn:schemas-microsoft-com:office:word" xmlns:w="http://schemas.openxmlformats.org/wordprocessingml/2006/main" xmlns:sl="http://schemas.openxmlformats.org/schemaLibrary/2006/main">
3− <w:zoom w:percent="90" />
3+ <w:zoom w:percent="100" />
44 <w:embedSystemFonts />
55 <w:proofState w:spelling="clean" w:grammar="clean" />
66 <w:stylePaneFormatFilter w:val="0004" />
@@ -17,31 +17,21 @@
1717 <w:characterSpacingControl w:val="doNotCompress" />
1818 <w:savePreviewPicture />
1919 <w:rsids>
20− <w:rsidRoot w:val="00590D07" />
21− <w:rsid w:val="00011C8B" />
22− <w:rsid w:val="004E29B3" />
23− <w:rsid w:val="00590D07" />
24− <w:rsid w:val="00784D58" />
25− <w:rsid w:val="008D6863" />
26− <w:rsid w:val="00B86B75" />
27− <w:rsid w:val="00BC48D5" />
28− <w:rsid w:val="00C36279" />
29− <w:rsid w:val="00E315A3" />
3020 </w:rsids>
3121 <m:mathPr>
32− <m:mathFont m:val="Lucida Grande" />
22+ <m:mathFont m:val="Cambria Math" />
3323 <m:brkBin m:val="before" />
3424 <m:brkBinSub m:val="--" />
35− <m:smallFrac m:val="false" />
36− <m:dispDef m:val="false" />
25+ <m:smallFrac m:val="0" />
26+ <m:dispDef />
3727 <m:lMargin m:val="0" />
3828 <m:rMargin m:val="0" />
3929 <m:wrapRight />
4030 <m:intLim m:val="subSup" />
41− <m:naryLim m:val="subSup" />
31+ <m:naryLim m:val="undOvr" />
4232 </m:mathPr>
4333 <w:themeFontLang w:val="en-US" />
4434 <w:clrSchemeMapping w:bg1="light1" w:t1="dark1" w:bg2="light2" w:t2="dark2" w:accent1="accent1" w:accent2="accent2" w:accent3="accent3" w:accent4="accent4" w:accent5="accent5" w:accent6="accent6" w:hyperlink="hyperlink" w:followedHyperlink="followedHyperlink" />
4535 <w:decimalSymbol w:val="." />
4636 <w:listSeparator w:val="," />
47−</w:settings>
37+</w:settings>
data/docx/word/styles.xml+15
@@ -350,6 +350,21 @@
350350 <w:right w:w="108" w:type="dxa" />
351351 </w:tblCellMar>
352352 </w:tblPr>
353+ <w:tblStylePr w:type="firstRow">
354+ <w:tblPr>
355+ <w:jc w:val="left"/>
356+ <w:tblInd w:w="0" w:type="dxa"/>
357+ </w:tblPr>
358+ <w:trPr>
359+ <w:jc w:val="left"/>
360+ </w:trPr>
361+ <w:tcPr>
362+ <w:vAlign w:val="bottom"/>
363+ <w:tcBorders>
364+ <w:bottom w:val="single"/>
365+ </w:tcBorders>
366+ </w:tcPr>
367+ </w:tblStylePr>
353368 </w:style>
354369 <w:style w:type="paragraph" w:customStyle="1" w:styleId="DefinitionTerm">
355370 <w:name w:val="Definition Term" />
data/jats.csldeleted−203
@@ -1,203 +0,0 @@
1−<?xml version="1.0" encoding="utf-8"?>
2−<style xmlns="http://purl.org/net/xbiblio/csl" class="in-text" version="1.0" default-locale="en-US">
3− <info>
4− <title>Journal Article Tag Suite</title>
5− <title-short>JATS</title-short>
6− <id>http://www.zotero.org/styles/journal-article-tag-suite</id>
7− <link href="https://github.com/MartinPaulEve/JATS-CSL/blob/master/jats.csl" rel="self"/>
8− <link rel="documentation" href="http://jats.nlm.nih.gov/archiving/tag-library/1.0/index.html"/>
9− <author>
10− <name>Martin Paul Eve</name>
11− <email>martin@martineve.com</email>
12− </author>
13− <category citation-format="numeric"/>
14− <category field="medicine"/>
15− <category field="biology"/>
16− <summary>Use this style to generate bibliographic data in Journal Article Tagging Suite (JATS) 1.0 XML format</summary>
17− <updated>2014-06-21T17:41:26+00:00</updated>
18− <rights license="http://creativecommons.org/licenses/by-sa/3.0/">This work is licensed under a Creative Commons Attribution-ShareAlike 3.0 License. Originally by Martin Fenner.</rights>
19− </info>
20− <locale xml:lang="en">
21− <terms>
22− <term name="et-al">{{jats}}&lt;etal/&gt;{{/jats}}</term>
23− </terms>
24− </locale>
25− <macro name="citation-number">
26− <text variable="citation-number" prefix="{{jats}}id=&quot;ref-{{/jats}}" suffix="{{jats}}&quot;&gt;{{/jats}}"/>
27− </macro>
28− <macro name="author">
29− <names variable="author" delimiter=" ">
30− <name prefix="{{jats}}&lt;name&gt;{{/jats}}" suffix="{{jats}}&lt;/name&gt;{{/jats}}" name-as-sort-order="all" sort-separator=" ">
31− <name-part name="family" text-case="capitalize-first" prefix="{{jats}}&lt;surname&gt;{{/jats}}" suffix="{{jats}}&lt;/surname&gt;{{/jats}}"/>
32− <name-part name="given" text-case="capitalize-first" prefix="{{jats}}&lt;given-names&gt;{{/jats}}" suffix="{{jats}}&lt;/given-names&gt;{{/jats}}"/>
33− </name>
34− <substitute>
35− <names variable="editor"/>
36− </substitute>
37− </names>
38− </macro>
39−
40− <macro name="editor" delimiter=" ">
41− <names variable="editor" prefix="{{jats}}&lt;person-group person-group-type=&quot;editor&quot;&gt;{{/jats}}" suffix="{{jats}}&lt;/person-group&gt;{{/jats}}">
42− <name prefix="{{jats}}&lt;name&gt;{{/jats}}" suffix="{{jats}}&lt;/name&gt;{{/jats}}" name-as-sort-order="all" sort-separator=" ">
43− <name-part name="family" text-case="capitalize-first" prefix="{{jats}}&lt;surname&gt;{{/jats}}" suffix="{{jats}}&lt;/surname&gt;{{/jats}}"/>
44− <name-part name="given" text-case="capitalize-first" prefix="{{jats}}&lt;given-names&gt;{{/jats}}" suffix="{{jats}}&lt;/given-names&gt;{{/jats}}"/>
45− </name>
46− <substitute>
47− <names variable="editor"/>
48− </substitute>
49− </names>
50− </macro>
51−
52− <macro name="editor">
53− <group delimiter=": ">
54− <names variable="editor">
55− <name prefix="{{jats}}&lt;name&gt;{{/jats}}" suffix="{{jats}}&lt;/name&gt;{{/jats}}" name-as-sort-order="all" sort-separator="">
56− <name-part name="family" text-case="capitalize-first" prefix="{{jats}}&lt;surname&gt;{{/jats}}" suffix="{{jats}}&lt;/surname&gt;{{/jats}}"/>
57− <name-part name="given" text-case="capitalize-first" prefix="{{jats}}&lt;given-names&gt;{{/jats}}" suffix="{{jats}}&lt;given-names&gt;{{/jats}}"/>
58− </name>
59− </names>
60− </group>
61− </macro>
62− <macro name="title">
63− <choose>
64− <if type="book" match="any">
65− <group prefix="{{jats}}&lt;source&gt;{{/jats}}" suffix="{{jats}}&lt;/source&gt;{{/jats}}">
66− <text variable="title"/>
67− </group>
68− </if>
69− <else>
70− <group prefix="{{jats}}&lt;article-title&gt;{{/jats}}" suffix="{{jats}}&lt;/article-title&gt;{{/jats}}">
71− <text variable="title"/>
72− </group>
73− </else>
74− </choose>
75− </macro>
76− <macro name="container-title">
77− <text variable="container-title" form="short" prefix="{{jats}}&lt;source&gt;{{/jats}}" suffix="{{jats}}&lt;/source&gt;{{/jats}}"/>
78− </macro>
79− <macro name="publisher">
80− <text variable="publisher" prefix="{{jats}}&lt;publisher-name&gt;{{/jats}}" suffix="{{jats}}&lt;/publisher-name&gt;{{/jats}}"/>
81− <text variable="publisher-place" prefix="{{jats}}&lt;publisher-loc&gt;{{/jats}}" suffix="{{jats}}&lt;/publisher-loc&gt;{{/jats}}"/>
82− </macro>
83− <macro name="link">
84− <choose>
85− <if match="any" variable="DOI">
86− <group prefix="{{jats}}&lt;pub-id pub-id-type=&quot;doi&quot;&gt;{{/jats}}" suffix="{{jats}}&lt;/pub-id&gt;{{/jats}}">
87− <text variable="DOI"/>
88− </group>
89− </if>
90− </choose>
91− <choose>
92− <if match="any" variable="PMID">
93− <group prefix="{{jats}}&lt;pub-id pub-id-type=&quot;pmid&quot;&gt;{{/jats}}" suffix="{{jats}}&lt;/pub-id&gt;{{/jats}}">
94− <text variable="PMID"/>
95− </group>
96− </if>
97− </choose>
98− <choose>
99− <if variable="URL" match="any">
100− <text variable="URL" />
101− </if>
102− </choose>
103− </macro>
104− <macro name="date">
105− <choose>
106− <if type="article-journal article-magazine article-newspaper report patent book" match="any">
107− <group prefix="{{jats}}&lt;date&gt;{{/jats}}" suffix="{{jats}}&lt;/date&gt;{{/jats}}">
108− <date variable="issued">
109− <date-part name="day" form="numeric-leading-zeros" prefix="{{jats}}&lt;day&gt;{{/jats}}" suffix="{{jats}}&lt;/day&gt;{{/jats}}"/>
110− <date-part name="month" form="numeric-leading-zeros" prefix="{{jats}}&lt;month&gt;{{/jats}}" suffix="{{jats}}&lt;/month&gt;{{/jats}}"/>
111− <date-part name="year" prefix="{{jats}}&lt;year&gt;{{/jats}}" suffix="{{jats}}&lt;/year&gt;{{/jats}}"/>
112− </date>
113− </group>
114− </if>
115− <else>
116− <group prefix="{{jats}}&lt;date-in-citation content-type=&quot;access-date&quot;{{/jats}}" suffix="{{jats}}&lt;/date-in-citation&gt;{{/jats}}">
117− <date variable="accessed" prefix="{{jats}} iso-8601-date=&quot;{{/jats}}" suffix="{{jats}}&quot;&gt;{{/jats}}">
118− <date-part name="year"/>
119− <date-part name="month" form="numeric-leading-zeros" prefix="{{jats}}-{{/jats}}"/>
120− <date-part name="day" form="numeric-leading-zeros" prefix="{{jats}}-{{/jats}}"/>
121− </date>
122− <date variable="accessed">
123− <date-part name="day" prefix="{{jats}}&lt;day&gt;{{/jats}}" suffix="{{jats}}&lt;/day&gt;{{/jats}}"/>
124− <date-part name="month" form="numeric-leading-zeros" prefix="{{jats}}&lt;month&gt;{{/jats}}" suffix="{{jats}}&lt;/month&gt;{{/jats}}"/>
125− <date-part name="year" prefix="{{jats}}&lt;year&gt;{{/jats}}" suffix="{{jats}}&lt;/year&gt;{{/jats}}"/>
126− </date>
127− </group>
128− </else>
129− </choose>
130− </macro>
131− <macro name="location">
132− <choose>
133− <if type="article-journal article-magazine" match="any">
134− <text variable="volume" prefix="{{jats}}&lt;volume&gt;{{/jats}}" suffix="{{jats}}&lt;/volume&gt;{{/jats}}"/>
135− <text variable="issue" prefix="{{jats}}&lt;issue&gt;{{/jats}}" suffix="{{jats}}&lt;/issue&gt;{{/jats}}"/>
136− </if>
137− </choose>
138− <choose>
139− <if type="article-journal article-magazine article-newspaper chapter" match="any">
140− <text variable="page-first" prefix="{{jats}}&lt;fpage&gt;{{/jats}}" suffix="{{jats}}&lt;/fpage&gt;{{/jats}}"/>
141− </if>
142− </choose>
143− </macro>
144− <macro name="publication-type">
145− <group prefix="{{jats}} publication-type=&quot;{{/jats}}" suffix="{{jats}}&quot;&gt;{{/jats}}">
146− <choose>
147− <if type="article-journal article-magazine article-newspaper" match="any">
148− <text value="journal"/>
149− </if>
150− <else-if type="book" match="any">
151− <text value="book"/>
152− </else-if>
153− <else-if type="chapter" match="any">
154− <text value="bookchapter"/>
155− </else-if>
156− <else-if type="dataset" match="any">
157− <text value="dataset"/>
158− </else-if>
159− <else-if type="patent" match="any">
160− <text value="patent"/>
161− </else-if>
162− <else-if type="report" match="any">
163− <text value="report"/>
164− </else-if>
165− <else-if type="review" match="any">
166− <text value="review"/>
167− </else-if>
168− <else>
169− <text value="standard"/>
170− </else>
171− </choose>
172− </group>
173− </macro>
174− <citation collapse="citation-number">
175− <sort>
176− <key variable="citation-number"/>
177− </sort>
178− <layout delimiter=",">
179− <group prefix="{{jats}}&lt;xref ref-type=&quot;bibr&quot; rid=&quot;{{/jats}}" suffix="{{jats}}&lt;/xref&gt;{{/jats}}">
180− <text variable="citation-number" prefix="{{jats}}ref-{{/jats}}" suffix="{{jats}}&quot;&gt;{{/jats}}"/>
181− <text variable="citation-number"/>
182− </group>
183− </layout>
184− </citation>
185− <bibliography sort-separator="">
186− <layout>
187− <group prefix="{{jats}}&lt;ref {{/jats}}" suffix="{{jats}}&lt;/ref&gt;{{/jats}}">
188− <text macro="citation-number"/>
189− <group prefix="{{jats}}&lt;element-citation{{/jats}}" suffix="{{jats}}&lt;/element-citation&gt;{{/jats}}">
190− <text macro="publication-type"/>
191− <text macro="author" prefix="{{jats}}&lt;person-group person-group-type=&quot;author&quot;&gt;{{/jats}}" suffix="{{jats}}&lt;/person-group&gt;{{/jats}}"/>
192− <text macro="title" />
193− <text macro="container-title"/>
194− <text macro="editor"/>
195− <text macro="publisher"/>
196− <text macro="date"/>
197− <text macro="location"/>
198− <text macro="link"/>
199− </group>
200− </group>
201− </layout>
202− </bibliography>
203−</style>
data/odt/styles.xml+1−1
@@ -350,7 +350,7 @@ xmlns:css3t="http://www.w3.org/TR/css3-text/" office:version="1.2">
350350 </style:style>
351351 <style:style style:name="First_20_paragraph"
352352 style:display-name="First paragraph" style:family="paragraph"
353− style:parent-style-name="Standard"
353+ style:parent-style-name="Text_20_body"
354354 style:next-style-name="Text_20_body" style:class="text" />
355355 <style:style style:name="Numbering_20_Symbols"
356356 style:display-name="Numbering Symbols" style:family="text" />
data/templates/article.jats_publishing+6−4
@@ -30,10 +30,10 @@ $if(journal.abbrev-title)$
3030$endif$
3131</journal-title-group>
3232$if(journal.pissn)$
33−<issn pub-type="ppub">$journal.pissn$</issn>
33+<issn publication-format="print">$journal.pissn$</issn>
3434$endif$
3535$if(journal.eissn)$
36−<issn pub-type="epub">$journal.eissn$</issn>
36+<issn publication-format="electronic">$journal.eissn$</issn>
3737$endif$
3838$-- At least one issn element is required; use empty issn as fallback
3939$if(journal.pissn)$
@@ -86,7 +86,7 @@ $endif$
8686$if(author)$
8787<contrib-group>
8888$for(author)$
89−<contrib contrib-type="author">
89+<contrib contrib-type="author"$if(author.equal-contrib)$ equal-contrib="true"$endif$>
9090$if(author.orcid)$
9191<contrib-id contrib-id-type="orcid">$author.orcid$</contrib-id>
9292$endif$
@@ -95,6 +95,8 @@ $if(author.surname)$
9595<surname>$author.surname$</surname>
9696<given-names>$author.given-names$</given-names>
9797</name>
98+$elseif(author.name)$
99+<string-name>$author.name$</string-name>
98100$else$
99101<string-name>$author$</string-name>
100102$endif$
@@ -128,7 +130,7 @@ $endif$
128130</author-notes>
129131$endif$
130132$if(date)$
131−<pub-date pub-type="epub"$if(date.iso-8601)$ iso-8601-date="$date.iso-8601$"$endif$>
133+<pub-date date-type="$if(date.type)$$date.type$$else$pub$endif$" publication-format="electronic"$if(date.iso-8601)$ iso-8601-date="$date.iso-8601$"$endif$>
132134$if(date.day)$
133135<day>$date.day$</day>
134136$endif$
data/templates/default.biblatexnew file+10
@@ -0,0 +1,10 @@
1+$for(header-includes)$
2+$header-includes$
3+$endfor$
4+$for(include-before)$
5+$include-before$
6+$endfor$
7+$body$
8+$for(include-after)$
9+$include-after$
10+$endfor$
data/templates/default.bibtexnew file+10
@@ -0,0 +1,10 @@
1+$for(header-includes)$
2+$header-includes$
3+$endfor$
4+$for(include-before)$
5+$include-before$
6+$endfor$
7+$body$
8+$for(include-after)$
9+$include-after$
10+$endfor$
data/templates/default.context+7−5
@@ -51,6 +51,7 @@ $endif$
5151
5252% use microtypography
5353\definefontfeature[default][default][script=latn, protrusion=quality, expansion=quality, itlc=yes, textitalics=yes, onum=yes, pnum=yes]
54+\definefontfeature[default:tnum][default][tnum=yes, pnum=no]
5455\definefontfeature[smallcaps][script=latn, protrusion=quality, expansion=quality, smcp=yes, onum=yes, pnum=yes]
5556\setupalign[hz,hanging]
5657\setupitaliccorrection[global, always]
@@ -97,11 +98,12 @@ $endif$
9798\setupitemize[autointro] % prevent orphan list intro
9899\setupitemize[indentnext=no]
99100
101+\defineitemgroup[enumerate]
102+\setupenumerate[each][fit][itemalign=left,distance=.5em,style={\feature[+][default:tnum]}]
103+
100104\setupfloat[figure][default={here,nonumber}]
101105\setupfloat[table][default={here,nonumber}]
102106
103−\setupthinrules[width=15em] % width of horizontal rules
104−
105107\setupxtable[frame=off]
106108\setupxtable[head][topframe=on,bottomframe=on]
107109\setupxtable[body][]
@@ -162,12 +164,12 @@ $endfor$
162164$if(toc)$
163165\completecontent
164166$endif$
165−$if(lot)$
166−\completelistoftables
167−$endif$
168167$if(lof)$
169168\completelistoffigures
170169$endif$
170+$if(lot)$
171+\completelistoftables
172+$endif$
171173
172174$body$
173175
data/templates/default.dzslides+1−1
@@ -171,7 +171,7 @@ $if(subtitle)$
171171 <h1 class="subtitle">$subtitle$</h1>
172172$endif$
173173 <footer>
174− $if(author)$<span class="author">$for(author)$$author$$sep$, $endfor$</span>$endif$ · $if(date)$<span class="date">$date$</span>$endif$
174+ $if(author)$<span class="author">$for(author)$$author$$sep$, $endfor$</span> · $endif$$if(institute)$<span class="institute">$for(institute)$$institute$$sep$, $endfor$</span> · $endif$$if(date)$<span class="date">$date$</span>$endif$
175175 </footer>
176176</section>
177177$endif$
data/templates/default.jats_articleauthoring+3−1
@@ -19,7 +19,7 @@ $endif$
1919$if(author)$
2020<contrib-group>
2121$for(author)$
22−<contrib contrib-type="author">
22+<contrib contrib-type="author"$if(author.equal-contrib)$ equal-contrib="true"$endif$>
2323$if(author.orcid)$
2424<contrib-id contrib-id-type="orcid">$author.orcid$</contrib-id>
2525$endif$
@@ -28,6 +28,8 @@ $if(author.surname)$
2828<surname>$author.surname$</surname>
2929<given-names>$author.given-names$</given-names>
3030</name>
31+$elseif(author.name)$
32+<string-name>$author.name$</string-name>
3133$else$
3234<string-name>$author$</string-name>
3335$endif$
data/templates/default.latex+65−40
@@ -2,7 +2,7 @@
22\PassOptionsToPackage{unicode$for(hyperrefoptions)$,$hyperrefoptions$$endfor$}{hyperref}
33\PassOptionsToPackage{hyphens}{url}
44$if(colorlinks)$
5−\PassOptionsToPackage{dvipsnames,svgnames*,x11names*}{xcolor}
5+\PassOptionsToPackage{dvipsnames,svgnames,x11names}{xcolor}
66$endif$
77$if(dir)$
88$if(latex-dir-rtl)$
@@ -98,14 +98,14 @@ $endif$
9898$if(linestretch)$
9999\usepackage{setspace}
100100$endif$
101−\usepackage{ifxetex,ifluatex}
102−\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
101+\usepackage{iftex}
102+\ifPDFTeX
103103 \usepackage[$if(fontenc)$$fontenc$$else$T1$endif$]{fontenc}
104104 \usepackage[utf8]{inputenc}
105105 \usepackage{textcomp} % provide euro and other symbols
106106\else % if luatex or xetex
107107$if(mathspec)$
108− \ifxetex
108+ \ifXeTeX
109109 \usepackage{mathspec}
110110 \else
111111 \usepackage{unicode-math}
@@ -129,7 +129,7 @@ $for(fontfamilies)$
129129$endfor$
130130$if(mathfont)$
131131$if(mathspec)$
132− \ifxetex
132+ \ifXeTeX
133133 \setmathfont(Digits,Latin,Greek)[$for(mathfontoptions)$$mathfontoptions$$sep$,$endfor$]{$mathfont$}
134134 \else
135135 \setmathfont[$for(mathfontoptions)$$mathfontoptions$$sep$,$endfor$]{$mathfont$}
@@ -139,23 +139,42 @@ $else$
139139$endif$
140140$endif$
141141$if(CJKmainfont)$
142− \ifxetex
142+ \ifXeTeX
143143 \usepackage{xeCJK}
144144 \setCJKmainfont[$for(CJKoptions)$$CJKoptions$$sep$,$endfor$]{$CJKmainfont$}
145145 \fi
146146$endif$
147147$if(luatexjapresetoptions)$
148− \ifluatex
148+ \ifLuaTeX
149149 \usepackage[$for(luatexjapresetoptions)$$luatexjapresetoptions$$sep$,$endfor$]{luatexja-preset}
150150 \fi
151151$endif$
152152$if(CJKmainfont)$
153− \ifluatex
153+ \ifLuaTeX
154154 \usepackage[$for(luatexjafontspecoptions)$$luatexjafontspecoptions$$sep$,$endfor$]{luatexja-fontspec}
155155 \setmainjfont[$for(CJKoptions)$$CJKoptions$$sep$,$endfor$]{$CJKmainfont$}
156156 \fi
157157$endif$
158158\fi
159+$if(zero-width-non-joiner)$
160+%% Support for zero-width non-joiner characters.
161+\makeatletter
162+\def\zerowidthnonjoiner{%
163+ % Prevent ligatures and adjust kerning, but still support hyphenating.
164+ \texorpdfstring{%
165+ \textormath{\nobreak\discretionary{-}{}{\kern.03em}%
166+ \ifvmode\else\nobreak\hskip\z@skip\fi}{}%
167+ }{}%
168+}
169+\makeatother
170+\ifPDFTeX
171+ \DeclareUnicodeCharacter{200C}{\zerowidthnonjoiner}
172+\else
173+ \catcode`^^^^200c=\active
174+ \protected\def ^^^^200c{\zerowidthnonjoiner}
175+\fi
176+%% End of ZWNJ support
177+$endif$
159178$if(beamer)$
160179$if(theme)$
161180\usetheme[$for(themeoptions)$$themeoptions$$sep$,$endfor$]{$theme$}
@@ -219,10 +238,10 @@ $if(keywords)$
219238$endif$
220239$if(colorlinks)$
221240 colorlinks=true,
222− linkcolor=$if(linkcolor)$$linkcolor$$else$Maroon$endif$,
223− filecolor=$if(filecolor)$$filecolor$$else$Maroon$endif$,
224− citecolor=$if(citecolor)$$citecolor$$else$Blue$endif$,
225− urlcolor=$if(urlcolor)$$urlcolor$$else$Blue$endif$,
241+ linkcolor={$if(linkcolor)$$linkcolor$$else$Maroon$endif$},
242+ filecolor={$if(filecolor)$$filecolor$$else$Maroon$endif$},
243+ citecolor={$if(citecolor)$$citecolor$$else$Blue$endif$},
244+ urlcolor={$if(urlcolor)$$urlcolor$$else$Blue$endif$},
226245$else$
227246 hidelinks,
228247$endif$
@@ -325,11 +344,36 @@ $endif$
325344$if(pagestyle)$
326345\pagestyle{$pagestyle$}
327346$endif$
347+$if(csl-refs)$
348+\newlength{\cslhangindent}
349+\setlength{\cslhangindent}{1.5em}
350+\newlength{\csllabelwidth}
351+\setlength{\csllabelwidth}{3em}
352+\newlength{\cslentryspacingunit} % times entry-spacing
353+\setlength{\cslentryspacingunit}{\parskip}
354+\newenvironment{CSLReferences}[2] % #1 hanging-ident, #2 entry spacing
355+ {% don't indent paragraphs
356+ \setlength{\parindent}{0pt}
357+ % turn on hanging indent if param 1 is 1
358+ \ifodd #1
359+ \let\oldpar\par
360+ \def\par{\hangindent=\cslhangindent\oldpar}
361+ \fi
362+ % set entry spacing
363+ \setlength{\parskip}{#2\cslentryspacingunit}
364+ }%
365+ {}
366+\usepackage{calc}
367+\newcommand{\CSLBlock}[1]{#1\hfill\break}
368+\newcommand{\CSLLeftMargin}[1]{\parbox[t]{\csllabelwidth}{#1}}
369+\newcommand{\CSLRightInline}[1]{\parbox[t]{\linewidth - \csllabelwidth}{#1}\break}
370+\newcommand{\CSLIndent}[1]{\hspace{\cslhangindent}#1}
371+$endif$
328372$for(header-includes)$
329373$header-includes$
330374$endfor$
331375$if(lang)$
332−\ifxetex
376+\ifXeTeX
333377 % Load polyglossia as late as possible: uses bidi with RTL langages (e.g. Hebrew, Arabic)
334378 \usepackage{polyglossia}
335379 \setmainlanguage[$for(polyglossia-lang.options)$$polyglossia-lang.options$$sep$,$endfor$]{$polyglossia-lang.name$}
@@ -346,15 +390,15 @@ $if(babel-newcommands)$
346390$endif$
347391\fi
348392$endif$
349−\ifluatex
393+\ifLuaTeX
350394 \usepackage{selnolig} % disable illegal ligatures
351395\fi
352396$if(dir)$
353−\ifxetex
397+\ifXeTeX
354398 % Load bidi as late as possible as it modifies e.g. graphicx
355399 \usepackage{bidi}
356400\fi
357−\ifnum 0\ifxetex 1\fi\ifluatex 1\fi=0 % if pdftex
401+\ifPDFTeX
358402 \TeXXeTstate=1
359403 \newcommand{\RL}[1]{\beginR #1\endR}
360404 \newcommand{\LR}[1]{\beginL #1\endL}
@@ -372,27 +416,8 @@ $for(bibliography)$
372416\addbibresource{$bibliography$}
373417$endfor$
374418$endif$
375−$if(csl-refs)$
376−\newlength{\cslhangindent}
377−\setlength{\cslhangindent}{1.5em}
378−\newlength{\csllabelwidth}
379−\setlength{\csllabelwidth}{3em}
380−\newenvironment{CSLReferences}[2] % #1 hanging-ident, #2 entry spacing
381− {% don't indent paragraphs
382− \setlength{\parindent}{0pt}
383− % turn on hanging indent if param 1 is 1
384− \ifodd #1 \everypar{\setlength{\hangindent}{\cslhangindent}}\ignorespaces\fi
385− % set entry spacing
386− \ifnum #2 > 0
387− \setlength{\parskip}{#2\baselineskip}
388− \fi
389− }%
390− {}
391−\usepackage{calc}
392−\newcommand{\CSLBlock}[1]{#1\hfill\break}
393−\newcommand{\CSLLeftMargin}[1]{\parbox[t]{\csllabelwidth}{#1}}
394−\newcommand{\CSLRightInline}[1]{\parbox[t]{\linewidth - \csllabelwidth}{#1}\break}
395−\newcommand{\CSLIndent}[1]{\hspace{\cslhangindent}#1}
419+$if(nocite-ids)$
420+\nocite{$for(nocite-ids)$$it$$sep$, $endfor$}
396421$endif$
397422$if(csquotes)$
398423\usepackage{csquotes}
@@ -469,12 +494,12 @@ $endif$
469494}
470495$endif$
471496$endif$
472−$if(lot)$
473−\listoftables
474−$endif$
475497$if(lof)$
476498\listoffigures
477499$endif$
500+$if(lot)$
501+\listoftables
502+$endif$
478503$if(linestretch)$
479504\setstretch{$linestretch$}
480505$endif$
data/templates/default.revealjs+79−125
@@ -52,6 +52,9 @@ $endif$
5252$for(author)$
5353 <p class="author">$author$</p>
5454$endfor$
55+$for(institute)$
56+ <p class="institute">$institute$</p>
57+$endfor$
5558$if(date)$
5659 <p class="date">$date$</p>
5760$endif$
@@ -59,7 +62,12 @@ $endif$
5962$endif$
6063$if(toc)$
6164<section id="$idprefix$TOC">
65+<nav role="doc-toc">
66+$if(toc-title)$
67+<h2 id="$idprefix$toc-title">$toc-title$</h2>
68+$endif$
6269$table-of-contents$
70+</nav>
6371</section>
6472$endif$
6573
@@ -69,7 +77,7 @@ $body$
6977
7078 <script src="$revealjs-url$/dist/reveal.js"></script>
7179
72− // reveal.js plugins
80+ <!-- reveal.js plugins -->
7381 <script src="$revealjs-url$/plugin/notes/notes.js"></script>
7482 <script src="$revealjs-url$/plugin/search/search.js"></script>
7583 <script src="$revealjs-url$/plugin/zoom/zoom.js"></script>
@@ -82,252 +90,198 @@ $endif$
8290 // Full list of configuration options available at:
8391 // https://revealjs.com/config/
8492 Reveal.initialize({
85−$if(controls)$
8693 // Display controls in the bottom right corner
8794 controls: $controls$,
88−$endif$
89−$if(controlsTutorial)$
95+
9096 // Help the user learn the controls by providing hints, for example by
9197 // bouncing the down arrow when they first encounter a vertical slide
9298 controlsTutorial: $controlsTutorial$,
93−$endif$
94−$if(controlsLayout)$
99+
95100 // Determines where controls appear, "edges" or "bottom-right"
96101 controlsLayout: '$controlsLayout$',
97−$endif$
98−$if(controlsBackArrows)$
102+
99103 // Visibility rule for backwards navigation arrows; "faded", "hidden"
100104 // or "visible"
101105 controlsBackArrows: '$controlsBackArrows$',
102−$endif$
103−$if(progress)$
106+
104107 // Display a presentation progress bar
105108 progress: $progress$,
106−$endif$
107−$if(slideNumber)$
109+
108110 // Display the page number of the current slide
109111 slideNumber: $slideNumber$,
110−$endif$
111−$if(showSlideNumber)$
112+
112113 // 'all', 'print', or 'speaker'
113114 showSlideNumber: '$showSlideNumber$',
114−$endif$
115−$if(hash)$
115+
116116 // Add the current slide number to the URL hash so that reloading the
117117 // page/copying the URL will return you to the same slide
118118 hash: $hash$,
119−$endif$
120−$if(hashOneBasedIndex)$
119+
121120 // Start with 1 for the hash rather than 0
122121 hashOneBasedIndex: $hashOneBasedIndex$,
123−$endif$
122+
123+ // Flags if we should monitor the hash and change slides accordingly
124+ respondToHashChanges: $respondToHashChanges$,
125+
124126 // Push each slide change to the browser history
125− history: $if(history)$$history$$else$true$endif$,
126−$if(keyboard)$
127+ history: $history$,
128+
127129 // Enable keyboard shortcuts for navigation
128130 keyboard: $keyboard$,
129−$endif$
130−$if(overview)$
131+
131132 // Enable the slide overview mode
132133 overview: $overview$,
133−$endif$
134−$if(center)$
134+
135+ // Disables the default reveal.js slide layout (scaling and centering)
136+ // so that you can use custom CSS layout
137+ disableLayout: false,
138+
135139 // Vertical centering of slides
136140 center: $center$,
137−$endif$
138−$if(touch)$
141+
139142 // Enables touch navigation on devices with touch input
140143 touch: $touch$,
141−$endif$
142−$if(loop)$
144+
143145 // Loop the presentation
144146 loop: $loop$,
145−$endif$
146−$if(rtl)$
147+
147148 // Change the presentation direction to be RTL
148149 rtl: $rtl$,
149−$endif$
150−$if(navigationMode)$
150+
151151 // see https://revealjs.com/vertical-slides/#navigation-mode
152152 navigationMode: '$navigationMode$',
153−$endif$
154−$if(shuffle)$
153+
155154 // Randomizes the order of slides each time the presentation loads
156155 shuffle: $shuffle$,
157−$endif$
158−$if(fragments)$
156+
159157 // Turns fragments on and off globally
160158 fragments: $fragments$,
161−$endif$
162−$if(fragmentInURL)$
159+
163160 // Flags whether to include the current fragment in the URL,
164161 // so that reloading brings you to the same fragment position
165162 fragmentInURL: $fragmentInURL$,
166−$endif$
167−$if(embedded)$
163+
168164 // Flags if the presentation is running in an embedded mode,
169165 // i.e. contained within a limited portion of the screen
170166 embedded: $embedded$,
171−$endif$
172−$if(help)$
167+
173168 // Flags if we should show a help overlay when the questionmark
174169 // key is pressed
175170 help: $help$,
176−$endif$
177−$if(pause)$
178− // Flags if it should be possible to pause the presentation (blackout)
179− pause: $pause$,
180−$endif$
181−$if(showNotes)$
171+
172+ // Flags if it should be possible to pause the presentation (blackout)
173+ pause: $pause$,
174+
182175 // Flags if speaker notes should be visible to all viewers
183176 showNotes: $showNotes$,
184−$endif$
185−$if(autoPlayMedia)$
186− // Global override for autoplaying embedded media (video/audio/iframe)
187− // - null: Media will only autoplay if data-autoplay is present
188− // - true: All media will autoplay, regardless of individual setting
189− // - false: No media will autoplay, regardless of individual setting
177+
178+ // Global override for autoplaying embedded media (null/true/false)
190179 autoPlayMedia: $autoPlayMedia$,
191−$endif$
192−$if(preloadIframes)$
193− // Global override for preloading lazy-loaded iframes
194− // - null: Iframes with data-src AND data-preload will be loaded when within
195− // the viewDistance, iframes with only data-src will be loaded when visible
196− // - true: All iframes with data-src will be loaded when within the viewDistance
197− // - false: All iframes with data-src will be loaded only when visible
180+
181+ // Global override for preloading lazy-loaded iframes (null/true/false)
198182 preloadIframes: $preloadIframes$,
199−$endif$
200−$if(autoSlide)$
183+
201184 // Number of milliseconds between automatically proceeding to the
202185 // next slide, disabled when set to 0, this value can be overwritten
203186 // by using a data-autoslide attribute on your slides
204187 autoSlide: $autoSlide$,
205−$endif$
206−$if(autoSlideStoppable)$
188+
207189 // Stop auto-sliding after user input
208190 autoSlideStoppable: $autoSlideStoppable$,
209−$endif$
210−$if(autoSlideMethod)$
191+
211192 // Use this method for navigation when auto-sliding
212193 autoSlideMethod: $autoSlideMethod$,
213−$endif$
214−$if(defaultTiming)$
194+
215195 // Specify the average time in seconds that you think you will spend
216196 // presenting each slide. This is used to show a pacing timer in the
217197 // speaker view
218198 defaultTiming: $defaultTiming$,
219−$endif$
220−$if(totalTime)$
221− // Specify the total time in seconds that is available to
222− // present. If this is set to a nonzero value, the pacing
223− // timer will work out the time available for each slide,
224− // instead of using the defaultTiming value
225− totalTime: $totalTime$,
226−$endif$
227−$if(minimumTimePerSlide)$
228− // Specify the minimum amount of time you want to allot to
229− // each slide, if using the totalTime calculation method. If
230− // the automated time allocation causes slide pacing to fall
231− // below this threshold, then you will see an alert in the
232− // speaker notes window
233− minimumTimePerSlide: $minimumTimePerSlide$,
234−$endif$
235−$if(mouseWheel)$
199+
236200 // Enable slide navigation via mouse wheel
237201 mouseWheel: $mouseWheel$,
238−$endif$
239−$if(rollingLinks)$
240− // Apply a 3D roll to links on hover
241− rollingLinks: $rollingLinks$,
242−$endif$
243−$if(hideInactiveCursor)$
202+
203+ // The display mode that will be used to show slides
204+ display: '$display$',
205+
244206 // Hide cursor if inactive
245207 hideInactiveCursor: $hideInactiveCursor$,
246−$endif$
247−$if(hideCursorTime)$
208+
248209 // Time before the cursor is hidden (in ms)
249210 hideCursorTime: $hideCursorTime$,
250−$endif$
251−$if(hideAddressBar)$
252− // Hides the address bar on mobile devices
253− hideAddressBar: $hideAddressBar$,
254−$endif$
255−$if(previewLinks)$
211+
256212 // Opens links in an iframe preview overlay
257213 previewLinks: $previewLinks$,
258−$endif$
259−$if(transition)$
260− // Transition style
261− transition: '$transition$', // none/fade/slide/convex/concave/zoom
262−$endif$
263−$if(transitionSpeed)$
264− // Transition speed
265− transitionSpeed: '$transitionSpeed$', // default/fast/slow
266−$endif$
267−$if(backgroundTransition)$
214+
215+ // Transition style (none/fade/slide/convex/concave/zoom)
216+ transition: '$transition$',
217+
218+ // Transition speed (default/fast/slow)
219+ transitionSpeed: '$transitionSpeed$',
220+
268221 // Transition style for full page slide backgrounds
269− backgroundTransition: '$backgroundTransition$', // none/fade/slide/convex/concave/zoom
270−$endif$
271−$if(viewDistance)$
222+ // (none/fade/slide/convex/concave/zoom)
223+ backgroundTransition: '$backgroundTransition$',
224+
272225 // Number of slides away from the current that are visible
273226 viewDistance: $viewDistance$,
274−$endif$
275−$if(mobileViewDistance)$
227+
276228 // Number of slides away from the current that are visible on mobile
277229 // devices. It is advisable to set this to a lower number than
278230 // viewDistance in order to save resources.
279231 mobileViewDistance: $mobileViewDistance$,
280−$endif$
281232$if(parallaxBackgroundImage)$
233+
282234 // Parallax background image
283235 parallaxBackgroundImage: '$parallaxBackgroundImage$', // e.g. "'https://s3.amazonaws.com/hakim-static/reveal-js/reveal-parallax-1.jpg'"
284236$else$
285237$if(background-image)$
238+
286239 // Parallax background image
287240 parallaxBackgroundImage: '$background-image$', // e.g. "'https://s3.amazonaws.com/hakim-static/reveal-js/reveal-parallax-1.jpg'"
288241$endif$
289242$endif$
290243$if(parallaxBackgroundSize)$
244+
291245 // Parallax background size
292246 parallaxBackgroundSize: '$parallaxBackgroundSize$', // CSS syntax, e.g. "2100px 900px"
293247$endif$
294248$if(parallaxBackgroundHorizontal)$
249+
295250 // Amount to move parallax background (horizontal and vertical) on slide change
296251 // Number, e.g. 100
297252 parallaxBackgroundHorizontal: $parallaxBackgroundHorizontal$,
298253$endif$
299254$if(parallaxBackgroundVertical)$
255+
300256 parallaxBackgroundVertical: $parallaxBackgroundVertical$,
301257$endif$
302258$if(width)$
259+
303260 // The "normal" size of the presentation, aspect ratio will be preserved
304261 // when the presentation is scaled to fit different resolutions. Can be
305262 // specified using percentage units.
306263 width: $width$,
307264$endif$
308265$if(height)$
266+
309267 height: $height$,
310268$endif$
311269$if(margin)$
270+
312271 // Factor of the display size that should remain empty around the content
313272 margin: $margin$,
314273$endif$
315274$if(minScale)$
275+
316276 // Bounds for smallest/largest possible scale to apply to content
317277 minScale: $minScale$,
318278$endif$
319279$if(maxScale)$
280+
320281 maxScale: $maxScale$,
321282$endif$
322−$if(zoomKey)$
323− // Modifier key used to click-zoom to part of the slide
324− zoomKey: '$zoomKey$',
325−$endif$
326−$if(display)$
327− // The display mode that will be used to show slides
328− display: '$display$',
329−$endif$
330283$if(mathjax)$
284+
331285 math: {
332286 mathjax: '$mathjaxurl$',
333287 config: 'TeX-AMS_HTML-full',
data/templates/default.s5+3
@@ -61,6 +61,9 @@ $endif$
6161$if(author)$
6262 <h3 class="author">$for(author)$$author$$sep$<br/>$endfor$</h3>
6363$endif$
64+$if(institute)$
65+ <h3 class="institute">$for(institute)$$institute$$sep$<br/>$endfor$</h3>
66+$endif$
6467$if(date)$
6568 <h4 class="date">$date$</h4>
6669$endif$
data/templates/default.slideous+7
@@ -63,9 +63,16 @@ $if(title)$
6363$if(subtitle)$
6464 <h1 class="subtitle">$subtitle$</h1>
6565$endif$
66+$if(author)$
6667 <p class="author">
6768$for(author)$$author$$sep$<br/>$endfor$
6869 </p>
70+$endif$
71+$if(institute)$
72+ <p class="institute">
73+$for(institute)$$institute$$sep$<br/>$endfor$
74+ </p>
75+$endif$
6976$if(date)$
7077 <p class="date">$date$</p>
7178$endif$
data/templates/default.slidy+5
@@ -52,6 +52,11 @@ $if(author)$
5252$for(author)$$author$$sep$<br/>$endfor$
5353 </p>
5454$endif$
55+$if(institute)$
56+ <p class="institute">
57+$for(institute)$$institute$$sep$<br/>$endfor$
58+ </p>
59+$endif$
5560$if(date)$
5661 <p class="date">$date$</p>
5762$endif$
data/templates/styles.html+2−1
@@ -14,7 +14,7 @@ body {
1414 padding-top: $if(margin-top)$$margin-top$$else$50px$endif$;
1515 padding-bottom: $if(margin-bottom)$$margin-bottom$$else$50px$endif$;
1616 hyphens: auto;
17− word-wrap: break-word;
17+ overflow-wrap: break-word;
1818 text-rendering: optimizeLegibility;
1919 font-kerning: normal;
2020}
@@ -93,6 +93,7 @@ $endif$
9393pre code {
9494 padding: 0;
9595 overflow: visible;
96+ overflow-wrap: normal;
9697}
9798.sourceCode {
9899 background-color: transparent;
default.nixnew file+13
@@ -0,0 +1,13 @@
1+# default.nix
2+let
3+ pkgs = import <nixpkgs> { };
4+in
5+ pkgs.haskellPackages.developPackage {
6+ root = ./.;
7+ modifier = drv:
8+ pkgs.haskell.lib.addBuildTools drv (with pkgs.haskellPackages;
9+ [ cabal-install
10+ ghcid
11+ ]);
12+ }
13+
doc/epub.md+9−2
@@ -141,8 +141,15 @@ Pandoc has an EPUB3 writer. It renders LaTeX math into MathML, which
141141EPUB3 readers are supposed to support (but unfortunately few do).
142142
143143Of course, this isn't much help if you want EPUB2 output (`pandoc -t epub2`)
144−or target readers that don't support MathML. Then you should try using the
145−`--webtex` option, which will use a web service to convert the TeX to an image.
144+or target readers that don't support MathML. Then you have two options:
145+
146+1. Use the [`--webtex`](https://pandoc.org/MANUAL.html#option--webtex) option,
147+ which will use a web service to convert the TeX to an image.
148+2. Use the [`--gladtex`](https://pandoc.org/MANUAL.html#option--gladtex) option
149+ to convert maths into SVG images on your local machine.
150+
151+Both GladTeX and WebTeX add the LaTeX source of the formula as alternative text
152+of the image, increasing accessibility for blind users.
146153
147154[KindleGen]: https://www.amazon.com/gp/feature.html?ie=UTF8&docId=1000765211
148155[EPUB]: https://en.wikipedia.org/wiki/EPUB
doc/faqs.mdnew file+128
@@ -0,0 +1,128 @@
1+% FAQs
2+
3+::: faqs
4+
5+## How can I convert a whole directory of files from Markdown to RTF?
6+
7+On linux or OSX:
8+
9+ for f in *.txt; do pandoc "$f" -s -o "${f%.txt}.rtf"; done
10+
11+
12+In Windows Powershell:
13+
14+ gci -r -i *.txt |foreach{$rtf=$_.directoryname+"\"+$_.basename+".rtf";pandoc -f markdown -s $_.name -o $rtf}
15+
16+## I used pandoc to convert a document to ICML (or OPML or RTF), and when I try to open it I'm told it's invalid. What have I done wrong?
17+
18+Be sure to use the `-s` or `--standalone` flag, or you just get a
19+fragment, not a full document with the required header:
20+
21+ pandoc -s -f markdown -t icml -o my.icml my.md
22+
23+## I get a blank document when I try to convert a markdown document in Chinese to PDF.
24+
25+By default, pandoc uses pdflatex to generate the PDF, and pdflatex
26+doesn't handle Chinese characters. But you can change the default to
27+use xelatex instead. You should also make sure you're using a font
28+with Chinese glyphs. For example:
29+
30+ pandoc -o c.pdf --pdf-engine=xelatex -V mainfont='Adobe Ming Std'
31+
32+## The Windows installer does a single user install, rather than installing pandoc for all users. How can I install pandoc for all users?
33+
34+Run the following command as admin:
35+
36+ msiexec /i pandoc-VERSION.msi ALLUSERS=1
37+
38+This will put pandoc in `C:\Program Files\Pandoc`.
39+You can install Pandoc to a different directory by setting APPLICATIONFOLDER parameter,
40+for example:
41+
42+ msiexec /i pandoc-1.11.1.msi ALLUSERS=1 APPLICATIONFOLDER="C:\Pandoc"
43+
44+## How do I change the margins in PDF output?
45+
46+The option
47+
48+ -V geometry:margin=1in
49+
50+will set the margins to one inch on each side. If you don't want uniform
51+margins, you can do something like
52+
53+ -V geometry:"top=2cm, bottom=1.5cm, left=1cm, right=1cm"
54+
55+Or
56+
57+ -V geometry:"left=3cm, width=10cm"
58+
59+For more options, see the documentation for the LaTeX [geometry
60+package](https://www.ctan.org/pkg/geometry).
61+
62+## How does pandoc compare to multimarkdown?
63+
64+Here is a [wiki
65+page](https://github.com/jgm/pandoc/wiki/Pandoc-vs-Multimarkdown)
66+comparing the two.
67+
68+## When I specify an image width of 50% and convert to LaTeX, pandoc sets the height to textheight and the aspect ratio isn't preserved. How can I prevent this?
69+
70+For example, if you convert an image with `{width="50%"}`, the LaTeX produced
71+will be `\includegraphics[width=0.5\textwidth,height=\textheight]`.
72+
73+This output presupposes the following code in pandoc's default latex
74+template:
75+
76+```
77+% Scale images if necessary, so that they will not overflow the page
78+% margins by default, and it is still possible to overwrite the defaults
79+% using explicit options in \includegraphics[width, height, ...]{}
80+\setkeys{Gin}{width=\maxwidth,height=\maxheight,keepaspectratio}
81+```
82+
83+If you don't have this in your custom template, you should
84+add it. If we didn't set the `height` explicitly in this way,
85+the image would not be resized correctly unless it was
86+being resized to smaller than its original size.
87+
88+## Pandoc sometimes uses too much memory. How can I limit the memory used by pandoc?
89+
90+`pandoc +RTS -M30m -RTS` will limit heap memory to 30MB.
91+When converting a document requires more than this, an out of
92+memory error will be issued.
93+
94+## When using `--include-in-header` with PDF or LaTeX output, how do I reference tex declarations coming after `$header-includes$` in the default template?
95+
96+For various reasons, the `$header-includes$` are not at the very
97+end of the LaTeX preamble. This poses a problem when the code
98+you are inserting depends on declarations in the preamble coming
99+after the `$header-includes$` location. For example, you might
100+want to reference the `\author` and `\title` metadata values
101+(set at the very bottom of the preamble) and present them in
102+margins. In that case you can wrap your code in `etoolbox`'s
103+`\AtEndPreamble`. The technique is demonstrated in [this
104+gist](https://gist.github.com/JohnLukeBentley/9dda6166b9ee5c4127afd2b8cd16b70a).
105+When using `\AtEndPreamble`, keep any `makeatletter` or
106+`makeatother` outside of the `\AtEndPreamble`, as shown in the
107+example.
108+
109+## How can I convert PDFs to other formats using pandoc?
110+
111+You can't. You can try opening the PDF in Word or Google Docs
112+and saving in a format from which pandoc can convert directly.
113+
114+## Do I really need to install a 1 GB TeX installation to produce a PDF using pandoc?
115+
116+No. You can get by with a relatively small TeX installation,
117+for example, by starting with MacTeX's Basic TeX distribution
118+and using the `tlmgr` tool to install a few packages required by pandoc
119+(see https://pandoc.org/MANUAL.html#creating-a-pdf).
120+
121+Or, you can produce PDFs via HTML and `wkhtmltopdf`,
122+or via groff ms and `pdfroff`. (These don't produce as nice
123+topography as TeX, particularly when it comes to math, but they
124+may be fine for many purposes.)
125+
126+
127+:::
128+
doc/filters.md+2
@@ -394,6 +394,8 @@ automatically assign it `Just` the target format or `Nothing`.
394394
395395We compile our script:
396396
397+ # first, make sure pandoc-types is installed:
398+ cabal install --lib pandoc-types --package-env .
397399 ghc --make handleRuby
398400
399401Then run it:
doc/jats.md+31−7
@@ -33,6 +33,11 @@ Metadata Values
3333
3434 See [`<given-names>`][elem:given-names].
3535
36+ `name`
37+ : full name of the author; included only as a fallback if
38+ `author.surname` is not available. Tagged with
39+ [`<string-name>`][elem:string-name].
40+
3641 `email`
3742 : the contributor's email address.
3843
@@ -45,6 +50,13 @@ Metadata Values
4550 list must also occur as the `id` of an affiliation listed in
4651 the top-level `affiliation` list.
4752
53+ `equal-contrib`
54+ : boolean attribute used to mark authors who contributed
55+ equally to the work. The
56+ [`equal-contrib`][attr:equal-contrib] attribute is added
57+ to the author's [`<contrib>`] element if this is set to a
58+ truthy value.
59+
4860 `cor-id`
4961 : identifier linking to the contributor's correspondence
5062 information. The info itself must be stored in as an item in
@@ -167,12 +179,13 @@ Metadata Values
167179 pass these components directly.
168180
169181 The publication date is recorded in the document via the
170− [`<pub-date>`][elem:pub-date] element and its sub-elements. The
171− [`pub-type`][attr:pub-type] attribute is always set to `epub`.
182+ [`<pub-date>`] element and its sub-elements. The
183+ [`publication-format`][attr:publication-format] attribute is
184+ always set to `electronic`.
172185
173186 `iso-8601`
174187 : ISO-8601 representation of the publication date. Used as the
175− value of the [`pub-date`][elem:pub-date] element's
188+ value of the [`<pub-date>`] element's
176189 [`iso-8601-date`][attr:iso-8601-date] attribute.
177190
178191 This value is set automatically if pandoc can parse the
@@ -186,6 +199,12 @@ Metadata Values
186199 The values are set automatically if pandoc can parse the
187200 `date` value as a date.
188201
202+ `type`
203+ : The type of event marked by this date. The value is set as
204+ the [`date-type`][attr:date-type] attribute on the
205+ [`<pub-date>`] element and defaults to "pub" if not
206+ specified.
207+
189208`article`
190209: information concerning the article that identifies or describes
191210 it. The key-value pairs within this map are typically used
@@ -295,12 +314,14 @@ Metadata Values
295314 `pissn`
296315 : ISSN identifier of the publication's print version. Used as
297316 content of the [`<issn>`][elem:issn] element with the
298− [`pub-type`][attr:pub-type] attribute set to `ppub`.
317+ [`publication-format`][attr:publication-format] attribute
318+ set to `print`.
299319
300320 `eissn`
301321 : ISSN identifier of the publication's electronic version.
302322 Used as content of the [`<issn>`][elem:issn] element with
303− the [`pub-type`][attr:pub-type] attribute set to `epub`.
323+ the [`publication-format`][attr:publication-format]
324+ attribute set to `electronic`.
304325
305326 `publisher-name`
306327 : name of the publishing entity (person, company, or other).
@@ -340,6 +361,8 @@ Required metadata values:
340361
341362[Ringgold]: https://ringgold.com/
342363[attr:content-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/content-type.html
364+[attr:date-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/date-type.html
365+[attr:equal-contrib]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/equal-contrib.html
343366[attr:fn-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/fn-type.html
344367[attr:institution-id-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/institution-id-type.html
345368[attr:iso-8601-date]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/iso-8601-date.html
@@ -347,7 +370,7 @@ Required metadata values:
347370[attr:kwd-group-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/kwd-group-type.html
348371[attr:license-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/license-type.html
349372[attr:pub-id-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/pub-id-type.html
350−[attr:pub-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/pub-type.html
373+[attr:publication-format]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/publication-format.html
351374[attr:ref-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/ref-type.html
352375[attr:rid]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/rid.html
353376[attr:subj-group-type]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/attribute/subj-group-type.html
@@ -372,7 +395,6 @@ Required metadata values:
372395[elem:license]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/license.html
373396[elem:notes]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/notes.html
374397[elem:permissions]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/permissions.html
375−[elem:pub-date]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/pub-date.html
376398[elem:publisher-loc]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/publisher-loc.html
377399[elem:publisher-name]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/publisher-name.html
378400[elem:string-name]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/string-name.html
@@ -384,7 +406,9 @@ Required metadata values:
384406[`<addr-line>`]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/addr-line.html
385407[`<aff>`]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/aff.html
386408[`<city>`]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/city.html
409+[`<contrib>`]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/contrib.html
387410[`<country>`]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/country.html
388411[`<institution-id>`]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/institution-id.html
389412[`<institution-wrap>`]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/institution-wrap.html
390413[`<institution>`]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/institution.html
414+[`<pub-date>`]: https://jats.nlm.nih.gov/publishing/tag-library/1.2/element/pub-date.html
doc/libraries.mdnew file+63
@@ -0,0 +1,63 @@
1+---
2+title: The pandoc family of libraries
3+author: John MacFarlane
4+---
5+
6+The following Haskell libraries have been developed to support
7+pandoc:
8+
9+[skylighting-core] and [skylighting]
10+: Syntax highlighting engine supporting over 140 languages.
11+
12+[citeproc]
13+: Citation processing using CSL stylesheets.
14+
15+[texmath]
16+: Conversion of math between tex, Word equation, MathML, and GNU eqn.
17+
18+[unicode-collation]
19+: Proper Unicode collation (sorting).
20+
21+[doclayout]
22+: Combinators for laying out a textual document, with support
23+ for line wrapping, tabular layout, and more.
24+
25+[doctemplates]
26+: Supports pandoc's templates.
27+
28+[commonmark], [commonmark-extensions], and [commonmark-pandoc]
29+: Efficient, standards-compliant parser for commonmark and extensions.
30+
31+[ipynb]
32+: Representation of Jupyter notebooks and conversion to and
33+ from JSON.
34+
35+[zip-archive]
36+: A pure zip file creator and extractor, used by pandoc for
37+ docx, ODT, and EPUB.
38+
39+[rfc5051]
40+: Simple unicode collation (used for citation sorting).
41+
42+[emojis]
43+: Conversion between emoji characters and aliases.
44+
45+[jira-wiki-markup]
46+: Support for parsing Jira wiki syntax.
47+
48+[skylighting]: https://hackage.haskell.org/package/skylighting
49+[skylighting-core]: https://hackage.haskell.org/package/skylighting-core
50+[citeproc]: https://hackage.haskell.org/package/citeproc
51+[texmath]: https://hackage.haskell.org/package/texmath
52+[doclayout]: https://hackage.haskell.org/package/doclayout
53+[doctemplates]: https://hackage.haskell.org/package/doctemplates
54+[commonmark]: https://hackage.haskell.org/package/commonmark
55+[commonmark-extensions]: https://hackage.haskell.org/package/commonmark-extensions
56+[commonmark-pandoc]: https://hackage.haskell.org/package/commonmark-pandoc
57+[ipynb]: https://hackage.haskell.org/package/ipynb
58+[zip-archive]: https://hackage.haskell.org/package/zip-archive
59+[rfc5051]: https://hackage.haskell.org/package/rfc5051
60+[emojis]: https://hackage.haskell.org/package/emojis
61+[jira-wiki-markup]: https://hackage.haskell.org/package/jira-wiki-markup
62+[unicode-collation]: https://hackage.haskell.org/package/unicode-collation
63+
doc/lua-filters.md+192−6
@@ -109,7 +109,7 @@ element filtering function. In other words, filter entries will
109109be called for each corresponding element in the document,
110110getting the respective element as input.
111111
112−The return of a filter function must one of the following:
112+The return of a filter function must be one of the following:
113113
114114- nil: this means that the object should remain unchanged.
115115- a pandoc object: this must be of the same type as the input
@@ -3084,7 +3084,8 @@ Clear-out the media bag, deleting all items.
30843084
30853085`insert (filepath, mime_type, contents)`
30863086
3087−Adds a new entry to pandoc's media bag.
3087+Adds a new entry to pandoc's media bag. Replaces any existing
3088+mediabag entry with the same `filepath`.
30883089
30893090Parameters:
30903091
@@ -3092,7 +3093,7 @@ Parameters:
30923093: filename and path relative to the output folder.
30933094
30943095`mime_type`:
3095−: the file's MIME type
3096+: the file's MIME type; use `nil` if unknown or unavailable.
30963097
30973098`contents`:
30983099: the binary contents of the file.
@@ -3176,12 +3177,22 @@ Usage:
31763177
31773178### fetch {#pandoc.mediabag.fetch}
31783179
3179−`fetch (source, base_url)`
3180+`fetch (source)`
31803181
31813182Fetches the given source from a URL or local file. Returns two
31823183values: the contents of the file and the MIME type (or an empty
31833184string).
31843185
3186+The function will first try to retrieve `source` from the
3187+mediabag; if that fails, it will try to download it or read it
3188+from the local file system while respecting pandoc's "resource
3189+path" setting.
3190+
3191+Parameters:
3192+
3193+`source`:
3194+: path to a resource; either a local file path or URI
3195+
31853196Returns:
31863197
31873198- the entries MIME type, or nil if the file was not found.
@@ -3190,11 +3201,11 @@ Returns:
31903201Usage:
31913202
31923203 local diagram_url = "https://pandoc.org/diagram.jpg"
3193− local mt, contents = pandoc.mediabag.fetch(diagram_url, ".")
3204+ local mt, contents = pandoc.mediabag.fetch(diagram_url)
31943205
31953206# Module pandoc.List
31963207
3197−The this module defines pandoc's list type. It comes with useful
3208+This module defines pandoc's list type. It comes with useful
31983209methods and convenience functions.
31993210
32003211## Constructor
@@ -3376,6 +3387,181 @@ methods and convenience functions.
33763387 `comp`:
33773388 : Comparison function as described above.
33783389
3390+# Module pandoc.path
3391+
3392+Module for file path manipulations.
3393+
3394+## Static Fields {#pandoc.path-fields}
3395+
3396+### separator {#pandoc.path.separator}
3397+
3398+The character that separates directories.
3399+
3400+### search_path_separator {#pandoc.path.search_path_separator}
3401+
3402+The character that is used to separate the entries in the `PATH`
3403+environment variable.
3404+
3405+## Functions {#pandoc.path-functions}
3406+
3407+### directory (filepath) {#pandoc.path.directory}
3408+
3409+Gets the directory name, i.e., removes the last directory
3410+separator and everything after from the given path.
3411+
3412+Parameters:
3413+
3414+filepath
3415+: path (string)
3416+
3417+Returns:
3418+
3419+- The filepath up to the last directory separator. (string)
3420+
3421+### filename (filepath) {#pandoc.path.filename}
3422+
3423+Get the file name.
3424+
3425+Parameters:
3426+
3427+filepath
3428+: path (string)
3429+
3430+Returns:
3431+
3432+- File name part of the input path. (string)
3433+
3434+### is_absolute (filepath) {#pandoc.path.is_absolute}
3435+
3436+Checks whether a path is absolute, i.e. not fixed to a root.
3437+
3438+Parameters:
3439+
3440+filepath
3441+: path (string)
3442+
3443+Returns:
3444+
3445+- `true` iff `filepath` is an absolute path, `false` otherwise.
3446+ (boolean)
3447+
3448+### is_relative (filepath) {#pandoc.path.is_relative}
3449+
3450+Checks whether a path is relative or fixed to a root.
3451+
3452+Parameters:
3453+
3454+filepath
3455+: path (string)
3456+
3457+Returns:
3458+
3459+- `true` iff `filepath` is a relative path, `false` otherwise.
3460+ (boolean)
3461+
3462+### join (filepaths) {#pandoc.path.join}
3463+
3464+Join path elements back together by the directory separator.
3465+
3466+Parameters:
3467+
3468+filepaths
3469+: path components (list of strings)
3470+
3471+Returns:
3472+
3473+- The joined path. (string)
3474+
3475+### make_relative (path, root[, unsafe]) {#pandoc.path.make_relative}
3476+
3477+Contract a filename, based on a relative path. Note that the
3478+resulting path will usually not introduce `..` paths, as the
3479+presence of symlinks means `../b` may not reach `a/b` if it starts
3480+from `a/c`. For a worked example see [this blog
3481+post](https://neilmitchell.blogspot.co.uk/2015/10/filepaths-are-subtle-symlinks-are-hard.html).
3482+
3483+Set `unsafe` to a truthy value to a allow `..` in paths.
3484+
3485+Parameters:
3486+
3487+path
3488+: path to be made relative (string)
3489+
3490+root
3491+: root path (string)
3492+
3493+unsafe
3494+: whether to allow `..` in the result. (boolean)
3495+
3496+Returns:
3497+
3498+- contracted filename (string)
3499+
3500+### normalize (filepath) {#pandoc.path.normalize}
3501+
3502+Normalizes a path.
3503+
3504+- `//` makes sense only as part of a (Windows) network drive;
3505+ elsewhere, multiple slashes are reduced to a single
3506+ `path.separator` (platform dependent).
3507+- `/` becomes `path.separator` (platform dependent)
3508+- `./` -\> ''
3509+- an empty path becomes `.`
3510+
3511+Parameters:
3512+
3513+filepath
3514+: path (string)
3515+
3516+Returns:
3517+
3518+- The normalized path. (string)
3519+
3520+### split (filepath) {#pandoc.path.split}
3521+
3522+Splits a path by the directory separator.
3523+
3524+Parameters:
3525+
3526+filepath
3527+: path (string)
3528+
3529+Returns:
3530+
3531+- List of all path components. (list of strings)
3532+
3533+### split_extension (filepath) {#pandoc.path.split_extension}
3534+
3535+Splits the last extension from a file path and returns the parts. The
3536+extension, if present, includes the leading separator; if the path has
3537+no extension, then the empty string is returned as the extension.
3538+
3539+Parameters:
3540+
3541+filepath
3542+: path (string)
3543+
3544+Returns:
3545+
3546+- filepath without extension (string)
3547+
3548+- extension or empty string (string)
3549+
3550+### split_search_path (search_path) {#pandoc.path.split_search_path}
3551+
3552+Takes a string and splits it on the `search_path_separator` character.
3553+Blank items are ignored on Windows, and converted to `.` on Posix. On
3554+Windows path elements are stripped of quotes.
3555+
3556+Parameters:
3557+
3558+search_path
3559+: platform-specific search path (string)
3560+
3561+Returns:
3562+
3563+- list of directories in search path (list of strings)
3564+
33793565# Module pandoc.system
33803566
33813567Access to system information and functionality.
doc/org.md+16−2
@@ -25,8 +25,7 @@ The following export keywords are supported:
2525- LANGUAGE: document language; included as plain-text metadata
2626 field `lang`. The value should be a [BCP47 language tag].
2727
28−- SELECT_TAGS: tags which select a tree for export. Currently
29− *unsupported*.
28+- SELECT_TAGS: tags which select a tree for export.
3029
3130- EXCLUDE\_TAGS: tags which prevent a subtree from being
3231 exported. Fully supported.
@@ -37,6 +36,21 @@ The following export keywords are supported:
3736 defaults to stdout unless a target has to be given as a command
3837 line option.
3938
39+::: {.alert .alert-info}
40+Pandoc tries to be compatible with org-mode when exporting an org document. If
41+you find some behavior confusing, please do refer to org-mode
42+[Export-Settings](https://orgmode.org/manual/Export-Settings.html)
43+documentation. For example, a common confusion
44+([#3214](https://github.com/jgm/pandoc/issues/3214 "Problem with headers lower
45+then 3 in org-mode reader"), [#5169](https://github.com/jgm/pandoc/issues/5169
46+"org mode headings past level three converted to numbered outline list"),
47+[#6145](https://github.com/jgm/pandoc/issues/6145 "Headers 4 levels deep render
48+differently"), [#7236](https://github.com/jgm/pandoc/issues/7236 "In Org mode,
49+Header with level > 3 are not recognized as headers")) is treatment of headers
50+with level > 3 differently because org-mode sets `org-export-headline-levels`
51+(configurable with `#+OPTIONS: H:3`) to 3 by default.
52+:::
53+
4054[BCP47 language tag]: https://tools.ietf.org/html/bcp47
4155
4256Format-specific options
doc/using-the-pandoc-api.md+25−12
@@ -14,7 +14,7 @@ and types is available at
1414Pandoc is structured as a set of *readers*, which translate
1515various input formats into an abstract syntax tree (the
1616Pandoc AST) representing a structured document, and a set of
17−*writers*, which render this AST into various input formats.
17+*writers*, which render this AST into various output formats.
1818Pictorially:
1919
2020```
@@ -90,8 +90,14 @@ Some notes:
9090Let's look at the types of `readMarkdown` and `writeRST`:
9191
9292```haskell
93−readMarkdown :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
94−writeRST :: PandocMonad m => WriterOptions -> Pandoc -> m Text
93+readMarkdown :: (PandocMonad m, ToSources a)
94+ => ReaderOptions
95+ -> a
96+ -> m Pandoc
97+writeRST :: PandocMonad m
98+ => WriterOptions
99+ -> Pandoc
100+ -> m Text
95101```
96102
97103The `PandocMonad m =>` part is a typeclass constraint.
@@ -133,7 +139,7 @@ report :: PandocMonad m => LogMessage -> m ()
133139-- | Fetch an image or other item from the local filesystem or the net.
134140-- Returns raw content and maybe mime type.
135141fetchItem :: PandocMonad m
136− => String
142+ => Text
137143 -> m (B.ByteString, Maybe MimeType)
138144
139145-- Set the resource path searched by 'fetchItem'.
@@ -155,6 +161,13 @@ Note that `PandocIO` is an instance of `MonadIO`, so you can
155161use `liftIO` to perform arbitrary IO operations inside a pandoc
156162conversion chain.
157163
164+`readMarkdown` is polymorphic in its second argument, which
165+can be any type that is an instance of the `ToSources`
166+typeclass. You can use `Text`, as in the example above.
167+But you can also use `[(FilePath, Text)]`, if the input comes
168+from multiple files and you want to track source positions
169+accurately.
170+
158171# Options
159172
160173The first argument of each reader or writer is for
@@ -200,8 +213,8 @@ of the Monoid typeclass and can easily be concatenated:
200213import Text.Pandoc.Builder
201214
202215mydoc :: Pandoc
203−mydoc = doc $ header 1 (text "Hello!")
204− <> para (emph (text "hello world") <> text ".")
216+mydoc = doc $ header 1 (text (T.pack "Hello!"))
217+ <> para (emph (text (T.pack "hello world")) <> text (T.pack "."))
205218
206219main :: IO ()
207220main = print mydoc
@@ -248,16 +261,16 @@ import qualified Data.Text as T
248261import Data.List (intersperse)
249262
250263data Station = Station{
251− address :: String
252− , name :: String
253− , cardsAccepted :: [String]
264+ address :: T.Text
265+ , name :: T.Text
266+ , cardsAccepted :: [T.Text]
254267 } deriving Show
255268
256269instance FromJSON Station where
257270 parseJSON (Object v) = Station <$>
258271 v .: "street_address" <*>
259272 v .: "station_name" <*>
260− (words <$> (v .:? "cards_accepted" .!= ""))
273+ (T.words <$> (v .:? "cards_accepted" .!= ""))
261274 parseJSON _ = mzero
262275
263276createLetter :: [Station] -> Pandoc
@@ -315,7 +328,7 @@ users to override the system defaults. If you want to disable
315328this behavior, use `setUserDataDir Nothing`.
316329
317330To render a template, use `renderTemplate'`, which takes two
318−arguments, a template (String) and a context (any instance
331+arguments, a template (Text) and a context (any instance
319332of ToJSON). If you want to create a context from the metadata
320333part of a Pandoc document, use `metaToJSON'` from
321334[Text.Pandoc.Writers.Shared]. If you also want to incorporate
@@ -413,7 +426,7 @@ concatenated together. Here's an example that returns a
413426list of the URLs linked to in a document:
414427
415428```haskell
416−listURLs :: Pandoc -> [String]
429+listURLs :: Pandoc -> [Text]
417430listURLs = query urls
418431 where urls (Link _ _ (src, _)) = [src]
419432 urls _ = []
linux/make_artifacts.sh+19−7
@@ -2,13 +2,23 @@ set -e
22
33MACHINE=$(uname -m)
44case "$MACHINE" in
5− x86_64) ARCHITECTURE=amd64;;
6− i686) ARCHITECTURE=i386;;
7− i386) ARCHITECTURE=i386;;
5+ x86_64) ARCHITECTURE=amd64;;
6+ i686) ARCHITECTURE=i386;;
7+ i386) ARCHITECTURE=i386;;
8+ aarch64) ARCHITECTURE=arm64;;
9+ *) ARCHITECTURE=unknown;;
810esac
911
1012ARTIFACTS="${ARTIFACTS:-/artifacts}"
1113
14+# This is our sentinel that tells us when we're done.
15+rm -f $ARTIFACTS/DONE
16+
17+clean_up() {
18+ echo "All done!" > "$ARTIFACTS/DONE"
19+}
20+trap clean_up EXIT
21+
1222# build binaries
1323
1424cabal --version
@@ -16,9 +26,9 @@ ghc --version
1626
1727cabal v2-update
1828cabal v2-clean
19−cabal v2-configure --enable-tests -f-export-dynamic -fembed_data_files --enable-executable-static --ghc-options '-optc-Os -optl=-pthread' pandoc
20−cabal v2-build
21−cabal v2-test -j1
29+cabal v2-configure --enable-tests -f-export-dynamic -fembed_data_files --enable-executable-static --ghc-options '-j4 +RTS -A256m -RTS -split-sections -optc-Os -optl=-pthread' pandoc
30+cabal v2-build -j4
31+cabal v2-test -j4
2232for f in $(find dist-newstyle -name 'pandoc' -type f -perm /400); do cp $f /artifacts/; done
2333
2434# make deb
@@ -68,5 +78,7 @@ mv pandoc $TARGET/bin
6878strip $TARGET/bin/pandoc
6979gzip -9 $TARGET/share/man/man1/pandoc.1
7080
71−tar cvzf $TARGET-linux-amd64.tar.gz $TARGET
81+tar cvzf $TARGET-linux-$ARCHITECTURE.tar.gz $TARGET
7282rm -r $TARGET
83+
84+exit 0
man/pandoc.1+358−89
@@ -1,7 +1,7 @@
11'\" t
2−.\" Automatically generated by Pandoc 2.11.3
2+.\" Automatically generated by Pandoc 2.14.0.1
33.\"
4−.TH "Pandoc User\[cq]s Guide" "" "December 17, 2020" "pandoc 2.11.3.1" ""
4+.TH "Pandoc User\[cq]s Guide" "" "June 20, 2021" "pandoc 2.14.0.3" ""
55.hy
66.SH NAME
77pandoc - general markup converter
@@ -172,13 +172,12 @@ You can then test it with \f[C]pdflatex test.tex\f[R].
172172When using LaTeX, the following packages need to be available (they are
173173included with all recent versions of TeX Live): \f[C]amsfonts\f[R],
174174\f[C]amsmath\f[R], \f[C]lm\f[R], \f[C]unicode-math\f[R],
175−\f[C]ifxetex\f[R], \f[C]ifluatex\f[R], \f[C]listings\f[R] (if the
176−\f[C]--listings\f[R] option is used), \f[C]fancyvrb\f[R],
177−\f[C]longtable\f[R], \f[C]booktabs\f[R], \f[C]graphicx\f[R] (if the
178−document contains images), \f[C]hyperref\f[R], \f[C]xcolor\f[R],
179−\f[C]ulem\f[R], \f[C]geometry\f[R] (with the \f[C]geometry\f[R] variable
180−set), \f[C]setspace\f[R] (with \f[C]linestretch\f[R]), and
181−\f[C]babel\f[R] (with \f[C]lang\f[R]).
175+\f[C]iftex\f[R], \f[C]listings\f[R] (if the \f[C]--listings\f[R] option
176+is used), \f[C]fancyvrb\f[R], \f[C]longtable\f[R], \f[C]booktabs\f[R],
177+\f[C]graphicx\f[R] (if the document contains images),
178+\f[C]hyperref\f[R], \f[C]xcolor\f[R], \f[C]ulem\f[R], \f[C]geometry\f[R]
179+(with the \f[C]geometry\f[R] variable set), \f[C]setspace\f[R] (with
180+\f[C]linestretch\f[R]), and \f[C]babel\f[R] (with \f[C]lang\f[R]).
182181The use of \f[C]xelatex\f[R] or \f[C]lualatex\f[R] as the PDF engine
183182requires \f[C]fontspec\f[R].
184183\f[C]lualatex\f[R] uses \f[C]selnolig\f[R].
@@ -320,6 +319,10 @@ Specify output format.
320319.IP \[bu] 2
321320\f[C]beamer\f[R] (LaTeX beamer slide show)
322321.IP \[bu] 2
322+\f[C]bibtex\f[R] (BibTeX bibliography)
323+.IP \[bu] 2
324+\f[C]biblatex\f[R] (BibLaTeX bibliography)
325+.IP \[bu] 2
323326\f[C]commonmark\f[R] (CommonMark Markdown)
324327.IP \[bu] 2
325328\f[C]commonmark_x\f[R] (CommonMark Markdown with extensions)
@@ -454,16 +457,15 @@ On *nix and macOS systems this will be the \f[C]pandoc\f[R] subdirectory
454457of the XDG data directory (by default, \f[C]$HOME/.local/share\f[R],
455458overridable by setting the \f[C]XDG_DATA_HOME\f[R] environment
456459variable).
457−If that directory does not exist, \f[C]$HOME/.pandoc\f[R] will be used
458−(for backwards compatibility).
459−In Windows the default user data directory is
460+If that directory does not exist and \f[C]$HOME/.pandoc\f[R] exists, it
461+will be used (for backwards compatibility).
462+On Windows the default user data directory is
460463\f[C]C:\[rs]Users\[rs]USERNAME\[rs]AppData\[rs]Roaming\[rs]pandoc\f[R].
461464You can find the default user data directory on your system by looking
462465at the output of \f[C]pandoc --version\f[R].
463−A \f[C]reference.odt\f[R], \f[C]reference.docx\f[R], \f[C]epub.css\f[R],
464−\f[C]templates\f[R], \f[C]slidy\f[R], \f[C]slideous\f[R], or
465−\f[C]s5\f[R] directory placed in this directory will override
466−pandoc\[cq]s normal defaults.
466+Data files placed in this directory (for example,
467+\f[C]reference.odt\f[R], \f[C]reference.docx\f[R], \f[C]epub.css\f[R],
468+\f[C]templates\f[R]) will override pandoc\[cq]s normal defaults.
467469.TP
468470\f[B]\f[CB]-d\f[B]\f[R] \f[I]FILE\f[R], \f[B]\f[CB]--defaults=\f[B]\f[R]\f[I]FILE\f[R]
469471Specify a set of default option settings.
@@ -623,8 +625,8 @@ above).
623625.IP "3." 3
624626\f[C]$PATH\f[R] (executable only)
625627.PP
626−Filters and Lua-filters are applied in the order specified on the
627−command line.
628+Filters, Lua-filters, and citeproc processing are applied in the order
629+specified on the command line.
628630.RE
629631.TP
630632\f[B]\f[CB]-L\f[B]\f[R] \f[I]SCRIPT\f[R], \f[B]\f[CB]--lua-filter=\f[B]\f[R]\f[I]SCRIPT\f[R]
@@ -641,29 +643,17 @@ The \f[C]pandoc\f[R] Lua module provides helper functions for element
641643creation.
642644It is always loaded into the script\[cq]s Lua environment.
643645.PP
644−The following is an example Lua script for macro-expansion:
645−.IP
646−.nf
647−\f[C]
648−function expand_hello_world(inline)
649− if inline.c == \[aq]{{helloworld}}\[aq] then
650− return pandoc.Emph{ pandoc.Str \[dq]Hello, World\[dq] }
651− else
652− return inline
653− end
654−end
655−
656−return {{Str = expand_hello_world}}
657−\f[R]
658−.fi
646+See the Lua filters documentation for further details.
659647.PP
660648In order of preference, pandoc will look for Lua filters in
661649.IP "1." 3
662−a specified full or relative path (executable or non-executable)
650+a specified full or relative path
663651.IP "2." 3
664−\f[C]$DATADIR/filters\f[R] (executable or non-executable) where
665−\f[C]$DATADIR\f[R] is the user data directory (see \f[C]--data-dir\f[R],
666−above).
652+\f[C]$DATADIR/filters\f[R] where \f[C]$DATADIR\f[R] is the user data
653+directory (see \f[C]--data-dir\f[R], above).
654+.PP
655+Filters, Lua filters, and citeproc processing are applied in the order
656+specified on the command line.
667657.RE
668658.TP
669659\f[B]\f[CB]-M\f[B]\f[R] \f[I]KEY\f[R][\f[B]\f[CB]=\f[B]\f[R]\f[I]VAL\f[R]], \f[B]\f[CB]--metadata=\f[B]\f[R]\f[I]KEY\f[R][\f[B]\f[CB]:\f[B]\f[R]\f[I]VAL\f[R]]
@@ -726,11 +716,11 @@ Extract images and other media contained in or linked from the source
726716document to the path \f[I]DIR\f[R], creating it if necessary, and adjust
727717the images references in the document so they point to the extracted
728718files.
729−If the source format is a binary container (docx, epub, or odt), the
730−media is extracted from the container and the original filenames are
731−used.
732−Otherwise the media is read from the file system or downloaded, and new
733−filenames are constructed based on SHA1 hashes of the contents.
719+Media are downloaded, read from the file system, or extracted from a
720+binary container (e.g.\ docx), as needed.
721+The original file paths are used if they are relative paths not
722+containing \f[C]..\f[R].
723+Otherwise filenames are constructed from the SHA1 hash of the contents.
734724.TP
735725\f[B]\f[CB]--abbreviations=\f[B]\f[R]\f[I]FILE\f[R]
736726Specifies a custom abbreviations file, with abbreviations one to a line.
@@ -740,9 +730,9 @@ system default.
740730To see the system default, use
741731\f[C]pandoc --print-default-data-file=abbreviations\f[R].
742732The only use pandoc makes of this list is in the Markdown reader.
743−Strings ending in a period that are found in this list will be followed
744−by a nonbreaking space, so that the period will not produce
745−sentence-ending space in formats like LaTeX.
733+Strings found in this list will be followed by a nonbreaking space, and
734+the period will not produce sentence-ending space in formats like LaTeX.
735+The strings may not contain spaces.
746736.SS General writer options
747737.TP
748738\f[B]\f[CB]-s\f[B]\f[R], \f[B]\f[CB]--standalone\f[B]\f[R]
@@ -933,6 +923,11 @@ Note that, if \f[C]--resource-path\f[R] is specified, the working
933923directory must be explicitly listed or it will not be searched.
934924For example: \f[C]--resource-path=.:test\f[R] will search the working
935925directory and the \f[C]test\f[R] subdirectory, in that order.
926+This option can be used repeatedly.
927+Search path components that come later on the command line will be
928+searched before those that come earlier, so
929+\f[C]--resource-path foo:bar --resource-path baz:bim\f[R] is equivalent
930+to \f[C]--resource-path baz:bim:foo:bar\f[R].
936931.TP
937932\f[B]\f[CB]--request-header=\f[B]\f[R]\f[I]NAME\f[R]\f[B]\f[CB]:\f[B]\f[R]\f[I]VAL\f[R]
938933Set the request header \f[I]NAME\f[R] to the value \f[I]VAL\f[R] when
@@ -1509,16 +1504,16 @@ inserted.
15091504.TP
15101505\f[B]\f[CB]--gladtex\f[B]\f[R]
15111506Enclose TeX math in \f[C]<eq>\f[R] tags in HTML output.
1512−The resulting HTML can then be processed by GladTeX to produce images of
1513−the typeset formulas and an HTML file with links to these images.
1514−So, the procedure is:
1507+The resulting HTML can then be processed by GladTeX to produce SVG
1508+images of the typeset formulas and an HTML file with these images
1509+embedded.
15151510.RS
15161511.IP
15171512.nf
15181513\f[C]
15191514pandoc -s --gladtex input.md -o myfile.htex
1520−gladtex -d myfile-images myfile.htex
1521−# produces myfile.html and images in myfile-images
1515+gladtex -d image_dir myfile.htex
1516+# produces myfile.html and images in image_dir
15221517\f[R]
15231518.fi
15241519.RE
@@ -1622,6 +1617,11 @@ T}@T{
16221617PandocPDFError
16231618T}
16241619T{
1620+44
1621+T}@T{
1622+PandocXMLError
1623+T}
1624+T{
1625162547
16261626T}@T{
16271627PandocPDFProgramNotFoundError
@@ -1682,6 +1682,11 @@ T}@T{
16821682PandocIpynbDecodingError
16831683T}
16841684T{
1685+94
1686+T}@T{
1687+PandocUnsupportedCharsetError
1688+T}
1689+T{
1685169097
16861691T}@T{
16871692PandocCouldNotFindDataFileError
@@ -1714,6 +1719,16 @@ input-files:
17141719- content.md
17151720# or you may use input-file: with a single value
17161721
1722+# Include options from the specified defaults files.
1723+# The files will be searched for first in the working directory
1724+# and then in the defaults subdirectory of the user data directory.
1725+# The files are included in the same order in which they appear in
1726+# the list. Options specified in this defaults file always have
1727+# priority over the included ones.
1728+defaults:
1729+- defsA
1730+- defsB
1731+
17171732template: letter
17181733standalone: true
17191734self-contained: false
@@ -1771,8 +1786,11 @@ data-dir:
17711786verbosity: INFO
17721787log-file: log.json
17731788
1774−# citeproc, natbib, or biblatex
1789+# citeproc, natbib, or biblatex. This only affects LaTeX
1790+# output. If you want to use citeproc to format citations,
1791+# you should also set \[aq]citeproc: true\[aq] (see above).
17751792cite-method: citeproc
1793+
17761794# part, chapter, section, or default:
17771795top-level-division: chapter
17781796abbreviations:
@@ -1873,6 +1891,38 @@ verbosity: INFO
18731891\f[R]
18741892.fi
18751893.PP
1894+In fields that expect a file path (or list of file paths), the following
1895+syntax may be used to interpolate environment variables:
1896+.IP
1897+.nf
1898+\f[C]
1899+csl: ${HOME}/mycsldir/special.csl
1900+\f[R]
1901+.fi
1902+.PP
1903+\f[C]${USERDATA}\f[R] may also be used; this will always resolve to the
1904+user data directory that is current when the defaults file is parsed,
1905+regardless of the setting of the environment variable
1906+\f[C]USERDATA\f[R].
1907+.PP
1908+\f[C]${.}\f[R] will resolve to the directory containing the default file
1909+itself.
1910+This allows you to refer to resources contained in that directory:
1911+.IP
1912+.nf
1913+\f[C]
1914+epub-cover-image: ${.}/cover.jpg
1915+epub-metadata: ${.}/meta.xml
1916+resource-path:
1917+- . # the working directory from which pandoc is run
1918+- ${.}/images # the images subdirectory of the directory
1919+ # containing this defaults file
1920+\f[R]
1921+.fi
1922+.PP
1923+This environment variable interpolation syntax \f[I]only\f[R] works in
1924+fields that expect file paths.
1925+.PP
18761926Default files can be placed in the \f[C]defaults\f[R] subdirectory of
18771927the user data directory and used from any directory.
18781928For example, one could create a file specifying defaults for writing
@@ -2132,9 +2182,9 @@ ${ styles.html() }
21322182\f[R]
21332183.fi
21342184.PP
2135−(If a partial is not found in the directory of the template, it will
2136−also be sought in the \f[C]templates\f[R] subdirectory of the user data
2137−directory.)
2185+(If a partial is not found in the directory of the template and the
2186+template path is given as a relative path, it will also be sought in the
2187+\f[C]templates\f[R] subdirectory of the user data directory.)
21382188.PP
21392189Partials may optionally be applied to variables using a colon:
21402190.IP
@@ -2541,10 +2591,9 @@ YAML metadata or with \f[C]-M classoption=fleqn\f[R].
25412591.SS Variables for HTML slides
25422592.PP
25432593These affect HTML output when [producing slide shows with pandoc].
2544−.PP
2545−All reveal.js configuration options are available as variables.
2546−To turn off boolean flags that default to true in reveal.js, use
2547−\f[C]0\f[R].
2594+.TP
2595+\f[B]\f[CB]institute\f[B]\f[R]
2596+author affiliations: can be a list when there are multiple authors
25482597.TP
25492598\f[B]\f[CB]revealjs-url\f[B]\f[R]
25502599base URL for reveal.js documents (defaults to
@@ -2563,6 +2612,10 @@ base URL for Slideous documents (defaults to \f[C]slideous\f[R])
25632612\f[B]\f[CB]title-slide-attributes\f[B]\f[R]
25642613additional attributes for the title slide of reveal.js slide shows.
25652614See background in reveal.js and beamer for an example.
2615+.PP
2616+All reveal.js configuration options are available as variables.
2617+To turn off boolean flags that default to true in reveal.js, use
2618+\f[C]0\f[R].
25662619.SS Variables for Beamer slides
25672620.PP
25682621These variables change the appearance of PDF slides using
@@ -3437,6 +3490,12 @@ output format.
34373490.PP
34383491Some aspects of Pandoc\[cq]s Markdown citation syntax are also accepted
34393492in \f[C]org\f[R] input.
3493+.SS Extension: \f[C]element_citations\f[R]
3494+.PP
3495+In the \f[C]jats\f[R] output formats, this causes reference items to be
3496+replaced with \f[C]<element-citation>\f[R] elements.
3497+These elements are not influenced by CSL styles, but all information on
3498+the item is included in tags.
34403499.SS Extension: \f[C]ntb\f[R]
34413500.PP
34423501In the \f[C]context\f[R] output format this enables the use of Natural
@@ -3920,6 +3979,9 @@ begin with a space.
39203979\f[R]
39213980.fi
39223981.PP
3982+Inline formatting (such as emphasis) is allowed in the content, but not
3983+block-level formatting (such as block quotes or lists).
3984+.PP
39233985This syntax is borrowed from reStructuredText.
39243986.SS Lists
39253987.SS Bullet lists
@@ -4213,7 +4275,7 @@ Term 2
42134275Note that space between items in a definition list is required.
42144276(A variant that loosens this requirement, but disallows \[lq]lazy\[rq]
42154277hard wrapping, can be activated with \f[C]compact_definition_lists\f[R]:
4216−see Non-pandoc extensions, below.)
4278+see Non-default extensions, below.)
42174279.SS Numbered example lists
42184280.SS Extension: \f[C]example_lists\f[R]
42194281.PP
@@ -4734,7 +4796,9 @@ All of the metadata will appear in a single block at the beginning of
47344796the document.
47354797.PP
47364798Note that YAML escaping rules must be followed.
4737−Thus, for example, if a title contains a colon, it must be quoted.
4799+Thus, for example, if a title contains a colon, it must be quoted, and
4800+if it contains a backslash escape, then it must be ensured that it is
4801+not treated as a YAML escape sequence.
47384802The pipe character (\f[C]|\f[R]) can be used to begin an indented block
47394803that will be interpreted literally, without need for escaping.
47404804This form is necessary when the field contains blank lines or
@@ -4756,6 +4820,13 @@ abstract: |
47564820\f[R]
47574821.fi
47584822.PP
4823+The literal block after the \f[C]|\f[R] must be indented relative to the
4824+line containing the \f[C]|\f[R].
4825+If it is not, the YAML will be invalid and pandoc will not interpret it
4826+as metadata.
4827+For an overview of the complex rules governing YAML, see the Wikipedia
4828+entry on YAML syntax.
4829+.PP
47594830Template variables will be set automatically from the metadata.
47604831Thus, for example, in writing HTML, the variable \f[C]abstract\f[R] will
47614832be set to the HTML equivalent of the Markdown in the \f[C]abstract\f[R]
@@ -4821,6 +4892,21 @@ header-includes:
48214892 \[ga]\[ga]\[ga]
48224893\f[R]
48234894.fi
4895+.PP
4896+Note: the \f[C]yaml_metadata_block\f[R] extension works with
4897+\f[C]commonmark\f[R] as well as \f[C]markdown\f[R] (and it is enabled by
4898+default in \f[C]gfm\f[R] and \f[C]commonmark_x\f[R]).
4899+However, in these formats the following restrictions apply:
4900+.IP \[bu] 2
4901+The YAML metadata block must occur at the beginning of the document (and
4902+there can be only one).
4903+If multiple files are given as arguments to pandoc, only the first can
4904+be a YAML metadata block.
4905+.IP \[bu] 2
4906+The leaf nodes of the YAML structure are parsed in isolation from each
4907+other and from the rest of the document.
4908+So, for example, you can\[cq]t use a reference link in these contexts if
4909+the link definition is somewhere else in the document.
48244910.SS Backslash escapes
48254911.SS Extension: \f[C]all_symbols_escapable\f[R]
48264912.PP
@@ -5040,7 +5126,9 @@ delimiters.
50405126.PP
50415127For display math, use \f[C]$$\f[R] delimiters.
50425128(In this case, the delimiters may be separated from the formula by
5043−whitespace.)
5129+whitespace.
5130+However, there can be no blank lines betwen the opening and closing
5131+\f[C]$$\f[R] delimiters.)
50445132.PP
50455133TeX math will be printed in all output formats.
50465134How it is rendered depends on the output format:
@@ -5166,8 +5254,9 @@ into
51665254.PP
51675255whereas \f[C]Markdown.pl\f[R] will preserve it as is.
51685256.PP
5169−There is one exception to this rule: text between \f[C]<script>\f[R] and
5170−\f[C]<style>\f[R] tags is not interpreted as Markdown.
5257+There is one exception to this rule: text between \f[C]<script>\f[R],
5258+\f[C]<style>\f[R], and \f[C]<textarea>\f[R] tags is not interpreted as
5259+Markdown.
51715260.PP
51725261This departure from standard Markdown should make it easier to mix
51735262Markdown with HTML block elements.
@@ -5698,27 +5787,70 @@ Inline and regular footnotes may be mixed freely.
56985787.SS Citation syntax
56995788.SS Extension: \f[C]citations\f[R]
57005789.PP
5701−Markdown citations go inside square brackets and are separated by
5702−semicolons.
5703−Each citation must have a key, composed of `\[at]' + the citation
5704−identifier from the database, and may optionally have a prefix, a
5705−locator, and a suffix.
5706−The citation key must begin with a letter, digit, or \f[C]_\f[R], and
5707−may contain alphanumerics, \f[C]_\f[R], and internal punctuation
5708−characters (\f[C]:.#$%&-+?<>\[ti]/\f[R]).
5709−Here are some examples:
5790+To cite a bibliographic item with an identifier foo, use the syntax
5791+\f[C]\[at]foo\f[R].
5792+Normal citations should be included in square brackets, with semicolons
5793+separating distinct items:
57105794.IP
57115795.nf
57125796\f[C]
5713−Blah blah [see \[at]doe99, pp. 33-35; also \[at]smith04, chap. 1].
5714−
5715−Blah blah [\[at]doe99, pp. 33-35, 38-39 and *passim*].
5797+Blah blah [\[at]doe99; \[at]smith2000; \[at]smith2004].
5798+\f[R]
5799+.fi
5800+.PP
5801+How this is rendered depends on the citation style.
5802+In an author-date style, it might render as
5803+.IP
5804+.nf
5805+\f[C]
5806+Blah blah (Doe 1999, Smith 2000, 2004).
5807+\f[R]
5808+.fi
5809+.PP
5810+In a footnote style, it might render as
5811+.IP
5812+.nf
5813+\f[C]
5814+Blah blah.[\[ha]1]
57165815
5717−Blah blah [\[at]smith04; \[at]doe99].
5816+[\[ha]1]: John Doe, \[dq]Frogs,\[dq] *Journal of Amphibians* 44 (1999);
5817+Susan Smith, \[dq]Flies,\[dq] *Journal of Insects* (2000);
5818+Susan Smith, \[dq]Bees,\[dq] *Journal of Insects* (2004).
57185819\f[R]
57195820.fi
57205821.PP
5721−\f[C]pandoc\f[R] detects locator terms in the CSL locale files.
5822+See the CSL user documentation for more information about CSL styles and
5823+how they affect rendering.
5824+.PP
5825+Unless a citation key start with a letter, digit, or \f[C]_\f[R], and
5826+contains only alphanumerics and internal punctuation characters
5827+(\f[C]:.#$%&-+?<>\[ti]/\f[R]), it must be surrounded by curly braces,
5828+which are not considered part of the key.
5829+In \f[C]\[at]Foo_bar.baz.\f[R], the key is \f[C]Foo_bar.baz\f[R].
5830+The final period is not \f[I]internal\f[R] punctuation, so it is not
5831+included in the key.
5832+In \f[C]\[at]{Foo_bar.baz.}\f[R], the key is \f[C]Foo_bar.baz.\f[R],
5833+including the final period.
5834+The curly braces are recommended if you use URLs as keys:
5835+\f[C][\[at]{https://example.com/bib?name=foobar&date=2000}, p. 33]\f[R].
5836+.PP
5837+Citation items may optionally include a prefix, a locator, and a suffix.
5838+In
5839+.IP
5840+.nf
5841+\f[C]
5842+Blah blah [see \[at]doe99, pp. 33-35 and *passim*; \[at]smith04, chap. 1].
5843+\f[R]
5844+.fi
5845+.PP
5846+The first item (\f[C]doe99\f[R]) has prefix \f[C]see\f[R], locator
5847+\f[C]pp. 33-35\f[R], and suffix \f[C]and *passim*\f[R].
5848+The second item (\f[C]smith04\f[R]) has locator \f[C]chap. 1\f[R] and no
5849+prefix or suffix.
5850+.PP
5851+Pandoc uses some heuristics to separate the locator from the rest of the
5852+subject.
5853+It is sensitive to the locator terms defined in the CSL locale files.
57225854Either abbreviated or unabbreviated forms are accepted.
57235855In the \f[C]en-US\f[R] locale, locator terms can be written in either
57245856singular or plural forms, as \f[C]book\f[R],
@@ -5740,14 +5872,15 @@ singular or plural forms, as \f[C]book\f[R],
57405872\f[C]\[sc]\f[R]/\f[C]\[sc]\[sc]\f[R].
57415873If no locator term is used, \[lq]page\[rq] is assumed.
57425874.PP
5743−\f[C]pandoc\f[R] will use heuristics to distinguish the locator from the
5744−suffix.
5745−In complex cases, the locator can be enclosed in curly braces:
5875+In complex cases, you can force something to be treated as a locator by
5876+enclosing it in curly braces or prevent parsing the suffix as locator by
5877+prepending curly braces:
57465878.IP
57475879.nf
57485880\f[C]
57495881[\[at]smith{ii, A, D-Z}, with a suffix]
57505882[\[at]smith, {pp. iv, vi-xi, (xv)-(xvii)} with suffix here]
5883+[\[at]smith{}, 99 years later]
57515884\f[R]
57525885.fi
57535886.PP
@@ -5761,7 +5894,8 @@ Smith says blah [-\[at]smith04].
57615894\f[R]
57625895.fi
57635896.PP
5764−You can also write an in-text citation, as follows:
5897+You can also write an author-in-text citation, by omitting the square
5898+brackets:
57655899.IP
57665900.nf
57675901\f[C]
@@ -5770,13 +5904,64 @@ You can also write an in-text citation, as follows:
57705904\[at]smith04 [p. 33] says blah.
57715905\f[R]
57725906.fi
5773−.SS Non-pandoc extensions
5907+.PP
5908+This will cause the author\[cq]s name to be rendered, followed by the
5909+bibliographical details.
5910+Use this form when you want to make the citation the subject of a
5911+sentence.
5912+.PP
5913+When you are using a note style, it is usually better to let citeproc
5914+create the footnotes from citations rather than writing an explicit
5915+note.
5916+If you do write an explicit note that contains a citation, note that
5917+normal citations will be put in parentheses, while author-in-text
5918+citations will not.
5919+For this reason, it is sometimes preferable to use the author-in-text
5920+style inside notes when using a note style.
5921+.SS Non-default extensions
57745922.PP
57755923The following Markdown syntax extensions are not enabled by default in
57765924pandoc, but may be enabled by adding \f[C]+EXTENSION\f[R] to the format
57775925name, where \f[C]EXTENSION\f[R] is the name of the extension.
57785926Thus, for example, \f[C]markdown+hard_line_breaks\f[R] is Markdown with
57795927hard line breaks.
5928+.SS Extension: \f[C]rebase_relative_paths\f[R]
5929+.PP
5930+Rewrite relative paths for Markdown links and images, depending on the
5931+path of the file containing the link or image link.
5932+For each link or image, pandoc will compute the directory of the
5933+containing file, relative to the working directory, and prepend the
5934+resulting path to the link or image path.
5935+.PP
5936+The use of this extension is best understood by example.
5937+Suppose you have a a subdirectory for each chapter of a book,
5938+\f[C]chap1\f[R], \f[C]chap2\f[R], \f[C]chap3\f[R].
5939+Each contains a file \f[C]text.md\f[R] and a number of images used in
5940+the chapter.
5941+You would like to have \f[C]![image](spider.jpg)\f[R] in
5942+\f[C]chap1/text.md\f[R] refer to \f[C]chap1/spider.jpg\f[R] and
5943+\f[C]![image](spider.jpg)\f[R] in \f[C]chap2/text.md\f[R] refer to
5944+\f[C]chap2/spider.jpg\f[R].
5945+To do this, use
5946+.IP
5947+.nf
5948+\f[C]
5949+pandoc chap*/*.md -f markdown+rebase_relative_paths
5950+\f[R]
5951+.fi
5952+.PP
5953+Without this extension, you would have to use
5954+\f[C]![image](chap1/spider.jpg)\f[R] in \f[C]chap1/text.md\f[R] and
5955+\f[C]![image](chap2/spider.jpg)\f[R] in \f[C]chap2/text.md\f[R].
5956+Links with relative paths will be rewritten in the same way as images.
5957+.PP
5958+Absolute paths and URLs are not changed.
5959+Neither are empty paths or paths consisting entirely of a fragment,
5960+e.g., \f[C]#foo\f[R].
5961+.PP
5962+Note that relative paths in reference links and images will be rewritten
5963+relative to the file containing the link reference definition, not the
5964+file containing the reference link or image itself, if these differ.
57805965.SS Extension: \f[C]attributes\f[R]
57815966.PP
57825967Allows attributes to be attached to any inline or block-level element.
@@ -6136,13 +6321,13 @@ pandoc chem.json -s -f csljson -t markdown
61366321\f[R]
61376322.fi
61386323.PP
6139−\f[C]pandoc\f[R] can also be used to produce CSL JSON bibliography from
6140−BibTeX, BibLaTeX, or markdown YAML:
6324+Indeed, \f[C]pandoc\f[R] can convert between any of these citation
6325+formats:
61416326.IP
61426327.nf
61436328\f[C]
61446329pandoc chem.bib -s -f biblatex -t csljson
6145−pandoc chem.yaml -s -f markdown -t csljson
6330+pandoc chem.yaml -s -f markdown -t biblatex
61466331\f[R]
61476332.fi
61486333.PP
@@ -6271,6 +6456,41 @@ The format of the file can be illustrated with an example:
62716456}
62726457\f[R]
62736458.fi
6459+.SS Citations in note styles
6460+.PP
6461+Pandoc\[cq]s citation processing is designed to allow you to move
6462+between author-date, numerical, and note styles without modifying the
6463+markdown source.
6464+When you\[cq]re using a note style, avoid inserting footnotes manually.
6465+Instead, insert citations just as you would in an author-date
6466+style\[em]for example,
6467+.IP
6468+.nf
6469+\f[C]
6470+Blah blah [\[at]foo, p. 33].
6471+\f[R]
6472+.fi
6473+.PP
6474+The footnote will be created automatically.
6475+Pandoc will take care of removing the space and moving the note before
6476+or after the period, depending on the setting of
6477+\f[C]notes-after-punctuation\f[R], as described below in Other relevant
6478+metadata fields.
6479+.PP
6480+In some cases you may need to put a citation inside a regular footnote.
6481+Normal citations in footnotes (such as \f[C][\[at]foo, p. 33]\f[R]) will
6482+be rendered in parentheses.
6483+In-text citations (such as \f[C]\[at]foo [p. 33]\f[R]) will be rendered
6484+without parentheses.
6485+(A comma will be added if appropriate.) Thus:
6486+.IP
6487+.nf
6488+\f[C]
6489+[\[ha]1]: Some studies [\[at]foo; \[at]bar, p. 33] show that
6490+frubulicious zoosnaps are quantical. For a survey
6491+of the literature, see \[at]baz [chap. 1].
6492+\f[R]
6493+.fi
62746494.SS Raw content in a style
62756495.PP
62766496To include raw content in a prefix, suffix, delimiter, or term, surround
@@ -6371,9 +6591,29 @@ entries (for author-date and numerical styles only).
63716591.TP
63726592\f[B]\f[CB]lang\f[B]\f[R]
63736593The \f[C]lang\f[R] field will affect how the style is localized, for
6374−example in the translation of labels and the use of quotation marks.
6594+example in the translation of labels, the use of quotation marks, and
6595+the way items are sorted.
63756596(For backwards compatibility, \f[C]locale\f[R] may be used instead of
63766597\f[C]lang\f[R], but this use is deprecated.)
6598+.RS
6599+.PP
6600+A BCP 47 language tag is expected: for example, \f[C]en\f[R],
6601+\f[C]de\f[R], \f[C]en-US\f[R], \f[C]fr-CA\f[R], \f[C]ug-Cyrl\f[R].
6602+The unicode extension syntax (after \f[C]-u-\f[R]) may be used to
6603+specify options for collation (sorting) more precisely.
6604+Here are some examples:
6605+.IP \[bu] 2
6606+\f[C]zh-u-co-pinyin\f[R] \[en] Chinese with the Pinyin collation.
6607+.IP \[bu] 2
6608+\f[C]es-u-co-trad\f[R] \[en] Spanish with the traditional collation
6609+(with \f[C]Ch\f[R] sorting after \f[C]C\f[R]).
6610+.IP \[bu] 2
6611+\f[C]fr-u-kb\f[R] \[en] French with \[lq]backwards\[rq] accent sorting
6612+(with \f[C]cot\['e]\f[R] sorting after \f[C]c\[^o]te\f[R]).
6613+.IP \[bu] 2
6614+\f[C]en-US-u-kf-upper\f[R] \[en] English with uppercase letters sorting
6615+before lower (default is lower before upper).
6616+.RE
63776617.TP
63786618\f[B]\f[CB]notes-after-punctuation\f[B]\f[R]
63796619If true (the default), pandoc will put footnote citations after
@@ -6494,6 +6734,11 @@ A heading at the slide level always starts a new slide.
64946734.IP \[bu] 2
64956735Headings \f[I]below\f[R] the slide level in the hierarchy create
64966736headings \f[I]within\f[R] a slide.
6737+(In beamer, a \[lq]block\[rq] will be created.
6738+If the heading has the class \f[C]example\f[R], an
6739+\f[C]exampleblock\f[R] environment will be used; if it has the class
6740+\f[C]alert\f[R], an \f[C]alertblock\f[R] will be used; otherwise a
6741+regular \f[C]block\f[R] will be used.)
64976742.IP \[bu] 2
64986743Headings \f[I]above\f[R] the slide level in the hierarchy create
64996744\[lq]title slides,\[rq] which just contain the section title and help to
@@ -6894,6 +7139,16 @@ A string value.
68947139\f[B]\f[CB]rights\f[B]\f[R]
68957140A string value.
68967141.TP
7142+\f[B]\f[CB]belongs-to-collection\f[B]\f[R]
7143+A string value.
7144+identifies the name of a collection to which the EPUB Publication
7145+belongs.
7146+.TP
7147+\f[B]\f[CB]group-position\f[B]\f[R]
7148+The \f[C]group-position\f[R] field indicates the numeric position in
7149+which the EPUB Publication belongs relative to other works belonging to
7150+the same \f[C]belongs-to-collection\f[R] field.
7151+.TP
68977152\f[B]\f[CB]cover-image\f[B]\f[R]
68987153A string value (path to cover image).
68997154.TP
@@ -7409,6 +7664,20 @@ need to specify a template manually using \f[C]--template\f[R] or add a
74097664new default template with the name
74107665\f[C]default.NAME_OF_CUSTOM_WRITER.lua\f[R] to the \f[C]templates\f[R]
74117666subdirectory of your user data directory (see Templates).
7667+.SH REPRODUCIBLE BUILDS
7668+.PP
7669+Some of the document formats pandoc targets (such as EPUB, docx, and
7670+ODT) include build timestamps in the generated document.
7671+That means that the files generated on successive builds will differ,
7672+even if the source does not.
7673+To avoid this, set the \f[C]SOURCE_DATE_EPOCH\f[R] environment variable,
7674+and the timestamp will be taken from it instead of the current time.
7675+\f[C]SOURCE_DATE_EPOCH\f[R] should contain an integer unix timestamp
7676+(specifying the number of second since midnight UTC January 1, 1970).
7677+.PP
7678+Some document formats also include a unique identifier.
7679+For EPUB, this can be set explicitly by setting the \f[C]identifier\f[R]
7680+metadata field (see EPUB Metadata, above).
74127681.SH A NOTE ON SECURITY
74137682.PP
74147683If you use pandoc to convert user-contributed content in a web
@@ -7439,16 +7708,16 @@ The HTML generated by pandoc is not guaranteed to be safe.
74397708If \f[C]raw_html\f[R] is enabled for the Markdown input, users can
74407709inject arbitrary HTML.
74417710Even if \f[C]raw_html\f[R] is disabled, users can include dangerous
7442−content in attributes for headings, spans, and code blocks.
7711+content in URLs and attributes.
74437712To be safe, you should run all the generated HTML through an HTML
74447713sanitizer.
74457714.SH AUTHORS
74467715.PP
7447−Copyright 2006\[en]2020 John MacFarlane (jgm\[at]berkeley.edu).
7716+Copyright 2006\[en]2021 John MacFarlane (jgm\[at]berkeley.edu).
74487717Released under the GPL, version 2 or greater.
74497718This software carries no warranty of any kind.
74507719(See COPYRIGHT for full copyright and warranty notices.) For a full list
74517720of contributors, see the file AUTHORS.md in the pandoc source code.
74527721.PP
74537722The Pandoc source code and all documentation may be downloaded
7454−from <http://pandoc.org>.
7723+from <https://pandoc.org>.
man/pandoc.1.after+1−1
@@ -1,3 +1,3 @@
11.PP
22The Pandoc source code and all documentation may be downloaded
3−from <http://pandoc.org>.
3+from <https://pandoc.org>.
pandoc.cabal+110−161
@@ -1,10 +1,10 @@
11cabal-version: 2.2
22name: pandoc
3−version: 2.11.3.1
3+version: 2.14.0.3
44build-type: Simple
55license: GPL-2.0-or-later
66license-file: COPYING.md
7−copyright: (c) 2006-2020 John MacFarlane
7+copyright: (c) 2006-2021 John MacFarlane
88author: John MacFarlane <jgm@berkeley.edu>
99maintainer: John MacFarlane <jgm@berkeley.edu>
1010bug-reports: https://github.com/jgm/pandoc/issues
@@ -12,7 +12,7 @@ stability: alpha
1212homepage: https://pandoc.org
1313category: Text
1414tested-with: GHC == 8.2.2, GHC == 8.4.4, GHC == 8.6.5,
15− GHC == 8.8.4, GHC == 8.10.2
15+ GHC == 8.8.4, GHC == 8.10.2, GHC == 9.0.1
1616synopsis: Conversion between markup formats
1717description: Pandoc is a Haskell library for converting from one markup
1818 format to another, and a command-line tool that uses
@@ -56,6 +56,8 @@ data-files:
5656 data/templates/default.icml
5757 data/templates/default.opml
5858 data/templates/default.latex
59+ data/templates/default.bibtex
60+ data/templates/default.biblatex
5961 data/templates/default.context
6062 data/templates/default.texinfo
6163 data/templates/default.jira
@@ -180,8 +182,6 @@ data-files:
180182 data/pandoc.List.lua
181183 -- bash completion template
182184 data/bash_completion.tpl
183− -- jats csl
184− data/jats.csl
185185 -- citeproc
186186 data/default.csl
187187 citeproc/biblatex-localization/*.lbx.strings
@@ -208,14 +208,25 @@ extra-source-files:
208208 test/command/*.md
209209 test/command/*.csl
210210 test/command/biblio.bib
211− test/command/biblatex-examples.bib
211+ test/command/averroes.bib
212212 test/command/A.txt
213213 test/command/B.txt
214214 test/command/C.txt
215215 test/command/D.txt
216216 test/command/01.csv
217+ test/command/chap1/spider.png
218+ test/command/chap2/spider.png
219+ test/command/chap1/text.md
220+ test/command/chap2/text.md
217221 test/command/defaults1.yaml
218222 test/command/defaults2.yaml
223+ test/command/defaults3.yaml
224+ test/command/defaults4.yaml
225+ test/command/defaults5.yaml
226+ test/command/defaults6.yaml
227+ test/command/defaults7.yaml
228+ test/command/defaults8.yaml
229+ test/command/defaults9.yaml
219230 test/command/3533-rst-csv-tables.csv
220231 test/command/3880.txt
221232 test/command/5182.txt
@@ -389,103 +400,116 @@ flag trypandoc
389400 Description: Build trypandoc cgi executable.
390401 Default: False
391402
403+common common-options
404+ default-language: Haskell2010
405+ build-depends: base >= 4.9 && < 5
406+ ghc-options: -Wall -fno-warn-unused-do-bind
407+ -Wincomplete-record-updates
408+ -Wnoncanonical-monad-instances
409+
410+ if impl(ghc < 8.4)
411+ hs-source-dirs: prelude
412+ other-modules: Prelude
413+ build-depends: base-compat >= 0.9
414+ other-extensions: NoImplicitPrelude
415+
416+ if os(windows)
417+ cpp-options: -D_WINDOWS
418+
419+ -- Later:
420+ -- -Wpartial-fields (currently used in Powerpoint writer)
421+ -- -Wmissing-export-lists (currently some Odt modules violate this)
422+ -- -Wredundant-constraints (problematic if we want to support older base)
423+ if impl(ghc >= 8.2)
424+ ghc-options: -Wcpp-undef
425+ if impl(ghc >= 8.4)
426+ ghc-options: -Wincomplete-uni-patterns
427+ -Widentities
428+ -fhide-source-paths
429+
430+common common-executable
431+ import: common-options
432+ build-depends: pandoc
433+ ghc-options: -rtsopts -with-rtsopts=-A8m -threaded
434+
435+
392436library
393− build-depends: base >= 4.9 && < 5,
394− Glob >= 0.7 && < 0.11,
437+ import: common-options
438+ build-depends: Glob >= 0.7 && < 0.11,
395439 HTTP >= 4000.0.5 && < 4000.4,
396440 HsYAML >= 0.2 && < 0.3,
397441 JuicyPixels >= 3.1.6.1 && < 3.4,
398442 SHA >= 1.6 && < 1.7,
399443 aeson >= 0.7 && < 1.6,
400444 aeson-pretty >= 0.8.5 && < 0.9,
401− attoparsec >= 0.12 && < 0.14,
402− base64-bytestring >= 0.1 && < 1.2,
445+ array >= 0.5 && < 0.6,
446+ attoparsec >= 0.12 && < 0.15,
447+ base64-bytestring >= 0.1 && < 1.3,
403448 binary >= 0.7 && < 0.11,
404449 blaze-html >= 0.9 && < 0.10,
405450 blaze-markup >= 0.8 && < 0.9,
406451 bytestring >= 0.9 && < 0.12,
407452 case-insensitive >= 1.2 && < 1.3,
408− citeproc >= 0.3.0.1 && < 0.4,
409− commonmark >= 0.1.1.2 && < 0.2,
410− commonmark-extensions >= 0.2.0.4 && < 0.3,
411− commonmark-pandoc >= 0.2 && < 0.3,
453+ citeproc >= 0.4.0.1 && < 0.4.1,
454+ commonmark >= 0.2.1 && < 0.3,
455+ commonmark-extensions >= 0.2.1.2 && < 0.3,
456+ commonmark-pandoc >= 0.2.1.1 && < 0.3,
412457 connection >= 0.3.1,
413458 containers >= 0.4.2.1 && < 0.7,
414459 data-default >= 0.4 && < 0.8,
415460 deepseq >= 1.3 && < 1.5,
416461 directory >= 1.2.3 && < 1.4,
417− doclayout >= 0.3 && < 0.4,
462+ doclayout >= 0.3.0.1 && < 0.4,
418463 doctemplates >= 0.9 && < 0.10,
419464 emojis >= 0.1 && < 0.2,
420465 exceptions >= 0.8 && < 0.11,
421466 file-embed >= 0.0 && < 0.1,
422467 filepath >= 1.1 && < 1.5,
423− haddock-library >= 1.8 && < 1.10,
468+ haddock-library >= 1.10 && < 1.11,
424469 hslua >= 1.1 && < 1.4,
470+ hslua-module-path >= 0.1.0 && < 0.2.0,
425471 hslua-module-system >= 0.2 && < 0.3,
426472 hslua-module-text >= 0.2.1 && < 0.4,
427473 http-client >= 0.4.30 && < 0.8,
428474 http-client-tls >= 0.2.4 && < 0.4,
429475 http-types >= 0.8 && < 0.13,
430476 ipynb >= 0.1 && < 0.2,
431− jira-wiki-markup >= 1.3.2 && < 1.4,
477+ jira-wiki-markup >= 1.4 && < 1.5,
432478 mtl >= 2.2 && < 2.3,
433479 network >= 2.6,
434− network-uri >= 2.6 && < 2.7,
480+ network-uri >= 2.6 && < 2.8,
435481 pandoc-types >= 1.22 && < 1.23,
436482 parsec >= 3.1 && < 3.2,
437483 process >= 1.2.3 && < 1.7,
438− random >= 1 && < 1.2,
439− safe >= 0.3 && < 0.4,
484+ random >= 1 && < 1.3,
485+ safe >= 0.3.18 && < 0.4,
440486 scientific >= 0.3 && < 0.4,
441− skylighting >= 0.10.2 && < 0.11,
442− skylighting-core >= 0.10.2 && < 0.11,
487+ skylighting >= 0.10.5.2 && < 0.10.6,
488+ skylighting-core >= 0.10.5.2 && < 0.10.6,
443489 split >= 0.2 && < 0.3,
444490 syb >= 0.1 && < 0.8,
445491 tagsoup >= 0.14.6 && < 0.15,
446492 temporary >= 1.1 && < 1.4,
447− texmath >= 0.12.1 && < 0.13,
493+ texmath >= 0.12.3 && < 0.12.4,
448494 text >= 1.1.1.0 && < 1.3,
449495 text-conversions >= 0.3 && < 0.4,
450− time >= 1.5 && < 1.10,
496+ time >= 1.5 && < 1.12,
451497 unicode-transforms >= 0.3 && < 0.4,
452498 unordered-containers >= 0.2 && < 0.3,
453499 xml >= 1.3.12 && < 1.4,
500+ xml-conduit >= 1.9.1.1 && < 1.10,
501+ unicode-collation >= 0.1.1 && < 0.2,
454502 zip-archive >= 0.2.3.4 && < 0.5,
455503 zlib >= 0.5 && < 0.7
456504 if os(windows) && arch(i386)
457505 build-depends: basement >= 0.0.10,
458506 foundation >= 0.0.23
459507 -- basement 0.0.9 won't build on 32-bit windows.
460− if impl(ghc < 8.4)
461− hs-source-dirs: prelude
462− other-modules: Prelude
463− build-depends: base-compat >= 0.9
464− if os(windows)
465− cpp-options: -D_WINDOWS
466− else
508+ if !os(windows)
467509 build-depends: unix >= 2.4 && < 2.8
468510 if flag(embed_data_files)
469511 cpp-options: -DEMBED_DATA_FILES
470512 other-modules: Text.Pandoc.Data
471− if os(windows)
472− cpp-options: -D_WINDOWS
473− ghc-options: -Wall -fno-warn-unused-do-bind
474− -Wincomplete-record-updates
475− -Wnoncanonical-monad-instances
476− -- Later:
477− -- -Wpartial-fields (currently used in Powerpoint writer)
478− -- -Wmissing-export-lists (currently some Odt modules violate this)
479− -- -Wredundant-constraints (problematic if we want to support older base)
480− if impl(ghc >= 8.2)
481− ghc-options: -Wcpp-undef
482− if impl(ghc >= 8.4)
483− ghc-options: -Wincomplete-uni-patterns
484− -Widentities
485− -fhide-source-paths
486−
487− default-language: Haskell2010
488− other-extensions: NoImplicitPrelude
489513 hs-source-dirs: src
490514
491515 exposed-modules: Text.Pandoc,
@@ -493,13 +517,13 @@ library
493517 Text.Pandoc.Options,
494518 Text.Pandoc.Extensions,
495519 Text.Pandoc.Shared,
520+ Text.Pandoc.Sources,
496521 Text.Pandoc.MediaBag,
497522 Text.Pandoc.Error,
498523 Text.Pandoc.Filter,
499524 Text.Pandoc.Readers,
500525 Text.Pandoc.Readers.HTML,
501526 Text.Pandoc.Readers.LaTeX,
502− Text.Pandoc.Readers.LaTeX.Types,
503527 Text.Pandoc.Readers.Markdown,
504528 Text.Pandoc.Readers.CommonMark,
505529 Text.Pandoc.Readers.Creole,
@@ -568,6 +592,7 @@ library
568592 Text.Pandoc.Writers.Shared,
569593 Text.Pandoc.Writers.OOXML,
570594 Text.Pandoc.Writers.AnnotatedTable,
595+ Text.Pandoc.Writers.BibTeX,
571596 Text.Pandoc.Lua,
572597 Text.Pandoc.PDF,
573598 Text.Pandoc.UTF8,
@@ -582,7 +607,6 @@ library
582607 Text.Pandoc.Asciify,
583608 Text.Pandoc.Emoji,
584609 Text.Pandoc.ImageSize,
585− Text.Pandoc.BCP47,
586610 Text.Pandoc.Class,
587611 Text.Pandoc.Citeproc
588612 other-modules: Text.Pandoc.App.CommandLineOptions,
@@ -607,9 +631,15 @@ library
607631 Text.Pandoc.Readers.HTML.Table,
608632 Text.Pandoc.Readers.HTML.TagCategories,
609633 Text.Pandoc.Readers.HTML.Types,
610− Text.Pandoc.Readers.LaTeX.Parsing,
634+ Text.Pandoc.Readers.LaTeX.Inline,
635+ Text.Pandoc.Readers.LaTeX.Citation,
611636 Text.Pandoc.Readers.LaTeX.Lang,
637+ Text.Pandoc.Readers.LaTeX.Macro,
638+ Text.Pandoc.Readers.LaTeX.Math,
639+ Text.Pandoc.Readers.LaTeX.Parsing,
612640 Text.Pandoc.Readers.LaTeX.SIunitx,
641+ Text.Pandoc.Readers.LaTeX.Table,
642+ Text.Pandoc.Readers.LaTeX.Types,
613643 Text.Pandoc.Readers.Odt.Base,
614644 Text.Pandoc.Readers.Odt.Namespaces,
615645 Text.Pandoc.Readers.Odt.StyleReader,
@@ -633,12 +663,21 @@ library
633663 Text.Pandoc.Readers.Metadata,
634664 Text.Pandoc.Readers.Roff,
635665 Text.Pandoc.Writers.Docx.StyleMap,
666+ Text.Pandoc.Writers.Docx.Table,
667+ Text.Pandoc.Writers.Docx.Types,
668+ Text.Pandoc.Writers.GridTable
669+ Text.Pandoc.Writers.JATS.References,
636670 Text.Pandoc.Writers.JATS.Table,
637671 Text.Pandoc.Writers.JATS.Types,
638672 Text.Pandoc.Writers.LaTeX.Caption,
639673 Text.Pandoc.Writers.LaTeX.Notes,
640674 Text.Pandoc.Writers.LaTeX.Table,
675+ Text.Pandoc.Writers.LaTeX.Lang,
641676 Text.Pandoc.Writers.LaTeX.Types,
677+ Text.Pandoc.Writers.LaTeX.Citation,
678+ Text.Pandoc.Writers.LaTeX.Util,
679+ Text.Pandoc.Writers.Markdown.Types,
680+ Text.Pandoc.Writers.Markdown.Inline,
642681 Text.Pandoc.Writers.Roff,
643682 Text.Pandoc.Writers.Powerpoint.Presentation,
644683 Text.Pandoc.Writers.Powerpoint.Output,
@@ -666,6 +705,10 @@ library
666705 Text.Pandoc.Lua.PandocLua,
667706 Text.Pandoc.Lua.Util,
668707 Text.Pandoc.Lua.Walk,
708+ Text.Pandoc.XML.Light,
709+ Text.Pandoc.XML.Light.Types,
710+ Text.Pandoc.XML.Light.Proc,
711+ Text.Pandoc.XML.Light.Output,
669712 Text.Pandoc.CSS,
670713 Text.Pandoc.CSV,
671714 Text.Pandoc.RoffChar,
@@ -684,116 +727,50 @@ library
684727 buildable: True
685728
686729executable pandoc
687− build-depends: pandoc, base >= 4.8 && < 5
688− if impl(ghc < 8.4)
689− hs-source-dirs: prelude
690− other-modules: Prelude
691− build-depends: base-compat >= 0.9
692− default-language: Haskell2010
693− other-extensions: NoImplicitPrelude
730+ import: common-executable
694731 hs-source-dirs: app
695732 main-is: pandoc.hs
696733 buildable: True
697734 other-modules: Paths_pandoc
698− ghc-options: -rtsopts -with-rtsopts=-K16m -threaded
699− -Wall -fno-warn-unused-do-bind
700− -Wincomplete-record-updates
701− -Wnoncanonical-monad-instances
702− if impl(ghc >= 8.2)
703− ghc-options: -Wcpp-undef
704− if impl(ghc >= 8.4)
705− ghc-options: -Wincomplete-uni-patterns
706− -Widentities
707− -fhide-source-paths
708735
709736executable trypandoc
737+ import: common-executable
710738 main-is: trypandoc.hs
711739 hs-source-dirs: trypandoc
712− default-language: Haskell2010
713− other-extensions: NoImplicitPrelude
714740 if flag(trypandoc)
715− build-depends: base, aeson, pandoc,
716− text, wai-extra, wai >= 0.3, http-types
741+ build-depends: aeson, http-types, text, wai >= 0.3, wai-extra >= 3.0.24
717742 buildable: True
718743 else
719744 buildable: False
720− if impl(ghc < 8.4)
721− hs-source-dirs: prelude
722− other-modules: Prelude
723− build-depends: base-compat >= 0.9
724− ghc-options: -rtsopts -with-rtsopts=-K16m -threaded
725− -Wall -fno-warn-unused-do-bind
726− -Wincomplete-record-updates
727− -Wnoncanonical-monad-instances
728− if impl(ghc >= 8.2)
729− ghc-options: -Wcpp-undef
730− if impl(ghc >= 8.4)
731− ghc-options: -Wincomplete-uni-patterns
732− -Widentities
733− -fhide-source-paths
734−
735−benchmark weigh-pandoc
736− type: exitcode-stdio-1.0
737− main-is: weigh-pandoc.hs
738− hs-source-dirs: benchmark
739− build-depends: pandoc,
740− base >= 4.8 && < 5,
741− mtl >= 2.2 && < 2.3,
742− text,
743− weigh >= 0.0 && < 0.1,
744− if impl(ghc < 8.4)
745− hs-source-dirs: prelude
746− other-modules: Prelude
747− build-depends: base-compat >= 0.9
748− default-language: Haskell2010
749− other-extensions: NoImplicitPrelude
750− ghc-options: -rtsopts -with-rtsopts=-K16m -threaded
751− -Wall -fno-warn-unused-do-bind
752− -Wincomplete-record-updates
753− -Wnoncanonical-monad-instances
754− if impl(ghc >= 8.2)
755− ghc-options: -Wcpp-undef
756− if impl(ghc >= 8.4)
757− ghc-options: -Wincomplete-uni-patterns
758− -Widentities
759− -fhide-source-paths
760−
761745
762746test-suite test-pandoc
747+ import: common-executable
763748 type: exitcode-stdio-1.0
764749 main-is: test-pandoc.hs
765750 hs-source-dirs: test
766751 build-depends: pandoc,
767− base >= 4.8 && < 5,
768752 Diff >= 0.2 && < 0.5,
769753 Glob >= 0.7 && < 0.11,
770754 QuickCheck >= 2.4 && < 2.15,
771− base64-bytestring >= 0.1 && < 1.2,
772755 bytestring >= 0.9 && < 0.12,
773756 containers >= 0.4.2.1 && < 0.7,
774757 directory >= 1.2.3 && < 1.4,
775758 doctemplates >= 0.9 && < 0.10,
776759 exceptions >= 0.8 && < 0.11,
777− executable-path >= 0.0 && < 0.1,
778760 filepath >= 1.1 && < 1.5,
779761 hslua >= 1.1 && < 1.4,
780762 mtl >= 2.2 && < 2.3,
781763 pandoc-types >= 1.22 && < 1.23,
782764 process >= 1.2.3 && < 1.7,
783− tasty >= 0.11 && < 1.4,
765+ tasty >= 0.11 && < 1.5,
784766 tasty-golden >= 2.3 && < 2.4,
785767 tasty-hunit >= 0.9 && < 0.11,
786768 tasty-lua >= 0.2 && < 0.3,
787769 tasty-quickcheck >= 0.8 && < 0.11,
788− temporary >= 1.1 && < 1.4,
789− text >= 1.1.1.0 && < 1.3,
790− time >= 1.5 && < 1.10,
770+ text >= 1.1.1.0 && < 1.3,
771+ time >= 1.5 && < 1.12,
791772 xml >= 1.3.12 && < 1.4,
792773 zip-archive >= 0.2.3.4 && < 0.5
793− if impl(ghc < 8.4)
794− hs-source-dirs: prelude
795− other-modules: Prelude
796− build-depends: base-compat >= 0.9
797774 other-modules: Tests.Old
798775 Tests.Command
799776 Tests.Helpers
@@ -849,47 +826,19 @@ test-suite test-pandoc
849826 Tests.Writers.OOXML
850827 Tests.Writers.Ms
851828 Tests.Writers.AnnotatedTable
852− if os(windows)
853− cpp-options: -D_WINDOWS
854− default-language: Haskell2010
855− other-extensions: NoImplicitPrelude
856− ghc-options: -rtsopts -with-rtsopts=-K16m -threaded
857− -Wall -fno-warn-unused-do-bind
858− -Wincomplete-record-updates
859− -Wnoncanonical-monad-instances
860− if impl(ghc >= 8.2)
861− ghc-options: -Wcpp-undef
862− if impl(ghc >= 8.4)
863− ghc-options: -Wincomplete-uni-patterns
864− -Widentities
865− -fhide-source-paths
866−
867829
868830benchmark benchmark-pandoc
831+ import: common-executable
869832 type: exitcode-stdio-1.0
870833 main-is: benchmark-pandoc.hs
871834 hs-source-dirs: benchmark
872− build-depends: pandoc,
873− base >= 4.8 && < 5,
874− bytestring,
835+ build-depends: bytestring,
875836 containers,
876− criterion >= 1.0 && < 1.6,
837+ -- gauge >= 0.2 && < 0.3,
838+ tasty-bench >= 0.2 && <= 0.3,
877839 mtl >= 2.2 && < 2.3,
878840 text >= 1.1.1.0 && < 1.3,
879− time
880− if impl(ghc < 8.4)
881− hs-source-dirs: prelude
882− other-modules: Prelude
883− build-depends: base-compat >= 0.9
884− default-language: Haskell2010
885− other-extensions: NoImplicitPrelude
886− ghc-options: -rtsopts -with-rtsopts=-K16m -threaded
887− -Wall -fno-warn-unused-do-bind
888− -Wincomplete-record-updates
889− -Wnoncanonical-monad-instances
890− if impl(ghc >= 8.2)
891− ghc-options: -Wcpp-undef
892− if impl(ghc >= 8.4)
893− ghc-options: -Wincomplete-uni-patterns
894− -Widentities
895− -fhide-source-paths
841+ time,
842+ deepseq
843+ -- we increase heap size to avoid benchmarking garbage collection:
844+ ghc-options: -rtsopts -with-rtsopts=-A8m -threaded
release.nixnew file+4
@@ -0,0 +1,4 @@
1+let
2+ pkgs = import <nixpkgs> { };
3+in
4+ pkgs.haskellPackages.callPackage ./project.nix { }
shell.nixnew file+106
@@ -0,0 +1,106 @@
1+{nixpkgs ? import <nixpkgs> {} }:
2+let
3+ inherit (nixpkgs) pkgs;
4+ inherit (pkgs) haskellPackages;
5+
6+ haskellDeps = ps: with ps; [
7+ Diff
8+ Glob
9+ HTTP
10+ HTTP
11+ HsYAML
12+ JuicyPixels
13+ QuickCheck
14+ SHA
15+ aeson
16+ aeson-pretty
17+ attoparsec
18+ base
19+ base-compat
20+ base64-bytestring
21+ binary
22+ blaze-html
23+ blaze-markup
24+ bytestring
25+ case-insensitive
26+ citeproc
27+ commonmark
28+ commonmark-extensions
29+ commonmark-pandoc
30+ conduit-extra
31+ connection
32+ connection
33+ containers
34+ data-default
35+ deepseq
36+ directory
37+ doclayout
38+ doctemplates
39+ emojis
40+ exceptions
41+ file-embed
42+ filepath
43+ haddock-library
44+ hsc2hs
45+ hslua
46+ hslua-module-system
47+ hslua-module-text
48+ http-client
49+ http-client
50+ http-client-tls
51+ http-client-tls
52+ http-types
53+ ipynb
54+ jira-wiki-markup
55+ mtl
56+ network
57+ network
58+ network-uri
59+ pandoc-types
60+ parsec
61+ process
62+ random
63+ safe
64+ scientific
65+ skylighting
66+ skylighting-core
67+ socks
68+ split
69+ streaming-commons
70+ syb
71+ tagsoup
72+ tasty
73+ tasty-golden
74+ tasty-hunit
75+ tasty-lua
76+ tasty-quickcheck
77+ temporary
78+ texmath
79+ text
80+ text-conversions
81+ time
82+ tls
83+ unicode-transforms
84+ unordered-containers
85+ weigh
86+ xml
87+ xml-conduit
88+ zip-archive
89+ zlib
90+ ];
91+
92+ ghc = haskellPackages.ghcWithPackages haskellDeps;
93+
94+ nixPackages = [
95+ pkgs.zlib
96+ ghc
97+ pkgs.gdb
98+ haskellPackages.ghcid
99+ haskellPackages.cabal2nix
100+ haskellPackages.cabal-install
101+ ];
102+in
103+pkgs.stdenv.mkDerivation {
104+ name = "env";
105+ buildInputs = nixPackages;
106+}
src/Text/Pandoc.hs+1−1
@@ -3,7 +3,7 @@
33{-# LANGUAGE ScopedTypeVariables #-}
44{- |
55 Module : Text.Pandoc
6− Copyright : Copyright (C) 2006-2020 John MacFarlane
6+ Copyright : Copyright (C) 2006-2021 John MacFarlane
77 License : GNU GPL, version 2 or above
88
99 Maintainer : John MacFarlane <jgm@berkeley.edu>
src/Text/Pandoc/App.hs+66−43
@@ -1,9 +1,10 @@
1+{-# LANGUAGE LambdaCase #-}
12{-# LANGUAGE OverloadedStrings #-}
23{-# LANGUAGE CPP #-}
34{-# LANGUAGE ScopedTypeVariables #-}
45{- |
56 Module : Text.Pandoc.App
6− Copyright : Copyright (C) 2006-2020 John MacFarlane
7+ Copyright : Copyright (C) 2006-2021 John MacFarlane
78 License : GNU GPL, version 2 or above
89
910 Maintainer : John MacFarlane <jgm@berkeley@edu>
@@ -19,6 +20,7 @@ module Text.Pandoc.App (
1920 , Filter(..)
2021 , defaultOpts
2122 , parseOptions
23+ , parseOptionsFromArgs
2224 , options
2325 , applyFilters
2426 ) where
@@ -27,6 +29,7 @@ import Control.Monad ( (>=>), when )
2729import Control.Monad.Trans ( MonadIO(..) )
2830import Control.Monad.Except (throwError)
2931import qualified Data.ByteString as BS
32+import qualified Data.ByteString.Char8 as B8
3033import qualified Data.ByteString.Lazy as BL
3134import Data.Char (toLower)
3235import Data.Maybe (fromMaybe, isJust, isNothing)
@@ -44,19 +47,21 @@ import System.FilePath ( takeBaseName, takeExtension )
4447import System.IO (nativeNewline, stdout)
4548import qualified System.IO as IO (Newline (..))
4649import Text.Pandoc
50+import Text.Pandoc.Builder (setMeta)
51+import Text.Pandoc.MIME (getCharset)
4752import Text.Pandoc.App.FormatHeuristics (formatFromFilePaths)
4853import Text.Pandoc.App.Opt (Opt (..), LineEnding (..), defaultOpts,
49− IpynbOutput (..) )
50−import Text.Pandoc.App.CommandLineOptions (parseOptions, options)
54+ IpynbOutput (..))
55+import Text.Pandoc.App.CommandLineOptions (parseOptions, parseOptionsFromArgs,
56+ options)
5157import Text.Pandoc.App.OutputSettings (OutputSettings (..), optToOutputSettings)
52−import Text.Pandoc.BCP47 (Lang (..), parseBCP47)
53−import Text.Pandoc.Builder (setMeta)
58+import Text.Collate.Lang (Lang (..), parseLang)
5459import Text.Pandoc.Filter (Filter (JSONFilter, LuaFilter), applyFilters)
5560import Text.Pandoc.PDF (makePDF)
56−import Text.Pandoc.SelfContained (makeDataURI, makeSelfContained)
61+import Text.Pandoc.SelfContained (makeSelfContained)
5762import Text.Pandoc.Shared (eastAsianLineBreakFilter, stripEmptyParagraphs,
5863 headerShift, isURI, tabFilter, uriPathToPath, filterIpynbOutput,
59− defaultUserDataDirs, tshow, findM)
64+ defaultUserDataDir, tshow)
6065import Text.Pandoc.Writers.Shared (lookupMetaString)
6166import Text.Pandoc.Readers.Markdown (yamlToMeta)
6267import qualified Text.Pandoc.UTF8 as UTF8
@@ -67,25 +72,29 @@ import System.Posix.Terminal (queryTerminal)
6772
6873convertWithOpts :: Opt -> IO ()
6974convertWithOpts opts = do
75+ datadir <- case optDataDir opts of
76+ Nothing -> do
77+ d <- defaultUserDataDir
78+ exists <- doesDirectoryExist d
79+ return $ if exists
80+ then Just d
81+ else Nothing
82+ Just _ -> return $ optDataDir opts
83+
7084 let outputFile = fromMaybe "-" (optOutputFile opts)
7185 let filters = optFilters opts
7286 let verbosity = optVerbosity opts
7387
7488 when (optDumpArgs opts) $
75− do UTF8.hPutStrLn stdout outputFile
76− mapM_ (UTF8.hPutStrLn stdout) (fromMaybe ["-"] $ optInputFiles opts)
89+ do UTF8.hPutStrLn stdout (T.pack outputFile)
90+ mapM_ (UTF8.hPutStrLn stdout . T.pack)
91+ (fromMaybe ["-"] $ optInputFiles opts)
7792 exitSuccess
7893
7994 let sources = case optInputFiles opts of
8095 Just xs | not (optIgnoreArgs opts) -> xs
8196 _ -> ["-"]
8297
83− datadir <- case optDataDir opts of
84− Nothing -> do
85− ds <- defaultUserDataDirs
86− findM doesDirectoryExist ds
87− Just _ -> return $ optDataDir opts
88−
8998 let runIO' :: PandocIO a -> IO a
9099 runIO' f = do
91100 (res, reports) <- runIOorExplode $ do
@@ -151,9 +160,11 @@ convertWithOpts opts = do
151160 else optTabStop opts)
152161
153162
154− let readSources :: [FilePath] -> PandocIO Text
155− readSources srcs = convertTabs . T.intercalate (T.pack "\n") <$>
156− mapM readSource srcs
163+ let readSources :: [FilePath] -> PandocIO [(FilePath, Text)]
164+ readSources srcs =
165+ mapM (\fp -> do
166+ t <- readSource fp
167+ return (if fp == "-" then "" else fp, convertTabs t)) srcs
157168
158169
159170 outputSettings <- optToOutputSettings opts
@@ -190,20 +201,9 @@ convertWithOpts opts = do
190201 Nothing -> readDataFile "abbreviations"
191202 Just f -> readFileStrict f
192203
193− metadata <- if format == "jats" &&
194− isNothing (lookupMeta "csl" (optMetadata opts)) &&
195− isNothing (lookupMeta "citation-style"
196− (optMetadata opts))
197− then do
198− jatsCSL <- readDataFile "jats.csl"
199− let jatsEncoded = makeDataURI
200− ("application/xml", jatsCSL)
201− return $ setMeta "csl" jatsEncoded $ optMetadata opts
202− else return $ optMetadata opts
203−
204204 case lookupMetaString "lang" (optMetadata opts) of
205− "" -> setTranslations $ Lang "en" "" "US" []
206− l -> case parseBCP47 l of
205+ "" -> setTranslations $ Lang "en" Nothing (Just "US") [] [] []
206+ l -> case parseLang l of
207207 Left _ -> report $ InvalidLang l
208208 Right l' -> setTranslations l'
209209
@@ -257,13 +257,17 @@ convertWithOpts opts = do
257257 let sourceToDoc :: [FilePath] -> PandocIO Pandoc
258258 sourceToDoc sources' =
259259 case reader of
260− TextReader r
261− | optFileScope opts || readerNameBase == "json" ->
262− mconcat <$> mapM (readSource >=> r readerOpts) sources'
263− | otherwise ->
264− readSources sources' >>= r readerOpts
265− ByteStringReader r ->
266− mconcat <$> mapM (readFile' >=> r readerOpts) sources'
260+ TextReader r
261+ | readerNameBase == "json" ->
262+ mconcat <$> mapM (readSource >=> r readerOpts) sources'
263+ | optFileScope opts ->
264+ -- Read source and convert tabs (see #6709)
265+ let readSource' = fmap convertTabs . readSource
266+ in mconcat <$> mapM (readSource' >=> r readerOpts) sources'
267+ | otherwise ->
268+ readSources sources' >>= r readerOpts
269+ ByteStringReader r ->
270+ mconcat <$> mapM (readFile' >=> r readerOpts) sources'
267271
268272
269273 when (readerNameBase == "markdown_github" ||
@@ -281,12 +285,21 @@ convertWithOpts opts = do
281285 report $ Deprecated "pandoc-citeproc filter"
282286 "Use --citeproc instead."
283287
288+ let cslMetadata =
289+ maybe id (setMeta "csl") (optCSL opts) .
290+ (case optBibliography opts of
291+ [] -> id
292+ xs -> setMeta "bibliography" xs) .
293+ maybe id (setMeta "citation-abbreviations")
294+ (optCitationAbbreviations opts) $ mempty
295+
284296 doc <- sourceToDoc sources >>=
285297 ( (if isJust (optExtractMedia opts)
286298 then fillMediaBag
287299 else return)
288300 >=> return . adjustMetadata (metadataFromFile <>)
289− >=> return . adjustMetadata (<> metadata)
301+ >=> return . adjustMetadata (<> optMetadata opts)
302+ >=> return . adjustMetadata (<> cslMetadata)
290303 >=> applyTransforms transforms
291304 >=> applyFilters readerOpts filters [T.unpack format]
292305 >=> maybe return extractMedia (optExtractMedia opts)
@@ -353,7 +366,18 @@ readSource src = case parseURI src of
353366 _ -> PandocAppError (tshow e))
354367
355368readURI :: FilePath -> PandocIO Text
356−readURI src = UTF8.toText . fst <$> openURL (T.pack src)
369+readURI src = do
370+ (bs, mt) <- openURL (T.pack src)
371+ case mt >>= getCharset of
372+ Just "UTF-8" -> return $ UTF8.toText bs
373+ Just "ISO-8859-1" -> return $ T.pack $ B8.unpack bs
374+ Just charset -> throwError $ PandocUnsupportedCharsetError charset
375+ Nothing -> liftIO $ -- try first as UTF-8, then as latin1
376+ E.catch (return $! UTF8.toText bs)
377+ (\case
378+ TSE.DecodeError{} ->
379+ return $ T.pack $ B8.unpack bs
380+ e -> E.throwIO e)
357381
358382readFile' :: MonadIO m => FilePath -> m BL.ByteString
359383readFile' "-" = liftIO BL.getContents
@@ -364,6 +388,5 @@ writeFnBinary "-" = liftIO . BL.putStr
364388writeFnBinary f = liftIO . BL.writeFile (UTF8.encodePath f)
365389
366390writerFn :: MonadIO m => IO.Newline -> FilePath -> Text -> m ()
367−-- TODO this implementation isn't maximally efficient:
368−writerFn eol "-" = liftIO . UTF8.putStrWith eol . T.unpack
369−writerFn eol f = liftIO . UTF8.writeFileWith eol f . T.unpack
391+writerFn eol "-" = liftIO . UTF8.putStrWith eol
392+writerFn eol f = liftIO . UTF8.writeFileWith eol f
src/Text/Pandoc/App/CommandLineOptions.hs+64−51
@@ -6,7 +6,7 @@
66{-# LANGUAGE FlexibleContexts #-}
77{- |
88 Module : Text.Pandoc.App.CommandLineOptions
9− Copyright : Copyright (C) 2006-2020 John MacFarlane
9+ Copyright : Copyright (C) 2006-2021 John MacFarlane
1010 License : GNU GPL, version 2 or above
1111
1212 Maintainer : John MacFarlane <jgm@berkeley@edu>
@@ -17,6 +17,7 @@ Does a pandoc conversion based on command-line options.
1717-}
1818module Text.Pandoc.App.CommandLineOptions (
1919 parseOptions
20+ , parseOptionsFromArgs
2021 , options
2122 , engines
2223 , lookupHighlightStyle
@@ -25,11 +26,12 @@ module Text.Pandoc.App.CommandLineOptions (
2526import Control.Monad
2627import Control.Monad.Trans
2728import Control.Monad.Except (throwError)
29+import Control.Monad.State.Strict
2830import Data.Aeson.Encode.Pretty (encodePretty', Config(..), keyOrder,
2931 defConfig, Indent(..), NumberFormat(..))
3032import Data.Bifunctor (second)
3133import Data.Char (toLower)
32−import Data.List (intercalate, sort)
34+import Data.List (intercalate, sort, foldl')
3335#ifdef _WINDOWS
3436#if MIN_VERSION_base(4,12,0)
3537import Data.List (isPrefixOf)
@@ -46,10 +48,13 @@ import System.FilePath
4648import System.IO (stdout)
4749import Text.DocTemplates (Context (..), ToContext (toVal), Val (..))
4850import Text.Pandoc
49−import Text.Pandoc.App.Opt (Opt (..), LineEnding (..), IpynbOutput (..), addMeta)
51+import Text.Pandoc.Builder (setMeta)
52+import Text.Pandoc.App.Opt (Opt (..), LineEnding (..), IpynbOutput (..),
53+ DefaultsState (..), applyDefaults,
54+ fullDefaultsPath)
5055import Text.Pandoc.Filter (Filter (..))
5156import Text.Pandoc.Highlighting (highlightingStyles)
52−import Text.Pandoc.Shared (ordNub, elemText, safeStrRead, defaultUserDataDirs, findM)
57+import Text.Pandoc.Shared (ordNub, elemText, safeStrRead, defaultUserDataDir)
5358import Text.Printf
5459
5560#ifdef EMBED_DATA_FILES
@@ -64,16 +69,19 @@ import qualified Data.ByteString as BS
6469import qualified Data.ByteString.Lazy as B
6570import qualified Data.Map as M
6671import qualified Data.Text as T
67−import qualified Data.YAML as Y
6872import qualified Text.Pandoc.UTF8 as UTF8
6973
7074parseOptions :: [OptDescr (Opt -> IO Opt)] -> Opt -> IO Opt
7175parseOptions options' defaults = do
7276 rawArgs <- map UTF8.decodeArg <$> getArgs
7377 prg <- getProgName
78+ parseOptionsFromArgs options' defaults prg rawArgs
7479
80+parseOptionsFromArgs
81+ :: [OptDescr (Opt -> IO Opt)] -> Opt -> String -> [String] -> IO Opt
82+parseOptionsFromArgs options' defaults prg rawArgs = do
7583 let (actions, args, unrecognizedOpts, errors) =
76− getOpt' Permute options' rawArgs
84+ getOpt' Permute options' (map UTF8.decodeArg rawArgs)
7785
7886 let unknownOptionErrors =
7987 foldr (handleUnrecognizedOption . takeWhile (/= '=')) []
@@ -85,7 +93,7 @@ parseOptions options' defaults = do
8593 ("Try " ++ prg ++ " --help for more information.")
8694
8795 -- thread option data structure through all supplied option actions
88− opts <- foldl (>>=) (return defaults) actions
96+ opts <- foldl' (>>=) (return defaults) actions
8997 let mbArgs = case args of
9098 [] -> Nothing
9199 xs -> Just xs
@@ -166,7 +174,11 @@ options =
166174
167175 , Option "d" ["defaults"]
168176 (ReqArg
169− (\arg opt -> applyDefaults opt arg
177+ (\arg opt -> runIOorExplode $ do
178+ let defsState = DefaultsState { curDefaults = Nothing,
179+ inheritanceGraph = [] }
180+ fp <- fullDefaultsPath (optDataDir opt) arg
181+ evalStateT (applyDefaults opt fp) defsState
170182 )
171183 "FILE")
172184 ""
@@ -276,7 +288,8 @@ options =
276288 , Option "" ["resource-path"]
277289 (ReqArg
278290 (\arg opt -> return opt { optResourcePath =
279− splitSearchPath arg })
291+ splitSearchPath arg ++
292+ optResourcePath opt })
280293 "SEARCHPATH")
281294 "" -- "Paths to search for images and other resources"
282295
@@ -801,10 +814,10 @@ options =
801814 map (\c -> ['-',c]) shorts ++
802815 map ("--" ++) longs
803816 let allopts = unwords (concatMap optnames options)
804− UTF8.hPutStrLn stdout $ printf tpl allopts
805− (unwords readersNames)
806− (unwords writersNames)
807− (unwords $ map (T.unpack . fst) highlightingStyles)
817+ UTF8.hPutStrLn stdout $ T.pack $ printf tpl allopts
818+ (T.unpack $ T.unwords readersNames)
819+ (T.unpack $ T.unwords writersNames)
820+ (T.unpack $ T.unwords $ map fst highlightingStyles)
808821 (unwords datafiles)
809822 exitSuccess ))
810823 "" -- "Print bash completion script"
@@ -843,7 +856,7 @@ options =
843856 else if extensionEnabled x allExts
844857 then '-'
845858 else ' ') : drop 4 (show x)
846− mapM_ (UTF8.hPutStrLn stdout . showExt)
859+ mapM_ (UTF8.hPutStrLn stdout . T.pack . showExt)
847860 [ex | ex <- extList, extensionEnabled ex allExts]
848861 exitSuccess )
849862 "FORMAT")
@@ -857,14 +870,14 @@ options =
857870 , sShortname s `notElem`
858871 [T.pack "Alert", T.pack "Alert_indent"]
859872 ]
860− mapM_ (UTF8.hPutStrLn stdout) (sort langs)
873+ mapM_ (UTF8.hPutStrLn stdout . T.pack) (sort langs)
861874 exitSuccess ))
862875 ""
863876
864877 , Option "" ["list-highlight-styles"]
865878 (NoArg
866879 (\_ -> do
867− mapM_ (UTF8.hPutStrLn stdout . T.unpack . fst) highlightingStyles
880+ mapM_ (UTF8.hPutStrLn stdout . fst) highlightingStyles
868881 exitSuccess ))
869882 ""
870883
@@ -882,7 +895,7 @@ options =
882895 | T.null t -> -- e.g. for docx, odt, json:
883896 E.throwIO $ PandocCouldNotFindDataFileError $ T.pack
884897 ("templates/default." ++ arg)
885− | otherwise -> write . T.unpack $ t
898+ | otherwise -> write t
886899 Left e -> E.throwIO e
887900 exitSuccess)
888901 "FORMAT")
@@ -928,12 +941,13 @@ options =
928941 (NoArg
929942 (\_ -> do
930943 prg <- getProgName
931− defaultDatadirs <- defaultUserDataDirs
932− UTF8.hPutStrLn stdout (prg ++ " " ++ T.unpack pandocVersion ++
933− compileInfo ++
934− "\nUser data directory: " ++
935− intercalate " or " defaultDatadirs ++
936− ('\n':copyrightMessage))
944+ defaultDatadir <- defaultUserDataDir
945+ UTF8.hPutStrLn stdout
946+ $ T.pack
947+ $ prg ++ " " ++ T.unpack pandocVersion ++
948+ compileInfo ++
949+ "\nUser data directory: " ++ defaultDatadir ++
950+ ('\n':copyrightMessage)
937951 exitSuccess ))
938952 "" -- "Print version"
939953
@@ -941,7 +955,7 @@ options =
941955 (NoArg
942956 (\_ -> do
943957 prg <- getProgName
944− UTF8.hPutStr stdout (usageMessage prg options)
958+ UTF8.hPutStr stdout (T.pack $ usageMessage prg options)
945959 exitSuccess ))
946960 "" -- "Show help"
947961 ]
@@ -962,7 +976,7 @@ usageMessage programName = usageInfo (programName ++ " [OPTIONS] [FILES]")
962976
963977copyrightMessage :: String
964978copyrightMessage = intercalate "\n" [
965− "Copyright (C) 2006-2020 John MacFarlane. Web: https://pandoc.org",
979+ "Copyright (C) 2006-2021 John MacFarlane. Web: https://pandoc.org",
966980 "This is free software; see the source for copying conditions. There is no",
967981 "warranty, not even for merchantability or fitness for a particular purpose." ]
968982
@@ -1002,38 +1016,16 @@ handleUnrecognizedOption "-R" = handleUnrecognizedOption "--parse-raw"
10021016handleUnrecognizedOption x =
10031017 (("Unknown option " ++ x ++ ".") :)
10041018
1005−readersNames :: [String]
1006−readersNames = sort (map (T.unpack . fst) (readers :: [(Text, Reader PandocIO)]))
1019+readersNames :: [Text]
1020+readersNames = sort (map fst (readers :: [(Text, Reader PandocIO)]))
10071021
1008−writersNames :: [String]
1022+writersNames :: [Text]
10091023writersNames = sort
1010− ("pdf" : map (T.unpack . fst) (writers :: [(Text, Writer PandocIO)]))
1024+ ("pdf" : map fst (writers :: [(Text, Writer PandocIO)]))
10111025
10121026splitField :: String -> (String, String)
10131027splitField = second (tailDef "true") . break (`elemText` ":=")
10141028
1015−-- | Apply defaults from --defaults file.
1016−applyDefaults :: Opt -> FilePath -> IO Opt
1017−applyDefaults opt file = runIOorExplode $ do
1018− let fp = if null (takeExtension file)
1019− then addExtension file "yaml"
1020− else file
1021− setVerbosity $ optVerbosity opt
1022− dataDirs <- liftIO defaultUserDataDirs
1023− let fps = fp : case optDataDir opt of
1024− Nothing -> map (</> ("defaults" </> fp))
1025− dataDirs
1026− Just dd -> [dd </> "defaults" </> fp]
1027− fp' <- fromMaybe fp <$> findM fileExists fps
1028− inp <- readFileLazy fp'
1029− case Y.decode1 inp of
1030− Right (f :: Opt -> Opt) -> return $ f opt
1031− Left (errpos, errmsg) -> throwError $
1032− PandocParseError $ T.pack $
1033− "Error parsing " ++ fp' ++ " line " ++
1034− show (Y.posLine errpos) ++ " column " ++
1035− show (Y.posColumn errpos) ++ ":\n" ++ errmsg
1036−
10371029lookupHighlightStyle :: PandocMonad m => String -> m Style
10381030lookupHighlightStyle s
10391031 | takeExtension s == ".theme" = -- attempt to load KDE theme
@@ -1062,6 +1054,27 @@ setVariable key val (Context ctx) = Context $ M.alter go key ctx
10621054 go (Just (ListVal xs)) = Just $ ListVal $ xs ++ [toVal val]
10631055 go (Just x) = Just $ ListVal [x, toVal val]
10641056
1057+addMeta :: String -> String -> Meta -> Meta
1058+addMeta k v meta =
1059+ case lookupMeta k' meta of
1060+ Nothing -> setMeta k' v' meta
1061+ Just (MetaList xs) ->
1062+ setMeta k' (MetaList (xs ++ [v'])) meta
1063+ Just x -> setMeta k' (MetaList [x, v']) meta
1064+ where
1065+ v' = readMetaValue v
1066+ k' = T.pack k
1067+
1068+readMetaValue :: String -> MetaValue
1069+readMetaValue s
1070+ | s == "true" = MetaBool True
1071+ | s == "True" = MetaBool True
1072+ | s == "TRUE" = MetaBool True
1073+ | s == "false" = MetaBool False
1074+ | s == "False" = MetaBool False
1075+ | s == "FALSE" = MetaBool False
1076+ | otherwise = MetaString $ T.pack s
1077+
10651078-- On Windows with ghc 8.6+, we need to rewrite paths
10661079-- beginning with \\ to \\?\UNC\. -- See #5127.
10671080normalizePath :: FilePath -> FilePath
src/Text/Pandoc/App/FormatHeuristics.hs+19−8
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.App.FormatHeuristics
4− Copyright : Copyright (C) 2006-2020 John MacFarlane
4+ Copyright : Copyright (C) 2006-2021 John MacFarlane
55 License : GNU GPL, version 2 or above
66
77 Maintainer : John MacFarlane <jgm@berkeley@edu>
@@ -15,18 +15,24 @@ module Text.Pandoc.App.FormatHeuristics
1515 ) where
1616
1717import Data.Char (toLower)
18+import Data.Foldable (asum)
1819import Data.Text (Text)
1920import System.FilePath (takeExtension)
2021
21−-- Determine default format based on file extensions.
22+-- | Determines default format based on file extensions; uses the format
23+-- of the first extension that's associated with a format.
24+--
25+-- Examples:
26+--
27+-- > formatFromFilePaths ["text.unknown", "no-extension"]
28+-- Nothing
29+--
30+-- > formatFromFilePaths ["my.md", "other.rst"]
31+-- Just "markdown"
2232formatFromFilePaths :: [FilePath] -> Maybe Text
23−formatFromFilePaths [] = Nothing
24−formatFromFilePaths (x:xs) =
25− case formatFromFilePath x of
26− Just f -> Just f
27− Nothing -> formatFromFilePaths xs
33+formatFromFilePaths = asum . map formatFromFilePath
2834
29−-- Determine format based on file extension
35+-- | Determines format based on file extension.
3036formatFromFilePath :: FilePath -> Maybe Text
3137formatFromFilePath x =
3238 case takeExtension (map toLower x) of
@@ -48,6 +54,11 @@ formatFromFilePath x =
4854 ".lhs" -> Just "markdown+lhs"
4955 ".ltx" -> Just "latex"
5056 ".markdown" -> Just "markdown"
57+ ".mkdn" -> Just "markdown"
58+ ".mkd" -> Just "markdown"
59+ ".mdwn" -> Just "markdown"
60+ ".mdown" -> Just "markdown"
61+ ".Rmd" -> Just "markdown"
5162 ".md" -> Just "markdown"
5263 ".ms" -> Just "ms"
5364 ".muse" -> Just "muse"
src/Text/Pandoc/App/Opt.hs+261−51
@@ -7,7 +7,7 @@
77{-# LANGUAGE FlexibleContexts #-}
88{- |
99 Module : Text.Pandoc.App.Opt
10− Copyright : Copyright (C) 2006-2020 John MacFarlane
10+ Copyright : Copyright (C) 2006-2021 John MacFarlane
1111 License : GNU GPL, version 2 or above
1212
1313 Maintainer : John MacFarlane <jgm@berkeley@edu>
@@ -20,21 +20,31 @@ module Text.Pandoc.App.Opt (
2020 Opt(..)
2121 , LineEnding (..)
2222 , IpynbOutput (..)
23+ , DefaultsState (..)
2324 , defaultOpts
24− , addMeta
25+ , applyDefaults
26+ , fullDefaultsPath
2527 ) where
28+import Control.Monad.Except (MonadIO, liftIO, throwError, (>=>), foldM)
29+import Control.Monad.State.Strict (StateT, modify, gets)
30+import System.FilePath ( addExtension, (</>), takeExtension, takeDirectory )
31+import System.Directory ( canonicalizePath )
2632import Data.Char (isLower, toLower)
33+import Data.Maybe (fromMaybe)
2734import GHC.Generics hiding (Meta)
28−import Text.Pandoc.Builder (setMeta)
2935import Text.Pandoc.Filter (Filter (..))
30−import Text.Pandoc.Logging (Verbosity (WARNING))
36+import Text.Pandoc.Logging (Verbosity (WARNING), LogMessage(..))
3137import Text.Pandoc.Options (TopLevelDivision (TopLevelDefault),
3238 TrackChanges (AcceptChanges),
3339 WrapOption (WrapAuto), HTMLMathMethod (PlainMath),
3440 ReferenceLocation (EndOfDocument),
3541 ObfuscationMethod (NoObfuscation),
3642 CiteMethod (Citeproc))
37−import Text.Pandoc.Shared (camelCaseStrToHyphenated)
43+import Text.Pandoc.Class (readFileLazy, fileExists, setVerbosity, report,
44+ PandocMonad(lookupEnv), getUserDataDir)
45+import Text.Pandoc.Error (PandocError (PandocParseError, PandocSomeError))
46+import Text.Pandoc.Shared (camelCaseStrToHyphenated, defaultUserDataDir,
47+ findM, ordNub)
3848import qualified Text.Pandoc.Parsing as P
3949import Text.Pandoc.Readers.Metadata (yamlMap)
4050import Text.Pandoc.Class.PandocPure
@@ -43,7 +53,7 @@ import Data.Text (Text, unpack)
4353import Data.Default (def)
4454import qualified Data.Text as T
4555import qualified Data.Map as M
46−import Text.Pandoc.Definition (Meta(..), MetaValue(..), lookupMeta)
56+import Text.Pandoc.Definition (Meta(..), MetaValue(..))
4757import Data.Aeson (defaultOptions, Options(..))
4858import Data.Aeson.TH (deriveJSON)
4959import Control.Applicative ((<|>))
@@ -147,19 +157,194 @@ data Opt = Opt
147157 , optNoCheckCertificate :: Bool -- ^ Disable certificate validation
148158 , optEol :: LineEnding -- ^ Style of line-endings to use
149159 , optStripComments :: Bool -- ^ Skip HTML comments
160+ , optCSL :: Maybe FilePath -- ^ CSL stylesheet
161+ , optBibliography :: [FilePath] -- ^ Bibliography files
162+ , optCitationAbbreviations :: Maybe FilePath -- ^ Citation abbreviations
150163 } deriving (Generic, Show)
151164
152165instance FromYAML (Opt -> Opt) where
153− parseYAML (Mapping _ _ m) =
154− foldr (.) id <$> mapM doOpt (M.toList m)
166+ parseYAML (Mapping _ _ m) = chain doOpt (M.toList m)
155167 parseYAML n = failAtNode n "Expected a mapping"
156168
169+data DefaultsState = DefaultsState
170+ {
171+ curDefaults :: Maybe FilePath -- currently parsed file
172+ , inheritanceGraph :: [[FilePath]] -- defaults file inheritance graph
173+ } deriving (Show)
174+
175+instance (PandocMonad m, MonadIO m)
176+ => FromYAML (Opt -> StateT DefaultsState m Opt) where
177+ parseYAML (Mapping _ _ m) = do
178+ let opts = M.mapKeys toText m
179+ dataDir <- case M.lookup "data-dir" opts of
180+ Nothing -> return Nothing
181+ Just v -> Just . unpack <$> parseYAML v
182+ f <- parseOptions (M.toList m)
183+ case M.lookup "defaults" opts of
184+ Just v -> do
185+ g <- parseDefaults v dataDir
186+ return $ g >=> f >=> resolveVarsInOpt
187+ Nothing -> return $ f >=> resolveVarsInOpt
188+ where
189+ toText (Scalar _ (SStr s)) = s
190+ toText _ = ""
191+ parseYAML n = failAtNode n "Expected a mapping"
192+
193+resolveVarsInOpt :: forall m. (PandocMonad m, MonadIO m)
194+ => Opt -> StateT DefaultsState m Opt
195+resolveVarsInOpt
196+ opt@Opt
197+ { optTemplate = oTemplate
198+ , optMetadataFiles = oMetadataFiles
199+ , optOutputFile = oOutputFile
200+ , optInputFiles = oInputFiles
201+ , optSyntaxDefinitions = oSyntaxDefinitions
202+ , optAbbreviations = oAbbreviations
203+ , optReferenceDoc = oReferenceDoc
204+ , optEpubMetadata = oEpubMetadata
205+ , optEpubFonts = oEpubFonts
206+ , optEpubCoverImage = oEpubCoverImage
207+ , optLogFile = oLogFile
208+ , optFilters = oFilters
209+ , optDataDir = oDataDir
210+ , optExtractMedia = oExtractMedia
211+ , optCss = oCss
212+ , optIncludeBeforeBody = oIncludeBeforeBody
213+ , optIncludeAfterBody = oIncludeAfterBody
214+ , optIncludeInHeader = oIncludeInHeader
215+ , optResourcePath = oResourcePath
216+ , optCSL = oCSL
217+ , optBibliography = oBibliography
218+ , optCitationAbbreviations = oCitationAbbreviations
219+ }
220+ = do
221+ oTemplate' <- mapM resolveVars oTemplate
222+ oMetadataFiles' <- mapM resolveVars oMetadataFiles
223+ oOutputFile' <- mapM resolveVars oOutputFile
224+ oInputFiles' <- mapM (mapM resolveVars) oInputFiles
225+ oSyntaxDefinitions' <- mapM resolveVars oSyntaxDefinitions
226+ oAbbreviations' <- mapM resolveVars oAbbreviations
227+ oReferenceDoc' <- mapM resolveVars oReferenceDoc
228+ oEpubMetadata' <- mapM resolveVars oEpubMetadata
229+ oEpubFonts' <- mapM resolveVars oEpubFonts
230+ oEpubCoverImage' <- mapM resolveVars oEpubCoverImage
231+ oLogFile' <- mapM resolveVars oLogFile
232+ oFilters' <- mapM resolveVarsInFilter oFilters
233+ oDataDir' <- mapM resolveVars oDataDir
234+ oExtractMedia' <- mapM resolveVars oExtractMedia
235+ oCss' <- mapM resolveVars oCss
236+ oIncludeBeforeBody' <- mapM resolveVars oIncludeBeforeBody
237+ oIncludeAfterBody' <- mapM resolveVars oIncludeAfterBody
238+ oIncludeInHeader' <- mapM resolveVars oIncludeInHeader
239+ oResourcePath' <- mapM resolveVars oResourcePath
240+ oCSL' <- mapM resolveVars oCSL
241+ oBibliography' <- mapM resolveVars oBibliography
242+ oCitationAbbreviations' <- mapM resolveVars oCitationAbbreviations
243+ return opt{ optTemplate = oTemplate'
244+ , optMetadataFiles = oMetadataFiles'
245+ , optOutputFile = oOutputFile'
246+ , optInputFiles = oInputFiles'
247+ , optSyntaxDefinitions = oSyntaxDefinitions'
248+ , optAbbreviations = oAbbreviations'
249+ , optReferenceDoc = oReferenceDoc'
250+ , optEpubMetadata = oEpubMetadata'
251+ , optEpubFonts = oEpubFonts'
252+ , optEpubCoverImage = oEpubCoverImage'
253+ , optLogFile = oLogFile'
254+ , optFilters = oFilters'
255+ , optDataDir = oDataDir'
256+ , optExtractMedia = oExtractMedia'
257+ , optCss = oCss'
258+ , optIncludeBeforeBody = oIncludeBeforeBody'
259+ , optIncludeAfterBody = oIncludeAfterBody'
260+ , optIncludeInHeader = oIncludeInHeader'
261+ , optResourcePath = oResourcePath'
262+ , optCSL = oCSL'
263+ , optBibliography = oBibliography'
264+ , optCitationAbbreviations = oCitationAbbreviations'
265+ }
266+
267+ where
268+ resolveVars :: FilePath -> StateT DefaultsState m FilePath
269+ resolveVars [] = return []
270+ resolveVars ('$':'{':xs) =
271+ let (ys, zs) = break (=='}') xs
272+ in if null zs
273+ then return $ '$':'{':xs
274+ else do
275+ val <- lookupEnv' ys
276+ (val ++) <$> resolveVars (drop 1 zs)
277+ resolveVars (c:cs) = (c:) <$> resolveVars cs
278+ lookupEnv' :: String -> StateT DefaultsState m String
279+ lookupEnv' "." = do
280+ mbCurDefaults <- gets curDefaults
281+ maybe (return "")
282+ (fmap takeDirectory . liftIO . canonicalizePath)
283+ mbCurDefaults
284+ lookupEnv' "USERDATA" = do
285+ mbodatadir <- mapM resolveVars oDataDir
286+ mbdatadir <- getUserDataDir
287+ defdatadir <- liftIO defaultUserDataDir
288+ return $ fromMaybe defdatadir (mbodatadir <|> mbdatadir)
289+ lookupEnv' v = do
290+ mbval <- fmap T.unpack <$> lookupEnv (T.pack v)
291+ case mbval of
292+ Nothing -> do
293+ report $ EnvironmentVariableUndefined (T.pack v)
294+ return mempty
295+ Just x -> return x
296+ resolveVarsInFilter (JSONFilter fp) =
297+ JSONFilter <$> resolveVars fp
298+ resolveVarsInFilter (LuaFilter fp) =
299+ LuaFilter <$> resolveVars fp
300+ resolveVarsInFilter CiteprocFilter = return CiteprocFilter
301+
302+
303+
304+parseDefaults :: (PandocMonad m, MonadIO m)
305+ => Node Pos
306+ -> Maybe FilePath
307+ -> Parser (Opt -> StateT DefaultsState m Opt)
308+parseDefaults n dataDir = parseDefsNames n >>= \ds -> return $ \o -> do
309+ -- get parent defaults:
310+ defsParent <- gets $ fromMaybe "" . curDefaults
311+ -- get child defaults:
312+ defsChildren <- mapM (fullDefaultsPath dataDir) ds
313+ -- expand parent in defaults inheritance graph by children:
314+ defsGraph <- gets inheritanceGraph
315+ let defsGraphExp = expand defsGraph defsChildren defsParent
316+ modify $ \defsState -> defsState{ inheritanceGraph = defsGraphExp }
317+ -- check for cyclic inheritance:
318+ if cyclic defsGraphExp
319+ then throwError $
320+ PandocSomeError $ T.pack $
321+ "Error: Circular defaults file reference in " ++
322+ "'" ++ defsParent ++ "'"
323+ else foldM applyDefaults o defsChildren
324+ where parseDefsNames x = (parseYAML x >>= \xs -> return $ map unpack xs)
325+ <|> (parseYAML x >>= \x' -> return [unpack x'])
326+
327+parseOptions :: Monad m
328+ => [(Node Pos, Node Pos)]
329+ -> Parser (Opt -> StateT DefaultsState m Opt)
330+parseOptions ns = do
331+ f <- chain doOpt' ns
332+ return $ return . f
333+
334+chain :: Monad m => (a -> m (b -> b)) -> [a] -> m (b -> b)
335+chain f = foldM g id
336+ where g o n = f n >>= \o' -> return $ o' . o
337+
338+doOpt' :: (Node Pos, Node Pos) -> Parser (Opt -> Opt)
339+doOpt' (k',v) = do
340+ k <- parseStringKey k'
341+ case k of
342+ "defaults" -> return id
343+ _ -> doOpt (k',v)
344+
157345doOpt :: (Node Pos, Node Pos) -> Parser (Opt -> Opt)
158346doOpt (k',v) = do
159− k <- case k' of
160− Scalar _ (SStr t) -> return t
161− Scalar _ _ -> failAtNode k' "Non-string key"
162− _ -> failAtNode k' "Non-scalar key"
347+ k <- parseStringKey k'
163348 case k of
164349 "tab-stop" ->
165350 parseYAML v >>= \x -> return (\o -> o{ optTabStop = x })
@@ -358,26 +543,18 @@ doOpt (k',v) = do
358543 (parseYAML v >>= \x -> return (\o -> o{ optCss = optCss o <>
359544 [unpack x] }))
360545 "bibliography" ->
361− do let addItem x o = o{ optMetadata =
362− addMeta "bibliography" (T.unpack x)
363− (optMetadata o) }
364− (parseYAML v >>= \(xs :: [Text]) -> return $ \o ->
365− foldr addItem o xs)
366− <|>
367− (parseYAML v >>= \(x :: Text) -> return $ \o -> addItem x o)
546+ (parseYAML v >>= \x -> return (\o ->
547+ o{ optBibliography = optBibliography o <>
548+ map unpack x }))
549+ <|>
550+ (parseYAML v >>= \x -> return (\o ->
551+ o{ optBibliography = optBibliography o <>
552+ [unpack x] }))
368553 "csl" ->
369− do let addItem x o = o{ optMetadata =
370− addMeta "csl" (T.unpack x)
371− (optMetadata o) }
372− (parseYAML v >>= \(xs :: [Text]) -> return $ \o ->
373− foldr addItem o xs)
374− <|>
375− (parseYAML v >>= \(x :: Text) -> return $ \o -> addItem x o)
554+ parseYAML v >>= \x -> return (\o -> o{ optCSL = unpack <$> x })
376555 "citation-abbreviations" ->
377− parseYAML v >>= \x ->
378− return (\o -> o{ optMetadata =
379− addMeta "citation-abbreviations" (T.unpack x)
380− (optMetadata o) })
556+ parseYAML v >>= \x -> return (\o -> o{ optCitationAbbreviations =
557+ unpack <$> x })
381558 "ipynb-output" ->
382559 parseYAML v >>= \x -> return (\o -> o{ optIpynbOutput = x })
383560 "include-before-body" ->
@@ -406,7 +583,8 @@ doOpt (k',v) = do
406583 optIncludeInHeader o <> [unpack x] }))
407584 "resource-path" ->
408585 parseYAML v >>= \x ->
409− return (\o -> o{ optResourcePath = map unpack x })
586+ return (\o -> o{ optResourcePath = map unpack x <>
587+ optResourcePath o })
410588 "request-headers" ->
411589 parseYAML v >>= \x ->
412590 return (\o -> o{ optRequestHeaders = x })
@@ -492,38 +670,70 @@ defaultOpts = Opt
492670 , optNoCheckCertificate = False
493671 , optEol = Native
494672 , optStripComments = False
673+ , optCSL = Nothing
674+ , optBibliography = []
675+ , optCitationAbbreviations = Nothing
495676 }
496677
678+parseStringKey :: Node Pos -> Parser Text
679+parseStringKey k = case k of
680+ Scalar _ (SStr t) -> return t
681+ Scalar _ _ -> failAtNode k "Non-string key"
682+ _ -> failAtNode k "Non-scalar key"
683+
497684yamlToMeta :: Node Pos -> Parser Meta
498685yamlToMeta (Mapping _ _ m) =
499686 either (fail . show) return $ runEverything (yamlMap pMetaString m)
500687 where
501688 pMetaString = pure . MetaString <$> P.manyChar P.anyChar
502− runEverything p = runPure (P.readWithM p def "")
689+ runEverything p =
690+ runPure (P.readWithM p (def :: P.ParserState) ("" :: Text))
503691 >>= fmap (Meta . flip P.runF def)
504692yamlToMeta _ = return mempty
505693
506−addMeta :: String -> String -> Meta -> Meta
507−addMeta k v meta =
508− case lookupMeta k' meta of
509− Nothing -> setMeta k' v' meta
510− Just (MetaList xs) ->
511− setMeta k' (MetaList (xs ++ [v'])) meta
512− Just x -> setMeta k' (MetaList [x, v']) meta
513− where
514− v' = readMetaValue v
515− k' = T.pack k
694+-- | Apply defaults from --defaults file.
695+applyDefaults :: (PandocMonad m, MonadIO m)
696+ => Opt
697+ -> FilePath
698+ -> StateT DefaultsState m Opt
699+applyDefaults opt file = do
700+ setVerbosity $ optVerbosity opt
701+ modify $ \defsState -> defsState{ curDefaults = Just file }
702+ inp <- readFileLazy file
703+ case decode1 inp of
704+ Right f -> f opt
705+ Left (errpos, errmsg) -> throwError $
706+ PandocParseError $ T.pack $
707+ "Error parsing " ++ file ++ " line " ++
708+ show (posLine errpos) ++ " column " ++
709+ show (posColumn errpos) ++ ":\n" ++ errmsg
516710
517−readMetaValue :: String -> MetaValue
518−readMetaValue s
519− | s == "true" = MetaBool True
520− | s == "True" = MetaBool True
521− | s == "TRUE" = MetaBool True
522− | s == "false" = MetaBool False
523− | s == "False" = MetaBool False
524− | s == "FALSE" = MetaBool False
525− | otherwise = MetaString $ T.pack s
711+fullDefaultsPath :: (PandocMonad m, MonadIO m)
712+ => Maybe FilePath
713+ -> FilePath
714+ -> m FilePath
715+fullDefaultsPath dataDir file = do
716+ let fp = if null (takeExtension file)
717+ then addExtension file "yaml"
718+ else file
719+ defaultDataDir <- liftIO defaultUserDataDir
720+ let defaultFp = fromMaybe defaultDataDir dataDir </> "defaults" </> fp
721+ fromMaybe fp <$> findM fileExists [fp, defaultFp]
526722
723+-- | In a list of lists, append another list in front of every list which
724+-- starts with specific element.
725+expand :: Ord a => [[a]] -> [a] -> a -> [[a]]
726+expand [] ns n = fmap (\x -> x : [n]) ns
727+expand ps ns n = concatMap (ext n ns) ps
728+ where
729+ ext x xs p = case p of
730+ (l : _) | x == l -> fmap (: p) xs
731+ _ -> [p]
732+
733+cyclic :: Ord a => [[a]] -> Bool
734+cyclic = any hasDuplicate
735+ where
736+ hasDuplicate xs = length (ordNub xs) /= length xs
527737
528738-- see https://github.com/jgm/pandoc/pull/4083
529739-- using generic deriving caused long compilation times
src/Text/Pandoc/App/OutputSettings.hs+3−3
@@ -5,7 +5,7 @@
55{-# LANGUAGE TupleSections #-}
66{- |
77 Module : Text.Pandoc.App
8− Copyright : Copyright (C) 2006-2020 John MacFarlane
8+ Copyright : Copyright (C) 2006-2021 John MacFarlane
99 License : GNU GPL, version 2 or above
1010
1111 Maintainer : John MacFarlane <jgm@berkeley@edu>
@@ -59,8 +59,8 @@ optToOutputSettings opts = do
5959 let outputFile = fromMaybe "-" (optOutputFile opts)
6060
6161 when (optDumpArgs opts) . liftIO $ do
62− UTF8.hPutStrLn stdout outputFile
63− mapM_ (UTF8.hPutStrLn stdout) (fromMaybe [] $ optInputFiles opts)
62+ UTF8.hPutStrLn stdout (T.pack outputFile)
63+ mapM_ (UTF8.hPutStrLn stdout . T.pack) (fromMaybe [] $ optInputFiles opts)
6464 exitSuccess
6565
6666 epubMetadata <- traverse readUtf8File $ optEpubMetadata opts
src/Text/Pandoc/Asciify.hs+14−391
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Asciify
3− Copyright : Copyright (C) 2013-2020 John MacFarlane
3+ Copyright : Copyright (C) 2013-2021 John MacFarlane
44 License : GNU GPL, version 2 or above
55
66 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -10,396 +10,19 @@
1010Function to convert accented latin letters to their unaccented
1111ascii equivalents (used in constructing HTML identifiers).
1212-}
13−module Text.Pandoc.Asciify (toAsciiChar)
13+module Text.Pandoc.Asciify (toAsciiChar, toAsciiText)
1414where
15−import Data.Char (isAscii)
16−import qualified Data.Map as M
15+import Data.Char (isAscii, isMark)
16+import qualified Data.Text.Normalize as TN
17+import Data.Text (Text)
18+import qualified Data.Text as T
1719
18−toAsciiChar :: Char -> Maybe Char
19−toAsciiChar c | isAscii c = Just c
20− | otherwise = M.lookup c asciiMap
20+toAsciiText :: Text -> Text
21+toAsciiText = T.filter isAscii . TN.normalize (TN.NFD)
2122
22−asciiMap :: M.Map Char Char
23−asciiMap = M.fromList
24− [('\192','A')
25− ,('\193','A')
26− ,('\194','A')
27− ,('\195','A')
28− ,('\196','A')
29− ,('\197','A')
30− ,('\199','C')
31− ,('\200','E')
32− ,('\201','E')
33− ,('\202','E')
34− ,('\203','E')
35− ,('\204','I')
36− ,('\205','I')
37− ,('\206','I')
38− ,('\207','I')
39− ,('\209','N')
40− ,('\210','O')
41− ,('\211','O')
42− ,('\212','O')
43− ,('\213','O')
44− ,('\214','O')
45− ,('\217','U')
46− ,('\218','U')
47− ,('\219','U')
48− ,('\220','U')
49− ,('\221','Y')
50− ,('\224','a')
51− ,('\225','a')
52− ,('\226','a')
53− ,('\227','a')
54− ,('\228','a')
55− ,('\229','a')
56− ,('\231','c')
57− ,('\232','e')
58− ,('\233','e')
59− ,('\234','e')
60− ,('\235','e')
61− ,('\236','i')
62− ,('\237','i')
63− ,('\238','i')
64− ,('\239','i')
65− ,('\241','n')
66− ,('\242','o')
67− ,('\243','o')
68− ,('\244','o')
69− ,('\245','o')
70− ,('\246','o')
71− ,('\249','u')
72− ,('\250','u')
73− ,('\251','u')
74− ,('\252','u')
75− ,('\253','y')
76− ,('\255','y')
77− ,('\256','A')
78− ,('\257','a')
79− ,('\258','A')
80− ,('\259','a')
81− ,('\260','A')
82− ,('\261','a')
83− ,('\262','C')
84− ,('\263','c')
85− ,('\264','C')
86− ,('\265','c')
87− ,('\266','C')
88− ,('\267','c')
89− ,('\268','C')
90− ,('\269','c')
91− ,('\270','D')
92− ,('\271','d')
93− ,('\274','E')
94− ,('\275','e')
95− ,('\276','E')
96− ,('\277','e')
97− ,('\278','E')
98− ,('\279','e')
99− ,('\280','E')
100− ,('\281','e')
101− ,('\282','E')
102− ,('\283','e')
103− ,('\284','G')
104− ,('\285','g')
105− ,('\286','G')
106− ,('\287','g')
107− ,('\288','G')
108− ,('\289','g')
109− ,('\290','G')
110− ,('\291','g')
111− ,('\292','H')
112− ,('\293','h')
113− ,('\296','I')
114− ,('\297','i')
115− ,('\298','I')
116− ,('\299','i')
117− ,('\300','I')
118− ,('\301','i')
119− ,('\302','I')
120− ,('\303','i')
121− ,('\304','I')
122− ,('\305','i')
123− ,('\308','J')
124− ,('\309','j')
125− ,('\310','K')
126− ,('\311','k')
127− ,('\313','L')
128− ,('\314','l')
129− ,('\315','L')
130− ,('\316','l')
131− ,('\317','L')
132− ,('\318','l')
133− ,('\323','N')
134− ,('\324','n')
135− ,('\325','N')
136− ,('\326','n')
137− ,('\327','N')
138− ,('\328','n')
139− ,('\332','O')
140− ,('\333','o')
141− ,('\334','O')
142− ,('\335','o')
143− ,('\336','O')
144− ,('\337','o')
145− ,('\340','R')
146− ,('\341','r')
147− ,('\342','R')
148− ,('\343','r')
149− ,('\344','R')
150− ,('\345','r')
151− ,('\346','S')
152− ,('\347','s')
153− ,('\348','S')
154− ,('\349','s')
155− ,('\350','S')
156− ,('\351','s')
157− ,('\352','S')
158− ,('\353','s')
159− ,('\354','T')
160− ,('\355','t')
161− ,('\356','T')
162− ,('\357','t')
163− ,('\360','U')
164− ,('\361','u')
165− ,('\362','U')
166− ,('\363','u')
167− ,('\364','U')
168− ,('\365','u')
169− ,('\366','U')
170− ,('\367','u')
171− ,('\368','U')
172− ,('\369','u')
173− ,('\370','U')
174− ,('\371','u')
175− ,('\372','W')
176− ,('\373','w')
177− ,('\374','Y')
178− ,('\375','y')
179− ,('\376','Y')
180− ,('\377','Z')
181− ,('\378','z')
182− ,('\379','Z')
183− ,('\380','z')
184− ,('\381','Z')
185− ,('\382','z')
186− ,('\416','O')
187− ,('\417','o')
188− ,('\431','U')
189− ,('\432','u')
190− ,('\461','A')
191− ,('\462','a')
192− ,('\463','I')
193− ,('\464','i')
194− ,('\465','O')
195− ,('\466','o')
196− ,('\467','U')
197− ,('\468','u')
198− ,('\486','G')
199− ,('\487','g')
200− ,('\488','K')
201− ,('\489','k')
202− ,('\490','O')
203− ,('\491','o')
204− ,('\496','j')
205− ,('\500','G')
206− ,('\501','g')
207− ,('\504','N')
208− ,('\505','n')
209− ,('\512','A')
210− ,('\513','a')
211− ,('\514','A')
212− ,('\515','a')
213− ,('\516','E')
214− ,('\517','e')
215− ,('\518','E')
216− ,('\519','e')
217− ,('\520','I')
218− ,('\521','i')
219− ,('\522','I')
220− ,('\523','i')
221− ,('\524','O')
222− ,('\525','o')
223− ,('\526','O')
224− ,('\527','o')
225− ,('\528','R')
226− ,('\529','r')
227− ,('\530','R')
228− ,('\531','r')
229− ,('\532','U')
230− ,('\533','u')
231− ,('\534','U')
232− ,('\535','u')
233− ,('\536','S')
234− ,('\537','s')
235− ,('\538','T')
236− ,('\539','t')
237− ,('\542','H')
238− ,('\543','h')
239− ,('\550','A')
240− ,('\551','a')
241− ,('\552','E')
242− ,('\553','e')
243− ,('\558','O')
244− ,('\559','o')
245− ,('\562','Y')
246− ,('\563','y')
247− ,('\894',';')
248− ,('\7680','A')
249− ,('\7681','a')
250− ,('\7682','B')
251− ,('\7683','b')
252− ,('\7684','B')
253− ,('\7685','b')
254− ,('\7686','B')
255− ,('\7687','b')
256− ,('\7690','D')
257− ,('\7691','d')
258− ,('\7692','D')
259− ,('\7693','d')
260− ,('\7694','D')
261− ,('\7695','d')
262− ,('\7696','D')
263− ,('\7697','d')
264− ,('\7698','D')
265− ,('\7699','d')
266− ,('\7704','E')
267− ,('\7705','e')
268− ,('\7706','E')
269− ,('\7707','e')
270− ,('\7710','F')
271− ,('\7711','f')
272− ,('\7712','G')
273− ,('\7713','g')
274− ,('\7714','H')
275− ,('\7715','h')
276− ,('\7716','H')
277− ,('\7717','h')
278− ,('\7718','H')
279− ,('\7719','h')
280− ,('\7720','H')
281− ,('\7721','h')
282− ,('\7722','H')
283− ,('\7723','h')
284− ,('\7724','I')
285− ,('\7725','i')
286− ,('\7728','K')
287− ,('\7729','k')
288− ,('\7730','K')
289− ,('\7731','k')
290− ,('\7732','K')
291− ,('\7733','k')
292− ,('\7734','L')
293− ,('\7735','l')
294− ,('\7738','L')
295− ,('\7739','l')
296− ,('\7740','L')
297− ,('\7741','l')
298− ,('\7742','M')
299− ,('\7743','m')
300− ,('\7744','M')
301− ,('\7745','m')
302− ,('\7746','M')
303− ,('\7747','m')
304− ,('\7748','N')
305− ,('\7749','n')
306− ,('\7750','N')
307− ,('\7751','n')
308− ,('\7752','N')
309− ,('\7753','n')
310− ,('\7754','N')
311− ,('\7755','n')
312− ,('\7764','P')
313− ,('\7765','p')
314− ,('\7766','P')
315− ,('\7767','p')
316− ,('\7768','R')
317− ,('\7769','r')
318− ,('\7770','R')
319− ,('\7771','r')
320− ,('\7774','R')
321− ,('\7775','r')
322− ,('\7776','S')
323− ,('\7777','s')
324− ,('\7778','S')
325− ,('\7779','s')
326− ,('\7786','T')
327− ,('\7787','t')
328− ,('\7788','T')
329− ,('\7789','t')
330− ,('\7790','T')
331− ,('\7791','t')
332− ,('\7792','T')
333− ,('\7793','t')
334− ,('\7794','U')
335− ,('\7795','u')
336− ,('\7796','U')
337− ,('\7797','u')
338− ,('\7798','U')
339− ,('\7799','u')
340− ,('\7804','V')
341− ,('\7805','v')
342− ,('\7806','V')
343− ,('\7807','v')
344− ,('\7808','W')
345− ,('\7809','w')
346− ,('\7810','W')
347− ,('\7811','w')
348− ,('\7812','W')
349− ,('\7813','w')
350− ,('\7814','W')
351− ,('\7815','w')
352− ,('\7816','W')
353− ,('\7817','w')
354− ,('\7818','X')
355− ,('\7819','x')
356− ,('\7820','X')
357− ,('\7821','x')
358− ,('\7822','Y')
359− ,('\7823','y')
360− ,('\7824','Z')
361− ,('\7825','z')
362− ,('\7826','Z')
363− ,('\7827','z')
364− ,('\7828','Z')
365− ,('\7829','z')
366− ,('\7830','h')
367− ,('\7831','t')
368− ,('\7832','w')
369− ,('\7833','y')
370− ,('\7840','A')
371− ,('\7841','a')
372− ,('\7842','A')
373− ,('\7843','a')
374− ,('\7864','E')
375− ,('\7865','e')
376− ,('\7866','E')
377− ,('\7867','e')
378− ,('\7868','E')
379− ,('\7869','e')
380− ,('\7880','I')
381− ,('\7881','i')
382− ,('\7882','I')
383− ,('\7883','i')
384− ,('\7884','O')
385− ,('\7885','o')
386− ,('\7886','O')
387− ,('\7887','o')
388− ,('\7908','U')
389− ,('\7909','u')
390− ,('\7910','U')
391− ,('\7911','u')
392− ,('\7922','Y')
393− ,('\7923','y')
394− ,('\7924','Y')
395− ,('\7925','y')
396− ,('\7926','Y')
397− ,('\7927','y')
398− ,('\7928','Y')
399− ,('\7929','y')
400− ,('\8175','`')
401− ,('\8490','K')
402− ,('\8800','=')
403− ,('\8814','<')
404− ,('\8815','>')
405− ]
23+toAsciiChar :: Char -> Maybe Char
24+toAsciiChar c = case T.unpack (TN.normalize TN.NFD (T.singleton c)) of
25+ (x:xs) | isAscii x
26+ , all isMark xs
27+ -> Just x
28+ _ -> Nothing
src/Text/Pandoc/BCP47.hsdeleted−112
@@ -1,112 +0,0 @@
1−{-# LANGUAGE OverloadedStrings #-}
2−{- |
3− Module : Text.Pandoc.BCP47
4− Copyright : Copyright (C) 2017–2020 John MacFarlane
5− License : GNU GPL, version 2 or above
6−
7− Maintainer : John MacFarlane <jgm@berkeley.edu>
8− Stability : alpha
9− Portability : portable
10−
11−Functions for parsing and rendering BCP47 language identifiers.
12−-}
13−module Text.Pandoc.BCP47 (
14− getLang
15− , parseBCP47
16− , Lang(..)
17− , renderLang
18− )
19−where
20−import Control.Monad (guard)
21−import Data.Char (isAlphaNum, isAscii, isLetter, isLower, isUpper)
22−import Text.Pandoc.Definition
23−import Text.Pandoc.Options
24−import Text.DocTemplates (FromContext(..))
25−import qualified Data.Text as T
26−import qualified Text.Parsec as P
27−
28−-- | Represents BCP 47 language/country code.
29−data Lang = Lang{ langLanguage :: T.Text
30− , langScript :: T.Text
31− , langRegion :: T.Text
32− , langVariants :: [T.Text] }
33− deriving (Eq, Ord, Show)
34−
35−-- | Render a Lang as BCP 47.
36−renderLang :: Lang -> T.Text
37−renderLang lang = T.intercalate "-" (langLanguage lang : filter (not . T.null)
38− ([langScript lang, langRegion lang] ++ langVariants lang))
39−
40−-- | Get the contents of the `lang` metadata field or variable.
41−getLang :: WriterOptions -> Meta -> Maybe T.Text
42−getLang opts meta =
43− case lookupContext "lang" (writerVariables opts) of
44− Just s -> Just s
45− _ ->
46− case lookupMeta "lang" meta of
47− Just (MetaBlocks [Para [Str s]]) -> Just s
48− Just (MetaBlocks [Plain [Str s]]) -> Just s
49− Just (MetaInlines [Str s]) -> Just s
50− Just (MetaString s) -> Just s
51− _ -> Nothing
52−
53−-- | Parse a BCP 47 string as a Lang. Currently we parse
54−-- extensions and private-use fields as "variants," even
55−-- though officially they aren't.
56−parseBCP47 :: T.Text -> Either T.Text Lang
57−parseBCP47 lang =
58− case P.parse bcp47 "lang" lang of
59− Right r -> Right r
60− Left e -> Left $ T.pack $ show e
61− where bcp47 = do
62− language <- pLanguage
63− script <- P.option "" pScript
64− region <- P.option "" pRegion
65− variants <- P.many (pVariant P.<|> pExtension P.<|> pPrivateUse)
66− P.eof
67− return Lang{ langLanguage = language
68− , langScript = script
69− , langRegion = region
70− , langVariants = variants }
71− asciiLetter = P.satisfy (\c -> isAscii c && isLetter c)
72− pLanguage = do
73− cs <- P.many1 asciiLetter
74− let lcs = length cs
75− guard $ lcs == 2 || lcs == 3
76− return $ T.toLower $ T.pack cs
77− pScript = P.try $ do
78− P.char '-'
79− x <- P.satisfy (\c -> isAscii c && isLetter c && isUpper c)
80− xs <- P.count 3
81− (P.satisfy (\c -> isAscii c && isLetter c && isLower c))
82− return $ T.toLower $ T.pack (x:xs)
83− pRegion = P.try $ do
84− P.char '-'
85− cs <- P.many1 asciiLetter
86− let lcs = length cs
87− guard $ lcs == 2 || lcs == 3
88− return $ T.toUpper $ T.pack cs
89− pVariant = P.try $ do
90− P.char '-'
91− ds <- P.option "" (P.count 1 P.digit)
92− cs <- P.many1 asciiLetter
93− let var = ds ++ cs
94− lv = length var
95− guard $ if null ds
96− then lv >= 5 && lv <= 8
97− else lv == 4
98− return $ T.toLower $ T.pack var
99− pExtension = P.try $ do
100− P.char '-'
101− cs <- P.many1 $ P.satisfy (\c -> isAscii c && isAlphaNum c)
102− let lcs = length cs
103− guard $ lcs >= 2 && lcs <= 8
104− return $ T.toLower $ T.pack cs
105− pPrivateUse = P.try $ do
106− P.char '-'
107− P.char 'x'
108− P.char '-'
109− cs <- P.many1 $ P.satisfy (\c -> isAscii c && isAlphaNum c)
110− guard $ not (null cs) && length cs <= 8
111− let var = "x-" ++ cs
112− return $ T.toLower $ T.pack var
src/Text/Pandoc/CSS.hs+1−1
@@ -1,6 +1,6 @@
11{- |
22Module : Text.Pandoc.CSS
3−Copyright : © 2006-2020 John MacFarlane <jgm@berkeley.edu>,
3+Copyright : © 2006-2021 John MacFarlane <jgm@berkeley.edu>,
44 2015-2016 Mauro Bieg,
55 2015 Ophir Lifshitz <hangfromthefloor@gmail.com>
66License : GNU GPL, version 2 or above
src/Text/Pandoc/CSV.hs+2−3
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.CSV
3− Copyright : Copyright (C) 2017–2020 John MacFarlane <jgm@berkeley.edu>
3+ Copyright : Copyright (C) 2017-2021 John MacFarlane <jgm@berkeley.edu>
44 License : GNU GPL, version 2 or above
55 Maintainer : John MacFarlane <jgm@berkeley.edu>
66 Stability : alpha
@@ -68,8 +68,7 @@ escaped opts = try $
6868
6969pCSVUnquotedCell :: CSVOptions -> Parser Text
7070pCSVUnquotedCell opts = T.pack <$>
71− many (satisfy (\c -> c /= csvDelim opts && c /= '\r' && c /= '\n'
72− && c /= csvQuote opts))
71+ many (satisfy (\c -> c /= csvDelim opts && c /= '\r' && c /= '\n'))
7372
7473pCSVDelim :: CSVOptions -> Parser ()
7574pCSVDelim opts = do
src/Text/Pandoc/Citeproc.hs+185−123
@@ -5,7 +5,10 @@
55{-# LANGUAGE FlexibleInstances #-}
66{-# LANGUAGE OverloadedStrings #-}
77module Text.Pandoc.Citeproc
8− ( processCitations )
8+ ( processCitations,
9+ getReferences,
10+ getStyle
11+ )
912where
1013
1114import Citeproc
@@ -15,39 +18,104 @@ import Text.Pandoc.Citeproc.CslJson (cslJsonToReferences)
1518import Text.Pandoc.Citeproc.BibTeX (readBibtexString, Variant(..))
1619import Text.Pandoc.Citeproc.MetaValue (metaValueToReference, metaValueToText)
1720import Text.Pandoc.Readers.Markdown (yamlToRefs)
18−import Text.Pandoc.Class (setResourcePath, getResourcePath, getUserDataDir)
19−import qualified Text.Pandoc.BCP47 as BCP47
20−import Data.ByteString (ByteString)
21−import qualified Data.ByteString.Lazy as L
21+import Text.Pandoc.Builder (Inlines, Many(..), deleteMeta, setMeta)
22+import qualified Text.Pandoc.Builder as B
2223import Text.Pandoc.Definition as Pandoc
23−import Text.Pandoc.Walk
24−import Text.Pandoc.Builder as B
25−import Text.Pandoc (PandocMonad(..), PandocError(..),
26− readDataFile, ReaderOptions(..), pandocExtensions,
27− report, LogMessage(..), fetchItem)
28−import Text.Pandoc.Shared (stringify, ordNub, blocksToInlines, tshow)
24+import Text.Pandoc.Class (PandocMonad(..), getResourcePath, getUserDataDir,
25+ fetchItem, readDataFile, report, setResourcePath)
26+import Text.Pandoc.Error (PandocError(..))
27+import Text.Pandoc.Extensions (pandocExtensions)
28+import Text.Pandoc.Logging (LogMessage(..))
29+import Text.Pandoc.Options (ReaderOptions(..))
30+import Text.Pandoc.Shared (stringify, ordNub, tshow)
2931import qualified Text.Pandoc.UTF8 as UTF8
32+import Text.Pandoc.Walk (query, walk, walkM)
33+import Control.Applicative ((<|>))
34+import Control.Monad.Except (catchError, throwError)
35+import Control.Monad.State (State, evalState, get, put, runState)
3036import Data.Aeson (eitherDecode)
31−import Data.Default
32−import Data.Ord ()
37+import Data.ByteString (ByteString)
38+import qualified Data.ByteString.Lazy as L
39+import Data.Char (isPunctuation, isUpper)
40+import Data.Default (Default(def))
41+import qualified Data.Foldable as Foldable
3342import qualified Data.Map as M
43+import Data.Maybe (mapMaybe, fromMaybe)
44+import Data.Ord ()
45+import qualified Data.Sequence as Seq
3446import qualified Data.Set as Set
35−import Data.Char (isPunctuation, isUpper)
3647import Data.Text (Text)
3748import qualified Data.Text as T
38−import Control.Monad.State
39−import qualified Data.Sequence as Seq
40−import qualified Data.Foldable as Foldable
41−import System.FilePath
42−import Control.Applicative
43−import Control.Monad.Except
44−import Data.Maybe (mapMaybe, fromMaybe)
49+import System.FilePath (takeExtension)
4550import Safe (lastMay, initSafe)
46−-- import Debug.Trace as Trace (trace, traceShowId)
4751
4852
49−processCitations :: PandocMonad m => Pandoc -> m Pandoc
53+processCitations :: PandocMonad m => Pandoc -> m Pandoc
5054processCitations (Pandoc meta bs) = do
55+ style <- getStyle (Pandoc meta bs)
56+
57+ mblang <- getLang meta
58+ let locale = Citeproc.mergeLocales mblang style
59+
60+ refs <- getReferences (Just locale) (Pandoc meta bs)
61+
62+ let otherIdsMap = foldr (\ref m ->
63+ case T.words . extractText <$>
64+ M.lookup "other-ids"
65+ (referenceVariables ref) of
66+ Nothing -> m
67+ Just ids -> foldr
68+ (\id' ->
69+ M.insert id' (referenceId ref)) m ids)
70+ M.empty refs
71+ let meta' = deleteMeta "nocite" meta
72+ let citations = getCitations locale otherIdsMap $ Pandoc meta' bs
73+
74+
75+ let linkCites = maybe False truish $ lookupMeta "link-citations" meta
76+ let opts = defaultCiteprocOptions{ linkCitations = linkCites }
77+ let result = Citeproc.citeproc opts style mblang refs citations
78+ mapM_ (report . CiteprocWarning) (resultWarnings result)
79+ let sopts = styleOptions style
80+ let classes = "references" : -- TODO remove this or keep for compatibility?
81+ "csl-bib-body" :
82+ ["hanging-indent" | styleHangingIndent sopts]
83+ let refkvs = (case styleEntrySpacing sopts of
84+ Just es | es > 0 -> (("entry-spacing",T.pack $ show es):)
85+ _ -> id) .
86+ (case styleLineSpacing sopts of
87+ Just ls | ls > 1 -> (("line-spacing",T.pack $ show ls):)
88+ _ -> id) $ []
89+ let bibs = mconcat $ map (\(ident, out) ->
90+ B.divWith ("ref-" <> ident,["csl-entry"],[]) . B.para .
91+ walk (convertQuotes locale) .
92+ insertSpace $ out)
93+ (resultBibliography result)
94+ let moveNotes = styleIsNoteStyle sopts &&
95+ maybe True truish (lookupMeta "notes-after-punctuation" meta)
96+ let cits = map (walk (convertQuotes locale)) $
97+ resultCitations result
98+
99+ let metanocites = lookupMeta "nocite" meta
100+ let Pandoc meta'' bs' =
101+ maybe id (setMeta "nocite") metanocites .
102+ walk (mvPunct moveNotes locale) .
103+ (if styleIsNoteStyle sopts
104+ then walk addNote . walk deNote
105+ else id) .
106+ evalState (walkM insertResolvedCitations $ Pandoc meta' bs)
107+ $ cits
108+ return $ Pandoc meta''
109+ $ insertRefs refkvs classes meta''
110+ (walk fixLinks $ B.toList bibs) bs'
111+
112+-- | Retrieve the CSL style specified by the csl or citation-style
113+-- metadata field in a pandoc document, or the default CSL style
114+-- if none is specified. Retrieve the parent style
115+-- if the style is a dependent style. Add abbreviations defined
116+-- in an abbreviation file if one has been specified.
117+getStyle :: PandocMonad m => Pandoc -> m (Style Inlines)
118+getStyle (Pandoc meta _) = do
51119 let cslfile = (lookupMeta "csl" meta <|> lookupMeta "citation-style" meta)
52120 >>= metaValueToText
53121
@@ -88,25 +156,44 @@ processCitations (Pandoc meta bs) = do
88156 catchError (getFile ".csl" basename) (\_ -> fst <$> fetchItem url)
89157
90158 styleRes <- Citeproc.parseStyle getParentStyle cslContents
91− style <-
92− case styleRes of
93− Left err -> throwError $ PandocAppError $ prettyCiteprocError err
94− Right style -> return style{ styleAbbreviations = mbAbbrevs }
95− mblang <- maybe (return Nothing) bcp47LangToIETF
96− ((lookupMeta "lang" meta <|> lookupMeta "locale" meta) >>=
97− metaValueToText)
98− let locale = Citeproc.mergeLocales mblang style
159+ case styleRes of
160+ Left err -> throwError $ PandocAppError $ prettyCiteprocError err
161+ Right style -> return style{ styleAbbreviations = mbAbbrevs }
162+
163+
164+-- Retrieve citeproc lang based on metadata.
165+getLang :: PandocMonad m => Meta -> m (Maybe Lang)
166+getLang meta = maybe (return Nothing) bcp47LangToIETF
167+ ((lookupMeta "lang" meta <|> lookupMeta "locale" meta) >>=
168+ metaValueToText)
169+
170+-- | Get references defined inline in the metadata and via an external
171+-- bibliography. Only references that are actually cited in the document
172+-- (either with a genuine citation or with `nocite`) are returned.
173+-- URL variables are converted to links.
174+getReferences :: PandocMonad m
175+ => Maybe Locale -> Pandoc -> m [Reference Inlines]
176+getReferences mblocale (Pandoc meta bs) = do
177+ locale <- case mblocale of
178+ Just l -> return l
179+ Nothing -> do
180+ mblang <- getLang meta
181+ case mblang of
182+ Just lang -> return $ either mempty id $ getLocale lang
183+ Nothing -> return mempty
184+
99185 let getCiteId (Cite cs _) = Set.fromList $ map B.citationId cs
100186 getCiteId _ = mempty
101187 let metanocites = lookupMeta "nocite" meta
102− let meta' = deleteMeta "nocite" meta
103188 let nocites = maybe mempty (query getCiteId) metanocites
104189 let citeIds = query getCiteId (Pandoc meta bs)
105190 let idpred = if "*" `Set.member` nocites
106191 then const True
107192 else (`Set.member` citeIds)
108193 let inlineRefs = case lookupMeta "references" meta of
109− Just (MetaList rs) -> mapMaybe metaValueToReference rs
194+ Just (MetaList rs) ->
195+ filter (idpred . unItemId . referenceId)
196+ $ mapMaybe metaValueToReference rs
110197 _ -> []
111198 externalRefs <- case lookupMeta "bibliography" meta of
112199 Just (MetaList xs) ->
@@ -118,59 +205,11 @@ processCitations (Pandoc meta bs) = do
118205 Just fp -> getRefsFromBib locale idpred fp
119206 Nothing -> return []
120207 Nothing -> return []
121− let refs = map (linkifyVariables . legacyDateRanges)
122− (externalRefs ++ inlineRefs)
123− -- note that inlineRefs can override externalRefs
124− let otherIdsMap = foldr (\ref m ->
125− case T.words . extractText <$>
126− M.lookup "other-ids"
127− (referenceVariables ref) of
128− Nothing -> m
129− Just ids -> foldr
130− (\id' ->
131− M.insert id' (referenceId ref)) m ids)
132− M.empty refs
133− -- TODO: issue warning if no refs defined
134− let citations = getCitations locale otherIdsMap $ Pandoc meta' bs
135− let linkCites = maybe False truish $ lookupMeta "link-citations" meta
136− let opts = defaultCiteprocOptions{ linkCitations = linkCites }
137− let result = Citeproc.citeproc opts style (localeLanguage locale)
138− refs citations
139− mapM_ (report . CiteprocWarning) (resultWarnings result)
140− let sopts = styleOptions style
141− let classes = "references" : -- TODO remove this or keep for compatibility?
142− "csl-bib-body" :
143− ["hanging-indent" | styleHangingIndent sopts]
144− let refkvs = (case styleEntrySpacing sopts of
145− Just es | es > 0 -> (("entry-spacing",T.pack $ show es):)
146− _ -> id) .
147− (case styleLineSpacing sopts of
148− Just ls | ls > 1 -> (("line-spacing",T.pack $ show ls):)
149− _ -> id) $ []
150− let bibs = mconcat $ map (\(ident, out) ->
151− B.divWith ("ref-" <> ident,["csl-entry"],[]) . B.para .
152− walk (convertQuotes locale) . insertSpace $ out)
153− (resultBibliography result)
154− let moveNotes = maybe True truish $
155− lookupMeta "notes-after-punctuation" meta
156− let cits = map (walk fixLinks . walk (convertQuotes locale)) $
157− resultCitations result
208+ return $ map (linkifyVariables . legacyDateRanges)
209+ (externalRefs ++ inlineRefs)
210+ -- note that inlineRefs can override externalRefs
158211
159− let fixQuotes = case localePunctuationInQuote locale of
160− Just True ->
161− B.toList . movePunctuationInsideQuotes . B.fromList
162− _ -> id
163212
164− let Pandoc meta'' bs' =
165− maybe id (setMeta "nocite") metanocites .
166− walk (map capitalizeNoteCitation .
167− fixQuotes . mvPunct moveNotes locale) .
168− walk deNote .
169− evalState (walkM insertResolvedCitations $ Pandoc meta' bs)
170− $ cits
171− return $ Pandoc meta''
172− $ insertRefs refkvs classes meta''
173− (walk fixLinks $ B.toList bibs) bs'
174213
175214-- If we have a span.csl-left-margin followed by span.csl-right-inline,
176215-- we insert a space. This ensures that they will be separated by a space,
@@ -251,7 +290,7 @@ insertResolvedCitations (Cite cs ils) = do
251290 [] -> return (Cite cs ils)
252291 (x:xs) -> do
253292 put xs
254− return $ Cite cs (B.toList x)
293+ return $ Cite cs (walk fixLinks $ B.toList x)
255294insertResolvedCitations x = return x
256295
257296getCitations :: Locale
@@ -333,7 +372,6 @@ formatFromExtension fp = case dropWhile (== '.') $ takeExtension fp of
333372
334373
335374isNote :: Inline -> Bool
336−isNote (Note _) = True
337375isNote (Cite _ [Note _]) = True
338376 -- the following allows citation styles that are "in-text" but use superscript
339377 -- references to be treated as if they are "notes" for the purposes of moving
@@ -346,6 +384,12 @@ isSpacy Space = True
346384isSpacy SoftBreak = True
347385isSpacy _ = False
348386
387+movePunctInsideQuotes :: Locale -> [Inline] -> [Inline]
388+movePunctInsideQuotes locale
389+ | localePunctuationInQuote locale == Just True
390+ = B.toList . movePunctuationInsideQuotes . B.fromList
391+ | otherwise
392+ = id
349393
350394mvPunct :: Bool -> Locale -> [Inline] -> [Inline]
351395mvPunct moveNotes locale (x : xs)
@@ -358,7 +402,8 @@ mvPunct moveNotes locale (q : s : x : ys)
358402 in if moveNotes
359403 then if T.null spunct
360404 then q : x : mvPunct moveNotes locale ys
361− else q : Str spunct : x : mvPunct moveNotes locale
405+ else movePunctInsideQuotes locale
406+ [q , Str spunct , x] ++ mvPunct moveNotes locale
362407 (B.toList
363408 (dropTextWhile isPunctuation (B.fromList ys)))
364409 else q : x : mvPunct moveNotes locale ys
@@ -370,9 +415,10 @@ mvPunct moveNotes locale (Cite cs ils : ys)
370415 , moveNotes
371416 = let s = stringify ys
372417 spunct = T.takeWhile isPunctuation s
373− in Cite cs (init ils
374− ++ [Str spunct | not (endWithPunct False (init ils))]
375− ++ [last ils]) :
418+ in Cite cs (movePunctInsideQuotes locale $
419+ init ils
420+ ++ [Str spunct | not (endWithPunct False (init ils))]
421+ ++ [last ils]) :
376422 mvPunct moveNotes locale
377423 (B.toList (dropTextWhile isPunctuation (B.fromList ys)))
378424mvPunct moveNotes locale (s : x : ys) | isSpacy s, isNote x =
@@ -388,7 +434,7 @@ mvPunct _ _ [] = []
388434-- move https://doi.org etc. prefix inside link text (#6723):
389435fixLinks :: [Inline] -> [Inline]
390436fixLinks (Str t : Link attr [Str u1] (u2,tit) : xs)
391− | t <> u1 == u2
437+ | u2 == t <> u1
392438 = Link attr [Str (t <> u1)] (u2,tit) : fixLinks xs
393439fixLinks (x:xs) = x : fixLinks xs
394440fixLinks [] = []
@@ -458,7 +504,8 @@ insertRefs refkvs refclasses meta refs bs =
458504 put True
459505 -- refHeader isn't used if you have an explicit references div
460506 let cs' = ordNub $ cs ++ refclasses
461− return $ Div ("refs",cs',kvs) (xs ++ refs)
507+ let kvs' = ordNub $ kvs ++ refkvs
508+ return $ Div ("refs",cs',kvs') (xs ++ refs)
462509 go x = return x
463510
464511refTitle :: Meta -> Maybe [Inline]
@@ -501,7 +548,7 @@ linkifyVariables ref =
501548 fixShortDOI x = let x' = extractText x
502549 in if "10/" `T.isPrefixOf` x'
503550 then TextVal $ T.drop 3 x'
504− -- see http://shortdoi.org
551+ -- see https://shortdoi.org
505552 else TextVal x'
506553 tolink pref x = let x' = extractText x
507554 x'' = if "://" `T.isInfixOf` x'
@@ -517,42 +564,62 @@ extractText (FancyVal x) = toText x
517564extractText (NumVal n) = T.pack (show n)
518565extractText _ = mempty
519566
520−capitalizeNoteCitation :: Inline -> Inline
521−capitalizeNoteCitation (Cite cs [Note [Para ils]]) =
522− Cite cs
523− [Note [Para $ B.toList $ addTextCase Nothing CapitalizeFirst
524− $ B.fromList ils]]
525−capitalizeNoteCitation x = x
526−
527−deNote :: [Inline] -> [Inline]
528−deNote [] = []
529−deNote (Note bs:rest) =
530− Note (walk go bs) : deNote rest
567+-- Here we take the Spans with class csl-note that are left
568+-- after deNote has removed nested ones, and convert them
569+-- into real notes.
570+addNote :: Inline -> Inline
571+addNote (Span ("",["csl-note"],[]) ils) =
572+ Note [Para $
573+ B.toList . addTextCase Nothing CapitalizeFirst . B.fromList $ ils]
574+addNote x = x
575+
576+-- Here we handle citation notes that occur inside footnotes
577+-- or other citation notes, in a note style. We don't want
578+-- notes inside notes, so we convert these to parenthesized
579+-- or comma-separated citations.
580+deNote :: Inline -> Inline
581+deNote (Note bs) =
582+ case bs of
583+ [Para (cit@(Cite (c:_) _) : ils)]
584+ | citationMode c /= AuthorInText ->
585+ -- if citation is first in note, no need to parenthesize.
586+ Note [Para (walk removeNotes $ cit : walk addParens ils)]
587+ _ -> Note (walk removeNotes . walk addParens $ bs)
588+
531589 where
532− go [] = []
533− go (Cite (c:cs) ils : zs)
590+ addParens [] = []
591+ addParens (Cite (c:cs) ils : zs)
534592 | citationMode c == AuthorInText
535− = Cite (c:cs) (concatMap (noteAfterComma (needsPeriod zs)) ils) : go zs
593+ = Cite (c:cs) (concatMap (noteAfterComma (needsPeriod zs)) ils) :
594+ addParens zs
536595 | otherwise
537− = Cite (c:cs) (concatMap noteInParens ils) : go zs
538− go (x:xs) = x : go xs
596+ = Cite (c:cs) (concatMap noteInParens ils) : addParens zs
597+ addParens (x:xs) = x : addParens xs
598+
599+ removeNotes (Span ("",["csl-note"],[]) ils) = Span ("",[],[]) ils
600+ removeNotes x = x
601+
539602 needsPeriod [] = True
540603 needsPeriod (Str t:_) = case T.uncons t of
541604 Nothing -> False
542605 Just (c,_) -> isUpper c
543606 needsPeriod (Space:zs) = needsPeriod zs
544607 needsPeriod _ = False
545− noteInParens (Note bs')
608+
609+ noteInParens (Span ("",["csl-note"],[]) ils)
546610 = Space : Str "(" :
547− removeFinalPeriod (blocksToInlines bs') ++ [Str ")"]
611+ removeFinalPeriod ils ++ [Str ")"]
548612 noteInParens x = [x]
549− noteAfterComma needsPer (Note bs')
613+
614+ noteAfterComma needsPer (Span ("",["csl-note"],[]) ils)
615+ | not (null ils)
550616 = Str "," : Space :
551− (if needsPer
552− then id
553− else removeFinalPeriod) (blocksToInlines bs')
617+ if needsPer
618+ then ils
619+ else removeFinalPeriod ils
554620 noteAfterComma _ x = [x]
555−deNote (x:xs) = x : deNote xs
621+
622+deNote x = x
556623
557624-- Note: we can't use dropTextWhileEnd indiscriminately,
558625-- because this would remove the final period on abbreviations like Ibid.
@@ -587,13 +654,8 @@ removeFinalPeriod ils =
587654
588655bcp47LangToIETF :: PandocMonad m => Text -> m (Maybe Lang)
589656bcp47LangToIETF bcplang =
590− case BCP47.parseBCP47 bcplang of
657+ case parseLang bcplang of
591658 Left _ -> do
592659 report $ InvalidLang bcplang
593660 return Nothing
594− Right lang ->
595− return $ Just
596− $ Lang (BCP47.langLanguage lang)
597− (if T.null (BCP47.langRegion lang)
598− then Nothing
599− else Just (BCP47.langRegion lang))
661+ Right lang -> return $ Just lang
src/Text/Pandoc/Citeproc/BibTeX.hs+297−56
@@ -1,4 +1,5 @@
11{-# LANGUAGE ViewPatterns #-}
2+{-# LANGUAGE LambdaCase #-}
23{-# LANGUAGE OverloadedStrings #-}
34{-# LANGUAGE ScopedTypeVariables #-}
45{-# LANGUAGE FlexibleContexts #-}
@@ -17,6 +18,7 @@
1718module Text.Pandoc.Citeproc.BibTeX
1819 ( Variant(..)
1920 , readBibtexString
21+ , writeBibtexString
2022 )
2123 where
2224
@@ -24,10 +26,11 @@ import Text.Pandoc.Definition
2426import Text.Pandoc.Builder as B
2527import Text.Pandoc.Readers.LaTeX (readLaTeX)
2628import Text.Pandoc.Extensions (Extension(..), extensionsFromList)
27−import Text.Pandoc.Options (ReaderOptions(..))
28−import Text.Pandoc.Class (runPure)
29+import Text.Pandoc.Options (ReaderOptions(..), WriterOptions)
2930import Text.Pandoc.Error (PandocError)
3031import Text.Pandoc.Shared (stringify)
32+import Text.Pandoc.Writers.LaTeX (writeLaTeX)
33+import Text.Pandoc.Class (runPure)
3134import qualified Text.Pandoc.Walk as Walk
3235import Citeproc.Types
3336import Citeproc.Pandoc ()
@@ -46,17 +49,21 @@ import qualified Data.Sequence as Seq
4649import Data.Char (isAlphaNum, isDigit, isLetter,
4750 isUpper, toLower, toUpper,
4851 isLower, isPunctuation)
49−import Data.List (foldl', intercalate)
52+import Data.List (foldl', intercalate, intersperse)
5053import Safe (readMay)
54+import Text.Printf (printf)
55+import Text.DocLayout (literal, hsep, nest, hang, Doc(..),
56+ braces, ($$), cr)
5157
5258data Variant = Bibtex | Biblatex
5359 deriving (Show, Eq, Ord)
5460
5561-- | Parse BibTeX or BibLaTeX into a list of 'Reference's.
56−readBibtexString :: Variant -- ^ bibtex or biblatex
62+readBibtexString :: ToSources a
63+ => Variant -- ^ bibtex or biblatex
5764 -> Locale -- ^ Locale
5865 -> (Text -> Bool) -- ^ Filter on citation ids
59− -> Text -- ^ bibtex/biblatex text
66+ -> a -- ^ bibtex/biblatex text
6067 -> Either ParseError [Reference Inlines]
6168readBibtexString variant locale idpred contents = do
6269 case runParser (((resolveCrossRefs variant <$> bibEntries) <* eof) >>=
@@ -64,17 +71,280 @@ readBibtexString variant locale idpred contents = do
6471 filter (\item -> idpred (identifier item) &&
6572 entryType item /= "xdata"))
6673 (fromMaybe defaultLang $ localeLanguage locale, Map.empty)
67− "" contents of
74+ (initialSourceName sources) sources of
6875 Left err -> Left err
6976 Right xs -> return xs
77+ where
78+ sources = toSources contents
79+
80+-- | Write BibTeX or BibLaTeX given given a 'Reference'.
81+writeBibtexString :: WriterOptions -- ^ options (for writing LaTex)
82+ -> Variant -- ^ bibtex or biblatex
83+ -> Maybe Lang -- ^ Language
84+ -> Reference Inlines -- ^ Reference to write
85+ -> Doc Text
86+writeBibtexString opts variant mblang ref =
87+ "@" <> bibtexType <> "{" <> literal (unItemId (referenceId ref)) <> ","
88+ $$ nest 2 (renderFields fs)
89+ $$ "}" <> cr
90+
91+ where
92+ bibtexType =
93+ case referenceType ref of
94+ "article-magazine" -> "article"
95+ "article-newspaper" -> "article"
96+ "article-journal" -> "article"
97+ "book" -> "book"
98+ "pamphlet" -> "booklet"
99+ "dataset" | variant == Biblatex -> "dataset"
100+ "webpage" | variant == Biblatex -> "online"
101+ "chapter" -> case getVariable "editor" of
102+ Just _ -> "incollection"
103+ Nothing -> "inbook"
104+ "entry-encyclopedia" | variant == Biblatex -> "inreference"
105+ | otherwise -> "inbook"
106+ "paper-conference" -> "inproceedings"
107+ "thesis" -> case getVariableAsText "genre" of
108+ Just "mathesis" -> "mastersthesis"
109+ _ -> "phdthesis"
110+ "patent" | variant == Biblatex -> "patent"
111+ "report" | variant == Biblatex -> "report"
112+ | otherwise -> "techreport"
113+ "speech" -> "unpublished"
114+ "manuscript" -> "unpublished"
115+ "graphic" | variant == Biblatex -> "artwork"
116+ "song" | variant == Biblatex -> "music"
117+ "legal_case" | variant == Biblatex -> "jurisdictionN"
118+ "legislation" | variant == Biblatex -> "legislation"
119+ "treaty" | variant == Biblatex -> "legal"
120+ "personal_communication" | variant == Biblatex -> "letter"
121+ "motion_picture" | variant == Biblatex -> "movie"
122+ "review" | variant == Biblatex -> "review"
123+ _ -> "misc"
124+
125+ mbSubtype =
126+ case referenceType ref of
127+ "article-magazine" -> Just "magazine"
128+ "article-newspaper" -> Just "newspaper"
129+ _ -> Nothing
130+
131+ fs =
132+ case variant of
133+ Biblatex ->
134+ [ "author"
135+ , "editor"
136+ , "translator"
137+ , "publisher"
138+ , "title"
139+ , "booktitle"
140+ , "journal"
141+ , "series"
142+ , "edition"
143+ , "volume"
144+ , "volumes"
145+ , "number"
146+ , "pages"
147+ , "date"
148+ , "eventdate"
149+ , "urldate"
150+ , "address"
151+ , "url"
152+ , "doi"
153+ , "isbn"
154+ , "issn"
155+ , "type"
156+ , "entrysubtype"
157+ , "note"
158+ , "langid"
159+ , "abstract"
160+ , "keywords"
161+ , "annote"
162+ ]
163+ Bibtex ->
164+ [ "author"
165+ , "editor"
166+ , "translator"
167+ , "publisher"
168+ , "title"
169+ , "booktitle"
170+ , "journal"
171+ , "series"
172+ , "edition"
173+ , "volume"
174+ , "number"
175+ , "pages"
176+ , "year"
177+ , "month"
178+ , "address"
179+ , "type"
180+ , "note"
181+ , "annote"
182+ ]
183+
184+ valToInlines (TextVal t) = B.text t
185+ valToInlines (FancyVal ils) = ils
186+ valToInlines (NumVal n) = B.text (T.pack $ show n)
187+ valToInlines (NamesVal names) =
188+ mconcat $ intersperse (B.space <> B.text "and" <> B.space)
189+ $ map renderName names
190+ valToInlines (DateVal date) = B.text $
191+ case dateLiteral date of
192+ Just t -> t
193+ Nothing -> T.intercalate "/" (map renderDatePart (dateParts date)) <>
194+ (if dateCirca date then "~" else mempty)
195+
196+ renderDatePart (DateParts xs) = T.intercalate "-" $
197+ map (T.pack . printf "%02d") xs
198+
199+ renderName name =
200+ case nameLiteral name of
201+ Just t -> B.text t
202+ Nothing -> spacedMaybes
203+ [ nameNonDroppingParticle name
204+ , nameFamily name
205+ , if nameCommaSuffix name
206+ then (", " <>) <$> nameSuffix name
207+ else nameSuffix name ]
208+ <>
209+ spacedMaybes
210+ [ (", " <>) <$> nameGiven name,
211+ nameDroppingParticle name ]
212+
213+ mblang' = case getVariableAsText "language" of
214+ Just l -> either (const Nothing) Just $ parseLang l
215+ Nothing -> mblang
216+
217+ titlecase = case mblang' of
218+ Just lang | langLanguage lang == "en"
219+ -> titlecase'
220+ Nothing -> titlecase'
221+ _ ->
222+ case variant of
223+ Bibtex -> B.spanWith nullAttr
224+ -- BibTex lacks a language field, so we wrap non-English
225+ -- titles in {} to protect case.
226+ Biblatex -> id
227+
228+ titlecase' = addTextCase mblang' TitleCase .
229+ (\ils -> B.fromList
230+ (case B.toList ils of
231+ Str t : xs -> Str t : Walk.walk spanAroundCapitalizedWords xs
232+ xs -> Walk.walk spanAroundCapitalizedWords xs))
233+
234+ -- protect capitalized words when we titlecase
235+ spanAroundCapitalizedWords (Str t)
236+ | not (T.all (\c -> isLower c || not (isLetter c)) t) =
237+ Span ("",["nocase"],[]) [Str t]
238+ spanAroundCapitalizedWords x = x
239+
240+ spacedMaybes = mconcat . intersperse B.space . mapMaybe (fmap B.text)
241+
242+ toLaTeX x =
243+ case runPure (writeLaTeX opts $ doc (B.plain x)) of
244+ Left _ -> Nothing
245+ Right t -> Just $ hsep . map literal $ T.words t
246+
247+ renderField :: Text -> Maybe (Doc Text)
248+ renderField name =
249+ (((literal name) <>) . hang 2 " = " . braces)
250+ <$> getContentsFor name
251+
252+ getVariable v = lookupVariable (toVariable v) ref
253+
254+ getVariableAsText v = (stringify . valToInlines) <$> getVariable v
255+
256+ getYear val =
257+ case val of
258+ DateVal date ->
259+ case dateLiteral date of
260+ Just t -> toLaTeX (B.text t)
261+ Nothing ->
262+ case dateParts date of
263+ [DateParts (y1:_), DateParts (y2:_)] ->
264+ Just $ literal (T.pack (printf "%04d" y1) <> "--" <>
265+ T.pack (printf "%04d" y2))
266+ [DateParts (y1:_)] ->
267+ Just $ literal (T.pack (printf "%04d" y1))
268+ _ -> Nothing
269+ _ -> Nothing
270+
271+ toMonth 1 = "jan"
272+ toMonth 2 = "feb"
273+ toMonth 3 = "mar"
274+ toMonth 4 = "apr"
275+ toMonth 5 = "may"
276+ toMonth 6 = "jun"
277+ toMonth 7 = "jul"
278+ toMonth 8 = "aug"
279+ toMonth 9 = "sep"
280+ toMonth 10 = "oct"
281+ toMonth 11 = "nov"
282+ toMonth 12 = "dec"
283+ toMonth x = T.pack $ show x
284+
285+ getMonth val =
286+ case val of
287+ DateVal date ->
288+ case dateParts date of
289+ [DateParts (_:m1:_), DateParts (_:m2:_)] ->
290+ Just $ literal (toMonth m1 <> "--" <> toMonth m2)
291+ [DateParts (_:m1:_)] -> Just $ literal (toMonth m1)
292+ _ -> Nothing
293+ _ -> Nothing
294+
295+ getContentsFor :: Text -> Maybe (Doc Text)
296+ getContentsFor "type" =
297+ getVariableAsText "genre" >>=
298+ \case
299+ "mathesis" -> Just "mastersthesis"
300+ "phdthesis" -> Just "phdthesis"
301+ _ -> Nothing
302+ getContentsFor "entrysubtype" = literal <$> mbSubtype
303+ getContentsFor "journal"
304+ | bibtexType `elem` ["article", "periodical", "suppperiodical", "review"]
305+ = getVariable "container-title" >>= toLaTeX . valToInlines
306+ | otherwise = Nothing
307+ getContentsFor "booktitle"
308+ | bibtexType `elem`
309+ ["inbook","incollection","inproceedings","inreference","bookinbook"]
310+ = (getVariable "volume-title" <|> getVariable "container-title")
311+ >>= toLaTeX . valToInlines
312+ | otherwise = Nothing
313+ getContentsFor "series" = getVariable "collection-title"
314+ >>= toLaTeX . valToInlines
315+ getContentsFor "address" = getVariable "publisher-place"
316+ >>= toLaTeX . valToInlines
317+ getContentsFor "date" = getVariable "issued" >>= toLaTeX . valToInlines
318+ getContentsFor "eventdate" = getVariable "event-date" >>= toLaTeX . valToInlines
319+ getContentsFor "urldate" = getVariable "accessed" >>= toLaTeX . valToInlines
320+ getContentsFor "year" = getVariable "issued" >>= getYear
321+ getContentsFor "month" = getVariable "issued" >>= getMonth
322+ getContentsFor "pages" = getVariable "page" >>= toLaTeX . valToInlines
323+ getContentsFor "langid" = getVariable "language" >>= toLaTeX . valToInlines
324+ getContentsFor "number" = (getVariable "number"
325+ <|> getVariable "collection-number"
326+ <|> getVariable "issue") >>= toLaTeX . valToInlines
327+
328+ getContentsFor x = getVariable x >>=
329+ if isURL x
330+ then Just . literal . stringify . valToInlines
331+ else toLaTeX .
332+ (if x == "title"
333+ then titlecase
334+ else id) .
335+ valToInlines
336+
337+ isURL x = x `elem` ["url","doi","issn","isbn"]
338+
339+ renderFields = mconcat . intersperse ("," <> cr) . mapMaybe renderField
70340
71341defaultLang :: Lang
72−defaultLang = Lang "en" (Just "US")
342+defaultLang = Lang "en" Nothing (Just "US") [] [] []
73343
74344-- a map of bibtex "string" macros
75345type StringMap = Map.Map Text Text
76346
77−type BibParser = Parser Text (Lang, StringMap)
347+type BibParser = Parser Sources (Lang, StringMap)
78348
79349data Item = Item{ identifier :: Text
80350 , sourcePos :: SourcePos
@@ -89,9 +359,7 @@ itemToReference locale variant item = do
89359 bib item $ do
90360 let lang = fromMaybe defaultLang $ localeLanguage locale
91361 modify $ \st -> st{ localeLang = lang,
92− untitlecase = case lang of
93− (Lang "en" _) -> True
94− _ -> False }
362+ untitlecase = langLanguage lang == "en" }
95363
96364 id' <- asks identifier
97365 otherIds <- (Just <$> getRawField "ids")
@@ -315,10 +583,10 @@ itemToReference locale variant item = do
315583 eprint <- getRawField "eprint"
316584 let baseUrl =
317585 case T.toLower etype of
318− "arxiv" -> "http://arxiv.org/abs/"
319− "jstor" -> "http://www.jstor.org/stable/"
320− "pubmed" -> "http://www.ncbi.nlm.nih.gov/pubmed/"
321− "googlebooks" -> "http://books.google.com?id="
586+ "arxiv" -> "https://arxiv.org/abs/"
587+ "jstor" -> "https://www.jstor.org/stable/"
588+ "pubmed" -> "https://www.ncbi.nlm.nih.gov/pubmed/"
589+ "googlebooks" -> "https://books.google.com?id="
322590 _ -> ""
323591 if T.null baseUrl
324592 then mzero
@@ -449,7 +717,7 @@ itemToReference locale variant item = do
449717
450718
451719bib :: Item -> Bib a -> BibParser a
452−bib entry m = fst <$> evalRWST m entry (BibState True (Lang "en" (Just "US")))
720+bib entry m = fst <$> evalRWST m entry (BibState True defaultLang)
453721
454722resolveCrossRefs :: Variant -> [Item] -> [Item]
455723resolveCrossRefs variant entries =
@@ -502,41 +770,10 @@ blocksToInlines bs =
502770 _ -> B.fromList $ Walk.query (:[]) bs
503771
504772adjustSpans :: Lang -> Inline -> Inline
505−adjustSpans lang (RawInline (Format "latex") s)
506− | s == "\\hyphen" || s == "\\hyphen " = Str "-"
507− | otherwise = parseRawLaTeX lang s
773+adjustSpans lang (Span ("",[],[("bibstring",s)]) _) = Str $ resolveKey' lang s
508774adjustSpans _ SoftBreak = Space
509775adjustSpans _ x = x
510776
511−parseRawLaTeX :: Lang -> Text -> Inline
512−parseRawLaTeX lang t@(T.stripPrefix "\\" -> Just xs) =
513− case parseLaTeX lang contents of
514− Right [Para ys] -> f command ys
515− Right [Plain ys] -> f command ys
516− Right [] -> f command []
517− _ -> RawInline (Format "latex") t
518− where (command', contents') = T.break (\c -> c =='{' || c =='\\') xs
519− command = T.strip command'
520− contents = T.drop 1 $ T.dropEnd 1 contents'
521− f "mkbibquote" ils = Span nullAttr [Quoted DoubleQuote ils]
522− f "mkbibemph" ils = Span nullAttr [Emph ils]
523− f "mkbibitalic" ils = Span nullAttr [Emph ils]
524− f "mkbibbold" ils = Span nullAttr [Strong ils]
525− f "mkbibparens" ils = Span nullAttr $
526− [Str "("] ++ ils ++ [Str ")"]
527− f "mkbibbrackets" ils = Span nullAttr $
528− [Str "["] ++ ils ++ [Str "]"]
529− -- ... both should be nestable & should work in year fields
530− f "autocap" ils = Span nullAttr ils
531− -- TODO: should work in year fields
532− f "textnormal" ils = Span ("",["nodecor"],[]) ils
533− f "bibstring" [Str s] = Str $ resolveKey' lang s
534− f "adddot" [] = Str "."
535− f "adddotspace" [] = Span nullAttr [Str ".", Space]
536− f "addabbrvspace" [] = Space
537− f _ ils = Span nullAttr ils
538−parseRawLaTeX _ t = RawInline (Format "latex") t
539−
540777latex' :: Text -> Bib [Block]
541778latex' t = do
542779 lang <- gets localeLang
@@ -572,7 +809,7 @@ bibEntries = do
572809 (bibComment <|> bibPreamble <|> bibString))
573810
574811bibSkip :: BibParser ()
575−bibSkip = () <$ take1WhileP (/='@')
812+bibSkip = skipMany1 (satisfy (/='@'))
576813
577814bibComment :: BibParser ()
578815bibComment = do
@@ -597,6 +834,9 @@ bibString = do
597834 updateState (\(l,m) -> (l, Map.insert k v m))
598835 return ()
599836
837+take1WhileP :: Monad m => (Char -> Bool) -> ParserT Sources u m Text
838+take1WhileP f = T.pack <$> many1 (satisfy f)
839+
600840inBraces :: BibParser Text
601841inBraces = do
602842 char '{'
@@ -628,7 +868,7 @@ fieldName = resolveAlias . T.toLower
628868
629869isBibtexKeyChar :: Char -> Bool
630870isBibtexKeyChar c =
631− isAlphaNum c || c `elem` (".:;?!`'()/*@_+=-[]*&" :: [Char])
871+ isAlphaNum c || c `elem` (".:;?!`'()$/*@_+=-[]*&" :: [Char])
632872
633873bibItem :: BibParser Item
634874bibItem = do
@@ -812,14 +1052,14 @@ getOldDate prefix = do
8121052 let dateparts = filter (\x -> x /= DateParts [])
8131053 $ map toDateParts [(year',month',day'),
8141054 (endyear',endmonth',endday')]
815− literal <- if null dateparts
816− then Just <$> getRawField (prefix <> "year")
817− else return Nothing
1055+ literal' <- if null dateparts
1056+ then Just <$> getRawField (prefix <> "year")
1057+ else return Nothing
8181058 return $
8191059 Date { dateParts = dateparts
8201060 , dateCirca = False
8211061 , dateSeason = Nothing
822− , dateLiteral = literal }
1062+ , dateLiteral = literal' }
8231063
8241064getRawField :: Text -> Bib Text
8251065getRawField f = do
@@ -1225,8 +1465,9 @@ resolveKey lang ils = Walk.walk go ils
12251465 go x = x
12261466
12271467resolveKey' :: Lang -> Text -> Text
1228−resolveKey' lang@(Lang l _) k =
1229− case Map.lookup l biblatexStringMap >>= Map.lookup (T.toLower k) of
1468+resolveKey' lang k =
1469+ case Map.lookup (langLanguage lang) biblatexStringMap >>=
1470+ Map.lookup (T.toLower k) of
12301471 Nothing -> k
12311472 Just (x, _) -> either (const k) stringify $ parseLaTeX lang x
12321473
src/Text/Pandoc/Citeproc/Data.hs+7−6
@@ -10,7 +10,7 @@ import qualified Data.Text.Encoding as TE
1010import qualified Data.Text as T
1111import Data.Text (Text)
1212import Text.Pandoc.Citeproc.Util (toIETF)
13−import Citeproc (Lang(..), parseLang)
13+import Text.Collate.Lang (Lang(..), parseLang)
1414
1515biblatexLocalizations :: [(FilePath, ByteString)]
1616biblatexLocalizations = $(embedDir "citeproc/biblatex-localization")
@@ -21,11 +21,12 @@ biblatexStringMap :: M.Map Text (M.Map Text (Text, Text))
2121biblatexStringMap = foldr go mempty biblatexLocalizations
2222 where
2323 go (fp, bs) =
24− let Lang lang _ = parseLang (toIETF $ T.takeWhile (/= '.') $ T.pack fp)
25− ls = T.lines $ TE.decodeUtf8 bs
26− in if length ls > 4
27− then M.insert lang (toStringMap $ map (T.splitOn "|") ls)
28− else id
24+ let ls = T.lines $ TE.decodeUtf8 bs
25+ in case parseLang (toIETF $ T.takeWhile (/= '.') $ T.pack fp) of
26+ Right lang | length ls > 4
27+ -> M.insert (langLanguage lang)
28+ (toStringMap $ map (T.splitOn "|") ls)
29+ _ -> id
2930 toStringMap = foldr go' mempty
3031 go' [term, x, y] = M.insert term (x, y)
3132 go' _ = id
src/Text/Pandoc/Citeproc/Locator.hs+20−4
@@ -7,6 +7,7 @@ where
77import Citeproc.Types
88import Data.Text (Text)
99import qualified Data.Text as T
10+import Data.List (foldl')
1011import Text.Parsec
1112import Text.Pandoc.Definition
1213import Text.Pandoc.Parsing (romanNumeral)
@@ -19,7 +20,7 @@ parseLocator :: Locale -> [Inline] -> (Maybe (Text, Text), [Inline])
1920parseLocator locale inp =
2021 case parse (pLocatorWords (toLocatorMap locale)) "suffix" $ splitInp inp of
2122 Right r -> r
22− Left _ -> (Nothing, inp)
23+ Left _ -> (Nothing, maybeAddComma inp)
2324
2425splitInp :: [Inline] -> [Inline]
2526splitInp = splitStrWhen (\c -> isSpace c || (isPunctuation c && c /= ':'))
@@ -41,9 +42,17 @@ pLocatorWords locMap = do
4142 -- i.e. the first one will be " 9"
4243 return $
4344 if T.null la && T.null lo
44− then (Nothing, s)
45+ then (Nothing, maybeAddComma s)
4546 else (Just (la, T.strip lo), s)
4647
48+maybeAddComma :: [Inline] -> [Inline]
49+maybeAddComma [] = []
50+maybeAddComma ils@(Space : _) = ils
51+maybeAddComma ils@(Str t : _)
52+ | Just (c, _) <- T.uncons t
53+ , isPunctuation c = ils
54+maybeAddComma ils = Str "," : Space : ils
55+
4756pLocatorDelimited :: LocatorMap -> LocatorParser (Text, Text)
4857pLocatorDelimited locMap = try $ do
4958 _ <- pMatchChar "{" (== '{')
@@ -96,7 +105,7 @@ pLocatorLabel' locMap lim = go ""
96105 t <- anyToken
97106 ts <- manyTill anyToken (try $ lookAhead lim)
98107 let s = acc <> stringify (t:ts)
99− case M.lookup (T.strip s) locMap of
108+ case M.lookup (T.toCaseFold $ T.strip s) locMap of
100109 -- try to find a longer one, or return this one
101110 Just l -> go s <|> return (l, False)
102111 Nothing -> go s
@@ -139,7 +148,7 @@ pBalancedBraces braces p = try $ do
139148 where
140149 except = notFollowedBy pBraces >> p
141150 -- outer and inner
142− surround = foldl (\a (open, close) -> sur open close except <|> a)
151+ surround = foldl' (\a (open, close) -> sur open close except <|> a)
143152 except
144153 braces
145154
@@ -180,6 +189,7 @@ pPageUnit = roman <|> plainUnit
180189 plainUnit = do
181190 ts <- many1 (notFollowedBy pSpace >>
182191 notFollowedBy pLocatorPunct >>
192+ notFollowedBy pMath >>
183193 anyToken)
184194 let s = stringify ts
185195 -- otherwise look for actual digits or -s
@@ -210,6 +220,12 @@ pMatchChar msg f = satisfyTok f' <?> msg
210220pSpace :: LocatorParser Inline
211221pSpace = satisfyTok (\t -> isSpacey t || t == Str "\160") <?> "space"
212222
223+pMath :: LocatorParser Inline
224+pMath = satisfyTok isMath
225+ where
226+ isMath (Math{}) = True
227+ isMath _ = False
228+
213229satisfyTok :: (Inline -> Bool) -> LocatorParser Inline
214230satisfyTok f = tokenPrim show (\sp _ _ -> sp) (\tok -> if f tok
215231 then Just tok
src/Text/Pandoc/Citeproc/MetaValue.hs+5−3
@@ -135,12 +135,13 @@ metaValueToVal k v
135135 MetaMap _ -> TextVal mempty
136136
137137metaValueToDate :: MetaValue -> Date
138−metaValueToDate (MetaMap m) =
139− Date
138+metaValueToDate (MetaMap m) = fromMaybe
139+ (Date
140140 { dateParts = dateparts
141141 , dateCirca = circa
142142 , dateSeason = season
143− , dateLiteral = literal }
143+ , dateLiteral = literal })
144+ rawdate
144145 where
145146 dateparts = case M.lookup "date-parts" m of
146147 Just (MetaList xs) ->
@@ -152,6 +153,7 @@ metaValueToDate (MetaMap m) =
152153 M.lookup "circa" m >>= metaValueToBool
153154 season = M.lookup "season" m >>= metaValueToInt
154155 literal = M.lookup "literal" m >>= metaValueToText
156+ rawdate = M.lookup "raw" m >>= metaValueToText >>= rawDateEDTF
155157metaValueToDate (MetaList xs) =
156158 Date{ dateParts = mapMaybe metaValueToDateParts xs
157159 , dateCirca = False
src/Text/Pandoc/Class/CommonState.hs+1−1
@@ -19,7 +19,7 @@ where
1919
2020import Data.Default (Default (def))
2121import Data.Text (Text)
22−import Text.Pandoc.BCP47 (Lang)
22+import Text.Collate.Lang (Lang)
2323import Text.Pandoc.MediaBag (MediaBag)
2424import Text.Pandoc.Logging (LogMessage, Verbosity (WARNING))
2525import Text.Pandoc.Translations (Translations)
src/Text/Pandoc/Class/IO.hs+24−25
@@ -62,7 +62,7 @@ import Text.Pandoc.Definition (Pandoc, Inline (Image))
6262import Text.Pandoc.Error (PandocError (..))
6363import Text.Pandoc.Logging (LogMessage (..), messageVerbosity, showLogMessage)
6464import Text.Pandoc.MIME (MimeType)
65−import Text.Pandoc.MediaBag (MediaBag, lookupMedia, mediaDirectory)
65+import Text.Pandoc.MediaBag (MediaBag, MediaItem(..), lookupMedia, mediaItems)
6666import Text.Pandoc.Walk (walk)
6767import qualified Control.Exception as E
6868import qualified Data.ByteString as B
@@ -183,7 +183,7 @@ getModificationTime = liftIOError System.Directory.getModificationTime
183183logOutput :: (PandocMonad m, MonadIO m) => LogMessage -> m ()
184184logOutput msg = liftIO $ do
185185 UTF8.hPutStr stderr $
186− "[" ++ show (messageVerbosity msg) ++ "] "
186+ "[" <> T.pack (show (messageVerbosity msg)) <> "] "
187187 alertIndent $ T.lines $ showLogMessage msg
188188
189189-- | Prints the list of lines to @stderr@, indenting every but the first
@@ -191,42 +191,41 @@ logOutput msg = liftIO $ do
191191alertIndent :: [Text] -> IO ()
192192alertIndent [] = return ()
193193alertIndent (l:ls) = do
194− UTF8.hPutStrLn stderr $ unpack l
194+ UTF8.hPutStrLn stderr l
195195 mapM_ go ls
196196 where go l' = do UTF8.hPutStr stderr " "
197− UTF8.hPutStrLn stderr $ unpack l'
197+ UTF8.hPutStrLn stderr l'
198198
199199-- | Extract media from the mediabag into a directory.
200200extractMedia :: (PandocMonad m, MonadIO m) => FilePath -> Pandoc -> m Pandoc
201201extractMedia dir d = do
202202 media <- getMediaBag
203− case [fp | (fp, _, _) <- mediaDirectory media] of
204− [] -> return d
205− fps -> do
206− mapM_ (writeMedia dir media) fps
207− return $ walk (adjustImagePath dir fps) d
203+ let items = mediaItems media
204+ if null items
205+ then return d
206+ else do
207+ mapM_ (writeMedia dir) items
208+ return $ walk (adjustImagePath dir media) d
208209
209210-- | Write the contents of a media bag to a path.
210211writeMedia :: (PandocMonad m, MonadIO m)
211− => FilePath -> MediaBag -> FilePath
212+ => FilePath
213+ -> (FilePath, MimeType, BL.ByteString)
212214 -> m ()
213−writeMedia dir mediabag subpath = do
214− -- we join and split to convert a/b/c to a\b\c on Windows;
215− -- in zip containers all paths use /
216− let fullpath = dir </> unEscapeString (normalise subpath)
217− let mbcontents = lookupMedia subpath mediabag
218− case mbcontents of
219− Nothing -> throwError $ PandocResourceNotFound $ pack subpath
220− Just (_, bs) -> do
221− report $ Extracting $ pack fullpath
222− liftIOError (createDirectoryIfMissing True) (takeDirectory fullpath)
223− logIOError $ BL.writeFile fullpath bs
215+writeMedia dir (fp, _mt, bs) = do
216+ -- we normalize to get proper path separators for the platform
217+ let fullpath = normalise $ dir </> fp
218+ liftIOError (createDirectoryIfMissing True) (takeDirectory fullpath)
219+ logIOError $ BL.writeFile fullpath bs
224220
225221-- | If the given Inline element is an image with a @src@ path equal to
226222-- one in the list of @paths@, then prepends @dir@ to the image source;
227223-- returns the element unchanged otherwise.
228−adjustImagePath :: FilePath -> [FilePath] -> Inline -> Inline
229−adjustImagePath dir paths (Image attr lab (src, tit))
230− | unpack src `elem` paths
231− = Image attr lab (pack (normalise $ dir </> unpack src), tit)
224+adjustImagePath :: FilePath -> MediaBag -> Inline -> Inline
225+adjustImagePath dir mediabag (Image attr lab (src, tit)) =
226+ case lookupMedia (T.unpack src) mediabag of
227+ Nothing -> Image attr lab (src, tit)
228+ Just item ->
229+ let fullpath = dir </> mediaPath item
230+ in Image attr lab (T.pack fullpath, tit)
232231adjustImagePath _ _ x = x
src/Text/Pandoc/Class/PandocMonad.hs+43−36
@@ -1,4 +1,5 @@
11{-# LANGUAGE CPP #-}
2+{-# LANGUAGE TupleSections #-}
23{-# LANGUAGE FlexibleContexts #-}
34{-# LANGUAGE FlexibleInstances #-}
45{-# LANGUAGE MultiParamTypeClasses #-}
@@ -37,7 +38,6 @@ module Text.Pandoc.Class.PandocMonad
3738 , setUserDataDir
3839 , getUserDataDir
3940 , fetchItem
40− , fetchMediaResource
4141 , getInputFiles
4242 , setInputFiles
4343 , getOutputFile
@@ -51,30 +51,31 @@ module Text.Pandoc.Class.PandocMonad
5151 , setTranslations
5252 , translateTerm
5353 , makeCanonical
54+ , getTimestamp
5455 ) where
5556
5657import Codec.Archive.Zip
5758import Control.Monad.Except (MonadError (catchError, throwError),
5859 MonadTrans, lift, when)
59−import Data.Digest.Pure.SHA (sha1, showDigest)
60−import Data.Maybe (fromMaybe)
60+import Data.List (foldl')
6161import Data.Time (UTCTime)
62−import Data.Time.Clock.POSIX (POSIXTime, utcTimeToPOSIXSeconds)
62+import Data.Time.Clock.POSIX (POSIXTime, utcTimeToPOSIXSeconds,
63+ posixSecondsToUTCTime)
6364import Data.Time.LocalTime (TimeZone, ZonedTime, utcToZonedTime)
6465import Network.URI ( escapeURIString, nonStrictRelativeTo,
6566 unEscapeString, parseURIReference, isAllowedInURI,
6667 parseURI, URI(..) )
67−import System.FilePath ((</>), (<.>), takeExtension, dropExtension,
68− isRelative, splitDirectories)
68+import System.FilePath ((</>), takeExtension, dropExtension,
69+ isRelative, splitDirectories, makeRelative)
6970import System.Random (StdGen)
70−import Text.Pandoc.BCP47 (Lang(..), parseBCP47, renderLang)
71+import Text.Collate.Lang (Lang(..), parseLang, renderLang)
7172import Text.Pandoc.Class.CommonState (CommonState (..))
7273import Text.Pandoc.Definition
7374import Text.Pandoc.Error
7475import Text.Pandoc.Logging
75−import Text.Pandoc.MIME (MimeType, getMimeType, extensionFromMimeType)
76−import Text.Pandoc.MediaBag (MediaBag, lookupMedia)
77−import Text.Pandoc.Shared (uriPathToPath)
76+import Text.Pandoc.MIME (MimeType, getMimeType)
77+import Text.Pandoc.MediaBag (MediaBag, lookupMedia, MediaItem(..))
78+import Text.Pandoc.Shared (uriPathToPath, safeRead)
7879import Text.Pandoc.Translations (Term(..), Translations, lookupTerm,
7980 readTranslations)
8081import Text.Pandoc.Walk (walkM)
@@ -175,6 +176,21 @@ report msg = do
175176 when (level <= verbosity) $ logOutput msg
176177 modifyCommonState $ \st -> st{ stLog = msg : stLog st }
177178
179+-- | Get the time from the @SOURCE_DATE_EPOCH@
180+-- environment variable. The variable should contain a
181+-- unix time stamp, the number of seconds since midnight Jan 01
182+-- 1970 UTC. If the variable is not set or cannot be
183+-- parsed as a unix time stamp, the current time is returned.
184+-- This function is designed to make possible reproducible
185+-- builds in formats that include a creation timestamp.
186+getTimestamp :: PandocMonad m => m UTCTime
187+getTimestamp = do
188+ mbSourceDateEpoch <- lookupEnv "SOURCE_DATE_EPOCH"
189+ case mbSourceDateEpoch >>= safeRead of
190+ Just (epoch :: Integer) ->
191+ return $ posixSecondsToUTCTime $ fromIntegral epoch
192+ Nothing -> getCurrentTime
193+
178194-- | Determine whether tracing is enabled. This affects
179195-- the behavior of 'trace'. If tracing is not enabled,
180196-- 'trace' does nothing.
@@ -267,7 +283,7 @@ readFileFromDirs (d:ds) f = catchError
267283toLang :: PandocMonad m => Maybe T.Text -> m (Maybe Lang)
268284toLang Nothing = return Nothing
269285toLang (Just s) =
270− case parseBCP47 s of
286+ case parseLang s of
271287 Left _ -> do
272288 report $ InvalidLang s
273289 return Nothing
@@ -358,7 +374,8 @@ fetchItem :: PandocMonad m
358374fetchItem s = do
359375 mediabag <- getMediaBag
360376 case lookupMedia (T.unpack s) mediabag of
361− Just (mime, bs) -> return (BL.toStrict bs, Just mime)
377+ Just item -> return (BL.toStrict (mediaContents item),
378+ Just (mediaMimeType item))
362379 Nothing -> downloadOrRead s
363380
364381-- | Returns the content and, if available, the MIME type of a resource.
@@ -393,9 +410,10 @@ downloadOrRead s = do
393410 _ -> readLocalFile fp -- get from local file system
394411 where readLocalFile f = do
395412 resourcePath <- getResourcePath
396− cont <- if isRelative f
397− then withPaths resourcePath readFileStrict f
398− else readFileStrict f
413+ (fp', cont) <- if isRelative f
414+ then withPaths resourcePath readFileStrict f
415+ else (f,) <$> readFileStrict f
416+ report $ LoadedResource f (makeRelative "." fp')
399417 return (cont, mime)
400418 httpcolon = URI{ uriScheme = "http:",
401419 uriAuthority = Nothing,
@@ -595,7 +613,7 @@ checkExistence fn = do
595613-- | Canonicalizes a file path by removing redundant @.@ and @..@.
596614makeCanonical :: FilePath -> FilePath
597615makeCanonical = Posix.joinPath . transformPathParts . splitDirectories
598− where transformPathParts = reverse . foldl go []
616+ where transformPathParts = reverse . foldl' go []
599617 go as "." = as
600618 go (_:as) ".." = as
601619 go as x = x : as
@@ -605,25 +623,13 @@ makeCanonical = Posix.joinPath . transformPathParts . splitDirectories
605623-- that filepath. Returns the result of the first successful execution
606624-- of the action, or throws a @PandocResourceNotFound@ exception if the
607625-- action errors for all filepaths.
608−withPaths :: PandocMonad m => [FilePath] -> (FilePath -> m a) -> FilePath -> m a
626+withPaths :: PandocMonad m
627+ => [FilePath] -> (FilePath -> m a) -> FilePath -> m (FilePath, a)
609628withPaths [] _ fp = throwError $ PandocResourceNotFound $ T.pack fp
610629withPaths (p:ps) action fp =
611− catchError (action (p </> fp))
630+ catchError ((p </> fp,) <$> action (p </> fp))
612631 (\_ -> withPaths ps action fp)
613632
614−-- | Fetch local or remote resource (like an image) and provide data suitable
615−-- for adding it to the MediaBag.
616−fetchMediaResource :: PandocMonad m
617− => T.Text -> m (FilePath, Maybe MimeType, BL.ByteString)
618−fetchMediaResource src = do
619− (bs, mt) <- downloadOrRead src
620− let ext = fromMaybe (T.pack $ takeExtension $ T.unpack src)
621− (mt >>= extensionFromMimeType)
622− let bs' = BL.fromChunks [bs]
623− let basename = showDigest $ sha1 bs'
624− let fname = basename <.> T.unpack ext
625− return (fname, mt, bs')
626−
627633-- | Traverse tree, filling media bag for any images that
628634-- aren't already in the media bag.
629635fillMediaBag :: PandocMonad m => Pandoc -> m Pandoc
@@ -631,12 +637,13 @@ fillMediaBag d = walkM handleImage d
631637 where handleImage :: PandocMonad m => Inline -> m Inline
632638 handleImage (Image attr lab (src, tit)) = catchError
633639 (do mediabag <- getMediaBag
634− case lookupMedia (T.unpack src) mediabag of
635− Just (_, _) -> return $ Image attr lab (src, tit)
640+ let fp = T.unpack src
641+ case lookupMedia fp mediabag of
642+ Just _ -> return ()
636643 Nothing -> do
637− (fname, mt, bs) <- fetchMediaResource src
638− insertMedia fname mt bs
639− return $ Image attr lab (T.pack fname, tit))
644+ (bs, mt) <- fetchItem src
645+ insertMedia fp mt (BL.fromStrict bs)
646+ return $ Image attr lab (src, tit))
640647 (\e ->
641648 case e of
642649 PandocResourceNotFound _ -> do
src/Text/Pandoc/Data.hs+1−1
@@ -1,7 +1,7 @@
11{-# LANGUAGE TemplateHaskell #-}
22{- |
33Module : Text.Pandoc.Data
4−Copyright : Copyright (C) 2013-2020 John MacFarlane
4+Copyright : Copyright (C) 2013-2021 John MacFarlane
55License : GNU GPL, version 2 or above
66
77Maintainer : John MacFarlane <jgm@berkeley@edu>
src/Text/Pandoc/Error.hs+102−49
@@ -3,7 +3,7 @@
33{-# LANGUAGE OverloadedStrings #-}
44{- |
55 Module : Text.Pandoc.Error
6− Copyright : Copyright (C) 2006-2020 John MacFarlane
6+ Copyright : Copyright (C) 2006-2021 John MacFarlane
77 License : GNU GPL, version 2 or above
88
99 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -16,38 +16,41 @@ encountered during parsing.
1616-}
1717module Text.Pandoc.Error (
1818 PandocError(..),
19+ renderError,
1920 handleError) where
2021
21−import Control.Exception (Exception)
22+import Control.Exception (Exception, displayException)
2223import Data.Typeable (Typeable)
2324import Data.Word (Word8)
2425import Data.Text (Text)
26+import Data.List (sortOn)
2527import qualified Data.Text as T
28+import Data.Ord (Down(..))
2629import GHC.Generics (Generic)
2730import Network.HTTP.Client (HttpException)
2831import System.Exit (ExitCode (..), exitWith)
2932import System.IO (stderr)
3033import qualified Text.Pandoc.UTF8 as UTF8
34+import Text.Pandoc.Sources (Sources(..))
3135import Text.Printf (printf)
3236import Text.Parsec.Error
3337import Text.Parsec.Pos hiding (Line)
3438import Text.Pandoc.Shared (tshow)
3539import Citeproc (CiteprocError, prettyCiteprocError)
3640
37−type Input = Text
38−
3941data PandocError = PandocIOError Text IOError
4042 | PandocHttpError Text HttpException
4143 | PandocShouldNeverHappenError Text
4244 | PandocSomeError Text
4345 | PandocParseError Text
44− | PandocParsecError Input ParseError
46+ | PandocParsecError Sources ParseError
4547 | PandocMakePDFError Text
4648 | PandocOptionError Text
4749 | PandocSyntaxMapError Text
4850 | PandocFailOnWarningError
4951 | PandocPDFProgramNotFoundError Text
5052 | PandocPDFError Text
53+ | PandocXMLError Text Text
5154 | PandocFilterError Text Text
5255 | PandocLuaError Text
5356 | PandocCouldNotFindDataFileError Text
@@ -58,6 +61,7 @@ data PandocError = PandocIOError Text IOError
5861 | PandocMacroLoop Text
5962 | PandocUTF8DecodingError Text Int Word8
6063 | PandocIpynbDecodingError Text
64+ | PandocUnsupportedCharsetError Text
6165 | PandocUnknownReaderError Text
6266 | PandocUnknownWriterError Text
6367 | PandocUnsupportedExtensionError Text Text
@@ -67,62 +71,70 @@ data PandocError = PandocIOError Text IOError
6771
6872instance Exception PandocError
6973
70−-- | Handle PandocError by exiting with an error message.
71−handleError :: Either PandocError a -> IO a
72−handleError (Right r) = return r
73−handleError (Left e) =
74+renderError :: PandocError -> Text
75+renderError e =
7476 case e of
75− PandocIOError _ err' -> ioError err'
76− PandocHttpError u err' -> err 61 $
77+ PandocIOError _ err' -> T.pack $ displayException err'
78+ PandocHttpError u err' ->
7779 "Could not fetch " <> u <> "\n" <> tshow err'
78− PandocShouldNeverHappenError s -> err 62 $
80+ PandocShouldNeverHappenError s ->
7981 "Something we thought was impossible happened!\n" <>
8082 "Please report this to pandoc's developers: " <> s
81− PandocSomeError s -> err 63 s
82− PandocParseError s -> err 64 s
83− PandocParsecError input err' ->
83+ PandocSomeError s -> s
84+ PandocParseError s -> s
85+ PandocParsecError (Sources inputs) err' ->
8486 let errPos = errorPos err'
8587 errLine = sourceLine errPos
8688 errColumn = sourceColumn errPos
87− ls = T.lines input <> [""]
88− errorInFile = if length ls > errLine - 1
89− then T.concat ["\n", ls !! (errLine - 1)
90− ,"\n", T.replicate (errColumn - 1) " "
91− ,"^"]
92− else ""
93− in err 65 $ "\nError at " <> tshow err' <>
94− -- if error comes from a chunk or included file,
95− -- then we won't get the right text this way:
96− if sourceName errPos == "source"
97− then errorInFile
98− else ""
99− PandocMakePDFError s -> err 66 s
100− PandocOptionError s -> err 6 s
101− PandocSyntaxMapError s -> err 67 s
102− PandocFailOnWarningError -> err 3 "Failing because there were warnings."
103− PandocPDFProgramNotFoundError pdfprog -> err 47 $
89+ errFile = sourceName errPos
90+ errorInFile =
91+ case sortOn (Down . sourceLine . fst)
92+ [ (pos,t)
93+ | (pos,t) <- inputs
94+ , sourceName pos == errFile
95+ , sourceLine pos <= errLine
96+ ] of
97+ [] -> ""
98+ ((pos,txt):_) ->
99+ let ls = T.lines txt <> [""]
100+ ln = (errLine - sourceLine pos) + 1
101+ in if length ls > ln && ln >= 1
102+ then T.concat ["\n", ls !! (ln - 1)
103+ ,"\n", T.replicate (errColumn - 1) " "
104+ ,"^"]
105+ else ""
106+ in "Error at " <> tshow err' <> errorInFile
107+ PandocMakePDFError s -> s
108+ PandocOptionError s -> s
109+ PandocSyntaxMapError s -> s
110+ PandocFailOnWarningError -> "Failing because there were warnings."
111+ PandocPDFProgramNotFoundError pdfprog ->
104112 pdfprog <> " not found. Please select a different --pdf-engine or install " <> pdfprog
105− PandocPDFError logmsg -> err 43 $ "Error producing PDF.\n" <> logmsg
106− PandocFilterError filtername msg -> err 83 $ "Error running filter " <>
113+ PandocPDFError logmsg -> "Error producing PDF.\n" <> logmsg
114+ PandocXMLError fp logmsg -> "Invalid XML" <>
115+ (if T.null fp then "" else " in " <> fp) <> ":\n" <> logmsg
116+ PandocFilterError filtername msg -> "Error running filter " <>
107117 filtername <> ":\n" <> msg
108− PandocLuaError msg -> err 84 $ "Error running Lua:\n" <> msg
109− PandocCouldNotFindDataFileError fn -> err 97 $
118+ PandocLuaError msg -> "Error running Lua:\n" <> msg
119+ PandocCouldNotFindDataFileError fn ->
110120 "Could not find data file " <> fn
111− PandocResourceNotFound fn -> err 99 $
121+ PandocResourceNotFound fn ->
112122 "File " <> fn <> " not found in resource path"
113− PandocTemplateError s -> err 5 $ "Error compiling template " <> s
114− PandocAppError s -> err 4 s
115− PandocEpubSubdirectoryError s -> err 31 $
123+ PandocTemplateError s -> "Error compiling template " <> s
124+ PandocAppError s -> s
125+ PandocEpubSubdirectoryError s ->
116126 "EPUB subdirectory name '" <> s <> "' contains illegal characters"
117− PandocMacroLoop s -> err 91 $
127+ PandocMacroLoop s ->
118128 "Loop encountered in expanding macro " <> s
119− PandocUTF8DecodingError f offset w -> err 92 $
129+ PandocUTF8DecodingError f offset w ->
120130 "UTF-8 decoding error in " <> f <> " at byte offset " <> tshow offset <>
121131 " (" <> T.pack (printf "%2x" w) <> ").\n" <>
122132 "The input must be a UTF-8 encoded text."
123− PandocIpynbDecodingError w -> err 93 $
133+ PandocIpynbDecodingError w ->
124134 "ipynb decoding error: " <> w
125− PandocUnknownReaderError r -> err 21 $
135+ PandocUnsupportedCharsetError charset ->
136+ "Unsupported charset " <> charset
137+ PandocUnknownReaderError r ->
126138 "Unknown input format " <> r <>
127139 case r of
128140 "doc" -> "\nPandoc can convert from DOCX, but not from DOC." <>
@@ -130,7 +142,7 @@ handleError (Left e) =
130142 " and convert that with pandoc."
131143 "pdf" -> "\nPandoc can convert to PDF, but not from PDF."
132144 _ -> ""
133− PandocUnknownWriterError w -> err 22 $
145+ PandocUnknownWriterError w ->
134146 "Unknown output format " <> w <>
135147 case w of
136148 "pdf" -> "To create a pdf using pandoc, use" <>
@@ -139,16 +151,57 @@ handleError (Left e) =
139151 ".pdf extension (-o filename.pdf)."
140152 "doc" -> "\nPandoc can convert to DOCX, but not to DOC."
141153 _ -> ""
142− PandocUnsupportedExtensionError ext f -> err 23 $
154+ PandocUnsupportedExtensionError ext f ->
143155 "The extension " <> ext <> " is not supported " <>
144156 "for " <> f
145− PandocCiteprocError e' -> err 24 $
157+ PandocCiteprocError e' ->
146158 prettyCiteprocError e'
147− PandocBibliographyError fp msg -> err 25 $
159+ PandocBibliographyError fp msg ->
148160 "Error reading bibliography file " <> fp <> ":\n" <> msg
149161
162+
163+-- | Handle PandocError by exiting with an error message.
164+handleError :: Either PandocError a -> IO a
165+handleError (Right r) = return r
166+handleError (Left e) =
167+ case e of
168+ PandocIOError _ err' -> ioError err'
169+ _ -> err exitCode (renderError e)
170+ where
171+ exitCode =
172+ case e of
173+ PandocIOError{} -> 1
174+ PandocHttpError{} -> 61
175+ PandocShouldNeverHappenError{} -> 62
176+ PandocSomeError{} -> 63
177+ PandocParseError{} -> 64
178+ PandocParsecError{} -> 65
179+ PandocMakePDFError{} -> 66
180+ PandocOptionError{} -> 6
181+ PandocSyntaxMapError{} -> 67
182+ PandocFailOnWarningError{} -> 3
183+ PandocPDFProgramNotFoundError{} -> 47
184+ PandocPDFError{} -> 43
185+ PandocXMLError{} -> 44
186+ PandocFilterError{} -> 83
187+ PandocLuaError{} -> 84
188+ PandocCouldNotFindDataFileError{} -> 97
189+ PandocResourceNotFound{} -> 99
190+ PandocTemplateError{} -> 5
191+ PandocAppError{} -> 4
192+ PandocEpubSubdirectoryError{} -> 31
193+ PandocMacroLoop{} -> 91
194+ PandocUTF8DecodingError{} -> 92
195+ PandocIpynbDecodingError{} -> 93
196+ PandocUnsupportedCharsetError{} -> 94
197+ PandocUnknownReaderError{} -> 21
198+ PandocUnknownWriterError{} -> 22
199+ PandocUnsupportedExtensionError{} -> 23
200+ PandocCiteprocError{} -> 24
201+ PandocBibliographyError{} -> 25
202+
150203err :: Int -> Text -> IO a
151204err exitCode msg = do
152− UTF8.hPutStrLn stderr (T.unpack msg)
205+ UTF8.hPutStrLn stderr msg
153206 exitWith $ ExitFailure exitCode
154207 return undefined
src/Text/Pandoc/Extensions.hs+26−4
@@ -6,7 +6,7 @@
66{-# LANGUAGE OverloadedStrings #-}
77{- |
88 Module : Text.Pandoc.Extensions
9− Copyright : Copyright (C) 2012-2020 John MacFarlane
9+ Copyright : Copyright (C) 2012-2021 John MacFarlane
1010 License : GNU GPL, version 2 or above
1111
1212 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -34,6 +34,7 @@ module Text.Pandoc.Extensions ( Extension(..)
3434where
3535import Data.Bits (clearBit, setBit, testBit, (.|.))
3636import Data.Data (Data)
37+import Data.List (foldl')
3738import qualified Data.Text as T
3839import Data.Typeable (Typeable)
3940import GHC.Generics (Generic)
@@ -88,6 +89,7 @@ data Extension =
8889 -- does not affect readers/writers directly; it causes
8990 -- the eastAsianLineBreakFilter to be applied after
9091 -- parsing, in Text.Pandoc.App.convertWithOpts.
92+ | Ext_element_citations -- ^ Use element-citation elements for JATS citations
9193 | Ext_emoji -- ^ Support emoji like :smile:
9294 | Ext_empty_paragraphs -- ^ Allow empty paragraphs
9395 | Ext_epub_html_exts -- ^ Recognise the EPUB extended version of HTML
@@ -134,6 +136,8 @@ data Extension =
134136 | Ext_raw_html -- ^ Allow raw HTML
135137 | Ext_raw_tex -- ^ Allow raw TeX (other than math)
136138 | Ext_raw_markdown -- ^ Parse markdown in ipynb as raw markdown
139+ | Ext_rebase_relative_paths -- ^ Rebase relative image and link paths,
140+ -- relative to directory of containing file
137141 | Ext_shortcut_reference_links -- ^ Shortcut reference links
138142 | Ext_simple_tables -- ^ Pandoc-style simple tables
139143 | Ext_smart -- ^ "Smart" quotes, apostrophes, ellipses, dashes
@@ -352,6 +356,7 @@ getDefaultExtensions "gfm" = extensionsFromList
352356 , Ext_strikeout
353357 , Ext_task_lists
354358 , Ext_emoji
359+ , Ext_yaml_metadata_block
355360 ]
356361getDefaultExtensions "commonmark" = extensionsFromList
357362 [Ext_raw_html]
@@ -377,10 +382,11 @@ getDefaultExtensions "commonmark_x" = extensionsFromList
377382 , Ext_raw_attribute
378383 , Ext_implicit_header_references
379384 , Ext_attributes
380− , Ext_fenced_code_attributes
385+ , Ext_yaml_metadata_block
381386 ]
382387getDefaultExtensions "org" = extensionsFromList
383388 [Ext_citations,
389+ Ext_task_lists,
384390 Ext_auto_identifiers]
385391getDefaultExtensions "html" = extensionsFromList
386392 [Ext_auto_identifiers,
@@ -412,6 +418,11 @@ getDefaultExtensions "textile" = extensionsFromList
412418 Ext_smart,
413419 Ext_raw_html,
414420 Ext_auto_identifiers]
421+getDefaultExtensions "jats" = extensionsFromList
422+ [Ext_auto_identifiers]
423+getDefaultExtensions "jats_archiving" = getDefaultExtensions "jats"
424+getDefaultExtensions "jats_publishing" = getDefaultExtensions "jats"
425+getDefaultExtensions "jats_articleauthoring" = getDefaultExtensions "jats"
415426getDefaultExtensions "opml" = pandocExtensions -- affects notes
416427getDefaultExtensions _ = extensionsFromList
417428 [Ext_auto_identifiers]
@@ -453,6 +464,7 @@ getAllExtensions f = universalExtensions <> getAll f
453464 , Ext_gutenberg
454465 , Ext_smart
455466 , Ext_literate_haskell
467+ , Ext_rebase_relative_paths
456468 ]
457469 getAll "markdown_strict" = allMarkdownExtensions
458470 getAll "markdown_phpextra" = allMarkdownExtensions
@@ -503,14 +515,16 @@ getAllExtensions f = universalExtensions <> getAll f
503515 , Ext_raw_attribute
504516 , Ext_implicit_header_references
505517 , Ext_attributes
506− , Ext_fenced_code_attributes
507518 , Ext_sourcepos
519+ , Ext_yaml_metadata_block
520+ , Ext_rebase_relative_paths
508521 ]
509522 getAll "commonmark_x" = getAll "commonmark"
510523 getAll "org" = autoIdExtensions <>
511524 extensionsFromList
512525 [ Ext_citations
513526 , Ext_smart
527+ , Ext_task_lists
514528 ]
515529 getAll "html" = autoIdExtensions <>
516530 extensionsFromList
@@ -554,6 +568,14 @@ getAllExtensions f = universalExtensions <> getAll f
554568 , Ext_smart
555569 , Ext_raw_tex
556570 ]
571+ getAll "jats" =
572+ extensionsFromList
573+ [ Ext_auto_identifiers
574+ , Ext_element_citations
575+ ]
576+ getAll "jats_archiving" = getAll "jats"
577+ getAll "jats_publishing" = getAll "jats"
578+ getAll "jats_articleauthoring" = getAll "jats"
557579 getAll "opml" = allMarkdownExtensions -- affects notes
558580 getAll "twiki" = autoIdExtensions <>
559581 extensionsFromList
@@ -579,7 +601,7 @@ parseFormatSpec :: T.Text
579601parseFormatSpec = parse formatSpec ""
580602 where formatSpec = do
581603 name <- formatName
582− (extsToEnable, extsToDisable) <- foldl (flip ($)) ([],[]) <$>
604+ (extsToEnable, extsToDisable) <- foldl' (flip ($)) ([],[]) <$>
583605 many extMod
584606 return (T.pack name, reverse extsToEnable, reverse extsToDisable)
585607 formatName = many1 $ noneOf "-+"
src/Text/Pandoc/Filter.hs+1−1
@@ -4,7 +4,7 @@
44{-# LANGUAGE OverloadedStrings #-}
55{- |
66 Module : Text.Pandoc.Filter
7− Copyright : Copyright (C) 2006-2020 John MacFarlane
7+ Copyright : Copyright (C) 2006-2021 John MacFarlane
88 License : GNU GPL, version 2 or above
99
1010 Maintainer : John MacFarlane <jgm@berkeley@edu>
src/Text/Pandoc/Filter/JSON.hs+1−1
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.Filter
4− Copyright : Copyright (C) 2006-2020 John MacFarlane
4+ Copyright : Copyright (C) 2006-2021 John MacFarlane
55 License : GNU GPL, version 2 or above
66
77 Maintainer : John MacFarlane <jgm@berkeley@edu>
src/Text/Pandoc/Filter/Lua.hs+1−1
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Filter.Lua
3− Copyright : Copyright (C) 2006-2020 John MacFarlane
3+ Copyright : Copyright (C) 2006-2021 John MacFarlane
44 License : GNU GPL, version 2 or above
55
66 Maintainer : John MacFarlane <jgm@berkeley@edu>
src/Text/Pandoc/Filter/Path.hs+1−1
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Filter.Path
3− Copyright : Copyright (C) 2006-2020 John MacFarlane
3+ Copyright : Copyright (C) 2006-2021 John MacFarlane
44 License : GNU GPL, version 2 or above
55
66 Maintainer : John MacFarlane <jgm@berkeley@edu>
src/Text/Pandoc/Highlighting.hs+6−6
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.Highlighting
4− Copyright : Copyright (C) 2008-2020 John MacFarlane
4+ Copyright : Copyright (C) 2008-2021 John MacFarlane
55 License : GNU GPL, version 2 or above
66
77 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -52,12 +52,12 @@ highlightingStyles =
5252 ("breezedark", breezeDark),
5353 ("haddock", haddock)]
5454
55−languages :: [T.Text]
56−languages = [T.toLower (sName s) | s <- M.elems defaultSyntaxMap]
55+languages :: SyntaxMap -> [T.Text]
56+languages syntaxmap = [T.toLower (sName s) | s <- M.elems syntaxmap]
5757
58−languagesByExtension :: T.Text -> [T.Text]
59−languagesByExtension ext =
60− [T.toLower (sName s) | s <- syntaxesByExtension defaultSyntaxMap (T.unpack ext)]
58+languagesByExtension :: SyntaxMap -> T.Text -> [T.Text]
59+languagesByExtension syntaxmap ext =
60+ [T.toLower (sName s) | s <- syntaxesByExtension syntaxmap (T.unpack ext)]
6161
6262highlight :: SyntaxMap
6363 -> (FormatOptions -> [SourceLine] -> a) -- ^ Formatter
src/Text/Pandoc/Image.hs+1−1
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings, ScopedTypeVariables, CPP #-}
22{- |
33Module : Text.Pandoc.Image
4−Copyright : Copyright (C) 2020 John MacFarlane
4+Copyright : Copyright (C) 2020-2021 John MacFarlane
55License : GNU GPL, version 2 or above
66
77Maintainer : John MacFarlane <jgm@berkeley.edu>
src/Text/Pandoc/ImageSize.hs+17−7
@@ -3,7 +3,7 @@
33{-# OPTIONS_GHC -fno-warn-type-defaults #-}
44{- |
55Module : Text.Pandoc.ImageSize
6−Copyright : Copyright (C) 2011-2020 John MacFarlane
6+Copyright : Copyright (C) 2011-2021 John MacFarlane
77License : GNU GPL, version 2 or above
88
99Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -44,8 +44,9 @@ import Numeric (showFFloat)
4444import Text.Pandoc.Definition
4545import Text.Pandoc.Options
4646import qualified Text.Pandoc.UTF8 as UTF8
47−import qualified Text.XML.Light as Xml
47+import Text.Pandoc.XML.Light hiding (Attr)
4848import qualified Data.Text as T
49+import qualified Data.Text.Lazy as TL
4950import qualified Data.Text.Encoding as TE
5051import Control.Applicative
5152import qualified Data.Attoparsec.ByteString.Char8 as A
@@ -56,7 +57,8 @@ import Codec.Picture (decodeImageWithMetadata)
5657-- quick and dirty functions to get image sizes
5758-- algorithms borrowed from wwwis.pl
5859
59−data ImageType = Png | Gif | Jpeg | Svg | Pdf | Eps | Emf deriving Show
60+data ImageType = Png | Gif | Jpeg | Svg | Pdf | Eps | Emf | Tiff
61+ deriving Show
6062data Direction = Width | Height
6163instance Show Direction where
6264 show Width = "width"
@@ -99,6 +101,8 @@ imageType :: ByteString -> Maybe ImageType
99101imageType img = case B.take 4 img of
100102 "\x89\x50\x4e\x47" -> return Png
101103 "\x47\x49\x46\x38" -> return Gif
104+ "\x49\x49\x2a\x00" -> return Tiff
105+ "\x4D\x4D\x00\x2a" -> return Tiff
102106 "\xff\xd8\xff\xe0" -> return Jpeg -- JFIF
103107 "\xff\xd8\xff\xe1" -> return Jpeg -- Exif
104108 "%PDF" -> return Pdf
@@ -123,6 +127,7 @@ imageSize opts img = checkDpi <$>
123127 Just Png -> getSize img
124128 Just Gif -> getSize img
125129 Just Jpeg -> getSize img
130+ Just Tiff -> getSize img
126131 Just Svg -> mbToEither "could not determine SVG size" $ svgSize opts img
127132 Just Eps -> mbToEither "could not determine EPS size" $ epsSize img
128133 Just Pdf -> mbToEither "could not determine PDF size" $ pdfSize img
@@ -327,13 +332,18 @@ getSize img =
327332
328333svgSize :: WriterOptions -> ByteString -> Maybe ImageSize
329334svgSize opts img = do
330− doc <- Xml.parseXMLDoc $ UTF8.toString img
335+ doc <- either (const mzero) return $ parseXMLElement
336+ $ TL.fromStrict $ UTF8.toText img
337+ let viewboxSize = do
338+ vb <- findAttrBy (== QName "viewBox" Nothing Nothing) doc
339+ [_,_,w,h] <- mapM safeRead (T.words vb)
340+ return (w,h)
331341 let dpi = fromIntegral $ writerDpi opts
332342 let dirToInt dir = do
333− dim <- Xml.findAttrBy (== Xml.QName dir Nothing Nothing) doc >>= lengthToDim . T.pack
343+ dim <- findAttrBy (== QName dir Nothing Nothing) doc >>= lengthToDim
334344 return $ inPixel opts dim
335− w <- dirToInt "width"
336− h <- dirToInt "height"
345+ w <- dirToInt "width" <|> (fst <$> viewboxSize)
346+ h <- dirToInt "height" <|> (snd <$> viewboxSize)
337347 return ImageSize {
338348 pxX = w
339349 , pxY = h
src/Text/Pandoc/Logging.hs+33−10
@@ -4,7 +4,7 @@
44{-# LANGUAGE OverloadedStrings #-}
55{- |
66 Module : Text.Pandoc.Logging
7− Copyright : Copyright (C) 2006-2020 John MacFarlane
7+ Copyright : Copyright (C) 2006-2021 John MacFarlane
88 License : GNU GPL, version 2 or above
99
1010 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -36,6 +36,7 @@ import Data.Typeable (Typeable)
3636import GHC.Generics (Generic)
3737import Text.Pandoc.Definition
3838import Text.Parsec.Pos
39+import Text.Pandoc.Shared (tshow)
3940
4041-- | Verbosity level.
4142data Verbosity = ERROR | WARNING | INFO
@@ -84,6 +85,7 @@ data LogMessage =
8485 | CouldNotParseCSS Text
8586 | Fetching Text
8687 | Extracting Text
88+ | LoadedResource FilePath FilePath
8789 | NoTitleElement Text
8890 | NoLangSpecified
8991 | InvalidLang Text
@@ -100,6 +102,8 @@ data LogMessage =
100102 | FilterCompleted FilePath Integer
101103 | CiteprocWarning Text
102104 | ATXHeadingInLHS Int Text
105+ | EnvironmentVariableUndefined Text
106+ | DuplicateAttribute Text Text
103107 deriving (Show, Eq, Data, Ord, Typeable, Generic)
104108
105109instance ToJSON LogMessage where
@@ -192,6 +196,9 @@ instance ToJSON LogMessage where
192196 ["path" .= fp]
193197 Extracting fp ->
194198 ["path" .= fp]
199+ LoadedResource orig found ->
200+ ["for" .= orig
201+ ,"from" .= found]
195202 NoTitleElement fallback ->
196203 ["fallback" .= fallback]
197204 NoLangSpecified -> []
@@ -229,13 +236,20 @@ instance ToJSON LogMessage where
229236 ATXHeadingInLHS lvl contents ->
230237 ["level" .= lvl
231238 ,"contents" .= contents]
239+ EnvironmentVariableUndefined var ->
240+ ["variable" .= var ]
241+ DuplicateAttribute attr val ->
242+ ["attribute" .= attr
243+ ,"value" .= val]
232244
233245showPos :: SourcePos -> Text
234246showPos pos = Text.pack $ sn ++ "line " ++
235247 show (sourceLine pos) ++ " column " ++ show (sourceColumn pos)
236− where sn = if sourceName pos == "source" || sourceName pos == ""
237− then ""
238− else sourceName pos ++ " "
248+ where
249+ sn' = sourceName pos
250+ sn = if sn' == "source" || sn' == "" || sn' == "-"
251+ then ""
252+ else sn' ++ " "
239253
240254encodeLogMessages :: [LogMessage] -> BL.ByteString
241255encodeLogMessages ms =
@@ -268,7 +282,7 @@ showLogMessage msg =
268282 ParsingUnescaped s pos ->
269283 "Parsing unescaped '" <> s <> "' at " <> showPos pos
270284 CouldNotLoadIncludeFile fp pos ->
271− "Could not load include file '" <> fp <> "' at " <> showPos pos
285+ "Could not load include file " <> fp <> " at " <> showPos pos
272286 MacroAlreadyDefined name pos ->
273287 "Macro '" <> name <> "' already defined, ignoring at " <> showPos pos
274288 InlineNotRendered il ->
@@ -280,18 +294,18 @@ showLogMessage msg =
280294 IgnoredIOError s ->
281295 "IO Error (ignored): " <> s
282296 CouldNotFetchResource fp s ->
283− "Could not fetch resource '" <> fp <> "'" <>
297+ "Could not fetch resource " <> fp <>
284298 if Text.null s then "" else ": " <> s
285299 CouldNotDetermineImageSize fp s ->
286− "Could not determine image size for '" <> fp <> "'" <>
300+ "Could not determine image size for " <> fp <>
287301 if Text.null s then "" else ": " <> s
288302 CouldNotConvertImage fp s ->
289− "Could not convert image '" <> fp <> "'" <>
303+ "Could not convert image " <> fp <>
290304 if Text.null s then "" else ": " <> s
291305 CouldNotDetermineMimeType fp ->
292− "Could not determine mime type for '" <> fp <> "'"
306+ "Could not determine mime type for " <> fp
293307 CouldNotConvertTeXMath s m ->
294− "Could not convert TeX math '" <> s <> "', rendering as TeX" <>
308+ "Could not convert TeX math " <> s <> ", rendering as TeX" <>
295309 if Text.null m then "" else ":\n" <> m
296310 CouldNotParseCSS m ->
297311 "Could not parse CSS" <> if Text.null m then "" else ":\n" <> m
@@ -299,6 +313,8 @@ showLogMessage msg =
299313 "Fetching " <> fp <> "..."
300314 Extracting fp ->
301315 "Extracting " <> fp <> "..."
316+ LoadedResource orig found ->
317+ "Loaded " <> Text.pack orig <> " from " <> Text.pack found
302318 NoTitleElement fallback ->
303319 "This document format requires a nonempty <title> element.\n" <>
304320 "Defaulting to '" <> fallback <> "' as the title.\n" <>
@@ -345,6 +361,10 @@ showLogMessage msg =
345361 if lvl < 3
346362 then " Consider using --markdown-headings=setext."
347363 else ""
364+ EnvironmentVariableUndefined var ->
365+ "Undefined environment variable " <> var <> " in defaults file."
366+ DuplicateAttribute attr val ->
367+ "Ignoring duplicate attribute " <> attr <> "=" <> tshow val <> "."
348368
349369messageVerbosity :: LogMessage -> Verbosity
350370messageVerbosity msg =
@@ -375,6 +395,7 @@ messageVerbosity msg =
375395 CouldNotParseCSS{} -> WARNING
376396 Fetching{} -> INFO
377397 Extracting{} -> INFO
398+ LoadedResource{} -> INFO
378399 NoTitleElement{} -> WARNING
379400 NoLangSpecified -> INFO
380401 InvalidLang{} -> WARNING
@@ -391,3 +412,5 @@ messageVerbosity msg =
391412 FilterCompleted{} -> INFO
392413 CiteprocWarning{} -> WARNING
393414 ATXHeadingInLHS{} -> WARNING
415+ EnvironmentVariableUndefined{}-> WARNING
416+ DuplicateAttribute{} -> WARNING
src/Text/Pandoc/Lua.hs+1−1
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Lua
3− Copyright : Copyright © 2017–2020 Albert Krewinkel
3+ Copyright : Copyright © 2017-2021 Albert Krewinkel
44 License : GNU GPL, version 2 or above
55
66 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/ErrorConversion.hs+1−1
@@ -3,7 +3,7 @@
33{-# LANGUAGE ScopedTypeVariables #-}
44{- |
55 Module : Text.Pandoc.Lua.ErrorConversion
6− Copyright : © 2020 Albert Krewinkel
6+ Copyright : © 2020-2021 Albert Krewinkel
77 License : GNU GPL, version 2 or above
88
99 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Filter.hs+6−3
@@ -1,8 +1,8 @@
11{-# LANGUAGE FlexibleContexts #-}
22{- |
33Module : Text.Pandoc.Lua.Filter
4−Copyright : © 2012–2020 John MacFarlane,
5− © 2017-2020 Albert Krewinkel
4+Copyright : © 2012-2021 John MacFarlane,
5+ © 2017-2021 Albert Krewinkel
66License : GNU GPL, version 2 or above
77Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
88Stability : alpha
@@ -13,7 +13,9 @@ module Text.Pandoc.Lua.Filter ( LuaFilterFunction
1313 , LuaFilter
1414 , runFilterFile
1515 , walkInlines
16+ , walkInlineLists
1617 , walkBlocks
18+ , walkBlockLists
1719 , module Text.Pandoc.Lua.Walk
1820 ) where
1921import Control.Applicative ((<|>))
@@ -22,6 +24,7 @@ import Control.Monad.Catch (finally, try)
2224import Data.Data (Data, DataType, dataTypeConstrs, dataTypeName, dataTypeOf,
2325 showConstr, toConstr, tyconUQname)
2426import Data.Foldable (foldrM)
27+import Data.List (foldl')
2528import Data.Map (Map)
2629import Data.Maybe (fromMaybe)
2730import Foreign.Lua (Lua, Peekable, Pushable, StackIndex)
@@ -204,7 +207,7 @@ walkMeta lf (Pandoc m bs) = do
204207
205208walkPandoc :: LuaFilter -> Pandoc -> Lua Pandoc
206209walkPandoc (LuaFilter fnMap) =
207− case foldl mplus Nothing (map (`Map.lookup` fnMap) pandocFilterNames) of
210+ case foldl' mplus Nothing (map (`Map.lookup` fnMap) pandocFilterNames) of
208211 Just fn -> \x -> runFilterFunction fn x *> singleElement x
209212 Nothing -> return
210213
src/Text/Pandoc/Lua/Global.hs+1−1
@@ -1,7 +1,7 @@
11{-# LANGUAGE DeriveDataTypeable #-}
22{- |
33 Module : Text.Pandoc.Lua
4− Copyright : Copyright © 2017-2020 Albert Krewinkel
4+ Copyright : Copyright © 2017-2021 Albert Krewinkel
55 License : GNU GPL, version 2 or above
66
77 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Init.hs+15−5
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Lua
3− Copyright : Copyright © 2017-2020 Albert Krewinkel
3+ Copyright : Copyright © 2017-2021 Albert Krewinkel
44 License : GNU GPL, version 2 or above
55
66 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
@@ -12,17 +12,18 @@ module Text.Pandoc.Lua.Init
1212 ( runLua
1313 ) where
1414
15+import Control.Monad (when)
1516import Control.Monad.Catch (try)
1617import Control.Monad.Trans (MonadIO (..))
1718import Data.Data (Data, dataTypeConstrs, dataTypeOf, showConstr)
1819import Foreign.Lua (Lua)
1920import GHC.IO.Encoding (getForeignEncoding, setForeignEncoding, utf8)
21+import Text.Pandoc.Class.PandocMonad (readDataFile)
2022import Text.Pandoc.Class.PandocIO (PandocIO)
2123import Text.Pandoc.Error (PandocError)
2224import Text.Pandoc.Lua.Packages (installPandocPackageSearcher)
23−import Text.Pandoc.Lua.PandocLua (PandocLua, liftPandocLua,
24− loadScriptFromDataDir, runPandocLua)
25−
25+import Text.Pandoc.Lua.PandocLua (PandocLua, liftPandocLua, runPandocLua)
26+import Text.Pandoc.Lua.Util (throwTopMessageAsError')
2627import qualified Foreign.Lua as Lua
2728import qualified Text.Pandoc.Definition as Pandoc
2829import qualified Text.Pandoc.Lua.Module.Pandoc as ModulePandoc
@@ -44,7 +45,7 @@ initLuaState = do
4445 liftPandocLua Lua.openlibs
4546 installPandocPackageSearcher
4647 initPandocModule
47− loadScriptFromDataDir "init.lua"
48+ loadInitScript "init.lua"
4849 where
4950 initPandocModule :: PandocLua ()
5051 initPandocModule = do
@@ -61,6 +62,15 @@ initLuaState = do
6162 -- assign module to global variable
6263 liftPandocLua $ Lua.setglobal "pandoc"
6364
65+ loadInitScript :: FilePath -> PandocLua ()
66+ loadInitScript scriptFile = do
67+ script <- readDataFile scriptFile
68+ status <- liftPandocLua $ Lua.dostring script
69+ when (status /= Lua.OK) . liftPandocLua $
70+ throwTopMessageAsError'
71+ (("Couldn't load '" ++ scriptFile ++ "'.\n") ++)
72+
73+
6474-- | AST elements are marshaled via normal constructor functions in the
6575-- @pandoc@ module. However, accessing Lua globals from Haskell is
6676-- expensive (due to error handling). Accessing the Lua registry is much
src/Text/Pandoc/Lua/Marshaling.hs+2−2
@@ -1,7 +1,7 @@
11{- |
22 Module : Text.Pandoc.Lua.Marshaling
3− Copyright : © 2012-2020 John MacFarlane
4− © 2017-2020 Albert Krewinkel
3+ Copyright : © 2012-2021 John MacFarlane
4+ © 2017-2021 Albert Krewinkel
55 License : GNU GPL, version 2 or above
66
77 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Marshaling/AST.hs+79−81
@@ -1,9 +1,10 @@
11{-# OPTIONS_GHC -fno-warn-orphans #-}
2+{-# LANGUAGE BangPatterns #-}
23{-# LANGUAGE LambdaCase #-}
34{- |
45 Module : Text.Pandoc.Lua.Marshaling.AST
5− Copyright : © 2012-2020 John MacFarlane
6− © 2017-2020 Albert Krewinkel
6+ Copyright : © 2012-2021 John MacFarlane
7+ © 2017-2021 Albert Krewinkel
78 License : GNU GPL, version 2 or above
89
910 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
@@ -17,6 +18,7 @@ module Text.Pandoc.Lua.Marshaling.AST
1718 ) where
1819
1920import Control.Applicative ((<|>))
21+import Control.Monad ((<$!>))
2022import Foreign.Lua (Lua, Peekable, Pushable, StackIndex)
2123import Text.Pandoc.Definition
2224import Text.Pandoc.Error (PandocError)
@@ -32,17 +34,16 @@ instance Pushable Pandoc where
3234 pushViaConstructor "Pandoc" blocks meta
3335
3436instance Peekable Pandoc where
35− peek idx = defineHowTo "get Pandoc value" $ do
36− blocks <- LuaUtil.rawField idx "blocks"
37− meta <- LuaUtil.rawField idx "meta"
38− return $ Pandoc meta blocks
37+ peek idx = defineHowTo "get Pandoc value" $! Pandoc
38+ <$!> LuaUtil.rawField idx "meta"
39+ <*> LuaUtil.rawField idx "blocks"
3940
4041instance Pushable Meta where
4142 push (Meta mmap) =
4243 pushViaConstructor "Meta" mmap
4344instance Peekable Meta where
44− peek idx = defineHowTo "get Meta value" $
45− Meta <$> Lua.peek idx
45+ peek idx = defineHowTo "get Meta value" $!
46+ Meta <$!> Lua.peek idx
4647
4748instance Pushable MetaValue where
4849 push = pushMetaValue
@@ -68,14 +69,13 @@ instance Pushable Citation where
6869 pushViaConstructor "Citation" cid mode prefix suffix noteNum hash
6970
7071instance Peekable Citation where
71− peek idx = do
72− id' <- LuaUtil.rawField idx "id"
73− prefix <- LuaUtil.rawField idx "prefix"
74− suffix <- LuaUtil.rawField idx "suffix"
75− mode <- LuaUtil.rawField idx "mode"
76− num <- LuaUtil.rawField idx "note_num"
77− hash <- LuaUtil.rawField idx "hash"
78− return $ Citation id' prefix suffix mode num hash
72+ peek idx = Citation
73+ <$!> LuaUtil.rawField idx "id"
74+ <*> LuaUtil.rawField idx "prefix"
75+ <*> LuaUtil.rawField idx "suffix"
76+ <*> LuaUtil.rawField idx "mode"
77+ <*> LuaUtil.rawField idx "note_num"
78+ <*> LuaUtil.rawField idx "hash"
7979
8080instance Pushable Alignment where
8181 push = Lua.push . show
@@ -90,7 +90,7 @@ instance Peekable CitationMode where
9090instance Pushable Format where
9191 push (Format f) = Lua.push f
9292instance Peekable Format where
93− peek idx = Format <$> Lua.peek idx
93+ peek idx = Format <$!> Lua.peek idx
9494
9595instance Pushable ListNumberDelim where
9696 push = Lua.push . show
@@ -130,26 +130,26 @@ peekMetaValue idx = defineHowTo "get MetaValue" $ do
130130 elementContent = Lua.peek idx
131131 luatype <- Lua.ltype idx
132132 case luatype of
133− Lua.TypeBoolean -> MetaBool <$> Lua.peek idx
134− Lua.TypeString -> MetaString <$> Lua.peek idx
133+ Lua.TypeBoolean -> MetaBool <$!> Lua.peek idx
134+ Lua.TypeString -> MetaString <$!> Lua.peek idx
135135 Lua.TypeTable -> do
136136 tag <- try $ LuaUtil.getTag idx
137137 case tag of
138− Right "MetaBlocks" -> MetaBlocks <$> elementContent
139− Right "MetaBool" -> MetaBool <$> elementContent
140− Right "MetaMap" -> MetaMap <$> elementContent
141− Right "MetaInlines" -> MetaInlines <$> elementContent
142− Right "MetaList" -> MetaList <$> elementContent
143− Right "MetaString" -> MetaString <$> elementContent
138+ Right "MetaBlocks" -> MetaBlocks <$!> elementContent
139+ Right "MetaBool" -> MetaBool <$!> elementContent
140+ Right "MetaMap" -> MetaMap <$!> elementContent
141+ Right "MetaInlines" -> MetaInlines <$!> elementContent
142+ Right "MetaList" -> MetaList <$!> elementContent
143+ Right "MetaString" -> MetaString <$!> elementContent
144144 Right t -> Lua.throwMessage ("Unknown meta tag: " <> t)
145145 Left _ -> do
146146 -- no meta value tag given, try to guess.
147147 len <- Lua.rawlen idx
148148 if len <= 0
149− then MetaMap <$> Lua.peek idx
150− else (MetaInlines <$> Lua.peek idx)
151− <|> (MetaBlocks <$> Lua.peek idx)
152− <|> (MetaList <$> Lua.peek idx)
149+ then MetaMap <$!> Lua.peek idx
150+ else (MetaInlines <$!> Lua.peek idx)
151+ <|> (MetaBlocks <$!> Lua.peek idx)
152+ <|> (MetaList <$!> Lua.peek idx)
153153 _ -> Lua.throwMessage "could not get meta value"
154154
155155-- | Push a block element to the top of the Lua stack.
@@ -174,25 +174,25 @@ pushBlock = \case
174174
175175-- | Return the value at the given index as block if possible.
176176peekBlock :: StackIndex -> Lua Block
177−peekBlock idx = defineHowTo "get Block value" $ do
177+peekBlock idx = defineHowTo "get Block value" $! do
178178 tag <- LuaUtil.getTag idx
179179 case tag of
180− "BlockQuote" -> BlockQuote <$> elementContent
181− "BulletList" -> BulletList <$> elementContent
182− "CodeBlock" -> withAttr CodeBlock <$> elementContent
183− "DefinitionList" -> DefinitionList <$> elementContent
184− "Div" -> withAttr Div <$> elementContent
180+ "BlockQuote" -> BlockQuote <$!> elementContent
181+ "BulletList" -> BulletList <$!> elementContent
182+ "CodeBlock" -> withAttr CodeBlock <$!> elementContent
183+ "DefinitionList" -> DefinitionList <$!> elementContent
184+ "Div" -> withAttr Div <$!> elementContent
185185 "Header" -> (\(lvl, LuaAttr attr, lst) -> Header lvl attr lst)
186− <$> elementContent
186+ <$!> elementContent
187187 "HorizontalRule" -> return HorizontalRule
188− "LineBlock" -> LineBlock <$> elementContent
188+ "LineBlock" -> LineBlock <$!> elementContent
189189 "OrderedList" -> (\(LuaListAttributes lstAttr, lst) ->
190190 OrderedList lstAttr lst)
191− <$> elementContent
191+ <$!> elementContent
192192 "Null" -> return Null
193− "Para" -> Para <$> elementContent
194− "Plain" -> Plain <$> elementContent
195− "RawBlock" -> uncurry RawBlock <$> elementContent
193+ "Para" -> Para <$!> elementContent
194+ "Plain" -> Plain <$!> elementContent
195+ "RawBlock" -> uncurry RawBlock <$!> elementContent
196196 "Table" -> (\(attr, capt, colSpecs, thead, tbodies, tfoot) ->
197197 Table (fromLuaAttr attr)
198198 capt
@@ -200,7 +200,7 @@ peekBlock idx = defineHowTo "get Block value" $ do
200200 thead
201201 tbodies
202202 tfoot)
203− <$> elementContent
203+ <$!> elementContent
204204 _ -> Lua.throwMessage ("Unknown block type: " <> tag)
205205 where
206206 -- Get the contents of an AST element.
@@ -222,15 +222,14 @@ pushCaption (Caption shortCaption longCaption) = do
222222
223223-- | Peek Caption element
224224peekCaption :: StackIndex -> Lua Caption
225−peekCaption idx = do
226− short <- Lua.fromOptional <$> LuaUtil.rawField idx "short"
227− long <- LuaUtil.rawField idx "long"
228− return $ Caption short long
225+peekCaption idx = Caption
226+ <$!> (Lua.fromOptional <$!> LuaUtil.rawField idx "short")
227+ <*> LuaUtil.rawField idx "long"
229228
230229instance Peekable ColWidth where
231230 peek idx = do
232− width <- Lua.fromOptional <$> Lua.peek idx
233− return $ maybe ColWidthDefault ColWidth width
231+ width <- Lua.fromOptional <$!> Lua.peek idx
232+ return $! maybe ColWidthDefault ColWidth width
234233
235234instance Pushable ColWidth where
236235 push = \case
@@ -252,12 +251,11 @@ instance Pushable TableBody where
252251 LuaUtil.addField "body" body
253252
254253instance Peekable TableBody where
255− peek idx = do
256− attr <- LuaUtil.rawField idx "attr"
257− rowHeadColumns <- LuaUtil.rawField idx "row_head_columns"
258− head' <- LuaUtil.rawField idx "head"
259− body <- LuaUtil.rawField idx "body"
260− return $ TableBody attr (RowHeadColumns rowHeadColumns) head' body
254+ peek idx = TableBody
255+ <$!> LuaUtil.rawField idx "attr"
256+ <*> (RowHeadColumns <$!> LuaUtil.rawField idx "row_head_columns")
257+ <*> LuaUtil.rawField idx "head"
258+ <*> LuaUtil.rawField idx "body"
261259
262260instance Pushable TableHead where
263261 push (TableHead attr rows) = Lua.push (attr, rows)
@@ -287,13 +285,12 @@ pushCell (Cell attr align (RowSpan rowSpan) (ColSpan colSpan) contents) = do
287285 LuaUtil.addField "contents" contents
288286
289287peekCell :: StackIndex -> Lua Cell
290−peekCell idx = do
291− attr <- fromLuaAttr <$> LuaUtil.rawField idx "attr"
292− align <- LuaUtil.rawField idx "alignment"
293− rowSpan <- LuaUtil.rawField idx "row_span"
294− colSpan <- LuaUtil.rawField idx "col_span"
295− contents <- LuaUtil.rawField idx "contents"
296− return $ Cell attr align (RowSpan rowSpan) (ColSpan colSpan) contents
288+peekCell idx = Cell
289+ <$!> (fromLuaAttr <$!> LuaUtil.rawField idx "attr")
290+ <*> LuaUtil.rawField idx "alignment"
291+ <*> (RowSpan <$!> LuaUtil.rawField idx "row_span")
292+ <*> (ColSpan <$!> LuaUtil.rawField idx "col_span")
293+ <*> LuaUtil.rawField idx "contents"
297294
298295-- | Push an inline element to the top of the lua stack.
299296pushInline :: Inline -> Lua ()
@@ -324,28 +321,29 @@ peekInline :: StackIndex -> Lua Inline
324321peekInline idx = defineHowTo "get Inline value" $ do
325322 tag <- LuaUtil.getTag idx
326323 case tag of
327− "Cite" -> uncurry Cite <$> elementContent
328− "Code" -> withAttr Code <$> elementContent
329− "Emph" -> Emph <$> elementContent
330− "Underline" -> Underline <$> elementContent
331− "Image" -> (\(LuaAttr attr, lst, tgt) -> Image attr lst tgt)
332− <$> elementContent
333− "Link" -> (\(LuaAttr attr, lst, tgt) -> Link attr lst tgt)
334− <$> elementContent
324+ "Cite" -> uncurry Cite <$!> elementContent
325+ "Code" -> withAttr Code <$!> elementContent
326+ "Emph" -> Emph <$!> elementContent
327+ "Underline" -> Underline <$!> elementContent
328+ "Image" -> (\(LuaAttr !attr, !lst, !tgt) -> Image attr lst tgt)
329+ <$!> elementContent
330+ "Link" -> (\(LuaAttr !attr, !lst, !tgt) -> Link attr lst tgt)
331+ <$!> elementContent
335332 "LineBreak" -> return LineBreak
336− "Note" -> Note <$> elementContent
337− "Math" -> uncurry Math <$> elementContent
338− "Quoted" -> uncurry Quoted <$> elementContent
339− "RawInline" -> uncurry RawInline <$> elementContent
340− "SmallCaps" -> SmallCaps <$> elementContent
333+ "Note" -> Note <$!> elementContent
334+ "Math" -> uncurry Math <$!> elementContent
335+ "Quoted" -> uncurry Quoted <$!> elementContent
336+ "RawInline" -> uncurry RawInline <$!> elementContent
337+ "SmallCaps" -> SmallCaps <$!> elementContent
341338 "SoftBreak" -> return SoftBreak
342339 "Space" -> return Space
343− "Span" -> withAttr Span <$> elementContent
344− "Str" -> Str <$> elementContent
345− "Strikeout" -> Strikeout <$> elementContent
346− "Strong" -> Strong <$> elementContent
347− "Subscript" -> Subscript <$> elementContent
348− "Superscript"-> Superscript <$> elementContent
340+ "Span" -> withAttr Span <$!> elementContent
341+ -- strict to Lua string is copied before gc
342+ "Str" -> Str <$!> elementContent
343+ "Strikeout" -> Strikeout <$!> elementContent
344+ "Strong" -> Strong <$!> elementContent
345+ "Subscript" -> Subscript <$!> elementContent
346+ "Superscript"-> Superscript <$!> elementContent
349347 _ -> Lua.throwMessage ("Unknown inline type: " <> tag)
350348 where
351349 -- Get the contents of an AST element.
@@ -366,7 +364,7 @@ instance Pushable LuaAttr where
366364 pushViaConstructor "Attr" id' classes kv
367365
368366instance Peekable LuaAttr where
369− peek idx = defineHowTo "get Attr value" (LuaAttr <$> Lua.peek idx)
367+ peek idx = defineHowTo "get Attr value" $! (LuaAttr <$!> Lua.peek idx)
370368
371369-- | Wrapper for ListAttributes
372370newtype LuaListAttributes = LuaListAttributes ListAttributes
src/Text/Pandoc/Lua/Marshaling/AnyValue.hs+1−1
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Lua.Marshaling.AnyValue
3− Copyright : © 2017-2020 Albert Krewinkel
3+ Copyright : © 2017-2021 Albert Krewinkel
44 License : GNU GPL, version 2 or above
55
66 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Marshaling/CommonState.hs+2−2
@@ -3,8 +3,8 @@
33{-# LANGUAGE OverloadedStrings #-}
44{- |
55 Module : Text.Pandoc.Lua.Marshaling.CommonState
6− Copyright : © 2012-2020 John MacFarlane
7− © 2017-2020 Albert Krewinkel
6+ Copyright : © 2012-2021 John MacFarlane
7+ © 2017-2021 Albert Krewinkel
88 License : GNU GPL, version 2 or above
99 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
1010 Stability : alpha
src/Text/Pandoc/Lua/Marshaling/Context.hs+2−2
@@ -1,8 +1,8 @@
11{-# OPTIONS_GHC -fno-warn-orphans #-}
22{- |
33 Module : Text.Pandoc.Lua.Marshaling.Context
4− Copyright : © 2012-2020 John MacFarlane
5− © 2017-2020 Albert Krewinkel
4+ Copyright : © 2012-2021 John MacFarlane
5+ © 2017-2021 Albert Krewinkel
66 License : GNU GPL, version 2 or above
77
88 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Marshaling/List.hs+2−2
@@ -4,8 +4,8 @@
44{-# LANGUAGE UndecidableInstances #-}
55{- |
66Module : Text.Pandoc.Lua.Marshaling.List
7−Copyright : © 2012-2020 John MacFarlane
8− © 2017-2020 Albert Krewinkel
7+Copyright : © 2012-2021 John MacFarlane
8+ © 2017-2021 Albert Krewinkel
99License : GNU GPL, version 2 or above
1010Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
1111Stability : alpha
src/Text/Pandoc/Lua/Marshaling/MediaBag.hs+2−2
@@ -1,7 +1,7 @@
11{- |
22 Module : Text.Pandoc.Lua.Marshaling.MediaBag
3− Copyright : © 2012-2020 John MacFarlane
4− © 2017-2020 Albert Krewinkel
3+ Copyright : © 2012-2021 John MacFarlane
4+ © 2017-2021 Albert Krewinkel
55 License : GNU GPL, version 2 or above
66 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
77 Stability : alpha
src/Text/Pandoc/Lua/Marshaling/PandocError.hs+1−1
@@ -4,7 +4,7 @@
44{-# LANGUAGE ScopedTypeVariables #-}
55{- |
66 Module : Text.Pandoc.Lua.Marshaling.PandocError
7− Copyright : © 2020 Albert Krewinkel
7+ Copyright : © 2020-2021 Albert Krewinkel
88 License : GNU GPL, version 2 or above
99
1010 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Marshaling/ReaderOptions.hs+2−2
@@ -4,8 +4,8 @@
44{-# OPTIONS_GHC -fno-warn-orphans #-}
55{- |
66 Module : Text.Pandoc.Lua.Marshaling.ReaderOptions
7− Copyright : © 2012-2020 John MacFarlane
8− © 2017-2020 Albert Krewinkel
7+ Copyright : © 2012-2021 John MacFarlane
8+ © 2017-2021 Albert Krewinkel
99 License : GNU GPL, version 2 or above
1010
1111 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Marshaling/SimpleTable.hs+1−1
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Lua.Marshaling.SimpleTable
3− Copyright : © 2020 Albert Krewinkel
3+ Copyright : © 2020-2021 Albert Krewinkel
44 License : GNU GPL, version 2 or above
55
66 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Marshaling/Version.hs+1−1
@@ -4,7 +4,7 @@
44{-# LANGUAGE ScopedTypeVariables #-}
55{- |
66 Module : Text.Pandoc.Lua.Marshaling.Version
7− Copyright : © 2019-2020 Albert Krewinkel
7+ Copyright : © 2019-2021 Albert Krewinkel
88 License : GNU GPL, version 2 or above
99
1010 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Module/MediaBag.hs+18−17
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.Lua.Module.MediaBag
4− Copyright : Copyright © 2017-2020 Albert Krewinkel
4+ Copyright : Copyright © 2017-2021 Albert Krewinkel
55 License : GNU GPL, version 2 or above
66
77 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
@@ -13,6 +13,7 @@ module Text.Pandoc.Lua.Module.MediaBag
1313 ( pushModule
1414 ) where
1515
16+import Prelude hiding (lookup)
1617import Control.Monad (zipWithM_)
1718import Foreign.Lua (Lua, NumResults, Optional)
1819import Text.Pandoc.Class.CommonState (CommonState (..))
@@ -36,10 +37,10 @@ pushModule = do
3637 liftPandocLua Lua.newtable
3738 addFunction "delete" delete
3839 addFunction "empty" empty
39− addFunction "insert" insertMediaFn
40+ addFunction "insert" insert
4041 addFunction "items" items
41− addFunction "lookup" lookupMediaFn
42− addFunction "list" mediaDirectoryFn
42+ addFunction "lookup" lookup
43+ addFunction "list" list
4344 addFunction "fetch" fetch
4445 return 1
4546
@@ -53,11 +54,11 @@ empty :: PandocLua NumResults
5354empty = 0 <$ modifyCommonState (\st -> st { stMediaBag = mempty })
5455
5556-- | Insert a new item into the media bag.
56−insertMediaFn :: FilePath
57− -> Optional MimeType
58− -> BL.ByteString
59− -> PandocLua NumResults
60−insertMediaFn fp optionalMime contents = do
57+insert :: FilePath
58+ -> Optional MimeType
59+ -> BL.ByteString
60+ -> PandocLua NumResults
61+insert fp optionalMime contents = do
6162 mb <- getMediaBag
6263 setMediaBag $ MB.insertMedia fp (Lua.fromOptional optionalMime) contents mb
6364 return (Lua.NumResults 0)
@@ -66,19 +67,19 @@ insertMediaFn fp optionalMime contents = do
6667items :: PandocLua NumResults
6768items = getMediaBag >>= liftPandocLua . pushIterator
6869
69−lookupMediaFn :: FilePath
70− -> PandocLua NumResults
71−lookupMediaFn fp = do
70+lookup :: FilePath
71+ -> PandocLua NumResults
72+lookup fp = do
7273 res <- MB.lookupMedia fp <$> getMediaBag
7374 liftPandocLua $ case res of
7475 Nothing -> 1 <$ Lua.pushnil
75− Just (mimeType, contents) -> do
76− Lua.push mimeType
77− Lua.push contents
76+ Just item -> do
77+ Lua.push $ MB.mediaMimeType item
78+ Lua.push $ MB.mediaContents item
7879 return 2
7980
80−mediaDirectoryFn :: PandocLua NumResults
81−mediaDirectoryFn = do
81+list :: PandocLua NumResults
82+list = do
8283 dirContents <- MB.mediaDirectory <$> getMediaBag
8384 liftPandocLua $ do
8485 Lua.newtable
src/Text/Pandoc/Lua/Module/Pandoc.hs+29−23
@@ -2,7 +2,7 @@
22{-# LANGUAGE OverloadedStrings #-}
33{- |
44 Module : Text.Pandoc.Lua.Module.Pandoc
5− Copyright : Copyright © 2017-2020 Albert Krewinkel
5+ Copyright : Copyright © 2017-2021 Albert Krewinkel
66 License : GNU GPL, version 2 or above
77
88 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
@@ -14,6 +14,7 @@ module Text.Pandoc.Lua.Module.Pandoc
1414 ( pushModule
1515 ) where
1616
17+import Prelude hiding (read)
1718import Control.Monad (when)
1819import Control.Monad.Except (throwError)
1920import Data.Default (Default (..))
@@ -22,10 +23,12 @@ import Foreign.Lua (Lua, NumResults, Optional, Peekable, Pushable)
2223import System.Exit (ExitCode (..))
2324import Text.Pandoc.Class.PandocIO (runIO)
2425import Text.Pandoc.Definition (Block, Inline)
25−import Text.Pandoc.Lua.Filter (walkInlines, walkBlocks, LuaFilter, SingletonsList (..))
26+import Text.Pandoc.Lua.Filter (LuaFilter, SingletonsList (..), walkInlines,
27+ walkInlineLists, walkBlocks, walkBlockLists)
2628import Text.Pandoc.Lua.Marshaling ()
29+import Text.Pandoc.Lua.Marshaling.List (List (..))
2730import Text.Pandoc.Lua.PandocLua (PandocLua, addFunction, liftPandocLua,
28− loadScriptFromDataDir)
31+ loadDefaultModule)
2932import Text.Pandoc.Walk (Walkable)
3033import Text.Pandoc.Options (ReaderOptions (readerExtensions))
3134import Text.Pandoc.Process (pipeProcess)
@@ -38,30 +41,33 @@ import qualified Foreign.Lua as Lua
3841import qualified Text.Pandoc.Lua.Util as LuaUtil
3942import Text.Pandoc.Error
4043
41−-- | Push the "pandoc" on the lua stack. Requires the `list` module to be
42−-- loaded.
44+-- | Push the "pandoc" package to the Lua stack. Requires the `List`
45+-- module to be loadable.
4346pushModule :: PandocLua NumResults
4447pushModule = do
45− loadScriptFromDataDir "pandoc.lua"
46− addFunction "read" readDoc
47− addFunction "pipe" pipeFn
48− addFunction "walk_block" walkBlock
49− addFunction "walk_inline" walkInline
48+ loadDefaultModule "pandoc"
49+ addFunction "read" read
50+ addFunction "pipe" pipe
51+ addFunction "walk_block" walk_block
52+ addFunction "walk_inline" walk_inline
5053 return 1
5154
5255walkElement :: (Walkable (SingletonsList Inline) a,
53− Walkable (SingletonsList Block) a)
56+ Walkable (SingletonsList Block) a,
57+ Walkable (List Inline) a,
58+ Walkable (List Block) a)
5459 => a -> LuaFilter -> PandocLua a
55−walkElement x f = liftPandocLua $ walkInlines f x >>= walkBlocks f
60+walkElement x f = liftPandocLua $
61+ walkInlines f x >>= walkInlineLists f >>= walkBlocks f >>= walkBlockLists f
5662
57−walkInline :: Inline -> LuaFilter -> PandocLua Inline
58−walkInline = walkElement
63+walk_inline :: Inline -> LuaFilter -> PandocLua Inline
64+walk_inline = walkElement
5965
60−walkBlock :: Block -> LuaFilter -> PandocLua Block
61−walkBlock = walkElement
66+walk_block :: Block -> LuaFilter -> PandocLua Block
67+walk_block = walkElement
6268
63−readDoc :: T.Text -> Optional T.Text -> PandocLua NumResults
64−readDoc content formatSpecOrNil = liftPandocLua $ do
69+read :: T.Text -> Optional T.Text -> PandocLua NumResults
70+read content formatSpecOrNil = liftPandocLua $ do
6571 let formatSpec = fromMaybe "markdown" (Lua.fromOptional formatSpecOrNil)
6672 res <- Lua.liftIO . runIO $
6773 getReader formatSpec >>= \(rdr,es) ->
@@ -79,11 +85,11 @@ readDoc content formatSpecOrNil = liftPandocLua $ do
7985 Left e -> Lua.raiseError $ show e
8086
8187-- | Pipes input through a command.
82−pipeFn :: String
83− -> [String]
84− -> BL.ByteString
85− -> PandocLua NumResults
86−pipeFn command args input = liftPandocLua $ do
88+pipe :: String -- ^ path to executable
89+ -> [String] -- ^ list of arguments
90+ -> BL.ByteString -- ^ input passed to process via stdin
91+ -> PandocLua NumResults
92+pipe command args input = liftPandocLua $ do
8793 (ec, output) <- Lua.liftIO $ pipeProcess Nothing command args input
8894 case ec of
8995 ExitSuccess -> 1 <$ Lua.push output
src/Text/Pandoc/Lua/Module/System.hs+1−1
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Lua.Module.System
3− Copyright : © 2019-2020 Albert Krewinkel
3+ Copyright : © 2019-2021 Albert Krewinkel
44 License : GNU GPL, version 2 or above
55
66 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Module/Types.hs+1−1
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Lua.Module.Types
3− Copyright : © 2019-2020 Albert Krewinkel
3+ Copyright : © 2019-2021 Albert Krewinkel
44 License : GNU GPL, version 2 or above
55
66 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Module/Utils.hs+2−2
@@ -2,7 +2,7 @@
22{-# LANGUAGE ScopedTypeVariables #-}
33{- |
44 Module : Text.Pandoc.Lua.Module.Utils
5− Copyright : Copyright © 2017-2020 Albert Krewinkel
5+ Copyright : Copyright © 2017-2021 Albert Krewinkel
66 License : GNU GPL, version 2 or above
77
88 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
@@ -146,7 +146,7 @@ from_simple_table (SimpleTable capt aligns widths head' body) = do
146146 nullAttr
147147 (Caption Nothing [Plain capt])
148148 (zipWith (\a w -> (a, toColWidth w)) aligns widths)
149− (TableHead nullAttr [blockListToRow head'])
149+ (TableHead nullAttr [blockListToRow head' | not (null head') ])
150150 [TableBody nullAttr 0 [] $ map blockListToRow body]
151151 (TableFoot nullAttr [])
152152 return (NumResults 1)
src/Text/Pandoc/Lua/Packages.hs+10−27
@@ -1,9 +1,6 @@
1−{-# LANGUAGE FlexibleContexts #-}
2−{-# LANGUAGE LambdaCase #-}
3−{-# LANGUAGE ScopedTypeVariables #-}
41{- |
52 Module : Text.Pandoc.Lua.Packages
6− Copyright : Copyright © 2017-2020 Albert Krewinkel
3+ Copyright : Copyright © 2017-2021 Albert Krewinkel
74 License : GNU GPL, version 2 or above
85
96 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
@@ -15,15 +12,12 @@ module Text.Pandoc.Lua.Packages
1512 ( installPandocPackageSearcher
1613 ) where
1714
18−import Control.Monad.Catch (try)
1915import Control.Monad (forM_)
20−import Data.ByteString (ByteString)
21−import Foreign.Lua (Lua, NumResults)
22−import Text.Pandoc.Error (PandocError)
23−import Text.Pandoc.Class.PandocMonad (readDataFile)
24−import Text.Pandoc.Lua.PandocLua (PandocLua, liftPandocLua)
16+import Foreign.Lua (NumResults)
17+import Text.Pandoc.Lua.PandocLua (PandocLua, liftPandocLua, loadDefaultModule)
2518
2619import qualified Foreign.Lua as Lua
20+import qualified Foreign.Lua.Module.Path as Path
2721import qualified Foreign.Lua.Module.Text as Text
2822import qualified Text.Pandoc.Lua.Module.Pandoc as Pandoc
2923import qualified Text.Pandoc.Lua.Module.MediaBag as MediaBag
@@ -50,28 +44,17 @@ pandocPackageSearcher pkgName =
5044 case pkgName of
5145 "pandoc" -> pushWrappedHsFun Pandoc.pushModule
5246 "pandoc.mediabag" -> pushWrappedHsFun MediaBag.pushModule
47+ "pandoc.path" -> pushWrappedHsFun Path.pushModule
5348 "pandoc.system" -> pushWrappedHsFun System.pushModule
5449 "pandoc.types" -> pushWrappedHsFun Types.pushModule
5550 "pandoc.utils" -> pushWrappedHsFun Utils.pushModule
5651 "text" -> pushWrappedHsFun Text.pushModule
57− _ -> searchPureLuaLoader
52+ "pandoc.List" -> pushWrappedHsFun (loadDefaultModule pkgName)
53+ _ -> reportPandocSearcherFailure
5854 where
5955 pushWrappedHsFun f = liftPandocLua $ do
6056 Lua.pushHaskellFunction f
6157 return 1
62− searchPureLuaLoader = do
63− let filename = pkgName ++ ".lua"
64− try (readDataFile filename) >>= \case
65− Right script -> pushWrappedHsFun (loadStringAsPackage pkgName script)
66− Left (_ :: PandocError) -> liftPandocLua $ do
67− Lua.push ("\n\tno file '" ++ filename ++ "' in pandoc's datadir")
68− return (1 :: NumResults)
69−
70−loadStringAsPackage :: String -> ByteString -> Lua NumResults
71−loadStringAsPackage pkgName script = do
72− status <- Lua.dostring script
73− if status == Lua.OK
74− then return (1 :: NumResults)
75− else do
76− msg <- Lua.popValue
77− Lua.raiseError ("Error while loading `" <> pkgName <> "`.\n" <> msg)
58+ reportPandocSearcherFailure = liftPandocLua $ do
59+ Lua.push ("\n\t" <> pkgName <> "is not one of pandoc's default packages")
60+ return (1 :: NumResults)
src/Text/Pandoc/Lua/PandocLua.hs+20−13
@@ -9,7 +9,7 @@
99{-# OPTIONS_GHC -fno-warn-orphans #-}
1010{- |
1111 Module : Text.Pandoc.Lua.PandocLua
12− Copyright : Copyright © 2020 Albert Krewinkel
12+ Copyright : Copyright © 2020-2021 Albert Krewinkel
1313 License : GNU GPL, version 2 or above
1414
1515 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
@@ -23,24 +23,23 @@ module Text.Pandoc.Lua.PandocLua
2323 , runPandocLua
2424 , liftPandocLua
2525 , addFunction
26− , loadScriptFromDataDir
26+ , loadDefaultModule
2727 ) where
2828
29−import Control.Monad (when)
3029import Control.Monad.Catch (MonadCatch, MonadMask, MonadThrow)
3130import Control.Monad.Except (MonadError (catchError, throwError))
3231import Control.Monad.IO.Class (MonadIO (liftIO))
3332import Foreign.Lua (Lua (..), NumResults, Pushable, ToHaskellFunction)
3433import Text.Pandoc.Class.PandocIO (PandocIO)
35−import Text.Pandoc.Class.PandocMonad (PandocMonad (..), readDataFile)
36−import Text.Pandoc.Error (PandocError)
34+import Text.Pandoc.Class.PandocMonad (PandocMonad (..), readDefaultDataFile)
35+import Text.Pandoc.Error (PandocError (PandocLuaError))
3736import Text.Pandoc.Lua.Global (Global (..), setGlobals)
3837import Text.Pandoc.Lua.ErrorConversion (errorConversion)
3938
4039import qualified Control.Monad.Catch as Catch
40+import qualified Data.Text as T
4141import qualified Foreign.Lua as Lua
4242import qualified Text.Pandoc.Class.IO as IO
43−import qualified Text.Pandoc.Lua.Util as LuaUtil
4443
4544-- | Type providing access to both, pandoc and Lua operations.
4645newtype PandocLua a = PandocLua { unPandocLua :: Lua a }
@@ -86,14 +85,22 @@ addFunction name fn = liftPandocLua $ do
8685 Lua.pushHaskellFunction fn
8786 Lua.rawset (-3)
8887
89−-- | Load a file from pandoc's data directory.
90−loadScriptFromDataDir :: FilePath -> PandocLua ()
91−loadScriptFromDataDir scriptFile = do
92− script <- readDataFile scriptFile
88+-- | Load a pure Lua module included with pandoc. Leaves the result on
89+-- the stack and returns @NumResults 1@.
90+--
91+-- The script is loaded from the default data directory. We do not load
92+-- from data directories supplied via command line, as this could cause
93+-- scripts to be executed even though they had not been passed explicitly.
94+loadDefaultModule :: String -> PandocLua NumResults
95+loadDefaultModule name = do
96+ script <- readDefaultDataFile (name <> ".lua")
9397 status <- liftPandocLua $ Lua.dostring script
94− when (status /= Lua.OK) . liftPandocLua $
95− LuaUtil.throwTopMessageAsError'
96− (("Couldn't load '" ++ scriptFile ++ "'.\n") ++)
98+ if status == Lua.OK
99+ then return (1 :: NumResults)
100+ else do
101+ msg <- liftPandocLua Lua.popValue
102+ let err = "Error while loading `" <> name <> "`.\n" <> msg
103+ throwError $ PandocLuaError (T.pack err)
97104
98105-- | Global variables which should always be set.
99106defaultGlobals :: PandocIO [Global]
src/Text/Pandoc/Lua/Util.hs+2−2
@@ -3,8 +3,8 @@
33{-# LANGUAGE OverloadedStrings #-}
44{- |
55 Module : Text.Pandoc.Lua.Util
6− Copyright : © 2012–2020 John MacFarlane,
7− © 2017-2020 Albert Krewinkel
6+ Copyright : © 2012-2021 John MacFarlane,
7+ © 2017-2021 Albert Krewinkel
88 License : GNU GPL, version 2 or above
99
1010 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
src/Text/Pandoc/Lua/Walk.hs+2−2
@@ -4,8 +4,8 @@
44{-# LANGUAGE MultiParamTypeClasses #-}
55{- |
66Module : Text.Pandoc.Lua.Walk
7−Copyright : © 2012–2020 John MacFarlane,
8− © 2017-2020 Albert Krewinkel
7+Copyright : © 2012-2021 John MacFarlane,
8+ © 2017-2021 Albert Krewinkel
99License : GNU GPL, version 2 or above
1010Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
1111Stability : alpha
src/Text/Pandoc/MIME.hs+26−3
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.MIME
4− Copyright : Copyright (C) 2011-2020 John MacFarlane
4+ Copyright : Copyright (C) 2011-2021 John MacFarlane
55 License : GNU GPL, version 2 or above
66
77 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -10,8 +10,13 @@
1010
1111Mime type lookup.
1212-}
13−module Text.Pandoc.MIME ( MimeType, getMimeType, getMimeTypeDef,
14− extensionFromMimeType, mediaCategory ) where
13+module Text.Pandoc.MIME (
14+ MimeType,
15+ getMimeType,
16+ getMimeTypeDef,
17+ getCharset,
18+ extensionFromMimeType,
19+ mediaCategory ) where
1520import Data.List (isPrefixOf, isSuffixOf)
1621import qualified Data.Map as M
1722import qualified Data.Text as T
@@ -38,6 +43,16 @@ getMimeTypeDef :: FilePath -> MimeType
3843getMimeTypeDef = fromMaybe "application/octet-stream" . getMimeType
3944
4045extensionFromMimeType :: MimeType -> Maybe T.Text
46+-- few special cases, where there are multiple options:
47+extensionFromMimeType "text/plain" = Just "txt"
48+extensionFromMimeType "video/quicktime" = Just "mov"
49+extensionFromMimeType "video/mpeg" = Just "mpeg"
50+extensionFromMimeType "video/dv" = Just "dv"
51+extensionFromMimeType "image/vnd.djvu" = Just "djvu"
52+extensionFromMimeType "image/tiff" = Just "tiff"
53+extensionFromMimeType "image/jpeg" = Just "jpg"
54+extensionFromMimeType "application/xml" = Just "xml"
55+extensionFromMimeType "application/ogg" = Just "ogg"
4156extensionFromMimeType mimetype =
4257 M.lookup (T.takeWhile (/=';') mimetype) reverseMimeTypes
4358 -- note: we just look up the basic mime type, dropping the content-encoding etc.
@@ -54,6 +69,14 @@ reverseMimeTypes = M.fromList $ map swap mimeTypesList
5469mimeTypes :: M.Map T.Text MimeType
5570mimeTypes = M.fromList mimeTypesList
5671
72+-- | Get the charset from a mime type, if one is present.
73+getCharset :: MimeType -> Maybe T.Text
74+getCharset mt =
75+ let (_,y) = T.breakOn "charset=" mt
76+ in if T.null y
77+ then Nothing
78+ else Just $ T.toUpper $ T.takeWhile (/= ';') $ T.drop 8 y
79+
5780-- | Collection of common mime types.
5881-- Except for first entry, list borrowed from
5982-- <https://github.com/Happstack/happstack-server/blob/master/src/Happstack/Server/FileServe/BuildingBlocks.hs happstack-server>
src/Text/Pandoc/MediaBag.hs+46−15
@@ -1,9 +1,10 @@
11{-# LANGUAGE CPP #-}
2+{-# LANGUAGE OverloadedStrings #-}
23{-# LANGUAGE DeriveDataTypeable #-}
34{-# LANGUAGE GeneralizedNewtypeDeriving #-}
45{- |
56 Module : Text.Pandoc.MediaBag
6− Copyright : Copyright (C) 2014-2015, 2017–2020 John MacFarlane
7+ Copyright : Copyright (C) 2014-2015, 2017-2021 John MacFarlane
78 License : GNU GPL, version 2 or above
89
910 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -14,6 +15,7 @@ Definition of a MediaBag object to hold binary resources, and an
1415interface for interacting with it.
1516-}
1617module Text.Pandoc.MediaBag (
18+ MediaItem(..),
1719 MediaBag,
1820 deleteMedia,
1921 lookupMedia,
@@ -24,29 +26,43 @@ module Text.Pandoc.MediaBag (
2426import qualified Data.ByteString.Lazy as BL
2527import Data.Data (Data)
2628import qualified Data.Map as M
27−import Data.Maybe (fromMaybe)
29+import Data.Maybe (fromMaybe, isNothing)
2830import Data.Typeable (Typeable)
2931import System.FilePath
30−import qualified System.FilePath.Posix as Posix
31−import Text.Pandoc.MIME (MimeType, getMimeTypeDef)
32+import Text.Pandoc.MIME (MimeType, getMimeTypeDef, extensionFromMimeType)
33+import Data.Text (Text)
34+import qualified Data.Text as T
35+import Data.Digest.Pure.SHA (sha1, showDigest)
36+import Network.URI (URI (..), parseURI)
37+
38+data MediaItem =
39+ MediaItem
40+ { mediaMimeType :: MimeType
41+ , mediaPath :: FilePath
42+ , mediaContents :: BL.ByteString
43+ } deriving (Eq, Ord, Show, Data, Typeable)
3244
3345-- | A container for a collection of binary resources, with names and
3446-- mime types. Note that a 'MediaBag' is a Monoid, so 'mempty'
3547-- can be used for an empty 'MediaBag', and '<>' can be used to append
3648-- two 'MediaBag's.
37−newtype MediaBag = MediaBag (M.Map [FilePath] (MimeType, BL.ByteString))
49+newtype MediaBag = MediaBag (M.Map Text MediaItem)
3850 deriving (Semigroup, Monoid, Data, Typeable)
3951
4052instance Show MediaBag where
4153 show bag = "MediaBag " ++ show (mediaDirectory bag)
4254
55+-- | We represent paths with /, in normalized form.
56+canonicalize :: FilePath -> Text
57+canonicalize = T.replace "\\" "/" . T.pack . normalise
58+
4359-- | Delete a media item from a 'MediaBag', or do nothing if no item corresponds
4460-- to the given path.
4561deleteMedia :: FilePath -- ^ relative path and canonical name of resource
4662 -> MediaBag
4763 -> MediaBag
4864deleteMedia fp (MediaBag mediamap) =
49− MediaBag $ M.delete (splitDirectories fp) mediamap
65+ MediaBag $ M.delete (canonicalize fp) mediamap
5066
5167-- | Insert a media item into a 'MediaBag', replacing any existing
5268-- value with the same name.
@@ -56,26 +72,41 @@ insertMedia :: FilePath -- ^ relative path and canonical name of resource
5672 -> MediaBag
5773 -> MediaBag
5874insertMedia fp mbMime contents (MediaBag mediamap) =
59− MediaBag (M.insert (splitDirectories fp) (mime, contents) mediamap)
60− where mime = fromMaybe fallback mbMime
75+ MediaBag (M.insert fp' mediaItem mediamap)
76+ where mediaItem = MediaItem{ mediaPath = newpath
77+ , mediaContents = contents
78+ , mediaMimeType = mt }
79+ fp' = canonicalize fp
80+ uri = parseURI fp
81+ newpath = if isRelative fp
82+ && isNothing uri
83+ && ".." `notElem` splitPath fp
84+ then T.unpack fp'
85+ else showDigest (sha1 contents) <> "." <> ext
6186 fallback = case takeExtension fp of
6287 ".gz" -> getMimeTypeDef $ dropExtension fp
6388 _ -> getMimeTypeDef fp
89+ mt = fromMaybe fallback mbMime
90+ path = maybe fp uriPath uri
91+ ext = case takeExtension path of
92+ '.':e -> e
93+ _ -> maybe "" T.unpack $ extensionFromMimeType mt
94+
6495
6596-- | Lookup a media item in a 'MediaBag', returning mime type and contents.
6697lookupMedia :: FilePath
6798 -> MediaBag
68− -> Maybe (MimeType, BL.ByteString)
69−lookupMedia fp (MediaBag mediamap) = M.lookup (splitDirectories fp) mediamap
99+ -> Maybe MediaItem
100+lookupMedia fp (MediaBag mediamap) = M.lookup (canonicalize fp) mediamap
70101
71102-- | Get a list of the file paths stored in a 'MediaBag', with
72103-- their corresponding mime types and the lengths in bytes of the contents.
73104mediaDirectory :: MediaBag -> [(FilePath, MimeType, Int)]
74−mediaDirectory (MediaBag mediamap) =
75− M.foldrWithKey (\fp (mime,contents) ->
76− ((Posix.joinPath fp, mime, fromIntegral $ BL.length contents):)) [] mediamap
105+mediaDirectory mediabag =
106+ map (\(fp, mt, bs) -> (fp, mt, fromIntegral (BL.length bs)))
107+ (mediaItems mediabag)
77108
78109mediaItems :: MediaBag -> [(FilePath, MimeType, BL.ByteString)]
79110mediaItems (MediaBag mediamap) =
80− M.foldrWithKey (\fp (mime,contents) ->
81− ((Posix.joinPath fp, mime, contents):)) [] mediamap
111+ map (\item -> (mediaPath item, mediaMimeType item, mediaContents item))
112+ (M.elems mediamap)
src/Text/Pandoc/Options.hs+18−18
@@ -6,7 +6,7 @@
66{-# LANGUAGE TemplateHaskell #-}
77{- |
88 Module : Text.Pandoc.Options
9− Copyright : Copyright (C) 2012-2020 John MacFarlane
9+ Copyright : Copyright (C) 2012-2021 John MacFarlane
1010 License : GNU GPL, version 2 or above
1111
1212 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -315,6 +315,23 @@ defaultMathJaxURL = "https://cdn.jsdelivr.net/npm/mathjax@3/es5/tex-chtml-full.j
315315defaultKaTeXURL :: Text
316316defaultKaTeXURL = "https://cdnjs.cloudflare.com/ajax/libs/KaTeX/0.11.1/"
317317
318+-- Update documentation in doc/filters.md if this is changed.
319+$(deriveJSON defaultOptions{ constructorTagModifier =
320+ camelCaseStrToHyphenated
321+ } ''TrackChanges)
322+
323+$(deriveJSON defaultOptions{ constructorTagModifier =
324+ camelCaseStrToHyphenated
325+ } ''WrapOption)
326+
327+$(deriveJSON defaultOptions{ constructorTagModifier =
328+ camelCaseStrToHyphenated . drop 8
329+ } ''TopLevelDivision)
330+
331+$(deriveJSON defaultOptions{ constructorTagModifier =
332+ camelCaseStrToHyphenated
333+ } ''ReferenceLocation)
334+
318335-- Update documentation in doc/filters.md if this is changed.
319336$(deriveJSON defaultOptions ''ReaderOptions)
320337
@@ -338,20 +355,3 @@ $(deriveJSON defaultOptions{ constructorTagModifier =
338355 } ''ObfuscationMethod)
339356
340357$(deriveJSON defaultOptions ''HTMLSlideVariant)
341−
342−-- Update documentation in doc/filters.md if this is changed.
343−$(deriveJSON defaultOptions{ constructorTagModifier =
344− camelCaseStrToHyphenated
345− } ''TrackChanges)
346−
347−$(deriveJSON defaultOptions{ constructorTagModifier =
348− camelCaseStrToHyphenated
349− } ''WrapOption)
350−
351−$(deriveJSON defaultOptions{ constructorTagModifier =
352− camelCaseStrToHyphenated . drop 8
353− } ''TopLevelDivision)
354−
355−$(deriveJSON defaultOptions{ constructorTagModifier =
356− camelCaseStrToHyphenated
357− } ''ReferenceLocation)
src/Text/Pandoc/PDF.hs+34−16
@@ -4,7 +4,7 @@
44{-# LANGUAGE ScopedTypeVariables #-}
55{- |
66 Module : Text.Pandoc.PDF
7− Copyright : Copyright (C) 2012-2020 John MacFarlane
7+ Copyright : Copyright (C) 2012-2021 John MacFarlane
88 License : GNU GPL, version 2 or above
99
1010 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -43,6 +43,7 @@ import Text.Pandoc.Definition
4343import Text.Pandoc.Error (PandocError (PandocPDFProgramNotFoundError))
4444import Text.Pandoc.MIME (getMimeType)
4545import Text.Pandoc.Options (HTMLMathMethod (..), WriterOptions (..))
46+import Text.Pandoc.Extensions (disableExtension, Extension(Ext_smart))
4647import Text.Pandoc.Process (pipeProcess)
4748import System.Process (readProcessWithExitCode)
4849import Text.Pandoc.Shared (inDirectory, stringify, tshow)
@@ -114,7 +115,10 @@ makePDF program pdfargs writer opts doc =
114115 runIOorExplode $ do
115116 putCommonState commonState
116117 doc' <- handleImages opts tmpdir doc
117− source <- writer opts doc'
118+ source <- writer opts{ writerExtensions = -- disable use of quote
119+ -- ligatures to avoid bad ligatures like ?`
120+ disableExtension Ext_smart
121+ (writerExtensions opts) } doc'
118122 res <- case baseProg of
119123 "context" -> context2pdf verbosity program pdfargs tmpdir source
120124 "tectonic" -> tectonic2pdf verbosity program pdfargs tmpdir source
@@ -198,7 +202,7 @@ convertImage opts tmpdir fname = do
198202 Just "image/svg+xml" -> E.catch (do
199203 (exit, _) <- pipeProcess Nothing "rsvg-convert"
200204 ["-f","pdf","-a","--dpi-x",dpi,"--dpi-y",dpi,
201− "-o",pdfOut,fname] BL.empty
205+ "-o",pdfOut,svgIn] BL.empty
202206 if exit == ExitSuccess
203207 then return $ Right pdfOut
204208 else return $ Left "conversion from SVG failed")
@@ -213,6 +217,7 @@ convertImage opts tmpdir fname = do
213217 where
214218 pngOut = replaceDirectory (replaceExtension fname ".png") tmpdir
215219 pdfOut = replaceDirectory (replaceExtension fname ".pdf") tmpdir
220+ svgIn = tmpdir </> fname
216221 mime = getMimeType fname
217222 doNothing = return (Right fname)
218223
@@ -266,7 +271,7 @@ missingCharacterWarnings verbosity log' = do
266271 | isAscii c = T.singleton c
267272 | otherwise = T.pack $ c : " (U+" ++ printf "%04X" (ord c) ++ ")"
268273 let addCodePoint = T.concatMap toCodePoint
269− let warnings = [ addCodePoint (T.pack $ utf8ToString (BC.drop 19 l))
274+ let warnings = [ addCodePoint (utf8ToText (BC.drop 19 l))
270275 | l <- ls
271276 , isMissingCharacterWarning l
272277 ]
@@ -310,7 +315,7 @@ runTectonic verbosity program args' tmpDir' source = do
310315 env <- liftIO getEnvironment
311316 when (verbosity >= INFO) $ liftIO $
312317 showVerboseInfo (Just tmpDir) program programArgs env
313− (utf8ToString sourceBL)
318+ (utf8ToText sourceBL)
314319 (exit, out) <- liftIO $ E.catch
315320 (pipeProcess (Just env) program programArgs sourceBL)
316321 (handlePDFProgramNotFound program)
@@ -381,7 +386,7 @@ runTeXProgram verbosity program args numRuns tmpDir' source = do
381386 (pipeProcess (Just env'') program programArgs BL.empty)
382387 (handlePDFProgramNotFound program)
383388 when (verbosity >= INFO) $ liftIO $ do
384− UTF8.hPutStrLn stderr $ "[makePDF] Run #" ++ show runNumber
389+ UTF8.hPutStrLn stderr $ "[makePDF] Run #" <> tshow runNumber
385390 BL.hPutStr stderr out
386391 UTF8.hPutStr stderr "\n"
387392 if runNumber < numRuns
@@ -401,7 +406,7 @@ generic2pdf :: Verbosity
401406generic2pdf verbosity program args source = do
402407 env' <- getEnvironment
403408 when (verbosity >= INFO) $
404− showVerboseInfo Nothing program args env' (T.unpack source)
409+ showVerboseInfo Nothing program args env' source
405410 (exit, out) <- E.catch
406411 (pipeProcess (Just env') program args
407412 (BL.fromStrict $ UTF8.fromText source))
@@ -490,19 +495,32 @@ showVerboseInfo :: Maybe FilePath
490495 -> String
491496 -> [String]
492497 -> [(String, String)]
493− -> String
498+ -> Text
494499 -> IO ()
495500showVerboseInfo mbTmpDir program programArgs env source = do
496501 case mbTmpDir of
497502 Just tmpDir -> do
498503 UTF8.hPutStrLn stderr "[makePDF] temp dir:"
499− UTF8.hPutStrLn stderr tmpDir
504+ UTF8.hPutStrLn stderr (T.pack tmpDir)
500505 Nothing -> return ()
501506 UTF8.hPutStrLn stderr "[makePDF] Command line:"
502− UTF8.hPutStrLn stderr $ program ++ " " ++ unwords (map show programArgs)
507+ UTF8.hPutStrLn stderr $
508+ T.pack program <> " " <> T.pack (unwords (map show programArgs))
503509 UTF8.hPutStr stderr "\n"
504− UTF8.hPutStrLn stderr "[makePDF] Environment:"
505− mapM_ (UTF8.hPutStrLn stderr . show) env
510+ UTF8.hPutStrLn stderr "[makePDF] Relevant environment variables:"
511+ -- we filter out irrelevant stuff to avoid leaking passwords and keys!
512+ let isRelevant ("PATH",_) = True
513+ isRelevant ("TMPDIR",_) = True
514+ isRelevant ("PWD",_) = True
515+ isRelevant ("LANG",_) = True
516+ isRelevant ("HOME",_) = True
517+ isRelevant ("LUA_PATH",_) = True
518+ isRelevant ("LUA_CPATH",_) = True
519+ isRelevant ("SHELL",_) = True
520+ isRelevant ("TEXINPUTS",_) = True
521+ isRelevant ("TEXMFOUTPUT",_) = True
522+ isRelevant _ = False
523+ mapM_ (UTF8.hPutStrLn stderr . tshow) (filter isRelevant env)
506524 UTF8.hPutStr stderr "\n"
507525 UTF8.hPutStrLn stderr "[makePDF] Source:"
508526 UTF8.hPutStrLn stderr source
@@ -513,8 +531,8 @@ handlePDFProgramNotFound program e
513531 E.throwIO $ PandocPDFProgramNotFoundError $ T.pack program
514532 | otherwise = E.throwIO e
515533
516−utf8ToString :: ByteString -> String
517−utf8ToString lbs =
534+utf8ToText :: ByteString -> Text
535+utf8ToText lbs =
518536 case decodeUtf8' lbs of
519− Left _ -> BC.unpack lbs -- if decoding fails, treat as latin1
520− Right t -> TL.unpack t
537+ Left _ -> T.pack $ BC.unpack lbs -- if decoding fails, treat as latin1
538+ Right t -> TL.toStrict t
src/Text/Pandoc/Parsing.hs+373−255
@@ -5,11 +5,10 @@
55{-# LANGUAGE GeneralizedNewtypeDeriving #-}
66{-# LANGUAGE IncoherentInstances #-}
77{-# LANGUAGE MultiParamTypeClasses #-}
8−{-# LANGUAGE ViewPatterns #-}
98{-# LANGUAGE OverloadedStrings #-}
109{- |
1110 Module : Text.Pandoc.Parsing
12− Copyright : Copyright (C) 2006-2020 John MacFarlane
11+ Copyright : Copyright (C) 2006-2021 John MacFarlane
1312 License : GNU GPL, version 2 or above
1413
1514 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -19,8 +18,7 @@
1918A utility library with parsers used in pandoc readers.
2019-}
2120
22−module Text.Pandoc.Parsing ( take1WhileP,
23− takeP,
21+module Text.Pandoc.Parsing ( module Text.Pandoc.Sources,
2422 countChar,
2523 textStr,
2624 anyLine,
@@ -105,14 +103,14 @@ module Text.Pandoc.Parsing ( take1WhileP,
105103 singleQuoteEnd,
106104 doubleQuoteStart,
107105 doubleQuoteEnd,
108− ellipses,
109106 apostrophe,
107+ doubleCloseQuote,
108+ ellipses,
110109 dash,
111110 nested,
112111 citeKey,
113112 Parser,
114113 ParserT,
115− F,
116114 Future(..),
117115 runF,
118116 askF,
@@ -123,7 +121,6 @@ module Text.Pandoc.Parsing ( take1WhileP,
123121 (<+?>),
124122 extractIdClass,
125123 insertIncludedFile,
126− insertIncludedFileF,
127124 -- * Re-exports from Text.Parsec
128125 Stream,
129126 runParser,
@@ -134,22 +131,10 @@ module Text.Pandoc.Parsing ( take1WhileP,
134131 getInput,
135132 setInput,
136133 unexpected,
137− char,
138− letter,
139− digit,
140− alphaNum,
141134 skipMany,
142135 skipMany1,
143− spaces,
144− space,
145− anyChar,
146− satisfy,
147− newline,
148− string,
149136 count,
150137 eof,
151− noneOf,
152− oneOf,
153138 lookAhead,
154139 notFollowedBy,
155140 many,
@@ -174,6 +159,8 @@ module Text.Pandoc.Parsing ( take1WhileP,
174159 SourcePos,
175160 getPosition,
176161 setPosition,
162+ sourceName,
163+ setSourceName,
177164 sourceColumn,
178165 sourceLine,
179166 setSourceColumn,
@@ -189,48 +176,141 @@ module Text.Pandoc.Parsing ( take1WhileP,
189176where
190177
191178import Control.Monad.Identity
179+ ( guard,
180+ join,
181+ unless,
182+ when,
183+ void,
184+ liftM2,
185+ liftM,
186+ Identity(..),
187+ MonadPlus(mzero) )
192188import Control.Monad.Reader
189+ ( asks, runReader, MonadReader(ask), Reader, ReaderT(ReaderT) )
193190import Data.Char (chr, isAlphaNum, isAscii, isAsciiUpper, isAsciiLower,
194− isPunctuation, isSpace, ord, toLower, toUpper)
195−import Data.Default
191+ isSpace, ord, toLower, toUpper)
192+import Data.Default ( Default(..) )
196193import Data.Functor (($>))
197194import Data.List (intercalate, transpose)
198195import qualified Data.Map as M
199−import Data.Maybe (mapMaybe, fromMaybe)
196+import Data.Maybe (fromMaybe)
200197import qualified Data.Set as Set
201−import Data.String
202198import Data.Text (Text)
203199import qualified Data.Text as T
204200import Text.HTML.TagSoup.Entity (lookupEntity)
205−import Text.Pandoc.Asciify (toAsciiChar)
201+import Text.Pandoc.Asciify (toAsciiText)
206202import Text.Pandoc.Builder (Blocks, HasMeta (..), Inlines, trimInlines)
207203import qualified Text.Pandoc.Builder as B
208204import Text.Pandoc.Class.PandocMonad (PandocMonad, readFileFromDirs, report)
209205import Text.Pandoc.Definition
206+ ( Target,
207+ nullMeta,
208+ nullAttr,
209+ Meta,
210+ ColWidth(ColWidthDefault, ColWidth),
211+ TableFoot(TableFoot),
212+ TableBody(TableBody),
213+ Attr,
214+ TableHead(TableHead),
215+ Row(..),
216+ Alignment(..),
217+ Inline(Str),
218+ ListNumberDelim(..),
219+ ListAttributes,
220+ ListNumberStyle(..) )
210221import Text.Pandoc.Logging
222+ ( LogMessage(CouldNotLoadIncludeFile, DuplicateIdentifier) )
211223import Text.Pandoc.Options
224+ ( extensionEnabled,
225+ Extension(Ext_old_dashes, Ext_tex_math_dollars,
226+ Ext_tex_math_single_backslash, Ext_tex_math_double_backslash,
227+ Ext_auto_identifiers, Ext_ascii_identifiers, Ext_smart),
228+ ReaderOptions(readerTabStop, readerColumns, readerExtensions) )
212229import Text.Pandoc.Readers.LaTeX.Types (Macro)
213230import Text.Pandoc.Shared
231+ ( uniqueIdent,
232+ tshow,
233+ mapLeft,
234+ compactify,
235+ trim,
236+ trimr,
237+ splitTextByIndices,
238+ safeRead,
239+ trimMath,
240+ schemes,
241+ escapeURI )
242+import Text.Pandoc.Sources
214243import qualified Text.Pandoc.UTF8 as UTF8 (putStrLn)
215244import Text.Pandoc.XML (fromEntities)
216−import Text.Parsec hiding (token)
217−import Text.Parsec.Pos (initialPos, newPos, updatePosString)
218−
219−import Control.Monad.Except
245+import Text.Parsec
246+ ( between,
247+ setSourceName,
248+ Parsec,
249+ Column,
250+ Line,
251+ incSourceLine,
252+ incSourceColumn,
253+ setSourceLine,
254+ setSourceColumn,
255+ sourceLine,
256+ sourceColumn,
257+ sourceName,
258+ setSourceName,
259+ setPosition,
260+ getPosition,
261+ updateState,
262+ setState,
263+ getState,
264+ optionMaybe,
265+ optional,
266+ option,
267+ endBy1,
268+ endBy,
269+ sepEndBy1,
270+ sepEndBy,
271+ sepBy1,
272+ sepBy,
273+ try,
274+ choice,
275+ (<?>),
276+ (<|>),
277+ manyTill,
278+ many1,
279+ many,
280+ notFollowedBy,
281+ lookAhead,
282+ eof,
283+ count,
284+ skipMany1,
285+ skipMany,
286+ unexpected,
287+ setInput,
288+ getInput,
289+ anyToken,
290+ tokenPrim,
291+ parse,
292+ runParserT,
293+ runParser,
294+ ParseError,
295+ ParsecT,
296+ SourcePos,
297+ Stream(..) )
298+import Text.Parsec.Pos (initialPos, newPos)
299+import Control.Monad.Except ( MonadError(throwError) )
220300import Text.Pandoc.Error
301+ ( PandocError(PandocParseError, PandocParsecError) )
221302
222303type Parser t s = Parsec t s
223304
224305type ParserT = ParsecT
225306
307+
226308-- | Reader monad wrapping the parser state. This is used to possibly delay
227309-- evaluation until all relevant information has been parsed and made available
228310-- in the parser state.
229311newtype Future s a = Future { runDelayed :: Reader s a }
230312 deriving (Monad, Applicative, Functor)
231313
232−type F = Future ParserState
233−
234314runF :: Future s a -> s -> a
235315runF = runReader . runDelayed
236316
@@ -253,70 +333,48 @@ instance (Semigroup a, Monoid a) => Monoid (Future s a) where
253333 mappend = (<>)
254334
255335-- | Like @count@, but packs its result
256−countChar :: (Stream s m Char, Monad m)
336+countChar :: (Stream s m Char, UpdateSourcePos s Char, Monad m)
257337 => Int
258338 -> ParsecT s st m Char
259339 -> ParsecT s st m Text
260340countChar n = fmap T.pack . count n
261341
262342-- | Like @string@, but uses @Text@.
263−textStr :: Stream s m Char => Text -> ParsecT s u m Text
343+textStr :: (Stream s m Char, UpdateSourcePos s Char)
344+ => Text -> ParsecT s u m Text
264345textStr t = string (T.unpack t) $> t
265346
266−-- | Parse characters while a predicate is true.
267−take1WhileP :: Monad m
268− => (Char -> Bool)
269− -> ParserT Text st m Text
270−take1WhileP f = do
271− -- needed to persuade parsec that this won't match an empty string:
272− c <- satisfy f
273− inp <- getInput
274− pos <- getPosition
275− let (t, rest) = T.span f inp
276− setInput rest
277− setPosition $
278− if f '\t' || f '\n'
279− then updatePosString pos $ T.unpack t
280− else incSourceColumn pos (T.length t)
281− return $ T.singleton c <> t
282−
283−-- Parse n characters of input (or the rest of the input if
284−-- there aren't n characters).
285−takeP :: Monad m => Int -> ParserT Text st m Text
286−takeP n = do
287− guard (n > 0)
288− -- faster than 'count n anyChar'
289− inp <- getInput
290− pos <- getPosition
291− let (xs, rest) = T.splitAt n inp
292− -- needed to persuade parsec that this won't match an empty string:
293− anyChar
294− setInput rest
295− setPosition $ updatePosString pos $ T.unpack xs
296− return xs
297−
298−-- | Parse any line of text
299−anyLine :: Monad m => ParserT Text st m Text
347+
348+-- | Parse any line of text, returning the contents without the
349+-- final newline.
350+anyLine :: Monad m => ParserT Sources st m Text
300351anyLine = do
301352 -- This is much faster than:
302353 -- manyTill anyChar newline
303354 inp <- getInput
304− pos <- getPosition
305− case T.break (=='\n') inp of
306− (this, T.uncons -> Just ('\n', rest)) -> do
307− -- needed to persuade parsec that this won't match an empty string:
308− anyChar
309− setInput rest
310− setPosition $ incSourceLine (setSourceColumn pos 1) 1
311− return this
312− _ -> mzero
355+ case inp of
356+ Sources [] -> mzero
357+ Sources ((fp,t):inps) ->
358+ -- we assume that lines don't span different input files
359+ case T.break (=='\n') t of
360+ (this, rest)
361+ | T.null rest
362+ , not (null inps) ->
363+ -- line may span different input files, so do it
364+ -- character by character
365+ T.pack <$> manyTill anyChar newline
366+ | otherwise -> do -- either end of inputs or newline in rest
367+ setInput $ Sources ((fp, rest):inps)
368+ char '\n' -- needed so parsec knows we won't match empty string
369+ -- and so source pos is updated
370+ return this
313371
314372-- | Parse any line, include the final newline in the output
315−anyLineNewline :: Monad m => ParserT Text st m Text
373+anyLineNewline :: Monad m => ParserT Sources st m Text
316374anyLineNewline = (<> "\n") <$> anyLine
317375
318376-- | Parse indent by specified number of spaces (or equiv. tabs)
319−indentWith :: Stream s m Char
377+indentWith :: (Stream s m Char, UpdateSourcePos s Char)
320378 => HasReaderOptions st
321379 => Int -> ParserT s st m Text
322380indentWith num = do
@@ -401,11 +459,13 @@ notFollowedBy' p = try $ join $ do a <- try p
401459 return (return ())
402460-- (This version due to Andrew Pimlott on the Haskell mailing list.)
403461
404−oneOfStrings' :: Stream s m Char => (Char -> Char -> Bool) -> [Text] -> ParserT s st m Text
462+oneOfStrings' :: (Stream s m Char, UpdateSourcePos s Char)
463+ => (Char -> Char -> Bool) -> [Text] -> ParserT s st m Text
405464oneOfStrings' f = fmap T.pack . oneOfStrings'' f . fmap T.unpack
406465
407466-- TODO: This should be re-implemented in a Text-aware way
408−oneOfStrings'' :: Stream s m Char => (Char -> Char -> Bool) -> [String] -> ParserT s st m String
467+oneOfStrings'' :: (Stream s m Char, UpdateSourcePos s Char)
468+ => (Char -> Char -> Bool) -> [String] -> ParserT s st m String
409469oneOfStrings'' _ [] = Prelude.fail "no strings"
410470oneOfStrings'' matches strs = try $ do
411471 c <- anyChar
@@ -420,14 +480,16 @@ oneOfStrings'' matches strs = try $ do
420480-- | Parses one of a list of strings. If the list contains
421481-- two strings one of which is a prefix of the other, the longer
422482-- string will be matched if possible.
423−oneOfStrings :: Stream s m Char => [Text] -> ParserT s st m Text
483+oneOfStrings :: (Stream s m Char, UpdateSourcePos s Char)
484+ => [Text] -> ParserT s st m Text
424485oneOfStrings = oneOfStrings' (==)
425486
426487-- | Parses one of a list of strings (tried in order), case insensitive.
427488
428489-- TODO: This will not be accurate with general Unicode (neither
429490-- Text.toLower nor Text.toCaseFold can be implemented with a map)
430−oneOfStringsCI :: Stream s m Char => [Text] -> ParserT s st m Text
491+oneOfStringsCI :: (Stream s m Char, UpdateSourcePos s Char)
492+ => [Text] -> ParserT s st m Text
431493oneOfStringsCI = oneOfStrings' ciMatch
432494 where ciMatch x y = toLower' x == toLower' y
433495 -- this optimizes toLower by checking common ASCII case
@@ -438,11 +500,13 @@ oneOfStringsCI = oneOfStrings' ciMatch
438500 | otherwise = toLower c
439501
440502-- | Parses a space or tab.
441−spaceChar :: Stream s m Char => ParserT s st m Char
503+spaceChar :: (Stream s m Char, UpdateSourcePos s Char)
504+ => ParserT s st m Char
442505spaceChar = satisfy $ \c -> c == ' ' || c == '\t'
443506
444507-- | Parses a nonspace, nonnewline character.
445−nonspaceChar :: Stream s m Char => ParserT s st m Char
508+nonspaceChar :: (Stream s m Char, UpdateSourcePos s Char)
509+ => ParserT s st m Char
446510nonspaceChar = satisfy (not . isSpaceChar)
447511
448512isSpaceChar :: Char -> Bool
@@ -453,21 +517,24 @@ isSpaceChar '\r' = True
453517isSpaceChar _ = False
454518
455519-- | Skips zero or more spaces or tabs.
456−skipSpaces :: Stream s m Char => ParserT s st m ()
520+skipSpaces :: (Stream s m Char, UpdateSourcePos s Char)
521+ => ParserT s st m ()
457522skipSpaces = skipMany spaceChar
458523
459524-- | Skips zero or more spaces or tabs, then reads a newline.
460−blankline :: Stream s m Char => ParserT s st m Char
525+blankline :: (Stream s m Char, UpdateSourcePos s Char)
526+ => ParserT s st m Char
461527blankline = try $ skipSpaces >> newline
462528
463529-- | Parses one or more blank lines and returns a string of newlines.
464−blanklines :: Stream s m Char => ParserT s st m Text
530+blanklines :: (Stream s m Char, UpdateSourcePos s Char)
531+ => ParserT s st m Text
465532blanklines = T.pack <$> many1 blankline
466533
467534-- | Gobble n spaces; if tabs are encountered, expand them
468535-- and gobble some or all of their spaces, leaving the rest.
469536gobbleSpaces :: (HasReaderOptions st, Monad m)
470− => Int -> ParserT Text st m ()
537+ => Int -> ParserT Sources st m ()
471538gobbleSpaces 0 = return ()
472539gobbleSpaces n
473540 | n < 0 = error "gobbleSpaces called with negative number"
@@ -475,18 +542,26 @@ gobbleSpaces n
475542 char ' ' <|> eatOneSpaceOfTab
476543 gobbleSpaces (n - 1)
477544
478−eatOneSpaceOfTab :: (HasReaderOptions st, Monad m) => ParserT Text st m Char
545+eatOneSpaceOfTab :: (HasReaderOptions st, Monad m) => ParserT Sources st m Char
479546eatOneSpaceOfTab = do
480− char '\t'
547+ lookAhead (char '\t')
548+ pos <- getPosition
481549 tabstop <- getOption readerTabStop
550+ -- replace the tab on the input stream with spaces
551+ let numSpaces = tabstop - ((sourceColumn pos - 1) `mod` tabstop)
482552 inp <- getInput
483− setInput $ T.replicate (tabstop - 1) " " <> inp
484− return ' '
553+ setInput $
554+ case inp of
555+ Sources [] -> error "eatOneSpaceOfTab - empty Sources list"
556+ Sources ((fp,t):rest) ->
557+ -- drop the tab and add spaces
558+ Sources ((fp, T.replicate numSpaces " " <> T.drop 1 t):rest)
559+ char ' '
485560
486561-- | Gobble up to n spaces; if tabs are encountered, expand them
487562-- and gobble some or all of their spaces, leaving the rest.
488563gobbleAtMostSpaces :: (HasReaderOptions st, Monad m)
489− => Int -> ParserT Text st m Int
564+ => Int -> ParserT Sources st m Int
490565gobbleAtMostSpaces 0 = return 0
491566gobbleAtMostSpaces n
492567 | n < 0 = error "gobbleAtMostSpaces called with negative number"
@@ -495,7 +570,8 @@ gobbleAtMostSpaces n
495570 (+ 1) <$> gobbleAtMostSpaces (n - 1)
496571
497572-- | Parses material enclosed between start and end parsers.
498−enclosed :: (Show end, Stream s m Char) => ParserT s st m t -- ^ start parser
573+enclosed :: (Show end, Stream s m Char, UpdateSourcePos s Char)
574+ => ParserT s st m t -- ^ start parser
499575 -> ParserT s st m end -- ^ end parser
500576 -> ParserT s st m a -- ^ content parser (to be used repeatedly)
501577 -> ParserT s st m [a]
@@ -503,39 +579,41 @@ enclosed start end parser = try $
503579 start >> notFollowedBy space >> many1Till parser end
504580
505581-- | Parse string, case insensitive.
506−stringAnyCase :: Stream s m Char => Text -> ParserT s st m Text
582+stringAnyCase :: (Stream s m Char, UpdateSourcePos s Char)
583+ => Text -> ParserT s st m Text
507584stringAnyCase = fmap T.pack . stringAnyCase' . T.unpack
508585
509−stringAnyCase' :: Stream s m Char => String -> ParserT s st m String
586+stringAnyCase' :: (Stream s m Char, UpdateSourcePos s Char)
587+ => String -> ParserT s st m String
510588stringAnyCase' [] = string ""
511589stringAnyCase' (x:xs) = do
512590 firstChar <- char (toUpper x) <|> char (toLower x)
513591 rest <- stringAnyCase' xs
514592 return (firstChar:rest)
515593
594+-- TODO rewrite by just adding to Sources stream?
516595-- | Parse contents of 'str' using 'parser' and return result.
517−parseFromString :: (Stream s m Char, IsString s)
518− => ParserT s st m r
596+parseFromString :: Monad m
597+ => ParserT Sources st m r
519598 -> Text
520− -> ParserT s st m r
599+ -> ParserT Sources st m r
521600parseFromString parser str = do
522601 oldPos <- getPosition
523602 setPosition $ initialPos "chunk"
524603 oldInput <- getInput
525− setInput $ fromString $ T.unpack str
604+ setInput $ toSources str
526605 result <- parser
527606 spaces
528− eof
529607 setInput oldInput
530608 setPosition oldPos
531609 return result
532610
533611-- | Like 'parseFromString' but specialized for 'ParserState'.
534612-- This resets 'stateLastStrPos', which is almost always what we want.
535−parseFromString' :: (Stream s m Char, IsString s, HasLastStrPosition u)
536− => ParserT s u m a
613+parseFromString' :: (Monad m, HasLastStrPosition u)
614+ => ParserT Sources u m a
537615 -> Text
538− -> ParserT s u m a
616+ -> ParserT Sources u m a
539617parseFromString' parser str = do
540618 oldLastStrPos <- getLastStrPos <$> getState
541619 updateState $ setLastStrPos Nothing
@@ -544,7 +622,7 @@ parseFromString' parser str = do
544622 return res
545623
546624-- | Parse raw line block up to and including blank lines.
547−lineClump :: Monad m => ParserT Text st m Text
625+lineClump :: Monad m => ParserT Sources st m Text
548626lineClump = blanklines
549627 <|> (T.unlines <$> many1 (notFollowedBy blankline >> anyLine))
550628
@@ -553,7 +631,7 @@ lineClump = blanklines
553631-- pairs of open and close, which must be different. For example,
554632-- @charsInBalanced '(' ')' anyChar@ will parse "(hello (there))"
555633-- and return "hello (there)".
556−charsInBalanced :: Stream s m Char => Char -> Char -> ParserT s st m Char
634+charsInBalanced :: (Stream s m Char, UpdateSourcePos s Char) => Char -> Char -> ParserT s st m Char
557635 -> ParserT s st m Text
558636charsInBalanced open close parser = try $ do
559637 char open
@@ -572,7 +650,7 @@ charsInBalanced open close parser = try $ do
572650-- Auxiliary functions for romanNumeral:
573651
574652-- | Parses a roman numeral (uppercase or lowercase), returns number.
575−romanNumeral :: Stream s m Char => Bool -- ^ Uppercase if true
653+romanNumeral :: (Stream s m Char, UpdateSourcePos s Char) => Bool -- ^ Uppercase if true
576654 -> ParserT s st m Int
577655romanNumeral upperCase = do
578656 let rchar uc = char $ if upperCase then uc else toLower uc
@@ -608,20 +686,19 @@ romanNumeral upperCase = do
608686
609687-- | Parses an email address; returns original and corresponding
610688-- escaped mailto: URI.
611−emailAddress :: Stream s m Char => ParserT s st m (Text, Text)
689+emailAddress :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m (Text, Text)
612690emailAddress = try $ toResult <$> mailbox <*> (char '@' *> domain)
613691 where toResult mbox dom = let full = fromEntities $ T.pack $ mbox ++ '@':dom
614692 in (full, escapeURI $ "mailto:" <> full)
615693 mailbox = intercalate "." <$> (emailWord `sepBy1'` dot)
616694 domain = intercalate "." <$> (subdomain `sepBy1'` dot)
617695 dot = char '.'
618− subdomain = many1 $ alphaNum <|> innerPunct
696+ subdomain = many1 $ alphaNum <|> innerPunct (=='-')
619697 -- this excludes some valid email addresses, since an
620698 -- email could contain e.g. '__', but gives better results
621699 -- for our purposes, when combined with markdown parsing:
622− innerPunct = try (satisfy (\c -> isEmailPunct c || c == '@')
623− <* notFollowedBy space
624− <* notFollowedBy (satisfy isPunctuation))
700+ innerPunct f = try (satisfy f
701+ <* notFollowedBy (satisfy (not . isAlphaNum)))
625702 -- technically an email address could begin with a symbol,
626703 -- but allowing this creates too many problems.
627704 -- See e.g. https://github.com/jgm/pandoc/issues/2940
@@ -632,11 +709,11 @@ emailAddress = try $ toResult <$> mailbox <*> (char '@' *> domain)
632709 isEmailPunct c = T.any (== c) "!\"#$%&'*+-/=?^_{|}~;"
633710
634711
635−uriScheme :: Stream s m Char => ParserT s st m Text
712+uriScheme :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m Text
636713uriScheme = oneOfStringsCI (Set.toList schemes)
637714
638715-- | Parses a URI. Returns pair of original and URI-escaped version.
639−uri :: Stream s m Char => ParserT s st m (Text, Text)
716+uri :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m (Text, Text)
640717uri = try $ do
641718 scheme <- uriScheme
642719 char ':'
@@ -679,7 +756,7 @@ uri = try $ do
679756 uriChunkBetween l r = try $ do chunk <- between (char l) (char r) uriChunk
680757 return (T.pack $ [l] ++ chunk ++ [r])
681758
682−mathInlineWith :: Stream s m Char => Text -> Text -> ParserT s st m Text
759+mathInlineWith :: (Stream s m Char, UpdateSourcePos s Char) => Text -> Text -> ParserT s st m Text
683760mathInlineWith op cl = try $ do
684761 textStr op
685762 when (op == "$") $ notFollowedBy space
@@ -700,10 +777,10 @@ mathInlineWith op cl = try $ do
700777 notFollowedBy digit -- to prevent capture of $5
701778 return $ trimMath $ T.concat words'
702779 where
703− inBalancedBraces :: Stream s m Char => Int -> Text -> ParserT s st m Text
780+ inBalancedBraces :: (Stream s m Char, UpdateSourcePos s Char) => Int -> Text -> ParserT s st m Text
704781 inBalancedBraces n = fmap T.pack . inBalancedBraces' n . T.unpack
705782
706− inBalancedBraces' :: Stream s m Char => Int -> String -> ParserT s st m String
783+ inBalancedBraces' :: (Stream s m Char, UpdateSourcePos s Char) => Int -> String -> ParserT s st m String
707784 inBalancedBraces' 0 "" = do
708785 c <- anyChar
709786 if c == '{'
@@ -720,13 +797,13 @@ mathInlineWith op cl = try $ do
720797 '{' -> inBalancedBraces' (numOpen + 1) (c:xs)
721798 _ -> inBalancedBraces' numOpen (c:xs)
722799
723−mathDisplayWith :: Stream s m Char => Text -> Text -> ParserT s st m Text
800+mathDisplayWith :: (Stream s m Char, UpdateSourcePos s Char) => Text -> Text -> ParserT s st m Text
724801mathDisplayWith op cl = try $ fmap T.pack $ do
725802 textStr op
726803 many1Till (satisfy (/= '\n') <|> (newline <* notFollowedBy' blankline))
727804 (try $ textStr cl)
728805
729−mathDisplay :: (HasReaderOptions st, Stream s m Char)
806+mathDisplay :: (HasReaderOptions st, Stream s m Char, UpdateSourcePos s Char)
730807 => ParserT s st m Text
731808mathDisplay =
732809 (guardEnabled Ext_tex_math_dollars >> mathDisplayWith "$$" "$$")
@@ -735,7 +812,7 @@ mathDisplay =
735812 <|> (guardEnabled Ext_tex_math_double_backslash >>
736813 mathDisplayWith "\\\\[" "\\\\]")
737814
738−mathInline :: (HasReaderOptions st , Stream s m Char)
815+mathInline :: (HasReaderOptions st, Stream s m Char, UpdateSourcePos s Char)
739816 => ParserT s st m Text
740817mathInline =
741818 (guardEnabled Ext_tex_math_dollars >> mathInlineWith "$" "$")
@@ -748,7 +825,7 @@ mathInline =
748825-- displacement (the difference between the source column at the end
749826-- and the source column at the beginning). Vertical displacement
750827-- (source row) is ignored.
751−withHorizDisplacement :: Stream s m Char
828+withHorizDisplacement :: (Stream s m Char, UpdateSourcePos s Char)
752829 => ParserT s st m a -- ^ Parser to apply
753830 -> ParserT s st m (a, Int) -- ^ (result, displacement)
754831withHorizDisplacement parser = do
@@ -760,30 +837,37 @@ withHorizDisplacement parser = do
760837-- | Applies a parser and returns the raw string that was parsed,
761838-- along with the value produced by the parser.
762839withRaw :: Monad m
763− => ParsecT Text st m a
764− -> ParsecT Text st m (a, Text)
840+ => ParsecT Sources st m a
841+ -> ParsecT Sources st m (a, Text)
765842withRaw parser = do
766− pos1 <- getPosition
767− inp <- getInput
843+ inps1 <- getInput
768844 result <- parser
769− pos2 <- getPosition
770− let (l1,c1) = (sourceLine pos1, sourceColumn pos1)
771− let (l2,c2) = (sourceLine pos2, sourceColumn pos2)
772− let inplines = take ((l2 - l1) + 1) $ T.lines inp
773− let raw = case inplines of
774− [] -> ""
775− [l] -> T.take (c2 - c1) l
776− ls -> T.unlines (init ls) <> T.take (c2 - 1) (last ls)
777− return (result, raw)
845+ inps2 <- getInput
846+ -- 'raw' is the difference between inps1 and inps2
847+ return (result, sourcesDifference inps1 inps2)
848+
849+sourcesDifference :: Sources -> Sources -> Text
850+sourcesDifference (Sources is1) (Sources is2) = go is1 is2
851+ where
852+ go inps1 inps2 =
853+ case (inps1, inps2) of
854+ ([], _) -> mempty
855+ (_, []) -> mconcat $ map snd inps1
856+ ((p1,t1):rest1, (p2, t2):rest2)
857+ | p1 == p2
858+ , t1 == t2 -> go rest1 rest2
859+ | p1 == p2
860+ , t1 /= t2 -> fromMaybe mempty $ T.stripSuffix t2 t1
861+ | otherwise -> t1 <> go rest1 inps2
778862
779863-- | Parses backslash, then applies character parser.
780−escaped :: Stream s m Char
864+escaped :: (Stream s m Char, UpdateSourcePos s Char)
781865 => ParserT s st m Char -- ^ Parser for character to escape
782866 -> ParserT s st m Char
783867escaped parser = try $ char '\\' >> parser
784868
785869-- | Parse character entity.
786−characterReference :: Stream s m Char => ParserT s st m Char
870+characterReference :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m Char
787871characterReference = try $ do
788872 char '&'
789873 ent <- many1Till nonspaceChar (char ';')
@@ -796,19 +880,19 @@ characterReference = try $ do
796880 _ -> Prelude.fail "entity not found"
797881
798882-- | Parses an uppercase roman numeral and returns (UpperRoman, number).
799−upperRoman :: Stream s m Char => ParserT s st m (ListNumberStyle, Int)
883+upperRoman :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m (ListNumberStyle, Int)
800884upperRoman = do
801885 num <- romanNumeral True
802886 return (UpperRoman, num)
803887
804888-- | Parses a lowercase roman numeral and returns (LowerRoman, number).
805−lowerRoman :: Stream s m Char => ParserT s st m (ListNumberStyle, Int)
889+lowerRoman :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m (ListNumberStyle, Int)
806890lowerRoman = do
807891 num <- romanNumeral False
808892 return (LowerRoman, num)
809893
810894-- | Parses a decimal numeral and returns (Decimal, number).
811−decimal :: Stream s m Char => ParserT s st m (ListNumberStyle, Int)
895+decimal :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m (ListNumberStyle, Int)
812896decimal = do
813897 num <- many1 digit
814898 return (Decimal, fromMaybe 1 $ safeRead $ T.pack num)
@@ -817,7 +901,7 @@ decimal = do
817901-- returns (DefaultStyle, [next example number]). The next
818902-- example number is incremented in parser state, and the label
819903-- (if present) is added to the label table.
820−exampleNum :: Stream s m Char
904+exampleNum :: (Stream s m Char, UpdateSourcePos s Char)
821905 => ParserT s ParserState m (ListNumberStyle, Int)
822906exampleNum = do
823907 char '@'
@@ -836,37 +920,37 @@ exampleNum = do
836920 return (Example, num)
837921
838922-- | Parses a '#' returns (DefaultStyle, 1).
839−defaultNum :: Stream s m Char => ParserT s st m (ListNumberStyle, Int)
923+defaultNum :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m (ListNumberStyle, Int)
840924defaultNum = do
841925 char '#'
842926 return (DefaultStyle, 1)
843927
844928-- | Parses a lowercase letter and returns (LowerAlpha, number).
845−lowerAlpha :: Stream s m Char => ParserT s st m (ListNumberStyle, Int)
929+lowerAlpha :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m (ListNumberStyle, Int)
846930lowerAlpha = do
847931 ch <- satisfy isAsciiLower
848932 return (LowerAlpha, ord ch - ord 'a' + 1)
849933
850934-- | Parses an uppercase letter and returns (UpperAlpha, number).
851−upperAlpha :: Stream s m Char => ParserT s st m (ListNumberStyle, Int)
935+upperAlpha :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m (ListNumberStyle, Int)
852936upperAlpha = do
853937 ch <- satisfy isAsciiUpper
854938 return (UpperAlpha, ord ch - ord 'A' + 1)
855939
856940-- | Parses a roman numeral i or I
857−romanOne :: Stream s m Char => ParserT s st m (ListNumberStyle, Int)
941+romanOne :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m (ListNumberStyle, Int)
858942romanOne = (char 'i' >> return (LowerRoman, 1)) <|>
859943 (char 'I' >> return (UpperRoman, 1))
860944
861945-- | Parses an ordered list marker and returns list attributes.
862−anyOrderedListMarker :: Stream s m Char => ParserT s ParserState m ListAttributes
946+anyOrderedListMarker :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s ParserState m ListAttributes
863947anyOrderedListMarker = choice
864948 [delimParser numParser | delimParser <- [inPeriod, inOneParen, inTwoParens],
865949 numParser <- [decimal, exampleNum, defaultNum, romanOne,
866950 lowerAlpha, lowerRoman, upperAlpha, upperRoman]]
867951
868952-- | Parses a list number (num) followed by a period, returns list attributes.
869−inPeriod :: Stream s m Char
953+inPeriod :: (Stream s m Char, UpdateSourcePos s Char)
870954 => ParserT s st m (ListNumberStyle, Int)
871955 -> ParserT s st m ListAttributes
872956inPeriod num = try $ do
@@ -878,7 +962,7 @@ inPeriod num = try $ do
878962 return (start, style, delim)
879963
880964-- | Parses a list number (num) followed by a paren, returns list attributes.
881−inOneParen :: Stream s m Char
965+inOneParen :: (Stream s m Char, UpdateSourcePos s Char)
882966 => ParserT s st m (ListNumberStyle, Int)
883967 -> ParserT s st m ListAttributes
884968inOneParen num = try $ do
@@ -887,7 +971,7 @@ inOneParen num = try $ do
887971 return (start, style, OneParen)
888972
889973-- | Parses a list number (num) enclosed in parens, returns list attributes.
890−inTwoParens :: Stream s m Char
974+inTwoParens :: (Stream s m Char, UpdateSourcePos s Char)
891975 => ParserT s st m (ListNumberStyle, Int)
892976 -> ParserT s st m ListAttributes
893977inTwoParens num = try $ do
@@ -898,7 +982,7 @@ inTwoParens num = try $ do
898982
899983-- | Parses an ordered list marker with a given style and delimiter,
900984-- returns number.
901−orderedListMarker :: Stream s m Char
985+orderedListMarker :: (Stream s m Char, UpdateSourcePos s Char)
902986 => ListNumberStyle
903987 -> ListNumberDelim
904988 -> ParserT s ParserState m Int
@@ -921,10 +1005,10 @@ orderedListMarker style delim = do
9211005 return start
9221006
9231007-- | Parses a character reference and returns a Str element.
924−charRef :: Stream s m Char => ParserT s st m Inline
1008+charRef :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m Inline
9251009charRef = Str . T.singleton <$> characterReference
9261010
927−lineBlockLine :: Monad m => ParserT Text st m Text
1011+lineBlockLine :: Monad m => ParserT Sources st m Text
9281012lineBlockLine = try $ do
9291013 char '|'
9301014 char ' '
@@ -934,11 +1018,11 @@ lineBlockLine = try $ do
9341018 continuations <- many (try $ char ' ' >> anyLine)
9351019 return $ white <> T.unwords (line : continuations)
9361020
937−blankLineBlockLine :: Stream s m Char => ParserT s st m Char
1021+blankLineBlockLine :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m Char
9381022blankLineBlockLine = try (char '|' >> blankline)
9391023
9401024-- | Parses an RST-style line block and returns a list of strings.
941−lineBlockLines :: Monad m => ParserT Text st m [Text]
1025+lineBlockLines :: Monad m => ParserT Sources st m [Text]
9421026lineBlockLines = try $ do
9431027 lines' <- many1 (lineBlockLine <|> (T.singleton <$> blankLineBlockLine))
9441028 skipMany blankline
@@ -946,7 +1030,8 @@ lineBlockLines = try $ do
9461030
9471031-- | Parse a table using 'headerParser', 'rowParser',
9481032-- 'lineParser', and 'footerParser'.
949−tableWith :: (Stream s m Char, HasReaderOptions st, Monad mf)
1033+tableWith :: (Stream s m Char, UpdateSourcePos s Char,
1034+ HasReaderOptions st, Monad mf)
9501035 => ParserT s st m (mf [Blocks], [Alignment], [Int])
9511036 -> ([Int] -> ParserT s st m (mf [Blocks]))
9521037 -> ParserT s st m sep
@@ -966,7 +1051,8 @@ tableWith headerParser rowParser lineParser footerParser = try $ do
9661051
9671052type TableComponents mf = ([Alignment], [Double], mf [Row], mf [Row])
9681053
969−tableWith' :: (Stream s m Char, HasReaderOptions st, Monad mf)
1054+tableWith' :: (Stream s m Char, UpdateSourcePos s Char,
1055+ HasReaderOptions st, Monad mf)
9701056 => ParserT s st m (mf [Blocks], [Alignment], [Int])
9711057 -> ([Int] -> ParserT s st m (mf [Blocks]))
9721058 -> ParserT s st m sep
@@ -1015,20 +1101,19 @@ widthsFromIndices numColumns' indices =
10151101-- (which may be grid), then the rows,
10161102-- which may be grid, separated by blank lines, and
10171103-- ending with a footer (dashed line followed by blank line).
1018−gridTableWith :: (Stream s m Char, HasReaderOptions st, HasLastStrPosition st,
1019− Monad mf, IsString s)
1020− => ParserT s st m (mf Blocks) -- ^ Block list parser
1104+gridTableWith :: (Monad m, HasReaderOptions st, HasLastStrPosition st, Monad mf)
1105+ => ParserT Sources st m (mf Blocks) -- ^ Block list parser
10211106 -> Bool -- ^ Headerless table
1022− -> ParserT s st m (mf Blocks)
1107+ -> ParserT Sources st m (mf Blocks)
10231108gridTableWith blocks headless =
10241109 tableWith (gridTableHeader headless blocks) (gridTableRow blocks)
10251110 (gridTableSep '-') gridTableFooter
10261111
1027−gridTableWith' :: (Stream s m Char, HasReaderOptions st, HasLastStrPosition st,
1028− Monad mf, IsString s)
1029− => ParserT s st m (mf Blocks) -- ^ Block list parser
1112+gridTableWith' :: (Monad m, HasReaderOptions st, HasLastStrPosition st,
1113+ Monad mf)
1114+ => ParserT Sources st m (mf Blocks) -- ^ Block list parser
10301115 -> Bool -- ^ Headerless table
1031− -> ParserT s st m (TableComponents mf)
1116+ -> ParserT Sources st m (TableComponents mf)
10321117gridTableWith' blocks headless =
10331118 tableWith' (gridTableHeader headless blocks) (gridTableRow blocks)
10341119 (gridTableSep '-') gridTableFooter
@@ -1037,7 +1122,7 @@ gridTableSplitLine :: [Int] -> Text -> [Text]
10371122gridTableSplitLine indices line = map removeFinalBar $ tail $
10381123 splitTextByIndices (init indices) $ trimr line
10391124
1040−gridPart :: Stream s m Char => Char -> ParserT s st m ((Int, Int), Alignment)
1125+gridPart :: Monad m => Char -> ParserT Sources st m ((Int, Int), Alignment)
10411126gridPart ch = do
10421127 leftColon <- option False (True <$ char ':')
10431128 dashes <- many1 (char ch)
@@ -1052,7 +1137,7 @@ gridPart ch = do
10521137 (False, False) -> AlignDefault
10531138 return ((lengthDashes, lengthDashes + 1), alignment)
10541139
1055−gridDashedLines :: Stream s m Char => Char -> ParserT s st m [((Int, Int), Alignment)]
1140+gridDashedLines :: Monad m => Char -> ParserT Sources st m [((Int, Int), Alignment)]
10561141gridDashedLines ch = try $ char '+' >> many1 (gridPart ch) <* blankline
10571142
10581143removeFinalBar :: Text -> Text
@@ -1061,47 +1146,47 @@ removeFinalBar = T.dropWhileEnd go . T.dropWhileEnd (=='|')
10611146 go c = T.any (== c) " \t"
10621147
10631148-- | Separator between rows of grid table.
1064−gridTableSep :: Stream s m Char => Char -> ParserT s st m Char
1149+gridTableSep :: Monad m => Char -> ParserT Sources st m Char
10651150gridTableSep ch = try $ gridDashedLines ch >> return '\n'
10661151
10671152-- | Parse header for a grid table.
1068−gridTableHeader :: (Stream s m Char, Monad mf, IsString s, HasLastStrPosition st)
1153+gridTableHeader :: (Monad m, Monad mf, HasLastStrPosition st)
10691154 => Bool -- ^ Headerless table
1070− -> ParserT s st m (mf Blocks)
1071− -> ParserT s st m (mf [Blocks], [Alignment], [Int])
1072−gridTableHeader headless blocks = try $ do
1155+ -> ParserT Sources st m (mf Blocks)
1156+ -> ParserT Sources st m (mf [Blocks], [Alignment], [Int])
1157+gridTableHeader True _ = do
1158+ optional blanklines
1159+ dashes <- gridDashedLines '-'
1160+ let aligns = map snd dashes
1161+ let lines' = map (snd . fst) dashes
1162+ let indices = scanl (+) 0 lines'
1163+ return (return [], aligns, indices)
1164+gridTableHeader False blocks = try $ do
10731165 optional blanklines
10741166 dashes <- gridDashedLines '-'
1075− rawContent <- if headless
1076− then return $ repeat ""
1077− else many1
1078− (notFollowedBy (gridTableSep '=') >> char '|' >>
1167+ rawContent <- many1 (notFollowedBy (gridTableSep '=') >> char '|' >>
10791168 T.pack <$> many1Till anyChar newline)
1080− underDashes <- if headless
1081− then return dashes
1082− else gridDashedLines '='
1169+ underDashes <- gridDashedLines '='
10831170 guard $ length dashes == length underDashes
10841171 let lines' = map (snd . fst) underDashes
10851172 let indices = scanl (+) 0 lines'
10861173 let aligns = map snd underDashes
1087− let rawHeads = if headless
1088− then replicate (length underDashes) ""
1089− else map (T.unlines . map trim) $ transpose
1174+ let rawHeads = map (T.unlines . map trim) $ transpose
10901175 $ map (gridTableSplitLine indices) rawContent
10911176 heads <- sequence <$> mapM (parseFromString' blocks . trim) rawHeads
10921177 return (heads, aligns, indices)
10931178
1094−gridTableRawLine :: Stream s m Char => [Int] -> ParserT s st m [Text]
1179+gridTableRawLine :: (Stream s m Char, UpdateSourcePos s Char) => [Int] -> ParserT s st m [Text]
10951180gridTableRawLine indices = do
10961181 char '|'
10971182 line <- many1Till anyChar newline
10981183 return (gridTableSplitLine indices $ T.pack line)
10991184
11001185-- | Parse row of grid table.
1101−gridTableRow :: (Stream s m Char, Monad mf, IsString s, HasLastStrPosition st)
1102− => ParserT s st m (mf Blocks)
1186+gridTableRow :: (Monad m, Monad mf, HasLastStrPosition st)
1187+ => ParserT Sources st m (mf Blocks)
11031188 -> [Int]
1104− -> ParserT s st m (mf [Blocks])
1189+ -> ParserT Sources st m (mf [Blocks])
11051190gridTableRow blocks indices = do
11061191 colLines <- many1 (gridTableRawLine indices)
11071192 let cols = map ((<> "\n") . T.unlines . removeOneLeadingSpace) $
@@ -1122,34 +1207,38 @@ removeOneLeadingSpace xs =
11221207 Just (c, _) -> c == ' '
11231208
11241209-- | Parse footer for a grid table.
1125−gridTableFooter :: Stream s m Char => ParserT s st m ()
1210+gridTableFooter :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m ()
11261211gridTableFooter = optional blanklines
11271212
11281213---
11291214
11301215-- | Removes the ParsecT layer from the monad transformer stack
1131−readWithM :: (Stream s m Char, ToText s)
1132− => ParserT s st m a -- ^ parser
1133− -> st -- ^ initial state
1134− -> s -- ^ input
1216+readWithM :: (Monad m, ToSources t)
1217+ => ParserT Sources st m a -- ^ parser
1218+ -> st -- ^ initial state
1219+ -> t -- ^ input
11351220 -> m (Either PandocError a)
11361221readWithM parser state input =
1137− mapLeft (PandocParsecError $ toText input) `liftM` runParserT parser state "source" input
1222+ mapLeft (PandocParsecError sources)
1223+ <$> runParserT parser state (initialSourceName sources) sources
1224+ where
1225+ sources = toSources input
11381226
11391227-- | Parse a string with a given parser and state
1140−readWith :: Parser Text st a
1228+readWith :: ToSources t
1229+ => Parser Sources st a
11411230 -> st
1142− -> Text
1231+ -> t
11431232 -> Either PandocError a
11441233readWith p t inp = runIdentity $ readWithM p t inp
11451234
11461235-- | Parse a string with @parser@ (for testing).
11471236testStringWith :: Show a
1148− => ParserT Text ParserState Identity a
1237+ => ParserT Sources ParserState Identity a
11491238 -> Text
11501239 -> IO ()
1151−testStringWith parser str = UTF8.putStrLn $ show $
1152− readWith parser defaultParserState str
1240+testStringWith parser str = UTF8.putStrLn $ tshow $
1241+ readWith parser defaultParserState (toSources str)
11531242
11541243-- | Parsing options.
11551244data ParserState = ParserState
@@ -1169,7 +1258,7 @@ data ParserState = ParserState
11691258 stateInNote :: Bool, -- ^ True if parsing note contents
11701259 stateNoteNumber :: Int, -- ^ Last note number for citations
11711260 stateMeta :: Meta, -- ^ Document metadata
1172− stateMeta' :: F Meta, -- ^ Document metadata
1261+ stateMeta' :: Future ParserState Meta, -- ^ Document metadata
11731262 stateCitations :: M.Map Text Text, -- ^ RST-style citations
11741263 stateHeaderTable :: [HeaderType], -- ^ Ordered list of header types used
11751264 stateIdentifiers :: Set.Set Text, -- ^ Header identifiers used
@@ -1348,7 +1437,7 @@ data QuoteContext
13481437
13491438type NoteTable = [(Text, Text)]
13501439
1351−type NoteTable' = M.Map Text (SourcePos, F Blocks)
1440+type NoteTable' = M.Map Text (SourcePos, Future ParserState Blocks)
13521441-- used in markdown reader
13531442
13541443newtype Key = Key Text deriving (Show, Read, Eq, Ord)
@@ -1383,7 +1472,7 @@ registerHeader (ident,classes,kvs) header' = do
13831472 then do
13841473 let id' = uniqueIdent exts (B.toList header') ids
13851474 let id'' = if Ext_ascii_identifiers `extensionEnabled` exts
1386− then T.pack $ mapMaybe toAsciiChar $ T.unpack id'
1475+ then toAsciiText id'
13871476 else id'
13881477 updateState $ updateIdentifierList $ Set.insert id'
13891478 updateState $ updateIdentifierList $ Set.insert id''
@@ -1396,34 +1485,42 @@ registerHeader (ident,classes,kvs) header' = do
13961485 updateState $ updateIdentifierList $ Set.insert ident
13971486 return (ident,classes,kvs)
13981487
1399−smartPunctuation :: (HasReaderOptions st, HasLastStrPosition st, HasQuoteContext st m, Stream s m Char)
1488+smartPunctuation :: (HasReaderOptions st, HasLastStrPosition st,
1489+ HasQuoteContext st m,
1490+ Stream s m Char, UpdateSourcePos s Char)
14001491 => ParserT s st m Inlines
14011492 -> ParserT s st m Inlines
14021493smartPunctuation inlineParser = do
14031494 guardEnabled Ext_smart
1404− choice [ quoted inlineParser, apostrophe, dash, ellipses ]
1405−
1406−apostrophe :: Stream s m Char => ParserT s st m Inlines
1407−apostrophe = (char '\'' <|> char '\8217') >> return (B.str "\x2019")
1495+ choice [ quoted inlineParser, apostrophe, doubleCloseQuote, dash, ellipses ]
14081496
1409−quoted :: (HasLastStrPosition st, HasQuoteContext st m, Stream s m Char)
1497+quoted :: (HasLastStrPosition st, HasQuoteContext st m,
1498+ Stream s m Char, UpdateSourcePos s Char)
14101499 => ParserT s st m Inlines
14111500 -> ParserT s st m Inlines
14121501quoted inlineParser = doubleQuoted inlineParser <|> singleQuoted inlineParser
14131502
1414−singleQuoted :: (HasLastStrPosition st, HasQuoteContext st m, Stream s m Char)
1503+singleQuoted :: (HasLastStrPosition st, HasQuoteContext st m,
1504+ Stream s m Char, UpdateSourcePos s Char)
14151505 => ParserT s st m Inlines
14161506 -> ParserT s st m Inlines
1417−singleQuoted inlineParser = try $ B.singleQuoted . mconcat
1418− <$ singleQuoteStart
1419− <*> withQuoteContext InSingleQuote (many1Till inlineParser singleQuoteEnd)
1420−
1421−doubleQuoted :: (HasQuoteContext st m, Stream s m Char)
1507+singleQuoted inlineParser = do
1508+ singleQuoteStart
1509+ (B.singleQuoted . mconcat <$>
1510+ try
1511+ (withQuoteContext InSingleQuote (many1Till inlineParser singleQuoteEnd)))
1512+ <|> pure "\8217"
1513+
1514+doubleQuoted :: (HasQuoteContext st m, HasLastStrPosition st,
1515+ Stream s m Char, UpdateSourcePos s Char)
14221516 => ParserT s st m Inlines
14231517 -> ParserT s st m Inlines
1424−doubleQuoted inlineParser = try $ B.doubleQuoted . mconcat
1425− <$ doubleQuoteStart
1426− <*> withQuoteContext InDoubleQuote (manyTill inlineParser doubleQuoteEnd)
1518+doubleQuoted inlineParser = do
1519+ doubleQuoteStart
1520+ (B.doubleQuoted . mconcat <$>
1521+ try
1522+ (withQuoteContext InDoubleQuote (manyTill inlineParser doubleQuoteEnd)))
1523+ <|> pure (B.str "\8220")
14271524
14281525failIfInQuoteContext :: (HasQuoteContext st m, Stream s m t)
14291526 => QuoteContext
@@ -1432,13 +1529,14 @@ failIfInQuoteContext context = do
14321529 context' <- getQuoteContext
14331530 when (context' == context) $ Prelude.fail "already inside quotes"
14341531
1435−charOrRef :: Stream s m Char => [Char] -> ParserT s st m Char
1532+charOrRef :: (Stream s m Char, UpdateSourcePos s Char) => [Char] -> ParserT s st m Char
14361533charOrRef cs =
14371534 oneOf cs <|> try (do c <- characterReference
14381535 guard (c `elem` cs)
14391536 return c)
14401537
1441−singleQuoteStart :: (HasLastStrPosition st, HasQuoteContext st m, Stream s m Char)
1538+singleQuoteStart :: (HasLastStrPosition st, HasQuoteContext st m,
1539+ Stream s m Char, UpdateSourcePos s Char)
14421540 => ParserT s st m ()
14431541singleQuoteStart = do
14441542 failIfInQuoteContext InSingleQuote
@@ -1446,30 +1544,39 @@ singleQuoteStart = do
14461544 guard =<< notAfterString
14471545 try $ do
14481546 charOrRef "'\8216\145"
1449− notFollowedBy (oneOf [' ', '\t', '\n'])
1547+ void $ lookAhead (satisfy (not . isSpaceChar))
14501548
1451−singleQuoteEnd :: Stream s m Char
1549+singleQuoteEnd :: (Stream s m Char, UpdateSourcePos s Char)
14521550 => ParserT s st m ()
14531551singleQuoteEnd = try $ do
14541552 charOrRef "'\8217\146"
14551553 notFollowedBy alphaNum
14561554
1457−doubleQuoteStart :: (HasQuoteContext st m, Stream s m Char)
1555+doubleQuoteStart :: (HasLastStrPosition st,
1556+ HasQuoteContext st m,
1557+ Stream s m Char, UpdateSourcePos s Char)
14581558 => ParserT s st m ()
14591559doubleQuoteStart = do
14601560 failIfInQuoteContext InDoubleQuote
1561+ guard =<< notAfterString
14611562 try $ do charOrRef "\"\8220\147"
1462− notFollowedBy (oneOf [' ', '\t', '\n'])
1563+ void $ lookAhead (satisfy (not . isSpaceChar))
14631564
1464−doubleQuoteEnd :: Stream s m Char
1565+doubleQuoteEnd :: (Stream s m Char, UpdateSourcePos s Char)
14651566 => ParserT s st m ()
14661567doubleQuoteEnd = void (charOrRef "\"\8221\148")
14671568
1468−ellipses :: Stream s m Char
1569+apostrophe :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m Inlines
1570+apostrophe = (char '\'' <|> char '\8217') >> return (B.str "\8217")
1571+
1572+doubleCloseQuote :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m Inlines
1573+doubleCloseQuote = B.str "\8221" <$ char '"'
1574+
1575+ellipses :: (Stream s m Char, UpdateSourcePos s Char)
14691576 => ParserT s st m Inlines
14701577ellipses = try (string "..." >> return (B.str "\8230"))
14711578
1472−dash :: (HasReaderOptions st, Stream s m Char)
1579+dash :: (HasReaderOptions st, Stream s m Char, UpdateSourcePos s Char)
14731580 => ParserT s st m Inlines
14741581dash = try $ do
14751582 oldDashes <- extensionEnabled Ext_old_dashes <$> getOption readerExtensions
@@ -1496,20 +1603,28 @@ nested p = do
14961603 updateState $ \st -> st{ stateMaxNestingLevel = nestlevel }
14971604 return res
14981605
1499−citeKey :: (Stream s m Char, HasLastStrPosition st)
1500− => ParserT s st m (Bool, Text)
1501−citeKey = try $ do
1606+citeKey :: (Stream s m Char, UpdateSourcePos s Char, HasLastStrPosition st)
1607+ => Bool -- ^ If True, allow expanded @{..} syntax.
1608+ -> ParserT s st m (Bool, Text)
1609+citeKey allowBraced = try $ do
15021610 guard =<< notAfterString
15031611 suppress_author <- option False (True <$ char '-')
15041612 char '@'
1613+ key <- simpleCiteIdentifier
1614+ <|> if allowBraced
1615+ then charsInBalanced '{' '}' (satisfy (not . isSpace))
1616+ else mzero
1617+ return (suppress_author, key)
1618+
1619+simpleCiteIdentifier :: (Stream s m Char, UpdateSourcePos s Char)
1620+ => ParserT s st m Text
1621+simpleCiteIdentifier = do
15051622 firstChar <- alphaNum <|> char '_' <|> char '*' -- @* for wildcard in nocite
15061623 let regchar = satisfy (\c -> isAlphaNum c || c == '_')
15071624 let internal p = try $ p <* lookAhead regchar
15081625 rest <- many $ regchar <|> internal (oneOf ":.#$%&-+?<>~/") <|>
15091626 try (oneOf ":/" <* lookAhead (char '/'))
1510− let key = firstChar:rest
1511− return (suppress_author, T.pack key)
1512−
1627+ return $ T.pack $ firstChar:rest
15131628
15141629token :: (Stream s m t)
15151630 => (t -> Text)
@@ -1529,12 +1644,15 @@ extractIdClass (ident, cls, kvs) = (ident', cls', kvs')
15291644 cls' = maybe cls T.words $ lookup "class" kvs
15301645 kvs' = filter (\(k,_) -> k /= "id" || k /= "class") kvs
15311646
1532−insertIncludedFile' :: (PandocMonad m, HasIncludeFiles st)
1533− => ParserT a st m (mf Blocks)
1534− -> (Text -> a)
1535− -> [FilePath] -> FilePath
1536− -> ParserT a st m (mf Blocks)
1537−insertIncludedFile' blocks totoks dirs f = do
1647+insertIncludedFile :: (PandocMonad m, HasIncludeFiles st)
1648+ => ParserT a st m b -- ^ parser to apply
1649+ -> (Text -> a) -- ^ convert Text to stream type
1650+ -> [FilePath] -- ^ search path (directories)
1651+ -> FilePath -- ^ path of file to include
1652+ -> Maybe Int -- ^ start line (negative counts from end)
1653+ -> Maybe Int -- ^ end line (negative counts from end)
1654+ -> ParserT a st m b
1655+insertIncludedFile parser toStream dirs f mbstartline mbendline = do
15381656 oldPos <- getPosition
15391657 oldInput <- getInput
15401658 containers <- getIncludeFiles <$> getState
@@ -1543,32 +1661,32 @@ insertIncludedFile' blocks totoks dirs f = do
15431661 updateState $ addIncludeFile $ T.pack f
15441662 mbcontents <- readFileFromDirs dirs f
15451663 contents <- case mbcontents of
1546− Just s -> return s
1664+ Just s -> return $ exciseLines mbstartline mbendline s
15471665 Nothing -> do
15481666 report $ CouldNotLoadIncludeFile (T.pack f) oldPos
15491667 return ""
1550− setPosition $ newPos f 1 1
1551− setInput $ totoks contents
1552− bs <- blocks
1668+ setInput $ toStream contents
1669+ setPosition $ newPos f (fromMaybe 1 mbstartline) 1
1670+ result <- parser
15531671 setInput oldInput
15541672 setPosition oldPos
15551673 updateState dropLatestIncludeFile
1556− return bs
1674+ return result
1675+
1676+exciseLines :: Maybe Int -> Maybe Int -> Text -> Text
1677+exciseLines Nothing Nothing t = t
1678+exciseLines mbstartline mbendline t =
1679+ T.unlines $ take (endline' - (startline' - 1))
1680+ $ drop (startline' - 1) contentLines
1681+ where
1682+ contentLines = T.lines t
1683+ numLines = length contentLines
1684+ startline' = case mbstartline of
1685+ Nothing -> 1
1686+ Just x | x >= 0 -> x
1687+ | otherwise -> numLines + x -- negative from end
1688+ endline' = case mbendline of
1689+ Nothing -> numLines
1690+ Just x | x >= 0 -> x
1691+ | otherwise -> numLines + x -- negative from end
15571692
1558−-- | Parse content of include file as blocks. Circular includes result in an
1559−-- @PandocParseError@.
1560−insertIncludedFile :: (PandocMonad m, HasIncludeFiles st)
1561− => ParserT [a] st m Blocks
1562− -> (Text -> [a])
1563− -> [FilePath] -> FilePath
1564− -> ParserT [a] st m Blocks
1565−insertIncludedFile blocks totoks dirs f =
1566− runIdentity <$> insertIncludedFile' (Identity <$> blocks) totoks dirs f
1567−
1568−-- | Parse content of include file as future blocks. Circular includes result in
1569−-- an @PandocParseError@.
1570−insertIncludedFileF :: (PandocMonad m, HasIncludeFiles st)
1571− => ParserT Text st m (Future st Blocks)
1572− -> [FilePath] -> FilePath
1573− -> ParserT Text st m (Future st Blocks)
1574−insertIncludedFileF p = insertIncludedFile' p id
src/Text/Pandoc/Process.hs+1−1
@@ -1,6 +1,6 @@
11{- |
22 Module : Text.Pandoc.Process
3− Copyright : Copyright (C) 2013-2020 John MacFarlane
3+ Copyright : Copyright (C) 2013-2021 John MacFarlane
44 License : GNU GPL, version 2 or above
55
66 Maintainer : John MacFarlane <jgm@berkeley.edu>
src/Text/Pandoc/Readers.hs+56−49
@@ -1,9 +1,10 @@
11{-# LANGUAGE MonoLocalBinds #-}
2+{-# LANGUAGE RankNTypes #-}
23{-# LANGUAGE ScopedTypeVariables #-}
34{-# LANGUAGE OverloadedStrings #-}
45{- |
56 Module : Text.Pandoc.Readers
6− Copyright : Copyright (C) 2006-2020 John MacFarlane
7+ Copyright : Copyright (C) 2006-2021 John MacFarlane
78 License : GNU GPL, version 2 or above
89
910 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -65,12 +66,14 @@ import Data.Aeson
6566import qualified Data.ByteString.Lazy as BL
6667import Data.Text (Text)
6768import qualified Data.Text as T
69+import Text.Pandoc.Shared (tshow)
6870import Text.Pandoc.Class
6971import Text.Pandoc.Definition
7072import Text.Pandoc.Error
7173import Text.Pandoc.Extensions
7274import Text.Pandoc.Options
7375import Text.Pandoc.Readers.CommonMark
76+import Text.Pandoc.Readers.Markdown
7477import Text.Pandoc.Readers.Creole
7578import Text.Pandoc.Readers.DocBook
7679import Text.Pandoc.Readers.Docx
@@ -83,7 +86,6 @@ import Text.Pandoc.Readers.HTML (readHtml)
8386import Text.Pandoc.Readers.JATS (readJATS)
8487import Text.Pandoc.Readers.Jira (readJira)
8588import Text.Pandoc.Readers.LaTeX
86−import Text.Pandoc.Readers.Markdown
8789import Text.Pandoc.Readers.MediaWiki
8890import Text.Pandoc.Readers.Muse
8991import Text.Pandoc.Readers.Native
@@ -101,59 +103,60 @@ import Text.Pandoc.Readers.CSV
101103import Text.Pandoc.Readers.CslJson
102104import Text.Pandoc.Readers.BibTeX
103105import qualified Text.Pandoc.UTF8 as UTF8
104−import Text.Parsec.Error
106+import Text.Pandoc.Sources (ToSources(..), sourcesToText)
105107
106−data Reader m = TextReader (ReaderOptions -> Text -> m Pandoc)
108+data Reader m = TextReader (forall a . ToSources a =>
109+ ReaderOptions -> a -> m Pandoc)
107110 | ByteStringReader (ReaderOptions -> BL.ByteString -> m Pandoc)
108111
109112-- | Association list of formats and readers.
110113readers :: PandocMonad m => [(Text, Reader m)]
111−readers = [ ("native" , TextReader readNative)
112− ,("json" , TextReader readJSON)
113− ,("markdown" , TextReader readMarkdown)
114− ,("markdown_strict" , TextReader readMarkdown)
115− ,("markdown_phpextra" , TextReader readMarkdown)
116− ,("markdown_github" , TextReader readMarkdown)
117− ,("markdown_mmd", TextReader readMarkdown)
118− ,("commonmark" , TextReader readCommonMark)
119− ,("commonmark_x" , TextReader readCommonMark)
120− ,("creole" , TextReader readCreole)
121− ,("dokuwiki" , TextReader readDokuWiki)
122− ,("gfm" , TextReader readCommonMark)
123− ,("rst" , TextReader readRST)
124− ,("mediawiki" , TextReader readMediaWiki)
125− ,("vimwiki" , TextReader readVimwiki)
126− ,("docbook" , TextReader readDocBook)
127− ,("opml" , TextReader readOPML)
128− ,("org" , TextReader readOrg)
129− ,("textile" , TextReader readTextile) -- TODO : textile+lhs
130− ,("html" , TextReader readHtml)
131− ,("jats" , TextReader readJATS)
132− ,("jira" , TextReader readJira)
133− ,("latex" , TextReader readLaTeX)
134− ,("haddock" , TextReader readHaddock)
135− ,("twiki" , TextReader readTWiki)
136− ,("tikiwiki" , TextReader readTikiWiki)
137− ,("docx" , ByteStringReader readDocx)
138− ,("odt" , ByteStringReader readOdt)
139− ,("t2t" , TextReader readTxt2Tags)
140− ,("epub" , ByteStringReader readEPUB)
141− ,("muse" , TextReader readMuse)
142− ,("man" , TextReader readMan)
143− ,("fb2" , TextReader readFB2)
144− ,("ipynb" , TextReader readIpynb)
145− ,("csv" , TextReader readCSV)
146− ,("csljson" , TextReader readCslJson)
147− ,("bibtex" , TextReader readBibTeX)
148− ,("biblatex" , TextReader readBibLaTeX)
114+readers = [("native" , TextReader readNative)
115+ ,("json" , TextReader readJSON)
116+ ,("markdown" , TextReader readMarkdown)
117+ ,("markdown_strict" , TextReader readMarkdown)
118+ ,("markdown_phpextra" , TextReader readMarkdown)
119+ ,("markdown_github" , TextReader readMarkdown)
120+ ,("markdown_mmd", TextReader readMarkdown)
121+ ,("commonmark" , TextReader readCommonMark)
122+ ,("commonmark_x" , TextReader readCommonMark)
123+ ,("creole" , TextReader readCreole)
124+ ,("dokuwiki" , TextReader readDokuWiki)
125+ ,("gfm" , TextReader readCommonMark)
126+ ,("rst" , TextReader readRST)
127+ ,("mediawiki" , TextReader readMediaWiki)
128+ ,("vimwiki" , TextReader readVimwiki)
129+ ,("docbook" , TextReader readDocBook)
130+ ,("opml" , TextReader readOPML)
131+ ,("org" , TextReader readOrg)
132+ ,("textile" , TextReader readTextile) -- TODO : textile+lhs
133+ ,("html" , TextReader readHtml)
134+ ,("jats" , TextReader readJATS)
135+ ,("jira" , TextReader readJira)
136+ ,("latex" , TextReader readLaTeX)
137+ ,("haddock" , TextReader readHaddock)
138+ ,("twiki" , TextReader readTWiki)
139+ ,("tikiwiki" , TextReader readTikiWiki)
140+ ,("docx" , ByteStringReader readDocx)
141+ ,("odt" , ByteStringReader readOdt)
142+ ,("t2t" , TextReader readTxt2Tags)
143+ ,("epub" , ByteStringReader readEPUB)
144+ ,("muse" , TextReader readMuse)
145+ ,("man" , TextReader readMan)
146+ ,("fb2" , TextReader readFB2)
147+ ,("ipynb" , TextReader readIpynb)
148+ ,("csv" , TextReader readCSV)
149+ ,("csljson" , TextReader readCslJson)
150+ ,("bibtex" , TextReader readBibTeX)
151+ ,("biblatex" , TextReader readBibLaTeX)
149152 ]
150153
151154-- | Retrieve reader, extensions based on formatSpec (format+extensions).
152155getReader :: PandocMonad m => Text -> m (Reader m, Extensions)
153156getReader s =
154157 case parseFormatSpec s of
155− Left e -> throwError $ PandocAppError
156− $ T.intercalate "\n" [T.pack m | Message m <- errorMessages e]
158+ Left e -> throwError $ PandocAppError $
159+ "Error parsing reader format " <> tshow s <> ": " <> tshow e
157160 Right (readerName, extsToEnable, extsToDisable) ->
158161 case lookup readerName readers of
159162 Nothing -> throwError $ PandocUnknownReaderError
@@ -173,9 +176,13 @@ getReader s =
173176 return (r, exts)
174177
175178-- | Read pandoc document from JSON format.
176−readJSON :: PandocMonad m
177− => ReaderOptions -> Text -> m Pandoc
178−readJSON _ t =
179− case eitherDecode' . BL.fromStrict . UTF8.fromText $ t of
179+readJSON :: (PandocMonad m, ToSources a)
180+ => ReaderOptions
181+ -> a
182+ -> m Pandoc
183+readJSON _ s =
184+ case eitherDecode' . BL.fromStrict . UTF8.fromText
185+ . sourcesToText . toSources $ s of
180186 Right doc -> return doc
181− Left e -> throwError $ PandocParseError ("JSON parse error: " <> T.pack e)
187+ Left e -> throwError $ PandocParseError ("JSON parse error: "
188+ <> T.pack e)
src/Text/Pandoc/Readers/BibTeX.hs+15−9
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.Readers.BibTeX
4− Copyright : Copyright (C) 2020 John MacFarlane
4+ Copyright : Copyright (C) 2020-2021 John MacFarlane
55 License : GNU GPL, version 2 or above
66
77 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -23,41 +23,47 @@ where
2323import Text.Pandoc.Options
2424import Text.Pandoc.Definition
2525import Text.Pandoc.Builder (setMeta, cite, str)
26−import Data.Text (Text)
2726import Citeproc (Lang(..), parseLang)
2827import Citeproc.Locale (getLocale)
2928import Text.Pandoc.Error (PandocError(..))
3029import Text.Pandoc.Class (PandocMonad, lookupEnv)
3130import Text.Pandoc.Citeproc.BibTeX as BibTeX
3231import Text.Pandoc.Citeproc.MetaValue (referenceToMetaValue)
32+import Text.Pandoc.Sources (ToSources(..))
3333import Control.Monad.Except (throwError)
3434
3535-- | Read BibTeX from an input string and return a Pandoc document.
3636-- The document will have only metadata, with an empty body.
3737-- The metadata will contain a `references` field with the
3838-- bibliography entries, and a `nocite` field with the wildcard `[@*]`.
39−readBibTeX :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
39+readBibTeX :: (PandocMonad m, ToSources a)
40+ => ReaderOptions -> a -> m Pandoc
4041readBibTeX = readBibTeX' BibTeX.Bibtex
4142
4243-- | Read BibLaTeX from an input string and return a Pandoc document.
4344-- The document will have only metadata, with an empty body.
4445-- The metadata will contain a `references` field with the
4546-- bibliography entries, and a `nocite` field with the wildcard `[@*]`.
46−readBibLaTeX :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
47+readBibLaTeX :: (PandocMonad m, ToSources a)
48+ => ReaderOptions -> a -> m Pandoc
4749readBibLaTeX = readBibTeX' BibTeX.Biblatex
4850
49−readBibTeX' :: PandocMonad m => Variant -> ReaderOptions -> Text -> m Pandoc
51+readBibTeX' :: (PandocMonad m, ToSources a)
52+ => Variant -> ReaderOptions -> a -> m Pandoc
5053readBibTeX' variant _opts t = do
51− lang <- maybe (Lang "en" (Just "US")) parseLang
52− <$> lookupEnv "LANG"
54+ mblangEnv <- lookupEnv "LANG"
55+ let defaultLang = Lang "en" Nothing (Just "US") [] [] []
56+ let lang = case mblangEnv of
57+ Nothing -> defaultLang
58+ Just l -> either (const defaultLang) id $ parseLang l
5359 locale <- case getLocale lang of
5460 Left e ->
55− case getLocale (Lang "en" (Just "US")) of
61+ case getLocale (Lang "en" Nothing (Just "US") [] [] []) of
5662 Right l -> return l
5763 Left _ -> throwError $ PandocCiteprocError e
5864 Right l -> return l
5965 case BibTeX.readBibtexString variant locale (const True) t of
60− Left e -> throwError $ PandocParsecError t e
66+ Left e -> throwError $ PandocParsecError (toSources t) e
6167 Right refs -> return $ setMeta "references"
6268 (map referenceToMetaValue refs)
6369 . setMeta "nocite"
src/Text/Pandoc/Readers/CSV.hs+8−8
@@ -3,7 +3,7 @@
33{-# LANGUAGE ScopedTypeVariables #-}
44{- |
55 Module : Text.Pandoc.Readers.CSV
6− Copyright : Copyright (C) 2006-2020 John MacFarlane
6+ Copyright : Copyright (C) 2006-2021 John MacFarlane
77 License : GNU GPL, version 2 or above
88
99 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -13,23 +13,23 @@
1313Conversion from CSV to a 'Pandoc' table.
1414-}
1515module Text.Pandoc.Readers.CSV ( readCSV ) where
16−import Data.Text (Text)
1716import qualified Data.Text as T
1817import Text.Pandoc.CSV (parseCSV, defaultCSVOptions)
1918import Text.Pandoc.Definition
2019import qualified Text.Pandoc.Builder as B
2120import Text.Pandoc.Class (PandocMonad)
22−import Text.Pandoc.Shared (crFilter)
2321import Text.Pandoc.Error
22+import Text.Pandoc.Sources (ToSources(..), sourcesToText)
2423import Text.Pandoc.Options (ReaderOptions)
2524import Control.Monad.Except (throwError)
2625
27−readCSV :: PandocMonad m
26+readCSV :: (PandocMonad m, ToSources a)
2827 => ReaderOptions -- ^ Reader options
29− -> Text -- ^ Text to parse (assuming @'\n'@ line endings)
28+ -> a
3029 -> m Pandoc
31−readCSV _opts s =
32− case parseCSV defaultCSVOptions (crFilter s) of
30+readCSV _opts s = do
31+ let txt = sourcesToText $ toSources s
32+ case parseCSV defaultCSVOptions txt of
3333 Right (r:rs) -> return $ B.doc $ B.table capt
3434 (zip aligns widths)
3535 (TableHead nullAttr hdrs)
@@ -45,4 +45,4 @@ readCSV _opts s =
4545 aligns = replicate numcols AlignDefault
4646 widths = replicate numcols ColWidthDefault
4747 Right [] -> return $ B.doc mempty
48− Left e -> throwError $ PandocParsecError s e
48+ Left e -> throwError $ PandocParsecError (toSources [("",txt)]) e
src/Text/Pandoc/Readers/CommonMark.hs+47−5
@@ -3,7 +3,7 @@
33{-# LANGUAGE ScopedTypeVariables #-}
44{- |
55 Module : Text.Pandoc.Readers.CommonMark
6− Copyright : Copyright (C) 2015-2020 John MacFarlane
6+ Copyright : Copyright (C) 2015-2021 John MacFarlane
77 License : GNU GPL, version 2 or above
88
99 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -26,19 +26,59 @@ import Text.Pandoc.Definition
2626import Text.Pandoc.Builder as B
2727import Text.Pandoc.Options
2828import Text.Pandoc.Error
29+import Text.Pandoc.Readers.Metadata (yamlMetaBlock)
2930import Control.Monad.Except
3031import Data.Functor.Identity (runIdentity)
3132import Data.Typeable
33+import Text.Pandoc.Parsing (runParserT, getInput,
34+ runF, defaultParserState, option, many1, anyChar,
35+ Sources(..), ToSources(..), ParserT, Future,
36+ sourceName)
37+import qualified Data.Text as T
3238
3339-- | Parse a CommonMark formatted string into a 'Pandoc' structure.
34−readCommonMark :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
40+readCommonMark :: (PandocMonad m, ToSources a)
41+ => ReaderOptions -> a -> m Pandoc
3542readCommonMark opts s
43+ | isEnabled Ext_yaml_metadata_block opts = do
44+ let sources = toSources s
45+ let toks = concatMap sourceToToks (unSources sources)
46+ res <- runParserT (do meta <- yamlMetaBlock (metaValueParser opts)
47+ rest <- getInput
48+ return (meta, rest))
49+ defaultParserState "YAML metadata" (toSources s)
50+ case res of
51+ Left _ -> readCommonMarkBody opts sources toks
52+ Right (meta, rest) -> do
53+ -- strip off metadata section and parse body
54+ let body = concatMap sourceToToks (unSources rest)
55+ Pandoc _ bs <- readCommonMarkBody opts sources body
56+ return $ Pandoc (runF meta defaultParserState) bs
57+ | otherwise = do
58+ let sources = toSources s
59+ let toks = concatMap sourceToToks (unSources sources)
60+ readCommonMarkBody opts sources toks
61+
62+sourceToToks :: (SourcePos, Text) -> [Tok]
63+sourceToToks (pos, s) = tokenize (sourceName pos) s
64+
65+metaValueParser :: Monad m
66+ => ReaderOptions -> ParserT Sources st m (Future st MetaValue)
67+metaValueParser opts = do
68+ inp <- option "" $ T.pack <$> many1 anyChar
69+ let toks = concatMap sourceToToks (unSources (toSources inp))
70+ case runIdentity (parseCommonmarkWith (specFor opts) toks) of
71+ Left _ -> mzero
72+ Right (Cm bls :: Cm () Blocks) -> return $ return $ B.toMetaValue bls
73+
74+readCommonMarkBody :: PandocMonad m => ReaderOptions -> Sources -> [Tok] -> m Pandoc
75+readCommonMarkBody opts s toks
3676 | isEnabled Ext_sourcepos opts =
37− case runIdentity (commonmarkWith (specFor opts) "" s) of
77+ case runIdentity (parseCommonmarkWith (specFor opts) toks) of
3878 Left err -> throwError $ PandocParsecError s err
3979 Right (Cm bls :: Cm SourceRange Blocks) -> return $ B.doc bls
4080 | otherwise =
41− case runIdentity (commonmarkWith (specFor opts) "" s) of
81+ case runIdentity (parseCommonmarkWith (specFor opts) toks) of
4282 Left err -> throwError $ PandocParsecError s err
4383 Right (Cm bls :: Cm () Blocks) -> return $ B.doc bls
4484
@@ -72,5 +112,7 @@ specFor opts = foldr ($) defaultSyntaxSpec exts
72112 | isEnabled Ext_implicit_header_references opts ] ++
73113 [ (footnoteSpec <>) | isEnabled Ext_footnotes opts ] ++
74114 [ (definitionListSpec <>) | isEnabled Ext_definition_lists opts ] ++
75− [ (taskListSpec <>) | isEnabled Ext_task_lists opts ]
115+ [ (taskListSpec <>) | isEnabled Ext_task_lists opts ] ++
116+ [ (rebaseRelativePathsSpec <>)
117+ | isEnabled Ext_rebase_relative_paths opts ]
76118
src/Text/Pandoc/Readers/Creole.hs+5−6
@@ -23,21 +23,20 @@ import Text.Pandoc.Class.PandocMonad (PandocMonad (..))
2323import Text.Pandoc.Definition
2424import Text.Pandoc.Options
2525import Text.Pandoc.Parsing hiding (enclosed)
26−import Text.Pandoc.Shared (crFilter)
27−
2826
2927-- | Read creole from an input string and return a Pandoc document.
30−readCreole :: PandocMonad m
28+readCreole :: (PandocMonad m, ToSources a)
3129 => ReaderOptions
32− -> Text
30+ -> a
3331 -> m Pandoc
3432readCreole opts s = do
35− res <- readWithM parseCreole def{ stateOptions = opts } $ crFilter s <> "\n\n"
33+ let sources = ensureFinalNewlines 2 (toSources s)
34+ res <- readWithM parseCreole def{ stateOptions = opts } sources
3635 case res of
3736 Left e -> throwError e
3837 Right d -> return d
3938
40−type CRLParser = ParserT Text ParserState
39+type CRLParser = ParserT Sources ParserState
4140
4241--
4342-- Utility functions
src/Text/Pandoc/Readers/CslJson.hs+6−5
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.Readers.CslJson
4− Copyright : Copyright (C) 2020 John MacFarlane
4+ Copyright : Copyright (C) 2020-2021 John MacFarlane
55 License : GNU GPL, version 2 or above
66
77 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -24,21 +24,22 @@ import Text.Pandoc.Options
2424import Text.Pandoc.Definition
2525import Text.Pandoc.Builder (setMeta, cite, str)
2626import qualified Text.Pandoc.UTF8 as UTF8
27−import Data.Text (Text)
2827import qualified Data.Text as T
2928import Text.Pandoc.Error (PandocError(..))
3029import Text.Pandoc.Class (PandocMonad)
3130import Text.Pandoc.Citeproc.CslJson (cslJsonToReferences)
3231import Text.Pandoc.Citeproc.MetaValue (referenceToMetaValue)
3332import Control.Monad.Except (throwError)
33+import Text.Pandoc.Sources (ToSources(..), sourcesToText)
3434
3535-- | Read CSL JSON from an input string and return a Pandoc document.
3636-- The document will have only metadata, with an empty body.
3737-- The metadata will contain a `references` field with the
3838-- bibliography entries, and a `nocite` field with the wildcard `[@*]`.
39−readCslJson :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
40−readCslJson _opts t =
41− case cslJsonToReferences (UTF8.fromText t) of
39+readCslJson :: (PandocMonad m, ToSources a)
40+ => ReaderOptions -> a -> m Pandoc
41+readCslJson _opts x =
42+ case cslJsonToReferences (UTF8.fromText $ sourcesToText $ toSources x) of
4243 Left e -> throwError $ PandocParseError $ T.pack e
4344 Right refs -> return $ setMeta "references"
4445 (map referenceToMetaValue refs)
src/Text/Pandoc/Readers/DocBook.hs+95−82
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.Readers.DocBook
4− Copyright : Copyright (C) 2006-2020 John MacFarlane
4+ Copyright : Copyright (C) 2006-2021 John MacFarlane
55 License : GNU GPL, version 2 or above
66
77 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -12,23 +12,28 @@ Conversion of DocBook XML to 'Pandoc' document.
1212-}
1313module Text.Pandoc.Readers.DocBook ( readDocBook ) where
1414import Control.Monad.State.Strict
15−import Data.Char (isSpace, toUpper)
15+import Data.Char (isSpace, isLetter)
1616import Data.Default
1717import Data.Either (rights)
1818import Data.Foldable (asum)
1919import Data.Generics
2020import Data.List (intersperse,elemIndex)
21+import Data.List.NonEmpty (nonEmpty)
2122import Data.Maybe (fromMaybe,mapMaybe)
2223import Data.Text (Text)
2324import qualified Data.Text as T
25+import qualified Data.Text.Lazy as TL
26+import Control.Monad.Except (throwError)
2427import Text.HTML.TagSoup.Entity (lookupEntity)
28+import Text.Pandoc.Error (PandocError(..))
2529import Text.Pandoc.Builder
2630import Text.Pandoc.Class.PandocMonad (PandocMonad, report)
2731import Text.Pandoc.Options
2832import Text.Pandoc.Logging (LogMessage(..))
29−import Text.Pandoc.Shared (crFilter, safeRead, extractSpaces)
33+import Text.Pandoc.Shared (safeRead, extractSpaces)
34+import Text.Pandoc.Sources (ToSources(..), sourcesToText)
3035import Text.TeXMath (readMathML, writeTeX)
31−import Text.XML.Light
36+import Text.Pandoc.XML.Light
3237
3338{-
3439
@@ -129,6 +134,7 @@ List of all DocBook tags, with [x] indicating implemented,
129134[ ] corpcredit - A corporation or organization credited in a document
130135[ ] corpname - The name of a corporation
131136[ ] country - The name of a country
137+[x] danger - An admonition set off from the text indicating hazardous situation
132138[ ] database - The name of a database, or part of a database
133139[x] date - The date of publication or revision of a document
134140[ ] dedication - A wrapper for the dedication section of a book
@@ -206,7 +212,7 @@ List of all DocBook tags, with [x] indicating implemented,
206212[x] info - A wrapper for information about a component or other block. (DocBook v5)
207213[x] informalequation - A displayed mathematical equation without a title
208214[x] informalexample - A displayed example without a title
209−[ ] informalfigure - A untitled figure
215+[x] informalfigure - An untitled figure
210216[ ] informaltable - A table without a title
211217[ ] initializer - The initializer for a FieldSynopsis
212218[x] inlineequation - A mathematical equation or expression occurring inline
@@ -535,24 +541,32 @@ instance Default DBState where
535541 , dbContent = [] }
536542
537543
538−readDocBook :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
544+readDocBook :: (PandocMonad m, ToSources a)
545+ => ReaderOptions
546+ -> a
547+ -> m Pandoc
539548readDocBook _ inp = do
540− let tree = normalizeTree . parseXML . handleInstructions $ crFilter inp
549+ let sources = toSources inp
550+ tree <- either (throwError . PandocXMLError "") return $
551+ parseXMLContents
552+ (TL.fromStrict . handleInstructions . sourcesToText $ sources)
541553 (bs, st') <- flip runStateT (def{ dbContent = tree }) $ mapM parseBlock tree
542554 return $ Pandoc (dbMeta st') (toList . mconcat $ bs)
543555
544−-- We treat <?asciidoc-br?> specially (issue #1236), converting it
545−-- to <br/>, since xml-light doesn't parse the instruction correctly.
546−-- Other xml instructions are simply removed from the input stream.
556+-- We treat certain processing instructions by converting them to tags
557+-- beginning "pi-".
547558handleInstructions :: Text -> Text
548−handleInstructions = T.pack . handleInstructions' . T.unpack
549−
550−handleInstructions' :: String -> String
551−handleInstructions' ('<':'?':'a':'s':'c':'i':'i':'d':'o':'c':'-':'b':'r':'?':'>':xs) = '<':'b':'r':'/':'>': handleInstructions' xs
552−handleInstructions' xs = case break (=='<') xs of
553− (ys, []) -> ys
554− ([], '<':zs) -> '<' : handleInstructions' zs
555− (ys, zs) -> ys ++ handleInstructions' zs
559+handleInstructions t =
560+ let (x,y) = T.breakOn "<?" t
561+ in if T.null y
562+ then x
563+ else
564+ let (w,z) = T.breakOn "?>" y
565+ in (if T.takeWhile (\c -> isLetter c || c == '-')
566+ (T.drop 2 w) `elem` ["asciidoc-br", "dbfo"]
567+ then x <> "<pi-" <> T.drop 2 w <> "/>"
568+ else x <> w <> T.take 2 z) <>
569+ handleInstructions (T.drop 2 z)
556570
557571getFigure :: PandocMonad m => Element -> DB m Blocks
558572getFigure e = do
@@ -565,32 +579,14 @@ getFigure e = do
565579 modify $ \st -> st{ dbFigureTitle = mempty, dbFigureId = mempty }
566580 return res
567581
568−-- normalize input, consolidating adjacent Text and CRef elements
569−normalizeTree :: [Content] -> [Content]
570−normalizeTree = everywhere (mkT go)
571− where go :: [Content] -> [Content]
572− go (Text (CData CDataRaw _ _):xs) = xs
573− go (Text (CData CDataText s1 z):Text (CData CDataText s2 _):xs) =
574− Text (CData CDataText (s1 ++ s2) z):xs
575− go (Text (CData CDataText s1 z):CRef r:xs) =
576− Text (CData CDataText (s1 ++ convertEntity r) z):xs
577− go (CRef r:Text (CData CDataText s1 z):xs) =
578− Text (CData CDataText (convertEntity r ++ s1) z):xs
579− go (CRef r1:CRef r2:xs) =
580− Text (CData CDataText (convertEntity r1 ++ convertEntity r2) Nothing):xs
581− go xs = xs
582−
583−convertEntity :: String -> String
584−convertEntity e = Data.Maybe.fromMaybe (map toUpper e) (lookupEntity e)
585−
586582-- convenience function to get an attribute value, defaulting to ""
587−attrValue :: String -> Element -> Text
583+attrValue :: Text -> Element -> Text
588584attrValue attr elt =
589− maybe "" T.pack (lookupAttrBy (\x -> qName x == attr) (elAttribs elt))
585+ fromMaybe "" (lookupAttrBy (\x -> qName x == attr) (elAttribs elt))
590586
591587-- convenience function
592588named :: Text -> Element -> Bool
593−named s e = qName (elName e) == T.unpack s
589+named s e = qName (elName e) == s
594590
595591--
596592
@@ -605,16 +601,24 @@ addMetadataFromElement e = do
605601 Nothing -> return ()
606602 Just z -> addMetaField "author" z
607603 addMetaField "subtitle" e
608− addMetaField "author" e
604+ addAuthor e
609605 addMetaField "date" e
610606 addMetaField "release" e
611607 addMetaField "releaseinfo" e
612608 return mempty
613− where addMetaField fieldname elt =
614− case filterChildren (named fieldname) elt of
615− [] -> return ()
616− [z] -> getInlines z >>= addMeta fieldname
617− zs -> mapM getInlines zs >>= addMeta fieldname
609+ where
610+ addAuthor elt =
611+ case filterChildren (named "author") elt of
612+ [] -> return ()
613+ [z] -> fromAuthor z >>= addMeta "author"
614+ zs -> mapM fromAuthor zs >>= addMeta "author"
615+ fromAuthor elt =
616+ mconcat . intersperse space <$> mapM getInlines (elChildren elt)
617+ addMetaField fieldname elt =
618+ case filterChildren (named fieldname) elt of
619+ [] -> return ()
620+ [z] -> getInlines z >>= addMeta fieldname
621+ zs -> mapM getInlines zs >>= addMeta fieldname
618622
619623addMeta :: PandocMonad m => ToMetaValue a => Text -> a -> DB m ()
620624addMeta field val = modify (setMeta field val)
@@ -627,7 +631,7 @@ isBlockElement :: Content -> Bool
627631isBlockElement (Elem e) = qName (elName e) `elem` blockTags
628632isBlockElement _ = False
629633
630−blockTags :: [String]
634+blockTags :: [Text]
631635blockTags =
632636 [ "abstract"
633637 , "ackno"
@@ -669,6 +673,7 @@ blockTags =
669673 , "index"
670674 , "info"
671675 , "informalexample"
676+ , "informalfigure"
672677 , "informaltable"
673678 , "itemizedlist"
674679 , "linegroup"
@@ -713,8 +718,8 @@ blockTags =
713718 , "variablelist"
714719 ] ++ admonitionTags
715720
716−admonitionTags :: [String]
717−admonitionTags = ["important","caution","note","tip","warning"]
721+admonitionTags :: [Text]
722+admonitionTags = ["caution","danger","important","note","tip","warning"]
718723
719724-- Trim leading and trailing newline characters
720725trimNl :: Text -> Text
@@ -771,10 +776,10 @@ getBlocks e = mconcat <$>
771776
772777parseBlock :: PandocMonad m => Content -> DB m Blocks
773778parseBlock (Text (CData CDataRaw _ _)) = return mempty -- DOCTYPE
774−parseBlock (Text (CData _ s _)) = if all isSpace s
779+parseBlock (Text (CData _ s _)) = if T.all isSpace s
775780 then return mempty
776− else return $ plain $ trimInlines $ text $ T.pack s
777−parseBlock (CRef x) = return $ plain $ str $ T.toUpper $ T.pack x
781+ else return $ plain $ trimInlines $ text s
782+parseBlock (CRef x) = return $ plain $ str $ T.toUpper x
778783parseBlock (Elem e) =
779784 case qName (elName e) of
780785 "toc" -> skip -- skip TOC, since in pandoc it's autogenerated
@@ -829,7 +834,7 @@ parseBlock (Elem e) =
829834 "refsect2" -> sect 2
830835 "refsect3" -> sect 3
831836 "refsection" -> gets dbSectionLevel >>= sect . (+1)
832− l | l `elem` admonitionTags -> parseAdmonition $ T.pack l
837+ l | l `elem` admonitionTags -> parseAdmonition l
833838 "area" -> skip
834839 "areaset" -> skip
835840 "areaspec" -> skip
@@ -855,6 +860,7 @@ parseBlock (Elem e) =
855860 "variablelist" -> definitionList <$> deflistitems
856861 "procedure" -> bulletList <$> steps
857862 "figure" -> getFigure e
863+ "informalfigure" -> getFigure e
858864 "mediaobject" -> para <$> getMediaobject e
859865 "caption" -> skip
860866 "info" -> addMetadataFromElement e
@@ -890,7 +896,11 @@ parseBlock (Elem e) =
890896 "subtitle" -> return mempty -- handled in parent element
891897 _ -> skip >> getBlocks e
892898 where skip = do
893− lift $ report $ IgnoredElement $ T.pack $ qName (elName e)
899+ let qn = qName $ elName e
900+ let name = if "pi-" `T.isPrefixOf` qn
901+ then "<?" <> qn <> "?>"
902+ else qn
903+ lift $ report $ IgnoredElement name
894904 return mempty
895905
896906 codeBlockWithLang = do
@@ -898,7 +908,7 @@ parseBlock (Elem e) =
898908 "" -> []
899909 x -> [x]
900910 return $ codeBlockWith (attrValue "id" e, classes', [])
901− $ trimNl $ T.pack $ strContentRecursive e
911+ $ trimNl $ strContentRecursive e
902912 parseBlockquote = do
903913 attrib <- case filterChild (named "attribution") e of
904914 Nothing -> return mempty
@@ -952,17 +962,16 @@ parseBlock (Elem e) =
952962 w <- findAttr (unqual "colwidth") c
953963 n <- safeRead $ "0" <> T.filter (\x ->
954964 (x >= '0' && x <= '9')
955− || x == '.') (T.pack w)
965+ || x == '.') w
956966 if n > 0 then Just n else Nothing
957− let numrows = case bodyrows of
958− [] -> 0
959− xs -> maximum $ map length xs
967+ let numrows = maybe 0 maximum $ nonEmpty
968+ $ map length bodyrows
960969 let aligns = case colspecs of
961970 [] -> replicate numrows AlignDefault
962971 cs -> map toAlignment cs
963972 let parseWidth s = safeRead (T.filter (\x -> (x >= '0' && x <= '9')
964973 || x == '.') s)
965− let textWidth = case filterChild (named "?dbfo") e of
974+ let textWidth = case filterChild (named "pi-dbfo") e of
966975 Just d -> case attrValue "table-width" d of
967976 "" -> 1.0
968977 w -> fromMaybe 100.0 (parseWidth w) / 100.0
@@ -1035,12 +1044,12 @@ parseMixed container conts = do
10351044 x <- parseMixed container rs
10361045 return $ p <> b <> x
10371046
1038−parseRow :: PandocMonad m => [String] -> Element -> DB m [Cell]
1047+parseRow :: PandocMonad m => [Text] -> Element -> DB m [Cell]
10391048parseRow cn = do
10401049 let isEntry x = named "entry" x || named "td" x || named "th" x
10411050 mapM (parseEntry cn) . filterChildren isEntry
10421051
1043−parseEntry :: PandocMonad m => [String] -> Element -> DB m Cell
1052+parseEntry :: PandocMonad m => [Text] -> Element -> DB m Cell
10441053parseEntry cn el = do
10451054 let colDistance sa ea = do
10461055 let iStrt = elemIndex sa cn
@@ -1062,7 +1071,7 @@ getInlines :: PandocMonad m => Element -> DB m Inlines
10621071getInlines e' = trimInlines . mconcat <$>
10631072 mapM parseInline (elContent e')
10641073
1065−strContentRecursive :: Element -> String
1074+strContentRecursive :: Element -> Text
10661075strContentRecursive = strContent .
10671076 (\e' -> e'{ elContent = map elementToStr $ elContent e' })
10681077
@@ -1071,16 +1080,16 @@ elementToStr (Elem e') = Text $ CData CDataText (strContentRecursive e') Nothing
10711080elementToStr x = x
10721081
10731082parseInline :: PandocMonad m => Content -> DB m Inlines
1074−parseInline (Text (CData _ s _)) = return $ text $ T.pack s
1083+parseInline (Text (CData _ s _)) = return $ text s
10751084parseInline (CRef ref) =
1076− return $ text $ maybe (T.toUpper $ T.pack ref) T.pack $ lookupEntity ref
1085+ return $ text $ maybe (T.toUpper ref) T.pack $ lookupEntity (T.unpack ref)
10771086parseInline (Elem e) =
10781087 case qName (elName e) of
10791088 "anchor" -> do
10801089 return $ spanWith (attrValue "id" e, [], []) mempty
10811090 "phrase" -> do
10821091 let ident = attrValue "id" e
1083− let classes = T.words $ attrValue "class" e
1092+ let classes = T.words $ attrValue "role" e
10841093 if ident /= "" || classes /= []
10851094 then innerInlines (spanWith (ident,classes,[]))
10861095 else innerInlines id
@@ -1125,7 +1134,7 @@ parseInline (Elem e) =
11251134 "userinput" -> codeWithLang
11261135 "systemitem" -> codeWithLang
11271136 "varargs" -> return $ code "(...)"
1128− "keycap" -> return (str $ T.pack $ strContent e)
1137+ "keycap" -> return (str $ strContent e)
11291138 "keycombo" -> keycombo <$>
11301139 mapM parseInline (elContent e)
11311140 "menuchoice" -> menuchoice <$>
@@ -1137,17 +1146,17 @@ parseInline (Elem e) =
11371146 let title = case attrValue "endterm" e of
11381147 "" -> maybe "???" xrefTitleByElem
11391148 (findElementById linkend content)
1140− endterm -> maybe "???" (T.pack . strContent)
1149+ endterm -> maybe "???" strContent
11411150 (findElementById endterm content)
11421151 return $ link ("#" <> linkend) "" (text title)
1143− "email" -> return $ link ("mailto:" <> T.pack (strContent e)) ""
1144− $ str $ T.pack $ strContent e
1145− "uri" -> return $ link (T.pack $ strContent e) "" $ str $ T.pack $ strContent e
1152+ "email" -> return $ link ("mailto:" <> strContent e) ""
1153+ $ str $ strContent e
1154+ "uri" -> return $ link (strContent e) "" $ str $ strContent e
11461155 "ulink" -> innerInlines (link (attrValue "url" e) "")
11471156 "link" -> do
11481157 ils <- innerInlines id
11491158 let href = case findAttr (QName "href" (Just "http://www.w3.org/1999/xlink") Nothing) e of
1150− Just h -> T.pack h
1159+ Just h -> h
11511160 _ -> "#" <> attrValue "linkend" e
11521161 let ils' = if ils == mempty then str href else ils
11531162 let attr = (attrValue "id" e, T.words $ attrValue "role" e, [])
@@ -1163,12 +1172,15 @@ parseInline (Elem e) =
11631172 "title" -> return mempty
11641173 "affiliation" -> skip
11651174 -- Note: this isn't a real docbook tag; it's what we convert
1166− -- <?asciidor-br?> to in handleInstructions, above. A kludge to
1167− -- work around xml-light's inability to parse an instruction.
1168− "br" -> return linebreak
1175+ -- <?asciidor-br?> to in handleInstructions, above.
1176+ "pi-asciidoc-br" -> return linebreak
11691177 _ -> skip >> innerInlines id
11701178 where skip = do
1171− lift $ report $ IgnoredElement $ T.pack $ qName (elName e)
1179+ let qn = qName $ elName e
1180+ let name = if "pi-" `T.isPrefixOf` qn
1181+ then "<?" <> qn <> "?>"
1182+ else qn
1183+ lift $ report $ IgnoredElement name
11721184 return mempty
11731185
11741186 innerInlines f = extractSpaces f . mconcat <$>
@@ -1177,7 +1189,7 @@ parseInline (Elem e) =
11771189 let classes' = case attrValue "language" e of
11781190 "" -> []
11791191 l -> [l]
1180− return $ codeWith (attrValue "id" e,classes',[]) $ T.pack $ strContentRecursive e
1192+ return $ codeWith (attrValue "id" e,classes',[]) $ strContentRecursive e
11811193 simpleList = mconcat . intersperse (str "," <> space) <$> mapM getInlines
11821194 (filterChildren (named "member") e)
11831195 segmentedList = do
@@ -1218,10 +1230,10 @@ parseInline (Elem e) =
12181230 "sect5" -> descendantContent "title" el
12191231 "cmdsynopsis" -> descendantContent "command" el
12201232 "funcsynopsis" -> descendantContent "function" el
1221− _ -> T.pack $ qName (elName el) ++ "_title"
1233+ _ -> qName (elName el) <> "_title"
12221234 where
12231235 xrefLabel = attrValue "xreflabel" el
1224− descendantContent name = maybe "???" (T.pack . strContent)
1236+ descendantContent name = maybe "???" strContent
12251237 . filterElementName (\n -> qName n == name)
12261238
12271239-- | Extract a math equation from an element
@@ -1241,8 +1253,9 @@ equation e constructor =
12411253 where
12421254 mathMLEquations :: [Text]
12431255 mathMLEquations = map writeTeX $ rights $ readMath
1244− (\x -> qName (elName x) == "math" && qPrefix (elName x) == Just "mml")
1245− (readMathML . T.pack . showElement)
1256+ (\x -> qName (elName x) == "math" &&
1257+ qURI (elName x) == Just "http://www.w3.org/1998/Math/MathML")
1258+ (readMathML . showElement)
12461259
12471260 latexEquations :: [Text]
12481261 latexEquations = readMath (\x -> qName (elName x) == "mathphrase")
@@ -1256,8 +1269,8 @@ equation e constructor =
12561269-- | Get the actual text stored in a CData block. 'showContent'
12571270-- returns the text still surrounded by the [[CDATA]] tags.
12581271showVerbatimCData :: Content -> Text
1259−showVerbatimCData (Text (CData _ d _)) = T.pack d
1260−showVerbatimCData c = T.pack $ showContent c
1272+showVerbatimCData (Text (CData _ d _)) = d
1273+showVerbatimCData c = showContent c
12611274
12621275
12631276-- | Set the prefix of a name to 'Nothing'
src/Text/Pandoc/Readers/Docx.hs+59−42
@@ -61,13 +61,14 @@ module Text.Pandoc.Readers.Docx
6161import Codec.Archive.Zip
6262import Control.Monad.Reader
6363import Control.Monad.State.Strict
64+import Data.Bifunctor (bimap, first)
6465import qualified Data.ByteString.Lazy as B
6566import Data.Default (Default)
66−import Data.List (delete, intersect)
67+import Data.List (delete, intersect, foldl')
6768import Data.Char (isSpace)
6869import qualified Data.Map as M
6970import qualified Data.Text as T
70−import Data.Maybe (isJust, fromMaybe)
71+import Data.Maybe (catMaybes, isJust, fromMaybe)
7172import Data.Sequence (ViewL (..), viewl)
7273import qualified Data.Sequence as Seq
7374import qualified Data.Set as Set
@@ -85,6 +86,7 @@ import Text.Pandoc.Class.PandocMonad (PandocMonad)
8586import qualified Text.Pandoc.Class.PandocMonad as P
8687import Text.Pandoc.Error
8788import Text.Pandoc.Logging
89+import Data.List.NonEmpty (nonEmpty)
8890
8991readDocx :: PandocMonad m
9092 => ReaderOptions
@@ -112,6 +114,7 @@ data DState = DState { docxAnchorMap :: M.Map T.Text T.Text
112114 -- restarting
113115 , docxListState :: M.Map (T.Text, T.Text) Integer
114116 , docxPrevPara :: Inlines
117+ , docxTableCaptions :: [Blocks]
115118 }
116119
117120instance Default DState where
@@ -122,6 +125,7 @@ instance Default DState where
122125 , docxDropCap = mempty
123126 , docxListState = M.empty
124127 , docxPrevPara = mempty
128+ , docxTableCaptions = []
125129 }
126130
127131data DEnv = DEnv { docxOptions :: ReaderOptions
@@ -490,15 +494,32 @@ singleParaToPlain blks
490494 singleton $ Plain ils
491495singleParaToPlain blks = blks
492496
493−cellToBlocks :: PandocMonad m => Docx.Cell -> DocxContext m Blocks
494−cellToBlocks (Docx.Cell bps) = do
497+cellToCell :: PandocMonad m => RowSpan -> Docx.Cell -> DocxContext m Pandoc.Cell
498+cellToCell rowSpan (Docx.Cell gridSpan _ bps) = do
495499 blks <- smushBlocks <$> mapM bodyPartToBlocks bps
496− return $ fromList $ blocksToDefinitions $ blocksToBullets $ toList blks
500+ let blks' = singleParaToPlain $ fromList $ blocksToDefinitions $ blocksToBullets $ toList blks
501+ return (cell AlignDefault rowSpan (ColSpan (fromIntegral gridSpan)) blks')
502+
503+rowsToRows :: PandocMonad m => [Docx.Row] -> DocxContext m [Pandoc.Row]
504+rowsToRows rows = do
505+ let rowspans = (fmap . fmap) (first RowSpan) (Docx.rowsToRowspans rows)
506+ cells <- traverse (traverse (uncurry cellToCell)) rowspans
507+ return (fmap (Pandoc.Row nullAttr) cells)
508+
509+splitHeaderRows :: Bool -> [Docx.Row] -> ([Docx.Row], [Docx.Row])
510+splitHeaderRows hasFirstRowFormatting rs = bimap reverse reverse $ fst
511+ $ if hasFirstRowFormatting
512+ then foldl' f ((take 1 rs, []), True) (drop 1 rs)
513+ else foldl' f (([], []), False) rs
514+ where
515+ f ((headerRows, bodyRows), previousRowWasHeader) r@(Docx.Row h cs)
516+ | h == HasTblHeader || (previousRowWasHeader && any isContinuationCell cs)
517+ = ((r : headerRows, bodyRows), True)
518+ | otherwise
519+ = ((headerRows, r : bodyRows), False)
520+
521+ isContinuationCell (Docx.Cell _ vm _) = vm == Docx.Continue
497522
498−rowToBlocksList :: PandocMonad m => Docx.Row -> DocxContext m [Blocks]
499−rowToBlocksList (Docx.Row cells) = do
500− blksList <- mapM cellToBlocks cells
501− return $ map singleParaToPlain blksList
502523
503524-- like trimInlines, but also take out linebreaks
504525trimSps :: Inlines -> Inlines
@@ -545,6 +566,11 @@ normalizeToClassName = T.map go . fromStyleName
545566 where go c | isSpace c = '-'
546567 | otherwise = c
547568
569+bodyPartToTableCaption :: PandocMonad m => BodyPart -> DocxContext m (Maybe Blocks)
570+bodyPartToTableCaption (TblCaption pPr parparts) =
571+ Just <$> bodyPartToBlocks (Paragraph pPr parparts)
572+bodyPartToTableCaption _ = pure Nothing
573+
548574bodyPartToBlocks :: PandocMonad m => BodyPart -> DocxContext m Blocks
549575bodyPartToBlocks (Paragraph pPr parparts)
550576 | Just True <- pBidi pPr = do
@@ -636,54 +662,43 @@ bodyPartToBlocks (ListItem pPr _ _ _ parparts) =
636662 let pPr' = pPr {pStyle = constructBogusParStyleData "list-paragraph": pStyle pPr}
637663 in
638664 bodyPartToBlocks $ Paragraph pPr' parparts
665+bodyPartToBlocks (TblCaption _ _) =
666+ return $ para mempty -- collected separately
639667bodyPartToBlocks (Tbl _ _ _ []) =
640668 return $ para mempty
641−bodyPartToBlocks (Tbl cap _ look parts@(r:rs)) = do
642− let cap' = simpleCaption $ plain $ text cap
643− (hdr, rows) = case firstRowFormatting look of
644− True | null rs -> (Nothing, [r])
645− | otherwise -> (Just r, rs)
646− False -> (Nothing, r:rs)
647−
648− cells <- mapM rowToBlocksList rows
669+bodyPartToBlocks (Tbl cap grid look parts) = do
670+ captions <- gets docxTableCaptions
671+ fullCaption <- case captions of
672+ c : cs -> do
673+ modify (\s -> s { docxTableCaptions = cs })
674+ return c
675+ [] -> return $ if T.null cap then mempty else plain (text cap)
676+ let shortCaption = if T.null cap then Nothing else Just (toList (text cap))
677+ cap' = caption shortCaption fullCaption
678+ (hdr, rows) = splitHeaderRows (firstRowFormatting look) parts
649679
650680 let width = maybe 0 maximum $ nonEmpty $ map rowLength parts
651− -- Data.List.NonEmpty is not available with ghc 7.10 so we roll out
652− -- our own, see
653− -- https://github.com/jgm/pandoc/pull/4361#issuecomment-365416155
654− nonEmpty [] = Nothing
655− nonEmpty l = Just l
656681 rowLength :: Docx.Row -> Int
657− rowLength (Docx.Row c) = length c
682+ rowLength (Docx.Row _ c) = sum (fmap (\(Docx.Cell gridSpan _ _) -> fromIntegral gridSpan) c)
658683
659− let toRow = Pandoc.Row nullAttr . map simpleCell
660− toHeaderRow l = [toRow l | not (null l)]
684+ headerCells <- rowsToRows hdr
685+ bodyCells <- rowsToRows rows
661686
662− -- pad cells. New Text.Pandoc.Builder will do that for us,
663− -- so this is for compatibility while we switch over.
664− let cells' = map (\row -> toRow $ take width (row ++ repeat mempty)) cells
665−
666− hdrCells <- case hdr of
667− Just r' -> toHeaderRow <$> rowToBlocksList r'
668− Nothing -> return []
669−
670− -- The two following variables (horizontal column alignment and
671− -- relative column widths) go to the default at the
672− -- moment. Width information is in the TblGrid field of the Tbl,
673− -- so should be possible. Alignment might be more difficult,
674− -- since there doesn't seem to be a column entity in docx.
687+ -- Horizontal column alignment goes to the default at the moment. Getting
688+ -- it might be difficult, since there doesn't seem to be a column entity
689+ -- in docx.
675690 let alignments = replicate width AlignDefault
676− widths = replicate width ColWidthDefault
691+ totalWidth = sum grid
692+ widths = (\w -> ColWidth (fromInteger w / fromInteger totalWidth)) <$> grid
677693
678694 return $ table cap'
679695 (zip alignments widths)
680− (TableHead nullAttr hdrCells)
681− [TableBody nullAttr 0 [] cells']
696+ (TableHead nullAttr headerCells)
697+ [TableBody nullAttr 0 [] bodyCells]
682698 (TableFoot nullAttr [])
683699bodyPartToBlocks (OMathPara e) =
684700 return $ para $ displayMath (writeTeX e)
685701
686−
687702-- replace targets with generated anchors.
688703rewriteLink' :: PandocMonad m => Inline -> DocxContext m Inline
689704rewriteLink' l@(Link attr ils (T.uncons -> Just ('#',target), title)) = do
@@ -719,6 +734,8 @@ bodyToOutput :: PandocMonad m => Body -> DocxContext m (Meta, [Block])
719734bodyToOutput (Body bps) = do
720735 let (metabps, blkbps) = sepBodyParts bps
721736 meta <- bodyPartsToMeta metabps
737+ captions <- catMaybes <$> mapM bodyPartToTableCaption blkbps
738+ modify (\s -> s { docxTableCaptions = captions })
722739 blks <- smushBlocks <$> mapM bodyPartToBlocks blkbps
723740 blks' <- rewriteLinks $ blocksToDefinitions $ blocksToBullets $ toList blks
724741 blks'' <- removeOrphanAnchors blks'
src/Text/Pandoc/Readers/Docx/Combine.hs+6−6
@@ -2,7 +2,7 @@
22{- |
33 Module : Text.Pandoc.Readers.Docx.Combine
44 Copyright : © 2014-2020 Jesse Rosenthal <jrosenthal@jhu.edu>,
5− 2014-2020 John MacFarlane <jgm@berkeley.edu>,
5+ 2014-2021 John MacFarlane <jgm@berkeley.edu>,
66 2020 Nikolay Yakimov <root@livid.pp.ru>
77 License : GNU GPL, version 2 or above
88
@@ -61,7 +61,7 @@ import Data.List
6161import Data.Bifunctor
6262import Data.Sequence ( ViewL (..), ViewR (..), viewl, viewr, spanr, spanl
6363 , (><), (|>) )
64−import Text.Pandoc.Builder
64+import Text.Pandoc.Builder as B
6565
6666data Modifier a = Modifier (a -> a)
6767 | AttrModifier (Attr -> a -> a) Attr
@@ -116,12 +116,12 @@ ilModifierAndInnards ils = case viewl $ unMany ils of
116116
117117inlinesL :: Inlines -> (Inlines, Inlines)
118118inlinesL ils = case viewl $ unMany ils of
119− (s :< sq) -> (singleton s, Many sq)
119+ (s :< sq) -> (B.singleton s, Many sq)
120120 _ -> (mempty, ils)
121121
122122inlinesR :: Inlines -> (Inlines, Inlines)
123123inlinesR ils = case viewr $ unMany ils of
124− (sq :> s) -> (Many sq, singleton s)
124+ (sq :> s) -> (Many sq, B.singleton s)
125125 _ -> (ils, mempty)
126126
127127combineInlines :: Inlines -> Inlines -> Inlines
@@ -182,7 +182,7 @@ isAttrModifier _ = False
182182
183183smushInlines :: [Inlines] -> Inlines
184184smushInlines xs = combineInlines xs' mempty
185− where xs' = foldl combineInlines mempty xs
185+ where xs' = foldl' combineInlines mempty xs
186186
187187smushBlocks :: [Blocks] -> Blocks
188−smushBlocks xs = foldl combineBlocks mempty xs
188+smushBlocks xs = foldl' combineBlocks mempty xs
src/Text/Pandoc/Readers/Docx/Parse.hs+214−120
@@ -33,7 +33,9 @@ module Text.Pandoc.Readers.Docx.Parse ( Docx(..)
3333 , ParStyle
3434 , CharStyle(cStyleData)
3535 , Row(..)
36+ , TblHeader(..)
3637 , Cell(..)
38+ , VMerge(..)
3739 , TrackedChange(..)
3840 , ChangeType(..)
3941 , ChangeInfo(..)
@@ -50,6 +52,7 @@ module Text.Pandoc.Readers.Docx.Parse ( Docx(..)
5052 , pHeading
5153 , constructBogusParStyleData
5254 , leftBiasedMergeRunStyle
55+ , rowsToRowspans
5356 ) where
5457import Text.Pandoc.Readers.Docx.Parse.Styles
5558import Codec.Archive.Zip
@@ -63,6 +66,7 @@ import Data.Char (chr, ord, readLitChar)
6366import Data.List
6467import qualified Data.Map as M
6568import qualified Data.Text as T
69+import Data.Text (Text)
6670import Data.Maybe
6771import System.FilePath
6872import Text.Pandoc.Readers.Docx.Util
@@ -72,8 +76,7 @@ import qualified Text.Pandoc.UTF8 as UTF8
7276import Text.TeXMath (Exp)
7377import Text.TeXMath.Readers.OMML (readOMML)
7478import Text.TeXMath.Unicode.Fonts (Font (..), getUnicode, textToFont)
75−import Text.XML.Light
76−import qualified Text.XML.Light.Cursor as XMLC
79+import Text.Pandoc.XML.Light
7780
7881data ReaderEnv = ReaderEnv { envNotes :: Notes
7982 , envComments :: Comments
@@ -127,37 +130,23 @@ mapD f xs =
127130 in
128131 concatMapM handler xs
129132
130−unwrap :: NameSpaces -> Content -> [Content]
131−unwrap ns (Elem element)
133+unwrapElement :: NameSpaces -> Element -> [Element]
134+unwrapElement ns element
132135 | isElem ns "w" "sdt" element
133136 , Just sdtContent <- findChildByName ns "w" "sdtContent" element
134− = concatMap (unwrap ns . Elem) (elChildren sdtContent)
137+ = concatMap (unwrapElement ns) (elChildren sdtContent)
135138 | isElem ns "w" "smartTag" element
136− = concatMap (unwrap ns . Elem) (elChildren element)
137−unwrap _ content = [content]
139+ = concatMap (unwrapElement ns) (elChildren element)
140+ | otherwise
141+ = [element{ elContent = concatMap (unwrapContent ns) (elContent element) }]
138142
139−unwrapChild :: NameSpaces -> Content -> Content
140−unwrapChild ns (Elem element) =
141− Elem $ element { elContent = concatMap (unwrap ns) (elContent element) }
142−unwrapChild _ content = content
143+unwrapContent :: NameSpaces -> Content -> [Content]
144+unwrapContent ns (Elem element) = map Elem $ unwrapElement ns element
145+unwrapContent _ content = [content]
143146
144−walkDocument' :: NameSpaces -> XMLC.Cursor -> XMLC.Cursor
145−walkDocument' ns cur =
146− let modifiedCur = XMLC.modifyContent (unwrapChild ns) cur
147− in
148− case XMLC.nextDF modifiedCur of
149− Just cur' -> walkDocument' ns cur'
150− Nothing -> XMLC.root modifiedCur
151−
152−walkDocument :: NameSpaces -> Element -> Maybe Element
147+walkDocument :: NameSpaces -> Element -> Element
153148walkDocument ns element =
154− let cur = XMLC.fromContent (Elem element)
155− cur' = walkDocument' ns cur
156− in
157− case XMLC.toTree cur' of
158− Elem element' -> Just element'
159− _ -> Nothing
160−
149+ element{ elContent = concatMap (unwrapContent ns) (elContent element) }
161150
162151newtype Docx = Docx Document
163152 deriving Show
@@ -239,6 +228,7 @@ defaultParagraphStyle = ParagraphStyle { pStyle = []
239228data BodyPart = Paragraph ParagraphStyle [ParPart]
240229 | ListItem ParagraphStyle T.Text T.Text (Maybe Level) [ParPart]
241230 | Tbl T.Text TblGrid TblLook [Row]
231+ | TblCaption ParagraphStyle [ParPart]
242232 | OMathPara [Exp]
243233 deriving Show
244234
@@ -250,12 +240,61 @@ newtype TblLook = TblLook {firstRowFormatting::Bool}
250240defaultTblLook :: TblLook
251241defaultTblLook = TblLook{firstRowFormatting = False}
252242
253−newtype Row = Row [Cell]
254− deriving Show
243+data Row = Row TblHeader [Cell] deriving Show
255244
256−newtype Cell = Cell [BodyPart]
245+data TblHeader = HasTblHeader | NoTblHeader deriving (Show, Eq)
246+
247+data Cell = Cell GridSpan VMerge [BodyPart]
257248 deriving Show
258249
250+type GridSpan = Integer
251+
252+data VMerge = Continue
253+ -- ^ This cell should be merged with the one above it
254+ | Restart
255+ -- ^ This cell should not be merged with the one above it
256+ deriving (Show, Eq)
257+
258+rowsToRowspans :: [Row] -> [[(Int, Cell)]]
259+rowsToRowspans rows = let
260+ removeMergedCells = fmap (filter (\(_, Cell _ vmerge _) -> vmerge == Restart))
261+ in removeMergedCells (foldr f [] rows)
262+ where
263+ f :: Row -> [[(Int, Cell)]] -> [[(Int, Cell)]]
264+ f (Row _ cells) acc = let
265+ spans = g cells Nothing (listToMaybe acc)
266+ in spans : acc
267+
268+ g ::
269+ -- | The current row
270+ [Cell] ->
271+ -- | Number of columns left below
272+ Maybe Integer ->
273+ -- | (rowspan so far, cell) for the row below this one
274+ Maybe [(Int, Cell)] ->
275+ -- | (rowspan so far, cell) for this row
276+ [(Int, Cell)]
277+ g cells _ Nothing = zip (repeat 1) cells
278+ g cells columnsLeftBelow (Just rowBelow) =
279+ case cells of
280+ [] -> []
281+ thisCell@(Cell thisGridSpan _ _) : restOfRow -> case rowBelow of
282+ [] -> zip (repeat 1) cells
283+ (spanSoFarBelow, Cell gridSpanBelow vmerge _) : _ ->
284+ let spanSoFar = case vmerge of
285+ Restart -> 1
286+ Continue -> 1 + spanSoFarBelow
287+ columnsToDrop = thisGridSpan + (gridSpanBelow - fromMaybe gridSpanBelow columnsLeftBelow)
288+ (newColumnsLeftBelow, restOfRowBelow) = dropColumns columnsToDrop rowBelow
289+ in (spanSoFar, thisCell) : g restOfRow (Just newColumnsLeftBelow) (Just restOfRowBelow)
290+
291+ dropColumns :: Integer -> [(a, Cell)] -> (Integer, [(a, Cell)])
292+ dropColumns n [] = (n, [])
293+ dropColumns n cells@((_, Cell gridSpan _ _) : otherCells) =
294+ if n < gridSpan
295+ then (gridSpan - n, cells)
296+ else dropColumns (n - gridSpan) otherCells
297+
259298leftBiasedMergeRunStyle :: RunStyle -> RunStyle -> RunStyle
260299leftBiasedMergeRunStyle a b = RunStyle
261300 { isBold = isBold a <|> isBold b
@@ -343,10 +382,16 @@ archiveToDocxWithWarnings archive = do
343382 Right doc -> Right (Docx doc, stateWarnings st)
344383 Left e -> Left e
345384
385+parseXMLFromEntry :: Entry -> Maybe Element
386+parseXMLFromEntry entry =
387+ case parseXMLElement (UTF8.toTextLazy (fromEntry entry)) of
388+ Left _ -> Nothing
389+ Right el -> Just el
390+
346391getDocumentXmlPath :: Archive -> Maybe FilePath
347392getDocumentXmlPath zf = do
348393 entry <- findEntryByPath "_rels/.rels" zf
349− relsElem <- (parseXMLDoc . UTF8.toStringLazy . fromEntry) entry
394+ relsElem <- parseXMLFromEntry entry
350395 let rels = filterChildrenName (\n -> qName n == "Relationship") relsElem
351396 rel <- find (\e -> findAttr (QName "Type" Nothing Nothing) e ==
352397 Just "http://schemas.openxmlformats.org/officeDocument/2006/relationships/officeDocument")
@@ -354,18 +399,18 @@ getDocumentXmlPath zf = do
354399 fp <- findAttr (QName "Target" Nothing Nothing) rel
355400 -- sometimes there will be a leading slash, which windows seems to
356401 -- have trouble with.
357− return $ case fp of
402+ return $ case T.unpack fp of
358403 '/' : fp' -> fp'
359− _ -> fp
404+ fp' -> fp'
360405
361406archiveToDocument :: Archive -> D Document
362407archiveToDocument zf = do
363408 docPath <- asks envDocXmlPath
364409 entry <- maybeToD $ findEntryByPath docPath zf
365− docElem <- maybeToD $ (parseXMLDoc . UTF8.toStringLazy . fromEntry) entry
410+ docElem <- maybeToD $ parseXMLFromEntry entry
366411 let namespaces = elemToNameSpaces docElem
367412 bodyElem <- maybeToD $ findChildByName namespaces "w" "body" docElem
368− let bodyElem' = fromMaybe bodyElem (walkDocument namespaces bodyElem)
413+ let bodyElem' = walkDocument namespaces bodyElem
369414 body <- elemToBody namespaces bodyElem'
370415 return $ Document namespaces body
371416
@@ -401,23 +446,24 @@ constructBogusParStyleData stName = ParStyle
401446archiveToNotes :: Archive -> Notes
402447archiveToNotes zf =
403448 let fnElem = findEntryByPath "word/footnotes.xml" zf
404− >>= (parseXMLDoc . UTF8.toStringLazy . fromEntry)
449+ >>= parseXMLFromEntry
405450 enElem = findEntryByPath "word/endnotes.xml" zf
406− >>= (parseXMLDoc . UTF8.toStringLazy . fromEntry)
407− fn_namespaces = maybe [] elemToNameSpaces fnElem
408− en_namespaces = maybe [] elemToNameSpaces enElem
409− ns = unionBy (\x y -> fst x == fst y) fn_namespaces en_namespaces
410− fn = fnElem >>= walkDocument ns >>= elemToNotes ns "footnote"
411− en = enElem >>= walkDocument ns >>= elemToNotes ns "endnote"
451+ >>= parseXMLFromEntry
452+ fn_namespaces = maybe mempty elemToNameSpaces fnElem
453+ en_namespaces = maybe mempty elemToNameSpaces enElem
454+ ns = M.union fn_namespaces en_namespaces
455+ fn = fnElem >>= elemToNotes ns "footnote" . walkDocument ns
456+ en = enElem >>= elemToNotes ns "endnote" . walkDocument ns
412457 in
413458 Notes ns fn en
414459
415460archiveToComments :: Archive -> Comments
416461archiveToComments zf =
417462 let cmtsElem = findEntryByPath "word/comments.xml" zf
418− >>= (parseXMLDoc . UTF8.toStringLazy . fromEntry)
419− cmts_namespaces = maybe [] elemToNameSpaces cmtsElem
420− cmts = elemToComments cmts_namespaces <$> (cmtsElem >>= walkDocument cmts_namespaces)
463+ >>= parseXMLFromEntry
464+ cmts_namespaces = maybe mempty elemToNameSpaces cmtsElem
465+ cmts = elemToComments cmts_namespaces . walkDocument cmts_namespaces <$>
466+ cmtsElem
421467 in
422468 case cmts of
423469 Just c -> Comments cmts_namespaces c
@@ -433,20 +479,26 @@ filePathToRelType path docXmlPath =
433479 then Just InDocument
434480 else Nothing
435481
436−relElemToRelationship :: DocumentLocation -> Element -> Maybe Relationship
437−relElemToRelationship relType element | qName (elName element) == "Relationship" =
482+relElemToRelationship :: FilePath -> DocumentLocation -> Element
483+ -> Maybe Relationship
484+relElemToRelationship fp relType element | qName (elName element) == "Relationship" =
438485 do
439− relId <- findAttrText (QName "Id" Nothing Nothing) element
440− target <- findAttrText (QName "Target" Nothing Nothing) element
441− return $ Relationship relType relId target
442−relElemToRelationship _ _ = Nothing
486+ relId <- findAttr (QName "Id" Nothing Nothing) element
487+ target <- findAttr (QName "Target" Nothing Nothing) element
488+ -- target may be relative (media/image1.jpeg) or absolute
489+ -- (/word/media/image1.jpeg); we need to relativize it (see #7374)
490+ let frontOfFp = T.pack $ takeWhile (/= '_') fp
491+ let target' = fromMaybe target $
492+ T.stripPrefix frontOfFp $ T.dropWhile (== '/') target
493+ return $ Relationship relType relId target'
494+relElemToRelationship _ _ _ = Nothing
443495
444496filePathToRelationships :: Archive -> FilePath -> FilePath -> [Relationship]
445497filePathToRelationships ar docXmlPath fp
446498 | Just relType <- filePathToRelType fp docXmlPath
447499 , Just entry <- findEntryByPath fp ar
448− , Just relElems <- (parseXMLDoc . UTF8.toStringLazy . fromEntry) entry =
449− mapMaybe (relElemToRelationship relType) $ elChildren relElems
500+ , Just relElems <- parseXMLFromEntry entry =
501+ mapMaybe (relElemToRelationship fp relType) $ elChildren relElems
450502filePathToRelationships _ _ _ = []
451503
452504archiveToRelationships :: Archive -> FilePath -> [Relationship]
@@ -478,10 +530,10 @@ lookupLevel numId ilvl (Numbering _ numbs absNumbs) = do
478530loElemToLevelOverride :: NameSpaces -> Element -> Maybe LevelOverride
479531loElemToLevelOverride ns element
480532 | isElem ns "w" "lvlOverride" element = do
481− ilvl <- findAttrTextByName ns "w" "ilvl" element
533+ ilvl <- findAttrByName ns "w" "ilvl" element
482534 let startOverride = findChildByName ns "w" "startOverride" element
483535 >>= findAttrByName ns "w" "val"
484− >>= (\s -> listToMaybe (map fst (reads s :: [(Integer, String)])))
536+ >>= stringToInteger
485537 lvl = findChildByName ns "w" "lvl" element
486538 >>= levelElemToLevel ns
487539 return $ LevelOverride ilvl startOverride lvl
@@ -490,9 +542,9 @@ loElemToLevelOverride _ _ = Nothing
490542numElemToNum :: NameSpaces -> Element -> Maybe Numb
491543numElemToNum ns element
492544 | isElem ns "w" "num" element = do
493− numId <- findAttrTextByName ns "w" "numId" element
545+ numId <- findAttrByName ns "w" "numId" element
494546 absNumId <- findChildByName ns "w" "abstractNumId" element
495− >>= findAttrTextByName ns "w" "val"
547+ >>= findAttrByName ns "w" "val"
496548 let lvlOverrides = mapMaybe
497549 (loElemToLevelOverride ns)
498550 (findChildrenByName ns "w" "lvlOverride" element)
@@ -502,7 +554,7 @@ numElemToNum _ _ = Nothing
502554absNumElemToAbsNum :: NameSpaces -> Element -> Maybe AbstractNumb
503555absNumElemToAbsNum ns element
504556 | isElem ns "w" "abstractNum" element = do
505− absNumId <- findAttrTextByName ns "w" "abstractNumId" element
557+ absNumId <- findAttrByName ns "w" "abstractNumId" element
506558 let levelElems = findChildrenByName ns "w" "lvl" element
507559 levels = mapMaybe (levelElemToLevel ns) levelElems
508560 return $ AbstractNumb absNumId levels
@@ -511,23 +563,23 @@ absNumElemToAbsNum _ _ = Nothing
511563levelElemToLevel :: NameSpaces -> Element -> Maybe Level
512564levelElemToLevel ns element
513565 | isElem ns "w" "lvl" element = do
514− ilvl <- findAttrTextByName ns "w" "ilvl" element
566+ ilvl <- findAttrByName ns "w" "ilvl" element
515567 fmt <- findChildByName ns "w" "numFmt" element
516− >>= findAttrTextByName ns "w" "val"
568+ >>= findAttrByName ns "w" "val"
517569 txt <- findChildByName ns "w" "lvlText" element
518− >>= findAttrTextByName ns "w" "val"
570+ >>= findAttrByName ns "w" "val"
519571 let start = findChildByName ns "w" "start" element
520572 >>= findAttrByName ns "w" "val"
521− >>= (\s -> listToMaybe (map fst (reads s :: [(Integer, String)])))
573+ >>= stringToInteger
522574 return (Level ilvl fmt txt start)
523575levelElemToLevel _ _ = Nothing
524576
525577archiveToNumbering' :: Archive -> Maybe Numbering
526578archiveToNumbering' zf =
527579 case findEntryByPath "word/numbering.xml" zf of
528− Nothing -> Just $ Numbering [] [] []
580+ Nothing -> Just $ Numbering mempty [] []
529581 Just entry -> do
530− numberingElem <- (parseXMLDoc . UTF8.toStringLazy . fromEntry) entry
582+ numberingElem <- parseXMLFromEntry entry
531583 let namespaces = elemToNameSpaces numberingElem
532584 numElems = findChildrenByName namespaces "w" "num" numberingElem
533585 absNumElems = findChildrenByName namespaces "w" "abstractNum" numberingElem
@@ -537,13 +589,13 @@ archiveToNumbering' zf =
537589
538590archiveToNumbering :: Archive -> Numbering
539591archiveToNumbering archive =
540− fromMaybe (Numbering [] [] []) (archiveToNumbering' archive)
592+ fromMaybe (Numbering mempty [] []) (archiveToNumbering' archive)
541593
542−elemToNotes :: NameSpaces -> String -> Element -> Maybe (M.Map T.Text Element)
594+elemToNotes :: NameSpaces -> Text -> Element -> Maybe (M.Map T.Text Element)
543595elemToNotes ns notetype element
544596 | isElem ns "w" (notetype <> "s") element =
545597 let pairs = mapMaybe
546− (\e -> findAttrTextByName ns "w" "id" e >>=
598+ (\e -> findAttrByName ns "w" "id" e >>=
547599 (\a -> Just (a, e)))
548600 (findChildrenByName ns "w" notetype element)
549601 in
@@ -555,7 +607,7 @@ elemToComments :: NameSpaces -> Element -> M.Map T.Text Element
555607elemToComments ns element
556608 | isElem ns "w" "comments" element =
557609 let pairs = mapMaybe
558− (\e -> findAttrTextByName ns "w" "id" e >>=
610+ (\e -> findAttrByName ns "w" "id" e >>=
559611 (\a -> Just (a, e)))
560612 (findChildrenByName ns "w" "comment" element)
561613 in
@@ -570,7 +622,7 @@ elemToTblGrid :: NameSpaces -> Element -> D TblGrid
570622elemToTblGrid ns element | isElem ns "w" "tblGrid" element =
571623 let cols = findChildrenByName ns "w" "gridCol" element
572624 in
573− mapD (\e -> maybeToD (findAttrByName ns "w" "val" e >>= stringToInteger))
625+ mapD (\e -> maybeToD (findAttrByName ns "w" "w" e >>= stringToInteger))
574626 cols
575627elemToTblGrid _ _ = throwError WrongElem
576628
@@ -594,14 +646,31 @@ elemToRow ns element | isElem ns "w" "tr" element =
594646 do
595647 let cellElems = findChildrenByName ns "w" "tc" element
596648 cells <- mapD (elemToCell ns) cellElems
597− return $ Row cells
649+ let hasTblHeader = maybe NoTblHeader (const HasTblHeader)
650+ (findChildByName ns "w" "trPr" element
651+ >>= findChildByName ns "w" "tblHeader")
652+ return $ Row hasTblHeader cells
598653elemToRow _ _ = throwError WrongElem
599654
600655elemToCell :: NameSpaces -> Element -> D Cell
601656elemToCell ns element | isElem ns "w" "tc" element =
602657 do
658+ let properties = findChildByName ns "w" "tcPr" element
659+ let gridSpan = properties
660+ >>= findChildByName ns "w" "gridSpan"
661+ >>= findAttrByName ns "w" "val"
662+ >>= stringToInteger
663+ let vMerge = case properties >>= findChildByName ns "w" "vMerge" of
664+ Nothing -> Restart
665+ Just e ->
666+ fromMaybe Continue $ do
667+ s <- findAttrByName ns "w" "val" e
668+ case s of
669+ "continue" -> Just Continue
670+ "restart" -> Just Restart
671+ _ -> Nothing
603672 cellContents <- mapD (elemToBodyPart ns) (elChildren element)
604− return $ Cell cellContents
673+ return $ Cell (fromMaybe 1 gridSpan) vMerge cellContents
605674elemToCell _ _ = throwError WrongElem
606675
607676elemToParIndentation :: NameSpaces -> Element -> Maybe ParIndentation
@@ -615,12 +684,12 @@ elemToParIndentation ns element | isElem ns "w" "ind" element =
615684 stringToInteger
616685 , hangingParIndent =
617686 findAttrByName ns "w" "hanging" element >>=
618− stringToInteger}
687+ stringToInteger }
619688elemToParIndentation _ _ = Nothing
620689
621−testBitMask :: String -> Int -> Bool
690+testBitMask :: Text -> Int -> Bool
622691testBitMask bitMaskS n =
623− case (reads ("0x" ++ bitMaskS) :: [(Int, String)]) of
692+ case (reads ("0x" ++ T.unpack bitMaskS) :: [(Int, String)]) of
624693 [] -> False
625694 ((n', _) : _) -> (n' .|. n) /= 0
626695
@@ -633,10 +702,9 @@ pNumInfo = getParStyleField numInfo . pStyle
633702elemToBodyPart :: NameSpaces -> Element -> D BodyPart
634703elemToBodyPart ns element
635704 | isElem ns "w" "p" element
636− , (c:_) <- findChildrenByName ns "m" "oMathPara" element =
637− do
638− expsLst <- eitherToD $ readOMML $ T.pack $ showElement c
639− return $ OMathPara expsLst
705+ , (c:_) <- findChildrenByName ns "m" "oMathPara" element = do
706+ expsLst <- eitherToD $ readOMML $ showElement c
707+ return $ OMathPara expsLst
640708elemToBodyPart ns element
641709 | isElem ns "w" "p" element
642710 , Just (numId, lvl) <- getNumInfo ns element = do
@@ -654,13 +722,31 @@ elemToBodyPart ns element
654722 Nothing | Just (numId, lvl) <- pNumInfo parstyle -> do
655723 levelInfo <- lookupLevel numId lvl <$> asks envNumbering
656724 return $ ListItem parstyle numId lvl levelInfo parparts
657− _ -> return $ Paragraph parstyle parparts
725+ _ -> let
726+ hasCaptionStyle = elem "Caption" (pStyleId <$> pStyle parstyle)
727+
728+ hasSimpleTableField = fromMaybe False $ do
729+ fldSimple <- findChildByName ns "w" "fldSimple" element
730+ instr <- findAttrByName ns "w" "instr" fldSimple
731+ pure ("Table" `elem` T.words instr)
732+
733+ hasComplexTableField = fromMaybe False $ do
734+ instrText <- findElementByName ns "w" "instrText" element
735+ pure ("Table" `elem` T.words (strContent instrText))
736+
737+ in if hasCaptionStyle && (hasSimpleTableField || hasComplexTableField)
738+ then return $ TblCaption parstyle parparts
739+ else return $ Paragraph parstyle parparts
740+
658741elemToBodyPart ns element
659742 | isElem ns "w" "tbl" element = do
660− let caption' = findChildByName ns "w" "tblPr" element
743+ let tblProperties = findChildByName ns "w" "tblPr" element
744+ caption = fromMaybe "" $ tblProperties
661745 >>= findChildByName ns "w" "tblCaption"
662− >>= findAttrTextByName ns "w" "val"
663− caption = fromMaybe "" caption'
746+ >>= findAttrByName ns "w" "val"
747+ description = fromMaybe "" $ tblProperties
748+ >>= findChildByName ns "w" "tblDescription"
749+ >>= findAttrByName ns "w" "val"
664750 grid' = case findChildByName ns "w" "tblGrid" element of
665751 Just g -> elemToTblGrid ns g
666752 Nothing -> return []
@@ -673,7 +759,7 @@ elemToBodyPart ns element
673759 grid <- grid'
674760 tblLook <- tblLook'
675761 rows <- mapD (elemToRow ns) (elChildren element)
676− return $ Tbl caption grid tblLook rows
762+ return $ Tbl (caption <> description) grid tblLook rows
677763elemToBodyPart _ _ = throwError WrongElem
678764
679765lookupRelationship :: DocumentLocation -> RelId -> [Relationship] -> Maybe Target
@@ -698,8 +784,8 @@ getTitleAndAlt :: NameSpaces -> Element -> (T.Text, T.Text)
698784getTitleAndAlt ns element =
699785 let mbDocPr = findChildByName ns "wp" "inline" element >>=
700786 findChildByName ns "wp" "docPr"
701− title = fromMaybe "" (mbDocPr >>= findAttrTextByName ns "" "title")
702− alt = fromMaybe "" (mbDocPr >>= findAttrTextByName ns "" "descr")
787+ title = fromMaybe "" (mbDocPr >>= findAttrByName ns "" "title")
788+ alt = fromMaybe "" (mbDocPr >>= findAttrByName ns "" "descr")
703789 in (title, alt)
704790
705791elemToParPart :: NameSpaces -> Element -> D ParPart
@@ -711,22 +797,29 @@ elemToParPart ns element
711797 = let (title, alt) = getTitleAndAlt ns drawingElem
712798 a_ns = "http://schemas.openxmlformats.org/drawingml/2006/main"
713799 drawing = findElement (QName "blip" (Just a_ns) (Just "a")) picElem
714− >>= findAttrTextByName ns "r" "embed"
800+ >>= findAttrByName ns "r" "embed"
715801 in
716802 case drawing of
717803 Just s -> expandDrawingId s >>= (\(fp, bs) -> return $ Drawing fp title alt bs $ elemToExtent drawingElem)
718804 Nothing -> throwError WrongElem
719−-- The below is an attempt to deal with images in deprecated vml format.
805+-- The two cases below are an attempt to deal with images in deprecated vml format.
806+-- Todo: check out title and attr for deprecated format.
720807elemToParPart ns element
721808 | isElem ns "w" "r" element
722809 , Just _ <- findChildByName ns "w" "pict" element =
723810 let drawing = findElement (elemName ns "v" "imagedata") element
724− >>= findAttrTextByName ns "r" "id"
811+ >>= findAttrByName ns "r" "id"
725812 in
726813 case drawing of
727− -- Todo: check out title and attr for deprecated format.
728814 Just s -> expandDrawingId s >>= (\(fp, bs) -> return $ Drawing fp "" "" bs Nothing)
729815 Nothing -> throwError WrongElem
816+elemToParPart ns element
817+ | isElem ns "w" "r" element
818+ , Just objectElem <- findChildByName ns "w" "object" element
819+ , Just shapeElem <- findChildByName ns "v" "shape" objectElem
820+ , Just imagedataElem <- findChildByName ns "v" "imagedata" shapeElem
821+ , Just drawingId <- findAttrByName ns "r" "id" imagedataElem
822+ = expandDrawingId drawingId >>= (\(fp, bs) -> return $ Drawing fp "" "" bs Nothing)
730823-- Chart
731824elemToParPart ns element
732825 | isElem ns "w" "r" element
@@ -790,7 +883,7 @@ elemToParPart ns element
790883 fldCharState <- gets stateFldCharState
791884 case fldCharState of
792885 FldCharOpen -> do
793− info <- eitherToD $ parseFieldInfo $ T.pack $ strContent instrText
886+ info <- eitherToD $ parseFieldInfo $ strContent instrText
794887 modify $ \st -> st{stateFldCharState = FldCharFieldInfo info}
795888 return NullParPart
796889 _ -> return NullParPart
@@ -811,48 +904,48 @@ elemToParPart ns element
811904 return $ ChangedRuns change runs
812905elemToParPart ns element
813906 | isElem ns "w" "bookmarkStart" element
814− , Just bmId <- findAttrTextByName ns "w" "id" element
815− , Just bmName <- findAttrTextByName ns "w" "name" element =
907+ , Just bmId <- findAttrByName ns "w" "id" element
908+ , Just bmName <- findAttrByName ns "w" "name" element =
816909 return $ BookMark bmId bmName
817910elemToParPart ns element
818911 | isElem ns "w" "hyperlink" element
819− , Just relId <- findAttrTextByName ns "r" "id" element = do
912+ , Just relId <- findAttrByName ns "r" "id" element = do
820913 location <- asks envLocation
821914 runs <- mapD (elemToRun ns) (elChildren element)
822915 rels <- asks envRelationships
823916 case lookupRelationship location relId rels of
824917 Just target ->
825− case findAttrTextByName ns "w" "anchor" element of
918+ case findAttrByName ns "w" "anchor" element of
826919 Just anchor -> return $ ExternalHyperLink (target <> "#" <> anchor) runs
827920 Nothing -> return $ ExternalHyperLink target runs
828921 Nothing -> return $ ExternalHyperLink "" runs
829922elemToParPart ns element
830923 | isElem ns "w" "hyperlink" element
831− , Just anchor <- findAttrTextByName ns "w" "anchor" element = do
924+ , Just anchor <- findAttrByName ns "w" "anchor" element = do
832925 runs <- mapD (elemToRun ns) (elChildren element)
833926 return $ InternalHyperLink anchor runs
834927elemToParPart ns element
835928 | isElem ns "w" "commentRangeStart" element
836− , Just cmtId <- findAttrTextByName ns "w" "id" element = do
929+ , Just cmtId <- findAttrByName ns "w" "id" element = do
837930 (Comments _ commentMap) <- asks envComments
838931 case M.lookup cmtId commentMap of
839932 Just cmtElem -> elemToCommentStart ns cmtElem
840933 Nothing -> throwError WrongElem
841934elemToParPart ns element
842935 | isElem ns "w" "commentRangeEnd" element
843− , Just cmtId <- findAttrTextByName ns "w" "id" element =
936+ , Just cmtId <- findAttrByName ns "w" "id" element =
844937 return $ CommentEnd cmtId
845938elemToParPart ns element
846939 | isElem ns "m" "oMath" element =
847− fmap PlainOMath (eitherToD $ readOMML $ T.pack $ showElement element)
940+ fmap PlainOMath (eitherToD $ readOMML $ showElement element)
848941elemToParPart _ _ = throwError WrongElem
849942
850943elemToCommentStart :: NameSpaces -> Element -> D ParPart
851944elemToCommentStart ns element
852945 | isElem ns "w" "comment" element
853− , Just cmtId <- findAttrTextByName ns "w" "id" element
854− , Just cmtAuthor <- findAttrTextByName ns "w" "author" element
855− , cmtDate <- findAttrTextByName ns "w" "date" element = do
946+ , Just cmtId <- findAttrByName ns "w" "id" element
947+ , Just cmtAuthor <- findAttrByName ns "w" "author" element
948+ , cmtDate <- findAttrByName ns "w" "date" element = do
856949 bps <- mapD (elemToBodyPart ns) (elChildren element)
857950 return $ CommentStart cmtId cmtAuthor cmtDate bps
858951elemToCommentStart _ _ = throwError WrongElem
@@ -871,7 +964,7 @@ elemToExtent drawingElem =
871964 where
872965 wp_ns = "http://schemas.openxmlformats.org/drawingml/2006/wordprocessingDrawing"
873966 getDim at = findElement (QName "extent" (Just wp_ns) (Just "wp")) drawingElem
874− >>= findAttr (QName at Nothing Nothing) >>= safeRead . T.pack
967+ >>= findAttr (QName at Nothing Nothing) >>= safeRead
875968
876969
877970childElemToRun :: NameSpaces -> Element -> D Run
@@ -882,7 +975,7 @@ childElemToRun ns element
882975 = let (title, alt) = getTitleAndAlt ns element
883976 a_ns = "http://schemas.openxmlformats.org/drawingml/2006/main"
884977 drawing = findElement (QName "blip" (Just a_ns) (Just "a")) picElem
885− >>= findAttrText (QName "embed" (lookup "r" ns) (Just "r"))
978+ >>= findAttr (QName "embed" (M.lookup "r" ns) (Just "r"))
886979 in
887980 case drawing of
888981 Just s -> expandDrawingId s >>=
@@ -895,7 +988,7 @@ childElemToRun ns element
895988 = return InlineChart
896989childElemToRun ns element
897990 | isElem ns "w" "footnoteReference" element
898− , Just fnId <- findAttrTextByName ns "w" "id" element = do
991+ , Just fnId <- findAttrByName ns "w" "id" element = do
899992 notes <- asks envNotes
900993 case lookupFootnote fnId notes of
901994 Just e -> do bps <- local (\r -> r {envLocation=InFootnote}) $ mapD (elemToBodyPart ns) (elChildren e)
@@ -903,7 +996,7 @@ childElemToRun ns element
903996 Nothing -> return $ Footnote []
904997childElemToRun ns element
905998 | isElem ns "w" "endnoteReference" element
906− , Just enId <- findAttrTextByName ns "w" "id" element = do
999+ , Just enId <- findAttrByName ns "w" "id" element = do
9071000 notes <- asks envNotes
9081001 case lookupEndnote enId notes of
9091002 Just e -> do bps <- local (\r -> r {envLocation=InEndnote}) $ mapD (elemToBodyPart ns) (elChildren e)
@@ -956,15 +1049,15 @@ getParStyleField _ _ = Nothing
9561049getTrackedChange :: NameSpaces -> Element -> Maybe TrackedChange
9571050getTrackedChange ns element
9581051 | isElem ns "w" "ins" element || isElem ns "w" "moveTo" element
959− , Just cId <- findAttrTextByName ns "w" "id" element
960− , Just cAuthor <- findAttrTextByName ns "w" "author" element
961− , mcDate <- findAttrTextByName ns "w" "date" element =
1052+ , Just cId <- findAttrByName ns "w" "id" element
1053+ , Just cAuthor <- findAttrByName ns "w" "author" element
1054+ , mcDate <- findAttrByName ns "w" "date" element =
9621055 Just $ TrackedChange Insertion (ChangeInfo cId cAuthor mcDate)
9631056getTrackedChange ns element
9641057 | isElem ns "w" "del" element || isElem ns "w" "moveFrom" element
965− , Just cId <- findAttrTextByName ns "w" "id" element
966− , Just cAuthor <- findAttrTextByName ns "w" "author" element
967− , mcDate <- findAttrTextByName ns "w" "date" element =
1058+ , Just cId <- findAttrByName ns "w" "id" element
1059+ , Just cAuthor <- findAttrByName ns "w" "author" element
1060+ , mcDate <- findAttrByName ns "w" "date" element =
9681061 Just $ TrackedChange Deletion (ChangeInfo cId cAuthor mcDate)
9691062getTrackedChange _ _ = Nothing
9701063
@@ -973,7 +1066,7 @@ elemToParagraphStyle ns element sty
9731066 | Just pPr <- findChildByName ns "w" "pPr" element =
9741067 let style =
9751068 mapMaybe
976− (fmap ParaStyleId . findAttrTextByName ns "w" "val")
1069+ (fmap ParaStyleId . findAttrByName ns "w" "val")
9771070 (findChildrenByName ns "w" "pStyle" pPr)
9781071 in ParagraphStyle
9791072 {pStyle = mapMaybe (`M.lookup` sty) style
@@ -1005,7 +1098,7 @@ elemToRunStyleD ns element
10051098 charStyles <- asks envCharStyles
10061099 let parentSty =
10071100 findChildByName ns "w" "rStyle" rPr >>=
1008− findAttrTextByName ns "w" "val" >>=
1101+ findAttrByName ns "w" "val" >>=
10091102 flip M.lookup charStyles . CharStyleId
10101103 return $ elemToRunStyle ns element parentSty
10111104elemToRunStyleD _ _ = return defaultRunStyle
@@ -1015,7 +1108,7 @@ elemToRunElem ns element
10151108 | isElem ns "w" "t" element
10161109 || isElem ns "w" "delText" element
10171110 || isElem ns "m" "t" element = do
1018− let str = T.pack $ strContent element
1111+ let str = strContent element
10191112 font <- asks envFont
10201113 case font of
10211114 Nothing -> return $ TextRun str
@@ -1037,14 +1130,14 @@ getSymChar :: NameSpaces -> Element -> RunElem
10371130getSymChar ns element
10381131 | Just s <- lowerFromPrivate <$> getCodepoint
10391132 , Just font <- getFont =
1040− case readLitChar ("\\x" ++ s) of
1133+ case readLitChar ("\\x" ++ T.unpack s) of
10411134 [(char, _)] -> TextRun . maybe "" T.singleton $ getUnicode font char
10421135 _ -> TextRun ""
10431136 where
10441137 getCodepoint = findAttrByName ns "w" "char" element
1045− getFont = textToFont . T.pack =<< findAttrByName ns "w" "font" element
1046− lowerFromPrivate ('F':xs) = '0':xs
1047− lowerFromPrivate xs = xs
1138+ getFont = textToFont =<< findAttrByName ns "w" "font" element
1139+ lowerFromPrivate t | "F" `T.isPrefixOf` t = "0" <> T.drop 1 t
1140+ | otherwise = t
10481141getSymChar _ _ = TextRun ""
10491142
10501143elemToRunElems :: NameSpaces -> Element -> D [RunElem]
@@ -1054,8 +1147,9 @@ elemToRunElems ns element
10541147 let qualName = elemName ns "w"
10551148 let font = do
10561149 fontElem <- findElement (qualName "rFonts") element
1057− textToFont . T.pack =<<
1058− foldr ((<|>) . (flip findAttr fontElem . qualName)) Nothing ["ascii", "hAnsi"]
1150+ textToFont =<<
1151+ foldr ((<|>) . (flip findAttr fontElem . qualName))
1152+ Nothing ["ascii", "hAnsi"]
10591153 local (setFont font) (mapD (elemToRunElem ns) (elChildren element))
10601154elemToRunElems _ _ = throwError WrongElem
10611155
src/Text/Pandoc/Readers/Docx/Parse/Styles.hs+32−25
@@ -48,11 +48,13 @@ import Data.Function (on)
4848import Data.String (IsString(..))
4949import qualified Data.Map as M
5050import qualified Data.Text as T
51+import qualified Data.Text.Read
52+import Data.Text (Text)
5153import Data.Maybe
5254import Data.Coerce
5355import Text.Pandoc.Readers.Docx.Util
5456import qualified Text.Pandoc.UTF8 as UTF8
55−import Text.XML.Light
57+import Text.Pandoc.XML.Light
5658
5759newtype CharStyleId = CharStyleId T.Text
5860 deriving (Show, Eq, Ord, IsString, FromStyleId)
@@ -108,7 +110,7 @@ data RunStyle = RunStyle { isBold :: Maybe Bool
108110 , isRTL :: Maybe Bool
109111 , isForceCTL :: Maybe Bool
110112 , rVertAlign :: Maybe VertAlign
111− , rUnderline :: Maybe String
113+ , rUnderline :: Maybe Text
112114 , rParentStyle :: Maybe CharStyle
113115 }
114116 deriving Show
@@ -135,19 +137,22 @@ defaultRunStyle = RunStyle { isBold = Nothing
135137 , rParentStyle = Nothing
136138 }
137139
138−archiveToStyles' :: (Ord k1, Ord k2, ElemToStyle a1, ElemToStyle a2) =>
139− (a1 -> k1) -> (a2 -> k2) -> Archive -> (M.Map k1 a1, M.Map k2 a2)
140+archiveToStyles'
141+ :: (Ord k1, Ord k2, ElemToStyle a1, ElemToStyle a2)
142+ => (a1 -> k1) -> (a2 -> k2) -> Archive -> (M.Map k1 a1, M.Map k2 a2)
140143archiveToStyles' conv1 conv2 zf =
141− let stylesElem = findEntryByPath "word/styles.xml" zf >>=
142− (parseXMLDoc . UTF8.toStringLazy . fromEntry)
143− in
144− case stylesElem of
145− Nothing -> (M.empty, M.empty)
146− Just styElem ->
147− let namespaces = elemToNameSpaces styElem
148− in
149− ( M.fromList $ map (\r -> (conv1 r, r)) $ buildBasedOnList namespaces styElem Nothing,
150− M.fromList $ map (\p -> (conv2 p, p)) $ buildBasedOnList namespaces styElem Nothing)
144+ case findEntryByPath "word/styles.xml" zf of
145+ Nothing -> (M.empty, M.empty)
146+ Just entry ->
147+ case parseXMLElement . UTF8.toTextLazy . fromEntry $ entry of
148+ Left _ -> (M.empty, M.empty)
149+ Right styElem ->
150+ let namespaces = elemToNameSpaces styElem
151+ in
152+ ( M.fromList $ map (\r -> (conv1 r, r)) $
153+ buildBasedOnList namespaces styElem Nothing,
154+ M.fromList $ map (\p -> (conv2 p, p)) $
155+ buildBasedOnList namespaces styElem Nothing)
151156
152157isBasedOnStyle :: (ElemToStyle a, FromStyleId (StyleId a)) => NameSpaces -> Element -> Maybe a -> Bool
153158isBasedOnStyle ns element parentStyle
@@ -155,7 +160,7 @@ isBasedOnStyle ns element parentStyle
155160 , Just styleType <- findAttrByName ns "w" "type" element
156161 , styleType == cStyleType parentStyle
157162 , Just basedOnVal <- findChildByName ns "w" "basedOn" element >>=
158− findAttrTextByName ns "w" "val"
163+ findAttrByName ns "w" "val"
159164 , Just ps <- parentStyle = basedOnVal == fromStyleId (getStyleId ps)
160165 | isElem ns "w" "style" element
161166 , Just styleType <- findAttrByName ns "w" "type" element
@@ -165,7 +170,7 @@ isBasedOnStyle ns element parentStyle
165170 | otherwise = False
166171
167172class HasStyleId a => ElemToStyle a where
168− cStyleType :: Maybe a -> String
173+ cStyleType :: Maybe a -> Text
169174 elemToStyle :: NameSpaces -> Element -> Maybe a -> Maybe a
170175
171176class FromStyleId (StyleId a) => HasStyleId a where
@@ -222,8 +227,10 @@ buildBasedOnList ns element rootStyle =
222227 stys -> stys ++
223228 concatMap (buildBasedOnList ns element . Just) stys
224229
225−stringToInteger :: String -> Maybe Integer
226−stringToInteger s = listToMaybe $ map fst (reads s :: [(Integer, String)])
230+stringToInteger :: Text -> Maybe Integer
231+stringToInteger s = case Data.Text.Read.decimal s of
232+ Right (x,_) -> Just x
233+ Left _ -> Nothing
227234
228235checkOnOff :: NameSpaces -> Element -> QName -> Maybe Bool
229236checkOnOff ns rPr tag
@@ -243,7 +250,7 @@ checkOnOff _ _ _ = Nothing
243250elemToCharStyle :: NameSpaces
244251 -> Element -> Maybe CharStyle -> Maybe CharStyle
245252elemToCharStyle ns element parentStyle
246− = CharStyle <$> (CharStyleId <$> findAttrTextByName ns "w" "styleId" element)
253+ = CharStyle <$> (CharStyleId <$> findAttrByName ns "w" "styleId" element)
247254 <*> getElementStyleName ns element
248255 <*> Just (elemToRunStyle ns element parentStyle)
249256
@@ -277,7 +284,7 @@ elemToRunStyle _ _ _ = defaultRunStyle
277284getHeaderLevel :: NameSpaces -> Element -> Maybe (ParaStyleName, Int)
278285getHeaderLevel ns element
279286 | Just styleName <- getElementStyleName ns element
280− , Just n <- stringToInteger . T.unpack =<<
287+ , Just n <- stringToInteger =<<
281288 (T.stripPrefix "heading " . T.toLower $
282289 fromStyleName styleName)
283290 , n > 0 = Just (styleName, fromInteger n)
@@ -285,8 +292,8 @@ getHeaderLevel _ _ = Nothing
285292
286293getElementStyleName :: Coercible T.Text a => NameSpaces -> Element -> Maybe a
287294getElementStyleName ns el = coerce <$>
288− ((findChildByName ns "w" "name" el >>= findAttrTextByName ns "w" "val")
289− <|> findAttrTextByName ns "w" "styleId" el)
295+ ((findChildByName ns "w" "name" el >>= findAttrByName ns "w" "val")
296+ <|> findAttrByName ns "w" "styleId" el)
290297
291298getNumInfo :: NameSpaces -> Element -> Maybe (T.Text, T.Text)
292299getNumInfo ns element = do
@@ -294,15 +301,15 @@ getNumInfo ns element = do
294301 findChildByName ns "w" "numPr"
295302 lvl = fromMaybe "0" (numPr >>=
296303 findChildByName ns "w" "ilvl" >>=
297− findAttrTextByName ns "w" "val")
304+ findAttrByName ns "w" "val")
298305 numId <- numPr >>=
299306 findChildByName ns "w" "numId" >>=
300− findAttrTextByName ns "w" "val"
307+ findAttrByName ns "w" "val"
301308 return (numId, lvl)
302309
303310elemToParStyleData :: NameSpaces -> Element -> Maybe ParStyle -> Maybe ParStyle
304311elemToParStyleData ns element parentStyle
305− | Just styleId <- findAttrTextByName ns "w" "styleId" element
312+ | Just styleId <- findAttrByName ns "w" "styleId" element
306313 , Just styleName <- getElementStyleName ns element
307314 = Just $ ParStyle
308315 {
src/Text/Pandoc/Readers/Docx/Util.hs+28−26
@@ -1,7 +1,8 @@
1+{-# LANGUAGE OverloadedStrings #-}
12{- |
23 Module : Text.Pandoc.Readers.Docx.StyleMaps
34 Copyright : © 2014-2020 Jesse Rosenthal <jrosenthal@jhu.edu>,
4− 2014-2020 John MacFarlane <jgm@berkeley.edu>,
5+ 2014-2021 John MacFarlane <jgm@berkeley.edu>,
56 2015 Nikolay Yakimov <root@livid.pp.ru>
67 License : GNU GPL, version 2 or above
78
@@ -18,51 +19,52 @@ module Text.Pandoc.Readers.Docx.Util (
1819 , elemToNameSpaces
1920 , findChildByName
2021 , findChildrenByName
21− , findAttrText
22+ , findElementByName
2223 , findAttrByName
23− , findAttrTextByName
2424 ) where
2525
26−import Data.Maybe (mapMaybe)
2726import qualified Data.Text as T
28−import Text.XML.Light
27+import Data.Text (Text)
28+import Text.Pandoc.XML.Light
29+import qualified Data.Map as M
2930
30−type NameSpaces = [(String, String)]
31+type NameSpaces = M.Map Text Text
3132
3233elemToNameSpaces :: Element -> NameSpaces
33−elemToNameSpaces = mapMaybe attrToNSPair . elAttribs
34+elemToNameSpaces = foldr (\(Attr qn val) ->
35+ case qn of
36+ QName s _ (Just "xmlns") -> M.insert s val
37+ _ -> id) mempty . elAttribs
3438
35−attrToNSPair :: Attr -> Maybe (String, String)
36−attrToNSPair (Attr (QName s _ (Just "xmlns")) val) = Just (s, val)
37−attrToNSPair _ = Nothing
38−
39−elemName :: NameSpaces -> String -> String -> QName
39+elemName :: NameSpaces -> Text -> Text -> QName
4040elemName ns prefix name =
41− QName name (lookup prefix ns) (if null prefix then Nothing else Just prefix)
41+ QName name (M.lookup prefix ns)
42+ (if T.null prefix then Nothing else Just prefix)
4243
43−isElem :: NameSpaces -> String -> String -> Element -> Bool
44+isElem :: NameSpaces -> Text -> Text -> Element -> Bool
4445isElem ns prefix name element =
45− let ns' = ns ++ elemToNameSpaces element
46+ let ns' = ns <> elemToNameSpaces element
4647 in qName (elName element) == name &&
47− qURI (elName element) == lookup prefix ns'
48+ qURI (elName element) == M.lookup prefix ns'
4849
49−findChildByName :: NameSpaces -> String -> String -> Element -> Maybe Element
50+findChildByName :: NameSpaces -> Text -> Text -> Element -> Maybe Element
5051findChildByName ns pref name el =
51− let ns' = ns ++ elemToNameSpaces el
52+ let ns' = ns <> elemToNameSpaces el
5253 in findChild (elemName ns' pref name) el
5354
54−findChildrenByName :: NameSpaces -> String -> String -> Element -> [Element]
55+findChildrenByName :: NameSpaces -> Text -> Text -> Element -> [Element]
5556findChildrenByName ns pref name el =
56− let ns' = ns ++ elemToNameSpaces el
57+ let ns' = ns <> elemToNameSpaces el
5758 in findChildren (elemName ns' pref name) el
5859
59−findAttrText :: QName -> Element -> Maybe T.Text
60−findAttrText x = fmap T.pack . findAttr x
60+-- | Like 'findChildrenByName', but searches descendants.
61+findElementByName :: NameSpaces -> Text -> Text -> Element -> Maybe Element
62+findElementByName ns pref name el =
63+ let ns' = ns <> elemToNameSpaces el
64+ in findElement (elemName ns' pref name) el
6165
62−findAttrByName :: NameSpaces -> String -> String -> Element -> Maybe String
66+findAttrByName :: NameSpaces -> Text -> Text -> Element -> Maybe Text
6367findAttrByName ns pref name el =
64− let ns' = ns ++ elemToNameSpaces el
68+ let ns' = ns <> elemToNameSpaces el
6569 in findAttr (elemName ns' pref name) el
6670
67−findAttrTextByName :: NameSpaces -> String -> String -> Element -> Maybe T.Text
68−findAttrTextByName a b c = fmap T.pack . findAttrByName a b c
src/Text/Pandoc/Readers/DokuWiki.hs+9−8
@@ -29,26 +29,27 @@ import Text.Pandoc.Definition
2929import Text.Pandoc.Error (PandocError (PandocParsecError))
3030import Text.Pandoc.Options
3131import Text.Pandoc.Parsing hiding (enclosed, nested)
32−import Text.Pandoc.Shared (crFilter, trim, stringify, tshow)
32+import Text.Pandoc.Shared (trim, stringify, tshow)
3333
3434-- | Read DokuWiki from an input string and return a Pandoc document.
35−readDokuWiki :: PandocMonad m
35+readDokuWiki :: (PandocMonad m, ToSources a)
3636 => ReaderOptions
37− -> Text
37+ -> a
3838 -> m Pandoc
3939readDokuWiki opts s = do
40− let input = crFilter s
41− res <- runParserT parseDokuWiki def {stateOptions = opts } "source" input
40+ let sources = toSources s
41+ res <- runParserT parseDokuWiki def {stateOptions = opts }
42+ (initialSourceName sources) sources
4243 case res of
43− Left e -> throwError $ PandocParsecError input e
44+ Left e -> throwError $ PandocParsecError sources e
4445 Right d -> return d
4546
46−type DWParser = ParserT Text ParserState
47+type DWParser = ParserT Sources ParserState
4748
4849-- * Utility functions
4950
5051-- | Parse end-of-line, which can be either a newline or end-of-file.
51−eol :: Stream s m Char => ParserT s st m ()
52+eol :: (Stream s m Char, UpdateSourcePos s Char) => ParserT s st m ()
5253eol = void newline <|> eof
5354
5455nested :: PandocMonad m => DWParser m a -> DWParser m a
src/Text/Pandoc/Readers/EPUB.hs+44−36
@@ -17,14 +17,14 @@ module Text.Pandoc.Readers.EPUB
1717 (readEPUB)
1818 where
1919
20−import Codec.Archive.Zip (Archive (..), Entry, findEntryByPath, fromEntry,
20+import Codec.Archive.Zip (Archive (..), Entry(..), findEntryByPath, fromEntry,
2121 toArchiveOrFail)
2222import Control.DeepSeq (NFData, deepseq)
2323import Control.Monad (guard, liftM, liftM2, mplus)
2424import Control.Monad.Except (throwError)
2525import qualified Data.ByteString.Lazy as BL (ByteString)
26−import Data.List (isInfixOf)
2726import qualified Data.Text as T
27+import Data.Text (Text)
2828import qualified Data.Map as M (Map, elems, fromList, lookup)
2929import Data.Maybe (mapMaybe)
3030import qualified Data.Text.Lazy as TL
@@ -40,12 +40,12 @@ import Text.Pandoc.Extensions (Extension (Ext_raw_html), enableExtension)
4040import Text.Pandoc.MIME (MimeType)
4141import Text.Pandoc.Options (ReaderOptions (..))
4242import Text.Pandoc.Readers.HTML (readHtml)
43−import Text.Pandoc.Shared (addMetaField, collapseFilePath, escapeURI)
44−import qualified Text.Pandoc.UTF8 as UTF8 (toStringLazy)
43+import Text.Pandoc.Shared (addMetaField, collapseFilePath, escapeURI, tshow)
44+import qualified Text.Pandoc.UTF8 as UTF8 (toTextLazy)
4545import Text.Pandoc.Walk (query, walk)
46−import Text.XML.Light
46+import Text.Pandoc.XML.Light
4747
48−type Items = M.Map String (FilePath, MimeType)
48+type Items = M.Map Text (FilePath, MimeType)
4949
5050readEPUB :: PandocMonad m => ReaderOptions -> BL.ByteString -> m Pandoc
5151readEPUB opts bytes = case toArchiveOrFail bytes of
@@ -125,26 +125,27 @@ imageToPandoc s = B.doc . B.para $ B.image (T.pack s) "" mempty
125125imageMimes :: [MimeType]
126126imageMimes = ["image/gif", "image/jpeg", "image/png"]
127127
128−type CoverId = String
128+type CoverId = Text
129129
130130type CoverImage = FilePath
131131
132−parseManifest :: (PandocMonad m) => Element -> Maybe CoverId -> m (Maybe CoverImage, Items)
132+parseManifest :: (PandocMonad m)
133+ => Element -> Maybe CoverId -> m (Maybe CoverImage, Items)
133134parseManifest content coverId = do
134135 manifest <- findElementE (dfName "manifest") content
135136 let items = findChildren (dfName "item") manifest
136137 r <- mapM parseItem items
137138 let cover = findAttr (emptyName "href") =<< filterChild findCover manifest
138− return (cover `mplus` coverId, M.fromList r)
139+ return (T.unpack <$> (cover `mplus` coverId), M.fromList r)
139140 where
140− findCover e = maybe False (isInfixOf "cover-image")
141+ findCover e = maybe False (T.isInfixOf "cover-image")
141142 (findAttr (emptyName "properties") e)
142143 || Just True == liftM2 (==) coverId (findAttr (emptyName "id") e)
143144 parseItem e = do
144145 uid <- findAttrE (emptyName "id") e
145146 href <- findAttrE (emptyName "href") e
146147 mime <- findAttrE (emptyName "media-type") e
147− return (uid, (href, T.pack mime))
148+ return (uid, (T.unpack href, mime))
148149
149150parseSpine :: PandocMonad m => Items -> Element -> m [(FilePath, MimeType)]
150151parseSpine is e = do
@@ -172,25 +173,25 @@ parseMeta content = do
172173-- http://www.idpf.org/epub/30/spec/epub30-publications.html#sec-metadata-elem
173174parseMetaItem :: Element -> Meta -> Meta
174175parseMetaItem e@(stripNamespace . elName -> field) meta =
175− addMetaField (renameMeta field) (B.str $ T.pack $ strContent e) meta
176+ addMetaField (renameMeta field) (B.str $ strContent e) meta
176177
177−renameMeta :: String -> T.Text
178+renameMeta :: Text -> Text
178179renameMeta "creator" = "author"
179−renameMeta s = T.pack s
180+renameMeta s = s
180181
181182getManifest :: PandocMonad m => Archive -> m (String, Element)
182183getManifest archive = do
183184 metaEntry <- findEntryByPathE ("META-INF" </> "container.xml") archive
184− docElem <- (parseXMLDocE . UTF8.toStringLazy . fromEntry) metaEntry
185+ docElem <- parseXMLDocE metaEntry
185186 let namespaces = mapMaybe attrToNSPair (elAttribs docElem)
186187 ns <- mkE "xmlns not in namespaces" (lookup "xmlns" namespaces)
187188 as <- fmap (map attrToPair . elAttribs)
188189 (findElementE (QName "rootfile" (Just ns) Nothing) docElem)
189− manifestFile <- mkE "Root not found" (lookup "full-path" as)
190+ manifestFile <- T.unpack <$> mkE "Root not found" (lookup "full-path" as)
190191 let rootdir = dropFileName manifestFile
191192 --mime <- lookup "media-type" as
192193 manifest <- findEntryByPathE manifestFile archive
193− (rootdir,) <$> (parseXMLDocE . UTF8.toStringLazy . fromEntry $ manifest)
194+ (rootdir,) <$> parseXMLDocE manifest
194195
195196-- Fixup
196197
@@ -200,7 +201,8 @@ fixInternalReferences pathToFile =
200201 . walk (fixBlockIRs filename)
201202 . walk (fixInlineIRs filename)
202203 where
203− (root, T.unpack . escapeURI . T.pack -> filename) = splitFileName pathToFile
204+ (root, T.unpack . escapeURI . T.pack -> filename) =
205+ splitFileName pathToFile
204206
205207fixInlineIRs :: String -> Inline -> Inline
206208fixInlineIRs s (Span as v) =
@@ -213,7 +215,7 @@ fixInlineIRs s (Link as is t) =
213215 Link (fixAttrs s as) is t
214216fixInlineIRs _ v = v
215217
216−prependHash :: [T.Text] -> Inline -> Inline
218+prependHash :: [Text] -> Inline -> Inline
217219prependHash ps l@(Link attr is (url, tit))
218220 | or [s `T.isPrefixOf` url | s <- ps] =
219221 Link attr is ("#" <> url, tit)
@@ -230,16 +232,17 @@ fixBlockIRs s (CodeBlock as code) =
230232fixBlockIRs _ b = b
231233
232234fixAttrs :: FilePath -> B.Attr -> B.Attr
233−fixAttrs s (ident, cs, kvs) = (addHash s ident, filter (not . T.null) cs, removeEPUBAttrs kvs)
235+fixAttrs s (ident, cs, kvs) =
236+ (addHash s ident, filter (not . T.null) cs, removeEPUBAttrs kvs)
234237
235−addHash :: String -> T.Text -> T.Text
238+addHash :: FilePath -> Text -> Text
236239addHash _ "" = ""
237240addHash s ident = T.pack (takeFileName s) <> "#" <> ident
238241
239−removeEPUBAttrs :: [(T.Text, T.Text)] -> [(T.Text, T.Text)]
242+removeEPUBAttrs :: [(Text, Text)] -> [(Text, Text)]
240243removeEPUBAttrs kvs = filter (not . isEPUBAttr) kvs
241244
242−isEPUBAttr :: (T.Text, a) -> Bool
245+isEPUBAttr :: (Text, a) -> Bool
243246isEPUBAttr (k, _) = "epub:" `T.isPrefixOf` k
244247
245248-- Library
@@ -256,39 +259,44 @@ uncurry3 f (a, b, c) = f a b c
256259
257260-- Utility
258261
259−stripNamespace :: QName -> String
262+stripNamespace :: QName -> Text
260263stripNamespace (QName v _ _) = v
261264
262−attrToNSPair :: Attr -> Maybe (String, String)
265+attrToNSPair :: Attr -> Maybe (Text, Text)
263266attrToNSPair (Attr (QName "xmlns" _ _) val) = Just ("xmlns", val)
264267attrToNSPair _ = Nothing
265268
266−attrToPair :: Attr -> (String, String)
269+attrToPair :: Attr -> (Text, Text)
267270attrToPair (Attr (QName name _ _) val) = (name, val)
268271
269−defaultNameSpace :: Maybe String
272+defaultNameSpace :: Maybe Text
270273defaultNameSpace = Just "http://www.idpf.org/2007/opf"
271274
272−dfName :: String -> QName
275+dfName :: Text -> QName
273276dfName s = QName s defaultNameSpace Nothing
274277
275−emptyName :: String -> QName
278+emptyName :: Text -> QName
276279emptyName s = QName s Nothing Nothing
277280
278281-- Convert Maybe interface to Either
279282
280−findAttrE :: PandocMonad m => QName -> Element -> m String
283+findAttrE :: PandocMonad m => QName -> Element -> m Text
281284findAttrE q e = mkE "findAttr" $ findAttr q e
282285
283286findEntryByPathE :: PandocMonad m => FilePath -> Archive -> m Entry
284287findEntryByPathE (normalise . unEscapeString -> path) a =
285− mkE ("No entry on path: " ++ path) $ findEntryByPath path a
288+ mkE ("No entry on path: " <> T.pack path) $ findEntryByPath path a
286289
287−parseXMLDocE :: PandocMonad m => String -> m Element
288−parseXMLDocE doc = mkE "Unable to parse XML doc" $ parseXMLDoc doc
290+parseXMLDocE :: PandocMonad m => Entry -> m Element
291+parseXMLDocE entry =
292+ either (throwError . PandocXMLError fp) return $ parseXMLElement doc
293+ where
294+ doc = UTF8.toTextLazy . fromEntry $ entry
295+ fp = T.pack $ eRelativePath entry
289296
290297findElementE :: PandocMonad m => QName -> Element -> m Element
291−findElementE e x = mkE ("Unable to find element: " ++ show e) $ findElement e x
298+findElementE e x =
299+ mkE ("Unable to find element: " <> tshow e) $ findElement e x
292300
293−mkE :: PandocMonad m => String -> Maybe a -> m a
294−mkE s = maybe (throwError . PandocParseError $ T.pack s) return
301+mkE :: PandocMonad m => Text -> Maybe a -> m a
302+mkE s = maybe (throwError . PandocParseError $ s) return
src/Text/Pandoc/Readers/FB2.hs+57−51
@@ -25,13 +25,13 @@ TODO:
2525module Text.Pandoc.Readers.FB2 ( readFB2 ) where
2626import Control.Monad.Except (throwError)
2727import Control.Monad.State.Strict
28−import Data.ByteString.Lazy.Char8 ( pack )
2928import Data.ByteString.Base64.Lazy
3029import Data.Functor
3130import Data.List (intersperse)
3231import qualified Data.Map as M
3332import Data.Text (Text)
3433import qualified Data.Text as T
34+import qualified Data.Text.Lazy as TL
3535import Data.Default
3636import Data.Maybe
3737import Text.HTML.TagSoup.Entity (lookupEntity)
@@ -40,8 +40,9 @@ import Text.Pandoc.Class.PandocMonad (PandocMonad, insertMedia, report)
4040import Text.Pandoc.Error
4141import Text.Pandoc.Logging
4242import Text.Pandoc.Options
43−import Text.Pandoc.Shared (crFilter)
44−import Text.XML.Light
43+import Text.Pandoc.XML.Light
44+import qualified Text.Pandoc.UTF8 as UTF8
45+import Text.Pandoc.Sources (ToSources(..), sourcesToText)
4546
4647type FB2 m = StateT FB2State m
4748
@@ -62,12 +63,15 @@ instance HasMeta FB2State where
6263 setMeta field v s = s {fb2Meta = setMeta field v (fb2Meta s)}
6364 deleteMeta field s = s {fb2Meta = deleteMeta field (fb2Meta s)}
6465
65−readFB2 :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
66+readFB2 :: (PandocMonad m, ToSources a)
67+ => ReaderOptions
68+ -> a
69+ -> m Pandoc
6670readFB2 _ inp =
67− case parseXMLDoc $ crFilter inp of
68− Nothing -> throwError $ PandocParseError "Not an XML document"
69− Just e -> do
70− (bs, st) <- runStateT (parseRootElement e) def
71+ case parseXMLElement $ TL.fromStrict $ sourcesToText $ toSources inp of
72+ Left msg -> throwError $ PandocXMLError "" msg
73+ Right el -> do
74+ (bs, st) <- runStateT (parseRootElement el) def
7175 let authors = if null $ fb2Authors st
7276 then id
7377 else setMeta "author" (map text $ reverse $ fb2Authors st)
@@ -83,12 +87,12 @@ removeHash t = case T.uncons t of
8387 Just ('#', xs) -> xs
8488 _ -> t
8589
86−convertEntity :: String -> Text
87−convertEntity e = maybe (T.toUpper $ T.pack e) T.pack $ lookupEntity e
90+convertEntity :: Text -> Text
91+convertEntity e = maybe (T.toUpper e) T.pack $ lookupEntity (T.unpack e)
8892
8993parseInline :: PandocMonad m => Content -> FB2 m Inlines
9094parseInline (Elem e) =
91− case T.pack $ qName $ elName e of
95+ case qName $ elName e of
9296 "strong" -> strong <$> parseStyleType e
9397 "emphasis" -> emph <$> parseStyleType e
9498 "style" -> parseNamedStyle e
@@ -96,12 +100,12 @@ parseInline (Elem e) =
96100 "strikethrough" -> strikeout <$> parseStyleType e
97101 "sub" -> subscript <$> parseStyleType e
98102 "sup" -> superscript <$> parseStyleType e
99− "code" -> pure $ code $ T.pack $ strContent e
103+ "code" -> pure $ code $ strContent e
100104 "image" -> parseInlineImageElement e
101105 name -> do
102106 report $ IgnoredElement name
103107 pure mempty
104−parseInline (Text x) = pure $ text $ T.pack $ cdData x
108+parseInline (Text x) = pure $ text $ cdData x
105109parseInline (CRef r) = pure $ str $ convertEntity r
106110
107111parseSubtitle :: PandocMonad m => Element -> FB2 m Blocks
@@ -111,7 +115,7 @@ parseSubtitle e = headerWith ("", ["unnumbered"], []) <$> gets fb2SectionLevel <
111115
112116parseRootElement :: PandocMonad m => Element -> FB2 m Blocks
113117parseRootElement e =
114− case T.pack $ qName $ elName e of
118+ case qName $ elName e of
115119 "FictionBook" -> do
116120 -- Parse notes before parsing the rest of the content.
117121 case filterChild isNotesBody e of
@@ -144,7 +148,7 @@ parseNote e =
144148 Just sectionId -> do
145149 content <- mconcat <$> mapM parseSectionChild (dropTitle $ elChildren e)
146150 oldNotes <- gets fb2Notes
147− modify $ \s -> s { fb2Notes = M.insert ("#" <> T.pack sectionId) content oldNotes }
151+ modify $ \s -> s { fb2Notes = M.insert ("#" <> sectionId) content oldNotes }
148152 pure ()
149153 where
150154 isTitle x = qName (elName x) == "title"
@@ -156,7 +160,7 @@ parseNote e =
156160-- | Parse a child of @\<FictionBook>@ element.
157161parseFictionBookChild :: PandocMonad m => Element -> FB2 m Blocks
158162parseFictionBookChild e =
159− case T.pack $ qName $ elName e of
163+ case qName $ elName e of
160164 "stylesheet" -> pure mempty -- stylesheet is ignored
161165 "description" -> mempty <$ mapM_ parseDescriptionChild (elChildren e)
162166 "body" -> if isNotesBody e
@@ -168,7 +172,7 @@ parseFictionBookChild e =
168172-- | Parse a child of @\<description>@ element.
169173parseDescriptionChild :: PandocMonad m => Element -> FB2 m ()
170174parseDescriptionChild e =
171− case T.pack $ qName $ elName e of
175+ case qName $ elName e of
172176 "title-info" -> mapM_ parseTitleInfoChild (elChildren e)
173177 "src-title-info" -> pure () -- ignore
174178 "document-info" -> pure ()
@@ -182,7 +186,7 @@ parseDescriptionChild e =
182186-- | Parse a child of @\<body>@ element.
183187parseBodyChild :: PandocMonad m => Element -> FB2 m Blocks
184188parseBodyChild e =
185− case T.pack $ qName $ elName e of
189+ case qName $ elName e of
186190 "image" -> parseImageElement e
187191 "title" -> header <$> gets fb2SectionLevel <*> parseTitleType (elContent e)
188192 "epigraph" -> parseEpigraph e
@@ -196,7 +200,10 @@ parseBinaryElement e =
196200 (Nothing, _) -> report $ IgnoredElement "binary without id attribute"
197201 (Just _, Nothing) ->
198202 report $ IgnoredElement "binary without content-type attribute"
199− (Just filename, contentType) -> insertMedia filename (T.pack <$> contentType) (decodeLenient (pack (strContent e)))
203+ (Just filename, contentType) ->
204+ insertMedia (T.unpack filename) contentType
205+ (decodeLenient
206+ (UTF8.fromTextLazy . TL.fromStrict . strContent $ e))
200207
201208-- * Type parsers
202209
@@ -206,13 +213,13 @@ parseAuthor e = T.unwords . catMaybes <$> mapM parseAuthorChild (elChildren e)
206213
207214parseAuthorChild :: PandocMonad m => Element -> FB2 m (Maybe Text)
208215parseAuthorChild e =
209− case T.pack $ qName $ elName e of
210− "first-name" -> pure $ Just $ T.pack $ strContent e
211− "middle-name" -> pure $ Just $ T.pack $ strContent e
212− "last-name" -> pure $ Just $ T.pack $ strContent e
213− "nickname" -> pure $ Just $ T.pack $ strContent e
214− "home-page" -> pure $ Just $ T.pack $ strContent e
215− "email" -> pure $ Just $ T.pack $ strContent e
216+ case qName $ elName e of
217+ "first-name" -> pure $ Just $ strContent e
218+ "middle-name" -> pure $ Just $ strContent e
219+ "last-name" -> pure $ Just $ strContent e
220+ "nickname" -> pure $ Just $ strContent e
221+ "home-page" -> pure $ Just $ strContent e
222+ "email" -> pure $ Just $ strContent e
216223 name -> do
217224 report $ IgnoredElement $ name <> " in author"
218225 pure Nothing
@@ -236,13 +243,13 @@ parseTitleContent _ = pure Nothing
236243parseImageElement :: PandocMonad m => Element -> FB2 m Blocks
237244parseImageElement e =
238245 case href of
239− Just src -> pure $ para $ imageWith (imgId, [], []) (removeHash $ T.pack src) title alt
246+ Just src -> pure $ para $ imageWith (imgId, [], []) (removeHash src) title alt
240247 Nothing -> do
241248 report $ IgnoredElement " image without href"
242249 pure mempty
243− where alt = maybe mempty (str . T.pack) $ findAttr (unqual "alt") e
244− title = maybe "" T.pack $ findAttr (unqual "title") e
245− imgId = maybe "" T.pack $ findAttr (unqual "id") e
250+ where alt = maybe mempty str $ findAttr (unqual "alt") e
251+ title = fromMaybe "" $ findAttr (unqual "title") e
252+ imgId = fromMaybe "" $ findAttr (unqual "id") e
246253 href = findAttr (QName "href" (Just "http://www.w3.org/1999/xlink") Nothing) e
247254
248255-- | Parse @pType@
@@ -256,7 +263,7 @@ parseCite e = blockQuote . mconcat <$> mapM parseCiteChild (elChildren e)
256263-- | Parse @citeType@ child
257264parseCiteChild :: PandocMonad m => Element -> FB2 m Blocks
258265parseCiteChild e =
259− case T.pack $ qName $ elName e of
266+ case qName $ elName e of
260267 "p" -> para <$> parsePType e
261268 "poem" -> parsePoem e
262269 "empty-line" -> pure horizontalRule
@@ -271,13 +278,13 @@ parsePoem e = mconcat <$> mapM parsePoemChild (elChildren e)
271278
272279parsePoemChild :: PandocMonad m => Element -> FB2 m Blocks
273280parsePoemChild e =
274− case T.pack $ qName $ elName e of
281+ case qName $ elName e of
275282 "title" -> parseTitle e
276283 "subtitle" -> parseSubtitle e
277284 "epigraph" -> parseEpigraph e
278285 "stanza" -> parseStanza e
279286 "text-author" -> para <$> parsePType e
280− "date" -> pure $ para $ text $ T.pack $ strContent e
287+ "date" -> pure $ para $ text $ strContent e
281288 name -> report (UnexpectedXmlElement name "poem") $> mempty
282289
283290parseStanza :: PandocMonad m => Element -> FB2 m Blocks
@@ -290,7 +297,7 @@ joinLineBlocks [] = []
290297
291298parseStanzaChild :: PandocMonad m => Element -> FB2 m Blocks
292299parseStanzaChild e =
293− case T.pack $ qName $ elName e of
300+ case qName $ elName e of
294301 "title" -> parseTitle e
295302 "subtitle" -> parseSubtitle e
296303 "v" -> lineBlock . (:[]) <$> parsePType e
@@ -300,11 +307,11 @@ parseStanzaChild e =
300307parseEpigraph :: PandocMonad m => Element -> FB2 m Blocks
301308parseEpigraph e =
302309 divWith (divId, ["epigraph"], []) . mconcat <$> mapM parseEpigraphChild (elChildren e)
303− where divId = maybe "" T.pack $ findAttr (unqual "id") e
310+ where divId = fromMaybe "" $ findAttr (unqual "id") e
304311
305312parseEpigraphChild :: PandocMonad m => Element -> FB2 m Blocks
306313parseEpigraphChild e =
307− case T.pack $ qName $ elName e of
314+ case qName $ elName e of
308315 "p" -> para <$> parsePType e
309316 "poem" -> parsePoem e
310317 "cite" -> parseCite e
@@ -318,7 +325,7 @@ parseAnnotation e = mconcat <$> mapM parseAnnotationChild (elChildren e)
318325
319326parseAnnotationChild :: PandocMonad m => Element -> FB2 m Blocks
320327parseAnnotationChild e =
321− case T.pack $ qName $ elName e of
328+ case qName $ elName e of
322329 "p" -> para <$> parsePType e
323330 "poem" -> parsePoem e
324331 "cite" -> parseCite e
@@ -332,14 +339,14 @@ parseSection :: PandocMonad m => Element -> FB2 m Blocks
332339parseSection e = do
333340 n <- gets fb2SectionLevel
334341 modify $ \st -> st{ fb2SectionLevel = n + 1 }
335− let sectionId = maybe "" T.pack $ findAttr (unqual "id") e
342+ let sectionId = fromMaybe "" $ findAttr (unqual "id") e
336343 bs <- divWith (sectionId, ["section"], []) . mconcat <$> mapM parseSectionChild (elChildren e)
337344 modify $ \st -> st{ fb2SectionLevel = n }
338345 pure bs
339346
340347parseSectionChild :: PandocMonad m => Element -> FB2 m Blocks
341348parseSectionChild e =
342− case T.pack $ qName $ elName e of
349+ case qName $ elName e of
343350 "title" -> parseBodyChild e
344351 "epigraph" -> parseEpigraph e
345352 "image" -> parseImageElement e
@@ -361,16 +368,16 @@ parseStyleType e = mconcat <$> mapM parseInline (elContent e)
361368parseNamedStyle :: PandocMonad m => Element -> FB2 m Inlines
362369parseNamedStyle e = do
363370 content <- mconcat <$> mapM parseNamedStyleChild (elContent e)
364− let lang = maybeToList $ ("lang",) . T.pack <$> findAttr (QName "lang" Nothing (Just "xml")) e
371+ let lang = maybeToList $ ("lang",) <$> findAttr (QName "lang" Nothing (Just "xml")) e
365372 case findAttr (unqual "name") e of
366− Just name -> pure $ spanWith ("", [T.pack name], lang) content
373+ Just name -> pure $ spanWith ("", [name], lang) content
367374 Nothing -> do
368375 report $ IgnoredElement "link without required name"
369376 pure mempty
370377
371378parseNamedStyleChild :: PandocMonad m => Content -> FB2 m Inlines
372379parseNamedStyleChild (Elem e) =
373− case T.pack $ qName (elName e) of
380+ case qName (elName e) of
374381 "strong" -> strong <$> parseStyleType e
375382 "emphasis" -> emph <$> parseStyleType e
376383 "style" -> parseNamedStyle e
@@ -378,7 +385,7 @@ parseNamedStyleChild (Elem e) =
378385 "strikethrough" -> strikeout <$> parseStyleType e
379386 "sub" -> subscript <$> parseStyleType e
380387 "sup" -> superscript <$> parseStyleType e
381− "code" -> pure $ code $ T.pack $ strContent e
388+ "code" -> pure $ code $ strContent e
382389 "image" -> parseInlineImageElement e
383390 name -> do
384391 report $ IgnoredElement $ name <> " in style"
@@ -390,7 +397,7 @@ parseLinkType :: PandocMonad m => Element -> FB2 m Inlines
390397parseLinkType e = do
391398 content <- mconcat <$> mapM parseStyleLinkType (elContent e)
392399 notes <- gets fb2Notes
393− case T.pack <$> findAttr (QName "href" (Just "http://www.w3.org/1999/xlink") Nothing) e of
400+ case findAttr (QName "href" (Just "http://www.w3.org/1999/xlink") Nothing) e of
394401 Just href -> case findAttr (QName "type" Nothing Nothing) e of
395402 Just "note" -> case M.lookup href notes of
396403 Nothing -> pure $ link href "" content
@@ -417,15 +424,14 @@ parseTable _ = pure mempty -- TODO: tables are not supported yet
417424-- | Parse @title-infoType@
418425parseTitleInfoChild :: PandocMonad m => Element -> FB2 m ()
419426parseTitleInfoChild e =
420− case T.pack $ qName (elName e) of
427+ case qName (elName e) of
421428 "genre" -> pure ()
422429 "author" -> parseAuthor e >>= \author -> modify (\st -> st {fb2Authors = author:fb2Authors st})
423− "book-title" -> modify (setMeta "title" (text $ T.pack $ strContent e))
430+ "book-title" -> modify (setMeta "title" (text $ strContent e))
424431 "annotation" -> parseAnnotation e >>= modify . setMeta "abstract"
425432 "keywords" -> modify (setMeta "keywords" (map (MetaString . trim) $ T.splitOn ","
426− $ T.pack
427433 $ strContent e))
428− "date" -> modify (setMeta "date" (text $ T.pack $ strContent e))
434+ "date" -> modify (setMeta "date" (text $ strContent e))
429435 "coverpage" -> parseCoverPage e
430436 "lang" -> pure ()
431437 "src-lang" -> pure ()
@@ -439,7 +445,7 @@ parseCoverPage e =
439445 Just img -> case href of
440446 Just src -> modify (setMeta "cover-image" (MetaString $ removeHash src))
441447 Nothing -> pure ()
442− where href = T.pack <$> findAttr (QName "href" (Just "http://www.w3.org/1999/xlink") Nothing) img
448+ where href = findAttr (QName "href" (Just "http://www.w3.org/1999/xlink") Nothing) img
443449 Nothing -> pure ()
444450
445451-- | Parse @inlineImageType@ element
@@ -452,5 +458,5 @@ parseInlineImageElement e =
452458 Nothing -> do
453459 report $ IgnoredElement "inline image without href"
454460 pure mempty
455− where alt = maybe mempty (str . T.pack) $ findAttr (unqual "alt") e
456− href = T.pack <$> findAttr (QName "href" (Just "http://www.w3.org/1999/xlink") Nothing) e
461+ where alt = maybe mempty str $ findAttr (unqual "alt") e
462+ href = findAttr (QName "href" (Just "http://www.w3.org/1999/xlink") Nothing) e
src/Text/Pandoc/Readers/HTML.hs+200−137
@@ -5,7 +5,7 @@
55{-# LANGUAGE ViewPatterns #-}
66{- |
77 Module : Text.Pandoc.Readers.HTML
8− Copyright : Copyright (C) 2006-2020 John MacFarlane
8+ Copyright : Copyright (C) 2006-2021 John MacFarlane
99 License : GNU GPL, version 2 or above
1010
1111 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -19,20 +19,20 @@ module Text.Pandoc.Readers.HTML ( readHtml
1919 , htmlInBalanced
2020 , isInlineTag
2121 , isBlockTag
22− , NamedTag(..)
2322 , isTextTag
2423 , isCommentTag
2524 ) where
2625
2726import Control.Applicative ((<|>))
2827import Control.Monad (guard, msum, mzero, unless, void)
29−import Control.Monad.Except (throwError)
28+import Control.Monad.Except (throwError, catchError)
3029import Control.Monad.Reader (ask, asks, lift, local, runReaderT)
3130import Data.ByteString.Base64 (encode)
3231import Data.Char (isAlphaNum, isLetter)
3332import Data.Default (Default (..), def)
3433import Data.Foldable (for_)
3534import Data.List.Split (splitWhen)
35+import Data.List (foldl')
3636import qualified Data.Map as M
3737import Data.Maybe (fromMaybe, isJust, isNothing)
3838import Data.Monoid (First (..))
@@ -62,21 +62,21 @@ import Text.Pandoc.Options (
6262 extensionEnabled)
6363import Text.Pandoc.Parsing hiding ((<|>))
6464import Text.Pandoc.Shared (
65− addMetaField, blocksToInlines', crFilter, escapeURI, extractSpaces,
65+ addMetaField, blocksToInlines', escapeURI, extractSpaces,
6666 htmlSpanLikeElements, renderTags', safeRead, tshow)
6767import Text.Pandoc.Walk
6868import Text.Parsec.Error
6969import Text.TeXMath (readMathML, writeTeX)
7070
7171-- | Convert HTML-formatted string to 'Pandoc' document.
72−readHtml :: PandocMonad m
72+readHtml :: (PandocMonad m, ToSources a)
7373 => ReaderOptions -- ^ Reader options
74− -> Text -- ^ String to parse (assumes @'\n'@ line endings)
74+ -> a -- ^ Input to parse
7575 -> m Pandoc
7676readHtml opts inp = do
7777 let tags = stripPrefixes $ canonicalizeTags $
7878 parseTagsOptions parseOptions{ optTagPosition = True }
79− (crFilter inp)
79+ (sourcesToText $ toSources inp)
8080 parseDoc = do
8181 blocks <- fixPlains False . mconcat <$> manyTill block eof
8282 meta <- stateMeta . parserState <$> getState
@@ -158,32 +158,65 @@ pHead = pInTags "head" $ pTitle <|> pMetaTag <|> pBaseTag <|> (mempty <$ pAny)
158158 return mempty
159159
160160block :: PandocMonad m => TagParser m Blocks
161−block = do
162− res <- choice
163− [ eSection
164− , eSwitch B.para block
165− , mempty <$ eFootnote
166− , mempty <$ eTOC
167− , mempty <$ eTitlePage
168− , pPara
169− , pHeader
170− , pBlockQuote
171− , pCodeBlock
172− , pList
173− , pHrule
174− , pTable block
175− , pHtml
176− , pHead
177− , pBody
178− , pLineBlock
179− , pDiv
180− , pPlain
181− , pFigure
182− , pIframe
183− , pRawHtmlBlock
184− ]
185− trace (T.take 60 $ tshow $ B.toList res)
186− return res
161+block = ((do
162+ tag <- lookAhead (pSatisfy isBlockTag)
163+ exts <- getOption readerExtensions
164+ case tag of
165+ TagOpen name attr ->
166+ let type' = fromMaybe "" $
167+ lookup "type" attr <|> lookup "epub:type" attr
168+ epubExts = extensionEnabled Ext_epub_html_exts exts
169+ in
170+ case name of
171+ _ | name `elem` sectioningContent
172+ , epubExts
173+ , "chapter" `T.isInfixOf` type'
174+ -> eSection
175+ _ | epubExts
176+ , type' `elem` ["footnote", "rearnote"]
177+ -> mempty <$ eFootnote
178+ _ | epubExts
179+ , type' == "toc"
180+ -> mempty <$ eTOC
181+ _ | "titlepage" `T.isInfixOf` type'
182+ , name `elem` ("section" : groupingContent)
183+ -> mempty <$ eTitlePage
184+ "p" -> pPara
185+ "h1" -> pHeader
186+ "h2" -> pHeader
187+ "h3" -> pHeader
188+ "h4" -> pHeader
189+ "h5" -> pHeader
190+ "h6" -> pHeader
191+ "blockquote" -> pBlockQuote
192+ "pre" -> pCodeBlock
193+ "ul" -> pBulletList
194+ "ol" -> pOrderedList
195+ "dl" -> pDefinitionList
196+ "table" -> pTable block
197+ "hr" -> pHrule
198+ "html" -> pHtml
199+ "head" -> pHead
200+ "body" -> pBody
201+ "div"
202+ | extensionEnabled Ext_line_blocks exts
203+ , Just "line-block" <- lookup "class" attr
204+ -> pLineBlock
205+ | otherwise
206+ -> pDiv
207+ "section" -> pDiv
208+ "header" -> pDiv
209+ "main" -> pDiv
210+ "figure" -> pFigure
211+ "iframe" -> pIframe
212+ "style" -> pRawHtmlBlock
213+ "textarea" -> pRawHtmlBlock
214+ "switch"
215+ | epubExts
216+ -> eSwitch B.para block
217+ _ -> mzero
218+ _ -> mzero) <|> pPlain <|> pRawHtmlBlock) >>= \res ->
219+ res <$ trace (T.take 60 $ tshow $ B.toList res)
187220
188221namespaces :: PandocMonad m => [(Text, TagParser m Inlines)]
189222namespaces = [(mathMLNamespace, pMath True)]
@@ -256,9 +289,6 @@ eTOC = try $ do
256289 guard $ (lookup "type" attr <|> lookup "epub:type" attr) == Just "toc"
257290 void (pInTags tag block)
258291
259−pList :: PandocMonad m => TagParser m Blocks
260−pList = pBulletList <|> pOrderedList <|> pDefinitionList
261−
262292pBulletList :: PandocMonad m => TagParser m Blocks
263293pBulletList = try $ do
264294 pSatisfy (matchTagOpen "ul" [])
@@ -332,7 +362,10 @@ pDefListItem = try $ do
332362 terms <- many1 (try $ skipMany nonItem >> pInTags "dt" inline)
333363 defs <- many1 (try $ skipMany nonItem >> pInTags "dd" block)
334364 skipMany nonItem
335− let term = foldl1 (\x y -> x <> B.linebreak <> y) $ map trimInlines terms
365+ let term = foldl' (\x y -> if null x
366+ then trimInlines y
367+ else x <> B.linebreak <> trimInlines y)
368+ mempty terms
336369 return (term, map (fixPlains True) defs)
337370
338371fixPlains :: Bool -> Blocks -> Blocks
@@ -369,13 +402,16 @@ pLineBlock = try $ do
369402 B.toList ils
370403 return $ B.lineBlock lns
371404
405+isDivLike :: Text -> Bool
406+isDivLike "div" = True
407+isDivLike "section" = True
408+isDivLike "header" = True
409+isDivLike "main" = True
410+isDivLike _ = False
411+
372412pDiv :: PandocMonad m => TagParser m Blocks
373413pDiv = try $ do
374414 guardEnabled Ext_native_divs
375− let isDivLike "div" = True
376− isDivLike "section" = True
377− isDivLike "main" = True
378− isDivLike _ = False
379415 TagOpen tag attr' <- lookAhead $ pSatisfy $ tagOpen isDivLike (const True)
380416 let (ident, classes, kvs) = toAttr attr'
381417 contents <- pInTags tag block
@@ -393,11 +429,17 @@ pIframe = try $ do
393429 tag <- pSatisfy (tagOpen (=="iframe") (isJust . lookup "src"))
394430 pCloses "iframe" <|> eof
395431 url <- canonicalizeUrl $ fromAttrib "src" tag
396− (bs, _) <- openURL url
397− let inp = UTF8.toText bs
398− opts <- readerOpts <$> getState
399− Pandoc _ contents <- readHtml opts inp
400− return $ B.divWith ("",["iframe"],[]) $ B.fromList contents
432+ if T.null url
433+ then ignore $ renderTags' [tag, TagClose "iframe"]
434+ else catchError
435+ (do (bs, _) <- openURL url
436+ let inp = UTF8.toText bs
437+ opts <- readerOpts <$> getState
438+ Pandoc _ contents <- readHtml opts inp
439+ return $ B.divWith ("",["iframe"],[]) $ B.fromList contents)
440+ (\e -> do
441+ logMessage $ CouldNotFetchResource url (renderError e)
442+ ignore $ renderTags' [tag, TagClose "iframe"])
401443
402444pRawHtmlBlock :: PandocMonad m => TagParser m Blocks
403445pRawHtmlBlock = do
@@ -459,17 +501,13 @@ pHeader = try $ do
459501 tagOpen (`elem` ["h1","h2","h3","h4","h5","h6"])
460502 (const True)
461503 let attr = toStringAttr attr'
462− let bodyTitle = TagOpen tagtype attr' ~== TagOpen ("h1" :: Text)
463− [("class","title")]
464504 level <- headerLevel tagtype
465505 contents <- trimInlines . mconcat <$> manyTill inline (pCloses tagtype <|> eof)
466506 let ident = fromMaybe "" $ lookup "id" attr
467507 let classes = maybe [] T.words $ lookup "class" attr
468508 let keyvals = [(k,v) | (k,v) <- attr, k /= "class", k /= "id"]
469509 attr'' <- registerHeader (ident, classes, keyvals) contents
470− return $ if bodyTitle
471− then mempty -- skip a representation of the title in the body
472− else B.headerWith attr'' level contents
510+ return $ B.headerWith attr'' level contents
473511
474512pHrule :: PandocMonad m => TagParser m Blocks
475513pHrule = do
@@ -519,7 +557,18 @@ pFigure = try $ do
519557pCodeBlock :: PandocMonad m => TagParser m Blocks
520558pCodeBlock = try $ do
521559 TagOpen _ attr' <- pSatisfy (matchTagOpen "pre" [])
522− let attr = toAttr attr'
560+ -- if the `pre` has no attributes, try if it is followed by a `code`
561+ -- element and use those attributes if possible.
562+ attr <- case attr' of
563+ _:_ -> pure (toAttr attr')
564+ [] -> option nullAttr $ do
565+ TagOpen _ codeAttr <- pSatisfy (matchTagOpen "code" [])
566+ pure $ toAttr
567+ [ (k, v') | (k, v) <- codeAttr
568+ -- strip language from class
569+ , let v' = if k == "class"
570+ then fromMaybe v (T.stripPrefix "language-" v)
571+ else v ]
523572 contents <- manyTill pAny (pCloses "pre" <|> eof)
524573 let rawText = T.concat $ map tagToText contents
525574 -- drop leading newline if any
@@ -538,31 +587,47 @@ tagToText (TagOpen "br" _) = "\n"
538587tagToText _ = ""
539588
540589inline :: PandocMonad m => TagParser m Inlines
541−inline = choice
542− [ eNoteref
543− , eSwitch id inline
544− , pTagText
545− , pQ
546− , pEmph
547− , pStrong
548− , pSuperscript
549− , pSubscript
550− , pSpanLike
551− , pSmall
552− , pStrikeout
553− , pUnderline
554− , pLineBreak
555− , pLink
556− , pImage
557− , pSvg
558− , pBdo
559− , pCode
560− , pCodeWithClass [("samp","sample"),("var","variable")]
561− , pSpan
562− , pMath False
563− , pScriptMath
564− , pRawHtmlInline
565− ]
590+inline = pTagText <|> do
591+ tag <- lookAhead (pSatisfy isInlineTag)
592+ exts <- getOption readerExtensions
593+ case tag of
594+ TagOpen name attr ->
595+ case name of
596+ "a" | extensionEnabled Ext_epub_html_exts exts
597+ , Just "noteref" <- lookup "type" attr <|> lookup "epub:type" attr
598+ , Just ('#',_) <- lookup "href" attr >>= T.uncons
599+ -> eNoteref
600+ | otherwise -> pLink
601+ "switch" -> eSwitch id inline
602+ "q" -> pQ
603+ "em" -> pEmph
604+ "i" -> pEmph
605+ "strong" -> pStrong
606+ "b" -> pStrong
607+ "sup" -> pSuperscript
608+ "sub" -> pSubscript
609+ "small" -> pSmall
610+ "s" -> pStrikeout
611+ "strike" -> pStrikeout
612+ "del" -> pStrikeout
613+ "u" -> pUnderline
614+ "ins" -> pUnderline
615+ "br" -> pLineBreak
616+ "img" -> pImage
617+ "svg" -> pSvg
618+ "bdo" -> pBdo
619+ "code" -> pCode
620+ "samp" -> pCodeWithClass "samp" "sample"
621+ "var" -> pCodeWithClass "var" "variable"
622+ "span" -> pSpan
623+ "math" -> pMath False
624+ "script"
625+ | Just x <- lookup "type" attr
626+ , "math/tex" `T.isPrefixOf` x -> pScriptMath
627+ _ | name `elem` htmlSpanLikeElements -> pSpanLike
628+ _ -> pRawHtmlInline
629+ TagText _ -> pTagText
630+ _ -> pRawHtmlInline
566631
567632pSelfClosing :: PandocMonad m
568633 => (Text -> Bool) -> ([Attribute Text] -> Bool)
@@ -573,27 +638,25 @@ pSelfClosing f g = do
573638 return open
574639
575640pQ :: PandocMonad m => TagParser m Inlines
576−pQ = choice $ map try [citedQuote, normalQuote]
577− where citedQuote = do
578− tag <- pSatisfy $ tagOpenLit "q" (any ((=="cite") . fst))
579−
580− url <- canonicalizeUrl $ fromAttrib "cite" tag
581− let uid = fromMaybe (fromAttrib "name" tag) $
582− maybeFromAttrib "id" tag
583− let cls = T.words $ fromAttrib "class" tag
584−
585− makeQuote $ B.spanWith (uid, cls, [("cite", escapeURI url)])
586− normalQuote = do
587− pSatisfy $ tagOpenLit "q" (const True)
588− makeQuote id
589− makeQuote wrapper = do
590− ctx <- asks quoteContext
591− let (constructor, innerContext) = case ctx of
592− InDoubleQuote -> (B.singleQuoted, InSingleQuote)
593− _ -> (B.doubleQuoted, InDoubleQuote)
594−
595− content <- withQuoteContext innerContext (mconcat <$> manyTill inline (pCloses "q"))
596− return $ extractSpaces (constructor . wrapper) content
641+pQ = do
642+ TagOpen _ attrs <- pSatisfy $ tagOpenLit "q" (const True)
643+ case lookup "cite" attrs of
644+ Just url -> do
645+ let uid = fromMaybe mempty $
646+ lookup "name" attrs <> lookup "id" attrs
647+ let cls = maybe [] T.words $ lookup "class" attrs
648+ url' <- canonicalizeUrl url
649+ makeQuote $ B.spanWith (uid, cls, [("cite", escapeURI url')])
650+ Nothing -> makeQuote id
651+ where
652+ makeQuote wrapper = do
653+ ctx <- asks quoteContext
654+ let (constructor, innerContext) = case ctx of
655+ InDoubleQuote -> (B.singleQuoted, InSingleQuote)
656+ _ -> (B.doubleQuoted, InDoubleQuote)
657+ content <- withQuoteContext innerContext
658+ (mconcat <$> manyTill inline (pCloses "q"))
659+ return $ extractSpaces (constructor . wrapper) content
597660
598661pEmph :: PandocMonad m => TagParser m Inlines
599662pEmph = pInlinesInTags "em" B.emph <|> pInlinesInTags "i" B.emph
@@ -684,13 +747,12 @@ pSvg = do
684747 UTF8.toText (encode $ UTF8.fromText rawText)
685748 return $ B.imageWith (ident,cls,[]) svgData mempty mempty
686749
687−pCodeWithClass :: PandocMonad m => [(T.Text,Text)] -> TagParser m Inlines
688−pCodeWithClass elemToClass = try $ do
689− let tagTest = flip elem . fmap fst $ elemToClass
690− TagOpen open attr' <- pSatisfy $ tagOpen tagTest (const True)
750+pCodeWithClass :: PandocMonad m => Text -> Text -> TagParser m Inlines
751+pCodeWithClass name class' = try $ do
752+ TagOpen open attr' <- pSatisfy $ tagOpen (== name) (const True)
691753 result <- manyTill pAny (pCloses open)
692754 let (ids,cs,kvs) = toAttr attr'
693− cs' = maybe cs (:cs) . lookup open $ elemToClass
755+ cs' = class' : cs
694756 return . B.codeWith (ids,cs',kvs) .
695757 T.unwords . T.lines . innerText $ result
696758
@@ -777,17 +839,19 @@ pInlinesInTags tagtype f = extractSpaces f <$> pInTags tagtype inline
777839
778840pTagText :: PandocMonad m => TagParser m Inlines
779841pTagText = try $ do
842+ pos <- getPosition
780843 (TagText str) <- pSatisfy isTagText
781844 st <- getState
782845 qu <- ask
783846 parsed <- lift $ lift $
784− flip runReaderT qu $ runParserT (many pTagContents) st "text" str
847+ flip runReaderT qu $ runParserT (many pTagContents) st "text"
848+ (Sources [(pos, str)])
785849 case parsed of
786850 Left _ -> throwError $ PandocParseError $
787851 "Could not parse `" <> str <> "'"
788852 Right result -> return $ mconcat result
789853
790−type InlinesParser m = HTMLParser m Text
854+type InlinesParser m = HTMLParser m Sources
791855
792856pTagContents :: PandocMonad m => InlinesParser m Inlines
793857pTagContents =
@@ -881,27 +945,23 @@ pSpace = many1 (satisfy isSpace) >>= \xs ->
881945 then return B.softbreak
882946 else return B.space
883947
884−class NamedTag a where
885− getTagName :: a -> Maybe Text
886−
887−instance NamedTag (Tag Text) where
888− getTagName (TagOpen t _) = Just t
889− getTagName (TagClose t) = Just t
890− getTagName _ = Nothing
891−
892−instance NamedTag (Tag String) where
893− getTagName (TagOpen t _) = Just (T.pack t)
894− getTagName (TagClose t) = Just (T.pack t)
895− getTagName _ = Nothing
896−
897−isInlineTag :: NamedTag (Tag a) => Tag a -> Bool
898−isInlineTag t =
899− isCommentTag t || case getTagName t of
900− Nothing -> False
901− Just x -> x `Set.notMember` blockTags ||
902− T.take 1 x == "?" -- processing instr.
903−
904−isBlockTag :: NamedTag (Tag a) => Tag a -> Bool
948+getTagName :: Tag Text -> Maybe Text
949+getTagName (TagOpen t _) = Just t
950+getTagName (TagClose t) = Just t
951+getTagName _ = Nothing
952+
953+isInlineTag :: Tag Text -> Bool
954+isInlineTag t = isCommentTag t || case t of
955+ TagOpen "script" _ -> "math/tex" `T.isPrefixOf` fromAttrib "type" t
956+ TagClose "script" -> True
957+ TagOpen name _ -> isInlineTagName name
958+ TagClose name -> isInlineTagName name
959+ _ -> False
960+ where isInlineTagName x =
961+ x `Set.notMember` blockTags ||
962+ T.take 1 x == "?" -- processing instr.
963+
964+isBlockTag :: Tag Text -> Bool
905965isBlockTag t = isBlockTagName || isTagComment t
906966 where isBlockTagName =
907967 case getTagName t of
@@ -912,10 +972,10 @@ isBlockTag t = isBlockTagName || isTagComment t
912972 || x `Set.member` eitherBlockOrInline
913973 Nothing -> False
914974
915−isTextTag :: Tag a -> Bool
975+isTextTag :: Tag Text -> Bool
916976isTextTag = tagText (const True)
917977
918−isCommentTag :: Tag a -> Bool
978+isCommentTag :: Tag Text -> Bool
919979isCommentTag = tagComment (const True)
920980
921981--- parsers for use in markdown, textile readers
@@ -923,13 +983,14 @@ isCommentTag = tagComment (const True)
923983-- | Matches a stretch of HTML in balanced tags.
924984htmlInBalanced :: Monad m
925985 => (Tag Text -> Bool)
926− -> ParserT Text st m Text
986+ -> ParserT Sources st m Text
927987htmlInBalanced f = try $ do
928988 lookAhead (char '<')
929− inp <- getInput
930− let ts = canonicalizeTags $
931− parseTagsOptions parseOptions{ optTagWarning = True,
932− optTagPosition = True } inp
989+ sources <- getInput
990+ let ts = canonicalizeTags
991+ $ parseTagsOptions parseOptions{ optTagWarning = True,
992+ optTagPosition = True }
993+ $ sourcesToText sources
933994 case ts of
934995 (TagPosition sr sc : t@(TagOpen tn _) : rest) -> do
935996 guard $ f t
@@ -964,22 +1025,24 @@ htmlInBalanced' tagname ts = fromMaybe [] $ go 0 ts
9641025 go n (t:ts') = (t :) <$> go n ts'
9651026 go _ [] = mzero
9661027
967−hasTagWarning :: [Tag a] -> Bool
1028+hasTagWarning :: [Tag Text] -> Bool
9681029hasTagWarning (TagWarning _:_) = True
9691030hasTagWarning _ = False
9701031
9711032-- | Matches a tag meeting a certain condition.
9721033htmlTag :: (HasReaderOptions st, Monad m)
9731034 => (Tag Text -> Bool)
974− -> ParserT Text st m (Tag Text, Text)
1035+ -> ParserT Sources st m (Tag Text, Text)
9751036htmlTag f = try $ do
9761037 lookAhead (char '<')
9771038 startpos <- getPosition
978− inp <- getInput
1039+ sources <- getInput
1040+ let inp = sourcesToText sources
9791041 let ts = canonicalizeTags $ parseTagsOptions
9801042 parseOptions{ optTagWarning = False
9811043 , optTagPosition = True }
982− (inp <> " ") -- add space to ensure that
1044+ (inp <> " ")
1045+ -- add space to ensure that
9831046 -- we get a TagPosition after the tag
9841047 (next, ln, col) <- case ts of
9851048 (TagPosition{} : next : TagPosition ln col : _)
src/Text/Pandoc/Readers/HTML/Parsing.hs+1−1
@@ -2,7 +2,7 @@
22{-# LANGUAGE OverloadedStrings #-}
33{- |
44 Module : Text.Pandoc.Readers.HTML.Parsing
5− Copyright : Copyright (C) 2006-2020 John MacFarlane
5+ Copyright : Copyright (C) 2006-2021 John MacFarlane
66 License : GNU GPL, version 2 or above
77
88 Maintainer : John MacFarlane <jgm@berkeley.edu>
src/Text/Pandoc/Readers/HTML/Table.hs+37−17
@@ -3,8 +3,8 @@
33{-# LANGUAGE ViewPatterns #-}
44{- |
55 Module : Text.Pandoc.Readers.HTML.Table
6− Copyright : © 2006-2020 John MacFarlane,
7− 2020 Albert Krewinkel
6+ Copyright : © 2006-2021 John MacFarlane,
7+ 2020-2021 Albert Krewinkel
88 License : GNU GPL, version 2 or above
99
1010 Maintainer : Albert Krewinkel <albert@zeitkraut.de>
@@ -17,6 +17,8 @@ module Text.Pandoc.Readers.HTML.Table (pTable) where
1717
1818import Control.Applicative ((<|>))
1919import Data.Maybe (fromMaybe)
20+import Data.Either (lefts, rights)
21+import Data.List.NonEmpty (nonEmpty)
2022import Data.Text (Text)
2123import Text.HTML.TagSoup
2224import Text.Pandoc.Builder (Blocks)
@@ -32,34 +34,51 @@ import Text.Pandoc.Shared (onlySimpleTableCells, safeRead)
3234import qualified Data.Text as T
3335import qualified Text.Pandoc.Builder as B
3436
35−-- | Parses a @<col>@ element, returning the column's width. Defaults to
36−-- @'ColWidthDefault'@ if the width is not set or cannot be determined.
37−pCol :: PandocMonad m => TagParser m ColWidth
37+-- | Parses a @<col>@ element, returning the column's width.
38+-- An Either value is used: Left i means a "relative length" with
39+-- integral value i (see https://www.w3.org/TR/html4/types.html#h-6.6);
40+-- Right w means a regular width. Defaults to @'Right ColWidthDefault'@
41+-- if the width is not set or cannot be determined.
42+pCol :: PandocMonad m => TagParser m (Either Int ColWidth)
3843pCol = try $ do
3944 TagOpen _ attribs' <- pSatisfy (matchTagOpen "col" [])
4045 let attribs = toStringAttr attribs'
4146 skipMany pBlank
4247 optional $ pSatisfy (matchTagClose "col")
4348 skipMany pBlank
44− let width = case lookup "width" attribs of
49+ return $ case lookup "width" attribs of
4550 Nothing -> case lookup "style" attribs of
4651 Just (T.stripPrefix "width:" -> Just xs) | T.any (== '%') xs ->
47− fromMaybe 0.0 $ safeRead (T.filter
48− (`notElem` (" \t\r\n%'\";" :: [Char])) xs)
49− _ -> 0.0
52+ maybe (Right ColWidthDefault) (Right . ColWidth . (/ 100.0))
53+ $ safeRead (T.filter
54+ (`notElem` (" \t\r\n%'\";" :: [Char])) xs)
55+ _ -> Right ColWidthDefault
56+ Just (T.unsnoc -> Just (xs, '*')) ->
57+ maybe (Left 1) Left $ safeRead xs
5058 Just (T.unsnoc -> Just (xs, '%')) ->
51− fromMaybe 0.0 $ safeRead xs
52− _ -> 0.0
53− if width > 0.0
54− then return $ ColWidth $ width / 100.0
55− else return ColWidthDefault
59+ maybe (Right ColWidthDefault)
60+ (Right . ColWidth . (/ 100.0)) $ safeRead xs
61+ _ -> Right ColWidthDefault
5662
57−pColgroup :: PandocMonad m => TagParser m [ColWidth]
63+pColgroup :: PandocMonad m => TagParser m [Either Int ColWidth]
5864pColgroup = try $ do
5965 pSatisfy (matchTagOpen "colgroup" [])
6066 skipMany pBlank
6167 manyTill pCol (pCloses "colgroup" <|> eof) <* skipMany pBlank
6268
69+resolveRelativeLengths :: [Either Int ColWidth] -> [ColWidth]
70+resolveRelativeLengths ws =
71+ let remaining = 1 - sum (map getColWidth $ rights ws)
72+ relatives = sum $ lefts ws
73+ relUnit = remaining / fromIntegral relatives
74+ toColWidth (Right x) = x
75+ toColWidth (Left i) = ColWidth (fromIntegral i * relUnit)
76+ in map toColWidth ws
77+
78+getColWidth :: ColWidth -> Double
79+getColWidth ColWidthDefault = 0
80+getColWidth (ColWidth w) = w
81+
6382data CellType
6483 = HeaderCell
6584 | BodyCell
@@ -181,7 +200,8 @@ pTable :: PandocMonad m
181200pTable block = try $ do
182201 TagOpen _ attribs <- pSatisfy (matchTagOpen "table" []) <* skipMany pBlank
183202 caption <- option mempty $ pInTags "caption" block <* skipMany pBlank
184− widths <- ((mconcat <$> many1 pColgroup) <|> many pCol) <* skipMany pBlank
203+ widths <- resolveRelativeLengths <$>
204+ ((mconcat <$> many1 pColgroup) <|> many pCol) <* skipMany pBlank
185205 thead <- pTableHead block <* skipMany pBlank
186206 topfoot <- optionMaybe (pTableFoot block) <* skipMany pBlank
187207 tbodies <- many (pTableBody block) <* skipMany pBlank
@@ -216,7 +236,7 @@ normalize widths head' bodies foot = do
216236 let rows = headRows head' <> concatMap bodyRows bodies <> footRows foot
217237 let cellWidth (Cell _ _ _ (ColSpan cs) _) = cs
218238 let rowLength = foldr (\cell acc -> cellWidth cell + acc) 0 . rowCells
219− let ncols = maximum (map rowLength rows)
239+ let ncols = maybe 0 maximum $ nonEmpty $ map rowLength rows
220240 let tblType = tableType (map rowCells rows)
221241 -- fail on empty table
222242 if null rows
src/Text/Pandoc/Readers/HTML/TagCategories.hs+1−1
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.Readers.HTML.TagCategories
4− Copyright : Copyright (C) 2006-2020 John MacFarlane
4+ Copyright : Copyright (C) 2006-2021 John MacFarlane
55 License : GNU GPL, version 2 or above
66
77 Maintainer : John MacFarlane <jgm@berkeley.edu>
src/Text/Pandoc/Readers/HTML/Types.hs+1−1
@@ -2,7 +2,7 @@
22{-# LANGUAGE MultiParamTypeClasses #-}
33{- |
44 Module : Text.Pandoc.Readers.HTML.Types
5− Copyright : Copyright (C) 2006-2020 John MacFarlane
5+ Copyright : Copyright (C) 2006-2021 John MacFarlane
66 License : GNU GPL, version 2 or above
77
88 Maintainer : John MacFarlane <jgm@berkeley.edu>
src/Text/Pandoc/Readers/Haddock.hs+11−7
@@ -17,8 +17,9 @@ module Text.Pandoc.Readers.Haddock
1717
1818import Control.Monad.Except (throwError)
1919import Data.List (intersperse)
20+import Data.List.NonEmpty (nonEmpty)
2021import Data.Maybe (fromMaybe)
21−import Data.Text (Text, unpack)
22+import Data.Text (unpack)
2223import qualified Data.Text as T
2324import Documentation.Haddock.Parser
2425import Documentation.Haddock.Types as H
@@ -28,15 +29,17 @@ import Text.Pandoc.Class.PandocMonad (PandocMonad)
2829import Text.Pandoc.Definition
2930import Text.Pandoc.Error
3031import Text.Pandoc.Options
31−import Text.Pandoc.Shared (crFilter, splitTextBy, trim)
32+import Text.Pandoc.Sources (ToSources(..), sourcesToText)
33+import Text.Pandoc.Shared (splitTextBy, trim)
3234
3335
3436-- | Parse Haddock markup and return a 'Pandoc' document.
35−readHaddock :: PandocMonad m
37+readHaddock :: (PandocMonad m, ToSources a)
3638 => ReaderOptions
37− -> Text
39+ -> a
3840 -> m Pandoc
39−readHaddock opts s = case readHaddockEither opts (unpack (crFilter s)) of
41+readHaddock opts s = case readHaddockEither opts
42+ (unpack . sourcesToText . toSources $ s) of
4043 Right result -> return result
4144 Left e -> throwError e
4245
@@ -92,7 +95,7 @@ docHToBlocks d' =
9295 then ([], map toCells bodyRows)
9396 else (toCells (head headerRows),
9497 map toCells (tail headerRows ++ bodyRows))
95− colspecs = replicate (maximum (map length body))
98+ colspecs = replicate (maybe 0 maximum (nonEmpty (map length body)))
9699 (AlignDefault, ColWidthDefault)
97100 in B.table B.emptyCaption
98101 colspecs
@@ -128,7 +131,8 @@ docHToInlines isCode d' =
128131 DocIdentifier s -> B.codeWith ("",["haskell","identifier"],[]) $ T.pack s
129132 _ -> mempty
130133 DocIdentifierUnchecked s -> B.codeWith ("",["haskell","identifier"],[]) $ T.pack s
131− DocModule s -> B.codeWith ("",["haskell","module"],[]) $ T.pack s
134+ DocModule s -> B.codeWith ("",["haskell","module"],[]) $
135+ T.pack (modLinkName s)
132136 DocWarning _ -> mempty -- TODO
133137 DocEmphasis d -> B.emph (docHToInlines isCode d)
134138 DocMonospaced (DocString s) -> B.code $ T.pack s
src/Text/Pandoc/Readers/Ipynb.hs+6−4
@@ -4,7 +4,7 @@
44{-# LANGUAGE ScopedTypeVariables #-}
55{- |
66 Module : Text.Pandoc.Readers.Ipynb
7− Copyright : Copyright (C) 2019-2020 John MacFarlane
7+ Copyright : Copyright (C) 2019-2021 John MacFarlane
88 License : GNU GPL, version 2 or above
99
1010 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -39,10 +39,12 @@ import Data.Aeson as Aeson
3939import Control.Monad.Except (throwError)
4040import Text.Pandoc.Readers.Markdown (readMarkdown)
4141import qualified Text.Pandoc.UTF8 as UTF8
42+import Text.Pandoc.Sources (ToSources(..), sourcesToText)
4243
43−readIpynb :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
44−readIpynb opts t = do
45− let src = BL.fromStrict (TE.encodeUtf8 t)
44+readIpynb :: (PandocMonad m, ToSources a)
45+ => ReaderOptions -> a -> m Pandoc
46+readIpynb opts x = do
47+ let src = BL.fromStrict . TE.encodeUtf8 . sourcesToText $ toSources x
4648 case eitherDecode src of
4749 Right (notebook4 :: Notebook NbV4) -> notebookToPandoc opts notebook4
4850 Left _ ->
src/Text/Pandoc/Readers/JATS.hs+37−51
@@ -14,7 +14,9 @@ Conversion of JATS XML to 'Pandoc' document.
1414
1515module Text.Pandoc.Readers.JATS ( readJATS ) where
1616import Control.Monad.State.Strict
17−import Data.Char (isDigit, isSpace, toUpper)
17+import Control.Monad.Except (throwError)
18+import Text.Pandoc.Error (PandocError(..))
19+import Data.Char (isDigit, isSpace)
1820import Data.Default
1921import Data.Generics
2022import Data.List (foldl', intersperse)
@@ -22,15 +24,17 @@ import qualified Data.Map as Map
2224import Data.Maybe (maybeToList, fromMaybe)
2325import Data.Text (Text)
2426import qualified Data.Text as T
27+import qualified Data.Text.Lazy as TL
2528import Text.HTML.TagSoup.Entity (lookupEntity)
2629import Text.Pandoc.Builder
2730import Text.Pandoc.Class.PandocMonad (PandocMonad)
2831import Text.Pandoc.Options
29−import Text.Pandoc.Shared (crFilter, safeRead, extractSpaces)
32+import Text.Pandoc.Shared (safeRead, extractSpaces)
3033import Text.TeXMath (readMathML, writeTeX)
31−import Text.XML.Light
34+import Text.Pandoc.XML.Light
3235import qualified Data.Set as S (fromList, member)
3336import Data.Set ((\\))
37+import Text.Pandoc.Sources (ToSources(..), sourcesToText)
3438
3539type JATS m = StateT JATSState m
3640
@@ -49,42 +53,28 @@ instance Default JATSState where
4953 , jatsContent = [] }
5054
5155
52−readJATS :: PandocMonad m => ReaderOptions -> Text -> m Pandoc
56+readJATS :: (PandocMonad m, ToSources a)
57+ => ReaderOptions
58+ -> a
59+ -> m Pandoc
5360readJATS _ inp = do
54− let tree = normalizeTree . parseXML
55− $ T.unpack $ crFilter inp
61+ let sources = toSources inp
62+ tree <- either (throwError . PandocXMLError "") return $
63+ parseXMLContents (TL.fromStrict . sourcesToText $ sources)
5664 (bs, st') <- flip runStateT (def{ jatsContent = tree }) $ mapM parseBlock tree
5765 return $ Pandoc (jatsMeta st') (toList . mconcat $ bs)
5866
59−-- normalize input, consolidating adjacent Text and CRef elements
60−normalizeTree :: [Content] -> [Content]
61−normalizeTree = everywhere (mkT go)
62− where go :: [Content] -> [Content]
63− go (Text (CData CDataRaw _ _):xs) = xs
64− go (Text (CData CDataText s1 z):Text (CData CDataText s2 _):xs) =
65− Text (CData CDataText (s1 ++ s2) z):xs
66− go (Text (CData CDataText s1 z):CRef r:xs) =
67− Text (CData CDataText (s1 ++ convertEntity r) z):xs
68− go (CRef r:Text (CData CDataText s1 z):xs) =
69− Text (CData CDataText (convertEntity r ++ s1) z):xs
70− go (CRef r1:CRef r2:xs) =
71− Text (CData CDataText (convertEntity r1 ++ convertEntity r2) Nothing):xs
72− go xs = xs
73−
74−convertEntity :: String -> String
75−convertEntity e = Data.Maybe.fromMaybe (map toUpper e) (lookupEntity e)
76−
7767-- convenience function to get an attribute value, defaulting to ""
78−attrValue :: String -> Element -> Text
68+attrValue :: Text -> Element -> Text
7969attrValue attr =
8070 fromMaybe "" . maybeAttrValue attr
8171
82−maybeAttrValue :: String -> Element -> Maybe Text
72+maybeAttrValue :: Text -> Element -> Maybe Text
8373maybeAttrValue attr elt =
84− T.pack <$> lookupAttrBy (\x -> qName x == attr) (elAttribs elt)
74+ lookupAttrBy (\x -> qName x == attr) (elAttribs elt)
8575
8676-- convenience function
87−named :: String -> Element -> Bool
77+named :: Text -> Element -> Bool
8878named s e = qName (elName e) == s
8979
9080--
@@ -150,10 +140,10 @@ getBlocks e = mconcat <$>
150140
151141parseBlock :: PandocMonad m => Content -> JATS m Blocks
152142parseBlock (Text (CData CDataRaw _ _)) = return mempty -- DOCTYPE
153−parseBlock (Text (CData _ s _)) = if all isSpace s
143+parseBlock (Text (CData _ s _)) = if T.all isSpace s
154144 then return mempty
155− else return $ plain $ trimInlines $ text $ T.pack s
156−parseBlock (CRef x) = return $ plain $ str $ T.toUpper $ T.pack x
145+ else return $ plain $ trimInlines $ text s
146+parseBlock (CRef x) = return $ plain $ str $ T.toUpper x
157147parseBlock (Elem e) =
158148 case qName (elName e) of
159149 "p" -> parseMixed para (elContent e)
@@ -202,7 +192,7 @@ parseBlock (Elem e) =
202192 "" -> []
203193 x -> [x]
204194 return $ codeBlockWith (attrValue "id" e, classes', [])
205− $ trimNl $ textContentRecursive e
195+ $ trimNl $ strContentRecursive e
206196 parseBlockquote = do
207197 attrib <- case filterChild (named "attribution") e of
208198 Nothing -> return mempty
@@ -266,7 +256,7 @@ parseBlock (Elem e) =
266256 Just "center" -> AlignCenter
267257 _ -> AlignDefault
268258 let toWidth c = do
269− w <- findAttrText (unqual "colwidth") c
259+ w <- findAttr (unqual "colwidth") c
270260 n <- safeRead $ "0" <> T.filter (\x -> isDigit x || x == '.') w
271261 if n > 0 then Just n else Nothing
272262 let numrows = foldl' max 0 $ map length bodyrows
@@ -437,16 +427,10 @@ parseRef e = do
437427 Nothing -> return $ Map.insert "id" (toMetaValue refId) mempty
438428 -- TODO handle mixed-citation
439429
440−findAttrText :: QName -> Element -> Maybe Text
441−findAttrText x = fmap T.pack . findAttr x
442−
443430textContent :: Element -> Text
444−textContent = T.pack . strContent
445−
446−textContentRecursive :: Element -> Text
447−textContentRecursive = T.pack . strContentRecursive
431+textContent = strContent
448432
449−strContentRecursive :: Element -> String
433+strContentRecursive :: Element -> Text
450434strContentRecursive = strContent .
451435 (\e' -> e'{ elContent = map elementToStr $ elContent e' })
452436
@@ -455,9 +439,9 @@ elementToStr (Elem e') = Text $ CData CDataText (strContentRecursive e') Nothing
455439elementToStr x = x
456440
457441parseInline :: PandocMonad m => Content -> JATS m Inlines
458−parseInline (Text (CData _ s _)) = return $ text $ T.pack s
459−parseInline (CRef ref) =
460− return . text . maybe (T.toUpper $ T.pack ref) T.pack $ lookupEntity ref
442+parseInline (Text (CData _ s _)) = return $ text s
443+parseInline (CRef ref) = return $ maybe (text $ T.toUpper ref) (text . T.pack)
444+ $ lookupEntity (T.unpack ref)
461445parseInline (Elem e) =
462446 case qName (elName e) of
463447 "italic" -> innerInlines emph
@@ -502,9 +486,9 @@ parseInline (Elem e) =
502486 else linkWith attr ("#" <> rid) "" ils
503487 "ext-link" -> do
504488 ils <- innerInlines id
505− let title = fromMaybe "" $ findAttrText (QName "title" (Just "http://www.w3.org/1999/xlink") Nothing) e
489+ let title = fromMaybe "" $ findAttr (QName "title" (Just "http://www.w3.org/1999/xlink") Nothing) e
506490 let href = case findAttr (QName "href" (Just "http://www.w3.org/1999/xlink") Nothing) e of
507− Just h -> T.pack h
491+ Just h -> h
508492 _ -> "#" <> attrValue "rid" e
509493 let ils' = if ils == mempty then str href else ils
510494 let attr = (attrValue "id" e, [], [])
@@ -512,7 +496,8 @@ parseInline (Elem e) =
512496
513497 "disp-formula" -> formula displayMath
514498 "inline-formula" -> formula math
515− "math" | qPrefix (elName e) == Just "mml" -> return . math $ mathML e
499+ "math" | qURI (elName e) == Just "http://www.w3.org/1998/Math/MathML"
500+ -> return . math $ mathML e
516501 "tex-math" -> return . math $ textContent e
517502
518503 "email" -> return $ link ("mailto:" <> textContent e) ""
@@ -524,7 +509,7 @@ parseInline (Elem e) =
524509 where innerInlines f = extractSpaces f . mconcat <$>
525510 mapM parseInline (elContent e)
526511 mathML x =
527− case readMathML . T.pack . showElement $ everywhere (mkT removePrefix) x of
512+ case readMathML . showElement $ everywhere (mkT removePrefix) x of
528513 Left _ -> mempty
529514 Right m -> writeTeX m
530515 formula constructor = do
@@ -535,11 +520,12 @@ parseInline (Elem e) =
535520 filterChildren isMathML whereToLook
536521 return . mconcat . take 1 . map constructor $ texMaths ++ mathMLs
537522
538− isMathML x = qName (elName x) == "math" &&
539− qPrefix (elName x) == Just "mml"
523+ isMathML x = qName (elName x) == "math" &&
524+ qURI (elName x) ==
525+ Just "http://www.w3.org/1998/Math/MathML"
540526 removePrefix elname = elname { qPrefix = Nothing }
541527 codeWithLang = do
542528 let classes' = case attrValue "language" e of
543529 "" -> []
544530 l -> [l]
545− return $ codeWith (attrValue "id" e,classes',[]) $ textContentRecursive e
531+ return $ codeWith (attrValue "id" e,classes',[]) $ strContentRecursive e
src/Text/Pandoc/Readers/Jira.hs+14−10
@@ -2,7 +2,7 @@
22{-# LANGUAGE OverloadedStrings #-}
33{- |
44 Module : Text.Pandoc.Readers.Org
5− Copyright : © 2019-2020 Albert Krewinkel
5+ Copyright : © 2019-2021 Albert Krewinkel
66 License : GNU GPL, version 2 or above
77
88 Maintainer : Albert Krewinkel <tarleb+pandoc@moltkeplatz.de>
@@ -20,18 +20,20 @@ import Text.Pandoc.Builder hiding (cell)
2020import Text.Pandoc.Error (PandocError (PandocParseError))
2121import Text.Pandoc.Options (ReaderOptions)
2222import Text.Pandoc.Shared (stringify)
23−
23+import Text.Pandoc.Sources (ToSources(..), sourcesToText)
2424import qualified Text.Jira.Markup as Jira
2525
2626-- | Read Jira wiki markup.
27−readJira :: PandocMonad m
27+readJira :: (PandocMonad m, ToSources a)
2828 => ReaderOptions
29− -> Text
29+ -> a
3030 -> m Pandoc
31−readJira _opts s = case parse s of
32− Right d -> return $ jiraToPandoc d
33− Left e -> throwError . PandocParseError $
34− "Jira parse error" `append` pack (show e)
31+readJira _opts inp = do
32+ let sources = toSources inp
33+ case parse (sourcesToText sources) of
34+ Right d -> return $ jiraToPandoc d
35+ Left e -> throwError . PandocParseError $
36+ "Jira parse error" `append` pack (show e)
3537
3638jiraToPandoc :: Jira.Doc -> Pandoc
3739jiraToPandoc (Jira.Doc blks) = doc $ foldMap jiraToPandocBlocks blks
@@ -71,10 +73,10 @@ toPandocCodeBlocks langMay params txt =
7173 Nothing -> []
7274 in codeBlockWith ("", classes, map paramToPair params) txt
7375
74−-- | Create a pandoc @'Div'@
76+-- | Create a pandoc @'Div'@ from a panel.
7577toPandocDiv :: [Jira.Parameter] -> [Jira.Block] -> Blocks
7678toPandocDiv params =
77− divWith ("", [], map paramToPair params) . foldMap jiraToPandocBlocks
79+ divWith ("", ["panel"], map paramToPair params) . foldMap jiraToPandocBlocks
7880
7981paramToPair :: Jira.Parameter -> (Text, Text)
8082paramToPair (Jira.Parameter key value) = (key, value)
@@ -170,6 +172,8 @@ jiraLinkToPandoc linkType alias url =
170172 Jira.Email -> link ("mailto:" <> url') "" alias'
171173 Jira.Attachment -> linkWith ("", ["attachment"], []) url' "" alias'
172174 Jira.User -> linkWith ("", ["user-account"], []) url' "" alias'
175+ Jira.SmartCard -> linkWith ("", ["smart-card"], []) url' "" alias'
176+ Jira.SmartLink -> linkWith ("", ["smart-link"], []) url' "" alias'
173177
174178-- | Get unicode representation of a Jira icon.
175179iconUnicode :: Jira.Icon -> Text
src/Text/Pandoc/Readers/LaTeX.hs+252−1415
@@ -1,14 +1,10 @@
1−{-# LANGUAGE BangPatterns #-}
2−{-# LANGUAGE CPP #-}
3−{-# LANGUAGE FlexibleInstances #-}
4−{-# LANGUAGE MultiParamTypeClasses #-}
51{-# LANGUAGE OverloadedStrings #-}
62{-# LANGUAGE PatternGuards #-}
73{-# LANGUAGE ScopedTypeVariables #-}
84{-# LANGUAGE ViewPatterns #-}
95{- |
106 Module : Text.Pandoc.Readers.LaTeX
11− Copyright : Copyright (C) 2006-2020 John MacFarlane
7+ Copyright : Copyright (C) 2006-2021 John MacFarlane
128 License : GNU GPL, version 2 or above
139
1410 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -22,50 +18,58 @@ module Text.Pandoc.Readers.LaTeX ( readLaTeX,
2218 applyMacros,
2319 rawLaTeXInline,
2420 rawLaTeXBlock,
25− inlineCommand,
26− tokenize,
27− untokenize
21+ inlineCommand
2822 ) where
2923
3024import Control.Applicative (many, optional, (<|>))
3125import Control.Monad
3226import Control.Monad.Except (throwError)
33−import Data.Char (isDigit, isLetter, toUpper, chr)
27+import Data.Char (isDigit, isLetter, isAlphaNum, toUpper, chr)
3428import Data.Default
35−import Data.Functor (($>))
3629import Data.List (intercalate)
3730import qualified Data.Map as M
3831import Data.Maybe (fromMaybe, maybeToList)
3932import qualified Data.Set as Set
40−import qualified Data.Sequence as Seq
4133import Data.Text (Text)
4234import qualified Data.Text as T
35+import Skylighting (defaultSyntaxMap)
4336import System.FilePath (addExtension, replaceExtension, takeExtension)
44−import Text.Pandoc.BCP47 (Lang (..), renderLang)
45−import Text.Pandoc.Builder
37+import Text.Collate.Lang (renderLang)
38+import Text.Pandoc.Builder as B
4639import Text.Pandoc.Class.PandocPure (PandocPure)
4740import Text.Pandoc.Class.PandocMonad (PandocMonad (..), getResourcePath,
48− readFileFromDirs, report, setResourcePath,
49− setTranslations, translateTerm)
41+ readFileFromDirs, report,
42+ setResourcePath)
5043import Text.Pandoc.Error (PandocError (PandocParseError, PandocParsecError))
51−import Text.Pandoc.Highlighting (fromListingsLanguage, languagesByExtension)
44+import Text.Pandoc.Highlighting (languagesByExtension)
5245import Text.Pandoc.ImageSize (numUnit, showFl)
5346import Text.Pandoc.Logging
5447import Text.Pandoc.Options
5548import Text.Pandoc.Parsing hiding (blankline, many, mathDisplay, mathInline,
5649 optional, space, spaces, withRaw, (<|>))
57−import Text.Pandoc.Readers.LaTeX.Types (ExpansionPoint (..), Macro (..),
58− ArgSpec (..), Tok (..), TokType (..))
50+import Text.Pandoc.Readers.LaTeX.Types (Tok (..), TokType (..))
5951import Text.Pandoc.Readers.LaTeX.Parsing
60−import Text.Pandoc.Readers.LaTeX.Lang (polyglossiaLangToBCP47,
61− babelLangToBCP47)
62−import Text.Pandoc.Readers.LaTeX.SIunitx
52+import Text.Pandoc.Readers.LaTeX.Citation (citationCommands, cites)
53+import Text.Pandoc.Readers.LaTeX.Math (dollarsMath, inlineEnvironments,
54+ inlineEnvironment,
55+ mathDisplay, mathInline,
56+ newtheorem, theoremstyle, proof,
57+ theoremEnvironment)
58+import Text.Pandoc.Readers.LaTeX.Table (tableEnvironments)
59+import Text.Pandoc.Readers.LaTeX.Macro (macroDef)
60+import Text.Pandoc.Readers.LaTeX.Lang (inlineLanguageCommands,
61+ enquoteCommands,
62+ babelLangToBCP47, setDefaultLanguage)
63+import Text.Pandoc.Readers.LaTeX.SIunitx (siunitxCommands)
64+import Text.Pandoc.Readers.LaTeX.Inline (acronymCommands, refCommands,
65+ nameCommands, charCommands,
66+ accentCommands,
67+ biblatexInlineCommands,
68+ verbCommands, rawInlineOr,
69+ listingsLanguage)
6370import Text.Pandoc.Shared
64−import qualified Text.Pandoc.Translations as Translations
6571import Text.Pandoc.Walk
66−import qualified Text.Pandoc.Builder as B
67−import qualified Data.Text.Normalize as Normalize
68−import Safe
72+import Data.List.NonEmpty (nonEmpty)
6973
7074-- for debugging:
7175-- import Text.Pandoc.Extensions (getDefaultExtensions)
@@ -73,16 +77,17 @@ import Safe
7377-- import Debug.Trace (traceShowId)
7478
7579-- | Parse LaTeX from string and return 'Pandoc' document.
76−readLaTeX :: PandocMonad m
80+readLaTeX :: (PandocMonad m, ToSources a)
7781 => ReaderOptions -- ^ Reader options
78− -> Text -- ^ String to parse (assumes @'\n'@ line endings)
82+ -> a -- ^ Input to parse
7983 -> m Pandoc
8084readLaTeX opts ltx = do
85+ let sources = toSources ltx
8186 parsed <- runParserT parseLaTeX def{ sOptions = opts } "source"
82− (tokenize "source" (crFilter ltx))
87+ (tokenizeSources sources)
8388 case parsed of
8489 Right result -> return result
85− Left e -> throwError $ PandocParsecError ltx e
90+ Left e -> throwError $ PandocParsecError sources e
8691
8792parseLaTeX :: PandocMonad m => LP m Pandoc
8893parseLaTeX = do
@@ -93,11 +98,7 @@ parseLaTeX = do
9398 let doc' = doc bs
9499 let headerLevel (Header n _ _) = [n]
95100 headerLevel _ = []
96−#if MIN_VERSION_safe(0,3,18)
97− let bottomLevel = minimumBound 1 $ query headerLevel doc'
98−#else
99− let bottomLevel = minimumDef 1 $ query headerLevel doc'
100−#endif
101+ let bottomLevel = maybe 1 minimum $ nonEmpty $ query headerLevel doc'
101102 let adjustHeaders m (Header n attr ils) = Header (n+m) attr ils
102103 adjustHeaders _ x = x
103104 let (Pandoc _ bs') =
@@ -132,11 +133,11 @@ resolveRefs _ x = x
132133
133134
134135rawLaTeXBlock :: (PandocMonad m, HasMacros s, HasReaderOptions s)
135− => ParserT Text s m Text
136+ => ParserT Sources s m Text
136137rawLaTeXBlock = do
137138 lookAhead (try (char '\\' >> letter))
138139 inp <- getInput
139− let toks = tokenize "source" inp
140+ let toks = tokenizeSources inp
140141 snd <$> (rawLaTeXParser toks False (macroDef (const mempty)) blocks
141142 <|> rawLaTeXParser toks True
142143 (do choice (map controlSeq
@@ -163,14 +164,14 @@ beginOrEndCommand = try $ do
163164 (txt <> untokenize rawargs)
164165
165166rawLaTeXInline :: (PandocMonad m, HasMacros s, HasReaderOptions s)
166− => ParserT Text s m Text
167+ => ParserT Sources s m Text
167168rawLaTeXInline = do
168169 lookAhead (try (char '\\' >> letter))
169170 inp <- getInput
170− let toks = tokenize "source" inp
171+ let toks = tokenizeSources inp
171172 raw <- snd <$>
172173 ( rawLaTeXParser toks True
173− (mempty <$ (controlSeq "input" >> skipMany opt >> braced))
174+ (mempty <$ (controlSeq "input" >> skipMany rawopt >> braced))
174175 inlines
175176 <|> rawLaTeXParser toks True (inlineEnvironment <|> inlineCommand')
176177 inlines
@@ -178,11 +179,11 @@ rawLaTeXInline = do
178179 finalbraces <- mconcat <$> many (try (string "{}")) -- see #5439
179180 return $ raw <> T.pack finalbraces
180181
181−inlineCommand :: PandocMonad m => ParserT Text ParserState m Inlines
182+inlineCommand :: PandocMonad m => ParserT Sources ParserState m Inlines
182183inlineCommand = do
183184 lookAhead (try (char '\\' >> letter))
184185 inp <- getInput
185− let toks = tokenize "source" inp
186+ let toks = tokenizeSources inp
186187 fst <$> rawLaTeXParser toks True (inlineEnvironment <|> inlineCommand')
187188 inlines
188189
@@ -191,12 +192,6 @@ inlineCommand = do
191192word :: PandocMonad m => LP m Inlines
192193word = str . untoken <$> satisfyTok isWordTok
193194
194−regularSymbol :: PandocMonad m => LP m Inlines
195−regularSymbol = str . untoken <$> satisfyTok isRegularSymbol
196− where isRegularSymbol (Tok _ Symbol t) = not $ T.any isSpecial t
197− isRegularSymbol _ = False
198− isSpecial c = c `Set.member` specialChars
199−
200195inlineGroup :: PandocMonad m => LP m Inlines
201196inlineGroup = do
202197 ils <- grouped inline
@@ -237,19 +232,6 @@ mkImage options (T.unpack -> src) = do
237232 _ -> return src
238233 return $ imageWith attr (T.pack src') "" alt
239234
240−doxspace :: PandocMonad m => LP m Inlines
241−doxspace =
242− (space <$ lookAhead (satisfyTok startsWithLetter)) <|> return mempty
243− where startsWithLetter (Tok _ Word t) =
244− case T.uncons t of
245− Just (c, _) | isLetter c -> True
246− _ -> False
247− startsWithLetter _ = False
248−
249−
250−lit :: Text -> LP m Inlines
251−lit = pure . str
252−
253235removeDoubleQuotes :: Text -> Text
254236removeDoubleQuotes t =
255237 Data.Maybe.fromMaybe t $ T.stripPrefix "\"" t >>= T.stripSuffix "\""
@@ -296,23 +278,14 @@ quoted' f starter ender = do
296278 cs -> cs)
297279 else lit startchs
298280
299−enquote :: PandocMonad m => Bool -> Maybe Text -> LP m Inlines
300−enquote starred mblang = do
301− skipopts
302− let lang = mblang >>= babelLangToBCP47
303− let langspan = case lang of
304− Nothing -> id
305− Just l -> spanWith ("",[],[("lang", renderLang l)])
306− quoteContext <- sQuoteContext <$> getState
307− if starred || quoteContext == InDoubleQuote
308− then singleQuoted . langspan <$> withQuoteContext InSingleQuote tok
309− else doubleQuoted . langspan <$> withQuoteContext InDoubleQuote tok
281+lit :: Text -> LP m Inlines
282+lit = pure . str
310283
311284blockquote :: PandocMonad m => Bool -> Maybe Text -> LP m Blocks
312285blockquote cvariant mblang = do
313286 citepar <- if cvariant
314287 then (\xs -> para (cite xs mempty))
315− <$> cites NormalCitation False
288+ <$> cites inline NormalCitation False
316289 else option mempty $ para <$> bracketed inline
317290 let lang = mblang >>= babelLangToBCP47
318291 let langdiv = case lang of
@@ -323,224 +296,13 @@ blockquote cvariant mblang = do
323296 optional $ symbolIn (".:;?!" :: [Char]) -- currently ignored
324297 return $ blockQuote . langdiv $ (bs <> citepar)
325298
326−doAcronym :: PandocMonad m => Text -> LP m Inlines
327−doAcronym form = do
328− acro <- braced
329− return . mconcat $ [spanWith ("",[],[("acronym-label", untokenize acro),
330− ("acronym-form", "singular+" <> form)])
331− $ str $ untokenize acro]
332−
333−doAcronymPlural :: PandocMonad m => Text -> LP m Inlines
334−doAcronymPlural form = do
335− acro <- braced
336− plural <- lit "s"
337− return . mconcat $ [spanWith ("",[],[("acronym-label", untokenize acro),
338− ("acronym-form", "plural+" <> form)]) $
339− mconcat [str $ untokenize acro, plural]]
340−
341−doverb :: PandocMonad m => LP m Inlines
342−doverb = do
343− Tok _ Symbol t <- anySymbol
344− marker <- case T.uncons t of
345− Just (c, ts) | T.null ts -> return c
346− _ -> mzero
347− withVerbatimMode $
348− code . untokenize <$>
349− manyTill (notFollowedBy newlineTok >> verbTok marker) (symbol marker)
350−
351−verbTok :: PandocMonad m => Char -> LP m Tok
352−verbTok stopchar = do
353− t@(Tok pos toktype txt) <- anyTok
354− case T.findIndex (== stopchar) txt of
355− Nothing -> return t
356− Just i -> do
357− let (t1, t2) = T.splitAt i txt
358− inp <- getInput
359− setInput $ Tok (incSourceColumn pos i) Symbol (T.singleton stopchar)
360− : totoks (incSourceColumn pos (i + 1)) (T.drop 1 t2) ++ inp
361− return $ Tok pos toktype t1
362−
363−listingsLanguage :: [(Text, Text)] -> Maybe Text
364−listingsLanguage opts =
365− case lookup "language" opts of
366− Nothing -> Nothing
367− Just l -> fromListingsLanguage l `mplus` Just l
368−
369−dolstinline :: PandocMonad m => LP m Inlines
370−dolstinline = do
371− options <- option [] keyvals
372− let classes = maybeToList $ listingsLanguage options
373− doinlinecode classes
374−
375−domintinline :: PandocMonad m => LP m Inlines
376−domintinline = do
377− skipopts
378− cls <- untokenize <$> braced
379− doinlinecode [cls]
380−
381−doinlinecode :: PandocMonad m => [Text] -> LP m Inlines
382−doinlinecode classes = do
383− Tok _ Symbol t <- anySymbol
384− marker <- case T.uncons t of
385− Just (c, ts) | T.null ts -> return c
386− _ -> mzero
387− let stopchar = if marker == '{' then '}' else marker
388− withVerbatimMode $
389− codeWith ("",classes,[]) . T.map nlToSpace . untokenize <$>
390− manyTill (verbTok stopchar) (symbol stopchar)
391−
392−nlToSpace :: Char -> Char
393−nlToSpace '\n' = ' '
394−nlToSpace x = x
395−
396−mathDisplay :: Text -> Inlines
397−mathDisplay = displayMath . trimMath
398−
399−mathInline :: Text -> Inlines
400−mathInline = math . trimMath
401−
402−dollarsMath :: PandocMonad m => LP m Inlines
403−dollarsMath = do
404− symbol '$'
405− display <- option False (True <$ symbol '$')
406− (do contents <- try $ untokenize <$> pDollarsMath 0
407− if display
408− then mathDisplay contents <$ symbol '$'
409− else return $ mathInline contents)
410− <|> (guard display >> return (mathInline ""))
411−
412−-- Int is number of embedded groupings
413−pDollarsMath :: PandocMonad m => Int -> LP m [Tok]
414−pDollarsMath n = do
415− tk@(Tok _ toktype t) <- anyTok
416− case toktype of
417− Symbol | t == "$"
418− , n == 0 -> return []
419− | t == "\\" -> do
420− tk' <- anyTok
421− (tk :) . (tk' :) <$> pDollarsMath n
422− | t == "{" -> (tk :) <$> pDollarsMath (n+1)
423− | t == "}" ->
424− if n > 0
425− then (tk :) <$> pDollarsMath (n-1)
426− else mzero
427− _ -> (tk :) <$> pDollarsMath n
428−
429−-- citations
430−
431−addPrefix :: [Inline] -> [Citation] -> [Citation]
432−addPrefix p (k:ks) = k {citationPrefix = p ++ citationPrefix k} : ks
433−addPrefix _ _ = []
434−
435−addSuffix :: [Inline] -> [Citation] -> [Citation]
436−addSuffix s ks@(_:_) =
437− let k = last ks
438− in init ks ++ [k {citationSuffix = citationSuffix k ++ s}]
439−addSuffix _ _ = []
440−
441−simpleCiteArgs :: PandocMonad m => LP m [Citation]
442−simpleCiteArgs = try $ do
443− first <- optionMaybe $ toList <$> opt
444− second <- optionMaybe $ toList <$> opt
445− keys <- try $ bgroup *> manyTill citationLabel egroup
446− let (pre, suf) = case (first , second ) of
447− (Just s , Nothing) -> (mempty, s )
448− (Just s , Just t ) -> (s , t )
449− _ -> (mempty, mempty)
450− conv k = Citation { citationId = k
451− , citationPrefix = []
452− , citationSuffix = []
453− , citationMode = NormalCitation
454− , citationHash = 0
455− , citationNoteNum = 0
456− }
457− return $ addPrefix pre $ addSuffix suf $ map conv keys
458−
459−citationLabel :: PandocMonad m => LP m Text
460−citationLabel = do
461− sp
462− untokenize <$>
463− (many1 (satisfyTok isWordTok <|> symbolIn bibtexKeyChar)
464− <* sp
465− <* optional (symbol ',')
466− <* sp)
467− where bibtexKeyChar = ".:;?!`'()/*@_+=-&[]" :: [Char]
468−
469−cites :: PandocMonad m => CitationMode -> Bool -> LP m [Citation]
470−cites mode multi = try $ do
471− cits <- if multi
472− then do
473− multiprenote <- optionMaybe $ toList <$> paropt
474− multipostnote <- optionMaybe $ toList <$> paropt
475− let (pre, suf) = case (multiprenote, multipostnote) of
476− (Just s , Nothing) -> (mempty, s)
477− (Nothing , Just t) -> (mempty, t)
478− (Just s , Just t ) -> (s, t)
479− _ -> (mempty, mempty)
480− tempCits <- many1 simpleCiteArgs
481− case tempCits of
482− (k:ks) -> case ks of
483− (_:_) -> return $ (addMprenote pre k : init ks) ++
484− [addMpostnote suf (last ks)]
485− _ -> return [addMprenote pre (addMpostnote suf k)]
486− _ -> return [[]]
487− else count 1 simpleCiteArgs
488− let cs = concat cits
489− return $ case mode of
490− AuthorInText -> case cs of
491− (c:rest) -> c {citationMode = mode} : rest
492− [] -> []
493− _ -> map (\a -> a {citationMode = mode}) cs
494− where mprenote (k:ks) = (k:ks) ++ [Space]
495− mprenote _ = mempty
496− mpostnote (k:ks) = [Str ",", Space] ++ (k:ks)
497− mpostnote _ = mempty
498− addMprenote mpn (k:ks) =
499− let mpnfinal = case citationPrefix k of
500− (_:_) -> mprenote mpn
501− _ -> mpn
502− in addPrefix mpnfinal (k:ks)
503− addMprenote _ _ = []
504− addMpostnote = addSuffix . mpostnote
505−
506−citation :: PandocMonad m => Text -> CitationMode -> Bool -> LP m Inlines
507−citation name mode multi = do
508− (c,raw) <- withRaw $ cites mode multi
509− return $ cite c (rawInline "latex" $ "\\" <> name <> untokenize raw)
510−
511−handleCitationPart :: Inlines -> [Citation]
512−handleCitationPart ils =
513− let isCite Cite{} = True
514− isCite _ = False
515− (pref, rest) = break isCite (toList ils)
516− in case rest of
517− (Cite cs _:suff) -> addPrefix pref $ addSuffix suff cs
518− _ -> []
519−
520−complexNatbibCitation :: PandocMonad m => CitationMode -> LP m Inlines
521−complexNatbibCitation mode = try $ do
522− (cs, raw) <-
523− withRaw $ concat <$> do
524− bgroup
525− items <- mconcat <$>
526− many1 (notFollowedBy (symbol ';') >> inline)
527− `sepBy1` symbol ';'
528− egroup
529− return $ map handleCitationPart items
530− case cs of
531− [] -> mzero
532− (c:cits) -> return $ cite (c{ citationMode = mode }:cits)
533− (rawInline "latex" $ "\\citetext" <> untokenize raw)
534−
535−inNote :: Inlines -> Inlines
536−inNote ils =
537− note $ para $ ils <> str "."
538−
539299inlineCommand' :: PandocMonad m => LP m Inlines
540300inlineCommand' = try $ do
541301 Tok _ (CtrlSeq name) cmd <- anyControlSeq
542302 guard $ name /= "begin" && name /= "end" && name /= "and"
543− star <- option "" ("*" <$ symbol '*' <* sp)
303+ star <- if T.all isAlphaNum name
304+ then option "" ("*" <$ symbol '*' <* sp)
305+ else pure ""
544306 overlay <- option "" overlaySpecification
545307 let name' = name <> star <> overlay
546308 let names = ordNub [name', name] -- check non-starred as fallback
@@ -551,28 +313,8 @@ inlineCommand' = try $ do
551313 <|> ignore rawcommand
552314 lookupListDefault raw names inlineCommands
553315
554−
555316tok :: PandocMonad m => LP m Inlines
556−tok = try $ spaces >> grouped inline <|> inlineCommand' <|> singleChar'
557− where singleChar' = do
558− Tok _ _ t <- singleChar
559− return $ str t
560−
561−opt :: PandocMonad m => LP m Inlines
562−opt = do
563− toks <- try (sp *> bracketedToks <* sp)
564− -- now parse the toks as inlines
565− st <- getState
566− parsed <- runParserT (mconcat <$> many inline) st "bracketed option" toks
567− case parsed of
568− Right result -> return result
569− Left e -> throwError $ PandocParsecError (untokenize toks) e
570−
571−paropt :: PandocMonad m => LP m Inlines
572−paropt = parenWrapped inline
573−
574−inBrackets :: Inlines -> Inlines
575−inBrackets x = str "[" <> x <> str "]"
317+tok = tokWith inline
576318
577319unescapeURL :: Text -> Text
578320unescapeURL = T.concat . go . T.splitOn "\\"
@@ -585,379 +327,107 @@ unescapeURL = T.concat . go . T.splitOn "\\"
585327 , isEscapable c = t
586328 | otherwise = "\\" <> t
587329
588−mathEnvWith :: PandocMonad m
589− => (Inlines -> a) -> Maybe Text -> Text -> LP m a
590−mathEnvWith f innerEnv name = f . mathDisplay . inner <$> mathEnv name
591− where inner x = case innerEnv of
592− Nothing -> x
593− Just y -> "\\begin{" <> y <> "}\n" <> x <>
594− "\\end{" <> y <> "}"
595−
596−mathEnv :: PandocMonad m => Text -> LP m Text
597−mathEnv name = do
598− skipopts
599− optional blankline
600− res <- manyTill anyTok (end_ name)
601− return $ stripTrailingNewlines $ untokenize res
602−
603−inlineEnvironment :: PandocMonad m => LP m Inlines
604−inlineEnvironment = try $ do
605− controlSeq "begin"
606− name <- untokenize <$> braced
607− M.findWithDefault mzero name inlineEnvironments
608−
609−inlineEnvironments :: PandocMonad m => M.Map Text (LP m Inlines)
610−inlineEnvironments = M.fromList [
611− ("displaymath", mathEnvWith id Nothing "displaymath")
612− , ("math", math <$> mathEnv "math")
613− , ("equation", mathEnvWith id Nothing "equation")
614− , ("equation*", mathEnvWith id Nothing "equation*")
615− , ("gather", mathEnvWith id (Just "gathered") "gather")
616− , ("gather*", mathEnvWith id (Just "gathered") "gather*")
617− , ("multline", mathEnvWith id (Just "gathered") "multline")
618− , ("multline*", mathEnvWith id (Just "gathered") "multline*")
619− , ("eqnarray", mathEnvWith id (Just "aligned") "eqnarray")
620− , ("eqnarray*", mathEnvWith id (Just "aligned") "eqnarray*")
621− , ("align", mathEnvWith id (Just "aligned") "align")
622− , ("align*", mathEnvWith id (Just "aligned") "align*")
623− , ("alignat", mathEnvWith id (Just "aligned") "alignat")
624− , ("alignat*", mathEnvWith id (Just "aligned") "alignat*")
625− , ("dmath", mathEnvWith id Nothing "dmath")
626− , ("dmath*", mathEnvWith id Nothing "dmath*")
627− , ("dgroup", mathEnvWith id (Just "aligned") "dgroup")
628− , ("dgroup*", mathEnvWith id (Just "aligned") "dgroup*")
629− , ("darray", mathEnvWith id (Just "aligned") "darray")
630− , ("darray*", mathEnvWith id (Just "aligned") "darray*")
631− ]
632−
633330inlineCommands :: PandocMonad m => M.Map Text (LP m Inlines)
634−inlineCommands = M.union inlineLanguageCommands $ M.fromList
635− [ ("emph", extractSpaces emph <$> tok)
636− , ("textit", extractSpaces emph <$> tok)
637− , ("textsl", extractSpaces emph <$> tok)
638− , ("textsc", extractSpaces smallcaps <$> tok)
639− , ("textsf", extractSpaces (spanWith ("",["sans-serif"],[])) <$> tok)
640− , ("textmd", extractSpaces (spanWith ("",["medium"],[])) <$> tok)
641− , ("textrm", extractSpaces (spanWith ("",["roman"],[])) <$> tok)
642− , ("textup", extractSpaces (spanWith ("",["upright"],[])) <$> tok)
643− , ("texttt", ttfamily)
644− , ("sout", extractSpaces strikeout <$> tok)
645− , ("alert", skipopts >> spanWith ("",["alert"],[]) <$> tok) -- beamer
646− , ("lq", return (str "‘"))
647− , ("rq", return (str "’"))
648− , ("textquoteleft", return (str "‘"))
649− , ("textquoteright", return (str "’"))
650− , ("textquotedblleft", return (str "“"))
651− , ("textquotedblright", return (str "”"))
652− , ("textsuperscript", extractSpaces superscript <$> tok)
653− , ("textsubscript", extractSpaces subscript <$> tok)
654− , ("textbackslash", lit "\\")
655− , ("backslash", lit "\\")
656− , ("slash", lit "/")
657− , ("textbf", extractSpaces strong <$> tok)
658− , ("textnormal", extractSpaces (spanWith ("",["nodecor"],[])) <$> tok)
659− , ("underline", underline <$> tok)
660− , ("ldots", lit "…")
661− , ("vdots", lit "\8942")
662− , ("dots", lit "…")
663− , ("mdots", lit "…")
664− , ("sim", lit "~")
665− , ("sep", lit ",")
666− , ("label", rawInlineOr "label" dolabel)
667− , ("ref", rawInlineOr "ref" $ doref "ref")
668− , ("cref", rawInlineOr "cref" $ doref "ref") -- from cleveref.sty
669− , ("vref", rawInlineOr "vref" $ doref "ref+page") -- from varioref.sty
670− , ("eqref", rawInlineOr "eqref" $ doref "eqref") -- from amsmath.sty
671− , ("mbox", rawInlineOr "mbox" $ processHBox <$> tok)
672− , ("hbox", rawInlineOr "hbox" $ processHBox <$> tok)
673− , ("lettrine", rawInlineOr "lettrine" lettrine)
674− , ("(", mathInline . untokenize <$> manyTill anyTok (controlSeq ")"))
675− , ("[", mathDisplay . untokenize <$> manyTill anyTok (controlSeq "]"))
676− , ("ensuremath", mathInline . untokenize <$> braced)
677− , ("texorpdfstring", const <$> tok <*> tok)
678− , ("P", lit "¶")
679− , ("S", lit "§")
680− , ("$", lit "$")
681− , ("%", lit "%")
682− , ("&", lit "&")
683− , ("#", lit "#")
684− , ("_", lit "_")
685− , ("{", lit "{")
686− , ("}", lit "}")
687− , ("qed", lit "\a0\x25FB")
688− -- old TeX commands
689− , ("em", extractSpaces emph <$> inlines)
690− , ("it", extractSpaces emph <$> inlines)
691− , ("sl", extractSpaces emph <$> inlines)
692− , ("bf", extractSpaces strong <$> inlines)
693− , ("tt", code . stringify . toList <$> inlines)
694− , ("rm", inlines)
695− , ("itshape", extractSpaces emph <$> inlines)
696− , ("slshape", extractSpaces emph <$> inlines)
697− , ("scshape", extractSpaces smallcaps <$> inlines)
698− , ("bfseries", extractSpaces strong <$> inlines)
699− , ("MakeUppercase", makeUppercase <$> tok)
700− , ("MakeTextUppercase", makeUppercase <$> tok) -- textcase
701− , ("uppercase", makeUppercase <$> tok)
702− , ("MakeLowercase", makeLowercase <$> tok)
703− , ("MakeTextLowercase", makeLowercase <$> tok)
704− , ("lowercase", makeLowercase <$> tok)
705− , ("/", pure mempty) -- italic correction
706− , ("aa", lit "å")
707− , ("AA", lit "Å")
708− , ("ss", lit "ß")
709− , ("o", lit "ø")
710− , ("O", lit "Ø")
711− , ("L", lit "Ł")
712− , ("l", lit "ł")
713− , ("ae", lit "æ")
714− , ("AE", lit "Æ")
715− , ("oe", lit "œ")
716− , ("OE", lit "Œ")
717− , ("pounds", lit "£")
718− , ("euro", lit "€")
719− , ("copyright", lit "©")
720− , ("textasciicircum", lit "^")
721− , ("textasciitilde", lit "~")
722− , ("H", accent '\779' Nothing) -- hungarumlaut
723− , ("`", accent '\768' (Just '`')) -- grave
724− , ("'", accent '\769' (Just '\'')) -- acute
725− , ("^", accent '\770' (Just '^')) -- circ
726− , ("~", accent '\771' (Just '~')) -- tilde
727− , ("\"", accent '\776' Nothing) -- umlaut
728− , (".", accent '\775' Nothing) -- dot
729− , ("=", accent '\772' Nothing) -- macron
730− , ("|", accent '\781' Nothing) -- vertical line above
731− , ("b", accent '\817' Nothing) -- macron below
732− , ("c", accent '\807' Nothing) -- cedilla
733− , ("G", accent '\783' Nothing) -- doublegrave
734− , ("h", accent '\777' Nothing) -- hookabove
735− , ("d", accent '\803' Nothing) -- dotbelow
736− , ("f", accent '\785' Nothing) -- inverted breve
737− , ("r", accent '\778' Nothing) -- ringabove
738− , ("t", accent '\865' Nothing) -- double inverted breve
739− , ("U", accent '\782' Nothing) -- double vertical line above
740− , ("v", accent '\780' Nothing) -- hacek
741− , ("u", accent '\774' Nothing) -- breve
742− , ("k", accent '\808' Nothing) -- ogonek
743− , ("textogonekcentered", accent '\808' Nothing) -- ogonek
744− , ("i", lit "ı") -- dotless i
745− , ("j", lit "ȷ") -- dotless j
746− , ("newtie", accent '\785' Nothing) -- inverted breve
747− , ("textcircled", accent '\8413' Nothing) -- combining circle
748− , ("\\", linebreak <$ (do inTableCell <- sInTableCell <$> getState
749− guard $ not inTableCell
750− optional opt
751− spaces))
752− , (",", lit "\8198")
753− , ("@", pure mempty)
754− , (" ", lit "\160")
755− , ("ps", pure $ str "PS." <> space)
756− , ("TeX", lit "TeX")
757− , ("LaTeX", lit "LaTeX")
758− , ("bar", lit "|")
759− , ("textless", lit "<")
760− , ("textgreater", lit ">")
761− , ("thanks", skipopts >> note <$> grouped block)
762− , ("footnote", skipopts >> note <$> grouped block)
763− , ("passthrough", tok) -- \passthrough macro used by latex writer
764− -- for listings
765− , ("verb", doverb)
766− , ("lstinline", dolstinline)
767− , ("mintinline", domintinline)
768− , ("Verb", doverb)
769− , ("url", (\url -> link url "" (str url)) . unescapeURL . untokenize <$>
770− bracedUrl)
771− , ("nolinkurl", code . unescapeURL . untokenize <$> bracedUrl)
772− , ("href", do url <- bracedUrl
773− sp
774− link (unescapeURL $ untokenize url) "" <$> tok)
775− , ("includegraphics", do options <- option [] keyvals
776− src <- braced
777− mkImage options . unescapeURL . removeDoubleQuotes $
778− untokenize src)
779− , ("enquote*", enquote True Nothing)
780− , ("enquote", enquote False Nothing)
781− -- foreignquote is supposed to use native quote marks
782− , ("foreignquote*", braced >>= enquote True . Just . untokenize)
783− , ("foreignquote", braced >>= enquote False . Just . untokenize)
784− -- hypehnquote uses regular quotes
785− , ("hyphenquote*", braced >>= enquote True . Just . untokenize)
786− , ("hyphenquote", braced >>= enquote False . Just . untokenize)
787− , ("figurename", doTerm Translations.Figure)
788− , ("prefacename", doTerm Translations.Preface)
789− , ("refname", doTerm Translations.References)
790− , ("bibname", doTerm Translations.Bibliography)
791− , ("chaptername", doTerm Translations.Chapter)
792− , ("partname", doTerm Translations.Part)
793− , ("contentsname", doTerm Translations.Contents)
794− , ("listfigurename", doTerm Translations.ListOfFigures)
795− , ("listtablename", doTerm Translations.ListOfTables)
796− , ("indexname", doTerm Translations.Index)
797− , ("abstractname", doTerm Translations.Abstract)
798− , ("tablename", doTerm Translations.Table)
799− , ("enclname", doTerm Translations.Encl)
800− , ("ccname", doTerm Translations.Cc)
801− , ("headtoname", doTerm Translations.To)
802− , ("pagename", doTerm Translations.Page)
803− , ("seename", doTerm Translations.See)
804− , ("seealsoname", doTerm Translations.SeeAlso)
805− , ("proofname", doTerm Translations.Proof)
806− , ("glossaryname", doTerm Translations.Glossary)
807− , ("lstlistingname", doTerm Translations.Listing)
808− , ("cite", citation "cite" NormalCitation False)
809− , ("Cite", citation "Cite" NormalCitation False)
810− , ("citep", citation "citep" NormalCitation False)
811− , ("citep*", citation "citep*" NormalCitation False)
812− , ("citeal", citation "citeal" NormalCitation False)
813− , ("citealp", citation "citealp" NormalCitation False)
814− , ("citealp*", citation "citealp*" NormalCitation False)
815− , ("autocite", citation "autocite" NormalCitation False)
816− , ("smartcite", citation "smartcite" NormalCitation False)
817− , ("footcite", inNote <$> citation "footcite" NormalCitation False)
818− , ("parencite", citation "parencite" NormalCitation False)
819− , ("supercite", citation "supercite" NormalCitation False)
820− , ("footcitetext", inNote <$> citation "footcitetext" NormalCitation False)
821− , ("citeyearpar", citation "citeyearpar" SuppressAuthor False)
822− , ("citeyear", citation "citeyear" SuppressAuthor False)
823− , ("autocite*", citation "autocite*" SuppressAuthor False)
824− , ("cite*", citation "cite*" SuppressAuthor False)
825− , ("parencite*", citation "parencite*" SuppressAuthor False)
826− , ("textcite", citation "textcite" AuthorInText False)
827− , ("citet", citation "citet" AuthorInText False)
828− , ("citet*", citation "citet*" AuthorInText False)
829− , ("citealt", citation "citealt" AuthorInText False)
830− , ("citealt*", citation "citealt*" AuthorInText False)
831− , ("textcites", citation "textcites" AuthorInText True)
832− , ("cites", citation "cites" NormalCitation True)
833− , ("autocites", citation "autocites" NormalCitation True)
834− , ("footcites", inNote <$> citation "footcites" NormalCitation True)
835− , ("parencites", citation "parencites" NormalCitation True)
836− , ("supercites", citation "supercites" NormalCitation True)
837− , ("footcitetexts", inNote <$> citation "footcitetexts" NormalCitation True)
838− , ("Autocite", citation "Autocite" NormalCitation False)
839− , ("Smartcite", citation "Smartcite" NormalCitation False)
840− , ("Footcite", inNote <$> citation "Footcite" NormalCitation False)
841− , ("Parencite", citation "Parencite" NormalCitation False)
842− , ("Supercite", citation "Supercite" NormalCitation False)
843− , ("Footcitetext", inNote <$> citation "Footcitetext" NormalCitation False)
844− , ("Citeyearpar", citation "Citeyearpar" SuppressAuthor False)
845− , ("Citeyear", citation "Citeyear" SuppressAuthor False)
846− , ("Autocite*", citation "Autocite*" SuppressAuthor False)
847− , ("Cite*", citation "Cite*" SuppressAuthor False)
848− , ("Parencite*", citation "Parencite*" SuppressAuthor False)
849− , ("Textcite", citation "Textcite" AuthorInText False)
850− , ("Textcites", citation "Textcites" AuthorInText True)
851− , ("Cites", citation "Cites" NormalCitation True)
852− , ("Autocites", citation "Autocites" NormalCitation True)
853− , ("Footcites", inNote <$> citation "Footcites" NormalCitation True)
854− , ("Parencites", citation "Parencites" NormalCitation True)
855− , ("Supercites", citation "Supercites" NormalCitation True)
856− , ("Footcitetexts", inNote <$> citation "Footcitetexts" NormalCitation True)
857− , ("citetext", complexNatbibCitation NormalCitation)
858− , ("citeauthor", (try (tok *> sp *> controlSeq "citetext") *>
859− complexNatbibCitation AuthorInText)
860− <|> citation "citeauthor" AuthorInText False)
861− , ("nocite", mempty <$ (citation "nocite" NormalCitation False >>=
862− addMeta "nocite"))
863− , ("hyperlink", hyperlink)
864− , ("hypertarget", hypertargetInline)
865− -- glossaries package
866− , ("gls", doAcronym "short")
867− , ("Gls", doAcronym "short")
868− , ("glsdesc", doAcronym "long")
869− , ("Glsdesc", doAcronym "long")
870− , ("GLSdesc", doAcronym "long")
871− , ("acrlong", doAcronym "long")
872− , ("Acrlong", doAcronym "long")
873− , ("acrfull", doAcronym "full")
874− , ("Acrfull", doAcronym "full")
875− , ("acrshort", doAcronym "abbrv")
876− , ("Acrshort", doAcronym "abbrv")
877− , ("glspl", doAcronymPlural "short")
878− , ("Glspl", doAcronymPlural "short")
879− , ("glsdescplural", doAcronymPlural "long")
880− , ("Glsdescplural", doAcronymPlural "long")
881− , ("GLSdescplural", doAcronymPlural "long")
882− -- acronyms package
883− , ("ac", doAcronym "short")
884− , ("acf", doAcronym "full")
885− , ("acs", doAcronym "abbrv")
886− , ("acl", doAcronym "long")
887− , ("acp", doAcronymPlural "short")
888− , ("acfp", doAcronymPlural "full")
889− , ("acsp", doAcronymPlural "abbrv")
890− , ("aclp", doAcronymPlural "long")
891− , ("Ac", doAcronym "short")
892− , ("Acf", doAcronym "full")
893− , ("Acs", doAcronym "abbrv")
894− , ("Acl", doAcronym "long")
895− , ("Acp", doAcronymPlural "short")
896− , ("Acfp", doAcronymPlural "full")
897− , ("Acsp", doAcronymPlural "abbrv")
898− , ("Aclp", doAcronymPlural "long")
899− -- siuntix
900− , ("si", skipopts *> dosi tok)
901− , ("SI", doSI tok)
902− , ("SIrange", doSIrange True tok)
903− , ("numrange", doSIrange False tok)
904− , ("numlist", doSInumlist)
905− , ("num", doSInum)
906− , ("ang", doSIang)
907− -- hyphenat
908− , ("bshyp", lit "\\\173")
909− , ("fshyp", lit "/\173")
910− , ("dothyp", lit ".\173")
911− , ("colonhyp", lit ":\173")
912− , ("hyp", lit "-")
913− , ("nohyphens", tok)
914− , ("textnhtt", ttfamily)
915− , ("nhttfamily", ttfamily)
916− -- LaTeX colors
917− , ("textcolor", coloredInline "color")
918− , ("colorbox", coloredInline "background-color")
919− -- fontawesome
920− , ("faCheck", lit "\10003")
921− , ("faClose", lit "\10007")
922− -- xspace
923− , ("xspace", doxspace)
924− -- etoolbox
925− , ("ifstrequal", ifstrequal)
926− , ("newtoggle", braced >>= newToggle)
927− , ("toggletrue", braced >>= setToggle True)
928− , ("togglefalse", braced >>= setToggle False)
929− , ("iftoggle", try $ ifToggle >> inline)
930− -- biblatex misc
931− , ("RN", romanNumeralUpper)
932− , ("Rn", romanNumeralLower)
933− -- babel
934− , ("foreignlanguage", foreignlanguage)
935− -- include
936− , ("input", rawInlineOr "input" $ include "input")
937− -- soul package
938− , ("ul", underline <$> tok)
939− -- ulem package
940− , ("uline", underline <$> tok)
941− -- plain tex stuff that should just be passed through as raw tex
942− , ("ifdim", ifdim)
943− ]
944−
945−accent :: PandocMonad m => Char -> Maybe Char -> LP m Inlines
946−accent combiningAccent fallBack = try $ do
947− ils <- tok
948− case toList ils of
949− (Str (T.uncons -> Just (x, xs)) : ys) -> return $ fromList $
950− -- try to normalize to the combined character:
951− Str (Normalize.normalize Normalize.NFC
952− (T.pack [x, combiningAccent]) <> xs) : ys
953− [Space] -> return $ str $ T.singleton $ fromMaybe combiningAccent fallBack
954− [] -> return $ str $ T.singleton $ fromMaybe combiningAccent fallBack
955− _ -> return ils
956−
331+inlineCommands = M.unions
332+ [ accentCommands tok
333+ , citationCommands inline
334+ , siunitxCommands tok
335+ , acronymCommands
336+ , refCommands
337+ , nameCommands
338+ , verbCommands
339+ , charCommands
340+ , enquoteCommands tok
341+ , inlineLanguageCommands tok
342+ , biblatexInlineCommands tok
343+ , rest ]
344+ where
345+ rest = M.fromList
346+ [ ("emph", extractSpaces emph <$> tok)
347+ , ("textit", extractSpaces emph <$> tok)
348+ , ("textsl", extractSpaces emph <$> tok)
349+ , ("textsc", extractSpaces smallcaps <$> tok)
350+ , ("textsf", extractSpaces (spanWith ("",["sans-serif"],[])) <$> tok)
351+ , ("textmd", extractSpaces (spanWith ("",["medium"],[])) <$> tok)
352+ , ("textrm", extractSpaces (spanWith ("",["roman"],[])) <$> tok)
353+ , ("textup", extractSpaces (spanWith ("",["upright"],[])) <$> tok)
354+ , ("texttt", ttfamily)
355+ , ("sout", extractSpaces strikeout <$> tok)
356+ , ("alert", skipopts >> spanWith ("",["alert"],[]) <$> tok) -- beamer
357+ , ("textsuperscript", extractSpaces superscript <$> tok)
358+ , ("textsubscript", extractSpaces subscript <$> tok)
359+ , ("textbf", extractSpaces strong <$> tok)
360+ , ("textnormal", extractSpaces (spanWith ("",["nodecor"],[])) <$> tok)
361+ , ("underline", underline <$> tok)
362+ , ("mbox", rawInlineOr "mbox" $ processHBox <$> tok)
363+ , ("hbox", rawInlineOr "hbox" $ processHBox <$> tok)
364+ , ("lettrine", rawInlineOr "lettrine" lettrine)
365+ , ("(", mathInline . untokenize <$> manyTill anyTok (controlSeq ")"))
366+ , ("[", mathDisplay . untokenize <$> manyTill anyTok (controlSeq "]"))
367+ , ("ensuremath", mathInline . untokenize <$> braced)
368+ , ("texorpdfstring", const <$> tok <*> tok)
369+ -- old TeX commands
370+ , ("em", extractSpaces emph <$> inlines)
371+ , ("it", extractSpaces emph <$> inlines)
372+ , ("sl", extractSpaces emph <$> inlines)
373+ , ("bf", extractSpaces strong <$> inlines)
374+ , ("tt", code . stringify . toList <$> inlines)
375+ , ("rm", inlines)
376+ , ("itshape", extractSpaces emph <$> inlines)
377+ , ("slshape", extractSpaces emph <$> inlines)
378+ , ("scshape", extractSpaces smallcaps <$> inlines)
379+ , ("bfseries", extractSpaces strong <$> inlines)
380+ , ("MakeUppercase", makeUppercase <$> tok)
381+ , ("MakeTextUppercase", makeUppercase <$> tok) -- textcase
382+ , ("uppercase", makeUppercase <$> tok)
383+ , ("MakeLowercase", makeLowercase <$> tok)
384+ , ("MakeTextLowercase", makeLowercase <$> tok)
385+ , ("lowercase", makeLowercase <$> tok)
386+ , ("thanks", skipopts >> note <$> grouped block)
387+ , ("footnote", skipopts >> note <$> grouped block)
388+ , ("passthrough", tok) -- \passthrough macro used by latex writer
389+ -- for listings
390+ , ("includegraphics", do options <- option [] keyvals
391+ src <- braced
392+ mkImage options .
393+ unescapeURL .
394+ removeDoubleQuotes $ untokenize src)
395+ -- hyperref
396+ , ("url", (\url -> link url "" (str url)) . unescapeURL . untokenize <$>
397+ bracedUrl)
398+ , ("nolinkurl", code . unescapeURL . untokenize <$> bracedUrl)
399+ , ("href", do url <- bracedUrl
400+ sp
401+ link (unescapeURL $ untokenize url) "" <$> tok)
402+ , ("hyperlink", hyperlink)
403+ , ("hyperref", hyperref)
404+ , ("hypertarget", hypertargetInline)
405+ -- hyphenat
406+ , ("nohyphens", tok)
407+ , ("textnhtt", ttfamily)
408+ , ("nhttfamily", ttfamily)
409+ -- LaTeX colors
410+ , ("textcolor", coloredInline "color")
411+ , ("colorbox", coloredInline "background-color")
412+ -- etoolbox
413+ , ("ifstrequal", ifstrequal)
414+ , ("newtoggle", braced >>= newToggle)
415+ , ("toggletrue", braced >>= setToggle True)
416+ , ("togglefalse", braced >>= setToggle False)
417+ , ("iftoggle", try $ ifToggle >> inline)
418+ -- include
419+ , ("input", rawInlineOr "input" $ include "input")
420+ -- soul package
421+ , ("ul", underline <$> tok)
422+ -- ulem package
423+ , ("uline", underline <$> tok)
424+ -- plain tex stuff that should just be passed through as raw tex
425+ , ("ifdim", ifdim)
426+ ]
957427
958428lettrine :: PandocMonad m => LP m Inlines
959429lettrine = do
960− optional opt
430+ optional rawopt
961431 x <- tok
962432 y <- tok
963433 return $ extractSpaces (spanWith ("",["lettrine"],[])) x <> smallcaps y
@@ -977,32 +447,18 @@ alterStr :: (Text -> Text) -> Inline -> Inline
977447alterStr f (Str xs) = Str (f xs)
978448alterStr _ x = x
979449
980−foreignlanguage :: PandocMonad m => LP m Inlines
981−foreignlanguage = do
982− babelLang <- untokenize <$> braced
983− case babelLangToBCP47 babelLang of
984− Just lang -> spanWith ("", [], [("lang", renderLang lang)]) <$> tok
985− _ -> tok
986−
987−inlineLanguageCommands :: PandocMonad m => M.Map Text (LP m Inlines)
988−inlineLanguageCommands = M.fromList $ mk <$> M.toList polyglossiaLangToBCP47
989− where
990− mk (polyglossia, bcp47Func) =
991− ("text" <> polyglossia, inlineLanguage bcp47Func)
992−
993−inlineLanguage :: PandocMonad m => (Text -> Lang) -> LP m Inlines
994−inlineLanguage bcp47Func = do
995− o <- option "" $ T.filter (\c -> c /= '[' && c /= ']')
996− <$> rawopt
997− let lang = renderLang $ bcp47Func o
998− extractSpaces (spanWith ("", [], [("lang", lang)])) <$> tok
999−
1000450hyperlink :: PandocMonad m => LP m Inlines
1001451hyperlink = try $ do
1002452 src <- untokenize <$> braced
1003453 lab <- tok
1004454 return $ link ("#" <> src) "" lab
1005455
456+hyperref :: PandocMonad m => LP m Inlines
457+hyperref = try $ do
458+ url <- (("#" <>) . untokenize <$> try (sp *> bracketedToks <* sp))
459+ <|> untokenize <$> (bracedUrl <* bracedUrl <* bracedUrl)
460+ link url "" <$> tok
461+
1006462hypertargetBlock :: PandocMonad m => LP m Blocks
1007463hypertargetBlock = try $ do
1008464 ref <- untokenize <$> braced
@@ -1017,31 +473,6 @@ hypertargetInline = try $ do
1017473 ils <- grouped inline
1018474 return $ spanWith (ref, [], []) ils
1019475
1020−romanNumeralUpper :: (PandocMonad m) => LP m Inlines
1021−romanNumeralUpper =
1022− str . toRomanNumeral <$> romanNumeralArg
1023−
1024−romanNumeralLower :: (PandocMonad m) => LP m Inlines
1025−romanNumeralLower =
1026− str . T.toLower . toRomanNumeral <$> romanNumeralArg
1027−
1028−romanNumeralArg :: (PandocMonad m) => LP m Int
1029−romanNumeralArg = spaces *> (parser <|> inBraces)
1030− where
1031− inBraces = do
1032− symbol '{'
1033− spaces
1034− res <- parser
1035− spaces
1036− symbol '}'
1037− return res
1038− parser = do
1039− Tok _ Word s <- satisfyTok isWordTok
1040− let (digits, rest) = T.span isDigit s
1041− unless (T.null rest) $
1042− Prelude.fail "Non-digits in argument to \\Rn or \\RN"
1043− safeRead digits
1044−
1045476newToggle :: (Monoid a, PandocMonad m) => [Tok] -> LP m a
1046477newToggle name = do
1047478 updateState $ \st ->
@@ -1072,9 +503,6 @@ ifToggle = do
1072503 report $ UndefinedToggle name' pos
1073504 return ()
1074505
1075−doTerm :: PandocMonad m => Translations.Term -> LP m Inlines
1076−doTerm term = str <$> translateTerm term
1077−
1078506ifstrequal :: (PandocMonad m, Monoid a) => LP m a
1079507ifstrequal = do
1080508 str1 <- tok
@@ -1095,13 +523,6 @@ coloredInline stylename = do
1095523ttfamily :: PandocMonad m => LP m Inlines
1096524ttfamily = code . stringify . toList <$> tok
1097525
1098−rawInlineOr :: PandocMonad m => Text -> LP m Inlines -> LP m Inlines
1099−rawInlineOr name' fallback = do
1100− parseRaw <- extensionEnabled Ext_raw_tex <$> getOption readerExtensions
1101− if parseRaw
1102− then rawInline "latex" <$> getRawCommand name' ("\\" <> name')
1103− else fallback
1104−
1105526processHBox :: Inlines -> Inlines
1106527processHBox = walk convert
1107528 where
@@ -1152,73 +573,74 @@ treatAsInline = Set.fromList
1152573 , "pagebreak"
1153574 ]
1154575
1155−label :: PandocMonad m => LP m ()
1156−label = do
1157− controlSeq "label"
1158− t <- braced
1159− updateState $ \st -> st{ sLastLabel = Just $ untokenize t }
1160−
1161−dolabel :: PandocMonad m => LP m Inlines
1162−dolabel = do
1163− v <- braced
1164− let refstr = untokenize v
1165− updateState $ \st ->
1166− st{ sLastLabel = Just refstr }
1167− return $ spanWith (refstr,[],[("label", refstr)])
1168− $ inBrackets $ str $ untokenize v
1169−
1170−doref :: PandocMonad m => Text -> LP m Inlines
1171−doref cls = do
1172− v <- braced
1173− let refstr = untokenize v
1174− return $ linkWith ("",[],[ ("reference-type", cls)
1175− , ("reference", refstr)])
1176− ("#" <> refstr)
1177− ""
1178− (inBrackets $ str refstr)
1179−
1180576lookupListDefault :: (Ord k) => v -> [k] -> M.Map k v -> v
1181577lookupListDefault d = (fromMaybe d .) . lookupList
1182578 where lookupList l m = msum $ map (`M.lookup` m) l
1183579
1184580inline :: PandocMonad m => LP m Inlines
1185−inline = (mempty <$ comment)
1186− <|> (space <$ whitespace)
1187− <|> (softbreak <$ endline)
1188− <|> word
1189− <|> macroDef (rawInline "latex")
1190− <|> inlineCommand'
1191− <|> inlineEnvironment
1192− <|> inlineGroup
1193− <|> (symbol '-' *>
1194− option (str "-") (symbol '-' *>
1195− option (str "–") (str "—" <$ symbol '-')))
1196− <|> doubleQuote
1197− <|> singleQuote
1198− <|> (str "”" <$ try (symbol '\'' >> symbol '\''))
1199− <|> (str "”" <$ symbol '”')
1200− <|> (str "’" <$ symbol '\'')
1201− <|> (str "’" <$ symbol '’')
1202− <|> (str "\160" <$ symbol '~')
1203− <|> dollarsMath
1204− <|> (guardEnabled Ext_literate_haskell *> symbol '|' *> doLHSverb)
1205− <|> (str . T.singleton <$> primEscape)
1206− <|> regularSymbol
1207− <|> (do res <- symbolIn "#^'`\"[]&"
1208− pos <- getPosition
1209− let s = untoken res
1210− report $ ParsingUnescaped s pos
1211− return $ str s)
581+inline = do
582+ Tok pos toktype t <- lookAhead anyTok
583+ let symbolAsString = str . untoken <$> anySymbol
584+ let unescapedSymbolAsString =
585+ do s <- untoken <$> anySymbol
586+ report $ ParsingUnescaped s pos
587+ return $ str s
588+ case toktype of
589+ Comment -> mempty <$ comment
590+ Spaces -> space <$ whitespace
591+ Newline -> softbreak <$ endline
592+ Word -> word
593+ Esc1 -> str . T.singleton <$> primEscape
594+ Esc2 -> str . T.singleton <$> primEscape
595+ Symbol ->
596+ case t of
597+ "-" -> symbol '-' *>
598+ option (str "-") (symbol '-' *>
599+ option (str "–") (str "—" <$ symbol '-'))
600+ "'" -> symbol '\'' *>
601+ option (str "’") (str "”" <$ symbol '\'')
602+ "~" -> str "\160" <$ symbol '~'
603+ "`" -> doubleQuote <|> singleQuote <|> symbolAsString
604+ "\"" -> doubleQuote <|> singleQuote <|> symbolAsString
605+ "“" -> doubleQuote <|> symbolAsString
606+ "‘" -> singleQuote <|> symbolAsString
607+ "$" -> dollarsMath <|> unescapedSymbolAsString
608+ "|" -> (guardEnabled Ext_literate_haskell *>
609+ symbol '|' *> doLHSverb) <|> symbolAsString
610+ "{" -> inlineGroup
611+ "#" -> unescapedSymbolAsString
612+ "&" -> unescapedSymbolAsString
613+ "_" -> unescapedSymbolAsString
614+ "^" -> unescapedSymbolAsString
615+ "\\" -> mzero
616+ "}" -> mzero
617+ _ -> symbolAsString
618+ CtrlSeq _ -> macroDef (rawInline "latex")
619+ <|> inlineCommand'
620+ <|> inlineEnvironment
621+ <|> inlineGroup
622+ _ -> mzero
1212623
1213624inlines :: PandocMonad m => LP m Inlines
1214625inlines = mconcat <$> many inline
1215626
627+opt :: PandocMonad m => LP m Inlines
628+opt = do
629+ toks <- try (sp *> bracketedToks <* sp)
630+ -- now parse the toks as inlines
631+ st <- getState
632+ parsed <- runParserT (mconcat <$> many inline) st "bracketed option" toks
633+ case parsed of
634+ Right result -> return result
635+ Left e -> throwError $ PandocParsecError (toSources toks) e
636+
1216637-- block elements:
1217638
1218639preamble :: PandocMonad m => LP m Blocks
1219640preamble = mconcat <$> many preambleBlock
1220641 where preambleBlock = (mempty <$ spaces1)
1221642 <|> macroDef (rawBlock "latex")
643+ <|> filecontents
1222644 <|> (mempty <$ blockCommand)
1223645 <|> (mempty <$ braced)
1224646 <|> (do notFollowedBy (begin_ "document")
@@ -1272,6 +694,16 @@ include name = do
1272694 mapM_ (insertIncluded defaultExt) fs
1273695 return mempty
1274696
697+readFileFromTexinputs :: PandocMonad m => FilePath -> LP m (Maybe Text)
698+readFileFromTexinputs fp = do
699+ fileContentsMap <- sFileContents <$> getState
700+ case M.lookup (T.pack fp) fileContentsMap of
701+ Just t -> return (Just t)
702+ Nothing -> do
703+ dirs <- map T.unpack . splitTextBy (==':') . fromMaybe "."
704+ <$> lookupEnv "TEXINPUTS"
705+ readFileFromDirs dirs fp
706+
1275707insertIncluded :: PandocMonad m
1276708 => FilePath
1277709 -> FilePath
@@ -1281,13 +713,12 @@ insertIncluded defaultExtension f' = do
1281713 ".tex" -> f'
1282714 ".sty" -> f'
1283715 _ -> addExtension f' defaultExtension
1284− dirs <- map T.unpack . splitTextBy (==':') . fromMaybe "." <$> lookupEnv "TEXINPUTS"
1285716 pos <- getPosition
1286717 containers <- getIncludeFiles <$> getState
1287718 when (T.pack f `elem` containers) $
1288719 throwError $ PandocParseError $ T.pack $ "Include file loop at " ++ show pos
1289720 updateState $ addIncludeFile $ T.pack f
1290− mbcontents <- readFileFromDirs dirs f
721+ mbcontents <- readFileFromTexinputs f
1291722 contents <- case mbcontents of
1292723 Just s -> return s
1293724 Nothing -> do
@@ -1296,10 +727,6 @@ insertIncluded defaultExtension f' = do
1296727 getInput >>= setInput . (tokenize f contents ++)
1297728 updateState dropLatestIncludeFile
1298729
1299−addMeta :: PandocMonad m => ToMetaValue a => Text -> a -> LP m ()
1300−addMeta field val = updateState $ \st ->
1301− st{ sMeta = addMetaField field val $ sMeta st }
1302−
1303730authors :: PandocMonad m => LP m ()
1304731authors = try $ do
1305732 bgroup
@@ -1308,150 +735,6 @@ authors = try $ do
1308735 egroup
1309736 addMeta "author" (map trimInlines auths)
1310737
1311−macroDef :: (PandocMonad m, Monoid a) => (Text -> a) -> LP m a
1312−macroDef constructor = do
1313− (_, s) <- withRaw (commandDef <|> environmentDef)
1314− (constructor (untokenize s) <$
1315− guardDisabled Ext_latex_macros)
1316− <|> return mempty
1317− where commandDef = do
1318− (name, macro') <- newcommand <|> letmacro <|> defmacro
1319− guardDisabled Ext_latex_macros <|>
1320− updateState (\s -> s{ sMacros = M.insert name macro' (sMacros s) })
1321− environmentDef = do
1322− mbenv <- newenvironment
1323− case mbenv of
1324− Nothing -> return ()
1325− Just (name, macro1, macro2) ->
1326− guardDisabled Ext_latex_macros <|>
1327− do updateState $ \s -> s{ sMacros =
1328− M.insert name macro1 (sMacros s) }
1329− updateState $ \s -> s{ sMacros =
1330− M.insert ("end" <> name) macro2 (sMacros s) }
1331− -- @\newenvironment{envname}[n-args][default]{begin}{end}@
1332− -- is equivalent to
1333− -- @\newcommand{\envname}[n-args][default]{begin}@
1334− -- @\newcommand{\endenvname}@
1335−
1336−letmacro :: PandocMonad m => LP m (Text, Macro)
1337−letmacro = do
1338− controlSeq "let"
1339− (name, contents) <- withVerbatimMode $ do
1340− Tok _ (CtrlSeq name) _ <- anyControlSeq
1341− optional $ symbol '='
1342− spaces
1343− -- we first parse in verbatim mode, and then expand macros,
1344− -- because we don't want \let\foo\bar to turn into
1345− -- \let\foo hello if we have previously \def\bar{hello}
1346− contents <- bracedOrToken
1347− return (name, contents)
1348− contents' <- doMacros' 0 contents
1349− return (name, Macro ExpandWhenDefined [] Nothing contents')
1350−
1351−defmacro :: PandocMonad m => LP m (Text, Macro)
1352−defmacro = try $
1353− -- we use withVerbatimMode, because macros are to be expanded
1354− -- at point of use, not point of definition
1355− withVerbatimMode $ do
1356− controlSeq "def"
1357− Tok _ (CtrlSeq name) _ <- anyControlSeq
1358− argspecs <- many (argspecArg <|> argspecPattern)
1359− contents <- bracedOrToken
1360− return (name, Macro ExpandWhenUsed argspecs Nothing contents)
1361−
1362−argspecArg :: PandocMonad m => LP m ArgSpec
1363−argspecArg = do
1364− Tok _ (Arg i) _ <- satisfyTok isArgTok
1365− return $ ArgNum i
1366−
1367−argspecPattern :: PandocMonad m => LP m ArgSpec
1368−argspecPattern =
1369− Pattern <$> many1 (satisfyTok (\(Tok _ toktype' txt) ->
1370− (toktype' == Symbol || toktype' == Word) &&
1371− (txt /= "{" && txt /= "\\" && txt /= "}")))
1372−
1373−newcommand :: PandocMonad m => LP m (Text, Macro)
1374−newcommand = do
1375− pos <- getPosition
1376− Tok _ (CtrlSeq mtype) _ <- controlSeq "newcommand" <|>
1377− controlSeq "renewcommand" <|>
1378− controlSeq "providecommand" <|>
1379− controlSeq "DeclareMathOperator" <|>
1380− controlSeq "DeclareRobustCommand"
1381− withVerbatimMode $ do
1382− Tok _ (CtrlSeq name) txt <- do
1383− optional (symbol '*')
1384− anyControlSeq <|>
1385− (symbol '{' *> spaces *> anyControlSeq <* spaces <* symbol '}')
1386− spaces
1387− numargs <- option 0 $ try bracketedNum
1388− let argspecs = map ArgNum [1..numargs]
1389− spaces
1390− optarg <- option Nothing $ Just <$> try bracketedToks
1391− spaces
1392− contents' <- bracedOrToken
1393− let contents =
1394− case mtype of
1395− "DeclareMathOperator" ->
1396− Tok pos (CtrlSeq "mathop") "\\mathop"
1397− : Tok pos Symbol "{"
1398− : Tok pos (CtrlSeq "mathrm") "\\mathrm"
1399− : Tok pos Symbol "{"
1400− : (contents' ++
1401− [ Tok pos Symbol "}", Tok pos Symbol "}" ])
1402− _ -> contents'
1403− macros <- sMacros <$> getState
1404− case M.lookup name macros of
1405− Just macro
1406− | mtype == "newcommand" -> do
1407− report $ MacroAlreadyDefined txt pos
1408− return (name, macro)
1409− | mtype == "providecommand" -> return (name, macro)
1410− _ -> return (name, Macro ExpandWhenUsed argspecs optarg contents)
1411−
1412−newenvironment :: PandocMonad m => LP m (Maybe (Text, Macro, Macro))
1413−newenvironment = do
1414− pos <- getPosition
1415− Tok _ (CtrlSeq mtype) _ <- controlSeq "newenvironment" <|>
1416− controlSeq "renewenvironment" <|>
1417− controlSeq "provideenvironment"
1418− withVerbatimMode $ do
1419− optional $ symbol '*'
1420− spaces
1421− name <- untokenize <$> braced
1422− spaces
1423− numargs <- option 0 $ try bracketedNum
1424− spaces
1425− optarg <- option Nothing $ Just <$> try bracketedToks
1426− let argspecs = map (\i -> ArgNum i) [1..numargs]
1427− startcontents <- spaces >> bracedOrToken
1428− endcontents <- spaces >> bracedOrToken
1429− macros <- sMacros <$> getState
1430− case M.lookup name macros of
1431− Just _
1432− | mtype == "newenvironment" -> do
1433− report $ MacroAlreadyDefined name pos
1434− return Nothing
1435− | mtype == "provideenvironment" ->
1436− return Nothing
1437− _ -> return $ Just (name,
1438− Macro ExpandWhenUsed argspecs optarg startcontents,
1439− Macro ExpandWhenUsed [] Nothing endcontents)
1440−
1441−bracketedNum :: PandocMonad m => LP m Int
1442−bracketedNum = do
1443− ds <- untokenize <$> bracketedToks
1444− case safeRead ds of
1445− Just i -> return i
1446− _ -> return 0
1447−
1448−setCaption :: PandocMonad m => LP m ()
1449−setCaption = try $ do
1450− skipopts
1451− ils <- tok
1452− optional $ try $ spaces *> label
1453− updateState $ \st -> st{ sCaption = Just ils }
1454−
1455738looseItem :: PandocMonad m => LP m Blocks
1456739looseItem = do
1457740 inListItem <- sInListItem <$> getState
@@ -1465,10 +748,6 @@ epigraph = do
1465748 p2 <- grouped block
1466749 return $ divWith ("", ["epigraph"], []) (p1 <> p2)
1467750
1468−resetCaption :: PandocMonad m => LP m ()
1469−resetCaption = updateState $ \st -> st{ sCaption = Nothing
1470− , sLastLabel = Nothing }
1471−
1472751section :: PandocMonad m => Attr -> Int -> LP m Blocks
1473752section (ident, classes, kvs) lvl = do
1474753 skipopts
@@ -1561,7 +840,7 @@ blockCommands = M.fromList
1561840 , ("address", mempty <$ (skipopts *> tok >>= addMeta "address"))
1562841 , ("signature", mempty <$ (skipopts *> authors))
1563842 , ("date", mempty <$ (skipopts *> tok >>= addMeta "date"))
1564− , ("newtheorem", newtheorem)
843+ , ("newtheorem", newtheorem inline)
1565844 , ("theoremstyle", theoremstyle)
1566845 -- KOMA-Script metadata commands
1567846 , ("extratitle", mempty <$ (skipopts *> tok >>= addMeta "extratitle"))
@@ -1609,7 +888,7 @@ blockCommands = M.fromList
1609888 , ("item", looseItem)
1610889 , ("documentclass", skipopts *> braced *> preamble)
1611890 , ("centerline", para . trimInlines <$> (skipopts *> tok))
1612− , ("caption", mempty <$ setCaption)
891+ , ("caption", mempty <$ setCaption inline)
1613892 , ("bibliography", mempty <$ (skipopts *> braced >>=
1614893 addMeta "bibliography" . splitBibs . untokenize))
1615894 , ("addbibresource", mempty <$ (skipopts *> braced >>=
@@ -1647,7 +926,8 @@ blockCommands = M.fromList
1647926
1648927
1649928environments :: PandocMonad m => M.Map Text (LP m Blocks)
1650−environments = M.fromList
929+environments = M.union (tableEnvironments blocks inline) $
930+ M.fromList
1651931 [ ("document", env "document" blocks <* skipMany anyTok)
1652932 , ("abstract", mempty <$ (env "abstract" blocks >>= addMeta "abstract"))
1653933 , ("sloppypar", env "sloppypar" blocks)
@@ -1657,13 +937,6 @@ environments = M.fromList
1657937 , ("figure", env "figure" $ skipopts *> figure)
1658938 , ("subfigure", env "subfigure" $ skipopts *> tok *> figure)
1659939 , ("center", divWith ("", ["center"], []) <$> env "center" blocks)
1660− , ("longtable", env "longtable" $
1661− resetCaption *> simpTable "longtable" False >>= addTableCaption)
1662− , ("table", env "table" $
1663− skipopts *> resetCaption *> blocks >>= addTableCaption)
1664− , ("tabular*", env "tabular*" $ simpTable "tabular*" True)
1665− , ("tabularx", env "tabularx" $ simpTable "tabularx" True)
1666− , ("tabular", env "tabular" $ simpTable "tabular" False)
1667940 , ("quote", blockQuote <$> env "quote" blocks)
1668941 , ("quotation", blockQuote <$> env "quotation" blocks)
1669942 , ("verse", blockQuote <$> env "verse" blocks)
@@ -1686,7 +959,7 @@ environments = M.fromList
1686959 , ("lilypond", rawVerbEnv "lilypond")
1687960 , ("ly", rawVerbEnv "ly")
1688961 -- amsthm
1689− , ("proof", proof)
962+ , ("proof", proof blocks opt)
1690963 -- etoolbox
1691964 , ("ifstrequal", ifstrequal)
1692965 , ("newtoggle", braced >>= newToggle)
@@ -1695,131 +968,29 @@ environments = M.fromList
1695968 , ("iftoggle", try $ ifToggle >> block)
1696969 ]
1697970
1698−theoremstyle :: PandocMonad m => LP m Blocks
1699−theoremstyle = do
1700− stylename <- untokenize <$> braced
1701− let mbstyle = case stylename of
1702− "plain" -> Just PlainStyle
1703− "definition" -> Just DefinitionStyle
1704− "remark" -> Just RemarkStyle
1705− _ -> Nothing
1706− case mbstyle of
1707− Nothing -> return ()
1708− Just sty -> updateState $ \s -> s{ sLastTheoremStyle = sty }
1709− return mempty
1710−
1711−newtheorem :: PandocMonad m => LP m Blocks
1712−newtheorem = do
1713− number <- option True (False <$ symbol '*' <* sp)
971+filecontents :: PandocMonad m => LP m Blocks
972+filecontents = try $ do
973+ controlSeq "begin"
1714974 name <- untokenize <$> braced
1715− sp
1716− series <- option Nothing $ Just . untokenize <$> bracketedToks
1717− sp
1718− showName <- tok
1719− sp
1720− syncTo <- option Nothing $ Just . untokenize <$> bracketedToks
1721− sty <- sLastTheoremStyle <$> getState
1722− let spec = TheoremSpec { theoremName = showName
1723− , theoremStyle = sty
1724− , theoremSeries = series
1725− , theoremSyncTo = syncTo
1726− , theoremNumber = number
1727− , theoremLastNum = DottedNum [0] }
1728− tmap <- sTheoremMap <$> getState
1729− updateState $ \s -> s{ sTheoremMap =
1730− M.insert name spec tmap }
975+ guard $ name == "filecontents" || name == "filecontents*"
976+ skipopts
977+ fp <- untokenize <$> braced
978+ txt <- verbEnv name
979+ updateState $ \st ->
980+ st{ sFileContents = M.insert fp txt (sFileContents st) }
1731981 return mempty
1732982
1733−proof :: PandocMonad m => LP m Blocks
1734−proof = do
1735− title <- option (B.text "Proof") opt
1736− bs <- env "proof" blocks
1737− return $
1738− B.divWith ("", ["proof"], []) $
1739− addQed $ addTitle (B.emph (title <> ".")) bs
1740−
1741−addTitle :: Inlines -> Blocks -> Blocks
1742−addTitle ils bs =
1743− case B.toList bs of
1744− (Para xs : rest)
1745− -> B.fromList (Para (B.toList ils ++ (Space : xs)) : rest)
1746− _ -> B.para ils <> bs
1747−
1748−addQed :: Blocks -> Blocks
1749−addQed bs =
1750− case Seq.viewr (B.unMany bs) of
1751− s Seq.:> Para ils
1752− -> B.Many (s Seq.|> Para (ils ++ B.toList qedSign))
1753− _ -> bs <> B.para qedSign
1754− where
1755− qedSign = B.str "\xa0\x25FB"
1756−
1757983environment :: PandocMonad m => LP m Blocks
1758984environment = try $ do
1759985 controlSeq "begin"
1760986 name <- untokenize <$> braced
1761987 M.findWithDefault mzero name environments <|>
1762− theoremEnvironment name <|>
988+ theoremEnvironment blocks opt name <|>
1763989 if M.member name (inlineEnvironments
1764990 :: M.Map Text (LP PandocPure Inlines))
1765991 then mzero
1766992 else try (rawEnv name) <|> rawVerbEnv name
1767993
1768−theoremEnvironment :: PandocMonad m => Text -> LP m Blocks
1769−theoremEnvironment name = do
1770− tmap <- sTheoremMap <$> getState
1771− case M.lookup name tmap of
1772− Nothing -> mzero
1773− Just tspec -> do
1774− optTitle <- option mempty $ (\x -> space <> "(" <> x <> ")") <$> opt
1775− mblabel <- option Nothing $ Just . untokenize <$>
1776− try (spaces >> controlSeq "label" >> spaces >> braced)
1777− bs <- env name blocks
1778− number <-
1779− if theoremNumber tspec
1780− then do
1781− let name' = fromMaybe name $ theoremSeries tspec
1782− num <- getNextNumber
1783− (maybe (DottedNum [0]) theoremLastNum .
1784− M.lookup name' . sTheoremMap)
1785− updateState $ \s ->
1786− s{ sTheoremMap =
1787− M.adjust
1788− (\spec -> spec{ theoremLastNum = num })
1789− name'
1790− (sTheoremMap s)
1791− }
1792−
1793− case mblabel of
1794− Just ident ->
1795− updateState $ \s ->
1796− s{ sLabels = M.insert ident
1797− (B.toList $
1798− theoremName tspec <> "\160" <>
1799− str (renderDottedNum num)) (sLabels s) }
1800− Nothing -> return ()
1801− return $ space <> B.text (renderDottedNum num)
1802− else return mempty
1803− let titleEmph = case theoremStyle tspec of
1804− PlainStyle -> B.strong
1805− DefinitionStyle -> B.strong
1806− RemarkStyle -> B.emph
1807− let title = titleEmph (theoremName tspec <> number)
1808− <> optTitle <> "." <> space
1809− return $ divWith (fromMaybe "" mblabel, [name], []) $ addTitle title
1810− $ case theoremStyle tspec of
1811− PlainStyle -> walk italicize bs
1812− _ -> bs
1813−
1814−italicize :: Block -> Block
1815−italicize x@(Para [Image{}]) = x -- see #6925
1816−italicize (Para ils) = Para [Emph ils]
1817−italicize (Plain ils) = Plain [Emph ils]
1818−italicize x = x
1819−
1820−env :: PandocMonad m => Text -> LP m a -> LP m a
1821−env name p = p <* end_ name
1822−
1823994rawEnv :: PandocMonad m => Text -> LP m Blocks
1824995rawEnv name = do
1825996 exts <- getOption readerExtensions
@@ -1827,15 +998,17 @@ rawEnv name = do
1827998 rawOptions <- mconcat <$> many rawopt
1828999 let beginCommand = "\\begin{" <> name <> "}" <> rawOptions
18291000 pos1 <- getPosition
1830− (bs, raw) <- withRaw $ env name blocks
18311001 if parseRaw
1832− then return $ rawBlock "latex"
1002+ then do
1003+ (_, raw) <- withRaw $ env name blocks
1004+ return $ rawBlock "latex"
18331005 $ beginCommand <> untokenize raw
18341006 else do
1007+ bs <- env name blocks
18351008 report $ SkippedContent beginCommand pos1
18361009 pos2 <- getPosition
18371010 report $ SkippedContent ("\\end{" <> name <> "}") pos2
1838− return bs
1011+ return $ divWith ("",[name],[]) bs
18391012
18401013rawVerbEnv :: PandocMonad m => Text -> LP m Blocks
18411014rawVerbEnv name = do
@@ -1894,8 +1067,7 @@ inputMinted = do
18941067 pos <- getPosition
18951068 attr <- mintedAttr
18961069 f <- T.filter (/='"') . untokenize <$> braced
1897− dirs <- map T.unpack . splitTextBy (==':') . fromMaybe "." <$> lookupEnv "TEXINPUTS"
1898− mbCode <- readFileFromDirs dirs (T.unpack f)
1070+ mbCode <- readFileFromTexinputs (T.unpack f)
18991071 rawcode <- case mbCode of
19001072 Just s -> return s
19011073 Nothing -> do
@@ -1981,8 +1153,7 @@ inputListing = do
19811153 pos <- getPosition
19821154 options <- option [] keyvals
19831155 f <- T.filter (/='"') . untokenize <$> braced
1984− dirs <- map T.unpack . splitTextBy (==':') . fromMaybe "." <$> lookupEnv "TEXINPUTS"
1985− mbCode <- readFileFromDirs dirs (T.unpack f)
1156+ mbCode <- readFileFromTexinputs (T.unpack f)
19861157 codeLines <- case mbCode of
19871158 Just s -> return $ T.lines s
19881159 Nothing -> do
@@ -1991,7 +1162,8 @@ inputListing = do
19911162 let (ident,classes,kvs) = parseListingsOptions options
19921163 let classes' =
19931164 (case listingsLanguage options of
1994− Nothing -> (take 1 (languagesByExtension (T.pack $ takeExtension $ T.unpack f)) <>)
1165+ Nothing -> (take 1 (languagesByExtension defaultSyntaxMap
1166+ (T.pack $ takeExtension $ T.unpack f)) <>)
19951167 Just _ -> id) classes
19961168 let firstline = fromMaybe 1 $ lookup "firstline" options >>= safeRead
19971169 let lastline = fromMaybe (length codeLines) $
@@ -2057,358 +1229,23 @@ orderedList' = try $ do
20571229 bs <- listenv "enumerate" (many item)
20581230 return $ orderedListWith (start, style, delim) bs
20591231
2060−-- tables
2061−
2062−hline :: PandocMonad m => LP m ()
2063−hline = try $ do
2064− spaces
2065− controlSeq "hline" <|>
2066− -- booktabs rules:
2067− controlSeq "toprule" <|>
2068− controlSeq "bottomrule" <|>
2069− controlSeq "midrule" <|>
2070− controlSeq "endhead" <|>
2071− controlSeq "endfirsthead"
2072− spaces
2073− optional opt
2074− return ()
2075−
2076−lbreak :: PandocMonad m => LP m Tok
2077−lbreak = (controlSeq "\\" <|> controlSeq "tabularnewline")
2078− <* skipopts <* spaces
2079−
2080−amp :: PandocMonad m => LP m Tok
2081−amp = symbol '&'
2082−
2083−-- Split a Word into individual Symbols (for parseAligns)
2084−splitWordTok :: PandocMonad m => LP m ()
2085−splitWordTok = do
2086− inp <- getInput
2087− case inp of
2088− (Tok spos Word t : rest) ->
2089− setInput $ map (Tok spos Symbol . T.singleton) (T.unpack t) <> rest
2090− _ -> return ()
2091−
2092−parseAligns :: PandocMonad m => LP m [(Alignment, ColWidth, ([Tok], [Tok]))]
2093−parseAligns = try $ do
2094− let maybeBar = skipMany
2095− (try $ sp *> (() <$ symbol '|' <|> () <$ (symbol '@' >> braced)))
2096− let cAlign = AlignCenter <$ symbol 'c'
2097− let lAlign = AlignLeft <$ symbol 'l'
2098− let rAlign = AlignRight <$ symbol 'r'
2099− let parAlign = AlignLeft <$ symbol 'p'
2100− -- aligns from tabularx
2101− let xAlign = AlignLeft <$ symbol 'X'
2102− let mAlign = AlignLeft <$ symbol 'm'
2103− let bAlign = AlignLeft <$ symbol 'b'
2104− let alignChar = splitWordTok *> ( cAlign <|> lAlign <|> rAlign <|> parAlign
2105− <|> xAlign <|> mAlign <|> bAlign )
2106− let alignPrefix = symbol '>' >> braced
2107− let alignSuffix = symbol '<' >> braced
2108− let colWidth = try $ do
2109− symbol '{'
2110− ds <- trim . untokenize <$> manyTill anyTok (controlSeq "linewidth")
2111− spaces
2112− symbol '}'
2113− return $ safeRead ds
2114− let alignSpec = do
2115− pref <- option [] alignPrefix
2116− spaces
2117− al <- alignChar
2118− width <- colWidth <|> option Nothing (do s <- untokenize <$> braced
2119− pos <- getPosition
2120− report $ SkippedContent s pos
2121− return Nothing)
2122− spaces
2123− suff <- option [] alignSuffix
2124− return (al, width, (pref, suff))
2125− let starAlign = do -- '*{2}{r}' == 'rr', we just expand like a macro
2126− symbol '*'
2127− spaces
2128− ds <- trim . untokenize <$> braced
2129− spaces
2130− spec <- braced
2131− case safeRead ds of
2132− Just n ->
2133− getInput >>= setInput . (mconcat (replicate n spec) ++)
2134− Nothing -> Prelude.fail $ "Could not parse " <> T.unpack ds <> " as number"
2135− bgroup
2136− spaces
2137− maybeBar
2138− aligns' <- many $ try $ spaces >> optional starAlign >>
2139− (alignSpec <* maybeBar)
2140− spaces
2141− egroup
2142− spaces
2143− return $ map toSpec aligns'
2144− where
2145− toColWidth (Just w) | w > 0 = ColWidth w
2146− toColWidth _ = ColWidthDefault
2147− toSpec (x, y, z) = (x, toColWidth y, z)
2148−
2149−-- N.B. this parser returns a Row that may have erroneous empty cells
2150−-- in it. See the note above fixTableHead for details.
2151−parseTableRow :: PandocMonad m
2152− => Text -- ^ table environment name
2153− -> [([Tok], [Tok])] -- ^ pref/suffixes
2154− -> LP m Row
2155−parseTableRow envname prefsufs = do
2156− notFollowedBy (spaces *> end_ envname)
2157− -- add prefixes and suffixes in token stream:
2158− let celltoks (pref, suff) = do
2159− prefpos <- getPosition
2160− contents <- mconcat <$>
2161− many ( snd <$> withRaw (controlSeq "parbox" >> parbox) -- #5711
2162− <|>
2163− snd <$> withRaw (inlineEnvironment <|> dollarsMath)
2164− <|>
2165− (do notFollowedBy
2166− (() <$ amp <|> () <$ lbreak <|> end_ envname)
2167− count 1 anyTok) )
2168−
2169− suffpos <- getPosition
2170− option [] (count 1 amp)
2171− return $ map (setpos prefpos) pref ++ contents ++ map (setpos suffpos) suff
2172− rawcells <- mapM celltoks prefsufs
2173− oldInput <- getInput
2174− cells <- mapM (\ts -> setInput ts >> parseTableCell) rawcells
2175− setInput oldInput
2176− spaces
2177− return $ Row nullAttr cells
2178−
2179−parseTableCell :: PandocMonad m => LP m Cell
2180−parseTableCell = do
2181− spaces
2182− updateState $ \st -> st{ sInTableCell = True }
2183− cell' <- multicolumnCell
2184− <|> multirowCell
2185− <|> parseSimpleCell
2186− <|> parseEmptyCell
2187− updateState $ \st -> st{ sInTableCell = False }
2188− spaces
2189− return cell'
2190− where
2191− -- The parsing of empty cells is important in LaTeX, especially when dealing
2192− -- with multirow/multicolumn. See #6603.
2193− parseEmptyCell = spaces $> emptyCell
2194−
2195−cellAlignment :: PandocMonad m => LP m Alignment
2196−cellAlignment = skipMany (symbol '|') *> alignment <* skipMany (symbol '|')
2197− where
2198− alignment = do
2199− c <- untoken <$> singleChar
2200− return $ case c of
2201− "l" -> AlignLeft
2202− "r" -> AlignRight
2203− "c" -> AlignCenter
2204− "*" -> AlignDefault
2205− _ -> AlignDefault
2206−
2207−plainify :: Blocks -> Blocks
2208−plainify bs = case toList bs of
2209− [Para ils] -> plain (fromList ils)
2210− _ -> bs
2211−
2212−multirowCell :: PandocMonad m => LP m Cell
2213−multirowCell = controlSeq "multirow" >> do
2214− -- Full prototype for \multirow macro is:
2215− -- \multirow[vpos]{nrows}[bigstruts]{width}[vmove]{text}
2216− -- However, everything except `nrows` and `text` make
2217− -- sense in the context of the Pandoc AST
2218− _ <- optional $ symbol '[' *> cellAlignment <* symbol ']' -- vertical position
2219− nrows <- fmap (fromMaybe 1 . safeRead . untokenize) braced
2220− _ <- optional $ symbol '[' *> manyTill anyTok (symbol ']') -- bigstrut-related
2221− _ <- symbol '{' *> manyTill anyTok (symbol '}') -- Cell width
2222− _ <- optional $ symbol '[' *> manyTill anyTok (symbol ']') -- Length used for fine-tuning
2223− content <- symbol '{' *> (plainify <$> blocks) <* symbol '}'
2224− return $ cell AlignDefault (RowSpan nrows) (ColSpan 1) content
2225−
2226−multicolumnCell :: PandocMonad m => LP m Cell
2227−multicolumnCell = controlSeq "multicolumn" >> do
2228− span' <- fmap (fromMaybe 1 . safeRead . untokenize) braced
2229− alignment <- symbol '{' *> cellAlignment <* symbol '}'
2230−
2231− let singleCell = do
2232− content <- plainify <$> blocks
2233− return $ cell alignment (RowSpan 1) (ColSpan span') content
2234−
2235− -- Two possible contents: either a \multirow cell, or content.
2236− -- E.g. \multicol{1}{c}{\multirow{2}{1em}{content}}
2237− -- Note that a \multirow cell can be nested in a \multicolumn,
2238− -- but not the other way around. See #6603
2239− let nestedCell = do
2240− (Cell _ _ (RowSpan rs) _ bs) <- multirowCell
2241− return $ cell
2242− alignment
2243− (RowSpan rs)
2244− (ColSpan span')
2245− (fromList bs)
2246−
2247− symbol '{' *> (nestedCell <|> singleCell) <* symbol '}'
2248−
2249−-- Parse a simple cell, i.e. not multirow/multicol
2250−parseSimpleCell :: PandocMonad m => LP m Cell
2251−parseSimpleCell = simpleCell <$> (plainify <$> blocks)
2252−
2253−-- LaTeX tables are stored with empty cells underneath multirow cells
2254−-- denoting the grid spaces taken up by them. More specifically, if a
2255−-- cell spans m rows, then it will overwrite all the cells in the
2256−-- columns it spans for (m-1) rows underneath it, requiring padding
2257−-- cells in these places. These padding cells need to be removed for
2258−-- proper table reading. See #6603.
2259−--
2260−-- These fixTable functions do not otherwise fix up malformed
2261−-- input tables: that is left to the table builder.
2262−fixTableHead :: TableHead -> TableHead
2263−fixTableHead (TableHead attr rows) = TableHead attr rows'
2264− where
2265− rows' = fixTableRows rows
2266−
2267−fixTableBody :: TableBody -> TableBody
2268−fixTableBody (TableBody attr rhc th tb)
2269− = TableBody attr rhc th' tb'
2270− where
2271− th' = fixTableRows th
2272− tb' = fixTableRows tb
2273−
2274−fixTableRows :: [Row] -> [Row]
2275−fixTableRows = fixTableRows' $ repeat Nothing
2276− where
2277− fixTableRows' oldHang (Row attr cells : rs)
2278− = let (newHang, cells') = fixTableRow oldHang cells
2279− rs' = fixTableRows' newHang rs
2280− in Row attr cells' : rs'
2281− fixTableRows' _ [] = []
2282−
2283−-- The overhang is represented as Just (relative cell dimensions) or
2284−-- Nothing for an empty grid space.
2285−fixTableRow :: [Maybe (ColSpan, RowSpan)] -> [Cell] -> ([Maybe (ColSpan, RowSpan)], [Cell])
2286−fixTableRow oldHang cells
2287− -- If there's overhang, drop cells until their total width meets the
2288− -- width of the occupied grid spaces (or we run out)
2289− | (n, prefHang, restHang) <- splitHang oldHang
2290− , n > 0
2291− = let cells' = dropToWidth getCellW n cells
2292− (restHang', cells'') = fixTableRow restHang cells'
2293− in (prefHang restHang', cells'')
2294− -- Otherwise record the overhang of a pending cell and fix the rest
2295− -- of the row
2296− | c@(Cell _ _ h w _):cells' <- cells
2297− = let h' = max 1 h
2298− w' = max 1 w
2299− oldHang' = dropToWidth getHangW w' oldHang
2300− (newHang, cells'') = fixTableRow oldHang' cells'
2301− in (toHang w' h' <> newHang, c : cells'')
2302− | otherwise
2303− = (oldHang, [])
2304− where
2305− getCellW (Cell _ _ _ w _) = w
2306− getHangW = maybe 1 fst
2307− getCS (ColSpan n) = n
2308−
2309− toHang c r
2310− | r > 1 = [Just (c, r)]
2311− | otherwise = replicate (getCS c) Nothing
2312−
2313− -- Take the prefix of the overhang list representing filled grid
2314− -- spaces. Also return the remainder and the length of this prefix.
2315− splitHang = splitHang' 0 id
2316−
2317− splitHang' !n l (Just (c, r):xs)
2318− = splitHang' (n + c) (l . (toHang c (r-1) ++)) xs
2319− splitHang' n l xs = (n, l, xs)
2320−
2321− -- Drop list items until the total width of the dropped items
2322− -- exceeds the passed width.
2323− dropToWidth _ n l | n < 1 = l
2324− dropToWidth wproj n (c:cs) = dropToWidth wproj (n - wproj c) cs
2325− dropToWidth _ _ [] = []
2326−
2327−simpTable :: PandocMonad m => Text -> Bool -> LP m Blocks
2328−simpTable envname hasWidthParameter = try $ do
2329− when hasWidthParameter $ () <$ (spaces >> tok)
2330− skipopts
2331− colspecs <- parseAligns
2332− let (aligns, widths, prefsufs) = unzip3 colspecs
2333− optional $ controlSeq "caption" *> setCaption
2334− spaces
2335− optional label
2336− spaces
2337− optional lbreak
2338− spaces
2339− skipMany hline
2340− spaces
2341− header' <- option [] . try . fmap (:[]) $
2342− parseTableRow envname prefsufs <* lbreak <* many1 hline
2343− spaces
2344− rows <- sepEndBy (parseTableRow envname prefsufs)
2345− (lbreak <* optional (skipMany hline))
2346− spaces
2347− optional $ controlSeq "caption" *> setCaption
2348− spaces
2349− optional label
2350− spaces
2351− optional lbreak
2352− spaces
2353− lookAhead $ controlSeq "end" -- make sure we're at end
2354− let th = fixTableHead $ TableHead nullAttr header'
2355− let tbs = [fixTableBody $ TableBody nullAttr 0 [] rows]
2356− let tf = TableFoot nullAttr []
2357− return $ table emptyCaption (zip aligns widths) th tbs tf
2358−
2359−addTableCaption :: PandocMonad m => Blocks -> LP m Blocks
2360−addTableCaption = walkM go
2361− where go (Table attr c spec th tb tf) = do
2362− st <- getState
2363− let mblabel = sLastLabel st
2364− capt <- case (sCaption st, mblabel) of
2365− (Just ils, Nothing) -> return $ caption Nothing (plain ils)
2366− (Just ils, Just lab) -> do
2367− num <- getNextNumber sLastTableNum
2368− setState
2369− st{ sLastTableNum = num
2370− , sLabels = M.insert lab
2371− [Str (renderDottedNum num)]
2372− (sLabels st) }
2373− return $ caption Nothing (plain ils) -- add number??
2374− (Nothing, _) -> return c
2375− let attr' = case (attr, mblabel) of
2376− ((_,classes,kvs), Just ident) ->
2377− (ident,classes,kvs)
2378− _ -> attr
2379− return $ addAttrDiv attr' $ Table nullAttr capt spec th tb tf
2380− go x = return x
2381−
2382−-- TODO: For now we add a Div to contain table attributes, since
2383−-- most writers don't do anything yet with attributes on Table.
2384−-- This can be removed when that changes.
2385−addAttrDiv :: Attr -> Block -> Block
2386−addAttrDiv ("",[],[]) b = b
2387−addAttrDiv attr b = Div attr [b]
2388−
23891232block :: PandocMonad m => LP m Blocks
23901233block = do
2391− res <- (mempty <$ spaces1)
2392− <|> environment
2393− <|> macroDef (rawBlock "latex")
2394− <|> blockCommand
2395− <|> paragraph
2396− <|> grouped block
1234+ Tok _ toktype _ <- lookAhead anyTok
1235+ res <- (case toktype of
1236+ Newline -> mempty <$ spaces1
1237+ Spaces -> mempty <$ spaces1
1238+ Comment -> mempty <$ spaces1
1239+ Word -> paragraph
1240+ CtrlSeq "begin" -> environment
1241+ CtrlSeq _ -> macroDef (rawBlock "latex")
1242+ <|> blockCommand
1243+ _ -> mzero)
1244+ <|> paragraph
1245+ <|> grouped block
23971246 trace (T.take 60 $ tshow $ B.toList res)
23981247 return res
23991248
24001249blocks :: PandocMonad m => LP m Blocks
24011250blocks = mconcat <$> many block
24021251
2403−setDefaultLanguage :: PandocMonad m => LP m Blocks
2404−setDefaultLanguage = do
2405− o <- option "" $ T.filter (\c -> c /= '[' && c /= ']')
2406− <$> rawopt
2407− polylang <- untokenize <$> braced
2408− case M.lookup polylang polyglossiaLangToBCP47 of
2409− Nothing -> return mempty -- TODO mzero? warning?
2410− Just langFunc -> do
2411− let l = langFunc o
2412− setTranslations l
2413− updateState $ setMeta "lang" $ str (renderLang l)
2414− return mempty
src/Text/Pandoc/Readers/LaTeX/Citation.hsnew file+210
@@ -0,0 +1,210 @@
1+{-# LANGUAGE ScopedTypeVariables #-}
2+{-# LANGUAGE OverloadedStrings #-}
3+module Text.Pandoc.Readers.LaTeX.Citation
4+ ( citationCommands
5+ , cites
6+ )
7+where
8+
9+import Text.Pandoc.Class
10+import Text.Pandoc.Readers.LaTeX.Parsing
11+import Text.Pandoc.Builder as B
12+import qualified Data.Map as M
13+import Data.Text (Text)
14+import Control.Applicative ((<|>), optional, many)
15+import Control.Monad (mzero)
16+import Control.Monad.Trans (lift)
17+import Control.Monad.Except (throwError)
18+import Text.Pandoc.Error (PandocError(PandocParsecError))
19+import Text.Pandoc.Parsing hiding (blankline, many, mathDisplay, mathInline,
20+ optional, space, spaces, withRaw, (<|>))
21+
22+citationCommands :: PandocMonad m => LP m Inlines -> M.Map Text (LP m Inlines)
23+citationCommands inline =
24+ let citation = citationWith inline
25+ tok = spaces *> grouped inline
26+ in M.fromList
27+ [ ("cite", citation "cite" NormalCitation False)
28+ , ("Cite", citation "Cite" NormalCitation False)
29+ , ("citep", citation "citep" NormalCitation False)
30+ , ("citep*", citation "citep*" NormalCitation False)
31+ , ("citeal", citation "citeal" NormalCitation False)
32+ , ("citealp", citation "citealp" NormalCitation False)
33+ , ("citealp*", citation "citealp*" NormalCitation False)
34+ , ("autocite", citation "autocite" NormalCitation False)
35+ , ("smartcite", citation "smartcite" NormalCitation False)
36+ , ("footcite", inNote <$> citation "footcite" NormalCitation False)
37+ , ("parencite", citation "parencite" NormalCitation False)
38+ , ("supercite", citation "supercite" NormalCitation False)
39+ , ("footcitetext", inNote <$> citation "footcitetext" NormalCitation False)
40+ , ("citeyearpar", citation "citeyearpar" SuppressAuthor False)
41+ , ("citeyear", citation "citeyear" SuppressAuthor False)
42+ , ("autocite*", citation "autocite*" SuppressAuthor False)
43+ , ("cite*", citation "cite*" SuppressAuthor False)
44+ , ("parencite*", citation "parencite*" SuppressAuthor False)
45+ , ("textcite", citation "textcite" AuthorInText False)
46+ , ("citet", citation "citet" AuthorInText False)
47+ , ("citet*", citation "citet*" AuthorInText False)
48+ , ("citealt", citation "citealt" AuthorInText False)
49+ , ("citealt*", citation "citealt*" AuthorInText False)
50+ , ("textcites", citation "textcites" AuthorInText True)
51+ , ("cites", citation "cites" NormalCitation True)
52+ , ("autocites", citation "autocites" NormalCitation True)
53+ , ("footcites", inNote <$> citation "footcites" NormalCitation True)
54+ , ("parencites", citation "parencites" NormalCitation True)
55+ , ("supercites", citation "supercites" NormalCitation True)
56+ , ("footcitetexts", inNote <$> citation "footcitetexts" NormalCitation True)
57+ , ("Autocite", citation "Autocite" NormalCitation False)
58+ , ("Smartcite", citation "Smartcite" NormalCitation False)
59+ , ("Footcite", inNote <$> citation "Footcite" NormalCitation False)
60+ , ("Parencite", citation "Parencite" NormalCitation False)
61+ , ("Supercite", citation "Supercite" NormalCitation False)
62+ , ("Footcitetext", inNote <$> citation "Footcitetext" NormalCitation False)
63+ , ("Citeyearpar", citation "Citeyearpar" SuppressAuthor False)
64+ , ("Citeyear", citation "Citeyear" SuppressAuthor False)
65+ , ("Autocite*", citation "Autocite*" SuppressAuthor False)
66+ , ("Cite*", citation "Cite*" SuppressAuthor False)
67+ , ("Parencite*", citation "Parencite*" SuppressAuthor False)
68+ , ("Textcite", citation "Textcite" AuthorInText False)
69+ , ("Textcites", citation "Textcites" AuthorInText True)
70+ , ("Cites", citation "Cites" NormalCitation True)
71+ , ("Autocites", citation "Autocites" NormalCitation True)
72+ , ("Footcites", inNote <$> citation "Footcites" NormalCitation True)
73+ , ("Parencites", citation "Parencites" NormalCitation True)
74+ , ("Supercites", citation "Supercites" NormalCitation True)
75+ , ("Footcitetexts", inNote <$> citation "Footcitetexts" NormalCitation True)
76+ , ("citetext", complexNatbibCitation inline NormalCitation)
77+ , ("citeauthor", (try (tok *> sp *> controlSeq "citetext") *>
78+ complexNatbibCitation inline AuthorInText)
79+ <|> citation "citeauthor" AuthorInText False)
80+ , ("nocite", mempty <$ (citation "nocite" NormalCitation False >>=
81+ addMeta "nocite"))
82+ ]
83+
84+-- citations
85+
86+addPrefix :: [Inline] -> [Citation] -> [Citation]
87+addPrefix p (k:ks) = k {citationPrefix = p ++ citationPrefix k} : ks
88+addPrefix _ _ = []
89+
90+addSuffix :: [Inline] -> [Citation] -> [Citation]
91+addSuffix s ks@(_:_) =
92+ let k = last ks
93+ in init ks ++ [k {citationSuffix = citationSuffix k ++ s}]
94+addSuffix _ _ = []
95+
96+simpleCiteArgs :: forall m . PandocMonad m => LP m Inlines -> LP m [Citation]
97+simpleCiteArgs inline = try $ do
98+ first <- optionMaybe $ toList <$> opt
99+ second <- optionMaybe $ toList <$> opt
100+ keys <- try $ bgroup *> manyTill citationLabel egroup
101+ let (pre, suf) = case (first , second ) of
102+ (Just s , Nothing) -> (mempty, s )
103+ (Just s , Just t ) -> (s , t )
104+ _ -> (mempty, mempty)
105+ conv k = Citation { citationId = k
106+ , citationPrefix = []
107+ , citationSuffix = []
108+ , citationMode = NormalCitation
109+ , citationHash = 0
110+ , citationNoteNum = 0
111+ }
112+ return $ addPrefix pre $ addSuffix suf $ map conv keys
113+ where
114+ opt :: PandocMonad m => LP m Inlines
115+ opt = do
116+ toks <- try (sp *> bracketedToks <* sp)
117+ -- now parse the toks as inlines
118+ st <- getState
119+ parsed <- lift $
120+ runParserT (mconcat <$> many inline) st "bracketed option" toks
121+ case parsed of
122+ Right result -> return result
123+ Left e -> throwError $ PandocParsecError (toSources toks) e
124+
125+
126+
127+citationLabel :: PandocMonad m => LP m Text
128+citationLabel = do
129+ sp
130+ untokenize <$>
131+ (many1 (satisfyTok isWordTok <|> symbolIn bibtexKeyChar)
132+ <* sp
133+ <* optional (symbol ',')
134+ <* sp)
135+ where bibtexKeyChar = ".:;?!`'()/*@_+=-&[]" :: [Char]
136+
137+cites :: PandocMonad m
138+ => LP m Inlines -> CitationMode -> Bool -> LP m [Citation]
139+cites inline mode multi = try $ do
140+ let paropt = parenWrapped inline
141+ cits <- if multi
142+ then do
143+ multiprenote <- optionMaybe $ toList <$> paropt
144+ multipostnote <- optionMaybe $ toList <$> paropt
145+ let (pre, suf) = case (multiprenote, multipostnote) of
146+ (Just s , Nothing) -> (mempty, s)
147+ (Nothing , Just t) -> (mempty, t)
148+ (Just s , Just t ) -> (s, t)
149+ _ -> (mempty, mempty)
150+ tempCits <- many1 $ simpleCiteArgs inline
151+ case tempCits of
152+ (k:ks) -> case ks of
153+ (_:_) -> return $ (addMprenote pre k : init ks) ++
154+ [addMpostnote suf (last ks)]
155+ _ -> return [addMprenote pre (addMpostnote suf k)]
156+ _ -> return [[]]
157+ else count 1 $ simpleCiteArgs inline
158+ let cs = concat cits
159+ return $ case mode of
160+ AuthorInText -> case cs of
161+ (c:rest) -> c {citationMode = mode} : rest
162+ [] -> []
163+ _ -> map (\a -> a {citationMode = mode}) cs
164+ where mprenote (k:ks) = (k:ks) ++ [Space]
165+ mprenote _ = mempty
166+ mpostnote (k:ks) = [Str ",", Space] ++ (k:ks)
167+ mpostnote _ = mempty
168+ addMprenote mpn (k:ks) =
169+ let mpnfinal = case citationPrefix k of
170+ (_:_) -> mprenote mpn
171+ _ -> mpn
172+ in addPrefix mpnfinal (k:ks)
173+ addMprenote _ _ = []
174+ addMpostnote = addSuffix . mpostnote
175+
176+citationWith :: PandocMonad m
177+ => LP m Inlines -> Text -> CitationMode -> Bool -> LP m Inlines
178+citationWith inline name mode multi = do
179+ (c,raw) <- withRaw $ cites inline mode multi
180+ return $ cite c (rawInline "latex" $ "\\" <> name <> untokenize raw)
181+
182+handleCitationPart :: Inlines -> [Citation]
183+handleCitationPart ils =
184+ let isCite Cite{} = True
185+ isCite _ = False
186+ (pref, rest) = break isCite (toList ils)
187+ in case rest of
188+ (Cite cs _:suff) -> addPrefix pref $ addSuffix suff cs
189+ _ -> []
190+
191+complexNatbibCitation :: PandocMonad m
192+ => LP m Inlines -> CitationMode -> LP m Inlines
193+complexNatbibCitation inline mode = try $ do
194+ (cs, raw) <-
195+ withRaw $ concat <$> do
196+ bgroup
197+ items <- mconcat <$>
198+ many1 (notFollowedBy (symbol ';') >> inline)
199+ `sepBy1` symbol ';'
200+ egroup
201+ return $ map handleCitationPart items
202+ case cs of
203+ [] -> mzero
204+ (c:cits) -> return $ cite (c{ citationMode = mode }:cits)
205+ (rawInline "latex" $ "\\citetext" <> untokenize raw)
206+
207+inNote :: Inlines -> Inlines
208+inNote ils =
209+ note $ para $ ils <> str "."
210+
src/Text/Pandoc/Readers/LaTeX/Inline.hsnew file+397
@@ -0,0 +1,397 @@
1+{-# LANGUAGE OverloadedStrings #-}
2+{-# LANGUAGE ViewPatterns #-}
3+{- |
4+ Module : Text.Pandoc.Readers.LaTeX.Inline
5+ Copyright : Copyright (C) 2006-2021 John MacFarlane
6+ License : GNU GPL, version 2 or above
7+
8+ Maintainer : John MacFarlane <jgm@berkeley.edu>
9+ Stability : alpha
10+ Portability : portable
11+-}
12+module Text.Pandoc.Readers.LaTeX.Inline
13+ ( acronymCommands
14+ , verbCommands
15+ , charCommands
16+ , accentCommands
17+ , nameCommands
18+ , biblatexInlineCommands
19+ , refCommands
20+ , rawInlineOr
21+ , listingsLanguage
22+ )
23+where
24+
25+import qualified Data.Map as M
26+import Data.Text (Text)
27+import qualified Data.Text as T
28+import Text.Pandoc.Builder
29+import Text.Pandoc.Shared (toRomanNumeral, safeRead)
30+import Text.Pandoc.Readers.LaTeX.Types (Tok (..), TokType (..))
31+import Control.Applicative (optional, (<|>))
32+import Control.Monad (guard, mzero, mplus, unless)
33+import Text.Pandoc.Class.PandocMonad (PandocMonad (..), translateTerm)
34+import Text.Pandoc.Readers.LaTeX.Parsing
35+import Text.Pandoc.Extensions (extensionEnabled, Extension(..))
36+import Text.Pandoc.Parsing (getOption, updateState, getState, notFollowedBy,
37+ manyTill, getInput, setInput, incSourceColumn,
38+ option, many1, try)
39+import Data.Char (isDigit)
40+import Text.Pandoc.Highlighting (fromListingsLanguage,)
41+import Data.Maybe (maybeToList, fromMaybe)
42+import Text.Pandoc.Options (ReaderOptions(..))
43+import qualified Data.Text.Normalize as Normalize
44+import qualified Text.Pandoc.Translations as Translations
45+
46+rawInlineOr :: PandocMonad m => Text -> LP m Inlines -> LP m Inlines
47+rawInlineOr name' fallback = do
48+ parseRaw <- extensionEnabled Ext_raw_tex <$> getOption readerExtensions
49+ if parseRaw
50+ then rawInline "latex" <$> getRawCommand name' ("\\" <> name')
51+ else fallback
52+
53+dolabel :: PandocMonad m => LP m Inlines
54+dolabel = do
55+ v <- braced
56+ let refstr = untokenize v
57+ updateState $ \st ->
58+ st{ sLastLabel = Just refstr }
59+ return $ spanWith (refstr,[],[("label", refstr)])
60+ $ inBrackets $ str $ untokenize v
61+
62+doref :: PandocMonad m => Text -> LP m Inlines
63+doref cls = do
64+ v <- braced
65+ let refstr = untokenize v
66+ return $ linkWith ("",[],[ ("reference-type", cls)
67+ , ("reference", refstr)])
68+ ("#" <> refstr)
69+ ""
70+ (inBrackets $ str refstr)
71+
72+inBrackets :: Inlines -> Inlines
73+inBrackets x = str "[" <> x <> str "]"
74+
75+doTerm :: PandocMonad m => Translations.Term -> LP m Inlines
76+doTerm term = str <$> translateTerm term
77+
78+lit :: Text -> LP m Inlines
79+lit = pure . str
80+
81+doverb :: PandocMonad m => LP m Inlines
82+doverb = do
83+ Tok _ Symbol t <- anySymbol
84+ marker <- case T.uncons t of
85+ Just (c, ts) | T.null ts -> return c
86+ _ -> mzero
87+ withVerbatimMode $
88+ code . untokenize <$>
89+ manyTill (notFollowedBy newlineTok >> verbTok marker) (symbol marker)
90+
91+verbTok :: PandocMonad m => Char -> LP m Tok
92+verbTok stopchar = do
93+ t@(Tok pos toktype txt) <- anyTok
94+ case T.findIndex (== stopchar) txt of
95+ Nothing -> return t
96+ Just i -> do
97+ let (t1, t2) = T.splitAt i txt
98+ inp <- getInput
99+ setInput $ Tok (incSourceColumn pos i) Symbol (T.singleton stopchar)
100+ : totoks (incSourceColumn pos (i + 1)) (T.drop 1 t2) ++ inp
101+ return $ Tok pos toktype t1
102+
103+listingsLanguage :: [(Text, Text)] -> Maybe Text
104+listingsLanguage opts =
105+ case lookup "language" opts of
106+ Nothing -> Nothing
107+ Just l -> fromListingsLanguage l `mplus` Just l
108+
109+dolstinline :: PandocMonad m => LP m Inlines
110+dolstinline = do
111+ options <- option [] keyvals
112+ let classes = maybeToList $ listingsLanguage options
113+ doinlinecode classes
114+
115+domintinline :: PandocMonad m => LP m Inlines
116+domintinline = do
117+ skipopts
118+ cls <- untokenize <$> braced
119+ doinlinecode [cls]
120+
121+doinlinecode :: PandocMonad m => [Text] -> LP m Inlines
122+doinlinecode classes = do
123+ Tok _ Symbol t <- anySymbol
124+ marker <- case T.uncons t of
125+ Just (c, ts) | T.null ts -> return c
126+ _ -> mzero
127+ let stopchar = if marker == '{' then '}' else marker
128+ withVerbatimMode $
129+ codeWith ("",classes,[]) . T.map nlToSpace . untokenize <$>
130+ manyTill (verbTok stopchar) (symbol stopchar)
131+
132+nlToSpace :: Char -> Char
133+nlToSpace '\n' = ' '
134+nlToSpace x = x
135+
136+romanNumeralUpper :: (PandocMonad m) => LP m Inlines
137+romanNumeralUpper =
138+ str . toRomanNumeral <$> romanNumeralArg
139+
140+romanNumeralLower :: (PandocMonad m) => LP m Inlines
141+romanNumeralLower =
142+ str . T.toLower . toRomanNumeral <$> romanNumeralArg
143+
144+romanNumeralArg :: (PandocMonad m) => LP m Int
145+romanNumeralArg = spaces *> (parser <|> inBraces)
146+ where
147+ inBraces = do
148+ symbol '{'
149+ spaces
150+ res <- parser
151+ spaces
152+ symbol '}'
153+ return res
154+ parser = do
155+ s <- untokenize <$> many1 (satisfyTok isWordTok)
156+ let (digits, rest) = T.span isDigit s
157+ unless (T.null rest) $
158+ Prelude.fail "Non-digits in argument to \\Rn or \\RN"
159+ safeRead digits
160+
161+accentWith :: PandocMonad m
162+ => LP m Inlines -> Char -> Maybe Char -> LP m Inlines
163+accentWith tok combiningAccent fallBack = try $ do
164+ ils <- tok
165+ case toList ils of
166+ (Str (T.uncons -> Just (x, xs)) : ys) -> return $ fromList $
167+ -- try to normalize to the combined character:
168+ Str (Normalize.normalize Normalize.NFC
169+ (T.pack [x, combiningAccent]) <> xs) : ys
170+ [Space] -> return $ str $ T.singleton
171+ $ fromMaybe combiningAccent fallBack
172+ [] -> return $ str $ T.singleton
173+ $ fromMaybe combiningAccent fallBack
174+ _ -> return ils
175+
176+
177+verbCommands :: PandocMonad m => M.Map Text (LP m Inlines)
178+verbCommands = M.fromList
179+ [ ("verb", doverb)
180+ , ("lstinline", dolstinline)
181+ , ("mintinline", domintinline)
182+ , ("Verb", doverb)
183+ ]
184+
185+accentCommands :: PandocMonad m => LP m Inlines -> M.Map Text (LP m Inlines)
186+accentCommands tok =
187+ let accent = accentWith tok
188+ in M.fromList
189+ [ ("aa", lit "å")
190+ , ("AA", lit "Å")
191+ , ("ss", lit "ß")
192+ , ("o", lit "ø")
193+ , ("O", lit "Ø")
194+ , ("L", lit "Ł")
195+ , ("l", lit "ł")
196+ , ("ae", lit "æ")
197+ , ("AE", lit "Æ")
198+ , ("oe", lit "œ")
199+ , ("OE", lit "Œ")
200+ , ("pounds", lit "£")
201+ , ("euro", lit "€")
202+ , ("copyright", lit "©")
203+ , ("textasciicircum", lit "^")
204+ , ("textasciitilde", lit "~")
205+ , ("H", accent '\779' Nothing) -- hungarumlaut
206+ , ("`", accent '\768' (Just '`')) -- grave
207+ , ("'", accent '\769' (Just '\'')) -- acute
208+ , ("^", accent '\770' (Just '^')) -- circ
209+ , ("~", accent '\771' (Just '~')) -- tilde
210+ , ("\"", accent '\776' Nothing) -- umlaut
211+ , (".", accent '\775' Nothing) -- dot
212+ , ("=", accent '\772' Nothing) -- macron
213+ , ("|", accent '\781' Nothing) -- vertical line above
214+ , ("b", accent '\817' Nothing) -- macron below
215+ , ("c", accent '\807' Nothing) -- cedilla
216+ , ("G", accent '\783' Nothing) -- doublegrave
217+ , ("h", accent '\777' Nothing) -- hookabove
218+ , ("d", accent '\803' Nothing) -- dotbelow
219+ , ("f", accent '\785' Nothing) -- inverted breve
220+ , ("r", accent '\778' Nothing) -- ringabove
221+ , ("t", accent '\865' Nothing) -- double inverted breve
222+ , ("U", accent '\782' Nothing) -- double vertical line above
223+ , ("v", accent '\780' Nothing) -- hacek
224+ , ("u", accent '\774' Nothing) -- breve
225+ , ("k", accent '\808' Nothing) -- ogonek
226+ , ("textogonekcentered", accent '\808' Nothing) -- ogonek
227+ , ("i", lit "ı") -- dotless i
228+ , ("j", lit "ȷ") -- dotless j
229+ , ("newtie", accent '\785' Nothing) -- inverted breve
230+ , ("textcircled", accent '\8413' Nothing) -- combining circle
231+ ]
232+
233+charCommands :: PandocMonad m => M.Map Text (LP m Inlines)
234+charCommands = M.fromList
235+ [ ("ldots", lit "…")
236+ , ("vdots", lit "\8942")
237+ , ("dots", lit "…")
238+ , ("mdots", lit "…")
239+ , ("sim", lit "~")
240+ , ("sep", lit ",")
241+ , ("P", lit "¶")
242+ , ("S", lit "§")
243+ , ("$", lit "$")
244+ , ("%", lit "%")
245+ , ("&", lit "&")
246+ , ("#", lit "#")
247+ , ("_", lit "_")
248+ , ("{", lit "{")
249+ , ("}", lit "}")
250+ , ("qed", lit "\a0\x25FB")
251+ , ("lq", return (str "‘"))
252+ , ("rq", return (str "’"))
253+ , ("textquoteleft", return (str "‘"))
254+ , ("textquoteright", return (str "’"))
255+ , ("textquotedblleft", return (str "“"))
256+ , ("textquotedblright", return (str "”"))
257+ , ("/", pure mempty) -- italic correction
258+ , ("\\", linebreak <$ (do inTableCell <- sInTableCell <$> getState
259+ guard $ not inTableCell
260+ optional rawopt
261+ spaces))
262+ , (",", lit "\8198")
263+ , ("@", pure mempty)
264+ , (" ", lit "\160")
265+ , ("ps", pure $ str "PS." <> space)
266+ , ("TeX", lit "TeX")
267+ , ("LaTeX", lit "LaTeX")
268+ , ("bar", lit "|")
269+ , ("textless", lit "<")
270+ , ("textgreater", lit ">")
271+ , ("textbackslash", lit "\\")
272+ , ("backslash", lit "\\")
273+ , ("slash", lit "/")
274+ -- fontawesome
275+ , ("faCheck", lit "\10003")
276+ , ("faClose", lit "\10007")
277+ -- hyphenat
278+ , ("bshyp", lit "\\\173")
279+ , ("fshyp", lit "/\173")
280+ , ("dothyp", lit ".\173")
281+ , ("colonhyp", lit ":\173")
282+ , ("hyp", lit "-")
283+ ]
284+
285+biblatexInlineCommands :: PandocMonad m
286+ => LP m Inlines -> M.Map Text (LP m Inlines)
287+biblatexInlineCommands tok = M.fromList
288+ -- biblatex misc
289+ [ ("RN", romanNumeralUpper)
290+ , ("Rn", romanNumeralLower)
291+ , ("mkbibquote", spanWith nullAttr . doubleQuoted <$> tok)
292+ , ("mkbibemph", spanWith nullAttr . emph <$> tok)
293+ , ("mkbibitalic", spanWith nullAttr . emph <$> tok)
294+ , ("mkbibbold", spanWith nullAttr . strong <$> tok)
295+ , ("mkbibparens",
296+ spanWith nullAttr . (\x -> str "(" <> x <> str ")") <$> tok)
297+ , ("mkbibbrackets",
298+ spanWith nullAttr . (\x -> str "[" <> x <> str "]") <$> tok)
299+ , ("autocap", spanWith nullAttr <$> tok)
300+ , ("textnormal", spanWith ("",["nodecor"],[]) <$> tok)
301+ , ("bibstring",
302+ (\x -> spanWith ("",[],[("bibstring",x)]) (str x)) . untokenize
303+ <$> braced)
304+ , ("adddot", pure (str "."))
305+ , ("adddotspace", pure (spanWith nullAttr (str "." <> space)))
306+ , ("addabbrvspace", pure space)
307+ , ("hyphen", pure (str "-"))
308+ ]
309+
310+nameCommands :: PandocMonad m => M.Map Text (LP m Inlines)
311+nameCommands = M.fromList
312+ [ ("figurename", doTerm Translations.Figure)
313+ , ("prefacename", doTerm Translations.Preface)
314+ , ("refname", doTerm Translations.References)
315+ , ("bibname", doTerm Translations.Bibliography)
316+ , ("chaptername", doTerm Translations.Chapter)
317+ , ("partname", doTerm Translations.Part)
318+ , ("contentsname", doTerm Translations.Contents)
319+ , ("listfigurename", doTerm Translations.ListOfFigures)
320+ , ("listtablename", doTerm Translations.ListOfTables)
321+ , ("indexname", doTerm Translations.Index)
322+ , ("abstractname", doTerm Translations.Abstract)
323+ , ("tablename", doTerm Translations.Table)
324+ , ("enclname", doTerm Translations.Encl)
325+ , ("ccname", doTerm Translations.Cc)
326+ , ("headtoname", doTerm Translations.To)
327+ , ("pagename", doTerm Translations.Page)
328+ , ("seename", doTerm Translations.See)
329+ , ("seealsoname", doTerm Translations.SeeAlso)
330+ , ("proofname", doTerm Translations.Proof)
331+ , ("glossaryname", doTerm Translations.Glossary)
332+ , ("lstlistingname", doTerm Translations.Listing)
333+ ]
334+
335+refCommands :: PandocMonad m => M.Map Text (LP m Inlines)
336+refCommands = M.fromList
337+ [ ("label", rawInlineOr "label" dolabel)
338+ , ("ref", rawInlineOr "ref" $ doref "ref")
339+ , ("cref", rawInlineOr "cref" $ doref "ref") -- from cleveref.sty
340+ , ("vref", rawInlineOr "vref" $ doref "ref+page") -- from varioref.sty
341+ , ("eqref", rawInlineOr "eqref" $ doref "eqref") -- from amsmath.sty
342+ ]
343+
344+acronymCommands :: PandocMonad m => M.Map Text (LP m Inlines)
345+acronymCommands = M.fromList
346+ -- glossaries package
347+ [ ("gls", doAcronym "short")
348+ , ("Gls", doAcronym "short")
349+ , ("glsdesc", doAcronym "long")
350+ , ("Glsdesc", doAcronym "long")
351+ , ("GLSdesc", doAcronym "long")
352+ , ("acrlong", doAcronym "long")
353+ , ("Acrlong", doAcronym "long")
354+ , ("acrfull", doAcronym "full")
355+ , ("Acrfull", doAcronym "full")
356+ , ("acrshort", doAcronym "abbrv")
357+ , ("Acrshort", doAcronym "abbrv")
358+ , ("glspl", doAcronymPlural "short")
359+ , ("Glspl", doAcronymPlural "short")
360+ , ("glsdescplural", doAcronymPlural "long")
361+ , ("Glsdescplural", doAcronymPlural "long")
362+ , ("GLSdescplural", doAcronymPlural "long")
363+ -- acronyms package
364+ , ("ac", doAcronym "short")
365+ , ("acf", doAcronym "full")
366+ , ("acs", doAcronym "abbrv")
367+ , ("acl", doAcronym "long")
368+ , ("acp", doAcronymPlural "short")
369+ , ("acfp", doAcronymPlural "full")
370+ , ("acsp", doAcronymPlural "abbrv")
371+ , ("aclp", doAcronymPlural "long")
372+ , ("Ac", doAcronym "short")
373+ , ("Acf", doAcronym "full")
374+ , ("Acs", doAcronym "abbrv")
375+ , ("Acl", doAcronym "long")
376+ , ("Acp", doAcronymPlural "short")
377+ , ("Acfp", doAcronymPlural "full")
378+ , ("Acsp", doAcronymPlural "abbrv")
379+ , ("Aclp", doAcronymPlural "long")
380+ ]
381+
382+doAcronym :: PandocMonad m => Text -> LP m Inlines
383+doAcronym form = do
384+ acro <- braced
385+ return . mconcat $ [spanWith ("",[],[("acronym-label", untokenize acro),
386+ ("acronym-form", "singular+" <> form)])
387+ $ str $ untokenize acro]
388+
389+doAcronymPlural :: PandocMonad m => Text -> LP m Inlines
390+doAcronymPlural form = do
391+ acro <- braced
392+ let plural = str "s"
393+ return . mconcat $ [spanWith ("",[],[("acronym-label", untokenize acro),
394+ ("acronym-form", "plural+" <> form)]) $
395+ mconcat [str $ untokenize acro, plural]]
396+
397+
src/Text/Pandoc/Readers/LaTeX/Lang.hs+200−121
@@ -1,7 +1,7 @@
11{-# LANGUAGE OverloadedStrings #-}
22{- |
33 Module : Text.Pandoc.Readers.LaTeX.Lang
4− Copyright : Copyright (C) 2018-2020 John MacFarlane
4+ Copyright : Copyright (C) 2018-2021 John MacFarlane
55 License : GNU GPL, version 2 or above
66
77 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -12,144 +12,223 @@ Functions for parsing polyglossia and babel language specifiers to
1212BCP47 'Lang'.
1313-}
1414module Text.Pandoc.Readers.LaTeX.Lang
15− ( polyglossiaLangToBCP47
15+ ( setDefaultLanguage
16+ , polyglossiaLangToBCP47
1617 , babelLangToBCP47
18+ , enquoteCommands
19+ , inlineLanguageCommands
1720 )
1821where
1922import qualified Data.Map as M
23+import Data.Text (Text)
2024import qualified Data.Text as T
21−import Text.Pandoc.BCP47 (Lang(..))
25+import Text.Pandoc.Shared (extractSpaces)
26+import Text.Collate.Lang (Lang(..), renderLang)
27+import Text.Pandoc.Class (PandocMonad(..), setTranslations)
28+import Text.Pandoc.Readers.LaTeX.Parsing
29+import Text.Pandoc.Parsing (updateState, option, getState, QuoteContext(..),
30+ withQuoteContext)
31+import Text.Pandoc.Builder (Blocks, Inlines, setMeta, str, spanWith,
32+ singleQuoted, doubleQuoted)
33+
34+enquote :: PandocMonad m
35+ => LP m Inlines
36+ -> Bool -> Maybe Text -> LP m Inlines
37+enquote tok starred mblang = do
38+ skipopts
39+ let lang = mblang >>= babelLangToBCP47
40+ let langspan = case lang of
41+ Nothing -> id
42+ Just l -> spanWith ("",[],[("lang", renderLang l)])
43+ quoteContext <- sQuoteContext <$> getState
44+ if starred || quoteContext == InDoubleQuote
45+ then singleQuoted . langspan <$> withQuoteContext InSingleQuote tok
46+ else doubleQuoted . langspan <$> withQuoteContext InDoubleQuote tok
47+
48+enquoteCommands :: PandocMonad m
49+ => LP m Inlines -> M.Map Text (LP m Inlines)
50+enquoteCommands tok = M.fromList
51+ [ ("enquote*", enquote tok True Nothing)
52+ , ("enquote", enquote tok False Nothing)
53+ -- foreignquote is supposed to use native quote marks
54+ , ("foreignquote*", braced >>= enquote tok True . Just . untokenize)
55+ , ("foreignquote", braced >>= enquote tok False . Just . untokenize)
56+ -- hypehnquote uses regular quotes
57+ , ("hyphenquote*", braced >>= enquote tok True . Just . untokenize)
58+ , ("hyphenquote", braced >>= enquote tok False . Just . untokenize)
59+ ]
60+
61+foreignlanguage :: PandocMonad m => LP m Inlines -> LP m Inlines
62+foreignlanguage tok = do
63+ babelLang <- untokenize <$> braced
64+ case babelLangToBCP47 babelLang of
65+ Just lang -> spanWith ("", [], [("lang", renderLang lang)]) <$> tok
66+ _ -> tok
67+
68+inlineLanguageCommands :: PandocMonad m
69+ => LP m Inlines -> M.Map Text (LP m Inlines)
70+inlineLanguageCommands tok =
71+ M.fromList $
72+ ("foreignlanguage", foreignlanguage tok) :
73+ (mk <$> M.toList polyglossiaLangToBCP47)
74+ where
75+ mk (polyglossia, bcp47Func) =
76+ ("text" <> polyglossia, inlineLanguage tok bcp47Func)
77+
78+inlineLanguage :: PandocMonad m
79+ => LP m Inlines -> (Text -> Lang) -> LP m Inlines
80+inlineLanguage tok bcp47Func = do
81+ o <- option "" $ T.filter (\c -> c /= '[' && c /= ']')
82+ <$> rawopt
83+ let lang = renderLang $ bcp47Func o
84+ extractSpaces (spanWith ("", [], [("lang", lang)])) <$> tok
85+
86+setDefaultLanguage :: PandocMonad m => LP m Blocks
87+setDefaultLanguage = do
88+ o <- option "" $ T.filter (\c -> c /= '[' && c /= ']')
89+ <$> rawopt
90+ polylang <- untokenize <$> braced
91+ case M.lookup polylang polyglossiaLangToBCP47 of
92+ Nothing -> return mempty -- TODO mzero? warning?
93+ Just langFunc -> do
94+ let l = langFunc o
95+ setTranslations l
96+ updateState $ setMeta "lang" $ str (renderLang l)
97+ return mempty
2298
2399polyglossiaLangToBCP47 :: M.Map T.Text (T.Text -> Lang)
24100polyglossiaLangToBCP47 = M.fromList
25101 [ ("arabic", \o -> case T.filter (/=' ') o of
26− "locale=algeria" -> Lang "ar" "" "DZ" []
27− "locale=mashriq" -> Lang "ar" "" "SY" []
28− "locale=libya" -> Lang "ar" "" "LY" []
29− "locale=morocco" -> Lang "ar" "" "MA" []
30− "locale=mauritania" -> Lang "ar" "" "MR" []
31− "locale=tunisia" -> Lang "ar" "" "TN" []
32− _ -> Lang "ar" "" "" [])
102+ "locale=algeria" -> Lang "ar" Nothing (Just "DZ") [] [] []
103+ "locale=mashriq" -> Lang "ar" Nothing (Just "SY") [] [] []
104+ "locale=libya" -> Lang "ar" Nothing (Just "LY") [] [] []
105+ "locale=morocco" -> Lang "ar" Nothing (Just "MA") [] [] []
106+ "locale=mauritania" -> Lang "ar" Nothing (Just "MR") [] [] []
107+ "locale=tunisia" -> Lang "ar" Nothing (Just "TN") [] [] []
108+ _ -> Lang "ar" Nothing (Just "") [] [] [])
33109 , ("german", \o -> case T.filter (/=' ') o of
34− "spelling=old" -> Lang "de" "" "DE" ["1901"]
110+ "spelling=old" -> Lang "de" Nothing (Just "DE") ["1901"] [] []
35111 "variant=austrian,spelling=old"
36− -> Lang "de" "" "AT" ["1901"]
37− "variant=austrian" -> Lang "de" "" "AT" []
112+ -> Lang "de" Nothing (Just "AT") ["1901"] [] []
113+ "variant=austrian" -> Lang "de" Nothing (Just "AT") [] [] []
38114 "variant=swiss,spelling=old"
39− -> Lang "de" "" "CH" ["1901"]
40− "variant=swiss" -> Lang "de" "" "CH" []
41− _ -> Lang "de" "" "" [])
42− , ("lsorbian", \_ -> Lang "dsb" "" "" [])
115+ -> Lang "de" Nothing (Just "CH") ["1901"] [] []
116+ "variant=swiss" -> Lang "de" Nothing (Just "CH") [] [] []
117+ _ -> Lang "de" Nothing Nothing [] [] [])
118+ , ("lsorbian", \_ -> Lang "dsb" Nothing Nothing [] [] [])
43119 , ("greek", \o -> case T.filter (/=' ') o of
44− "variant=poly" -> Lang "el" "" "polyton" []
45− "variant=ancient" -> Lang "grc" "" "" []
46− _ -> Lang "el" "" "" [])
120+ "variant=poly" -> Lang "el" Nothing (Just "polyton") [] [] []
121+ "variant=ancient" -> Lang "grc" Nothing Nothing [] [] []
122+ _ -> Lang "el" Nothing Nothing [] [] [])
47123 , ("english", \o -> case T.filter (/=' ') o of
48− "variant=australian" -> Lang "en" "" "AU" []
49− "variant=canadian" -> Lang "en" "" "CA" []
50− "variant=british" -> Lang "en" "" "GB" []
51− "variant=newzealand" -> Lang "en" "" "NZ" []
52− "variant=american" -> Lang "en" "" "US" []
53− _ -> Lang "en" "" "" [])
54− , ("usorbian", \_ -> Lang "hsb" "" "" [])
124+ "variant=australian" -> Lang "en" Nothing (Just "AU") [] [] []
125+ "variant=canadian" -> Lang "en" Nothing (Just "CA") [] [] []
126+ "variant=british" -> Lang "en" Nothing (Just "GB") [] [] []
127+ "variant=newzealand" -> Lang "en" Nothing (Just "NZ") [] [] []
128+ "variant=american" -> Lang "en" Nothing (Just "US") [] [] []
129+ _ -> Lang "en" Nothing (Just "") [] [] [])
130+ , ("usorbian", \_ -> Lang "hsb" Nothing Nothing [] [] [])
55131 , ("latin", \o -> case T.filter (/=' ') o of
56− "variant=classic" -> Lang "la" "" "" ["x-classic"]
57− _ -> Lang "la" "" "" [])
58− , ("slovenian", \_ -> Lang "sl" "" "" [])
59− , ("serbianc", \_ -> Lang "sr" "cyrl" "" [])
60− , ("pinyin", \_ -> Lang "zh" "Latn" "" ["pinyin"])
61− , ("afrikaans", \_ -> Lang "af" "" "" [])
62− , ("amharic", \_ -> Lang "am" "" "" [])
63− , ("assamese", \_ -> Lang "as" "" "" [])
64− , ("asturian", \_ -> Lang "ast" "" "" [])
65− , ("bulgarian", \_ -> Lang "bg" "" "" [])
66− , ("bengali", \_ -> Lang "bn" "" "" [])
67− , ("tibetan", \_ -> Lang "bo" "" "" [])
68− , ("breton", \_ -> Lang "br" "" "" [])
69− , ("catalan", \_ -> Lang "ca" "" "" [])
70− , ("welsh", \_ -> Lang "cy" "" "" [])
71− , ("czech", \_ -> Lang "cs" "" "" [])
72− , ("coptic", \_ -> Lang "cop" "" "" [])
73− , ("danish", \_ -> Lang "da" "" "" [])
74− , ("divehi", \_ -> Lang "dv" "" "" [])
75− , ("esperanto", \_ -> Lang "eo" "" "" [])
76− , ("spanish", \_ -> Lang "es" "" "" [])
77− , ("estonian", \_ -> Lang "et" "" "" [])
78− , ("basque", \_ -> Lang "eu" "" "" [])
79− , ("farsi", \_ -> Lang "fa" "" "" [])
80− , ("finnish", \_ -> Lang "fi" "" "" [])
81− , ("french", \_ -> Lang "fr" "" "" [])
82− , ("friulan", \_ -> Lang "fur" "" "" [])
83− , ("irish", \_ -> Lang "ga" "" "" [])
84− , ("scottish", \_ -> Lang "gd" "" "" [])
85− , ("ethiopic", \_ -> Lang "gez" "" "" [])
86− , ("galician", \_ -> Lang "gl" "" "" [])
87− , ("hebrew", \_ -> Lang "he" "" "" [])
88− , ("hindi", \_ -> Lang "hi" "" "" [])
89− , ("croatian", \_ -> Lang "hr" "" "" [])
90− , ("magyar", \_ -> Lang "hu" "" "" [])
91− , ("armenian", \_ -> Lang "hy" "" "" [])
92− , ("interlingua", \_ -> Lang "ia" "" "" [])
93− , ("indonesian", \_ -> Lang "id" "" "" [])
94− , ("icelandic", \_ -> Lang "is" "" "" [])
95− , ("italian", \_ -> Lang "it" "" "" [])
96− , ("japanese", \_ -> Lang "jp" "" "" [])
97− , ("khmer", \_ -> Lang "km" "" "" [])
98− , ("kurmanji", \_ -> Lang "kmr" "" "" [])
99− , ("kannada", \_ -> Lang "kn" "" "" [])
100− , ("korean", \_ -> Lang "ko" "" "" [])
101− , ("lao", \_ -> Lang "lo" "" "" [])
102− , ("lithuanian", \_ -> Lang "lt" "" "" [])
103− , ("latvian", \_ -> Lang "lv" "" "" [])
104− , ("malayalam", \_ -> Lang "ml" "" "" [])
105− , ("mongolian", \_ -> Lang "mn" "" "" [])
106− , ("marathi", \_ -> Lang "mr" "" "" [])
107− , ("dutch", \_ -> Lang "nl" "" "" [])
108− , ("nynorsk", \_ -> Lang "nn" "" "" [])
109− , ("norsk", \_ -> Lang "no" "" "" [])
110− , ("nko", \_ -> Lang "nqo" "" "" [])
111− , ("occitan", \_ -> Lang "oc" "" "" [])
112− , ("panjabi", \_ -> Lang "pa" "" "" [])
113− , ("polish", \_ -> Lang "pl" "" "" [])
114− , ("piedmontese", \_ -> Lang "pms" "" "" [])
115− , ("portuguese", \_ -> Lang "pt" "" "" [])
116− , ("romansh", \_ -> Lang "rm" "" "" [])
117− , ("romanian", \_ -> Lang "ro" "" "" [])
118− , ("russian", \_ -> Lang "ru" "" "" [])
119− , ("sanskrit", \_ -> Lang "sa" "" "" [])
120− , ("samin", \_ -> Lang "se" "" "" [])
121− , ("slovak", \_ -> Lang "sk" "" "" [])
122− , ("albanian", \_ -> Lang "sq" "" "" [])
123− , ("serbian", \_ -> Lang "sr" "" "" [])
124− , ("swedish", \_ -> Lang "sv" "" "" [])
125− , ("syriac", \_ -> Lang "syr" "" "" [])
126− , ("tamil", \_ -> Lang "ta" "" "" [])
127− , ("telugu", \_ -> Lang "te" "" "" [])
128− , ("thai", \_ -> Lang "th" "" "" [])
129− , ("turkmen", \_ -> Lang "tk" "" "" [])
130− , ("turkish", \_ -> Lang "tr" "" "" [])
131− , ("ukrainian", \_ -> Lang "uk" "" "" [])
132− , ("urdu", \_ -> Lang "ur" "" "" [])
133− , ("vietnamese", \_ -> Lang "vi" "" "" [])
132+ "variant=classic" -> Lang "la" Nothing Nothing ["x-classic"] [] []
133+ _ -> Lang "la" Nothing Nothing [] [] [])
134+ , ("slovenian", \_ -> Lang "sl" Nothing Nothing [] [] [])
135+ , ("serbianc", \_ -> Lang "sr" (Just "Cyrl") Nothing [] [] [])
136+ , ("pinyin", \_ -> Lang "zh" (Just "Latn") Nothing ["pinyin"] [] [])
137+ , ("afrikaans", \_ -> simpleLang "af")
138+ , ("amharic", \_ -> simpleLang "am")
139+ , ("assamese", \_ -> simpleLang "as")
140+ , ("asturian", \_ -> simpleLang "ast")
141+ , ("bulgarian", \_ -> simpleLang "bg")
142+ , ("bengali", \_ -> simpleLang "bn")
143+ , ("tibetan", \_ -> simpleLang "bo")
144+ , ("breton", \_ -> simpleLang "br")
145+ , ("catalan", \_ -> simpleLang "ca")
146+ , ("welsh", \_ -> simpleLang "cy")
147+ , ("czech", \_ -> simpleLang "cs")
148+ , ("coptic", \_ -> simpleLang "cop")
149+ , ("danish", \_ -> simpleLang "da")
150+ , ("divehi", \_ -> simpleLang "dv")
151+ , ("esperanto", \_ -> simpleLang "eo")
152+ , ("spanish", \_ -> simpleLang "es")
153+ , ("estonian", \_ -> simpleLang "et")
154+ , ("basque", \_ -> simpleLang "eu")
155+ , ("farsi", \_ -> simpleLang "fa")
156+ , ("finnish", \_ -> simpleLang "fi")
157+ , ("french", \_ -> simpleLang "fr")
158+ , ("friulan", \_ -> simpleLang "fur")
159+ , ("irish", \_ -> simpleLang "ga")
160+ , ("scottish", \_ -> simpleLang "gd")
161+ , ("ethiopic", \_ -> simpleLang "gez")
162+ , ("galician", \_ -> simpleLang "gl")
163+ , ("hebrew", \_ -> simpleLang "he")
164+ , ("hindi", \_ -> simpleLang "hi")
165+ , ("croatian", \_ -> simpleLang "hr")
166+ , ("magyar", \_ -> simpleLang "hu")
167+ , ("armenian", \_ -> simpleLang "hy")
168+ , ("interlingua", \_ -> simpleLang "ia")
169+ , ("indonesian", \_ -> simpleLang "id")
170+ , ("icelandic", \_ -> simpleLang "is")
171+ , ("italian", \_ -> simpleLang "it")
172+ , ("japanese", \_ -> simpleLang "jp")
173+ , ("khmer", \_ -> simpleLang "km")
174+ , ("kurmanji", \_ -> simpleLang "kmr")
175+ , ("kannada", \_ -> simpleLang "kn")
176+ , ("korean", \_ -> simpleLang "ko")
177+ , ("lao", \_ -> simpleLang "lo")
178+ , ("lithuanian", \_ -> simpleLang "lt")
179+ , ("latvian", \_ -> simpleLang "lv")
180+ , ("malayalam", \_ -> simpleLang "ml")
181+ , ("mongolian", \_ -> simpleLang "mn")
182+ , ("marathi", \_ -> simpleLang "mr")
183+ , ("dutch", \_ -> simpleLang "nl")
184+ , ("nynorsk", \_ -> simpleLang "nn")
185+ , ("norsk", \_ -> simpleLang "no")
186+ , ("nko", \_ -> simpleLang "nqo")
187+ , ("occitan", \_ -> simpleLang "oc")
188+ , ("panjabi", \_ -> simpleLang "pa")
189+ , ("polish", \_ -> simpleLang "pl")
190+ , ("piedmontese", \_ -> simpleLang "pms")
191+ , ("portuguese", \_ -> simpleLang "pt")
192+ , ("romansh", \_ -> simpleLang "rm")
193+ , ("romanian", \_ -> simpleLang "ro")
194+ , ("russian", \_ -> simpleLang "ru")
195+ , ("sanskrit", \_ -> simpleLang "sa")
196+ , ("samin", \_ -> simpleLang "se")
197+ , ("slovak", \_ -> simpleLang "sk")
198+ , ("albanian", \_ -> simpleLang "sq")
199+ , ("serbian", \_ -> simpleLang "sr")
200+ , ("swedish", \_ -> simpleLang "sv")
201+ , ("syriac", \_ -> simpleLang "syr")
202+ , ("tamil", \_ -> simpleLang "ta")
203+ , ("telugu", \_ -> simpleLang "te")
204+ , ("thai", \_ -> simpleLang "th")
205+ , ("turkmen", \_ -> simpleLang "tk")
206+ , ("turkish", \_ -> simpleLang "tr")
207+ , ("ukrainian", \_ -> simpleLang "uk")
208+ , ("urdu", \_ -> simpleLang "ur")
209+ , ("vietnamese", \_ -> simpleLang "vi")
134210 ]
135211
212+simpleLang :: Text -> Lang
213+simpleLang l = Lang l Nothing Nothing [] [] []
214+
136215babelLangToBCP47 :: T.Text -> Maybe Lang
137216babelLangToBCP47 s =
138217 case s of
139− "austrian" -> Just $ Lang "de" "" "AT" ["1901"]
140− "naustrian" -> Just $ Lang "de" "" "AT" []
141− "swissgerman" -> Just $ Lang "de" "" "CH" ["1901"]
142− "nswissgerman" -> Just $ Lang "de" "" "CH" []
143− "german" -> Just $ Lang "de" "" "DE" ["1901"]
144− "ngerman" -> Just $ Lang "de" "" "DE" []
145− "lowersorbian" -> Just $ Lang "dsb" "" "" []
146− "uppersorbian" -> Just $ Lang "hsb" "" "" []
147− "polutonikogreek" -> Just $ Lang "el" "" "" ["polyton"]
148− "slovene" -> Just $ Lang "sl" "" "" []
149− "australian" -> Just $ Lang "en" "" "AU" []
150− "canadian" -> Just $ Lang "en" "" "CA" []
151− "british" -> Just $ Lang "en" "" "GB" []
152− "newzealand" -> Just $ Lang "en" "" "NZ" []
153− "american" -> Just $ Lang "en" "" "US" []
154− "classiclatin" -> Just $ Lang "la" "" "" ["x-classic"]
218+ "austrian" -> Just $ Lang "de" Nothing (Just "AT") ["1901"] [] []
219+ "naustrian" -> Just $ Lang "de" Nothing (Just "AT") [] [] []
220+ "swissgerman" -> Just $ Lang "de" Nothing (Just "CH") ["1901"] [] []
221+ "nswissgerman" -> Just $ Lang "de" Nothing (Just "CH") [] [] []
222+ "german" -> Just $ Lang "de" Nothing (Just "DE") ["1901"] [] []
223+ "ngerman" -> Just $ Lang "de" Nothing (Just "DE") [] [] []
224+ "lowersorbian" -> Just $ Lang "dsb" Nothing Nothing [] [] []
225+ "uppersorbian" -> Just $ Lang "hsb" Nothing Nothing [] [] []
226+ "polutonikogreek" -> Just $ Lang "el" Nothing Nothing ["polyton"] [] []
227+ "slovene" -> Just $ simpleLang "sl"
228+ "australian" -> Just $ Lang "en" Nothing (Just "AU") [] [] []
229+ "canadian" -> Just $ Lang "en" Nothing (Just "CA") [] [] []
230+ "british" -> Just $ Lang "en" Nothing (Just "GB") [] [] []
231+ "newzealand" -> Just $ Lang "en" Nothing (Just "NZ") [] [] []
232+ "american" -> Just $ Lang "en" Nothing (Just "US") [] [] []
233+ "classiclatin" -> Just $ Lang "la" Nothing Nothing ["x-classic"] [] []
155234 _ -> ($ "") <$> M.lookup s polyglossiaLangToBCP47
src/Text/Pandoc/Readers/LaTeX/Macro.hsnew file+184
@@ -0,0 +1,184 @@
1+{-# LANGUAGE OverloadedStrings #-}
2+module Text.Pandoc.Readers.LaTeX.Macro
3+ ( macroDef
4+ )
5+where
6+import Text.Pandoc.Extensions (Extension(..))
7+import Text.Pandoc.Logging (LogMessage(MacroAlreadyDefined))
8+import Text.Pandoc.Readers.LaTeX.Parsing
9+import Text.Pandoc.Readers.LaTeX.Types
10+import Text.Pandoc.Class
11+import Text.Pandoc.Shared (safeRead)
12+import Text.Pandoc.Parsing hiding (blankline, mathDisplay, mathInline,
13+ optional, space, spaces, withRaw, (<|>))
14+import Control.Applicative ((<|>), optional)
15+import qualified Data.Map as M
16+import Data.Text (Text)
17+import qualified Data.Text as T
18+
19+macroDef :: (PandocMonad m, Monoid a) => (Text -> a) -> LP m a
20+macroDef constructor = do
21+ (_, s) <- withRaw (commandDef <|> environmentDef)
22+ (constructor (untokenize s) <$
23+ guardDisabled Ext_latex_macros)
24+ <|> return mempty
25+ where commandDef = do
26+ nameMacroPairs <- newcommand <|> letmacro <|> defmacro <|> newif
27+ guardDisabled Ext_latex_macros <|>
28+ mapM_ (\(name, macro') ->
29+ updateState (\s -> s{ sMacros = M.insert name macro'
30+ (sMacros s) })) nameMacroPairs
31+ environmentDef = do
32+ mbenv <- newenvironment
33+ case mbenv of
34+ Nothing -> return ()
35+ Just (name, macro1, macro2) ->
36+ guardDisabled Ext_latex_macros <|>
37+ do updateState $ \s -> s{ sMacros =
38+ M.insert name macro1 (sMacros s) }
39+ updateState $ \s -> s{ sMacros =
40+ M.insert ("end" <> name) macro2 (sMacros s) }
41+ -- @\newenvironment{envname}[n-args][default]{begin}{end}@
42+ -- is equivalent to
43+ -- @\newcommand{\envname}[n-args][default]{begin}@
44+ -- @\newcommand{\endenvname}@
45+
46+letmacro :: PandocMonad m => LP m [(Text, Macro)]
47+letmacro = do
48+ controlSeq "let"
49+ (name, contents) <- withVerbatimMode $ do
50+ Tok _ (CtrlSeq name) _ <- anyControlSeq
51+ optional $ symbol '='
52+ spaces
53+ -- we first parse in verbatim mode, and then expand macros,
54+ -- because we don't want \let\foo\bar to turn into
55+ -- \let\foo hello if we have previously \def\bar{hello}
56+ contents <- bracedOrToken
57+ return (name, contents)
58+ contents' <- doMacros' 0 contents
59+ return [(name, Macro ExpandWhenDefined [] Nothing contents')]
60+
61+defmacro :: PandocMonad m => LP m [(Text, Macro)]
62+defmacro = do
63+ -- we use withVerbatimMode, because macros are to be expanded
64+ -- at point of use, not point of definition
65+ controlSeq "def"
66+ withVerbatimMode $ do
67+ Tok _ (CtrlSeq name) _ <- anyControlSeq
68+ argspecs <- many (argspecArg <|> argspecPattern)
69+ contents <- bracedOrToken
70+ return [(name, Macro ExpandWhenUsed argspecs Nothing contents)]
71+
72+-- \newif\iffoo' defines:
73+-- \iffoo to be \iffalse
74+-- \footrue to be a command that defines \iffoo to be \iftrue
75+-- \foofalse to be a command that defines \iffoo to be \iffalse
76+newif :: PandocMonad m => LP m [(Text, Macro)]
77+newif = do
78+ controlSeq "newif"
79+ withVerbatimMode $ do
80+ Tok pos (CtrlSeq name) _ <- anyControlSeq
81+ -- \def\iffoo\iffalse
82+ -- \def\footrue{\def\iffoo\iftrue}
83+ -- \def\foofalse{\def\iffoo\iffalse}
84+ let base = T.drop 2 name
85+ return [ (name, Macro ExpandWhenUsed [] Nothing
86+ [Tok pos (CtrlSeq "iffalse") "\\iffalse"])
87+ , (base <> "true",
88+ Macro ExpandWhenUsed [] Nothing
89+ [ Tok pos (CtrlSeq "def") "\\def"
90+ , Tok pos (CtrlSeq name) ("\\" <> name)
91+ , Tok pos (CtrlSeq "iftrue") "\\iftrue"
92+ ])
93+ , (base <> "false",
94+ Macro ExpandWhenUsed [] Nothing
95+ [ Tok pos (CtrlSeq "def") "\\def"
96+ , Tok pos (CtrlSeq name) ("\\" <> name)
97+ , Tok pos (CtrlSeq "iffalse") "\\iffalse"
98+ ])
99+ ]
100+
101+argspecArg :: PandocMonad m => LP m ArgSpec
102+argspecArg = do
103+ Tok _ (Arg i) _ <- satisfyTok isArgTok
104+ return $ ArgNum i
105+
106+argspecPattern :: PandocMonad m => LP m ArgSpec
107+argspecPattern =
108+ Pattern <$> many1 (satisfyTok (\(Tok _ toktype' txt) ->
109+ (toktype' == Symbol || toktype' == Word) &&
110+ (txt /= "{" && txt /= "\\" && txt /= "}")))
111+
112+newcommand :: PandocMonad m => LP m [(Text, Macro)]
113+newcommand = do
114+ Tok pos (CtrlSeq mtype) _ <- controlSeq "newcommand" <|>
115+ controlSeq "renewcommand" <|>
116+ controlSeq "providecommand" <|>
117+ controlSeq "DeclareMathOperator" <|>
118+ controlSeq "DeclareRobustCommand"
119+ withVerbatimMode $ do
120+ Tok _ (CtrlSeq name) txt <- do
121+ optional (symbol '*')
122+ anyControlSeq <|>
123+ (symbol '{' *> spaces *> anyControlSeq <* spaces <* symbol '}')
124+ spaces
125+ numargs <- option 0 $ try bracketedNum
126+ let argspecs = map ArgNum [1..numargs]
127+ spaces
128+ optarg <- option Nothing $ Just <$> try bracketedToks
129+ spaces
130+ contents' <- bracedOrToken
131+ let contents =
132+ case mtype of
133+ "DeclareMathOperator" ->
134+ Tok pos (CtrlSeq "mathop") "\\mathop"
135+ : Tok pos Symbol "{"
136+ : Tok pos (CtrlSeq "mathrm") "\\mathrm"
137+ : Tok pos Symbol "{"
138+ : (contents' ++
139+ [ Tok pos Symbol "}", Tok pos Symbol "}" ])
140+ _ -> contents'
141+ macros <- sMacros <$> getState
142+ case M.lookup name macros of
143+ Just macro
144+ | mtype == "newcommand" -> do
145+ report $ MacroAlreadyDefined txt pos
146+ return [(name, macro)]
147+ | mtype == "providecommand" -> return [(name, macro)]
148+ _ -> return [(name, Macro ExpandWhenUsed argspecs optarg contents)]
149+
150+newenvironment :: PandocMonad m => LP m (Maybe (Text, Macro, Macro))
151+newenvironment = do
152+ pos <- getPosition
153+ Tok _ (CtrlSeq mtype) _ <- controlSeq "newenvironment" <|>
154+ controlSeq "renewenvironment" <|>
155+ controlSeq "provideenvironment"
156+ withVerbatimMode $ do
157+ optional $ symbol '*'
158+ spaces
159+ name <- untokenize <$> braced
160+ spaces
161+ numargs <- option 0 $ try bracketedNum
162+ spaces
163+ optarg <- option Nothing $ Just <$> try bracketedToks
164+ let argspecs = map (\i -> ArgNum i) [1..numargs]
165+ startcontents <- spaces >> bracedOrToken
166+ endcontents <- spaces >> bracedOrToken
167+ macros <- sMacros <$> getState
168+ case M.lookup name macros of
169+ Just _
170+ | mtype == "newenvironment" -> do
171+ report $ MacroAlreadyDefined name pos
172+ return Nothing
173+ | mtype == "provideenvironment" ->
174+ return Nothing
175+ _ -> return $ Just (name,
176+ Macro ExpandWhenUsed argspecs optarg startcontents,
177+ Macro ExpandWhenUsed [] Nothing endcontents)
178+
179+bracketedNum :: PandocMonad m => LP m Int
180+bracketedNum = do
181+ ds <- untokenize <$> bracketedToks
182+ case safeRead ds of
183+ Just i -> return i
184+ _ -> return 0
src/Text/Pandoc/Readers/LaTeX/Math.hsnew file+221
@@ -0,0 +1,221 @@
1+{-# LANGUAGE OverloadedStrings #-}
2+module Text.Pandoc.Readers.LaTeX.Math
3+ ( dollarsMath
4+ , inlineEnvironments
5+ , inlineEnvironment
6+ , mathInline
7+ , mathDisplay
8+ , theoremstyle
9+ , theoremEnvironment
10+ , newtheorem
11+ , proof
12+ )
13+where
14+import Data.Maybe (fromMaybe)
15+import Text.Pandoc.Walk (walk)
16+import Text.Pandoc.Builder as B
17+import qualified Data.Sequence as Seq
18+import Text.Pandoc.Readers.LaTeX.Parsing
19+import Text.Pandoc.Readers.LaTeX.Types
20+import Text.Pandoc.Class
21+import Text.Pandoc.Shared (trimMath, stripTrailingNewlines)
22+import Text.Pandoc.Parsing hiding (blankline, mathDisplay, mathInline,
23+ optional, space, spaces, withRaw, (<|>))
24+import Control.Applicative ((<|>), optional)
25+import Control.Monad (guard, mzero)
26+import qualified Data.Map as M
27+import Data.Text (Text)
28+
29+dollarsMath :: PandocMonad m => LP m Inlines
30+dollarsMath = do
31+ symbol '$'
32+ display <- option False (True <$ symbol '$')
33+ (do contents <- try $ untokenize <$> pDollarsMath 0
34+ if display
35+ then mathDisplay contents <$ symbol '$'
36+ else return $ mathInline contents)
37+ <|> (guard display >> return (mathInline ""))
38+
39+-- Int is number of embedded groupings
40+pDollarsMath :: PandocMonad m => Int -> LP m [Tok]
41+pDollarsMath n = do
42+ tk@(Tok _ toktype t) <- anyTok
43+ case toktype of
44+ Symbol | t == "$"
45+ , n == 0 -> return []
46+ | t == "\\" -> do
47+ tk' <- anyTok
48+ (tk :) . (tk' :) <$> pDollarsMath n
49+ | t == "{" -> (tk :) <$> pDollarsMath (n+1)
50+ | t == "}" ->
51+ if n > 0
52+ then (tk :) <$> pDollarsMath (n-1)
53+ else mzero
54+ _ -> (tk :) <$> pDollarsMath n
55+
56+mathDisplay :: Text -> Inlines
57+mathDisplay = displayMath . trimMath
58+
59+mathInline :: Text -> Inlines
60+mathInline = math . trimMath
61+
62+mathEnvWith :: PandocMonad m
63+ => (Inlines -> a) -> Maybe Text -> Text -> LP m a
64+mathEnvWith f innerEnv name = f . mathDisplay . inner <$> mathEnv name
65+ where inner x = case innerEnv of
66+ Nothing -> x
67+ Just y -> "\\begin{" <> y <> "}\n" <> x <>
68+ "\\end{" <> y <> "}"
69+
70+mathEnv :: PandocMonad m => Text -> LP m Text
71+mathEnv name = do
72+ skipopts
73+ optional blankline
74+ res <- manyTill anyTok (end_ name)
75+ return $ stripTrailingNewlines $ untokenize res
76+
77+inlineEnvironment :: PandocMonad m => LP m Inlines
78+inlineEnvironment = try $ do
79+ controlSeq "begin"
80+ name <- untokenize <$> braced
81+ M.findWithDefault mzero name inlineEnvironments
82+
83+inlineEnvironments :: PandocMonad m => M.Map Text (LP m Inlines)
84+inlineEnvironments = M.fromList [
85+ ("displaymath", mathEnvWith id Nothing "displaymath")
86+ , ("math", math <$> mathEnv "math")
87+ , ("equation", mathEnvWith id Nothing "equation")
88+ , ("equation*", mathEnvWith id Nothing "equation*")
89+ , ("gather", mathEnvWith id (Just "gathered") "gather")
90+ , ("gather*", mathEnvWith id (Just "gathered") "gather*")
91+ , ("multline", mathEnvWith id (Just "gathered") "multline")
92+ , ("multline*", mathEnvWith id (Just "gathered") "multline*")
93+ , ("eqnarray", mathEnvWith id (Just "aligned") "eqnarray")
94+ , ("eqnarray*", mathEnvWith id (Just "aligned") "eqnarray*")
95+ , ("align", mathEnvWith id (Just "aligned") "align")
96+ , ("align*", mathEnvWith id (Just "aligned") "align*")
97+ , ("alignat", mathEnvWith id (Just "aligned") "alignat")
98+ , ("alignat*", mathEnvWith id (Just "aligned") "alignat*")
99+ , ("dmath", mathEnvWith id Nothing "dmath")
100+ , ("dmath*", mathEnvWith id Nothing "dmath*")
101+ , ("dgroup", mathEnvWith id (Just "aligned") "dgroup")
102+ , ("dgroup*", mathEnvWith id (Just "aligned") "dgroup*")
103+ , ("darray", mathEnvWith id (Just "aligned") "darray")
104+ , ("darray*", mathEnvWith id (Just "aligned") "darray*")
105+ ]
106+
107+theoremstyle :: PandocMonad m => LP m Blocks
108+theoremstyle = do
109+ stylename <- untokenize <$> braced
110+ let mbstyle = case stylename of
111+ "plain" -> Just PlainStyle
112+ "definition" -> Just DefinitionStyle
113+ "remark" -> Just RemarkStyle
114+ _ -> Nothing
115+ case mbstyle of
116+ Nothing -> return ()
117+ Just sty -> updateState $ \s -> s{ sLastTheoremStyle = sty }
118+ return mempty
119+
120+newtheorem :: PandocMonad m => LP m Inlines -> LP m Blocks
121+newtheorem inline = do
122+ number <- option True (False <$ symbol '*' <* sp)
123+ name <- untokenize <$> braced
124+ sp
125+ series <- option Nothing $ Just . untokenize <$> bracketedToks
126+ sp
127+ showName <- tokWith inline
128+ sp
129+ syncTo <- option Nothing $ Just . untokenize <$> bracketedToks
130+ sty <- sLastTheoremStyle <$> getState
131+ let spec = TheoremSpec { theoremName = showName
132+ , theoremStyle = sty
133+ , theoremSeries = series
134+ , theoremSyncTo = syncTo
135+ , theoremNumber = number
136+ , theoremLastNum = DottedNum [0] }
137+ tmap <- sTheoremMap <$> getState
138+ updateState $ \s -> s{ sTheoremMap =
139+ M.insert name spec tmap }
140+ return mempty
141+
142+theoremEnvironment :: PandocMonad m
143+ => LP m Blocks -> LP m Inlines -> Text -> LP m Blocks
144+theoremEnvironment blocks opt name = do
145+ tmap <- sTheoremMap <$> getState
146+ case M.lookup name tmap of
147+ Nothing -> mzero
148+ Just tspec -> do
149+ optTitle <- option mempty $ (\x -> space <> "(" <> x <> ")") <$> opt
150+ mblabel <- option Nothing $ Just . untokenize <$>
151+ try (spaces >> controlSeq "label" >> spaces >> braced)
152+ bs <- env name blocks
153+ number <-
154+ if theoremNumber tspec
155+ then do
156+ let name' = fromMaybe name $ theoremSeries tspec
157+ num <- getNextNumber
158+ (maybe (DottedNum [0]) theoremLastNum .
159+ M.lookup name' . sTheoremMap)
160+ updateState $ \s ->
161+ s{ sTheoremMap =
162+ M.adjust
163+ (\spec -> spec{ theoremLastNum = num })
164+ name'
165+ (sTheoremMap s)
166+ }
167+
168+ case mblabel of
169+ Just ident ->
170+ updateState $ \s ->
171+ s{ sLabels = M.insert ident
172+ (B.toList $
173+ theoremName tspec <> "\160" <>
174+ str (renderDottedNum num)) (sLabels s) }
175+ Nothing -> return ()
176+ return $ space <> B.text (renderDottedNum num)
177+ else return mempty
178+ let titleEmph = case theoremStyle tspec of
179+ PlainStyle -> B.strong
180+ DefinitionStyle -> B.strong
181+ RemarkStyle -> B.emph
182+ let title = titleEmph (theoremName tspec <> number)
183+ <> optTitle <> "." <> space
184+ return $ divWith (fromMaybe "" mblabel, [name], []) $ addTitle title
185+ $ case theoremStyle tspec of
186+ PlainStyle -> walk italicize bs
187+ _ -> bs
188+
189+
190+
191+proof :: PandocMonad m => LP m Blocks -> LP m Inlines -> LP m Blocks
192+proof blocks opt = do
193+ title <- option (B.text "Proof") opt
194+ bs <- env "proof" blocks
195+ return $
196+ B.divWith ("", ["proof"], []) $
197+ addQed $ addTitle (B.emph (title <> ".")) bs
198+
199+addTitle :: Inlines -> Blocks -> Blocks
200+addTitle ils bs =
201+ case B.toList bs of
202+ (Para xs : rest)
203+ -> B.fromList (Para (B.toList ils ++ (Space : xs)) : rest)
204+ _ -> B.para ils <> bs
205+
206+addQed :: Blocks -> Blocks
207+addQed bs =
208+ case Seq.viewr (B.unMany bs) of
209+ s Seq.:> Para ils
210+ -> B.Many (s Seq.|> Para (ils ++ B.toList qedSign))
211+ _ -> bs <> B.para qedSign
212+ where
213+ qedSign = B.str "\xa0\x25FB"
214+
215+italicize :: Block -> Block
216+italicize x@(Para [Image{}]) = x -- see #6925
217+italicize (Para ils) = Para [Emph ils]
218+italicize (Plain ils) = Plain [Emph ils]
219+italicize x = x
220+
221+
src/Text/Pandoc/Readers/LaTeX/Parsing.hs+182−54
@@ -5,7 +5,7 @@
55{-# LANGUAGE ScopedTypeVariables #-}
66{- |
77 Module : Text.Pandoc.Readers.LaTeX.Parsing
8− Copyright : Copyright (C) 2006-2020 John MacFarlane
8+ Copyright : Copyright (C) 2006-2021 John MacFarlane
99 License : GNU GPL, version 2 or above
1010
1111 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -27,11 +27,14 @@ module Text.Pandoc.Readers.LaTeX.Parsing
2727 , rawLaTeXParser
2828 , applyMacros
2929 , tokenize
30+ , tokenizeSources
3031 , untokenize
3132 , untoken
3233 , totoks
3334 , toksToString
3435 , satisfyTok
36+ , parseFromToks
37+ , disablingWithRaw
3538 , doMacros
3639 , doMacros'
3740 , setpos
@@ -52,6 +55,7 @@ module Text.Pandoc.Readers.LaTeX.Parsing
5255 , comment
5356 , anyTok
5457 , singleChar
58+ , tokWith
5559 , specialChars
5660 , endline
5761 , blankline
@@ -78,6 +82,11 @@ module Text.Pandoc.Readers.LaTeX.Parsing
7882 , rawopt
7983 , overlaySpecification
8084 , getNextNumber
85+ , label
86+ , setCaption
87+ , resetCaption
88+ , env
89+ , addMeta
8190 ) where
8291
8392import Control.Applicative (many, (<|>))
@@ -87,13 +96,15 @@ import Control.Monad.Trans (lift)
8796import Data.Char (chr, isAlphaNum, isDigit, isLetter, ord)
8897import Data.Default
8998import Data.List (intercalate)
99+import qualified Data.IntMap as IntMap
90100import qualified Data.Map as M
91101import qualified Data.Set as Set
92102import Data.Text (Text)
93103import qualified Data.Text as T
94104import Text.Pandoc.Builder
95105import Text.Pandoc.Class.PandocMonad (PandocMonad, report)
96−import Text.Pandoc.Error (PandocError (PandocMacroLoop))
106+import Text.Pandoc.Error
107+ (PandocError (PandocMacroLoop,PandocShouldNeverHappenError))
97108import Text.Pandoc.Logging
98109import Text.Pandoc.Options
99110import Text.Pandoc.Parsing hiding (blankline, many, mathDisplay, mathInline,
@@ -102,7 +113,6 @@ import Text.Pandoc.Readers.LaTeX.Types (ExpansionPoint (..), Macro (..),
102113 ArgSpec (..), Tok (..), TokType (..))
103114import Text.Pandoc.Shared
104115import Text.Parsec.Pos
105−-- import Debug.Trace
106116
107117newtype DottedNum = DottedNum [Int]
108118 deriving (Show, Eq)
@@ -151,7 +161,9 @@ data LaTeXState = LaTeXState{ sOptions :: ReaderOptions
151161 , sLabels :: M.Map Text [Inline]
152162 , sHasChapters :: Bool
153163 , sToggles :: M.Map Text Bool
154− , sExpanded :: Bool
164+ , sFileContents :: M.Map Text Text
165+ , sEnableWithRaw :: Bool
166+ , sRawTokens :: IntMap.IntMap [Tok]
155167 }
156168 deriving Show
157169
@@ -176,7 +188,9 @@ defaultLaTeXState = LaTeXState{ sOptions = def
176188 , sLabels = M.empty
177189 , sHasChapters = False
178190 , sToggles = M.empty
179− , sExpanded = False
191+ , sFileContents = M.empty
192+ , sEnableWithRaw = True
193+ , sRawTokens = IntMap.empty
180194 }
181195
182196instance PandocMonad m => HasQuoteContext LaTeXState m where
@@ -234,7 +248,7 @@ withVerbatimMode parser = do
234248
235249rawLaTeXParser :: (PandocMonad m, HasMacros s, HasReaderOptions s)
236250 => [Tok] -> Bool -> LP m a -> LP m a
237− -> ParserT Text s m (a, Text)
251+ -> ParserT Sources s m (a, Text)
238252rawLaTeXParser toks retokenize parser valParser = do
239253 pstate <- getState
240254 let lstate = def{ sOptions = extractReaderOptions pstate }
@@ -246,7 +260,6 @@ rawLaTeXParser toks retokenize parser valParser = do
246260 Right toks' -> do
247261 res <- lift $ runParserT (do when retokenize $ do
248262 -- retokenize, applying macros
249− doMacros
250263 ts <- many (satisfyTok (const True))
251264 setInput ts
252265 rawparser)
@@ -255,7 +268,7 @@ rawLaTeXParser toks retokenize parser valParser = do
255268 Left _ -> mzero
256269 Right ((val, raw), st) -> do
257270 updateState (updateMacros (sMacros st <>))
258− _ <- takeP (T.length (untokenize toks'))
271+ void $ count (T.length (untokenize toks')) anyChar
259272 let result = untokenize raw
260273 -- ensure we end with space if input did, see #4442
261274 let result' =
@@ -268,7 +281,7 @@ rawLaTeXParser toks retokenize parser valParser = do
268281 return (val, result')
269282
270283applyMacros :: (PandocMonad m, HasMacros s, HasReaderOptions s)
271− => Text -> ParserT Text s m Text
284+ => Text -> ParserT Sources s m Text
272285applyMacros s = (guardDisabled Ext_latex_macros >> return s) <|>
273286 do let retokenize = untokenize <$> many (satisfyTok (const True))
274287 pstate <- getState
@@ -279,6 +292,20 @@ applyMacros s = (guardDisabled Ext_latex_macros >> return s) <|>
279292 Left e -> Prelude.fail (show e)
280293 Right s' -> return s'
281294
295+{-
296+When tokenize or untokenize change, test with this
297+QuickCheck property:
298+
299+> tokUntokRoundtrip :: String -> Bool
300+> tokUntokRoundtrip s =
301+> let t = T.pack s in untokenize (tokenize "random" t) == t
302+-}
303+
304+tokenizeSources :: Sources -> [Tok]
305+tokenizeSources = concatMap tokenizeSource . unSources
306+ where
307+ tokenizeSource (pos, t) = totoks pos t
308+
282309tokenize :: SourceName -> Text -> [Tok]
283310tokenize sourcename = totoks (initialPos sourcename)
284311
@@ -402,12 +429,31 @@ untoken t = untokenAccum t mempty
402429toksToString :: [Tok] -> String
403430toksToString = T.unpack . untokenize
404431
432+parseFromToks :: PandocMonad m => LP m a -> [Tok] -> LP m a
433+parseFromToks parser toks = do
434+ oldInput <- getInput
435+ setInput toks
436+ result <- disablingWithRaw parser
437+ setInput oldInput
438+ return result
439+
440+disablingWithRaw :: PandocMonad m => LP m a -> LP m a
441+disablingWithRaw parser = do
442+ oldEnableWithRaw <- sEnableWithRaw <$> getState
443+ updateState $ \st -> st{ sEnableWithRaw = False }
444+ result <- parser
445+ updateState $ \st -> st{ sEnableWithRaw = oldEnableWithRaw }
446+ return result
447+
405448satisfyTok :: PandocMonad m => (Tok -> Bool) -> LP m Tok
406449satisfyTok f = do
407450 doMacros -- apply macros on remaining input stream
408451 res <- tokenPrim (T.unpack . untoken) updatePos matcher
409− updateState $ \st -> st{ sExpanded = False }
410− return res
452+ updateState $ \st ->
453+ if sEnableWithRaw st
454+ then st{ sRawTokens = IntMap.map (res:) $ sRawTokens st }
455+ else st
456+ return $! res
411457 where matcher t | f t = Just t
412458 | otherwise = Nothing
413459 updatePos :: SourcePos -> Tok -> [Tok] -> SourcePos
@@ -416,27 +462,24 @@ satisfyTok f = do
416462
417463doMacros :: PandocMonad m => LP m ()
418464doMacros = do
419− expanded <- sExpanded <$> getState
420− verbatimMode <- sVerbatimMode <$> getState
421− unless (expanded || verbatimMode) $ do
422− getInput >>= doMacros' 1 >>= setInput
423− updateState $ \st -> st{ sExpanded = True }
465+ st <- getState
466+ unless (sVerbatimMode st) $
467+ getInput >>= doMacros' 1 >>= setInput
424468
425469doMacros' :: PandocMonad m => Int -> [Tok] -> LP m [Tok]
426470doMacros' n inp =
427471 case inp of
428472 Tok spos (CtrlSeq "begin") _ : Tok _ Symbol "{" :
429473 Tok _ Word name : Tok _ Symbol "}" : ts
430− -> handleMacros n spos name ts
474+ -> handleMacros n spos name ts <|> return inp
431475 Tok spos (CtrlSeq "end") _ : Tok _ Symbol "{" :
432476 Tok _ Word name : Tok _ Symbol "}" : ts
433− -> handleMacros n spos ("end" <> name) ts
477+ -> handleMacros n spos ("end" <> name) ts <|> return inp
434478 Tok _ (CtrlSeq "expandafter") _ : t : ts
435479 -> combineTok t <$> doMacros' n ts
436480 Tok spos (CtrlSeq name) _ : ts
437− -> handleMacros n spos name ts
481+ -> handleMacros n spos name ts <|> return inp
438482 _ -> return inp
439− <|> return inp
440483
441484 where
442485 combineTok (Tok spos (CtrlSeq name) x) (Tok _ Word w : ts)
@@ -482,7 +525,7 @@ doMacros' n inp =
482525 $ throwError $ PandocMacroLoop name
483526 macros <- sMacros <$> getState
484527 case M.lookup name macros of
485− Nothing -> mzero
528+ Nothing -> trySpecialMacro name ts
486529 Just (Macro expansionPoint argspecs optarg newtoks) -> do
487530 let getargs' = do
488531 args <-
@@ -510,6 +553,41 @@ doMacros' n inp =
510553 ExpandWhenUsed -> doMacros' (n' + 1) result
511554 ExpandWhenDefined -> return result
512555
556+-- | Certain macros do low-level tex manipulations that can't
557+-- be represented in our Macro type, so we handle them here.
558+trySpecialMacro :: PandocMonad m => Text -> [Tok] -> LP m [Tok]
559+trySpecialMacro "xspace" ts = do
560+ ts' <- doMacros' 1 ts
561+ case ts' of
562+ Tok pos Word t : _
563+ | startsWithAlphaNum t -> return $ Tok pos Spaces " " : ts'
564+ _ -> return ts'
565+trySpecialMacro "iftrue" ts = handleIf True ts
566+trySpecialMacro "iffalse" ts = handleIf False ts
567+trySpecialMacro _ _ = mzero
568+
569+handleIf :: PandocMonad m => Bool -> [Tok] -> LP m [Tok]
570+handleIf b ts = do
571+ res' <- lift $ runParserT (ifParser b) defaultLaTeXState "tokens" ts
572+ case res' of
573+ Left _ -> Prelude.fail "Could not parse conditional"
574+ Right ts' -> return ts'
575+
576+ifParser :: PandocMonad m => Bool -> LP m [Tok]
577+ifParser b = do
578+ ifToks <- many (notFollowedBy (controlSeq "else" <|> controlSeq "fi")
579+ *> anyTok)
580+ elseToks <- (controlSeq "else" >> manyTill anyTok (controlSeq "fi"))
581+ <|> ([] <$ controlSeq "fi")
582+ rest <- getInput
583+ return $ (if b then ifToks else elseToks) ++ rest
584+
585+startsWithAlphaNum :: Text -> Bool
586+startsWithAlphaNum t =
587+ case T.uncons t of
588+ Just (c, _) | isAlphaNum c -> True
589+ _ -> False
590+
513591setpos :: SourcePos -> Tok -> Tok
514592setpos spos (Tok _ tt txt) = Tok spos tt txt
515593
@@ -592,18 +670,22 @@ isCommentTok _ = False
592670anyTok :: PandocMonad m => LP m Tok
593671anyTok = satisfyTok (const True)
594672
673+singleCharTok :: PandocMonad m => LP m Tok
674+singleCharTok =
675+ satisfyTok $ \case
676+ Tok _ Word t -> T.length t == 1
677+ Tok _ Symbol t -> not (T.any (`Set.member` specialChars) t)
678+ _ -> False
679+
595680singleChar :: PandocMonad m => LP m Tok
596−singleChar = try $ do
597− Tok pos toktype t <- satisfyTok (tokTypeIn [Word, Symbol])
598− guard $ not $ toktype == Symbol &&
599− T.any (`Set.member` specialChars) t
600− if T.length t > 1
601− then do
602− let (t1, t2) = (T.take 1 t, T.drop 1 t)
603− inp <- getInput
604− setInput $ Tok (incSourceColumn pos 1) toktype t2 : inp
605− return $ Tok pos toktype t1
606− else return $ Tok pos toktype t
681+singleChar = singleCharTok <|> singleCharFromWord
682+ where
683+ singleCharFromWord = do
684+ Tok pos toktype t <- disablingWithRaw $ satisfyTok isWordTok
685+ let (t1, t2) = (T.take 1 t, T.drop 1 t)
686+ inp <- getInput
687+ setInput $ Tok pos toktype t1 : Tok (incSourceColumn pos 1) toktype t2 : inp
688+ anyTok
607689
608690specialChars :: Set.Set Char
609691specialChars = Set.fromList "#$%&~_^\\{}"
@@ -646,28 +728,25 @@ grouped parser = try $ do
646728 -- {{a,b}} should be parsed the same as {a,b}
647729 try (grouped parser <* egroup) <|> (mconcat <$> manyTill parser egroup)
648730
649−braced' :: PandocMonad m => LP m Tok -> Int -> LP m [Tok]
650−braced' getTok n =
651− handleEgroup <|> handleBgroup <|> handleOther
652− where handleEgroup = do
653− t <- symbol '}'
654− if n == 1
655− then return []
656− else (t:) <$> braced' getTok (n - 1)
657− handleBgroup = do
658− t <- symbol '{'
659− (t:) <$> braced' getTok (n + 1)
660− handleOther = do
661− t <- getTok
662− (t:) <$> braced' getTok n
731+braced' :: PandocMonad m => LP m Tok -> LP m [Tok]
732+braced' getTok = symbol '{' *> go (1 :: Int)
733+ where
734+ go n = do
735+ t <- getTok
736+ case t of
737+ Tok _ Symbol "}"
738+ | n > 1 -> (t:) <$> go (n - 1)
739+ | otherwise -> return []
740+ Tok _ Symbol "{" -> (t:) <$> go (n + 1)
741+ _ -> (t:) <$> go n
663742
664743braced :: PandocMonad m => LP m [Tok]
665−braced = symbol '{' *> braced' anyTok 1
744+braced = braced' anyTok
666745
667746-- URLs require special handling, because they can contain %
668747-- characters. So we retonenize comments as we go...
669748bracedUrl :: PandocMonad m => LP m [Tok]
670−bracedUrl = bgroup *> braced' (retokenizeComment >> anyTok) 1
749+bracedUrl = braced' (retokenizeComment >> anyTok)
671750
672751-- For handling URLs, which allow literal % characters...
673752retokenizeComment :: PandocMonad m => LP m ()
@@ -723,16 +802,29 @@ ignore raw = do
723802
724803withRaw :: PandocMonad m => LP m a -> LP m (a, [Tok])
725804withRaw parser = do
726− inp <- getInput
805+ rawTokensMap <- sRawTokens <$> getState
806+ let key = case IntMap.lookupMax rawTokensMap of
807+ Nothing -> 0
808+ Just (n,_) -> n + 1
809+ -- insert empty list at key
810+ updateState $ \st -> st{ sRawTokens =
811+ IntMap.insert key [] $ sRawTokens st }
727812 result <- parser
728− nxtpos <- option Nothing ((\(Tok pos' _ _) -> Just pos') <$> lookAhead anyTok)
729− let raw = takeWhile (\(Tok pos _ _) -> maybe True
730− (\p -> sourceName p /= sourceName pos || pos < p) nxtpos) inp
813+ mbRevToks <- IntMap.lookup key . sRawTokens <$> getState
814+ raw <- case mbRevToks of
815+ Just revtoks -> do
816+ updateState $ \st -> st{ sRawTokens =
817+ IntMap.delete key $ sRawTokens st}
818+ return $ reverse revtoks
819+ Nothing ->
820+ throwError $ PandocShouldNeverHappenError $
821+ "sRawTokens has nothing at key " <> T.pack (show key)
731822 return (result, raw)
732823
733824keyval :: PandocMonad m => LP m (Text, Text)
734825keyval = try $ do
735− Tok _ Word key <- satisfyTok isWordTok
826+ key <- untokenize <$> many1 (notFollowedBy (symbol '=') >>
827+ (symbol '-' <|> symbol '_' <|> satisfyTok isWordTok))
736828 sp
737829 val <- option mempty $ do
738830 symbol '='
@@ -792,7 +884,7 @@ getRawCommand name txt = do
792884 (_, rawargs) <- withRaw $
793885 case name of
794886 "write" -> do
795− void $ satisfyTok isWordTok -- digits
887+ void $ many $ satisfyTok isDigitTok -- digits
796888 void braced
797889 "titleformat" -> do
798890 void braced
@@ -807,6 +899,10 @@ getRawCommand name txt = do
807899 void $ many braced
808900 return $ txt <> untokenize rawargs
809901
902+isDigitTok :: Tok -> Bool
903+isDigitTok (Tok _ Word t) = T.all isDigit t
904+isDigitTok _ = False
905+
810906skipopts :: PandocMonad m => LP m ()
811907skipopts = skipMany (void overlaySpecification <|> void rawopt)
812908
@@ -874,3 +970,35 @@ getNextNumber getCurrentNum = do
874970 Just n -> [n, 1]
875971 Nothing -> [1]
876972
973+label :: PandocMonad m => LP m ()
974+label = do
975+ controlSeq "label"
976+ t <- braced
977+ updateState $ \st -> st{ sLastLabel = Just $ untokenize t }
978+
979+setCaption :: PandocMonad m => LP m Inlines -> LP m ()
980+setCaption inline = try $ do
981+ skipopts
982+ ils <- tokWith inline
983+ optional $ try $ spaces *> label
984+ updateState $ \st -> st{ sCaption = Just ils }
985+
986+resetCaption :: PandocMonad m => LP m ()
987+resetCaption = updateState $ \st -> st{ sCaption = Nothing
988+ , sLastLabel = Nothing }
989+
990+env :: PandocMonad m => Text -> LP m a -> LP m a
991+env name p = p <* end_ name
992+
993+tokWith :: PandocMonad m => LP m Inlines -> LP m Inlines
994+tokWith inlineParser = try $ spaces >>
995+ grouped inlineParser
996+ <|> (lookAhead anyControlSeq >> inlineParser)
997+ <|> singleChar'
998+ where singleChar' = do
999+ Tok _ _ t <- singleChar
1000+ return $ str t
1001+
1002+addMeta :: PandocMonad m => ToMetaValue a => Text -> a -> LP m ()
1003+addMeta field val = updateState $ \st ->
1004+ st{ sMeta = addMetaField field val $ sMeta st }
src/Text/Pandoc/Readers/LaTeX/SIunitx.hs+152−71
@@ -1,12 +1,7 @@
1+{-# LANGUAGE ScopedTypeVariables #-}
12{-# LANGUAGE OverloadedStrings #-}
23module Text.Pandoc.Readers.LaTeX.SIunitx
3− ( dosi
4− , doSI
5− , doSIrange
6− , doSInum
7− , doSInumlist
8− , doSIang
9− )
4+ ( siunitxCommands )
105where
116import Text.Pandoc.Builder
127import Text.Pandoc.Readers.LaTeX.Parsing
@@ -15,14 +10,32 @@ import Text.Pandoc.Class
1510import Text.Pandoc.Parsing hiding (blankline, mathDisplay, mathInline,
1611 optional, space, spaces, withRaw, (<|>))
1712import Control.Applicative ((<|>))
13+import Control.Monad (void)
1814import qualified Data.Map as M
1915import Data.Char (isDigit)
2016import Data.Text (Text)
2117import qualified Data.Text as T
2218import Data.List (intersperse)
19+import qualified Data.Sequence as Seq
20+import Text.Pandoc.Walk (walk)
21+
22+siunitxCommands :: PandocMonad m
23+ => LP m Inlines -> M.Map Text (LP m Inlines)
24+siunitxCommands tok = M.fromList
25+ [ ("si", dosi tok)
26+ , ("SI", doSI tok)
27+ , ("SIrange", doSIrange True tok)
28+ , ("numrange", doSIrange False tok)
29+ , ("numlist", doSInumlist)
30+ , ("SIlist", doSIlist tok)
31+ , ("num", doSInum)
32+ , ("ang", doSIang)
33+ ]
2334
2435dosi :: PandocMonad m => LP m Inlines -> LP m Inlines
25−dosi tok = grouped (siUnit tok) <|> siUnit tok
36+dosi tok = do
37+ options <- option [] keyvals
38+ grouped (siUnit options tok) <|> siUnit options tok
2639
2740-- converts e.g. \SI{1}[\$]{} to "$ 1" or \SI{1}{\euro} to "1 €"
2841doSI :: PandocMonad m => LP m Inlines -> LP m Inlines
@@ -57,23 +70,50 @@ doSInumlist = do
5770 mconcat (intersperse (str "," <> space) (init xs)) <>
5871 text ", & " <> last xs
5972
73+doSIlist :: PandocMonad m => LP m Inlines -> LP m Inlines
74+doSIlist tok = do
75+ options <- option [] keyvals
76+ nums <- map tonum . T.splitOn ";" . untokenize <$> braced
77+ unit <- grouped (siUnit options tok) <|> siUnit options tok
78+ let xs = map (<> (str "\xa0" <> unit)) nums
79+ case xs of
80+ [] -> return mempty
81+ [x] -> return x
82+ _ -> return $
83+ mconcat (intersperse (str "," <> space) (init xs)) <>
84+ text ", & " <> last xs
85+
6086parseNum :: Parser Text () Inlines
6187parseNum = (mconcat <$> many parseNumPart) <* eof
6288
89+minus :: Text
90+minus = "\x2212"
91+
92+hyphenToMinus :: Inline -> Inline
93+hyphenToMinus (Str t) = Str (T.replace "-" minus t)
94+hyphenToMinus x = x
95+
6396parseNumPart :: Parser Text () Inlines
6497parseNumPart =
6598 parseDecimalNum <|>
6699 parseComma <|>
67100 parsePlusMinus <|>
101+ parsePM <|>
68102 parseI <|>
69103 parseExp <|>
70104 parseX <|>
71105 parseSpace
72106 where
73− parseDecimalNum = do
74− pref <- option mempty $ (mempty <$ char '+') <|> ("\x2212" <$ char '-')
75− basenum <- (pref <>) . T.pack
76− <$> many1 (satisfy (\c -> isDigit c || c == '.'))
107+ parseDecimalNum, parsePlusMinus, parsePM,
108+ parseComma, parseI, parseX,
109+ parseExp, parseSpace :: Parser Text () Inlines
110+ parseDecimalNum = try $ do
111+ pref <- option mempty $ (mempty <$ char '+') <|> (minus <$ char '-')
112+ basenum' <- many1 (satisfy (\c -> isDigit c || c == '.'))
113+ let basenum = pref <> T.pack
114+ (case basenum' of
115+ '.':_ -> '0':basenum'
116+ _ -> basenum')
77117 uncertainty <- option mempty $ T.pack <$> parseParens
78118 if T.null uncertainty
79119 then return $ str basenum
@@ -91,6 +131,7 @@ parseNumPart =
91131 | otherwise -> "." <> t
92132 parseComma = str "." <$ char ','
93133 parsePlusMinus = str "\xa0\xb1\xa0" <$ try (string "+-")
134+ parsePM = str "\xa0\xb1\xa0" <$ try (string "\\pm")
94135 parseParens =
95136 char '(' *> many1 (satisfy (\c -> isDigit c || c == '.')) <* char ')'
96137 parseI = str "i" <$ char 'i'
@@ -103,11 +144,14 @@ doSIang :: PandocMonad m => LP m Inlines
103144doSIang = do
104145 skipopts
105146 ps <- T.splitOn ";" . untokenize <$> braced
147+ let dropPlus t = case T.uncons t of
148+ Just ('+',t') -> t'
149+ _ -> t
106150 case ps ++ repeat "" of
107151 (d:m:s:_) -> return $
108− (if T.null d then mempty else str d <> str "\xb0") <>
109− (if T.null m then mempty else str m <> str "\x2032") <>
110− (if T.null s then mempty else str s <> str "\x2033")
152+ (if T.null d then mempty else str (dropPlus d) <> str "\xb0") <>
153+ (if T.null m then mempty else str (dropPlus m) <> str "\x2032") <>
154+ (if T.null s then mempty else str (dropPlus s) <> str "\x2033")
111155 _ -> return mempty
112156
113157-- converts e.g. \SIrange{100}{200}{\ms} to "100 ms--200 ms"
@@ -136,40 +180,99 @@ doSIrange includeUnits tok = do
136180emptyOr160 :: Inlines -> Inlines
137181emptyOr160 x = if x == mempty then x else str "\160"
138182
139−siUnit :: PandocMonad m => LP m Inlines -> LP m Inlines
140−siUnit tok = try (do
141− Tok _ (CtrlSeq name) _ <- anyControlSeq
142− case name of
143− "square" -> do
144− unit <- siUnit tok
145− return $ unit <> superscript "2"
146− "cubic" -> do
147− unit <- siUnit tok
148− return $ unit <> superscript "3"
149− "raisetothe" -> do
150− n <- tok
151− unit <- siUnit tok
152− return $ unit <> superscript n
153− _ ->
154− case M.lookup name siUnitMap of
155− Just il ->
156− option il $
157− choice
158− [ (il <> superscript "2") <$ controlSeq "squared"
159− , (il <> superscript "3") <$ controlSeq "cubed"
160− , (\n -> il <> superscript n) <$> (controlSeq "tothe" *> tok)
161− ]
162− Nothing -> fail "not an siunit unit command")
163− <|> (lookAhead anyControlSeq >> tok)
164− <|> (do Tok _ Word t <- satisfyTok isWordTok
165− return $ str t)
166− <|> (symbol '^' *> (superscript <$> tok))
167− <|> (symbol '_' *> (subscript <$> tok))
168− <|> ("\xa0" <$ symbol '.')
169− <|> ("\xa0" <$ symbol '~')
170− <|> tok
171− <|> (do Tok _ _ t <- anyTok
172− return (str t))
183+siUnit :: forall m. PandocMonad m => [(Text,Text)] -> LP m Inlines -> LP m Inlines
184+siUnit options tok = mconcat . intersperse (str "\xa0") <$> many1 siUnitPart
185+ where
186+ siUnitPart :: LP m Inlines
187+ siUnitPart = try $ do
188+ skipMany (void (symbol '.') <|> void (symbol '~') <|> spaces1)
189+ x <- ((siPrefix <*> siBase)
190+ <|> (do u <- siBase <|> tok
191+ option u $ siSuffix <*> pure u))
192+ option x (siInfix x)
193+ siInfix :: Inlines -> LP m Inlines
194+ siInfix u1 = try $
195+ (do _ <- controlSeq "per"
196+ u2 <- siUnitPart
197+ let useSlash = lookup "per-mode" options == Just "symbol"
198+ if useSlash
199+ then return (u1 <> str "/" <> u2)
200+ else return (u1 <> str "\xa0" <> negateExponent u2))
201+ <|> (do _ <- symbol '/'
202+ u2 <- siUnitPart
203+ return (u1 <> str "/" <> u2))
204+ siPrefix :: LP m (Inlines -> Inlines)
205+ siPrefix =
206+ (do _ <- controlSeq "square"
207+ skipopts
208+ return (<> superscript "2"))
209+ <|> (do _ <- controlSeq "cubic"
210+ skipopts
211+ return (<> superscript "3"))
212+ <|> (do _ <- controlSeq "raisetothe"
213+ skipopts
214+ n <- walk hyphenToMinus <$> tok
215+ return (<> superscript n))
216+ siSuffix :: LP m (Inlines -> Inlines)
217+ siSuffix =
218+ (do _ <- controlSeq "squared"
219+ skipopts
220+ return (<> superscript "2"))
221+ <|> (do _ <- controlSeq "cubed"
222+ skipopts
223+ return (<> superscript "3"))
224+ <|> (do _ <- controlSeq "tothe"
225+ skipopts
226+ n <- walk hyphenToMinus <$> tok
227+ return (<> superscript n))
228+ <|> (symbol '^' *> (do n <- walk hyphenToMinus <$> tok
229+ return (<> superscript n)))
230+ <|> (symbol '_' *> (do n <- walk hyphenToMinus <$> tok
231+ return (<> subscript n)))
232+ negateExponent :: Inlines -> Inlines
233+ negateExponent ils =
234+ case Seq.viewr (unMany ils) of
235+ xs Seq.:> Superscript ss -> (Many xs) <>
236+ superscript (str minus <> fromList ss)
237+ _ -> ils <> superscript (str (minus <> "1"))
238+ siBase :: LP m Inlines
239+ siBase =
240+ ((try
241+ (do Tok _ (CtrlSeq name) _ <- anyControlSeq
242+ case M.lookup name siUnitModifierMap of
243+ Just il -> (il <>) <$> siBase
244+ Nothing ->
245+ case M.lookup name siUnitMap of
246+ Just il -> pure il
247+ Nothing -> fail "not a unit command"))
248+ <|> (do Tok _ Word t <- satisfyTok isWordTok
249+ return $ str t)
250+ )
251+
252+siUnitModifierMap :: M.Map Text Inlines
253+siUnitModifierMap = M.fromList
254+ [ ("atto", str "a")
255+ , ("centi", str "c")
256+ , ("deca", str "d")
257+ , ("deci", str "d")
258+ , ("deka", str "d")
259+ , ("exa", str "E")
260+ , ("femto", str "f")
261+ , ("giga", str "G")
262+ , ("hecto", str "h")
263+ , ("kilo", str "k")
264+ , ("mega", str "M")
265+ , ("micro", str "μ")
266+ , ("milli", str "m")
267+ , ("nano", str "n")
268+ , ("peta", str "P")
269+ , ("pico", str "p")
270+ , ("tera", str "T")
271+ , ("yocto", str "y")
272+ , ("yotta", str "Y")
273+ , ("zepto", str "z")
274+ , ("zetta", str "Z")
275+ ]
173276
174277siUnitMap :: M.Map Text Inlines
175278siUnitMap = M.fromList
@@ -269,7 +372,6 @@ siUnitMap = M.fromList
269372 , ("arcsecond", str "″")
270373 , ("astronomicalunit", str "ua")
271374 , ("atomicmassunit", str "u")
272− , ("atto", str "a")
273375 , ("bar", str "bar")
274376 , ("barn", str "b")
275377 , ("becquerel", str "Bq")
@@ -277,51 +379,38 @@ siUnitMap = M.fromList
277379 , ("bohr", emph (str "a") <> subscript (str "0"))
278380 , ("candela", str "cd")
279381 , ("celsius", str "°C")
280− , ("centi", str "c")
281382 , ("clight", emph (str "c") <> subscript (str "0"))
282383 , ("coulomb", str "C")
283384 , ("dalton", str "Da")
284385 , ("day", str "d")
285− , ("deca", str "d")
286− , ("deci", str "d")
287386 , ("decibel", str "db")
288387 , ("degreeCelsius",str "°C")
289388 , ("degree", str "°")
290− , ("deka", str "d")
291389 , ("electronmass", emph (str "m") <> subscript (str "e"))
292390 , ("electronvolt", str "eV")
293391 , ("elementarycharge", emph (str "e"))
294− , ("exa", str "E")
295392 , ("farad", str "F")
296− , ("femto", str "f")
297− , ("giga", str "G")
298393 , ("gram", str "g")
299394 , ("gray", str "Gy")
300395 , ("hartree", emph (str "E") <> subscript (str "h"))
301396 , ("hectare", str "ha")
302− , ("hecto", str "h")
303397 , ("henry", str "H")
304398 , ("hertz", str "Hz")
305399 , ("hour", str "h")
306400 , ("joule", str "J")
307401 , ("katal", str "kat")
308402 , ("kelvin", str "K")
309− , ("kilo", str "k")
310403 , ("kilogram", str "kg")
311404 , ("knot", str "kn")
312405 , ("liter", str "L")
313406 , ("litre", str "l")
314407 , ("lumen", str "lm")
315408 , ("lux", str "lx")
316− , ("mega", str "M")
317409 , ("meter", str "m")
318410 , ("metre", str "m")
319− , ("micro", str "μ")
320− , ("milli", str "m")
321411 , ("minute", str "min")
322412 , ("mmHg", str "mmHg")
323413 , ("mole", str "mol")
324− , ("nano", str "n")
325414 , ("nauticalmile", str "M")
326415 , ("neper", str "Np")
327416 , ("newton", str "N")
@@ -329,25 +418,17 @@ siUnitMap = M.fromList
329418 , ("Pa", str "Pa")
330419 , ("pascal", str "Pa")
331420 , ("percent", str "%")
332− , ("per", str "/")
333− , ("peta", str "P")
334− , ("pico", str "p")
335421 , ("planckbar", emph (str "\x210f"))
336422 , ("radian", str "rad")
337423 , ("second", str "s")
338424 , ("siemens", str "S")
339425 , ("sievert", str "Sv")
340426 , ("steradian", str "sr")
341− , ("tera", str "T")
342427 , ("tesla", str "T")
343428 , ("tonne", str "t")
344429 , ("volt", str "V")
345430 , ("watt", str "W")
346431 , ("weber", str "Wb")
347− , ("yocto", str "y")
348− , ("yotta", str "Y")
349− , ("zepto", str "z")
350− , ("zetta", str "Z")
351432 ]
352433
353434
src/Text/Pandoc/Readers/LaTeX/Table.hsnew file+378
@@ -0,0 +1,378 @@
1+{-# LANGUAGE BangPatterns #-}
2+{-# LANGUAGE OverloadedStrings #-}
3+module Text.Pandoc.Readers.LaTeX.Table
4+ ( tableEnvironments )
5+where
6+
7+import Data.Functor (($>))
8+import Text.Pandoc.Class
9+import Text.Pandoc.Readers.LaTeX.Parsing
10+import Text.Pandoc.Readers.LaTeX.Types
11+import Text.Pandoc.Builder as B
12+import qualified Data.Map as M
13+import Data.Text (Text)
14+import Data.Maybe (fromMaybe)
15+import qualified Data.Text as T
16+import Control.Applicative ((<|>), optional, many)
17+import Control.Monad (when, void)
18+import Text.Pandoc.Shared (safeRead, trim)
19+import Text.Pandoc.Logging (LogMessage(SkippedContent))
20+import Text.Pandoc.Walk (walkM)
21+import Text.Pandoc.Parsing hiding (blankline, many, mathDisplay, mathInline,
22+ optional, space, spaces, withRaw, (<|>))
23+
24+tableEnvironments :: PandocMonad m
25+ => LP m Blocks
26+ -> LP m Inlines
27+ -> M.Map Text (LP m Blocks)
28+tableEnvironments blocks inline =
29+ M.fromList
30+ [ ("longtable", env "longtable" $
31+ resetCaption *>
32+ simpTable blocks inline "longtable" False >>= addTableCaption)
33+ , ("table", env "table" $
34+ skipopts *> resetCaption *> blocks >>= addTableCaption)
35+ , ("tabular*", env "tabular*" $ simpTable blocks inline "tabular*" True)
36+ , ("tabularx", env "tabularx" $ simpTable blocks inline "tabularx" True)
37+ , ("tabular", env "tabular" $ simpTable blocks inline "tabular" False)
38+ ]
39+
40+hline :: PandocMonad m => LP m ()
41+hline = try $ do
42+ spaces
43+ controlSeq "hline" <|>
44+ -- booktabs rules:
45+ controlSeq "toprule" <|>
46+ controlSeq "bottomrule" <|>
47+ controlSeq "midrule" <|>
48+ controlSeq "endhead" <|>
49+ controlSeq "endfirsthead"
50+ spaces
51+ optional rawopt
52+ return ()
53+
54+lbreak :: PandocMonad m => LP m Tok
55+lbreak = (controlSeq "\\" <|> controlSeq "tabularnewline")
56+ <* skipopts <* spaces
57+
58+amp :: PandocMonad m => LP m Tok
59+amp = symbol '&'
60+
61+-- Split a Word into individual Symbols (for parseAligns)
62+splitWordTok :: PandocMonad m => LP m ()
63+splitWordTok = do
64+ inp <- getInput
65+ case inp of
66+ (Tok spos Word t : rest) ->
67+ setInput $ map (Tok spos Symbol . T.singleton) (T.unpack t) <> rest
68+ _ -> return ()
69+
70+parseAligns :: PandocMonad m => LP m [(Alignment, ColWidth, ([Tok], [Tok]))]
71+parseAligns = try $ do
72+ let maybeBar = skipMany
73+ (try $ sp *> (() <$ symbol '|' <|> () <$ (symbol '@' >> braced)))
74+ let cAlign = AlignCenter <$ symbol 'c'
75+ let lAlign = AlignLeft <$ symbol 'l'
76+ let rAlign = AlignRight <$ symbol 'r'
77+ let parAlign = AlignLeft <$ symbol 'p'
78+ -- aligns from tabularx
79+ let xAlign = AlignLeft <$ symbol 'X'
80+ let mAlign = AlignLeft <$ symbol 'm'
81+ let bAlign = AlignLeft <$ symbol 'b'
82+ let alignChar = splitWordTok *> ( cAlign <|> lAlign <|> rAlign <|> parAlign
83+ <|> xAlign <|> mAlign <|> bAlign )
84+ let alignPrefix = symbol '>' >> braced
85+ let alignSuffix = symbol '<' >> braced
86+ let colWidth = try $ do
87+ symbol '{'
88+ ds <- trim . untokenize <$> manyTill anyTok (controlSeq "linewidth")
89+ spaces
90+ symbol '}'
91+ return $ safeRead ds
92+ let alignSpec = do
93+ pref <- option [] alignPrefix
94+ spaces
95+ al <- alignChar
96+ width <- colWidth <|> option Nothing (do s <- untokenize <$> braced
97+ pos <- getPosition
98+ report $ SkippedContent s pos
99+ return Nothing)
100+ spaces
101+ suff <- option [] alignSuffix
102+ return (al, width, (pref, suff))
103+ let starAlign = do -- '*{2}{r}' == 'rr', we just expand like a macro
104+ symbol '*'
105+ spaces
106+ ds <- trim . untokenize <$> braced
107+ spaces
108+ spec <- braced
109+ case safeRead ds of
110+ Just n ->
111+ getInput >>= setInput . (mconcat (replicate n spec) ++)
112+ Nothing -> Prelude.fail $ "Could not parse " <> T.unpack ds <> " as number"
113+ bgroup
114+ spaces
115+ maybeBar
116+ aligns' <- many $ try $ spaces >> optional starAlign >>
117+ (alignSpec <* maybeBar)
118+ spaces
119+ egroup
120+ spaces
121+ return $ map toSpec aligns'
122+ where
123+ toColWidth (Just w) | w > 0 = ColWidth w
124+ toColWidth _ = ColWidthDefault
125+ toSpec (x, y, z) = (x, toColWidth y, z)
126+
127+-- N.B. this parser returns a Row that may have erroneous empty cells
128+-- in it. See the note above fixTableHead for details.
129+parseTableRow :: PandocMonad m
130+ => LP m Blocks -- ^ block parser
131+ -> LP m Inlines -- ^ inline parser
132+ -> Text -- ^ table environment name
133+ -> [([Tok], [Tok])] -- ^ pref/suffixes
134+ -> LP m Row
135+parseTableRow blocks inline envname prefsufs = do
136+ notFollowedBy (spaces *> end_ envname)
137+ -- contexts that can contain & that is not colsep:
138+ let canContainAmp (Tok _ (CtrlSeq "begin") _) = True
139+ canContainAmp (Tok _ (CtrlSeq "verb") _) = True
140+ canContainAmp (Tok _ (CtrlSeq "Verb") _) = True
141+ canContainAmp _ = False
142+ -- add prefixes and suffixes in token stream:
143+ let celltoks (pref, suff) = do
144+ prefpos <- getPosition
145+ contents <- mconcat <$>
146+ many ( snd <$> withRaw
147+ ((lookAhead (controlSeq "parbox") >>
148+ void blocks) -- #5711
149+ <|>
150+ (lookAhead (satisfyTok canContainAmp) >> void inline)
151+ <|>
152+ (lookAhead (symbol '$') >> void inline))
153+ <|>
154+ (do notFollowedBy
155+ (() <$ amp <|> () <$ lbreak <|> end_ envname)
156+ count 1 anyTok) )
157+
158+ suffpos <- getPosition
159+ option [] (count 1 amp)
160+ return $ map (setpos prefpos) pref ++ contents ++ map (setpos suffpos) suff
161+ rawcells <- mapM celltoks prefsufs
162+ cells <- mapM (parseFromToks (parseTableCell blocks)) rawcells
163+ spaces
164+ return $ Row nullAttr cells
165+
166+parseTableCell :: PandocMonad m => LP m Blocks -> LP m Cell
167+parseTableCell blocks = do
168+ spaces
169+ updateState $ \st -> st{ sInTableCell = True }
170+ cell' <- multicolumnCell blocks
171+ <|> multirowCell blocks
172+ <|> parseSimpleCell
173+ <|> parseEmptyCell
174+ updateState $ \st -> st{ sInTableCell = False }
175+ spaces
176+ return cell'
177+ where
178+ -- The parsing of empty cells is important in LaTeX, especially when dealing
179+ -- with multirow/multicolumn. See #6603.
180+ parseEmptyCell = spaces $> emptyCell
181+ parseSimpleCell = simpleCell <$> (plainify <$> blocks)
182+
183+
184+cellAlignment :: PandocMonad m => LP m Alignment
185+cellAlignment = skipMany (symbol '|') *> alignment <* skipMany (symbol '|')
186+ where
187+ alignment = do
188+ c <- untoken <$> singleChar
189+ return $ case c of
190+ "l" -> AlignLeft
191+ "r" -> AlignRight
192+ "c" -> AlignCenter
193+ "*" -> AlignDefault
194+ _ -> AlignDefault
195+
196+plainify :: Blocks -> Blocks
197+plainify bs = case toList bs of
198+ [Para ils] -> plain (fromList ils)
199+ _ -> bs
200+
201+multirowCell :: PandocMonad m => LP m Blocks -> LP m Cell
202+multirowCell blocks = controlSeq "multirow" >> do
203+ -- Full prototype for \multirow macro is:
204+ -- \multirow[vpos]{nrows}[bigstruts]{width}[vmove]{text}
205+ -- However, everything except `nrows` and `text` make
206+ -- sense in the context of the Pandoc AST
207+ _ <- optional $ symbol '[' *> cellAlignment <* symbol ']' -- vertical position
208+ nrows <- fmap (fromMaybe 1 . safeRead . untokenize) braced
209+ _ <- optional $ symbol '[' *> manyTill anyTok (symbol ']') -- bigstrut-related
210+ _ <- symbol '{' *> manyTill anyTok (symbol '}') -- Cell width
211+ _ <- optional $ symbol '[' *> manyTill anyTok (symbol ']') -- Length used for fine-tuning
212+ content <- symbol '{' *> (plainify <$> blocks) <* symbol '}'
213+ return $ cell AlignDefault (RowSpan nrows) (ColSpan 1) content
214+
215+multicolumnCell :: PandocMonad m => LP m Blocks -> LP m Cell
216+multicolumnCell blocks = controlSeq "multicolumn" >> do
217+ span' <- fmap (fromMaybe 1 . safeRead . untokenize) braced
218+ alignment <- symbol '{' *> cellAlignment <* symbol '}'
219+
220+ let singleCell = do
221+ content <- plainify <$> blocks
222+ return $ cell alignment (RowSpan 1) (ColSpan span') content
223+
224+ -- Two possible contents: either a \multirow cell, or content.
225+ -- E.g. \multicol{1}{c}{\multirow{2}{1em}{content}}
226+ -- Note that a \multirow cell can be nested in a \multicolumn,
227+ -- but not the other way around. See #6603
228+ let nestedCell = do
229+ (Cell _ _ (RowSpan rs) _ bs) <- multirowCell blocks
230+ return $ cell
231+ alignment
232+ (RowSpan rs)
233+ (ColSpan span')
234+ (fromList bs)
235+
236+ symbol '{' *> (nestedCell <|> singleCell) <* symbol '}'
237+
238+-- LaTeX tables are stored with empty cells underneath multirow cells
239+-- denoting the grid spaces taken up by them. More specifically, if a
240+-- cell spans m rows, then it will overwrite all the cells in the
241+-- columns it spans for (m-1) rows underneath it, requiring padding
242+-- cells in these places. These padding cells need to be removed for
243+-- proper table reading. See #6603.
244+--
245+-- These fixTable functions do not otherwise fix up malformed
246+-- input tables: that is left to the table builder.
247+fixTableHead :: TableHead -> TableHead
248+fixTableHead (TableHead attr rows) = TableHead attr rows'
249+ where
250+ rows' = fixTableRows rows
251+
252+fixTableBody :: TableBody -> TableBody
253+fixTableBody (TableBody attr rhc th tb)
254+ = TableBody attr rhc th' tb'
255+ where
256+ th' = fixTableRows th
257+ tb' = fixTableRows tb
258+
259+fixTableRows :: [Row] -> [Row]
260+fixTableRows = fixTableRows' $ repeat Nothing
261+ where
262+ fixTableRows' oldHang (Row attr cells : rs)
263+ = let (newHang, cells') = fixTableRow oldHang cells
264+ rs' = fixTableRows' newHang rs
265+ in Row attr cells' : rs'
266+ fixTableRows' _ [] = []
267+
268+-- The overhang is represented as Just (relative cell dimensions) or
269+-- Nothing for an empty grid space.
270+fixTableRow :: [Maybe (ColSpan, RowSpan)] -> [Cell] -> ([Maybe (ColSpan, RowSpan)], [Cell])
271+fixTableRow oldHang cells
272+ -- If there's overhang, drop cells until their total width meets the
273+ -- width of the occupied grid spaces (or we run out)
274+ | (n, prefHang, restHang) <- splitHang oldHang
275+ , n > 0
276+ = let cells' = dropToWidth getCellW n cells
277+ (restHang', cells'') = fixTableRow restHang cells'
278+ in (prefHang restHang', cells'')
279+ -- Otherwise record the overhang of a pending cell and fix the rest
280+ -- of the row
281+ | c@(Cell _ _ h w _):cells' <- cells
282+ = let h' = max 1 h
283+ w' = max 1 w
284+ oldHang' = dropToWidth getHangW w' oldHang
285+ (newHang, cells'') = fixTableRow oldHang' cells'
286+ in (toHang w' h' <> newHang, c : cells'')
287+ | otherwise
288+ = (oldHang, [])
289+ where
290+ getCellW (Cell _ _ _ w _) = w
291+ getHangW = maybe 1 fst
292+ getCS (ColSpan n) = n
293+
294+ toHang c r
295+ | r > 1 = [Just (c, r)]
296+ | otherwise = replicate (getCS c) Nothing
297+
298+ -- Take the prefix of the overhang list representing filled grid
299+ -- spaces. Also return the remainder and the length of this prefix.
300+ splitHang = splitHang' 0 id
301+
302+ splitHang' !n l (Just (c, r):xs)
303+ = splitHang' (n + c) (l . (toHang c (r-1) ++)) xs
304+ splitHang' n l xs = (n, l, xs)
305+
306+ -- Drop list items until the total width of the dropped items
307+ -- exceeds the passed width.
308+ dropToWidth _ n l | n < 1 = l
309+ dropToWidth wproj n (c:cs) = dropToWidth wproj (n - wproj c) cs
310+ dropToWidth _ _ [] = []
311+
312+simpTable :: PandocMonad m
313+ => LP m Blocks
314+ -> LP m Inlines
315+ -> Text
316+ -> Bool
317+ -> LP m Blocks
318+simpTable blocks inline envname hasWidthParameter = try $ do
319+ when hasWidthParameter $ () <$ tokWith inline
320+ skipopts
321+ colspecs <- parseAligns
322+ let (aligns, widths, prefsufs) = unzip3 colspecs
323+ optional $ controlSeq "caption" *> setCaption inline
324+ spaces
325+ optional label
326+ spaces
327+ optional lbreak
328+ spaces
329+ skipMany hline
330+ spaces
331+ header' <- option [] . try . fmap (:[]) $
332+ parseTableRow blocks inline envname prefsufs <*
333+ lbreak <* many1 hline
334+ spaces
335+ rows <- sepEndBy (parseTableRow blocks inline envname prefsufs)
336+ (lbreak <* optional (skipMany hline))
337+ spaces
338+ optional $ controlSeq "caption" *> setCaption inline
339+ spaces
340+ optional label
341+ spaces
342+ optional lbreak
343+ spaces
344+ lookAhead $ controlSeq "end" -- make sure we're at end
345+ let th = fixTableHead $ TableHead nullAttr header'
346+ let tbs = [fixTableBody $ TableBody nullAttr 0 [] rows]
347+ let tf = TableFoot nullAttr []
348+ return $ table emptyCaption (zip aligns widths) th tbs tf
349+
350+addTableCaption :: PandocMonad m => Blocks -> LP m Blocks
351+addTableCaption = walkM go
352+ where go (Table attr c spec th tb tf) = do
353+ st <- getState
354+ let mblabel = sLastLabel st
355+ capt <- case (sCaption st, mblabel) of
356+ (Just ils, Nothing) -> return $ caption Nothing (plain ils)
357+ (Just ils, Just lab) -> do
358+ num <- getNextNumber sLastTableNum
359+ setState
360+ st{ sLastTableNum = num
361+ , sLabels = M.insert lab
362+ [Str (renderDottedNum num)]
363+ (sLabels st) }
364+ return $ caption Nothing (plain ils) -- add number??
365+ (Nothing, _) -> return c
366+ let attr' = case (attr, mblabel) of
367+ ((_,classes,kvs), Just ident) ->
368+ (ident,classes,kvs)
369+ _ -> attr
370+ return $ addAttrDiv attr' $ Table nullAttr capt spec th tb tf
371+ go x = return x
372+
373+-- TODO: For now we add a Div to contain table attributes, since
374+-- most writers don't do anything yet with attributes on Table.
375+-- This can be removed when that changes.
376+addAttrDiv :: Attr -> Block -> Block
377+addAttrDiv ("",[],[]) b = b
378+addAttrDiv attr b = Div attr [b]
src/Text/Pandoc/Readers/LaTeX/Types.hs+15−2
@@ -1,6 +1,7 @@
1+{-# LANGUAGE FlexibleInstances #-}
12{- |
23 Module : Text.Pandoc.Readers.LaTeX.Types
3− Copyright : Copyright (C) 2017-2020 John MacFarlane
4+ Copyright : Copyright (C) 2017-2021 John MacFarlane
45 License : GNU GPL, version 2 or above
56
67 Maintainer : John MacFarlane <jgm@berkeley.edu>
@@ -18,7 +19,9 @@ module Text.Pandoc.Readers.LaTeX.Types ( Tok(..)
1819 )
1920where
2021import Data.Text (Text)
21−import Text.Parsec.Pos (SourcePos)
22+import Text.Parsec.Pos (SourcePos, sourceName)
23+import Text.Pandoc.Sources
24+import Data.List (groupBy)
2225
2326data TokType = CtrlSeq Text | Spaces | Newline | Symbol | Word | Comment |
2427 Esc1 | Esc2 | Arg Int
@@ -27,6 +30,16 @@ data TokType = CtrlSeq Text | Spaces | Newline | Symbol | Word | Comment |
2730data Tok = Tok SourcePos TokType Text
2831 deriving (Eq, Ord, Show)
2932
33+instance ToSources [Tok] where
34+ toSources = Sources
35+ . map (\ts -> case ts of
36+ Tok p _ _ : _ -> (p, mconcat $ map tokToText ts)
37+ _ -> error "toSources [Tok] encountered empty group")
38+ . groupBy (\(Tok p1 _ _) (Tok p2 _ _) -> sourceName p1 == sourceName p2)
39+
40+tokToText :: Tok -> Text
41+tokToText (Tok _ _ t) = t
42+
3043data ExpansionPoint = ExpandWhenDefined | ExpandWhenUsed
3144 deriving (Eq, Ord, Show)
3245
src/Text/Pandoc/Readers/Man.hs+18−11
@@ -20,7 +20,7 @@ import Control.Monad (liftM, mzero, guard, void)
2020import Control.Monad.Trans (lift)
2121import Control.Monad.Except (throwError)
2222import Data.Maybe (catMaybes, isJust)
23−import Data.List (intersperse, intercalate)
23+import Data.List (intersperse)
2424import qualified Data.Text as T
2525import Text.Pandoc.Builder as B
2626import Text.Pandoc.Class.PandocMonad (PandocMonad(..), report)
@@ -29,9 +29,8 @@ import Text.Pandoc.Logging (LogMessage(..))
2929import Text.Pandoc.Options
3030import Text.Pandoc.Parsing
3131import Text.Pandoc.Walk (query)
32−import Text.Pandoc.Shared (crFilter, mapLeft)
32+import Text.Pandoc.Shared (mapLeft)
3333import Text.Pandoc.Readers.Roff -- TODO explicit imports
34−import Text.Parsec hiding (tokenPrim)
3534import qualified Text.Parsec as Parsec
3635import Text.Parsec.Pos (updatePosString)
3736import qualified Data.Foldable as Foldable
@@ -50,13 +49,20 @@ type ManParser m = ParserT [RoffToken] ManState m
5049
5150
5251-- | Read man (troff) from an input string and return a Pandoc document.
53−readMan :: PandocMonad m => ReaderOptions -> T.Text -> m Pandoc
54−readMan opts txt = do
55− tokenz <- lexRoff (initialPos "input") (crFilter txt)
52+readMan :: (PandocMonad m, ToSources a)
53+ => ReaderOptions
54+ -> a
55+ -> m Pandoc
56+readMan opts s = do
57+ let Sources inps = toSources s
58+ tokenz <- mconcat <$> mapM (uncurry lexRoff) inps
5659 let state = def {readerOptions = opts} :: ManState
60+ let fixError (PandocParsecError _ e) = PandocParsecError (Sources inps) e
61+ fixError e = e
5762 eitherdoc <- readWithMTokens parseMan state
5863 (Foldable.toList . unRoffTokens $ tokenz)
59− either throwError return eitherdoc
64+ either (throwError . fixError) return eitherdoc
65+
6066
6167readWithMTokens :: PandocMonad m
6268 => ParserT [RoffToken] ManState m a -- ^ parser
@@ -64,9 +70,10 @@ readWithMTokens :: PandocMonad m
6470 -> [RoffToken] -- ^ input
6571 -> m (Either PandocError a)
6672readWithMTokens parser state input =
67− let leftF = PandocParsecError . T.pack . intercalate "\n" $ show <$> input
73+ let leftF = PandocParsecError mempty
6874 in mapLeft leftF `liftM` runParserT parser state "source" input
6975
76+
7077parseMan :: PandocMonad m => ManParser m Pandoc
7178parseMan = do
7279 bs <- many parseBlock <* eof
@@ -89,7 +96,7 @@ parseBlock = choice [ parseList
8996
9097parseTable :: PandocMonad m => ManParser m Blocks
9198parseTable = do
92− modifyState $ \st -> st { tableCellsPlain = True }
99+ updateState $ \st -> st { tableCellsPlain = True }
93100 let isTbl Tbl{} = True
94101 isTbl _ = False
95102 Tbl _opts rows pos <- msatisfy isTbl
@@ -135,7 +142,7 @@ parseTable = do
135142 case res' of
136143 Left _ -> Prelude.fail "Could not parse table cell"
137144 Right x -> do
138− modifyState $ \s -> s{ tableCellsPlain = False }
145+ updateState $ \s -> s{ tableCellsPlain = False }
139146 return x
140147 Right x -> return x
141148
@@ -222,7 +229,7 @@ parseTitle = do
222229 setMeta "section" (linePartsToInlines y)
223230 [x] -> setMeta "title" (linePartsToInlines x)
224231 [] -> id
225− modifyState $ \st -> st{ metadata = adjustMeta $ metadata st }
232+ updateState $ \st -> st{ metadata = adjustMeta $ metadata st }
226233 return mempty
227234
228235linePartsToInlines :: [LinePart] -> Inlines
src/Text/Pandoc/Readers/Markdown.hs+287−239

This diff is too big to show here.

src/Text/Pandoc/Readers/MediaWiki.hs+21−14

This diff is too big to show here.

src/Text/Pandoc/Readers/Metadata.hs+45−18

This diff is too big to show here.

src/Text/Pandoc/Readers/Muse.hs+10−9

This diff is too big to show here.

src/Text/Pandoc/Readers/Native.hs+8−6

This diff is too big to show here.

src/Text/Pandoc/Readers/OPML.hs+22−36

This diff is too big to show here.

src/Text/Pandoc/Readers/Odt.hs+28−25

This diff is too big to show here.

src/Text/Pandoc/Readers/Odt/Arrows/State.hs+2−2

This diff is too big to show here.

src/Text/Pandoc/Readers/Odt/ContentReader.hs+19−13

This diff is too big to show here.

src/Text/Pandoc/Readers/Odt/Generic/Namespaces.hs+2−1

This diff is too big to show here.

src/Text/Pandoc/Readers/Odt/Generic/Utils.hs+9−24

This diff is too big to show here.

src/Text/Pandoc/Readers/Odt/Generic/XMLConverter.hs+13−14

This diff is too big to show here.

src/Text/Pandoc/Readers/Odt/Namespaces.hs+6−5

This diff is too big to show here.

src/Text/Pandoc/Readers/Odt/StyleReader.hs+21−17

This diff is too big to show here.

src/Text/Pandoc/Readers/Org.hs+5−8

This diff is too big to show here.

src/Text/Pandoc/Readers/Org/BlockStarts.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Readers/Org/Blocks.hs+59−21

This diff is too big to show here.

src/Text/Pandoc/Readers/Org/DocumentTree.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Readers/Org/ExportSettings.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Readers/Org/Inlines.hs+9−8

This diff is too big to show here.

src/Text/Pandoc/Readers/Org/Meta.hs+11−5

This diff is too big to show here.

src/Text/Pandoc/Readers/Org/ParserState.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Readers/Org/Parsing.hs+6−6

This diff is too big to show here.

src/Text/Pandoc/Readers/Org/Shared.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Readers/RST.hs+85−116

This diff is too big to show here.

src/Text/Pandoc/Readers/Roff.hs+16−18

This diff is too big to show here.

src/Text/Pandoc/Readers/TWiki.hs+7−27

This diff is too big to show here.

src/Text/Pandoc/Readers/Textile.hs+96−90

This diff is too big to show here.

src/Text/Pandoc/Readers/TikiWiki.hs+6−6

This diff is too big to show here.

src/Text/Pandoc/Readers/Txt2Tags.hs+48−22

This diff is too big to show here.

src/Text/Pandoc/Readers/Vimwiki.hs+12−7

This diff is too big to show here.

src/Text/Pandoc/RoffChar.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/SelfContained.hs+5−8

This diff is too big to show here.

src/Text/Pandoc/Shared.hs+52−101

This diff is too big to show here.

src/Text/Pandoc/Slides.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Sources.hsnew file+195

This diff is too big to show here.

src/Text/Pandoc/Templates.hs+5−1

This diff is too big to show here.

src/Text/Pandoc/Translations.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/UTF8.hs+21−24

This diff is too big to show here.

src/Text/Pandoc/UUID.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers.hs+9−4

This diff is too big to show here.

src/Text/Pandoc/Writers/AsciiDoc.hs+11−5

This diff is too big to show here.

src/Text/Pandoc/Writers/BibTeX.hsnew file+61

This diff is too big to show here.

src/Text/Pandoc/Writers/CommonMark.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/ConTeXt.hs+45−39

This diff is too big to show here.

src/Text/Pandoc/Writers/CslJson.hs+10−9

This diff is too big to show here.

src/Text/Pandoc/Writers/Custom.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/Docbook.hs+51−8

This diff is too big to show here.

src/Text/Pandoc/Writers/Docx.hs+177−357

This diff is too big to show here.

src/Text/Pandoc/Writers/Docx/StyleMap.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/Docx/Table.hsnew file+227

This diff is too big to show here.

src/Text/Pandoc/Writers/Docx/Types.hsnew file+185

This diff is too big to show here.

src/Text/Pandoc/Writers/DokuWiki.hs+4−2

This diff is too big to show here.

src/Text/Pandoc/Writers/EPUB.hs+261−211

This diff is too big to show here.

src/Text/Pandoc/Writers/FB2.hs+61−77

This diff is too big to show here.

src/Text/Pandoc/Writers/GridTable.hsnew file+157

This diff is too big to show here.

src/Text/Pandoc/Writers/HTML.hs+115−38

This diff is too big to show here.

src/Text/Pandoc/Writers/Haddock.hs+13−2

This diff is too big to show here.

src/Text/Pandoc/Writers/Ipynb.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/JATS.hs+99−56

This diff is too big to show here.

src/Text/Pandoc/Writers/JATS/References.hsnew file+164

This diff is too big to show here.

src/Text/Pandoc/Writers/JATS/Table.hs+26−6

This diff is too big to show here.

src/Text/Pandoc/Writers/JATS/Types.hs+15−4

This diff is too big to show here.

src/Text/Pandoc/Writers/Jira.hs+42−17

This diff is too big to show here.

src/Text/Pandoc/Writers/LaTeX.hs+54−566

This diff is too big to show here.

src/Text/Pandoc/Writers/LaTeX/Caption.hs+1−2

This diff is too big to show here.

src/Text/Pandoc/Writers/LaTeX/Citation.hsnew file+181

This diff is too big to show here.

src/Text/Pandoc/Writers/LaTeX/Lang.hsnew file+192

This diff is too big to show here.

src/Text/Pandoc/Writers/LaTeX/Notes.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/LaTeX/Table.hs+23−10

This diff is too big to show here.

src/Text/Pandoc/Writers/LaTeX/Types.hs+2−1

This diff is too big to show here.

src/Text/Pandoc/Writers/LaTeX/Util.hsnew file+275

This diff is too big to show here.

src/Text/Pandoc/Writers/Man.hs+3−3

This diff is too big to show here.

src/Text/Pandoc/Writers/Markdown.hs+83−662

This diff is too big to show here.

src/Text/Pandoc/Writers/Markdown/Inline.hsnew file+616

This diff is too big to show here.

src/Text/Pandoc/Writers/Markdown/Types.hsnew file+81

This diff is too big to show here.

src/Text/Pandoc/Writers/MediaWiki.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/Ms.hs+28−10

This diff is too big to show here.

src/Text/Pandoc/Writers/Muse.hs+6−3

This diff is too big to show here.

src/Text/Pandoc/Writers/Native.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/ODT.hs+28−22

This diff is too big to show here.

src/Text/Pandoc/Writers/OOXML.hs+22−22

This diff is too big to show here.

src/Text/Pandoc/Writers/OPML.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/OpenDocument.hs+50−54

This diff is too big to show here.

src/Text/Pandoc/Writers/Org.hs+39−19

This diff is too big to show here.

src/Text/Pandoc/Writers/Powerpoint.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/Powerpoint/Output.hs+148−129

This diff is too big to show here.

src/Text/Pandoc/Writers/Powerpoint/Presentation.hs+2−3

This diff is too big to show here.

src/Text/Pandoc/Writers/RST.hs+17−13

This diff is too big to show here.

src/Text/Pandoc/Writers/RTF.hs+17−11

This diff is too big to show here.

src/Text/Pandoc/Writers/Roff.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/Shared.hs+19−4

This diff is too big to show here.

src/Text/Pandoc/Writers/TEI.hs+5−4

This diff is too big to show here.

src/Text/Pandoc/Writers/Texinfo.hs+23−16

This diff is too big to show here.

src/Text/Pandoc/Writers/Textile.hs+1−1

This diff is too big to show here.

src/Text/Pandoc/Writers/ZimWiki.hs+5−3

This diff is too big to show here.

src/Text/Pandoc/XML.hs+42−20

This diff is too big to show here.

src/Text/Pandoc/XML/Light.hsnew file+89

This diff is too big to show here.

src/Text/Pandoc/XML/Light/Output.hsnew file+234

This diff is too big to show here.

src/Text/Pandoc/XML/Light/Proc.hsnew file+139

This diff is too big to show here.

src/Text/Pandoc/XML/Light/Types.hsnew file+193

This diff is too big to show here.

stack.yaml+11−18

This diff is too big to show here.

test/Tests/Command.hs+34−39

This diff is too big to show here.

test/Tests/Helpers.hs+25−36

This diff is too big to show here.

test/Tests/Lua.hs+1−3

This diff is too big to show here.

test/Tests/Lua/Module.hs+3−1

This diff is too big to show here.

test/Tests/Old.hs+12−26

This diff is too big to show here.

test/Tests/Readers/Creole.hs+1−3

This diff is too big to show here.

test/Tests/Readers/Docx.hs+25−7

This diff is too big to show here.

test/Tests/Readers/DokuWiki.hs−2

This diff is too big to show here.

test/Tests/Readers/EPUB.hs+2−4

This diff is too big to show here.

test/Tests/Readers/FB2.hs−2

This diff is too big to show here.

test/Tests/Readers/HTML.hs+21−8

This diff is too big to show here.

test/Tests/Readers/JATS.hs+1−2

This diff is too big to show here.

test/Tests/Readers/Jira.hs+22−2

This diff is too big to show here.

test/Tests/Readers/LaTeX.hs+2−21

This diff is too big to show here.

test/Tests/Readers/Man.hs+1−2

This diff is too big to show here.

test/Tests/Readers/Markdown.hs+4−6

This diff is too big to show here.

test/Tests/Readers/Muse.hs−2

This diff is too big to show here.

test/Tests/Readers/Odt.hs+2−3

This diff is too big to show here.

test/Tests/Readers/Org.hs+1−1

This diff is too big to show here.

test/Tests/Readers/Org/Block.hs+1−3

This diff is too big to show here.

test/Tests/Readers/Org/Block/CodeBlock.hs+4−6

This diff is too big to show here.

test/Tests/Readers/Org/Block/Figure.hs+1−3

This diff is too big to show here.

test/Tests/Readers/Org/Block/Header.hs+1−3

This diff is too big to show here.

test/Tests/Readers/Org/Block/List.hs+14−3

This diff is too big to show here.

test/Tests/Readers/Org/Block/Table.hs+1−3

This diff is too big to show here.

test/Tests/Readers/Org/Directive.hs+1−3

This diff is too big to show here.

test/Tests/Readers/Org/Inline.hs+3−5

This diff is too big to show here.

test/Tests/Readers/Org/Inline/Citation.hs+41−3

This diff is too big to show here.

test/Tests/Readers/Org/Inline/Note.hs+1−3

This diff is too big to show here.

test/Tests/Readers/Org/Inline/Smart.hs+1−3

This diff is too big to show here.

test/Tests/Readers/Org/Meta.hs+11−3

This diff is too big to show here.

test/Tests/Readers/Org/Shared.hs+1−3

This diff is too big to show here.

test/Tests/Readers/RST.hs+1−3

This diff is too big to show here.

test/Tests/Readers/Txt2Tags.hs+1−3

This diff is too big to show here.

test/Tests/Shared.hs+1−3

This diff is too big to show here.

test/Tests/Writers/AnnotatedTable.hs−1

This diff is too big to show here.

test/Tests/Writers/AsciiDoc.hs−2

This diff is too big to show here.

test/Tests/Writers/ConTeXt.hs+122−115

This diff is too big to show here.

test/Tests/Writers/Docbook.hs+43−8

This diff is too big to show here.

test/Tests/Writers/Docx.hs+5−2

This diff is too big to show here.

test/Tests/Writers/FB2.hs−2

This diff is too big to show here.

test/Tests/Writers/HTML.hs+57−54

This diff is too big to show here.

test/Tests/Writers/JATS.hs+135−93

This diff is too big to show here.

test/Tests/Writers/Jira.hs+60−1

This diff is too big to show here.

test/Tests/Writers/LaTeX.hs−2

This diff is too big to show here.

test/Tests/Writers/Markdown.hs−2

This diff is too big to show here.

test/Tests/Writers/Ms.hs−2

This diff is too big to show here.

test/Tests/Writers/Muse.hs−1

This diff is too big to show here.

test/Tests/Writers/Native.hs−2

This diff is too big to show here.

test/Tests/Writers/OOXML.hs+5−4

This diff is too big to show here.

test/Tests/Writers/Org.hs+46−13

This diff is too big to show here.

test/Tests/Writers/Plain.hs−2

This diff is too big to show here.

test/Tests/Writers/Powerpoint.hs−2

This diff is too big to show here.

test/Tests/Writers/RST.hs−2

This diff is too big to show here.

test/Tests/Writers/TEI.hs−2

This diff is too big to show here.

test/command/1279.md+1−1

This diff is too big to show here.

test/command/2103.mdnew file+8

This diff is too big to show here.

test/command/2378.md+3−3

This diff is too big to show here.

test/command/2874.md+2−2

This diff is too big to show here.

test/command/3516.md+1−5

This diff is too big to show here.

test/command/3752.mdnew file+60

This diff is too big to show here.

test/command/3792.md+1−1

This diff is too big to show here.

test/command/4063.mdnew file+29

This diff is too big to show here.

test/command/4164.md+1−1

This diff is too big to show here.

test/command/4442.md+1−1

This diff is too big to show here.

test/command/4465.mdnew file+15

This diff is too big to show here.

test/command/5304.md+1−1

This diff is too big to show here.

test/command/5321.md+2−2

This diff is too big to show here.

test/command/5367.md+7−3

This diff is too big to show here.

test/command/5519.md+1−1

This diff is too big to show here.

test/command/5686.md+1−1

This diff is too big to show here.

test/command/5708.md+1−5

This diff is too big to show here.

test/command/6026.mdnew file+19

This diff is too big to show here.

test/command/6541.mdnew file+27

This diff is too big to show here.

test/command/6620.md+3

This diff is too big to show here.

test/command/6658.mdnew file+75

This diff is too big to show here.

test/command/6709.mdnew file+11

This diff is too big to show here.

test/command/6890.md+2−2

This diff is too big to show here.

test/command/6925.md+2−2

This diff is too big to show here.

test/command/6958.mdnew file+10

This diff is too big to show here.

test/command/6993.mdnew file+21

This diff is too big to show here.

test/command/7003.mdnew file+37

This diff is too big to show here.

test/command/7006.mdnew file+20

This diff is too big to show here.

test/command/7009.mdnew file+8

This diff is too big to show here.

test/command/7016.mdnew file+48

This diff is too big to show here.

test/command/7041.mdnew file+23

This diff is too big to show here.

test/command/7042.mdnew file+146

This diff is too big to show here.

test/command/7058.mdnew file+6

This diff is too big to show here.

test/command/7064.mdnew file+32

This diff is too big to show here.

test/command/7067.mdnew file+90

This diff is too big to show here.

test/command/7080.mdnew file+8

This diff is too big to show here.

test/command/7092.mdnew file+8

This diff is too big to show here.

test/command/7099.mdnew file+18

This diff is too big to show here.

test/command/7112.mdnew file+15

This diff is too big to show here.

test/command/7129.mdnew file+27

This diff is too big to show here.

test/command/7132.mdnew file+10

This diff is too big to show here.

test/command/7134.mdnew file+16

This diff is too big to show here.

test/command/7145.mdnew file+12

This diff is too big to show here.

test/command/7155.mdnew file+15

This diff is too big to show here.

test/command/7173.mdnew file+137

This diff is too big to show here.

test/command/7208.mdnew file+6

This diff is too big to show here.

test/command/7214.mdnew file+28

This diff is too big to show here.

test/command/7216.mdnew file+19

This diff is too big to show here.

test/command/7266.mdnew file+10

This diff is too big to show here.

test/command/7272.mdnew file+27

This diff is too big to show here.

test/command/7278.mdnew file+38

This diff is too big to show here.

test/command/7282.mdnew file+7

This diff is too big to show here.

test/command/7288.mdnew file+40

This diff is too big to show here.

test/command/7299.mdnew file+23

This diff is too big to show here.

test/command/7321.mdnew file+24

This diff is too big to show here.

test/command/7323.mdnew file+29

This diff is too big to show here.

test/command/7324.mdnew file+25

This diff is too big to show here.

test/command/7339.mdnew file+11

This diff is too big to show here.

test/command/7340.mdnew file+6

This diff is too big to show here.

test/command/7376.mdnew file+16

This diff is too big to show here.

test/command/7394.mdnew file+85

This diff is too big to show here.

test/command/7397.mdnew file+14

This diff is too big to show here.

test/command/7400.mdnew file+9

This diff is too big to show here.

test/command/7416.mdnew file+19

This diff is too big to show here.

test/command/averroes.bibnew file+95

This diff is too big to show here.

test/command/biblatex-baez-article.md+1−1

This diff is too big to show here.

test/command/biblatex-baez-online.md+1−1

This diff is too big to show here.

test/command/biblatex-basic.md+5−3

This diff is too big to show here.

test/command/biblatex-cotton.md+3−3

This diff is too big to show here.

test/command/biblatex-examples.bibdeleted−1674

This diff is too big to show here.

test/command/biblatex-inproceedings.md+1−1

This diff is too big to show here.

test/command/biblatex-itzhaki.md+1−1

This diff is too big to show here.

test/command/biblatex-jaffe.md+2−2

This diff is too big to show here.

test/command/biblatex-moraux.md+1−1

This diff is too big to show here.

test/command/biblatex-murray.md+3−2

This diff is too big to show here.

test/command/biblatex-nussbaum.mddeleted−63

This diff is too big to show here.

test/command/biblatex-quotes.md+1−1

This diff is too big to show here.

test/command/biblatex-sigfridsson.md+1−1

This diff is too big to show here.

test/command/biblatex-spiegelberg.md+2−2

This diff is too big to show here.

test/command/biblatex-test-case-conversion.md+2−2

This diff is too big to show here.

test/command/biblatex-textnormal.md+1−1

This diff is too big to show here.

test/command/biblatex-wassenberg.md+1−1

This diff is too big to show here.

test/command/biblatex-wilde.md+1−1

This diff is too big to show here.

test/command/chap1/spider.pngnew file

Binary file not shown.

test/command/chap1/text.mdnew file+17

This diff is too big to show here.

test/command/chap2/spider.pngnew file

Binary file not shown.

test/command/chap2/text.mdnew file+5

This diff is too big to show here.

test/command/defaults-inheritance-1.mdnew file+6

This diff is too big to show here.

test/command/defaults-inheritance-2.mdnew file+5

This diff is too big to show here.

test/command/defaults-inheritance-3.mdnew file+6

This diff is too big to show here.

test/command/defaults3.yamlnew file+4

This diff is too big to show here.

test/command/defaults4.yamlnew file+3

This diff is too big to show here.

test/command/defaults5.yamlnew file+2

This diff is too big to show here.

test/command/defaults6.yamlnew file+2

This diff is too big to show here.

test/command/defaults7.yamlnew file+2

This diff is too big to show here.

test/command/defaults8.yamlnew file+2

This diff is too big to show here.

test/command/defaults9.yamlnew file+1

This diff is too big to show here.

test/command/duplicate_attributes.mdnew file+7

This diff is too big to show here.

test/command/mmd-metadata.mdnew file+20

This diff is too big to show here.

test/command/newif.mdnew file+55

This diff is too big to show here.

test/command/pandoc-citeproc-119.md+1−1

This diff is too big to show here.

test/command/pandoc-citeproc-320a.md+3−3

This diff is too big to show here.

test/command/pandoc-citeproc-322.md+1−1

This diff is too big to show here.

test/command/pandoc-citeproc-356.md+2−7

This diff is too big to show here.

test/command/pandoc-citeproc-65.md+1−1

This diff is too big to show here.

test/command/pandoc-citeproc-70.md+2−2

This diff is too big to show here.

test/command/pandoc-citeproc-chicago-author-date.md+7−7

This diff is too big to show here.

test/command/pandoc-citeproc-chicago-fullnote-bibliography.md+15−18

This diff is too big to show here.

test/command/svg.md+15−17

This diff is too big to show here.

test/command/toc.md+2−2

This diff is too big to show here.

test/command/unicode-collation.mdnew file+130

This diff is too big to show here.

test/docbook-reader.docbook+9−2

This diff is too big to show here.

test/docx/0_level_headers.native+2−2

This diff is too big to show here.

test/docx/golden/block_quotes.docx

Binary file not shown.

test/docx/golden/codeblock.docx

Binary file not shown.

test/docx/golden/comments.docx

Binary file not shown.

test/docx/golden/custom_style_no_reference.docx

Binary file not shown.

test/docx/golden/custom_style_preserve.docx

Binary file not shown.

test/docx/golden/custom_style_reference.docx

Binary file not shown.

test/docx/golden/definition_list.docx

Binary file not shown.

test/docx/golden/document-properties-short-desc.docx

Binary file not shown.

test/docx/golden/document-properties.docx

Binary file not shown.

test/docx/golden/headers.docx

Binary file not shown.

test/docx/golden/image.docx

Binary file not shown.

test/docx/golden/inline_code.docx

Binary file not shown.

test/docx/golden/inline_formatting.docx

Binary file not shown.

test/docx/golden/inline_images.docx

Binary file not shown.

test/docx/golden/link_in_notes.docx

Binary file not shown.

test/docx/golden/links.docx

Binary file not shown.

test/docx/golden/lists.docx

Binary file not shown.

test/docx/golden/lists_continuing.docx

Binary file not shown.

test/docx/golden/lists_multiple_initial.docx

Binary file not shown.

test/docx/golden/lists_restarting.docx

Binary file not shown.

test/docx/golden/nested_anchors_in_header.docx

Binary file not shown.

test/docx/golden/notes.docx

Binary file not shown.

test/docx/golden/raw-blocks.docx

Binary file not shown.

test/docx/golden/raw-bookmarks.docx

Binary file not shown.

test/docx/golden/table_one_row.docx

Binary file not shown.

test/docx/golden/table_with_list_cell.docx

Binary file not shown.

test/docx/golden/tables-default-widths.docxnew file

Binary file not shown.

test/docx/golden/tables.docx

Binary file not shown.

test/docx/golden/track_changes_deletion.docx

Binary file not shown.

test/docx/golden/track_changes_insertion.docx

Binary file not shown.

test/docx/golden/track_changes_move.docx

Binary file not shown.

test/docx/golden/track_changes_scrubbed_metadata.docx

Binary file not shown.

test/docx/golden/unicode.docx

Binary file not shown.

test/docx/golden/verbatim_subsuper.docx

Binary file not shown.

test/docx/image_vml_as_object.docxnew file

Binary file not shown.

test/docx/image_vml_as_object.nativenew file+2

This diff is too big to show here.

test/docx/sdt_elements.native+9−10

This diff is too big to show here.

test/docx/table_captions_no_field.docxnew file

Binary file not shown.

test/docx/table_captions_no_field.nativenew file+34

This diff is too big to show here.

test/docx/table_captions_with_field.docxnew file

Binary file not shown.

test/docx/table_captions_with_field.nativenew file+54

This diff is too big to show here.

test/docx/table_header_rowspan.docxnew file

Binary file not shown.

test/docx/table_header_rowspan.nativenew file+189

This diff is too big to show here.

test/docx/table_one_header_row.docxnew file

Binary file not shown.

test/docx/table_one_header_row.nativenew file+18

This diff is too big to show here.

test/docx/table_one_row.docx

Binary file not shown.

test/docx/table_one_row.native+4−4

This diff is too big to show here.

test/docx/table_variable_width.native+12−17

This diff is too big to show here.

test/docx/table_with_list_cell.native+3−3

This diff is too big to show here.

test/docx/tables-default-widths.nativenew file+92

This diff is too big to show here.

test/docx/tables.native+9−9

This diff is too big to show here.

test/epub/features.native+2−2

This diff is too big to show here.

test/epub/wasteland.epub

Binary file not shown.

test/fb2/images.fb2+1−1

This diff is too big to show here.

test/fb2/test.jpg

Binary file not shown.

test/html-reader.native+2−1

This diff is too big to show here.

test/jats-reader.native+1−1

This diff is too big to show here.

test/jats-reader.xml+1

This diff is too big to show here.

test/jira-reader.native+1−1

This diff is too big to show here.

test/lhs-test.html+2−1

This diff is too big to show here.

test/lhs-test.html+lhs+2−1

This diff is too big to show here.

test/lhs-test.latex+3−3

This diff is too big to show here.

test/lhs-test.latex+lhs+3−3

This diff is too big to show here.

test/lua/module/pandoc-path.luanew file+44

This diff is too big to show here.

test/markdown-reader-more.native+5−33

This diff is too big to show here.

test/odt/native/tab.nativenew file+1

This diff is too big to show here.

test/odt/odt/tab.odtnew file

Binary file not shown.

test/pipe-tables.native+2−10

This diff is too big to show here.

test/pptx/code-custom.pptx

Binary file not shown.

test/pptx/code-custom_templated.pptx

Binary file not shown.

test/pptx/code.pptx

Binary file not shown.

test/pptx/code_templated.pptx

Binary file not shown.

test/pptx/document-properties-short-desc.pptx

Binary file not shown.

test/pptx/document-properties-short-desc_templated.pptx

Binary file not shown.

test/pptx/document-properties.pptx

Binary file not shown.

test/pptx/document-properties_templated.pptx

Binary file not shown.

test/pptx/endnotes.pptx

Binary file not shown.

test/pptx/endnotes_templated.pptx

Binary file not shown.

test/pptx/endnotes_toc.pptx

Binary file not shown.

test/pptx/endnotes_toc_templated.pptx

Binary file not shown.

test/pptx/images.pptx

Binary file not shown.

test/pptx/images_templated.pptx

Binary file not shown.

test/pptx/inline_formatting.pptx

Binary file not shown.

test/pptx/inline_formatting_templated.pptx

Binary file not shown.

test/pptx/lists.pptx

Binary file not shown.

test/pptx/lists_templated.pptx

Binary file not shown.

test/pptx/raw_ooxml.pptx

Binary file not shown.

test/pptx/raw_ooxml_templated.pptx

Binary file not shown.

test/pptx/reference_depth.pptx

Binary file not shown.

test/pptx/remove_empty_slides.pptx

Binary file not shown.

test/pptx/remove_empty_slides_templated.pptx

Binary file not shown.

test/pptx/slide_breaks.pptx

Binary file not shown.

test/pptx/slide_breaks_slide_level_1.pptx

Binary file not shown.

test/pptx/slide_breaks_slide_level_1_templated.pptx

Binary file not shown.

test/pptx/slide_breaks_templated.pptx

Binary file not shown.

test/pptx/slide_breaks_toc.pptx

Binary file not shown.

test/pptx/slide_breaks_toc_templated.pptx

Binary file not shown.

test/pptx/speaker_notes.pptx

Binary file not shown.

test/pptx/speaker_notes_after_metadata.pptx

Binary file not shown.

test/pptx/speaker_notes_after_metadata_templated.pptx

Binary file not shown.

test/pptx/speaker_notes_afterheader.pptx

Binary file not shown.

test/pptx/speaker_notes_afterheader_templated.pptx

Binary file not shown.

test/pptx/speaker_notes_afterseps.pptx

Binary file not shown.

test/pptx/speaker_notes_afterseps_templated.pptx

Binary file not shown.

test/pptx/speaker_notes_templated.pptx

Binary file not shown.

test/pptx/start_numbering_at.pptx

Binary file not shown.

test/pptx/start_numbering_at_templated.pptx

Binary file not shown.

test/pptx/tables.pptx

Binary file not shown.

test/pptx/tables_templated.pptx

Binary file not shown.

test/pptx/two_column.pptx

Binary file not shown.

test/pptx/two_column_templated.pptx

Binary file not shown.

test/rst-reader.native+3−21

This diff is too big to show here.

test/tables-rstsubset.native+1−9

This diff is too big to show here.

test/tables.latex+50−35

This diff is too big to show here.

test/tables.ms+44

This diff is too big to show here.

test/tables/nordics.latex+28−12

This diff is too big to show here.

test/tables/planets.latex+15−21

This diff is too big to show here.

test/tables/students.latex+19−11

This diff is too big to show here.

test/test-pandoc.hs+18−8

This diff is too big to show here.

test/txt2tags.native+1−1

This diff is too big to show here.

test/txt2tags.t2t+1−1

This diff is too big to show here.

test/writer.context+42−40

This diff is too big to show here.

test/writer.html4+2−1

This diff is too big to show here.

test/writer.html5+2−1

This diff is too big to show here.

test/writer.jats_archiving+10−8

This diff is too big to show here.

test/writer.jats_articleauthoring+1−1

This diff is too big to show here.

test/writer.jats_publishing+10−8

This diff is too big to show here.

test/writer.jira+23−35

This diff is too big to show here.

test/writer.latex+3−3

This diff is too big to show here.

test/writer.ms+1−1

This diff is too big to show here.

test/writer.opml+1−1

This diff is too big to show here.

test/writer.org+7−7

This diff is too big to show here.

test/writer.zimwiki+3−3

This diff is too big to show here.

test/writers-lang-and-dir.context+4−2

This diff is too big to show here.

test/writers-lang-and-dir.latex+6−6

This diff is too big to show here.

tools/build-arm.shnew file+110

This diff is too big to show here.

tools/changelog-helper.sh+11−4

This diff is too big to show here.

tools/parseTimings.plnew file+33

This diff is too big to show here.

trypandoc/index.html+1−1

This diff is too big to show here.

trypandoc/trypandoc.hs+3−8

This diff is too big to show here.

windows/Makefile+1−1

This diff is too big to show here.

windows/pandoc.wxs+2−2

This diff is too big to show here.