From f88ded2ba8fc9c452717674542fcaef231b3f3e9 Mon Sep 17 00:00:00 2001 From: Roman Grundkiewicz Date: Tue, 23 Feb 2021 16:25:30 +0000 Subject: [PATCH] Add documentation platform based on Sphinx+Doxygen+Breathe+Exhale (#803) * Setup Sphinx+Doxygen+Breathe+Exhale * Remove links to modindex and search pages * Enable Doxygen autobrief --- CHANGELOG.md | 3 + doc/.gitignore | 4 ++ doc/Makefile | 23 ++++++++ doc/README.md | 51 ++++++++++++++++ doc/_static/css/custom.css | 4 ++ doc/conf.py | 118 +++++++++++++++++++++++++++++++++++++ doc/contributing.rst | 1 + doc/graph.md | 3 + doc/index.rst | 48 +++++++++++++++ doc/install.rst | 3 + doc/make.bat | 35 +++++++++++ doc/operators.md | 3 + doc/requirements.txt | 6 ++ 13 files changed, 302 insertions(+) create mode 100644 doc/.gitignore create mode 100644 doc/Makefile create mode 100644 doc/README.md create mode 100644 doc/_static/css/custom.css create mode 100644 doc/conf.py create mode 100644 doc/contributing.rst create mode 100644 doc/graph.md create mode 100644 doc/index.rst create mode 100644 doc/install.rst create mode 100644 doc/make.bat create mode 100644 doc/operators.md create mode 100644 doc/requirements.txt diff --git a/CHANGELOG.md b/CHANGELOG.md index ec988df7..a9b3997c 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,6 +8,9 @@ and this project adheres to [Semantic Versioning](http://semver.org/spec/v2.0.0. ## [Unreleased] +### Added +- Developer documentation framework based on Sphinx+Doxygen+Breathe+Exhale + ### Fixed - Fix building server with Boost 1.75 diff --git a/doc/.gitignore b/doc/.gitignore new file mode 100644 index 00000000..4d192b77 --- /dev/null +++ b/doc/.gitignore @@ -0,0 +1,4 @@ +api +build +doxygen +venv diff --git a/doc/Makefile b/doc/Makefile new file mode 100644 index 00000000..84310d9d --- /dev/null +++ b/doc/Makefile @@ -0,0 +1,23 @@ +# You can set these variables from the command line, and also +# from the environment for the first two. +SPHINXOPTS ?= +SPHINXBUILD ?= sphinx-build +SOURCEDIR = . +BUILDDIR = build + +# Put it first so that "make" without argument is like "make help". +help: + @$(SPHINXBUILD) -M help "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +.PHONY: clean help Makefile + +# Catch-all target: route all unknown targets to Sphinx using the new +# "make mode" option. $(O) is meant as a shortcut for $(SPHINXOPTS). +%: Makefile + @$(SPHINXBUILD) -M $@ "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) + +# Clean target as recommended by Exhale +# https://exhale.readthedocs.io/en/latest/usage.html#optional-create-a-proper-clean-target +clean: + rm -rf doxygen/ api/ + @$(SPHINXBUILD) -M clean "$(SOURCEDIR)" "$(BUILDDIR)" $(SPHINXOPTS) $(O) diff --git a/doc/README.md b/doc/README.md new file mode 100644 index 00000000..87d86ba1 --- /dev/null +++ b/doc/README.md @@ -0,0 +1,51 @@ +# Marian NMT code documentation and library API + +This directory contains code documentation and library API for developers of Marian NMT. + +The documentation is generated using +[Sphinx](https://www.sphinx-doc.org/en/master/usage/quickstart.html) + +[Breathe](https://breathe.readthedocs.io/en/latest/directives.html) + +[Doxygen](http://www.doxygen.nl/manual/docblocks.html) + +[Exhale](https://exhale.readthedocs.io/en/latest/usage.html). +The documentation source code is written in `.rst` or `.md` files with special directives that allow +to reference to C++ source code and documentation. The source documents are then build into static +HTML pages. + + +## Installation + +On Ubuntu 20.04, install the following packages: + + sudo apt-get install python3 python3-pip python3-setuptools doxygen + +Then set up a Python environment and install modules: + + pip3 install virtualenv + virtualenv venv -p python3 + source venv/bin/activate + pip install -r requirements.txt + +Documentation building should also work on Windows, but it has not been tested. + + +## Generation + +The documentation can be generated by running: + + make html + +The website will be generated into `build/html` and accessible by opening _index.html_ in your +browser. + +Directories: + +- `build` - automatically output directory for HTML documentation +- `doxygen` - automatically generated Doxygen XML files +- `api` - automatic library API generated with Exhale +- `.rst` and `.md` files in this directory and its subdirectories are documentation source files +- `_static` - custom CSS and JavaScript files + + +## Writing documentation + +To be documented... diff --git a/doc/_static/css/custom.css b/doc/_static/css/custom.css new file mode 100644 index 00000000..8352655e --- /dev/null +++ b/doc/_static/css/custom.css @@ -0,0 +1,4 @@ +.wy-body-for-nav > .wy-grid-for-nav > .wy-nav-side { + border-bottom: 5px solid #28bbee; + /*background-color: #494d55;*/ +} diff --git a/doc/conf.py b/doc/conf.py new file mode 100644 index 00000000..f194f374 --- /dev/null +++ b/doc/conf.py @@ -0,0 +1,118 @@ +# Configuration file for the Sphinx documentation builder. +# +# This file only contains a selection of the most common options. For a full +# list see the documentation: +# https://www.sphinx-doc.org/en/master/usage/configuration.html + +# -- Path setup -------------------------------------------------------------- + +# If extensions (or modules to document with autodoc) are in another directory, +# add these directories to sys.path here. If the directory is relative to the +# documentation root, use os.path.abspath to make it absolute, like shown here. +# +import os +import datetime +import sys + +sys.path.insert(0, os.path.abspath('.')) + + +# -- Project information ----------------------------------------------------- + +project = 'Marian NMT' +copyright = '2021, Marian NMT Team' +author = 'Marian NMT Team' + +# The full version, including alpha/beta/rc tags +# TODO: add GitHub commit hash to the version +version_file = os.path.join(os.path.dirname(os.path.dirname(__file__)), 'VERSION') +with open(os.path.abspath(version_file)) as f: + version = f.read().strip() +release = version + ' ' + str(datetime.date.today()) + + +# -- General configuration --------------------------------------------------- + +# Add any Sphinx extension module names here, as strings. They can be +# extensions coming with Sphinx (named 'sphinx.ext.*') or your custom +# ones. +extensions = [ + 'sphinx.ext.imgmath', + 'sphinx.ext.todo', + 'breathe', + 'exhale', + 'recommonmark', +] + +# Add any paths that contain templates here, relative to this directory. +templates_path = ['_templates'] + +# List of patterns, relative to source directory, that match files and +# directories to ignore when looking for source files. +# This pattern also affects html_static_path and html_extra_path. +exclude_patterns = [ + 'build', + 'doxygen', + 'venv', + 'README.md', +] + + +# -- Options for HTML output ------------------------------------------------- + +# The theme to use for HTML and HTML Help pages. See the documentation for +# a list of builtin themes. +# +html_theme = 'sphinx_rtd_theme' +htmlhelp_basename = 'marian' + +# Add any paths that contain custom static files (such as style sheets) here, +# relative to this directory. They are copied after the builtin static files, +# so a file named "default.css" will overwrite the builtin "default.css". +html_static_path = ['_static'] +html_css_files = ['css/custom.css'] + +# The base URL which points to the root of the HTML documentation +html_baseurl = 'http://marian-nmt.github.io/docs/api' + + +# -- Extension configuration ------------------------------------------------- + +breathe_projects = { 'marian': './doxygen/xml' } +breathe_default_project = 'marian' + +doxygen_config = """ +INPUT = ../src +EXCLUDE += ../src/3rd_party +EXCLUDE += ../src/tests +EXCLUDE_PATTERNS = *.inc *.md *.txt +ENABLE_PREPROCESSING = YES +JAVADOC_AUTOBRIEF = YES +WARN_IF_UNDOCUMENTED = NO +""" + +exhale_args = { + 'containmentFolder' : './api', + 'rootFileName' : 'library_index.rst', + 'rootFileTitle' : 'Library API', + 'doxygenStripFromPath' : '..', + 'createTreeView' : True, + 'exhaleExecutesDoxygen' : True, + 'exhaleDoxygenStdin' : doxygen_config.strip(), +} + +primary_domain = 'cpp' +highlight_language = 'cpp' + +# A trick to include markdown files from outside the source directory using +# 'mdinclude'. Warning: all other markdown files not included via 'mdinclude' +# will be rendered using recommonmark as recommended by Sphinx +from m2r import MdInclude + +def setup(app): + # from m2r to make `mdinclude` work + app.add_config_value('no_underscore_emphasis', False, 'env') + app.add_config_value('m2r_parse_relative_links', False, 'env') + app.add_config_value('m2r_anonymous_references', False, 'env') + app.add_config_value('m2r_disable_inline_math', False, 'env') + app.add_directive('mdinclude', MdInclude) diff --git a/doc/contributing.rst b/doc/contributing.rst new file mode 100644 index 00000000..4fc50161 --- /dev/null +++ b/doc/contributing.rst @@ -0,0 +1 @@ +.. mdinclude:: ../CONTRIBUTING.md diff --git a/doc/graph.md b/doc/graph.md new file mode 100644 index 00000000..51dec9db --- /dev/null +++ b/doc/graph.md @@ -0,0 +1,3 @@ +# Expression graphs + +... diff --git a/doc/index.rst b/doc/index.rst new file mode 100644 index 00000000..241135b9 --- /dev/null +++ b/doc/index.rst @@ -0,0 +1,48 @@ +Welcome to Marian's documentation! +================================== + +|buildgpu| |buildcpu| |tests| |release| |license| + +Marian is an efficient and self-contained Neural Machine Translation framework with an integrated +automatic differentiation engine based on dynamic computation graphs, written entirely in C++. + +This is developer documentation. User documentation is available at https://marian-nmt.github.io/docs/ + +.. toctree:: + :maxdepth: 2 + :caption: Contents: + + install + graph + operators + + api/library_index + + contributing + + +Indices and tables +------------------ + +* :ref:`genindex` + + +.. |buildgpu| image:: https://img.shields.io/jenkins/s/http/vali.inf.ed.ac.uk/jenkins/view/marian/job/marian-dev-cuda-10.1.svg?label=CUDAC%20Build + :target: http://vali.inf.ed.ac.uk/jenkins/job/marian-dev/ + :alt: GPU build status + +.. |buildcpu| image:: https://img.shields.io/jenkins/s/http/vali.inf.ed.ac.uk/jenkins/view/marian/job/marian-dev-cpu.svg?label=CPU%20Build + :target: http://vali.inf.ed.ac.uk/jenkins/job/marian-dev-cpu/ + :alt: CPU build status + +.. |tests| image:: https://img.shields.io/jenkins/s/http/vali.inf.ed.ac.uk/jenkins/view/marian/job/marian-regression-tests.svg?label=Tests + :target: http://vali.inf.ed.ac.uk/jenkins/job/marian-regression-tests/ + :alt: Tests status + +.. |release| image:: https://img.shields.io/github/release/marian-nmt/marian.svg?label=Release + :target: https://github.com/marian-nmt/marian/releases + :alt: Latest release + +.. |license| image:: https://img.shields.io/badge/License-MIT-blue.svg + :target: ../LICENSE.md + :alt: License: MIT diff --git a/doc/install.rst b/doc/install.rst new file mode 100644 index 00000000..cd7a81df --- /dev/null +++ b/doc/install.rst @@ -0,0 +1,3 @@ +Installation +============ + diff --git a/doc/make.bat b/doc/make.bat new file mode 100644 index 00000000..6247f7e2 --- /dev/null +++ b/doc/make.bat @@ -0,0 +1,35 @@ +@ECHO OFF + +pushd %~dp0 + +REM Command file for Sphinx documentation + +if "%SPHINXBUILD%" == "" ( + set SPHINXBUILD=sphinx-build +) +set SOURCEDIR=source +set BUILDDIR=build + +if "%1" == "" goto help + +%SPHINXBUILD% >NUL 2>NUL +if errorlevel 9009 ( + echo. + echo.The 'sphinx-build' command was not found. Make sure you have Sphinx + echo.installed, then set the SPHINXBUILD environment variable to point + echo.to the full path of the 'sphinx-build' executable. Alternatively you + echo.may add the Sphinx directory to PATH. + echo. + echo.If you don't have Sphinx installed, grab it from + echo.http://sphinx-doc.org/ + exit /b 1 +) + +%SPHINXBUILD% -M %1 %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% +goto end + +:help +%SPHINXBUILD% -M help %SOURCEDIR% %BUILDDIR% %SPHINXOPTS% %O% + +:end +popd diff --git a/doc/operators.md b/doc/operators.md new file mode 100644 index 00000000..70d880e2 --- /dev/null +++ b/doc/operators.md @@ -0,0 +1,3 @@ +# Graph operations + +... diff --git a/doc/requirements.txt b/doc/requirements.txt new file mode 100644 index 00000000..8d56e683 --- /dev/null +++ b/doc/requirements.txt @@ -0,0 +1,6 @@ +sphinx==2.4.4 +breathe==4.13.0 +exhale +sphinx_rtd_theme +recommonmark +m2r