on master help sign in
doc: describe Makefile targets

With the aim to simplify development cycles, started with PR #1756 a Makefile based boilerplate was added. This patch adds the missing developer documentation. Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

commitf09459b98ae877d7dfd40d85c0edf058c65913d5 authorMarkus Heiser <markus.heiser@darmarit.de> dateWed 18 Dec 2019, 16:11 (7 years ago) parentda56bda
6 files changed+257−7
Makefile+4 docs/conf.py+1 docs/dev/contribution_guide.rst+10−5 docs/dev/index.rst+1 docs/dev/makefile.rst+217 docs/dev/quickstart.rst+24−2
Makefile+4
@@ -8,6 +8,9 @@ PYOBJECTS = searx
88DOC = docs
99PY_SETUP_EXTRAS ?= \[test\]
1010
11+PYDIST=./dist/py
12+PYBUILD=./build/py
13+
1114include utils/makefile.include
1215include utils/makefile.python
1316include utils/makefile.sphinx
@@ -23,6 +26,7 @@ help:
2326 @echo ' install - developer install (./local)'
2427 @echo ' uninstall - uninstall (./local)'
2528 @echo ' gh-pages - build docs & deploy on gh-pages branch'
29+ @echo ' clean - drop builds and environments'
2630 @echo ''
2731 @$(MAKE) -s -f utils/makefile.include make-help
2832 @echo ''
docs/conf.py+1
@@ -32,6 +32,7 @@ extlinks['origin'] = (GIT_URL + '/blob/master/%s', 'git://')
3232extlinks['patch'] = (GIT_URL + '/commit/%s', '#')
3333extlinks['search'] = (SEARX_URL + '/%s', '#')
3434extlinks['docs'] = (DOCS_URL + '/%s', 'docs: ')
35+extlinks['pypi'] = ('https://pypi.org/project/%s', 'PyPi: ')
3536
3637extensions = [
3738 'sphinx.ext.extlinks',
docs/dev/contribution_guide.rst+10−5
@@ -79,6 +79,8 @@ Translation currently takes place on :ref:`transifex <translation>`.
7979 Please, do not update translation files in the repo.
8080
8181
82+.. _contrib docs:
83+
8284Documentation
8385=============
8486
@@ -91,7 +93,7 @@ Documentation
9193
9294The documentation is built using Sphinx_. So in order to be able to generate
9395the required files, you have to install it on your system. Much easier, use
94−Makefile our targets.
96+our :ref:`makefile`.
9597
9698Here is an example which makes a complete rebuild:
9799
@@ -101,6 +103,7 @@ Here is an example which makes a complete rebuild:
101103 ...
102104 The HTML pages are in dist/docs.
103105
106+.. _make docs-live:
104107
105108live build
106109----------
@@ -110,9 +113,10 @@ live build
110113 It is recommended to assert a complete rebuild before deploying (use
111114 ``docs-clean``).
112115
113−Live build is like WYSIWYG, If you want to edit the documentation, its
114−recommended to use. The Makefile target ``docs-live`` builds the docs, opens URL
115−in your favorite browser and rebuilds every time a reST file has been changed.
116+Live build is like WYSIWYG. If you want to edit the documentation, its
117+recommended to use. The Makefile target ``docs-live`` builds the docs, opens
118+URL in your favorite browser and rebuilds every time a reST file has been
119+changed.
116120
117121.. code:: sh
118122
@@ -123,12 +127,13 @@ in your favorite browser and rebuilds every time a reST file has been changed.
123127 ... Start watching changes
124128
125129
130+.. _deploy on github.io:
126131
127132deploy on github.io
128133-------------------
129134
130135To deploy documentation at :docs:`github.io <.>` use Makefile target
131−``gh-pages``, which will builds the documentation, clones searx into a sub
136+:ref:`make gh-pages`, which will builds the documentation, clones searx into a sub
132137folder ``gh-pages``, cleans it, copies the doc build into and runs all the
133138needed git add, commit and push:
134139
docs/dev/index.rst+1
@@ -11,3 +11,4 @@ Developer documentation
1111 search_api
1212 plugins
1313 translation
14+ makefile
docs/dev/makefile.rstnew file+217
@@ -0,0 +1,217 @@
1+.. _makefile:
2+
3+================
4+Makefile Targets
5+================
6+
7+.. sidebar:: build environment
8+
9+ Before looking deeper at the targets, first read about :ref:`makefile setup`
10+ and :ref:`make pyenv`.
11+
12+With the aim to simplify development cycles, started with :pull:`1756` a
13+``Makefile`` based boilerplate was added.
14+
15+The usage is simple, just type ``make {target-name}`` to *build* a target.
16+Calling the ``help`` target gives a first overview::
17+
18+ $ make help
19+ test - run developer tests
20+ docs - build documentation
21+ docs-live - autobuild HTML documentation while editing
22+ run - run developer instance
23+ install - developer install (./local)
24+ uninstall - uninstall (./local)
25+ gh-pages - build docs & deploy on gh-pages branch
26+ clean - drop builds and environments
27+ ...
28+
29+.. contents:: Contents
30+ :depth: 2
31+ :local:
32+ :backlinks: entry
33+
34+
35+.. _makefile setup:
36+
37+Setup
38+=====
39+
40+.. _git stash: https://git-scm.com/docs/git-stash
41+
42+The main setup is done in the :origin:`Makefile`::
43+
44+ export GIT_URL=https://github.com/asciimoo/searx
45+ export SEARX_URL=https://searx.me
46+ export DOCS_URL=https://asciimoo.github.io/searx
47+
48+.. sidebar:: fork & upstream
49+
50+ Commit changes in your (local) branch, fork or whatever, but do not push them
51+ upstream / `git stash`_ is your friend.
52+
53+:GIT_URL: Changes this, to point to your searx fork.
54+
55+:SEARX_URL: Changes this, to point to your searx instance.
56+
57+:DOCS_URL: If you host your own (branded) documentation, change this URL.
58+
59+.. _make pyenv:
60+
61+Python environment
62+==================
63+
64+.. sidebar:: activate environment
65+
66+ ``source ./local/py3/bin/activate``
67+
68+With Makefile we do no longer need to build up the virualenv manually (as
69+described in the :ref:`devquickstart` guide). Jump into your git working tree
70+and release a ``make pyenv``:
71+
72+.. code:: sh
73+
74+ $ cd ~/searx-clone
75+ $ make pyenv
76+ PYENV usage: source ./local/py3/bin/activate
77+ ...
78+
79+With target ``pyenv`` a development environment (aka virtualenv) was build up in
80+``./local/py3/``. To make a *developer install** of searx (:origin:`setup.py`)
81+into this environment make target ``install``
82+
83+.. code:: sh
84+
85+ $ make install
86+ PYENV usage: source ./local/py3/bin/activate
87+ PYENV using virtualenv from ./local/py3
88+ PYENV install .
89+
90+You have never to think about intermediate targets like ``pyenv`` or
91+``install``, the ``Makefile`` chains them as requisites. Just run your main
92+target.
93+
94+.. sidebar:: drop environment
95+
96+ To get rid of the existing environment before re-build use :ref:`clean target
97+ <make clean>` first.
98+
99+If you think, something goes wrong with your ./local environment or you change
100+the :origin:`setup.py` file (or the requirements listed in
101+:origin:`requirements-dev.txt` and :origin:`requirements.txt`), you have to call
102+:ref:`make clean`.
103+
104+
105+.. _make run:
106+
107+``make run``
108+============
109+
110+To get up a running a developer instance simply call ``make run``. This enables
111+*debug* option in :origin:`searx/settings.yml`, starts ``./searx/webapp.py``
112+instance, disables *debug* option and opens the site (xdg-open):
113+
114+.. code:: sh
115+
116+ $ make run
117+ PYENV usage: source ./local/py3/bin/activate
118+ PYENV install .
119+ ./local/py3/bin/python ./searx/webapp.py
120+ ...
121+ INFO:werkzeug: * Running on http://127.0.0.1:8888/ (Press CTRL+C to quit)
122+ ...
123+
124+.. _make clean:
125+
126+``make clean``
127+==============
128+
129+Drop all intermediate files, all builds, but keep sources untouched. Includes
130+target ``pyclean`` which drops ./local environment. Before calling ``make
131+clean`` stop all processes using :ref:`make pyenv`.
132+
133+.. code:: sh
134+
135+ $ make clean
136+ CLEAN pyclean
137+ CLEAN clean
138+
139+.. _make docs:
140+
141+``make docs docs-live docs-clean``
142+==================================
143+
144+We describe the usage of the ``doc*`` targets in the :ref:`How to contribute /
145+Documentation <contrib docs>` section. If you want to edit the documentation
146+read our :ref:`make docs-live` section. If you are working in your own brand,
147+adjust your :ref:`Makefile setup <makefile setup>`.
148+
149+
150+.. _make gh-pages:
151+
152+``make gh-pages``
153+=================
154+
155+To deploy on github.io first adjust your :ref:`Makefile setup <makefile
156+setup>`. For any further read :ref:`deploy on github.io`.
157+
158+.. _make test:
159+
160+``make test``
161+=============
162+
163+Runs a series of tests: ``test.pep8``, ``test.unit``, ``test.robot`` and does
164+additional :ref:`pylint checks <make pylint>`. You can run tests selective,
165+e.g.:
166+
167+.. code:: sh
168+
169+ $ make test.pep8 test.unit
170+ . ./local/py3/bin/activate; ./manage.sh pep8_check
171+ [!] Running pep8 check
172+ . ./local/py3/bin/activate; ./manage.sh unit_tests
173+ [!] Running unit tests
174+
175+.. _make pylint:
176+
177+``make pylint``
178+===============
179+
180+.. _Pylint: https://www.pylint.org/
181+
182+Before commiting its recommend to do some (more) linting. Pylint_ is known as
183+one of the best source-code, bug and quality checker for the Python programming
184+language. Pylint_ is not yet a quality gate within our searx project (like
185+:ref:`test.pep8 <make test>` it is), but Pylint_ can help to improve code
186+quality anyway. The pylint profile we use at searx project is found in
187+project's root folder :origin:`.pylintrc`.
188+
189+Code quality is a ongoing process. Don't try to fix all messages from Pylint,
190+run Pylint and check if your changed lines are bringing up new messages. If so,
191+fix it. By this, code quality gets incremental better and if there comes the
192+day, the linting is balanced out, we might decide to add Pylint as a quality
193+gate.
194+
195+
196+``make pybuild``
197+================
198+
199+.. _PyPi: https://pypi.org/
200+.. _twine: https://twine.readthedocs.io/en/latest/
201+
202+Build Python packages in ``./dist/py``.
203+
204+.. code:: sh
205+
206+ $ make pybuild
207+ ...
208+ BUILD pybuild
209+ running sdist
210+ running egg_info
211+ ...
212+ $ ls ./dist/py/
213+ searx-0.15.0-py3-none-any.whl searx-0.15.0.tar.gz
214+
215+To upload packages to PyPi_, there is also a ``upload-pypi`` target. It needs
216+twine_ to be installed. Since you are not the owner of :pypi:`searx` you will
217+never need the latter.
docs/dev/quickstart.rst+24−2
@@ -4,15 +4,23 @@
44Development Quickstart
55======================
66
7+.. sidebar:: :ref:`makefile`
8+
9+ For additional developer purpose there are :ref:`makefile`.
10+
711This quickstart guide gets your environment set up with searx. Furthermore, it
812gives a short introduction to the ``manage.sh`` script.
913
1014How to setup your development environment
1115=========================================
1216
17+.. sidebar:: :ref:`make pyenv <make pyenv>`
18+
19+ Alternatively use the :ref:`make pyenv`.
20+
1321First, clone the source code of searx to the desired folder. In this case the
1422source is cloned to ``~/myprojects/searx``. Then create and activate the
15−searx-ve virtualenv and install the required packages using manage.sh.
23+searx-ve virtualenv and install the required packages using ``manage.sh``.
1624
1725.. code:: sh
1826
@@ -27,6 +35,10 @@ searx-ve virtualenv and install the required packages using manage.sh.
2735How to run tests
2836================
2937
38+.. sidebar:: :ref:`make test.unit <make test>`
39+
40+ Alternatively use the ``test.pep8``, ``test.unit``, ``test.robot`` targets.
41+
3042Tests can be run using the ``manage.sh`` script. Following tests and checks are
3143available:
3244
@@ -41,7 +53,8 @@ For example unit tests are run with the command below:
4153
4254 ./manage.sh unit_tests
4355
44−For further test options, please consult the help of the ``manage.sh`` script.
56+For further test options, please consult the help of the ``manage.sh`` script or
57+read :ref:`make test`.
4558
4659
4760How to compile styles and javascript
@@ -97,6 +110,11 @@ After installing grunt, the files can be built using the following command:
97110Tips for debugging/development
98111==============================
99112
113+.. sidebar:: :ref:`make run`
114+
115+ Makefile target ``run`` already enables debug option for your developer
116+ session / see :ref:`make run`.
117+
100118Turn on debug logging
101119 Whether you are working on a new engine or trying to eliminate a bug, it is
102120 always a good idea to turn on debug logging. When debug logging is enabled a
@@ -104,6 +122,10 @@ Turn on debug logging
104122 message. It can be turned on by setting ``debug: False`` to ``debug: True`` in
105123 :origin:`settings.yml <searx/settings.yml>`.
106124
125+.. sidebar:: :ref:`make test`
126+
127+ Alternatively use the :ref:`make test` targets.
128+
107129Run ``./manage.sh tests`` before creating a PR.
108130 Failing build on Travis is common because of PEP8 checks. So a new commit
109131 must be created containing these format fixes. This phase can be skipped if