git.iko.soy/searx
on master help sign in
doc: proofread of the all reST sources (no content change)

Normalize reST sources with best practice and KISS in mind. to name a few points: - simplify reST tables - make use of ``literal`` markup for monospace rendering - fix code-blocks for better rendering in HTML - normalize section header markup - limit all lines to a maximum of 79 characters - add option -H to the sudo command used in code blocks - drop useless indentation of lists - ... [1] https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

commite9fff4fde6d7a8bec3fae087d2afe1fce2145f22 authorMarkus Heiser <markus.heiser@darmarit.de> dateThu 12 Dec 2019, 19:20 (7 years ago) parent0011890
17 files changed+716−769
compare with master
unifiedsplit
docs/admin/api.rst+81−79 docs/admin/filtron.rst+29−26 docs/admin/morty.rst+12−7 docs/blog/admin.rst+27−26 docs/blog/blog.rst+1 docs/blog/intro-offline.rst+38−26 docs/blog/python3.rst+31−17 docs/conf.py+6−2 docs/dev/contribution_guide.rst+47−56 docs/dev/engine_overview.rst+202−251 docs/dev/plugins.rst+24−21 docs/dev/quickstart.rst+47−45 docs/dev/search_api.rst+67−159 docs/dev/translation.rst+27−17 docs/index.rst+18−16 docs/user/own-instance.rst+52−18 docs/user/search_syntax.rst+7−3
docs/admin/api.rst+81−79
@@ -1,94 +1,96 @@
11.. _adminapi:
22
3+==================
34Administration API
4−------------------
5+==================
56
67Get configuration data
7−~~~~~~~~~~~~~~~~~~~~~~
8+======================
89
9−.. code:: sh
10+.. code:: http
1011
11− GET /config
12+ GET /config HTTP/1.1
1213
1314Sample response
14−```````````````
15+---------------
16+
17+.. code:: json
18+
19+ {
20+ "autocomplete": "",
21+ "categories": [
22+ "map",
23+ "it",
24+ "images",
25+ ],
26+ "default_locale": "",
27+ "default_theme": "oscar",
28+ "engines": [
29+ {
30+ "categories": [
31+ "map"
32+ ],
33+ "enabled": true,
34+ "name": "openstreetmap",
35+ "shortcut": "osm"
36+ },
37+ {
38+ "categories": [
39+ "it"
40+ ],
41+ "enabled": true,
42+ "name": "arch linux wiki",
43+ "shortcut": "al"
44+ },
45+ {
46+ "categories": [
47+ "images"
48+ ],
49+ "enabled": true,
50+ "name": "google images",
51+ "shortcut": "goi"
52+ },
53+ {
54+ "categories": [
55+ "it"
56+ ],
57+ "enabled": false,
58+ "name": "bitbucket",
59+ "shortcut": "bb"
60+ },
61+ ],
62+ "instance_name": "searx",
63+ "locales": {
64+ "de": "Deutsch (German)",
65+ "en": "English",
66+ "eo": "Esperanto (Esperanto)",
67+ },
68+ "plugins": [
69+ {
70+ "enabled": true,
71+ "name": "HTTPS rewrite"
72+ },
73+ {
74+ "enabled": false,
75+ "name": "Vim-like hotkeys"
76+ }
77+ ],
78+ "safe_search": 0
79+ }
1580
16−.. code:: sh
17−
18− {
19− "autocomplete": "",
20− "categories": [
21− "map",
22− "it",
23− "images",
24− ],
25− "default_locale": "",
26− "default_theme": "oscar",
27− "engines": [
28− {
29− "categories": [
30− "map"
31− ],
32− "enabled": true,
33− "name": "openstreetmap",
34− "shortcut": "osm"
35− },
36− {
37− "categories": [
38− "it"
39− ],
40− "enabled": true,
41− "name": "arch linux wiki",
42− "shortcut": "al"
43− },
44− {
45− "categories": [
46− "images"
47− ],
48− "enabled": true,
49− "name": "google images",
50− "shortcut": "goi"
51− },
52− {
53− "categories": [
54− "it"
55− ],
56− "enabled": false,
57− "name": "bitbucket",
58− "shortcut": "bb"
59− },
60− ],
61− "instance_name": "searx",
62− "locales": {
63− "de": "Deutsch (German)",
64− "en": "English",
65− "eo": "Esperanto (Esperanto)",
66− },
67− "plugins": [
68− {
69− "enabled": true,
70− "name": "HTTPS rewrite"
71− },
72− {
73− "enabled": false,
74− "name": "Vim-like hotkeys"
75− }
76− ],
77− "safe_search": 0
78− }
7981
8082Embed search bar
81−----------------
83+================
8284
83−The search bar can be embedded into websites. Just paste the example into the HTML of the site.
84−URL of the searx instance and values are customizable.
85+The search bar can be embedded into websites. Just paste the example into the
86+HTML of the site. URL of the searx instance and values are customizable.
8587
8688.. code:: html
87−
88− <form method="post" action="https://searx.me/">
89− <!-- search query --> <input type="text" name="q" />
90− <!-- categories --> <input type="hidden" name="categories" value="general,social media" />
91− <!-- language --> <input type="hidden" name="lang" value="all" />
92− <!-- locale --> <input type="hidden" name="locale" value="en" />
93− <!-- date filter --> <input type="hidden" name="time_range" value="month" />
94− </form>
89+
90+ <form method="post" action="https://searx.me/">
91+ <!-- search --> <input type="text" name="q" />
92+ <!-- categories --> <input type="hidden" name="categories" value="general,social media" />
93+ <!-- language --> <input type="hidden" name="lang" value="all" />
94+ <!-- locale --> <input type="hidden" name="locale" value="en" />
95+ <!-- date filter --> <input type="hidden" name="time_range" value="month" />
96+ </form>
docs/admin/filtron.rst+29−26
@@ -1,25 +1,26 @@
1+==========================
12How to protect an instance
23==========================
34
4−Searx depens on external search services. To avoid the abuse of these services it is advised to limit the number of requests processed by searx.
5−
6−An application firewall, ``filtron`` solves exactly this problem. Information on how to install it can be found at the `project page of filtron <https://github.com/asciimoo/filtron>`__.
7−
8−Sample configuration of filtron
9−-------------------------------
5+Searx depens on external search services. To avoid the abuse of these services
6+it is advised to limit the number of requests processed by searx.
107
11−An example configuration can be find below. This configuration limits the access of
8+An application firewall, ``filtron`` solves exactly this problem. Information
9+on how to install it can be found at the `project page of filtron
10+<https://github.com/asciimoo/filtron>`__.
1211
13− * scripts or applications (roboagent limit)
1412
15− * webcrawlers (botlimit)
16−
17− * IPs which send too many requests (IP limit)
18−
19− * too many json, csv, etc. requests (rss/json limit)
13+Sample configuration of filtron
14+===============================
2015
21− * the same UserAgent of if too many requests (useragent limit)
16+An example configuration can be find below. This configuration limits the access
17+of:
2218
19+- scripts or applications (roboagent limit)
20+- webcrawlers (botlimit)
21+- IPs which send too many requests (IP limit)
22+- too many json, csv, etc. requests (rss/json limit)
23+- the same UserAgent of if too many requests (useragent limit)
2324
2425.. code:: json
2526
@@ -120,26 +121,28 @@ An example configuration can be find below. This configuration limits the access
120121
121122
122123Route request through filtron
123−-----------------------------
124+=============================
124125
125126Filtron can be started using the following command:
126127
127−.. code:: bash
128+.. code:: sh
128129
129− $ filtron -rules rules.json
130+ $ filtron -rules rules.json
130131
131−It listens on 127.0.0.1:4004 and forwards filtered requests to 127.0.0.1:8888 by default.
132+It listens on ``127.0.0.1:4004`` and forwards filtered requests to
133+``127.0.0.1:8888`` by default.
132134
133135Use it along with ``nginx`` with the following example configuration.
134136
135137.. code:: nginx
136138
137− location / {
138− proxy_set_header Host $http_host;
139− proxy_set_header X-Real-IP $remote_addr;
140− proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
141− proxy_set_header X-Scheme $scheme;
142− proxy_pass http://127.0.0.1:4004/;
143− }
139+ location / {
140+ proxy_set_header Host $http_host;
141+ proxy_set_header X-Real-IP $remote_addr;
142+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
143+ proxy_set_header X-Scheme $scheme;
144+ proxy_pass http://127.0.0.1:4004/;
145+ }
144146
145−Requests are coming from port 4004 going through filtron and then forwarded to port 8888 where a searx is being run.
147+Requests are coming from port 4004 going through filtron and then forwarded to
148+port 8888 where a searx is being run.
docs/admin/morty.rst+12−7
@@ -1,14 +1,17 @@
1+=========================
12How to setup result proxy
23=========================
34
4−By default searx can only act as an image proxy for result images,
5−but it is possible to proxify all the result URLs with an external service,
6−`morty <https://github.com/asciimoo/morty>`__.
5+.. _morty: https://github.com/asciimoo/morty
6+.. _morty's README: https://github.com/asciimoo/morty
77
8−To use this feature, morty has to be installed and activated in searx's ``settings.yml``.
8+By default searx can only act as an image proxy for result images, but it is
9+possible to proxify all the result URLs with an external service, morty_.
910
10−Add the following snippet to your ``settings.yml`` and restart searx:
11+To use this feature, morty has to be installed and activated in searx's
12+``settings.yml``.
1113
14+Add the following snippet to your ``settings.yml`` and restart searx:
1215
1316.. code:: yaml
1417
@@ -16,6 +19,8 @@ Add the following snippet to your ``settings.yml`` and restart searx:
1619 url : http://127.0.0.1:3000/
1720 key : your_morty_proxy_key
1821
19−``url`` is the address of the running morty service
22+``url``
23+ Is the address of the running morty service.
2024
21−``key`` is an optional argument, see `morty's README <https://github.com/asciimoo/morty>`__ for more information.
25+``key``
26+ Is an optional argument, see `morty's README`_ for more information.
docs/blog/admin.rst+27−26
@@ -1,42 +1,43 @@
1−Searx admin interface: manage your instance from your browser
1+=============================================================
2+Searx admin interface
23=============================================================
34
4−One of the biggest advantages of searx is being extremely customizable. But at first it can be daunting to newcomers.
5−A barrier of taking advantage of this feature is our ugly settings file which is sometimes hard to understand and edit.
5+.. _searx-admin: https://github.com/kvch/searx-admin#searx-admin
6+.. _NLnet Foundation: https://nlnet.nl/
67
7−To make self-hosting searx more accessible a new tool is introduced, called ``searx-admin``.
8−It is a web application which is capable of managing your instance and manipulating its settings via a web UI.
9−It aims to replace editing of ``settings.yml`` for less experienced administrators or people
10−who prefer graphical admin interfaces.
8+ manage your instance from your browser
119
12−.. figure:: searx-admin-engines.png
13− :scale: 50 %
14− :alt: Screenshot of engine list
15− :align: center
16− :figclass: align-center
17−
18− Configuration page of engines
10+.. sidebar:: Installation
1911
12+ Installation guide can be found in the repository of searx-admin_.
2013
21−Since ``searx-admin`` acts as a supervisor for searx, we have decided to implement it
22−as a standalone tool instead of part of searx. Another reason for making it a standalone
23−tool is that the codebase and dependencies of searx should not grow because of a fully optional feature,
24−which does not affect existing instances.
14+One of the biggest advantages of searx is being extremely customizable. But at
15+first it can be daunting to newcomers. A barrier of taking advantage of this
16+feature is our ugly settings file which is sometimes hard to understand and
17+edit.
2518
19+To make self-hosting searx more accessible a new tool is introduced, called
20+``searx-admin``. It is a web application which is capable of managing your
21+instance and manipulating its settings via a web UI. It aims to replace editing
22+of ``settings.yml`` for less experienced administrators or people who prefer
23+graphical admin interfaces.
2624
27−Installation
28−------------
25+.. figure:: searx-admin-engines.png
26+ :alt: Screenshot of engine list
2927
30−Installation guide can be found in the repository of searx-admin:
31−https://github.com/kvch/searx-admin#installation--usage
28+ Configuration page of engines
3229
33−Acknowledgements
34−----------------
30+Since ``searx-admin`` acts as a supervisor for searx, we have decided to
31+implement it as a standalone tool instead of part of searx. Another reason for
32+making it a standalone tool is that the codebase and dependencies of searx
33+should not grow because of a fully optional feature, which does not affect
34+existing instances.
3535
36−This development was sponsored by `NLnet Foundation`_.
3736
38−.. _NLnet Foundation: https://nlnet.nl/
37+Acknowledgements
38+================
3939
40+This development was sponsored by `NLnet Foundation`_.
4041
4142| Happy hacking.
4243| kvch // 2017.08.22 21:25
docs/blog/blog.rst+1
@@ -1,3 +1,4 @@
1+====
12Blog
23====
34
docs/blog/intro-offline.rst+38−26
@@ -1,58 +1,70 @@
1+===============================
12Preparation for offline engines
23===============================
34
45Offline engines
5−---------------
6+===============
67
7−To extend the functionality of searx, offline engines are going to be introduced. An offline engine is an engine which does not need Internet connection to perform a search and does not use HTTP to communicate.
8+To extend the functionality of searx, offline engines are going to be
9+introduced. An offline engine is an engine which does not need Internet
10+connection to perform a search and does not use HTTP to communicate.
811
9−Offline engines can be configured as online engines, by adding those to the `engines` list of `settings.yml`. Thus, searx finds the engine file and imports it.
12+Offline engines can be configured as online engines, by adding those to the
13+`engines` list of :origin:`settings.yml <searx/settings.yml>`. Thus, searx
14+finds the engine file and imports it.
1015
1116Example skeleton for the new engines:
1217
1318.. code:: python
1419
15− from subprocess import PIPE, Popen
20+ from subprocess import PIPE, Popen
1621
17− categories = ['general']
18− offline = True
22+ categories = ['general']
23+ offline = True
1924
20− def init(settings):
21− pass
25+ def init(settings):
26+ pass
2227
23− def search(query, params):
24− process = Popen(['ls', query], stdout=PIPE)
25− return_code = process.wait()
26− if return_code != 0:
27− raise RuntimeError('non-zero return code', return_code)
28+ def search(query, params):
29+ process = Popen(['ls', query], stdout=PIPE)
30+ return_code = process.wait()
31+ if return_code != 0:
32+ raise RuntimeError('non-zero return code', return_code)
2833
29− results = []
30− line = process.stdout.readline()
31− while line:
32− result = parse_line(line)
33− results.append(results)
34+ results = []
35+ line = process.stdout.readline()
36+ while line:
37+ result = parse_line(line)
38+ results.append(results)
3439
35− line = process.stdout.readline()
40+ line = process.stdout.readline()
3641
37− return results
42+ return results
3843
3944
4045Development progress
41−--------------------
46+====================
4247
43−First, a proposal has been created as a Github issue. Then it was moved to the wiki as a design document. You can read it here: :wiki:`Offline-engines`.
48+First, a proposal has been created as a Github issue. Then it was moved to the
49+wiki as a design document. You can read it here: :wiki:`Offline-engines`.
4450
45−In this development step, searx core was prepared to accept and perform offline searches. Offline search requests are scheduled together with regular offline requests.
51+In this development step, searx core was prepared to accept and perform offline
52+searches. Offline search requests are scheduled together with regular offline
53+requests.
4654
47−As offline searches can return arbitrary results depending on the engine, the current result templates were insufficient to present such results. Thus, a new template is introduced which is caplable of presenting arbitrary key value pairs as a table. You can check out the pull request for more details: https://github.com/asciimoo/searx/pull/1700
55+As offline searches can return arbitrary results depending on the engine, the
56+current result templates were insufficient to present such results. Thus, a new
57+template is introduced which is caplable of presenting arbitrary key value pairs
58+as a table. You can check out the pull request for more details see
59+:pull:`1700`.
4860
4961Next steps
50−----------
62+==========
5163
5264Today, it is possible to create/run an offline engine. However, it is going to be publicly available for everyone who knows the searx instance. So the next step is to introduce token based access for engines. This way administrators are able to limit the access to private engines.
5365
5466Acknowledgement
55−---------------
67+===============
5668
5769This development was sponsored by `Search and Discovery Fund`_ of `NLnet Foundation`_ .
5870
docs/blog/python3.rst+31−17
@@ -1,7 +1,17 @@
1−Introducing Python3 support
2−===========================
1+============================
2+Introducing Python 3 support
3+============================
34
4−As most operation systems are coming with Python3 installed by default. So it is time for searx to support Python3. But don't worry support of Python2.7 won't be dropped.
5+.. _Python 2.7 clock: https://pythonclock.org/
6+
7+.. sidebar:: Python 2.7 to 3 upgrade
8+
9+ This chapter exists of historical reasons. Python 2.7 release schedule ends
10+ (`Python 2.7 clock`_) after 11 years Python 3 exists
11+
12+As most operation systems are coming with Python3 installed by default. So it is
13+time for searx to support Python3. But don't worry support of Python2.7 won't be
14+dropped.
515
616.. image:: searxpy3.png
717 :scale: 50 %
@@ -9,13 +19,14 @@ As most operation systems are coming with Python3 installed by default. So it is
919 :align: center
1020
1121
12−How to run searx using Python3
13−------------------------------
22+How to run searx using Python 3
23+===============================
1424
15−Please make sure that you run at least Python3.5.
25+Please make sure that you run at least Python 3.5.
1626
17−To run searx, first a Python3 virtualenv should be created. After entering the virtualenv,
18−dependencies must be installed. Then run searx with python3 instead of the usual python command.
27+To run searx, first a Python3 virtualenv should be created. After entering the
28+virtualenv, dependencies must be installed. Then run searx with python3 instead
29+of the usual python command.
1930
2031.. code:: sh
2132
@@ -25,25 +36,28 @@ dependencies must be installed. Then run searx with python3 instead of the usual
2536 python3 searx/webapp.py
2637
2738
28−If you want to run searx using Python2.7, you don't have to do anything differently as before.
39+If you want to run searx using Python2.7, you don't have to do anything
40+differently as before.
2941
3042Fun facts
31−---------
43+=========
3244
3345- 115 files were changed when implementing the support for both Python versions.
3446
35−- All of the dependencies was compatible except for the robotframework used for browser tests. Thus, these tests were migrated to splinter. So from now on both versions are being tested on Travis and can be tested locally.
47+- All of the dependencies was compatible except for the robotframework used for
48+ browser tests. Thus, these tests were migrated to splinter. So from now on
49+ both versions are being tested on Travis and can be tested locally.
3650
37−If you found bugs...
38−--------------------
51+If you found bugs
52+=================
3953
40−...please open an issue on `GitHub`_. Make sure that you mention your Python version in your issue,
41−so we can investigate it properly.
54+Please open an issue on `GitHub`_. Make sure that you mention your Python
55+version in your issue, so we can investigate it properly.
4256
4357.. _GitHub: https://github.com/asciimoo/searx/issues
4458
45−Acknowledgement
46−---------------
59+Acknowledgment
60+==============
4761
4862This development was sponsored by `NLnet Foundation`_.
4963
docs/conf.py+6−2
@@ -20,12 +20,16 @@ master_doc = "index"
2020source_suffix = '.rst'
2121
2222# usage:: lorem :patch:`f373169` ipsum
23−
2423extlinks = {}
24+
25+# upstream links
26+extlinks['wiki'] = ('https://github.com/asciimoo/searx/wiki/%s', ' ')
27+extlinks['pull'] = ('https://github.com/asciimoo/searx/pull/%s', 'PR ')
28+
29+# links to custom brand
2530extlinks['origin'] = (GIT_URL + '/blob/master/%s', 'git://')
2631extlinks['patch'] = (GIT_URL + '/commit/%s', '#')
2732extlinks['search'] = (SEARX_URL + '/%s', '#')
28−extlinks['wiki'] = ('https://github.com/asciimoo/searx/wiki/%s', ' ')
2933
3034extensions = [
3135 'sphinx.ext.extlinks',
docs/dev/contribution_guide.rst+47−56
@@ -1,66 +1,68 @@
1+=================
12How to contribute
2−-----------------
3+=================
34
45Prime directives: Privacy, Hackability
5−~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
6+======================================
67
7−Searx has two prime directives, privacy-by-design and hackability. The
8+Searx has two prime directives, **privacy-by-design and hackability** . The
89hackability comes in three levels:
910
10−- support of search engines
11−- plugins to alter search behaviour
12−- hacking searx itself
11+- support of search engines
12+- plugins to alter search behaviour
13+- hacking searx itself
1314
14−Note the lack of "world domination" among the directives.
15−Searx has no intention of wide mass-adoption, rounded
16−corners, etc. The prime directive "privacy" deserves a separate
17−chapter, as it's quite uncommon unfortunately.
15+Note the lack of "world domination" among the directives. Searx has no
16+intention of wide mass-adoption, rounded corners, etc. The prime directive
17+"privacy" deserves a separate chapter, as it's quite uncommon unfortunately.
1818
1919Privacy-by-design
20−^^^^^^^^^^^^^^^^^
20+-----------------
2121
22−Searx was born out of the need for a privacy-respecting search tool
23−which can be extended easily to maximize both its search and its
24−privacy protecting capabilities.
22+Searx was born out of the need for a **privacy-respecting** search tool which
23+can be extended easily to maximize both, its search and its privacy protecting
24+capabilities.
2525
26−A few widely used features work differently or turned off by default or not implemented
27−at all as a consequence of privacy-by-design.
26+A few widely used features work differently or turned off by default or not
27+implemented at all **as a consequence of privacy-by-design**.
2828
29−If a feature reduces the privacy preserving aspects of searx, it
30−should be switched off by default or should not implemented at all.
31−There are plenty of search engines already providing such features.
32−If a feature reduces the protection of searx, users must be
33−informed about the effect of choosing to enable it. Features
34−that protect privacy but differ from the expectations of the
35−user should also be explained.
29+If a feature reduces the privacy preserving aspects of searx, it should be
30+switched off by default or should not implemented at all. There are plenty of
31+search engines already providing such features. If a feature reduces the
32+protection of searx, users must be informed about the effect of choosing to
33+enable it. Features that protect privacy but differ from the expectations of
34+the user should also be explained.
3635
37−Also, if you think that something works weird with searx,
38−it's might be because of the tool you use is designed in a way to interfere with
39−the privacy respect. Submitting a bugreport to the vendor of the tool that
40−misbehaves might be a good feedback to reconsider the disrespect to
41−its customers (e.g. GET vs POST requests in various browsers).
36+Also, if you think that something works weird with searx, it's might be because
37+of the tool you use is designed in a way to interfere with the privacy respect.
38+Submitting a bugreport to the vendor of the tool that misbehaves might be a good
39+feedback to reconsider the disrespect to its customers (e.g. ``GET`` vs ``POST``
40+requests in various browsers).
4241
43−Remember the other prime directive of searx is to be hackable, so if the
44−above privacy concerns do not fancy you, simply fork it.
42+Remember the other prime directive of searx is to be hackable, so if the above
43+privacy concerns do not fancy you, simply fork it.
4544
46−Happy hacking.
45+ *Happy hacking.*
4746
4847Code
49−~~~~
48+====
49+
50+.. _PEP8: https://www.python.org/dev/peps/pep-0008/
51+
5052
5153In order to submit a patch, please follow the steps below:
5254
5355- Follow coding conventions.
5456
55− - PEP8 standards apply, except the convention of line length
56−
57+ - PEP8_ standards apply, except the convention of line length
5758 - Maximum line length is 120 characters
5859
59−- Check if your code breaks existing tests. If so, update the tests or fix your code.
60+- Check if your code breaks existing tests. If so, update the tests or fix your
61+ code.
6062
6163- If your code can be unit-tested, add unit tests.
6264
63−- Add yourself to the AUTHORS file.
65+- Add yourself to the :origin:`AUTHORS.rst` file.
6466
6567- Create a pull request.
6668
@@ -68,31 +70,20 @@ For more help on getting started with searx development, see :ref:`devquickstart
6870
6971
7072Translation
71−~~~~~~~~~~~
73+===========
7274
73−Translation currently takes place on
74−`transifex <https://transifex.com/projects/p/searx>`__.
75+Translation currently takes place on :ref:`transifex <translation>`.
7576
76−**Please, do not update translation files in the repo.**
77+.. caution::
7778
78−Documentation
79−~~~~~~~~~~~~~
79+ Please, do not update translation files in the repo.
8080
81−The documentation is built using Sphinx. So in order to be able to generate the required
82−files, you have to install it on your system. (It can be installed easily using pip.)
8381
84−1. Checkout the gh-pages branch.
85−
86−2. Edit the rst file you wish to update. Or create a new rst file and place it under the appropriate folder.
87−
88−3. Build the documentation using Sphinx.
89−
90−4. Add the updated and created files of these extension:
91−
92− - .rst
82+Documentation
83+=============
9384
94− - .html
85+.. admonition:: ToDo
9586
96− - .txt
87+ docs are no longer in gh-branch
9788
98−6. Create a pull request.
89+ Update this chapter and document the /docs workflow!!
docs/dev/engine_overview.rst+202−251
@@ -1,21 +1,25 @@
1+
2+.. _engines-dev:
3+
4+===============
15Engine overview
26===============
37
8+.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
49
5−searx is a `metasearch-engine <https://en.wikipedia.org/wiki/Metasearch_engine>`__,
6−so it uses different search engines to provide better results.
7−
8−Because there is no general search API which could be used for every
9−search engine, an adapter has to be built between searx and the
10−external search engines. Adapters are stored under the folder
11−:origin:`searx/engines`.
10+searx is a metasearch-engine_, so it uses different search engines to provide
11+better results.
1212
13+Because there is no general search API which could be used for every search
14+engine, an adapter has to be built between searx and the external search
15+engines. Adapters are stored under the folder :origin:`searx/engines`.
1316
1417.. contents::
1518 :depth: 3
19+ :backlinks: entry
1620
1721general engine configuration
18−----------------------------
22+============================
1923
2024It is required to tell searx the type of results the engine provides. The
2125arguments can be set in the engine file or in the settings file
@@ -23,292 +27,239 @@ arguments can be set in the engine file or in the settings file
2327the ones in the engine file.
2428
2529It does not matter if an option is stored in the engine file or in the
26−settings. However, the standard way is the following:
30+settings. However, the standard way is the following:
2731
2832
2933engine file
30−~~~~~~~~~~~
31−
32−+----------------------+-----------+-----------------------------------------+
33−| argument | type | information |
34−+======================+===========+=========================================+
35−| categories | list | pages, in which the engine is working |
36−+----------------------+-----------+-----------------------------------------+
37−| paging | boolean | support multible pages |
38−+----------------------+-----------+-----------------------------------------+
39−| language\_support | boolean | support language choosing |
40−+----------------------+-----------+-----------------------------------------+
41−| time\_range\_support | boolean | support search time range |
42−+----------------------+-----------+-----------------------------------------+
43−| offline | boolean | engine runs offline |
44−+----------------------+-----------+-----------------------------------------+
34+-----------
35+
36+======================= =========== ===========================================
37+argument type information
38+======================= =========== ===========================================
39+categories list pages, in which the engine is working
40+paging boolean support multible pages
41+language_support boolean support language choosing
42+time_range_support boolean support search time range
43+offline boolean engine runs offline
44+======================= =========== ===========================================
45+
4546
4647settings.yml
47−~~~~~~~~~~~~
48−
49−+------------+----------+-----------------------------------------------+
50−| argument | type | information |
51−+============+==========+===============================================+
52−| name | string | name of search-engine |
53−+------------+----------+-----------------------------------------------+
54−| engine | string | name of searx-engine (filename without .py) |
55−+------------+----------+-----------------------------------------------+
56−| shortcut | string | shortcut of search-engine |
57−+------------+----------+-----------------------------------------------+
58−| timeout | string | specific timeout for search-engine |
59−+------------+----------+-----------------------------------------------+
48+------------
49+
50+======================= =========== ===========================================
51+argument type information
52+======================= =========== ===========================================
53+name string name of search-engine
54+engine string name of searx-engine
55+ (filename without ``.py``)
56+shortcut string shortcut of search-engine
57+timeout string specific timeout for search-engine
58+======================= =========== ===========================================
59+
6060
6161overrides
62−~~~~~~~~~
63−
64−A few of the options have default values in the engine, but are
65−often overwritten by the settings. If ``None`` is assigned to an option
66−in the engine file, it has to be redefined in the settings,
67−otherwise searx will not start with that engine.
68−
69−The naming of overrides is arbitrary. But the recommended
70−overrides are the following:
71−
72−+-----------------------+----------+----------------------------------------------------------------+
73−| argument | type | information |
74−+=======================+==========+================================================================+
75−| base\_url | string | base-url, can be overwritten to use same engine on other URL |
76−+-----------------------+----------+----------------------------------------------------------------+
77−| number\_of\_results | int | maximum number of results per request |
78−+-----------------------+----------+----------------------------------------------------------------+
79−| language | string | ISO code of language and country like en\_US |
80−+-----------------------+----------+----------------------------------------------------------------+
81−| api\_key | string | api-key if required by engine |
82−+-----------------------+----------+----------------------------------------------------------------+
62+---------
63+
64+A few of the options have default values in the engine, but are often
65+overwritten by the settings. If ``None`` is assigned to an option in the engine
66+file, it has to be redefined in the settings, otherwise searx will not start
67+with that engine.
68+
69+The naming of overrides is arbitrary. But the recommended overrides are the
70+following:
71+
72+======================= =========== ===========================================
73+argument type information
74+======================= =========== ===========================================
75+base_url string base-url, can be overwritten to use same
76+ engine on other URL
77+number_of_results int maximum number of results per request
78+language string ISO code of language and country like en_US
79+api_key string api-key if required by engine
80+======================= =========== ===========================================
8381
8482example code
85−~~~~~~~~~~~~
83+------------
8684
8785.. code:: python
8886
89− # engine dependent config
90− categories = ['general']
91− paging = True
92− language_support = True
87+ # engine dependent config
88+ categories = ['general']
89+ paging = True
90+ language_support = True
91+
9392
9493making a request
95−----------------
94+================
9695
97−To perform a search an URL have to be specified. In addition to
98−specifying an URL, arguments can be passed to the query.
96+To perform a search an URL have to be specified. In addition to specifying an
97+URL, arguments can be passed to the query.
9998
10099passed arguments
101−~~~~~~~~~~~~~~~~
100+----------------
102101
103−These arguments can be used to construct the search query. Furthermore,
102+These arguments can be used to construct the search query. Furthermore,
104103parameters with default value can be redefined for special purposes.
105104
106−+----------------------+------------+------------------------------------------------------------------------+
107−| argument | type | default-value, information |
108−+======================+============+========================================================================+
109−| url | string | ``''`` |
110−+----------------------+------------+------------------------------------------------------------------------+
111−| method | string | ``'GET'`` |
112−+----------------------+------------+------------------------------------------------------------------------+
113−| headers | set | ``{}`` |
114−+----------------------+------------+------------------------------------------------------------------------+
115−| data | set | ``{}`` |
116−+----------------------+------------+------------------------------------------------------------------------+
117−| cookies | set | ``{}`` |
118−+----------------------+------------+------------------------------------------------------------------------+
119−| verify | boolean | ``True`` |
120−+----------------------+------------+------------------------------------------------------------------------+
121−| headers.User-Agent | string | a random User-Agent |
122−+----------------------+------------+------------------------------------------------------------------------+
123−| category | string | current category, like ``'general'`` |
124−+----------------------+------------+------------------------------------------------------------------------+
125−| started | datetime | current date-time |
126−+----------------------+------------+------------------------------------------------------------------------+
127−| pageno | int | current pagenumber |
128−+----------------------+------------+------------------------------------------------------------------------+
129−| language | string | specific language code like ``'en_US'``, or ``'all'`` if unspecified |
130−+----------------------+------------+------------------------------------------------------------------------+
105+====================== ============ ========================================================================
106+argument type default-value, information
107+====================== ============ ========================================================================
108+url string ``''``
109+method string ``'GET'``
110+headers set ``{}``
111+data set ``{}``
112+cookies set ``{}``
113+verify boolean ``True``
114+headers.User-Agent string a random User-Agent
115+category string current category, like ``'general'``
116+started datetime current date-time
117+pageno int current pagenumber
118+language string specific language code like ``'en_US'``, or ``'all'`` if unspecified
119+====================== ============ ========================================================================
131120
132121parsed arguments
133−~~~~~~~~~~~~~~~~
134−
135−The function ``def request(query, params):`` always returns the
136−``params`` variable. Inside searx, the following paramters can be
137−used to specify a search request:
138−
139−+------------+-----------+---------------------------------------------------------+
140−| argument | type | information |
141−+============+===========+=========================================================+
142−| url | string | requested url |
143−+------------+-----------+---------------------------------------------------------+
144−| method | string | HTTP request method |
145−+------------+-----------+---------------------------------------------------------+
146−| headers | set | HTTP header information |
147−+------------+-----------+---------------------------------------------------------+
148−| data | set | HTTP data information (parsed if ``method != 'GET'``) |
149−+------------+-----------+---------------------------------------------------------+
150−| cookies | set | HTTP cookies |
151−+------------+-----------+---------------------------------------------------------+
152−| verify | boolean | Performing SSL-Validity check |
153−+------------+-----------+---------------------------------------------------------+
122+----------------
123+
124+The function ``def request(query, params):`` always returns the ``params``
125+variable. Inside searx, the following paramters can be used to specify a search
126+request:
127+
128+============ =========== =========================================================
129+argument type information
130+============ =========== =========================================================
131+url string requested url
132+method string HTTP request method
133+headers set HTTP header information
134+data set HTTP data information (parsed if ``method != 'GET'``)
135+cookies set HTTP cookies
136+verify boolean Performing SSL-Validity check
137+============ =========== =========================================================
138+
154139
155140example code
156−~~~~~~~~~~~~
141+------------
157142
158143.. code:: python
159144
160− # search-url
161− base_url = 'https://example.com/'
162− search_string = 'search?{query}&page={page}'
145+ # search-url
146+ base_url = 'https://example.com/'
147+ search_string = 'search?{query}&page={page}'
163148
164− # do search-request
165− def request(query, params):
166− search_path = search_string.format(
167− query=urlencode({'q': query}),
168− page=params['pageno'])
149+ # do search-request
150+ def request(query, params):
151+ search_path = search_string.format(
152+ query=urlencode({'q': query}),
153+ page=params['pageno'])
169154
170− params['url'] = base_url + search_path
155+ params['url'] = base_url + search_path
156+
157+ return params
171158
172− return params
173159
174160returned results
175−----------------
161+================
176162
177−Searx is able to return results of different media-types.
178−Currently the following media-types are supported:
163+Searx is able to return results of different media-types. Currently the
164+following media-types are supported:
179165
180−- default
181−- images
182−- videos
183−- torrent
184−- map
166+- default_
167+- images_
168+- videos_
169+- torrent_
170+- map_
185171
186−To set another media-type as default, the parameter
187−``template`` must be set to the desired type.
172+To set another media-type as default, the parameter ``template`` must be set to
173+the desired type.
188174
189175default
190−~~~~~~~
191−
192−+--------------------+---------------------------------------------------------------------------------------------------------------+
193−| result-parameter | information |
194−+====================+===============================================================================================================+
195−| url | string, url of the result |
196−+--------------------+---------------------------------------------------------------------------------------------------------------+
197−| title | string, title of the result |
198−+--------------------+---------------------------------------------------------------------------------------------------------------+
199−| content | string, general result-text |
200−+--------------------+---------------------------------------------------------------------------------------------------------------+
201−| publishedDate | :py:class:`datetime.datetime`, time of publish |
202−+--------------------+---------------------------------------------------------------------------------------------------------------+
176+-------
177+
178+========================= =====================================================
179+result-parameter information
180+========================= =====================================================
181+url string, url of the result
182+title string, title of the result
183+content string, general result-text
184+publishedDate :py:class:`datetime.datetime`, time of publish
185+========================= =====================================================
203186
204187images
205−~~~~~~
206−
207−to use this template, the parameter
208−
209−+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
210−| result-parameter | information |
211−+====================+=======================================================================================================================================+
212−| template | is set to ``images.html`` |
213−+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
214−| url | string, url to the result site |
215−+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
216−| title | string, title of the result *(partly implemented)* |
217−+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
218−| content | *(partly implemented)* |
219−+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
220−| publishedDate | :py:class:`datetime.datetime`, time of publish *(partly implemented)* |
221−+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
222−| img\_src | string, url to the result image |
223−+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
224−| thumbnail\_src | string, url to a small-preview image |
225−+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
188+------
189+
190+To use this template, the parameter:
191+
192+========================= =====================================================
193+result-parameter information
194+========================= =====================================================
195+template is set to ``images.html``
196+url string, url to the result site
197+title string, title of the result *(partly implemented)*
198+content *(partly implemented)*
199+publishedDate :py:class:`datetime.datetime`,
200+ time of publish *(partly implemented)*
201+img\_src string, url to the result image
202+thumbnail\_src string, url to a small-preview image
203+========================= =====================================================
226204
227205videos
228−~~~~~~
229−
230−+--------------------+--------------------------------------------------------------------------------------------------------------+
231−| result-parameter | information |
232−+====================+==============================================================================================================+
233−| template | is set to ``videos.html`` |
234−+--------------------+--------------------------------------------------------------------------------------------------------------+
235−| url | string, url of the result |
236−+--------------------+--------------------------------------------------------------------------------------------------------------+
237−| title | string, title of the result |
238−+--------------------+--------------------------------------------------------------------------------------------------------------+
239−| content | *(not implemented yet)* |
240−+--------------------+--------------------------------------------------------------------------------------------------------------+
241−| publishedDate | :py:class:`datetime.datetime`, time of publish |
242−+--------------------+--------------------------------------------------------------------------------------------------------------+
243−| thumbnail | string, url to a small-preview image |
244−+--------------------+--------------------------------------------------------------------------------------------------------------+
206+------
207+
208+========================= =====================================================
209+result-parameter information
210+========================= =====================================================
211+template is set to ``videos.html``
212+url string, url of the result
213+title string, title of the result
214+content *(not implemented yet)*
215+publishedDate :py:class:`datetime.datetime`, time of publish
216+thumbnail string, url to a small-preview image
217+========================= =====================================================
245218
246219torrent
247−~~~~~~~
248−
249−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
250−| result-parameter | information |
251−+==================+=======================================================================================================================================+
252−| template | is set to ``torrent.html`` |
253−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
254−| url | string, url of the result |
255−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
256−| title | string, title of the result |
257−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
258−| content | string, general result-text |
259−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
260−| publishedDate | :py:class:`datetime.datetime`, time of publish *(not implemented yet)* |
261−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
262−| seed | int, number of seeder |
263−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
264−| leech | int, number of leecher |
265−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
266−| filesize | int, size of file in bytes |
267−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
268−| files | int, number of files |
269−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
270−| magnetlink | string, `magnetlink <https://en.wikipedia.org/wiki/Magnet_URI_scheme>`__ of the result |
271−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
272−| torrentfile | string, torrentfile of the result |
273−+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
220+-------
221+
222+.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
223+
224+========================= =====================================================
225+result-parameter information
226+========================= =====================================================
227+template is set to ``torrent.html``
228+url string, url of the result
229+title string, title of the result
230+content string, general result-text
231+publishedDate :py:class:`datetime.datetime`,
232+ time of publish *(not implemented yet)*
233+seed int, number of seeder
234+leech int, number of leecher
235+filesize int, size of file in bytes
236+files int, number of files
237+magnetlink string, magnetlink_ of the result
238+torrentfile string, torrentfile of the result
239+========================= =====================================================
274240
275241
276242map
277−~~~
278−
279−+-------------------------+--------------------------------------------------------------------------------------------------------------+
280−| result-parameter | information |
281−+=========================+==============================================================================================================+
282−| url | string, url of the result |
283−+-------------------------+--------------------------------------------------------------------------------------------------------------+
284−| title | string, title of the result |
285−+-------------------------+--------------------------------------------------------------------------------------------------------------+
286−| content | string, general result-text |
287−+-------------------------+--------------------------------------------------------------------------------------------------------------+
288−| publishedDate | :py:class:`datetime.datetime`, time of publish |
289−+-------------------------+--------------------------------------------------------------------------------------------------------------+
290−| latitude | latitude of result (in decimal format) |
291−+-------------------------+--------------------------------------------------------------------------------------------------------------+
292−| longitude | longitude of result (in decimal format) |
293−+-------------------------+--------------------------------------------------------------------------------------------------------------+
294−| boundingbox | boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) |
295−+-------------------------+--------------------------------------------------------------------------------------------------------------+
296−| geojson | geojson of result (http://geojson.org) |
297−+-------------------------+--------------------------------------------------------------------------------------------------------------+
298−| osm.type | type of osm-object (if OSM-Result) |
299−+-------------------------+--------------------------------------------------------------------------------------------------------------+
300−| osm.id | id of osm-object (if OSM-Result) |
301−+-------------------------+--------------------------------------------------------------------------------------------------------------+
302−| address.name | name of object |
303−+-------------------------+--------------------------------------------------------------------------------------------------------------+
304−| address.road | street name of object |
305−+-------------------------+--------------------------------------------------------------------------------------------------------------+
306−| address.house\_number | house number of object |
307−+-------------------------+--------------------------------------------------------------------------------------------------------------+
308−| address.locality | city, place of object |
309−+-------------------------+--------------------------------------------------------------------------------------------------------------+
310−| address.postcode | postcode of object |
311−+-------------------------+--------------------------------------------------------------------------------------------------------------+
312−| address.country | country of object |
313−+-------------------------+--------------------------------------------------------------------------------------------------------------+
314−
243+---
244+
245+========================= =====================================================
246+result-parameter information
247+========================= =====================================================
248+url string, url of the result
249+title string, title of the result
250+content string, general result-text
251+publishedDate :py:class:`datetime.datetime`, time of publish
252+latitude latitude of result (in decimal format)
253+longitude longitude of result (in decimal format)
254+boundingbox boundingbox of result (array of 4. values
255+ ``[lat-min, lat-max, lon-min, lon-max]``)
256+geojson geojson of result (http://geojson.org)
257+osm.type type of osm-object (if OSM-Result)
258+osm.id id of osm-object (if OSM-Result)
259+address.name name of object
260+address.road street name of object
261+address.house_number house number of object
262+address.locality city, place of object
263+address.postcode postcode of object
264+address.country country of object
265+========================= =====================================================
docs/dev/plugins.rst+24−21
@@ -1,45 +1,48 @@
1+=======
12Plugins
2−-------
3+=======
34
4−Plugins can extend or replace functionality of various components of
5−searx.
5+Plugins can extend or replace functionality of various components of searx.
66
77Example plugin
8−~~~~~~~~~~~~~~
8+==============
99
1010.. code:: python
1111
12− name = 'Example plugin'
13− description = 'This plugin extends the suggestions with the word "example"'
14− default_on = False # disabled by default
12+ name = 'Example plugin'
13+ description = 'This plugin extends the suggestions with the word "example"'
14+ default_on = False # disabled by default
1515
16− js_dependencies = tuple() # optional, list of static js files
17− css_dependencies = tuple() # optional, list of static css files
16+ js_dependencies = tuple() # optional, list of static js files
17+ css_dependencies = tuple() # optional, list of static css files
1818
1919
20− # attach callback to the post search hook
21− # request: flask request object
22− # ctx: the whole local context of the post search hook
23− def post_search(request, ctx):
24− ctx['search'].suggestions.add('example')
25− return True
20+ # attach callback to the post search hook
21+ # request: flask request object
22+ # ctx: the whole local context of the post search hook
23+ def post_search(request, ctx):
24+ ctx['search'].suggestions.add('example')
25+ return True
2626
2727Plugin entry points
28−~~~~~~~~~~~~~~~~~~~
28+===================
2929
30−Entry points (hooks) define when a plugin runs. Right now only three hooks are implemented. So feel free to implement a hook if it fits the behaviour of your plugin.
30+Entry points (hooks) define when a plugin runs. Right now only three hooks are
31+implemented. So feel free to implement a hook if it fits the behaviour of your
32+plugin.
3133
3234Pre search hook
33−```````````````
35+---------------
3436
3537Runs BEFORE the search request. Function to implement: ``pre_search``
3638
3739Post search hook
38−````````````````
40+----------------
3941
4042Runs AFTER the search request. Function to implement: ``post_search``
4143
4244Result hook
43−```````````
45+-----------
4446
45−Runs when a new result is added to the result list. Function to implement: ``on_result``
47+Runs when a new result is added to the result list. Function to implement:
48+``on_result``
docs/dev/quickstart.rst+47−45
@@ -1,17 +1,18 @@
11.. _devquickstart:
22
3+======================
34Development Quickstart
4−----------------------
5+======================
56
6−This quickstart guide gets your environment set up with searx. Furthermore, it gives a
7−short introduction to the new manage.sh script.
7+This quickstart guide gets your environment set up with searx. Furthermore, it
8+gives a short introduction to the ``manage.sh`` script.
89
910How to setup your development environment
10−~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
11+=========================================
1112
12−First, clone the source code of searx to the desired folder. In this case the source
13−is cloned to ~/myprojects/searx. Then create and activate the searx-ve
14−virtualenv and install the required packages using manage.sh.
13+First, clone the source code of searx to the desired folder. In this case the
14+source is cloned to ``~/myprojects/searx``. Then create and activate the
15+searx-ve virtualenv and install the required packages using manage.sh.
1516
1617.. code:: sh
1718
@@ -24,85 +25,86 @@ virtualenv and install the required packages using manage.sh.
2425
2526
2627How to run tests
27−~~~~~~~~~~~~~~~~
28+================
2829
29−Tests can be run using the manage.sh script.
30−
31−Following tests and checks are available:
30+Tests can be run using the ``manage.sh`` script. Following tests and checks are
31+available:
3232
3333- Unit tests
34−
3534- Selenium tests
36−
3735- PEP8 validation
38−
3936- Unit test coverage check
4037
4138For example unit tests are run with the command below:
4239
4340.. code:: sh
4441
45− ./manage.sh unit_tests
42+ ./manage.sh unit_tests
4643
47−For further test options, please consult the help of the manage.sh script.
44+For further test options, please consult the help of the ``manage.sh`` script.
4845
4946
5047How to compile styles and javascript
51−~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
48+====================================
49+
50+.. _less: http://lesscss.org/
51+.. _NodeJS: https://nodejs.org
5252
5353How to build styles
54−^^^^^^^^^^^^^^^^^^^
54+-------------------
5555
56−Less is required to build the styles of searx. Less can be installed using either NodeJS or Apt.
56+Less_ is required to build the styles of searx. Less_ can be installed using
57+either NodeJS_ or Apt.
5758
5859.. code:: sh
5960
60− sudo apt-get install nodejs
61− sudo npm install -g less
62−
61+ sudo -H apt-get install nodejs
62+ sudo -H npm install -g less
6363
6464OR
6565
6666.. code:: sh
6767
68− sudo apt-get install node-less
68+ sudo -H apt-get install node-less
6969
70−After satisfying the requirements styles can be build using manage.sh
70+After satisfying the requirements styles can be build using ``manage.sh``
7171
7272.. code:: sh
7373
74− ./manage.sh styles
74+ ./manage.sh styles
7575
7676
7777How to build the source of the oscar theme
78−^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
78+==========================================
7979
80−Grunt must be installed in order to build the javascript sources. It depends on NodeJS, so first
81−Node has to be installed.
80+.. _grunt: https://gruntjs.com/
81+
82+Grunt_ must be installed in order to build the javascript sources. It depends on
83+NodeJS, so first Node has to be installed.
8284
8385.. code:: sh
8486
85− sudo apt-get install nodejs
86− sudo npm install -g grunt-cli
87+ sudo -H apt-get install nodejs
88+ sudo -H npm install -g grunt-cli
8789
88−After installing grunt, the files can be built using the following command:
90+After installing grunt, the files can be built using the following command:
8991
9092.. code:: sh
9193
92− ./manage.sh grunt_build
93−
94+ ./manage.sh grunt_build
9495
9596
9697Tips for debugging/development
97−~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
98−
99−1. Turn on debug logging
100− Whether you are working on a new engine or trying to eliminate a bug, it is always a good idea
101− to turn on debug logging. When debug logging is enabled a stack trace appears,
102− instead of the cryptic ``Internal Server Error`` message. It can be turned on by setting
103− ``debug: False`` to ``debug: True`` in settings.yml.
104−
105−2. Run ``./manage.sh tests`` before creating a PR.
106− Failing build on Travis is common because of PEP8 checks. So a new commit must be created
107− containing these format fixes. This phase can be skipped if ``./manage.sh tests`` is run
108− locally before creating a PR.
98+==============================
99+
100+Turn on debug logging
101+ Whether you are working on a new engine or trying to eliminate a bug, it is
102+ always a good idea to turn on debug logging. When debug logging is enabled a
103+ stack trace appears, instead of the cryptic ``Internal Server Error``
104+ message. It can be turned on by setting ``debug: False`` to ``debug: True`` in
105+ :origin:`settings.yml <searx/settings.yml>`.
106+
107+Run ``./manage.sh tests`` before creating a PR.
108+ Failing build on Travis is common because of PEP8 checks. So a new commit
109+ must be created containing these format fixes. This phase can be skipped if
110+ ``./manage.sh tests`` is run locally before creating a PR.
docs/dev/search_api.rst+67−159
@@ -1,3 +1,4 @@
1+==========
12Search API
23==========
34
@@ -11,194 +12,101 @@ Furthermore, two enpoints ``/`` and ``/search`` are available for querying.
1112``GET /search``
1213
1314Parameters
14−~~~~~~~~~~
15−
16−.. code:: sh
17−
18− q
19−
20−The search query. This string is passed to external search services.
21−Thus, searx supports syntax of each search service. For example, ``site:github.com searx`` is a valid
22−query for Google. However, if simply the query above is passed to any search engine which does not filter its
23−results based on this syntax, you might not get the results you wanted.
24−
25−
26−See more at :doc:`/user/search_syntax`
27−
28−Required.
29−
30−.. code:: sh
31−
32− categories
33−
34−Comma separated list, specifies the active search categories
35−
36−Optional.
37−
38−.. code:: sh
39−
40− engines
41−
42−Comma separated list, specifies the active search engines.
43−
44−Optional.
45−
46−.. code:: sh
47−
48− lang
49−
50−Code of the language.
51−
52−Optional.
53−
54−Default: ``all``
55−
56−.. code:: sh
57−
58− pageno
59−
60−Search page number.
61−
62−Optional.
63−
64−Default: ``1``
65−
66−.. code:: sh
67−
68− time_range
69−
70−Time range of search for engines which support it. See if an engine supports time range search in the preferences page of an instance.
71−
72−Optional.
73−
74−Possible: ``day``, ``month``, ``year``
75−
76−.. code:: sh
77−
78− format
79−
80−Output format of results.
81−
82−Optional.
83−
84−Possible: ``json``, ``csv``, ``rss``
85−
86−.. code:: sh
87−
88− results_on_new_tab
89−
90−Open search results on new tab.
91−
92−Optional.
93−
94−Default: ``0``
95−
96−Possible: ``0``, ``1``
97−
98−.. code:: sh
99−
100− image_proxy
101−
102−Proxy image results through searx.
103−
104−Optional.
105−
106−Default: ``False``
107−
108−Possible: ``True``, ``False``
109−
110−.. code:: sh
111−
112− autocomplete
113−
114−Service which completes words as you type.
115−
116−Optional.
117−
118−Default: empty
119−
120−Possible: ``google``, ``dbpedia``, ``duckduckgo``, ``startpage``, ``wikipedia``
121−
122−.. code:: sh
123−
124− safesearch
125−
126−Filter search results of engines which support safe search. See if an engine supports safe search in the preferences page of an instance.
127−
128−Optional.
129−
130−Default: ``None``
131−
132−Possible: ``0``, ``1``, ``None``
133−
134−.. code:: sh
135−
136− theme
137−
138−Theme of instance.
139−
140−Optional.
15+==========
14116
142−Default: ``oscar``
17+``q`` : required
18+ The search query. This string is passed to external search services. Thus,
19+ searx supports syntax of each search service. For example, ``site:github.com
20+ searx`` is a valid query for Google. However, if simply the query above is
21+ passed to any search engine which does not filter its results based on this
22+ syntax, you might not get the results you wanted.
14323
144−Possible: ``oscar``, ``simple``, ``legacy``, ``pix-art``, ``courgette``
24+ See more at :ref:`search-syntax`
14525
146−Please note, available themes depend on an instance. It is possible that an instance administrator deleted, created or renamed themes on his/her instance. See the available options in the preferences page of the instance.
26+``categories`` : optional
27+ Comma separated list, specifies the active search categories
14728
148−.. code:: sh
29+``engines``: optional
30+ Comma separated list, specifies the active search engines.
14931
150− oscar-style
32+``lang``: default ``all``
33+ Code of the language.
15134
152−Style of Oscar theme. It is only parsed if the theme of an instance is ``oscar``.
35+``pageno``: default ``1``
36+ Search page number.
15337
154−Optional.
38+``time_range``: optional
39+ [ ``day``, ``month``, ``year`` ]
15540
156−Default: ``logicodev``
41+ Time range of search for engines which support it. See if an engine supports
42+ time range search in the preferences page of an instance.
15743
158−Possible: ``pointhi``, ``logicodev``
44+``format``: optional
45+ [ ``json``, ``csv``, ``rss`` ]
15946
160−Please note, available styles depend on an instance. It is possible that an instance administrator deleted, created or renamed styles on his/her instance. See the available options in the preferences page of the instance.
47+ Output format of results.
16148
162−.. code:: sh
49+``results_on_new_tab``: default ``0``
50+ [ ``0``, ``1`` ]
16351
164− enabled_plugins
52+ Open search results on new tab.
16553
166−List of enabled plugins.
54+``image_proxy``: default ``False``
55+ [ ``True``, ``False`` ]
16756
168−Optional.
57+ Proxy image results through searx.
16958
170−Default: ``HTTPS_rewrite``, ``Self_Informations``, ``Search_on_category_select``, ``Tracker_URL_remover``
59+``autocomplete``: default *empty*
60+ [ ``google``, ``dbpedia``, ``duckduckgo``, ``startpage``, ``wikipedia`` ]
17161
172−Possible: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, ``Search_on_category_select``
62+ Service which completes words as you type.
17363
174−.. code:: sh
64+``safesearch``: default ``None``
65+ [ ``0``, ``1``, ``None`` ]
17566
176− disabled_plugins
67+ Filter search results of engines which support safe search. See if an engine
68+ supports safe search in the preferences page of an instance.
17769
178−List of disabled plugins.
70+``theme``: default ``oscar``
71+ [ ``oscar``, ``simple``, ``legacy``, ``pix-art``, ``courgette`` ]
17972
180−Optional.
73+ Theme of instance.
18174
182−Default: ``DOAI_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys``
75+ Please note, available themes depend on an instance. It is possible that an
76+ instance administrator deleted, created or renamed themes on his/her instance.
77+ See the available options in the preferences page of the instance.
18378
184−Possible: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``, ``Search_on_category_select``
79+``oscar-style``: default ``logicodev``
80+ [ ``pointhi``, ``logicodev`` ]
18581
186−.. code:: sh
82+ Style of Oscar theme. It is only parsed if the theme of an instance is
83+ ``oscar``.
18784
188− enabled_engines
85+ Please note, available styles depend on an instance. It is possible that an
86+ instance administrator deleted, created or renamed styles on his/her
87+ instance. See the available options in the preferences page of the instance.
18988
190−List of enabled engines.
89+``enabled_plugins``: optional
90+ List of enabled plugins.
19191
192−Optional.
92+ :default: ``HTTPS_rewrite``, ``Self_Informations``,
93+ ``Search_on_category_select``, ``Tracker_URL_remover``
19394
194−Possible: all engines
95+ :values: [ ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``,
96+ ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``,
97+ ``Search_on_category_select`` ]
19598
196−.. code:: sh
99+``disabled_plugins``: optional
100+ List of disabled plugins.
197101
198− disabled_engines
102+ :default: ``DOAI_rewrite``, ``Infinite_scroll``, ``Vim-like_hotkeys``
103+ :values: ``DOAI_rewrite``, ``HTTPS_rewrite``, ``Infinite_scroll``,
104+ ``Vim-like_hotkeys``, ``Self_Informations``, ``Tracker_URL_remover``,
105+ ``Search_on_category_select``
199106
200−List of disabled engines.
107+``enabled_engines``: optional : *all* :origin:`engines <searx/engines>`
108+ List of enabled engines.
201109
202−Optional.
110+``disabled_engines``: optional : *all* :origin:`engines <searx/engines>`
111+ List of disabled engines.
203112
204−Possible: all engines
docs/dev/translation.rst+27−17
@@ -1,28 +1,36 @@
1+.. _translation:
2+
3+===========
14Translation
25===========
36
4−Requirements
5−------------
7+.. _searx@transifex: https://www.transifex.com/asciimoo/searx/
8+
9+Translation currently takes place on `searx@transifex`_
610
7− * Transifex account
11+Requirements
12+============
813
9− * Installed CLI tool of Transifex
14+* Transifex account
15+* Installed CLI tool of Transifex
1016
1117Init Transifex project
12−----------------------
18+======================
1319
14−After installing ``transifex`` using pip, run the following command to initialize the project.
20+After installing ``transifex`` using pip, run the following command to
21+initialize the project.
1522
16−.. code:: shell
23+.. code:: sh
1724
18− tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/
25+ tx init # Transifex instance: https://www.transifex.com/asciimoo/searx/
1926
2027
21−After ``$HOME/.transifexrc`` is created, get a Transifex API key and insert it into the configuration file.
28+After ``$HOME/.transifexrc`` is created, get a Transifex API key and insert it
29+into the configuration file.
2230
2331Create a configuration file for ``tx`` named ``$HOME/.tx/config``.
2432
25−.. code:: shell
33+.. code:: ini
2634
2735 [main]
2836 host = https://www.transifex.com
@@ -42,20 +50,22 @@ Then run ``tx set``:
4250
4351
4452Update translations
45−-------------------
53+===================
4654
4755To retrieve the latest translations, pull it from Transifex.
4856
49−.. code:: shell
57+.. code:: sh
5058
51− tx pull -a
59+ tx pull -a
5260
53−Then check the new languages. If strings translated are not enough, delete those folders, because
54−those should not be compiled. Call the command below to compile the ``.po`` files.
61+Then check the new languages. If strings translated are not enough, delete those
62+folders, because those should not be compiled. Call the command below to compile
63+the ``.po`` files.
5564
5665.. code:: shell
5766
58− pybabel compile -d searx/translations
67+ pybabel compile -d searx/translations
5968
6069
61−After the compilation is finished commit the ``.po`` and ``.mo`` files and create a PR.
70+After the compilation is finished commit the ``.po`` and ``.mo`` files and
71+create a PR.
docs/index.rst+18−16
@@ -3,22 +3,24 @@ Welcome to searx
33
44Search without being tracked.
55
6−Searx is a free internet metasearch engine which aggregates results from more than 70 search services. Users are neither tracked nor profiled. Additionally, searx can be used over Tor for online anonymity.
7−
8−Get started with searx by using one of the :wiki:`Searx-instances`. If you don't trust anyone, you can set up your own, see :ref:`installation`.
9−
10−
11−Features
12−--------
13−
14− - Self hosted
15− - No user tracking
16− - No user profiling
17− - About 70 supported search engines
18− - Easy integration with any search engine
19− - Cookies are not used by default
20− - Secure, encrypted connections (HTTPS/SSL)
21− - Hosted by organisations, such as La Quadrature du Net, which promote digital rights
6+.. sidebar:: Features
7+
8+ - Self hosted
9+ - No user tracking
10+ - No user profiling
11+ - About 70 supported search engines
12+ - Easy integration with any search engine
13+ - Cookies are not used by default
14+ - Secure, encrypted connections (HTTPS/SSL)
15+ - Hosted by organizations, such as *La Quadrature du Net*, which promote
16+ digital rights
17+
18+Searx is a free internet metasearch engine which aggregates results from more
19+than 70 search services. Users are neither tracked nor profiled. Additionally,
20+searx can be used over Tor for online anonymity.
21+
22+Get started with searx by using one of the :wiki:`Searx-instances`. If you
23+don't trust anyone, you can set up your own, see :ref:`installation`.
2224
2325
2426User documentation
docs/user/own-instance.rst+52−18
@@ -1,43 +1,77 @@
1+===========================
12Why use a private instance?
23===========================
34
4−"Is it worth to run my own instance?" is a common question among searx users. Before answering this question, see what options a searx user has.
5+"Is it worth to run my own instance?" is a common question among searx users.
6+Before answering this question, see what options a searx user has.
57
6−Public instances are open to everyone who has access to its URL. Usually, these are operated by unknown parties (from the users' point of view). Private instances can be used by a select group of people. It is for example a searx of group of friends or a company which can be accessed through VPN. Also it can be single user one which runs on the user's laptop.
8+Public instances are open to everyone who has access to its URL. Usually, these
9+are operated by unknown parties (from the users' point of view). Private
10+instances can be used by a select group of people. It is for example a searx of
11+group of friends or a company which can be accessed through VPN. Also it can be
12+single user one which runs on the user's laptop.
713
8−To gain more insight on how these instances work let's dive into how searx protects its users.
14+To gain more insight on how these instances work let's dive into how searx
15+protects its users.
916
1017How does searx protect privacy?
11−-------------------------------
18+===============================
1219
13−Searx protects the privacy of its users in multiple ways regardless of the type of the instance (private, public). Removal of private data from search requests comes in three forms:
20+Searx protects the privacy of its users in multiple ways regardless of the type
21+of the instance (private, public). Removal of private data from search requests
22+comes in three forms:
1423
1524 1. removal of private data from requests going to search services
16− 2. not forwarding anything from a third party services through search services (e.g. advertisement)
25+ 2. not forwarding anything from a third party services through search services
26+ (e.g. advertisement)
1727 3. removal of private data from requests going to the result pages
1828
19−Removing private data means not sending cookies to external search engines and generating a random browser profile for every request. Thus, it does not matter if a public or private instance handles the request, because it is anonymized in both cases. IP addresses will be the IP of the instance. But searx can be configured to use proxy or Tor. `Result proxy <https://github.com/asciimoo/morty>`__ is supported, too.
29+Removing private data means not sending cookies to external search engines and
30+generating a random browser profile for every request. Thus, it does not matter
31+if a public or private instance handles the request, because it is anonymized in
32+both cases. IP addresses will be the IP of the instance. But searx can be
33+configured to use proxy or Tor. `Result proxy
34+<https://github.com/asciimoo/morty>`__ is supported, too.
35+
36+Searx does not serve ads or tracking content unlike most search services. So
37+private data is not forwarded to third parties who might monetize it. Besides
38+protecting users from search services, both referring page and search query are
39+hidden from visited result pages.
2040
21−Searx does not serve ads or tracking content unlike most search services. So private data is not forwarded to third parties who might monetize it. Besides protecting users from search services, both referring page and search query are hidden from visited result pages.
2241
2342What are the consequences of using public instances?
24−^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
43+----------------------------------------------------
2544
26−If someone uses a public instance, he/she has to trust the administrator of that instance.
27−This means that the user of the public instance does not know whether his/her requests are logged, aggregated and sent or sold to a third party.
45+If someone uses a public instance, he/she has to trust the administrator of that
46+instance. This means that the user of the public instance does not know whether
47+his/her requests are logged, aggregated and sent or sold to a third party.
2848
29−Also, public instances without proper protection are more vulnerable to abusing the search service, In this case the external service in exchange returns CAPTCHAs or bans the IP of the instance. Thus, search requests return less results.
49+Also, public instances without proper protection are more vulnerable to abusing
50+the search service, In this case the external service in exchange returns
51+CAPTCHAs or bans the IP of the instance. Thus, search requests return less
52+results.
3053
3154I see. What about private instances?
32−^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
55+------------------------------------
3356
34−If users run their own instances, everything is in their control: the source code, logging settings and private data. Unknown instance administrators do not have to be trusted.
57+If users run their own instances, everything is in their control: the source
58+code, logging settings and private data. Unknown instance administrators do not
59+have to be trusted.
3560
36−Furthermore, as the default settings of their instance is editable, there is no need to use cookies to tailor searx to their needs. So preferences will not be reset to defaults when clearing browser cookies. As settings are stored on their computer, it will not be accessible to others as long as their computer is not compromised.
61+Furthermore, as the default settings of their instance is editable, there is no
62+need to use cookies to tailor searx to their needs. So preferences will not be
63+reset to defaults when clearing browser cookies. As settings are stored on
64+their computer, it will not be accessible to others as long as their computer is
65+not compromised.
3766
3867Conclusion
39−----------
68+==========
4069
41−Always use an instance which is operated by people you trust. The privacy features of searx are available to users no matter what kind of instance they use.
70+Always use an instance which is operated by people you trust. The privacy
71+features of searx are available to users no matter what kind of instance they
72+use.
4273
43−If someone is on the go or just wants to try searx for the first time public instances are the best choices. Additionally, public instance are making a world a better place, because those who cannot or do not want to run an instance, have access to a privacy respecting search service.
74+If someone is on the go or just wants to try searx for the first time public
75+instances are the best choices. Additionally, public instance are making a
76+world a better place, because those who cannot or do not want to run an
77+instance, have access to a privacy respecting search service.
docs/user/search_syntax.rst+7−3
@@ -1,8 +1,12 @@
1+
2+.. _search-syntax:
3+
4+=============
15Search syntax
26=============
37
4−Searx allows you to modify the default categories, engines and search
5−language via the search query.
8+Searx allows you to modify the default categories, engines and search language
9+via the search query.
610
711Prefix ``!``
812 to set Category/engine
@@ -22,7 +26,7 @@ See the :search:`/preferences page <preferences>` for the list of engines,
2226categories and languages.
2327
2428Examples
25−~~~~~~~~
29+========
2630
2731Search in wikipedia for ``qwer``:
2832