@@ -0,0 +1,315 @@
1+Engine overview
2+===============
3+
4+
5+searx is a `metasearch-engine <https://en.wikipedia.org/wiki/Metasearch_engine>`__,
6+so it uses different search engines to provide better results.
7+
8+Because there is no general search API which could be used for every
9+search engine, an adapter has to be built between searx and the
10+external search engines. Adapters are stored under the folder
11+`searx/engines
12+<https://github.com/asciimoo/searx/tree/master/searx/engines>`__.
13+
14+
15+.. contents::
16+ :depth: 3
17+
18+general engine configuration
19+----------------------------
20+
21+It is required to tell searx the type of results the engine provides. The
22+arguments can be set in the engine file or in the settings file
23+(normally ``settings.yml``). The arguments in the settings file override
24+the ones in the engine file.
25+
26+It does not matter if an option is stored in the engine file or in the
27+settings. However, the standard way is the following:
28+
29+
30+engine file
31+~~~~~~~~~~~
32+
33++----------------------+-----------+-----------------------------------------+
34+| argument | type | information |
35++======================+===========+=========================================+
36+| categories | list | pages, in which the engine is working |
37++----------------------+-----------+-----------------------------------------+
38+| paging | boolean | support multible pages |
39++----------------------+-----------+-----------------------------------------+
40+| language\_support | boolean | support language choosing |
41++----------------------+-----------+-----------------------------------------+
42+| time\_range\_support | boolean | support search time range |
43++----------------------+-----------+-----------------------------------------+
44+| offline | boolean | engine runs offline |
45++----------------------+-----------+-----------------------------------------+
46+
47+settings.yml
48+~~~~~~~~~~~~
49+
50++------------+----------+-----------------------------------------------+
51+| argument | type | information |
52++============+==========+===============================================+
53+| name | string | name of search-engine |
54++------------+----------+-----------------------------------------------+
55+| engine | string | name of searx-engine (filename without .py) |
56++------------+----------+-----------------------------------------------+
57+| shortcut | string | shortcut of search-engine |
58++------------+----------+-----------------------------------------------+
59+| timeout | string | specific timeout for search-engine |
60++------------+----------+-----------------------------------------------+
61+
62+overrides
63+~~~~~~~~~
64+
65+A few of the options have default values in the engine, but are
66+often overwritten by the settings. If ``None`` is assigned to an option
67+in the engine file, it has to be redefined in the settings,
68+otherwise searx will not start with that engine.
69+
70+The naming of overrides is arbitrary. But the recommended
71+overrides are the following:
72+
73++-----------------------+----------+----------------------------------------------------------------+
74+| argument | type | information |
75++=======================+==========+================================================================+
76+| base\_url | string | base-url, can be overwritten to use same engine on other URL |
77++-----------------------+----------+----------------------------------------------------------------+
78+| number\_of\_results | int | maximum number of results per request |
79++-----------------------+----------+----------------------------------------------------------------+
80+| language | string | ISO code of language and country like en\_US |
81++-----------------------+----------+----------------------------------------------------------------+
82+| api\_key | string | api-key if required by engine |
83++-----------------------+----------+----------------------------------------------------------------+
84+
85+example code
86+~~~~~~~~~~~~
87+
88+.. code:: python
89+
90+ # engine dependent config
91+ categories = ['general']
92+ paging = True
93+ language_support = True
94+
95+making a request
96+----------------
97+
98+To perform a search an URL have to be specified. In addition to
99+specifying an URL, arguments can be passed to the query.
100+
101+passed arguments
102+~~~~~~~~~~~~~~~~
103+
104+These arguments can be used to construct the search query. Furthermore,
105+parameters with default value can be redefined for special purposes.
106+
107++----------------------+------------+------------------------------------------------------------------------+
108+| argument | type | default-value, information |
109++======================+============+========================================================================+
110+| url | string | ``''`` |
111++----------------------+------------+------------------------------------------------------------------------+
112+| method | string | ``'GET'`` |
113++----------------------+------------+------------------------------------------------------------------------+
114+| headers | set | ``{}`` |
115++----------------------+------------+------------------------------------------------------------------------+
116+| data | set | ``{}`` |
117++----------------------+------------+------------------------------------------------------------------------+
118+| cookies | set | ``{}`` |
119++----------------------+------------+------------------------------------------------------------------------+
120+| verify | boolean | ``True`` |
121++----------------------+------------+------------------------------------------------------------------------+
122+| headers.User-Agent | string | a random User-Agent |
123++----------------------+------------+------------------------------------------------------------------------+
124+| category | string | current category, like ``'general'`` |
125++----------------------+------------+------------------------------------------------------------------------+
126+| started | datetime | current date-time |
127++----------------------+------------+------------------------------------------------------------------------+
128+| pageno | int | current pagenumber |
129++----------------------+------------+------------------------------------------------------------------------+
130+| language | string | specific language code like ``'en_US'``, or ``'all'`` if unspecified |
131++----------------------+------------+------------------------------------------------------------------------+
132+
133+parsed arguments
134+~~~~~~~~~~~~~~~~
135+
136+The function ``def request(query, params):`` always returns the
137+``params`` variable. Inside searx, the following paramters can be
138+used to specify a search request:
139+
140++------------+-----------+---------------------------------------------------------+
141+| argument | type | information |
142++============+===========+=========================================================+
143+| url | string | requested url |
144++------------+-----------+---------------------------------------------------------+
145+| method | string | HTTP request method |
146++------------+-----------+---------------------------------------------------------+
147+| headers | set | HTTP header information |
148++------------+-----------+---------------------------------------------------------+
149+| data | set | HTTP data information (parsed if ``method != 'GET'``) |
150++------------+-----------+---------------------------------------------------------+
151+| cookies | set | HTTP cookies |
152++------------+-----------+---------------------------------------------------------+
153+| verify | boolean | Performing SSL-Validity check |
154++------------+-----------+---------------------------------------------------------+
155+
156+example code
157+~~~~~~~~~~~~
158+
159+.. code:: python
160+
161+ # search-url
162+ base_url = 'https://example.com/'
163+ search_string = 'search?{query}&page={page}'
164+
165+ # do search-request
166+ def request(query, params):
167+ search_path = search_string.format(
168+ query=urlencode({'q': query}),
169+ page=params['pageno'])
170+
171+ params['url'] = base_url + search_path
172+
173+ return params
174+
175+returned results
176+----------------
177+
178+Searx is able to return results of different media-types.
179+Currently the following media-types are supported:
180+
181+- default
182+- images
183+- videos
184+- torrent
185+- map
186+
187+To set another media-type as default, the parameter
188+``template`` must be set to the desired type.
189+
190+default
191+~~~~~~~
192+
193++--------------------+---------------------------------------------------------------------------------------------------------------+
194+| result-parameter | information |
195++====================+===============================================================================================================+
196+| url | string, url of the result |
197++--------------------+---------------------------------------------------------------------------------------------------------------+
198+| title | string, title of the result |
199++--------------------+---------------------------------------------------------------------------------------------------------------+
200+| content | string, general result-text |
201++--------------------+---------------------------------------------------------------------------------------------------------------+
202+| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish |
203++--------------------+---------------------------------------------------------------------------------------------------------------+
204+
205+images
206+~~~~~~
207+
208+to use this template, the parameter
209+
210++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
211+| result-parameter | information |
212++====================+=======================================================================================================================================+
213+| template | is set to ``images.html`` |
214++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
215+| url | string, url to the result site |
216++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
217+| title | string, title of the result *(partly implemented)* |
218++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
219+| content | *(partly implemented)* |
220++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
221+| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish *(partly implemented)* |
222++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
223+| img\_src | string, url to the result image |
224++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
225+| thumbnail\_src | string, url to a small-preview image |
226++--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
227+
228+videos
229+~~~~~~
230+
231++--------------------+--------------------------------------------------------------------------------------------------------------+
232+| result-parameter | information |
233++====================+==============================================================================================================+
234+| template | is set to ``videos.html`` |
235++--------------------+--------------------------------------------------------------------------------------------------------------+
236+| url | string, url of the result |
237++--------------------+--------------------------------------------------------------------------------------------------------------+
238+| title | string, title of the result |
239++--------------------+--------------------------------------------------------------------------------------------------------------+
240+| content | *(not implemented yet)* |
241++--------------------+--------------------------------------------------------------------------------------------------------------+
242+| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish |
243++--------------------+--------------------------------------------------------------------------------------------------------------+
244+| thumbnail | string, url to a small-preview image |
245++--------------------+--------------------------------------------------------------------------------------------------------------+
246+
247+torrent
248+~~~~~~~
249+
250++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
251+| result-parameter | information |
252++==================+=======================================================================================================================================+
253+| template | is set to ``torrent.html`` |
254++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
255+| url | string, url of the result |
256++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
257+| title | string, title of the result |
258++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
259+| content | string, general result-text |
260++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
261+| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish *(not implemented yet)* |
262++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
263+| seed | int, number of seeder |
264++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
265+| leech | int, number of leecher |
266++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
267+| filesize | int, size of file in bytes |
268++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
269+| files | int, number of files |
270++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
271+| magnetlink | string, `magnetlink <https://en.wikipedia.org/wiki/Magnet_URI_scheme>`__ of the result |
272++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
273+| torrentfile | string, torrentfile of the result |
274++------------------+---------------------------------------------------------------------------------------------------------------------------------------+
275+
276+
277+map
278+~~~
279+
280++-------------------------+--------------------------------------------------------------------------------------------------------------+
281+| result-parameter | information |
282++=========================+==============================================================================================================+
283+| url | string, url of the result |
284++-------------------------+--------------------------------------------------------------------------------------------------------------+
285+| title | string, title of the result |
286++-------------------------+--------------------------------------------------------------------------------------------------------------+
287+| content | string, general result-text |
288++-------------------------+--------------------------------------------------------------------------------------------------------------+
289+| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish |
290++-------------------------+--------------------------------------------------------------------------------------------------------------+
291+| latitude | latitude of result (in decimal format) |
292++-------------------------+--------------------------------------------------------------------------------------------------------------+
293+| longitude | longitude of result (in decimal format) |
294++-------------------------+--------------------------------------------------------------------------------------------------------------+
295+| boundingbox | boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) |
296++-------------------------+--------------------------------------------------------------------------------------------------------------+
297+| geojson | geojson of result (http://geojson.org) |
298++-------------------------+--------------------------------------------------------------------------------------------------------------+
299+| osm.type | type of osm-object (if OSM-Result) |
300++-------------------------+--------------------------------------------------------------------------------------------------------------+
301+| osm.id | id of osm-object (if OSM-Result) |
302++-------------------------+--------------------------------------------------------------------------------------------------------------+
303+| address.name | name of object |
304++-------------------------+--------------------------------------------------------------------------------------------------------------+
305+| address.road | street name of object |
306++-------------------------+--------------------------------------------------------------------------------------------------------------+
307+| address.house\_number | house number of object |
308++-------------------------+--------------------------------------------------------------------------------------------------------------+
309+| address.locality | city, place of object |
310++-------------------------+--------------------------------------------------------------------------------------------------------------+
311+| address.postcode | postcode of object |
312++-------------------------+--------------------------------------------------------------------------------------------------------------+
313+| address.country | country of object |
314++-------------------------+--------------------------------------------------------------------------------------------------------------+
315+