on 60f36cc help sign in
searx/docs/dev/engine_overview.rst
313 lines 25 kB history raw
1Engine overview
2===============
5searx is a `metasearch-engine <https://en.wikipedia.org/wiki/Metasearch_engine>`__,
6so it uses different search engines to provide better results.
8Because there is no general search API which could be used for every
9search engine, an adapter has to be built between searx and the
10external search engines. Adapters are stored under the folder
11`searx/engines
12<https://github.com/asciimoo/searx/tree/master/searx/engines>`__.
15.. contents::
16 :depth: 3
18general engine configuration
19----------------------------
21It is required to tell searx the type of results the engine provides. The
22arguments can be set in the engine file or in the settings file
23(normally ``settings.yml``). The arguments in the settings file override
24the ones in the engine file.
26It does not matter if an option is stored in the engine file or in the
27settings. However, the standard way is the following:
30engine file
31~~~~~~~~~~~
33+----------------------+-----------+-----------------------------------------+
34| argument | type | information |
35+======================+===========+=========================================+
36| categories | list | pages, in which the engine is working |
37+----------------------+-----------+-----------------------------------------+
38| paging | boolean | support multible pages |
39+----------------------+-----------+-----------------------------------------+
40| language\_support | boolean | support language choosing |
41+----------------------+-----------+-----------------------------------------+
42| time\_range\_support | boolean | support search time range |
43+----------------------+-----------+-----------------------------------------+
45settings.yml
46~~~~~~~~~~~~
48+------------+----------+-----------------------------------------------+
49| argument | type | information |
50+============+==========+===============================================+
51| name | string | name of search-engine |
52+------------+----------+-----------------------------------------------+
53| engine | string | name of searx-engine (filename without .py) |
54+------------+----------+-----------------------------------------------+
55| shortcut | string | shortcut of search-engine |
56+------------+----------+-----------------------------------------------+
57| timeout | string | specific timeout for search-engine |
58+------------+----------+-----------------------------------------------+
60overrides
61~~~~~~~~~
63A few of the options have default values in the engine, but are
64often overwritten by the settings. If ``None`` is assigned to an option
65in the engine file, it has to be redefined in the settings,
66otherwise searx will not start with that engine.
68The naming of overrides is arbitrary. But the recommended
69overrides are the following:
71+-----------------------+----------+----------------------------------------------------------------+
72| argument | type | information |
73+=======================+==========+================================================================+
74| base\_url | string | base-url, can be overwritten to use same engine on other URL |
75+-----------------------+----------+----------------------------------------------------------------+
76| number\_of\_results | int | maximum number of results per request |
77+-----------------------+----------+----------------------------------------------------------------+
78| language | string | ISO code of language and country like en\_US |
79+-----------------------+----------+----------------------------------------------------------------+
80| api\_key | string | api-key if required by engine |
81+-----------------------+----------+----------------------------------------------------------------+
83example code
84~~~~~~~~~~~~
86.. code:: python
88 # engine dependent config
89 categories = ['general']
90 paging = True
91 language_support = True
93making a request
94----------------
96To perform a search an URL have to be specified. In addition to
97specifying an URL, arguments can be passed to the query.
99passed arguments
100~~~~~~~~~~~~~~~~
102These arguments can be used to construct the search query. Furthermore,
103parameters with default value can be redefined for special purposes.
105+----------------------+------------+------------------------------------------------------------------------+
106| argument | type | default-value, information |
107+======================+============+========================================================================+
108| url | string | ``''`` |
109+----------------------+------------+------------------------------------------------------------------------+
110| method | string | ``'GET'`` |
111+----------------------+------------+------------------------------------------------------------------------+
112| headers | set | ``{}`` |
113+----------------------+------------+------------------------------------------------------------------------+
114| data | set | ``{}`` |
115+----------------------+------------+------------------------------------------------------------------------+
116| cookies | set | ``{}`` |
117+----------------------+------------+------------------------------------------------------------------------+
118| verify | boolean | ``True`` |
119+----------------------+------------+------------------------------------------------------------------------+
120| headers.User-Agent | string | a random User-Agent |
121+----------------------+------------+------------------------------------------------------------------------+
122| category | string | current category, like ``'general'`` |
123+----------------------+------------+------------------------------------------------------------------------+
124| started | datetime | current date-time |
125+----------------------+------------+------------------------------------------------------------------------+
126| pageno | int | current pagenumber |
127+----------------------+------------+------------------------------------------------------------------------+
128| language | string | specific language code like ``'en_US'``, or ``'all'`` if unspecified |
129+----------------------+------------+------------------------------------------------------------------------+
131parsed arguments
132~~~~~~~~~~~~~~~~
134The function ``def request(query, params):`` always returns the
135``params`` variable. Inside searx, the following paramters can be
136used to specify a search request:
138+------------+-----------+---------------------------------------------------------+
139| argument | type | information |
140+============+===========+=========================================================+
141| url | string | requested url |
142+------------+-----------+---------------------------------------------------------+
143| method | string | HTTP request method |
144+------------+-----------+---------------------------------------------------------+
145| headers | set | HTTP header information |
146+------------+-----------+---------------------------------------------------------+
147| data | set | HTTP data information (parsed if ``method != 'GET'``) |
148+------------+-----------+---------------------------------------------------------+
149| cookies | set | HTTP cookies |
150+------------+-----------+---------------------------------------------------------+
151| verify | boolean | Performing SSL-Validity check |
152+------------+-----------+---------------------------------------------------------+
154example code
155~~~~~~~~~~~~
157.. code:: python
159 # search-url
160 base_url = 'https://example.com/'
161 search_string = 'search?{query}&page={page}'
163 # do search-request
164 def request(query, params):
165 search_path = search_string.format(
166 query=urlencode({'q': query}),
167 page=params['pageno'])
169 params['url'] = base_url + search_path
171 return params
173returned results
174----------------
176Searx is able to return results of different media-types.
177Currently the following media-types are supported:
179- default
180- images
181- videos
182- torrent
183- map
185To set another media-type as default, the parameter
186``template`` must be set to the desired type.
188default
189~~~~~~~
191+--------------------+---------------------------------------------------------------------------------------------------------------+
192| result-parameter | information |
193+====================+===============================================================================================================+
194| url | string, url of the result |
195+--------------------+---------------------------------------------------------------------------------------------------------------+
196| title | string, title of the result |
197+--------------------+---------------------------------------------------------------------------------------------------------------+
198| content | string, general result-text |
199+--------------------+---------------------------------------------------------------------------------------------------------------+
200| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish |
201+--------------------+---------------------------------------------------------------------------------------------------------------+
203images
204~~~~~~
206to use this template, the parameter
208+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
209| result-parameter | information |
210+====================+=======================================================================================================================================+
211| template | is set to ``images.html`` |
212+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
213| url | string, url to the result site |
214+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
215| title | string, title of the result *(partly implemented)* |
216+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
217| content | *(partly implemented)* |
218+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
219| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish *(partly implemented)* |
220+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
221| img\_src | string, url to the result image |
222+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
223| thumbnail\_src | string, url to a small-preview image |
224+--------------------+---------------------------------------------------------------------------------------------------------------------------------------+
226videos
227~~~~~~
229+--------------------+--------------------------------------------------------------------------------------------------------------+
230| result-parameter | information |
231+====================+==============================================================================================================+
232| template | is set to ``videos.html`` |
233+--------------------+--------------------------------------------------------------------------------------------------------------+
234| url | string, url of the result |
235+--------------------+--------------------------------------------------------------------------------------------------------------+
236| title | string, title of the result |
237+--------------------+--------------------------------------------------------------------------------------------------------------+
238| content | *(not implemented yet)* |
239+--------------------+--------------------------------------------------------------------------------------------------------------+
240| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish |
241+--------------------+--------------------------------------------------------------------------------------------------------------+
242| thumbnail | string, url to a small-preview image |
243+--------------------+--------------------------------------------------------------------------------------------------------------+
245torrent
246~~~~~~~
248+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
249| result-parameter | information |
250+==================+=======================================================================================================================================+
251| template | is set to ``torrent.html`` |
252+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
253| url | string, url of the result |
254+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
255| title | string, title of the result |
256+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
257| content | string, general result-text |
258+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
259| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish *(not implemented yet)* |
260+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
261| seed | int, number of seeder |
262+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
263| leech | int, number of leecher |
264+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
265| filesize | int, size of file in bytes |
266+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
267| files | int, number of files |
268+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
269| magnetlink | string, `magnetlink <https://en.wikipedia.org/wiki/Magnet_URI_scheme>`__ of the result |
270+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
271| torrentfile | string, torrentfile of the result |
272+------------------+---------------------------------------------------------------------------------------------------------------------------------------+
275map
276~~~
278+-------------------------+--------------------------------------------------------------------------------------------------------------+
279| result-parameter | information |
280+=========================+==============================================================================================================+
281| url | string, url of the result |
282+-------------------------+--------------------------------------------------------------------------------------------------------------+
283| title | string, title of the result |
284+-------------------------+--------------------------------------------------------------------------------------------------------------+
285| content | string, general result-text |
286+-------------------------+--------------------------------------------------------------------------------------------------------------+
287| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, time of publish |
288+-------------------------+--------------------------------------------------------------------------------------------------------------+
289| latitude | latitude of result (in decimal format) |
290+-------------------------+--------------------------------------------------------------------------------------------------------------+
291| longitude | longitude of result (in decimal format) |
292+-------------------------+--------------------------------------------------------------------------------------------------------------+
293| boundingbox | boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) |
294+-------------------------+--------------------------------------------------------------------------------------------------------------+
295| geojson | geojson of result (http://geojson.org) |
296+-------------------------+--------------------------------------------------------------------------------------------------------------+
297| osm.type | type of osm-object (if OSM-Result) |
298+-------------------------+--------------------------------------------------------------------------------------------------------------+
299| osm.id | id of osm-object (if OSM-Result) |
300+-------------------------+--------------------------------------------------------------------------------------------------------------+
301| address.name | name of object |
302+-------------------------+--------------------------------------------------------------------------------------------------------------+
303| address.road | street name of object |
304+-------------------------+--------------------------------------------------------------------------------------------------------------+
305| address.house\_number | house number of object |
306+-------------------------+--------------------------------------------------------------------------------------------------------------+
307| address.locality | city, place of object |
308+-------------------------+--------------------------------------------------------------------------------------------------------------+
309| address.postcode | postcode of object |
310+-------------------------+--------------------------------------------------------------------------------------------------------------+
311| address.country | country of object |
312+-------------------------+--------------------------------------------------------------------------------------------------------------+