git.iko.soy/searx
on 2fded29 help sign in
searx/docs/dev/engine_overview.rst
311 lines 27 kB history raw
1Engine overview
2===============
5searx is a `metasearch-engine <https://en.wikipedia.org/wiki/Metasearch_engine>`__,
6so it uses different search engines to provide better results.
8Because there is no general search API which could be used for every
9search engine, an adapter has to be built between searx and the
10external search engines. Adapters are stored under the folder
11`searx/engines
12<https://github.com/asciimoo/searx/tree/master/searx/engines>`__.
15.. contents::
16 :depth: 3
18general engine configuration
19----------------------------
21It is required to tell searx the type of results the engine provides. The
22arguments can be set in the engine file or in the settings file
23(normally ``settings.yml``). The arguments in the settings file override
24the ones in the engine file.
26It does not matter if an option is stored in the engine file or in the
27settings. However, the standard way is the following:
30engine file
31~~~~~~~~~~~
33+---------------------+-----------+-----------------------------------------+
34| argument | type | information |
35+=====================+===========+=========================================+
36| categories | list | pages, in which the engine is working |
37+---------------------+-----------+-----------------------------------------+
38| paging | boolean | support multible pages |
39+---------------------+-----------+-----------------------------------------+
40| language\_support | boolean | support language choosing |
41+---------------------+-----------+-----------------------------------------+
43settings.yml
44~~~~~~~~~~~~
46+------------+----------+-----------------------------------------------+
47| argument | type | information |
48+============+==========+===============================================+
49| name | string | name of search-engine |
50+------------+----------+-----------------------------------------------+
51| engine | string | name of searx-engine (filename without .py) |
52+------------+----------+-----------------------------------------------+
53| shortcut | string | shortcut of search-engine |
54+------------+----------+-----------------------------------------------+
55| timeout | string | specific timeout for search-engine |
56+------------+----------+-----------------------------------------------+
58overrides
59~~~~~~~~~
61A few of the options have default values in the engine, but are
62often overwritten by the settings. If ``None`` is assigned to an option
63in the engine file, it has to be redefined in the settings,
64otherwise searx will not start with that engine.
66The naming of overrides is arbitrary. But the recommended
67overrides are the following:
69+-----------------------+----------+----------------------------------------------------------------+
70| argument | type | information |
71+=======================+==========+================================================================+
72| base\_url | string | base-url, can be overwritten to use same engine on other URL |
73+-----------------------+----------+----------------------------------------------------------------+
74| number\_of\_results | int | maximum number of results per request |
75+-----------------------+----------+----------------------------------------------------------------+
76| language | string | ISO code of language and country like en\_US |
77+-----------------------+----------+----------------------------------------------------------------+
78| api\_key | string | api-key if required by engine |
79+-----------------------+----------+----------------------------------------------------------------+
81example code
82~~~~~~~~~~~~
84.. code:: python
86 # engine dependent config
87 categories = ['general']
88 paging = True
89 language_support = True
91making a request
92----------------
94To perform a search an URL have to be specified. In addition to
95specifying an URL, arguments can be passed to the query.
97passed arguments
98~~~~~~~~~~~~~~~~
100These arguments can be used to construct the search query. Furthermore,
101parameters with default value can be redefined for special purposes.
103+----------------------+------------+------------------------------------------------------------------------+
104| argument | type | default-value, information |
105+======================+============+========================================================================+
106| url | string | ``''`` |
107+----------------------+------------+------------------------------------------------------------------------+
108| method | string | ``'GET'`` |
109+----------------------+------------+------------------------------------------------------------------------+
110| headers | set | ``{}`` |
111+----------------------+------------+------------------------------------------------------------------------+
112| data | set | ``{}`` |
113+----------------------+------------+------------------------------------------------------------------------+
114| cookies | set | ``{}`` |
115+----------------------+------------+------------------------------------------------------------------------+
116| verify | boolean | ``True`` |
117+----------------------+------------+------------------------------------------------------------------------+
118| headers.User-Agent | string | a random User-Agent |
119+----------------------+------------+------------------------------------------------------------------------+
120| category | string | current category, like ``'general'`` |
121+----------------------+------------+------------------------------------------------------------------------+
122| started | datetime | current date-time |
123+----------------------+------------+------------------------------------------------------------------------+
124| pageno | int | current pagenumber |
125+----------------------+------------+------------------------------------------------------------------------+
126| language | string | specific language code like ``'en_US'``, or ``'all'`` if unspecified |
127+----------------------+------------+------------------------------------------------------------------------+
129parsed arguments
130~~~~~~~~~~~~~~~~
132The function ``def request(query, params):`` always returns the
133``params`` variable. Inside searx, the following paramters can be
134used to specify a search request:
136+------------+-----------+---------------------------------------------------------+
137| argument | type | information |
138+============+===========+=========================================================+
139| url | string | requested url |
140+------------+-----------+---------------------------------------------------------+
141| method | string | HTTP request method |
142+------------+-----------+---------------------------------------------------------+
143| headers | set | HTTP header information |
144+------------+-----------+---------------------------------------------------------+
145| data | set | HTTP data information (parsed if ``method != 'GET'``) |
146+------------+-----------+---------------------------------------------------------+
147| cookies | set | HTTP cookies |
148+------------+-----------+---------------------------------------------------------+
149| verify | boolean | Performing SSL-Validity check |
150+------------+-----------+---------------------------------------------------------+
152example code
153~~~~~~~~~~~~
155.. code:: python
157 # search-url
158 base_url = 'https://example.com/'
159 search_string = 'search?{query}&page={page}'
161 # do search-request
162 def request(query, params):
163 search_path = search_string.format(
164 query=urlencode({'q': query}),
165 page=params['pageno'])
167 params['url'] = base_url + search_path
169 return params
171returned results
172----------------
174Searx is able to return results of different media-types.
175Currently the following media-types are supported:
177- default
178- images
179- videos
180- torrent
181- map
183To set another media-type as default, the parameter
184``template`` must be set to the desired type.
186default
187~~~~~~~
189+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
190| result-parameter | information |
191+====================+====================================================================================================================================+
192| url | string, which is representing the url of the result |
193+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
194| title | string, which is representing the title of the result |
195+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
196| content | string, which is giving a general result-text |
197+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
198| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, represent when the result is published |
199+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
201images
202~~~~~~
204to use this template, the parameter
206+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
207| result-parameter | information |
208+====================+===========================================================================================================================================================+
209| template | is set to ``images.html`` |
210+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
211| url | string, which is representing the url to the result site |
212+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
213| title | string, which is representing the title of the result *(partly implemented)* |
214+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
215| content | *(partly implemented)* |
216+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
217| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, represent when the result is published *(partly implemented)* |
218+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
219| img\_src | string, which is representing the url to the result image |
220+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
221| thumbnail\_src | string, which is representing the url to a small-preview image |
222+--------------------+-----------------------------------------------------------------------------------------------------------------------------------------------------------+
224videos
225~~~~~~
227+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
228| result-parameter | information |
229+====================+====================================================================================================================================+
230| template | is set to ``videos.html`` |
231+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
232| url | string, which is representing the url of the result |
233+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
234| title | string, which is representing the title of the result |
235+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
236| content | *(not implemented yet)* |
237+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
238| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, represent when the result is published |
239+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
240| thumbnail | string, which is representing the url to a small-preview image |
241+--------------------+------------------------------------------------------------------------------------------------------------------------------------+
243torrent
244~~~~~~~
246+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
247| result-parameter | information |
248+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
249| template | is set to ```torrent.html``` |
250+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
251| url | string, which is representing the url of the result |
252+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
253| title | string, which is representing the title of the result |
254+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
255| content | string, which is giving a general result-text |
256+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
257| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, represent when the result is published *(not implemented yet)* |
258+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
259| seed | int, number of seeder |
260+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
261| leech | int, number of leecher |
262+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
263| filesize | int, size of file in bytes |
264+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
265| files | int, number of files |
266+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
267| magnetlink | string, which is the `magnetlink <https://en.wikipedia.org/wiki/Magnet_URI_scheme>`__ of the result |
268+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
269| torrentfile | string, which is the torrentfile of the result |
270+------------------+----------------------------------------------------------------------------------------------------------------------------------------------------------+
273map
274~~~
276+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
277| result-parameter | information |
278+=========================+====================================================================================================================================+
279| url | string, which is representing the url of the result |
280+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
281| title | string, which is representing the title of the result |
282+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
283| content | string, which is giving a general result-text |
284+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
285| publishedDate | `datetime.datetime <https://docs.python.org/2/library/datetime.html#datetime-objects>`__, represent when the result is published |
286+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
287| latitude | latitude of result (in decimal format) |
288+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
289| longitude | longitude of result (in decimal format) |
290+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
291| boundingbox | boundingbox of result (array of 4. values ``[lat-min, lat-max, lon-min, lon-max]``) |
292+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
293| geojson | geojson of result (http://geojson.org) |
294+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
295| osm.type | type of osm-object (if OSM-Result) |
296+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
297| osm.id | id of osm-object (if OSM-Result) |
298+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
299| address.name | name of object |
300+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
301| address.road | street adress of object |
302+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
303| address.house\_number | house number of object |
304+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
305| address.locality | city, place of object |
306+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
307| address.postcode | postcode of object |
308+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+
309| address.country | country of object |
310+-------------------------+------------------------------------------------------------------------------------------------------------------------------------+