@@ -0,0 +1,265 @@
1+
2+.. _engines-dev:
3+
4+===============
5+Engine overview
6+===============
7+
8+.. _metasearch-engine: https://en.wikipedia.org/wiki/Metasearch_engine
9+
10+searx is a metasearch-engine_, so it uses different search engines to provide
11+better results.
12+
13+Because there is no general search API which could be used for every search
14+engine, an adapter has to be built between searx and the external search
15+engines. Adapters are stored under the folder :origin:`searx/engines`.
16+
17+.. contents::
18+ :depth: 3
19+ :backlinks: entry
20+
21+general engine configuration
22+============================
23+
24+It is required to tell searx the type of results the engine provides. The
25+arguments can be set in the engine file or in the settings file
26+(normally ``settings.yml``). The arguments in the settings file override
27+the ones in the engine file.
28+
29+It does not matter if an option is stored in the engine file or in the
30+settings. However, the standard way is the following:
31+
32+
33+engine file
34+-----------
35+
36+======================= =========== ===========================================
37+argument type information
38+======================= =========== ===========================================
39+categories list pages, in which the engine is working
40+paging boolean support multible pages
41+language_support boolean support language choosing
42+time_range_support boolean support search time range
43+offline boolean engine runs offline
44+======================= =========== ===========================================
45+
46+
47+settings.yml
48+------------
49+
50+======================= =========== ===========================================
51+argument type information
52+======================= =========== ===========================================
53+name string name of search-engine
54+engine string name of searx-engine
55+ (filename without ``.py``)
56+shortcut string shortcut of search-engine
57+timeout string specific timeout for search-engine
58+======================= =========== ===========================================
59+
60+
61+overrides
62+---------
63+
64+A few of the options have default values in the engine, but are often
65+overwritten by the settings. If ``None`` is assigned to an option in the engine
66+file, it has to be redefined in the settings, otherwise searx will not start
67+with that engine.
68+
69+The naming of overrides is arbitrary. But the recommended overrides are the
70+following:
71+
72+======================= =========== ===========================================
73+argument type information
74+======================= =========== ===========================================
75+base_url string base-url, can be overwritten to use same
76+ engine on other URL
77+number_of_results int maximum number of results per request
78+language string ISO code of language and country like en_US
79+api_key string api-key if required by engine
80+======================= =========== ===========================================
81+
82+example code
83+------------
84+
85+.. code:: python
86+
87+ # engine dependent config
88+ categories = ['general']
89+ paging = True
90+ language_support = True
91+
92+
93+making a request
94+================
95+
96+To perform a search an URL have to be specified. In addition to specifying an
97+URL, arguments can be passed to the query.
98+
99+passed arguments
100+----------------
101+
102+These arguments can be used to construct the search query. Furthermore,
103+parameters with default value can be redefined for special purposes.
104+
105+====================== ============ ========================================================================
106+argument type default-value, information
107+====================== ============ ========================================================================
108+url string ``''``
109+method string ``'GET'``
110+headers set ``{}``
111+data set ``{}``
112+cookies set ``{}``
113+verify boolean ``True``
114+headers.User-Agent string a random User-Agent
115+category string current category, like ``'general'``
116+started datetime current date-time
117+pageno int current pagenumber
118+language string specific language code like ``'en_US'``, or ``'all'`` if unspecified
119+====================== ============ ========================================================================
120+
121+parsed arguments
122+----------------
123+
124+The function ``def request(query, params):`` always returns the ``params``
125+variable. Inside searx, the following paramters can be used to specify a search
126+request:
127+
128+============ =========== =========================================================
129+argument type information
130+============ =========== =========================================================
131+url string requested url
132+method string HTTP request method
133+headers set HTTP header information
134+data set HTTP data information (parsed if ``method != 'GET'``)
135+cookies set HTTP cookies
136+verify boolean Performing SSL-Validity check
137+============ =========== =========================================================
138+
139+
140+example code
141+------------
142+
143+.. code:: python
144+
145+ # search-url
146+ base_url = 'https://example.com/'
147+ search_string = 'search?{query}&page={page}'
148+
149+ # do search-request
150+ def request(query, params):
151+ search_path = search_string.format(
152+ query=urlencode({'q': query}),
153+ page=params['pageno'])
154+
155+ params['url'] = base_url + search_path
156+
157+ return params
158+
159+
160+returned results
161+================
162+
163+Searx is able to return results of different media-types. Currently the
164+following media-types are supported:
165+
166+- default_
167+- images_
168+- videos_
169+- torrent_
170+- map_
171+
172+To set another media-type as default, the parameter ``template`` must be set to
173+the desired type.
174+
175+default
176+-------
177+
178+========================= =====================================================
179+result-parameter information
180+========================= =====================================================
181+url string, url of the result
182+title string, title of the result
183+content string, general result-text
184+publishedDate :py:class:`datetime.datetime`, time of publish
185+========================= =====================================================
186+
187+images
188+------
189+
190+To use this template, the parameter:
191+
192+========================= =====================================================
193+result-parameter information
194+========================= =====================================================
195+template is set to ``images.html``
196+url string, url to the result site
197+title string, title of the result *(partly implemented)*
198+content *(partly implemented)*
199+publishedDate :py:class:`datetime.datetime`,
200+ time of publish *(partly implemented)*
201+img\_src string, url to the result image
202+thumbnail\_src string, url to a small-preview image
203+========================= =====================================================
204+
205+videos
206+------
207+
208+========================= =====================================================
209+result-parameter information
210+========================= =====================================================
211+template is set to ``videos.html``
212+url string, url of the result
213+title string, title of the result
214+content *(not implemented yet)*
215+publishedDate :py:class:`datetime.datetime`, time of publish
216+thumbnail string, url to a small-preview image
217+========================= =====================================================
218+
219+torrent
220+-------
221+
222+.. _magnetlink: https://en.wikipedia.org/wiki/Magnet_URI_scheme
223+
224+========================= =====================================================
225+result-parameter information
226+========================= =====================================================
227+template is set to ``torrent.html``
228+url string, url of the result
229+title string, title of the result
230+content string, general result-text
231+publishedDate :py:class:`datetime.datetime`,
232+ time of publish *(not implemented yet)*
233+seed int, number of seeder
234+leech int, number of leecher
235+filesize int, size of file in bytes
236+files int, number of files
237+magnetlink string, magnetlink_ of the result
238+torrentfile string, torrentfile of the result
239+========================= =====================================================
240+
241+
242+map
243+---
244+
245+========================= =====================================================
246+result-parameter information
247+========================= =====================================================
248+url string, url of the result
249+title string, title of the result
250+content string, general result-text
251+publishedDate :py:class:`datetime.datetime`, time of publish
252+latitude latitude of result (in decimal format)
253+longitude longitude of result (in decimal format)
254+boundingbox boundingbox of result (array of 4. values
255+ ``[lat-min, lat-max, lon-min, lon-max]``)
256+geojson geojson of result (http://geojson.org)
257+osm.type type of osm-object (if OSM-Result)
258+osm.id id of osm-object (if OSM-Result)
259+address.name name of object
260+address.road street name of object
261+address.house_number house number of object
262+address.locality city, place of object
263+address.postcode postcode of object
264+address.country country of object
265+========================= =====================================================