d93ac96 · searx · git.iko.soy

docs/admin/settings.rst+51−13

compare with master

unifiedsplit

@@ -130,14 +130,12 @@ Global Settings

130 request_timeout : 2.0 # default timeout in seconds, can be override by engine

131 # max_request_timeout: 10.0 # the maximum timeout in seconds

132 useragent_suffix : "" # informations like an email address to the administrator

133− pool_connections : 100 # Number of different hosts

133+ pool_connections : 100 # Maximum number of allowable connections, or None for no limits. The default is 100.

134− pool_maxsize : 10 # Number of simultaneous requests by host

134+ pool_maxsize : 10 # Number of allowable keep-alive connections, or None to always allow. The default is 10.

135+ enable_http2: True # See https://www.python-httpx.org/http2/

135 # uncomment below section if you want to use a proxy

136 # uncomment below section if you want to use a proxy

136 # proxies:

137 # proxies:

137− # http:

138+ # all://:

138− # - http://proxy1:8080

139− # - http://proxy2:8080

140− # https:

141 # - http://proxy1:8080

139 # - http://proxy1:8080

142 # - http://proxy2:8080

140 # - http://proxy2:8080

143 # uncomment below section only if you have more than one network interface

141 # uncomment below section only if you have more than one network interface

@@ -145,6 +143,7 @@ Global Settings

145 # source_ips:

143 # source_ips:

146 # - 1.1.1.1

144 # - 1.1.1.1

147 # - 1.1.1.2

145 # - 1.1.1.2

146+ # - fe80::/126

148

147

149

148

150``request_timeout`` :

149``request_timeout`` :

@@ -157,20 +156,46 @@ Global Settings

157 Suffix to the user-agent searx uses to send requests to others engines. If an

156 Suffix to the user-agent searx uses to send requests to others engines. If an

158 engine wish to block you, a contact info here may be useful to avoid that.

157 engine wish to block you, a contact info here may be useful to avoid that.

159

158

160−.. _requests proxies: https://requests.readthedocs.io/en/latest/user/advanced/#proxies

159+``keepalive_expiry``:

161−.. _PySocks: https://pypi.org/project/PySocks/

160+ Number of seconds to keep a connection in the pool. By default 5.0 seconds.

161+

162+.. _httpx proxies: https://www.python-httpx.org/advanced/#http-proxying

162

163

163``proxies`` :

164``proxies`` :

164− Define one or more proxies you wish to use, see `requests proxies`_.

165+ Define one or more proxies you wish to use, see `httpx proxies`_.

165 If there are more than one proxy for one protocol (http, https),

166 If there are more than one proxy for one protocol (http, https),

166 requests to the engines are distributed in a round-robin fashion.

167 requests to the engines are distributed in a round-robin fashion.

167

168

168− - Proxy: `see <https://2.python-requests.org/en/latest/user/advanced/#proxies>`__.

169− - SOCKS proxies are also supported: `see <https://2.python-requests.org/en/latest/user/advanced/#socks>`__

170−

171``source_ips`` :

169``source_ips`` :

172 If you use multiple network interfaces, define from which IP the requests must

170 If you use multiple network interfaces, define from which IP the requests must

173− be made. This parameter is ignored when ``proxies`` is set.

171+ be made. Example:

172+

173+ * ``0.0.0.0`` any local IPv4 address.

174+ * ``::`` any local IPv6 address.

175+ * ``192.168.0.1``

176+ * ``[ 192.168.0.1, 192.168.0.2 ]`` these two specific IP addresses

177+ * ``fe80::60a2:1691:e5a2:ee1f``

178+ * ``fe80::60a2:1691:e5a2:ee1f/126`` all IP addresses in this network.

179+ * ``[ 192.168.0.1, fe80::/126 ]``

180+

181+``retries`` :

182+ Number of retry in case of an HTTP error.

183+ On each retry, searx uses an different proxy and source ip.

184+

185+``retry_on_http_error`` :

186+ Retry request on some HTTP status code.

187+

188+ Example:

189+

190+ * ``true`` : on HTTP status code between 400 and 599.

191+ * ``403`` : on HTTP status code 403.

192+ * ``[403, 429]``: on HTTP status code 403 and 429.

193+

194+``enable_http2`` :

195+ Enable by default. Set to ``False`` to disable HTTP/2.

196+

197+``max_redirects`` :

198+ 30 by default. Maximum redirect before it is an error.

174

199

175

200

176``locales:``

201``locales:``

@@ -216,6 +241,13 @@ Engine settings

216 api_key : 'apikey'

241 api_key : 'apikey'

217 disabled : True

242 disabled : True

218 language : en_US

243 language : en_US

244+ #enable_http: False

245+ #enable_http2: False

246+ #retries: 1

247+ #retry_on_http_error: True # or 403 or [404, 429]

248+ #max_connections: 100

249+ #max_keepalive_connections: 10

250+ #keepalive_expiry: 5.0

219 #proxies:

251 #proxies:

220 # http:

252 # http:

221 # - http://proxy1:8080

253 # - http://proxy1:8080

@@ -270,6 +302,12 @@ Engine settings

270``display_error_messages`` : default ``True``

302``display_error_messages`` : default ``True``

271 When an engine returns an error, the message is displayed on the user interface.

303 When an engine returns an error, the message is displayed on the user interface.

272

304

305+``network``: optional

306+ Use the network configuration from another engine.

307+ In addition, there are two default networks:

308+ * ``ipv4`` set ``local_addresses`` to ``0.0.0.0`` (use only IPv4 local addresses)

309+ * ``ipv6`` set ``local_addresses`` to ``::`` (use only IPv6 local addresses)

310+

273.. note::

311.. note::

274

312

275 A few more options are possible, but they are pretty specific to some

313 A few more options are possible, but they are pretty specific to some

requirements-dev.txt+1

@@ -16,3 +16,4 @@ sphinx-tabs==2.1.0

16sphinxcontrib-programoutput==0.17

17sphinx-autobuild==2021.3.14

18linuxdoc==20210324

19+aiounittest==1.4.0

requirements.txt+6−2

@@ -2,11 +2,15 @@ certifi==2020.12.05

2babel==2.9.1

3flask-babel==2.0.0

4flask==1.1.2

5−idna==2.10

6jinja2==2.11.3

5jinja2==2.11.3

7lxml==4.6.3

6lxml==4.6.3

8pygments==2.8.0

7pygments==2.8.0

9python-dateutil==2.8.1

8python-dateutil==2.8.1

10pyyaml==5.4.1

9pyyaml==5.4.1

11−requests[socks]==2.25.1

10+httpx[http2]==0.17.1

11+Brotli==1.0.9

12+uvloop==0.15.2; python_version >= '3.7'

13+uvloop==0.14.0; python_version < '3.7'

14+httpx-socks[asyncio]==0.3.1

12langdetect==1.0.8

15langdetect==1.0.8

16+setproctitle==1.2.2

searx/autocomplete.py+4−3

@@ -20,10 +20,11 @@ from lxml import etree

20from json import loads

21from urllib.parse import urlencode

22

23−from requests import RequestException

23+from httpx import HTTPError

24+

24

25

25from searx import settings

26from searx import settings

26−from searx.poolrequests import get as http_get

27+from searx.network import get as http_get

27from searx.exceptions import SearxEngineResponseException

28from searx.exceptions import SearxEngineResponseException

28

29

30

@@ -136,5 +137,5 @@ def search_autocomplete(backend_name, query, lang):

136

137

137 try:

138 try:

138 return backend(query, lang)

139 return backend(query, lang)

139− except (RequestException, SearxEngineResponseException):

140+ except (HTTPError, SearxEngineResponseException):

140 return []

141 return []

searx/engines/__init__.py+3−3

@@ -27,7 +27,7 @@ from searx import settings

27from searx import logger

28from searx.data import ENGINES_LANGUAGES

29from searx.exceptions import SearxEngineResponseException

30−from searx.poolrequests import get, get_proxy_cycles

30+from searx.network import get, initialize as initialize_network, set_context_network_name

31from searx.utils import load_module, match_language, get_engine_from_settings, gen_useragent

32

33

@@ -89,8 +89,6 @@ def load_engine(engine_data):

89 engine.categories = []

90 else:

91 engine.categories = list(map(str.strip, param_value.split(',')))

92− elif param_name == 'proxies':

93− engine.proxies = get_proxy_cycles(param_value)

94 else:

92 else:

95 setattr(engine, param_name, param_value)

93 setattr(engine, param_name, param_value)

96

94

@@ -289,9 +287,11 @@ def load_engines(engine_list):

289

287

290def initialize_engines(engine_list):

288def initialize_engines(engine_list):

291 load_engines(engine_list)

289 load_engines(engine_list)

290+ initialize_network(engine_list, settings['outgoing'])

292

291

293 def engine_init(engine_name, init_fn):

292 def engine_init(engine_name, init_fn):

294 try:

293 try:

294+ set_context_network_name(engine_name)

295 init_fn(get_engine_from_settings(engine_name))

296 except SearxEngineResponseException as exc:

297 logger.warn('%s engine: Fail to initialize // %s', engine_name, exc)

searx/engines/dictzone.py+1−1

@@ -52,7 +52,7 @@ def response(resp):

52 to_results.append(to_result.text_content())

53

54 results.append({

55− 'url': urljoin(resp.url, '?%d' % k),

55+ 'url': urljoin(str(resp.url), '?%d' % k),

56 'title': from_result.text_content(),

57 'content': '; '.join(to_results)

58 })

searx/engines/duckduckgo.py+1−1

@@ -6,7 +6,7 @@

6from lxml.html import fromstring

7from json import loads

8from searx.utils import extract_text, match_language, eval_xpath, dict_subset

9−from searx.poolrequests import get

9+from searx.network import get

10

11# about

12about = {

searx/engines/duckduckgo_images.py+1−1

@@ -8,7 +8,7 @@ from urllib.parse import urlencode

8from searx.exceptions import SearxEngineAPIException

9from searx.engines.duckduckgo import get_region_code

10from searx.engines.duckduckgo import _fetch_supported_languages, supported_languages_url # NOQA # pylint: disable=unused-import

11−from searx.poolrequests import get

11+from searx.network import get

12

13# about

14about = {

searx/engines/elasticsearch.py+1−2

@@ -4,7 +4,6 @@

4"""

5

6from json import loads, dumps

7−from requests.auth import HTTPBasicAuth

8from searx.exceptions import SearxEngineAPIException

7from searx.exceptions import SearxEngineAPIException

9

8

10

9

@@ -32,7 +31,7 @@ def request(query, params):

32 return params

31 return params

33

32

34 if username and password:

33 if username and password:

35− params['auth'] = HTTPBasicAuth(username, password)

34+ params['auth'] = (username, password)

36

35

37 params['url'] = search_url

36 params['url'] = search_url

38 params['method'] = 'GET'

37 params['method'] = 'GET'

searx/engines/gigablast.py+1−1

@@ -8,7 +8,7 @@ import re

8from json import loads

9from urllib.parse import urlencode

10# from searx import logger

11−from searx.poolrequests import get

11+from searx.network import get

12

13# about

14about = {

searx/engines/google.py+2−3

@@ -10,7 +10,7 @@ Definitions`_.

10

11# pylint: disable=invalid-name, missing-function-docstring

12

13−from urllib.parse import urlencode, urlparse

13+from urllib.parse import urlencode

14from lxml import html

15from searx import logger

16from searx.utils import match_language, extract_text, eval_xpath, eval_xpath_list, eval_xpath_getindex

@@ -186,8 +186,7 @@ def get_lang_info(params, lang_list, custom_aliases):

186 return ret_val

187

188def detect_google_sorry(resp):

189− resp_url = urlparse(resp.url)

190− if resp_url.netloc == 'sorry.google.com' or resp_url.path.startswith('/sorry'):

189+ if resp.url.host == 'sorry.google.com' or resp.url.path.startswith('/sorry'):

191 raise SearxEngineCaptchaException()

190 raise SearxEngineCaptchaException()

192

191

193

192

searx/engines/pubmed.py+1−1

@@ -7,7 +7,7 @@ from flask_babel import gettext

7from lxml import etree

8from datetime import datetime

9from urllib.parse import urlencode

10−from searx.poolrequests import get

10+from searx.network import get

11

12# about

13about = {

searx/engines/qwant.py+1−1

@@ -8,7 +8,7 @@ from json import loads

8from urllib.parse import urlencode

9from searx.utils import html_to_text, match_language

10from searx.exceptions import SearxEngineAPIException, SearxEngineCaptchaException

11−from searx.raise_for_httperror import raise_for_httperror

11+from searx.network import raise_for_httperror

12

13# about

14about = {

searx/engines/seznam.py+3−4

@@ -3,9 +3,9 @@

3 Seznam

4"""

5

6−from urllib.parse import urlencode, urlparse

6+from urllib.parse import urlencode

7from lxml import html

8−from searx.poolrequests import get

8+from searx.network import get

9from searx.exceptions import SearxEngineAccessDeniedException

10from searx.utils import (

11 extract_text,

@@ -46,8 +46,7 @@ def request(query, params):

46

47

48def response(resp):

49− resp_url = urlparse(resp.url)

50− if resp_url.path.startswith('/verify'):

49+ if resp.url.path.startswith('/verify'):

51 raise SearxEngineAccessDeniedException()

50 raise SearxEngineAccessDeniedException()

52

51

53 results = []

52 results = []

searx/engines/sjp.py+1−1

@@ -6,7 +6,7 @@

6from lxml.html import fromstring

7from searx import logger

8from searx.utils import extract_text

9−from searx.raise_for_httperror import raise_for_httperror

9+from searx.network import raise_for_httperror

10

11logger = logger.getChild('sjp engine')

12

searx/engines/soundcloud.py+1−1

@@ -9,7 +9,7 @@ from lxml import html

9from dateutil import parser

10from urllib.parse import quote_plus, urlencode

11from searx import logger

12−from searx.poolrequests import get as http_get

12+from searx.network import get as http_get

13

14# about

15about = {

searx/engines/spotify.py+3−2

@@ -5,9 +5,10 @@

5

6from json import loads

7from urllib.parse import urlencode

8−import requests

9import base64

8import base64

10

9

10+from searx.network import post as http_post

11+

11# about

12# about

12about = {

13about = {

13 "website": 'https://www.spotify.com',

14 "website": 'https://www.spotify.com',

@@ -38,7 +39,7 @@ def request(query, params):

38

39

39 params['url'] = search_url.format(query=urlencode({'q': query}), offset=offset)

40 params['url'] = search_url.format(query=urlencode({'q': query}), offset=offset)

40

41

41− r = requests.post(

42+ r = http_post(

42 'https://accounts.spotify.com/api/token',

43 'https://accounts.spotify.com/api/token',

43 data={'grant_type': 'client_credentials'},

44 data={'grant_type': 'client_credentials'},

44 headers={'Authorization': 'Basic ' + base64.b64encode(

45 headers={'Authorization': 'Basic ' + base64.b64encode(

searx/engines/stackoverflow.py+2−3

@@ -3,7 +3,7 @@

3 Stackoverflow (IT)

4"""

5

6−from urllib.parse import urlencode, urljoin, urlparse

6+from urllib.parse import urlencode, urljoin

7from lxml import html

8from searx.utils import extract_text

9from searx.exceptions import SearxEngineCaptchaException

@@ -41,8 +41,7 @@ def request(query, params):

41

42# get response from search-request

43def response(resp):

44− resp_url = urlparse(resp.url)

45− if resp_url.path.startswith('/nocaptcha'):

44+ if resp.url.path.startswith('/nocaptcha'):

46 raise SearxEngineCaptchaException()

45 raise SearxEngineCaptchaException()

47

46

48 results = []

47 results = []

searx/engines/wikidata.py+1−1

@@ -12,7 +12,7 @@ from babel.dates import format_datetime, format_date, format_time, get_datetime_

12

13from searx import logger

14from searx.data import WIKIDATA_UNITS

15−from searx.poolrequests import post, get

15+from searx.network import post, get

16from searx.utils import match_language, searx_useragent, get_string_replaces_function

17from searx.external_urls import get_external_url, get_earth_coordinates_url, area_to_osm_zoom

18from searx.engines.wikipedia import _fetch_supported_languages, supported_languages_url # NOQA # pylint: disable=unused-import

searx/engines/wikipedia.py+1−1

@@ -7,7 +7,7 @@ from urllib.parse import quote

7from json import loads

8from lxml.html import fromstring

9from searx.utils import match_language, searx_useragent

10−from searx.raise_for_httperror import raise_for_httperror

10+from searx.network import raise_for_httperror

11

12# about

13about = {

searx/engines/wolframalpha_noapi.py+1−1

@@ -7,7 +7,7 @@ from json import loads

7from time import time

8from urllib.parse import urlencode

9

10−from searx.poolrequests import get as http_get

10+from searx.network import get as http_get

11

12# about

13about = {

searx/engines/wordnik.py+1−1

@@ -6,7 +6,7 @@

6from lxml.html import fromstring

7from searx import logger

8from searx.utils import extract_text

9−from searx.raise_for_httperror import raise_for_httperror

9+from searx.network import raise_for_httperror

10

11logger = logger.getChild('Wordnik engine')

12

searx/engines/yacy.py+2−2

@@ -7,7 +7,7 @@ from json import loads

7from dateutil import parser

8from urllib.parse import urlencode

9

10−from requests.auth import HTTPDigestAuth

10+from httpx import DigestAuth

11

12from searx.utils import html_to_text

13

@@ -56,7 +56,7 @@ def request(query, params):

56 search_type=search_type)

57

58 if http_digest_auth_user and http_digest_auth_pass:

59− params['auth'] = HTTPDigestAuth(http_digest_auth_user, http_digest_auth_pass)

59+ params['auth'] = DigestAuth(http_digest_auth_user, http_digest_auth_pass)

60

61 # add language tag if specified

62 if params['language'] != 'all':

searx/engines/yggtorrent.py+1−1

@@ -8,7 +8,7 @@ from operator import itemgetter

8from datetime import datetime

9from urllib.parse import quote

10from searx.utils import extract_text, get_torrent_size

11−from searx.poolrequests import get as http_get

11+from searx.network import get as http_get

12

13# about

14about = {

searx/metrology/error_recorder.py+9−9

@@ -3,7 +3,7 @@ import inspect

3import logging

4from json import JSONDecodeError

5from urllib.parse import urlparse

6−from requests.exceptions import RequestException

6+from httpx import HTTPError, HTTPStatusError

7from searx.exceptions import (SearxXPathSyntaxException, SearxEngineXPathException, SearxEngineAPIException,

8 SearxEngineAccessDeniedException)

9from searx import logger

@@ -60,28 +60,28 @@ def get_trace(traces):

60 return traces[-1]

61

62

63−def get_hostname(exc: RequestException) -> typing.Optional[None]:

63+def get_hostname(exc: HTTPError) -> typing.Optional[None]:

64 url = exc.request.url

65 if url is None and exc.response is not None:

66 url = exc.response.url

67 return urlparse(url).netloc

68

69

70−def get_request_exception_messages(exc: RequestException)\

70+def get_request_exception_messages(exc: HTTPError)\

71 -> typing.Tuple[typing.Optional[str], typing.Optional[str], typing.Optional[str]]:

72 url = None

73 status_code = None

74 reason = None

75 hostname = None

76− if exc.request is not None:

76+ if hasattr(exc, 'request') and exc.request is not None:

77 url = exc.request.url

78− if url is None and exc.response is not None:

78+ if url is None and hasattr(exc, 'response') and exc.respones is not None:

79 url = exc.response.url

80 if url is not None:

81− hostname = str(urlparse(url).netloc)

81+ hostname = url.host

82− if exc.response is not None:

82+ if isinstance(exc, HTTPStatusError):

83 status_code = str(exc.response.status_code)

84− reason = exc.response.reason

84+ reason = exc.response.reason_phrase

85 return (status_code, reason, hostname)

86

87

@@ -92,7 +92,7 @@ def get_messages(exc, filename) -> typing.Tuple:

92 return (str(exc), )

93 if isinstance(exc, ValueError) and 'lxml' in filename:

94 return (str(exc), )

95− if isinstance(exc, RequestException):

95+ if isinstance(exc, HTTPError):

96 return get_request_exception_messages(exc)

97 if isinstance(exc, SearxXPathSyntaxException):

98 return (exc.xpath_str, exc.message)

searx/network/__init__.pynew file+189

@@ -0,0 +1,189 @@

1+# SPDX-License-Identifier: AGPL-3.0-or-later

2+

3+import asyncio

4+import threading

5+import concurrent.futures

6+from time import time

7+

8+import httpx

9+import h2.exceptions

10+

11+from .network import get_network, initialize

12+from .client import get_loop

13+from .raise_for_httperror import raise_for_httperror

14+

15+# queue.SimpleQueue: Support Python 3.6

16+try:

17+ from queue import SimpleQueue

18+except ImportError:

19+ from queue import Empty

20+ from collections import deque

21+

22+ class SimpleQueue:

23+ """Minimal backport of queue.SimpleQueue"""

24+

25+ def __init__(self):

26+ self._queue = deque()

27+ self._count = threading.Semaphore(0)

28+

29+ def put(self, item):

30+ self._queue.append(item)

31+ self._count.release()

32+

33+ def get(self):

34+ if not self._count.acquire(True):

35+ raise Empty

36+ return self._queue.popleft()

37+

38+

39+THREADLOCAL = threading.local()

40+

41+

42+def reset_time_for_thread():

43+ THREADLOCAL.total_time = 0

44+

45+

46+def get_time_for_thread():

47+ return THREADLOCAL.total_time

48+

49+

50+def set_timeout_for_thread(timeout, start_time=None):

51+ THREADLOCAL.timeout = timeout

52+ THREADLOCAL.start_time = start_time

53+

54+

55+def set_context_network_name(network_name):

56+ THREADLOCAL.network = get_network(network_name)

57+

58+

59+def get_context_network():

60+ try:

61+ return THREADLOCAL.network

62+ except AttributeError:

63+ return get_network()

64+

65+

66+def request(method, url, **kwargs):

67+ """same as requests/requests/api.py request(...)"""

68+ time_before_request = time()

69+

70+ # timeout (httpx)

71+ if 'timeout' in kwargs:

72+ timeout = kwargs['timeout']

73+ else:

74+ timeout = getattr(THREADLOCAL, 'timeout', None)

75+ if timeout is not None:

76+ kwargs['timeout'] = timeout

77+

78+ # 2 minutes timeout for the requests without timeout

79+ timeout = timeout or 120

80+

81+ # ajdust actual timeout

82+ timeout += 0.2 # overhead

83+ start_time = getattr(THREADLOCAL, 'start_time', time_before_request)

84+ if start_time:

85+ timeout -= time() - start_time

86+

87+ # raise_for_error

88+ check_for_httperror = True

89+ if 'raise_for_httperror' in kwargs:

90+ check_for_httperror = kwargs['raise_for_httperror']

91+ del kwargs['raise_for_httperror']

92+

93+ # requests compatibility

94+ if isinstance(url, bytes):

95+ url = url.decode()

96+

97+ # network

98+ network = get_context_network()

99+

100+ # do request

101+ future = asyncio.run_coroutine_threadsafe(network.request(method, url, **kwargs), get_loop())

102+ try:

103+ response = future.result(timeout)

104+ except concurrent.futures.TimeoutError as e:

105+ raise httpx.TimeoutException('Timeout', request=None) from e

106+

107+ # requests compatibility

108+ # see also https://www.python-httpx.org/compatibility/#checking-for-4xx5xx-responses

109+ response.ok = not response.is_error

110+

111+ # update total_time.

112+ # See get_time_for_thread() and reset_time_for_thread()

113+ if hasattr(THREADLOCAL, 'total_time'):

114+ time_after_request = time()

115+ THREADLOCAL.total_time += time_after_request - time_before_request

116+

117+ # raise an exception

118+ if check_for_httperror:

119+ raise_for_httperror(response)

120+

121+ return response

122+

123+

124+def get(url, **kwargs):

125+ kwargs.setdefault('allow_redirects', True)

126+ return request('get', url, **kwargs)

127+

128+

129+def options(url, **kwargs):

130+ kwargs.setdefault('allow_redirects', True)

131+ return request('options', url, **kwargs)

132+

133+

134+def head(url, **kwargs):

135+ kwargs.setdefault('allow_redirects', False)

136+ return request('head', url, **kwargs)

137+

138+

139+def post(url, data=None, **kwargs):

140+ return request('post', url, data=data, **kwargs)

141+

142+

143+def put(url, data=None, **kwargs):

144+ return request('put', url, data=data, **kwargs)

145+

146+

147+def patch(url, data=None, **kwargs):

148+ return request('patch', url, data=data, **kwargs)

149+

150+

151+def delete(url, **kwargs):

152+ return request('delete', url, **kwargs)

153+

154+

155+async def stream_chunk_to_queue(network, q, method, url, **kwargs):

156+ try:

157+ async with network.stream(method, url, **kwargs) as response:

158+ q.put(response)

159+ async for chunk in response.aiter_bytes(65536):

160+ if len(chunk) > 0:

161+ q.put(chunk)

162+ except (httpx.HTTPError, OSError, h2.exceptions.ProtocolError) as e:

163+ q.put(e)

164+ finally:

165+ q.put(None)

166+

167+

168+def stream(method, url, **kwargs):

169+ """Replace httpx.stream.

170+

171+ Usage:

172+ stream = poolrequests.stream(...)

173+ response = next(stream)

174+ for chunk in stream:

175+ ...

176+

177+ httpx.Client.stream requires to write the httpx.HTTPTransport version of the

178+ the httpx.AsyncHTTPTransport declared above.

179+ """

180+ q = SimpleQueue()

181+ future = asyncio.run_coroutine_threadsafe(stream_chunk_to_queue(get_network(), q, method, url, **kwargs),

182+ get_loop())

183+ chunk_or_exception = q.get()

184+ while chunk_or_exception is not None:

185+ if isinstance(chunk_or_exception, Exception):

186+ raise chunk_or_exception

187+ yield chunk_or_exception

188+ chunk_or_exception = q.get()

189+ return future.result()

searx/network/client.pynew file+218

@@ -0,0 +1,218 @@

1+# SPDX-License-Identifier: AGPL-3.0-or-later

2+

3+import asyncio

4+import logging

5+import threading

6+

7+import httpcore

8+import httpx

9+from httpx_socks import AsyncProxyTransport

10+from python_socks import parse_proxy_url

11+import python_socks._errors

12+

13+from searx import logger

14+

15+# Optional uvloop (support Python 3.6)

16+try:

17+ import uvloop

18+except ImportError:

19+ pass

20+else:

21+ uvloop.install()

22+

23+

24+logger = logger.getChild('searx.http.client')

25+LOOP = None

26+TRANSPORT_KWARGS = {

27+ 'backend': 'asyncio',

28+ 'trust_env': False,

29+}

30+

31+

32+async def close_connections_for_url(connection_pool: httpcore.AsyncConnectionPool, url: httpcore._utils.URL):

33+ origin = httpcore._utils.url_to_origin(url)

34+ logger.debug('Drop connections for %r', origin)

35+ connections_to_close = connection_pool._connections_for_origin(origin)

36+ for connection in connections_to_close:

37+ await connection_pool._remove_from_pool(connection)

38+ try:

39+ await connection.aclose()

40+ except httpcore.NetworkError as e:

41+ logger.warning('Error closing an existing connection', exc_info=e)

42+

43+

44+class AsyncHTTPTransportNoHttp(httpcore.AsyncHTTPTransport):

45+ """Block HTTP request"""

46+

47+ async def arequest(self, method, url, headers=None, stream=None, ext=None):

48+ raise httpcore.UnsupportedProtocol("HTTP protocol is disabled")

49+

50+

51+class AsyncProxyTransportFixed(AsyncProxyTransport):

52+ """Fix httpx_socks.AsyncProxyTransport

53+

54+ Map python_socks exceptions to httpcore.ProxyError

55+

56+ Map socket.gaierror to httpcore.ConnectError

57+

58+ Note: keepalive_expiry is ignored, AsyncProxyTransport should call:

59+ * self._keepalive_sweep()

60+ * self._response_closed(self, connection)

61+

62+ Note: AsyncProxyTransport inherit from AsyncConnectionPool

63+

64+ Note: the API is going to change on httpx 0.18.0

65+ see https://github.com/encode/httpx/pull/1522

66+ """

67+

68+ async def arequest(self, method, url, headers=None, stream=None, ext=None):

69+ retry = 2

70+ while retry > 0:

71+ retry -= 1

72+ try:

73+ return await super().arequest(method, url, headers, stream, ext)

74+ except (python_socks._errors.ProxyConnectionError,

75+ python_socks._errors.ProxyTimeoutError,

76+ python_socks._errors.ProxyError) as e:

77+ raise httpcore.ProxyError(e)

78+ except OSError as e:

79+ # socket.gaierror when DNS resolution fails

80+ raise httpcore.NetworkError(e)

81+ except httpcore.RemoteProtocolError as e:

82+ # in case of httpcore.RemoteProtocolError: Server disconnected

83+ await close_connections_for_url(self, url)

84+ logger.warning('httpcore.RemoteProtocolError: retry', exc_info=e)

85+ # retry

86+ except (httpcore.NetworkError, httpcore.ProtocolError) as e:

87+ # httpcore.WriteError on HTTP/2 connection leaves a new opened stream

88+ # then each new request creates a new stream and raise the same WriteError

89+ await close_connections_for_url(self, url)

90+ raise e

91+

92+

93+class AsyncHTTPTransportFixed(httpx.AsyncHTTPTransport):

94+ """Fix httpx.AsyncHTTPTransport"""

95+

96+ async def arequest(self, method, url, headers=None, stream=None, ext=None):

97+ retry = 2

98+ while retry > 0:

99+ retry -= 1

100+ try:

101+ return await super().arequest(method, url, headers, stream, ext)

102+ except OSError as e:

103+ # socket.gaierror when DNS resolution fails

104+ raise httpcore.ConnectError(e)

105+ except httpcore.CloseError as e:

106+ # httpcore.CloseError: [Errno 104] Connection reset by peer

107+ # raised by _keepalive_sweep()

108+ # from https://github.com/encode/httpcore/blob/4b662b5c42378a61e54d673b4c949420102379f5/httpcore/_backends/asyncio.py#L198 # noqa

109+ await close_connections_for_url(self._pool, url)

110+ logger.warning('httpcore.CloseError: retry', exc_info=e)

111+ # retry

112+ except httpcore.RemoteProtocolError as e:

113+ # in case of httpcore.RemoteProtocolError: Server disconnected

114+ await close_connections_for_url(self._pool, url)

115+ logger.warning('httpcore.RemoteProtocolError: retry', exc_info=e)

116+ # retry

117+ except (httpcore.ProtocolError, httpcore.NetworkError) as e:

118+ await close_connections_for_url(self._pool, url)

119+ raise e

120+

121+

122+def get_transport_for_socks_proxy(verify, http2, local_address, proxy_url, limit, retries):

123+ # support socks5h (requests compatibility):

124+ # https://requests.readthedocs.io/en/master/user/advanced/#socks

125+ # socks5:// hostname is resolved on client side

126+ # socks5h:// hostname is resolved on proxy side

127+ rdns = False

128+ socks5h = 'socks5h://'

129+ if proxy_url.startswith(socks5h):

130+ proxy_url = 'socks5://' + proxy_url[len(socks5h):]

131+ rdns = True

132+

133+ proxy_type, proxy_host, proxy_port, proxy_username, proxy_password = parse_proxy_url(proxy_url)

134+

135+ return AsyncProxyTransportFixed(proxy_type=proxy_type, proxy_host=proxy_host, proxy_port=proxy_port,

136+ username=proxy_username, password=proxy_password,

137+ rdns=rdns,

138+ loop=get_loop(),

139+ verify=verify,

140+ http2=http2,

141+ local_address=local_address,

142+ max_connections=limit.max_connections,

143+ max_keepalive_connections=limit.max_keepalive_connections,

144+ keepalive_expiry=limit.keepalive_expiry,

145+ retries=retries,

146+ **TRANSPORT_KWARGS)

147+

148+

149+def get_transport(verify, http2, local_address, proxy_url, limit, retries):

150+ return AsyncHTTPTransportFixed(verify=verify,

151+ http2=http2,

152+ local_address=local_address,

153+ proxy=httpx._config.Proxy(proxy_url) if proxy_url else None,

154+ limits=limit,

155+ retries=retries,

156+ **TRANSPORT_KWARGS)

157+

158+

159+def iter_proxies(proxies):

160+ # https://www.python-httpx.org/compatibility/#proxy-keys

161+ if isinstance(proxies, str):

162+ yield 'all://', proxies

163+ elif isinstance(proxies, dict):

164+ for pattern, proxy_url in proxies.items():

165+ yield pattern, proxy_url

166+

167+

168+def new_client(enable_http, verify, enable_http2,

169+ max_connections, max_keepalive_connections, keepalive_expiry,

170+ proxies, local_address, retries, max_redirects):

171+ limit = httpx.Limits(max_connections=max_connections,

172+ max_keepalive_connections=max_keepalive_connections,

173+ keepalive_expiry=keepalive_expiry)

174+ # See https://www.python-httpx.org/advanced/#routing

175+ mounts = {}

176+ for pattern, proxy_url in iter_proxies(proxies):

177+ if not enable_http and (pattern == 'http' or pattern.startswith('http://')):

178+ continue

179+ if proxy_url.startswith('socks4://') \

180+ or proxy_url.startswith('socks5://') \

181+ or proxy_url.startswith('socks5h://'):

182+ mounts[pattern] = get_transport_for_socks_proxy(verify, enable_http2, local_address, proxy_url, limit,

183+ retries)

184+ else:

185+ mounts[pattern] = get_transport(verify, enable_http2, local_address, proxy_url, limit, retries)

186+

187+ if not enable_http:

188+ mounts['http://'] = AsyncHTTPTransportNoHttp()

189+

190+ transport = get_transport(verify, enable_http2, local_address, None, limit, retries)

191+ return httpx.AsyncClient(transport=transport, mounts=mounts, max_redirects=max_redirects)

192+

193+

194+def get_loop():

195+ global LOOP

196+ return LOOP

197+

198+

199+def init():

200+ # log

201+ for logger_name in ('hpack.hpack', 'hpack.table'):

202+ logging.getLogger(logger_name).setLevel(logging.WARNING)

203+

204+ # loop

205+ def loop_thread():

206+ global LOOP

207+ LOOP = asyncio.new_event_loop()

208+ LOOP.run_forever()

209+

210+ th = threading.Thread(

211+ target=loop_thread,

212+ name='asyncio_loop',

213+ daemon=True,

214+ )

215+ th.start()

216+

217+

218+init()

searx/network/network.pynew file+303

@@ -0,0 +1,303 @@

1+# SPDX-License-Identifier: AGPL-3.0-or-later

2+

3+import atexit

4+import asyncio

5+import ipaddress

6+from itertools import cycle

7+

8+import httpx

9+

10+from .client import new_client, get_loop

11+

12+

13+DEFAULT_NAME = '__DEFAULT__'

14+NETWORKS = {}

15+# requests compatibility when reading proxy settings from settings.yml

16+PROXY_PATTERN_MAPPING = {

17+ 'http': 'http://',

18+ 'https': 'https://',

19+ 'socks4': 'socks4://',

20+ 'socks5': 'socks5://',

21+ 'socks5h': 'socks5h://',

22+ 'http:': 'http://',

23+ 'https:': 'https://',

24+ 'socks4:': 'socks4://',

25+ 'socks5:': 'socks5://',

26+ 'socks5h:': 'socks5h://',

27+}

28+

29+ADDRESS_MAPPING = {

30+ 'ipv4': '0.0.0.0',

31+ 'ipv6': '::'

32+}

33+

34+

35+class Network:

36+

37+ __slots__ = ('enable_http', 'verify', 'enable_http2',

38+ 'max_connections', 'max_keepalive_connections', 'keepalive_expiry',

39+ 'local_addresses', 'proxies', 'max_redirects', 'retries', 'retry_on_http_error',

40+ '_local_addresses_cycle', '_proxies_cycle', '_clients')

41+

42+ def __init__(self,

43+ enable_http=True,

44+ verify=True,

45+ enable_http2=False,

46+ max_connections=None,

47+ max_keepalive_connections=None,

48+ keepalive_expiry=None,

49+ proxies=None,

50+ local_addresses=None,

51+ retries=0,

52+ retry_on_http_error=None,

53+ max_redirects=30):

54+ self.enable_http = enable_http

55+ self.verify = verify

56+ self.enable_http2 = enable_http2

57+ self.max_connections = max_connections

58+ self.max_keepalive_connections = max_keepalive_connections

59+ self.keepalive_expiry = keepalive_expiry

60+ self.proxies = proxies

61+ self.local_addresses = local_addresses

62+ self.retries = retries

63+ self.retry_on_http_error = retry_on_http_error

64+ self.max_redirects = max_redirects

65+ self._local_addresses_cycle = self.get_ipaddress_cycle()

66+ self._proxies_cycle = self.get_proxy_cycles()

67+ self._clients = {}

68+ self.check_parameters()

69+

70+ def check_parameters(self):

71+ for address in self.iter_ipaddresses():

72+ if '/' in address:

73+ ipaddress.ip_network(address, False)

74+ else:

75+ ipaddress.ip_address(address)

76+

77+ if self.proxies is not None and not isinstance(self.proxies, (str, dict)):

78+ raise ValueError('proxies type has to be str, dict or None')

79+

80+ def iter_ipaddresses(self):

81+ local_addresses = self.local_addresses

82+ if not local_addresses:

83+ return

84+ elif isinstance(local_addresses, str):

85+ local_addresses = [local_addresses]

86+ for address in local_addresses:

87+ yield address

88+

89+ def get_ipaddress_cycle(self):

90+ while True:

91+ count = 0

92+ for address in self.iter_ipaddresses():

93+ if '/' in address:

94+ for a in ipaddress.ip_network(address, False).hosts():

95+ yield str(a)

96+ count += 1

97+ else:

98+ a = ipaddress.ip_address(address)

99+ yield str(a)

100+ count += 1

101+ if count == 0:

102+ yield None

103+

104+ def iter_proxies(self):

105+ if not self.proxies:

106+ return

107+ # https://www.python-httpx.org/compatibility/#proxy-keys

108+ if isinstance(self.proxies, str):

109+ yield 'all://', [self.proxies]

110+ else:

111+ for pattern, proxy_url in self.proxies.items():

112+ pattern = PROXY_PATTERN_MAPPING.get(pattern, pattern)

113+ if isinstance(proxy_url, str):

114+ proxy_url = [proxy_url]

115+ yield pattern, proxy_url

116+

117+ def get_proxy_cycles(self):

118+ proxy_settings = {}

119+ for pattern, proxy_urls in self.iter_proxies():

120+ proxy_settings[pattern] = cycle(proxy_urls)

121+ while True:

122+ yield tuple((pattern, next(proxy_url_cycle)) for pattern, proxy_url_cycle in proxy_settings.items())

123+

124+ def get_client(self, verify=None, max_redirects=None):

125+ verify = self.verify if verify is None else verify

126+ max_redirects = self.max_redirects if max_redirects is None else max_redirects

127+ local_address = next(self._local_addresses_cycle)

128+ proxies = next(self._proxies_cycle) # is a tuple so it can be part of the key

129+ key = (verify, max_redirects, local_address, proxies)

130+ if key not in self._clients or self._clients[key].is_closed:

131+ self._clients[key] = new_client(self.enable_http,

132+ verify,

133+ self.enable_http2,

134+ self.max_connections,

135+ self.max_keepalive_connections,

136+ self.keepalive_expiry,

137+ dict(proxies),

138+ local_address,

139+ 0,

140+ max_redirects)

141+ return self._clients[key]

142+

143+ async def aclose(self):

144+ async def close_client(client):

145+ try:

146+ await client.aclose()

147+ except httpx.HTTPError:

148+ pass

149+ await asyncio.gather(*[close_client(client) for client in self._clients.values()], return_exceptions=False)

150+

151+ @staticmethod

152+ def get_kwargs_clients(kwargs):

153+ kwargs_clients = {}

154+ if 'verify' in kwargs:

155+ kwargs_clients['verify'] = kwargs.pop('verify')

156+ if 'max_redirects' in kwargs:

157+ kwargs_clients['max_redirects'] = kwargs.pop('max_redirects')

158+ return kwargs_clients

159+

160+ def is_valid_respones(self, response):

161+ if (self.retry_on_http_error is True and 400 <= response.status_code <= 599) \

162+ or (isinstance(self.retry_on_http_error, list) and response.status_code in self.retry_on_http_error) \

163+ or (isinstance(self.retry_on_http_error, int) and response.status_code == self.retry_on_http_error):

164+ return False

165+ return True

166+

167+ async def request(self, method, url, **kwargs):

168+ retries = self.retries

169+ while retries >= 0: # pragma: no cover

170+ kwargs_clients = Network.get_kwargs_clients(kwargs)

171+ client = self.get_client(**kwargs_clients)

172+ try:

173+ response = await client.request(method, url, **kwargs)

174+ if self.is_valid_respones(response) or retries <= 0:

175+ return response

176+ except (httpx.RequestError, httpx.HTTPStatusError) as e:

177+ if retries <= 0:

178+ raise e

179+ retries -= 1

180+

181+ def stream(self, method, url, **kwargs):

182+ retries = self.retries

183+ while retries >= 0: # pragma: no cover

184+ kwargs_clients = Network.get_kwargs_clients(kwargs)

185+ client = self.get_client(**kwargs_clients)

186+ try:

187+ response = client.stream(method, url, **kwargs)

188+ if self.is_valid_respones(response) or retries <= 0:

189+ return response

190+ except (httpx.RequestError, httpx.HTTPStatusError) as e:

191+ if retries <= 0:

192+ raise e

193+ retries -= 1

194+

195+ @classmethod

196+ async def aclose_all(cls):

197+ await asyncio.gather(*[network.aclose() for network in NETWORKS.values()], return_exceptions=False)

198+

199+

200+def get_network(name=None):

201+ global NETWORKS

202+ return NETWORKS[name or DEFAULT_NAME]

203+

204+

205+def initialize(settings_engines=None, settings_outgoing=None):

206+ from searx.engines import engines

207+ from searx import settings

208+

209+ global NETWORKS

210+

211+ settings_engines = settings_engines or settings.get('engines')

212+ settings_outgoing = settings_outgoing or settings.get('outgoing')

213+

214+ # default parameters for AsyncHTTPTransport

215+ # see https://github.com/encode/httpx/blob/e05a5372eb6172287458b37447c30f650047e1b8/httpx/_transports/default.py#L108-L121 # noqa

216+ default_params = {

217+ 'enable_http': False,

218+ 'verify': True,

219+ 'enable_http2': settings_outgoing.get('enable_http2', True),

220+ # Magic number kept from previous code

221+ 'max_connections': settings_outgoing.get('pool_connections', 100),

222+ # Picked from constructor

223+ 'max_keepalive_connections': settings_outgoing.get('pool_maxsize', 10),

224+ #

225+ 'keepalive_expiry': settings_outgoing.get('keepalive_expiry', 5.0),

226+ 'local_addresses': settings_outgoing.get('source_ips'),

227+ 'proxies': settings_outgoing.get('proxies'),

228+ # default maximum redirect

229+ # from https://github.com/psf/requests/blob/8c211a96cdbe9fe320d63d9e1ae15c5c07e179f8/requests/models.py#L55

230+ 'max_redirects': settings_outgoing.get('max_redirects', 30),

231+ #

232+ 'retries': settings_outgoing.get('retries', 0),

233+ 'retry_on_http_error': None,

234+ }

235+

236+ def new_network(params):

237+ nonlocal default_params

238+ result = {}

239+ result.update(default_params)

240+ result.update(params)

241+ return Network(**result)

242+

243+ def iter_networks():

244+ nonlocal settings_engines

245+ for engine_spec in settings_engines:

246+ engine_name = engine_spec['name']

247+ engine = engines.get(engine_name)

248+ if engine is None:

249+ continue

250+ network = getattr(engine, 'network', None)

251+ yield engine_name, engine, network

252+

253+ if NETWORKS:

254+ done()

255+ NETWORKS.clear()

256+ NETWORKS[DEFAULT_NAME] = new_network({})

257+ NETWORKS['ipv4'] = new_network({'local_addresses': '0.0.0.0'})

258+ NETWORKS['ipv6'] = new_network({'local_addresses': '::'})

259+

260+ # define networks from outgoing.networks

261+ for network_name, network in settings_outgoing.get('networks', {}).items():

262+ NETWORKS[network_name] = new_network(network)

263+

264+ # define networks from engines.[i].network (except references)

265+ for engine_name, engine, network in iter_networks():

266+ if network is None:

267+ network = {}

268+ for attribute_name, attribute_value in default_params.items():

269+ if hasattr(engine, attribute_name):

270+ network[attribute_name] = getattr(engine, attribute_name)

271+ else:

272+ network[attribute_name] = attribute_value

273+ NETWORKS[engine_name] = new_network(network)

274+ elif isinstance(network, dict):

275+ NETWORKS[engine_name] = new_network(network)

276+

277+ # define networks from engines.[i].network (references)

278+ for engine_name, engine, network in iter_networks():

279+ if isinstance(network, str):

280+ NETWORKS[engine_name] = NETWORKS[network]

281+

282+

283+@atexit.register

284+def done():

285+ """Close all HTTP client

286+

287+ Avoid a warning at exit

288+ see https://github.com/encode/httpx/blob/1a6e254f72d9fd5694a1c10a28927e193ab4f76b/httpx/_client.py#L1785

289+

290+ Note: since Network.aclose has to be async, it is not possible to call this method on Network.__del__

291+ So Network.aclose is called here using atexit.register

292+ """

293+ try:

294+ loop = get_loop()

295+ if loop:

296+ future = asyncio.run_coroutine_threadsafe(Network.aclose_all(), loop)

297+ # wait 3 seconds to close the HTTP clients

298+ future.result(3)

299+ finally:

300+ NETWORKS.clear()

301+

302+

303+NETWORKS[DEFAULT_NAME] = Network()

searx/raise_for_httperror.py → searx/network/raise_for_httperror.pyrenamed

The content is the same.

searx/poolrequests.pydeleted−235

@@ -1,235 +0,0 @@

1−import sys

2−from time import time

3−from itertools import cycle

4−from threading import local

5−

6−import requests

7−

8−from searx import settings

9−from searx import logger

10−from searx.raise_for_httperror import raise_for_httperror

11−

12−

13−logger = logger.getChild('poolrequests')

14−

15−

16−try:

17− import ssl

18− if ssl.OPENSSL_VERSION_INFO[0:3] < (1, 0, 2):

19− # https://github.com/certifi/python-certifi#1024-bit-root-certificates

20− logger.critical('You are using an old openssl version({0}), please upgrade above 1.0.2!'

21− .format(ssl.OPENSSL_VERSION))

22− sys.exit(1)

23−except ImportError:

24− ssl = None

25−if not getattr(ssl, "HAS_SNI", False):

26− try:

27− import OpenSSL # pylint: disable=unused-import

28− except ImportError:

29− logger.critical("ssl doesn't support SNI and the pyopenssl module is not installed.\n"

30− "Some HTTPS connections will fail")

31− sys.exit(1)

32−

33−

34−class HTTPAdapterWithConnParams(requests.adapters.HTTPAdapter):

35−

36− def __init__(self, pool_connections=requests.adapters.DEFAULT_POOLSIZE,

37− pool_maxsize=requests.adapters.DEFAULT_POOLSIZE,

38− max_retries=requests.adapters.DEFAULT_RETRIES,

39− pool_block=requests.adapters.DEFAULT_POOLBLOCK,

40− **conn_params):

41− if max_retries == requests.adapters.DEFAULT_RETRIES:

42− self.max_retries = requests.adapters.Retry(0, read=False)

43− else:

44− self.max_retries = requests.adapters.Retry.from_int(max_retries)

45− self.config = {}

46− self.proxy_manager = {}

47−

48− super().__init__()

49−

50− self._pool_connections = pool_connections

51− self._pool_maxsize = pool_maxsize

52− self._pool_block = pool_block

53− self._conn_params = conn_params

54−

55− self.init_poolmanager(pool_connections, pool_maxsize, block=pool_block, **conn_params)

56−

57− def __setstate__(self, state):

58− # Can't handle by adding 'proxy_manager' to self.__attrs__ because

59− # because self.poolmanager uses a lambda function, which isn't pickleable.

60− self.proxy_manager = {}

61− self.config = {}

62−

63− for attr, value in state.items():

64− setattr(self, attr, value)

65−

66− self.init_poolmanager(self._pool_connections, self._pool_maxsize,

67− block=self._pool_block, **self._conn_params)

68−

69−

70−threadLocal = local()

71−connect = settings['outgoing'].get('pool_connections', 100) # Magic number kept from previous code

72−maxsize = settings['outgoing'].get('pool_maxsize', requests.adapters.DEFAULT_POOLSIZE) # Picked from constructor

73−if settings['outgoing'].get('source_ips'):

74− http_adapters = cycle(HTTPAdapterWithConnParams(pool_connections=connect, pool_maxsize=maxsize,

75− source_address=(source_ip, 0))

76− for source_ip in settings['outgoing']['source_ips'])

77− https_adapters = cycle(HTTPAdapterWithConnParams(pool_connections=connect, pool_maxsize=maxsize,

78− source_address=(source_ip, 0))

79− for source_ip in settings['outgoing']['source_ips'])

80−else:

81− http_adapters = cycle((HTTPAdapterWithConnParams(pool_connections=connect, pool_maxsize=maxsize), ))

82− https_adapters = cycle((HTTPAdapterWithConnParams(pool_connections=connect, pool_maxsize=maxsize), ))

83−

84−

85−class SessionSinglePool(requests.Session):

86−

87− def __init__(self):

88− super().__init__()

89−

90− # reuse the same adapters

91− self.adapters.clear()

92−

93− https_adapter = threadLocal.__dict__.setdefault('https_adapter', next(https_adapters))

94− self.mount('https://', https_adapter)

95− if get_enable_http_protocol():

96− http_adapter = threadLocal.__dict__.setdefault('http_adapter', next(http_adapters))

97− self.mount('http://', http_adapter)

98−

99− def close(self):

100− """Call super, but clear adapters since there are managed globaly"""

101− self.adapters.clear()

102− super().close()

103−

104−

105−def set_timeout_for_thread(timeout, start_time=None):

106− threadLocal.timeout = timeout

107− threadLocal.start_time = start_time

108−

109−

110−def set_enable_http_protocol(enable_http):

111− threadLocal.enable_http = enable_http

112−

113−

114−def get_enable_http_protocol():

115− try:

116− return threadLocal.enable_http

117− except AttributeError:

118− return False

119−

120−

121−def reset_time_for_thread():

122− threadLocal.total_time = 0

123−

124−

125−def get_time_for_thread():

126− return threadLocal.total_time

127−

128−

129−def get_proxy_cycles(proxy_settings):

130− if not proxy_settings:

131− return None

132− # Backwards compatibility for single proxy in settings.yml

133− for protocol, proxy in proxy_settings.items():

134− if isinstance(proxy, str):

135− proxy_settings[protocol] = [proxy]

136−

137− for protocol in proxy_settings:

138− proxy_settings[protocol] = cycle(proxy_settings[protocol])

139− return proxy_settings

140−

141−

142−GLOBAL_PROXY_CYCLES = get_proxy_cycles(settings['outgoing'].get('proxies'))

143−

144−

145−def get_proxies(proxy_cycles):

146− if proxy_cycles:

147− return {protocol: next(proxy_cycle) for protocol, proxy_cycle in proxy_cycles.items()}

148− return None

149−

150−

151−def get_global_proxies():

152− return get_proxies(GLOBAL_PROXY_CYCLES)

153−

154−

155−def request(method, url, **kwargs):

156− """same as requests/requests/api.py request(...)"""

157− time_before_request = time()

158−

159− # session start

160− session = SessionSinglePool()

161−

162− # proxies

163− if not kwargs.get('proxies'):

164− kwargs['proxies'] = get_global_proxies()

165−

166− # timeout

167− if 'timeout' in kwargs:

168− timeout = kwargs['timeout']

169− else:

170− timeout = getattr(threadLocal, 'timeout', None)

171− if timeout is not None:

172− kwargs['timeout'] = timeout

173−

174− # raise_for_error

175− check_for_httperror = True

176− if 'raise_for_httperror' in kwargs:

177− check_for_httperror = kwargs['raise_for_httperror']

178− del kwargs['raise_for_httperror']

179−

180− # do request

181− response = session.request(method=method, url=url, **kwargs)

182−

183− time_after_request = time()

184−

185− # is there a timeout for this engine ?

186− if timeout is not None:

187− timeout_overhead = 0.2 # seconds

188− # start_time = when the user request started

189− start_time = getattr(threadLocal, 'start_time', time_before_request)

190− search_duration = time_after_request - start_time

191− if search_duration > timeout + timeout_overhead:

192− raise requests.exceptions.Timeout(response=response)

193−

194− # session end

195− session.close()

196−

197− if hasattr(threadLocal, 'total_time'):

198− threadLocal.total_time += time_after_request - time_before_request

199−

200− # raise an exception

201− if check_for_httperror:

202− raise_for_httperror(response)

203−

204− return response

205−

206−

207−def get(url, **kwargs):

208− kwargs.setdefault('allow_redirects', True)

209− return request('get', url, **kwargs)

210−

211−

212−def options(url, **kwargs):

213− kwargs.setdefault('allow_redirects', True)

214− return request('options', url, **kwargs)

215−

216−

217−def head(url, **kwargs):

218− kwargs.setdefault('allow_redirects', False)

219− return request('head', url, **kwargs)

220−

221−

222−def post(url, data=None, **kwargs):

223− return request('post', url, data=data, **kwargs)

224−

225−

226−def put(url, data=None, **kwargs):

227− return request('put', url, data=data, **kwargs)

228−

229−

230−def patch(url, data=None, **kwargs):

231− return request('patch', url, data=data, **kwargs)

232−

233−

234−def delete(url, **kwargs):

235− return request('delete', url, **kwargs)

searx/search/checker/impl.py+6−6

@@ -11,9 +11,9 @@ from urllib.parse import urlparse

11import re

12from langdetect import detect_langs

13from langdetect.lang_detect_exception import LangDetectException

14−import requests.exceptions

14+import httpx

15

16−from searx import poolrequests, logger

16+from searx import network, logger

17from searx.results import ResultContainer

18from searx.search.models import SearchQuery, EngineRef

19from searx.search.processors import EngineProcessor

@@ -75,8 +75,8 @@ def _is_url_image(image_url):

75 while retry > 0:

76 a = time()

77 try:

78− poolrequests.set_timeout_for_thread(10.0, time())

78+ network.set_timeout_for_thread(10.0, time())

79− r = poolrequests.get(image_url, timeout=10.0, allow_redirects=True, headers={

79+ r = network.get(image_url, timeout=10.0, allow_redirects=True, headers={

80 'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64; rv:84.0) Gecko/20100101 Firefox/84.0',

81 'Accept': 'text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8',

82 'Accept-Language': 'en-US;q=0.5,en;q=0.3',

@@ -90,10 +90,10 @@ def _is_url_image(image_url):

90 if r.headers["content-type"].startswith('image/'):

91 return True

92 return False

93− except requests.exceptions.Timeout:

93+ except httpx.TimeoutException:

94 logger.error('Timeout for %s: %i', image_url, int(time() - a))

95 retry -= 1

96− except requests.exceptions.RequestException:

96+ except httpx.HTTPError:

97 logger.exception('Exception for %s', image_url)

98 return False

99

searx/search/processors/online.py+19−23

@@ -1,12 +1,12 @@

1# SPDX-License-Identifier: AGPL-3.0-or-later

2

3−from urllib.parse import urlparse

4from time import time

3from time import time

5import threading

4import threading

5+import asyncio

6

7−import requests.exceptions

7+import httpx

8

9−import searx.poolrequests as poolrequests

9+import searx.network

10from searx.engines import settings

11from searx import logger

12from searx.utils import gen_useragent

@@ -64,10 +64,6 @@ class OnlineProcessor(EngineProcessor):

64 auth=params['auth']

65 )

66

67− # setting engine based proxies

68− if hasattr(self.engine, 'proxies'):

69− request_args['proxies'] = poolrequests.get_proxies(self.engine.proxies)

70−

71 # max_redirects

67 # max_redirects

72 max_redirects = params.get('max_redirects')

68 max_redirects = params.get('max_redirects')

73 if max_redirects:

69 if max_redirects:

@@ -85,9 +81,9 @@ class OnlineProcessor(EngineProcessor):

85

81

86 # specific type of request (GET or POST)

82 # specific type of request (GET or POST)

87 if params['method'] == 'GET':

83 if params['method'] == 'GET':

88− req = poolrequests.get

84+ req = searx.network.get

89 else:

85 else:

90− req = poolrequests.post

86+ req = searx.network.post

91

87

92 request_args['data'] = params['data']

88 request_args['data'] = params['data']

93

89

@@ -99,8 +95,8 @@ class OnlineProcessor(EngineProcessor):

99 # unexpected redirect : record an error

95 # unexpected redirect : record an error

100 # but the engine might still return valid results.

96 # but the engine might still return valid results.

101 status_code = str(response.status_code or '')

97 status_code = str(response.status_code or '')

102− reason = response.reason or ''

98+ reason = response.reason_phrase or ''

103− hostname = str(urlparse(response.url or '').netloc)

99+ hostname = response.url.host

104 record_error(self.engine_name,

100 record_error(self.engine_name,

105 '{} redirects, maximum: {}'.format(len(response.history), soft_max_redirects),

101 '{} redirects, maximum: {}'.format(len(response.history), soft_max_redirects),

106 (status_code, reason, hostname))

102 (status_code, reason, hostname))

@@ -128,14 +124,14 @@ class OnlineProcessor(EngineProcessor):

128

124

129 def search(self, query, params, result_container, start_time, timeout_limit):

125 def search(self, query, params, result_container, start_time, timeout_limit):

130 # set timeout for all HTTP requests

126 # set timeout for all HTTP requests

131− poolrequests.set_timeout_for_thread(timeout_limit, start_time=start_time)

127+ searx.network.set_timeout_for_thread(timeout_limit, start_time=start_time)

132 # reset the HTTP total time

128 # reset the HTTP total time

133− poolrequests.reset_time_for_thread()

129+ searx.network.reset_time_for_thread()

134− # enable HTTP only if explicitly enabled

130+ # set the network

135− poolrequests.set_enable_http_protocol(self.engine.enable_http)

131+ searx.network.set_context_network_name(self.engine_name)

136

132

137 # suppose everything will be alright

133 # suppose everything will be alright

138− requests_exception = False

134+ http_exception = False

139 suspended_time = None

135 suspended_time = None

140

136

141 try:

137 try:

@@ -149,7 +145,7 @@ class OnlineProcessor(EngineProcessor):

149

145

150 # update engine time when there is no exception

146 # update engine time when there is no exception

151 engine_time = time() - start_time

147 engine_time = time() - start_time

152− page_load_time = poolrequests.get_time_for_thread()

148+ page_load_time = searx.network.get_time_for_thread()

153 result_container.add_timing(self.engine_name, engine_time, page_load_time)

149 result_container.add_timing(self.engine_name, engine_time, page_load_time)

154 with threading.RLock():

150 with threading.RLock():

155 self.engine.stats['engine_time'] += engine_time

151 self.engine.stats['engine_time'] += engine_time

@@ -162,27 +158,27 @@ class OnlineProcessor(EngineProcessor):

162

158

163 # Timing

159 # Timing

164 engine_time = time() - start_time

160 engine_time = time() - start_time

165− page_load_time = poolrequests.get_time_for_thread()

161+ page_load_time = searx.network.get_time_for_thread()

166 result_container.add_timing(self.engine_name, engine_time, page_load_time)

162 result_container.add_timing(self.engine_name, engine_time, page_load_time)

167

163

168 # Record the errors

164 # Record the errors

169 with threading.RLock():

165 with threading.RLock():

170 self.engine.stats['errors'] += 1

166 self.engine.stats['errors'] += 1

171

167

172− if (issubclass(e.__class__, requests.exceptions.Timeout)):

168+ if (issubclass(e.__class__, (httpx.TimeoutException, asyncio.TimeoutError))):

173 result_container.add_unresponsive_engine(self.engine_name, 'HTTP timeout')

169 result_container.add_unresponsive_engine(self.engine_name, 'HTTP timeout')

174 # requests timeout (connect or read)

170 # requests timeout (connect or read)

175 logger.error("engine {0} : HTTP requests timeout"

171 logger.error("engine {0} : HTTP requests timeout"

176 "(search duration : {1} s, timeout: {2} s) : {3}"

172 "(search duration : {1} s, timeout: {2} s) : {3}"

177 .format(self.engine_name, engine_time, timeout_limit, e.__class__.__name__))

173 .format(self.engine_name, engine_time, timeout_limit, e.__class__.__name__))

178− requests_exception = True

174+ http_exception = True

179− elif (issubclass(e.__class__, requests.exceptions.RequestException)):

175+ elif (issubclass(e.__class__, (httpx.HTTPError, httpx.StreamError))):

180 result_container.add_unresponsive_engine(self.engine_name, 'HTTP error')

176 result_container.add_unresponsive_engine(self.engine_name, 'HTTP error')

181 # other requests exception

177 # other requests exception

182 logger.exception("engine {0} : requests exception"

178 logger.exception("engine {0} : requests exception"

183 "(search duration : {1} s, timeout: {2} s) : {3}"

179 "(search duration : {1} s, timeout: {2} s) : {3}"

184 .format(self.engine_name, engine_time, timeout_limit, e))

180 .format(self.engine_name, engine_time, timeout_limit, e))

185− requests_exception = True

181+ http_exception = True

186 elif (issubclass(e.__class__, SearxEngineCaptchaException)):

182 elif (issubclass(e.__class__, SearxEngineCaptchaException)):

187 result_container.add_unresponsive_engine(self.engine_name, 'CAPTCHA required')

183 result_container.add_unresponsive_engine(self.engine_name, 'CAPTCHA required')

188 logger.exception('engine {0} : CAPTCHA'.format(self.engine_name))

184 logger.exception('engine {0} : CAPTCHA'.format(self.engine_name))

@@ -206,7 +202,7 @@ class OnlineProcessor(EngineProcessor):

206 # suspend the engine if there is an HTTP error

202 # suspend the engine if there is an HTTP error

207 # or suspended_time is defined

203 # or suspended_time is defined

208 with threading.RLock():

204 with threading.RLock():

209− if requests_exception or suspended_time:

205+ if http_exception or suspended_time:

210 # update continuous_errors / suspend_end_time

206 # update continuous_errors / suspend_end_time

211 self.engine.continuous_errors += 1

207 self.engine.continuous_errors += 1

212 if suspended_time is None:

208 if suspended_time is None:

searx/settings.yml+8−7

@@ -67,19 +67,17 @@ ui:

67# key : !!binary "your_morty_proxy_key"

68

69outgoing: # communication with search engines

70− request_timeout : 2.0 # default timeout in seconds, can be override by engine

70+ request_timeout : 3.0 # default timeout in seconds, can be override by engine

71 # max_request_timeout: 10.0 # the maximum timeout in seconds

72 useragent_suffix : "" # suffix of searx_useragent, could contain informations like an email address to the administrator

73− pool_connections : 100 # Number of different hosts

73+ pool_connections : 100 # The maximum number of concurrent connections that may be established.

74− pool_maxsize : 10 # Number of simultaneous requests by host

74+ pool_maxsize : 20 # Allow the connection pool to maintain keep-alive connections below this point.

75+ enable_http2: True # See https://www.python-httpx.org/http2/

75# uncomment below section if you want to use a proxy

76# uncomment below section if you want to use a proxy

76# see https://2.python-requests.org/en/latest/user/advanced/#proxies

77# see https://2.python-requests.org/en/latest/user/advanced/#proxies

77# SOCKS proxies are also supported: see https://2.python-requests.org/en/latest/user/advanced/#socks

78# SOCKS proxies are also supported: see https://2.python-requests.org/en/latest/user/advanced/#socks

78# proxies:

79# proxies:

79−# http:

80+# all://:

80−# - http://proxy1:8080

81−# - http://proxy2:8080

82−# https:

83# - http://proxy1:8080

81# - http://proxy1:8080

84# - http://proxy2:8080

82# - http://proxy2:8080

85# using_tor_proxy : True

83# using_tor_proxy : True

@@ -89,6 +87,7 @@ outgoing: # communication with search engines

89# source_ips:

87# source_ips:

90# - 1.1.1.1

88# - 1.1.1.1

91# - 1.1.1.2

89# - 1.1.1.2

90+# - fe80::/126

92

91

93# External plugin configuration

92# External plugin configuration

94# See http://searx.github.io/searx/dev/plugins.html for more details

93# See http://searx.github.io/searx/dev/plugins.html for more details

@@ -887,11 +886,13 @@ engines:

887 engine : qwant

886 engine : qwant

888 shortcut : qwi

887 shortcut : qwi

889 categories : images

888 categories : images

889+ network: qwant

890

891 - name : qwant news

892 engine : qwant

893 shortcut : qwn

894 categories : news

895+ network: qwant

895

896

896# - name: library

897# - name: library

897# engine: recoll

898# engine: recoll

searx/testing.py+2−2

@@ -10,8 +10,8 @@ import traceback

10

11from os.path import dirname, join, abspath, realpath

12

13−from unittest import TestCase

14from splinter import Browser

13from splinter import Browser

14+import aiounittest

15

16

17class SearxTestLayer:

@@ -82,7 +82,7 @@ def run_robot_tests(tests):

82 test(browser)

83

84

85−class SearxTestCase(TestCase):

85+class SearxTestCase(aiounittest.AsyncTestCase):

86 """Base test case for non-robot tests."""

87

88 layer = SearxTestLayer

searx/utils.py+1−1

@@ -45,7 +45,7 @@ def searx_useragent():

45 """Return the searx User Agent"""

46 return 'searx/{searx_version} {suffix}'.format(

47 searx_version=VERSION_STRING,

48− suffix=settings['outgoing'].get('useragent_suffix', ''))

48+ suffix=settings['outgoing'].get('useragent_suffix', '')).strip()

49

50

51def gen_useragent(os=None):

searx/webapp.py+62−32

@@ -26,12 +26,26 @@ if __name__ == '__main__':

26 from os.path import realpath, dirname

27 sys.path.append(realpath(dirname(realpath(__file__)) + '/../'))

28

29+# set Unix thread name

30+try:

31+ import setproctitle

32+except ImportError:

33+ pass

34+else:

35+ import threading

36+ old_thread_init = threading.Thread.__init__

37+

38+ def new_thread_init(self, *args, **kwargs):

39+ old_thread_init(self, *args, **kwargs)

40+ setproctitle.setthreadtitle(self._name)

41+ threading.Thread.__init__ = new_thread_init

42+

29import hashlib

43import hashlib

30import hmac

44import hmac

31import json

45import json

32import os

46import os

33

47

34−import requests

48+import httpx

35

49

36from searx import logger

50from searx import logger

37logger = logger.getChild('webapp')

51logger = logger.getChild('webapp')

@@ -40,6 +54,7 @@ from datetime import datetime, timedelta

40from time import time

54from time import time

41from html import escape

55from html import escape

42from io import StringIO

56from io import StringIO

57+import urllib

43from urllib.parse import urlencode, urlparse

58from urllib.parse import urlencode, urlparse

44

59

45from pygments import highlight

60from pygments import highlight

@@ -79,7 +94,7 @@ from searx.plugins import plugins

79from searx.plugins.oa_doi_rewrite import get_doi_resolver

94from searx.plugins.oa_doi_rewrite import get_doi_resolver

80from searx.preferences import Preferences, ValidationException, LANGUAGE_CODES

95from searx.preferences import Preferences, ValidationException, LANGUAGE_CODES

81from searx.answerers import answerers

96from searx.answerers import answerers

82−from searx.poolrequests import get_global_proxies

97+from searx.network import stream as http_stream

83from searx.answerers import ask

98from searx.answerers import ask

84from searx.metrology.error_recorder import errors_per_engines

99from searx.metrology.error_recorder import errors_per_engines

85

100

@@ -209,7 +224,10 @@ def get_locale():

209 request.form['use-translation'] = 'oc'

224 request.form['use-translation'] = 'oc'

210 locale = 'fr_FR'

225 locale = 'fr_FR'

211

226

227+ logger.debug(

212− logger.debug("%s uses locale `%s` from %s", request.url, locale, locale_source)

228+ "%s uses locale `%s` from %s", urllib.parse.quote(request.url), locale, locale_source

229+ )

230+

213 return locale

231 return locale

214

232

215

233

@@ -899,50 +917,62 @@ def _is_selected_language_supported(engine, preferences):

899

917

900@app.route('/image_proxy', methods=['GET'])

918@app.route('/image_proxy', methods=['GET'])

901def image_proxy():

919def image_proxy():

902− url = request.args.get('url').encode()

920+ url = request.args.get('url')

903

921

904 if not url:

922 if not url:

905 return '', 400

923 return '', 400

906

924

907− h = new_hmac(settings['server']['secret_key'], url)

925+ h = new_hmac(settings['server']['secret_key'], url.encode())

908

926

909 if h != request.args.get('h'):

927 if h != request.args.get('h'):

910 return '', 400

928 return '', 400

911

929

912− headers = dict_subset(request.headers, {'If-Modified-Since', 'If-None-Match'})

930+ maximum_size = 5 * 1024 * 1024

913− headers['User-Agent'] = gen_useragent()

914−

915− resp = requests.get(url,

916− stream=True,

917− timeout=settings['outgoing']['request_timeout'],

918− headers=headers,

919− proxies=get_global_proxies())

920−

921− if resp.status_code == 304:

922− return '', resp.status_code

923

931

924− if resp.status_code != 200:

932+ try:

925− logger.debug('image-proxy: wrong response code: {0}'.format(resp.status_code))

933+ headers = dict_subset(request.headers, {'If-Modified-Since', 'If-None-Match'})

926− if resp.status_code >= 400:

934+ headers['User-Agent'] = gen_useragent()

935+ stream = http_stream(

936+ method='GET',

937+ url=url,

938+ headers=headers,

939+ timeout=settings['outgoing']['request_timeout'],

940+ allow_redirects=True,

941+ max_redirects=20)

942+

943+ resp = next(stream)

944+ content_length = resp.headers.get('Content-Length')

945+ if content_length and content_length.isdigit() and int(content_length) > maximum_size:

946+ return 'Max size', 400

947+

948+ if resp.status_code == 304:

927 return '', resp.status_code

949 return '', resp.status_code

928− return '', 400

929

950

930− if not resp.headers.get('content-type', '').startswith('image/'):

951+ if resp.status_code != 200:

931− logger.debug('image-proxy: wrong content-type: {0}'.format(resp.headers.get('content-type')))

952+ logger.debug('image-proxy: wrong response code: {0}'.format(resp.status_code))

953+ if resp.status_code >= 400:

954+ return '', resp.status_code

932− return '', 400

955+ return '', 400

933

956

934− img = b''

957+ if not resp.headers.get('content-type', '').startswith('image/'):

935− chunk_counter = 0

958+ logger.debug('image-proxy: wrong content-type: {0}'.format(resp.headers.get('content-type')))

959+ return '', 400

936

960

937− for chunk in resp.iter_content(1024 * 1024):

961+ headers = dict_subset(resp.headers, {'Content-Length', 'Length', 'Date', 'Last-Modified', 'Expires', 'Etag'})

938− chunk_counter += 1

939− if chunk_counter > 5:

940− return '', 502 # Bad gateway - file is too big (>5M)

941− img += chunk

942

962

943− headers = dict_subset(resp.headers, {'Content-Length', 'Length', 'Date', 'Last-Modified', 'Expires', 'Etag'})

963+ total_length = 0

944

964

965+ def forward_chunk():

966+ nonlocal total_length

967+ for chunk in stream:

968+ total_length += len(chunk)

969+ if total_length > maximum_size:

970+ break

971+ yield chunk

972+

945− return Response(img, mimetype=resp.headers['content-type'], headers=headers)

973+ return Response(forward_chunk(), mimetype=resp.headers['Content-Type'], headers=headers)

974+ except httpx.HTTPError:

975+ return '', 400

946

976

947

977

948@app.route('/stats', methods=['GET'])

978@app.route('/stats', methods=['GET'])

searx_extra/update/update_engine_descriptions.py+3−3

@@ -10,7 +10,7 @@ from searx.engines.wikidata import send_wikidata_query

10from searx.utils import extract_text

11import searx

12import searx.search

13−import searx.poolrequests

13+import searx.network

14

15SPARQL_WIKIPEDIA_ARTICLE = """

16SELECT DISTINCT ?item ?name

@@ -59,7 +59,7 @@ def get_wikipedia_summary(language, pageid):

59 search_url = 'https://{language}.wikipedia.org/api/rest_v1/page/summary/{title}'

60 url = search_url.format(title=quote(pageid), language=language)

61 try:

62− response = searx.poolrequests.get(url)

62+ response = searx.network.get(url)

63 response.raise_for_status()

64 api_result = json.loads(response.text)

65 return api_result.get('extract')

@@ -89,7 +89,7 @@ def get_website_description(url, lang1, lang2=None):

89 lang_list.append(lang2)

90 headers['Accept-Language'] = f'{",".join(lang_list)};q=0.8'

91 try:

92− response = searx.poolrequests.get(url, headers=headers, timeout=10)

92+ response = searx.network.get(url, headers=headers, timeout=10)

93 response.raise_for_status()

94 except Exception:

95 return (None, None)

searx_extra/update/update_external_bangs.py+3−3

@@ -17,7 +17,7 @@ import json

17import re

18from os.path import join

19

20−import requests

20+import httpx

21

22from searx import searx_dir # pylint: disable=E0401 C0413

23

@@ -30,7 +30,7 @@ HTTP_COLON = 'http:'

30

31

32def get_bang_url():

33− response = requests.get(URL_BV1)

33+ response = httpx.get(URL_BV1)

34 response.raise_for_status()

35

36 r = RE_BANG_VERSION.findall(response.text)

@@ -38,7 +38,7 @@ def get_bang_url():

38

39

40def fetch_ddg_bangs(url):

41− response = requests.get(url)

41+ response = httpx.get(url)

42 response.raise_for_status()

43 return json.loads(response.content.decode())

44

tests/unit/network/__init__.pynew file

No changes to the content.

tests/unit/network/test_network.pynew file+236

@@ -0,0 +1,236 @@

1+# SPDX-License-Identifier: AGPL-3.0-or-later

2+

3+from mock import patch

4+

5+import httpx

6+

7+from searx.network.network import Network, NETWORKS

8+from searx.testing import SearxTestCase

9+

10+

11+class TestNetwork(SearxTestCase):

12+

13+ def test_simple(self):

14+ network = Network()

15+

16+ self.assertEqual(next(network._local_addresses_cycle), None)

17+ self.assertEqual(next(network._proxies_cycle), ())

18+

19+ def test_ipaddress_cycle(self):

20+ network = NETWORKS['ipv6']

21+ self.assertEqual(next(network._local_addresses_cycle), '::')

22+ self.assertEqual(next(network._local_addresses_cycle), '::')

23+

24+ network = NETWORKS['ipv4']

25+ self.assertEqual(next(network._local_addresses_cycle), '0.0.0.0')

26+ self.assertEqual(next(network._local_addresses_cycle), '0.0.0.0')

27+

28+ network = Network(local_addresses=['192.168.0.1', '192.168.0.2'])

29+ self.assertEqual(next(network._local_addresses_cycle), '192.168.0.1')

30+ self.assertEqual(next(network._local_addresses_cycle), '192.168.0.2')

31+ self.assertEqual(next(network._local_addresses_cycle), '192.168.0.1')

32+

33+ network = Network(local_addresses=['192.168.0.0/30'])

34+ self.assertEqual(next(network._local_addresses_cycle), '192.168.0.1')

35+ self.assertEqual(next(network._local_addresses_cycle), '192.168.0.2')

36+ self.assertEqual(next(network._local_addresses_cycle), '192.168.0.1')

37+ self.assertEqual(next(network._local_addresses_cycle), '192.168.0.2')

38+

39+ network = Network(local_addresses=['fe80::/10'])

40+ self.assertEqual(next(network._local_addresses_cycle), 'fe80::1')

41+ self.assertEqual(next(network._local_addresses_cycle), 'fe80::2')

42+ self.assertEqual(next(network._local_addresses_cycle), 'fe80::3')

43+

44+ with self.assertRaises(ValueError):

45+ Network(local_addresses=['not_an_ip_address'])

46+

47+ def test_proxy_cycles(self):

48+ network = Network(proxies='http://localhost:1337')

49+ self.assertEqual(next(network._proxies_cycle), (('all://', 'http://localhost:1337'),))

50+

51+ network = Network(proxies={

52+ 'https': 'http://localhost:1337',

53+ 'http': 'http://localhost:1338'

54+ })

55+ self.assertEqual(next(network._proxies_cycle),

56+ (('https://', 'http://localhost:1337'), ('http://', 'http://localhost:1338')))

57+ self.assertEqual(next(network._proxies_cycle),

58+ (('https://', 'http://localhost:1337'), ('http://', 'http://localhost:1338')))

59+

60+ network = Network(proxies={

61+ 'https': ['http://localhost:1337', 'http://localhost:1339'],

62+ 'http': 'http://localhost:1338'

63+ })

64+ self.assertEqual(next(network._proxies_cycle),

65+ (('https://', 'http://localhost:1337'), ('http://', 'http://localhost:1338')))

66+ self.assertEqual(next(network._proxies_cycle),

67+ (('https://', 'http://localhost:1339'), ('http://', 'http://localhost:1338')))

68+

69+ with self.assertRaises(ValueError):

70+ Network(proxies=1)

71+

72+ def test_get_kwargs_clients(self):

73+ kwargs = {

74+ 'verify': True,

75+ 'max_redirects': 5,

76+ 'timeout': 2,

77+ }

78+ kwargs_client = Network.get_kwargs_clients(kwargs)

79+

80+ self.assertEqual(len(kwargs_client), 2)

81+ self.assertEqual(len(kwargs), 1)

82+

83+ self.assertEqual(kwargs['timeout'], 2)

84+

85+ self.assertTrue(kwargs_client['verify'])

86+ self.assertEqual(kwargs_client['max_redirects'], 5)

87+

88+ async def test_get_client(self):

89+ network = Network(verify=True)

90+ client1 = network.get_client()

91+ client2 = network.get_client(verify=True)

92+ client3 = network.get_client(max_redirects=10)

93+ client4 = network.get_client(verify=True)

94+ client5 = network.get_client(verify=False)

95+ client6 = network.get_client(max_redirects=10)

96+

97+ self.assertEqual(client1, client2)

98+ self.assertEqual(client1, client4)

99+ self.assertNotEqual(client1, client3)

100+ self.assertNotEqual(client1, client5)

101+ self.assertEqual(client3, client6)

102+

103+ await network.aclose()

104+

105+ async def test_aclose(self):

106+ network = Network(verify=True)

107+ network.get_client()

108+ await network.aclose()

109+

110+ async def test_request(self):

111+ a_text = 'Lorem Ipsum'

112+ response = httpx.Response(status_code=200, text=a_text)

113+ with patch.object(httpx.AsyncClient, 'request', return_value=response):

114+ network = Network(enable_http=True)

115+ response = await network.request('GET', 'https://example.com/')

116+ self.assertEqual(response.text, a_text)

117+ await network.aclose()

118+

119+

120+class TestNetworkRequestRetries(SearxTestCase):

121+

122+ TEXT = 'Lorem Ipsum'

123+

124+ @classmethod

125+ def get_response_404_then_200(cls):

126+ first = True

127+

128+ async def get_response(*args, **kwargs):

129+ nonlocal first

130+ if first:

131+ first = False

132+ return httpx.Response(status_code=403, text=TestNetworkRequestRetries.TEXT)

133+ return httpx.Response(status_code=200, text=TestNetworkRequestRetries.TEXT)

134+ return get_response

135+

136+ async def test_retries_ok(self):

137+ with patch.object(httpx.AsyncClient, 'request', new=TestNetworkRequestRetries.get_response_404_then_200()):

138+ network = Network(enable_http=True, retries=1, retry_on_http_error=403)

139+ response = await network.request('GET', 'https://example.com/')

140+ self.assertEqual(response.text, TestNetworkRequestRetries.TEXT)

141+ await network.aclose()

142+

143+ async def test_retries_fail_int(self):

144+ with patch.object(httpx.AsyncClient, 'request', new=TestNetworkRequestRetries.get_response_404_then_200()):

145+ network = Network(enable_http=True, retries=0, retry_on_http_error=403)

146+ response = await network.request('GET', 'https://example.com/')

147+ self.assertEqual(response.status_code, 403)

148+ await network.aclose()

149+

150+ async def test_retries_fail_list(self):

151+ with patch.object(httpx.AsyncClient, 'request', new=TestNetworkRequestRetries.get_response_404_then_200()):

152+ network = Network(enable_http=True, retries=0, retry_on_http_error=[403, 429])

153+ response = await network.request('GET', 'https://example.com/')

154+ self.assertEqual(response.status_code, 403)

155+ await network.aclose()

156+

157+ async def test_retries_fail_bool(self):

158+ with patch.object(httpx.AsyncClient, 'request', new=TestNetworkRequestRetries.get_response_404_then_200()):

159+ network = Network(enable_http=True, retries=0, retry_on_http_error=True)

160+ response = await network.request('GET', 'https://example.com/')

161+ self.assertEqual(response.status_code, 403)

162+ await network.aclose()

163+

164+ async def test_retries_exception_then_200(self):

165+ request_count = 0

166+

167+ async def get_response(*args, **kwargs):

168+ nonlocal request_count

169+ request_count += 1

170+ if request_count < 3:

171+ raise httpx.RequestError('fake exception', request=None)

172+ return httpx.Response(status_code=200, text=TestNetworkRequestRetries.TEXT)

173+

174+ with patch.object(httpx.AsyncClient, 'request', new=get_response):

175+ network = Network(enable_http=True, retries=2)

176+ response = await network.request('GET', 'https://example.com/')

177+ self.assertEqual(response.status_code, 200)

178+ self.assertEqual(response.text, TestNetworkRequestRetries.TEXT)

179+ await network.aclose()

180+

181+ async def test_retries_exception(self):

182+ async def get_response(*args, **kwargs):

183+ raise httpx.RequestError('fake exception', request=None)

184+

185+ with patch.object(httpx.AsyncClient, 'request', new=get_response):

186+ network = Network(enable_http=True, retries=0)

187+ with self.assertRaises(httpx.RequestError):

188+ await network.request('GET', 'https://example.com/')

189+ await network.aclose()

190+

191+

192+class TestNetworkStreamRetries(SearxTestCase):

193+

194+ TEXT = 'Lorem Ipsum'

195+

196+ @classmethod

197+ def get_response_exception_then_200(cls):

198+ first = True

199+

200+ def stream(*args, **kwargs):

201+ nonlocal first

202+ if first:

203+ first = False

204+ raise httpx.RequestError('fake exception', request=None)

205+ return httpx.Response(status_code=200, text=TestNetworkStreamRetries.TEXT)

206+ return stream

207+

208+ async def test_retries_ok(self):

209+ with patch.object(httpx.AsyncClient, 'stream', new=TestNetworkStreamRetries.get_response_exception_then_200()):

210+ network = Network(enable_http=True, retries=1, retry_on_http_error=403)

211+ response = network.stream('GET', 'https://example.com/')

212+ self.assertEqual(response.text, TestNetworkStreamRetries.TEXT)

213+ await network.aclose()

214+

215+ async def test_retries_fail(self):

216+ with patch.object(httpx.AsyncClient, 'stream', new=TestNetworkStreamRetries.get_response_exception_then_200()):

217+ network = Network(enable_http=True, retries=0, retry_on_http_error=403)

218+ with self.assertRaises(httpx.RequestError):

219+ network.stream('GET', 'https://example.com/')

220+ await network.aclose()

221+

222+ async def test_retries_exception(self):

223+ first = True

224+

225+ def stream(*args, **kwargs):

226+ nonlocal first

227+ if first:

228+ first = False

229+ return httpx.Response(status_code=403, text=TestNetworkRequestRetries.TEXT)

230+ return httpx.Response(status_code=200, text=TestNetworkRequestRetries.TEXT)

231+

232+ with patch.object(httpx.AsyncClient, 'stream', new=stream):

233+ network = Network(enable_http=True, retries=0, retry_on_http_error=403)

234+ response = network.stream('GET', 'https://example.com/')

235+ self.assertEqual(response.status_code, 403)

236+ await network.aclose()

tests/unit/test_engines_init.py+1−1

@@ -13,7 +13,7 @@ class TestEnginesInit(SearxTestCase):

13 engine_list = [{'engine': 'dummy', 'name': 'engine1', 'shortcut': 'e1'},

14 {'engine': 'dummy', 'name': 'engine2', 'shortcut': 'e2'}]

15

16− engines.initialize_engines(engine_list)

16+ engines.load_engines(engine_list)

17 self.assertEqual(len(engines.engines), 2)

18 self.assertIn('engine1', engines.engines)

19 self.assertIn('engine2', engines.engines)

tests/unit/test_poolrequests.pydeleted−89

@@ -1,89 +0,0 @@

1−from unittest.mock import patch

2−from requests.models import Response

3−

4−from searx.testing import SearxTestCase

5−

6−import searx.poolrequests

7−from searx.poolrequests import get_proxy_cycles, get_proxies

8−

9−

10−CONFIG = {'http': ['http://localhost:9090', 'http://localhost:9092'],

11− 'https': ['http://localhost:9091', 'http://localhost:9093']}

12−

13−

14−class TestProxy(SearxTestCase):

15−

16− def test_noconfig(self):

17− cycles = get_proxy_cycles(None)

18− self.assertIsNone(cycles)

19−

20− cycles = get_proxy_cycles(False)

21− self.assertIsNone(cycles)

22−

23− def test_oldconfig(self):

24− config = {

25− 'http': 'http://localhost:9090',

26− 'https': 'http://localhost:9091',

27− }

28− cycles = get_proxy_cycles(config)

29− self.assertEqual(next(cycles['http']), 'http://localhost:9090')

30− self.assertEqual(next(cycles['http']), 'http://localhost:9090')

31− self.assertEqual(next(cycles['https']), 'http://localhost:9091')

32− self.assertEqual(next(cycles['https']), 'http://localhost:9091')

33−

34− def test_one_proxy(self):

35− config = {

36− 'http': ['http://localhost:9090'],

37− 'https': ['http://localhost:9091'],

38− }

39− cycles = get_proxy_cycles(config)

40− self.assertEqual(next(cycles['http']), 'http://localhost:9090')

41− self.assertEqual(next(cycles['http']), 'http://localhost:9090')

42− self.assertEqual(next(cycles['https']), 'http://localhost:9091')

43− self.assertEqual(next(cycles['https']), 'http://localhost:9091')

44−

45− def test_multiple_proxies(self):

46− cycles = get_proxy_cycles(CONFIG)

47− self.assertEqual(next(cycles['http']), 'http://localhost:9090')

48− self.assertEqual(next(cycles['http']), 'http://localhost:9092')

49− self.assertEqual(next(cycles['http']), 'http://localhost:9090')

50− self.assertEqual(next(cycles['https']), 'http://localhost:9091')

51− self.assertEqual(next(cycles['https']), 'http://localhost:9093')

52− self.assertEqual(next(cycles['https']), 'http://localhost:9091')

53−

54− def test_getproxies_none(self):

55− self.assertIsNone(get_proxies(None))

56−

57− def test_getproxies_config(self):

58− cycles = get_proxy_cycles(CONFIG)

59− self.assertEqual(get_proxies(cycles), {

60− 'http': 'http://localhost:9090',

61− 'https': 'http://localhost:9091'

62− })

63− self.assertEqual(get_proxies(cycles), {

64− 'http': 'http://localhost:9092',

65− 'https': 'http://localhost:9093'

66− })

67−

68− @patch('searx.poolrequests.get_global_proxies')

69− def test_request(self, mock_get_global_proxies):

70− method = 'GET'

71− url = 'http://localhost'

72− custom_proxies = {

73− 'https': 'http://localhost:1080'

74− }

75− global_proxies = {

76− 'http': 'http://localhost:9092',

77− 'https': 'http://localhost:9093'

78− }

79− mock_get_global_proxies.return_value = global_proxies

80−

81− # check the global proxies usage

82− with patch.object(searx.poolrequests.SessionSinglePool, 'request', return_value=Response()) as mock_method:

83− searx.poolrequests.request(method, url)

84− mock_method.assert_called_once_with(method=method, url=url, proxies=global_proxies)

85−

86− # check if the proxies parameter overrides the global proxies

87− with patch.object(searx.poolrequests.SessionSinglePool, 'request', return_value=Response()) as mock_method:

88− searx.poolrequests.request(method, url, proxies=custom_proxies)

89− mock_method.assert_called_once_with(method=method, url=url, proxies=custom_proxies)

tests/unit/test_query.py+10−5

@@ -1,7 +1,11 @@

1+from mock import patch

2+

1from searx.search import initialize

3from searx.search import initialize

2from searx.query import RawTextQuery

4from searx.query import RawTextQuery

3from searx.testing import SearxTestCase

5from searx.testing import SearxTestCase

4

6

7+import searx.engines

8+

5

9

6TEST_ENGINES = [

10TEST_ENGINES = [

7 {

11 {

@@ -277,9 +281,10 @@ class TestBang(SearxTestCase):

277 self.assertEqual(query.getQuery(), '!dum the query')

281 self.assertEqual(query.getQuery(), '!dum the query')

278

282

279 def test_bang_autocomplete_empty(self):

283 def test_bang_autocomplete_empty(self):

284+ with patch.object(searx.engines, 'initialize_engines', searx.engines.load_engines):

280− initialize()

285+ initialize()

281− query = RawTextQuery('the query !', [])

286+ query = RawTextQuery('the query !', [])

282− self.assertEqual(query.autocomplete_list, ['!images', '!wikipedia', '!osm'])

287+ self.assertEqual(query.autocomplete_list, ['!images', '!wikipedia', '!osm'])

283

288

284− query = RawTextQuery('the query ?', ['osm'])

289+ query = RawTextQuery('the query ?', ['osm'])

285− self.assertEqual(query.autocomplete_list, ['?images', '?wikipedia'])

290+ self.assertEqual(query.autocomplete_list, ['?images', '?wikipedia'])

tests/unit/test_webapp.py+6−1

@@ -3,14 +3,19 @@

3import json

4from urllib.parse import ParseResult

5from mock import Mock

6−from searx import webapp

7from searx.testing import SearxTestCase

6from searx.testing import SearxTestCase

8from searx.search import Search

7from searx.search import Search

8+import searx.engines

9

10

11class ViewsTestCase(SearxTestCase):

12

13 def setUp(self):

14+ # skip init function (no external HTTP request)

15+ self.setattr4test(searx.engines, 'initialize_engines', searx.engines.load_engines)

16+

17+ from searx import webapp # pylint disable=import-outside-toplevel

18+

14 webapp.app.config['TESTING'] = True # to get better error messages

19 webapp.app.config['TESTING'] = True # to get better error messages

15 self.app = webapp.app.test_client()

20 self.app = webapp.app.test_client()

16

21