925bb56 · searx · git.iko.soy

git.iko.soy/searx

Merge pull request #2352 from dalf/no_http

Remove HTTP connections as much as possible

commit925bb561a2797b6d3cc5be748c2361721b8e1f08 authorAlexandre Flament <alex@al-f.net> dateSun 6 Dec 2020, 10:18 (6 years ago) parentcdceec1 38d3233

13 files changed+28−114

compare with master

unifiedsplit

CONTRIBUTING.md+2−2 dockerfiles/docker-entrypoint.sh+1−1 docs/admin/buildhosts.rst+1−1 docs/admin/installation-nginx.rst+1−1 docs/admin/settings.rst+1−1 docs/conf.py+3−3 docs/dev/contribution_guide.rst+2−2 docs/dev/engine_overview.rst+1−1 docs/dev/quickstart.rst+1−1 docs/dev/reST.rst+12−12 searx/autocomplete.py+2−3 searx/engines/filecrop.py−85 utils/makefile.python+1−1

CONTRIBUTING.md+2−2

@@ -2,8 +2,8 @@

33## Resources in the documentation

5−* [Development quickstart](http://searx.github.io/searx/dev/contribution_guide.html)

6−* [Contribution guide](http://searx.github.io/searx/dev/contribution_guide.html)

5+* [Development quickstart](https://searx.github.io/searx/dev/contribution_guide.html)

6+* [Contribution guide](https://searx.github.io/searx/dev/contribution_guide.html)

88## Submitting PRs

dockerfiles/docker-entrypoint.sh+1−1

@@ -66,7 +66,7 @@ patch_searx_settings() {

6666 CONF="$1"

6767

6868 # Make sure that there is trailing slash at the end of BASE_URL

69− # see http://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Shell-Parameter-Expansion

69+ # see https://www.gnu.org/savannah-checkouts/gnu/bash/manual/bash.html#Shell-Parameter-Expansion

7070 export BASE_URL="${BASE_URL%/}/"

7171

7272 # update settings.yml

docs/admin/buildhosts.rst+1−1

@@ -67,7 +67,7 @@ to ``imgmath``:

6767If your docs build (``make docs``) shows warnings like this::

6868

6969 WARNING: dot(1) not found, for better output quality install \

70− graphviz from http://www.graphviz.org

70+ graphviz from https://www.graphviz.org

7171 ..

7272 WARNING: LaTeX command 'latex' cannot be run (needed for math \

7373 display), check the imgmath_latex setting

docs/admin/installation-nginx.rst+1−1

@@ -9,7 +9,7 @@ Install with nginx

99.. _nginx server configuration:

1010 https://docs.nginx.com/nginx/admin-guide/web-server/web-server/#setting-up-virtual-servers

1111.. _nginx beginners guide:

12− http://nginx.org/en/docs/beginners_guide.html

12+ https://nginx.org/en/docs/beginners_guide.html

1313.. _Getting Started wiki:

1414 https://www.nginx.com/resources/wiki/start/

1515.. _uWSGI support from nginx:

docs/admin/settings.rst+1−1

@@ -112,7 +112,7 @@ Global Settings

112112 specific instance of searx, a locale can be defined using an ISO language

113113 code, like ``fr``, ``en``, ``de``.

114114

115−.. _requests proxies: http://requests.readthedocs.io/en/latest/user/advanced/#proxies

115+.. _requests proxies: https://requests.readthedocs.io/en/latest/user/advanced/#proxies

116116.. _PySocks: https://pypi.org/project/PySocks/

117117

118118``proxies`` :

docs/conf.py+3−3

@@ -49,11 +49,11 @@ extlinks['man'] = ('https://manpages.debian.org/jump?q=%s', '')

4949#extlinks['role'] = (

5050# 'https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html#role-%s', '')

5151extlinks['duref'] = (

52− 'http://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#%s', '')

52+ 'https://docutils.sourceforge.net/docs/ref/rst/restructuredtext.html#%s', '')

5353extlinks['durole'] = (

54− 'http://docutils.sourceforge.net/docs/ref/rst/roles.html#%s', '')

54+ 'https://docutils.sourceforge.net/docs/ref/rst/roles.html#%s', '')

5555extlinks['dudir'] = (

56− 'http://docutils.sourceforge.net/docs/ref/rst/directives.html#%s', '')

56+ 'https://docutils.sourceforge.net/docs/ref/rst/directives.html#%s', '')

5757extlinks['ctan'] = (

5858 'https://ctan.org/pkg/%s', 'CTAN: ')

5959

docs/dev/contribution_guide.rst+2−2

@@ -117,8 +117,8 @@ Translation currently takes place on :ref:`transifex <translation>`.

117117Documentation

118118=============

119119

120−.. _Sphinx: http://www.sphinx-doc.org

121−.. _reST: http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html

120+.. _Sphinx: https://www.sphinx-doc.org

121+.. _reST: https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html

122122

123123.. sidebar:: The reST sources

124124

docs/dev/engine_overview.rst+1−1

@@ -265,7 +265,7 @@ latitude latitude of result (in decimal format)

265265longitude longitude of result (in decimal format)

266266boundingbox boundingbox of result (array of 4. values

267267 ``[lat-min, lat-max, lon-min, lon-max]``)

268−geojson geojson of result (http://geojson.org)

268+geojson geojson of result (https://geojson.org/)

269269osm.type type of osm-object (if OSM-Result)

270270osm.id id of osm-object (if OSM-Result)

271271address.name name of object

docs/dev/quickstart.rst+1−1

@@ -60,7 +60,7 @@ read :ref:`make test`.

6060How to compile styles and javascript

6161====================================

6262

63−.. _less: http://lesscss.org/

63+.. _less: https://lesscss.org/

6464.. _NodeJS: https://nodejs.org

6565

6666How to build styles

docs/dev/reST.rst+12−12

@@ -1391,27 +1391,27 @@ The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac``

13911391

13921392.. _readability: https://docs.python-guide.org/writing/style/

13931393.. _Sphinx-Primer:

1394− http://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html

1394+ https://www.sphinx-doc.org/en/master/usage/restructuredtext/basics.html

13951395.. _reST: https://docutils.sourceforge.io/rst.html

13961396.. _Sphinx Roles:

13971397 https://www.sphinx-doc.org/en/master/usage/restructuredtext/roles.html

1398−.. _Sphinx: http://www.sphinx-doc.org

1399−.. _`sphinx-doc FAQ`: http://www.sphinx-doc.org/en/stable/faq.html

1398+.. _Sphinx: https://www.sphinx-doc.org

1399+.. _`sphinx-doc FAQ`: https://www.sphinx-doc.org/en/stable/faq.html

14001400.. _Sphinx markup constructs:

1401− http://www.sphinx-doc.org/en/stable/markup/index.html

1401+ https://www.sphinx-doc.org/en/stable/markup/index.html

14021402.. _`sphinx cross references`:

1403− http://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations

1403+ https://www.sphinx-doc.org/en/stable/markup/inline.html#cross-referencing-arbitrary-locations

14041404.. _sphinx.ext.extlinks:

14051405 https://www.sphinx-doc.org/en/master/usage/extensions/extlinks.html

1406−.. _intersphinx: http://www.sphinx-doc.org/en/stable/ext/intersphinx.html

1407−.. _sphinx config: http://www.sphinx-doc.org/en/stable/config.html

1408−.. _Sphinx's autodoc: http://www.sphinx-doc.org/en/stable/ext/autodoc.html

1406+.. _intersphinx: https://www.sphinx-doc.org/en/stable/ext/intersphinx.html

1407+.. _sphinx config: https://www.sphinx-doc.org/en/stable/config.html

1408+.. _Sphinx's autodoc: https://www.sphinx-doc.org/en/stable/ext/autodoc.html

14091409.. _Sphinx's Python domain:

1410− http://www.sphinx-doc.org/en/stable/domains.html#the-python-domain

1410+ https://www.sphinx-doc.org/en/stable/domains.html#the-python-domain

14111411.. _Sphinx's C domain:

1412− http://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs

1412+ https://www.sphinx-doc.org/en/stable/domains.html#cross-referencing-c-constructs

14131413.. _doctree:

1414− http://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases

1414+ https://www.sphinx-doc.org/en/master/extdev/tutorial.html?highlight=doctree#build-phases

14151415.. _docutils: http://docutils.sourceforge.net/docs/index.html

14161416.. _docutils FAQ: http://docutils.sourceforge.net/FAQ.html

14171417.. _linuxdoc: https://return42.github.io/linuxdoc

@@ -1424,5 +1424,5 @@ The next example shows the difference of ``\tfrac`` (*textstyle*) and ``\dfrac``

14241424.. _ImageMagick: https://www.imagemagick.org

14251425

14261426.. _`Emacs Table Mode`: https://www.emacswiki.org/emacs/TableMode

1427−.. _`Online Tables Generator`: http://www.tablesgenerator.com/text_tables

1427+.. _`Online Tables Generator`: https://www.tablesgenerator.com/text_tables

14281428.. _`OASIS XML Exchange Table Model`: https://www.oasis-open.org/specs/tm9901.html

searx/autocomplete.py+2−3

@@ -113,7 +113,7 @@ def searx_bang(full_query):

113113

114114def dbpedia(query, lang):

115115 # dbpedia autocompleter, no HTTPS

116− autocomplete_url = 'http://lookup.dbpedia.org/api/search.asmx/KeywordSearch?'

116+ autocomplete_url = 'https://lookup.dbpedia.org/api/search.asmx/KeywordSearch?'

117117

118118 response = get(autocomplete_url + urlencode(dict(QueryString=query)))

119119

@@ -121,8 +121,7 @@ def dbpedia(query, lang):

121121

122122 if response.ok:

123123 dom = etree.fromstring(response.content)

124− results = dom.xpath('//a:Result/a:Label//text()',

125− namespaces={'a': 'http://lookup.dbpedia.org/'})

124+ results = dom.xpath('//Result/Label//text()')

126125

127126 return results

128127

searx/engines/filecrop.pydeleted−85

@@ -1,85 +0,0 @@

1−from html.parser import HTMLParser

2−from urllib.parse import urlencode

3−

4−

5−url = 'http://www.filecrop.com/'

6−search_url = url + '/search.php?{query}&size_i=0&size_f=100000000&engine_r=1&engine_d=1&engine_e=1&engine_4=1&engine_m=1&pos={index}' # noqa

7−

8−paging = True

9−

10−

11−class FilecropResultParser(HTMLParser): # pylint: disable=W0223 # (see https://bugs.python.org/issue31844)

12−

13− def __init__(self):

14− HTMLParser.__init__(self)

15− self.__start_processing = False

16−

17− self.results = []

18− self.result = {}

19−

20− self.tr_counter = 0

21− self.data_counter = 0

22−

23− def handle_starttag(self, tag, attrs):

24−

25− if tag == 'tr':

26− if ('bgcolor', '#edeff5') in attrs or\

27− ('bgcolor', '#ffffff') in attrs:

28− self.__start_processing = True

29−

30− if not self.__start_processing:

31− return

32−

33− if tag == 'label':

34− self.result['title'] = [attr[1] for attr in attrs

35− if attr[0] == 'title'][0]

36− elif tag == 'a' and ('rel', 'nofollow') in attrs\

37− and ('class', 'sourcelink') in attrs:

38− if 'content' in self.result:

39− self.result['content'] += [attr[1] for attr in attrs

40− if attr[0] == 'title'][0]

41− else:

42− self.result['content'] = [attr[1] for attr in attrs

43− if attr[0] == 'title'][0]

44− self.result['content'] += ' '

45− elif tag == 'a':

46− self.result['url'] = url + [attr[1] for attr in attrs

47− if attr[0] == 'href'][0]

48−

49− def handle_endtag(self, tag):

50− if self.__start_processing is False:

51− return

52−

53− if tag == 'tr':

54− self.tr_counter += 1

55−

56− if self.tr_counter == 2:

57− self.__start_processing = False

58− self.tr_counter = 0

59− self.data_counter = 0

60− self.results.append(self.result)

61− self.result = {}

62−

63− def handle_data(self, data):

64− if not self.__start_processing:

65− return

66−

67− if 'content' in self.result:

68− self.result['content'] += data + ' '

69− else:

70− self.result['content'] = data + ' '

71−

72− self.data_counter += 1

73−

74−

75−def request(query, params):

76− index = 1 + (params['pageno'] - 1) * 30

77− params['url'] = search_url.format(query=urlencode({'w': query}), index=index)

78− return params

79−

80−

81−def response(resp):

82− parser = FilecropResultParser()

83− parser.feed(resp.text)

84−

85− return parser.results

utils/makefile.python+1−1

@@ -252,7 +252,7 @@ pyenv-python: pyenv-install

252252# PyPi is required and since uploads via setuptools is not recommended, we have

253253# to imstall / use twine ... its really a mess.

254254#

255−# [1] http://python-packaging.readthedocs.io/en/latest/dependencies.html#packages-not-on-pypi

255+# [1] https://python-packaging.readthedocs.io/en/latest/dependencies.html#packages-not-on-pypi

256256# [2] https://github.com/pypa/pip/pull/1519

257257

258258# https://github.com/pypa/twine