git.iko.soy/searx
on master help sign in
[enh] settings.yml: add use_default_settings option (2nd version)
commitb4b81a5e1a74f03926e3c2e6f8c5fade99f7eabb authorAlexandre Flament <alex@al-f.net> dateFri 27 Nov 2020, 19:32 (6 years ago) parent1cfe7f2
14 files changed+437−249
compare with master
unifiedsplit
Makefile+1−16 docs/admin/settings.rst+25−42 searx/__init__.py+2−2 searx/settings.py−91 searx/settings_loader.py+129 tests/unit/settings/empty_settings.yml tests/unit/settings/syntaxerror_settings.yml+2 tests/unit/settings/user_settings.yml+111 tests/unit/settings/user_settings_keep_only.yml+14 tests/unit/settings/user_settings_remove.yml+10 tests/unit/settings/user_settings_remove2.yml+15 tests/unit/settings/user_settings_simple.yml+6 tests/unit/test_settings_loader.py+122 utils/update_user_settings.py−98
Makefile+1−16
@@ -266,19 +266,4 @@ test.clean:
266266travis.codecov:
267267 $(Q)$(PY_ENV_BIN)/python -m pip install codecov
268268
269−
270−# user-settings
271−# -------------
272−
273−PHONY += user-settings.create user-settings.update
274−
275−user-settings.update: pyenvinstall
276− $(Q)$(PY_ENV_ACT); pip install ruamel.yaml
277− $(Q)$(PY_ENV_ACT); python utils/update_user_settings.py ${SEARX_SETTINGS_PATH}
278−
279−user-settings.update.engines: pyenvinstall
280− $(Q)$(PY_ENV_ACT); pip install ruamel.yaml
281− $(Q)$(PY_ENV_ACT); python utils/update_user_settings.py --add-engines ${SEARX_SETTINGS_PATH}
282−
283−
284−.PHONY: $(PHONY)
269+.PHONY: $(PHONY)
docs/admin/settings.rst+25−42
@@ -235,68 +235,51 @@ In the following example, the actual settings are the default settings defined i
235235
236236.. code-block:: yaml
237237
238− use_default_settings: true
238+ use_default_settings: True
239239 server:
240240 secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
241− server:
242241 bind_address: "0.0.0.0"
243242
244−With ``use_default_settings: True``, each settings can be override in a similar way with one exception, the ``engines`` section:
245−
246−* If the ``engines`` section is not defined in the user settings, searx uses the engines from the default setttings (the above example).
247−* If the ``engines`` section is defined then:
248−
249− * searx loads only the engines declare in the user setttings.
250− * searx merges the configuration according to the engine name.
243+With ``use_default_settings: True``, each settings can be override in a similar way, the ``engines`` section is merged according to the engine ``name``.
251244
252−In the following example, only three engines are available. Each engine configuration is merged with the default configuration.
245+In this example, searx will load all the engine and the arch linux wiki engine has a :ref:`token<private engines>`:
253246
254247.. code-block:: yaml
255248
256− use_default_settings: true
249+ use_default_settings: True
257250 server:
258251 secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
259252 engines:
260− - name: wikipedia
261− - name: wikidata
262− - name: ddg definitions
253+ - name: arch linux wiki
254+ tokens: ['$ecretValue']
263255
264−Another example where four engines are available. The arch linux wiki engine has a :ref:`token<private engines>`.
256+It is possible to remove some engines from the default settings. The following example is similar to the above one, but searx doesn't load the the google engine:
265257
266258.. code-block:: yaml
267259
268− use_default_settings: true
260+ use_default_settings:
261+ engines:
262+ remove:
263+ - google
269264 server:
270265 secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
271266 engines:
272267 - name: arch linux wiki
273268 tokens: ['$ecretValue']
274− - name: wikipedia
275− - name: wikidata
276− - name: ddg definitions
277−
278−automatic update
279−----------------
280−
281−The following comand creates or updates a minimal user settings (a secret key is defined if it is not already the case):
282−
283−.. code-block:: sh
284−
285− make SEARX_SETTINGS_PATH=/etc/searx/settings.yml user-settings.update
286−
287−Set ``SEARX_SETTINGS_PATH`` to your user settings path.
288269
289−As soon the user settings contains an ``engines`` section, it becomes difficult to keep the engine list updated.
290−The following command creates or updates the user settings including the ``engines`` section:
270+As an alternative, it is possible to specify the engines to keep. In the following example, searx has only two engines:
291271
292−.. code-block:: sh
293−
294− make SEARX_SETTINGS_PATH=/etc/searx/settings.yml user-settings.update.engines
295−
296−After that ``/etc/searx/settings.yml``
297−
298−* has a ``secret key``
299−* has a ``engine`` section if it is not already the case, moreover the command:
272+.. code-block:: yaml
300273
301− * has deleted engines that do not exist in the default settings.
302− * has added engines that exist in the default settings but are not declare in the user settings.
274+ use_default_settings:
275+ engines:
276+ keep_only:
277+ - google
278+ - duckduckgo
279+ server:
280+ secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
281+ engines:
282+ - name: google
283+ tokens: ['$ecretValue']
284+ - name: duckduckgo
285+ tokens: ['$ecretValue']
searx/__init__.py+2−2
@@ -16,7 +16,7 @@ along with searx. If not, see < http://www.gnu.org/licenses/ >.
1616'''
1717
1818import logging
19−import searx.settings
19+import searx.settings_loader
2020from os import environ
2121from os.path import realpath, dirname, join, abspath, isfile
2222
@@ -24,7 +24,7 @@ from os.path import realpath, dirname, join, abspath, isfile
2424searx_dir = abspath(dirname(__file__))
2525engine_dir = dirname(realpath(__file__))
2626static_path = abspath(join(dirname(__file__), 'static'))
27−settings, settings_load_message = searx.settings.load_settings()
27+settings, settings_load_message = searx.settings_loader.load_settings()
2828
2929if settings['ui']['static_path']:
3030 static_path = settings['ui']['static_path']
searx/settings.pydeleted−91
@@ -1,91 +0,0 @@
1−import collections.abc
2−
3−import yaml
4−from searx.exceptions import SearxSettingsException
5−from os import environ
6−from os.path import dirname, join, abspath, isfile
7−
8−
9−searx_dir = abspath(dirname(__file__))
10−
11−
12−def check_settings_yml(file_name):
13− if isfile(file_name):
14− return file_name
15− else:
16− return None
17−
18−
19−def load_yaml(file_name):
20− try:
21− with open(file_name, 'r', encoding='utf-8') as settings_yaml:
22− settings = yaml.safe_load(settings_yaml)
23− if not isinstance(settings, dict) or len(settings) == 0:
24− raise SearxSettingsException('Empty file', file_name)
25− return settings
26− except IOError as e:
27− raise SearxSettingsException(e, file_name)
28− except yaml.YAMLError as e:
29− raise SearxSettingsException(e, file_name)
30−
31−
32−def get_default_settings_path():
33− return check_settings_yml(join(searx_dir, 'settings.yml'))
34−
35−
36−def get_user_settings_path():
37− # find location of settings.yml
38− if 'SEARX_SETTINGS_PATH' in environ:
39− # if possible set path to settings using the
40− # enviroment variable SEARX_SETTINGS_PATH
41− return check_settings_yml(environ['SEARX_SETTINGS_PATH'])
42− else:
43− # if not, get it from searx code base or last solution from /etc/searx
44− return check_settings_yml('/etc/searx/settings.yml')
45−
46−
47−def update_dict(d, u):
48− for k, v in u.items():
49− if isinstance(v, collections.abc.Mapping):
50− d[k] = update_dict(d.get(k, {}), v)
51− else:
52− d[k] = v
53− return d
54−
55−
56−def update_settings(default_settings, user_settings):
57− for k, v in user_settings.items():
58− if k == 'use_default_settings':
59− continue
60− elif k == 'engines':
61− default_engines = default_settings[k]
62− default_engines_dict = dict((definition['name'], definition) for definition in default_engines)
63− default_settings[k] = [update_dict(default_engines_dict[definition['name']], definition)
64− for definition in v]
65− else:
66− update_dict(default_settings[k], v)
67−
68− return default_settings
69−
70−
71−def load_settings(load_user_setttings=True):
72− default_settings_path = get_default_settings_path()
73− user_settings_path = get_user_settings_path()
74− if user_settings_path is None or not load_user_setttings:
75− # no user settings
76− return (load_yaml(default_settings_path),
77− 'load the default settings from {}'.format(default_settings_path))
78−
79− # user settings
80− user_settings = load_yaml(user_settings_path)
81− if user_settings.get('use_default_settings'):
82− # the user settings are merged with the default configuration
83− default_settings = load_yaml(default_settings_path)
84− update_settings(default_settings, user_settings)
85− return (default_settings,
86− 'merge the default settings ( {} ) and the user setttings ( {} )'
87− .format(default_settings_path, user_settings_path))
88−
89− # the user settings, fully replace the default configuration
90− return (user_settings,
91− 'load the user settings from {}'.format(user_settings_path))
searx/settings_loader.pynew file+129
@@ -0,0 +1,129 @@
1+# SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+from os import environ
4+from os.path import dirname, join, abspath, isfile
5+from collections.abc import Mapping
6+from itertools import filterfalse
7+
8+import yaml
9+
10+from searx.exceptions import SearxSettingsException
11+
12+
13+searx_dir = abspath(dirname(__file__))
14+
15+
16+def check_settings_yml(file_name):
17+ if isfile(file_name):
18+ return file_name
19+ return None
20+
21+
22+def load_yaml(file_name):
23+ try:
24+ with open(file_name, 'r', encoding='utf-8') as settings_yaml:
25+ return yaml.safe_load(settings_yaml)
26+ except IOError as e:
27+ raise SearxSettingsException(e, file_name)
28+ except yaml.YAMLError as e:
29+ raise SearxSettingsException(e, file_name)
30+
31+
32+def get_default_settings_path():
33+ return check_settings_yml(join(searx_dir, 'settings.yml'))
34+
35+
36+def get_user_settings_path():
37+ # find location of settings.yml
38+ if 'SEARX_SETTINGS_PATH' in environ:
39+ # if possible set path to settings using the
40+ # enviroment variable SEARX_SETTINGS_PATH
41+ return check_settings_yml(environ['SEARX_SETTINGS_PATH'])
42+
43+ # if not, get it from searx code base or last solution from /etc/searx
44+ return check_settings_yml('/etc/searx/settings.yml')
45+
46+
47+def update_dict(default_dict, user_dict):
48+ for k, v in user_dict.items():
49+ if isinstance(v, Mapping):
50+ default_dict[k] = update_dict(default_dict.get(k, {}), v)
51+ else:
52+ default_dict[k] = v
53+ return default_dict
54+
55+
56+def update_settings(default_settings, user_settings):
57+ # merge everything except the engines
58+ for k, v in user_settings.items():
59+ if k not in ('use_default_settings', 'engines'):
60+ update_dict(default_settings[k], v)
61+
62+ # parse the engines
63+ remove_engines = None
64+ keep_only_engines = None
65+ use_default_settings = user_settings.get('use_default_settings')
66+ if isinstance(use_default_settings, dict):
67+ remove_engines = use_default_settings.get('engines', {}).get('remove')
68+ keep_only_engines = use_default_settings.get('engines', {}).get('keep_only')
69+
70+ if 'engines' in user_settings or remove_engines is not None or keep_only_engines is not None:
71+ engines = default_settings['engines']
72+
73+ # parse "use_default_settings.engines.remove"
74+ if remove_engines is not None:
75+ engines = list(filterfalse(lambda engine: (engine.get('name')) in remove_engines, engines))
76+
77+ # parse "use_default_settings.engines.keep_only"
78+ if keep_only_engines is not None:
79+ engines = list(filter(lambda engine: (engine.get('name')) in keep_only_engines, engines))
80+
81+ # parse "engines"
82+ user_engines = user_settings.get('engines')
83+ if user_engines:
84+ engines_dict = dict((definition['name'], definition) for definition in engines)
85+ for user_engine in user_engines:
86+ default_engine = engines_dict.get(user_engine['name'])
87+ if default_engine:
88+ update_dict(default_engine, user_engine)
89+ else:
90+ engines.append(user_engine)
91+
92+ # store the result
93+ default_settings['engines'] = engines
94+
95+ return default_settings
96+
97+
98+def is_use_default_settings(user_settings):
99+ use_default_settings = user_settings.get('use_default_settings')
100+ if use_default_settings is True:
101+ return True
102+ if isinstance(use_default_settings, dict):
103+ return True
104+ if use_default_settings is False or use_default_settings is None:
105+ return False
106+ raise ValueError('Invalid value for use_default_settings')
107+
108+
109+def load_settings(load_user_setttings=True):
110+ default_settings_path = get_default_settings_path()
111+ user_settings_path = get_user_settings_path()
112+ if user_settings_path is None or not load_user_setttings:
113+ # no user settings
114+ return (load_yaml(default_settings_path),
115+ 'load the default settings from {}'.format(default_settings_path))
116+
117+ # user settings
118+ user_settings = load_yaml(user_settings_path)
119+ if is_use_default_settings(user_settings):
120+ # the user settings are merged with the default configuration
121+ default_settings = load_yaml(default_settings_path)
122+ update_settings(default_settings, user_settings)
123+ return (default_settings,
124+ 'merge the default settings ( {} ) and the user setttings ( {} )'
125+ .format(default_settings_path, user_settings_path))
126+
127+ # the user settings, fully replace the default configuration
128+ return (user_settings,
129+ 'load the user settings from {}'.format(user_settings_path))
tests/unit/settings/empty_settings.ymlnew file

No changes to the content.

tests/unit/settings/syntaxerror_settings.ymlnew file+2
@@ -0,0 +1,2 @@
1+Test:
2+ **********
tests/unit/settings/user_settings.ymlnew file+111
@@ -0,0 +1,111 @@
1+general:
2+ debug : False
3+ instance_name : "searx"
4+
5+search:
6+ safe_search : 0
7+ autocomplete : ""
8+ default_lang : ""
9+ ban_time_on_fail : 5
10+ max_ban_time_on_fail : 120
11+
12+server:
13+ port : 9000
14+ bind_address : "0.0.0.0"
15+ secret_key : "user_settings_secret"
16+ base_url : False
17+ image_proxy : False
18+ http_protocol_version : "1.0"
19+ method: "POST"
20+ default_http_headers:
21+ X-Content-Type-Options : nosniff
22+ X-XSS-Protection : 1; mode=block
23+ X-Download-Options : noopen
24+ X-Robots-Tag : noindex, nofollow
25+ Referrer-Policy : no-referrer
26+
27+ui:
28+ static_path : ""
29+ templates_path : ""
30+ default_theme : oscar
31+ default_locale : ""
32+ theme_args :
33+ oscar_style : logicodev
34+
35+engines:
36+ - name : wikidata
37+ engine : wikidata
38+ shortcut : wd
39+ timeout : 3.0
40+ weight : 2
41+
42+ - name : wikibooks
43+ engine : mediawiki
44+ shortcut : wb
45+ categories : general
46+ base_url : "https://{language}.wikibooks.org/"
47+ number_of_results : 5
48+ search_type : text
49+
50+ - name : wikinews
51+ engine : mediawiki
52+ shortcut : wn
53+ categories : news
54+ base_url : "https://{language}.wikinews.org/"
55+ number_of_results : 5
56+ search_type : text
57+
58+ - name : wikiquote
59+ engine : mediawiki
60+ shortcut : wq
61+ categories : general
62+ base_url : "https://{language}.wikiquote.org/"
63+ number_of_results : 5
64+ search_type : text
65+
66+locales:
67+ en : English
68+ ar : العَرَبِيَّة (Arabic)
69+ bg : Български (Bulgarian)
70+ bo : བོད་སྐད་ (Tibetian)
71+ ca : Català (Catalan)
72+ cs : Čeština (Czech)
73+ cy : Cymraeg (Welsh)
74+ da : Dansk (Danish)
75+ de : Deutsch (German)
76+ el_GR : Ελληνικά (Greek_Greece)
77+ eo : Esperanto (Esperanto)
78+ es : Español (Spanish)
79+ et : Eesti (Estonian)
80+ eu : Euskara (Basque)
81+ fa_IR : (fārsī) فارسى (Persian)
82+ fi : Suomi (Finnish)
83+ fil : Wikang Filipino (Filipino)
84+ fr : Français (French)
85+ gl : Galego (Galician)
86+ he : עברית (Hebrew)
87+ hr : Hrvatski (Croatian)
88+ hu : Magyar (Hungarian)
89+ ia : Interlingua (Interlingua)
90+ it : Italiano (Italian)
91+ ja : 日本語 (Japanese)
92+ lt : Lietuvių (Lithuanian)
93+ nl : Nederlands (Dutch)
94+ nl_BE : Vlaams (Dutch_Belgium)
95+ oc : Lenga D'òc (Occitan)
96+ pl : Polski (Polish)
97+ pt : Português (Portuguese)
98+ pt_BR : Português (Portuguese_Brazil)
99+ ro : Română (Romanian)
100+ ru : Русский (Russian)
101+ sk : Slovenčina (Slovak)
102+ sl : Slovenski (Slovene)
103+ sr : српски (Serbian)
104+ sv : Svenska (Swedish)
105+ te : తెలుగు (telugu)
106+ ta : தமிழ் (Tamil)
107+ tr : Türkçe (Turkish)
108+ uk : українська мова (Ukrainian)
109+ vi : tiếng việt (Vietnamese)
110+ zh : 中文 (Chinese)
111+ zh_TW : 國語 (Taiwanese Mandarin)
tests/unit/settings/user_settings_keep_only.ymlnew file+14
@@ -0,0 +1,14 @@
1+use_default_settings:
2+ engines:
3+ keep_only:
4+ - wikibooks
5+ - wikinews
6+server:
7+ secret_key: "user_secret_key"
8+ bind_address: "0.0.0.0"
9+ default_http_headers:
10+ Custom-Header: Custom-Value
11+engines:
12+ - name: wikipedia
13+ - name: newengine
14+ engine: dummy
tests/unit/settings/user_settings_remove.ymlnew file+10
@@ -0,0 +1,10 @@
1+use_default_settings:
2+ engines:
3+ remove:
4+ - wikibooks
5+ - wikinews
6+server:
7+ secret_key: "user_secret_key"
8+ bind_address: "0.0.0.0"
9+ default_http_headers:
10+ Custom-Header: Custom-Value
tests/unit/settings/user_settings_remove2.ymlnew file+15
@@ -0,0 +1,15 @@
1+use_default_settings:
2+ engines:
3+ remove:
4+ - wikibooks
5+ - wikinews
6+server:
7+ secret_key: "user_secret_key"
8+ bind_address: "0.0.0.0"
9+ default_http_headers:
10+ Custom-Header: Custom-Value
11+engines:
12+ - name: wikipedia
13+ tokens: ['secret_token']
14+ - name: newengine
15+ engine: dummy
tests/unit/settings/user_settings_simple.ymlnew file+6
@@ -0,0 +1,6 @@
1+use_default_settings: True
2+server:
3+ secret_key: "user_secret_key"
4+ bind_address: "0.0.0.0"
5+ default_http_headers:
6+ Custom-Header: Custom-Value
tests/unit/test_settings_loader.pynew file+122
@@ -0,0 +1,122 @@
1+# SPDX-License-Identifier: AGPL-3.0-or-later
2+
3+from os.path import dirname, join, abspath
4+from unittest.mock import patch
5+
6+from searx.testing import SearxTestCase
7+from searx.exceptions import SearxSettingsException
8+from searx import settings_loader
9+
10+
11+test_dir = abspath(dirname(__file__))
12+
13+
14+class TestLoad(SearxTestCase):
15+
16+ def test_load_zero(self):
17+ with self.assertRaises(SearxSettingsException):
18+ settings_loader.load_yaml('/dev/zero')
19+
20+ with self.assertRaises(SearxSettingsException):
21+ settings_loader.load_yaml(join(test_dir, '/settings/syntaxerror_settings.yml'))
22+
23+ with self.assertRaises(SearxSettingsException):
24+ settings_loader.load_yaml(join(test_dir, '/settings/empty_settings.yml'))
25+
26+ def test_check_settings_yml(self):
27+ self.assertIsNone(settings_loader.check_settings_yml('/dev/zero'))
28+
29+ bad_settings_path = join(test_dir, 'settings/syntaxerror_settings.yml')
30+ self.assertEqual(settings_loader.check_settings_yml(bad_settings_path), bad_settings_path)
31+
32+
33+class TestDefaultSettings(SearxTestCase):
34+
35+ def test_load(self):
36+ settings, msg = settings_loader.load_settings(load_user_setttings=False)
37+ self.assertTrue(msg.startswith('load the default settings from'))
38+ self.assertFalse(settings['general']['debug'])
39+ self.assertTrue(isinstance(settings['general']['instance_name'], str))
40+ self.assertEqual(settings['server']['secret_key'], "ultrasecretkey")
41+ self.assertTrue(isinstance(settings['server']['port'], int))
42+ self.assertTrue(isinstance(settings['server']['bind_address'], str))
43+ self.assertTrue(isinstance(settings['engines'], list))
44+ self.assertTrue(isinstance(settings['locales'], dict))
45+ self.assertTrue(isinstance(settings['doi_resolvers'], dict))
46+ self.assertTrue(isinstance(settings['default_doi_resolver'], str))
47+
48+
49+class TestUserSettings(SearxTestCase):
50+
51+ def test_is_use_default_settings(self):
52+ self.assertFalse(settings_loader.is_use_default_settings({}))
53+ self.assertTrue(settings_loader.is_use_default_settings({'use_default_settings': True}))
54+ self.assertTrue(settings_loader.is_use_default_settings({'use_default_settings': {}}))
55+ with self.assertRaises(ValueError):
56+ self.assertFalse(settings_loader.is_use_default_settings({'use_default_settings': 1}))
57+ with self.assertRaises(ValueError):
58+ self.assertFalse(settings_loader.is_use_default_settings({'use_default_settings': 0}))
59+
60+ def test_user_settings_not_found(self):
61+ with patch.dict(settings_loader.environ,
62+ {'SEARX_SETTINGS_PATH': '/dev/null'}):
63+ settings, msg = settings_loader.load_settings()
64+ self.assertTrue(msg.startswith('load the default settings from'))
65+ self.assertEqual(settings['server']['secret_key'], "ultrasecretkey")
66+
67+ def test_user_settings(self):
68+ with patch.dict(settings_loader.environ,
69+ {'SEARX_SETTINGS_PATH': join(test_dir, 'settings/user_settings_simple.yml')}):
70+ settings, msg = settings_loader.load_settings()
71+ self.assertTrue(msg.startswith('merge the default settings'))
72+ self.assertEqual(settings['server']['secret_key'], "user_secret_key")
73+ self.assertEqual(settings['server']['default_http_headers']['Custom-Header'], "Custom-Value")
74+
75+ def test_user_settings_remove(self):
76+ with patch.dict(settings_loader.environ,
77+ {'SEARX_SETTINGS_PATH': join(test_dir, 'settings/user_settings_remove.yml')}):
78+ settings, msg = settings_loader.load_settings()
79+ self.assertTrue(msg.startswith('merge the default settings'))
80+ self.assertEqual(settings['server']['secret_key'], "user_secret_key")
81+ self.assertEqual(settings['server']['default_http_headers']['Custom-Header'], "Custom-Value")
82+ engine_names = [engine['name'] for engine in settings['engines']]
83+ self.assertNotIn('wikinews', engine_names)
84+ self.assertNotIn('wikibooks', engine_names)
85+ self.assertIn('wikipedia', engine_names)
86+
87+ def test_user_settings_remove2(self):
88+ with patch.dict(settings_loader.environ,
89+ {'SEARX_SETTINGS_PATH': join(test_dir, 'settings/user_settings_remove2.yml')}):
90+ settings, msg = settings_loader.load_settings()
91+ self.assertTrue(msg.startswith('merge the default settings'))
92+ self.assertEqual(settings['server']['secret_key'], "user_secret_key")
93+ self.assertEqual(settings['server']['default_http_headers']['Custom-Header'], "Custom-Value")
94+ engine_names = [engine['name'] for engine in settings['engines']]
95+ self.assertNotIn('wikinews', engine_names)
96+ self.assertNotIn('wikibooks', engine_names)
97+ self.assertIn('wikipedia', engine_names)
98+ wikipedia = list(filter(lambda engine: (engine.get('name')) == 'wikipedia', settings['engines']))
99+ self.assertEqual(wikipedia[0]['engine'], 'wikipedia')
100+ self.assertEqual(wikipedia[0]['tokens'], ['secret_token'])
101+ newengine = list(filter(lambda engine: (engine.get('name')) == 'newengine', settings['engines']))
102+ self.assertEqual(newengine[0]['engine'], 'dummy')
103+
104+ def test_user_settings_keep_only(self):
105+ with patch.dict(settings_loader.environ,
106+ {'SEARX_SETTINGS_PATH': join(test_dir, 'settings/user_settings_keep_only.yml')}):
107+ settings, msg = settings_loader.load_settings()
108+ self.assertTrue(msg.startswith('merge the default settings'))
109+ engine_names = [engine['name'] for engine in settings['engines']]
110+ self.assertEqual(engine_names, ['wikibooks', 'wikinews', 'wikipedia', 'newengine'])
111+ # wikipedia has been removed, then added again with the "engine" section of user_settings_keep_only.yml
112+ self.assertEqual(len(settings['engines'][2]), 1)
113+
114+ def test_custom_settings(self):
115+ with patch.dict(settings_loader.environ,
116+ {'SEARX_SETTINGS_PATH': join(test_dir, 'settings/user_settings.yml')}):
117+ settings, msg = settings_loader.load_settings()
118+ self.assertTrue(msg.startswith('load the user settings from'))
119+ self.assertEqual(settings['server']['port'], 9000)
120+ self.assertEqual(settings['server']['secret_key'], "user_settings_secret")
121+ engine_names = [engine['name'] for engine in settings['engines']]
122+ self.assertEqual(engine_names, ['wikidata', 'wikibooks', 'wikinews', 'wikiquote'])
utils/update_user_settings.pydeleted−98
@@ -1,98 +0,0 @@
1−#!/usr/bin/env python
2−
3−# set path
4−from sys import path
5−from os.path import realpath, dirname, join
6−path.append(realpath(dirname(realpath(__file__)) + '/../'))
7−
8−import argparse
9−import sys
10−import string
11−import ruamel.yaml
12−import secrets
13−import collections
14−from ruamel.yaml.scalarstring import SingleQuotedScalarString, DoubleQuotedScalarString
15−from searx.settings import load_settings, check_settings_yml, get_default_settings_path
16−from searx.exceptions import SearxSettingsException
17−
18−
19−RANDOM_STRING_LETTERS = string.ascii_lowercase + string.digits + string.ascii_uppercase
20−
21−
22−def get_random_string():
23− r = [secrets.choice(RANDOM_STRING_LETTERS) for _ in range(64)]
24− return ''.join(r)
25−
26−
27−def main(prog_arg):
28− yaml = ruamel.yaml.YAML()
29− yaml.preserve_quotes = True
30− yaml.indent(mapping=4, sequence=1, offset=2)
31− user_settings_path = prog_args.get('user-settings-yaml')
32−
33− try:
34− default_settings, _ = load_settings(False)
35− if check_settings_yml(user_settings_path):
36− with open(user_settings_path, 'r', encoding='utf-8') as f:
37− user_settings = yaml.load(f.read())
38− new_user_settings = False
39− else:
40− user_settings = yaml.load('use_default_settings: True')
41− new_user_settings = True
42− except SearxSettingsException as e:
43− sys.stderr.write(str(e))
44− return
45−
46− if not new_user_settings and not user_settings.get('use_default_settings'):
47− sys.stderr.write('settings.yml already exists and use_default_settings is not True')
48− return
49−
50− user_settings['use_default_settings'] = True
51− use_default_settings_comment = "settings based on " + get_default_settings_path()
52− user_settings.yaml_add_eol_comment(use_default_settings_comment, 'use_default_settings')
53−
54− if user_settings.get('server', {}).get('secret_key') in [None, 'ultrasecretkey']:
55− user_settings.setdefault('server', {})['secret_key'] = DoubleQuotedScalarString(get_random_string())
56−
57− user_engines = user_settings.get('engines')
58− if user_engines:
59− has_user_engines = True
60− user_engines_dict = dict((definition['name'], definition) for definition in user_engines)
61− else:
62− has_user_engines = False
63− user_engines_dict = {}
64− user_engines = []
65−
66− # remove old engines
67− if prog_arg.get('add-engines') or has_user_engines:
68− default_engines_dict = dict((definition['name'], definition) for definition in default_settings['engines'])
69− for i, engine in enumerate(user_engines):
70− if engine['name'] not in default_engines_dict:
71− del user_engines[i]
72−
73− # add new engines
74− if prog_arg.get('add-engines'):
75− for engine in default_settings.get('engines', {}):
76− if engine['name'] not in user_engines_dict:
77− user_engines.append({'name': engine['name']})
78− user_settings['engines'] = user_engines
79−
80− # output
81− if prog_arg.get('dry-run'):
82− yaml.dump(user_settings, sys.stdout)
83− else:
84− with open(user_settings_path, 'w', encoding='utf-8') as f:
85− yaml.dump(user_settings, f)
86−
87−
88−def parse_args():
89− parser = argparse.ArgumentParser(description='Update user settings.yml')
90− parser.add_argument('--add-engines', dest='add-engines', default=False, action='store_true', help='Add new engines')
91− parser.add_argument('--dry-run', dest='dry-run', default=False, action='store_true', help='Dry run')
92− parser.add_argument('user-settings-yaml', type=str)
93− return vars(parser.parse_args())
94−
95−
96−if __name__ == '__main__':
97− prog_args = parse_args()
98− main(prog_args)