on master help sign in
[enh] settings.yml: add use_default_settings option

This change is backward compatible with the existing configurations. If a settings.yml loaded from an user defined location (SEARX_SETTINGS_PATH or /etc/searx/settings.yml), then this settings can relied on the default settings.yml with this option: user_default_settings:True

commit1cfe7f2a7543b2994a1afd0d81da1962d04423b0 authorAlexandre Flament <alex@al-f.net> dateTue 3 Nov 2020, 15:29 (6 years ago) parent6ada5ba
7 files changed+312−29
Makefile+16−1 docs/admin/settings.rst+94 docs/blog/private-engines.rst+2 searx/__init__.py+3−28 searx/exceptions.py+8 searx/settings.py+91 utils/update_user_settings.py+98
Makefile+16−1
@@ -266,4 +266,19 @@ test.clean:
266266travis.codecov:
267267 $(Q)$(PY_ENV_BIN)/python -m pip install codecov
268268
269−.PHONY: $(PHONY)
269+
270+# user-settings
271+# -------------
272+
273+PHONY += user-settings.create user-settings.update
274+
275+user-settings.update: pyenvinstall
276+ $(Q)$(PY_ENV_ACT); pip install ruamel.yaml
277+ $(Q)$(PY_ENV_ACT); python utils/update_user_settings.py ${SEARX_SETTINGS_PATH}
278+
279+user-settings.update.engines: pyenvinstall
280+ $(Q)$(PY_ENV_ACT); pip install ruamel.yaml
281+ $(Q)$(PY_ENV_ACT); python utils/update_user_settings.py --add-engines ${SEARX_SETTINGS_PATH}
282+
283+
284+.PHONY: $(PHONY)
docs/admin/settings.rst+94
@@ -206,3 +206,97 @@ Engine settings
206206
207207 A few more options are possible, but they are pretty specific to some
208208 engines, and so won't be described here.
209+
210+
211+.. _settings location:
212+
213+settings.yml location
214+=====================
215+
216+First, searx will try to load settings.yml from these locations:
217+
218+1. the full path specified in the ``SEARX_SETTINGS_PATH`` environment variable.
219+2. ``/etc/searx/settings.yml``
220+
221+If these files don't exist (or are empty or can't be read), searx uses the :origin:`searx/settings.yml` file.
222+
223+.. _ settings use_default_settings:
224+
225+use_default_settings
226+====================
227+
228+.. note::
229+
230+ If searx is cloned from a git repository, most probably there is no need to have an user settings.
231+
232+The user defined settings.yml can relied on the default configuration :origin:`searx/settings.yml` using ``use_default_settings: True``.
233+
234+In the following example, the actual settings are the default settings defined in :origin:`searx/settings.yml` with the exception of the ``secret_key`` and the ``bind_address``:
235+
236+.. code-block:: yaml
237+
238+ use_default_settings: true
239+ server:
240+ secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
241+ server:
242+ bind_address: "0.0.0.0"
243+
244+With ``use_default_settings: True``, each settings can be override in a similar way with one exception, the ``engines`` section:
245+
246+* If the ``engines`` section is not defined in the user settings, searx uses the engines from the default setttings (the above example).
247+* If the ``engines`` section is defined then:
248+
249+ * searx loads only the engines declare in the user setttings.
250+ * searx merges the configuration according to the engine name.
251+
252+In the following example, only three engines are available. Each engine configuration is merged with the default configuration.
253+
254+.. code-block:: yaml
255+
256+ use_default_settings: true
257+ server:
258+ secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
259+ engines:
260+ - name: wikipedia
261+ - name: wikidata
262+ - name: ddg definitions
263+
264+Another example where four engines are available. The arch linux wiki engine has a :ref:`token<private engines>`.
265+
266+.. code-block:: yaml
267+
268+ use_default_settings: true
269+ server:
270+ secret_key: "uvys6bRhKHUdFF5CqbJonSDSRN8H0sCBziNSrDGNVdpz7IeZhveVart3yvghoKHA"
271+ engines:
272+ - name: arch linux wiki
273+ tokens: ['$ecretValue']
274+ - name: wikipedia
275+ - name: wikidata
276+ - name: ddg definitions
277+
278+automatic update
279+----------------
280+
281+The following comand creates or updates a minimal user settings (a secret key is defined if it is not already the case):
282+
283+.. code-block:: sh
284+
285+ make SEARX_SETTINGS_PATH=/etc/searx/settings.yml user-settings.update
286+
287+Set ``SEARX_SETTINGS_PATH`` to your user settings path.
288+
289+As soon the user settings contains an ``engines`` section, it becomes difficult to keep the engine list updated.
290+The following command creates or updates the user settings including the ``engines`` section:
291+
292+.. code-block:: sh
293+
294+ make SEARX_SETTINGS_PATH=/etc/searx/settings.yml user-settings.update.engines
295+
296+After that ``/etc/searx/settings.yml``
297+
298+* has a ``secret key``
299+* has a ``engine`` section if it is not already the case, moreover the command:
300+
301+ * has deleted engines that do not exist in the default settings.
302+ * has added engines that exist in the default settings but are not declare in the user settings.
docs/blog/private-engines.rst+2
@@ -7,6 +7,8 @@ enabled engines on their instances. It might be because they do not want to
77expose some private information through an offline engine. Or they
88would rather share engines only with their trusted friends or colleagues.
99
10+.. _private engines:
11+
1012Private engines
1113===============
1214
searx/__init__.py+3−28
@@ -16,39 +16,15 @@ along with searx. If not, see < http://www.gnu.org/licenses/ >.
1616'''
1717
1818import logging
19+import searx.settings
1920from os import environ
2021from os.path import realpath, dirname, join, abspath, isfile
21−from io import open
22−from yaml import safe_load
2322
2423
2524searx_dir = abspath(dirname(__file__))
2625engine_dir = dirname(realpath(__file__))
2726static_path = abspath(join(dirname(__file__), 'static'))
28−
29−
30−def check_settings_yml(file_name):
31− if isfile(file_name):
32− return file_name
33− else:
34− return None
35−
36−
37−# find location of settings.yml
38−if 'SEARX_SETTINGS_PATH' in environ:
39− # if possible set path to settings using the
40− # enviroment variable SEARX_SETTINGS_PATH
41− settings_path = check_settings_yml(environ['SEARX_SETTINGS_PATH'])
42−else:
43− # if not, get it from searx code base or last solution from /etc/searx
44− settings_path = check_settings_yml(join(searx_dir, 'settings.yml')) or check_settings_yml('/etc/searx/settings.yml')
45−
46−if not settings_path:
47− raise Exception('settings.yml not found')
48−
49−# load settings
50−with open(settings_path, 'r', encoding='utf-8') as settings_yaml:
51− settings = safe_load(settings_yaml)
27+settings, settings_load_message = searx.settings.load_settings()
5228
5329if settings['ui']['static_path']:
5430 static_path = settings['ui']['static_path']
@@ -58,7 +34,6 @@ enable debug if
5834the environnement variable SEARX_DEBUG is 1 or true
5935(whatever the value in settings.yml)
6036or general.debug=True in settings.yml
61−
6237disable debug if
6338the environnement variable SEARX_DEBUG is 0 or false
6439(whatever the value in settings.yml)
@@ -78,7 +53,7 @@ else:
7853 logging.basicConfig(level=logging.WARNING)
7954
8055logger = logging.getLogger('searx')
81−logger.debug('read configuration from %s', settings_path)
56+logger.info(settings_load_message)
8257logger.info('Initialisation done')
8358
8459if 'SEARX_SECRET' in environ:
searx/exceptions.py+8
@@ -31,3 +31,11 @@ class SearxParameterException(SearxException):
3131 self.message = message
3232 self.parameter_name = name
3333 self.parameter_value = value
34+
35+
36+class SearxSettingsException(SearxException):
37+
38+ def __init__(self, message, filename):
39+ super().__init__(message)
40+ self.message = message
41+ self.filename = filename
searx/settings.pynew file+91
@@ -0,0 +1,91 @@
1+import collections.abc
2+
3+import yaml
4+from searx.exceptions import SearxSettingsException
5+from os import environ
6+from os.path import dirname, join, abspath, isfile
7+
8+
9+searx_dir = abspath(dirname(__file__))
10+
11+
12+def check_settings_yml(file_name):
13+ if isfile(file_name):
14+ return file_name
15+ else:
16+ return None
17+
18+
19+def load_yaml(file_name):
20+ try:
21+ with open(file_name, 'r', encoding='utf-8') as settings_yaml:
22+ settings = yaml.safe_load(settings_yaml)
23+ if not isinstance(settings, dict) or len(settings) == 0:
24+ raise SearxSettingsException('Empty file', file_name)
25+ return settings
26+ except IOError as e:
27+ raise SearxSettingsException(e, file_name)
28+ except yaml.YAMLError as e:
29+ raise SearxSettingsException(e, file_name)
30+
31+
32+def get_default_settings_path():
33+ return check_settings_yml(join(searx_dir, 'settings.yml'))
34+
35+
36+def get_user_settings_path():
37+ # find location of settings.yml
38+ if 'SEARX_SETTINGS_PATH' in environ:
39+ # if possible set path to settings using the
40+ # enviroment variable SEARX_SETTINGS_PATH
41+ return check_settings_yml(environ['SEARX_SETTINGS_PATH'])
42+ else:
43+ # if not, get it from searx code base or last solution from /etc/searx
44+ return check_settings_yml('/etc/searx/settings.yml')
45+
46+
47+def update_dict(d, u):
48+ for k, v in u.items():
49+ if isinstance(v, collections.abc.Mapping):
50+ d[k] = update_dict(d.get(k, {}), v)
51+ else:
52+ d[k] = v
53+ return d
54+
55+
56+def update_settings(default_settings, user_settings):
57+ for k, v in user_settings.items():
58+ if k == 'use_default_settings':
59+ continue
60+ elif k == 'engines':
61+ default_engines = default_settings[k]
62+ default_engines_dict = dict((definition['name'], definition) for definition in default_engines)
63+ default_settings[k] = [update_dict(default_engines_dict[definition['name']], definition)
64+ for definition in v]
65+ else:
66+ update_dict(default_settings[k], v)
67+
68+ return default_settings
69+
70+
71+def load_settings(load_user_setttings=True):
72+ default_settings_path = get_default_settings_path()
73+ user_settings_path = get_user_settings_path()
74+ if user_settings_path is None or not load_user_setttings:
75+ # no user settings
76+ return (load_yaml(default_settings_path),
77+ 'load the default settings from {}'.format(default_settings_path))
78+
79+ # user settings
80+ user_settings = load_yaml(user_settings_path)
81+ if user_settings.get('use_default_settings'):
82+ # the user settings are merged with the default configuration
83+ default_settings = load_yaml(default_settings_path)
84+ update_settings(default_settings, user_settings)
85+ return (default_settings,
86+ 'merge the default settings ( {} ) and the user setttings ( {} )'
87+ .format(default_settings_path, user_settings_path))
88+
89+ # the user settings, fully replace the default configuration
90+ return (user_settings,
91+ 'load the user settings from {}'.format(user_settings_path))
utils/update_user_settings.pynew file+98
@@ -0,0 +1,98 @@
1+#!/usr/bin/env python
2+
3+# set path
4+from sys import path
5+from os.path import realpath, dirname, join
6+path.append(realpath(dirname(realpath(__file__)) + '/../'))
7+
8+import argparse
9+import sys
10+import string
11+import ruamel.yaml
12+import secrets
13+import collections
14+from ruamel.yaml.scalarstring import SingleQuotedScalarString, DoubleQuotedScalarString
15+from searx.settings import load_settings, check_settings_yml, get_default_settings_path
16+from searx.exceptions import SearxSettingsException
17+
18+
19+RANDOM_STRING_LETTERS = string.ascii_lowercase + string.digits + string.ascii_uppercase
20+
21+
22+def get_random_string():
23+ r = [secrets.choice(RANDOM_STRING_LETTERS) for _ in range(64)]
24+ return ''.join(r)
25+
26+
27+def main(prog_arg):
28+ yaml = ruamel.yaml.YAML()
29+ yaml.preserve_quotes = True
30+ yaml.indent(mapping=4, sequence=1, offset=2)
31+ user_settings_path = prog_args.get('user-settings-yaml')
32+
33+ try:
34+ default_settings, _ = load_settings(False)
35+ if check_settings_yml(user_settings_path):
36+ with open(user_settings_path, 'r', encoding='utf-8') as f:
37+ user_settings = yaml.load(f.read())
38+ new_user_settings = False
39+ else:
40+ user_settings = yaml.load('use_default_settings: True')
41+ new_user_settings = True
42+ except SearxSettingsException as e:
43+ sys.stderr.write(str(e))
44+ return
45+
46+ if not new_user_settings and not user_settings.get('use_default_settings'):
47+ sys.stderr.write('settings.yml already exists and use_default_settings is not True')
48+ return
49+
50+ user_settings['use_default_settings'] = True
51+ use_default_settings_comment = "settings based on " + get_default_settings_path()
52+ user_settings.yaml_add_eol_comment(use_default_settings_comment, 'use_default_settings')
53+
54+ if user_settings.get('server', {}).get('secret_key') in [None, 'ultrasecretkey']:
55+ user_settings.setdefault('server', {})['secret_key'] = DoubleQuotedScalarString(get_random_string())
56+
57+ user_engines = user_settings.get('engines')
58+ if user_engines:
59+ has_user_engines = True
60+ user_engines_dict = dict((definition['name'], definition) for definition in user_engines)
61+ else:
62+ has_user_engines = False
63+ user_engines_dict = {}
64+ user_engines = []
65+
66+ # remove old engines
67+ if prog_arg.get('add-engines') or has_user_engines:
68+ default_engines_dict = dict((definition['name'], definition) for definition in default_settings['engines'])
69+ for i, engine in enumerate(user_engines):
70+ if engine['name'] not in default_engines_dict:
71+ del user_engines[i]
72+
73+ # add new engines
74+ if prog_arg.get('add-engines'):
75+ for engine in default_settings.get('engines', {}):
76+ if engine['name'] not in user_engines_dict:
77+ user_engines.append({'name': engine['name']})
78+ user_settings['engines'] = user_engines
79+
80+ # output
81+ if prog_arg.get('dry-run'):
82+ yaml.dump(user_settings, sys.stdout)
83+ else:
84+ with open(user_settings_path, 'w', encoding='utf-8') as f:
85+ yaml.dump(user_settings, f)
86+
87+
88+def parse_args():
89+ parser = argparse.ArgumentParser(description='Update user settings.yml')
90+ parser.add_argument('--add-engines', dest='add-engines', default=False, action='store_true', help='Add new engines')
91+ parser.add_argument('--dry-run', dest='dry-run', default=False, action='store_true', help='Dry run')
92+ parser.add_argument('user-settings-yaml', type=str)
93+ return vars(parser.parse_args())
94+
95+
96+if __name__ == '__main__':
97+ prog_args = parse_args()
98+ main(prog_args)