on master help sign in
fix fetch_languages for bing

Bing has a list of regions that it supports and some of these regions may have more than one possible language. In some cases, like Switzerland, these languages are always shown as options, so there is no issue. But in other cases, like Andorra, Bing will only show one language at the time, either the region's default or the request's language if the latter is supported by that region. For example, if the HTTP request is in French, Andorra will appear as fr-AD but if the same page is requested in any other language Andorra will appear as ca-AD. This is specially a problem when Bing assumes that the request is in English because it overrides enough language codes to make several major languages like Arabic dissappear from the languages.py file. To avoid that issue, I set the Accept-Language header to a language that's only supported in one region to hopefully avoid these overrides.

commit9b6ffed061570e2540e219d66e5100a1572c07c8 authorMarc Abonce Seguin <marc-abonce@mailbox.org> dateThu 25 Feb 2021, 23:20 (6 years ago) parent728e096
1 file changed+6−2
searx/engines/__init__.py+6−2
searx/engines/__init__.py+6−2
@@ -27,7 +27,7 @@ from searx import settings
2727from searx import logger
2828from searx.data import ENGINES_LANGUAGES
2929from searx.poolrequests import get, get_proxy_cycles
30−from searx.utils import load_module, match_language, get_engine_from_settings
30+from searx.utils import load_module, match_language, get_engine_from_settings, gen_useragent
3131
3232
3333logger = logger.getChild('engines')
@@ -131,8 +131,12 @@ def load_engine(engine_data):
131131
132132 # assign language fetching method if auxiliary method exists
133133 if hasattr(engine, '_fetch_supported_languages'):
134+ headers = {
135+ 'User-Agent': gen_useragent(),
136+ 'Accept-Language': 'ja-JP,ja;q=0.8,en-US;q=0.5,en;q=0.3', # bing needs a non-English language
137+ }
134138 setattr(engine, 'fetch_supported_languages',
135− lambda: engine._fetch_supported_languages(get(engine.supported_languages_url)))
139+ lambda: engine._fetch_supported_languages(get(engine.supported_languages_url, headers=headers)))
136140
137141 engine.stats = {
138142 'sent_search_count': 0, # sent search