on master help sign in
[fix] googel engine - "some results are invalids: invalid content"

Fix google issues listet in the `/stats?engine=google` and message:: some results are invalids: invalid content The log is:: DEBUG searx : result: invalid content: {'url': 'https://de.wikipedia.org/wiki/Foo', 'title': 'Foo - Wikipedia', 'content': None, 'engine': 'google'} WARNING searx.engines.google : ErrorContext('searx/search/processors/abstract.py', 111, 'result_container.extend(self.engine_name, search_results)', None, 'some results are invalids: invalid content', ()) True Signed-off-by: Markus Heiser <markus.heiser@darmarit.de>

commit99128537a8b6d011f545b4c3598290c83d9f1e2f authorMarkus Heiser <markus.heiser@darmarit.de> dateTue 18 Jan 2022, 13:23 (5 years ago) parent5e45b5a
1 file changed+8−5
searx/engines/google.py+8−5
searx/engines/google.py+8−5
@@ -291,24 +291,27 @@ def response(resp):
291291 title_tag = eval_xpath_getindex(result, title_xpath, 0, default=None)
292292 if title_tag is None:
293293 # this not one of the common google results *section*
294− logger.debug('ingoring <div class="g" ../> section: missing title')
294+ logger.debug('ingoring item from the result_xpath list: missing title')
295295 continue
296296 title = extract_text(title_tag)
297297 url = eval_xpath_getindex(result, href_xpath, 0, None)
298298 if url is None:
299299 continue
300300 content = extract_text(eval_xpath_getindex(result, content_xpath, 0, default=None), allow_none=True)
301+ if content is None:
302+ logger.debug('ingoring item from the result_xpath list: missing content of title "%s"', title)
303+ continue
304+
305+ logger.debug('add link to results: %s', title)
306+
301307 results.append({
302308 'url': url,
303309 'title': title,
304310 'content': content
305311 })
312+
306313 except Exception as e: # pylint: disable=broad-except
307314 logger.error(e, exc_info=True)
308− # from lxml import etree
309− # logger.debug(etree.tostring(result, pretty_print=True))
310− # import pdb
311− # pdb.set_trace()
312315 continue
313316
314317 # parse suggestion