blog/search-indexer-engines.html · searx

git.iko.soy/searx

on gh-pages help sign in

searx/blog/search-indexer-engines.html

197 lines 12 kB history raw

2<!DOCTYPE html>

4<html>

5 <head>

6 <meta charset="utf-8" />

7 <meta name="viewport" content="width=device-width, initial-scale=1.0" />

8 <meta name="viewport" content="width=device-width, initial-scale=1">

9 <title>Query your local search engines — Searx Documentation (Searx-1.0.0.tex)</title>

10 <link rel="stylesheet" type="text/css" href="../_static/pygments.css" />

11 <link rel="stylesheet" type="text/css" href="../_static/searx.css" />

12 <link rel="stylesheet" type="text/css" href="../_static/tabs.css" />

13 <script data-url_root="../" id="documentation_options" src="../_static/documentation_options.js"></script>

14 <script src="../_static/jquery.js"></script>

15 <script src="../_static/underscore.js"></script>

16 <script src="../_static/doctools.js"></script>

17 <link rel="index" title="Index" href="../genindex.html" />

18 <link rel="search" title="Search" href="../search.html" />

19 <link rel="next" title="Running shell commands to fetch results" href="command-line-engines.html" />

20 <link rel="prev" title="Query SQL servers" href="sql-engines.html" />

21 </head><body>

22 <div class="related" role="navigation" aria-label="related navigation">

23 <h3>Navigation</h3>

24 <ul>

25 <li class="right" style="margin-right: 10px">

26 <a href="../genindex.html" title="General Index"

27 accesskey="I">index</a></li>

28 <li class="right" >

29 <a href="../py-modindex.html" title="Python Module Index"

30 >modules</a> |</li>

31 <li class="right" >

32 <a href="command-line-engines.html" title="Running shell commands to fetch results"

33 accesskey="N">next</a> |</li>

34 <li class="right" >

35 <a href="sql-engines.html" title="Query SQL servers"

36 accesskey="P">previous</a> |</li>

37 <li class="nav-item nav-item-0"><a href="../index.html">Searx Documentation (Searx-1.0.0.tex)</a> »</li>

38 <li class="nav-item nav-item-1"><a href="index.html" accesskey="U">Blog</a> »</li>

39 <li class="nav-item nav-item-this"><a href="">Query your local search engines</a></li>

40 </ul>

41 </div>

43 <div class="document">

44 <div class="documentwrapper">

45 <div class="bodywrapper">

46 <div class="body" role="main">

48 <div class="section" id="query-your-local-search-engines">

49<h1>Query your local search engines<a class="headerlink" href="#query-your-local-search-engines" title="Permalink to this headline">¶</a></h1>

50<p>From now on, searx lets you to query your locally running search engines. The following

51ones are supported now:</p>

52<ul class="simple">

53<li><p><a class="reference external" href="https://www.elastic.co/elasticsearch/">Elasticsearch</a></p></li>

54<li><p><a class="reference external" href="https://www.meilisearch.com/">Meilisearch</a></p></li>

55<li><p><a class="reference external" href="https://solr.apache.org/">Solr</a></p></li>

56</ul>

57<p>All of the engines above are added to <code class="docutils literal notranslate"><span class="pre">settings.yml</span></code> just commented out, as you have to

58<code class="docutils literal notranslate"><span class="pre">base_url</span></code> for all them.</p>

59<p>Please note that if you are not using HTTPS to access these engines, you have to enable

60HTTP requests by setting <code class="docutils literal notranslate"><span class="pre">enable_http</span></code> to <code class="docutils literal notranslate"><span class="pre">True</span></code>.</p>

61<p>Futhermore, if you do not want to expose these engines on a public instance, you can

62still add them and limit the access by setting <code class="docutils literal notranslate"><span class="pre">tokens</span></code> as described in the <a class="reference external" href="private-engines.html#private-engines">blog post about

63private engines</a>.</p>

64<div class="section" id="configuring-searx-for-search-engines">

65<h2>Configuring searx for search engines<a class="headerlink" href="#configuring-searx-for-search-engines" title="Permalink to this headline">¶</a></h2>

66<p>Each search engine is powerful, capable of full-text search.</p>

67<div class="section" id="elasticsearch">

68<h3>Elasticsearch<a class="headerlink" href="#elasticsearch" title="Permalink to this headline">¶</a></h3>

69<p>Elasticsearch supports numerous ways to query the data it is storing. At the moment

70the engine supports the most popular search methods: <code class="docutils literal notranslate"><span class="pre">match</span></code>, <code class="docutils literal notranslate"><span class="pre">simple_query_string</span></code>, <code class="docutils literal notranslate"><span class="pre">term</span></code> and <code class="docutils literal notranslate"><span class="pre">terms</span></code>.</p>

71<p>If none of the methods fit your use case, you can select <code class="docutils literal notranslate"><span class="pre">custom</span></code> query type and provide the JSON payload

72searx has to submit to Elasticsearch in <code class="docutils literal notranslate"><span class="pre">custom_query_json</span></code>.</p>

73<p>The following is an example configuration for an Elasticsearch instance with authentication

74configured to read from <code class="docutils literal notranslate"><span class="pre">my-index</span></code> index.</p>

75<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span> <span class="nt">name </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">elasticsearch</span>

76 <span class="nt">shortcut </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">es</span>

77 <span class="nt">engine </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">elasticsearch</span>

78 <span class="nt">base_url </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">http://localhost:9200</span>

79 <span class="nt">username </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">elastic</span>

80 <span class="nt">password </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">changeme</span>

81 <span class="nt">index </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">my-index</span>

82 <span class="nt">query_type </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">match</span>

83 <span class="nt">enable_http </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">True</span>

84</pre></div>

85</div>

86</div>

87<div class="section" id="meilisearch">

88<h3>Meilisearch<a class="headerlink" href="#meilisearch" title="Permalink to this headline">¶</a></h3>

89<p>This search engine is aimed at individuals and small companies. It is designed for

90small-scale (less than 10 million documents) data collections. E.g. it is great for storing

91web pages you have visited and searching in the contents later.</p>

92<p>The engine supports faceted search, so you can search in a subset of documents of the collection.

93Futhermore, you can search in Meilisearch instances that require authentication by setting <code class="docutils literal notranslate"><span class="pre">auth_token</span></code>.</p>

94<p>Here is a simple example to query a Meilisearch instance:</p>

95<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span> <span class="nt">name </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">meilisearch</span>

96 <span class="nt">engine </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">meilisearch</span>

97 <span class="nt">shortcut</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">mes</span>

98 <span class="nt">base_url </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">http://localhost:7700</span>

99 <span class="nt">index </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">my-index</span>

100 <span class="nt">enable_http</span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">True</span>

101</pre></div>

102</div>

103</div>

104<div class="section" id="solr">

105<h3>Solr<a class="headerlink" href="#solr" title="Permalink to this headline">¶</a></h3>

106<p>Solr is a popular search engine based on Lucene, just like Elasticsearch.

107But instead of searching in indices, you can search in collections.</p>

108<p>This is an example configuration for searching in the collection <code class="docutils literal notranslate"><span class="pre">my-collection</span></code> and get

109the results in ascending order.</p>

110<div class="highlight-yaml notranslate"><div class="highlight"><pre><span></span><span class="p p-Indicator">-</span> <span class="nt">name </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">solr</span>

111 <span class="nt">engine </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">solr</span>

112 <span class="nt">shortcut </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">slr</span>

113 <span class="nt">base_url </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">http://localhost:8983</span>

114 <span class="nt">collection </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">my-collection</span>

115 <span class="nt">sort </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">asc</span>

116 <span class="nt">enable_http </span><span class="p">:</span> <span class="l l-Scalar l-Scalar-Plain">True</span>

117</pre></div>

118</div>

119</div>

120</div>

121<div class="section" id="next-steps">

122<h2>Next steps<a class="headerlink" href="#next-steps" title="Permalink to this headline">¶</a></h2>

123<p>The next step is to add support for various SQL databases.</p>

124</div>

125<div class="section" id="acknowledgement">

126<h2>Acknowledgement<a class="headerlink" href="#acknowledgement" title="Permalink to this headline">¶</a></h2>

127<p>This development was sponsored by <a class="reference external" href="https://nlnet.nl/discovery">Search and Discovery Fund</a> of <a class="reference external" href="https://nlnet.nl/">NLnet Foundation</a> .</p>

128<div class="line-block">

129<div class="line">Happy hacking.</div>

130<div class="line">kvch // 2021.04.07 23:16</div>

131</div>

132</div>

133</div>

134

135

136 <div class="clearer"></div>

137 </div>

138 </div>

139 </div>