@@ -0,0 +1,114 @@
1+How to protect an instance
2+==========================
3+
4+Searx depens on external search services. To avoid the abuse of these services it is advised to limit the number of requests processed by searx.
5+
6+An application firewall, ``filtron`` solves exactly this problem. Information on how to install it can be found at the `project page of filtron <https://github.com/asciimoo/filtron>`__.
7+
8+Sample configuration of filtron
9+-------------------------------
10+
11+An example configuration can be find below. This configuration limits the access of
12+
13+ * scripts or applications (roboagent limit)
14+
15+ * webcrawlers (botlimit)
16+
17+ * IPs which send too many requests (IP limit)
18+
19+ * too many json, csv, etc. requests (rss/json limit)
20+
21+ * the same UserAgent of if too many requests (useragent limit)
22+
23+
24+.. code:: json
25+
26+ [
27+ {
28+ "name": "search request",
29+ "filters": ["Param:q", "Path=^(/|/search)$"],
30+ "interval": <time-interval-in-sec>,
31+ "limit": <max-request-number-in-interval>,
32+ "subrules": [
33+ {
34+ "name": "roboagent limit",
35+ "interval": <time-interval-in-sec>,
36+ "limit": <max-request-number-in-interval>,
37+ "filters": ["Header:User-Agent=(curl|cURL|Wget|python-requests|Scrapy|FeedFetcher|Go-http-client)"],
38+ "actions": [
39+ {"name": "block",
40+ "params": {"message": "Rate limit exceeded"}}
41+ ]
42+ },
43+ {
44+ "name": "botlimit",
45+ "limit": 0,
46+ "stop": true,
47+ "filters": ["Header:User-Agent=(Googlebot|bingbot|Baiduspider|yacybot|YandexMobileBot|YandexBot|Yahoo! Slurp|MJ12bot|AhrefsBot|archive.org_bot|msnbot|MJ12bot|SeznamBot|linkdexbot|Netvibes|SMTBot|zgrab|James BOT)"],
48+ "actions": [
49+ {"name": "block",
50+ "params": {"message": "Rate limit exceeded"}}
51+ ]
52+ },
53+ {
54+ "name": "IP limit",
55+ "interval": <time-interval-in-sec>,
56+ "limit": <max-request-number-in-interval>,
57+ "stop": true,
58+ "aggregations": ["Header:X-Forwarded-For"],
59+ "actions": [
60+ {"name": "block",
61+ "params": {"message": "Rate limit exceeded"}}
62+ ]
63+ },
64+ {
65+ "name": "rss/json limit",
66+ "interval": <time-interval-in-sec>,
67+ "limit": <max-request-number-in-interval>,
68+ "stop": true,
69+ "filters": ["Param:format=(csv|json|rss)"],
70+ "actions": [
71+ {"name": "block",
72+ "params": {"message": "Rate limit exceeded"}}
73+ ]
74+ },
75+ {
76+ "name": "useragent limit",
77+ "interval": <time-interval-in-sec>,
78+ "limit": <max-request-number-in-interval>,
79+ "aggregations": ["Header:User-Agent"],
80+ "actions": [
81+ {"name": "block",
82+ "params": {"message": "Rate limit exceeded"}}
83+ ]
84+ }
85+ ]
86+ }
87+ ]
88+
89+
90+
91+Route request through filtron
92+-----------------------------
93+
94+Filtron can be started using the following command:
95+
96+.. code:: bash
97+
98+ $ filtron -rules rules.json
99+
100+It listens on 127.0.0.1:4004 and forwards filtered requests to 127.0.0.1:8888 by default.
101+
102+Use it along with ``nginx`` with the following example configuration.
103+
104+.. code:: bash
105+
106+ location / {
107+ proxy_set_header Host $http_host;
108+ proxy_set_header X-Real-IP $remote_addr;
109+ proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
110+ proxy_set_header X-Scheme $scheme;
111+ proxy_pass http://127.0.0.1:4004/;
112+ }
113+
114+Requests are coming from port 4004 going through filtron and then forwarded to port 8888 where a searx is being run.