我正在使用 nginx，如果我的答案正确的话，我总是会被网络爬虫找到。
我尝试配置 fail2ban，但是 fail2ban 无法检测到 IP 地址。
之所以没有被检测到，是因为它看起来像是一个合法的访问者。以下是示例日志：
116.73.68.36 - - [19/Jul/2013:23:57:47 +0800] "GET /sites/default/files/download/rhenz23/it4cai.zip HTTP/1.1" 206 14628884 "http://www.mysite.com/php/5297/computer-aided-instruction.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/kariuki/institute.zip HTTP/1.1" 206 14510149 "http://www.mysite.com/php/5040/automatic-online-examination-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 16357796 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/tovi/online_lot_reservation_system.zip HTTP/1.1" 206 14850935 "http://www.mysite.com/php/4088/online-lot-reservation.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 17217908 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:49 +0800] "GET /sites/default/files/download/sanbunna9/online_gues_house.zip HTTP/1.1" 206 17594389 "http://www.mysite.com/php/5235/online-guest-house.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:50 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 17070214 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/welmarie/online_product_reservation_system.zip HTTP/1.1" 206 15074810 "http://www.mysite.com/php/3969/online-product-reservation-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/mindgamez/system1_0.zip HTTP/1.1" 206 15232701 "http://www.mysite.com/php/4094/online-membership-and-billing-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:51 +0800] "GET /sites/default/files/download/mindgamez/system1.zip HTTP/1.1" 206 15555605 "http://www.mysite.com/php/4171/online-management-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:53 +0800] "GET /sites/default/files/download/nexus_00/ompaaps.zip HTTP/1.1" 206 16379516 "http://www.mysite.com/php/4948/online-music-publishing-and-audio-playing-system-updated.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:53 +0800] "GET /sites/default/files/download/carol_janine_crislyn/chmscnet_0.zip HTTP/1.1" 206 17671134 "http://www.mysite.com/php/4178/social-networking-site-chmscnet.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:57:59 +0800] "GET /sites/default/files/download/malyn30/socialnetworkingsite.zip HTTP/1.1" 206 16711108 "http://www.mysite.com/php/3971/sample-simple-social-networking-site.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:04 +0800] "GET /sites/default/files/download/Franziholic/franzdarylduetes.zip HTTP/1.1" 206 17718916 "http://www.mysite.com/php/5408/reyans-burger-online-ordering-system-using-php.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:07 +0800] "GET /sites/default/files/download/kariuki/institute.zip HTTP/1.1" 206 16876180 "http://www.mysite.com/php/5040/automatic-online-examination-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:08 +0800] "GET /sites/default/files/download/carol_janine_crislyn/chmscnet_0.zip HTTP/1.1" 206 16685045 "http://www.mysite.com/php/4178/social-networking-site-chmscnet.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:15 +0800] "GET /sites/default/files/download/jkev/psits_voting_system.zip HTTP/1.1" 206 17465518 "http://www.mysite.com/php/5442/drag-and-drop-voting-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
116.73.68.36 - - [19/Jul/2013:23:58:32 +0800] "GET /sites/default/files/download/may_ann/onlineschedulingsystem.zip HTTP/1.1" 206 12997278 "http://www.mysite.com/php/scheduling-system.html" "Mozilla/5.0 (Windows NT 5.1; rv:22.0) Gecko/20100101 Firefox/22.0"
顺便说一句，我使用以下命令测试了我的 fail2ban 设置，并且运行良好。
fail2ban-regex /var/log/ispconfig/httpd/mysite.com/yesterday-access.log /etc/fail2ban/filter.d/apache-badbots.conf
事实上它检测到了两个 IP 地址，日志如下：
5.9.23.42 - - [19/Jul/2013:07:52:26 +0800] "GET / HTTP/1.1" 200 11220 "-" "Mozilla/3.0 (compatible; Indy Library)"
124.122.67.67 - - [19/Jul/2013:02:43:30 +0800] "GET / HTTP/1.1" 200 59663 "-" "Mozilla/4.0 (compatible; MSIE 5.0; Windows NT; DigExt; DTS Agent"
124.122.67.67 - - [19/Jul/2013:02:43:40 +0800] "-" 400 0 "-" "-"
那么为什么 fail2ban 没有检测到它？这真的是一个机器人吗？
我也使用 cloudflare 来阻止这种机器人，但我需要手动阻止 IP 地址或将设置更改为“我受到攻击”。如果我一直在监视我的服务器，这种方法很有效。但如果我不在电脑旁怎么办？那么我该如何阻止这种爬虫呢？
请帮忙。
更新：
这是我的 jail.local 设置：
[nginx-badbots]
enabled = true
filter = apache-badbots
action = iptables-multiport[name=BadBots, port="http,https"]
logpath = /var/log/ispconfig/httpd/mysite.com/access.log
bantime = 86400 # 1 day
maxretry = 1
答案1
就像 Micheal Hampton 所说的那样，你的 jail.conf 中针对 fail2ban 的规则是什么？
也许这有帮助。：http://codelog.climens.net/2011/02/13/using-fail2ban-with-nginx-in-debian/
您还可以在 nginx 上使用 dos-deflate 代替 fail2ban 和 limit_req：http://wiki.nginx.org/HttpLimitReqModule