
在 GCP 上的 bitnami moodle 的 apache 访问日志中,有多个条目(每秒添加一次!!)如下所示。这导致非常高的出口流量。我很感激任何解决此问题的指示。
52.70.240.171 - - [17/Mar/2024:17:34:25 +0530] "GET /lib/requirejs.php/1684130153/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.imageUrl(%22t/expanded%22)+%22/%22/%22 HTTP/1.1" 200 315815
52.70.240.171 - - [17/Mar/2024:17:34:26 +0530] "GET /lib/requirejs.php/1684130153/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.imageUrl(%22t/expanded%22)+%22/%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.imageUrl(%22t/expanded%22)+%22/%22/ HTTP/1.1" 200 315815
23.22.35.162 - - [17/Mar/2024:17:34:26 +0530] "GET /lib/requirejs.php/1684130153/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+h.link+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+b.params.cookiePolicyUrl+%22/%22%22+h.link+%22/%22%22+b.imageUrl(%22t/collapsed%22)+%22/%22/%22 HTTP/1.1" 200 315815
我不确定是什么原因造成的。有没有办法找出这个问题的根源?提前感谢您的帮助。
答案1
IP 的反向查找显示主机名,例如:
dig -x 52.70.240.171
-->52-70-240-171.crawl.amazonbot.amazon.
亚马逊爬行机器人行为良好,并会尊重robots.txt
如果您需要完全或部分阻止访问,请使用排除指令。
此外:Amazonbot 支持链接级rel=nofollow
指令。将这些指令包含在 HTML 中,以使 Amazonbot 能够跟踪而不是从您的网站抓取特定链接。
经常将 User-Agent 字符串记录到您的 Web 服务器日志中有助于更快地识别机器人和爬虫。