Nginx 和 php-fpm 因高负载而失败:“端口 80 上可能发生 SYN 泛滥”和“104:对等方重置连接”

Nginx 和 php-fpm 因高负载而失败:“端口 80 上可能发生 SYN 泛滥”和“104:对等方重置连接”

我有一个负载相当重的 Web 服务器,使用:

Ubuntu server
nginx
php-fpm + apc

昨天我的服务器发生了一些奇怪的事情。它崩溃了并且停止了响应,在我重新启动它之后,网页开始加载得非常非常慢,在大多数情况下都显示“请求超时”。

我检查了一下/var/log/syslog,发现有很多类似的消息:TCP: Possible SYN flooding on port 80. Sending cookies.

页面本地加载大约需要 2 分钟:

time wget -O /dev/null mysite.net
--2012-12-21 13:17:15--  http://mysite.net/
Resolving ficbook.net... 85.254.49.180
Connecting to mysite.net|85.254.49.180|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 1311 (1.3K) [text/html]
Saving to: `/dev/null'

100%[========================================================================================================>] 1,311       --.-K/s   in 0s      

2012-12-21 13:19:18 (181 MB/s) - `/dev/null' saved [1311/1311]


real    2m2.438s
user    0m0.000s
sys 0m0.000s

我不确定这是否真的是 SYN Flood 攻击。如果是,为什么 cookies 没有帮助?以下是来自 netstat 的信息:

netstat -tuna | grep :80 | grep SYN_RECV
tcp        0      0 85.254.49.180:80        92.37.173.66:3214       SYN_RECV   
tcp        0      0 85.254.49.180:80        81.26.91.4:49471        SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4940      SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.140.242:23259    SYN_RECV   
tcp        0      0 85.254.49.180:80        94.139.229.219:49827    SYN_RECV   
tcp        0      0 85.254.49.180:80        95.67.233.125:51267     SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.2.69:7051        SYN_RECV   
tcp        0      0 85.254.49.180:80        95.67.239.40:54497      SYN_RECV   
tcp        0      0 85.254.49.180:80        195.91.229.193:58981    SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4925      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:59086      SYN_RECV   
tcp        0      0 85.254.49.180:80        92.113.26.124:3887      SYN_RECV   
tcp        0      0 85.254.49.180:80        77.34.83.254:26963      SYN_RECV   
tcp        0      0 85.254.49.180:80        195.208.64.130:3542     SYN_RECV   
tcp        0      0 85.254.49.180:80        81.26.91.4:49480        SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53130     SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4939      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:41696      SYN_RECV   
tcp        0      0 85.254.49.180:80        178.45.39.169:41758     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.66.37:51534     SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.9.197:8249       SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:3531       SYN_RECV   
tcp        0      0 85.254.49.180:80        178.34.206.52:3409      SYN_RECV   
tcp        0      0 85.254.49.180:80        193.188.254.93:50317    SYN_RECV   
tcp        0      0 85.254.49.180:80        217.66.152.162:8883     SYN_RECV   
tcp        0      0 85.254.49.180:80        109.198.235.10:56382    SYN_RECV   
tcp        0      0 85.254.49.180:80        95.53.159.39:2256       SYN_RECV   
tcp        0      0 85.254.49.180:80        188.232.13.175:49819    SYN_RECV   
tcp        0      0 85.254.49.180:80        88.203.2.27:64080       SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.64.52:12382     SYN_RECV   
tcp        0      0 85.254.49.180:80        92.124.76.189:3416      SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:30532      SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53131     SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.123.1:44943      SYN_RECV   
tcp        0      0 85.254.49.180:80        176.51.255.3:1642       SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.165.112:56906     SYN_RECV   
tcp        0      0 85.254.49.180:80        88.203.2.27:64081       SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.66.37:51533     SYN_RECV   
tcp        0      0 85.254.49.180:80        176.51.211.131:1699     SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:22233      SYN_RECV   
tcp        0      0 85.254.49.180:80        211.167.112.18:58353    SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.66.32:38640     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.144.185.150:64421   SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4928      SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4927      SYN_RECV   
tcp        0      0 85.254.49.180:80        94.153.254.218:1084     SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:30384      SYN_RECV   
tcp        0      0 85.254.49.180:80        46.201.3.189:51032      SYN_RECV   
tcp        0      0 85.254.49.180:80        109.187.107.41:50565    SYN_RECV   
tcp        0      0 85.254.49.180:80        91.146.60.86:49266      SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53134     SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.238.25:2515       SYN_RECV   
tcp        0      0 85.254.49.180:80        176.102.16.8:54291      SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4918      SYN_RECV   
tcp        0      0 85.254.49.180:80        95.153.164.165:26752    SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.239.76:46519      SYN_RECV   
tcp        0      0 85.254.49.180:80        94.139.229.219:49826    SYN_RECV   
tcp        0      0 85.254.49.180:80        188.239.193.48:49418    SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4919      SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.66.32:38639     SYN_RECV   
tcp        0      0 85.254.49.180:80        95.67.233.125:51266     SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.235.172:59092     SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.136.21:44804     SYN_RECV   
tcp        0      0 85.254.49.180:80        95.109.193.247:1206     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.112.11.130:2714     SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4941      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:52640      SYN_RECV   
tcp        0      0 85.254.49.180:80        37.79.93.27:64801       SYN_RECV   
tcp        0      0 85.254.49.180:80        91.203.96.76:45132      SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.238.25:2513       SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.235.172:60092     SYN_RECV   
tcp        0      0 85.254.49.180:80        188.239.193.48:49416    SYN_RECV   
tcp        0      0 85.254.49.180:80        178.130.42.68:60373     SYN_RECV   
tcp        0      0 85.254.49.180:80        80.239.243.181:58110    SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53128     SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.9.197:18870      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:53380      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.135.63.40:58845      SYN_RECV   
tcp        0      0 85.254.49.180:80        80.239.243.110:52234    SYN_RECV   
tcp        0      0 85.254.49.180:80        46.201.3.189:51028      SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:53457      SYN_RECV   
tcp        0      0 85.254.49.180:80        85.235.176.138:12101    SYN_RECV   
tcp        0      0 85.254.49.180:80        109.187.107.41:50567    SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.48.29:4172       SYN_RECV   
tcp        0      0 85.254.49.180:80        188.232.13.175:49820    SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:6651       SYN_RECV   
tcp        0      0 85.254.49.180:80        91.198.143.6:45591      SYN_RECV   
tcp        0      0 85.254.49.180:80        85.235.176.138:50667    SYN_RECV   
tcp        0      0 85.254.49.180:80        176.209.98.72:53653     SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.239.71:49701      SYN_RECV   
tcp        0      0 85.254.49.180:80        188.232.13.175:49817    SYN_RECV   
tcp        0      0 85.254.49.180:80        188.239.193.48:49417    SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:54175      SYN_RECV   
tcp        0      0 85.254.49.180:80        61.147.79.111:51039     SYN_RECV   
tcp        0      0 85.254.49.180:80        88.154.3.228:58854      SYN_RECV   
tcp        0      0 85.254.49.180:80        87.253.29.234:53135     SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4938      SYN_RECV   
tcp        0      0 85.254.49.180:80        62.122.51.139:4942      SYN_RECV   
tcp        0      0 85.254.49.180:80        176.209.98.72:53662     SYN_RECV   
tcp        0      0 85.254.49.180:80        2.74.51.158:1092        SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.140.242:48178    SYN_RECV   
tcp        0      0 85.254.49.180:80        213.87.129.42:29549     SYN_RECV   
tcp        0      0 85.254.49.180:80        37.29.88.202:28428      SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.235.172:50983     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.118.64.52:12381     SYN_RECV   
tcp        0      0 85.254.49.180:80        85.26.235.172:55459     SYN_RECV   
tcp        0      0 85.254.49.180:80        84.244.12.209:64975     SYN_RECV   
tcp        0      0 85.254.49.180:80        83.149.2.121:10768      SYN_RECV   
tcp        0      0 85.254.49.180:80        84.240.248.206:3494     SYN_RECV   
tcp        0      0 85.254.49.180:80        195.91.229.193:52428    SYN_RECV   
tcp        0      0 85.254.49.180:80        95.109.193.247:1202     SYN_RECV   
tcp        0      0 85.254.49.180:80        79.105.204.56:56822     SYN_RECV   
tcp        0      0 85.254.49.180:80        85.15.184.141:56335     SYN_RECV   
tcp        0      0 85.254.49.180:80        164.177.225.31:50584    SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.238.25:2511       SYN_RECV   
tcp        0      0 85.254.49.180:80        84.240.248.206:3493     SYN_RECV   
tcp        0      0 85.254.49.180:80        80.83.239.76:26950      SYN_RECV   
tcp        0      0 85.254.49.180:80        84.240.248.206:3495     SYN_RECV   
tcp        0      0 85.254.49.180:80        217.144.185.150:58141   SYN_RECV   
tcp        0      0 85.254.49.180:80        178.215.97.15:13346     SYN_RECV 

我尝试禁用 syn cookies,但没有效果。似乎该服务器正在限制连接数,如果您查看“每分钟点击数”指标,它看起来如下:

在此处输入图片描述

昨天,在崩溃之前,一切都运行良好。我会感谢任何有关问题可能是什么或如何诊断的信息或建议。

更新

我非常确定这不是一次攻击。当我重新启动 nginx 时,一切都正常工作了几个小时,然后,系统日志再次充满了:

Possible SYN flooding on port 80
Possible SYN flooding on port 9000

然后nginx的错误日志首先得到很多104错误:

2013/01/08 20:28:24 [error] 959#0: *2387458 recv() failed (104: Connection reset by peer) while reading response header from upstream

然后是 110:

2013/01/08 21:27:19 [error] 30349#0: *760749 upstream timed out (110: Connection timed out) while connecting to upstream

这种情况发生在晚上,当负载达到一定量(每秒约 800 次)时就会出现问题。

关闭 syn cookies 和调整积压没有任何效果。

互联网上有很多类似的说法,但找不到真正的答案。请帮忙!

答案1

听起来你的上游服务器出了问题,导致 nginx 显得非常慢。

当 nginx 速度缓慢时,请求是否仅当代理到您的 php-fpm + apc 时才会花费很长时间?您是否尝试过定义非代理location,并查看是否会出现任何问题?

您的 php-fpm + apc 是否内存不足、连接/文件描述符不足或工作线程/进程不足?您是否在使用 OpenVZ?或者任何其他可能因设计不当而导致的内核级虚拟化?如果没有,您是否可能遇到任何其他进程或内存限制?您可以通过进入su运行 php-fpm / apc 的用户并运行limit来检查限制tcsh

您可能应该发布 nginx 和 php-fpm + apc 的整个配置,否则,这将是一场大猜谜游戏。我不是 php 专家,但我有根据的猜测是,您的 php-fpm+apc 端存在某种连接或工作线程限制,而您的 nginx 代理超出了该限制。

另外,我看到你有一张很好的图表,显示了某天情况突然变糟的情况;在这件事发生之前的几天内,你是否做过任何更改或升级?

答案2

是否可以使用以下方式限制传入连接的速率IP表或者nginx

答案3

我强烈推荐让你设置一些默认的 Iptables 链。尝试丢弃 SYN 泛洪并记录所有丢弃的数据包。你甚至不知道你的网络层

  • 阻止洪水攻击,丢弃包裹
  • 记录所有丢弃的数据包
  • 请告诉我们此事的最新进展

最后,对于具有网络分析经验的人来说,可以轻松检测到服务器崩溃并立即阻止攻击。

相关内容