我们有以下设置:AWS 应用程序 ELB 具有 4 个 EC2 实例的目标组。在每个 EC2 实例上,都有一个 Apache2,它会转发到 Tomcat。
它工作得非常好。但有时,ELB 似乎拒绝与健康实例对话,而是响应 502。ELB 访问日志中的内容如下:
https 2018-03-25T09:36:13.797244Z app/XXXXXX 178.162.220.236:50254 172.31.24.81:1443 0.001 0.031 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d8d-2f54c360354a57532ccc55ef" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:14.169090Z app/XXXXXX 178.162.220.236:50270 172.31.22.59:1443 0.001 0.096 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d8e-748fc1e34bee608baa1efa81" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:18.785982Z app/XXXXXX 178.162.220.236:50330 172.31.19.151:1443 0.001 0.034 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d92-34ac7ac52b69c34fdfdbce33" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:28.457341Z app/XXXXXX 178.162.220.236:50808 172.31.28.152:1443 0.001 0.075 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d9c-66b8f0a5c6be9ff6c098dfe4" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:30.615879Z app/XXXXXX 178.162.220.236:50900 172.31.24.81:1443 0.001 0.032 0.000 204 204 314 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d9e-d1d310c5d55ad09295bb08b9" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:30.955285Z app/XXXXXX 178.162.220.236:50914 172.31.19.151:1443 0.001 0.091 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d9e-079416df92b2cd27865c8b93" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:36.843389Z app/XXXXXX 178.162.220.236:51090 172.31.28.152:1443 0.001 0.081 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da4-917b032b1fb8c1c0dc914354" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:40.432293Z app/XXXXXX 178.162.220.236:51184 172.31.24.81:1443 0.002 0.076 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da8-1ee52e1fa688cc81071fc8d1" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:41.398389Z app/XXXXXX 178.162.220.236:51208 172.31.19.151:1443 0.001 0.033 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da9-2ee889d47ead389675818ea6" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:41.775607Z app/XXXXXX 178.162.220.236:51216 172.31.28.152:1443 0.000 0.000 -1 502 - 314 293 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da9-fad6ea5fa8cc956c24d65206" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:46.954552Z app/XXXXXX 178.162.220.236:50912 172.31.22.59:1443 0.001 16.118 0.000 204 204 293 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d9e-9c0827e71b334dd908000419" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:50.750077Z app/XXXXXX 178.162.220.236:51188 172.31.22.59:1443 0.001 10.096 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da8-f99e3281d1e0e2a73c7c4ba9" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:52.336104Z app/XXXXXX 178.162.220.236:51510 172.31.24.81:1443 0.001 0.115 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76db4-2640adae88a8639f196613e9" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:55.563397Z app/XXXXXX 178.162.220.236:51604 172.31.22.59:1443 0.000 0.104 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76db7-ecff4da4c9414383ce128eae" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:58.824749Z app/XXXXXX 178.162.220.236:51682 172.31.19.151:1443 0.001 0.034 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76dba-425a808b5efc4c3cb4cb7f53" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:37:09.004924Z app/XXXXXX 178.162.220.236:51994 172.31.28.152:1443 0.001 0.135 0.000 204 204 655 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76dc4-5d8dce64e6fcb2aa069e9018" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:11.146813Z app/XXXXXX 178.162.220.236:50212 172.31.24.81:1443 0.001 0.031 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d8b-ce30a6a810a89110465a9944" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
请注意第 10 行中的 502。elb_status_code
还请注意 -1 target_status_code
。
apache2 日志和 tomcat 日志中都找不到该请求的踪迹。所有四个实例始终保持健康。同一 IP(本例中为 172.31.28.152)正确处理了前后请求。
发送系统在我们的控制之下。所有请求都是统一的,因此不是某些格式错误的请求未被正确处理。它们看起来都一样。
知道为什么会发生这种情况吗?我认为 AWS 不出售有故障的负载均衡器...
答案1
解决方案在评论中。ELB 的默认连接 KeepAlive 为 60 秒。Apache2 的默认连接 KeepAlive 为 5 秒。如果 5 秒结束,Apache2 将关闭其连接并重置与 ELB 的连接。但是,如果请求恰好在正确的时间到达,ELB 将接受它,决定将其转发到哪个主机,然后在那一刻,Apache 关闭连接。这将导致上述 502 错误代码。
解决方案是:当您具有级联代理/LB 时,要么调整它们的 KeepAlive 超时时间,或者 - 最好 - 甚至使它们在您到达线路的末端时变得更长一些。
我们将 ELB 超时设置为 60 秒,将 Apache2 超时设置为 120 秒。问题解决了。