☰

AWS 应用程序 ELB 有时会响应 502 而不调用后端

tag-icon • tag-icon amazon-web-services amazon-elb
AWS 应用程序 ELB 有时会响应 502 而不调用后端

我们有以下设置：AWS 应用程序 ELB 具有 4 个 EC2 实例的目标组。在每个 EC2 实例上，都有一个 Apache2，它会转发到 Tomcat。

它工作得非常好。但有时，ELB 似乎拒绝与健康实例对话，而是响应 502。ELB 访问日志中的内容如下：

https 2018-03-25T09:36:13.797244Z app/XXXXXX 178.162.220.236:50254 172.31.24.81:1443 0.001 0.031 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d8d-2f54c360354a57532ccc55ef" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:14.169090Z app/XXXXXX 178.162.220.236:50270 172.31.22.59:1443 0.001 0.096 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d8e-748fc1e34bee608baa1efa81" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:18.785982Z app/XXXXXX 178.162.220.236:50330 172.31.19.151:1443 0.001 0.034 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d92-34ac7ac52b69c34fdfdbce33" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:28.457341Z app/XXXXXX 178.162.220.236:50808 172.31.28.152:1443 0.001 0.075 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d9c-66b8f0a5c6be9ff6c098dfe4" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:30.615879Z app/XXXXXX 178.162.220.236:50900 172.31.24.81:1443 0.001 0.032 0.000 204 204 314 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d9e-d1d310c5d55ad09295bb08b9" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:30.955285Z app/XXXXXX 178.162.220.236:50914 172.31.19.151:1443 0.001 0.091 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d9e-079416df92b2cd27865c8b93" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:36.843389Z app/XXXXXX 178.162.220.236:51090 172.31.28.152:1443 0.001 0.081 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da4-917b032b1fb8c1c0dc914354" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:40.432293Z app/XXXXXX 178.162.220.236:51184 172.31.24.81:1443 0.002 0.076 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da8-1ee52e1fa688cc81071fc8d1" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:41.398389Z app/XXXXXX 178.162.220.236:51208 172.31.19.151:1443 0.001 0.033 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da9-2ee889d47ead389675818ea6" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:41.775607Z app/XXXXXX 178.162.220.236:51216 172.31.28.152:1443 0.000 0.000 -1 502 - 314 293 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da9-fad6ea5fa8cc956c24d65206" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:46.954552Z app/XXXXXX 178.162.220.236:50912 172.31.22.59:1443 0.001 16.118 0.000 204 204 293 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d9e-9c0827e71b334dd908000419" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:50.750077Z app/XXXXXX 178.162.220.236:51188 172.31.22.59:1443 0.001 10.096 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76da8-f99e3281d1e0e2a73c7c4ba9" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:52.336104Z app/XXXXXX 178.162.220.236:51510 172.31.24.81:1443 0.001 0.115 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76db4-2640adae88a8639f196613e9" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:55.563397Z app/XXXXXX 178.162.220.236:51604 172.31.22.59:1443 0.000 0.104 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76db7-ecff4da4c9414383ce128eae" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:58.824749Z app/XXXXXX 178.162.220.236:51682 172.31.19.151:1443 0.001 0.034 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76dba-425a808b5efc4c3cb4cb7f53" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:37:09.004924Z app/XXXXXX 178.162.220.236:51994 172.31.28.152:1443 0.001 0.135 0.000 204 204 655 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76dc4-5d8dce64e6fcb2aa069e9018" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0
https 2018-03-25T09:36:11.146813Z app/XXXXXX 178.162.220.236:50212 172.31.24.81:1443 0.001 0.031 0.000 204 204 374 210 "POST https://XXXXXX.com:443/api/event?apikey=XXXXXX HTTP/1.1" "-" ECDHE-RSA-AES128-GCM-SHA256 TLSv1.2 arn:aws:elasticloadbalancing:eu-central-1:709725828154:targetgroup/XXXXXX "Root=1-5ab76d8b-ce30a6a810a89110465a9944" "XXXXXX.com" "arn:aws:acm:eu-central-1:709725828154:certificate/91f68487-7f5d-4ed5-80b9-4a9839ebe0f1" 0

请注意第 10 行中的 502。elb_status_code还请注意 -1 target_status_code。

apache2 日志和 tomcat 日志中都找不到该请求的踪迹。所有四个实例始终保持健康。同一 IP（本例中为 172.31.28.152）正确处理了前后请求。

发送系统在我们的控制之下。所有请求都是统一的，因此不是某些格式错误的请求未被正确处理。它们看起来都一样。

知道为什么会发生这种情况吗？我认为 AWS 不出售有故障的负载均衡器...

答案1

解决方案在评论中。ELB 的默认连接 KeepAlive 为 60 秒。Apache2 的默认连接 KeepAlive 为 5 秒。如果 5 秒结束，Apache2 将关闭其连接并重置与 ELB 的连接。但是，如果请求恰好在正确的时间到达，ELB 将接受它，决定将其转发到哪个主机，然后在那一刻，Apache 关闭连接。这将导致上述 502 错误代码。

解决方案是：当您具有级联代理/LB 时，要么调整它们的 KeepAlive 超时时间，或者 - 最好 - 甚至使它们在您到达线路的末端时变得更长一些。

我们将 ELB 超时设置为 60 秒，将 Apache2 超时设置为 120 秒。问题解决了。

相关内容