我们确实遇到了一个奇怪的生产超时问题nginx
。超时说明如下:
"upstream timed out(110: Connection timed out)"
这里,上游服务器jetty
与 nginx 运行在同一台主机上。Jetty 运行在8080
nginx 运行的端口上443
。
检查了上述错误后,我验证了 jetty 日志和 nginx 日志。尽管 jetty 在不到一秒的时间内返回响应,但发送给 nginx 的响应delayed
接近60 secs
。并且 nginx 的计时是在 60 秒后触发超时警报,http 响应代码为504
。这是正在发生的事情intermittently
。
以下是 jetty 和 nginx 的超时请求日志。
码头日志:
{"evt":1494519426927,"intelId":"50","intelSeq":112506,"intelVer":"1","time":"2017-05-11T16:17:06.927Z","uiCorrelationIdV1":"SUI-1494519425839-42047","threadName":"qtp754853679-357","wResource":"http://m.xxx.com/search/facet/women/womens-handbags-9780510203/shoulder//N-53f3Z7hk9Z1z0roil","wMethod":"GET","wStatus":200,"wDurationMicros":1087801,"wJlpLocation":"","wFwdFor":"NADA","wHostHdr":"m.xxxx.com","wReferer":"https://m.xxxx.com/search/facet/women/womens-handbags-9780510203//N-53f3Z7hk9?search-term=包袋&sortBy=priceLow&facet=手提包%20Style","wHttpVer":"HTTP/1.0","wWsgClientIp":"80.192.191.2","wSrcIP":"127.0.0.1","wUserAgent":"Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 类似 Mac OS X) AppleWebKit/603.1.30 (KHTML, 类似 Gecko) Version/10.0 Mobile/14E304 Safari/602.1","intelCropped":false,"intelLength":834}
Nginx 日志:
2017-05-11T17:18:09+01:00 intelId="56" intelVer="2" wMethod="GET" wResource="/search/facet/women/womens-handbags-9780510203/across-body/shoulder//N-53f3Z7hk9Z1z0roq4Z1z0roil?search-term=Bags&sortBy=priceLow&facet=Handbag%20Style" wStatus="504" wCacheStatus="MISS" wSrcIP="172.17.233.135" wSize="176" wDurationSeconds="60.001" wHostHdr="m.xxx.com" wReferer="https://m.xxxx.com/search/facet/women/womens-handbags-9780510203/shoulder//N-53f3Z7hk9Z1z0roil?search-term=Bags&sortBy=priceLow&facet=Handbag%20Style" wSSL="on" wSSLver="771" wSSLciph="TLS1.2-ECDHE-RSA-AES256-GCM-SHA384" wWsgClientIp="80.192.191.2" wFwdFor="-" wJlpLocation="-" wProtocol="HTTP/1.1" wUpstreamAddr="127.0.0.1:8080" wPort=443 s_vi="[CS]v1|2B4D4CA80501261B-600001064000B11D[CE]" s_ppv="jl%253Asearch%2C14%2C100%2C5789%2C414%2C628%2C414%2C736%2C3%2CP" perceivedUser="true" wUiCorrelationIdV1="-" wUserAgent="Mozilla/5.0 (iPhone; CPU iPhone OS 10_3_1 类似 Mac OS X) AppleWebKit/603.1.30 (KHTML, 类似 Gecko) Version/10.0 Mobile/14E304 Safari/602.1" deviceType="mobile"
从以上两个日志中,我们可以推断 jetty 在 时返回了 200 响应2017-05-11T16:17:06.927Z
,但 nginx 在 时收到了该2017-05-11T17:18:09+01:00
响应。这 60 多秒导致了超时。这有点奇怪,因为 nginx 和 jetty 都托管在同一台主机上。
如果有人可以帮助我们调试问题或提供建议,那就太好了。
提前谢谢了。