k8s环境(4节点,rke 1.21.5)
我们注意到,不同的 k8s pod 之间的套接字数据传输存在随机显著的延迟。在某些情况下,延迟可能长达 15 秒。
通过分析tcpdump,我们发现在某些情况下,服务器端需要很长时间才能向客户端回复ACK。
以下是一个套接字的服务器端 TCP 转储:10.42.40.2:51702(客户端)<-> 10.42.18.64:9099(服务器端)
13:24:05.485173 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1353130:1361578, ack 615, win 3068, options [nop,nop,TS val 497835713 ecr 1074832978], length 8448
**13:24:05.489375 IP 10.42.18.64.9099 > 10.42.40.2.51702: Flags [.], ack 1330602, win 1285, options [nop,nop,TS val 1074832982 ecr 497835641], length 0**
13:24:05.489420 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1361578:1362986, ack 615, win 3068, options [nop,nop,TS val 497835717 ecr 1074832982], length 1408
13:24:05.489424 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1362986:1367210, ack 615, win 3068, options [nop,nop,TS val 497835717 ecr 1074832982], length 4224
13:24:05.489482 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1367210:1412266, ack 615, win 3068, options [nop,nop,TS val 497835717 ecr 1074832982], length 45056
13:24:05.489532 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [P.], seq 1412266:1420714, ack 615, win 3068, options [nop,nop,TS val 497835717 ecr 1074832982], length 8448
13:24:05.489534 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1420714:1457322, ack 615, win 3068, options [nop,nop,TS val 497835717 ecr 1074832982], length 36608
13:24:05.489573 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1457322:1472810, ack 615, win 3068, options [nop,nop,TS val 497835717 ecr 1074832982], length 15488
13:24:05.489688 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1472810:1475626, ack 615, win 3068, options [nop,nop,TS val 497835717 ecr 1074832982], length 2816
13:24:05.489741 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1475626:1485482, ack 615, win 3068, options [nop,nop,TS val 497835717 ecr 1074832982], length 9856
13:24:05.671207 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1485482:1486890, ack 615, win 3068, options [nop,nop,TS val 497835899 ecr 1074832982], length 1408
13:24:06.155201 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1330602:1332010, ack 615, win 3068, options [nop,nop,TS val 497836383 ecr 1074832982], length 1408
13:24:07.115190 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1330602:1332010, ack 615, win 3068, options [nop,nop,TS val 497837343 ecr 1074832982], length 1408
13:24:09.003161 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1330602:1332010, ack 615, win 3068, options [nop,nop,TS val 497839231 ecr 1074832982], length 1408
13:24:12.843114 IP 10.42.40.2.51702 > 10.42.18.64.9099: Flags [.], seq 1330602:1332010, ack 615, win 3068, options [nop,nop,TS val 497843071 ecr 1074832982], length 1408
**13:24:17.658342 IP 10.42.18.64.9099 > 10.42.40.2.51702: Flags [.], ack 1361578, win 1255, options [nop,nop,TS val 1074845151 ecr 497835713], length 0**
13:24:17.658357 IP 10.42.18.64.9099 > 10.42.40.2.51702: Flags [.], ack 1486890, win 1133, options [nop,nop,TS val 1074845151 ecr 497835717,nop,nop,sack 1 {1330602:1332010}], length 0
13:24:17.658360 IP 10.42.18.64.9099 > 10.42.40.2.51702: Flags [.], ack 1486890, win 1133, options [nop,nop,TS val 1074845151 ecr 497835717,nop,nop,sack 1 {1330602:1332010}], length 0
13:24:17.658365 IP 10.42.18.64.9099 > 10.42.40.2.51702: Flags [.], ack 1486890, win 1133, options [nop,nop,TS val 1074845151 ecr 497835717,nop,nop,sack 1 {1330602:1332010}], length 0
根据 tcpdump,服务器 (9099) 在 13:24:05.489375 回复了 ack 1330602。然后,在接下来的 12 秒内,它直到 13:24:17.658342 才回复任何 ack。我认为这阻塞了套接字并阻止客户端传输更多字节。在 13:24:17 之后,套接字恢复正常,字节继续刷新到其中。
我们尝试了 TCP_NODELAY 和 TCP_QUICKACK,但都不能解决问题。(我不这么认为)您能解释一下为什么这里回复 TCP ACK 需要这么长时间吗?