我有一个名为 的 k8s 节点edge1，它有两个 pod，一个是名为 的客户端 pod net-tool-edge1，一个是名为 的服务 pod nginx-edge1。还有一个名为nginx
由于某种原因，该节点没有 kube-proxy，而是由代理为服务生成 IPVS。
今天发现net-tool-edge1无法访问服务nginx，没有任何反应。而当我使用tcpdump抓包之后，我发现IPVS并没有按预期工作。
Podnet-work-tool的IP是10.234.67.29，Podnginx-edge1的IP是10.234.67.28，servicenginx有clusterIP 10.234.39.157。
的输出ipvsadm list：
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.234.14.175:80 rr
-> 10.234.67.28:80 Masq 1 0 0
TCP 10.234.39.157:80 rr
-> 10.234.67.28:80 Masq 1 0 0
TCP 10.234.50.96:80 rr
-> 10.22.48.15:80 Masq 1 0 0
以下是 tcpdump 出来的net-tool-edge1：
02:45:46.016748 ARP, Request who-has 10.234.67.1 tell 10.234.67.29, length 28
02:45:46.016858 ARP, Request who-has 10.234.67.1 tell 10.234.67.29, length 28
02:45:46.016862 ARP, Reply 10.234.67.1 is-at 96:b9:e1:32:f0:fa, length 28
02:45:46.016864 IP 10.234.67.29.52704 > 169.254.25.10.53: 6768+ A? nginx.fabedge-e2e-test.svc.cluster.local. (58)
02:45:46.016953 IP 10.234.67.29.52704 > 169.254.25.10.53: 7300+ AAAA? nginx.fabedge-e2e-test.svc.cluster.local. (58)
02:45:46.025844 IP 169.254.25.10.53 > 10.234.67.29.52704: 6768*- 1/0/0 A 10.234.39.157 (114)
02:45:47.023403 IP 169.254.25.10.53 > 10.234.67.29.52704: 7300*- 0/1/0 (151)
02:45:47.023958 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
02:45:47.024040 ARP, Request who-has 10.234.67.28 tell 10.234.67.1, length 28
02:45:47.024149 ARP, Request who-has 10.234.67.29 tell 10.234.67.28, length 28
02:45:47.024153 ARP, Reply 10.234.67.29 is-at f2:3e:7d:a6:f5:1d, length 28
02:45:47.024162 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3004459791, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 688253 ecr 688253,nop,wscale 7], length 0
02:45:47.024180 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
02:45:48.026571 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
02:45:48.026687 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3020124674, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 689256 ecr 689256,nop,wscale 7], length 0
02:45:48.026702 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
02:45:51.026582 ARP, Request who-has 10.234.67.29 tell 10.234.67.1, length 28
02:45:51.026595 ARP, Reply 10.234.67.29 is-at f2:3e:7d:a6:f5:1d, length 28
02:45:52.034599 ARP, Request who-has 10.234.67.28 tell 10.234.67.29, length 28
02:45:52.034668 ARP, Reply 10.234.67.28 is-at 92:01:73:4f:50:2f, length 28
我们可以看到，在 pod 中，net-tool-edge1请求的目标 IP 是 10.234.39.157，但响应数据包的源 IP 是10.234.67.28，因此 net-tool-edge1 发送了 RST 数据包。
这是edge1节点的tcpdump输出：
10:45:47.023961 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
10:45:47.023978 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
10:45:47.024063 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
10:45:47.024064 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 688253 ecr 0,nop,wscale 7], length 0
10:45:47.024160 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3004459791, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 688253 ecr 688253,nop,wscale 7], length 0
10:45:47.024161 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3004459791, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 688253 ecr 688253,nop,wscale 7], length 0
10:45:47.024185 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
10:45:47.024186 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
10:45:48.026585 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
10:45:48.026598 IP 10.234.67.29.57824 > 10.234.39.157.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
10:45:48.026636 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
10:45:48.026640 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [S], seq 1920875836, win 27200, options [mss 1360,sackOK,TS val 689256 ecr 0,nop,wscale 7], length 0
10:45:48.026684 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3020124674, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 689256 ecr 689256,nop,wscale 7], length 0
10:45:48.026686 IP 10.234.67.28.80 > 10.234.67.29.57824: Flags [S.], seq 3020124674, ack 1920875837, win 26960, options [mss 1360,sackOK,TS val 689256 ecr 689256,nop,wscale 7], length 0
10:45:48.026703 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
10:45:48.026704 IP 10.234.67.29.57824 > 10.234.67.28.80: Flags [R], seq 1920875837, win 0, length 0
这里我们可以看到，当 net-tool-edge1 向 发送请求时10.234.39.157，IPVS（或其他内核模块）将目标 IP 更改为10.234.67.28，但是当 nginx-edge1 发送响应数据包时，IPVS 并未将源 IP 更改为10.234.39.157。
我还有另一个名为edge2的节点，它具有相同的设置，但edge2上的一切都运行良好。事实上，在我重新启动edge1之前，一切都运行良好。
我用谷歌搜索了很多，但没有发现任何有用的东西。欢迎任何帮助、任何提示和任何文件，并提前致谢。
有一些 iptables 规则：
[root@edge1 ~]# iptables -S
-P INPUT ACCEPT
-P FORWARD DROP
-P OUTPUT ACCEPT
-N DOCKER
-N DOCKER-ISOLATION-STAGE-1
-N DOCKER-ISOLATION-STAGE-2
-N DOCKER-USER
-N FABEDGE-FORWARD
-A INPUT -d 169.254.25.10/32 -p udp -m udp --dport 53 -j ACCEPT
-A INPUT -d 169.254.25.10/32 -p tcp -m tcp --dport 53 -j ACCEPT
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A FORWARD -j FABEDGE-FORWARD
-A OUTPUT -s 169.254.25.10/32 -p udp -m udp --sport 53 -j ACCEPT
-A OUTPUT -s 169.254.25.10/32 -p tcp -m tcp --sport 53 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
-A FABEDGE-FORWARD -s 10.234.67.0/24 -j ACCEPT
-A FABEDGE-FORWARD -d 10.234.67.0/24 -j ACCEPT
[root@edge1 ~]# iptables -t nat -S
-P PREROUTING ACCEPT
-P INPUT ACCEPT
-P OUTPUT ACCEPT
-P POSTROUTING ACCEPT
-N DOCKER
-N FABEDGE-NAT-OUTGOING
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -j FABEDGE-NAT-OUTGOING
-A DOCKER -i docker0 -j RETURN
-A FABEDGE-NAT-OUTGOING -s 10.234.67.0/24 -m set --match-set FABEDGE-PEER-CIDR dst -j RETURN
-A FABEDGE-NAT-OUTGOING -s 10.234.67.0/24 -d 10.234.67.0/24 -j RETURN
-A FABEDGE-NAT-OUTGOING -s 10.234.67.0/24 -j MASQUERADE
[root@edge1 ~]# ipset list
Name: FABEDGE-PEER-CIDR
Type: hash:net
Revision: 6
Header: family inet hashsize 1024 maxelem 65536
Size in memory: 952
References: 1
Number of entries: 9
Members:
10.234.66.0/24
10.234.0.0/18
10.22.48.28
10.22.48.16
10.22.48.34
10.234.64.0/24
10.234.68.0/24
10.234.65.0/24
10.22.48.17
答案1
重启后，名为“net.bridge.bridge-nf-call-iptables”的参数似乎更改为 0。重置为1后，问题就解决了。
我应该提到的是，有一个名为 br-fabedge 的桥接设备，net-tool-edge1 和 nginx-edge1 的数据包将通过它。
当节点edge1做dnat时，内核会在conntrack表中保存一些信息，当响应包返回时，内核会根据conntrack表更改源地址。但这发生在第 3 层。
但是当nginx-edge1通过br-fabedge发送响应数据包时，这种情况发生在layer2上，内核不会引用connetrack表。
但是设置bridge-nf-call-iptables=1将使br-fabedge引用conntrack表并更改源地址。
我的解释可能不太准确，请指教。