过去几天我遇到了一个奇怪的问题。我们有 3 个 Kubernetes 实例设置，所有 3 个实例都使用 Ubuntu 20.04 LTS。
ping 一直运行良好，突然停止工作，节点本身没有任何更新/安装。我想知道网络网关端是否有任何变化，但网关端也没有变化。
- ufw 已禁用
| 节点 | Ping 状态 |
|---|---|
| 节点 51（主） | 对节点 52 进行 ping 操作，无法 Ping 至节点 53 |
| 节点 52（工作者） | 可以 Ping 到节点 51，也可以 Ping 到节点 53 |
| 节点 53（工人） | 向节点 51 执行 ping 操作，Ping 至节点 52 |
节点 1 网络和路线详细信息
default via 172.16.25.1 dev eno1 proto static
10.244.0.0/24 dev cni0 proto kernel scope link src 10.244.0.1
10.244.2.0/24 via 10.244.2.0 dev flannel.1 onlink
10.244.5.0/24 via 10.244.5.0 dev flannel.1 onlink
172.16.25.0/24 dev eno1 proto kernel scope link src 172.16.25.51
172.16.25.0/24 via 172.16.25.1 dev eno1 proto static
eno1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.16.25.51 netmask 255.255.255.0 broadcast 172.16.25.255
inet6 fe80::b67a:f1ff:fee2:9334 prefixlen 64 scopeid 0x20<link>
ether b4:7a:f1:e2:93:34 txqueuelen 1000 (Ethernet)
RX packets 4175094283 bytes 3583980311063 (3.5 TB)
RX errors 0 dropped 154659 overruns 0 frame 0
TX packets 4347655900 bytes 4081889938622 (4.0 TB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eno1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether b4:7a:f1:e2:93:34 brd ff:ff:ff:ff:ff:ff
inet 172.16.25.51/24 brd 172.16.25.255 scope global eno1
valid_lft forever preferred_lft forever
inet6 fe80::b67a:f1ff:fee2:9334/64 scope link
valid_lft forever preferred_lft forever
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.25.1 0.0.0.0 UG 0 0 0 eno1
10.244.0.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
10.244.2.0 10.244.2.0 255.255.255.0 UG 0 0 0 flannel.1
10.244.5.0 10.244.5.0 255.255.255.0 UG 0 0 0 flannel.1
172.16.25.0 0.0.0.0 255.255.255.0 U 0 0 0 eno1
172.16.25.0 172.16.25.1 255.255.255.0 UG 0 0 0 eno1
节点 1 iptable 条目
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N KUBE-EXTERNAL-SERVICES
-N KUBE-FIREWALL
-N KUBE-FORWARD
-N KUBE-KUBELET-CANARY
-N KUBE-NODEPORTS
-N KUBE-PROXY-CANARY
-N KUBE-SERVICES
-A INPUT -m comment --comment "kubernetes health check service ports" -j KUBE-NODEPORTS
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
节点 3 网络和路线详细信息
default via 172.16.25.1 dev eno2 proto static
10.244.0.0/24 via 10.244.0.0 dev flannel.1 onlink
10.244.2.0/24 dev cni0 proto kernel scope link src 10.244.2.1
10.244.5.0/24 via 10.244.5.0 dev flannel.1 onlink
172.16.25.0/24 dev eno2 proto kernel scope link src 172.16.25.53
172.16.25.0/24 via 172.16.25.1 dev eno2 proto static
eno2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST> mtu 1500
inet 172.16.25.53 netmask 255.255.255.0 broadcast 172.16.25.255
inet6 fe80::b67a:f1ff:fee2:94f1 prefixlen 64 scopeid 0x20<link>
ether b4:7a:f1:e2:94:f1 txqueuelen 1000 (Ethernet)
RX packets 3211362535 bytes 3081309266071 (3.0 TB)
RX errors 0 dropped 121709 overruns 0 frame 0
TX packets 2936433937 bytes 2631825614517 (2.6 TB)
TX errors 0 dropped 0 overruns 0 carrier 0 collisions 0
eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq state UP group default qlen 1000
link/ether b4:7a:f1:e2:94:f1 brd ff:ff:ff:ff:ff:ff
inet 172.16.25.53/24 brd 172.16.25.255 scope global eno2
valid_lft forever preferred_lft forever
inet6 fe80::b67a:f1ff:fee2:94f1/64 scope link
valid_lft forever preferred_lft forever
Kernel IP routing table
Destination Gateway Genmask Flags Metric Ref Use Iface
0.0.0.0 172.16.25.1 0.0.0.0 UG 0 0 0 eno2
10.244.0.0 10.244.0.0 255.255.255.0 UG 0 0 0 flannel.1
10.244.2.0 0.0.0.0 255.255.255.0 U 0 0 0 cni0
10.244.5.0 10.244.5.0 255.255.255.0 UG 0 0 0 flannel.1
172.16.25.0 0.0.0.0 255.255.255.0 U 0 0 0 eno2
172.16.25.0 172.16.25.1 255.255.255.0 UG 0 0 0 eno2
节点 3 iptables 条目
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-N KUBE-EXTERNAL-SERVICES
-N KUBE-FIREWALL
-N KUBE-FORWARD
-N KUBE-KUBELET-CANARY
-N KUBE-NODEPORTS
-N KUBE-PROXY-CANARY
-N KUBE-SERVICES
-A INPUT -m comment --comment "kubernetes health check service ports" -j KUBE-NODEPORTS
-A INPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A INPUT -j KUBE-FIREWALL
-A INPUT -p icmp -j ACCEPT
-A FORWARD -m comment --comment "kubernetes forwarding rules" -j KUBE-FORWARD
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A FORWARD -m conntrack --ctstate NEW -m comment --comment "kubernetes externally-visible service portals" -j KUBE-EXTERNAL-SERVICES
-A FORWARD -s 10.244.0.0/16 -j ACCEPT
-A FORWARD -d 10.244.0.0/16 -j ACCEPT
-A OUTPUT -m conntrack --ctstate NEW -m comment --comment "kubernetes service portals" -j KUBE-SERVICES
-A OUTPUT -j KUBE-FIREWALL
-A KUBE-FIREWALL -m comment --comment "kubernetes firewall for dropping marked packets" -m mark --mark 0x8000/0x8000 -j DROP
-A KUBE-FIREWALL ! -s 127.0.0.0/8 -d 127.0.0.0/8 -m comment --comment "block incoming localnet connections" -m conntrack ! --ctstate RELATED,ESTABLISHED,DNAT -j DROP
-A KUBE-FORWARD -m conntrack --ctstate INVALID -j DROP
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding rules" -m mark --mark 0x4000/0x4000 -j ACCEPT
-A KUBE-FORWARD -m comment --comment "kubernetes forwarding conntrack rule" -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
[更新] 运行ip neighbor命令显示具有 REACHABLE 状态的正确条目。
从节点 1
$ ip neigh show | grep 172.16.25.5
172.16.25.52 dev eno1 lladdr b4:7a:f1:e2:8e:24 REACHABLE
172.16.25.53 dev eno1 lladdr b4:7a:f1:e2:94:f1 REACHABLE
从节点 3
$ ip neigh show | grep 172.16.25.5
172.16.25.52 dev eno2 lladdr b4:7a:f1:e2:8e:24 REACHABLE
172.16.25.51 dev eno2 lladdr b4:7a:f1:e2:8e:24 REACHABLE
[更新] 运行 wireshark 后
根据 wireshark 扫描，节点 53 确实收到了 ping 请求并返回了响应。但节点 51 从未收到响应。
| 节点 | Wireshark 扫描 |
|---|---|
| 节点 51（主） | Ping 到节点 53，但缺少响应 |
| 节点 53（工人） | 从 51 接收 Ping，并返回响应 |