我们有一个内部 1.17.5 K8s 集群 - 5 个节点。启用 IPTables 后，我无法在集群上部署、收集日志等任何内容。
[root@lpdkubpoc01a ~]# kubectl run --generator=run-pod/v1 --rm utils -it --image quaytest.phx.aexp.com/sanupin/utils:1.0 bash
[root@lpdkubpoc01a couchbase-autonomous-operator-kubernetes_2.0.1-linux-x86_64]# kubectl exec -it hello-0 bash
[root@lpdkubpoc01a ~]# kubectl logs kube-proxy-kqs7m --tail 10 -n kube-system
[root@lpdkubpoc01a ~]# kubectl logs couchbase-operator-d9696755c-tqx57
当启用 IPTables 时，所有上述操作都会挂起。
API 服务器日志（我可以获取此日志，因为它与我的控制平面位于同一台虚拟机上，并且我已登录）显示连接到端口 10250 时出现问题
Trace[1253082920]: [11.90845011s] [11.906664621s] Transformed response object
E0728 21:39:10.658466 1 status.go:71] apiserver received an error that is not an metav1.Status: &url.Error{Op:"Get", URL:"https://10.22.77.12:10250/containerLogs/default/couchbase-operator-d9696755c-tqx57/couchbase-operator", Err:(*errors.errorString)(0xc000098260)}
I0728 21:39:10.658761 1 trace.go:116] Trace[128874851]: "Get" url:/api/v1/namespaces/default/pods/couchbase-operator-d9696755c-tqx57/log,user-agent:kubectl/v1.17.5 (linux/amd64) kubernetes/e0fccaf,client:10.22.76.244 (started: 2020-07-28 21:39:06.353221799 +0000 UTC m=+80919.504899548) (total time: 4.30550213s):
Trace[128874851]: [4.305499605s] [4.303636525s] Transformed response object
我在所有节点上配置了 10250：
[root@lpdkubpoc01a ~]# iptables -L | grep 10250
ACCEPT tcp -- anywhere anywhere tcp dpt:10250
ACCEPT tcp -- anywhere anywhere tcp spt:10250
[root@lpdkubpoc01a ~]# iptables -L | grep 10250
ACCEPT tcp -- anywhere anywhere tcp dpt:10250
ACCEPT tcp -- anywhere anywhere tcp spt:10250
[root@lpdkubpoc01a ~]# iptables -L | grep 10250
ACCEPT tcp -- anywhere anywhere tcp dpt:10250
ACCEPT tcp -- anywhere anywhere tcp spt:10250
但目前还无法访问任何日志。
我正在运行 calico pod 网络：
[root@lpdkubpoc01a couchbase-autonomous-operator-kubernetes_2.0.1-linux-x86_64]# kubectl get pods -n kube-system -owide
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
calico-kube-controllers-58c67bc699-g2dzw 1/1 Running 0 47h 192.168.76.195 lpdkubpoc01a.phx.aexp.com <none> <none>
calico-node-9khc6 1/1 Running 0 47h 10.22.77.15 lpdkubpoc01e.phx.aexp.com <none> <none>
calico-node-fc9kp 1/1 Running 0 47h 10.22.77.12 lpdkubpoc01c.phx.aexp.com <none> <none>
calico-node-htxbh 1/1 Running 0 47h 10.22.76.245 lpdkubpoc01b.phx.aexp.com <none> <none>
calico-node-q59vd 1/1 Running 0 47h 10.22.77.13 lpdkubpoc01d.phx.aexp.com <none> <none>
calico-node-zkwtr 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
coredns-598947db54-dtsjk 1/1 Running 0 47h 192.168.76.193 lpdkubpoc01a.phx.aexp.com <none> <none>
coredns-598947db54-mrjjl 1/1 Running 0 47h 192.168.76.194 lpdkubpoc01a.phx.aexp.com <none> <none>
etcd-lpdkubpoc01a.phx.aexp.com 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
kube-apiserver-lpdkubpoc01a.phx.aexp.com 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
kube-controller-manager-lpdkubpoc01a.phx.aexp.com 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
kube-proxy-2z5rx 1/1 Running 0 47h 10.22.76.245 lpdkubpoc01b.phx.aexp.com <none> <none>
kube-proxy-55jgf 1/1 Running 0 47h 10.22.77.15 lpdkubpoc01e.phx.aexp.com <none> <none>
kube-proxy-f5k5f 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
kube-proxy-gskwj 1/1 Running 0 47h 10.22.77.13 lpdkubpoc01d.phx.aexp.com <none> <none>
kube-proxy-kqs7m 1/1 Running 0 47h 10.22.77.12 lpdkubpoc01c.phx.aexp.com <none> <none>
kube-scheduler-lpdkubpoc01a.phx.aexp.com 1/1 Running 0 47h 10.22.76.244 lpdkubpoc01a.phx.aexp.com <none> <none>
下面是我在服务器上的 IPTables 配置，该服务器恰好托管一个 hello world pod：
当IPTables被禁用时，执行任何操作都没有问题。
编辑：
以下是完全不起作用的防火墙
[root@lpdkubpoc01b ~]# iptables -nvL
Chain INPUT (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
1459K 268M cali-INPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Cz_u1IQiXIMmKD4c */
790K 62M KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
790K 62M KUBE-EXTERNAL-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes externally-visible service portals */
2 144 ACCEPT icmp -- * * 0.0.0.0/0 0.0.0.0/0 /* 001 accept all icmp - Puppet Managed by fw_base */
221K 16M ACCEPT all -- lo * 0.0.0.0/0 0.0.0.0/0 /* 002 accept all to lo interface */
0 0 REJECT all -- !lo * 0.0.0.0/0 127.0.0.0/8 /* 003 reject local traffic not on loopback interface */ reject-with icmp-port-unreachable
474K 193M ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* 004 accept related established rules */ state RELATED,ESTABLISHED
2 128 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 22 /* 005 ssh - port 22 */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 8089 /* 006 splunk client - port 8089 */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 9898 /* 007 tripwire client - port 9898 */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 5666 /* 009 nrpe/nagios client - port 5666 */
789 47340 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 multiport dports 17472 /* 110 allow taniumclient access - port 17472 */
10 600 LOGIT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp flags:0x17/0x02 /* 990 forward new SYN input to LOGIT chain */
771K 61M DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* 999 drop everything else */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp dpt:10250
Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
pkts bytes target prot opt in out source destination
0 0 cali-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:wUHhoiAYhphO9Mso */
0 0 KUBE-FORWARD all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */
0 0 KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
Chain OUTPUT (policy ACCEPT 10956 packets, 1349K bytes)
pkts bytes target prot opt in out source destination
783K 84M cali-OUTPUT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tVnHkvAo15HuiPy0 */
129K 25M KUBE-SERVICES all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate NEW /* kubernetes service portals */
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 tcp spt:10250
Chain KUBE-EXTERNAL-SERVICES (1 references)
pkts bytes target prot opt in out source destination
Chain KUBE-FIREWALL (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-FORWARD (1 references)
pkts bytes target prot opt in out source destination
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 ctstate INVALID
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* kubernetes forwarding rules */ mark match 0x4000/0x4000
0 0 ACCEPT all -- * * 192.168.0.0/16 0.0.0.0/0 /* kubernetes forwarding conntrack pod source rule */ ctstate RELATED,ESTABLISHED
0 0 ACCEPT all -- * * 0.0.0.0/0 192.168.0.0/16 /* kubernetes forwarding conntrack pod destination rule */ ctstate RELATED,ESTABLISHED
Chain KUBE-KUBELET-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-PROXY-CANARY (0 references)
pkts bytes target prot opt in out source destination
Chain KUBE-SERVICES (3 references)
pkts bytes target prot opt in out source destination
Chain LOGIT (1 references)
pkts bytes target prot opt in out source destination
10 600 LOG tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* 991 configure LOGIT chain to log everything as DROP INBOUND TCP */ LOG flags 0 level 4 prefix "DROP INBOUND TCP "
Chain cali-FORWARD (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:vjrMJCRpqwy5oRoX */ MARK and 0xfff1ffff
0 0 cali-from-hep-forward all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:A_sPAO0mcxbT9mOV */ mark match 0x0/0x10000
0 0 cali-from-wl-dispatch all -- cali+ * 0.0.0.0/0 0.0.0.0/0 /* cali:8ZoYfO5HKXWbB3pk */
0 0 cali-to-wl-dispatch all -- * cali+ 0.0.0.0/0 0.0.0.0/0 /* cali:jdEuaPBe14V2hutn */
0 0 cali-to-hep-forward all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:12bc6HljsMKsmfr- */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:MH9kMp5aNICL-Olv */ /* Policy explicitly accepted packet. */ mark match 0x10000/0x10000
Chain cali-INPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT 4 -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:PajejrV4aFdkZojI */ /* Allow IPIP packets from Calico hosts */ match-set cali40all-hosts-net src ADDRTYPE match dst-type LOCAL
0 0 DROP 4 -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:_wjq-Yrma8Ly1Svo */ /* Drop IPIP packets from non-Calico hosts */
0 0 cali-wl-to-host all -- cali+ * 0.0.0.0/0 0.0.0.0/0 [goto] /* cali:8TZGxLWh_Eiz66wc */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:6McIeIDvPdL6PE1T */ mark match 0x10000/0x10000
1466K 269M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:YGPbrUms7NId8xVa */ MARK and 0xfff0ffff
1466K 269M cali-from-host-endpoint all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:2gmY7Bg2i0i84Wk_ */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:q-Vz2ZT9iGE331LL */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000
Chain cali-OUTPUT (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Mq1_rAdXXH3YkrzW */ mark match 0x10000/0x10000
0 0 RETURN all -- * cali+ 0.0.0.0/0 0.0.0.0/0 /* cali:69FkRTJDvD5Vu6Vl */
0 0 ACCEPT 4 -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:AnEsmO6bDZbQntWW */ /* Allow IPIP packets to other Calico hosts */ match-set cali40all-hosts-net dst ADDRTYPE match src-type LOCAL
787K 85M MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:9e9Uf3GU5tX--Lxy */ MARK and 0xfff0ffff
787K 85M cali-to-host-endpoint all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:OB2pzPrvQn6PC89t */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:tvSSMDBWrme3CUqM */ /* Host endpoint policy accepted packet. */ mark match 0x10000/0x10000
Chain cali-failsafe-in (0 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:wWFQM43tJU7wwnFZ */ multiport dports 22
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:LwNV--R8MjeUYacw */ multiport dports 68
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:QOO5NUOqOSS1_Iw0 */ multiport dports 179
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:cwZWoBSwVeIAZmVN */ multiport dports 2379
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:7FbNXT91kugE_upR */ multiport dports 2380
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ywE9WYUBEpve70WT */ multiport dports 6666
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:l-WQSVBf_lygPR0J */ multiport dports 6667
Chain cali-failsafe-out (0 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:82hjfji-wChFhAqL */ multiport dports 53
0 0 ACCEPT udp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:TNM3RfEjbNr72hgH */ multiport dports 67
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ycxKitIl4u3dK0HR */ multiport dports 179
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:hxjEWyxdkXXkdvut */ multiport dports 2379
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:cA_GLtruuvG88KiO */ multiport dports 2380
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Sb1hkLYFMrKS6r01 */ multiport dports 6666
0 0 ACCEPT tcp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:UwLSebGONJUG4yG- */ multiport dports 6667
Chain cali-from-hep-forward (1 references)
pkts bytes target prot opt in out source destination
Chain cali-from-host-endpoint (1 references)
pkts bytes target prot opt in out source destination
Chain cali-from-wl-dispatch (2 references)
pkts bytes target prot opt in out source destination
0 0 cali-fw-cali5bdd8f7a3d4 all -- cali5bdd8f7a3d4 * 0.0.0.0/0 0.0.0.0/0 [goto] /* cali:Miz_dfm_OqFqStOj */
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:j9w09IE3nQzJkJrt */ /* Unknown interface */
Chain cali-fw-cali5bdd8f7a3d4 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:RQw05lu9TEo6E9J7 */ ctstate RELATED,ESTABLISHED
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:01JI5c18EIipS498 */ ctstate INVALID
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:VyKbvKvpg3t6bqdZ */ MARK and 0xfffeffff
0 0 DROP udp -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:rxc0MECUB43_dUKZ */ /* Drop VXLAN encapped packets originating in pods */ multiport dports 4789
0 0 DROP 4 -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:IhtTke9qggxJCRvo */ /* Drop IPinIP encapped packets originating in pods */
0 0 cali-pro-kns.default all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:V5EIVhFIRYorU3ee */
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Gx-cKFSZy-GfPVMs */ /* Return if profile accepted */ mark match 0x10000/0x10000
0 0 cali-pro-ksa.default.default all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:FSUAC6Xrp8hOklGS */
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:80aV6ux2xgqiIrzU */ /* Return if profile accepted */ mark match 0x10000/0x10000
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:dSsEIrLZxSINZx51 */ /* Drop if no profiles matched */
Chain cali-pri-kns.default (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:7Fnh7Pv3_98FtLW7 */ MARK or 0x10000
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ZbV6bJXWSRefjK0u */ mark match 0x10000/0x10000
Chain cali-pri-ksa.default.default (1 references)
pkts bytes target prot opt in out source destination
Chain cali-pro-kns.default (1 references)
pkts bytes target prot opt in out source destination
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:oLzzje5WExbgfib5 */ MARK or 0x10000
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:4goskqvxh5xcGw3s */ mark match 0x10000/0x10000
Chain cali-pro-ksa.default.default (1 references)
pkts bytes target prot opt in out source destination
Chain cali-to-hep-forward (1 references)
pkts bytes target prot opt in out source destination
Chain cali-to-host-endpoint (1 references)
pkts bytes target prot opt in out source destination
Chain cali-to-wl-dispatch (1 references)
pkts bytes target prot opt in out source destination
0 0 cali-tw-cali5bdd8f7a3d4 all -- * cali5bdd8f7a3d4 0.0.0.0/0 0.0.0.0/0 [goto] /* cali:SsXGJ85OfhKFm0ei */
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:pP7bpa3eH6NFcNsD */ /* Unknown interface */
Chain cali-tw-cali5bdd8f7a3d4 (1 references)
pkts bytes target prot opt in out source destination
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:uNKFNt79CGfLzpK9 */ ctstate RELATED,ESTABLISHED
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ObEnKmaWBF0EWLU3 */ ctstate INVALID
0 0 MARK all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:JMSA5XB5i9j9eeav */ MARK and 0xfffeffff
0 0 cali-pri-kns.default all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:ogNpbuRSm1qaUhka */
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:owLwcUjuD59m1Twf */ /* Return if profile accepted */ mark match 0x10000/0x10000
0 0 cali-pri-ksa.default.default all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:89--cr3F1NqYju12 */
0 0 RETURN all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:KbMS7iLpFxJMiJWe */ /* Return if profile accepted */ mark match 0x10000/0x10000
0 0 DROP all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:u5nFxLY0XdpnCUkZ */ /* Drop if no profiles matched */
Chain cali-wl-to-host (1 references)
pkts bytes target prot opt in out source destination
0 0 cali-from-wl-dispatch all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:Ee9Sbo10IpVujdIY */
0 0 ACCEPT all -- * * 0.0.0.0/0 0.0.0.0/0 /* cali:nSZbcOoG1xPONxb8 */ /* Configured DefaultEndpointToHostAction */
此外，主机网络是 10.22.76.0/23
Pod 网络为 192.168.0.0/16
请帮忙！
答案1
接受到 TCP 端口 10250 的流量的规则永远不会匹配，因为它位于 INPUT 链的末尾，并且出现在 DROP 所有内容的规则之后。它应该被上移，位于记录和丢弃流量的规则之前。
答案2
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD_direct 0 -s 10.43.0.0/16 -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD_direct 1 -d 10.43.0.0/16 -j ACCEPT
sudo firewall-cmd --permanent --direct --add-rule ipv4 filter FORWARD_direct 2 -m comment --comment "Policy explicitly accepted packet." -m mark --mark 0x10000/0x10000 -j ACCEPT
sudo firewall-cmd --reload
其中 10.43.0.0/16 是我的 K8s 集群网络。在我的情况中，这是 calico 错误，将在 3.18 版本中修复。Iptables 会覆盖 calico 创建的规则，因此您应该再次为 calico 重写 iptables 规则。