将 WireGuard 从 docker 转换为 k8s,握手正常但无法访问互联网

将 WireGuard 从 docker 转换为 k8s,握手正常但无法访问互联网

问题:我在 Docker 中有一个可用的 WireGuard 设置(请参阅指南:关联),但在将配置移植到具有主机网络的 Kubernetes 时,很难实现客户端的互联网访问。我可以进行握手,甚至可以 ping 主机的 LAN IP,但似乎无法到达默认网关。

请注意,我使用 21421 作为外部端口,并将流量转发到 51820。我的 wireguard 子网是 10.14.14.0/24 和 2601:204:xxxx:xxxc::/64;我的 LAN 子网是 10.0.0.0/24 和 2601:204:xxxx:xxx0::/64。

配置映射.yaml

apiVersion: v1
kind: ConfigMap
metadata:
  name: wireguard-config
data:
  PUID: "1000"
  PGID: "1000"
  TZ: "America/Los_Angeles"
  SERVERURL: my.website.addr
  SERVERPORT: "21421"
  PEERS: pphone,wphone,tablet,laptop,trouter
  PEERDNS: 75.75.75.75,75.75.76.76,2001:558:feed::1,2001:558:feed::2
  INTERNAL_SUBNET: 10.14.14.0/24
  ALLOWEDIPS: 0.0.0.0/0, ::/0
  PERSISTENTKEEPALIVE_PEERS: all 

部署.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: wireguard
spec:
  selector:
    matchLabels:
      app: wireguard
  replicas: 1
  template:
    metadata:
      labels:
        app: wireguard
    spec:
      nodeSelector:
        kubernetes.io/hostname: obsidiana
      hostNetwork: true
      containers:
      - name: wireguard
        image: linuxserver/wireguard:latest
        securityContext:
          privileged: true 
          capabilities:
            add:
              - NET_ADMIN
              - SYS_MODULE
        volumeMounts:
          - name: wireguard-configfiles
            mountPath: /config
          - name: lib-modules
            mountPath: /lib/modules
        envFrom:
          - configMapRef:
              name: wireguard-config
      volumes:
        - name: wireguard-configfiles
          hostPath:
            path: /srv/wireguard/config
        - name: lib-modules
          hostPath:
            path: /lib/modules

此外,以下是主机上的 IP 路由(请注意 wireguard 子网 10.14.14.0/24 和 2601:204:xxxx:xxxc::/64 的存在):

atom@obsidiana [10:53:18] [/srv/wireguard] 
-> % ip -c route
default via 10.0.0.1 dev enp3s0 
default via 10.0.0.1 dev enp3s0 proto dhcp src 10.0.0.238 metric 100 
10.0.0.0/24 dev enp3s0 proto kernel scope link src 10.0.0.238 metric 100 
10.0.0.1 dev enp3s0 proto dhcp scope link src 10.0.0.238 metric 100 
10.14.14.2 dev wg0 scope link 
10.14.14.3 dev wg0 scope link 
10.14.14.4 dev wg0 scope link 
10.14.14.5 dev wg0 scope link 
10.14.14.6 dev wg0 scope link 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown 
172.18.0.0/16 dev br-1b4d200d1cbb proto kernel scope link src 172.18.0.1 linkdown 
172.19.0.0/16 dev br-a1be084c54c9 proto kernel scope link src 172.19.0.1 linkdown 
172.21.0.0/16 dev br-4d301d3707dd proto kernel scope link src 172.21.0.1 
172.25.0.0/16 dev br-8745f19da673 proto kernel scope link src 172.25.0.1 
172.26.0.0/16 dev br-d9ec277ec93b proto kernel scope link src 172.26.0.1 
172.27.0.0/16 dev br-8a6e7b3004eb proto kernel scope link src 172.27.0.1 
192.168.48.0/20 dev br-45b26225ad0a proto kernel scope link src 192.168.48.1 linkdown 
192.168.67.0/24 dev br-2fe8a6223784 proto kernel scope link src 192.168.67.1 linkdown 
192.168.122.0/24 dev virbr0 proto kernel scope link src 192.168.122.1 linkdown 
blackhole 192.168.139.128/26 proto 80 
192.168.139.154 dev cali151eafd1c9f scope link 
192.168.139.160 dev calia50db85314e scope link 
192.168.139.164 dev calia28aed46668 scope link 
192.168.139.166 dev calib00d4512918 scope link 
192.168.139.167 dev cali2018d45df2e scope link 
192.168.139.168 dev cali339a2a73fab scope link 
192.168.139.169 dev calia8fc0d7cff4 scope link 
192.168.139.170 dev cali5d667b293c0 scope link 
192.168.139.172 dev calic7ba6791d16 scope link 
192.168.139.173 dev calif47c6967706 scope link 
192.168.139.174 dev caliaeb0ffaab04 scope link 
192.168.139.175 dev caliaf5a7cc0076 scope link 
192.168.139.176 dev cali4497ec7f2ec scope link 
192.168.176.0/20 dev br-3606b1dbef9e proto kernel scope link src 192.168.176.1 
192.168.190.64/26 via 10.0.0.1 dev enp3s0 proto 80 onlink 

atom@obsidiana [10:57:51] [/srv/wireguard] 
-> % ip -c -6 route
::1 dev lo proto kernel metric 256 pref medium
2601:204:xxxx:xxx0::/64 dev enp3s0 proto ra metric 100 expires 3588sec pref medium
2601:204:xxxx:xxxc::1 dev wg0 proto kernel metric 256 pref medium
2601:204:xxxx:xxxc::2 dev wg0 metric 1024 pref medium
2601:204:xxxx:xxxc::3 dev wg0 metric 1024 pref medium
2601:204:xxxx:xxxc::4 dev wg0 metric 1024 pref medium
2601:204:xxxx:xxxc::5 dev wg0 metric 1024 pref medium
2601:204:xxxx:xxxc::6 dev wg0 metric 1024 pref medium
fd2b:938d:7743:1::/64 proto ra metric 100 expires 1655sec pref medium
    nexthop via fe80::d358:7828:fa79:4a97 dev enp3s0 weight 1 
    nexthop via fe80::d9c7:c6cc:58c8:1181 dev enp3s0 weight 1 
fe80::/64 dev enp3s0 proto kernel metric 256 pref medium
fe80::/64 dev br-45b26225ad0a proto kernel metric 256 linkdown pref medium
fe80::/64 dev br-4d301d3707dd proto kernel metric 256 pref medium
fe80::/64 dev br-8745f19da673 proto kernel metric 256 pref medium
fe80::/64 dev vethca97195 proto kernel metric 256 pref medium
fe80::/64 dev br-d9ec277ec93b proto kernel metric 256 pref medium
fe80::/64 dev veth3e9a2b2 proto kernel metric 256 pref medium
fe80::/64 dev br-3606b1dbef9e proto kernel metric 256 pref medium
fe80::/64 dev veth5f2e53f proto kernel metric 256 pref medium
fe80::/64 dev br-8a6e7b3004eb proto kernel metric 256 pref medium
fe80::/64 dev veth42b0ce5 proto kernel metric 256 pref medium
fe80::/64 dev veth4730c27 proto kernel metric 256 pref medium
fe80::/64 dev cali151eafd1c9f proto kernel metric 256 pref medium
fe80::/64 dev calia50db85314e proto kernel metric 256 pref medium
fe80::/64 dev calib00d4512918 proto kernel metric 256 pref medium
fe80::/64 dev cali2018d45df2e proto kernel metric 256 pref medium
fe80::/64 dev cali339a2a73fab proto kernel metric 256 pref medium
fe80::/64 dev calia28aed46668 proto kernel metric 256 pref medium
fe80::/64 dev cali5d667b293c0 proto kernel metric 256 pref medium
fe80::/64 dev calia8fc0d7cff4 proto kernel metric 256 pref medium
fe80::/64 dev calif47c6967706 proto kernel metric 256 pref medium
fe80::/64 dev caliaeb0ffaab04 proto kernel metric 256 pref medium
fe80::/64 dev caliaf5a7cc0076 proto kernel metric 256 pref medium
fe80::/64 dev cali4497ec7f2ec proto kernel metric 256 pref medium
fe80::/64 dev calic7ba6791d16 proto kernel metric 256 pref medium
fe80::/64 dev veth3c7f6d9 proto kernel metric 256 pref medium
default via fe80::6cf2:67ff:fed0:9b95 dev enp3s0 proto ra metric 100 expires 1788sec pref medium

我已经调整了主机上的防火墙规则以适应主机网络(请注意 wg0 的存在,以及 wireguard 子网 10.14.14.0/24、2601:204:xxxx:xxxc::/64)。

trusted (active)
  target: ACCEPT
  icmp-block-inversion: no
  interfaces: enp3s0 wg0
  sources: 2601:204:xxxx:xxx0::/64 2601:204:xxxx:xxxc::/64 10.14.14.0/24 10.0.0.0/24 192.168.0.0/16
  services: 
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

在具有活动客户端的网关/路由器上运行tcpdump -i br0 udp and port 51820显示双向流量(br0 是 LAN iface,obsidiana 是托管 WireGuard 的 PC):

listening on br0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
16:10:52.858477 IP obsidiana.51820 > 172.56.168.229.41909: UDP, length 32
16:10:52.858919 IP obsidiana.51820 > 172.56.168.229.41909: UDP, length 148
16:10:53.810684 IP 172.56.168.229.41909 > obsidiana.51820: UDP, length 92
16:10:53.810900 IP obsidiana.51820 > 172.56.168.229.41909: UDP, length 32
16:10:55.867321 IP 108.147.99.17.35334 > obsidiana.51820: UDP, length 148
16:10:55.867700 IP obsidiana.51820 > 108.147.99.17.35334: UDP, length 92
16:10:55.948070 IP 108.147.99.17.35334 > obsidiana.51820: UDP, length 96
16:10:55.948476 IP 108.147.99.17.35334 > obsidiana.51820: UDP, length 96
16:10:56.272068 IP 108.147.99.17.35334 > obsidiana.51820: UDP, length 128

我还可以看到来自路由器的双向流量tcpdump -i enp10s0 udp and port 21421(enp10s0 是 WAN,21421 是 wireguard 的外部端口):

18:03:54.241853 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 112
18:03:54.248918 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 112
18:03:54.669307 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:54.679954 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:55.269114 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 96
18:03:55.285552 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 96
18:03:55.758942 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:55.774862 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:55.835307 IP c-73-151-158-xxx.hsd1.ca.comcast.net.21421 > 172.56.168.229.41909: UDP, length 32
18:03:56.769571 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:56.774526 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:56.859496 IP c-73-151-158-xxx.hsd1.ca.comcast.net.21421 > 108.147.99.18.60458: UDP, length 32
18:03:57.688746 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:57.691103 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:58.776023 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:58.776023 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:59.791058 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128
18:03:59.791058 IP 108.147.99.18.60458 > c-73-151-158-xxx.hsd1.ca.comcast.net.21421: UDP, length 128

最后但同样重要的一点是,以下是默认网关(firewalld)的相关防火墙设置:

➜  ~ sudo firewall-cmd --list-all --zone=home
home (active)
  target: default
  icmp-block-inversion: no
  interfaces: br0 wg0
  sources: 192.168.0.0/16 10.0.0.0/24 2601:204:xxxx:xxx0::/64 2601:204:xxxx:xxxc::/64
  services: dhcp dhcpv6-client dns dropbox-lansync elasticsearch grafana http iperf kibana kube-apiserver kube-repo kubelet mdns netbootxyz plex remote-wireguard samba-client ssh upnp wireguard
  ports: 6667/udp 49152/tcp 9101/tcp 9093/tcp 5353/udp
  protocols: igmp
  forward: yes
  masquerade: no
  forward-ports: 
  source-ports: 
  icmp-blocks: 
  rich rules: 

➜  ~ sudo firewall-cmd --list-all --zone=external
external (active)
  target: default
  icmp-block-inversion: no
  interfaces: enp10s0
  sources: 
  services: dhcpv6-client shadowsocks
  ports: 
  protocols: 
  forward: yes
  masquerade: no
  forward-ports: 
    port=21421:proto=udp:toport=51820:toaddr=10.0.0.238
  source-ports: 
  icmp-blocks: 
  rich rules: 

有什么想法可能出现什么问题吗?

相关内容