昨天我意识到 Proxmox 服务器上的一个 LXC 容器需要访问 github.com,因此我必须添加 IPv4 支持(真的是 Github?)。我在 vmbr0 接口后面的内部网络中添加了一个 /29 子网。但是,无法将数据包路由回容器。当然,从外部连接也是如此。
无论如何,我过去已经在其他几台 Proxmox 服务器上这样做过。从来没有遇到过问题。此外,IPv6 运行正常,没有任何问题。据我所知,我在至少另外 2 台服务器上进行了相同的设置,只是 IPv4 子网不同。
Proxmox服务器上的接口配置如下:
auto enp35s0 # public interface
iface enp35s0 inet static
address N.M.173.126 # public server address provided by ISP
netmask 26
gateway N.M.173.65
up route add -net N.M.173.64 netmask 255.255.255.192 gw N.M.173.65 dev enp35s0
auto vmbr0
iface vmbr0 inet static
address X.Y.163.145 # subnet assigned by ISP to this server is X.Y.163.144/29
netmask 29
bridge-ports none
bridge-stp off
bridge-fd 0
这是路由表:
default via N.M.173.65 dev enp35s0
X.Y.163.144/29 dev vmbr0 proto kernel scope link src X.Y.163.145
N.M.173.64/26 via N.M.173.65 dev enp35s0
N.M.173.64/26 dev enp35s0 proto kernel scope link src N.M.173.126
现在,LXC 容器在其 eth0 接口上分配了 IP XY163.146:
auto lo
iface lo inet loopback
auto eth0
iface eth0 inet static
address X.Y.163.146
netmask 255.255.255.248
gateway X.Y.163.145
基本网络工作正常。Proxmox (.145) 可以 ping 容器 (.146),容器可以 ping 虚拟机管理程序。
现在,当从容器 ping github.com 时,我可以在 Proxmox 服务器的 enp35s0 上使用 tcpdump 进行监听。我看到 ICMP 请求发出,回复进来(Github.com 位于 140.82.118.3):
11:54:35.131596 IP X.Y.163.146 > 140.82.118.3: ICMP echo request, id 1204, seq 5, length 64
11:54:35.143779 IP 140.82.118.3 > X.Y.163.146: ICMP echo reply, id 1204, seq 5, length 64
但在 vmbr0 接口上监听时,我只看到传出的数据包。这意味着路由回内部网络不起作用。
为了测试目的,我停止了 pve-firewall。iptables 如下所示:
Chain PREROUTING (policy ACCEPT 18825 packets, 6190K bytes)
pkts bytes target prot opt in out source destination
19953 6512K LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4
Chain INPUT (policy ACCEPT 16343 packets, 6057K bytes)
pkts bytes target prot opt in out source destination
Chain FORWARD (policy ACCEPT 200 packets, 16320 bytes)
pkts bytes target prot opt in out source destination
200 16320 LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4
Chain OUTPUT (policy ACCEPT 14420 packets, 5998K bytes)
pkts bytes target prot opt in out source destination
Chain POSTROUTING (policy ACCEPT 14620 packets, 6015K bytes)
pkts bytes target prot opt in out source destination
14659 6027K LOG all -- * * 0.0.0.0/0 0.0.0.0/0 LOG flags 0 level 4
其余一切都是空的并且政策是接受。
当然,IPv4 转发已启用。
cat /proc/sys/net/ipv4/ip_forward
1
有人能看出我遗漏了什么吗?任何帮助进一步解决此问题的帮助都将不胜感激。
/编辑:我为所有链添加了日志规则:
对应规则:
iptables -t mangle -A PREROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "MANGLE:PREROUTING:IN:GITHUB"
iptables -t mangle -A FORWARD -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "MANGLE:FORWARD:IN:GITHUB"
iptables -t mangle -A POSTROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "MANGLE:POSTROUTING:IN:GITHUB"
iptables -t nat -A PREROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "NAT:PREROUTING:IN:GITHUB"
iptables -t filter -A FORWARD -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "FILTER:FORWARD:IN:GITHUB"
iptables -t nat -A POSTROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "NAT:POSTROUTING:IN:GITHUB"
iptables -t raw -A PREROUTING -s 140.82.0.0/16 -p icmp -j LOG --log-prefix "RAW:PREROUTING:IN:GITHUB"
iptables -t mangle -A PREROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:PREROUTING:OUT:PASSBOLT"
iptables -t mangle -A FORWARD -s X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:FORWARD:OUT:PASSBOLT"
iptables -t mangle -A POSTROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:POSTROUTING:OUT:PASSBOLT"
iptables -t nat -A PREROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "NAT:PREROUTING:OUT:PASSBOLT"
iptables -t filter -A FORWARD -s X.Y.163.146 -p icmp -j LOG --log-prefix "FILTER:FORWARD:OUT:PASSBOLT"
iptables -t nat -A POSTROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "NAT:POSTROUTING:OUT:PASSBOLT"
iptables -t raw -A PREROUTING -s X.Y.163.146 -p icmp -j LOG --log-prefix "RAW:PREROUTING:OUT:PASSBOLT"
iptables -t mangle -A PREROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:PREROUTING:IN:PASSBOLT"
iptables -t mangle -A FORWARD -d X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:FORWARD:IN:PASSBOLT"
iptables -t mangle -A POSTROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "MANGLE:POSTROUTING:IN:PASSBOLT"
iptables -t nat -A PREROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "NAT:PREROUTING:IN:PASSBOLT"
iptables -t filter -A FORWARD -d X.Y.163.146 -p icmp -j LOG --log-prefix "FILTER:FORWARD:IN:PASSBOLT"
iptables -t nat -A POSTROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "NAT:POSTROUTING:IN:PASSBOLT"
iptables -t raw -A PREROUTING -d X.Y.163.146 -p icmp -j LOG --log-prefix "RAW:PREROUTING:IN:PASSBOLT"
一次 ping 的输出为:
Mar 5 15:35:39 proxmox kernel: [18347.757914] RAW:PREROUTING:OUT:LXC IN=fwbr1005i0 OUT= PHYSIN=veth1005i0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.758170] MANGLE:PREROUTING:OUT:LXC IN=fwbr1005i0 OUT= PHYSIN=veth1005i0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.758426] MANGLE:FORWARD:OUT:LXC IN=fwbr1005i0 OUT=fwbr1005i0 PHYSIN=veth1005i0 PHYSOUT=fwln1005i0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.758686] FILTER:FORWARD:OUT:LXC IN=fwbr1005i0 OUT=fwbr1005i0 PHYSIN=veth1005i0 PHYSOUT=fwln1005i0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.758963] MANGLE:POSTROUTING:OUT:LXC IN= OUT=fwbr1005i0 PHYSIN=veth1005i0 PHYSOUT=fwln1005i0 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.759189] RAW:PREROUTING:OUT:LXC IN=vmbr0 OUT= PHYSIN=fwpr1005p0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.759416] MANGLE:PREROUTING:OUT:LXC IN=vmbr0 OUT= PHYSIN=fwpr1005p0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=64 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.759642] MANGLE:FORWARD:OUT:LXC IN=vmbr0 OUT=enp35s0 PHYSIN=fwpr1005p0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.761541] FILTER:FORWARD:OUT:LXC TIN=vmbr0 OUT=enp35s0 PHYSIN=fwpr1005p0 MAC=2a:d5:27:f3:e7:23:d6:fb:c7:cc:e2:b0:08:00 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.761791] MANGLE:POSTROUTING:OUT:LXC IN= OUT=enp35s0 PHYSIN=fwpr1005p0 SRC=X.Y.163.146 DST=140.82.118.4 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=53622 DF PROTO=ICMP TYPE=8 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.773606] RAW:PREROUTING:IN:GITHUB IN=enp35s0 OUT= MAC=a8:a1:59:0e:aa:e7:80:7f:f8:79:1c:96:08:00 SRC=140.82.118.4 DST=X.Y.163.146 LEN=84 TOS=0x00 PREC=0x00 TTL=58 ID=34871 PROTO=ICMP TYPE=0 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.773831] RAW:PREROUTING:IN:LXC IN=enp35s0 OUT= MAC=a8:a1:59:0e:aa:e7:80:7f:f8:79:1c:96:08:00 SRC=140.82.118.4 DST=X.Y.163.146 LEN=84 TOS=0x00 PREC=0x00 TTL=58 ID=34871 PROTO=ICMP TYPE=0 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.774051] MANGLE:PREROUTING:IN:GITHUB IN=enp35s0 OUT= MAC=a8:a1:59:0e:aa:e7:80:7f:f8:79:1c:96:08:00 SRC=140.82.118.4 DST=X.Y.163.146 LEN=84 TOS=0x00 PREC=0x00 TTL=58 ID=34871 PROTO=ICMP TYPE=0 CODE=0 ID=2554 SEQ=1
Mar 5 15:35:39 proxmox kernel: [18347.774288] MANGLE:PREROUTING:IN:LXC IN=enp35s0 OUT= MAC=a8:a1:59:0e:aa:e7:80:7f:f8:79:1c:96:08:00 SRC=140.82.118.4 DST=X.Y.163.146 LEN=84 TOS=0x00 PREC=0x00 TTL=58 ID=34871 PROTO=ICMP TYPE=0 CODE=0 ID=2554 SEQ=1
因此似乎触发了“MANGLE:PREROUTING:IN:LXC”规则。但是为什么数据包没有到达 FORWARD 链?为了以防万一,我还为 INPUT 链添加了一条日志规则。那里也没有条目。似乎数据包在没有任何通知/规则的情况下被丢弃了?!
答案1
感谢所有考虑这个问题的人。
所以解决办法是echo 1 > /proc/sys/net/ipv4/conf/enp35s0/forwarding
。有人能解释一下原因吗?