带有 Veth 的 Linux 网络桥接器无法发送出站数据包

带有 Veth 的 Linux 网络桥接器无法发送出站数据包

我遇到了一个问题,我在 Ubuntu 18.04 上创建的 Linux 网络桥接器无法访问 Internet。我在 Linux 中有一个网络命名空间,我想在其中运行一个应用程序。我希望此应用程序能够将出站数据包发送到 Internet。因此,我设置了一个 veth 对并将对等体放在网络命名空间内。Veth1是主机/默认网络命名空间上的 veth,veth2是自定义网络命名空间(测试)内的 veth。然后我在主机上设置了一个 Linux 桥接器并添加veth1到它。以下是我为实现此目的而运行的命令:

# Create namespace.
ip netns add test

# Put up loopback interface.
ip netns exec test ip link set lo up

# Create veth pair.
ip link add veth1 type veth peer name veth2

# Put veth2 inside namespace.
ip link set veth2 netns test

# Add IP address to veth2 inside namespace.
ip netns exec test ip addr add 172.20.0.2/16 dev veth2

# Put veth2 up.
ip netns exec test ip link set veth2 up

# Delete default route in namespace.
ip netns exec test ip route delete default

# Add veth2 to default route in namespace.
ip netns exec test ip route add default dev veth2

# Create bridge br0.
ip link add br0 type bridge

# Add veth1 to bridge (I've also tried 'brctl addif br0 veth1').
ip link set veth1 master br0

# Add IP to br0.
ip addr add 172.20.0.1/16 dev br0

# Put br0 up.
ip link set br0 up

最初,我试图让这个功能适用于一个我没有创建的应用程序。该应用程序通过veth2网络命名空间内的接口发送出站数据包,因为这是默认路由。然而,它发送的只是 ARP 请求(who-has),它从未收到任何响应。因此,我决定创建自己的使用AF_PACKET套接字的 C 程序。这里是任何想知道的人的代码。它所做的只是绑定到特定接口并向命令行中指定的目标发送一个空的 UDP 数据包。我还做了它,以便您可以在命令行中设置源 IP。我想指出的另一件事是程序检索网关的 MAC 地址并将其用作以太网报头的目标 MAC(我不确定将目标 MAC 设置为什么,并将其设置为网关 MAC 地址应该有效,因为 ARP 请求不应该发送到网络之外的 IP)。

在网络命名空间内执行程序时如下:

ip netns exec test ./test_veth veth2 10.50.0.11 10.50.0.3

流量从未到达。我可以看到和通过 的10.50.0.3流量。以下是 的示例:veth1br0tcpdumpbr0

root@netvm02:/home/roy# tcpdump -i br0 -nne
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on br0, link-type EN10MB (Ethernet), capture size 262144 bytes
14:29:13.928570 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:29:14.928741 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:29:15.928957 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:29:16.929181 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:29:17.929412 42:7d:2a:5e:8c:78 > 00:00:00:00:00:00, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0

当我在默认网络命名空间内运行程序并附加到 时veth,我最终看不到 上的流量br0。这可能是因为我的程序将目标 MAC 设置为网关:

root@netvm02:/home/roy# tcpdump -i veth1 -nne
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on veth1, link-type EN10MB (Ethernet), capture size 262144 bytes
14:30:58.397476 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:30:59.397707 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:31:00.398022 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:31:01.398295 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
14:31:02.398544 02:a2:0f:2a:7b:bf > 78:8a:20:ba:e1:f9, ethertype IPv4 (0x0800), length 42: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0

我也尝试过连接该程序br0,但10.50.0.3仍然看不到流量。因此,我假设桥有问题。

如果我将其附加到主接口(ens18在本例中),我可以看到以下流量10.50.0.3

root@test02:/home/roy# tcpdump -i any host 10.50.0.11 and udp -nne
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
22:17:59.964569  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
22:18:00.964726  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
22:18:01.965059  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
22:18:02.965271  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0
22:18:03.965544  In 78:8a:20:ba:e1:f9 ethertype IPv4 (0x0800), length 58: 10.50.0.11.15000 > 10.50.0.3.25000: UDP, length 0

我还尝试通过(bridge-utils)将物理接口(ens18)添加到网桥:brctl

brctl addif br0 ens18

这导致虚拟机无法发送任何出站数据包并且与虚拟机的连接丢失。

我尝试通过以下方式伪装两者172.20.0.0/16br0界面:

iptables -t nat -A POSTROUTING -s 172.20.0.0/16 -j MASQUERADE
iptables -t nat -A POSTROUTING -o br0 -j MASQUERADE

不幸的是,这些都不起作用。奇怪的是,在运行程序时,我没有看到任何数据包被这些规则处理iptables -t nat -L -n -v

Chain POSTROUTING (policy ACCEPT 5 packets, 355 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 MASQUERADE  all  --  *      *       172.20.0.0/16        0.0.0.0/0
    0     0 MASQUERADE  all  --  *      br0     0.0.0.0/0            0.0.0.0/0

我还尝试将程序的源 IP 设置为,172.20.0.2看看第一条规则是否会处理数据包。遗憾的是,它没有。

我也尝试过设置net.ipv4.ip_forward1via sysctl net.ipv4.ip_forward=1。不过,我也没有成功。

以下是我在 IPTables 中尝试的转发规则:

Chain FORWARD (policy ACCEPT 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 ACCEPT     all  --  A      A       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br0    br0     0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br0    !br0    0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  A      br0     0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br0    A       0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  ens18  br0     0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  br0    ens18   0.0.0.0/0            0.0.0.0/0

我知道其中很多可能都没用,但我只是想尝试一下,看看它们是否有什么不同。

以下是更多信息,包括详细信息ifconfig

root@netvm02:/home/roy# ifconfig
veth1: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        ether 02:a2:0f:2a:7b:bf  txqueuelen 1000  (Ethernet)
        RX packets 3655  bytes 154906 (154.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2380  bytes 101548 (101.5 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

br0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.0.1  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::185a:96ff:fe62:d174  prefixlen 64  scopeid 0x20<link>
        ether 02:a2:0f:2a:7b:bf  txqueuelen 1000  (Ethernet)
        RX packets 726  bytes 55088 (55.0 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 276  bytes 12624 (12.6 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

ens18: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 10.50.0.11  netmask 255.255.255.0  broadcast 10.50.0.255
        inet6 fe80::e087:deff:fe1f:d504  prefixlen 64  scopeid 0x20<link>
        ether e2:87:de:1f:d5:04  txqueuelen 1000  (Ethernet)
        RX packets 1423812  bytes 306465717 (306.4 MB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1694988587  bytes 2103526747383 (2.1 TB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 2436  bytes 223919 (223.9 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2436  bytes 223919 (223.9 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@netvm02:/home/roy# ip netns exec test ifconfig
veth2: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.20.0.2  netmask 255.255.0.0  broadcast 0.0.0.0
        inet6 fe80::407d:2aff:fe5e:8c78  prefixlen 64  scopeid 0x20<link>
        ether 42:7d:2a:5e:8c:78  txqueuelen 1000  (Ethernet)
        RX packets 2380  bytes 101548 (101.5 KB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 3677  bytes 155830 (155.8 KB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 0  bytes 0 (0.0 B)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

root@netvm02:/home/roy# ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
2: ens18: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
    link/ether e2:87:de:1f:d5:04 brd ff:ff:ff:ff:ff:ff
    inet 10.50.0.11/24 brd 10.50.0.255 scope global dynamic ens18
       valid_lft 80490sec preferred_lft 80490sec
    inet6 fe80::e087:deff:fe1f:d504/64 scope link
       valid_lft forever preferred_lft forever
4: veth1@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master br0 state UP group default qlen 1000
    link/ether 02:a2:0f:2a:7b:bf brd ff:ff:ff:ff:ff:ff link-netnsid 0
5: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 02:a2:0f:2a:7b:bf brd ff:ff:ff:ff:ff:ff
    inet 172.20.0.1/16 scope global br0
       valid_lft forever preferred_lft forever
    inet6 fe80::185a:96ff:fe62:d174/64 scope link
       valid_lft forever preferred_lft forever

root@netvm02:/home/roy# ip netns exec test ip addr
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
       valid_lft forever preferred_lft forever
    inet6 ::1/128 scope host
       valid_lft forever preferred_lft forever
3: veth2@if4: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000
    link/ether 42:7d:2a:5e:8c:78 brd ff:ff:ff:ff:ff:ff link-netnsid 0
    inet 172.20.0.2/16 scope global veth2
       valid_lft forever preferred_lft forever
    inet6 fe80::407d:2aff:fe5e:8c78/64 scope link
       valid_lft forever preferred_lft forever


root@netvm02:/home/roy# ip route
default via 10.50.0.1 dev ens18 proto dhcp src 10.50.0.11 metric 100
10.50.0.0/24 dev ens18 proto kernel scope link src 10.50.0.11
10.50.0.1 dev ens18 proto dhcp scope link src 10.50.0.11 metric 100
172.20.0.0/16 dev br0 proto kernel scope link src 172.20.0.1

root@netvm02:/home/roy# ip netns exec test ip route
default dev veth2 scope link
172.20.0.0/16 dev veth2 proto kernel scope link src 172.20.0.2

root@netvm02:/home/roy# brctl show
bridge name     bridge id               STP enabled     interfaces
br0             8000.02a20f2a7bbf       no              veth1

此外,10.50.0.1110.50.0.3都是我家用服务器上运行 ProxMox 的虚拟机。它们在主接口 (ens18) 上使用 DHCP,但具有来自我的 Edge 路由器的静态 IP 映射。

在此之前,我还没有接触过桥梁或交通工具,因此我可能遗漏了一些东西。

我只希望来自的流量br0能够到达互联网。在上面,我正在测试本地网络上的连接,但我计划运行的应用程序将向网络外的 IP 发送数据包。

如果您需要任何其他信息,请告诉我!

非常感谢您的帮助,并感谢您的时间!

答案1

您必须将单独的网络命名空间视为不同的主机,并将 veth 对之间的连接视为外部数据包进入的线路。因此,您必须激活路由。主命名空间中的 iptables 将在 PREROUTING 和 POSTROUTING 以及 INPUT 和 OUTPUT 中看到数据包。

因此要设置出站功能(eth0用您的出站接口替换):

# Activate router functions
# Has side effects: e.g. net.ipv4.conf.all.accept_redirects=0,secure_redirects=1
# Resets ipv4 kernel interface 'all' config values to default for HOST or ROUTER
# https://www.kernel.org/doc/Documentation/networking/ip-sysctl.txt
echo 1 > /proc/sys/net/ipv4/ip_forward
# Set a gateway for the 'inside' namespace
# You have to specify an ip which will be the next hop
# This ip must be on the network segment of the main namespace veth
ip netns exec test ip route add default via 172.20.0.1
# Masquerade outgoing connections (you can limit to tcp with `-p tcp`)
iptables -t nat -A POSTROUTING -s 172.20.0.1 -o eth0 -j MASQUERADE
# If default FORWARD policy is DROP
# Let packets move from the outward interface
# to the virtual ethernet pair and vice versa
iptables -A FORWARD -i eth0 -o br0 -j ACCEPT
iptables -A FORWARD -o eth0 -i br0 -j ACCEPT
# Setup a resolver (replace with your own DNS, does not work with a loopback resolver)
mkdir -p /etc/netns/test
echo nameserver dns-ip > /etc/netns/test/resolv.conf
# Maybe give it its own hosts file, to do edits
cp /etc/hosts /etc/netns/test/hosts

现在你可以测试ip netns exec test ping example.com

相关内容