具有“公共”IP、桥接网络的 docker 容器

具有“公共”IP、桥接网络的 docker 容器

我基本上遵循了这个指南: 具有公共 IP 的 Docker 容器

我们在另一个地方已经有类似的设置,但我无法让它在新环境中工作。遗憾的是,我的前任没有记录任何内容,所以我试图对设置进行逆向工程。

Docker 主机: 10.10.60.41/24

使用docker桥接网络: docker network create --subnet=10.60.0.0/16 --opt "com.docker.network.bridge.name"="br-ext" ext

Docker 主机上的路由:

#  ip r
default via 10.10.60.1 dev br0 proto static 
10.10.60.0/24 dev br0 proto kernel scope link src 10.10.60.41 
10.60.0.0/16 dev br-ext proto kernel scope link src 10.60.0.1 
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdow

运行docker容器: docker run --network=ext -itd --name=web nginx

该 Docker 容器已10.60.0.2分配 IP。

ping 10.60.0.2或者curl 10.80.0.2从docker主机工作正常...正如预期的那样。

但无法从网络访问 docker 容器。已设置10.60.0.0/16到 docker 主机主 IP 的网络路由。10.10.60.41

# iptables -L
Chain INPUT (policy ACCEPT)
target     prot opt source               destination         

Chain FORWARD (policy ACCEPT)
target     prot opt source               destination         
DOCKER-USER  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-1  all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere             ctstate RELATED,ESTABLISHED
DOCKER     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            
ACCEPT     all  --  anywhere             anywhere            

Chain OUTPUT (policy ACCEPT)
target     prot opt source               destination         

Chain DOCKER (2 references)
target     prot opt source               destination         

Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target     prot opt source               destination         
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
DOCKER-ISOLATION-STAGE-2  all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target     prot opt source               destination         
DROP       all  --  anywhere             anywhere            
DROP       all  --  anywhere             anywhere            
RETURN     all  --  anywhere             anywhere            

Chain DOCKER-USER (1 references)
target     prot opt source               destination         
RETURN     all  --  anywhere             anywhere
# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 35363 packets, 2140K bytes)
 pkts bytes target     prot opt in     out     source               destination         
 140K 8413K DOCKER     all  --  *      *       0.0.0.0/0            0.0.0.0/0            ADDRTYPE match dst-type LOCAL

Chain INPUT (policy ACCEPT 24828 packets, 1495K bytes)
 pkts bytes target     prot opt in     out     source               destination         

Chain OUTPUT (policy ACCEPT 286 packets, 19813 bytes)
 pkts bytes target     prot opt in     out     source               destination         
    6   504 DOCKER     all  --  *      *       0.0.0.0/0           !127.0.0.0/8          ADDRTYPE match dst-type LOCAL

Chain POSTROUTING (policy ACCEPT 10799 packets, 659K bytes)
 pkts bytes target     prot opt in     out     source               destination         
    6   504 MASQUERADE  all  --  *      !br-ext  10.60.0.0/16         0.0.0.0/0           
    0     0 MASQUERADE  all  --  *      !docker0  172.17.0.0/16        0.0.0.0/0           

Chain DOCKER (2 references)
 pkts bytes target     prot opt in     out     source               destination         
    2   168 RETURN     all  --  br-ext *       0.0.0.0/0            0.0.0.0/0           
    0     0 RETURN     all  --  docker0 *       0.0.0.0/0            0.0.0.0/0
# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1

除了子网等之外,这两个设置基本相同。但看起来我在这里遗漏了一些东西......任何帮助都将不胜感激。

提前致谢并祝您有美好的一天!

=====

编辑-回答 larsks

是的,数据包可以到达主机/容器:10.10.60.6 > 10.60.1.25

# tcpdump -n -i any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
15:34:52.257656 IP 10.10.60.6 > 10.60.1.25: ICMP echo request, id 879, seq 1, length 64
15:34:52.257731 IP 10.10.60.6 > 10.60.1.25: ICMP echo request, id 879, seq 1, length 64
15:34:52.257741 IP 10.10.60.6 > 10.60.1.25: ICMP echo request, id 879, seq 1, length 64
15:34:52.257799 IP 10.60.1.25 > 10.10.60.6: ICMP echo reply, id 879, seq 1, length 64
15:34:52.257799 IP 10.60.1.25 > 10.10.60.6: ICMP echo reply, id 879, seq 1, length 64
15:34:52.257826 IP 10.60.1.25 > 10.10.60.6: ICMP echo reply, id 879, seq 1, length 64

即使发送了 ICMP 回复

10.10.60.6在发送 ICMP 请求的主机上,没有回复

# tcpdump -i any icmp and host 10.60.1.25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
15:36:52.042690 IP vpnconnect > 10.60.1.25: ICMP echo request, id 879, seq 118, length 64
15:36:53.066672 IP vpnconnect > 10.60.1.25: ICMP echo request, id 879, seq 119, length 64
15:36:54.090729 IP vpnconnect > 10.60.1.25: ICMP echo request, id 879, seq 120, length 64
15:36:55.114713 IP vpnconnect > 10.60.1.25: ICMP echo request, id 879, seq 121, length 6

附加信息:当从其中一个docker容器发送ICMP请求到时10.10.60.6,这有效

$ ping 10.10.60.6
PING 10.10.60.6 (10.10.60.6): 56 data bytes
64 bytes from 10.10.60.6: seq=0 ttl=42 time=1.051 ms
64 bytes from 10.10.60.6: seq=1 ttl=42 time=0.738 ms

看起来10.10.60.6像这样:

# tcpdump -i any icmp and host 10.10.60.41
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
15:40:57.489752 IP 10.10.60.41 > host: ICMP echo request, id 42, seq 38, length 64
15:40:57.489771 IP host > 10.10.60.41: ICMP echo reply, id 42, seq 38, length 64

对 Docker 主机的请求和回复(?)

答案1

我在虚拟机中重现了您的环境,以便查看问题。您可以找到完整的配置这里


在此配置中,我有以下三个节点:

  • node1是docker主机@ 10.10.60.41
  • node2路由器位于10.10.60.1
  • node3是网络上位于 的一个随机其他主机10.10.60.20

我已经按照您的示例创建了 docker 网络,以便在node1容器主机上,我可以br-ext像这样配置网桥:

node# ip addr show br-ext
5: br-ext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
    link/ether 02:42:83:d4:10:54 brd ff:ff:ff:ff:ff:ff
    inet 10.60.0.1/16 brd 10.60.255.255 scope global br-ext
       valid_lft forever preferred_lft forever
    inet6 fe80::42:83ff:fed4:1054/64 scope link
       valid_lft forever preferred_lft forever

以及以下路由表(你可以忽略到的路由192.168.121.0/24;这是 vagrant 处理配置的方式的产物):

default via 10.10.60.1 dev eth1
10.10.60.0/24 dev eth1 proto kernel scope link src 10.10.60.41 metric 101
10.60.0.0/16 dev br-ext proto kernel scope link src 10.60.0.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.12 metric 100

node2在网络路由器上,我有:

default via 192.168.121.1 dev eth0 proto dhcp metric 100
10.10.60.0/24 dev eth1 proto kernel scope link src 10.10.60.1 metric 101
10.60.0.0/16 via 10.10.60.41 dev eth1 proto static metric 101
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.181 metric 100

node3,我有:

default via 10.10.60.1 dev eth1
10.10.60.0/24 dev eth1 proto kernel scope link src 10.10.60.20 metric 101
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.184 metric 100

有了上述配置,如果我运行node1

node1# tcpdump -n -i any icmp

然后node3我继续跑:

node3# ping 10.60.0.2

我在输出中看到tcpdump

14:12:25.777825 eth1  In  IP 10.10.60.1 > 10.60.0.2: ICMP echo request, id 2, seq 1, length 64
14:12:26.836689 eth1  In  IP 10.10.60.1 > 10.60.0.2: ICMP echo request, id 2, seq 2, length 64
14:12:27.860833 eth1  In  IP 10.10.60.1 > 10.60.0.2: ICMP echo request, id 2, seq 3, length 64

因此,ICMP 回显请求显示在 上node1,这意味着我们的网络路由是正确的(node3正在通过 发送请求node2,网络路由器正在正确地将它们传递给node1)...但我们没有看到任何回复。可能是什么原因?

我们可以使用一种机制来诊断这个问题,就是在我们的 netfilter 配置中启用数据包跟踪node1

node1# iptables -t raw -A PREROUTING -s 10.10.60.20 -j TRACE

现在,当我们尝试pingnode3到时node1,我们会看到以下内核日志:

Aug 15 14:19:05 node1 kernel: TRACE: raw:PREROUTING:policy:2 IN=eth1 OUT= MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: nat:PREROUTING:policy:2 IN=eth1 OUT= MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:rule:1 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:DOCKER-USER:return:1 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:rule:2 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:3 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:rule:4 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:DOCKER:return:1 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:policy:11 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1

(此时请不要忘记禁用跟踪:iptables -t raw -F PREROUTING。)

查看上述日志,我们看到我们的数据包最终进入了FORWARD链,但无法匹配任何终端规则并最终掉落到底部,由默认策略进行处理:

Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:policy:11 ...

DROPDocker 安装已将其设置为:

node1# iptables -S FORWARD
-P FORWARD DROP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-ext -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-ext -j DOCKER
-A FORWARD -i br-ext ! -o br-ext -j ACCEPT
-A FORWARD -i br-ext -o br-ext -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT

这里的解决方案是添加一条新规则,明确允许从主机网络到容器网络的流量。在node1

node1# iptables -A FORWARD -s 10.10.60.0/24 -d 10.60.0.0/16 -j ACCEPT

有了这个规则,我现在就可以成功地从 pingnode3nginx容器node1

node3# ping -c2 10.60.0.2
PING 10.60.0.2 (10.60.0.2) 56(84) bytes of data.
64 bytes from 10.60.0.2: icmp_seq=1 ttl=63 time=0.417 ms
64 bytes from 10.60.0.2: icmp_seq=2 ttl=63 time=0.328 ms

--- 10.60.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1047ms
rtt min/avg/max/mdev = 0.328/0.372/0.417/0.044 ms

我还可以成功访问容器中运行的 Nginx 实例web

node3# curl 10.60.0.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
.
.
.

反向连接同样有效;node1

node1# docker exec -it apt-get update
node1# docker exec -it apt-get -y install iputils-ping
node1# docker exec -it web ping -c2 10.10.60.20
PING 10.10.60.20 (10.10.60.20) 56(84) bytes of data.
64 bytes from 10.10.60.20: icmp_seq=1 ttl=63 time=0.304 ms
64 bytes from 10.10.60.20: icmp_seq=2 ttl=63 time=0.378 ms

--- 10.10.60.20 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.304/0.341/0.378/0.037 ms

这可能与您的配置不完全匹配:例如,在您的输出中,您的FORWARD链具有默认策略ACCEPT,这表明可能发生了其他事情……但希望它能为您提供一些可以解决问题的方法。

如果您想通过有关您的环境的更多信息来更新您的问题,我很乐意再看一遍。

答案2

默认情况下,您无法从外部主机 ping Docker 容器。
默认情况下,Docker 容器内运行的任何服务都不是“已发布”的(Docker 术语),无法从外部访问。您必须在运行容器时明确定义/允许要发布的服务。
就您而言,我没有在您的 docker run 命令中看到任何端口发布,而且您正在使用 nginx,所以我想您至少应该发布 80 个端口。

相关内容