我基本上遵循了这个指南: 具有公共 IP 的 Docker 容器
我们在另一个地方已经有类似的设置,但我无法让它在新环境中工作。遗憾的是,我的前任没有记录任何内容,所以我试图对设置进行逆向工程。
Docker 主机:
10.10.60.41/24
使用docker桥接网络:
docker network create --subnet=10.60.0.0/16 --opt "com.docker.network.bridge.name"="br-ext" ext
Docker 主机上的路由:
# ip r
default via 10.10.60.1 dev br0 proto static
10.10.60.0/24 dev br0 proto kernel scope link src 10.10.60.41
10.60.0.0/16 dev br-ext proto kernel scope link src 10.60.0.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdow
运行docker容器:
docker run --network=ext -itd --name=web nginx
该 Docker 容器已10.60.0.2
分配 IP。
ping 10.60.0.2
或者curl 10.80.0.2
从docker主机工作正常...正如预期的那样。
但无法从网络访问 docker 容器。已设置10.60.0.0/16
到 docker 主机主 IP 的网络路由。10.10.60.41
# iptables -L
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain FORWARD (policy ACCEPT)
target prot opt source destination
DOCKER-USER all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-1 all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere ctstate RELATED,ESTABLISHED
DOCKER all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
ACCEPT all -- anywhere anywhere
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain DOCKER (2 references)
target prot opt source destination
Chain DOCKER-ISOLATION-STAGE-1 (1 references)
target prot opt source destination
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
DOCKER-ISOLATION-STAGE-2 all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-ISOLATION-STAGE-2 (2 references)
target prot opt source destination
DROP all -- anywhere anywhere
DROP all -- anywhere anywhere
RETURN all -- anywhere anywhere
Chain DOCKER-USER (1 references)
target prot opt source destination
RETURN all -- anywhere anywhere
# iptables -t nat -L -n -v
Chain PREROUTING (policy ACCEPT 35363 packets, 2140K bytes)
pkts bytes target prot opt in out source destination
140K 8413K DOCKER all -- * * 0.0.0.0/0 0.0.0.0/0 ADDRTYPE match dst-type LOCAL
Chain INPUT (policy ACCEPT 24828 packets, 1495K bytes)
pkts bytes target prot opt in out source destination
Chain OUTPUT (policy ACCEPT 286 packets, 19813 bytes)
pkts bytes target prot opt in out source destination
6 504 DOCKER all -- * * 0.0.0.0/0 !127.0.0.0/8 ADDRTYPE match dst-type LOCAL
Chain POSTROUTING (policy ACCEPT 10799 packets, 659K bytes)
pkts bytes target prot opt in out source destination
6 504 MASQUERADE all -- * !br-ext 10.60.0.0/16 0.0.0.0/0
0 0 MASQUERADE all -- * !docker0 172.17.0.0/16 0.0.0.0/0
Chain DOCKER (2 references)
pkts bytes target prot opt in out source destination
2 168 RETURN all -- br-ext * 0.0.0.0/0 0.0.0.0/0
0 0 RETURN all -- docker0 * 0.0.0.0/0 0.0.0.0/0
# sysctl net.ipv4.ip_forward
net.ipv4.ip_forward = 1
除了子网等之外,这两个设置基本相同。但看起来我在这里遗漏了一些东西......任何帮助都将不胜感激。
提前致谢并祝您有美好的一天!
=====
编辑-回答 larsks
是的,数据包可以到达主机/容器:10.10.60.6 > 10.60.1.25
# tcpdump -n -i any icmp
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked v1), capture size 262144 bytes
15:34:52.257656 IP 10.10.60.6 > 10.60.1.25: ICMP echo request, id 879, seq 1, length 64
15:34:52.257731 IP 10.10.60.6 > 10.60.1.25: ICMP echo request, id 879, seq 1, length 64
15:34:52.257741 IP 10.10.60.6 > 10.60.1.25: ICMP echo request, id 879, seq 1, length 64
15:34:52.257799 IP 10.60.1.25 > 10.10.60.6: ICMP echo reply, id 879, seq 1, length 64
15:34:52.257799 IP 10.60.1.25 > 10.10.60.6: ICMP echo reply, id 879, seq 1, length 64
15:34:52.257826 IP 10.60.1.25 > 10.10.60.6: ICMP echo reply, id 879, seq 1, length 64
即使发送了 ICMP 回复
10.10.60.6
在发送 ICMP 请求的主机上,没有回复
# tcpdump -i any icmp and host 10.60.1.25
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
15:36:52.042690 IP vpnconnect > 10.60.1.25: ICMP echo request, id 879, seq 118, length 64
15:36:53.066672 IP vpnconnect > 10.60.1.25: ICMP echo request, id 879, seq 119, length 64
15:36:54.090729 IP vpnconnect > 10.60.1.25: ICMP echo request, id 879, seq 120, length 64
15:36:55.114713 IP vpnconnect > 10.60.1.25: ICMP echo request, id 879, seq 121, length 6
附加信息:当从其中一个docker容器发送ICMP请求到时10.10.60.6
,这有效
$ ping 10.10.60.6
PING 10.10.60.6 (10.10.60.6): 56 data bytes
64 bytes from 10.10.60.6: seq=0 ttl=42 time=1.051 ms
64 bytes from 10.10.60.6: seq=1 ttl=42 time=0.738 ms
看起来10.10.60.6
像这样:
# tcpdump -i any icmp and host 10.10.60.41
tcpdump: verbose output suppressed, use -v or -vv for full protocol decode
listening on any, link-type LINUX_SLL (Linux cooked), capture size 262144 bytes
15:40:57.489752 IP 10.10.60.41 > host: ICMP echo request, id 42, seq 38, length 64
15:40:57.489771 IP host > 10.10.60.41: ICMP echo reply, id 42, seq 38, length 64
对 Docker 主机的请求和回复(?)
答案1
我在虚拟机中重现了您的环境,以便查看问题。您可以找到完整的配置这里。
在此配置中,我有以下三个节点:
node1
是docker主机@10.10.60.41
。node2
路由器位于10.10.60.1
node3
是网络上位于 的一个随机其他主机10.10.60.20
。
我已经按照您的示例创建了 docker 网络,以便在node1
容器主机上,我可以br-ext
像这样配置网桥:
node# ip addr show br-ext
5: br-ext: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default
link/ether 02:42:83:d4:10:54 brd ff:ff:ff:ff:ff:ff
inet 10.60.0.1/16 brd 10.60.255.255 scope global br-ext
valid_lft forever preferred_lft forever
inet6 fe80::42:83ff:fed4:1054/64 scope link
valid_lft forever preferred_lft forever
以及以下路由表(你可以忽略到的路由192.168.121.0/24
;这是 vagrant 处理配置的方式的产物):
default via 10.10.60.1 dev eth1
10.10.60.0/24 dev eth1 proto kernel scope link src 10.10.60.41 metric 101
10.60.0.0/16 dev br-ext proto kernel scope link src 10.60.0.1
172.17.0.0/16 dev docker0 proto kernel scope link src 172.17.0.1 linkdown
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.12 metric 100
node2
在网络路由器上,我有:
default via 192.168.121.1 dev eth0 proto dhcp metric 100
10.10.60.0/24 dev eth1 proto kernel scope link src 10.10.60.1 metric 101
10.60.0.0/16 via 10.10.60.41 dev eth1 proto static metric 101
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.181 metric 100
在node3
,我有:
default via 10.10.60.1 dev eth1
10.10.60.0/24 dev eth1 proto kernel scope link src 10.10.60.20 metric 101
192.168.121.0/24 dev eth0 proto kernel scope link src 192.168.121.184 metric 100
有了上述配置,如果我运行node1
:
node1# tcpdump -n -i any icmp
然后node3
我继续跑:
node3# ping 10.60.0.2
我在输出中看到tcpdump
:
14:12:25.777825 eth1 In IP 10.10.60.1 > 10.60.0.2: ICMP echo request, id 2, seq 1, length 64
14:12:26.836689 eth1 In IP 10.10.60.1 > 10.60.0.2: ICMP echo request, id 2, seq 2, length 64
14:12:27.860833 eth1 In IP 10.10.60.1 > 10.60.0.2: ICMP echo request, id 2, seq 3, length 64
因此,ICMP 回显请求显示在 上node1
,这意味着我们的网络路由是正确的(node3
正在通过 发送请求node2
,网络路由器正在正确地将它们传递给node1
)...但我们没有看到任何回复。可能是什么原因?
我们可以使用一种机制来诊断这个问题,就是在我们的 netfilter 配置中启用数据包跟踪node1
:
node1# iptables -t raw -A PREROUTING -s 10.10.60.20 -j TRACE
现在,当我们尝试ping
从node3
到时node1
,我们会看到以下内核日志:
Aug 15 14:19:05 node1 kernel: TRACE: raw:PREROUTING:policy:2 IN=eth1 OUT= MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: nat:PREROUTING:policy:2 IN=eth1 OUT= MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=63 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:rule:1 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:DOCKER-USER:return:1 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:rule:2 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:DOCKER-ISOLATION-STAGE-1:return:3 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:rule:4 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:DOCKER:return:1 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:policy:11 IN=eth1 OUT=br-ext MAC=52:54:00:95:12:24:52:54:00:f4:d3:e4:08:00 SRC=10.10.60.20 DST=10.60.0.2 LEN=84 TOS=0x00 PREC=0x00 TTL=62 ID=20048 DF PROTO=ICMP TYPE=8 CODE=0 ID=3 SEQ=1
(此时请不要忘记禁用跟踪:iptables -t raw -F PREROUTING
。)
查看上述日志,我们看到我们的数据包最终进入了FORWARD
链,但无法匹配任何终端规则并最终掉落到底部,由默认策略进行处理:
Aug 15 14:19:05 node1 kernel: TRACE: filter:FORWARD:policy:11 ...
DROP
Docker 安装已将其设置为:
node1# iptables -S FORWARD
-P FORWARD DROP
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-ext -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-ext -j DOCKER
-A FORWARD -i br-ext ! -o br-ext -j ACCEPT
-A FORWARD -i br-ext -o br-ext -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
这里的解决方案是添加一条新规则,明确允许从主机网络到容器网络的流量。在node1
:
node1# iptables -A FORWARD -s 10.10.60.0/24 -d 10.60.0.0/16 -j ACCEPT
有了这个规则,我现在就可以成功地从 pingnode3
到nginx
容器node1
:
node3# ping -c2 10.60.0.2
PING 10.60.0.2 (10.60.0.2) 56(84) bytes of data.
64 bytes from 10.60.0.2: icmp_seq=1 ttl=63 time=0.417 ms
64 bytes from 10.60.0.2: icmp_seq=2 ttl=63 time=0.328 ms
--- 10.60.0.2 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1047ms
rtt min/avg/max/mdev = 0.328/0.372/0.417/0.044 ms
我还可以成功访问容器中运行的 Nginx 实例web
:
node3# curl 10.60.0.2
<!DOCTYPE html>
<html>
<head>
<title>Welcome to nginx!</title>
.
.
.
反向连接同样有效;node1
:
node1# docker exec -it apt-get update
node1# docker exec -it apt-get -y install iputils-ping
node1# docker exec -it web ping -c2 10.10.60.20
PING 10.10.60.20 (10.10.60.20) 56(84) bytes of data.
64 bytes from 10.10.60.20: icmp_seq=1 ttl=63 time=0.304 ms
64 bytes from 10.10.60.20: icmp_seq=2 ttl=63 time=0.378 ms
--- 10.10.60.20 ping statistics ---
2 packets transmitted, 2 received, 0% packet loss, time 1024ms
rtt min/avg/max/mdev = 0.304/0.341/0.378/0.037 ms
这可能与您的配置不完全匹配:例如,在您的输出中,您的FORWARD
链具有默认策略ACCEPT
,这表明可能发生了其他事情……但希望它能为您提供一些可以解决问题的方法。
如果您想通过有关您的环境的更多信息来更新您的问题,我很乐意再看一遍。
答案2
默认情况下,您无法从外部主机 ping Docker 容器。
默认情况下,Docker 容器内运行的任何服务都不是“已发布”的(Docker 术语),无法从外部访问。您必须在运行容器时明确定义/允许要发布的服务。
就您而言,我没有在您的 docker run 命令中看到任何端口发布,而且您正在使用 nginx,所以我想您至少应该发布 80 个端口。