我正在尝试将容器 B（10.10.1.2，使用 mysql 的自定义二进制文件并在端口 4242 上运行）的 tcp 流量通过容器 A（10.10.1.3，haproxy，以透明代理模式设置）路由到外部世界，它们10.10.1.0/24在桥接模式下共享同一个用户定义的 docker 网络
容器B不暴露端口，只有容器A暴露。
在容器B上：
# ip route show
default via 10.10.1.1 dev eth0
10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.2
# ip route replace default via 10.10.1.3
# ip route show
default via 10.10.1.3 dev eth0
10.10.1.0/24 dev eth0 proto kernel scope link src 10.10.1.2
在容器A（haproxy）上：
sysctl -w net.ipv4.ip_forward=1
sysctl -w net.ipv4.ip_nonlocal_bind=1
iptables -t mangle -N DIVERT
iptables -t mangle -A PREROUTING -p tcp -m socket -j DIVERT
iptables -t mangle -A DIVERT -j MARK --set-mark 1
iptables -t mangle -A DIVERT -j ACCEPT
ip rule add fwmark 1 lookup 100
ip route add local 0.0.0.0/0 dev lo table 100
net.ipv4.ip_forward所有地方都设置为 1，Docker 使用默认选项运行。
但是当我改变路线时；因为它必须回复容器 A；容器 B 无法到达任何地方，只是出现了简单的ping google.com超时。
我不是专家，但根据我在 3 个部分上运行的 tcpdump 命令，它们之间存在交换。也许有些数据包没有通过？
我没有遗漏什么吗？它应该非常简单，但我无法工作，我不明白为什么。
谢谢
编辑：
尝试 ping/curl google.com 时，haproxy 容器上的 tcpdump
15:12:46.576882 IP (tos 0x0, ttl 64, id 24297, offset 0, flags [DF], proto UDP (17), length 56)
myproject_containerb_1.myproject_mynetwork.52246 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0x68e7!] 46224+ A? google.com. (28)
15:12:46.576945 IP (tos 0x0, ttl 63, id 24297, offset 0, flags [DF], proto UDP (17), length 56)
myproject_containerb_1.myproject_mynetwork.52246 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0x68e7!] 46224+ A? google.com. (28)
15:12:46.578608 IP (tos 0x0, ttl 64, id 1536, offset 0, flags [DF], proto UDP (17), length 72)
28768a341dc6.35740 > cdns.ovh.net.53: [bad udp cksum 0x0270 -> 0xa31b!] 56294+ PTR? 99.33.186.213.in-addr.arpa. (44)
15:12:46.578846 IP (tos 0x0, ttl 64, id 24298, offset 0, flags [DF], proto UDP (17), length 56)
myproject_containerb_1.myproject_mynetwork.54291 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0xa70e!] 28241+ AAAA? google.com. (28)
15:12:46.578854 IP (tos 0x0, ttl 63, id 24298, offset 0, flags [DF], proto UDP (17), length 56)
myproject_containerb_1.myproject_mynetwork.54291 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0xa70e!] 28241+ AAAA? google.com. (28)
15:12:46.580544 IP (tos 0x0, ttl 56, id 23868, offset 0, flags [none], proto UDP (17), length 98)
cdns.ovh.net.53 > 28768a341dc6.35740: [udp sum ok] 56294 q: PTR? 99.33.186.213.in-addr.arpa. 1/0/0 99.33.186.213.in-addr.arpa. PTR cdns.ovh.net. (70)
15:12:48.559050 IP (tos 0x0, ttl 64, id 19298, offset 0, flags [DF], proto TCP (6), length 60)
15:12:48.797205 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 28768a341dc6 tell myproject_containerb_1.myproject_mynetwork, length 28
15:12:48.797226 ARP, Ethernet (len 6), IPv4 (len 4), Reply 28768a341dc6 is-at 02:42:0a:0a:01:03 (oui Unknown), length 28
15:12:51.580010 IP (tos 0x0, ttl 64, id 24376, offset 0, flags [DF], proto UDP (17), length 56)
myproject_containerb_1.myproject_mynetwork.35939 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0xeebe!] 28241+ AAAA? google.com. (28)
15:12:51.580178 IP (tos 0x0, ttl 63, id 24376, offset 0, flags [DF], proto UDP (17), length 56)
myproject_containerb_1.myproject_mynetwork.35939 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0xeebe!] 28241+ AAAA? google.com. (28)
15:12:51.580799 IP (tos 0x0, ttl 64, id 24377, offset 0, flags [DF], proto UDP (17), length 56)
myproject_containerb_1.myproject_mynetwork.51485 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0x6be0!] 46224+ A? google.com. (28)
15:12:51.580820 IP (tos 0x0, ttl 63, id 24377, offset 0, flags [DF], proto UDP (17), length 56)
myproject_containerb_1.myproject_mynetwork.51485 > cdns.ovh.net.53: [bad udp cksum 0x025f -> 0x6be0!] 46224+ A? google.com. (28)
15:12:51.613200 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 10.10.1.1 tell 28768a341dc6, length 28
15:12:51.613286 ARP, Ethernet (len 6), IPv4 (len 4), Request who-has 28768a341dc6 tell 10.10.1.1, length 28
15:12:51.613300 ARP, Ethernet (len 6), IPv4 (len 4), Reply 28768a341dc6 is-at 02:42:0a:0a:01:03 (oui Unknown), length 28
15:12:51.613319 ARP, Ethernet (len 6), IPv4 (len 4), Reply 10.10.1.1 is-at 02:42:38:6d:07:86 (oui Unknown), length 28
15:12:51.613928 IP (tos 0x0, ttl 64, id 2532, offset 0, flags [DF], proto UDP (17), length 68)
28768a341dc6.58522 > cdns.ovh.net.53: [bad udp cksum 0x026c -> 0x5a50!] 17457+ PTR? 1.1.10.10.in-addr.arpa. (40)
15:12:51.615887 IP (tos 0x0, ttl 56, id 844, offset 0, flags [none], proto UDP (17), length 127)
cdns.ovh.net.53 > 28768a341dc6.58522: [udp sum ok] 17457 NXDomain* q: PTR? 1.1.10.10.in-addr.arpa. 0/1/0 ns: 10.in-addr.arpa. SOA localhost. nobody.invalid. 1 3600 1200 604800 10800 (99)
15:12:56.584820 IP (tos 0x0, ttl 64, id 24781, offset 0, flags [DF], proto UDP (17), length 71)
myproject_containerb_1.myproject_mynetwork.60514 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0xdfaf!] 13663+ A? google.com.openstacklocal. (43)
15:12:56.584847 IP (tos 0x0, ttl 63, id 24781, offset 0, flags [DF], proto UDP (17), length 71)
myproject_containerb_1.myproject_mynetwork.60514 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0xdfaf!] 13663+ A? google.com.openstacklocal. (43)
15:12:56.587511 IP (tos 0x0, ttl 64, id 24782, offset 0, flags [DF], proto UDP (17), length 71)
myproject_containerb_1.myproject_mynetwork.36461 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x2d86!] 10878+ AAAA? google.com.openstacklocal. (43)
15:12:56.587522 IP (tos 0x0, ttl 63, id 24782, offset 0, flags [DF], proto UDP (17), length 71)
myproject_containerb_1.myproject_mynetwork.36461 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x2d86!] 10878+ AAAA? google.com.openstacklocal. (43)
15:12:58.560867 IP (tos 0x0, ttl 64, id 12123, offset 0, flags [DF], proto TCP (6), length 60)
15:12:59.549284 IP (tos 0xc0, ttl 64, id 40262, offset 0, flags [none], proto ICMP (1), length 94)
28768a341dc6 > myproject_containerb_1.myproject_mynetwork: ICMP redirect ns3012985.ip-149-202-74.eu to host 10.10.1.1, length 74
IP (tos 0x0, ttl 63, id 61603, offset 0, flags [DF], proto TCP (6), length 66)
15:12:59.551076 IP (tos 0x0, ttl 64, id 3564, offset 0, flags [DF], proto UDP (17), length 73)
28768a341dc6.34089 > cdns.ovh.net.53: [bad udp cksum 0x0271 -> 0xe0da!] 3332+ PTR? 155.74.202.149.in-addr.arpa. (45)
15:12:59.637667 IP (tos 0x0, ttl 56, id 1829, offset 0, flags [none], proto UDP (17), length 113)
cdns.ovh.net.53 > 28768a341dc6.34089: [udp sum ok] 3332 q: PTR? 155.74.202.149.in-addr.arpa. 1/0/0 155.74.202.149.in-addr.arpa. PTR ns3012985.ip-149-202-74.eu. (85)
15:13:01.591313 IP (tos 0x0, ttl 64, id 24864, offset 0, flags [DF], proto UDP (17), length 71)
myproject_containerb_1.myproject_mynetwork.51194 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x0418!] 13663+ A? google.com.openstacklocal. (43)
15:13:01.591352 IP (tos 0x0, ttl 63, id 24864, offset 0, flags [DF], proto UDP (17), length 71)
myproject_containerb_1.myproject_mynetwork.51194 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x0418!] 13663+ A? google.com.openstacklocal. (43)
15:13:01.592629 IP (tos 0x0, ttl 64, id 24865, offset 0, flags [DF], proto UDP (17), length 71)
myproject_containerb_1.myproject_mynetwork.42581 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x159e!] 10878+ AAAA? google.com.openstacklocal. (43)
15:13:01.592638 IP (tos 0x0, ttl 63, id 24865, offset 0, flags [DF], proto UDP (17), length 71)
myproject_containerb_1.myproject_mynetwork.42581 > cdns.ovh.net.53: [bad udp cksum 0x026e -> 0x159e!] 10878+ AAAA? google.com.openstacklocal. (43)
15:13:03.555973 IP (tos 0x0, ttl 64, id 21583, offset 0, flags [DF], proto TCP (6), length 60)
28768a341dc6.34260 > myproject_containerb_1.myproject_mynetwork.8081: Flags [S], cksum 0x1647 (incorrect -> 0xeed8), seq 2021736358, win 29200, options [mss 1460,sackOK,TS val 126381033 ecr 0,nop,wscale 7], length 0
编辑2：
ping 8.8.8.8 时，haproxy 容器的另一个日志
17:07:20.714651 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 1, length 64
17:07:20.714683 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 1, length 64
17:07:20.715559 IP 6eb4896e0f04.53502 > cdns.ovh.net.53: 49544+ PTR? 8.8.8.8.in-addr.arpa. (38)
17:07:20.717433 IP cdns.ovh.net.53 > 6eb4896e0f04.53502: 49544 1/0/0 PTR google-public-dns-a.google.com. (82)
17:07:20.718841 IP 6eb4896e0f04.46346 > cdns.ovh.net.53: 61016+ PTR? 99.33.186.213.in-addr.arpa. (44)
17:07:20.720686 IP cdns.ovh.net.53 > 6eb4896e0f04.46346: 61016 1/0/0 PTR cdns.ovh.net. (70)
17:07:21.725413 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 2, length 64
17:07:21.725573 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:21.725579 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 2, length 64
17:07:21.727624 IP 6eb4896e0f04.45201 > cdns.ovh.net.53: 21323+ PTR? 1.1.10.10.in-addr.arpa. (40)
17:07:21.729570 IP cdns.ovh.net.53 > 6eb4896e0f04.45201: 21323 NXDomain* 0/1/0 (99)
17:07:22.726499 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 3, length 64
17:07:22.726527 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:22.726530 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 3, length 64
17:07:23.741248 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 4, length 64
17:07:23.741277 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:23.741281 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 4, length 64
17:07:24.765267 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 5, length 64
17:07:24.765314 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:24.765319 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 5, length 64
17:07:25.725183 ARP, Request who-has 6eb4896e0f04 tell 10.10.1.1, length 28
17:07:25.725213 ARP, Reply 6eb4896e0f04 is-at 02:42:0a:0a:01:03 (oui Unknown), length 28
17:07:25.766514 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 6, length 64
17:07:25.766543 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:25.766546 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 6, length 64
17:07:26.767908 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 7, length 64
17:07:26.767930 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 7, length 64
17:07:27.773201 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 8, length 64
17:07:27.773226 IP 6eb4896e0f04 > myproject_containerb_1.myproject_mynetwork: ICMP redirect google-public-dns-a.google.com to host 10.10.1.1, length 92
17:07:27.773230 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 8, length 64
17:07:28.774537 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 9, length 64
17:07:28.774555 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 9, length 64
17:07:29.789239 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 10, length 64
17:07:29.789260 IP myproject_containerb_1.myproject_mynetwork > google-public-dns-a.google.com: ICMP echo request, id 32, seq 10, length 64
编辑3：
# Generated by iptables-save v1.6.0 on Mon Jan 28 21:38:44 2019
*mangle
:PREROUTING ACCEPT [1095680:979414720]
:INPUT ACCEPT [1087344:977384769]
:FORWARD ACCEPT [8231:2022517]
:OUTPUT ACCEPT [733683:74635430]
:POSTROUTING ACCEPT [741914:76657947]
COMMIT
# Completed on Mon Jan 28 21:38:44 2019
# Generated by iptables-save v1.6.0 on Mon Jan 28 21:38:44 2019
*nat
:PREROUTING ACCEPT [2457:111645]
:INPUT ACCEPT [1538:56511]
:OUTPUT ACCEPT [15:1088]
:POSTROUTING ACCEPT [917:55188]
:DOCKER - [0:0]
-A PREROUTING -m addrtype --dst-type LOCAL -j DOCKER
-A OUTPUT ! -d 127.0.0.0/8 -m addrtype --dst-type LOCAL -j DOCKER
-A POSTROUTING -s 10.10.1.0/24 ! -o br-a52f541ced84 -j MASQUERADE
-A POSTROUTING -s 172.17.0.0/16 ! -o docker0 -j MASQUERADE
-A POSTROUTING -s 10.10.1.3/32 -d 10.10.1.3/32 -p tcp -m tcp --dport 4242 -j MASQUERADE
-A DOCKER -i br-a52f541ced84 -j RETURN
-A DOCKER -i docker0 -j RETURN
-A DOCKER ! -i br-a52f541ced84 -p tcp -m tcp --dport 8081 -j DNAT --to-destination 10.10.1.3:4242
COMMIT
# Completed on Mon Jan 28 21:38:44 2019
# Generated by iptables-save v1.6.0 on Mon Jan 28 21:38:44 2019
*filter
:INPUT ACCEPT [11:784]
:FORWARD DROP [0:0]
:OUTPUT ACCEPT [15:2336]
:DOCKER - [0:0]
:DOCKER-ISOLATION-STAGE-1 - [0:0]
:DOCKER-ISOLATION-STAGE-2 - [0:0]
:DOCKER-USER - [0:0]
-A FORWARD -j DOCKER-USER
-A FORWARD -j DOCKER-ISOLATION-STAGE-1
-A FORWARD -o br-a52f541ced84 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o br-a52f541ced84 -j DOCKER
-A FORWARD -i br-a52f541ced84 ! -o br-a52f541ced84 -j ACCEPT
-A FORWARD -i br-a52f541ced84 -o br-a52f541ced84 -j ACCEPT
-A FORWARD -o docker0 -m conntrack --ctstate RELATED,ESTABLISHED -j ACCEPT
-A FORWARD -o docker0 -j DOCKER
-A FORWARD -i docker0 ! -o docker0 -j ACCEPT
-A FORWARD -i docker0 -o docker0 -j ACCEPT
-A DOCKER -d 10.10.1.3/32 ! -i br-a52f541ced84 -o br-a52f541ced84 -p tcp -m tcp --dport 4242 -j ACCEPT
-A DOCKER-ISOLATION-STAGE-1 -i br-a52f541ced84 ! -o br-a52f541ced84 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -i docker0 ! -o docker0 -j DOCKER-ISOLATION-STAGE-2
-A DOCKER-ISOLATION-STAGE-1 -j RETURN
-A DOCKER-ISOLATION-STAGE-2 -o br-a52f541ced84 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -o docker0 -j DROP
-A DOCKER-ISOLATION-STAGE-2 -j RETURN
-A DOCKER-USER -j RETURN
COMMIT
# Completed on Mon Jan 28 21:38:44 2019
答案1
在最新版本的 docker-compose 中，可以使用
network_mode: "container:name"
version: '3'
services:
nginx_proxy:
image: nginx
ports:
- "8080:80"
webapp:
image: your-webapp-image
network_mode: "container:nginx_proxy"
无需摆弄静态路由和 iptables
答案2
尝试在容器 A 上使用 MASQUERADE 而不是 mangle：
#!/bin/bash echo 1 > /proc/sys/net/ipv4/ip_forward IPT=/sbin/iptables PUB_IF=enp3s0 #(or whichever is your container A interface) $IPT -F $IPT -t nat -F $IPT -t nat -A POSTROUTING -o $PUB_IF -j MASQUERADE $IPT -t nat -A POSTROUTING -j MASQUERADE $IPT -A INPUT -m state --state ESTABLISHED,RELATED -j ACCEPT $IPT -A INPUT -m state ! --state NEW -i $PUB_IF -j ACCEPT $IPT -A FORWARD -m state --state ESTABLISHED,RELATED -j ACCEPT $IPT -A FORWARD -m state ! --state NEW -i $PUB_IF -j ACCEPT $IPT -A FORWARD -i $PUB_IF -j ACCEPT
还要先检查 ping 8.8.8.8（或任何你拥有的 DNS），并尝试使用 ip route get 8.8.8.8 获取 ip 的路径
答案3
您的容器需要特定的功能才能运行这些命令。请在您的 docker run 命令中添加此参数：
–cap-add = NET_ADMIN