我的拓扑包括 HostA 运行 OpenVPN 客户端连接到 HostB 上的服务器。HostB 有 OpenVPN 客户端连接到 HostC 上的服务器。两个隧道都已打开,我可以通过每个隧道发送 curl 请求,但我无法将流量从 HostA 路由到 HostB,再路由到 HostC。例如:
Public Private Client Tunnel Server Tunnel
HostA 1.1.1.1 10.120.177.168 10.8.0.6 10.8.0.1
HostB 2.2.2.2 172.30.24.54
HostC 3.3.3.3 10.140.17.141 10.9.0.6 10.9.0.1
HostA $ ip address
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc prio state UP group default qlen 1000
inet 10.120.177.168/26 brd 10.120.177.191 scope global eth0
20: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast state UNKNOWN group default qlen 100
inet 10.8.0.6 peer 255.255.255.0/32 scope global tun0
HostB $ ip address
4: eth0@if14: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1480 qdisc noqueue state UP group default
inet 172.30.24.54/32 scope global eth0
18: tun010140017141: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
inet 10.9.0.6 peer 10.9.0.5/32 scope global tun010140017141
20: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UNKNOWN group default qlen 100
inet 10.8.0.1 peer 10.8.0.2/32 scope global tun0
HostC $ ip address
2: eth0: <BROADCAST,MULTICAST,PROMISC,UP,LOWER_UP> mtu 1500 qdisc prio state UP group default qlen 1000
inet 10.140.17.141/26 brd 10.140.17.191 scope global eth0
227: tun0: <POINTOPOINT,MULTICAST,NOARP,UP,LOWER_UP> mtu 32000 qdisc pfifo_fast state UNKNOWN group default qlen 100
inet 10.9.0.1 peer 10.9.0.2/32 scope global tun0
HostA $ cat /etc/openvpn/client.conf
client
remote 2.2.2.2 443
proto tcp
dev tun
nobind
HostB $ cat /etc/openvpn/server/server.conf
port 443
proto tcp4
dev tun
server 10.8.0.0 255.255.255.0
push "route 10.140.17.141 255.255.255.255 10.8.0.6"
client-config-dir ccd
route 10.120.177.168 255.255.255.255
HostB $ cat /etc/openvpn/server/ccd/gateway_10_120_177_168
ifconfig-push 10.8.0.6 255.255.255.0
iroute 10.120.177.168 255.255.255.255
HostB $ cat /etc/openvpn/client/tun010140017141/client.ovpn
client
remote 10.140.17.141 1194
proto tcp
dev tun010140017141
nobind
HostC $ cat /etc/openvpn/server.conf
port 1194
proto tcp
dev tun
server 10.9.0.0 255.255.255.0
- 由于这样可行,我省略了与证书和日志相关的配置部分。
内核路由:
HostA $ ip route
default via 10.120.177.129 dev eth0
10.0.0.0/8 via 10.120.177.129 dev eth0
10.8.0.1 via 255.255.255.0 dev tun0
10.120.177.128/26 dev eth0 proto kernel scope link src 10.120.177.168
10.140.17.141 via 10.8.0.6 dev tun0
255.255.255.0 dev tun0 proto kernel scope link src 10.8.0.6
HostB $ ip rule list table my_routing_table
32764: from 10.140.17.141 lookup my_routing_table
32765: from 10.120.177.168 lookup my_routing_table
HostB $ ip route list table my_routing_table
10.120.177.168 dev tun0 scope link
10.140.17.141 dev tun010140017141 scope link
HostB $ ip route
default via 169.254.1.1 dev eth0
10.8.0.0/24 via 10.8.0.2 dev tun0
10.8.0.2 dev tun0 proto kernel scope link src 10.8.0.1
10.9.0.1 via 10.9.0.5 dev tun010140017141
10.9.0.5 dev tun010140017141 proto kernel scope link src 10.9.0.6
10.120.177.168 via 10.8.0.2 dev tun0
169.254.1.1 dev eth0 scope link
HostC $ ip route
default via 10.140.17.129 dev eth0
10.0.0.0/8 via 10.140.17.129 dev eth0
10.9.0.0/24 via 10.9.0.2 dev tun0
10.9.0.2 dev tun0 proto kernel scope link src 10.9.0.1
10.120.177.168 via 10.140.17.129 dev eth0
10.140.17.128/26 dev eth0 proto kernel scope link src 10.140.17.141
192.168.0.0/16 via 10.140.17.129 dev eth0
我想要实现:
HostA $ curl -v https://10.140.17.141
但回应是
* NSS error -5961 (PR_CONNECT_RESET_ERROR)
流量确实来自 HostA,通过隧道到达 HostB
HostB $ tcpdump --interface=tun0 -n host 10.140.17.141
12:55:07.039521 IP 10.8.0.6.44150 > 10.140.17.141.https: Flags [S], seq 2877997232, win 42340, options [mss 1358,sackOK,TS val 3699256805 ecr 0,nop,wscale 12], length 0
12:55:07.039629 IP 10.140.17.141.https > 10.8.0.6.44150: Flags [S.], seq 1620656485, ack 2877997233, win 65535, options [mss 1460,sackOK,TS val 19053367 ecr 3699256805,nop,wscale 9], length 0
12:55:07.046763 IP 10.8.0.6.44150 > 10.140.17.141.https: Flags [.], ack 1, win 11, options [nop,nop,TS val 3699256831 ecr 19053367], length 0
12:55:07.205307 IP 10.8.0.6.44150 > 10.140.17.141.https: Flags [P.], seq 1:178, ack 1, win 11, options [nop,nop,TS val 3699256988 ecr 19053367], length 177
12:55:07.205377 IP 10.140.17.141.https > 10.8.0.6.44150: Flags [.], ack 178, win 131, options [nop,nop,TS val 19053533 ecr 3699256988], length 0
12:55:07.206243 IP 10.140.17.141.https > 10.8.0.6.44150: Flags [R.], seq 1, ack 178, win 131, options [nop,nop,TS val 19053534 ecr 3699256988], length 0
12:55:07.392219 IP 10.8.0.6.44154 > 10.140.17.141.https: Flags [S], seq 3637655375, win 42340, options [mss 1358,sackOK,TS val 3699257178 ecr 0,nop,wscale 12], length 0
12:55:07.392277 IP 10.140.17.141.https > 10.8.0.6.44154: Flags [S.], seq 1233055697, ack 3637655376, win 65535, options [mss 1460,sackOK,TS val 19053720 ecr 3699257178,nop,wscale 9], length 0
12:55:07.412840 IP 10.8.0.6.44154 > 10.140.17.141.https: Flags [.], ack 1, win 11, options [nop,nop,TS val 3699257199 ecr 19053720], length 0
12:55:07.419864 IP 10.8.0.6.44154 > 10.140.17.141.https: Flags [P.], seq 1:319, ack 1, win 11, options [nop,nop,TS val 3699257206 ecr 19053720], length 318
12:55:07.419895 IP 10.140.17.141.https > 10.8.0.6.44154: Flags [.], ack 319, win 131, options [nop,nop,TS val 19053747 ecr 3699257206], length 0
12:55:07.420618 IP 10.140.17.141.https > 10.8.0.6.44154: Flags [R.], seq 1, ack 319, win 131, options [nop,nop,TS val 19053748 ecr 3699257206], length 0
但无法到达另一个接口
HostB $ tcpdump --interface=tun010140017141 -n host 10.140.17.141
nothing
我很困惑为什么 tcpdump 表明 10.140.17.141 正在响应。
我能够使用隧道从 HostB 与 HostC 通信
HostB $ curl -v https://10.9.0.1
< HTTP/1.1 200 OK
它响应我期望的页面
curl -v https://10.140.17.141
为了证明它正在使用隧道
HostB $ tcpdump --interface=tun010140017141 -n
13:09:45.950403 IP 10.9.0.6.45884 > 10.9.0.1.https: Flags [S], seq 226912329, win 65535, options [mss 1460,sackOK,TS val 879082581 ecr 0,nop,wscale 9], length 0
13:09:45.995483 IP 10.9.0.1.https > 10.9.0.6.45884: Flags [S.], seq 845398301, ack 226912330, win 31948, options [mss 1358,sackOK,TS val 3700256992 ecr 879082581,nop,wscale 12], length 0
13:09:45.995550 IP 10.9.0.6.45884 > 10.9.0.1.https: Flags [.], ack 1, win 128, options [nop,nop,TS val 879082626 ecr 3700256992], length 0
13:09:45.995727 IP 10.9.0.6.45884 > 10.9.0.1.https: Flags [P.], seq 1:518, ack 1, win 128, options [nop,nop,TS val 879082626 ecr 3700256992], length 517
13:09:46.043310 IP 10.9.0.1.https > 10.9.0.6.45884: Flags [.], ack 518, win 9, options [nop,nop,TS val 3700257036 ecr 879082626], length 0
13:09:46.043468 IP 10.9.0.1.https > 10.9.0.6.45884: Flags [.], seq 1:1347, ack 518, win 9, options [nop,nop,TS val 3700257038 ecr 879082626], length 1346
...
由于 HostB 有路由
10.140.17.141 dev tun010140017141 scope link
为什么请求未被转发?
答案1
解决方案涉及在客户端配置目录文件中使用 nftables 和 openvpn iroute 命令进行网络地址转换和伪装。以下是更改:
HostB $ nft list ruleset
table inet filter {
set cluster-ips {
type ipv4_addr
elements = { 10.140.17.141 }
}
set tunnel-nets {
type ipv4_addr
elements = { 10.120.177.168 }
}
set ports-allowed {
type inet_service
elements = { 443, /* other ports needed by applications */ }
}
chain INPUT {
type filter hook input priority filter; policy accept;
ip saddr @tunnel-nets tcp dport @ports-allowed ip daddr @cluster-ips accept
ip saddr @tunnel-nets udp dport @ports-allowed ip daddr @cluster-ips accept
}
chain OUTPUT {
type filter hook output priority filter; policy accept;
ip saddr @cluster-ips ip daddr @tunnel-nets accept
}
chain FORWARD {
type filter hook forward priority filter; policy accept;
iifname "tun0" oifname "tun010*" ct state established,related counter
iifname "tun010*" oifname "tun0" ct state established,related counter
}
}
table ip nat {
set downstream_ports_allowed {
type inet_service
elements = { 443, /* other ports needed by applications */ }
}
set upstream_ports_allowed {
type inet_service
elements = { 22, /* other ports needed by applications */ }
}
chain prerouting {
type nat hook prerouting priority dstnat; policy accept;
ip daddr 10.140.17.141 iifname "tun0" tcp dport @downstream_ports_allowed counter dnat to 10.141.67.5
ip daddr 10.120.177.168 iifname "tun010*" tcp dport @upstream_ports_allowed counter dnat to 10.120.177.168
}
chain postrouting {
type nat hook postrouting priority srcnat; policy accept;
oifname "tun0" masquerade
oifname "tun010*" masquerade
}
}
HostB $ cat /etc/openvpn/server/server.conf
client-config-dir ccd
topology subnet
push "route 10.140.17.141 255.255.255.255"
route 10.120.177.168 255.255.255.255
HostB $ cat /etc/openvpn/server/ccd/gateway_10_120_177_168
iroute 10.120.177.168 255.255.255.255
HostC $ cat /etc/openvpn/server.conf
client-config-dir ccd
topology subnet
route 10.120.177.168
HostC $ cat /etc/openvpn/ccd/tun010140017141
iroute 10.120.177.168