我正在设置以下设置:
- 我使用 vyos 作为路由器/防火墙
- 在 vyos 上,我还以 DR 模式设置了 IPVS。这意味着 IPVS 将请求数据包 MAC 地址更改为 realserver MAC 地址,并将带有原始 SRC/DST IP 的数据包转发到 realserver。
- realserver 接收数据包并直接回复客户端 IP 地址。realserver 在环回上也有 VIP 地址(arp 数据包被禁用)。realserver 配置应该没问题,因为回复将发送到 vyos。
- 由于 vyos 位于 realservers 和客户端之间的路径上并且还充当路由器,因此回复数据包应该通过 vyos 路由,但不应再由 IPVS 处理,而只是通过 vyos 路由。
我知道这不是最标准的解决方案,在大多数情况下,IPVS 与路由器/防火墙分离到不同的主机,并且 IPVS 主机在 DR 模式下根本看不到重放数据包。但就我而言,我尝试将 IPVS 功能集成到 vyos 路由器/防火墙中。
问题是,在 vyos 中,我看到 vyos realservers 接口中有 syn-ack 回复数据包,但它们并没有转发到客户端接口。
Vyos 接口:
vyos@vyos-1:~$ show interfaces
Codes: S - State, L - Link, u - Up, D - Down, A - Admin Down
Interface IP Address S/L Description
--------- ---------- --- -----------
dum3 10.10.100.3/32 u/u IPVS-VIP
eth0 10.10.10.1/24 u/u IPVS-realservers
eth2 10.10.5.1/24 u/u client
lo 127.0.0.1/8 u/u
::1/128
vyos 上的 IPVS:
root@vyos-1:/home/vyos# ipvsadm -Ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
-> RemoteAddress:Port Forward Weight ActiveConn InActConn
TCP 10.10.100.3:80 sh
-> 10.10.10.10:80 Route 1 0 0
-> 10.10.10.20:80 Route 1 0 0
RP 过滤器已禁用
root@vyos-1:/home/vyos# sysctl -a | grep \\.rp_filter
net.ipv4.conf.all.rp_filter = 0
net.ipv4.conf.default.rp_filter = 0
net.ipv4.conf.dum3.rp_filter = 0
net.ipv4.conf.eth0.rp_filter = 0
net.ipv4.conf.eth1.rp_filter = 0
net.ipv4.conf.eth2.rp_filter = 0
net.ipv4.conf.lo.rp_filter = 0
net.ipv4.conf.pim6reg.rp_filter = 0
使用 curl 从客户端(10.10.5.5)进行测试:
curl http://10.10.100.3
vyos 中的 tcpdump
root@vyos-1:/home/vyos# tcpdump -nnn -i any port 80
tcpdump: data link type LINUX_SLL2
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on any, link-type LINUX_SLL2 (Linux cooked v2), snapshot length 262144 bytes
//Request packet from client to IPVS VIP on eth2
19:08:13.582552 eth2 In IP 10.10.5.5.34200 > 10.10.100.3.80: Flags [S], seq 4055297170, win 64240, options [mss 1460,sackOK,TS val 2156865110 ecr 0,nop,wscale 7], length 0
//Request packet from IPVS to realserver on eth0
19:08:13.582708 eth0 Out IP 10.10.5.5.34200 > 10.10.100.3.80: Flags [S], seq 4055297170, win 64240, options [mss 1460,sackOK,TS val 2156865110 ecr 0,nop,wscale 7], length 0
//Reply packet from realserver to client on eth0
19:08:13.583239 eth0 In IP 10.10.100.3.80 > 10.10.5.5.34200: Flags [S.], seq 2282149395, ack 4055297171, win 65160, options [mss 1460,sackOK,TS val 1876404975 ecr 2156865110,nop,wscale 7], length 0
//Previous reply packet is not forwarded to client interface eth2
//Retry packets from client because client did not get reply:
19:08:14.582700 eth2 In IP 10.10.5.5.34200 > 10.10.100.3.80: Flags [S], seq 4055297170, win 64240, options [mss 1460,sackOK,TS val 2156866111 ecr 0,nop,wscale 7], length 0
19:08:14.582851 eth0 Out IP 10.10.5.5.34200 > 10.10.100.3.80: Flags [S], seq 4055297170, win 64240, options [mss 1460,sackOK,TS val 2156866111 ecr 0,nop,wscale 7], length 0
19:08:14.583211 eth0 In IP 10.10.100.3.80 > 10.10.5.5.34200: Flags [S.], seq 2282149395, ack 4055297171, win 65160, options [mss 1460,sackOK,TS val 1876405975 ecr 2156865110,nop,wscale 7], length 0
19:08:15.604264 eth0 In IP 10.10.100.3.80 > 10.10.5.5.34200: Flags [S.], seq 2282149395, ack 4055297171, win 65160, options [mss 1460,sackOK,TS val 1876406996 ecr 2156865110,nop,wscale 7], length 0
19:08:16.598753 eth2 In IP 10.10.5.5.34200 > 10.10.100.3.80: Flags [S], seq 4055297170, win 64240, options [mss 1460,sackOK,TS val 2156868127 ecr 0,nop,wscale 7], length 0
19:08:16.598902 eth0 Out IP 10.10.5.5.34200 > 10.10.100.3.80: Flags [S], seq 4055297170, win 64240, options [mss 1460,sackOK,TS val 2156868127 ecr 0,nop,wscale 7], length 0
19:08:16.599546 eth0 In IP 10.10.100.3.80 > 10.10.5.5.34200: Flags [S.], seq 2282149395, ack 4055297171, win 65160, options [mss 1460,sackOK,TS val 1876407991 ecr 2156865110,nop,wscale 7], length 0
19:08:18.612826 eth0 In IP 10.10.100.3.80 > 10.10.5.5.34200: Flags [S.], seq 2282149395, ack 4055297171, win 65160, options [mss 1460,sackOK,TS val 1876410004 ecr 2156865110,nop,wscale 7], length 0
从防火墙的角度来看,没有拒绝规则。重放数据包在 PREROUTING 链中可见/记录,但在 FORWARD 链中不可见。我怀疑问题可能在于重放数据包源 IP(10.10.100.3)也在 vyos 虚拟接口中,这就是为什么 vyos 认为这应该是本地路由数据包或其他东西……并且没有转发到 FORWARDING 链。
有什么建议可以解释为什么回复数据包未在 vyos 中转发以及如何使其工作?