Keepalived 不路由某些 TCP 数据包

Keepalived 不路由某些 TCP 数据包

我正在尝试为我们的传出 squid 代理实现基于 NAT 的 keepalived 负载均衡器。我已设置了 2 个共享 VIP 的 keepalived VM:

[root@keepalived-01 ~]# ip a
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 16436 qdisc noqueue state UNKNOWN 
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
    inet 127.0.0.1/8 scope host lo
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc pfifo_fast 
state UP qlen 1000
    link/ether 52:54:b3:65:10:17 brd ff:ff:ff:ff:ff:ff
    inet 10.65.10.17/24 brd 10.65.10.255 scope global eth0
    inet 10.65.10.101/32 scope global eth0
[root@keepalived-01 ~]#

然后将流量转发到 squid 代理:

[root@keepalived-01 ~]# ipvsadm -ln
IP Virtual Server version 1.2.1 (size=4096)
Prot LocalAddress:Port Scheduler Flags
  -> RemoteAddress:Port           Forward Weight ActiveConn InActConn
TCP  10.65.10.101:3128 rr
  -> 10.65.10.12:3128             Masq    1      0          0         
  -> 10.65.10.13:3128             Masq    1      0          0         
[root@keepalived-01 ~]#

我的 iptables 规则是:

[root@keepalived-01 ~]# iptables -S
-P INPUT ACCEPT
-P FORWARD ACCEPT
-P OUTPUT ACCEPT
-A INPUT -m state --state RELATED,ESTABLISHED -j ACCEPT 
-A INPUT -i lo -j ACCEPT 
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG NONE -j DROP 
-A INPUT -p tcp -m tcp ! --tcp-flags FIN,SYN,RST,ACK SYN -m state --state NEW -j DROP 
-A INPUT -p tcp -m tcp --tcp-flags FIN,SYN,RST,PSH,ACK,URG FIN,SYN,RST,PSH,ACK,URG -j DROP 
-A INPUT -p icmp -m icmp --icmp-type 8 -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 22 -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 5666 -j ACCEPT 
-A INPUT -s 10.2.100.5/32 -j ACCEPT 
-A INPUT -d 224.0.0.0/8 -i eth0 -p vrrp -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 9100 -j ACCEPT 
-A INPUT -p tcp -m tcp --dport 3128 -j ACCEPT 
-A OUTPUT -d 224.0.0.0/8 -o eth0 -p vrrp -j ACCEPT 
[root@keepalived-01 ~]#

keepalived 机器上已经启用了 ip_forwarding。

VIP 故障转移工作正常,squid 代理的健康检查也运行良好。keepalived 配置如下:

Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: ------< Global definitions >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  Router ID = keepalived-01.tekno.veritrans.co.id
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  Smtp server connection timeout = 30
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  Email notification from = [email protected]
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  VRRP IPv4 mcast group = 224.0.0.18
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  VRRP IPv6 mcast group = 224.0.0.18
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  SNMP Trap disabled
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: ------< SSL definitions >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  Using autogen SSL context
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: ------< Global definitions >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: ------< LVS Topology >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  System is compiled with LVS v1.2.1
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  VIP = 10.65.10.101, VPORT = 3128
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:  Router ID = keepalived-01.tekno.veritrans.co.id
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    delay_loop = 6, lb_algo = rr
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    protocol = TCP
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    alpha is OFF, omega is OFF
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    quorum = 1, hysteresis = 0
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    lb_kind = NAT
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    RIP = 10.65.10.13, RPORT = 3128, WEIGHT = 1
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    RIP = 10.65.10.12, RPORT = 3128, WEIGHT = 1
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: ------< Health checkers >------
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  [10.65.10.13]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    Keepalive method = TCP_CHECK
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    Connection dest = [10.65.10.12]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    Connection timeout = 5
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:  [10.65.10.12]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    Keepalive method = TCP_CHECK
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    Connection dest = [10.65.10.12]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]:    Connection timeout = 5
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Using LinkWatch kernel netlink reflector...
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Activating healthchecker for service [10.65.10.13]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:  Smtp server connection timeout = 30
Jan 15 13:15:55 keepalived-01 Keepalived_healthcheckers[27562]: Activating healthchecker for service [10.65.10.12]:3128
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:  Email notification from = [email protected]
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:  VRRP IPv4 mcast group = 224.0.0.18
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:  VRRP IPv6 mcast group = 224.0.0.18
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:  SNMP Trap disabled
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: ------< VRRP Topology >------
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:  VRRP Instance = squid
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:    Want State = MASTER
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:    Runing on device = eth0
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:    Virtual Router ID = 1
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:    Priority = 100
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:    Advert interval = 1sec
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:    Authentication type = SIMPLE_PASSWORD
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:    Password = MIdk33pA
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:    Virtual IP = 1
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]:      10.65.10.101/32 dev eth0 scope global
Jan 15 13:15:55 keepalived-01 Keepalived_vrrp[27563]: Using LinkWatch kernel netlink reflector...
Jan 15 13:15:55 keepalived-01 bash: root(ishan): /usr/sbin/keepalived -d
Jan 15 13:15:56 keepalived-01 Keepalived_vrrp[27563]: VRRP_Instance(squid) Transition to MASTER STATE

我面临的问题是,当我到达 VIP 时,SYN数据包到达 squid 代理,然后我收到数据SYN.包。但是当我发送时.,VIP 不会将其转发给 squid 代理。所有后续P.数据包也不会传输。但是当我断开连接时,F数据包到达代理。已经卡了很长时间了。任何帮助都将不胜感激。

正在尝试连接:

[root@ns1 ~]# curl -x http://10.65.10.101:3128 google.com -vvv
* About to connect() to proxy 10.65.10.101 port 3128 (#0)
*   Trying 10.65.10.101... connected
* Connected to 10.65.10.101 (10.65.10.101) port 3128 (#0)
> GET http://google.com HTTP/1.1
> User-Agent: curl/7.19.7 (x86_64-redhat-linux-gnu) libcurl/7.19.7 NSS/3.14.0.0 zlib/1.2.3 libidn/1.18 libssh2/1.4.2
> Host: google.com
> Accept: */*
> Proxy-Connection: Keep-Alive
> 
^C
[root@ns1 ~]#

发起连接自:10.65.10.14
VIP:10.65.10.101
代理:10.65.10.12

tcpdump贵宾:

13:21:15.718171 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [S], seq 2999891684, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
13:21:15.718200 IP 10.65.10.14.33299 > 10.65.10.12.squid: Flags [S], seq 2999891684, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
13:21:15.718451 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:15.718467 IP 10.65.10.101.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:15.718610 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [.], ack 1, win 229, length 0
13:21:15.718682 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:15.919022 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:16.321066 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:16.918216 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:16.918239 IP 10.65.10.101.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:16.918488 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [.], ack 1, win 229, length 0
13:21:17.124999 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:18.733058 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:18.918241 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:18.918263 IP 10.65.10.101.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:18.918491 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [.], ack 1, win 229, length 0
13:21:20.950227 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [F.], seq 212, ack 1, win 229, length 0
13:21:20.950256 IP 10.65.10.14.33299 > 10.65.10.12.squid: Flags [F.], seq 212, ack 1, win 229, length 0
13:21:20.950541 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [.], ack 1, win 115, options [nop,nop,sack 1 {212:213}], length 0
13:21:20.950562 IP 10.65.10.101.squid > 10.65.10.14.33299: Flags [.], ack 1, win 115, options [nop,nop,sack 1 {212:213}], length 0
13:21:21.949007 IP 10.65.10.14.33299 > 10.65.10.101.squid: Flags [P.], seq 1:212, ack 1, win 229, length 211
13:21:27.416158 IP 10.65.10.14.33300 > 10.65.10.101.squid: Flags [S], seq 3034291567, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
13:21:27.416186 IP 10.65.10.14.33300 > 10.65.10.13.squid: Flags [S], seq 3034291567, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0

tcpdump在 Squid 代理上:

13:21:15.718678 IP 10.65.10.14.33299 > 10.65.10.12.squid: Flags [S], seq 2999891684, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 6], length 0
13:21:15.718727 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:16.918423 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:18.918448 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [S.], seq 2385228924, ack 2999891685, win 14600, options [mss 1460,nop,nop,sackOK,nop,wscale 7], length 0
13:21:20.950749 IP 10.65.10.14.33299 > 10.65.10.12.squid: Flags [F.], seq 212, ack 1, win 229, length 0
13:21:20.950784 IP 10.65.10.12.squid > 10.65.10.14.33299: Flags [.], ack 1, win 115, options [nop,nop,sack 1 {212:213}], length 0

相关内容