我有一台安装了 Ubuntu 20.04 的 Gateworks SBC。我的 SOC 通过 USB 转以太网适配器 (eth0) 作为 DHCP 客户端 (192.168.88.102) 连接到我的 Mikrotik 路由器 (DHCP 服务器)，后者充当我的互联网网关 (Masquerade)。
我在 AWS 上安装了一个虚拟 Mikrotik 实例，IP 地址为 18.xxx。我在那里配置了一个带证书的 Ipsec Ikev2 隧道。Roadwarrior 客户端应使用虚拟 IP 192.168.11.0/24（模板）连接。我的配置与 Mikrotik Ipsec Wiki 相同。
通过 Strongswan，我将 SBC 连接到 Mikrotik AWS 路由器。我的 SOC 充当 roadwarrior 客户端（虚拟 IP 192.168.11.254）。连接已建立，SA 已分配。我可以从 SOC ping 到 8.8.8.8，数据包通过 Mikrotik IPSEC 路由器进行隧道传输并返回。到目前为止一切都很顺利。
提示：我现在断开 USB 至以太网适配器以避免任何问题。
我的 Gateworks SBC 已安装 Toby L210 4G USB 调制解调器。Ubuntu 已识别调制解调器。我按照 u-blox 的描述将调制解调器配置为在路由器模式下工作。使用 Modemmanager，我发送命令以连接到我的电信提供商。然后我发送命令“dhclient wwan0”来创建 wwan0 接口并从调制解调器的 DHCP 服务器获取 IP 地址 192.168.1.100。192.168.1.1 是调制解调器的网关地址。Ping 到 google 工作正常。我的调制解调器有一个内部 IP 地址 100.64.xx（运营商级 NAT）
现在我启动 Strongswan（ipsec 启动测试）。再次建立连接，SA 已分发。但我无法 ping 8.8.8.8 有趣的是，如果我 ping 8.8.8.8，我会在我的 Mikrotik IPSEC 服务器（数据包嗅探器）上看到传入数据包，该服务器正在转发到 8.8.8.8 并将答案发送回隧道。所以我猜我的 SBC 发出的数据包可以正常工作...
使用 4G 调制解调器进行 Ping 转储，Ping 没有得到响应
1 0.000000000 192.168.1.100 18.xxx.xxx.xxx ESP 174 ESP (SPI=0x0b2a5a40)
2 0.063911734 18.xxx.xxx.xxx 192.168.1.100 ESP 174 ESP (SPI=0xcbb1d2c0)
3 0.486664666 18.xxx.xxx.xxx 192.168.1.100 ISAKMP 202 INFORMATIONAL MID=65 Responder Request
4 0.488833047 192.168.1.100 18.xxx.xxx.xxx ISAKMP 122 INFORMATIONAL MID=65 Initiator Response
5 1.019959021 192.168.1.100 18.xxx.xxx.xxx ESP 174 ESP (SPI=0x0b2a5a40)
6 1.088668527 18.xxx.xxx.xxx 192.168.1.100 ESP 174 ESP (SPI=0xcbb1d2c0)
7 2.043874128 192.168.1.100 18.xxx.xxx.xxx ESP 174 ESP (SPI=0x0b2a5a40)
8 2.133560094 18.xxx.xxx.xxx 192.168.1.100 ESP 174 ESP (SPI=0xcbb1d2c0)
9 3.067978240 192.168.1.100 18.xxx.xxx.xxx ESP 174 ESP (SPI=0x0b2a5a40)
10 3.163589336 18.xxx.xxx.xxx 192.168.1.100 ESP 174 ESP (SPI=0xcbb1d2c0)
11 4.091881681 192.168.1.100 18.xxx.xxx.xxx ESP 174 ESP (SPI=0x0b2a5a40)
12 4.168360691 18.xxx.xxx.xxx 192.168.1.100 ESP 174 ESP (SPI=0xcbb1d2c0)
13 5.115993793 192.168.1.100 18.xxx.xxx.xxx ESP 174 ESP (SPI=0x0b2a5a40)
14 5.193667829 18.xxx.xxx.xxx 192.168.1.100 ESP 174 ESP (SPI=0xcbb1d2c0)
15 5.566650670 18.xxx.xxx.xxx 192.168.1.100 ISAKMP 186 INFORMATIONAL MID=66 Responder Request
16 5.568814384 192.168.1.100 18.xxx.xxx.xxx ISAKMP 122 INFORMATIONAL MID=66 Initiator Response
使用 eth0 进行 Ping 转储，Ping 得到答复
2 0.227289053 192.168.88.102 18.xxx.xxx.xxx ESP 174 ESP (SPI=0x05bce76a)
3 0.245391122 18.xxx.xxx.xxx 192.168.88.102 ESP 174 ESP (SPI=0xc7c756b9)
4 0.245391122 8.8.8.8 192.168.11.254 ICMP 98 Echo (ping) reply id=0x0002, seq=9/2304, ttl=111
9 1.229264659 192.168.88.102 18.xxx.xxx.xxx ESP 174 ESP (SPI=0x05bce76a)
10 1.243413640 18.xxx.xxx.xxx 192.168.88.102 ESP 174 ESP (SPI=0xc7c756b9)
11 1.243413640 8.8.8.8 192.168.11.254 ICMP 98 Echo (ping) reply id=0x0002, seq=10/2560, ttl=111
12 1.651801718 18.xxx.xxx.xxx 192.168.88.102 ISAKMP 186 INFORMATIONAL MID=04 Responder Request
13 1.654115436 192.168.88.102 18.xxx.xxx.xxx ISAKMP 122 INFORMATIONAL MID=04 Initiator Response
IPSec配置文件
# ipsec.conf - strongSwan IPsec configuration file
# basic configuration
config setup
# strictcrlpolicy=yes
# uniqueids = no
# Add connections here.
# Sample VPN connections
#conn sample-self-signed
# leftsubnet=10.1.0.0/16
# leftcert=selfCert.der
# leftsendcert=never
# right=192.168.0.2
# rightsubnet=10.2.0.0/16
# rightcert=peerCert.der
# auto=start
#conn sample-with-ca-cert
# leftsubnet=10.1.0.0/16
# leftcert=myCert.pem
# right=192.168.0.2
# rightsubnet=10.2.0.0/16
# rightid="C=CH, O=Linux strongSwan CN=peer name"
# auto=start
conn test
keyexchange=ikev2
ike=aes128-sha1-modp2048
esp=aes128-sha1-modp2048
ikelifetime = 24h
lifetime = 30m
dpddelay = 120s
left=%defaultroute
leftsourceip=%modeconfig
leftcert=cert_export_Ventana.pem
leftid=VentanaTest
leftfirewall=yes
right=18.xxx.xxx.xxx
rightsubnet=0.0.0.0/0
rightid="18.xxx.xxx.xxx"
auto=add
ipsec 状态全部
Status of IKE charon daemon (strongSwan 5.8.2, Linux 5.4.45-g334256aa8a89, armv7l):
uptime: 6 minutes, since Dec 12 20:06:32 2020
malloc: sbrk 1347584, mmap 0, used 457608, free 889976
worker threads: 11 of 16 idle, 5/0/0/0 working, job queue: 0/0/0/0, scheduled: 3
loaded plugins: charon aes rc2 sha2 sha1 md5 mgf1 random nonce x509 revocation constraints pubkey pkcs1 pkcs7 pkcs8 pkcs12 pgp dnskey sshkey pem openssl fips-prf gmp agent xcbc hmac gcm drbg attr kernel-netlink resolve socket-default connmark farp stroke updown eap-identity eap-aka eap-md5 eap-gtc eap-mschapv2 eap-dynamic eap-radius eap-tls eap-ttls eap-peap eap-tnc xauth-generic xauth-eap xauth-pam tnc-tnccs dhcp lookip error-notify certexpire led addrblock unity counters
Listening IP addresses:
192.168.1.100
Connections:
test: %any...18.xxx.xxx.xxx IKEv2
test: local: [CN=VentanaTest] uses public key authentication
test: cert: "CN=VentanaTest"
test: remote: [18.xxx.xxx.xxx] uses public key authentication
test: child: dynamic === 0.0.0.0/0 TUNNEL
Security Associations (1 up, 0 connecting):
test[1]: ESTABLISHED 65 seconds ago, 192.168.1.100[CN=VentanaTest]...18.xxx.xxx.xxx[18.xxx.xxx.xxx]
test[1]: IKEv2 SPIs: 0f0ebcd1d3f24257_i* 0bdabee4cebd469b_r, public key reauthentication in 23 hours
test[1]: IKE proposal: AES_CBC_128/HMAC_SHA1_96/PRF_HMAC_SHA1/MODP_2048
test{1}: INSTALLED, TUNNEL, reqid 1, ESP in UDP SPIs: c91f7359_i 005ce3a7_o
test{1}: AES_CBC_128/HMAC_SHA1_96, 0 bytes_i, 5068 bytes_o, (66pkts, 58s ago), rekeying in 14 minutes
test{1}: 192.168.11.254/32 === 0.0.0.0/0
ip xfrm 策略
src 192.168.11.254/32 dst 0.0.0.0/0
dir out priority 383615
tmpl src 192.168.1.100 dst 18.xxx.xxx.xxx
proto esp spi 0x005ce3a7 reqid 1 mode tunnel
src 0.0.0.0/0 dst 192.168.11.254/32
dir fwd priority 383615
tmpl src 18.xxx.xxx.xxx dst 192.168.1.100
proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 192.168.11.254/32
dir in priority 383615
tmpl src 18.xxx.xxx.xxx dst 192.168.1.100
proto esp reqid 1 mode tunnel
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket in priority 0
src 0.0.0.0/0 dst 0.0.0.0/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
src ::/0 dst ::/0
socket in priority 0
src ::/0 dst ::/0
socket out priority 0
iptables-保存
# Generated by iptables-save v1.8.4 on Sat Dec 12 20:24:03 2020
*filter
:INPUT ACCEPT [235:34857]
:FORWARD ACCEPT [0:0]
:OUTPUT ACCEPT [224:27000]
-A INPUT -d 192.168.11.254/32 -i wwan0 -m policy --dir in --pol ipsec --reqid 1 --proto esp -j ACCEPT
-A FORWARD -d 192.168.11.254/32 -i wwan0 -m policy --dir in --pol ipsec --reqid 1 --proto esp -j ACCEPT
-A FORWARD -s 192.168.11.254/32 -o wwan0 -m policy --dir out --pol ipsec --reqid 1 --proto esp -j ACCEPT
-A OUTPUT -s 192.168.11.254/32 -o wwan0 -m policy --dir out --pol ipsec --reqid 1 --proto esp -j ACCEPT
COMMIT
ip 路由显示表全部
default via 192.168.1.1 dev wwan0 table 220 proto static src 192.168.11.254
default via 192.168.1.1 dev wwan0
192.168.1.0/24 dev wwan0 proto kernel scope link src 192.168.1.100
broadcast 127.0.0.0 dev lo table local proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo table local proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo table local proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo table local proto kernel scope link src 127.0.0.1
broadcast 192.168.1.0 dev wwan0 table local proto kernel scope link src 192.168.1.100
local 192.168.1.100 dev wwan0 table local proto kernel scope host src 192.168.1.100
broadcast 192.168.1.255 dev wwan0 table local proto kernel scope link src 192.168.1.100
local 192.168.11.254 dev wwan0 table local proto kernel scope host src 192.168.11.254
::1 dev lo proto kernel metric 256 pref medium
fe80::/64 dev wwan0 proto kernel metric 256 pref medium
local ::1 dev lo table local proto kernel metric 0 pref medium
local fe80::8c85:a8ff:fe59:1d79 dev wwan0 table local proto kernel metric 0 pref medium
ff00::/8 dev wwan0 table local metric 256 pref medium
IP地址
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN group default qlen 1000
link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
inet 127.0.0.1/8 scope host lo
valid_lft forever preferred_lft forever
inet6 ::1/128 scope host
valid_lft forever preferred_lft forever
2: wwan0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc fq_codel state UP group default qlen 1000
link/ether 8e:85:a8:59:1d:79 brd ff:ff:ff:ff:ff:ff
inet 192.168.1.100/24 brd 192.168.1.255 scope global dynamic wwan0
valid_lft 42250sec preferred_lft 42250sec
inet 192.168.11.254/32 scope global wwan0
valid_lft forever preferred_lft forever
inet6 fe80::8c85:a8ff:fe59:1d79/64 scope link
valid_lft forever preferred_lft forever
ip -s xfrm 状态 在对 8.8.8.8 执行相同的 ping 操作后
root@focal-ventana:~# ip -s xfrm state
src 192.168.1.100 dst 18.xxx.xxx.xxx
proto esp spi 0x0ce2b2ec(216183532) reqid 1(0x00000001) mode tunnel
replay-window 0 seq 0x00000000 flag af-unspec (0x00100000)
auth-trunc hmac(sha1) 0x960b069b0a13ba25dcb30b85f8d5c7ce62efce48 (160 bits) 96
enc cbc(aes) 0x032cc1a7afb2035dd4b6124bf0a845f7 (128 bits)
encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
anti-replay context: seq 0x0, oseq 0xd, bitmap 0x00000000
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 2921(sec), hard 3600(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
1092(bytes), 13(packets)
add 2020-12-14 20:26:13 use 2020-12-14 20:36:15
stats:
replay-window 0 replay 0 failed 0
src 18.xxx.xx.x dst 192.168.1.100
proto esp spi 0xc56be3e6(3312182246) reqid 1(0x00000001) mode tunnel
replay-window 32 seq 0x00000000 flag af-unspec (0x00100000)
auth-trunc hmac(sha1) 0xa6789f18e159b6abc44c20850bada3b903c9a161 (160 bits) 96
enc cbc(aes) 0xf6b72416ee9c739502a67dc14883334b (128 bits)
encap type espinudp sport 4500 dport 4500 addr 0.0.0.0
anti-replay context: seq 0x0, oseq 0x0, bitmap 0x00000000
lifetime config:
limit: soft (INF)(bytes), hard (INF)(bytes)
limit: soft (INF)(packets), hard (INF)(packets)
expire add: soft 2582(sec), hard 3600(sec)
expire use: soft 0(sec), hard 0(sec)
lifetime current:
0(bytes), 0(packets)
add 2020-12-14 20:26:13 use -
stats:
replay-window 0 replay 0 failed 0
/proc/net/xfrm_stat
root@focal-ventana:~# vi /proc/net/xfrm_stat
XfrmInError 0
XfrmInBufferError 0
XfrmInHdrError 0
XfrmInNoStates 0
XfrmInStateProtoError 0
XfrmInError 0
XfrmInBufferError 0
XfrmInHdrError 0
XfrmInNoStates 0
XfrmInStateProtoError 0
XfrmInStateModeError 0
XfrmInStateSeqError 0
XfrmInStateExpired 0
XfrmInStateMismatch 0
XfrmInStateInvalid 0
XfrmInTmplMismatch 0
XfrmInNoPols 0
XfrmInPolBlock 0
XfrmInPolError 0
XfrmOutError 0
XfrmOutBundleGenError 0
XfrmOutBundleCheckError 0
XfrmOutNoStates 0
XfrmOutStateProtoError 0
XfrmOutStateModeError 0
XfrmOutStateSeqError 0
XfrmOutStateExpired 0
XfrmOutPolBlock 0
XfrmOutPolDead 0
XfrmOutPolError 0
XfrmFwdHdrError 0
XfrmOutStateInvalid 0
XfrmAcquireError 0
iptables -t nat -L
Chain PREROUTING (policy ACCEPT)
target prot opt source destination
Chain INPUT (policy ACCEPT)
target prot opt source destination
Chain OUTPUT (policy ACCEPT)
target prot opt source destination
Chain POSTROUTING (policy ACCEPT)
target prot opt source destination
答案1
因此，它正在发挥作用。
Toby L210 4G 调制解调器出厂默认使用 RNDIS 作为虚拟以太网链路。即使没有 IPSEC，RNDIS 也给我的网络流量带来了问题，所以我改用了 CDC-ECM。
来自 ublox AT 命令手册：
On a Linux system, to reach the maximum throughput it is necessary to recompile the kernel. Apply the following modification in /kernel/drivers/net/usb/usbnet.c:
Replace "size_t size = dev->rx_urb_size;"with "size_t size = (16*1024);
"If it is not possible to recompile the kernel, then try to use CDC-ECM.
现在 CDC-ECM 导致我的 IPSEC 在某处丢失了传入流量，因此我切换回 RNDIS 并停用驱动程序优化。
The RNDIS interface may not work if Linux kernel version of the DTE is higher than 4.8 and RNDIS driver optimization is enabled (AT+UDCONF=67,1).
The reported behavior is due to the “rndis_host:
Set valid random MAC on buggy devices” patch introduced in the Kernel version v4.8-rc1:
https://github.com/torvalds/linux/commit/a5a18bdf7453d505783e40e47ebb84bfdd35f93b#diff6dfe16ca0bdcdc21501ef11cfd202a10
The patch presents an incompatibility with the RNDIS driver optimization, which leads to a TCP and UDP traffic lost.
The issue is not present if the RNDIS driver optimization is disabled (AT+UDCONF=67,0), however this may lead to lower throughput.