语境
学校项目：在 kubernetes 节点 minikube 上托管多个服务，包括使用负载均衡器在端口 21 上的 ftps 服务器。学校要求我们基于 Alpine 制作自己的图像。
我目前所做的
我将 metallb 设置为负载均衡器。
我构建了一个图像，在其中设置了 vsftpd 并生成了证书和密钥：
openssl req -x509 -subj "/C=FR/ST=PARIS/L=Paris/O=42_Corp/OU=myname/CN=mywebsite" -nodes -days 365 -newkey rsa:2048 -keyout /etc/ssl/private/vsftpd.key -out /etc/ssl/certs/vsftpd.crt
旁注：我使用 Supervisor 在同一个容器中运行 Telegraf 以获取仪表板。
我使用这个 yml 部署该图像：
metadata:
name: ftps
spec:
serviceName: "ftps"
selector:
matchLabels:
app: ftps
replicas: 2 # tells deployment to run 2 pods matching the template
template:
metadata:
labels:
app: ftps
spec:
containers:
- name: ftps
image: ft_service/ftps
imagePullPolicy: Never
volumeMounts:
- name: ftps-data
mountPath: /mnt/ftps_data
volumes:
- name: ftps-data
---
apiVersion: v1
kind: Service
metadata:
labels:
app: ftps
name: ftps
namespace: default
spec:
type: LoadBalancer
externalTrafficPolicy: Cluster
ports:
- name: port-20
port: 20
protocol: TCP
targetPort: 20
- name: port-21
port: 21
protocol: TCP
targetPort: 21
selector:
app: ftps
然后我有一个 vsftp 的 conf 文件，其中包含以下内容：
local_enable=YES
write_enable=YES
dirmessage_enable=YES
xferlog_enable=YES
connect_from_port_20=YES
chroot_local_user=YES
allow_writeable_chroot=YES
listen=YES
rsa_cert_file=/etc/ssl/certs/vsftpd.crt
rsa_private_key_file=/etc/ssl/private/vsftpd.key
ssl_enable=YES
allow_anon_ssl=NO
force_local_data_ssl=YES
force_local_logins_ssl=YES
ssl_tlsv1=YES
ssl_sslv2=NO
ssl_sslv3=NO
require_ssl_reuse=NO
ssl_ciphers=HIGH
seccomp_sandbox=NO
（最后的“seccomp_sandbox=NO”似乎解决了我一开始遇到的一些问题）
这让我...
一旦我部署了所有东西，我就会得到我的 pod：
default pod/ftps-1 1/1 Running 0 7m55s
我得到我的服务：
default service/ftps LoadBalancer 10.104.132.127 192.168.1.241 20:31268/TCP,21:31581/TCP 8m43s
然后当我跑步时minikube service ftps --url，我得到
http://172.17.0.2:31268
http://172.17.0.2:31581
我遇到的问题
我使用 lftp（我尝试了其他几个，结果几乎相同，所以我排除了客户端问题......但我对整个 ftp 事情还很陌生，所以......我让你判断）
1. 使用 lftp 和 ftps://
lftp -u user,password ftps://172.17.0.2 -p 31581
然后ls
ls: Fatal error: gnutls_handshake: An unexpected TLS packet was received
下面是使用调试的非常长的版本（由于帖子限制，我截断了许多“GNUTLS：ASSERT：extensions.c[_gnutls_get_extension]:65”行）：
lftp [email protected]:~> debug 99999999
lftp [email protected]:~> ls
FileCopy(0x563eb3eed780) enters state INITIAL
FileCopy(0x563eb3eed780) enters state DO_COPY
---- dns cache hit
---- attempt number 1 (max_retries=1000)
---- Connecting to 172.17.0.2 (172.17.0.2) port 31581
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
[...]
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: common.c[x509_read_value]:698
GNUTLS: ASSERT: common.c[x509_read_value]:698
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: ASSERT: extensions.c[_gnutls_get_extension]:65
GNUTLS: REC[0x563eb42050a0]: Allocating epoch #0
GNUTLS: ASSERT: constate.c[_gnutls_epoch_get]:600
GNUTLS: REC[0x563eb42050a0]: Allocating epoch #1
GNUTLS: HSK[0x563eb42050a0]: Adv. version: 3.3
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_256_GCM_SHA384 (C0.2C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_CAMELLIA_256_GCM_SHA384 (C0.87)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_CHACHA20_POLY1305 (CC.A9)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_256_CCM (C0.AD)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_256_CBC_SHA1 (C0.0A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_128_GCM_SHA256 (C0.2B)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_CAMELLIA_128_GCM_SHA256 (C0.86)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_128_CCM (C0.AC)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_AES_128_CBC_SHA1 (C0.09)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_ECDSA_3DES_EDE_CBC_SHA1 (C0.08)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_256_GCM_SHA384 (C0.30)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_CAMELLIA_256_GCM_SHA384 (C0.8B)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_CHACHA20_POLY1305 (CC.A8)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_256_CBC_SHA1 (C0.14)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_128_GCM_SHA256 (C0.2F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_CAMELLIA_128_GCM_SHA256 (C0.8A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_AES_128_CBC_SHA1 (C0.13)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_ECDHE_RSA_3DES_EDE_CBC_SHA1 (C0.12)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_256_GCM_SHA384 (00.9D)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_256_GCM_SHA384 (C0.7B)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_256_CCM (C0.9D)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_256_CBC_SHA1 (00.35)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_256_CBC_SHA1 (00.84)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_128_GCM_SHA256 (00.9C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_128_GCM_SHA256 (C0.7A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_128_CCM (C0.9C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_AES_128_CBC_SHA1 (00.2F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_CAMELLIA_128_CBC_SHA1 (00.41)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_RSA_3DES_EDE_CBC_SHA1 (00.0A)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_256_GCM_SHA384 (00.9F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_256_GCM_SHA384 (C0.7D)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CHACHA20_POLY1305 (CC.AA)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_256_CCM (C0.9F)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_256_CBC_SHA1 (00.39)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_256_CBC_SHA1 (00.88)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_128_GCM_SHA256 (00.9E)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_128_GCM_SHA256 (C0.7C)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_128_CCM (C0.9E)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_AES_128_CBC_SHA1 (00.33)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_CAMELLIA_128_CBC_SHA1 (00.45)
GNUTLS: HSK[0x563eb42050a0]: Keeping ciphersuite: GNUTLS_DHE_RSA_3DES_EDE_CBC_SHA1 (00.16)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Extended Master Secret (0 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Encrypt-then-MAC (0 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension OCSP Status Request (5 bytes)
GNUTLS: HSK[0x563eb42050a0]: sent server name: '172.17.0.2'
GNUTLS: EXT[0x563eb42050a0]: Sending extension Server Name Indication (15 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Safe Renegotiation (1 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Session Ticket (0 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Supported curves (12 bytes)
GNUTLS: EXT[0x563eb42050a0]: Sending extension Supported ECC Point Formats (2 bytes)
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (4.1) RSA-SHA256
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (4.3) ECDSA-SHA256
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (5.1) RSA-SHA384
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (5.3) ECDSA-SHA384
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (6.1) RSA-SHA512
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (6.3) ECDSA-SHA512
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (3.1) RSA-SHA224
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (3.3) ECDSA-SHA224
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (2.1) RSA-SHA1
GNUTLS: EXT[0x563eb42050a0]: sent signature algo (2.3) ECDSA-SHA1
GNUTLS: EXT[0x563eb42050a0]: Sending extension Signature Algorithms (22 bytes)
GNUTLS: HSK[0x563eb42050a0]: CLIENT HELLO was queued [220 bytes]
GNUTLS: REC[0x563eb42050a0]: Preparing Packet Handshake(22) with length: 220 and min pad: 0
GNUTLS: ENC[0x563eb42050a0]: cipher: NULL, MAC: MAC-NULL, Epoch: 0
GNUTLS: REC[0x563eb42050a0]: Sent Packet[1] Handshake(22) in epoch 0 and length: 225
GNUTLS: ASSERT: buffers.c[get_last_packet]:1160
GNUTLS: ASSERT: buffers.c[_gnutls_io_read_buffered]:587
GNUTLS: ASSERT: buffers.c[get_last_packet]:1160
GNUTLS: ASSERT: buffers.c[_gnutls_io_read_buffered]:587
GNUTLS: ASSERT: buffers.c[get_last_packet]:1160
GNUTLS: REC[0x563eb42050a0]: SSL 48.48 Unknown Packet packet received. Epoch 0, length: 8271
GNUTLS: ASSERT: record.c[check_recv_type]:580
GNUTLS: Received record packet of unknown type 53
GNUTLS: ASSERT: record.c[recv_headers]:1093
GNUTLS: ASSERT: record.c[_gnutls_recv_in_buffers]:1175
GNUTLS: ASSERT: buffers.c[_gnutls_handshake_io_recv_int]:1415
GNUTLS: ASSERT: handshake.c[_gnutls_recv_handshake]:1474
GNUTLS: ASSERT: handshake.c[handshake_client]:2841
**** gnutls_handshake: An unexpected TLS packet was received.
GNUTLS: REC[0x563eb42050a0]: Start of epoch cleanup
GNUTLS: REC[0x563eb42050a0]: End of epoch cleanup
GNUTLS: REC[0x563eb42050a0]: Epoch #0 freed
GNUTLS: REC[0x563eb42050a0]: Epoch #1 freed
---- Closing control socket
ls: Fatal error: gnutls_handshake: An unexpected TLS packet was received.
lftp [email protected]:~> exit
2. 使用 lftp 和 ftp://
如果我从上面的 lftp 命令中的 ftp 中删除 s，我会得到：
➜ repo git:(master) ✗ lftp -u user,password ftp://172.17.0.2 -p 31581
lftp [email protected]:~> set ssl:verify-certificate false
lftp [email protected]:~> ls
Interrupt
lftp [email protected]:/> debug 9
lftp [email protected]:/> ls
---- Connecting to 172.17.0.2 (172.17.0.2) port 31581
<--- 220 (vsFTPd 3.0.3)
---> FEAT
<--- 211-Features:
<--- AUTH TLS
<--- EPRT
<--- EPSV
<--- MDTM
<--- PASV
<--- PBSZ
<--- PROT
<--- REST STREAM
<--- SIZE
<--- TVFS
<--- UTF8
<--- 211 End
---> AUTH TLS
<--- 234 Proceed with negotiation.
---> OPTS UTF8 ON
Certificate: C=FR,ST=PARIS,L=Paris,O=42_Corp,OU=user,CN=email
Issued by: C=FR,ST=PARIS,L=Paris,O=42_Corp,OU=user,CN=email
WARNING: Certificate verification: Not trusted (28:51:50:4B:83:40:96:E7:2C:9A:1E:9B:38:C5:25:46:B4:CC:4C:24)
WARNING: Certificate verification: certificate common name doesn't match requested host name ‘172.17.0.2’ (28:51:50:4B:83:40:96:E7:2C:9A:1E:9B:38:C5:25:46:B4:CC:4C:24)
<--- 200 Always in UTF8 mode.
---> USER ftp42
<--- 331 Please specify the password.
---> PASS 42
<--- 230 Login successful.
---> PBSZ 0
<--- 200 PBSZ set to 0.
---> PROT P
<--- 200 PROT now Private.
---> PASV
<--- 227 Entering Passive Mode (172,18,0,6,194,222).
---- Connecting data socket to (172.18.0.6) port 49886
Interrupt
---> LIST
---> ABOR
---- Closing aborted data socket
lftp [email protected]:/>
<--- 425 Failed to establish connection.
<--- 225 No transfer to ABOR.
lftp [email protected]:/>
总结
我猜我的问题出在 SSL 配置上，但在哪里呢？...我遗漏了什么？感谢您阅读这篇很长的帖子 :)
答案1
ftps 始终需要额外的端口（被动模式），正如您在 lftp 日志中看到的那样，此时它会失败。我通过向 ftps k8s 服务添加端口并在 vsftpd 的配置中设置特定的额外 2 个端口来解决这个问题。