我已经在 Kubernetes 集群中设置了身份验证服务，该集群位于 Windows 11 Pro 机器上的 Docker Desktop + WSL2 环境中。
例如，可以通过来自同一台 Windows 机器（WSL2 层之外）的 CURL 和 PostMan 请求来访问它。
然而，问题是，无法从同一 LAN 网络内的 Linux 机器访问该其余端点。
为了验证与该身份验证服务的连接是否有效，我设置了一个类型为“NodePort”的服务。使用从“kubectl get svc”获取的外部端口，curl 请求在 Windows 和 Linux 计算机上均能正常工作，并且服务会给出正确的答案。
但请注意，NodePort 服务完全绕过了带有假证书的 K8s ingress 的 TSL/SSL，因此该请求 - 虽然成功 - 仅适用于通过 HTTP 的请求。
因为我需要使用来自 auth 服务的响应中的 cookie，所以通过 HTTPS 的请求至关重要，但是 K8s Ingress 根本没有发送响应。
在 Windows 机器上，我打开了相关端口 80、443（以及 22，以确保万无一失），并在 Windows 防火墙的连接规则中将 Linux 机器添加到可信任的计算机中。我还尝试暂时禁用它，以查看它是否真的阻止了连接尝试。但是，请求仍然被默默地丢弃（可能是由 Ingress 丢弃的）。
但在做出任何假设之前，让我向您展示我的 LAN 概览，其中包含机器之间的 POST 请求以及所有相关的 K8s yaml 文件：
auth部署和服务定义如下：
apiVersion: apps/v1
kind: Deployment
metadata:
name: gts-auth-depl
spec:
replicas: 1
selector:
matchLabels:
app: gts-auth
template:
metadata:
labels:
app: gts-auth
spec:
containers:
- name: gts-auth
image: creativewarlock/gts-auth
imagePullPolicy: IfNotPresent
env:
- name: MONGO_URI
value: 'mongodb://gts-auth-mongo-srv:27017/auth'
- name: JWT_KEY
valueFrom:
secretKeyRef:
name: gts-jwt-secret
key: JWT_KEY
imagePullSecrets:
- name: docker-hub
---
apiVersion: v1
kind: Service
metadata:
name: gts-auth-srv
spec:
type: ClusterIP
selector:
app: gts-auth
ports:
- name: gts-auth
protocol: TCP
port: 3000
targetPort: 3000
身份验证和 Web 客户端的入口（虽然不相关，但只是为了展示）：
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
name: gts-ingress-srv
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/use-regex: 'true'
spec:
rules:
- host: gts.dev # Entry in System32/etc/hosts: 127.0.0.1 gts.dev (alternatively: localhost)
http:
paths:
- path: /api/users/?(.*) # will match /api/users/signin
pathType: Prefix
backend:
service:
name: gts-auth-srv
port:
number: 3000
- path: /?(.*) # map all other requests to web client
pathType: Prefix
backend:
service:
name: gts-client-srv
port:
number: 8080
用于外部/直接访问 auth 服务的 NodePort 服务：
apiVersion: v1
kind: Service
metadata:
name: gts-auth-srv-ext
spec:
type: NodePort
selector:
app: gts-auth
ports:
- name: gts-auth-srv
protocol: TCP
port: 3000
targetPort: 3000
Kubernetes 对 Ingress 的描述如下：
k describe pod ingress-nginx-controller-6bf7bc7f94-krv2c -n ingress
-nginx
Name: ingress-nginx-controller-6bf7bc7f94-krv2c
Namespace: ingress-nginx
Priority: 0
Node: docker-desktop/192.168.65.4
Start Time: Thu, 11 Aug 2022 12:31:42 +0200
Labels: app.kubernetes.io/component=controller
app.kubernetes.io/instance=ingress-nginx
app.kubernetes.io/name=ingress-nginx
pod-template-hash=6bf7bc7f94
Annotations: <none>
Status: Running
IP: 10.1.28.235
IPs:
IP: 10.1.28.235
Controlled By: ReplicaSet/ingress-nginx-controller-6bf7bc7f94
Containers:
controller:
Container ID: docker://0cef9bec68d309011a32e18bf4f3a2a7ec7050b3db6cc7bafd87565e370f6695
Image: registry.k8s.io/ingress-nginx/controller:v1.3.0@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
Image ID: docker-pullable://registry.k8s.io/ingress-nginx/controller@sha256:d1707ca76d3b044ab8a28277a2466a02100ee9f58a86af1535a3edf9323ea1b5
Ports: 80/TCP, 443/TCP, 8443/TCP
Host Ports: 0/TCP, 0/TCP, 0/TCP
Args:
/nginx-ingress-controller
--publish-service=$(POD_NAMESPACE)/ingress-nginx-controller
--election-id=ingress-controller-leader
--controller-class=k8s.io/ingress-nginx
--ingress-class=nginx
--configmap=$(POD_NAMESPACE)/ingress-nginx-controller
--validating-webhook=:8443
--validating-webhook-certificate=/usr/local/certificates/cert
--validating-webhook-key=/usr/local/certificates/key
State: Running
Started: Sun, 19 Feb 2023 11:50:07 +0100
Last State: Terminated
Reason: Error
Exit Code: 255
Started: Sat, 18 Feb 2023 11:44:18 +0100
Finished: Sun, 19 Feb 2023 11:49:36 +0100
Ready: True
Restart Count: 132
Requests:
cpu: 100m
memory: 90Mi
Liveness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=5
Readiness: http-get http://:10254/healthz delay=10s timeout=1s period=10s #success=1 #failure=3
Environment:
POD_NAME: ingress-nginx-controller-6bf7bc7f94-krv2c (v1:metadata.name)
POD_NAMESPACE: ingress-nginx (v1:metadata.namespace)
LD_PRELOAD: /usr/local/lib/libmimalloc.so
Mounts:
/usr/local/certificates/ from webhook-cert (ro)
/var/run/secrets/kubernetes.io/serviceaccount from kube-api-access-hpxhw (ro)
Conditions:
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Volumes:
webhook-cert:
Type: Secret (a volume populated by a Secret)
SecretName: ingress-nginx-admission
Optional: false
kube-api-access-hpxhw:
ConfigMapOptional: <nil> DownwardAPI: true
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Normal RELOAD 54m (x3 over 95m) nginx-ingress-controller NGINX reload triggered due to a change in configuration
以下是 Ingress 控制器的日志。最后一行显示了来自 Windows 计算机的 curl 请求的响应。
k logs -f ingress-nginx-controller-6bf7bc7f94-krv2c -n ingress-nginx
W0219 10:50:08.009613 7 client_config.go:617] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0219 10:50:08.010796 7 main.go:230] "Creating API client" host="https://10.96.0.1:443"
W0219 10:50:30.162337 7 main.go:271] Initial connection to the Kubernetes API server was retried 1 times.
I0219 10:50:30.162386 7 main.go:274] "Running in Kubernetes cluster" major="1" minor="24" git="v1.24.1" state="clean" commit="3ddd0f45aa91e2f30c70734b175631bec5b5825a" platform="linux/amd64"
I0219 10:50:30.278395 7 main.go:104] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0219 10:50:30.299891 7 ssl.go:531] "loading tls certificate" path="/usr/local/certificates/cert" key="/usr/local/certificates/key"
I0219 10:50:30.314983 7 nginx.go:258] "Starting NGINX Ingress controller"
I0219 10:50:30.321843 7 event.go:285] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-controller", UID:"407f816e-98b9-4ab6-a08f-95efafa21ddc", APIVersion:"v1", ResourceVersion:"612", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-controller
I0219 10:50:31.516611 7 nginx.go:301] "Starting NGINX process"
I0219 10:50:31.516698 7 leaderelection.go:248] attempting to acquire leader lease ingress-nginx/ingress-controller-leader...
I0219 10:50:31.517522 7 nginx.go:321] "Starting validation webhook" address=":8443" certPath="/usr/local/certificates/cert" keyPath="/usr/local/certificates/key"
I0219 10:50:31.517893 7 controller.go:167] "Configuration changes detected, backend reload required"
I0219 10:50:31.543621 7 leaderelection.go:258] successfully acquired lease ingress-nginx/ingress-controller-leader
I0219 10:50:31.543648 7 status.go:84] "New leader elected" identity="ingress-nginx-controller-6bf7bc7f94-krv2c"
I0219 10:50:31.610775 7 controller.go:184] "Backend successfully reloaded"
I0219 10:50:31.610910 7 controller.go:195] "Initial sync, sleeping for 1 second"
192.168.65.4 - - [19/Feb/2023:13:09:46 +0000] "POST /api/users/signin/ HTTP/1.1" 200 150 "-" "curl/7.83.1" 259 0.054 [default-gts-auth-srv-3000] [] 10.1.28.246:3000 150 0.050 200 5b8761d90f76b4c160b0a2effc980b85
来自我的 Linux 机器“Kaspar”的 curl HTTPS 请求未记录在 Ingress pod 中，并且发送者从 NGINX 收到 404 HTML 文档：
Blacky@Kaspar:~$ curl -k -H 'Content-Type: application/json' https://192.168.0.57/api/users/signin/ -X POST -d '{ "email": "[email protected]", "password": "iHhRCF9Dm3wKT" }'
<html>
<head><title>404 Not Found</title></head>
<body>
<center><h1>404 Not Found</h1></center>
<hr><center>nginx</center>
</body>
</html>
在 Windows 防火墙中，我将 Linux 机器添加到可信任的计算机，并通过 powershell 脚本明确设置了这些规则：
$wsl_ip = (ubuntu.exe -c "ifconfig eth0 | grep 'inet '").trim().split()| where {$_}
$regex = [regex] "\b(?:(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\.){3}(?:25[0-5]|2[0-4][0-9]|[01]?[0-9][0-9]?)\b"
$ip_array = $regex.Matches($wsl_ip) | %{ $_.value }
$wsl_ip = $ip_array[0]
Write-Host "WSL Machine IP: ""$wsl_ip"""
New-NetFireWallRule -DisplayName 'WSL SSL' -Direction Outbound -LocalPort 443 -Action Allow -Protocol TCP
New-NetFireWallRule -DisplayName 'WSL SSL' -Direction Inbound -LocalPort 443 -Action Allow -Protocol TCP
#### ------------ Clear all portproxy rules ------------ ####
netsh int portproxy reset all
#### ------------ SSH rule - port 22 ------------ ####
netsh interface portproxy add v4tov4 listenport=22 listenaddress=0.0.0.0 connectport=22 connectaddress=$wsl_ip
#### ------------ Webserver SSL rule - port 443 ------------ ####
netsh interface portproxy add v4tov4 listenport=443 listenaddress=0.0.0.0 connectport=443 connectaddress=$wsl_ip
#### ------------ Webserver SSL rule - port 80 ------------ ####
netsh interface portproxy add v4tov4 listenport=80 listenaddress=0.0.0.0 connectport=80 connectaddress=$wsl_ip
所以我的主要问题是：
Kubernetes 中的（自签名）伪造证书是否造成了问题？我是否应该使用适当的 letsEncrypt 证书在我的 LAN 中使用？
由于 curl/postman 请求在 Windows 机器上（WSL2 层之外）工作，我首先认为这是一个配置 Windows 防火墙的问题，但似乎 K8s Ingress 没有正确响应该请求。但是，我缺乏分析 Ingress 内部情况的工具。除了“kubectl logs -f <ingress...> -n ingress-nginx”之外还有其他方法吗？
任何关于如何从我的 LAN 内的另一台机器执行 HTTPS 请求的建议都非常受欢迎。