我遇到了一个非常奇怪的情况，kube-proxy某些节点不接受连接，坦率地说，我很困惑。
背景：
- kubernetes 集群安装在AWS EC2使用科普斯
- 中的集群
eu-central-1分布在三个可用区（a、b、c） - kubernetes 版本是，
v1.14.6但是这也被观察到了v.1.13.x - 日志中似乎没有任何错误
kube-proxy，事实上输出看起来非常相似 kube-proxy做监听给定的端口，但请求超时ingress该问题似乎仅与服务、内部负载均衡器和 HTTPS（443）端口有关
以下为示例。两个节点均位于同一可用区域。
健康节点：
curl --connect-timeout 5 -sD - -k https://localhost:32028/healthz
HTTP/2 200
date: Sun, 13 Oct 2019 13:51:32 GMT
content-type: text/html
content-length: 0
sudo netstat -taupen | grep kube-proxy | grep LISTEN
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 0 21284 3162/kube-proxy
tcp6 0 0 :::32618 :::* LISTEN 0 23820 3162/kube-proxy
tcp6 0 0 :::32012 :::* LISTEN 0 21359 3162/kube-proxy
tcp6 0 0 :::10256 :::* LISTEN 0 21280 3162/kube-proxy
tcp6 0 0 :::30259 :::* LISTEN 0 21358 3162/kube-proxy
tcp6 0 0 :::30844 :::* LISTEN 0 21361 3162/kube-proxy
tcp6 0 0 :::32028 :::* LISTEN 0 21360 3162/kube-proxy
tcp6 0 0 :::30048 :::* LISTEN 0 21357 3162/kube-proxy
不健康节点：
curl -v --connect-timeout 5 -sD - -k https://localhost:32028/healthz'
* Trying 127.0.0.1...
* TCP_NODELAY set
* Connection timed out after 5001 milliseconds
* Curl_http_done: called premature == 1
* stopped the pause stream!
* Closing connection 0
sudo netstat -taupen | grep kube-proxy | grep LISTEN
tcp 0 0 127.0.0.1:10249 0.0.0.0:* LISTEN 0 23611 2726/kube-proxy
tcp6 0 0 :::32618 :::* LISTEN 0 22662 2726/kube-proxy
tcp6 0 0 :::32012 :::* LISTEN 0 22654 2726/kube-proxy
tcp6 0 0 :::10256 :::* LISTEN 0 21872 2726/kube-proxy
tcp6 0 0 :::30259 :::* LISTEN 0 22653 2726/kube-proxy
tcp6 0 0 :::32028 :::* LISTEN 0 22656 2726/kube-proxy
tcp6 0 0 :::30844 :::* LISTEN 0 22655 2726/kube-proxy
tcp6 0 0 :::30048 :::* LISTEN 0 22652 2726/kube-proxy
环境：
kubectl get services nginx-ingress
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
nginx-ingress LoadBalancer 100.67.138.99 xxxx-yyyy.elb.eu-central-1.amazonaws.com 80:30259/TCP,443:32028/TCP,22:32012/TCP 47h
kubectl version
Client Version: version.Info{Major:"1", Minor:"16", GitVersion:"v1.16.1", GitCommit:"d647ddbd755faf07169599a625faf302ffc34458", GitTreeState:"clean", BuildDate:"2019-10-02T23:49:07Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"darwin/amd64"}
Server Version: version.Info{Major:"1", Minor:"14", GitVersion:"v1.14.6", GitCommit:"96fac5cd13a5dc064f7d9f4f23030a6aeface6cc", GitTreeState:"clean", BuildDate:"2019-08-19T11:05:16Z", GoVersion:"go1.12.9", Compiler:"gc", Platform:"linux/amd64"}
kubectl get deployment nginx-ingress -oyaml：
apiVersion: extensions/v1beta1
kind: Deployment
metadata:
annotations:
checksum/config: 0f815cbf49129a18dacd05c1f35c0e2c0a36d0ad4f8f0272828a558c49c40aed
configmap.reloader.stakater.com/reload: ingress-nginx,tcp-services,udp-services
deployment.kubernetes.io/revision: "5"
creationTimestamp: "2019-10-11T14:16:23Z"
generation: 18
labels:
app: nginx-ingress
chart: nginx-ingress-0.26.1
heritage: Tiller
k8s-addon: ingress-nginx.addons.k8s.io
k8s-app: nginx-ingress-controller
release: nginx-ingress-private
name: nginx-ingress
namespace: backend
resourceVersion: "85333311"
selfLink: /apis/extensions/v1beta1/namespaces/backend/deployments/nginx-ingress
uid: b4a6f226-ec31-11e9-bd40-066623cdec10
spec:
progressDeadlineSeconds: 600
replicas: 3
revisionHistoryLimit: 10
selector:
matchLabels:
app: nginx-ingress
release: nginx-ingress-private
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
creationTimestamp: null
labels:
app: nginx-ingress
k8s-addon: ingress-nginx.addons.k8s.io
k8s-app: nginx-ingress-controller
release: nginx-ingress-private
spec:
containers:
- args:
- /nginx-ingress-controller
- --logtostderr=true
- --stderrthreshold=0
- --http-port=80
- --https-port=443
- --healthz-port=10254
- --default-backend-service=$(POD_NAMESPACE)/nginx-ingress-default
- --configmap=$(POD_NAMESPACE)/ingress-nginx
- --tcp-services-configmap=$(POD_NAMESPACE)/tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/udp-services
- --publish-service=$(POD_NAMESPACE)/nginx-ingress
- --default-ssl-certificate=$(POD_NAMESPACE)/certs-tls
- --ingress-class=private
env:
- name: POD_NAME
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
apiVersion: v1
fieldPath: metadata.namespace
- name: STAKATER_TCP_SERVICES_CONFIGMAP
value: 060d83d51ec7d01fe8af12484189acf792044690
- name: STAKATER_UDP_SERVICES_CONFIGMAP
value: da39a3ee5e6b4b0d3255bfef95601890afd80709
- name: STAKATER_INGRESS_NGINX_CONFIGMAP
value: 6a393dab54c63f4117785f635b7c00c64e140853
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:0.26.1
imagePullPolicy: IfNotPresent
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 30
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 5
name: nginx-ingress
ports:
- containerPort: 80
name: http
protocol: TCP
- containerPort: 443
name: https
protocol: TCP
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources:
limits:
cpu: 250m
memory: 512Mi
requests:
cpu: 10m
memory: 50Mi
securityContext:
capabilities:
add:
- NET_BIND_SERVICE
drop:
- ALL
runAsUser: 33
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
dnsPolicy: ClusterFirst
restartPolicy: Always
schedulerName: default-scheduler
securityContext: {}
serviceAccount: nginx-ingress
serviceAccountName: nginx-ingress
terminationGracePeriodSeconds: 60
status:
availableReplicas: 3
conditions:
- lastTransitionTime: "2019-10-11T14:16:24Z"
lastUpdateTime: "2019-10-11T21:55:50Z"
message: ReplicaSet "nginx-ingress-6d994d7b96" has successfully progressed.
reason: NewReplicaSetAvailable
status: "True"
type: Progressing
- lastTransitionTime: "2019-10-13T11:05:46Z"
lastUpdateTime: "2019-10-13T11:05:46Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
observedGeneration: 18
readyReplicas: 3
replicas: 3
updatedReplicas: 3
/etc/kubernetes/manifests/kube-proxy.manifest：
apiVersion: v1
kind: Pod
metadata:
annotations:
scheduler.alpha.kubernetes.io/critical-pod: ""
creationTimestamp: null
labels:
k8s-app: kube-proxy
tier: node
name: kube-proxy
namespace: kube-system
spec:
containers:
- command:
- /bin/sh
- -c
- mkfifo /tmp/pipe; (tee -a /var/log/kube-proxy.log < /tmp/pipe & ) ; exec /usr/local/bin/kube-proxy
--cluster-cidr=100.96.0.0/11 --conntrack-max-per-core=131072 --hostname-override=ip-aaa-bbb-ccc-ddd.eu-central-1.compute.internal
--kubeconfig=/var/lib/kube-proxy/kubeconfig --master=https://api.xxx.yyy.zzz
--oom-score-adj=-998 --resource-container="" --v=2 > /tmp/pipe 2>&1
image: k8s.gcr.io/kube-proxy:v1.14.6
name: kube-proxy
resources:
requests:
cpu: 100m
securityContext:
privileged: true
volumeMounts:
- mountPath: /var/lib/kube-proxy/kubeconfig
name: kubeconfig
readOnly: true
- mountPath: /var/log/kube-proxy.log
name: logfile
- mountPath: /lib/modules
name: modules
readOnly: true
- mountPath: /etc/ssl/certs
name: ssl-certs-hosts
readOnly: true
- mountPath: /run/xtables.lock
name: iptableslock
hostNetwork: true
priorityClassName: system-node-critical
tolerations:
- key: CriticalAddonsOnly
operator: Exists
volumes:
- hostPath:
path: /var/lib/kube-proxy/kubeconfig
name: kubeconfig
- hostPath:
path: /var/log/kube-proxy.log
name: logfile
- hostPath:
path: /lib/modules
name: modules
- hostPath:
path: /usr/share/ca-certificates
name: ssl-certs-hosts
- hostPath:
path: /run/xtables.lock
type: FileOrCreate
name: iptableslock
status: {}
kubectl get service nginx-ingress -oyaml：
apiVersion: v1
kind: Service
metadata:
annotations:
dns.alpha.kubernetes.io/internal: private.xxx.yyy.zzz
external-dns.alpha.kubernetes.io/hostname: private.xxx.yyy.zzz
kubernetes.io/ingress.class: private
service.beta.kubernetes.io/aws-load-balancer-cross-zone-load-balancing-enabled: "true"
service.beta.kubernetes.io/aws-load-balancer-internal: 0.0.0.0/0
service.beta.kubernetes.io/aws-load-balancer-type: nlb
service.beta.kubernetes.io/do-loadbalancer-enable-proxy-protocol: "false"
creationTimestamp: "2019-10-11T14:16:23Z"
labels:
app: nginx-ingress
chart: nginx-ingress-0.26.1
heritage: Tiller
k8s-addon: ingress-nginx.addons.k8s.io
release: nginx-ingress-private
name: nginx-ingress
namespace: backend
resourceVersion: "84591201"
selfLink: /api/v1/namespaces/backend/services/nginx-ingress
uid: b4a1b347-ec31-11e9-bd40-066623cdec10
spec:
clusterIP: 100.xxx.yyy.zzz
externalTrafficPolicy: Local
healthCheckNodePort: 32618
ports:
- name: http
nodePort: 30259
port: 80
protocol: TCP
targetPort: http
- name: https
nodePort: 32028
port: 443
protocol: TCP
targetPort: https
- name: ssh
nodePort: 32012
port: 22
protocol: TCP
targetPort: 22
selector:
app: nginx-ingress
release: nginx-ingress-private
sessionAffinity: None
type: LoadBalancer
status:
loadBalancer:
ingress:
- hostname: xxx-yyy.elb.eu-central-1.amazonaws.com
kubectl get pods -n backend -o wide：
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
drone-drone-server-59586dd487-rmv9b 1/1 Running 0 2d10h aaa.bbb.13.19 ip-172-xxx-yyy-180.eu-central-1.compute.internal <none> <none>
kube-slack-848c9646fd-8w5mw 1/1 Running 0 2d10h aaa.bbb.13.10 ip-172-xxx-yyy-180.eu-central-1.compute.internal <none> <none>
nginx-ingress-5nlzg 1/1 Running 0 9h aaa.bbb.14.164 ip-172-xxx-yyy-201.eu-central-1.compute.internal <none> <none>
nginx-ingress-7xb54 1/1 Running 0 9h aaa.bbb.15.120 ip-172-xxx-yyy-156.eu-central-1.compute.internal <none> <none>
nginx-ingress-default-589975445-qdjvz 1/1 Running 0 9h aaa.bbb.14.150 ip-172-xxx-yyy-201.eu-central-1.compute.internal <none> <none>
nginx-ingress-jqtd6 1/1 Running 0 9h aaa.bbb.12.84 ip-172-xxx-yyy-29.eu-central-1.compute.internal <none> <none>
nginx-ingress-z9nt8 1/1 Running 0 9h aaa.bbb.13.57 ip-172-xxx-yyy-180.eu-central-1.compute.internal <none> <none>
sonarqube-sonarqube-746cbc858b-ks87c 1/1 Running 0 2d10h aaa.bbb.12.13 ip-172-xxx-yyy-29.eu-central-1.compute.internal <none> <none>
youtrack-0 1/1 Running 0 2d10h aaa.bbb.12.52 ip-172-xxx-yyy-29.eu-central-1.compute.internal <none> <none>
如果您能提供任何关于可能出现错误或如何进一步调试问题的想法，我们将不胜感激。
更新：看到“不健康”/不工作的节点只是 nginx 所在的节点不是已部署。将 nginx pod 部署为DeamonSet而不是Deployment解决了该问题。这仍然没有回答为什么此问题仅在严格的特定条件下出现（非普通 HTTP 端口 + 内部负载均衡器）以及如何解决它的问题。
答案1
我想说它按照设计运行。
请注意，通过在 Ingress Controller 的“nginx-ingress”服务（LB）中指定以下注释：
externalTrafficPolicy: Local
您正在强制入口流量仅由节点本地的这些 Pod（入口控制器的副本）提供服务。在 AWS 中，NLB 不知道哪些节点（负载均衡器的注册目标）没有自己的 Nginx-Ingress Controller 副本，因此有时 LB 分布式流量会失败。
为了避免此行为，您需要从 LB 健康检查探测中手动删除这些节点。
为什么从“Deployment”切换到“DeamonSet”可以解决您的问题？因为它确保每个节点都运行一个 Ingress Controller Pod 的副本。
请注意，这是云提供商负载均衡器的特定行为，记录在案这里。
如果您不关心保留源客户端 IP，只需将“ externalTrafficPolicy”从“ Local”更改为“ Cluster”即可解决您的问题。