我遇到很多问题，似乎源于将我的 kubernetes 集群升级到最新版本（1.23.5）。我最初遇到了集群本身和节点的一些问题，但这些问题似乎已经得到解决，集群看起来很健康，至少在我运行时是这么显示的kops validate cluster。我现在面临的问题是我的 ingress-nginx pod 没有运行，这意味着我的负载均衡器没有任何指向，因此即使应用程序 pod 正在顺利运行，我也无法访问我的应用程序。我使用 helm 创建了 ingress-nginx 资源，并将我尝试用于升级的文件粘贴到下面。我尝试了多种方法来解决这个问题，我认为我主要缺少的是 IngressClass 的东西，我试图在多个地方包含它，但我不知道如何做到这一点。我的集群只有一个入口控制器，并且在部署中为应用程序的每个实例定义了一个 Ingress 实例。您还将看到 AppVersion 是 0.24.0，我尝试通过多种方式改变它并在 deploy.yaml 中使用不同的图像。
rbac.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ .Chart.Name }}-serviceaccount
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: {{ .Chart.Name }}-clusterrole
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ .Chart.Name }}-role
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ .Chart.Name }}-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ .Chart.Name }}-role
subjects:
- kind: ServiceAccount
name: {{ .Chart.Name }}-serviceaccount
namespace: {{ .Release.Namespace }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: {{ .Chart.Name }}-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ .Chart.Name }}-clusterrole
subjects:
- kind: ServiceAccount
name: {{ .Chart.Name }}-serviceaccount
namespace: {{ .Release.Namespace }}
服务.yaml
---
# Main service ingesting http traffic
kind: Service
apiVersion: v1
metadata:
name: loadbalancer-ingress
labels:
app.kubernetes.io/managed-by: Helm
annotations:
{{- if .Values.loadbalancer.cert }}
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {{ .Values.loadbalancer.cert | quote }}
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "{{- range .Values.loadbalancer.ports -}}{{- if .ssl -}}{{ .name }},{{- end -}}{{- end -}}"
{{- end }}
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: {{ .Values.loadbalancer.backend_protocol | quote }}
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
type: LoadBalancer
selector:
pod: {{ .Chart.Name }}
ports:
{{- range .Values.loadbalancer.ports }}
- name: {{ .name }}
port: {{ .port }}
targetPort: {{ .targetPort }}
{{- end }}
---
# Dummy service to stop the controller from nagging about ingress-nginx service
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/managed-by: Helm
spec:
ports:
- name: http
port: 10254
targetPort: 10254
selector:
pod: {{ .Chart.Name }}
---
部署.yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: {{ .Chart.Name }}-controller
spec:
selector:
matchLabels:
pod: {{ .Chart.Name }}
template:
metadata:
labels:
pod: {{ .Chart.Name }}
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
fluentbit.io/parser: k8s-nginx-ingress
spec:
serviceAccountName: {{ .Chart.Name }}-serviceaccount
containers:
- name: {{ .Chart.Name }}-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:{{ .Chart.AppVersion }}
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/{{ .Chart.Name }}-nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/{{ .Chart.Name }}-tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/{{ .Chart.Name }}-udp-services
- --publish-service=$(POD_NAMESPACE)/loadbalancer-ingress
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: metrics
containerPort: 10254
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
配置映射.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
name: {{ .Chart.Name }}-nginx-configuration
data:
use-proxy-protocol: "false"
use-forwarded-headers: "true"
server-tokens: "false"
---
kind: ConfigMap
apiVersion: v1
metadata:
name: {{ .Chart.Name }}-tcp-services
---
kind: ConfigMap
apiVersion: v1
metadata:
name: {{ .Chart.Name }}-udp-services
图表.yaml
name: ingress-nginx
description: Cluster - Ingress Controller
version: 1
apiVersion: v1
appVersion: "0.24.0"
值.yaml
loadbalancer:
backend_protocol: http
cert: <my-cert>
ports:
- name: http
port: 80
targetPort: 80
ssl: false
- name: https
port: 443
targetPort: 80
ssl: true
我正在运行的命令。
helm upgrade ingress-nginx --install --namespace ingress-nginx ./
我目前得到的输出。
W0327 19:53:47.472827 8 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0327 19:53:47.473136 8 main.go:241] "Creating API client" host="https://100.64.0.1:443"
I0327 19:53:47.487201 8 main.go:285] "Running in Kubernetes cluster" major="1" minor="23" git="v1.23.5" state="clean" commit="c285e781331a3785a7f436042c65c5641ce8a9e9" platform="linux/amd64"
I0327 19:53:47.684135 8 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0327 19:53:47.689215 8 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
E0327 19:53:47.692044 8 main.go:124] "Searching IngressClass" err="ingressclasses.networking.k8s.io \"nginx\" is forbidden: User \"system:serviceaccount:ingress-nginx:ingress-nginx-serviceaccount\" cannot get resource \"ingressclasses\" in API group \"networking.k8s.io\" at the cluster scope" class="nginx"
W0327 19:53:47.692070 8 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0327 19:53:47.739577 8 nginx.go:254] "Starting NGINX Ingress controller"
I0327 19:53:47.755865 8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-tcp-services", UID:"6115a34f-4f95-4f99-970a-b65477e45808", APIVersion:"v1", ResourceVersion:"103400810", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-tcp-services
I0327 19:53:47.756010 8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-udp-services", UID:"fa04d653-a070-4934-a606-a60a7f98ad6a", APIVersion:"v1", ResourceVersion:"103400812", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-udp-services
I0327 19:53:47.756196 8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-nginx-configuration", UID:"3af77ed0-e71c-49e9-bac3-b7c3fada40df", APIVersion:"v1", ResourceVersion:"103400808", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-nginx-configuration
E0327 19:53:48.844980 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:53:50.385656 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:53:52.811461 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:53:57.052727 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:54:05.784219 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
I0327 19:54:26.690574 8 main.go:187] "Received SIGTERM, shutting down"
I0327 19:54:26.690593 8 nginx.go:372] "Shutting down controller queues"
E0327 19:54:26.690778 8 store.go:178] timed out waiting for caches to sync
I0327 19:54:26.690835 8 nginx.go:296] "Starting NGINX process"
I0327 19:54:26.691321 8 queue.go:78] "queue has been shutdown, failed to enqueue" key="&ObjectMeta{Name:initial-sync,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}"
I0327 19:54:26.691353 8 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0327 19:54:26.718477 8 status.go:84] "New leader elected" identity="ingress-nginx-controller-72b9j"
I0327 19:54:26.733451 8 nginx.go:388] "Stopping NGINX process"
2022/03/27 19:54:26 [notice] 28#28: signal process started
I0327 19:54:27.738884 8 nginx.go:401] "NGINX process has stopped"
I0327 19:54:27.738926 8 main.go:195] "Handled quit, awaiting Pod deletion"
I0327 19:54:37.739197 8 main.go:198] "Exiting" code=0
很高兴提供任何其他有用的详细信息。我非常感谢您的帮助！
编辑：
该集群位于 AWS 上，并使用以下 k0ps 命令创建。
kops create cluster --node-count 2 --node-size t2.medium --zones ap-southeast-2a,ap-southeast-2c --master-size t2.small --master-zones ap-southeast-2c --master-count 1 --networking=calico --authorization RBAC -o yaml --dry-run > my-cluster.yaml