![Kubernetes coredns 未收到请求](https://linux22.com/image/770686/Kubernetes%20coredns%20%E6%9C%AA%E6%94%B6%E5%88%B0%E8%AF%B7%E6%B1%82.png)
我已经设置了一个 kubernetes 集群,单节点,debian 11。但是,我的 CoreDNS 似乎无法解决任何问题。我注意到 portainer 无法加载资源。
http: proxy error: dial tcp: lookup kubernetes.default.svc on 10.96.0.10:53: read udp 10.244.0.4:57589->10.96.0.10:53: i/o timeout
由于我的 DNS 超时,我检查了服务:
root@dmvandenberg:~/kubernetes# kubectl get svc -n kube-system -o wide
NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR
kube-dns ClusterIP 10.96.0.10 <none> 53/UDP,53/TCP,9153/TCP 78m k8s-app=kube-dns
root@dmvandenberg:~/kubernetes# kubectl get pods --selector=k8s-app=kube-dns -o wide -n kube-system
NAME READY STATUS RESTARTS AGE IP NODE NOMINATED NODE READINESS GATES
coredns-78fcd69978-2b6cq 1/1 Running 0 79m 10.244.0.2 dmvandenberg.nl <none> <none>
coredns-78fcd69978-swprh 1/1 Running 0 79m 10.244.0.3 dmvandenberg.nl <none> <none>
我已经用以下文件设置了我的集群:
cat init.sh init2.sh
kubeadm init --pod-network-cidr=10.244.0.0/16 --ignore-preflight-errors=all
mkdir -p $HOME/.kube
sudo cp -i /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config
kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml
kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml
kubectl taint nodes --all node-role.kubernetes.io/master-
kubectl create -f localstorage.yml --save-config
kubectl create -f pvportainer.yml --save-config
kubectl patch storageclass local-storage -p '{"metadata": {"annotations":{"storageclass.kubernetes.io/is-default-class":"true"}}}'
kubectl apply -n portainer -f https://raw.githubusercontent.com/portainer/k8s/master/deploy/manifests/portainer/portainer.yaml
我也尝试过用kubectl apply -f https://github.com/coreos/flannel/raw/master/Documentation/kube-flannel.yml
而不是kubectl create -f https://docs.projectcalico.org/manifests/tigera-operator.yaml kubectl create -f https://docs.projectcalico.org/manifests/custom-resources.yaml
。
root@dmvandenberg:~/kubernetes# cat localstorage.yml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-storage
provisioner: kubernetes.io/no-provisioner
volumeBindingMode: WaitForFirstConsumer
root@dmvandenberg:~/kubernetes# cat pvportainer.yml
apiVersion: v1
kind: PersistentVolume
metadata:
name: portainer
spec:
capacity:
storage: 11Gi
accessModes:
- ReadWriteOnce
persistentVolumeReclaimPolicy: Retain
storageClassName: local-storage
local:
path: /dockerdirs/pvportainer
nodeAffinity:
required:
nodeSelectorTerms:
- matchExpressions:
- key: kubernetes.io/hostname
operator: In
values:
- dmvandenberg.nl
我使用以下命令和输出将问题缩小到 DNS 解析:
root@dmvandenberg:~/kubernetes# kubectl logs --namespace=kube-system -l k8s-app=kube-dns -f & tcpdump -ani cni0 udp port 53
[5] 9505
tcpdump: verbose output suppressed, use -v[v]... for full protocol decode
listening on cni0, link-type EN10MB (Ethernet), snapshot length 262144 bytes
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.4
linux/amd64, go1.16.4, 053c4d5
.:53
[INFO] plugin/reload: Running configuration MD5 = db32ca3650231d74073ff4cf814959a7
CoreDNS-1.8.4
linux/amd64, go1.16.4, 053c4d5
21:21:07.629395 IP 10.244.0.4.44224 > 10.244.0.2.53: 3488+ AAAA? kubernetes.default.svc.portainer.svc.cluster.local. (68)
21:21:07.629667 IP 10.244.0.4.43161 > 10.244.0.2.53: 433+ A? kubernetes.default.svc.portainer.svc.cluster.local. (68)
21:21:12.630395 IP 10.244.0.4.54508 > 10.244.0.3.53: 61466+ AAAA? kubernetes.default.svc.portainer.svc.cluster.local. (68)
21:21:12.630453 IP 10.244.0.4.46088 > 10.244.0.2.53: 55999+ A? kubernetes.default.svc.portainer.svc.cluster.local. (68)
^C
4 packets captured
4 packets received by filter
0 packets dropped by kernel
我本以为会看到对 DNS 查询的回复,但却没有看到任何回复。我在互联网上找到了有关向 coredns 的核心文件添加“log”的信息,所以我尝试了一下,但没有看到任何日志行出现。这让我相信,正如 tcpdump 所示,UDP 消息没有被 coredns 读取/接收。
我按照所有步骤进行了https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/,但这并没有让我取得任何进展。
但我在这之后陷入了困境。我该如何继续调试?可能出了什么问题?
编辑:我已尝试按照本指南操作:https://www.oueta.com/linux/create-a-debian-11-kubernetes-cluster-with-kubeadm/ 我在不同的界面上看到了完全相同的结果:
16:56:06.482769 cali6bd455d068f In IP 172.20.122.129.60650 > 10.96.0.10.53: 31215+ AAAA? kubernetes.default.svc.portainer.svc.cluster.local. (68)
16:56:06.482980 cali6bd455d068f In IP 172.20.122.129.35119 > 10.96.0.10.53: 8608+ A? kubernetes.default.svc.portainer.svc.cluster.local. (68)
16:56:11.483200 cali6bd455d068f In IP 172.20.122.129.57079 > 10.96.0.10.53: 61639+ AAAA? kubernetes.default.svc.portainer.svc.cluster.local. (68)
16:56:11.483309 cali6bd455d068f In IP 172.20.122.129.38249 > 10.96.0.10.53: 14976+ A? kubernetes.default.svc.portainer.svc.cluster.local. (68)
16:56:16.484367 cali6bd455d068f In IP 172.20.122.129.57768 > 10.96.0.10.53: 55396+ AAAA? kubernetes.default.svc.svc.cluster.local. (58)
16:56:16.484488 cali6bd455d068f In IP 172.20.122.129.53058 > 10.96.0.10.53: 50700+ A? kubernetes.default.svc.svc.cluster.local. (58)
16:56:21.484644 cali6bd455d068f In IP 172.20.122.129.58857 > 10.96.0.10.53: 18986+ AAAA? kubernetes.default.svc.svc.cluster.local. (58)
16:56:21.484702 cali6bd455d068f In IP 172.20.122.129.36861 > 10.96.0.10.53: 44020+ A? kubernetes.default.svc.svc.cluster.local. (58)
在整个接口上运行 tcpdump 表明,考虑到发回的 ACK 消息,TCP 似乎确实有效。我注意到的是,没有从 10.96.0.10(服务)到 pod 的流量,但我不知道这是否是必需的?
17:03:29.224602 IP 169.254.167.173.36088 > 172.20.122.129.9000: Flags [.], ack 1, win 169, options [nop,nop,TS val 4014670766 ecr 4073454542], length 0
17:03:29.224869 IP 169.254.167.173.36088 > 172.20.122.129.9000: Flags [P.], seq 1:107, ack 1, win 169, options [nop,nop,TS val 4014670766 ecr 4073454542], length 106
17:03:29.224887 IP 172.20.122.129.9000 > 169.254.167.173.36088: Flags [.], ack 107, win 167, options [nop,nop,TS val 4073454542 ecr 4014670766], length 0
17:03:29.225273 IP 172.20.122.129.9000 > 169.254.167.173.36088: Flags [P.], seq 1:818, ack 107, win 167, options [nop,nop,TS val 4073454543 ecr 4014670766], length 817
17:03:29.225341 IP 169.254.167.173.36088 > 172.20.122.129.9000: Flags [.], ack 818, win 166, options [nop,nop,TS val 4014670767 ecr 4073454543], length 0
17:03:29.225399 IP 172.20.122.129.9000 > 169.254.167.173.36088: Flags [.], seq 818:7958, ack 107, win 167, options [nop,nop,TS val 4073454543 ecr 4014670767], length 7140
17:03:29.225422 IP 169.254.167.173.36088 > 172.20.122.129.9000: Flags [.], ack 7958, win 155, options [nop,nop,TS val 4014670767 ecr 4073454543], length 0
17:03:29.225430 IP 172.20.122.129.9000 > 169.254.167.173.36088: Flags [.], seq 7958:15098, ack 107, win 167, options [nop,nop,TS val 4073454543 ecr 4014670767], length 7140
17:03:29.225448 IP 169.254.167.173.36088 > 172.20.122.129.9000: Flags [.], ack 15098, win 138, options [nop,nop,TS val 4014670767 ecr 4073454543], length 0
17:03:29.225457 IP 172.20.122.129.9000 > 169.254.167.173.36088: Flags [P.], seq 15098:23486, ack 107, win 167, options [nop,nop,TS val 4073454543 ecr 4014670767], length 8388
17:03:29.225474 IP 169.254.167.173.36088 > 172.20.122.129.9000: Flags [.], ack 23486, win 119, options [nop,nop,TS val 4014670767 ecr 4073454543], length 0
17:03:29.225564 IP 172.20.122.129.9000 > 169.254.167.173.36088: Flags [F.], seq 23486, ack 107, win 167, options [nop,nop,TS val 4073454543 ecr 4014670767], length 0
17:03:29.225609 IP 169.254.167.173.36088 > 172.20.122.129.9000: Flags [R.], seq 107, ack 23486, win 166, options [nop,nop,TS val 4014670767 ecr 4073454543], length 0
17:03:29.524333 IP 172.20.122.129.9000 > 169.254.167.173.9984: Flags [.], ack 3370092883, win 166, options [nop,nop,TS val 4073454842 ecr 1976747960], length 0
17:03:29.524564 IP 169.254.167.173.9984 > 172.20.122.129.9000: Flags [.], ack 1, win 171, options [nop,nop,TS val 1976763065 ecr 4073424519], length 0
17:03:34.218598 IP 172.20.122.129.45239 > 10.96.0.10.53: 23854+ AAAA? kubernetes.default.svc. (40)
17:03:34.219065 IP 172.20.122.129.38604 > 10.96.0.10.53: 24098+ A? kubernetes.default.svc. (40)
17:03:34.388311 IP 172.20.122.129.9000 > 169.254.167.173.7394: Flags [.], ack 917, win 166, options [nop,nop,TS val 4073459706 ecr 1976752753], length 0
17:03:34.388402 IP 169.254.167.173.7394 > 172.20.122.129.9000: Flags [.], ack 1, win 171, options [nop,nop,TS val 1976767929 ecr 4073444530], length 0
17:03:34.388314 IP 172.20.122.129.9000 > 169.254.167.173.3949: Flags [.], ack 917, win 166, options [nop,nop,TS val 4073459706 ecr 1976752753], length 0
17:03:34.388424 IP 169.254.167.173.3949 > 172.20.122.129.9000: Flags [.], ack 1, win 171, options [nop,nop,TS val 1976767929 ecr 4073444530], length 0
17:03:34.388288 IP 172.20.122.129.9000 > 169.254.167.173.26855: Flags [.], ack 917, win 166, options [nop,nop,TS val 4073459706 ecr 1976752752], length 0
17:03:34.388544 IP 169.254.167.173.26855 > 172.20.122.129.9000: Flags [.], ack 1, win 171, options [nop,nop,TS val 1976767929 ecr 4073444529], length 0
17:03:39.216823 IP 169.254.167.173.36182 > 172.20.122.129.9000: Flags [S], seq 2192346809, win 43200, options [mss 1440,sackOK,TS val 4014680758 ecr 0,nop,wscale 8], length 0
17:03:39.216889 IP 172.20.122.129.9000 > 169.254.167.173.36182: Flags [S.], seq 1678785660, ack 2192346810, win 42840, options [mss 1440,sackOK,TS val 4073464535 ecr 4014680758,nop,wscale 8]
, length 0