☰

在 conjure-up k8s cdk 之后 coredns 正在运行但尚未准备就绪

tag-icon • tag-icon juju kubernetes
在 conjure-up k8s cdk 之后 coredns 正在运行但尚未准备就绪

V1.18.2我已经使用 conjure-up（使用 bionic）部署了 Kubernetes （CDK）

coredns通过 解决，/etc/resolv.conf如下所示configmap：

Name:         coredns
Namespace:    kube-system
Labels:       cdk-addons=true
Annotations:  
Data
====
Corefile:
----
.:53 {
    errors
    health {
      lameduck 5s
    }
    ready
    kubernetes cluster.local in-addr.arpa ip6.arpa {
      fallthrough in-addr.arpa ip6.arpa
    }
    prometheus :9153
    forward . /etc/resolv.conf
    cache 30
    loop
    reload
    loadbalance
}

Events:  <none>

这里有一个已知问题https://kubernetes.io/docs/tasks/administer-cluster/dns-debugging-resolution/#known-issues关于/etc/resolv.conf而不是/run/systemd/resolve/resolv.conf

我编辑了coredns配置图以指向它/run/systemd/resolve/resolv.conf，但设置被恢复了。

我也尝试设置kubelet-extra-config为{resolvConf: /run/systemd/resolve/resolv.conf}，重新启动了服务器，没有变化：

kubelet-extra-config:
    default: '{}'
    description: |
      Extra configuration to be passed to kubelet. Any values specified in this
      config will be merged into a KubeletConfiguration file that is passed to
      the kubelet service via the --config flag. This can be used to override
      values provided by the charm.
      Requires Kubernetes 1.10+.
      The value for this config must be a YAML mapping that can be safely
      merged with a KubeletConfiguration file. For example:
        {evictionHard: {memory.available: 200Mi}}
      For more information about KubeletConfiguration, see upstream docs:
      https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/
    source: user
    type: string
    value: '{resolvConf: /run/systemd/resolve/resolv.conf}'

kubelet但我可以在检查配置时看到配置的变化https://kubernetes.io/docs/tasks/administer-cluster/reconfigure-kubelet/

...
"resolvConf": "/run/systemd/resolve/resolv.conf",
...

这是我在 coredns pod 中遇到的错误：

E0429 09:16:42.172959       1 reflector.go:153] pkg/mod/k8s.io/[email protected]/tools/cache/reflector.go:105: Failed to list *v1.Endpoints: Get https://10.152.183.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.152.183.1:443: i/o timeout
[INFO] plugin/ready: Still waiting on: "kubernetes"

查看 kubernetes 服务：

default                           kubernetes                               ClusterIP   10.152.183.1     <none>        443/TCP                  4h42m   <none>

部署如下coredns：

Name:                   coredns
Namespace:              kube-system
CreationTimestamp:      Wed, 29 Apr 2020 09:15:07 +0000
Labels:                 cdk-addons=true
                        cdk-restart-on-ca-change=true
                        k8s-app=kube-dns
                        kubernetes.io/name=CoreDNS
Annotations:            deployment.kubernetes.io/revision: 1
Selector:               k8s-app=kube-dns
Replicas:               1 desired | 1 updated | 1 total | 0 available | 1 unavailable
StrategyType:           RollingUpdate
MinReadySeconds:        0
RollingUpdateStrategy:  1 max unavailable, 25% max surge
Pod Template:
  Labels:           k8s-app=kube-dns
  Service Account:  coredns
  Containers:
   coredns:
    Image:       rocks.canonical.com:443/cdk/coredns/coredns-amd64:1.6.7
    Ports:       53/UDP, 53/TCP, 9153/TCP
    Host Ports:  0/UDP, 0/TCP, 0/TCP
    Args:
      -conf
      /etc/coredns/Corefile
    Limits:
      memory:  170Mi
    Requests:
      cpu:        100m
      memory:     70Mi
    Liveness:     http-get http://:8080/health delay=60s timeout=5s period=10s #success=1 #failure=5
    Readiness:    http-get http://:8181/ready delay=0s timeout=1s period=10s #success=1 #failure=3
    Environment:  <none>
    Mounts:
      /etc/coredns from config-volume (ro)
  Volumes:
   config-volume:
    Type:               ConfigMap (a volume populated by a ConfigMap)
    Name:               coredns
    Optional:           false
  Priority Class Name:  system-cluster-critical
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    False   ProgressDeadlineExceeded
OldReplicaSets:  <none>
NewReplicaSet:   coredns-6b59b8bd9f (1/1 replicas created)
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  11m   deployment-controller  Scaled up replica set coredns-6b59b8bd9f to 1

有人可以帮忙吗？

答案1

对我来说，解决方案是设置一些 iptables 规则，然后再次启用 dns，所以我的错误非常相似：

E0303 11:36:04.387981       1 reflector.go:134] pkg/mod/k8s.io/[email protected]+incompatible/tools/cache/reflector.go:95: Failed to list *v1.Endpoints: Get https://10.152.183.1:443/api/v1/endpoints?limit=500&resourceVersion=0: dial tcp 10.152.183.1:443: i/o timeout
2021-03-03T11:36:10.799Z [INFO] plugin/ready: Still waiting on: "kubernetes"

iptables 规则：

sudo iptables -D  INPUT -j REJECT --reject-with icmp-host-prohibited
sudo iptables -D  FORWARD -j REJECT --reject-with icmp-host-prohibited

重新启用 kubedns 后我有一个健康的日志：

kubectl logs --namespace=kube-system -l k8s-app=kube-dns
.:53
2021-03-03T11:37:14.709Z [INFO] plugin/reload: Running configuration MD5 = 5d839962c224ea2e9fb32222b6a237d1
2021-03-03T11:37:14.709Z [INFO] CoreDNS-1.5.0
2021-03-03T11:37:14.710Z [INFO] linux/amd64, go1.12.2, e3f9a80
CoreDNS-1.5.0
linux/amd64, go1.12.2, e3f9a80

相关内容