我们曾尝试在我们的 kubernetes 集群中设置指标服务器，但一直失败。
我有点不确定我哪里做错了。集群已在现有硬件上使用 kubeadm 设置和升级。我发现在很多操作过程中，kubernetes 尝试与 metrics-server 通信但失败了。
有谁经历过这种情况和/或可以帮助我找出原因吗？
以下是来自 metric-server 日志的一些输出：
I0201 09:20:32.016226 1 manager.go:150] ScrapeMetrics: time: 216.595261ms, nodes: 5, pods: 49
I0201 09:20:32.016257 1 manager.go:115] ...Storing metrics...
I0201 09:20:32.016319 1 manager.go:126] ...Cycle complete
E0201 09:20:32.596639 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:32.596839 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (615.212µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
E0201 09:20:32.636449 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:32.636590 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (460.541µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
I0201 09:20:37.552609 1 request.go:897] Request Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0201 09:20:37.552813 1 round_trippers.go:386] curl -k -v -XPOST -H "Accept: application/json, */*" -H "Content-Type: application/json" -H "User-Agent: metrics-server/v0.0.0 (linux/amd64) kubernetes/$Format" -H "Authorization: Bearer eyJhbGciOiJSUzI1NiIsImtpZCI6IiJ9.eyJpc3MiOiJrdWJlcm5ldGVzL3NlcnZpY2VhY2NvdW50Iiwia3ViZXJuZXRlcy5pby9zZXJ2aWNlYWNjb3VudC9uYW1lc3BhY2UiOiJrdWJlLXN5c3RlbSIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VjcmV0Lm5hbWUiOiJtZXRyaWNzLXNlcnZlci10b2tlbi10bTRmdCIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50Lm5hbWUiOiJtZXRyaWNzLXNlcnZlciIsImt1YmVybmV0ZXMuaW8vc2VydmljZWFjY291bnQvc2VydmljZS1hY2NvdW50LnVpZCI6ImMyN2VjNDAxLTI1NjMtMTFlOS04Y2M2LTFjYzFkZWVhNzdjOCIsInN1YiI6InN5c3RlbTpzZXJ2aWNlYWNjb3VudDprdWJlLXN5c3RlbTptZXRyaWNzLXNlcnZlciJ9.YF4IaGtM_IlRZ8Xzik3AEDnv6-Q4YQBamBjna_gLydhVehH4gmq_Y4y0Nrcqt4Ana9HwNcLx0jGV4GU-njUfzrb0uS9eKl2Eeh6bLTkwafKAv7cF8SwP0rBLuhIl6FDgwBU4d95MQAqOxvMdnlSquJmYOiuIT25OxD_wPJ2PYjdXbuxxSChvrLrtGwa5URbzNvN9deMWSugbz2B1knCu8YAlKPx31bUEa27YFCZIrtydRjY2E1Qzl8hkJiEuom8v_sRLTvnJyYcOU6ARWqwJT570JeubMO5_GcvnpVpmBmh8QFr8_BLTJJfiEleFNs9YmBgWIr3xDwjEBDmn5ndjrQ" 'https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews'
I0201 09:20:37.572204 1 round_trippers.go:405] POST https://10.96.0.1:443/apis/authorization.k8s.io/v1beta1/subjectaccessreviews 201 Created in 19 milliseconds
I0201 09:20:37.572235 1 round_trippers.go:411] Response Headers:
I0201 09:20:37.572245 1 round_trippers.go:414] Content-Type: application/json
I0201 09:20:37.572254 1 round_trippers.go:414] Content-Length: 260
I0201 09:20:37.572262 1 round_trippers.go:414] Date: Fri, 01 Feb 2019 09:20:37 GMT
I0201 09:20:37.572323 1 request.go:897] Response Body: {"kind":"SubjectAccessReview","apiVersion":"authorization.k8s.io/v1beta1","metadata":{"creationTimestamp":null},"spec":{"nonResourceAttributes":{"path":"/","verb":"get"},"user":"system:anonymous","group":["system:unauthenticated"]},"status":{"allowed":false}}
I0201 09:20:37.572465 1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:37.572580 1 wrap.go:42] GET /: (20.227877ms) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
I0201 09:20:39.404760 1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:39.404908 1 wrap.go:42] GET /: (321.809µs) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
I0201 09:20:39.451089 1 authorization.go:73] Forbidden: "/", Reason: ""
I0201 09:20:39.451212 1 wrap.go:42] GET /: (283.995µs) 403 [[Go-http-client/2.0] 10.46.0.0:44198]
E0201 09:20:40.708131 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:40.708327 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (544.441µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/controller-discovery] 10.46.0.0:44210]
E0201 09:20:40.955975 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:40.956151 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (574.914µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/system:serviceaccount:kube-system:generic-garbage-collector] 10.46.0.0:44210]
E0201 09:20:41.785405 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:41.785570 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (579.992µs) 401 [[kube-controller-manager/v1.13.2 (linux/amd64) kubernetes/cff46ab/system:serviceaccount:kube-system:generic-garbage-collector] 10.46.0.0:44210]
E0201 09:20:42.065074 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:42.065248 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (566.86µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
E0201 09:20:42.305102 1 authentication.go:62] Unable to authenticate the request due to an error: [x509: certificate signed by unknown authority, x509: certificate signed by unknown authority (possibly because of "crypto/rsa: verification error" while trying to verify candidate authority certificate "kubernetes")]
I0201 09:20:42.305272 1 wrap.go:42] GET /apis/metrics.k8s.io/v1beta1?timeout=32s: (552.597µs) 401 [[kubectl/v1.13.2 (linux/amd64) kubernetes/cff46ab] 10.46.0.0:44210]
这是来自 kube-apiserver 日志的内容：
I0201 09:22:14.652152 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:22:19.688846 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:22:49.751772 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:23:19.816917 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:23:49.896396 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
I0201 09:24:14.314774 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0201 09:24:14.317317 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 401, Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
, Header: map[Content-Length:[129] Date:[Fri, 01 Feb 2019 09:24:14 GMT] Content-Type:[application/json]]
I0201 09:24:14.317368 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:24:19.960927 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:24:50.037553 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
I0201 09:25:14.317811 1 controller.go:105] OpenAPI AggregationController: Processing item v1beta1.metrics.k8s.io
E0201 09:25:14.320556 1 controller.go:111] loading OpenAPI spec for "v1beta1.metrics.k8s.io" failed with: failed to retrieve openAPI spec, http error: ResponseCode: 401, Body: {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"Unauthorized","reason":"Unauthorized","code":401}
, Header: map[Content-Length:[129] Date:[Fri, 01 Feb 2019 09:25:14 GMT] Content-Type:[application/json]]
I0201 09:25:14.320623 1 controller.go:119] OpenAPI AggregationController: action for item v1beta1.metrics.k8s.io: Rate Limited Requeue.
E0201 09:25:20.110375 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
E0201 09:25:50.172368 1 memcache.go:135] couldn't get resource list for metrics.k8s.io/v1beta1: Unauthorized
答案1
继https://github.com/kubernetes-incubator/metrics-server/issues/67， https://github.com/kubernetes-incubator/metrics-server/issues/146和https://github.com/kubernetes-incubator/metrics-server/issues/131您可能需要尝试使用下一个解决方案：
对于未来感到困惑的读者来说：在使用 kubeadm 部署的 Kubernetes 1.13 集群上，一旦我使用以下内容更新部署规范，指标服务器就开始工作：
command:
- /metrics-server
- --kubelet-insecure-tls
- --kubelet-preferred-address-types=InternalIP
（此后，需要等待几分钟，kubectl top 才真正有足够的数据来显示任何内容。）
或者至少尝试修改 metrics-server 部署为
command:
- /metrics-server
- --kubelet-insecure-tls
答案2
此问题已报告于https://github.com/kubernetes/kubernetes/issues/69277并进一步讨论了https://github.com/kubernetes/kubernetes/issues/61879。如上所述，多主服务器设置应在外部生成 ca crt/key 文件并将其设置在 /etc/kubernetes/pki/* 文件夹中，以便 Kubeadm 可以使用 ca 文件颁发服务器证书和客户端证书。希望这对您有所帮助。