我有一个包含 1 个控制面板和 2 个节点的集群。
Istio 作为 Service Mesh 安装。
我确实通过 istio ingress 请求管理。
我希望它通过在 Kubernetes HPA 和 istio prometheus 之间共享指标来自动扩展，但我做不到。
kube-system 上的我的 pod
root@ubuntu-master:~# kubectl get pods -n kube-system
NAME READY STATUS RESTARTS AGE
coredns-78fcd69978-pk69f 1/1 Running 1 (2d11h ago) 2d12h
coredns-78fcd69978-t5dkx 1/1 Running 1 (2d11h ago) 2d12h
etcd-ubuntu-master 1/1 Running 1 (2d11h ago) 48d
kube-apiserver-ubuntu-master 1/1 Running 2 (2d11h ago) 48d
kube-controller-manager-ubuntu-master 1/1 Running 3 (2d11h ago) 48d
kube-proxy-72q2r 1/1 Running 0 2d10h
kube-proxy-8qgr9 1/1 Running 1 (2d11h ago) 48d
kube-proxy-t4wgr 1/1 Running 0 2d10h
kube-scheduler-ubuntu-master 1/1 Running 3 (2d11h ago) 48d
metrics-server-84b4bfc7fb-h8gd2 1/1 Running 0 18h
我在 istio-system 上的 pod
root@ubuntu-master:~# kubectl get pods -n istio-system
NAME READY STATUS RESTARTS AGE
grafana-6ccd56f4b6-bw5md 1/1 Running 0 2d12h
istio-ingressgateway-57c665985b-wj5gr 1/1 Running 0 2d12h
istiod-78cc776c5b-qkr6b 1/1 Running 0 2d12h
jaeger-5d44bc5c5d-db2pj 1/1 Running 0 2d12h
kiali-79b86ff5bc-mj8bn 1/1 Running 0 2d12h
prometheus-64fd8ccd65-22znf 2/2 Running 0 2d12h
prometheus-adapter-6d9c6c8fdf-lxfbp 1/1 Running 0 17h
Prometheus UI 结果：
指标服务器响应；
root@ubuntu-master:~# kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1"
{"kind":"APIResourceList","apiVersion":"v1","groupVersion":"custom.metrics.k8s.io/v1beta1","resources":[{"name":"namespaces/network_transmit_packets_dropped","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/processes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/sockets","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/spec_memory_swap_limit_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"jobs.batch/kiali_single_validation_processing_duration_seconds_sum","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"jobs.batch/kiali_validation_processing_duration_seconds_count","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/kubelet_container_log_filesystem_used_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/cpu_system","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"pods/cpu_user","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},{"name":"namespaces/fs_reads_bytes","singularName":"","namespaced":false,"kind":"MetricValueList","verbs":["get"]},{"name":"services/kiali_validation_processing_duration_seconds_sum","singularName":"","namespaced":true,"kind":"MetricValueList","verbs":["get"]},......
这是我的 HPA 定义
root@ubuntu-master:~# kubectl describe hpa html-pdf-v2-hpa -n html-pdf-v2
Name: html-pdf-v2-hpa
Namespace: html-pdf-v2
Labels: <none>
Annotations: metric-config.object.istio-requests-total.prometheus/query:
sum(
rate(
istio_requests_total{
destination_workload="html-pdf-deploy",
destination_workload_namespace="html-pdf-v2"
}[1m]
)
) /
count(
count(
container_memory_usage_bytes{
namespace="html-pdf-v2",
pod=~"html-pdf-deploy.*"
}
) by (pod)
)
CreationTimestamp: Wed, 19 Jan 2022 18:11:14 +0300
Reference: Deployment/html-pdf-deploy
Metrics: ( current / target )
"istio-requests-total" on Pod/html-pdf-deploy (target value): <unknown> / 10
Min replicas: 1
Max replicas: 10
Deployment pods: 5 current / 0 desired
Conditions:
Type Status Reason Message
---- ------ ------ -------
AbleToScale True SucceededGetScale the HPA controller was able to get the target's current scale
ScalingActive False FailedGetObjectMetric the HPA was unable to compute the replica count: unable to get metric istio-requests-total: Pod on html-pdf-v2 html-pdf-deploy/unable to fetch metrics from custom metrics API: the server could not find the metric istio-requests-total for pods
Events:
Type Reason Age From Message
---- ------ ---- ---- -------
Warning FailedGetObjectMetric 55s (x4255 over 17h) horizontal-pod-autoscaler unable to get metric istio-requests-total: Pod on html-pdf-v2 html-pdf-deploy/unable to fetch metrics from custom metrics API: the server could not find the metric istio-requests-total for pods
kubectl top pods 结果
root@ubuntu-master:~# kubectl top pods -n istio-system
NAME CPU(cores) MEMORY(bytes)
grafana-6ccd56f4b6-bw5md 2m 39Mi
istio-ingressgateway-57c665985b-wj5gr 10m 85Mi
istiod-78cc776c5b-qkr6b 7m 61Mi
jaeger-5d44bc5c5d-db2pj 3m 809Mi
kiali-79b86ff5bc-mj8bn 4m 1125Mi
prometheus-64fd8ccd65-22znf 34m 744Mi
prometheus-adapter-6d9c6c8fdf-lxfbp 40m 76Mi
HPA Yaml。
apiVersion: autoscaling/v2beta1
kind: HorizontalPodAutoscaler
metadata:
name: html-pdf-v2-hpa
namespace: html-pdf-v2
annotations:
metric-config.object.istio-requests-total.prometheus/query: |
sum(
rate(
istio_requests_total{
destination_workload="html-pdf-deploy",
destination_workload_namespace="html-pdf-v2"
}[1m]
)
) /
count(
count(
container_memory_usage_bytes{
namespace="html-pdf-v2",
pod=~"html-pdf-deploy.*"
}
) by (pod)
)
spec:
maxReplicas: 10
minReplicas: 1
scaleTargetRef:
apiVersion: apps/v1
kind: Deployment
name: html-pdf-deploy
metrics:
- type: Object
object:
metricName: istio-requests-total
target:
apiVersion: v1
kind: Pod
name: html-pdf-deploy
targetValue: 10
我担心自己哪里做错了或者我是否走在正确的道路上。
第一次发帖，我很期待得到答案。希望我解释得正确。
谢谢
答案1
@Orgenus 你找到解决方案了吗？我遇到了和你描述的情况一样的情况……我意识到，从 Prometheus 中我看到的指标是 istio_requests_total，而如果我运行“kubectl get --raw "/apis/custom.metrics.k8s.io/v1beta1/" | jq | grep pods/istio”，我会得到一个没有总数的“istio_requests”