我有一个 AKS 集群，正在其中运行基于 Python-Django 的测试 Web 应用程序。我还配置了 Grafana 和 Prometheus。我需要使用 OpenTelemetry 从测试应用程序获取指标数据并在 Grafana 中将其可视化。
我有一个使用 OTEL CRD 以 Kubernetes 操作员形式部署的 OpenTelemetry Collector（参考：https://opentelemetry.io/docs/kubernetes/operator/automatic/）。
我已经部署了 OpenTelemetry CRD 并部署了收集器。收集器的类型为OpenTelemetryCollector。我已将导出器配置为Promethus，OTEL 收集器正在为 Prometheus 提供服务，并且它们全部已连接。我能够在 Grafana 中查看基于 OTEL 收集器的面板。
我的目标是使用 OTEL 收集器在 Grafana 中获取基于应用程序的指标，例如pod_cpu_usage、等。pod_memory_usage
以下是我的 OTEL Collector YAML 配置：
otel-collector.yml
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: otel
namespace: otel
labels:
opentelemetry.io/name: otelcol
app.kubernetes.io/instance: opentelemetry
app.kubernetes.io/name: otelcol
app.kubernetes.io/version: "0.82.0"
annotations:
otel_collector: "true"
prometheus.io/port: "9464"
prometheus.io/scrape: "true"
spec:
config: |
receivers:
prometheus:
config:
scrape_configs:
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: integrations/kubernetes/cadvisor
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc.cluster.local:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$${1}/proxy/metrics/cadvisor
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
server_name: kubernetes
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
- bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
job_name: integrations/kubernetes/kubelet
kubernetes_sd_configs:
- role: node
relabel_configs:
- replacement: kubernetes.default.svc.cluster.local:443
target_label: __address__
- regex: (.+)
replacement: /api/v1/nodes/$${1}/proxy/metrics
source_labels:
- __meta_kubernetes_node_name
target_label: __metrics_path__
scheme: https
tls_config:
ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
insecure_skip_verify: false
server_name: kubernetes
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
- job_name: integrations/kubernetes/kube-state-metrics
kubernetes_sd_configs:
- role: pod
relabel_configs:
- action: keep
regex: kube-state-metrics
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
- job_name: integrations/node_exporter
kubernetes_sd_configs:
- namespaces:
names:
- default
role: pod
relabel_configs:
- action: keep
regex: prometheus-node-exporter.*
source_labels:
- __meta_kubernetes_pod_label_app_kubernetes_io_name
- action: replace
source_labels:
- __meta_kubernetes_pod_node_name
target_label: instance
- action: replace
source_labels:
- __meta_kubernetes_namespace
target_label: namespace
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
- job_name: 'otel-collector'
scrape_interval: 10s
static_configs:
- targets: [ '0.0.0.0:8888' ]
otlp:
protocols:
grpc:
endpoint: '0.0.0.0:4317'
http:
cors:
allowed_origins:
- http://*
- https://*
endpoint: '0.0.0.0:4318'
processors:
batch: {}
filter/ottl:
error_mode: ignore
metrics:
metric:
- name == "rpc.server.duration"
memory_limiter:
check_interval: 5s
limit_percentage: 80
spike_limit_percentage: 25
resource:
attributes:
- action: insert
from_attribute: k8s.pod.uid
key: service.instance.id
exporters:
logging: {}
prometheus:
enable_open_metrics: true
endpoint: '0.0.0.0:9464'
resource_to_telemetry_conversion:
enabled: true
extensions:
health_check: {}
memory_ballast:
size_in_percentage: 40
service:
pipelines:
metrics:
receivers: [prometheus, otlp]
processors: [batch, filter/ottl, memory_limiter, resource]
exporters: [prometheus, logging]
以下是当前的 Instrumentation 配置 YAML（参考：https://github.com/open-telemetry/opentelemetry-operator#deployment-modes） 我有：
otel-instrumentation.yml
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: python-instrumentation
namespace: trojanwall
spec:
exporter:
endpoint: http://10.244.0.7:4318
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: "1"
python:
env:
- name: OTEL_LOGS_EXPORTER
value: otlp_proto_http
- name: OTEL_PYTHON_LOGGING_AUTO_INSTRUMENTATION_ENABLED
value: 'true'
我需要为 Python-Django 应用程序实现自动检测，而无需修改任何应用程序源代码。
此后，我在 Python-Django 测试应用程序部署 YAML 文件中添加了以下注释：
annotations:
instrumentation.opentelemetry.io/inject-python: "true"
但是，我的应用程序特定指标均未在 Grafana 中显示。
如何使用 OpenTelemetry Collector 和 Auto-Instrumentation 正确执行此操作？
有人可以提供关于如何正确执行此操作的见解吗？