Prometheus 未在加载 WAL 段的情况下启动

Prometheus 未在加载 WAL 段的情况下启动

Kubernetes Azure AKS 中安装了 Prometheus,默认存储为 StandardSSD。Prometheus 版本 v2.47.0

卷存储大小为 1 Gb。数据保留配置为 1 小时。

该卷上的最大数据大小约为 500Mb。

Prometheus 无法启动。

看起来容器正在被 Kubernetes 重新启动,因为 Prometheus 没有上线。

有没有办法不加载 WAL?只需清理数据并启动服务器?

Prometheus 日志

ts=2023-09-26T12:18:22.504Z caller=main.go:583 level=info msg="Starting Prometheus Server" mode=server version="(version=2.47.0, branch=HEAD, revision=efa34a5840661c29c2e362efa76bc3a70dccb335)"
ts=2023-09-26T12:18:22.504Z caller=main.go:588 level=info build_context="(go=go1.21.0, platform=linux/amd64, user=root@409eb5e6b30c, date=20230906-10:20:07, tags=netgo,builtinassets,stringlabels)"
ts=2023-09-26T12:18:22.504Z caller=main.go:589 level=info host_details="(Linux 6.2.0-1012-azure #12~22.04.1-Ubuntu SMP Thu Sep  7 14:07:14 UTC 2023 x86_64 prometheus-server-c6dc89b74-sm4qn (none))"
ts=2023-09-26T12:18:22.505Z caller=main.go:590 level=info fd_limits="(soft=1048576, hard=1048576)"
ts=2023-09-26T12:18:22.505Z caller=main.go:591 level=info vm_limits="(soft=unlimited, hard=unlimited)"
ts=2023-09-26T12:18:22.507Z caller=web.go:566 level=info component=web msg="Start listening for connections" address=0.0.0.0:9090
ts=2023-09-26T12:18:22.508Z caller=main.go:1024 level=info msg="Starting TSDB ..."
ts=2023-09-26T12:18:22.508Z caller=repair.go:56 level=info component=tsdb msg="Found healthy block" mint=1695074402397 maxt=1695081600000 ulid=01HANF73XBWEF5B6WXZ81FHC74
ts=2023-09-26T12:18:22.509Z caller=tls_config.go:274 level=info component=web msg="Listening on" address=[::]:9090
ts=2023-09-26T12:18:22.509Z caller=tls_config.go:277 level=info component=web msg="TLS is disabled." http2=false address=[::]:9090
ts=2023-09-26T12:18:22.542Z caller=dir_locker.go:77 level=warn component=tsdb msg="A lockfile from a previous execution already existed. It was replaced" file=/data/lock
ts=2023-09-26T12:18:22.597Z caller=head.go:600 level=info component=tsdb msg="Replaying on-disk memory mappable chunks if any"
ts=2023-09-26T12:18:22.755Z caller=head.go:681 level=info component=tsdb msg="On-disk memory mappable chunks replay completed" duration=157.95156ms
ts=2023-09-26T12:18:22.755Z caller=head.go:689 level=info component=tsdb msg="Replaying WAL, this may take a while"
ts=2023-09-26T12:18:25.156Z caller=head.go:725 level=info component=tsdb msg="WAL checkpoint loaded"
ts=2023-09-26T12:08:41.845Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=99 maxSegment=1752
ts=2023-09-26T12:08:41.845Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=100 maxSegment=1752
ts=2023-09-26T12:08:41.846Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=101 maxSegment=1752
ts=2023-09-26T12:08:41.846Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=102 maxSegment=1752
ts=2023-09-26T12:08:41.848Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=103 maxSegment=1752
ts=2023-09-26T12:08:41.848Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=104 maxSegment=1752
ts=2023-09-26T12:08:41.848Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=105 maxSegment=1752
ts=2023-09-26T12:08:41.852Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=106 maxSegment=1752
ts=2023-09-26T12:08:41.853Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=107 maxSegment=1752
ts=2023-09-26T12:08:41.854Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=108 maxSegment=1752
ts=2023-09-26T12:08:41.854Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=109 maxSegment=1752
ts=2023-09-26T12:08:41.856Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=110 maxSegment=1752
ts=2023-09-26T12:08:41.856Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=111 maxSegment=1752
ts=2023-09-26T12:08:41.856Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=112 maxSegment=1752
ts=2023-09-26T12:08:41.858Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=113 maxSegment=1752
ts=2023-09-26T12:08:41.859Z caller=head.go:760 level=info component=tsdb msg="WAL segment loaded" segment=114 maxSegment=1752

相关内容