![Rsyslog 输出到 Kafka 超时](https://linux22.com/image/790438/Rsyslog%20%E8%BE%93%E5%87%BA%E5%88%B0%20Kafka%20%E8%B6%85%E6%97%B6.png)
我在使用 omkafka 模块设置 Rsyslog 输出到 Kafka 时遇到了麻烦。我有一个在生产环境中运行的强大的 Kafka 集群。在当前工作流程中,Rsyslog 在 UDP:514 上接收事件并将其写入文件。Filebeat 读取多个文件、进行预处理并发送到 Kafka 中的不同主题。此过程按预期工作,但我想简化并让 Rsyslog 直接发送到 Kafka。我尝试在现有环境中的测试设备上进行几次尝试,但没有成功。为了简化,我创建了一个全新的 VM,其中默认安装了 OS、Kafka 和 Rsyslog,并且无法在该 VM 上发送 Rsyslog -> Kafka。我确信我遗漏了一些简单的东西,但我就是看不到它。任何帮助都将不胜感激。
以下是一些其他详细信息:
==== 操作系统和应用程序版本 ====
cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.8 (Ootpa)"
java -version
openjdk version "11.0.21" 2023-10-17 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.21.0.9-1) (build 11.0.21+9-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.21.0.9-1) (build 11.0.21+9-LTS, mixed mode, sharing)
/kafka/bin/kafka-topics.sh --version
3.6.0
rsyslogd -v
rsyslogd 8.2310.0 (aka 2023.10) compiled with:
PLATFORM: x86_64-redhat-linux-gnu
PLATFORM (lsb_release -d):
FEATURE_REGEXP: Yes
GSSAPI Kerberos 5 support: Yes
FEATURE_DEBUG (debug build, slow code): No
32bit Atomic operations supported: Yes
64bit Atomic operations supported: Yes
memory allocator: system default
Runtime Instrumentation (slow code): No
uuid support: Yes
systemd support: Yes
Config file: /etc/rsyslog.conf
PID file: /var/run/syslogd.pid
Number of Bits in RainerScript integers: 64
====卡夫卡====
Kafka 安装使用了此处 KRaft 版本快速入门说明中的 100% 默认设置:Apache Kafka 快速入门。Kafka 正在监听 localhost:9092。
$ netstat -ntlp | grep 9092
tcp6 0 0 :::9092 :::* LISTEN 7187/java
我已经测试过,可以使用内置的生产者和消费者脚本发送和接收事件,如下所示:
$ /kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test
Created topic test.
$ /kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a kafka test sent from the kafka console producer
$ /kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a kafka test sent from the kafka console producer
==== RSYSLOG ====
Rsyslog 安装使用默认设置。我安装了一些我认为 Rsyslog omkafka 模块使用的依赖项:
$ yum list installed | grep kafka
librdkafka.x86_64 0.11.4-3.el8 @rhel-8-for-x86_64-appstream-rpms
rsyslog-kafka.x86_64 8.2310.0-1.el8 @rsyslog_v8
我使用以下基本配置将测试事件发送到 Kafka。它还会将它们写入文件以确认它们已正确生成。
# /etc/rsyslog.d/kafka-sender.conf
module(load="omkafka")
if $msg contains "kafka-test" then {
action(
type="omkafka"
broker=["localhost:9092"]
topic="test"
errorFile="/var/log/remote-hosts/errors_kafka.log"
)
action(type="omfile" file="/var/log/remote-hosts/kafka.log")
}
我现在正在生成测试事件。事件按预期记录到 kafka.log 中,这意味着 Rsyslog 正在拾取它们并正确过滤,但 Kafka 上没有收到任何事件。
$ logger
kafka-test This is a test message to send from rsyslog to kafka
kafka-test This is a test message to send from rsyslog to kafka 2
$ cat /var/log/remote-hosts/kafka.log
Oct 30 05:55:20 localhost anthony[76955]: kafka-test This is a test message to send from rsyslog to kafka
Oct 30 05:55:23 localhost anthony[76955]: kafka-test This is a test message to send from rsyslog to kafka 2
这些事件似乎已排队,直到我重新启动 rsyslog 才会写入错误日志:
$ cat /var/log/remote-hosts/errors_kafka.log
{ "errcode": -152, "errmsg": "Local: Purged in queue", "data": "2023-10-30T05:55:20.506080-04:00 localhost anthony[76955]: kafka-test This is a test message to send from rsyslog to kafka\n" }
{ "errcode": -152, "errmsg": "Local: Purged in queue", "data": "2023-10-30T05:55:23.275050-04:00 localhost anthony[76955]: kafka-test This is a test message to send from rsyslog to kafka 2\n" }
我检查了 rsyslog 日志,发现有一些错误,但在网上找不到关于这些错误的太多信息。显然无法连接到 Kafka。
Oct 30 05:54:43 localhost.localdomain rsyslogd[76655]: imjournal: journal files changed, reloading... [v8.2310.0 try https://www.rsyslog.com/e/0 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: omkafka: action will suspended due to kafka error -195: Local: Broker transport failure [v8.2310.0 try https://www.rsyslog.com/e/2422 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: omkafka: action will suspended due to kafka error -187: Local: All broker connections are down [v8.2310.0 try https://www.rsyslog.com/e/2422 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: omkafka: action will suspended due to kafka error -195: Local: Broker transport failure [v8.2310.0 try https://www.rsyslog.com/e/2422 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: omkafka: action will suspended due to kafka error -195: Local: Broker transport failure [v8.2310.0 try https://www.rsyslog.com/e/2422 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: action 'action-0-omkafka' suspended (module 'omkafka'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2310.0 try https://www.rsyslog.com/e/2007 ]
我没有主意。有人有建议吗?如果需要,很乐意提供更多信息。
提前致谢。
-安东尼