Rsyslog 输出到 Kafka 超时

Question

我在使用 omkafka 模块设置 Rsyslog 输出到 Kafka 时遇到了麻烦。我有一个在生产环境中运行的强大的 Kafka 集群。在当前工作流程中，Rsyslog 在 UDP:514 上接收事件并将其写入文件。Filebeat 读取多个文件、进行预处理并发送到 Kafka 中的不同主题。此过程按预期工作，但我想简化并让 Rsyslog 直接发送到 Kafka。我尝试在现有环境中的测试设备上进行几次尝试，但没有成功。为了简化，我创建了一个全新的 VM，其中默认安装了 OS、Kafka 和 Rsyslog，并且无法在该 VM 上发送 Rsyslog -> Kafka。我确信我遗漏了一些简单的东西，但我就是看不到它。任何帮助都将不胜感激。

以下是一些其他详细信息：

==== 操作系统和应用程序版本 ====

cat /etc/os-release
NAME="Red Hat Enterprise Linux"
VERSION="8.8 (Ootpa)"

java -version
openjdk version "11.0.21" 2023-10-17 LTS
OpenJDK Runtime Environment (Red_Hat-11.0.21.0.9-1) (build 11.0.21+9-LTS)
OpenJDK 64-Bit Server VM (Red_Hat-11.0.21.0.9-1) (build 11.0.21+9-LTS, mixed mode, sharing)

/kafka/bin/kafka-topics.sh --version
3.6.0

rsyslogd -v
rsyslogd  8.2310.0 (aka 2023.10) compiled with:
        PLATFORM:                               x86_64-redhat-linux-gnu
        PLATFORM (lsb_release -d):
        FEATURE_REGEXP:                         Yes
        GSSAPI Kerberos 5 support:              Yes
        FEATURE_DEBUG (debug build, slow code): No
        32bit Atomic operations supported:      Yes
        64bit Atomic operations supported:      Yes
        memory allocator:                       system default
        Runtime Instrumentation (slow code):    No
        uuid support:                           Yes
        systemd support:                        Yes
        Config file:                            /etc/rsyslog.conf
        PID file:                               /var/run/syslogd.pid
        Number of Bits in RainerScript integers: 64

====卡夫卡====

Kafka 安装使用了此处 KRaft 版本快速入门说明中的 100% 默认设置：Apache Kafka 快速入门。Kafka 正在监听 localhost:9092。

$ netstat -ntlp | grep 9092
tcp6       0      0 :::9092                 :::*                    LISTEN      7187/java

我已经测试过，可以使用内置的生产者和消费者脚本发送和接收事件，如下所示：

$ /kafka/bin/kafka-topics.sh --bootstrap-server localhost:9092 --create --topic test
Created topic test.

$ /kafka/bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test
This is a kafka test sent from the kafka console producer

$ /kafka/bin/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning
This is a kafka test sent from the kafka console producer

==== RSYSLOG ====

Rsyslog 安装使用默认设置。我安装了一些我认为 Rsyslog omkafka 模块使用的依赖项：

$ yum list installed | grep kafka
librdkafka.x86_64                             0.11.4-3.el8                              @rhel-8-for-x86_64-appstream-rpms
rsyslog-kafka.x86_64                          8.2310.0-1.el8                            @rsyslog_v8

我使用以下基本配置将测试事件发送到 Kafka。它还会将它们写入文件以确认它们已正确生成。

# /etc/rsyslog.d/kafka-sender.conf

module(load="omkafka") 

if $msg contains "kafka-test" then {

    action(
            type="omkafka"
            broker=["localhost:9092"]
            topic="test"
            errorFile="/var/log/remote-hosts/errors_kafka.log"
      )

    action(type="omfile" file="/var/log/remote-hosts/kafka.log")
}

我现在正在生成测试事件。事件按预期记录到 kafka.log 中，这意味着 Rsyslog 正在拾取它们并正确过滤，但 Kafka 上没有收到任何事件。

$ logger
kafka-test This is a test message to send from rsyslog to kafka
kafka-test This is a test message to send from rsyslog to kafka 2

$ cat /var/log/remote-hosts/kafka.log
Oct 30 05:55:20 localhost anthony[76955]: kafka-test This is a test message to send from rsyslog to kafka
Oct 30 05:55:23 localhost anthony[76955]: kafka-test This is a test message to send from rsyslog to kafka 2

这些事件似乎已排队，直到我重新启动 rsyslog 才会写入错误日志：

$ cat /var/log/remote-hosts/errors_kafka.log
{ "errcode": -152, "errmsg": "Local: Purged in queue", "data": "2023-10-30T05:55:20.506080-04:00 localhost anthony[76955]: kafka-test This is a test message to send from rsyslog to kafka\n" }
{ "errcode": -152, "errmsg": "Local: Purged in queue", "data": "2023-10-30T05:55:23.275050-04:00 localhost anthony[76955]: kafka-test This is a test message to send from rsyslog to kafka 2\n" }

我检查了 rsyslog 日志，发现有一些错误，但在网上找不到关于这些错误的太多信息。显然无法连接到 Kafka。

Oct 30 05:54:43 localhost.localdomain rsyslogd[76655]: imjournal: journal files changed, reloading...  [v8.2310.0 try https://www.rsyslog.com/e/0 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: omkafka: action will suspended due to kafka error -195: Local: Broker transport failure [v8.2310.0 try https://www.rsyslog.com/e/2422 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: omkafka: action will suspended due to kafka error -187: Local: All broker connections are down [v8.2310.0 try https://www.rsyslog.com/e/2422 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: omkafka: action will suspended due to kafka error -195: Local: Broker transport failure [v8.2310.0 try https://www.rsyslog.com/e/2422 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: omkafka: action will suspended due to kafka error -195: Local: Broker transport failure [v8.2310.0 try https://www.rsyslog.com/e/2422 ]
Oct 30 05:55:23 localhost.localdomain rsyslogd[76655]: action 'action-0-omkafka' suspended (module 'omkafka'), retry 0. There should be messages before this one giving the reason for suspension. [v8.2310.0 try https://www.rsyslog.com/e/2007 ]

我没有主意。有人有建议吗？如果需要，很乐意提供更多信息。

提前致谢。

-安东尼