☰

Kafka Broker 通过 docker compose 在本地运行，但在配置几乎相同的 kubernetes 中失败

tag-icon • tag-icon docker kubernetes kafka
Kafka Broker 通过 docker compose 在本地运行，但在配置几乎相同的 kubernetes 中失败

Kafka broker 通过 docker-compose 在本地运行时运行良好。

version: "3"
services:
  zookeeper:
    image: 'bitnami/zookeeper:latest'
    ports:
      - '2181:2181'
    environment:
      - ALLOW_ANONYMOUS_LOGIN=yes
  kafka:
    image: 'bitnami/kafka:latest'
    ports:
      - '9092:9092'
    environment:
      - KAFKA_BROKER_ID=1
      - KAFKA_CFG_LISTENERS=PLAINTEXT://:9092
      - KAFKA_CFG_ADVERTISED_LISTENERS=PLAINTEXT://127.0.0.1:9092
      - KAFKA_CFG_ZOOKEEPER_CONNECT=zookeeper:2181
      - ALLOW_PLAINTEXT_LISTENER=yes
    depends_on:
      - zookeeper

但是当通过 kubernetes 运行时

apiVersion: v1
kind: Service
metadata:
  name: zevrant-kafka-zookeeper-service
spec:
  ports:
    - port: 443
      targetPort: 2181
  selector:
    app: zevrant-kafka-zookeeper-service
---
apiVersion: v1
kind: Service
metadata:
  name: zevrant-kafka-service
spec:
  type: NodePort
  ports:Connection to node -1 (/<IP_ADDRESS>:30129) could not be established. Broker may not be available.
    - port: 443
      targetPort: 9092
      nodePort: 30129
  selector:
    app: zevrant-kafka-service
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zevrant-kafka-zookeeper-service-deployment
  labels:
    app: zevrant-kafka-zookeeper-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zevrant-kafka-zookeeper-service
  template:
    metadata:
      labels:
        app: zevrant-kafka-zookeeper-service
    spec:
      volumes:
        - name: zookeeper-data
          nfs:
            server: <IP_ADDRESS>
            path: /zookeeper
      imagePullSecrets:
        - name: regcred
      nodeSelector:
        architecture: amd64
      containers:
        - name: zevrant-kafka-zookeeper-service
          image: bitnami/zookeeper:3.7.0
          env:
            - name: ALLOW_ANONYMOUS_LOGIN
              value: "yes"
          ports:
            - containerPort: 2181
          volumeMounts:
            - name: zookeeper-data
              mountPath: /bitnami/zookeeper
        - name: zevrant-kafka-zookeper-ui-service
          image: elkozmon/zoonavigator:1.1.0
          ports:
            - containerPort: 9000
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: zevrant-kafka-service-daemonset
  labels:
    app: zevrant-kafka-service
spec:
  replicas: 1
  selector:
    matchLabels:
      app: zevrant-kafka-service
  template:
    metadata:
      labels:
        app: zevrant-kafka-service
    spec:
      volumes:
        - name: kafka-storage
          hostPath:
            path: /opt/kafka/dev/
      imagePullSecrets:
        - name: regcred
      nodeSelector:
        architecture: amd64
      containers:
        - name: zevrant-kafka-broker-service
          image: bitnami/kafka:2.8.0
          volumeMounts:
            - name: kafka-storage
              mountPath: /bitnami/kafka
          ports:
            - containerPort: 9092
          env:
            - name: ALLOW_PLAINTEXT_LISTENER
              value: "yes"
            - name: KAFKA_CFG_ZOOKEEPER_CONNECT
              value: "zevrant-kafka-zookeeper-service:443"
            - name: KAFKA_BROKER_ID
              value: "1"
            - name: KAFKA_CFG_LISTENERS
              value: "PLAINTEXT://0.0.0.0:9092"
            - name: KAFKA_CFG_ADVERTISED_LISTENERS
              value: "PLAINTEXT://<IP_ADDRESS>:30129"

代理启动但反复发送错误

INFO [Admin Manager on Broker 1]: Error processing create topic request CreatableTopic(name='zevrant-video-stream', numPartitions=1, replicationFactor=1, assignments=[], configs=[]) (kafka.server.ZkAdminManager)

除了连接详细信息之外，我在两个集群上都使用相同的命令来创建主题，并且在两种情况下，我都从正在运行的代理容器内部运行该命令。

本地

kafka-topics.sh --create --topic zevrant-video-stream --zookeeper localhost:2181 --partitions 1 --replication-factor 1 --config retention.ms=86400000

Kubernetes

kafka-topics.sh --create --topic zevrant-video-stream --zookeeper zevrant-kafka-zookeeper-service:443 --partitions 1 --replication-factor 1 --config retention.ms=86400000

我不太清楚为什么这在一个地方有效，而在其他地方无效。

尝试将生产者连接到我收到的 kubernetes 代理

Connection to node -1 (/<IP_ADDRESS>:30129) could not be established. Broker may not be available.

当向 zookeeper 查询 broker 信息时，zookeeper 显示该 broker 已连接

zkCli.sh get /brokers/ids/1


{"listener_security_protocol_map":{"PLAINTEXT":"PLAINTEXT"},"endpoints":["PLAINTEXT://<IP_ADDRESS>:9092"],"jmx_port":-1,"features":{},"host":"<IP_ADDRESS>","timestamp":"1622398259745","port":9092,"version":5}

什么原因会导致代理已连接、在线，但不可用？

答案1

事实证明，正如所讨论的那样，在封闭网络内运行需要额外的配置这里

相关内容