docker-compose 上 Redis Sentinel 的故障转移问题

Question

我想在 docker 上为 Redis Sentinel 准备基本设置。一切看起来都很好，复制工作正常，但是故障转移不起作用。为了测试它，我创建了一些简单的 python 应用程序并从 docker compose 中杀死 redis-master。我尝试找到一些解决方案，但可能遇到的一个问题是https://redis.io/docs/management/sentinel/#sentinel-docker-nat-and-possible-issues。我尝试使用容器 IP 或不同的公告选项，但仍然不确定如何解决它。我尝试通过简单地暂停容器来测试故障转移docker-compose pause redis-master。Sentinel 检测到主服务器已关闭sdown，但此后没有选举新的主服务器。

这是我的应用程序树：

.
├── app
│   ├── Dockerfile
│   └── redis_app.py
├── compose.yaml
└── sentinel
    ├── Dockerfile
    ├── sentinel.conf
    └── sentinel-entrypoint.sh

compose.yml

version: '3.2'

services:
    redis-master:
        image: redis:7.0.5
        command: redis-server
        ports:
          - 16379:6379

    redis-slave-1:
        image: redis:7.0.5
        command: redis-server --slaveof redis-master 6379
        ports:
          - 26379:6379
        links:
          - redis-master

    redis-slave-2:
        image: redis:7.0.5
        command: redis-server --slaveof redis-master 6379
        ports:
          - 36379:6379
        links:
          - redis-master

    sentinel-1:
        build: sentinel
        environment:
          - SENTINEL_DOWN_AFTER=5000
          - SENTINEL_FAILOVER=500
          - SENTINEL_QUORUM=2
        depends_on:
          - redis-master
          - redis-slave-1
          - redis-slave-2
        links:
          - redis-master

    redis_cluster_app:
        build: app
        environment:
          - SENTINEL_DOWN_AFTER=5000
          - SENTINEL_FAILOVER=500
          - SENTINEL_QUORUM=2
        depends_on:
          - redis-master
          - redis-slave-1
          - redis-slave-2
        command: /bin/bash -c "echo 'Waiting for redis to run..' && sleep 60 && python redis_app.py"
        volumes:
          - ./app:/usr/src/app/redis_app/

sentinel/Dockerfile

FROM redis:7.0.5

EXPOSE 26379

ADD sentinel.conf /etc/redis/sentinel.conf

RUN chown redis:redis /etc/redis/sentinel.conf

COPY sentinel-entrypoint.sh /usr/local/bin/

RUN chmod +x /usr/local/bin/sentinel-entrypoint.sh

ENTRYPOINT ["sentinel-entrypoint.sh"]

sentinel/sentinel.conf

port 26379

dir /tmp

sentinel resolve-hostnames yes
sentinel monitor mymaster redis-master 6379 $SENTINEL_QUORUM
sentinel down-after-milliseconds mymaster $SENTINEL_DOWN_AFTER
sentinel parallel-syncs mymaster 1
sentinel failover-timeout mymaster $SENTINEL_FAILOVER
sentinel announce-port 26379

sentinel/sentinel-entrypoint.sh

#!/bin/sh

sed -i "s/\$SENTINEL_QUORUM/$SENTINEL_QUORUM/g" /etc/redis/sentinel.conf
sed -i "s/\$SENTINEL_DOWN_AFTER/$SENTINEL_DOWN_AFTER/g" /etc/redis/sentinel.conf
sed -i "s/\$SENTINEL_FAILOVER/$SENTINEL_FAILOVER/g" /etc/redis/sentinel.conf

应用程序/Dockerfile

# basic python image
FROM python:3.11

# install redis to access redis APIs
RUN pip install redis

# Without this setting, Python never prints anything out.
ENV PYTHONUNBUFFERED=1

# declare the source directory
WORKDIR /usr/src/app/

# copy the file
COPY redis_app.py .

# start command
CMD [ "python", "redis_app.py" ]

应用程序/redis_app.py

import time
from redis import RedisError, sentinel, ReadOnlyError
import sys

ERROR_KEY_NOT_FOUND = "Key not found in redis"


class RedisDriver:
    def __init__(self, redis_config):
        self.service = redis_config["service_name"]
        self.__connect(redis_config)

    def __connect(self, redis_config):
        self.connection = sentinel.Sentinel(
            [
                (redis_config["sentinel_host"], redis_config["sentinel_port"]),
            ],
            socket_timeout=0.5,
        )
        # print(self.connection.master_for("mymaster"))
        # print(self.connection.discover_slaves("mymaster"))

    def set(self, key, value):
        key_str = str(key)
        val_str = str(value)
        try:
            master = self.connection.master_for(self.service)
            master.set(key_str, val_str)
            return {"success": True}
        except RedisError as err:
            error_str = "Error while connecting to redis : " + str(err)
            return {"success": False, "error": error_str}

    def get(self, key):
        key_str = str(key)
        try:
            master = self.connection.master_for(self.service)
            value = master.get(key_str)
        except RedisError as err:
            error_str = "Error while retrieving value from redis : " + str(err)
            return {"success": False, "error": error_str}

        if value is not None:
            return {"success": True, "value": value}
        else:
            return {"success": False, "error": ERROR_KEY_NOT_FOUND}

    def delete(self, key):
        key_str = str(key)
        try:
            master = self.connection.master_for(self.service)
            value = master.delete(key_str)
        except RedisError as err:
            error_str = "Error while deleting key from redis : " + str(err)
            return {"success": False, "error": error_str}

        return {"success": True}


if __name__ == "__main__":
    print("*" * 75)
    redis_config = {
        "service_name": "mymaster",
        "sentinel_host": "sentinel-1",
        "sentinel_port": 26379,
    }

    redis_driver = RedisDriver(redis_config)
    while True:
        result = redis_driver.set("hello", "world")
        print(result)

        if result["success"]:
            result = redis_driver.get("hello")
            print(result)

        slave = redis_driver.connection.slave_for(redis_driver.service)
        try:
            slave.set("slave", "slave")
        except ReadOnlyError:
            print("Slave is readonly")
        print("******** SLEEPING *********")
        time.sleep(10)

        print("******** AGAIN *********")
        result = redis_driver.set("hello2", "world2")
        print(result)

        if result["success"]:
            result = redis_driver.get("hello2")
            print(result)
        redis_driver.delete("hello")
        time.sleep(10)

当我运行时，docker-compose up --build一切似乎都运行良好，在无限循环中我不断修改密钥，然后在另一个终端中运行docker-compose pause redis-master但故障转移不起作用。

一些日志，sdwon master sentinel 没有选择新的 master