Redis一主一从Docker方式部署通过keepalived和HAVIP实现高可用

一 主从高可用要求

1 主节点挂掉之后，从节点能在秒级内完成晋升为主
2 挂掉的节点恢复后，能追加为从节点
3 升主、恢复、挂掉的时候都会有日志和钉钉机器人消息通知，确保及时感知到redis的状态，尽早人工干预恢复。ke’y# 二 机器情况
Centos7.2的2核8G的机器

机器	redis的docker挂载目录	keepalived脚本目录	日志位置
master 10.0.32.72	/data/redis/data，/data/redis/redis.conf, /data/redis/logs	/etc/keepalived/scripts	/var/log/keepalived-redis*
slave 10.0.32.73	/data/redis/data，/data/redis/redis.conf, /data/redis/logs	/etc/keepalived/scripts	/var/log/keepalived-redis*

三 redis.conf配置

两台机器都创建目录

mkdir -p /data/redis/data
mkdir -p /data/redis/logs
chmod 777 /data/redis/data
chmod 777 /data/redis/logs

master 10.0.32.72 /data/redis/redis.conf

cat > /data/redis/redis.conf << EOF
bind 0.0.0.0
daemonize no
port 6379
tcp-backlog 511
timeout 300
tcp-keepalive 60
databases 16
 
save 900 1
save 300 10
save 60 10000
 
dbfilename dump.rdb
 
loglevel notice
logfile "/var/log/redis-server.log"
dir "/data"
appendonly yes
appendfilename appendonly.aof
maxmemory-policy volatile-lru
maxmemory 6442450944
appendfsync everysec
 
requirepass 【redis密码】
masterauth 【redis密码】
 
 
EOF

slave 10.0.32.73 /data/redis/redis.conf

cat > /data/redis/redis.conf << EOF
bind 0.0.0.0
daemonize no
port 6379
tcp-backlog 511
timeout 300
tcp-keepalive 60
databases 16
 
save 900 1
save 300 10
save 60 10000
 
dbfilename dump.rdb
 
loglevel notice
logfile "/var/log/redis-server.log"
dir "/data"
appendonly yes
appendfilename appendonly.aof
maxmemory-policy volatile-lru
maxmemory 6442450944
appendfsync everysec
 
requirepass 【redis密码】
masterauth 【redis密码】
 
slaveof 10.0.32.72 6379
EOF

四 Docker启动redis

主从启动命令一致如下：

docker run -dit --name redis --restart=always -v /data/redis/redis.conf:/etc/redis.conf -v /data/redis/data/:/data/ -v /data/redis/logs:/var/log -p 6379:6379 redis:4.0.1 redis-server /etc/redis.conf

启动redis之后copy redis-cli 到宿主机的 /usr/local/bin/目录下

docker cp redis:/usr/local/bin/redis-cli /usr/local/bin/

五 Keepalived配置

master 10.0.32.72 /etc/keepalived/keepalived.conf

! Configuration File for keepalived
 
global_defs {
   router_id redis100
}
vrrp_script chk_redis
{
    script "/etc/keepalived/scripts/redis_check.sh 127.0.0.1 6379 econ202101"
    interval 2
    timeout 2
    fall 3
}
vrrp_instance redis {
    state MASTER # master set to SLAVE also
    interface eth0  ## 修改为实际网卡名称
    virtual_router_id 50
    priority  150
    nopreempt # no seize,must add
    advert_int 1
    authentication {   #all node must same
       auth_type PASS
       auth_pass redis1111
    }
    virtual_ipaddress { 
      10.0.32.74 dev eth0 label eth0:havip   ## 修改为实际分配的虚拟IP地址 10.0.32.74 为阿里专有云分配的havip
    }
    unicast_src_ip 10.0.32.72
    unicast_peer {
        10.0.32.73
    }
    track_script {
       chk_redis
    }
    notify_master "/etc/keepalived/scripts/redis_master.sh 127.0.0.1 10.0.32.73 6379 econ202101"     ## 10.0.32.73应修改为实际的slave IP地址
    notify_backup "/etc/keepalived/scripts/redis_backup.sh 127.0.0.1 10.0.32.73 6379 econ202101"     ## 10.0.32.73应修改为实际的slave IP地址
    notify_fault /etc/keepalived/scripts/redis_fault.sh
    notify_stop /etc/keepalived/scripts/redis_stop.sh
}

slave 10.0.32.73 /etc/keepalived/keepalived.conf

! Configuration File for keepalived
 
global_defs {
   router_id redis100
}
vrrp_script chk_redis
{
    script "/etc/keepalived/scripts/redis_check.sh 127.0.0.1 6379 econ202101"
    interval 2
    timeout 2
    fall 3
}
vrrp_instance redis {
    state BACKUP # master set to SLAVE also
    interface eth0
    virtual_router_id 50
    priority  100
    nopreempt # no seize,must add
    advert_int 1
    authentication {   #all node must same
       auth_type PASS
       auth_pass redis1111
    }
    virtual_ipaddress { 
      10.0.32.74 dev eth0 label eth0:havip   ## 修改为实际分配的虚拟IP地址 10.0.32.74 为阿里专有云分配的havip
    }
    unicast_src_ip 10.0.32.73
    unicast_peer {
        10.0.32.72
    }
    track_script {
       chk_redis
    }
    notify_master "/etc/keepalived/scripts/redis_master.sh 127.0.0.1 10.0.32.72 6379 econ202101"     ## 10.0.32.72应修改为实际的master IP地址
    notify_backup "/etc/keepalived/scripts/redis_backup.sh 127.0.0.1 10.0.32.72 6379 econ202101"     ## 10.0.32.72应修改为实际的master IP地址
    notify_fault /etc/keepalived/scripts/redis_fault.sh
    notify_stop /etc/keepalived/scripts/redis_stop.sh
}

keepalived的脚本主备都是一致的

/etc/keepalived/scripts/redis_check.sh

#!/bin/bash
ALIVE=`/usr/local/bin/redis-cli -h $1 -p $2 -a $3 PING`
LOGFILE="/var/log/keepalived-redis-check.log"
echo "[CHECK]" >> $LOGFILE
date >> $LOGFILE
if [ "$ALIVE" == "PONG" ]; then :
    echo "Success: redis-cli -h $1 -p $2 PING $ALIVE" >> $LOGFILE 2>&1
    exit 0
else
    echo "Failed: redis-cli -h $1 -p $2 PING $ALIVE" >> $LOGFILE 2>&1
    exit 1
fi

/etc/keepalived/scripts/redis_master.sh

#!/bin/bash
REDISCLI="/usr/local/bin/redis-cli -h $1 -p $3 -a $4"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[master]" >> $LOGFILE
date >> $LOGFILE
echo "Being master...." >> $LOGFILE
echo "Run MASTER cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF $2 $3 >> $LOGFILE
sleep 10 #delay 10 s wait data async cancel sync
echo "Run SLAVEOF NO ONE cmd ..." >> $LOGFILE
$REDISCLI SLAVEOF NO ONE >> $LOGFILE
echo "send dingding msg ..." >> $LOGFILE
curl -s -XPOST -H 'Content-Type: application/json' http://10.0.42.22:9999/robot/send?access_token=69824bb93b6357d3ff36e9fb484ef8f40926eac02d96031faa7770e62bf09cd4 -d "{\"msgtype\":\"markdown\",\"markdown\":{\"title\":\"生产应用redis告警\",\"text\":\"生产应用redis已自动切换主从 \\n\\n>告警机器: <font color=\\\"comment\\\">10.0.32.72,10.0.32.73</font> \\n\\n>人工干预: <font color=\\\"comment\\\">请检查是否切换成功并人工干预恢复故障节点</font>\"}}" >> $LOGFILE

/etc/keepalived/scripts/redis_backup.sh

#!/bin/bash
REDISCLI="/usr/local/bin/redis-cli -a $4"
LOGFILE="/var/log/keepalived-redis-state.log"
echo "[BACKUP]" >> $LOGFILE
date >> $LOGFILE
echo "Being slave...." >> $LOGFILE 2>&1
echo "Run SLAVEOF cmd ..." >> $LOGFILE 2>&1
$REDISCLI SLAVEOF $2 $3 >> $LOGFILE 
sleep 10 #delay 10 s wait data async cancel sync
echo "send dingding msg ..." >> $LOGFILE
curl -s -XPOST -H 'Content-Type: application/json' http://10.0.42.22:9999/robot/send?access_token=69824bb93b6357d3ff36e9fb484ef8f40926eac02d96031faa7770e62bf09cd4 -d "{\"msgtype\":\"markdown\",\"markdown\":{\"title\":\"生产应用redis告警\",\"text\":\"生产应用redis节点已恢复并追加为slave节点 \\n\\n>告警机器: <font color=\\\"comment\\\">10.0.32.72,10.0.32.73</font> \\n\\n>人工干预: <font color=\\\"comment\\\">请检查是否成功恢复主从</font>\"}}" >> $LOGFILE

/etc/keepalived/scripts/redis_fault.sh

#!/bin/bash
LOGFILE=/var/log/keepalived-redis-state.log
echo "[fault]" >> $LOGFILE
date >> $LOGFILE
echo "send dingding msg ..." >> $LOGFILE
curl -s -XPOST -H 'Content-Type: application/json' http://10.0.42.22:9999/robot/send?access_token=69824bb93b6357d3ff36e9fb484ef8f40926eac02d96031faa7770e62bf09cd4 -d "{\"msgtype\":\"markdown\",\"markdown\":{\"title\":\"生产应用redis告警\",\"text\":\"生产应用redis有节点发生故障 \\n\\n>告警机器: <font color=\\\"comment\\\">10.0.32.72,10.0.32.73</font> \\n\\n>人工干预: <font color=\\\"comment\\\">请检查并恢复故障节点</font>\"}}" >> $LOGFILE

/etc/keepalived/scripts/redis_stop.sh

#!/bin/bash
LOGFILE=/var/log/keepalived-redis-state.log
echo "[stop]" >> $LOGFILE
date >> $LOGFILE

六 启动顺序

1. 先启动master 10.0.32.72 的 redis
2. 再启动slave 10.0.32.73 的redis
3. 启动master 10.0.32.72 的 keepalived
4. 启动slave 10.0.32.73 的 keepalived

七 检查

1 两台机器 redis-cli 进入后执行 info Replication 查看主从信息
2 人为操作主节点挂掉，检查从节点是否会晋升为主节点，反之亦然。